Compare commits
1
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
 panxiao81
|
0e3564bf72
|
@@ -1,157 +0,0 @@
|
|||||||
# Nexus Repository POC
|
|
||||||
|
|
||||||
本目录声明一个 Nexus Repository Community Edition POC，用来验证一次性 CI runner 通过
|
|
||||||
网络服务复用 Ansible Galaxy、Go Modules 与 OCI/BuildKit 缓存。Nexus 固定为 `3.96.1`，
|
|
||||||
镜像固定到官方 multi-arch index digest；LAN 入口为
|
|
||||||
`https://nexus.ad.ddupan.top`。
|
|
||||||
|
|
||||||
## POC 边界
|
|
||||||
|
|
||||||
- 单副本 Deployment，`Recreate` 更新，避免一个 RWO 卷被两个 Pod 同时挂载。
|
|
||||||
- `/nexus-data` 使用 `localpv-zfs-ceph` 上的 50 GiB RWO PVC。
|
|
||||||
- 资源预算为 250m/2 GiB request、2 CPU/4 GiB limit；JVM heap 上限 2 GiB。
|
|
||||||
- 使用容器默认的 embedded H2。它只用于 POC；正式接管 OCI 制品前必须迁移到外部
|
|
||||||
PostgreSQL，并验证备份恢复。
|
|
||||||
- 入口只在 LAN wildcard Gateway 上发布，不创建公网 DNS 或 Cloudflare route。
|
|
||||||
- 不套 Authelia forward-auth；它会破坏 Go、Ansible 与 OCI 非浏览器客户端。
|
|
||||||
- 现有 zot 保持不变。Nexus 完成 OCI、BuildKit cache 和恢复验收前不得迁移或删除 zot。
|
|
||||||
|
|
||||||
Terraform provider 创建 `ansible-public`、`go-public`、最小匿名权限与 OCI Bearer Token
|
|
||||||
Realm。Nexus 3.94 才加入的原生 OCI repository 已有 REST API，但当前锁定的 community
|
|
||||||
provider 尚未暴露 OCI resource；`terraform/reconcile-oci.sh` 因此根据 3.96.1 实例 Swagger
|
|
||||||
固定的 JSON schema，幂等调和 `oci-hosted`、`oci-proxy` 与 `oci-public`。不得绕过该入口在
|
|
||||||
UI 中创建无人管理的长期 repository。
|
|
||||||
|
|
||||||
## 部署
|
|
||||||
|
|
||||||
Flux 从 `clusters/homelab/apps/nexus.yaml` 协调本目录，并依赖 Envoy Gateway 与 OpenEBS。
|
|
||||||
合并前只渲染配置，不直接 apply：
|
|
||||||
|
|
||||||
```bash
|
|
||||||
kubectl kustomize apps/nexus
|
|
||||||
```
|
|
||||||
|
|
||||||
合并并由 Flux 部署后检查：
|
|
||||||
|
|
||||||
```bash
|
|
||||||
kubectl -n flux-system get kustomization nexus
|
|
||||||
kubectl -n nexus get pod,pvc,service,httproute
|
|
||||||
kubectl -n nexus logs deployment/nexus --tail=100
|
|
||||||
```
|
|
||||||
|
|
||||||
启动可能需要数分钟，startup probe 允许最多十分钟。不要因初次启动较慢反复删除 Pod；
|
|
||||||
先确认 PVC 已 Bound、Pod 没有 OOM，以及日志仍在推进。
|
|
||||||
|
|
||||||
DNS 期望状态已加入 `infrastructure/dns/records.yml`，需从 Samba AD Ansible root 以
|
|
||||||
`--check --diff` 核对后再按其 README 应用 DNS tag。没有 DNS 时可先用 port-forward
|
|
||||||
验证应用，但不能据此宣称 Gateway 路径已通过。
|
|
||||||
|
|
||||||
## 首次初始化与 Terraform
|
|
||||||
|
|
||||||
初始管理员密码生成在 PVC 的 `/nexus-data/admin.password`。只在交互式终端中读取并立即
|
|
||||||
完成首次密码轮换；不得把密码复制进 shell tracing、工单、Git 或命令参数。随后将
|
|
||||||
Terraform 管理账号的凭据存入 OpenBao，由 CI 通过 Terraform input variable 注入以下
|
|
||||||
环境变量：
|
|
||||||
|
|
||||||
```text
|
|
||||||
TF_VAR_nexus_url=https://nexus.ad.ddupan.top
|
|
||||||
TF_VAR_nexus_username=admin
|
|
||||||
TF_VAR_nexus_password=<OpenBao kv/infra/nexus 的 admin_password 字段>
|
|
||||||
```
|
|
||||||
|
|
||||||
`terraform/` 使用 `sonatype-nexus-community/sonatyperepo` 1.17.0，当前声明：
|
|
||||||
|
|
||||||
- `ansible-galaxy-proxy` → `https://galaxy.ansible.com`
|
|
||||||
- `ansible-public` group
|
|
||||||
- `go-proxy` → `https://proxy.golang.org`
|
|
||||||
- `go-public` group
|
|
||||||
|
|
||||||
provider credential 不写入 HCL 或 tfvars。正式 apply 前还必须为这个独立 Terraform root
|
|
||||||
配置远端 backend；本地 state 只允许用于可丢弃的 POC，不提交。验证命令：
|
|
||||||
|
|
||||||
```bash
|
|
||||||
terraform -chdir=apps/nexus/terraform init -backend=false
|
|
||||||
terraform -chdir=apps/nexus/terraform validate
|
|
||||||
```
|
|
||||||
|
|
||||||
先以 `--check` 查看 OCI repository 漂移，再明确 apply；脚本只从上述环境变量取得凭据，
|
|
||||||
用临时 `0600` netrc 调用 REST API，退出时删除：
|
|
||||||
|
|
||||||
```bash
|
|
||||||
apps/nexus/terraform/reconcile-oci.sh --check
|
|
||||||
apps/nexus/terraform/reconcile-oci.sh --apply
|
|
||||||
```
|
|
||||||
|
|
||||||
Terraform 同时把内置 `anonymous` 用户从默认的全仓库 `nx-anonymous` 角色收窄到
|
|
||||||
`ansible-public`、其返回制品 URL 使用的 `ansible-galaxy-proxy`、`go-public`，以及
|
|
||||||
`oci-public`/`oci-proxy` 的 `browse/read` 权限。`oci-hosted` 不向匿名用户开放。首次接管
|
|
||||||
已有实例时先执行
|
|
||||||
`terraform import sonatyperepo_user.anonymous anonymous,default`，再 apply；不要先启用默认的
|
|
||||||
全仓库匿名读取。
|
|
||||||
|
|
||||||
2026-09-20 的 POC 现场验收已确认：Flux 与 Pod Ready、PVC Bound、HTTPRoute 通过 HTTPS
|
|
||||||
返回 Nexus 状态 200，Samba DNS 已幂等收敛。全新客户端目录通过匿名入口下载
|
|
||||||
`community.general:11.2.0` 时冷缓存为 8.49 秒、热缓存为 1.89 秒，两次 tarball SHA-256
|
|
||||||
一致；`golang.org/x/[email protected]` 为 2.92 秒与 1.51 秒。
|
|
||||||
|
|
||||||
## 客户端验收
|
|
||||||
|
|
||||||
先验证冷缓存，再原样重复命令验证热缓存；记录 Nexus 请求、上游流量和耗时，不只观察
|
|
||||||
命令成功。Ansible 配置的 URL 必须以 `/` 结尾：
|
|
||||||
|
|
||||||
```ini
|
|
||||||
[galaxy]
|
|
||||||
server_list = nexus
|
|
||||||
|
|
||||||
[galaxy_server.nexus]
|
|
||||||
url = https://nexus.ad.ddupan.top/repository/ansible-public/
|
|
||||||
```
|
|
||||||
|
|
||||||
```bash
|
|
||||||
ansible-galaxy collection install -r collections/requirements.yml \
|
|
||||||
-p .ansible/collections
|
|
||||||
```
|
|
||||||
|
|
||||||
Go POC 使用：
|
|
||||||
|
|
||||||
```bash
|
|
||||||
GOPROXY=https://nexus.ad.ddupan.top/repository/go-public/ go mod download
|
|
||||||
```
|
|
||||||
|
|
||||||
私有 `git.ddupan.top/*` module 的 `GOPRIVATE`、凭据与是否允许 `direct` fallback 在实际
|
|
||||||
workflow 中单独决定；不要让私有 module path 意外发往公共 proxy。
|
|
||||||
|
|
||||||
OCI 使用 path-based routing：匿名公共拉取地址形如
|
|
||||||
`nexus.ad.ddupan.top/oci-public/library/alpine:3.22`，认证写入地址形如
|
|
||||||
`nexus.ad.ddupan.top/oci-hosted/<namespace>/<image>:<tag>`。2026-09-20 现场验收结果：
|
|
||||||
|
|
||||||
- `oci-public` 匿名代理拉取 Alpine 冷缓存 4.75 秒、热缓存 0.80 秒，digest 一致；
|
|
||||||
- `oci-hosted` 认证 push/pull 成功，匿名 pull 返回 401；
|
|
||||||
- amd64/arm64 OCI image index push 成功，两个平台 manifest 可见；
|
|
||||||
- Helm chart push/pull digest 与本地 tarball SHA-256 一致；
|
|
||||||
- Cosign 3.1.3 使用一次性密钥签名并验证成功，OCI 1.1 referrers API 返回一个
|
|
||||||
`application/vnd.dev.sigstore.bundle.v0.3+json` artifact；
|
|
||||||
- BuildKit `registry` cache 以 `mode=max` 导出成功，销毁首个 builder 后由新 builder 导入，
|
|
||||||
两个 `RUN` step 均明确命中 `CACHED`。
|
|
||||||
|
|
||||||
本机安装的测试客户端包括 `/usr/local/bin/cosign` 3.1.3；安装时核对官方 Linux amd64
|
|
||||||
binary SHA-256 `4629c757b7618056f8ddd7e2625ae9fdd94c0372a65049520bc7d9df9efc7f71`。
|
|
||||||
上述结果仍不代表备份恢复、外部 PostgreSQL 或正式 publisher service account 已完成；
|
|
||||||
这些项目通过前不得迁移或删除 zot。
|
|
||||||
|
|
||||||
## 数据与恢复
|
|
||||||
|
|
||||||
POC 的数据库、配置、blob、初始管理员状态都位于 `nexus-data` PVC。删除 Deployment
|
|
||||||
不会删除 PVC；删除 PVC 会永久删除整个 POC。当前没有独立备份，不能将它用于唯一副本的
|
|
||||||
正式制品。
|
|
||||||
|
|
||||||
恢复验证至少包括：停止写入、取得一致备份、在独立 PVC/实例恢复、登录、列出 repository、
|
|
||||||
拉取已缓存的 Ansible/Go 制品，并核对 OCI digest/referrers。正式化时再把数据库迁移至
|
|
||||||
外部 PostgreSQL，并分别定义数据库与 blob 的备份、恢复顺序和 RPO。
|
|
||||||
|
|
||||||
参考：
|
|
||||||
|
|
||||||
- [Nexus OCI repositories](https://help.sonatype.com/en/oci-repositories.html)
|
|
||||||
- [Nexus Ansible repositories](https://help.sonatype.com/en/ansible-repositories.html)
|
|
||||||
- [Nexus Go repositories](https://help.sonatype.com/en/go-repositories.html)
|
|
||||||
- [官方容器镜像](https://hub.docker.com/r/sonatype/nexus3)
|
|
||||||
@@ -1,82 +0,0 @@
|
|||||||
apiVersion: apps/v1
|
|
||||||
kind: Deployment
|
|
||||||
metadata:
|
|
||||||
name: nexus
|
|
||||||
namespace: nexus
|
|
||||||
labels:
|
|
||||||
app.kubernetes.io/name: nexus
|
|
||||||
spec:
|
|
||||||
replicas: 1
|
|
||||||
strategy:
|
|
||||||
type: Recreate
|
|
||||||
selector:
|
|
||||||
matchLabels:
|
|
||||||
app.kubernetes.io/name: nexus
|
|
||||||
template:
|
|
||||||
metadata:
|
|
||||||
labels:
|
|
||||||
app.kubernetes.io/name: nexus
|
|
||||||
spec:
|
|
||||||
automountServiceAccountToken: false
|
|
||||||
securityContext:
|
|
||||||
fsGroup: 200
|
|
||||||
fsGroupChangePolicy: OnRootMismatch
|
|
||||||
runAsGroup: 200
|
|
||||||
runAsNonRoot: true
|
|
||||||
runAsUser: 200
|
|
||||||
seccompProfile:
|
|
||||||
type: RuntimeDefault
|
|
||||||
terminationGracePeriodSeconds: 120
|
|
||||||
containers:
|
|
||||||
- name: nexus
|
|
||||||
image: docker.io/sonatype/nexus3:3.96.1@sha256:56142f13432cf072e017aebb2025f201e42ae36ff40bb82618c702504c61f7dd
|
|
||||||
imagePullPolicy: IfNotPresent
|
|
||||||
env:
|
|
||||||
- name: INSTALL4J_ADD_VM_PARAMS
|
|
||||||
value: >-
|
|
||||||
-Xms1024m -Xmx2048m -XX:MaxDirectMemorySize=1024m
|
|
||||||
-Djava.util.prefs.userRoot=/nexus-data/javaprefs
|
|
||||||
ports:
|
|
||||||
- name: http
|
|
||||||
containerPort: 8081
|
|
||||||
protocol: TCP
|
|
||||||
resources:
|
|
||||||
requests:
|
|
||||||
cpu: 250m
|
|
||||||
memory: 2Gi
|
|
||||||
limits:
|
|
||||||
cpu: "2"
|
|
||||||
memory: 4Gi
|
|
||||||
securityContext:
|
|
||||||
allowPrivilegeEscalation: false
|
|
||||||
capabilities:
|
|
||||||
drop:
|
|
||||||
- ALL
|
|
||||||
startupProbe:
|
|
||||||
httpGet:
|
|
||||||
path: /service/rest/v1/status
|
|
||||||
port: http
|
|
||||||
failureThreshold: 60
|
|
||||||
periodSeconds: 10
|
|
||||||
timeoutSeconds: 5
|
|
||||||
readinessProbe:
|
|
||||||
httpGet:
|
|
||||||
path: /service/rest/v1/status
|
|
||||||
port: http
|
|
||||||
failureThreshold: 6
|
|
||||||
periodSeconds: 10
|
|
||||||
timeoutSeconds: 5
|
|
||||||
livenessProbe:
|
|
||||||
httpGet:
|
|
||||||
path: /service/rest/v1/status
|
|
||||||
port: http
|
|
||||||
failureThreshold: 6
|
|
||||||
periodSeconds: 30
|
|
||||||
timeoutSeconds: 5
|
|
||||||
volumeMounts:
|
|
||||||
- name: data
|
|
||||||
mountPath: /nexus-data
|
|
||||||
volumes:
|
|
||||||
- name: data
|
|
||||||
persistentVolumeClaim:
|
|
||||||
claimName: nexus-data
|
|
||||||
@@ -1,16 +0,0 @@
|
|||||||
apiVersion: gateway.networking.k8s.io/v1
|
|
||||||
kind: HTTPRoute
|
|
||||||
metadata:
|
|
||||||
name: nexus
|
|
||||||
namespace: nexus
|
|
||||||
spec:
|
|
||||||
parentRefs:
|
|
||||||
- name: eg
|
|
||||||
namespace: envoy-gateway-system
|
|
||||||
sectionName: https
|
|
||||||
hostnames:
|
|
||||||
- nexus.ad.ddupan.top
|
|
||||||
rules:
|
|
||||||
- backendRefs:
|
|
||||||
- name: nexus
|
|
||||||
port: 8081
|
|
||||||
@@ -1,9 +0,0 @@
|
|||||||
apiVersion: kustomize.config.k8s.io/v1beta1
|
|
||||||
kind: Kustomization
|
|
||||||
resources:
|
|
||||||
- namespace.yaml
|
|
||||||
- pvc.yaml
|
|
||||||
- deployment.yaml
|
|
||||||
- service.yaml
|
|
||||||
- httproute.yaml
|
|
||||||
- networkpolicy.yaml
|
|
||||||
@@ -1,8 +0,0 @@
|
|||||||
apiVersion: v1
|
|
||||||
kind: Namespace
|
|
||||||
metadata:
|
|
||||||
name: nexus
|
|
||||||
labels:
|
|
||||||
pod-security.kubernetes.io/enforce: restricted
|
|
||||||
pod-security.kubernetes.io/audit: restricted
|
|
||||||
pod-security.kubernetes.io/warn: restricted
|
|
||||||
@@ -1,23 +0,0 @@
|
|||||||
apiVersion: networking.k8s.io/v1
|
|
||||||
kind: NetworkPolicy
|
|
||||||
metadata:
|
|
||||||
name: nexus-ingress
|
|
||||||
namespace: nexus
|
|
||||||
spec:
|
|
||||||
podSelector:
|
|
||||||
matchLabels:
|
|
||||||
app.kubernetes.io/name: nexus
|
|
||||||
policyTypes:
|
|
||||||
- Ingress
|
|
||||||
ingress:
|
|
||||||
- from:
|
|
||||||
- namespaceSelector:
|
|
||||||
matchLabels:
|
|
||||||
kubernetes.io/metadata.name: envoy-gateway-system
|
|
||||||
podSelector:
|
|
||||||
matchLabels:
|
|
||||||
gateway.envoyproxy.io/owning-gateway-name: eg
|
|
||||||
gateway.envoyproxy.io/owning-gateway-namespace: envoy-gateway-system
|
|
||||||
ports:
|
|
||||||
- protocol: TCP
|
|
||||||
port: 8081
|
|
||||||
@@ -1,12 +0,0 @@
|
|||||||
apiVersion: v1
|
|
||||||
kind: PersistentVolumeClaim
|
|
||||||
metadata:
|
|
||||||
name: nexus-data
|
|
||||||
namespace: nexus
|
|
||||||
spec:
|
|
||||||
accessModes:
|
|
||||||
- ReadWriteOnce
|
|
||||||
resources:
|
|
||||||
requests:
|
|
||||||
storage: 50Gi
|
|
||||||
storageClassName: localpv-zfs-ceph
|
|
||||||
@@ -1,14 +0,0 @@
|
|||||||
apiVersion: v1
|
|
||||||
kind: Service
|
|
||||||
metadata:
|
|
||||||
name: nexus
|
|
||||||
namespace: nexus
|
|
||||||
spec:
|
|
||||||
type: ClusterIP
|
|
||||||
selector:
|
|
||||||
app.kubernetes.io/name: nexus
|
|
||||||
ports:
|
|
||||||
- name: http
|
|
||||||
port: 8081
|
|
||||||
protocol: TCP
|
|
||||||
targetPort: http
|
|
||||||
@@ -1,6 +0,0 @@
|
|||||||
.terraform/
|
|
||||||
*.tfstate
|
|
||||||
*.tfstate.*
|
|
||||||
*.tfplan
|
|
||||||
crash.log
|
|
||||||
crash.*.log
|
|
||||||
Generated
-24
@@ -1,24 +0,0 @@
|
|||||||
# This file is maintained automatically by "terraform init".
|
|
||||||
# Manual edits may be lost in future updates.
|
|
||||||
|
|
||||||
provider "registry.terraform.io/sonatype-nexus-community/sonatyperepo" {
|
|
||||||
version = "1.17.0"
|
|
||||||
constraints = "1.17.0"
|
|
||||||
hashes = [
|
|
||||||
"h1:uKhvhhhI7B+HBsh0zq/ybqKt+EnOGyI6rjcRCtj79ZA=",
|
|
||||||
"zh:0dde99e7b343fa01f8eefc378171fb8621bedb20f59157d6cc8e3d46c738105f",
|
|
||||||
"zh:3315929df254a3a6ac27c8c846c2006f7d2a91fadc014351bc4d617f948e5bf9",
|
|
||||||
"zh:36be5a455af3ce4e187de26753de63e78c1ee9a32dba0135c6cf96a6c1fff25f",
|
|
||||||
"zh:3f73f7ff57b8c339a7c7ac37653e2dc0b2dd9dcc3f3a538788e7e3ac838337b2",
|
|
||||||
"zh:40286ecca4c22ab7ae90618ac6d2743f5055199dac81cf5204a4a397c784d439",
|
|
||||||
"zh:4d24e5c0195fb3155b1967583ee64cfeda402d7cc7f3c73369438f6c69f4245b",
|
|
||||||
"zh:828a9d7aceaac36af7f9c07af43ec8d20a89148780645d170ffb1c68b2da792d",
|
|
||||||
"zh:a5ab04de3fe626ec57c832618c6f990abd6610f81e132621651e0b180b970cff",
|
|
||||||
"zh:a959fa6090a8c0f53739879184e7346423494aee598003df0d1ab4a22b2eee91",
|
|
||||||
"zh:bdda26c2f03f918bbe59e75abea44868fafda019c3a543725331195df126350b",
|
|
||||||
"zh:d8048e149ee97ba62971e6a79355d59887bc6d10fcf72cc2feff3d0a2582670c",
|
|
||||||
"zh:dd36f9988af4e1ca5b1ca7b7bb6f658df9a220dfcda7fec7392fedfe9064f652",
|
|
||||||
"zh:dda2688d46c7e539fe97e8fe9d3ec81fb364170e018d9c6a681364c8955d4e9d",
|
|
||||||
"zh:e6b519afe2dea1c0434f766eb6bc9ba78cc5b6ef2c311c2ca3c65cb24744f31f",
|
|
||||||
]
|
|
||||||
}
|
|
||||||
@@ -1,17 +0,0 @@
|
|||||||
{
|
|
||||||
"name": "oci-hosted",
|
|
||||||
"online": true,
|
|
||||||
"storage": {
|
|
||||||
"blobStoreName": "default",
|
|
||||||
"strictContentTypeValidation": true,
|
|
||||||
"writePolicy": "ALLOW",
|
|
||||||
"latestPolicy": false
|
|
||||||
},
|
|
||||||
"oci": {
|
|
||||||
"v1Enabled": false,
|
|
||||||
"forceBasicAuth": false,
|
|
||||||
"pathEnabled": true
|
|
||||||
},
|
|
||||||
"component": { "proprietaryComponents": false },
|
|
||||||
"cosign": { "enforcement": "NONE" }
|
|
||||||
}
|
|
||||||
@@ -1,27 +0,0 @@
|
|||||||
{
|
|
||||||
"name": "oci-proxy",
|
|
||||||
"online": true,
|
|
||||||
"storage": {
|
|
||||||
"blobStoreName": "default",
|
|
||||||
"strictContentTypeValidation": true
|
|
||||||
},
|
|
||||||
"oci": {
|
|
||||||
"v1Enabled": false,
|
|
||||||
"forceBasicAuth": false,
|
|
||||||
"pathEnabled": true
|
|
||||||
},
|
|
||||||
"ociProxy": {
|
|
||||||
"indexType": "HUB",
|
|
||||||
"cacheForeignLayers": false,
|
|
||||||
"foreignLayerUrlWhitelist": []
|
|
||||||
},
|
|
||||||
"proxy": {
|
|
||||||
"remoteUrl": "https://registry-1.docker.io",
|
|
||||||
"contentMaxAge": 1440,
|
|
||||||
"metadataMaxAge": 60,
|
|
||||||
"preserveEncodedCharacters": false
|
|
||||||
},
|
|
||||||
"negativeCache": { "enabled": true, "timeToLive": 60 },
|
|
||||||
"httpClient": { "blocked": false, "autoBlock": true },
|
|
||||||
"cosign": { "enforcement": "NONE" }
|
|
||||||
}
|
|
||||||
@@ -1,15 +0,0 @@
|
|||||||
{
|
|
||||||
"name": "oci-public",
|
|
||||||
"online": true,
|
|
||||||
"storage": {
|
|
||||||
"blobStoreName": "default",
|
|
||||||
"strictContentTypeValidation": true
|
|
||||||
},
|
|
||||||
"group": { "memberNames": ["oci-proxy"] },
|
|
||||||
"oci": {
|
|
||||||
"v1Enabled": false,
|
|
||||||
"forceBasicAuth": false,
|
|
||||||
"pathEnabled": true
|
|
||||||
},
|
|
||||||
"cosign": { "enforcement": "NONE" }
|
|
||||||
}
|
|
||||||
@@ -1,80 +0,0 @@
|
|||||||
#!/usr/bin/env bash
|
|
||||||
set -euo pipefail
|
|
||||||
|
|
||||||
mode="${1:---check}"
|
|
||||||
case "$mode" in
|
|
||||||
--check | --apply) ;;
|
|
||||||
*) echo "usage: $0 [--check|--apply]" >&2; exit 2 ;;
|
|
||||||
esac
|
|
||||||
|
|
||||||
: "${TF_VAR_nexus_url:?set TF_VAR_nexus_url}"
|
|
||||||
: "${TF_VAR_nexus_username:?set TF_VAR_nexus_username}"
|
|
||||||
: "${TF_VAR_nexus_password:?set TF_VAR_nexus_password}"
|
|
||||||
|
|
||||||
script_dir="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")" && pwd)"
|
|
||||||
auth_file="$(mktemp /tmp/nexus-oci-auth.XXXXXX)"
|
|
||||||
trap 'rm -f -- "$auth_file"' EXIT
|
|
||||||
chmod 0600 "$auth_file"
|
|
||||||
printf 'machine %s\nlogin %s\npassword %s\n' \
|
|
||||||
"${TF_VAR_nexus_url#*://}" "$TF_VAR_nexus_username" \
|
|
||||||
"$TF_VAR_nexus_password" >"$auth_file"
|
|
||||||
|
|
||||||
drift=0
|
|
||||||
for entry in \
|
|
||||||
"hosted:$script_dir/oci/oci-hosted.json" \
|
|
||||||
"proxy:$script_dir/oci/oci-proxy.json" \
|
|
||||||
"group:$script_dir/oci/oci-public.json"; do
|
|
||||||
repository_type="${entry%%:*}"
|
|
||||||
desired_file="${entry#*:}"
|
|
||||||
repository_name="$(jq -er '.name' "$desired_file")"
|
|
||||||
endpoint="$TF_VAR_nexus_url/service/rest/v1/repositories/oci/$repository_type"
|
|
||||||
current_file="$(mktemp /tmp/nexus-oci-current.XXXXXX)"
|
|
||||||
|
|
||||||
status="$(curl --silent --show-error --netrc-file "$auth_file" \
|
|
||||||
--output "$current_file" --write-out '%{http_code}' \
|
|
||||||
"$endpoint/$repository_name")"
|
|
||||||
|
|
||||||
if [[ "$status" == 404 ]]; then
|
|
||||||
drift=1
|
|
||||||
if [[ "$mode" == --apply ]]; then
|
|
||||||
curl --fail --silent --show-error --netrc-file "$auth_file" \
|
|
||||||
--header 'Content-Type: application/json' \
|
|
||||||
--data-binary "@$desired_file" "$endpoint"
|
|
||||||
echo "created $repository_name"
|
|
||||||
else
|
|
||||||
echo "missing $repository_name" >&2
|
|
||||||
fi
|
|
||||||
elif [[ "$status" == 200 ]]; then
|
|
||||||
if jq -e --slurpfile desired "$desired_file" '
|
|
||||||
def subset($actual; $wanted):
|
|
||||||
if ($wanted | type) == "object" then
|
|
||||||
all($wanted | keys[];
|
|
||||||
($actual[.] != null) and subset($actual[.]; $wanted[.]))
|
|
||||||
else
|
|
||||||
$actual == $wanted
|
|
||||||
end;
|
|
||||||
subset(.; $desired[0])
|
|
||||||
' "$current_file" >/dev/null; then
|
|
||||||
echo "in sync $repository_name"
|
|
||||||
else
|
|
||||||
drift=1
|
|
||||||
if [[ "$mode" == --apply ]]; then
|
|
||||||
curl --fail --silent --show-error --netrc-file "$auth_file" \
|
|
||||||
--request PUT --header 'Content-Type: application/json' \
|
|
||||||
--data-binary "@$desired_file" "$endpoint/$repository_name"
|
|
||||||
echo "updated $repository_name"
|
|
||||||
else
|
|
||||||
echo "drifted $repository_name" >&2
|
|
||||||
fi
|
|
||||||
fi
|
|
||||||
else
|
|
||||||
cat "$current_file" >&2
|
|
||||||
echo "unexpected HTTP $status for $repository_name" >&2
|
|
||||||
exit 1
|
|
||||||
fi
|
|
||||||
rm -f -- "$current_file"
|
|
||||||
done
|
|
||||||
|
|
||||||
if [[ "$mode" == --check && "$drift" -ne 0 ]]; then
|
|
||||||
exit 1
|
|
||||||
fi
|
|
||||||
@@ -1,64 +0,0 @@
|
|||||||
locals {
|
|
||||||
proxy_http_client = {
|
|
||||||
auto_block = true
|
|
||||||
blocked = false
|
|
||||||
}
|
|
||||||
|
|
||||||
proxy_negative_cache = {
|
|
||||||
enabled = true
|
|
||||||
time_to_live = 60
|
|
||||||
}
|
|
||||||
|
|
||||||
repository_storage = {
|
|
||||||
blob_store_name = "default"
|
|
||||||
strict_content_type_validation = true
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
resource "sonatyperepo_repository_ansiblegalaxy_proxy" "galaxy" {
|
|
||||||
name = "ansible-galaxy-proxy"
|
|
||||||
online = true
|
|
||||||
|
|
||||||
http_client = local.proxy_http_client
|
|
||||||
negative_cache = local.proxy_negative_cache
|
|
||||||
proxy = {
|
|
||||||
remote_url = "https://galaxy.ansible.com"
|
|
||||||
content_max_age = 1440
|
|
||||||
metadata_max_age = 60
|
|
||||||
}
|
|
||||||
storage = local.repository_storage
|
|
||||||
}
|
|
||||||
|
|
||||||
resource "sonatyperepo_repository_ansiblegalaxy_group" "public" {
|
|
||||||
name = "ansible-public"
|
|
||||||
online = true
|
|
||||||
|
|
||||||
group = {
|
|
||||||
member_names = [sonatyperepo_repository_ansiblegalaxy_proxy.galaxy.name]
|
|
||||||
}
|
|
||||||
storage = local.repository_storage
|
|
||||||
}
|
|
||||||
|
|
||||||
resource "sonatyperepo_repository_go_proxy" "public" {
|
|
||||||
name = "go-proxy"
|
|
||||||
online = true
|
|
||||||
|
|
||||||
http_client = local.proxy_http_client
|
|
||||||
negative_cache = local.proxy_negative_cache
|
|
||||||
proxy = {
|
|
||||||
remote_url = "https://proxy.golang.org"
|
|
||||||
content_max_age = 1440
|
|
||||||
metadata_max_age = 60
|
|
||||||
}
|
|
||||||
storage = local.repository_storage
|
|
||||||
}
|
|
||||||
|
|
||||||
resource "sonatyperepo_repository_go_group" "public" {
|
|
||||||
name = "go-public"
|
|
||||||
online = true
|
|
||||||
|
|
||||||
group = {
|
|
||||||
member_names = [sonatyperepo_repository_go_proxy.public.name]
|
|
||||||
}
|
|
||||||
storage = local.repository_storage
|
|
||||||
}
|
|
||||||
@@ -1,75 +0,0 @@
|
|||||||
resource "sonatyperepo_privilege_repository_view" "anonymous_ansible" {
|
|
||||||
name = "ci-anonymous-ansible-read"
|
|
||||||
description = "Anonymous read access to the Ansible Galaxy group"
|
|
||||||
actions = ["BROWSE", "READ"]
|
|
||||||
format = "ansiblegalaxy"
|
|
||||||
repository = sonatyperepo_repository_ansiblegalaxy_group.public.name
|
|
||||||
}
|
|
||||||
|
|
||||||
resource "sonatyperepo_privilege_repository_view" "anonymous_ansible_proxy" {
|
|
||||||
name = "ci-anonymous-ansible-proxy-read"
|
|
||||||
description = "Anonymous artifact read access to the Ansible Galaxy proxy"
|
|
||||||
actions = ["BROWSE", "READ"]
|
|
||||||
format = "ansiblegalaxy"
|
|
||||||
repository = sonatyperepo_repository_ansiblegalaxy_proxy.galaxy.name
|
|
||||||
}
|
|
||||||
|
|
||||||
resource "sonatyperepo_privilege_repository_view" "anonymous_go" {
|
|
||||||
name = "ci-anonymous-go-read"
|
|
||||||
description = "Anonymous read access to the Go module group"
|
|
||||||
actions = ["BROWSE", "READ"]
|
|
||||||
format = "go"
|
|
||||||
repository = sonatyperepo_repository_go_group.public.name
|
|
||||||
}
|
|
||||||
|
|
||||||
resource "sonatyperepo_privilege_repository_view" "anonymous_oci_public" {
|
|
||||||
name = "ci-anonymous-oci-public-read"
|
|
||||||
description = "Anonymous read access to the public OCI group"
|
|
||||||
actions = ["BROWSE", "READ"]
|
|
||||||
format = "oci"
|
|
||||||
repository = "oci-public"
|
|
||||||
}
|
|
||||||
|
|
||||||
resource "sonatyperepo_privilege_repository_view" "anonymous_oci_proxy" {
|
|
||||||
name = "ci-anonymous-oci-proxy-read"
|
|
||||||
description = "Anonymous read access to the OCI proxy member"
|
|
||||||
actions = ["BROWSE", "READ"]
|
|
||||||
format = "oci"
|
|
||||||
repository = "oci-proxy"
|
|
||||||
}
|
|
||||||
|
|
||||||
resource "sonatyperepo_role" "anonymous_ci" {
|
|
||||||
id = "ci-anonymous-read"
|
|
||||||
name = "CI anonymous read"
|
|
||||||
description = "Read-only access to public CI dependency proxy groups"
|
|
||||||
privileges = [
|
|
||||||
sonatyperepo_privilege_repository_view.anonymous_ansible.name,
|
|
||||||
sonatyperepo_privilege_repository_view.anonymous_ansible_proxy.name,
|
|
||||||
sonatyperepo_privilege_repository_view.anonymous_go.name,
|
|
||||||
sonatyperepo_privilege_repository_view.anonymous_oci_public.name,
|
|
||||||
sonatyperepo_privilege_repository_view.anonymous_oci_proxy.name,
|
|
||||||
]
|
|
||||||
roles = []
|
|
||||||
}
|
|
||||||
|
|
||||||
resource "sonatyperepo_user" "anonymous" {
|
|
||||||
user_id = "anonymous"
|
|
||||||
first_name = "Anonymous"
|
|
||||||
last_name = "User"
|
|
||||||
email_address = "[email protected]"
|
|
||||||
status = "active"
|
|
||||||
roles = [sonatyperepo_role.anonymous_ci.id]
|
|
||||||
}
|
|
||||||
|
|
||||||
resource "sonatyperepo_system_anonymous_access" "ci" {
|
|
||||||
enabled = true
|
|
||||||
user_id = sonatyperepo_user.anonymous.user_id
|
|
||||||
realm_name = "NexusAuthorizingRealm"
|
|
||||||
}
|
|
||||||
|
|
||||||
resource "sonatyperepo_security_realms" "active" {
|
|
||||||
active = [
|
|
||||||
"NexusAuthenticatingRealm",
|
|
||||||
"OciBearerToken",
|
|
||||||
]
|
|
||||||
}
|
|
||||||
@@ -1,33 +0,0 @@
|
|||||||
terraform {
|
|
||||||
required_version = ">= 1.11.0"
|
|
||||||
|
|
||||||
required_providers {
|
|
||||||
sonatyperepo = {
|
|
||||||
source = "sonatype-nexus-community/sonatyperepo"
|
|
||||||
version = "1.17.0"
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
provider "sonatyperepo" {
|
|
||||||
url = var.nexus_url
|
|
||||||
username = var.nexus_username
|
|
||||||
password = var.nexus_password
|
|
||||||
}
|
|
||||||
|
|
||||||
variable "nexus_url" {
|
|
||||||
description = "Nexus Repository base URL"
|
|
||||||
type = string
|
|
||||||
}
|
|
||||||
|
|
||||||
variable "nexus_username" {
|
|
||||||
description = "Nexus Terraform management username"
|
|
||||||
type = string
|
|
||||||
sensitive = true
|
|
||||||
}
|
|
||||||
|
|
||||||
variable "nexus_password" {
|
|
||||||
description = "Nexus Terraform management password"
|
|
||||||
type = string
|
|
||||||
sensitive = true
|
|
||||||
}
|
|
||||||
@@ -53,6 +53,4 @@ sudo k3s kubectl -n flux-system get gitrepositories,kustomizations
|
|||||||
- SPIRE 已按 hardened chart 内部 fork `0.30.2-ddupan.1`（基于上游 `0.30.2`，SPIRE
|
- SPIRE 已按 hardened chart 内部 fork `0.30.2-ddupan.1`（基于上游 `0.30.2`，SPIRE
|
||||||
`1.15.3`）声明，使用共享
|
`1.15.3`）声明，使用共享
|
||||||
PostgreSQL 与独立 signing-key PVC；首次上线和 OpenBao JWT-SVID PoC 尚待合并后验证；
|
PostgreSQL 与独立 signing-key PVC；首次上线和 OpenBao JWT-SVID PoC 尚待合并后验证；
|
||||||
- Nexus Repository CE POC 已加入 GitOps 声明，计划验证 Ansible Galaxy、Go Modules 与
|
|
||||||
OCI/BuildKit 缓存；尚未部署或完成现场验收，现有 zot 保持不变；
|
|
||||||
- root Kustomization 与所有 brownfield 子 Kustomization 继续保持 `prune: false`。
|
- root Kustomization 与所有 brownfield 子 Kustomization 继续保持 `prune: false`。
|
||||||
|
|||||||
@@ -6,12 +6,11 @@ metadata:
|
|||||||
spec:
|
spec:
|
||||||
dependsOn:
|
dependsOn:
|
||||||
- name: external-secrets
|
- name: external-secrets
|
||||||
|
- name: nats
|
||||||
- name: spire
|
- name: spire
|
||||||
interval: 10m
|
interval: 10m
|
||||||
path: ./platform/dynamic-runner
|
path: ./platform/dynamic-runner
|
||||||
# The runner backends are replaceable. Prune is required when a retired
|
prune: false
|
||||||
# worker is removed from the component; otherwise it keeps consuming work.
|
|
||||||
prune: true
|
|
||||||
sourceRef:
|
sourceRef:
|
||||||
kind: GitRepository
|
kind: GitRepository
|
||||||
name: flux-system
|
name: flux-system
|
||||||
|
|||||||
@@ -6,7 +6,7 @@ metadata:
|
|||||||
spec:
|
spec:
|
||||||
interval: 10m
|
interval: 10m
|
||||||
path: ./platform/gitea-runner
|
path: ./platform/gitea-runner
|
||||||
prune: true
|
prune: false
|
||||||
sourceRef:
|
sourceRef:
|
||||||
kind: GitRepository
|
kind: GitRepository
|
||||||
name: flux-system
|
name: flux-system
|
||||||
|
|||||||
@@ -1,22 +0,0 @@
|
|||||||
apiVersion: kustomize.toolkit.fluxcd.io/v1
|
|
||||||
kind: Kustomization
|
|
||||||
metadata:
|
|
||||||
name: nexus
|
|
||||||
namespace: flux-system
|
|
||||||
spec:
|
|
||||||
dependsOn:
|
|
||||||
- name: envoy-gateway
|
|
||||||
- name: openebs
|
|
||||||
healthChecks:
|
|
||||||
- apiVersion: apps/v1
|
|
||||||
kind: Deployment
|
|
||||||
name: nexus
|
|
||||||
namespace: nexus
|
|
||||||
interval: 10m
|
|
||||||
path: ./apps/nexus
|
|
||||||
prune: false
|
|
||||||
sourceRef:
|
|
||||||
kind: GitRepository
|
|
||||||
name: flux-system
|
|
||||||
timeout: 15m
|
|
||||||
wait: true
|
|
||||||
@@ -15,4 +15,3 @@ resources:
|
|||||||
- apps/spire.yaml
|
- apps/spire.yaml
|
||||||
- apps/observability.yaml
|
- apps/observability.yaml
|
||||||
- apps/zot.yaml
|
- apps/zot.yaml
|
||||||
- apps/nexus.yaml
|
|
||||||
|
|||||||
@@ -15,8 +15,7 @@ Root bootstrap 已完成。后续按依赖顺序分别引入：
|
|||||||
1. 监控 CRD、kube-state-metrics 以及 kubelet/cAdvisor 抓取配置；
|
1. 监控 CRD、kube-state-metrics 以及 kubelet/cAdvisor 抓取配置；
|
||||||
2. SPIRE Agent、SPIFFE CSI Driver 与 workload registration；
|
2. SPIRE Agent、SPIFFE CSI Driver 与 workload registration；
|
||||||
3. Kata Containers、`block-plain` RuntimeClass；
|
3. Kata Containers、`block-plain` RuntimeClass；
|
||||||
4. 独立 External Secrets Operator 与 sandbox 专用 OpenBao auth backend；
|
4. OpenSandbox operator/server 及 `ci-pod`、`ci-vm` Pools。
|
||||||
5. OpenSandbox controller/server；CI Pool 与 runner 调度器随后独立接入。
|
|
||||||
|
|
||||||
每一阶段单独合并并等待对应 Flux Kustomization Ready，不在 bootstrap 时一次性部署。
|
每一阶段单独合并并等待对应 Flux Kustomization Ready，不在 bootstrap 时一次性部署。
|
||||||
第一阶段监控拆为 `monitoring-operator` 与依赖它的 `monitoring`，防止 VM CR 在
|
第一阶段监控拆为 `monitoring-operator` 与依赖它的 `monitoring`，防止 VM CR 在
|
||||||
@@ -43,25 +42,15 @@ attestation。Server 使用 external bundle publisher 持续维护 sandbox
|
|||||||
显式关闭 Server 与 OIDC Provider，只部署 Agent DaemonSet 和 SPIFFE CSI Driver；因此
|
显式关闭 Server 与 OIDC Provider，只部署 Agent DaemonSet 和 SPIFFE CSI Driver；因此
|
||||||
不会产生第二个 trust root。
|
不会产生第二个 trust root。
|
||||||
|
|
||||||
`spire-smoke` namespace、ServiceAccount 和 `sandbox-spire-smoke` ClusterSPIFFEID 只用于
|
`spire-smoke` namespace、ServiceAccount 和 `sandbox-spire-smoke` ClusterSPIFFEID 是
|
||||||
普通 Pod 的 CSI 回归夹具，稳定身份为 `spiffe://ddupan.top/sandbox/smoke`。Kata guest
|
普通 Pod 与后续 Kata guest 的回归夹具，稳定身份为
|
||||||
不能复用 node Agent 暴露的 Unix socket；virtio-fs 只能呈现 socket 路径，不能把连接
|
`spiffe://ddupan.top/sandbox/smoke`。测试 Pod 临时创建并在验收后删除，身份声明保留。
|
||||||
跨过 VM 边界。Kata workload 必须使用 guest 内 Agent，具体约束见
|
|
||||||
`platform/sandbox-kata/README.md`。测试 Pod 临时创建并在验收后删除，普通 Pod 的身份
|
|
||||||
声明保留。
|
|
||||||
|
|
||||||
Kata 阶段使用官方 4.1.0 `kata-deploy` chart 的短生命周期 `job` 模式，逐节点安装并
|
Kata 阶段使用官方 4.1.0 `kata-deploy` chart 的短生命周期 `job` 模式，逐节点安装并
|
||||||
重启 K3s。只启用 `kata-clh-runtime-rs`，不创建默认 `kata` 别名；该 handler 的
|
重启 K3s。只启用 `kata-clh-runtime-rs`，不创建默认 `kata` 别名；该 handler 的
|
||||||
`emptyDir` 固定使用 `block-plain`，为 Docker/BuildKit overlay2 与 kind 提供 guest
|
`emptyDir` 固定使用 `block-plain`，为 Docker/BuildKit overlay2 与 kind 提供 guest
|
||||||
内块设备文件系统。详细限制与上线验收见 `platform/sandbox-kata/README.md`。
|
内块设备文件系统。详细限制与上线验收见 `platform/sandbox-kata/README.md`。
|
||||||
|
|
||||||
Sandbox 的 ESO 通过独立 `auth/kubernetes-sandbox` 向 OpenBao 证明 ServiceAccount
|
|
||||||
身份，只能读取共享的 `kv/k8s/opensandbox-api`。它不保存 reviewer JWT 或长期 Bao token；相关
|
|
||||||
Terraform 与 Flux 边界见 `platform/sandbox-external-secrets/README.md`。
|
|
||||||
|
|
||||||
OpenSandbox 阶段固定官方源码 commit 与 umbrella chart `0.2.2`，只部署 controller、
|
|
||||||
ClusterIP server 和 CRD。API key 由 ESO 从 OpenBao 投影，明文不进入 Git。
|
|
||||||
|
|
||||||
## 监控边界
|
## 监控边界
|
||||||
|
|
||||||
这里只管理 sandbox LXC 内的 Kubernetes 监控，不负责 PVE 宿主监控。LXC 与宿主共享
|
这里只管理 sandbox LXC 内的 Kubernetes 监控，不负责 PVE 宿主监控。LXC 与宿主共享
|
||||||
|
|||||||
@@ -1,17 +0,0 @@
|
|||||||
---
|
|
||||||
apiVersion: kustomize.toolkit.fluxcd.io/v1
|
|
||||||
kind: Kustomization
|
|
||||||
metadata:
|
|
||||||
name: external-secrets-operator
|
|
||||||
namespace: flux-system
|
|
||||||
spec:
|
|
||||||
dependsOn:
|
|
||||||
- name: monitoring-operator
|
|
||||||
interval: 10m
|
|
||||||
path: ./platform/sandbox-external-secrets/operator
|
|
||||||
prune: true
|
|
||||||
sourceRef:
|
|
||||||
kind: GitRepository
|
|
||||||
name: flux-system
|
|
||||||
timeout: 10m
|
|
||||||
wait: true
|
|
||||||
@@ -1,17 +0,0 @@
|
|||||||
---
|
|
||||||
apiVersion: kustomize.toolkit.fluxcd.io/v1
|
|
||||||
kind: Kustomization
|
|
||||||
metadata:
|
|
||||||
name: external-secrets
|
|
||||||
namespace: flux-system
|
|
||||||
spec:
|
|
||||||
dependsOn:
|
|
||||||
- name: external-secrets-operator
|
|
||||||
interval: 10m
|
|
||||||
path: ./platform/sandbox-external-secrets/config
|
|
||||||
prune: true
|
|
||||||
sourceRef:
|
|
||||||
kind: GitRepository
|
|
||||||
name: flux-system
|
|
||||||
timeout: 10m
|
|
||||||
wait: true
|
|
||||||
+2
-3
@@ -2,14 +2,13 @@
|
|||||||
apiVersion: kustomize.toolkit.fluxcd.io/v1
|
apiVersion: kustomize.toolkit.fluxcd.io/v1
|
||||||
kind: Kustomization
|
kind: Kustomization
|
||||||
metadata:
|
metadata:
|
||||||
name: ci-runners
|
name: opensandbox-pools
|
||||||
namespace: flux-system
|
namespace: flux-system
|
||||||
spec:
|
spec:
|
||||||
dependsOn:
|
dependsOn:
|
||||||
- name: opensandbox
|
- name: opensandbox
|
||||||
- name: spire-agents
|
|
||||||
interval: 10m
|
interval: 10m
|
||||||
path: ./platform/sandbox-ci-runners
|
path: ./platform/sandbox-opensandbox-pools
|
||||||
prune: true
|
prune: true
|
||||||
sourceRef:
|
sourceRef:
|
||||||
kind: GitRepository
|
kind: GitRepository
|
||||||
@@ -6,14 +6,13 @@ metadata:
|
|||||||
namespace: flux-system
|
namespace: flux-system
|
||||||
spec:
|
spec:
|
||||||
dependsOn:
|
dependsOn:
|
||||||
- name: external-secrets
|
|
||||||
- name: kata
|
- name: kata
|
||||||
- name: monitoring-operator
|
- name: spire-agents
|
||||||
interval: 10m
|
interval: 10m
|
||||||
path: ./platform/sandbox-opensandbox
|
path: ./platform/sandbox-opensandbox
|
||||||
prune: true
|
prune: true
|
||||||
sourceRef:
|
sourceRef:
|
||||||
kind: GitRepository
|
kind: GitRepository
|
||||||
name: flux-system
|
name: flux-system
|
||||||
timeout: 15m
|
timeout: 20m
|
||||||
wait: true
|
wait: true
|
||||||
|
|||||||
@@ -7,7 +7,5 @@ resources:
|
|||||||
- apps/spire-bootstrap.yaml
|
- apps/spire-bootstrap.yaml
|
||||||
- apps/spire-agents.yaml
|
- apps/spire-agents.yaml
|
||||||
- apps/kata.yaml
|
- apps/kata.yaml
|
||||||
- apps/external-secrets-operator.yaml
|
|
||||||
- apps/external-secrets.yaml
|
|
||||||
- apps/opensandbox.yaml
|
- apps/opensandbox.yaml
|
||||||
- apps/ci-runners.yaml
|
- apps/opensandbox-pools.yaml
|
||||||
|
|||||||
@@ -18,7 +18,6 @@ homelab_dns:
|
|||||||
- { zone: ad.ddupan.top, name: metrics-write, type: A, values: [192.168.10.127] }
|
- { zone: ad.ddupan.top, name: metrics-write, type: A, values: [192.168.10.127] }
|
||||||
- { zone: ad.ddupan.top, name: netbox, type: A, values: [192.168.10.127] }
|
- { zone: ad.ddupan.top, name: netbox, type: A, values: [192.168.10.127] }
|
||||||
- { zone: ad.ddupan.top, name: nats, type: A, values: [192.168.10.127] }
|
- { zone: ad.ddupan.top, name: nats, type: A, values: [192.168.10.127] }
|
||||||
- { zone: ad.ddupan.top, name: nexus, type: A, values: [192.168.10.127] }
|
|
||||||
- { zone: ad.ddupan.top, name: s3, type: A, values: [192.168.10.127] }
|
- { zone: ad.ddupan.top, name: s3, type: A, values: [192.168.10.127] }
|
||||||
- { zone: ad.ddupan.top, name: spire-oidc, type: A, values: [192.168.10.127] }
|
- { zone: ad.ddupan.top, name: spire-oidc, type: A, values: [192.168.10.127] }
|
||||||
- { zone: ad.ddupan.top, name: spire-server, type: A, values: [192.168.10.127] }
|
- { zone: ad.ddupan.top, name: spire-server, type: A, values: [192.168.10.127] }
|
||||||
|
|||||||
@@ -51,7 +51,7 @@ few things Terraform must not own.**
|
|||||||
| secrets engine mounts (`kv`, `pki`, `ssh-client-signer`) | the daemon, Raft, TLS files, systemd |
|
| secrets engine mounts (`kv`, `pki`, `ssh-client-signer`) | the daemon, Raft, TLS files, systemd |
|
||||||
| PKI role, issuing/CRL URLs, cluster paths, **ACME** | `bao operator init` / unseal (manual, PGP-wrapped) |
|
| PKI role, issuing/CRL URLs, cluster paths, **ACME** | `bao operator init` / unseal (manual, PGP-wrapped) |
|
||||||
| SSH signing role (`ai-agent`) | **PKI root CA + SSH CA signing key** |
|
| SSH signing role (`ai-agent`) | **PKI root CA + SSH CA signing key** |
|
||||||
| OIDC/Kubernetes auth mount、config 与 role | OIDC **client secret** (`auth/oidc/config`) |
|
| OIDC auth *mount* and *role* | OIDC **client secret** (`auth/oidc/config`) |
|
||||||
| all policies | snapshot token + script + systemd timer |
|
| all policies | snapshot token + script + systemd timer |
|
||||||
| | host-level CA trust distribution (`openbao_ssh_ca_trust`) |
|
| | host-level CA trust distribution (`openbao_ssh_ca_trust`) |
|
||||||
|
|
||||||
@@ -92,13 +92,6 @@ then `VAULT_ADDR`/`VAULT_TOKEN`), mirroring how `smtp-relay/terraform` uses
|
|||||||
native `openbao/openbao` provider is published only to the OpenTofu registry and
|
native `openbao/openbao` provider is published only to the OpenTofu registry and
|
||||||
cannot be resolved by the HashiCorp `terraform` CLI.
|
cannot be resolved by the HashiCorp `terraform` CLI.
|
||||||
|
|
||||||
Sandbox 集群使用独立的 `auth/kubernetes-sandbox`。其 API 地址、公开 Kubernetes CA、
|
|
||||||
ESO role 与只读 `kv/k8s/opensandbox-api` policy 全部由 Terraform 管理；CA 位于
|
|
||||||
`terraform/certs/sandbox-kubernetes-ca.crt`。集群重建并轮换 CA 后，先更新该文件并
|
|
||||||
apply，再让 Flux 恢复 ESO reconciliation。该 backend 不保存 reviewer JWT，而是使用
|
|
||||||
ESO 的短期登录 JWT 执行 TokenReview。该组资源已于 2026-09-18 apply，随后复验 plan
|
|
||||||
为 zero-diff。
|
|
||||||
|
|
||||||
## DNS
|
## DNS
|
||||||
|
|
||||||
`bao.ad.ddupan.top` is an **internal-only** name — not a public Cloudflare record and
|
`bao.ad.ddupan.top` is an **internal-only** name — not a public Cloudflare record and
|
||||||
|
|||||||
@@ -59,33 +59,3 @@ resource "vault_kubernetes_auth_backend_role" "external_secrets" {
|
|||||||
# in a long TTL and every extra hour is a longer-lived credential in memory.
|
# in a long TTL and every extra hour is a longer-lived credential in memory.
|
||||||
token_ttl = 3600
|
token_ttl = 3600
|
||||||
}
|
}
|
||||||
|
|
||||||
# A Kubernetes auth mount can validate identities from only the API server it is
|
|
||||||
# configured against. The sandbox cluster therefore cannot reuse auth/kubernetes,
|
|
||||||
# whose TokenReview endpoint belongs to homelab.
|
|
||||||
resource "vault_auth_backend" "sandbox_kubernetes" {
|
|
||||||
type = "kubernetes"
|
|
||||||
path = "kubernetes-sandbox"
|
|
||||||
}
|
|
||||||
|
|
||||||
resource "vault_kubernetes_auth_backend_config" "sandbox" {
|
|
||||||
backend = vault_auth_backend.sandbox_kubernetes.path
|
|
||||||
kubernetes_host = "https://10.60.0.13:6443"
|
|
||||||
kubernetes_ca_cert = file("${path.module}/certs/sandbox-kubernetes-ca.crt")
|
|
||||||
disable_local_ca_jwt = true
|
|
||||||
|
|
||||||
# Deliberately omit token_reviewer_jwt. OpenBao uses the login JWT for
|
|
||||||
# TokenReview; the sandbox external-secrets ServiceAccount is bound only to
|
|
||||||
# system:auth-delegator and all issued JWTs remain short-lived.
|
|
||||||
}
|
|
||||||
|
|
||||||
resource "vault_kubernetes_auth_backend_role" "sandbox_external_secrets" {
|
|
||||||
backend = vault_auth_backend.sandbox_kubernetes.path
|
|
||||||
role_name = "external-secrets"
|
|
||||||
|
|
||||||
bound_service_account_names = ["external-secrets"]
|
|
||||||
bound_service_account_namespaces = ["external-secrets"]
|
|
||||||
|
|
||||||
token_policies = [vault_policy.sandbox_external_secrets.name]
|
|
||||||
token_ttl = 3600
|
|
||||||
}
|
|
||||||
|
|||||||
@@ -1,10 +0,0 @@
|
|||||||
-----BEGIN CERTIFICATE-----
|
|
||||||
MIIBdjCCAR2gAwIBAgIBADAKBggqhkjOPQQDAjAjMSEwHwYDVQQDDBhrM3Mtc2Vy
|
|
||||||
dmVyLWNhQDE3ODk2NTI4NTcwHhcNMjYwOTE3MTI0NzM3WhcNMzYwOTE0MTI0NzM3
|
|
||||||
WjAjMSEwHwYDVQQDDBhrM3Mtc2VydmVyLWNhQDE3ODk2NTI4NTcwWTATBgcqhkjO
|
|
||||||
PQIBBggqhkjOPQMBBwNCAAR4SbqzTXZnlZdUPz7viN6+dYbB1Maw44Qiepn9r5XG
|
|
||||||
sOzkYkN8t1aG3Ugo8TqQ3xJaKkM89n1Rluj0vbOhiNajo0IwQDAOBgNVHQ8BAf8E
|
|
||||||
BAMCAqQwDwYDVR0TAQH/BAUwAwEB/zAdBgNVHQ4EFgQUyLSGoKAKAJuiniuRdBLG
|
|
||||||
XYaDQC8wCgYIKoZIzj0EAwIDRwAwRAIgFkVzyUZexk/ynnxBEOg+3foJv3WKqAei
|
|
||||||
hTSRjO1gL0UCIFbBKR7BMrJJAgW3DJFeeBM+b+tTg93jNx55qZACbFOL
|
|
||||||
-----END CERTIFICATE-----
|
|
||||||
@@ -35,10 +35,3 @@ resource "vault_policy" "external_secrets" {
|
|||||||
name = "external-secrets"
|
name = "external-secrets"
|
||||||
policy = file("${path.module}/policies/external-secrets.hcl")
|
policy = file("${path.module}/policies/external-secrets.hcl")
|
||||||
}
|
}
|
||||||
|
|
||||||
# The sandbox cluster has its own Kubernetes auth backend and a deliberately
|
|
||||||
# narrower KV view than the homelab ESO instance.
|
|
||||||
resource "vault_policy" "sandbox_external_secrets" {
|
|
||||||
name = "sandbox-external-secrets"
|
|
||||||
policy = file("${path.module}/policies/sandbox-external-secrets.hcl")
|
|
||||||
}
|
|
||||||
|
|||||||
@@ -1,9 +0,0 @@
|
|||||||
# Read only the shared OpenSandbox control-plane API key. The same Bao object is
|
|
||||||
# consumed by the server in sandbox and, later, by the scheduler in homelab.
|
|
||||||
path "kv/data/k8s/opensandbox-api" {
|
|
||||||
capabilities = ["read"]
|
|
||||||
}
|
|
||||||
|
|
||||||
path "kv/metadata/k8s/opensandbox-api" {
|
|
||||||
capabilities = ["read"]
|
|
||||||
}
|
|
||||||
@@ -83,11 +83,6 @@ vyos_postgresql_primary_address: "10.60.0.11"
|
|||||||
vyos_sandbox_k3s_api_address: "10.60.0.13"
|
vyos_sandbox_k3s_api_address: "10.60.0.13"
|
||||||
vyos_sandbox_k3s_api_interface: eth1
|
vyos_sandbox_k3s_api_interface: eth1
|
||||||
vyos_sandbox_k3s_api_port: 6443
|
vyos_sandbox_k3s_api_port: 6443
|
||||||
vyos_opensandbox_api_port: 8080
|
|
||||||
vyos_opensandbox_node_port: 30080
|
|
||||||
# Lifecycle create is synchronous and a cold Kata image pull can exceed the
|
|
||||||
# HAProxy 50-second default. Keep this below OpenSandbox's overall timeout.
|
|
||||||
vyos_opensandbox_api_timeout: 600
|
|
||||||
vyos_sandbox_k3s_servers:
|
vyos_sandbox_k3s_servers:
|
||||||
- name: sandbox1
|
- name: sandbox1
|
||||||
address: "10.60.0.11"
|
address: "10.60.0.11"
|
||||||
|
|||||||
@@ -15,20 +15,6 @@ set interfaces ethernet {{ i.iface }} address {{ i.address }}
|
|||||||
set interfaces ethernet {{ i.iface }} description '{{ i.description }}'
|
set interfaces ethernet {{ i.iface }} description '{{ i.description }}'
|
||||||
{% endfor %}
|
{% endfor %}
|
||||||
|
|
||||||
{# OpenSandbox stays on a NodePort; VyOS owns the stable routed frontend. #}
|
|
||||||
set load-balancing haproxy service opensandbox-api listen-address {{ vyos_sandbox_k3s_api_address }}
|
|
||||||
set load-balancing haproxy service opensandbox-api port {{ vyos_opensandbox_api_port }}
|
|
||||||
set load-balancing haproxy service opensandbox-api mode 'tcp'
|
|
||||||
set load-balancing haproxy service opensandbox-api backend 'opensandbox-api'
|
|
||||||
set load-balancing haproxy service opensandbox-api timeout client {{ vyos_opensandbox_api_timeout }}
|
|
||||||
set load-balancing haproxy backend opensandbox-api mode 'tcp'
|
|
||||||
set load-balancing haproxy backend opensandbox-api timeout server {{ vyos_opensandbox_api_timeout }}
|
|
||||||
{% for server in vyos_sandbox_k3s_servers %}
|
|
||||||
set load-balancing haproxy backend opensandbox-api server {{ server.name }} address {{ server.address }}
|
|
||||||
set load-balancing haproxy backend opensandbox-api server {{ server.name }} port {{ vyos_opensandbox_node_port }}
|
|
||||||
set load-balancing haproxy backend opensandbox-api server {{ server.name }} check
|
|
||||||
{% endfor %}
|
|
||||||
|
|
||||||
{# --- default route out; OSPF carries the rest --- #}
|
{# --- default route out; OSPF carries the rest --- #}
|
||||||
set protocols static route 0.0.0.0/0 next-hop {{ vyos_lan_gateway }}
|
set protocols static route 0.0.0.0/0 next-hop {{ vyos_lan_gateway }}
|
||||||
|
|
||||||
|
|||||||
@@ -18,8 +18,7 @@ Flux 管理以下 Kubernetes 资源：
|
|||||||
- Kata Containers 和 CI 专用的 `block-plain` RuntimeClass；
|
- Kata Containers 和 CI 专用的 `block-plain` RuntimeClass；
|
||||||
- SPIRE Agent、SPIFFE CSI Driver 与 workload identity 声明；
|
- SPIRE Agent、SPIFFE CSI Driver 与 workload identity 声明；
|
||||||
- vmagent、kube-state-metrics、kubelet/cAdvisor scrape 配置和告警；
|
- vmagent、kube-state-metrics、kubelet/cAdvisor scrape 配置和告警；
|
||||||
- sandbox External Secrets Operator、OpenSandbox controller/server；CI Pool 与 runner
|
- OpenSandbox operator/server、`ci-pod` 与 `ci-vm` Pools。
|
||||||
调度器由 runner 项目接入。
|
|
||||||
|
|
||||||
同一个对象只能有一个 owner。Ansible 不直接部署上述集群内 workload；Flux 不管理
|
同一个对象只能有一个 owner。Ansible 不直接部署上述集群内 workload；Flux 不管理
|
||||||
LXC、K3s datastore 或 K3s 本身。
|
LXC、K3s datastore 或 K3s 本身。
|
||||||
@@ -93,13 +92,6 @@ ansible-playbook site.yml
|
|||||||
ansible-playbook k3s.yml
|
ansible-playbook k3s.yml
|
||||||
```
|
```
|
||||||
|
|
||||||
只 reconcile Flux controllers 与 root sync manifest（不触碰 LXC、PostgreSQL 或
|
|
||||||
K3s lifecycle）：
|
|
||||||
|
|
||||||
```bash
|
|
||||||
ansible-playbook flux.yml
|
|
||||||
```
|
|
||||||
|
|
||||||
K3s 外部 datastore URI 由运行时 `SANDBOX_K3S_DB_PASSWORD` 生成，密码在 URI 中
|
K3s 外部 datastore URI 由运行时 `SANDBOX_K3S_DB_PASSWORD` 生成，密码在 URI 中
|
||||||
进行 URL 编码，最终仅持久化于节点 root 可读的 `/etc/rancher/k3s/config.yaml`
|
进行 URL 编码，最终仅持久化于节点 root 可读的 `/etc/rancher/k3s/config.yaml`
|
||||||
（mode `0600`）。首节点生成的 K3s join token 仅在同一次 Ansible run 内传给第二节点；
|
（mode `0600`）。首节点生成的 K3s join token 仅在同一次 Ansible run 内传给第二节点；
|
||||||
@@ -114,9 +106,7 @@ ansible-playbook verify.yml
|
|||||||
当前已经声明 LXC 生命周期、最小 OS baseline、PostgreSQL 和 K3s，包括系统级
|
当前已经声明 LXC 生命周期、最小 OS baseline、PostgreSQL 和 K3s，包括系统级
|
||||||
homelab CA trust。Flux `v2.9.5` controllers 与 root sync 也由 Ansible 通过 K3s
|
homelab CA trust。Flux `v2.9.5` controllers 与 root sync 也由 Ansible 通过 K3s
|
||||||
server manifests 管理；root 使用 homelab CA 访问公开 Gitea 仓库，不保存 Git token。
|
server manifests 管理；root 使用 homelab CA 访问公开 Gitea 仓库，不保存 Git token。
|
||||||
集群内 workload 由 `clusters/sandbox/` 分阶段纳入 Flux。root Kustomization 的健康检查
|
集群内 workload 由 `clusters/sandbox/` 分阶段纳入 Flux。
|
||||||
timeout 为 40 分钟，用于覆盖 Kata 等首次安装时会逐节点重启 K3s 的子
|
|
||||||
Kustomization；各子项仍保留自己的更短 timeout，故障会在对应子项先行暴露。
|
|
||||||
|
|
||||||
## SPIRE 跨集群 bootstrap
|
## SPIRE 跨集群 bootstrap
|
||||||
|
|
||||||
|
|||||||
@@ -1,6 +0,0 @@
|
|||||||
---
|
|
||||||
- name: Reconcile Flux controllers and root sync
|
|
||||||
hosts: sandbox1
|
|
||||||
gather_facts: false
|
|
||||||
roles:
|
|
||||||
- sandbox_flux
|
|
||||||
+1
-1
@@ -35,5 +35,5 @@ spec:
|
|||||||
sourceRef:
|
sourceRef:
|
||||||
kind: GitRepository
|
kind: GitRepository
|
||||||
name: flux-system
|
name: flux-system
|
||||||
timeout: 40m
|
timeout: 3m
|
||||||
wait: true
|
wait: true
|
||||||
|
|||||||
@@ -1,103 +1,51 @@
|
|||||||
# Gitea dynamic runner
|
# Gitea dynamic runner controller
|
||||||
|
|
||||||
本目录部署单副本 Go controller，在同一进程运行 RunnerService scheduler、原生
|
此目录只管理 homelab 中的 controller 部署。controller、worker、Cloud Hypervisor
|
||||||
Kubernetes Pod worker、OpenSandbox VM worker 和 SPIFFE mTLS facade：
|
launcher 和 guest runner 的源码与发布位于独立仓库
|
||||||
|
`panxiao81/gitea-dynamic-runner`。
|
||||||
|
|
||||||
|
当前 bootstrap controller 接收 Gitea `workflow_job` webhook，将 `[self-hosted, pod]` 和
|
||||||
|
`[self-hosted, vm]` 的 queued job 分别发布到 NATS。Pod worker 在本集群创建一次性
|
||||||
|
privileged host runner；Docker、BuildKit 和 kind 由 workflow 自行 setup。内部
|
||||||
|
endpoint：
|
||||||
|
|
||||||
```text
|
```text
|
||||||
Gitea RunnerService -> scheduler -> JetStream ci.runner.pod
|
http://dynamic-runner-controller.dynamic-runner.svc.cluster.local:8787/webhook
|
||||||
|
|
|
||||||
v
|
|
||||||
homelab Kubernetes Pod
|
|
||||||
|
|
|
||||||
SPIFFE mTLS RunnerService facade
|
|
||||||
|
|
|
||||||
v
|
|
||||||
Gitea
|
|
||||||
```
|
```
|
||||||
|
|
||||||
Pod backend 不经过 OpenSandbox。VM backend 后续启用时才访问 VyOS 暴露的
|
OpenBao 路径：
|
||||||
OpenSandbox Lifecycle API；本目录不修改 sandbox 平台侧 ESO、Bao Terraform 或
|
|
||||||
OpenSandbox chart 所有权边界。
|
|
||||||
|
|
||||||
## Canary 安全边界
|
- `kv/k8s/nats.ci_producer_password`：已有 NATS producer 密码。
|
||||||
|
- `kv/k8s/nats.ci_worker_password`：已有 NATS worker 密码。
|
||||||
|
- `kv/k8s/dynamic-runner.webhook_secret`：Gitea webhook HMAC secret。
|
||||||
|
- `kv/k8s/gitea-runner.token`：现有 instance runner registration token。
|
||||||
|
|
||||||
- Deployment 为单副本，滚动策略固定 `maxSurge: 1`、`maxUnavailable: 0`，保证新
|
首期 controller 与 runner 镜像由 laptop 本机构建后导入 k3s containerd，作为 CI
|
||||||
facade Ready 后才终止旧实例。scheduler 必须通过 Kubernetes Lease 保持单 leader，
|
发布链路建立前的 bootstrap。部署使用 `imagePullPolicy: Never`。正式发布 workflow
|
||||||
不能依赖 Recreate 避免重复领取。
|
获得专用 SPIFFE ID 后，必须将 image 改为 zot digest 并移除本地导入步骤。
|
||||||
- scheduler 使用一个 runner registration，并按总容量启动并发 `FetchTask` goroutine；
|
|
||||||
`POD_CAPACITY=4` 与 `VM_CAPACITY=1` 分别限制两个 durable consumer 和 backend
|
|
||||||
admission pool。池满时 assignment 保持 JetStream pending，任一 backend 不占用
|
|
||||||
另一方的执行槽位，也不会创建超出容量的 workload。
|
|
||||||
- rollout 重叠期间只有持有 `Lease/dynamic-runner-scheduler` 的 controller 执行
|
|
||||||
`FetchTask`；所有 Ready 实例都可通过 backend metadata 恢复 claim 并服务 facade。
|
|
||||||
- executor 镜像使用 digest；Pod 以 UID 2000 运行，SPIRE `ClusterStaticEntry` 同时绑定
|
|
||||||
具体 Pod UID 与 `unix:uid:2000`。
|
|
||||||
- facade 通过仅内网可路由的 `192.168.10.127:30443` NodePort 提供给 sandbox executor；
|
|
||||||
双方使用 Workload API X509-SVID mTLS，并按 SPIFFE ID 而不是 IP/DNS 名验证服务端。
|
|
||||||
该入口不经过公网或 Cloudflare Tunnel。
|
|
||||||
- controller 的 SPIFFE ID 固定为
|
|
||||||
`spiffe://ddupan.top/ns/dynamic-runner/sa/dynamic-runner-controller`。
|
|
||||||
- NATS 保留既有最小权限分离：`ci-producer` 仅 publish，`ci-worker` 仅 pull/ACK。
|
|
||||||
|
|
||||||
facade claim registry 在启动时从 Pod/OpenSandbox metadata 恢复；scheduler leadership
|
## 身份绑定
|
||||||
由 Kubernetes Lease 持久化协调。Deployment 仍保持 `replicas: 1`，滚动更新期间允许
|
|
||||||
一个额外 Pod 提供 facade 连续性。
|
|
||||||
|
|
||||||
## Secret 边界
|
queued webhook 只负责创建没有业务身份的 Pod。runner 实际领取任务后，Gitea 的
|
||||||
|
`in_progress` webhook 会携带实际 `runner_name`；controller 将 binding 消息发布到
|
||||||
|
NATS，Pod worker 再给对应 Pod 添加：
|
||||||
|
|
||||||
`ExternalSecret/dynamic-runner` 从既有 `ClusterSecretStore/openbao` 读取：
|
```text
|
||||||
|
ci.ddupan.top/identity-bound=true
|
||||||
- `kv/k8s/nats`：producer/worker 密码；
|
ci.ddupan.top/spiffe-path=<owner>/<repository>/<percent-encoded-job-name>
|
||||||
- `kv/k8s/opensandbox-api:api_key`：保留给后续 VM worker；
|
|
||||||
- `kv/k8s/gitea-runner:token`：保留的 runner registration token；
|
|
||||||
- `kv/k8s/dynamic-runner`：scheduler UUID/token、facade HMAC key 和回滚所需 webhook secret。
|
|
||||||
|
|
||||||
scheduler credential 由官方 Gitea Runner v3.5.0 一次注册生成；它只挂载到 controller，
|
|
||||||
不会进入 executor。facade capability key 至少 32 字节，controller 为每个 assignment
|
|
||||||
确定性生成独立 capability。不要打印 Secret、创建静态 Bao token或把 credential 写入
|
|
||||||
Git。OpenBao 写入使用本机 SPIFFE JWT 换取的短期 `local-development` token。
|
|
||||||
|
|
||||||
## Workflow 身份与依赖配置
|
|
||||||
|
|
||||||
workflow 可复用 [`panxiao81/ci-actions`](https://git.ddupan.top/panxiao81/ci-actions)
|
|
||||||
中的 `spiffe-openbao-login@v1` 和 `setup-nexus@v1`。这不改变 runner 的权限边界：
|
|
||||||
runner 仅提供 Node.js 20、`spire-agent` 与 Workload API socket，workflow 负责声明 Bao
|
|
||||||
role、audience 和具体用途，目标服务 policy 决定是否授权。短期 Bao token 会进入
|
|
||||||
Actions job 临时文件，因此这些 Action 只允许在本目录管理的一次性 Pod/VM executor
|
|
||||||
中使用，不能迁移到共享或持久 runner。
|
|
||||||
|
|
||||||
匿名读取 Nexus public repository 只需 `setup-nexus@v1`，不应为了依赖下载额外申请
|
|
||||||
Bao 凭据；需要发布制品时再为对应 repository 建立独立 service account 与最小权限
|
|
||||||
policy。
|
|
||||||
|
|
||||||
## 首次验收
|
|
||||||
|
|
||||||
合并后先观察 Flux 与 controller，不要立即开启 VM：
|
|
||||||
|
|
||||||
```bash
|
|
||||||
flux reconcile kustomization dynamic-runner --with-source
|
|
||||||
kubectl -n dynamic-runner wait externalsecret/dynamic-runner \
|
|
||||||
--for=condition=Ready --timeout=2m
|
|
||||||
kubectl -n dynamic-runner rollout status deploy/dynamic-runner-controller --timeout=5m
|
|
||||||
kubectl -n dynamic-runner logs deploy/dynamic-runner-controller -f
|
|
||||||
```
|
```
|
||||||
|
|
||||||
确认 scheduler 只领取一条 `[self-hosted,pod]` task，然后验证：
|
`ClusterSPIFFEID/gitea-dynamic-runner` 只匹配已经绑定的 Pod，并签发
|
||||||
|
`spiffe://ddupan.top/ci/<owner>/<repository>/<job-name>`。runner 的 job-start hook 在
|
||||||
|
SVID 可用之前不会放行第一步，因此不能根据 queued 事件错配身份。
|
||||||
|
|
||||||
1. assignment message 进入并离开 durable `pod` consumer；
|
每个 runner Pod 使用 `gitea-dynamic-runner` ServiceAccount。该 ServiceAccount 没有
|
||||||
2. `gitea-task-<task-id>` Pod 创建，取得实际 Pod UID；
|
Kubernetes API 权限；只有 `dynamic-runner-pod-worker` ServiceAccount 能在本 namespace
|
||||||
3. 同名 `ClusterStaticEntry` 的 parent ID 包含该 UID，SPIFFE ID 使用 repository/job key；
|
create/get/patch/delete Pod。
|
||||||
4. 官方 runner v3.5.0 经 facade claim 精确 task，Gitea 实时收到日志和终态；
|
|
||||||
5. Pod consumer 达到 capacity 时 VM consumer 仍可独立接受任务。
|
|
||||||
|
|
||||||
Pod 与 VM 都在 Gitea 接受终态后先把 terminal marker 写入各自 backend metadata，再由
|
长期实现将由兼容 Gitea RunnerService 的 scheduler 直接领取 task，再交给 Pod/VM
|
||||||
lifecycle reconciler 删除执行器。首次 VM 测试仍须观察 BatchSandbox、Pod 与
|
executor；届时删除 webhook、临时 runner 注册和 identity binding 消息。跟踪见
|
||||||
ClusterStaticEntry 全部消失；完整自动清理通过前不得提高 `POD_CAPACITY` 或
|
`panxiao81/gitea-dynamic-runner` issue #7。
|
||||||
`VM_CAPACITY`。
|
|
||||||
|
|
||||||
## 回滚
|
Gitea webhook 只订阅 `workflow_job`，content type 使用 JSON，secret 与 Bao 中值
|
||||||
|
一致。不要启用 `send_everything`，否则 controller 会收到无关仓库事件。
|
||||||
若 controller 在领取 task 前失败，回滚到前一 commit 的 Python controller/Pod worker
|
|
||||||
manifests。若已经创建 `gitea-task-*` Pod，先保留现场并核对 Gitea task 状态，不能直接
|
|
||||||
重启 scheduler 造成重复执行。scheduler registration 和 capability key 保留在 Bao，
|
|
||||||
回滚不需要删除或打印它们。
|
|
||||||
|
|||||||
@@ -1,16 +1,17 @@
|
|||||||
apiVersion: spire.spiffe.io/v1alpha1
|
apiVersion: spire.spiffe.io/v1alpha1
|
||||||
kind: ClusterSPIFFEID
|
kind: ClusterSPIFFEID
|
||||||
metadata:
|
metadata:
|
||||||
name: dynamic-runner-controller
|
name: gitea-dynamic-runner
|
||||||
spec:
|
spec:
|
||||||
className: spire-mgmt-spire
|
className: spire-mgmt-spire
|
||||||
spiffeIDTemplate: spiffe://ddupan.top/ns/dynamic-runner/sa/dynamic-runner-controller
|
spiffeIDTemplate: 'spiffe://{{ .TrustDomain }}/ci/{{ index .PodMeta.Annotations "ci.ddupan.top/spiffe-path" }}'
|
||||||
namespaceSelector:
|
namespaceSelector:
|
||||||
matchLabels:
|
matchLabels:
|
||||||
kubernetes.io/metadata.name: dynamic-runner
|
kubernetes.io/metadata.name: dynamic-runner
|
||||||
podSelector:
|
podSelector:
|
||||||
matchLabels:
|
matchLabels:
|
||||||
app.kubernetes.io/name: dynamic-runner-controller
|
app.kubernetes.io/name: gitea-dynamic-runner
|
||||||
|
ci.ddupan.top/identity-bound: "true"
|
||||||
workloadSelectorTemplates:
|
workloadSelectorTemplates:
|
||||||
- k8s:ns:dynamic-runner
|
- k8s:ns:dynamic-runner
|
||||||
- k8s:sa:dynamic-runner-controller
|
- k8s:sa:gitea-dynamic-runner
|
||||||
|
|||||||
@@ -5,11 +5,6 @@ metadata:
|
|||||||
namespace: dynamic-runner
|
namespace: dynamic-runner
|
||||||
spec:
|
spec:
|
||||||
replicas: 1
|
replicas: 1
|
||||||
strategy:
|
|
||||||
type: RollingUpdate
|
|
||||||
rollingUpdate:
|
|
||||||
maxSurge: 1
|
|
||||||
maxUnavailable: 0
|
|
||||||
selector:
|
selector:
|
||||||
matchLabels:
|
matchLabels:
|
||||||
app.kubernetes.io/name: dynamic-runner-controller
|
app.kubernetes.io/name: dynamic-runner-controller
|
||||||
@@ -19,10 +14,17 @@ spec:
|
|||||||
app.kubernetes.io/name: dynamic-runner-controller
|
app.kubernetes.io/name: dynamic-runner-controller
|
||||||
spec:
|
spec:
|
||||||
serviceAccountName: dynamic-runner-controller
|
serviceAccountName: dynamic-runner-controller
|
||||||
|
automountServiceAccountToken: false
|
||||||
initContainers:
|
initContainers:
|
||||||
- name: fetch-internal-ca
|
- name: fetch-internal-ca
|
||||||
image: curlimages/curl:8.16.0@sha256:463eaf6072688fe96ac64fa623fe73e1dbe25d8ad6c34404a669ad3ce1f104b6
|
image: curlimages/curl:8.16.0@sha256:463eaf6072688fe96ac64fa623fe73e1dbe25d8ad6c34404a669ad3ce1f104b6
|
||||||
args: [--fail, --silent, --show-error, --output, /trust/ca.pem, https://bao.ad.ddupan.top:8200/v1/pki/ca/pem]
|
args:
|
||||||
|
- --fail
|
||||||
|
- --silent
|
||||||
|
- --show-error
|
||||||
|
- --output
|
||||||
|
- /trust/ca.pem
|
||||||
|
- https://bao.ad.ddupan.top:8200/v1/pki/ca/pem
|
||||||
securityContext:
|
securityContext:
|
||||||
allowPrivilegeEscalation: false
|
allowPrivilegeEscalation: false
|
||||||
capabilities:
|
capabilities:
|
||||||
@@ -38,89 +40,40 @@ spec:
|
|||||||
mountPath: /trust
|
mountPath: /trust
|
||||||
containers:
|
containers:
|
||||||
- name: controller
|
- name: controller
|
||||||
image: zot.ad.ddupan.top/panxiao81/gitea-dynamic-runner-controller@sha256:9372ef869353dc7f79186bfae4191c1be0e52159c166489e76efd339e60c51b8
|
# Bootstrap import on laptop. Replace with a zot digest after the
|
||||||
imagePullPolicy: IfNotPresent
|
# repository's image publishing workflow has a dedicated identity.
|
||||||
args: [controller]
|
image: gitea-dynamic-runner-controller:0.3.0-bootstrap
|
||||||
|
imagePullPolicy: Never
|
||||||
env:
|
env:
|
||||||
- name: COMPONENTS
|
|
||||||
value: scheduler,pod-worker,vm-worker
|
|
||||||
- name: GITEA_INSTANCE_URL
|
|
||||||
value: https://git.ddupan.top
|
|
||||||
- name: GITEA_RUNNER_UUID_FILE
|
|
||||||
value: /run/dynamic-runner-secrets/scheduler-uuid
|
|
||||||
- name: GITEA_RUNNER_TOKEN_FILE
|
|
||||||
value: /run/dynamic-runner-secrets/scheduler-token
|
|
||||||
- name: NATS_URL
|
- name: NATS_URL
|
||||||
value: tls://nats.ad.ddupan.top:4222
|
value: tls://nats.ad.ddupan.top:4222
|
||||||
- name: NATS_CA_FILE
|
- name: NATS_CA_FILE
|
||||||
value: /run/trust/ca.pem
|
value: /run/trust/ca.pem
|
||||||
- name: NATS_PRODUCER_PASSWORD_FILE
|
- name: NATS_PASSWORD_FILE
|
||||||
value: /run/dynamic-runner-secrets/nats-password
|
value: /run/dynamic-runner-secrets/nats-password
|
||||||
- name: NATS_WORKER_PASSWORD_FILE
|
- name: WEBHOOK_SECRET_FILE
|
||||||
value: /run/dynamic-runner-secrets/nats-worker-password
|
value: /run/dynamic-runner-secrets/webhook-secret
|
||||||
- name: RUNNER_FACADE_CAPABILITY_KEY_FILE
|
|
||||||
value: /run/dynamic-runner-secrets/facade-capability-key
|
|
||||||
- name: RUNNER_FACADE_LISTEN
|
|
||||||
value: :8443
|
|
||||||
- name: RUNNER_FACADE_URL
|
|
||||||
value: https://192.168.10.127:30443
|
|
||||||
- name: RUNNER_FACADE_SPIFFE_ID
|
|
||||||
value: spiffe://ddupan.top/ns/dynamic-runner/sa/dynamic-runner-controller
|
|
||||||
- name: SPIFFE_ENDPOINT_SOCKET
|
|
||||||
value: unix:///run/spire/agent-sockets/spire-agent.sock
|
|
||||||
- name: POD_NAMESPACE
|
|
||||||
valueFrom:
|
|
||||||
fieldRef:
|
|
||||||
fieldPath: metadata.namespace
|
|
||||||
- name: POD_EXECUTOR_IMAGE
|
|
||||||
value: zot.ad.ddupan.top/panxiao81/gitea-dynamic-runner-runner@sha256:08f83c9993645dce376140fdde7e29e809795166b2bb32e50cebdbefaf3cf303
|
|
||||||
- name: POD_SERVICE_ACCOUNT
|
|
||||||
value: gitea-dynamic-runner
|
|
||||||
- name: POD_EXECUTOR_UID
|
|
||||||
value: "2000"
|
|
||||||
- name: POD_CAPACITY
|
|
||||||
value: "4"
|
|
||||||
- name: OPENSANDBOX_API
|
|
||||||
value: http://10.60.0.13:8080
|
|
||||||
- name: OPENSANDBOX_API_KEY_FILE
|
|
||||||
value: /run/dynamic-runner-secrets/opensandbox-api-key
|
|
||||||
- name: OPENSANDBOX_POOL
|
|
||||||
value: ci-vm
|
|
||||||
- name: VM_CAPACITY
|
|
||||||
value: "1"
|
|
||||||
- name: VM_RUNNER_LABEL
|
|
||||||
value: vm-dev
|
|
||||||
- name: VM_TIMEOUT_SECONDS
|
|
||||||
value: "14400"
|
|
||||||
- name: SPIRE_CLUSTER
|
|
||||||
value: homelab
|
|
||||||
- name: SPIRE_CLASS
|
|
||||||
value: spire-mgmt-spire
|
|
||||||
- name: SPIRE_AGENT_ID
|
|
||||||
value: spiffe://ddupan.top/spire/agent/k8s_psat/homelab/cd2d0233-c4ea-4031-8327-e7e359e766dd
|
|
||||||
- name: SSL_CERT_FILE
|
|
||||||
value: /run/trust/ca.pem
|
|
||||||
ports:
|
ports:
|
||||||
- name: facade
|
- name: http
|
||||||
containerPort: 8443
|
containerPort: 8787
|
||||||
readinessProbe:
|
readinessProbe:
|
||||||
httpGet: null
|
httpGet:
|
||||||
tcpSocket:
|
path: /healthz
|
||||||
port: facade
|
port: http
|
||||||
periodSeconds: 5
|
periodSeconds: 5
|
||||||
livenessProbe:
|
livenessProbe:
|
||||||
httpGet: null
|
httpGet:
|
||||||
tcpSocket:
|
path: /healthz
|
||||||
port: facade
|
port: http
|
||||||
initialDelaySeconds: 15
|
initialDelaySeconds: 10
|
||||||
periodSeconds: 10
|
periodSeconds: 10
|
||||||
resources:
|
resources:
|
||||||
requests:
|
requests:
|
||||||
cpu: 50m
|
cpu: 25m
|
||||||
memory: 64Mi
|
memory: 32Mi
|
||||||
limits:
|
limits:
|
||||||
cpu: 500m
|
cpu: 250m
|
||||||
memory: 256Mi
|
memory: 128Mi
|
||||||
securityContext:
|
securityContext:
|
||||||
allowPrivilegeEscalation: false
|
allowPrivilegeEscalation: false
|
||||||
capabilities:
|
capabilities:
|
||||||
@@ -138,9 +91,6 @@ spec:
|
|||||||
- name: trust
|
- name: trust
|
||||||
mountPath: /run/trust
|
mountPath: /run/trust
|
||||||
readOnly: true
|
readOnly: true
|
||||||
- name: spire-agent-socket
|
|
||||||
mountPath: /run/spire/agent-sockets
|
|
||||||
readOnly: true
|
|
||||||
securityContext:
|
securityContext:
|
||||||
fsGroup: 65532
|
fsGroup: 65532
|
||||||
fsGroupChangePolicy: OnRootMismatch
|
fsGroupChangePolicy: OnRootMismatch
|
||||||
@@ -154,7 +104,3 @@ spec:
|
|||||||
- name: trust
|
- name: trust
|
||||||
emptyDir:
|
emptyDir:
|
||||||
sizeLimit: 1Mi
|
sizeLimit: 1Mi
|
||||||
- name: spire-agent-socket
|
|
||||||
csi:
|
|
||||||
driver: csi.spiffe.io
|
|
||||||
readOnly: true
|
|
||||||
|
|||||||
@@ -20,10 +20,6 @@ spec:
|
|||||||
remoteRef:
|
remoteRef:
|
||||||
key: k8s/nats
|
key: k8s/nats
|
||||||
property: ci_worker_password
|
property: ci_worker_password
|
||||||
- secretKey: opensandbox-api-key
|
|
||||||
remoteRef:
|
|
||||||
key: k8s/opensandbox-api
|
|
||||||
property: api_key
|
|
||||||
- secretKey: webhook-secret
|
- secretKey: webhook-secret
|
||||||
remoteRef:
|
remoteRef:
|
||||||
key: k8s/dynamic-runner
|
key: k8s/dynamic-runner
|
||||||
@@ -32,15 +28,3 @@ spec:
|
|||||||
remoteRef:
|
remoteRef:
|
||||||
key: k8s/gitea-runner
|
key: k8s/gitea-runner
|
||||||
property: token
|
property: token
|
||||||
- secretKey: scheduler-uuid
|
|
||||||
remoteRef:
|
|
||||||
key: k8s/dynamic-runner
|
|
||||||
property: scheduler_uuid
|
|
||||||
- secretKey: scheduler-token
|
|
||||||
remoteRef:
|
|
||||||
key: k8s/dynamic-runner
|
|
||||||
property: scheduler_token
|
|
||||||
- secretKey: facade-capability-key
|
|
||||||
remoteRef:
|
|
||||||
key: k8s/dynamic-runner
|
|
||||||
property: facade_capability_key
|
|
||||||
|
|||||||
@@ -6,4 +6,5 @@ resources:
|
|||||||
- rbac.yaml
|
- rbac.yaml
|
||||||
- clusterspiffeid.yaml
|
- clusterspiffeid.yaml
|
||||||
- deployment.yaml
|
- deployment.yaml
|
||||||
|
- pod-worker-deployment.yaml
|
||||||
- service.yaml
|
- service.yaml
|
||||||
|
|||||||
@@ -0,0 +1,100 @@
|
|||||||
|
apiVersion: apps/v1
|
||||||
|
kind: Deployment
|
||||||
|
metadata:
|
||||||
|
name: dynamic-runner-pod-worker
|
||||||
|
namespace: dynamic-runner
|
||||||
|
spec:
|
||||||
|
replicas: 1
|
||||||
|
selector:
|
||||||
|
matchLabels:
|
||||||
|
app.kubernetes.io/name: dynamic-runner-pod-worker
|
||||||
|
template:
|
||||||
|
metadata:
|
||||||
|
labels:
|
||||||
|
app.kubernetes.io/name: dynamic-runner-pod-worker
|
||||||
|
spec:
|
||||||
|
serviceAccountName: dynamic-runner-pod-worker
|
||||||
|
initContainers:
|
||||||
|
- name: fetch-internal-ca
|
||||||
|
image: curlimages/curl:8.16.0@sha256:463eaf6072688fe96ac64fa623fe73e1dbe25d8ad6c34404a669ad3ce1f104b6
|
||||||
|
args:
|
||||||
|
- --fail
|
||||||
|
- --silent
|
||||||
|
- --show-error
|
||||||
|
- --output
|
||||||
|
- /trust/ca.pem
|
||||||
|
- https://bao.ad.ddupan.top:8200/v1/pki/ca/pem
|
||||||
|
securityContext:
|
||||||
|
allowPrivilegeEscalation: false
|
||||||
|
capabilities:
|
||||||
|
drop: [ALL]
|
||||||
|
readOnlyRootFilesystem: true
|
||||||
|
runAsNonRoot: true
|
||||||
|
runAsUser: 101
|
||||||
|
runAsGroup: 102
|
||||||
|
seccompProfile:
|
||||||
|
type: RuntimeDefault
|
||||||
|
volumeMounts:
|
||||||
|
- name: trust
|
||||||
|
mountPath: /trust
|
||||||
|
containers:
|
||||||
|
- name: pod-worker
|
||||||
|
image: gitea-dynamic-runner-controller:0.3.0-bootstrap
|
||||||
|
imagePullPolicy: Never
|
||||||
|
command: [/venv/bin/gitea-dynamic-runner-pod-worker]
|
||||||
|
env:
|
||||||
|
- name: NATS_URL
|
||||||
|
value: tls://nats.ad.ddupan.top:4222
|
||||||
|
- name: NATS_CA_FILE
|
||||||
|
value: /run/trust/ca.pem
|
||||||
|
- name: NATS_PASSWORD_FILE
|
||||||
|
value: /run/dynamic-runner-secrets/nats-worker-password
|
||||||
|
- name: RUNNER_NAMESPACE
|
||||||
|
valueFrom:
|
||||||
|
fieldRef:
|
||||||
|
fieldPath: metadata.namespace
|
||||||
|
- name: RUNNER_IMAGE
|
||||||
|
value: zot.ad.ddupan.top/panxiao81/gitea-dynamic-runner-runner@sha256:4c61f6315453d68a827ee9542f1345ed86576eaaf62325eb803c8fe3f06ddf2a
|
||||||
|
- name: RUNNER_SERVICE_ACCOUNT
|
||||||
|
value: gitea-dynamic-runner
|
||||||
|
- name: RUNNER_TOKEN_SECRET
|
||||||
|
value: dynamic-runner
|
||||||
|
- name: RUNNER_CAPACITY
|
||||||
|
value: "4"
|
||||||
|
resources:
|
||||||
|
requests:
|
||||||
|
cpu: 25m
|
||||||
|
memory: 32Mi
|
||||||
|
limits:
|
||||||
|
cpu: 250m
|
||||||
|
memory: 128Mi
|
||||||
|
securityContext:
|
||||||
|
allowPrivilegeEscalation: false
|
||||||
|
capabilities:
|
||||||
|
drop: [ALL]
|
||||||
|
readOnlyRootFilesystem: true
|
||||||
|
runAsNonRoot: true
|
||||||
|
runAsUser: 65532
|
||||||
|
runAsGroup: 65532
|
||||||
|
seccompProfile:
|
||||||
|
type: RuntimeDefault
|
||||||
|
volumeMounts:
|
||||||
|
- name: secret
|
||||||
|
mountPath: /run/dynamic-runner-secrets
|
||||||
|
readOnly: true
|
||||||
|
- name: trust
|
||||||
|
mountPath: /run/trust
|
||||||
|
readOnly: true
|
||||||
|
securityContext:
|
||||||
|
fsGroup: 65532
|
||||||
|
fsGroupChangePolicy: OnRootMismatch
|
||||||
|
seccompProfile:
|
||||||
|
type: RuntimeDefault
|
||||||
|
volumes:
|
||||||
|
- name: secret
|
||||||
|
secret:
|
||||||
|
secretName: dynamic-runner
|
||||||
|
defaultMode: 0400
|
||||||
|
- name: trust
|
||||||
|
emptyDir:
|
||||||
|
sizeLimit: 1Mi
|
||||||
@@ -6,6 +6,12 @@ metadata:
|
|||||||
---
|
---
|
||||||
apiVersion: v1
|
apiVersion: v1
|
||||||
kind: ServiceAccount
|
kind: ServiceAccount
|
||||||
|
metadata:
|
||||||
|
name: dynamic-runner-pod-worker
|
||||||
|
namespace: dynamic-runner
|
||||||
|
---
|
||||||
|
apiVersion: v1
|
||||||
|
kind: ServiceAccount
|
||||||
metadata:
|
metadata:
|
||||||
name: gitea-dynamic-runner
|
name: gitea-dynamic-runner
|
||||||
namespace: dynamic-runner
|
namespace: dynamic-runner
|
||||||
@@ -13,48 +19,23 @@ metadata:
|
|||||||
apiVersion: rbac.authorization.k8s.io/v1
|
apiVersion: rbac.authorization.k8s.io/v1
|
||||||
kind: Role
|
kind: Role
|
||||||
metadata:
|
metadata:
|
||||||
name: dynamic-runner-controller
|
name: dynamic-runner-pod-worker
|
||||||
namespace: dynamic-runner
|
namespace: dynamic-runner
|
||||||
rules:
|
rules:
|
||||||
- apiGroups: [""]
|
- apiGroups: [""]
|
||||||
resources: [pods]
|
resources: [pods]
|
||||||
verbs: [create, get, list, watch, patch, delete]
|
verbs: [create, get, patch, delete]
|
||||||
- apiGroups: [coordination.k8s.io]
|
|
||||||
resources: [leases]
|
|
||||||
verbs: [create, get, list, watch, update, patch]
|
|
||||||
---
|
---
|
||||||
apiVersion: rbac.authorization.k8s.io/v1
|
apiVersion: rbac.authorization.k8s.io/v1
|
||||||
kind: RoleBinding
|
kind: RoleBinding
|
||||||
metadata:
|
metadata:
|
||||||
name: dynamic-runner-controller
|
name: dynamic-runner-pod-worker
|
||||||
namespace: dynamic-runner
|
namespace: dynamic-runner
|
||||||
roleRef:
|
roleRef:
|
||||||
apiGroup: rbac.authorization.k8s.io
|
apiGroup: rbac.authorization.k8s.io
|
||||||
kind: Role
|
kind: Role
|
||||||
name: dynamic-runner-controller
|
name: dynamic-runner-pod-worker
|
||||||
subjects:
|
subjects:
|
||||||
- kind: ServiceAccount
|
- kind: ServiceAccount
|
||||||
name: dynamic-runner-controller
|
name: dynamic-runner-pod-worker
|
||||||
namespace: dynamic-runner
|
|
||||||
---
|
|
||||||
apiVersion: rbac.authorization.k8s.io/v1
|
|
||||||
kind: ClusterRole
|
|
||||||
metadata:
|
|
||||||
name: dynamic-runner-spiffe-entries
|
|
||||||
rules:
|
|
||||||
- apiGroups: [spire.spiffe.io]
|
|
||||||
resources: [clusterstaticentries]
|
|
||||||
verbs: [create, get, delete]
|
|
||||||
---
|
|
||||||
apiVersion: rbac.authorization.k8s.io/v1
|
|
||||||
kind: ClusterRoleBinding
|
|
||||||
metadata:
|
|
||||||
name: dynamic-runner-spiffe-entries
|
|
||||||
roleRef:
|
|
||||||
apiGroup: rbac.authorization.k8s.io
|
|
||||||
kind: ClusterRole
|
|
||||||
name: dynamic-runner-spiffe-entries
|
|
||||||
subjects:
|
|
||||||
- kind: ServiceAccount
|
|
||||||
name: dynamic-runner-controller
|
|
||||||
namespace: dynamic-runner
|
namespace: dynamic-runner
|
||||||
|
|||||||
@@ -4,11 +4,9 @@ metadata:
|
|||||||
name: dynamic-runner-controller
|
name: dynamic-runner-controller
|
||||||
namespace: dynamic-runner
|
namespace: dynamic-runner
|
||||||
spec:
|
spec:
|
||||||
type: NodePort
|
|
||||||
selector:
|
selector:
|
||||||
app.kubernetes.io/name: dynamic-runner-controller
|
app.kubernetes.io/name: dynamic-runner-controller
|
||||||
ports:
|
ports:
|
||||||
- name: facade
|
- name: http
|
||||||
port: 8443
|
port: 8787
|
||||||
targetPort: facade
|
targetPort: http
|
||||||
nodePort: 30443
|
|
||||||
|
|||||||
@@ -5,7 +5,7 @@ existingSecret: gitea-runner-token
|
|||||||
existingSecretKey: token
|
existingSecretKey: token
|
||||||
|
|
||||||
statefulset:
|
statefulset:
|
||||||
replicas: 0
|
replicas: 1
|
||||||
timezone: Etc/UTC
|
timezone: Etc/UTC
|
||||||
serviceAccountName: gitea-actions
|
serviceAccountName: gitea-actions
|
||||||
extraVolumes:
|
extraVolumes:
|
||||||
|
|||||||
@@ -1,66 +0,0 @@
|
|||||||
# OpenSandbox Gitea runner Pools
|
|
||||||
|
|
||||||
本目录只部署零预热的 `ci-vm` Pool、sandbox 内的 SPIFFE identity controller，以及仅供
|
|
||||||
内网 VyOS 转发的 OpenSandbox NodePort。`ci-vm` 使用 `kata-clh-runtime-rs`，每个 VM
|
|
||||||
执行单一任务并在结束后删除。普通 Pod job 直接运行在 homelab，不进入 OpenSandbox。
|
|
||||||
|
|
||||||
Pool 中 task-executor 接收 Lifecycle API 下发的进程环境。guest-local SPIRE Agent 用
|
|
||||||
Pod-bound PSAT 向中央 SPIRE 注册；identity controller 从 BatchSandbox allocation
|
|
||||||
取得真实 Pod UID，再创建精确的 `ClusterStaticEntry`。runner 只有拿到请求中的完整
|
|
||||||
repository/task SVID 后才领取一次性 Gitea registration token。
|
|
||||||
|
|
||||||
这些 `ClusterStaticEntry` 位于 sandbox 集群，由 central SPIRE Server 内的
|
|
||||||
`spire-controller-manager-sandbox` 通过受限 external kubeconfig reconcile。必须在
|
|
||||||
`platform/spire/values.yaml` 显式启用 external controller-manager 的
|
|
||||||
`reconcile.clusterStaticEntries`（chart 默认关闭）；仅看到 CR 存在但没有 status，不算
|
|
||||||
身份链路就绪。
|
|
||||||
|
|
||||||
## 部署依赖
|
|
||||||
|
|
||||||
- OpenSandbox chart 和 CRD 已 Ready；
|
|
||||||
- RuntimeClass `kata-clh-runtime-rs` 已存在；
|
|
||||||
- 中央 SPIRE 已发布 `ConfigMap/opensandbox/spire-bundle-pem`；
|
|
||||||
- VyOS `10.60.0.13:8080` 转发 sandbox1/2 的 NodePort `30080`；
|
|
||||||
- runner/controller 镜像均使用 Zot digest，而不是可变 tag。
|
|
||||||
|
|
||||||
本目录不读取 OpenBao，也不修改 OpenSandbox 平台侧 ExternalSecret、ClusterSecretStore
|
|
||||||
或 Bao policy。OpenSandbox API key 只存在于平台 server Secret 和 homelab controller
|
|
||||||
Secret，两边由各自身份读取同一 Bao 资源。
|
|
||||||
|
|
||||||
## 上线验收
|
|
||||||
|
|
||||||
先确认 Secret 和 API 认证，命令不得输出 key：
|
|
||||||
|
|
||||||
```bash
|
|
||||||
kubectl -n dynamic-runner wait externalsecret/dynamic-runner --for=condition=Ready --timeout=2m
|
|
||||||
kubectl -n dynamic-runner exec deploy/dynamic-runner-controller -- \
|
|
||||||
wget -qO- http://10.60.0.13:8080/health
|
|
||||||
```
|
|
||||||
|
|
||||||
然后触发一个 `runs-on: [self-hosted, vm]` 的最小 workflow，并同时观察：
|
|
||||||
|
|
||||||
```bash
|
|
||||||
kubectl -n dynamic-runner logs deploy/dynamic-runner-controller -f
|
|
||||||
kubectl -n opensandbox get pool,batchsandbox,pod -w
|
|
||||||
kubectl get clusterstaticentry -l app.kubernetes.io/component=opensandbox-identity -w
|
|
||||||
kubectl -n opensandbox logs deploy/opensandbox-identity -f
|
|
||||||
```
|
|
||||||
|
|
||||||
合格证据必须同时包含：
|
|
||||||
|
|
||||||
1. Lifecycle create 成功并选择 `ci-vm`；
|
|
||||||
2. 分配 Pod 的 `runtimeClassName` 为 `kata-clh-runtime-rs`；
|
|
||||||
3. entry 的 parent ID 包含该 Pod 的 UID，SPIFFE ID 使用 repository/task；
|
|
||||||
4. Gitea 显示临时 runner 带 `self-hosted,vm` labels 并完成真实任务；
|
|
||||||
5. 任务后 BatchSandbox、Pod、ClusterStaticEntry 和临时 runner 均消失。
|
|
||||||
|
|
||||||
清理超时对象时只调用 Lifecycle DELETE，不直接删除 Pool Pod。若 controller 已不可用，
|
|
||||||
可从 OpenSandbox API 按 metadata 定位 sandbox 后执行 DELETE；不要绕过 API 伪造状态。
|
|
||||||
|
|
||||||
## 故障定位
|
|
||||||
|
|
||||||
- `401`：检查 homelab ExternalSecret Ready 和文件挂载，不打印 Secret；
|
|
||||||
- `PoolCapacityExhausted`：检查 `ci-vm` 的 `poolMax` 及残留 BatchSandbox；
|
|
||||||
- runner 等待 SVID：核对 allocation Pod UID、ClusterStaticEntry parentID、guest Agent 日志；
|
|
||||||
- runner 等待 token：核对 `192.168.10.127:8787` 的 sandbox 到 homelab 路由；
|
|
||||||
- Docker 任务失败：检查 `docker` sidecar 和 `/run/docker/docker.sock` 的 group 2000。
|
|
||||||
@@ -1,31 +0,0 @@
|
|||||||
apiVersion: v1
|
|
||||||
kind: ConfigMap
|
|
||||||
metadata:
|
|
||||||
name: gitea-ci-spire-agent
|
|
||||||
namespace: opensandbox
|
|
||||||
data:
|
|
||||||
agent.conf: |
|
|
||||||
agent {
|
|
||||||
data_dir = "/run/spire/data"
|
|
||||||
log_level = "INFO"
|
|
||||||
server_address = "spire-server.ad.ddupan.top"
|
|
||||||
server_port = "8081"
|
|
||||||
socket_path = "/run/spire/agent-sockets/spire-agent.sock"
|
|
||||||
trust_bundle_path = "/run/spire/bundle/bundle.pem"
|
|
||||||
trust_domain = "ddupan.top"
|
|
||||||
}
|
|
||||||
|
|
||||||
plugins {
|
|
||||||
NodeAttestor "k8s_psat" {
|
|
||||||
plugin_data {
|
|
||||||
cluster = "sandbox-kata"
|
|
||||||
token_path = "/run/spire/tokens/token"
|
|
||||||
}
|
|
||||||
}
|
|
||||||
KeyManager "memory" {
|
|
||||||
plugin_data {}
|
|
||||||
}
|
|
||||||
WorkloadAttestor "unix" {
|
|
||||||
plugin_data {}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
@@ -1,41 +0,0 @@
|
|||||||
apiVersion: apps/v1
|
|
||||||
kind: Deployment
|
|
||||||
metadata:
|
|
||||||
name: opensandbox-identity
|
|
||||||
namespace: opensandbox
|
|
||||||
spec:
|
|
||||||
replicas: 1
|
|
||||||
selector:
|
|
||||||
matchLabels:
|
|
||||||
app.kubernetes.io/name: opensandbox-identity
|
|
||||||
template:
|
|
||||||
metadata:
|
|
||||||
labels:
|
|
||||||
app.kubernetes.io/name: opensandbox-identity
|
|
||||||
spec:
|
|
||||||
serviceAccountName: opensandbox-identity
|
|
||||||
containers:
|
|
||||||
- name: controller
|
|
||||||
image: zot.ad.ddupan.top/panxiao81/gitea-dynamic-runner-controller@sha256:dfbfaf2a7aa5951d1dc4e7f941a8cd05ceb4a860ac1ab23eb8be234861edcb55
|
|
||||||
imagePullPolicy: IfNotPresent
|
|
||||||
command: [/venv/bin/gitea-dynamic-runner-opensandbox-identity]
|
|
||||||
resources:
|
|
||||||
requests:
|
|
||||||
cpu: 10m
|
|
||||||
memory: 32Mi
|
|
||||||
limits:
|
|
||||||
cpu: 100m
|
|
||||||
memory: 96Mi
|
|
||||||
securityContext:
|
|
||||||
allowPrivilegeEscalation: false
|
|
||||||
capabilities:
|
|
||||||
drop: [ALL]
|
|
||||||
readOnlyRootFilesystem: true
|
|
||||||
runAsNonRoot: true
|
|
||||||
runAsUser: 65532
|
|
||||||
runAsGroup: 65532
|
|
||||||
seccompProfile:
|
|
||||||
type: RuntimeDefault
|
|
||||||
securityContext:
|
|
||||||
seccompProfile:
|
|
||||||
type: RuntimeDefault
|
|
||||||
@@ -1,22 +0,0 @@
|
|||||||
apiVersion: kustomize.config.k8s.io/v1beta1
|
|
||||||
kind: Kustomization
|
|
||||||
resources:
|
|
||||||
- service.yaml
|
|
||||||
- agent-config.yaml
|
|
||||||
- rbac.yaml
|
|
||||||
- identity-controller.yaml
|
|
||||||
- pools.yaml
|
|
||||||
patches:
|
|
||||||
# Pod jobs run directly in the homelab cluster. Keep OpenSandbox VM-only.
|
|
||||||
- target:
|
|
||||||
group: sandbox.opensandbox.io
|
|
||||||
version: v1alpha1
|
|
||||||
kind: Pool
|
|
||||||
name: ci-pod
|
|
||||||
patch: |-
|
|
||||||
apiVersion: sandbox.opensandbox.io/v1alpha1
|
|
||||||
kind: Pool
|
|
||||||
metadata:
|
|
||||||
name: ci-pod
|
|
||||||
namespace: opensandbox
|
|
||||||
$patch: delete
|
|
||||||
@@ -1,301 +0,0 @@
|
|||||||
---
|
|
||||||
apiVersion: sandbox.opensandbox.io/v1alpha1
|
|
||||||
kind: Pool
|
|
||||||
metadata:
|
|
||||||
name: ci-vm
|
|
||||||
namespace: opensandbox
|
|
||||||
labels:
|
|
||||||
app.kubernetes.io/name: gitea-ci-vm
|
|
||||||
app.kubernetes.io/component: runner-pool
|
|
||||||
spec:
|
|
||||||
capacitySpec:
|
|
||||||
bufferMax: 0
|
|
||||||
bufferMin: 0
|
|
||||||
poolMax: 2
|
|
||||||
poolMin: 0
|
|
||||||
recycleStrategy:
|
|
||||||
type: Delete
|
|
||||||
template:
|
|
||||||
metadata:
|
|
||||||
labels:
|
|
||||||
app.kubernetes.io/name: gitea-ci-vm
|
|
||||||
ci.ddupan.top/backend: vm
|
|
||||||
spec:
|
|
||||||
runtimeClassName: kata-clh-runtime-rs
|
|
||||||
serviceAccountName: gitea-ci
|
|
||||||
restartPolicy: Never
|
|
||||||
terminationGracePeriodSeconds: 30
|
|
||||||
shareProcessNamespace: true
|
|
||||||
securityContext:
|
|
||||||
fsGroup: 2000
|
|
||||||
fsGroupChangePolicy: OnRootMismatch
|
|
||||||
initContainers:
|
|
||||||
- name: task-executor-installer
|
|
||||||
image: sandbox-registry.cn-zhangjiakou.cr.aliyuncs.com/opensandbox/task-executor:v0.1.0
|
|
||||||
command: [/bin/sh, -c]
|
|
||||||
args:
|
|
||||||
- cp /workspace/server /opt/opensandbox/task-executor && chmod 0755 /opt/opensandbox/task-executor
|
|
||||||
volumeMounts:
|
|
||||||
- name: opensandbox-bin
|
|
||||||
mountPath: /opt/opensandbox
|
|
||||||
- name: execd-installer
|
|
||||||
image: sandbox-registry.cn-zhangjiakou.cr.aliyuncs.com/opensandbox/execd:v1.0.22
|
|
||||||
command: [/bin/sh, -c]
|
|
||||||
args:
|
|
||||||
- cp ./execd /opt/opensandbox/execd && cp ./bootstrap.sh /opt/opensandbox/bootstrap.sh && chmod 0755 /opt/opensandbox/execd /opt/opensandbox/bootstrap.sh
|
|
||||||
volumeMounts:
|
|
||||||
- name: opensandbox-bin
|
|
||||||
mountPath: /opt/opensandbox
|
|
||||||
containers:
|
|
||||||
- name: sandbox
|
|
||||||
image: zot.ad.ddupan.top/panxiao81/gitea-dynamic-runner-runner@sha256:08f83c9993645dce376140fdde7e29e809795166b2bb32e50cebdbefaf3cf303
|
|
||||||
imagePullPolicy: IfNotPresent
|
|
||||||
command: [/opt/opensandbox/task-executor]
|
|
||||||
args:
|
|
||||||
- -listen-addr=0.0.0.0:5758
|
|
||||||
- -log-dir=/tmp
|
|
||||||
- -data-dir=/tmp/tasks
|
|
||||||
env:
|
|
||||||
- name: SANDBOX_MAIN_CONTAINER
|
|
||||||
value: sandbox
|
|
||||||
- name: EXECD_ENVS
|
|
||||||
value: /opt/opensandbox/.env
|
|
||||||
- name: EXECD
|
|
||||||
value: /opt/opensandbox/execd
|
|
||||||
- name: GITEA_INSTANCE_URL
|
|
||||||
value: https://git.ddupan.top
|
|
||||||
- name: HOME
|
|
||||||
value: /data
|
|
||||||
- name: DOCKER_HOST
|
|
||||||
value: unix:///run/docker/docker.sock
|
|
||||||
ports:
|
|
||||||
- name: task-executor
|
|
||||||
containerPort: 5758
|
|
||||||
securityContext:
|
|
||||||
allowPrivilegeEscalation: false
|
|
||||||
capabilities:
|
|
||||||
drop: [ALL]
|
|
||||||
runAsNonRoot: true
|
|
||||||
runAsUser: 2000
|
|
||||||
runAsGroup: 2000
|
|
||||||
volumeMounts:
|
|
||||||
- name: opensandbox-bin
|
|
||||||
mountPath: /opt/opensandbox
|
|
||||||
- name: runner-data
|
|
||||||
mountPath: /data
|
|
||||||
- name: docker-socket
|
|
||||||
mountPath: /run/docker
|
|
||||||
- name: spire-socket
|
|
||||||
mountPath: /run/spire/agent-sockets
|
|
||||||
- name: spire-agent
|
|
||||||
image: ghcr.io/spiffe/spire-agent:1.15.3@sha256:41b0dcd8b258a69db9e2768292a060766fb76fd866e4bc925849981ea1b825ff
|
|
||||||
args: [-config, /run/spire/config/agent.conf]
|
|
||||||
securityContext:
|
|
||||||
allowPrivilegeEscalation: false
|
|
||||||
capabilities:
|
|
||||||
drop: [ALL]
|
|
||||||
volumeMounts:
|
|
||||||
- name: spire-config
|
|
||||||
mountPath: /run/spire/config
|
|
||||||
readOnly: true
|
|
||||||
- name: spire-bundle
|
|
||||||
mountPath: /run/spire/bundle
|
|
||||||
readOnly: true
|
|
||||||
- name: spire-token
|
|
||||||
mountPath: /run/spire/tokens
|
|
||||||
readOnly: true
|
|
||||||
- name: spire-data
|
|
||||||
mountPath: /run/spire/data
|
|
||||||
- name: spire-socket
|
|
||||||
mountPath: /run/spire/agent-sockets
|
|
||||||
- name: docker
|
|
||||||
image: docker.io/library/docker:29.1.5-dind
|
|
||||||
command: [/bin/sh, -c]
|
|
||||||
args:
|
|
||||||
- test -e /dev/kmsg || mknod /dev/kmsg c 1 11; exec dockerd --host=unix:///run/docker/docker.sock --group=2000 --storage-driver=overlay2
|
|
||||||
securityContext:
|
|
||||||
privileged: true
|
|
||||||
volumeMounts:
|
|
||||||
- name: docker-socket
|
|
||||||
mountPath: /run/docker
|
|
||||||
- name: docker-data
|
|
||||||
mountPath: /var/lib/docker
|
|
||||||
volumes:
|
|
||||||
- name: opensandbox-bin
|
|
||||||
emptyDir: {}
|
|
||||||
- name: runner-data
|
|
||||||
emptyDir: {}
|
|
||||||
- name: docker-data
|
|
||||||
emptyDir: {}
|
|
||||||
- name: docker-socket
|
|
||||||
emptyDir:
|
|
||||||
medium: Memory
|
|
||||||
- name: spire-data
|
|
||||||
emptyDir:
|
|
||||||
medium: Memory
|
|
||||||
- name: spire-socket
|
|
||||||
emptyDir:
|
|
||||||
medium: Memory
|
|
||||||
- name: spire-config
|
|
||||||
configMap:
|
|
||||||
name: gitea-ci-spire-agent
|
|
||||||
- name: spire-bundle
|
|
||||||
configMap:
|
|
||||||
name: spire-bundle-pem
|
|
||||||
- name: spire-token
|
|
||||||
projected:
|
|
||||||
sources:
|
|
||||||
- serviceAccountToken:
|
|
||||||
audience: spire-server
|
|
||||||
expirationSeconds: 3600
|
|
||||||
path: token
|
|
||||||
---
|
|
||||||
apiVersion: sandbox.opensandbox.io/v1alpha1
|
|
||||||
kind: Pool
|
|
||||||
metadata:
|
|
||||||
name: ci-pod
|
|
||||||
namespace: opensandbox
|
|
||||||
labels:
|
|
||||||
app.kubernetes.io/name: gitea-ci-pod
|
|
||||||
app.kubernetes.io/component: runner-pool
|
|
||||||
spec:
|
|
||||||
capacitySpec:
|
|
||||||
bufferMax: 0
|
|
||||||
bufferMin: 0
|
|
||||||
poolMax: 4
|
|
||||||
poolMin: 0
|
|
||||||
recycleStrategy:
|
|
||||||
type: Delete
|
|
||||||
template:
|
|
||||||
metadata:
|
|
||||||
labels:
|
|
||||||
app.kubernetes.io/name: gitea-ci-pod
|
|
||||||
ci.ddupan.top/backend: pod
|
|
||||||
spec:
|
|
||||||
serviceAccountName: gitea-ci
|
|
||||||
restartPolicy: Never
|
|
||||||
terminationGracePeriodSeconds: 30
|
|
||||||
shareProcessNamespace: true
|
|
||||||
securityContext:
|
|
||||||
fsGroup: 2000
|
|
||||||
fsGroupChangePolicy: OnRootMismatch
|
|
||||||
initContainers:
|
|
||||||
- name: task-executor-installer
|
|
||||||
image: sandbox-registry.cn-zhangjiakou.cr.aliyuncs.com/opensandbox/task-executor:v0.1.0
|
|
||||||
command: [/bin/sh, -c]
|
|
||||||
args:
|
|
||||||
- cp /workspace/server /opt/opensandbox/task-executor && chmod 0755 /opt/opensandbox/task-executor
|
|
||||||
volumeMounts:
|
|
||||||
- name: opensandbox-bin
|
|
||||||
mountPath: /opt/opensandbox
|
|
||||||
- name: execd-installer
|
|
||||||
image: sandbox-registry.cn-zhangjiakou.cr.aliyuncs.com/opensandbox/execd:v1.0.22
|
|
||||||
command: [/bin/sh, -c]
|
|
||||||
args:
|
|
||||||
- cp ./execd /opt/opensandbox/execd && cp ./bootstrap.sh /opt/opensandbox/bootstrap.sh && chmod 0755 /opt/opensandbox/execd /opt/opensandbox/bootstrap.sh
|
|
||||||
volumeMounts:
|
|
||||||
- name: opensandbox-bin
|
|
||||||
mountPath: /opt/opensandbox
|
|
||||||
containers:
|
|
||||||
- name: sandbox
|
|
||||||
image: zot.ad.ddupan.top/panxiao81/gitea-dynamic-runner-runner@sha256:a45875fd2d0e67429b0b7bc3914735581669c705bb4e1a05d712134d2bceb86f
|
|
||||||
imagePullPolicy: IfNotPresent
|
|
||||||
command: [/opt/opensandbox/task-executor]
|
|
||||||
args:
|
|
||||||
- -listen-addr=0.0.0.0:5758
|
|
||||||
- -log-dir=/tmp
|
|
||||||
- -data-dir=/tmp/tasks
|
|
||||||
env:
|
|
||||||
- name: SANDBOX_MAIN_CONTAINER
|
|
||||||
value: sandbox
|
|
||||||
- name: EXECD_ENVS
|
|
||||||
value: /opt/opensandbox/.env
|
|
||||||
- name: EXECD
|
|
||||||
value: /opt/opensandbox/execd
|
|
||||||
- name: GITEA_INSTANCE_URL
|
|
||||||
value: https://git.ddupan.top
|
|
||||||
- name: HOME
|
|
||||||
value: /data
|
|
||||||
- name: DOCKER_HOST
|
|
||||||
value: unix:///run/docker/docker.sock
|
|
||||||
ports:
|
|
||||||
- name: task-executor
|
|
||||||
containerPort: 5758
|
|
||||||
securityContext:
|
|
||||||
allowPrivilegeEscalation: false
|
|
||||||
capabilities:
|
|
||||||
drop: [ALL]
|
|
||||||
runAsNonRoot: true
|
|
||||||
runAsUser: 2000
|
|
||||||
runAsGroup: 2000
|
|
||||||
volumeMounts:
|
|
||||||
- name: opensandbox-bin
|
|
||||||
mountPath: /opt/opensandbox
|
|
||||||
- name: runner-data
|
|
||||||
mountPath: /data
|
|
||||||
- name: docker-socket
|
|
||||||
mountPath: /run/docker
|
|
||||||
- name: spire-socket
|
|
||||||
mountPath: /run/spire/agent-sockets
|
|
||||||
- name: spire-agent
|
|
||||||
image: ghcr.io/spiffe/spire-agent:1.15.3@sha256:41b0dcd8b258a69db9e2768292a060766fb76fd866e4bc925849981ea1b825ff
|
|
||||||
args: [-config, /run/spire/config/agent.conf]
|
|
||||||
securityContext:
|
|
||||||
allowPrivilegeEscalation: false
|
|
||||||
capabilities:
|
|
||||||
drop: [ALL]
|
|
||||||
volumeMounts:
|
|
||||||
- name: spire-config
|
|
||||||
mountPath: /run/spire/config
|
|
||||||
readOnly: true
|
|
||||||
- name: spire-bundle
|
|
||||||
mountPath: /run/spire/bundle
|
|
||||||
readOnly: true
|
|
||||||
- name: spire-token
|
|
||||||
mountPath: /run/spire/tokens
|
|
||||||
readOnly: true
|
|
||||||
- name: spire-data
|
|
||||||
mountPath: /run/spire/data
|
|
||||||
- name: spire-socket
|
|
||||||
mountPath: /run/spire/agent-sockets
|
|
||||||
- name: docker
|
|
||||||
image: docker.io/library/docker:29.1.5-dind
|
|
||||||
command: [/bin/sh, -c]
|
|
||||||
args:
|
|
||||||
- test -e /dev/kmsg || mknod /dev/kmsg c 1 11; exec dockerd --host=unix:///run/docker/docker.sock --group=2000 --storage-driver=overlay2
|
|
||||||
securityContext:
|
|
||||||
privileged: true
|
|
||||||
volumeMounts:
|
|
||||||
- name: docker-socket
|
|
||||||
mountPath: /run/docker
|
|
||||||
- name: docker-data
|
|
||||||
mountPath: /var/lib/docker
|
|
||||||
volumes:
|
|
||||||
- name: opensandbox-bin
|
|
||||||
emptyDir: {}
|
|
||||||
- name: runner-data
|
|
||||||
emptyDir: {}
|
|
||||||
- name: docker-data
|
|
||||||
emptyDir: {}
|
|
||||||
- name: docker-socket
|
|
||||||
emptyDir:
|
|
||||||
medium: Memory
|
|
||||||
- name: spire-data
|
|
||||||
emptyDir:
|
|
||||||
medium: Memory
|
|
||||||
- name: spire-socket
|
|
||||||
emptyDir:
|
|
||||||
medium: Memory
|
|
||||||
- name: spire-config
|
|
||||||
configMap:
|
|
||||||
name: gitea-ci-spire-agent
|
|
||||||
- name: spire-bundle
|
|
||||||
configMap:
|
|
||||||
name: spire-bundle-pem
|
|
||||||
- name: spire-token
|
|
||||||
projected:
|
|
||||||
sources:
|
|
||||||
- serviceAccountToken:
|
|
||||||
audience: spire-server
|
|
||||||
expirationSeconds: 3600
|
|
||||||
path: token
|
|
||||||
@@ -1,87 +0,0 @@
|
|||||||
---
|
|
||||||
apiVersion: v1
|
|
||||||
kind: ServiceAccount
|
|
||||||
metadata:
|
|
||||||
name: gitea-ci
|
|
||||||
namespace: opensandbox
|
|
||||||
---
|
|
||||||
apiVersion: v1
|
|
||||||
kind: ServiceAccount
|
|
||||||
metadata:
|
|
||||||
name: opensandbox-identity
|
|
||||||
namespace: opensandbox
|
|
||||||
---
|
|
||||||
apiVersion: rbac.authorization.k8s.io/v1
|
|
||||||
kind: Role
|
|
||||||
metadata:
|
|
||||||
name: opensandbox-identity
|
|
||||||
namespace: opensandbox
|
|
||||||
rules:
|
|
||||||
- apiGroups: [""]
|
|
||||||
resources: [pods]
|
|
||||||
verbs: [get, list, watch]
|
|
||||||
- apiGroups: [sandbox.opensandbox.io]
|
|
||||||
resources: [batchsandboxes]
|
|
||||||
verbs: [get, list, watch]
|
|
||||||
---
|
|
||||||
apiVersion: rbac.authorization.k8s.io/v1
|
|
||||||
kind: RoleBinding
|
|
||||||
metadata:
|
|
||||||
name: opensandbox-identity
|
|
||||||
namespace: opensandbox
|
|
||||||
roleRef:
|
|
||||||
apiGroup: rbac.authorization.k8s.io
|
|
||||||
kind: Role
|
|
||||||
name: opensandbox-identity
|
|
||||||
subjects:
|
|
||||||
- kind: ServiceAccount
|
|
||||||
name: opensandbox-identity
|
|
||||||
namespace: opensandbox
|
|
||||||
---
|
|
||||||
apiVersion: rbac.authorization.k8s.io/v1
|
|
||||||
kind: ClusterRole
|
|
||||||
metadata:
|
|
||||||
name: opensandbox-identity
|
|
||||||
rules:
|
|
||||||
- apiGroups: [spire.spiffe.io]
|
|
||||||
resources: [clusterstaticentries]
|
|
||||||
verbs: [create, delete, get, list, watch]
|
|
||||||
---
|
|
||||||
apiVersion: rbac.authorization.k8s.io/v1
|
|
||||||
kind: ClusterRoleBinding
|
|
||||||
metadata:
|
|
||||||
name: opensandbox-identity
|
|
||||||
roleRef:
|
|
||||||
apiGroup: rbac.authorization.k8s.io
|
|
||||||
kind: ClusterRole
|
|
||||||
name: opensandbox-identity
|
|
||||||
subjects:
|
|
||||||
- kind: ServiceAccount
|
|
||||||
name: opensandbox-identity
|
|
||||||
namespace: opensandbox
|
|
||||||
---
|
|
||||||
# The central SPIRE external controller uses this existing sandbox credential
|
|
||||||
# to publish a PEM bundle for guest-local Agents. It cannot read Secrets here.
|
|
||||||
apiVersion: rbac.authorization.k8s.io/v1
|
|
||||||
kind: Role
|
|
||||||
metadata:
|
|
||||||
name: spire-runner-bundle-publisher
|
|
||||||
namespace: opensandbox
|
|
||||||
rules:
|
|
||||||
- apiGroups: [""]
|
|
||||||
resources: [configmaps]
|
|
||||||
verbs: [create, delete, get, list, patch, update, watch]
|
|
||||||
---
|
|
||||||
apiVersion: rbac.authorization.k8s.io/v1
|
|
||||||
kind: RoleBinding
|
|
||||||
metadata:
|
|
||||||
name: spire-runner-bundle-publisher
|
|
||||||
namespace: opensandbox
|
|
||||||
roleRef:
|
|
||||||
apiGroup: rbac.authorization.k8s.io
|
|
||||||
kind: Role
|
|
||||||
name: spire-runner-bundle-publisher
|
|
||||||
subjects:
|
|
||||||
- kind: ServiceAccount
|
|
||||||
name: spire-controller-manager
|
|
||||||
namespace: spire-system
|
|
||||||
@@ -1,15 +0,0 @@
|
|||||||
apiVersion: v1
|
|
||||||
kind: Service
|
|
||||||
metadata:
|
|
||||||
name: opensandbox-server-internal
|
|
||||||
namespace: opensandbox-system
|
|
||||||
spec:
|
|
||||||
type: NodePort
|
|
||||||
selector:
|
|
||||||
app.kubernetes.io/instance: opensandbox
|
|
||||||
app.kubernetes.io/name: opensandbox-server
|
|
||||||
ports:
|
|
||||||
- name: http
|
|
||||||
port: 80
|
|
||||||
targetPort: http
|
|
||||||
nodePort: 30080
|
|
||||||
@@ -1,56 +0,0 @@
|
|||||||
# Sandbox External Secrets Operator
|
|
||||||
|
|
||||||
本目录在 sandbox 集群部署独立的 External Secrets Operator `2.8.0`，并通过
|
|
||||||
`ClusterSecretStore/openbao` 读取 OpenBao KV v2 中共享的 OpenSandbox API key。它不复用
|
|
||||||
homelab 集群的 ESO Pod、ServiceAccount 或 Kubernetes auth backend。
|
|
||||||
|
|
||||||
## 当前状态
|
|
||||||
|
|
||||||
- OpenBao `auth/kubernetes-sandbox`、backend config、`external-secrets` role 与
|
|
||||||
`sandbox-external-secrets` policy 已于 2026-09-18 由 Terraform 创建；apply 后 plan
|
|
||||||
为 zero-diff；
|
|
||||||
- `kv/k8s/opensandbox-api` 已由本机 `spiffe://ddupan.top/dev/panxiao81` 身份生成并写入，
|
|
||||||
值未输出或落盘；
|
|
||||||
- sandbox ESO operator、`ClusterSecretStore/openbao` 与 OpenSandbox `ExternalSecret`
|
|
||||||
由 Flux 管理；
|
|
||||||
- 线上 `ClusterSecretStore/openbao` 为 `Valid/Ready`，`ExternalSecret/opensandbox-api-key`
|
|
||||||
为 `SecretSynced/Ready`；
|
|
||||||
- OpenSandbox 已切换到 API key：无 key 请求返回 `401`，正确 key 请求返回 `200`；
|
|
||||||
- homelab runner 对同一 key 的投影不在本目录，留给 runner 项目管理。
|
|
||||||
|
|
||||||
OpenBao 的 `auth/kubernetes-sandbox`、对应 role、policy、sandbox API 地址与公开 CA
|
|
||||||
完全由 `infrastructure/openbao/terraform/` 管理。CA 文件提交到 Git 是刻意设计：CA
|
|
||||||
是公开信任材料，版本化后集群重建造成的 trust root 变化会产生可审计的 Terraform diff。
|
|
||||||
|
|
||||||
ESO 使用 TokenRequest 生成短期 ServiceAccount JWT。OpenBao 未配置长期
|
|
||||||
`token_reviewer_jwt`，而是使用登录 JWT 调用 sandbox TokenReview；因此
|
|
||||||
`external-secrets` ServiceAccount 仅额外绑定内建 `system:auth-delegator`。
|
|
||||||
|
|
||||||
## 重建顺序
|
|
||||||
|
|
||||||
1. 在 OpenBao 写入 OpenSandbox API key：
|
|
||||||
|
|
||||||
```bash
|
|
||||||
openssl rand -hex 32 | bao kv put kv/k8s/opensandbox-api api_key=-
|
|
||||||
```
|
|
||||||
|
|
||||||
2. 在 `infrastructure/openbao/terraform` 执行 `terraform plan` 和 `terraform apply`，
|
|
||||||
创建 `kubernetes-sandbox` auth mount、backend config、role 与只允许读取
|
|
||||||
`kv/k8s/opensandbox-api` 的最小权限 policy。
|
|
||||||
3. 合并 Flux 变更。依次等待 `flux-system/external-secrets-operator`、
|
|
||||||
`flux-system/external-secrets` Ready，再等待 `flux-system/opensandbox` 滚动完成。
|
|
||||||
|
|
||||||
operator 与配置拆成两个 Flux Kustomization，确保全新集群先安装 CRD，再声明
|
|
||||||
`ClusterSecretStore`；不要为了减少目录而把两层重新合并。
|
|
||||||
|
|
||||||
## 验收
|
|
||||||
|
|
||||||
```bash
|
|
||||||
kubectl get clustersecretstore openbao
|
|
||||||
kubectl -n opensandbox-system get externalsecret opensandbox-api-key
|
|
||||||
kubectl -n opensandbox-system get secret opensandbox-api-key
|
|
||||||
```
|
|
||||||
|
|
||||||
只检查 Secret 是否存在及 key 名，不输出 `data`。`ClusterSecretStore` 或
|
|
||||||
`ExternalSecret` 不 Ready 时，先检查 `auth/kubernetes-sandbox`，不要临时创建静态
|
|
||||||
Bao token Secret。
|
|
||||||
@@ -1,13 +0,0 @@
|
|||||||
---
|
|
||||||
apiVersion: rbac.authorization.k8s.io/v1
|
|
||||||
kind: ClusterRoleBinding
|
|
||||||
metadata:
|
|
||||||
name: sandbox-external-secrets-token-review
|
|
||||||
roleRef:
|
|
||||||
apiGroup: rbac.authorization.k8s.io
|
|
||||||
kind: ClusterRole
|
|
||||||
name: system:auth-delegator
|
|
||||||
subjects:
|
|
||||||
- kind: ServiceAccount
|
|
||||||
name: external-secrets
|
|
||||||
namespace: external-secrets
|
|
||||||
@@ -1,18 +0,0 @@
|
|||||||
---
|
|
||||||
apiVersion: external-secrets.io/v1
|
|
||||||
kind: ClusterSecretStore
|
|
||||||
metadata:
|
|
||||||
name: openbao
|
|
||||||
spec:
|
|
||||||
provider:
|
|
||||||
vault:
|
|
||||||
server: https://bao.ad.ddupan.top:8200
|
|
||||||
path: kv
|
|
||||||
version: v2
|
|
||||||
auth:
|
|
||||||
kubernetes:
|
|
||||||
mountPath: kubernetes-sandbox
|
|
||||||
role: external-secrets
|
|
||||||
serviceAccountRef:
|
|
||||||
name: external-secrets
|
|
||||||
namespace: external-secrets
|
|
||||||
@@ -1,35 +0,0 @@
|
|||||||
---
|
|
||||||
apiVersion: helm.toolkit.fluxcd.io/v2
|
|
||||||
kind: HelmRelease
|
|
||||||
metadata:
|
|
||||||
name: external-secrets
|
|
||||||
namespace: external-secrets
|
|
||||||
spec:
|
|
||||||
chart:
|
|
||||||
spec:
|
|
||||||
chart: external-secrets
|
|
||||||
interval: 1h
|
|
||||||
sourceRef:
|
|
||||||
kind: HelmRepository
|
|
||||||
name: external-secrets
|
|
||||||
version: 2.8.0
|
|
||||||
driftDetection:
|
|
||||||
mode: enabled
|
|
||||||
install:
|
|
||||||
crds: CreateReplace
|
|
||||||
strategy:
|
|
||||||
name: RetryOnFailure
|
|
||||||
retryInterval: 5m
|
|
||||||
interval: 30m
|
|
||||||
releaseName: external-secrets
|
|
||||||
targetNamespace: external-secrets
|
|
||||||
timeout: 10m
|
|
||||||
upgrade:
|
|
||||||
crds: CreateReplace
|
|
||||||
strategy:
|
|
||||||
name: RetryOnFailure
|
|
||||||
retryInterval: 5m
|
|
||||||
valuesFrom:
|
|
||||||
- kind: ConfigMap
|
|
||||||
name: external-secrets-values
|
|
||||||
valuesKey: values.yaml
|
|
||||||
@@ -1,8 +0,0 @@
|
|||||||
---
|
|
||||||
apiVersion: kustomize.config.k8s.io/v1beta1
|
|
||||||
kind: Kustomization
|
|
||||||
resources:
|
|
||||||
- namespace.yaml
|
|
||||||
- repository.yaml
|
|
||||||
- values.yaml
|
|
||||||
- helmrelease.yaml
|
|
||||||
@@ -1,5 +0,0 @@
|
|||||||
---
|
|
||||||
apiVersion: v1
|
|
||||||
kind: Namespace
|
|
||||||
metadata:
|
|
||||||
name: external-secrets
|
|
||||||
@@ -1,9 +0,0 @@
|
|||||||
---
|
|
||||||
apiVersion: source.toolkit.fluxcd.io/v1
|
|
||||||
kind: HelmRepository
|
|
||||||
metadata:
|
|
||||||
name: external-secrets
|
|
||||||
namespace: external-secrets
|
|
||||||
spec:
|
|
||||||
interval: 1h
|
|
||||||
url: https://charts.external-secrets.io
|
|
||||||
@@ -1,31 +0,0 @@
|
|||||||
---
|
|
||||||
apiVersion: v1
|
|
||||||
kind: ConfigMap
|
|
||||||
metadata:
|
|
||||||
name: external-secrets-values
|
|
||||||
namespace: external-secrets
|
|
||||||
data:
|
|
||||||
values.yaml: |
|
|
||||||
replicaCount: 1
|
|
||||||
|
|
||||||
webhook:
|
|
||||||
replicaCount: 1
|
|
||||||
resources:
|
|
||||||
requests: {cpu: 10m, memory: 32Mi}
|
|
||||||
limits: {memory: 128Mi}
|
|
||||||
|
|
||||||
certController:
|
|
||||||
replicaCount: 1
|
|
||||||
resources:
|
|
||||||
requests: {cpu: 10m, memory: 32Mi}
|
|
||||||
limits: {memory: 128Mi}
|
|
||||||
|
|
||||||
resources:
|
|
||||||
requests: {cpu: 10m, memory: 64Mi}
|
|
||||||
limits: {memory: 256Mi}
|
|
||||||
|
|
||||||
serviceAccount:
|
|
||||||
create: true
|
|
||||||
name: external-secrets
|
|
||||||
|
|
||||||
installCRDs: true
|
|
||||||
@@ -28,32 +28,6 @@ Pod 删除后的 backing-file/VMM 回收和真实构建基准必须作为上线
|
|||||||
由 `VMPodScrape` 写入中央 VictoriaMetrics。它不拥有 Kubernetes API 凭据或 host 写
|
由 `VMPodScrape` 写入中央 VictoriaMetrics。它不拥有 Kubernetes API 凭据或 host 写
|
||||||
权限。
|
权限。
|
||||||
|
|
||||||
## Guest 内 SPIFFE 身份
|
|
||||||
|
|
||||||
Kata guest 不能直接使用 node SPIRE Agent 的 CSI socket。Unix socket 的路径即使通过
|
|
||||||
virtio-fs 出现在 guest 中，连接也不能跨 VM 边界。CI Pod 应以 native sidecar 在同一
|
|
||||||
guest 内启动临时 SPIRE Agent，并满足以下约束：
|
|
||||||
|
|
||||||
1. 外层 Pod 挂载 `audience=spire-server` 的 Pod-bound projected ServiceAccount token；
|
|
||||||
2. 内层 Agent 使用 `k8s_psat` 向中央 Server attestation，Server cluster profile 必须
|
|
||||||
启用 `use_pod_uid_for_agent_id`，使 Agent ID 包含 Pod UID；
|
|
||||||
3. 调度器为该具体 Agent 创建 registration entry；SPIFFE ID 使用仓库和任务名等稳定的
|
|
||||||
业务语义，Pod UID 只作为 parent binding，不进入业务身份；
|
|
||||||
4. Agent 与 workload 通过 guest 内 `emptyDir` 上的 Unix socket 通信；Pod 必须设置
|
|
||||||
`shareProcessNamespace: true`，否则 `unix` workload attestor 无法从共享 `/proc`
|
|
||||||
解析客户端的 `SO_PEERCRED` PID；
|
|
||||||
5. 调度器必须在 registration entry 已同步到 Agent 后才放行 workload。Pod 删除后同步
|
|
||||||
删除 entry，并 evict 对应的临时 Agent 记录。
|
|
||||||
|
|
||||||
2026-09-17 的 live PoC 已验证：以 Pod UID 作为 parent 的临时 Agent 成功注册，
|
|
||||||
`unix:uid:2000` workload 从 guest-local socket 获得
|
|
||||||
`spiffe://ddupan.top/ci/poc/task/build` X.509-SVID，Pod 正常退出。PoC 资源随后全部清理，
|
|
||||||
中央 SPIRE 恢复声明式配置。
|
|
||||||
|
|
||||||
SPIRE 1.15.3 已支持 `use_pod_uid_for_agent_id`，但当前使用的 hardened chart 0.30.2
|
|
||||||
尚未把它暴露到 values/template。正式部署应先给 chart 补齐该字段并向上游提交，随后
|
|
||||||
采用包含修复的 release；不得把手工修改 Server ConfigMap 作为运行方案。
|
|
||||||
|
|
||||||
## 上线验收
|
## 上线验收
|
||||||
|
|
||||||
Flux reconciliation 完成后至少确认：
|
Flux reconciliation 完成后至少确认：
|
||||||
@@ -61,8 +35,8 @@ Flux reconciliation 完成后至少确认：
|
|||||||
1. 两个节点重新回到 Ready，`RuntimeClass/kata-clh-runtime-rs` 存在；
|
1. 两个节点重新回到 Ready，`RuntimeClass/kata-clh-runtime-rs` 存在；
|
||||||
2. Kata Pod 内核与 LXC host 内核不同，且 `/dev/kvm` 可用；
|
2. Kata Pod 内核与 LXC host 内核不同，且 `/dev/kvm` 可用；
|
||||||
3. Cloud Hypervisor API 的 `vm.info.config.memory.shared` 为 `true`；
|
3. Cloud Hypervisor API 的 `vm.info.config.memory.shared` 为 `true`；
|
||||||
4. Kata Pod 内层 Agent 的 ID 包含该 Pod UID；只有调度器创建的业务 entry 所匹配的
|
4. `spire-smoke` ServiceAccount 的 Kata Pod 可获得
|
||||||
workload UID 能从 guest-local socket 获得预期 SPIFFE ID；
|
`spiffe://ddupan.top/sandbox/smoke`，错误 ServiceAccount 无法获得身份；
|
||||||
5. block-backed `emptyDir` 上 Docker 使用 `overlay2`，BuildKit 与 kind smoke test
|
5. block-backed `emptyDir` 上 Docker 使用 `overlay2`，BuildKit 与 kind smoke test
|
||||||
通过；kind 的 dockerd bootstrap 需要先在 guest 内执行
|
通过；kind 的 dockerd bootstrap 需要先在 guest 内执行
|
||||||
`mknod /dev/kmsg c 1 11`；
|
`mknod /dev/kmsg c 1 11`；
|
||||||
|
|||||||
+1
-3
@@ -1,6 +1,4 @@
|
|||||||
---
|
|
||||||
apiVersion: kustomize.config.k8s.io/v1beta1
|
apiVersion: kustomize.config.k8s.io/v1beta1
|
||||||
kind: Kustomization
|
kind: Kustomization
|
||||||
resources:
|
resources:
|
||||||
- auth-delegator.yaml
|
- pools.yaml
|
||||||
- clustersecretstore.yaml
|
|
||||||
@@ -0,0 +1,123 @@
|
|||||||
|
---
|
||||||
|
apiVersion: sandbox.opensandbox.io/v1alpha1
|
||||||
|
kind: Pool
|
||||||
|
metadata:
|
||||||
|
name: ci-pod
|
||||||
|
namespace: opensandbox
|
||||||
|
spec:
|
||||||
|
template:
|
||||||
|
metadata:
|
||||||
|
labels:
|
||||||
|
ci.ddupan.top/backend: pod
|
||||||
|
spec:
|
||||||
|
containers:
|
||||||
|
- name: sandbox
|
||||||
|
image: sandbox-registry.cn-zhangjiakou.cr.aliyuncs.com/opensandbox/code-interpreter:v1.1.0
|
||||||
|
command: [/opt/opensandbox/task-executor]
|
||||||
|
args: [-listen-addr=0.0.0.0:5758, -log-dir=/tmp]
|
||||||
|
env:
|
||||||
|
- name: SANDBOX_MAIN_CONTAINER
|
||||||
|
value: sandbox
|
||||||
|
- name: EXECD_ENVS
|
||||||
|
value: /opt/opensandbox/.env
|
||||||
|
- name: EXECD
|
||||||
|
value: /opt/opensandbox/execd
|
||||||
|
resources:
|
||||||
|
requests:
|
||||||
|
cpu: 100m
|
||||||
|
memory: 256Mi
|
||||||
|
limits:
|
||||||
|
cpu: "2"
|
||||||
|
memory: 4Gi
|
||||||
|
volumeMounts:
|
||||||
|
- name: opensandbox-bin
|
||||||
|
mountPath: /opt/opensandbox
|
||||||
|
- name: sandbox-storage
|
||||||
|
mountPath: /var/lib/sandbox
|
||||||
|
initContainers:
|
||||||
|
- name: task-executor-installer
|
||||||
|
image: sandbox-registry.cn-zhangjiakou.cr.aliyuncs.com/opensandbox/task-executor:v0.1.0
|
||||||
|
command: [/bin/sh, -c]
|
||||||
|
args: [cp /workspace/server /opt/opensandbox/task-executor && chmod 0755 /opt/opensandbox/task-executor]
|
||||||
|
volumeMounts:
|
||||||
|
- name: opensandbox-bin
|
||||||
|
mountPath: /opt/opensandbox
|
||||||
|
- name: execd-installer
|
||||||
|
image: sandbox-registry.cn-zhangjiakou.cr.aliyuncs.com/opensandbox/execd:v1.0.22
|
||||||
|
command: [/bin/sh, -c]
|
||||||
|
args: [cp ./execd /opt/opensandbox/execd && cp ./bootstrap.sh /opt/opensandbox/bootstrap.sh && chmod 0755 /opt/opensandbox/execd /opt/opensandbox/bootstrap.sh]
|
||||||
|
volumeMounts:
|
||||||
|
- name: opensandbox-bin
|
||||||
|
mountPath: /opt/opensandbox
|
||||||
|
volumes:
|
||||||
|
- name: opensandbox-bin
|
||||||
|
emptyDir: {}
|
||||||
|
- name: sandbox-storage
|
||||||
|
emptyDir: {}
|
||||||
|
capacitySpec:
|
||||||
|
bufferMax: 1
|
||||||
|
bufferMin: 0
|
||||||
|
poolMax: 4
|
||||||
|
poolMin: 0
|
||||||
|
---
|
||||||
|
apiVersion: sandbox.opensandbox.io/v1alpha1
|
||||||
|
kind: Pool
|
||||||
|
metadata:
|
||||||
|
name: ci-vm
|
||||||
|
namespace: opensandbox
|
||||||
|
spec:
|
||||||
|
template:
|
||||||
|
metadata:
|
||||||
|
labels:
|
||||||
|
ci.ddupan.top/backend: vm
|
||||||
|
spec:
|
||||||
|
runtimeClassName: kata-clh-runtime-rs
|
||||||
|
containers:
|
||||||
|
- name: sandbox
|
||||||
|
image: sandbox-registry.cn-zhangjiakou.cr.aliyuncs.com/opensandbox/code-interpreter:v1.1.0
|
||||||
|
command: [/opt/opensandbox/task-executor]
|
||||||
|
args: [-listen-addr=0.0.0.0:5758, -log-dir=/tmp]
|
||||||
|
env:
|
||||||
|
- name: SANDBOX_MAIN_CONTAINER
|
||||||
|
value: sandbox
|
||||||
|
- name: EXECD_ENVS
|
||||||
|
value: /opt/opensandbox/.env
|
||||||
|
- name: EXECD
|
||||||
|
value: /opt/opensandbox/execd
|
||||||
|
resources:
|
||||||
|
requests:
|
||||||
|
cpu: 250m
|
||||||
|
memory: 512Mi
|
||||||
|
limits:
|
||||||
|
cpu: "4"
|
||||||
|
memory: 8Gi
|
||||||
|
volumeMounts:
|
||||||
|
- name: opensandbox-bin
|
||||||
|
mountPath: /opt/opensandbox
|
||||||
|
- name: sandbox-storage
|
||||||
|
mountPath: /var/lib/sandbox
|
||||||
|
initContainers:
|
||||||
|
- name: task-executor-installer
|
||||||
|
image: sandbox-registry.cn-zhangjiakou.cr.aliyuncs.com/opensandbox/task-executor:v0.1.0
|
||||||
|
command: [/bin/sh, -c]
|
||||||
|
args: [cp /workspace/server /opt/opensandbox/task-executor && chmod 0755 /opt/opensandbox/task-executor]
|
||||||
|
volumeMounts:
|
||||||
|
- name: opensandbox-bin
|
||||||
|
mountPath: /opt/opensandbox
|
||||||
|
- name: execd-installer
|
||||||
|
image: sandbox-registry.cn-zhangjiakou.cr.aliyuncs.com/opensandbox/execd:v1.0.22
|
||||||
|
command: [/bin/sh, -c]
|
||||||
|
args: [cp ./execd /opt/opensandbox/execd && cp ./bootstrap.sh /opt/opensandbox/bootstrap.sh && chmod 0755 /opt/opensandbox/execd /opt/opensandbox/bootstrap.sh]
|
||||||
|
volumeMounts:
|
||||||
|
- name: opensandbox-bin
|
||||||
|
mountPath: /opt/opensandbox
|
||||||
|
volumes:
|
||||||
|
- name: opensandbox-bin
|
||||||
|
emptyDir: {}
|
||||||
|
- name: sandbox-storage
|
||||||
|
emptyDir: {}
|
||||||
|
capacitySpec:
|
||||||
|
bufferMax: 1
|
||||||
|
bufferMin: 0
|
||||||
|
poolMax: 2
|
||||||
|
poolMin: 0
|
||||||
@@ -1,44 +1,19 @@
|
|||||||
# Sandbox OpenSandbox
|
# OpenSandbox
|
||||||
|
|
||||||
本目录在独立 sandbox k3s 集群部署 OpenSandbox controller、server 与 CRD。Flux 从
|
Flux installs the upstream all-in-one OpenSandbox chart pinned to
|
||||||
上游 commit `8f01e935c2cabba778cf37a152033fae062fa0f4` 构建官方 umbrella chart
|
`helm/opensandbox/0.2.2` (`8f01e935`). The API is cluster-internal and intentionally runs a
|
||||||
`0.2.2`；该源码渲染结果已与 release `opensandbox-0.2.2.tgz` 对比一致。不要改为跟随
|
single replica until shared server state and HA behaviour have been validated.
|
||||||
浮动 branch 或 tag。
|
|
||||||
|
|
||||||
server 只提供集群内 `opensandbox-server.opensandbox-system.svc:80` ClusterIP，不部署
|
`ci-pod` uses `runc`; `ci-vm` uses the separately managed
|
||||||
Gateway、Ingress 或 LoadBalancer。sandbox workload 位于 `opensandbox` namespace，
|
`kata-clh-runtime-rs` RuntimeClass. Both Pools start at zero and create capacity
|
||||||
默认使用 `kata-clh-runtime-rs`；CI Pool、runner 镜像、动态 SPIFFE registration 均由
|
on demand. They currently use the upstream interpreter image to validate the
|
||||||
runner 项目后续声明，本目录不预制。
|
Lifecycle API and Pool allocation independently of the CI scheduler cutover.
|
||||||
|
|
||||||
## API 认证
|
The dynamic runner worker, runner image, guest-local SPIRE Agent and Docker
|
||||||
|
sidecar are introduced only after this layer is Ready. In particular, do not
|
||||||
|
mount the host SPIFFE CSI socket into `ci-vm`: Unix sockets do not cross the
|
||||||
|
Kata VM boundary.
|
||||||
|
|
||||||
`ExternalSecret/opensandbox-api-key` 从 OpenBao
|
Smoke test both backends through the same API by creating sandboxes with
|
||||||
`kv/k8s/opensandbox-api:api_key` 投影同名 Secret。这个路径不归属于某个 Kubernetes
|
`extensions.poolRef` set to `ci-pod` and `ci-vm`, then confirm their
|
||||||
集群：sandbox server 与 homelab runner 调度器分别通过自己的 Bao 身份读取。server 只通过
|
BatchSandboxes, Pods and VMMs disappear after deletion.
|
||||||
`secretKeyRef` 读取：
|
|
||||||
|
|
||||||
```yaml
|
|
||||||
- name: OPENSANDBOX_SERVER_API_KEY
|
|
||||||
valueFrom:
|
|
||||||
secretKeyRef:
|
|
||||||
name: opensandbox-api-key
|
|
||||||
key: api-key
|
|
||||||
```
|
|
||||||
|
|
||||||
仓库与 Helm values 均不保存 API key。OpenSandbox 不支持更丰富的原生 workload
|
|
||||||
authentication；runner 后续读取同一 Bao 路径并在请求头中使用 API key。
|
|
||||||
`opensandbox-values` 带 Flux watch label，values 变化会立即触发 Helm reconcile，不依赖
|
|
||||||
30 分钟的 HelmRelease interval。
|
|
||||||
|
|
||||||
## 验收
|
|
||||||
|
|
||||||
合并后等待 `flux-system/opensandbox` 与 `opensandbox-system/opensandbox` Ready，并确认：
|
|
||||||
|
|
||||||
```bash
|
|
||||||
kubectl get crd batchsandboxes.sandbox.opensandbox.io pools.sandbox.opensandbox.io
|
|
||||||
kubectl -n opensandbox-system get deploy,pod,svc
|
|
||||||
kubectl get runtimeclass kata-clh-runtime-rs
|
|
||||||
```
|
|
||||||
|
|
||||||
控制面上线不创建 CI Pool，也不产生 sandbox workload。首个 runner 集成应另行提交 Pool
|
|
||||||
与完整的 Lifecycle API smoke test。
|
|
||||||
|
|||||||
@@ -1,19 +0,0 @@
|
|||||||
---
|
|
||||||
apiVersion: external-secrets.io/v1
|
|
||||||
kind: ExternalSecret
|
|
||||||
metadata:
|
|
||||||
name: opensandbox-api-key
|
|
||||||
namespace: opensandbox-system
|
|
||||||
spec:
|
|
||||||
refreshInterval: 1h
|
|
||||||
secretStoreRef:
|
|
||||||
kind: ClusterSecretStore
|
|
||||||
name: openbao
|
|
||||||
target:
|
|
||||||
name: opensandbox-api-key
|
|
||||||
creationPolicy: Owner
|
|
||||||
data:
|
|
||||||
- secretKey: api-key
|
|
||||||
remoteRef:
|
|
||||||
key: k8s/opensandbox-api
|
|
||||||
property: api_key
|
|
||||||
@@ -1,4 +1,3 @@
|
|||||||
---
|
|
||||||
apiVersion: helm.toolkit.fluxcd.io/v2
|
apiVersion: helm.toolkit.fluxcd.io/v2
|
||||||
kind: HelmRelease
|
kind: HelmRelease
|
||||||
metadata:
|
metadata:
|
||||||
@@ -16,7 +15,6 @@ spec:
|
|||||||
driftDetection:
|
driftDetection:
|
||||||
mode: enabled
|
mode: enabled
|
||||||
install:
|
install:
|
||||||
crds: CreateReplace
|
|
||||||
strategy:
|
strategy:
|
||||||
name: RetryOnFailure
|
name: RetryOnFailure
|
||||||
retryInterval: 5m
|
retryInterval: 5m
|
||||||
@@ -25,11 +23,9 @@ spec:
|
|||||||
targetNamespace: opensandbox-system
|
targetNamespace: opensandbox-system
|
||||||
timeout: 15m
|
timeout: 15m
|
||||||
upgrade:
|
upgrade:
|
||||||
crds: CreateReplace
|
|
||||||
strategy:
|
strategy:
|
||||||
name: RetryOnFailure
|
name: RetryOnFailure
|
||||||
retryInterval: 5m
|
retryInterval: 5m
|
||||||
valuesFrom:
|
valuesFrom:
|
||||||
- kind: ConfigMap
|
- kind: ConfigMap
|
||||||
name: opensandbox-values
|
name: opensandbox-values
|
||||||
valuesKey: values.yaml
|
|
||||||
|
|||||||
@@ -1,11 +1,7 @@
|
|||||||
---
|
|
||||||
apiVersion: kustomize.config.k8s.io/v1beta1
|
apiVersion: kustomize.config.k8s.io/v1beta1
|
||||||
kind: Kustomization
|
kind: Kustomization
|
||||||
resources:
|
resources:
|
||||||
- namespace.yaml
|
- namespaces.yaml
|
||||||
- external-secret.yaml
|
|
||||||
- repository.yaml
|
- repository.yaml
|
||||||
- template.yaml
|
|
||||||
- values.yaml
|
- values.yaml
|
||||||
- helmrelease.yaml
|
- helmrelease.yaml
|
||||||
- monitor-scrape.yaml
|
|
||||||
|
|||||||
@@ -1,16 +0,0 @@
|
|||||||
---
|
|
||||||
apiVersion: operator.victoriametrics.com/v1beta1
|
|
||||||
kind: VMPodScrape
|
|
||||||
metadata:
|
|
||||||
name: opensandbox-controller
|
|
||||||
namespace: monitoring
|
|
||||||
spec:
|
|
||||||
namespaceSelector:
|
|
||||||
matchNames:
|
|
||||||
- opensandbox-system
|
|
||||||
podMetricsEndpoints:
|
|
||||||
- interval: 30s
|
|
||||||
port: metrics
|
|
||||||
selector:
|
|
||||||
matchLabels:
|
|
||||||
app.kubernetes.io/name: opensandbox
|
|
||||||
@@ -1,14 +0,0 @@
|
|||||||
---
|
|
||||||
apiVersion: v1
|
|
||||||
kind: Namespace
|
|
||||||
metadata:
|
|
||||||
name: opensandbox-system
|
|
||||||
---
|
|
||||||
apiVersion: v1
|
|
||||||
kind: Namespace
|
|
||||||
metadata:
|
|
||||||
name: opensandbox
|
|
||||||
labels:
|
|
||||||
pod-security.kubernetes.io/enforce: privileged
|
|
||||||
pod-security.kubernetes.io/audit: restricted
|
|
||||||
pod-security.kubernetes.io/warn: restricted
|
|
||||||
@@ -0,0 +1,10 @@
|
|||||||
|
---
|
||||||
|
apiVersion: v1
|
||||||
|
kind: Namespace
|
||||||
|
metadata:
|
||||||
|
name: opensandbox-system
|
||||||
|
---
|
||||||
|
apiVersion: v1
|
||||||
|
kind: Namespace
|
||||||
|
metadata:
|
||||||
|
name: opensandbox
|
||||||
@@ -1,4 +1,3 @@
|
|||||||
---
|
|
||||||
apiVersion: source.toolkit.fluxcd.io/v1
|
apiVersion: source.toolkit.fluxcd.io/v1
|
||||||
kind: GitRepository
|
kind: GitRepository
|
||||||
metadata:
|
metadata:
|
||||||
@@ -7,5 +6,6 @@ metadata:
|
|||||||
spec:
|
spec:
|
||||||
interval: 1h
|
interval: 1h
|
||||||
ref:
|
ref:
|
||||||
commit: 8f01e935c2cabba778cf37a152033fae062fa0f4
|
tag: helm/opensandbox/0.2.2
|
||||||
url: https://github.com/opensandbox-group/OpenSandbox.git
|
timeout: 60s
|
||||||
|
url: https://github.com/alibaba/OpenSandbox.git
|
||||||
|
|||||||
@@ -1,17 +0,0 @@
|
|||||||
---
|
|
||||||
apiVersion: v1
|
|
||||||
kind: ConfigMap
|
|
||||||
metadata:
|
|
||||||
name: opensandbox-batchsandbox-template
|
|
||||||
namespace: opensandbox-system
|
|
||||||
data:
|
|
||||||
batchsandbox-template.yaml: |
|
|
||||||
metadata:
|
|
||||||
labels:
|
|
||||||
ddupan.top/workload-class: sandbox
|
|
||||||
spec:
|
|
||||||
replicas: 1
|
|
||||||
template:
|
|
||||||
spec:
|
|
||||||
restartPolicy: Never
|
|
||||||
terminationGracePeriodSeconds: 30
|
|
||||||
@@ -1,52 +1,41 @@
|
|||||||
---
|
|
||||||
apiVersion: v1
|
apiVersion: v1
|
||||||
kind: ConfigMap
|
kind: ConfigMap
|
||||||
metadata:
|
metadata:
|
||||||
name: opensandbox-values
|
name: opensandbox-values
|
||||||
namespace: opensandbox-system
|
namespace: opensandbox-system
|
||||||
labels:
|
|
||||||
reconcile.fluxcd.io/watch: Enabled
|
|
||||||
data:
|
data:
|
||||||
values.yaml: |
|
values.yaml: |
|
||||||
opensandbox-controller:
|
opensandbox-controller:
|
||||||
controller:
|
controller:
|
||||||
|
logLevel: info
|
||||||
|
replicaCount: 1
|
||||||
metrics:
|
metrics:
|
||||||
enabled: true
|
enabled: true
|
||||||
port: 8080
|
|
||||||
secure: false
|
secure: false
|
||||||
|
port: 8080
|
||||||
|
resources:
|
||||||
|
requests:
|
||||||
|
cpu: 25m
|
||||||
|
memory: 64Mi
|
||||||
|
limits:
|
||||||
|
cpu: 500m
|
||||||
|
memory: 256Mi
|
||||||
|
|
||||||
opensandbox-server:
|
opensandbox-server:
|
||||||
server:
|
server:
|
||||||
replicaCount: 1
|
replicaCount: 1
|
||||||
env:
|
|
||||||
- name: OPENSANDBOX_SERVER_API_KEY
|
|
||||||
valueFrom:
|
|
||||||
secretKeyRef:
|
|
||||||
name: opensandbox-api-key
|
|
||||||
key: api-key
|
|
||||||
resources:
|
resources:
|
||||||
limits:
|
|
||||||
cpu: "1"
|
|
||||||
memory: 1Gi
|
|
||||||
requests:
|
requests:
|
||||||
cpu: 100m
|
cpu: 100m
|
||||||
memory: 256Mi
|
memory: 256Mi
|
||||||
volumeMounts:
|
limits:
|
||||||
- name: batchsandbox-template
|
cpu: "1"
|
||||||
mountPath: /etc/opensandbox/batchsandbox-template.yaml
|
memory: 1Gi
|
||||||
subPath: batchsandbox-template.yaml
|
|
||||||
readOnly: true
|
|
||||||
volumes:
|
|
||||||
- name: batchsandbox-template
|
|
||||||
configMap:
|
|
||||||
name: opensandbox-batchsandbox-template
|
|
||||||
|
|
||||||
configToml: |
|
configToml: |
|
||||||
[server]
|
[server]
|
||||||
host = "0.0.0.0"
|
host = "0.0.0.0"
|
||||||
port = 80
|
port = 80
|
||||||
api_key = ""
|
api_key = ""
|
||||||
max_sandbox_timeout_seconds = 86400
|
|
||||||
|
|
||||||
[log]
|
[log]
|
||||||
level = "INFO"
|
level = "INFO"
|
||||||
@@ -55,10 +44,6 @@ data:
|
|||||||
type = "kubernetes"
|
type = "kubernetes"
|
||||||
execd_image = "sandbox-registry.cn-zhangjiakou.cr.aliyuncs.com/opensandbox/execd:v1.0.22"
|
execd_image = "sandbox-registry.cn-zhangjiakou.cr.aliyuncs.com/opensandbox/execd:v1.0.22"
|
||||||
|
|
||||||
[storage]
|
|
||||||
allowed_host_paths = []
|
|
||||||
volume_default_size = "1Gi"
|
|
||||||
|
|
||||||
[kubernetes]
|
[kubernetes]
|
||||||
kubeconfig_path = ""
|
kubeconfig_path = ""
|
||||||
namespace = "opensandbox"
|
namespace = "opensandbox"
|
||||||
@@ -67,17 +52,11 @@ data:
|
|||||||
informer_watch_timeout_seconds = 60
|
informer_watch_timeout_seconds = 60
|
||||||
snapshot_create_timeout_seconds = 900
|
snapshot_create_timeout_seconds = 900
|
||||||
workload_provider = "batchsandbox"
|
workload_provider = "batchsandbox"
|
||||||
image_pull_policy = "IfNotPresent"
|
batchsandbox_template_file = "/etc/opensandbox/example.batchsandbox-template.yaml"
|
||||||
batchsandbox_template_file = "/etc/opensandbox/batchsandbox-template.yaml"
|
|
||||||
|
|
||||||
[egress]
|
[egress]
|
||||||
image = "sandbox-registry.cn-zhangjiakou.cr.aliyuncs.com/opensandbox/egress:v1.1.6"
|
image = "sandbox-registry.cn-zhangjiakou.cr.aliyuncs.com/opensandbox/egress:v1.1.6"
|
||||||
mode = "dns+nft"
|
mode = "dns+nft"
|
||||||
disable_ipv6 = true
|
|
||||||
|
|
||||||
[secure_runtime]
|
|
||||||
type = "kata"
|
|
||||||
k8s_runtime_class = "kata-clh-runtime-rs"
|
|
||||||
|
|
||||||
opensandbox-node-agent:
|
opensandbox-node-agent:
|
||||||
enabled: false
|
enabled: false
|
||||||
|
|||||||
@@ -74,12 +74,6 @@ Agent 已启用 Unix workload attestor，并为本机用户 `panxiao81`（UID `1
|
|||||||
export SPIFFE_ENDPOINT_SOCKET=unix:///run/spire/agent-sockets/spire-agent.sock
|
export SPIFFE_ENDPOINT_SOCKET=unix:///run/spire/agent-sockets/spire-agent.sock
|
||||||
```
|
```
|
||||||
|
|
||||||
宿主机已安装与 Agent Pod 同版本的 `/usr/local/bin/spire-agent` 1.15.3，供本地进程从
|
|
||||||
Workload API 获取 JWT-SVID。二进制来自 SPIRE 官方 `linux-amd64-musl` release，安装时
|
|
||||||
核对 tarball SHA-256
|
|
||||||
`ca1a4d1155317bdd2afc7f36663828a10410c7c840e54725b90b4064b0a301c7`。升级 chart 时应
|
|
||||||
同步升级这个 CLI 并重新核对官方 checksum，不能长期混用版本。
|
|
||||||
|
|
||||||
该身份仅按 Unix UID 匹配，不是 SPIRE admin，也不会匹配 `sudo` 后以 root 运行的
|
该身份仅按 Unix UID 匹配，不是 SPIRE admin，也不会匹配 `sudo` 后以 root 运行的
|
||||||
进程。`ClusterStaticEntry.spec.parentID` 绑定当前 `laptop` Kubernetes node UID；若
|
进程。`ClusterStaticEntry.spec.parentID` 绑定当前 `laptop` Kubernetes node UID；若
|
||||||
节点被删除后重建，需从 `spire-server agent list` 取得新 Agent ID 并同步更新该字段。
|
节点被删除后重建，需从 `spire-server agent list` 取得新 Agent ID 并同步更新该字段。
|
||||||
|
|||||||
@@ -274,24 +274,14 @@ CI job/Agent 不应接收长期 `BAO_TOKEN`。标准启动顺序是：
|
|||||||
5. 在同一进程树中以环境变量调用 `tofu`、Ansible 或其他工具；
|
5. 在同一进程树中以环境变量调用 `tofu`、Ansible 或其他工具；
|
||||||
6. cleanup 尝试 `revoke-self`，随后销毁 job/VM/容器。
|
6. cleanup 尝试 `revoke-self`，随后销毁 job/VM/容器。
|
||||||
|
|
||||||
Gitea Actions 可使用
|
通用 credential-exec 包装器未来应负责步骤 3–6。它必须满足：
|
||||||
[`panxiao81/ci-actions/spiffe-openbao-login@v1`](https://git.ddupan.top/panxiao81/ci-actions/src/tag/v1/spiffe-openbao-login)
|
|
||||||
完成步骤 3、4 和 cleanup 时的 `revoke-self`。runner 只负责提供 Node.js 20、
|
|
||||||
`spire-agent` 与 Workload API socket；role、audience 及是否登录由 workflow 明确声明，
|
|
||||||
目标侧 Bao policy 仍负责最终授权。该 Action 必须满足：
|
|
||||||
|
|
||||||
- 不把 JWT-SVID 或 Bao token 写到 stdout/stderr；
|
- 不把 JWT-SVID 或 Bao token 写到 stdout/stderr；
|
||||||
- 不把凭据传入命令行参数，避免出现在进程列表；
|
- 不把凭据传入命令行参数，避免出现在进程列表；
|
||||||
- job 结束后销毁 runner 及其 Actions 临时文件；
|
- 子进程退出后清除环境和临时文件；
|
||||||
- 不尝试把短期 token 上传到 Actions Secret 或 artifact；
|
- 不尝试把短期 token 上传到 Actions Secret 或 artifact；
|
||||||
- role、audience 和目标命令由受审查的 pipeline 配置决定。
|
- role、audience 和目标命令由受审查的 pipeline 配置决定。
|
||||||
|
|
||||||
Action 会按 GitHub Actions 协议把短期 token 写入 `GITHUB_ENV` 和 `GITHUB_STATE`，因此
|
|
||||||
只允许用于一次性 Pod/VM runner，不能用于共享或持久 runner。只读取 Nexus public
|
|
||||||
repository 时不需要 Bao 登录，可直接使用
|
|
||||||
[`panxiao81/ci-actions/setup-nexus@v1`](https://git.ddupan.top/panxiao81/ci-actions/src/tag/v1/setup-nexus)
|
|
||||||
配置 Ansible Galaxy、Go module proxy 与 OCI endpoint。
|
|
||||||
|
|
||||||
Kubernetes 以外的执行环境不能伪造 ServiceAccount。未来应分别使用 host SPIRE
|
Kubernetes 以外的执行环境不能伪造 ServiceAccount。未来应分别使用 host SPIRE
|
||||||
Agent、TPM/DevID、cloud instance identity、GitHub OIDC 等初始证明接入同一信任模型。
|
Agent、TPM/DevID、cloud instance identity、GitHub OIDC 等初始证明接入同一信任模型。
|
||||||
|
|
||||||
|
|||||||
@@ -43,10 +43,6 @@ spire-server:
|
|||||||
externalSecret:
|
externalSecret:
|
||||||
name: spire-external-kubeconfigs
|
name: spire-external-kubeconfigs
|
||||||
key: sandbox-controller
|
key: sandbox-controller
|
||||||
sandbox-runner-bundle:
|
|
||||||
externalSecret:
|
|
||||||
name: spire-external-kubeconfigs
|
|
||||||
key: sandbox-controller
|
|
||||||
nodeAttestor:
|
nodeAttestor:
|
||||||
externalK8sPSAT:
|
externalK8sPSAT:
|
||||||
enabled: true
|
enabled: true
|
||||||
@@ -59,19 +55,12 @@ spire-server:
|
|||||||
kubeConfigName: sandbox
|
kubeConfigName: sandbox
|
||||||
serviceAccountAllowList:
|
serviceAccountAllowList:
|
||||||
- spire-smoke:spire-smoke
|
- spire-smoke:spire-smoke
|
||||||
- opensandbox:gitea-ci
|
|
||||||
usePodUIDForAgentID: true
|
usePodUIDForAgentID: true
|
||||||
externalControllerManagers:
|
externalControllerManagers:
|
||||||
enabled: true
|
enabled: true
|
||||||
clusters:
|
clusters:
|
||||||
sandbox:
|
sandbox:
|
||||||
kubeConfigName: sandbox-controller
|
kubeConfigName: sandbox-controller
|
||||||
# Dynamic OpenSandbox runner identities are exact Pod-UID-bound
|
|
||||||
# ClusterStaticEntries created in the sandbox cluster. The chart
|
|
||||||
# defaults this reconciler to false, so enable the central registration
|
|
||||||
# path explicitly.
|
|
||||||
reconcile:
|
|
||||||
clusterStaticEntries: true
|
|
||||||
bundlePublisher:
|
bundlePublisher:
|
||||||
externalK8sConfigMap:
|
externalK8sConfigMap:
|
||||||
enabled: true
|
enabled: true
|
||||||
@@ -82,12 +71,6 @@ spire-server:
|
|||||||
configMapName: spire-bundle
|
configMapName: spire-bundle
|
||||||
configMapKey: bundle.spiffe
|
configMapKey: bundle.spiffe
|
||||||
format: spiffe
|
format: spiffe
|
||||||
sandbox-runner-bundle:
|
|
||||||
kubeConfigName: sandbox-runner-bundle
|
|
||||||
namespace: opensandbox
|
|
||||||
configMapName: spire-bundle-pem
|
|
||||||
configMapKey: bundle.pem
|
|
||||||
format: pem
|
|
||||||
persistence:
|
persistence:
|
||||||
# PostgreSQL stores registrations, but the disk KeyManager still needs durable
|
# PostgreSQL stores registrations, but the disk KeyManager still needs durable
|
||||||
# storage for the trust-domain signing keys.
|
# storage for the trust-domain signing keys.
|
||||||
|
|||||||
Reference in New Issue
Block a user