Author SHA1 Message Date
panxiao81 e325a703dd Merge 部署内置 SPIRE CLI 的 runner 镜像
yaml / yaml (push) Successful in 16s
2026-09-24 07:29:53 +00:00
panxiao81 130849040c chore(runner): 部署内置 SPIRE CLI 镜像
yaml / yaml (pull_request) Successful in 22s
2026-09-24 07:29:36 +00:00
panxiao81 bcd034355e Merge 部署 runner 工作目录修复镜像
yaml / yaml (push) Failing after 8s
2026-09-24 06:57:48 +00:00
panxiao81 506e0cb983 chore(runner): 部署工作目录修复镜像
yaml / yaml (pull_request) Failing after 8s
2026-09-24 06:57:30 +00:00
panxiao81 c07709e078 Merge 为 OpenSandbox runner 挂载独立工作目录
yaml / yaml (push) Failing after 8s
2026-09-24 06:29:37 +00:00
panxiao81 e1979ebbb8 fix(runner): 挂载独立工作目录
yaml / yaml (pull_request) Failing after 7s
2026-09-24 06:29:02 +00:00
panxiao81 b02b2aec96 Merge 滚动部署最终 Docker runner 镜像
yaml / yaml (push) Successful in 18s
2026-09-24 06:12:24 +00:00
panxiao81 ff92037a28 chore(runner): 滚动部署最终 Docker 运行环境镜像
yaml / yaml (pull_request) Successful in 19s
2026-09-24 06:11:49 +00:00
panxiao81 9360eb2ea0 Merge 自动 Docker runner 镜像 rollout
yaml / yaml (push) Successful in 19s
2026-09-21 17:43:19 +00:00
panxiao81 0c7eb371e5 chore: rollout job-local Docker runner image
yaml / yaml (pull_request) Successful in 28s
2026-09-21 17:40:12 +00:00
panxiao81 757c21fc0b Merge pull request 'feat: 启用生产 VM runner 标签' (#136) from rollout/vm-runner-production into main
yaml / yaml (push) Successful in 12s
Reviewed-on: #136
2026-09-21 15:23:31 +00:00
panxiao81 c8e163f01e feat: 启用生产 VM runner 标签
yaml / yaml (pull_request) Successful in 21s
2026-09-21 15:22:36 +00:00
panxiao81 59fe28faa5 Merge Kata kind executor 最低资源配置
yaml / yaml (push) Successful in 20s
2026-09-21 12:36:20 +00:00
panxiao81 69c90b9fdf fix: 为 Kata kind executor 配置最低资源
yaml / yaml (pull_request) Successful in 19s
2026-09-21 12:33:35 +00:00
panxiao81 a6a861cc7e Merge 统一 Pod 与 VM executor 运行模型
yaml / yaml (push) Successful in 19s
2026-09-21 12:06:02 +00:00
panxiao81 5c9a09edde refactor: 统一 Pod 与 VM executor 运行模型
yaml / yaml (pull_request) Successful in 31s
2026-09-21 12:01:55 +00:00
panxiao81 cfb9c2d150 Merge Kata DinD guest ext4 存储修复
yaml / yaml (push) Successful in 16s
2026-09-21 11:50:26 +00:00
panxiao81 1ddec34149 fix: 为 Kata DinD 使用 guest ext4 数据盘
yaml / yaml (pull_request) Successful in 21s
2026-09-21 11:48:57 +00:00
panxiao81 db63503ec9 Merge Kata kind 资源规格
yaml / yaml (push) Successful in 16s
2026-09-21 11:34:00 +00:00
panxiao81 9953fae93d fix: 为 Kata kind 任务分配明确资源
yaml / yaml (pull_request) Successful in 25s
2026-09-21 11:31:57 +00:00
panxiao81 56598d6c84 Merge Kata DinD cgroup nesting 修复
yaml / yaml (push) Successful in 16s
2026-09-21 11:16:48 +00:00
panxiao81 834fdcb771 fix: 恢复 Kata DinD cgroup 初始化
yaml / yaml (pull_request) Successful in 17s
2026-09-21 11:14:52 +00:00
panxiao81 47ff73cefb Merge pull request '彻底删除旧静态 Gitea Runner' (#130) from retire/static-gitea-runner-phase2 into main
yaml / yaml (push) Successful in 19s
2026-09-21 10:23:05 +00:00
panxiao81 bb24c76704 chore: 删除旧静态 Gitea Runner
yaml / yaml (pull_request) Successful in 18s
2026-09-21 10:21:33 +00:00
panxiao81 b0791e4095 Merge pull request '退役旧静态 Gitea Runner(第一阶段)' (#129) from retire/static-gitea-runner-phase1 into main
yaml / yaml (push) Successful in 19s
2026-09-21 10:18:44 +00:00
panxiao81 26179f700f chore: 开始退役旧静态 Gitea Runner
yaml / yaml (pull_request) Successful in 16s
2026-09-21 10:17:06 +00:00
panxiao81 1ac99f4476 Merge pull request '部署 Runner 结构化生命周期日志' (#128) from deploy/structured-runner-logs into main
yaml / yaml (push) Successful in 23s
2026-09-21 09:46:47 +00:00
panxiao81 891c6b90e7 deploy: 更新 Runner 结构化日志镜像
yaml / yaml (pull_request) Successful in 21s
2026-09-21 09:45:33 +00:00
panxiao81 c2314256cc Merge pull request '为 sandbox executor 暴露内网 Runner facade' (#127) from fix/expose-runner-facade-internal into main
yaml / yaml (push) Successful in 26s
2026-09-21 09:18:16 +00:00
panxiao81 8244cdad79 fix: 为 sandbox 暴露 Runner facade
yaml / yaml (pull_request) Successful in 16s
2026-09-21 09:11:29 +00:00
panxiao81 23841556d3 Merge VM 集成测试入口隔离
yaml / yaml (push) Successful in 24s
2026-09-21 09:00:23 +00:00
panxiao81 4daaa38495 deploy: 隔离 VM 集成测试入口
yaml / yaml (pull_request) Successful in 12s
2026-09-21 08:58:21 +00:00
panxiao81 da0630cd29 Merge OpenSandbox Runner 标记修复部署
yaml / yaml (push) Successful in 46s
2026-09-21 08:35:24 +00:00
panxiao81 d0fcfdeddc deploy: 补回 OpenSandbox Runner 标记
yaml / yaml (pull_request) Successful in 23s
2026-09-21 08:34:37 +00:00
panxiao81 ecf395cc8f Merge OpenSandbox metadata 修复部署
yaml / yaml (push) Successful in 58s
2026-09-21 08:07:33 +00:00
panxiao81 9f646d51e7 deploy: 修复 OpenSandbox metadata 编码
yaml / yaml (pull_request) Successful in 56s
2026-09-21 08:04:15 +00:00
panxiao81 5c7e7ff8db Merge Runner 后端独立容量池部署
yaml / yaml (push) Successful in 34s
2026-09-21 07:35:32 +00:00
panxiao81 d6b97355d6 deploy: 启用 Runner 后端容量池
yaml / yaml (pull_request) Successful in 48s
2026-09-21 07:33:31 +00:00
panxiao81 b82ba4d5a0 Merge Pod Runner 四并发恢复
yaml / yaml (push) Successful in 22s
2026-09-21 06:57:37 +00:00
panxiao81 9dc7aabc94 恢复 Pod Runner 四并发
yaml / yaml (pull_request) Successful in 21s
2026-09-21 06:57:09 +00:00
panxiao81 b7c95fd0b8 Merge Runner 无中断滚动与容量隔离
yaml / yaml (push) Successful in 24s
2026-09-21 06:51:39 +00:00
panxiao81 098ff4e27f 更新 Runner 容量隔离镜像
yaml / yaml (pull_request) Successful in 27s
2026-09-21 06:51:37 +00:00
panxiao81 dfc55e1d9e 启用 Runner 无中断滚动更新
yaml / yaml (pull_request) Successful in 29s
2026-09-21 06:39:11 +00:00
panxiao81 18d9f83413 Merge Runner claim 恢复镜像
yaml / yaml (push) Successful in 34s
2026-09-21 06:24:31 +00:00
panxiao81 e5e2ab1b89 更新 Runner claim 恢复镜像
yaml / yaml (pull_request) Successful in 15s
2026-09-21 06:21:20 +00:00
panxiao81 0b236c43d9 Merge OpenSandbox VM Runner canary
yaml / yaml (push) Successful in 14s
2026-09-21 06:12:35 +00:00
panxiao81 762ce6458d 启用 OpenSandbox VM Runner canary
yaml / yaml (pull_request) Successful in 17s
2026-09-21 06:10:59 +00:00
panxiao81 15ef512cfc Merge runner Gitea proxy Host fix
yaml / yaml (push) Failing after 31s
更新 Runner Gitea 反代 Host 修复镜像
2026-09-21 05:42:33 +00:00
panxiao81 a13a5118cf 更新 Runner Gitea 反代 Host 修复镜像
yaml / yaml (pull_request) Failing after 33s
2026-09-21 05:41:50 +00:00
panxiao81 25bc7ee7d6 Merge runner cleanup and checkout proxy fix
yaml / yaml (push) Failing after 41s
更新 Runner 自动清理与 checkout 修复镜像
2026-09-21 05:31:26 +00:00
panxiao81 3d41e0e29b 更新 Runner 自动清理与 checkout 修复镜像
yaml / yaml (pull_request) Failing after 37s
2026-09-21 05:30:09 +00:00
panxiao81 3169fcabd3 Merge pull request '记录 CI 身份 Action 使用入口' (#116) from docs/ci-actions into main 2026-09-21 03:31:03 +00:00
panxiao81 0a42a3ad04 记录 CI 身份 Action 使用入口 2026-09-21 03:30:31 +00:00
panxiao81 9ec034c947 Merge runner facade readiness fix
yaml / yaml (push) Failing after 5m30s
更新 Runner facade 就绪等待镜像
2026-09-20 21:37:42 +00:00
panxiao81 ad9b305aba 更新 Runner facade 就绪等待镜像
yaml / yaml (pull_request) Failing after 5m26s
2026-09-20 21:37:19 +00:00
panxiao81 9696df1f51 Merge pull request #114
yaml / yaml (push) Failing after 10m47s
修正动态 Runner 的 SPIRE agent 绑定
2026-09-20 21:31:11 +00:00
panxiao81 a335bdbc38 修正动态 Runner 的 SPIRE agent 绑定
yaml / yaml (pull_request) Failing after 11m40s
2026-09-20 21:30:48 +00:00
panxiao81 b6623069ad Merge pull request #113
yaml / yaml (push) Failing after 11m36s
更新动态 Runner SPIFFE socket 修复镜像
2026-09-20 21:15:15 +00:00
panxiao81 4e3ef3c33f 更新动态 Runner SPIFFE socket 修复镜像
yaml / yaml (pull_request) Failing after 9m48s
2026-09-20 21:14:08 +00:00
panxiao81 8eac82c272 合并无重叠 scheduler 滚动策略
yaml / yaml (push) Failing after 1m1s
修复 Flux SSA strategy 冲突。
2026-09-20 21:05:02 +00:00
panxiao81 2b232c4cd2 使用无重叠滚动策略部署 scheduler
yaml / yaml (pull_request) Successful in 14s
2026-09-20 21:04:45 +00:00
panxiao81 b51dc3751b 合并 Recreate 策略修复
yaml / yaml (push) Successful in 15s
修复 Flux dry-run 阻塞,生产资源此前尚未切换。
2026-09-20 21:03:06 +00:00
panxiao81 616c87bafd 清理 Recreate 策略遗留字段
yaml / yaml (pull_request) Successful in 14s
2026-09-20 21:02:49 +00:00
panxiao81 0663642583 合并 Go 动态 Runner Pod canary
yaml / yaml (push) Successful in 20s
已完成 yamllint、kustomize 与生产 API server-side dry-run;按单副本 single-flight canary 部署。
2026-09-20 21:01:36 +00:00
panxiao81 68191e0ff4 部署 Go 动态 Runner Pod canary
yaml / yaml (pull_request) Successful in 18s
2026-09-20 21:00:28 +00:00
panxiao81 c368373e74 Merge pull request '纳管并验收 Nexus OCI 仓库' (#109) from feat/nexus-oci-poc into main
terraform / validate (push) Successful in 54s
2026-09-20 20:47:03 +00:00
panxiao81 7bdd152db9 纳管并验收 Nexus OCI 仓库
terraform / validate (pull_request) Successful in 1m21s
2026-09-20 20:46:26 +00:00
panxiao81 ab985c7979 Merge pull request '完成 Nexus Ansible 与 Go 缓存验收' (#108) from feat/nexus-poc-verification into main
terraform / validate (push) Successful in 1m48s
2026-09-20 20:32:45 +00:00
panxiao81 cb137f2044 完成 Nexus Ansible 与 Go 缓存验收
terraform / validate (pull_request) Successful in 49s
2026-09-20 20:28:23 +00:00
panxiao81 83e4ac2f01 Merge pull request '新增 Nexus 统一制品仓库 POC' (#103) from feat/nexus-poc into main
ansible / collection-test (push) Successful in 4m14s
ansible / lint (push) Successful in 5m15s
yaml / yaml (push) Successful in 35s
terraform / validate (push) Successful in 1m8s
Reviewed-on: #103
2026-09-20 19:49:49 +00:00
panxiao81 1df682bc6d Merge pull request '限制 OpenSandbox 仅调度 VM' (#107) from fix/enable-native-pod-worker into main
yaml / yaml (push) Successful in 20s
2026-09-20 17:19:36 +00:00
panxiao81 eb52197eee 限制 OpenSandbox 仅调度 VM
yaml / yaml (pull_request) Successful in 19s
2026-09-20 17:19:12 +00:00
panxiao81 9b9f44333e Merge pull request '在 homelab 启用原生 Pod Worker' (#106) from fix/enable-native-pod-worker into main
yaml / yaml (push) Successful in 24s
2026-09-20 17:16:28 +00:00
panxiao81 8896d58b89 在 homelab 启用原生 Pod Worker
yaml / yaml (pull_request) Successful in 24s
2026-09-20 17:16:26 +00:00
panxiao81 0fd7f87d7e Merge pull request '停止 OpenSandbox 消费 Pod 任务' (#105) from fix/separate-pod-opensandbox into main
yaml / yaml (push) Failing after 0s
2026-09-20 17:13:43 +00:00
panxiao81 89bd7864e6 停止 OpenSandbox 消费 Pod 任务
yaml / yaml (pull_request) Failing after 0s
2026-09-20 17:13:21 +00:00
panxiao81 93f803a0d9 合并 Runner completed ACK 镜像修复
yaml / yaml (push) Failing after 0s
2026-09-18 19:46:33 +00:00
panxiao81 f7d9579221 部署 Runner completed ACK 修复
yaml / yaml (pull_request) Failing after 0s
2026-09-18 19:46:30 +00:00
panxiao81 9fe910a4c9 合并 OpenSandbox Runner 部署修复
ansible / lint (push) Failing after 0s
ansible / collection-test (push) Failing after 0s
yaml / yaml (push) Failing after 0s
2026-09-18 19:34:25 +00:00
panxiao81 28c20fd3e5 恢复持久事件驱动 Runner 调度
ansible / lint (pull_request) Failing after 0s
ansible / collection-test (pull_request) Failing after 0s
yaml / yaml (pull_request) Failing after 0s
2026-09-18 19:27:25 +00:00
panxiao81 110dbcc2c3 新增 Nexus 统一制品仓库 POC
ansible / lint (pull_request) Failing after 0s
ansible / collection-test (pull_request) Failing after 0s
yaml / yaml (pull_request) Failing after 0s
terraform / validate (pull_request) Failing after 0s
2026-09-18 19:11:30 +00:00
panxiao81 28f1de35cb 启用 sandbox 静态身份登记
ansible / lint (pull_request) Failing after 2s
ansible / collection-test (pull_request) Failing after 1s
yaml / yaml (pull_request) Failing after 1s
2026-09-18 19:06:13 +00:00
panxiao81 ad4ed1f0d6 重新触发 OpenSandbox 集成验证
ansible / lint (pull_request) Failing after 0s
ansible / collection-test (pull_request) Failing after 0s
yaml / yaml (pull_request) Failing after 2s
2026-09-18 19:02:44 +00:00
panxiao81 611a5a3d3a 修复 Pool task-executor 数据目录与 guest 身份镜像
ansible / lint (pull_request) Failing after 0s
ansible / collection-test (pull_request) Failing after 0s
yaml / yaml (pull_request) Failing after 0s
2026-09-18 18:59:02 +00:00
panxiao81 7582990655 修复 OpenSandbox 冷启动超时与旧 worker 清理
ansible / lint (pull_request) Failing after 0s
ansible / collection-test (pull_request) Failing after 0s
yaml / yaml (pull_request) Failing after 0s
2026-09-18 18:53:26 +00:00
panxiao81 ec3ce94e12 Merge pull request '接入 OpenSandbox 动态 Runner' (#101) from feat/opensandbox-runner-integration into main
yaml / yaml (push) Successful in 23s
ansible / collection-test (push) Successful in 1m35s
ansible / lint (push) Successful in 2m55s
2026-09-18 18:45:24 +00:00
panxiao81 6eb4875db6 接入 OpenSandbox 动态 Runner
ansible / collection-test (pull_request) Successful in 2m58s
ansible / lint (pull_request) Successful in 4m33s
yaml / yaml (pull_request) Successful in 29s
2026-09-18 18:33:00 +00:00
panxiao81 518dba6036 Merge pull request 确保 OpenSandbox values 立即生效
yaml / yaml (push) Successful in 17s
ansible / collection-test (push) Successful in 1m15s
ansible / lint (push) Successful in 2m7s
2026-09-18 17:52:51 +00:00
panxiao81 4c823f8181 确保 OpenSandbox values 立即生效
yaml / yaml (pull_request) Successful in 27s
ansible / collection-test (pull_request) Successful in 1m16s
ansible / lint (pull_request) Successful in 2m33s
2026-09-18 17:49:26 +00:00
panxiao81 8290082fb4 Merge pull request 为 sandbox 接入 OpenBao External Secrets
yaml / yaml (push) Successful in 15s
terraform / validate (push) Successful in 42s
2026-09-18 17:41:51 +00:00
panxiao81 159a75b710 记录 sandbox ESO 当前状态
yaml / yaml (pull_request) Successful in 16s
terraform / validate (pull_request) Successful in 39s
2026-09-18 17:40:24 +00:00
panxiao81 e0e629794b 为 sandbox 接入 OpenBao External Secrets
yaml / yaml (pull_request) Successful in 17s
terraform / validate (pull_request) Successful in 49s
2026-09-18 17:34:53 +00:00
panxiao81 90ba945d85 Merge pull request 部署 OpenSandbox 控制面
yaml / yaml (push) Successful in 18s
2026-09-18 16:38:13 +00:00
panxiao81 d526fd75d3 部署 OpenSandbox 控制面
yaml / yaml (pull_request) Successful in 17s
2026-09-18 16:36:21 +00:00
panxiao81 18cb2858b9 Merge pull request '延长 sandbox Flux 根同步超时' (#93) from fix/sandbox-flux-root-timeout into main
ansible / collection-test (push) Successful in 1m13s
ansible / lint (push) Successful in 2m23s
Reviewed-on: #93
2026-09-18 16:18:46 +00:00
panxiao81 99d1ec1d6f Merge pull request '记录 Kata guest 内 SPIFFE 身份方案' (#95) from docs/kata-inner-spire into main
Reviewed-on: #95
2026-09-18 16:18:29 +00:00
panxiao81 583dab526a docs: 明确 SPIRE chart 字段缺口 2026-09-17 18:54:00 +00:00
panxiao81 3dbd4c5f31 docs: 记录 Kata guest SPIFFE 身份方案 2026-09-17 18:47:01 +00:00
panxiao81 e67bce5121 fix: 延长 sandbox Flux 根同步超时
ansible / collection-test (pull_request) Successful in 1m8s
ansible / lint (pull_request) Successful in 2m56s
2026-09-17 18:20:33 +00:00
81 changed files with 2132 additions and 463 deletions
+157
View File
@@ -0,0 +1,157 @@
# Nexus Repository POC
本目录声明一个 Nexus Repository Community Edition POC,用来验证一次性 CI runner 通过
网络服务复用 Ansible Galaxy、Go Modules 与 OCI/BuildKit 缓存。Nexus 固定为 `3.96.1`,
镜像固定到官方 multi-arch index digest;LAN 入口为
`https://nexus.ad.ddupan.top`。
## POC 边界
- 单副本 Deployment,`Recreate` 更新,避免一个 RWO 卷被两个 Pod 同时挂载。
- `/nexus-data` 使用 `localpv-zfs-ceph` 上的 50 GiB RWO PVC。
- 资源预算为 250m/2 GiB request、2 CPU/4 GiB limit;JVM heap 上限 2 GiB。
- 使用容器默认的 embedded H2。它只用于 POC;正式接管 OCI 制品前必须迁移到外部
PostgreSQL,并验证备份恢复。
- 入口只在 LAN wildcard Gateway 上发布,不创建公网 DNS 或 Cloudflare route。
- 不套 Authelia forward-auth;它会破坏 Go、Ansible 与 OCI 非浏览器客户端。
- 现有 zot 保持不变。Nexus 完成 OCI、BuildKit cache 和恢复验收前不得迁移或删除 zot。
Terraform provider 创建 `ansible-public`、`go-public`、最小匿名权限与 OCI Bearer Token
Realm。Nexus 3.94 才加入的原生 OCI repository 已有 REST API,但当前锁定的 community
provider 尚未暴露 OCI resource;`terraform/reconcile-oci.sh` 因此根据 3.96.1 实例 Swagger
固定的 JSON schema,幂等调和 `oci-hosted`、`oci-proxy` 与 `oci-public`。不得绕过该入口在
UI 中创建无人管理的长期 repository。
## 部署
Flux 从 `clusters/homelab/apps/nexus.yaml` 协调本目录,并依赖 Envoy Gateway 与 OpenEBS。
合并前只渲染配置,不直接 apply:
```bash
kubectl kustomize apps/nexus
```
合并并由 Flux 部署后检查:
```bash
kubectl -n flux-system get kustomization nexus
kubectl -n nexus get pod,pvc,service,httproute
kubectl -n nexus logs deployment/nexus --tail=100
```
启动可能需要数分钟,startup probe 允许最多十分钟。不要因初次启动较慢反复删除 Pod;
先确认 PVC 已 Bound、Pod 没有 OOM,以及日志仍在推进。
DNS 期望状态已加入 `infrastructure/dns/records.yml`,需从 Samba AD Ansible root 以
`--check --diff` 核对后再按其 README 应用 DNS tag。没有 DNS 时可先用 port-forward
验证应用,但不能据此宣称 Gateway 路径已通过。
## 首次初始化与 Terraform
初始管理员密码生成在 PVC 的 `/nexus-data/admin.password`。只在交互式终端中读取并立即
完成首次密码轮换;不得把密码复制进 shell tracing、工单、Git 或命令参数。随后将
Terraform 管理账号的凭据存入 OpenBao,由 CI 通过 Terraform input variable 注入以下
环境变量:
```text
TF_VAR_nexus_url=https://nexus.ad.ddupan.top
TF_VAR_nexus_username=admin
TF_VAR_nexus_password=<OpenBao kv/infra/nexus 的 admin_password 字段>
```
`terraform/` 使用 `sonatype-nexus-community/sonatyperepo` 1.17.0,当前声明:
- `ansible-galaxy-proxy` → `https://galaxy.ansible.com`
- `ansible-public` group
- `go-proxy` → `https://proxy.golang.org`
- `go-public` group
provider credential 不写入 HCL 或 tfvars。正式 apply 前还必须为这个独立 Terraform root
配置远端 backend;本地 state 只允许用于可丢弃的 POC,不提交。验证命令:
```bash
terraform -chdir=apps/nexus/terraform init -backend=false
terraform -chdir=apps/nexus/terraform validate
```
先以 `--check` 查看 OCI repository 漂移,再明确 apply;脚本只从上述环境变量取得凭据,
用临时 `0600` netrc 调用 REST API,退出时删除:
```bash
apps/nexus/terraform/reconcile-oci.sh --check
apps/nexus/terraform/reconcile-oci.sh --apply
```
Terraform 同时把内置 `anonymous` 用户从默认的全仓库 `nx-anonymous` 角色收窄到
`ansible-public`、其返回制品 URL 使用的 `ansible-galaxy-proxy`、`go-public`,以及
`oci-public`/`oci-proxy` 的 `browse/read` 权限。`oci-hosted` 不向匿名用户开放。首次接管
已有实例时先执行
`terraform import sonatyperepo_user.anonymous anonymous,default`,再 apply;不要先启用默认的
全仓库匿名读取。
2026-09-20 的 POC 现场验收已确认:Flux 与 Pod Ready、PVC Bound、HTTPRoute 通过 HTTPS
返回 Nexus 状态 200,Samba DNS 已幂等收敛。全新客户端目录通过匿名入口下载
`community.general:11.2.0` 时冷缓存为 8.49 秒、热缓存为 1.89 秒,两次 tarball SHA-256
一致;`golang.org/x/[email protected]` 为 2.92 秒与 1.51 秒。
## 客户端验收
先验证冷缓存,再原样重复命令验证热缓存;记录 Nexus 请求、上游流量和耗时,不只观察
命令成功。Ansible 配置的 URL 必须以 `/` 结尾:
```ini
[galaxy]
server_list = nexus
[galaxy_server.nexus]
url = https://nexus.ad.ddupan.top/repository/ansible-public/
```
```bash
ansible-galaxy collection install -r collections/requirements.yml \
-p .ansible/collections
```
Go POC 使用:
```bash
GOPROXY=https://nexus.ad.ddupan.top/repository/go-public/ go mod download
```
私有 `git.ddupan.top/*` module 的 `GOPRIVATE`、凭据与是否允许 `direct` fallback 在实际
workflow 中单独决定;不要让私有 module path 意外发往公共 proxy。
OCI 使用 path-based routing:匿名公共拉取地址形如
`nexus.ad.ddupan.top/oci-public/library/alpine:3.22`,认证写入地址形如
`nexus.ad.ddupan.top/oci-hosted/<namespace>/<image>:<tag>`。2026-09-20 现场验收结果:
- `oci-public` 匿名代理拉取 Alpine 冷缓存 4.75 秒、热缓存 0.80 秒,digest 一致;
- `oci-hosted` 认证 push/pull 成功,匿名 pull 返回 401;
- amd64/arm64 OCI image index push 成功,两个平台 manifest 可见;
- Helm chart push/pull digest 与本地 tarball SHA-256 一致;
- Cosign 3.1.3 使用一次性密钥签名并验证成功,OCI 1.1 referrers API 返回一个
`application/vnd.dev.sigstore.bundle.v0.3+json` artifact;
- BuildKit `registry` cache 以 `mode=max` 导出成功,销毁首个 builder 后由新 builder 导入,
两个 `RUN` step 均明确命中 `CACHED`。
本机安装的测试客户端包括 `/usr/local/bin/cosign` 3.1.3;安装时核对官方 Linux amd64
binary SHA-256 `4629c757b7618056f8ddd7e2625ae9fdd94c0372a65049520bc7d9df9efc7f71`。
上述结果仍不代表备份恢复、外部 PostgreSQL 或正式 publisher service account 已完成;
这些项目通过前不得迁移或删除 zot。
## 数据与恢复
POC 的数据库、配置、blob、初始管理员状态都位于 `nexus-data` PVC。删除 Deployment
不会删除 PVC;删除 PVC 会永久删除整个 POC。当前没有独立备份,不能将它用于唯一副本的
正式制品。
恢复验证至少包括:停止写入、取得一致备份、在独立 PVC/实例恢复、登录、列出 repository、
拉取已缓存的 Ansible/Go 制品,并核对 OCI digest/referrers。正式化时再把数据库迁移至
外部 PostgreSQL,并分别定义数据库与 blob 的备份、恢复顺序和 RPO。
参考:
- [Nexus OCI repositories](https://help.sonatype.com/en/oci-repositories.html)
- [Nexus Ansible repositories](https://help.sonatype.com/en/ansible-repositories.html)
- [Nexus Go repositories](https://help.sonatype.com/en/go-repositories.html)
- [官方容器镜像](https://hub.docker.com/r/sonatype/nexus3)
+82
View File
@@ -0,0 +1,82 @@
apiVersion: apps/v1
kind: Deployment
metadata:
name: nexus
namespace: nexus
labels:
app.kubernetes.io/name: nexus
spec:
replicas: 1
strategy:
type: Recreate
selector:
matchLabels:
app.kubernetes.io/name: nexus
template:
metadata:
labels:
app.kubernetes.io/name: nexus
spec:
automountServiceAccountToken: false
securityContext:
fsGroup: 200
fsGroupChangePolicy: OnRootMismatch
runAsGroup: 200
runAsNonRoot: true
runAsUser: 200
seccompProfile:
type: RuntimeDefault
terminationGracePeriodSeconds: 120
containers:
- name: nexus
image: docker.io/sonatype/nexus3:3.96.1@sha256:56142f13432cf072e017aebb2025f201e42ae36ff40bb82618c702504c61f7dd
imagePullPolicy: IfNotPresent
env:
- name: INSTALL4J_ADD_VM_PARAMS
value: >-
-Xms1024m -Xmx2048m -XX:MaxDirectMemorySize=1024m
-Djava.util.prefs.userRoot=/nexus-data/javaprefs
ports:
- name: http
containerPort: 8081
protocol: TCP
resources:
requests:
cpu: 250m
memory: 2Gi
limits:
cpu: "2"
memory: 4Gi
securityContext:
allowPrivilegeEscalation: false
capabilities:
drop:
- ALL
startupProbe:
httpGet:
path: /service/rest/v1/status
port: http
failureThreshold: 60
periodSeconds: 10
timeoutSeconds: 5
readinessProbe:
httpGet:
path: /service/rest/v1/status
port: http
failureThreshold: 6
periodSeconds: 10
timeoutSeconds: 5
livenessProbe:
httpGet:
path: /service/rest/v1/status
port: http
failureThreshold: 6
periodSeconds: 30
timeoutSeconds: 5
volumeMounts:
- name: data
mountPath: /nexus-data
volumes:
- name: data
persistentVolumeClaim:
claimName: nexus-data
+16
View File
@@ -0,0 +1,16 @@
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
name: nexus
namespace: nexus
spec:
parentRefs:
- name: eg
namespace: envoy-gateway-system
sectionName: https
hostnames:
- nexus.ad.ddupan.top
rules:
- backendRefs:
- name: nexus
port: 8081
+9
View File
@@ -0,0 +1,9 @@
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- namespace.yaml
- pvc.yaml
- deployment.yaml
- service.yaml
- httproute.yaml
- networkpolicy.yaml
+8
View File
@@ -0,0 +1,8 @@
apiVersion: v1
kind: Namespace
metadata:
name: nexus
labels:
pod-security.kubernetes.io/enforce: restricted
pod-security.kubernetes.io/audit: restricted
pod-security.kubernetes.io/warn: restricted
+23
View File
@@ -0,0 +1,23 @@
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: nexus-ingress
namespace: nexus
spec:
podSelector:
matchLabels:
app.kubernetes.io/name: nexus
policyTypes:
- Ingress
ingress:
- from:
- namespaceSelector:
matchLabels:
kubernetes.io/metadata.name: envoy-gateway-system
podSelector:
matchLabels:
gateway.envoyproxy.io/owning-gateway-name: eg
gateway.envoyproxy.io/owning-gateway-namespace: envoy-gateway-system
ports:
- protocol: TCP
port: 8081
+12
View File
@@ -0,0 +1,12 @@
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: nexus-data
namespace: nexus
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 50Gi
storageClassName: localpv-zfs-ceph
+14
View File
@@ -0,0 +1,14 @@
apiVersion: v1
kind: Service
metadata:
name: nexus
namespace: nexus
spec:
type: ClusterIP
selector:
app.kubernetes.io/name: nexus
ports:
- name: http
port: 8081
protocol: TCP
targetPort: http
+6
View File
@@ -0,0 +1,6 @@
.terraform/
*.tfstate
*.tfstate.*
*.tfplan
crash.log
crash.*.log
+24
View File
@@ -0,0 +1,24 @@
# This file is maintained automatically by "terraform init".
# Manual edits may be lost in future updates.
provider "registry.terraform.io/sonatype-nexus-community/sonatyperepo" {
version = "1.17.0"
constraints = "1.17.0"
hashes = [
"h1:uKhvhhhI7B+HBsh0zq/ybqKt+EnOGyI6rjcRCtj79ZA=",
"zh:0dde99e7b343fa01f8eefc378171fb8621bedb20f59157d6cc8e3d46c738105f",
"zh:3315929df254a3a6ac27c8c846c2006f7d2a91fadc014351bc4d617f948e5bf9",
"zh:36be5a455af3ce4e187de26753de63e78c1ee9a32dba0135c6cf96a6c1fff25f",
"zh:3f73f7ff57b8c339a7c7ac37653e2dc0b2dd9dcc3f3a538788e7e3ac838337b2",
"zh:40286ecca4c22ab7ae90618ac6d2743f5055199dac81cf5204a4a397c784d439",
"zh:4d24e5c0195fb3155b1967583ee64cfeda402d7cc7f3c73369438f6c69f4245b",
"zh:828a9d7aceaac36af7f9c07af43ec8d20a89148780645d170ffb1c68b2da792d",
"zh:a5ab04de3fe626ec57c832618c6f990abd6610f81e132621651e0b180b970cff",
"zh:a959fa6090a8c0f53739879184e7346423494aee598003df0d1ab4a22b2eee91",
"zh:bdda26c2f03f918bbe59e75abea44868fafda019c3a543725331195df126350b",
"zh:d8048e149ee97ba62971e6a79355d59887bc6d10fcf72cc2feff3d0a2582670c",
"zh:dd36f9988af4e1ca5b1ca7b7bb6f658df9a220dfcda7fec7392fedfe9064f652",
"zh:dda2688d46c7e539fe97e8fe9d3ec81fb364170e018d9c6a681364c8955d4e9d",
"zh:e6b519afe2dea1c0434f766eb6bc9ba78cc5b6ef2c311c2ca3c65cb24744f31f",
]
}
+17
View File
@@ -0,0 +1,17 @@
{
"name": "oci-hosted",
"online": true,
"storage": {
"blobStoreName": "default",
"strictContentTypeValidation": true,
"writePolicy": "ALLOW",
"latestPolicy": false
},
"oci": {
"v1Enabled": false,
"forceBasicAuth": false,
"pathEnabled": true
},
"component": { "proprietaryComponents": false },
"cosign": { "enforcement": "NONE" }
}
+27
View File
@@ -0,0 +1,27 @@
{
"name": "oci-proxy",
"online": true,
"storage": {
"blobStoreName": "default",
"strictContentTypeValidation": true
},
"oci": {
"v1Enabled": false,
"forceBasicAuth": false,
"pathEnabled": true
},
"ociProxy": {
"indexType": "HUB",
"cacheForeignLayers": false,
"foreignLayerUrlWhitelist": []
},
"proxy": {
"remoteUrl": "https://registry-1.docker.io",
"contentMaxAge": 1440,
"metadataMaxAge": 60,
"preserveEncodedCharacters": false
},
"negativeCache": { "enabled": true, "timeToLive": 60 },
"httpClient": { "blocked": false, "autoBlock": true },
"cosign": { "enforcement": "NONE" }
}
+15
View File
@@ -0,0 +1,15 @@
{
"name": "oci-public",
"online": true,
"storage": {
"blobStoreName": "default",
"strictContentTypeValidation": true
},
"group": { "memberNames": ["oci-proxy"] },
"oci": {
"v1Enabled": false,
"forceBasicAuth": false,
"pathEnabled": true
},
"cosign": { "enforcement": "NONE" }
}
+80
View File
@@ -0,0 +1,80 @@
#!/usr/bin/env bash
set -euo pipefail
mode="${1:---check}"
case "$mode" in
--check | --apply) ;;
*) echo "usage: $0 [--check|--apply]" >&2; exit 2 ;;
esac
: "${TF_VAR_nexus_url:?set TF_VAR_nexus_url}"
: "${TF_VAR_nexus_username:?set TF_VAR_nexus_username}"
: "${TF_VAR_nexus_password:?set TF_VAR_nexus_password}"
script_dir="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")" && pwd)"
auth_file="$(mktemp /tmp/nexus-oci-auth.XXXXXX)"
trap 'rm -f -- "$auth_file"' EXIT
chmod 0600 "$auth_file"
printf 'machine %s\nlogin %s\npassword %s\n' \
"${TF_VAR_nexus_url#*://}" "$TF_VAR_nexus_username" \
"$TF_VAR_nexus_password" >"$auth_file"
drift=0
for entry in \
"hosted:$script_dir/oci/oci-hosted.json" \
"proxy:$script_dir/oci/oci-proxy.json" \
"group:$script_dir/oci/oci-public.json"; do
repository_type="${entry%%:*}"
desired_file="${entry#*:}"
repository_name="$(jq -er '.name' "$desired_file")"
endpoint="$TF_VAR_nexus_url/service/rest/v1/repositories/oci/$repository_type"
current_file="$(mktemp /tmp/nexus-oci-current.XXXXXX)"
status="$(curl --silent --show-error --netrc-file "$auth_file" \
--output "$current_file" --write-out '%{http_code}' \
"$endpoint/$repository_name")"
if [[ "$status" == 404 ]]; then
drift=1
if [[ "$mode" == --apply ]]; then
curl --fail --silent --show-error --netrc-file "$auth_file" \
--header 'Content-Type: application/json' \
--data-binary "@$desired_file" "$endpoint"
echo "created $repository_name"
else
echo "missing $repository_name" >&2
fi
elif [[ "$status" == 200 ]]; then
if jq -e --slurpfile desired "$desired_file" '
def subset($actual; $wanted):
if ($wanted | type) == "object" then
all($wanted | keys[];
($actual[.] != null) and subset($actual[.]; $wanted[.]))
else
$actual == $wanted
end;
subset(.; $desired[0])
' "$current_file" >/dev/null; then
echo "in sync $repository_name"
else
drift=1
if [[ "$mode" == --apply ]]; then
curl --fail --silent --show-error --netrc-file "$auth_file" \
--request PUT --header 'Content-Type: application/json' \
--data-binary "@$desired_file" "$endpoint/$repository_name"
echo "updated $repository_name"
else
echo "drifted $repository_name" >&2
fi
fi
else
cat "$current_file" >&2
echo "unexpected HTTP $status for $repository_name" >&2
exit 1
fi
rm -f -- "$current_file"
done
if [[ "$mode" == --check && "$drift" -ne 0 ]]; then
exit 1
fi
+64
View File
@@ -0,0 +1,64 @@
locals {
proxy_http_client = {
auto_block = true
blocked = false
}
proxy_negative_cache = {
enabled = true
time_to_live = 60
}
repository_storage = {
blob_store_name = "default"
strict_content_type_validation = true
}
}
resource "sonatyperepo_repository_ansiblegalaxy_proxy" "galaxy" {
name = "ansible-galaxy-proxy"
online = true
http_client = local.proxy_http_client
negative_cache = local.proxy_negative_cache
proxy = {
remote_url = "https://galaxy.ansible.com"
content_max_age = 1440
metadata_max_age = 60
}
storage = local.repository_storage
}
resource "sonatyperepo_repository_ansiblegalaxy_group" "public" {
name = "ansible-public"
online = true
group = {
member_names = [sonatyperepo_repository_ansiblegalaxy_proxy.galaxy.name]
}
storage = local.repository_storage
}
resource "sonatyperepo_repository_go_proxy" "public" {
name = "go-proxy"
online = true
http_client = local.proxy_http_client
negative_cache = local.proxy_negative_cache
proxy = {
remote_url = "https://proxy.golang.org"
content_max_age = 1440
metadata_max_age = 60
}
storage = local.repository_storage
}
resource "sonatyperepo_repository_go_group" "public" {
name = "go-public"
online = true
group = {
member_names = [sonatyperepo_repository_go_proxy.public.name]
}
storage = local.repository_storage
}
+75
View File
@@ -0,0 +1,75 @@
resource "sonatyperepo_privilege_repository_view" "anonymous_ansible" {
name = "ci-anonymous-ansible-read"
description = "Anonymous read access to the Ansible Galaxy group"
actions = ["BROWSE", "READ"]
format = "ansiblegalaxy"
repository = sonatyperepo_repository_ansiblegalaxy_group.public.name
}
resource "sonatyperepo_privilege_repository_view" "anonymous_ansible_proxy" {
name = "ci-anonymous-ansible-proxy-read"
description = "Anonymous artifact read access to the Ansible Galaxy proxy"
actions = ["BROWSE", "READ"]
format = "ansiblegalaxy"
repository = sonatyperepo_repository_ansiblegalaxy_proxy.galaxy.name
}
resource "sonatyperepo_privilege_repository_view" "anonymous_go" {
name = "ci-anonymous-go-read"
description = "Anonymous read access to the Go module group"
actions = ["BROWSE", "READ"]
format = "go"
repository = sonatyperepo_repository_go_group.public.name
}
resource "sonatyperepo_privilege_repository_view" "anonymous_oci_public" {
name = "ci-anonymous-oci-public-read"
description = "Anonymous read access to the public OCI group"
actions = ["BROWSE", "READ"]
format = "oci"
repository = "oci-public"
}
resource "sonatyperepo_privilege_repository_view" "anonymous_oci_proxy" {
name = "ci-anonymous-oci-proxy-read"
description = "Anonymous read access to the OCI proxy member"
actions = ["BROWSE", "READ"]
format = "oci"
repository = "oci-proxy"
}
resource "sonatyperepo_role" "anonymous_ci" {
id = "ci-anonymous-read"
name = "CI anonymous read"
description = "Read-only access to public CI dependency proxy groups"
privileges = [
sonatyperepo_privilege_repository_view.anonymous_ansible.name,
sonatyperepo_privilege_repository_view.anonymous_ansible_proxy.name,
sonatyperepo_privilege_repository_view.anonymous_go.name,
sonatyperepo_privilege_repository_view.anonymous_oci_public.name,
sonatyperepo_privilege_repository_view.anonymous_oci_proxy.name,
]
roles = []
}
resource "sonatyperepo_user" "anonymous" {
user_id = "anonymous"
first_name = "Anonymous"
last_name = "User"
email_address = "[email protected]"
status = "active"
roles = [sonatyperepo_role.anonymous_ci.id]
}
resource "sonatyperepo_system_anonymous_access" "ci" {
enabled = true
user_id = sonatyperepo_user.anonymous.user_id
realm_name = "NexusAuthorizingRealm"
}
resource "sonatyperepo_security_realms" "active" {
active = [
"NexusAuthenticatingRealm",
"OciBearerToken",
]
}
+33
View File
@@ -0,0 +1,33 @@
terraform {
required_version = ">= 1.11.0"
required_providers {
sonatyperepo = {
source = "sonatype-nexus-community/sonatyperepo"
version = "1.17.0"
}
}
}
provider "sonatyperepo" {
url = var.nexus_url
username = var.nexus_username
password = var.nexus_password
}
variable "nexus_url" {
description = "Nexus Repository base URL"
type = string
}
variable "nexus_username" {
description = "Nexus Terraform management username"
type = string
sensitive = true
}
variable "nexus_password" {
description = "Nexus Terraform management password"
type = string
sensitive = true
}
+2
View File
@@ -53,4 +53,6 @@ sudo k3s kubectl -n flux-system get gitrepositories,kustomizations
- SPIRE 已按 hardened chart 内部 fork `0.30.2-ddupan.1`(基于上游 `0.30.2`,SPIRE - SPIRE 已按 hardened chart 内部 fork `0.30.2-ddupan.1`(基于上游 `0.30.2`,SPIRE
`1.15.3`)声明,使用共享 `1.15.3`)声明,使用共享
PostgreSQL 与独立 signing-key PVC;首次上线和 OpenBao JWT-SVID PoC 尚待合并后验证; PostgreSQL 与独立 signing-key PVC;首次上线和 OpenBao JWT-SVID PoC 尚待合并后验证;
- Nexus Repository CE POC 已加入 GitOps 声明,计划验证 Ansible Galaxy、Go Modules 与
OCI/BuildKit 缓存;尚未部署或完成现场验收,现有 zot 保持不变;
- root Kustomization 与所有 brownfield 子 Kustomization 继续保持 `prune: false`。 - root Kustomization 与所有 brownfield 子 Kustomization 继续保持 `prune: false`。
+3 -2
View File
@@ -6,11 +6,12 @@ metadata:
spec: spec:
dependsOn: dependsOn:
- name: external-secrets - name: external-secrets
- name: nats
- name: spire - name: spire
interval: 10m interval: 10m
path: ./platform/dynamic-runner path: ./platform/dynamic-runner
prune: false # The runner backends are replaceable. Prune is required when a retired
# worker is removed from the component; otherwise it keeps consuming work.
prune: true
sourceRef: sourceRef:
kind: GitRepository kind: GitRepository
name: flux-system name: flux-system
+22
View File
@@ -0,0 +1,22 @@
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
name: nexus
namespace: flux-system
spec:
dependsOn:
- name: envoy-gateway
- name: openebs
healthChecks:
- apiVersion: apps/v1
kind: Deployment
name: nexus
namespace: nexus
interval: 10m
path: ./apps/nexus
prune: false
sourceRef:
kind: GitRepository
name: flux-system
timeout: 15m
wait: true
+1 -1
View File
@@ -7,7 +7,6 @@ resources:
- apps/envoy-gateway.yaml - apps/envoy-gateway.yaml
- apps/external-secrets.yaml - apps/external-secrets.yaml
- apps/gitea.yaml - apps/gitea.yaml
- apps/gitea-actions.yaml
- apps/http-echo.yaml - apps/http-echo.yaml
- apps/openebs.yaml - apps/openebs.yaml
- apps/nats.yaml - apps/nats.yaml
@@ -15,3 +14,4 @@ resources:
- apps/spire.yaml - apps/spire.yaml
- apps/observability.yaml - apps/observability.yaml
- apps/zot.yaml - apps/zot.yaml
- apps/nexus.yaml
+15 -4
View File
@@ -15,7 +15,8 @@ Root bootstrap 已完成。后续按依赖顺序分别引入:
1. 监控 CRD、kube-state-metrics 以及 kubelet/cAdvisor 抓取配置; 1. 监控 CRD、kube-state-metrics 以及 kubelet/cAdvisor 抓取配置;
2. SPIRE Agent、SPIFFE CSI Driver 与 workload registration; 2. SPIRE Agent、SPIFFE CSI Driver 与 workload registration;
3. Kata Containers、`block-plain` RuntimeClass; 3. Kata Containers、`block-plain` RuntimeClass;
4. OpenSandbox operator/server 及 `ci-pod`、`ci-vm` Pools。 4. 独立 External Secrets Operator 与 sandbox 专用 OpenBao auth backend;
5. OpenSandbox controller/server;CI Pool 与 runner 调度器随后独立接入。
每一阶段单独合并并等待对应 Flux Kustomization Ready,不在 bootstrap 时一次性部署。 每一阶段单独合并并等待对应 Flux Kustomization Ready,不在 bootstrap 时一次性部署。
第一阶段监控拆为 `monitoring-operator` 与依赖它的 `monitoring`,防止 VM CR 在 第一阶段监控拆为 `monitoring-operator` 与依赖它的 `monitoring`,防止 VM CR 在
@@ -42,15 +43,25 @@ attestation。Server 使用 external bundle publisher 持续维护 sandbox
显式关闭 Server 与 OIDC Provider,只部署 Agent DaemonSet 和 SPIFFE CSI Driver;因此 显式关闭 Server 与 OIDC Provider,只部署 Agent DaemonSet 和 SPIFFE CSI Driver;因此
不会产生第二个 trust root。 不会产生第二个 trust root。
`spire-smoke` namespace、ServiceAccount 和 `sandbox-spire-smoke` ClusterSPIFFEID 是 `spire-smoke` namespace、ServiceAccount 和 `sandbox-spire-smoke` ClusterSPIFFEID 只用于
普通 Pod 与后续 Kata guest 的回归夹具,稳定身份为 普通 Pod 的 CSI 回归夹具,稳定身份为 `spiffe://ddupan.top/sandbox/smoke`。Kata guest
`spiffe://ddupan.top/sandbox/smoke`。测试 Pod 临时创建并在验收后删除,身份声明保留。 不能复用 node Agent 暴露的 Unix socket;virtio-fs 只能呈现 socket 路径,不能把连接
跨过 VM 边界。Kata workload 必须使用 guest 内 Agent,具体约束见
`platform/sandbox-kata/README.md`。测试 Pod 临时创建并在验收后删除,普通 Pod 的身份
声明保留。
Kata 阶段使用官方 4.1.0 `kata-deploy` chart 的短生命周期 `job` 模式,逐节点安装并 Kata 阶段使用官方 4.1.0 `kata-deploy` chart 的短生命周期 `job` 模式,逐节点安装并
重启 K3s。只启用 `kata-clh-runtime-rs`,不创建默认 `kata` 别名;该 handler 的 重启 K3s。只启用 `kata-clh-runtime-rs`,不创建默认 `kata` 别名;该 handler 的
`emptyDir` 固定使用 `block-plain`,为 Docker/BuildKit overlay2 与 kind 提供 guest `emptyDir` 固定使用 `block-plain`,为 Docker/BuildKit overlay2 与 kind 提供 guest
内块设备文件系统。详细限制与上线验收见 `platform/sandbox-kata/README.md`。 内块设备文件系统。详细限制与上线验收见 `platform/sandbox-kata/README.md`。
Sandbox 的 ESO 通过独立 `auth/kubernetes-sandbox` 向 OpenBao 证明 ServiceAccount
身份,只能读取共享的 `kv/k8s/opensandbox-api`。它不保存 reviewer JWT 或长期 Bao token;相关
Terraform 与 Flux 边界见 `platform/sandbox-external-secrets/README.md`。
OpenSandbox 阶段固定官方源码 commit 与 umbrella chart `0.2.2`,只部署 controller、
ClusterIP server 和 CRD。API key 由 ESO 从 OpenBao 投影,明文不进入 Git。
## 监控边界 ## 监控边界
这里只管理 sandbox LXC 内的 Kubernetes 监控,不负责 PVE 宿主监控。LXC 与宿主共享 这里只管理 sandbox LXC 内的 Kubernetes 监控,不负责 PVE 宿主监控。LXC 与宿主共享
@@ -1,14 +1,18 @@
---
apiVersion: kustomize.toolkit.fluxcd.io/v1 apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization kind: Kustomization
metadata: metadata:
name: gitea-actions name: ci-runners
namespace: flux-system namespace: flux-system
spec: spec:
dependsOn:
- name: opensandbox
- name: spire-agents
interval: 10m interval: 10m
path: ./platform/gitea-runner path: ./platform/sandbox-ci-runners
prune: false prune: true
sourceRef: sourceRef:
kind: GitRepository kind: GitRepository
name: flux-system name: flux-system
timeout: 3m timeout: 20m
wait: false wait: true
@@ -0,0 +1,17 @@
---
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
name: external-secrets-operator
namespace: flux-system
spec:
dependsOn:
- name: monitoring-operator
interval: 10m
path: ./platform/sandbox-external-secrets/operator
prune: true
sourceRef:
kind: GitRepository
name: flux-system
timeout: 10m
wait: true
@@ -0,0 +1,17 @@
---
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
name: external-secrets
namespace: flux-system
spec:
dependsOn:
- name: external-secrets-operator
interval: 10m
path: ./platform/sandbox-external-secrets/config
prune: true
sourceRef:
kind: GitRepository
name: flux-system
timeout: 10m
wait: true
+19
View File
@@ -0,0 +1,19 @@
---
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
name: opensandbox
namespace: flux-system
spec:
dependsOn:
- name: external-secrets
- name: kata
- name: monitoring-operator
interval: 10m
path: ./platform/sandbox-opensandbox
prune: true
sourceRef:
kind: GitRepository
name: flux-system
timeout: 15m
wait: true
+4
View File
@@ -7,3 +7,7 @@ resources:
- apps/spire-bootstrap.yaml - apps/spire-bootstrap.yaml
- apps/spire-agents.yaml - apps/spire-agents.yaml
- apps/kata.yaml - apps/kata.yaml
- apps/external-secrets-operator.yaml
- apps/external-secrets.yaml
- apps/opensandbox.yaml
- apps/ci-runners.yaml
+1
View File
@@ -18,6 +18,7 @@ homelab_dns:
- { zone: ad.ddupan.top, name: metrics-write, type: A, values: [192.168.10.127] } - { zone: ad.ddupan.top, name: metrics-write, type: A, values: [192.168.10.127] }
- { zone: ad.ddupan.top, name: netbox, type: A, values: [192.168.10.127] } - { zone: ad.ddupan.top, name: netbox, type: A, values: [192.168.10.127] }
- { zone: ad.ddupan.top, name: nats, type: A, values: [192.168.10.127] } - { zone: ad.ddupan.top, name: nats, type: A, values: [192.168.10.127] }
- { zone: ad.ddupan.top, name: nexus, type: A, values: [192.168.10.127] }
- { zone: ad.ddupan.top, name: s3, type: A, values: [192.168.10.127] } - { zone: ad.ddupan.top, name: s3, type: A, values: [192.168.10.127] }
- { zone: ad.ddupan.top, name: spire-oidc, type: A, values: [192.168.10.127] } - { zone: ad.ddupan.top, name: spire-oidc, type: A, values: [192.168.10.127] }
- { zone: ad.ddupan.top, name: spire-server, type: A, values: [192.168.10.127] } - { zone: ad.ddupan.top, name: spire-server, type: A, values: [192.168.10.127] }
+8 -1
View File
@@ -51,7 +51,7 @@ few things Terraform must not own.**
| secrets engine mounts (`kv`, `pki`, `ssh-client-signer`) | the daemon, Raft, TLS files, systemd | | secrets engine mounts (`kv`, `pki`, `ssh-client-signer`) | the daemon, Raft, TLS files, systemd |
| PKI role, issuing/CRL URLs, cluster paths, **ACME** | `bao operator init` / unseal (manual, PGP-wrapped) | | PKI role, issuing/CRL URLs, cluster paths, **ACME** | `bao operator init` / unseal (manual, PGP-wrapped) |
| SSH signing role (`ai-agent`) | **PKI root CA + SSH CA signing key** | | SSH signing role (`ai-agent`) | **PKI root CA + SSH CA signing key** |
| OIDC auth *mount* and *role* | OIDC **client secret** (`auth/oidc/config`) | | OIDC/Kubernetes auth mount、config 与 role | OIDC **client secret** (`auth/oidc/config`) |
| all policies | snapshot token + script + systemd timer | | all policies | snapshot token + script + systemd timer |
| | host-level CA trust distribution (`openbao_ssh_ca_trust`) | | | host-level CA trust distribution (`openbao_ssh_ca_trust`) |
@@ -92,6 +92,13 @@ then `VAULT_ADDR`/`VAULT_TOKEN`), mirroring how `smtp-relay/terraform` uses
native `openbao/openbao` provider is published only to the OpenTofu registry and native `openbao/openbao` provider is published only to the OpenTofu registry and
cannot be resolved by the HashiCorp `terraform` CLI. cannot be resolved by the HashiCorp `terraform` CLI.
Sandbox 集群使用独立的 `auth/kubernetes-sandbox`。其 API 地址、公开 Kubernetes CA、
ESO role 与只读 `kv/k8s/opensandbox-api` policy 全部由 Terraform 管理;CA 位于
`terraform/certs/sandbox-kubernetes-ca.crt`。集群重建并轮换 CA 后,先更新该文件并
apply,再让 Flux 恢复 ESO reconciliation。该 backend 不保存 reviewer JWT,而是使用
ESO 的短期登录 JWT 执行 TokenReview。该组资源已于 2026-09-18 apply,随后复验 plan
为 zero-diff。
## DNS ## DNS
`bao.ad.ddupan.top` is an **internal-only** name — not a public Cloudflare record and `bao.ad.ddupan.top` is an **internal-only** name — not a public Cloudflare record and
+30
View File
@@ -59,3 +59,33 @@ resource "vault_kubernetes_auth_backend_role" "external_secrets" {
# in a long TTL and every extra hour is a longer-lived credential in memory. # in a long TTL and every extra hour is a longer-lived credential in memory.
token_ttl = 3600 token_ttl = 3600
} }
# A Kubernetes auth mount can validate identities from only the API server it is
# configured against. The sandbox cluster therefore cannot reuse auth/kubernetes,
# whose TokenReview endpoint belongs to homelab.
resource "vault_auth_backend" "sandbox_kubernetes" {
type = "kubernetes"
path = "kubernetes-sandbox"
}
resource "vault_kubernetes_auth_backend_config" "sandbox" {
backend = vault_auth_backend.sandbox_kubernetes.path
kubernetes_host = "https://10.60.0.13:6443"
kubernetes_ca_cert = file("${path.module}/certs/sandbox-kubernetes-ca.crt")
disable_local_ca_jwt = true
# Deliberately omit token_reviewer_jwt. OpenBao uses the login JWT for
# TokenReview; the sandbox external-secrets ServiceAccount is bound only to
# system:auth-delegator and all issued JWTs remain short-lived.
}
resource "vault_kubernetes_auth_backend_role" "sandbox_external_secrets" {
backend = vault_auth_backend.sandbox_kubernetes.path
role_name = "external-secrets"
bound_service_account_names = ["external-secrets"]
bound_service_account_namespaces = ["external-secrets"]
token_policies = [vault_policy.sandbox_external_secrets.name]
token_ttl = 3600
}
@@ -0,0 +1,10 @@
-----BEGIN CERTIFICATE-----
MIIBdjCCAR2gAwIBAgIBADAKBggqhkjOPQQDAjAjMSEwHwYDVQQDDBhrM3Mtc2Vy
dmVyLWNhQDE3ODk2NTI4NTcwHhcNMjYwOTE3MTI0NzM3WhcNMzYwOTE0MTI0NzM3
WjAjMSEwHwYDVQQDDBhrM3Mtc2VydmVyLWNhQDE3ODk2NTI4NTcwWTATBgcqhkjO
PQIBBggqhkjOPQMBBwNCAAR4SbqzTXZnlZdUPz7viN6+dYbB1Maw44Qiepn9r5XG
sOzkYkN8t1aG3Ugo8TqQ3xJaKkM89n1Rluj0vbOhiNajo0IwQDAOBgNVHQ8BAf8E
BAMCAqQwDwYDVR0TAQH/BAUwAwEB/zAdBgNVHQ4EFgQUyLSGoKAKAJuiniuRdBLG
XYaDQC8wCgYIKoZIzj0EAwIDRwAwRAIgFkVzyUZexk/ynnxBEOg+3foJv3WKqAei
hTSRjO1gL0UCIFbBKR7BMrJJAgW3DJFeeBM+b+tTg93jNx55qZACbFOL
-----END CERTIFICATE-----
@@ -35,3 +35,10 @@ resource "vault_policy" "external_secrets" {
name = "external-secrets" name = "external-secrets"
policy = file("${path.module}/policies/external-secrets.hcl") policy = file("${path.module}/policies/external-secrets.hcl")
} }
# The sandbox cluster has its own Kubernetes auth backend and a deliberately
# narrower KV view than the homelab ESO instance.
resource "vault_policy" "sandbox_external_secrets" {
name = "sandbox-external-secrets"
policy = file("${path.module}/policies/sandbox-external-secrets.hcl")
}
@@ -0,0 +1,9 @@
# Read only the shared OpenSandbox control-plane API key. The same Bao object is
# consumed by the server in sandbox and, later, by the scheduler in homelab.
path "kv/data/k8s/opensandbox-api" {
capabilities = ["read"]
}
path "kv/metadata/k8s/opensandbox-api" {
capabilities = ["read"]
}
@@ -83,6 +83,11 @@ vyos_postgresql_primary_address: "10.60.0.11"
vyos_sandbox_k3s_api_address: "10.60.0.13" vyos_sandbox_k3s_api_address: "10.60.0.13"
vyos_sandbox_k3s_api_interface: eth1 vyos_sandbox_k3s_api_interface: eth1
vyos_sandbox_k3s_api_port: 6443 vyos_sandbox_k3s_api_port: 6443
vyos_opensandbox_api_port: 8080
vyos_opensandbox_node_port: 30080
# Lifecycle create is synchronous and a cold Kata image pull can exceed the
# HAProxy 50-second default. Keep this below OpenSandbox's overall timeout.
vyos_opensandbox_api_timeout: 600
vyos_sandbox_k3s_servers: vyos_sandbox_k3s_servers:
- name: sandbox1 - name: sandbox1
address: "10.60.0.11" address: "10.60.0.11"
@@ -15,6 +15,20 @@ set interfaces ethernet {{ i.iface }} address {{ i.address }}
set interfaces ethernet {{ i.iface }} description '{{ i.description }}' set interfaces ethernet {{ i.iface }} description '{{ i.description }}'
{% endfor %} {% endfor %}
{# OpenSandbox stays on a NodePort; VyOS owns the stable routed frontend. #}
set load-balancing haproxy service opensandbox-api listen-address {{ vyos_sandbox_k3s_api_address }}
set load-balancing haproxy service opensandbox-api port {{ vyos_opensandbox_api_port }}
set load-balancing haproxy service opensandbox-api mode 'tcp'
set load-balancing haproxy service opensandbox-api backend 'opensandbox-api'
set load-balancing haproxy service opensandbox-api timeout client {{ vyos_opensandbox_api_timeout }}
set load-balancing haproxy backend opensandbox-api mode 'tcp'
set load-balancing haproxy backend opensandbox-api timeout server {{ vyos_opensandbox_api_timeout }}
{% for server in vyos_sandbox_k3s_servers %}
set load-balancing haproxy backend opensandbox-api server {{ server.name }} address {{ server.address }}
set load-balancing haproxy backend opensandbox-api server {{ server.name }} port {{ vyos_opensandbox_node_port }}
set load-balancing haproxy backend opensandbox-api server {{ server.name }} check
{% endfor %}
{# --- default route out; OSPF carries the rest --- #} {# --- default route out; OSPF carries the rest --- #}
set protocols static route 0.0.0.0/0 next-hop {{ vyos_lan_gateway }} set protocols static route 0.0.0.0/0 next-hop {{ vyos_lan_gateway }}
+12 -2
View File
@@ -18,7 +18,8 @@ Flux 管理以下 Kubernetes 资源:
- Kata Containers 和 CI 专用的 `block-plain` RuntimeClass; - Kata Containers 和 CI 专用的 `block-plain` RuntimeClass;
- SPIRE Agent、SPIFFE CSI Driver 与 workload identity 声明; - SPIRE Agent、SPIFFE CSI Driver 与 workload identity 声明;
- vmagent、kube-state-metrics、kubelet/cAdvisor scrape 配置和告警; - vmagent、kube-state-metrics、kubelet/cAdvisor scrape 配置和告警;
- OpenSandbox operator/server、`ci-pod` 与 `ci-vm` Pools。 - sandbox External Secrets Operator、OpenSandbox controller/server;CI Pool 与 runner
调度器由 runner 项目接入。
同一个对象只能有一个 owner。Ansible 不直接部署上述集群内 workload;Flux 不管理 同一个对象只能有一个 owner。Ansible 不直接部署上述集群内 workload;Flux 不管理
LXC、K3s datastore 或 K3s 本身。 LXC、K3s datastore 或 K3s 本身。
@@ -92,6 +93,13 @@ ansible-playbook site.yml
ansible-playbook k3s.yml ansible-playbook k3s.yml
``` ```
只 reconcile Flux controllers 与 root sync manifest(不触碰 LXC、PostgreSQL 或
K3s lifecycle):
```bash
ansible-playbook flux.yml
```
K3s 外部 datastore URI 由运行时 `SANDBOX_K3S_DB_PASSWORD` 生成,密码在 URI 中 K3s 外部 datastore URI 由运行时 `SANDBOX_K3S_DB_PASSWORD` 生成,密码在 URI 中
进行 URL 编码,最终仅持久化于节点 root 可读的 `/etc/rancher/k3s/config.yaml` 进行 URL 编码,最终仅持久化于节点 root 可读的 `/etc/rancher/k3s/config.yaml`
(mode `0600`)。首节点生成的 K3s join token 仅在同一次 Ansible run 内传给第二节点; (mode `0600`)。首节点生成的 K3s join token 仅在同一次 Ansible run 内传给第二节点;
@@ -106,7 +114,9 @@ ansible-playbook verify.yml
当前已经声明 LXC 生命周期、最小 OS baseline、PostgreSQL 和 K3s,包括系统级 当前已经声明 LXC 生命周期、最小 OS baseline、PostgreSQL 和 K3s,包括系统级
homelab CA trust。Flux `v2.9.5` controllers 与 root sync 也由 Ansible 通过 K3s homelab CA trust。Flux `v2.9.5` controllers 与 root sync 也由 Ansible 通过 K3s
server manifests 管理;root 使用 homelab CA 访问公开 Gitea 仓库,不保存 Git token。 server manifests 管理;root 使用 homelab CA 访问公开 Gitea 仓库,不保存 Git token。
集群内 workload 由 `clusters/sandbox/` 分阶段纳入 Flux。 集群内 workload 由 `clusters/sandbox/` 分阶段纳入 Flux。root Kustomization 的健康检查
timeout 为 40 分钟,用于覆盖 Kata 等首次安装时会逐节点重启 K3s 的子
Kustomization;各子项仍保留自己的更短 timeout,故障会在对应子项先行暴露。
## SPIRE 跨集群 bootstrap ## SPIRE 跨集群 bootstrap
@@ -0,0 +1,6 @@
---
- name: Reconcile Flux controllers and root sync
hosts: sandbox1
gather_facts: false
roles:
- sandbox_flux
@@ -35,5 +35,5 @@ spec:
sourceRef: sourceRef:
kind: GitRepository kind: GitRepository
name: flux-system name: flux-system
timeout: 3m timeout: 40m
wait: true wait: true
+92 -36
View File
@@ -1,51 +1,107 @@
# Gitea dynamic runner controller # Gitea dynamic runner
此目录只管理 homelab 中的 controller 部署。controller、worker、Cloud Hypervisor 本目录部署单副本 Go controller,在同一进程运行 RunnerService scheduler、原生
launcher 和 guest runner 的源码与发布位于独立仓库 Kubernetes Pod worker、OpenSandbox VM worker 和 SPIFFE mTLS facade:
`panxiao81/gitea-dynamic-runner`。
当前 bootstrap controller 接收 Gitea `workflow_job` webhook,将 `[self-hosted, pod]` 和
`[self-hosted, vm]` 的 queued job 分别发布到 NATS。Pod worker 在本集群创建一次性
privileged host runner;Docker、BuildKit 和 kind 由 workflow 自行 setup。内部
endpoint:
```text ```text
http://dynamic-runner-controller.dynamic-runner.svc.cluster.local:8787/webhook Gitea RunnerService -> scheduler -> JetStream ci.runner.pod
|
v
homelab Kubernetes Pod
|
SPIFFE mTLS RunnerService facade
|
v
Gitea
``` ```
OpenBao 路径: Pod backend 不经过 OpenSandbox。VM backend 后续启用时才访问 VyOS 暴露的
OpenSandbox Lifecycle API;本目录不修改 sandbox 平台侧 ESO、Bao Terraform 或
OpenSandbox chart 所有权边界。
- `kv/k8s/nats.ci_producer_password`:已有 NATS producer 密码。 ## Canary 安全边界
- `kv/k8s/nats.ci_worker_password`:已有 NATS worker 密码。
- `kv/k8s/dynamic-runner.webhook_secret`:Gitea webhook HMAC secret。
- `kv/k8s/gitea-runner.token`:现有 instance runner registration token。
首期 controller 与 runner 镜像由 laptop 本机构建后导入 k3s containerd,作为 CI - Deployment 为单副本,滚动策略固定 `maxSurge: 1`、`maxUnavailable: 0`,保证新
发布链路建立前的 bootstrap。部署使用 `imagePullPolicy: Never`。正式发布 workflow facade Ready 后才终止旧实例。scheduler 必须通过 Kubernetes Lease 保持单 leader,
获得专用 SPIFFE ID 后,必须将 image 改为 zot digest 并移除本地导入步骤。 不能依赖 Recreate 避免重复领取。
- scheduler 使用一个 runner registration,并按总容量启动并发 `FetchTask` goroutine;
`POD_CAPACITY=4` 与 `VM_CAPACITY=1` 分别限制两个 durable consumer 和 backend
admission pool。池满时 assignment 保持 JetStream pending,任一 backend 不占用
另一方的执行槽位,也不会创建超出容量的 workload。
- rollout 重叠期间只有持有 `Lease/dynamic-runner-scheduler` 的 controller 执行
`FetchTask`;所有 Ready 实例都可通过 backend metadata 恢复 claim 并服务 facade。
- executor 镜像使用 digest;Pod 以 UID 2000 运行,SPIRE `ClusterStaticEntry` 同时绑定
具体 Pod UID 与 `unix:uid:2000`。
- facade 通过仅内网可路由的 `192.168.10.127:30443` NodePort 提供给 sandbox executor;
双方使用 Workload API X509-SVID mTLS,并按 SPIFFE ID 而不是 IP/DNS 名验证服务端。
该入口不经过公网或 Cloudflare Tunnel。
- controller 的 SPIFFE ID 固定为
`spiffe://ddupan.top/ns/dynamic-runner/sa/dynamic-runner-controller`。
- NATS 保留既有最小权限分离:`ci-producer` 仅 publish,`ci-worker` 仅 pull/ACK。
## 身份绑定 facade claim registry 在启动时从 Pod/OpenSandbox metadata 恢复;scheduler leadership
由 Kubernetes Lease 持久化协调。Deployment 仍保持 `replicas: 1`,滚动更新期间允许
一个额外 Pod 提供 facade 连续性。
queued webhook 只负责创建没有业务身份的 Pod。runner 实际领取任务后,Gitea 的 ## Secret 边界
`in_progress` webhook 会携带实际 `runner_name`;controller 将 binding 消息发布到
NATS,Pod worker 再给对应 Pod 添加:
```text `ExternalSecret/dynamic-runner` 从既有 `ClusterSecretStore/openbao` 读取:
ci.ddupan.top/identity-bound=true
ci.ddupan.top/spiffe-path=<owner>/<repository>/<percent-encoded-job-name> - `kv/k8s/nats`:producer/worker 密码;
- `kv/k8s/opensandbox-api:api_key`:保留给后续 VM worker;
- `kv/k8s/gitea-runner:token`:保留的 runner registration token;
- `kv/k8s/dynamic-runner`:scheduler UUID/token、facade HMAC key 和回滚所需 webhook secret。
scheduler credential 由官方 Gitea Runner v3.5.0 一次注册生成;它只挂载到 controller,
不会进入 executor。facade capability key 至少 32 字节,controller 为每个 assignment
确定性生成独立 capability。不要打印 Secret、创建静态 Bao token或把 credential 写入
Git。OpenBao 写入使用本机 SPIFFE JWT 换取的短期 `local-development` token。
## Workflow 身份与依赖配置
workflow 可复用 [`panxiao81/ci-actions`](https://git.ddupan.top/panxiao81/ci-actions)
中的 `spiffe-openbao-login@v1` 和 `setup-nexus@v1`。这不改变 runner 的权限边界:
runner 仅提供 Node.js 20、`spire-agent` 与 Workload API socket,workflow 负责声明 Bao
role、audience 和具体用途,目标服务 policy 决定是否授权。短期 Bao token 会进入
Actions job 临时文件,因此这些 Action 只允许在本目录管理的一次性 Pod/VM executor
中使用,不能迁移到共享或持久 runner。
匿名读取 Nexus public repository 只需 `setup-nexus@v1`,不应为了依赖下载额外申请
Bao 凭据;需要发布制品时再为对应 repository 建立独立 service account 与最小权限
policy。
## 首次验收
合并后先观察 Flux 与 controller,不要立即开启 VM:
```bash
flux reconcile kustomization dynamic-runner --with-source
kubectl -n dynamic-runner wait externalsecret/dynamic-runner \
--for=condition=Ready --timeout=2m
kubectl -n dynamic-runner rollout status deploy/dynamic-runner-controller --timeout=5m
kubectl -n dynamic-runner logs deploy/dynamic-runner-controller -f
``` ```
`ClusterSPIFFEID/gitea-dynamic-runner` 只匹配已经绑定的 Pod,并签发 确认 scheduler 只领取一条 `[self-hosted,pod]` task,然后验证:
`spiffe://ddupan.top/ci/<owner>/<repository>/<job-name>`。runner 的 job-start hook 在
SVID 可用之前不会放行第一步,因此不能根据 queued 事件错配身份。
每个 runner Pod 使用 `gitea-dynamic-runner` ServiceAccount。该 ServiceAccount 没有 1. assignment message 进入并离开 durable `pod` consumer;
Kubernetes API 权限;只有 `dynamic-runner-pod-worker` ServiceAccount 能在本 namespace 2. `gitea-task-<task-id>` Pod 创建,取得实际 Pod UID;
create/get/patch/delete Pod。 3. 同名 `ClusterStaticEntry` 的 parent ID 包含该 UID,SPIFFE ID 使用 repository/job key;
4. 官方 runner v3.5.0 经 facade claim 精确 task,Gitea 实时收到日志和终态;
5. Pod consumer 达到 capacity 时 VM consumer 仍可独立接受任务。
长期实现将由兼容 Gitea RunnerService 的 scheduler 直接领取 task,再交给 Pod/VM Pod 与 VM 都在 Gitea 接受终态后先把 terminal marker 写入各自 backend metadata,再由
executor;届时删除 webhook、临时 runner 注册和 identity binding 消息。跟踪见 lifecycle reconciler 删除执行器。首次 VM 测试仍须观察 BatchSandbox、Pod 与
`panxiao81/gitea-dynamic-runner` issue #7。 ClusterStaticEntry 全部消失;完整自动清理通过前不得提高 `POD_CAPACITY` 或
`VM_CAPACITY`。
Gitea webhook 只订阅 `workflow_job`,content type 使用 JSON,secret 与 Bao 中值 VM backend 已通过 `vm-dev` canary 完成 Docker、kind、SPIFFE 和完整生命周期测试。
一致。不要启用 `send_everything`,否则 controller 会收到无关仓库事件。 生产调度标签为 `[self-hosted, vm]`,初始保持 `VM_CAPACITY=1`;扩容前先观察实际任务的
资源水位、等待时间以及 OpenSandbox 是否存在 terminal sandbox 残留。
## 回滚
若 controller 在领取 task 前失败,回滚到前一 commit 的 Python controller/Pod worker
manifests。若已经创建 `gitea-task-*` Pod,先保留现场并核对 Gitea task 状态,不能直接
重启 scheduler 造成重复执行。scheduler registration 和 capability key 保留在 Bao,
回滚不需要删除或打印它们。
+4 -5
View File
@@ -1,17 +1,16 @@
apiVersion: spire.spiffe.io/v1alpha1 apiVersion: spire.spiffe.io/v1alpha1
kind: ClusterSPIFFEID kind: ClusterSPIFFEID
metadata: metadata:
name: gitea-dynamic-runner name: dynamic-runner-controller
spec: spec:
className: spire-mgmt-spire className: spire-mgmt-spire
spiffeIDTemplate: 'spiffe://{{ .TrustDomain }}/ci/{{ index .PodMeta.Annotations "ci.ddupan.top/spiffe-path" }}' spiffeIDTemplate: spiffe://ddupan.top/ns/dynamic-runner/sa/dynamic-runner-controller
namespaceSelector: namespaceSelector:
matchLabels: matchLabels:
kubernetes.io/metadata.name: dynamic-runner kubernetes.io/metadata.name: dynamic-runner
podSelector: podSelector:
matchLabels: matchLabels:
app.kubernetes.io/name: gitea-dynamic-runner app.kubernetes.io/name: dynamic-runner-controller
ci.ddupan.top/identity-bound: "true"
workloadSelectorTemplates: workloadSelectorTemplates:
- k8s:ns:dynamic-runner - k8s:ns:dynamic-runner
- k8s:sa:gitea-dynamic-runner - k8s:sa:dynamic-runner-controller
+82 -28
View File
@@ -5,6 +5,11 @@ metadata:
namespace: dynamic-runner namespace: dynamic-runner
spec: spec:
replicas: 1 replicas: 1
strategy:
type: RollingUpdate
rollingUpdate:
maxSurge: 1
maxUnavailable: 0
selector: selector:
matchLabels: matchLabels:
app.kubernetes.io/name: dynamic-runner-controller app.kubernetes.io/name: dynamic-runner-controller
@@ -14,17 +19,10 @@ spec:
app.kubernetes.io/name: dynamic-runner-controller app.kubernetes.io/name: dynamic-runner-controller
spec: spec:
serviceAccountName: dynamic-runner-controller serviceAccountName: dynamic-runner-controller
automountServiceAccountToken: false
initContainers: initContainers:
- name: fetch-internal-ca - name: fetch-internal-ca
image: curlimages/curl:8.16.0@sha256:463eaf6072688fe96ac64fa623fe73e1dbe25d8ad6c34404a669ad3ce1f104b6 image: curlimages/curl:8.16.0@sha256:463eaf6072688fe96ac64fa623fe73e1dbe25d8ad6c34404a669ad3ce1f104b6
args: args: [--fail, --silent, --show-error, --output, /trust/ca.pem, https://bao.ad.ddupan.top:8200/v1/pki/ca/pem]
- --fail
- --silent
- --show-error
- --output
- /trust/ca.pem
- https://bao.ad.ddupan.top:8200/v1/pki/ca/pem
securityContext: securityContext:
allowPrivilegeEscalation: false allowPrivilegeEscalation: false
capabilities: capabilities:
@@ -40,40 +38,89 @@ spec:
mountPath: /trust mountPath: /trust
containers: containers:
- name: controller - name: controller
# Bootstrap import on laptop. Replace with a zot digest after the image: zot.ad.ddupan.top/panxiao81/gitea-dynamic-runner-controller@sha256:7374a08a238ca1c76a9aee33c57520ae0edaa9b5b443708487d5b3a69a61da4e
# repository's image publishing workflow has a dedicated identity. imagePullPolicy: IfNotPresent
image: gitea-dynamic-runner-controller:0.3.0-bootstrap args: [controller]
imagePullPolicy: Never
env: env:
- name: COMPONENTS
value: scheduler,pod-worker,vm-worker
- name: GITEA_INSTANCE_URL
value: https://git.ddupan.top
- name: GITEA_RUNNER_UUID_FILE
value: /run/dynamic-runner-secrets/scheduler-uuid
- name: GITEA_RUNNER_TOKEN_FILE
value: /run/dynamic-runner-secrets/scheduler-token
- name: NATS_URL - name: NATS_URL
value: tls://nats.ad.ddupan.top:4222 value: tls://nats.ad.ddupan.top:4222
- name: NATS_CA_FILE - name: NATS_CA_FILE
value: /run/trust/ca.pem value: /run/trust/ca.pem
- name: NATS_PASSWORD_FILE - name: NATS_PRODUCER_PASSWORD_FILE
value: /run/dynamic-runner-secrets/nats-password value: /run/dynamic-runner-secrets/nats-password
- name: WEBHOOK_SECRET_FILE - name: NATS_WORKER_PASSWORD_FILE
value: /run/dynamic-runner-secrets/webhook-secret value: /run/dynamic-runner-secrets/nats-worker-password
- name: RUNNER_FACADE_CAPABILITY_KEY_FILE
value: /run/dynamic-runner-secrets/facade-capability-key
- name: RUNNER_FACADE_LISTEN
value: :8443
- name: RUNNER_FACADE_URL
value: https://192.168.10.127:30443
- name: RUNNER_FACADE_SPIFFE_ID
value: spiffe://ddupan.top/ns/dynamic-runner/sa/dynamic-runner-controller
- name: SPIFFE_ENDPOINT_SOCKET
value: unix:///run/spire/agent-sockets/spire-agent.sock
- name: POD_NAMESPACE
valueFrom:
fieldRef:
fieldPath: metadata.namespace
- name: POD_EXECUTOR_IMAGE
value: zot.ad.ddupan.top/panxiao81/gitea-dynamic-runner-runner@sha256:325d75a208b1a1c6f3b4d4705e47bbc58b34733edcd12f689c60769ab6772a59
- name: POD_SERVICE_ACCOUNT
value: gitea-dynamic-runner
- name: POD_EXECUTOR_UID
value: "2000"
- name: POD_CAPACITY
value: "4"
- name: OPENSANDBOX_API
value: http://10.60.0.13:8080
- name: OPENSANDBOX_API_KEY_FILE
value: /run/dynamic-runner-secrets/opensandbox-api-key
- name: OPENSANDBOX_POOL
value: ci-vm
- name: VM_CAPACITY
value: "1"
- name: VM_RUNNER_LABEL
value: vm
- name: VM_TIMEOUT_SECONDS
value: "14400"
- name: SPIRE_CLUSTER
value: homelab
- name: SPIRE_CLASS
value: spire-mgmt-spire
- name: SPIRE_AGENT_ID
value: spiffe://ddupan.top/spire/agent/k8s_psat/homelab/cd2d0233-c4ea-4031-8327-e7e359e766dd
- name: SSL_CERT_FILE
value: /run/trust/ca.pem
ports: ports:
- name: http - name: facade
containerPort: 8787 containerPort: 8443
readinessProbe: readinessProbe:
httpGet: httpGet: null
path: /healthz tcpSocket:
port: http port: facade
periodSeconds: 5 periodSeconds: 5
livenessProbe: livenessProbe:
httpGet: httpGet: null
path: /healthz tcpSocket:
port: http port: facade
initialDelaySeconds: 10 initialDelaySeconds: 15
periodSeconds: 10 periodSeconds: 10
resources: resources:
requests: requests:
cpu: 25m cpu: 50m
memory: 32Mi memory: 64Mi
limits: limits:
cpu: 250m cpu: 500m
memory: 128Mi memory: 256Mi
securityContext: securityContext:
allowPrivilegeEscalation: false allowPrivilegeEscalation: false
capabilities: capabilities:
@@ -91,6 +138,9 @@ spec:
- name: trust - name: trust
mountPath: /run/trust mountPath: /run/trust
readOnly: true readOnly: true
- name: spire-agent-socket
mountPath: /run/spire/agent-sockets
readOnly: true
securityContext: securityContext:
fsGroup: 65532 fsGroup: 65532
fsGroupChangePolicy: OnRootMismatch fsGroupChangePolicy: OnRootMismatch
@@ -104,3 +154,7 @@ spec:
- name: trust - name: trust
emptyDir: emptyDir:
sizeLimit: 1Mi sizeLimit: 1Mi
- name: spire-agent-socket
csi:
driver: csi.spiffe.io
readOnly: true
@@ -20,6 +20,10 @@ spec:
remoteRef: remoteRef:
key: k8s/nats key: k8s/nats
property: ci_worker_password property: ci_worker_password
- secretKey: opensandbox-api-key
remoteRef:
key: k8s/opensandbox-api
property: api_key
- secretKey: webhook-secret - secretKey: webhook-secret
remoteRef: remoteRef:
key: k8s/dynamic-runner key: k8s/dynamic-runner
@@ -28,3 +32,15 @@ spec:
remoteRef: remoteRef:
key: k8s/gitea-runner key: k8s/gitea-runner
property: token property: token
- secretKey: scheduler-uuid
remoteRef:
key: k8s/dynamic-runner
property: scheduler_uuid
- secretKey: scheduler-token
remoteRef:
key: k8s/dynamic-runner
property: scheduler_token
- secretKey: facade-capability-key
remoteRef:
key: k8s/dynamic-runner
property: facade_capability_key
@@ -6,5 +6,4 @@ resources:
- rbac.yaml - rbac.yaml
- clusterspiffeid.yaml - clusterspiffeid.yaml
- deployment.yaml - deployment.yaml
- pod-worker-deployment.yaml
- service.yaml - service.yaml
@@ -1,100 +0,0 @@
apiVersion: apps/v1
kind: Deployment
metadata:
name: dynamic-runner-pod-worker
namespace: dynamic-runner
spec:
replicas: 1
selector:
matchLabels:
app.kubernetes.io/name: dynamic-runner-pod-worker
template:
metadata:
labels:
app.kubernetes.io/name: dynamic-runner-pod-worker
spec:
serviceAccountName: dynamic-runner-pod-worker
initContainers:
- name: fetch-internal-ca
image: curlimages/curl:8.16.0@sha256:463eaf6072688fe96ac64fa623fe73e1dbe25d8ad6c34404a669ad3ce1f104b6
args:
- --fail
- --silent
- --show-error
- --output
- /trust/ca.pem
- https://bao.ad.ddupan.top:8200/v1/pki/ca/pem
securityContext:
allowPrivilegeEscalation: false
capabilities:
drop: [ALL]
readOnlyRootFilesystem: true
runAsNonRoot: true
runAsUser: 101
runAsGroup: 102
seccompProfile:
type: RuntimeDefault
volumeMounts:
- name: trust
mountPath: /trust
containers:
- name: pod-worker
image: gitea-dynamic-runner-controller:0.3.0-bootstrap
imagePullPolicy: Never
command: [/venv/bin/gitea-dynamic-runner-pod-worker]
env:
- name: NATS_URL
value: tls://nats.ad.ddupan.top:4222
- name: NATS_CA_FILE
value: /run/trust/ca.pem
- name: NATS_PASSWORD_FILE
value: /run/dynamic-runner-secrets/nats-worker-password
- name: RUNNER_NAMESPACE
valueFrom:
fieldRef:
fieldPath: metadata.namespace
- name: RUNNER_IMAGE
value: zot.ad.ddupan.top/panxiao81/gitea-dynamic-runner-runner@sha256:4c61f6315453d68a827ee9542f1345ed86576eaaf62325eb803c8fe3f06ddf2a
- name: RUNNER_SERVICE_ACCOUNT
value: gitea-dynamic-runner
- name: RUNNER_TOKEN_SECRET
value: dynamic-runner
- name: RUNNER_CAPACITY
value: "4"
resources:
requests:
cpu: 25m
memory: 32Mi
limits:
cpu: 250m
memory: 128Mi
securityContext:
allowPrivilegeEscalation: false
capabilities:
drop: [ALL]
readOnlyRootFilesystem: true
runAsNonRoot: true
runAsUser: 65532
runAsGroup: 65532
seccompProfile:
type: RuntimeDefault
volumeMounts:
- name: secret
mountPath: /run/dynamic-runner-secrets
readOnly: true
- name: trust
mountPath: /run/trust
readOnly: true
securityContext:
fsGroup: 65532
fsGroupChangePolicy: OnRootMismatch
seccompProfile:
type: RuntimeDefault
volumes:
- name: secret
secret:
secretName: dynamic-runner
defaultMode: 0400
- name: trust
emptyDir:
sizeLimit: 1Mi
+30 -11
View File
@@ -6,12 +6,6 @@ metadata:
--- ---
apiVersion: v1 apiVersion: v1
kind: ServiceAccount kind: ServiceAccount
metadata:
name: dynamic-runner-pod-worker
namespace: dynamic-runner
---
apiVersion: v1
kind: ServiceAccount
metadata: metadata:
name: gitea-dynamic-runner name: gitea-dynamic-runner
namespace: dynamic-runner namespace: dynamic-runner
@@ -19,23 +13,48 @@ metadata:
apiVersion: rbac.authorization.k8s.io/v1 apiVersion: rbac.authorization.k8s.io/v1
kind: Role kind: Role
metadata: metadata:
name: dynamic-runner-pod-worker name: dynamic-runner-controller
namespace: dynamic-runner namespace: dynamic-runner
rules: rules:
- apiGroups: [""] - apiGroups: [""]
resources: [pods] resources: [pods]
verbs: [create, get, patch, delete] verbs: [create, get, list, watch, patch, delete]
- apiGroups: [coordination.k8s.io]
resources: [leases]
verbs: [create, get, list, watch, update, patch]
--- ---
apiVersion: rbac.authorization.k8s.io/v1 apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding kind: RoleBinding
metadata: metadata:
name: dynamic-runner-pod-worker name: dynamic-runner-controller
namespace: dynamic-runner namespace: dynamic-runner
roleRef: roleRef:
apiGroup: rbac.authorization.k8s.io apiGroup: rbac.authorization.k8s.io
kind: Role kind: Role
name: dynamic-runner-pod-worker name: dynamic-runner-controller
subjects: subjects:
- kind: ServiceAccount - kind: ServiceAccount
name: dynamic-runner-pod-worker name: dynamic-runner-controller
namespace: dynamic-runner
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: dynamic-runner-spiffe-entries
rules:
- apiGroups: [spire.spiffe.io]
resources: [clusterstaticentries]
verbs: [create, get, delete]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: dynamic-runner-spiffe-entries
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: dynamic-runner-spiffe-entries
subjects:
- kind: ServiceAccount
name: dynamic-runner-controller
namespace: dynamic-runner namespace: dynamic-runner
+5 -3
View File
@@ -4,9 +4,11 @@ metadata:
name: dynamic-runner-controller name: dynamic-runner-controller
namespace: dynamic-runner namespace: dynamic-runner
spec: spec:
type: NodePort
selector: selector:
app.kubernetes.io/name: dynamic-runner-controller app.kubernetes.io/name: dynamic-runner-controller
ports: ports:
- name: http - name: facade
port: 8787 port: 8443
targetPort: http targetPort: facade
nodePort: 30443
-114
View File
@@ -1,114 +0,0 @@
# Gitea Actions runner
This is the bootstrap runner for Gitea Actions. One persistent runner Pod accepts
up to four jobs; each job runs in a dynamically created container inside a
Docker-in-Docker daemon. The official chart runs DinD privileged. Rootless DinD
would still be privileged and is blocked by the node's AppArmor user-namespace
policy, so this deployment uses regular DinD instead of weakening that host-wide
policy. Only trusted workflows may target this runner.
DinD 同时使用 `--mtu=1450` 和
`--default-network-opt=bridge=com.docker.network.driver.mtu=1450`,与 k3s Pod 的
`eth0` 一致。前者只覆盖 Docker 默认 bridge;act 为每个 job 创建 user-defined
bridge,必须由后者设置默认 MTU。不要在未验证节点 Pod MTU 的情况下删除或修改这
两个参数:MTU 1500 的 job 容器虽然能够解析 GitHub、甚至建立 TCP 连接,但较大的
TLS 数据包会在嵌套网络路径中丢失,表现为 `github.com` / `api.github.com` 超时或
`setup-go` 每次请求卡满 6 分钟后重试。Pod 网络和默认 Docker bridge 正常不代表
Actions job bridge 正常。
The runner is registered at instance scope so it is available to every repository
on this Gitea instance. Repository permissions and protected-branch review are
therefore the security boundary; do not enable Actions for untrusted repositories.
The runner registration token is authoritative in OpenBao at
`kv/k8s/gitea-runner`. External Secrets Operator projects its `token` property to
the `gitea-runner-token` Secret. Never put the token in this directory or a Helm
command line.
## SPIRE 与 OCI 发布
runner Pod 使用专用 ServiceAccount `gitea-actions`,并由精确匹配 namespace、
ServiceAccount 隐含的 Pod、以及 chart labels 的 `ClusterSPIFFEID` 获得:
```text
spiffe://ddupan.top/ci/gitea-actions
```
SPIFFE CSI socket 同时只读挂载到 runner 和 DinD。act 的 volume allowlist 只允许
`/run/spire/agent-sockets`;workflow 仍必须在 job container 中显式请求该 bind
mount。原因是 bind mount 由 DinD 内的 dockerd 解析,只挂 runner 容器无法让 job
访问 Workload API。
该身份不是通用 registry 管理员。zot 只对明确列出的 CI 镜像仓库授予
`read/create/update`,不授予 delete 或其他仓库写入。workflow 应获取
`aud=zot` 的短期 JWT-SVID,并经 stdin 传给 registry client,不得把 JWT、X.509
SVID 或 Docker auth 写入 workspace/artifact。
## Flux 接管状态
该 release 最初通过下述 review-first 流程手动 bootstrap。下一个 GitOps 阶段将
使用 Flux `HelmRelease` 接管它,并首先固定现有 chart `0.1.1`,不在接管 PR 中升级。
迁移前审计发现:Helm 保存的 user-supplied values 和 release manifest 仍描述失败的
rootless DinD 尝试,但 live StatefulSet 与本目录 `values.yaml` 都已经使用 regular
DinD。首次 reconcile 的验收条件是修正 Helm 存储状态,同时 live Pod spec、PVC
identity、runner capacity 和在线状态保持不变。接管稳定后再用独立 PR 升级 chart。
接管分两阶段:第一阶段提交 `suspend: true` 的 HelmRelease、HelmRepository 和由
`values.yaml` 生成的 ConfigMap。Flux 只登记这些对象,不执行 Helm action。合并后
检查 HelmRepository Ready,并用固定 chart 重复比较期望清单与 live StatefulSet;
第二阶段解除 suspend。第一阶段已经确认 source Ready、完整 chart render 与 live
资源零差异,且登记过程中现有 runner 没有 rollout。失败重试使用
`RetryOnFailure`,不会用 stored rootless release 做 rollback。
## 历史 review-first bootstrap
这是 Flux 安装前执行过的一次性手动部署流程,保留用于恢复和审计:
1. Merge the reviewed PR.
2. As a Gitea site administrator, create an instance-scoped runner registration
token under **Site Administration → Actions → Runners**.
3. Store it as the `token` property at `kv/k8s/gitea-runner` without exposing it
in shell history:
```bash
read -rsp 'Runner token: ' runner_token
printf '%s' "$runner_token" | bao kv put kv/k8s/gitea-runner token=-
unset runner_token
```
4. From the updated `main`, create the namespace and ExternalSecret, then wait
for `SecretSynced=True`:
```bash
KUBECONFIG="$HOME/.kube/config" k3s kubectl apply \
-f platform/gitea-runner/namespace.yaml
KUBECONFIG="$HOME/.kube/config" k3s kubectl apply \
-f platform/gitea-runner/external-secret.yaml
KUBECONFIG="$HOME/.kube/config" k3s kubectl wait \
--namespace gitea-actions \
--for=condition=Ready externalsecret/gitea-runner-token \
--timeout=60s
```
5. Install chart `actions` version `0.1.1` from
`https://dl.gitea.com/charts/` with this `values.yaml`:
```bash
helm repo add gitea-charts https://dl.gitea.com/charts/
helm repo update gitea-charts
helm upgrade --install gitea-actions gitea-charts/actions \
--namespace gitea-actions \
--version 0.1.1 \
--values platform/gitea-runner/values.yaml \
--wait --timeout 10m
```
6. Confirm the runner is online, then re-run the queued lint workflow.
Do not deploy from an unmerged feature branch. Do not use `--set` for the token.
The 1 GiB PVC preserves `.runner` identity. Docker image layers are ephemeral;
the Pod has a 20 GiB ephemeral-storage limit. Terraform apply jobs must use a
workflow concurrency group because runner capacity does not serialize access to
a shared state.
@@ -1,17 +0,0 @@
apiVersion: spire.spiffe.io/v1alpha1
kind: ClusterSPIFFEID
metadata:
name: gitea-actions
spec:
className: spire-mgmt-spire
spiffeIDTemplate: spiffe://{{ .TrustDomain }}/ci/gitea-actions
namespaceSelector:
matchLabels:
kubernetes.io/metadata.name: gitea-actions
podSelector:
matchLabels:
app.kubernetes.io/instance: gitea-actions
app.kubernetes.io/name: actions-runner
workloadSelectorTemplates:
- k8s:ns:gitea-actions
- k8s:sa:gitea-actions
@@ -1,8 +0,0 @@
apiVersion: source.toolkit.fluxcd.io/v1
kind: HelmRepository
metadata:
name: gitea-charts
namespace: gitea-actions
spec:
interval: 1h
url: https://dl.gitea.com/charts/
-18
View File
@@ -1,18 +0,0 @@
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
generatorOptions:
disableNameSuffixHash: true
labels:
reconcile.fluxcd.io/watch: Enabled
configMapGenerator:
- name: gitea-actions-values
namespace: gitea-actions
files:
- values.yaml=values.yaml
resources:
- namespace.yaml
- serviceaccount.yaml
- clusterspiffeid.yaml
- external-secret.yaml
- helmrepository.yaml
- helmrelease.yaml
@@ -1,6 +0,0 @@
apiVersion: v1
kind: ServiceAccount
metadata:
name: gitea-actions
namespace: gitea-actions
automountServiceAccountToken: false
-81
View File
@@ -1,81 +0,0 @@
enabled: true
giteaRootURL: http://gitea-http.gitea.svc.cluster.local:3000
existingSecret: gitea-runner-token
existingSecretKey: token
statefulset:
replicas: 1
timezone: Etc/UTC
serviceAccountName: gitea-actions
extraVolumes:
- name: spiffe-workload-api
csi:
driver: csi.spiffe.io
readOnly: true
securityContext:
fsGroup: 1000
# Chart 0.1.1 applies this block to both runner and DinD containers.
resources:
requests:
cpu: 250m
memory: 512Mi
ephemeral-storage: 2Gi
limits:
cpu: "4"
memory: 6Gi
ephemeral-storage: 20Gi
persistence:
size: 1Gi
runner:
registry: docker.io
repository: gitea/runner
tag: 2.3.0
pullPolicy: IfNotPresent
extraVolumeMounts:
- name: spiffe-workload-api
mountPath: /run/spire/agent-sockets
readOnly: true
config: |
log:
level: info
runner:
file: .runner
capacity: 4
timeout: 3h
shutdown_timeout: 3h
labels:
- self-hosted:docker://docker.gitea.com/runner-images:ubuntu-latest
cache:
enabled: false
container:
require_docker: true
docker_timeout: 300s
# Workflows must still request this exact bind mount explicitly. The
# allowlist prevents arbitrary host paths from reaching job containers.
valid_volumes:
- /run/spire/agent-sockets
dind:
# The node enforces AppArmor's unprivileged-userns restriction, which blocks
# rootlesskit even though this chart must run DinD privileged either way.
rootless: false
registry: docker.io
repository: docker
tag: 29.7.1-dind
pullPolicy: IfNotPresent
# Bind mounts are resolved by dockerd, so the CSI socket must exist in the
# DinD container as well as in the runner container.
extraVolumeMounts:
- name: spiffe-workload-api
mountPath: /run/spire/agent-sockets
readOnly: true
# k3s uses a 1450-byte pod MTU. Without matching it here, nested Actions
# networks advertise 1500 and GitHub TLS packets disappear on the outer
# overlay path while direct pod traffic remains healthy.
extraArgs:
- --mtu=1450
# --mtu only changes Docker's default bridge. act creates a user-defined
# bridge per job, so give every new bridge the same explicit default.
- --default-network-opt=bridge=com.docker.network.driver.mtu=1450
+84
View File
@@ -0,0 +1,84 @@
# OpenSandbox Gitea runner Pools
本目录只部署零预热的 `ci-vm` Pool、sandbox 内的 SPIFFE identity controller,以及仅供
内网 VyOS 转发的 OpenSandbox NodePort。`ci-vm` 使用 `kata-clh-runtime-rs`,每个 VM
执行单一任务并在结束后删除。普通 Pod job 直接运行在 homelab,不进入 OpenSandbox。
Pool 中 task-executor 接收 Lifecycle API 下发的进程环境。guest-local SPIRE Agent 用
Pod-bound PSAT 向中央 SPIRE 注册;identity controller 从 BatchSandbox allocation
取得真实 Pod UID,再创建精确的 `ClusterStaticEntry`。runner 只有拿到请求中的完整
repository/task SVID 后才领取一次性 Gitea registration token。
VM 与 Pod backend 使用同一个 `gitea-dynamic-runner-runner` executor 镜像。Pool 不运行
常驻 `docker:dind` sidecar;需要 Docker 的 workflow 在 privileged executor 内按任务启动
daemon。Kata VM 中的 workflow 必须先把稀疏 ext4 镜像 loop-mount 到
`/var/lib/docker`,并完成 cgroup v2 nesting 初始化。
`ci-vm` executor 固定请求并限制为 2 CPU/3GiB。真实 kind canary 表明 1 CPU/约 2GiB
虽然能最终启动全部 control-plane 容器,但无法在 kubeadm 超时前提供可用的 API server;
该规格是 nested Kubernetes 任务的容量下限,不是用资源掩盖存储阻塞。
这些 `ClusterStaticEntry` 位于 sandbox 集群,由 central SPIRE Server 内的
`spire-controller-manager-sandbox` 通过受限 external kubeconfig reconcile。必须在
`platform/spire/values.yaml` 显式启用 external controller-manager 的
`reconcile.clusterStaticEntries`(chart 默认关闭);仅看到 CR 存在但没有 status,不算
身份链路就绪。
## 部署依赖
- OpenSandbox chart 和 CRD 已 Ready;
- RuntimeClass `kata-clh-runtime-rs` 已存在;
- 中央 SPIRE 已发布 `ConfigMap/opensandbox/spire-bundle-pem`;
- VyOS `10.60.0.13:8080` 转发 sandbox1/2 的 NodePort `30080`;
- runner/controller 镜像均使用 Zot digest,而不是可变 tag。
本目录不读取 OpenBao,也不修改 OpenSandbox 平台侧 ExternalSecret、ClusterSecretStore
或 Bao policy。OpenSandbox API key 只存在于平台 server Secret 和 homelab controller
Secret,两边由各自身份读取同一 Bao 资源。
## 上线验收
先确认 Secret 和 API 认证,命令不得输出 key:
```bash
kubectl -n dynamic-runner wait externalsecret/dynamic-runner --for=condition=Ready --timeout=2m
kubectl -n dynamic-runner exec deploy/dynamic-runner-controller -- \
wget -qO- http://10.60.0.13:8080/health
```
然后触发一个 `runs-on: [self-hosted, vm]` 的最小 workflow,并同时观察:
```bash
kubectl -n dynamic-runner logs deploy/dynamic-runner-controller -f
kubectl -n opensandbox get pool,batchsandbox,pod -w
kubectl get clusterstaticentry -l app.kubernetes.io/component=opensandbox-identity -w
kubectl -n opensandbox logs deploy/opensandbox-identity -f
```
合格证据必须同时包含:
1. Lifecycle create 成功并选择 `ci-vm`;
2. 分配 Pod 的 `runtimeClassName` 为 `kata-clh-runtime-rs`;
3. entry 的 parent ID 包含该 Pod 的 UID,SPIFFE ID 使用 repository/task;
4. Gitea 显示临时 runner 带 `self-hosted,vm` labels 并完成真实任务;
5. 任务后 BatchSandbox、Pod、ClusterStaticEntry 和临时 runner 均消失。
清理超时对象时只调用 Lifecycle DELETE,不直接删除 Pool Pod。若 controller 已不可用,
可从 OpenSandbox API 按 metadata 定位 sandbox 后执行 DELETE;不要绕过 API 伪造状态。
## 故障定位
- `401`:检查 homelab ExternalSecret Ready 和文件挂载,不打印 Secret;
- `PoolCapacityExhausted`:检查 `ci-vm` 的 `poolMax` 及残留 BatchSandbox;
- runner 等待 SVID:核对 allocation Pod UID、ClusterStaticEntry parentID、guest Agent 日志;
- runner 等待 token:核对 `192.168.10.127:8787` 的 sandbox 到 homelab 路由;
- Docker 任务失败:检查 workflow 的 job-local dockerd 日志及
`/var/run/docker.sock`;Pool 不提供共享或常驻 daemon。
- kind node 的 systemd 报 `Failed to create /init.scope` 或 `Structure needs
cleaning`:确认 workflow 启动 dockerd 前完成 cgroup v2 nesting 初始化。否则
子容器的私有 cgroup namespace 根会变成 `threaded`,systemd 无法创建 domain
cgroup。
- kind 的 kubeadm 卡在 `CreateContainer`,而 nested containerd goroutine 停在
`bbolt` 的 `fdatasync`:不要直接把 Kata shared mount 用作 Docker 数据目录。
workflow 必须在 guest 内创建稀疏 ext4 镜像并 loop-mount 到
`/var/lib/docker`;该镜像随 Sandbox 删除,不得在节点侧遗留。
@@ -0,0 +1,31 @@
apiVersion: v1
kind: ConfigMap
metadata:
name: gitea-ci-spire-agent
namespace: opensandbox
data:
agent.conf: |
agent {
data_dir = "/run/spire/data"
log_level = "INFO"
server_address = "spire-server.ad.ddupan.top"
server_port = "8081"
socket_path = "/run/spire/agent-sockets/spire-agent.sock"
trust_bundle_path = "/run/spire/bundle/bundle.pem"
trust_domain = "ddupan.top"
}
plugins {
NodeAttestor "k8s_psat" {
plugin_data {
cluster = "sandbox-kata"
token_path = "/run/spire/tokens/token"
}
}
KeyManager "memory" {
plugin_data {}
}
WorkloadAttestor "unix" {
plugin_data {}
}
}
@@ -0,0 +1,41 @@
apiVersion: apps/v1
kind: Deployment
metadata:
name: opensandbox-identity
namespace: opensandbox
spec:
replicas: 1
selector:
matchLabels:
app.kubernetes.io/name: opensandbox-identity
template:
metadata:
labels:
app.kubernetes.io/name: opensandbox-identity
spec:
serviceAccountName: opensandbox-identity
containers:
- name: controller
image: zot.ad.ddupan.top/panxiao81/gitea-dynamic-runner-controller@sha256:dfbfaf2a7aa5951d1dc4e7f941a8cd05ceb4a860ac1ab23eb8be234861edcb55
imagePullPolicy: IfNotPresent
command: [/venv/bin/gitea-dynamic-runner-opensandbox-identity]
resources:
requests:
cpu: 10m
memory: 32Mi
limits:
cpu: 100m
memory: 96Mi
securityContext:
allowPrivilegeEscalation: false
capabilities:
drop: [ALL]
readOnlyRootFilesystem: true
runAsNonRoot: true
runAsUser: 65532
runAsGroup: 65532
seccompProfile:
type: RuntimeDefault
securityContext:
seccompProfile:
type: RuntimeDefault
@@ -0,0 +1,22 @@
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- service.yaml
- agent-config.yaml
- rbac.yaml
- identity-controller.yaml
- pools.yaml
patches:
# Pod jobs run directly in the homelab cluster. Keep OpenSandbox VM-only.
- target:
group: sandbox.opensandbox.io
version: v1alpha1
kind: Pool
name: ci-pod
patch: |-
apiVersion: sandbox.opensandbox.io/v1alpha1
kind: Pool
metadata:
name: ci-pod
namespace: opensandbox
$patch: delete
+293
View File
@@ -0,0 +1,293 @@
---
apiVersion: sandbox.opensandbox.io/v1alpha1
kind: Pool
metadata:
name: ci-vm
namespace: opensandbox
labels:
app.kubernetes.io/name: gitea-ci-vm
app.kubernetes.io/component: runner-pool
spec:
capacitySpec:
bufferMax: 0
bufferMin: 0
poolMax: 2
poolMin: 0
recycleStrategy:
type: Delete
template:
metadata:
labels:
app.kubernetes.io/name: gitea-ci-vm
ci.ddupan.top/backend: vm
spec:
runtimeClassName: kata-clh-runtime-rs
serviceAccountName: gitea-ci
restartPolicy: Never
terminationGracePeriodSeconds: 30
shareProcessNamespace: true
securityContext:
fsGroup: 2000
fsGroupChangePolicy: OnRootMismatch
initContainers:
- name: task-executor-installer
image: sandbox-registry.cn-zhangjiakou.cr.aliyuncs.com/opensandbox/task-executor:v0.1.0
command: [/bin/sh, -c]
args:
- cp /workspace/server /opt/opensandbox/task-executor && chmod 0755 /opt/opensandbox/task-executor
volumeMounts:
- name: opensandbox-bin
mountPath: /opt/opensandbox
- name: execd-installer
image: sandbox-registry.cn-zhangjiakou.cr.aliyuncs.com/opensandbox/execd:v1.0.22
command: [/bin/sh, -c]
args:
- cp ./execd /opt/opensandbox/execd && cp ./bootstrap.sh /opt/opensandbox/bootstrap.sh && chmod 0755 /opt/opensandbox/execd /opt/opensandbox/bootstrap.sh
volumeMounts:
- name: opensandbox-bin
mountPath: /opt/opensandbox
containers:
- name: sandbox
image: zot.ad.ddupan.top/panxiao81/gitea-dynamic-runner-runner@sha256:325d75a208b1a1c6f3b4d4705e47bbc58b34733edcd12f689c60769ab6772a59
imagePullPolicy: IfNotPresent
command: [/opt/opensandbox/task-executor]
args:
- -listen-addr=0.0.0.0:5758
- -log-dir=/tmp
- -data-dir=/tmp/tasks
env:
- name: SANDBOX_MAIN_CONTAINER
value: sandbox
- name: EXECD_ENVS
value: /opt/opensandbox/.env
- name: EXECD
value: /opt/opensandbox/execd
- name: GITEA_INSTANCE_URL
value: https://git.ddupan.top
- name: HOME
value: /data
ports:
- name: task-executor
containerPort: 5758
securityContext:
privileged: true
runAsNonRoot: true
runAsUser: 2000
runAsGroup: 2000
resources:
requests:
cpu: "2"
memory: 3Gi
limits:
cpu: "2"
memory: 3Gi
volumeMounts:
- name: opensandbox-bin
mountPath: /opt/opensandbox
- name: runner-data
mountPath: /data
- name: runner-workspace
mountPath: /workspace
- name: spire-socket
mountPath: /run/spire/agent-sockets
- name: spire-agent
image: ghcr.io/spiffe/spire-agent:1.15.3@sha256:41b0dcd8b258a69db9e2768292a060766fb76fd866e4bc925849981ea1b825ff
args: [-config, /run/spire/config/agent.conf]
securityContext:
allowPrivilegeEscalation: false
capabilities:
drop: [ALL]
volumeMounts:
- name: spire-config
mountPath: /run/spire/config
readOnly: true
- name: spire-bundle
mountPath: /run/spire/bundle
readOnly: true
- name: spire-token
mountPath: /run/spire/tokens
readOnly: true
- name: spire-data
mountPath: /run/spire/data
- name: spire-socket
mountPath: /run/spire/agent-sockets
volumes:
- name: opensandbox-bin
emptyDir: {}
- name: runner-data
emptyDir: {}
- name: runner-workspace
emptyDir: {}
- name: spire-data
emptyDir:
medium: Memory
- name: spire-socket
emptyDir:
medium: Memory
- name: spire-config
configMap:
name: gitea-ci-spire-agent
- name: spire-bundle
configMap:
name: spire-bundle-pem
- name: spire-token
projected:
sources:
- serviceAccountToken:
audience: spire-server
expirationSeconds: 3600
path: token
---
apiVersion: sandbox.opensandbox.io/v1alpha1
kind: Pool
metadata:
name: ci-pod
namespace: opensandbox
labels:
app.kubernetes.io/name: gitea-ci-pod
app.kubernetes.io/component: runner-pool
spec:
capacitySpec:
bufferMax: 0
bufferMin: 0
poolMax: 4
poolMin: 0
recycleStrategy:
type: Delete
template:
metadata:
labels:
app.kubernetes.io/name: gitea-ci-pod
ci.ddupan.top/backend: pod
spec:
serviceAccountName: gitea-ci
restartPolicy: Never
terminationGracePeriodSeconds: 30
shareProcessNamespace: true
securityContext:
fsGroup: 2000
fsGroupChangePolicy: OnRootMismatch
initContainers:
- name: task-executor-installer
image: sandbox-registry.cn-zhangjiakou.cr.aliyuncs.com/opensandbox/task-executor:v0.1.0
command: [/bin/sh, -c]
args:
- cp /workspace/server /opt/opensandbox/task-executor && chmod 0755 /opt/opensandbox/task-executor
volumeMounts:
- name: opensandbox-bin
mountPath: /opt/opensandbox
- name: execd-installer
image: sandbox-registry.cn-zhangjiakou.cr.aliyuncs.com/opensandbox/execd:v1.0.22
command: [/bin/sh, -c]
args:
- cp ./execd /opt/opensandbox/execd && cp ./bootstrap.sh /opt/opensandbox/bootstrap.sh && chmod 0755 /opt/opensandbox/execd /opt/opensandbox/bootstrap.sh
volumeMounts:
- name: opensandbox-bin
mountPath: /opt/opensandbox
containers:
- name: sandbox
image: zot.ad.ddupan.top/panxiao81/gitea-dynamic-runner-runner@sha256:325d75a208b1a1c6f3b4d4705e47bbc58b34733edcd12f689c60769ab6772a59
imagePullPolicy: IfNotPresent
command: [/opt/opensandbox/task-executor]
args:
- -listen-addr=0.0.0.0:5758
- -log-dir=/tmp
- -data-dir=/tmp/tasks
env:
- name: SANDBOX_MAIN_CONTAINER
value: sandbox
- name: EXECD_ENVS
value: /opt/opensandbox/.env
- name: EXECD
value: /opt/opensandbox/execd
- name: GITEA_INSTANCE_URL
value: https://git.ddupan.top
- name: HOME
value: /data
- name: DOCKER_HOST
value: unix:///run/docker/docker.sock
ports:
- name: task-executor
containerPort: 5758
securityContext:
allowPrivilegeEscalation: false
capabilities:
drop: [ALL]
runAsNonRoot: true
runAsUser: 2000
runAsGroup: 2000
volumeMounts:
- name: opensandbox-bin
mountPath: /opt/opensandbox
- name: runner-data
mountPath: /data
- name: runner-workspace
mountPath: /workspace
- name: docker-socket
mountPath: /run/docker
- name: spire-socket
mountPath: /run/spire/agent-sockets
- name: spire-agent
image: ghcr.io/spiffe/spire-agent:1.15.3@sha256:41b0dcd8b258a69db9e2768292a060766fb76fd866e4bc925849981ea1b825ff
args: [-config, /run/spire/config/agent.conf]
securityContext:
allowPrivilegeEscalation: false
capabilities:
drop: [ALL]
volumeMounts:
- name: spire-config
mountPath: /run/spire/config
readOnly: true
- name: spire-bundle
mountPath: /run/spire/bundle
readOnly: true
- name: spire-token
mountPath: /run/spire/tokens
readOnly: true
- name: spire-data
mountPath: /run/spire/data
- name: spire-socket
mountPath: /run/spire/agent-sockets
- name: docker
image: docker.io/library/docker:29.1.5-dind
command: [/bin/sh, -c]
args:
- test -e /dev/kmsg || mknod /dev/kmsg c 1 11; exec dockerd --host=unix:///run/docker/docker.sock --group=2000 --storage-driver=overlay2
securityContext:
privileged: true
volumeMounts:
- name: docker-socket
mountPath: /run/docker
- name: docker-data
mountPath: /var/lib/docker
volumes:
- name: opensandbox-bin
emptyDir: {}
- name: runner-data
emptyDir: {}
- name: runner-workspace
emptyDir: {}
- name: docker-data
emptyDir: {}
- name: docker-socket
emptyDir:
medium: Memory
- name: spire-data
emptyDir:
medium: Memory
- name: spire-socket
emptyDir:
medium: Memory
- name: spire-config
configMap:
name: gitea-ci-spire-agent
- name: spire-bundle
configMap:
name: spire-bundle-pem
- name: spire-token
projected:
sources:
- serviceAccountToken:
audience: spire-server
expirationSeconds: 3600
path: token
+87
View File
@@ -0,0 +1,87 @@
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: gitea-ci
namespace: opensandbox
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: opensandbox-identity
namespace: opensandbox
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
name: opensandbox-identity
namespace: opensandbox
rules:
- apiGroups: [""]
resources: [pods]
verbs: [get, list, watch]
- apiGroups: [sandbox.opensandbox.io]
resources: [batchsandboxes]
verbs: [get, list, watch]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: opensandbox-identity
namespace: opensandbox
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: Role
name: opensandbox-identity
subjects:
- kind: ServiceAccount
name: opensandbox-identity
namespace: opensandbox
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: opensandbox-identity
rules:
- apiGroups: [spire.spiffe.io]
resources: [clusterstaticentries]
verbs: [create, delete, get, list, watch]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: opensandbox-identity
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: opensandbox-identity
subjects:
- kind: ServiceAccount
name: opensandbox-identity
namespace: opensandbox
---
# The central SPIRE external controller uses this existing sandbox credential
# to publish a PEM bundle for guest-local Agents. It cannot read Secrets here.
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
name: spire-runner-bundle-publisher
namespace: opensandbox
rules:
- apiGroups: [""]
resources: [configmaps]
verbs: [create, delete, get, list, patch, update, watch]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: spire-runner-bundle-publisher
namespace: opensandbox
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: Role
name: spire-runner-bundle-publisher
subjects:
- kind: ServiceAccount
name: spire-controller-manager
namespace: spire-system
+15
View File
@@ -0,0 +1,15 @@
apiVersion: v1
kind: Service
metadata:
name: opensandbox-server-internal
namespace: opensandbox-system
spec:
type: NodePort
selector:
app.kubernetes.io/instance: opensandbox
app.kubernetes.io/name: opensandbox-server
ports:
- name: http
port: 80
targetPort: http
nodePort: 30080
@@ -0,0 +1,56 @@
# Sandbox External Secrets Operator
本目录在 sandbox 集群部署独立的 External Secrets Operator `2.8.0`,并通过
`ClusterSecretStore/openbao` 读取 OpenBao KV v2 中共享的 OpenSandbox API key。它不复用
homelab 集群的 ESO Pod、ServiceAccount 或 Kubernetes auth backend。
## 当前状态
- OpenBao `auth/kubernetes-sandbox`、backend config、`external-secrets` role 与
`sandbox-external-secrets` policy 已于 2026-09-18 由 Terraform 创建;apply 后 plan
为 zero-diff;
- `kv/k8s/opensandbox-api` 已由本机 `spiffe://ddupan.top/dev/panxiao81` 身份生成并写入,
值未输出或落盘;
- sandbox ESO operator、`ClusterSecretStore/openbao` 与 OpenSandbox `ExternalSecret`
由 Flux 管理;
- 线上 `ClusterSecretStore/openbao` 为 `Valid/Ready`,`ExternalSecret/opensandbox-api-key`
为 `SecretSynced/Ready`;
- OpenSandbox 已切换到 API key:无 key 请求返回 `401`,正确 key 请求返回 `200`;
- homelab runner 对同一 key 的投影不在本目录,留给 runner 项目管理。
OpenBao 的 `auth/kubernetes-sandbox`、对应 role、policy、sandbox API 地址与公开 CA
完全由 `infrastructure/openbao/terraform/` 管理。CA 文件提交到 Git 是刻意设计:CA
是公开信任材料,版本化后集群重建造成的 trust root 变化会产生可审计的 Terraform diff。
ESO 使用 TokenRequest 生成短期 ServiceAccount JWT。OpenBao 未配置长期
`token_reviewer_jwt`,而是使用登录 JWT 调用 sandbox TokenReview;因此
`external-secrets` ServiceAccount 仅额外绑定内建 `system:auth-delegator`。
## 重建顺序
1. 在 OpenBao 写入 OpenSandbox API key:
```bash
openssl rand -hex 32 | bao kv put kv/k8s/opensandbox-api api_key=-
```
2. 在 `infrastructure/openbao/terraform` 执行 `terraform plan` 和 `terraform apply`,
创建 `kubernetes-sandbox` auth mount、backend config、role 与只允许读取
`kv/k8s/opensandbox-api` 的最小权限 policy。
3. 合并 Flux 变更。依次等待 `flux-system/external-secrets-operator`、
`flux-system/external-secrets` Ready,再等待 `flux-system/opensandbox` 滚动完成。
operator 与配置拆成两个 Flux Kustomization,确保全新集群先安装 CRD,再声明
`ClusterSecretStore`;不要为了减少目录而把两层重新合并。
## 验收
```bash
kubectl get clustersecretstore openbao
kubectl -n opensandbox-system get externalsecret opensandbox-api-key
kubectl -n opensandbox-system get secret opensandbox-api-key
```
只检查 Secret 是否存在及 key 名,不输出 `data`。`ClusterSecretStore` 或
`ExternalSecret` 不 Ready 时,先检查 `auth/kubernetes-sandbox`,不要临时创建静态
Bao token Secret。
@@ -0,0 +1,13 @@
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: sandbox-external-secrets-token-review
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: system:auth-delegator
subjects:
- kind: ServiceAccount
name: external-secrets
namespace: external-secrets
@@ -0,0 +1,18 @@
---
apiVersion: external-secrets.io/v1
kind: ClusterSecretStore
metadata:
name: openbao
spec:
provider:
vault:
server: https://bao.ad.ddupan.top:8200
path: kv
version: v2
auth:
kubernetes:
mountPath: kubernetes-sandbox
role: external-secrets
serviceAccountRef:
name: external-secrets
namespace: external-secrets
@@ -0,0 +1,6 @@
---
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- auth-delegator.yaml
- clustersecretstore.yaml
@@ -1,31 +1,35 @@
---
apiVersion: helm.toolkit.fluxcd.io/v2 apiVersion: helm.toolkit.fluxcd.io/v2
kind: HelmRelease kind: HelmRelease
metadata: metadata:
name: gitea-actions name: external-secrets
namespace: gitea-actions namespace: external-secrets
spec: spec:
chart: chart:
spec: spec:
chart: actions chart: external-secrets
interval: 1h interval: 1h
sourceRef: sourceRef:
kind: HelmRepository kind: HelmRepository
name: gitea-charts name: external-secrets
version: 0.1.1 version: 2.8.0
driftDetection: driftDetection:
mode: enabled mode: enabled
install: install:
crds: CreateReplace
strategy: strategy:
name: RetryOnFailure name: RetryOnFailure
retryInterval: 5m retryInterval: 5m
interval: 30m interval: 30m
releaseName: gitea-actions releaseName: external-secrets
targetNamespace: gitea-actions targetNamespace: external-secrets
timeout: 10m timeout: 10m
upgrade: upgrade:
crds: CreateReplace
strategy: strategy:
name: RetryOnFailure name: RetryOnFailure
retryInterval: 5m retryInterval: 5m
valuesFrom: valuesFrom:
- kind: ConfigMap - kind: ConfigMap
name: gitea-actions-values name: external-secrets-values
valuesKey: values.yaml
@@ -0,0 +1,8 @@
---
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- namespace.yaml
- repository.yaml
- values.yaml
- helmrelease.yaml
@@ -1,4 +1,5 @@
---
apiVersion: v1 apiVersion: v1
kind: Namespace kind: Namespace
metadata: metadata:
name: gitea-actions name: external-secrets
@@ -0,0 +1,9 @@
---
apiVersion: source.toolkit.fluxcd.io/v1
kind: HelmRepository
metadata:
name: external-secrets
namespace: external-secrets
spec:
interval: 1h
url: https://charts.external-secrets.io
@@ -0,0 +1,31 @@
---
apiVersion: v1
kind: ConfigMap
metadata:
name: external-secrets-values
namespace: external-secrets
data:
values.yaml: |
replicaCount: 1
webhook:
replicaCount: 1
resources:
requests: {cpu: 10m, memory: 32Mi}
limits: {memory: 128Mi}
certController:
replicaCount: 1
resources:
requests: {cpu: 10m, memory: 32Mi}
limits: {memory: 128Mi}
resources:
requests: {cpu: 10m, memory: 64Mi}
limits: {memory: 256Mi}
serviceAccount:
create: true
name: external-secrets
installCRDs: true
+28 -2
View File
@@ -28,6 +28,32 @@ Pod 删除后的 backing-file/VMM 回收和真实构建基准必须作为上线
由 `VMPodScrape` 写入中央 VictoriaMetrics。它不拥有 Kubernetes API 凭据或 host 写 由 `VMPodScrape` 写入中央 VictoriaMetrics。它不拥有 Kubernetes API 凭据或 host 写
权限。 权限。
## Guest 内 SPIFFE 身份
Kata guest 不能直接使用 node SPIRE Agent 的 CSI socket。Unix socket 的路径即使通过
virtio-fs 出现在 guest 中,连接也不能跨 VM 边界。CI Pod 应以 native sidecar 在同一
guest 内启动临时 SPIRE Agent,并满足以下约束:
1. 外层 Pod 挂载 `audience=spire-server` 的 Pod-bound projected ServiceAccount token;
2. 内层 Agent 使用 `k8s_psat` 向中央 Server attestation,Server cluster profile 必须
启用 `use_pod_uid_for_agent_id`,使 Agent ID 包含 Pod UID;
3. 调度器为该具体 Agent 创建 registration entry;SPIFFE ID 使用仓库和任务名等稳定的
业务语义,Pod UID 只作为 parent binding,不进入业务身份;
4. Agent 与 workload 通过 guest 内 `emptyDir` 上的 Unix socket 通信;Pod 必须设置
`shareProcessNamespace: true`,否则 `unix` workload attestor 无法从共享 `/proc`
解析客户端的 `SO_PEERCRED` PID;
5. 调度器必须在 registration entry 已同步到 Agent 后才放行 workload。Pod 删除后同步
删除 entry,并 evict 对应的临时 Agent 记录。
2026-09-17 的 live PoC 已验证:以 Pod UID 作为 parent 的临时 Agent 成功注册,
`unix:uid:2000` workload 从 guest-local socket 获得
`spiffe://ddupan.top/ci/poc/task/build` X.509-SVID,Pod 正常退出。PoC 资源随后全部清理,
中央 SPIRE 恢复声明式配置。
SPIRE 1.15.3 已支持 `use_pod_uid_for_agent_id`,但当前使用的 hardened chart 0.30.2
尚未把它暴露到 values/template。正式部署应先给 chart 补齐该字段并向上游提交,随后
采用包含修复的 release;不得把手工修改 Server ConfigMap 作为运行方案。
## 上线验收 ## 上线验收
Flux reconciliation 完成后至少确认: Flux reconciliation 完成后至少确认:
@@ -35,8 +61,8 @@ Flux reconciliation 完成后至少确认:
1. 两个节点重新回到 Ready,`RuntimeClass/kata-clh-runtime-rs` 存在; 1. 两个节点重新回到 Ready,`RuntimeClass/kata-clh-runtime-rs` 存在;
2. Kata Pod 内核与 LXC host 内核不同,且 `/dev/kvm` 可用; 2. Kata Pod 内核与 LXC host 内核不同,且 `/dev/kvm` 可用;
3. Cloud Hypervisor API 的 `vm.info.config.memory.shared` 为 `true`; 3. Cloud Hypervisor API 的 `vm.info.config.memory.shared` 为 `true`;
4. `spire-smoke` ServiceAccount 的 Kata Pod 可获得 4. Kata Pod 内层 Agent 的 ID 包含该 Pod UID;只有调度器创建的业务 entry 所匹配的
`spiffe://ddupan.top/sandbox/smoke`,错误 ServiceAccount 无法获得身份; workload UID 能从 guest-local socket 获得预期 SPIFFE ID;
5. block-backed `emptyDir` 上 Docker 使用 `overlay2`,BuildKit 与 kind smoke test 5. block-backed `emptyDir` 上 Docker 使用 `overlay2`,BuildKit 与 kind smoke test
通过;kind 的 dockerd bootstrap 需要先在 guest 内执行 通过;kind 的 dockerd bootstrap 需要先在 guest 内执行
`mknod /dev/kmsg c 1 11`; `mknod /dev/kmsg c 1 11`;
+44
View File
@@ -0,0 +1,44 @@
# Sandbox OpenSandbox
本目录在独立 sandbox k3s 集群部署 OpenSandbox controller、server 与 CRD。Flux 从
上游 commit `8f01e935c2cabba778cf37a152033fae062fa0f4` 构建官方 umbrella chart
`0.2.2`;该源码渲染结果已与 release `opensandbox-0.2.2.tgz` 对比一致。不要改为跟随
浮动 branch 或 tag。
server 只提供集群内 `opensandbox-server.opensandbox-system.svc:80` ClusterIP,不部署
Gateway、Ingress 或 LoadBalancer。sandbox workload 位于 `opensandbox` namespace,
默认使用 `kata-clh-runtime-rs`;CI Pool、runner 镜像、动态 SPIFFE registration 均由
runner 项目后续声明,本目录不预制。
## API 认证
`ExternalSecret/opensandbox-api-key` 从 OpenBao
`kv/k8s/opensandbox-api:api_key` 投影同名 Secret。这个路径不归属于某个 Kubernetes
集群:sandbox server 与 homelab runner 调度器分别通过自己的 Bao 身份读取。server 只通过
`secretKeyRef` 读取:
```yaml
- name: OPENSANDBOX_SERVER_API_KEY
valueFrom:
secretKeyRef:
name: opensandbox-api-key
key: api-key
```
仓库与 Helm values 均不保存 API key。OpenSandbox 不支持更丰富的原生 workload
authentication;runner 后续读取同一 Bao 路径并在请求头中使用 API key。
`opensandbox-values` 带 Flux watch label,values 变化会立即触发 Helm reconcile,不依赖
30 分钟的 HelmRelease interval。
## 验收
合并后等待 `flux-system/opensandbox` 与 `opensandbox-system/opensandbox` Ready,并确认:
```bash
kubectl get crd batchsandboxes.sandbox.opensandbox.io pools.sandbox.opensandbox.io
kubectl -n opensandbox-system get deploy,pod,svc
kubectl get runtimeclass kata-clh-runtime-rs
```
控制面上线不创建 CI Pool,也不产生 sandbox workload。首个 runner 集成应另行提交 Pool
与完整的 Lifecycle API smoke test。
@@ -1,18 +1,19 @@
---
apiVersion: external-secrets.io/v1 apiVersion: external-secrets.io/v1
kind: ExternalSecret kind: ExternalSecret
metadata: metadata:
name: gitea-runner-token name: opensandbox-api-key
namespace: gitea-actions namespace: opensandbox-system
spec: spec:
refreshInterval: 1h refreshInterval: 1h
secretStoreRef: secretStoreRef:
kind: ClusterSecretStore kind: ClusterSecretStore
name: openbao name: openbao
target: target:
name: opensandbox-api-key
creationPolicy: Owner creationPolicy: Owner
name: gitea-runner-token
data: data:
- secretKey: token - secretKey: api-key
remoteRef: remoteRef:
key: k8s/gitea-runner key: k8s/opensandbox-api
property: token property: api_key
@@ -0,0 +1,35 @@
---
apiVersion: helm.toolkit.fluxcd.io/v2
kind: HelmRelease
metadata:
name: opensandbox
namespace: opensandbox-system
spec:
chart:
spec:
chart: ./kubernetes/charts/opensandbox
interval: 1h
reconcileStrategy: Revision
sourceRef:
kind: GitRepository
name: opensandbox
driftDetection:
mode: enabled
install:
crds: CreateReplace
strategy:
name: RetryOnFailure
retryInterval: 5m
interval: 30m
releaseName: opensandbox
targetNamespace: opensandbox-system
timeout: 15m
upgrade:
crds: CreateReplace
strategy:
name: RetryOnFailure
retryInterval: 5m
valuesFrom:
- kind: ConfigMap
name: opensandbox-values
valuesKey: values.yaml
@@ -0,0 +1,11 @@
---
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- namespace.yaml
- external-secret.yaml
- repository.yaml
- template.yaml
- values.yaml
- helmrelease.yaml
- monitor-scrape.yaml
@@ -0,0 +1,16 @@
---
apiVersion: operator.victoriametrics.com/v1beta1
kind: VMPodScrape
metadata:
name: opensandbox-controller
namespace: monitoring
spec:
namespaceSelector:
matchNames:
- opensandbox-system
podMetricsEndpoints:
- interval: 30s
port: metrics
selector:
matchLabels:
app.kubernetes.io/name: opensandbox
@@ -0,0 +1,14 @@
---
apiVersion: v1
kind: Namespace
metadata:
name: opensandbox-system
---
apiVersion: v1
kind: Namespace
metadata:
name: opensandbox
labels:
pod-security.kubernetes.io/enforce: privileged
pod-security.kubernetes.io/audit: restricted
pod-security.kubernetes.io/warn: restricted
@@ -0,0 +1,11 @@
---
apiVersion: source.toolkit.fluxcd.io/v1
kind: GitRepository
metadata:
name: opensandbox
namespace: opensandbox-system
spec:
interval: 1h
ref:
commit: 8f01e935c2cabba778cf37a152033fae062fa0f4
url: https://github.com/opensandbox-group/OpenSandbox.git
@@ -0,0 +1,17 @@
---
apiVersion: v1
kind: ConfigMap
metadata:
name: opensandbox-batchsandbox-template
namespace: opensandbox-system
data:
batchsandbox-template.yaml: |
metadata:
labels:
ddupan.top/workload-class: sandbox
spec:
replicas: 1
template:
spec:
restartPolicy: Never
terminationGracePeriodSeconds: 30
+83
View File
@@ -0,0 +1,83 @@
---
apiVersion: v1
kind: ConfigMap
metadata:
name: opensandbox-values
namespace: opensandbox-system
labels:
reconcile.fluxcd.io/watch: Enabled
data:
values.yaml: |
opensandbox-controller:
controller:
metrics:
enabled: true
port: 8080
secure: false
opensandbox-server:
server:
replicaCount: 1
env:
- name: OPENSANDBOX_SERVER_API_KEY
valueFrom:
secretKeyRef:
name: opensandbox-api-key
key: api-key
resources:
limits:
cpu: "1"
memory: 1Gi
requests:
cpu: 100m
memory: 256Mi
volumeMounts:
- name: batchsandbox-template
mountPath: /etc/opensandbox/batchsandbox-template.yaml
subPath: batchsandbox-template.yaml
readOnly: true
volumes:
- name: batchsandbox-template
configMap:
name: opensandbox-batchsandbox-template
configToml: |
[server]
host = "0.0.0.0"
port = 80
api_key = ""
max_sandbox_timeout_seconds = 86400
[log]
level = "INFO"
[runtime]
type = "kubernetes"
execd_image = "sandbox-registry.cn-zhangjiakou.cr.aliyuncs.com/opensandbox/execd:v1.0.22"
[storage]
allowed_host_paths = []
volume_default_size = "1Gi"
[kubernetes]
kubeconfig_path = ""
namespace = "opensandbox"
informer_enabled = true
informer_resync_seconds = 300
informer_watch_timeout_seconds = 60
snapshot_create_timeout_seconds = 900
workload_provider = "batchsandbox"
image_pull_policy = "IfNotPresent"
batchsandbox_template_file = "/etc/opensandbox/batchsandbox-template.yaml"
[egress]
image = "sandbox-registry.cn-zhangjiakou.cr.aliyuncs.com/opensandbox/egress:v1.1.6"
mode = "dns+nft"
disable_ipv6 = true
[secure_runtime]
type = "kata"
k8s_runtime_class = "kata-clh-runtime-rs"
opensandbox-node-agent:
enabled: false
+6
View File
@@ -74,6 +74,12 @@ Agent 已启用 Unix workload attestor,并为本机用户 `panxiao81`(UID `1
export SPIFFE_ENDPOINT_SOCKET=unix:///run/spire/agent-sockets/spire-agent.sock export SPIFFE_ENDPOINT_SOCKET=unix:///run/spire/agent-sockets/spire-agent.sock
``` ```
宿主机已安装与 Agent Pod 同版本的 `/usr/local/bin/spire-agent` 1.15.3,供本地进程从
Workload API 获取 JWT-SVID。二进制来自 SPIRE 官方 `linux-amd64-musl` release,安装时
核对 tarball SHA-256
`ca1a4d1155317bdd2afc7f36663828a10410c7c840e54725b90b4064b0a301c7`。升级 chart 时应
同步升级这个 CLI 并重新核对官方 checksum,不能长期混用版本。
该身份仅按 Unix UID 匹配,不是 SPIRE admin,也不会匹配 `sudo` 后以 root 运行的 该身份仅按 Unix UID 匹配,不是 SPIRE admin,也不会匹配 `sudo` 后以 root 运行的
进程。`ClusterStaticEntry.spec.parentID` 绑定当前 `laptop` Kubernetes node UID;若 进程。`ClusterStaticEntry.spec.parentID` 绑定当前 `laptop` Kubernetes node UID;若
节点被删除后重建,需从 `spire-server agent list` 取得新 Agent ID 并同步更新该字段。 节点被删除后重建,需从 `spire-server agent list` 取得新 Agent ID 并同步更新该字段。
+12 -2
View File
@@ -274,14 +274,24 @@ CI job/Agent 不应接收长期 `BAO_TOKEN`。标准启动顺序是:
5. 在同一进程树中以环境变量调用 `tofu`、Ansible 或其他工具; 5. 在同一进程树中以环境变量调用 `tofu`、Ansible 或其他工具;
6. cleanup 尝试 `revoke-self`,随后销毁 job/VM/容器。 6. cleanup 尝试 `revoke-self`,随后销毁 job/VM/容器。
通用 credential-exec 包装器未来应负责步骤 3–6。它必须满足: Gitea Actions 可使用
[`panxiao81/ci-actions/spiffe-openbao-login@v1`](https://git.ddupan.top/panxiao81/ci-actions/src/tag/v1/spiffe-openbao-login)
完成步骤 3、4 和 cleanup 时的 `revoke-self`。runner 只负责提供 Node.js 20、
`spire-agent` 与 Workload API socket;role、audience 及是否登录由 workflow 明确声明,
目标侧 Bao policy 仍负责最终授权。该 Action 必须满足:
- 不把 JWT-SVID 或 Bao token 写到 stdout/stderr; - 不把 JWT-SVID 或 Bao token 写到 stdout/stderr;
- 不把凭据传入命令行参数,避免出现在进程列表; - 不把凭据传入命令行参数,避免出现在进程列表;
- 子进程退出后清除环境和临时文件; - job 结束后销毁 runner 及其 Actions 临时文件;
- 不尝试把短期 token 上传到 Actions Secret 或 artifact; - 不尝试把短期 token 上传到 Actions Secret 或 artifact;
- role、audience 和目标命令由受审查的 pipeline 配置决定。 - role、audience 和目标命令由受审查的 pipeline 配置决定。
Action 会按 GitHub Actions 协议把短期 token 写入 `GITHUB_ENV` 和 `GITHUB_STATE`,因此
只允许用于一次性 Pod/VM runner,不能用于共享或持久 runner。只读取 Nexus public
repository 时不需要 Bao 登录,可直接使用
[`panxiao81/ci-actions/setup-nexus@v1`](https://git.ddupan.top/panxiao81/ci-actions/src/tag/v1/setup-nexus)
配置 Ansible Galaxy、Go module proxy 与 OCI endpoint。
Kubernetes 以外的执行环境不能伪造 ServiceAccount。未来应分别使用 host SPIRE Kubernetes 以外的执行环境不能伪造 ServiceAccount。未来应分别使用 host SPIRE
Agent、TPM/DevID、cloud instance identity、GitHub OIDC 等初始证明接入同一信任模型。 Agent、TPM/DevID、cloud instance identity、GitHub OIDC 等初始证明接入同一信任模型。
+17
View File
@@ -43,6 +43,10 @@ spire-server:
externalSecret: externalSecret:
name: spire-external-kubeconfigs name: spire-external-kubeconfigs
key: sandbox-controller key: sandbox-controller
sandbox-runner-bundle:
externalSecret:
name: spire-external-kubeconfigs
key: sandbox-controller
nodeAttestor: nodeAttestor:
externalK8sPSAT: externalK8sPSAT:
enabled: true enabled: true
@@ -55,12 +59,19 @@ spire-server:
kubeConfigName: sandbox kubeConfigName: sandbox
serviceAccountAllowList: serviceAccountAllowList:
- spire-smoke:spire-smoke - spire-smoke:spire-smoke
- opensandbox:gitea-ci
usePodUIDForAgentID: true usePodUIDForAgentID: true
externalControllerManagers: externalControllerManagers:
enabled: true enabled: true
clusters: clusters:
sandbox: sandbox:
kubeConfigName: sandbox-controller kubeConfigName: sandbox-controller
# Dynamic OpenSandbox runner identities are exact Pod-UID-bound
# ClusterStaticEntries created in the sandbox cluster. The chart
# defaults this reconciler to false, so enable the central registration
# path explicitly.
reconcile:
clusterStaticEntries: true
bundlePublisher: bundlePublisher:
externalK8sConfigMap: externalK8sConfigMap:
enabled: true enabled: true
@@ -71,6 +82,12 @@ spire-server:
configMapName: spire-bundle configMapName: spire-bundle
configMapKey: bundle.spiffe configMapKey: bundle.spiffe
format: spiffe format: spiffe
sandbox-runner-bundle:
kubeConfigName: sandbox-runner-bundle
namespace: opensandbox
configMapName: spire-bundle-pem
configMapKey: bundle.pem
format: pem
persistence: persistence:
# PostgreSQL stores registrations, but the disk KeyManager still needs durable # PostgreSQL stores registrations, but the disk KeyManager still needs durable
# storage for the trust-domain signing keys. # storage for the trust-domain signing keys.