Author SHA1 Message Date
panxiao81 1ac99f4476 Merge pull request '部署 Runner 结构化生命周期日志' (#128) from deploy/structured-runner-logs into main
yaml / yaml (push) Successful in 23s
2026-09-21 09:46:47 +00:00
panxiao81 891c6b90e7 deploy: 更新 Runner 结构化日志镜像
yaml / yaml (pull_request) Successful in 21s
2026-09-21 09:45:33 +00:00
panxiao81 c2314256cc Merge pull request '为 sandbox executor 暴露内网 Runner facade' (#127) from fix/expose-runner-facade-internal into main
yaml / yaml (push) Successful in 26s
2026-09-21 09:18:16 +00:00
panxiao81 8244cdad79 fix: 为 sandbox 暴露 Runner facade
yaml / yaml (pull_request) Successful in 16s
2026-09-21 09:11:29 +00:00
panxiao81 23841556d3 Merge VM 集成测试入口隔离
yaml / yaml (push) Successful in 24s
2026-09-21 09:00:23 +00:00
panxiao81 4daaa38495 deploy: 隔离 VM 集成测试入口
yaml / yaml (pull_request) Successful in 12s
2026-09-21 08:58:21 +00:00
panxiao81 da0630cd29 Merge OpenSandbox Runner 标记修复部署
yaml / yaml (push) Successful in 46s
2026-09-21 08:35:24 +00:00
panxiao81 d0fcfdeddc deploy: 补回 OpenSandbox Runner 标记
yaml / yaml (pull_request) Successful in 23s
2026-09-21 08:34:37 +00:00
panxiao81 ecf395cc8f Merge OpenSandbox metadata 修复部署
yaml / yaml (push) Successful in 58s
2026-09-21 08:07:33 +00:00
panxiao81 9f646d51e7 deploy: 修复 OpenSandbox metadata 编码
yaml / yaml (pull_request) Successful in 56s
2026-09-21 08:04:15 +00:00
panxiao81 5c7e7ff8db Merge Runner 后端独立容量池部署
yaml / yaml (push) Successful in 34s
2026-09-21 07:35:32 +00:00
panxiao81 d6b97355d6 deploy: 启用 Runner 后端容量池
yaml / yaml (pull_request) Successful in 48s
2026-09-21 07:33:31 +00:00
panxiao81 b82ba4d5a0 Merge Pod Runner 四并发恢复
yaml / yaml (push) Successful in 22s
2026-09-21 06:57:37 +00:00
panxiao81 9dc7aabc94 恢复 Pod Runner 四并发
yaml / yaml (pull_request) Successful in 21s
2026-09-21 06:57:09 +00:00
panxiao81 b7c95fd0b8 Merge Runner 无中断滚动与容量隔离
yaml / yaml (push) Successful in 24s
2026-09-21 06:51:39 +00:00
panxiao81 098ff4e27f 更新 Runner 容量隔离镜像
yaml / yaml (pull_request) Successful in 27s
2026-09-21 06:51:37 +00:00
panxiao81 dfc55e1d9e 启用 Runner 无中断滚动更新
yaml / yaml (pull_request) Successful in 29s
2026-09-21 06:39:11 +00:00
panxiao81 18d9f83413 Merge Runner claim 恢复镜像
yaml / yaml (push) Successful in 34s
2026-09-21 06:24:31 +00:00
panxiao81 e5e2ab1b89 更新 Runner claim 恢复镜像
yaml / yaml (pull_request) Successful in 15s
2026-09-21 06:21:20 +00:00
panxiao81 0b236c43d9 Merge OpenSandbox VM Runner canary
yaml / yaml (push) Successful in 14s
2026-09-21 06:12:35 +00:00
panxiao81 762ce6458d 启用 OpenSandbox VM Runner canary
yaml / yaml (pull_request) Successful in 17s
2026-09-21 06:10:59 +00:00
panxiao81 15ef512cfc Merge runner Gitea proxy Host fix
yaml / yaml (push) Failing after 31s
更新 Runner Gitea 反代 Host 修复镜像
2026-09-21 05:42:33 +00:00
panxiao81 a13a5118cf 更新 Runner Gitea 反代 Host 修复镜像
yaml / yaml (pull_request) Failing after 33s
2026-09-21 05:41:50 +00:00
panxiao81 25bc7ee7d6 Merge runner cleanup and checkout proxy fix
yaml / yaml (push) Failing after 41s
更新 Runner 自动清理与 checkout 修复镜像
2026-09-21 05:31:26 +00:00
panxiao81 3d41e0e29b 更新 Runner 自动清理与 checkout 修复镜像
yaml / yaml (pull_request) Failing after 37s
2026-09-21 05:30:09 +00:00
panxiao81 3169fcabd3 Merge pull request '记录 CI 身份 Action 使用入口' (#116) from docs/ci-actions into main 2026-09-21 03:31:03 +00:00
panxiao81 0a42a3ad04 记录 CI 身份 Action 使用入口 2026-09-21 03:30:31 +00:00
panxiao81 9ec034c947 Merge runner facade readiness fix
yaml / yaml (push) Failing after 5m30s
更新 Runner facade 就绪等待镜像
2026-09-20 21:37:42 +00:00
panxiao81 ad9b305aba 更新 Runner facade 就绪等待镜像
yaml / yaml (pull_request) Failing after 5m26s
2026-09-20 21:37:19 +00:00
panxiao81 9696df1f51 Merge pull request #114
yaml / yaml (push) Failing after 10m47s
修正动态 Runner 的 SPIRE agent 绑定
2026-09-20 21:31:11 +00:00
panxiao81 a335bdbc38 修正动态 Runner 的 SPIRE agent 绑定
yaml / yaml (pull_request) Failing after 11m40s
2026-09-20 21:30:48 +00:00
panxiao81 b6623069ad Merge pull request #113
yaml / yaml (push) Failing after 11m36s
更新动态 Runner SPIFFE socket 修复镜像
2026-09-20 21:15:15 +00:00
panxiao81 4e3ef3c33f 更新动态 Runner SPIFFE socket 修复镜像
yaml / yaml (pull_request) Failing after 9m48s
2026-09-20 21:14:08 +00:00
panxiao81 8eac82c272 合并无重叠 scheduler 滚动策略
yaml / yaml (push) Failing after 1m1s
修复 Flux SSA strategy 冲突。
2026-09-20 21:05:02 +00:00
panxiao81 2b232c4cd2 使用无重叠滚动策略部署 scheduler
yaml / yaml (pull_request) Successful in 14s
2026-09-20 21:04:45 +00:00
panxiao81 b51dc3751b 合并 Recreate 策略修复
yaml / yaml (push) Successful in 15s
修复 Flux dry-run 阻塞,生产资源此前尚未切换。
2026-09-20 21:03:06 +00:00
panxiao81 616c87bafd 清理 Recreate 策略遗留字段
yaml / yaml (pull_request) Successful in 14s
2026-09-20 21:02:49 +00:00
panxiao81 0663642583 合并 Go 动态 Runner Pod canary
yaml / yaml (push) Successful in 20s
已完成 yamllint、kustomize 与生产 API server-side dry-run;按单副本 single-flight canary 部署。
2026-09-20 21:01:36 +00:00
panxiao81 68191e0ff4 部署 Go 动态 Runner Pod canary
yaml / yaml (pull_request) Successful in 18s
2026-09-20 21:00:28 +00:00
panxiao81 c368373e74 Merge pull request '纳管并验收 Nexus OCI 仓库' (#109) from feat/nexus-oci-poc into main
terraform / validate (push) Successful in 54s
2026-09-20 20:47:03 +00:00
panxiao81 7bdd152db9 纳管并验收 Nexus OCI 仓库
terraform / validate (pull_request) Successful in 1m21s
2026-09-20 20:46:26 +00:00
panxiao81 ab985c7979 Merge pull request '完成 Nexus Ansible 与 Go 缓存验收' (#108) from feat/nexus-poc-verification into main
terraform / validate (push) Successful in 1m48s
2026-09-20 20:32:45 +00:00
panxiao81 cb137f2044 完成 Nexus Ansible 与 Go 缓存验收
terraform / validate (pull_request) Successful in 49s
2026-09-20 20:28:23 +00:00
panxiao81 83e4ac2f01 Merge pull request '新增 Nexus 统一制品仓库 POC' (#103) from feat/nexus-poc into main
ansible / collection-test (push) Successful in 4m14s
ansible / lint (push) Successful in 5m15s
yaml / yaml (push) Successful in 35s
terraform / validate (push) Successful in 1m8s
Reviewed-on: #103
2026-09-20 19:49:49 +00:00
panxiao81 1df682bc6d Merge pull request '限制 OpenSandbox 仅调度 VM' (#107) from fix/enable-native-pod-worker into main
yaml / yaml (push) Successful in 20s
2026-09-20 17:19:36 +00:00
panxiao81 eb52197eee 限制 OpenSandbox 仅调度 VM
yaml / yaml (pull_request) Successful in 19s
2026-09-20 17:19:12 +00:00
panxiao81 9b9f44333e Merge pull request '在 homelab 启用原生 Pod Worker' (#106) from fix/enable-native-pod-worker into main
yaml / yaml (push) Successful in 24s
2026-09-20 17:16:28 +00:00
panxiao81 8896d58b89 在 homelab 启用原生 Pod Worker
yaml / yaml (pull_request) Successful in 24s
2026-09-20 17:16:26 +00:00
panxiao81 0fd7f87d7e Merge pull request '停止 OpenSandbox 消费 Pod 任务' (#105) from fix/separate-pod-opensandbox into main
yaml / yaml (push) Failing after 0s
2026-09-20 17:13:43 +00:00
panxiao81 89bd7864e6 停止 OpenSandbox 消费 Pod 任务
yaml / yaml (pull_request) Failing after 0s
2026-09-20 17:13:21 +00:00
panxiao81 93f803a0d9 合并 Runner completed ACK 镜像修复
yaml / yaml (push) Failing after 0s
2026-09-18 19:46:33 +00:00
panxiao81 f7d9579221 部署 Runner completed ACK 修复
yaml / yaml (pull_request) Failing after 0s
2026-09-18 19:46:30 +00:00
panxiao81 9fe910a4c9 合并 OpenSandbox Runner 部署修复
ansible / lint (push) Failing after 0s
ansible / collection-test (push) Failing after 0s
yaml / yaml (push) Failing after 0s
2026-09-18 19:34:25 +00:00
panxiao81 28c20fd3e5 恢复持久事件驱动 Runner 调度
ansible / lint (pull_request) Failing after 0s
ansible / collection-test (pull_request) Failing after 0s
yaml / yaml (pull_request) Failing after 0s
2026-09-18 19:27:25 +00:00
panxiao81 110dbcc2c3 新增 Nexus 统一制品仓库 POC
ansible / lint (pull_request) Failing after 0s
ansible / collection-test (pull_request) Failing after 0s
yaml / yaml (pull_request) Failing after 0s
terraform / validate (pull_request) Failing after 0s
2026-09-18 19:11:30 +00:00
panxiao81 28f1de35cb 启用 sandbox 静态身份登记
ansible / lint (pull_request) Failing after 2s
ansible / collection-test (pull_request) Failing after 1s
yaml / yaml (pull_request) Failing after 1s
2026-09-18 19:06:13 +00:00
panxiao81 ad4ed1f0d6 重新触发 OpenSandbox 集成验证
ansible / lint (pull_request) Failing after 0s
ansible / collection-test (pull_request) Failing after 0s
yaml / yaml (pull_request) Failing after 2s
2026-09-18 19:02:44 +00:00
panxiao81 611a5a3d3a 修复 Pool task-executor 数据目录与 guest 身份镜像
ansible / lint (pull_request) Failing after 0s
ansible / collection-test (pull_request) Failing after 0s
yaml / yaml (pull_request) Failing after 0s
2026-09-18 18:59:02 +00:00
panxiao81 7582990655 修复 OpenSandbox 冷启动超时与旧 worker 清理
ansible / lint (pull_request) Failing after 0s
ansible / collection-test (pull_request) Failing after 0s
yaml / yaml (pull_request) Failing after 0s
2026-09-18 18:53:26 +00:00
panxiao81 ec3ce94e12 Merge pull request '接入 OpenSandbox 动态 Runner' (#101) from feat/opensandbox-runner-integration into main
yaml / yaml (push) Successful in 23s
ansible / collection-test (push) Successful in 1m35s
ansible / lint (push) Successful in 2m55s
2026-09-18 18:45:24 +00:00
panxiao81 6eb4875db6 接入 OpenSandbox 动态 Runner
ansible / collection-test (pull_request) Successful in 2m58s
ansible / lint (pull_request) Successful in 4m33s
yaml / yaml (pull_request) Successful in 29s
2026-09-18 18:33:00 +00:00
panxiao81 518dba6036 Merge pull request 确保 OpenSandbox values 立即生效
yaml / yaml (push) Successful in 17s
ansible / collection-test (push) Successful in 1m15s
ansible / lint (push) Successful in 2m7s
2026-09-18 17:52:51 +00:00
panxiao81 4c823f8181 确保 OpenSandbox values 立即生效
yaml / yaml (pull_request) Successful in 27s
ansible / collection-test (pull_request) Successful in 1m16s
ansible / lint (pull_request) Successful in 2m33s
2026-09-18 17:49:26 +00:00
panxiao81 8290082fb4 Merge pull request 为 sandbox 接入 OpenBao External Secrets
yaml / yaml (push) Successful in 15s
terraform / validate (push) Successful in 42s
2026-09-18 17:41:51 +00:00
panxiao81 159a75b710 记录 sandbox ESO 当前状态
yaml / yaml (pull_request) Successful in 16s
terraform / validate (pull_request) Successful in 39s
2026-09-18 17:40:24 +00:00
panxiao81 e0e629794b 为 sandbox 接入 OpenBao External Secrets
yaml / yaml (pull_request) Successful in 17s
terraform / validate (pull_request) Successful in 49s
2026-09-18 17:34:53 +00:00
panxiao81 90ba945d85 Merge pull request 部署 OpenSandbox 控制面
yaml / yaml (push) Successful in 18s
2026-09-18 16:38:13 +00:00
panxiao81 d526fd75d3 部署 OpenSandbox 控制面
yaml / yaml (pull_request) Successful in 17s
2026-09-18 16:36:21 +00:00
panxiao81 18cb2858b9 Merge pull request '延长 sandbox Flux 根同步超时' (#93) from fix/sandbox-flux-root-timeout into main
ansible / collection-test (push) Successful in 1m13s
ansible / lint (push) Successful in 2m23s
Reviewed-on: #93
2026-09-18 16:18:46 +00:00
panxiao81 99d1ec1d6f Merge pull request '记录 Kata guest 内 SPIFFE 身份方案' (#95) from docs/kata-inner-spire into main
Reviewed-on: #95
2026-09-18 16:18:29 +00:00
panxiao81 583dab526a docs: 明确 SPIRE chart 字段缺口 2026-09-17 18:54:00 +00:00
panxiao81 3dbd4c5f31 docs: 记录 Kata guest SPIFFE 身份方案 2026-09-17 18:47:01 +00:00
panxiao81 e67bce5121 fix: 延长 sandbox Flux 根同步超时
ansible / collection-test (pull_request) Successful in 1m8s
ansible / lint (pull_request) Successful in 2m56s
2026-09-17 18:20:33 +00:00
75 changed files with 2165 additions and 198 deletions
+157
View File
@@ -0,0 +1,157 @@
# Nexus Repository POC
本目录声明一个 Nexus Repository Community Edition POC,用来验证一次性 CI runner 通过
网络服务复用 Ansible Galaxy、Go Modules 与 OCI/BuildKit 缓存。Nexus 固定为 `3.96.1`,
镜像固定到官方 multi-arch index digest;LAN 入口为
`https://nexus.ad.ddupan.top`。
## POC 边界
- 单副本 Deployment,`Recreate` 更新,避免一个 RWO 卷被两个 Pod 同时挂载。
- `/nexus-data` 使用 `localpv-zfs-ceph` 上的 50 GiB RWO PVC。
- 资源预算为 250m/2 GiB request、2 CPU/4 GiB limit;JVM heap 上限 2 GiB。
- 使用容器默认的 embedded H2。它只用于 POC;正式接管 OCI 制品前必须迁移到外部
PostgreSQL,并验证备份恢复。
- 入口只在 LAN wildcard Gateway 上发布,不创建公网 DNS 或 Cloudflare route。
- 不套 Authelia forward-auth;它会破坏 Go、Ansible 与 OCI 非浏览器客户端。
- 现有 zot 保持不变。Nexus 完成 OCI、BuildKit cache 和恢复验收前不得迁移或删除 zot。
Terraform provider 创建 `ansible-public`、`go-public`、最小匿名权限与 OCI Bearer Token
Realm。Nexus 3.94 才加入的原生 OCI repository 已有 REST API,但当前锁定的 community
provider 尚未暴露 OCI resource;`terraform/reconcile-oci.sh` 因此根据 3.96.1 实例 Swagger
固定的 JSON schema,幂等调和 `oci-hosted`、`oci-proxy` 与 `oci-public`。不得绕过该入口在
UI 中创建无人管理的长期 repository。
## 部署
Flux 从 `clusters/homelab/apps/nexus.yaml` 协调本目录,并依赖 Envoy Gateway 与 OpenEBS。
合并前只渲染配置,不直接 apply:
```bash
kubectl kustomize apps/nexus
```
合并并由 Flux 部署后检查:
```bash
kubectl -n flux-system get kustomization nexus
kubectl -n nexus get pod,pvc,service,httproute
kubectl -n nexus logs deployment/nexus --tail=100
```
启动可能需要数分钟,startup probe 允许最多十分钟。不要因初次启动较慢反复删除 Pod;
先确认 PVC 已 Bound、Pod 没有 OOM,以及日志仍在推进。
DNS 期望状态已加入 `infrastructure/dns/records.yml`,需从 Samba AD Ansible root 以
`--check --diff` 核对后再按其 README 应用 DNS tag。没有 DNS 时可先用 port-forward
验证应用,但不能据此宣称 Gateway 路径已通过。
## 首次初始化与 Terraform
初始管理员密码生成在 PVC 的 `/nexus-data/admin.password`。只在交互式终端中读取并立即
完成首次密码轮换;不得把密码复制进 shell tracing、工单、Git 或命令参数。随后将
Terraform 管理账号的凭据存入 OpenBao,由 CI 通过 Terraform input variable 注入以下
环境变量:
```text
TF_VAR_nexus_url=https://nexus.ad.ddupan.top
TF_VAR_nexus_username=admin
TF_VAR_nexus_password=<OpenBao kv/infra/nexus 的 admin_password 字段>
```
`terraform/` 使用 `sonatype-nexus-community/sonatyperepo` 1.17.0,当前声明:
- `ansible-galaxy-proxy` → `https://galaxy.ansible.com`
- `ansible-public` group
- `go-proxy` → `https://proxy.golang.org`
- `go-public` group
provider credential 不写入 HCL 或 tfvars。正式 apply 前还必须为这个独立 Terraform root
配置远端 backend;本地 state 只允许用于可丢弃的 POC,不提交。验证命令:
```bash
terraform -chdir=apps/nexus/terraform init -backend=false
terraform -chdir=apps/nexus/terraform validate
```
先以 `--check` 查看 OCI repository 漂移,再明确 apply;脚本只从上述环境变量取得凭据,
用临时 `0600` netrc 调用 REST API,退出时删除:
```bash
apps/nexus/terraform/reconcile-oci.sh --check
apps/nexus/terraform/reconcile-oci.sh --apply
```
Terraform 同时把内置 `anonymous` 用户从默认的全仓库 `nx-anonymous` 角色收窄到
`ansible-public`、其返回制品 URL 使用的 `ansible-galaxy-proxy`、`go-public`,以及
`oci-public`/`oci-proxy` 的 `browse/read` 权限。`oci-hosted` 不向匿名用户开放。首次接管
已有实例时先执行
`terraform import sonatyperepo_user.anonymous anonymous,default`,再 apply;不要先启用默认的
全仓库匿名读取。
2026-09-20 的 POC 现场验收已确认:Flux 与 Pod Ready、PVC Bound、HTTPRoute 通过 HTTPS
返回 Nexus 状态 200,Samba DNS 已幂等收敛。全新客户端目录通过匿名入口下载
`community.general:11.2.0` 时冷缓存为 8.49 秒、热缓存为 1.89 秒,两次 tarball SHA-256
一致;`golang.org/x/[email protected]` 为 2.92 秒与 1.51 秒。
## 客户端验收
先验证冷缓存,再原样重复命令验证热缓存;记录 Nexus 请求、上游流量和耗时,不只观察
命令成功。Ansible 配置的 URL 必须以 `/` 结尾:
```ini
[galaxy]
server_list = nexus
[galaxy_server.nexus]
url = https://nexus.ad.ddupan.top/repository/ansible-public/
```
```bash
ansible-galaxy collection install -r collections/requirements.yml \
-p .ansible/collections
```
Go POC 使用:
```bash
GOPROXY=https://nexus.ad.ddupan.top/repository/go-public/ go mod download
```
私有 `git.ddupan.top/*` module 的 `GOPRIVATE`、凭据与是否允许 `direct` fallback 在实际
workflow 中单独决定;不要让私有 module path 意外发往公共 proxy。
OCI 使用 path-based routing:匿名公共拉取地址形如
`nexus.ad.ddupan.top/oci-public/library/alpine:3.22`,认证写入地址形如
`nexus.ad.ddupan.top/oci-hosted/<namespace>/<image>:<tag>`。2026-09-20 现场验收结果:
- `oci-public` 匿名代理拉取 Alpine 冷缓存 4.75 秒、热缓存 0.80 秒,digest 一致;
- `oci-hosted` 认证 push/pull 成功,匿名 pull 返回 401;
- amd64/arm64 OCI image index push 成功,两个平台 manifest 可见;
- Helm chart push/pull digest 与本地 tarball SHA-256 一致;
- Cosign 3.1.3 使用一次性密钥签名并验证成功,OCI 1.1 referrers API 返回一个
`application/vnd.dev.sigstore.bundle.v0.3+json` artifact;
- BuildKit `registry` cache 以 `mode=max` 导出成功,销毁首个 builder 后由新 builder 导入,
两个 `RUN` step 均明确命中 `CACHED`。
本机安装的测试客户端包括 `/usr/local/bin/cosign` 3.1.3;安装时核对官方 Linux amd64
binary SHA-256 `4629c757b7618056f8ddd7e2625ae9fdd94c0372a65049520bc7d9df9efc7f71`。
上述结果仍不代表备份恢复、外部 PostgreSQL 或正式 publisher service account 已完成;
这些项目通过前不得迁移或删除 zot。
## 数据与恢复
POC 的数据库、配置、blob、初始管理员状态都位于 `nexus-data` PVC。删除 Deployment
不会删除 PVC;删除 PVC 会永久删除整个 POC。当前没有独立备份,不能将它用于唯一副本的
正式制品。
恢复验证至少包括:停止写入、取得一致备份、在独立 PVC/实例恢复、登录、列出 repository、
拉取已缓存的 Ansible/Go 制品,并核对 OCI digest/referrers。正式化时再把数据库迁移至
外部 PostgreSQL,并分别定义数据库与 blob 的备份、恢复顺序和 RPO。
参考:
- [Nexus OCI repositories](https://help.sonatype.com/en/oci-repositories.html)
- [Nexus Ansible repositories](https://help.sonatype.com/en/ansible-repositories.html)
- [Nexus Go repositories](https://help.sonatype.com/en/go-repositories.html)
- [官方容器镜像](https://hub.docker.com/r/sonatype/nexus3)
+82
View File
@@ -0,0 +1,82 @@
apiVersion: apps/v1
kind: Deployment
metadata:
name: nexus
namespace: nexus
labels:
app.kubernetes.io/name: nexus
spec:
replicas: 1
strategy:
type: Recreate
selector:
matchLabels:
app.kubernetes.io/name: nexus
template:
metadata:
labels:
app.kubernetes.io/name: nexus
spec:
automountServiceAccountToken: false
securityContext:
fsGroup: 200
fsGroupChangePolicy: OnRootMismatch
runAsGroup: 200
runAsNonRoot: true
runAsUser: 200
seccompProfile:
type: RuntimeDefault
terminationGracePeriodSeconds: 120
containers:
- name: nexus
image: docker.io/sonatype/nexus3:3.96.1@sha256:56142f13432cf072e017aebb2025f201e42ae36ff40bb82618c702504c61f7dd
imagePullPolicy: IfNotPresent
env:
- name: INSTALL4J_ADD_VM_PARAMS
value: >-
-Xms1024m -Xmx2048m -XX:MaxDirectMemorySize=1024m
-Djava.util.prefs.userRoot=/nexus-data/javaprefs
ports:
- name: http
containerPort: 8081
protocol: TCP
resources:
requests:
cpu: 250m
memory: 2Gi
limits:
cpu: "2"
memory: 4Gi
securityContext:
allowPrivilegeEscalation: false
capabilities:
drop:
- ALL
startupProbe:
httpGet:
path: /service/rest/v1/status
port: http
failureThreshold: 60
periodSeconds: 10
timeoutSeconds: 5
readinessProbe:
httpGet:
path: /service/rest/v1/status
port: http
failureThreshold: 6
periodSeconds: 10
timeoutSeconds: 5
livenessProbe:
httpGet:
path: /service/rest/v1/status
port: http
failureThreshold: 6
periodSeconds: 30
timeoutSeconds: 5
volumeMounts:
- name: data
mountPath: /nexus-data
volumes:
- name: data
persistentVolumeClaim:
claimName: nexus-data
+16
View File
@@ -0,0 +1,16 @@
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
name: nexus
namespace: nexus
spec:
parentRefs:
- name: eg
namespace: envoy-gateway-system
sectionName: https
hostnames:
- nexus.ad.ddupan.top
rules:
- backendRefs:
- name: nexus
port: 8081
+9
View File
@@ -0,0 +1,9 @@
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- namespace.yaml
- pvc.yaml
- deployment.yaml
- service.yaml
- httproute.yaml
- networkpolicy.yaml
+8
View File
@@ -0,0 +1,8 @@
apiVersion: v1
kind: Namespace
metadata:
name: nexus
labels:
pod-security.kubernetes.io/enforce: restricted
pod-security.kubernetes.io/audit: restricted
pod-security.kubernetes.io/warn: restricted
+23
View File
@@ -0,0 +1,23 @@
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: nexus-ingress
namespace: nexus
spec:
podSelector:
matchLabels:
app.kubernetes.io/name: nexus
policyTypes:
- Ingress
ingress:
- from:
- namespaceSelector:
matchLabels:
kubernetes.io/metadata.name: envoy-gateway-system
podSelector:
matchLabels:
gateway.envoyproxy.io/owning-gateway-name: eg
gateway.envoyproxy.io/owning-gateway-namespace: envoy-gateway-system
ports:
- protocol: TCP
port: 8081
+12
View File
@@ -0,0 +1,12 @@
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: nexus-data
namespace: nexus
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 50Gi
storageClassName: localpv-zfs-ceph
+14
View File
@@ -0,0 +1,14 @@
apiVersion: v1
kind: Service
metadata:
name: nexus
namespace: nexus
spec:
type: ClusterIP
selector:
app.kubernetes.io/name: nexus
ports:
- name: http
port: 8081
protocol: TCP
targetPort: http
+6
View File
@@ -0,0 +1,6 @@
.terraform/
*.tfstate
*.tfstate.*
*.tfplan
crash.log
crash.*.log
+24
View File
@@ -0,0 +1,24 @@
# This file is maintained automatically by "terraform init".
# Manual edits may be lost in future updates.
provider "registry.terraform.io/sonatype-nexus-community/sonatyperepo" {
version = "1.17.0"
constraints = "1.17.0"
hashes = [
"h1:uKhvhhhI7B+HBsh0zq/ybqKt+EnOGyI6rjcRCtj79ZA=",
"zh:0dde99e7b343fa01f8eefc378171fb8621bedb20f59157d6cc8e3d46c738105f",
"zh:3315929df254a3a6ac27c8c846c2006f7d2a91fadc014351bc4d617f948e5bf9",
"zh:36be5a455af3ce4e187de26753de63e78c1ee9a32dba0135c6cf96a6c1fff25f",
"zh:3f73f7ff57b8c339a7c7ac37653e2dc0b2dd9dcc3f3a538788e7e3ac838337b2",
"zh:40286ecca4c22ab7ae90618ac6d2743f5055199dac81cf5204a4a397c784d439",
"zh:4d24e5c0195fb3155b1967583ee64cfeda402d7cc7f3c73369438f6c69f4245b",
"zh:828a9d7aceaac36af7f9c07af43ec8d20a89148780645d170ffb1c68b2da792d",
"zh:a5ab04de3fe626ec57c832618c6f990abd6610f81e132621651e0b180b970cff",
"zh:a959fa6090a8c0f53739879184e7346423494aee598003df0d1ab4a22b2eee91",
"zh:bdda26c2f03f918bbe59e75abea44868fafda019c3a543725331195df126350b",
"zh:d8048e149ee97ba62971e6a79355d59887bc6d10fcf72cc2feff3d0a2582670c",
"zh:dd36f9988af4e1ca5b1ca7b7bb6f658df9a220dfcda7fec7392fedfe9064f652",
"zh:dda2688d46c7e539fe97e8fe9d3ec81fb364170e018d9c6a681364c8955d4e9d",
"zh:e6b519afe2dea1c0434f766eb6bc9ba78cc5b6ef2c311c2ca3c65cb24744f31f",
]
}
+17
View File
@@ -0,0 +1,17 @@
{
"name": "oci-hosted",
"online": true,
"storage": {
"blobStoreName": "default",
"strictContentTypeValidation": true,
"writePolicy": "ALLOW",
"latestPolicy": false
},
"oci": {
"v1Enabled": false,
"forceBasicAuth": false,
"pathEnabled": true
},
"component": { "proprietaryComponents": false },
"cosign": { "enforcement": "NONE" }
}
+27
View File
@@ -0,0 +1,27 @@
{
"name": "oci-proxy",
"online": true,
"storage": {
"blobStoreName": "default",
"strictContentTypeValidation": true
},
"oci": {
"v1Enabled": false,
"forceBasicAuth": false,
"pathEnabled": true
},
"ociProxy": {
"indexType": "HUB",
"cacheForeignLayers": false,
"foreignLayerUrlWhitelist": []
},
"proxy": {
"remoteUrl": "https://registry-1.docker.io",
"contentMaxAge": 1440,
"metadataMaxAge": 60,
"preserveEncodedCharacters": false
},
"negativeCache": { "enabled": true, "timeToLive": 60 },
"httpClient": { "blocked": false, "autoBlock": true },
"cosign": { "enforcement": "NONE" }
}
+15
View File
@@ -0,0 +1,15 @@
{
"name": "oci-public",
"online": true,
"storage": {
"blobStoreName": "default",
"strictContentTypeValidation": true
},
"group": { "memberNames": ["oci-proxy"] },
"oci": {
"v1Enabled": false,
"forceBasicAuth": false,
"pathEnabled": true
},
"cosign": { "enforcement": "NONE" }
}
+80
View File
@@ -0,0 +1,80 @@
#!/usr/bin/env bash
set -euo pipefail
mode="${1:---check}"
case "$mode" in
--check | --apply) ;;
*) echo "usage: $0 [--check|--apply]" >&2; exit 2 ;;
esac
: "${TF_VAR_nexus_url:?set TF_VAR_nexus_url}"
: "${TF_VAR_nexus_username:?set TF_VAR_nexus_username}"
: "${TF_VAR_nexus_password:?set TF_VAR_nexus_password}"
script_dir="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")" && pwd)"
auth_file="$(mktemp /tmp/nexus-oci-auth.XXXXXX)"
trap 'rm -f -- "$auth_file"' EXIT
chmod 0600 "$auth_file"
printf 'machine %s\nlogin %s\npassword %s\n' \
"${TF_VAR_nexus_url#*://}" "$TF_VAR_nexus_username" \
"$TF_VAR_nexus_password" >"$auth_file"
drift=0
for entry in \
"hosted:$script_dir/oci/oci-hosted.json" \
"proxy:$script_dir/oci/oci-proxy.json" \
"group:$script_dir/oci/oci-public.json"; do
repository_type="${entry%%:*}"
desired_file="${entry#*:}"
repository_name="$(jq -er '.name' "$desired_file")"
endpoint="$TF_VAR_nexus_url/service/rest/v1/repositories/oci/$repository_type"
current_file="$(mktemp /tmp/nexus-oci-current.XXXXXX)"
status="$(curl --silent --show-error --netrc-file "$auth_file" \
--output "$current_file" --write-out '%{http_code}' \
"$endpoint/$repository_name")"
if [[ "$status" == 404 ]]; then
drift=1
if [[ "$mode" == --apply ]]; then
curl --fail --silent --show-error --netrc-file "$auth_file" \
--header 'Content-Type: application/json' \
--data-binary "@$desired_file" "$endpoint"
echo "created $repository_name"
else
echo "missing $repository_name" >&2
fi
elif [[ "$status" == 200 ]]; then
if jq -e --slurpfile desired "$desired_file" '
def subset($actual; $wanted):
if ($wanted | type) == "object" then
all($wanted | keys[];
($actual[.] != null) and subset($actual[.]; $wanted[.]))
else
$actual == $wanted
end;
subset(.; $desired[0])
' "$current_file" >/dev/null; then
echo "in sync $repository_name"
else
drift=1
if [[ "$mode" == --apply ]]; then
curl --fail --silent --show-error --netrc-file "$auth_file" \
--request PUT --header 'Content-Type: application/json' \
--data-binary "@$desired_file" "$endpoint/$repository_name"
echo "updated $repository_name"
else
echo "drifted $repository_name" >&2
fi
fi
else
cat "$current_file" >&2
echo "unexpected HTTP $status for $repository_name" >&2
exit 1
fi
rm -f -- "$current_file"
done
if [[ "$mode" == --check && "$drift" -ne 0 ]]; then
exit 1
fi
+64
View File
@@ -0,0 +1,64 @@
locals {
proxy_http_client = {
auto_block = true
blocked = false
}
proxy_negative_cache = {
enabled = true
time_to_live = 60
}
repository_storage = {
blob_store_name = "default"
strict_content_type_validation = true
}
}
resource "sonatyperepo_repository_ansiblegalaxy_proxy" "galaxy" {
name = "ansible-galaxy-proxy"
online = true
http_client = local.proxy_http_client
negative_cache = local.proxy_negative_cache
proxy = {
remote_url = "https://galaxy.ansible.com"
content_max_age = 1440
metadata_max_age = 60
}
storage = local.repository_storage
}
resource "sonatyperepo_repository_ansiblegalaxy_group" "public" {
name = "ansible-public"
online = true
group = {
member_names = [sonatyperepo_repository_ansiblegalaxy_proxy.galaxy.name]
}
storage = local.repository_storage
}
resource "sonatyperepo_repository_go_proxy" "public" {
name = "go-proxy"
online = true
http_client = local.proxy_http_client
negative_cache = local.proxy_negative_cache
proxy = {
remote_url = "https://proxy.golang.org"
content_max_age = 1440
metadata_max_age = 60
}
storage = local.repository_storage
}
resource "sonatyperepo_repository_go_group" "public" {
name = "go-public"
online = true
group = {
member_names = [sonatyperepo_repository_go_proxy.public.name]
}
storage = local.repository_storage
}
+75
View File
@@ -0,0 +1,75 @@
resource "sonatyperepo_privilege_repository_view" "anonymous_ansible" {
name = "ci-anonymous-ansible-read"
description = "Anonymous read access to the Ansible Galaxy group"
actions = ["BROWSE", "READ"]
format = "ansiblegalaxy"
repository = sonatyperepo_repository_ansiblegalaxy_group.public.name
}
resource "sonatyperepo_privilege_repository_view" "anonymous_ansible_proxy" {
name = "ci-anonymous-ansible-proxy-read"
description = "Anonymous artifact read access to the Ansible Galaxy proxy"
actions = ["BROWSE", "READ"]
format = "ansiblegalaxy"
repository = sonatyperepo_repository_ansiblegalaxy_proxy.galaxy.name
}
resource "sonatyperepo_privilege_repository_view" "anonymous_go" {
name = "ci-anonymous-go-read"
description = "Anonymous read access to the Go module group"
actions = ["BROWSE", "READ"]
format = "go"
repository = sonatyperepo_repository_go_group.public.name
}
resource "sonatyperepo_privilege_repository_view" "anonymous_oci_public" {
name = "ci-anonymous-oci-public-read"
description = "Anonymous read access to the public OCI group"
actions = ["BROWSE", "READ"]
format = "oci"
repository = "oci-public"
}
resource "sonatyperepo_privilege_repository_view" "anonymous_oci_proxy" {
name = "ci-anonymous-oci-proxy-read"
description = "Anonymous read access to the OCI proxy member"
actions = ["BROWSE", "READ"]
format = "oci"
repository = "oci-proxy"
}
resource "sonatyperepo_role" "anonymous_ci" {
id = "ci-anonymous-read"
name = "CI anonymous read"
description = "Read-only access to public CI dependency proxy groups"
privileges = [
sonatyperepo_privilege_repository_view.anonymous_ansible.name,
sonatyperepo_privilege_repository_view.anonymous_ansible_proxy.name,
sonatyperepo_privilege_repository_view.anonymous_go.name,
sonatyperepo_privilege_repository_view.anonymous_oci_public.name,
sonatyperepo_privilege_repository_view.anonymous_oci_proxy.name,
]
roles = []
}
resource "sonatyperepo_user" "anonymous" {
user_id = "anonymous"
first_name = "Anonymous"
last_name = "User"
email_address = "[email protected]"
status = "active"
roles = [sonatyperepo_role.anonymous_ci.id]
}
resource "sonatyperepo_system_anonymous_access" "ci" {
enabled = true
user_id = sonatyperepo_user.anonymous.user_id
realm_name = "NexusAuthorizingRealm"
}
resource "sonatyperepo_security_realms" "active" {
active = [
"NexusAuthenticatingRealm",
"OciBearerToken",
]
}
+33
View File
@@ -0,0 +1,33 @@
terraform {
required_version = ">= 1.11.0"
required_providers {
sonatyperepo = {
source = "sonatype-nexus-community/sonatyperepo"
version = "1.17.0"
}
}
}
provider "sonatyperepo" {
url = var.nexus_url
username = var.nexus_username
password = var.nexus_password
}
variable "nexus_url" {
description = "Nexus Repository base URL"
type = string
}
variable "nexus_username" {
description = "Nexus Terraform management username"
type = string
sensitive = true
}
variable "nexus_password" {
description = "Nexus Terraform management password"
type = string
sensitive = true
}
+2
View File
@@ -53,4 +53,6 @@ sudo k3s kubectl -n flux-system get gitrepositories,kustomizations
- SPIRE 已按 hardened chart 内部 fork `0.30.2-ddupan.1`(基于上游 `0.30.2`,SPIRE
`1.15.3`)声明,使用共享
PostgreSQL 与独立 signing-key PVC;首次上线和 OpenBao JWT-SVID PoC 尚待合并后验证;
- Nexus Repository CE POC 已加入 GitOps 声明,计划验证 Ansible Galaxy、Go Modules 与
OCI/BuildKit 缓存;尚未部署或完成现场验收,现有 zot 保持不变;
- root Kustomization 与所有 brownfield 子 Kustomization 继续保持 `prune: false`。
+3 -2
View File
@@ -6,11 +6,12 @@ metadata:
spec:
dependsOn:
- name: external-secrets
- name: nats
- name: spire
interval: 10m
path: ./platform/dynamic-runner
prune: false
# The runner backends are replaceable. Prune is required when a retired
# worker is removed from the component; otherwise it keeps consuming work.
prune: true
sourceRef:
kind: GitRepository
name: flux-system
+22
View File
@@ -0,0 +1,22 @@
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
name: nexus
namespace: flux-system
spec:
dependsOn:
- name: envoy-gateway
- name: openebs
healthChecks:
- apiVersion: apps/v1
kind: Deployment
name: nexus
namespace: nexus
interval: 10m
path: ./apps/nexus
prune: false
sourceRef:
kind: GitRepository
name: flux-system
timeout: 15m
wait: true
+1
View File
@@ -15,3 +15,4 @@ resources:
- apps/spire.yaml
- apps/observability.yaml
- apps/zot.yaml
- apps/nexus.yaml
+15 -4
View File
@@ -15,7 +15,8 @@ Root bootstrap 已完成。后续按依赖顺序分别引入:
1. 监控 CRD、kube-state-metrics 以及 kubelet/cAdvisor 抓取配置;
2. SPIRE Agent、SPIFFE CSI Driver 与 workload registration;
3. Kata Containers、`block-plain` RuntimeClass;
4. OpenSandbox operator/server 及 `ci-pod`、`ci-vm` Pools。
4. 独立 External Secrets Operator 与 sandbox 专用 OpenBao auth backend;
5. OpenSandbox controller/server;CI Pool 与 runner 调度器随后独立接入。
每一阶段单独合并并等待对应 Flux Kustomization Ready,不在 bootstrap 时一次性部署。
第一阶段监控拆为 `monitoring-operator` 与依赖它的 `monitoring`,防止 VM CR 在
@@ -42,15 +43,25 @@ attestation。Server 使用 external bundle publisher 持续维护 sandbox
显式关闭 Server 与 OIDC Provider,只部署 Agent DaemonSet 和 SPIFFE CSI Driver;因此
不会产生第二个 trust root。
`spire-smoke` namespace、ServiceAccount 和 `sandbox-spire-smoke` ClusterSPIFFEID 是
普通 Pod 与后续 Kata guest 的回归夹具,稳定身份为
`spiffe://ddupan.top/sandbox/smoke`。测试 Pod 临时创建并在验收后删除,身份声明保留。
`spire-smoke` namespace、ServiceAccount 和 `sandbox-spire-smoke` ClusterSPIFFEID 只用于
普通 Pod 的 CSI 回归夹具,稳定身份为 `spiffe://ddupan.top/sandbox/smoke`。Kata guest
不能复用 node Agent 暴露的 Unix socket;virtio-fs 只能呈现 socket 路径,不能把连接
跨过 VM 边界。Kata workload 必须使用 guest 内 Agent,具体约束见
`platform/sandbox-kata/README.md`。测试 Pod 临时创建并在验收后删除,普通 Pod 的身份
声明保留。
Kata 阶段使用官方 4.1.0 `kata-deploy` chart 的短生命周期 `job` 模式,逐节点安装并
重启 K3s。只启用 `kata-clh-runtime-rs`,不创建默认 `kata` 别名;该 handler 的
`emptyDir` 固定使用 `block-plain`,为 Docker/BuildKit overlay2 与 kind 提供 guest
内块设备文件系统。详细限制与上线验收见 `platform/sandbox-kata/README.md`。
Sandbox 的 ESO 通过独立 `auth/kubernetes-sandbox` 向 OpenBao 证明 ServiceAccount
身份,只能读取共享的 `kv/k8s/opensandbox-api`。它不保存 reviewer JWT 或长期 Bao token;相关
Terraform 与 Flux 边界见 `platform/sandbox-external-secrets/README.md`。
OpenSandbox 阶段固定官方源码 commit 与 umbrella chart `0.2.2`,只部署 controller、
ClusterIP server 和 CRD。API key 由 ESO 从 OpenBao 投影,明文不进入 Git。
## 监控边界
这里只管理 sandbox LXC 内的 Kubernetes 监控,不负责 PVE 宿主监控。LXC 与宿主共享
+18
View File
@@ -0,0 +1,18 @@
---
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
name: ci-runners
namespace: flux-system
spec:
dependsOn:
- name: opensandbox
- name: spire-agents
interval: 10m
path: ./platform/sandbox-ci-runners
prune: true
sourceRef:
kind: GitRepository
name: flux-system
timeout: 20m
wait: true
@@ -0,0 +1,17 @@
---
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
name: external-secrets-operator
namespace: flux-system
spec:
dependsOn:
- name: monitoring-operator
interval: 10m
path: ./platform/sandbox-external-secrets/operator
prune: true
sourceRef:
kind: GitRepository
name: flux-system
timeout: 10m
wait: true
@@ -0,0 +1,17 @@
---
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
name: external-secrets
namespace: flux-system
spec:
dependsOn:
- name: external-secrets-operator
interval: 10m
path: ./platform/sandbox-external-secrets/config
prune: true
sourceRef:
kind: GitRepository
name: flux-system
timeout: 10m
wait: true
+19
View File
@@ -0,0 +1,19 @@
---
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
name: opensandbox
namespace: flux-system
spec:
dependsOn:
- name: external-secrets
- name: kata
- name: monitoring-operator
interval: 10m
path: ./platform/sandbox-opensandbox
prune: true
sourceRef:
kind: GitRepository
name: flux-system
timeout: 15m
wait: true
+4
View File
@@ -7,3 +7,7 @@ resources:
- apps/spire-bootstrap.yaml
- apps/spire-agents.yaml
- apps/kata.yaml
- apps/external-secrets-operator.yaml
- apps/external-secrets.yaml
- apps/opensandbox.yaml
- apps/ci-runners.yaml
+1
View File
@@ -18,6 +18,7 @@ homelab_dns:
- { zone: ad.ddupan.top, name: metrics-write, type: A, values: [192.168.10.127] }
- { zone: ad.ddupan.top, name: netbox, type: A, values: [192.168.10.127] }
- { zone: ad.ddupan.top, name: nats, type: A, values: [192.168.10.127] }
- { zone: ad.ddupan.top, name: nexus, type: A, values: [192.168.10.127] }
- { zone: ad.ddupan.top, name: s3, type: A, values: [192.168.10.127] }
- { zone: ad.ddupan.top, name: spire-oidc, type: A, values: [192.168.10.127] }
- { zone: ad.ddupan.top, name: spire-server, type: A, values: [192.168.10.127] }
+8 -1
View File
@@ -51,7 +51,7 @@ few things Terraform must not own.**
| secrets engine mounts (`kv`, `pki`, `ssh-client-signer`) | the daemon, Raft, TLS files, systemd |
| PKI role, issuing/CRL URLs, cluster paths, **ACME** | `bao operator init` / unseal (manual, PGP-wrapped) |
| SSH signing role (`ai-agent`) | **PKI root CA + SSH CA signing key** |
| OIDC auth *mount* and *role* | OIDC **client secret** (`auth/oidc/config`) |
| OIDC/Kubernetes auth mount、config 与 role | OIDC **client secret** (`auth/oidc/config`) |
| all policies | snapshot token + script + systemd timer |
| | host-level CA trust distribution (`openbao_ssh_ca_trust`) |
@@ -92,6 +92,13 @@ then `VAULT_ADDR`/`VAULT_TOKEN`), mirroring how `smtp-relay/terraform` uses
native `openbao/openbao` provider is published only to the OpenTofu registry and
cannot be resolved by the HashiCorp `terraform` CLI.
Sandbox 集群使用独立的 `auth/kubernetes-sandbox`。其 API 地址、公开 Kubernetes CA、
ESO role 与只读 `kv/k8s/opensandbox-api` policy 全部由 Terraform 管理;CA 位于
`terraform/certs/sandbox-kubernetes-ca.crt`。集群重建并轮换 CA 后,先更新该文件并
apply,再让 Flux 恢复 ESO reconciliation。该 backend 不保存 reviewer JWT,而是使用
ESO 的短期登录 JWT 执行 TokenReview。该组资源已于 2026-09-18 apply,随后复验 plan
为 zero-diff。
## DNS
`bao.ad.ddupan.top` is an **internal-only** name — not a public Cloudflare record and
+30
View File
@@ -59,3 +59,33 @@ resource "vault_kubernetes_auth_backend_role" "external_secrets" {
# in a long TTL and every extra hour is a longer-lived credential in memory.
token_ttl = 3600
}
# A Kubernetes auth mount can validate identities from only the API server it is
# configured against. The sandbox cluster therefore cannot reuse auth/kubernetes,
# whose TokenReview endpoint belongs to homelab.
resource "vault_auth_backend" "sandbox_kubernetes" {
type = "kubernetes"
path = "kubernetes-sandbox"
}
resource "vault_kubernetes_auth_backend_config" "sandbox" {
backend = vault_auth_backend.sandbox_kubernetes.path
kubernetes_host = "https://10.60.0.13:6443"
kubernetes_ca_cert = file("${path.module}/certs/sandbox-kubernetes-ca.crt")
disable_local_ca_jwt = true
# Deliberately omit token_reviewer_jwt. OpenBao uses the login JWT for
# TokenReview; the sandbox external-secrets ServiceAccount is bound only to
# system:auth-delegator and all issued JWTs remain short-lived.
}
resource "vault_kubernetes_auth_backend_role" "sandbox_external_secrets" {
backend = vault_auth_backend.sandbox_kubernetes.path
role_name = "external-secrets"
bound_service_account_names = ["external-secrets"]
bound_service_account_namespaces = ["external-secrets"]
token_policies = [vault_policy.sandbox_external_secrets.name]
token_ttl = 3600
}
@@ -0,0 +1,10 @@
-----BEGIN CERTIFICATE-----
MIIBdjCCAR2gAwIBAgIBADAKBggqhkjOPQQDAjAjMSEwHwYDVQQDDBhrM3Mtc2Vy
dmVyLWNhQDE3ODk2NTI4NTcwHhcNMjYwOTE3MTI0NzM3WhcNMzYwOTE0MTI0NzM3
WjAjMSEwHwYDVQQDDBhrM3Mtc2VydmVyLWNhQDE3ODk2NTI4NTcwWTATBgcqhkjO
PQIBBggqhkjOPQMBBwNCAAR4SbqzTXZnlZdUPz7viN6+dYbB1Maw44Qiepn9r5XG
sOzkYkN8t1aG3Ugo8TqQ3xJaKkM89n1Rluj0vbOhiNajo0IwQDAOBgNVHQ8BAf8E
BAMCAqQwDwYDVR0TAQH/BAUwAwEB/zAdBgNVHQ4EFgQUyLSGoKAKAJuiniuRdBLG
XYaDQC8wCgYIKoZIzj0EAwIDRwAwRAIgFkVzyUZexk/ynnxBEOg+3foJv3WKqAei
hTSRjO1gL0UCIFbBKR7BMrJJAgW3DJFeeBM+b+tTg93jNx55qZACbFOL
-----END CERTIFICATE-----
@@ -35,3 +35,10 @@ resource "vault_policy" "external_secrets" {
name = "external-secrets"
policy = file("${path.module}/policies/external-secrets.hcl")
}
# The sandbox cluster has its own Kubernetes auth backend and a deliberately
# narrower KV view than the homelab ESO instance.
resource "vault_policy" "sandbox_external_secrets" {
name = "sandbox-external-secrets"
policy = file("${path.module}/policies/sandbox-external-secrets.hcl")
}
@@ -0,0 +1,9 @@
# Read only the shared OpenSandbox control-plane API key. The same Bao object is
# consumed by the server in sandbox and, later, by the scheduler in homelab.
path "kv/data/k8s/opensandbox-api" {
capabilities = ["read"]
}
path "kv/metadata/k8s/opensandbox-api" {
capabilities = ["read"]
}
@@ -83,6 +83,11 @@ vyos_postgresql_primary_address: "10.60.0.11"
vyos_sandbox_k3s_api_address: "10.60.0.13"
vyos_sandbox_k3s_api_interface: eth1
vyos_sandbox_k3s_api_port: 6443
vyos_opensandbox_api_port: 8080
vyos_opensandbox_node_port: 30080
# Lifecycle create is synchronous and a cold Kata image pull can exceed the
# HAProxy 50-second default. Keep this below OpenSandbox's overall timeout.
vyos_opensandbox_api_timeout: 600
vyos_sandbox_k3s_servers:
- name: sandbox1
address: "10.60.0.11"
@@ -15,6 +15,20 @@ set interfaces ethernet {{ i.iface }} address {{ i.address }}
set interfaces ethernet {{ i.iface }} description '{{ i.description }}'
{% endfor %}
{# OpenSandbox stays on a NodePort; VyOS owns the stable routed frontend. #}
set load-balancing haproxy service opensandbox-api listen-address {{ vyos_sandbox_k3s_api_address }}
set load-balancing haproxy service opensandbox-api port {{ vyos_opensandbox_api_port }}
set load-balancing haproxy service opensandbox-api mode 'tcp'
set load-balancing haproxy service opensandbox-api backend 'opensandbox-api'
set load-balancing haproxy service opensandbox-api timeout client {{ vyos_opensandbox_api_timeout }}
set load-balancing haproxy backend opensandbox-api mode 'tcp'
set load-balancing haproxy backend opensandbox-api timeout server {{ vyos_opensandbox_api_timeout }}
{% for server in vyos_sandbox_k3s_servers %}
set load-balancing haproxy backend opensandbox-api server {{ server.name }} address {{ server.address }}
set load-balancing haproxy backend opensandbox-api server {{ server.name }} port {{ vyos_opensandbox_node_port }}
set load-balancing haproxy backend opensandbox-api server {{ server.name }} check
{% endfor %}
{# --- default route out; OSPF carries the rest --- #}
set protocols static route 0.0.0.0/0 next-hop {{ vyos_lan_gateway }}
+12 -2
View File
@@ -18,7 +18,8 @@ Flux 管理以下 Kubernetes 资源:
- Kata Containers 和 CI 专用的 `block-plain` RuntimeClass;
- SPIRE Agent、SPIFFE CSI Driver 与 workload identity 声明;
- vmagent、kube-state-metrics、kubelet/cAdvisor scrape 配置和告警;
- OpenSandbox operator/server、`ci-pod` 与 `ci-vm` Pools。
- sandbox External Secrets Operator、OpenSandbox controller/server;CI Pool 与 runner
调度器由 runner 项目接入。
同一个对象只能有一个 owner。Ansible 不直接部署上述集群内 workload;Flux 不管理
LXC、K3s datastore 或 K3s 本身。
@@ -92,6 +93,13 @@ ansible-playbook site.yml
ansible-playbook k3s.yml
```
只 reconcile Flux controllers 与 root sync manifest(不触碰 LXC、PostgreSQL 或
K3s lifecycle):
```bash
ansible-playbook flux.yml
```
K3s 外部 datastore URI 由运行时 `SANDBOX_K3S_DB_PASSWORD` 生成,密码在 URI 中
进行 URL 编码,最终仅持久化于节点 root 可读的 `/etc/rancher/k3s/config.yaml`
(mode `0600`)。首节点生成的 K3s join token 仅在同一次 Ansible run 内传给第二节点;
@@ -106,7 +114,9 @@ ansible-playbook verify.yml
当前已经声明 LXC 生命周期、最小 OS baseline、PostgreSQL 和 K3s,包括系统级
homelab CA trust。Flux `v2.9.5` controllers 与 root sync 也由 Ansible 通过 K3s
server manifests 管理;root 使用 homelab CA 访问公开 Gitea 仓库,不保存 Git token。
集群内 workload 由 `clusters/sandbox/` 分阶段纳入 Flux。
集群内 workload 由 `clusters/sandbox/` 分阶段纳入 Flux。root Kustomization 的健康检查
timeout 为 40 分钟,用于覆盖 Kata 等首次安装时会逐节点重启 K3s 的子
Kustomization;各子项仍保留自己的更短 timeout,故障会在对应子项先行暴露。
## SPIRE 跨集群 bootstrap
@@ -0,0 +1,6 @@
---
- name: Reconcile Flux controllers and root sync
hosts: sandbox1
gather_facts: false
roles:
- sandbox_flux
@@ -35,5 +35,5 @@ spec:
sourceRef:
kind: GitRepository
name: flux-system
timeout: 3m
timeout: 40m
wait: true
+88 -36
View File
@@ -1,51 +1,103 @@
# Gitea dynamic runner controller
# Gitea dynamic runner
此目录只管理 homelab 中的 controller 部署。controller、worker、Cloud Hypervisor
launcher 和 guest runner 的源码与发布位于独立仓库
`panxiao81/gitea-dynamic-runner`。
当前 bootstrap controller 接收 Gitea `workflow_job` webhook,将 `[self-hosted, pod]` 和
`[self-hosted, vm]` 的 queued job 分别发布到 NATS。Pod worker 在本集群创建一次性
privileged host runner;Docker、BuildKit 和 kind 由 workflow 自行 setup。内部
endpoint:
本目录部署单副本 Go controller,在同一进程运行 RunnerService scheduler、原生
Kubernetes Pod worker、OpenSandbox VM worker 和 SPIFFE mTLS facade:
```text
http://dynamic-runner-controller.dynamic-runner.svc.cluster.local:8787/webhook
Gitea RunnerService -> scheduler -> JetStream ci.runner.pod
|
v
homelab Kubernetes Pod
|
SPIFFE mTLS RunnerService facade
|
v
Gitea
```
OpenBao 路径:
Pod backend 不经过 OpenSandbox。VM backend 后续启用时才访问 VyOS 暴露的
OpenSandbox Lifecycle API;本目录不修改 sandbox 平台侧 ESO、Bao Terraform 或
OpenSandbox chart 所有权边界。
- `kv/k8s/nats.ci_producer_password`:已有 NATS producer 密码。
- `kv/k8s/nats.ci_worker_password`:已有 NATS worker 密码。
- `kv/k8s/dynamic-runner.webhook_secret`:Gitea webhook HMAC secret。
- `kv/k8s/gitea-runner.token`:现有 instance runner registration token。
## Canary 安全边界
首期 controller 与 runner 镜像由 laptop 本机构建后导入 k3s containerd,作为 CI
发布链路建立前的 bootstrap。部署使用 `imagePullPolicy: Never`。正式发布 workflow
获得专用 SPIFFE ID 后,必须将 image 改为 zot digest 并移除本地导入步骤。
- Deployment 为单副本,滚动策略固定 `maxSurge: 1`、`maxUnavailable: 0`,保证新
facade Ready 后才终止旧实例。scheduler 必须通过 Kubernetes Lease 保持单 leader,
不能依赖 Recreate 避免重复领取。
- scheduler 使用一个 runner registration,并按总容量启动并发 `FetchTask` goroutine;
`POD_CAPACITY=4` 与 `VM_CAPACITY=1` 分别限制两个 durable consumer 和 backend
admission pool。池满时 assignment 保持 JetStream pending,任一 backend 不占用
另一方的执行槽位,也不会创建超出容量的 workload。
- rollout 重叠期间只有持有 `Lease/dynamic-runner-scheduler` 的 controller 执行
`FetchTask`;所有 Ready 实例都可通过 backend metadata 恢复 claim 并服务 facade。
- executor 镜像使用 digest;Pod 以 UID 2000 运行,SPIRE `ClusterStaticEntry` 同时绑定
具体 Pod UID 与 `unix:uid:2000`。
- facade 通过仅内网可路由的 `192.168.10.127:30443` NodePort 提供给 sandbox executor;
双方使用 Workload API X509-SVID mTLS,并按 SPIFFE ID 而不是 IP/DNS 名验证服务端。
该入口不经过公网或 Cloudflare Tunnel。
- controller 的 SPIFFE ID 固定为
`spiffe://ddupan.top/ns/dynamic-runner/sa/dynamic-runner-controller`。
- NATS 保留既有最小权限分离:`ci-producer` 仅 publish,`ci-worker` 仅 pull/ACK。
## 身份绑定
facade claim registry 在启动时从 Pod/OpenSandbox metadata 恢复;scheduler leadership
由 Kubernetes Lease 持久化协调。Deployment 仍保持 `replicas: 1`,滚动更新期间允许
一个额外 Pod 提供 facade 连续性。
queued webhook 只负责创建没有业务身份的 Pod。runner 实际领取任务后,Gitea 的
`in_progress` webhook 会携带实际 `runner_name`;controller 将 binding 消息发布到
NATS,Pod worker 再给对应 Pod 添加:
## Secret 边界
```text
ci.ddupan.top/identity-bound=true
ci.ddupan.top/spiffe-path=<owner>/<repository>/<percent-encoded-job-name>
`ExternalSecret/dynamic-runner` 从既有 `ClusterSecretStore/openbao` 读取:
- `kv/k8s/nats`:producer/worker 密码;
- `kv/k8s/opensandbox-api:api_key`:保留给后续 VM worker;
- `kv/k8s/gitea-runner:token`:保留的 runner registration token;
- `kv/k8s/dynamic-runner`:scheduler UUID/token、facade HMAC key 和回滚所需 webhook secret。
scheduler credential 由官方 Gitea Runner v3.5.0 一次注册生成;它只挂载到 controller,
不会进入 executor。facade capability key 至少 32 字节,controller 为每个 assignment
确定性生成独立 capability。不要打印 Secret、创建静态 Bao token或把 credential 写入
Git。OpenBao 写入使用本机 SPIFFE JWT 换取的短期 `local-development` token。
## Workflow 身份与依赖配置
workflow 可复用 [`panxiao81/ci-actions`](https://git.ddupan.top/panxiao81/ci-actions)
中的 `spiffe-openbao-login@v1` 和 `setup-nexus@v1`。这不改变 runner 的权限边界:
runner 仅提供 Node.js 20、`spire-agent` 与 Workload API socket,workflow 负责声明 Bao
role、audience 和具体用途,目标服务 policy 决定是否授权。短期 Bao token 会进入
Actions job 临时文件,因此这些 Action 只允许在本目录管理的一次性 Pod/VM executor
中使用,不能迁移到共享或持久 runner。
匿名读取 Nexus public repository 只需 `setup-nexus@v1`,不应为了依赖下载额外申请
Bao 凭据;需要发布制品时再为对应 repository 建立独立 service account 与最小权限
policy。
## 首次验收
合并后先观察 Flux 与 controller,不要立即开启 VM:
```bash
flux reconcile kustomization dynamic-runner --with-source
kubectl -n dynamic-runner wait externalsecret/dynamic-runner \
--for=condition=Ready --timeout=2m
kubectl -n dynamic-runner rollout status deploy/dynamic-runner-controller --timeout=5m
kubectl -n dynamic-runner logs deploy/dynamic-runner-controller -f
```
`ClusterSPIFFEID/gitea-dynamic-runner` 只匹配已经绑定的 Pod,并签发
`spiffe://ddupan.top/ci/<owner>/<repository>/<job-name>`。runner 的 job-start hook 在
SVID 可用之前不会放行第一步,因此不能根据 queued 事件错配身份。
确认 scheduler 只领取一条 `[self-hosted,pod]` task,然后验证:
每个 runner Pod 使用 `gitea-dynamic-runner` ServiceAccount。该 ServiceAccount 没有
Kubernetes API 权限;只有 `dynamic-runner-pod-worker` ServiceAccount 能在本 namespace
create/get/patch/delete Pod。
1. assignment message 进入并离开 durable `pod` consumer;
2. `gitea-task-<task-id>` Pod 创建,取得实际 Pod UID;
3. 同名 `ClusterStaticEntry` 的 parent ID 包含该 UID,SPIFFE ID 使用 repository/job key;
4. 官方 runner v3.5.0 经 facade claim 精确 task,Gitea 实时收到日志和终态;
5. Pod consumer 达到 capacity 时 VM consumer 仍可独立接受任务。
长期实现将由兼容 Gitea RunnerService 的 scheduler 直接领取 task,再交给 Pod/VM
executor;届时删除 webhook、临时 runner 注册和 identity binding 消息。跟踪见
`panxiao81/gitea-dynamic-runner` issue #7。
Pod 与 VM 都在 Gitea 接受终态后先把 terminal marker 写入各自 backend metadata,再由
lifecycle reconciler 删除执行器。首次 VM 测试仍须观察 BatchSandbox、Pod 与
ClusterStaticEntry 全部消失;完整自动清理通过前不得提高 `POD_CAPACITY` 或
`VM_CAPACITY`。
Gitea webhook 只订阅 `workflow_job`,content type 使用 JSON,secret 与 Bao 中值
一致。不要启用 `send_everything`,否则 controller 会收到无关仓库事件。
## 回滚
若 controller 在领取 task 前失败,回滚到前一 commit 的 Python controller/Pod worker
manifests。若已经创建 `gitea-task-*` Pod,先保留现场并核对 Gitea task 状态,不能直接
重启 scheduler 造成重复执行。scheduler registration 和 capability key 保留在 Bao,
回滚不需要删除或打印它们。
+4 -5
View File
@@ -1,17 +1,16 @@
apiVersion: spire.spiffe.io/v1alpha1
kind: ClusterSPIFFEID
metadata:
name: gitea-dynamic-runner
name: dynamic-runner-controller
spec:
className: spire-mgmt-spire
spiffeIDTemplate: 'spiffe://{{ .TrustDomain }}/ci/{{ index .PodMeta.Annotations "ci.ddupan.top/spiffe-path" }}'
spiffeIDTemplate: spiffe://ddupan.top/ns/dynamic-runner/sa/dynamic-runner-controller
namespaceSelector:
matchLabels:
kubernetes.io/metadata.name: dynamic-runner
podSelector:
matchLabels:
app.kubernetes.io/name: gitea-dynamic-runner
ci.ddupan.top/identity-bound: "true"
app.kubernetes.io/name: dynamic-runner-controller
workloadSelectorTemplates:
- k8s:ns:dynamic-runner
- k8s:sa:gitea-dynamic-runner
- k8s:sa:dynamic-runner-controller
+82 -28
View File
@@ -5,6 +5,11 @@ metadata:
namespace: dynamic-runner
spec:
replicas: 1
strategy:
type: RollingUpdate
rollingUpdate:
maxSurge: 1
maxUnavailable: 0
selector:
matchLabels:
app.kubernetes.io/name: dynamic-runner-controller
@@ -14,17 +19,10 @@ spec:
app.kubernetes.io/name: dynamic-runner-controller
spec:
serviceAccountName: dynamic-runner-controller
automountServiceAccountToken: false
initContainers:
- name: fetch-internal-ca
image: curlimages/curl:8.16.0@sha256:463eaf6072688fe96ac64fa623fe73e1dbe25d8ad6c34404a669ad3ce1f104b6
args:
- --fail
- --silent
- --show-error
- --output
- /trust/ca.pem
- https://bao.ad.ddupan.top:8200/v1/pki/ca/pem
args: [--fail, --silent, --show-error, --output, /trust/ca.pem, https://bao.ad.ddupan.top:8200/v1/pki/ca/pem]
securityContext:
allowPrivilegeEscalation: false
capabilities:
@@ -40,40 +38,89 @@ spec:
mountPath: /trust
containers:
- name: controller
# Bootstrap import on laptop. Replace with a zot digest after the
# repository's image publishing workflow has a dedicated identity.
image: gitea-dynamic-runner-controller:0.3.0-bootstrap
imagePullPolicy: Never
image: zot.ad.ddupan.top/panxiao81/gitea-dynamic-runner-controller@sha256:9372ef869353dc7f79186bfae4191c1be0e52159c166489e76efd339e60c51b8
imagePullPolicy: IfNotPresent
args: [controller]
env:
- name: COMPONENTS
value: scheduler,pod-worker,vm-worker
- name: GITEA_INSTANCE_URL
value: https://git.ddupan.top
- name: GITEA_RUNNER_UUID_FILE
value: /run/dynamic-runner-secrets/scheduler-uuid
- name: GITEA_RUNNER_TOKEN_FILE
value: /run/dynamic-runner-secrets/scheduler-token
- name: NATS_URL
value: tls://nats.ad.ddupan.top:4222
- name: NATS_CA_FILE
value: /run/trust/ca.pem
- name: NATS_PASSWORD_FILE
- name: NATS_PRODUCER_PASSWORD_FILE
value: /run/dynamic-runner-secrets/nats-password
- name: WEBHOOK_SECRET_FILE
value: /run/dynamic-runner-secrets/webhook-secret
- name: NATS_WORKER_PASSWORD_FILE
value: /run/dynamic-runner-secrets/nats-worker-password
- name: RUNNER_FACADE_CAPABILITY_KEY_FILE
value: /run/dynamic-runner-secrets/facade-capability-key
- name: RUNNER_FACADE_LISTEN
value: :8443
- name: RUNNER_FACADE_URL
value: https://192.168.10.127:30443
- name: RUNNER_FACADE_SPIFFE_ID
value: spiffe://ddupan.top/ns/dynamic-runner/sa/dynamic-runner-controller
- name: SPIFFE_ENDPOINT_SOCKET
value: unix:///run/spire/agent-sockets/spire-agent.sock
- name: POD_NAMESPACE
valueFrom:
fieldRef:
fieldPath: metadata.namespace
- name: POD_EXECUTOR_IMAGE
value: zot.ad.ddupan.top/panxiao81/gitea-dynamic-runner-runner@sha256:08f83c9993645dce376140fdde7e29e809795166b2bb32e50cebdbefaf3cf303
- name: POD_SERVICE_ACCOUNT
value: gitea-dynamic-runner
- name: POD_EXECUTOR_UID
value: "2000"
- name: POD_CAPACITY
value: "4"
- name: OPENSANDBOX_API
value: http://10.60.0.13:8080
- name: OPENSANDBOX_API_KEY_FILE
value: /run/dynamic-runner-secrets/opensandbox-api-key
- name: OPENSANDBOX_POOL
value: ci-vm
- name: VM_CAPACITY
value: "1"
- name: VM_RUNNER_LABEL
value: vm-dev
- name: VM_TIMEOUT_SECONDS
value: "14400"
- name: SPIRE_CLUSTER
value: homelab
- name: SPIRE_CLASS
value: spire-mgmt-spire
- name: SPIRE_AGENT_ID
value: spiffe://ddupan.top/spire/agent/k8s_psat/homelab/cd2d0233-c4ea-4031-8327-e7e359e766dd
- name: SSL_CERT_FILE
value: /run/trust/ca.pem
ports:
- name: http
containerPort: 8787
- name: facade
containerPort: 8443
readinessProbe:
httpGet:
path: /healthz
port: http
httpGet: null
tcpSocket:
port: facade
periodSeconds: 5
livenessProbe:
httpGet:
path: /healthz
port: http
initialDelaySeconds: 10
httpGet: null
tcpSocket:
port: facade
initialDelaySeconds: 15
periodSeconds: 10
resources:
requests:
cpu: 25m
memory: 32Mi
cpu: 50m
memory: 64Mi
limits:
cpu: 250m
memory: 128Mi
cpu: 500m
memory: 256Mi
securityContext:
allowPrivilegeEscalation: false
capabilities:
@@ -91,6 +138,9 @@ spec:
- name: trust
mountPath: /run/trust
readOnly: true
- name: spire-agent-socket
mountPath: /run/spire/agent-sockets
readOnly: true
securityContext:
fsGroup: 65532
fsGroupChangePolicy: OnRootMismatch
@@ -104,3 +154,7 @@ spec:
- name: trust
emptyDir:
sizeLimit: 1Mi
- name: spire-agent-socket
csi:
driver: csi.spiffe.io
readOnly: true
@@ -20,6 +20,10 @@ spec:
remoteRef:
key: k8s/nats
property: ci_worker_password
- secretKey: opensandbox-api-key
remoteRef:
key: k8s/opensandbox-api
property: api_key
- secretKey: webhook-secret
remoteRef:
key: k8s/dynamic-runner
@@ -28,3 +32,15 @@ spec:
remoteRef:
key: k8s/gitea-runner
property: token
- secretKey: scheduler-uuid
remoteRef:
key: k8s/dynamic-runner
property: scheduler_uuid
- secretKey: scheduler-token
remoteRef:
key: k8s/dynamic-runner
property: scheduler_token
- secretKey: facade-capability-key
remoteRef:
key: k8s/dynamic-runner
property: facade_capability_key
@@ -6,5 +6,4 @@ resources:
- rbac.yaml
- clusterspiffeid.yaml
- deployment.yaml
- pod-worker-deployment.yaml
- service.yaml
@@ -1,100 +0,0 @@
apiVersion: apps/v1
kind: Deployment
metadata:
name: dynamic-runner-pod-worker
namespace: dynamic-runner
spec:
replicas: 1
selector:
matchLabels:
app.kubernetes.io/name: dynamic-runner-pod-worker
template:
metadata:
labels:
app.kubernetes.io/name: dynamic-runner-pod-worker
spec:
serviceAccountName: dynamic-runner-pod-worker
initContainers:
- name: fetch-internal-ca
image: curlimages/curl:8.16.0@sha256:463eaf6072688fe96ac64fa623fe73e1dbe25d8ad6c34404a669ad3ce1f104b6
args:
- --fail
- --silent
- --show-error
- --output
- /trust/ca.pem
- https://bao.ad.ddupan.top:8200/v1/pki/ca/pem
securityContext:
allowPrivilegeEscalation: false
capabilities:
drop: [ALL]
readOnlyRootFilesystem: true
runAsNonRoot: true
runAsUser: 101
runAsGroup: 102
seccompProfile:
type: RuntimeDefault
volumeMounts:
- name: trust
mountPath: /trust
containers:
- name: pod-worker
image: gitea-dynamic-runner-controller:0.3.0-bootstrap
imagePullPolicy: Never
command: [/venv/bin/gitea-dynamic-runner-pod-worker]
env:
- name: NATS_URL
value: tls://nats.ad.ddupan.top:4222
- name: NATS_CA_FILE
value: /run/trust/ca.pem
- name: NATS_PASSWORD_FILE
value: /run/dynamic-runner-secrets/nats-worker-password
- name: RUNNER_NAMESPACE
valueFrom:
fieldRef:
fieldPath: metadata.namespace
- name: RUNNER_IMAGE
value: zot.ad.ddupan.top/panxiao81/gitea-dynamic-runner-runner@sha256:4c61f6315453d68a827ee9542f1345ed86576eaaf62325eb803c8fe3f06ddf2a
- name: RUNNER_SERVICE_ACCOUNT
value: gitea-dynamic-runner
- name: RUNNER_TOKEN_SECRET
value: dynamic-runner
- name: RUNNER_CAPACITY
value: "4"
resources:
requests:
cpu: 25m
memory: 32Mi
limits:
cpu: 250m
memory: 128Mi
securityContext:
allowPrivilegeEscalation: false
capabilities:
drop: [ALL]
readOnlyRootFilesystem: true
runAsNonRoot: true
runAsUser: 65532
runAsGroup: 65532
seccompProfile:
type: RuntimeDefault
volumeMounts:
- name: secret
mountPath: /run/dynamic-runner-secrets
readOnly: true
- name: trust
mountPath: /run/trust
readOnly: true
securityContext:
fsGroup: 65532
fsGroupChangePolicy: OnRootMismatch
seccompProfile:
type: RuntimeDefault
volumes:
- name: secret
secret:
secretName: dynamic-runner
defaultMode: 0400
- name: trust
emptyDir:
sizeLimit: 1Mi
+30 -11
View File
@@ -6,12 +6,6 @@ metadata:
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: dynamic-runner-pod-worker
namespace: dynamic-runner
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: gitea-dynamic-runner
namespace: dynamic-runner
@@ -19,23 +13,48 @@ metadata:
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
name: dynamic-runner-pod-worker
name: dynamic-runner-controller
namespace: dynamic-runner
rules:
- apiGroups: [""]
resources: [pods]
verbs: [create, get, patch, delete]
verbs: [create, get, list, watch, patch, delete]
- apiGroups: [coordination.k8s.io]
resources: [leases]
verbs: [create, get, list, watch, update, patch]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: dynamic-runner-pod-worker
name: dynamic-runner-controller
namespace: dynamic-runner
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: Role
name: dynamic-runner-pod-worker
name: dynamic-runner-controller
subjects:
- kind: ServiceAccount
name: dynamic-runner-pod-worker
name: dynamic-runner-controller
namespace: dynamic-runner
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: dynamic-runner-spiffe-entries
rules:
- apiGroups: [spire.spiffe.io]
resources: [clusterstaticentries]
verbs: [create, get, delete]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: dynamic-runner-spiffe-entries
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: dynamic-runner-spiffe-entries
subjects:
- kind: ServiceAccount
name: dynamic-runner-controller
namespace: dynamic-runner
+5 -3
View File
@@ -4,9 +4,11 @@ metadata:
name: dynamic-runner-controller
namespace: dynamic-runner
spec:
type: NodePort
selector:
app.kubernetes.io/name: dynamic-runner-controller
ports:
- name: http
port: 8787
targetPort: http
- name: facade
port: 8443
targetPort: facade
nodePort: 30443
+66
View File
@@ -0,0 +1,66 @@
# OpenSandbox Gitea runner Pools
本目录只部署零预热的 `ci-vm` Pool、sandbox 内的 SPIFFE identity controller,以及仅供
内网 VyOS 转发的 OpenSandbox NodePort。`ci-vm` 使用 `kata-clh-runtime-rs`,每个 VM
执行单一任务并在结束后删除。普通 Pod job 直接运行在 homelab,不进入 OpenSandbox。
Pool 中 task-executor 接收 Lifecycle API 下发的进程环境。guest-local SPIRE Agent 用
Pod-bound PSAT 向中央 SPIRE 注册;identity controller 从 BatchSandbox allocation
取得真实 Pod UID,再创建精确的 `ClusterStaticEntry`。runner 只有拿到请求中的完整
repository/task SVID 后才领取一次性 Gitea registration token。
这些 `ClusterStaticEntry` 位于 sandbox 集群,由 central SPIRE Server 内的
`spire-controller-manager-sandbox` 通过受限 external kubeconfig reconcile。必须在
`platform/spire/values.yaml` 显式启用 external controller-manager 的
`reconcile.clusterStaticEntries`(chart 默认关闭);仅看到 CR 存在但没有 status,不算
身份链路就绪。
## 部署依赖
- OpenSandbox chart 和 CRD 已 Ready;
- RuntimeClass `kata-clh-runtime-rs` 已存在;
- 中央 SPIRE 已发布 `ConfigMap/opensandbox/spire-bundle-pem`;
- VyOS `10.60.0.13:8080` 转发 sandbox1/2 的 NodePort `30080`;
- runner/controller 镜像均使用 Zot digest,而不是可变 tag。
本目录不读取 OpenBao,也不修改 OpenSandbox 平台侧 ExternalSecret、ClusterSecretStore
或 Bao policy。OpenSandbox API key 只存在于平台 server Secret 和 homelab controller
Secret,两边由各自身份读取同一 Bao 资源。
## 上线验收
先确认 Secret 和 API 认证,命令不得输出 key:
```bash
kubectl -n dynamic-runner wait externalsecret/dynamic-runner --for=condition=Ready --timeout=2m
kubectl -n dynamic-runner exec deploy/dynamic-runner-controller -- \
wget -qO- http://10.60.0.13:8080/health
```
然后触发一个 `runs-on: [self-hosted, vm]` 的最小 workflow,并同时观察:
```bash
kubectl -n dynamic-runner logs deploy/dynamic-runner-controller -f
kubectl -n opensandbox get pool,batchsandbox,pod -w
kubectl get clusterstaticentry -l app.kubernetes.io/component=opensandbox-identity -w
kubectl -n opensandbox logs deploy/opensandbox-identity -f
```
合格证据必须同时包含:
1. Lifecycle create 成功并选择 `ci-vm`;
2. 分配 Pod 的 `runtimeClassName` 为 `kata-clh-runtime-rs`;
3. entry 的 parent ID 包含该 Pod 的 UID,SPIFFE ID 使用 repository/task;
4. Gitea 显示临时 runner 带 `self-hosted,vm` labels 并完成真实任务;
5. 任务后 BatchSandbox、Pod、ClusterStaticEntry 和临时 runner 均消失。
清理超时对象时只调用 Lifecycle DELETE,不直接删除 Pool Pod。若 controller 已不可用,
可从 OpenSandbox API 按 metadata 定位 sandbox 后执行 DELETE;不要绕过 API 伪造状态。
## 故障定位
- `401`:检查 homelab ExternalSecret Ready 和文件挂载,不打印 Secret;
- `PoolCapacityExhausted`:检查 `ci-vm` 的 `poolMax` 及残留 BatchSandbox;
- runner 等待 SVID:核对 allocation Pod UID、ClusterStaticEntry parentID、guest Agent 日志;
- runner 等待 token:核对 `192.168.10.127:8787` 的 sandbox 到 homelab 路由;
- Docker 任务失败:检查 `docker` sidecar 和 `/run/docker/docker.sock` 的 group 2000。
@@ -0,0 +1,31 @@
apiVersion: v1
kind: ConfigMap
metadata:
name: gitea-ci-spire-agent
namespace: opensandbox
data:
agent.conf: |
agent {
data_dir = "/run/spire/data"
log_level = "INFO"
server_address = "spire-server.ad.ddupan.top"
server_port = "8081"
socket_path = "/run/spire/agent-sockets/spire-agent.sock"
trust_bundle_path = "/run/spire/bundle/bundle.pem"
trust_domain = "ddupan.top"
}
plugins {
NodeAttestor "k8s_psat" {
plugin_data {
cluster = "sandbox-kata"
token_path = "/run/spire/tokens/token"
}
}
KeyManager "memory" {
plugin_data {}
}
WorkloadAttestor "unix" {
plugin_data {}
}
}
@@ -0,0 +1,41 @@
apiVersion: apps/v1
kind: Deployment
metadata:
name: opensandbox-identity
namespace: opensandbox
spec:
replicas: 1
selector:
matchLabels:
app.kubernetes.io/name: opensandbox-identity
template:
metadata:
labels:
app.kubernetes.io/name: opensandbox-identity
spec:
serviceAccountName: opensandbox-identity
containers:
- name: controller
image: zot.ad.ddupan.top/panxiao81/gitea-dynamic-runner-controller@sha256:dfbfaf2a7aa5951d1dc4e7f941a8cd05ceb4a860ac1ab23eb8be234861edcb55
imagePullPolicy: IfNotPresent
command: [/venv/bin/gitea-dynamic-runner-opensandbox-identity]
resources:
requests:
cpu: 10m
memory: 32Mi
limits:
cpu: 100m
memory: 96Mi
securityContext:
allowPrivilegeEscalation: false
capabilities:
drop: [ALL]
readOnlyRootFilesystem: true
runAsNonRoot: true
runAsUser: 65532
runAsGroup: 65532
seccompProfile:
type: RuntimeDefault
securityContext:
seccompProfile:
type: RuntimeDefault
@@ -0,0 +1,22 @@
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- service.yaml
- agent-config.yaml
- rbac.yaml
- identity-controller.yaml
- pools.yaml
patches:
# Pod jobs run directly in the homelab cluster. Keep OpenSandbox VM-only.
- target:
group: sandbox.opensandbox.io
version: v1alpha1
kind: Pool
name: ci-pod
patch: |-
apiVersion: sandbox.opensandbox.io/v1alpha1
kind: Pool
metadata:
name: ci-pod
namespace: opensandbox
$patch: delete
+301
View File
@@ -0,0 +1,301 @@
---
apiVersion: sandbox.opensandbox.io/v1alpha1
kind: Pool
metadata:
name: ci-vm
namespace: opensandbox
labels:
app.kubernetes.io/name: gitea-ci-vm
app.kubernetes.io/component: runner-pool
spec:
capacitySpec:
bufferMax: 0
bufferMin: 0
poolMax: 2
poolMin: 0
recycleStrategy:
type: Delete
template:
metadata:
labels:
app.kubernetes.io/name: gitea-ci-vm
ci.ddupan.top/backend: vm
spec:
runtimeClassName: kata-clh-runtime-rs
serviceAccountName: gitea-ci
restartPolicy: Never
terminationGracePeriodSeconds: 30
shareProcessNamespace: true
securityContext:
fsGroup: 2000
fsGroupChangePolicy: OnRootMismatch
initContainers:
- name: task-executor-installer
image: sandbox-registry.cn-zhangjiakou.cr.aliyuncs.com/opensandbox/task-executor:v0.1.0
command: [/bin/sh, -c]
args:
- cp /workspace/server /opt/opensandbox/task-executor && chmod 0755 /opt/opensandbox/task-executor
volumeMounts:
- name: opensandbox-bin
mountPath: /opt/opensandbox
- name: execd-installer
image: sandbox-registry.cn-zhangjiakou.cr.aliyuncs.com/opensandbox/execd:v1.0.22
command: [/bin/sh, -c]
args:
- cp ./execd /opt/opensandbox/execd && cp ./bootstrap.sh /opt/opensandbox/bootstrap.sh && chmod 0755 /opt/opensandbox/execd /opt/opensandbox/bootstrap.sh
volumeMounts:
- name: opensandbox-bin
mountPath: /opt/opensandbox
containers:
- name: sandbox
image: zot.ad.ddupan.top/panxiao81/gitea-dynamic-runner-runner@sha256:08f83c9993645dce376140fdde7e29e809795166b2bb32e50cebdbefaf3cf303
imagePullPolicy: IfNotPresent
command: [/opt/opensandbox/task-executor]
args:
- -listen-addr=0.0.0.0:5758
- -log-dir=/tmp
- -data-dir=/tmp/tasks
env:
- name: SANDBOX_MAIN_CONTAINER
value: sandbox
- name: EXECD_ENVS
value: /opt/opensandbox/.env
- name: EXECD
value: /opt/opensandbox/execd
- name: GITEA_INSTANCE_URL
value: https://git.ddupan.top
- name: HOME
value: /data
- name: DOCKER_HOST
value: unix:///run/docker/docker.sock
ports:
- name: task-executor
containerPort: 5758
securityContext:
allowPrivilegeEscalation: false
capabilities:
drop: [ALL]
runAsNonRoot: true
runAsUser: 2000
runAsGroup: 2000
volumeMounts:
- name: opensandbox-bin
mountPath: /opt/opensandbox
- name: runner-data
mountPath: /data
- name: docker-socket
mountPath: /run/docker
- name: spire-socket
mountPath: /run/spire/agent-sockets
- name: spire-agent
image: ghcr.io/spiffe/spire-agent:1.15.3@sha256:41b0dcd8b258a69db9e2768292a060766fb76fd866e4bc925849981ea1b825ff
args: [-config, /run/spire/config/agent.conf]
securityContext:
allowPrivilegeEscalation: false
capabilities:
drop: [ALL]
volumeMounts:
- name: spire-config
mountPath: /run/spire/config
readOnly: true
- name: spire-bundle
mountPath: /run/spire/bundle
readOnly: true
- name: spire-token
mountPath: /run/spire/tokens
readOnly: true
- name: spire-data
mountPath: /run/spire/data
- name: spire-socket
mountPath: /run/spire/agent-sockets
- name: docker
image: docker.io/library/docker:29.1.5-dind
command: [/bin/sh, -c]
args:
- test -e /dev/kmsg || mknod /dev/kmsg c 1 11; exec dockerd --host=unix:///run/docker/docker.sock --group=2000 --storage-driver=overlay2
securityContext:
privileged: true
volumeMounts:
- name: docker-socket
mountPath: /run/docker
- name: docker-data
mountPath: /var/lib/docker
volumes:
- name: opensandbox-bin
emptyDir: {}
- name: runner-data
emptyDir: {}
- name: docker-data
emptyDir: {}
- name: docker-socket
emptyDir:
medium: Memory
- name: spire-data
emptyDir:
medium: Memory
- name: spire-socket
emptyDir:
medium: Memory
- name: spire-config
configMap:
name: gitea-ci-spire-agent
- name: spire-bundle
configMap:
name: spire-bundle-pem
- name: spire-token
projected:
sources:
- serviceAccountToken:
audience: spire-server
expirationSeconds: 3600
path: token
---
apiVersion: sandbox.opensandbox.io/v1alpha1
kind: Pool
metadata:
name: ci-pod
namespace: opensandbox
labels:
app.kubernetes.io/name: gitea-ci-pod
app.kubernetes.io/component: runner-pool
spec:
capacitySpec:
bufferMax: 0
bufferMin: 0
poolMax: 4
poolMin: 0
recycleStrategy:
type: Delete
template:
metadata:
labels:
app.kubernetes.io/name: gitea-ci-pod
ci.ddupan.top/backend: pod
spec:
serviceAccountName: gitea-ci
restartPolicy: Never
terminationGracePeriodSeconds: 30
shareProcessNamespace: true
securityContext:
fsGroup: 2000
fsGroupChangePolicy: OnRootMismatch
initContainers:
- name: task-executor-installer
image: sandbox-registry.cn-zhangjiakou.cr.aliyuncs.com/opensandbox/task-executor:v0.1.0
command: [/bin/sh, -c]
args:
- cp /workspace/server /opt/opensandbox/task-executor && chmod 0755 /opt/opensandbox/task-executor
volumeMounts:
- name: opensandbox-bin
mountPath: /opt/opensandbox
- name: execd-installer
image: sandbox-registry.cn-zhangjiakou.cr.aliyuncs.com/opensandbox/execd:v1.0.22
command: [/bin/sh, -c]
args:
- cp ./execd /opt/opensandbox/execd && cp ./bootstrap.sh /opt/opensandbox/bootstrap.sh && chmod 0755 /opt/opensandbox/execd /opt/opensandbox/bootstrap.sh
volumeMounts:
- name: opensandbox-bin
mountPath: /opt/opensandbox
containers:
- name: sandbox
image: zot.ad.ddupan.top/panxiao81/gitea-dynamic-runner-runner@sha256:a45875fd2d0e67429b0b7bc3914735581669c705bb4e1a05d712134d2bceb86f
imagePullPolicy: IfNotPresent
command: [/opt/opensandbox/task-executor]
args:
- -listen-addr=0.0.0.0:5758
- -log-dir=/tmp
- -data-dir=/tmp/tasks
env:
- name: SANDBOX_MAIN_CONTAINER
value: sandbox
- name: EXECD_ENVS
value: /opt/opensandbox/.env
- name: EXECD
value: /opt/opensandbox/execd
- name: GITEA_INSTANCE_URL
value: https://git.ddupan.top
- name: HOME
value: /data
- name: DOCKER_HOST
value: unix:///run/docker/docker.sock
ports:
- name: task-executor
containerPort: 5758
securityContext:
allowPrivilegeEscalation: false
capabilities:
drop: [ALL]
runAsNonRoot: true
runAsUser: 2000
runAsGroup: 2000
volumeMounts:
- name: opensandbox-bin
mountPath: /opt/opensandbox
- name: runner-data
mountPath: /data
- name: docker-socket
mountPath: /run/docker
- name: spire-socket
mountPath: /run/spire/agent-sockets
- name: spire-agent
image: ghcr.io/spiffe/spire-agent:1.15.3@sha256:41b0dcd8b258a69db9e2768292a060766fb76fd866e4bc925849981ea1b825ff
args: [-config, /run/spire/config/agent.conf]
securityContext:
allowPrivilegeEscalation: false
capabilities:
drop: [ALL]
volumeMounts:
- name: spire-config
mountPath: /run/spire/config
readOnly: true
- name: spire-bundle
mountPath: /run/spire/bundle
readOnly: true
- name: spire-token
mountPath: /run/spire/tokens
readOnly: true
- name: spire-data
mountPath: /run/spire/data
- name: spire-socket
mountPath: /run/spire/agent-sockets
- name: docker
image: docker.io/library/docker:29.1.5-dind
command: [/bin/sh, -c]
args:
- test -e /dev/kmsg || mknod /dev/kmsg c 1 11; exec dockerd --host=unix:///run/docker/docker.sock --group=2000 --storage-driver=overlay2
securityContext:
privileged: true
volumeMounts:
- name: docker-socket
mountPath: /run/docker
- name: docker-data
mountPath: /var/lib/docker
volumes:
- name: opensandbox-bin
emptyDir: {}
- name: runner-data
emptyDir: {}
- name: docker-data
emptyDir: {}
- name: docker-socket
emptyDir:
medium: Memory
- name: spire-data
emptyDir:
medium: Memory
- name: spire-socket
emptyDir:
medium: Memory
- name: spire-config
configMap:
name: gitea-ci-spire-agent
- name: spire-bundle
configMap:
name: spire-bundle-pem
- name: spire-token
projected:
sources:
- serviceAccountToken:
audience: spire-server
expirationSeconds: 3600
path: token
+87
View File
@@ -0,0 +1,87 @@
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: gitea-ci
namespace: opensandbox
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: opensandbox-identity
namespace: opensandbox
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
name: opensandbox-identity
namespace: opensandbox
rules:
- apiGroups: [""]
resources: [pods]
verbs: [get, list, watch]
- apiGroups: [sandbox.opensandbox.io]
resources: [batchsandboxes]
verbs: [get, list, watch]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: opensandbox-identity
namespace: opensandbox
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: Role
name: opensandbox-identity
subjects:
- kind: ServiceAccount
name: opensandbox-identity
namespace: opensandbox
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: opensandbox-identity
rules:
- apiGroups: [spire.spiffe.io]
resources: [clusterstaticentries]
verbs: [create, delete, get, list, watch]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: opensandbox-identity
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: opensandbox-identity
subjects:
- kind: ServiceAccount
name: opensandbox-identity
namespace: opensandbox
---
# The central SPIRE external controller uses this existing sandbox credential
# to publish a PEM bundle for guest-local Agents. It cannot read Secrets here.
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
name: spire-runner-bundle-publisher
namespace: opensandbox
rules:
- apiGroups: [""]
resources: [configmaps]
verbs: [create, delete, get, list, patch, update, watch]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: spire-runner-bundle-publisher
namespace: opensandbox
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: Role
name: spire-runner-bundle-publisher
subjects:
- kind: ServiceAccount
name: spire-controller-manager
namespace: spire-system
+15
View File
@@ -0,0 +1,15 @@
apiVersion: v1
kind: Service
metadata:
name: opensandbox-server-internal
namespace: opensandbox-system
spec:
type: NodePort
selector:
app.kubernetes.io/instance: opensandbox
app.kubernetes.io/name: opensandbox-server
ports:
- name: http
port: 80
targetPort: http
nodePort: 30080
@@ -0,0 +1,56 @@
# Sandbox External Secrets Operator
本目录在 sandbox 集群部署独立的 External Secrets Operator `2.8.0`,并通过
`ClusterSecretStore/openbao` 读取 OpenBao KV v2 中共享的 OpenSandbox API key。它不复用
homelab 集群的 ESO Pod、ServiceAccount 或 Kubernetes auth backend。
## 当前状态
- OpenBao `auth/kubernetes-sandbox`、backend config、`external-secrets` role 与
`sandbox-external-secrets` policy 已于 2026-09-18 由 Terraform 创建;apply 后 plan
为 zero-diff;
- `kv/k8s/opensandbox-api` 已由本机 `spiffe://ddupan.top/dev/panxiao81` 身份生成并写入,
值未输出或落盘;
- sandbox ESO operator、`ClusterSecretStore/openbao` 与 OpenSandbox `ExternalSecret`
由 Flux 管理;
- 线上 `ClusterSecretStore/openbao` 为 `Valid/Ready`,`ExternalSecret/opensandbox-api-key`
为 `SecretSynced/Ready`;
- OpenSandbox 已切换到 API key:无 key 请求返回 `401`,正确 key 请求返回 `200`;
- homelab runner 对同一 key 的投影不在本目录,留给 runner 项目管理。
OpenBao 的 `auth/kubernetes-sandbox`、对应 role、policy、sandbox API 地址与公开 CA
完全由 `infrastructure/openbao/terraform/` 管理。CA 文件提交到 Git 是刻意设计:CA
是公开信任材料,版本化后集群重建造成的 trust root 变化会产生可审计的 Terraform diff。
ESO 使用 TokenRequest 生成短期 ServiceAccount JWT。OpenBao 未配置长期
`token_reviewer_jwt`,而是使用登录 JWT 调用 sandbox TokenReview;因此
`external-secrets` ServiceAccount 仅额外绑定内建 `system:auth-delegator`。
## 重建顺序
1. 在 OpenBao 写入 OpenSandbox API key:
```bash
openssl rand -hex 32 | bao kv put kv/k8s/opensandbox-api api_key=-
```
2. 在 `infrastructure/openbao/terraform` 执行 `terraform plan` 和 `terraform apply`,
创建 `kubernetes-sandbox` auth mount、backend config、role 与只允许读取
`kv/k8s/opensandbox-api` 的最小权限 policy。
3. 合并 Flux 变更。依次等待 `flux-system/external-secrets-operator`、
`flux-system/external-secrets` Ready,再等待 `flux-system/opensandbox` 滚动完成。
operator 与配置拆成两个 Flux Kustomization,确保全新集群先安装 CRD,再声明
`ClusterSecretStore`;不要为了减少目录而把两层重新合并。
## 验收
```bash
kubectl get clustersecretstore openbao
kubectl -n opensandbox-system get externalsecret opensandbox-api-key
kubectl -n opensandbox-system get secret opensandbox-api-key
```
只检查 Secret 是否存在及 key 名,不输出 `data`。`ClusterSecretStore` 或
`ExternalSecret` 不 Ready 时,先检查 `auth/kubernetes-sandbox`,不要临时创建静态
Bao token Secret。
@@ -0,0 +1,13 @@
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: sandbox-external-secrets-token-review
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: system:auth-delegator
subjects:
- kind: ServiceAccount
name: external-secrets
namespace: external-secrets
@@ -0,0 +1,18 @@
---
apiVersion: external-secrets.io/v1
kind: ClusterSecretStore
metadata:
name: openbao
spec:
provider:
vault:
server: https://bao.ad.ddupan.top:8200
path: kv
version: v2
auth:
kubernetes:
mountPath: kubernetes-sandbox
role: external-secrets
serviceAccountRef:
name: external-secrets
namespace: external-secrets
@@ -0,0 +1,6 @@
---
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- auth-delegator.yaml
- clustersecretstore.yaml
@@ -0,0 +1,35 @@
---
apiVersion: helm.toolkit.fluxcd.io/v2
kind: HelmRelease
metadata:
name: external-secrets
namespace: external-secrets
spec:
chart:
spec:
chart: external-secrets
interval: 1h
sourceRef:
kind: HelmRepository
name: external-secrets
version: 2.8.0
driftDetection:
mode: enabled
install:
crds: CreateReplace
strategy:
name: RetryOnFailure
retryInterval: 5m
interval: 30m
releaseName: external-secrets
targetNamespace: external-secrets
timeout: 10m
upgrade:
crds: CreateReplace
strategy:
name: RetryOnFailure
retryInterval: 5m
valuesFrom:
- kind: ConfigMap
name: external-secrets-values
valuesKey: values.yaml
@@ -0,0 +1,8 @@
---
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- namespace.yaml
- repository.yaml
- values.yaml
- helmrelease.yaml
@@ -0,0 +1,5 @@
---
apiVersion: v1
kind: Namespace
metadata:
name: external-secrets
@@ -0,0 +1,9 @@
---
apiVersion: source.toolkit.fluxcd.io/v1
kind: HelmRepository
metadata:
name: external-secrets
namespace: external-secrets
spec:
interval: 1h
url: https://charts.external-secrets.io
@@ -0,0 +1,31 @@
---
apiVersion: v1
kind: ConfigMap
metadata:
name: external-secrets-values
namespace: external-secrets
data:
values.yaml: |
replicaCount: 1
webhook:
replicaCount: 1
resources:
requests: {cpu: 10m, memory: 32Mi}
limits: {memory: 128Mi}
certController:
replicaCount: 1
resources:
requests: {cpu: 10m, memory: 32Mi}
limits: {memory: 128Mi}
resources:
requests: {cpu: 10m, memory: 64Mi}
limits: {memory: 256Mi}
serviceAccount:
create: true
name: external-secrets
installCRDs: true
+28 -2
View File
@@ -28,6 +28,32 @@ Pod 删除后的 backing-file/VMM 回收和真实构建基准必须作为上线
由 `VMPodScrape` 写入中央 VictoriaMetrics。它不拥有 Kubernetes API 凭据或 host 写
权限。
## Guest 内 SPIFFE 身份
Kata guest 不能直接使用 node SPIRE Agent 的 CSI socket。Unix socket 的路径即使通过
virtio-fs 出现在 guest 中,连接也不能跨 VM 边界。CI Pod 应以 native sidecar 在同一
guest 内启动临时 SPIRE Agent,并满足以下约束:
1. 外层 Pod 挂载 `audience=spire-server` 的 Pod-bound projected ServiceAccount token;
2. 内层 Agent 使用 `k8s_psat` 向中央 Server attestation,Server cluster profile 必须
启用 `use_pod_uid_for_agent_id`,使 Agent ID 包含 Pod UID;
3. 调度器为该具体 Agent 创建 registration entry;SPIFFE ID 使用仓库和任务名等稳定的
业务语义,Pod UID 只作为 parent binding,不进入业务身份;
4. Agent 与 workload 通过 guest 内 `emptyDir` 上的 Unix socket 通信;Pod 必须设置
`shareProcessNamespace: true`,否则 `unix` workload attestor 无法从共享 `/proc`
解析客户端的 `SO_PEERCRED` PID;
5. 调度器必须在 registration entry 已同步到 Agent 后才放行 workload。Pod 删除后同步
删除 entry,并 evict 对应的临时 Agent 记录。
2026-09-17 的 live PoC 已验证:以 Pod UID 作为 parent 的临时 Agent 成功注册,
`unix:uid:2000` workload 从 guest-local socket 获得
`spiffe://ddupan.top/ci/poc/task/build` X.509-SVID,Pod 正常退出。PoC 资源随后全部清理,
中央 SPIRE 恢复声明式配置。
SPIRE 1.15.3 已支持 `use_pod_uid_for_agent_id`,但当前使用的 hardened chart 0.30.2
尚未把它暴露到 values/template。正式部署应先给 chart 补齐该字段并向上游提交,随后
采用包含修复的 release;不得把手工修改 Server ConfigMap 作为运行方案。
## 上线验收
Flux reconciliation 完成后至少确认:
@@ -35,8 +61,8 @@ Flux reconciliation 完成后至少确认:
1. 两个节点重新回到 Ready,`RuntimeClass/kata-clh-runtime-rs` 存在;
2. Kata Pod 内核与 LXC host 内核不同,且 `/dev/kvm` 可用;
3. Cloud Hypervisor API 的 `vm.info.config.memory.shared` 为 `true`;
4. `spire-smoke` ServiceAccount 的 Kata Pod 可获得
`spiffe://ddupan.top/sandbox/smoke`,错误 ServiceAccount 无法获得身份;
4. Kata Pod 内层 Agent 的 ID 包含该 Pod UID;只有调度器创建的业务 entry 所匹配的
workload UID 能从 guest-local socket 获得预期 SPIFFE ID;
5. block-backed `emptyDir` 上 Docker 使用 `overlay2`,BuildKit 与 kind smoke test
通过;kind 的 dockerd bootstrap 需要先在 guest 内执行
`mknod /dev/kmsg c 1 11`;
+44
View File
@@ -0,0 +1,44 @@
# Sandbox OpenSandbox
本目录在独立 sandbox k3s 集群部署 OpenSandbox controller、server 与 CRD。Flux 从
上游 commit `8f01e935c2cabba778cf37a152033fae062fa0f4` 构建官方 umbrella chart
`0.2.2`;该源码渲染结果已与 release `opensandbox-0.2.2.tgz` 对比一致。不要改为跟随
浮动 branch 或 tag。
server 只提供集群内 `opensandbox-server.opensandbox-system.svc:80` ClusterIP,不部署
Gateway、Ingress 或 LoadBalancer。sandbox workload 位于 `opensandbox` namespace,
默认使用 `kata-clh-runtime-rs`;CI Pool、runner 镜像、动态 SPIFFE registration 均由
runner 项目后续声明,本目录不预制。
## API 认证
`ExternalSecret/opensandbox-api-key` 从 OpenBao
`kv/k8s/opensandbox-api:api_key` 投影同名 Secret。这个路径不归属于某个 Kubernetes
集群:sandbox server 与 homelab runner 调度器分别通过自己的 Bao 身份读取。server 只通过
`secretKeyRef` 读取:
```yaml
- name: OPENSANDBOX_SERVER_API_KEY
valueFrom:
secretKeyRef:
name: opensandbox-api-key
key: api-key
```
仓库与 Helm values 均不保存 API key。OpenSandbox 不支持更丰富的原生 workload
authentication;runner 后续读取同一 Bao 路径并在请求头中使用 API key。
`opensandbox-values` 带 Flux watch label,values 变化会立即触发 Helm reconcile,不依赖
30 分钟的 HelmRelease interval。
## 验收
合并后等待 `flux-system/opensandbox` 与 `opensandbox-system/opensandbox` Ready,并确认:
```bash
kubectl get crd batchsandboxes.sandbox.opensandbox.io pools.sandbox.opensandbox.io
kubectl -n opensandbox-system get deploy,pod,svc
kubectl get runtimeclass kata-clh-runtime-rs
```
控制面上线不创建 CI Pool,也不产生 sandbox workload。首个 runner 集成应另行提交 Pool
与完整的 Lifecycle API smoke test。
@@ -0,0 +1,19 @@
---
apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
name: opensandbox-api-key
namespace: opensandbox-system
spec:
refreshInterval: 1h
secretStoreRef:
kind: ClusterSecretStore
name: openbao
target:
name: opensandbox-api-key
creationPolicy: Owner
data:
- secretKey: api-key
remoteRef:
key: k8s/opensandbox-api
property: api_key
@@ -0,0 +1,35 @@
---
apiVersion: helm.toolkit.fluxcd.io/v2
kind: HelmRelease
metadata:
name: opensandbox
namespace: opensandbox-system
spec:
chart:
spec:
chart: ./kubernetes/charts/opensandbox
interval: 1h
reconcileStrategy: Revision
sourceRef:
kind: GitRepository
name: opensandbox
driftDetection:
mode: enabled
install:
crds: CreateReplace
strategy:
name: RetryOnFailure
retryInterval: 5m
interval: 30m
releaseName: opensandbox
targetNamespace: opensandbox-system
timeout: 15m
upgrade:
crds: CreateReplace
strategy:
name: RetryOnFailure
retryInterval: 5m
valuesFrom:
- kind: ConfigMap
name: opensandbox-values
valuesKey: values.yaml
@@ -0,0 +1,11 @@
---
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- namespace.yaml
- external-secret.yaml
- repository.yaml
- template.yaml
- values.yaml
- helmrelease.yaml
- monitor-scrape.yaml
@@ -0,0 +1,16 @@
---
apiVersion: operator.victoriametrics.com/v1beta1
kind: VMPodScrape
metadata:
name: opensandbox-controller
namespace: monitoring
spec:
namespaceSelector:
matchNames:
- opensandbox-system
podMetricsEndpoints:
- interval: 30s
port: metrics
selector:
matchLabels:
app.kubernetes.io/name: opensandbox
@@ -0,0 +1,14 @@
---
apiVersion: v1
kind: Namespace
metadata:
name: opensandbox-system
---
apiVersion: v1
kind: Namespace
metadata:
name: opensandbox
labels:
pod-security.kubernetes.io/enforce: privileged
pod-security.kubernetes.io/audit: restricted
pod-security.kubernetes.io/warn: restricted
@@ -0,0 +1,11 @@
---
apiVersion: source.toolkit.fluxcd.io/v1
kind: GitRepository
metadata:
name: opensandbox
namespace: opensandbox-system
spec:
interval: 1h
ref:
commit: 8f01e935c2cabba778cf37a152033fae062fa0f4
url: https://github.com/opensandbox-group/OpenSandbox.git
@@ -0,0 +1,17 @@
---
apiVersion: v1
kind: ConfigMap
metadata:
name: opensandbox-batchsandbox-template
namespace: opensandbox-system
data:
batchsandbox-template.yaml: |
metadata:
labels:
ddupan.top/workload-class: sandbox
spec:
replicas: 1
template:
spec:
restartPolicy: Never
terminationGracePeriodSeconds: 30
+83
View File
@@ -0,0 +1,83 @@
---
apiVersion: v1
kind: ConfigMap
metadata:
name: opensandbox-values
namespace: opensandbox-system
labels:
reconcile.fluxcd.io/watch: Enabled
data:
values.yaml: |
opensandbox-controller:
controller:
metrics:
enabled: true
port: 8080
secure: false
opensandbox-server:
server:
replicaCount: 1
env:
- name: OPENSANDBOX_SERVER_API_KEY
valueFrom:
secretKeyRef:
name: opensandbox-api-key
key: api-key
resources:
limits:
cpu: "1"
memory: 1Gi
requests:
cpu: 100m
memory: 256Mi
volumeMounts:
- name: batchsandbox-template
mountPath: /etc/opensandbox/batchsandbox-template.yaml
subPath: batchsandbox-template.yaml
readOnly: true
volumes:
- name: batchsandbox-template
configMap:
name: opensandbox-batchsandbox-template
configToml: |
[server]
host = "0.0.0.0"
port = 80
api_key = ""
max_sandbox_timeout_seconds = 86400
[log]
level = "INFO"
[runtime]
type = "kubernetes"
execd_image = "sandbox-registry.cn-zhangjiakou.cr.aliyuncs.com/opensandbox/execd:v1.0.22"
[storage]
allowed_host_paths = []
volume_default_size = "1Gi"
[kubernetes]
kubeconfig_path = ""
namespace = "opensandbox"
informer_enabled = true
informer_resync_seconds = 300
informer_watch_timeout_seconds = 60
snapshot_create_timeout_seconds = 900
workload_provider = "batchsandbox"
image_pull_policy = "IfNotPresent"
batchsandbox_template_file = "/etc/opensandbox/batchsandbox-template.yaml"
[egress]
image = "sandbox-registry.cn-zhangjiakou.cr.aliyuncs.com/opensandbox/egress:v1.1.6"
mode = "dns+nft"
disable_ipv6 = true
[secure_runtime]
type = "kata"
k8s_runtime_class = "kata-clh-runtime-rs"
opensandbox-node-agent:
enabled: false
+6
View File
@@ -74,6 +74,12 @@ Agent 已启用 Unix workload attestor,并为本机用户 `panxiao81`(UID `1
export SPIFFE_ENDPOINT_SOCKET=unix:///run/spire/agent-sockets/spire-agent.sock
```
宿主机已安装与 Agent Pod 同版本的 `/usr/local/bin/spire-agent` 1.15.3,供本地进程从
Workload API 获取 JWT-SVID。二进制来自 SPIRE 官方 `linux-amd64-musl` release,安装时
核对 tarball SHA-256
`ca1a4d1155317bdd2afc7f36663828a10410c7c840e54725b90b4064b0a301c7`。升级 chart 时应
同步升级这个 CLI 并重新核对官方 checksum,不能长期混用版本。
该身份仅按 Unix UID 匹配,不是 SPIRE admin,也不会匹配 `sudo` 后以 root 运行的
进程。`ClusterStaticEntry.spec.parentID` 绑定当前 `laptop` Kubernetes node UID;若
节点被删除后重建,需从 `spire-server agent list` 取得新 Agent ID 并同步更新该字段。
+12 -2
View File
@@ -274,14 +274,24 @@ CI job/Agent 不应接收长期 `BAO_TOKEN`。标准启动顺序是:
5. 在同一进程树中以环境变量调用 `tofu`、Ansible 或其他工具;
6. cleanup 尝试 `revoke-self`,随后销毁 job/VM/容器。
通用 credential-exec 包装器未来应负责步骤 3–6。它必须满足:
Gitea Actions 可使用
[`panxiao81/ci-actions/spiffe-openbao-login@v1`](https://git.ddupan.top/panxiao81/ci-actions/src/tag/v1/spiffe-openbao-login)
完成步骤 3、4 和 cleanup 时的 `revoke-self`。runner 只负责提供 Node.js 20、
`spire-agent` 与 Workload API socket;role、audience 及是否登录由 workflow 明确声明,
目标侧 Bao policy 仍负责最终授权。该 Action 必须满足:
- 不把 JWT-SVID 或 Bao token 写到 stdout/stderr;
- 不把凭据传入命令行参数,避免出现在进程列表;
- 子进程退出后清除环境和临时文件;
- job 结束后销毁 runner 及其 Actions 临时文件;
- 不尝试把短期 token 上传到 Actions Secret 或 artifact;
- role、audience 和目标命令由受审查的 pipeline 配置决定。
Action 会按 GitHub Actions 协议把短期 token 写入 `GITHUB_ENV` 和 `GITHUB_STATE`,因此
只允许用于一次性 Pod/VM runner,不能用于共享或持久 runner。只读取 Nexus public
repository 时不需要 Bao 登录,可直接使用
[`panxiao81/ci-actions/setup-nexus@v1`](https://git.ddupan.top/panxiao81/ci-actions/src/tag/v1/setup-nexus)
配置 Ansible Galaxy、Go module proxy 与 OCI endpoint。
Kubernetes 以外的执行环境不能伪造 ServiceAccount。未来应分别使用 host SPIRE
Agent、TPM/DevID、cloud instance identity、GitHub OIDC 等初始证明接入同一信任模型。
+17
View File
@@ -43,6 +43,10 @@ spire-server:
externalSecret:
name: spire-external-kubeconfigs
key: sandbox-controller
sandbox-runner-bundle:
externalSecret:
name: spire-external-kubeconfigs
key: sandbox-controller
nodeAttestor:
externalK8sPSAT:
enabled: true
@@ -55,12 +59,19 @@ spire-server:
kubeConfigName: sandbox
serviceAccountAllowList:
- spire-smoke:spire-smoke
- opensandbox:gitea-ci
usePodUIDForAgentID: true
externalControllerManagers:
enabled: true
clusters:
sandbox:
kubeConfigName: sandbox-controller
# Dynamic OpenSandbox runner identities are exact Pod-UID-bound
# ClusterStaticEntries created in the sandbox cluster. The chart
# defaults this reconciler to false, so enable the central registration
# path explicitly.
reconcile:
clusterStaticEntries: true
bundlePublisher:
externalK8sConfigMap:
enabled: true
@@ -71,6 +82,12 @@ spire-server:
configMapName: spire-bundle
configMapKey: bundle.spiffe
format: spiffe
sandbox-runner-bundle:
kubeConfigName: sandbox-runner-bundle
namespace: opensandbox
configMapName: spire-bundle-pem
configMapKey: bundle.pem
format: pem
persistence:
# PostgreSQL stores registrations, but the disk KeyManager still needs durable
# storage for the trust-domain signing keys.