panxiao81/homelab-infra
1
0
Fork 0
Code Issues 10 Pull Requests Actions Packages Projects Releases Wiki Activity

Compare commits

base: panxiao81/homelab-infra:ab985c797913a6a627f734fafec93f45ffd60a2b
Branches Tags
panxiao81/homelab-infra:main panxiao81/homelab-infra:feat/ayatori-database-environments panxiao81/homelab-infra:feat/monitoring-next-batch panxiao81/homelab-infra:feat/opensandbox-ci-runner panxiao81/homelab-infra:poc/kind-on-kata
..
compare: panxiao81/homelab-infra:feat/opensandbox-ci-runner
Branches Tags
panxiao81/homelab-infra:main panxiao81/homelab-infra:feat/ayatori-database-environments panxiao81/homelab-infra:feat/monitoring-next-batch panxiao81/homelab-infra:feat/opensandbox-ci-runner panxiao81/homelab-infra:poc/kind-on-kata
1 Commits
ab985c7979 .. feat/opensandbox-ci-runner
Author SHA1 Message Date
panxiao81 0e3564bf72 feat: 部署 OpenSandbox API 与双运行时 Pool
yaml / yaml (pull_request) Successful in 20s
Details
2026-09-18 01:10:07 +00:00
69 changed files with 227 additions and 1664 deletions
Expand all files Collapse all files
apps/nexus/README.md
-135
View File
@@ -1,135 +0,0 @@
# Nexus Repository POC
本目录声明一个 Nexus Repository Community Edition POC，用来验证一次性 CI runner 通过
网络服务复用 Ansible Galaxy、Go Modules 与 OCI/BuildKit 缓存。Nexus 固定为 `3.96.1`，
镜像固定到官方 multi-arch index digest；LAN 入口为
`https://nexus.ad.ddupan.top`。
## POC 边界
- 单副本 Deployment，`Recreate` 更新，避免一个 RWO 卷被两个 Pod 同时挂载。
- `/nexus-data` 使用 `localpv-zfs-ceph` 上的 50 GiB RWO PVC。
- 资源预算为 250m/2 GiB request、2 CPU/4 GiB limit；JVM heap 上限 2 GiB。
- 使用容器默认的 embedded H2。它只用于 POC；正式接管 OCI 制品前必须迁移到外部
PostgreSQL，并验证备份恢复。
- 入口只在 LAN wildcard Gateway 上发布，不创建公网 DNS 或 Cloudflare route。
- 不套 Authelia forward-auth；它会破坏 Go、Ansible 与 OCI 非浏览器客户端。
- 现有 zot 保持不变。Nexus 完成 OCI、BuildKit cache 和恢复验收前不得迁移或删除 zot。
POC 先由 Terraform 创建 `ansible-public` 和 `go-public`。Nexus 3.94 才加入的原生 OCI
repository 已有 REST API，但当前锁定的 community provider 尚未暴露 OCI resource；不能
因此在 UI 中创建一个无人管理的长期 repository。POC 上线后先根据实例 Swagger 固定
请求 schema，再补 Terraform provider resource 或受测试的声明式 REST 适配。
## 部署
Flux 从 `clusters/homelab/apps/nexus.yaml` 协调本目录，并依赖 Envoy Gateway 与 OpenEBS。
合并前只渲染配置，不直接 apply：
```bash
kubectl kustomize apps/nexus
```
合并并由 Flux 部署后检查：
```bash
kubectl -n flux-system get kustomization nexus
kubectl -n nexus get pod,pvc,service,httproute
kubectl -n nexus logs deployment/nexus --tail=100
```
启动可能需要数分钟，startup probe 允许最多十分钟。不要因初次启动较慢反复删除 Pod；
先确认 PVC 已 Bound、Pod 没有 OOM，以及日志仍在推进。
DNS 期望状态已加入 `infrastructure/dns/records.yml`，需从 Samba AD Ansible root 以
`--check --diff` 核对后再按其 README 应用 DNS tag。没有 DNS 时可先用 port-forward
验证应用，但不能据此宣称 Gateway 路径已通过。
## 首次初始化与 Terraform
初始管理员密码生成在 PVC 的 `/nexus-data/admin.password`。只在交互式终端中读取并立即
完成首次密码轮换；不得把密码复制进 shell tracing、工单、Git 或命令参数。随后将
Terraform 管理账号的凭据存入 OpenBao，由 CI 通过 Terraform input variable 注入以下
环境变量：
```text
TF_VAR_nexus_url=https://nexus.ad.ddupan.top
TF_VAR_nexus_username=admin
TF_VAR_nexus_password=<OpenBao kv/infra/nexus 的 admin_password 字段>
```
`terraform/` 使用 `sonatype-nexus-community/sonatyperepo` 1.17.0，当前声明：
- `ansible-galaxy-proxy` → `https://galaxy.ansible.com`
- `ansible-public` group
- `go-proxy` → `https://proxy.golang.org`
- `go-public` group
provider credential 不写入 HCL 或 tfvars。正式 apply 前还必须为这个独立 Terraform root
配置远端 backend；本地 state 只允许用于可丢弃的 POC，不提交。验证命令：
```bash
terraform -chdir=apps/nexus/terraform init -backend=false
terraform -chdir=apps/nexus/terraform validate
```
Terraform 同时把内置 `anonymous` 用户从默认的全仓库 `nx-anonymous` 角色收窄到
`ansible-public`、其返回制品 URL 使用的 `ansible-galaxy-proxy`，以及 `go-public` 的
`browse/read` 权限。首次接管已有实例时先执行
`terraform import sonatyperepo_user.anonymous anonymous,default`，再 apply；不要先启用默认的
全仓库匿名读取。
2026-09-20 的 POC 现场验收已确认：Flux 与 Pod Ready、PVC Bound、HTTPRoute 通过 HTTPS
返回 Nexus 状态 200，Samba DNS 已幂等收敛。全新客户端目录通过匿名入口下载
`community.general:11.2.0` 时冷缓存为 8.49 秒、热缓存为 1.89 秒，两次 tarball SHA-256
一致；`golang.org/x/[email protected]` 为 2.92 秒与 1.51 秒。该结果只证明 Ansible/Go
代理链路，不代表 OCI、BuildKit cache、备份恢复或正式数据库已经验收。
## 客户端验收
先验证冷缓存，再原样重复命令验证热缓存；记录 Nexus 请求、上游流量和耗时，不只观察
命令成功。Ansible 配置的 URL 必须以 `/` 结尾：
```ini
[galaxy]
server_list = nexus
[galaxy_server.nexus]
url = https://nexus.ad.ddupan.top/repository/ansible-public/
```
```bash
ansible-galaxy collection install -r collections/requirements.yml \
-p .ansible/collections
```
Go POC 使用：
```bash
GOPROXY=https://nexus.ad.ddupan.top/repository/go-public/ go mod download
```
私有 `git.ddupan.top/*` module 的 `GOPRIVATE`、凭据与是否允许 `direct` fallback 在实际
workflow 中单独决定；不要让私有 module path 意外发往公共 proxy。
OCI 阶段至少验证普通 push/pull、multi-arch、OCI 1.1 referrers、Cosign、Helm OCI，
以及 BuildKit `registry` cache 的 `mode=max` 导入导出。Nexus 上游已有 BuildKit cache
兼容性问题记录，因此该项失败时不能用普通镜像 push 成功代替。全部通过并完成备份恢复
演练后，才评估 zot 迁移。
## 数据与恢复
POC 的数据库、配置、blob、初始管理员状态都位于 `nexus-data` PVC。删除 Deployment
不会删除 PVC；删除 PVC 会永久删除整个 POC。当前没有独立备份，不能将它用于唯一副本的
正式制品。
恢复验证至少包括：停止写入、取得一致备份、在独立 PVC/实例恢复、登录、列出 repository、
拉取已缓存的 Ansible/Go 制品，并核对 OCI digest/referrers。正式化时再把数据库迁移至
外部 PostgreSQL，并分别定义数据库与 blob 的备份、恢复顺序和 RPO。
参考：
- [Nexus OCI repositories](https://help.sonatype.com/en/oci-repositories.html)
- [Nexus Ansible repositories](https://help.sonatype.com/en/ansible-repositories.html)
- [Nexus Go repositories](https://help.sonatype.com/en/go-repositories.html)
- [官方容器镜像](https://hub.docker.com/r/sonatype/nexus3)
apps/nexus/deployment.yaml
-82
View File
@@ -1,82 +0,0 @@
apiVersion: apps/v1
kind: Deployment
metadata:
name: nexus
namespace: nexus
labels:
app.kubernetes.io/name: nexus
spec:
replicas: 1
strategy:
type: Recreate
selector:
matchLabels:
app.kubernetes.io/name: nexus
template:
metadata:
labels:
app.kubernetes.io/name: nexus
spec:
automountServiceAccountToken: false
securityContext:
fsGroup: 200
fsGroupChangePolicy: OnRootMismatch
runAsGroup: 200
runAsNonRoot: true
runAsUser: 200
seccompProfile:
type: RuntimeDefault
terminationGracePeriodSeconds: 120
containers:
- name: nexus
image: docker.io/sonatype/nexus3:3.96.1@sha256:56142f13432cf072e017aebb2025f201e42ae36ff40bb82618c702504c61f7dd
imagePullPolicy: IfNotPresent
env:
- name: INSTALL4J_ADD_VM_PARAMS
value: >-
-Xms1024m -Xmx2048m -XX:MaxDirectMemorySize=1024m
-Djava.util.prefs.userRoot=/nexus-data/javaprefs
ports:
- name: http
containerPort: 8081
protocol: TCP
resources:
requests:
cpu: 250m
memory: 2Gi
limits:
cpu: "2"
memory: 4Gi
securityContext:
allowPrivilegeEscalation: false
capabilities:
drop:
- ALL
startupProbe:
httpGet:
path: /service/rest/v1/status
port: http
failureThreshold: 60
periodSeconds: 10
timeoutSeconds: 5
readinessProbe:
httpGet:
path: /service/rest/v1/status
port: http
failureThreshold: 6
periodSeconds: 10
timeoutSeconds: 5
livenessProbe:
httpGet:
path: /service/rest/v1/status
port: http
failureThreshold: 6
periodSeconds: 30
timeoutSeconds: 5
volumeMounts:
- name: data
mountPath: /nexus-data
volumes:
- name: data
persistentVolumeClaim:
claimName: nexus-data
apps/nexus/httproute.yaml
-16
View File
@@ -1,16 +0,0 @@
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
name: nexus
namespace: nexus
spec:
parentRefs:
- name: eg
namespace: envoy-gateway-system
sectionName: https
hostnames:
- nexus.ad.ddupan.top
rules:
- backendRefs:
- name: nexus
port: 8081
apps/nexus/kustomization.yaml
-9
View File
@@ -1,9 +0,0 @@
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- namespace.yaml
- pvc.yaml
- deployment.yaml
- service.yaml
- httproute.yaml
- networkpolicy.yaml
apps/nexus/namespace.yaml
-8
View File
@@ -1,8 +0,0 @@
apiVersion: v1
kind: Namespace
metadata:
name: nexus
labels:
pod-security.kubernetes.io/enforce: restricted
pod-security.kubernetes.io/audit: restricted
pod-security.kubernetes.io/warn: restricted
apps/nexus/networkpolicy.yaml
-23
View File
@@ -1,23 +0,0 @@
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: nexus-ingress
namespace: nexus
spec:
podSelector:
matchLabels:
app.kubernetes.io/name: nexus
policyTypes:
- Ingress
ingress:
- from:
- namespaceSelector:
matchLabels:
kubernetes.io/metadata.name: envoy-gateway-system
podSelector:
matchLabels:
gateway.envoyproxy.io/owning-gateway-name: eg
gateway.envoyproxy.io/owning-gateway-namespace: envoy-gateway-system
ports:
- protocol: TCP
port: 8081
apps/nexus/pvc.yaml
-12
View File
@@ -1,12 +0,0 @@
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: nexus-data
namespace: nexus
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 50Gi
storageClassName: localpv-zfs-ceph
apps/nexus/service.yaml
-14
View File
@@ -1,14 +0,0 @@
apiVersion: v1
kind: Service
metadata:
name: nexus
namespace: nexus
spec:
type: ClusterIP
selector:
app.kubernetes.io/name: nexus
ports:
- name: http
port: 8081
protocol: TCP
targetPort: http
apps/nexus/terraform/.gitignore
-6
View File
@@ -1,6 +0,0 @@
.terraform/
*.tfstate
*.tfstate.*
*.tfplan
crash.log
crash.*.log
apps/nexus/terraform/.terraform.lock.hcl
Generated
-24
View File
@@ -1,24 +0,0 @@
# This file is maintained automatically by "terraform init".
# Manual edits may be lost in future updates.
provider "registry.terraform.io/sonatype-nexus-community/sonatyperepo" {
version = "1.17.0"
constraints = "1.17.0"
hashes = [
"h1:uKhvhhhI7B+HBsh0zq/ybqKt+EnOGyI6rjcRCtj79ZA=",
"zh:0dde99e7b343fa01f8eefc378171fb8621bedb20f59157d6cc8e3d46c738105f",
"zh:3315929df254a3a6ac27c8c846c2006f7d2a91fadc014351bc4d617f948e5bf9",
"zh:36be5a455af3ce4e187de26753de63e78c1ee9a32dba0135c6cf96a6c1fff25f",
"zh:3f73f7ff57b8c339a7c7ac37653e2dc0b2dd9dcc3f3a538788e7e3ac838337b2",
"zh:40286ecca4c22ab7ae90618ac6d2743f5055199dac81cf5204a4a397c784d439",
"zh:4d24e5c0195fb3155b1967583ee64cfeda402d7cc7f3c73369438f6c69f4245b",
"zh:828a9d7aceaac36af7f9c07af43ec8d20a89148780645d170ffb1c68b2da792d",
"zh:a5ab04de3fe626ec57c832618c6f990abd6610f81e132621651e0b180b970cff",
"zh:a959fa6090a8c0f53739879184e7346423494aee598003df0d1ab4a22b2eee91",
"zh:bdda26c2f03f918bbe59e75abea44868fafda019c3a543725331195df126350b",
"zh:d8048e149ee97ba62971e6a79355d59887bc6d10fcf72cc2feff3d0a2582670c",
"zh:dd36f9988af4e1ca5b1ca7b7bb6f658df9a220dfcda7fec7392fedfe9064f652",
"zh:dda2688d46c7e539fe97e8fe9d3ec81fb364170e018d9c6a681364c8955d4e9d",
"zh:e6b519afe2dea1c0434f766eb6bc9ba78cc5b6ef2c311c2ca3c65cb24744f31f",
]
}
apps/nexus/terraform/repositories.tf
-64
View File
@@ -1,64 +0,0 @@
locals {
proxy_http_client = {
auto_block = true
blocked = false
}
proxy_negative_cache = {
enabled = true
time_to_live = 60
}
repository_storage = {
blob_store_name = "default"
strict_content_type_validation = true
}
}
resource "sonatyperepo_repository_ansiblegalaxy_proxy" "galaxy" {
name = "ansible-galaxy-proxy"
online = true
http_client = local.proxy_http_client
negative_cache = local.proxy_negative_cache
proxy = {
remote_url = "https://galaxy.ansible.com"
content_max_age = 1440
metadata_max_age = 60
}
storage = local.repository_storage
}
resource "sonatyperepo_repository_ansiblegalaxy_group" "public" {
name = "ansible-public"
online = true
group = {
member_names = [sonatyperepo_repository_ansiblegalaxy_proxy.galaxy.name]
}
storage = local.repository_storage
}
resource "sonatyperepo_repository_go_proxy" "public" {
name = "go-proxy"
online = true
http_client = local.proxy_http_client
negative_cache = local.proxy_negative_cache
proxy = {
remote_url = "https://proxy.golang.org"
content_max_age = 1440
metadata_max_age = 60
}
storage = local.repository_storage
}
resource "sonatyperepo_repository_go_group" "public" {
name = "go-public"
online = true
group = {
member_names = [sonatyperepo_repository_go_proxy.public.name]
}
storage = local.repository_storage
}
apps/nexus/terraform/security.tf
-50
View File
@@ -1,50 +0,0 @@
resource "sonatyperepo_privilege_repository_view" "anonymous_ansible" {
name = "ci-anonymous-ansible-read"
description = "Anonymous read access to the Ansible Galaxy group"
actions = ["BROWSE", "READ"]
format = "ansiblegalaxy"
repository = sonatyperepo_repository_ansiblegalaxy_group.public.name
}
resource "sonatyperepo_privilege_repository_view" "anonymous_ansible_proxy" {
name = "ci-anonymous-ansible-proxy-read"
description = "Anonymous artifact read access to the Ansible Galaxy proxy"
actions = ["BROWSE", "READ"]
format = "ansiblegalaxy"
repository = sonatyperepo_repository_ansiblegalaxy_proxy.galaxy.name
}
resource "sonatyperepo_privilege_repository_view" "anonymous_go" {
name = "ci-anonymous-go-read"
description = "Anonymous read access to the Go module group"
actions = ["BROWSE", "READ"]
format = "go"
repository = sonatyperepo_repository_go_group.public.name
}
resource "sonatyperepo_role" "anonymous_ci" {
id = "ci-anonymous-read"
name = "CI anonymous read"
description = "Read-only access to public CI dependency proxy groups"
privileges = [
sonatyperepo_privilege_repository_view.anonymous_ansible.name,
sonatyperepo_privilege_repository_view.anonymous_ansible_proxy.name,
sonatyperepo_privilege_repository_view.anonymous_go.name,
]
roles = []
}
resource "sonatyperepo_user" "anonymous" {
user_id = "anonymous"
first_name = "Anonymous"
last_name = "User"
email_address = "[email protected]"
status = "active"
roles = [sonatyperepo_role.anonymous_ci.id]
}
resource "sonatyperepo_system_anonymous_access" "ci" {
enabled = true
user_id = sonatyperepo_user.anonymous.user_id
realm_name = "NexusAuthorizingRealm"
}
apps/nexus/terraform/versions.tf
-33
View File
@@ -1,33 +0,0 @@
terraform {
required_version = ">= 1.11.0"
required_providers {
sonatyperepo = {
source = "sonatype-nexus-community/sonatyperepo"
version = "1.17.0"
}
}
}
provider "sonatyperepo" {
url = var.nexus_url
username = var.nexus_username
password = var.nexus_password
}
variable "nexus_url" {
description = "Nexus Repository base URL"
type = string
}
variable "nexus_username" {
description = "Nexus Terraform management username"
type = string
sensitive = true
}
variable "nexus_password" {
description = "Nexus Terraform management password"
type = string
sensitive = true
}
clusters/homelab/README.md
-2
View File
@@ -53,6 +53,4 @@ sudo k3s kubectl -n flux-system get gitrepositories,kustomizations
- SPIRE 已按 hardened chart 内部 fork `0.30.2-ddupan.1`（基于上游 `0.30.2`，SPIRE
`1.15.3`）声明，使用共享
PostgreSQL 与独立 signing-key PVC；首次上线和 OpenBao JWT-SVID PoC 尚待合并后验证；
- Nexus Repository CE POC 已加入 GitOps 声明，计划验证 Ansible Galaxy、Go Modules 与
OCI/BuildKit 缓存；尚未部署或完成现场验收，现有 zot 保持不变；
- root Kustomization 与所有 brownfield 子 Kustomization 继续保持 `prune: false`。
clusters/homelab/apps/dynamic-runner.yaml
+2 -3
View File
@@ -6,12 +6,11 @@ metadata:
spec:
dependsOn:
- name: external-secrets
- name: nats
- name: spire
interval: 10m
path: ./platform/dynamic-runner
# The runner backends are replaceable. Prune is required when a retired
# worker is removed from the component; otherwise it keeps consuming work.
prune: true
prune: false
sourceRef:
kind: GitRepository
name: flux-system
clusters/homelab/apps/nexus.yaml
-22
View File
@@ -1,22 +0,0 @@
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
name: nexus
namespace: flux-system
spec:
dependsOn:
- name: envoy-gateway
- name: openebs
healthChecks:
- apiVersion: apps/v1
kind: Deployment
name: nexus
namespace: nexus
interval: 10m
path: ./apps/nexus
prune: false
sourceRef:
kind: GitRepository
name: flux-system
timeout: 15m
wait: true
clusters/homelab/kustomization.yaml
-1
View File
@@ -15,4 +15,3 @@ resources:
- apps/spire.yaml
- apps/observability.yaml
- apps/zot.yaml
- apps/nexus.yaml
clusters/sandbox/README.md
+4 -15
View File
@@ -15,8 +15,7 @@ Root bootstrap 已完成。后续按依赖顺序分别引入：
1. 监控 CRD、kube-state-metrics 以及 kubelet/cAdvisor 抓取配置；
2. SPIRE Agent、SPIFFE CSI Driver 与 workload registration；
3. Kata Containers、`block-plain` RuntimeClass；
4. 独立 External Secrets Operator 与 sandbox 专用 OpenBao auth backend；
5. OpenSandbox controller/server；CI Pool 与 runner 调度器随后独立接入。
4. OpenSandbox operator/server 及 `ci-pod`、`ci-vm` Pools。
每一阶段单独合并并等待对应 Flux Kustomization Ready，不在 bootstrap 时一次性部署。
第一阶段监控拆为 `monitoring-operator` 与依赖它的 `monitoring`，防止 VM CR 在
@@ -43,25 +42,15 @@ attestation。Server 使用 external bundle publisher 持续维护 sandbox
显式关闭 Server 与 OIDC Provider，只部署 Agent DaemonSet 和 SPIFFE CSI Driver；因此
不会产生第二个 trust root。
`spire-smoke` namespace、ServiceAccount 和 `sandbox-spire-smoke` ClusterSPIFFEID 只用于
普通 Pod 的 CSI 回归夹具，稳定身份为 `spiffe://ddupan.top/sandbox/smoke`。Kata guest
不能复用 node Agent 暴露的 Unix socket；virtio-fs 只能呈现 socket 路径，不能把连接
跨过 VM 边界。Kata workload 必须使用 guest 内 Agent，具体约束见
`platform/sandbox-kata/README.md`。测试 Pod 临时创建并在验收后删除，普通 Pod 的身份
声明保留。
`spire-smoke` namespace、ServiceAccount 和 `sandbox-spire-smoke` ClusterSPIFFEID 是
普通 Pod 与后续 Kata guest 的回归夹具，稳定身份为
`spiffe://ddupan.top/sandbox/smoke`。测试 Pod 临时创建并在验收后删除，身份声明保留。
Kata 阶段使用官方 4.1.0 `kata-deploy` chart 的短生命周期 `job` 模式，逐节点安装并
重启 K3s。只启用 `kata-clh-runtime-rs`，不创建默认 `kata` 别名；该 handler 的
`emptyDir` 固定使用 `block-plain`，为 Docker/BuildKit overlay2 与 kind 提供 guest
内块设备文件系统。详细限制与上线验收见 `platform/sandbox-kata/README.md`。
Sandbox 的 ESO 通过独立 `auth/kubernetes-sandbox` 向 OpenBao 证明 ServiceAccount
身份，只能读取共享的 `kv/k8s/opensandbox-api`。它不保存 reviewer JWT 或长期 Bao token；相关
Terraform 与 Flux 边界见 `platform/sandbox-external-secrets/README.md`。
OpenSandbox 阶段固定官方源码 commit 与 umbrella chart `0.2.2`，只部署 controller、
ClusterIP server 和 CRD。API key 由 ESO 从 OpenBao 投影，明文不进入 Git。
## 监控边界
这里只管理 sandbox LXC 内的 Kubernetes 监控，不负责 PVE 宿主监控。LXC 与宿主共享
clusters/sandbox/apps/external-secrets-operator.yaml
-17
View File
@@ -1,17 +0,0 @@
---
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
name: external-secrets-operator
namespace: flux-system
spec:
dependsOn:
- name: monitoring-operator
interval: 10m
path: ./platform/sandbox-external-secrets/operator
prune: true
sourceRef:
kind: GitRepository
name: flux-system
timeout: 10m
wait: true
clusters/sandbox/apps/external-secrets.yaml
-17
View File
@@ -1,17 +0,0 @@
---
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
name: external-secrets
namespace: flux-system
spec:
dependsOn:
- name: external-secrets-operator
interval: 10m
path: ./platform/sandbox-external-secrets/config
prune: true
sourceRef:
kind: GitRepository
name: flux-system
timeout: 10m
wait: true
clusters/sandbox/apps/ci-runners.yaml → clusters/sandbox/apps/opensandbox-pools.yaml
+2 -3
View File
@@ -2,14 +2,13 @@
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
name: ci-runners
name: opensandbox-pools
namespace: flux-system
spec:
dependsOn:
- name: opensandbox
- name: spire-agents
interval: 10m
path: ./platform/sandbox-ci-runners
path: ./platform/sandbox-opensandbox-pools
prune: true
sourceRef:
kind: GitRepository
clusters/sandbox/apps/opensandbox.yaml
+2 -3
View File
@@ -6,14 +6,13 @@ metadata:
namespace: flux-system
spec:
dependsOn:
- name: external-secrets
- name: kata
- name: monitoring-operator
- name: spire-agents
interval: 10m
path: ./platform/sandbox-opensandbox
prune: true
sourceRef:
kind: GitRepository
name: flux-system
timeout: 15m
timeout: 20m
wait: true
clusters/sandbox/kustomization.yaml
+1 -3
View File
@@ -7,7 +7,5 @@ resources:
- apps/spire-bootstrap.yaml
- apps/spire-agents.yaml
- apps/kata.yaml
- apps/external-secrets-operator.yaml
- apps/external-secrets.yaml
- apps/opensandbox.yaml
- apps/ci-runners.yaml
- apps/opensandbox-pools.yaml
infrastructure/dns/records.yml
-1
View File
@@ -18,7 +18,6 @@ homelab_dns:
- { zone: ad.ddupan.top, name: metrics-write, type: A, values: [192.168.10.127] }
- { zone: ad.ddupan.top, name: netbox, type: A, values: [192.168.10.127] }
- { zone: ad.ddupan.top, name: nats, type: A, values: [192.168.10.127] }
- { zone: ad.ddupan.top, name: nexus, type: A, values: [192.168.10.127] }
- { zone: ad.ddupan.top, name: s3, type: A, values: [192.168.10.127] }
- { zone: ad.ddupan.top, name: spire-oidc, type: A, values: [192.168.10.127] }
- { zone: ad.ddupan.top, name: spire-server, type: A, values: [192.168.10.127] }
infrastructure/openbao/README.md
+1 -8
View File
@@ -51,7 +51,7 @@ few things Terraform must not own.**
| secrets engine mounts (`kv`, `pki`, `ssh-client-signer`) | the daemon, Raft, TLS files, systemd |
| PKI role, issuing/CRL URLs, cluster paths, **ACME** | `bao operator init` / unseal (manual, PGP-wrapped) |
| SSH signing role (`ai-agent`) | **PKI root CA + SSH CA signing key** |
| OIDC/Kubernetes auth mount、config 与 role | OIDC **client secret** (`auth/oidc/config`) |
| OIDC auth *mount* and *role* | OIDC **client secret** (`auth/oidc/config`) |
| all policies | snapshot token + script + systemd timer |
| | host-level CA trust distribution (`openbao_ssh_ca_trust`) |
@@ -92,13 +92,6 @@ then `VAULT_ADDR`/`VAULT_TOKEN`), mirroring how `smtp-relay/terraform` uses
native `openbao/openbao` provider is published only to the OpenTofu registry and
cannot be resolved by the HashiCorp `terraform` CLI.
Sandbox 集群使用独立的 `auth/kubernetes-sandbox`。其 API 地址、公开 Kubernetes CA、
ESO role 与只读 `kv/k8s/opensandbox-api` policy 全部由 Terraform 管理；CA 位于
`terraform/certs/sandbox-kubernetes-ca.crt`。集群重建并轮换 CA 后，先更新该文件并
apply，再让 Flux 恢复 ESO reconciliation。该 backend 不保存 reviewer JWT，而是使用
ESO 的短期登录 JWT 执行 TokenReview。该组资源已于 2026-09-18 apply，随后复验 plan
为 zero-diff。
## DNS
`bao.ad.ddupan.top` is an **internal-only** name — not a public Cloudflare record and
infrastructure/openbao/terraform/auth.tf
-30
View File
@@ -59,33 +59,3 @@ resource "vault_kubernetes_auth_backend_role" "external_secrets" {
# in a long TTL and every extra hour is a longer-lived credential in memory.
token_ttl = 3600
}
# A Kubernetes auth mount can validate identities from only the API server it is
# configured against. The sandbox cluster therefore cannot reuse auth/kubernetes,
# whose TokenReview endpoint belongs to homelab.
resource "vault_auth_backend" "sandbox_kubernetes" {
type = "kubernetes"
path = "kubernetes-sandbox"
}
resource "vault_kubernetes_auth_backend_config" "sandbox" {
backend = vault_auth_backend.sandbox_kubernetes.path
kubernetes_host = "https://10.60.0.13:6443"
kubernetes_ca_cert = file("${path.module}/certs/sandbox-kubernetes-ca.crt")
disable_local_ca_jwt = true
# Deliberately omit token_reviewer_jwt. OpenBao uses the login JWT for
# TokenReview; the sandbox external-secrets ServiceAccount is bound only to
# system:auth-delegator and all issued JWTs remain short-lived.
}
resource "vault_kubernetes_auth_backend_role" "sandbox_external_secrets" {
backend = vault_auth_backend.sandbox_kubernetes.path
role_name = "external-secrets"
bound_service_account_names = ["external-secrets"]
bound_service_account_namespaces = ["external-secrets"]
token_policies = [vault_policy.sandbox_external_secrets.name]
token_ttl = 3600
}
infrastructure/openbao/terraform/certs/sandbox-kubernetes-ca.crt
-10
View File
@@ -1,10 +0,0 @@
-----BEGIN CERTIFICATE-----
MIIBdjCCAR2gAwIBAgIBADAKBggqhkjOPQQDAjAjMSEwHwYDVQQDDBhrM3Mtc2Vy
dmVyLWNhQDE3ODk2NTI4NTcwHhcNMjYwOTE3MTI0NzM3WhcNMzYwOTE0MTI0NzM3
WjAjMSEwHwYDVQQDDBhrM3Mtc2VydmVyLWNhQDE3ODk2NTI4NTcwWTATBgcqhkjO
PQIBBggqhkjOPQMBBwNCAAR4SbqzTXZnlZdUPz7viN6+dYbB1Maw44Qiepn9r5XG
sOzkYkN8t1aG3Ugo8TqQ3xJaKkM89n1Rluj0vbOhiNajo0IwQDAOBgNVHQ8BAf8E
BAMCAqQwDwYDVR0TAQH/BAUwAwEB/zAdBgNVHQ4EFgQUyLSGoKAKAJuiniuRdBLG
XYaDQC8wCgYIKoZIzj0EAwIDRwAwRAIgFkVzyUZexk/ynnxBEOg+3foJv3WKqAei
hTSRjO1gL0UCIFbBKR7BMrJJAgW3DJFeeBM+b+tTg93jNx55qZACbFOL
-----END CERTIFICATE-----
infrastructure/openbao/terraform/policies.tf
-7
View File
@@ -35,10 +35,3 @@ resource "vault_policy" "external_secrets" {
name = "external-secrets"
policy = file("${path.module}/policies/external-secrets.hcl")
}
# The sandbox cluster has its own Kubernetes auth backend and a deliberately
# narrower KV view than the homelab ESO instance.
resource "vault_policy" "sandbox_external_secrets" {
name = "sandbox-external-secrets"
policy = file("${path.module}/policies/sandbox-external-secrets.hcl")
}
infrastructure/openbao/terraform/policies/sandbox-external-secrets.hcl
-9
View File
@@ -1,9 +0,0 @@
# Read only the shared OpenSandbox control-plane API key. The same Bao object is
# consumed by the server in sandbox and, later, by the scheduler in homelab.
path "kv/data/k8s/opensandbox-api" {
capabilities = ["read"]
}
path "kv/metadata/k8s/opensandbox-api" {
capabilities = ["read"]
}
infrastructure/proxmox/ansible/roles/vyos_router/defaults/main.yml
-5
View File
@@ -83,11 +83,6 @@ vyos_postgresql_primary_address: "10.60.0.11"
vyos_sandbox_k3s_api_address: "10.60.0.13"
vyos_sandbox_k3s_api_interface: eth1
vyos_sandbox_k3s_api_port: 6443
vyos_opensandbox_api_port: 8080
vyos_opensandbox_node_port: 30080
# Lifecycle create is synchronous and a cold Kata image pull can exceed the
# HAProxy 50-second default. Keep this below OpenSandbox's overall timeout.
vyos_opensandbox_api_timeout: 600
vyos_sandbox_k3s_servers:
- name: sandbox1
address: "10.60.0.11"
infrastructure/proxmox/ansible/roles/vyos_router/templates/vyos.conf.j2
-14
View File
@@ -15,20 +15,6 @@ set interfaces ethernet {{ i.iface }} address {{ i.address }}
set interfaces ethernet {{ i.iface }} description '{{ i.description }}'
{% endfor %}
{# OpenSandbox stays on a NodePort; VyOS owns the stable routed frontend. #}
set load-balancing haproxy service opensandbox-api listen-address {{ vyos_sandbox_k3s_api_address }}
set load-balancing haproxy service opensandbox-api port {{ vyos_opensandbox_api_port }}
set load-balancing haproxy service opensandbox-api mode 'tcp'
set load-balancing haproxy service opensandbox-api backend 'opensandbox-api'
set load-balancing haproxy service opensandbox-api timeout client {{ vyos_opensandbox_api_timeout }}
set load-balancing haproxy backend opensandbox-api mode 'tcp'
set load-balancing haproxy backend opensandbox-api timeout server {{ vyos_opensandbox_api_timeout }}
{% for server in vyos_sandbox_k3s_servers %}
set load-balancing haproxy backend opensandbox-api server {{ server.name }} address {{ server.address }}
set load-balancing haproxy backend opensandbox-api server {{ server.name }} port {{ vyos_opensandbox_node_port }}
set load-balancing haproxy backend opensandbox-api server {{ server.name }} check
{% endfor %}
{# --- default route out; OSPF carries the rest --- #}
set protocols static route 0.0.0.0/0 next-hop {{ vyos_lan_gateway }}
infrastructure/sandbox-cluster/README.md
+2 -12
View File
@@ -18,8 +18,7 @@ Flux 管理以下 Kubernetes 资源：
- Kata Containers 和 CI 专用的 `block-plain` RuntimeClass；
- SPIRE Agent、SPIFFE CSI Driver 与 workload identity 声明；
- vmagent、kube-state-metrics、kubelet/cAdvisor scrape 配置和告警；
- sandbox External Secrets Operator、OpenSandbox controller/server；CI Pool 与 runner
调度器由 runner 项目接入。
- OpenSandbox operator/server、`ci-pod` 与 `ci-vm` Pools。
同一个对象只能有一个 owner。Ansible 不直接部署上述集群内 workload；Flux 不管理
LXC、K3s datastore 或 K3s 本身。
@@ -93,13 +92,6 @@ ansible-playbook site.yml
ansible-playbook k3s.yml
```
只 reconcile Flux controllers 与 root sync manifest（不触碰 LXC、PostgreSQL 或
K3s lifecycle）：
```bash
ansible-playbook flux.yml
```
K3s 外部 datastore URI 由运行时 `SANDBOX_K3S_DB_PASSWORD` 生成，密码在 URI 中
进行 URL 编码，最终仅持久化于节点 root 可读的 `/etc/rancher/k3s/config.yaml`
（mode `0600`）。首节点生成的 K3s join token 仅在同一次 Ansible run 内传给第二节点；
@@ -114,9 +106,7 @@ ansible-playbook verify.yml
当前已经声明 LXC 生命周期、最小 OS baseline、PostgreSQL 和 K3s，包括系统级
homelab CA trust。Flux `v2.9.5` controllers 与 root sync 也由 Ansible 通过 K3s
server manifests 管理；root 使用 homelab CA 访问公开 Gitea 仓库，不保存 Git token。
集群内 workload 由 `clusters/sandbox/` 分阶段纳入 Flux。root Kustomization 的健康检查
timeout 为 40 分钟，用于覆盖 Kata 等首次安装时会逐节点重启 K3s 的子
Kustomization；各子项仍保留自己的更短 timeout，故障会在对应子项先行暴露。
集群内 workload 由 `clusters/sandbox/` 分阶段纳入 Flux。
## SPIRE 跨集群 bootstrap
infrastructure/sandbox-cluster/ansible/flux.yml
-6
View File
@@ -1,6 +0,0 @@
---
- name: Reconcile Flux controllers and root sync
hosts: sandbox1
gather_facts: false
roles:
- sandbox_flux
infrastructure/sandbox-cluster/ansible/roles/sandbox_flux/templates/flux-sync.yaml.j2
+1 -1
View File
@@ -35,5 +35,5 @@ spec:
sourceRef:
kind: GitRepository
name: flux-system
timeout: 40m
timeout: 3m
wait: true
platform/dynamic-runner/README.md
+36 -29
View File
@@ -1,44 +1,51 @@
# Gitea dynamic runner controller
controller 接收 Gitea `workflow_job` webhook，并将带 `[self-hosted, pod]` 或
`[self-hosted, vm]` 的 queued job 持久化到 NATS JetStream。Pod workload 在 homelab
集群由原生 Kubernetes worker 创建；只有 VM workload 通过 OpenSandbox Lifecycle API。
此目录只管理 homelab 中的 controller 部署。controller、worker、Cloud Hypervisor
launcher 和 guest runner 的源码与发布位于独立仓库
`panxiao81/gitea-dynamic-runner`。
当前 bootstrap controller 接收 Gitea `workflow_job` webhook，将 `[self-hosted, pod]` 和
`[self-hosted, vm]` 的 queued job 分别发布到 NATS。Pod worker 在本集群创建一次性
privileged host runner；Docker、BuildKit 和 kind 由 workflow 自行 setup。内部
endpoint：
```text
Gitea -> controller -> NATS -> pod worker -> homelab Kubernetes Pod
`-> VM worker -> OpenSandbox -> Kata/microVM
http://dynamic-runner-controller.dynamic-runner.svc.cluster.local:8787/webhook
```
`dynamic-runner-pod-worker` 使用 durable `pod`，最多并发创建 4 个一次性 privileged
Pod。任务 Pod、ServiceAccount、SPIFFE CSI socket 和生命周期都位于 homelab 的
`dynamic-runner` namespace，不依赖 sandbox 集群或 OpenSandbox API。
OpenBao 路径：
`10.60.0.13:8080` 是 VyOS 上的内网 HAProxy frontend，后端是 sandbox 节点的
`30080` NodePort。它只依赖现有跨网段路由，不暴露公网，也不经过 Cloudflare
Tunnel。controller 自身的 `192.168.10.127:8787` 仅供 Gitea webhook 和 sandbox
通过一次性 nonce 领取 registration token。
- `kv/k8s/nats.ci_producer_password`：已有 NATS producer 密码。
- `kv/k8s/nats.ci_worker_password`：已有 NATS worker 密码。
- `kv/k8s/dynamic-runner.webhook_secret`：Gitea webhook HMAC secret。
- `kv/k8s/gitea-runner.token`：现有 instance runner registration token。
## Secret 边界
首期 controller 与 runner 镜像由 laptop 本机构建后导入 k3s containerd，作为 CI
发布链路建立前的 bootstrap。部署使用 `imagePullPolicy: Never`。正式发布 workflow
获得专用 SPIFFE ID 后，必须将 image 改为 zot digest 并移除本地导入步骤。
homelab 的 `ExternalSecret/dynamic-runner` 从既有 `ClusterSecretStore/openbao` 读取：
## 身份绑定
- `kv/k8s/opensandbox-api:api_key` -> `opensandbox-api-key`；
- `kv/k8s/dynamic-runner:webhook_secret` -> `webhook-secret`；
- `kv/k8s/gitea-runner:token` -> `token`。
queued webhook 只负责创建没有业务身份的 Pod。runner 实际领取任务后，Gitea 的
`in_progress` webhook 会携带实际 `runner_name`；controller 将 binding 消息发布到
NATS，Pod worker 再给对应 Pod 添加：
API key 以只读文件挂载，controller 通过 `OPEN-SANDBOX-API-KEY` header 使用。不要把
key 复制到 Git、Lifecycle 请求、BatchSandbox 或 sandbox 集群 Secret。本目录不创建
Bao token，也不拥有 sandbox 平台侧的 ESO/Bao Terraform。
```text
ci.ddupan.top/identity-bound=true
ci.ddupan.top/spiffe-path=<owner>/<repository>/<percent-encoded-job-name>
```
## 调度与身份
`ClusterSPIFFEID/gitea-dynamic-runner` 只匹配已经绑定的 Pod，并签发
`spiffe://ddupan.top/ci/<owner>/<repository>/<job-name>`。runner 的 job-start hook 在
SVID 可用之前不会放行第一步，因此不能根据 queued 事件错配身份。
Lifecycle 请求只按 `ci-pod` / `ci-vm` 选择 Pool，并携带稳定的
`spiffe://ddupan.top/ci/<owner>/<repository>/<task>`。job ID 只用于日志和诊断 metadata。
runner labels 始终以 `self-hosted` 开头。
每个 runner Pod 使用 `gitea-dynamic-runner` ServiceAccount。该 ServiceAccount 没有
Kubernetes API 权限；只有 `dynamic-runner-pod-worker` ServiceAccount 能在本 namespace
create/get/patch/delete Pod。
registration token 通过 controller 内存中的单次 nonce URL 投递。成功领取后 nonce
立即失效；沙箱结束、创建失败或超时时也会撤销。OpenSandbox DELETE 是最终的正常
回收路径，OpenSandbox timeout 是 controller 异常退出时的兜底。
长期实现将由兼容 Gitea RunnerService 的 scheduler 直接领取 task，再交给 Pod/VM
executor；届时删除 webhook、临时 runner 注册和 identity binding 消息。跟踪见
`panxiao81/gitea-dynamic-runner` issue #7。
完整部署和验收步骤见
[`platform/sandbox-ci-runners/README.md`](../sandbox-ci-runners/README.md)。
Gitea webhook 只订阅 `workflow_job`，content type 使用 JSON，secret 与 Bao 中值
一致。不要启用 `send_everything`，否则 controller 会收到无关仓库事件。
platform/dynamic-runner/deployment.yaml
+5 -17
View File
@@ -40,31 +40,19 @@ spec:
mountPath: /trust
containers:
- name: controller
image: zot.ad.ddupan.top/panxiao81/gitea-dynamic-runner-controller@sha256:9f23e84934643bbe859dc0f8da418421dc8ab7b0958fdf1fdc91d562c30e246f
imagePullPolicy: IfNotPresent
# Bootstrap import on laptop. Replace with a zot digest after the
# repository's image publishing workflow has a dedicated identity.
image: gitea-dynamic-runner-controller:0.3.0-bootstrap
imagePullPolicy: Never
env:
- name: NATS_URL
value: tls://nats.ad.ddupan.top:4222
- name: NATS_CA_FILE
value: /run/trust/ca.pem
- name: NATS_PRODUCER_PASSWORD_FILE
- name: NATS_PASSWORD_FILE
value: /run/dynamic-runner-secrets/nats-password
- name: NATS_WORKER_PASSWORD_FILE
value: /run/dynamic-runner-secrets/nats-worker-password
- name: POD_CONSUMER_ENABLED
value: "false"
- name: VM_CONSUMER_ENABLED
value: "false"
- name: WEBHOOK_SECRET_FILE
value: /run/dynamic-runner-secrets/webhook-secret
- name: REGISTRATION_TOKEN_FILE
value: /run/dynamic-runner-secrets/token
- name: OPENSANDBOX_API
value: http://10.60.0.13:8080
- name: OPENSANDBOX_API_KEY_FILE
value: /run/dynamic-runner-secrets/opensandbox-api-key
- name: RUNNER_TOKEN_BASE_URL
value: http://192.168.10.127:8787/token
ports:
- name: http
containerPort: 8787
platform/dynamic-runner/external-secret.yaml
-4
View File
@@ -20,10 +20,6 @@ spec:
remoteRef:
key: k8s/nats
property: ci_worker_password
- secretKey: opensandbox-api-key
remoteRef:
key: k8s/opensandbox-api
property: api_key
- secretKey: webhook-secret
remoteRef:
key: k8s/dynamic-runner
platform/dynamic-runner/pod-worker-deployment.yaml
+2 -2
View File
@@ -39,8 +39,8 @@ spec:
mountPath: /trust
containers:
- name: pod-worker
image: zot.ad.ddupan.top/panxiao81/gitea-dynamic-runner-controller@sha256:9f23e84934643bbe859dc0f8da418421dc8ab7b0958fdf1fdc91d562c30e246f
imagePullPolicy: IfNotPresent
image: gitea-dynamic-runner-controller:0.3.0-bootstrap
imagePullPolicy: Never
command: [/venv/bin/gitea-dynamic-runner-pod-worker]
env:
- name: NATS_URL
platform/dynamic-runner/service.yaml
-2
View File
@@ -4,8 +4,6 @@ metadata:
name: dynamic-runner-controller
namespace: dynamic-runner
spec:
type: LoadBalancer
loadBalancerIP: 192.168.10.127
selector:
app.kubernetes.io/name: dynamic-runner-controller
ports:
platform/sandbox-ci-runners/README.md
-66
View File
@@ -1,66 +0,0 @@
# OpenSandbox Gitea runner Pools
本目录只部署零预热的 `ci-vm` Pool、sandbox 内的 SPIFFE identity controller，以及仅供
内网 VyOS 转发的 OpenSandbox NodePort。`ci-vm` 使用 `kata-clh-runtime-rs`，每个 VM
执行单一任务并在结束后删除。普通 Pod job 直接运行在 homelab，不进入 OpenSandbox。
Pool 中 task-executor 接收 Lifecycle API 下发的进程环境。guest-local SPIRE Agent 用
Pod-bound PSAT 向中央 SPIRE 注册；identity controller 从 BatchSandbox allocation
取得真实 Pod UID，再创建精确的 `ClusterStaticEntry`。runner 只有拿到请求中的完整
repository/task SVID 后才领取一次性 Gitea registration token。
这些 `ClusterStaticEntry` 位于 sandbox 集群，由 central SPIRE Server 内的
`spire-controller-manager-sandbox` 通过受限 external kubeconfig reconcile。必须在
`platform/spire/values.yaml` 显式启用 external controller-manager 的
`reconcile.clusterStaticEntries`（chart 默认关闭）；仅看到 CR 存在但没有 status，不算
身份链路就绪。
## 部署依赖
- OpenSandbox chart 和 CRD 已 Ready；
- RuntimeClass `kata-clh-runtime-rs` 已存在；
- 中央 SPIRE 已发布 `ConfigMap/opensandbox/spire-bundle-pem`；
- VyOS `10.60.0.13:8080` 转发 sandbox1/2 的 NodePort `30080`；
- runner/controller 镜像均使用 Zot digest，而不是可变 tag。
本目录不读取 OpenBao，也不修改 OpenSandbox 平台侧 ExternalSecret、ClusterSecretStore
或 Bao policy。OpenSandbox API key 只存在于平台 server Secret 和 homelab controller
Secret，两边由各自身份读取同一 Bao 资源。
## 上线验收
先确认 Secret 和 API 认证，命令不得输出 key：
```bash
kubectl -n dynamic-runner wait externalsecret/dynamic-runner --for=condition=Ready --timeout=2m
kubectl -n dynamic-runner exec deploy/dynamic-runner-controller -- \
wget -qO- http://10.60.0.13:8080/health
```
然后触发一个 `runs-on: [self-hosted, vm]` 的最小 workflow，并同时观察：
```bash
kubectl -n dynamic-runner logs deploy/dynamic-runner-controller -f
kubectl -n opensandbox get pool,batchsandbox,pod -w
kubectl get clusterstaticentry -l app.kubernetes.io/component=opensandbox-identity -w
kubectl -n opensandbox logs deploy/opensandbox-identity -f
```
合格证据必须同时包含：
1. Lifecycle create 成功并选择 `ci-vm`；
2. 分配 Pod 的 `runtimeClassName` 为 `kata-clh-runtime-rs`；
3. entry 的 parent ID 包含该 Pod 的 UID，SPIFFE ID 使用 repository/task；
4. Gitea 显示临时 runner 带 `self-hosted,vm` labels 并完成真实任务；
5. 任务后 BatchSandbox、Pod、ClusterStaticEntry 和临时 runner 均消失。
清理超时对象时只调用 Lifecycle DELETE，不直接删除 Pool Pod。若 controller 已不可用，
可从 OpenSandbox API 按 metadata 定位 sandbox 后执行 DELETE；不要绕过 API 伪造状态。
## 故障定位
- `401`：检查 homelab ExternalSecret Ready 和文件挂载，不打印 Secret；
- `PoolCapacityExhausted`：检查 `ci-vm` 的 `poolMax` 及残留 BatchSandbox；
- runner 等待 SVID：核对 allocation Pod UID、ClusterStaticEntry parentID、guest Agent 日志；
- runner 等待 token：核对 `192.168.10.127:8787` 的 sandbox 到 homelab 路由；
- Docker 任务失败：检查 `docker` sidecar 和 `/run/docker/docker.sock` 的 group 2000。
platform/sandbox-ci-runners/agent-config.yaml
-31
View File
@@ -1,31 +0,0 @@
apiVersion: v1
kind: ConfigMap
metadata:
name: gitea-ci-spire-agent
namespace: opensandbox
data:
agent.conf: |
agent {
data_dir = "/run/spire/data"
log_level = "INFO"
server_address = "spire-server.ad.ddupan.top"
server_port = "8081"
socket_path = "/run/spire/agent-sockets/spire-agent.sock"
trust_bundle_path = "/run/spire/bundle/bundle.pem"
trust_domain = "ddupan.top"
}
plugins {
NodeAttestor "k8s_psat" {
plugin_data {
cluster = "sandbox-kata"
token_path = "/run/spire/tokens/token"
}
}
KeyManager "memory" {
plugin_data {}
}
WorkloadAttestor "unix" {
plugin_data {}
}
}
platform/sandbox-ci-runners/identity-controller.yaml
-41
View File
@@ -1,41 +0,0 @@
apiVersion: apps/v1
kind: Deployment
metadata:
name: opensandbox-identity
namespace: opensandbox
spec:
replicas: 1
selector:
matchLabels:
app.kubernetes.io/name: opensandbox-identity
template:
metadata:
labels:
app.kubernetes.io/name: opensandbox-identity
spec:
serviceAccountName: opensandbox-identity
containers:
- name: controller
image: zot.ad.ddupan.top/panxiao81/gitea-dynamic-runner-controller@sha256:dfbfaf2a7aa5951d1dc4e7f941a8cd05ceb4a860ac1ab23eb8be234861edcb55
imagePullPolicy: IfNotPresent
command: [/venv/bin/gitea-dynamic-runner-opensandbox-identity]
resources:
requests:
cpu: 10m
memory: 32Mi
limits:
cpu: 100m
memory: 96Mi
securityContext:
allowPrivilegeEscalation: false
capabilities:
drop: [ALL]
readOnlyRootFilesystem: true
runAsNonRoot: true
runAsUser: 65532
runAsGroup: 65532
seccompProfile:
type: RuntimeDefault
securityContext:
seccompProfile:
type: RuntimeDefault
platform/sandbox-ci-runners/kustomization.yaml
-22
View File
@@ -1,22 +0,0 @@
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- service.yaml
- agent-config.yaml
- rbac.yaml
- identity-controller.yaml
- pools.yaml
patches:
# Pod jobs run directly in the homelab cluster. Keep OpenSandbox VM-only.
- target:
group: sandbox.opensandbox.io
version: v1alpha1
kind: Pool
name: ci-pod
patch: |-
apiVersion: sandbox.opensandbox.io/v1alpha1
kind: Pool
metadata:
name: ci-pod
namespace: opensandbox
$patch: delete
platform/sandbox-ci-runners/pools.yaml
-301
View File
@@ -1,301 +0,0 @@
---
apiVersion: sandbox.opensandbox.io/v1alpha1
kind: Pool
metadata:
name: ci-vm
namespace: opensandbox
labels:
app.kubernetes.io/name: gitea-ci-vm
app.kubernetes.io/component: runner-pool
spec:
capacitySpec:
bufferMax: 0
bufferMin: 0
poolMax: 2
poolMin: 0
recycleStrategy:
type: Delete
template:
metadata:
labels:
app.kubernetes.io/name: gitea-ci-vm
ci.ddupan.top/backend: vm
spec:
runtimeClassName: kata-clh-runtime-rs
serviceAccountName: gitea-ci
restartPolicy: Never
terminationGracePeriodSeconds: 30
shareProcessNamespace: true
securityContext:
fsGroup: 2000
fsGroupChangePolicy: OnRootMismatch
initContainers:
- name: task-executor-installer
image: sandbox-registry.cn-zhangjiakou.cr.aliyuncs.com/opensandbox/task-executor:v0.1.0
command: [/bin/sh, -c]
args:
- cp /workspace/server /opt/opensandbox/task-executor && chmod 0755 /opt/opensandbox/task-executor
volumeMounts:
- name: opensandbox-bin
mountPath: /opt/opensandbox
- name: execd-installer
image: sandbox-registry.cn-zhangjiakou.cr.aliyuncs.com/opensandbox/execd:v1.0.22
command: [/bin/sh, -c]
args:
- cp ./execd /opt/opensandbox/execd && cp ./bootstrap.sh /opt/opensandbox/bootstrap.sh && chmod 0755 /opt/opensandbox/execd /opt/opensandbox/bootstrap.sh
volumeMounts:
- name: opensandbox-bin
mountPath: /opt/opensandbox
containers:
- name: sandbox
image: zot.ad.ddupan.top/panxiao81/gitea-dynamic-runner-runner@sha256:a45875fd2d0e67429b0b7bc3914735581669c705bb4e1a05d712134d2bceb86f
imagePullPolicy: IfNotPresent
command: [/opt/opensandbox/task-executor]
args:
- -listen-addr=0.0.0.0:5758
- -log-dir=/tmp
- -data-dir=/tmp/tasks
env:
- name: SANDBOX_MAIN_CONTAINER
value: sandbox
- name: EXECD_ENVS
value: /opt/opensandbox/.env
- name: EXECD
value: /opt/opensandbox/execd
- name: GITEA_INSTANCE_URL
value: https://git.ddupan.top
- name: HOME
value: /data
- name: DOCKER_HOST
value: unix:///run/docker/docker.sock
ports:
- name: task-executor
containerPort: 5758
securityContext:
allowPrivilegeEscalation: false
capabilities:
drop: [ALL]
runAsNonRoot: true
runAsUser: 2000
runAsGroup: 2000
volumeMounts:
- name: opensandbox-bin
mountPath: /opt/opensandbox
- name: runner-data
mountPath: /data
- name: docker-socket
mountPath: /run/docker
- name: spire-socket
mountPath: /run/spire/agent-sockets
- name: spire-agent
image: ghcr.io/spiffe/spire-agent:1.15.3@sha256:41b0dcd8b258a69db9e2768292a060766fb76fd866e4bc925849981ea1b825ff
args: [-config, /run/spire/config/agent.conf]
securityContext:
allowPrivilegeEscalation: false
capabilities:
drop: [ALL]
volumeMounts:
- name: spire-config
mountPath: /run/spire/config
readOnly: true
- name: spire-bundle
mountPath: /run/spire/bundle
readOnly: true
- name: spire-token
mountPath: /run/spire/tokens
readOnly: true
- name: spire-data
mountPath: /run/spire/data
- name: spire-socket
mountPath: /run/spire/agent-sockets
- name: docker
image: docker.io/library/docker:29.1.5-dind
command: [/bin/sh, -c]
args:
- test -e /dev/kmsg || mknod /dev/kmsg c 1 11; exec dockerd --host=unix:///run/docker/docker.sock --group=2000 --storage-driver=overlay2
securityContext:
privileged: true
volumeMounts:
- name: docker-socket
mountPath: /run/docker
- name: docker-data
mountPath: /var/lib/docker
volumes:
- name: opensandbox-bin
emptyDir: {}
- name: runner-data
emptyDir: {}
- name: docker-data
emptyDir: {}
- name: docker-socket
emptyDir:
medium: Memory
- name: spire-data
emptyDir:
medium: Memory
- name: spire-socket
emptyDir:
medium: Memory
- name: spire-config
configMap:
name: gitea-ci-spire-agent
- name: spire-bundle
configMap:
name: spire-bundle-pem
- name: spire-token
projected:
sources:
- serviceAccountToken:
audience: spire-server
expirationSeconds: 3600
path: token
---
apiVersion: sandbox.opensandbox.io/v1alpha1
kind: Pool
metadata:
name: ci-pod
namespace: opensandbox
labels:
app.kubernetes.io/name: gitea-ci-pod
app.kubernetes.io/component: runner-pool
spec:
capacitySpec:
bufferMax: 0
bufferMin: 0
poolMax: 4
poolMin: 0
recycleStrategy:
type: Delete
template:
metadata:
labels:
app.kubernetes.io/name: gitea-ci-pod
ci.ddupan.top/backend: pod
spec:
serviceAccountName: gitea-ci
restartPolicy: Never
terminationGracePeriodSeconds: 30
shareProcessNamespace: true
securityContext:
fsGroup: 2000
fsGroupChangePolicy: OnRootMismatch
initContainers:
- name: task-executor-installer
image: sandbox-registry.cn-zhangjiakou.cr.aliyuncs.com/opensandbox/task-executor:v0.1.0
command: [/bin/sh, -c]
args:
- cp /workspace/server /opt/opensandbox/task-executor && chmod 0755 /opt/opensandbox/task-executor
volumeMounts:
- name: opensandbox-bin
mountPath: /opt/opensandbox
- name: execd-installer
image: sandbox-registry.cn-zhangjiakou.cr.aliyuncs.com/opensandbox/execd:v1.0.22
command: [/bin/sh, -c]
args:
- cp ./execd /opt/opensandbox/execd && cp ./bootstrap.sh /opt/opensandbox/bootstrap.sh && chmod 0755 /opt/opensandbox/execd /opt/opensandbox/bootstrap.sh
volumeMounts:
- name: opensandbox-bin
mountPath: /opt/opensandbox
containers:
- name: sandbox
image: zot.ad.ddupan.top/panxiao81/gitea-dynamic-runner-runner@sha256:a45875fd2d0e67429b0b7bc3914735581669c705bb4e1a05d712134d2bceb86f
imagePullPolicy: IfNotPresent
command: [/opt/opensandbox/task-executor]
args:
- -listen-addr=0.0.0.0:5758
- -log-dir=/tmp
- -data-dir=/tmp/tasks
env:
- name: SANDBOX_MAIN_CONTAINER
value: sandbox
- name: EXECD_ENVS
value: /opt/opensandbox/.env
- name: EXECD
value: /opt/opensandbox/execd
- name: GITEA_INSTANCE_URL
value: https://git.ddupan.top
- name: HOME
value: /data
- name: DOCKER_HOST
value: unix:///run/docker/docker.sock
ports:
- name: task-executor
containerPort: 5758
securityContext:
allowPrivilegeEscalation: false
capabilities:
drop: [ALL]
runAsNonRoot: true
runAsUser: 2000
runAsGroup: 2000
volumeMounts:
- name: opensandbox-bin
mountPath: /opt/opensandbox
- name: runner-data
mountPath: /data
- name: docker-socket
mountPath: /run/docker
- name: spire-socket
mountPath: /run/spire/agent-sockets
- name: spire-agent
image: ghcr.io/spiffe/spire-agent:1.15.3@sha256:41b0dcd8b258a69db9e2768292a060766fb76fd866e4bc925849981ea1b825ff
args: [-config, /run/spire/config/agent.conf]
securityContext:
allowPrivilegeEscalation: false
capabilities:
drop: [ALL]
volumeMounts:
- name: spire-config
mountPath: /run/spire/config
readOnly: true
- name: spire-bundle
mountPath: /run/spire/bundle
readOnly: true
- name: spire-token
mountPath: /run/spire/tokens
readOnly: true
- name: spire-data
mountPath: /run/spire/data
- name: spire-socket
mountPath: /run/spire/agent-sockets
- name: docker
image: docker.io/library/docker:29.1.5-dind
command: [/bin/sh, -c]
args:
- test -e /dev/kmsg || mknod /dev/kmsg c 1 11; exec dockerd --host=unix:///run/docker/docker.sock --group=2000 --storage-driver=overlay2
securityContext:
privileged: true
volumeMounts:
- name: docker-socket
mountPath: /run/docker
- name: docker-data
mountPath: /var/lib/docker
volumes:
- name: opensandbox-bin
emptyDir: {}
- name: runner-data
emptyDir: {}
- name: docker-data
emptyDir: {}
- name: docker-socket
emptyDir:
medium: Memory
- name: spire-data
emptyDir:
medium: Memory
- name: spire-socket
emptyDir:
medium: Memory
- name: spire-config
configMap:
name: gitea-ci-spire-agent
- name: spire-bundle
configMap:
name: spire-bundle-pem
- name: spire-token
projected:
sources:
- serviceAccountToken:
audience: spire-server
expirationSeconds: 3600
path: token
platform/sandbox-ci-runners/rbac.yaml
-87
View File
@@ -1,87 +0,0 @@
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: gitea-ci
namespace: opensandbox
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: opensandbox-identity
namespace: opensandbox
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
name: opensandbox-identity
namespace: opensandbox
rules:
- apiGroups: [""]
resources: [pods]
verbs: [get, list, watch]
- apiGroups: [sandbox.opensandbox.io]
resources: [batchsandboxes]
verbs: [get, list, watch]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: opensandbox-identity
namespace: opensandbox
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: Role
name: opensandbox-identity
subjects:
- kind: ServiceAccount
name: opensandbox-identity
namespace: opensandbox
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: opensandbox-identity
rules:
- apiGroups: [spire.spiffe.io]
resources: [clusterstaticentries]
verbs: [create, delete, get, list, watch]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: opensandbox-identity
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: opensandbox-identity
subjects:
- kind: ServiceAccount
name: opensandbox-identity
namespace: opensandbox
---
# The central SPIRE external controller uses this existing sandbox credential
# to publish a PEM bundle for guest-local Agents. It cannot read Secrets here.
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
name: spire-runner-bundle-publisher
namespace: opensandbox
rules:
- apiGroups: [""]
resources: [configmaps]
verbs: [create, delete, get, list, patch, update, watch]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: spire-runner-bundle-publisher
namespace: opensandbox
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: Role
name: spire-runner-bundle-publisher
subjects:
- kind: ServiceAccount
name: spire-controller-manager
namespace: spire-system
platform/sandbox-ci-runners/service.yaml
-15
View File
@@ -1,15 +0,0 @@
apiVersion: v1
kind: Service
metadata:
name: opensandbox-server-internal
namespace: opensandbox-system
spec:
type: NodePort
selector:
app.kubernetes.io/instance: opensandbox
app.kubernetes.io/name: opensandbox-server
ports:
- name: http
port: 80
targetPort: http
nodePort: 30080
platform/sandbox-external-secrets/README.md
-56
View File
@@ -1,56 +0,0 @@
# Sandbox External Secrets Operator
本目录在 sandbox 集群部署独立的 External Secrets Operator `2.8.0`，并通过
`ClusterSecretStore/openbao` 读取 OpenBao KV v2 中共享的 OpenSandbox API key。它不复用
homelab 集群的 ESO Pod、ServiceAccount 或 Kubernetes auth backend。
## 当前状态
- OpenBao `auth/kubernetes-sandbox`、backend config、`external-secrets` role 与
`sandbox-external-secrets` policy 已于 2026-09-18 由 Terraform 创建；apply 后 plan
为 zero-diff；
- `kv/k8s/opensandbox-api` 已由本机 `spiffe://ddupan.top/dev/panxiao81` 身份生成并写入，
值未输出或落盘；
- sandbox ESO operator、`ClusterSecretStore/openbao` 与 OpenSandbox `ExternalSecret`
由 Flux 管理；
- 线上 `ClusterSecretStore/openbao` 为 `Valid/Ready`，`ExternalSecret/opensandbox-api-key`
为 `SecretSynced/Ready`；
- OpenSandbox 已切换到 API key：无 key 请求返回 `401`，正确 key 请求返回 `200`；
- homelab runner 对同一 key 的投影不在本目录，留给 runner 项目管理。
OpenBao 的 `auth/kubernetes-sandbox`、对应 role、policy、sandbox API 地址与公开 CA
完全由 `infrastructure/openbao/terraform/` 管理。CA 文件提交到 Git 是刻意设计：CA
是公开信任材料，版本化后集群重建造成的 trust root 变化会产生可审计的 Terraform diff。
ESO 使用 TokenRequest 生成短期 ServiceAccount JWT。OpenBao 未配置长期
`token_reviewer_jwt`，而是使用登录 JWT 调用 sandbox TokenReview；因此
`external-secrets` ServiceAccount 仅额外绑定内建 `system:auth-delegator`。
## 重建顺序
1. 在 OpenBao 写入 OpenSandbox API key：
```bash
openssl rand -hex 32 | bao kv put kv/k8s/opensandbox-api api_key=-
```
2. 在 `infrastructure/openbao/terraform` 执行 `terraform plan` 和 `terraform apply`，
创建 `kubernetes-sandbox` auth mount、backend config、role 与只允许读取
`kv/k8s/opensandbox-api` 的最小权限 policy。
3. 合并 Flux 变更。依次等待 `flux-system/external-secrets-operator`、
`flux-system/external-secrets` Ready，再等待 `flux-system/opensandbox` 滚动完成。
operator 与配置拆成两个 Flux Kustomization，确保全新集群先安装 CRD，再声明
`ClusterSecretStore`；不要为了减少目录而把两层重新合并。
## 验收
```bash
kubectl get clustersecretstore openbao
kubectl -n opensandbox-system get externalsecret opensandbox-api-key
kubectl -n opensandbox-system get secret opensandbox-api-key
```
只检查 Secret 是否存在及 key 名，不输出 `data`。`ClusterSecretStore` 或
`ExternalSecret` 不 Ready 时，先检查 `auth/kubernetes-sandbox`，不要临时创建静态
Bao token Secret。
platform/sandbox-external-secrets/config/auth-delegator.yaml
-13
View File
@@ -1,13 +0,0 @@
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: sandbox-external-secrets-token-review
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: system:auth-delegator
subjects:
- kind: ServiceAccount
name: external-secrets
namespace: external-secrets
platform/sandbox-external-secrets/config/clustersecretstore.yaml
-18
View File
@@ -1,18 +0,0 @@
---
apiVersion: external-secrets.io/v1
kind: ClusterSecretStore
metadata:
name: openbao
spec:
provider:
vault:
server: https://bao.ad.ddupan.top:8200
path: kv
version: v2
auth:
kubernetes:
mountPath: kubernetes-sandbox
role: external-secrets
serviceAccountRef:
name: external-secrets
namespace: external-secrets
platform/sandbox-external-secrets/operator/helmrelease.yaml
-35
View File
@@ -1,35 +0,0 @@
---
apiVersion: helm.toolkit.fluxcd.io/v2
kind: HelmRelease
metadata:
name: external-secrets
namespace: external-secrets
spec:
chart:
spec:
chart: external-secrets
interval: 1h
sourceRef:
kind: HelmRepository
name: external-secrets
version: 2.8.0
driftDetection:
mode: enabled
install:
crds: CreateReplace
strategy:
name: RetryOnFailure
retryInterval: 5m
interval: 30m
releaseName: external-secrets
targetNamespace: external-secrets
timeout: 10m
upgrade:
crds: CreateReplace
strategy:
name: RetryOnFailure
retryInterval: 5m
valuesFrom:
- kind: ConfigMap
name: external-secrets-values
valuesKey: values.yaml
platform/sandbox-external-secrets/operator/kustomization.yaml
-8
View File
@@ -1,8 +0,0 @@
---
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- namespace.yaml
- repository.yaml
- values.yaml
- helmrelease.yaml
platform/sandbox-external-secrets/operator/namespace.yaml
-5
View File
@@ -1,5 +0,0 @@
---
apiVersion: v1
kind: Namespace
metadata:
name: external-secrets
platform/sandbox-external-secrets/operator/repository.yaml
-9
View File
@@ -1,9 +0,0 @@
---
apiVersion: source.toolkit.fluxcd.io/v1
kind: HelmRepository
metadata:
name: external-secrets
namespace: external-secrets
spec:
interval: 1h
url: https://charts.external-secrets.io
platform/sandbox-external-secrets/operator/values.yaml
-31
View File
@@ -1,31 +0,0 @@
---
apiVersion: v1
kind: ConfigMap
metadata:
name: external-secrets-values
namespace: external-secrets
data:
values.yaml: |
replicaCount: 1
webhook:
replicaCount: 1
resources:
requests: {cpu: 10m, memory: 32Mi}
limits: {memory: 128Mi}
certController:
replicaCount: 1
resources:
requests: {cpu: 10m, memory: 32Mi}
limits: {memory: 128Mi}
resources:
requests: {cpu: 10m, memory: 64Mi}
limits: {memory: 256Mi}
serviceAccount:
create: true
name: external-secrets
installCRDs: true
platform/sandbox-kata/README.md
+2 -28
View File
@@ -28,32 +28,6 @@ Pod 删除后的 backing-file/VMM 回收和真实构建基准必须作为上线
由 `VMPodScrape` 写入中央 VictoriaMetrics。它不拥有 Kubernetes API 凭据或 host 写
权限。
## Guest 内 SPIFFE 身份
Kata guest 不能直接使用 node SPIRE Agent 的 CSI socket。Unix socket 的路径即使通过
virtio-fs 出现在 guest 中，连接也不能跨 VM 边界。CI Pod 应以 native sidecar 在同一
guest 内启动临时 SPIRE Agent，并满足以下约束：
1. 外层 Pod 挂载 `audience=spire-server` 的 Pod-bound projected ServiceAccount token；
2. 内层 Agent 使用 `k8s_psat` 向中央 Server attestation，Server cluster profile 必须
启用 `use_pod_uid_for_agent_id`，使 Agent ID 包含 Pod UID；
3. 调度器为该具体 Agent 创建 registration entry；SPIFFE ID 使用仓库和任务名等稳定的
业务语义，Pod UID 只作为 parent binding，不进入业务身份；
4. Agent 与 workload 通过 guest 内 `emptyDir` 上的 Unix socket 通信；Pod 必须设置
`shareProcessNamespace: true`，否则 `unix` workload attestor 无法从共享 `/proc`
解析客户端的 `SO_PEERCRED` PID；
5. 调度器必须在 registration entry 已同步到 Agent 后才放行 workload。Pod 删除后同步
删除 entry，并 evict 对应的临时 Agent 记录。
2026-09-17 的 live PoC 已验证：以 Pod UID 作为 parent 的临时 Agent 成功注册，
`unix:uid:2000` workload 从 guest-local socket 获得
`spiffe://ddupan.top/ci/poc/task/build` X.509-SVID，Pod 正常退出。PoC 资源随后全部清理，
中央 SPIRE 恢复声明式配置。
SPIRE 1.15.3 已支持 `use_pod_uid_for_agent_id`，但当前使用的 hardened chart 0.30.2
尚未把它暴露到 values/template。正式部署应先给 chart 补齐该字段并向上游提交，随后
采用包含修复的 release；不得把手工修改 Server ConfigMap 作为运行方案。
## 上线验收
Flux reconciliation 完成后至少确认：
@@ -61,8 +35,8 @@ Flux reconciliation 完成后至少确认：
1. 两个节点重新回到 Ready，`RuntimeClass/kata-clh-runtime-rs` 存在；
2. Kata Pod 内核与 LXC host 内核不同，且 `/dev/kvm` 可用；
3. Cloud Hypervisor API 的 `vm.info.config.memory.shared` 为 `true`；
4. Kata Pod 内层 Agent 的 ID 包含该 Pod UID；只有调度器创建的业务 entry 所匹配的
workload UID 能从 guest-local socket 获得预期 SPIFFE ID；
4. `spire-smoke` ServiceAccount 的 Kata Pod 可获得
`spiffe://ddupan.top/sandbox/smoke`，错误 ServiceAccount 无法获得身份；
5. block-backed `emptyDir` 上 Docker 使用 `overlay2`，BuildKit 与 kind smoke test
通过；kind 的 dockerd bootstrap 需要先在 guest 内执行
`mknod /dev/kmsg c 1 11`；
platform/sandbox-external-secrets/config/kustomization.yaml → platform/sandbox-opensandbox-pools/kustomization.yaml
+1 -3
View File
@@ -1,6 +1,4 @@
---
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- auth-delegator.yaml
- clustersecretstore.yaml
- pools.yaml
platform/sandbox-opensandbox-pools/pools.yaml
+123
View File
@@ -0,0 +1,123 @@
---
apiVersion: sandbox.opensandbox.io/v1alpha1
kind: Pool
metadata:
name: ci-pod
namespace: opensandbox
spec:
template:
metadata:
labels:
ci.ddupan.top/backend: pod
spec:
containers:
- name: sandbox
image: sandbox-registry.cn-zhangjiakou.cr.aliyuncs.com/opensandbox/code-interpreter:v1.1.0
command: [/opt/opensandbox/task-executor]
args: [-listen-addr=0.0.0.0:5758, -log-dir=/tmp]
env:
- name: SANDBOX_MAIN_CONTAINER
value: sandbox
- name: EXECD_ENVS
value: /opt/opensandbox/.env
- name: EXECD
value: /opt/opensandbox/execd
resources:
requests:
cpu: 100m
memory: 256Mi
limits:
cpu: "2"
memory: 4Gi
volumeMounts:
- name: opensandbox-bin
mountPath: /opt/opensandbox
- name: sandbox-storage
mountPath: /var/lib/sandbox
initContainers:
- name: task-executor-installer
image: sandbox-registry.cn-zhangjiakou.cr.aliyuncs.com/opensandbox/task-executor:v0.1.0
command: [/bin/sh, -c]
args: [cp /workspace/server /opt/opensandbox/task-executor && chmod 0755 /opt/opensandbox/task-executor]
volumeMounts:
- name: opensandbox-bin
mountPath: /opt/opensandbox
- name: execd-installer
image: sandbox-registry.cn-zhangjiakou.cr.aliyuncs.com/opensandbox/execd:v1.0.22
command: [/bin/sh, -c]
args: [cp ./execd /opt/opensandbox/execd && cp ./bootstrap.sh /opt/opensandbox/bootstrap.sh && chmod 0755 /opt/opensandbox/execd /opt/opensandbox/bootstrap.sh]
volumeMounts:
- name: opensandbox-bin
mountPath: /opt/opensandbox
volumes:
- name: opensandbox-bin
emptyDir: {}
- name: sandbox-storage
emptyDir: {}
capacitySpec:
bufferMax: 1
bufferMin: 0
poolMax: 4
poolMin: 0
---
apiVersion: sandbox.opensandbox.io/v1alpha1
kind: Pool
metadata:
name: ci-vm
namespace: opensandbox
spec:
template:
metadata:
labels:
ci.ddupan.top/backend: vm
spec:
runtimeClassName: kata-clh-runtime-rs
containers:
- name: sandbox
image: sandbox-registry.cn-zhangjiakou.cr.aliyuncs.com/opensandbox/code-interpreter:v1.1.0
command: [/opt/opensandbox/task-executor]
args: [-listen-addr=0.0.0.0:5758, -log-dir=/tmp]
env:
- name: SANDBOX_MAIN_CONTAINER
value: sandbox
- name: EXECD_ENVS
value: /opt/opensandbox/.env
- name: EXECD
value: /opt/opensandbox/execd
resources:
requests:
cpu: 250m
memory: 512Mi
limits:
cpu: "4"
memory: 8Gi
volumeMounts:
- name: opensandbox-bin
mountPath: /opt/opensandbox
- name: sandbox-storage
mountPath: /var/lib/sandbox
initContainers:
- name: task-executor-installer
image: sandbox-registry.cn-zhangjiakou.cr.aliyuncs.com/opensandbox/task-executor:v0.1.0
command: [/bin/sh, -c]
args: [cp /workspace/server /opt/opensandbox/task-executor && chmod 0755 /opt/opensandbox/task-executor]
volumeMounts:
- name: opensandbox-bin
mountPath: /opt/opensandbox
- name: execd-installer
image: sandbox-registry.cn-zhangjiakou.cr.aliyuncs.com/opensandbox/execd:v1.0.22
command: [/bin/sh, -c]
args: [cp ./execd /opt/opensandbox/execd && cp ./bootstrap.sh /opt/opensandbox/bootstrap.sh && chmod 0755 /opt/opensandbox/execd /opt/opensandbox/bootstrap.sh]
volumeMounts:
- name: opensandbox-bin
mountPath: /opt/opensandbox
volumes:
- name: opensandbox-bin
emptyDir: {}
- name: sandbox-storage
emptyDir: {}
capacitySpec:
bufferMax: 1
bufferMin: 0
poolMax: 2
poolMin: 0
platform/sandbox-opensandbox/README.md
+15 -40
View File
@@ -1,44 +1,19 @@
# Sandbox OpenSandbox
# OpenSandbox
本目录在独立 sandbox k3s 集群部署 OpenSandbox controller、server 与 CRD。Flux 从
上游 commit `8f01e935c2cabba778cf37a152033fae062fa0f4` 构建官方 umbrella chart
`0.2.2`；该源码渲染结果已与 release `opensandbox-0.2.2.tgz` 对比一致。不要改为跟随
浮动 branch 或 tag。
Flux installs the upstream all-in-one OpenSandbox chart pinned to
`helm/opensandbox/0.2.2` (`8f01e935`). The API is cluster-internal and intentionally runs a
single replica until shared server state and HA behaviour have been validated.
server 只提供集群内 `opensandbox-server.opensandbox-system.svc:80` ClusterIP，不部署
Gateway、Ingress 或 LoadBalancer。sandbox workload 位于 `opensandbox` namespace，
默认使用 `kata-clh-runtime-rs`；CI Pool、runner 镜像、动态 SPIFFE registration 均由
runner 项目后续声明，本目录不预制。
`ci-pod` uses `runc`; `ci-vm` uses the separately managed
`kata-clh-runtime-rs` RuntimeClass. Both Pools start at zero and create capacity
on demand. They currently use the upstream interpreter image to validate the
Lifecycle API and Pool allocation independently of the CI scheduler cutover.
## API 认证
The dynamic runner worker, runner image, guest-local SPIRE Agent and Docker
sidecar are introduced only after this layer is Ready. In particular, do not
mount the host SPIFFE CSI socket into `ci-vm`: Unix sockets do not cross the
Kata VM boundary.
`ExternalSecret/opensandbox-api-key` 从 OpenBao
`kv/k8s/opensandbox-api:api_key` 投影同名 Secret。这个路径不归属于某个 Kubernetes
集群：sandbox server 与 homelab runner 调度器分别通过自己的 Bao 身份读取。server 只通过
`secretKeyRef` 读取：
```yaml
- name: OPENSANDBOX_SERVER_API_KEY
valueFrom:
secretKeyRef:
name: opensandbox-api-key
key: api-key
```
仓库与 Helm values 均不保存 API key。OpenSandbox 不支持更丰富的原生 workload
authentication；runner 后续读取同一 Bao 路径并在请求头中使用 API key。
`opensandbox-values` 带 Flux watch label，values 变化会立即触发 Helm reconcile，不依赖
30 分钟的 HelmRelease interval。
## 验收
合并后等待 `flux-system/opensandbox` 与 `opensandbox-system/opensandbox` Ready，并确认：
```bash
kubectl get crd batchsandboxes.sandbox.opensandbox.io pools.sandbox.opensandbox.io
kubectl -n opensandbox-system get deploy,pod,svc
kubectl get runtimeclass kata-clh-runtime-rs
```
控制面上线不创建 CI Pool，也不产生 sandbox workload。首个 runner 集成应另行提交 Pool
与完整的 Lifecycle API smoke test。
Smoke test both backends through the same API by creating sandboxes with
`extensions.poolRef` set to `ci-pod` and `ci-vm`, then confirm their
BatchSandboxes, Pods and VMMs disappear after deletion.
platform/sandbox-opensandbox/external-secret.yaml
-19
View File
@@ -1,19 +0,0 @@
---
apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
name: opensandbox-api-key
namespace: opensandbox-system
spec:
refreshInterval: 1h
secretStoreRef:
kind: ClusterSecretStore
name: openbao
target:
name: opensandbox-api-key
creationPolicy: Owner
data:
- secretKey: api-key
remoteRef:
key: k8s/opensandbox-api
property: api_key
platform/sandbox-opensandbox/helmrelease.yaml
-4
View File
@@ -1,4 +1,3 @@
---
apiVersion: helm.toolkit.fluxcd.io/v2
kind: HelmRelease
metadata:
@@ -16,7 +15,6 @@ spec:
driftDetection:
mode: enabled
install:
crds: CreateReplace
strategy:
name: RetryOnFailure
retryInterval: 5m
@@ -25,11 +23,9 @@ spec:
targetNamespace: opensandbox-system
timeout: 15m
upgrade:
crds: CreateReplace
strategy:
name: RetryOnFailure
retryInterval: 5m
valuesFrom:
- kind: ConfigMap
name: opensandbox-values
valuesKey: values.yaml
platform/sandbox-opensandbox/kustomization.yaml
+1 -5
View File
@@ -1,11 +1,7 @@
---
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- namespace.yaml
- external-secret.yaml
- namespaces.yaml
- repository.yaml
- template.yaml
- values.yaml
- helmrelease.yaml
- monitor-scrape.yaml
platform/sandbox-opensandbox/monitor-scrape.yaml
-16
View File
@@ -1,16 +0,0 @@
---
apiVersion: operator.victoriametrics.com/v1beta1
kind: VMPodScrape
metadata:
name: opensandbox-controller
namespace: monitoring
spec:
namespaceSelector:
matchNames:
- opensandbox-system
podMetricsEndpoints:
- interval: 30s
port: metrics
selector:
matchLabels:
app.kubernetes.io/name: opensandbox
platform/sandbox-opensandbox/namespace.yaml
-14
View File
@@ -1,14 +0,0 @@
---
apiVersion: v1
kind: Namespace
metadata:
name: opensandbox-system
---
apiVersion: v1
kind: Namespace
metadata:
name: opensandbox
labels:
pod-security.kubernetes.io/enforce: privileged
pod-security.kubernetes.io/audit: restricted
pod-security.kubernetes.io/warn: restricted
platform/sandbox-opensandbox/namespaces.yaml
+10
View File
@@ -0,0 +1,10 @@
---
apiVersion: v1
kind: Namespace
metadata:
name: opensandbox-system
---
apiVersion: v1
kind: Namespace
metadata:
name: opensandbox
platform/sandbox-opensandbox/repository.yaml
+3 -3
View File
@@ -1,4 +1,3 @@
---
apiVersion: source.toolkit.fluxcd.io/v1
kind: GitRepository
metadata:
@@ -7,5 +6,6 @@ metadata:
spec:
interval: 1h
ref:
commit: 8f01e935c2cabba778cf37a152033fae062fa0f4
url: https://github.com/opensandbox-group/OpenSandbox.git
tag: helm/opensandbox/0.2.2
timeout: 60s
url: https://github.com/alibaba/OpenSandbox.git
platform/sandbox-opensandbox/template.yaml
-17
View File
@@ -1,17 +0,0 @@
---
apiVersion: v1
kind: ConfigMap
metadata:
name: opensandbox-batchsandbox-template
namespace: opensandbox-system
data:
batchsandbox-template.yaml: |
metadata:
labels:
ddupan.top/workload-class: sandbox
spec:
replicas: 1
template:
spec:
restartPolicy: Never
terminationGracePeriodSeconds: 30
platform/sandbox-opensandbox/values.yaml
+14 -35
View File
@@ -1,52 +1,41 @@
---
apiVersion: v1
kind: ConfigMap
metadata:
name: opensandbox-values
namespace: opensandbox-system
labels:
reconcile.fluxcd.io/watch: Enabled
data:
values.yaml: |
opensandbox-controller:
controller:
logLevel: info
replicaCount: 1
metrics:
enabled: true
port: 8080
secure: false
port: 8080
resources:
requests:
cpu: 25m
memory: 64Mi
limits:
cpu: 500m
memory: 256Mi
opensandbox-server:
server:
replicaCount: 1
env:
- name: OPENSANDBOX_SERVER_API_KEY
valueFrom:
secretKeyRef:
name: opensandbox-api-key
key: api-key
resources:
limits:
cpu: "1"
memory: 1Gi
requests:
cpu: 100m
memory: 256Mi
volumeMounts:
- name: batchsandbox-template
mountPath: /etc/opensandbox/batchsandbox-template.yaml
subPath: batchsandbox-template.yaml
readOnly: true
volumes:
- name: batchsandbox-template
configMap:
name: opensandbox-batchsandbox-template
limits:
cpu: "1"
memory: 1Gi
configToml: |
[server]
host = "0.0.0.0"
port = 80
api_key = ""
max_sandbox_timeout_seconds = 86400
[log]
level = "INFO"
@@ -55,10 +44,6 @@ data:
type = "kubernetes"
execd_image = "sandbox-registry.cn-zhangjiakou.cr.aliyuncs.com/opensandbox/execd:v1.0.22"
[storage]
allowed_host_paths = []
volume_default_size = "1Gi"
[kubernetes]
kubeconfig_path = ""
namespace = "opensandbox"
@@ -67,17 +52,11 @@ data:
informer_watch_timeout_seconds = 60
snapshot_create_timeout_seconds = 900
workload_provider = "batchsandbox"
image_pull_policy = "IfNotPresent"
batchsandbox_template_file = "/etc/opensandbox/batchsandbox-template.yaml"
batchsandbox_template_file = "/etc/opensandbox/example.batchsandbox-template.yaml"
[egress]
image = "sandbox-registry.cn-zhangjiakou.cr.aliyuncs.com/opensandbox/egress:v1.1.6"
mode = "dns+nft"
disable_ipv6 = true
[secure_runtime]
type = "kata"
k8s_runtime_class = "kata-clh-runtime-rs"
opensandbox-node-agent:
enabled: false
platform/spire/README.md
-6
View File
@@ -74,12 +74,6 @@ Agent 已启用 Unix workload attestor，并为本机用户 `panxiao81`（UID `1
export SPIFFE_ENDPOINT_SOCKET=unix:///run/spire/agent-sockets/spire-agent.sock
```
宿主机已安装与 Agent Pod 同版本的 `/usr/local/bin/spire-agent` 1.15.3，供本地进程从
Workload API 获取 JWT-SVID。二进制来自 SPIRE 官方 `linux-amd64-musl` release，安装时
核对 tarball SHA-256
`ca1a4d1155317bdd2afc7f36663828a10410c7c840e54725b90b4064b0a301c7`。升级 chart 时应
同步升级这个 CLI 并重新核对官方 checksum，不能长期混用版本。
该身份仅按 Unix UID 匹配，不是 SPIRE admin，也不会匹配 `sudo` 后以 root 运行的
进程。`ClusterStaticEntry.spec.parentID` 绑定当前 `laptop` Kubernetes node UID；若
节点被删除后重建，需从 `spire-server agent list` 取得新 Agent ID 并同步更新该字段。
platform/spire/values.yaml
-17
View File
@@ -43,10 +43,6 @@ spire-server:
externalSecret:
name: spire-external-kubeconfigs
key: sandbox-controller
sandbox-runner-bundle:
externalSecret:
name: spire-external-kubeconfigs
key: sandbox-controller
nodeAttestor:
externalK8sPSAT:
enabled: true
@@ -59,19 +55,12 @@ spire-server:
kubeConfigName: sandbox
serviceAccountAllowList:
- spire-smoke:spire-smoke
- opensandbox:gitea-ci
usePodUIDForAgentID: true
externalControllerManagers:
enabled: true
clusters:
sandbox:
kubeConfigName: sandbox-controller
# Dynamic OpenSandbox runner identities are exact Pod-UID-bound
# ClusterStaticEntries created in the sandbox cluster. The chart
# defaults this reconciler to false, so enable the central registration
# path explicitly.
reconcile:
clusterStaticEntries: true
bundlePublisher:
externalK8sConfigMap:
enabled: true
@@ -82,12 +71,6 @@ spire-server:
configMapName: spire-bundle
configMapKey: bundle.spiffe
format: spiffe
sandbox-runner-bundle:
kubeConfigName: sandbox-runner-bundle
namespace: opensandbox
configMapName: spire-bundle-pem
configMapKey: bundle.pem
format: pem
persistence:
# PostgreSQL stores registrations, but the disk KeyManager still needs durable
# storage for the trust-domain signing keys.