Compare commits
99
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
e325a703dd | ||
|
|
130849040c
|
||
|
|
bcd034355e | ||
|
|
506e0cb983
|
||
|
|
c07709e078 | ||
|
|
e1979ebbb8
|
||
|
|
b02b2aec96 | ||
|
|
ff92037a28
|
||
|
|
9360eb2ea0 | ||
|
|
0c7eb371e5
|
||
|
|
757c21fc0b | ||
|
|
c8e163f01e
|
||
|
|
59fe28faa5 | ||
|
|
69c90b9fdf
|
||
|
|
a6a861cc7e | ||
|
|
5c9a09edde
|
||
|
|
cfb9c2d150 | ||
|
|
1ddec34149
|
||
|
|
db63503ec9 | ||
|
|
9953fae93d
|
||
|
|
56598d6c84 | ||
|
|
834fdcb771
|
||
|
|
47ff73cefb | ||
|
|
bb24c76704
|
||
|
|
b0791e4095 | ||
|
|
26179f700f
|
||
|
|
1ac99f4476 | ||
|
|
891c6b90e7
|
||
|
|
c2314256cc | ||
|
|
8244cdad79
|
||
|
|
23841556d3 | ||
|
|
4daaa38495
|
||
|
|
da0630cd29 | ||
|
|
d0fcfdeddc
|
||
|
|
ecf395cc8f | ||
|
|
9f646d51e7
|
||
|
|
5c7e7ff8db | ||
|
|
d6b97355d6
|
||
|
|
b82ba4d5a0 | ||
|
|
9dc7aabc94
|
||
|
|
b7c95fd0b8 | ||
|
|
098ff4e27f
|
||
|
|
dfc55e1d9e
|
||
|
|
18d9f83413 | ||
|
|
e5e2ab1b89
|
||
|
|
0b236c43d9 | ||
|
|
762ce6458d
|
||
|
|
15ef512cfc | ||
|
|
a13a5118cf
|
||
|
|
25bc7ee7d6 | ||
|
|
3d41e0e29b
|
||
|
|
3169fcabd3 | ||
|
|
0a42a3ad04
|
||
|
|
9ec034c947 | ||
|
|
ad9b305aba
|
||
|
|
9696df1f51 | ||
|
|
a335bdbc38
|
||
|
|
b6623069ad | ||
|
|
4e3ef3c33f
|
||
|
|
8eac82c272 | ||
|
|
2b232c4cd2
|
||
|
|
b51dc3751b | ||
|
|
616c87bafd
|
||
|
|
0663642583 | ||
|
|
68191e0ff4
|
||
|
|
c368373e74 | ||
|
|
7bdd152db9
|
||
|
|
ab985c7979 | ||
|
|
cb137f2044
|
||
|
|
83e4ac2f01 | ||
|
|
1df682bc6d | ||
|
|
eb52197eee
|
||
|
|
9b9f44333e | ||
|
|
8896d58b89
|
||
|
|
0fd7f87d7e | ||
|
|
89bd7864e6
|
||
|
|
93f803a0d9 | ||
|
|
f7d9579221 | ||
|
|
9fe910a4c9 | ||
|
|
28c20fd3e5 | ||
|
|
110dbcc2c3
|
||
|
|
28f1de35cb | ||
|
|
ad4ed1f0d6 | ||
|
|
611a5a3d3a
|
||
|
|
7582990655
|
||
|
|
ec3ce94e12 | ||
|
|
6eb4875db6
|
||
|
|
518dba6036 | ||
|
|
4c823f8181
|
||
|
|
8290082fb4 | ||
|
|
159a75b710
|
||
|
|
e0e629794b
|
||
|
|
90ba945d85 | ||
|
|
d526fd75d3
|
||
|
|
18cb2858b9 | ||
|
|
99d1ec1d6f | ||
|
|
583dab526a
|
||
|
|
3dbd4c5f31
|
||
|
|
e67bce5121
|
@@ -0,0 +1,157 @@
|
|||||||
|
# Nexus Repository POC
|
||||||
|
|
||||||
|
本目录声明一个 Nexus Repository Community Edition POC,用来验证一次性 CI runner 通过
|
||||||
|
网络服务复用 Ansible Galaxy、Go Modules 与 OCI/BuildKit 缓存。Nexus 固定为 `3.96.1`,
|
||||||
|
镜像固定到官方 multi-arch index digest;LAN 入口为
|
||||||
|
`https://nexus.ad.ddupan.top`。
|
||||||
|
|
||||||
|
## POC 边界
|
||||||
|
|
||||||
|
- 单副本 Deployment,`Recreate` 更新,避免一个 RWO 卷被两个 Pod 同时挂载。
|
||||||
|
- `/nexus-data` 使用 `localpv-zfs-ceph` 上的 50 GiB RWO PVC。
|
||||||
|
- 资源预算为 250m/2 GiB request、2 CPU/4 GiB limit;JVM heap 上限 2 GiB。
|
||||||
|
- 使用容器默认的 embedded H2。它只用于 POC;正式接管 OCI 制品前必须迁移到外部
|
||||||
|
PostgreSQL,并验证备份恢复。
|
||||||
|
- 入口只在 LAN wildcard Gateway 上发布,不创建公网 DNS 或 Cloudflare route。
|
||||||
|
- 不套 Authelia forward-auth;它会破坏 Go、Ansible 与 OCI 非浏览器客户端。
|
||||||
|
- 现有 zot 保持不变。Nexus 完成 OCI、BuildKit cache 和恢复验收前不得迁移或删除 zot。
|
||||||
|
|
||||||
|
Terraform provider 创建 `ansible-public`、`go-public`、最小匿名权限与 OCI Bearer Token
|
||||||
|
Realm。Nexus 3.94 才加入的原生 OCI repository 已有 REST API,但当前锁定的 community
|
||||||
|
provider 尚未暴露 OCI resource;`terraform/reconcile-oci.sh` 因此根据 3.96.1 实例 Swagger
|
||||||
|
固定的 JSON schema,幂等调和 `oci-hosted`、`oci-proxy` 与 `oci-public`。不得绕过该入口在
|
||||||
|
UI 中创建无人管理的长期 repository。
|
||||||
|
|
||||||
|
## 部署
|
||||||
|
|
||||||
|
Flux 从 `clusters/homelab/apps/nexus.yaml` 协调本目录,并依赖 Envoy Gateway 与 OpenEBS。
|
||||||
|
合并前只渲染配置,不直接 apply:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
kubectl kustomize apps/nexus
|
||||||
|
```
|
||||||
|
|
||||||
|
合并并由 Flux 部署后检查:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
kubectl -n flux-system get kustomization nexus
|
||||||
|
kubectl -n nexus get pod,pvc,service,httproute
|
||||||
|
kubectl -n nexus logs deployment/nexus --tail=100
|
||||||
|
```
|
||||||
|
|
||||||
|
启动可能需要数分钟,startup probe 允许最多十分钟。不要因初次启动较慢反复删除 Pod;
|
||||||
|
先确认 PVC 已 Bound、Pod 没有 OOM,以及日志仍在推进。
|
||||||
|
|
||||||
|
DNS 期望状态已加入 `infrastructure/dns/records.yml`,需从 Samba AD Ansible root 以
|
||||||
|
`--check --diff` 核对后再按其 README 应用 DNS tag。没有 DNS 时可先用 port-forward
|
||||||
|
验证应用,但不能据此宣称 Gateway 路径已通过。
|
||||||
|
|
||||||
|
## 首次初始化与 Terraform
|
||||||
|
|
||||||
|
初始管理员密码生成在 PVC 的 `/nexus-data/admin.password`。只在交互式终端中读取并立即
|
||||||
|
完成首次密码轮换;不得把密码复制进 shell tracing、工单、Git 或命令参数。随后将
|
||||||
|
Terraform 管理账号的凭据存入 OpenBao,由 CI 通过 Terraform input variable 注入以下
|
||||||
|
环境变量:
|
||||||
|
|
||||||
|
```text
|
||||||
|
TF_VAR_nexus_url=https://nexus.ad.ddupan.top
|
||||||
|
TF_VAR_nexus_username=admin
|
||||||
|
TF_VAR_nexus_password=<OpenBao kv/infra/nexus 的 admin_password 字段>
|
||||||
|
```
|
||||||
|
|
||||||
|
`terraform/` 使用 `sonatype-nexus-community/sonatyperepo` 1.17.0,当前声明:
|
||||||
|
|
||||||
|
- `ansible-galaxy-proxy` → `https://galaxy.ansible.com`
|
||||||
|
- `ansible-public` group
|
||||||
|
- `go-proxy` → `https://proxy.golang.org`
|
||||||
|
- `go-public` group
|
||||||
|
|
||||||
|
provider credential 不写入 HCL 或 tfvars。正式 apply 前还必须为这个独立 Terraform root
|
||||||
|
配置远端 backend;本地 state 只允许用于可丢弃的 POC,不提交。验证命令:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
terraform -chdir=apps/nexus/terraform init -backend=false
|
||||||
|
terraform -chdir=apps/nexus/terraform validate
|
||||||
|
```
|
||||||
|
|
||||||
|
先以 `--check` 查看 OCI repository 漂移,再明确 apply;脚本只从上述环境变量取得凭据,
|
||||||
|
用临时 `0600` netrc 调用 REST API,退出时删除:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
apps/nexus/terraform/reconcile-oci.sh --check
|
||||||
|
apps/nexus/terraform/reconcile-oci.sh --apply
|
||||||
|
```
|
||||||
|
|
||||||
|
Terraform 同时把内置 `anonymous` 用户从默认的全仓库 `nx-anonymous` 角色收窄到
|
||||||
|
`ansible-public`、其返回制品 URL 使用的 `ansible-galaxy-proxy`、`go-public`,以及
|
||||||
|
`oci-public`/`oci-proxy` 的 `browse/read` 权限。`oci-hosted` 不向匿名用户开放。首次接管
|
||||||
|
已有实例时先执行
|
||||||
|
`terraform import sonatyperepo_user.anonymous anonymous,default`,再 apply;不要先启用默认的
|
||||||
|
全仓库匿名读取。
|
||||||
|
|
||||||
|
2026-09-20 的 POC 现场验收已确认:Flux 与 Pod Ready、PVC Bound、HTTPRoute 通过 HTTPS
|
||||||
|
返回 Nexus 状态 200,Samba DNS 已幂等收敛。全新客户端目录通过匿名入口下载
|
||||||
|
`community.general:11.2.0` 时冷缓存为 8.49 秒、热缓存为 1.89 秒,两次 tarball SHA-256
|
||||||
|
一致;`golang.org/x/[email protected]` 为 2.92 秒与 1.51 秒。
|
||||||
|
|
||||||
|
## 客户端验收
|
||||||
|
|
||||||
|
先验证冷缓存,再原样重复命令验证热缓存;记录 Nexus 请求、上游流量和耗时,不只观察
|
||||||
|
命令成功。Ansible 配置的 URL 必须以 `/` 结尾:
|
||||||
|
|
||||||
|
```ini
|
||||||
|
[galaxy]
|
||||||
|
server_list = nexus
|
||||||
|
|
||||||
|
[galaxy_server.nexus]
|
||||||
|
url = https://nexus.ad.ddupan.top/repository/ansible-public/
|
||||||
|
```
|
||||||
|
|
||||||
|
```bash
|
||||||
|
ansible-galaxy collection install -r collections/requirements.yml \
|
||||||
|
-p .ansible/collections
|
||||||
|
```
|
||||||
|
|
||||||
|
Go POC 使用:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
GOPROXY=https://nexus.ad.ddupan.top/repository/go-public/ go mod download
|
||||||
|
```
|
||||||
|
|
||||||
|
私有 `git.ddupan.top/*` module 的 `GOPRIVATE`、凭据与是否允许 `direct` fallback 在实际
|
||||||
|
workflow 中单独决定;不要让私有 module path 意外发往公共 proxy。
|
||||||
|
|
||||||
|
OCI 使用 path-based routing:匿名公共拉取地址形如
|
||||||
|
`nexus.ad.ddupan.top/oci-public/library/alpine:3.22`,认证写入地址形如
|
||||||
|
`nexus.ad.ddupan.top/oci-hosted/<namespace>/<image>:<tag>`。2026-09-20 现场验收结果:
|
||||||
|
|
||||||
|
- `oci-public` 匿名代理拉取 Alpine 冷缓存 4.75 秒、热缓存 0.80 秒,digest 一致;
|
||||||
|
- `oci-hosted` 认证 push/pull 成功,匿名 pull 返回 401;
|
||||||
|
- amd64/arm64 OCI image index push 成功,两个平台 manifest 可见;
|
||||||
|
- Helm chart push/pull digest 与本地 tarball SHA-256 一致;
|
||||||
|
- Cosign 3.1.3 使用一次性密钥签名并验证成功,OCI 1.1 referrers API 返回一个
|
||||||
|
`application/vnd.dev.sigstore.bundle.v0.3+json` artifact;
|
||||||
|
- BuildKit `registry` cache 以 `mode=max` 导出成功,销毁首个 builder 后由新 builder 导入,
|
||||||
|
两个 `RUN` step 均明确命中 `CACHED`。
|
||||||
|
|
||||||
|
本机安装的测试客户端包括 `/usr/local/bin/cosign` 3.1.3;安装时核对官方 Linux amd64
|
||||||
|
binary SHA-256 `4629c757b7618056f8ddd7e2625ae9fdd94c0372a65049520bc7d9df9efc7f71`。
|
||||||
|
上述结果仍不代表备份恢复、外部 PostgreSQL 或正式 publisher service account 已完成;
|
||||||
|
这些项目通过前不得迁移或删除 zot。
|
||||||
|
|
||||||
|
## 数据与恢复
|
||||||
|
|
||||||
|
POC 的数据库、配置、blob、初始管理员状态都位于 `nexus-data` PVC。删除 Deployment
|
||||||
|
不会删除 PVC;删除 PVC 会永久删除整个 POC。当前没有独立备份,不能将它用于唯一副本的
|
||||||
|
正式制品。
|
||||||
|
|
||||||
|
恢复验证至少包括:停止写入、取得一致备份、在独立 PVC/实例恢复、登录、列出 repository、
|
||||||
|
拉取已缓存的 Ansible/Go 制品,并核对 OCI digest/referrers。正式化时再把数据库迁移至
|
||||||
|
外部 PostgreSQL,并分别定义数据库与 blob 的备份、恢复顺序和 RPO。
|
||||||
|
|
||||||
|
参考:
|
||||||
|
|
||||||
|
- [Nexus OCI repositories](https://help.sonatype.com/en/oci-repositories.html)
|
||||||
|
- [Nexus Ansible repositories](https://help.sonatype.com/en/ansible-repositories.html)
|
||||||
|
- [Nexus Go repositories](https://help.sonatype.com/en/go-repositories.html)
|
||||||
|
- [官方容器镜像](https://hub.docker.com/r/sonatype/nexus3)
|
||||||
@@ -0,0 +1,82 @@
|
|||||||
|
apiVersion: apps/v1
|
||||||
|
kind: Deployment
|
||||||
|
metadata:
|
||||||
|
name: nexus
|
||||||
|
namespace: nexus
|
||||||
|
labels:
|
||||||
|
app.kubernetes.io/name: nexus
|
||||||
|
spec:
|
||||||
|
replicas: 1
|
||||||
|
strategy:
|
||||||
|
type: Recreate
|
||||||
|
selector:
|
||||||
|
matchLabels:
|
||||||
|
app.kubernetes.io/name: nexus
|
||||||
|
template:
|
||||||
|
metadata:
|
||||||
|
labels:
|
||||||
|
app.kubernetes.io/name: nexus
|
||||||
|
spec:
|
||||||
|
automountServiceAccountToken: false
|
||||||
|
securityContext:
|
||||||
|
fsGroup: 200
|
||||||
|
fsGroupChangePolicy: OnRootMismatch
|
||||||
|
runAsGroup: 200
|
||||||
|
runAsNonRoot: true
|
||||||
|
runAsUser: 200
|
||||||
|
seccompProfile:
|
||||||
|
type: RuntimeDefault
|
||||||
|
terminationGracePeriodSeconds: 120
|
||||||
|
containers:
|
||||||
|
- name: nexus
|
||||||
|
image: docker.io/sonatype/nexus3:3.96.1@sha256:56142f13432cf072e017aebb2025f201e42ae36ff40bb82618c702504c61f7dd
|
||||||
|
imagePullPolicy: IfNotPresent
|
||||||
|
env:
|
||||||
|
- name: INSTALL4J_ADD_VM_PARAMS
|
||||||
|
value: >-
|
||||||
|
-Xms1024m -Xmx2048m -XX:MaxDirectMemorySize=1024m
|
||||||
|
-Djava.util.prefs.userRoot=/nexus-data/javaprefs
|
||||||
|
ports:
|
||||||
|
- name: http
|
||||||
|
containerPort: 8081
|
||||||
|
protocol: TCP
|
||||||
|
resources:
|
||||||
|
requests:
|
||||||
|
cpu: 250m
|
||||||
|
memory: 2Gi
|
||||||
|
limits:
|
||||||
|
cpu: "2"
|
||||||
|
memory: 4Gi
|
||||||
|
securityContext:
|
||||||
|
allowPrivilegeEscalation: false
|
||||||
|
capabilities:
|
||||||
|
drop:
|
||||||
|
- ALL
|
||||||
|
startupProbe:
|
||||||
|
httpGet:
|
||||||
|
path: /service/rest/v1/status
|
||||||
|
port: http
|
||||||
|
failureThreshold: 60
|
||||||
|
periodSeconds: 10
|
||||||
|
timeoutSeconds: 5
|
||||||
|
readinessProbe:
|
||||||
|
httpGet:
|
||||||
|
path: /service/rest/v1/status
|
||||||
|
port: http
|
||||||
|
failureThreshold: 6
|
||||||
|
periodSeconds: 10
|
||||||
|
timeoutSeconds: 5
|
||||||
|
livenessProbe:
|
||||||
|
httpGet:
|
||||||
|
path: /service/rest/v1/status
|
||||||
|
port: http
|
||||||
|
failureThreshold: 6
|
||||||
|
periodSeconds: 30
|
||||||
|
timeoutSeconds: 5
|
||||||
|
volumeMounts:
|
||||||
|
- name: data
|
||||||
|
mountPath: /nexus-data
|
||||||
|
volumes:
|
||||||
|
- name: data
|
||||||
|
persistentVolumeClaim:
|
||||||
|
claimName: nexus-data
|
||||||
@@ -0,0 +1,16 @@
|
|||||||
|
apiVersion: gateway.networking.k8s.io/v1
|
||||||
|
kind: HTTPRoute
|
||||||
|
metadata:
|
||||||
|
name: nexus
|
||||||
|
namespace: nexus
|
||||||
|
spec:
|
||||||
|
parentRefs:
|
||||||
|
- name: eg
|
||||||
|
namespace: envoy-gateway-system
|
||||||
|
sectionName: https
|
||||||
|
hostnames:
|
||||||
|
- nexus.ad.ddupan.top
|
||||||
|
rules:
|
||||||
|
- backendRefs:
|
||||||
|
- name: nexus
|
||||||
|
port: 8081
|
||||||
@@ -0,0 +1,9 @@
|
|||||||
|
apiVersion: kustomize.config.k8s.io/v1beta1
|
||||||
|
kind: Kustomization
|
||||||
|
resources:
|
||||||
|
- namespace.yaml
|
||||||
|
- pvc.yaml
|
||||||
|
- deployment.yaml
|
||||||
|
- service.yaml
|
||||||
|
- httproute.yaml
|
||||||
|
- networkpolicy.yaml
|
||||||
@@ -0,0 +1,8 @@
|
|||||||
|
apiVersion: v1
|
||||||
|
kind: Namespace
|
||||||
|
metadata:
|
||||||
|
name: nexus
|
||||||
|
labels:
|
||||||
|
pod-security.kubernetes.io/enforce: restricted
|
||||||
|
pod-security.kubernetes.io/audit: restricted
|
||||||
|
pod-security.kubernetes.io/warn: restricted
|
||||||
@@ -0,0 +1,23 @@
|
|||||||
|
apiVersion: networking.k8s.io/v1
|
||||||
|
kind: NetworkPolicy
|
||||||
|
metadata:
|
||||||
|
name: nexus-ingress
|
||||||
|
namespace: nexus
|
||||||
|
spec:
|
||||||
|
podSelector:
|
||||||
|
matchLabels:
|
||||||
|
app.kubernetes.io/name: nexus
|
||||||
|
policyTypes:
|
||||||
|
- Ingress
|
||||||
|
ingress:
|
||||||
|
- from:
|
||||||
|
- namespaceSelector:
|
||||||
|
matchLabels:
|
||||||
|
kubernetes.io/metadata.name: envoy-gateway-system
|
||||||
|
podSelector:
|
||||||
|
matchLabels:
|
||||||
|
gateway.envoyproxy.io/owning-gateway-name: eg
|
||||||
|
gateway.envoyproxy.io/owning-gateway-namespace: envoy-gateway-system
|
||||||
|
ports:
|
||||||
|
- protocol: TCP
|
||||||
|
port: 8081
|
||||||
@@ -0,0 +1,12 @@
|
|||||||
|
apiVersion: v1
|
||||||
|
kind: PersistentVolumeClaim
|
||||||
|
metadata:
|
||||||
|
name: nexus-data
|
||||||
|
namespace: nexus
|
||||||
|
spec:
|
||||||
|
accessModes:
|
||||||
|
- ReadWriteOnce
|
||||||
|
resources:
|
||||||
|
requests:
|
||||||
|
storage: 50Gi
|
||||||
|
storageClassName: localpv-zfs-ceph
|
||||||
@@ -0,0 +1,14 @@
|
|||||||
|
apiVersion: v1
|
||||||
|
kind: Service
|
||||||
|
metadata:
|
||||||
|
name: nexus
|
||||||
|
namespace: nexus
|
||||||
|
spec:
|
||||||
|
type: ClusterIP
|
||||||
|
selector:
|
||||||
|
app.kubernetes.io/name: nexus
|
||||||
|
ports:
|
||||||
|
- name: http
|
||||||
|
port: 8081
|
||||||
|
protocol: TCP
|
||||||
|
targetPort: http
|
||||||
@@ -0,0 +1,6 @@
|
|||||||
|
.terraform/
|
||||||
|
*.tfstate
|
||||||
|
*.tfstate.*
|
||||||
|
*.tfplan
|
||||||
|
crash.log
|
||||||
|
crash.*.log
|
||||||
Generated
+24
@@ -0,0 +1,24 @@
|
|||||||
|
# This file is maintained automatically by "terraform init".
|
||||||
|
# Manual edits may be lost in future updates.
|
||||||
|
|
||||||
|
provider "registry.terraform.io/sonatype-nexus-community/sonatyperepo" {
|
||||||
|
version = "1.17.0"
|
||||||
|
constraints = "1.17.0"
|
||||||
|
hashes = [
|
||||||
|
"h1:uKhvhhhI7B+HBsh0zq/ybqKt+EnOGyI6rjcRCtj79ZA=",
|
||||||
|
"zh:0dde99e7b343fa01f8eefc378171fb8621bedb20f59157d6cc8e3d46c738105f",
|
||||||
|
"zh:3315929df254a3a6ac27c8c846c2006f7d2a91fadc014351bc4d617f948e5bf9",
|
||||||
|
"zh:36be5a455af3ce4e187de26753de63e78c1ee9a32dba0135c6cf96a6c1fff25f",
|
||||||
|
"zh:3f73f7ff57b8c339a7c7ac37653e2dc0b2dd9dcc3f3a538788e7e3ac838337b2",
|
||||||
|
"zh:40286ecca4c22ab7ae90618ac6d2743f5055199dac81cf5204a4a397c784d439",
|
||||||
|
"zh:4d24e5c0195fb3155b1967583ee64cfeda402d7cc7f3c73369438f6c69f4245b",
|
||||||
|
"zh:828a9d7aceaac36af7f9c07af43ec8d20a89148780645d170ffb1c68b2da792d",
|
||||||
|
"zh:a5ab04de3fe626ec57c832618c6f990abd6610f81e132621651e0b180b970cff",
|
||||||
|
"zh:a959fa6090a8c0f53739879184e7346423494aee598003df0d1ab4a22b2eee91",
|
||||||
|
"zh:bdda26c2f03f918bbe59e75abea44868fafda019c3a543725331195df126350b",
|
||||||
|
"zh:d8048e149ee97ba62971e6a79355d59887bc6d10fcf72cc2feff3d0a2582670c",
|
||||||
|
"zh:dd36f9988af4e1ca5b1ca7b7bb6f658df9a220dfcda7fec7392fedfe9064f652",
|
||||||
|
"zh:dda2688d46c7e539fe97e8fe9d3ec81fb364170e018d9c6a681364c8955d4e9d",
|
||||||
|
"zh:e6b519afe2dea1c0434f766eb6bc9ba78cc5b6ef2c311c2ca3c65cb24744f31f",
|
||||||
|
]
|
||||||
|
}
|
||||||
@@ -0,0 +1,17 @@
|
|||||||
|
{
|
||||||
|
"name": "oci-hosted",
|
||||||
|
"online": true,
|
||||||
|
"storage": {
|
||||||
|
"blobStoreName": "default",
|
||||||
|
"strictContentTypeValidation": true,
|
||||||
|
"writePolicy": "ALLOW",
|
||||||
|
"latestPolicy": false
|
||||||
|
},
|
||||||
|
"oci": {
|
||||||
|
"v1Enabled": false,
|
||||||
|
"forceBasicAuth": false,
|
||||||
|
"pathEnabled": true
|
||||||
|
},
|
||||||
|
"component": { "proprietaryComponents": false },
|
||||||
|
"cosign": { "enforcement": "NONE" }
|
||||||
|
}
|
||||||
@@ -0,0 +1,27 @@
|
|||||||
|
{
|
||||||
|
"name": "oci-proxy",
|
||||||
|
"online": true,
|
||||||
|
"storage": {
|
||||||
|
"blobStoreName": "default",
|
||||||
|
"strictContentTypeValidation": true
|
||||||
|
},
|
||||||
|
"oci": {
|
||||||
|
"v1Enabled": false,
|
||||||
|
"forceBasicAuth": false,
|
||||||
|
"pathEnabled": true
|
||||||
|
},
|
||||||
|
"ociProxy": {
|
||||||
|
"indexType": "HUB",
|
||||||
|
"cacheForeignLayers": false,
|
||||||
|
"foreignLayerUrlWhitelist": []
|
||||||
|
},
|
||||||
|
"proxy": {
|
||||||
|
"remoteUrl": "https://registry-1.docker.io",
|
||||||
|
"contentMaxAge": 1440,
|
||||||
|
"metadataMaxAge": 60,
|
||||||
|
"preserveEncodedCharacters": false
|
||||||
|
},
|
||||||
|
"negativeCache": { "enabled": true, "timeToLive": 60 },
|
||||||
|
"httpClient": { "blocked": false, "autoBlock": true },
|
||||||
|
"cosign": { "enforcement": "NONE" }
|
||||||
|
}
|
||||||
@@ -0,0 +1,15 @@
|
|||||||
|
{
|
||||||
|
"name": "oci-public",
|
||||||
|
"online": true,
|
||||||
|
"storage": {
|
||||||
|
"blobStoreName": "default",
|
||||||
|
"strictContentTypeValidation": true
|
||||||
|
},
|
||||||
|
"group": { "memberNames": ["oci-proxy"] },
|
||||||
|
"oci": {
|
||||||
|
"v1Enabled": false,
|
||||||
|
"forceBasicAuth": false,
|
||||||
|
"pathEnabled": true
|
||||||
|
},
|
||||||
|
"cosign": { "enforcement": "NONE" }
|
||||||
|
}
|
||||||
Executable
+80
@@ -0,0 +1,80 @@
|
|||||||
|
#!/usr/bin/env bash
|
||||||
|
set -euo pipefail
|
||||||
|
|
||||||
|
mode="${1:---check}"
|
||||||
|
case "$mode" in
|
||||||
|
--check | --apply) ;;
|
||||||
|
*) echo "usage: $0 [--check|--apply]" >&2; exit 2 ;;
|
||||||
|
esac
|
||||||
|
|
||||||
|
: "${TF_VAR_nexus_url:?set TF_VAR_nexus_url}"
|
||||||
|
: "${TF_VAR_nexus_username:?set TF_VAR_nexus_username}"
|
||||||
|
: "${TF_VAR_nexus_password:?set TF_VAR_nexus_password}"
|
||||||
|
|
||||||
|
script_dir="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")" && pwd)"
|
||||||
|
auth_file="$(mktemp /tmp/nexus-oci-auth.XXXXXX)"
|
||||||
|
trap 'rm -f -- "$auth_file"' EXIT
|
||||||
|
chmod 0600 "$auth_file"
|
||||||
|
printf 'machine %s\nlogin %s\npassword %s\n' \
|
||||||
|
"${TF_VAR_nexus_url#*://}" "$TF_VAR_nexus_username" \
|
||||||
|
"$TF_VAR_nexus_password" >"$auth_file"
|
||||||
|
|
||||||
|
drift=0
|
||||||
|
for entry in \
|
||||||
|
"hosted:$script_dir/oci/oci-hosted.json" \
|
||||||
|
"proxy:$script_dir/oci/oci-proxy.json" \
|
||||||
|
"group:$script_dir/oci/oci-public.json"; do
|
||||||
|
repository_type="${entry%%:*}"
|
||||||
|
desired_file="${entry#*:}"
|
||||||
|
repository_name="$(jq -er '.name' "$desired_file")"
|
||||||
|
endpoint="$TF_VAR_nexus_url/service/rest/v1/repositories/oci/$repository_type"
|
||||||
|
current_file="$(mktemp /tmp/nexus-oci-current.XXXXXX)"
|
||||||
|
|
||||||
|
status="$(curl --silent --show-error --netrc-file "$auth_file" \
|
||||||
|
--output "$current_file" --write-out '%{http_code}' \
|
||||||
|
"$endpoint/$repository_name")"
|
||||||
|
|
||||||
|
if [[ "$status" == 404 ]]; then
|
||||||
|
drift=1
|
||||||
|
if [[ "$mode" == --apply ]]; then
|
||||||
|
curl --fail --silent --show-error --netrc-file "$auth_file" \
|
||||||
|
--header 'Content-Type: application/json' \
|
||||||
|
--data-binary "@$desired_file" "$endpoint"
|
||||||
|
echo "created $repository_name"
|
||||||
|
else
|
||||||
|
echo "missing $repository_name" >&2
|
||||||
|
fi
|
||||||
|
elif [[ "$status" == 200 ]]; then
|
||||||
|
if jq -e --slurpfile desired "$desired_file" '
|
||||||
|
def subset($actual; $wanted):
|
||||||
|
if ($wanted | type) == "object" then
|
||||||
|
all($wanted | keys[];
|
||||||
|
($actual[.] != null) and subset($actual[.]; $wanted[.]))
|
||||||
|
else
|
||||||
|
$actual == $wanted
|
||||||
|
end;
|
||||||
|
subset(.; $desired[0])
|
||||||
|
' "$current_file" >/dev/null; then
|
||||||
|
echo "in sync $repository_name"
|
||||||
|
else
|
||||||
|
drift=1
|
||||||
|
if [[ "$mode" == --apply ]]; then
|
||||||
|
curl --fail --silent --show-error --netrc-file "$auth_file" \
|
||||||
|
--request PUT --header 'Content-Type: application/json' \
|
||||||
|
--data-binary "@$desired_file" "$endpoint/$repository_name"
|
||||||
|
echo "updated $repository_name"
|
||||||
|
else
|
||||||
|
echo "drifted $repository_name" >&2
|
||||||
|
fi
|
||||||
|
fi
|
||||||
|
else
|
||||||
|
cat "$current_file" >&2
|
||||||
|
echo "unexpected HTTP $status for $repository_name" >&2
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
rm -f -- "$current_file"
|
||||||
|
done
|
||||||
|
|
||||||
|
if [[ "$mode" == --check && "$drift" -ne 0 ]]; then
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
@@ -0,0 +1,64 @@
|
|||||||
|
locals {
|
||||||
|
proxy_http_client = {
|
||||||
|
auto_block = true
|
||||||
|
blocked = false
|
||||||
|
}
|
||||||
|
|
||||||
|
proxy_negative_cache = {
|
||||||
|
enabled = true
|
||||||
|
time_to_live = 60
|
||||||
|
}
|
||||||
|
|
||||||
|
repository_storage = {
|
||||||
|
blob_store_name = "default"
|
||||||
|
strict_content_type_validation = true
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
resource "sonatyperepo_repository_ansiblegalaxy_proxy" "galaxy" {
|
||||||
|
name = "ansible-galaxy-proxy"
|
||||||
|
online = true
|
||||||
|
|
||||||
|
http_client = local.proxy_http_client
|
||||||
|
negative_cache = local.proxy_negative_cache
|
||||||
|
proxy = {
|
||||||
|
remote_url = "https://galaxy.ansible.com"
|
||||||
|
content_max_age = 1440
|
||||||
|
metadata_max_age = 60
|
||||||
|
}
|
||||||
|
storage = local.repository_storage
|
||||||
|
}
|
||||||
|
|
||||||
|
resource "sonatyperepo_repository_ansiblegalaxy_group" "public" {
|
||||||
|
name = "ansible-public"
|
||||||
|
online = true
|
||||||
|
|
||||||
|
group = {
|
||||||
|
member_names = [sonatyperepo_repository_ansiblegalaxy_proxy.galaxy.name]
|
||||||
|
}
|
||||||
|
storage = local.repository_storage
|
||||||
|
}
|
||||||
|
|
||||||
|
resource "sonatyperepo_repository_go_proxy" "public" {
|
||||||
|
name = "go-proxy"
|
||||||
|
online = true
|
||||||
|
|
||||||
|
http_client = local.proxy_http_client
|
||||||
|
negative_cache = local.proxy_negative_cache
|
||||||
|
proxy = {
|
||||||
|
remote_url = "https://proxy.golang.org"
|
||||||
|
content_max_age = 1440
|
||||||
|
metadata_max_age = 60
|
||||||
|
}
|
||||||
|
storage = local.repository_storage
|
||||||
|
}
|
||||||
|
|
||||||
|
resource "sonatyperepo_repository_go_group" "public" {
|
||||||
|
name = "go-public"
|
||||||
|
online = true
|
||||||
|
|
||||||
|
group = {
|
||||||
|
member_names = [sonatyperepo_repository_go_proxy.public.name]
|
||||||
|
}
|
||||||
|
storage = local.repository_storage
|
||||||
|
}
|
||||||
@@ -0,0 +1,75 @@
|
|||||||
|
resource "sonatyperepo_privilege_repository_view" "anonymous_ansible" {
|
||||||
|
name = "ci-anonymous-ansible-read"
|
||||||
|
description = "Anonymous read access to the Ansible Galaxy group"
|
||||||
|
actions = ["BROWSE", "READ"]
|
||||||
|
format = "ansiblegalaxy"
|
||||||
|
repository = sonatyperepo_repository_ansiblegalaxy_group.public.name
|
||||||
|
}
|
||||||
|
|
||||||
|
resource "sonatyperepo_privilege_repository_view" "anonymous_ansible_proxy" {
|
||||||
|
name = "ci-anonymous-ansible-proxy-read"
|
||||||
|
description = "Anonymous artifact read access to the Ansible Galaxy proxy"
|
||||||
|
actions = ["BROWSE", "READ"]
|
||||||
|
format = "ansiblegalaxy"
|
||||||
|
repository = sonatyperepo_repository_ansiblegalaxy_proxy.galaxy.name
|
||||||
|
}
|
||||||
|
|
||||||
|
resource "sonatyperepo_privilege_repository_view" "anonymous_go" {
|
||||||
|
name = "ci-anonymous-go-read"
|
||||||
|
description = "Anonymous read access to the Go module group"
|
||||||
|
actions = ["BROWSE", "READ"]
|
||||||
|
format = "go"
|
||||||
|
repository = sonatyperepo_repository_go_group.public.name
|
||||||
|
}
|
||||||
|
|
||||||
|
resource "sonatyperepo_privilege_repository_view" "anonymous_oci_public" {
|
||||||
|
name = "ci-anonymous-oci-public-read"
|
||||||
|
description = "Anonymous read access to the public OCI group"
|
||||||
|
actions = ["BROWSE", "READ"]
|
||||||
|
format = "oci"
|
||||||
|
repository = "oci-public"
|
||||||
|
}
|
||||||
|
|
||||||
|
resource "sonatyperepo_privilege_repository_view" "anonymous_oci_proxy" {
|
||||||
|
name = "ci-anonymous-oci-proxy-read"
|
||||||
|
description = "Anonymous read access to the OCI proxy member"
|
||||||
|
actions = ["BROWSE", "READ"]
|
||||||
|
format = "oci"
|
||||||
|
repository = "oci-proxy"
|
||||||
|
}
|
||||||
|
|
||||||
|
resource "sonatyperepo_role" "anonymous_ci" {
|
||||||
|
id = "ci-anonymous-read"
|
||||||
|
name = "CI anonymous read"
|
||||||
|
description = "Read-only access to public CI dependency proxy groups"
|
||||||
|
privileges = [
|
||||||
|
sonatyperepo_privilege_repository_view.anonymous_ansible.name,
|
||||||
|
sonatyperepo_privilege_repository_view.anonymous_ansible_proxy.name,
|
||||||
|
sonatyperepo_privilege_repository_view.anonymous_go.name,
|
||||||
|
sonatyperepo_privilege_repository_view.anonymous_oci_public.name,
|
||||||
|
sonatyperepo_privilege_repository_view.anonymous_oci_proxy.name,
|
||||||
|
]
|
||||||
|
roles = []
|
||||||
|
}
|
||||||
|
|
||||||
|
resource "sonatyperepo_user" "anonymous" {
|
||||||
|
user_id = "anonymous"
|
||||||
|
first_name = "Anonymous"
|
||||||
|
last_name = "User"
|
||||||
|
email_address = "[email protected]"
|
||||||
|
status = "active"
|
||||||
|
roles = [sonatyperepo_role.anonymous_ci.id]
|
||||||
|
}
|
||||||
|
|
||||||
|
resource "sonatyperepo_system_anonymous_access" "ci" {
|
||||||
|
enabled = true
|
||||||
|
user_id = sonatyperepo_user.anonymous.user_id
|
||||||
|
realm_name = "NexusAuthorizingRealm"
|
||||||
|
}
|
||||||
|
|
||||||
|
resource "sonatyperepo_security_realms" "active" {
|
||||||
|
active = [
|
||||||
|
"NexusAuthenticatingRealm",
|
||||||
|
"OciBearerToken",
|
||||||
|
]
|
||||||
|
}
|
||||||
@@ -0,0 +1,33 @@
|
|||||||
|
terraform {
|
||||||
|
required_version = ">= 1.11.0"
|
||||||
|
|
||||||
|
required_providers {
|
||||||
|
sonatyperepo = {
|
||||||
|
source = "sonatype-nexus-community/sonatyperepo"
|
||||||
|
version = "1.17.0"
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
provider "sonatyperepo" {
|
||||||
|
url = var.nexus_url
|
||||||
|
username = var.nexus_username
|
||||||
|
password = var.nexus_password
|
||||||
|
}
|
||||||
|
|
||||||
|
variable "nexus_url" {
|
||||||
|
description = "Nexus Repository base URL"
|
||||||
|
type = string
|
||||||
|
}
|
||||||
|
|
||||||
|
variable "nexus_username" {
|
||||||
|
description = "Nexus Terraform management username"
|
||||||
|
type = string
|
||||||
|
sensitive = true
|
||||||
|
}
|
||||||
|
|
||||||
|
variable "nexus_password" {
|
||||||
|
description = "Nexus Terraform management password"
|
||||||
|
type = string
|
||||||
|
sensitive = true
|
||||||
|
}
|
||||||
@@ -53,4 +53,6 @@ sudo k3s kubectl -n flux-system get gitrepositories,kustomizations
|
|||||||
- SPIRE 已按 hardened chart 内部 fork `0.30.2-ddupan.1`(基于上游 `0.30.2`,SPIRE
|
- SPIRE 已按 hardened chart 内部 fork `0.30.2-ddupan.1`(基于上游 `0.30.2`,SPIRE
|
||||||
`1.15.3`)声明,使用共享
|
`1.15.3`)声明,使用共享
|
||||||
PostgreSQL 与独立 signing-key PVC;首次上线和 OpenBao JWT-SVID PoC 尚待合并后验证;
|
PostgreSQL 与独立 signing-key PVC;首次上线和 OpenBao JWT-SVID PoC 尚待合并后验证;
|
||||||
|
- Nexus Repository CE POC 已加入 GitOps 声明,计划验证 Ansible Galaxy、Go Modules 与
|
||||||
|
OCI/BuildKit 缓存;尚未部署或完成现场验收,现有 zot 保持不变;
|
||||||
- root Kustomization 与所有 brownfield 子 Kustomization 继续保持 `prune: false`。
|
- root Kustomization 与所有 brownfield 子 Kustomization 继续保持 `prune: false`。
|
||||||
|
|||||||
@@ -6,11 +6,12 @@ metadata:
|
|||||||
spec:
|
spec:
|
||||||
dependsOn:
|
dependsOn:
|
||||||
- name: external-secrets
|
- name: external-secrets
|
||||||
- name: nats
|
|
||||||
- name: spire
|
- name: spire
|
||||||
interval: 10m
|
interval: 10m
|
||||||
path: ./platform/dynamic-runner
|
path: ./platform/dynamic-runner
|
||||||
prune: false
|
# The runner backends are replaceable. Prune is required when a retired
|
||||||
|
# worker is removed from the component; otherwise it keeps consuming work.
|
||||||
|
prune: true
|
||||||
sourceRef:
|
sourceRef:
|
||||||
kind: GitRepository
|
kind: GitRepository
|
||||||
name: flux-system
|
name: flux-system
|
||||||
|
|||||||
@@ -0,0 +1,22 @@
|
|||||||
|
apiVersion: kustomize.toolkit.fluxcd.io/v1
|
||||||
|
kind: Kustomization
|
||||||
|
metadata:
|
||||||
|
name: nexus
|
||||||
|
namespace: flux-system
|
||||||
|
spec:
|
||||||
|
dependsOn:
|
||||||
|
- name: envoy-gateway
|
||||||
|
- name: openebs
|
||||||
|
healthChecks:
|
||||||
|
- apiVersion: apps/v1
|
||||||
|
kind: Deployment
|
||||||
|
name: nexus
|
||||||
|
namespace: nexus
|
||||||
|
interval: 10m
|
||||||
|
path: ./apps/nexus
|
||||||
|
prune: false
|
||||||
|
sourceRef:
|
||||||
|
kind: GitRepository
|
||||||
|
name: flux-system
|
||||||
|
timeout: 15m
|
||||||
|
wait: true
|
||||||
@@ -7,7 +7,6 @@ resources:
|
|||||||
- apps/envoy-gateway.yaml
|
- apps/envoy-gateway.yaml
|
||||||
- apps/external-secrets.yaml
|
- apps/external-secrets.yaml
|
||||||
- apps/gitea.yaml
|
- apps/gitea.yaml
|
||||||
- apps/gitea-actions.yaml
|
|
||||||
- apps/http-echo.yaml
|
- apps/http-echo.yaml
|
||||||
- apps/openebs.yaml
|
- apps/openebs.yaml
|
||||||
- apps/nats.yaml
|
- apps/nats.yaml
|
||||||
@@ -15,3 +14,4 @@ resources:
|
|||||||
- apps/spire.yaml
|
- apps/spire.yaml
|
||||||
- apps/observability.yaml
|
- apps/observability.yaml
|
||||||
- apps/zot.yaml
|
- apps/zot.yaml
|
||||||
|
- apps/nexus.yaml
|
||||||
|
|||||||
@@ -15,7 +15,8 @@ Root bootstrap 已完成。后续按依赖顺序分别引入:
|
|||||||
1. 监控 CRD、kube-state-metrics 以及 kubelet/cAdvisor 抓取配置;
|
1. 监控 CRD、kube-state-metrics 以及 kubelet/cAdvisor 抓取配置;
|
||||||
2. SPIRE Agent、SPIFFE CSI Driver 与 workload registration;
|
2. SPIRE Agent、SPIFFE CSI Driver 与 workload registration;
|
||||||
3. Kata Containers、`block-plain` RuntimeClass;
|
3. Kata Containers、`block-plain` RuntimeClass;
|
||||||
4. OpenSandbox operator/server 及 `ci-pod`、`ci-vm` Pools。
|
4. 独立 External Secrets Operator 与 sandbox 专用 OpenBao auth backend;
|
||||||
|
5. OpenSandbox controller/server;CI Pool 与 runner 调度器随后独立接入。
|
||||||
|
|
||||||
每一阶段单独合并并等待对应 Flux Kustomization Ready,不在 bootstrap 时一次性部署。
|
每一阶段单独合并并等待对应 Flux Kustomization Ready,不在 bootstrap 时一次性部署。
|
||||||
第一阶段监控拆为 `monitoring-operator` 与依赖它的 `monitoring`,防止 VM CR 在
|
第一阶段监控拆为 `monitoring-operator` 与依赖它的 `monitoring`,防止 VM CR 在
|
||||||
@@ -42,15 +43,25 @@ attestation。Server 使用 external bundle publisher 持续维护 sandbox
|
|||||||
显式关闭 Server 与 OIDC Provider,只部署 Agent DaemonSet 和 SPIFFE CSI Driver;因此
|
显式关闭 Server 与 OIDC Provider,只部署 Agent DaemonSet 和 SPIFFE CSI Driver;因此
|
||||||
不会产生第二个 trust root。
|
不会产生第二个 trust root。
|
||||||
|
|
||||||
`spire-smoke` namespace、ServiceAccount 和 `sandbox-spire-smoke` ClusterSPIFFEID 是
|
`spire-smoke` namespace、ServiceAccount 和 `sandbox-spire-smoke` ClusterSPIFFEID 只用于
|
||||||
普通 Pod 与后续 Kata guest 的回归夹具,稳定身份为
|
普通 Pod 的 CSI 回归夹具,稳定身份为 `spiffe://ddupan.top/sandbox/smoke`。Kata guest
|
||||||
`spiffe://ddupan.top/sandbox/smoke`。测试 Pod 临时创建并在验收后删除,身份声明保留。
|
不能复用 node Agent 暴露的 Unix socket;virtio-fs 只能呈现 socket 路径,不能把连接
|
||||||
|
跨过 VM 边界。Kata workload 必须使用 guest 内 Agent,具体约束见
|
||||||
|
`platform/sandbox-kata/README.md`。测试 Pod 临时创建并在验收后删除,普通 Pod 的身份
|
||||||
|
声明保留。
|
||||||
|
|
||||||
Kata 阶段使用官方 4.1.0 `kata-deploy` chart 的短生命周期 `job` 模式,逐节点安装并
|
Kata 阶段使用官方 4.1.0 `kata-deploy` chart 的短生命周期 `job` 模式,逐节点安装并
|
||||||
重启 K3s。只启用 `kata-clh-runtime-rs`,不创建默认 `kata` 别名;该 handler 的
|
重启 K3s。只启用 `kata-clh-runtime-rs`,不创建默认 `kata` 别名;该 handler 的
|
||||||
`emptyDir` 固定使用 `block-plain`,为 Docker/BuildKit overlay2 与 kind 提供 guest
|
`emptyDir` 固定使用 `block-plain`,为 Docker/BuildKit overlay2 与 kind 提供 guest
|
||||||
内块设备文件系统。详细限制与上线验收见 `platform/sandbox-kata/README.md`。
|
内块设备文件系统。详细限制与上线验收见 `platform/sandbox-kata/README.md`。
|
||||||
|
|
||||||
|
Sandbox 的 ESO 通过独立 `auth/kubernetes-sandbox` 向 OpenBao 证明 ServiceAccount
|
||||||
|
身份,只能读取共享的 `kv/k8s/opensandbox-api`。它不保存 reviewer JWT 或长期 Bao token;相关
|
||||||
|
Terraform 与 Flux 边界见 `platform/sandbox-external-secrets/README.md`。
|
||||||
|
|
||||||
|
OpenSandbox 阶段固定官方源码 commit 与 umbrella chart `0.2.2`,只部署 controller、
|
||||||
|
ClusterIP server 和 CRD。API key 由 ESO 从 OpenBao 投影,明文不进入 Git。
|
||||||
|
|
||||||
## 监控边界
|
## 监控边界
|
||||||
|
|
||||||
这里只管理 sandbox LXC 内的 Kubernetes 监控,不负责 PVE 宿主监控。LXC 与宿主共享
|
这里只管理 sandbox LXC 内的 Kubernetes 监控,不负责 PVE 宿主监控。LXC 与宿主共享
|
||||||
|
|||||||
+3
-2
@@ -2,13 +2,14 @@
|
|||||||
apiVersion: kustomize.toolkit.fluxcd.io/v1
|
apiVersion: kustomize.toolkit.fluxcd.io/v1
|
||||||
kind: Kustomization
|
kind: Kustomization
|
||||||
metadata:
|
metadata:
|
||||||
name: opensandbox-pools
|
name: ci-runners
|
||||||
namespace: flux-system
|
namespace: flux-system
|
||||||
spec:
|
spec:
|
||||||
dependsOn:
|
dependsOn:
|
||||||
- name: opensandbox
|
- name: opensandbox
|
||||||
|
- name: spire-agents
|
||||||
interval: 10m
|
interval: 10m
|
||||||
path: ./platform/sandbox-opensandbox-pools
|
path: ./platform/sandbox-ci-runners
|
||||||
prune: true
|
prune: true
|
||||||
sourceRef:
|
sourceRef:
|
||||||
kind: GitRepository
|
kind: GitRepository
|
||||||
@@ -0,0 +1,17 @@
|
|||||||
|
---
|
||||||
|
apiVersion: kustomize.toolkit.fluxcd.io/v1
|
||||||
|
kind: Kustomization
|
||||||
|
metadata:
|
||||||
|
name: external-secrets-operator
|
||||||
|
namespace: flux-system
|
||||||
|
spec:
|
||||||
|
dependsOn:
|
||||||
|
- name: monitoring-operator
|
||||||
|
interval: 10m
|
||||||
|
path: ./platform/sandbox-external-secrets/operator
|
||||||
|
prune: true
|
||||||
|
sourceRef:
|
||||||
|
kind: GitRepository
|
||||||
|
name: flux-system
|
||||||
|
timeout: 10m
|
||||||
|
wait: true
|
||||||
+8
-5
@@ -1,14 +1,17 @@
|
|||||||
|
---
|
||||||
apiVersion: kustomize.toolkit.fluxcd.io/v1
|
apiVersion: kustomize.toolkit.fluxcd.io/v1
|
||||||
kind: Kustomization
|
kind: Kustomization
|
||||||
metadata:
|
metadata:
|
||||||
name: gitea-actions
|
name: external-secrets
|
||||||
namespace: flux-system
|
namespace: flux-system
|
||||||
spec:
|
spec:
|
||||||
|
dependsOn:
|
||||||
|
- name: external-secrets-operator
|
||||||
interval: 10m
|
interval: 10m
|
||||||
path: ./platform/gitea-runner
|
path: ./platform/sandbox-external-secrets/config
|
||||||
prune: false
|
prune: true
|
||||||
sourceRef:
|
sourceRef:
|
||||||
kind: GitRepository
|
kind: GitRepository
|
||||||
name: flux-system
|
name: flux-system
|
||||||
timeout: 3m
|
timeout: 10m
|
||||||
wait: false
|
wait: true
|
||||||
@@ -6,13 +6,14 @@ metadata:
|
|||||||
namespace: flux-system
|
namespace: flux-system
|
||||||
spec:
|
spec:
|
||||||
dependsOn:
|
dependsOn:
|
||||||
|
- name: external-secrets
|
||||||
- name: kata
|
- name: kata
|
||||||
- name: spire-agents
|
- name: monitoring-operator
|
||||||
interval: 10m
|
interval: 10m
|
||||||
path: ./platform/sandbox-opensandbox
|
path: ./platform/sandbox-opensandbox
|
||||||
prune: true
|
prune: true
|
||||||
sourceRef:
|
sourceRef:
|
||||||
kind: GitRepository
|
kind: GitRepository
|
||||||
name: flux-system
|
name: flux-system
|
||||||
timeout: 20m
|
timeout: 15m
|
||||||
wait: true
|
wait: true
|
||||||
|
|||||||
@@ -7,5 +7,7 @@ resources:
|
|||||||
- apps/spire-bootstrap.yaml
|
- apps/spire-bootstrap.yaml
|
||||||
- apps/spire-agents.yaml
|
- apps/spire-agents.yaml
|
||||||
- apps/kata.yaml
|
- apps/kata.yaml
|
||||||
|
- apps/external-secrets-operator.yaml
|
||||||
|
- apps/external-secrets.yaml
|
||||||
- apps/opensandbox.yaml
|
- apps/opensandbox.yaml
|
||||||
- apps/opensandbox-pools.yaml
|
- apps/ci-runners.yaml
|
||||||
|
|||||||
@@ -18,6 +18,7 @@ homelab_dns:
|
|||||||
- { zone: ad.ddupan.top, name: metrics-write, type: A, values: [192.168.10.127] }
|
- { zone: ad.ddupan.top, name: metrics-write, type: A, values: [192.168.10.127] }
|
||||||
- { zone: ad.ddupan.top, name: netbox, type: A, values: [192.168.10.127] }
|
- { zone: ad.ddupan.top, name: netbox, type: A, values: [192.168.10.127] }
|
||||||
- { zone: ad.ddupan.top, name: nats, type: A, values: [192.168.10.127] }
|
- { zone: ad.ddupan.top, name: nats, type: A, values: [192.168.10.127] }
|
||||||
|
- { zone: ad.ddupan.top, name: nexus, type: A, values: [192.168.10.127] }
|
||||||
- { zone: ad.ddupan.top, name: s3, type: A, values: [192.168.10.127] }
|
- { zone: ad.ddupan.top, name: s3, type: A, values: [192.168.10.127] }
|
||||||
- { zone: ad.ddupan.top, name: spire-oidc, type: A, values: [192.168.10.127] }
|
- { zone: ad.ddupan.top, name: spire-oidc, type: A, values: [192.168.10.127] }
|
||||||
- { zone: ad.ddupan.top, name: spire-server, type: A, values: [192.168.10.127] }
|
- { zone: ad.ddupan.top, name: spire-server, type: A, values: [192.168.10.127] }
|
||||||
|
|||||||
@@ -51,7 +51,7 @@ few things Terraform must not own.**
|
|||||||
| secrets engine mounts (`kv`, `pki`, `ssh-client-signer`) | the daemon, Raft, TLS files, systemd |
|
| secrets engine mounts (`kv`, `pki`, `ssh-client-signer`) | the daemon, Raft, TLS files, systemd |
|
||||||
| PKI role, issuing/CRL URLs, cluster paths, **ACME** | `bao operator init` / unseal (manual, PGP-wrapped) |
|
| PKI role, issuing/CRL URLs, cluster paths, **ACME** | `bao operator init` / unseal (manual, PGP-wrapped) |
|
||||||
| SSH signing role (`ai-agent`) | **PKI root CA + SSH CA signing key** |
|
| SSH signing role (`ai-agent`) | **PKI root CA + SSH CA signing key** |
|
||||||
| OIDC auth *mount* and *role* | OIDC **client secret** (`auth/oidc/config`) |
|
| OIDC/Kubernetes auth mount、config 与 role | OIDC **client secret** (`auth/oidc/config`) |
|
||||||
| all policies | snapshot token + script + systemd timer |
|
| all policies | snapshot token + script + systemd timer |
|
||||||
| | host-level CA trust distribution (`openbao_ssh_ca_trust`) |
|
| | host-level CA trust distribution (`openbao_ssh_ca_trust`) |
|
||||||
|
|
||||||
@@ -92,6 +92,13 @@ then `VAULT_ADDR`/`VAULT_TOKEN`), mirroring how `smtp-relay/terraform` uses
|
|||||||
native `openbao/openbao` provider is published only to the OpenTofu registry and
|
native `openbao/openbao` provider is published only to the OpenTofu registry and
|
||||||
cannot be resolved by the HashiCorp `terraform` CLI.
|
cannot be resolved by the HashiCorp `terraform` CLI.
|
||||||
|
|
||||||
|
Sandbox 集群使用独立的 `auth/kubernetes-sandbox`。其 API 地址、公开 Kubernetes CA、
|
||||||
|
ESO role 与只读 `kv/k8s/opensandbox-api` policy 全部由 Terraform 管理;CA 位于
|
||||||
|
`terraform/certs/sandbox-kubernetes-ca.crt`。集群重建并轮换 CA 后,先更新该文件并
|
||||||
|
apply,再让 Flux 恢复 ESO reconciliation。该 backend 不保存 reviewer JWT,而是使用
|
||||||
|
ESO 的短期登录 JWT 执行 TokenReview。该组资源已于 2026-09-18 apply,随后复验 plan
|
||||||
|
为 zero-diff。
|
||||||
|
|
||||||
## DNS
|
## DNS
|
||||||
|
|
||||||
`bao.ad.ddupan.top` is an **internal-only** name — not a public Cloudflare record and
|
`bao.ad.ddupan.top` is an **internal-only** name — not a public Cloudflare record and
|
||||||
|
|||||||
@@ -59,3 +59,33 @@ resource "vault_kubernetes_auth_backend_role" "external_secrets" {
|
|||||||
# in a long TTL and every extra hour is a longer-lived credential in memory.
|
# in a long TTL and every extra hour is a longer-lived credential in memory.
|
||||||
token_ttl = 3600
|
token_ttl = 3600
|
||||||
}
|
}
|
||||||
|
|
||||||
|
# A Kubernetes auth mount can validate identities from only the API server it is
|
||||||
|
# configured against. The sandbox cluster therefore cannot reuse auth/kubernetes,
|
||||||
|
# whose TokenReview endpoint belongs to homelab.
|
||||||
|
resource "vault_auth_backend" "sandbox_kubernetes" {
|
||||||
|
type = "kubernetes"
|
||||||
|
path = "kubernetes-sandbox"
|
||||||
|
}
|
||||||
|
|
||||||
|
resource "vault_kubernetes_auth_backend_config" "sandbox" {
|
||||||
|
backend = vault_auth_backend.sandbox_kubernetes.path
|
||||||
|
kubernetes_host = "https://10.60.0.13:6443"
|
||||||
|
kubernetes_ca_cert = file("${path.module}/certs/sandbox-kubernetes-ca.crt")
|
||||||
|
disable_local_ca_jwt = true
|
||||||
|
|
||||||
|
# Deliberately omit token_reviewer_jwt. OpenBao uses the login JWT for
|
||||||
|
# TokenReview; the sandbox external-secrets ServiceAccount is bound only to
|
||||||
|
# system:auth-delegator and all issued JWTs remain short-lived.
|
||||||
|
}
|
||||||
|
|
||||||
|
resource "vault_kubernetes_auth_backend_role" "sandbox_external_secrets" {
|
||||||
|
backend = vault_auth_backend.sandbox_kubernetes.path
|
||||||
|
role_name = "external-secrets"
|
||||||
|
|
||||||
|
bound_service_account_names = ["external-secrets"]
|
||||||
|
bound_service_account_namespaces = ["external-secrets"]
|
||||||
|
|
||||||
|
token_policies = [vault_policy.sandbox_external_secrets.name]
|
||||||
|
token_ttl = 3600
|
||||||
|
}
|
||||||
|
|||||||
@@ -0,0 +1,10 @@
|
|||||||
|
-----BEGIN CERTIFICATE-----
|
||||||
|
MIIBdjCCAR2gAwIBAgIBADAKBggqhkjOPQQDAjAjMSEwHwYDVQQDDBhrM3Mtc2Vy
|
||||||
|
dmVyLWNhQDE3ODk2NTI4NTcwHhcNMjYwOTE3MTI0NzM3WhcNMzYwOTE0MTI0NzM3
|
||||||
|
WjAjMSEwHwYDVQQDDBhrM3Mtc2VydmVyLWNhQDE3ODk2NTI4NTcwWTATBgcqhkjO
|
||||||
|
PQIBBggqhkjOPQMBBwNCAAR4SbqzTXZnlZdUPz7viN6+dYbB1Maw44Qiepn9r5XG
|
||||||
|
sOzkYkN8t1aG3Ugo8TqQ3xJaKkM89n1Rluj0vbOhiNajo0IwQDAOBgNVHQ8BAf8E
|
||||||
|
BAMCAqQwDwYDVR0TAQH/BAUwAwEB/zAdBgNVHQ4EFgQUyLSGoKAKAJuiniuRdBLG
|
||||||
|
XYaDQC8wCgYIKoZIzj0EAwIDRwAwRAIgFkVzyUZexk/ynnxBEOg+3foJv3WKqAei
|
||||||
|
hTSRjO1gL0UCIFbBKR7BMrJJAgW3DJFeeBM+b+tTg93jNx55qZACbFOL
|
||||||
|
-----END CERTIFICATE-----
|
||||||
@@ -35,3 +35,10 @@ resource "vault_policy" "external_secrets" {
|
|||||||
name = "external-secrets"
|
name = "external-secrets"
|
||||||
policy = file("${path.module}/policies/external-secrets.hcl")
|
policy = file("${path.module}/policies/external-secrets.hcl")
|
||||||
}
|
}
|
||||||
|
|
||||||
|
# The sandbox cluster has its own Kubernetes auth backend and a deliberately
|
||||||
|
# narrower KV view than the homelab ESO instance.
|
||||||
|
resource "vault_policy" "sandbox_external_secrets" {
|
||||||
|
name = "sandbox-external-secrets"
|
||||||
|
policy = file("${path.module}/policies/sandbox-external-secrets.hcl")
|
||||||
|
}
|
||||||
|
|||||||
@@ -0,0 +1,9 @@
|
|||||||
|
# Read only the shared OpenSandbox control-plane API key. The same Bao object is
|
||||||
|
# consumed by the server in sandbox and, later, by the scheduler in homelab.
|
||||||
|
path "kv/data/k8s/opensandbox-api" {
|
||||||
|
capabilities = ["read"]
|
||||||
|
}
|
||||||
|
|
||||||
|
path "kv/metadata/k8s/opensandbox-api" {
|
||||||
|
capabilities = ["read"]
|
||||||
|
}
|
||||||
@@ -83,6 +83,11 @@ vyos_postgresql_primary_address: "10.60.0.11"
|
|||||||
vyos_sandbox_k3s_api_address: "10.60.0.13"
|
vyos_sandbox_k3s_api_address: "10.60.0.13"
|
||||||
vyos_sandbox_k3s_api_interface: eth1
|
vyos_sandbox_k3s_api_interface: eth1
|
||||||
vyos_sandbox_k3s_api_port: 6443
|
vyos_sandbox_k3s_api_port: 6443
|
||||||
|
vyos_opensandbox_api_port: 8080
|
||||||
|
vyos_opensandbox_node_port: 30080
|
||||||
|
# Lifecycle create is synchronous and a cold Kata image pull can exceed the
|
||||||
|
# HAProxy 50-second default. Keep this below OpenSandbox's overall timeout.
|
||||||
|
vyos_opensandbox_api_timeout: 600
|
||||||
vyos_sandbox_k3s_servers:
|
vyos_sandbox_k3s_servers:
|
||||||
- name: sandbox1
|
- name: sandbox1
|
||||||
address: "10.60.0.11"
|
address: "10.60.0.11"
|
||||||
|
|||||||
@@ -15,6 +15,20 @@ set interfaces ethernet {{ i.iface }} address {{ i.address }}
|
|||||||
set interfaces ethernet {{ i.iface }} description '{{ i.description }}'
|
set interfaces ethernet {{ i.iface }} description '{{ i.description }}'
|
||||||
{% endfor %}
|
{% endfor %}
|
||||||
|
|
||||||
|
{# OpenSandbox stays on a NodePort; VyOS owns the stable routed frontend. #}
|
||||||
|
set load-balancing haproxy service opensandbox-api listen-address {{ vyos_sandbox_k3s_api_address }}
|
||||||
|
set load-balancing haproxy service opensandbox-api port {{ vyos_opensandbox_api_port }}
|
||||||
|
set load-balancing haproxy service opensandbox-api mode 'tcp'
|
||||||
|
set load-balancing haproxy service opensandbox-api backend 'opensandbox-api'
|
||||||
|
set load-balancing haproxy service opensandbox-api timeout client {{ vyos_opensandbox_api_timeout }}
|
||||||
|
set load-balancing haproxy backend opensandbox-api mode 'tcp'
|
||||||
|
set load-balancing haproxy backend opensandbox-api timeout server {{ vyos_opensandbox_api_timeout }}
|
||||||
|
{% for server in vyos_sandbox_k3s_servers %}
|
||||||
|
set load-balancing haproxy backend opensandbox-api server {{ server.name }} address {{ server.address }}
|
||||||
|
set load-balancing haproxy backend opensandbox-api server {{ server.name }} port {{ vyos_opensandbox_node_port }}
|
||||||
|
set load-balancing haproxy backend opensandbox-api server {{ server.name }} check
|
||||||
|
{% endfor %}
|
||||||
|
|
||||||
{# --- default route out; OSPF carries the rest --- #}
|
{# --- default route out; OSPF carries the rest --- #}
|
||||||
set protocols static route 0.0.0.0/0 next-hop {{ vyos_lan_gateway }}
|
set protocols static route 0.0.0.0/0 next-hop {{ vyos_lan_gateway }}
|
||||||
|
|
||||||
|
|||||||
@@ -18,7 +18,8 @@ Flux 管理以下 Kubernetes 资源:
|
|||||||
- Kata Containers 和 CI 专用的 `block-plain` RuntimeClass;
|
- Kata Containers 和 CI 专用的 `block-plain` RuntimeClass;
|
||||||
- SPIRE Agent、SPIFFE CSI Driver 与 workload identity 声明;
|
- SPIRE Agent、SPIFFE CSI Driver 与 workload identity 声明;
|
||||||
- vmagent、kube-state-metrics、kubelet/cAdvisor scrape 配置和告警;
|
- vmagent、kube-state-metrics、kubelet/cAdvisor scrape 配置和告警;
|
||||||
- OpenSandbox operator/server、`ci-pod` 与 `ci-vm` Pools。
|
- sandbox External Secrets Operator、OpenSandbox controller/server;CI Pool 与 runner
|
||||||
|
调度器由 runner 项目接入。
|
||||||
|
|
||||||
同一个对象只能有一个 owner。Ansible 不直接部署上述集群内 workload;Flux 不管理
|
同一个对象只能有一个 owner。Ansible 不直接部署上述集群内 workload;Flux 不管理
|
||||||
LXC、K3s datastore 或 K3s 本身。
|
LXC、K3s datastore 或 K3s 本身。
|
||||||
@@ -92,6 +93,13 @@ ansible-playbook site.yml
|
|||||||
ansible-playbook k3s.yml
|
ansible-playbook k3s.yml
|
||||||
```
|
```
|
||||||
|
|
||||||
|
只 reconcile Flux controllers 与 root sync manifest(不触碰 LXC、PostgreSQL 或
|
||||||
|
K3s lifecycle):
|
||||||
|
|
||||||
|
```bash
|
||||||
|
ansible-playbook flux.yml
|
||||||
|
```
|
||||||
|
|
||||||
K3s 外部 datastore URI 由运行时 `SANDBOX_K3S_DB_PASSWORD` 生成,密码在 URI 中
|
K3s 外部 datastore URI 由运行时 `SANDBOX_K3S_DB_PASSWORD` 生成,密码在 URI 中
|
||||||
进行 URL 编码,最终仅持久化于节点 root 可读的 `/etc/rancher/k3s/config.yaml`
|
进行 URL 编码,最终仅持久化于节点 root 可读的 `/etc/rancher/k3s/config.yaml`
|
||||||
(mode `0600`)。首节点生成的 K3s join token 仅在同一次 Ansible run 内传给第二节点;
|
(mode `0600`)。首节点生成的 K3s join token 仅在同一次 Ansible run 内传给第二节点;
|
||||||
@@ -106,7 +114,9 @@ ansible-playbook verify.yml
|
|||||||
当前已经声明 LXC 生命周期、最小 OS baseline、PostgreSQL 和 K3s,包括系统级
|
当前已经声明 LXC 生命周期、最小 OS baseline、PostgreSQL 和 K3s,包括系统级
|
||||||
homelab CA trust。Flux `v2.9.5` controllers 与 root sync 也由 Ansible 通过 K3s
|
homelab CA trust。Flux `v2.9.5` controllers 与 root sync 也由 Ansible 通过 K3s
|
||||||
server manifests 管理;root 使用 homelab CA 访问公开 Gitea 仓库,不保存 Git token。
|
server manifests 管理;root 使用 homelab CA 访问公开 Gitea 仓库,不保存 Git token。
|
||||||
集群内 workload 由 `clusters/sandbox/` 分阶段纳入 Flux。
|
集群内 workload 由 `clusters/sandbox/` 分阶段纳入 Flux。root Kustomization 的健康检查
|
||||||
|
timeout 为 40 分钟,用于覆盖 Kata 等首次安装时会逐节点重启 K3s 的子
|
||||||
|
Kustomization;各子项仍保留自己的更短 timeout,故障会在对应子项先行暴露。
|
||||||
|
|
||||||
## SPIRE 跨集群 bootstrap
|
## SPIRE 跨集群 bootstrap
|
||||||
|
|
||||||
|
|||||||
@@ -0,0 +1,6 @@
|
|||||||
|
---
|
||||||
|
- name: Reconcile Flux controllers and root sync
|
||||||
|
hosts: sandbox1
|
||||||
|
gather_facts: false
|
||||||
|
roles:
|
||||||
|
- sandbox_flux
|
||||||
+1
-1
@@ -35,5 +35,5 @@ spec:
|
|||||||
sourceRef:
|
sourceRef:
|
||||||
kind: GitRepository
|
kind: GitRepository
|
||||||
name: flux-system
|
name: flux-system
|
||||||
timeout: 3m
|
timeout: 40m
|
||||||
wait: true
|
wait: true
|
||||||
|
|||||||
@@ -1,51 +1,107 @@
|
|||||||
# Gitea dynamic runner controller
|
# Gitea dynamic runner
|
||||||
|
|
||||||
此目录只管理 homelab 中的 controller 部署。controller、worker、Cloud Hypervisor
|
本目录部署单副本 Go controller,在同一进程运行 RunnerService scheduler、原生
|
||||||
launcher 和 guest runner 的源码与发布位于独立仓库
|
Kubernetes Pod worker、OpenSandbox VM worker 和 SPIFFE mTLS facade:
|
||||||
`panxiao81/gitea-dynamic-runner`。
|
|
||||||
|
|
||||||
当前 bootstrap controller 接收 Gitea `workflow_job` webhook,将 `[self-hosted, pod]` 和
|
|
||||||
`[self-hosted, vm]` 的 queued job 分别发布到 NATS。Pod worker 在本集群创建一次性
|
|
||||||
privileged host runner;Docker、BuildKit 和 kind 由 workflow 自行 setup。内部
|
|
||||||
endpoint:
|
|
||||||
|
|
||||||
```text
|
```text
|
||||||
http://dynamic-runner-controller.dynamic-runner.svc.cluster.local:8787/webhook
|
Gitea RunnerService -> scheduler -> JetStream ci.runner.pod
|
||||||
|
|
|
||||||
|
v
|
||||||
|
homelab Kubernetes Pod
|
||||||
|
|
|
||||||
|
SPIFFE mTLS RunnerService facade
|
||||||
|
|
|
||||||
|
v
|
||||||
|
Gitea
|
||||||
```
|
```
|
||||||
|
|
||||||
OpenBao 路径:
|
Pod backend 不经过 OpenSandbox。VM backend 后续启用时才访问 VyOS 暴露的
|
||||||
|
OpenSandbox Lifecycle API;本目录不修改 sandbox 平台侧 ESO、Bao Terraform 或
|
||||||
|
OpenSandbox chart 所有权边界。
|
||||||
|
|
||||||
- `kv/k8s/nats.ci_producer_password`:已有 NATS producer 密码。
|
## Canary 安全边界
|
||||||
- `kv/k8s/nats.ci_worker_password`:已有 NATS worker 密码。
|
|
||||||
- `kv/k8s/dynamic-runner.webhook_secret`:Gitea webhook HMAC secret。
|
|
||||||
- `kv/k8s/gitea-runner.token`:现有 instance runner registration token。
|
|
||||||
|
|
||||||
首期 controller 与 runner 镜像由 laptop 本机构建后导入 k3s containerd,作为 CI
|
- Deployment 为单副本,滚动策略固定 `maxSurge: 1`、`maxUnavailable: 0`,保证新
|
||||||
发布链路建立前的 bootstrap。部署使用 `imagePullPolicy: Never`。正式发布 workflow
|
facade Ready 后才终止旧实例。scheduler 必须通过 Kubernetes Lease 保持单 leader,
|
||||||
获得专用 SPIFFE ID 后,必须将 image 改为 zot digest 并移除本地导入步骤。
|
不能依赖 Recreate 避免重复领取。
|
||||||
|
- scheduler 使用一个 runner registration,并按总容量启动并发 `FetchTask` goroutine;
|
||||||
|
`POD_CAPACITY=4` 与 `VM_CAPACITY=1` 分别限制两个 durable consumer 和 backend
|
||||||
|
admission pool。池满时 assignment 保持 JetStream pending,任一 backend 不占用
|
||||||
|
另一方的执行槽位,也不会创建超出容量的 workload。
|
||||||
|
- rollout 重叠期间只有持有 `Lease/dynamic-runner-scheduler` 的 controller 执行
|
||||||
|
`FetchTask`;所有 Ready 实例都可通过 backend metadata 恢复 claim 并服务 facade。
|
||||||
|
- executor 镜像使用 digest;Pod 以 UID 2000 运行,SPIRE `ClusterStaticEntry` 同时绑定
|
||||||
|
具体 Pod UID 与 `unix:uid:2000`。
|
||||||
|
- facade 通过仅内网可路由的 `192.168.10.127:30443` NodePort 提供给 sandbox executor;
|
||||||
|
双方使用 Workload API X509-SVID mTLS,并按 SPIFFE ID 而不是 IP/DNS 名验证服务端。
|
||||||
|
该入口不经过公网或 Cloudflare Tunnel。
|
||||||
|
- controller 的 SPIFFE ID 固定为
|
||||||
|
`spiffe://ddupan.top/ns/dynamic-runner/sa/dynamic-runner-controller`。
|
||||||
|
- NATS 保留既有最小权限分离:`ci-producer` 仅 publish,`ci-worker` 仅 pull/ACK。
|
||||||
|
|
||||||
## 身份绑定
|
facade claim registry 在启动时从 Pod/OpenSandbox metadata 恢复;scheduler leadership
|
||||||
|
由 Kubernetes Lease 持久化协调。Deployment 仍保持 `replicas: 1`,滚动更新期间允许
|
||||||
|
一个额外 Pod 提供 facade 连续性。
|
||||||
|
|
||||||
queued webhook 只负责创建没有业务身份的 Pod。runner 实际领取任务后,Gitea 的
|
## Secret 边界
|
||||||
`in_progress` webhook 会携带实际 `runner_name`;controller 将 binding 消息发布到
|
|
||||||
NATS,Pod worker 再给对应 Pod 添加:
|
|
||||||
|
|
||||||
```text
|
`ExternalSecret/dynamic-runner` 从既有 `ClusterSecretStore/openbao` 读取:
|
||||||
ci.ddupan.top/identity-bound=true
|
|
||||||
ci.ddupan.top/spiffe-path=<owner>/<repository>/<percent-encoded-job-name>
|
- `kv/k8s/nats`:producer/worker 密码;
|
||||||
|
- `kv/k8s/opensandbox-api:api_key`:保留给后续 VM worker;
|
||||||
|
- `kv/k8s/gitea-runner:token`:保留的 runner registration token;
|
||||||
|
- `kv/k8s/dynamic-runner`:scheduler UUID/token、facade HMAC key 和回滚所需 webhook secret。
|
||||||
|
|
||||||
|
scheduler credential 由官方 Gitea Runner v3.5.0 一次注册生成;它只挂载到 controller,
|
||||||
|
不会进入 executor。facade capability key 至少 32 字节,controller 为每个 assignment
|
||||||
|
确定性生成独立 capability。不要打印 Secret、创建静态 Bao token或把 credential 写入
|
||||||
|
Git。OpenBao 写入使用本机 SPIFFE JWT 换取的短期 `local-development` token。
|
||||||
|
|
||||||
|
## Workflow 身份与依赖配置
|
||||||
|
|
||||||
|
workflow 可复用 [`panxiao81/ci-actions`](https://git.ddupan.top/panxiao81/ci-actions)
|
||||||
|
中的 `spiffe-openbao-login@v1` 和 `setup-nexus@v1`。这不改变 runner 的权限边界:
|
||||||
|
runner 仅提供 Node.js 20、`spire-agent` 与 Workload API socket,workflow 负责声明 Bao
|
||||||
|
role、audience 和具体用途,目标服务 policy 决定是否授权。短期 Bao token 会进入
|
||||||
|
Actions job 临时文件,因此这些 Action 只允许在本目录管理的一次性 Pod/VM executor
|
||||||
|
中使用,不能迁移到共享或持久 runner。
|
||||||
|
|
||||||
|
匿名读取 Nexus public repository 只需 `setup-nexus@v1`,不应为了依赖下载额外申请
|
||||||
|
Bao 凭据;需要发布制品时再为对应 repository 建立独立 service account 与最小权限
|
||||||
|
policy。
|
||||||
|
|
||||||
|
## 首次验收
|
||||||
|
|
||||||
|
合并后先观察 Flux 与 controller,不要立即开启 VM:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
flux reconcile kustomization dynamic-runner --with-source
|
||||||
|
kubectl -n dynamic-runner wait externalsecret/dynamic-runner \
|
||||||
|
--for=condition=Ready --timeout=2m
|
||||||
|
kubectl -n dynamic-runner rollout status deploy/dynamic-runner-controller --timeout=5m
|
||||||
|
kubectl -n dynamic-runner logs deploy/dynamic-runner-controller -f
|
||||||
```
|
```
|
||||||
|
|
||||||
`ClusterSPIFFEID/gitea-dynamic-runner` 只匹配已经绑定的 Pod,并签发
|
确认 scheduler 只领取一条 `[self-hosted,pod]` task,然后验证:
|
||||||
`spiffe://ddupan.top/ci/<owner>/<repository>/<job-name>`。runner 的 job-start hook 在
|
|
||||||
SVID 可用之前不会放行第一步,因此不能根据 queued 事件错配身份。
|
|
||||||
|
|
||||||
每个 runner Pod 使用 `gitea-dynamic-runner` ServiceAccount。该 ServiceAccount 没有
|
1. assignment message 进入并离开 durable `pod` consumer;
|
||||||
Kubernetes API 权限;只有 `dynamic-runner-pod-worker` ServiceAccount 能在本 namespace
|
2. `gitea-task-<task-id>` Pod 创建,取得实际 Pod UID;
|
||||||
create/get/patch/delete Pod。
|
3. 同名 `ClusterStaticEntry` 的 parent ID 包含该 UID,SPIFFE ID 使用 repository/job key;
|
||||||
|
4. 官方 runner v3.5.0 经 facade claim 精确 task,Gitea 实时收到日志和终态;
|
||||||
|
5. Pod consumer 达到 capacity 时 VM consumer 仍可独立接受任务。
|
||||||
|
|
||||||
长期实现将由兼容 Gitea RunnerService 的 scheduler 直接领取 task,再交给 Pod/VM
|
Pod 与 VM 都在 Gitea 接受终态后先把 terminal marker 写入各自 backend metadata,再由
|
||||||
executor;届时删除 webhook、临时 runner 注册和 identity binding 消息。跟踪见
|
lifecycle reconciler 删除执行器。首次 VM 测试仍须观察 BatchSandbox、Pod 与
|
||||||
`panxiao81/gitea-dynamic-runner` issue #7。
|
ClusterStaticEntry 全部消失;完整自动清理通过前不得提高 `POD_CAPACITY` 或
|
||||||
|
`VM_CAPACITY`。
|
||||||
|
|
||||||
Gitea webhook 只订阅 `workflow_job`,content type 使用 JSON,secret 与 Bao 中值
|
VM backend 已通过 `vm-dev` canary 完成 Docker、kind、SPIFFE 和完整生命周期测试。
|
||||||
一致。不要启用 `send_everything`,否则 controller 会收到无关仓库事件。
|
生产调度标签为 `[self-hosted, vm]`,初始保持 `VM_CAPACITY=1`;扩容前先观察实际任务的
|
||||||
|
资源水位、等待时间以及 OpenSandbox 是否存在 terminal sandbox 残留。
|
||||||
|
|
||||||
|
## 回滚
|
||||||
|
|
||||||
|
若 controller 在领取 task 前失败,回滚到前一 commit 的 Python controller/Pod worker
|
||||||
|
manifests。若已经创建 `gitea-task-*` Pod,先保留现场并核对 Gitea task 状态,不能直接
|
||||||
|
重启 scheduler 造成重复执行。scheduler registration 和 capability key 保留在 Bao,
|
||||||
|
回滚不需要删除或打印它们。
|
||||||
|
|||||||
@@ -1,17 +1,16 @@
|
|||||||
apiVersion: spire.spiffe.io/v1alpha1
|
apiVersion: spire.spiffe.io/v1alpha1
|
||||||
kind: ClusterSPIFFEID
|
kind: ClusterSPIFFEID
|
||||||
metadata:
|
metadata:
|
||||||
name: gitea-dynamic-runner
|
name: dynamic-runner-controller
|
||||||
spec:
|
spec:
|
||||||
className: spire-mgmt-spire
|
className: spire-mgmt-spire
|
||||||
spiffeIDTemplate: 'spiffe://{{ .TrustDomain }}/ci/{{ index .PodMeta.Annotations "ci.ddupan.top/spiffe-path" }}'
|
spiffeIDTemplate: spiffe://ddupan.top/ns/dynamic-runner/sa/dynamic-runner-controller
|
||||||
namespaceSelector:
|
namespaceSelector:
|
||||||
matchLabels:
|
matchLabels:
|
||||||
kubernetes.io/metadata.name: dynamic-runner
|
kubernetes.io/metadata.name: dynamic-runner
|
||||||
podSelector:
|
podSelector:
|
||||||
matchLabels:
|
matchLabels:
|
||||||
app.kubernetes.io/name: gitea-dynamic-runner
|
app.kubernetes.io/name: dynamic-runner-controller
|
||||||
ci.ddupan.top/identity-bound: "true"
|
|
||||||
workloadSelectorTemplates:
|
workloadSelectorTemplates:
|
||||||
- k8s:ns:dynamic-runner
|
- k8s:ns:dynamic-runner
|
||||||
- k8s:sa:gitea-dynamic-runner
|
- k8s:sa:dynamic-runner-controller
|
||||||
|
|||||||
@@ -5,6 +5,11 @@ metadata:
|
|||||||
namespace: dynamic-runner
|
namespace: dynamic-runner
|
||||||
spec:
|
spec:
|
||||||
replicas: 1
|
replicas: 1
|
||||||
|
strategy:
|
||||||
|
type: RollingUpdate
|
||||||
|
rollingUpdate:
|
||||||
|
maxSurge: 1
|
||||||
|
maxUnavailable: 0
|
||||||
selector:
|
selector:
|
||||||
matchLabels:
|
matchLabels:
|
||||||
app.kubernetes.io/name: dynamic-runner-controller
|
app.kubernetes.io/name: dynamic-runner-controller
|
||||||
@@ -14,17 +19,10 @@ spec:
|
|||||||
app.kubernetes.io/name: dynamic-runner-controller
|
app.kubernetes.io/name: dynamic-runner-controller
|
||||||
spec:
|
spec:
|
||||||
serviceAccountName: dynamic-runner-controller
|
serviceAccountName: dynamic-runner-controller
|
||||||
automountServiceAccountToken: false
|
|
||||||
initContainers:
|
initContainers:
|
||||||
- name: fetch-internal-ca
|
- name: fetch-internal-ca
|
||||||
image: curlimages/curl:8.16.0@sha256:463eaf6072688fe96ac64fa623fe73e1dbe25d8ad6c34404a669ad3ce1f104b6
|
image: curlimages/curl:8.16.0@sha256:463eaf6072688fe96ac64fa623fe73e1dbe25d8ad6c34404a669ad3ce1f104b6
|
||||||
args:
|
args: [--fail, --silent, --show-error, --output, /trust/ca.pem, https://bao.ad.ddupan.top:8200/v1/pki/ca/pem]
|
||||||
- --fail
|
|
||||||
- --silent
|
|
||||||
- --show-error
|
|
||||||
- --output
|
|
||||||
- /trust/ca.pem
|
|
||||||
- https://bao.ad.ddupan.top:8200/v1/pki/ca/pem
|
|
||||||
securityContext:
|
securityContext:
|
||||||
allowPrivilegeEscalation: false
|
allowPrivilegeEscalation: false
|
||||||
capabilities:
|
capabilities:
|
||||||
@@ -40,40 +38,89 @@ spec:
|
|||||||
mountPath: /trust
|
mountPath: /trust
|
||||||
containers:
|
containers:
|
||||||
- name: controller
|
- name: controller
|
||||||
# Bootstrap import on laptop. Replace with a zot digest after the
|
image: zot.ad.ddupan.top/panxiao81/gitea-dynamic-runner-controller@sha256:7374a08a238ca1c76a9aee33c57520ae0edaa9b5b443708487d5b3a69a61da4e
|
||||||
# repository's image publishing workflow has a dedicated identity.
|
imagePullPolicy: IfNotPresent
|
||||||
image: gitea-dynamic-runner-controller:0.3.0-bootstrap
|
args: [controller]
|
||||||
imagePullPolicy: Never
|
|
||||||
env:
|
env:
|
||||||
|
- name: COMPONENTS
|
||||||
|
value: scheduler,pod-worker,vm-worker
|
||||||
|
- name: GITEA_INSTANCE_URL
|
||||||
|
value: https://git.ddupan.top
|
||||||
|
- name: GITEA_RUNNER_UUID_FILE
|
||||||
|
value: /run/dynamic-runner-secrets/scheduler-uuid
|
||||||
|
- name: GITEA_RUNNER_TOKEN_FILE
|
||||||
|
value: /run/dynamic-runner-secrets/scheduler-token
|
||||||
- name: NATS_URL
|
- name: NATS_URL
|
||||||
value: tls://nats.ad.ddupan.top:4222
|
value: tls://nats.ad.ddupan.top:4222
|
||||||
- name: NATS_CA_FILE
|
- name: NATS_CA_FILE
|
||||||
value: /run/trust/ca.pem
|
value: /run/trust/ca.pem
|
||||||
- name: NATS_PASSWORD_FILE
|
- name: NATS_PRODUCER_PASSWORD_FILE
|
||||||
value: /run/dynamic-runner-secrets/nats-password
|
value: /run/dynamic-runner-secrets/nats-password
|
||||||
- name: WEBHOOK_SECRET_FILE
|
- name: NATS_WORKER_PASSWORD_FILE
|
||||||
value: /run/dynamic-runner-secrets/webhook-secret
|
value: /run/dynamic-runner-secrets/nats-worker-password
|
||||||
|
- name: RUNNER_FACADE_CAPABILITY_KEY_FILE
|
||||||
|
value: /run/dynamic-runner-secrets/facade-capability-key
|
||||||
|
- name: RUNNER_FACADE_LISTEN
|
||||||
|
value: :8443
|
||||||
|
- name: RUNNER_FACADE_URL
|
||||||
|
value: https://192.168.10.127:30443
|
||||||
|
- name: RUNNER_FACADE_SPIFFE_ID
|
||||||
|
value: spiffe://ddupan.top/ns/dynamic-runner/sa/dynamic-runner-controller
|
||||||
|
- name: SPIFFE_ENDPOINT_SOCKET
|
||||||
|
value: unix:///run/spire/agent-sockets/spire-agent.sock
|
||||||
|
- name: POD_NAMESPACE
|
||||||
|
valueFrom:
|
||||||
|
fieldRef:
|
||||||
|
fieldPath: metadata.namespace
|
||||||
|
- name: POD_EXECUTOR_IMAGE
|
||||||
|
value: zot.ad.ddupan.top/panxiao81/gitea-dynamic-runner-runner@sha256:325d75a208b1a1c6f3b4d4705e47bbc58b34733edcd12f689c60769ab6772a59
|
||||||
|
- name: POD_SERVICE_ACCOUNT
|
||||||
|
value: gitea-dynamic-runner
|
||||||
|
- name: POD_EXECUTOR_UID
|
||||||
|
value: "2000"
|
||||||
|
- name: POD_CAPACITY
|
||||||
|
value: "4"
|
||||||
|
- name: OPENSANDBOX_API
|
||||||
|
value: http://10.60.0.13:8080
|
||||||
|
- name: OPENSANDBOX_API_KEY_FILE
|
||||||
|
value: /run/dynamic-runner-secrets/opensandbox-api-key
|
||||||
|
- name: OPENSANDBOX_POOL
|
||||||
|
value: ci-vm
|
||||||
|
- name: VM_CAPACITY
|
||||||
|
value: "1"
|
||||||
|
- name: VM_RUNNER_LABEL
|
||||||
|
value: vm
|
||||||
|
- name: VM_TIMEOUT_SECONDS
|
||||||
|
value: "14400"
|
||||||
|
- name: SPIRE_CLUSTER
|
||||||
|
value: homelab
|
||||||
|
- name: SPIRE_CLASS
|
||||||
|
value: spire-mgmt-spire
|
||||||
|
- name: SPIRE_AGENT_ID
|
||||||
|
value: spiffe://ddupan.top/spire/agent/k8s_psat/homelab/cd2d0233-c4ea-4031-8327-e7e359e766dd
|
||||||
|
- name: SSL_CERT_FILE
|
||||||
|
value: /run/trust/ca.pem
|
||||||
ports:
|
ports:
|
||||||
- name: http
|
- name: facade
|
||||||
containerPort: 8787
|
containerPort: 8443
|
||||||
readinessProbe:
|
readinessProbe:
|
||||||
httpGet:
|
httpGet: null
|
||||||
path: /healthz
|
tcpSocket:
|
||||||
port: http
|
port: facade
|
||||||
periodSeconds: 5
|
periodSeconds: 5
|
||||||
livenessProbe:
|
livenessProbe:
|
||||||
httpGet:
|
httpGet: null
|
||||||
path: /healthz
|
tcpSocket:
|
||||||
port: http
|
port: facade
|
||||||
initialDelaySeconds: 10
|
initialDelaySeconds: 15
|
||||||
periodSeconds: 10
|
periodSeconds: 10
|
||||||
resources:
|
resources:
|
||||||
requests:
|
requests:
|
||||||
cpu: 25m
|
cpu: 50m
|
||||||
memory: 32Mi
|
memory: 64Mi
|
||||||
limits:
|
limits:
|
||||||
cpu: 250m
|
cpu: 500m
|
||||||
memory: 128Mi
|
memory: 256Mi
|
||||||
securityContext:
|
securityContext:
|
||||||
allowPrivilegeEscalation: false
|
allowPrivilegeEscalation: false
|
||||||
capabilities:
|
capabilities:
|
||||||
@@ -91,6 +138,9 @@ spec:
|
|||||||
- name: trust
|
- name: trust
|
||||||
mountPath: /run/trust
|
mountPath: /run/trust
|
||||||
readOnly: true
|
readOnly: true
|
||||||
|
- name: spire-agent-socket
|
||||||
|
mountPath: /run/spire/agent-sockets
|
||||||
|
readOnly: true
|
||||||
securityContext:
|
securityContext:
|
||||||
fsGroup: 65532
|
fsGroup: 65532
|
||||||
fsGroupChangePolicy: OnRootMismatch
|
fsGroupChangePolicy: OnRootMismatch
|
||||||
@@ -104,3 +154,7 @@ spec:
|
|||||||
- name: trust
|
- name: trust
|
||||||
emptyDir:
|
emptyDir:
|
||||||
sizeLimit: 1Mi
|
sizeLimit: 1Mi
|
||||||
|
- name: spire-agent-socket
|
||||||
|
csi:
|
||||||
|
driver: csi.spiffe.io
|
||||||
|
readOnly: true
|
||||||
|
|||||||
@@ -20,6 +20,10 @@ spec:
|
|||||||
remoteRef:
|
remoteRef:
|
||||||
key: k8s/nats
|
key: k8s/nats
|
||||||
property: ci_worker_password
|
property: ci_worker_password
|
||||||
|
- secretKey: opensandbox-api-key
|
||||||
|
remoteRef:
|
||||||
|
key: k8s/opensandbox-api
|
||||||
|
property: api_key
|
||||||
- secretKey: webhook-secret
|
- secretKey: webhook-secret
|
||||||
remoteRef:
|
remoteRef:
|
||||||
key: k8s/dynamic-runner
|
key: k8s/dynamic-runner
|
||||||
@@ -28,3 +32,15 @@ spec:
|
|||||||
remoteRef:
|
remoteRef:
|
||||||
key: k8s/gitea-runner
|
key: k8s/gitea-runner
|
||||||
property: token
|
property: token
|
||||||
|
- secretKey: scheduler-uuid
|
||||||
|
remoteRef:
|
||||||
|
key: k8s/dynamic-runner
|
||||||
|
property: scheduler_uuid
|
||||||
|
- secretKey: scheduler-token
|
||||||
|
remoteRef:
|
||||||
|
key: k8s/dynamic-runner
|
||||||
|
property: scheduler_token
|
||||||
|
- secretKey: facade-capability-key
|
||||||
|
remoteRef:
|
||||||
|
key: k8s/dynamic-runner
|
||||||
|
property: facade_capability_key
|
||||||
|
|||||||
@@ -6,5 +6,4 @@ resources:
|
|||||||
- rbac.yaml
|
- rbac.yaml
|
||||||
- clusterspiffeid.yaml
|
- clusterspiffeid.yaml
|
||||||
- deployment.yaml
|
- deployment.yaml
|
||||||
- pod-worker-deployment.yaml
|
|
||||||
- service.yaml
|
- service.yaml
|
||||||
|
|||||||
@@ -1,100 +0,0 @@
|
|||||||
apiVersion: apps/v1
|
|
||||||
kind: Deployment
|
|
||||||
metadata:
|
|
||||||
name: dynamic-runner-pod-worker
|
|
||||||
namespace: dynamic-runner
|
|
||||||
spec:
|
|
||||||
replicas: 1
|
|
||||||
selector:
|
|
||||||
matchLabels:
|
|
||||||
app.kubernetes.io/name: dynamic-runner-pod-worker
|
|
||||||
template:
|
|
||||||
metadata:
|
|
||||||
labels:
|
|
||||||
app.kubernetes.io/name: dynamic-runner-pod-worker
|
|
||||||
spec:
|
|
||||||
serviceAccountName: dynamic-runner-pod-worker
|
|
||||||
initContainers:
|
|
||||||
- name: fetch-internal-ca
|
|
||||||
image: curlimages/curl:8.16.0@sha256:463eaf6072688fe96ac64fa623fe73e1dbe25d8ad6c34404a669ad3ce1f104b6
|
|
||||||
args:
|
|
||||||
- --fail
|
|
||||||
- --silent
|
|
||||||
- --show-error
|
|
||||||
- --output
|
|
||||||
- /trust/ca.pem
|
|
||||||
- https://bao.ad.ddupan.top:8200/v1/pki/ca/pem
|
|
||||||
securityContext:
|
|
||||||
allowPrivilegeEscalation: false
|
|
||||||
capabilities:
|
|
||||||
drop: [ALL]
|
|
||||||
readOnlyRootFilesystem: true
|
|
||||||
runAsNonRoot: true
|
|
||||||
runAsUser: 101
|
|
||||||
runAsGroup: 102
|
|
||||||
seccompProfile:
|
|
||||||
type: RuntimeDefault
|
|
||||||
volumeMounts:
|
|
||||||
- name: trust
|
|
||||||
mountPath: /trust
|
|
||||||
containers:
|
|
||||||
- name: pod-worker
|
|
||||||
image: gitea-dynamic-runner-controller:0.3.0-bootstrap
|
|
||||||
imagePullPolicy: Never
|
|
||||||
command: [/venv/bin/gitea-dynamic-runner-pod-worker]
|
|
||||||
env:
|
|
||||||
- name: NATS_URL
|
|
||||||
value: tls://nats.ad.ddupan.top:4222
|
|
||||||
- name: NATS_CA_FILE
|
|
||||||
value: /run/trust/ca.pem
|
|
||||||
- name: NATS_PASSWORD_FILE
|
|
||||||
value: /run/dynamic-runner-secrets/nats-worker-password
|
|
||||||
- name: RUNNER_NAMESPACE
|
|
||||||
valueFrom:
|
|
||||||
fieldRef:
|
|
||||||
fieldPath: metadata.namespace
|
|
||||||
- name: RUNNER_IMAGE
|
|
||||||
value: zot.ad.ddupan.top/panxiao81/gitea-dynamic-runner-runner@sha256:4c61f6315453d68a827ee9542f1345ed86576eaaf62325eb803c8fe3f06ddf2a
|
|
||||||
- name: RUNNER_SERVICE_ACCOUNT
|
|
||||||
value: gitea-dynamic-runner
|
|
||||||
- name: RUNNER_TOKEN_SECRET
|
|
||||||
value: dynamic-runner
|
|
||||||
- name: RUNNER_CAPACITY
|
|
||||||
value: "4"
|
|
||||||
resources:
|
|
||||||
requests:
|
|
||||||
cpu: 25m
|
|
||||||
memory: 32Mi
|
|
||||||
limits:
|
|
||||||
cpu: 250m
|
|
||||||
memory: 128Mi
|
|
||||||
securityContext:
|
|
||||||
allowPrivilegeEscalation: false
|
|
||||||
capabilities:
|
|
||||||
drop: [ALL]
|
|
||||||
readOnlyRootFilesystem: true
|
|
||||||
runAsNonRoot: true
|
|
||||||
runAsUser: 65532
|
|
||||||
runAsGroup: 65532
|
|
||||||
seccompProfile:
|
|
||||||
type: RuntimeDefault
|
|
||||||
volumeMounts:
|
|
||||||
- name: secret
|
|
||||||
mountPath: /run/dynamic-runner-secrets
|
|
||||||
readOnly: true
|
|
||||||
- name: trust
|
|
||||||
mountPath: /run/trust
|
|
||||||
readOnly: true
|
|
||||||
securityContext:
|
|
||||||
fsGroup: 65532
|
|
||||||
fsGroupChangePolicy: OnRootMismatch
|
|
||||||
seccompProfile:
|
|
||||||
type: RuntimeDefault
|
|
||||||
volumes:
|
|
||||||
- name: secret
|
|
||||||
secret:
|
|
||||||
secretName: dynamic-runner
|
|
||||||
defaultMode: 0400
|
|
||||||
- name: trust
|
|
||||||
emptyDir:
|
|
||||||
sizeLimit: 1Mi
|
|
||||||
@@ -6,12 +6,6 @@ metadata:
|
|||||||
---
|
---
|
||||||
apiVersion: v1
|
apiVersion: v1
|
||||||
kind: ServiceAccount
|
kind: ServiceAccount
|
||||||
metadata:
|
|
||||||
name: dynamic-runner-pod-worker
|
|
||||||
namespace: dynamic-runner
|
|
||||||
---
|
|
||||||
apiVersion: v1
|
|
||||||
kind: ServiceAccount
|
|
||||||
metadata:
|
metadata:
|
||||||
name: gitea-dynamic-runner
|
name: gitea-dynamic-runner
|
||||||
namespace: dynamic-runner
|
namespace: dynamic-runner
|
||||||
@@ -19,23 +13,48 @@ metadata:
|
|||||||
apiVersion: rbac.authorization.k8s.io/v1
|
apiVersion: rbac.authorization.k8s.io/v1
|
||||||
kind: Role
|
kind: Role
|
||||||
metadata:
|
metadata:
|
||||||
name: dynamic-runner-pod-worker
|
name: dynamic-runner-controller
|
||||||
namespace: dynamic-runner
|
namespace: dynamic-runner
|
||||||
rules:
|
rules:
|
||||||
- apiGroups: [""]
|
- apiGroups: [""]
|
||||||
resources: [pods]
|
resources: [pods]
|
||||||
verbs: [create, get, patch, delete]
|
verbs: [create, get, list, watch, patch, delete]
|
||||||
|
- apiGroups: [coordination.k8s.io]
|
||||||
|
resources: [leases]
|
||||||
|
verbs: [create, get, list, watch, update, patch]
|
||||||
---
|
---
|
||||||
apiVersion: rbac.authorization.k8s.io/v1
|
apiVersion: rbac.authorization.k8s.io/v1
|
||||||
kind: RoleBinding
|
kind: RoleBinding
|
||||||
metadata:
|
metadata:
|
||||||
name: dynamic-runner-pod-worker
|
name: dynamic-runner-controller
|
||||||
namespace: dynamic-runner
|
namespace: dynamic-runner
|
||||||
roleRef:
|
roleRef:
|
||||||
apiGroup: rbac.authorization.k8s.io
|
apiGroup: rbac.authorization.k8s.io
|
||||||
kind: Role
|
kind: Role
|
||||||
name: dynamic-runner-pod-worker
|
name: dynamic-runner-controller
|
||||||
subjects:
|
subjects:
|
||||||
- kind: ServiceAccount
|
- kind: ServiceAccount
|
||||||
name: dynamic-runner-pod-worker
|
name: dynamic-runner-controller
|
||||||
|
namespace: dynamic-runner
|
||||||
|
---
|
||||||
|
apiVersion: rbac.authorization.k8s.io/v1
|
||||||
|
kind: ClusterRole
|
||||||
|
metadata:
|
||||||
|
name: dynamic-runner-spiffe-entries
|
||||||
|
rules:
|
||||||
|
- apiGroups: [spire.spiffe.io]
|
||||||
|
resources: [clusterstaticentries]
|
||||||
|
verbs: [create, get, delete]
|
||||||
|
---
|
||||||
|
apiVersion: rbac.authorization.k8s.io/v1
|
||||||
|
kind: ClusterRoleBinding
|
||||||
|
metadata:
|
||||||
|
name: dynamic-runner-spiffe-entries
|
||||||
|
roleRef:
|
||||||
|
apiGroup: rbac.authorization.k8s.io
|
||||||
|
kind: ClusterRole
|
||||||
|
name: dynamic-runner-spiffe-entries
|
||||||
|
subjects:
|
||||||
|
- kind: ServiceAccount
|
||||||
|
name: dynamic-runner-controller
|
||||||
namespace: dynamic-runner
|
namespace: dynamic-runner
|
||||||
|
|||||||
@@ -4,9 +4,11 @@ metadata:
|
|||||||
name: dynamic-runner-controller
|
name: dynamic-runner-controller
|
||||||
namespace: dynamic-runner
|
namespace: dynamic-runner
|
||||||
spec:
|
spec:
|
||||||
|
type: NodePort
|
||||||
selector:
|
selector:
|
||||||
app.kubernetes.io/name: dynamic-runner-controller
|
app.kubernetes.io/name: dynamic-runner-controller
|
||||||
ports:
|
ports:
|
||||||
- name: http
|
- name: facade
|
||||||
port: 8787
|
port: 8443
|
||||||
targetPort: http
|
targetPort: facade
|
||||||
|
nodePort: 30443
|
||||||
|
|||||||
@@ -1,114 +0,0 @@
|
|||||||
# Gitea Actions runner
|
|
||||||
|
|
||||||
This is the bootstrap runner for Gitea Actions. One persistent runner Pod accepts
|
|
||||||
up to four jobs; each job runs in a dynamically created container inside a
|
|
||||||
Docker-in-Docker daemon. The official chart runs DinD privileged. Rootless DinD
|
|
||||||
would still be privileged and is blocked by the node's AppArmor user-namespace
|
|
||||||
policy, so this deployment uses regular DinD instead of weakening that host-wide
|
|
||||||
policy. Only trusted workflows may target this runner.
|
|
||||||
|
|
||||||
DinD 同时使用 `--mtu=1450` 和
|
|
||||||
`--default-network-opt=bridge=com.docker.network.driver.mtu=1450`,与 k3s Pod 的
|
|
||||||
`eth0` 一致。前者只覆盖 Docker 默认 bridge;act 为每个 job 创建 user-defined
|
|
||||||
bridge,必须由后者设置默认 MTU。不要在未验证节点 Pod MTU 的情况下删除或修改这
|
|
||||||
两个参数:MTU 1500 的 job 容器虽然能够解析 GitHub、甚至建立 TCP 连接,但较大的
|
|
||||||
TLS 数据包会在嵌套网络路径中丢失,表现为 `github.com` / `api.github.com` 超时或
|
|
||||||
`setup-go` 每次请求卡满 6 分钟后重试。Pod 网络和默认 Docker bridge 正常不代表
|
|
||||||
Actions job bridge 正常。
|
|
||||||
|
|
||||||
The runner is registered at instance scope so it is available to every repository
|
|
||||||
on this Gitea instance. Repository permissions and protected-branch review are
|
|
||||||
therefore the security boundary; do not enable Actions for untrusted repositories.
|
|
||||||
|
|
||||||
The runner registration token is authoritative in OpenBao at
|
|
||||||
`kv/k8s/gitea-runner`. External Secrets Operator projects its `token` property to
|
|
||||||
the `gitea-runner-token` Secret. Never put the token in this directory or a Helm
|
|
||||||
command line.
|
|
||||||
|
|
||||||
## SPIRE 与 OCI 发布
|
|
||||||
|
|
||||||
runner Pod 使用专用 ServiceAccount `gitea-actions`,并由精确匹配 namespace、
|
|
||||||
ServiceAccount 隐含的 Pod、以及 chart labels 的 `ClusterSPIFFEID` 获得:
|
|
||||||
|
|
||||||
```text
|
|
||||||
spiffe://ddupan.top/ci/gitea-actions
|
|
||||||
```
|
|
||||||
|
|
||||||
SPIFFE CSI socket 同时只读挂载到 runner 和 DinD。act 的 volume allowlist 只允许
|
|
||||||
`/run/spire/agent-sockets`;workflow 仍必须在 job container 中显式请求该 bind
|
|
||||||
mount。原因是 bind mount 由 DinD 内的 dockerd 解析,只挂 runner 容器无法让 job
|
|
||||||
访问 Workload API。
|
|
||||||
|
|
||||||
该身份不是通用 registry 管理员。zot 只对明确列出的 CI 镜像仓库授予
|
|
||||||
`read/create/update`,不授予 delete 或其他仓库写入。workflow 应获取
|
|
||||||
`aud=zot` 的短期 JWT-SVID,并经 stdin 传给 registry client,不得把 JWT、X.509
|
|
||||||
SVID 或 Docker auth 写入 workspace/artifact。
|
|
||||||
|
|
||||||
## Flux 接管状态
|
|
||||||
|
|
||||||
该 release 最初通过下述 review-first 流程手动 bootstrap。下一个 GitOps 阶段将
|
|
||||||
使用 Flux `HelmRelease` 接管它,并首先固定现有 chart `0.1.1`,不在接管 PR 中升级。
|
|
||||||
|
|
||||||
迁移前审计发现:Helm 保存的 user-supplied values 和 release manifest 仍描述失败的
|
|
||||||
rootless DinD 尝试,但 live StatefulSet 与本目录 `values.yaml` 都已经使用 regular
|
|
||||||
DinD。首次 reconcile 的验收条件是修正 Helm 存储状态,同时 live Pod spec、PVC
|
|
||||||
identity、runner capacity 和在线状态保持不变。接管稳定后再用独立 PR 升级 chart。
|
|
||||||
|
|
||||||
接管分两阶段:第一阶段提交 `suspend: true` 的 HelmRelease、HelmRepository 和由
|
|
||||||
`values.yaml` 生成的 ConfigMap。Flux 只登记这些对象,不执行 Helm action。合并后
|
|
||||||
检查 HelmRepository Ready,并用固定 chart 重复比较期望清单与 live StatefulSet;
|
|
||||||
第二阶段解除 suspend。第一阶段已经确认 source Ready、完整 chart render 与 live
|
|
||||||
资源零差异,且登记过程中现有 runner 没有 rollout。失败重试使用
|
|
||||||
`RetryOnFailure`,不会用 stored rootless release 做 rollback。
|
|
||||||
|
|
||||||
## 历史 review-first bootstrap
|
|
||||||
|
|
||||||
这是 Flux 安装前执行过的一次性手动部署流程,保留用于恢复和审计:
|
|
||||||
|
|
||||||
1. Merge the reviewed PR.
|
|
||||||
2. As a Gitea site administrator, create an instance-scoped runner registration
|
|
||||||
token under **Site Administration → Actions → Runners**.
|
|
||||||
3. Store it as the `token` property at `kv/k8s/gitea-runner` without exposing it
|
|
||||||
in shell history:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
read -rsp 'Runner token: ' runner_token
|
|
||||||
printf '%s' "$runner_token" | bao kv put kv/k8s/gitea-runner token=-
|
|
||||||
unset runner_token
|
|
||||||
```
|
|
||||||
|
|
||||||
4. From the updated `main`, create the namespace and ExternalSecret, then wait
|
|
||||||
for `SecretSynced=True`:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
KUBECONFIG="$HOME/.kube/config" k3s kubectl apply \
|
|
||||||
-f platform/gitea-runner/namespace.yaml
|
|
||||||
KUBECONFIG="$HOME/.kube/config" k3s kubectl apply \
|
|
||||||
-f platform/gitea-runner/external-secret.yaml
|
|
||||||
KUBECONFIG="$HOME/.kube/config" k3s kubectl wait \
|
|
||||||
--namespace gitea-actions \
|
|
||||||
--for=condition=Ready externalsecret/gitea-runner-token \
|
|
||||||
--timeout=60s
|
|
||||||
```
|
|
||||||
|
|
||||||
5. Install chart `actions` version `0.1.1` from
|
|
||||||
`https://dl.gitea.com/charts/` with this `values.yaml`:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
helm repo add gitea-charts https://dl.gitea.com/charts/
|
|
||||||
helm repo update gitea-charts
|
|
||||||
helm upgrade --install gitea-actions gitea-charts/actions \
|
|
||||||
--namespace gitea-actions \
|
|
||||||
--version 0.1.1 \
|
|
||||||
--values platform/gitea-runner/values.yaml \
|
|
||||||
--wait --timeout 10m
|
|
||||||
```
|
|
||||||
|
|
||||||
6. Confirm the runner is online, then re-run the queued lint workflow.
|
|
||||||
|
|
||||||
Do not deploy from an unmerged feature branch. Do not use `--set` for the token.
|
|
||||||
|
|
||||||
The 1 GiB PVC preserves `.runner` identity. Docker image layers are ephemeral;
|
|
||||||
the Pod has a 20 GiB ephemeral-storage limit. Terraform apply jobs must use a
|
|
||||||
workflow concurrency group because runner capacity does not serialize access to
|
|
||||||
a shared state.
|
|
||||||
@@ -1,17 +0,0 @@
|
|||||||
apiVersion: spire.spiffe.io/v1alpha1
|
|
||||||
kind: ClusterSPIFFEID
|
|
||||||
metadata:
|
|
||||||
name: gitea-actions
|
|
||||||
spec:
|
|
||||||
className: spire-mgmt-spire
|
|
||||||
spiffeIDTemplate: spiffe://{{ .TrustDomain }}/ci/gitea-actions
|
|
||||||
namespaceSelector:
|
|
||||||
matchLabels:
|
|
||||||
kubernetes.io/metadata.name: gitea-actions
|
|
||||||
podSelector:
|
|
||||||
matchLabels:
|
|
||||||
app.kubernetes.io/instance: gitea-actions
|
|
||||||
app.kubernetes.io/name: actions-runner
|
|
||||||
workloadSelectorTemplates:
|
|
||||||
- k8s:ns:gitea-actions
|
|
||||||
- k8s:sa:gitea-actions
|
|
||||||
@@ -1,8 +0,0 @@
|
|||||||
apiVersion: source.toolkit.fluxcd.io/v1
|
|
||||||
kind: HelmRepository
|
|
||||||
metadata:
|
|
||||||
name: gitea-charts
|
|
||||||
namespace: gitea-actions
|
|
||||||
spec:
|
|
||||||
interval: 1h
|
|
||||||
url: https://dl.gitea.com/charts/
|
|
||||||
@@ -1,18 +0,0 @@
|
|||||||
apiVersion: kustomize.config.k8s.io/v1beta1
|
|
||||||
kind: Kustomization
|
|
||||||
generatorOptions:
|
|
||||||
disableNameSuffixHash: true
|
|
||||||
labels:
|
|
||||||
reconcile.fluxcd.io/watch: Enabled
|
|
||||||
configMapGenerator:
|
|
||||||
- name: gitea-actions-values
|
|
||||||
namespace: gitea-actions
|
|
||||||
files:
|
|
||||||
- values.yaml=values.yaml
|
|
||||||
resources:
|
|
||||||
- namespace.yaml
|
|
||||||
- serviceaccount.yaml
|
|
||||||
- clusterspiffeid.yaml
|
|
||||||
- external-secret.yaml
|
|
||||||
- helmrepository.yaml
|
|
||||||
- helmrelease.yaml
|
|
||||||
@@ -1,6 +0,0 @@
|
|||||||
apiVersion: v1
|
|
||||||
kind: ServiceAccount
|
|
||||||
metadata:
|
|
||||||
name: gitea-actions
|
|
||||||
namespace: gitea-actions
|
|
||||||
automountServiceAccountToken: false
|
|
||||||
@@ -1,81 +0,0 @@
|
|||||||
enabled: true
|
|
||||||
|
|
||||||
giteaRootURL: http://gitea-http.gitea.svc.cluster.local:3000
|
|
||||||
existingSecret: gitea-runner-token
|
|
||||||
existingSecretKey: token
|
|
||||||
|
|
||||||
statefulset:
|
|
||||||
replicas: 1
|
|
||||||
timezone: Etc/UTC
|
|
||||||
serviceAccountName: gitea-actions
|
|
||||||
extraVolumes:
|
|
||||||
- name: spiffe-workload-api
|
|
||||||
csi:
|
|
||||||
driver: csi.spiffe.io
|
|
||||||
readOnly: true
|
|
||||||
securityContext:
|
|
||||||
fsGroup: 1000
|
|
||||||
# Chart 0.1.1 applies this block to both runner and DinD containers.
|
|
||||||
resources:
|
|
||||||
requests:
|
|
||||||
cpu: 250m
|
|
||||||
memory: 512Mi
|
|
||||||
ephemeral-storage: 2Gi
|
|
||||||
limits:
|
|
||||||
cpu: "4"
|
|
||||||
memory: 6Gi
|
|
||||||
ephemeral-storage: 20Gi
|
|
||||||
persistence:
|
|
||||||
size: 1Gi
|
|
||||||
|
|
||||||
runner:
|
|
||||||
registry: docker.io
|
|
||||||
repository: gitea/runner
|
|
||||||
tag: 2.3.0
|
|
||||||
pullPolicy: IfNotPresent
|
|
||||||
extraVolumeMounts:
|
|
||||||
- name: spiffe-workload-api
|
|
||||||
mountPath: /run/spire/agent-sockets
|
|
||||||
readOnly: true
|
|
||||||
config: |
|
|
||||||
log:
|
|
||||||
level: info
|
|
||||||
runner:
|
|
||||||
file: .runner
|
|
||||||
capacity: 4
|
|
||||||
timeout: 3h
|
|
||||||
shutdown_timeout: 3h
|
|
||||||
labels:
|
|
||||||
- self-hosted:docker://docker.gitea.com/runner-images:ubuntu-latest
|
|
||||||
cache:
|
|
||||||
enabled: false
|
|
||||||
container:
|
|
||||||
require_docker: true
|
|
||||||
docker_timeout: 300s
|
|
||||||
# Workflows must still request this exact bind mount explicitly. The
|
|
||||||
# allowlist prevents arbitrary host paths from reaching job containers.
|
|
||||||
valid_volumes:
|
|
||||||
- /run/spire/agent-sockets
|
|
||||||
|
|
||||||
dind:
|
|
||||||
# The node enforces AppArmor's unprivileged-userns restriction, which blocks
|
|
||||||
# rootlesskit even though this chart must run DinD privileged either way.
|
|
||||||
rootless: false
|
|
||||||
registry: docker.io
|
|
||||||
repository: docker
|
|
||||||
tag: 29.7.1-dind
|
|
||||||
pullPolicy: IfNotPresent
|
|
||||||
# Bind mounts are resolved by dockerd, so the CSI socket must exist in the
|
|
||||||
# DinD container as well as in the runner container.
|
|
||||||
extraVolumeMounts:
|
|
||||||
- name: spiffe-workload-api
|
|
||||||
mountPath: /run/spire/agent-sockets
|
|
||||||
readOnly: true
|
|
||||||
# k3s uses a 1450-byte pod MTU. Without matching it here, nested Actions
|
|
||||||
# networks advertise 1500 and GitHub TLS packets disappear on the outer
|
|
||||||
# overlay path while direct pod traffic remains healthy.
|
|
||||||
extraArgs:
|
|
||||||
- --mtu=1450
|
|
||||||
# --mtu only changes Docker's default bridge. act creates a user-defined
|
|
||||||
# bridge per job, so give every new bridge the same explicit default.
|
|
||||||
- --default-network-opt=bridge=com.docker.network.driver.mtu=1450
|
|
||||||
@@ -0,0 +1,84 @@
|
|||||||
|
# OpenSandbox Gitea runner Pools
|
||||||
|
|
||||||
|
本目录只部署零预热的 `ci-vm` Pool、sandbox 内的 SPIFFE identity controller,以及仅供
|
||||||
|
内网 VyOS 转发的 OpenSandbox NodePort。`ci-vm` 使用 `kata-clh-runtime-rs`,每个 VM
|
||||||
|
执行单一任务并在结束后删除。普通 Pod job 直接运行在 homelab,不进入 OpenSandbox。
|
||||||
|
|
||||||
|
Pool 中 task-executor 接收 Lifecycle API 下发的进程环境。guest-local SPIRE Agent 用
|
||||||
|
Pod-bound PSAT 向中央 SPIRE 注册;identity controller 从 BatchSandbox allocation
|
||||||
|
取得真实 Pod UID,再创建精确的 `ClusterStaticEntry`。runner 只有拿到请求中的完整
|
||||||
|
repository/task SVID 后才领取一次性 Gitea registration token。
|
||||||
|
|
||||||
|
VM 与 Pod backend 使用同一个 `gitea-dynamic-runner-runner` executor 镜像。Pool 不运行
|
||||||
|
常驻 `docker:dind` sidecar;需要 Docker 的 workflow 在 privileged executor 内按任务启动
|
||||||
|
daemon。Kata VM 中的 workflow 必须先把稀疏 ext4 镜像 loop-mount 到
|
||||||
|
`/var/lib/docker`,并完成 cgroup v2 nesting 初始化。
|
||||||
|
|
||||||
|
`ci-vm` executor 固定请求并限制为 2 CPU/3GiB。真实 kind canary 表明 1 CPU/约 2GiB
|
||||||
|
虽然能最终启动全部 control-plane 容器,但无法在 kubeadm 超时前提供可用的 API server;
|
||||||
|
该规格是 nested Kubernetes 任务的容量下限,不是用资源掩盖存储阻塞。
|
||||||
|
|
||||||
|
这些 `ClusterStaticEntry` 位于 sandbox 集群,由 central SPIRE Server 内的
|
||||||
|
`spire-controller-manager-sandbox` 通过受限 external kubeconfig reconcile。必须在
|
||||||
|
`platform/spire/values.yaml` 显式启用 external controller-manager 的
|
||||||
|
`reconcile.clusterStaticEntries`(chart 默认关闭);仅看到 CR 存在但没有 status,不算
|
||||||
|
身份链路就绪。
|
||||||
|
|
||||||
|
## 部署依赖
|
||||||
|
|
||||||
|
- OpenSandbox chart 和 CRD 已 Ready;
|
||||||
|
- RuntimeClass `kata-clh-runtime-rs` 已存在;
|
||||||
|
- 中央 SPIRE 已发布 `ConfigMap/opensandbox/spire-bundle-pem`;
|
||||||
|
- VyOS `10.60.0.13:8080` 转发 sandbox1/2 的 NodePort `30080`;
|
||||||
|
- runner/controller 镜像均使用 Zot digest,而不是可变 tag。
|
||||||
|
|
||||||
|
本目录不读取 OpenBao,也不修改 OpenSandbox 平台侧 ExternalSecret、ClusterSecretStore
|
||||||
|
或 Bao policy。OpenSandbox API key 只存在于平台 server Secret 和 homelab controller
|
||||||
|
Secret,两边由各自身份读取同一 Bao 资源。
|
||||||
|
|
||||||
|
## 上线验收
|
||||||
|
|
||||||
|
先确认 Secret 和 API 认证,命令不得输出 key:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
kubectl -n dynamic-runner wait externalsecret/dynamic-runner --for=condition=Ready --timeout=2m
|
||||||
|
kubectl -n dynamic-runner exec deploy/dynamic-runner-controller -- \
|
||||||
|
wget -qO- http://10.60.0.13:8080/health
|
||||||
|
```
|
||||||
|
|
||||||
|
然后触发一个 `runs-on: [self-hosted, vm]` 的最小 workflow,并同时观察:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
kubectl -n dynamic-runner logs deploy/dynamic-runner-controller -f
|
||||||
|
kubectl -n opensandbox get pool,batchsandbox,pod -w
|
||||||
|
kubectl get clusterstaticentry -l app.kubernetes.io/component=opensandbox-identity -w
|
||||||
|
kubectl -n opensandbox logs deploy/opensandbox-identity -f
|
||||||
|
```
|
||||||
|
|
||||||
|
合格证据必须同时包含:
|
||||||
|
|
||||||
|
1. Lifecycle create 成功并选择 `ci-vm`;
|
||||||
|
2. 分配 Pod 的 `runtimeClassName` 为 `kata-clh-runtime-rs`;
|
||||||
|
3. entry 的 parent ID 包含该 Pod 的 UID,SPIFFE ID 使用 repository/task;
|
||||||
|
4. Gitea 显示临时 runner 带 `self-hosted,vm` labels 并完成真实任务;
|
||||||
|
5. 任务后 BatchSandbox、Pod、ClusterStaticEntry 和临时 runner 均消失。
|
||||||
|
|
||||||
|
清理超时对象时只调用 Lifecycle DELETE,不直接删除 Pool Pod。若 controller 已不可用,
|
||||||
|
可从 OpenSandbox API 按 metadata 定位 sandbox 后执行 DELETE;不要绕过 API 伪造状态。
|
||||||
|
|
||||||
|
## 故障定位
|
||||||
|
|
||||||
|
- `401`:检查 homelab ExternalSecret Ready 和文件挂载,不打印 Secret;
|
||||||
|
- `PoolCapacityExhausted`:检查 `ci-vm` 的 `poolMax` 及残留 BatchSandbox;
|
||||||
|
- runner 等待 SVID:核对 allocation Pod UID、ClusterStaticEntry parentID、guest Agent 日志;
|
||||||
|
- runner 等待 token:核对 `192.168.10.127:8787` 的 sandbox 到 homelab 路由;
|
||||||
|
- Docker 任务失败:检查 workflow 的 job-local dockerd 日志及
|
||||||
|
`/var/run/docker.sock`;Pool 不提供共享或常驻 daemon。
|
||||||
|
- kind node 的 systemd 报 `Failed to create /init.scope` 或 `Structure needs
|
||||||
|
cleaning`:确认 workflow 启动 dockerd 前完成 cgroup v2 nesting 初始化。否则
|
||||||
|
子容器的私有 cgroup namespace 根会变成 `threaded`,systemd 无法创建 domain
|
||||||
|
cgroup。
|
||||||
|
- kind 的 kubeadm 卡在 `CreateContainer`,而 nested containerd goroutine 停在
|
||||||
|
`bbolt` 的 `fdatasync`:不要直接把 Kata shared mount 用作 Docker 数据目录。
|
||||||
|
workflow 必须在 guest 内创建稀疏 ext4 镜像并 loop-mount 到
|
||||||
|
`/var/lib/docker`;该镜像随 Sandbox 删除,不得在节点侧遗留。
|
||||||
@@ -0,0 +1,31 @@
|
|||||||
|
apiVersion: v1
|
||||||
|
kind: ConfigMap
|
||||||
|
metadata:
|
||||||
|
name: gitea-ci-spire-agent
|
||||||
|
namespace: opensandbox
|
||||||
|
data:
|
||||||
|
agent.conf: |
|
||||||
|
agent {
|
||||||
|
data_dir = "/run/spire/data"
|
||||||
|
log_level = "INFO"
|
||||||
|
server_address = "spire-server.ad.ddupan.top"
|
||||||
|
server_port = "8081"
|
||||||
|
socket_path = "/run/spire/agent-sockets/spire-agent.sock"
|
||||||
|
trust_bundle_path = "/run/spire/bundle/bundle.pem"
|
||||||
|
trust_domain = "ddupan.top"
|
||||||
|
}
|
||||||
|
|
||||||
|
plugins {
|
||||||
|
NodeAttestor "k8s_psat" {
|
||||||
|
plugin_data {
|
||||||
|
cluster = "sandbox-kata"
|
||||||
|
token_path = "/run/spire/tokens/token"
|
||||||
|
}
|
||||||
|
}
|
||||||
|
KeyManager "memory" {
|
||||||
|
plugin_data {}
|
||||||
|
}
|
||||||
|
WorkloadAttestor "unix" {
|
||||||
|
plugin_data {}
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,41 @@
|
|||||||
|
apiVersion: apps/v1
|
||||||
|
kind: Deployment
|
||||||
|
metadata:
|
||||||
|
name: opensandbox-identity
|
||||||
|
namespace: opensandbox
|
||||||
|
spec:
|
||||||
|
replicas: 1
|
||||||
|
selector:
|
||||||
|
matchLabels:
|
||||||
|
app.kubernetes.io/name: opensandbox-identity
|
||||||
|
template:
|
||||||
|
metadata:
|
||||||
|
labels:
|
||||||
|
app.kubernetes.io/name: opensandbox-identity
|
||||||
|
spec:
|
||||||
|
serviceAccountName: opensandbox-identity
|
||||||
|
containers:
|
||||||
|
- name: controller
|
||||||
|
image: zot.ad.ddupan.top/panxiao81/gitea-dynamic-runner-controller@sha256:dfbfaf2a7aa5951d1dc4e7f941a8cd05ceb4a860ac1ab23eb8be234861edcb55
|
||||||
|
imagePullPolicy: IfNotPresent
|
||||||
|
command: [/venv/bin/gitea-dynamic-runner-opensandbox-identity]
|
||||||
|
resources:
|
||||||
|
requests:
|
||||||
|
cpu: 10m
|
||||||
|
memory: 32Mi
|
||||||
|
limits:
|
||||||
|
cpu: 100m
|
||||||
|
memory: 96Mi
|
||||||
|
securityContext:
|
||||||
|
allowPrivilegeEscalation: false
|
||||||
|
capabilities:
|
||||||
|
drop: [ALL]
|
||||||
|
readOnlyRootFilesystem: true
|
||||||
|
runAsNonRoot: true
|
||||||
|
runAsUser: 65532
|
||||||
|
runAsGroup: 65532
|
||||||
|
seccompProfile:
|
||||||
|
type: RuntimeDefault
|
||||||
|
securityContext:
|
||||||
|
seccompProfile:
|
||||||
|
type: RuntimeDefault
|
||||||
@@ -0,0 +1,22 @@
|
|||||||
|
apiVersion: kustomize.config.k8s.io/v1beta1
|
||||||
|
kind: Kustomization
|
||||||
|
resources:
|
||||||
|
- service.yaml
|
||||||
|
- agent-config.yaml
|
||||||
|
- rbac.yaml
|
||||||
|
- identity-controller.yaml
|
||||||
|
- pools.yaml
|
||||||
|
patches:
|
||||||
|
# Pod jobs run directly in the homelab cluster. Keep OpenSandbox VM-only.
|
||||||
|
- target:
|
||||||
|
group: sandbox.opensandbox.io
|
||||||
|
version: v1alpha1
|
||||||
|
kind: Pool
|
||||||
|
name: ci-pod
|
||||||
|
patch: |-
|
||||||
|
apiVersion: sandbox.opensandbox.io/v1alpha1
|
||||||
|
kind: Pool
|
||||||
|
metadata:
|
||||||
|
name: ci-pod
|
||||||
|
namespace: opensandbox
|
||||||
|
$patch: delete
|
||||||
@@ -0,0 +1,293 @@
|
|||||||
|
---
|
||||||
|
apiVersion: sandbox.opensandbox.io/v1alpha1
|
||||||
|
kind: Pool
|
||||||
|
metadata:
|
||||||
|
name: ci-vm
|
||||||
|
namespace: opensandbox
|
||||||
|
labels:
|
||||||
|
app.kubernetes.io/name: gitea-ci-vm
|
||||||
|
app.kubernetes.io/component: runner-pool
|
||||||
|
spec:
|
||||||
|
capacitySpec:
|
||||||
|
bufferMax: 0
|
||||||
|
bufferMin: 0
|
||||||
|
poolMax: 2
|
||||||
|
poolMin: 0
|
||||||
|
recycleStrategy:
|
||||||
|
type: Delete
|
||||||
|
template:
|
||||||
|
metadata:
|
||||||
|
labels:
|
||||||
|
app.kubernetes.io/name: gitea-ci-vm
|
||||||
|
ci.ddupan.top/backend: vm
|
||||||
|
spec:
|
||||||
|
runtimeClassName: kata-clh-runtime-rs
|
||||||
|
serviceAccountName: gitea-ci
|
||||||
|
restartPolicy: Never
|
||||||
|
terminationGracePeriodSeconds: 30
|
||||||
|
shareProcessNamespace: true
|
||||||
|
securityContext:
|
||||||
|
fsGroup: 2000
|
||||||
|
fsGroupChangePolicy: OnRootMismatch
|
||||||
|
initContainers:
|
||||||
|
- name: task-executor-installer
|
||||||
|
image: sandbox-registry.cn-zhangjiakou.cr.aliyuncs.com/opensandbox/task-executor:v0.1.0
|
||||||
|
command: [/bin/sh, -c]
|
||||||
|
args:
|
||||||
|
- cp /workspace/server /opt/opensandbox/task-executor && chmod 0755 /opt/opensandbox/task-executor
|
||||||
|
volumeMounts:
|
||||||
|
- name: opensandbox-bin
|
||||||
|
mountPath: /opt/opensandbox
|
||||||
|
- name: execd-installer
|
||||||
|
image: sandbox-registry.cn-zhangjiakou.cr.aliyuncs.com/opensandbox/execd:v1.0.22
|
||||||
|
command: [/bin/sh, -c]
|
||||||
|
args:
|
||||||
|
- cp ./execd /opt/opensandbox/execd && cp ./bootstrap.sh /opt/opensandbox/bootstrap.sh && chmod 0755 /opt/opensandbox/execd /opt/opensandbox/bootstrap.sh
|
||||||
|
volumeMounts:
|
||||||
|
- name: opensandbox-bin
|
||||||
|
mountPath: /opt/opensandbox
|
||||||
|
containers:
|
||||||
|
- name: sandbox
|
||||||
|
image: zot.ad.ddupan.top/panxiao81/gitea-dynamic-runner-runner@sha256:325d75a208b1a1c6f3b4d4705e47bbc58b34733edcd12f689c60769ab6772a59
|
||||||
|
imagePullPolicy: IfNotPresent
|
||||||
|
command: [/opt/opensandbox/task-executor]
|
||||||
|
args:
|
||||||
|
- -listen-addr=0.0.0.0:5758
|
||||||
|
- -log-dir=/tmp
|
||||||
|
- -data-dir=/tmp/tasks
|
||||||
|
env:
|
||||||
|
- name: SANDBOX_MAIN_CONTAINER
|
||||||
|
value: sandbox
|
||||||
|
- name: EXECD_ENVS
|
||||||
|
value: /opt/opensandbox/.env
|
||||||
|
- name: EXECD
|
||||||
|
value: /opt/opensandbox/execd
|
||||||
|
- name: GITEA_INSTANCE_URL
|
||||||
|
value: https://git.ddupan.top
|
||||||
|
- name: HOME
|
||||||
|
value: /data
|
||||||
|
ports:
|
||||||
|
- name: task-executor
|
||||||
|
containerPort: 5758
|
||||||
|
securityContext:
|
||||||
|
privileged: true
|
||||||
|
runAsNonRoot: true
|
||||||
|
runAsUser: 2000
|
||||||
|
runAsGroup: 2000
|
||||||
|
resources:
|
||||||
|
requests:
|
||||||
|
cpu: "2"
|
||||||
|
memory: 3Gi
|
||||||
|
limits:
|
||||||
|
cpu: "2"
|
||||||
|
memory: 3Gi
|
||||||
|
volumeMounts:
|
||||||
|
- name: opensandbox-bin
|
||||||
|
mountPath: /opt/opensandbox
|
||||||
|
- name: runner-data
|
||||||
|
mountPath: /data
|
||||||
|
- name: runner-workspace
|
||||||
|
mountPath: /workspace
|
||||||
|
- name: spire-socket
|
||||||
|
mountPath: /run/spire/agent-sockets
|
||||||
|
- name: spire-agent
|
||||||
|
image: ghcr.io/spiffe/spire-agent:1.15.3@sha256:41b0dcd8b258a69db9e2768292a060766fb76fd866e4bc925849981ea1b825ff
|
||||||
|
args: [-config, /run/spire/config/agent.conf]
|
||||||
|
securityContext:
|
||||||
|
allowPrivilegeEscalation: false
|
||||||
|
capabilities:
|
||||||
|
drop: [ALL]
|
||||||
|
volumeMounts:
|
||||||
|
- name: spire-config
|
||||||
|
mountPath: /run/spire/config
|
||||||
|
readOnly: true
|
||||||
|
- name: spire-bundle
|
||||||
|
mountPath: /run/spire/bundle
|
||||||
|
readOnly: true
|
||||||
|
- name: spire-token
|
||||||
|
mountPath: /run/spire/tokens
|
||||||
|
readOnly: true
|
||||||
|
- name: spire-data
|
||||||
|
mountPath: /run/spire/data
|
||||||
|
- name: spire-socket
|
||||||
|
mountPath: /run/spire/agent-sockets
|
||||||
|
volumes:
|
||||||
|
- name: opensandbox-bin
|
||||||
|
emptyDir: {}
|
||||||
|
- name: runner-data
|
||||||
|
emptyDir: {}
|
||||||
|
- name: runner-workspace
|
||||||
|
emptyDir: {}
|
||||||
|
- name: spire-data
|
||||||
|
emptyDir:
|
||||||
|
medium: Memory
|
||||||
|
- name: spire-socket
|
||||||
|
emptyDir:
|
||||||
|
medium: Memory
|
||||||
|
- name: spire-config
|
||||||
|
configMap:
|
||||||
|
name: gitea-ci-spire-agent
|
||||||
|
- name: spire-bundle
|
||||||
|
configMap:
|
||||||
|
name: spire-bundle-pem
|
||||||
|
- name: spire-token
|
||||||
|
projected:
|
||||||
|
sources:
|
||||||
|
- serviceAccountToken:
|
||||||
|
audience: spire-server
|
||||||
|
expirationSeconds: 3600
|
||||||
|
path: token
|
||||||
|
---
|
||||||
|
apiVersion: sandbox.opensandbox.io/v1alpha1
|
||||||
|
kind: Pool
|
||||||
|
metadata:
|
||||||
|
name: ci-pod
|
||||||
|
namespace: opensandbox
|
||||||
|
labels:
|
||||||
|
app.kubernetes.io/name: gitea-ci-pod
|
||||||
|
app.kubernetes.io/component: runner-pool
|
||||||
|
spec:
|
||||||
|
capacitySpec:
|
||||||
|
bufferMax: 0
|
||||||
|
bufferMin: 0
|
||||||
|
poolMax: 4
|
||||||
|
poolMin: 0
|
||||||
|
recycleStrategy:
|
||||||
|
type: Delete
|
||||||
|
template:
|
||||||
|
metadata:
|
||||||
|
labels:
|
||||||
|
app.kubernetes.io/name: gitea-ci-pod
|
||||||
|
ci.ddupan.top/backend: pod
|
||||||
|
spec:
|
||||||
|
serviceAccountName: gitea-ci
|
||||||
|
restartPolicy: Never
|
||||||
|
terminationGracePeriodSeconds: 30
|
||||||
|
shareProcessNamespace: true
|
||||||
|
securityContext:
|
||||||
|
fsGroup: 2000
|
||||||
|
fsGroupChangePolicy: OnRootMismatch
|
||||||
|
initContainers:
|
||||||
|
- name: task-executor-installer
|
||||||
|
image: sandbox-registry.cn-zhangjiakou.cr.aliyuncs.com/opensandbox/task-executor:v0.1.0
|
||||||
|
command: [/bin/sh, -c]
|
||||||
|
args:
|
||||||
|
- cp /workspace/server /opt/opensandbox/task-executor && chmod 0755 /opt/opensandbox/task-executor
|
||||||
|
volumeMounts:
|
||||||
|
- name: opensandbox-bin
|
||||||
|
mountPath: /opt/opensandbox
|
||||||
|
- name: execd-installer
|
||||||
|
image: sandbox-registry.cn-zhangjiakou.cr.aliyuncs.com/opensandbox/execd:v1.0.22
|
||||||
|
command: [/bin/sh, -c]
|
||||||
|
args:
|
||||||
|
- cp ./execd /opt/opensandbox/execd && cp ./bootstrap.sh /opt/opensandbox/bootstrap.sh && chmod 0755 /opt/opensandbox/execd /opt/opensandbox/bootstrap.sh
|
||||||
|
volumeMounts:
|
||||||
|
- name: opensandbox-bin
|
||||||
|
mountPath: /opt/opensandbox
|
||||||
|
containers:
|
||||||
|
- name: sandbox
|
||||||
|
image: zot.ad.ddupan.top/panxiao81/gitea-dynamic-runner-runner@sha256:325d75a208b1a1c6f3b4d4705e47bbc58b34733edcd12f689c60769ab6772a59
|
||||||
|
imagePullPolicy: IfNotPresent
|
||||||
|
command: [/opt/opensandbox/task-executor]
|
||||||
|
args:
|
||||||
|
- -listen-addr=0.0.0.0:5758
|
||||||
|
- -log-dir=/tmp
|
||||||
|
- -data-dir=/tmp/tasks
|
||||||
|
env:
|
||||||
|
- name: SANDBOX_MAIN_CONTAINER
|
||||||
|
value: sandbox
|
||||||
|
- name: EXECD_ENVS
|
||||||
|
value: /opt/opensandbox/.env
|
||||||
|
- name: EXECD
|
||||||
|
value: /opt/opensandbox/execd
|
||||||
|
- name: GITEA_INSTANCE_URL
|
||||||
|
value: https://git.ddupan.top
|
||||||
|
- name: HOME
|
||||||
|
value: /data
|
||||||
|
- name: DOCKER_HOST
|
||||||
|
value: unix:///run/docker/docker.sock
|
||||||
|
ports:
|
||||||
|
- name: task-executor
|
||||||
|
containerPort: 5758
|
||||||
|
securityContext:
|
||||||
|
allowPrivilegeEscalation: false
|
||||||
|
capabilities:
|
||||||
|
drop: [ALL]
|
||||||
|
runAsNonRoot: true
|
||||||
|
runAsUser: 2000
|
||||||
|
runAsGroup: 2000
|
||||||
|
volumeMounts:
|
||||||
|
- name: opensandbox-bin
|
||||||
|
mountPath: /opt/opensandbox
|
||||||
|
- name: runner-data
|
||||||
|
mountPath: /data
|
||||||
|
- name: runner-workspace
|
||||||
|
mountPath: /workspace
|
||||||
|
- name: docker-socket
|
||||||
|
mountPath: /run/docker
|
||||||
|
- name: spire-socket
|
||||||
|
mountPath: /run/spire/agent-sockets
|
||||||
|
- name: spire-agent
|
||||||
|
image: ghcr.io/spiffe/spire-agent:1.15.3@sha256:41b0dcd8b258a69db9e2768292a060766fb76fd866e4bc925849981ea1b825ff
|
||||||
|
args: [-config, /run/spire/config/agent.conf]
|
||||||
|
securityContext:
|
||||||
|
allowPrivilegeEscalation: false
|
||||||
|
capabilities:
|
||||||
|
drop: [ALL]
|
||||||
|
volumeMounts:
|
||||||
|
- name: spire-config
|
||||||
|
mountPath: /run/spire/config
|
||||||
|
readOnly: true
|
||||||
|
- name: spire-bundle
|
||||||
|
mountPath: /run/spire/bundle
|
||||||
|
readOnly: true
|
||||||
|
- name: spire-token
|
||||||
|
mountPath: /run/spire/tokens
|
||||||
|
readOnly: true
|
||||||
|
- name: spire-data
|
||||||
|
mountPath: /run/spire/data
|
||||||
|
- name: spire-socket
|
||||||
|
mountPath: /run/spire/agent-sockets
|
||||||
|
- name: docker
|
||||||
|
image: docker.io/library/docker:29.1.5-dind
|
||||||
|
command: [/bin/sh, -c]
|
||||||
|
args:
|
||||||
|
- test -e /dev/kmsg || mknod /dev/kmsg c 1 11; exec dockerd --host=unix:///run/docker/docker.sock --group=2000 --storage-driver=overlay2
|
||||||
|
securityContext:
|
||||||
|
privileged: true
|
||||||
|
volumeMounts:
|
||||||
|
- name: docker-socket
|
||||||
|
mountPath: /run/docker
|
||||||
|
- name: docker-data
|
||||||
|
mountPath: /var/lib/docker
|
||||||
|
volumes:
|
||||||
|
- name: opensandbox-bin
|
||||||
|
emptyDir: {}
|
||||||
|
- name: runner-data
|
||||||
|
emptyDir: {}
|
||||||
|
- name: runner-workspace
|
||||||
|
emptyDir: {}
|
||||||
|
- name: docker-data
|
||||||
|
emptyDir: {}
|
||||||
|
- name: docker-socket
|
||||||
|
emptyDir:
|
||||||
|
medium: Memory
|
||||||
|
- name: spire-data
|
||||||
|
emptyDir:
|
||||||
|
medium: Memory
|
||||||
|
- name: spire-socket
|
||||||
|
emptyDir:
|
||||||
|
medium: Memory
|
||||||
|
- name: spire-config
|
||||||
|
configMap:
|
||||||
|
name: gitea-ci-spire-agent
|
||||||
|
- name: spire-bundle
|
||||||
|
configMap:
|
||||||
|
name: spire-bundle-pem
|
||||||
|
- name: spire-token
|
||||||
|
projected:
|
||||||
|
sources:
|
||||||
|
- serviceAccountToken:
|
||||||
|
audience: spire-server
|
||||||
|
expirationSeconds: 3600
|
||||||
|
path: token
|
||||||
@@ -0,0 +1,87 @@
|
|||||||
|
---
|
||||||
|
apiVersion: v1
|
||||||
|
kind: ServiceAccount
|
||||||
|
metadata:
|
||||||
|
name: gitea-ci
|
||||||
|
namespace: opensandbox
|
||||||
|
---
|
||||||
|
apiVersion: v1
|
||||||
|
kind: ServiceAccount
|
||||||
|
metadata:
|
||||||
|
name: opensandbox-identity
|
||||||
|
namespace: opensandbox
|
||||||
|
---
|
||||||
|
apiVersion: rbac.authorization.k8s.io/v1
|
||||||
|
kind: Role
|
||||||
|
metadata:
|
||||||
|
name: opensandbox-identity
|
||||||
|
namespace: opensandbox
|
||||||
|
rules:
|
||||||
|
- apiGroups: [""]
|
||||||
|
resources: [pods]
|
||||||
|
verbs: [get, list, watch]
|
||||||
|
- apiGroups: [sandbox.opensandbox.io]
|
||||||
|
resources: [batchsandboxes]
|
||||||
|
verbs: [get, list, watch]
|
||||||
|
---
|
||||||
|
apiVersion: rbac.authorization.k8s.io/v1
|
||||||
|
kind: RoleBinding
|
||||||
|
metadata:
|
||||||
|
name: opensandbox-identity
|
||||||
|
namespace: opensandbox
|
||||||
|
roleRef:
|
||||||
|
apiGroup: rbac.authorization.k8s.io
|
||||||
|
kind: Role
|
||||||
|
name: opensandbox-identity
|
||||||
|
subjects:
|
||||||
|
- kind: ServiceAccount
|
||||||
|
name: opensandbox-identity
|
||||||
|
namespace: opensandbox
|
||||||
|
---
|
||||||
|
apiVersion: rbac.authorization.k8s.io/v1
|
||||||
|
kind: ClusterRole
|
||||||
|
metadata:
|
||||||
|
name: opensandbox-identity
|
||||||
|
rules:
|
||||||
|
- apiGroups: [spire.spiffe.io]
|
||||||
|
resources: [clusterstaticentries]
|
||||||
|
verbs: [create, delete, get, list, watch]
|
||||||
|
---
|
||||||
|
apiVersion: rbac.authorization.k8s.io/v1
|
||||||
|
kind: ClusterRoleBinding
|
||||||
|
metadata:
|
||||||
|
name: opensandbox-identity
|
||||||
|
roleRef:
|
||||||
|
apiGroup: rbac.authorization.k8s.io
|
||||||
|
kind: ClusterRole
|
||||||
|
name: opensandbox-identity
|
||||||
|
subjects:
|
||||||
|
- kind: ServiceAccount
|
||||||
|
name: opensandbox-identity
|
||||||
|
namespace: opensandbox
|
||||||
|
---
|
||||||
|
# The central SPIRE external controller uses this existing sandbox credential
|
||||||
|
# to publish a PEM bundle for guest-local Agents. It cannot read Secrets here.
|
||||||
|
apiVersion: rbac.authorization.k8s.io/v1
|
||||||
|
kind: Role
|
||||||
|
metadata:
|
||||||
|
name: spire-runner-bundle-publisher
|
||||||
|
namespace: opensandbox
|
||||||
|
rules:
|
||||||
|
- apiGroups: [""]
|
||||||
|
resources: [configmaps]
|
||||||
|
verbs: [create, delete, get, list, patch, update, watch]
|
||||||
|
---
|
||||||
|
apiVersion: rbac.authorization.k8s.io/v1
|
||||||
|
kind: RoleBinding
|
||||||
|
metadata:
|
||||||
|
name: spire-runner-bundle-publisher
|
||||||
|
namespace: opensandbox
|
||||||
|
roleRef:
|
||||||
|
apiGroup: rbac.authorization.k8s.io
|
||||||
|
kind: Role
|
||||||
|
name: spire-runner-bundle-publisher
|
||||||
|
subjects:
|
||||||
|
- kind: ServiceAccount
|
||||||
|
name: spire-controller-manager
|
||||||
|
namespace: spire-system
|
||||||
@@ -0,0 +1,15 @@
|
|||||||
|
apiVersion: v1
|
||||||
|
kind: Service
|
||||||
|
metadata:
|
||||||
|
name: opensandbox-server-internal
|
||||||
|
namespace: opensandbox-system
|
||||||
|
spec:
|
||||||
|
type: NodePort
|
||||||
|
selector:
|
||||||
|
app.kubernetes.io/instance: opensandbox
|
||||||
|
app.kubernetes.io/name: opensandbox-server
|
||||||
|
ports:
|
||||||
|
- name: http
|
||||||
|
port: 80
|
||||||
|
targetPort: http
|
||||||
|
nodePort: 30080
|
||||||
@@ -0,0 +1,56 @@
|
|||||||
|
# Sandbox External Secrets Operator
|
||||||
|
|
||||||
|
本目录在 sandbox 集群部署独立的 External Secrets Operator `2.8.0`,并通过
|
||||||
|
`ClusterSecretStore/openbao` 读取 OpenBao KV v2 中共享的 OpenSandbox API key。它不复用
|
||||||
|
homelab 集群的 ESO Pod、ServiceAccount 或 Kubernetes auth backend。
|
||||||
|
|
||||||
|
## 当前状态
|
||||||
|
|
||||||
|
- OpenBao `auth/kubernetes-sandbox`、backend config、`external-secrets` role 与
|
||||||
|
`sandbox-external-secrets` policy 已于 2026-09-18 由 Terraform 创建;apply 后 plan
|
||||||
|
为 zero-diff;
|
||||||
|
- `kv/k8s/opensandbox-api` 已由本机 `spiffe://ddupan.top/dev/panxiao81` 身份生成并写入,
|
||||||
|
值未输出或落盘;
|
||||||
|
- sandbox ESO operator、`ClusterSecretStore/openbao` 与 OpenSandbox `ExternalSecret`
|
||||||
|
由 Flux 管理;
|
||||||
|
- 线上 `ClusterSecretStore/openbao` 为 `Valid/Ready`,`ExternalSecret/opensandbox-api-key`
|
||||||
|
为 `SecretSynced/Ready`;
|
||||||
|
- OpenSandbox 已切换到 API key:无 key 请求返回 `401`,正确 key 请求返回 `200`;
|
||||||
|
- homelab runner 对同一 key 的投影不在本目录,留给 runner 项目管理。
|
||||||
|
|
||||||
|
OpenBao 的 `auth/kubernetes-sandbox`、对应 role、policy、sandbox API 地址与公开 CA
|
||||||
|
完全由 `infrastructure/openbao/terraform/` 管理。CA 文件提交到 Git 是刻意设计:CA
|
||||||
|
是公开信任材料,版本化后集群重建造成的 trust root 变化会产生可审计的 Terraform diff。
|
||||||
|
|
||||||
|
ESO 使用 TokenRequest 生成短期 ServiceAccount JWT。OpenBao 未配置长期
|
||||||
|
`token_reviewer_jwt`,而是使用登录 JWT 调用 sandbox TokenReview;因此
|
||||||
|
`external-secrets` ServiceAccount 仅额外绑定内建 `system:auth-delegator`。
|
||||||
|
|
||||||
|
## 重建顺序
|
||||||
|
|
||||||
|
1. 在 OpenBao 写入 OpenSandbox API key:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
openssl rand -hex 32 | bao kv put kv/k8s/opensandbox-api api_key=-
|
||||||
|
```
|
||||||
|
|
||||||
|
2. 在 `infrastructure/openbao/terraform` 执行 `terraform plan` 和 `terraform apply`,
|
||||||
|
创建 `kubernetes-sandbox` auth mount、backend config、role 与只允许读取
|
||||||
|
`kv/k8s/opensandbox-api` 的最小权限 policy。
|
||||||
|
3. 合并 Flux 变更。依次等待 `flux-system/external-secrets-operator`、
|
||||||
|
`flux-system/external-secrets` Ready,再等待 `flux-system/opensandbox` 滚动完成。
|
||||||
|
|
||||||
|
operator 与配置拆成两个 Flux Kustomization,确保全新集群先安装 CRD,再声明
|
||||||
|
`ClusterSecretStore`;不要为了减少目录而把两层重新合并。
|
||||||
|
|
||||||
|
## 验收
|
||||||
|
|
||||||
|
```bash
|
||||||
|
kubectl get clustersecretstore openbao
|
||||||
|
kubectl -n opensandbox-system get externalsecret opensandbox-api-key
|
||||||
|
kubectl -n opensandbox-system get secret opensandbox-api-key
|
||||||
|
```
|
||||||
|
|
||||||
|
只检查 Secret 是否存在及 key 名,不输出 `data`。`ClusterSecretStore` 或
|
||||||
|
`ExternalSecret` 不 Ready 时,先检查 `auth/kubernetes-sandbox`,不要临时创建静态
|
||||||
|
Bao token Secret。
|
||||||
@@ -0,0 +1,13 @@
|
|||||||
|
---
|
||||||
|
apiVersion: rbac.authorization.k8s.io/v1
|
||||||
|
kind: ClusterRoleBinding
|
||||||
|
metadata:
|
||||||
|
name: sandbox-external-secrets-token-review
|
||||||
|
roleRef:
|
||||||
|
apiGroup: rbac.authorization.k8s.io
|
||||||
|
kind: ClusterRole
|
||||||
|
name: system:auth-delegator
|
||||||
|
subjects:
|
||||||
|
- kind: ServiceAccount
|
||||||
|
name: external-secrets
|
||||||
|
namespace: external-secrets
|
||||||
@@ -0,0 +1,18 @@
|
|||||||
|
---
|
||||||
|
apiVersion: external-secrets.io/v1
|
||||||
|
kind: ClusterSecretStore
|
||||||
|
metadata:
|
||||||
|
name: openbao
|
||||||
|
spec:
|
||||||
|
provider:
|
||||||
|
vault:
|
||||||
|
server: https://bao.ad.ddupan.top:8200
|
||||||
|
path: kv
|
||||||
|
version: v2
|
||||||
|
auth:
|
||||||
|
kubernetes:
|
||||||
|
mountPath: kubernetes-sandbox
|
||||||
|
role: external-secrets
|
||||||
|
serviceAccountRef:
|
||||||
|
name: external-secrets
|
||||||
|
namespace: external-secrets
|
||||||
+3
-1
@@ -1,4 +1,6 @@
|
|||||||
|
---
|
||||||
apiVersion: kustomize.config.k8s.io/v1beta1
|
apiVersion: kustomize.config.k8s.io/v1beta1
|
||||||
kind: Kustomization
|
kind: Kustomization
|
||||||
resources:
|
resources:
|
||||||
- pools.yaml
|
- auth-delegator.yaml
|
||||||
|
- clustersecretstore.yaml
|
||||||
+12
-8
@@ -1,31 +1,35 @@
|
|||||||
|
---
|
||||||
apiVersion: helm.toolkit.fluxcd.io/v2
|
apiVersion: helm.toolkit.fluxcd.io/v2
|
||||||
kind: HelmRelease
|
kind: HelmRelease
|
||||||
metadata:
|
metadata:
|
||||||
name: gitea-actions
|
name: external-secrets
|
||||||
namespace: gitea-actions
|
namespace: external-secrets
|
||||||
spec:
|
spec:
|
||||||
chart:
|
chart:
|
||||||
spec:
|
spec:
|
||||||
chart: actions
|
chart: external-secrets
|
||||||
interval: 1h
|
interval: 1h
|
||||||
sourceRef:
|
sourceRef:
|
||||||
kind: HelmRepository
|
kind: HelmRepository
|
||||||
name: gitea-charts
|
name: external-secrets
|
||||||
version: 0.1.1
|
version: 2.8.0
|
||||||
driftDetection:
|
driftDetection:
|
||||||
mode: enabled
|
mode: enabled
|
||||||
install:
|
install:
|
||||||
|
crds: CreateReplace
|
||||||
strategy:
|
strategy:
|
||||||
name: RetryOnFailure
|
name: RetryOnFailure
|
||||||
retryInterval: 5m
|
retryInterval: 5m
|
||||||
interval: 30m
|
interval: 30m
|
||||||
releaseName: gitea-actions
|
releaseName: external-secrets
|
||||||
targetNamespace: gitea-actions
|
targetNamespace: external-secrets
|
||||||
timeout: 10m
|
timeout: 10m
|
||||||
upgrade:
|
upgrade:
|
||||||
|
crds: CreateReplace
|
||||||
strategy:
|
strategy:
|
||||||
name: RetryOnFailure
|
name: RetryOnFailure
|
||||||
retryInterval: 5m
|
retryInterval: 5m
|
||||||
valuesFrom:
|
valuesFrom:
|
||||||
- kind: ConfigMap
|
- kind: ConfigMap
|
||||||
name: gitea-actions-values
|
name: external-secrets-values
|
||||||
|
valuesKey: values.yaml
|
||||||
@@ -0,0 +1,8 @@
|
|||||||
|
---
|
||||||
|
apiVersion: kustomize.config.k8s.io/v1beta1
|
||||||
|
kind: Kustomization
|
||||||
|
resources:
|
||||||
|
- namespace.yaml
|
||||||
|
- repository.yaml
|
||||||
|
- values.yaml
|
||||||
|
- helmrelease.yaml
|
||||||
+2
-1
@@ -1,4 +1,5 @@
|
|||||||
|
---
|
||||||
apiVersion: v1
|
apiVersion: v1
|
||||||
kind: Namespace
|
kind: Namespace
|
||||||
metadata:
|
metadata:
|
||||||
name: gitea-actions
|
name: external-secrets
|
||||||
@@ -0,0 +1,9 @@
|
|||||||
|
---
|
||||||
|
apiVersion: source.toolkit.fluxcd.io/v1
|
||||||
|
kind: HelmRepository
|
||||||
|
metadata:
|
||||||
|
name: external-secrets
|
||||||
|
namespace: external-secrets
|
||||||
|
spec:
|
||||||
|
interval: 1h
|
||||||
|
url: https://charts.external-secrets.io
|
||||||
@@ -0,0 +1,31 @@
|
|||||||
|
---
|
||||||
|
apiVersion: v1
|
||||||
|
kind: ConfigMap
|
||||||
|
metadata:
|
||||||
|
name: external-secrets-values
|
||||||
|
namespace: external-secrets
|
||||||
|
data:
|
||||||
|
values.yaml: |
|
||||||
|
replicaCount: 1
|
||||||
|
|
||||||
|
webhook:
|
||||||
|
replicaCount: 1
|
||||||
|
resources:
|
||||||
|
requests: {cpu: 10m, memory: 32Mi}
|
||||||
|
limits: {memory: 128Mi}
|
||||||
|
|
||||||
|
certController:
|
||||||
|
replicaCount: 1
|
||||||
|
resources:
|
||||||
|
requests: {cpu: 10m, memory: 32Mi}
|
||||||
|
limits: {memory: 128Mi}
|
||||||
|
|
||||||
|
resources:
|
||||||
|
requests: {cpu: 10m, memory: 64Mi}
|
||||||
|
limits: {memory: 256Mi}
|
||||||
|
|
||||||
|
serviceAccount:
|
||||||
|
create: true
|
||||||
|
name: external-secrets
|
||||||
|
|
||||||
|
installCRDs: true
|
||||||
@@ -28,6 +28,32 @@ Pod 删除后的 backing-file/VMM 回收和真实构建基准必须作为上线
|
|||||||
由 `VMPodScrape` 写入中央 VictoriaMetrics。它不拥有 Kubernetes API 凭据或 host 写
|
由 `VMPodScrape` 写入中央 VictoriaMetrics。它不拥有 Kubernetes API 凭据或 host 写
|
||||||
权限。
|
权限。
|
||||||
|
|
||||||
|
## Guest 内 SPIFFE 身份
|
||||||
|
|
||||||
|
Kata guest 不能直接使用 node SPIRE Agent 的 CSI socket。Unix socket 的路径即使通过
|
||||||
|
virtio-fs 出现在 guest 中,连接也不能跨 VM 边界。CI Pod 应以 native sidecar 在同一
|
||||||
|
guest 内启动临时 SPIRE Agent,并满足以下约束:
|
||||||
|
|
||||||
|
1. 外层 Pod 挂载 `audience=spire-server` 的 Pod-bound projected ServiceAccount token;
|
||||||
|
2. 内层 Agent 使用 `k8s_psat` 向中央 Server attestation,Server cluster profile 必须
|
||||||
|
启用 `use_pod_uid_for_agent_id`,使 Agent ID 包含 Pod UID;
|
||||||
|
3. 调度器为该具体 Agent 创建 registration entry;SPIFFE ID 使用仓库和任务名等稳定的
|
||||||
|
业务语义,Pod UID 只作为 parent binding,不进入业务身份;
|
||||||
|
4. Agent 与 workload 通过 guest 内 `emptyDir` 上的 Unix socket 通信;Pod 必须设置
|
||||||
|
`shareProcessNamespace: true`,否则 `unix` workload attestor 无法从共享 `/proc`
|
||||||
|
解析客户端的 `SO_PEERCRED` PID;
|
||||||
|
5. 调度器必须在 registration entry 已同步到 Agent 后才放行 workload。Pod 删除后同步
|
||||||
|
删除 entry,并 evict 对应的临时 Agent 记录。
|
||||||
|
|
||||||
|
2026-09-17 的 live PoC 已验证:以 Pod UID 作为 parent 的临时 Agent 成功注册,
|
||||||
|
`unix:uid:2000` workload 从 guest-local socket 获得
|
||||||
|
`spiffe://ddupan.top/ci/poc/task/build` X.509-SVID,Pod 正常退出。PoC 资源随后全部清理,
|
||||||
|
中央 SPIRE 恢复声明式配置。
|
||||||
|
|
||||||
|
SPIRE 1.15.3 已支持 `use_pod_uid_for_agent_id`,但当前使用的 hardened chart 0.30.2
|
||||||
|
尚未把它暴露到 values/template。正式部署应先给 chart 补齐该字段并向上游提交,随后
|
||||||
|
采用包含修复的 release;不得把手工修改 Server ConfigMap 作为运行方案。
|
||||||
|
|
||||||
## 上线验收
|
## 上线验收
|
||||||
|
|
||||||
Flux reconciliation 完成后至少确认:
|
Flux reconciliation 完成后至少确认:
|
||||||
@@ -35,8 +61,8 @@ Flux reconciliation 完成后至少确认:
|
|||||||
1. 两个节点重新回到 Ready,`RuntimeClass/kata-clh-runtime-rs` 存在;
|
1. 两个节点重新回到 Ready,`RuntimeClass/kata-clh-runtime-rs` 存在;
|
||||||
2. Kata Pod 内核与 LXC host 内核不同,且 `/dev/kvm` 可用;
|
2. Kata Pod 内核与 LXC host 内核不同,且 `/dev/kvm` 可用;
|
||||||
3. Cloud Hypervisor API 的 `vm.info.config.memory.shared` 为 `true`;
|
3. Cloud Hypervisor API 的 `vm.info.config.memory.shared` 为 `true`;
|
||||||
4. `spire-smoke` ServiceAccount 的 Kata Pod 可获得
|
4. Kata Pod 内层 Agent 的 ID 包含该 Pod UID;只有调度器创建的业务 entry 所匹配的
|
||||||
`spiffe://ddupan.top/sandbox/smoke`,错误 ServiceAccount 无法获得身份;
|
workload UID 能从 guest-local socket 获得预期 SPIFFE ID;
|
||||||
5. block-backed `emptyDir` 上 Docker 使用 `overlay2`,BuildKit 与 kind smoke test
|
5. block-backed `emptyDir` 上 Docker 使用 `overlay2`,BuildKit 与 kind smoke test
|
||||||
通过;kind 的 dockerd bootstrap 需要先在 guest 内执行
|
通过;kind 的 dockerd bootstrap 需要先在 guest 内执行
|
||||||
`mknod /dev/kmsg c 1 11`;
|
`mknod /dev/kmsg c 1 11`;
|
||||||
|
|||||||
@@ -1,123 +0,0 @@
|
|||||||
---
|
|
||||||
apiVersion: sandbox.opensandbox.io/v1alpha1
|
|
||||||
kind: Pool
|
|
||||||
metadata:
|
|
||||||
name: ci-pod
|
|
||||||
namespace: opensandbox
|
|
||||||
spec:
|
|
||||||
template:
|
|
||||||
metadata:
|
|
||||||
labels:
|
|
||||||
ci.ddupan.top/backend: pod
|
|
||||||
spec:
|
|
||||||
containers:
|
|
||||||
- name: sandbox
|
|
||||||
image: sandbox-registry.cn-zhangjiakou.cr.aliyuncs.com/opensandbox/code-interpreter:v1.1.0
|
|
||||||
command: [/opt/opensandbox/task-executor]
|
|
||||||
args: [-listen-addr=0.0.0.0:5758, -log-dir=/tmp]
|
|
||||||
env:
|
|
||||||
- name: SANDBOX_MAIN_CONTAINER
|
|
||||||
value: sandbox
|
|
||||||
- name: EXECD_ENVS
|
|
||||||
value: /opt/opensandbox/.env
|
|
||||||
- name: EXECD
|
|
||||||
value: /opt/opensandbox/execd
|
|
||||||
resources:
|
|
||||||
requests:
|
|
||||||
cpu: 100m
|
|
||||||
memory: 256Mi
|
|
||||||
limits:
|
|
||||||
cpu: "2"
|
|
||||||
memory: 4Gi
|
|
||||||
volumeMounts:
|
|
||||||
- name: opensandbox-bin
|
|
||||||
mountPath: /opt/opensandbox
|
|
||||||
- name: sandbox-storage
|
|
||||||
mountPath: /var/lib/sandbox
|
|
||||||
initContainers:
|
|
||||||
- name: task-executor-installer
|
|
||||||
image: sandbox-registry.cn-zhangjiakou.cr.aliyuncs.com/opensandbox/task-executor:v0.1.0
|
|
||||||
command: [/bin/sh, -c]
|
|
||||||
args: [cp /workspace/server /opt/opensandbox/task-executor && chmod 0755 /opt/opensandbox/task-executor]
|
|
||||||
volumeMounts:
|
|
||||||
- name: opensandbox-bin
|
|
||||||
mountPath: /opt/opensandbox
|
|
||||||
- name: execd-installer
|
|
||||||
image: sandbox-registry.cn-zhangjiakou.cr.aliyuncs.com/opensandbox/execd:v1.0.22
|
|
||||||
command: [/bin/sh, -c]
|
|
||||||
args: [cp ./execd /opt/opensandbox/execd && cp ./bootstrap.sh /opt/opensandbox/bootstrap.sh && chmod 0755 /opt/opensandbox/execd /opt/opensandbox/bootstrap.sh]
|
|
||||||
volumeMounts:
|
|
||||||
- name: opensandbox-bin
|
|
||||||
mountPath: /opt/opensandbox
|
|
||||||
volumes:
|
|
||||||
- name: opensandbox-bin
|
|
||||||
emptyDir: {}
|
|
||||||
- name: sandbox-storage
|
|
||||||
emptyDir: {}
|
|
||||||
capacitySpec:
|
|
||||||
bufferMax: 1
|
|
||||||
bufferMin: 0
|
|
||||||
poolMax: 4
|
|
||||||
poolMin: 0
|
|
||||||
---
|
|
||||||
apiVersion: sandbox.opensandbox.io/v1alpha1
|
|
||||||
kind: Pool
|
|
||||||
metadata:
|
|
||||||
name: ci-vm
|
|
||||||
namespace: opensandbox
|
|
||||||
spec:
|
|
||||||
template:
|
|
||||||
metadata:
|
|
||||||
labels:
|
|
||||||
ci.ddupan.top/backend: vm
|
|
||||||
spec:
|
|
||||||
runtimeClassName: kata-clh-runtime-rs
|
|
||||||
containers:
|
|
||||||
- name: sandbox
|
|
||||||
image: sandbox-registry.cn-zhangjiakou.cr.aliyuncs.com/opensandbox/code-interpreter:v1.1.0
|
|
||||||
command: [/opt/opensandbox/task-executor]
|
|
||||||
args: [-listen-addr=0.0.0.0:5758, -log-dir=/tmp]
|
|
||||||
env:
|
|
||||||
- name: SANDBOX_MAIN_CONTAINER
|
|
||||||
value: sandbox
|
|
||||||
- name: EXECD_ENVS
|
|
||||||
value: /opt/opensandbox/.env
|
|
||||||
- name: EXECD
|
|
||||||
value: /opt/opensandbox/execd
|
|
||||||
resources:
|
|
||||||
requests:
|
|
||||||
cpu: 250m
|
|
||||||
memory: 512Mi
|
|
||||||
limits:
|
|
||||||
cpu: "4"
|
|
||||||
memory: 8Gi
|
|
||||||
volumeMounts:
|
|
||||||
- name: opensandbox-bin
|
|
||||||
mountPath: /opt/opensandbox
|
|
||||||
- name: sandbox-storage
|
|
||||||
mountPath: /var/lib/sandbox
|
|
||||||
initContainers:
|
|
||||||
- name: task-executor-installer
|
|
||||||
image: sandbox-registry.cn-zhangjiakou.cr.aliyuncs.com/opensandbox/task-executor:v0.1.0
|
|
||||||
command: [/bin/sh, -c]
|
|
||||||
args: [cp /workspace/server /opt/opensandbox/task-executor && chmod 0755 /opt/opensandbox/task-executor]
|
|
||||||
volumeMounts:
|
|
||||||
- name: opensandbox-bin
|
|
||||||
mountPath: /opt/opensandbox
|
|
||||||
- name: execd-installer
|
|
||||||
image: sandbox-registry.cn-zhangjiakou.cr.aliyuncs.com/opensandbox/execd:v1.0.22
|
|
||||||
command: [/bin/sh, -c]
|
|
||||||
args: [cp ./execd /opt/opensandbox/execd && cp ./bootstrap.sh /opt/opensandbox/bootstrap.sh && chmod 0755 /opt/opensandbox/execd /opt/opensandbox/bootstrap.sh]
|
|
||||||
volumeMounts:
|
|
||||||
- name: opensandbox-bin
|
|
||||||
mountPath: /opt/opensandbox
|
|
||||||
volumes:
|
|
||||||
- name: opensandbox-bin
|
|
||||||
emptyDir: {}
|
|
||||||
- name: sandbox-storage
|
|
||||||
emptyDir: {}
|
|
||||||
capacitySpec:
|
|
||||||
bufferMax: 1
|
|
||||||
bufferMin: 0
|
|
||||||
poolMax: 2
|
|
||||||
poolMin: 0
|
|
||||||
@@ -1,19 +1,44 @@
|
|||||||
# OpenSandbox
|
# Sandbox OpenSandbox
|
||||||
|
|
||||||
Flux installs the upstream all-in-one OpenSandbox chart pinned to
|
本目录在独立 sandbox k3s 集群部署 OpenSandbox controller、server 与 CRD。Flux 从
|
||||||
`helm/opensandbox/0.2.2` (`8f01e935`). The API is cluster-internal and intentionally runs a
|
上游 commit `8f01e935c2cabba778cf37a152033fae062fa0f4` 构建官方 umbrella chart
|
||||||
single replica until shared server state and HA behaviour have been validated.
|
`0.2.2`;该源码渲染结果已与 release `opensandbox-0.2.2.tgz` 对比一致。不要改为跟随
|
||||||
|
浮动 branch 或 tag。
|
||||||
|
|
||||||
`ci-pod` uses `runc`; `ci-vm` uses the separately managed
|
server 只提供集群内 `opensandbox-server.opensandbox-system.svc:80` ClusterIP,不部署
|
||||||
`kata-clh-runtime-rs` RuntimeClass. Both Pools start at zero and create capacity
|
Gateway、Ingress 或 LoadBalancer。sandbox workload 位于 `opensandbox` namespace,
|
||||||
on demand. They currently use the upstream interpreter image to validate the
|
默认使用 `kata-clh-runtime-rs`;CI Pool、runner 镜像、动态 SPIFFE registration 均由
|
||||||
Lifecycle API and Pool allocation independently of the CI scheduler cutover.
|
runner 项目后续声明,本目录不预制。
|
||||||
|
|
||||||
The dynamic runner worker, runner image, guest-local SPIRE Agent and Docker
|
## API 认证
|
||||||
sidecar are introduced only after this layer is Ready. In particular, do not
|
|
||||||
mount the host SPIFFE CSI socket into `ci-vm`: Unix sockets do not cross the
|
|
||||||
Kata VM boundary.
|
|
||||||
|
|
||||||
Smoke test both backends through the same API by creating sandboxes with
|
`ExternalSecret/opensandbox-api-key` 从 OpenBao
|
||||||
`extensions.poolRef` set to `ci-pod` and `ci-vm`, then confirm their
|
`kv/k8s/opensandbox-api:api_key` 投影同名 Secret。这个路径不归属于某个 Kubernetes
|
||||||
BatchSandboxes, Pods and VMMs disappear after deletion.
|
集群:sandbox server 与 homelab runner 调度器分别通过自己的 Bao 身份读取。server 只通过
|
||||||
|
`secretKeyRef` 读取:
|
||||||
|
|
||||||
|
```yaml
|
||||||
|
- name: OPENSANDBOX_SERVER_API_KEY
|
||||||
|
valueFrom:
|
||||||
|
secretKeyRef:
|
||||||
|
name: opensandbox-api-key
|
||||||
|
key: api-key
|
||||||
|
```
|
||||||
|
|
||||||
|
仓库与 Helm values 均不保存 API key。OpenSandbox 不支持更丰富的原生 workload
|
||||||
|
authentication;runner 后续读取同一 Bao 路径并在请求头中使用 API key。
|
||||||
|
`opensandbox-values` 带 Flux watch label,values 变化会立即触发 Helm reconcile,不依赖
|
||||||
|
30 分钟的 HelmRelease interval。
|
||||||
|
|
||||||
|
## 验收
|
||||||
|
|
||||||
|
合并后等待 `flux-system/opensandbox` 与 `opensandbox-system/opensandbox` Ready,并确认:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
kubectl get crd batchsandboxes.sandbox.opensandbox.io pools.sandbox.opensandbox.io
|
||||||
|
kubectl -n opensandbox-system get deploy,pod,svc
|
||||||
|
kubectl get runtimeclass kata-clh-runtime-rs
|
||||||
|
```
|
||||||
|
|
||||||
|
控制面上线不创建 CI Pool,也不产生 sandbox workload。首个 runner 集成应另行提交 Pool
|
||||||
|
与完整的 Lifecycle API smoke test。
|
||||||
|
|||||||
+7
-6
@@ -1,18 +1,19 @@
|
|||||||
|
---
|
||||||
apiVersion: external-secrets.io/v1
|
apiVersion: external-secrets.io/v1
|
||||||
kind: ExternalSecret
|
kind: ExternalSecret
|
||||||
metadata:
|
metadata:
|
||||||
name: gitea-runner-token
|
name: opensandbox-api-key
|
||||||
namespace: gitea-actions
|
namespace: opensandbox-system
|
||||||
spec:
|
spec:
|
||||||
refreshInterval: 1h
|
refreshInterval: 1h
|
||||||
secretStoreRef:
|
secretStoreRef:
|
||||||
kind: ClusterSecretStore
|
kind: ClusterSecretStore
|
||||||
name: openbao
|
name: openbao
|
||||||
target:
|
target:
|
||||||
|
name: opensandbox-api-key
|
||||||
creationPolicy: Owner
|
creationPolicy: Owner
|
||||||
name: gitea-runner-token
|
|
||||||
data:
|
data:
|
||||||
- secretKey: token
|
- secretKey: api-key
|
||||||
remoteRef:
|
remoteRef:
|
||||||
key: k8s/gitea-runner
|
key: k8s/opensandbox-api
|
||||||
property: token
|
property: api_key
|
||||||
@@ -1,3 +1,4 @@
|
|||||||
|
---
|
||||||
apiVersion: helm.toolkit.fluxcd.io/v2
|
apiVersion: helm.toolkit.fluxcd.io/v2
|
||||||
kind: HelmRelease
|
kind: HelmRelease
|
||||||
metadata:
|
metadata:
|
||||||
@@ -15,6 +16,7 @@ spec:
|
|||||||
driftDetection:
|
driftDetection:
|
||||||
mode: enabled
|
mode: enabled
|
||||||
install:
|
install:
|
||||||
|
crds: CreateReplace
|
||||||
strategy:
|
strategy:
|
||||||
name: RetryOnFailure
|
name: RetryOnFailure
|
||||||
retryInterval: 5m
|
retryInterval: 5m
|
||||||
@@ -23,9 +25,11 @@ spec:
|
|||||||
targetNamespace: opensandbox-system
|
targetNamespace: opensandbox-system
|
||||||
timeout: 15m
|
timeout: 15m
|
||||||
upgrade:
|
upgrade:
|
||||||
|
crds: CreateReplace
|
||||||
strategy:
|
strategy:
|
||||||
name: RetryOnFailure
|
name: RetryOnFailure
|
||||||
retryInterval: 5m
|
retryInterval: 5m
|
||||||
valuesFrom:
|
valuesFrom:
|
||||||
- kind: ConfigMap
|
- kind: ConfigMap
|
||||||
name: opensandbox-values
|
name: opensandbox-values
|
||||||
|
valuesKey: values.yaml
|
||||||
|
|||||||
@@ -1,7 +1,11 @@
|
|||||||
|
---
|
||||||
apiVersion: kustomize.config.k8s.io/v1beta1
|
apiVersion: kustomize.config.k8s.io/v1beta1
|
||||||
kind: Kustomization
|
kind: Kustomization
|
||||||
resources:
|
resources:
|
||||||
- namespaces.yaml
|
- namespace.yaml
|
||||||
|
- external-secret.yaml
|
||||||
- repository.yaml
|
- repository.yaml
|
||||||
|
- template.yaml
|
||||||
- values.yaml
|
- values.yaml
|
||||||
- helmrelease.yaml
|
- helmrelease.yaml
|
||||||
|
- monitor-scrape.yaml
|
||||||
|
|||||||
@@ -0,0 +1,16 @@
|
|||||||
|
---
|
||||||
|
apiVersion: operator.victoriametrics.com/v1beta1
|
||||||
|
kind: VMPodScrape
|
||||||
|
metadata:
|
||||||
|
name: opensandbox-controller
|
||||||
|
namespace: monitoring
|
||||||
|
spec:
|
||||||
|
namespaceSelector:
|
||||||
|
matchNames:
|
||||||
|
- opensandbox-system
|
||||||
|
podMetricsEndpoints:
|
||||||
|
- interval: 30s
|
||||||
|
port: metrics
|
||||||
|
selector:
|
||||||
|
matchLabels:
|
||||||
|
app.kubernetes.io/name: opensandbox
|
||||||
@@ -0,0 +1,14 @@
|
|||||||
|
---
|
||||||
|
apiVersion: v1
|
||||||
|
kind: Namespace
|
||||||
|
metadata:
|
||||||
|
name: opensandbox-system
|
||||||
|
---
|
||||||
|
apiVersion: v1
|
||||||
|
kind: Namespace
|
||||||
|
metadata:
|
||||||
|
name: opensandbox
|
||||||
|
labels:
|
||||||
|
pod-security.kubernetes.io/enforce: privileged
|
||||||
|
pod-security.kubernetes.io/audit: restricted
|
||||||
|
pod-security.kubernetes.io/warn: restricted
|
||||||
@@ -1,10 +0,0 @@
|
|||||||
---
|
|
||||||
apiVersion: v1
|
|
||||||
kind: Namespace
|
|
||||||
metadata:
|
|
||||||
name: opensandbox-system
|
|
||||||
---
|
|
||||||
apiVersion: v1
|
|
||||||
kind: Namespace
|
|
||||||
metadata:
|
|
||||||
name: opensandbox
|
|
||||||
@@ -1,3 +1,4 @@
|
|||||||
|
---
|
||||||
apiVersion: source.toolkit.fluxcd.io/v1
|
apiVersion: source.toolkit.fluxcd.io/v1
|
||||||
kind: GitRepository
|
kind: GitRepository
|
||||||
metadata:
|
metadata:
|
||||||
@@ -6,6 +7,5 @@ metadata:
|
|||||||
spec:
|
spec:
|
||||||
interval: 1h
|
interval: 1h
|
||||||
ref:
|
ref:
|
||||||
tag: helm/opensandbox/0.2.2
|
commit: 8f01e935c2cabba778cf37a152033fae062fa0f4
|
||||||
timeout: 60s
|
url: https://github.com/opensandbox-group/OpenSandbox.git
|
||||||
url: https://github.com/alibaba/OpenSandbox.git
|
|
||||||
|
|||||||
@@ -0,0 +1,17 @@
|
|||||||
|
---
|
||||||
|
apiVersion: v1
|
||||||
|
kind: ConfigMap
|
||||||
|
metadata:
|
||||||
|
name: opensandbox-batchsandbox-template
|
||||||
|
namespace: opensandbox-system
|
||||||
|
data:
|
||||||
|
batchsandbox-template.yaml: |
|
||||||
|
metadata:
|
||||||
|
labels:
|
||||||
|
ddupan.top/workload-class: sandbox
|
||||||
|
spec:
|
||||||
|
replicas: 1
|
||||||
|
template:
|
||||||
|
spec:
|
||||||
|
restartPolicy: Never
|
||||||
|
terminationGracePeriodSeconds: 30
|
||||||
@@ -1,41 +1,52 @@
|
|||||||
|
---
|
||||||
apiVersion: v1
|
apiVersion: v1
|
||||||
kind: ConfigMap
|
kind: ConfigMap
|
||||||
metadata:
|
metadata:
|
||||||
name: opensandbox-values
|
name: opensandbox-values
|
||||||
namespace: opensandbox-system
|
namespace: opensandbox-system
|
||||||
|
labels:
|
||||||
|
reconcile.fluxcd.io/watch: Enabled
|
||||||
data:
|
data:
|
||||||
values.yaml: |
|
values.yaml: |
|
||||||
opensandbox-controller:
|
opensandbox-controller:
|
||||||
controller:
|
controller:
|
||||||
logLevel: info
|
|
||||||
replicaCount: 1
|
|
||||||
metrics:
|
metrics:
|
||||||
enabled: true
|
enabled: true
|
||||||
secure: false
|
|
||||||
port: 8080
|
port: 8080
|
||||||
resources:
|
secure: false
|
||||||
requests:
|
|
||||||
cpu: 25m
|
|
||||||
memory: 64Mi
|
|
||||||
limits:
|
|
||||||
cpu: 500m
|
|
||||||
memory: 256Mi
|
|
||||||
|
|
||||||
opensandbox-server:
|
opensandbox-server:
|
||||||
server:
|
server:
|
||||||
replicaCount: 1
|
replicaCount: 1
|
||||||
|
env:
|
||||||
|
- name: OPENSANDBOX_SERVER_API_KEY
|
||||||
|
valueFrom:
|
||||||
|
secretKeyRef:
|
||||||
|
name: opensandbox-api-key
|
||||||
|
key: api-key
|
||||||
resources:
|
resources:
|
||||||
requests:
|
|
||||||
cpu: 100m
|
|
||||||
memory: 256Mi
|
|
||||||
limits:
|
limits:
|
||||||
cpu: "1"
|
cpu: "1"
|
||||||
memory: 1Gi
|
memory: 1Gi
|
||||||
|
requests:
|
||||||
|
cpu: 100m
|
||||||
|
memory: 256Mi
|
||||||
|
volumeMounts:
|
||||||
|
- name: batchsandbox-template
|
||||||
|
mountPath: /etc/opensandbox/batchsandbox-template.yaml
|
||||||
|
subPath: batchsandbox-template.yaml
|
||||||
|
readOnly: true
|
||||||
|
volumes:
|
||||||
|
- name: batchsandbox-template
|
||||||
|
configMap:
|
||||||
|
name: opensandbox-batchsandbox-template
|
||||||
|
|
||||||
configToml: |
|
configToml: |
|
||||||
[server]
|
[server]
|
||||||
host = "0.0.0.0"
|
host = "0.0.0.0"
|
||||||
port = 80
|
port = 80
|
||||||
api_key = ""
|
api_key = ""
|
||||||
|
max_sandbox_timeout_seconds = 86400
|
||||||
|
|
||||||
[log]
|
[log]
|
||||||
level = "INFO"
|
level = "INFO"
|
||||||
@@ -44,6 +55,10 @@ data:
|
|||||||
type = "kubernetes"
|
type = "kubernetes"
|
||||||
execd_image = "sandbox-registry.cn-zhangjiakou.cr.aliyuncs.com/opensandbox/execd:v1.0.22"
|
execd_image = "sandbox-registry.cn-zhangjiakou.cr.aliyuncs.com/opensandbox/execd:v1.0.22"
|
||||||
|
|
||||||
|
[storage]
|
||||||
|
allowed_host_paths = []
|
||||||
|
volume_default_size = "1Gi"
|
||||||
|
|
||||||
[kubernetes]
|
[kubernetes]
|
||||||
kubeconfig_path = ""
|
kubeconfig_path = ""
|
||||||
namespace = "opensandbox"
|
namespace = "opensandbox"
|
||||||
@@ -52,11 +67,17 @@ data:
|
|||||||
informer_watch_timeout_seconds = 60
|
informer_watch_timeout_seconds = 60
|
||||||
snapshot_create_timeout_seconds = 900
|
snapshot_create_timeout_seconds = 900
|
||||||
workload_provider = "batchsandbox"
|
workload_provider = "batchsandbox"
|
||||||
batchsandbox_template_file = "/etc/opensandbox/example.batchsandbox-template.yaml"
|
image_pull_policy = "IfNotPresent"
|
||||||
|
batchsandbox_template_file = "/etc/opensandbox/batchsandbox-template.yaml"
|
||||||
|
|
||||||
[egress]
|
[egress]
|
||||||
image = "sandbox-registry.cn-zhangjiakou.cr.aliyuncs.com/opensandbox/egress:v1.1.6"
|
image = "sandbox-registry.cn-zhangjiakou.cr.aliyuncs.com/opensandbox/egress:v1.1.6"
|
||||||
mode = "dns+nft"
|
mode = "dns+nft"
|
||||||
|
disable_ipv6 = true
|
||||||
|
|
||||||
|
[secure_runtime]
|
||||||
|
type = "kata"
|
||||||
|
k8s_runtime_class = "kata-clh-runtime-rs"
|
||||||
|
|
||||||
opensandbox-node-agent:
|
opensandbox-node-agent:
|
||||||
enabled: false
|
enabled: false
|
||||||
|
|||||||
@@ -74,6 +74,12 @@ Agent 已启用 Unix workload attestor,并为本机用户 `panxiao81`(UID `1
|
|||||||
export SPIFFE_ENDPOINT_SOCKET=unix:///run/spire/agent-sockets/spire-agent.sock
|
export SPIFFE_ENDPOINT_SOCKET=unix:///run/spire/agent-sockets/spire-agent.sock
|
||||||
```
|
```
|
||||||
|
|
||||||
|
宿主机已安装与 Agent Pod 同版本的 `/usr/local/bin/spire-agent` 1.15.3,供本地进程从
|
||||||
|
Workload API 获取 JWT-SVID。二进制来自 SPIRE 官方 `linux-amd64-musl` release,安装时
|
||||||
|
核对 tarball SHA-256
|
||||||
|
`ca1a4d1155317bdd2afc7f36663828a10410c7c840e54725b90b4064b0a301c7`。升级 chart 时应
|
||||||
|
同步升级这个 CLI 并重新核对官方 checksum,不能长期混用版本。
|
||||||
|
|
||||||
该身份仅按 Unix UID 匹配,不是 SPIRE admin,也不会匹配 `sudo` 后以 root 运行的
|
该身份仅按 Unix UID 匹配,不是 SPIRE admin,也不会匹配 `sudo` 后以 root 运行的
|
||||||
进程。`ClusterStaticEntry.spec.parentID` 绑定当前 `laptop` Kubernetes node UID;若
|
进程。`ClusterStaticEntry.spec.parentID` 绑定当前 `laptop` Kubernetes node UID;若
|
||||||
节点被删除后重建,需从 `spire-server agent list` 取得新 Agent ID 并同步更新该字段。
|
节点被删除后重建,需从 `spire-server agent list` 取得新 Agent ID 并同步更新该字段。
|
||||||
|
|||||||
@@ -274,14 +274,24 @@ CI job/Agent 不应接收长期 `BAO_TOKEN`。标准启动顺序是:
|
|||||||
5. 在同一进程树中以环境变量调用 `tofu`、Ansible 或其他工具;
|
5. 在同一进程树中以环境变量调用 `tofu`、Ansible 或其他工具;
|
||||||
6. cleanup 尝试 `revoke-self`,随后销毁 job/VM/容器。
|
6. cleanup 尝试 `revoke-self`,随后销毁 job/VM/容器。
|
||||||
|
|
||||||
通用 credential-exec 包装器未来应负责步骤 3–6。它必须满足:
|
Gitea Actions 可使用
|
||||||
|
[`panxiao81/ci-actions/spiffe-openbao-login@v1`](https://git.ddupan.top/panxiao81/ci-actions/src/tag/v1/spiffe-openbao-login)
|
||||||
|
完成步骤 3、4 和 cleanup 时的 `revoke-self`。runner 只负责提供 Node.js 20、
|
||||||
|
`spire-agent` 与 Workload API socket;role、audience 及是否登录由 workflow 明确声明,
|
||||||
|
目标侧 Bao policy 仍负责最终授权。该 Action 必须满足:
|
||||||
|
|
||||||
- 不把 JWT-SVID 或 Bao token 写到 stdout/stderr;
|
- 不把 JWT-SVID 或 Bao token 写到 stdout/stderr;
|
||||||
- 不把凭据传入命令行参数,避免出现在进程列表;
|
- 不把凭据传入命令行参数,避免出现在进程列表;
|
||||||
- 子进程退出后清除环境和临时文件;
|
- job 结束后销毁 runner 及其 Actions 临时文件;
|
||||||
- 不尝试把短期 token 上传到 Actions Secret 或 artifact;
|
- 不尝试把短期 token 上传到 Actions Secret 或 artifact;
|
||||||
- role、audience 和目标命令由受审查的 pipeline 配置决定。
|
- role、audience 和目标命令由受审查的 pipeline 配置决定。
|
||||||
|
|
||||||
|
Action 会按 GitHub Actions 协议把短期 token 写入 `GITHUB_ENV` 和 `GITHUB_STATE`,因此
|
||||||
|
只允许用于一次性 Pod/VM runner,不能用于共享或持久 runner。只读取 Nexus public
|
||||||
|
repository 时不需要 Bao 登录,可直接使用
|
||||||
|
[`panxiao81/ci-actions/setup-nexus@v1`](https://git.ddupan.top/panxiao81/ci-actions/src/tag/v1/setup-nexus)
|
||||||
|
配置 Ansible Galaxy、Go module proxy 与 OCI endpoint。
|
||||||
|
|
||||||
Kubernetes 以外的执行环境不能伪造 ServiceAccount。未来应分别使用 host SPIRE
|
Kubernetes 以外的执行环境不能伪造 ServiceAccount。未来应分别使用 host SPIRE
|
||||||
Agent、TPM/DevID、cloud instance identity、GitHub OIDC 等初始证明接入同一信任模型。
|
Agent、TPM/DevID、cloud instance identity、GitHub OIDC 等初始证明接入同一信任模型。
|
||||||
|
|
||||||
|
|||||||
@@ -43,6 +43,10 @@ spire-server:
|
|||||||
externalSecret:
|
externalSecret:
|
||||||
name: spire-external-kubeconfigs
|
name: spire-external-kubeconfigs
|
||||||
key: sandbox-controller
|
key: sandbox-controller
|
||||||
|
sandbox-runner-bundle:
|
||||||
|
externalSecret:
|
||||||
|
name: spire-external-kubeconfigs
|
||||||
|
key: sandbox-controller
|
||||||
nodeAttestor:
|
nodeAttestor:
|
||||||
externalK8sPSAT:
|
externalK8sPSAT:
|
||||||
enabled: true
|
enabled: true
|
||||||
@@ -55,12 +59,19 @@ spire-server:
|
|||||||
kubeConfigName: sandbox
|
kubeConfigName: sandbox
|
||||||
serviceAccountAllowList:
|
serviceAccountAllowList:
|
||||||
- spire-smoke:spire-smoke
|
- spire-smoke:spire-smoke
|
||||||
|
- opensandbox:gitea-ci
|
||||||
usePodUIDForAgentID: true
|
usePodUIDForAgentID: true
|
||||||
externalControllerManagers:
|
externalControllerManagers:
|
||||||
enabled: true
|
enabled: true
|
||||||
clusters:
|
clusters:
|
||||||
sandbox:
|
sandbox:
|
||||||
kubeConfigName: sandbox-controller
|
kubeConfigName: sandbox-controller
|
||||||
|
# Dynamic OpenSandbox runner identities are exact Pod-UID-bound
|
||||||
|
# ClusterStaticEntries created in the sandbox cluster. The chart
|
||||||
|
# defaults this reconciler to false, so enable the central registration
|
||||||
|
# path explicitly.
|
||||||
|
reconcile:
|
||||||
|
clusterStaticEntries: true
|
||||||
bundlePublisher:
|
bundlePublisher:
|
||||||
externalK8sConfigMap:
|
externalK8sConfigMap:
|
||||||
enabled: true
|
enabled: true
|
||||||
@@ -71,6 +82,12 @@ spire-server:
|
|||||||
configMapName: spire-bundle
|
configMapName: spire-bundle
|
||||||
configMapKey: bundle.spiffe
|
configMapKey: bundle.spiffe
|
||||||
format: spiffe
|
format: spiffe
|
||||||
|
sandbox-runner-bundle:
|
||||||
|
kubeConfigName: sandbox-runner-bundle
|
||||||
|
namespace: opensandbox
|
||||||
|
configMapName: spire-bundle-pem
|
||||||
|
configMapKey: bundle.pem
|
||||||
|
format: pem
|
||||||
persistence:
|
persistence:
|
||||||
# PostgreSQL stores registrations, but the disk KeyManager still needs durable
|
# PostgreSQL stores registrations, but the disk KeyManager still needs durable
|
||||||
# storage for the trust-domain signing keys.
|
# storage for the trust-domain signing keys.
|
||||||
|
|||||||
Reference in New Issue
Block a user