diff --git a/platform/dynamic-runner/README.md b/platform/dynamic-runner/README.md index e61c76b..d2ad9e3 100644 --- a/platform/dynamic-runner/README.md +++ b/platform/dynamic-runner/README.md @@ -1,44 +1,84 @@ -# Gitea dynamic runner controller +# Gitea dynamic runner -controller 接收 Gitea `workflow_job` webhook,并将带 `[self-hosted, pod]` 或 -`[self-hosted, vm]` 的 queued job 持久化到 NATS JetStream。Pod workload 在 homelab -集群由原生 Kubernetes worker 创建;只有 VM workload 通过 OpenSandbox Lifecycle API。 +本目录部署单副本 Go controller,默认在同一进程运行 RunnerService scheduler、原生 +Kubernetes Pod worker 和 SPIFFE mTLS facade。首轮生产 canary 只启用 +`scheduler,pod-worker`,OpenSandbox VM worker 保持关闭: ```text -Gitea -> controller -> NATS -> pod worker -> homelab Kubernetes Pod - `-> VM worker -> OpenSandbox -> Kata/microVM +Gitea RunnerService -> scheduler -> JetStream ci.runner.pod + | + v + homelab Kubernetes Pod + | + SPIFFE mTLS RunnerService facade + | + v + Gitea ``` -`dynamic-runner-pod-worker` 使用 durable `pod`,最多并发创建 4 个一次性 privileged -Pod。任务 Pod、ServiceAccount、SPIFFE CSI socket 和生命周期都位于 homelab 的 -`dynamic-runner` namespace,不依赖 sandbox 集群或 OpenSandbox API。 +Pod backend 不经过 OpenSandbox。VM backend 后续启用时才访问 VyOS 暴露的 +OpenSandbox Lifecycle API;本目录不修改 sandbox 平台侧 ESO、Bao Terraform 或 +OpenSandbox chart 所有权边界。 -`10.60.0.13:8080` 是 VyOS 上的内网 HAProxy frontend,后端是 sandbox 节点的 -`30080` NodePort。它只依赖现有跨网段路由,不暴露公网,也不经过 Cloudflare -Tunnel。controller 自身的 `192.168.10.127:8787` 仅供 Gitea webhook 和 sandbox -通过一次性 nonce 领取 registration token。 +## Canary 安全边界 + +- Deployment 为单副本 `Recreate`,避免两个 scheduler 共享同一个 Gitea runner 身份。 +- controller 的 single-flight gate 只允许一个已领取 task 运行;Gitea 接受终态 + `UpdateTask` 后才领取下一条。`POD_CAPACITY=1` 同时限制 worker 创建并发。 +- executor 镜像使用 digest;Pod 以 UID 2000 运行,SPIRE `ClusterStaticEntry` 同时绑定 + 具体 Pod UID 与 `unix:uid:2000`。 +- facade 只有 ClusterIP,executor 通过 + `dynamic-runner-controller.dynamic-runner.svc:8443` 访问;双方使用 Workload API + X509-SVID mTLS,不再保留公网 webhook/token endpoint。 +- controller 的 SPIFFE ID 固定为 + `spiffe://ddupan.top/ns/dynamic-runner/sa/dynamic-runner-controller`。 +- NATS 保留既有最小权限分离:`ci-producer` 仅 publish,`ci-worker` 仅 pull/ACK。 + +当前 facade claim registry 与 single-flight gate 是进程内状态。异常重启时应先检查 +现存 assignment Pod 和 Gitea task,再人工决定是否恢复 scheduler;不能通过扩大副本数 +规避。完成 backend-driven recovery 前保持 `replicas: 1`。 ## Secret 边界 -homelab 的 `ExternalSecret/dynamic-runner` 从既有 `ClusterSecretStore/openbao` 读取: +`ExternalSecret/dynamic-runner` 从既有 `ClusterSecretStore/openbao` 读取: -- `kv/k8s/opensandbox-api:api_key` -> `opensandbox-api-key`; -- `kv/k8s/dynamic-runner:webhook_secret` -> `webhook-secret`; -- `kv/k8s/gitea-runner:token` -> `token`。 +- `kv/k8s/nats`:producer/worker 密码; +- `kv/k8s/opensandbox-api:api_key`:保留给后续 VM worker; +- `kv/k8s/gitea-runner:token`:保留的 runner registration token; +- `kv/k8s/dynamic-runner`:scheduler UUID/token、facade HMAC key 和回滚所需 webhook secret。 -API key 以只读文件挂载,controller 通过 `OPEN-SANDBOX-API-KEY` header 使用。不要把 -key 复制到 Git、Lifecycle 请求、BatchSandbox 或 sandbox 集群 Secret。本目录不创建 -Bao token,也不拥有 sandbox 平台侧的 ESO/Bao Terraform。 +scheduler credential 由官方 Gitea Runner v3.5.0 一次注册生成;它只挂载到 controller, +不会进入 executor。facade capability key 至少 32 字节,controller 为每个 assignment +确定性生成独立 capability。不要打印 Secret、创建静态 Bao token或把 credential 写入 +Git。OpenBao 写入使用本机 SPIFFE JWT 换取的短期 `local-development` token。 -## 调度与身份 +## 首次验收 -Lifecycle 请求只按 `ci-pod` / `ci-vm` 选择 Pool,并携带稳定的 -`spiffe://ddupan.top/ci///`。job ID 只用于日志和诊断 metadata。 -runner labels 始终以 `self-hosted` 开头。 +合并后先观察 Flux 与 controller,不要立即开启 VM: -registration token 通过 controller 内存中的单次 nonce URL 投递。成功领取后 nonce -立即失效;沙箱结束、创建失败或超时时也会撤销。OpenSandbox DELETE 是最终的正常 -回收路径,OpenSandbox timeout 是 controller 异常退出时的兜底。 +```bash +flux reconcile kustomization dynamic-runner --with-source +kubectl -n dynamic-runner wait externalsecret/dynamic-runner \ + --for=condition=Ready --timeout=2m +kubectl -n dynamic-runner rollout status deploy/dynamic-runner-controller --timeout=5m +kubectl -n dynamic-runner logs deploy/dynamic-runner-controller -f +``` -完整部署和验收步骤见 -[`platform/sandbox-ci-runners/README.md`](../sandbox-ci-runners/README.md)。 +确认 scheduler 只领取一条 `[self-hosted,pod]` task,然后验证: + +1. assignment message 进入并离开 durable `pod` consumer; +2. `gitea-task-` Pod 创建,取得实际 Pod UID; +3. 同名 `ClusterStaticEntry` 的 parent ID 包含该 UID,SPIFFE ID 使用 repository/job key; +4. 官方 runner v3.5.0 经 facade claim 精确 task,Gitea 实时收到日志和终态; +5. terminal gate 释放后才领取下一条任务。 + +当前代码尚未完成 ACK 后 backend lifecycle reconciler,因此 canary 成功后可能留下 +Completed Pod/ClusterStaticEntry。首次测试要人工核对并删除已终态资源;在完整自动清理 +通过前不得提高并发或启用 VM worker。 + +## 回滚 + +若 controller 在领取 task 前失败,回滚到前一 commit 的 Python controller/Pod worker +manifests。若已经创建 `gitea-task-*` Pod,先保留现场并核对 Gitea task 状态,不能直接 +重启 scheduler 造成重复执行。scheduler registration 和 capability key 保留在 Bao, +回滚不需要删除或打印它们。 diff --git a/platform/dynamic-runner/clusterspiffeid.yaml b/platform/dynamic-runner/clusterspiffeid.yaml index 57bda53..178eb35 100644 --- a/platform/dynamic-runner/clusterspiffeid.yaml +++ b/platform/dynamic-runner/clusterspiffeid.yaml @@ -1,17 +1,16 @@ apiVersion: spire.spiffe.io/v1alpha1 kind: ClusterSPIFFEID metadata: - name: gitea-dynamic-runner + name: dynamic-runner-controller spec: className: spire-mgmt-spire - spiffeIDTemplate: 'spiffe://{{ .TrustDomain }}/ci/{{ index .PodMeta.Annotations "ci.ddupan.top/spiffe-path" }}' + spiffeIDTemplate: spiffe://ddupan.top/ns/dynamic-runner/sa/dynamic-runner-controller namespaceSelector: matchLabels: kubernetes.io/metadata.name: dynamic-runner podSelector: matchLabels: - app.kubernetes.io/name: gitea-dynamic-runner - ci.ddupan.top/identity-bound: "true" + app.kubernetes.io/name: dynamic-runner-controller workloadSelectorTemplates: - k8s:ns:dynamic-runner - - k8s:sa:gitea-dynamic-runner + - k8s:sa:dynamic-runner-controller diff --git a/platform/dynamic-runner/deployment.yaml b/platform/dynamic-runner/deployment.yaml index f47a256..aa1225a 100644 --- a/platform/dynamic-runner/deployment.yaml +++ b/platform/dynamic-runner/deployment.yaml @@ -5,6 +5,8 @@ metadata: namespace: dynamic-runner spec: replicas: 1 + strategy: + type: Recreate selector: matchLabels: app.kubernetes.io/name: dynamic-runner-controller @@ -14,17 +16,10 @@ spec: app.kubernetes.io/name: dynamic-runner-controller spec: serviceAccountName: dynamic-runner-controller - automountServiceAccountToken: false initContainers: - name: fetch-internal-ca image: curlimages/curl:8.16.0@sha256:463eaf6072688fe96ac64fa623fe73e1dbe25d8ad6c34404a669ad3ce1f104b6 - args: - - --fail - - --silent - - --show-error - - --output - - /trust/ca.pem - - https://bao.ad.ddupan.top:8200/v1/pki/ca/pem + args: [--fail, --silent, --show-error, --output, /trust/ca.pem, https://bao.ad.ddupan.top:8200/v1/pki/ca/pem] securityContext: allowPrivilegeEscalation: false capabilities: @@ -40,9 +35,18 @@ spec: mountPath: /trust containers: - name: controller - image: zot.ad.ddupan.top/panxiao81/gitea-dynamic-runner-controller@sha256:9f23e84934643bbe859dc0f8da418421dc8ab7b0958fdf1fdc91d562c30e246f + image: zot.ad.ddupan.top/panxiao81/gitea-dynamic-runner-controller@sha256:83ac5dc53116d5f60692709b5e5f585ff3e3c52487083db02ec66af1e2fb8eba imagePullPolicy: IfNotPresent + args: [controller] env: + - name: COMPONENTS + value: scheduler,pod-worker + - name: GITEA_INSTANCE_URL + value: https://git.ddupan.top + - name: GITEA_RUNNER_UUID_FILE + value: /run/dynamic-runner-secrets/scheduler-uuid + - name: GITEA_RUNNER_TOKEN_FILE + value: /run/dynamic-runner-secrets/scheduler-token - name: NATS_URL value: tls://nats.ad.ddupan.top:4222 - name: NATS_CA_FILE @@ -51,41 +55,55 @@ spec: value: /run/dynamic-runner-secrets/nats-password - name: NATS_WORKER_PASSWORD_FILE value: /run/dynamic-runner-secrets/nats-worker-password - - name: POD_CONSUMER_ENABLED - value: "false" - - name: VM_CONSUMER_ENABLED - value: "false" - - name: WEBHOOK_SECRET_FILE - value: /run/dynamic-runner-secrets/webhook-secret - - name: REGISTRATION_TOKEN_FILE - value: /run/dynamic-runner-secrets/token - - name: OPENSANDBOX_API - value: http://10.60.0.13:8080 - - name: OPENSANDBOX_API_KEY_FILE - value: /run/dynamic-runner-secrets/opensandbox-api-key - - name: RUNNER_TOKEN_BASE_URL - value: http://192.168.10.127:8787/token + - name: RUNNER_FACADE_CAPABILITY_KEY_FILE + value: /run/dynamic-runner-secrets/facade-capability-key + - name: RUNNER_FACADE_LISTEN + value: :8443 + - name: RUNNER_FACADE_URL + value: https://dynamic-runner-controller.dynamic-runner.svc:8443 + - name: RUNNER_FACADE_SPIFFE_ID + value: spiffe://ddupan.top/ns/dynamic-runner/sa/dynamic-runner-controller + - name: SPIFFE_ENDPOINT_SOCKET + value: unix:///run/spire/agent-sockets/spire-agent.sock + - name: POD_NAMESPACE + valueFrom: + fieldRef: + fieldPath: metadata.namespace + - name: POD_EXECUTOR_IMAGE + value: zot.ad.ddupan.top/panxiao81/gitea-dynamic-runner-runner@sha256:59bdc035b9372a6ff16337e11a5c6c64db6b7ab3664ca07b85f3a060ad238a4a + - name: POD_SERVICE_ACCOUNT + value: gitea-dynamic-runner + - name: POD_EXECUTOR_UID + value: "2000" + - name: POD_CAPACITY + value: "1" + - name: SPIRE_CLUSTER + value: homelab + - name: SPIRE_CLASS + value: spire-mgmt-spire + - name: SSL_CERT_FILE + value: /run/trust/ca.pem ports: - - name: http - containerPort: 8787 + - name: facade + containerPort: 8443 readinessProbe: - httpGet: - path: /healthz - port: http + httpGet: null + tcpSocket: + port: facade periodSeconds: 5 livenessProbe: - httpGet: - path: /healthz - port: http - initialDelaySeconds: 10 + httpGet: null + tcpSocket: + port: facade + initialDelaySeconds: 15 periodSeconds: 10 resources: requests: - cpu: 25m - memory: 32Mi + cpu: 50m + memory: 64Mi limits: - cpu: 250m - memory: 128Mi + cpu: 500m + memory: 256Mi securityContext: allowPrivilegeEscalation: false capabilities: @@ -103,6 +121,9 @@ spec: - name: trust mountPath: /run/trust readOnly: true + - name: spire-agent-socket + mountPath: /run/spire/agent-sockets + readOnly: true securityContext: fsGroup: 65532 fsGroupChangePolicy: OnRootMismatch @@ -116,3 +137,7 @@ spec: - name: trust emptyDir: sizeLimit: 1Mi + - name: spire-agent-socket + csi: + driver: csi.spiffe.io + readOnly: true diff --git a/platform/dynamic-runner/external-secret.yaml b/platform/dynamic-runner/external-secret.yaml index 6d602ee..1126780 100644 --- a/platform/dynamic-runner/external-secret.yaml +++ b/platform/dynamic-runner/external-secret.yaml @@ -32,3 +32,15 @@ spec: remoteRef: key: k8s/gitea-runner property: token + - secretKey: scheduler-uuid + remoteRef: + key: k8s/dynamic-runner + property: scheduler_uuid + - secretKey: scheduler-token + remoteRef: + key: k8s/dynamic-runner + property: scheduler_token + - secretKey: facade-capability-key + remoteRef: + key: k8s/dynamic-runner + property: facade_capability_key diff --git a/platform/dynamic-runner/kustomization.yaml b/platform/dynamic-runner/kustomization.yaml index 6bf792e..00d940d 100644 --- a/platform/dynamic-runner/kustomization.yaml +++ b/platform/dynamic-runner/kustomization.yaml @@ -6,5 +6,4 @@ resources: - rbac.yaml - clusterspiffeid.yaml - deployment.yaml - - pod-worker-deployment.yaml - service.yaml diff --git a/platform/dynamic-runner/pod-worker-deployment.yaml b/platform/dynamic-runner/pod-worker-deployment.yaml deleted file mode 100644 index 1b012e1..0000000 --- a/platform/dynamic-runner/pod-worker-deployment.yaml +++ /dev/null @@ -1,100 +0,0 @@ -apiVersion: apps/v1 -kind: Deployment -metadata: - name: dynamic-runner-pod-worker - namespace: dynamic-runner -spec: - replicas: 1 - selector: - matchLabels: - app.kubernetes.io/name: dynamic-runner-pod-worker - template: - metadata: - labels: - app.kubernetes.io/name: dynamic-runner-pod-worker - spec: - serviceAccountName: dynamic-runner-pod-worker - initContainers: - - name: fetch-internal-ca - image: curlimages/curl:8.16.0@sha256:463eaf6072688fe96ac64fa623fe73e1dbe25d8ad6c34404a669ad3ce1f104b6 - args: - - --fail - - --silent - - --show-error - - --output - - /trust/ca.pem - - https://bao.ad.ddupan.top:8200/v1/pki/ca/pem - securityContext: - allowPrivilegeEscalation: false - capabilities: - drop: [ALL] - readOnlyRootFilesystem: true - runAsNonRoot: true - runAsUser: 101 - runAsGroup: 102 - seccompProfile: - type: RuntimeDefault - volumeMounts: - - name: trust - mountPath: /trust - containers: - - name: pod-worker - image: zot.ad.ddupan.top/panxiao81/gitea-dynamic-runner-controller@sha256:9f23e84934643bbe859dc0f8da418421dc8ab7b0958fdf1fdc91d562c30e246f - imagePullPolicy: IfNotPresent - command: [/venv/bin/gitea-dynamic-runner-pod-worker] - env: - - name: NATS_URL - value: tls://nats.ad.ddupan.top:4222 - - name: NATS_CA_FILE - value: /run/trust/ca.pem - - name: NATS_PASSWORD_FILE - value: /run/dynamic-runner-secrets/nats-worker-password - - name: RUNNER_NAMESPACE - valueFrom: - fieldRef: - fieldPath: metadata.namespace - - name: RUNNER_IMAGE - value: zot.ad.ddupan.top/panxiao81/gitea-dynamic-runner-runner@sha256:4c61f6315453d68a827ee9542f1345ed86576eaaf62325eb803c8fe3f06ddf2a - - name: RUNNER_SERVICE_ACCOUNT - value: gitea-dynamic-runner - - name: RUNNER_TOKEN_SECRET - value: dynamic-runner - - name: RUNNER_CAPACITY - value: "4" - resources: - requests: - cpu: 25m - memory: 32Mi - limits: - cpu: 250m - memory: 128Mi - securityContext: - allowPrivilegeEscalation: false - capabilities: - drop: [ALL] - readOnlyRootFilesystem: true - runAsNonRoot: true - runAsUser: 65532 - runAsGroup: 65532 - seccompProfile: - type: RuntimeDefault - volumeMounts: - - name: secret - mountPath: /run/dynamic-runner-secrets - readOnly: true - - name: trust - mountPath: /run/trust - readOnly: true - securityContext: - fsGroup: 65532 - fsGroupChangePolicy: OnRootMismatch - seccompProfile: - type: RuntimeDefault - volumes: - - name: secret - secret: - secretName: dynamic-runner - defaultMode: 0400 - - name: trust - emptyDir: - sizeLimit: 1Mi diff --git a/platform/dynamic-runner/rbac.yaml b/platform/dynamic-runner/rbac.yaml index 7d96083..5dd3a7b 100644 --- a/platform/dynamic-runner/rbac.yaml +++ b/platform/dynamic-runner/rbac.yaml @@ -6,12 +6,6 @@ metadata: --- apiVersion: v1 kind: ServiceAccount -metadata: - name: dynamic-runner-pod-worker - namespace: dynamic-runner ---- -apiVersion: v1 -kind: ServiceAccount metadata: name: gitea-dynamic-runner namespace: dynamic-runner @@ -19,23 +13,45 @@ metadata: apiVersion: rbac.authorization.k8s.io/v1 kind: Role metadata: - name: dynamic-runner-pod-worker + name: dynamic-runner-controller namespace: dynamic-runner rules: - apiGroups: [""] resources: [pods] - verbs: [create, get, patch, delete] + verbs: [create, get, list, watch, patch, delete] --- apiVersion: rbac.authorization.k8s.io/v1 kind: RoleBinding metadata: - name: dynamic-runner-pod-worker + name: dynamic-runner-controller namespace: dynamic-runner roleRef: apiGroup: rbac.authorization.k8s.io kind: Role - name: dynamic-runner-pod-worker + name: dynamic-runner-controller subjects: - kind: ServiceAccount - name: dynamic-runner-pod-worker + name: dynamic-runner-controller + namespace: dynamic-runner +--- +apiVersion: rbac.authorization.k8s.io/v1 +kind: ClusterRole +metadata: + name: dynamic-runner-spiffe-entries +rules: + - apiGroups: [spire.spiffe.io] + resources: [clusterstaticentries] + verbs: [create, get, delete] +--- +apiVersion: rbac.authorization.k8s.io/v1 +kind: ClusterRoleBinding +metadata: + name: dynamic-runner-spiffe-entries +roleRef: + apiGroup: rbac.authorization.k8s.io + kind: ClusterRole + name: dynamic-runner-spiffe-entries +subjects: + - kind: ServiceAccount + name: dynamic-runner-controller namespace: dynamic-runner diff --git a/platform/dynamic-runner/service.yaml b/platform/dynamic-runner/service.yaml index 33df06c..bcef3db 100644 --- a/platform/dynamic-runner/service.yaml +++ b/platform/dynamic-runner/service.yaml @@ -4,11 +4,10 @@ metadata: name: dynamic-runner-controller namespace: dynamic-runner spec: - type: LoadBalancer - loadBalancerIP: 192.168.10.127 + type: ClusterIP selector: app.kubernetes.io/name: dynamic-runner-controller ports: - - name: http - port: 8787 - targetPort: http + - name: facade + port: 8443 + targetPort: facade