部署 Go 动态 Runner Pod canary

2026-09-20 21:00:28 +00:00
parent ab985c7979
commit 68191e0ff4
8 changed files with 177 additions and 187 deletions
@@ -1,44 +1,84 @@
-# Gitea dynamic runner controller
+# Gitea dynamic runner

-controller 接收 Gitea `workflow_job` webhook，并将带 `[self-hosted, pod]` 或
-`[self-hosted, vm]` 的 queued job 持久化到 NATS JetStream。Pod workload 在 homelab
-集群由原生 Kubernetes worker 创建；只有 VM workload 通过 OpenSandbox Lifecycle API。
+本目录部署单副本 Go controller，默认在同一进程运行 RunnerService scheduler、原生
+Kubernetes Pod worker 和 SPIFFE mTLS facade。首轮生产 canary 只启用
+`scheduler,pod-worker`，OpenSandbox VM worker 保持关闭：

 ```text
-Gitea -> controller -> NATS -> pod worker -> homelab Kubernetes Pod
-                           `-> VM worker  -> OpenSandbox -> Kata/microVM
+Gitea RunnerService -> scheduler -> JetStream ci.runner.pod
+                                      |
+                                      v
+                              homelab Kubernetes Pod
+                                      |
+                         SPIFFE mTLS RunnerService facade
+                                      |
+                                      v
+                                     Gitea
 ```

-`dynamic-runner-pod-worker` 使用 durable `pod`，最多并发创建 4 个一次性 privileged
-Pod。任务 Pod、ServiceAccount、SPIFFE CSI socket 和生命周期都位于 homelab 的
-`dynamic-runner` namespace，不依赖 sandbox 集群或 OpenSandbox API。
+Pod backend 不经过 OpenSandbox。VM backend 后续启用时才访问 VyOS 暴露的
+OpenSandbox Lifecycle API；本目录不修改 sandbox 平台侧 ESO、Bao Terraform 或
+OpenSandbox chart 所有权边界。

-`10.60.0.13:8080` 是 VyOS 上的内网 HAProxy frontend，后端是 sandbox 节点的
-`30080` NodePort。它只依赖现有跨网段路由，不暴露公网，也不经过 Cloudflare
-Tunnel。controller 自身的 `192.168.10.127:8787` 仅供 Gitea webhook 和 sandbox
-通过一次性 nonce 领取 registration token。
+## Canary 安全边界
+
+- Deployment 为单副本 `Recreate`，避免两个 scheduler 共享同一个 Gitea runner 身份。
+- controller 的 single-flight gate 只允许一个已领取 task 运行；Gitea 接受终态
+  `UpdateTask` 后才领取下一条。`POD_CAPACITY=1` 同时限制 worker 创建并发。
+- executor 镜像使用 digest；Pod 以 UID 2000 运行，SPIRE `ClusterStaticEntry` 同时绑定
+  具体 Pod UID 与 `unix:uid:2000`。
+- facade 只有 ClusterIP，executor 通过
+  `dynamic-runner-controller.dynamic-runner.svc:8443` 访问；双方使用 Workload API
+  X509-SVID mTLS，不再保留公网 webhook/token endpoint。
+- controller 的 SPIFFE ID 固定为
+  `spiffe://ddupan.top/ns/dynamic-runner/sa/dynamic-runner-controller`。
+- NATS 保留既有最小权限分离：`ci-producer` 仅 publish，`ci-worker` 仅 pull/ACK。
+
+当前 facade claim registry 与 single-flight gate 是进程内状态。异常重启时应先检查
+现存 assignment Pod 和 Gitea task，再人工决定是否恢复 scheduler；不能通过扩大副本数
+规避。完成 backend-driven recovery 前保持 `replicas: 1`。

 ## Secret 边界

-homelab 的 `ExternalSecret/dynamic-runner` 从既有 `ClusterSecretStore/openbao` 读取：
+`ExternalSecret/dynamic-runner` 从既有 `ClusterSecretStore/openbao` 读取：

- `kv/k8s/opensandbox-api:api_key` -> `opensandbox-api-key`；
- `kv/k8s/dynamic-runner:webhook_secret` -> `webhook-secret`；
- `kv/k8s/gitea-runner:token` -> `token`。
+- `kv/k8s/nats`：producer/worker 密码；
+- `kv/k8s/opensandbox-api:api_key`：保留给后续 VM worker；
+- `kv/k8s/gitea-runner:token`：保留的 runner registration token；
+- `kv/k8s/dynamic-runner`：scheduler UUID/token、facade HMAC key 和回滚所需 webhook secret。

-API key 以只读文件挂载，controller 通过 `OPEN-SANDBOX-API-KEY` header 使用。不要把
-key 复制到 Git、Lifecycle 请求、BatchSandbox 或 sandbox 集群 Secret。本目录不创建
-Bao token，也不拥有 sandbox 平台侧的 ESO/Bao Terraform。
+scheduler credential 由官方 Gitea Runner v3.5.0 一次注册生成；它只挂载到 controller，
+不会进入 executor。facade capability key 至少 32 字节，controller 为每个 assignment
+确定性生成独立 capability。不要打印 Secret、创建静态 Bao token或把 credential 写入
+Git。OpenBao 写入使用本机 SPIFFE JWT 换取的短期 `local-development` token。

-## 调度与身份
+## 首次验收

-Lifecycle 请求只按 `ci-pod` / `ci-vm` 选择 Pool，并携带稳定的
-`spiffe://ddupan.top/ci/<owner>/<repository>/<task>`。job ID 只用于日志和诊断 metadata。
-runner labels 始终以 `self-hosted` 开头。
+合并后先观察 Flux 与 controller，不要立即开启 VM：

-registration token 通过 controller 内存中的单次 nonce URL 投递。成功领取后 nonce
-立即失效；沙箱结束、创建失败或超时时也会撤销。OpenSandbox DELETE 是最终的正常
-回收路径，OpenSandbox timeout 是 controller 异常退出时的兜底。
+```bash
+flux reconcile kustomization dynamic-runner --with-source
+kubectl -n dynamic-runner wait externalsecret/dynamic-runner \
+  --for=condition=Ready --timeout=2m
+kubectl -n dynamic-runner rollout status deploy/dynamic-runner-controller --timeout=5m
+kubectl -n dynamic-runner logs deploy/dynamic-runner-controller -f
+```

-完整部署和验收步骤见
-[`platform/sandbox-ci-runners/README.md`](../sandbox-ci-runners/README.md)。
+确认 scheduler 只领取一条 `[self-hosted,pod]` task，然后验证：
+
+1. assignment message 进入并离开 durable `pod` consumer；
+2. `gitea-task-<task-id>` Pod 创建，取得实际 Pod UID；
+3. 同名 `ClusterStaticEntry` 的 parent ID 包含该 UID，SPIFFE ID 使用 repository/job key；
+4. 官方 runner v3.5.0 经 facade claim 精确 task，Gitea 实时收到日志和终态；
+5. terminal gate 释放后才领取下一条任务。
+
+当前代码尚未完成 ACK 后 backend lifecycle reconciler，因此 canary 成功后可能留下
+Completed Pod/ClusterStaticEntry。首次测试要人工核对并删除已终态资源；在完整自动清理
+通过前不得提高并发或启用 VM worker。
+
+## 回滚
+
+若 controller 在领取 task 前失败，回滚到前一 commit 的 Python controller/Pod worker
+manifests。若已经创建 `gitea-task-*` Pod，先保留现场并核对 Gitea task 状态，不能直接
+重启 scheduler 造成重复执行。scheduler registration 和 capability key 保留在 Bao，
+回滚不需要删除或打印它们。
@@ -1,17 +1,16 @@
 apiVersion: spire.spiffe.io/v1alpha1
 kind: ClusterSPIFFEID
 metadata:
-  name: gitea-dynamic-runner
+  name: dynamic-runner-controller
 spec:
  className: spire-mgmt-spire
-  spiffeIDTemplate: 'spiffe://{{ .TrustDomain }}/ci/{{ index .PodMeta.Annotations "ci.ddupan.top/spiffe-path" }}'
+  spiffeIDTemplate: spiffe://ddupan.top/ns/dynamic-runner/sa/dynamic-runner-controller
  namespaceSelector:
    matchLabels:
      kubernetes.io/metadata.name: dynamic-runner
  podSelector:
    matchLabels:
-      app.kubernetes.io/name: gitea-dynamic-runner
-      ci.ddupan.top/identity-bound: "true"
+      app.kubernetes.io/name: dynamic-runner-controller
  workloadSelectorTemplates:
    - k8s:ns:dynamic-runner
-    - k8s:sa:gitea-dynamic-runner
+    - k8s:sa:dynamic-runner-controller
@@ -5,6 +5,8 @@ metadata:
  namespace: dynamic-runner
 spec:
  replicas: 1
+  strategy:
+    type: Recreate
  selector:
    matchLabels:
      app.kubernetes.io/name: dynamic-runner-controller
@@ -14,17 +16,10 @@ spec:
        app.kubernetes.io/name: dynamic-runner-controller
    spec:
      serviceAccountName: dynamic-runner-controller
-      automountServiceAccountToken: false
      initContainers:
        - name: fetch-internal-ca
          image: curlimages/curl:8.16.0@sha256:463eaf6072688fe96ac64fa623fe73e1dbe25d8ad6c34404a669ad3ce1f104b6
-          args:
-            - --fail
-            - --silent
-            - --show-error
-            - --output
-            - /trust/ca.pem
-            - https://bao.ad.ddupan.top:8200/v1/pki/ca/pem
+          args: [--fail, --silent, --show-error, --output, /trust/ca.pem, https://bao.ad.ddupan.top:8200/v1/pki/ca/pem]
          securityContext:
            allowPrivilegeEscalation: false
            capabilities:
@@ -40,9 +35,18 @@ spec:
              mountPath: /trust
      containers:
        - name: controller
-          image: zot.ad.ddupan.top/panxiao81/gitea-dynamic-runner-controller@sha256:9f23e84934643bbe859dc0f8da418421dc8ab7b0958fdf1fdc91d562c30e246f
+          image: zot.ad.ddupan.top/panxiao81/gitea-dynamic-runner-controller@sha256:83ac5dc53116d5f60692709b5e5f585ff3e3c52487083db02ec66af1e2fb8eba
          imagePullPolicy: IfNotPresent
+          args: [controller]
          env:
+            - name: COMPONENTS
+              value: scheduler,pod-worker
+            - name: GITEA_INSTANCE_URL
+              value: https://git.ddupan.top
+            - name: GITEA_RUNNER_UUID_FILE
+              value: /run/dynamic-runner-secrets/scheduler-uuid
+            - name: GITEA_RUNNER_TOKEN_FILE
+              value: /run/dynamic-runner-secrets/scheduler-token
            - name: NATS_URL
              value: tls://nats.ad.ddupan.top:4222
            - name: NATS_CA_FILE
@@ -51,41 +55,55 @@ spec:
              value: /run/dynamic-runner-secrets/nats-password
            - name: NATS_WORKER_PASSWORD_FILE
              value: /run/dynamic-runner-secrets/nats-worker-password
-            - name: POD_CONSUMER_ENABLED
-              value: "false"
-            - name: VM_CONSUMER_ENABLED
-              value: "false"
-            - name: WEBHOOK_SECRET_FILE
-              value: /run/dynamic-runner-secrets/webhook-secret
-            - name: REGISTRATION_TOKEN_FILE
-              value: /run/dynamic-runner-secrets/token
-            - name: OPENSANDBOX_API
-              value: http://10.60.0.13:8080
-            - name: OPENSANDBOX_API_KEY_FILE
-              value: /run/dynamic-runner-secrets/opensandbox-api-key
-            - name: RUNNER_TOKEN_BASE_URL
-              value: http://192.168.10.127:8787/token
+            - name: RUNNER_FACADE_CAPABILITY_KEY_FILE
+              value: /run/dynamic-runner-secrets/facade-capability-key
+            - name: RUNNER_FACADE_LISTEN
+              value: :8443
+            - name: RUNNER_FACADE_URL
+              value: https://dynamic-runner-controller.dynamic-runner.svc:8443
+            - name: RUNNER_FACADE_SPIFFE_ID
+              value: spiffe://ddupan.top/ns/dynamic-runner/sa/dynamic-runner-controller
+            - name: SPIFFE_ENDPOINT_SOCKET
+              value: unix:///run/spire/agent-sockets/spire-agent.sock
+            - name: POD_NAMESPACE
+              valueFrom:
+                fieldRef:
+                  fieldPath: metadata.namespace
+            - name: POD_EXECUTOR_IMAGE
+              value: zot.ad.ddupan.top/panxiao81/gitea-dynamic-runner-runner@sha256:59bdc035b9372a6ff16337e11a5c6c64db6b7ab3664ca07b85f3a060ad238a4a
+            - name: POD_SERVICE_ACCOUNT
+              value: gitea-dynamic-runner
+            - name: POD_EXECUTOR_UID
+              value: "2000"
+            - name: POD_CAPACITY
+              value: "1"
+            - name: SPIRE_CLUSTER
+              value: homelab
+            - name: SPIRE_CLASS
+              value: spire-mgmt-spire
+            - name: SSL_CERT_FILE
+              value: /run/trust/ca.pem
          ports:
-            - name: http
-              containerPort: 8787
+            - name: facade
+              containerPort: 8443
          readinessProbe:
-            httpGet:
-              path: /healthz
-              port: http
+            httpGet: null
+            tcpSocket:
+              port: facade
            periodSeconds: 5
          livenessProbe:
-            httpGet:
-              path: /healthz
-              port: http
-            initialDelaySeconds: 10
+            httpGet: null
+            tcpSocket:
+              port: facade
+            initialDelaySeconds: 15
            periodSeconds: 10
          resources:
            requests:
-              cpu: 25m
-              memory: 32Mi
+              cpu: 50m
+              memory: 64Mi
            limits:
-              cpu: 250m
-              memory: 128Mi
+              cpu: 500m
+              memory: 256Mi
          securityContext:
            allowPrivilegeEscalation: false
            capabilities:
@@ -103,6 +121,9 @@ spec:
            - name: trust
              mountPath: /run/trust
              readOnly: true
+            - name: spire-agent-socket
+              mountPath: /run/spire/agent-sockets
+              readOnly: true
      securityContext:
        fsGroup: 65532
        fsGroupChangePolicy: OnRootMismatch
@@ -116,3 +137,7 @@ spec:
        - name: trust
          emptyDir:
            sizeLimit: 1Mi
+        - name: spire-agent-socket
+          csi:
+            driver: csi.spiffe.io
+            readOnly: true
@@ -32,3 +32,15 @@ spec:
      remoteRef:
        key: k8s/gitea-runner
        property: token
+    - secretKey: scheduler-uuid
+      remoteRef:
+        key: k8s/dynamic-runner
+        property: scheduler_uuid
+    - secretKey: scheduler-token
+      remoteRef:
+        key: k8s/dynamic-runner
+        property: scheduler_token
+    - secretKey: facade-capability-key
+      remoteRef:
+        key: k8s/dynamic-runner
+        property: facade_capability_key
@@ -6,5 +6,4 @@ resources:
  - rbac.yaml
  - clusterspiffeid.yaml
  - deployment.yaml
-  - pod-worker-deployment.yaml
  - service.yaml
@@ -1,100 +0,0 @@
-apiVersion: apps/v1
-kind: Deployment
-metadata:
-  name: dynamic-runner-pod-worker
-  namespace: dynamic-runner
-spec:
-  replicas: 1
-  selector:
-    matchLabels:
-      app.kubernetes.io/name: dynamic-runner-pod-worker
-  template:
-    metadata:
-      labels:
-        app.kubernetes.io/name: dynamic-runner-pod-worker
-    spec:
-      serviceAccountName: dynamic-runner-pod-worker
-      initContainers:
-        - name: fetch-internal-ca
-          image: curlimages/curl:8.16.0@sha256:463eaf6072688fe96ac64fa623fe73e1dbe25d8ad6c34404a669ad3ce1f104b6
-          args:
-            - --fail
-            - --silent
-            - --show-error
-            - --output
-            - /trust/ca.pem
-            - https://bao.ad.ddupan.top:8200/v1/pki/ca/pem
-          securityContext:
-            allowPrivilegeEscalation: false
-            capabilities:
-              drop: [ALL]
-            readOnlyRootFilesystem: true
-            runAsNonRoot: true
-            runAsUser: 101
-            runAsGroup: 102
-            seccompProfile:
-              type: RuntimeDefault
-          volumeMounts:
-            - name: trust
-              mountPath: /trust
-      containers:
-        - name: pod-worker
-          image: zot.ad.ddupan.top/panxiao81/gitea-dynamic-runner-controller@sha256:9f23e84934643bbe859dc0f8da418421dc8ab7b0958fdf1fdc91d562c30e246f
-          imagePullPolicy: IfNotPresent
-          command: [/venv/bin/gitea-dynamic-runner-pod-worker]
-          env:
-            - name: NATS_URL
-              value: tls://nats.ad.ddupan.top:4222
-            - name: NATS_CA_FILE
-              value: /run/trust/ca.pem
-            - name: NATS_PASSWORD_FILE
-              value: /run/dynamic-runner-secrets/nats-worker-password
-            - name: RUNNER_NAMESPACE
-              valueFrom:
-                fieldRef:
-                  fieldPath: metadata.namespace
-            - name: RUNNER_IMAGE
-              value: zot.ad.ddupan.top/panxiao81/gitea-dynamic-runner-runner@sha256:4c61f6315453d68a827ee9542f1345ed86576eaaf62325eb803c8fe3f06ddf2a
-            - name: RUNNER_SERVICE_ACCOUNT
-              value: gitea-dynamic-runner
-            - name: RUNNER_TOKEN_SECRET
-              value: dynamic-runner
-            - name: RUNNER_CAPACITY
-              value: "4"
-          resources:
-            requests:
-              cpu: 25m
-              memory: 32Mi
-            limits:
-              cpu: 250m
-              memory: 128Mi
-          securityContext:
-            allowPrivilegeEscalation: false
-            capabilities:
-              drop: [ALL]
-            readOnlyRootFilesystem: true
-            runAsNonRoot: true
-            runAsUser: 65532
-            runAsGroup: 65532
-            seccompProfile:
-              type: RuntimeDefault
-          volumeMounts:
-            - name: secret
-              mountPath: /run/dynamic-runner-secrets
-              readOnly: true
-            - name: trust
-              mountPath: /run/trust
-              readOnly: true
-      securityContext:
-        fsGroup: 65532
-        fsGroupChangePolicy: OnRootMismatch
-        seccompProfile:
-          type: RuntimeDefault
-      volumes:
-        - name: secret
-          secret:
-            secretName: dynamic-runner
-            defaultMode: 0400
-        - name: trust
-          emptyDir:
-            sizeLimit: 1Mi
@@ -6,12 +6,6 @@ metadata:
 ---
 apiVersion: v1
 kind: ServiceAccount
-metadata:
-  name: dynamic-runner-pod-worker
-  namespace: dynamic-runner
---
-apiVersion: v1
-kind: ServiceAccount
 metadata:
  name: gitea-dynamic-runner
  namespace: dynamic-runner
@@ -19,23 +13,45 @@ metadata:
 apiVersion: rbac.authorization.k8s.io/v1
 kind: Role
 metadata:
-  name: dynamic-runner-pod-worker
+  name: dynamic-runner-controller
  namespace: dynamic-runner
 rules:
  - apiGroups: [""]
    resources: [pods]
-    verbs: [create, get, patch, delete]
+    verbs: [create, get, list, watch, patch, delete]
 ---
 apiVersion: rbac.authorization.k8s.io/v1
 kind: RoleBinding
 metadata:
-  name: dynamic-runner-pod-worker
+  name: dynamic-runner-controller
  namespace: dynamic-runner
 roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
-  name: dynamic-runner-pod-worker
+  name: dynamic-runner-controller
 subjects:
  - kind: ServiceAccount
-    name: dynamic-runner-pod-worker
+    name: dynamic-runner-controller
+    namespace: dynamic-runner
+---
+apiVersion: rbac.authorization.k8s.io/v1
+kind: ClusterRole
+metadata:
+  name: dynamic-runner-spiffe-entries
+rules:
+  - apiGroups: [spire.spiffe.io]
+    resources: [clusterstaticentries]
+    verbs: [create, get, delete]
+---
+apiVersion: rbac.authorization.k8s.io/v1
+kind: ClusterRoleBinding
+metadata:
+  name: dynamic-runner-spiffe-entries
+roleRef:
+  apiGroup: rbac.authorization.k8s.io
+  kind: ClusterRole
+  name: dynamic-runner-spiffe-entries
+subjects:
+  - kind: ServiceAccount
+    name: dynamic-runner-controller
    namespace: dynamic-runner
@@ -4,11 +4,10 @@ metadata:
  name: dynamic-runner-controller
  namespace: dynamic-runner
 spec:
-  type: LoadBalancer
-  loadBalancerIP: 192.168.10.127
+  type: ClusterIP
  selector:
    app.kubernetes.io/name: dynamic-runner-controller
  ports:
-    - name: http
-      port: 8787
-      targetPort: http
+    - name: facade
+      port: 8443
+      targetPort: facade