feat: 部署 OpenSandbox API 与双运行时 Pool

2026-09-18 01:10:07 +00:00
13 changed files with 309 additions and 34 deletions
@@ -42,12 +42,9 @@ attestation。Server 使用 external bundle publisher 持续维护 sandbox
 显式关闭 Server 与 OIDC Provider，只部署 Agent DaemonSet 和 SPIFFE CSI Driver；因此
 不会产生第二个 trust root。

-`spire-smoke` namespace、ServiceAccount 和 `sandbox-spire-smoke` ClusterSPIFFEID 只用于
-普通 Pod 的 CSI 回归夹具，稳定身份为 `spiffe://ddupan.top/sandbox/smoke`。Kata guest
-不能复用 node Agent 暴露的 Unix socket；virtio-fs 只能呈现 socket 路径，不能把连接
-跨过 VM 边界。Kata workload 必须使用 guest 内 Agent，具体约束见
-`platform/sandbox-kata/README.md`。测试 Pod 临时创建并在验收后删除，普通 Pod 的身份
-声明保留。
+`spire-smoke` namespace、ServiceAccount 和 `sandbox-spire-smoke` ClusterSPIFFEID 是
+普通 Pod 与后续 Kata guest 的回归夹具，稳定身份为
+`spiffe://ddupan.top/sandbox/smoke`。测试 Pod 临时创建并在验收后删除，身份声明保留。

 Kata 阶段使用官方 4.1.0 `kata-deploy` chart 的短生命周期 `job` 模式，逐节点安装并
 重启 K3s。只启用 `kata-clh-runtime-rs`，不创建默认 `kata` 别名；该 handler 的
@@ -0,0 +1,17 @@
+---
+apiVersion: kustomize.toolkit.fluxcd.io/v1
+kind: Kustomization
+metadata:
+  name: opensandbox-pools
+  namespace: flux-system
+spec:
+  dependsOn:
+    - name: opensandbox
+  interval: 10m
+  path: ./platform/sandbox-opensandbox-pools
+  prune: true
+  sourceRef:
+    kind: GitRepository
+    name: flux-system
+  timeout: 20m
+  wait: true
@@ -0,0 +1,18 @@
+---
+apiVersion: kustomize.toolkit.fluxcd.io/v1
+kind: Kustomization
+metadata:
+  name: opensandbox
+  namespace: flux-system
+spec:
+  dependsOn:
+    - name: kata
+    - name: spire-agents
+  interval: 10m
+  path: ./platform/sandbox-opensandbox
+  prune: true
+  sourceRef:
+    kind: GitRepository
+    name: flux-system
+  timeout: 20m
+  wait: true
@@ -7,3 +7,5 @@ resources:
  - apps/spire-bootstrap.yaml
  - apps/spire-agents.yaml
  - apps/kata.yaml
+  - apps/opensandbox.yaml
+  - apps/opensandbox-pools.yaml
@@ -28,32 +28,6 @@ Pod 删除后的 backing-file/VMM 回收和真实构建基准必须作为上线
 由 `VMPodScrape` 写入中央 VictoriaMetrics。它不拥有 Kubernetes API 凭据或 host 写
 权限。

-## Guest 内 SPIFFE 身份
-
-Kata guest 不能直接使用 node SPIRE Agent 的 CSI socket。Unix socket 的路径即使通过
-virtio-fs 出现在 guest 中，连接也不能跨 VM 边界。CI Pod 应以 native sidecar 在同一
-guest 内启动临时 SPIRE Agent，并满足以下约束：
-
-1. 外层 Pod 挂载 `audience=spire-server` 的 Pod-bound projected ServiceAccount token；
-2. 内层 Agent 使用 `k8s_psat` 向中央 Server attestation，Server cluster profile 必须
-   启用 `use_pod_uid_for_agent_id`，使 Agent ID 包含 Pod UID；
-3. 调度器为该具体 Agent 创建 registration entry；SPIFFE ID 使用仓库和任务名等稳定的
-   业务语义，Pod UID 只作为 parent binding，不进入业务身份；
-4. Agent 与 workload 通过 guest 内 `emptyDir` 上的 Unix socket 通信；Pod 必须设置
-   `shareProcessNamespace: true`，否则 `unix` workload attestor 无法从共享 `/proc`
-   解析客户端的 `SO_PEERCRED` PID；
-5. 调度器必须在 registration entry 已同步到 Agent 后才放行 workload。Pod 删除后同步
-   删除 entry，并 evict 对应的临时 Agent 记录。
-
-2026-09-17 的 live PoC 已验证：以 Pod UID 作为 parent 的临时 Agent 成功注册，
-`unix:uid:2000` workload 从 guest-local socket 获得
-`spiffe://ddupan.top/ci/poc/task/build` X.509-SVID，Pod 正常退出。PoC 资源随后全部清理，
-中央 SPIRE 恢复声明式配置。
-
-SPIRE 1.15.3 已支持 `use_pod_uid_for_agent_id`，但当前使用的 hardened chart 0.30.2
-尚未把它暴露到 values/template。正式部署应先给 chart 补齐该字段并向上游提交，随后
-采用包含修复的 release；不得把手工修改 Server ConfigMap 作为运行方案。
-
 ## 上线验收

 Flux reconciliation 完成后至少确认：
@@ -61,8 +35,8 @@ Flux reconciliation 完成后至少确认：
 1. 两个节点重新回到 Ready，`RuntimeClass/kata-clh-runtime-rs` 存在；
 2. Kata Pod 内核与 LXC host 内核不同，且 `/dev/kvm` 可用；
 3. Cloud Hypervisor API 的 `vm.info.config.memory.shared` 为 `true`；
-4. Kata Pod 内层 Agent 的 ID 包含该 Pod UID；只有调度器创建的业务 entry 所匹配的
-   workload UID 能从 guest-local socket 获得预期 SPIFFE ID；
+4. `spire-smoke` ServiceAccount 的 Kata Pod 可获得
+   `spiffe://ddupan.top/sandbox/smoke`，错误 ServiceAccount 无法获得身份；
 5. block-backed `emptyDir` 上 Docker 使用 `overlay2`，BuildKit 与 kind smoke test
   通过；kind 的 dockerd bootstrap 需要先在 guest 内执行
   `mknod /dev/kmsg c 1 11`；
@@ -0,0 +1,4 @@
+apiVersion: kustomize.config.k8s.io/v1beta1
+kind: Kustomization
+resources:
+  - pools.yaml
@@ -0,0 +1,123 @@
+---
+apiVersion: sandbox.opensandbox.io/v1alpha1
+kind: Pool
+metadata:
+  name: ci-pod
+  namespace: opensandbox
+spec:
+  template:
+    metadata:
+      labels:
+        ci.ddupan.top/backend: pod
+    spec:
+      containers:
+        - name: sandbox
+          image: sandbox-registry.cn-zhangjiakou.cr.aliyuncs.com/opensandbox/code-interpreter:v1.1.0
+          command: [/opt/opensandbox/task-executor]
+          args: [-listen-addr=0.0.0.0:5758, -log-dir=/tmp]
+          env:
+            - name: SANDBOX_MAIN_CONTAINER
+              value: sandbox
+            - name: EXECD_ENVS
+              value: /opt/opensandbox/.env
+            - name: EXECD
+              value: /opt/opensandbox/execd
+          resources:
+            requests:
+              cpu: 100m
+              memory: 256Mi
+            limits:
+              cpu: "2"
+              memory: 4Gi
+          volumeMounts:
+            - name: opensandbox-bin
+              mountPath: /opt/opensandbox
+            - name: sandbox-storage
+              mountPath: /var/lib/sandbox
+      initContainers:
+        - name: task-executor-installer
+          image: sandbox-registry.cn-zhangjiakou.cr.aliyuncs.com/opensandbox/task-executor:v0.1.0
+          command: [/bin/sh, -c]
+          args: [cp /workspace/server /opt/opensandbox/task-executor && chmod 0755 /opt/opensandbox/task-executor]
+          volumeMounts:
+            - name: opensandbox-bin
+              mountPath: /opt/opensandbox
+        - name: execd-installer
+          image: sandbox-registry.cn-zhangjiakou.cr.aliyuncs.com/opensandbox/execd:v1.0.22
+          command: [/bin/sh, -c]
+          args: [cp ./execd /opt/opensandbox/execd && cp ./bootstrap.sh /opt/opensandbox/bootstrap.sh && chmod 0755 /opt/opensandbox/execd /opt/opensandbox/bootstrap.sh]
+          volumeMounts:
+            - name: opensandbox-bin
+              mountPath: /opt/opensandbox
+      volumes:
+        - name: opensandbox-bin
+          emptyDir: {}
+        - name: sandbox-storage
+          emptyDir: {}
+  capacitySpec:
+    bufferMax: 1
+    bufferMin: 0
+    poolMax: 4
+    poolMin: 0
+---
+apiVersion: sandbox.opensandbox.io/v1alpha1
+kind: Pool
+metadata:
+  name: ci-vm
+  namespace: opensandbox
+spec:
+  template:
+    metadata:
+      labels:
+        ci.ddupan.top/backend: vm
+    spec:
+      runtimeClassName: kata-clh-runtime-rs
+      containers:
+        - name: sandbox
+          image: sandbox-registry.cn-zhangjiakou.cr.aliyuncs.com/opensandbox/code-interpreter:v1.1.0
+          command: [/opt/opensandbox/task-executor]
+          args: [-listen-addr=0.0.0.0:5758, -log-dir=/tmp]
+          env:
+            - name: SANDBOX_MAIN_CONTAINER
+              value: sandbox
+            - name: EXECD_ENVS
+              value: /opt/opensandbox/.env
+            - name: EXECD
+              value: /opt/opensandbox/execd
+          resources:
+            requests:
+              cpu: 250m
+              memory: 512Mi
+            limits:
+              cpu: "4"
+              memory: 8Gi
+          volumeMounts:
+            - name: opensandbox-bin
+              mountPath: /opt/opensandbox
+            - name: sandbox-storage
+              mountPath: /var/lib/sandbox
+      initContainers:
+        - name: task-executor-installer
+          image: sandbox-registry.cn-zhangjiakou.cr.aliyuncs.com/opensandbox/task-executor:v0.1.0
+          command: [/bin/sh, -c]
+          args: [cp /workspace/server /opt/opensandbox/task-executor && chmod 0755 /opt/opensandbox/task-executor]
+          volumeMounts:
+            - name: opensandbox-bin
+              mountPath: /opt/opensandbox
+        - name: execd-installer
+          image: sandbox-registry.cn-zhangjiakou.cr.aliyuncs.com/opensandbox/execd:v1.0.22
+          command: [/bin/sh, -c]
+          args: [cp ./execd /opt/opensandbox/execd && cp ./bootstrap.sh /opt/opensandbox/bootstrap.sh && chmod 0755 /opt/opensandbox/execd /opt/opensandbox/bootstrap.sh]
+          volumeMounts:
+            - name: opensandbox-bin
+              mountPath: /opt/opensandbox
+      volumes:
+        - name: opensandbox-bin
+          emptyDir: {}
+        - name: sandbox-storage
+          emptyDir: {}
+  capacitySpec:
+    bufferMax: 1
+    bufferMin: 0
+    poolMax: 2
+    poolMin: 0
@@ -0,0 +1,19 @@
+# OpenSandbox
+
+Flux installs the upstream all-in-one OpenSandbox chart pinned to
+`helm/opensandbox/0.2.2` (`8f01e935`). The API is cluster-internal and intentionally runs a
+single replica until shared server state and HA behaviour have been validated.
+
+`ci-pod` uses `runc`; `ci-vm` uses the separately managed
+`kata-clh-runtime-rs` RuntimeClass. Both Pools start at zero and create capacity
+on demand. They currently use the upstream interpreter image to validate the
+Lifecycle API and Pool allocation independently of the CI scheduler cutover.
+
+The dynamic runner worker, runner image, guest-local SPIRE Agent and Docker
+sidecar are introduced only after this layer is Ready. In particular, do not
+mount the host SPIFFE CSI socket into `ci-vm`: Unix sockets do not cross the
+Kata VM boundary.
+
+Smoke test both backends through the same API by creating sandboxes with
+`extensions.poolRef` set to `ci-pod` and `ci-vm`, then confirm their
+BatchSandboxes, Pods and VMMs disappear after deletion.
@@ -0,0 +1,31 @@
+apiVersion: helm.toolkit.fluxcd.io/v2
+kind: HelmRelease
+metadata:
+  name: opensandbox
+  namespace: opensandbox-system
+spec:
+  chart:
+    spec:
+      chart: ./kubernetes/charts/opensandbox
+      interval: 1h
+      reconcileStrategy: Revision
+      sourceRef:
+        kind: GitRepository
+        name: opensandbox
+  driftDetection:
+    mode: enabled
+  install:
+    strategy:
+      name: RetryOnFailure
+      retryInterval: 5m
+  interval: 30m
+  releaseName: opensandbox
+  targetNamespace: opensandbox-system
+  timeout: 15m
+  upgrade:
+    strategy:
+      name: RetryOnFailure
+      retryInterval: 5m
+  valuesFrom:
+    - kind: ConfigMap
+      name: opensandbox-values
@@ -0,0 +1,7 @@
+apiVersion: kustomize.config.k8s.io/v1beta1
+kind: Kustomization
+resources:
+  - namespaces.yaml
+  - repository.yaml
+  - values.yaml
+  - helmrelease.yaml
@@ -0,0 +1,10 @@
+---
+apiVersion: v1
+kind: Namespace
+metadata:
+  name: opensandbox-system
+---
+apiVersion: v1
+kind: Namespace
+metadata:
+  name: opensandbox
@@ -0,0 +1,11 @@
+apiVersion: source.toolkit.fluxcd.io/v1
+kind: GitRepository
+metadata:
+  name: opensandbox
+  namespace: opensandbox-system
+spec:
+  interval: 1h
+  ref:
+    tag: helm/opensandbox/0.2.2
+  timeout: 60s
+  url: https://github.com/alibaba/OpenSandbox.git
@@ -0,0 +1,62 @@
+apiVersion: v1
+kind: ConfigMap
+metadata:
+  name: opensandbox-values
+  namespace: opensandbox-system
+data:
+  values.yaml: |
+    opensandbox-controller:
+      controller:
+        logLevel: info
+        replicaCount: 1
+        metrics:
+          enabled: true
+          secure: false
+          port: 8080
+        resources:
+          requests:
+            cpu: 25m
+            memory: 64Mi
+          limits:
+            cpu: 500m
+            memory: 256Mi
+
+    opensandbox-server:
+      server:
+        replicaCount: 1
+        resources:
+          requests:
+            cpu: 100m
+            memory: 256Mi
+          limits:
+            cpu: "1"
+            memory: 1Gi
+      configToml: |
+        [server]
+        host = "0.0.0.0"
+        port = 80
+        api_key = ""
+
+        [log]
+        level = "INFO"
+
+        [runtime]
+        type = "kubernetes"
+        execd_image = "sandbox-registry.cn-zhangjiakou.cr.aliyuncs.com/opensandbox/execd:v1.0.22"
+
+        [kubernetes]
+        kubeconfig_path = ""
+        namespace = "opensandbox"
+        informer_enabled = true
+        informer_resync_seconds = 300
+        informer_watch_timeout_seconds = 60
+        snapshot_create_timeout_seconds = 900
+        workload_provider = "batchsandbox"
+        batchsandbox_template_file = "/etc/opensandbox/example.batchsandbox-template.yaml"
+
+        [egress]
+        image = "sandbox-registry.cn-zhangjiakou.cr.aliyuncs.com/opensandbox/egress:v1.1.6"
+        mode = "dns+nft"
+
+    opensandbox-node-agent:
+      enabled: false