重构为直接 OpenSandbox 生命周期调度
This commit is contained in:
+39
-42
@@ -1,55 +1,52 @@
|
||||
# OpenSandbox Kata runner
|
||||
# OpenSandbox runner
|
||||
|
||||
The OpenSandbox worker replaces the legacy direct Cloud Hypervisor launcher for
|
||||
jobs labelled `self-hosted, vm`. It runs in the sandbox Kubernetes cluster and
|
||||
uses two local control planes:
|
||||
Gitea 的 `workflow_job` webhook 只负责发现带 `self-hosted,pod` 或
|
||||
`self-hosted,vm` label 的 queued job。controller 不再把请求写入 NATS，而是携带由
|
||||
homelab ExternalSecret 挂载的 API key 直接调用 OpenSandbox Lifecycle API：
|
||||
|
||||
1. OpenSandbox Lifecycle API creates a sandbox from the `ci-vm` Pool.
|
||||
2. Kubernetes API exposes the concrete BatchSandbox allocation and manages its
|
||||
short-lived `ClusterStaticEntry`.
|
||||
```text
|
||||
http://10.60.0.13:8080/v1/sandboxes
|
||||
```
|
||||
|
||||
## Identity ordering
|
||||
请求通过 `extensions.poolRef` 选择 `ci-pod` 或 `ci-vm`。地址是 VyOS HAProxy 的
|
||||
内网 TCP frontend，backend 为 sandbox 两个节点上的固定 NodePort；不通过公网或
|
||||
Cloudflare Tunnel。OpenSandbox key 只从 OpenBao `kv/k8s/opensandbox-api.api_key`
|
||||
进入 homelab Secret，进程通过 `OPENSANDBOX_API_KEY_FILE` 读取。
|
||||
|
||||
The worker sends the stable repository/job identity and a generated runner name
|
||||
as task environment. Once OpenSandbox has allocated a Pool Pod, the worker reads
|
||||
the Pod UID and creates an entry with:
|
||||
## 身份顺序
|
||||
|
||||
- parent: `spiffe://ddupan.top/spire/agent/k8s_psat/sandbox-kata/pod/<pod-uid>`;
|
||||
- workload: `spiffe://ddupan.top/ci/<owner>/<repository>/<job>`;
|
||||
- selector: `unix:uid:2000`.
|
||||
Lifecycle 请求把稳定的 repository/task SPIFFE ID 放入 task environment。sandbox
|
||||
集群内的 `opensandbox-identity` controller 读取 BatchSandbox allocation 得到实际
|
||||
Pod UID，然后创建：
|
||||
|
||||
The Pool must run the runner task as UID 2000 and set
|
||||
`shareProcessNamespace: true`. Its guest-local SPIRE Agent uses a Pod-bound PSAT
|
||||
and exposes the Workload API through the shared `spire-agent-socket` emptyDir.
|
||||
The runner image starts through `gitea-opensandbox-runner`, which waits until the
|
||||
exact expected SVID is available before it registers with Gitea. This prevents a
|
||||
job from starting between Pod allocation and entry reconciliation.
|
||||
- parent：`spiffe://ddupan.top/spire/agent/k8s_psat/sandbox-kata/pod/<pod-uid>`；
|
||||
- workload：`spiffe://ddupan.top/ci/<owner>/<repository>/<task>`；
|
||||
- selector：`unix:uid:2000`。
|
||||
|
||||
The UID selector is the boundary between containers in the same Kata Pod. The
|
||||
SPIRE Agent and privileged Docker daemon must not run as UID 2000. The runner may
|
||||
access Docker only through a group-owned Unix socket.
|
||||
job ID 只进入诊断 label，不进入业务身份。Pool 内 runner 进程固定使用 UID 2000，
|
||||
Pod 设置 `shareProcessNamespace: true`；guest-local SPIRE Agent 使用 Pod-bound PSAT，
|
||||
通过内存 emptyDir 暴露 Workload API。`gitea-opensandbox-runner` 等待精确 SVID，随后
|
||||
通过一次性 nonce URL 领取 Gitea registration token。token 不进入 Lifecycle 请求、
|
||||
BatchSandbox、镜像或 sandbox Secret。
|
||||
|
||||
## Required Pool contract
|
||||
identity controller 只能读取 opensandbox namespace 的 BatchSandbox/Pod，并维护带
|
||||
自身 label 的 ClusterStaticEntry。它不持有 OpenSandbox API key、Gitea token 或 Bao
|
||||
凭据。BatchSandbox 消失后，对应 entry 在下一次 reconcile 删除；Pod 删除后，按 Pod
|
||||
UID attestation 的临时 Agent 失去父级。
|
||||
|
||||
The `ci-vm` Pool template owns infrastructure that callers cannot override in
|
||||
Pool mode:
|
||||
## Pool 契约
|
||||
|
||||
- `runtimeClassName: kata-clh-runtime-rs` with block-backed emptyDir storage;
|
||||
- runner image containing `gitea-opensandbox-runner` and SPIRE CLI;
|
||||
- guest-local SPIRE Agent sidecar and projected audience `spire-server` token;
|
||||
- `shareProcessNamespace: true`;
|
||||
- runner UID 2000 and a distinct UID for every sidecar;
|
||||
- ephemeral Gitea registration token delivery;
|
||||
- Docker/BuildKit storage and socket entirely inside the Kata guest.
|
||||
`ci-vm` 使用 `kata-clh-runtime-rs`；`ci-pod` 使用默认 runc。两者都要求：
|
||||
|
||||
The worker ServiceAccount needs read access to BatchSandboxes and Pods and
|
||||
create/get/delete access to ClusterStaticEntries. OpenSandbox API credentials,
|
||||
when enabled, are mounted from a Secret and read from
|
||||
`OPENSANDBOX_API_KEY_FILE`.
|
||||
- runner 镜像包含 Gitea Runner、Node.js action userspace、SPIRE CLI 和 identity gate；
|
||||
- runner UID 2000，SPIRE Agent 与 privileged dockerd 使用不同 UID；
|
||||
- Docker socket 通过 group 2000 共享，Docker 数据仅存在于 sandbox emptyDir；
|
||||
- `self-hosted` 必须是所有 runner labels 的前缀；
|
||||
- ephemeral/once runner 完成一项任务后退出。
|
||||
|
||||
## Cleanup
|
||||
## 清理与恢复
|
||||
|
||||
On success, failure, timeout, or cancellation the worker deletes the
|
||||
ClusterStaticEntry before deleting the sandbox. Both deletes accept `404`, so a
|
||||
JetStream redelivery can safely repeat cleanup. The SPIRE Agent registration is
|
||||
bound to the Pod UID and is removed by SPIRE after the Pod disappears.
|
||||
controller 监控 Lifecycle 状态，在终止、失败、超时或取消时调用 DELETE。API delete、
|
||||
identity entry delete 均接受对象已不存在。controller 重启时，OpenSandbox timeout
|
||||
仍是最终回收边界；后续可基于 metadata list 恢复主动监控，但不得为此重新引入消息
|
||||
队列。
|
||||
|
||||
Reference in New Issue
Block a user