diff --git a/clusters/sandbox/README.md b/clusters/sandbox/README.md index 527fb17..430fe24 100644 --- a/clusters/sandbox/README.md +++ b/clusters/sandbox/README.md @@ -46,6 +46,11 @@ attestation。Server 使用 external bundle publisher 持续维护 sandbox 普通 Pod 与后续 Kata guest 的回归夹具,稳定身份为 `spiffe://ddupan.top/sandbox/smoke`。测试 Pod 临时创建并在验收后删除,身份声明保留。 +Kata 阶段使用官方 4.1.0 `kata-deploy` chart 的短生命周期 `job` 模式,逐节点安装并 +重启 K3s。只启用 `kata-clh-runtime-rs`,不创建默认 `kata` 别名;该 handler 的 +`emptyDir` 固定使用 `block-plain`,为 Docker/BuildKit overlay2 与 kind 提供 guest +内块设备文件系统。详细限制与上线验收见 `platform/sandbox-kata/README.md`。 + ## 监控边界 这里只管理 sandbox LXC 内的 Kubernetes 监控,不负责 PVE 宿主监控。LXC 与宿主共享 diff --git a/clusters/sandbox/apps/kata.yaml b/clusters/sandbox/apps/kata.yaml new file mode 100644 index 0000000..bf04446 --- /dev/null +++ b/clusters/sandbox/apps/kata.yaml @@ -0,0 +1,18 @@ +--- +apiVersion: kustomize.toolkit.fluxcd.io/v1 +kind: Kustomization +metadata: + name: kata + namespace: flux-system +spec: + dependsOn: + - name: monitoring-operator + - name: spire-agents + interval: 10m + path: ./platform/sandbox-kata + prune: true + sourceRef: + kind: GitRepository + name: flux-system + timeout: 35m + wait: true diff --git a/clusters/sandbox/kustomization.yaml b/clusters/sandbox/kustomization.yaml index 5a47e5a..fa2ce88 100644 --- a/clusters/sandbox/kustomization.yaml +++ b/clusters/sandbox/kustomization.yaml @@ -6,3 +6,4 @@ resources: - apps/monitoring.yaml - apps/spire-bootstrap.yaml - apps/spire-agents.yaml + - apps/kata.yaml diff --git a/platform/sandbox-kata/README.md b/platform/sandbox-kata/README.md new file mode 100644 index 0000000..c5e24e3 --- /dev/null +++ b/platform/sandbox-kata/README.md @@ -0,0 +1,48 @@ +# Sandbox Kata Containers + +本目录通过 Flux 安装 Kata Containers 4.1.0,只启用 Cloud Hypervisor 的 Rust +runtime,并创建明确命名的 `kata-clh-runtime-rs` RuntimeClass。OpenSandbox 的 VM +Pool 必须显式选择该 RuntimeClass;不创建含义不明确的 `kata` 默认别名。 +OCI chart 同时固定到已验证的 4.1.0 artifact digest,升级时必须重新执行本页验收。 + +安装使用官方 `kata-deploy` chart 的 `job` 模式。每次 install/upgrade 由 dispatcher +逐个节点创建短生命周期、可修改 host 的安装 Job,写入 Kata artifacts 和 K3s containerd +配置并重启对应节点的 K3s;节点恢复 Ready 后才继续下一节点。安装结束后不保留拥有 +host 写权限的 DaemonSet。新增节点后必须触发 HelmRelease upgrade,使 dispatcher +重新枚举节点。 + +`clh-runtime-rs` 固定使用: + +```toml +[runtime] +emptydir_mode = "block-plain" +``` + +因此普通 `emptyDir` 会在 kubelet volume 目录创建稀疏 backing file,并作为块设备 +热插拔给 guest。Docker/BuildKit 可以在 guest ext4 上使用原生 overlay2,避免 +virtio-fs 作为 overlayfs upperdir 时的限制。Kata 当前不会用 +`emptyDir.sizeLimit` 决定该虚拟盘容量;实际容量取决于节点 rootfs。节点磁盘占用、 +Pod 删除后的 backing-file/VMM 回收和真实构建基准必须作为上线验收项目。 + +`kata-monitor` 常驻每个节点,只读访问 K3s containerd socket 与 Kata sandbox 状态, +由 `VMPodScrape` 写入中央 VictoriaMetrics。它不拥有 Kubernetes API 凭据或 host 写 +权限。 + +## 上线验收 + +Flux reconciliation 完成后至少确认: + +1. 两个节点重新回到 Ready,`RuntimeClass/kata-clh-runtime-rs` 存在; +2. Kata Pod 内核与 LXC host 内核不同,且 `/dev/kvm` 可用; +3. Cloud Hypervisor API 的 `vm.info.config.memory.shared` 为 `true`; +4. `spire-smoke` ServiceAccount 的 Kata Pod 可获得 + `spiffe://ddupan.top/sandbox/smoke`,错误 ServiceAccount 无法获得身份; +5. block-backed `emptyDir` 上 Docker 使用 `overlay2`,BuildKit 与 kind smoke test + 通过;kind 的 dockerd bootstrap 需要先在 guest 内执行 + `mknod /dev/kmsg c 1 11`; +6. 删除测试 Pod 后,Cloud Hypervisor 进程、backing file 和临时数据全部回收,且 + LXC 没有新增 OOM 事件; +7. 中央 VictoriaMetrics 中两个 `kata-monitor` target 均为 `up=1`。 + +不要依赖手工修改 `/opt/kata` 或 K3s containerd 配置;任何修复都必须回写 chart +values 并由 Flux reconciliation。 diff --git a/platform/sandbox-kata/helmrelease.yaml b/platform/sandbox-kata/helmrelease.yaml new file mode 100644 index 0000000..94a27b4 --- /dev/null +++ b/platform/sandbox-kata/helmrelease.yaml @@ -0,0 +1,40 @@ +--- +apiVersion: helm.toolkit.fluxcd.io/v2 +kind: HelmRelease +metadata: + name: kata-deploy + namespace: kata-system +spec: + chartRef: + kind: OCIRepository + name: kata-deploy + driftDetection: + mode: enabled + install: + strategy: + name: RetryOnFailure + retryInterval: 5m + interval: 30m + postRenderers: + - kustomize: + patches: + - target: + kind: DaemonSet + name: kata-monitor + patch: | + - op: add + path: /spec/template/spec/automountServiceAccountToken + value: false + - op: add + path: /spec/template/spec/containers/0/ports/0/name + value: metrics + releaseName: kata-deploy + targetNamespace: kata-system + timeout: 30m + upgrade: + strategy: + name: RetryOnFailure + retryInterval: 5m + valuesFrom: + - kind: ConfigMap + name: kata-deploy-values diff --git a/platform/sandbox-kata/kustomization.yaml b/platform/sandbox-kata/kustomization.yaml new file mode 100644 index 0000000..b2a4e51 --- /dev/null +++ b/platform/sandbox-kata/kustomization.yaml @@ -0,0 +1,9 @@ +--- +apiVersion: kustomize.config.k8s.io/v1beta1 +kind: Kustomization +resources: + - namespace.yaml + - repository.yaml + - values.yaml + - helmrelease.yaml + - monitor-scrape.yaml diff --git a/platform/sandbox-kata/monitor-scrape.yaml b/platform/sandbox-kata/monitor-scrape.yaml new file mode 100644 index 0000000..1d56e28 --- /dev/null +++ b/platform/sandbox-kata/monitor-scrape.yaml @@ -0,0 +1,17 @@ +--- +apiVersion: operator.victoriametrics.com/v1beta1 +kind: VMPodScrape +metadata: + name: kata-monitor + namespace: monitoring +spec: + namespaceSelector: + matchNames: + - kata-system + podMetricsEndpoints: + - interval: 30s + port: metrics + selector: + matchLabels: + app.kubernetes.io/instance: kata-deploy + app.kubernetes.io/name: kata-monitor diff --git a/platform/sandbox-kata/namespace.yaml b/platform/sandbox-kata/namespace.yaml new file mode 100644 index 0000000..1ba20e4 --- /dev/null +++ b/platform/sandbox-kata/namespace.yaml @@ -0,0 +1,5 @@ +--- +apiVersion: v1 +kind: Namespace +metadata: + name: kata-system diff --git a/platform/sandbox-kata/repository.yaml b/platform/sandbox-kata/repository.yaml new file mode 100644 index 0000000..860c811 --- /dev/null +++ b/platform/sandbox-kata/repository.yaml @@ -0,0 +1,11 @@ +--- +apiVersion: source.toolkit.fluxcd.io/v1 +kind: OCIRepository +metadata: + name: kata-deploy + namespace: kata-system +spec: + interval: 1h + ref: + digest: sha256:33f102f6db70083de4fc8238af4439c4245a601bcb6a72e49bc80529098aefc0 + url: oci://ghcr.io/kata-containers/kata-deploy-charts/kata-deploy diff --git a/platform/sandbox-kata/values.yaml b/platform/sandbox-kata/values.yaml new file mode 100644 index 0000000..0221d72 --- /dev/null +++ b/platform/sandbox-kata/values.yaml @@ -0,0 +1,44 @@ +--- +apiVersion: v1 +kind: ConfigMap +metadata: + name: kata-deploy-values + namespace: kata-system +data: + values.yaml: | + deploymentMode: job + k8sDistribution: k3s + + job: + parallelism: 1 + + snapshotter: + setup: [] + + shims: + disableAll: true + clh-runtime-rs: + enabled: true + dropIn: | + [runtime] + emptydir_mode = "block-plain" + + defaultShim: + amd64: clh-runtime-rs + + runtimeClasses: + enabled: true + createDefault: false + + node-feature-discovery: + enabled: false + + monitor: + enabled: true + resources: + requests: + cpu: 25m + memory: 64Mi + limits: + cpu: 200m + memory: 192Mi