feat: 部署 OpenSandbox API 与双运行时 Pool

合并内部 chart fork 与 sandbox-kata Pod UID PSAT profile。

clusters/homelab/README.md

@@ -50,6 +50,7 @@ sudo k3s kubectl -n flux-system get gitrepositories,kustomizations
 - VictoriaMetrics Operator 已固定现有 chart `0.66.2` 并完成分阶段 Flux HelmRelease
   接管；Metrics、Logs、Traces 与 Grafana 也已统一完成 Flux 接管；
 - External Secrets Operator 已固定 chart `2.8.0` 并完成分阶段接管；
-- SPIRE 已按官方 hardened chart `0.30.2`（SPIRE `1.15.3`）声明，使用共享
+- SPIRE 已按 hardened chart 内部 fork `0.30.2-ddupan.1`（基于上游 `0.30.2`，SPIRE
+  `1.15.3`）声明，使用共享
   PostgreSQL 与独立 signing-key PVC；首次上线和 OpenBao JWT-SVID PoC 尚待合并后验证；
 - root Kustomization 与所有 brownfield 子 Kustomization 继续保持 `prune: false`。

clusters/sandbox/README.md

@@ -0,0 +1,61 @@
+# Sandbox 集群
+
+这里是 OpenSandbox、CI 和 AI Agent workload 所在双节点 k3s 集群的 Flux
+reconciliation 入口。LXC、PostgreSQL、K3s、固定版本的 Flux controllers 与 root
+sync 由 `infrastructure/sandbox-cluster/` 中的 Ansible 管理；本目录只组合集群内
+workload。
+
+Flux 通过 `https://git.ddupan.top/panxiao81/homelab-infra.git` 读取公开仓库。
+Ansible 将 homelab CA 注入 `GitRepository/flux-system` 引用的同名 Secret，不使用
+长期 Git 凭据。root Kustomization 从 `./clusters/sandbox` 开始 reconciliation，
+初始保持 `prune: false`。
+
+Root bootstrap 已完成。后续按依赖顺序分别引入：
+
+1. 监控 CRD、kube-state-metrics 以及 kubelet/cAdvisor 抓取配置；
+2. SPIRE Agent、SPIFFE CSI Driver 与 workload registration；
+3. Kata Containers、`block-plain` RuntimeClass；
+4. OpenSandbox operator/server 及 `ci-pod`、`ci-vm` Pools。
+
+每一阶段单独合并并等待对应 Flux Kustomization Ready，不在 bootstrap 时一次性部署。
+第一阶段监控拆为 `monitoring-operator` 与依赖它的 `monitoring`，防止 VM CR 在
+VictoriaMetrics Operator CRD Ready 前进入 reconciliation。
+
+SPIRE 阶段先由 `spire-bootstrap` 安装 CRD，并声明按上游 k8s_psat Server plugin
+要求收窄的 reviewer：它可以调用 TokenReview，并只读查询用于证明的 Pod 与 Node。
+Agent ServiceAccount 留给后续 HelmRelease 创建，避免两个声明方争夺同一资源。随后运行
+`infrastructure/sandbox-cluster/ansible/spire-bootstrap.yml`：playbook 从 sandbox
+读取 reviewer token，在内存中组成受限 kubeconfig，再通过 stdin reconcile 到 central
+集群的 `spire-server/spire-external-kubeconfigs` Secret。凭据不写入仓库、日志或控制机
+文件；该 Secret 准备完成后，才能启用 central external PSAT/controller-manager 和
+sandbox Agent/CSI。
+
+External controller-manager 使用独立的 `spire-controller-manager` ServiceAccount；其
+RBAC 与上游 controller-manager 所需权限一致，用于读取 workload selectors、维护
+SPIFFE CR status/finalizer 和 leader election。它不复用只允许 TokenReview 的 Server
+reviewer。Ansible 将两份 kubeconfig 写入同一个 central Secret 的不同 key，便于 central
+chart 分别绑定 `sandbox` 与 `sandbox-controller`。
+
+Central SPIRE Server 通过内网 `spire-server.ad.ddupan.top:8081` 接收 sandbox Agent
+attestation。Server 使用 external bundle publisher 持续维护 sandbox
+`spire-system/spire-bundle`，Agent 不固定或复制 trust bundle。Sandbox HelmRelease
+显式关闭 Server 与 OIDC Provider，只部署 Agent DaemonSet 和 SPIFFE CSI Driver；因此
+不会产生第二个 trust root。
+
+`spire-smoke` namespace、ServiceAccount 和 `sandbox-spire-smoke` ClusterSPIFFEID 是
+普通 Pod 与后续 Kata guest 的回归夹具，稳定身份为
+`spiffe://ddupan.top/sandbox/smoke`。测试 Pod 临时创建并在验收后删除，身份声明保留。
+
+Kata 阶段使用官方 4.1.0 `kata-deploy` chart 的短生命周期 `job` 模式，逐节点安装并
+重启 K3s。只启用 `kata-clh-runtime-rs`，不创建默认 `kata` 别名；该 handler 的
+`emptyDir` 固定使用 `block-plain`，为 Docker/BuildKit overlay2 与 kind 提供 guest
+内块设备文件系统。详细限制与上线验收见 `platform/sandbox-kata/README.md`。
+
+## 监控边界
+
+这里只管理 sandbox LXC 内的 Kubernetes 监控，不负责 PVE 宿主监控。LXC 与宿主共享
+内核，即使 lxcfs 虚拟化了内存和 uptime，容器内 `/proc/stat` 仍是宿主 CPU 视图；
+在 LXC 内运行 node_exporter 会生成混合语义并重复采集宿主指标，因此禁止部署。
+
+Sandbox 节点与 workload 指标来自 kubelet/cAdvisor 和 kube-state-metrics；K3s 或 LXC
+特有但上述接口未覆盖的指标，应使用目标明确的 collector，不以 node_exporter 补齐。

clusters/sandbox/apps/kata.yaml

@@ -0,0 +1,18 @@
+---
+apiVersion: kustomize.toolkit.fluxcd.io/v1
+kind: Kustomization
+metadata:
+  name: kata
+  namespace: flux-system
+spec:
+  dependsOn:
+    - name: monitoring-operator
+    - name: spire-agents
+  interval: 10m
+  path: ./platform/sandbox-kata
+  prune: true
+  sourceRef:
+    kind: GitRepository
+    name: flux-system
+  timeout: 35m
+  wait: true

clusters/sandbox/apps/monitoring-operator.yaml

@@ -0,0 +1,15 @@
+---
+apiVersion: kustomize.toolkit.fluxcd.io/v1
+kind: Kustomization
+metadata:
+  name: monitoring-operator
+  namespace: flux-system
+spec:
+  interval: 10m
+  path: ./platform/sandbox-monitoring/operator
+  prune: true
+  sourceRef:
+    kind: GitRepository
+    name: flux-system
+  timeout: 10m
+  wait: true

clusters/sandbox/apps/monitoring.yaml

@@ -0,0 +1,17 @@
+---
+apiVersion: kustomize.toolkit.fluxcd.io/v1
+kind: Kustomization
+metadata:
+  name: monitoring
+  namespace: flux-system
+spec:
+  dependsOn:
+    - name: monitoring-operator
+  interval: 10m
+  path: ./platform/sandbox-monitoring/workloads
+  prune: true
+  sourceRef:
+    kind: GitRepository
+    name: flux-system
+  timeout: 10m
+  wait: true

clusters/sandbox/apps/opensandbox-pools.yaml

@@ -0,0 +1,17 @@
+---
+apiVersion: kustomize.toolkit.fluxcd.io/v1
+kind: Kustomization
+metadata:
+  name: opensandbox-pools
+  namespace: flux-system
+spec:
+  dependsOn:
+    - name: opensandbox
+  interval: 10m
+  path: ./platform/sandbox-opensandbox-pools
+  prune: true
+  sourceRef:
+    kind: GitRepository
+    name: flux-system
+  timeout: 20m
+  wait: true

clusters/sandbox/apps/opensandbox.yaml

@@ -0,0 +1,18 @@
+---
+apiVersion: kustomize.toolkit.fluxcd.io/v1
+kind: Kustomization
+metadata:
+  name: opensandbox
+  namespace: flux-system
+spec:
+  dependsOn:
+    - name: kata
+    - name: spire-agents
+  interval: 10m
+  path: ./platform/sandbox-opensandbox
+  prune: true
+  sourceRef:
+    kind: GitRepository
+    name: flux-system
+  timeout: 20m
+  wait: true

clusters/sandbox/apps/spire-agents.yaml

@@ -0,0 +1,18 @@
+---
+apiVersion: kustomize.toolkit.fluxcd.io/v1
+kind: Kustomization
+metadata:
+  name: spire-agents
+  namespace: flux-system
+spec:
+  dependsOn:
+    - name: spire-bootstrap
+    - name: monitoring-operator
+  interval: 10m
+  path: ./platform/sandbox-spire/agents
+  prune: true
+  sourceRef:
+    kind: GitRepository
+    name: flux-system
+  timeout: 15m
+  wait: true

clusters/sandbox/apps/spire-bootstrap.yaml

@@ -0,0 +1,15 @@
+---
+apiVersion: kustomize.toolkit.fluxcd.io/v1
+kind: Kustomization
+metadata:
+  name: spire-bootstrap
+  namespace: flux-system
+spec:
+  interval: 10m
+  path: ./platform/sandbox-spire/bootstrap
+  prune: true
+  sourceRef:
+    kind: GitRepository
+    name: flux-system
+  timeout: 10m
+  wait: true

clusters/sandbox/kustomization.yaml

@@ -0,0 +1,11 @@
+---
+apiVersion: kustomize.config.k8s.io/v1beta1
+kind: Kustomization
+resources:
+  - apps/monitoring-operator.yaml
+  - apps/monitoring.yaml
+  - apps/spire-bootstrap.yaml
+  - apps/spire-agents.yaml
+  - apps/kata.yaml
+  - apps/opensandbox.yaml
+  - apps/opensandbox-pools.yaml

infrastructure/dns/records.yml

@@ -15,10 +15,12 @@ homelab_dns:
       - { zone: ad.ddupan.top, name: sandbox-k8s, type: A, values: [10.60.0.13] }
       - { zone: ad.ddupan.top, name: retrolab, type: A, values: [10.60.0.10] }
       - { zone: ad.ddupan.top, name: grafana, type: A, values: [192.168.10.127] }
+      - { zone: ad.ddupan.top, name: metrics-write, type: A, values: [192.168.10.127] }
       - { zone: ad.ddupan.top, name: netbox, type: A, values: [192.168.10.127] }
       - { zone: ad.ddupan.top, name: nats, type: A, values: [192.168.10.127] }
       - { zone: ad.ddupan.top, name: s3, type: A, values: [192.168.10.127] }
       - { zone: ad.ddupan.top, name: spire-oidc, type: A, values: [192.168.10.127] }
+      - { zone: ad.ddupan.top, name: spire-server, type: A, values: [192.168.10.127] }
       - { zone: ad.ddupan.top, name: zot, type: A, values: [192.168.10.127] }
       - { zone: ad.ddupan.top, name: zot-push, type: A, values: [192.168.10.127] }
 

infrastructure/proxmox/ansible/roles/vyos_router/defaults/main.yml

@@ -77,3 +77,14 @@ vyos_ospf_networks:
 vyos_postgresql_listen_address: "10.60.0.1"
 vyos_postgresql_port: 5432
 vyos_postgresql_primary_address: "10.60.0.11"
+
+# Stable API endpoint for the two-node sandbox K3s control plane. HAProxy owns
+# this /32 on the existing labnet interface and performs only TCP health checks.
+vyos_sandbox_k3s_api_address: "10.60.0.13"
+vyos_sandbox_k3s_api_interface: eth1
+vyos_sandbox_k3s_api_port: 6443
+vyos_sandbox_k3s_servers:
+  - name: sandbox1
+    address: "10.60.0.11"
+  - name: sandbox2
+    address: "10.60.0.12"

infrastructure/proxmox/ansible/roles/vyos_router/tasks/main.yml

@@ -3,6 +3,23 @@
 # `set` lines against the running config and issues only the differences, so
 # re-runs are no-ops. Supports --check and --diff.
 
+- name: Read current VyOS configuration commands
+  vyos.vyos.vyos_command:
+    commands:
+      - show configuration commands
+  register: _vyos_running_commands
+  changed_when: false
+
+- name: Remove obsolete HAProxy PostgreSQL protocol check
+  vyos.vyos.vyos_config:
+    lines:
+      - delete load-balancing haproxy backend sandbox-postgresql-primary health-check
+    save: true
+    comment: Remove unsupported PostgreSQL protocol check
+  when: >-
+    "set load-balancing haproxy backend sandbox-postgresql-primary health-check 'pgsql'"
+    in _vyos_running_commands.stdout[0]
+
 - name: Render the desired configuration
   ansible.builtin.set_fact:
     _vyos_lines: >-

infrastructure/proxmox/ansible/roles/vyos_router/templates/vyos.conf.j2

@@ -35,11 +35,23 @@ set load-balancing haproxy service sandbox-postgresql port {{ vyos_postgresql_po
 set load-balancing haproxy service sandbox-postgresql mode 'tcp'
 set load-balancing haproxy service sandbox-postgresql backend 'sandbox-postgresql-primary'
 set load-balancing haproxy backend sandbox-postgresql-primary mode 'tcp'
-set load-balancing haproxy backend sandbox-postgresql-primary health-check 'pgsql'
 set load-balancing haproxy backend sandbox-postgresql-primary server primary address {{ vyos_postgresql_primary_address }}
 set load-balancing haproxy backend sandbox-postgresql-primary server primary port {{ vyos_postgresql_port }}
 set load-balancing haproxy backend sandbox-postgresql-primary server primary check
 
+{# --- Sandbox K3s API: stable address with both control-plane nodes --- #}
+set interfaces ethernet {{ vyos_sandbox_k3s_api_interface }} address '{{ vyos_sandbox_k3s_api_address }}/32'
+set load-balancing haproxy service sandbox-k3s-api listen-address {{ vyos_sandbox_k3s_api_address }}
+set load-balancing haproxy service sandbox-k3s-api port {{ vyos_sandbox_k3s_api_port }}
+set load-balancing haproxy service sandbox-k3s-api mode 'tcp'
+set load-balancing haproxy service sandbox-k3s-api backend 'sandbox-k3s-api'
+set load-balancing haproxy backend sandbox-k3s-api mode 'tcp'
+{% for server in vyos_sandbox_k3s_servers %}
+set load-balancing haproxy backend sandbox-k3s-api server {{ server.name }} address {{ server.address }}
+set load-balancing haproxy backend sandbox-k3s-api server {{ server.name }} port {{ vyos_sandbox_k3s_api_port }}
+set load-balancing haproxy backend sandbox-k3s-api server {{ server.name }} check
+{% endfor %}
+
 {# --- management --- #}
 set service ssh port 22
 

infrastructure/sandbox-cluster/README.md

@@ -0,0 +1,142 @@
+# Sandbox 集群
+
+该目录管理供 OpenSandbox、CI 和 AI Agent workload 使用的独立双节点 k3s 集群。
+基础设施和节点生命周期由 Ansible 管理；Kubernetes API 可用后，集群内组件由
+`clusters/sandbox/` 下的 Flux desired state 管理。
+
+## 所有权边界
+
+Ansible 管理以下持久状态：
+
+- pve1/pve2 上的 privileged LXC、磁盘、固定网络与 KVM/vhost/TUN/kmsg 设备；
+- LXC OS 基线、内部 CA、外部 PostgreSQL 与 K3s；
+- 固定版本的 Flux controllers 和 root sync bootstrap；
+- 只读状态验证和数据库切换 runbook。
+
+Flux 管理以下 Kubernetes 资源：
+
+- Kata Containers 和 CI 专用的 `block-plain` RuntimeClass；
+- SPIRE Agent、SPIFFE CSI Driver 与 workload identity 声明；
+- vmagent、kube-state-metrics、kubelet/cAdvisor scrape 配置和告警；
+- OpenSandbox operator/server、`ci-pod` 与 `ci-vm` Pools。
+
+同一个对象只能有一个 owner。Ansible 不直接部署上述集群内 workload；Flux 不管理
+LXC、K3s datastore 或 K3s 本身。
+
+监控范围仅覆盖 sandbox LXC 内的 Kubernetes 与 workload。不得在 LXC 内部署
+node_exporter：LXC 的 `/proc` 是 lxcfs 虚拟视图与宿主内核视图的混合，尤其
+`/proc/stat` 会重复暴露 PVE 宿主 CPU 数据。PVE 自身的 node_exporter 或其他宿主
+监控不属于本目录；sandbox 使用 kubelet/cAdvisor 和 kube-state-metrics。
+
+## 声明拓扑
+
+| 对象 | PVE 节点 | VMID | 地址 | 资源 |
+|---|---:|---:|---|---|
+| `sandbox1` | pve1 | 148 | `10.60.0.11/24` | 4 vCPU / 6 GiB / 2 GiB swap |
+| `sandbox2` | pve2 | 149 | `10.60.0.12/24` | 4 vCPU / 4 GiB / 2 GiB swap |
+| K3s API VIP | VyOS | — | `10.60.0.13:6443` | HAProxy TCP LB |
+
+节点与 API VIP 均位于现有 PVE `labnet`（VLAN 100，`10.60.0.0/24`），通过
+VyOS `10.60.0.1` 路由；不为 sandbox 新建 VNet，也不占用 `192.168.10.0/24`
+地址。需要从 LAN 访问的服务统一经 VyOS 路由或 LB 暴露。两个 LXC 均使用
+`pve-rg-hdd` 上的 32 GiB rootfs；不为 `/var/lib/kubelet` 单独创建 volume。Kata
+`block-plain` 产生的数据随 Pod 生命周期清理，当前规模没有额外磁盘故障域的需求。
+K3s API 的 `10.60.0.13/32` 由 VyOS 现有 labnet interface 持有，HAProxy 以 TCP
+健康检查把 `6443` 分发到两个 control-plane 节点；它与 PostgreSQL 主从切换逻辑无关。
+LXC 内通过 `/etc/tmpfiles.d/kmsg.conf` 持久维护 `/dev/kmsg -> /dev/console`；否则
+kubelet 会因 LXC 不提供真实 host `/dev/kmsg` 而反复退出。
+每个 LXC 还以只读 bind mount 使用宿主的 `/lib/modules`。LXC 与 PVE 宿主共享内核，
+guest 若看不到对应版本的模块目录，K3s 无法加载 `br_netfilter` 和 `overlay`，Flannel
+也不会生成节点的 subnet 配置。
+
+## PostgreSQL 写入口与切换
+
+K3s 使用外部 PostgreSQL，首期采用 primary + synchronous standby。VyOS 在 labnet
+gateway `10.60.0.1:5432` 提供固定 TCP 入口，backend 只包含当前声明的 primary；
+不把普通 TCP 或 PostgreSQL 存活检查等同于“节点可写”，也不自动把流量切到 standby。
+VyOS 2025.11 的 PostgreSQL protocol check 会生成缺少必需 `user` 参数的 HAProxy
+配置，因此这里只使用基础 TCP check；真正的可写性由 `verify.yml` 通过 SQL 验证。
+
+数据库切换必须由 Ansible runbook 受控完成：先隔离旧 primary，再提升 standby，最后
+更新 VyOS backend。首期不部署 PgBouncer、Patroni、repmgr 或额外 DCS，也不宣称两节点
+PostgreSQL 能够自动 HA。以后具备第三个仲裁节点时再重新评估自动 failover。
+
+数据库凭据位于 Bao `kv/infra/sandbox-postgresql`，包含 K3s 登录密码和 physical
+replication 密码。运行 Ansible 前由本机 SPIFFE identity 获取短期 Bao token，再把
+两个值注入 `SANDBOX_K3S_DB_PASSWORD` 与 `SANDBOX_REPLICATION_PASSWORD`；凭据不写入
+inventory、Git 或 Ansible fact cache。首次创建 standby 只允许覆盖不含任何业务库的
+Ubuntu 默认空集群，之后重复执行不会 reseed。
+
+## Ansible
+
+安装固定依赖：
+
+```bash
+cd infrastructure/sandbox-cluster/ansible
+uv venv .venv
+uv pip install --python .venv/bin/python -r requirements.txt
+source .venv/bin/activate
+ansible-galaxy collection install -r requirements.yml
+export PVE_API_PYTHON="$PWD/.venv/bin/python"
+```
+
+部署入口最终为：
+
+```bash
+ansible-playbook site.yml
+```
+
+只安装或 reconcile K3s（LXC、OS baseline 和 PostgreSQL 已就绪时）：
+
+```bash
+ansible-playbook k3s.yml
+```
+
+K3s 外部 datastore URI 由运行时 `SANDBOX_K3S_DB_PASSWORD` 生成，密码在 URI 中
+进行 URL 编码，最终仅持久化于节点 root 可读的 `/etc/rancher/k3s/config.yaml`
+（mode `0600`）。首节点生成的 K3s join token 仅在同一次 Ansible run 内传给第二节点；
+不在 inventory 或 Git 中维护副本。首期关闭内建 Traefik 和 ServiceLB。
+
+部署后使用同一组运行时凭据执行只读验收：
+
+```bash
+ansible-playbook verify.yml
+```
+
+当前已经声明 LXC 生命周期、最小 OS baseline、PostgreSQL 和 K3s，包括系统级
+homelab CA trust。Flux `v2.9.5` controllers 与 root sync 也由 Ansible 通过 K3s
+server manifests 管理；root 使用 homelab CA 访问公开 Gitea 仓库，不保存 Git token。
+集群内 workload 由 `clusters/sandbox/` 分阶段纳入 Flux。
+
+## SPIRE 跨集群 bootstrap
+
+Sandbox 复用 homelab 的 SPIRE Server 与 `ddupan.top` trust domain。Flux 首先安装
+SPIRE CRD，并创建供 k8s_psat 使用的 reviewer。它按上游 Server chart 的权限模型调用
+TokenReview，并以 `get/list` 读取用于证明的 Pod 与 Node；不具有修改 workload 的权限。
+在该 Kustomization Ready 后运行：
+
+```bash
+cd infrastructure/sandbox-cluster/ansible
+ansible-playbook spire-bootstrap.yml
+```
+
+Playbook 不把 reviewer token 或生成的 kubeconfig 落盘，而是将目标 Secret manifest
+通过 stdin 交给本机 homelab `k3s kubectl`。目标 Secret
+`spire-server/spire-external-kubeconfigs` 由 Ansible 单独拥有；Flux 和人工操作不得写入。
+第二次运行必须为零变更。
+
+Secret 的 `sandbox` key 仅供 SPIRE Server 的 external PSAT plugin 验证 token 与对应的
+Pod/Node；
+`sandbox-controller` key 供 external controller-manager 读取 Pod/Node、reconcile SPIFFE
+CR 及执行 leader election。两者使用不同 ServiceAccount，不得合并权限或互换。
+
+## 已验证的 Kata CI 前置条件
+
+- Cloud Hypervisor 必须报告 `vm.info.config.memory.shared=true`；
+- CI RuntimeClass 必须使用 `[runtime] emptydir_mode = "block-plain"`；
+- dockerd bootstrap 在 guest 内创建 `/dev/kmsg`：`mknod /dev/kmsg c 1 11`；
+- `/var/lib/docker` 必须是 guest block device 上的 ext4，Docker driver 必须为
+  `overlay2`，不能静默退化到 `vfs` 或 `fuse-overlayfs`；
+- Pod 删除后必须不存在遗留 `disk.img`、VMM 或临时 credential。
+
+PoC 的完整数据和陷阱见 `../kata-lxc-lab/README.md`。

infrastructure/sandbox-cluster/ansible/ansible.cfg

@@ -0,0 +1,11 @@
+[defaults]
+inventory = inventory/hosts.yml
+host_key_checking = True
+interpreter_python = auto_silent
+retry_files_enabled = False
+local_tmp = /tmp/ansible-sandbox-cluster-local
+remote_tmp = /tmp
+roles_path = roles
+
+[ssh_connection]
+pipelining = True

infrastructure/sandbox-cluster/ansible/inventory/group_vars/all.yml

@@ -0,0 +1,61 @@
+---
+sandbox_pve_api_host: "{{ lookup('env', 'PROXMOX_HOST') }}"
+sandbox_pve_api_user: "{{ lookup('env', 'PROXMOX_USER') }}"
+sandbox_pve_api_token_id: "{{ lookup('env', 'PROXMOX_TOKEN_ID') }}"
+sandbox_pve_api_token_secret: "{{ lookup('env', 'PROXMOX_TOKEN_SECRET') }}"
+sandbox_pve_validate_certs: true
+sandbox_pve_api_python_interpreter: "{{ lookup('env', 'PVE_API_PYTHON') | default(ansible_playbook_python, true) }}"
+
+sandbox_bridge: labnet
+sandbox_gateway: 10.60.0.1
+sandbox_nameserver: 192.168.10.5
+sandbox_search_domain: ad.ddupan.top
+sandbox_lxc_template: laptop:vztmpl/ubuntu-24.04-standard_24.04-2_amd64.tar.zst
+sandbox_rootfs_storage: pve-rg-hdd
+sandbox_rootfs_gb: 32
+sandbox_lxc_cores: 4
+sandbox_lxc_swap_mb: 2048
+sandbox_lxc_pubkey: "{{ lookup('file', '~/.ssh/id_ed25519.pub') }}"
+sandbox_internal_ca_url: https://bao.ad.ddupan.top:8200/v1/pki/ca/pem
+sandbox_base_packages:
+  - ca-certificates
+  - curl
+  - jq
+  - openssh-server
+
+sandbox_postgresql_version: 16
+sandbox_postgresql_primary_address: 10.60.0.11
+sandbox_postgresql_standby_address: 10.60.0.12
+sandbox_postgresql_lb_address: 10.60.0.1
+sandbox_postgresql_database: k3s
+sandbox_postgresql_user: k3s
+sandbox_postgresql_replication_user: k3s_replication
+sandbox_postgresql_replication_slot: sandbox2
+sandbox_postgresql_rotate_passwords: false
+sandbox_postgresql_k3s_password: "{{ lookup('env', 'SANDBOX_K3S_DB_PASSWORD') }}"
+sandbox_postgresql_replication_password: "{{ lookup('env', 'SANDBOX_REPLICATION_PASSWORD') }}"
+
+# This is rendered only at playbook runtime from the Bao-provided password. K3s
+# necessarily persists its datastore credential, but config.yaml is root-only.
+sandbox_k3s_datastore_password_encoded: >-
+  {{ sandbox_postgresql_k3s_password | urlencode | replace('/', '%2F') }}
+sandbox_k3s_datastore_endpoint: >-
+  {{
+    [
+      'postgres://', sandbox_postgresql_user, ':', sandbox_k3s_datastore_password_encoded,
+      '@', sandbox_postgresql_lb_address, ':5432/', sandbox_postgresql_database, '?sslmode=disable'
+    ] | join
+  }}
+server_config_yaml: |
+  datastore-endpoint: {{ sandbox_k3s_datastore_endpoint }}
+  disable:
+    - servicelb
+    - traefik
+user_kubectl: false
+
+sandbox_flux_version: v2.9.5
+sandbox_flux_install_url: >-
+  https://github.com/fluxcd/flux2/releases/download/{{ sandbox_flux_version }}/install.yaml
+sandbox_flux_repository_url: https://git.ddupan.top/panxiao81/homelab-infra.git
+sandbox_flux_repository_branch: main
+sandbox_flux_path: ./clusters/sandbox

infrastructure/sandbox-cluster/ansible/inventory/hosts.yml

@@ -0,0 +1,48 @@
+---
+all:
+  children:
+    sandbox_pve:
+      hosts:
+        pve1:
+          ansible_host: 192.168.10.4
+          sandbox_lxc:
+            vmid: 148
+            hostname: sandbox1
+            address: 10.60.0.11/24
+            memory_mb: 6144
+        pve2:
+          ansible_host: 192.168.10.7
+          sandbox_lxc:
+            vmid: 149
+            hostname: sandbox2
+            address: 10.60.0.12/24
+            memory_mb: 4096
+
+    k3s_cluster:
+      children:
+        server:
+          hosts:
+            sandbox1:
+              ansible_host: 10.60.0.11
+            sandbox2:
+              ansible_host: 10.60.0.12
+        agent:
+          hosts: {}
+      vars:
+        ansible_user: root
+        k3s_version: v1.36.4+k3s1
+        api_endpoint: 10.60.0.13
+        k3s_server_config_mode: "0600"
+        use_external_database: true
+
+    postgres_primary:
+      hosts:
+        sandbox1:
+
+    postgres_standby:
+      hosts:
+        sandbox2:
+
+  vars:
+    ansible_user: root
+    ansible_python_interpreter: /usr/bin/python3

infrastructure/sandbox-cluster/ansible/k3s.yml

@@ -0,0 +1,17 @@
+---
+- name: Validate runtime K3s datastore credential
+  hosts: k3s_cluster
+  gather_facts: false
+  tasks:
+    - name: Require the Bao-provided PostgreSQL password
+      ansible.builtin.assert:
+        that:
+          - sandbox_postgresql_k3s_password | length > 0
+          - server_config_yaml is defined
+          - (server_config_yaml | from_yaml)['datastore-endpoint'] == sandbox_k3s_datastore_endpoint
+        fail_msg: SANDBOX_K3S_DB_PASSWORD must be populated from Bao
+        quiet: true
+      no_log: true
+
+- name: Install the sandbox K3s control plane
+  ansible.builtin.import_playbook: k3s.orchestration.site

infrastructure/sandbox-cluster/ansible/requirements.txt

@@ -0,0 +1,4 @@
+ansible-core==2.21.2
+proxmoxer==2.3.0
+requests==2.34.2
+netaddr==1.3.0

infrastructure/sandbox-cluster/ansible/requirements.yml

@@ -0,0 +1,15 @@
+---
+collections:
+  - name: community.proxmox
+    version: 2.0.0
+  - name: community.postgresql
+    version: 4.2.0
+  - name: ansible.posix
+    version: 2.2.2
+  - name: community.general
+    version: 13.4.0
+  - name: community.library_inventory_filtering_v1
+    version: 1.1.5
+  - name: https://github.com/k3s-io/k3s-ansible.git
+    type: git
+    version: 1a600b60d37e0f8a6e2e79b0e474147b5b108ae5

infrastructure/sandbox-cluster/ansible/roles/sandbox_base/handlers/main.yml

@@ -0,0 +1,5 @@
+---
+- name: Update sandbox CA certificates
+  ansible.builtin.command:
+    cmd: update-ca-certificates
+  changed_when: true

infrastructure/sandbox-cluster/ansible/roles/sandbox_base/tasks/main.yml

@@ -0,0 +1,58 @@
+---
+- name: Install sandbox base packages
+  ansible.builtin.apt:
+    name: "{{ sandbox_base_packages }}"
+    state: present
+    update_cache: true
+    cache_valid_time: 3600
+
+- name: Fetch the homelab internal CA
+  ansible.builtin.uri:
+    url: "{{ sandbox_internal_ca_url }}"
+    return_content: true
+    validate_certs: true
+  register: sandbox_internal_ca
+  changed_when: false
+  retries: 3
+  delay: 5
+  until: sandbox_internal_ca is succeeded
+
+- name: Validate the homelab internal CA response
+  ansible.builtin.assert:
+    that:
+      - "'BEGIN CERTIFICATE' in sandbox_internal_ca.content"
+    fail_msg: OpenBao did not return a PEM certificate
+    quiet: true
+
+- name: Install the homelab internal CA
+  ansible.builtin.copy:
+    dest: /usr/local/share/ca-certificates/ddupan-internal-ca.crt
+    content: "{{ sandbox_internal_ca.content }}"
+    owner: root
+    group: root
+    mode: "0644"
+  notify: Update sandbox CA certificates
+
+- name: Enable the SSH service
+  ansible.builtin.service:
+    name: ssh
+    enabled: true
+    state: started
+
+# LXC does not expose the host kernel log device. Kubelet only needs a writable
+# kmsg-compatible character device, and the container console is the established
+# LXC mapping for this purpose. tmpfiles recreates the link after every boot.
+- name: Persist the LXC kubelet kmsg mapping
+  ansible.builtin.copy:
+    dest: /etc/tmpfiles.d/kmsg.conf
+    content: "L+ /dev/kmsg - - - - /dev/console\n"
+    owner: root
+    group: root
+    mode: "0644"
+
+- name: Expose the LXC console as kmsg
+  ansible.builtin.file:
+    src: /dev/console
+    dest: /dev/kmsg
+    state: link
+    force: true

infrastructure/sandbox-cluster/ansible/roles/sandbox_flux/tasks/main.yml

@@ -0,0 +1,30 @@
+---
+- name: Wait for the K3s server manifest directory
+  ansible.builtin.wait_for:
+    path: /var/lib/rancher/k3s/server/manifests
+    state: present
+    timeout: 120
+
+- name: Download the pinned Flux controller manifest
+  ansible.builtin.get_url:
+    url: "{{ sandbox_flux_install_url }}"
+    dest: /var/lib/rancher/k3s/server/manifests/flux-install.yaml
+    owner: root
+    group: root
+    mode: "0644"
+    timeout: 120
+
+- name: Read the installed homelab CA
+  ansible.builtin.slurp:
+    src: /usr/local/share/ca-certificates/ddupan-internal-ca.crt
+  register: sandbox_flux_ca
+  no_log: true
+
+- name: Install the Flux source and root sync manifest
+  ansible.builtin.template:
+    src: flux-sync.yaml.j2
+    dest: /var/lib/rancher/k3s/server/manifests/flux-sync.yaml
+    owner: root
+    group: root
+    mode: "0600"
+  no_log: true

infrastructure/sandbox-cluster/ansible/roles/sandbox_flux/templates/flux-sync.yaml.j2

@@ -0,0 +1,39 @@
+---
+apiVersion: v1
+kind: Secret
+metadata:
+  name: flux-system
+  namespace: flux-system
+type: Opaque
+stringData:
+  caFile: |
+{{ sandbox_flux_ca.content | b64decode | indent(4, true) }}
+---
+apiVersion: source.toolkit.fluxcd.io/v1
+kind: GitRepository
+metadata:
+  name: flux-system
+  namespace: flux-system
+spec:
+  interval: 1m
+  ref:
+    branch: {{ sandbox_flux_repository_branch }}
+  secretRef:
+    name: flux-system
+  timeout: 60s
+  url: {{ sandbox_flux_repository_url }}
+---
+apiVersion: kustomize.toolkit.fluxcd.io/v1
+kind: Kustomization
+metadata:
+  name: flux-system
+  namespace: flux-system
+spec:
+  interval: 10m
+  path: {{ sandbox_flux_path }}
+  prune: false
+  sourceRef:
+    kind: GitRepository
+    name: flux-system
+  timeout: 3m
+  wait: true

infrastructure/sandbox-cluster/ansible/roles/sandbox_lxc/tasks/main.yml

@@ -0,0 +1,214 @@
+---
+- name: Validate sandbox declaration and API inputs
+  ansible.builtin.assert:
+    that:
+      - sandbox_lxc.vmid | int > 100
+      - sandbox_lxc.hostname | length > 0
+      - sandbox_lxc.address is match('^10\.60\.0\.[0-9]{1,3}/24$')
+      - sandbox_lxc.memory_mb | int >= 4096
+      - sandbox_pve_api_host | length > 0
+      - sandbox_pve_api_user | length > 0
+      - sandbox_pve_api_token_id | length > 0
+      - sandbox_pve_api_token_secret | length > 0
+    fail_msg: Invalid sandbox declaration or missing Proxmox API environment
+  no_log: true
+
+- name: Check whether the LXC already exists
+  ansible.builtin.stat:
+    path: /etc/pve/lxc/{{ sandbox_lxc.vmid }}.conf
+  register: sandbox_lxc_config
+
+- name: Read current sandbox LXC runtime state
+  community.proxmox.proxmox_vm_info:
+    api_host: "{{ sandbox_pve_api_host }}"
+    api_user: "{{ sandbox_pve_api_user }}"
+    api_token_id: "{{ sandbox_pve_api_token_id }}"
+    api_token_secret: "{{ sandbox_pve_api_token_secret }}"
+    validate_certs: "{{ sandbox_pve_validate_certs }}"
+    ca_path: /etc/ssl/certs/ca-certificates.crt
+    api_timeout: 120
+    vmid: "{{ sandbox_lxc.vmid }}"
+  register: sandbox_lxc_info
+  delegate_to: localhost
+  no_log: true
+  vars:
+    ansible_python_interpreter: "{{ sandbox_pve_api_python_interpreter }}"
+
+- name: Read current sandbox LXC configuration
+  ansible.builtin.uri:
+    url: >-
+      https://{{ sandbox_pve_api_host }}:8006/api2/json/nodes/{{ inventory_hostname }}/lxc/{{ sandbox_lxc.vmid }}/config
+    headers:
+      Authorization: >-
+        PVEAPIToken={{ sandbox_pve_api_user }}!{{ sandbox_pve_api_token_id }}={{ sandbox_pve_api_token_secret }}
+    ca_path: /etc/ssl/certs/ca-certificates.crt
+    return_content: true
+    status_code: [200, 404, 500]
+  register: sandbox_lxc_current_config
+  delegate_to: localhost
+  no_log: true
+
+- name: Reconcile sandbox LXC through the PVE API
+  community.proxmox.proxmox:
+    api_host: "{{ sandbox_pve_api_host }}"
+    api_user: "{{ sandbox_pve_api_user }}"
+    api_token_id: "{{ sandbox_pve_api_token_id }}"
+    api_token_secret: "{{ sandbox_pve_api_token_secret }}"
+    validate_certs: "{{ sandbox_pve_validate_certs }}"
+    ca_path: /etc/ssl/certs/ca-certificates.crt
+    api_timeout: 120
+    node: "{{ inventory_hostname }}"
+    vmid: "{{ sandbox_lxc.vmid }}"
+    hostname: "{{ sandbox_lxc.hostname }}"
+    ostemplate: "{{ sandbox_lxc_template }}"
+    ostype: ubuntu
+    cmode: shell
+    disk_volume:
+      storage: "{{ sandbox_rootfs_storage }}"
+      size: "{{ sandbox_rootfs_gb if not sandbox_lxc_config.stat.exists else omit }}"
+    cores: "{{ sandbox_lxc_cores }}"
+    memory: "{{ sandbox_lxc.memory_mb }}"
+    swap: "{{ sandbox_lxc_swap_mb }}"
+    netif:
+      net0: >-
+        {{
+          'name=eth0,bridge=' ~ sandbox_bridge ~
+          ',ip=' ~ sandbox_lxc.address ~
+          ',gw=' ~ sandbox_gateway ~
+          ',type=veth' ~
+          (',hwaddr=' ~ _sandbox_existing_mac if _sandbox_existing_mac | length > 0 else '')
+        }}
+    nameserver: "{{ sandbox_nameserver }}"
+    searchdomain: "{{ sandbox_search_domain }}"
+    unprivileged: false
+    onboot: true
+    startup: ["order=30", "up=30", "down=60"]
+    tags: [ansible, sandbox, k3s, kata]
+    pubkey: "{{ sandbox_lxc_pubkey }}"
+    update: true
+    state: present
+    timeout: 120
+  register: sandbox_lxc_api
+  when: not sandbox_lxc_config.stat.exists or _sandbox_api_drift
+  delegate_to: localhost
+  no_log: true
+  vars:
+    ansible_python_interpreter: "{{ sandbox_pve_api_python_interpreter }}"
+    _sandbox_existing_mac: >-
+      {{ sandbox_lxc_current_config.json.data.net0.split('hwaddr=')[1].split(',')[0]
+         if 'hwaddr=' in (sandbox_lxc_current_config.json.data.net0 | default(''))
+         else '' }}
+    _sandbox_api_drift: >-
+      {{
+        sandbox_lxc_current_config.status != 200 or
+        sandbox_lxc_current_config.json.data.hostname | default('') != sandbox_lxc.hostname or
+        sandbox_lxc_current_config.json.data.cores | default(0) | int != sandbox_lxc_cores | int or
+        sandbox_lxc_current_config.json.data.memory | default(0) | int != sandbox_lxc.memory_mb | int or
+        sandbox_lxc_current_config.json.data.swap | default(0) | int != sandbox_lxc_swap_mb | int or
+        sandbox_lxc_current_config.json.data.nameserver | default('') != sandbox_nameserver or
+        sandbox_lxc_current_config.json.data.searchdomain | default('') != sandbox_search_domain or
+        sandbox_lxc_current_config.json.data.cmode | default('') != 'shell' or
+        sandbox_lxc_current_config.json.data.onboot | default(0) | int != 1 or
+        sandbox_lxc_current_config.json.data.tags | default('') != 'ansible;k3s;kata;sandbox' or
+        ('bridge=' ~ sandbox_bridge) not in (sandbox_lxc_current_config.json.data.net0 | default('')) or
+        ('ip=' ~ sandbox_lxc.address) not in (sandbox_lxc_current_config.json.data.net0 | default('')) or
+        ('gw=' ~ sandbox_gateway) not in (sandbox_lxc_current_config.json.data.net0 | default('')) or
+        (sandbox_rootfs_storage ~ ':') not in (sandbox_lxc_current_config.json.data.rootfs | default(''))
+      }}
+
+- name: Grow existing sandbox rootfs through the PVE resize API
+  ansible.builtin.uri:
+    url: >-
+      https://{{ sandbox_pve_api_host }}:8006/api2/json/nodes/{{ inventory_hostname }}/lxc/{{ sandbox_lxc.vmid }}/resize
+    method: PUT
+    headers:
+      Authorization: >-
+        PVEAPIToken={{ sandbox_pve_api_user }}!{{ sandbox_pve_api_token_id }}={{ sandbox_pve_api_token_secret }}
+    body_format: form-urlencoded
+    body:
+      disk: rootfs
+      size: "{{ sandbox_rootfs_gb }}G"
+    ca_path: /etc/ssl/certs/ca-certificates.crt
+    status_code: 200
+  when:
+    - sandbox_lxc_config.stat.exists
+    - _sandbox_current_rootfs_gb | int < sandbox_rootfs_gb | int
+  delegate_to: localhost
+  no_log: true
+  vars:
+    _sandbox_current_rootfs_gb: >-
+      {{
+        sandbox_lxc_current_config.json.data.rootfs |
+        default('') |
+        regex_findall('size=([0-9]+)G') |
+        first |
+        default('0')
+      }}
+
+# These properties are not exposed by community.proxmox. Every retained line
+# is required by the already validated nested Kata/kind runtime.
+- name: Reconcile Kata-specific native LXC properties
+  ansible.builtin.lineinfile:
+    path: /etc/pve/lxc/{{ sandbox_lxc.vmid }}.conf
+    regexp: "^{{ item.key | regex_escape }}:"
+    line: "{{ item.key }}: {{ item.value }}"
+  loop:
+    - { key: features, value: "nesting=1,keyctl=1,fuse=1,mknod=1,force_rw_sys=1" }
+    - { key: dev0, value: "path=/dev/kvm,mode=0660" }
+    - { key: dev1, value: "path=/dev/vhost-net,mode=0660" }
+    - { key: dev2, value: "path=/dev/vhost-vsock,mode=0660" }
+    - { key: dev3, value: "path=/dev/net/tun,mode=0666" }
+    - { key: lxc.apparmor.profile, value: unconfined }
+    - { key: lxc.cgroup2.devices.allow, value: a }
+    - { key: lxc.cap.drop, value: "" }
+    - { key: lxc.mount.auto, value: "proc:rw sys:rw" }
+    - { key: lxc.mount.entry, value: "/lib/modules usr/lib/modules none bind,ro,create=dir 0 0" }
+  loop_control:
+    label: "{{ item.key }}"
+  register: sandbox_lxc_native
+
+- name: Start sandbox LXC
+  community.proxmox.proxmox:
+    api_host: "{{ sandbox_pve_api_host }}"
+    api_user: "{{ sandbox_pve_api_user }}"
+    api_token_id: "{{ sandbox_pve_api_token_id }}"
+    api_token_secret: "{{ sandbox_pve_api_token_secret }}"
+    validate_certs: "{{ sandbox_pve_validate_certs }}"
+    ca_path: /etc/ssl/certs/ca-certificates.crt
+    api_timeout: 120
+    vmid: "{{ sandbox_lxc.vmid }}"
+    state: started
+    timeout: 120
+  when: >-
+    not _sandbox_was_running or
+    not (sandbox_lxc_api is changed or sandbox_lxc_native is changed)
+  delegate_to: localhost
+  no_log: true
+  vars:
+    ansible_python_interpreter: "{{ sandbox_pve_api_python_interpreter }}"
+    _sandbox_was_running: >-
+      {{ (sandbox_lxc_info.proxmox_vms | default([]) | first | default({})).status |
+         default('absent') == 'running' }}
+
+- name: Restart existing sandbox LXC after configuration changes
+  community.proxmox.proxmox:
+    api_host: "{{ sandbox_pve_api_host }}"
+    api_user: "{{ sandbox_pve_api_user }}"
+    api_token_id: "{{ sandbox_pve_api_token_id }}"
+    api_token_secret: "{{ sandbox_pve_api_token_secret }}"
+    validate_certs: "{{ sandbox_pve_validate_certs }}"
+    ca_path: /etc/ssl/certs/ca-certificates.crt
+    api_timeout: 120
+    vmid: "{{ sandbox_lxc.vmid }}"
+    state: restarted
+    timeout: 120
+  when:
+    - _sandbox_was_running
+    - sandbox_lxc_api is changed or sandbox_lxc_native is changed
+  delegate_to: localhost
+  no_log: true
+  vars:
+    ansible_python_interpreter: "{{ sandbox_pve_api_python_interpreter }}"
+    _sandbox_was_running: >-
+      {{ (sandbox_lxc_info.proxmox_vms | default([]) | first | default({})).status |
+         default('absent') == 'running' }}

infrastructure/sandbox-cluster/ansible/roles/sandbox_postgresql/handlers/main.yml

@@ -0,0 +1,6 @@
+---
+- name: Restart sandbox PostgreSQL
+  ansible.builtin.service:
+    name: postgresql
+    enabled: true
+    state: restarted

infrastructure/sandbox-cluster/ansible/roles/sandbox_postgresql/tasks/main.yml

@@ -0,0 +1,50 @@
+---
+- name: Validate sandbox PostgreSQL inputs
+  ansible.builtin.assert:
+    that:
+      - sandbox_postgresql_k3s_password | length >= 32
+      - sandbox_postgresql_replication_password | length >= 32
+      - inventory_hostname in groups['postgres_primary'] or inventory_hostname in groups['postgres_standby']
+    fail_msg: Missing Bao-provided PostgreSQL credentials or invalid inventory role
+  no_log: true
+
+- name: Install PostgreSQL packages
+  ansible.builtin.apt:
+    name:
+      - postgresql-{{ sandbox_postgresql_version }}
+      - postgresql-client-{{ sandbox_postgresql_version }}
+      - python3-psycopg2
+    state: present
+    update_cache: true
+    cache_valid_time: 3600
+
+- name: Check whether synchronous replication has completed bootstrap
+  ansible.builtin.stat:
+    path: /etc/postgresql/{{ sandbox_postgresql_version }}/main/sandbox-synchronous-ready
+  register: sandbox_postgresql_synchronous_marker
+
+- name: Install sandbox PostgreSQL access policy
+  ansible.builtin.template:
+    src: pg_hba.conf.j2
+    dest: /etc/postgresql/{{ sandbox_postgresql_version }}/main/pg_hba.conf
+    owner: postgres
+    group: postgres
+    mode: "0640"
+  notify: Restart sandbox PostgreSQL
+
+- name: Install sandbox PostgreSQL server settings
+  ansible.builtin.template:
+    src: sandbox.conf.j2
+    dest: /etc/postgresql/{{ sandbox_postgresql_version }}/main/conf.d/99-sandbox.conf
+    owner: postgres
+    group: postgres
+    mode: "0644"
+  notify: Restart sandbox PostgreSQL
+
+- name: Configure sandbox PostgreSQL primary
+  ansible.builtin.include_tasks: primary.yml
+  when: inventory_hostname in groups['postgres_primary']
+
+- name: Configure sandbox PostgreSQL standby
+  ansible.builtin.include_tasks: standby.yml
+  when: inventory_hostname in groups['postgres_standby']

infrastructure/sandbox-cluster/ansible/roles/sandbox_postgresql/tasks/primary.yml

@@ -0,0 +1,45 @@
+---
+- name: Apply primary PostgreSQL configuration before replication setup
+  ansible.builtin.meta: flush_handlers
+
+- name: Create K3s PostgreSQL login role
+  community.postgresql.postgresql_user:
+    name: "{{ sandbox_postgresql_user }}"
+    password: "{{ sandbox_postgresql_k3s_password }}"
+    no_password_changes: "{{ not sandbox_postgresql_rotate_passwords }}"
+    role_attr_flags: LOGIN,NOSUPERUSER,NOCREATEDB,NOCREATEROLE,NOREPLICATION
+    state: present
+  become: true
+  become_user: postgres
+  no_log: true
+  environment:
+    PGOPTIONS: -c password_encryption=scram-sha-256
+
+- name: Create K3s PostgreSQL database
+  community.postgresql.postgresql_db:
+    name: "{{ sandbox_postgresql_database }}"
+    owner: "{{ sandbox_postgresql_user }}"
+    state: present
+  become: true
+  become_user: postgres
+
+- name: Create PostgreSQL replication login role
+  community.postgresql.postgresql_user:
+    name: "{{ sandbox_postgresql_replication_user }}"
+    password: "{{ sandbox_postgresql_replication_password }}"
+    no_password_changes: "{{ not sandbox_postgresql_rotate_passwords }}"
+    role_attr_flags: LOGIN,REPLICATION,NOSUPERUSER,NOCREATEDB,NOCREATEROLE
+    state: present
+  become: true
+  become_user: postgres
+  no_log: true
+  environment:
+    PGOPTIONS: -c password_encryption=scram-sha-256
+
+- name: Create physical replication slot for sandbox2
+  community.postgresql.postgresql_slot:
+    name: "{{ sandbox_postgresql_replication_slot }}"
+    slot_type: physical
+    state: present
+  become: true
+  become_user: postgres

infrastructure/sandbox-cluster/ansible/roles/sandbox_postgresql/tasks/standby.yml

@@ -0,0 +1,114 @@
+---
+- name: Detect whether sandbox2 is already a standby
+  community.postgresql.postgresql_query:
+    login_db: postgres
+    query: SELECT pg_is_in_recovery() AS in_recovery
+  become: true
+  become_user: postgres
+  register: sandbox_postgresql_recovery_state
+
+- name: Inspect databases before the initial standby seed
+  community.postgresql.postgresql_query:
+    login_db: postgres
+    query: >-
+      SELECT datname FROM pg_database
+      WHERE NOT datistemplate AND datname <> 'postgres'
+  become: true
+  become_user: postgres
+  register: sandbox_postgresql_existing_databases
+  when: not sandbox_postgresql_recovery_state.query_result[0].in_recovery
+
+- name: Refuse to overwrite a non-empty PostgreSQL node
+  ansible.builtin.assert:
+    that:
+      - sandbox_postgresql_existing_databases.query_result | length == 0
+    fail_msg: Refusing to reseed sandbox2 because it contains non-system databases
+  when: not sandbox_postgresql_recovery_state.query_result[0].in_recovery
+
+- name: Stop PostgreSQL before the initial standby seed
+  ansible.builtin.service:
+    name: postgresql
+    state: stopped
+  when: not sandbox_postgresql_recovery_state.query_result[0].in_recovery
+
+- name: Remove the verified-empty standby data directory
+  ansible.builtin.file:
+    path: /var/lib/postgresql/{{ sandbox_postgresql_version }}/main
+    state: absent
+  when: not sandbox_postgresql_recovery_state.query_result[0].in_recovery
+
+- name: Recreate the standby data directory
+  ansible.builtin.file:
+    path: /var/lib/postgresql/{{ sandbox_postgresql_version }}/main
+    state: directory
+    owner: postgres
+    group: postgres
+    mode: "0700"
+  when: not sandbox_postgresql_recovery_state.query_result[0].in_recovery
+
+- name: Install standby replication password file
+  ansible.builtin.copy:
+    dest: /var/lib/postgresql/.pgpass
+    content: >-
+      {{ sandbox_postgresql_primary_address }}:5432:*:{{ sandbox_postgresql_replication_user }}:{{ sandbox_postgresql_replication_password }}
+    owner: postgres
+    group: postgres
+    mode: "0600"
+  no_log: true
+
+- name: Seed sandbox2 from the PostgreSQL primary
+  ansible.builtin.command:
+    argv:
+      - /usr/bin/pg_basebackup
+      - --host={{ sandbox_postgresql_primary_address }}
+      - --username={{ sandbox_postgresql_replication_user }}
+      - --pgdata=/var/lib/postgresql/{{ sandbox_postgresql_version }}/main
+      - --format=plain
+      - --wal-method=stream
+      - --write-recovery-conf
+      - --slot={{ sandbox_postgresql_replication_slot }}
+  become: true
+  become_user: postgres
+  environment:
+    PGPASSFILE: /var/lib/postgresql/.pgpass
+  when: not sandbox_postgresql_recovery_state.query_result[0].in_recovery
+  no_log: true
+
+- name: Set the standby connection identity
+  ansible.builtin.lineinfile:
+    path: /var/lib/postgresql/{{ sandbox_postgresql_version }}/main/postgresql.auto.conf
+    regexp: ^primary_conninfo =
+    line: >-
+      primary_conninfo = 'host={{ sandbox_postgresql_primary_address }} port=5432
+      user={{ sandbox_postgresql_replication_user }} application_name=sandbox2
+      passfile=/var/lib/postgresql/.pgpass'
+    owner: postgres
+    group: postgres
+    mode: "0600"
+  no_log: true
+
+- name: Start PostgreSQL standby
+  ansible.builtin.service:
+    name: postgresql
+    enabled: true
+    state: started
+
+- name: Wait for sandbox2 to enter recovery
+  community.postgresql.postgresql_query:
+    login_db: postgres
+    query: SELECT pg_is_in_recovery() AS in_recovery
+  become: true
+  become_user: postgres
+  register: sandbox_postgresql_standby_ready
+  retries: 12
+  delay: 5
+  until: sandbox_postgresql_standby_ready.query_result[0].in_recovery
+
+- name: Mark synchronous replication bootstrap complete on the primary
+  ansible.builtin.copy:
+    dest: /etc/postgresql/{{ sandbox_postgresql_version }}/main/sandbox-synchronous-ready
+    content: "sandbox2\n"
+    owner: postgres
+    group: postgres
+    mode: "0644"
+  delegate_to: "{{ groups['postgres_primary'][0] }}"

infrastructure/sandbox-cluster/ansible/roles/sandbox_postgresql/templates/pg_hba.conf.j2

@@ -0,0 +1,5 @@
+# Ansible managed
+local   all             postgres                                peer
+local   all             all                                     peer
+host    {{ sandbox_postgresql_database }} {{ sandbox_postgresql_user }} 10.60.0.0/24 scram-sha-256
+host    replication     {{ sandbox_postgresql_replication_user }} 10.60.0.0/24 scram-sha-256

infrastructure/sandbox-cluster/ansible/roles/sandbox_postgresql/templates/sandbox.conf.j2

@@ -0,0 +1,11 @@
+# Ansible managed
+listen_addresses = '{{ ansible_host }},127.0.0.1'
+password_encryption = 'scram-sha-256'
+wal_level = 'replica'
+max_wal_senders = 10
+max_replication_slots = 10
+hot_standby = on
+{% if inventory_hostname in groups['postgres_primary'] and sandbox_postgresql_synchronous_marker.stat.exists %}
+synchronous_standby_names = 'FIRST 1 (sandbox2)'
+synchronous_commit = 'remote_apply'
+{% endif %}

infrastructure/sandbox-cluster/ansible/roles/sandbox_spire_bootstrap/defaults/main.yml

@@ -0,0 +1,7 @@
+---
+sandbox_spire_bootstrap_api_server: https://10.60.0.13:6443
+sandbox_spire_bootstrap_source_namespace: spire-system
+sandbox_spire_bootstrap_source_secret: spire-server-token-reviewer-token
+sandbox_spire_bootstrap_controller_secret: spire-controller-manager-token
+sandbox_spire_bootstrap_target_namespace: spire-server
+sandbox_spire_bootstrap_target_secret: spire-external-kubeconfigs

infrastructure/sandbox-cluster/ansible/roles/sandbox_spire_bootstrap/tasks/main.yml

@@ -0,0 +1,115 @@
+---
+- name: Wait for the sandbox SPIRE token reviewer credential
+  ansible.builtin.command:
+    argv:
+      - k3s
+      - kubectl
+      - --namespace
+      - "{{ sandbox_spire_bootstrap_source_namespace }}"
+      - get
+      - secret
+      - "{{ sandbox_spire_bootstrap_source_secret }}"
+      - --output=json
+  register: sandbox_spire_bootstrap_reviewer_secret
+  changed_when: false
+  retries: 60
+  delay: 10
+  until:
+    - sandbox_spire_bootstrap_reviewer_secret.rc == 0
+    - (sandbox_spire_bootstrap_reviewer_secret.stdout | from_json).data.token is defined
+    - (sandbox_spire_bootstrap_reviewer_secret.stdout | from_json).data['ca.crt'] is defined
+  no_log: true
+
+- name: Wait for the sandbox SPIRE controller credential
+  ansible.builtin.command:
+    argv:
+      - k3s
+      - kubectl
+      - --namespace
+      - "{{ sandbox_spire_bootstrap_source_namespace }}"
+      - get
+      - secret
+      - "{{ sandbox_spire_bootstrap_controller_secret }}"
+      - --output=json
+  register: sandbox_spire_bootstrap_controller_secret_result
+  changed_when: false
+  retries: 60
+  delay: 10
+  until:
+    - sandbox_spire_bootstrap_controller_secret_result.rc == 0
+    - (sandbox_spire_bootstrap_controller_secret_result.stdout | from_json).data.token is defined
+    - (sandbox_spire_bootstrap_controller_secret_result.stdout | from_json).data['ca.crt'] is defined
+  no_log: true
+
+- name: Extract the sandbox TokenReview credential data
+  ansible.builtin.set_fact:
+    sandbox_spire_bootstrap_secret_data: >-
+      {{ (sandbox_spire_bootstrap_reviewer_secret.stdout | from_json).data }}
+    sandbox_spire_bootstrap_controller_data: >-
+      {{ (sandbox_spire_bootstrap_controller_secret_result.stdout | from_json).data }}
+  no_log: true
+
+- name: Build the restricted sandbox TokenReview kubeconfig
+  ansible.builtin.set_fact:
+    sandbox_spire_bootstrap_kubeconfig: |
+      apiVersion: v1
+      kind: Config
+      clusters:
+        - name: sandbox
+          cluster:
+            server: {{ sandbox_spire_bootstrap_api_server }}
+            certificate-authority-data: {{ sandbox_spire_bootstrap_secret_data['ca.crt'] }}
+      users:
+        - name: spire-server-token-reviewer
+          user:
+            token: {{ sandbox_spire_bootstrap_secret_data.token | b64decode }}
+      contexts:
+        - name: sandbox
+          context:
+            cluster: sandbox
+            user: spire-server-token-reviewer
+      current-context: sandbox
+    sandbox_spire_bootstrap_controller_kubeconfig: |
+      apiVersion: v1
+      kind: Config
+      clusters:
+        - name: sandbox
+          cluster:
+            server: {{ sandbox_spire_bootstrap_api_server }}
+            certificate-authority-data: {{ sandbox_spire_bootstrap_controller_data['ca.crt'] }}
+      users:
+        - name: spire-controller-manager
+          user:
+            token: {{ sandbox_spire_bootstrap_controller_data.token | b64decode }}
+      contexts:
+        - name: sandbox
+          context:
+            cluster: sandbox
+            user: spire-controller-manager
+      current-context: sandbox
+  no_log: true
+
+- name: Reconcile the central SPIRE external kubeconfig Secret
+  ansible.builtin.command:
+    argv:
+      - k3s
+      - kubectl
+      - apply
+      - --filename=-
+    stdin: |
+      apiVersion: v1
+      kind: Secret
+      metadata:
+        name: {{ sandbox_spire_bootstrap_target_secret }}
+        namespace: {{ sandbox_spire_bootstrap_target_namespace }}
+      type: Opaque
+      data:
+        sandbox: {{ sandbox_spire_bootstrap_kubeconfig | b64encode }}
+        sandbox-controller: {{ sandbox_spire_bootstrap_controller_kubeconfig | b64encode }}
+  delegate_to: localhost
+  become: true
+  register: sandbox_spire_bootstrap_target
+  changed_when: >-
+    ' created' in sandbox_spire_bootstrap_target.stdout or
+    ' configured' in sandbox_spire_bootstrap_target.stdout
+  no_log: true

infrastructure/sandbox-cluster/ansible/site.yml

@@ -0,0 +1,60 @@
+---
+- name: Reconcile sandbox LXC nodes
+  hosts: sandbox_pve
+  gather_facts: false
+  serial: 1
+  roles:
+    - sandbox_lxc
+
+- name: Reconcile sandbox node OS baseline
+  hosts: k3s_cluster
+  gather_facts: false
+  roles:
+    - sandbox_base
+
+- name: Reconcile sandbox PostgreSQL primary
+  hosts: postgres_primary
+  gather_facts: false
+  roles:
+    - sandbox_postgresql
+
+- name: Reconcile sandbox PostgreSQL synchronous standby
+  hosts: postgres_standby
+  gather_facts: false
+  roles:
+    - sandbox_postgresql
+
+- name: Finalize synchronous PostgreSQL settings
+  hosts: postgres_primary
+  gather_facts: false
+  roles:
+    - sandbox_postgresql
+
+- name: Validate runtime K3s datastore credential
+  hosts: k3s_cluster
+  gather_facts: false
+  tasks:
+    - name: Require the Bao-provided PostgreSQL password
+      ansible.builtin.assert:
+        that:
+          - sandbox_postgresql_k3s_password | length > 0
+          - server_config_yaml is defined
+          - (server_config_yaml | from_yaml)['datastore-endpoint'] == sandbox_k3s_datastore_endpoint
+        fail_msg: SANDBOX_K3S_DB_PASSWORD must be populated from Bao
+        quiet: true
+      no_log: true
+
+- name: Install the sandbox K3s control plane
+  ansible.builtin.import_playbook: k3s.orchestration.site
+
+- name: Reconcile Flux controllers and root sync
+  hosts: sandbox1
+  gather_facts: false
+  roles:
+    - sandbox_flux
+
+- name: Reconcile central SPIRE access to sandbox Kubernetes
+  hosts: sandbox1
+  gather_facts: false
+  roles:
+    - sandbox_spire_bootstrap

infrastructure/sandbox-cluster/ansible/spire-bootstrap.yml

@@ -0,0 +1,6 @@
+---
+- name: Reconcile central SPIRE access to sandbox Kubernetes
+  hosts: sandbox1
+  gather_facts: false
+  roles:
+    - sandbox_spire_bootstrap

infrastructure/sandbox-cluster/ansible/verify.yml

@@ -0,0 +1,149 @@
+---
+- name: Verify the PostgreSQL datastore through the VyOS endpoint
+  hosts: k3s_cluster
+  gather_facts: false
+  tasks:
+    - name: Query the K3s database through the stable endpoint
+      community.postgresql.postgresql_query:
+        login_host: "{{ sandbox_postgresql_lb_address }}"
+        login_db: "{{ sandbox_postgresql_database }}"
+        login_user: "{{ sandbox_postgresql_user }}"
+        login_password: "{{ sandbox_postgresql_k3s_password }}"
+        query: SELECT NOT pg_is_in_recovery() AS writable
+      register: sandbox_datastore_endpoint
+      no_log: true
+
+    - name: Assert the stable datastore endpoint is writable
+      ansible.builtin.assert:
+        that:
+          - sandbox_datastore_endpoint.query_result[0].writable
+        fail_msg: The VyOS datastore endpoint is not connected to a writable PostgreSQL primary
+        quiet: true
+
+- name: Verify synchronous PostgreSQL replication
+  hosts: postgres_primary
+  gather_facts: false
+  tasks:
+    - name: Query replication state on the primary
+      community.postgresql.postgresql_query:
+        login_db: postgres
+        query: >-
+          SELECT application_name, state, sync_state
+          FROM pg_stat_replication
+          WHERE application_name = 'sandbox2'
+      become: true
+      become_user: postgres
+      register: sandbox_replication_state
+
+    - name: Assert sandbox2 is a synchronous streaming standby
+      ansible.builtin.assert:
+        that:
+          - sandbox_replication_state.query_result | length == 1
+          - sandbox_replication_state.query_result[0].state == 'streaming'
+          - sandbox_replication_state.query_result[0].sync_state == 'sync'
+        fail_msg: sandbox2 is not synchronously streaming from sandbox1
+        quiet: true
+
+- name: Verify PostgreSQL standby recovery state
+  hosts: postgres_standby
+  gather_facts: false
+  tasks:
+    - name: Query recovery state on the standby
+      community.postgresql.postgresql_query:
+        login_db: postgres
+        query: SELECT pg_is_in_recovery() AS in_recovery
+      become: true
+      become_user: postgres
+      register: sandbox_standby_state
+
+    - name: Assert sandbox2 remains in recovery
+      ansible.builtin.assert:
+        that:
+          - sandbox_standby_state.query_result[0].in_recovery
+        fail_msg: sandbox2 is not operating as a PostgreSQL standby
+        quiet: true
+
+- name: Verify K3s node prerequisites
+  hosts: k3s_cluster
+  gather_facts: false
+  tasks:
+    - name: Read the kmsg mapping
+      ansible.builtin.command:
+        cmd: readlink /dev/kmsg
+      register: sandbox_kmsg_target
+      changed_when: false
+
+    - name: Inspect the host kernel module mount
+      ansible.builtin.shell:
+        cmd: >-
+          set -o pipefail &&
+          findmnt -rn -T "/lib/modules/$(uname -r)" -o OPTIONS |
+          grep -Eq '(^|,)ro(,|$)'
+        executable: /bin/bash
+      changed_when: false
+
+    - name: Assert required LXC kernel integration
+      ansible.builtin.assert:
+        that:
+          - sandbox_kmsg_target.stdout == '/dev/console'
+        fail_msg: LXC is missing its persistent kmsg or read-only host module mapping
+        quiet: true
+
+    - name: Read K3s service state
+      ansible.builtin.systemd_service:
+        name: k3s
+      register: sandbox_k3s_service
+
+    - name: Assert K3s is active
+      ansible.builtin.assert:
+        that:
+          - sandbox_k3s_service.status.ActiveState == 'active'
+          - sandbox_k3s_service.status.SubState == 'running'
+        fail_msg: K3s is not running
+        quiet: true
+
+- name: Verify the K3s control plane and API VIP
+  hosts: sandbox1
+  gather_facts: false
+  tasks:
+    - name: Wait for all declared nodes
+      ansible.builtin.command:
+        cmd: k3s kubectl wait --for=condition=Ready nodes --all --timeout=120s
+      changed_when: false
+
+    - name: Read control-plane nodes
+      ansible.builtin.command:
+        cmd: >-
+          k3s kubectl get nodes
+          -l node-role.kubernetes.io/control-plane=true
+          -o name
+      register: sandbox_control_plane_nodes
+      changed_when: false
+
+    - name: Assert both control-plane nodes are registered
+      ansible.builtin.assert:
+        that:
+          - sandbox_control_plane_nodes.stdout_lines | length == 2
+        fail_msg: The sandbox cluster does not contain both control-plane nodes
+        quiet: true
+
+    - name: Wait for all kube-system Pods
+      ansible.builtin.command:
+        cmd: >-
+          k3s kubectl wait --namespace kube-system
+          --for=condition=Ready pods --all --timeout=120s
+      changed_when: false
+
+    - name: Verify the authenticated API path through VyOS
+      ansible.builtin.command:
+        cmd: >-
+          k3s kubectl --server=https://10.60.0.13:6443 get --raw=/livez
+      register: sandbox_api_livez
+      changed_when: false
+
+    - name: Assert the API VIP is live
+      ansible.builtin.assert:
+        that:
+          - sandbox_api_livez.stdout == 'ok'
+        fail_msg: The K3s API is not healthy through the VyOS VIP
+        quiet: true

platform/observability/metrics/kustomization.yaml

@@ -2,6 +2,7 @@ apiVersion: kustomize.config.k8s.io/v1beta1
 kind: Kustomization
 resources:
   - vmsingle.yaml
+  - vmsingle-write-route.yaml
   - vmagent.yaml
   - vmalert.yaml
   - vmalertmanager.yaml

platform/observability/metrics/vmsingle-write-route.yaml

@@ -0,0 +1,24 @@
+---
+# Internal-only remote_write ingress for vmagent instances in other homelab
+# clusters. Expose only the write endpoint, not VictoriaMetrics query/admin APIs.
+apiVersion: gateway.networking.k8s.io/v1
+kind: HTTPRoute
+metadata:
+  name: vmsingle-remote-write
+  namespace: monitoring
+spec:
+  parentRefs:
+    - name: eg
+      namespace: envoy-gateway-system
+      sectionName: https
+  hostnames:
+    - metrics-write.ad.ddupan.top
+  rules:
+    - matches:
+        - method: POST
+          path:
+            type: Exact
+            value: /api/v1/write
+      backendRefs:
+        - name: vmsingle-main
+          port: 8428

platform/sandbox-kata/README.md

@@ -0,0 +1,48 @@
+# Sandbox Kata Containers
+
+本目录通过 Flux 安装 Kata Containers 4.1.0，只启用 Cloud Hypervisor 的 Rust
+runtime，并创建明确命名的 `kata-clh-runtime-rs` RuntimeClass。OpenSandbox 的 VM
+Pool 必须显式选择该 RuntimeClass；不创建含义不明确的 `kata` 默认别名。
+OCI chart 同时固定到已验证的 4.1.0 artifact digest，升级时必须重新执行本页验收。
+
+安装使用官方 `kata-deploy` chart 的 `job` 模式。每次 install/upgrade 由 dispatcher
+逐个节点创建短生命周期、可修改 host 的安装 Job，写入 Kata artifacts 和 K3s containerd
+配置并重启对应节点的 K3s；节点恢复 Ready 后才继续下一节点。安装结束后不保留拥有
+host 写权限的 DaemonSet。新增节点后必须触发 HelmRelease upgrade，使 dispatcher
+重新枚举节点。
+
+`clh-runtime-rs` 固定使用：
+
+```toml
+[runtime]
+emptydir_mode = "block-plain"
+```
+
+因此普通 `emptyDir` 会在 kubelet volume 目录创建稀疏 backing file，并作为块设备
+热插拔给 guest。Docker/BuildKit 可以在 guest ext4 上使用原生 overlay2，避免
+virtio-fs 作为 overlayfs upperdir 时的限制。Kata 当前不会用
+`emptyDir.sizeLimit` 决定该虚拟盘容量；实际容量取决于节点 rootfs。节点磁盘占用、
+Pod 删除后的 backing-file/VMM 回收和真实构建基准必须作为上线验收项目。
+
+`kata-monitor` 常驻每个节点，只读访问 K3s containerd socket 与 Kata sandbox 状态，
+由 `VMPodScrape` 写入中央 VictoriaMetrics。它不拥有 Kubernetes API 凭据或 host 写
+权限。
+
+## 上线验收
+
+Flux reconciliation 完成后至少确认：
+
+1. 两个节点重新回到 Ready，`RuntimeClass/kata-clh-runtime-rs` 存在；
+2. Kata Pod 内核与 LXC host 内核不同，且 `/dev/kvm` 可用；
+3. Cloud Hypervisor API 的 `vm.info.config.memory.shared` 为 `true`；
+4. `spire-smoke` ServiceAccount 的 Kata Pod 可获得
+   `spiffe://ddupan.top/sandbox/smoke`，错误 ServiceAccount 无法获得身份；
+5. block-backed `emptyDir` 上 Docker 使用 `overlay2`，BuildKit 与 kind smoke test
+   通过；kind 的 dockerd bootstrap 需要先在 guest 内执行
+   `mknod /dev/kmsg c 1 11`；
+6. 删除测试 Pod 后，Cloud Hypervisor 进程、backing file 和临时数据全部回收，且
+   LXC 没有新增 OOM 事件；
+7. 中央 VictoriaMetrics 中两个 `kata-monitor` target 均为 `up=1`。
+
+不要依赖手工修改 `/opt/kata` 或 K3s containerd 配置；任何修复都必须回写 chart
+values 并由 Flux reconciliation。

platform/sandbox-kata/helmrelease.yaml

@@ -0,0 +1,40 @@
+---
+apiVersion: helm.toolkit.fluxcd.io/v2
+kind: HelmRelease
+metadata:
+  name: kata-deploy
+  namespace: kata-system
+spec:
+  chartRef:
+    kind: OCIRepository
+    name: kata-deploy
+  driftDetection:
+    mode: enabled
+  install:
+    strategy:
+      name: RetryOnFailure
+      retryInterval: 5m
+  interval: 30m
+  postRenderers:
+    - kustomize:
+        patches:
+          - target:
+              kind: DaemonSet
+              name: kata-monitor
+            patch: |
+              - op: add
+                path: /spec/template/spec/automountServiceAccountToken
+                value: false
+              - op: add
+                path: /spec/template/spec/containers/0/ports/0/name
+                value: metrics
+  releaseName: kata-deploy
+  targetNamespace: kata-system
+  timeout: 30m
+  upgrade:
+    strategy:
+      name: RetryOnFailure
+      retryInterval: 5m
+  valuesFrom:
+    - kind: ConfigMap
+      name: kata-deploy-values

platform/sandbox-kata/kustomization.yaml

@@ -0,0 +1,9 @@
+---
+apiVersion: kustomize.config.k8s.io/v1beta1
+kind: Kustomization
+resources:
+  - namespace.yaml
+  - repository.yaml
+  - values.yaml
+  - helmrelease.yaml
+  - monitor-scrape.yaml

platform/sandbox-kata/monitor-scrape.yaml

@@ -0,0 +1,17 @@
+---
+apiVersion: operator.victoriametrics.com/v1beta1
+kind: VMPodScrape
+metadata:
+  name: kata-monitor
+  namespace: monitoring
+spec:
+  namespaceSelector:
+    matchNames:
+      - kata-system
+  podMetricsEndpoints:
+    - interval: 30s
+      port: metrics
+  selector:
+    matchLabels:
+      app.kubernetes.io/instance: kata-deploy
+      app.kubernetes.io/name: kata-monitor

platform/sandbox-kata/namespace.yaml

@@ -0,0 +1,5 @@
+---
+apiVersion: v1
+kind: Namespace
+metadata:
+  name: kata-system

platform/sandbox-kata/repository.yaml

@@ -0,0 +1,11 @@
+---
+apiVersion: source.toolkit.fluxcd.io/v1
+kind: OCIRepository
+metadata:
+  name: kata-deploy
+  namespace: kata-system
+spec:
+  interval: 1h
+  ref:
+    digest: sha256:33f102f6db70083de4fc8238af4439c4245a601bcb6a72e49bc80529098aefc0
+  url: oci://ghcr.io/kata-containers/kata-deploy-charts/kata-deploy

platform/sandbox-kata/values.yaml

@@ -0,0 +1,44 @@
+---
+apiVersion: v1
+kind: ConfigMap
+metadata:
+  name: kata-deploy-values
+  namespace: kata-system
+data:
+  values.yaml: |
+    deploymentMode: job
+    k8sDistribution: k3s
+
+    job:
+      parallelism: 1
+
+    snapshotter:
+      setup: []
+
+    shims:
+      disableAll: true
+      clh-runtime-rs:
+        enabled: true
+        dropIn: |
+          [runtime]
+          emptydir_mode = "block-plain"
+
+    defaultShim:
+      amd64: clh-runtime-rs
+
+    runtimeClasses:
+      enabled: true
+      createDefault: false
+
+    node-feature-discovery:
+      enabled: false
+
+    monitor:
+      enabled: true
+      resources:
+        requests:
+          cpu: 25m
+          memory: 64Mi
+        limits:
+          cpu: 200m
+          memory: 192Mi

platform/sandbox-monitoring/README.md

@@ -0,0 +1,16 @@
+# Sandbox 监控
+
+该目录只采集 sandbox Kubernetes 与 workload 指标：
+
+- kubelet 与 cAdvisor；
+- kube-state-metrics；
+- vmagent 自身运行状态；
+- SPIRE Agent attestation、SVID 与连接状态。
+
+不得在 sandbox LXC 内部署 node_exporter。LXC 的 `/proc/stat` 暴露 PVE 宿主 CPU
+视图，会产生重复且语义混合的指标。PVE 宿主监控不属于本目录。
+
+vmagent 为所有远端样本增加 `cluster=sandbox`，并通过仅允许
+`POST /api/v1/write` 的 `metrics-write.ad.ddupan.top` 路由写入 homelab VMSingle。
+Operator 与 workload 拆成两个 Flux Kustomization，确保 VM CRD Ready 后再创建
+VMAgent、VMNodeScrape 和 VMServiceScrape。

platform/sandbox-monitoring/operator/helmrelease.yaml

@@ -0,0 +1,32 @@
+---
+apiVersion: helm.toolkit.fluxcd.io/v2
+kind: HelmRelease
+metadata:
+  name: vm-operator
+  namespace: monitoring
+spec:
+  chart:
+    spec:
+      chart: victoria-metrics-operator
+      interval: 1h
+      sourceRef:
+        kind: HelmRepository
+        name: vm
+      version: 0.66.2
+  install:
+    crds: CreateReplace
+    strategy:
+      name: RetryOnFailure
+      retryInterval: 5m
+  interval: 30m
+  releaseName: vm-operator
+  targetNamespace: monitoring
+  timeout: 10m
+  upgrade:
+    crds: CreateReplace
+    strategy:
+      name: RetryOnFailure
+      retryInterval: 5m
+  valuesFrom:
+    - kind: ConfigMap
+      name: vm-operator-values

platform/sandbox-monitoring/operator/kustomization.yaml

@@ -0,0 +1,8 @@
+---
+apiVersion: kustomize.config.k8s.io/v1beta1
+kind: Kustomization
+resources:
+  - namespace.yaml
+  - repositories.yaml
+  - values.yaml
+  - helmrelease.yaml

platform/sandbox-monitoring/operator/namespace.yaml

@@ -0,0 +1,5 @@
+---
+apiVersion: v1
+kind: Namespace
+metadata:
+  name: monitoring

platform/sandbox-monitoring/operator/repositories.yaml

@@ -0,0 +1,18 @@
+---
+apiVersion: source.toolkit.fluxcd.io/v1
+kind: HelmRepository
+metadata:
+  name: vm
+  namespace: monitoring
+spec:
+  interval: 1h
+  url: https://victoriametrics.github.io/helm-charts/
+---
+apiVersion: source.toolkit.fluxcd.io/v1
+kind: HelmRepository
+metadata:
+  name: prometheus-community
+  namespace: monitoring
+spec:
+  interval: 1h
+  url: https://prometheus-community.github.io/helm-charts

platform/sandbox-monitoring/operator/values.yaml

@@ -0,0 +1,23 @@
+---
+apiVersion: v1
+kind: ConfigMap
+metadata:
+  name: vm-operator-values
+  namespace: monitoring
+data:
+  values.yaml: |
+    crds:
+      plain: true
+      cleanup:
+        enabled: false
+    operator:
+      disable_prometheus_converter: true
+    serviceMonitor:
+      enabled: false
+    resources:
+      requests:
+        cpu: 25m
+        memory: 96Mi
+      limits:
+        cpu: 250m
+        memory: 256Mi

platform/sandbox-monitoring/workloads/kube-state-metrics-scrape.yaml

@@ -0,0 +1,14 @@
+---
+apiVersion: operator.victoriametrics.com/v1beta1
+kind: VMServiceScrape
+metadata:
+  name: kube-state-metrics
+  namespace: monitoring
+spec:
+  endpoints:
+    - interval: 30s
+      port: http
+  selector:
+    matchLabels:
+      app.kubernetes.io/instance: kube-state-metrics
+      app.kubernetes.io/name: kube-state-metrics

platform/sandbox-monitoring/workloads/kube-state-metrics.yaml

@@ -0,0 +1,39 @@
+---
+apiVersion: helm.toolkit.fluxcd.io/v2
+kind: HelmRelease
+metadata:
+  name: kube-state-metrics
+  namespace: monitoring
+spec:
+  chart:
+    spec:
+      chart: kube-state-metrics
+      interval: 1h
+      sourceRef:
+        kind: HelmRepository
+        name: prometheus-community
+        namespace: monitoring
+      version: 8.3.0
+  install:
+    strategy:
+      name: RetryOnFailure
+      retryInterval: 5m
+  interval: 30m
+  releaseName: kube-state-metrics
+  targetNamespace: monitoring
+  timeout: 10m
+  upgrade:
+    strategy:
+      name: RetryOnFailure
+      retryInterval: 5m
+  values:
+    prometheus:
+      monitor:
+        enabled: false
+    resources:
+      requests:
+        cpu: 25m
+        memory: 64Mi
+      limits:
+        cpu: 250m
+        memory: 256Mi

platform/sandbox-monitoring/workloads/kubelet-scrapes.yaml

@@ -0,0 +1,41 @@
+---
+apiVersion: operator.victoriametrics.com/v1beta1
+kind: VMNodeScrape
+metadata:
+  name: kubelet
+  namespace: monitoring
+spec:
+  bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token
+  honorLabels: true
+  honorTimestamps: false
+  interval: 30s
+  scheme: https
+  tlsConfig:
+    caFile: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
+    insecureSkipVerify: true
+  relabelConfigs:
+    - action: labelmap
+      regex: __meta_kubernetes_node_label_(.+)
+    - targetLabel: job
+      replacement: kubelet
+---
+apiVersion: operator.victoriametrics.com/v1beta1
+kind: VMNodeScrape
+metadata:
+  name: cadvisor
+  namespace: monitoring
+spec:
+  bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token
+  honorLabels: true
+  honorTimestamps: false
+  interval: 30s
+  path: /metrics/cadvisor
+  scheme: https
+  tlsConfig:
+    caFile: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
+    insecureSkipVerify: true
+  relabelConfigs:
+    - action: labelmap
+      regex: __meta_kubernetes_node_label_(.+)
+    - targetLabel: job
+      replacement: cadvisor

platform/sandbox-monitoring/workloads/kustomization.yaml

@@ -0,0 +1,9 @@
+---
+apiVersion: kustomize.config.k8s.io/v1beta1
+kind: Kustomization
+resources:
+  - vmagent.yaml
+  - kubelet-scrapes.yaml
+  - kube-state-metrics.yaml
+  - kube-state-metrics-scrape.yaml
+  - spire-agent-scrape.yaml

platform/sandbox-monitoring/workloads/spire-agent-scrape.yaml

@@ -0,0 +1,17 @@
+---
+apiVersion: operator.victoriametrics.com/v1beta1
+kind: VMPodScrape
+metadata:
+  name: spire-agent
+  namespace: monitoring
+spec:
+  namespaceSelector:
+    matchNames:
+      - spire-system
+  podMetricsEndpoints:
+    - interval: 30s
+      port: prom
+  selector:
+    matchLabels:
+      app.kubernetes.io/instance: sandbox-spire
+      app.kubernetes.io/name: agent

platform/sandbox-monitoring/workloads/vmagent.yaml

@@ -0,0 +1,21 @@
+---
+apiVersion: operator.victoriametrics.com/v1beta1
+kind: VMAgent
+metadata:
+  name: sandbox
+  namespace: monitoring
+spec:
+  externalLabels:
+    cluster: sandbox
+  remoteWrite:
+    - url: https://metrics-write.ad.ddupan.top/api/v1/write
+  replicaCount: 1
+  resources:
+    requests:
+      cpu: 50m
+      memory: 128Mi
+    limits:
+      cpu: 500m
+      memory: 512Mi
+  scrapeInterval: 30s
+  selectAllByDefault: true

platform/sandbox-opensandbox-pools/kustomization.yaml

@@ -0,0 +1,4 @@
+apiVersion: kustomize.config.k8s.io/v1beta1
+kind: Kustomization
+resources:
+  - pools.yaml

platform/sandbox-opensandbox-pools/pools.yaml

@@ -0,0 +1,123 @@
+---
+apiVersion: sandbox.opensandbox.io/v1alpha1
+kind: Pool
+metadata:
+  name: ci-pod
+  namespace: opensandbox
+spec:
+  template:
+    metadata:
+      labels:
+        ci.ddupan.top/backend: pod
+    spec:
+      containers:
+        - name: sandbox
+          image: sandbox-registry.cn-zhangjiakou.cr.aliyuncs.com/opensandbox/code-interpreter:v1.1.0
+          command: [/opt/opensandbox/task-executor]
+          args: [-listen-addr=0.0.0.0:5758, -log-dir=/tmp]
+          env:
+            - name: SANDBOX_MAIN_CONTAINER
+              value: sandbox
+            - name: EXECD_ENVS
+              value: /opt/opensandbox/.env
+            - name: EXECD
+              value: /opt/opensandbox/execd
+          resources:
+            requests:
+              cpu: 100m
+              memory: 256Mi
+            limits:
+              cpu: "2"
+              memory: 4Gi
+          volumeMounts:
+            - name: opensandbox-bin
+              mountPath: /opt/opensandbox
+            - name: sandbox-storage
+              mountPath: /var/lib/sandbox
+      initContainers:
+        - name: task-executor-installer
+          image: sandbox-registry.cn-zhangjiakou.cr.aliyuncs.com/opensandbox/task-executor:v0.1.0
+          command: [/bin/sh, -c]
+          args: [cp /workspace/server /opt/opensandbox/task-executor && chmod 0755 /opt/opensandbox/task-executor]
+          volumeMounts:
+            - name: opensandbox-bin
+              mountPath: /opt/opensandbox
+        - name: execd-installer
+          image: sandbox-registry.cn-zhangjiakou.cr.aliyuncs.com/opensandbox/execd:v1.0.22
+          command: [/bin/sh, -c]
+          args: [cp ./execd /opt/opensandbox/execd && cp ./bootstrap.sh /opt/opensandbox/bootstrap.sh && chmod 0755 /opt/opensandbox/execd /opt/opensandbox/bootstrap.sh]
+          volumeMounts:
+            - name: opensandbox-bin
+              mountPath: /opt/opensandbox
+      volumes:
+        - name: opensandbox-bin
+          emptyDir: {}
+        - name: sandbox-storage
+          emptyDir: {}
+  capacitySpec:
+    bufferMax: 1
+    bufferMin: 0
+    poolMax: 4
+    poolMin: 0
+---
+apiVersion: sandbox.opensandbox.io/v1alpha1
+kind: Pool
+metadata:
+  name: ci-vm
+  namespace: opensandbox
+spec:
+  template:
+    metadata:
+      labels:
+        ci.ddupan.top/backend: vm
+    spec:
+      runtimeClassName: kata-clh-runtime-rs
+      containers:
+        - name: sandbox
+          image: sandbox-registry.cn-zhangjiakou.cr.aliyuncs.com/opensandbox/code-interpreter:v1.1.0
+          command: [/opt/opensandbox/task-executor]
+          args: [-listen-addr=0.0.0.0:5758, -log-dir=/tmp]
+          env:
+            - name: SANDBOX_MAIN_CONTAINER
+              value: sandbox
+            - name: EXECD_ENVS
+              value: /opt/opensandbox/.env
+            - name: EXECD
+              value: /opt/opensandbox/execd
+          resources:
+            requests:
+              cpu: 250m
+              memory: 512Mi
+            limits:
+              cpu: "4"
+              memory: 8Gi
+          volumeMounts:
+            - name: opensandbox-bin
+              mountPath: /opt/opensandbox
+            - name: sandbox-storage
+              mountPath: /var/lib/sandbox
+      initContainers:
+        - name: task-executor-installer
+          image: sandbox-registry.cn-zhangjiakou.cr.aliyuncs.com/opensandbox/task-executor:v0.1.0
+          command: [/bin/sh, -c]
+          args: [cp /workspace/server /opt/opensandbox/task-executor && chmod 0755 /opt/opensandbox/task-executor]
+          volumeMounts:
+            - name: opensandbox-bin
+              mountPath: /opt/opensandbox
+        - name: execd-installer
+          image: sandbox-registry.cn-zhangjiakou.cr.aliyuncs.com/opensandbox/execd:v1.0.22
+          command: [/bin/sh, -c]
+          args: [cp ./execd /opt/opensandbox/execd && cp ./bootstrap.sh /opt/opensandbox/bootstrap.sh && chmod 0755 /opt/opensandbox/execd /opt/opensandbox/bootstrap.sh]
+          volumeMounts:
+            - name: opensandbox-bin
+              mountPath: /opt/opensandbox
+      volumes:
+        - name: opensandbox-bin
+          emptyDir: {}
+        - name: sandbox-storage
+          emptyDir: {}
+  capacitySpec:
+    bufferMax: 1
+    bufferMin: 0
+    poolMax: 2
+    poolMin: 0

platform/sandbox-opensandbox/README.md

@@ -0,0 +1,19 @@
+# OpenSandbox
+
+Flux installs the upstream all-in-one OpenSandbox chart pinned to
+`helm/opensandbox/0.2.2` (`8f01e935`). The API is cluster-internal and intentionally runs a
+single replica until shared server state and HA behaviour have been validated.
+
+`ci-pod` uses `runc`; `ci-vm` uses the separately managed
+`kata-clh-runtime-rs` RuntimeClass. Both Pools start at zero and create capacity
+on demand. They currently use the upstream interpreter image to validate the
+Lifecycle API and Pool allocation independently of the CI scheduler cutover.
+
+The dynamic runner worker, runner image, guest-local SPIRE Agent and Docker
+sidecar are introduced only after this layer is Ready. In particular, do not
+mount the host SPIFFE CSI socket into `ci-vm`: Unix sockets do not cross the
+Kata VM boundary.
+
+Smoke test both backends through the same API by creating sandboxes with
+`extensions.poolRef` set to `ci-pod` and `ci-vm`, then confirm their
+BatchSandboxes, Pods and VMMs disappear after deletion.

platform/sandbox-opensandbox/helmrelease.yaml

@@ -0,0 +1,31 @@
+apiVersion: helm.toolkit.fluxcd.io/v2
+kind: HelmRelease
+metadata:
+  name: opensandbox
+  namespace: opensandbox-system
+spec:
+  chart:
+    spec:
+      chart: ./kubernetes/charts/opensandbox
+      interval: 1h
+      reconcileStrategy: Revision
+      sourceRef:
+        kind: GitRepository
+        name: opensandbox
+  driftDetection:
+    mode: enabled
+  install:
+    strategy:
+      name: RetryOnFailure
+      retryInterval: 5m
+  interval: 30m
+  releaseName: opensandbox
+  targetNamespace: opensandbox-system
+  timeout: 15m
+  upgrade:
+    strategy:
+      name: RetryOnFailure
+      retryInterval: 5m
+  valuesFrom:
+    - kind: ConfigMap
+      name: opensandbox-values

platform/sandbox-opensandbox/kustomization.yaml

@@ -0,0 +1,7 @@
+apiVersion: kustomize.config.k8s.io/v1beta1
+kind: Kustomization
+resources:
+  - namespaces.yaml
+  - repository.yaml
+  - values.yaml
+  - helmrelease.yaml

platform/sandbox-opensandbox/namespaces.yaml

@@ -0,0 +1,10 @@
+---
+apiVersion: v1
+kind: Namespace
+metadata:
+  name: opensandbox-system
+---
+apiVersion: v1
+kind: Namespace
+metadata:
+  name: opensandbox

platform/sandbox-opensandbox/repository.yaml

@@ -0,0 +1,11 @@
+apiVersion: source.toolkit.fluxcd.io/v1
+kind: GitRepository
+metadata:
+  name: opensandbox
+  namespace: opensandbox-system
+spec:
+  interval: 1h
+  ref:
+    tag: helm/opensandbox/0.2.2
+  timeout: 60s
+  url: https://github.com/alibaba/OpenSandbox.git

platform/sandbox-opensandbox/values.yaml

@@ -0,0 +1,62 @@
+apiVersion: v1
+kind: ConfigMap
+metadata:
+  name: opensandbox-values
+  namespace: opensandbox-system
+data:
+  values.yaml: |
+    opensandbox-controller:
+      controller:
+        logLevel: info
+        replicaCount: 1
+        metrics:
+          enabled: true
+          secure: false
+          port: 8080
+        resources:
+          requests:
+            cpu: 25m
+            memory: 64Mi
+          limits:
+            cpu: 500m
+            memory: 256Mi
+
+    opensandbox-server:
+      server:
+        replicaCount: 1
+        resources:
+          requests:
+            cpu: 100m
+            memory: 256Mi
+          limits:
+            cpu: "1"
+            memory: 1Gi
+      configToml: |
+        [server]
+        host = "0.0.0.0"
+        port = 80
+        api_key = ""
+
+        [log]
+        level = "INFO"
+
+        [runtime]
+        type = "kubernetes"
+        execd_image = "sandbox-registry.cn-zhangjiakou.cr.aliyuncs.com/opensandbox/execd:v1.0.22"
+
+        [kubernetes]
+        kubeconfig_path = ""
+        namespace = "opensandbox"
+        informer_enabled = true
+        informer_resync_seconds = 300
+        informer_watch_timeout_seconds = 60
+        snapshot_create_timeout_seconds = 900
+        workload_provider = "batchsandbox"
+        batchsandbox_template_file = "/etc/opensandbox/example.batchsandbox-template.yaml"
+
+        [egress]
+        image = "sandbox-registry.cn-zhangjiakou.cr.aliyuncs.com/opensandbox/egress:v1.1.6"
+        mode = "dns+nft"
+
+    opensandbox-node-agent:
+      enabled: false

platform/sandbox-spire/agents/kustomization.yaml

@@ -0,0 +1,7 @@
+---
+apiVersion: kustomize.config.k8s.io/v1beta1
+kind: Kustomization
+resources:
+  - values.yaml
+  - release.yaml
+  - smoke-identity.yaml

platform/sandbox-spire/agents/release.yaml

@@ -0,0 +1,35 @@
+---
+apiVersion: helm.toolkit.fluxcd.io/v2
+kind: HelmRelease
+metadata:
+  name: sandbox-spire
+  namespace: spire-mgmt
+spec:
+  chart:
+    spec:
+      chart: spire
+      interval: 1h
+      sourceRef:
+        kind: HelmRepository
+        name: spiffe-hardened
+      version: 0.30.2
+  dependsOn:
+    - name: spire-crds
+      namespace: spire-mgmt
+  driftDetection:
+    mode: enabled
+  install:
+    strategy:
+      name: RetryOnFailure
+      retryInterval: 5m
+  interval: 30m
+  releaseName: sandbox-spire
+  targetNamespace: spire-mgmt
+  timeout: 15m
+  upgrade:
+    strategy:
+      name: RetryOnFailure
+      retryInterval: 5m
+  valuesFrom:
+    - kind: ConfigMap
+      name: sandbox-spire-values

platform/sandbox-spire/agents/smoke-identity.yaml

@@ -0,0 +1,28 @@
+---
+apiVersion: v1
+kind: Namespace
+metadata:
+  name: spire-smoke
+---
+apiVersion: v1
+kind: ServiceAccount
+metadata:
+  name: spire-smoke
+  namespace: spire-smoke
+---
+apiVersion: spire.spiffe.io/v1alpha1
+kind: ClusterSPIFFEID
+metadata:
+  name: sandbox-spire-smoke
+spec:
+  className: spire-mgmt-spire
+  namespaceSelector:
+    matchLabels:
+      kubernetes.io/metadata.name: spire-smoke
+  podSelector:
+    matchLabels:
+      app.kubernetes.io/name: spire-smoke
+  spiffeIDTemplate: spiffe://{{ .TrustDomain }}/sandbox/smoke
+  workloadSelectorTemplates:
+    - k8s:ns:spire-smoke
+    - k8s:sa:spire-smoke

platform/sandbox-spire/agents/values.yaml

@@ -0,0 +1,86 @@
+---
+apiVersion: v1
+kind: ConfigMap
+metadata:
+  name: sandbox-spire-values
+  namespace: spire-mgmt
+data:
+  values.yaml: |
+    global:
+      k8s:
+        clusterDomain: cluster.local
+      spire:
+        bundleConfigMap: spire-bundle
+        clusterName: sandbox
+        trustDomain: ddupan.top
+        namespaces:
+          create: false
+          system:
+            name: spire-system
+          server:
+            name: spire-server
+        recommendations:
+          enabled: true
+          namespaceLayout: true
+          namespacePSS: true
+          priorityClassName: true
+          strictMode: true
+          securityContexts: true
+          prometheus: false
+
+    spire-server:
+      enabled: false
+
+    spire-agent:
+      enabled: true
+      serviceAccount:
+        name: spire-agent
+      server:
+        address: spire-server.ad.ddupan.top
+        port: 8081
+      nodeAttestor:
+        k8sPSAT:
+          enabled: true
+      workloadAttestors:
+        k8s:
+          enabled: true
+        unix:
+          enabled: false
+      telemetry:
+        prometheus:
+          enabled: true
+          podMonitor:
+            enabled: false
+      resources:
+        requests:
+          cpu: 25m
+          memory: 64Mi
+        limits:
+          cpu: 250m
+          memory: 192Mi
+
+    spiffe-csi-driver:
+      enabled: true
+      resources:
+        requests:
+          cpu: 10m
+          memory: 32Mi
+        limits:
+          cpu: 100m
+          memory: 96Mi
+
+    spiffe-oidc-discovery-provider:
+      enabled: false
+
+    upstream:
+      enabled: false
+    tornjak-frontend:
+      enabled: false
+    spire-identity-exchange:
+      enabled: false
+    spike-keeper:
+      enabled: false
+    spike-nexus:
+      enabled: false
+    spike-pilot:
+      enabled: false

platform/sandbox-spire/bootstrap/controller-manager.yaml

@@ -0,0 +1,162 @@
+---
+apiVersion: v1
+kind: ServiceAccount
+metadata:
+  name: spire-controller-manager
+  namespace: spire-system
+---
+apiVersion: v1
+kind: Secret
+metadata:
+  name: spire-controller-manager-token
+  namespace: spire-system
+  annotations:
+    kubernetes.io/service-account.name: spire-controller-manager
+type: kubernetes.io/service-account-token
+---
+apiVersion: rbac.authorization.k8s.io/v1
+kind: ClusterRole
+metadata:
+  name: spire-controller-manager
+rules:
+  - apiGroups:
+      - ""
+    resources:
+      - endpoints
+      - namespaces
+      - nodes
+      - pods
+    verbs:
+      - get
+      - list
+      - watch
+  - apiGroups:
+      - spire.spiffe.io
+    resources:
+      - clusterfederatedtrustdomains
+      - clusterspiffeids
+      - clusterstaticentries
+    verbs:
+      - create
+      - delete
+      - get
+      - list
+      - patch
+      - update
+      - watch
+  - apiGroups:
+      - spire.spiffe.io
+    resources:
+      - clusterfederatedtrustdomains/finalizers
+      - clusterspiffeids/finalizers
+      - clusterstaticentries/finalizers
+    verbs:
+      - update
+  - apiGroups:
+      - spire.spiffe.io
+    resources:
+      - clusterfederatedtrustdomains/status
+      - clusterspiffeids/status
+      - clusterstaticentries/status
+    verbs:
+      - get
+      - patch
+      - update
+---
+apiVersion: rbac.authorization.k8s.io/v1
+kind: ClusterRoleBinding
+metadata:
+  name: spire-controller-manager
+roleRef:
+  apiGroup: rbac.authorization.k8s.io
+  kind: ClusterRole
+  name: spire-controller-manager
+subjects:
+  - kind: ServiceAccount
+    name: spire-controller-manager
+    namespace: spire-system
+---
+apiVersion: rbac.authorization.k8s.io/v1
+kind: Role
+metadata:
+  name: spire-bundle-publisher
+  namespace: spire-system
+rules:
+  - apiGroups:
+      - ""
+    resources:
+      - configmaps
+    verbs:
+      - create
+      - delete
+      - get
+      - list
+      - patch
+      - update
+      - watch
+---
+apiVersion: rbac.authorization.k8s.io/v1
+kind: RoleBinding
+metadata:
+  name: spire-bundle-publisher
+  namespace: spire-system
+roleRef:
+  apiGroup: rbac.authorization.k8s.io
+  kind: Role
+  name: spire-bundle-publisher
+subjects:
+  - kind: ServiceAccount
+    name: spire-controller-manager
+    namespace: spire-system
+---
+apiVersion: rbac.authorization.k8s.io/v1
+kind: Role
+metadata:
+  name: spire-controller-manager-leader-election
+  namespace: spire-server
+rules:
+  - apiGroups:
+      - ""
+    resources:
+      - configmaps
+    verbs:
+      - create
+      - delete
+      - get
+      - list
+      - patch
+      - update
+      - watch
+  - apiGroups:
+      - coordination.k8s.io
+    resources:
+      - leases
+    verbs:
+      - create
+      - delete
+      - get
+      - list
+      - patch
+      - update
+      - watch
+  - apiGroups:
+      - ""
+    resources:
+      - events
+    verbs:
+      - create
+      - patch
+---
+apiVersion: rbac.authorization.k8s.io/v1
+kind: RoleBinding
+metadata:
+  name: spire-controller-manager-leader-election
+  namespace: spire-server
+roleRef:
+  apiGroup: rbac.authorization.k8s.io
+  kind: Role
+  name: spire-controller-manager-leader-election
+subjects:
+  - kind: ServiceAccount
+    name: spire-controller-manager
+    namespace: spire-system

platform/sandbox-spire/bootstrap/crds.yaml

@@ -0,0 +1,31 @@
+---
+apiVersion: helm.toolkit.fluxcd.io/v2
+kind: HelmRelease
+metadata:
+  name: spire-crds
+  namespace: spire-mgmt
+spec:
+  chart:
+    spec:
+      chart: spire-crds
+      interval: 1h
+      sourceRef:
+        kind: HelmRepository
+        name: spiffe-hardened
+      version: 0.6.1
+  driftDetection:
+    mode: enabled
+  install:
+    crds: CreateReplace
+    strategy:
+      name: RetryOnFailure
+      retryInterval: 5m
+  interval: 30m
+  releaseName: spire-crds
+  targetNamespace: spire-mgmt
+  timeout: 10m
+  upgrade:
+    crds: CreateReplace
+    strategy:
+      name: RetryOnFailure
+      retryInterval: 5m

platform/sandbox-spire/bootstrap/kustomization.yaml

@@ -0,0 +1,9 @@
+---
+apiVersion: kustomize.config.k8s.io/v1beta1
+kind: Kustomization
+resources:
+  - namespaces.yaml
+  - repository.yaml
+  - crds.yaml
+  - token-reviewer.yaml
+  - controller-manager.yaml

platform/sandbox-spire/bootstrap/namespaces.yaml

@@ -0,0 +1,15 @@
+---
+apiVersion: v1
+kind: Namespace
+metadata:
+  name: spire-mgmt
+---
+apiVersion: v1
+kind: Namespace
+metadata:
+  name: spire-system
+---
+apiVersion: v1
+kind: Namespace
+metadata:
+  name: spire-server

platform/sandbox-spire/bootstrap/repository.yaml

@@ -0,0 +1,9 @@
+---
+apiVersion: source.toolkit.fluxcd.io/v1
+kind: HelmRepository
+metadata:
+  name: spiffe-hardened
+  namespace: spire-mgmt
+spec:
+  interval: 1h
+  url: https://spiffe.github.io/helm-charts-hardened/

platform/sandbox-spire/bootstrap/token-reviewer.yaml

@@ -0,0 +1,51 @@
+---
+apiVersion: v1
+kind: ServiceAccount
+metadata:
+  name: spire-server-token-reviewer
+  namespace: spire-system
+---
+apiVersion: v1
+kind: Secret
+metadata:
+  name: spire-server-token-reviewer-token
+  namespace: spire-system
+  annotations:
+    kubernetes.io/service-account.name: spire-server-token-reviewer
+type: kubernetes.io/service-account-token
+---
+apiVersion: rbac.authorization.k8s.io/v1
+kind: ClusterRole
+metadata:
+  name: spire-server-token-reviewer
+rules:
+  - apiGroups:
+      - authentication.k8s.io
+    resources:
+      - tokenreviews
+    verbs:
+      - get
+      - list
+      - watch
+      - create
+  - apiGroups:
+      - ""
+    resources:
+      - nodes
+      - pods
+    verbs:
+      - get
+      - list
+---
+apiVersion: rbac.authorization.k8s.io/v1
+kind: ClusterRoleBinding
+metadata:
+  name: spire-server-token-reviewer
+roleRef:
+  apiGroup: rbac.authorization.k8s.io
+  kind: ClusterRole
+  name: spire-server-token-reviewer
+subjects:
+  - kind: ServiceAccount
+    name: spire-server-token-reviewer
+    namespace: spire-system

platform/spire/README.md

@@ -11,7 +11,8 @@ Authelia 提供；SPIRE 不替代人类 OIDC，也不承担目标服务的资源
 Flux 安装 SPIFFE hardened charts：
 
 - `spire-crds` `0.6.1`；
-- `spire` `0.30.2`（SPIRE `1.15.3`）；
+- 内部 fork 的 `spire` `0.30.2-ddupan.1`（SPIRE `1.15.3`），固定 Git tag
+  `spire-0.30.2-ddupan.1`；
 - SPIRE Server、Agent、Controller Manager、SPIFFE CSI Driver；
 - OIDC Discovery Provider。
 
@@ -19,6 +20,17 @@ Flux 安装 SPIFFE hardened charts：
 API 或 Broker API。Trust domain 是 `ddupan.top`，Kubernetes cluster name 是
 `homelab`。
 
+同一 Server 也接受 cluster name 为 `sandbox` 的 external PSAT attestation。SPIRE gRPC
+只通过内网 `spire-server.ad.ddupan.top:8081` 暴露；external PSAT、external
+controller-manager 与 bundle publisher 使用由 sandbox Ansible bootstrap 的独立、受限
+kubeconfig。Sandbox 不运行第二套 Server 或 OIDC Provider。
+
+内部 fork 仅在上游 `spire-0.30.2` 基础上暴露
+`use_pod_uid_for_agent_id`。现有 `sandbox` profile 保持 node UID 模式，供 DaemonSet
+Agent 使用；独立的 `sandbox-kata` profile 复用同一 kubeconfig，但启用 Pod UID 模式，
+供每个 Kata guest 内的临时 Agent 使用。不得把现有 `sandbox` profile 切换为 Pod UID，
+否则会改变常驻 Agent 的 parent ID。
+
 ## PostgreSQL bootstrap
 
 SPIRE registration datastore 使用共享 CloudNativePG：

platform/spire/RUNBOOK.md

@@ -41,7 +41,7 @@ SPIRE Server（trust domain: ddupan.top）
 
 | 项目 | 当前值 |
 |---|---|
-| SPIRE chart | `0.30.2` |
+| SPIRE chart | `0.30.2-ddupan.1`（内部 fork，基于 `0.30.2`） |
 | SPIRE | `1.15.3` |
 | SPIRE CRDs chart | `0.6.1` |
 | trust domain | `ddupan.top` |

platform/spire/gitrepository-fork.yaml

@@ -0,0 +1,11 @@
+apiVersion: source.toolkit.fluxcd.io/v1
+kind: GitRepository
+metadata:
+  name: spiffe-hardened-fork
+  namespace: spire-mgmt
+spec:
+  interval: 1h
+  ref:
+    tag: spire-0.30.2-ddupan.1
+  timeout: 60s
+  url: http://gitea-http.gitea.svc.cluster.local:3000/panxiao81/helm-charts-hardened.git

platform/spire/helmrelease.yaml

@@ -6,12 +6,12 @@ metadata:
 spec:
   chart:
     spec:
-      chart: spire
+      chart: ./charts/spire
       interval: 1h
+      reconcileStrategy: Revision
       sourceRef:
-        kind: HelmRepository
+        kind: GitRepository
-        name: spiffe-hardened
+        name: spiffe-hardened-fork
-      version: 0.30.2
   dependsOn:
     - name: spire-crds
       namespace: spire-mgmt

platform/spire/kustomization.yaml

@@ -12,6 +12,7 @@ configMapGenerator:
 resources:
   - namespaces.yaml
   - helmrepository.yaml
+  - gitrepository-fork.yaml
   - helmrelease-crds.yaml
   - helmrelease.yaml
   - httproute.yaml

platform/spire/values.yaml

@@ -30,6 +30,47 @@ spire-server:
   kind: statefulset
   replicaCount: 1
   auditLogEnabled: true
+  service:
+    type: LoadBalancer
+    port: 8081
+    loadBalancerIP: 192.168.10.127
+  kubeConfigs:
+    sandbox:
+      externalSecret:
+        name: spire-external-kubeconfigs
+        key: sandbox
+    sandbox-controller:
+      externalSecret:
+        name: spire-external-kubeconfigs
+        key: sandbox-controller
+  nodeAttestor:
+    externalK8sPSAT:
+      enabled: true
+      clusters:
+        sandbox:
+          kubeConfigName: sandbox
+          serviceAccountAllowList:
+            - spire-system:spire-agent
+        sandbox-kata:
+          kubeConfigName: sandbox
+          serviceAccountAllowList:
+            - spire-smoke:spire-smoke
+          usePodUIDForAgentID: true
+  externalControllerManagers:
+    enabled: true
+    clusters:
+      sandbox:
+        kubeConfigName: sandbox-controller
+  bundlePublisher:
+    externalK8sConfigMap:
+      enabled: true
+      clusters:
+        sandbox:
+          kubeConfigName: sandbox-controller
+          namespace: spire-system
+          configMapName: spire-bundle
+          configMapKey: bundle.spiffe
+          format: spiffe
   persistence:
     # PostgreSQL stores registrations, but the disk KeyManager still needs durable
     # storage for the trust-domain signing keys.
@@ -65,6 +106,11 @@ spire-server:
           enabled: false
 
 spire-agent:
+  server:
+    # Keep the Agent endpoint aligned with spire-server.service.port. The
+    # chart defaults this to 443, which only remained unnoticed while the
+    # Agent's pre-upgrade gRPC connection stayed alive.
+    port: 8081
   nodeAttestor:
     k8sPSAT:
       enabled: true