Author SHA1 Message Date
panxiao81 54dd52f0ed 声明式管理 Proxmox API 对象与 Samba DNS 上游
yaml / yaml (pull_request) Failing after 25s
terraform / validate (pull_request) Successful in 57s
ansible / collection-test (pull_request) Successful in 1m34s
ansible / lint (pull_request) Successful in 4m1s
2026-09-17 13:18:59 +00:00
panxiao81 76ec49ff48 恢复 OCI Terraform 与站点网络 IaC 2026-09-17 13:18:00 +00:00
panxiao81 00671e1729 统一 DN42 DNS 与宿主 Docker 地址池 2026-09-17 13:17:15 +00:00
84 changed files with 12 additions and 2669 deletions
+1 -1
View File
@@ -58,4 +58,4 @@ networks:
default:
ipam:
config:
- subnet: 172.28.0.0/24
- subnet: 172.28.0.0/24
+1 -1
View File
@@ -18,4 +18,4 @@ networks:
default:
ipam:
config:
- subnet: 172.28.1.0/24
- subnet: 172.28.1.0/24
+1 -2
View File
@@ -50,7 +50,6 @@ sudo k3s kubectl -n flux-system get gitrepositories,kustomizations
- VictoriaMetrics Operator 已固定现有 chart `0.66.2` 并完成分阶段 Flux HelmRelease
接管;Metrics、Logs、Traces 与 Grafana 也已统一完成 Flux 接管;
- External Secrets Operator 已固定 chart `2.8.0` 并完成分阶段接管;
- SPIRE 已按 hardened chart 内部 fork `0.30.2-ddupan.1`(基于上游 `0.30.2`,SPIRE
`1.15.3`)声明,使用共享
- SPIRE 已按官方 hardened chart `0.30.2`(SPIRE `1.15.3`)声明,使用共享
PostgreSQL 与独立 signing-key PVC;首次上线和 OpenBao JWT-SVID PoC 尚待合并后验证;
- root Kustomization 与所有 brownfield 子 Kustomization 继续保持 `prune: false`。
-61
View File
@@ -1,61 +0,0 @@
# Sandbox 集群
这里是 OpenSandbox、CI 和 AI Agent workload 所在双节点 k3s 集群的 Flux
reconciliation 入口。LXC、PostgreSQL、K3s、固定版本的 Flux controllers 与 root
sync 由 `infrastructure/sandbox-cluster/` 中的 Ansible 管理;本目录只组合集群内
workload。
Flux 通过 `https://git.ddupan.top/panxiao81/homelab-infra.git` 读取公开仓库。
Ansible 将 homelab CA 注入 `GitRepository/flux-system` 引用的同名 Secret,不使用
长期 Git 凭据。root Kustomization 从 `./clusters/sandbox` 开始 reconciliation,
初始保持 `prune: false`。
Root bootstrap 已完成。后续按依赖顺序分别引入:
1. 监控 CRD、kube-state-metrics 以及 kubelet/cAdvisor 抓取配置;
2. SPIRE Agent、SPIFFE CSI Driver 与 workload registration;
3. Kata Containers、`block-plain` RuntimeClass;
4. OpenSandbox operator/server 及 `ci-pod`、`ci-vm` Pools。
每一阶段单独合并并等待对应 Flux Kustomization Ready,不在 bootstrap 时一次性部署。
第一阶段监控拆为 `monitoring-operator` 与依赖它的 `monitoring`,防止 VM CR 在
VictoriaMetrics Operator CRD Ready 前进入 reconciliation。
SPIRE 阶段先由 `spire-bootstrap` 安装 CRD,并声明按上游 k8s_psat Server plugin
要求收窄的 reviewer:它可以调用 TokenReview,并只读查询用于证明的 Pod 与 Node。
Agent ServiceAccount 留给后续 HelmRelease 创建,避免两个声明方争夺同一资源。随后运行
`infrastructure/sandbox-cluster/ansible/spire-bootstrap.yml`:playbook 从 sandbox
读取 reviewer token,在内存中组成受限 kubeconfig,再通过 stdin reconcile 到 central
集群的 `spire-server/spire-external-kubeconfigs` Secret。凭据不写入仓库、日志或控制机
文件;该 Secret 准备完成后,才能启用 central external PSAT/controller-manager 和
sandbox Agent/CSI。
External controller-manager 使用独立的 `spire-controller-manager` ServiceAccount;其
RBAC 与上游 controller-manager 所需权限一致,用于读取 workload selectors、维护
SPIFFE CR status/finalizer 和 leader election。它不复用只允许 TokenReview 的 Server
reviewer。Ansible 将两份 kubeconfig 写入同一个 central Secret 的不同 key,便于 central
chart 分别绑定 `sandbox` 与 `sandbox-controller`。
Central SPIRE Server 通过内网 `spire-server.ad.ddupan.top:8081` 接收 sandbox Agent
attestation。Server 使用 external bundle publisher 持续维护 sandbox
`spire-system/spire-bundle`,Agent 不固定或复制 trust bundle。Sandbox HelmRelease
显式关闭 Server 与 OIDC Provider,只部署 Agent DaemonSet 和 SPIFFE CSI Driver;因此
不会产生第二个 trust root。
`spire-smoke` namespace、ServiceAccount 和 `sandbox-spire-smoke` ClusterSPIFFEID 是
普通 Pod 与后续 Kata guest 的回归夹具,稳定身份为
`spiffe://ddupan.top/sandbox/smoke`。测试 Pod 临时创建并在验收后删除,身份声明保留。
Kata 阶段使用官方 4.1.0 `kata-deploy` chart 的短生命周期 `job` 模式,逐节点安装并
重启 K3s。只启用 `kata-clh-runtime-rs`,不创建默认 `kata` 别名;该 handler 的
`emptyDir` 固定使用 `block-plain`,为 Docker/BuildKit overlay2 与 kind 提供 guest
内块设备文件系统。详细限制与上线验收见 `platform/sandbox-kata/README.md`。
## 监控边界
这里只管理 sandbox LXC 内的 Kubernetes 监控,不负责 PVE 宿主监控。LXC 与宿主共享
内核,即使 lxcfs 虚拟化了内存和 uptime,容器内 `/proc/stat` 仍是宿主 CPU 视图;
在 LXC 内运行 node_exporter 会生成混合语义并重复采集宿主指标,因此禁止部署。
Sandbox 节点与 workload 指标来自 kubelet/cAdvisor 和 kube-state-metrics;K3s 或 LXC
特有但上述接口未覆盖的指标,应使用目标明确的 collector,不以 node_exporter 补齐。
-18
View File
@@ -1,18 +0,0 @@
---
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
name: kata
namespace: flux-system
spec:
dependsOn:
- name: monitoring-operator
- name: spire-agents
interval: 10m
path: ./platform/sandbox-kata
prune: true
sourceRef:
kind: GitRepository
name: flux-system
timeout: 35m
wait: true
@@ -1,15 +0,0 @@
---
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
name: monitoring-operator
namespace: flux-system
spec:
interval: 10m
path: ./platform/sandbox-monitoring/operator
prune: true
sourceRef:
kind: GitRepository
name: flux-system
timeout: 10m
wait: true
-17
View File
@@ -1,17 +0,0 @@
---
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
name: monitoring
namespace: flux-system
spec:
dependsOn:
- name: monitoring-operator
interval: 10m
path: ./platform/sandbox-monitoring/workloads
prune: true
sourceRef:
kind: GitRepository
name: flux-system
timeout: 10m
wait: true
@@ -1,17 +0,0 @@
---
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
name: opensandbox-pools
namespace: flux-system
spec:
dependsOn:
- name: opensandbox
interval: 10m
path: ./platform/sandbox-opensandbox-pools
prune: true
sourceRef:
kind: GitRepository
name: flux-system
timeout: 20m
wait: true
-18
View File
@@ -1,18 +0,0 @@
---
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
name: opensandbox
namespace: flux-system
spec:
dependsOn:
- name: kata
- name: spire-agents
interval: 10m
path: ./platform/sandbox-opensandbox
prune: true
sourceRef:
kind: GitRepository
name: flux-system
timeout: 20m
wait: true
-18
View File
@@ -1,18 +0,0 @@
---
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
name: spire-agents
namespace: flux-system
spec:
dependsOn:
- name: spire-bootstrap
- name: monitoring-operator
interval: 10m
path: ./platform/sandbox-spire/agents
prune: true
sourceRef:
kind: GitRepository
name: flux-system
timeout: 15m
wait: true
@@ -1,15 +0,0 @@
---
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
name: spire-bootstrap
namespace: flux-system
spec:
interval: 10m
path: ./platform/sandbox-spire/bootstrap
prune: true
sourceRef:
kind: GitRepository
name: flux-system
timeout: 10m
wait: true
-11
View File
@@ -1,11 +0,0 @@
---
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- apps/monitoring-operator.yaml
- apps/monitoring.yaml
- apps/spire-bootstrap.yaml
- apps/spire-agents.yaml
- apps/kata.yaml
- apps/opensandbox.yaml
- apps/opensandbox-pools.yaml
+2 -4
View File
@@ -15,12 +15,10 @@ homelab_dns:
- { zone: ad.ddupan.top, name: sandbox-k8s, type: A, values: [10.60.0.13] }
- { zone: ad.ddupan.top, name: retrolab, type: A, values: [10.60.0.10] }
- { zone: ad.ddupan.top, name: grafana, type: A, values: [192.168.10.127] }
- { zone: ad.ddupan.top, name: metrics-write, type: A, values: [192.168.10.127] }
- { zone: ad.ddupan.top, name: netbox, type: A, values: [192.168.10.127] }
- { zone: ad.ddupan.top, name: nats, type: A, values: [192.168.10.127] }
- { zone: ad.ddupan.top, name: s3, type: A, values: [192.168.10.127] }
- { zone: ad.ddupan.top, name: spire-oidc, type: A, values: [192.168.10.127] }
- { zone: ad.ddupan.top, name: spire-server, type: A, values: [192.168.10.127] }
- { zone: ad.ddupan.top, name: zot, type: A, values: [192.168.10.127] }
- { zone: ad.ddupan.top, name: zot-push, type: A, values: [192.168.10.127] }
@@ -62,14 +60,14 @@ homelab_dns:
- name: oci-arm.ddupan.top
type: A
values:
- 129.225.138.179
- 129.225.138.179
proxied: false
ttl: 300
terraform: { managed: false }
- name: oci-amd.ddupan.top
type: A
values:
- 129.225.176.134
- 129.225.176.134
proxied: false
ttl: 300
terraform: { managed: false }
@@ -77,14 +77,3 @@ vyos_ospf_networks:
vyos_postgresql_listen_address: "10.60.0.1"
vyos_postgresql_port: 5432
vyos_postgresql_primary_address: "10.60.0.11"
# Stable API endpoint for the two-node sandbox K3s control plane. HAProxy owns
# this /32 on the existing labnet interface and performs only TCP health checks.
vyos_sandbox_k3s_api_address: "10.60.0.13"
vyos_sandbox_k3s_api_interface: eth1
vyos_sandbox_k3s_api_port: 6443
vyos_sandbox_k3s_servers:
- name: sandbox1
address: "10.60.0.11"
- name: sandbox2
address: "10.60.0.12"
@@ -3,23 +3,6 @@
# `set` lines against the running config and issues only the differences, so
# re-runs are no-ops. Supports --check and --diff.
- name: Read current VyOS configuration commands
vyos.vyos.vyos_command:
commands:
- show configuration commands
register: _vyos_running_commands
changed_when: false
- name: Remove obsolete HAProxy PostgreSQL protocol check
vyos.vyos.vyos_config:
lines:
- delete load-balancing haproxy backend sandbox-postgresql-primary health-check
save: true
comment: Remove unsupported PostgreSQL protocol check
when: >-
"set load-balancing haproxy backend sandbox-postgresql-primary health-check 'pgsql'"
in _vyos_running_commands.stdout[0]
- name: Render the desired configuration
ansible.builtin.set_fact:
_vyos_lines: >-
@@ -35,23 +35,11 @@ set load-balancing haproxy service sandbox-postgresql port {{ vyos_postgresql_po
set load-balancing haproxy service sandbox-postgresql mode 'tcp'
set load-balancing haproxy service sandbox-postgresql backend 'sandbox-postgresql-primary'
set load-balancing haproxy backend sandbox-postgresql-primary mode 'tcp'
set load-balancing haproxy backend sandbox-postgresql-primary health-check 'pgsql'
set load-balancing haproxy backend sandbox-postgresql-primary server primary address {{ vyos_postgresql_primary_address }}
set load-balancing haproxy backend sandbox-postgresql-primary server primary port {{ vyos_postgresql_port }}
set load-balancing haproxy backend sandbox-postgresql-primary server primary check
{# --- Sandbox K3s API: stable address with both control-plane nodes --- #}
set interfaces ethernet {{ vyos_sandbox_k3s_api_interface }} address '{{ vyos_sandbox_k3s_api_address }}/32'
set load-balancing haproxy service sandbox-k3s-api listen-address {{ vyos_sandbox_k3s_api_address }}
set load-balancing haproxy service sandbox-k3s-api port {{ vyos_sandbox_k3s_api_port }}
set load-balancing haproxy service sandbox-k3s-api mode 'tcp'
set load-balancing haproxy service sandbox-k3s-api backend 'sandbox-k3s-api'
set load-balancing haproxy backend sandbox-k3s-api mode 'tcp'
{% for server in vyos_sandbox_k3s_servers %}
set load-balancing haproxy backend sandbox-k3s-api server {{ server.name }} address {{ server.address }}
set load-balancing haproxy backend sandbox-k3s-api server {{ server.name }} port {{ vyos_sandbox_k3s_api_port }}
set load-balancing haproxy backend sandbox-k3s-api server {{ server.name }} check
{% endfor %}
{# --- management --- #}
set service ssh port 22
-142
View File
@@ -1,142 +0,0 @@
# Sandbox 集群
该目录管理供 OpenSandbox、CI 和 AI Agent workload 使用的独立双节点 k3s 集群。
基础设施和节点生命周期由 Ansible 管理;Kubernetes API 可用后,集群内组件由
`clusters/sandbox/` 下的 Flux desired state 管理。
## 所有权边界
Ansible 管理以下持久状态:
- pve1/pve2 上的 privileged LXC、磁盘、固定网络与 KVM/vhost/TUN/kmsg 设备;
- LXC OS 基线、内部 CA、外部 PostgreSQL 与 K3s;
- 固定版本的 Flux controllers 和 root sync bootstrap;
- 只读状态验证和数据库切换 runbook。
Flux 管理以下 Kubernetes 资源:
- Kata Containers 和 CI 专用的 `block-plain` RuntimeClass;
- SPIRE Agent、SPIFFE CSI Driver 与 workload identity 声明;
- vmagent、kube-state-metrics、kubelet/cAdvisor scrape 配置和告警;
- OpenSandbox operator/server、`ci-pod` 与 `ci-vm` Pools。
同一个对象只能有一个 owner。Ansible 不直接部署上述集群内 workload;Flux 不管理
LXC、K3s datastore 或 K3s 本身。
监控范围仅覆盖 sandbox LXC 内的 Kubernetes 与 workload。不得在 LXC 内部署
node_exporter:LXC 的 `/proc` 是 lxcfs 虚拟视图与宿主内核视图的混合,尤其
`/proc/stat` 会重复暴露 PVE 宿主 CPU 数据。PVE 自身的 node_exporter 或其他宿主
监控不属于本目录;sandbox 使用 kubelet/cAdvisor 和 kube-state-metrics。
## 声明拓扑
| 对象 | PVE 节点 | VMID | 地址 | 资源 |
|---|---:|---:|---|---|
| `sandbox1` | pve1 | 148 | `10.60.0.11/24` | 4 vCPU / 6 GiB / 2 GiB swap |
| `sandbox2` | pve2 | 149 | `10.60.0.12/24` | 4 vCPU / 4 GiB / 2 GiB swap |
| K3s API VIP | VyOS | — | `10.60.0.13:6443` | HAProxy TCP LB |
节点与 API VIP 均位于现有 PVE `labnet`(VLAN 100,`10.60.0.0/24`),通过
VyOS `10.60.0.1` 路由;不为 sandbox 新建 VNet,也不占用 `192.168.10.0/24`
地址。需要从 LAN 访问的服务统一经 VyOS 路由或 LB 暴露。两个 LXC 均使用
`pve-rg-hdd` 上的 32 GiB rootfs;不为 `/var/lib/kubelet` 单独创建 volume。Kata
`block-plain` 产生的数据随 Pod 生命周期清理,当前规模没有额外磁盘故障域的需求。
K3s API 的 `10.60.0.13/32` 由 VyOS 现有 labnet interface 持有,HAProxy 以 TCP
健康检查把 `6443` 分发到两个 control-plane 节点;它与 PostgreSQL 主从切换逻辑无关。
LXC 内通过 `/etc/tmpfiles.d/kmsg.conf` 持久维护 `/dev/kmsg -> /dev/console`;否则
kubelet 会因 LXC 不提供真实 host `/dev/kmsg` 而反复退出。
每个 LXC 还以只读 bind mount 使用宿主的 `/lib/modules`。LXC 与 PVE 宿主共享内核,
guest 若看不到对应版本的模块目录,K3s 无法加载 `br_netfilter` 和 `overlay`,Flannel
也不会生成节点的 subnet 配置。
## PostgreSQL 写入口与切换
K3s 使用外部 PostgreSQL,首期采用 primary + synchronous standby。VyOS 在 labnet
gateway `10.60.0.1:5432` 提供固定 TCP 入口,backend 只包含当前声明的 primary;
不把普通 TCP 或 PostgreSQL 存活检查等同于“节点可写”,也不自动把流量切到 standby。
VyOS 2025.11 的 PostgreSQL protocol check 会生成缺少必需 `user` 参数的 HAProxy
配置,因此这里只使用基础 TCP check;真正的可写性由 `verify.yml` 通过 SQL 验证。
数据库切换必须由 Ansible runbook 受控完成:先隔离旧 primary,再提升 standby,最后
更新 VyOS backend。首期不部署 PgBouncer、Patroni、repmgr 或额外 DCS,也不宣称两节点
PostgreSQL 能够自动 HA。以后具备第三个仲裁节点时再重新评估自动 failover。
数据库凭据位于 Bao `kv/infra/sandbox-postgresql`,包含 K3s 登录密码和 physical
replication 密码。运行 Ansible 前由本机 SPIFFE identity 获取短期 Bao token,再把
两个值注入 `SANDBOX_K3S_DB_PASSWORD` 与 `SANDBOX_REPLICATION_PASSWORD`;凭据不写入
inventory、Git 或 Ansible fact cache。首次创建 standby 只允许覆盖不含任何业务库的
Ubuntu 默认空集群,之后重复执行不会 reseed。
## Ansible
安装固定依赖:
```bash
cd infrastructure/sandbox-cluster/ansible
uv venv .venv
uv pip install --python .venv/bin/python -r requirements.txt
source .venv/bin/activate
ansible-galaxy collection install -r requirements.yml
export PVE_API_PYTHON="$PWD/.venv/bin/python"
```
部署入口最终为:
```bash
ansible-playbook site.yml
```
只安装或 reconcile K3s(LXC、OS baseline 和 PostgreSQL 已就绪时):
```bash
ansible-playbook k3s.yml
```
K3s 外部 datastore URI 由运行时 `SANDBOX_K3S_DB_PASSWORD` 生成,密码在 URI 中
进行 URL 编码,最终仅持久化于节点 root 可读的 `/etc/rancher/k3s/config.yaml`
(mode `0600`)。首节点生成的 K3s join token 仅在同一次 Ansible run 内传给第二节点;
不在 inventory 或 Git 中维护副本。首期关闭内建 Traefik 和 ServiceLB。
部署后使用同一组运行时凭据执行只读验收:
```bash
ansible-playbook verify.yml
```
当前已经声明 LXC 生命周期、最小 OS baseline、PostgreSQL 和 K3s,包括系统级
homelab CA trust。Flux `v2.9.5` controllers 与 root sync 也由 Ansible 通过 K3s
server manifests 管理;root 使用 homelab CA 访问公开 Gitea 仓库,不保存 Git token。
集群内 workload 由 `clusters/sandbox/` 分阶段纳入 Flux。
## SPIRE 跨集群 bootstrap
Sandbox 复用 homelab 的 SPIRE Server 与 `ddupan.top` trust domain。Flux 首先安装
SPIRE CRD,并创建供 k8s_psat 使用的 reviewer。它按上游 Server chart 的权限模型调用
TokenReview,并以 `get/list` 读取用于证明的 Pod 与 Node;不具有修改 workload 的权限。
在该 Kustomization Ready 后运行:
```bash
cd infrastructure/sandbox-cluster/ansible
ansible-playbook spire-bootstrap.yml
```
Playbook 不把 reviewer token 或生成的 kubeconfig 落盘,而是将目标 Secret manifest
通过 stdin 交给本机 homelab `k3s kubectl`。目标 Secret
`spire-server/spire-external-kubeconfigs` 由 Ansible 单独拥有;Flux 和人工操作不得写入。
第二次运行必须为零变更。
Secret 的 `sandbox` key 仅供 SPIRE Server 的 external PSAT plugin 验证 token 与对应的
Pod/Node;
`sandbox-controller` key 供 external controller-manager 读取 Pod/Node、reconcile SPIFFE
CR 及执行 leader election。两者使用不同 ServiceAccount,不得合并权限或互换。
## 已验证的 Kata CI 前置条件
- Cloud Hypervisor 必须报告 `vm.info.config.memory.shared=true`;
- CI RuntimeClass 必须使用 `[runtime] emptydir_mode = "block-plain"`;
- dockerd bootstrap 在 guest 内创建 `/dev/kmsg`:`mknod /dev/kmsg c 1 11`;
- `/var/lib/docker` 必须是 guest block device 上的 ext4,Docker driver 必须为
`overlay2`,不能静默退化到 `vfs` 或 `fuse-overlayfs`;
- Pod 删除后必须不存在遗留 `disk.img`、VMM 或临时 credential。
PoC 的完整数据和陷阱见 `../kata-lxc-lab/README.md`。
@@ -1,11 +0,0 @@
[defaults]
inventory = inventory/hosts.yml
host_key_checking = True
interpreter_python = auto_silent
retry_files_enabled = False
local_tmp = /tmp/ansible-sandbox-cluster-local
remote_tmp = /tmp
roles_path = roles
[ssh_connection]
pipelining = True
@@ -1,61 +0,0 @@
---
sandbox_pve_api_host: "{{ lookup('env', 'PROXMOX_HOST') }}"
sandbox_pve_api_user: "{{ lookup('env', 'PROXMOX_USER') }}"
sandbox_pve_api_token_id: "{{ lookup('env', 'PROXMOX_TOKEN_ID') }}"
sandbox_pve_api_token_secret: "{{ lookup('env', 'PROXMOX_TOKEN_SECRET') }}"
sandbox_pve_validate_certs: true
sandbox_pve_api_python_interpreter: "{{ lookup('env', 'PVE_API_PYTHON') | default(ansible_playbook_python, true) }}"
sandbox_bridge: labnet
sandbox_gateway: 10.60.0.1
sandbox_nameserver: 192.168.10.5
sandbox_search_domain: ad.ddupan.top
sandbox_lxc_template: laptop:vztmpl/ubuntu-24.04-standard_24.04-2_amd64.tar.zst
sandbox_rootfs_storage: pve-rg-hdd
sandbox_rootfs_gb: 32
sandbox_lxc_cores: 4
sandbox_lxc_swap_mb: 2048
sandbox_lxc_pubkey: "{{ lookup('file', '~/.ssh/id_ed25519.pub') }}"
sandbox_internal_ca_url: https://bao.ad.ddupan.top:8200/v1/pki/ca/pem
sandbox_base_packages:
- ca-certificates
- curl
- jq
- openssh-server
sandbox_postgresql_version: 16
sandbox_postgresql_primary_address: 10.60.0.11
sandbox_postgresql_standby_address: 10.60.0.12
sandbox_postgresql_lb_address: 10.60.0.1
sandbox_postgresql_database: k3s
sandbox_postgresql_user: k3s
sandbox_postgresql_replication_user: k3s_replication
sandbox_postgresql_replication_slot: sandbox2
sandbox_postgresql_rotate_passwords: false
sandbox_postgresql_k3s_password: "{{ lookup('env', 'SANDBOX_K3S_DB_PASSWORD') }}"
sandbox_postgresql_replication_password: "{{ lookup('env', 'SANDBOX_REPLICATION_PASSWORD') }}"
# This is rendered only at playbook runtime from the Bao-provided password. K3s
# necessarily persists its datastore credential, but config.yaml is root-only.
sandbox_k3s_datastore_password_encoded: >-
{{ sandbox_postgresql_k3s_password | urlencode | replace('/', '%2F') }}
sandbox_k3s_datastore_endpoint: >-
{{
[
'postgres://', sandbox_postgresql_user, ':', sandbox_k3s_datastore_password_encoded,
'@', sandbox_postgresql_lb_address, ':5432/', sandbox_postgresql_database, '?sslmode=disable'
] | join
}}
server_config_yaml: |
datastore-endpoint: {{ sandbox_k3s_datastore_endpoint }}
disable:
- servicelb
- traefik
user_kubectl: false
sandbox_flux_version: v2.9.5
sandbox_flux_install_url: >-
https://github.com/fluxcd/flux2/releases/download/{{ sandbox_flux_version }}/install.yaml
sandbox_flux_repository_url: https://git.ddupan.top/panxiao81/homelab-infra.git
sandbox_flux_repository_branch: main
sandbox_flux_path: ./clusters/sandbox
@@ -1,48 +0,0 @@
---
all:
children:
sandbox_pve:
hosts:
pve1:
ansible_host: 192.168.10.4
sandbox_lxc:
vmid: 148
hostname: sandbox1
address: 10.60.0.11/24
memory_mb: 6144
pve2:
ansible_host: 192.168.10.7
sandbox_lxc:
vmid: 149
hostname: sandbox2
address: 10.60.0.12/24
memory_mb: 4096
k3s_cluster:
children:
server:
hosts:
sandbox1:
ansible_host: 10.60.0.11
sandbox2:
ansible_host: 10.60.0.12
agent:
hosts: {}
vars:
ansible_user: root
k3s_version: v1.36.4+k3s1
api_endpoint: 10.60.0.13
k3s_server_config_mode: "0600"
use_external_database: true
postgres_primary:
hosts:
sandbox1:
postgres_standby:
hosts:
sandbox2:
vars:
ansible_user: root
ansible_python_interpreter: /usr/bin/python3
@@ -1,17 +0,0 @@
---
- name: Validate runtime K3s datastore credential
hosts: k3s_cluster
gather_facts: false
tasks:
- name: Require the Bao-provided PostgreSQL password
ansible.builtin.assert:
that:
- sandbox_postgresql_k3s_password | length > 0
- server_config_yaml is defined
- (server_config_yaml | from_yaml)['datastore-endpoint'] == sandbox_k3s_datastore_endpoint
fail_msg: SANDBOX_K3S_DB_PASSWORD must be populated from Bao
quiet: true
no_log: true
- name: Install the sandbox K3s control plane
ansible.builtin.import_playbook: k3s.orchestration.site
@@ -1,4 +0,0 @@
ansible-core==2.21.2
proxmoxer==2.3.0
requests==2.34.2
netaddr==1.3.0
@@ -1,15 +0,0 @@
---
collections:
- name: community.proxmox
version: 2.0.0
- name: community.postgresql
version: 4.2.0
- name: ansible.posix
version: 2.2.2
- name: community.general
version: 13.4.0
- name: community.library_inventory_filtering_v1
version: 1.1.5
- name: https://github.com/k3s-io/k3s-ansible.git
type: git
version: 1a600b60d37e0f8a6e2e79b0e474147b5b108ae5
@@ -1,5 +0,0 @@
---
- name: Update sandbox CA certificates
ansible.builtin.command:
cmd: update-ca-certificates
changed_when: true
@@ -1,58 +0,0 @@
---
- name: Install sandbox base packages
ansible.builtin.apt:
name: "{{ sandbox_base_packages }}"
state: present
update_cache: true
cache_valid_time: 3600
- name: Fetch the homelab internal CA
ansible.builtin.uri:
url: "{{ sandbox_internal_ca_url }}"
return_content: true
validate_certs: true
register: sandbox_internal_ca
changed_when: false
retries: 3
delay: 5
until: sandbox_internal_ca is succeeded
- name: Validate the homelab internal CA response
ansible.builtin.assert:
that:
- "'BEGIN CERTIFICATE' in sandbox_internal_ca.content"
fail_msg: OpenBao did not return a PEM certificate
quiet: true
- name: Install the homelab internal CA
ansible.builtin.copy:
dest: /usr/local/share/ca-certificates/ddupan-internal-ca.crt
content: "{{ sandbox_internal_ca.content }}"
owner: root
group: root
mode: "0644"
notify: Update sandbox CA certificates
- name: Enable the SSH service
ansible.builtin.service:
name: ssh
enabled: true
state: started
# LXC does not expose the host kernel log device. Kubelet only needs a writable
# kmsg-compatible character device, and the container console is the established
# LXC mapping for this purpose. tmpfiles recreates the link after every boot.
- name: Persist the LXC kubelet kmsg mapping
ansible.builtin.copy:
dest: /etc/tmpfiles.d/kmsg.conf
content: "L+ /dev/kmsg - - - - /dev/console\n"
owner: root
group: root
mode: "0644"
- name: Expose the LXC console as kmsg
ansible.builtin.file:
src: /dev/console
dest: /dev/kmsg
state: link
force: true
@@ -1,30 +0,0 @@
---
- name: Wait for the K3s server manifest directory
ansible.builtin.wait_for:
path: /var/lib/rancher/k3s/server/manifests
state: present
timeout: 120
- name: Download the pinned Flux controller manifest
ansible.builtin.get_url:
url: "{{ sandbox_flux_install_url }}"
dest: /var/lib/rancher/k3s/server/manifests/flux-install.yaml
owner: root
group: root
mode: "0644"
timeout: 120
- name: Read the installed homelab CA
ansible.builtin.slurp:
src: /usr/local/share/ca-certificates/ddupan-internal-ca.crt
register: sandbox_flux_ca
no_log: true
- name: Install the Flux source and root sync manifest
ansible.builtin.template:
src: flux-sync.yaml.j2
dest: /var/lib/rancher/k3s/server/manifests/flux-sync.yaml
owner: root
group: root
mode: "0600"
no_log: true
@@ -1,39 +0,0 @@
---
apiVersion: v1
kind: Secret
metadata:
name: flux-system
namespace: flux-system
type: Opaque
stringData:
caFile: |
{{ sandbox_flux_ca.content | b64decode | indent(4, true) }}
---
apiVersion: source.toolkit.fluxcd.io/v1
kind: GitRepository
metadata:
name: flux-system
namespace: flux-system
spec:
interval: 1m
ref:
branch: {{ sandbox_flux_repository_branch }}
secretRef:
name: flux-system
timeout: 60s
url: {{ sandbox_flux_repository_url }}
---
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
name: flux-system
namespace: flux-system
spec:
interval: 10m
path: {{ sandbox_flux_path }}
prune: false
sourceRef:
kind: GitRepository
name: flux-system
timeout: 3m
wait: true
@@ -1,214 +0,0 @@
---
- name: Validate sandbox declaration and API inputs
ansible.builtin.assert:
that:
- sandbox_lxc.vmid | int > 100
- sandbox_lxc.hostname | length > 0
- sandbox_lxc.address is match('^10\.60\.0\.[0-9]{1,3}/24$')
- sandbox_lxc.memory_mb | int >= 4096
- sandbox_pve_api_host | length > 0
- sandbox_pve_api_user | length > 0
- sandbox_pve_api_token_id | length > 0
- sandbox_pve_api_token_secret | length > 0
fail_msg: Invalid sandbox declaration or missing Proxmox API environment
no_log: true
- name: Check whether the LXC already exists
ansible.builtin.stat:
path: /etc/pve/lxc/{{ sandbox_lxc.vmid }}.conf
register: sandbox_lxc_config
- name: Read current sandbox LXC runtime state
community.proxmox.proxmox_vm_info:
api_host: "{{ sandbox_pve_api_host }}"
api_user: "{{ sandbox_pve_api_user }}"
api_token_id: "{{ sandbox_pve_api_token_id }}"
api_token_secret: "{{ sandbox_pve_api_token_secret }}"
validate_certs: "{{ sandbox_pve_validate_certs }}"
ca_path: /etc/ssl/certs/ca-certificates.crt
api_timeout: 120
vmid: "{{ sandbox_lxc.vmid }}"
register: sandbox_lxc_info
delegate_to: localhost
no_log: true
vars:
ansible_python_interpreter: "{{ sandbox_pve_api_python_interpreter }}"
- name: Read current sandbox LXC configuration
ansible.builtin.uri:
url: >-
https://{{ sandbox_pve_api_host }}:8006/api2/json/nodes/{{ inventory_hostname }}/lxc/{{ sandbox_lxc.vmid }}/config
headers:
Authorization: >-
PVEAPIToken={{ sandbox_pve_api_user }}!{{ sandbox_pve_api_token_id }}={{ sandbox_pve_api_token_secret }}
ca_path: /etc/ssl/certs/ca-certificates.crt
return_content: true
status_code: [200, 404, 500]
register: sandbox_lxc_current_config
delegate_to: localhost
no_log: true
- name: Reconcile sandbox LXC through the PVE API
community.proxmox.proxmox:
api_host: "{{ sandbox_pve_api_host }}"
api_user: "{{ sandbox_pve_api_user }}"
api_token_id: "{{ sandbox_pve_api_token_id }}"
api_token_secret: "{{ sandbox_pve_api_token_secret }}"
validate_certs: "{{ sandbox_pve_validate_certs }}"
ca_path: /etc/ssl/certs/ca-certificates.crt
api_timeout: 120
node: "{{ inventory_hostname }}"
vmid: "{{ sandbox_lxc.vmid }}"
hostname: "{{ sandbox_lxc.hostname }}"
ostemplate: "{{ sandbox_lxc_template }}"
ostype: ubuntu
cmode: shell
disk_volume:
storage: "{{ sandbox_rootfs_storage }}"
size: "{{ sandbox_rootfs_gb if not sandbox_lxc_config.stat.exists else omit }}"
cores: "{{ sandbox_lxc_cores }}"
memory: "{{ sandbox_lxc.memory_mb }}"
swap: "{{ sandbox_lxc_swap_mb }}"
netif:
net0: >-
{{
'name=eth0,bridge=' ~ sandbox_bridge ~
',ip=' ~ sandbox_lxc.address ~
',gw=' ~ sandbox_gateway ~
',type=veth' ~
(',hwaddr=' ~ _sandbox_existing_mac if _sandbox_existing_mac | length > 0 else '')
}}
nameserver: "{{ sandbox_nameserver }}"
searchdomain: "{{ sandbox_search_domain }}"
unprivileged: false
onboot: true
startup: ["order=30", "up=30", "down=60"]
tags: [ansible, sandbox, k3s, kata]
pubkey: "{{ sandbox_lxc_pubkey }}"
update: true
state: present
timeout: 120
register: sandbox_lxc_api
when: not sandbox_lxc_config.stat.exists or _sandbox_api_drift
delegate_to: localhost
no_log: true
vars:
ansible_python_interpreter: "{{ sandbox_pve_api_python_interpreter }}"
_sandbox_existing_mac: >-
{{ sandbox_lxc_current_config.json.data.net0.split('hwaddr=')[1].split(',')[0]
if 'hwaddr=' in (sandbox_lxc_current_config.json.data.net0 | default(''))
else '' }}
_sandbox_api_drift: >-
{{
sandbox_lxc_current_config.status != 200 or
sandbox_lxc_current_config.json.data.hostname | default('') != sandbox_lxc.hostname or
sandbox_lxc_current_config.json.data.cores | default(0) | int != sandbox_lxc_cores | int or
sandbox_lxc_current_config.json.data.memory | default(0) | int != sandbox_lxc.memory_mb | int or
sandbox_lxc_current_config.json.data.swap | default(0) | int != sandbox_lxc_swap_mb | int or
sandbox_lxc_current_config.json.data.nameserver | default('') != sandbox_nameserver or
sandbox_lxc_current_config.json.data.searchdomain | default('') != sandbox_search_domain or
sandbox_lxc_current_config.json.data.cmode | default('') != 'shell' or
sandbox_lxc_current_config.json.data.onboot | default(0) | int != 1 or
sandbox_lxc_current_config.json.data.tags | default('') != 'ansible;k3s;kata;sandbox' or
('bridge=' ~ sandbox_bridge) not in (sandbox_lxc_current_config.json.data.net0 | default('')) or
('ip=' ~ sandbox_lxc.address) not in (sandbox_lxc_current_config.json.data.net0 | default('')) or
('gw=' ~ sandbox_gateway) not in (sandbox_lxc_current_config.json.data.net0 | default('')) or
(sandbox_rootfs_storage ~ ':') not in (sandbox_lxc_current_config.json.data.rootfs | default(''))
}}
- name: Grow existing sandbox rootfs through the PVE resize API
ansible.builtin.uri:
url: >-
https://{{ sandbox_pve_api_host }}:8006/api2/json/nodes/{{ inventory_hostname }}/lxc/{{ sandbox_lxc.vmid }}/resize
method: PUT
headers:
Authorization: >-
PVEAPIToken={{ sandbox_pve_api_user }}!{{ sandbox_pve_api_token_id }}={{ sandbox_pve_api_token_secret }}
body_format: form-urlencoded
body:
disk: rootfs
size: "{{ sandbox_rootfs_gb }}G"
ca_path: /etc/ssl/certs/ca-certificates.crt
status_code: 200
when:
- sandbox_lxc_config.stat.exists
- _sandbox_current_rootfs_gb | int < sandbox_rootfs_gb | int
delegate_to: localhost
no_log: true
vars:
_sandbox_current_rootfs_gb: >-
{{
sandbox_lxc_current_config.json.data.rootfs |
default('') |
regex_findall('size=([0-9]+)G') |
first |
default('0')
}}
# These properties are not exposed by community.proxmox. Every retained line
# is required by the already validated nested Kata/kind runtime.
- name: Reconcile Kata-specific native LXC properties
ansible.builtin.lineinfile:
path: /etc/pve/lxc/{{ sandbox_lxc.vmid }}.conf
regexp: "^{{ item.key | regex_escape }}:"
line: "{{ item.key }}: {{ item.value }}"
loop:
- { key: features, value: "nesting=1,keyctl=1,fuse=1,mknod=1,force_rw_sys=1" }
- { key: dev0, value: "path=/dev/kvm,mode=0660" }
- { key: dev1, value: "path=/dev/vhost-net,mode=0660" }
- { key: dev2, value: "path=/dev/vhost-vsock,mode=0660" }
- { key: dev3, value: "path=/dev/net/tun,mode=0666" }
- { key: lxc.apparmor.profile, value: unconfined }
- { key: lxc.cgroup2.devices.allow, value: a }
- { key: lxc.cap.drop, value: "" }
- { key: lxc.mount.auto, value: "proc:rw sys:rw" }
- { key: lxc.mount.entry, value: "/lib/modules usr/lib/modules none bind,ro,create=dir 0 0" }
loop_control:
label: "{{ item.key }}"
register: sandbox_lxc_native
- name: Start sandbox LXC
community.proxmox.proxmox:
api_host: "{{ sandbox_pve_api_host }}"
api_user: "{{ sandbox_pve_api_user }}"
api_token_id: "{{ sandbox_pve_api_token_id }}"
api_token_secret: "{{ sandbox_pve_api_token_secret }}"
validate_certs: "{{ sandbox_pve_validate_certs }}"
ca_path: /etc/ssl/certs/ca-certificates.crt
api_timeout: 120
vmid: "{{ sandbox_lxc.vmid }}"
state: started
timeout: 120
when: >-
not _sandbox_was_running or
not (sandbox_lxc_api is changed or sandbox_lxc_native is changed)
delegate_to: localhost
no_log: true
vars:
ansible_python_interpreter: "{{ sandbox_pve_api_python_interpreter }}"
_sandbox_was_running: >-
{{ (sandbox_lxc_info.proxmox_vms | default([]) | first | default({})).status |
default('absent') == 'running' }}
- name: Restart existing sandbox LXC after configuration changes
community.proxmox.proxmox:
api_host: "{{ sandbox_pve_api_host }}"
api_user: "{{ sandbox_pve_api_user }}"
api_token_id: "{{ sandbox_pve_api_token_id }}"
api_token_secret: "{{ sandbox_pve_api_token_secret }}"
validate_certs: "{{ sandbox_pve_validate_certs }}"
ca_path: /etc/ssl/certs/ca-certificates.crt
api_timeout: 120
vmid: "{{ sandbox_lxc.vmid }}"
state: restarted
timeout: 120
when:
- _sandbox_was_running
- sandbox_lxc_api is changed or sandbox_lxc_native is changed
delegate_to: localhost
no_log: true
vars:
ansible_python_interpreter: "{{ sandbox_pve_api_python_interpreter }}"
_sandbox_was_running: >-
{{ (sandbox_lxc_info.proxmox_vms | default([]) | first | default({})).status |
default('absent') == 'running' }}
@@ -1,6 +0,0 @@
---
- name: Restart sandbox PostgreSQL
ansible.builtin.service:
name: postgresql
enabled: true
state: restarted
@@ -1,50 +0,0 @@
---
- name: Validate sandbox PostgreSQL inputs
ansible.builtin.assert:
that:
- sandbox_postgresql_k3s_password | length >= 32
- sandbox_postgresql_replication_password | length >= 32
- inventory_hostname in groups['postgres_primary'] or inventory_hostname in groups['postgres_standby']
fail_msg: Missing Bao-provided PostgreSQL credentials or invalid inventory role
no_log: true
- name: Install PostgreSQL packages
ansible.builtin.apt:
name:
- postgresql-{{ sandbox_postgresql_version }}
- postgresql-client-{{ sandbox_postgresql_version }}
- python3-psycopg2
state: present
update_cache: true
cache_valid_time: 3600
- name: Check whether synchronous replication has completed bootstrap
ansible.builtin.stat:
path: /etc/postgresql/{{ sandbox_postgresql_version }}/main/sandbox-synchronous-ready
register: sandbox_postgresql_synchronous_marker
- name: Install sandbox PostgreSQL access policy
ansible.builtin.template:
src: pg_hba.conf.j2
dest: /etc/postgresql/{{ sandbox_postgresql_version }}/main/pg_hba.conf
owner: postgres
group: postgres
mode: "0640"
notify: Restart sandbox PostgreSQL
- name: Install sandbox PostgreSQL server settings
ansible.builtin.template:
src: sandbox.conf.j2
dest: /etc/postgresql/{{ sandbox_postgresql_version }}/main/conf.d/99-sandbox.conf
owner: postgres
group: postgres
mode: "0644"
notify: Restart sandbox PostgreSQL
- name: Configure sandbox PostgreSQL primary
ansible.builtin.include_tasks: primary.yml
when: inventory_hostname in groups['postgres_primary']
- name: Configure sandbox PostgreSQL standby
ansible.builtin.include_tasks: standby.yml
when: inventory_hostname in groups['postgres_standby']
@@ -1,45 +0,0 @@
---
- name: Apply primary PostgreSQL configuration before replication setup
ansible.builtin.meta: flush_handlers
- name: Create K3s PostgreSQL login role
community.postgresql.postgresql_user:
name: "{{ sandbox_postgresql_user }}"
password: "{{ sandbox_postgresql_k3s_password }}"
no_password_changes: "{{ not sandbox_postgresql_rotate_passwords }}"
role_attr_flags: LOGIN,NOSUPERUSER,NOCREATEDB,NOCREATEROLE,NOREPLICATION
state: present
become: true
become_user: postgres
no_log: true
environment:
PGOPTIONS: -c password_encryption=scram-sha-256
- name: Create K3s PostgreSQL database
community.postgresql.postgresql_db:
name: "{{ sandbox_postgresql_database }}"
owner: "{{ sandbox_postgresql_user }}"
state: present
become: true
become_user: postgres
- name: Create PostgreSQL replication login role
community.postgresql.postgresql_user:
name: "{{ sandbox_postgresql_replication_user }}"
password: "{{ sandbox_postgresql_replication_password }}"
no_password_changes: "{{ not sandbox_postgresql_rotate_passwords }}"
role_attr_flags: LOGIN,REPLICATION,NOSUPERUSER,NOCREATEDB,NOCREATEROLE
state: present
become: true
become_user: postgres
no_log: true
environment:
PGOPTIONS: -c password_encryption=scram-sha-256
- name: Create physical replication slot for sandbox2
community.postgresql.postgresql_slot:
name: "{{ sandbox_postgresql_replication_slot }}"
slot_type: physical
state: present
become: true
become_user: postgres
@@ -1,114 +0,0 @@
---
- name: Detect whether sandbox2 is already a standby
community.postgresql.postgresql_query:
login_db: postgres
query: SELECT pg_is_in_recovery() AS in_recovery
become: true
become_user: postgres
register: sandbox_postgresql_recovery_state
- name: Inspect databases before the initial standby seed
community.postgresql.postgresql_query:
login_db: postgres
query: >-
SELECT datname FROM pg_database
WHERE NOT datistemplate AND datname <> 'postgres'
become: true
become_user: postgres
register: sandbox_postgresql_existing_databases
when: not sandbox_postgresql_recovery_state.query_result[0].in_recovery
- name: Refuse to overwrite a non-empty PostgreSQL node
ansible.builtin.assert:
that:
- sandbox_postgresql_existing_databases.query_result | length == 0
fail_msg: Refusing to reseed sandbox2 because it contains non-system databases
when: not sandbox_postgresql_recovery_state.query_result[0].in_recovery
- name: Stop PostgreSQL before the initial standby seed
ansible.builtin.service:
name: postgresql
state: stopped
when: not sandbox_postgresql_recovery_state.query_result[0].in_recovery
- name: Remove the verified-empty standby data directory
ansible.builtin.file:
path: /var/lib/postgresql/{{ sandbox_postgresql_version }}/main
state: absent
when: not sandbox_postgresql_recovery_state.query_result[0].in_recovery
- name: Recreate the standby data directory
ansible.builtin.file:
path: /var/lib/postgresql/{{ sandbox_postgresql_version }}/main
state: directory
owner: postgres
group: postgres
mode: "0700"
when: not sandbox_postgresql_recovery_state.query_result[0].in_recovery
- name: Install standby replication password file
ansible.builtin.copy:
dest: /var/lib/postgresql/.pgpass
content: >-
{{ sandbox_postgresql_primary_address }}:5432:*:{{ sandbox_postgresql_replication_user }}:{{ sandbox_postgresql_replication_password }}
owner: postgres
group: postgres
mode: "0600"
no_log: true
- name: Seed sandbox2 from the PostgreSQL primary
ansible.builtin.command:
argv:
- /usr/bin/pg_basebackup
- --host={{ sandbox_postgresql_primary_address }}
- --username={{ sandbox_postgresql_replication_user }}
- --pgdata=/var/lib/postgresql/{{ sandbox_postgresql_version }}/main
- --format=plain
- --wal-method=stream
- --write-recovery-conf
- --slot={{ sandbox_postgresql_replication_slot }}
become: true
become_user: postgres
environment:
PGPASSFILE: /var/lib/postgresql/.pgpass
when: not sandbox_postgresql_recovery_state.query_result[0].in_recovery
no_log: true
- name: Set the standby connection identity
ansible.builtin.lineinfile:
path: /var/lib/postgresql/{{ sandbox_postgresql_version }}/main/postgresql.auto.conf
regexp: ^primary_conninfo =
line: >-
primary_conninfo = 'host={{ sandbox_postgresql_primary_address }} port=5432
user={{ sandbox_postgresql_replication_user }} application_name=sandbox2
passfile=/var/lib/postgresql/.pgpass'
owner: postgres
group: postgres
mode: "0600"
no_log: true
- name: Start PostgreSQL standby
ansible.builtin.service:
name: postgresql
enabled: true
state: started
- name: Wait for sandbox2 to enter recovery
community.postgresql.postgresql_query:
login_db: postgres
query: SELECT pg_is_in_recovery() AS in_recovery
become: true
become_user: postgres
register: sandbox_postgresql_standby_ready
retries: 12
delay: 5
until: sandbox_postgresql_standby_ready.query_result[0].in_recovery
- name: Mark synchronous replication bootstrap complete on the primary
ansible.builtin.copy:
dest: /etc/postgresql/{{ sandbox_postgresql_version }}/main/sandbox-synchronous-ready
content: "sandbox2\n"
owner: postgres
group: postgres
mode: "0644"
delegate_to: "{{ groups['postgres_primary'][0] }}"
@@ -1,5 +0,0 @@
# Ansible managed
local all postgres peer
local all all peer
host {{ sandbox_postgresql_database }} {{ sandbox_postgresql_user }} 10.60.0.0/24 scram-sha-256
host replication {{ sandbox_postgresql_replication_user }} 10.60.0.0/24 scram-sha-256
@@ -1,11 +0,0 @@
# Ansible managed
listen_addresses = '{{ ansible_host }},127.0.0.1'
password_encryption = 'scram-sha-256'
wal_level = 'replica'
max_wal_senders = 10
max_replication_slots = 10
hot_standby = on
{% if inventory_hostname in groups['postgres_primary'] and sandbox_postgresql_synchronous_marker.stat.exists %}
synchronous_standby_names = 'FIRST 1 (sandbox2)'
synchronous_commit = 'remote_apply'
{% endif %}
@@ -1,7 +0,0 @@
---
sandbox_spire_bootstrap_api_server: https://10.60.0.13:6443
sandbox_spire_bootstrap_source_namespace: spire-system
sandbox_spire_bootstrap_source_secret: spire-server-token-reviewer-token
sandbox_spire_bootstrap_controller_secret: spire-controller-manager-token
sandbox_spire_bootstrap_target_namespace: spire-server
sandbox_spire_bootstrap_target_secret: spire-external-kubeconfigs
@@ -1,115 +0,0 @@
---
- name: Wait for the sandbox SPIRE token reviewer credential
ansible.builtin.command:
argv:
- k3s
- kubectl
- --namespace
- "{{ sandbox_spire_bootstrap_source_namespace }}"
- get
- secret
- "{{ sandbox_spire_bootstrap_source_secret }}"
- --output=json
register: sandbox_spire_bootstrap_reviewer_secret
changed_when: false
retries: 60
delay: 10
until:
- sandbox_spire_bootstrap_reviewer_secret.rc == 0
- (sandbox_spire_bootstrap_reviewer_secret.stdout | from_json).data.token is defined
- (sandbox_spire_bootstrap_reviewer_secret.stdout | from_json).data['ca.crt'] is defined
no_log: true
- name: Wait for the sandbox SPIRE controller credential
ansible.builtin.command:
argv:
- k3s
- kubectl
- --namespace
- "{{ sandbox_spire_bootstrap_source_namespace }}"
- get
- secret
- "{{ sandbox_spire_bootstrap_controller_secret }}"
- --output=json
register: sandbox_spire_bootstrap_controller_secret_result
changed_when: false
retries: 60
delay: 10
until:
- sandbox_spire_bootstrap_controller_secret_result.rc == 0
- (sandbox_spire_bootstrap_controller_secret_result.stdout | from_json).data.token is defined
- (sandbox_spire_bootstrap_controller_secret_result.stdout | from_json).data['ca.crt'] is defined
no_log: true
- name: Extract the sandbox TokenReview credential data
ansible.builtin.set_fact:
sandbox_spire_bootstrap_secret_data: >-
{{ (sandbox_spire_bootstrap_reviewer_secret.stdout | from_json).data }}
sandbox_spire_bootstrap_controller_data: >-
{{ (sandbox_spire_bootstrap_controller_secret_result.stdout | from_json).data }}
no_log: true
- name: Build the restricted sandbox TokenReview kubeconfig
ansible.builtin.set_fact:
sandbox_spire_bootstrap_kubeconfig: |
apiVersion: v1
kind: Config
clusters:
- name: sandbox
cluster:
server: {{ sandbox_spire_bootstrap_api_server }}
certificate-authority-data: {{ sandbox_spire_bootstrap_secret_data['ca.crt'] }}
users:
- name: spire-server-token-reviewer
user:
token: {{ sandbox_spire_bootstrap_secret_data.token | b64decode }}
contexts:
- name: sandbox
context:
cluster: sandbox
user: spire-server-token-reviewer
current-context: sandbox
sandbox_spire_bootstrap_controller_kubeconfig: |
apiVersion: v1
kind: Config
clusters:
- name: sandbox
cluster:
server: {{ sandbox_spire_bootstrap_api_server }}
certificate-authority-data: {{ sandbox_spire_bootstrap_controller_data['ca.crt'] }}
users:
- name: spire-controller-manager
user:
token: {{ sandbox_spire_bootstrap_controller_data.token | b64decode }}
contexts:
- name: sandbox
context:
cluster: sandbox
user: spire-controller-manager
current-context: sandbox
no_log: true
- name: Reconcile the central SPIRE external kubeconfig Secret
ansible.builtin.command:
argv:
- k3s
- kubectl
- apply
- --filename=-
stdin: |
apiVersion: v1
kind: Secret
metadata:
name: {{ sandbox_spire_bootstrap_target_secret }}
namespace: {{ sandbox_spire_bootstrap_target_namespace }}
type: Opaque
data:
sandbox: {{ sandbox_spire_bootstrap_kubeconfig | b64encode }}
sandbox-controller: {{ sandbox_spire_bootstrap_controller_kubeconfig | b64encode }}
delegate_to: localhost
become: true
register: sandbox_spire_bootstrap_target
changed_when: >-
' created' in sandbox_spire_bootstrap_target.stdout or
' configured' in sandbox_spire_bootstrap_target.stdout
no_log: true
@@ -1,60 +0,0 @@
---
- name: Reconcile sandbox LXC nodes
hosts: sandbox_pve
gather_facts: false
serial: 1
roles:
- sandbox_lxc
- name: Reconcile sandbox node OS baseline
hosts: k3s_cluster
gather_facts: false
roles:
- sandbox_base
- name: Reconcile sandbox PostgreSQL primary
hosts: postgres_primary
gather_facts: false
roles:
- sandbox_postgresql
- name: Reconcile sandbox PostgreSQL synchronous standby
hosts: postgres_standby
gather_facts: false
roles:
- sandbox_postgresql
- name: Finalize synchronous PostgreSQL settings
hosts: postgres_primary
gather_facts: false
roles:
- sandbox_postgresql
- name: Validate runtime K3s datastore credential
hosts: k3s_cluster
gather_facts: false
tasks:
- name: Require the Bao-provided PostgreSQL password
ansible.builtin.assert:
that:
- sandbox_postgresql_k3s_password | length > 0
- server_config_yaml is defined
- (server_config_yaml | from_yaml)['datastore-endpoint'] == sandbox_k3s_datastore_endpoint
fail_msg: SANDBOX_K3S_DB_PASSWORD must be populated from Bao
quiet: true
no_log: true
- name: Install the sandbox K3s control plane
ansible.builtin.import_playbook: k3s.orchestration.site
- name: Reconcile Flux controllers and root sync
hosts: sandbox1
gather_facts: false
roles:
- sandbox_flux
- name: Reconcile central SPIRE access to sandbox Kubernetes
hosts: sandbox1
gather_facts: false
roles:
- sandbox_spire_bootstrap
@@ -1,6 +0,0 @@
---
- name: Reconcile central SPIRE access to sandbox Kubernetes
hosts: sandbox1
gather_facts: false
roles:
- sandbox_spire_bootstrap
@@ -1,149 +0,0 @@
---
- name: Verify the PostgreSQL datastore through the VyOS endpoint
hosts: k3s_cluster
gather_facts: false
tasks:
- name: Query the K3s database through the stable endpoint
community.postgresql.postgresql_query:
login_host: "{{ sandbox_postgresql_lb_address }}"
login_db: "{{ sandbox_postgresql_database }}"
login_user: "{{ sandbox_postgresql_user }}"
login_password: "{{ sandbox_postgresql_k3s_password }}"
query: SELECT NOT pg_is_in_recovery() AS writable
register: sandbox_datastore_endpoint
no_log: true
- name: Assert the stable datastore endpoint is writable
ansible.builtin.assert:
that:
- sandbox_datastore_endpoint.query_result[0].writable
fail_msg: The VyOS datastore endpoint is not connected to a writable PostgreSQL primary
quiet: true
- name: Verify synchronous PostgreSQL replication
hosts: postgres_primary
gather_facts: false
tasks:
- name: Query replication state on the primary
community.postgresql.postgresql_query:
login_db: postgres
query: >-
SELECT application_name, state, sync_state
FROM pg_stat_replication
WHERE application_name = 'sandbox2'
become: true
become_user: postgres
register: sandbox_replication_state
- name: Assert sandbox2 is a synchronous streaming standby
ansible.builtin.assert:
that:
- sandbox_replication_state.query_result | length == 1
- sandbox_replication_state.query_result[0].state == 'streaming'
- sandbox_replication_state.query_result[0].sync_state == 'sync'
fail_msg: sandbox2 is not synchronously streaming from sandbox1
quiet: true
- name: Verify PostgreSQL standby recovery state
hosts: postgres_standby
gather_facts: false
tasks:
- name: Query recovery state on the standby
community.postgresql.postgresql_query:
login_db: postgres
query: SELECT pg_is_in_recovery() AS in_recovery
become: true
become_user: postgres
register: sandbox_standby_state
- name: Assert sandbox2 remains in recovery
ansible.builtin.assert:
that:
- sandbox_standby_state.query_result[0].in_recovery
fail_msg: sandbox2 is not operating as a PostgreSQL standby
quiet: true
- name: Verify K3s node prerequisites
hosts: k3s_cluster
gather_facts: false
tasks:
- name: Read the kmsg mapping
ansible.builtin.command:
cmd: readlink /dev/kmsg
register: sandbox_kmsg_target
changed_when: false
- name: Inspect the host kernel module mount
ansible.builtin.shell:
cmd: >-
set -o pipefail &&
findmnt -rn -T "/lib/modules/$(uname -r)" -o OPTIONS |
grep -Eq '(^|,)ro(,|$)'
executable: /bin/bash
changed_when: false
- name: Assert required LXC kernel integration
ansible.builtin.assert:
that:
- sandbox_kmsg_target.stdout == '/dev/console'
fail_msg: LXC is missing its persistent kmsg or read-only host module mapping
quiet: true
- name: Read K3s service state
ansible.builtin.systemd_service:
name: k3s
register: sandbox_k3s_service
- name: Assert K3s is active
ansible.builtin.assert:
that:
- sandbox_k3s_service.status.ActiveState == 'active'
- sandbox_k3s_service.status.SubState == 'running'
fail_msg: K3s is not running
quiet: true
- name: Verify the K3s control plane and API VIP
hosts: sandbox1
gather_facts: false
tasks:
- name: Wait for all declared nodes
ansible.builtin.command:
cmd: k3s kubectl wait --for=condition=Ready nodes --all --timeout=120s
changed_when: false
- name: Read control-plane nodes
ansible.builtin.command:
cmd: >-
k3s kubectl get nodes
-l node-role.kubernetes.io/control-plane=true
-o name
register: sandbox_control_plane_nodes
changed_when: false
- name: Assert both control-plane nodes are registered
ansible.builtin.assert:
that:
- sandbox_control_plane_nodes.stdout_lines | length == 2
fail_msg: The sandbox cluster does not contain both control-plane nodes
quiet: true
- name: Wait for all kube-system Pods
ansible.builtin.command:
cmd: >-
k3s kubectl wait --namespace kube-system
--for=condition=Ready pods --all --timeout=120s
changed_when: false
- name: Verify the authenticated API path through VyOS
ansible.builtin.command:
cmd: >-
k3s kubectl --server=https://10.60.0.13:6443 get --raw=/livez
register: sandbox_api_livez
changed_when: false
- name: Assert the API VIP is live
ansible.builtin.assert:
that:
- sandbox_api_livez.stdout == 'ok'
fail_msg: The K3s API is not healthy through the VyOS VIP
quiet: true
@@ -2,7 +2,6 @@ apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- vmsingle.yaml
- vmsingle-write-route.yaml
- vmagent.yaml
- vmalert.yaml
- vmalertmanager.yaml
@@ -1,24 +0,0 @@
---
# Internal-only remote_write ingress for vmagent instances in other homelab
# clusters. Expose only the write endpoint, not VictoriaMetrics query/admin APIs.
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
name: vmsingle-remote-write
namespace: monitoring
spec:
parentRefs:
- name: eg
namespace: envoy-gateway-system
sectionName: https
hostnames:
- metrics-write.ad.ddupan.top
rules:
- matches:
- method: POST
path:
type: Exact
value: /api/v1/write
backendRefs:
- name: vmsingle-main
port: 8428
-48
View File
@@ -1,48 +0,0 @@
# Sandbox Kata Containers
本目录通过 Flux 安装 Kata Containers 4.1.0,只启用 Cloud Hypervisor 的 Rust
runtime,并创建明确命名的 `kata-clh-runtime-rs` RuntimeClass。OpenSandbox 的 VM
Pool 必须显式选择该 RuntimeClass;不创建含义不明确的 `kata` 默认别名。
OCI chart 同时固定到已验证的 4.1.0 artifact digest,升级时必须重新执行本页验收。
安装使用官方 `kata-deploy` chart 的 `job` 模式。每次 install/upgrade 由 dispatcher
逐个节点创建短生命周期、可修改 host 的安装 Job,写入 Kata artifacts 和 K3s containerd
配置并重启对应节点的 K3s;节点恢复 Ready 后才继续下一节点。安装结束后不保留拥有
host 写权限的 DaemonSet。新增节点后必须触发 HelmRelease upgrade,使 dispatcher
重新枚举节点。
`clh-runtime-rs` 固定使用:
```toml
[runtime]
emptydir_mode = "block-plain"
```
因此普通 `emptyDir` 会在 kubelet volume 目录创建稀疏 backing file,并作为块设备
热插拔给 guest。Docker/BuildKit 可以在 guest ext4 上使用原生 overlay2,避免
virtio-fs 作为 overlayfs upperdir 时的限制。Kata 当前不会用
`emptyDir.sizeLimit` 决定该虚拟盘容量;实际容量取决于节点 rootfs。节点磁盘占用、
Pod 删除后的 backing-file/VMM 回收和真实构建基准必须作为上线验收项目。
`kata-monitor` 常驻每个节点,只读访问 K3s containerd socket 与 Kata sandbox 状态,
由 `VMPodScrape` 写入中央 VictoriaMetrics。它不拥有 Kubernetes API 凭据或 host 写
权限。
## 上线验收
Flux reconciliation 完成后至少确认:
1. 两个节点重新回到 Ready,`RuntimeClass/kata-clh-runtime-rs` 存在;
2. Kata Pod 内核与 LXC host 内核不同,且 `/dev/kvm` 可用;
3. Cloud Hypervisor API 的 `vm.info.config.memory.shared` 为 `true`;
4. `spire-smoke` ServiceAccount 的 Kata Pod 可获得
`spiffe://ddupan.top/sandbox/smoke`,错误 ServiceAccount 无法获得身份;
5. block-backed `emptyDir` 上 Docker 使用 `overlay2`,BuildKit 与 kind smoke test
通过;kind 的 dockerd bootstrap 需要先在 guest 内执行
`mknod /dev/kmsg c 1 11`;
6. 删除测试 Pod 后,Cloud Hypervisor 进程、backing file 和临时数据全部回收,且
LXC 没有新增 OOM 事件;
7. 中央 VictoriaMetrics 中两个 `kata-monitor` target 均为 `up=1`。
不要依赖手工修改 `/opt/kata` 或 K3s containerd 配置;任何修复都必须回写 chart
values 并由 Flux reconciliation。
-40
View File
@@ -1,40 +0,0 @@
---
apiVersion: helm.toolkit.fluxcd.io/v2
kind: HelmRelease
metadata:
name: kata-deploy
namespace: kata-system
spec:
chartRef:
kind: OCIRepository
name: kata-deploy
driftDetection:
mode: enabled
install:
strategy:
name: RetryOnFailure
retryInterval: 5m
interval: 30m
postRenderers:
- kustomize:
patches:
- target:
kind: DaemonSet
name: kata-monitor
patch: |
- op: add
path: /spec/template/spec/automountServiceAccountToken
value: false
- op: add
path: /spec/template/spec/containers/0/ports/0/name
value: metrics
releaseName: kata-deploy
targetNamespace: kata-system
timeout: 30m
upgrade:
strategy:
name: RetryOnFailure
retryInterval: 5m
valuesFrom:
- kind: ConfigMap
name: kata-deploy-values
-9
View File
@@ -1,9 +0,0 @@
---
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- namespace.yaml
- repository.yaml
- values.yaml
- helmrelease.yaml
- monitor-scrape.yaml
-17
View File
@@ -1,17 +0,0 @@
---
apiVersion: operator.victoriametrics.com/v1beta1
kind: VMPodScrape
metadata:
name: kata-monitor
namespace: monitoring
spec:
namespaceSelector:
matchNames:
- kata-system
podMetricsEndpoints:
- interval: 30s
port: metrics
selector:
matchLabels:
app.kubernetes.io/instance: kata-deploy
app.kubernetes.io/name: kata-monitor
-5
View File
@@ -1,5 +0,0 @@
---
apiVersion: v1
kind: Namespace
metadata:
name: kata-system
-11
View File
@@ -1,11 +0,0 @@
---
apiVersion: source.toolkit.fluxcd.io/v1
kind: OCIRepository
metadata:
name: kata-deploy
namespace: kata-system
spec:
interval: 1h
ref:
digest: sha256:33f102f6db70083de4fc8238af4439c4245a601bcb6a72e49bc80529098aefc0
url: oci://ghcr.io/kata-containers/kata-deploy-charts/kata-deploy
-44
View File
@@ -1,44 +0,0 @@
---
apiVersion: v1
kind: ConfigMap
metadata:
name: kata-deploy-values
namespace: kata-system
data:
values.yaml: |
deploymentMode: job
k8sDistribution: k3s
job:
parallelism: 1
snapshotter:
setup: []
shims:
disableAll: true
clh-runtime-rs:
enabled: true
dropIn: |
[runtime]
emptydir_mode = "block-plain"
defaultShim:
amd64: clh-runtime-rs
runtimeClasses:
enabled: true
createDefault: false
node-feature-discovery:
enabled: false
monitor:
enabled: true
resources:
requests:
cpu: 25m
memory: 64Mi
limits:
cpu: 200m
memory: 192Mi
-16
View File
@@ -1,16 +0,0 @@
# Sandbox 监控
该目录只采集 sandbox Kubernetes 与 workload 指标:
- kubelet 与 cAdvisor;
- kube-state-metrics;
- vmagent 自身运行状态;
- SPIRE Agent attestation、SVID 与连接状态。
不得在 sandbox LXC 内部署 node_exporter。LXC 的 `/proc/stat` 暴露 PVE 宿主 CPU
视图,会产生重复且语义混合的指标。PVE 宿主监控不属于本目录。
vmagent 为所有远端样本增加 `cluster=sandbox`,并通过仅允许
`POST /api/v1/write` 的 `metrics-write.ad.ddupan.top` 路由写入 homelab VMSingle。
Operator 与 workload 拆成两个 Flux Kustomization,确保 VM CRD Ready 后再创建
VMAgent、VMNodeScrape 和 VMServiceScrape。
@@ -1,32 +0,0 @@
---
apiVersion: helm.toolkit.fluxcd.io/v2
kind: HelmRelease
metadata:
name: vm-operator
namespace: monitoring
spec:
chart:
spec:
chart: victoria-metrics-operator
interval: 1h
sourceRef:
kind: HelmRepository
name: vm
version: 0.66.2
install:
crds: CreateReplace
strategy:
name: RetryOnFailure
retryInterval: 5m
interval: 30m
releaseName: vm-operator
targetNamespace: monitoring
timeout: 10m
upgrade:
crds: CreateReplace
strategy:
name: RetryOnFailure
retryInterval: 5m
valuesFrom:
- kind: ConfigMap
name: vm-operator-values
@@ -1,8 +0,0 @@
---
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- namespace.yaml
- repositories.yaml
- values.yaml
- helmrelease.yaml
@@ -1,5 +0,0 @@
---
apiVersion: v1
kind: Namespace
metadata:
name: monitoring
@@ -1,18 +0,0 @@
---
apiVersion: source.toolkit.fluxcd.io/v1
kind: HelmRepository
metadata:
name: vm
namespace: monitoring
spec:
interval: 1h
url: https://victoriametrics.github.io/helm-charts/
---
apiVersion: source.toolkit.fluxcd.io/v1
kind: HelmRepository
metadata:
name: prometheus-community
namespace: monitoring
spec:
interval: 1h
url: https://prometheus-community.github.io/helm-charts
@@ -1,23 +0,0 @@
---
apiVersion: v1
kind: ConfigMap
metadata:
name: vm-operator-values
namespace: monitoring
data:
values.yaml: |
crds:
plain: true
cleanup:
enabled: false
operator:
disable_prometheus_converter: true
serviceMonitor:
enabled: false
resources:
requests:
cpu: 25m
memory: 96Mi
limits:
cpu: 250m
memory: 256Mi
@@ -1,14 +0,0 @@
---
apiVersion: operator.victoriametrics.com/v1beta1
kind: VMServiceScrape
metadata:
name: kube-state-metrics
namespace: monitoring
spec:
endpoints:
- interval: 30s
port: http
selector:
matchLabels:
app.kubernetes.io/instance: kube-state-metrics
app.kubernetes.io/name: kube-state-metrics
@@ -1,39 +0,0 @@
---
apiVersion: helm.toolkit.fluxcd.io/v2
kind: HelmRelease
metadata:
name: kube-state-metrics
namespace: monitoring
spec:
chart:
spec:
chart: kube-state-metrics
interval: 1h
sourceRef:
kind: HelmRepository
name: prometheus-community
namespace: monitoring
version: 8.3.0
install:
strategy:
name: RetryOnFailure
retryInterval: 5m
interval: 30m
releaseName: kube-state-metrics
targetNamespace: monitoring
timeout: 10m
upgrade:
strategy:
name: RetryOnFailure
retryInterval: 5m
values:
prometheus:
monitor:
enabled: false
resources:
requests:
cpu: 25m
memory: 64Mi
limits:
cpu: 250m
memory: 256Mi
@@ -1,41 +0,0 @@
---
apiVersion: operator.victoriametrics.com/v1beta1
kind: VMNodeScrape
metadata:
name: kubelet
namespace: monitoring
spec:
bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token
honorLabels: true
honorTimestamps: false
interval: 30s
scheme: https
tlsConfig:
caFile: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
insecureSkipVerify: true
relabelConfigs:
- action: labelmap
regex: __meta_kubernetes_node_label_(.+)
- targetLabel: job
replacement: kubelet
---
apiVersion: operator.victoriametrics.com/v1beta1
kind: VMNodeScrape
metadata:
name: cadvisor
namespace: monitoring
spec:
bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token
honorLabels: true
honorTimestamps: false
interval: 30s
path: /metrics/cadvisor
scheme: https
tlsConfig:
caFile: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
insecureSkipVerify: true
relabelConfigs:
- action: labelmap
regex: __meta_kubernetes_node_label_(.+)
- targetLabel: job
replacement: cadvisor
@@ -1,9 +0,0 @@
---
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- vmagent.yaml
- kubelet-scrapes.yaml
- kube-state-metrics.yaml
- kube-state-metrics-scrape.yaml
- spire-agent-scrape.yaml
@@ -1,17 +0,0 @@
---
apiVersion: operator.victoriametrics.com/v1beta1
kind: VMPodScrape
metadata:
name: spire-agent
namespace: monitoring
spec:
namespaceSelector:
matchNames:
- spire-system
podMetricsEndpoints:
- interval: 30s
port: prom
selector:
matchLabels:
app.kubernetes.io/instance: sandbox-spire
app.kubernetes.io/name: agent
@@ -1,21 +0,0 @@
---
apiVersion: operator.victoriametrics.com/v1beta1
kind: VMAgent
metadata:
name: sandbox
namespace: monitoring
spec:
externalLabels:
cluster: sandbox
remoteWrite:
- url: https://metrics-write.ad.ddupan.top/api/v1/write
replicaCount: 1
resources:
requests:
cpu: 50m
memory: 128Mi
limits:
cpu: 500m
memory: 512Mi
scrapeInterval: 30s
selectAllByDefault: true
@@ -1,4 +0,0 @@
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- pools.yaml
@@ -1,123 +0,0 @@
---
apiVersion: sandbox.opensandbox.io/v1alpha1
kind: Pool
metadata:
name: ci-pod
namespace: opensandbox
spec:
template:
metadata:
labels:
ci.ddupan.top/backend: pod
spec:
containers:
- name: sandbox
image: sandbox-registry.cn-zhangjiakou.cr.aliyuncs.com/opensandbox/code-interpreter:v1.1.0
command: [/opt/opensandbox/task-executor]
args: [-listen-addr=0.0.0.0:5758, -log-dir=/tmp]
env:
- name: SANDBOX_MAIN_CONTAINER
value: sandbox
- name: EXECD_ENVS
value: /opt/opensandbox/.env
- name: EXECD
value: /opt/opensandbox/execd
resources:
requests:
cpu: 100m
memory: 256Mi
limits:
cpu: "2"
memory: 4Gi
volumeMounts:
- name: opensandbox-bin
mountPath: /opt/opensandbox
- name: sandbox-storage
mountPath: /var/lib/sandbox
initContainers:
- name: task-executor-installer
image: sandbox-registry.cn-zhangjiakou.cr.aliyuncs.com/opensandbox/task-executor:v0.1.0
command: [/bin/sh, -c]
args: [cp /workspace/server /opt/opensandbox/task-executor && chmod 0755 /opt/opensandbox/task-executor]
volumeMounts:
- name: opensandbox-bin
mountPath: /opt/opensandbox
- name: execd-installer
image: sandbox-registry.cn-zhangjiakou.cr.aliyuncs.com/opensandbox/execd:v1.0.22
command: [/bin/sh, -c]
args: [cp ./execd /opt/opensandbox/execd && cp ./bootstrap.sh /opt/opensandbox/bootstrap.sh && chmod 0755 /opt/opensandbox/execd /opt/opensandbox/bootstrap.sh]
volumeMounts:
- name: opensandbox-bin
mountPath: /opt/opensandbox
volumes:
- name: opensandbox-bin
emptyDir: {}
- name: sandbox-storage
emptyDir: {}
capacitySpec:
bufferMax: 1
bufferMin: 0
poolMax: 4
poolMin: 0
---
apiVersion: sandbox.opensandbox.io/v1alpha1
kind: Pool
metadata:
name: ci-vm
namespace: opensandbox
spec:
template:
metadata:
labels:
ci.ddupan.top/backend: vm
spec:
runtimeClassName: kata-clh-runtime-rs
containers:
- name: sandbox
image: sandbox-registry.cn-zhangjiakou.cr.aliyuncs.com/opensandbox/code-interpreter:v1.1.0
command: [/opt/opensandbox/task-executor]
args: [-listen-addr=0.0.0.0:5758, -log-dir=/tmp]
env:
- name: SANDBOX_MAIN_CONTAINER
value: sandbox
- name: EXECD_ENVS
value: /opt/opensandbox/.env
- name: EXECD
value: /opt/opensandbox/execd
resources:
requests:
cpu: 250m
memory: 512Mi
limits:
cpu: "4"
memory: 8Gi
volumeMounts:
- name: opensandbox-bin
mountPath: /opt/opensandbox
- name: sandbox-storage
mountPath: /var/lib/sandbox
initContainers:
- name: task-executor-installer
image: sandbox-registry.cn-zhangjiakou.cr.aliyuncs.com/opensandbox/task-executor:v0.1.0
command: [/bin/sh, -c]
args: [cp /workspace/server /opt/opensandbox/task-executor && chmod 0755 /opt/opensandbox/task-executor]
volumeMounts:
- name: opensandbox-bin
mountPath: /opt/opensandbox
- name: execd-installer
image: sandbox-registry.cn-zhangjiakou.cr.aliyuncs.com/opensandbox/execd:v1.0.22
command: [/bin/sh, -c]
args: [cp ./execd /opt/opensandbox/execd && cp ./bootstrap.sh /opt/opensandbox/bootstrap.sh && chmod 0755 /opt/opensandbox/execd /opt/opensandbox/bootstrap.sh]
volumeMounts:
- name: opensandbox-bin
mountPath: /opt/opensandbox
volumes:
- name: opensandbox-bin
emptyDir: {}
- name: sandbox-storage
emptyDir: {}
capacitySpec:
bufferMax: 1
bufferMin: 0
poolMax: 2
poolMin: 0
-19
View File
@@ -1,19 +0,0 @@
# OpenSandbox
Flux installs the upstream all-in-one OpenSandbox chart pinned to
`helm/opensandbox/0.2.2` (`8f01e935`). The API is cluster-internal and intentionally runs a
single replica until shared server state and HA behaviour have been validated.
`ci-pod` uses `runc`; `ci-vm` uses the separately managed
`kata-clh-runtime-rs` RuntimeClass. Both Pools start at zero and create capacity
on demand. They currently use the upstream interpreter image to validate the
Lifecycle API and Pool allocation independently of the CI scheduler cutover.
The dynamic runner worker, runner image, guest-local SPIRE Agent and Docker
sidecar are introduced only after this layer is Ready. In particular, do not
mount the host SPIFFE CSI socket into `ci-vm`: Unix sockets do not cross the
Kata VM boundary.
Smoke test both backends through the same API by creating sandboxes with
`extensions.poolRef` set to `ci-pod` and `ci-vm`, then confirm their
BatchSandboxes, Pods and VMMs disappear after deletion.
@@ -1,31 +0,0 @@
apiVersion: helm.toolkit.fluxcd.io/v2
kind: HelmRelease
metadata:
name: opensandbox
namespace: opensandbox-system
spec:
chart:
spec:
chart: ./kubernetes/charts/opensandbox
interval: 1h
reconcileStrategy: Revision
sourceRef:
kind: GitRepository
name: opensandbox
driftDetection:
mode: enabled
install:
strategy:
name: RetryOnFailure
retryInterval: 5m
interval: 30m
releaseName: opensandbox
targetNamespace: opensandbox-system
timeout: 15m
upgrade:
strategy:
name: RetryOnFailure
retryInterval: 5m
valuesFrom:
- kind: ConfigMap
name: opensandbox-values
@@ -1,7 +0,0 @@
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- namespaces.yaml
- repository.yaml
- values.yaml
- helmrelease.yaml
@@ -1,10 +0,0 @@
---
apiVersion: v1
kind: Namespace
metadata:
name: opensandbox-system
---
apiVersion: v1
kind: Namespace
metadata:
name: opensandbox
@@ -1,11 +0,0 @@
apiVersion: source.toolkit.fluxcd.io/v1
kind: GitRepository
metadata:
name: opensandbox
namespace: opensandbox-system
spec:
interval: 1h
ref:
tag: helm/opensandbox/0.2.2
timeout: 60s
url: https://github.com/alibaba/OpenSandbox.git
-62
View File
@@ -1,62 +0,0 @@
apiVersion: v1
kind: ConfigMap
metadata:
name: opensandbox-values
namespace: opensandbox-system
data:
values.yaml: |
opensandbox-controller:
controller:
logLevel: info
replicaCount: 1
metrics:
enabled: true
secure: false
port: 8080
resources:
requests:
cpu: 25m
memory: 64Mi
limits:
cpu: 500m
memory: 256Mi
opensandbox-server:
server:
replicaCount: 1
resources:
requests:
cpu: 100m
memory: 256Mi
limits:
cpu: "1"
memory: 1Gi
configToml: |
[server]
host = "0.0.0.0"
port = 80
api_key = ""
[log]
level = "INFO"
[runtime]
type = "kubernetes"
execd_image = "sandbox-registry.cn-zhangjiakou.cr.aliyuncs.com/opensandbox/execd:v1.0.22"
[kubernetes]
kubeconfig_path = ""
namespace = "opensandbox"
informer_enabled = true
informer_resync_seconds = 300
informer_watch_timeout_seconds = 60
snapshot_create_timeout_seconds = 900
workload_provider = "batchsandbox"
batchsandbox_template_file = "/etc/opensandbox/example.batchsandbox-template.yaml"
[egress]
image = "sandbox-registry.cn-zhangjiakou.cr.aliyuncs.com/opensandbox/egress:v1.1.6"
mode = "dns+nft"
opensandbox-node-agent:
enabled: false
@@ -1,7 +0,0 @@
---
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- values.yaml
- release.yaml
- smoke-identity.yaml
@@ -1,35 +0,0 @@
---
apiVersion: helm.toolkit.fluxcd.io/v2
kind: HelmRelease
metadata:
name: sandbox-spire
namespace: spire-mgmt
spec:
chart:
spec:
chart: spire
interval: 1h
sourceRef:
kind: HelmRepository
name: spiffe-hardened
version: 0.30.2
dependsOn:
- name: spire-crds
namespace: spire-mgmt
driftDetection:
mode: enabled
install:
strategy:
name: RetryOnFailure
retryInterval: 5m
interval: 30m
releaseName: sandbox-spire
targetNamespace: spire-mgmt
timeout: 15m
upgrade:
strategy:
name: RetryOnFailure
retryInterval: 5m
valuesFrom:
- kind: ConfigMap
name: sandbox-spire-values
@@ -1,28 +0,0 @@
---
apiVersion: v1
kind: Namespace
metadata:
name: spire-smoke
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: spire-smoke
namespace: spire-smoke
---
apiVersion: spire.spiffe.io/v1alpha1
kind: ClusterSPIFFEID
metadata:
name: sandbox-spire-smoke
spec:
className: spire-mgmt-spire
namespaceSelector:
matchLabels:
kubernetes.io/metadata.name: spire-smoke
podSelector:
matchLabels:
app.kubernetes.io/name: spire-smoke
spiffeIDTemplate: spiffe://{{ .TrustDomain }}/sandbox/smoke
workloadSelectorTemplates:
- k8s:ns:spire-smoke
- k8s:sa:spire-smoke
-86
View File
@@ -1,86 +0,0 @@
---
apiVersion: v1
kind: ConfigMap
metadata:
name: sandbox-spire-values
namespace: spire-mgmt
data:
values.yaml: |
global:
k8s:
clusterDomain: cluster.local
spire:
bundleConfigMap: spire-bundle
clusterName: sandbox
trustDomain: ddupan.top
namespaces:
create: false
system:
name: spire-system
server:
name: spire-server
recommendations:
enabled: true
namespaceLayout: true
namespacePSS: true
priorityClassName: true
strictMode: true
securityContexts: true
prometheus: false
spire-server:
enabled: false
spire-agent:
enabled: true
serviceAccount:
name: spire-agent
server:
address: spire-server.ad.ddupan.top
port: 8081
nodeAttestor:
k8sPSAT:
enabled: true
workloadAttestors:
k8s:
enabled: true
unix:
enabled: false
telemetry:
prometheus:
enabled: true
podMonitor:
enabled: false
resources:
requests:
cpu: 25m
memory: 64Mi
limits:
cpu: 250m
memory: 192Mi
spiffe-csi-driver:
enabled: true
resources:
requests:
cpu: 10m
memory: 32Mi
limits:
cpu: 100m
memory: 96Mi
spiffe-oidc-discovery-provider:
enabled: false
upstream:
enabled: false
tornjak-frontend:
enabled: false
spire-identity-exchange:
enabled: false
spike-keeper:
enabled: false
spike-nexus:
enabled: false
spike-pilot:
enabled: false
@@ -1,162 +0,0 @@
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: spire-controller-manager
namespace: spire-system
---
apiVersion: v1
kind: Secret
metadata:
name: spire-controller-manager-token
namespace: spire-system
annotations:
kubernetes.io/service-account.name: spire-controller-manager
type: kubernetes.io/service-account-token
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: spire-controller-manager
rules:
- apiGroups:
- ""
resources:
- endpoints
- namespaces
- nodes
- pods
verbs:
- get
- list
- watch
- apiGroups:
- spire.spiffe.io
resources:
- clusterfederatedtrustdomains
- clusterspiffeids
- clusterstaticentries
verbs:
- create
- delete
- get
- list
- patch
- update
- watch
- apiGroups:
- spire.spiffe.io
resources:
- clusterfederatedtrustdomains/finalizers
- clusterspiffeids/finalizers
- clusterstaticentries/finalizers
verbs:
- update
- apiGroups:
- spire.spiffe.io
resources:
- clusterfederatedtrustdomains/status
- clusterspiffeids/status
- clusterstaticentries/status
verbs:
- get
- patch
- update
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: spire-controller-manager
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: spire-controller-manager
subjects:
- kind: ServiceAccount
name: spire-controller-manager
namespace: spire-system
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
name: spire-bundle-publisher
namespace: spire-system
rules:
- apiGroups:
- ""
resources:
- configmaps
verbs:
- create
- delete
- get
- list
- patch
- update
- watch
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: spire-bundle-publisher
namespace: spire-system
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: Role
name: spire-bundle-publisher
subjects:
- kind: ServiceAccount
name: spire-controller-manager
namespace: spire-system
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
name: spire-controller-manager-leader-election
namespace: spire-server
rules:
- apiGroups:
- ""
resources:
- configmaps
verbs:
- create
- delete
- get
- list
- patch
- update
- watch
- apiGroups:
- coordination.k8s.io
resources:
- leases
verbs:
- create
- delete
- get
- list
- patch
- update
- watch
- apiGroups:
- ""
resources:
- events
verbs:
- create
- patch
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: spire-controller-manager-leader-election
namespace: spire-server
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: Role
name: spire-controller-manager-leader-election
subjects:
- kind: ServiceAccount
name: spire-controller-manager
namespace: spire-system
@@ -1,31 +0,0 @@
---
apiVersion: helm.toolkit.fluxcd.io/v2
kind: HelmRelease
metadata:
name: spire-crds
namespace: spire-mgmt
spec:
chart:
spec:
chart: spire-crds
interval: 1h
sourceRef:
kind: HelmRepository
name: spiffe-hardened
version: 0.6.1
driftDetection:
mode: enabled
install:
crds: CreateReplace
strategy:
name: RetryOnFailure
retryInterval: 5m
interval: 30m
releaseName: spire-crds
targetNamespace: spire-mgmt
timeout: 10m
upgrade:
crds: CreateReplace
strategy:
name: RetryOnFailure
retryInterval: 5m
@@ -1,9 +0,0 @@
---
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- namespaces.yaml
- repository.yaml
- crds.yaml
- token-reviewer.yaml
- controller-manager.yaml
@@ -1,15 +0,0 @@
---
apiVersion: v1
kind: Namespace
metadata:
name: spire-mgmt
---
apiVersion: v1
kind: Namespace
metadata:
name: spire-system
---
apiVersion: v1
kind: Namespace
metadata:
name: spire-server
@@ -1,9 +0,0 @@
---
apiVersion: source.toolkit.fluxcd.io/v1
kind: HelmRepository
metadata:
name: spiffe-hardened
namespace: spire-mgmt
spec:
interval: 1h
url: https://spiffe.github.io/helm-charts-hardened/
@@ -1,51 +0,0 @@
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: spire-server-token-reviewer
namespace: spire-system
---
apiVersion: v1
kind: Secret
metadata:
name: spire-server-token-reviewer-token
namespace: spire-system
annotations:
kubernetes.io/service-account.name: spire-server-token-reviewer
type: kubernetes.io/service-account-token
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: spire-server-token-reviewer
rules:
- apiGroups:
- authentication.k8s.io
resources:
- tokenreviews
verbs:
- get
- list
- watch
- create
- apiGroups:
- ""
resources:
- nodes
- pods
verbs:
- get
- list
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: spire-server-token-reviewer
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: spire-server-token-reviewer
subjects:
- kind: ServiceAccount
name: spire-server-token-reviewer
namespace: spire-system
+1 -13
View File
@@ -11,8 +11,7 @@ Authelia 提供;SPIRE 不替代人类 OIDC,也不承担目标服务的资源
Flux 安装 SPIFFE hardened charts:
- `spire-crds` `0.6.1`;
- 内部 fork 的 `spire` `0.30.2-ddupan.1`(SPIRE `1.15.3`),固定 Git tag
`spire-0.30.2-ddupan.1`;
- `spire` `0.30.2`(SPIRE `1.15.3`);
- SPIRE Server、Agent、Controller Manager、SPIFFE CSI Driver;
- OIDC Discovery Provider。
@@ -20,17 +19,6 @@ Flux 安装 SPIFFE hardened charts:
API 或 Broker API。Trust domain 是 `ddupan.top`,Kubernetes cluster name 是
`homelab`。
同一 Server 也接受 cluster name 为 `sandbox` 的 external PSAT attestation。SPIRE gRPC
只通过内网 `spire-server.ad.ddupan.top:8081` 暴露;external PSAT、external
controller-manager 与 bundle publisher 使用由 sandbox Ansible bootstrap 的独立、受限
kubeconfig。Sandbox 不运行第二套 Server 或 OIDC Provider。
内部 fork 仅在上游 `spire-0.30.2` 基础上暴露
`use_pod_uid_for_agent_id`。现有 `sandbox` profile 保持 node UID 模式,供 DaemonSet
Agent 使用;独立的 `sandbox-kata` profile 复用同一 kubeconfig,但启用 Pod UID 模式,
供每个 Kata guest 内的临时 Agent 使用。不得把现有 `sandbox` profile 切换为 Pod UID,
否则会改变常驻 Agent 的 parent ID。
## PostgreSQL bootstrap
SPIRE registration datastore 使用共享 CloudNativePG:
+1 -1
View File
@@ -41,7 +41,7 @@ SPIRE Server(trust domain: ddupan.top)
| 项目 | 当前值 |
|---|---|
| SPIRE chart | `0.30.2-ddupan.1`(内部 fork,基于 `0.30.2`) |
| SPIRE chart | `0.30.2` |
| SPIRE | `1.15.3` |
| SPIRE CRDs chart | `0.6.1` |
| trust domain | `ddupan.top` |
-11
View File
@@ -1,11 +0,0 @@
apiVersion: source.toolkit.fluxcd.io/v1
kind: GitRepository
metadata:
name: spiffe-hardened-fork
namespace: spire-mgmt
spec:
interval: 1h
ref:
tag: spire-0.30.2-ddupan.1
timeout: 60s
url: http://gitea-http.gitea.svc.cluster.local:3000/panxiao81/helm-charts-hardened.git
+4 -4
View File
@@ -6,12 +6,12 @@ metadata:
spec:
chart:
spec:
chart: ./charts/spire
chart: spire
interval: 1h
reconcileStrategy: Revision
sourceRef:
kind: GitRepository
name: spiffe-hardened-fork
kind: HelmRepository
name: spiffe-hardened
version: 0.30.2
dependsOn:
- name: spire-crds
namespace: spire-mgmt
-1
View File
@@ -12,7 +12,6 @@ configMapGenerator:
resources:
- namespaces.yaml
- helmrepository.yaml
- gitrepository-fork.yaml
- helmrelease-crds.yaml
- helmrelease.yaml
- httproute.yaml
-46
View File
@@ -30,47 +30,6 @@ spire-server:
kind: statefulset
replicaCount: 1
auditLogEnabled: true
service:
type: LoadBalancer
port: 8081
loadBalancerIP: 192.168.10.127
kubeConfigs:
sandbox:
externalSecret:
name: spire-external-kubeconfigs
key: sandbox
sandbox-controller:
externalSecret:
name: spire-external-kubeconfigs
key: sandbox-controller
nodeAttestor:
externalK8sPSAT:
enabled: true
clusters:
sandbox:
kubeConfigName: sandbox
serviceAccountAllowList:
- spire-system:spire-agent
sandbox-kata:
kubeConfigName: sandbox
serviceAccountAllowList:
- spire-smoke:spire-smoke
usePodUIDForAgentID: true
externalControllerManagers:
enabled: true
clusters:
sandbox:
kubeConfigName: sandbox-controller
bundlePublisher:
externalK8sConfigMap:
enabled: true
clusters:
sandbox:
kubeConfigName: sandbox-controller
namespace: spire-system
configMapName: spire-bundle
configMapKey: bundle.spiffe
format: spiffe
persistence:
# PostgreSQL stores registrations, but the disk KeyManager still needs durable
# storage for the trust-domain signing keys.
@@ -106,11 +65,6 @@ spire-server:
enabled: false
spire-agent:
server:
# Keep the Agent endpoint aligned with spire-server.service.port. The
# chart defaults this to 443, which only remained unnoticed while the
# Agent's pre-upgrade gRPC connection stayed alive.
port: 8081
nodeAttestor:
k8sPSAT:
enabled: true