Compare commits
3
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
54dd52f0ed
|
||
|
|
76ec49ff48
|
||
|
|
00671e1729
|
@@ -58,4 +58,4 @@ networks:
|
|||||||
default:
|
default:
|
||||||
ipam:
|
ipam:
|
||||||
config:
|
config:
|
||||||
- subnet: 172.28.0.0/24
|
- subnet: 172.28.0.0/24
|
||||||
|
|||||||
@@ -18,4 +18,4 @@ networks:
|
|||||||
default:
|
default:
|
||||||
ipam:
|
ipam:
|
||||||
config:
|
config:
|
||||||
- subnet: 172.28.1.0/24
|
- subnet: 172.28.1.0/24
|
||||||
|
|||||||
@@ -50,7 +50,6 @@ sudo k3s kubectl -n flux-system get gitrepositories,kustomizations
|
|||||||
- VictoriaMetrics Operator 已固定现有 chart `0.66.2` 并完成分阶段 Flux HelmRelease
|
- VictoriaMetrics Operator 已固定现有 chart `0.66.2` 并完成分阶段 Flux HelmRelease
|
||||||
接管;Metrics、Logs、Traces 与 Grafana 也已统一完成 Flux 接管;
|
接管;Metrics、Logs、Traces 与 Grafana 也已统一完成 Flux 接管;
|
||||||
- External Secrets Operator 已固定 chart `2.8.0` 并完成分阶段接管;
|
- External Secrets Operator 已固定 chart `2.8.0` 并完成分阶段接管;
|
||||||
- SPIRE 已按 hardened chart 内部 fork `0.30.2-ddupan.1`(基于上游 `0.30.2`,SPIRE
|
- SPIRE 已按官方 hardened chart `0.30.2`(SPIRE `1.15.3`)声明,使用共享
|
||||||
`1.15.3`)声明,使用共享
|
|
||||||
PostgreSQL 与独立 signing-key PVC;首次上线和 OpenBao JWT-SVID PoC 尚待合并后验证;
|
PostgreSQL 与独立 signing-key PVC;首次上线和 OpenBao JWT-SVID PoC 尚待合并后验证;
|
||||||
- root Kustomization 与所有 brownfield 子 Kustomization 继续保持 `prune: false`。
|
- root Kustomization 与所有 brownfield 子 Kustomization 继续保持 `prune: false`。
|
||||||
|
|||||||
@@ -1,61 +0,0 @@
|
|||||||
# Sandbox 集群
|
|
||||||
|
|
||||||
这里是 OpenSandbox、CI 和 AI Agent workload 所在双节点 k3s 集群的 Flux
|
|
||||||
reconciliation 入口。LXC、PostgreSQL、K3s、固定版本的 Flux controllers 与 root
|
|
||||||
sync 由 `infrastructure/sandbox-cluster/` 中的 Ansible 管理;本目录只组合集群内
|
|
||||||
workload。
|
|
||||||
|
|
||||||
Flux 通过 `https://git.ddupan.top/panxiao81/homelab-infra.git` 读取公开仓库。
|
|
||||||
Ansible 将 homelab CA 注入 `GitRepository/flux-system` 引用的同名 Secret,不使用
|
|
||||||
长期 Git 凭据。root Kustomization 从 `./clusters/sandbox` 开始 reconciliation,
|
|
||||||
初始保持 `prune: false`。
|
|
||||||
|
|
||||||
Root bootstrap 已完成。后续按依赖顺序分别引入:
|
|
||||||
|
|
||||||
1. 监控 CRD、kube-state-metrics 以及 kubelet/cAdvisor 抓取配置;
|
|
||||||
2. SPIRE Agent、SPIFFE CSI Driver 与 workload registration;
|
|
||||||
3. Kata Containers、`block-plain` RuntimeClass;
|
|
||||||
4. OpenSandbox operator/server 及 `ci-pod`、`ci-vm` Pools。
|
|
||||||
|
|
||||||
每一阶段单独合并并等待对应 Flux Kustomization Ready,不在 bootstrap 时一次性部署。
|
|
||||||
第一阶段监控拆为 `monitoring-operator` 与依赖它的 `monitoring`,防止 VM CR 在
|
|
||||||
VictoriaMetrics Operator CRD Ready 前进入 reconciliation。
|
|
||||||
|
|
||||||
SPIRE 阶段先由 `spire-bootstrap` 安装 CRD,并声明按上游 k8s_psat Server plugin
|
|
||||||
要求收窄的 reviewer:它可以调用 TokenReview,并只读查询用于证明的 Pod 与 Node。
|
|
||||||
Agent ServiceAccount 留给后续 HelmRelease 创建,避免两个声明方争夺同一资源。随后运行
|
|
||||||
`infrastructure/sandbox-cluster/ansible/spire-bootstrap.yml`:playbook 从 sandbox
|
|
||||||
读取 reviewer token,在内存中组成受限 kubeconfig,再通过 stdin reconcile 到 central
|
|
||||||
集群的 `spire-server/spire-external-kubeconfigs` Secret。凭据不写入仓库、日志或控制机
|
|
||||||
文件;该 Secret 准备完成后,才能启用 central external PSAT/controller-manager 和
|
|
||||||
sandbox Agent/CSI。
|
|
||||||
|
|
||||||
External controller-manager 使用独立的 `spire-controller-manager` ServiceAccount;其
|
|
||||||
RBAC 与上游 controller-manager 所需权限一致,用于读取 workload selectors、维护
|
|
||||||
SPIFFE CR status/finalizer 和 leader election。它不复用只允许 TokenReview 的 Server
|
|
||||||
reviewer。Ansible 将两份 kubeconfig 写入同一个 central Secret 的不同 key,便于 central
|
|
||||||
chart 分别绑定 `sandbox` 与 `sandbox-controller`。
|
|
||||||
|
|
||||||
Central SPIRE Server 通过内网 `spire-server.ad.ddupan.top:8081` 接收 sandbox Agent
|
|
||||||
attestation。Server 使用 external bundle publisher 持续维护 sandbox
|
|
||||||
`spire-system/spire-bundle`,Agent 不固定或复制 trust bundle。Sandbox HelmRelease
|
|
||||||
显式关闭 Server 与 OIDC Provider,只部署 Agent DaemonSet 和 SPIFFE CSI Driver;因此
|
|
||||||
不会产生第二个 trust root。
|
|
||||||
|
|
||||||
`spire-smoke` namespace、ServiceAccount 和 `sandbox-spire-smoke` ClusterSPIFFEID 是
|
|
||||||
普通 Pod 与后续 Kata guest 的回归夹具,稳定身份为
|
|
||||||
`spiffe://ddupan.top/sandbox/smoke`。测试 Pod 临时创建并在验收后删除,身份声明保留。
|
|
||||||
|
|
||||||
Kata 阶段使用官方 4.1.0 `kata-deploy` chart 的短生命周期 `job` 模式,逐节点安装并
|
|
||||||
重启 K3s。只启用 `kata-clh-runtime-rs`,不创建默认 `kata` 别名;该 handler 的
|
|
||||||
`emptyDir` 固定使用 `block-plain`,为 Docker/BuildKit overlay2 与 kind 提供 guest
|
|
||||||
内块设备文件系统。详细限制与上线验收见 `platform/sandbox-kata/README.md`。
|
|
||||||
|
|
||||||
## 监控边界
|
|
||||||
|
|
||||||
这里只管理 sandbox LXC 内的 Kubernetes 监控,不负责 PVE 宿主监控。LXC 与宿主共享
|
|
||||||
内核,即使 lxcfs 虚拟化了内存和 uptime,容器内 `/proc/stat` 仍是宿主 CPU 视图;
|
|
||||||
在 LXC 内运行 node_exporter 会生成混合语义并重复采集宿主指标,因此禁止部署。
|
|
||||||
|
|
||||||
Sandbox 节点与 workload 指标来自 kubelet/cAdvisor 和 kube-state-metrics;K3s 或 LXC
|
|
||||||
特有但上述接口未覆盖的指标,应使用目标明确的 collector,不以 node_exporter 补齐。
|
|
||||||
@@ -1,18 +0,0 @@
|
|||||||
---
|
|
||||||
apiVersion: kustomize.toolkit.fluxcd.io/v1
|
|
||||||
kind: Kustomization
|
|
||||||
metadata:
|
|
||||||
name: kata
|
|
||||||
namespace: flux-system
|
|
||||||
spec:
|
|
||||||
dependsOn:
|
|
||||||
- name: monitoring-operator
|
|
||||||
- name: spire-agents
|
|
||||||
interval: 10m
|
|
||||||
path: ./platform/sandbox-kata
|
|
||||||
prune: true
|
|
||||||
sourceRef:
|
|
||||||
kind: GitRepository
|
|
||||||
name: flux-system
|
|
||||||
timeout: 35m
|
|
||||||
wait: true
|
|
||||||
@@ -1,15 +0,0 @@
|
|||||||
---
|
|
||||||
apiVersion: kustomize.toolkit.fluxcd.io/v1
|
|
||||||
kind: Kustomization
|
|
||||||
metadata:
|
|
||||||
name: monitoring-operator
|
|
||||||
namespace: flux-system
|
|
||||||
spec:
|
|
||||||
interval: 10m
|
|
||||||
path: ./platform/sandbox-monitoring/operator
|
|
||||||
prune: true
|
|
||||||
sourceRef:
|
|
||||||
kind: GitRepository
|
|
||||||
name: flux-system
|
|
||||||
timeout: 10m
|
|
||||||
wait: true
|
|
||||||
@@ -1,17 +0,0 @@
|
|||||||
---
|
|
||||||
apiVersion: kustomize.toolkit.fluxcd.io/v1
|
|
||||||
kind: Kustomization
|
|
||||||
metadata:
|
|
||||||
name: monitoring
|
|
||||||
namespace: flux-system
|
|
||||||
spec:
|
|
||||||
dependsOn:
|
|
||||||
- name: monitoring-operator
|
|
||||||
interval: 10m
|
|
||||||
path: ./platform/sandbox-monitoring/workloads
|
|
||||||
prune: true
|
|
||||||
sourceRef:
|
|
||||||
kind: GitRepository
|
|
||||||
name: flux-system
|
|
||||||
timeout: 10m
|
|
||||||
wait: true
|
|
||||||
@@ -1,17 +0,0 @@
|
|||||||
---
|
|
||||||
apiVersion: kustomize.toolkit.fluxcd.io/v1
|
|
||||||
kind: Kustomization
|
|
||||||
metadata:
|
|
||||||
name: opensandbox-pools
|
|
||||||
namespace: flux-system
|
|
||||||
spec:
|
|
||||||
dependsOn:
|
|
||||||
- name: opensandbox
|
|
||||||
interval: 10m
|
|
||||||
path: ./platform/sandbox-opensandbox-pools
|
|
||||||
prune: true
|
|
||||||
sourceRef:
|
|
||||||
kind: GitRepository
|
|
||||||
name: flux-system
|
|
||||||
timeout: 20m
|
|
||||||
wait: true
|
|
||||||
@@ -1,18 +0,0 @@
|
|||||||
---
|
|
||||||
apiVersion: kustomize.toolkit.fluxcd.io/v1
|
|
||||||
kind: Kustomization
|
|
||||||
metadata:
|
|
||||||
name: opensandbox
|
|
||||||
namespace: flux-system
|
|
||||||
spec:
|
|
||||||
dependsOn:
|
|
||||||
- name: kata
|
|
||||||
- name: spire-agents
|
|
||||||
interval: 10m
|
|
||||||
path: ./platform/sandbox-opensandbox
|
|
||||||
prune: true
|
|
||||||
sourceRef:
|
|
||||||
kind: GitRepository
|
|
||||||
name: flux-system
|
|
||||||
timeout: 20m
|
|
||||||
wait: true
|
|
||||||
@@ -1,18 +0,0 @@
|
|||||||
---
|
|
||||||
apiVersion: kustomize.toolkit.fluxcd.io/v1
|
|
||||||
kind: Kustomization
|
|
||||||
metadata:
|
|
||||||
name: spire-agents
|
|
||||||
namespace: flux-system
|
|
||||||
spec:
|
|
||||||
dependsOn:
|
|
||||||
- name: spire-bootstrap
|
|
||||||
- name: monitoring-operator
|
|
||||||
interval: 10m
|
|
||||||
path: ./platform/sandbox-spire/agents
|
|
||||||
prune: true
|
|
||||||
sourceRef:
|
|
||||||
kind: GitRepository
|
|
||||||
name: flux-system
|
|
||||||
timeout: 15m
|
|
||||||
wait: true
|
|
||||||
@@ -1,15 +0,0 @@
|
|||||||
---
|
|
||||||
apiVersion: kustomize.toolkit.fluxcd.io/v1
|
|
||||||
kind: Kustomization
|
|
||||||
metadata:
|
|
||||||
name: spire-bootstrap
|
|
||||||
namespace: flux-system
|
|
||||||
spec:
|
|
||||||
interval: 10m
|
|
||||||
path: ./platform/sandbox-spire/bootstrap
|
|
||||||
prune: true
|
|
||||||
sourceRef:
|
|
||||||
kind: GitRepository
|
|
||||||
name: flux-system
|
|
||||||
timeout: 10m
|
|
||||||
wait: true
|
|
||||||
@@ -1,11 +0,0 @@
|
|||||||
---
|
|
||||||
apiVersion: kustomize.config.k8s.io/v1beta1
|
|
||||||
kind: Kustomization
|
|
||||||
resources:
|
|
||||||
- apps/monitoring-operator.yaml
|
|
||||||
- apps/monitoring.yaml
|
|
||||||
- apps/spire-bootstrap.yaml
|
|
||||||
- apps/spire-agents.yaml
|
|
||||||
- apps/kata.yaml
|
|
||||||
- apps/opensandbox.yaml
|
|
||||||
- apps/opensandbox-pools.yaml
|
|
||||||
@@ -15,12 +15,10 @@ homelab_dns:
|
|||||||
- { zone: ad.ddupan.top, name: sandbox-k8s, type: A, values: [10.60.0.13] }
|
- { zone: ad.ddupan.top, name: sandbox-k8s, type: A, values: [10.60.0.13] }
|
||||||
- { zone: ad.ddupan.top, name: retrolab, type: A, values: [10.60.0.10] }
|
- { zone: ad.ddupan.top, name: retrolab, type: A, values: [10.60.0.10] }
|
||||||
- { zone: ad.ddupan.top, name: grafana, type: A, values: [192.168.10.127] }
|
- { zone: ad.ddupan.top, name: grafana, type: A, values: [192.168.10.127] }
|
||||||
- { zone: ad.ddupan.top, name: metrics-write, type: A, values: [192.168.10.127] }
|
|
||||||
- { zone: ad.ddupan.top, name: netbox, type: A, values: [192.168.10.127] }
|
- { zone: ad.ddupan.top, name: netbox, type: A, values: [192.168.10.127] }
|
||||||
- { zone: ad.ddupan.top, name: nats, type: A, values: [192.168.10.127] }
|
- { zone: ad.ddupan.top, name: nats, type: A, values: [192.168.10.127] }
|
||||||
- { zone: ad.ddupan.top, name: s3, type: A, values: [192.168.10.127] }
|
- { zone: ad.ddupan.top, name: s3, type: A, values: [192.168.10.127] }
|
||||||
- { zone: ad.ddupan.top, name: spire-oidc, type: A, values: [192.168.10.127] }
|
- { zone: ad.ddupan.top, name: spire-oidc, type: A, values: [192.168.10.127] }
|
||||||
- { zone: ad.ddupan.top, name: spire-server, type: A, values: [192.168.10.127] }
|
|
||||||
- { zone: ad.ddupan.top, name: zot, type: A, values: [192.168.10.127] }
|
- { zone: ad.ddupan.top, name: zot, type: A, values: [192.168.10.127] }
|
||||||
- { zone: ad.ddupan.top, name: zot-push, type: A, values: [192.168.10.127] }
|
- { zone: ad.ddupan.top, name: zot-push, type: A, values: [192.168.10.127] }
|
||||||
|
|
||||||
@@ -62,14 +60,14 @@ homelab_dns:
|
|||||||
- name: oci-arm.ddupan.top
|
- name: oci-arm.ddupan.top
|
||||||
type: A
|
type: A
|
||||||
values:
|
values:
|
||||||
- 129.225.138.179
|
- 129.225.138.179
|
||||||
proxied: false
|
proxied: false
|
||||||
ttl: 300
|
ttl: 300
|
||||||
terraform: { managed: false }
|
terraform: { managed: false }
|
||||||
- name: oci-amd.ddupan.top
|
- name: oci-amd.ddupan.top
|
||||||
type: A
|
type: A
|
||||||
values:
|
values:
|
||||||
- 129.225.176.134
|
- 129.225.176.134
|
||||||
proxied: false
|
proxied: false
|
||||||
ttl: 300
|
ttl: 300
|
||||||
terraform: { managed: false }
|
terraform: { managed: false }
|
||||||
|
|||||||
@@ -77,14 +77,3 @@ vyos_ospf_networks:
|
|||||||
vyos_postgresql_listen_address: "10.60.0.1"
|
vyos_postgresql_listen_address: "10.60.0.1"
|
||||||
vyos_postgresql_port: 5432
|
vyos_postgresql_port: 5432
|
||||||
vyos_postgresql_primary_address: "10.60.0.11"
|
vyos_postgresql_primary_address: "10.60.0.11"
|
||||||
|
|
||||||
# Stable API endpoint for the two-node sandbox K3s control plane. HAProxy owns
|
|
||||||
# this /32 on the existing labnet interface and performs only TCP health checks.
|
|
||||||
vyos_sandbox_k3s_api_address: "10.60.0.13"
|
|
||||||
vyos_sandbox_k3s_api_interface: eth1
|
|
||||||
vyos_sandbox_k3s_api_port: 6443
|
|
||||||
vyos_sandbox_k3s_servers:
|
|
||||||
- name: sandbox1
|
|
||||||
address: "10.60.0.11"
|
|
||||||
- name: sandbox2
|
|
||||||
address: "10.60.0.12"
|
|
||||||
|
|||||||
@@ -3,23 +3,6 @@
|
|||||||
# `set` lines against the running config and issues only the differences, so
|
# `set` lines against the running config and issues only the differences, so
|
||||||
# re-runs are no-ops. Supports --check and --diff.
|
# re-runs are no-ops. Supports --check and --diff.
|
||||||
|
|
||||||
- name: Read current VyOS configuration commands
|
|
||||||
vyos.vyos.vyos_command:
|
|
||||||
commands:
|
|
||||||
- show configuration commands
|
|
||||||
register: _vyos_running_commands
|
|
||||||
changed_when: false
|
|
||||||
|
|
||||||
- name: Remove obsolete HAProxy PostgreSQL protocol check
|
|
||||||
vyos.vyos.vyos_config:
|
|
||||||
lines:
|
|
||||||
- delete load-balancing haproxy backend sandbox-postgresql-primary health-check
|
|
||||||
save: true
|
|
||||||
comment: Remove unsupported PostgreSQL protocol check
|
|
||||||
when: >-
|
|
||||||
"set load-balancing haproxy backend sandbox-postgresql-primary health-check 'pgsql'"
|
|
||||||
in _vyos_running_commands.stdout[0]
|
|
||||||
|
|
||||||
- name: Render the desired configuration
|
- name: Render the desired configuration
|
||||||
ansible.builtin.set_fact:
|
ansible.builtin.set_fact:
|
||||||
_vyos_lines: >-
|
_vyos_lines: >-
|
||||||
|
|||||||
@@ -35,23 +35,11 @@ set load-balancing haproxy service sandbox-postgresql port {{ vyos_postgresql_po
|
|||||||
set load-balancing haproxy service sandbox-postgresql mode 'tcp'
|
set load-balancing haproxy service sandbox-postgresql mode 'tcp'
|
||||||
set load-balancing haproxy service sandbox-postgresql backend 'sandbox-postgresql-primary'
|
set load-balancing haproxy service sandbox-postgresql backend 'sandbox-postgresql-primary'
|
||||||
set load-balancing haproxy backend sandbox-postgresql-primary mode 'tcp'
|
set load-balancing haproxy backend sandbox-postgresql-primary mode 'tcp'
|
||||||
|
set load-balancing haproxy backend sandbox-postgresql-primary health-check 'pgsql'
|
||||||
set load-balancing haproxy backend sandbox-postgresql-primary server primary address {{ vyos_postgresql_primary_address }}
|
set load-balancing haproxy backend sandbox-postgresql-primary server primary address {{ vyos_postgresql_primary_address }}
|
||||||
set load-balancing haproxy backend sandbox-postgresql-primary server primary port {{ vyos_postgresql_port }}
|
set load-balancing haproxy backend sandbox-postgresql-primary server primary port {{ vyos_postgresql_port }}
|
||||||
set load-balancing haproxy backend sandbox-postgresql-primary server primary check
|
set load-balancing haproxy backend sandbox-postgresql-primary server primary check
|
||||||
|
|
||||||
{# --- Sandbox K3s API: stable address with both control-plane nodes --- #}
|
|
||||||
set interfaces ethernet {{ vyos_sandbox_k3s_api_interface }} address '{{ vyos_sandbox_k3s_api_address }}/32'
|
|
||||||
set load-balancing haproxy service sandbox-k3s-api listen-address {{ vyos_sandbox_k3s_api_address }}
|
|
||||||
set load-balancing haproxy service sandbox-k3s-api port {{ vyos_sandbox_k3s_api_port }}
|
|
||||||
set load-balancing haproxy service sandbox-k3s-api mode 'tcp'
|
|
||||||
set load-balancing haproxy service sandbox-k3s-api backend 'sandbox-k3s-api'
|
|
||||||
set load-balancing haproxy backend sandbox-k3s-api mode 'tcp'
|
|
||||||
{% for server in vyos_sandbox_k3s_servers %}
|
|
||||||
set load-balancing haproxy backend sandbox-k3s-api server {{ server.name }} address {{ server.address }}
|
|
||||||
set load-balancing haproxy backend sandbox-k3s-api server {{ server.name }} port {{ vyos_sandbox_k3s_api_port }}
|
|
||||||
set load-balancing haproxy backend sandbox-k3s-api server {{ server.name }} check
|
|
||||||
{% endfor %}
|
|
||||||
|
|
||||||
{# --- management --- #}
|
{# --- management --- #}
|
||||||
set service ssh port 22
|
set service ssh port 22
|
||||||
|
|
||||||
|
|||||||
@@ -1,142 +0,0 @@
|
|||||||
# Sandbox 集群
|
|
||||||
|
|
||||||
该目录管理供 OpenSandbox、CI 和 AI Agent workload 使用的独立双节点 k3s 集群。
|
|
||||||
基础设施和节点生命周期由 Ansible 管理;Kubernetes API 可用后,集群内组件由
|
|
||||||
`clusters/sandbox/` 下的 Flux desired state 管理。
|
|
||||||
|
|
||||||
## 所有权边界
|
|
||||||
|
|
||||||
Ansible 管理以下持久状态:
|
|
||||||
|
|
||||||
- pve1/pve2 上的 privileged LXC、磁盘、固定网络与 KVM/vhost/TUN/kmsg 设备;
|
|
||||||
- LXC OS 基线、内部 CA、外部 PostgreSQL 与 K3s;
|
|
||||||
- 固定版本的 Flux controllers 和 root sync bootstrap;
|
|
||||||
- 只读状态验证和数据库切换 runbook。
|
|
||||||
|
|
||||||
Flux 管理以下 Kubernetes 资源:
|
|
||||||
|
|
||||||
- Kata Containers 和 CI 专用的 `block-plain` RuntimeClass;
|
|
||||||
- SPIRE Agent、SPIFFE CSI Driver 与 workload identity 声明;
|
|
||||||
- vmagent、kube-state-metrics、kubelet/cAdvisor scrape 配置和告警;
|
|
||||||
- OpenSandbox operator/server、`ci-pod` 与 `ci-vm` Pools。
|
|
||||||
|
|
||||||
同一个对象只能有一个 owner。Ansible 不直接部署上述集群内 workload;Flux 不管理
|
|
||||||
LXC、K3s datastore 或 K3s 本身。
|
|
||||||
|
|
||||||
监控范围仅覆盖 sandbox LXC 内的 Kubernetes 与 workload。不得在 LXC 内部署
|
|
||||||
node_exporter:LXC 的 `/proc` 是 lxcfs 虚拟视图与宿主内核视图的混合,尤其
|
|
||||||
`/proc/stat` 会重复暴露 PVE 宿主 CPU 数据。PVE 自身的 node_exporter 或其他宿主
|
|
||||||
监控不属于本目录;sandbox 使用 kubelet/cAdvisor 和 kube-state-metrics。
|
|
||||||
|
|
||||||
## 声明拓扑
|
|
||||||
|
|
||||||
| 对象 | PVE 节点 | VMID | 地址 | 资源 |
|
|
||||||
|---|---:|---:|---|---|
|
|
||||||
| `sandbox1` | pve1 | 148 | `10.60.0.11/24` | 4 vCPU / 6 GiB / 2 GiB swap |
|
|
||||||
| `sandbox2` | pve2 | 149 | `10.60.0.12/24` | 4 vCPU / 4 GiB / 2 GiB swap |
|
|
||||||
| K3s API VIP | VyOS | — | `10.60.0.13:6443` | HAProxy TCP LB |
|
|
||||||
|
|
||||||
节点与 API VIP 均位于现有 PVE `labnet`(VLAN 100,`10.60.0.0/24`),通过
|
|
||||||
VyOS `10.60.0.1` 路由;不为 sandbox 新建 VNet,也不占用 `192.168.10.0/24`
|
|
||||||
地址。需要从 LAN 访问的服务统一经 VyOS 路由或 LB 暴露。两个 LXC 均使用
|
|
||||||
`pve-rg-hdd` 上的 32 GiB rootfs;不为 `/var/lib/kubelet` 单独创建 volume。Kata
|
|
||||||
`block-plain` 产生的数据随 Pod 生命周期清理,当前规模没有额外磁盘故障域的需求。
|
|
||||||
K3s API 的 `10.60.0.13/32` 由 VyOS 现有 labnet interface 持有,HAProxy 以 TCP
|
|
||||||
健康检查把 `6443` 分发到两个 control-plane 节点;它与 PostgreSQL 主从切换逻辑无关。
|
|
||||||
LXC 内通过 `/etc/tmpfiles.d/kmsg.conf` 持久维护 `/dev/kmsg -> /dev/console`;否则
|
|
||||||
kubelet 会因 LXC 不提供真实 host `/dev/kmsg` 而反复退出。
|
|
||||||
每个 LXC 还以只读 bind mount 使用宿主的 `/lib/modules`。LXC 与 PVE 宿主共享内核,
|
|
||||||
guest 若看不到对应版本的模块目录,K3s 无法加载 `br_netfilter` 和 `overlay`,Flannel
|
|
||||||
也不会生成节点的 subnet 配置。
|
|
||||||
|
|
||||||
## PostgreSQL 写入口与切换
|
|
||||||
|
|
||||||
K3s 使用外部 PostgreSQL,首期采用 primary + synchronous standby。VyOS 在 labnet
|
|
||||||
gateway `10.60.0.1:5432` 提供固定 TCP 入口,backend 只包含当前声明的 primary;
|
|
||||||
不把普通 TCP 或 PostgreSQL 存活检查等同于“节点可写”,也不自动把流量切到 standby。
|
|
||||||
VyOS 2025.11 的 PostgreSQL protocol check 会生成缺少必需 `user` 参数的 HAProxy
|
|
||||||
配置,因此这里只使用基础 TCP check;真正的可写性由 `verify.yml` 通过 SQL 验证。
|
|
||||||
|
|
||||||
数据库切换必须由 Ansible runbook 受控完成:先隔离旧 primary,再提升 standby,最后
|
|
||||||
更新 VyOS backend。首期不部署 PgBouncer、Patroni、repmgr 或额外 DCS,也不宣称两节点
|
|
||||||
PostgreSQL 能够自动 HA。以后具备第三个仲裁节点时再重新评估自动 failover。
|
|
||||||
|
|
||||||
数据库凭据位于 Bao `kv/infra/sandbox-postgresql`,包含 K3s 登录密码和 physical
|
|
||||||
replication 密码。运行 Ansible 前由本机 SPIFFE identity 获取短期 Bao token,再把
|
|
||||||
两个值注入 `SANDBOX_K3S_DB_PASSWORD` 与 `SANDBOX_REPLICATION_PASSWORD`;凭据不写入
|
|
||||||
inventory、Git 或 Ansible fact cache。首次创建 standby 只允许覆盖不含任何业务库的
|
|
||||||
Ubuntu 默认空集群,之后重复执行不会 reseed。
|
|
||||||
|
|
||||||
## Ansible
|
|
||||||
|
|
||||||
安装固定依赖:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
cd infrastructure/sandbox-cluster/ansible
|
|
||||||
uv venv .venv
|
|
||||||
uv pip install --python .venv/bin/python -r requirements.txt
|
|
||||||
source .venv/bin/activate
|
|
||||||
ansible-galaxy collection install -r requirements.yml
|
|
||||||
export PVE_API_PYTHON="$PWD/.venv/bin/python"
|
|
||||||
```
|
|
||||||
|
|
||||||
部署入口最终为:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
ansible-playbook site.yml
|
|
||||||
```
|
|
||||||
|
|
||||||
只安装或 reconcile K3s(LXC、OS baseline 和 PostgreSQL 已就绪时):
|
|
||||||
|
|
||||||
```bash
|
|
||||||
ansible-playbook k3s.yml
|
|
||||||
```
|
|
||||||
|
|
||||||
K3s 外部 datastore URI 由运行时 `SANDBOX_K3S_DB_PASSWORD` 生成,密码在 URI 中
|
|
||||||
进行 URL 编码,最终仅持久化于节点 root 可读的 `/etc/rancher/k3s/config.yaml`
|
|
||||||
(mode `0600`)。首节点生成的 K3s join token 仅在同一次 Ansible run 内传给第二节点;
|
|
||||||
不在 inventory 或 Git 中维护副本。首期关闭内建 Traefik 和 ServiceLB。
|
|
||||||
|
|
||||||
部署后使用同一组运行时凭据执行只读验收:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
ansible-playbook verify.yml
|
|
||||||
```
|
|
||||||
|
|
||||||
当前已经声明 LXC 生命周期、最小 OS baseline、PostgreSQL 和 K3s,包括系统级
|
|
||||||
homelab CA trust。Flux `v2.9.5` controllers 与 root sync 也由 Ansible 通过 K3s
|
|
||||||
server manifests 管理;root 使用 homelab CA 访问公开 Gitea 仓库,不保存 Git token。
|
|
||||||
集群内 workload 由 `clusters/sandbox/` 分阶段纳入 Flux。
|
|
||||||
|
|
||||||
## SPIRE 跨集群 bootstrap
|
|
||||||
|
|
||||||
Sandbox 复用 homelab 的 SPIRE Server 与 `ddupan.top` trust domain。Flux 首先安装
|
|
||||||
SPIRE CRD,并创建供 k8s_psat 使用的 reviewer。它按上游 Server chart 的权限模型调用
|
|
||||||
TokenReview,并以 `get/list` 读取用于证明的 Pod 与 Node;不具有修改 workload 的权限。
|
|
||||||
在该 Kustomization Ready 后运行:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
cd infrastructure/sandbox-cluster/ansible
|
|
||||||
ansible-playbook spire-bootstrap.yml
|
|
||||||
```
|
|
||||||
|
|
||||||
Playbook 不把 reviewer token 或生成的 kubeconfig 落盘,而是将目标 Secret manifest
|
|
||||||
通过 stdin 交给本机 homelab `k3s kubectl`。目标 Secret
|
|
||||||
`spire-server/spire-external-kubeconfigs` 由 Ansible 单独拥有;Flux 和人工操作不得写入。
|
|
||||||
第二次运行必须为零变更。
|
|
||||||
|
|
||||||
Secret 的 `sandbox` key 仅供 SPIRE Server 的 external PSAT plugin 验证 token 与对应的
|
|
||||||
Pod/Node;
|
|
||||||
`sandbox-controller` key 供 external controller-manager 读取 Pod/Node、reconcile SPIFFE
|
|
||||||
CR 及执行 leader election。两者使用不同 ServiceAccount,不得合并权限或互换。
|
|
||||||
|
|
||||||
## 已验证的 Kata CI 前置条件
|
|
||||||
|
|
||||||
- Cloud Hypervisor 必须报告 `vm.info.config.memory.shared=true`;
|
|
||||||
- CI RuntimeClass 必须使用 `[runtime] emptydir_mode = "block-plain"`;
|
|
||||||
- dockerd bootstrap 在 guest 内创建 `/dev/kmsg`:`mknod /dev/kmsg c 1 11`;
|
|
||||||
- `/var/lib/docker` 必须是 guest block device 上的 ext4,Docker driver 必须为
|
|
||||||
`overlay2`,不能静默退化到 `vfs` 或 `fuse-overlayfs`;
|
|
||||||
- Pod 删除后必须不存在遗留 `disk.img`、VMM 或临时 credential。
|
|
||||||
|
|
||||||
PoC 的完整数据和陷阱见 `../kata-lxc-lab/README.md`。
|
|
||||||
@@ -1,11 +0,0 @@
|
|||||||
[defaults]
|
|
||||||
inventory = inventory/hosts.yml
|
|
||||||
host_key_checking = True
|
|
||||||
interpreter_python = auto_silent
|
|
||||||
retry_files_enabled = False
|
|
||||||
local_tmp = /tmp/ansible-sandbox-cluster-local
|
|
||||||
remote_tmp = /tmp
|
|
||||||
roles_path = roles
|
|
||||||
|
|
||||||
[ssh_connection]
|
|
||||||
pipelining = True
|
|
||||||
@@ -1,61 +0,0 @@
|
|||||||
---
|
|
||||||
sandbox_pve_api_host: "{{ lookup('env', 'PROXMOX_HOST') }}"
|
|
||||||
sandbox_pve_api_user: "{{ lookup('env', 'PROXMOX_USER') }}"
|
|
||||||
sandbox_pve_api_token_id: "{{ lookup('env', 'PROXMOX_TOKEN_ID') }}"
|
|
||||||
sandbox_pve_api_token_secret: "{{ lookup('env', 'PROXMOX_TOKEN_SECRET') }}"
|
|
||||||
sandbox_pve_validate_certs: true
|
|
||||||
sandbox_pve_api_python_interpreter: "{{ lookup('env', 'PVE_API_PYTHON') | default(ansible_playbook_python, true) }}"
|
|
||||||
|
|
||||||
sandbox_bridge: labnet
|
|
||||||
sandbox_gateway: 10.60.0.1
|
|
||||||
sandbox_nameserver: 192.168.10.5
|
|
||||||
sandbox_search_domain: ad.ddupan.top
|
|
||||||
sandbox_lxc_template: laptop:vztmpl/ubuntu-24.04-standard_24.04-2_amd64.tar.zst
|
|
||||||
sandbox_rootfs_storage: pve-rg-hdd
|
|
||||||
sandbox_rootfs_gb: 32
|
|
||||||
sandbox_lxc_cores: 4
|
|
||||||
sandbox_lxc_swap_mb: 2048
|
|
||||||
sandbox_lxc_pubkey: "{{ lookup('file', '~/.ssh/id_ed25519.pub') }}"
|
|
||||||
sandbox_internal_ca_url: https://bao.ad.ddupan.top:8200/v1/pki/ca/pem
|
|
||||||
sandbox_base_packages:
|
|
||||||
- ca-certificates
|
|
||||||
- curl
|
|
||||||
- jq
|
|
||||||
- openssh-server
|
|
||||||
|
|
||||||
sandbox_postgresql_version: 16
|
|
||||||
sandbox_postgresql_primary_address: 10.60.0.11
|
|
||||||
sandbox_postgresql_standby_address: 10.60.0.12
|
|
||||||
sandbox_postgresql_lb_address: 10.60.0.1
|
|
||||||
sandbox_postgresql_database: k3s
|
|
||||||
sandbox_postgresql_user: k3s
|
|
||||||
sandbox_postgresql_replication_user: k3s_replication
|
|
||||||
sandbox_postgresql_replication_slot: sandbox2
|
|
||||||
sandbox_postgresql_rotate_passwords: false
|
|
||||||
sandbox_postgresql_k3s_password: "{{ lookup('env', 'SANDBOX_K3S_DB_PASSWORD') }}"
|
|
||||||
sandbox_postgresql_replication_password: "{{ lookup('env', 'SANDBOX_REPLICATION_PASSWORD') }}"
|
|
||||||
|
|
||||||
# This is rendered only at playbook runtime from the Bao-provided password. K3s
|
|
||||||
# necessarily persists its datastore credential, but config.yaml is root-only.
|
|
||||||
sandbox_k3s_datastore_password_encoded: >-
|
|
||||||
{{ sandbox_postgresql_k3s_password | urlencode | replace('/', '%2F') }}
|
|
||||||
sandbox_k3s_datastore_endpoint: >-
|
|
||||||
{{
|
|
||||||
[
|
|
||||||
'postgres://', sandbox_postgresql_user, ':', sandbox_k3s_datastore_password_encoded,
|
|
||||||
'@', sandbox_postgresql_lb_address, ':5432/', sandbox_postgresql_database, '?sslmode=disable'
|
|
||||||
] | join
|
|
||||||
}}
|
|
||||||
server_config_yaml: |
|
|
||||||
datastore-endpoint: {{ sandbox_k3s_datastore_endpoint }}
|
|
||||||
disable:
|
|
||||||
- servicelb
|
|
||||||
- traefik
|
|
||||||
user_kubectl: false
|
|
||||||
|
|
||||||
sandbox_flux_version: v2.9.5
|
|
||||||
sandbox_flux_install_url: >-
|
|
||||||
https://github.com/fluxcd/flux2/releases/download/{{ sandbox_flux_version }}/install.yaml
|
|
||||||
sandbox_flux_repository_url: https://git.ddupan.top/panxiao81/homelab-infra.git
|
|
||||||
sandbox_flux_repository_branch: main
|
|
||||||
sandbox_flux_path: ./clusters/sandbox
|
|
||||||
@@ -1,48 +0,0 @@
|
|||||||
---
|
|
||||||
all:
|
|
||||||
children:
|
|
||||||
sandbox_pve:
|
|
||||||
hosts:
|
|
||||||
pve1:
|
|
||||||
ansible_host: 192.168.10.4
|
|
||||||
sandbox_lxc:
|
|
||||||
vmid: 148
|
|
||||||
hostname: sandbox1
|
|
||||||
address: 10.60.0.11/24
|
|
||||||
memory_mb: 6144
|
|
||||||
pve2:
|
|
||||||
ansible_host: 192.168.10.7
|
|
||||||
sandbox_lxc:
|
|
||||||
vmid: 149
|
|
||||||
hostname: sandbox2
|
|
||||||
address: 10.60.0.12/24
|
|
||||||
memory_mb: 4096
|
|
||||||
|
|
||||||
k3s_cluster:
|
|
||||||
children:
|
|
||||||
server:
|
|
||||||
hosts:
|
|
||||||
sandbox1:
|
|
||||||
ansible_host: 10.60.0.11
|
|
||||||
sandbox2:
|
|
||||||
ansible_host: 10.60.0.12
|
|
||||||
agent:
|
|
||||||
hosts: {}
|
|
||||||
vars:
|
|
||||||
ansible_user: root
|
|
||||||
k3s_version: v1.36.4+k3s1
|
|
||||||
api_endpoint: 10.60.0.13
|
|
||||||
k3s_server_config_mode: "0600"
|
|
||||||
use_external_database: true
|
|
||||||
|
|
||||||
postgres_primary:
|
|
||||||
hosts:
|
|
||||||
sandbox1:
|
|
||||||
|
|
||||||
postgres_standby:
|
|
||||||
hosts:
|
|
||||||
sandbox2:
|
|
||||||
|
|
||||||
vars:
|
|
||||||
ansible_user: root
|
|
||||||
ansible_python_interpreter: /usr/bin/python3
|
|
||||||
@@ -1,17 +0,0 @@
|
|||||||
---
|
|
||||||
- name: Validate runtime K3s datastore credential
|
|
||||||
hosts: k3s_cluster
|
|
||||||
gather_facts: false
|
|
||||||
tasks:
|
|
||||||
- name: Require the Bao-provided PostgreSQL password
|
|
||||||
ansible.builtin.assert:
|
|
||||||
that:
|
|
||||||
- sandbox_postgresql_k3s_password | length > 0
|
|
||||||
- server_config_yaml is defined
|
|
||||||
- (server_config_yaml | from_yaml)['datastore-endpoint'] == sandbox_k3s_datastore_endpoint
|
|
||||||
fail_msg: SANDBOX_K3S_DB_PASSWORD must be populated from Bao
|
|
||||||
quiet: true
|
|
||||||
no_log: true
|
|
||||||
|
|
||||||
- name: Install the sandbox K3s control plane
|
|
||||||
ansible.builtin.import_playbook: k3s.orchestration.site
|
|
||||||
@@ -1,4 +0,0 @@
|
|||||||
ansible-core==2.21.2
|
|
||||||
proxmoxer==2.3.0
|
|
||||||
requests==2.34.2
|
|
||||||
netaddr==1.3.0
|
|
||||||
@@ -1,15 +0,0 @@
|
|||||||
---
|
|
||||||
collections:
|
|
||||||
- name: community.proxmox
|
|
||||||
version: 2.0.0
|
|
||||||
- name: community.postgresql
|
|
||||||
version: 4.2.0
|
|
||||||
- name: ansible.posix
|
|
||||||
version: 2.2.2
|
|
||||||
- name: community.general
|
|
||||||
version: 13.4.0
|
|
||||||
- name: community.library_inventory_filtering_v1
|
|
||||||
version: 1.1.5
|
|
||||||
- name: https://github.com/k3s-io/k3s-ansible.git
|
|
||||||
type: git
|
|
||||||
version: 1a600b60d37e0f8a6e2e79b0e474147b5b108ae5
|
|
||||||
@@ -1,5 +0,0 @@
|
|||||||
---
|
|
||||||
- name: Update sandbox CA certificates
|
|
||||||
ansible.builtin.command:
|
|
||||||
cmd: update-ca-certificates
|
|
||||||
changed_when: true
|
|
||||||
@@ -1,58 +0,0 @@
|
|||||||
---
|
|
||||||
- name: Install sandbox base packages
|
|
||||||
ansible.builtin.apt:
|
|
||||||
name: "{{ sandbox_base_packages }}"
|
|
||||||
state: present
|
|
||||||
update_cache: true
|
|
||||||
cache_valid_time: 3600
|
|
||||||
|
|
||||||
- name: Fetch the homelab internal CA
|
|
||||||
ansible.builtin.uri:
|
|
||||||
url: "{{ sandbox_internal_ca_url }}"
|
|
||||||
return_content: true
|
|
||||||
validate_certs: true
|
|
||||||
register: sandbox_internal_ca
|
|
||||||
changed_when: false
|
|
||||||
retries: 3
|
|
||||||
delay: 5
|
|
||||||
until: sandbox_internal_ca is succeeded
|
|
||||||
|
|
||||||
- name: Validate the homelab internal CA response
|
|
||||||
ansible.builtin.assert:
|
|
||||||
that:
|
|
||||||
- "'BEGIN CERTIFICATE' in sandbox_internal_ca.content"
|
|
||||||
fail_msg: OpenBao did not return a PEM certificate
|
|
||||||
quiet: true
|
|
||||||
|
|
||||||
- name: Install the homelab internal CA
|
|
||||||
ansible.builtin.copy:
|
|
||||||
dest: /usr/local/share/ca-certificates/ddupan-internal-ca.crt
|
|
||||||
content: "{{ sandbox_internal_ca.content }}"
|
|
||||||
owner: root
|
|
||||||
group: root
|
|
||||||
mode: "0644"
|
|
||||||
notify: Update sandbox CA certificates
|
|
||||||
|
|
||||||
- name: Enable the SSH service
|
|
||||||
ansible.builtin.service:
|
|
||||||
name: ssh
|
|
||||||
enabled: true
|
|
||||||
state: started
|
|
||||||
|
|
||||||
# LXC does not expose the host kernel log device. Kubelet only needs a writable
|
|
||||||
# kmsg-compatible character device, and the container console is the established
|
|
||||||
# LXC mapping for this purpose. tmpfiles recreates the link after every boot.
|
|
||||||
- name: Persist the LXC kubelet kmsg mapping
|
|
||||||
ansible.builtin.copy:
|
|
||||||
dest: /etc/tmpfiles.d/kmsg.conf
|
|
||||||
content: "L+ /dev/kmsg - - - - /dev/console\n"
|
|
||||||
owner: root
|
|
||||||
group: root
|
|
||||||
mode: "0644"
|
|
||||||
|
|
||||||
- name: Expose the LXC console as kmsg
|
|
||||||
ansible.builtin.file:
|
|
||||||
src: /dev/console
|
|
||||||
dest: /dev/kmsg
|
|
||||||
state: link
|
|
||||||
force: true
|
|
||||||
@@ -1,30 +0,0 @@
|
|||||||
---
|
|
||||||
- name: Wait for the K3s server manifest directory
|
|
||||||
ansible.builtin.wait_for:
|
|
||||||
path: /var/lib/rancher/k3s/server/manifests
|
|
||||||
state: present
|
|
||||||
timeout: 120
|
|
||||||
|
|
||||||
- name: Download the pinned Flux controller manifest
|
|
||||||
ansible.builtin.get_url:
|
|
||||||
url: "{{ sandbox_flux_install_url }}"
|
|
||||||
dest: /var/lib/rancher/k3s/server/manifests/flux-install.yaml
|
|
||||||
owner: root
|
|
||||||
group: root
|
|
||||||
mode: "0644"
|
|
||||||
timeout: 120
|
|
||||||
|
|
||||||
- name: Read the installed homelab CA
|
|
||||||
ansible.builtin.slurp:
|
|
||||||
src: /usr/local/share/ca-certificates/ddupan-internal-ca.crt
|
|
||||||
register: sandbox_flux_ca
|
|
||||||
no_log: true
|
|
||||||
|
|
||||||
- name: Install the Flux source and root sync manifest
|
|
||||||
ansible.builtin.template:
|
|
||||||
src: flux-sync.yaml.j2
|
|
||||||
dest: /var/lib/rancher/k3s/server/manifests/flux-sync.yaml
|
|
||||||
owner: root
|
|
||||||
group: root
|
|
||||||
mode: "0600"
|
|
||||||
no_log: true
|
|
||||||
@@ -1,39 +0,0 @@
|
|||||||
---
|
|
||||||
apiVersion: v1
|
|
||||||
kind: Secret
|
|
||||||
metadata:
|
|
||||||
name: flux-system
|
|
||||||
namespace: flux-system
|
|
||||||
type: Opaque
|
|
||||||
stringData:
|
|
||||||
caFile: |
|
|
||||||
{{ sandbox_flux_ca.content | b64decode | indent(4, true) }}
|
|
||||||
---
|
|
||||||
apiVersion: source.toolkit.fluxcd.io/v1
|
|
||||||
kind: GitRepository
|
|
||||||
metadata:
|
|
||||||
name: flux-system
|
|
||||||
namespace: flux-system
|
|
||||||
spec:
|
|
||||||
interval: 1m
|
|
||||||
ref:
|
|
||||||
branch: {{ sandbox_flux_repository_branch }}
|
|
||||||
secretRef:
|
|
||||||
name: flux-system
|
|
||||||
timeout: 60s
|
|
||||||
url: {{ sandbox_flux_repository_url }}
|
|
||||||
---
|
|
||||||
apiVersion: kustomize.toolkit.fluxcd.io/v1
|
|
||||||
kind: Kustomization
|
|
||||||
metadata:
|
|
||||||
name: flux-system
|
|
||||||
namespace: flux-system
|
|
||||||
spec:
|
|
||||||
interval: 10m
|
|
||||||
path: {{ sandbox_flux_path }}
|
|
||||||
prune: false
|
|
||||||
sourceRef:
|
|
||||||
kind: GitRepository
|
|
||||||
name: flux-system
|
|
||||||
timeout: 3m
|
|
||||||
wait: true
|
|
||||||
@@ -1,214 +0,0 @@
|
|||||||
---
|
|
||||||
- name: Validate sandbox declaration and API inputs
|
|
||||||
ansible.builtin.assert:
|
|
||||||
that:
|
|
||||||
- sandbox_lxc.vmid | int > 100
|
|
||||||
- sandbox_lxc.hostname | length > 0
|
|
||||||
- sandbox_lxc.address is match('^10\.60\.0\.[0-9]{1,3}/24$')
|
|
||||||
- sandbox_lxc.memory_mb | int >= 4096
|
|
||||||
- sandbox_pve_api_host | length > 0
|
|
||||||
- sandbox_pve_api_user | length > 0
|
|
||||||
- sandbox_pve_api_token_id | length > 0
|
|
||||||
- sandbox_pve_api_token_secret | length > 0
|
|
||||||
fail_msg: Invalid sandbox declaration or missing Proxmox API environment
|
|
||||||
no_log: true
|
|
||||||
|
|
||||||
- name: Check whether the LXC already exists
|
|
||||||
ansible.builtin.stat:
|
|
||||||
path: /etc/pve/lxc/{{ sandbox_lxc.vmid }}.conf
|
|
||||||
register: sandbox_lxc_config
|
|
||||||
|
|
||||||
- name: Read current sandbox LXC runtime state
|
|
||||||
community.proxmox.proxmox_vm_info:
|
|
||||||
api_host: "{{ sandbox_pve_api_host }}"
|
|
||||||
api_user: "{{ sandbox_pve_api_user }}"
|
|
||||||
api_token_id: "{{ sandbox_pve_api_token_id }}"
|
|
||||||
api_token_secret: "{{ sandbox_pve_api_token_secret }}"
|
|
||||||
validate_certs: "{{ sandbox_pve_validate_certs }}"
|
|
||||||
ca_path: /etc/ssl/certs/ca-certificates.crt
|
|
||||||
api_timeout: 120
|
|
||||||
vmid: "{{ sandbox_lxc.vmid }}"
|
|
||||||
register: sandbox_lxc_info
|
|
||||||
delegate_to: localhost
|
|
||||||
no_log: true
|
|
||||||
vars:
|
|
||||||
ansible_python_interpreter: "{{ sandbox_pve_api_python_interpreter }}"
|
|
||||||
|
|
||||||
- name: Read current sandbox LXC configuration
|
|
||||||
ansible.builtin.uri:
|
|
||||||
url: >-
|
|
||||||
https://{{ sandbox_pve_api_host }}:8006/api2/json/nodes/{{ inventory_hostname }}/lxc/{{ sandbox_lxc.vmid }}/config
|
|
||||||
headers:
|
|
||||||
Authorization: >-
|
|
||||||
PVEAPIToken={{ sandbox_pve_api_user }}!{{ sandbox_pve_api_token_id }}={{ sandbox_pve_api_token_secret }}
|
|
||||||
ca_path: /etc/ssl/certs/ca-certificates.crt
|
|
||||||
return_content: true
|
|
||||||
status_code: [200, 404, 500]
|
|
||||||
register: sandbox_lxc_current_config
|
|
||||||
delegate_to: localhost
|
|
||||||
no_log: true
|
|
||||||
|
|
||||||
- name: Reconcile sandbox LXC through the PVE API
|
|
||||||
community.proxmox.proxmox:
|
|
||||||
api_host: "{{ sandbox_pve_api_host }}"
|
|
||||||
api_user: "{{ sandbox_pve_api_user }}"
|
|
||||||
api_token_id: "{{ sandbox_pve_api_token_id }}"
|
|
||||||
api_token_secret: "{{ sandbox_pve_api_token_secret }}"
|
|
||||||
validate_certs: "{{ sandbox_pve_validate_certs }}"
|
|
||||||
ca_path: /etc/ssl/certs/ca-certificates.crt
|
|
||||||
api_timeout: 120
|
|
||||||
node: "{{ inventory_hostname }}"
|
|
||||||
vmid: "{{ sandbox_lxc.vmid }}"
|
|
||||||
hostname: "{{ sandbox_lxc.hostname }}"
|
|
||||||
ostemplate: "{{ sandbox_lxc_template }}"
|
|
||||||
ostype: ubuntu
|
|
||||||
cmode: shell
|
|
||||||
disk_volume:
|
|
||||||
storage: "{{ sandbox_rootfs_storage }}"
|
|
||||||
size: "{{ sandbox_rootfs_gb if not sandbox_lxc_config.stat.exists else omit }}"
|
|
||||||
cores: "{{ sandbox_lxc_cores }}"
|
|
||||||
memory: "{{ sandbox_lxc.memory_mb }}"
|
|
||||||
swap: "{{ sandbox_lxc_swap_mb }}"
|
|
||||||
netif:
|
|
||||||
net0: >-
|
|
||||||
{{
|
|
||||||
'name=eth0,bridge=' ~ sandbox_bridge ~
|
|
||||||
',ip=' ~ sandbox_lxc.address ~
|
|
||||||
',gw=' ~ sandbox_gateway ~
|
|
||||||
',type=veth' ~
|
|
||||||
(',hwaddr=' ~ _sandbox_existing_mac if _sandbox_existing_mac | length > 0 else '')
|
|
||||||
}}
|
|
||||||
nameserver: "{{ sandbox_nameserver }}"
|
|
||||||
searchdomain: "{{ sandbox_search_domain }}"
|
|
||||||
unprivileged: false
|
|
||||||
onboot: true
|
|
||||||
startup: ["order=30", "up=30", "down=60"]
|
|
||||||
tags: [ansible, sandbox, k3s, kata]
|
|
||||||
pubkey: "{{ sandbox_lxc_pubkey }}"
|
|
||||||
update: true
|
|
||||||
state: present
|
|
||||||
timeout: 120
|
|
||||||
register: sandbox_lxc_api
|
|
||||||
when: not sandbox_lxc_config.stat.exists or _sandbox_api_drift
|
|
||||||
delegate_to: localhost
|
|
||||||
no_log: true
|
|
||||||
vars:
|
|
||||||
ansible_python_interpreter: "{{ sandbox_pve_api_python_interpreter }}"
|
|
||||||
_sandbox_existing_mac: >-
|
|
||||||
{{ sandbox_lxc_current_config.json.data.net0.split('hwaddr=')[1].split(',')[0]
|
|
||||||
if 'hwaddr=' in (sandbox_lxc_current_config.json.data.net0 | default(''))
|
|
||||||
else '' }}
|
|
||||||
_sandbox_api_drift: >-
|
|
||||||
{{
|
|
||||||
sandbox_lxc_current_config.status != 200 or
|
|
||||||
sandbox_lxc_current_config.json.data.hostname | default('') != sandbox_lxc.hostname or
|
|
||||||
sandbox_lxc_current_config.json.data.cores | default(0) | int != sandbox_lxc_cores | int or
|
|
||||||
sandbox_lxc_current_config.json.data.memory | default(0) | int != sandbox_lxc.memory_mb | int or
|
|
||||||
sandbox_lxc_current_config.json.data.swap | default(0) | int != sandbox_lxc_swap_mb | int or
|
|
||||||
sandbox_lxc_current_config.json.data.nameserver | default('') != sandbox_nameserver or
|
|
||||||
sandbox_lxc_current_config.json.data.searchdomain | default('') != sandbox_search_domain or
|
|
||||||
sandbox_lxc_current_config.json.data.cmode | default('') != 'shell' or
|
|
||||||
sandbox_lxc_current_config.json.data.onboot | default(0) | int != 1 or
|
|
||||||
sandbox_lxc_current_config.json.data.tags | default('') != 'ansible;k3s;kata;sandbox' or
|
|
||||||
('bridge=' ~ sandbox_bridge) not in (sandbox_lxc_current_config.json.data.net0 | default('')) or
|
|
||||||
('ip=' ~ sandbox_lxc.address) not in (sandbox_lxc_current_config.json.data.net0 | default('')) or
|
|
||||||
('gw=' ~ sandbox_gateway) not in (sandbox_lxc_current_config.json.data.net0 | default('')) or
|
|
||||||
(sandbox_rootfs_storage ~ ':') not in (sandbox_lxc_current_config.json.data.rootfs | default(''))
|
|
||||||
}}
|
|
||||||
|
|
||||||
- name: Grow existing sandbox rootfs through the PVE resize API
|
|
||||||
ansible.builtin.uri:
|
|
||||||
url: >-
|
|
||||||
https://{{ sandbox_pve_api_host }}:8006/api2/json/nodes/{{ inventory_hostname }}/lxc/{{ sandbox_lxc.vmid }}/resize
|
|
||||||
method: PUT
|
|
||||||
headers:
|
|
||||||
Authorization: >-
|
|
||||||
PVEAPIToken={{ sandbox_pve_api_user }}!{{ sandbox_pve_api_token_id }}={{ sandbox_pve_api_token_secret }}
|
|
||||||
body_format: form-urlencoded
|
|
||||||
body:
|
|
||||||
disk: rootfs
|
|
||||||
size: "{{ sandbox_rootfs_gb }}G"
|
|
||||||
ca_path: /etc/ssl/certs/ca-certificates.crt
|
|
||||||
status_code: 200
|
|
||||||
when:
|
|
||||||
- sandbox_lxc_config.stat.exists
|
|
||||||
- _sandbox_current_rootfs_gb | int < sandbox_rootfs_gb | int
|
|
||||||
delegate_to: localhost
|
|
||||||
no_log: true
|
|
||||||
vars:
|
|
||||||
_sandbox_current_rootfs_gb: >-
|
|
||||||
{{
|
|
||||||
sandbox_lxc_current_config.json.data.rootfs |
|
|
||||||
default('') |
|
|
||||||
regex_findall('size=([0-9]+)G') |
|
|
||||||
first |
|
|
||||||
default('0')
|
|
||||||
}}
|
|
||||||
|
|
||||||
# These properties are not exposed by community.proxmox. Every retained line
|
|
||||||
# is required by the already validated nested Kata/kind runtime.
|
|
||||||
- name: Reconcile Kata-specific native LXC properties
|
|
||||||
ansible.builtin.lineinfile:
|
|
||||||
path: /etc/pve/lxc/{{ sandbox_lxc.vmid }}.conf
|
|
||||||
regexp: "^{{ item.key | regex_escape }}:"
|
|
||||||
line: "{{ item.key }}: {{ item.value }}"
|
|
||||||
loop:
|
|
||||||
- { key: features, value: "nesting=1,keyctl=1,fuse=1,mknod=1,force_rw_sys=1" }
|
|
||||||
- { key: dev0, value: "path=/dev/kvm,mode=0660" }
|
|
||||||
- { key: dev1, value: "path=/dev/vhost-net,mode=0660" }
|
|
||||||
- { key: dev2, value: "path=/dev/vhost-vsock,mode=0660" }
|
|
||||||
- { key: dev3, value: "path=/dev/net/tun,mode=0666" }
|
|
||||||
- { key: lxc.apparmor.profile, value: unconfined }
|
|
||||||
- { key: lxc.cgroup2.devices.allow, value: a }
|
|
||||||
- { key: lxc.cap.drop, value: "" }
|
|
||||||
- { key: lxc.mount.auto, value: "proc:rw sys:rw" }
|
|
||||||
- { key: lxc.mount.entry, value: "/lib/modules usr/lib/modules none bind,ro,create=dir 0 0" }
|
|
||||||
loop_control:
|
|
||||||
label: "{{ item.key }}"
|
|
||||||
register: sandbox_lxc_native
|
|
||||||
|
|
||||||
- name: Start sandbox LXC
|
|
||||||
community.proxmox.proxmox:
|
|
||||||
api_host: "{{ sandbox_pve_api_host }}"
|
|
||||||
api_user: "{{ sandbox_pve_api_user }}"
|
|
||||||
api_token_id: "{{ sandbox_pve_api_token_id }}"
|
|
||||||
api_token_secret: "{{ sandbox_pve_api_token_secret }}"
|
|
||||||
validate_certs: "{{ sandbox_pve_validate_certs }}"
|
|
||||||
ca_path: /etc/ssl/certs/ca-certificates.crt
|
|
||||||
api_timeout: 120
|
|
||||||
vmid: "{{ sandbox_lxc.vmid }}"
|
|
||||||
state: started
|
|
||||||
timeout: 120
|
|
||||||
when: >-
|
|
||||||
not _sandbox_was_running or
|
|
||||||
not (sandbox_lxc_api is changed or sandbox_lxc_native is changed)
|
|
||||||
delegate_to: localhost
|
|
||||||
no_log: true
|
|
||||||
vars:
|
|
||||||
ansible_python_interpreter: "{{ sandbox_pve_api_python_interpreter }}"
|
|
||||||
_sandbox_was_running: >-
|
|
||||||
{{ (sandbox_lxc_info.proxmox_vms | default([]) | first | default({})).status |
|
|
||||||
default('absent') == 'running' }}
|
|
||||||
|
|
||||||
- name: Restart existing sandbox LXC after configuration changes
|
|
||||||
community.proxmox.proxmox:
|
|
||||||
api_host: "{{ sandbox_pve_api_host }}"
|
|
||||||
api_user: "{{ sandbox_pve_api_user }}"
|
|
||||||
api_token_id: "{{ sandbox_pve_api_token_id }}"
|
|
||||||
api_token_secret: "{{ sandbox_pve_api_token_secret }}"
|
|
||||||
validate_certs: "{{ sandbox_pve_validate_certs }}"
|
|
||||||
ca_path: /etc/ssl/certs/ca-certificates.crt
|
|
||||||
api_timeout: 120
|
|
||||||
vmid: "{{ sandbox_lxc.vmid }}"
|
|
||||||
state: restarted
|
|
||||||
timeout: 120
|
|
||||||
when:
|
|
||||||
- _sandbox_was_running
|
|
||||||
- sandbox_lxc_api is changed or sandbox_lxc_native is changed
|
|
||||||
delegate_to: localhost
|
|
||||||
no_log: true
|
|
||||||
vars:
|
|
||||||
ansible_python_interpreter: "{{ sandbox_pve_api_python_interpreter }}"
|
|
||||||
_sandbox_was_running: >-
|
|
||||||
{{ (sandbox_lxc_info.proxmox_vms | default([]) | first | default({})).status |
|
|
||||||
default('absent') == 'running' }}
|
|
||||||
@@ -1,6 +0,0 @@
|
|||||||
---
|
|
||||||
- name: Restart sandbox PostgreSQL
|
|
||||||
ansible.builtin.service:
|
|
||||||
name: postgresql
|
|
||||||
enabled: true
|
|
||||||
state: restarted
|
|
||||||
@@ -1,50 +0,0 @@
|
|||||||
---
|
|
||||||
- name: Validate sandbox PostgreSQL inputs
|
|
||||||
ansible.builtin.assert:
|
|
||||||
that:
|
|
||||||
- sandbox_postgresql_k3s_password | length >= 32
|
|
||||||
- sandbox_postgresql_replication_password | length >= 32
|
|
||||||
- inventory_hostname in groups['postgres_primary'] or inventory_hostname in groups['postgres_standby']
|
|
||||||
fail_msg: Missing Bao-provided PostgreSQL credentials or invalid inventory role
|
|
||||||
no_log: true
|
|
||||||
|
|
||||||
- name: Install PostgreSQL packages
|
|
||||||
ansible.builtin.apt:
|
|
||||||
name:
|
|
||||||
- postgresql-{{ sandbox_postgresql_version }}
|
|
||||||
- postgresql-client-{{ sandbox_postgresql_version }}
|
|
||||||
- python3-psycopg2
|
|
||||||
state: present
|
|
||||||
update_cache: true
|
|
||||||
cache_valid_time: 3600
|
|
||||||
|
|
||||||
- name: Check whether synchronous replication has completed bootstrap
|
|
||||||
ansible.builtin.stat:
|
|
||||||
path: /etc/postgresql/{{ sandbox_postgresql_version }}/main/sandbox-synchronous-ready
|
|
||||||
register: sandbox_postgresql_synchronous_marker
|
|
||||||
|
|
||||||
- name: Install sandbox PostgreSQL access policy
|
|
||||||
ansible.builtin.template:
|
|
||||||
src: pg_hba.conf.j2
|
|
||||||
dest: /etc/postgresql/{{ sandbox_postgresql_version }}/main/pg_hba.conf
|
|
||||||
owner: postgres
|
|
||||||
group: postgres
|
|
||||||
mode: "0640"
|
|
||||||
notify: Restart sandbox PostgreSQL
|
|
||||||
|
|
||||||
- name: Install sandbox PostgreSQL server settings
|
|
||||||
ansible.builtin.template:
|
|
||||||
src: sandbox.conf.j2
|
|
||||||
dest: /etc/postgresql/{{ sandbox_postgresql_version }}/main/conf.d/99-sandbox.conf
|
|
||||||
owner: postgres
|
|
||||||
group: postgres
|
|
||||||
mode: "0644"
|
|
||||||
notify: Restart sandbox PostgreSQL
|
|
||||||
|
|
||||||
- name: Configure sandbox PostgreSQL primary
|
|
||||||
ansible.builtin.include_tasks: primary.yml
|
|
||||||
when: inventory_hostname in groups['postgres_primary']
|
|
||||||
|
|
||||||
- name: Configure sandbox PostgreSQL standby
|
|
||||||
ansible.builtin.include_tasks: standby.yml
|
|
||||||
when: inventory_hostname in groups['postgres_standby']
|
|
||||||
@@ -1,45 +0,0 @@
|
|||||||
---
|
|
||||||
- name: Apply primary PostgreSQL configuration before replication setup
|
|
||||||
ansible.builtin.meta: flush_handlers
|
|
||||||
|
|
||||||
- name: Create K3s PostgreSQL login role
|
|
||||||
community.postgresql.postgresql_user:
|
|
||||||
name: "{{ sandbox_postgresql_user }}"
|
|
||||||
password: "{{ sandbox_postgresql_k3s_password }}"
|
|
||||||
no_password_changes: "{{ not sandbox_postgresql_rotate_passwords }}"
|
|
||||||
role_attr_flags: LOGIN,NOSUPERUSER,NOCREATEDB,NOCREATEROLE,NOREPLICATION
|
|
||||||
state: present
|
|
||||||
become: true
|
|
||||||
become_user: postgres
|
|
||||||
no_log: true
|
|
||||||
environment:
|
|
||||||
PGOPTIONS: -c password_encryption=scram-sha-256
|
|
||||||
|
|
||||||
- name: Create K3s PostgreSQL database
|
|
||||||
community.postgresql.postgresql_db:
|
|
||||||
name: "{{ sandbox_postgresql_database }}"
|
|
||||||
owner: "{{ sandbox_postgresql_user }}"
|
|
||||||
state: present
|
|
||||||
become: true
|
|
||||||
become_user: postgres
|
|
||||||
|
|
||||||
- name: Create PostgreSQL replication login role
|
|
||||||
community.postgresql.postgresql_user:
|
|
||||||
name: "{{ sandbox_postgresql_replication_user }}"
|
|
||||||
password: "{{ sandbox_postgresql_replication_password }}"
|
|
||||||
no_password_changes: "{{ not sandbox_postgresql_rotate_passwords }}"
|
|
||||||
role_attr_flags: LOGIN,REPLICATION,NOSUPERUSER,NOCREATEDB,NOCREATEROLE
|
|
||||||
state: present
|
|
||||||
become: true
|
|
||||||
become_user: postgres
|
|
||||||
no_log: true
|
|
||||||
environment:
|
|
||||||
PGOPTIONS: -c password_encryption=scram-sha-256
|
|
||||||
|
|
||||||
- name: Create physical replication slot for sandbox2
|
|
||||||
community.postgresql.postgresql_slot:
|
|
||||||
name: "{{ sandbox_postgresql_replication_slot }}"
|
|
||||||
slot_type: physical
|
|
||||||
state: present
|
|
||||||
become: true
|
|
||||||
become_user: postgres
|
|
||||||
@@ -1,114 +0,0 @@
|
|||||||
---
|
|
||||||
- name: Detect whether sandbox2 is already a standby
|
|
||||||
community.postgresql.postgresql_query:
|
|
||||||
login_db: postgres
|
|
||||||
query: SELECT pg_is_in_recovery() AS in_recovery
|
|
||||||
become: true
|
|
||||||
become_user: postgres
|
|
||||||
register: sandbox_postgresql_recovery_state
|
|
||||||
|
|
||||||
- name: Inspect databases before the initial standby seed
|
|
||||||
community.postgresql.postgresql_query:
|
|
||||||
login_db: postgres
|
|
||||||
query: >-
|
|
||||||
SELECT datname FROM pg_database
|
|
||||||
WHERE NOT datistemplate AND datname <> 'postgres'
|
|
||||||
become: true
|
|
||||||
become_user: postgres
|
|
||||||
register: sandbox_postgresql_existing_databases
|
|
||||||
when: not sandbox_postgresql_recovery_state.query_result[0].in_recovery
|
|
||||||
|
|
||||||
- name: Refuse to overwrite a non-empty PostgreSQL node
|
|
||||||
ansible.builtin.assert:
|
|
||||||
that:
|
|
||||||
- sandbox_postgresql_existing_databases.query_result | length == 0
|
|
||||||
fail_msg: Refusing to reseed sandbox2 because it contains non-system databases
|
|
||||||
when: not sandbox_postgresql_recovery_state.query_result[0].in_recovery
|
|
||||||
|
|
||||||
- name: Stop PostgreSQL before the initial standby seed
|
|
||||||
ansible.builtin.service:
|
|
||||||
name: postgresql
|
|
||||||
state: stopped
|
|
||||||
when: not sandbox_postgresql_recovery_state.query_result[0].in_recovery
|
|
||||||
|
|
||||||
- name: Remove the verified-empty standby data directory
|
|
||||||
ansible.builtin.file:
|
|
||||||
path: /var/lib/postgresql/{{ sandbox_postgresql_version }}/main
|
|
||||||
state: absent
|
|
||||||
when: not sandbox_postgresql_recovery_state.query_result[0].in_recovery
|
|
||||||
|
|
||||||
- name: Recreate the standby data directory
|
|
||||||
ansible.builtin.file:
|
|
||||||
path: /var/lib/postgresql/{{ sandbox_postgresql_version }}/main
|
|
||||||
state: directory
|
|
||||||
owner: postgres
|
|
||||||
group: postgres
|
|
||||||
mode: "0700"
|
|
||||||
when: not sandbox_postgresql_recovery_state.query_result[0].in_recovery
|
|
||||||
|
|
||||||
- name: Install standby replication password file
|
|
||||||
ansible.builtin.copy:
|
|
||||||
dest: /var/lib/postgresql/.pgpass
|
|
||||||
content: >-
|
|
||||||
{{ sandbox_postgresql_primary_address }}:5432:*:{{ sandbox_postgresql_replication_user }}:{{ sandbox_postgresql_replication_password }}
|
|
||||||
owner: postgres
|
|
||||||
group: postgres
|
|
||||||
mode: "0600"
|
|
||||||
no_log: true
|
|
||||||
|
|
||||||
- name: Seed sandbox2 from the PostgreSQL primary
|
|
||||||
ansible.builtin.command:
|
|
||||||
argv:
|
|
||||||
- /usr/bin/pg_basebackup
|
|
||||||
- --host={{ sandbox_postgresql_primary_address }}
|
|
||||||
- --username={{ sandbox_postgresql_replication_user }}
|
|
||||||
- --pgdata=/var/lib/postgresql/{{ sandbox_postgresql_version }}/main
|
|
||||||
- --format=plain
|
|
||||||
- --wal-method=stream
|
|
||||||
- --write-recovery-conf
|
|
||||||
- --slot={{ sandbox_postgresql_replication_slot }}
|
|
||||||
become: true
|
|
||||||
become_user: postgres
|
|
||||||
environment:
|
|
||||||
PGPASSFILE: /var/lib/postgresql/.pgpass
|
|
||||||
when: not sandbox_postgresql_recovery_state.query_result[0].in_recovery
|
|
||||||
no_log: true
|
|
||||||
|
|
||||||
- name: Set the standby connection identity
|
|
||||||
ansible.builtin.lineinfile:
|
|
||||||
path: /var/lib/postgresql/{{ sandbox_postgresql_version }}/main/postgresql.auto.conf
|
|
||||||
regexp: ^primary_conninfo =
|
|
||||||
line: >-
|
|
||||||
primary_conninfo = 'host={{ sandbox_postgresql_primary_address }} port=5432
|
|
||||||
user={{ sandbox_postgresql_replication_user }} application_name=sandbox2
|
|
||||||
passfile=/var/lib/postgresql/.pgpass'
|
|
||||||
owner: postgres
|
|
||||||
group: postgres
|
|
||||||
mode: "0600"
|
|
||||||
no_log: true
|
|
||||||
|
|
||||||
- name: Start PostgreSQL standby
|
|
||||||
ansible.builtin.service:
|
|
||||||
name: postgresql
|
|
||||||
enabled: true
|
|
||||||
state: started
|
|
||||||
|
|
||||||
- name: Wait for sandbox2 to enter recovery
|
|
||||||
community.postgresql.postgresql_query:
|
|
||||||
login_db: postgres
|
|
||||||
query: SELECT pg_is_in_recovery() AS in_recovery
|
|
||||||
become: true
|
|
||||||
become_user: postgres
|
|
||||||
register: sandbox_postgresql_standby_ready
|
|
||||||
retries: 12
|
|
||||||
delay: 5
|
|
||||||
until: sandbox_postgresql_standby_ready.query_result[0].in_recovery
|
|
||||||
|
|
||||||
- name: Mark synchronous replication bootstrap complete on the primary
|
|
||||||
ansible.builtin.copy:
|
|
||||||
dest: /etc/postgresql/{{ sandbox_postgresql_version }}/main/sandbox-synchronous-ready
|
|
||||||
content: "sandbox2\n"
|
|
||||||
owner: postgres
|
|
||||||
group: postgres
|
|
||||||
mode: "0644"
|
|
||||||
delegate_to: "{{ groups['postgres_primary'][0] }}"
|
|
||||||
-5
@@ -1,5 +0,0 @@
|
|||||||
# Ansible managed
|
|
||||||
local all postgres peer
|
|
||||||
local all all peer
|
|
||||||
host {{ sandbox_postgresql_database }} {{ sandbox_postgresql_user }} 10.60.0.0/24 scram-sha-256
|
|
||||||
host replication {{ sandbox_postgresql_replication_user }} 10.60.0.0/24 scram-sha-256
|
|
||||||
-11
@@ -1,11 +0,0 @@
|
|||||||
# Ansible managed
|
|
||||||
listen_addresses = '{{ ansible_host }},127.0.0.1'
|
|
||||||
password_encryption = 'scram-sha-256'
|
|
||||||
wal_level = 'replica'
|
|
||||||
max_wal_senders = 10
|
|
||||||
max_replication_slots = 10
|
|
||||||
hot_standby = on
|
|
||||||
{% if inventory_hostname in groups['postgres_primary'] and sandbox_postgresql_synchronous_marker.stat.exists %}
|
|
||||||
synchronous_standby_names = 'FIRST 1 (sandbox2)'
|
|
||||||
synchronous_commit = 'remote_apply'
|
|
||||||
{% endif %}
|
|
||||||
@@ -1,7 +0,0 @@
|
|||||||
---
|
|
||||||
sandbox_spire_bootstrap_api_server: https://10.60.0.13:6443
|
|
||||||
sandbox_spire_bootstrap_source_namespace: spire-system
|
|
||||||
sandbox_spire_bootstrap_source_secret: spire-server-token-reviewer-token
|
|
||||||
sandbox_spire_bootstrap_controller_secret: spire-controller-manager-token
|
|
||||||
sandbox_spire_bootstrap_target_namespace: spire-server
|
|
||||||
sandbox_spire_bootstrap_target_secret: spire-external-kubeconfigs
|
|
||||||
@@ -1,115 +0,0 @@
|
|||||||
---
|
|
||||||
- name: Wait for the sandbox SPIRE token reviewer credential
|
|
||||||
ansible.builtin.command:
|
|
||||||
argv:
|
|
||||||
- k3s
|
|
||||||
- kubectl
|
|
||||||
- --namespace
|
|
||||||
- "{{ sandbox_spire_bootstrap_source_namespace }}"
|
|
||||||
- get
|
|
||||||
- secret
|
|
||||||
- "{{ sandbox_spire_bootstrap_source_secret }}"
|
|
||||||
- --output=json
|
|
||||||
register: sandbox_spire_bootstrap_reviewer_secret
|
|
||||||
changed_when: false
|
|
||||||
retries: 60
|
|
||||||
delay: 10
|
|
||||||
until:
|
|
||||||
- sandbox_spire_bootstrap_reviewer_secret.rc == 0
|
|
||||||
- (sandbox_spire_bootstrap_reviewer_secret.stdout | from_json).data.token is defined
|
|
||||||
- (sandbox_spire_bootstrap_reviewer_secret.stdout | from_json).data['ca.crt'] is defined
|
|
||||||
no_log: true
|
|
||||||
|
|
||||||
- name: Wait for the sandbox SPIRE controller credential
|
|
||||||
ansible.builtin.command:
|
|
||||||
argv:
|
|
||||||
- k3s
|
|
||||||
- kubectl
|
|
||||||
- --namespace
|
|
||||||
- "{{ sandbox_spire_bootstrap_source_namespace }}"
|
|
||||||
- get
|
|
||||||
- secret
|
|
||||||
- "{{ sandbox_spire_bootstrap_controller_secret }}"
|
|
||||||
- --output=json
|
|
||||||
register: sandbox_spire_bootstrap_controller_secret_result
|
|
||||||
changed_when: false
|
|
||||||
retries: 60
|
|
||||||
delay: 10
|
|
||||||
until:
|
|
||||||
- sandbox_spire_bootstrap_controller_secret_result.rc == 0
|
|
||||||
- (sandbox_spire_bootstrap_controller_secret_result.stdout | from_json).data.token is defined
|
|
||||||
- (sandbox_spire_bootstrap_controller_secret_result.stdout | from_json).data['ca.crt'] is defined
|
|
||||||
no_log: true
|
|
||||||
|
|
||||||
- name: Extract the sandbox TokenReview credential data
|
|
||||||
ansible.builtin.set_fact:
|
|
||||||
sandbox_spire_bootstrap_secret_data: >-
|
|
||||||
{{ (sandbox_spire_bootstrap_reviewer_secret.stdout | from_json).data }}
|
|
||||||
sandbox_spire_bootstrap_controller_data: >-
|
|
||||||
{{ (sandbox_spire_bootstrap_controller_secret_result.stdout | from_json).data }}
|
|
||||||
no_log: true
|
|
||||||
|
|
||||||
- name: Build the restricted sandbox TokenReview kubeconfig
|
|
||||||
ansible.builtin.set_fact:
|
|
||||||
sandbox_spire_bootstrap_kubeconfig: |
|
|
||||||
apiVersion: v1
|
|
||||||
kind: Config
|
|
||||||
clusters:
|
|
||||||
- name: sandbox
|
|
||||||
cluster:
|
|
||||||
server: {{ sandbox_spire_bootstrap_api_server }}
|
|
||||||
certificate-authority-data: {{ sandbox_spire_bootstrap_secret_data['ca.crt'] }}
|
|
||||||
users:
|
|
||||||
- name: spire-server-token-reviewer
|
|
||||||
user:
|
|
||||||
token: {{ sandbox_spire_bootstrap_secret_data.token | b64decode }}
|
|
||||||
contexts:
|
|
||||||
- name: sandbox
|
|
||||||
context:
|
|
||||||
cluster: sandbox
|
|
||||||
user: spire-server-token-reviewer
|
|
||||||
current-context: sandbox
|
|
||||||
sandbox_spire_bootstrap_controller_kubeconfig: |
|
|
||||||
apiVersion: v1
|
|
||||||
kind: Config
|
|
||||||
clusters:
|
|
||||||
- name: sandbox
|
|
||||||
cluster:
|
|
||||||
server: {{ sandbox_spire_bootstrap_api_server }}
|
|
||||||
certificate-authority-data: {{ sandbox_spire_bootstrap_controller_data['ca.crt'] }}
|
|
||||||
users:
|
|
||||||
- name: spire-controller-manager
|
|
||||||
user:
|
|
||||||
token: {{ sandbox_spire_bootstrap_controller_data.token | b64decode }}
|
|
||||||
contexts:
|
|
||||||
- name: sandbox
|
|
||||||
context:
|
|
||||||
cluster: sandbox
|
|
||||||
user: spire-controller-manager
|
|
||||||
current-context: sandbox
|
|
||||||
no_log: true
|
|
||||||
|
|
||||||
- name: Reconcile the central SPIRE external kubeconfig Secret
|
|
||||||
ansible.builtin.command:
|
|
||||||
argv:
|
|
||||||
- k3s
|
|
||||||
- kubectl
|
|
||||||
- apply
|
|
||||||
- --filename=-
|
|
||||||
stdin: |
|
|
||||||
apiVersion: v1
|
|
||||||
kind: Secret
|
|
||||||
metadata:
|
|
||||||
name: {{ sandbox_spire_bootstrap_target_secret }}
|
|
||||||
namespace: {{ sandbox_spire_bootstrap_target_namespace }}
|
|
||||||
type: Opaque
|
|
||||||
data:
|
|
||||||
sandbox: {{ sandbox_spire_bootstrap_kubeconfig | b64encode }}
|
|
||||||
sandbox-controller: {{ sandbox_spire_bootstrap_controller_kubeconfig | b64encode }}
|
|
||||||
delegate_to: localhost
|
|
||||||
become: true
|
|
||||||
register: sandbox_spire_bootstrap_target
|
|
||||||
changed_when: >-
|
|
||||||
' created' in sandbox_spire_bootstrap_target.stdout or
|
|
||||||
' configured' in sandbox_spire_bootstrap_target.stdout
|
|
||||||
no_log: true
|
|
||||||
@@ -1,60 +0,0 @@
|
|||||||
---
|
|
||||||
- name: Reconcile sandbox LXC nodes
|
|
||||||
hosts: sandbox_pve
|
|
||||||
gather_facts: false
|
|
||||||
serial: 1
|
|
||||||
roles:
|
|
||||||
- sandbox_lxc
|
|
||||||
|
|
||||||
- name: Reconcile sandbox node OS baseline
|
|
||||||
hosts: k3s_cluster
|
|
||||||
gather_facts: false
|
|
||||||
roles:
|
|
||||||
- sandbox_base
|
|
||||||
|
|
||||||
- name: Reconcile sandbox PostgreSQL primary
|
|
||||||
hosts: postgres_primary
|
|
||||||
gather_facts: false
|
|
||||||
roles:
|
|
||||||
- sandbox_postgresql
|
|
||||||
|
|
||||||
- name: Reconcile sandbox PostgreSQL synchronous standby
|
|
||||||
hosts: postgres_standby
|
|
||||||
gather_facts: false
|
|
||||||
roles:
|
|
||||||
- sandbox_postgresql
|
|
||||||
|
|
||||||
- name: Finalize synchronous PostgreSQL settings
|
|
||||||
hosts: postgres_primary
|
|
||||||
gather_facts: false
|
|
||||||
roles:
|
|
||||||
- sandbox_postgresql
|
|
||||||
|
|
||||||
- name: Validate runtime K3s datastore credential
|
|
||||||
hosts: k3s_cluster
|
|
||||||
gather_facts: false
|
|
||||||
tasks:
|
|
||||||
- name: Require the Bao-provided PostgreSQL password
|
|
||||||
ansible.builtin.assert:
|
|
||||||
that:
|
|
||||||
- sandbox_postgresql_k3s_password | length > 0
|
|
||||||
- server_config_yaml is defined
|
|
||||||
- (server_config_yaml | from_yaml)['datastore-endpoint'] == sandbox_k3s_datastore_endpoint
|
|
||||||
fail_msg: SANDBOX_K3S_DB_PASSWORD must be populated from Bao
|
|
||||||
quiet: true
|
|
||||||
no_log: true
|
|
||||||
|
|
||||||
- name: Install the sandbox K3s control plane
|
|
||||||
ansible.builtin.import_playbook: k3s.orchestration.site
|
|
||||||
|
|
||||||
- name: Reconcile Flux controllers and root sync
|
|
||||||
hosts: sandbox1
|
|
||||||
gather_facts: false
|
|
||||||
roles:
|
|
||||||
- sandbox_flux
|
|
||||||
|
|
||||||
- name: Reconcile central SPIRE access to sandbox Kubernetes
|
|
||||||
hosts: sandbox1
|
|
||||||
gather_facts: false
|
|
||||||
roles:
|
|
||||||
- sandbox_spire_bootstrap
|
|
||||||
@@ -1,6 +0,0 @@
|
|||||||
---
|
|
||||||
- name: Reconcile central SPIRE access to sandbox Kubernetes
|
|
||||||
hosts: sandbox1
|
|
||||||
gather_facts: false
|
|
||||||
roles:
|
|
||||||
- sandbox_spire_bootstrap
|
|
||||||
@@ -1,149 +0,0 @@
|
|||||||
---
|
|
||||||
- name: Verify the PostgreSQL datastore through the VyOS endpoint
|
|
||||||
hosts: k3s_cluster
|
|
||||||
gather_facts: false
|
|
||||||
tasks:
|
|
||||||
- name: Query the K3s database through the stable endpoint
|
|
||||||
community.postgresql.postgresql_query:
|
|
||||||
login_host: "{{ sandbox_postgresql_lb_address }}"
|
|
||||||
login_db: "{{ sandbox_postgresql_database }}"
|
|
||||||
login_user: "{{ sandbox_postgresql_user }}"
|
|
||||||
login_password: "{{ sandbox_postgresql_k3s_password }}"
|
|
||||||
query: SELECT NOT pg_is_in_recovery() AS writable
|
|
||||||
register: sandbox_datastore_endpoint
|
|
||||||
no_log: true
|
|
||||||
|
|
||||||
- name: Assert the stable datastore endpoint is writable
|
|
||||||
ansible.builtin.assert:
|
|
||||||
that:
|
|
||||||
- sandbox_datastore_endpoint.query_result[0].writable
|
|
||||||
fail_msg: The VyOS datastore endpoint is not connected to a writable PostgreSQL primary
|
|
||||||
quiet: true
|
|
||||||
|
|
||||||
- name: Verify synchronous PostgreSQL replication
|
|
||||||
hosts: postgres_primary
|
|
||||||
gather_facts: false
|
|
||||||
tasks:
|
|
||||||
- name: Query replication state on the primary
|
|
||||||
community.postgresql.postgresql_query:
|
|
||||||
login_db: postgres
|
|
||||||
query: >-
|
|
||||||
SELECT application_name, state, sync_state
|
|
||||||
FROM pg_stat_replication
|
|
||||||
WHERE application_name = 'sandbox2'
|
|
||||||
become: true
|
|
||||||
become_user: postgres
|
|
||||||
register: sandbox_replication_state
|
|
||||||
|
|
||||||
- name: Assert sandbox2 is a synchronous streaming standby
|
|
||||||
ansible.builtin.assert:
|
|
||||||
that:
|
|
||||||
- sandbox_replication_state.query_result | length == 1
|
|
||||||
- sandbox_replication_state.query_result[0].state == 'streaming'
|
|
||||||
- sandbox_replication_state.query_result[0].sync_state == 'sync'
|
|
||||||
fail_msg: sandbox2 is not synchronously streaming from sandbox1
|
|
||||||
quiet: true
|
|
||||||
|
|
||||||
- name: Verify PostgreSQL standby recovery state
|
|
||||||
hosts: postgres_standby
|
|
||||||
gather_facts: false
|
|
||||||
tasks:
|
|
||||||
- name: Query recovery state on the standby
|
|
||||||
community.postgresql.postgresql_query:
|
|
||||||
login_db: postgres
|
|
||||||
query: SELECT pg_is_in_recovery() AS in_recovery
|
|
||||||
become: true
|
|
||||||
become_user: postgres
|
|
||||||
register: sandbox_standby_state
|
|
||||||
|
|
||||||
- name: Assert sandbox2 remains in recovery
|
|
||||||
ansible.builtin.assert:
|
|
||||||
that:
|
|
||||||
- sandbox_standby_state.query_result[0].in_recovery
|
|
||||||
fail_msg: sandbox2 is not operating as a PostgreSQL standby
|
|
||||||
quiet: true
|
|
||||||
|
|
||||||
- name: Verify K3s node prerequisites
|
|
||||||
hosts: k3s_cluster
|
|
||||||
gather_facts: false
|
|
||||||
tasks:
|
|
||||||
- name: Read the kmsg mapping
|
|
||||||
ansible.builtin.command:
|
|
||||||
cmd: readlink /dev/kmsg
|
|
||||||
register: sandbox_kmsg_target
|
|
||||||
changed_when: false
|
|
||||||
|
|
||||||
- name: Inspect the host kernel module mount
|
|
||||||
ansible.builtin.shell:
|
|
||||||
cmd: >-
|
|
||||||
set -o pipefail &&
|
|
||||||
findmnt -rn -T "/lib/modules/$(uname -r)" -o OPTIONS |
|
|
||||||
grep -Eq '(^|,)ro(,|$)'
|
|
||||||
executable: /bin/bash
|
|
||||||
changed_when: false
|
|
||||||
|
|
||||||
- name: Assert required LXC kernel integration
|
|
||||||
ansible.builtin.assert:
|
|
||||||
that:
|
|
||||||
- sandbox_kmsg_target.stdout == '/dev/console'
|
|
||||||
fail_msg: LXC is missing its persistent kmsg or read-only host module mapping
|
|
||||||
quiet: true
|
|
||||||
|
|
||||||
- name: Read K3s service state
|
|
||||||
ansible.builtin.systemd_service:
|
|
||||||
name: k3s
|
|
||||||
register: sandbox_k3s_service
|
|
||||||
|
|
||||||
- name: Assert K3s is active
|
|
||||||
ansible.builtin.assert:
|
|
||||||
that:
|
|
||||||
- sandbox_k3s_service.status.ActiveState == 'active'
|
|
||||||
- sandbox_k3s_service.status.SubState == 'running'
|
|
||||||
fail_msg: K3s is not running
|
|
||||||
quiet: true
|
|
||||||
|
|
||||||
- name: Verify the K3s control plane and API VIP
|
|
||||||
hosts: sandbox1
|
|
||||||
gather_facts: false
|
|
||||||
tasks:
|
|
||||||
- name: Wait for all declared nodes
|
|
||||||
ansible.builtin.command:
|
|
||||||
cmd: k3s kubectl wait --for=condition=Ready nodes --all --timeout=120s
|
|
||||||
changed_when: false
|
|
||||||
|
|
||||||
- name: Read control-plane nodes
|
|
||||||
ansible.builtin.command:
|
|
||||||
cmd: >-
|
|
||||||
k3s kubectl get nodes
|
|
||||||
-l node-role.kubernetes.io/control-plane=true
|
|
||||||
-o name
|
|
||||||
register: sandbox_control_plane_nodes
|
|
||||||
changed_when: false
|
|
||||||
|
|
||||||
- name: Assert both control-plane nodes are registered
|
|
||||||
ansible.builtin.assert:
|
|
||||||
that:
|
|
||||||
- sandbox_control_plane_nodes.stdout_lines | length == 2
|
|
||||||
fail_msg: The sandbox cluster does not contain both control-plane nodes
|
|
||||||
quiet: true
|
|
||||||
|
|
||||||
- name: Wait for all kube-system Pods
|
|
||||||
ansible.builtin.command:
|
|
||||||
cmd: >-
|
|
||||||
k3s kubectl wait --namespace kube-system
|
|
||||||
--for=condition=Ready pods --all --timeout=120s
|
|
||||||
changed_when: false
|
|
||||||
|
|
||||||
- name: Verify the authenticated API path through VyOS
|
|
||||||
ansible.builtin.command:
|
|
||||||
cmd: >-
|
|
||||||
k3s kubectl --server=https://10.60.0.13:6443 get --raw=/livez
|
|
||||||
register: sandbox_api_livez
|
|
||||||
changed_when: false
|
|
||||||
|
|
||||||
- name: Assert the API VIP is live
|
|
||||||
ansible.builtin.assert:
|
|
||||||
that:
|
|
||||||
- sandbox_api_livez.stdout == 'ok'
|
|
||||||
fail_msg: The K3s API is not healthy through the VyOS VIP
|
|
||||||
quiet: true
|
|
||||||
@@ -2,7 +2,6 @@ apiVersion: kustomize.config.k8s.io/v1beta1
|
|||||||
kind: Kustomization
|
kind: Kustomization
|
||||||
resources:
|
resources:
|
||||||
- vmsingle.yaml
|
- vmsingle.yaml
|
||||||
- vmsingle-write-route.yaml
|
|
||||||
- vmagent.yaml
|
- vmagent.yaml
|
||||||
- vmalert.yaml
|
- vmalert.yaml
|
||||||
- vmalertmanager.yaml
|
- vmalertmanager.yaml
|
||||||
|
|||||||
@@ -1,24 +0,0 @@
|
|||||||
---
|
|
||||||
# Internal-only remote_write ingress for vmagent instances in other homelab
|
|
||||||
# clusters. Expose only the write endpoint, not VictoriaMetrics query/admin APIs.
|
|
||||||
apiVersion: gateway.networking.k8s.io/v1
|
|
||||||
kind: HTTPRoute
|
|
||||||
metadata:
|
|
||||||
name: vmsingle-remote-write
|
|
||||||
namespace: monitoring
|
|
||||||
spec:
|
|
||||||
parentRefs:
|
|
||||||
- name: eg
|
|
||||||
namespace: envoy-gateway-system
|
|
||||||
sectionName: https
|
|
||||||
hostnames:
|
|
||||||
- metrics-write.ad.ddupan.top
|
|
||||||
rules:
|
|
||||||
- matches:
|
|
||||||
- method: POST
|
|
||||||
path:
|
|
||||||
type: Exact
|
|
||||||
value: /api/v1/write
|
|
||||||
backendRefs:
|
|
||||||
- name: vmsingle-main
|
|
||||||
port: 8428
|
|
||||||
@@ -1,48 +0,0 @@
|
|||||||
# Sandbox Kata Containers
|
|
||||||
|
|
||||||
本目录通过 Flux 安装 Kata Containers 4.1.0,只启用 Cloud Hypervisor 的 Rust
|
|
||||||
runtime,并创建明确命名的 `kata-clh-runtime-rs` RuntimeClass。OpenSandbox 的 VM
|
|
||||||
Pool 必须显式选择该 RuntimeClass;不创建含义不明确的 `kata` 默认别名。
|
|
||||||
OCI chart 同时固定到已验证的 4.1.0 artifact digest,升级时必须重新执行本页验收。
|
|
||||||
|
|
||||||
安装使用官方 `kata-deploy` chart 的 `job` 模式。每次 install/upgrade 由 dispatcher
|
|
||||||
逐个节点创建短生命周期、可修改 host 的安装 Job,写入 Kata artifacts 和 K3s containerd
|
|
||||||
配置并重启对应节点的 K3s;节点恢复 Ready 后才继续下一节点。安装结束后不保留拥有
|
|
||||||
host 写权限的 DaemonSet。新增节点后必须触发 HelmRelease upgrade,使 dispatcher
|
|
||||||
重新枚举节点。
|
|
||||||
|
|
||||||
`clh-runtime-rs` 固定使用:
|
|
||||||
|
|
||||||
```toml
|
|
||||||
[runtime]
|
|
||||||
emptydir_mode = "block-plain"
|
|
||||||
```
|
|
||||||
|
|
||||||
因此普通 `emptyDir` 会在 kubelet volume 目录创建稀疏 backing file,并作为块设备
|
|
||||||
热插拔给 guest。Docker/BuildKit 可以在 guest ext4 上使用原生 overlay2,避免
|
|
||||||
virtio-fs 作为 overlayfs upperdir 时的限制。Kata 当前不会用
|
|
||||||
`emptyDir.sizeLimit` 决定该虚拟盘容量;实际容量取决于节点 rootfs。节点磁盘占用、
|
|
||||||
Pod 删除后的 backing-file/VMM 回收和真实构建基准必须作为上线验收项目。
|
|
||||||
|
|
||||||
`kata-monitor` 常驻每个节点,只读访问 K3s containerd socket 与 Kata sandbox 状态,
|
|
||||||
由 `VMPodScrape` 写入中央 VictoriaMetrics。它不拥有 Kubernetes API 凭据或 host 写
|
|
||||||
权限。
|
|
||||||
|
|
||||||
## 上线验收
|
|
||||||
|
|
||||||
Flux reconciliation 完成后至少确认:
|
|
||||||
|
|
||||||
1. 两个节点重新回到 Ready,`RuntimeClass/kata-clh-runtime-rs` 存在;
|
|
||||||
2. Kata Pod 内核与 LXC host 内核不同,且 `/dev/kvm` 可用;
|
|
||||||
3. Cloud Hypervisor API 的 `vm.info.config.memory.shared` 为 `true`;
|
|
||||||
4. `spire-smoke` ServiceAccount 的 Kata Pod 可获得
|
|
||||||
`spiffe://ddupan.top/sandbox/smoke`,错误 ServiceAccount 无法获得身份;
|
|
||||||
5. block-backed `emptyDir` 上 Docker 使用 `overlay2`,BuildKit 与 kind smoke test
|
|
||||||
通过;kind 的 dockerd bootstrap 需要先在 guest 内执行
|
|
||||||
`mknod /dev/kmsg c 1 11`;
|
|
||||||
6. 删除测试 Pod 后,Cloud Hypervisor 进程、backing file 和临时数据全部回收,且
|
|
||||||
LXC 没有新增 OOM 事件;
|
|
||||||
7. 中央 VictoriaMetrics 中两个 `kata-monitor` target 均为 `up=1`。
|
|
||||||
|
|
||||||
不要依赖手工修改 `/opt/kata` 或 K3s containerd 配置;任何修复都必须回写 chart
|
|
||||||
values 并由 Flux reconciliation。
|
|
||||||
@@ -1,40 +0,0 @@
|
|||||||
---
|
|
||||||
apiVersion: helm.toolkit.fluxcd.io/v2
|
|
||||||
kind: HelmRelease
|
|
||||||
metadata:
|
|
||||||
name: kata-deploy
|
|
||||||
namespace: kata-system
|
|
||||||
spec:
|
|
||||||
chartRef:
|
|
||||||
kind: OCIRepository
|
|
||||||
name: kata-deploy
|
|
||||||
driftDetection:
|
|
||||||
mode: enabled
|
|
||||||
install:
|
|
||||||
strategy:
|
|
||||||
name: RetryOnFailure
|
|
||||||
retryInterval: 5m
|
|
||||||
interval: 30m
|
|
||||||
postRenderers:
|
|
||||||
- kustomize:
|
|
||||||
patches:
|
|
||||||
- target:
|
|
||||||
kind: DaemonSet
|
|
||||||
name: kata-monitor
|
|
||||||
patch: |
|
|
||||||
- op: add
|
|
||||||
path: /spec/template/spec/automountServiceAccountToken
|
|
||||||
value: false
|
|
||||||
- op: add
|
|
||||||
path: /spec/template/spec/containers/0/ports/0/name
|
|
||||||
value: metrics
|
|
||||||
releaseName: kata-deploy
|
|
||||||
targetNamespace: kata-system
|
|
||||||
timeout: 30m
|
|
||||||
upgrade:
|
|
||||||
strategy:
|
|
||||||
name: RetryOnFailure
|
|
||||||
retryInterval: 5m
|
|
||||||
valuesFrom:
|
|
||||||
- kind: ConfigMap
|
|
||||||
name: kata-deploy-values
|
|
||||||
@@ -1,9 +0,0 @@
|
|||||||
---
|
|
||||||
apiVersion: kustomize.config.k8s.io/v1beta1
|
|
||||||
kind: Kustomization
|
|
||||||
resources:
|
|
||||||
- namespace.yaml
|
|
||||||
- repository.yaml
|
|
||||||
- values.yaml
|
|
||||||
- helmrelease.yaml
|
|
||||||
- monitor-scrape.yaml
|
|
||||||
@@ -1,17 +0,0 @@
|
|||||||
---
|
|
||||||
apiVersion: operator.victoriametrics.com/v1beta1
|
|
||||||
kind: VMPodScrape
|
|
||||||
metadata:
|
|
||||||
name: kata-monitor
|
|
||||||
namespace: monitoring
|
|
||||||
spec:
|
|
||||||
namespaceSelector:
|
|
||||||
matchNames:
|
|
||||||
- kata-system
|
|
||||||
podMetricsEndpoints:
|
|
||||||
- interval: 30s
|
|
||||||
port: metrics
|
|
||||||
selector:
|
|
||||||
matchLabels:
|
|
||||||
app.kubernetes.io/instance: kata-deploy
|
|
||||||
app.kubernetes.io/name: kata-monitor
|
|
||||||
@@ -1,5 +0,0 @@
|
|||||||
---
|
|
||||||
apiVersion: v1
|
|
||||||
kind: Namespace
|
|
||||||
metadata:
|
|
||||||
name: kata-system
|
|
||||||
@@ -1,11 +0,0 @@
|
|||||||
---
|
|
||||||
apiVersion: source.toolkit.fluxcd.io/v1
|
|
||||||
kind: OCIRepository
|
|
||||||
metadata:
|
|
||||||
name: kata-deploy
|
|
||||||
namespace: kata-system
|
|
||||||
spec:
|
|
||||||
interval: 1h
|
|
||||||
ref:
|
|
||||||
digest: sha256:33f102f6db70083de4fc8238af4439c4245a601bcb6a72e49bc80529098aefc0
|
|
||||||
url: oci://ghcr.io/kata-containers/kata-deploy-charts/kata-deploy
|
|
||||||
@@ -1,44 +0,0 @@
|
|||||||
---
|
|
||||||
apiVersion: v1
|
|
||||||
kind: ConfigMap
|
|
||||||
metadata:
|
|
||||||
name: kata-deploy-values
|
|
||||||
namespace: kata-system
|
|
||||||
data:
|
|
||||||
values.yaml: |
|
|
||||||
deploymentMode: job
|
|
||||||
k8sDistribution: k3s
|
|
||||||
|
|
||||||
job:
|
|
||||||
parallelism: 1
|
|
||||||
|
|
||||||
snapshotter:
|
|
||||||
setup: []
|
|
||||||
|
|
||||||
shims:
|
|
||||||
disableAll: true
|
|
||||||
clh-runtime-rs:
|
|
||||||
enabled: true
|
|
||||||
dropIn: |
|
|
||||||
[runtime]
|
|
||||||
emptydir_mode = "block-plain"
|
|
||||||
|
|
||||||
defaultShim:
|
|
||||||
amd64: clh-runtime-rs
|
|
||||||
|
|
||||||
runtimeClasses:
|
|
||||||
enabled: true
|
|
||||||
createDefault: false
|
|
||||||
|
|
||||||
node-feature-discovery:
|
|
||||||
enabled: false
|
|
||||||
|
|
||||||
monitor:
|
|
||||||
enabled: true
|
|
||||||
resources:
|
|
||||||
requests:
|
|
||||||
cpu: 25m
|
|
||||||
memory: 64Mi
|
|
||||||
limits:
|
|
||||||
cpu: 200m
|
|
||||||
memory: 192Mi
|
|
||||||
@@ -1,16 +0,0 @@
|
|||||||
# Sandbox 监控
|
|
||||||
|
|
||||||
该目录只采集 sandbox Kubernetes 与 workload 指标:
|
|
||||||
|
|
||||||
- kubelet 与 cAdvisor;
|
|
||||||
- kube-state-metrics;
|
|
||||||
- vmagent 自身运行状态;
|
|
||||||
- SPIRE Agent attestation、SVID 与连接状态。
|
|
||||||
|
|
||||||
不得在 sandbox LXC 内部署 node_exporter。LXC 的 `/proc/stat` 暴露 PVE 宿主 CPU
|
|
||||||
视图,会产生重复且语义混合的指标。PVE 宿主监控不属于本目录。
|
|
||||||
|
|
||||||
vmagent 为所有远端样本增加 `cluster=sandbox`,并通过仅允许
|
|
||||||
`POST /api/v1/write` 的 `metrics-write.ad.ddupan.top` 路由写入 homelab VMSingle。
|
|
||||||
Operator 与 workload 拆成两个 Flux Kustomization,确保 VM CRD Ready 后再创建
|
|
||||||
VMAgent、VMNodeScrape 和 VMServiceScrape。
|
|
||||||
@@ -1,32 +0,0 @@
|
|||||||
---
|
|
||||||
apiVersion: helm.toolkit.fluxcd.io/v2
|
|
||||||
kind: HelmRelease
|
|
||||||
metadata:
|
|
||||||
name: vm-operator
|
|
||||||
namespace: monitoring
|
|
||||||
spec:
|
|
||||||
chart:
|
|
||||||
spec:
|
|
||||||
chart: victoria-metrics-operator
|
|
||||||
interval: 1h
|
|
||||||
sourceRef:
|
|
||||||
kind: HelmRepository
|
|
||||||
name: vm
|
|
||||||
version: 0.66.2
|
|
||||||
install:
|
|
||||||
crds: CreateReplace
|
|
||||||
strategy:
|
|
||||||
name: RetryOnFailure
|
|
||||||
retryInterval: 5m
|
|
||||||
interval: 30m
|
|
||||||
releaseName: vm-operator
|
|
||||||
targetNamespace: monitoring
|
|
||||||
timeout: 10m
|
|
||||||
upgrade:
|
|
||||||
crds: CreateReplace
|
|
||||||
strategy:
|
|
||||||
name: RetryOnFailure
|
|
||||||
retryInterval: 5m
|
|
||||||
valuesFrom:
|
|
||||||
- kind: ConfigMap
|
|
||||||
name: vm-operator-values
|
|
||||||
@@ -1,8 +0,0 @@
|
|||||||
---
|
|
||||||
apiVersion: kustomize.config.k8s.io/v1beta1
|
|
||||||
kind: Kustomization
|
|
||||||
resources:
|
|
||||||
- namespace.yaml
|
|
||||||
- repositories.yaml
|
|
||||||
- values.yaml
|
|
||||||
- helmrelease.yaml
|
|
||||||
@@ -1,5 +0,0 @@
|
|||||||
---
|
|
||||||
apiVersion: v1
|
|
||||||
kind: Namespace
|
|
||||||
metadata:
|
|
||||||
name: monitoring
|
|
||||||
@@ -1,18 +0,0 @@
|
|||||||
---
|
|
||||||
apiVersion: source.toolkit.fluxcd.io/v1
|
|
||||||
kind: HelmRepository
|
|
||||||
metadata:
|
|
||||||
name: vm
|
|
||||||
namespace: monitoring
|
|
||||||
spec:
|
|
||||||
interval: 1h
|
|
||||||
url: https://victoriametrics.github.io/helm-charts/
|
|
||||||
---
|
|
||||||
apiVersion: source.toolkit.fluxcd.io/v1
|
|
||||||
kind: HelmRepository
|
|
||||||
metadata:
|
|
||||||
name: prometheus-community
|
|
||||||
namespace: monitoring
|
|
||||||
spec:
|
|
||||||
interval: 1h
|
|
||||||
url: https://prometheus-community.github.io/helm-charts
|
|
||||||
@@ -1,23 +0,0 @@
|
|||||||
---
|
|
||||||
apiVersion: v1
|
|
||||||
kind: ConfigMap
|
|
||||||
metadata:
|
|
||||||
name: vm-operator-values
|
|
||||||
namespace: monitoring
|
|
||||||
data:
|
|
||||||
values.yaml: |
|
|
||||||
crds:
|
|
||||||
plain: true
|
|
||||||
cleanup:
|
|
||||||
enabled: false
|
|
||||||
operator:
|
|
||||||
disable_prometheus_converter: true
|
|
||||||
serviceMonitor:
|
|
||||||
enabled: false
|
|
||||||
resources:
|
|
||||||
requests:
|
|
||||||
cpu: 25m
|
|
||||||
memory: 96Mi
|
|
||||||
limits:
|
|
||||||
cpu: 250m
|
|
||||||
memory: 256Mi
|
|
||||||
@@ -1,14 +0,0 @@
|
|||||||
---
|
|
||||||
apiVersion: operator.victoriametrics.com/v1beta1
|
|
||||||
kind: VMServiceScrape
|
|
||||||
metadata:
|
|
||||||
name: kube-state-metrics
|
|
||||||
namespace: monitoring
|
|
||||||
spec:
|
|
||||||
endpoints:
|
|
||||||
- interval: 30s
|
|
||||||
port: http
|
|
||||||
selector:
|
|
||||||
matchLabels:
|
|
||||||
app.kubernetes.io/instance: kube-state-metrics
|
|
||||||
app.kubernetes.io/name: kube-state-metrics
|
|
||||||
@@ -1,39 +0,0 @@
|
|||||||
---
|
|
||||||
apiVersion: helm.toolkit.fluxcd.io/v2
|
|
||||||
kind: HelmRelease
|
|
||||||
metadata:
|
|
||||||
name: kube-state-metrics
|
|
||||||
namespace: monitoring
|
|
||||||
spec:
|
|
||||||
chart:
|
|
||||||
spec:
|
|
||||||
chart: kube-state-metrics
|
|
||||||
interval: 1h
|
|
||||||
sourceRef:
|
|
||||||
kind: HelmRepository
|
|
||||||
name: prometheus-community
|
|
||||||
namespace: monitoring
|
|
||||||
version: 8.3.0
|
|
||||||
install:
|
|
||||||
strategy:
|
|
||||||
name: RetryOnFailure
|
|
||||||
retryInterval: 5m
|
|
||||||
interval: 30m
|
|
||||||
releaseName: kube-state-metrics
|
|
||||||
targetNamespace: monitoring
|
|
||||||
timeout: 10m
|
|
||||||
upgrade:
|
|
||||||
strategy:
|
|
||||||
name: RetryOnFailure
|
|
||||||
retryInterval: 5m
|
|
||||||
values:
|
|
||||||
prometheus:
|
|
||||||
monitor:
|
|
||||||
enabled: false
|
|
||||||
resources:
|
|
||||||
requests:
|
|
||||||
cpu: 25m
|
|
||||||
memory: 64Mi
|
|
||||||
limits:
|
|
||||||
cpu: 250m
|
|
||||||
memory: 256Mi
|
|
||||||
@@ -1,41 +0,0 @@
|
|||||||
---
|
|
||||||
apiVersion: operator.victoriametrics.com/v1beta1
|
|
||||||
kind: VMNodeScrape
|
|
||||||
metadata:
|
|
||||||
name: kubelet
|
|
||||||
namespace: monitoring
|
|
||||||
spec:
|
|
||||||
bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token
|
|
||||||
honorLabels: true
|
|
||||||
honorTimestamps: false
|
|
||||||
interval: 30s
|
|
||||||
scheme: https
|
|
||||||
tlsConfig:
|
|
||||||
caFile: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
|
|
||||||
insecureSkipVerify: true
|
|
||||||
relabelConfigs:
|
|
||||||
- action: labelmap
|
|
||||||
regex: __meta_kubernetes_node_label_(.+)
|
|
||||||
- targetLabel: job
|
|
||||||
replacement: kubelet
|
|
||||||
---
|
|
||||||
apiVersion: operator.victoriametrics.com/v1beta1
|
|
||||||
kind: VMNodeScrape
|
|
||||||
metadata:
|
|
||||||
name: cadvisor
|
|
||||||
namespace: monitoring
|
|
||||||
spec:
|
|
||||||
bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token
|
|
||||||
honorLabels: true
|
|
||||||
honorTimestamps: false
|
|
||||||
interval: 30s
|
|
||||||
path: /metrics/cadvisor
|
|
||||||
scheme: https
|
|
||||||
tlsConfig:
|
|
||||||
caFile: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
|
|
||||||
insecureSkipVerify: true
|
|
||||||
relabelConfigs:
|
|
||||||
- action: labelmap
|
|
||||||
regex: __meta_kubernetes_node_label_(.+)
|
|
||||||
- targetLabel: job
|
|
||||||
replacement: cadvisor
|
|
||||||
@@ -1,9 +0,0 @@
|
|||||||
---
|
|
||||||
apiVersion: kustomize.config.k8s.io/v1beta1
|
|
||||||
kind: Kustomization
|
|
||||||
resources:
|
|
||||||
- vmagent.yaml
|
|
||||||
- kubelet-scrapes.yaml
|
|
||||||
- kube-state-metrics.yaml
|
|
||||||
- kube-state-metrics-scrape.yaml
|
|
||||||
- spire-agent-scrape.yaml
|
|
||||||
@@ -1,17 +0,0 @@
|
|||||||
---
|
|
||||||
apiVersion: operator.victoriametrics.com/v1beta1
|
|
||||||
kind: VMPodScrape
|
|
||||||
metadata:
|
|
||||||
name: spire-agent
|
|
||||||
namespace: monitoring
|
|
||||||
spec:
|
|
||||||
namespaceSelector:
|
|
||||||
matchNames:
|
|
||||||
- spire-system
|
|
||||||
podMetricsEndpoints:
|
|
||||||
- interval: 30s
|
|
||||||
port: prom
|
|
||||||
selector:
|
|
||||||
matchLabels:
|
|
||||||
app.kubernetes.io/instance: sandbox-spire
|
|
||||||
app.kubernetes.io/name: agent
|
|
||||||
@@ -1,21 +0,0 @@
|
|||||||
---
|
|
||||||
apiVersion: operator.victoriametrics.com/v1beta1
|
|
||||||
kind: VMAgent
|
|
||||||
metadata:
|
|
||||||
name: sandbox
|
|
||||||
namespace: monitoring
|
|
||||||
spec:
|
|
||||||
externalLabels:
|
|
||||||
cluster: sandbox
|
|
||||||
remoteWrite:
|
|
||||||
- url: https://metrics-write.ad.ddupan.top/api/v1/write
|
|
||||||
replicaCount: 1
|
|
||||||
resources:
|
|
||||||
requests:
|
|
||||||
cpu: 50m
|
|
||||||
memory: 128Mi
|
|
||||||
limits:
|
|
||||||
cpu: 500m
|
|
||||||
memory: 512Mi
|
|
||||||
scrapeInterval: 30s
|
|
||||||
selectAllByDefault: true
|
|
||||||
@@ -1,4 +0,0 @@
|
|||||||
apiVersion: kustomize.config.k8s.io/v1beta1
|
|
||||||
kind: Kustomization
|
|
||||||
resources:
|
|
||||||
- pools.yaml
|
|
||||||
@@ -1,123 +0,0 @@
|
|||||||
---
|
|
||||||
apiVersion: sandbox.opensandbox.io/v1alpha1
|
|
||||||
kind: Pool
|
|
||||||
metadata:
|
|
||||||
name: ci-pod
|
|
||||||
namespace: opensandbox
|
|
||||||
spec:
|
|
||||||
template:
|
|
||||||
metadata:
|
|
||||||
labels:
|
|
||||||
ci.ddupan.top/backend: pod
|
|
||||||
spec:
|
|
||||||
containers:
|
|
||||||
- name: sandbox
|
|
||||||
image: sandbox-registry.cn-zhangjiakou.cr.aliyuncs.com/opensandbox/code-interpreter:v1.1.0
|
|
||||||
command: [/opt/opensandbox/task-executor]
|
|
||||||
args: [-listen-addr=0.0.0.0:5758, -log-dir=/tmp]
|
|
||||||
env:
|
|
||||||
- name: SANDBOX_MAIN_CONTAINER
|
|
||||||
value: sandbox
|
|
||||||
- name: EXECD_ENVS
|
|
||||||
value: /opt/opensandbox/.env
|
|
||||||
- name: EXECD
|
|
||||||
value: /opt/opensandbox/execd
|
|
||||||
resources:
|
|
||||||
requests:
|
|
||||||
cpu: 100m
|
|
||||||
memory: 256Mi
|
|
||||||
limits:
|
|
||||||
cpu: "2"
|
|
||||||
memory: 4Gi
|
|
||||||
volumeMounts:
|
|
||||||
- name: opensandbox-bin
|
|
||||||
mountPath: /opt/opensandbox
|
|
||||||
- name: sandbox-storage
|
|
||||||
mountPath: /var/lib/sandbox
|
|
||||||
initContainers:
|
|
||||||
- name: task-executor-installer
|
|
||||||
image: sandbox-registry.cn-zhangjiakou.cr.aliyuncs.com/opensandbox/task-executor:v0.1.0
|
|
||||||
command: [/bin/sh, -c]
|
|
||||||
args: [cp /workspace/server /opt/opensandbox/task-executor && chmod 0755 /opt/opensandbox/task-executor]
|
|
||||||
volumeMounts:
|
|
||||||
- name: opensandbox-bin
|
|
||||||
mountPath: /opt/opensandbox
|
|
||||||
- name: execd-installer
|
|
||||||
image: sandbox-registry.cn-zhangjiakou.cr.aliyuncs.com/opensandbox/execd:v1.0.22
|
|
||||||
command: [/bin/sh, -c]
|
|
||||||
args: [cp ./execd /opt/opensandbox/execd && cp ./bootstrap.sh /opt/opensandbox/bootstrap.sh && chmod 0755 /opt/opensandbox/execd /opt/opensandbox/bootstrap.sh]
|
|
||||||
volumeMounts:
|
|
||||||
- name: opensandbox-bin
|
|
||||||
mountPath: /opt/opensandbox
|
|
||||||
volumes:
|
|
||||||
- name: opensandbox-bin
|
|
||||||
emptyDir: {}
|
|
||||||
- name: sandbox-storage
|
|
||||||
emptyDir: {}
|
|
||||||
capacitySpec:
|
|
||||||
bufferMax: 1
|
|
||||||
bufferMin: 0
|
|
||||||
poolMax: 4
|
|
||||||
poolMin: 0
|
|
||||||
---
|
|
||||||
apiVersion: sandbox.opensandbox.io/v1alpha1
|
|
||||||
kind: Pool
|
|
||||||
metadata:
|
|
||||||
name: ci-vm
|
|
||||||
namespace: opensandbox
|
|
||||||
spec:
|
|
||||||
template:
|
|
||||||
metadata:
|
|
||||||
labels:
|
|
||||||
ci.ddupan.top/backend: vm
|
|
||||||
spec:
|
|
||||||
runtimeClassName: kata-clh-runtime-rs
|
|
||||||
containers:
|
|
||||||
- name: sandbox
|
|
||||||
image: sandbox-registry.cn-zhangjiakou.cr.aliyuncs.com/opensandbox/code-interpreter:v1.1.0
|
|
||||||
command: [/opt/opensandbox/task-executor]
|
|
||||||
args: [-listen-addr=0.0.0.0:5758, -log-dir=/tmp]
|
|
||||||
env:
|
|
||||||
- name: SANDBOX_MAIN_CONTAINER
|
|
||||||
value: sandbox
|
|
||||||
- name: EXECD_ENVS
|
|
||||||
value: /opt/opensandbox/.env
|
|
||||||
- name: EXECD
|
|
||||||
value: /opt/opensandbox/execd
|
|
||||||
resources:
|
|
||||||
requests:
|
|
||||||
cpu: 250m
|
|
||||||
memory: 512Mi
|
|
||||||
limits:
|
|
||||||
cpu: "4"
|
|
||||||
memory: 8Gi
|
|
||||||
volumeMounts:
|
|
||||||
- name: opensandbox-bin
|
|
||||||
mountPath: /opt/opensandbox
|
|
||||||
- name: sandbox-storage
|
|
||||||
mountPath: /var/lib/sandbox
|
|
||||||
initContainers:
|
|
||||||
- name: task-executor-installer
|
|
||||||
image: sandbox-registry.cn-zhangjiakou.cr.aliyuncs.com/opensandbox/task-executor:v0.1.0
|
|
||||||
command: [/bin/sh, -c]
|
|
||||||
args: [cp /workspace/server /opt/opensandbox/task-executor && chmod 0755 /opt/opensandbox/task-executor]
|
|
||||||
volumeMounts:
|
|
||||||
- name: opensandbox-bin
|
|
||||||
mountPath: /opt/opensandbox
|
|
||||||
- name: execd-installer
|
|
||||||
image: sandbox-registry.cn-zhangjiakou.cr.aliyuncs.com/opensandbox/execd:v1.0.22
|
|
||||||
command: [/bin/sh, -c]
|
|
||||||
args: [cp ./execd /opt/opensandbox/execd && cp ./bootstrap.sh /opt/opensandbox/bootstrap.sh && chmod 0755 /opt/opensandbox/execd /opt/opensandbox/bootstrap.sh]
|
|
||||||
volumeMounts:
|
|
||||||
- name: opensandbox-bin
|
|
||||||
mountPath: /opt/opensandbox
|
|
||||||
volumes:
|
|
||||||
- name: opensandbox-bin
|
|
||||||
emptyDir: {}
|
|
||||||
- name: sandbox-storage
|
|
||||||
emptyDir: {}
|
|
||||||
capacitySpec:
|
|
||||||
bufferMax: 1
|
|
||||||
bufferMin: 0
|
|
||||||
poolMax: 2
|
|
||||||
poolMin: 0
|
|
||||||
@@ -1,19 +0,0 @@
|
|||||||
# OpenSandbox
|
|
||||||
|
|
||||||
Flux installs the upstream all-in-one OpenSandbox chart pinned to
|
|
||||||
`helm/opensandbox/0.2.2` (`8f01e935`). The API is cluster-internal and intentionally runs a
|
|
||||||
single replica until shared server state and HA behaviour have been validated.
|
|
||||||
|
|
||||||
`ci-pod` uses `runc`; `ci-vm` uses the separately managed
|
|
||||||
`kata-clh-runtime-rs` RuntimeClass. Both Pools start at zero and create capacity
|
|
||||||
on demand. They currently use the upstream interpreter image to validate the
|
|
||||||
Lifecycle API and Pool allocation independently of the CI scheduler cutover.
|
|
||||||
|
|
||||||
The dynamic runner worker, runner image, guest-local SPIRE Agent and Docker
|
|
||||||
sidecar are introduced only after this layer is Ready. In particular, do not
|
|
||||||
mount the host SPIFFE CSI socket into `ci-vm`: Unix sockets do not cross the
|
|
||||||
Kata VM boundary.
|
|
||||||
|
|
||||||
Smoke test both backends through the same API by creating sandboxes with
|
|
||||||
`extensions.poolRef` set to `ci-pod` and `ci-vm`, then confirm their
|
|
||||||
BatchSandboxes, Pods and VMMs disappear after deletion.
|
|
||||||
@@ -1,31 +0,0 @@
|
|||||||
apiVersion: helm.toolkit.fluxcd.io/v2
|
|
||||||
kind: HelmRelease
|
|
||||||
metadata:
|
|
||||||
name: opensandbox
|
|
||||||
namespace: opensandbox-system
|
|
||||||
spec:
|
|
||||||
chart:
|
|
||||||
spec:
|
|
||||||
chart: ./kubernetes/charts/opensandbox
|
|
||||||
interval: 1h
|
|
||||||
reconcileStrategy: Revision
|
|
||||||
sourceRef:
|
|
||||||
kind: GitRepository
|
|
||||||
name: opensandbox
|
|
||||||
driftDetection:
|
|
||||||
mode: enabled
|
|
||||||
install:
|
|
||||||
strategy:
|
|
||||||
name: RetryOnFailure
|
|
||||||
retryInterval: 5m
|
|
||||||
interval: 30m
|
|
||||||
releaseName: opensandbox
|
|
||||||
targetNamespace: opensandbox-system
|
|
||||||
timeout: 15m
|
|
||||||
upgrade:
|
|
||||||
strategy:
|
|
||||||
name: RetryOnFailure
|
|
||||||
retryInterval: 5m
|
|
||||||
valuesFrom:
|
|
||||||
- kind: ConfigMap
|
|
||||||
name: opensandbox-values
|
|
||||||
@@ -1,7 +0,0 @@
|
|||||||
apiVersion: kustomize.config.k8s.io/v1beta1
|
|
||||||
kind: Kustomization
|
|
||||||
resources:
|
|
||||||
- namespaces.yaml
|
|
||||||
- repository.yaml
|
|
||||||
- values.yaml
|
|
||||||
- helmrelease.yaml
|
|
||||||
@@ -1,10 +0,0 @@
|
|||||||
---
|
|
||||||
apiVersion: v1
|
|
||||||
kind: Namespace
|
|
||||||
metadata:
|
|
||||||
name: opensandbox-system
|
|
||||||
---
|
|
||||||
apiVersion: v1
|
|
||||||
kind: Namespace
|
|
||||||
metadata:
|
|
||||||
name: opensandbox
|
|
||||||
@@ -1,11 +0,0 @@
|
|||||||
apiVersion: source.toolkit.fluxcd.io/v1
|
|
||||||
kind: GitRepository
|
|
||||||
metadata:
|
|
||||||
name: opensandbox
|
|
||||||
namespace: opensandbox-system
|
|
||||||
spec:
|
|
||||||
interval: 1h
|
|
||||||
ref:
|
|
||||||
tag: helm/opensandbox/0.2.2
|
|
||||||
timeout: 60s
|
|
||||||
url: https://github.com/alibaba/OpenSandbox.git
|
|
||||||
@@ -1,62 +0,0 @@
|
|||||||
apiVersion: v1
|
|
||||||
kind: ConfigMap
|
|
||||||
metadata:
|
|
||||||
name: opensandbox-values
|
|
||||||
namespace: opensandbox-system
|
|
||||||
data:
|
|
||||||
values.yaml: |
|
|
||||||
opensandbox-controller:
|
|
||||||
controller:
|
|
||||||
logLevel: info
|
|
||||||
replicaCount: 1
|
|
||||||
metrics:
|
|
||||||
enabled: true
|
|
||||||
secure: false
|
|
||||||
port: 8080
|
|
||||||
resources:
|
|
||||||
requests:
|
|
||||||
cpu: 25m
|
|
||||||
memory: 64Mi
|
|
||||||
limits:
|
|
||||||
cpu: 500m
|
|
||||||
memory: 256Mi
|
|
||||||
|
|
||||||
opensandbox-server:
|
|
||||||
server:
|
|
||||||
replicaCount: 1
|
|
||||||
resources:
|
|
||||||
requests:
|
|
||||||
cpu: 100m
|
|
||||||
memory: 256Mi
|
|
||||||
limits:
|
|
||||||
cpu: "1"
|
|
||||||
memory: 1Gi
|
|
||||||
configToml: |
|
|
||||||
[server]
|
|
||||||
host = "0.0.0.0"
|
|
||||||
port = 80
|
|
||||||
api_key = ""
|
|
||||||
|
|
||||||
[log]
|
|
||||||
level = "INFO"
|
|
||||||
|
|
||||||
[runtime]
|
|
||||||
type = "kubernetes"
|
|
||||||
execd_image = "sandbox-registry.cn-zhangjiakou.cr.aliyuncs.com/opensandbox/execd:v1.0.22"
|
|
||||||
|
|
||||||
[kubernetes]
|
|
||||||
kubeconfig_path = ""
|
|
||||||
namespace = "opensandbox"
|
|
||||||
informer_enabled = true
|
|
||||||
informer_resync_seconds = 300
|
|
||||||
informer_watch_timeout_seconds = 60
|
|
||||||
snapshot_create_timeout_seconds = 900
|
|
||||||
workload_provider = "batchsandbox"
|
|
||||||
batchsandbox_template_file = "/etc/opensandbox/example.batchsandbox-template.yaml"
|
|
||||||
|
|
||||||
[egress]
|
|
||||||
image = "sandbox-registry.cn-zhangjiakou.cr.aliyuncs.com/opensandbox/egress:v1.1.6"
|
|
||||||
mode = "dns+nft"
|
|
||||||
|
|
||||||
opensandbox-node-agent:
|
|
||||||
enabled: false
|
|
||||||
@@ -1,7 +0,0 @@
|
|||||||
---
|
|
||||||
apiVersion: kustomize.config.k8s.io/v1beta1
|
|
||||||
kind: Kustomization
|
|
||||||
resources:
|
|
||||||
- values.yaml
|
|
||||||
- release.yaml
|
|
||||||
- smoke-identity.yaml
|
|
||||||
@@ -1,35 +0,0 @@
|
|||||||
---
|
|
||||||
apiVersion: helm.toolkit.fluxcd.io/v2
|
|
||||||
kind: HelmRelease
|
|
||||||
metadata:
|
|
||||||
name: sandbox-spire
|
|
||||||
namespace: spire-mgmt
|
|
||||||
spec:
|
|
||||||
chart:
|
|
||||||
spec:
|
|
||||||
chart: spire
|
|
||||||
interval: 1h
|
|
||||||
sourceRef:
|
|
||||||
kind: HelmRepository
|
|
||||||
name: spiffe-hardened
|
|
||||||
version: 0.30.2
|
|
||||||
dependsOn:
|
|
||||||
- name: spire-crds
|
|
||||||
namespace: spire-mgmt
|
|
||||||
driftDetection:
|
|
||||||
mode: enabled
|
|
||||||
install:
|
|
||||||
strategy:
|
|
||||||
name: RetryOnFailure
|
|
||||||
retryInterval: 5m
|
|
||||||
interval: 30m
|
|
||||||
releaseName: sandbox-spire
|
|
||||||
targetNamespace: spire-mgmt
|
|
||||||
timeout: 15m
|
|
||||||
upgrade:
|
|
||||||
strategy:
|
|
||||||
name: RetryOnFailure
|
|
||||||
retryInterval: 5m
|
|
||||||
valuesFrom:
|
|
||||||
- kind: ConfigMap
|
|
||||||
name: sandbox-spire-values
|
|
||||||
@@ -1,28 +0,0 @@
|
|||||||
---
|
|
||||||
apiVersion: v1
|
|
||||||
kind: Namespace
|
|
||||||
metadata:
|
|
||||||
name: spire-smoke
|
|
||||||
---
|
|
||||||
apiVersion: v1
|
|
||||||
kind: ServiceAccount
|
|
||||||
metadata:
|
|
||||||
name: spire-smoke
|
|
||||||
namespace: spire-smoke
|
|
||||||
---
|
|
||||||
apiVersion: spire.spiffe.io/v1alpha1
|
|
||||||
kind: ClusterSPIFFEID
|
|
||||||
metadata:
|
|
||||||
name: sandbox-spire-smoke
|
|
||||||
spec:
|
|
||||||
className: spire-mgmt-spire
|
|
||||||
namespaceSelector:
|
|
||||||
matchLabels:
|
|
||||||
kubernetes.io/metadata.name: spire-smoke
|
|
||||||
podSelector:
|
|
||||||
matchLabels:
|
|
||||||
app.kubernetes.io/name: spire-smoke
|
|
||||||
spiffeIDTemplate: spiffe://{{ .TrustDomain }}/sandbox/smoke
|
|
||||||
workloadSelectorTemplates:
|
|
||||||
- k8s:ns:spire-smoke
|
|
||||||
- k8s:sa:spire-smoke
|
|
||||||
@@ -1,86 +0,0 @@
|
|||||||
---
|
|
||||||
apiVersion: v1
|
|
||||||
kind: ConfigMap
|
|
||||||
metadata:
|
|
||||||
name: sandbox-spire-values
|
|
||||||
namespace: spire-mgmt
|
|
||||||
data:
|
|
||||||
values.yaml: |
|
|
||||||
global:
|
|
||||||
k8s:
|
|
||||||
clusterDomain: cluster.local
|
|
||||||
spire:
|
|
||||||
bundleConfigMap: spire-bundle
|
|
||||||
clusterName: sandbox
|
|
||||||
trustDomain: ddupan.top
|
|
||||||
namespaces:
|
|
||||||
create: false
|
|
||||||
system:
|
|
||||||
name: spire-system
|
|
||||||
server:
|
|
||||||
name: spire-server
|
|
||||||
recommendations:
|
|
||||||
enabled: true
|
|
||||||
namespaceLayout: true
|
|
||||||
namespacePSS: true
|
|
||||||
priorityClassName: true
|
|
||||||
strictMode: true
|
|
||||||
securityContexts: true
|
|
||||||
prometheus: false
|
|
||||||
|
|
||||||
spire-server:
|
|
||||||
enabled: false
|
|
||||||
|
|
||||||
spire-agent:
|
|
||||||
enabled: true
|
|
||||||
serviceAccount:
|
|
||||||
name: spire-agent
|
|
||||||
server:
|
|
||||||
address: spire-server.ad.ddupan.top
|
|
||||||
port: 8081
|
|
||||||
nodeAttestor:
|
|
||||||
k8sPSAT:
|
|
||||||
enabled: true
|
|
||||||
workloadAttestors:
|
|
||||||
k8s:
|
|
||||||
enabled: true
|
|
||||||
unix:
|
|
||||||
enabled: false
|
|
||||||
telemetry:
|
|
||||||
prometheus:
|
|
||||||
enabled: true
|
|
||||||
podMonitor:
|
|
||||||
enabled: false
|
|
||||||
resources:
|
|
||||||
requests:
|
|
||||||
cpu: 25m
|
|
||||||
memory: 64Mi
|
|
||||||
limits:
|
|
||||||
cpu: 250m
|
|
||||||
memory: 192Mi
|
|
||||||
|
|
||||||
spiffe-csi-driver:
|
|
||||||
enabled: true
|
|
||||||
resources:
|
|
||||||
requests:
|
|
||||||
cpu: 10m
|
|
||||||
memory: 32Mi
|
|
||||||
limits:
|
|
||||||
cpu: 100m
|
|
||||||
memory: 96Mi
|
|
||||||
|
|
||||||
spiffe-oidc-discovery-provider:
|
|
||||||
enabled: false
|
|
||||||
|
|
||||||
upstream:
|
|
||||||
enabled: false
|
|
||||||
tornjak-frontend:
|
|
||||||
enabled: false
|
|
||||||
spire-identity-exchange:
|
|
||||||
enabled: false
|
|
||||||
spike-keeper:
|
|
||||||
enabled: false
|
|
||||||
spike-nexus:
|
|
||||||
enabled: false
|
|
||||||
spike-pilot:
|
|
||||||
enabled: false
|
|
||||||
@@ -1,162 +0,0 @@
|
|||||||
---
|
|
||||||
apiVersion: v1
|
|
||||||
kind: ServiceAccount
|
|
||||||
metadata:
|
|
||||||
name: spire-controller-manager
|
|
||||||
namespace: spire-system
|
|
||||||
---
|
|
||||||
apiVersion: v1
|
|
||||||
kind: Secret
|
|
||||||
metadata:
|
|
||||||
name: spire-controller-manager-token
|
|
||||||
namespace: spire-system
|
|
||||||
annotations:
|
|
||||||
kubernetes.io/service-account.name: spire-controller-manager
|
|
||||||
type: kubernetes.io/service-account-token
|
|
||||||
---
|
|
||||||
apiVersion: rbac.authorization.k8s.io/v1
|
|
||||||
kind: ClusterRole
|
|
||||||
metadata:
|
|
||||||
name: spire-controller-manager
|
|
||||||
rules:
|
|
||||||
- apiGroups:
|
|
||||||
- ""
|
|
||||||
resources:
|
|
||||||
- endpoints
|
|
||||||
- namespaces
|
|
||||||
- nodes
|
|
||||||
- pods
|
|
||||||
verbs:
|
|
||||||
- get
|
|
||||||
- list
|
|
||||||
- watch
|
|
||||||
- apiGroups:
|
|
||||||
- spire.spiffe.io
|
|
||||||
resources:
|
|
||||||
- clusterfederatedtrustdomains
|
|
||||||
- clusterspiffeids
|
|
||||||
- clusterstaticentries
|
|
||||||
verbs:
|
|
||||||
- create
|
|
||||||
- delete
|
|
||||||
- get
|
|
||||||
- list
|
|
||||||
- patch
|
|
||||||
- update
|
|
||||||
- watch
|
|
||||||
- apiGroups:
|
|
||||||
- spire.spiffe.io
|
|
||||||
resources:
|
|
||||||
- clusterfederatedtrustdomains/finalizers
|
|
||||||
- clusterspiffeids/finalizers
|
|
||||||
- clusterstaticentries/finalizers
|
|
||||||
verbs:
|
|
||||||
- update
|
|
||||||
- apiGroups:
|
|
||||||
- spire.spiffe.io
|
|
||||||
resources:
|
|
||||||
- clusterfederatedtrustdomains/status
|
|
||||||
- clusterspiffeids/status
|
|
||||||
- clusterstaticentries/status
|
|
||||||
verbs:
|
|
||||||
- get
|
|
||||||
- patch
|
|
||||||
- update
|
|
||||||
---
|
|
||||||
apiVersion: rbac.authorization.k8s.io/v1
|
|
||||||
kind: ClusterRoleBinding
|
|
||||||
metadata:
|
|
||||||
name: spire-controller-manager
|
|
||||||
roleRef:
|
|
||||||
apiGroup: rbac.authorization.k8s.io
|
|
||||||
kind: ClusterRole
|
|
||||||
name: spire-controller-manager
|
|
||||||
subjects:
|
|
||||||
- kind: ServiceAccount
|
|
||||||
name: spire-controller-manager
|
|
||||||
namespace: spire-system
|
|
||||||
---
|
|
||||||
apiVersion: rbac.authorization.k8s.io/v1
|
|
||||||
kind: Role
|
|
||||||
metadata:
|
|
||||||
name: spire-bundle-publisher
|
|
||||||
namespace: spire-system
|
|
||||||
rules:
|
|
||||||
- apiGroups:
|
|
||||||
- ""
|
|
||||||
resources:
|
|
||||||
- configmaps
|
|
||||||
verbs:
|
|
||||||
- create
|
|
||||||
- delete
|
|
||||||
- get
|
|
||||||
- list
|
|
||||||
- patch
|
|
||||||
- update
|
|
||||||
- watch
|
|
||||||
---
|
|
||||||
apiVersion: rbac.authorization.k8s.io/v1
|
|
||||||
kind: RoleBinding
|
|
||||||
metadata:
|
|
||||||
name: spire-bundle-publisher
|
|
||||||
namespace: spire-system
|
|
||||||
roleRef:
|
|
||||||
apiGroup: rbac.authorization.k8s.io
|
|
||||||
kind: Role
|
|
||||||
name: spire-bundle-publisher
|
|
||||||
subjects:
|
|
||||||
- kind: ServiceAccount
|
|
||||||
name: spire-controller-manager
|
|
||||||
namespace: spire-system
|
|
||||||
---
|
|
||||||
apiVersion: rbac.authorization.k8s.io/v1
|
|
||||||
kind: Role
|
|
||||||
metadata:
|
|
||||||
name: spire-controller-manager-leader-election
|
|
||||||
namespace: spire-server
|
|
||||||
rules:
|
|
||||||
- apiGroups:
|
|
||||||
- ""
|
|
||||||
resources:
|
|
||||||
- configmaps
|
|
||||||
verbs:
|
|
||||||
- create
|
|
||||||
- delete
|
|
||||||
- get
|
|
||||||
- list
|
|
||||||
- patch
|
|
||||||
- update
|
|
||||||
- watch
|
|
||||||
- apiGroups:
|
|
||||||
- coordination.k8s.io
|
|
||||||
resources:
|
|
||||||
- leases
|
|
||||||
verbs:
|
|
||||||
- create
|
|
||||||
- delete
|
|
||||||
- get
|
|
||||||
- list
|
|
||||||
- patch
|
|
||||||
- update
|
|
||||||
- watch
|
|
||||||
- apiGroups:
|
|
||||||
- ""
|
|
||||||
resources:
|
|
||||||
- events
|
|
||||||
verbs:
|
|
||||||
- create
|
|
||||||
- patch
|
|
||||||
---
|
|
||||||
apiVersion: rbac.authorization.k8s.io/v1
|
|
||||||
kind: RoleBinding
|
|
||||||
metadata:
|
|
||||||
name: spire-controller-manager-leader-election
|
|
||||||
namespace: spire-server
|
|
||||||
roleRef:
|
|
||||||
apiGroup: rbac.authorization.k8s.io
|
|
||||||
kind: Role
|
|
||||||
name: spire-controller-manager-leader-election
|
|
||||||
subjects:
|
|
||||||
- kind: ServiceAccount
|
|
||||||
name: spire-controller-manager
|
|
||||||
namespace: spire-system
|
|
||||||
@@ -1,31 +0,0 @@
|
|||||||
---
|
|
||||||
apiVersion: helm.toolkit.fluxcd.io/v2
|
|
||||||
kind: HelmRelease
|
|
||||||
metadata:
|
|
||||||
name: spire-crds
|
|
||||||
namespace: spire-mgmt
|
|
||||||
spec:
|
|
||||||
chart:
|
|
||||||
spec:
|
|
||||||
chart: spire-crds
|
|
||||||
interval: 1h
|
|
||||||
sourceRef:
|
|
||||||
kind: HelmRepository
|
|
||||||
name: spiffe-hardened
|
|
||||||
version: 0.6.1
|
|
||||||
driftDetection:
|
|
||||||
mode: enabled
|
|
||||||
install:
|
|
||||||
crds: CreateReplace
|
|
||||||
strategy:
|
|
||||||
name: RetryOnFailure
|
|
||||||
retryInterval: 5m
|
|
||||||
interval: 30m
|
|
||||||
releaseName: spire-crds
|
|
||||||
targetNamespace: spire-mgmt
|
|
||||||
timeout: 10m
|
|
||||||
upgrade:
|
|
||||||
crds: CreateReplace
|
|
||||||
strategy:
|
|
||||||
name: RetryOnFailure
|
|
||||||
retryInterval: 5m
|
|
||||||
@@ -1,9 +0,0 @@
|
|||||||
---
|
|
||||||
apiVersion: kustomize.config.k8s.io/v1beta1
|
|
||||||
kind: Kustomization
|
|
||||||
resources:
|
|
||||||
- namespaces.yaml
|
|
||||||
- repository.yaml
|
|
||||||
- crds.yaml
|
|
||||||
- token-reviewer.yaml
|
|
||||||
- controller-manager.yaml
|
|
||||||
@@ -1,15 +0,0 @@
|
|||||||
---
|
|
||||||
apiVersion: v1
|
|
||||||
kind: Namespace
|
|
||||||
metadata:
|
|
||||||
name: spire-mgmt
|
|
||||||
---
|
|
||||||
apiVersion: v1
|
|
||||||
kind: Namespace
|
|
||||||
metadata:
|
|
||||||
name: spire-system
|
|
||||||
---
|
|
||||||
apiVersion: v1
|
|
||||||
kind: Namespace
|
|
||||||
metadata:
|
|
||||||
name: spire-server
|
|
||||||
@@ -1,9 +0,0 @@
|
|||||||
---
|
|
||||||
apiVersion: source.toolkit.fluxcd.io/v1
|
|
||||||
kind: HelmRepository
|
|
||||||
metadata:
|
|
||||||
name: spiffe-hardened
|
|
||||||
namespace: spire-mgmt
|
|
||||||
spec:
|
|
||||||
interval: 1h
|
|
||||||
url: https://spiffe.github.io/helm-charts-hardened/
|
|
||||||
@@ -1,51 +0,0 @@
|
|||||||
---
|
|
||||||
apiVersion: v1
|
|
||||||
kind: ServiceAccount
|
|
||||||
metadata:
|
|
||||||
name: spire-server-token-reviewer
|
|
||||||
namespace: spire-system
|
|
||||||
---
|
|
||||||
apiVersion: v1
|
|
||||||
kind: Secret
|
|
||||||
metadata:
|
|
||||||
name: spire-server-token-reviewer-token
|
|
||||||
namespace: spire-system
|
|
||||||
annotations:
|
|
||||||
kubernetes.io/service-account.name: spire-server-token-reviewer
|
|
||||||
type: kubernetes.io/service-account-token
|
|
||||||
---
|
|
||||||
apiVersion: rbac.authorization.k8s.io/v1
|
|
||||||
kind: ClusterRole
|
|
||||||
metadata:
|
|
||||||
name: spire-server-token-reviewer
|
|
||||||
rules:
|
|
||||||
- apiGroups:
|
|
||||||
- authentication.k8s.io
|
|
||||||
resources:
|
|
||||||
- tokenreviews
|
|
||||||
verbs:
|
|
||||||
- get
|
|
||||||
- list
|
|
||||||
- watch
|
|
||||||
- create
|
|
||||||
- apiGroups:
|
|
||||||
- ""
|
|
||||||
resources:
|
|
||||||
- nodes
|
|
||||||
- pods
|
|
||||||
verbs:
|
|
||||||
- get
|
|
||||||
- list
|
|
||||||
---
|
|
||||||
apiVersion: rbac.authorization.k8s.io/v1
|
|
||||||
kind: ClusterRoleBinding
|
|
||||||
metadata:
|
|
||||||
name: spire-server-token-reviewer
|
|
||||||
roleRef:
|
|
||||||
apiGroup: rbac.authorization.k8s.io
|
|
||||||
kind: ClusterRole
|
|
||||||
name: spire-server-token-reviewer
|
|
||||||
subjects:
|
|
||||||
- kind: ServiceAccount
|
|
||||||
name: spire-server-token-reviewer
|
|
||||||
namespace: spire-system
|
|
||||||
@@ -11,8 +11,7 @@ Authelia 提供;SPIRE 不替代人类 OIDC,也不承担目标服务的资源
|
|||||||
Flux 安装 SPIFFE hardened charts:
|
Flux 安装 SPIFFE hardened charts:
|
||||||
|
|
||||||
- `spire-crds` `0.6.1`;
|
- `spire-crds` `0.6.1`;
|
||||||
- 内部 fork 的 `spire` `0.30.2-ddupan.1`(SPIRE `1.15.3`),固定 Git tag
|
- `spire` `0.30.2`(SPIRE `1.15.3`);
|
||||||
`spire-0.30.2-ddupan.1`;
|
|
||||||
- SPIRE Server、Agent、Controller Manager、SPIFFE CSI Driver;
|
- SPIRE Server、Agent、Controller Manager、SPIFFE CSI Driver;
|
||||||
- OIDC Discovery Provider。
|
- OIDC Discovery Provider。
|
||||||
|
|
||||||
@@ -20,17 +19,6 @@ Flux 安装 SPIFFE hardened charts:
|
|||||||
API 或 Broker API。Trust domain 是 `ddupan.top`,Kubernetes cluster name 是
|
API 或 Broker API。Trust domain 是 `ddupan.top`,Kubernetes cluster name 是
|
||||||
`homelab`。
|
`homelab`。
|
||||||
|
|
||||||
同一 Server 也接受 cluster name 为 `sandbox` 的 external PSAT attestation。SPIRE gRPC
|
|
||||||
只通过内网 `spire-server.ad.ddupan.top:8081` 暴露;external PSAT、external
|
|
||||||
controller-manager 与 bundle publisher 使用由 sandbox Ansible bootstrap 的独立、受限
|
|
||||||
kubeconfig。Sandbox 不运行第二套 Server 或 OIDC Provider。
|
|
||||||
|
|
||||||
内部 fork 仅在上游 `spire-0.30.2` 基础上暴露
|
|
||||||
`use_pod_uid_for_agent_id`。现有 `sandbox` profile 保持 node UID 模式,供 DaemonSet
|
|
||||||
Agent 使用;独立的 `sandbox-kata` profile 复用同一 kubeconfig,但启用 Pod UID 模式,
|
|
||||||
供每个 Kata guest 内的临时 Agent 使用。不得把现有 `sandbox` profile 切换为 Pod UID,
|
|
||||||
否则会改变常驻 Agent 的 parent ID。
|
|
||||||
|
|
||||||
## PostgreSQL bootstrap
|
## PostgreSQL bootstrap
|
||||||
|
|
||||||
SPIRE registration datastore 使用共享 CloudNativePG:
|
SPIRE registration datastore 使用共享 CloudNativePG:
|
||||||
|
|||||||
@@ -41,7 +41,7 @@ SPIRE Server(trust domain: ddupan.top)
|
|||||||
|
|
||||||
| 项目 | 当前值 |
|
| 项目 | 当前值 |
|
||||||
|---|---|
|
|---|---|
|
||||||
| SPIRE chart | `0.30.2-ddupan.1`(内部 fork,基于 `0.30.2`) |
|
| SPIRE chart | `0.30.2` |
|
||||||
| SPIRE | `1.15.3` |
|
| SPIRE | `1.15.3` |
|
||||||
| SPIRE CRDs chart | `0.6.1` |
|
| SPIRE CRDs chart | `0.6.1` |
|
||||||
| trust domain | `ddupan.top` |
|
| trust domain | `ddupan.top` |
|
||||||
|
|||||||
@@ -1,11 +0,0 @@
|
|||||||
apiVersion: source.toolkit.fluxcd.io/v1
|
|
||||||
kind: GitRepository
|
|
||||||
metadata:
|
|
||||||
name: spiffe-hardened-fork
|
|
||||||
namespace: spire-mgmt
|
|
||||||
spec:
|
|
||||||
interval: 1h
|
|
||||||
ref:
|
|
||||||
tag: spire-0.30.2-ddupan.1
|
|
||||||
timeout: 60s
|
|
||||||
url: http://gitea-http.gitea.svc.cluster.local:3000/panxiao81/helm-charts-hardened.git
|
|
||||||
@@ -6,12 +6,12 @@ metadata:
|
|||||||
spec:
|
spec:
|
||||||
chart:
|
chart:
|
||||||
spec:
|
spec:
|
||||||
chart: ./charts/spire
|
chart: spire
|
||||||
interval: 1h
|
interval: 1h
|
||||||
reconcileStrategy: Revision
|
|
||||||
sourceRef:
|
sourceRef:
|
||||||
kind: GitRepository
|
kind: HelmRepository
|
||||||
name: spiffe-hardened-fork
|
name: spiffe-hardened
|
||||||
|
version: 0.30.2
|
||||||
dependsOn:
|
dependsOn:
|
||||||
- name: spire-crds
|
- name: spire-crds
|
||||||
namespace: spire-mgmt
|
namespace: spire-mgmt
|
||||||
|
|||||||
@@ -12,7 +12,6 @@ configMapGenerator:
|
|||||||
resources:
|
resources:
|
||||||
- namespaces.yaml
|
- namespaces.yaml
|
||||||
- helmrepository.yaml
|
- helmrepository.yaml
|
||||||
- gitrepository-fork.yaml
|
|
||||||
- helmrelease-crds.yaml
|
- helmrelease-crds.yaml
|
||||||
- helmrelease.yaml
|
- helmrelease.yaml
|
||||||
- httproute.yaml
|
- httproute.yaml
|
||||||
|
|||||||
@@ -30,47 +30,6 @@ spire-server:
|
|||||||
kind: statefulset
|
kind: statefulset
|
||||||
replicaCount: 1
|
replicaCount: 1
|
||||||
auditLogEnabled: true
|
auditLogEnabled: true
|
||||||
service:
|
|
||||||
type: LoadBalancer
|
|
||||||
port: 8081
|
|
||||||
loadBalancerIP: 192.168.10.127
|
|
||||||
kubeConfigs:
|
|
||||||
sandbox:
|
|
||||||
externalSecret:
|
|
||||||
name: spire-external-kubeconfigs
|
|
||||||
key: sandbox
|
|
||||||
sandbox-controller:
|
|
||||||
externalSecret:
|
|
||||||
name: spire-external-kubeconfigs
|
|
||||||
key: sandbox-controller
|
|
||||||
nodeAttestor:
|
|
||||||
externalK8sPSAT:
|
|
||||||
enabled: true
|
|
||||||
clusters:
|
|
||||||
sandbox:
|
|
||||||
kubeConfigName: sandbox
|
|
||||||
serviceAccountAllowList:
|
|
||||||
- spire-system:spire-agent
|
|
||||||
sandbox-kata:
|
|
||||||
kubeConfigName: sandbox
|
|
||||||
serviceAccountAllowList:
|
|
||||||
- spire-smoke:spire-smoke
|
|
||||||
usePodUIDForAgentID: true
|
|
||||||
externalControllerManagers:
|
|
||||||
enabled: true
|
|
||||||
clusters:
|
|
||||||
sandbox:
|
|
||||||
kubeConfigName: sandbox-controller
|
|
||||||
bundlePublisher:
|
|
||||||
externalK8sConfigMap:
|
|
||||||
enabled: true
|
|
||||||
clusters:
|
|
||||||
sandbox:
|
|
||||||
kubeConfigName: sandbox-controller
|
|
||||||
namespace: spire-system
|
|
||||||
configMapName: spire-bundle
|
|
||||||
configMapKey: bundle.spiffe
|
|
||||||
format: spiffe
|
|
||||||
persistence:
|
persistence:
|
||||||
# PostgreSQL stores registrations, but the disk KeyManager still needs durable
|
# PostgreSQL stores registrations, but the disk KeyManager still needs durable
|
||||||
# storage for the trust-domain signing keys.
|
# storage for the trust-domain signing keys.
|
||||||
@@ -106,11 +65,6 @@ spire-server:
|
|||||||
enabled: false
|
enabled: false
|
||||||
|
|
||||||
spire-agent:
|
spire-agent:
|
||||||
server:
|
|
||||||
# Keep the Agent endpoint aligned with spire-server.service.port. The
|
|
||||||
# chart defaults this to 443, which only remained unnoticed while the
|
|
||||||
# Agent's pre-upgrade gRPC connection stayed alive.
|
|
||||||
port: 8081
|
|
||||||
nodeAttestor:
|
nodeAttestor:
|
||||||
k8sPSAT:
|
k8sPSAT:
|
||||||
enabled: true
|
enabled: true
|
||||||
|
|||||||
Reference in New Issue
Block a user