Compare commits
3
Commits
main
..
54dd52f0ed
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
54dd52f0ed
|
||
|
|
76ec49ff48
|
||
|
|
00671e1729
|
@@ -16,6 +16,9 @@ on:
|
||||
- '.ansible-lint'
|
||||
- '.gitea/workflows/ansible.yml'
|
||||
|
||||
env:
|
||||
ANSIBLE_COLLECTIONS_PATH: /root/.ansible/collections
|
||||
|
||||
jobs:
|
||||
lint:
|
||||
runs-on: [self-hosted, pod]
|
||||
@@ -27,7 +30,6 @@ jobs:
|
||||
python3 -m pip install --user --break-system-packages \
|
||||
--index-url https://pypi.org/simple --quiet uv==0.11.7
|
||||
echo "$HOME/.local/bin" >> "$GITHUB_PATH"
|
||||
echo "ANSIBLE_COLLECTIONS_PATH=$HOME/.ansible/collections" >> "$GITHUB_ENV"
|
||||
|
||||
- name: Install ansible-lint and collections
|
||||
run: |
|
||||
@@ -48,7 +50,7 @@ jobs:
|
||||
- name: ansible-lint
|
||||
run: |
|
||||
export PATH="$HOME/.local/bin:$PATH"
|
||||
export ANSIBLE_COLLECTIONS_PATH="$PWD/infrastructure/samba-ad/ansible/collections:$HOME/.ansible/collections"
|
||||
export ANSIBLE_COLLECTIONS_PATH="$PWD/infrastructure/samba-ad/ansible/collections:/root/.ansible/collections"
|
||||
# 静态检查不应依赖生产 vault 凭据。一次性 checkout 可以去掉加密变量文件;
|
||||
# syntax-check 只验证结构,不需要解析变量的运行时值。
|
||||
rm -f \
|
||||
|
||||
@@ -1,22 +0,0 @@
|
||||
name: hydra-login
|
||||
on:
|
||||
pull_request:
|
||||
paths:
|
||||
- 'apps/hydra/login-consent/**'
|
||||
- '.gitea/workflows/hydra.yml'
|
||||
workflow_dispatch:
|
||||
jobs:
|
||||
verify:
|
||||
runs-on: [self-hosted, pod]
|
||||
steps:
|
||||
- uses: actions/checkout@v4
|
||||
- uses: actions/setup-go@v5
|
||||
with:
|
||||
go-version-file: apps/hydra/login-consent/go.mod
|
||||
cache-dependency-path: apps/hydra/login-consent/go.sum
|
||||
- name: Test authentication boundaries
|
||||
working-directory: apps/hydra/login-consent
|
||||
run: |
|
||||
go test -race ./...
|
||||
go vet ./...
|
||||
CGO_ENABLED=0 go build -trimpath .
|
||||
@@ -58,4 +58,4 @@ networks:
|
||||
default:
|
||||
ipam:
|
||||
config:
|
||||
- subnet: 172.28.0.0/24
|
||||
- subnet: 172.28.0.0/24
|
||||
|
||||
@@ -44,15 +44,3 @@ API、OIDC、Git/Flux 和 runner 均已验证。第二跳按明确决定跳过
|
||||
结构,或者 release notes 指出相关 breaking migration 时,才恢复停机一致备份、分阶段
|
||||
suspend、详细日志审计和扩展验收。出现启动失败或 migration error 时也立即升级为完整
|
||||
故障流程。
|
||||
|
||||
## Hydra 人类登录 PoC
|
||||
|
||||
新增 `hydra` OIDC 登录源,旧 `authelia` 入口保留。Hydra 通过通用 OIDC Login/Consent
|
||||
适配器转到现有 Authelia 完成人类认证;不是 Gitea 直接验证 LDAP 或 SPIFFE。
|
||||
入口为 `https://git.ddupan.top/user/oauth2/hydra`,需 LAN/Tailscale 可达 Hydra 内网域名。
|
||||
|
||||
新 client secret 通过 `ExternalSecret/gitea-hydra-oidc` 从 OpenBao 投射。沿用
|
||||
preferred_username、已验证邮箱与 groups;当前仍映射 gitea-admins,不在本轮切换组模型。
|
||||
先部署并验证 Hydra discovery 后再接入本配置,避免 Gitea init 因上游不可达而失败。
|
||||
|
||||
实际登录验收与部署状态见 wiki;依赖和回退见 [Hydra README](../hydra/README.md)。
|
||||
|
||||
@@ -116,13 +116,6 @@ gitea:
|
||||
scopes: openid profile email groups
|
||||
groupClaimName: groups
|
||||
adminGroup: gitea-admins
|
||||
- name: hydra
|
||||
provider: openidConnect
|
||||
existingSecret: gitea-hydra-oidc
|
||||
autoDiscoverUrl: https://hydra.ad.ddupan.top/.well-known/openid-configuration
|
||||
scopes: openid profile email groups
|
||||
groupClaimName: groups
|
||||
adminGroup: gitea-admins
|
||||
|
||||
persistence:
|
||||
size: 20Gi
|
||||
|
||||
@@ -1,22 +0,0 @@
|
||||
apiVersion: external-secrets.io/v1
|
||||
kind: ExternalSecret
|
||||
metadata:
|
||||
name: gitea-hydra-oidc
|
||||
namespace: gitea
|
||||
spec:
|
||||
refreshInterval: 1h
|
||||
secretStoreRef:
|
||||
kind: ClusterSecretStore
|
||||
name: openbao
|
||||
target:
|
||||
name: gitea-hydra-oidc
|
||||
creationPolicy: Owner
|
||||
template:
|
||||
data:
|
||||
key: gitea
|
||||
secret: "{{ .client_secret }}"
|
||||
data:
|
||||
- secretKey: client_secret
|
||||
remoteRef:
|
||||
key: k8s/hydra
|
||||
property: gitea_client_secret
|
||||
@@ -16,4 +16,3 @@ resources:
|
||||
- helmrepository.yaml
|
||||
- helmrelease.yaml
|
||||
- httproute.yaml
|
||||
- hydra-external-secret.yaml
|
||||
|
||||
@@ -1,102 +0,0 @@
|
||||
# Hydra 与 OIDC Login/Consent PoC
|
||||
|
||||
本目录提供独立 Hydra 签发服务,以及一个薄的 **OIDC 上游适配器**。当前上游配置为
|
||||
Authelia;适配器不连接 LDAP,也不管理用户目录。Samba AD、密码和 MFA 继续由现有
|
||||
Authelia 链路负责。第一轮只接入人类和 Gitea,不实现 agent 动态授权。
|
||||
|
||||
```text
|
||||
Gitea → Hydra → OIDC Login/Consent → Authelia → Samba AD
|
||||
← OIDC ← 经验证的上游身份 ← OIDC callback
|
||||
```
|
||||
|
||||
目标入口:
|
||||
|
||||
- `https://hydra.ad.ddupan.top`:Hydra 公共 OAuth2/OIDC endpoint。
|
||||
- `https://hydra-login.ad.ddupan.top`:上游 OIDC 登录及 consent 适配器。
|
||||
- `hydra-admin.hydra.svc.cluster.local:4445`:仅集群内管理接口,无 HTTPRoute。
|
||||
|
||||
均为 LAN/Tailscale 入口,复用 Envoy `eg/https` wildcard TLS。没有增加公网 tunnel。
|
||||
部署及实际验收状态以 wiki 和对应 PR 为准,文件存在不表示登录已验收。
|
||||
|
||||
## 首次使用与边界
|
||||
|
||||
在 Gitea 登录页选择 `hydra`,跳转到 Authelia 完成现有人类认证,再返回原有 Gitea
|
||||
账号。旧 `authelia` 登录源保留。Gitea 的账号关联和资源权限仍由 Gitea 维护。
|
||||
|
||||
适配器要求验证上游 issuer、audience、签名、过期时间和 nonce,使用 PKCE S256,
|
||||
并把单次 state 绑定到 Secure/HttpOnly/SameSite=Lax cookie。短期登录事务只存内存,
|
||||
最多 1024 个、10 分钟过期;单副本重启后正在登录的用户需重试,不存人类密码或 token。
|
||||
|
||||
Hydra subject 为上游 `(issuer, sub)` 的 SHA-256 加 `human:` 前缀,与可变邮箱/用户名
|
||||
分离。第一轮要求上游返回经过验证的 email 及 preferred_username;这些 claims 必须
|
||||
明确配置进 ID token。更换 issuer 会改变本 PoC 的 subject,正式迁移前需要身份绑定设计。
|
||||
|
||||
仅为显式 `ALLOWED_CLIENTS=gitea` 自动 consent,scope 限于 openid/profile/email/groups;
|
||||
拒绝额外 access-token audience,不发 refresh token。只按实际请求 scope 释放 claims。
|
||||
这不是通用的无人确认授权服务。组当前透传,沿用 Gitea 的 gitea-admins 映射;统一组
|
||||
模型和 agent 认证均在后续阶段。不存在对 Authelia 专有协议的调用。
|
||||
|
||||
NetworkPolicy 限制公共端口只接收 Envoy 流量,Hydra admin 只允许适配器访问。
|
||||
Hydra 使用正式模式,TLS 由 Envoy 终止;不使用 `--dev`。管理操作使用受控
|
||||
`kubectl port-forward`,不要将 admin 接口暴露到 Gateway。
|
||||
|
||||
## 依赖、秘密与初始化
|
||||
|
||||
依赖共享 CloudNativePG、OpenBao/ESO、Authelia OIDC、Envoy、Samba DNS、zot 镜像仓库。
|
||||
Hydra 使用独立 `hydra` database/role,不与其他应用共享数据库角色。
|
||||
|
||||
`kv/k8s/hydra` 保存 dsn、system_secret、upstream_client_secret、upstream_client_digest、
|
||||
gitea_client_secret;通过 ExternalSecret 投射,值不写入 Git。Bootstrap 创建角色及数据库
|
||||
后才启动 Hydra migration。system_secret 必须持久保存,不得在重启时随机重建。
|
||||
|
||||
Authelia 中新增 confidential client `hydra-login`:
|
||||
|
||||
- redirect URI:`https://hydra-login.ad.ddupan.top/callback`;
|
||||
- authorization policy:two_factor;grant:authorization_code;PKCE:S256;
|
||||
- token endpoint auth:client_secret_basic;scope:openid/profile/email/groups;
|
||||
- claims policy:把 preferred_username、name、email、email_verified、groups 放入 ID token;
|
||||
- client secret 的 PBKDF2 digest 存入 Authelia,原值仅供适配器使用。
|
||||
|
||||
Authelia 尚非 Flux 管理。修改 Helm values 时保留所有已有 clients 与 secret 引用,
|
||||
通过 `--reuse-values` 和最小 overlay 增加客户端,不能以本目录配置覆盖其完整 values。
|
||||
|
||||
Hydra 中注册 confidential client `gitea`,redirect URI 为
|
||||
`https://git.ddupan.top/user/oauth2/hydra/callback`,grant/response 为 authorization_code/code,
|
||||
scope 为 openid/profile/email/groups,token endpoint auth 为 client_secret_basic。
|
||||
Gitea 启动时读取 OIDC discovery,所以应先确认 Hydra 健康和 discovery 可达,再接入 Gitea。
|
||||
|
||||
## 构建与检查
|
||||
|
||||
```bash
|
||||
cd apps/hydra/login-consent
|
||||
go test -race ./...
|
||||
go vet ./...
|
||||
CGO_ENABLED=0 go build -trimpath -ldflags='-s -w' -o login-consent .
|
||||
docker build -t hydra-login-consent:VERSION .
|
||||
```
|
||||
|
||||
Go module 独立,依赖由 go.sum 锁定;Dockerfile 固定基础镜像 digest。
|
||||
使用已授权的短期 SPIFFE zot 凭据发布镜像,部署使用匿名拉取入口与不可变 digest。
|
||||
不把 registry 凭据写入源码或 build args。
|
||||
|
||||
```bash
|
||||
kubectl kustomize apps/hydra
|
||||
sudo k3s kubectl -n hydra get deployment,pod,externalsecret,httproute
|
||||
sudo k3s kubectl -n hydra logs deployment/hydra -c migrate
|
||||
sudo k3s kubectl -n hydra logs deployment/hydra-login
|
||||
```
|
||||
|
||||
日志不输出上游 token、授权 code、challenge 或秘密。登录失败先查两端 Pod 状态、
|
||||
DNS/discovery 连通性、client redirect URI 和 scope,再由用户重新发起登录。
|
||||
不要在故障排查中关闭签名验证、MFA 或 state/nonce 校验。
|
||||
|
||||
## 恢复与回退
|
||||
|
||||
保留共享 PostgreSQL 中 Hydra 数据及 OpenBao 秘密;数据库持有 clients、会话及签名密钥,
|
||||
单独重建 Deployment 不能替代恢复数据库。先恢复依赖,再启动 Hydra 和适配器。
|
||||
当前恢复仍依赖 homelab 共享基础设施,不能声称已完成独立灾备。
|
||||
|
||||
第一轮不切换 Authelia 的主入口。撤回 Gitea 的新增 Hydra 登录源即可回到旧入口;
|
||||
先撤消费者,再考虑停用 Hydra。不要删除旧 Authelia 登录源、用户或数据库作为回退手段。
|
||||
|
||||
跨服务设计见 [独立 IAM 草案](https://git.ddupan.top/panxiao81/homelab-wiki/src/branch/main/architecture/independent-iam-draft.md)。
|
||||
@@ -1,95 +0,0 @@
|
||||
apiVersion: apps/v1
|
||||
kind: Deployment
|
||||
metadata:
|
||||
name: hydra
|
||||
namespace: hydra
|
||||
spec:
|
||||
replicas: 1
|
||||
selector:
|
||||
matchLabels:
|
||||
app: hydra
|
||||
template:
|
||||
metadata:
|
||||
labels:
|
||||
app: hydra
|
||||
spec:
|
||||
automountServiceAccountToken: false
|
||||
securityContext:
|
||||
runAsNonRoot: true
|
||||
runAsUser: 1000
|
||||
runAsGroup: 1000
|
||||
seccompProfile:
|
||||
type: RuntimeDefault
|
||||
initContainers:
|
||||
- name: migrate
|
||||
image: docker.io/oryd/hydra:v26.2.0@sha256:ff67c7fb5f95074fa53374d41151713554960504b340cd3f95b09e65deaea2a9
|
||||
args:
|
||||
- migrate
|
||||
- sql
|
||||
- -e
|
||||
- --yes
|
||||
env:
|
||||
- name: DSN
|
||||
valueFrom:
|
||||
secretKeyRef:
|
||||
name: hydra
|
||||
key: dsn
|
||||
securityContext: &id002
|
||||
allowPrivilegeEscalation: false
|
||||
readOnlyRootFilesystem: true
|
||||
capabilities:
|
||||
drop:
|
||||
- ALL
|
||||
resources: &id001
|
||||
requests:
|
||||
cpu: 50m
|
||||
memory: 64Mi
|
||||
limits:
|
||||
memory: 256Mi
|
||||
containers:
|
||||
- name: hydra
|
||||
image: docker.io/oryd/hydra:v26.2.0@sha256:ff67c7fb5f95074fa53374d41151713554960504b340cd3f95b09e65deaea2a9
|
||||
args:
|
||||
- serve
|
||||
- all
|
||||
- --config
|
||||
- /etc/hydra/hydra.yaml
|
||||
- --sqa-opt-out
|
||||
env:
|
||||
- name: DSN
|
||||
valueFrom:
|
||||
secretKeyRef:
|
||||
name: hydra
|
||||
key: dsn
|
||||
- name: SECRETS_SYSTEM
|
||||
valueFrom:
|
||||
secretKeyRef:
|
||||
name: hydra
|
||||
key: system_secret
|
||||
ports:
|
||||
- name: public
|
||||
containerPort: 4444
|
||||
- name: admin
|
||||
containerPort: 4445
|
||||
resources: *id001
|
||||
securityContext: *id002
|
||||
volumeMounts:
|
||||
- name: config
|
||||
mountPath: /etc/hydra
|
||||
readOnly: true
|
||||
readinessProbe:
|
||||
httpGet:
|
||||
path: /health/ready
|
||||
port: admin
|
||||
initialDelaySeconds: 5
|
||||
periodSeconds: 5
|
||||
livenessProbe:
|
||||
httpGet:
|
||||
path: /health/alive
|
||||
port: admin
|
||||
initialDelaySeconds: 20
|
||||
periodSeconds: 20
|
||||
volumes:
|
||||
- name: config
|
||||
configMap:
|
||||
name: hydra-config
|
||||
@@ -1,16 +0,0 @@
|
||||
apiVersion: external-secrets.io/v1
|
||||
kind: ExternalSecret
|
||||
metadata:
|
||||
name: hydra
|
||||
namespace: hydra
|
||||
spec:
|
||||
refreshInterval: 1h
|
||||
secretStoreRef:
|
||||
kind: ClusterSecretStore
|
||||
name: openbao
|
||||
target:
|
||||
name: hydra
|
||||
creationPolicy: Owner
|
||||
dataFrom:
|
||||
- extract:
|
||||
key: k8s/hydra
|
||||
@@ -1,33 +0,0 @@
|
||||
apiVersion: gateway.networking.k8s.io/v1
|
||||
kind: HTTPRoute
|
||||
metadata:
|
||||
name: hydra-public
|
||||
namespace: hydra
|
||||
spec:
|
||||
parentRefs:
|
||||
- name: eg
|
||||
namespace: envoy-gateway-system
|
||||
sectionName: https
|
||||
hostnames:
|
||||
- hydra.ad.ddupan.top
|
||||
rules:
|
||||
- backendRefs:
|
||||
- name: hydra-public
|
||||
port: 4444
|
||||
---
|
||||
apiVersion: gateway.networking.k8s.io/v1
|
||||
kind: HTTPRoute
|
||||
metadata:
|
||||
name: hydra-login
|
||||
namespace: hydra
|
||||
spec:
|
||||
parentRefs:
|
||||
- name: eg
|
||||
namespace: envoy-gateway-system
|
||||
sectionName: https
|
||||
hostnames:
|
||||
- hydra-login.ad.ddupan.top
|
||||
rules:
|
||||
- backendRefs:
|
||||
- name: hydra-login
|
||||
port: 8080
|
||||
@@ -1,25 +0,0 @@
|
||||
serve:
|
||||
public:
|
||||
port: 4444
|
||||
admin:
|
||||
port: 4445
|
||||
tls:
|
||||
allow_termination_from:
|
||||
- 10.42.0.0/16
|
||||
cookies:
|
||||
same_site_mode: Lax
|
||||
urls:
|
||||
self:
|
||||
issuer: https://hydra.ad.ddupan.top
|
||||
public: https://hydra.ad.ddupan.top
|
||||
login: https://hydra-login.ad.ddupan.top/login
|
||||
consent: https://hydra-login.ad.ddupan.top/consent
|
||||
ttl:
|
||||
access_token: 15m
|
||||
id_token: 15m
|
||||
auth_code: 5m
|
||||
log:
|
||||
level: info
|
||||
leak_sensitive_values: false
|
||||
oauth2:
|
||||
expose_internal_errors: false
|
||||
@@ -1,15 +0,0 @@
|
||||
apiVersion: kustomize.config.k8s.io/v1beta1
|
||||
kind: Kustomization
|
||||
resources:
|
||||
- namespace.yaml
|
||||
- external-secret.yaml
|
||||
- deployment.yaml
|
||||
- login-deployment.yaml
|
||||
- services.yaml
|
||||
- httproutes.yaml
|
||||
- networkpolicy.yaml
|
||||
configMapGenerator:
|
||||
- name: hydra-config
|
||||
namespace: hydra
|
||||
files:
|
||||
- hydra.yaml
|
||||
@@ -1 +0,0 @@
|
||||
/login-consent
|
||||
@@ -1,4 +0,0 @@
|
||||
FROM gcr.io/distroless/static-debian12:nonroot@sha256:afa5c872c891853ca7fcf1f12c3edb23f7eeef36189728842dd51042ff57f7ab
|
||||
COPY login-consent /login-consent
|
||||
USER 65532:65532
|
||||
ENTRYPOINT ["/login-consent"]
|
||||
@@ -1,13 +0,0 @@
|
||||
module git.ddupan.top/panxiao81/homelab-infra/apps/hydra/login-consent
|
||||
|
||||
go 1.26.0
|
||||
|
||||
require (
|
||||
github.com/coreos/go-oidc/v3 v3.14.1
|
||||
golang.org/x/oauth2 v0.37.0
|
||||
)
|
||||
|
||||
require (
|
||||
github.com/go-jose/go-jose/v4 v4.0.5 // indirect
|
||||
golang.org/x/crypto v0.36.0 // indirect
|
||||
)
|
||||
@@ -1,18 +0,0 @@
|
||||
github.com/coreos/go-oidc/v3 v3.14.1 h1:9ePWwfdwC4QKRlCXsJGou56adA/owXczOzwKdOumLqk=
|
||||
github.com/coreos/go-oidc/v3 v3.14.1/go.mod h1:HaZ3szPaZ0e4r6ebqvsLWlk2Tn+aejfmrfah6hnSYEU=
|
||||
github.com/davecgh/go-spew v1.1.1 h1:vj9j/u1bqnvCEfJOwUhtlOARqs3+rkHYY13jYWTU97c=
|
||||
github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
|
||||
github.com/go-jose/go-jose/v4 v4.0.5 h1:M6T8+mKZl/+fNNuFHvGIzDz7BTLQPIounk/b9dw3AaE=
|
||||
github.com/go-jose/go-jose/v4 v4.0.5/go.mod h1:s3P1lRrkT8igV8D9OjyL4WRyHvjB6a4JSllnOrmmBOA=
|
||||
github.com/google/go-cmp v0.6.0 h1:ofyhxvXcZhMsU5ulbFiLKl/XBFqE1GSq7atu8tAmTRI=
|
||||
github.com/google/go-cmp v0.6.0/go.mod h1:17dUlkBOakJ0+DkrSSNjCkIjxS6bF9zb3elmeNGIjoY=
|
||||
github.com/pmezard/go-difflib v1.0.0 h1:4DBwDE0NGyQoBHbLQYPwSUPoCMWR5BEzIk/f1lZbAQM=
|
||||
github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4=
|
||||
github.com/stretchr/testify v1.10.0 h1:Xv5erBjTwe/5IxqUQTdXv5kgmIvbHo3QQyRwhJsOfJA=
|
||||
github.com/stretchr/testify v1.10.0/go.mod h1:r2ic/lqez/lEtzL7wO/rwa5dbSLXVDPFyf8C91i36aY=
|
||||
golang.org/x/crypto v0.36.0 h1:AnAEvhDddvBdpY+uR+MyHmuZzzNqXSe/GvuDeob5L34=
|
||||
golang.org/x/crypto v0.36.0/go.mod h1:Y4J0ReaxCR1IMaabaSMugxJES1EpwhBHhv2bDHklZvc=
|
||||
golang.org/x/oauth2 v0.37.0 h1:JUlcxA8oAtauLfiH8FX2/FkAWHAdi0QtGCGc+hofE98=
|
||||
golang.org/x/oauth2 v0.37.0/go.mod h1:IxwZNxUULJmpBFf9K/9NTMSIfZZuvuTy1gGxhigP/58=
|
||||
gopkg.in/yaml.v3 v3.0.1 h1:fxVm/GzAzEWqLHuvctI91KS9hhNmmWOoWu0XTYJS7CA=
|
||||
gopkg.in/yaml.v3 v3.0.1/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=
|
||||
@@ -1,285 +0,0 @@
|
||||
// Login/Consent adapter for a single trusted upstream and first-party clients.
|
||||
package main
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"context"
|
||||
"crypto/rand"
|
||||
"crypto/sha256"
|
||||
"crypto/subtle"
|
||||
"encoding/base64"
|
||||
"encoding/hex"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"fmt"
|
||||
"io"
|
||||
"log"
|
||||
"net/http"
|
||||
"net/url"
|
||||
"os"
|
||||
"strings"
|
||||
"sync"
|
||||
"time"
|
||||
|
||||
"github.com/coreos/go-oidc/v3/oidc"
|
||||
"golang.org/x/oauth2"
|
||||
)
|
||||
|
||||
const cookieName = "__Host-hydra-login"
|
||||
|
||||
type pending struct {
|
||||
Challenge, Nonce, Verifier string
|
||||
Expires time.Time
|
||||
}
|
||||
type claims struct {
|
||||
Username string `json:"preferred_username"`
|
||||
Email string `json:"email"`
|
||||
EmailVerified bool `json:"email_verified"`
|
||||
Name string `json:"name"`
|
||||
Groups []string `json:"groups"`
|
||||
}
|
||||
type flowRequest struct {
|
||||
Client struct {
|
||||
ID string `json:"client_id"`
|
||||
} `json:"client"`
|
||||
Subject string `json:"subject"`
|
||||
Scopes []string `json:"requested_scope"`
|
||||
Audience []string `json:"requested_access_token_audience"`
|
||||
Context claims `json:"context"`
|
||||
}
|
||||
type app struct {
|
||||
admin, public string
|
||||
client *http.Client
|
||||
oauth oauth2.Config
|
||||
verifier *oidc.IDTokenVerifier
|
||||
allowed map[string]bool
|
||||
mu sync.Mutex
|
||||
pending map[string]pending
|
||||
}
|
||||
|
||||
func required(key string) string {
|
||||
v := os.Getenv(key)
|
||||
if v == "" {
|
||||
log.Fatalf("missing %s", key)
|
||||
}
|
||||
return v
|
||||
}
|
||||
func random() string {
|
||||
b := make([]byte, 32)
|
||||
if _, err := rand.Read(b); err != nil {
|
||||
panic(err)
|
||||
}
|
||||
return base64.RawURLEncoding.EncodeToString(b)
|
||||
}
|
||||
func (a *app) api(ctx context.Context, method, path string, in, out any) error {
|
||||
var body io.Reader
|
||||
if in != nil {
|
||||
b, err := json.Marshal(in)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
body = bytes.NewReader(b)
|
||||
}
|
||||
req, err := http.NewRequestWithContext(ctx, method, a.admin+path, body)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
req.Header.Set("Content-Type", "application/json")
|
||||
resp, err := a.client.Do(req)
|
||||
if err != nil {
|
||||
return errors.New("Hydra unavailable")
|
||||
}
|
||||
defer resp.Body.Close()
|
||||
if resp.StatusCode < 200 || resp.StatusCode >= 300 {
|
||||
return fmt.Errorf("Hydra status %d", resp.StatusCode)
|
||||
}
|
||||
if out != nil {
|
||||
return json.NewDecoder(io.LimitReader(resp.Body, 1<<20)).Decode(out)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
func (a *app) request(r *http.Request, kind, challenge string) (flowRequest, error) {
|
||||
var f flowRequest
|
||||
if challenge == "" || len(challenge) > 8192 {
|
||||
return f, errors.New("missing or invalid challenge")
|
||||
}
|
||||
err := a.api(r.Context(), http.MethodGet, "/admin/oauth2/auth/requests/"+kind+"?"+kind+"_challenge="+url.QueryEscape(challenge), nil, &f)
|
||||
if err != nil {
|
||||
return f, err
|
||||
}
|
||||
if !a.allowed[f.Client.ID] {
|
||||
return f, errors.New("client not allowed")
|
||||
}
|
||||
return f, nil
|
||||
}
|
||||
func (a *app) accept(w http.ResponseWriter, r *http.Request, kind, challenge string, body any) {
|
||||
var result struct {
|
||||
Redirect string `json:"redirect_to"`
|
||||
}
|
||||
if err := a.api(r.Context(), http.MethodPut, "/admin/oauth2/auth/requests/"+kind+"/accept?"+kind+"_challenge="+url.QueryEscape(challenge), body, &result); err != nil {
|
||||
fail(w, 502)
|
||||
return
|
||||
}
|
||||
// Only Hydra's own authorization endpoint can receive a challenge verifier.
|
||||
u, err := url.Parse(result.Redirect)
|
||||
p, _ := url.Parse(a.public)
|
||||
if err != nil || u.Scheme != p.Scheme || u.Host != p.Host || u.User != nil || u.Path != "/oauth2/auth" {
|
||||
fail(w, 502)
|
||||
return
|
||||
}
|
||||
http.Redirect(w, r, result.Redirect, http.StatusSeeOther)
|
||||
}
|
||||
func fail(w http.ResponseWriter, status int) { http.Error(w, http.StatusText(status), status) }
|
||||
func (a *app) login(w http.ResponseWriter, r *http.Request) {
|
||||
challenge := r.URL.Query().Get("login_challenge")
|
||||
if _, err := a.request(r, "login", challenge); err != nil {
|
||||
fail(w, 403)
|
||||
return
|
||||
}
|
||||
state := random()
|
||||
p := pending{challenge, random(), oauth2.GenerateVerifier(), time.Now().Add(10 * time.Minute)}
|
||||
a.mu.Lock()
|
||||
for k, v := range a.pending {
|
||||
if time.Now().After(v.Expires) {
|
||||
delete(a.pending, k)
|
||||
}
|
||||
}
|
||||
if len(a.pending) >= 1024 {
|
||||
a.mu.Unlock()
|
||||
fail(w, 503)
|
||||
return
|
||||
}
|
||||
a.pending[state] = p
|
||||
a.mu.Unlock()
|
||||
http.SetCookie(w, &http.Cookie{Name: cookieName, Value: state, Path: "/", Secure: true, HttpOnly: true, SameSite: http.SameSiteLaxMode, MaxAge: 600})
|
||||
http.Redirect(w, r, a.oauth.AuthCodeURL(state, oidc.Nonce(p.Nonce), oauth2.S256ChallengeOption(p.Verifier)), http.StatusSeeOther)
|
||||
}
|
||||
func (a *app) take(r *http.Request) (pending, error) {
|
||||
state := r.URL.Query().Get("state")
|
||||
cookie, err := r.Cookie(cookieName)
|
||||
if err != nil || state == "" || subtle.ConstantTimeCompare([]byte(cookie.Value), []byte(state)) != 1 {
|
||||
return pending{}, errors.New("state mismatch")
|
||||
}
|
||||
a.mu.Lock()
|
||||
defer a.mu.Unlock()
|
||||
p, ok := a.pending[state]
|
||||
delete(a.pending, state)
|
||||
if !ok || time.Now().After(p.Expires) {
|
||||
return pending{}, errors.New("expired or used state")
|
||||
}
|
||||
return p, nil
|
||||
}
|
||||
func (a *app) callback(w http.ResponseWriter, r *http.Request) {
|
||||
p, err := a.take(r)
|
||||
if err != nil {
|
||||
fail(w, 403)
|
||||
return
|
||||
}
|
||||
http.SetCookie(w, &http.Cookie{Name: cookieName, Path: "/", Secure: true, HttpOnly: true, SameSite: http.SameSiteLaxMode, MaxAge: -1})
|
||||
if r.URL.Query().Get("error") != "" || r.URL.Query().Get("code") == "" {
|
||||
fail(w, 403)
|
||||
return
|
||||
}
|
||||
ctx := oidc.ClientContext(r.Context(), a.client)
|
||||
token, err := a.oauth.Exchange(ctx, r.URL.Query().Get("code"), oauth2.VerifierOption(p.Verifier))
|
||||
if err != nil {
|
||||
fail(w, 502)
|
||||
return
|
||||
}
|
||||
raw, ok := token.Extra("id_token").(string)
|
||||
if !ok {
|
||||
fail(w, 502)
|
||||
return
|
||||
}
|
||||
id, err := a.verifier.Verify(ctx, raw)
|
||||
if err != nil || id.Nonce != p.Nonce || id.Subject == "" {
|
||||
fail(w, 403)
|
||||
return
|
||||
}
|
||||
var c claims
|
||||
if id.Claims(&c) != nil || c.Username == "" || c.Email == "" || !c.EmailVerified {
|
||||
fail(w, 403)
|
||||
return
|
||||
}
|
||||
if _, err := a.request(r, "login", p.Challenge); err != nil {
|
||||
fail(w, 403)
|
||||
return
|
||||
}
|
||||
// Stable identity is tied to the verified upstream issuer+subject, never email.
|
||||
sum := sha256.Sum256([]byte(id.Issuer + "\x00" + id.Subject))
|
||||
a.accept(w, r, "login", p.Challenge, map[string]any{"subject": "human:" + hex.EncodeToString(sum[:]), "remember": false, "context": c})
|
||||
}
|
||||
func consentSession(f flowRequest) (map[string]any, error) {
|
||||
if !strings.HasPrefix(f.Subject, "human:") || f.Context.Username == "" || f.Context.Email == "" || !f.Context.EmailVerified {
|
||||
return nil, errors.New("invalid identity context")
|
||||
}
|
||||
allowed := map[string]bool{"openid": true, "profile": true, "email": true, "groups": true}
|
||||
session := map[string]any{"principal_type": "human"}
|
||||
for _, scope := range f.Scopes {
|
||||
if !allowed[scope] {
|
||||
return nil, errors.New("scope not allowed")
|
||||
}
|
||||
switch scope {
|
||||
case "profile":
|
||||
session["preferred_username"] = f.Context.Username
|
||||
session["name"] = f.Context.Name
|
||||
case "email":
|
||||
session["email"] = f.Context.Email
|
||||
session["email_verified"] = true
|
||||
case "groups":
|
||||
session["groups"] = f.Context.Groups
|
||||
}
|
||||
}
|
||||
if len(f.Audience) > 0 {
|
||||
return nil, errors.New("access token audience not allowed")
|
||||
}
|
||||
return session, nil
|
||||
}
|
||||
func (a *app) consent(w http.ResponseWriter, r *http.Request) {
|
||||
challenge := r.URL.Query().Get("consent_challenge")
|
||||
f, err := a.request(r, "consent", challenge)
|
||||
if err != nil {
|
||||
fail(w, 403)
|
||||
return
|
||||
}
|
||||
session, err := consentSession(f)
|
||||
if err != nil {
|
||||
fail(w, 403)
|
||||
return
|
||||
}
|
||||
// Explicit policy for pre-approved first-party clients only; no generic auto-consent.
|
||||
a.accept(w, r, "consent", challenge, map[string]any{"grant_scope": f.Scopes, "remember": false, "session": map[string]any{"id_token": session}})
|
||||
}
|
||||
func (a *app) handler() http.Handler {
|
||||
mux := http.NewServeMux()
|
||||
mux.HandleFunc("GET /healthz", func(w http.ResponseWriter, r *http.Request) { w.WriteHeader(200) })
|
||||
mux.HandleFunc("GET /login", a.login)
|
||||
mux.HandleFunc("GET /callback", a.callback)
|
||||
mux.HandleFunc("GET /consent", a.consent)
|
||||
return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
w.Header().Set("Cache-Control", "no-store")
|
||||
w.Header().Set("Referrer-Policy", "no-referrer")
|
||||
w.Header().Set("X-Content-Type-Options", "nosniff")
|
||||
w.Header().Set("Content-Security-Policy", "default-src 'none'; frame-ancestors 'none'")
|
||||
mux.ServeHTTP(w, r)
|
||||
})
|
||||
}
|
||||
func main() {
|
||||
client := &http.Client{Timeout: 15 * time.Second, CheckRedirect: func(r *http.Request, via []*http.Request) error { return http.ErrUseLastResponse }}
|
||||
issuer := required("UPSTREAM_ISSUER")
|
||||
ctx := oidc.ClientContext(context.Background(), client)
|
||||
provider, err := oidc.NewProvider(ctx, issuer)
|
||||
if err != nil {
|
||||
log.Fatal("upstream discovery failed")
|
||||
}
|
||||
clientID := required("UPSTREAM_CLIENT_ID")
|
||||
a := &app{admin: required("HYDRA_ADMIN_URL"), public: required("HYDRA_PUBLIC_URL"), client: client, allowed: map[string]bool{}, pending: map[string]pending{},
|
||||
oauth: oauth2.Config{ClientID: clientID, ClientSecret: required("UPSTREAM_CLIENT_SECRET"), RedirectURL: required("CALLBACK_URL"), Endpoint: provider.Endpoint(), Scopes: []string{"openid", "profile", "email", "groups"}},
|
||||
verifier: provider.Verifier(&oidc.Config{ClientID: clientID})}
|
||||
for _, id := range strings.Split(required("ALLOWED_CLIENTS"), ",") {
|
||||
a.allowed[id] = true
|
||||
}
|
||||
s := http.Server{Addr: ":8080", Handler: a.handler(), ReadHeaderTimeout: 5 * time.Second, ReadTimeout: 20 * time.Second, WriteTimeout: 45 * time.Second, IdleTimeout: 60 * time.Second, MaxHeaderBytes: 16384}
|
||||
log.Print("login/consent adapter listening on :8080")
|
||||
log.Fatal(s.ListenAndServe())
|
||||
}
|
||||
@@ -1,107 +0,0 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"net/url"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"golang.org/x/oauth2"
|
||||
)
|
||||
|
||||
func TestStateBoundToCookieSingleUseAndExpiry(t *testing.T) {
|
||||
a := &app{pending: map[string]pending{"valid": {Challenge: "challenge", Expires: time.Now().Add(time.Minute)}, "expired": {Expires: time.Now().Add(-time.Minute)}}}
|
||||
request := func(state, cookie string) *http.Request {
|
||||
r := httptest.NewRequest("GET", "https://login.example/callback?state="+state, nil)
|
||||
if cookie != "" {
|
||||
r.AddCookie(&http.Cookie{Name: cookieName, Value: cookie})
|
||||
}
|
||||
return r
|
||||
}
|
||||
for _, r := range []*http.Request{request("valid", ""), request("valid", "other"), request("expired", "expired")} {
|
||||
if _, err := a.take(r); err == nil {
|
||||
t.Fatal("invalid state accepted")
|
||||
}
|
||||
}
|
||||
if p, err := a.take(request("valid", "valid")); err != nil || p.Challenge != "challenge" {
|
||||
t.Fatal("valid state rejected")
|
||||
}
|
||||
if _, err := a.take(request("valid", "valid")); err == nil {
|
||||
t.Fatal("replayed state accepted")
|
||||
}
|
||||
}
|
||||
func TestConsentRejectsPrivilegeExpansionAndFiltersClaims(t *testing.T) {
|
||||
f := flowRequest{Subject: "human:known", Scopes: []string{"openid", "email"}, Context: claims{Username: "alice", Email: "[email protected]", EmailVerified: true, Groups: []string{"operators"}}}
|
||||
s, err := consentSession(f)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if _, ok := s["groups"]; ok {
|
||||
t.Fatal("groups leaked without scope")
|
||||
}
|
||||
if _, ok := s["preferred_username"]; ok {
|
||||
t.Fatal("profile leaked without scope")
|
||||
}
|
||||
for _, scope := range []string{"admin", "offline_access", "unknown"} {
|
||||
bad := f
|
||||
bad.Scopes = append([]string{"openid"}, scope)
|
||||
if _, err := consentSession(bad); err == nil {
|
||||
t.Fatalf("accepted %s", scope)
|
||||
}
|
||||
}
|
||||
f.Audience = []string{"other-service"}
|
||||
if _, err := consentSession(f); err == nil {
|
||||
t.Fatal("unexpected audience accepted")
|
||||
}
|
||||
f.Audience = nil
|
||||
f.Context.EmailVerified = false
|
||||
if _, err := consentSession(f); err == nil {
|
||||
t.Fatal("unverified email accepted")
|
||||
}
|
||||
}
|
||||
func TestLoginValidatesClientAndUsesPKCEAndNonce(t *testing.T) {
|
||||
admin := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
json.NewEncoder(w).Encode(map[string]any{"client": map[string]string{"client_id": r.URL.Query().Get("login_challenge")}})
|
||||
}))
|
||||
defer admin.Close()
|
||||
a := &app{admin: admin.URL, client: admin.Client(), allowed: map[string]bool{"gitea": true}, pending: map[string]pending{}, oauth: oauth2.Config{ClientID: "hydra-login", RedirectURL: "https://login.example/callback", Endpoint: oauth2.Endpoint{AuthURL: "https://upstream.example/authorize"}}}
|
||||
w := httptest.NewRecorder()
|
||||
a.handler().ServeHTTP(w, httptest.NewRequest("GET", "https://login.example/login?login_challenge=rogue", nil))
|
||||
if w.Code != 403 {
|
||||
t.Fatal("unknown client accepted")
|
||||
}
|
||||
w = httptest.NewRecorder()
|
||||
a.handler().ServeHTTP(w, httptest.NewRequest("GET", "https://login.example/login?login_challenge=gitea", nil))
|
||||
if w.Code != 303 {
|
||||
t.Fatalf("status %d", w.Code)
|
||||
}
|
||||
u, _ := url.Parse(w.Header().Get("Location"))
|
||||
q := u.Query()
|
||||
if q.Get("code_challenge_method") != "S256" || q.Get("code_challenge") == "" || q.Get("nonce") == "" || q.Get("state") == "" {
|
||||
t.Fatal("missing protocol binding")
|
||||
}
|
||||
cookies := w.Result().Cookies()
|
||||
if len(cookies) != 1 || !cookies[0].Secure || !cookies[0].HttpOnly || cookies[0].SameSite != http.SameSiteLaxMode || cookies[0].Value != q.Get("state") {
|
||||
t.Fatal("unsafe cookie")
|
||||
}
|
||||
if w.Header().Get("Cache-Control") != "no-store" {
|
||||
t.Fatal("missing cache protection")
|
||||
}
|
||||
}
|
||||
func TestHydraRedirectCannotLeaveTrustedOrigin(t *testing.T) {
|
||||
for _, target := range []string{"https://evil.example/oauth2/auth", "https://[email protected]/oauth2/auth", "https://hydra.example/other"} {
|
||||
admin := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
json.NewEncoder(w).Encode(map[string]string{"redirect_to": target})
|
||||
}))
|
||||
a := &app{admin: admin.URL, public: "https://hydra.example", client: admin.Client()}
|
||||
w := httptest.NewRecorder()
|
||||
a.accept(w, httptest.NewRequest("GET", "https://login.example/login", nil), "login", "challenge", map[string]string{"subject": "human:test"})
|
||||
if w.Code != 502 || strings.Contains(w.Header().Get("Location"), "evil") {
|
||||
t.Fatal("untrusted redirect accepted")
|
||||
}
|
||||
admin.Close()
|
||||
}
|
||||
}
|
||||
@@ -1,69 +0,0 @@
|
||||
apiVersion: apps/v1
|
||||
kind: Deployment
|
||||
metadata:
|
||||
name: hydra-login
|
||||
namespace: hydra
|
||||
spec:
|
||||
replicas: 1
|
||||
strategy:
|
||||
type: Recreate
|
||||
selector:
|
||||
matchLabels:
|
||||
app: hydra-login
|
||||
template:
|
||||
metadata:
|
||||
labels:
|
||||
app: hydra-login
|
||||
spec:
|
||||
automountServiceAccountToken: false
|
||||
securityContext:
|
||||
runAsNonRoot: true
|
||||
runAsUser: 65532
|
||||
runAsGroup: 65532
|
||||
seccompProfile:
|
||||
type: RuntimeDefault
|
||||
containers:
|
||||
- name: login-consent
|
||||
image: zot.ad.ddupan.top/iam/oidc-login-consent@sha256:fede9b9e93c457c4b7a8a6022d9df86ff5d5900d3f6b6c4f3439851a0ae1e944
|
||||
env:
|
||||
- name: HYDRA_ADMIN_URL
|
||||
value: http://hydra-admin.hydra.svc.cluster.local:4445
|
||||
- name: HYDRA_PUBLIC_URL
|
||||
value: https://hydra.ad.ddupan.top
|
||||
- name: UPSTREAM_ISSUER
|
||||
value: https://auth.ddupan.top
|
||||
- name: UPSTREAM_CLIENT_ID
|
||||
value: hydra-login
|
||||
- name: CALLBACK_URL
|
||||
value: https://hydra-login.ad.ddupan.top/callback
|
||||
- name: ALLOWED_CLIENTS
|
||||
value: gitea
|
||||
- name: UPSTREAM_CLIENT_SECRET
|
||||
valueFrom:
|
||||
secretKeyRef:
|
||||
name: hydra
|
||||
key: upstream_client_secret
|
||||
ports:
|
||||
- name: http
|
||||
containerPort: 8080
|
||||
resources:
|
||||
requests:
|
||||
cpu: 50m
|
||||
memory: 64Mi
|
||||
limits:
|
||||
memory: 256Mi
|
||||
securityContext:
|
||||
allowPrivilegeEscalation: false
|
||||
readOnlyRootFilesystem: true
|
||||
capabilities:
|
||||
drop:
|
||||
- ALL
|
||||
readinessProbe:
|
||||
httpGet:
|
||||
path: /healthz
|
||||
port: http
|
||||
livenessProbe:
|
||||
httpGet:
|
||||
path: /healthz
|
||||
port: http
|
||||
initialDelaySeconds: 10
|
||||
@@ -1,6 +0,0 @@
|
||||
apiVersion: v1
|
||||
kind: Namespace
|
||||
metadata:
|
||||
name: hydra
|
||||
labels:
|
||||
pod-security.kubernetes.io/enforce: restricted
|
||||
@@ -1,46 +0,0 @@
|
||||
apiVersion: networking.k8s.io/v1
|
||||
kind: NetworkPolicy
|
||||
metadata:
|
||||
name: hydra
|
||||
namespace: hydra
|
||||
spec:
|
||||
podSelector:
|
||||
matchLabels:
|
||||
app: hydra
|
||||
policyTypes:
|
||||
- Ingress
|
||||
ingress:
|
||||
- from:
|
||||
- namespaceSelector:
|
||||
matchLabels:
|
||||
kubernetes.io/metadata.name: envoy-gateway-system
|
||||
ports:
|
||||
- port: 4444
|
||||
protocol: TCP
|
||||
- from:
|
||||
- podSelector:
|
||||
matchLabels:
|
||||
app: hydra-login
|
||||
ports:
|
||||
- port: 4445
|
||||
protocol: TCP
|
||||
---
|
||||
apiVersion: networking.k8s.io/v1
|
||||
kind: NetworkPolicy
|
||||
metadata:
|
||||
name: hydra-login
|
||||
namespace: hydra
|
||||
spec:
|
||||
podSelector:
|
||||
matchLabels:
|
||||
app: hydra-login
|
||||
policyTypes:
|
||||
- Ingress
|
||||
ingress:
|
||||
- from:
|
||||
- namespaceSelector:
|
||||
matchLabels:
|
||||
kubernetes.io/metadata.name: envoy-gateway-system
|
||||
ports:
|
||||
- port: 8080
|
||||
protocol: TCP
|
||||
@@ -1,35 +0,0 @@
|
||||
apiVersion: v1
|
||||
kind: Service
|
||||
metadata:
|
||||
name: hydra-public
|
||||
namespace: hydra
|
||||
spec:
|
||||
selector:
|
||||
app: hydra
|
||||
ports:
|
||||
- port: 4444
|
||||
targetPort: 4444
|
||||
---
|
||||
apiVersion: v1
|
||||
kind: Service
|
||||
metadata:
|
||||
name: hydra-admin
|
||||
namespace: hydra
|
||||
spec:
|
||||
selector:
|
||||
app: hydra
|
||||
ports:
|
||||
- port: 4445
|
||||
targetPort: 4445
|
||||
---
|
||||
apiVersion: v1
|
||||
kind: Service
|
||||
metadata:
|
||||
name: hydra-login
|
||||
namespace: hydra
|
||||
spec:
|
||||
selector:
|
||||
app: hydra-login
|
||||
ports:
|
||||
- port: 8080
|
||||
targetPort: 8080
|
||||
@@ -1,157 +0,0 @@
|
||||
# Nexus Repository POC
|
||||
|
||||
本目录声明一个 Nexus Repository Community Edition POC,用来验证一次性 CI runner 通过
|
||||
网络服务复用 Ansible Galaxy、Go Modules 与 OCI/BuildKit 缓存。Nexus 固定为 `3.96.1`,
|
||||
镜像固定到官方 multi-arch index digest;LAN 入口为
|
||||
`https://nexus.ad.ddupan.top`。
|
||||
|
||||
## POC 边界
|
||||
|
||||
- 单副本 Deployment,`Recreate` 更新,避免一个 RWO 卷被两个 Pod 同时挂载。
|
||||
- `/nexus-data` 使用 `localpv-zfs-ceph` 上的 50 GiB RWO PVC。
|
||||
- 资源预算为 250m/2 GiB request、2 CPU/4 GiB limit;JVM heap 上限 2 GiB。
|
||||
- 使用容器默认的 embedded H2。它只用于 POC;正式接管 OCI 制品前必须迁移到外部
|
||||
PostgreSQL,并验证备份恢复。
|
||||
- 入口只在 LAN wildcard Gateway 上发布,不创建公网 DNS 或 Cloudflare route。
|
||||
- 不套 Authelia forward-auth;它会破坏 Go、Ansible 与 OCI 非浏览器客户端。
|
||||
- 现有 zot 保持不变。Nexus 完成 OCI、BuildKit cache 和恢复验收前不得迁移或删除 zot。
|
||||
|
||||
Terraform provider 创建 `ansible-public`、`go-public`、最小匿名权限与 OCI Bearer Token
|
||||
Realm。Nexus 3.94 才加入的原生 OCI repository 已有 REST API,但当前锁定的 community
|
||||
provider 尚未暴露 OCI resource;`terraform/reconcile-oci.sh` 因此根据 3.96.1 实例 Swagger
|
||||
固定的 JSON schema,幂等调和 `oci-hosted`、`oci-proxy` 与 `oci-public`。不得绕过该入口在
|
||||
UI 中创建无人管理的长期 repository。
|
||||
|
||||
## 部署
|
||||
|
||||
Flux 从 `clusters/homelab/apps/nexus.yaml` 协调本目录,并依赖 Envoy Gateway 与 OpenEBS。
|
||||
合并前只渲染配置,不直接 apply:
|
||||
|
||||
```bash
|
||||
kubectl kustomize apps/nexus
|
||||
```
|
||||
|
||||
合并并由 Flux 部署后检查:
|
||||
|
||||
```bash
|
||||
kubectl -n flux-system get kustomization nexus
|
||||
kubectl -n nexus get pod,pvc,service,httproute
|
||||
kubectl -n nexus logs deployment/nexus --tail=100
|
||||
```
|
||||
|
||||
启动可能需要数分钟,startup probe 允许最多十分钟。不要因初次启动较慢反复删除 Pod;
|
||||
先确认 PVC 已 Bound、Pod 没有 OOM,以及日志仍在推进。
|
||||
|
||||
DNS 期望状态已加入 `infrastructure/dns/records.yml`,需从 Samba AD Ansible root 以
|
||||
`--check --diff` 核对后再按其 README 应用 DNS tag。没有 DNS 时可先用 port-forward
|
||||
验证应用,但不能据此宣称 Gateway 路径已通过。
|
||||
|
||||
## 首次初始化与 Terraform
|
||||
|
||||
初始管理员密码生成在 PVC 的 `/nexus-data/admin.password`。只在交互式终端中读取并立即
|
||||
完成首次密码轮换;不得把密码复制进 shell tracing、工单、Git 或命令参数。随后将
|
||||
Terraform 管理账号的凭据存入 OpenBao,由 CI 通过 Terraform input variable 注入以下
|
||||
环境变量:
|
||||
|
||||
```text
|
||||
TF_VAR_nexus_url=https://nexus.ad.ddupan.top
|
||||
TF_VAR_nexus_username=admin
|
||||
TF_VAR_nexus_password=<OpenBao kv/infra/nexus 的 admin_password 字段>
|
||||
```
|
||||
|
||||
`terraform/` 使用 `sonatype-nexus-community/sonatyperepo` 1.17.0,当前声明:
|
||||
|
||||
- `ansible-galaxy-proxy` → `https://galaxy.ansible.com`
|
||||
- `ansible-public` group
|
||||
- `go-proxy` → `https://proxy.golang.org`
|
||||
- `go-public` group
|
||||
|
||||
provider credential 不写入 HCL 或 tfvars。正式 apply 前还必须为这个独立 Terraform root
|
||||
配置远端 backend;本地 state 只允许用于可丢弃的 POC,不提交。验证命令:
|
||||
|
||||
```bash
|
||||
terraform -chdir=apps/nexus/terraform init -backend=false
|
||||
terraform -chdir=apps/nexus/terraform validate
|
||||
```
|
||||
|
||||
先以 `--check` 查看 OCI repository 漂移,再明确 apply;脚本只从上述环境变量取得凭据,
|
||||
用临时 `0600` netrc 调用 REST API,退出时删除:
|
||||
|
||||
```bash
|
||||
apps/nexus/terraform/reconcile-oci.sh --check
|
||||
apps/nexus/terraform/reconcile-oci.sh --apply
|
||||
```
|
||||
|
||||
Terraform 同时把内置 `anonymous` 用户从默认的全仓库 `nx-anonymous` 角色收窄到
|
||||
`ansible-public`、其返回制品 URL 使用的 `ansible-galaxy-proxy`、`go-public`,以及
|
||||
`oci-public`/`oci-proxy` 的 `browse/read` 权限。`oci-hosted` 不向匿名用户开放。首次接管
|
||||
已有实例时先执行
|
||||
`terraform import sonatyperepo_user.anonymous anonymous,default`,再 apply;不要先启用默认的
|
||||
全仓库匿名读取。
|
||||
|
||||
2026-09-20 的 POC 现场验收已确认:Flux 与 Pod Ready、PVC Bound、HTTPRoute 通过 HTTPS
|
||||
返回 Nexus 状态 200,Samba DNS 已幂等收敛。全新客户端目录通过匿名入口下载
|
||||
`community.general:11.2.0` 时冷缓存为 8.49 秒、热缓存为 1.89 秒,两次 tarball SHA-256
|
||||
一致;`golang.org/x/[email protected]` 为 2.92 秒与 1.51 秒。
|
||||
|
||||
## 客户端验收
|
||||
|
||||
先验证冷缓存,再原样重复命令验证热缓存;记录 Nexus 请求、上游流量和耗时,不只观察
|
||||
命令成功。Ansible 配置的 URL 必须以 `/` 结尾:
|
||||
|
||||
```ini
|
||||
[galaxy]
|
||||
server_list = nexus
|
||||
|
||||
[galaxy_server.nexus]
|
||||
url = https://nexus.ad.ddupan.top/repository/ansible-public/
|
||||
```
|
||||
|
||||
```bash
|
||||
ansible-galaxy collection install -r collections/requirements.yml \
|
||||
-p .ansible/collections
|
||||
```
|
||||
|
||||
Go POC 使用:
|
||||
|
||||
```bash
|
||||
GOPROXY=https://nexus.ad.ddupan.top/repository/go-public/ go mod download
|
||||
```
|
||||
|
||||
私有 `git.ddupan.top/*` module 的 `GOPRIVATE`、凭据与是否允许 `direct` fallback 在实际
|
||||
workflow 中单独决定;不要让私有 module path 意外发往公共 proxy。
|
||||
|
||||
OCI 使用 path-based routing:匿名公共拉取地址形如
|
||||
`nexus.ad.ddupan.top/oci-public/library/alpine:3.22`,认证写入地址形如
|
||||
`nexus.ad.ddupan.top/oci-hosted/<namespace>/<image>:<tag>`。2026-09-20 现场验收结果:
|
||||
|
||||
- `oci-public` 匿名代理拉取 Alpine 冷缓存 4.75 秒、热缓存 0.80 秒,digest 一致;
|
||||
- `oci-hosted` 认证 push/pull 成功,匿名 pull 返回 401;
|
||||
- amd64/arm64 OCI image index push 成功,两个平台 manifest 可见;
|
||||
- Helm chart push/pull digest 与本地 tarball SHA-256 一致;
|
||||
- Cosign 3.1.3 使用一次性密钥签名并验证成功,OCI 1.1 referrers API 返回一个
|
||||
`application/vnd.dev.sigstore.bundle.v0.3+json` artifact;
|
||||
- BuildKit `registry` cache 以 `mode=max` 导出成功,销毁首个 builder 后由新 builder 导入,
|
||||
两个 `RUN` step 均明确命中 `CACHED`。
|
||||
|
||||
本机安装的测试客户端包括 `/usr/local/bin/cosign` 3.1.3;安装时核对官方 Linux amd64
|
||||
binary SHA-256 `4629c757b7618056f8ddd7e2625ae9fdd94c0372a65049520bc7d9df9efc7f71`。
|
||||
上述结果仍不代表备份恢复、外部 PostgreSQL 或正式 publisher service account 已完成;
|
||||
这些项目通过前不得迁移或删除 zot。
|
||||
|
||||
## 数据与恢复
|
||||
|
||||
POC 的数据库、配置、blob、初始管理员状态都位于 `nexus-data` PVC。删除 Deployment
|
||||
不会删除 PVC;删除 PVC 会永久删除整个 POC。当前没有独立备份,不能将它用于唯一副本的
|
||||
正式制品。
|
||||
|
||||
恢复验证至少包括:停止写入、取得一致备份、在独立 PVC/实例恢复、登录、列出 repository、
|
||||
拉取已缓存的 Ansible/Go 制品,并核对 OCI digest/referrers。正式化时再把数据库迁移至
|
||||
外部 PostgreSQL,并分别定义数据库与 blob 的备份、恢复顺序和 RPO。
|
||||
|
||||
参考:
|
||||
|
||||
- [Nexus OCI repositories](https://help.sonatype.com/en/oci-repositories.html)
|
||||
- [Nexus Ansible repositories](https://help.sonatype.com/en/ansible-repositories.html)
|
||||
- [Nexus Go repositories](https://help.sonatype.com/en/go-repositories.html)
|
||||
- [官方容器镜像](https://hub.docker.com/r/sonatype/nexus3)
|
||||
@@ -1,82 +0,0 @@
|
||||
apiVersion: apps/v1
|
||||
kind: Deployment
|
||||
metadata:
|
||||
name: nexus
|
||||
namespace: nexus
|
||||
labels:
|
||||
app.kubernetes.io/name: nexus
|
||||
spec:
|
||||
replicas: 1
|
||||
strategy:
|
||||
type: Recreate
|
||||
selector:
|
||||
matchLabels:
|
||||
app.kubernetes.io/name: nexus
|
||||
template:
|
||||
metadata:
|
||||
labels:
|
||||
app.kubernetes.io/name: nexus
|
||||
spec:
|
||||
automountServiceAccountToken: false
|
||||
securityContext:
|
||||
fsGroup: 200
|
||||
fsGroupChangePolicy: OnRootMismatch
|
||||
runAsGroup: 200
|
||||
runAsNonRoot: true
|
||||
runAsUser: 200
|
||||
seccompProfile:
|
||||
type: RuntimeDefault
|
||||
terminationGracePeriodSeconds: 120
|
||||
containers:
|
||||
- name: nexus
|
||||
image: docker.io/sonatype/nexus3:3.96.1@sha256:56142f13432cf072e017aebb2025f201e42ae36ff40bb82618c702504c61f7dd
|
||||
imagePullPolicy: IfNotPresent
|
||||
env:
|
||||
- name: INSTALL4J_ADD_VM_PARAMS
|
||||
value: >-
|
||||
-Xms1024m -Xmx2048m -XX:MaxDirectMemorySize=1024m
|
||||
-Djava.util.prefs.userRoot=/nexus-data/javaprefs
|
||||
ports:
|
||||
- name: http
|
||||
containerPort: 8081
|
||||
protocol: TCP
|
||||
resources:
|
||||
requests:
|
||||
cpu: 250m
|
||||
memory: 2Gi
|
||||
limits:
|
||||
cpu: "2"
|
||||
memory: 4Gi
|
||||
securityContext:
|
||||
allowPrivilegeEscalation: false
|
||||
capabilities:
|
||||
drop:
|
||||
- ALL
|
||||
startupProbe:
|
||||
httpGet:
|
||||
path: /service/rest/v1/status
|
||||
port: http
|
||||
failureThreshold: 60
|
||||
periodSeconds: 10
|
||||
timeoutSeconds: 5
|
||||
readinessProbe:
|
||||
httpGet:
|
||||
path: /service/rest/v1/status
|
||||
port: http
|
||||
failureThreshold: 6
|
||||
periodSeconds: 10
|
||||
timeoutSeconds: 5
|
||||
livenessProbe:
|
||||
httpGet:
|
||||
path: /service/rest/v1/status
|
||||
port: http
|
||||
failureThreshold: 6
|
||||
periodSeconds: 30
|
||||
timeoutSeconds: 5
|
||||
volumeMounts:
|
||||
- name: data
|
||||
mountPath: /nexus-data
|
||||
volumes:
|
||||
- name: data
|
||||
persistentVolumeClaim:
|
||||
claimName: nexus-data
|
||||
@@ -1,16 +0,0 @@
|
||||
apiVersion: gateway.networking.k8s.io/v1
|
||||
kind: HTTPRoute
|
||||
metadata:
|
||||
name: nexus
|
||||
namespace: nexus
|
||||
spec:
|
||||
parentRefs:
|
||||
- name: eg
|
||||
namespace: envoy-gateway-system
|
||||
sectionName: https
|
||||
hostnames:
|
||||
- nexus.ad.ddupan.top
|
||||
rules:
|
||||
- backendRefs:
|
||||
- name: nexus
|
||||
port: 8081
|
||||
@@ -1,9 +0,0 @@
|
||||
apiVersion: kustomize.config.k8s.io/v1beta1
|
||||
kind: Kustomization
|
||||
resources:
|
||||
- namespace.yaml
|
||||
- pvc.yaml
|
||||
- deployment.yaml
|
||||
- service.yaml
|
||||
- httproute.yaml
|
||||
- networkpolicy.yaml
|
||||
@@ -1,8 +0,0 @@
|
||||
apiVersion: v1
|
||||
kind: Namespace
|
||||
metadata:
|
||||
name: nexus
|
||||
labels:
|
||||
pod-security.kubernetes.io/enforce: restricted
|
||||
pod-security.kubernetes.io/audit: restricted
|
||||
pod-security.kubernetes.io/warn: restricted
|
||||
@@ -1,23 +0,0 @@
|
||||
apiVersion: networking.k8s.io/v1
|
||||
kind: NetworkPolicy
|
||||
metadata:
|
||||
name: nexus-ingress
|
||||
namespace: nexus
|
||||
spec:
|
||||
podSelector:
|
||||
matchLabels:
|
||||
app.kubernetes.io/name: nexus
|
||||
policyTypes:
|
||||
- Ingress
|
||||
ingress:
|
||||
- from:
|
||||
- namespaceSelector:
|
||||
matchLabels:
|
||||
kubernetes.io/metadata.name: envoy-gateway-system
|
||||
podSelector:
|
||||
matchLabels:
|
||||
gateway.envoyproxy.io/owning-gateway-name: eg
|
||||
gateway.envoyproxy.io/owning-gateway-namespace: envoy-gateway-system
|
||||
ports:
|
||||
- protocol: TCP
|
||||
port: 8081
|
||||
@@ -1,12 +0,0 @@
|
||||
apiVersion: v1
|
||||
kind: PersistentVolumeClaim
|
||||
metadata:
|
||||
name: nexus-data
|
||||
namespace: nexus
|
||||
spec:
|
||||
accessModes:
|
||||
- ReadWriteOnce
|
||||
resources:
|
||||
requests:
|
||||
storage: 50Gi
|
||||
storageClassName: localpv-zfs-ceph
|
||||
@@ -1,14 +0,0 @@
|
||||
apiVersion: v1
|
||||
kind: Service
|
||||
metadata:
|
||||
name: nexus
|
||||
namespace: nexus
|
||||
spec:
|
||||
type: ClusterIP
|
||||
selector:
|
||||
app.kubernetes.io/name: nexus
|
||||
ports:
|
||||
- name: http
|
||||
port: 8081
|
||||
protocol: TCP
|
||||
targetPort: http
|
||||
@@ -1,6 +0,0 @@
|
||||
.terraform/
|
||||
*.tfstate
|
||||
*.tfstate.*
|
||||
*.tfplan
|
||||
crash.log
|
||||
crash.*.log
|
||||
Generated
-24
@@ -1,24 +0,0 @@
|
||||
# This file is maintained automatically by "terraform init".
|
||||
# Manual edits may be lost in future updates.
|
||||
|
||||
provider "registry.terraform.io/sonatype-nexus-community/sonatyperepo" {
|
||||
version = "1.17.0"
|
||||
constraints = "1.17.0"
|
||||
hashes = [
|
||||
"h1:uKhvhhhI7B+HBsh0zq/ybqKt+EnOGyI6rjcRCtj79ZA=",
|
||||
"zh:0dde99e7b343fa01f8eefc378171fb8621bedb20f59157d6cc8e3d46c738105f",
|
||||
"zh:3315929df254a3a6ac27c8c846c2006f7d2a91fadc014351bc4d617f948e5bf9",
|
||||
"zh:36be5a455af3ce4e187de26753de63e78c1ee9a32dba0135c6cf96a6c1fff25f",
|
||||
"zh:3f73f7ff57b8c339a7c7ac37653e2dc0b2dd9dcc3f3a538788e7e3ac838337b2",
|
||||
"zh:40286ecca4c22ab7ae90618ac6d2743f5055199dac81cf5204a4a397c784d439",
|
||||
"zh:4d24e5c0195fb3155b1967583ee64cfeda402d7cc7f3c73369438f6c69f4245b",
|
||||
"zh:828a9d7aceaac36af7f9c07af43ec8d20a89148780645d170ffb1c68b2da792d",
|
||||
"zh:a5ab04de3fe626ec57c832618c6f990abd6610f81e132621651e0b180b970cff",
|
||||
"zh:a959fa6090a8c0f53739879184e7346423494aee598003df0d1ab4a22b2eee91",
|
||||
"zh:bdda26c2f03f918bbe59e75abea44868fafda019c3a543725331195df126350b",
|
||||
"zh:d8048e149ee97ba62971e6a79355d59887bc6d10fcf72cc2feff3d0a2582670c",
|
||||
"zh:dd36f9988af4e1ca5b1ca7b7bb6f658df9a220dfcda7fec7392fedfe9064f652",
|
||||
"zh:dda2688d46c7e539fe97e8fe9d3ec81fb364170e018d9c6a681364c8955d4e9d",
|
||||
"zh:e6b519afe2dea1c0434f766eb6bc9ba78cc5b6ef2c311c2ca3c65cb24744f31f",
|
||||
]
|
||||
}
|
||||
@@ -1,17 +0,0 @@
|
||||
{
|
||||
"name": "oci-hosted",
|
||||
"online": true,
|
||||
"storage": {
|
||||
"blobStoreName": "default",
|
||||
"strictContentTypeValidation": true,
|
||||
"writePolicy": "ALLOW",
|
||||
"latestPolicy": false
|
||||
},
|
||||
"oci": {
|
||||
"v1Enabled": false,
|
||||
"forceBasicAuth": false,
|
||||
"pathEnabled": true
|
||||
},
|
||||
"component": { "proprietaryComponents": false },
|
||||
"cosign": { "enforcement": "NONE" }
|
||||
}
|
||||
@@ -1,27 +0,0 @@
|
||||
{
|
||||
"name": "oci-proxy",
|
||||
"online": true,
|
||||
"storage": {
|
||||
"blobStoreName": "default",
|
||||
"strictContentTypeValidation": true
|
||||
},
|
||||
"oci": {
|
||||
"v1Enabled": false,
|
||||
"forceBasicAuth": false,
|
||||
"pathEnabled": true
|
||||
},
|
||||
"ociProxy": {
|
||||
"indexType": "HUB",
|
||||
"cacheForeignLayers": false,
|
||||
"foreignLayerUrlWhitelist": []
|
||||
},
|
||||
"proxy": {
|
||||
"remoteUrl": "https://registry-1.docker.io",
|
||||
"contentMaxAge": 1440,
|
||||
"metadataMaxAge": 60,
|
||||
"preserveEncodedCharacters": false
|
||||
},
|
||||
"negativeCache": { "enabled": true, "timeToLive": 60 },
|
||||
"httpClient": { "blocked": false, "autoBlock": true },
|
||||
"cosign": { "enforcement": "NONE" }
|
||||
}
|
||||
@@ -1,15 +0,0 @@
|
||||
{
|
||||
"name": "oci-public",
|
||||
"online": true,
|
||||
"storage": {
|
||||
"blobStoreName": "default",
|
||||
"strictContentTypeValidation": true
|
||||
},
|
||||
"group": { "memberNames": ["oci-proxy"] },
|
||||
"oci": {
|
||||
"v1Enabled": false,
|
||||
"forceBasicAuth": false,
|
||||
"pathEnabled": true
|
||||
},
|
||||
"cosign": { "enforcement": "NONE" }
|
||||
}
|
||||
@@ -1,80 +0,0 @@
|
||||
#!/usr/bin/env bash
|
||||
set -euo pipefail
|
||||
|
||||
mode="${1:---check}"
|
||||
case "$mode" in
|
||||
--check | --apply) ;;
|
||||
*) echo "usage: $0 [--check|--apply]" >&2; exit 2 ;;
|
||||
esac
|
||||
|
||||
: "${TF_VAR_nexus_url:?set TF_VAR_nexus_url}"
|
||||
: "${TF_VAR_nexus_username:?set TF_VAR_nexus_username}"
|
||||
: "${TF_VAR_nexus_password:?set TF_VAR_nexus_password}"
|
||||
|
||||
script_dir="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")" && pwd)"
|
||||
auth_file="$(mktemp /tmp/nexus-oci-auth.XXXXXX)"
|
||||
trap 'rm -f -- "$auth_file"' EXIT
|
||||
chmod 0600 "$auth_file"
|
||||
printf 'machine %s\nlogin %s\npassword %s\n' \
|
||||
"${TF_VAR_nexus_url#*://}" "$TF_VAR_nexus_username" \
|
||||
"$TF_VAR_nexus_password" >"$auth_file"
|
||||
|
||||
drift=0
|
||||
for entry in \
|
||||
"hosted:$script_dir/oci/oci-hosted.json" \
|
||||
"proxy:$script_dir/oci/oci-proxy.json" \
|
||||
"group:$script_dir/oci/oci-public.json"; do
|
||||
repository_type="${entry%%:*}"
|
||||
desired_file="${entry#*:}"
|
||||
repository_name="$(jq -er '.name' "$desired_file")"
|
||||
endpoint="$TF_VAR_nexus_url/service/rest/v1/repositories/oci/$repository_type"
|
||||
current_file="$(mktemp /tmp/nexus-oci-current.XXXXXX)"
|
||||
|
||||
status="$(curl --silent --show-error --netrc-file "$auth_file" \
|
||||
--output "$current_file" --write-out '%{http_code}' \
|
||||
"$endpoint/$repository_name")"
|
||||
|
||||
if [[ "$status" == 404 ]]; then
|
||||
drift=1
|
||||
if [[ "$mode" == --apply ]]; then
|
||||
curl --fail --silent --show-error --netrc-file "$auth_file" \
|
||||
--header 'Content-Type: application/json' \
|
||||
--data-binary "@$desired_file" "$endpoint"
|
||||
echo "created $repository_name"
|
||||
else
|
||||
echo "missing $repository_name" >&2
|
||||
fi
|
||||
elif [[ "$status" == 200 ]]; then
|
||||
if jq -e --slurpfile desired "$desired_file" '
|
||||
def subset($actual; $wanted):
|
||||
if ($wanted | type) == "object" then
|
||||
all($wanted | keys[];
|
||||
($actual[.] != null) and subset($actual[.]; $wanted[.]))
|
||||
else
|
||||
$actual == $wanted
|
||||
end;
|
||||
subset(.; $desired[0])
|
||||
' "$current_file" >/dev/null; then
|
||||
echo "in sync $repository_name"
|
||||
else
|
||||
drift=1
|
||||
if [[ "$mode" == --apply ]]; then
|
||||
curl --fail --silent --show-error --netrc-file "$auth_file" \
|
||||
--request PUT --header 'Content-Type: application/json' \
|
||||
--data-binary "@$desired_file" "$endpoint/$repository_name"
|
||||
echo "updated $repository_name"
|
||||
else
|
||||
echo "drifted $repository_name" >&2
|
||||
fi
|
||||
fi
|
||||
else
|
||||
cat "$current_file" >&2
|
||||
echo "unexpected HTTP $status for $repository_name" >&2
|
||||
exit 1
|
||||
fi
|
||||
rm -f -- "$current_file"
|
||||
done
|
||||
|
||||
if [[ "$mode" == --check && "$drift" -ne 0 ]]; then
|
||||
exit 1
|
||||
fi
|
||||
@@ -1,64 +0,0 @@
|
||||
locals {
|
||||
proxy_http_client = {
|
||||
auto_block = true
|
||||
blocked = false
|
||||
}
|
||||
|
||||
proxy_negative_cache = {
|
||||
enabled = true
|
||||
time_to_live = 60
|
||||
}
|
||||
|
||||
repository_storage = {
|
||||
blob_store_name = "default"
|
||||
strict_content_type_validation = true
|
||||
}
|
||||
}
|
||||
|
||||
resource "sonatyperepo_repository_ansiblegalaxy_proxy" "galaxy" {
|
||||
name = "ansible-galaxy-proxy"
|
||||
online = true
|
||||
|
||||
http_client = local.proxy_http_client
|
||||
negative_cache = local.proxy_negative_cache
|
||||
proxy = {
|
||||
remote_url = "https://galaxy.ansible.com"
|
||||
content_max_age = 1440
|
||||
metadata_max_age = 60
|
||||
}
|
||||
storage = local.repository_storage
|
||||
}
|
||||
|
||||
resource "sonatyperepo_repository_ansiblegalaxy_group" "public" {
|
||||
name = "ansible-public"
|
||||
online = true
|
||||
|
||||
group = {
|
||||
member_names = [sonatyperepo_repository_ansiblegalaxy_proxy.galaxy.name]
|
||||
}
|
||||
storage = local.repository_storage
|
||||
}
|
||||
|
||||
resource "sonatyperepo_repository_go_proxy" "public" {
|
||||
name = "go-proxy"
|
||||
online = true
|
||||
|
||||
http_client = local.proxy_http_client
|
||||
negative_cache = local.proxy_negative_cache
|
||||
proxy = {
|
||||
remote_url = "https://proxy.golang.org"
|
||||
content_max_age = 1440
|
||||
metadata_max_age = 60
|
||||
}
|
||||
storage = local.repository_storage
|
||||
}
|
||||
|
||||
resource "sonatyperepo_repository_go_group" "public" {
|
||||
name = "go-public"
|
||||
online = true
|
||||
|
||||
group = {
|
||||
member_names = [sonatyperepo_repository_go_proxy.public.name]
|
||||
}
|
||||
storage = local.repository_storage
|
||||
}
|
||||
@@ -1,75 +0,0 @@
|
||||
resource "sonatyperepo_privilege_repository_view" "anonymous_ansible" {
|
||||
name = "ci-anonymous-ansible-read"
|
||||
description = "Anonymous read access to the Ansible Galaxy group"
|
||||
actions = ["BROWSE", "READ"]
|
||||
format = "ansiblegalaxy"
|
||||
repository = sonatyperepo_repository_ansiblegalaxy_group.public.name
|
||||
}
|
||||
|
||||
resource "sonatyperepo_privilege_repository_view" "anonymous_ansible_proxy" {
|
||||
name = "ci-anonymous-ansible-proxy-read"
|
||||
description = "Anonymous artifact read access to the Ansible Galaxy proxy"
|
||||
actions = ["BROWSE", "READ"]
|
||||
format = "ansiblegalaxy"
|
||||
repository = sonatyperepo_repository_ansiblegalaxy_proxy.galaxy.name
|
||||
}
|
||||
|
||||
resource "sonatyperepo_privilege_repository_view" "anonymous_go" {
|
||||
name = "ci-anonymous-go-read"
|
||||
description = "Anonymous read access to the Go module group"
|
||||
actions = ["BROWSE", "READ"]
|
||||
format = "go"
|
||||
repository = sonatyperepo_repository_go_group.public.name
|
||||
}
|
||||
|
||||
resource "sonatyperepo_privilege_repository_view" "anonymous_oci_public" {
|
||||
name = "ci-anonymous-oci-public-read"
|
||||
description = "Anonymous read access to the public OCI group"
|
||||
actions = ["BROWSE", "READ"]
|
||||
format = "oci"
|
||||
repository = "oci-public"
|
||||
}
|
||||
|
||||
resource "sonatyperepo_privilege_repository_view" "anonymous_oci_proxy" {
|
||||
name = "ci-anonymous-oci-proxy-read"
|
||||
description = "Anonymous read access to the OCI proxy member"
|
||||
actions = ["BROWSE", "READ"]
|
||||
format = "oci"
|
||||
repository = "oci-proxy"
|
||||
}
|
||||
|
||||
resource "sonatyperepo_role" "anonymous_ci" {
|
||||
id = "ci-anonymous-read"
|
||||
name = "CI anonymous read"
|
||||
description = "Read-only access to public CI dependency proxy groups"
|
||||
privileges = [
|
||||
sonatyperepo_privilege_repository_view.anonymous_ansible.name,
|
||||
sonatyperepo_privilege_repository_view.anonymous_ansible_proxy.name,
|
||||
sonatyperepo_privilege_repository_view.anonymous_go.name,
|
||||
sonatyperepo_privilege_repository_view.anonymous_oci_public.name,
|
||||
sonatyperepo_privilege_repository_view.anonymous_oci_proxy.name,
|
||||
]
|
||||
roles = []
|
||||
}
|
||||
|
||||
resource "sonatyperepo_user" "anonymous" {
|
||||
user_id = "anonymous"
|
||||
first_name = "Anonymous"
|
||||
last_name = "User"
|
||||
email_address = "[email protected]"
|
||||
status = "active"
|
||||
roles = [sonatyperepo_role.anonymous_ci.id]
|
||||
}
|
||||
|
||||
resource "sonatyperepo_system_anonymous_access" "ci" {
|
||||
enabled = true
|
||||
user_id = sonatyperepo_user.anonymous.user_id
|
||||
realm_name = "NexusAuthorizingRealm"
|
||||
}
|
||||
|
||||
resource "sonatyperepo_security_realms" "active" {
|
||||
active = [
|
||||
"NexusAuthenticatingRealm",
|
||||
"OciBearerToken",
|
||||
]
|
||||
}
|
||||
@@ -1,33 +0,0 @@
|
||||
terraform {
|
||||
required_version = ">= 1.11.0"
|
||||
|
||||
required_providers {
|
||||
sonatyperepo = {
|
||||
source = "sonatype-nexus-community/sonatyperepo"
|
||||
version = "1.17.0"
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
provider "sonatyperepo" {
|
||||
url = var.nexus_url
|
||||
username = var.nexus_username
|
||||
password = var.nexus_password
|
||||
}
|
||||
|
||||
variable "nexus_url" {
|
||||
description = "Nexus Repository base URL"
|
||||
type = string
|
||||
}
|
||||
|
||||
variable "nexus_username" {
|
||||
description = "Nexus Terraform management username"
|
||||
type = string
|
||||
sensitive = true
|
||||
}
|
||||
|
||||
variable "nexus_password" {
|
||||
description = "Nexus Terraform management password"
|
||||
type = string
|
||||
sensitive = true
|
||||
}
|
||||
@@ -18,4 +18,4 @@ networks:
|
||||
default:
|
||||
ipam:
|
||||
config:
|
||||
- subnet: 172.28.1.0/24
|
||||
- subnet: 172.28.1.0/24
|
||||
|
||||
+4
-28
@@ -70,11 +70,11 @@ configFiles:
|
||||
},
|
||||
"accessControl": {
|
||||
"repositories": {
|
||||
"panxiao81/backstage": {
|
||||
"panxiao81/gitea-dynamic-runner-controller": {
|
||||
"policies": [
|
||||
{
|
||||
"users": [
|
||||
"spiffe://ddupan.top/ci/panxiao81/backstage/image",
|
||||
"spiffe://ddupan.top/ci/panxiao81/gitea-dynamic-runner/publish-images",
|
||||
"spiffe://ddupan.top/dev/panxiao81"
|
||||
],
|
||||
"actions": [
|
||||
@@ -88,42 +88,18 @@ configFiles:
|
||||
"read"
|
||||
]
|
||||
},
|
||||
"panxiao81/gitea-dynamic-runner-controller": {
|
||||
"policies": [
|
||||
{
|
||||
"users": [
|
||||
"spiffe://ddupan.top/ci/panxiao81/gitea-dynamic-runner/publish-images"
|
||||
],
|
||||
"actions": [
|
||||
"read",
|
||||
"create",
|
||||
"update"
|
||||
]
|
||||
},
|
||||
{
|
||||
"users": ["spiffe://ddupan.top/dev/panxiao81"],
|
||||
"actions": ["read", "create", "update", "delete"]
|
||||
}
|
||||
],
|
||||
"defaultPolicy": [
|
||||
"read"
|
||||
]
|
||||
},
|
||||
"panxiao81/gitea-dynamic-runner-runner": {
|
||||
"policies": [
|
||||
{
|
||||
"users": [
|
||||
"spiffe://ddupan.top/ci/panxiao81/gitea-dynamic-runner/publish-images"
|
||||
"spiffe://ddupan.top/ci/panxiao81/gitea-dynamic-runner/publish-images",
|
||||
"spiffe://ddupan.top/dev/panxiao81"
|
||||
],
|
||||
"actions": [
|
||||
"read",
|
||||
"create",
|
||||
"update"
|
||||
]
|
||||
},
|
||||
{
|
||||
"users": ["spiffe://ddupan.top/dev/panxiao81"],
|
||||
"actions": ["read", "create", "update", "delete"]
|
||||
}
|
||||
],
|
||||
"defaultPolicy": [
|
||||
|
||||
@@ -50,9 +50,6 @@ sudo k3s kubectl -n flux-system get gitrepositories,kustomizations
|
||||
- VictoriaMetrics Operator 已固定现有 chart `0.66.2` 并完成分阶段 Flux HelmRelease
|
||||
接管;Metrics、Logs、Traces 与 Grafana 也已统一完成 Flux 接管;
|
||||
- External Secrets Operator 已固定 chart `2.8.0` 并完成分阶段接管;
|
||||
- SPIRE 已按 hardened chart 内部 fork `0.30.2-ddupan.1`(基于上游 `0.30.2`,SPIRE
|
||||
`1.15.3`)声明,使用共享
|
||||
- SPIRE 已按官方 hardened chart `0.30.2`(SPIRE `1.15.3`)声明,使用共享
|
||||
PostgreSQL 与独立 signing-key PVC;首次上线和 OpenBao JWT-SVID PoC 尚待合并后验证;
|
||||
- Nexus Repository CE POC 已加入 GitOps 声明,计划验证 Ansible Galaxy、Go Modules 与
|
||||
OCI/BuildKit 缓存;尚未部署或完成现场验收,现有 zot 保持不变;
|
||||
- root Kustomization 与所有 brownfield 子 Kustomization 继续保持 `prune: false`。
|
||||
|
||||
@@ -6,12 +6,11 @@ metadata:
|
||||
spec:
|
||||
dependsOn:
|
||||
- name: external-secrets
|
||||
- name: nats
|
||||
- name: spire
|
||||
interval: 10m
|
||||
path: ./platform/dynamic-runner
|
||||
# The runner backends are replaceable. Prune is required when a retired
|
||||
# worker is removed from the component; otherwise it keeps consuming work.
|
||||
prune: true
|
||||
prune: false
|
||||
sourceRef:
|
||||
kind: GitRepository
|
||||
name: flux-system
|
||||
|
||||
+5
-6
@@ -1,15 +1,14 @@
|
||||
---
|
||||
apiVersion: kustomize.toolkit.fluxcd.io/v1
|
||||
kind: Kustomization
|
||||
metadata:
|
||||
name: spire-bootstrap
|
||||
name: gitea-actions
|
||||
namespace: flux-system
|
||||
spec:
|
||||
interval: 10m
|
||||
path: ./platform/sandbox-spire/bootstrap
|
||||
prune: true
|
||||
path: ./platform/gitea-runner
|
||||
prune: false
|
||||
sourceRef:
|
||||
kind: GitRepository
|
||||
name: flux-system
|
||||
timeout: 10m
|
||||
wait: true
|
||||
timeout: 3m
|
||||
wait: false
|
||||
@@ -1,17 +0,0 @@
|
||||
apiVersion: kustomize.toolkit.fluxcd.io/v1
|
||||
kind: Kustomization
|
||||
metadata:
|
||||
name: hydra
|
||||
namespace: flux-system
|
||||
spec:
|
||||
dependsOn:
|
||||
- name: envoy-gateway
|
||||
- name: external-secrets
|
||||
interval: 10m
|
||||
path: ./apps/hydra
|
||||
prune: false
|
||||
sourceRef:
|
||||
kind: GitRepository
|
||||
name: flux-system
|
||||
timeout: 5m
|
||||
wait: true
|
||||
@@ -1,22 +0,0 @@
|
||||
apiVersion: kustomize.toolkit.fluxcd.io/v1
|
||||
kind: Kustomization
|
||||
metadata:
|
||||
name: nexus
|
||||
namespace: flux-system
|
||||
spec:
|
||||
dependsOn:
|
||||
- name: envoy-gateway
|
||||
- name: openebs
|
||||
healthChecks:
|
||||
- apiVersion: apps/v1
|
||||
kind: Deployment
|
||||
name: nexus
|
||||
namespace: nexus
|
||||
interval: 10m
|
||||
path: ./apps/nexus
|
||||
prune: false
|
||||
sourceRef:
|
||||
kind: GitRepository
|
||||
name: flux-system
|
||||
timeout: 15m
|
||||
wait: true
|
||||
@@ -7,6 +7,7 @@ resources:
|
||||
- apps/envoy-gateway.yaml
|
||||
- apps/external-secrets.yaml
|
||||
- apps/gitea.yaml
|
||||
- apps/gitea-actions.yaml
|
||||
- apps/http-echo.yaml
|
||||
- apps/openebs.yaml
|
||||
- apps/nats.yaml
|
||||
@@ -14,5 +15,3 @@ resources:
|
||||
- apps/spire.yaml
|
||||
- apps/observability.yaml
|
||||
- apps/zot.yaml
|
||||
- apps/nexus.yaml
|
||||
- apps/hydra.yaml
|
||||
|
||||
@@ -1,72 +0,0 @@
|
||||
# Sandbox 集群
|
||||
|
||||
这里是 OpenSandbox、CI 和 AI Agent workload 所在双节点 k3s 集群的 Flux
|
||||
reconciliation 入口。LXC、PostgreSQL、K3s、固定版本的 Flux controllers 与 root
|
||||
sync 由 `infrastructure/sandbox-cluster/` 中的 Ansible 管理;本目录只组合集群内
|
||||
workload。
|
||||
|
||||
Flux 通过 `https://git.ddupan.top/panxiao81/homelab-infra.git` 读取公开仓库。
|
||||
Ansible 将 homelab CA 注入 `GitRepository/flux-system` 引用的同名 Secret,不使用
|
||||
长期 Git 凭据。root Kustomization 从 `./clusters/sandbox` 开始 reconciliation,
|
||||
初始保持 `prune: false`。
|
||||
|
||||
Root bootstrap 已完成。后续按依赖顺序分别引入:
|
||||
|
||||
1. 监控 CRD、kube-state-metrics 以及 kubelet/cAdvisor 抓取配置;
|
||||
2. SPIRE Agent、SPIFFE CSI Driver 与 workload registration;
|
||||
3. Kata Containers、`block-plain` RuntimeClass;
|
||||
4. 独立 External Secrets Operator 与 sandbox 专用 OpenBao auth backend;
|
||||
5. OpenSandbox controller/server;CI Pool 与 runner 调度器随后独立接入。
|
||||
|
||||
每一阶段单独合并并等待对应 Flux Kustomization Ready,不在 bootstrap 时一次性部署。
|
||||
第一阶段监控拆为 `monitoring-operator` 与依赖它的 `monitoring`,防止 VM CR 在
|
||||
VictoriaMetrics Operator CRD Ready 前进入 reconciliation。
|
||||
|
||||
SPIRE 阶段先由 `spire-bootstrap` 安装 CRD,并声明按上游 k8s_psat Server plugin
|
||||
要求收窄的 reviewer:它可以调用 TokenReview,并只读查询用于证明的 Pod 与 Node。
|
||||
Agent ServiceAccount 留给后续 HelmRelease 创建,避免两个声明方争夺同一资源。随后运行
|
||||
`infrastructure/sandbox-cluster/ansible/spire-bootstrap.yml`:playbook 从 sandbox
|
||||
读取 reviewer token,在内存中组成受限 kubeconfig,再通过 stdin reconcile 到 central
|
||||
集群的 `spire-server/spire-external-kubeconfigs` Secret。凭据不写入仓库、日志或控制机
|
||||
文件;该 Secret 准备完成后,才能启用 central external PSAT/controller-manager 和
|
||||
sandbox Agent/CSI。
|
||||
|
||||
External controller-manager 使用独立的 `spire-controller-manager` ServiceAccount;其
|
||||
RBAC 与上游 controller-manager 所需权限一致,用于读取 workload selectors、维护
|
||||
SPIFFE CR status/finalizer 和 leader election。它不复用只允许 TokenReview 的 Server
|
||||
reviewer。Ansible 将两份 kubeconfig 写入同一个 central Secret 的不同 key,便于 central
|
||||
chart 分别绑定 `sandbox` 与 `sandbox-controller`。
|
||||
|
||||
Central SPIRE Server 通过内网 `spire-server.ad.ddupan.top:8081` 接收 sandbox Agent
|
||||
attestation。Server 使用 external bundle publisher 持续维护 sandbox
|
||||
`spire-system/spire-bundle`,Agent 不固定或复制 trust bundle。Sandbox HelmRelease
|
||||
显式关闭 Server 与 OIDC Provider,只部署 Agent DaemonSet 和 SPIFFE CSI Driver;因此
|
||||
不会产生第二个 trust root。
|
||||
|
||||
`spire-smoke` namespace、ServiceAccount 和 `sandbox-spire-smoke` ClusterSPIFFEID 只用于
|
||||
普通 Pod 的 CSI 回归夹具,稳定身份为 `spiffe://ddupan.top/sandbox/smoke`。Kata guest
|
||||
不能复用 node Agent 暴露的 Unix socket;virtio-fs 只能呈现 socket 路径,不能把连接
|
||||
跨过 VM 边界。Kata workload 必须使用 guest 内 Agent,具体约束见
|
||||
`platform/sandbox-kata/README.md`。测试 Pod 临时创建并在验收后删除,普通 Pod 的身份
|
||||
声明保留。
|
||||
|
||||
Kata 阶段使用官方 4.1.0 `kata-deploy` chart 的短生命周期 `job` 模式,逐节点安装并
|
||||
重启 K3s。只启用 `kata-clh-runtime-rs`,不创建默认 `kata` 别名;该 handler 的
|
||||
`emptyDir` 固定使用 `block-plain`,为 Docker/BuildKit overlay2 与 kind 提供 guest
|
||||
内块设备文件系统。详细限制与上线验收见 `platform/sandbox-kata/README.md`。
|
||||
|
||||
Sandbox 的 ESO 通过独立 `auth/kubernetes-sandbox` 向 OpenBao 证明 ServiceAccount
|
||||
身份,只能读取共享的 `kv/k8s/opensandbox-api`。它不保存 reviewer JWT 或长期 Bao token;相关
|
||||
Terraform 与 Flux 边界见 `platform/sandbox-external-secrets/README.md`。
|
||||
|
||||
OpenSandbox 阶段固定官方源码 commit 与 umbrella chart `0.2.2`,只部署 controller、
|
||||
ClusterIP server 和 CRD。API key 由 ESO 从 OpenBao 投影,明文不进入 Git。
|
||||
|
||||
## 监控边界
|
||||
|
||||
这里只管理 sandbox LXC 内的 Kubernetes 监控,不负责 PVE 宿主监控。LXC 与宿主共享
|
||||
内核,即使 lxcfs 虚拟化了内存和 uptime,容器内 `/proc/stat` 仍是宿主 CPU 视图;
|
||||
在 LXC 内运行 node_exporter 会生成混合语义并重复采集宿主指标,因此禁止部署。
|
||||
|
||||
Sandbox 节点与 workload 指标来自 kubelet/cAdvisor 和 kube-state-metrics;K3s 或 LXC
|
||||
特有但上述接口未覆盖的指标,应使用目标明确的 collector,不以 node_exporter 补齐。
|
||||
@@ -1,18 +0,0 @@
|
||||
---
|
||||
apiVersion: kustomize.toolkit.fluxcd.io/v1
|
||||
kind: Kustomization
|
||||
metadata:
|
||||
name: ci-runners
|
||||
namespace: flux-system
|
||||
spec:
|
||||
dependsOn:
|
||||
- name: opensandbox
|
||||
- name: spire-agents
|
||||
interval: 10m
|
||||
path: ./platform/sandbox-ci-runners
|
||||
prune: true
|
||||
sourceRef:
|
||||
kind: GitRepository
|
||||
name: flux-system
|
||||
timeout: 20m
|
||||
wait: true
|
||||
@@ -1,17 +0,0 @@
|
||||
---
|
||||
apiVersion: kustomize.toolkit.fluxcd.io/v1
|
||||
kind: Kustomization
|
||||
metadata:
|
||||
name: external-secrets-operator
|
||||
namespace: flux-system
|
||||
spec:
|
||||
dependsOn:
|
||||
- name: monitoring-operator
|
||||
interval: 10m
|
||||
path: ./platform/sandbox-external-secrets/operator
|
||||
prune: true
|
||||
sourceRef:
|
||||
kind: GitRepository
|
||||
name: flux-system
|
||||
timeout: 10m
|
||||
wait: true
|
||||
@@ -1,17 +0,0 @@
|
||||
---
|
||||
apiVersion: kustomize.toolkit.fluxcd.io/v1
|
||||
kind: Kustomization
|
||||
metadata:
|
||||
name: external-secrets
|
||||
namespace: flux-system
|
||||
spec:
|
||||
dependsOn:
|
||||
- name: external-secrets-operator
|
||||
interval: 10m
|
||||
path: ./platform/sandbox-external-secrets/config
|
||||
prune: true
|
||||
sourceRef:
|
||||
kind: GitRepository
|
||||
name: flux-system
|
||||
timeout: 10m
|
||||
wait: true
|
||||
@@ -1,18 +0,0 @@
|
||||
---
|
||||
apiVersion: kustomize.toolkit.fluxcd.io/v1
|
||||
kind: Kustomization
|
||||
metadata:
|
||||
name: kata
|
||||
namespace: flux-system
|
||||
spec:
|
||||
dependsOn:
|
||||
- name: monitoring-operator
|
||||
- name: spire-agents
|
||||
interval: 10m
|
||||
path: ./platform/sandbox-kata
|
||||
prune: true
|
||||
sourceRef:
|
||||
kind: GitRepository
|
||||
name: flux-system
|
||||
timeout: 35m
|
||||
wait: true
|
||||
@@ -1,15 +0,0 @@
|
||||
---
|
||||
apiVersion: kustomize.toolkit.fluxcd.io/v1
|
||||
kind: Kustomization
|
||||
metadata:
|
||||
name: monitoring-operator
|
||||
namespace: flux-system
|
||||
spec:
|
||||
interval: 10m
|
||||
path: ./platform/sandbox-monitoring/operator
|
||||
prune: true
|
||||
sourceRef:
|
||||
kind: GitRepository
|
||||
name: flux-system
|
||||
timeout: 10m
|
||||
wait: true
|
||||
@@ -1,17 +0,0 @@
|
||||
---
|
||||
apiVersion: kustomize.toolkit.fluxcd.io/v1
|
||||
kind: Kustomization
|
||||
metadata:
|
||||
name: monitoring
|
||||
namespace: flux-system
|
||||
spec:
|
||||
dependsOn:
|
||||
- name: monitoring-operator
|
||||
interval: 10m
|
||||
path: ./platform/sandbox-monitoring/workloads
|
||||
prune: true
|
||||
sourceRef:
|
||||
kind: GitRepository
|
||||
name: flux-system
|
||||
timeout: 10m
|
||||
wait: true
|
||||
@@ -1,19 +0,0 @@
|
||||
---
|
||||
apiVersion: kustomize.toolkit.fluxcd.io/v1
|
||||
kind: Kustomization
|
||||
metadata:
|
||||
name: opensandbox
|
||||
namespace: flux-system
|
||||
spec:
|
||||
dependsOn:
|
||||
- name: external-secrets
|
||||
- name: kata
|
||||
- name: monitoring-operator
|
||||
interval: 10m
|
||||
path: ./platform/sandbox-opensandbox
|
||||
prune: true
|
||||
sourceRef:
|
||||
kind: GitRepository
|
||||
name: flux-system
|
||||
timeout: 15m
|
||||
wait: true
|
||||
@@ -1,18 +0,0 @@
|
||||
---
|
||||
apiVersion: kustomize.toolkit.fluxcd.io/v1
|
||||
kind: Kustomization
|
||||
metadata:
|
||||
name: spire-agents
|
||||
namespace: flux-system
|
||||
spec:
|
||||
dependsOn:
|
||||
- name: spire-bootstrap
|
||||
- name: monitoring-operator
|
||||
interval: 10m
|
||||
path: ./platform/sandbox-spire/agents
|
||||
prune: true
|
||||
sourceRef:
|
||||
kind: GitRepository
|
||||
name: flux-system
|
||||
timeout: 15m
|
||||
wait: true
|
||||
@@ -1,13 +0,0 @@
|
||||
---
|
||||
apiVersion: kustomize.config.k8s.io/v1beta1
|
||||
kind: Kustomization
|
||||
resources:
|
||||
- apps/monitoring-operator.yaml
|
||||
- apps/monitoring.yaml
|
||||
- apps/spire-bootstrap.yaml
|
||||
- apps/spire-agents.yaml
|
||||
- apps/kata.yaml
|
||||
- apps/external-secrets-operator.yaml
|
||||
- apps/external-secrets.yaml
|
||||
- apps/opensandbox.yaml
|
||||
- apps/ci-runners.yaml
|
||||
@@ -15,15 +15,10 @@ homelab_dns:
|
||||
- { zone: ad.ddupan.top, name: sandbox-k8s, type: A, values: [10.60.0.13] }
|
||||
- { zone: ad.ddupan.top, name: retrolab, type: A, values: [10.60.0.10] }
|
||||
- { zone: ad.ddupan.top, name: grafana, type: A, values: [192.168.10.127] }
|
||||
- { zone: ad.ddupan.top, name: metrics-write, type: A, values: [192.168.10.127] }
|
||||
- { zone: ad.ddupan.top, name: netbox, type: A, values: [192.168.10.127] }
|
||||
- { zone: ad.ddupan.top, name: nats, type: A, values: [192.168.10.127] }
|
||||
- { zone: ad.ddupan.top, name: nexus, type: A, values: [192.168.10.127] }
|
||||
- { zone: ad.ddupan.top, name: s3, type: A, values: [192.168.10.127] }
|
||||
- { zone: ad.ddupan.top, name: hydra, type: A, values: [192.168.10.127] }
|
||||
- { zone: ad.ddupan.top, name: hydra-login, type: A, values: [192.168.10.127] }
|
||||
- { zone: ad.ddupan.top, name: spire-oidc, type: A, values: [192.168.10.127] }
|
||||
- { zone: ad.ddupan.top, name: spire-server, type: A, values: [192.168.10.127] }
|
||||
- { zone: ad.ddupan.top, name: zot, type: A, values: [192.168.10.127] }
|
||||
- { zone: ad.ddupan.top, name: zot-push, type: A, values: [192.168.10.127] }
|
||||
|
||||
@@ -65,14 +60,14 @@ homelab_dns:
|
||||
- name: oci-arm.ddupan.top
|
||||
type: A
|
||||
values:
|
||||
- 129.225.138.179
|
||||
- 129.225.138.179
|
||||
proxied: false
|
||||
ttl: 300
|
||||
terraform: { managed: false }
|
||||
- name: oci-amd.ddupan.top
|
||||
type: A
|
||||
values:
|
||||
- 129.225.176.134
|
||||
- 129.225.176.134
|
||||
proxied: false
|
||||
ttl: 300
|
||||
terraform: { managed: false }
|
||||
|
||||
@@ -51,7 +51,7 @@ few things Terraform must not own.**
|
||||
| secrets engine mounts (`kv`, `pki`, `ssh-client-signer`) | the daemon, Raft, TLS files, systemd |
|
||||
| PKI role, issuing/CRL URLs, cluster paths, **ACME** | `bao operator init` / unseal (manual, PGP-wrapped) |
|
||||
| SSH signing role (`ai-agent`) | **PKI root CA + SSH CA signing key** |
|
||||
| OIDC/Kubernetes auth mount、config 与 role | OIDC **client secret** (`auth/oidc/config`) |
|
||||
| OIDC auth *mount* and *role* | OIDC **client secret** (`auth/oidc/config`) |
|
||||
| all policies | snapshot token + script + systemd timer |
|
||||
| | host-level CA trust distribution (`openbao_ssh_ca_trust`) |
|
||||
|
||||
@@ -92,13 +92,6 @@ then `VAULT_ADDR`/`VAULT_TOKEN`), mirroring how `smtp-relay/terraform` uses
|
||||
native `openbao/openbao` provider is published only to the OpenTofu registry and
|
||||
cannot be resolved by the HashiCorp `terraform` CLI.
|
||||
|
||||
Sandbox 集群使用独立的 `auth/kubernetes-sandbox`。其 API 地址、公开 Kubernetes CA、
|
||||
ESO role 与只读 `kv/k8s/opensandbox-api` policy 全部由 Terraform 管理;CA 位于
|
||||
`terraform/certs/sandbox-kubernetes-ca.crt`。集群重建并轮换 CA 后,先更新该文件并
|
||||
apply,再让 Flux 恢复 ESO reconciliation。该 backend 不保存 reviewer JWT,而是使用
|
||||
ESO 的短期登录 JWT 执行 TokenReview。该组资源已于 2026-09-18 apply,随后复验 plan
|
||||
为 zero-diff。
|
||||
|
||||
## DNS
|
||||
|
||||
`bao.ad.ddupan.top` is an **internal-only** name — not a public Cloudflare record and
|
||||
|
||||
@@ -59,33 +59,3 @@ resource "vault_kubernetes_auth_backend_role" "external_secrets" {
|
||||
# in a long TTL and every extra hour is a longer-lived credential in memory.
|
||||
token_ttl = 3600
|
||||
}
|
||||
|
||||
# A Kubernetes auth mount can validate identities from only the API server it is
|
||||
# configured against. The sandbox cluster therefore cannot reuse auth/kubernetes,
|
||||
# whose TokenReview endpoint belongs to homelab.
|
||||
resource "vault_auth_backend" "sandbox_kubernetes" {
|
||||
type = "kubernetes"
|
||||
path = "kubernetes-sandbox"
|
||||
}
|
||||
|
||||
resource "vault_kubernetes_auth_backend_config" "sandbox" {
|
||||
backend = vault_auth_backend.sandbox_kubernetes.path
|
||||
kubernetes_host = "https://10.60.0.13:6443"
|
||||
kubernetes_ca_cert = file("${path.module}/certs/sandbox-kubernetes-ca.crt")
|
||||
disable_local_ca_jwt = true
|
||||
|
||||
# Deliberately omit token_reviewer_jwt. OpenBao uses the login JWT for
|
||||
# TokenReview; the sandbox external-secrets ServiceAccount is bound only to
|
||||
# system:auth-delegator and all issued JWTs remain short-lived.
|
||||
}
|
||||
|
||||
resource "vault_kubernetes_auth_backend_role" "sandbox_external_secrets" {
|
||||
backend = vault_auth_backend.sandbox_kubernetes.path
|
||||
role_name = "external-secrets"
|
||||
|
||||
bound_service_account_names = ["external-secrets"]
|
||||
bound_service_account_namespaces = ["external-secrets"]
|
||||
|
||||
token_policies = [vault_policy.sandbox_external_secrets.name]
|
||||
token_ttl = 3600
|
||||
}
|
||||
|
||||
@@ -1,10 +0,0 @@
|
||||
-----BEGIN CERTIFICATE-----
|
||||
MIIBdjCCAR2gAwIBAgIBADAKBggqhkjOPQQDAjAjMSEwHwYDVQQDDBhrM3Mtc2Vy
|
||||
dmVyLWNhQDE3ODk2NTI4NTcwHhcNMjYwOTE3MTI0NzM3WhcNMzYwOTE0MTI0NzM3
|
||||
WjAjMSEwHwYDVQQDDBhrM3Mtc2VydmVyLWNhQDE3ODk2NTI4NTcwWTATBgcqhkjO
|
||||
PQIBBggqhkjOPQMBBwNCAAR4SbqzTXZnlZdUPz7viN6+dYbB1Maw44Qiepn9r5XG
|
||||
sOzkYkN8t1aG3Ugo8TqQ3xJaKkM89n1Rluj0vbOhiNajo0IwQDAOBgNVHQ8BAf8E
|
||||
BAMCAqQwDwYDVR0TAQH/BAUwAwEB/zAdBgNVHQ4EFgQUyLSGoKAKAJuiniuRdBLG
|
||||
XYaDQC8wCgYIKoZIzj0EAwIDRwAwRAIgFkVzyUZexk/ynnxBEOg+3foJv3WKqAei
|
||||
hTSRjO1gL0UCIFbBKR7BMrJJAgW3DJFeeBM+b+tTg93jNx55qZACbFOL
|
||||
-----END CERTIFICATE-----
|
||||
@@ -35,10 +35,3 @@ resource "vault_policy" "external_secrets" {
|
||||
name = "external-secrets"
|
||||
policy = file("${path.module}/policies/external-secrets.hcl")
|
||||
}
|
||||
|
||||
# The sandbox cluster has its own Kubernetes auth backend and a deliberately
|
||||
# narrower KV view than the homelab ESO instance.
|
||||
resource "vault_policy" "sandbox_external_secrets" {
|
||||
name = "sandbox-external-secrets"
|
||||
policy = file("${path.module}/policies/sandbox-external-secrets.hcl")
|
||||
}
|
||||
|
||||
@@ -1,9 +0,0 @@
|
||||
# Read only the shared OpenSandbox control-plane API key. The same Bao object is
|
||||
# consumed by the server in sandbox and, later, by the scheduler in homelab.
|
||||
path "kv/data/k8s/opensandbox-api" {
|
||||
capabilities = ["read"]
|
||||
}
|
||||
|
||||
path "kv/metadata/k8s/opensandbox-api" {
|
||||
capabilities = ["read"]
|
||||
}
|
||||
@@ -77,19 +77,3 @@ vyos_ospf_networks:
|
||||
vyos_postgresql_listen_address: "10.60.0.1"
|
||||
vyos_postgresql_port: 5432
|
||||
vyos_postgresql_primary_address: "10.60.0.11"
|
||||
|
||||
# Stable API endpoint for the two-node sandbox K3s control plane. HAProxy owns
|
||||
# this /32 on the existing labnet interface and performs only TCP health checks.
|
||||
vyos_sandbox_k3s_api_address: "10.60.0.13"
|
||||
vyos_sandbox_k3s_api_interface: eth1
|
||||
vyos_sandbox_k3s_api_port: 6443
|
||||
vyos_opensandbox_api_port: 8080
|
||||
vyos_opensandbox_node_port: 30080
|
||||
# Lifecycle create is synchronous and a cold Kata image pull can exceed the
|
||||
# HAProxy 50-second default. Keep this below OpenSandbox's overall timeout.
|
||||
vyos_opensandbox_api_timeout: 600
|
||||
vyos_sandbox_k3s_servers:
|
||||
- name: sandbox1
|
||||
address: "10.60.0.11"
|
||||
- name: sandbox2
|
||||
address: "10.60.0.12"
|
||||
|
||||
@@ -3,23 +3,6 @@
|
||||
# `set` lines against the running config and issues only the differences, so
|
||||
# re-runs are no-ops. Supports --check and --diff.
|
||||
|
||||
- name: Read current VyOS configuration commands
|
||||
vyos.vyos.vyos_command:
|
||||
commands:
|
||||
- show configuration commands
|
||||
register: _vyos_running_commands
|
||||
changed_when: false
|
||||
|
||||
- name: Remove obsolete HAProxy PostgreSQL protocol check
|
||||
vyos.vyos.vyos_config:
|
||||
lines:
|
||||
- delete load-balancing haproxy backend sandbox-postgresql-primary health-check
|
||||
save: true
|
||||
comment: Remove unsupported PostgreSQL protocol check
|
||||
when: >-
|
||||
"set load-balancing haproxy backend sandbox-postgresql-primary health-check 'pgsql'"
|
||||
in _vyos_running_commands.stdout[0]
|
||||
|
||||
- name: Render the desired configuration
|
||||
ansible.builtin.set_fact:
|
||||
_vyos_lines: >-
|
||||
|
||||
@@ -15,20 +15,6 @@ set interfaces ethernet {{ i.iface }} address {{ i.address }}
|
||||
set interfaces ethernet {{ i.iface }} description '{{ i.description }}'
|
||||
{% endfor %}
|
||||
|
||||
{# OpenSandbox stays on a NodePort; VyOS owns the stable routed frontend. #}
|
||||
set load-balancing haproxy service opensandbox-api listen-address {{ vyos_sandbox_k3s_api_address }}
|
||||
set load-balancing haproxy service opensandbox-api port {{ vyos_opensandbox_api_port }}
|
||||
set load-balancing haproxy service opensandbox-api mode 'tcp'
|
||||
set load-balancing haproxy service opensandbox-api backend 'opensandbox-api'
|
||||
set load-balancing haproxy service opensandbox-api timeout client {{ vyos_opensandbox_api_timeout }}
|
||||
set load-balancing haproxy backend opensandbox-api mode 'tcp'
|
||||
set load-balancing haproxy backend opensandbox-api timeout server {{ vyos_opensandbox_api_timeout }}
|
||||
{% for server in vyos_sandbox_k3s_servers %}
|
||||
set load-balancing haproxy backend opensandbox-api server {{ server.name }} address {{ server.address }}
|
||||
set load-balancing haproxy backend opensandbox-api server {{ server.name }} port {{ vyos_opensandbox_node_port }}
|
||||
set load-balancing haproxy backend opensandbox-api server {{ server.name }} check
|
||||
{% endfor %}
|
||||
|
||||
{# --- default route out; OSPF carries the rest --- #}
|
||||
set protocols static route 0.0.0.0/0 next-hop {{ vyos_lan_gateway }}
|
||||
|
||||
@@ -49,23 +35,11 @@ set load-balancing haproxy service sandbox-postgresql port {{ vyos_postgresql_po
|
||||
set load-balancing haproxy service sandbox-postgresql mode 'tcp'
|
||||
set load-balancing haproxy service sandbox-postgresql backend 'sandbox-postgresql-primary'
|
||||
set load-balancing haproxy backend sandbox-postgresql-primary mode 'tcp'
|
||||
set load-balancing haproxy backend sandbox-postgresql-primary health-check 'pgsql'
|
||||
set load-balancing haproxy backend sandbox-postgresql-primary server primary address {{ vyos_postgresql_primary_address }}
|
||||
set load-balancing haproxy backend sandbox-postgresql-primary server primary port {{ vyos_postgresql_port }}
|
||||
set load-balancing haproxy backend sandbox-postgresql-primary server primary check
|
||||
|
||||
{# --- Sandbox K3s API: stable address with both control-plane nodes --- #}
|
||||
set interfaces ethernet {{ vyos_sandbox_k3s_api_interface }} address '{{ vyos_sandbox_k3s_api_address }}/32'
|
||||
set load-balancing haproxy service sandbox-k3s-api listen-address {{ vyos_sandbox_k3s_api_address }}
|
||||
set load-balancing haproxy service sandbox-k3s-api port {{ vyos_sandbox_k3s_api_port }}
|
||||
set load-balancing haproxy service sandbox-k3s-api mode 'tcp'
|
||||
set load-balancing haproxy service sandbox-k3s-api backend 'sandbox-k3s-api'
|
||||
set load-balancing haproxy backend sandbox-k3s-api mode 'tcp'
|
||||
{% for server in vyos_sandbox_k3s_servers %}
|
||||
set load-balancing haproxy backend sandbox-k3s-api server {{ server.name }} address {{ server.address }}
|
||||
set load-balancing haproxy backend sandbox-k3s-api server {{ server.name }} port {{ vyos_sandbox_k3s_api_port }}
|
||||
set load-balancing haproxy backend sandbox-k3s-api server {{ server.name }} check
|
||||
{% endfor %}
|
||||
|
||||
{# --- management --- #}
|
||||
set service ssh port 22
|
||||
|
||||
|
||||
@@ -1,152 +0,0 @@
|
||||
# Sandbox 集群
|
||||
|
||||
该目录管理供 OpenSandbox、CI 和 AI Agent workload 使用的独立双节点 k3s 集群。
|
||||
基础设施和节点生命周期由 Ansible 管理;Kubernetes API 可用后,集群内组件由
|
||||
`clusters/sandbox/` 下的 Flux desired state 管理。
|
||||
|
||||
## 所有权边界
|
||||
|
||||
Ansible 管理以下持久状态:
|
||||
|
||||
- pve1/pve2 上的 privileged LXC、磁盘、固定网络与 KVM/vhost/TUN/kmsg 设备;
|
||||
- LXC OS 基线、内部 CA、外部 PostgreSQL 与 K3s;
|
||||
- 固定版本的 Flux controllers 和 root sync bootstrap;
|
||||
- 只读状态验证和数据库切换 runbook。
|
||||
|
||||
Flux 管理以下 Kubernetes 资源:
|
||||
|
||||
- Kata Containers 和 CI 专用的 `block-plain` RuntimeClass;
|
||||
- SPIRE Agent、SPIFFE CSI Driver 与 workload identity 声明;
|
||||
- vmagent、kube-state-metrics、kubelet/cAdvisor scrape 配置和告警;
|
||||
- sandbox External Secrets Operator、OpenSandbox controller/server;CI Pool 与 runner
|
||||
调度器由 runner 项目接入。
|
||||
|
||||
同一个对象只能有一个 owner。Ansible 不直接部署上述集群内 workload;Flux 不管理
|
||||
LXC、K3s datastore 或 K3s 本身。
|
||||
|
||||
监控范围仅覆盖 sandbox LXC 内的 Kubernetes 与 workload。不得在 LXC 内部署
|
||||
node_exporter:LXC 的 `/proc` 是 lxcfs 虚拟视图与宿主内核视图的混合,尤其
|
||||
`/proc/stat` 会重复暴露 PVE 宿主 CPU 数据。PVE 自身的 node_exporter 或其他宿主
|
||||
监控不属于本目录;sandbox 使用 kubelet/cAdvisor 和 kube-state-metrics。
|
||||
|
||||
## 声明拓扑
|
||||
|
||||
| 对象 | PVE 节点 | VMID | 地址 | 资源 |
|
||||
|---|---:|---:|---|---|
|
||||
| `sandbox1` | pve1 | 148 | `10.60.0.11/24` | 4 vCPU / 6 GiB / 2 GiB swap |
|
||||
| `sandbox2` | pve2 | 149 | `10.60.0.12/24` | 4 vCPU / 4 GiB / 2 GiB swap |
|
||||
| K3s API VIP | VyOS | — | `10.60.0.13:6443` | HAProxy TCP LB |
|
||||
|
||||
节点与 API VIP 均位于现有 PVE `labnet`(VLAN 100,`10.60.0.0/24`),通过
|
||||
VyOS `10.60.0.1` 路由;不为 sandbox 新建 VNet,也不占用 `192.168.10.0/24`
|
||||
地址。需要从 LAN 访问的服务统一经 VyOS 路由或 LB 暴露。两个 LXC 均使用
|
||||
`pve-rg-hdd` 上的 32 GiB rootfs;不为 `/var/lib/kubelet` 单独创建 volume。Kata
|
||||
`block-plain` 产生的数据随 Pod 生命周期清理,当前规模没有额外磁盘故障域的需求。
|
||||
K3s API 的 `10.60.0.13/32` 由 VyOS 现有 labnet interface 持有,HAProxy 以 TCP
|
||||
健康检查把 `6443` 分发到两个 control-plane 节点;它与 PostgreSQL 主从切换逻辑无关。
|
||||
LXC 内通过 `/etc/tmpfiles.d/kmsg.conf` 持久维护 `/dev/kmsg -> /dev/console`;否则
|
||||
kubelet 会因 LXC 不提供真实 host `/dev/kmsg` 而反复退出。
|
||||
每个 LXC 还以只读 bind mount 使用宿主的 `/lib/modules`。LXC 与 PVE 宿主共享内核,
|
||||
guest 若看不到对应版本的模块目录,K3s 无法加载 `br_netfilter` 和 `overlay`,Flannel
|
||||
也不会生成节点的 subnet 配置。
|
||||
|
||||
## PostgreSQL 写入口与切换
|
||||
|
||||
K3s 使用外部 PostgreSQL,首期采用 primary + synchronous standby。VyOS 在 labnet
|
||||
gateway `10.60.0.1:5432` 提供固定 TCP 入口,backend 只包含当前声明的 primary;
|
||||
不把普通 TCP 或 PostgreSQL 存活检查等同于“节点可写”,也不自动把流量切到 standby。
|
||||
VyOS 2025.11 的 PostgreSQL protocol check 会生成缺少必需 `user` 参数的 HAProxy
|
||||
配置,因此这里只使用基础 TCP check;真正的可写性由 `verify.yml` 通过 SQL 验证。
|
||||
|
||||
数据库切换必须由 Ansible runbook 受控完成:先隔离旧 primary,再提升 standby,最后
|
||||
更新 VyOS backend。首期不部署 PgBouncer、Patroni、repmgr 或额外 DCS,也不宣称两节点
|
||||
PostgreSQL 能够自动 HA。以后具备第三个仲裁节点时再重新评估自动 failover。
|
||||
|
||||
数据库凭据位于 Bao `kv/infra/sandbox-postgresql`,包含 K3s 登录密码和 physical
|
||||
replication 密码。运行 Ansible 前由本机 SPIFFE identity 获取短期 Bao token,再把
|
||||
两个值注入 `SANDBOX_K3S_DB_PASSWORD` 与 `SANDBOX_REPLICATION_PASSWORD`;凭据不写入
|
||||
inventory、Git 或 Ansible fact cache。首次创建 standby 只允许覆盖不含任何业务库的
|
||||
Ubuntu 默认空集群,之后重复执行不会 reseed。
|
||||
|
||||
## Ansible
|
||||
|
||||
安装固定依赖:
|
||||
|
||||
```bash
|
||||
cd infrastructure/sandbox-cluster/ansible
|
||||
uv venv .venv
|
||||
uv pip install --python .venv/bin/python -r requirements.txt
|
||||
source .venv/bin/activate
|
||||
ansible-galaxy collection install -r requirements.yml
|
||||
export PVE_API_PYTHON="$PWD/.venv/bin/python"
|
||||
```
|
||||
|
||||
部署入口最终为:
|
||||
|
||||
```bash
|
||||
ansible-playbook site.yml
|
||||
```
|
||||
|
||||
只安装或 reconcile K3s(LXC、OS baseline 和 PostgreSQL 已就绪时):
|
||||
|
||||
```bash
|
||||
ansible-playbook k3s.yml
|
||||
```
|
||||
|
||||
只 reconcile Flux controllers 与 root sync manifest(不触碰 LXC、PostgreSQL 或
|
||||
K3s lifecycle):
|
||||
|
||||
```bash
|
||||
ansible-playbook flux.yml
|
||||
```
|
||||
|
||||
K3s 外部 datastore URI 由运行时 `SANDBOX_K3S_DB_PASSWORD` 生成,密码在 URI 中
|
||||
进行 URL 编码,最终仅持久化于节点 root 可读的 `/etc/rancher/k3s/config.yaml`
|
||||
(mode `0600`)。首节点生成的 K3s join token 仅在同一次 Ansible run 内传给第二节点;
|
||||
不在 inventory 或 Git 中维护副本。首期关闭内建 Traefik 和 ServiceLB。
|
||||
|
||||
部署后使用同一组运行时凭据执行只读验收:
|
||||
|
||||
```bash
|
||||
ansible-playbook verify.yml
|
||||
```
|
||||
|
||||
当前已经声明 LXC 生命周期、最小 OS baseline、PostgreSQL 和 K3s,包括系统级
|
||||
homelab CA trust。Flux `v2.9.5` controllers 与 root sync 也由 Ansible 通过 K3s
|
||||
server manifests 管理;root 使用 homelab CA 访问公开 Gitea 仓库,不保存 Git token。
|
||||
集群内 workload 由 `clusters/sandbox/` 分阶段纳入 Flux。root Kustomization 的健康检查
|
||||
timeout 为 40 分钟,用于覆盖 Kata 等首次安装时会逐节点重启 K3s 的子
|
||||
Kustomization;各子项仍保留自己的更短 timeout,故障会在对应子项先行暴露。
|
||||
|
||||
## SPIRE 跨集群 bootstrap
|
||||
|
||||
Sandbox 复用 homelab 的 SPIRE Server 与 `ddupan.top` trust domain。Flux 首先安装
|
||||
SPIRE CRD,并创建供 k8s_psat 使用的 reviewer。它按上游 Server chart 的权限模型调用
|
||||
TokenReview,并以 `get/list` 读取用于证明的 Pod 与 Node;不具有修改 workload 的权限。
|
||||
在该 Kustomization Ready 后运行:
|
||||
|
||||
```bash
|
||||
cd infrastructure/sandbox-cluster/ansible
|
||||
ansible-playbook spire-bootstrap.yml
|
||||
```
|
||||
|
||||
Playbook 不把 reviewer token 或生成的 kubeconfig 落盘,而是将目标 Secret manifest
|
||||
通过 stdin 交给本机 homelab `k3s kubectl`。目标 Secret
|
||||
`spire-server/spire-external-kubeconfigs` 由 Ansible 单独拥有;Flux 和人工操作不得写入。
|
||||
第二次运行必须为零变更。
|
||||
|
||||
Secret 的 `sandbox` key 仅供 SPIRE Server 的 external PSAT plugin 验证 token 与对应的
|
||||
Pod/Node;
|
||||
`sandbox-controller` key 供 external controller-manager 读取 Pod/Node、reconcile SPIFFE
|
||||
CR 及执行 leader election。两者使用不同 ServiceAccount,不得合并权限或互换。
|
||||
|
||||
## 已验证的 Kata CI 前置条件
|
||||
|
||||
- Cloud Hypervisor 必须报告 `vm.info.config.memory.shared=true`;
|
||||
- CI RuntimeClass 必须使用 `[runtime] emptydir_mode = "block-plain"`;
|
||||
- dockerd bootstrap 在 guest 内创建 `/dev/kmsg`:`mknod /dev/kmsg c 1 11`;
|
||||
- `/var/lib/docker` 必须是 guest block device 上的 ext4,Docker driver 必须为
|
||||
`overlay2`,不能静默退化到 `vfs` 或 `fuse-overlayfs`;
|
||||
- Pod 删除后必须不存在遗留 `disk.img`、VMM 或临时 credential。
|
||||
|
||||
PoC 的完整数据和陷阱见 `../kata-lxc-lab/README.md`。
|
||||
@@ -1,11 +0,0 @@
|
||||
[defaults]
|
||||
inventory = inventory/hosts.yml
|
||||
host_key_checking = True
|
||||
interpreter_python = auto_silent
|
||||
retry_files_enabled = False
|
||||
local_tmp = /tmp/ansible-sandbox-cluster-local
|
||||
remote_tmp = /tmp
|
||||
roles_path = roles
|
||||
|
||||
[ssh_connection]
|
||||
pipelining = True
|
||||
@@ -1,6 +0,0 @@
|
||||
---
|
||||
- name: Reconcile Flux controllers and root sync
|
||||
hosts: sandbox1
|
||||
gather_facts: false
|
||||
roles:
|
||||
- sandbox_flux
|
||||
@@ -1,61 +0,0 @@
|
||||
---
|
||||
sandbox_pve_api_host: "{{ lookup('env', 'PROXMOX_HOST') }}"
|
||||
sandbox_pve_api_user: "{{ lookup('env', 'PROXMOX_USER') }}"
|
||||
sandbox_pve_api_token_id: "{{ lookup('env', 'PROXMOX_TOKEN_ID') }}"
|
||||
sandbox_pve_api_token_secret: "{{ lookup('env', 'PROXMOX_TOKEN_SECRET') }}"
|
||||
sandbox_pve_validate_certs: true
|
||||
sandbox_pve_api_python_interpreter: "{{ lookup('env', 'PVE_API_PYTHON') | default(ansible_playbook_python, true) }}"
|
||||
|
||||
sandbox_bridge: labnet
|
||||
sandbox_gateway: 10.60.0.1
|
||||
sandbox_nameserver: 192.168.10.5
|
||||
sandbox_search_domain: ad.ddupan.top
|
||||
sandbox_lxc_template: laptop:vztmpl/ubuntu-24.04-standard_24.04-2_amd64.tar.zst
|
||||
sandbox_rootfs_storage: pve-rg-hdd
|
||||
sandbox_rootfs_gb: 32
|
||||
sandbox_lxc_cores: 4
|
||||
sandbox_lxc_swap_mb: 2048
|
||||
sandbox_lxc_pubkey: "{{ lookup('file', '~/.ssh/id_ed25519.pub') }}"
|
||||
sandbox_internal_ca_url: https://bao.ad.ddupan.top:8200/v1/pki/ca/pem
|
||||
sandbox_base_packages:
|
||||
- ca-certificates
|
||||
- curl
|
||||
- jq
|
||||
- openssh-server
|
||||
|
||||
sandbox_postgresql_version: 16
|
||||
sandbox_postgresql_primary_address: 10.60.0.11
|
||||
sandbox_postgresql_standby_address: 10.60.0.12
|
||||
sandbox_postgresql_lb_address: 10.60.0.1
|
||||
sandbox_postgresql_database: k3s
|
||||
sandbox_postgresql_user: k3s
|
||||
sandbox_postgresql_replication_user: k3s_replication
|
||||
sandbox_postgresql_replication_slot: sandbox2
|
||||
sandbox_postgresql_rotate_passwords: false
|
||||
sandbox_postgresql_k3s_password: "{{ lookup('env', 'SANDBOX_K3S_DB_PASSWORD') }}"
|
||||
sandbox_postgresql_replication_password: "{{ lookup('env', 'SANDBOX_REPLICATION_PASSWORD') }}"
|
||||
|
||||
# This is rendered only at playbook runtime from the Bao-provided password. K3s
|
||||
# necessarily persists its datastore credential, but config.yaml is root-only.
|
||||
sandbox_k3s_datastore_password_encoded: >-
|
||||
{{ sandbox_postgresql_k3s_password | urlencode | replace('/', '%2F') }}
|
||||
sandbox_k3s_datastore_endpoint: >-
|
||||
{{
|
||||
[
|
||||
'postgres://', sandbox_postgresql_user, ':', sandbox_k3s_datastore_password_encoded,
|
||||
'@', sandbox_postgresql_lb_address, ':5432/', sandbox_postgresql_database, '?sslmode=disable'
|
||||
] | join
|
||||
}}
|
||||
server_config_yaml: |
|
||||
datastore-endpoint: {{ sandbox_k3s_datastore_endpoint }}
|
||||
disable:
|
||||
- servicelb
|
||||
- traefik
|
||||
user_kubectl: false
|
||||
|
||||
sandbox_flux_version: v2.9.5
|
||||
sandbox_flux_install_url: >-
|
||||
https://github.com/fluxcd/flux2/releases/download/{{ sandbox_flux_version }}/install.yaml
|
||||
sandbox_flux_repository_url: https://git.ddupan.top/panxiao81/homelab-infra.git
|
||||
sandbox_flux_repository_branch: main
|
||||
sandbox_flux_path: ./clusters/sandbox
|
||||
@@ -1,48 +0,0 @@
|
||||
---
|
||||
all:
|
||||
children:
|
||||
sandbox_pve:
|
||||
hosts:
|
||||
pve1:
|
||||
ansible_host: 192.168.10.4
|
||||
sandbox_lxc:
|
||||
vmid: 148
|
||||
hostname: sandbox1
|
||||
address: 10.60.0.11/24
|
||||
memory_mb: 6144
|
||||
pve2:
|
||||
ansible_host: 192.168.10.7
|
||||
sandbox_lxc:
|
||||
vmid: 149
|
||||
hostname: sandbox2
|
||||
address: 10.60.0.12/24
|
||||
memory_mb: 4096
|
||||
|
||||
k3s_cluster:
|
||||
children:
|
||||
server:
|
||||
hosts:
|
||||
sandbox1:
|
||||
ansible_host: 10.60.0.11
|
||||
sandbox2:
|
||||
ansible_host: 10.60.0.12
|
||||
agent:
|
||||
hosts: {}
|
||||
vars:
|
||||
ansible_user: root
|
||||
k3s_version: v1.36.4+k3s1
|
||||
api_endpoint: 10.60.0.13
|
||||
k3s_server_config_mode: "0600"
|
||||
use_external_database: true
|
||||
|
||||
postgres_primary:
|
||||
hosts:
|
||||
sandbox1:
|
||||
|
||||
postgres_standby:
|
||||
hosts:
|
||||
sandbox2:
|
||||
|
||||
vars:
|
||||
ansible_user: root
|
||||
ansible_python_interpreter: /usr/bin/python3
|
||||
@@ -1,17 +0,0 @@
|
||||
---
|
||||
- name: Validate runtime K3s datastore credential
|
||||
hosts: k3s_cluster
|
||||
gather_facts: false
|
||||
tasks:
|
||||
- name: Require the Bao-provided PostgreSQL password
|
||||
ansible.builtin.assert:
|
||||
that:
|
||||
- sandbox_postgresql_k3s_password | length > 0
|
||||
- server_config_yaml is defined
|
||||
- (server_config_yaml | from_yaml)['datastore-endpoint'] == sandbox_k3s_datastore_endpoint
|
||||
fail_msg: SANDBOX_K3S_DB_PASSWORD must be populated from Bao
|
||||
quiet: true
|
||||
no_log: true
|
||||
|
||||
- name: Install the sandbox K3s control plane
|
||||
ansible.builtin.import_playbook: k3s.orchestration.site
|
||||
@@ -1,4 +0,0 @@
|
||||
ansible-core==2.21.2
|
||||
proxmoxer==2.3.0
|
||||
requests==2.34.2
|
||||
netaddr==1.3.0
|
||||
@@ -1,15 +0,0 @@
|
||||
---
|
||||
collections:
|
||||
- name: community.proxmox
|
||||
version: 2.0.0
|
||||
- name: community.postgresql
|
||||
version: 4.2.0
|
||||
- name: ansible.posix
|
||||
version: 2.2.2
|
||||
- name: community.general
|
||||
version: 13.4.0
|
||||
- name: community.library_inventory_filtering_v1
|
||||
version: 1.1.5
|
||||
- name: https://github.com/k3s-io/k3s-ansible.git
|
||||
type: git
|
||||
version: 1a600b60d37e0f8a6e2e79b0e474147b5b108ae5
|
||||
@@ -1,5 +0,0 @@
|
||||
---
|
||||
- name: Update sandbox CA certificates
|
||||
ansible.builtin.command:
|
||||
cmd: update-ca-certificates
|
||||
changed_when: true
|
||||
@@ -1,58 +0,0 @@
|
||||
---
|
||||
- name: Install sandbox base packages
|
||||
ansible.builtin.apt:
|
||||
name: "{{ sandbox_base_packages }}"
|
||||
state: present
|
||||
update_cache: true
|
||||
cache_valid_time: 3600
|
||||
|
||||
- name: Fetch the homelab internal CA
|
||||
ansible.builtin.uri:
|
||||
url: "{{ sandbox_internal_ca_url }}"
|
||||
return_content: true
|
||||
validate_certs: true
|
||||
register: sandbox_internal_ca
|
||||
changed_when: false
|
||||
retries: 3
|
||||
delay: 5
|
||||
until: sandbox_internal_ca is succeeded
|
||||
|
||||
- name: Validate the homelab internal CA response
|
||||
ansible.builtin.assert:
|
||||
that:
|
||||
- "'BEGIN CERTIFICATE' in sandbox_internal_ca.content"
|
||||
fail_msg: OpenBao did not return a PEM certificate
|
||||
quiet: true
|
||||
|
||||
- name: Install the homelab internal CA
|
||||
ansible.builtin.copy:
|
||||
dest: /usr/local/share/ca-certificates/ddupan-internal-ca.crt
|
||||
content: "{{ sandbox_internal_ca.content }}"
|
||||
owner: root
|
||||
group: root
|
||||
mode: "0644"
|
||||
notify: Update sandbox CA certificates
|
||||
|
||||
- name: Enable the SSH service
|
||||
ansible.builtin.service:
|
||||
name: ssh
|
||||
enabled: true
|
||||
state: started
|
||||
|
||||
# LXC does not expose the host kernel log device. Kubelet only needs a writable
|
||||
# kmsg-compatible character device, and the container console is the established
|
||||
# LXC mapping for this purpose. tmpfiles recreates the link after every boot.
|
||||
- name: Persist the LXC kubelet kmsg mapping
|
||||
ansible.builtin.copy:
|
||||
dest: /etc/tmpfiles.d/kmsg.conf
|
||||
content: "L+ /dev/kmsg - - - - /dev/console\n"
|
||||
owner: root
|
||||
group: root
|
||||
mode: "0644"
|
||||
|
||||
- name: Expose the LXC console as kmsg
|
||||
ansible.builtin.file:
|
||||
src: /dev/console
|
||||
dest: /dev/kmsg
|
||||
state: link
|
||||
force: true
|
||||
@@ -1,30 +0,0 @@
|
||||
---
|
||||
- name: Wait for the K3s server manifest directory
|
||||
ansible.builtin.wait_for:
|
||||
path: /var/lib/rancher/k3s/server/manifests
|
||||
state: present
|
||||
timeout: 120
|
||||
|
||||
- name: Download the pinned Flux controller manifest
|
||||
ansible.builtin.get_url:
|
||||
url: "{{ sandbox_flux_install_url }}"
|
||||
dest: /var/lib/rancher/k3s/server/manifests/flux-install.yaml
|
||||
owner: root
|
||||
group: root
|
||||
mode: "0644"
|
||||
timeout: 120
|
||||
|
||||
- name: Read the installed homelab CA
|
||||
ansible.builtin.slurp:
|
||||
src: /usr/local/share/ca-certificates/ddupan-internal-ca.crt
|
||||
register: sandbox_flux_ca
|
||||
no_log: true
|
||||
|
||||
- name: Install the Flux source and root sync manifest
|
||||
ansible.builtin.template:
|
||||
src: flux-sync.yaml.j2
|
||||
dest: /var/lib/rancher/k3s/server/manifests/flux-sync.yaml
|
||||
owner: root
|
||||
group: root
|
||||
mode: "0600"
|
||||
no_log: true
|
||||
@@ -1,39 +0,0 @@
|
||||
---
|
||||
apiVersion: v1
|
||||
kind: Secret
|
||||
metadata:
|
||||
name: flux-system
|
||||
namespace: flux-system
|
||||
type: Opaque
|
||||
stringData:
|
||||
caFile: |
|
||||
{{ sandbox_flux_ca.content | b64decode | indent(4, true) }}
|
||||
---
|
||||
apiVersion: source.toolkit.fluxcd.io/v1
|
||||
kind: GitRepository
|
||||
metadata:
|
||||
name: flux-system
|
||||
namespace: flux-system
|
||||
spec:
|
||||
interval: 1m
|
||||
ref:
|
||||
branch: {{ sandbox_flux_repository_branch }}
|
||||
secretRef:
|
||||
name: flux-system
|
||||
timeout: 60s
|
||||
url: {{ sandbox_flux_repository_url }}
|
||||
---
|
||||
apiVersion: kustomize.toolkit.fluxcd.io/v1
|
||||
kind: Kustomization
|
||||
metadata:
|
||||
name: flux-system
|
||||
namespace: flux-system
|
||||
spec:
|
||||
interval: 10m
|
||||
path: {{ sandbox_flux_path }}
|
||||
prune: false
|
||||
sourceRef:
|
||||
kind: GitRepository
|
||||
name: flux-system
|
||||
timeout: 40m
|
||||
wait: true
|
||||
@@ -1,214 +0,0 @@
|
||||
---
|
||||
- name: Validate sandbox declaration and API inputs
|
||||
ansible.builtin.assert:
|
||||
that:
|
||||
- sandbox_lxc.vmid | int > 100
|
||||
- sandbox_lxc.hostname | length > 0
|
||||
- sandbox_lxc.address is match('^10\.60\.0\.[0-9]{1,3}/24$')
|
||||
- sandbox_lxc.memory_mb | int >= 4096
|
||||
- sandbox_pve_api_host | length > 0
|
||||
- sandbox_pve_api_user | length > 0
|
||||
- sandbox_pve_api_token_id | length > 0
|
||||
- sandbox_pve_api_token_secret | length > 0
|
||||
fail_msg: Invalid sandbox declaration or missing Proxmox API environment
|
||||
no_log: true
|
||||
|
||||
- name: Check whether the LXC already exists
|
||||
ansible.builtin.stat:
|
||||
path: /etc/pve/lxc/{{ sandbox_lxc.vmid }}.conf
|
||||
register: sandbox_lxc_config
|
||||
|
||||
- name: Read current sandbox LXC runtime state
|
||||
community.proxmox.proxmox_vm_info:
|
||||
api_host: "{{ sandbox_pve_api_host }}"
|
||||
api_user: "{{ sandbox_pve_api_user }}"
|
||||
api_token_id: "{{ sandbox_pve_api_token_id }}"
|
||||
api_token_secret: "{{ sandbox_pve_api_token_secret }}"
|
||||
validate_certs: "{{ sandbox_pve_validate_certs }}"
|
||||
ca_path: /etc/ssl/certs/ca-certificates.crt
|
||||
api_timeout: 120
|
||||
vmid: "{{ sandbox_lxc.vmid }}"
|
||||
register: sandbox_lxc_info
|
||||
delegate_to: localhost
|
||||
no_log: true
|
||||
vars:
|
||||
ansible_python_interpreter: "{{ sandbox_pve_api_python_interpreter }}"
|
||||
|
||||
- name: Read current sandbox LXC configuration
|
||||
ansible.builtin.uri:
|
||||
url: >-
|
||||
https://{{ sandbox_pve_api_host }}:8006/api2/json/nodes/{{ inventory_hostname }}/lxc/{{ sandbox_lxc.vmid }}/config
|
||||
headers:
|
||||
Authorization: >-
|
||||
PVEAPIToken={{ sandbox_pve_api_user }}!{{ sandbox_pve_api_token_id }}={{ sandbox_pve_api_token_secret }}
|
||||
ca_path: /etc/ssl/certs/ca-certificates.crt
|
||||
return_content: true
|
||||
status_code: [200, 404, 500]
|
||||
register: sandbox_lxc_current_config
|
||||
delegate_to: localhost
|
||||
no_log: true
|
||||
|
||||
- name: Reconcile sandbox LXC through the PVE API
|
||||
community.proxmox.proxmox:
|
||||
api_host: "{{ sandbox_pve_api_host }}"
|
||||
api_user: "{{ sandbox_pve_api_user }}"
|
||||
api_token_id: "{{ sandbox_pve_api_token_id }}"
|
||||
api_token_secret: "{{ sandbox_pve_api_token_secret }}"
|
||||
validate_certs: "{{ sandbox_pve_validate_certs }}"
|
||||
ca_path: /etc/ssl/certs/ca-certificates.crt
|
||||
api_timeout: 120
|
||||
node: "{{ inventory_hostname }}"
|
||||
vmid: "{{ sandbox_lxc.vmid }}"
|
||||
hostname: "{{ sandbox_lxc.hostname }}"
|
||||
ostemplate: "{{ sandbox_lxc_template }}"
|
||||
ostype: ubuntu
|
||||
cmode: shell
|
||||
disk_volume:
|
||||
storage: "{{ sandbox_rootfs_storage }}"
|
||||
size: "{{ sandbox_rootfs_gb if not sandbox_lxc_config.stat.exists else omit }}"
|
||||
cores: "{{ sandbox_lxc_cores }}"
|
||||
memory: "{{ sandbox_lxc.memory_mb }}"
|
||||
swap: "{{ sandbox_lxc_swap_mb }}"
|
||||
netif:
|
||||
net0: >-
|
||||
{{
|
||||
'name=eth0,bridge=' ~ sandbox_bridge ~
|
||||
',ip=' ~ sandbox_lxc.address ~
|
||||
',gw=' ~ sandbox_gateway ~
|
||||
',type=veth' ~
|
||||
(',hwaddr=' ~ _sandbox_existing_mac if _sandbox_existing_mac | length > 0 else '')
|
||||
}}
|
||||
nameserver: "{{ sandbox_nameserver }}"
|
||||
searchdomain: "{{ sandbox_search_domain }}"
|
||||
unprivileged: false
|
||||
onboot: true
|
||||
startup: ["order=30", "up=30", "down=60"]
|
||||
tags: [ansible, sandbox, k3s, kata]
|
||||
pubkey: "{{ sandbox_lxc_pubkey }}"
|
||||
update: true
|
||||
state: present
|
||||
timeout: 120
|
||||
register: sandbox_lxc_api
|
||||
when: not sandbox_lxc_config.stat.exists or _sandbox_api_drift
|
||||
delegate_to: localhost
|
||||
no_log: true
|
||||
vars:
|
||||
ansible_python_interpreter: "{{ sandbox_pve_api_python_interpreter }}"
|
||||
_sandbox_existing_mac: >-
|
||||
{{ sandbox_lxc_current_config.json.data.net0.split('hwaddr=')[1].split(',')[0]
|
||||
if 'hwaddr=' in (sandbox_lxc_current_config.json.data.net0 | default(''))
|
||||
else '' }}
|
||||
_sandbox_api_drift: >-
|
||||
{{
|
||||
sandbox_lxc_current_config.status != 200 or
|
||||
sandbox_lxc_current_config.json.data.hostname | default('') != sandbox_lxc.hostname or
|
||||
sandbox_lxc_current_config.json.data.cores | default(0) | int != sandbox_lxc_cores | int or
|
||||
sandbox_lxc_current_config.json.data.memory | default(0) | int != sandbox_lxc.memory_mb | int or
|
||||
sandbox_lxc_current_config.json.data.swap | default(0) | int != sandbox_lxc_swap_mb | int or
|
||||
sandbox_lxc_current_config.json.data.nameserver | default('') != sandbox_nameserver or
|
||||
sandbox_lxc_current_config.json.data.searchdomain | default('') != sandbox_search_domain or
|
||||
sandbox_lxc_current_config.json.data.cmode | default('') != 'shell' or
|
||||
sandbox_lxc_current_config.json.data.onboot | default(0) | int != 1 or
|
||||
sandbox_lxc_current_config.json.data.tags | default('') != 'ansible;k3s;kata;sandbox' or
|
||||
('bridge=' ~ sandbox_bridge) not in (sandbox_lxc_current_config.json.data.net0 | default('')) or
|
||||
('ip=' ~ sandbox_lxc.address) not in (sandbox_lxc_current_config.json.data.net0 | default('')) or
|
||||
('gw=' ~ sandbox_gateway) not in (sandbox_lxc_current_config.json.data.net0 | default('')) or
|
||||
(sandbox_rootfs_storage ~ ':') not in (sandbox_lxc_current_config.json.data.rootfs | default(''))
|
||||
}}
|
||||
|
||||
- name: Grow existing sandbox rootfs through the PVE resize API
|
||||
ansible.builtin.uri:
|
||||
url: >-
|
||||
https://{{ sandbox_pve_api_host }}:8006/api2/json/nodes/{{ inventory_hostname }}/lxc/{{ sandbox_lxc.vmid }}/resize
|
||||
method: PUT
|
||||
headers:
|
||||
Authorization: >-
|
||||
PVEAPIToken={{ sandbox_pve_api_user }}!{{ sandbox_pve_api_token_id }}={{ sandbox_pve_api_token_secret }}
|
||||
body_format: form-urlencoded
|
||||
body:
|
||||
disk: rootfs
|
||||
size: "{{ sandbox_rootfs_gb }}G"
|
||||
ca_path: /etc/ssl/certs/ca-certificates.crt
|
||||
status_code: 200
|
||||
when:
|
||||
- sandbox_lxc_config.stat.exists
|
||||
- _sandbox_current_rootfs_gb | int < sandbox_rootfs_gb | int
|
||||
delegate_to: localhost
|
||||
no_log: true
|
||||
vars:
|
||||
_sandbox_current_rootfs_gb: >-
|
||||
{{
|
||||
sandbox_lxc_current_config.json.data.rootfs |
|
||||
default('') |
|
||||
regex_findall('size=([0-9]+)G') |
|
||||
first |
|
||||
default('0')
|
||||
}}
|
||||
|
||||
# These properties are not exposed by community.proxmox. Every retained line
|
||||
# is required by the already validated nested Kata/kind runtime.
|
||||
- name: Reconcile Kata-specific native LXC properties
|
||||
ansible.builtin.lineinfile:
|
||||
path: /etc/pve/lxc/{{ sandbox_lxc.vmid }}.conf
|
||||
regexp: "^{{ item.key | regex_escape }}:"
|
||||
line: "{{ item.key }}: {{ item.value }}"
|
||||
loop:
|
||||
- { key: features, value: "nesting=1,keyctl=1,fuse=1,mknod=1,force_rw_sys=1" }
|
||||
- { key: dev0, value: "path=/dev/kvm,mode=0660" }
|
||||
- { key: dev1, value: "path=/dev/vhost-net,mode=0660" }
|
||||
- { key: dev2, value: "path=/dev/vhost-vsock,mode=0660" }
|
||||
- { key: dev3, value: "path=/dev/net/tun,mode=0666" }
|
||||
- { key: lxc.apparmor.profile, value: unconfined }
|
||||
- { key: lxc.cgroup2.devices.allow, value: a }
|
||||
- { key: lxc.cap.drop, value: "" }
|
||||
- { key: lxc.mount.auto, value: "proc:rw sys:rw" }
|
||||
- { key: lxc.mount.entry, value: "/lib/modules usr/lib/modules none bind,ro,create=dir 0 0" }
|
||||
loop_control:
|
||||
label: "{{ item.key }}"
|
||||
register: sandbox_lxc_native
|
||||
|
||||
- name: Start sandbox LXC
|
||||
community.proxmox.proxmox:
|
||||
api_host: "{{ sandbox_pve_api_host }}"
|
||||
api_user: "{{ sandbox_pve_api_user }}"
|
||||
api_token_id: "{{ sandbox_pve_api_token_id }}"
|
||||
api_token_secret: "{{ sandbox_pve_api_token_secret }}"
|
||||
validate_certs: "{{ sandbox_pve_validate_certs }}"
|
||||
ca_path: /etc/ssl/certs/ca-certificates.crt
|
||||
api_timeout: 120
|
||||
vmid: "{{ sandbox_lxc.vmid }}"
|
||||
state: started
|
||||
timeout: 120
|
||||
when: >-
|
||||
not _sandbox_was_running or
|
||||
not (sandbox_lxc_api is changed or sandbox_lxc_native is changed)
|
||||
delegate_to: localhost
|
||||
no_log: true
|
||||
vars:
|
||||
ansible_python_interpreter: "{{ sandbox_pve_api_python_interpreter }}"
|
||||
_sandbox_was_running: >-
|
||||
{{ (sandbox_lxc_info.proxmox_vms | default([]) | first | default({})).status |
|
||||
default('absent') == 'running' }}
|
||||
|
||||
- name: Restart existing sandbox LXC after configuration changes
|
||||
community.proxmox.proxmox:
|
||||
api_host: "{{ sandbox_pve_api_host }}"
|
||||
api_user: "{{ sandbox_pve_api_user }}"
|
||||
api_token_id: "{{ sandbox_pve_api_token_id }}"
|
||||
api_token_secret: "{{ sandbox_pve_api_token_secret }}"
|
||||
validate_certs: "{{ sandbox_pve_validate_certs }}"
|
||||
ca_path: /etc/ssl/certs/ca-certificates.crt
|
||||
api_timeout: 120
|
||||
vmid: "{{ sandbox_lxc.vmid }}"
|
||||
state: restarted
|
||||
timeout: 120
|
||||
when:
|
||||
- _sandbox_was_running
|
||||
- sandbox_lxc_api is changed or sandbox_lxc_native is changed
|
||||
delegate_to: localhost
|
||||
no_log: true
|
||||
vars:
|
||||
ansible_python_interpreter: "{{ sandbox_pve_api_python_interpreter }}"
|
||||
_sandbox_was_running: >-
|
||||
{{ (sandbox_lxc_info.proxmox_vms | default([]) | first | default({})).status |
|
||||
default('absent') == 'running' }}
|
||||
@@ -1,6 +0,0 @@
|
||||
---
|
||||
- name: Restart sandbox PostgreSQL
|
||||
ansible.builtin.service:
|
||||
name: postgresql
|
||||
enabled: true
|
||||
state: restarted
|
||||
@@ -1,50 +0,0 @@
|
||||
---
|
||||
- name: Validate sandbox PostgreSQL inputs
|
||||
ansible.builtin.assert:
|
||||
that:
|
||||
- sandbox_postgresql_k3s_password | length >= 32
|
||||
- sandbox_postgresql_replication_password | length >= 32
|
||||
- inventory_hostname in groups['postgres_primary'] or inventory_hostname in groups['postgres_standby']
|
||||
fail_msg: Missing Bao-provided PostgreSQL credentials or invalid inventory role
|
||||
no_log: true
|
||||
|
||||
- name: Install PostgreSQL packages
|
||||
ansible.builtin.apt:
|
||||
name:
|
||||
- postgresql-{{ sandbox_postgresql_version }}
|
||||
- postgresql-client-{{ sandbox_postgresql_version }}
|
||||
- python3-psycopg2
|
||||
state: present
|
||||
update_cache: true
|
||||
cache_valid_time: 3600
|
||||
|
||||
- name: Check whether synchronous replication has completed bootstrap
|
||||
ansible.builtin.stat:
|
||||
path: /etc/postgresql/{{ sandbox_postgresql_version }}/main/sandbox-synchronous-ready
|
||||
register: sandbox_postgresql_synchronous_marker
|
||||
|
||||
- name: Install sandbox PostgreSQL access policy
|
||||
ansible.builtin.template:
|
||||
src: pg_hba.conf.j2
|
||||
dest: /etc/postgresql/{{ sandbox_postgresql_version }}/main/pg_hba.conf
|
||||
owner: postgres
|
||||
group: postgres
|
||||
mode: "0640"
|
||||
notify: Restart sandbox PostgreSQL
|
||||
|
||||
- name: Install sandbox PostgreSQL server settings
|
||||
ansible.builtin.template:
|
||||
src: sandbox.conf.j2
|
||||
dest: /etc/postgresql/{{ sandbox_postgresql_version }}/main/conf.d/99-sandbox.conf
|
||||
owner: postgres
|
||||
group: postgres
|
||||
mode: "0644"
|
||||
notify: Restart sandbox PostgreSQL
|
||||
|
||||
- name: Configure sandbox PostgreSQL primary
|
||||
ansible.builtin.include_tasks: primary.yml
|
||||
when: inventory_hostname in groups['postgres_primary']
|
||||
|
||||
- name: Configure sandbox PostgreSQL standby
|
||||
ansible.builtin.include_tasks: standby.yml
|
||||
when: inventory_hostname in groups['postgres_standby']
|
||||
@@ -1,45 +0,0 @@
|
||||
---
|
||||
- name: Apply primary PostgreSQL configuration before replication setup
|
||||
ansible.builtin.meta: flush_handlers
|
||||
|
||||
- name: Create K3s PostgreSQL login role
|
||||
community.postgresql.postgresql_user:
|
||||
name: "{{ sandbox_postgresql_user }}"
|
||||
password: "{{ sandbox_postgresql_k3s_password }}"
|
||||
no_password_changes: "{{ not sandbox_postgresql_rotate_passwords }}"
|
||||
role_attr_flags: LOGIN,NOSUPERUSER,NOCREATEDB,NOCREATEROLE,NOREPLICATION
|
||||
state: present
|
||||
become: true
|
||||
become_user: postgres
|
||||
no_log: true
|
||||
environment:
|
||||
PGOPTIONS: -c password_encryption=scram-sha-256
|
||||
|
||||
- name: Create K3s PostgreSQL database
|
||||
community.postgresql.postgresql_db:
|
||||
name: "{{ sandbox_postgresql_database }}"
|
||||
owner: "{{ sandbox_postgresql_user }}"
|
||||
state: present
|
||||
become: true
|
||||
become_user: postgres
|
||||
|
||||
- name: Create PostgreSQL replication login role
|
||||
community.postgresql.postgresql_user:
|
||||
name: "{{ sandbox_postgresql_replication_user }}"
|
||||
password: "{{ sandbox_postgresql_replication_password }}"
|
||||
no_password_changes: "{{ not sandbox_postgresql_rotate_passwords }}"
|
||||
role_attr_flags: LOGIN,REPLICATION,NOSUPERUSER,NOCREATEDB,NOCREATEROLE
|
||||
state: present
|
||||
become: true
|
||||
become_user: postgres
|
||||
no_log: true
|
||||
environment:
|
||||
PGOPTIONS: -c password_encryption=scram-sha-256
|
||||
|
||||
- name: Create physical replication slot for sandbox2
|
||||
community.postgresql.postgresql_slot:
|
||||
name: "{{ sandbox_postgresql_replication_slot }}"
|
||||
slot_type: physical
|
||||
state: present
|
||||
become: true
|
||||
become_user: postgres
|
||||
@@ -1,114 +0,0 @@
|
||||
---
|
||||
- name: Detect whether sandbox2 is already a standby
|
||||
community.postgresql.postgresql_query:
|
||||
login_db: postgres
|
||||
query: SELECT pg_is_in_recovery() AS in_recovery
|
||||
become: true
|
||||
become_user: postgres
|
||||
register: sandbox_postgresql_recovery_state
|
||||
|
||||
- name: Inspect databases before the initial standby seed
|
||||
community.postgresql.postgresql_query:
|
||||
login_db: postgres
|
||||
query: >-
|
||||
SELECT datname FROM pg_database
|
||||
WHERE NOT datistemplate AND datname <> 'postgres'
|
||||
become: true
|
||||
become_user: postgres
|
||||
register: sandbox_postgresql_existing_databases
|
||||
when: not sandbox_postgresql_recovery_state.query_result[0].in_recovery
|
||||
|
||||
- name: Refuse to overwrite a non-empty PostgreSQL node
|
||||
ansible.builtin.assert:
|
||||
that:
|
||||
- sandbox_postgresql_existing_databases.query_result | length == 0
|
||||
fail_msg: Refusing to reseed sandbox2 because it contains non-system databases
|
||||
when: not sandbox_postgresql_recovery_state.query_result[0].in_recovery
|
||||
|
||||
- name: Stop PostgreSQL before the initial standby seed
|
||||
ansible.builtin.service:
|
||||
name: postgresql
|
||||
state: stopped
|
||||
when: not sandbox_postgresql_recovery_state.query_result[0].in_recovery
|
||||
|
||||
- name: Remove the verified-empty standby data directory
|
||||
ansible.builtin.file:
|
||||
path: /var/lib/postgresql/{{ sandbox_postgresql_version }}/main
|
||||
state: absent
|
||||
when: not sandbox_postgresql_recovery_state.query_result[0].in_recovery
|
||||
|
||||
- name: Recreate the standby data directory
|
||||
ansible.builtin.file:
|
||||
path: /var/lib/postgresql/{{ sandbox_postgresql_version }}/main
|
||||
state: directory
|
||||
owner: postgres
|
||||
group: postgres
|
||||
mode: "0700"
|
||||
when: not sandbox_postgresql_recovery_state.query_result[0].in_recovery
|
||||
|
||||
- name: Install standby replication password file
|
||||
ansible.builtin.copy:
|
||||
dest: /var/lib/postgresql/.pgpass
|
||||
content: >-
|
||||
{{ sandbox_postgresql_primary_address }}:5432:*:{{ sandbox_postgresql_replication_user }}:{{ sandbox_postgresql_replication_password }}
|
||||
owner: postgres
|
||||
group: postgres
|
||||
mode: "0600"
|
||||
no_log: true
|
||||
|
||||
- name: Seed sandbox2 from the PostgreSQL primary
|
||||
ansible.builtin.command:
|
||||
argv:
|
||||
- /usr/bin/pg_basebackup
|
||||
- --host={{ sandbox_postgresql_primary_address }}
|
||||
- --username={{ sandbox_postgresql_replication_user }}
|
||||
- --pgdata=/var/lib/postgresql/{{ sandbox_postgresql_version }}/main
|
||||
- --format=plain
|
||||
- --wal-method=stream
|
||||
- --write-recovery-conf
|
||||
- --slot={{ sandbox_postgresql_replication_slot }}
|
||||
become: true
|
||||
become_user: postgres
|
||||
environment:
|
||||
PGPASSFILE: /var/lib/postgresql/.pgpass
|
||||
when: not sandbox_postgresql_recovery_state.query_result[0].in_recovery
|
||||
no_log: true
|
||||
|
||||
- name: Set the standby connection identity
|
||||
ansible.builtin.lineinfile:
|
||||
path: /var/lib/postgresql/{{ sandbox_postgresql_version }}/main/postgresql.auto.conf
|
||||
regexp: ^primary_conninfo =
|
||||
line: >-
|
||||
primary_conninfo = 'host={{ sandbox_postgresql_primary_address }} port=5432
|
||||
user={{ sandbox_postgresql_replication_user }} application_name=sandbox2
|
||||
passfile=/var/lib/postgresql/.pgpass'
|
||||
owner: postgres
|
||||
group: postgres
|
||||
mode: "0600"
|
||||
no_log: true
|
||||
|
||||
- name: Start PostgreSQL standby
|
||||
ansible.builtin.service:
|
||||
name: postgresql
|
||||
enabled: true
|
||||
state: started
|
||||
|
||||
- name: Wait for sandbox2 to enter recovery
|
||||
community.postgresql.postgresql_query:
|
||||
login_db: postgres
|
||||
query: SELECT pg_is_in_recovery() AS in_recovery
|
||||
become: true
|
||||
become_user: postgres
|
||||
register: sandbox_postgresql_standby_ready
|
||||
retries: 12
|
||||
delay: 5
|
||||
until: sandbox_postgresql_standby_ready.query_result[0].in_recovery
|
||||
|
||||
- name: Mark synchronous replication bootstrap complete on the primary
|
||||
ansible.builtin.copy:
|
||||
dest: /etc/postgresql/{{ sandbox_postgresql_version }}/main/sandbox-synchronous-ready
|
||||
content: "sandbox2\n"
|
||||
owner: postgres
|
||||
group: postgres
|
||||
mode: "0644"
|
||||
delegate_to: "{{ groups['postgres_primary'][0] }}"
|
||||
-5
@@ -1,5 +0,0 @@
|
||||
# Ansible managed
|
||||
local all postgres peer
|
||||
local all all peer
|
||||
host {{ sandbox_postgresql_database }} {{ sandbox_postgresql_user }} 10.60.0.0/24 scram-sha-256
|
||||
host replication {{ sandbox_postgresql_replication_user }} 10.60.0.0/24 scram-sha-256
|
||||
-11
@@ -1,11 +0,0 @@
|
||||
# Ansible managed
|
||||
listen_addresses = '{{ ansible_host }},127.0.0.1'
|
||||
password_encryption = 'scram-sha-256'
|
||||
wal_level = 'replica'
|
||||
max_wal_senders = 10
|
||||
max_replication_slots = 10
|
||||
hot_standby = on
|
||||
{% if inventory_hostname in groups['postgres_primary'] and sandbox_postgresql_synchronous_marker.stat.exists %}
|
||||
synchronous_standby_names = 'FIRST 1 (sandbox2)'
|
||||
synchronous_commit = 'remote_apply'
|
||||
{% endif %}
|
||||
@@ -1,7 +0,0 @@
|
||||
---
|
||||
sandbox_spire_bootstrap_api_server: https://10.60.0.13:6443
|
||||
sandbox_spire_bootstrap_source_namespace: spire-system
|
||||
sandbox_spire_bootstrap_source_secret: spire-server-token-reviewer-token
|
||||
sandbox_spire_bootstrap_controller_secret: spire-controller-manager-token
|
||||
sandbox_spire_bootstrap_target_namespace: spire-server
|
||||
sandbox_spire_bootstrap_target_secret: spire-external-kubeconfigs
|
||||
@@ -1,115 +0,0 @@
|
||||
---
|
||||
- name: Wait for the sandbox SPIRE token reviewer credential
|
||||
ansible.builtin.command:
|
||||
argv:
|
||||
- k3s
|
||||
- kubectl
|
||||
- --namespace
|
||||
- "{{ sandbox_spire_bootstrap_source_namespace }}"
|
||||
- get
|
||||
- secret
|
||||
- "{{ sandbox_spire_bootstrap_source_secret }}"
|
||||
- --output=json
|
||||
register: sandbox_spire_bootstrap_reviewer_secret
|
||||
changed_when: false
|
||||
retries: 60
|
||||
delay: 10
|
||||
until:
|
||||
- sandbox_spire_bootstrap_reviewer_secret.rc == 0
|
||||
- (sandbox_spire_bootstrap_reviewer_secret.stdout | from_json).data.token is defined
|
||||
- (sandbox_spire_bootstrap_reviewer_secret.stdout | from_json).data['ca.crt'] is defined
|
||||
no_log: true
|
||||
|
||||
- name: Wait for the sandbox SPIRE controller credential
|
||||
ansible.builtin.command:
|
||||
argv:
|
||||
- k3s
|
||||
- kubectl
|
||||
- --namespace
|
||||
- "{{ sandbox_spire_bootstrap_source_namespace }}"
|
||||
- get
|
||||
- secret
|
||||
- "{{ sandbox_spire_bootstrap_controller_secret }}"
|
||||
- --output=json
|
||||
register: sandbox_spire_bootstrap_controller_secret_result
|
||||
changed_when: false
|
||||
retries: 60
|
||||
delay: 10
|
||||
until:
|
||||
- sandbox_spire_bootstrap_controller_secret_result.rc == 0
|
||||
- (sandbox_spire_bootstrap_controller_secret_result.stdout | from_json).data.token is defined
|
||||
- (sandbox_spire_bootstrap_controller_secret_result.stdout | from_json).data['ca.crt'] is defined
|
||||
no_log: true
|
||||
|
||||
- name: Extract the sandbox TokenReview credential data
|
||||
ansible.builtin.set_fact:
|
||||
sandbox_spire_bootstrap_secret_data: >-
|
||||
{{ (sandbox_spire_bootstrap_reviewer_secret.stdout | from_json).data }}
|
||||
sandbox_spire_bootstrap_controller_data: >-
|
||||
{{ (sandbox_spire_bootstrap_controller_secret_result.stdout | from_json).data }}
|
||||
no_log: true
|
||||
|
||||
- name: Build the restricted sandbox TokenReview kubeconfig
|
||||
ansible.builtin.set_fact:
|
||||
sandbox_spire_bootstrap_kubeconfig: |
|
||||
apiVersion: v1
|
||||
kind: Config
|
||||
clusters:
|
||||
- name: sandbox
|
||||
cluster:
|
||||
server: {{ sandbox_spire_bootstrap_api_server }}
|
||||
certificate-authority-data: {{ sandbox_spire_bootstrap_secret_data['ca.crt'] }}
|
||||
users:
|
||||
- name: spire-server-token-reviewer
|
||||
user:
|
||||
token: {{ sandbox_spire_bootstrap_secret_data.token | b64decode }}
|
||||
contexts:
|
||||
- name: sandbox
|
||||
context:
|
||||
cluster: sandbox
|
||||
user: spire-server-token-reviewer
|
||||
current-context: sandbox
|
||||
sandbox_spire_bootstrap_controller_kubeconfig: |
|
||||
apiVersion: v1
|
||||
kind: Config
|
||||
clusters:
|
||||
- name: sandbox
|
||||
cluster:
|
||||
server: {{ sandbox_spire_bootstrap_api_server }}
|
||||
certificate-authority-data: {{ sandbox_spire_bootstrap_controller_data['ca.crt'] }}
|
||||
users:
|
||||
- name: spire-controller-manager
|
||||
user:
|
||||
token: {{ sandbox_spire_bootstrap_controller_data.token | b64decode }}
|
||||
contexts:
|
||||
- name: sandbox
|
||||
context:
|
||||
cluster: sandbox
|
||||
user: spire-controller-manager
|
||||
current-context: sandbox
|
||||
no_log: true
|
||||
|
||||
- name: Reconcile the central SPIRE external kubeconfig Secret
|
||||
ansible.builtin.command:
|
||||
argv:
|
||||
- k3s
|
||||
- kubectl
|
||||
- apply
|
||||
- --filename=-
|
||||
stdin: |
|
||||
apiVersion: v1
|
||||
kind: Secret
|
||||
metadata:
|
||||
name: {{ sandbox_spire_bootstrap_target_secret }}
|
||||
namespace: {{ sandbox_spire_bootstrap_target_namespace }}
|
||||
type: Opaque
|
||||
data:
|
||||
sandbox: {{ sandbox_spire_bootstrap_kubeconfig | b64encode }}
|
||||
sandbox-controller: {{ sandbox_spire_bootstrap_controller_kubeconfig | b64encode }}
|
||||
delegate_to: localhost
|
||||
become: true
|
||||
register: sandbox_spire_bootstrap_target
|
||||
changed_when: >-
|
||||
' created' in sandbox_spire_bootstrap_target.stdout or
|
||||
' configured' in sandbox_spire_bootstrap_target.stdout
|
||||
no_log: true
|
||||
@@ -1,60 +0,0 @@
|
||||
---
|
||||
- name: Reconcile sandbox LXC nodes
|
||||
hosts: sandbox_pve
|
||||
gather_facts: false
|
||||
serial: 1
|
||||
roles:
|
||||
- sandbox_lxc
|
||||
|
||||
- name: Reconcile sandbox node OS baseline
|
||||
hosts: k3s_cluster
|
||||
gather_facts: false
|
||||
roles:
|
||||
- sandbox_base
|
||||
|
||||
- name: Reconcile sandbox PostgreSQL primary
|
||||
hosts: postgres_primary
|
||||
gather_facts: false
|
||||
roles:
|
||||
- sandbox_postgresql
|
||||
|
||||
- name: Reconcile sandbox PostgreSQL synchronous standby
|
||||
hosts: postgres_standby
|
||||
gather_facts: false
|
||||
roles:
|
||||
- sandbox_postgresql
|
||||
|
||||
- name: Finalize synchronous PostgreSQL settings
|
||||
hosts: postgres_primary
|
||||
gather_facts: false
|
||||
roles:
|
||||
- sandbox_postgresql
|
||||
|
||||
- name: Validate runtime K3s datastore credential
|
||||
hosts: k3s_cluster
|
||||
gather_facts: false
|
||||
tasks:
|
||||
- name: Require the Bao-provided PostgreSQL password
|
||||
ansible.builtin.assert:
|
||||
that:
|
||||
- sandbox_postgresql_k3s_password | length > 0
|
||||
- server_config_yaml is defined
|
||||
- (server_config_yaml | from_yaml)['datastore-endpoint'] == sandbox_k3s_datastore_endpoint
|
||||
fail_msg: SANDBOX_K3S_DB_PASSWORD must be populated from Bao
|
||||
quiet: true
|
||||
no_log: true
|
||||
|
||||
- name: Install the sandbox K3s control plane
|
||||
ansible.builtin.import_playbook: k3s.orchestration.site
|
||||
|
||||
- name: Reconcile Flux controllers and root sync
|
||||
hosts: sandbox1
|
||||
gather_facts: false
|
||||
roles:
|
||||
- sandbox_flux
|
||||
|
||||
- name: Reconcile central SPIRE access to sandbox Kubernetes
|
||||
hosts: sandbox1
|
||||
gather_facts: false
|
||||
roles:
|
||||
- sandbox_spire_bootstrap
|
||||
@@ -1,6 +0,0 @@
|
||||
---
|
||||
- name: Reconcile central SPIRE access to sandbox Kubernetes
|
||||
hosts: sandbox1
|
||||
gather_facts: false
|
||||
roles:
|
||||
- sandbox_spire_bootstrap
|
||||
@@ -1,149 +0,0 @@
|
||||
---
|
||||
- name: Verify the PostgreSQL datastore through the VyOS endpoint
|
||||
hosts: k3s_cluster
|
||||
gather_facts: false
|
||||
tasks:
|
||||
- name: Query the K3s database through the stable endpoint
|
||||
community.postgresql.postgresql_query:
|
||||
login_host: "{{ sandbox_postgresql_lb_address }}"
|
||||
login_db: "{{ sandbox_postgresql_database }}"
|
||||
login_user: "{{ sandbox_postgresql_user }}"
|
||||
login_password: "{{ sandbox_postgresql_k3s_password }}"
|
||||
query: SELECT NOT pg_is_in_recovery() AS writable
|
||||
register: sandbox_datastore_endpoint
|
||||
no_log: true
|
||||
|
||||
- name: Assert the stable datastore endpoint is writable
|
||||
ansible.builtin.assert:
|
||||
that:
|
||||
- sandbox_datastore_endpoint.query_result[0].writable
|
||||
fail_msg: The VyOS datastore endpoint is not connected to a writable PostgreSQL primary
|
||||
quiet: true
|
||||
|
||||
- name: Verify synchronous PostgreSQL replication
|
||||
hosts: postgres_primary
|
||||
gather_facts: false
|
||||
tasks:
|
||||
- name: Query replication state on the primary
|
||||
community.postgresql.postgresql_query:
|
||||
login_db: postgres
|
||||
query: >-
|
||||
SELECT application_name, state, sync_state
|
||||
FROM pg_stat_replication
|
||||
WHERE application_name = 'sandbox2'
|
||||
become: true
|
||||
become_user: postgres
|
||||
register: sandbox_replication_state
|
||||
|
||||
- name: Assert sandbox2 is a synchronous streaming standby
|
||||
ansible.builtin.assert:
|
||||
that:
|
||||
- sandbox_replication_state.query_result | length == 1
|
||||
- sandbox_replication_state.query_result[0].state == 'streaming'
|
||||
- sandbox_replication_state.query_result[0].sync_state == 'sync'
|
||||
fail_msg: sandbox2 is not synchronously streaming from sandbox1
|
||||
quiet: true
|
||||
|
||||
- name: Verify PostgreSQL standby recovery state
|
||||
hosts: postgres_standby
|
||||
gather_facts: false
|
||||
tasks:
|
||||
- name: Query recovery state on the standby
|
||||
community.postgresql.postgresql_query:
|
||||
login_db: postgres
|
||||
query: SELECT pg_is_in_recovery() AS in_recovery
|
||||
become: true
|
||||
become_user: postgres
|
||||
register: sandbox_standby_state
|
||||
|
||||
- name: Assert sandbox2 remains in recovery
|
||||
ansible.builtin.assert:
|
||||
that:
|
||||
- sandbox_standby_state.query_result[0].in_recovery
|
||||
fail_msg: sandbox2 is not operating as a PostgreSQL standby
|
||||
quiet: true
|
||||
|
||||
- name: Verify K3s node prerequisites
|
||||
hosts: k3s_cluster
|
||||
gather_facts: false
|
||||
tasks:
|
||||
- name: Read the kmsg mapping
|
||||
ansible.builtin.command:
|
||||
cmd: readlink /dev/kmsg
|
||||
register: sandbox_kmsg_target
|
||||
changed_when: false
|
||||
|
||||
- name: Inspect the host kernel module mount
|
||||
ansible.builtin.shell:
|
||||
cmd: >-
|
||||
set -o pipefail &&
|
||||
findmnt -rn -T "/lib/modules/$(uname -r)" -o OPTIONS |
|
||||
grep -Eq '(^|,)ro(,|$)'
|
||||
executable: /bin/bash
|
||||
changed_when: false
|
||||
|
||||
- name: Assert required LXC kernel integration
|
||||
ansible.builtin.assert:
|
||||
that:
|
||||
- sandbox_kmsg_target.stdout == '/dev/console'
|
||||
fail_msg: LXC is missing its persistent kmsg or read-only host module mapping
|
||||
quiet: true
|
||||
|
||||
- name: Read K3s service state
|
||||
ansible.builtin.systemd_service:
|
||||
name: k3s
|
||||
register: sandbox_k3s_service
|
||||
|
||||
- name: Assert K3s is active
|
||||
ansible.builtin.assert:
|
||||
that:
|
||||
- sandbox_k3s_service.status.ActiveState == 'active'
|
||||
- sandbox_k3s_service.status.SubState == 'running'
|
||||
fail_msg: K3s is not running
|
||||
quiet: true
|
||||
|
||||
- name: Verify the K3s control plane and API VIP
|
||||
hosts: sandbox1
|
||||
gather_facts: false
|
||||
tasks:
|
||||
- name: Wait for all declared nodes
|
||||
ansible.builtin.command:
|
||||
cmd: k3s kubectl wait --for=condition=Ready nodes --all --timeout=120s
|
||||
changed_when: false
|
||||
|
||||
- name: Read control-plane nodes
|
||||
ansible.builtin.command:
|
||||
cmd: >-
|
||||
k3s kubectl get nodes
|
||||
-l node-role.kubernetes.io/control-plane=true
|
||||
-o name
|
||||
register: sandbox_control_plane_nodes
|
||||
changed_when: false
|
||||
|
||||
- name: Assert both control-plane nodes are registered
|
||||
ansible.builtin.assert:
|
||||
that:
|
||||
- sandbox_control_plane_nodes.stdout_lines | length == 2
|
||||
fail_msg: The sandbox cluster does not contain both control-plane nodes
|
||||
quiet: true
|
||||
|
||||
- name: Wait for all kube-system Pods
|
||||
ansible.builtin.command:
|
||||
cmd: >-
|
||||
k3s kubectl wait --namespace kube-system
|
||||
--for=condition=Ready pods --all --timeout=120s
|
||||
changed_when: false
|
||||
|
||||
- name: Verify the authenticated API path through VyOS
|
||||
ansible.builtin.command:
|
||||
cmd: >-
|
||||
k3s kubectl --server=https://10.60.0.13:6443 get --raw=/livez
|
||||
register: sandbox_api_livez
|
||||
changed_when: false
|
||||
|
||||
- name: Assert the API VIP is live
|
||||
ansible.builtin.assert:
|
||||
that:
|
||||
- sandbox_api_livez.stdout == 'ok'
|
||||
fail_msg: The K3s API is not healthy through the VyOS VIP
|
||||
quiet: true
|
||||
@@ -1,107 +1,51 @@
|
||||
# Gitea dynamic runner
|
||||
# Gitea dynamic runner controller
|
||||
|
||||
本目录部署单副本 Go controller,在同一进程运行 RunnerService scheduler、原生
|
||||
Kubernetes Pod worker、OpenSandbox VM worker 和 SPIFFE mTLS facade:
|
||||
此目录只管理 homelab 中的 controller 部署。controller、worker、Cloud Hypervisor
|
||||
launcher 和 guest runner 的源码与发布位于独立仓库
|
||||
`panxiao81/gitea-dynamic-runner`。
|
||||
|
||||
当前 bootstrap controller 接收 Gitea `workflow_job` webhook,将 `[self-hosted, pod]` 和
|
||||
`[self-hosted, vm]` 的 queued job 分别发布到 NATS。Pod worker 在本集群创建一次性
|
||||
privileged host runner;Docker、BuildKit 和 kind 由 workflow 自行 setup。内部
|
||||
endpoint:
|
||||
|
||||
```text
|
||||
Gitea RunnerService -> scheduler -> JetStream ci.runner.pod
|
||||
|
|
||||
v
|
||||
homelab Kubernetes Pod
|
||||
|
|
||||
SPIFFE mTLS RunnerService facade
|
||||
|
|
||||
v
|
||||
Gitea
|
||||
http://dynamic-runner-controller.dynamic-runner.svc.cluster.local:8787/webhook
|
||||
```
|
||||
|
||||
Pod backend 不经过 OpenSandbox。VM backend 后续启用时才访问 VyOS 暴露的
|
||||
OpenSandbox Lifecycle API;本目录不修改 sandbox 平台侧 ESO、Bao Terraform 或
|
||||
OpenSandbox chart 所有权边界。
|
||||
OpenBao 路径:
|
||||
|
||||
## Canary 安全边界
|
||||
- `kv/k8s/nats.ci_producer_password`:已有 NATS producer 密码。
|
||||
- `kv/k8s/nats.ci_worker_password`:已有 NATS worker 密码。
|
||||
- `kv/k8s/dynamic-runner.webhook_secret`:Gitea webhook HMAC secret。
|
||||
- `kv/k8s/gitea-runner.token`:现有 instance runner registration token。
|
||||
|
||||
- Deployment 为单副本,滚动策略固定 `maxSurge: 1`、`maxUnavailable: 0`,保证新
|
||||
facade Ready 后才终止旧实例。scheduler 必须通过 Kubernetes Lease 保持单 leader,
|
||||
不能依赖 Recreate 避免重复领取。
|
||||
- scheduler 使用一个 runner registration,并按总容量启动并发 `FetchTask` goroutine;
|
||||
`POD_CAPACITY=4` 与 `VM_CAPACITY=1` 分别限制两个 durable consumer 和 backend
|
||||
admission pool。池满时 assignment 保持 JetStream pending,任一 backend 不占用
|
||||
另一方的执行槽位,也不会创建超出容量的 workload。
|
||||
- rollout 重叠期间只有持有 `Lease/dynamic-runner-scheduler` 的 controller 执行
|
||||
`FetchTask`;所有 Ready 实例都可通过 backend metadata 恢复 claim 并服务 facade。
|
||||
- executor 镜像使用 digest;Pod 以 UID 2000 运行,SPIRE `ClusterStaticEntry` 同时绑定
|
||||
具体 Pod UID 与 `unix:uid:2000`。
|
||||
- facade 通过仅内网可路由的 `192.168.10.127:30443` NodePort 提供给 sandbox executor;
|
||||
双方使用 Workload API X509-SVID mTLS,并按 SPIFFE ID 而不是 IP/DNS 名验证服务端。
|
||||
该入口不经过公网或 Cloudflare Tunnel。
|
||||
- controller 的 SPIFFE ID 固定为
|
||||
`spiffe://ddupan.top/ns/dynamic-runner/sa/dynamic-runner-controller`。
|
||||
- NATS 保留既有最小权限分离:`ci-producer` 仅 publish,`ci-worker` 仅 pull/ACK。
|
||||
首期 controller 与 runner 镜像由 laptop 本机构建后导入 k3s containerd,作为 CI
|
||||
发布链路建立前的 bootstrap。部署使用 `imagePullPolicy: Never`。正式发布 workflow
|
||||
获得专用 SPIFFE ID 后,必须将 image 改为 zot digest 并移除本地导入步骤。
|
||||
|
||||
facade claim registry 在启动时从 Pod/OpenSandbox metadata 恢复;scheduler leadership
|
||||
由 Kubernetes Lease 持久化协调。Deployment 仍保持 `replicas: 1`,滚动更新期间允许
|
||||
一个额外 Pod 提供 facade 连续性。
|
||||
## 身份绑定
|
||||
|
||||
## Secret 边界
|
||||
queued webhook 只负责创建没有业务身份的 Pod。runner 实际领取任务后,Gitea 的
|
||||
`in_progress` webhook 会携带实际 `runner_name`;controller 将 binding 消息发布到
|
||||
NATS,Pod worker 再给对应 Pod 添加:
|
||||
|
||||
`ExternalSecret/dynamic-runner` 从既有 `ClusterSecretStore/openbao` 读取:
|
||||
|
||||
- `kv/k8s/nats`:producer/worker 密码;
|
||||
- `kv/k8s/opensandbox-api:api_key`:保留给后续 VM worker;
|
||||
- `kv/k8s/gitea-runner:token`:保留的 runner registration token;
|
||||
- `kv/k8s/dynamic-runner`:scheduler UUID/token、facade HMAC key 和回滚所需 webhook secret。
|
||||
|
||||
scheduler credential 由官方 Gitea Runner v3.5.0 一次注册生成;它只挂载到 controller,
|
||||
不会进入 executor。facade capability key 至少 32 字节,controller 为每个 assignment
|
||||
确定性生成独立 capability。不要打印 Secret、创建静态 Bao token或把 credential 写入
|
||||
Git。OpenBao 写入使用本机 SPIFFE JWT 换取的短期 `local-development` token。
|
||||
|
||||
## Workflow 身份与依赖配置
|
||||
|
||||
workflow 可复用 [`panxiao81/ci-actions`](https://git.ddupan.top/panxiao81/ci-actions)
|
||||
中的 `spiffe-openbao-login@v1` 和 `setup-nexus@v1`。这不改变 runner 的权限边界:
|
||||
runner 仅提供 Node.js 20、`spire-agent` 与 Workload API socket,workflow 负责声明 Bao
|
||||
role、audience 和具体用途,目标服务 policy 决定是否授权。短期 Bao token 会进入
|
||||
Actions job 临时文件,因此这些 Action 只允许在本目录管理的一次性 Pod/VM executor
|
||||
中使用,不能迁移到共享或持久 runner。
|
||||
|
||||
匿名读取 Nexus public repository 只需 `setup-nexus@v1`,不应为了依赖下载额外申请
|
||||
Bao 凭据;需要发布制品时再为对应 repository 建立独立 service account 与最小权限
|
||||
policy。
|
||||
|
||||
## 首次验收
|
||||
|
||||
合并后先观察 Flux 与 controller,不要立即开启 VM:
|
||||
|
||||
```bash
|
||||
flux reconcile kustomization dynamic-runner --with-source
|
||||
kubectl -n dynamic-runner wait externalsecret/dynamic-runner \
|
||||
--for=condition=Ready --timeout=2m
|
||||
kubectl -n dynamic-runner rollout status deploy/dynamic-runner-controller --timeout=5m
|
||||
kubectl -n dynamic-runner logs deploy/dynamic-runner-controller -f
|
||||
```text
|
||||
ci.ddupan.top/identity-bound=true
|
||||
ci.ddupan.top/spiffe-path=<owner>/<repository>/<percent-encoded-job-name>
|
||||
```
|
||||
|
||||
确认 scheduler 只领取一条 `[self-hosted,pod]` task,然后验证:
|
||||
`ClusterSPIFFEID/gitea-dynamic-runner` 只匹配已经绑定的 Pod,并签发
|
||||
`spiffe://ddupan.top/ci/<owner>/<repository>/<job-name>`。runner 的 job-start hook 在
|
||||
SVID 可用之前不会放行第一步,因此不能根据 queued 事件错配身份。
|
||||
|
||||
1. assignment message 进入并离开 durable `pod` consumer;
|
||||
2. `gitea-task-<task-id>` Pod 创建,取得实际 Pod UID;
|
||||
3. 同名 `ClusterStaticEntry` 的 parent ID 包含该 UID,SPIFFE ID 使用 repository/job key;
|
||||
4. 官方 runner v3.5.0 经 facade claim 精确 task,Gitea 实时收到日志和终态;
|
||||
5. Pod consumer 达到 capacity 时 VM consumer 仍可独立接受任务。
|
||||
每个 runner Pod 使用 `gitea-dynamic-runner` ServiceAccount。该 ServiceAccount 没有
|
||||
Kubernetes API 权限;只有 `dynamic-runner-pod-worker` ServiceAccount 能在本 namespace
|
||||
create/get/patch/delete Pod。
|
||||
|
||||
Pod 与 VM 都在 Gitea 接受终态后先把 terminal marker 写入各自 backend metadata,再由
|
||||
lifecycle reconciler 删除执行器。首次 VM 测试仍须观察 BatchSandbox、Pod 与
|
||||
ClusterStaticEntry 全部消失;完整自动清理通过前不得提高 `POD_CAPACITY` 或
|
||||
`VM_CAPACITY`。
|
||||
长期实现将由兼容 Gitea RunnerService 的 scheduler 直接领取 task,再交给 Pod/VM
|
||||
executor;届时删除 webhook、临时 runner 注册和 identity binding 消息。跟踪见
|
||||
`panxiao81/gitea-dynamic-runner` issue #7。
|
||||
|
||||
VM backend 已通过 `vm-dev` canary 完成 Docker、kind、SPIFFE 和完整生命周期测试。
|
||||
生产调度标签为 `[self-hosted, vm]`,初始保持 `VM_CAPACITY=1`;扩容前先观察实际任务的
|
||||
资源水位、等待时间以及 OpenSandbox 是否存在 terminal sandbox 残留。
|
||||
|
||||
## 回滚
|
||||
|
||||
若 controller 在领取 task 前失败,回滚到前一 commit 的 Python controller/Pod worker
|
||||
manifests。若已经创建 `gitea-task-*` Pod,先保留现场并核对 Gitea task 状态,不能直接
|
||||
重启 scheduler 造成重复执行。scheduler registration 和 capability key 保留在 Bao,
|
||||
回滚不需要删除或打印它们。
|
||||
Gitea webhook 只订阅 `workflow_job`,content type 使用 JSON,secret 与 Bao 中值
|
||||
一致。不要启用 `send_everything`,否则 controller 会收到无关仓库事件。
|
||||
|
||||
@@ -1,16 +1,17 @@
|
||||
apiVersion: spire.spiffe.io/v1alpha1
|
||||
kind: ClusterSPIFFEID
|
||||
metadata:
|
||||
name: dynamic-runner-controller
|
||||
name: gitea-dynamic-runner
|
||||
spec:
|
||||
className: spire-mgmt-spire
|
||||
spiffeIDTemplate: spiffe://ddupan.top/ns/dynamic-runner/sa/dynamic-runner-controller
|
||||
spiffeIDTemplate: 'spiffe://{{ .TrustDomain }}/ci/{{ index .PodMeta.Annotations "ci.ddupan.top/spiffe-path" }}'
|
||||
namespaceSelector:
|
||||
matchLabels:
|
||||
kubernetes.io/metadata.name: dynamic-runner
|
||||
podSelector:
|
||||
matchLabels:
|
||||
app.kubernetes.io/name: dynamic-runner-controller
|
||||
app.kubernetes.io/name: gitea-dynamic-runner
|
||||
ci.ddupan.top/identity-bound: "true"
|
||||
workloadSelectorTemplates:
|
||||
- k8s:ns:dynamic-runner
|
||||
- k8s:sa:dynamic-runner-controller
|
||||
- k8s:sa:gitea-dynamic-runner
|
||||
|
||||
@@ -5,11 +5,6 @@ metadata:
|
||||
namespace: dynamic-runner
|
||||
spec:
|
||||
replicas: 1
|
||||
strategy:
|
||||
type: RollingUpdate
|
||||
rollingUpdate:
|
||||
maxSurge: 1
|
||||
maxUnavailable: 0
|
||||
selector:
|
||||
matchLabels:
|
||||
app.kubernetes.io/name: dynamic-runner-controller
|
||||
@@ -19,10 +14,17 @@ spec:
|
||||
app.kubernetes.io/name: dynamic-runner-controller
|
||||
spec:
|
||||
serviceAccountName: dynamic-runner-controller
|
||||
automountServiceAccountToken: false
|
||||
initContainers:
|
||||
- name: fetch-internal-ca
|
||||
image: curlimages/curl:8.16.0@sha256:463eaf6072688fe96ac64fa623fe73e1dbe25d8ad6c34404a669ad3ce1f104b6
|
||||
args: [--fail, --silent, --show-error, --output, /trust/ca.pem, https://bao.ad.ddupan.top:8200/v1/pki/ca/pem]
|
||||
args:
|
||||
- --fail
|
||||
- --silent
|
||||
- --show-error
|
||||
- --output
|
||||
- /trust/ca.pem
|
||||
- https://bao.ad.ddupan.top:8200/v1/pki/ca/pem
|
||||
securityContext:
|
||||
allowPrivilegeEscalation: false
|
||||
capabilities:
|
||||
@@ -38,89 +40,40 @@ spec:
|
||||
mountPath: /trust
|
||||
containers:
|
||||
- name: controller
|
||||
image: zot.ad.ddupan.top/panxiao81/gitea-dynamic-runner-controller@sha256:7374a08a238ca1c76a9aee33c57520ae0edaa9b5b443708487d5b3a69a61da4e
|
||||
imagePullPolicy: IfNotPresent
|
||||
args: [controller]
|
||||
# Bootstrap import on laptop. Replace with a zot digest after the
|
||||
# repository's image publishing workflow has a dedicated identity.
|
||||
image: gitea-dynamic-runner-controller:0.3.0-bootstrap
|
||||
imagePullPolicy: Never
|
||||
env:
|
||||
- name: COMPONENTS
|
||||
value: scheduler,pod-worker,vm-worker
|
||||
- name: GITEA_INSTANCE_URL
|
||||
value: https://git.ddupan.top
|
||||
- name: GITEA_RUNNER_UUID_FILE
|
||||
value: /run/dynamic-runner-secrets/scheduler-uuid
|
||||
- name: GITEA_RUNNER_TOKEN_FILE
|
||||
value: /run/dynamic-runner-secrets/scheduler-token
|
||||
- name: NATS_URL
|
||||
value: tls://nats.ad.ddupan.top:4222
|
||||
- name: NATS_CA_FILE
|
||||
value: /run/trust/ca.pem
|
||||
- name: NATS_PRODUCER_PASSWORD_FILE
|
||||
- name: NATS_PASSWORD_FILE
|
||||
value: /run/dynamic-runner-secrets/nats-password
|
||||
- name: NATS_WORKER_PASSWORD_FILE
|
||||
value: /run/dynamic-runner-secrets/nats-worker-password
|
||||
- name: RUNNER_FACADE_CAPABILITY_KEY_FILE
|
||||
value: /run/dynamic-runner-secrets/facade-capability-key
|
||||
- name: RUNNER_FACADE_LISTEN
|
||||
value: :8443
|
||||
- name: RUNNER_FACADE_URL
|
||||
value: https://192.168.10.127:30443
|
||||
- name: RUNNER_FACADE_SPIFFE_ID
|
||||
value: spiffe://ddupan.top/ns/dynamic-runner/sa/dynamic-runner-controller
|
||||
- name: SPIFFE_ENDPOINT_SOCKET
|
||||
value: unix:///run/spire/agent-sockets/spire-agent.sock
|
||||
- name: POD_NAMESPACE
|
||||
valueFrom:
|
||||
fieldRef:
|
||||
fieldPath: metadata.namespace
|
||||
- name: POD_EXECUTOR_IMAGE
|
||||
value: zot.ad.ddupan.top/panxiao81/gitea-dynamic-runner-runner@sha256:325d75a208b1a1c6f3b4d4705e47bbc58b34733edcd12f689c60769ab6772a59
|
||||
- name: POD_SERVICE_ACCOUNT
|
||||
value: gitea-dynamic-runner
|
||||
- name: POD_EXECUTOR_UID
|
||||
value: "2000"
|
||||
- name: POD_CAPACITY
|
||||
value: "4"
|
||||
- name: OPENSANDBOX_API
|
||||
value: http://10.60.0.13:8080
|
||||
- name: OPENSANDBOX_API_KEY_FILE
|
||||
value: /run/dynamic-runner-secrets/opensandbox-api-key
|
||||
- name: OPENSANDBOX_POOL
|
||||
value: ci-vm
|
||||
- name: VM_CAPACITY
|
||||
value: "1"
|
||||
- name: VM_RUNNER_LABEL
|
||||
value: vm
|
||||
- name: VM_TIMEOUT_SECONDS
|
||||
value: "14400"
|
||||
- name: SPIRE_CLUSTER
|
||||
value: homelab
|
||||
- name: SPIRE_CLASS
|
||||
value: spire-mgmt-spire
|
||||
- name: SPIRE_AGENT_ID
|
||||
value: spiffe://ddupan.top/spire/agent/k8s_psat/homelab/cd2d0233-c4ea-4031-8327-e7e359e766dd
|
||||
- name: SSL_CERT_FILE
|
||||
value: /run/trust/ca.pem
|
||||
- name: WEBHOOK_SECRET_FILE
|
||||
value: /run/dynamic-runner-secrets/webhook-secret
|
||||
ports:
|
||||
- name: facade
|
||||
containerPort: 8443
|
||||
- name: http
|
||||
containerPort: 8787
|
||||
readinessProbe:
|
||||
httpGet: null
|
||||
tcpSocket:
|
||||
port: facade
|
||||
httpGet:
|
||||
path: /healthz
|
||||
port: http
|
||||
periodSeconds: 5
|
||||
livenessProbe:
|
||||
httpGet: null
|
||||
tcpSocket:
|
||||
port: facade
|
||||
initialDelaySeconds: 15
|
||||
httpGet:
|
||||
path: /healthz
|
||||
port: http
|
||||
initialDelaySeconds: 10
|
||||
periodSeconds: 10
|
||||
resources:
|
||||
requests:
|
||||
cpu: 50m
|
||||
memory: 64Mi
|
||||
cpu: 25m
|
||||
memory: 32Mi
|
||||
limits:
|
||||
cpu: 500m
|
||||
memory: 256Mi
|
||||
cpu: 250m
|
||||
memory: 128Mi
|
||||
securityContext:
|
||||
allowPrivilegeEscalation: false
|
||||
capabilities:
|
||||
@@ -138,9 +91,6 @@ spec:
|
||||
- name: trust
|
||||
mountPath: /run/trust
|
||||
readOnly: true
|
||||
- name: spire-agent-socket
|
||||
mountPath: /run/spire/agent-sockets
|
||||
readOnly: true
|
||||
securityContext:
|
||||
fsGroup: 65532
|
||||
fsGroupChangePolicy: OnRootMismatch
|
||||
@@ -154,7 +104,3 @@ spec:
|
||||
- name: trust
|
||||
emptyDir:
|
||||
sizeLimit: 1Mi
|
||||
- name: spire-agent-socket
|
||||
csi:
|
||||
driver: csi.spiffe.io
|
||||
readOnly: true
|
||||
|
||||
@@ -20,10 +20,6 @@ spec:
|
||||
remoteRef:
|
||||
key: k8s/nats
|
||||
property: ci_worker_password
|
||||
- secretKey: opensandbox-api-key
|
||||
remoteRef:
|
||||
key: k8s/opensandbox-api
|
||||
property: api_key
|
||||
- secretKey: webhook-secret
|
||||
remoteRef:
|
||||
key: k8s/dynamic-runner
|
||||
@@ -32,15 +28,3 @@ spec:
|
||||
remoteRef:
|
||||
key: k8s/gitea-runner
|
||||
property: token
|
||||
- secretKey: scheduler-uuid
|
||||
remoteRef:
|
||||
key: k8s/dynamic-runner
|
||||
property: scheduler_uuid
|
||||
- secretKey: scheduler-token
|
||||
remoteRef:
|
||||
key: k8s/dynamic-runner
|
||||
property: scheduler_token
|
||||
- secretKey: facade-capability-key
|
||||
remoteRef:
|
||||
key: k8s/dynamic-runner
|
||||
property: facade_capability_key
|
||||
|
||||
@@ -6,4 +6,5 @@ resources:
|
||||
- rbac.yaml
|
||||
- clusterspiffeid.yaml
|
||||
- deployment.yaml
|
||||
- pod-worker-deployment.yaml
|
||||
- service.yaml
|
||||
|
||||
@@ -0,0 +1,100 @@
|
||||
apiVersion: apps/v1
|
||||
kind: Deployment
|
||||
metadata:
|
||||
name: dynamic-runner-pod-worker
|
||||
namespace: dynamic-runner
|
||||
spec:
|
||||
replicas: 1
|
||||
selector:
|
||||
matchLabels:
|
||||
app.kubernetes.io/name: dynamic-runner-pod-worker
|
||||
template:
|
||||
metadata:
|
||||
labels:
|
||||
app.kubernetes.io/name: dynamic-runner-pod-worker
|
||||
spec:
|
||||
serviceAccountName: dynamic-runner-pod-worker
|
||||
initContainers:
|
||||
- name: fetch-internal-ca
|
||||
image: curlimages/curl:8.16.0@sha256:463eaf6072688fe96ac64fa623fe73e1dbe25d8ad6c34404a669ad3ce1f104b6
|
||||
args:
|
||||
- --fail
|
||||
- --silent
|
||||
- --show-error
|
||||
- --output
|
||||
- /trust/ca.pem
|
||||
- https://bao.ad.ddupan.top:8200/v1/pki/ca/pem
|
||||
securityContext:
|
||||
allowPrivilegeEscalation: false
|
||||
capabilities:
|
||||
drop: [ALL]
|
||||
readOnlyRootFilesystem: true
|
||||
runAsNonRoot: true
|
||||
runAsUser: 101
|
||||
runAsGroup: 102
|
||||
seccompProfile:
|
||||
type: RuntimeDefault
|
||||
volumeMounts:
|
||||
- name: trust
|
||||
mountPath: /trust
|
||||
containers:
|
||||
- name: pod-worker
|
||||
image: gitea-dynamic-runner-controller:0.3.0-bootstrap
|
||||
imagePullPolicy: Never
|
||||
command: [/venv/bin/gitea-dynamic-runner-pod-worker]
|
||||
env:
|
||||
- name: NATS_URL
|
||||
value: tls://nats.ad.ddupan.top:4222
|
||||
- name: NATS_CA_FILE
|
||||
value: /run/trust/ca.pem
|
||||
- name: NATS_PASSWORD_FILE
|
||||
value: /run/dynamic-runner-secrets/nats-worker-password
|
||||
- name: RUNNER_NAMESPACE
|
||||
valueFrom:
|
||||
fieldRef:
|
||||
fieldPath: metadata.namespace
|
||||
- name: RUNNER_IMAGE
|
||||
value: zot.ad.ddupan.top/panxiao81/gitea-dynamic-runner-runner@sha256:4c61f6315453d68a827ee9542f1345ed86576eaaf62325eb803c8fe3f06ddf2a
|
||||
- name: RUNNER_SERVICE_ACCOUNT
|
||||
value: gitea-dynamic-runner
|
||||
- name: RUNNER_TOKEN_SECRET
|
||||
value: dynamic-runner
|
||||
- name: RUNNER_CAPACITY
|
||||
value: "4"
|
||||
resources:
|
||||
requests:
|
||||
cpu: 25m
|
||||
memory: 32Mi
|
||||
limits:
|
||||
cpu: 250m
|
||||
memory: 128Mi
|
||||
securityContext:
|
||||
allowPrivilegeEscalation: false
|
||||
capabilities:
|
||||
drop: [ALL]
|
||||
readOnlyRootFilesystem: true
|
||||
runAsNonRoot: true
|
||||
runAsUser: 65532
|
||||
runAsGroup: 65532
|
||||
seccompProfile:
|
||||
type: RuntimeDefault
|
||||
volumeMounts:
|
||||
- name: secret
|
||||
mountPath: /run/dynamic-runner-secrets
|
||||
readOnly: true
|
||||
- name: trust
|
||||
mountPath: /run/trust
|
||||
readOnly: true
|
||||
securityContext:
|
||||
fsGroup: 65532
|
||||
fsGroupChangePolicy: OnRootMismatch
|
||||
seccompProfile:
|
||||
type: RuntimeDefault
|
||||
volumes:
|
||||
- name: secret
|
||||
secret:
|
||||
secretName: dynamic-runner
|
||||
defaultMode: 0400
|
||||
- name: trust
|
||||
emptyDir:
|
||||
sizeLimit: 1Mi
|
||||
@@ -6,6 +6,12 @@ metadata:
|
||||
---
|
||||
apiVersion: v1
|
||||
kind: ServiceAccount
|
||||
metadata:
|
||||
name: dynamic-runner-pod-worker
|
||||
namespace: dynamic-runner
|
||||
---
|
||||
apiVersion: v1
|
||||
kind: ServiceAccount
|
||||
metadata:
|
||||
name: gitea-dynamic-runner
|
||||
namespace: dynamic-runner
|
||||
@@ -13,48 +19,23 @@ metadata:
|
||||
apiVersion: rbac.authorization.k8s.io/v1
|
||||
kind: Role
|
||||
metadata:
|
||||
name: dynamic-runner-controller
|
||||
name: dynamic-runner-pod-worker
|
||||
namespace: dynamic-runner
|
||||
rules:
|
||||
- apiGroups: [""]
|
||||
resources: [pods]
|
||||
verbs: [create, get, list, watch, patch, delete]
|
||||
- apiGroups: [coordination.k8s.io]
|
||||
resources: [leases]
|
||||
verbs: [create, get, list, watch, update, patch]
|
||||
verbs: [create, get, patch, delete]
|
||||
---
|
||||
apiVersion: rbac.authorization.k8s.io/v1
|
||||
kind: RoleBinding
|
||||
metadata:
|
||||
name: dynamic-runner-controller
|
||||
name: dynamic-runner-pod-worker
|
||||
namespace: dynamic-runner
|
||||
roleRef:
|
||||
apiGroup: rbac.authorization.k8s.io
|
||||
kind: Role
|
||||
name: dynamic-runner-controller
|
||||
name: dynamic-runner-pod-worker
|
||||
subjects:
|
||||
- kind: ServiceAccount
|
||||
name: dynamic-runner-controller
|
||||
namespace: dynamic-runner
|
||||
---
|
||||
apiVersion: rbac.authorization.k8s.io/v1
|
||||
kind: ClusterRole
|
||||
metadata:
|
||||
name: dynamic-runner-spiffe-entries
|
||||
rules:
|
||||
- apiGroups: [spire.spiffe.io]
|
||||
resources: [clusterstaticentries]
|
||||
verbs: [create, get, delete]
|
||||
---
|
||||
apiVersion: rbac.authorization.k8s.io/v1
|
||||
kind: ClusterRoleBinding
|
||||
metadata:
|
||||
name: dynamic-runner-spiffe-entries
|
||||
roleRef:
|
||||
apiGroup: rbac.authorization.k8s.io
|
||||
kind: ClusterRole
|
||||
name: dynamic-runner-spiffe-entries
|
||||
subjects:
|
||||
- kind: ServiceAccount
|
||||
name: dynamic-runner-controller
|
||||
name: dynamic-runner-pod-worker
|
||||
namespace: dynamic-runner
|
||||
|
||||
@@ -4,11 +4,9 @@ metadata:
|
||||
name: dynamic-runner-controller
|
||||
namespace: dynamic-runner
|
||||
spec:
|
||||
type: NodePort
|
||||
selector:
|
||||
app.kubernetes.io/name: dynamic-runner-controller
|
||||
ports:
|
||||
- name: facade
|
||||
port: 8443
|
||||
targetPort: facade
|
||||
nodePort: 30443
|
||||
- name: http
|
||||
port: 8787
|
||||
targetPort: http
|
||||
|
||||
@@ -0,0 +1,114 @@
|
||||
# Gitea Actions runner
|
||||
|
||||
This is the bootstrap runner for Gitea Actions. One persistent runner Pod accepts
|
||||
up to four jobs; each job runs in a dynamically created container inside a
|
||||
Docker-in-Docker daemon. The official chart runs DinD privileged. Rootless DinD
|
||||
would still be privileged and is blocked by the node's AppArmor user-namespace
|
||||
policy, so this deployment uses regular DinD instead of weakening that host-wide
|
||||
policy. Only trusted workflows may target this runner.
|
||||
|
||||
DinD 同时使用 `--mtu=1450` 和
|
||||
`--default-network-opt=bridge=com.docker.network.driver.mtu=1450`,与 k3s Pod 的
|
||||
`eth0` 一致。前者只覆盖 Docker 默认 bridge;act 为每个 job 创建 user-defined
|
||||
bridge,必须由后者设置默认 MTU。不要在未验证节点 Pod MTU 的情况下删除或修改这
|
||||
两个参数:MTU 1500 的 job 容器虽然能够解析 GitHub、甚至建立 TCP 连接,但较大的
|
||||
TLS 数据包会在嵌套网络路径中丢失,表现为 `github.com` / `api.github.com` 超时或
|
||||
`setup-go` 每次请求卡满 6 分钟后重试。Pod 网络和默认 Docker bridge 正常不代表
|
||||
Actions job bridge 正常。
|
||||
|
||||
The runner is registered at instance scope so it is available to every repository
|
||||
on this Gitea instance. Repository permissions and protected-branch review are
|
||||
therefore the security boundary; do not enable Actions for untrusted repositories.
|
||||
|
||||
The runner registration token is authoritative in OpenBao at
|
||||
`kv/k8s/gitea-runner`. External Secrets Operator projects its `token` property to
|
||||
the `gitea-runner-token` Secret. Never put the token in this directory or a Helm
|
||||
command line.
|
||||
|
||||
## SPIRE 与 OCI 发布
|
||||
|
||||
runner Pod 使用专用 ServiceAccount `gitea-actions`,并由精确匹配 namespace、
|
||||
ServiceAccount 隐含的 Pod、以及 chart labels 的 `ClusterSPIFFEID` 获得:
|
||||
|
||||
```text
|
||||
spiffe://ddupan.top/ci/gitea-actions
|
||||
```
|
||||
|
||||
SPIFFE CSI socket 同时只读挂载到 runner 和 DinD。act 的 volume allowlist 只允许
|
||||
`/run/spire/agent-sockets`;workflow 仍必须在 job container 中显式请求该 bind
|
||||
mount。原因是 bind mount 由 DinD 内的 dockerd 解析,只挂 runner 容器无法让 job
|
||||
访问 Workload API。
|
||||
|
||||
该身份不是通用 registry 管理员。zot 只对明确列出的 CI 镜像仓库授予
|
||||
`read/create/update`,不授予 delete 或其他仓库写入。workflow 应获取
|
||||
`aud=zot` 的短期 JWT-SVID,并经 stdin 传给 registry client,不得把 JWT、X.509
|
||||
SVID 或 Docker auth 写入 workspace/artifact。
|
||||
|
||||
## Flux 接管状态
|
||||
|
||||
该 release 最初通过下述 review-first 流程手动 bootstrap。下一个 GitOps 阶段将
|
||||
使用 Flux `HelmRelease` 接管它,并首先固定现有 chart `0.1.1`,不在接管 PR 中升级。
|
||||
|
||||
迁移前审计发现:Helm 保存的 user-supplied values 和 release manifest 仍描述失败的
|
||||
rootless DinD 尝试,但 live StatefulSet 与本目录 `values.yaml` 都已经使用 regular
|
||||
DinD。首次 reconcile 的验收条件是修正 Helm 存储状态,同时 live Pod spec、PVC
|
||||
identity、runner capacity 和在线状态保持不变。接管稳定后再用独立 PR 升级 chart。
|
||||
|
||||
接管分两阶段:第一阶段提交 `suspend: true` 的 HelmRelease、HelmRepository 和由
|
||||
`values.yaml` 生成的 ConfigMap。Flux 只登记这些对象,不执行 Helm action。合并后
|
||||
检查 HelmRepository Ready,并用固定 chart 重复比较期望清单与 live StatefulSet;
|
||||
第二阶段解除 suspend。第一阶段已经确认 source Ready、完整 chart render 与 live
|
||||
资源零差异,且登记过程中现有 runner 没有 rollout。失败重试使用
|
||||
`RetryOnFailure`,不会用 stored rootless release 做 rollback。
|
||||
|
||||
## 历史 review-first bootstrap
|
||||
|
||||
这是 Flux 安装前执行过的一次性手动部署流程,保留用于恢复和审计:
|
||||
|
||||
1. Merge the reviewed PR.
|
||||
2. As a Gitea site administrator, create an instance-scoped runner registration
|
||||
token under **Site Administration → Actions → Runners**.
|
||||
3. Store it as the `token` property at `kv/k8s/gitea-runner` without exposing it
|
||||
in shell history:
|
||||
|
||||
```bash
|
||||
read -rsp 'Runner token: ' runner_token
|
||||
printf '%s' "$runner_token" | bao kv put kv/k8s/gitea-runner token=-
|
||||
unset runner_token
|
||||
```
|
||||
|
||||
4. From the updated `main`, create the namespace and ExternalSecret, then wait
|
||||
for `SecretSynced=True`:
|
||||
|
||||
```bash
|
||||
KUBECONFIG="$HOME/.kube/config" k3s kubectl apply \
|
||||
-f platform/gitea-runner/namespace.yaml
|
||||
KUBECONFIG="$HOME/.kube/config" k3s kubectl apply \
|
||||
-f platform/gitea-runner/external-secret.yaml
|
||||
KUBECONFIG="$HOME/.kube/config" k3s kubectl wait \
|
||||
--namespace gitea-actions \
|
||||
--for=condition=Ready externalsecret/gitea-runner-token \
|
||||
--timeout=60s
|
||||
```
|
||||
|
||||
5. Install chart `actions` version `0.1.1` from
|
||||
`https://dl.gitea.com/charts/` with this `values.yaml`:
|
||||
|
||||
```bash
|
||||
helm repo add gitea-charts https://dl.gitea.com/charts/
|
||||
helm repo update gitea-charts
|
||||
helm upgrade --install gitea-actions gitea-charts/actions \
|
||||
--namespace gitea-actions \
|
||||
--version 0.1.1 \
|
||||
--values platform/gitea-runner/values.yaml \
|
||||
--wait --timeout 10m
|
||||
```
|
||||
|
||||
6. Confirm the runner is online, then re-run the queued lint workflow.
|
||||
|
||||
Do not deploy from an unmerged feature branch. Do not use `--set` for the token.
|
||||
|
||||
The 1 GiB PVC preserves `.runner` identity. Docker image layers are ephemeral;
|
||||
the Pod has a 20 GiB ephemeral-storage limit. Terraform apply jobs must use a
|
||||
workflow concurrency group because runner capacity does not serialize access to
|
||||
a shared state.
|
||||
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user