3 Commits
Author SHA1 Message Date
panxiao81 54dd52f0ed 声明式管理 Proxmox API 对象与 Samba DNS 上游
yaml / yaml (pull_request) Failing after 25s
terraform / validate (pull_request) Successful in 57s
ansible / collection-test (pull_request) Successful in 1m34s
ansible / lint (pull_request) Successful in 4m1s
2026-09-17 13:18:59 +00:00
panxiao81 76ec49ff48 恢复 OCI Terraform 与站点网络 IaC 2026-09-17 13:18:00 +00:00
panxiao81 00671e1729 统一 DN42 DNS 与宿主 Docker 地址池 2026-09-17 13:17:15 +00:00
315 changed files with 547 additions and 12509 deletions
+4 -2
View File
@@ -16,6 +16,9 @@ on:
- '.ansible-lint' - '.ansible-lint'
- '.gitea/workflows/ansible.yml' - '.gitea/workflows/ansible.yml'
env:
ANSIBLE_COLLECTIONS_PATH: /root/.ansible/collections
jobs: jobs:
lint: lint:
runs-on: [self-hosted, pod] runs-on: [self-hosted, pod]
@@ -27,7 +30,6 @@ jobs:
python3 -m pip install --user --break-system-packages \ python3 -m pip install --user --break-system-packages \
--index-url https://pypi.org/simple --quiet uv==0.11.7 --index-url https://pypi.org/simple --quiet uv==0.11.7
echo "$HOME/.local/bin" >> "$GITHUB_PATH" echo "$HOME/.local/bin" >> "$GITHUB_PATH"
echo "ANSIBLE_COLLECTIONS_PATH=$HOME/.ansible/collections" >> "$GITHUB_ENV"
- name: Install ansible-lint and collections - name: Install ansible-lint and collections
run: | run: |
@@ -48,7 +50,7 @@ jobs:
- name: ansible-lint - name: ansible-lint
run: | run: |
export PATH="$HOME/.local/bin:$PATH" export PATH="$HOME/.local/bin:$PATH"
export ANSIBLE_COLLECTIONS_PATH="$PWD/infrastructure/samba-ad/ansible/collections:$HOME/.ansible/collections" export ANSIBLE_COLLECTIONS_PATH="$PWD/infrastructure/samba-ad/ansible/collections:/root/.ansible/collections"
# 静态检查不应依赖生产 vault 凭据。一次性 checkout 可以去掉加密变量文件; # 静态检查不应依赖生产 vault 凭据。一次性 checkout 可以去掉加密变量文件;
# syntax-check 只验证结构,不需要解析变量的运行时值。 # syntax-check 只验证结构,不需要解析变量的运行时值。
rm -f \ rm -f \
-22
View File
@@ -1,22 +0,0 @@
name: hydra-login
on:
pull_request:
paths:
- 'apps/hydra/login-consent/**'
- '.gitea/workflows/hydra.yml'
workflow_dispatch:
jobs:
verify:
runs-on: [self-hosted, pod]
steps:
- uses: actions/checkout@v4
- uses: actions/setup-go@v5
with:
go-version-file: apps/hydra/login-consent/go.mod
cache-dependency-path: apps/hydra/login-consent/go.sum
- name: Test authentication boundaries
working-directory: apps/hydra/login-consent
run: |
go test -race ./...
go vet ./...
CGO_ENABLED=0 go build -trimpath .
-20
View File
@@ -1,20 +0,0 @@
# Backstage
Backstage 作为 homelab 的只读开发者门户运行。应用源码、插件、测试和镜像构建归
`panxiao81/backstage` 仓库管理;本目录只保存 Kubernetes/Flux 部署声明,并通过
OCI digest 固定镜像。
## 外部前置
- OpenBao `kv/k8s/backstage` 必须包含以下与容器环境变量同名的字段:
`BACKEND_SECRET`、`AUTH_SESSION_SECRET`、`AUTH_OIDC_CLIENT_ID`、
`AUTH_OIDC_CLIENT_SECRET`、`POSTGRES_PASSWORD`、`GITEA_TOKEN`。
- PostgreSQL 需要在共享集群中预先创建由 `backstage` 角色拥有的 `backstage`
数据库;密码必须与 OpenBao 中的 `POSTGRES_PASSWORD` 一致。
- Authelia OIDC 客户端回调地址为
`https://backstage.ad.ddupan.top/api/auth/oidc/handler/frame`。
- AD DNS 需要将 `backstage.ad.ddupan.top` 指向 Envoy Gateway
`192.168.10.127`。
Flux 等待 ExternalSecret 和 Deployment 就绪;任何前置缺失都会使该
Kustomization 保持 NotReady,而不会回退到明文 Secret。
-85
View File
@@ -1,85 +0,0 @@
apiVersion: apps/v1
kind: Deployment
metadata:
name: backstage
namespace: backstage
labels:
app.kubernetes.io/name: backstage
backstage.io/kubernetes-id: homelab-backstage
spec:
replicas: 1
strategy:
type: Recreate
selector:
matchLabels:
app.kubernetes.io/name: backstage
template:
metadata:
labels:
app.kubernetes.io/name: backstage
backstage.io/kubernetes-id: homelab-backstage
spec:
serviceAccountName: backstage
securityContext:
fsGroup: 1000
runAsNonRoot: true
seccompProfile:
type: RuntimeDefault
containers:
- name: backstage
image: zot.ad.ddupan.top/panxiao81/backstage@sha256:e5a12550726f19a680bc7c40e2cc07cc624318f9279ee121814d293a006ef210
imagePullPolicy: IfNotPresent
env:
- name: BACKSTAGE_BASE_URL
value: https://backstage.ad.ddupan.top
- name: POSTGRES_HOST
value: shared-postgresql-rw.shared-db.svc.cluster.local
- name: POSTGRES_PORT
value: "5432"
- name: POSTGRES_USER
value: backstage
- name: POSTGRES_DATABASE
value: backstage
- name: GITEA_HOST
value: git.ddupan.top
envFrom:
- secretRef:
name: backstage
ports:
- containerPort: 7007
name: http
protocol: TCP
readinessProbe:
httpGet:
path: /.backstage/health/v1/readiness
port: http
initialDelaySeconds: 10
periodSeconds: 10
timeoutSeconds: 3
livenessProbe:
httpGet:
path: /.backstage/health/v1/liveness
port: http
initialDelaySeconds: 30
periodSeconds: 20
timeoutSeconds: 3
resources:
requests:
cpu: 100m
memory: 512Mi
limits:
cpu: "1"
memory: 1Gi
securityContext:
allowPrivilegeEscalation: false
capabilities:
drop: [ALL]
readOnlyRootFilesystem: true
runAsNonRoot: true
runAsUser: 1000
volumeMounts:
- mountPath: /tmp
name: tmp
volumes:
- emptyDir: {}
name: tmp
-16
View File
@@ -1,16 +0,0 @@
apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
name: backstage
namespace: backstage
spec:
refreshInterval: 1h
secretStoreRef:
kind: ClusterSecretStore
name: openbao
target:
creationPolicy: Owner
name: backstage
dataFrom:
- extract:
key: k8s/backstage
-20
View File
@@ -1,20 +0,0 @@
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
name: backstage
namespace: backstage
spec:
parentRefs:
- name: eg
namespace: envoy-gateway-system
sectionName: https
hostnames:
- backstage.ad.ddupan.top
rules:
- matches:
- path:
type: PathPrefix
value: /
backendRefs:
- name: backstage
port: 7007
-11
View File
@@ -1,11 +0,0 @@
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- namespace.yaml
- serviceaccount.yaml
- rbac.yaml
- external-secret.yaml
- deployment.yaml
- service.yaml
- networkpolicy.yaml
- httproute.yaml
-7
View File
@@ -1,7 +0,0 @@
apiVersion: v1
kind: Namespace
metadata:
name: backstage
labels:
pod-security.kubernetes.io/enforce: restricted
pod-security.kubernetes.io/enforce-version: v1.36
-24
View File
@@ -1,24 +0,0 @@
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: backstage-ingress
namespace: backstage
spec:
podSelector:
matchLabels:
app.kubernetes.io/name: backstage
policyTypes: [Ingress]
ingress:
- from:
- namespaceSelector:
matchLabels:
kubernetes.io/metadata.name: envoy-gateway-system
ports:
- port: 7007
protocol: TCP
- from:
- ipBlock:
cidr: 192.168.10.127/32
ports:
- port: 7007
protocol: TCP
-51
View File
@@ -1,51 +0,0 @@
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: backstage-read-only
rules:
- apiGroups: [""]
resources:
- configmaps
- limitranges
- pods
- pods/log
- resourcequotas
- services
verbs: [get, list, watch]
- apiGroups: [apps]
resources:
- daemonsets
- deployments
- replicasets
- statefulsets
verbs: [get, list, watch]
- apiGroups: [autoscaling]
resources:
- horizontalpodautoscalers
verbs: [get, list, watch]
- apiGroups: [batch]
resources:
- cronjobs
- jobs
verbs: [get, list, watch]
- apiGroups: [networking.k8s.io]
resources:
- ingresses
verbs: [get, list, watch]
- apiGroups: [metrics.k8s.io]
resources:
- pods
verbs: [get, list]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: backstage-read-only
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: backstage-read-only
subjects:
- kind: ServiceAccount
name: backstage
namespace: backstage
-15
View File
@@ -1,15 +0,0 @@
apiVersion: v1
kind: Service
metadata:
name: backstage
namespace: backstage
labels:
backstage.io/kubernetes-id: homelab-backstage
spec:
selector:
app.kubernetes.io/name: backstage
ports:
- name: http
port: 7007
protocol: TCP
targetPort: http
-6
View File
@@ -1,6 +0,0 @@
apiVersion: v1
kind: ServiceAccount
metadata:
name: backstage
namespace: backstage
automountServiceAccountToken: true
-12
View File
@@ -44,15 +44,3 @@ API、OIDC、Git/Flux 和 runner 均已验证。第二跳按明确决定跳过
结构,或者 release notes 指出相关 breaking migration 时,才恢复停机一致备份、分阶段 结构,或者 release notes 指出相关 breaking migration 时,才恢复停机一致备份、分阶段
suspend、详细日志审计和扩展验收。出现启动失败或 migration error 时也立即升级为完整 suspend、详细日志审计和扩展验收。出现启动失败或 migration error 时也立即升级为完整
故障流程。 故障流程。
## Hydra 人类登录 PoC
新增 `hydra` OIDC 登录源,旧 `authelia` 入口保留。Hydra 通过通用 OIDC Login/Consent
适配器转到现有 Authelia 完成人类认证;不是 Gitea 直接验证 LDAP 或 SPIFFE。
入口为 `https://git.ddupan.top/user/oauth2/hydra`,需 LAN/Tailscale 可达 Hydra 内网域名。
新 client secret 通过 `ExternalSecret/gitea-hydra-oidc` 从 OpenBao 投射。沿用
preferred_username、已验证邮箱与 groups;当前仍映射 gitea-admins,不在本轮切换组模型。
先部署并验证 Hydra discovery 后再接入本配置,避免 Gitea init 因上游不可达而失败。
实际登录验收与部署状态见 wiki;依赖和回退见 [Hydra README](../hydra/README.md)。
-7
View File
@@ -116,13 +116,6 @@ gitea:
scopes: openid profile email groups scopes: openid profile email groups
groupClaimName: groups groupClaimName: groups
adminGroup: gitea-admins adminGroup: gitea-admins
- name: hydra
provider: openidConnect
existingSecret: gitea-hydra-oidc
autoDiscoverUrl: https://hydra.ad.ddupan.top/.well-known/openid-configuration
scopes: openid profile email groups
groupClaimName: groups
adminGroup: gitea-admins
persistence: persistence:
size: 20Gi size: 20Gi
-22
View File
@@ -1,22 +0,0 @@
apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
name: gitea-hydra-oidc
namespace: gitea
spec:
refreshInterval: 1h
secretStoreRef:
kind: ClusterSecretStore
name: openbao
target:
name: gitea-hydra-oidc
creationPolicy: Owner
template:
data:
key: gitea
secret: "{{ .client_secret }}"
data:
- secretKey: client_secret
remoteRef:
key: k8s/hydra
property: gitea_client_secret
-1
View File
@@ -16,4 +16,3 @@ resources:
- helmrepository.yaml - helmrepository.yaml
- helmrelease.yaml - helmrelease.yaml
- httproute.yaml - httproute.yaml
- hydra-external-secret.yaml
-102
View File
@@ -1,102 +0,0 @@
# Hydra 与 OIDC Login/Consent PoC
本目录提供独立 Hydra 签发服务,以及一个薄的 **OIDC 上游适配器**。当前上游配置为
Authelia;适配器不连接 LDAP,也不管理用户目录。Samba AD、密码和 MFA 继续由现有
Authelia 链路负责。第一轮只接入人类和 Gitea,不实现 agent 动态授权。
```text
Gitea → Hydra → OIDC Login/Consent → Authelia → Samba AD
← OIDC ← 经验证的上游身份 ← OIDC callback
```
目标入口:
- `https://hydra.ad.ddupan.top`:Hydra 公共 OAuth2/OIDC endpoint。
- `https://hydra-login.ad.ddupan.top`:上游 OIDC 登录及 consent 适配器。
- `hydra-admin.hydra.svc.cluster.local:4445`:仅集群内管理接口,无 HTTPRoute。
均为 LAN/Tailscale 入口,复用 Envoy `eg/https` wildcard TLS。没有增加公网 tunnel。
部署及实际验收状态以 wiki 和对应 PR 为准,文件存在不表示登录已验收。
## 首次使用与边界
在 Gitea 登录页选择 `hydra`,跳转到 Authelia 完成现有人类认证,再返回原有 Gitea
账号。旧 `authelia` 登录源保留。Gitea 的账号关联和资源权限仍由 Gitea 维护。
适配器要求验证上游 issuer、audience、签名、过期时间和 nonce,使用 PKCE S256,
并把单次 state 绑定到 Secure/HttpOnly/SameSite=Lax cookie。短期登录事务只存内存,
最多 1024 个、10 分钟过期;单副本重启后正在登录的用户需重试,不存人类密码或 token。
Hydra subject 为上游 `(issuer, sub)` 的 SHA-256 加 `human:` 前缀,与可变邮箱/用户名
分离。第一轮要求上游返回经过验证的 email 及 preferred_username;这些 claims 必须
明确配置进 ID token。更换 issuer 会改变本 PoC 的 subject,正式迁移前需要身份绑定设计。
仅为显式 `ALLOWED_CLIENTS=gitea` 自动 consent,scope 限于 openid/profile/email/groups;
拒绝额外 access-token audience,不发 refresh token。只按实际请求 scope 释放 claims。
这不是通用的无人确认授权服务。组当前透传,沿用 Gitea 的 gitea-admins 映射;统一组
模型和 agent 认证均在后续阶段。不存在对 Authelia 专有协议的调用。
NetworkPolicy 限制公共端口只接收 Envoy 流量,Hydra admin 只允许适配器访问。
Hydra 使用正式模式,TLS 由 Envoy 终止;不使用 `--dev`。管理操作使用受控
`kubectl port-forward`,不要将 admin 接口暴露到 Gateway。
## 依赖、秘密与初始化
依赖共享 CloudNativePG、OpenBao/ESO、Authelia OIDC、Envoy、Samba DNS、zot 镜像仓库。
Hydra 使用独立 `hydra` database/role,不与其他应用共享数据库角色。
`kv/k8s/hydra` 保存 dsn、system_secret、upstream_client_secret、upstream_client_digest、
gitea_client_secret;通过 ExternalSecret 投射,值不写入 Git。Bootstrap 创建角色及数据库
后才启动 Hydra migration。system_secret 必须持久保存,不得在重启时随机重建。
Authelia 中新增 confidential client `hydra-login`:
- redirect URI:`https://hydra-login.ad.ddupan.top/callback`;
- authorization policy:two_factor;grant:authorization_code;PKCE:S256;
- token endpoint auth:client_secret_basic;scope:openid/profile/email/groups;
- claims policy:把 preferred_username、name、email、email_verified、groups 放入 ID token;
- client secret 的 PBKDF2 digest 存入 Authelia,原值仅供适配器使用。
Authelia 尚非 Flux 管理。修改 Helm values 时保留所有已有 clients 与 secret 引用,
通过 `--reuse-values` 和最小 overlay 增加客户端,不能以本目录配置覆盖其完整 values。
Hydra 中注册 confidential client `gitea`,redirect URI 为
`https://git.ddupan.top/user/oauth2/hydra/callback`,grant/response 为 authorization_code/code,
scope 为 openid/profile/email/groups,token endpoint auth 为 client_secret_basic。
Gitea 启动时读取 OIDC discovery,所以应先确认 Hydra 健康和 discovery 可达,再接入 Gitea。
## 构建与检查
```bash
cd apps/hydra/login-consent
go test -race ./...
go vet ./...
CGO_ENABLED=0 go build -trimpath -ldflags='-s -w' -o login-consent .
docker build -t hydra-login-consent:VERSION .
```
Go module 独立,依赖由 go.sum 锁定;Dockerfile 固定基础镜像 digest。
使用已授权的短期 SPIFFE zot 凭据发布镜像,部署使用匿名拉取入口与不可变 digest。
不把 registry 凭据写入源码或 build args。
```bash
kubectl kustomize apps/hydra
sudo k3s kubectl -n hydra get deployment,pod,externalsecret,httproute
sudo k3s kubectl -n hydra logs deployment/hydra -c migrate
sudo k3s kubectl -n hydra logs deployment/hydra-login
```
日志不输出上游 token、授权 code、challenge 或秘密。登录失败先查两端 Pod 状态、
DNS/discovery 连通性、client redirect URI 和 scope,再由用户重新发起登录。
不要在故障排查中关闭签名验证、MFA 或 state/nonce 校验。
## 恢复与回退
保留共享 PostgreSQL 中 Hydra 数据及 OpenBao 秘密;数据库持有 clients、会话及签名密钥,
单独重建 Deployment 不能替代恢复数据库。先恢复依赖,再启动 Hydra 和适配器。
当前恢复仍依赖 homelab 共享基础设施,不能声称已完成独立灾备。
第一轮不切换 Authelia 的主入口。撤回 Gitea 的新增 Hydra 登录源即可回到旧入口;
先撤消费者,再考虑停用 Hydra。不要删除旧 Authelia 登录源、用户或数据库作为回退手段。
跨服务设计见 [独立 IAM 草案](https://git.ddupan.top/panxiao81/homelab-wiki/src/branch/main/architecture/independent-iam-draft.md)。
-95
View File
@@ -1,95 +0,0 @@
apiVersion: apps/v1
kind: Deployment
metadata:
name: hydra
namespace: hydra
spec:
replicas: 1
selector:
matchLabels:
app: hydra
template:
metadata:
labels:
app: hydra
spec:
automountServiceAccountToken: false
securityContext:
runAsNonRoot: true
runAsUser: 1000
runAsGroup: 1000
seccompProfile:
type: RuntimeDefault
initContainers:
- name: migrate
image: docker.io/oryd/hydra:v26.2.0@sha256:ff67c7fb5f95074fa53374d41151713554960504b340cd3f95b09e65deaea2a9
args:
- migrate
- sql
- -e
- --yes
env:
- name: DSN
valueFrom:
secretKeyRef:
name: hydra
key: dsn
securityContext: &id002
allowPrivilegeEscalation: false
readOnlyRootFilesystem: true
capabilities:
drop:
- ALL
resources: &id001
requests:
cpu: 50m
memory: 64Mi
limits:
memory: 256Mi
containers:
- name: hydra
image: docker.io/oryd/hydra:v26.2.0@sha256:ff67c7fb5f95074fa53374d41151713554960504b340cd3f95b09e65deaea2a9
args:
- serve
- all
- --config
- /etc/hydra/hydra.yaml
- --sqa-opt-out
env:
- name: DSN
valueFrom:
secretKeyRef:
name: hydra
key: dsn
- name: SECRETS_SYSTEM
valueFrom:
secretKeyRef:
name: hydra
key: system_secret
ports:
- name: public
containerPort: 4444
- name: admin
containerPort: 4445
resources: *id001
securityContext: *id002
volumeMounts:
- name: config
mountPath: /etc/hydra
readOnly: true
readinessProbe:
httpGet:
path: /health/ready
port: admin
initialDelaySeconds: 5
periodSeconds: 5
livenessProbe:
httpGet:
path: /health/alive
port: admin
initialDelaySeconds: 20
periodSeconds: 20
volumes:
- name: config
configMap:
name: hydra-config
-16
View File
@@ -1,16 +0,0 @@
apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
name: hydra
namespace: hydra
spec:
refreshInterval: 1h
secretStoreRef:
kind: ClusterSecretStore
name: openbao
target:
name: hydra
creationPolicy: Owner
dataFrom:
- extract:
key: k8s/hydra
-33
View File
@@ -1,33 +0,0 @@
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
name: hydra-public
namespace: hydra
spec:
parentRefs:
- name: eg
namespace: envoy-gateway-system
sectionName: https
hostnames:
- hydra.ad.ddupan.top
rules:
- backendRefs:
- name: hydra-public
port: 4444
---
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
name: hydra-login
namespace: hydra
spec:
parentRefs:
- name: eg
namespace: envoy-gateway-system
sectionName: https
hostnames:
- hydra-login.ad.ddupan.top
rules:
- backendRefs:
- name: hydra-login
port: 8080
-25
View File
@@ -1,25 +0,0 @@
serve:
public:
port: 4444
admin:
port: 4445
tls:
allow_termination_from:
- 10.42.0.0/16
cookies:
same_site_mode: Lax
urls:
self:
issuer: https://hydra.ad.ddupan.top
public: https://hydra.ad.ddupan.top
login: https://hydra-login.ad.ddupan.top/login
consent: https://hydra-login.ad.ddupan.top/consent
ttl:
access_token: 15m
id_token: 15m
auth_code: 5m
log:
level: info
leak_sensitive_values: false
oauth2:
expose_internal_errors: false
-15
View File
@@ -1,15 +0,0 @@
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- namespace.yaml
- external-secret.yaml
- deployment.yaml
- login-deployment.yaml
- services.yaml
- httproutes.yaml
- networkpolicy.yaml
configMapGenerator:
- name: hydra-config
namespace: hydra
files:
- hydra.yaml
-1
View File
@@ -1 +0,0 @@
/login-consent
-4
View File
@@ -1,4 +0,0 @@
FROM gcr.io/distroless/static-debian12:nonroot@sha256:afa5c872c891853ca7fcf1f12c3edb23f7eeef36189728842dd51042ff57f7ab
COPY login-consent /login-consent
USER 65532:65532
ENTRYPOINT ["/login-consent"]
-13
View File
@@ -1,13 +0,0 @@
module git.ddupan.top/panxiao81/homelab-infra/apps/hydra/login-consent
go 1.26.0
require (
github.com/coreos/go-oidc/v3 v3.14.1
golang.org/x/oauth2 v0.37.0
)
require (
github.com/go-jose/go-jose/v4 v4.0.5 // indirect
golang.org/x/crypto v0.36.0 // indirect
)
-18
View File
@@ -1,18 +0,0 @@
github.com/coreos/go-oidc/v3 v3.14.1 h1:9ePWwfdwC4QKRlCXsJGou56adA/owXczOzwKdOumLqk=
github.com/coreos/go-oidc/v3 v3.14.1/go.mod h1:HaZ3szPaZ0e4r6ebqvsLWlk2Tn+aejfmrfah6hnSYEU=
github.com/davecgh/go-spew v1.1.1 h1:vj9j/u1bqnvCEfJOwUhtlOARqs3+rkHYY13jYWTU97c=
github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
github.com/go-jose/go-jose/v4 v4.0.5 h1:M6T8+mKZl/+fNNuFHvGIzDz7BTLQPIounk/b9dw3AaE=
github.com/go-jose/go-jose/v4 v4.0.5/go.mod h1:s3P1lRrkT8igV8D9OjyL4WRyHvjB6a4JSllnOrmmBOA=
github.com/google/go-cmp v0.6.0 h1:ofyhxvXcZhMsU5ulbFiLKl/XBFqE1GSq7atu8tAmTRI=
github.com/google/go-cmp v0.6.0/go.mod h1:17dUlkBOakJ0+DkrSSNjCkIjxS6bF9zb3elmeNGIjoY=
github.com/pmezard/go-difflib v1.0.0 h1:4DBwDE0NGyQoBHbLQYPwSUPoCMWR5BEzIk/f1lZbAQM=
github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4=
github.com/stretchr/testify v1.10.0 h1:Xv5erBjTwe/5IxqUQTdXv5kgmIvbHo3QQyRwhJsOfJA=
github.com/stretchr/testify v1.10.0/go.mod h1:r2ic/lqez/lEtzL7wO/rwa5dbSLXVDPFyf8C91i36aY=
golang.org/x/crypto v0.36.0 h1:AnAEvhDddvBdpY+uR+MyHmuZzzNqXSe/GvuDeob5L34=
golang.org/x/crypto v0.36.0/go.mod h1:Y4J0ReaxCR1IMaabaSMugxJES1EpwhBHhv2bDHklZvc=
golang.org/x/oauth2 v0.37.0 h1:JUlcxA8oAtauLfiH8FX2/FkAWHAdi0QtGCGc+hofE98=
golang.org/x/oauth2 v0.37.0/go.mod h1:IxwZNxUULJmpBFf9K/9NTMSIfZZuvuTy1gGxhigP/58=
gopkg.in/yaml.v3 v3.0.1 h1:fxVm/GzAzEWqLHuvctI91KS9hhNmmWOoWu0XTYJS7CA=
gopkg.in/yaml.v3 v3.0.1/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=
-285
View File
@@ -1,285 +0,0 @@
// Login/Consent adapter for a single trusted upstream and first-party clients.
package main
import (
"bytes"
"context"
"crypto/rand"
"crypto/sha256"
"crypto/subtle"
"encoding/base64"
"encoding/hex"
"encoding/json"
"errors"
"fmt"
"io"
"log"
"net/http"
"net/url"
"os"
"strings"
"sync"
"time"
"github.com/coreos/go-oidc/v3/oidc"
"golang.org/x/oauth2"
)
const cookieName = "__Host-hydra-login"
type pending struct {
Challenge, Nonce, Verifier string
Expires time.Time
}
type claims struct {
Username string `json:"preferred_username"`
Email string `json:"email"`
EmailVerified bool `json:"email_verified"`
Name string `json:"name"`
Groups []string `json:"groups"`
}
type flowRequest struct {
Client struct {
ID string `json:"client_id"`
} `json:"client"`
Subject string `json:"subject"`
Scopes []string `json:"requested_scope"`
Audience []string `json:"requested_access_token_audience"`
Context claims `json:"context"`
}
type app struct {
admin, public string
client *http.Client
oauth oauth2.Config
verifier *oidc.IDTokenVerifier
allowed map[string]bool
mu sync.Mutex
pending map[string]pending
}
func required(key string) string {
v := os.Getenv(key)
if v == "" {
log.Fatalf("missing %s", key)
}
return v
}
func random() string {
b := make([]byte, 32)
if _, err := rand.Read(b); err != nil {
panic(err)
}
return base64.RawURLEncoding.EncodeToString(b)
}
func (a *app) api(ctx context.Context, method, path string, in, out any) error {
var body io.Reader
if in != nil {
b, err := json.Marshal(in)
if err != nil {
return err
}
body = bytes.NewReader(b)
}
req, err := http.NewRequestWithContext(ctx, method, a.admin+path, body)
if err != nil {
return err
}
req.Header.Set("Content-Type", "application/json")
resp, err := a.client.Do(req)
if err != nil {
return errors.New("Hydra unavailable")
}
defer resp.Body.Close()
if resp.StatusCode < 200 || resp.StatusCode >= 300 {
return fmt.Errorf("Hydra status %d", resp.StatusCode)
}
if out != nil {
return json.NewDecoder(io.LimitReader(resp.Body, 1<<20)).Decode(out)
}
return nil
}
func (a *app) request(r *http.Request, kind, challenge string) (flowRequest, error) {
var f flowRequest
if challenge == "" || len(challenge) > 8192 {
return f, errors.New("missing or invalid challenge")
}
err := a.api(r.Context(), http.MethodGet, "/admin/oauth2/auth/requests/"+kind+"?"+kind+"_challenge="+url.QueryEscape(challenge), nil, &f)
if err != nil {
return f, err
}
if !a.allowed[f.Client.ID] {
return f, errors.New("client not allowed")
}
return f, nil
}
func (a *app) accept(w http.ResponseWriter, r *http.Request, kind, challenge string, body any) {
var result struct {
Redirect string `json:"redirect_to"`
}
if err := a.api(r.Context(), http.MethodPut, "/admin/oauth2/auth/requests/"+kind+"/accept?"+kind+"_challenge="+url.QueryEscape(challenge), body, &result); err != nil {
fail(w, 502)
return
}
// Only Hydra's own authorization endpoint can receive a challenge verifier.
u, err := url.Parse(result.Redirect)
p, _ := url.Parse(a.public)
if err != nil || u.Scheme != p.Scheme || u.Host != p.Host || u.User != nil || u.Path != "/oauth2/auth" {
fail(w, 502)
return
}
http.Redirect(w, r, result.Redirect, http.StatusSeeOther)
}
func fail(w http.ResponseWriter, status int) { http.Error(w, http.StatusText(status), status) }
func (a *app) login(w http.ResponseWriter, r *http.Request) {
challenge := r.URL.Query().Get("login_challenge")
if _, err := a.request(r, "login", challenge); err != nil {
fail(w, 403)
return
}
state := random()
p := pending{challenge, random(), oauth2.GenerateVerifier(), time.Now().Add(10 * time.Minute)}
a.mu.Lock()
for k, v := range a.pending {
if time.Now().After(v.Expires) {
delete(a.pending, k)
}
}
if len(a.pending) >= 1024 {
a.mu.Unlock()
fail(w, 503)
return
}
a.pending[state] = p
a.mu.Unlock()
http.SetCookie(w, &http.Cookie{Name: cookieName, Value: state, Path: "/", Secure: true, HttpOnly: true, SameSite: http.SameSiteLaxMode, MaxAge: 600})
http.Redirect(w, r, a.oauth.AuthCodeURL(state, oidc.Nonce(p.Nonce), oauth2.S256ChallengeOption(p.Verifier)), http.StatusSeeOther)
}
func (a *app) take(r *http.Request) (pending, error) {
state := r.URL.Query().Get("state")
cookie, err := r.Cookie(cookieName)
if err != nil || state == "" || subtle.ConstantTimeCompare([]byte(cookie.Value), []byte(state)) != 1 {
return pending{}, errors.New("state mismatch")
}
a.mu.Lock()
defer a.mu.Unlock()
p, ok := a.pending[state]
delete(a.pending, state)
if !ok || time.Now().After(p.Expires) {
return pending{}, errors.New("expired or used state")
}
return p, nil
}
func (a *app) callback(w http.ResponseWriter, r *http.Request) {
p, err := a.take(r)
if err != nil {
fail(w, 403)
return
}
http.SetCookie(w, &http.Cookie{Name: cookieName, Path: "/", Secure: true, HttpOnly: true, SameSite: http.SameSiteLaxMode, MaxAge: -1})
if r.URL.Query().Get("error") != "" || r.URL.Query().Get("code") == "" {
fail(w, 403)
return
}
ctx := oidc.ClientContext(r.Context(), a.client)
token, err := a.oauth.Exchange(ctx, r.URL.Query().Get("code"), oauth2.VerifierOption(p.Verifier))
if err != nil {
fail(w, 502)
return
}
raw, ok := token.Extra("id_token").(string)
if !ok {
fail(w, 502)
return
}
id, err := a.verifier.Verify(ctx, raw)
if err != nil || id.Nonce != p.Nonce || id.Subject == "" {
fail(w, 403)
return
}
var c claims
if id.Claims(&c) != nil || c.Username == "" || c.Email == "" || !c.EmailVerified {
fail(w, 403)
return
}
if _, err := a.request(r, "login", p.Challenge); err != nil {
fail(w, 403)
return
}
// Stable identity is tied to the verified upstream issuer+subject, never email.
sum := sha256.Sum256([]byte(id.Issuer + "\x00" + id.Subject))
a.accept(w, r, "login", p.Challenge, map[string]any{"subject": "human:" + hex.EncodeToString(sum[:]), "remember": false, "context": c})
}
func consentSession(f flowRequest) (map[string]any, error) {
if !strings.HasPrefix(f.Subject, "human:") || f.Context.Username == "" || f.Context.Email == "" || !f.Context.EmailVerified {
return nil, errors.New("invalid identity context")
}
allowed := map[string]bool{"openid": true, "profile": true, "email": true, "groups": true}
session := map[string]any{"principal_type": "human"}
for _, scope := range f.Scopes {
if !allowed[scope] {
return nil, errors.New("scope not allowed")
}
switch scope {
case "profile":
session["preferred_username"] = f.Context.Username
session["name"] = f.Context.Name
case "email":
session["email"] = f.Context.Email
session["email_verified"] = true
case "groups":
session["groups"] = f.Context.Groups
}
}
if len(f.Audience) > 0 {
return nil, errors.New("access token audience not allowed")
}
return session, nil
}
func (a *app) consent(w http.ResponseWriter, r *http.Request) {
challenge := r.URL.Query().Get("consent_challenge")
f, err := a.request(r, "consent", challenge)
if err != nil {
fail(w, 403)
return
}
session, err := consentSession(f)
if err != nil {
fail(w, 403)
return
}
// Explicit policy for pre-approved first-party clients only; no generic auto-consent.
a.accept(w, r, "consent", challenge, map[string]any{"grant_scope": f.Scopes, "remember": false, "session": map[string]any{"id_token": session}})
}
func (a *app) handler() http.Handler {
mux := http.NewServeMux()
mux.HandleFunc("GET /healthz", func(w http.ResponseWriter, r *http.Request) { w.WriteHeader(200) })
mux.HandleFunc("GET /login", a.login)
mux.HandleFunc("GET /callback", a.callback)
mux.HandleFunc("GET /consent", a.consent)
return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
w.Header().Set("Cache-Control", "no-store")
w.Header().Set("Referrer-Policy", "no-referrer")
w.Header().Set("X-Content-Type-Options", "nosniff")
w.Header().Set("Content-Security-Policy", "default-src 'none'; frame-ancestors 'none'")
mux.ServeHTTP(w, r)
})
}
func main() {
client := &http.Client{Timeout: 15 * time.Second, CheckRedirect: func(r *http.Request, via []*http.Request) error { return http.ErrUseLastResponse }}
issuer := required("UPSTREAM_ISSUER")
ctx := oidc.ClientContext(context.Background(), client)
provider, err := oidc.NewProvider(ctx, issuer)
if err != nil {
log.Fatal("upstream discovery failed")
}
clientID := required("UPSTREAM_CLIENT_ID")
a := &app{admin: required("HYDRA_ADMIN_URL"), public: required("HYDRA_PUBLIC_URL"), client: client, allowed: map[string]bool{}, pending: map[string]pending{},
oauth: oauth2.Config{ClientID: clientID, ClientSecret: required("UPSTREAM_CLIENT_SECRET"), RedirectURL: required("CALLBACK_URL"), Endpoint: provider.Endpoint(), Scopes: []string{"openid", "profile", "email", "groups"}},
verifier: provider.Verifier(&oidc.Config{ClientID: clientID})}
for _, id := range strings.Split(required("ALLOWED_CLIENTS"), ",") {
a.allowed[id] = true
}
s := http.Server{Addr: ":8080", Handler: a.handler(), ReadHeaderTimeout: 5 * time.Second, ReadTimeout: 20 * time.Second, WriteTimeout: 45 * time.Second, IdleTimeout: 60 * time.Second, MaxHeaderBytes: 16384}
log.Print("login/consent adapter listening on :8080")
log.Fatal(s.ListenAndServe())
}
-107
View File
@@ -1,107 +0,0 @@
package main
import (
"encoding/json"
"net/http"
"net/http/httptest"
"net/url"
"strings"
"testing"
"time"
"golang.org/x/oauth2"
)
func TestStateBoundToCookieSingleUseAndExpiry(t *testing.T) {
a := &app{pending: map[string]pending{"valid": {Challenge: "challenge", Expires: time.Now().Add(time.Minute)}, "expired": {Expires: time.Now().Add(-time.Minute)}}}
request := func(state, cookie string) *http.Request {
r := httptest.NewRequest("GET", "https://login.example/callback?state="+state, nil)
if cookie != "" {
r.AddCookie(&http.Cookie{Name: cookieName, Value: cookie})
}
return r
}
for _, r := range []*http.Request{request("valid", ""), request("valid", "other"), request("expired", "expired")} {
if _, err := a.take(r); err == nil {
t.Fatal("invalid state accepted")
}
}
if p, err := a.take(request("valid", "valid")); err != nil || p.Challenge != "challenge" {
t.Fatal("valid state rejected")
}
if _, err := a.take(request("valid", "valid")); err == nil {
t.Fatal("replayed state accepted")
}
}
func TestConsentRejectsPrivilegeExpansionAndFiltersClaims(t *testing.T) {
f := flowRequest{Subject: "human:known", Scopes: []string{"openid", "email"}, Context: claims{Username: "alice", Email: "[email protected]", EmailVerified: true, Groups: []string{"operators"}}}
s, err := consentSession(f)
if err != nil {
t.Fatal(err)
}
if _, ok := s["groups"]; ok {
t.Fatal("groups leaked without scope")
}
if _, ok := s["preferred_username"]; ok {
t.Fatal("profile leaked without scope")
}
for _, scope := range []string{"admin", "offline_access", "unknown"} {
bad := f
bad.Scopes = append([]string{"openid"}, scope)
if _, err := consentSession(bad); err == nil {
t.Fatalf("accepted %s", scope)
}
}
f.Audience = []string{"other-service"}
if _, err := consentSession(f); err == nil {
t.Fatal("unexpected audience accepted")
}
f.Audience = nil
f.Context.EmailVerified = false
if _, err := consentSession(f); err == nil {
t.Fatal("unverified email accepted")
}
}
func TestLoginValidatesClientAndUsesPKCEAndNonce(t *testing.T) {
admin := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
json.NewEncoder(w).Encode(map[string]any{"client": map[string]string{"client_id": r.URL.Query().Get("login_challenge")}})
}))
defer admin.Close()
a := &app{admin: admin.URL, client: admin.Client(), allowed: map[string]bool{"gitea": true}, pending: map[string]pending{}, oauth: oauth2.Config{ClientID: "hydra-login", RedirectURL: "https://login.example/callback", Endpoint: oauth2.Endpoint{AuthURL: "https://upstream.example/authorize"}}}
w := httptest.NewRecorder()
a.handler().ServeHTTP(w, httptest.NewRequest("GET", "https://login.example/login?login_challenge=rogue", nil))
if w.Code != 403 {
t.Fatal("unknown client accepted")
}
w = httptest.NewRecorder()
a.handler().ServeHTTP(w, httptest.NewRequest("GET", "https://login.example/login?login_challenge=gitea", nil))
if w.Code != 303 {
t.Fatalf("status %d", w.Code)
}
u, _ := url.Parse(w.Header().Get("Location"))
q := u.Query()
if q.Get("code_challenge_method") != "S256" || q.Get("code_challenge") == "" || q.Get("nonce") == "" || q.Get("state") == "" {
t.Fatal("missing protocol binding")
}
cookies := w.Result().Cookies()
if len(cookies) != 1 || !cookies[0].Secure || !cookies[0].HttpOnly || cookies[0].SameSite != http.SameSiteLaxMode || cookies[0].Value != q.Get("state") {
t.Fatal("unsafe cookie")
}
if w.Header().Get("Cache-Control") != "no-store" {
t.Fatal("missing cache protection")
}
}
func TestHydraRedirectCannotLeaveTrustedOrigin(t *testing.T) {
for _, target := range []string{"https://evil.example/oauth2/auth", "https://[email protected]/oauth2/auth", "https://hydra.example/other"} {
admin := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
json.NewEncoder(w).Encode(map[string]string{"redirect_to": target})
}))
a := &app{admin: admin.URL, public: "https://hydra.example", client: admin.Client()}
w := httptest.NewRecorder()
a.accept(w, httptest.NewRequest("GET", "https://login.example/login", nil), "login", "challenge", map[string]string{"subject": "human:test"})
if w.Code != 502 || strings.Contains(w.Header().Get("Location"), "evil") {
t.Fatal("untrusted redirect accepted")
}
admin.Close()
}
}
-69
View File
@@ -1,69 +0,0 @@
apiVersion: apps/v1
kind: Deployment
metadata:
name: hydra-login
namespace: hydra
spec:
replicas: 1
strategy:
type: Recreate
selector:
matchLabels:
app: hydra-login
template:
metadata:
labels:
app: hydra-login
spec:
automountServiceAccountToken: false
securityContext:
runAsNonRoot: true
runAsUser: 65532
runAsGroup: 65532
seccompProfile:
type: RuntimeDefault
containers:
- name: login-consent
image: zot.ad.ddupan.top/iam/oidc-login-consent@sha256:fede9b9e93c457c4b7a8a6022d9df86ff5d5900d3f6b6c4f3439851a0ae1e944
env:
- name: HYDRA_ADMIN_URL
value: http://hydra-admin.hydra.svc.cluster.local:4445
- name: HYDRA_PUBLIC_URL
value: https://hydra.ad.ddupan.top
- name: UPSTREAM_ISSUER
value: https://auth.ddupan.top
- name: UPSTREAM_CLIENT_ID
value: hydra-login
- name: CALLBACK_URL
value: https://hydra-login.ad.ddupan.top/callback
- name: ALLOWED_CLIENTS
value: gitea
- name: UPSTREAM_CLIENT_SECRET
valueFrom:
secretKeyRef:
name: hydra
key: upstream_client_secret
ports:
- name: http
containerPort: 8080
resources:
requests:
cpu: 50m
memory: 64Mi
limits:
memory: 256Mi
securityContext:
allowPrivilegeEscalation: false
readOnlyRootFilesystem: true
capabilities:
drop:
- ALL
readinessProbe:
httpGet:
path: /healthz
port: http
livenessProbe:
httpGet:
path: /healthz
port: http
initialDelaySeconds: 10
-6
View File
@@ -1,6 +0,0 @@
apiVersion: v1
kind: Namespace
metadata:
name: hydra
labels:
pod-security.kubernetes.io/enforce: restricted
-46
View File
@@ -1,46 +0,0 @@
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: hydra
namespace: hydra
spec:
podSelector:
matchLabels:
app: hydra
policyTypes:
- Ingress
ingress:
- from:
- namespaceSelector:
matchLabels:
kubernetes.io/metadata.name: envoy-gateway-system
ports:
- port: 4444
protocol: TCP
- from:
- podSelector:
matchLabels:
app: hydra-login
ports:
- port: 4445
protocol: TCP
---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: hydra-login
namespace: hydra
spec:
podSelector:
matchLabels:
app: hydra-login
policyTypes:
- Ingress
ingress:
- from:
- namespaceSelector:
matchLabels:
kubernetes.io/metadata.name: envoy-gateway-system
ports:
- port: 8080
protocol: TCP
-35
View File
@@ -1,35 +0,0 @@
apiVersion: v1
kind: Service
metadata:
name: hydra-public
namespace: hydra
spec:
selector:
app: hydra
ports:
- port: 4444
targetPort: 4444
---
apiVersion: v1
kind: Service
metadata:
name: hydra-admin
namespace: hydra
spec:
selector:
app: hydra
ports:
- port: 4445
targetPort: 4445
---
apiVersion: v1
kind: Service
metadata:
name: hydra-login
namespace: hydra
spec:
selector:
app: hydra-login
ports:
- port: 8080
targetPort: 8080
-157
View File
@@ -1,157 +0,0 @@
# Nexus Repository POC
本目录声明一个 Nexus Repository Community Edition POC,用来验证一次性 CI runner 通过
网络服务复用 Ansible Galaxy、Go Modules 与 OCI/BuildKit 缓存。Nexus 固定为 `3.96.1`,
镜像固定到官方 multi-arch index digest;LAN 入口为
`https://nexus.ad.ddupan.top`。
## POC 边界
- 单副本 Deployment,`Recreate` 更新,避免一个 RWO 卷被两个 Pod 同时挂载。
- `/nexus-data` 使用 `localpv-zfs-ceph` 上的 50 GiB RWO PVC。
- 资源预算为 250m/2 GiB request、2 CPU/4 GiB limit;JVM heap 上限 2 GiB。
- 使用容器默认的 embedded H2。它只用于 POC;正式接管 OCI 制品前必须迁移到外部
PostgreSQL,并验证备份恢复。
- 入口只在 LAN wildcard Gateway 上发布,不创建公网 DNS 或 Cloudflare route。
- 不套 Authelia forward-auth;它会破坏 Go、Ansible 与 OCI 非浏览器客户端。
- 现有 zot 保持不变。Nexus 完成 OCI、BuildKit cache 和恢复验收前不得迁移或删除 zot。
Terraform provider 创建 `ansible-public`、`go-public`、最小匿名权限与 OCI Bearer Token
Realm。Nexus 3.94 才加入的原生 OCI repository 已有 REST API,但当前锁定的 community
provider 尚未暴露 OCI resource;`terraform/reconcile-oci.sh` 因此根据 3.96.1 实例 Swagger
固定的 JSON schema,幂等调和 `oci-hosted`、`oci-proxy` 与 `oci-public`。不得绕过该入口在
UI 中创建无人管理的长期 repository。
## 部署
Flux 从 `clusters/homelab/apps/nexus.yaml` 协调本目录,并依赖 Envoy Gateway 与 OpenEBS。
合并前只渲染配置,不直接 apply:
```bash
kubectl kustomize apps/nexus
```
合并并由 Flux 部署后检查:
```bash
kubectl -n flux-system get kustomization nexus
kubectl -n nexus get pod,pvc,service,httproute
kubectl -n nexus logs deployment/nexus --tail=100
```
启动可能需要数分钟,startup probe 允许最多十分钟。不要因初次启动较慢反复删除 Pod;
先确认 PVC 已 Bound、Pod 没有 OOM,以及日志仍在推进。
DNS 期望状态已加入 `infrastructure/dns/records.yml`,需从 Samba AD Ansible root 以
`--check --diff` 核对后再按其 README 应用 DNS tag。没有 DNS 时可先用 port-forward
验证应用,但不能据此宣称 Gateway 路径已通过。
## 首次初始化与 Terraform
初始管理员密码生成在 PVC 的 `/nexus-data/admin.password`。只在交互式终端中读取并立即
完成首次密码轮换;不得把密码复制进 shell tracing、工单、Git 或命令参数。随后将
Terraform 管理账号的凭据存入 OpenBao,由 CI 通过 Terraform input variable 注入以下
环境变量:
```text
TF_VAR_nexus_url=https://nexus.ad.ddupan.top
TF_VAR_nexus_username=admin
TF_VAR_nexus_password=<OpenBao kv/infra/nexus 的 admin_password 字段>
```
`terraform/` 使用 `sonatype-nexus-community/sonatyperepo` 1.17.0,当前声明:
- `ansible-galaxy-proxy` → `https://galaxy.ansible.com`
- `ansible-public` group
- `go-proxy` → `https://proxy.golang.org`
- `go-public` group
provider credential 不写入 HCL 或 tfvars。正式 apply 前还必须为这个独立 Terraform root
配置远端 backend;本地 state 只允许用于可丢弃的 POC,不提交。验证命令:
```bash
terraform -chdir=apps/nexus/terraform init -backend=false
terraform -chdir=apps/nexus/terraform validate
```
先以 `--check` 查看 OCI repository 漂移,再明确 apply;脚本只从上述环境变量取得凭据,
用临时 `0600` netrc 调用 REST API,退出时删除:
```bash
apps/nexus/terraform/reconcile-oci.sh --check
apps/nexus/terraform/reconcile-oci.sh --apply
```
Terraform 同时把内置 `anonymous` 用户从默认的全仓库 `nx-anonymous` 角色收窄到
`ansible-public`、其返回制品 URL 使用的 `ansible-galaxy-proxy`、`go-public`,以及
`oci-public`/`oci-proxy` 的 `browse/read` 权限。`oci-hosted` 不向匿名用户开放。首次接管
已有实例时先执行
`terraform import sonatyperepo_user.anonymous anonymous,default`,再 apply;不要先启用默认的
全仓库匿名读取。
2026-09-20 的 POC 现场验收已确认:Flux 与 Pod Ready、PVC Bound、HTTPRoute 通过 HTTPS
返回 Nexus 状态 200,Samba DNS 已幂等收敛。全新客户端目录通过匿名入口下载
`community.general:11.2.0` 时冷缓存为 8.49 秒、热缓存为 1.89 秒,两次 tarball SHA-256
一致;`golang.org/x/[email protected]` 为 2.92 秒与 1.51 秒。
## 客户端验收
先验证冷缓存,再原样重复命令验证热缓存;记录 Nexus 请求、上游流量和耗时,不只观察
命令成功。Ansible 配置的 URL 必须以 `/` 结尾:
```ini
[galaxy]
server_list = nexus
[galaxy_server.nexus]
url = https://nexus.ad.ddupan.top/repository/ansible-public/
```
```bash
ansible-galaxy collection install -r collections/requirements.yml \
-p .ansible/collections
```
Go POC 使用:
```bash
GOPROXY=https://nexus.ad.ddupan.top/repository/go-public/ go mod download
```
私有 `git.ddupan.top/*` module 的 `GOPRIVATE`、凭据与是否允许 `direct` fallback 在实际
workflow 中单独决定;不要让私有 module path 意外发往公共 proxy。
OCI 使用 path-based routing:匿名公共拉取地址形如
`nexus.ad.ddupan.top/oci-public/library/alpine:3.22`,认证写入地址形如
`nexus.ad.ddupan.top/oci-hosted/<namespace>/<image>:<tag>`。2026-09-20 现场验收结果:
- `oci-public` 匿名代理拉取 Alpine 冷缓存 4.75 秒、热缓存 0.80 秒,digest 一致;
- `oci-hosted` 认证 push/pull 成功,匿名 pull 返回 401;
- amd64/arm64 OCI image index push 成功,两个平台 manifest 可见;
- Helm chart push/pull digest 与本地 tarball SHA-256 一致;
- Cosign 3.1.3 使用一次性密钥签名并验证成功,OCI 1.1 referrers API 返回一个
`application/vnd.dev.sigstore.bundle.v0.3+json` artifact;
- BuildKit `registry` cache 以 `mode=max` 导出成功,销毁首个 builder 后由新 builder 导入,
两个 `RUN` step 均明确命中 `CACHED`。
本机安装的测试客户端包括 `/usr/local/bin/cosign` 3.1.3;安装时核对官方 Linux amd64
binary SHA-256 `4629c757b7618056f8ddd7e2625ae9fdd94c0372a65049520bc7d9df9efc7f71`。
上述结果仍不代表备份恢复、外部 PostgreSQL 或正式 publisher service account 已完成;
这些项目通过前不得迁移或删除 zot。
## 数据与恢复
POC 的数据库、配置、blob、初始管理员状态都位于 `nexus-data` PVC。删除 Deployment
不会删除 PVC;删除 PVC 会永久删除整个 POC。当前没有独立备份,不能将它用于唯一副本的
正式制品。
恢复验证至少包括:停止写入、取得一致备份、在独立 PVC/实例恢复、登录、列出 repository、
拉取已缓存的 Ansible/Go 制品,并核对 OCI digest/referrers。正式化时再把数据库迁移至
外部 PostgreSQL,并分别定义数据库与 blob 的备份、恢复顺序和 RPO。
参考:
- [Nexus OCI repositories](https://help.sonatype.com/en/oci-repositories.html)
- [Nexus Ansible repositories](https://help.sonatype.com/en/ansible-repositories.html)
- [Nexus Go repositories](https://help.sonatype.com/en/go-repositories.html)
- [官方容器镜像](https://hub.docker.com/r/sonatype/nexus3)
-82
View File
@@ -1,82 +0,0 @@
apiVersion: apps/v1
kind: Deployment
metadata:
name: nexus
namespace: nexus
labels:
app.kubernetes.io/name: nexus
spec:
replicas: 1
strategy:
type: Recreate
selector:
matchLabels:
app.kubernetes.io/name: nexus
template:
metadata:
labels:
app.kubernetes.io/name: nexus
spec:
automountServiceAccountToken: false
securityContext:
fsGroup: 200
fsGroupChangePolicy: OnRootMismatch
runAsGroup: 200
runAsNonRoot: true
runAsUser: 200
seccompProfile:
type: RuntimeDefault
terminationGracePeriodSeconds: 120
containers:
- name: nexus
image: docker.io/sonatype/nexus3:3.96.1@sha256:56142f13432cf072e017aebb2025f201e42ae36ff40bb82618c702504c61f7dd
imagePullPolicy: IfNotPresent
env:
- name: INSTALL4J_ADD_VM_PARAMS
value: >-
-Xms1024m -Xmx2048m -XX:MaxDirectMemorySize=1024m
-Djava.util.prefs.userRoot=/nexus-data/javaprefs
ports:
- name: http
containerPort: 8081
protocol: TCP
resources:
requests:
cpu: 250m
memory: 2Gi
limits:
cpu: "2"
memory: 4Gi
securityContext:
allowPrivilegeEscalation: false
capabilities:
drop:
- ALL
startupProbe:
httpGet:
path: /service/rest/v1/status
port: http
failureThreshold: 60
periodSeconds: 10
timeoutSeconds: 5
readinessProbe:
httpGet:
path: /service/rest/v1/status
port: http
failureThreshold: 6
periodSeconds: 10
timeoutSeconds: 5
livenessProbe:
httpGet:
path: /service/rest/v1/status
port: http
failureThreshold: 6
periodSeconds: 30
timeoutSeconds: 5
volumeMounts:
- name: data
mountPath: /nexus-data
volumes:
- name: data
persistentVolumeClaim:
claimName: nexus-data
-16
View File
@@ -1,16 +0,0 @@
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
name: nexus
namespace: nexus
spec:
parentRefs:
- name: eg
namespace: envoy-gateway-system
sectionName: https
hostnames:
- nexus.ad.ddupan.top
rules:
- backendRefs:
- name: nexus
port: 8081
-9
View File
@@ -1,9 +0,0 @@
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- namespace.yaml
- pvc.yaml
- deployment.yaml
- service.yaml
- httproute.yaml
- networkpolicy.yaml
-8
View File
@@ -1,8 +0,0 @@
apiVersion: v1
kind: Namespace
metadata:
name: nexus
labels:
pod-security.kubernetes.io/enforce: restricted
pod-security.kubernetes.io/audit: restricted
pod-security.kubernetes.io/warn: restricted
-23
View File
@@ -1,23 +0,0 @@
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: nexus-ingress
namespace: nexus
spec:
podSelector:
matchLabels:
app.kubernetes.io/name: nexus
policyTypes:
- Ingress
ingress:
- from:
- namespaceSelector:
matchLabels:
kubernetes.io/metadata.name: envoy-gateway-system
podSelector:
matchLabels:
gateway.envoyproxy.io/owning-gateway-name: eg
gateway.envoyproxy.io/owning-gateway-namespace: envoy-gateway-system
ports:
- protocol: TCP
port: 8081
-12
View File
@@ -1,12 +0,0 @@
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: nexus-data
namespace: nexus
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 50Gi
storageClassName: localpv-zfs-ceph
-14
View File
@@ -1,14 +0,0 @@
apiVersion: v1
kind: Service
metadata:
name: nexus
namespace: nexus
spec:
type: ClusterIP
selector:
app.kubernetes.io/name: nexus
ports:
- name: http
port: 8081
protocol: TCP
targetPort: http
-6
View File
@@ -1,6 +0,0 @@
.terraform/
*.tfstate
*.tfstate.*
*.tfplan
crash.log
crash.*.log
-24
View File
@@ -1,24 +0,0 @@
# This file is maintained automatically by "terraform init".
# Manual edits may be lost in future updates.
provider "registry.terraform.io/sonatype-nexus-community/sonatyperepo" {
version = "1.17.0"
constraints = "1.17.0"
hashes = [
"h1:uKhvhhhI7B+HBsh0zq/ybqKt+EnOGyI6rjcRCtj79ZA=",
"zh:0dde99e7b343fa01f8eefc378171fb8621bedb20f59157d6cc8e3d46c738105f",
"zh:3315929df254a3a6ac27c8c846c2006f7d2a91fadc014351bc4d617f948e5bf9",
"zh:36be5a455af3ce4e187de26753de63e78c1ee9a32dba0135c6cf96a6c1fff25f",
"zh:3f73f7ff57b8c339a7c7ac37653e2dc0b2dd9dcc3f3a538788e7e3ac838337b2",
"zh:40286ecca4c22ab7ae90618ac6d2743f5055199dac81cf5204a4a397c784d439",
"zh:4d24e5c0195fb3155b1967583ee64cfeda402d7cc7f3c73369438f6c69f4245b",
"zh:828a9d7aceaac36af7f9c07af43ec8d20a89148780645d170ffb1c68b2da792d",
"zh:a5ab04de3fe626ec57c832618c6f990abd6610f81e132621651e0b180b970cff",
"zh:a959fa6090a8c0f53739879184e7346423494aee598003df0d1ab4a22b2eee91",
"zh:bdda26c2f03f918bbe59e75abea44868fafda019c3a543725331195df126350b",
"zh:d8048e149ee97ba62971e6a79355d59887bc6d10fcf72cc2feff3d0a2582670c",
"zh:dd36f9988af4e1ca5b1ca7b7bb6f658df9a220dfcda7fec7392fedfe9064f652",
"zh:dda2688d46c7e539fe97e8fe9d3ec81fb364170e018d9c6a681364c8955d4e9d",
"zh:e6b519afe2dea1c0434f766eb6bc9ba78cc5b6ef2c311c2ca3c65cb24744f31f",
]
}
-17
View File
@@ -1,17 +0,0 @@
{
"name": "oci-hosted",
"online": true,
"storage": {
"blobStoreName": "default",
"strictContentTypeValidation": true,
"writePolicy": "ALLOW",
"latestPolicy": false
},
"oci": {
"v1Enabled": false,
"forceBasicAuth": false,
"pathEnabled": true
},
"component": { "proprietaryComponents": false },
"cosign": { "enforcement": "NONE" }
}
-27
View File
@@ -1,27 +0,0 @@
{
"name": "oci-proxy",
"online": true,
"storage": {
"blobStoreName": "default",
"strictContentTypeValidation": true
},
"oci": {
"v1Enabled": false,
"forceBasicAuth": false,
"pathEnabled": true
},
"ociProxy": {
"indexType": "HUB",
"cacheForeignLayers": false,
"foreignLayerUrlWhitelist": []
},
"proxy": {
"remoteUrl": "https://registry-1.docker.io",
"contentMaxAge": 1440,
"metadataMaxAge": 60,
"preserveEncodedCharacters": false
},
"negativeCache": { "enabled": true, "timeToLive": 60 },
"httpClient": { "blocked": false, "autoBlock": true },
"cosign": { "enforcement": "NONE" }
}
-15
View File
@@ -1,15 +0,0 @@
{
"name": "oci-public",
"online": true,
"storage": {
"blobStoreName": "default",
"strictContentTypeValidation": true
},
"group": { "memberNames": ["oci-proxy"] },
"oci": {
"v1Enabled": false,
"forceBasicAuth": false,
"pathEnabled": true
},
"cosign": { "enforcement": "NONE" }
}
-80
View File
@@ -1,80 +0,0 @@
#!/usr/bin/env bash
set -euo pipefail
mode="${1:---check}"
case "$mode" in
--check | --apply) ;;
*) echo "usage: $0 [--check|--apply]" >&2; exit 2 ;;
esac
: "${TF_VAR_nexus_url:?set TF_VAR_nexus_url}"
: "${TF_VAR_nexus_username:?set TF_VAR_nexus_username}"
: "${TF_VAR_nexus_password:?set TF_VAR_nexus_password}"
script_dir="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")" && pwd)"
auth_file="$(mktemp /tmp/nexus-oci-auth.XXXXXX)"
trap 'rm -f -- "$auth_file"' EXIT
chmod 0600 "$auth_file"
printf 'machine %s\nlogin %s\npassword %s\n' \
"${TF_VAR_nexus_url#*://}" "$TF_VAR_nexus_username" \
"$TF_VAR_nexus_password" >"$auth_file"
drift=0
for entry in \
"hosted:$script_dir/oci/oci-hosted.json" \
"proxy:$script_dir/oci/oci-proxy.json" \
"group:$script_dir/oci/oci-public.json"; do
repository_type="${entry%%:*}"
desired_file="${entry#*:}"
repository_name="$(jq -er '.name' "$desired_file")"
endpoint="$TF_VAR_nexus_url/service/rest/v1/repositories/oci/$repository_type"
current_file="$(mktemp /tmp/nexus-oci-current.XXXXXX)"
status="$(curl --silent --show-error --netrc-file "$auth_file" \
--output "$current_file" --write-out '%{http_code}' \
"$endpoint/$repository_name")"
if [[ "$status" == 404 ]]; then
drift=1
if [[ "$mode" == --apply ]]; then
curl --fail --silent --show-error --netrc-file "$auth_file" \
--header 'Content-Type: application/json' \
--data-binary "@$desired_file" "$endpoint"
echo "created $repository_name"
else
echo "missing $repository_name" >&2
fi
elif [[ "$status" == 200 ]]; then
if jq -e --slurpfile desired "$desired_file" '
def subset($actual; $wanted):
if ($wanted | type) == "object" then
all($wanted | keys[];
($actual[.] != null) and subset($actual[.]; $wanted[.]))
else
$actual == $wanted
end;
subset(.; $desired[0])
' "$current_file" >/dev/null; then
echo "in sync $repository_name"
else
drift=1
if [[ "$mode" == --apply ]]; then
curl --fail --silent --show-error --netrc-file "$auth_file" \
--request PUT --header 'Content-Type: application/json' \
--data-binary "@$desired_file" "$endpoint/$repository_name"
echo "updated $repository_name"
else
echo "drifted $repository_name" >&2
fi
fi
else
cat "$current_file" >&2
echo "unexpected HTTP $status for $repository_name" >&2
exit 1
fi
rm -f -- "$current_file"
done
if [[ "$mode" == --check && "$drift" -ne 0 ]]; then
exit 1
fi
-64
View File
@@ -1,64 +0,0 @@
locals {
proxy_http_client = {
auto_block = true
blocked = false
}
proxy_negative_cache = {
enabled = true
time_to_live = 60
}
repository_storage = {
blob_store_name = "default"
strict_content_type_validation = true
}
}
resource "sonatyperepo_repository_ansiblegalaxy_proxy" "galaxy" {
name = "ansible-galaxy-proxy"
online = true
http_client = local.proxy_http_client
negative_cache = local.proxy_negative_cache
proxy = {
remote_url = "https://galaxy.ansible.com"
content_max_age = 1440
metadata_max_age = 60
}
storage = local.repository_storage
}
resource "sonatyperepo_repository_ansiblegalaxy_group" "public" {
name = "ansible-public"
online = true
group = {
member_names = [sonatyperepo_repository_ansiblegalaxy_proxy.galaxy.name]
}
storage = local.repository_storage
}
resource "sonatyperepo_repository_go_proxy" "public" {
name = "go-proxy"
online = true
http_client = local.proxy_http_client
negative_cache = local.proxy_negative_cache
proxy = {
remote_url = "https://proxy.golang.org"
content_max_age = 1440
metadata_max_age = 60
}
storage = local.repository_storage
}
resource "sonatyperepo_repository_go_group" "public" {
name = "go-public"
online = true
group = {
member_names = [sonatyperepo_repository_go_proxy.public.name]
}
storage = local.repository_storage
}
-75
View File
@@ -1,75 +0,0 @@
resource "sonatyperepo_privilege_repository_view" "anonymous_ansible" {
name = "ci-anonymous-ansible-read"
description = "Anonymous read access to the Ansible Galaxy group"
actions = ["BROWSE", "READ"]
format = "ansiblegalaxy"
repository = sonatyperepo_repository_ansiblegalaxy_group.public.name
}
resource "sonatyperepo_privilege_repository_view" "anonymous_ansible_proxy" {
name = "ci-anonymous-ansible-proxy-read"
description = "Anonymous artifact read access to the Ansible Galaxy proxy"
actions = ["BROWSE", "READ"]
format = "ansiblegalaxy"
repository = sonatyperepo_repository_ansiblegalaxy_proxy.galaxy.name
}
resource "sonatyperepo_privilege_repository_view" "anonymous_go" {
name = "ci-anonymous-go-read"
description = "Anonymous read access to the Go module group"
actions = ["BROWSE", "READ"]
format = "go"
repository = sonatyperepo_repository_go_group.public.name
}
resource "sonatyperepo_privilege_repository_view" "anonymous_oci_public" {
name = "ci-anonymous-oci-public-read"
description = "Anonymous read access to the public OCI group"
actions = ["BROWSE", "READ"]
format = "oci"
repository = "oci-public"
}
resource "sonatyperepo_privilege_repository_view" "anonymous_oci_proxy" {
name = "ci-anonymous-oci-proxy-read"
description = "Anonymous read access to the OCI proxy member"
actions = ["BROWSE", "READ"]
format = "oci"
repository = "oci-proxy"
}
resource "sonatyperepo_role" "anonymous_ci" {
id = "ci-anonymous-read"
name = "CI anonymous read"
description = "Read-only access to public CI dependency proxy groups"
privileges = [
sonatyperepo_privilege_repository_view.anonymous_ansible.name,
sonatyperepo_privilege_repository_view.anonymous_ansible_proxy.name,
sonatyperepo_privilege_repository_view.anonymous_go.name,
sonatyperepo_privilege_repository_view.anonymous_oci_public.name,
sonatyperepo_privilege_repository_view.anonymous_oci_proxy.name,
]
roles = []
}
resource "sonatyperepo_user" "anonymous" {
user_id = "anonymous"
first_name = "Anonymous"
last_name = "User"
email_address = "[email protected]"
status = "active"
roles = [sonatyperepo_role.anonymous_ci.id]
}
resource "sonatyperepo_system_anonymous_access" "ci" {
enabled = true
user_id = sonatyperepo_user.anonymous.user_id
realm_name = "NexusAuthorizingRealm"
}
resource "sonatyperepo_security_realms" "active" {
active = [
"NexusAuthenticatingRealm",
"OciBearerToken",
]
}
-33
View File
@@ -1,33 +0,0 @@
terraform {
required_version = ">= 1.11.0"
required_providers {
sonatyperepo = {
source = "sonatype-nexus-community/sonatyperepo"
version = "1.17.0"
}
}
}
provider "sonatyperepo" {
url = var.nexus_url
username = var.nexus_username
password = var.nexus_password
}
variable "nexus_url" {
description = "Nexus Repository base URL"
type = string
}
variable "nexus_username" {
description = "Nexus Terraform management username"
type = string
sensitive = true
}
variable "nexus_password" {
description = "Nexus Terraform management password"
type = string
sensitive = true
}
+4 -28
View File
@@ -70,11 +70,11 @@ configFiles:
}, },
"accessControl": { "accessControl": {
"repositories": { "repositories": {
"panxiao81/backstage": { "panxiao81/gitea-dynamic-runner-controller": {
"policies": [ "policies": [
{ {
"users": [ "users": [
"spiffe://ddupan.top/ci/panxiao81/backstage/image", "spiffe://ddupan.top/ci/panxiao81/gitea-dynamic-runner/publish-images",
"spiffe://ddupan.top/dev/panxiao81" "spiffe://ddupan.top/dev/panxiao81"
], ],
"actions": [ "actions": [
@@ -88,42 +88,18 @@ configFiles:
"read" "read"
] ]
}, },
"panxiao81/gitea-dynamic-runner-controller": {
"policies": [
{
"users": [
"spiffe://ddupan.top/ci/panxiao81/gitea-dynamic-runner/publish-images"
],
"actions": [
"read",
"create",
"update"
]
},
{
"users": ["spiffe://ddupan.top/dev/panxiao81"],
"actions": ["read", "create", "update", "delete"]
}
],
"defaultPolicy": [
"read"
]
},
"panxiao81/gitea-dynamic-runner-runner": { "panxiao81/gitea-dynamic-runner-runner": {
"policies": [ "policies": [
{ {
"users": [ "users": [
"spiffe://ddupan.top/ci/panxiao81/gitea-dynamic-runner/publish-images" "spiffe://ddupan.top/ci/panxiao81/gitea-dynamic-runner/publish-images",
"spiffe://ddupan.top/dev/panxiao81"
], ],
"actions": [ "actions": [
"read", "read",
"create", "create",
"update" "update"
] ]
},
{
"users": ["spiffe://ddupan.top/dev/panxiao81"],
"actions": ["read", "create", "update", "delete"]
} }
], ],
"defaultPolicy": [ "defaultPolicy": [
+1 -4
View File
@@ -50,9 +50,6 @@ sudo k3s kubectl -n flux-system get gitrepositories,kustomizations
- VictoriaMetrics Operator 已固定现有 chart `0.66.2` 并完成分阶段 Flux HelmRelease - VictoriaMetrics Operator 已固定现有 chart `0.66.2` 并完成分阶段 Flux HelmRelease
接管;Metrics、Logs、Traces 与 Grafana 也已统一完成 Flux 接管; 接管;Metrics、Logs、Traces 与 Grafana 也已统一完成 Flux 接管;
- External Secrets Operator 已固定 chart `2.8.0` 并完成分阶段接管; - External Secrets Operator 已固定 chart `2.8.0` 并完成分阶段接管;
- SPIRE 已按 hardened chart 内部 fork `0.30.2-ddupan.1`(基于上游 `0.30.2`,SPIRE - SPIRE 已按官方 hardened chart `0.30.2`(SPIRE `1.15.3`)声明,使用共享
`1.15.3`)声明,使用共享
PostgreSQL 与独立 signing-key PVC;首次上线和 OpenBao JWT-SVID PoC 尚待合并后验证; PostgreSQL 与独立 signing-key PVC;首次上线和 OpenBao JWT-SVID PoC 尚待合并后验证;
- Nexus Repository CE POC 已加入 GitOps 声明,计划验证 Ansible Galaxy、Go Modules 与
OCI/BuildKit 缓存;尚未部署或完成现场验收,现有 zot 保持不变;
- root Kustomization 与所有 brownfield 子 Kustomization 继续保持 `prune: false`。 - root Kustomization 与所有 brownfield 子 Kustomization 继续保持 `prune: false`。
-22
View File
@@ -1,22 +0,0 @@
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
name: backstage
namespace: flux-system
spec:
dependsOn:
- name: envoy-gateway
- name: external-secrets
healthChecks:
- apiVersion: apps/v1
kind: Deployment
name: backstage
namespace: backstage
interval: 10m
path: ./apps/backstage
prune: true
sourceRef:
kind: GitRepository
name: flux-system
timeout: 5m
wait: true
+2 -3
View File
@@ -6,12 +6,11 @@ metadata:
spec: spec:
dependsOn: dependsOn:
- name: external-secrets - name: external-secrets
- name: nats
- name: spire - name: spire
interval: 10m interval: 10m
path: ./platform/dynamic-runner path: ./platform/dynamic-runner
# The runner backends are replaceable. Prune is required when a retired prune: false
# worker is removed from the component; otherwise it keeps consuming work.
prune: true
sourceRef: sourceRef:
kind: GitRepository kind: GitRepository
name: flux-system name: flux-system
@@ -1,15 +1,14 @@
---
apiVersion: kustomize.toolkit.fluxcd.io/v1 apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization kind: Kustomization
metadata: metadata:
name: spire-bootstrap name: gitea-actions
namespace: flux-system namespace: flux-system
spec: spec:
interval: 10m interval: 10m
path: ./platform/sandbox-spire/bootstrap path: ./platform/gitea-runner
prune: true prune: false
sourceRef: sourceRef:
kind: GitRepository kind: GitRepository
name: flux-system name: flux-system
timeout: 10m timeout: 3m
wait: true wait: false
-17
View File
@@ -1,17 +0,0 @@
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
name: hydra
namespace: flux-system
spec:
dependsOn:
- name: envoy-gateway
- name: external-secrets
interval: 10m
path: ./apps/hydra
prune: false
sourceRef:
kind: GitRepository
name: flux-system
timeout: 5m
wait: true
-22
View File
@@ -1,22 +0,0 @@
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
name: nexus
namespace: flux-system
spec:
dependsOn:
- name: envoy-gateway
- name: openebs
healthChecks:
- apiVersion: apps/v1
kind: Deployment
name: nexus
namespace: nexus
interval: 10m
path: ./apps/nexus
prune: false
sourceRef:
kind: GitRepository
name: flux-system
timeout: 15m
wait: true
+1 -3
View File
@@ -7,7 +7,7 @@ resources:
- apps/envoy-gateway.yaml - apps/envoy-gateway.yaml
- apps/external-secrets.yaml - apps/external-secrets.yaml
- apps/gitea.yaml - apps/gitea.yaml
- apps/backstage.yaml - apps/gitea-actions.yaml
- apps/http-echo.yaml - apps/http-echo.yaml
- apps/openebs.yaml - apps/openebs.yaml
- apps/nats.yaml - apps/nats.yaml
@@ -15,5 +15,3 @@ resources:
- apps/spire.yaml - apps/spire.yaml
- apps/observability.yaml - apps/observability.yaml
- apps/zot.yaml - apps/zot.yaml
- apps/nexus.yaml
- apps/hydra.yaml
-72
View File
@@ -1,72 +0,0 @@
# Sandbox 集群
这里是 OpenSandbox、CI 和 AI Agent workload 所在双节点 k3s 集群的 Flux
reconciliation 入口。LXC、PostgreSQL、K3s、固定版本的 Flux controllers 与 root
sync 由 `infrastructure/sandbox-cluster/` 中的 Ansible 管理;本目录只组合集群内
workload。
Flux 通过 `https://git.ddupan.top/panxiao81/homelab-infra.git` 读取公开仓库。
Ansible 将 homelab CA 注入 `GitRepository/flux-system` 引用的同名 Secret,不使用
长期 Git 凭据。root Kustomization 从 `./clusters/sandbox` 开始 reconciliation,
初始保持 `prune: false`。
Root bootstrap 已完成。后续按依赖顺序分别引入:
1. 监控 CRD、kube-state-metrics 以及 kubelet/cAdvisor 抓取配置;
2. SPIRE Agent、SPIFFE CSI Driver 与 workload registration;
3. Kata Containers、`block-plain` RuntimeClass;
4. 独立 External Secrets Operator 与 sandbox 专用 OpenBao auth backend;
5. OpenSandbox controller/server;CI Pool 与 runner 调度器随后独立接入。
每一阶段单独合并并等待对应 Flux Kustomization Ready,不在 bootstrap 时一次性部署。
第一阶段监控拆为 `monitoring-operator` 与依赖它的 `monitoring`,防止 VM CR 在
VictoriaMetrics Operator CRD Ready 前进入 reconciliation。
SPIRE 阶段先由 `spire-bootstrap` 安装 CRD,并声明按上游 k8s_psat Server plugin
要求收窄的 reviewer:它可以调用 TokenReview,并只读查询用于证明的 Pod 与 Node。
Agent ServiceAccount 留给后续 HelmRelease 创建,避免两个声明方争夺同一资源。随后运行
`infrastructure/sandbox-cluster/ansible/spire-bootstrap.yml`:playbook 从 sandbox
读取 reviewer token,在内存中组成受限 kubeconfig,再通过 stdin reconcile 到 central
集群的 `spire-server/spire-external-kubeconfigs` Secret。凭据不写入仓库、日志或控制机
文件;该 Secret 准备完成后,才能启用 central external PSAT/controller-manager 和
sandbox Agent/CSI。
External controller-manager 使用独立的 `spire-controller-manager` ServiceAccount;其
RBAC 与上游 controller-manager 所需权限一致,用于读取 workload selectors、维护
SPIFFE CR status/finalizer 和 leader election。它不复用只允许 TokenReview 的 Server
reviewer。Ansible 将两份 kubeconfig 写入同一个 central Secret 的不同 key,便于 central
chart 分别绑定 `sandbox` 与 `sandbox-controller`。
Central SPIRE Server 通过内网 `spire-server.ad.ddupan.top:8081` 接收 sandbox Agent
attestation。Server 使用 external bundle publisher 持续维护 sandbox
`spire-system/spire-bundle`,Agent 不固定或复制 trust bundle。Sandbox HelmRelease
显式关闭 Server 与 OIDC Provider,只部署 Agent DaemonSet 和 SPIFFE CSI Driver;因此
不会产生第二个 trust root。
`spire-smoke` namespace、ServiceAccount 和 `sandbox-spire-smoke` ClusterSPIFFEID 只用于
普通 Pod 的 CSI 回归夹具,稳定身份为 `spiffe://ddupan.top/sandbox/smoke`。Kata guest
不能复用 node Agent 暴露的 Unix socket;virtio-fs 只能呈现 socket 路径,不能把连接
跨过 VM 边界。Kata workload 必须使用 guest 内 Agent,具体约束见
`platform/sandbox-kata/README.md`。测试 Pod 临时创建并在验收后删除,普通 Pod 的身份
声明保留。
Kata 阶段使用官方 4.1.0 `kata-deploy` chart 的短生命周期 `job` 模式,逐节点安装并
重启 K3s。只启用 `kata-clh-runtime-rs`,不创建默认 `kata` 别名;该 handler 的
`emptyDir` 固定使用 `block-plain`,为 Docker/BuildKit overlay2 与 kind 提供 guest
内块设备文件系统。详细限制与上线验收见 `platform/sandbox-kata/README.md`。
Sandbox 的 ESO 通过独立 `auth/kubernetes-sandbox` 向 OpenBao 证明 ServiceAccount
身份,只能读取共享的 `kv/k8s/opensandbox-api`。它不保存 reviewer JWT 或长期 Bao token;相关
Terraform 与 Flux 边界见 `platform/sandbox-external-secrets/README.md`。
OpenSandbox 阶段固定官方源码 commit 与 umbrella chart `0.2.2`,只部署 controller、
ClusterIP server 和 CRD。API key 由 ESO 从 OpenBao 投影,明文不进入 Git。
## 监控边界
这里只管理 sandbox LXC 内的 Kubernetes 监控,不负责 PVE 宿主监控。LXC 与宿主共享
内核,即使 lxcfs 虚拟化了内存和 uptime,容器内 `/proc/stat` 仍是宿主 CPU 视图;
在 LXC 内运行 node_exporter 会生成混合语义并重复采集宿主指标,因此禁止部署。
Sandbox 节点与 workload 指标来自 kubelet/cAdvisor 和 kube-state-metrics;K3s 或 LXC
特有但上述接口未覆盖的指标,应使用目标明确的 collector,不以 node_exporter 补齐。
-18
View File
@@ -1,18 +0,0 @@
---
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
name: ci-runners
namespace: flux-system
spec:
dependsOn:
- name: opensandbox
- name: spire-agents
interval: 10m
path: ./platform/sandbox-ci-runners
prune: true
sourceRef:
kind: GitRepository
name: flux-system
timeout: 20m
wait: true
@@ -1,17 +0,0 @@
---
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
name: external-secrets-operator
namespace: flux-system
spec:
dependsOn:
- name: monitoring-operator
interval: 10m
path: ./platform/sandbox-external-secrets/operator
prune: true
sourceRef:
kind: GitRepository
name: flux-system
timeout: 10m
wait: true
@@ -1,17 +0,0 @@
---
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
name: external-secrets
namespace: flux-system
spec:
dependsOn:
- name: external-secrets-operator
interval: 10m
path: ./platform/sandbox-external-secrets/config
prune: true
sourceRef:
kind: GitRepository
name: flux-system
timeout: 10m
wait: true
-18
View File
@@ -1,18 +0,0 @@
---
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
name: kata
namespace: flux-system
spec:
dependsOn:
- name: monitoring-operator
- name: spire-agents
interval: 10m
path: ./platform/sandbox-kata
prune: true
sourceRef:
kind: GitRepository
name: flux-system
timeout: 35m
wait: true
@@ -1,15 +0,0 @@
---
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
name: monitoring-operator
namespace: flux-system
spec:
interval: 10m
path: ./platform/sandbox-monitoring/operator
prune: true
sourceRef:
kind: GitRepository
name: flux-system
timeout: 10m
wait: true
-17
View File
@@ -1,17 +0,0 @@
---
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
name: monitoring
namespace: flux-system
spec:
dependsOn:
- name: monitoring-operator
interval: 10m
path: ./platform/sandbox-monitoring/workloads
prune: true
sourceRef:
kind: GitRepository
name: flux-system
timeout: 10m
wait: true
-19
View File
@@ -1,19 +0,0 @@
---
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
name: opensandbox
namespace: flux-system
spec:
dependsOn:
- name: external-secrets
- name: kata
- name: monitoring-operator
interval: 10m
path: ./platform/sandbox-opensandbox
prune: true
sourceRef:
kind: GitRepository
name: flux-system
timeout: 15m
wait: true
-18
View File
@@ -1,18 +0,0 @@
---
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
name: spire-agents
namespace: flux-system
spec:
dependsOn:
- name: spire-bootstrap
- name: monitoring-operator
interval: 10m
path: ./platform/sandbox-spire/agents
prune: true
sourceRef:
kind: GitRepository
name: flux-system
timeout: 15m
wait: true
-13
View File
@@ -1,13 +0,0 @@
---
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- apps/monitoring-operator.yaml
- apps/monitoring.yaml
- apps/spire-bootstrap.yaml
- apps/spire-agents.yaml
- apps/kata.yaml
- apps/external-secrets-operator.yaml
- apps/external-secrets.yaml
- apps/opensandbox.yaml
- apps/ci-runners.yaml
-8
View File
@@ -15,21 +15,13 @@ homelab_dns:
- { zone: ad.ddupan.top, name: sandbox-k8s, type: A, values: [10.60.0.13] } - { zone: ad.ddupan.top, name: sandbox-k8s, type: A, values: [10.60.0.13] }
- { zone: ad.ddupan.top, name: retrolab, type: A, values: [10.60.0.10] } - { zone: ad.ddupan.top, name: retrolab, type: A, values: [10.60.0.10] }
- { zone: ad.ddupan.top, name: grafana, type: A, values: [192.168.10.127] } - { zone: ad.ddupan.top, name: grafana, type: A, values: [192.168.10.127] }
- { zone: ad.ddupan.top, name: metrics-write, type: A, values: [192.168.10.127] }
- { zone: ad.ddupan.top, name: netbox, type: A, values: [192.168.10.127] } - { zone: ad.ddupan.top, name: netbox, type: A, values: [192.168.10.127] }
- { zone: ad.ddupan.top, name: nats, type: A, values: [192.168.10.127] } - { zone: ad.ddupan.top, name: nats, type: A, values: [192.168.10.127] }
- { zone: ad.ddupan.top, name: nexus, type: A, values: [192.168.10.127] }
- { zone: ad.ddupan.top, name: s3, type: A, values: [192.168.10.127] } - { zone: ad.ddupan.top, name: s3, type: A, values: [192.168.10.127] }
- { zone: ad.ddupan.top, name: hydra, type: A, values: [192.168.10.127] }
- { zone: ad.ddupan.top, name: hydra-login, type: A, values: [192.168.10.127] }
- { zone: ad.ddupan.top, name: spire-oidc, type: A, values: [192.168.10.127] } - { zone: ad.ddupan.top, name: spire-oidc, type: A, values: [192.168.10.127] }
- { zone: ad.ddupan.top, name: spire-server, type: A, values: [192.168.10.127] }
- { zone: ad.ddupan.top, name: zot, type: A, values: [192.168.10.127] } - { zone: ad.ddupan.top, name: zot, type: A, values: [192.168.10.127] }
- { zone: ad.ddupan.top, name: zot-push, type: A, values: [192.168.10.127] } - { zone: ad.ddupan.top, name: zot-push, type: A, values: [192.168.10.127] }
- { zone: ad.ddupan.top, name: pg-prod, type: A, values: [192.168.10.2] }
- { zone: ad.ddupan.top, name: pg-dev, type: A, values: [192.168.10.127] }
split_horizon: split_horizon:
# backends records the current adoption boundary. obj is deliberately not # backends records the current adoption boundary. obj is deliberately not
# emitted to CoreDNS yet, preserving the current pod resolver behaviour. # emitted to CoreDNS yet, preserving the current pod resolver behaviour.
-6
View File
@@ -1,6 +0,0 @@
.terraform/
*.tfstate*
*.tfplan
*.tfvars
!*.tfvars.example
__pycache__/
-163
View File
@@ -1,163 +0,0 @@
# Homelab shared etcd
独立于 PostgreSQL 与 k3s 的三成员协调服务。Ansible 管主机、证书文件和 etcd 账号;
独立 Terraform root 管现有 Bao PKI 下的签发角色与 policy,不管理 CA 或秘密值。
旧 Ansible Vault 不在本次迁移范围。
状态:2026-09-25 已部署三成员 etcd,启用 Bao mTLS 和 RBAC;三个端点健康检查通过。
PVE 两个 LXC 的 rootfs 已通过 `pct move-volume` 从 local-lvm 迁入 `pve-rg` SSD DRBD 池,
保留容器与 etcd 数据。当前 PVE LXC 迁卷要求停止容器,维护入口按成员串行执行,回归健康后再处理下一个。
| 成员 | 承载 | 地址 | 资源 |
| --- | --- | --- | --- |
| etcd-laptop | laptop systemd | 192.168.10.127 | MemoryHigh 256 MiB / MemoryMax 384 MiB |
| etcd-pve1 | pve1 新无特权 LXC 150 | 10.60.0.20 | 1 core / 1536 MiB / 8 GiB SSD DRBD |
| etcd-pve2 | pve2 新无特权 LXC 151 | 10.60.0.21 | 1 core / 768 MiB / 8 GiB SSD DRBD |
资源限额是初始测试预算,不是已测容量。LXC 已增加 PG standby/备份的独立 HDD mp0 与内存上限,生产 PG standby 与备份仓库已启动。
etcd 自身的 MemoryHigh/MemoryMax 仍为 256/384 MiB。labnet 依赖现有 VyOS 路由;切换 bare-metal 时复用服务角色,更换主机 inventory。
## 接入方式与认证
客户端使用三个 `https://地址:2379` endpoint,校验 Bao 中央 CA,出示独立客户端证书。
成员间 2380 也开启 mTLS,额外限制 peer CN 为 `homelab-etcd-peer`。
管理员证书 CN 为 `root`,只保存在受管成员 root 可读的文件中;对应 etcd root 用户不设置密码。
服务进程只能读取自己的 server/peer 私钥,不能读取管理员私钥。
Patroni 走 etcd3 gateway,使用无 CN 的客户端证书及独立 username/password 完成 RBAC。
实测带 CN 的证书会被 gateway 拒绝;管理员 CN=root 证书仅用于原生 etcdctl。
成员另有无 CN 的 gateway 证书供内部 gateway→gRPC 连接及密码核对使用,不能将管理员证书复用到 HTTP API。首个消费者为
`patroni-pg-prod`,只允许 `/homelab/patroni/pg-prod/`。其 Bao KV v2 路径为
`kv/infra/etcd/consumers/patroni-pg-prod`,包含 username/password/prefix;不在本文复制实际值。
配置 Patroni 时对应 `namespace: /homelab/patroni/` 与 `scope: pg-prod`。
第一次接入应由管理流程交付 CA、客户端证书及秘密引用,然后使用自己的身份对自己的 prefix
执行一次 put/get/delete。跨 prefix 应被拒绝;不要把 root 证书复制给应用。
生产消费者证书已签发,Patroni 已接入并验证主从自动切换。
## 安装与维护入口
需要 Ansible、community.crypto,以及目标机 Python/systemd。当前二进制固定为 etcd 3.7.2
linux-amd64,并固定官方发布资产 SHA-256;版本升级必须单独评估兼容性、备份与 quorum。
Terraform 使用已登录的 Vault 兼容 provider 身份;Ansible 读取控制端 `BAO_TOKEN` 环境变量,
token 不下发目标机。`homelab-etcd-provisioner` policy 不自动绑定到现有身份,先由 Bao 管理流程授权。
```bash
cd infrastructure/etcd/terraform
terraform init
terraform plan
# 审阅计划后 apply;只新增此服务的 PKI roles/policies。
cd ../ansible
ansible-galaxy collection install -r requirements.yml
ansible-playbook lxc.yml --check
ansible-playbook lxc.yml
ansible-playbook site.yml
ansible-playbook bootstrap-auth.yml -e etcd_bootstrap_auth=true
ansible-playbook consumers.yml
ansible-playbook verify.yml
```
Terraform 使用现有 S3 tfstate bucket,独立 key `etcd/terraform.tfstate`,启用原生 lockfile;不复用 Bao 的 state。
从仓库根目录运行 `python3 infrastructure/etcd/run.py terraform <子命令>` 或
`python3 infrastructure/etcd/run.py ansible <playbook.yml>`,复用当前 Bao 会话并在内存中取得所需凭据。
`lxc.yml` 是创建入口,不接管已有未知 VMID,也不自动调整运行中容器的网络与容量;配置漂移会报错。
部署前核对模板与 `pve-rg` 存储可用性、IP 冲突、SSH host key、laptop sudo 和网络访问控制。
数据库角色不会创建/删除这些 LXC 或 etcd 成员。
`site.yml` 管安装、CSR 本地生成、Bao 签发、配置和逐成员激活。全体健康检查通过才记录激活指纹;
上次中断后重跑仍能识别尚未激活的磁盘配置。已有集群每个成员激活后检查健康,再处理下一个。
首次建群先让三个服务启动,再检查 quorum。`bootstrap-auth.yml` 与日常收敛分开,认证已启用时不重建。
首次认证启用前应限制客户端网络,不能把“已有中央 CA 签名”当作业务授权。
`consumers.yml` 首次生成 48 字符随机密码并以 KV v2 CAS=0 写入,之后只复用。
元数据存在但数据被删除、Bao 403/超时、或 etcd 用户存在但秘密丢失时均失败,不自动换密码。
并发 CAS 冲突会中止此次执行,重跑读取胜出的版本。既有额外角色或不同 prefix 权限也报错,
不默默扩大权限。账号删除、prefix 迁移和密码轮换均不包含在普通收敛里。
密码使用 stdin 送给 etcdctl,涉及秘密的任务 `no_log: true`;禁止用 `--diff` 打印未来消费者秘密文件。
证书有效期 60 天,`site.yml` 在剩余不足 14 天时续签;需要 Bao 的控制端身份。
续签 IaC 已实现于 `terraform/renewal.tf`、`controller/` 和 `ansible/controller.yml`:
独立 cert auth 挂载信任中央 CA,并限制 laptop 的 DNS SAN;用现有 peer 客户端证书登录,
换取 10 分钟 token,仅允许签发 server/peer/admin/gateway 四类证书,不授予 KV 读取权限。
控制端每日运行现有健康检查与串行部署,结束撤销 token;不保存长期 token,不依赖人工 OIDC 会话。
控制程序安装为 root 管理的固定副本,以 panxiao81 身份复用现有 Ansible/SSH/sudo 环境。
**机器身份和 timer 已启用**(2026-09-25):维护者恢复 OIDC 登录后,Terraform 新增三个
续签资源,`controller.yml` 登录预检通过。首次实际运行三成员均 `changed=0`,服务 Result=success。
Bao cert auth 需要非空 CN 作为 identity alias,不能使用无 CN 的 gateway 证书;因此登录使用
同一主机已有 peer 证书,仍由 `allowed_dns_sans=etcd-laptop` 限制身份,未扩大到其他成员。
手动触发 `systemctl start homelab-etcd-renew.service`,检查 journal 和
`/var/lib/homelab-etcd-controller/last-success`;每日 timer 04:10 UTC 加随机延迟。
本流程也续签自身登录所用 peer 证书;若超过有效期仍未修复,则需要管理身份重新签发。
成员启动本身仅用本地证书,不实时依赖 Bao。
## 备份与故障处理
每个成员有每日一次的 `homelab-etcd-snapshot.timer`,快照保存在各自
`/var/backups/homelab-etcd`,root-only,保留最近 3 个成功快照;snapshot status 验证成功才淘汰旧快照。
首备可执行 `systemctl start homelab-etcd-snapshot.service`,用 journal 和 `etcdutl snapshot status`
检查,不根据文件名推断备份成功。三处本地快照不等于异地备份。
quota 初始 256 MiB,按 1 小时保留进行自动 compaction;defrag 另择维护窗口逐成员做,
不能三个成员同时执行。metrics 绑定 loopback 与成员内网地址的 2381 端口,独立 listener 不提供 KV API;
与既有 LAN exporter 一样通过内网 HTTP 采集,禁止将端口映射到公网。
现有 VictoriaMetrics 使用 `platform/observability/metrics/scrapes/shared-etcd.yaml` 采集,
对应 VMRule 覆盖成员不可采集、少于两个可采集成员、无 leader、容量、WAL fsync 和频繁选举。
采集失败不等于 quorum 丢失,需结合管理员 endpoint health 判断。
备份年龄、证书到期与续签任务告警尚待补齐。既有 Alertmanager 已接入 Telegram,
维护者已收到本次成员无 leader 与频繁选举通知;接收器配置由现有监控栈维护。
### DRBD I/O 故障导致成员根卷只读
`SharedEtcdNoLeader` 是单成员指标,不等于全体没有 leader。先检查三个 endpoint,
并检查宿主机内核的 DRBD quorum、PingAck、I/O error 和 ext4 journal 日志。
`findmnt` 显示 `rw,emergency_ro` 仍表示文件系统已因错误停止写入;DRBD 恢复 quorum 不会自动修复它。
2026-09-25 17:16 UTC 起,CT150 根卷发生 quorum 短暂丢失、journal I/O 错误和 emergency_ro,
另外两个 etcd 成员保持健康。期间两个新 HDD 数据卷初始同步伴随多个 DRBD 资源 PingAck 超时。
只对本项目两个新数据卷设置 `DrbdOptions/PeerDevice/c-max-rate=10240`(10 MiB/s),
入口为 `../shared-postgresql/ansible/limit-resync.yml`,创建卷流程也复用该任务。
限速后短期未见新增超时,尚不构成唯一根因证明;未调整全局协议、quorum 或超时。
恢复顺序:确认另外两个成员健康并保存快照,正常停止故障容器,确认根卷卸载、DRBD quorum
和副本 UpToDate,再执行 `pct fsck 150 --device rootfs --force 1`。fsck 返回 1 表示已修复,
PVE 包装命令仍会报非零;需再次检查返回 0 才启动容器。不能在已挂载卷上执行 fsck,不能直接强制 remount。
本次修复后 CT150 启动,三个 endpoint 健康且 Raft term/index 一致;两个容器根卷与 mp0 均可写。
频繁选举告警含 15 分钟历史窗口,应核对最新计数和健康状态,不通过关闭规则消除通知。
- 单成员故障:先确认其他两成员仍健康,修复原成员;普通部署不删数据目录。
- 成员永久丢失:需单独执行 member remove/add 流程并同步 inventory,不用重新 init 覆盖。
- 全集群丢失:先恢复 Bao/信任根与恢复凭据,隔离全部旧成员,用同一有效快照恢复新逻辑集群。
开放客户端前逐个协调消费者状态。Patroni 必须核对真实主库、timeline、lease/DCS 状态;
不因旧快照宣称某节点为主就允许它写入。完整跨服务灾难恢复演练尚未完成。
## 验证
```bash
# 本机 loopback 三节点,临时测试 CA 与假的 Bao HTTP 服务,不接触线上秘密。
ETCD_TEST_BIN=/path/to/etcd-v3.7.2-linux-amd64 python3 tests/integration.py
cd terraform && terraform validate
cd ../ansible && ansible-playbook --syntax-check site.yml
```
集成测试验证实际模板启动、mTLS、认证初始化幂等、消费者幂等、CAS 首次创建、秘密缺失失败关闭、
prefix 隔离、gateway 账号登录和单成员故障。假 Bao 不验证真实 PKI 签发或中央 policy 的权限,
也不证明 PVE SSD DRBD 在业务负载下的延迟、systemd 内存预算、在线证书轮换和生产网络符合要求。
源码边界与 PostgreSQL 后续设计见 [研究记录](../shared-postgresql/RESEARCH.md)。
2026-09-25 验证结果:Terraform validate 通过;Ansible lint 零文件级问题;隔离三成员测试通过,
认证初始化与消费者第二次执行均 `changed=0`,包含 gateway 写入、密码漂移拒绝、快照离线恢复、
停止一成员后继续写入。测试低负载 RSS 约 36.6–40.4 MiB/成员,仅作开销参考。
现场验证:Bao PKI roles/policies 已应用,三成员已签发证书并启动;认证初始化与
`patroni-pg-prod` 账号创建成功,随机密码已保存 Bao,实际 gateway 登录验证通过。
两台 LXC 的 rootfs 均为 `pve-rg`,运行状态正常;迁移后全部端点成功提交健康探测。
现有监控已接入;自动续签已启用;备份/证书年龄告警尚待补齐。PostgreSQL 部署状态见其 README。
监控接入验收(2026-09-25):三个 `up{job="shared-etcd"}` 均为 1,六条规则 health=ok,
三端点健康检查通过。新增续签调度四项失败/成功路径测试通过,Terraform validate 与 Ansible lint 通过,
隔离三成员测试再次通过。[IaC PR #159](https://git.ddupan.top/panxiao81/homelab-infra/pulls/159) 已合并。
2026-09-25 合并后核实:Flux observability Ready,应用版本 `f6d12d6`,
etcd/PG 的两份 VMRule 与两份 VMStaticScrape 均已纳入 inventory,状态 operational;
三个 etcd 与两个 Patroni 抓取目标均 up=1,相关 11 条告警 health=ok、inactive。备份/证书年龄告警不包含在这六条规则中。
-6
View File
@@ -1,6 +0,0 @@
[defaults]
inventory = inventory/hosts.yml
roles_path = roles
retry_files_enabled = False
host_key_checking = True
interpreter_python = auto_silent
@@ -1,60 +0,0 @@
---
# 与常规 site 分离,防止重建/恢复时默默创建新的认证域。
- name: 初始化共享 etcd 认证
hosts: etcd[0]
become: true
gather_facts: false
pre_tasks:
- name: 加载共享默认参数
ansible.builtin.import_role:
name: shared_etcd
tasks_from: context
environment:
ETCDCTL_ENDPOINTS: "https://{{ etcd_address }}:{{ etcd_client_port }}"
ETCDCTL_CACERT: "{{ etcd_config_dir }}/ca.crt"
ETCDCTL_CERT: "{{ etcd_config_dir }}/admin.crt"
ETCDCTL_KEY: "{{ etcd_config_dir }}/admin.key"
tasks:
- name: 读取认证状态
ansible.builtin.command:
argv: ["{{ etcd_install_dir }}/etcdctl", --write-out=json, auth, status]
changed_when: false
register: etcd_auth_status
check_mode: false
- name: 初始化管理员与认证
when: not ((etcd_auth_status.stdout | from_json).enabled | default(false))
block:
- name: 要求显式初始化参数
ansible.builtin.assert:
that: etcd_bootstrap_auth | default(false) | bool
fail_msg: 首次初始化需要 -e etcd_bootstrap_auth=true;常规运行不能重建认证。
- name: 读取现有用户
ansible.builtin.command:
argv: ["{{ etcd_install_dir }}/etcdctl", --write-out=json, user, list]
changed_when: false
register: etcd_users
- name: 创建仅证书认证的 root 用户
ansible.builtin.command:
argv: ["{{ etcd_install_dir }}/etcdctl", user, add, root, --no-password]
when: "'root' not in ((etcd_users.stdout | from_json).users | default([], true))"
changed_when: true
- name: 读取管理员角色
ansible.builtin.command:
argv: ["{{ etcd_install_dir }}/etcdctl", --write-out=json, user, get, root]
changed_when: false
register: etcd_root_roles
- name: 授予 root 管理角色
ansible.builtin.command:
argv: ["{{ etcd_install_dir }}/etcdctl", user, grant-role, root, root]
when: "'root' not in ((etcd_root_roles.stdout | from_json).roles | default([], true))"
changed_when: true
- name: 开启认证
ansible.builtin.command:
argv: ["{{ etcd_install_dir }}/etcdctl", auth, enable]
changed_when: true
-37
View File
@@ -1,37 +0,0 @@
---
- name: 收敛消费者账号及独立 prefix
hosts: etcd[0]
become: true
gather_facts: false
pre_tasks:
- name: 加载共享默认参数
ansible.builtin.import_role:
name: shared_etcd
tasks_from: context
environment:
ETCDCTL_ENDPOINTS: "https://{{ etcd_address }}:{{ etcd_client_port }}"
ETCDCTL_CACERT: "{{ etcd_config_dir }}/ca.crt"
ETCDCTL_CERT: "{{ etcd_config_dir }}/admin.crt"
ETCDCTL_KEY: "{{ etcd_config_dir }}/admin.key"
tasks:
- name: 检查认证状态
ansible.builtin.command:
argv: ["{{ etcd_install_dir }}/etcdctl", --write-out=json, auth, status]
register: etcd_auth_status
changed_when: false
check_mode: false
- name: 要求认证已启用
ansible.builtin.assert:
that:
- ((etcd_auth_status.stdout | from_json).enabled | default(false))
- etcd_consumers | map(attribute='name') | unique | length == etcd_consumers | length
- etcd_bao_token | length > 0
no_log: true
- name: 逐个收敛消费者
ansible.builtin.include_tasks: tasks/consumer.yml
loop: "{{ etcd_consumers }}"
loop_control:
loop_var: etcd_consumer
label: "{{ etcd_consumer.name }}"
@@ -1,74 +0,0 @@
---
# 仅在 Terraform 身份配置完成后部署;预检失败不启用定时器。
- name: 安装独立于人工会话的续签调度
hosts: etcd-laptop
become: true
gather_facts: false
tasks:
- name: 创建 root 管理的程序目录
ansible.builtin.file:
path: /opt/homelab-etcd-controller
state: directory
mode: '0755'
- name: 安装证书登录与调度程序
ansible.builtin.copy:
src: "../controller/{{ item }}"
dest: "/opt/homelab-etcd-controller/{{ item }}"
owner: root
group: root
mode: '0755'
loop: [login.py, renew.py]
- name: 以机器身份预检(不输出 token)
ansible.builtin.command:
argv: [/usr/bin/python3, /opt/homelab-etcd-controller/login.py]
changed_when: false
no_log: true
- name: 安装固定的 Ansible 配置副本
ansible.builtin.copy:
src: "{{ playbook_dir }}/"
dest: /opt/homelab-etcd-controller/ansible/
owner: root
group: root
mode: preserve
- name: 创建调度状态目录
ansible.builtin.file:
path: /var/lib/homelab-etcd-controller
state: directory
owner: panxiao81
group: panxiao81
mode: '0700'
- name: 安装续签 oneshot unit
ansible.builtin.copy:
dest: /etc/systemd/system/homelab-etcd-renew.service
mode: '0644'
content: |
[Unit]
Description=Renew shared etcd certificates through Bao machine identity
After=network-online.target
[Service]
Type=oneshot
User=panxiao81
Environment=HOME=/home/panxiao81
Environment=PATH=/home/panxiao81/.local/bin:/usr/local/bin:/usr/bin:/bin
ExecStart=/usr/bin/python3 /opt/homelab-etcd-controller/renew.py
TimeoutStartSec=30min
UMask=0077
- name: 安装每日续签 timer
ansible.builtin.copy:
dest: /etc/systemd/system/homelab-etcd-renew.timer
mode: '0644'
content: |
[Unit]
Description=Daily shared etcd certificate renewal check
[Timer]
OnCalendar=*-*-* 04:10:00 UTC
RandomizedDelaySec=15min
Persistent=true
[Install]
WantedBy=timers.target
- name: 启用续签调度
ansible.builtin.systemd_service:
name: homelab-etcd-renew.timer
daemon_reload: true
enabled: true
state: started
@@ -1,20 +0,0 @@
---
etcd_bao_url: https://bao.ad.ddupan.top:8200
etcd_bao_pki_mount: pki
etcd_bao_kv_mount: kv
etcd_bao_secret_base: infra/etcd/consumers
# 只在控制端使用,不下发 Bao token。也不读取或复制历史 .vault_pass。
etcd_bao_token: "{{ lookup('env', 'BAO_TOKEN') }}"
etcd_consumers:
- name: patroni-pg-prod
prefix: /homelab/patroni/pg-prod/
etcd_lxc_template: laptop:vztmpl/ubuntu-24.04-standard_24.04-2_amd64.tar.zst
etcd_lxc_storage: pve-rg
etcd_lxc_bridge: labnet
etcd_lxc_gateway: 10.60.0.1
etcd_lxc_memory: 512
etcd_lxc_disk_gb: 8
etcd_lxc_pubkey: "{{ lookup('file', '~/.ssh/id_ed25519.pub') }}"
# 与现有 LAN exporter 相同的内网采集边界;禁止映射到公网。
etcd_metrics_urls: "http://127.0.0.1:2381,http://{{ etcd_address }}:2381"
@@ -1,33 +0,0 @@
---
# 已部署成员;共置 PG standby/备份的承载预算,数据库数据用独立 HDD mp0。
all:
vars:
ansible_user: root
children:
etcd:
hosts:
etcd-laptop:
ansible_connection: local
ansible_host: 192.168.10.127
ansible_user: panxiao81
etcd_address: 192.168.10.127
etcd-pve1:
ansible_host: 10.60.0.20
etcd_address: 10.60.0.20
etcd-pve2:
ansible_host: 10.60.0.21
etcd_address: 10.60.0.21
etcd_pve:
hosts:
pve1:
ansible_host: 192.168.10.4
etcd_lxc_vmid: 150
etcd_lxc_hostname: etcd-pve1
etcd_lxc_address: 10.60.0.20/24
etcd_lxc_memory: 1536
pve2:
ansible_host: 192.168.10.7
etcd_lxc_vmid: 151
etcd_lxc_hostname: etcd-pve2
etcd_lxc_address: 10.60.0.21/24
etcd_lxc_memory: 768
-126
View File
@@ -1,126 +0,0 @@
---
# 只创建已声明且尚不存在的 LXC;不接管未知 VMID,不重启现有容器。
- name: 创建独立 etcd LXC
hosts: etcd_pve
become: true
gather_facts: false
tasks:
- name: 读取全局资源,避免 VMID 在其他节点已占用
ansible.builtin.command:
argv: [pvesh, get, /cluster/resources, --type, vm, --output-format, json]
register: etcd_pve_resources
changed_when: false
check_mode: false
- name: 保存同号资源
ansible.builtin.set_fact:
etcd_lxc_existing: >-
{{ etcd_pve_resources.stdout
| from_json
| selectattr('vmid', 'equalto', etcd_lxc_vmid)
| list }}
- name: 拒绝接管未知资源
ansible.builtin.assert:
that:
- >-
etcd_lxc_existing | length == 0 or
(etcd_lxc_existing[0].type == 'lxc' and etcd_lxc_existing[0].node == inventory_hostname
and etcd_lxc_existing[0].name == etcd_lxc_hostname
and 'shared-etcd' in (etcd_lxc_existing[0].tags | default('')))
- name: 新建无特权 LXC
when: etcd_lxc_existing | length == 0 and not ansible_check_mode
block:
- name: 暂存 SSH 公钥
ansible.builtin.copy:
content: "{{ etcd_lxc_pubkey }}\n"
dest: /run/shared-etcd-bootstrap.pub
mode: '0600'
- name: 创建声明的容器
ansible.builtin.command:
argv:
- pct
- create
- "{{ etcd_lxc_vmid }}"
- "{{ etcd_lxc_template }}"
- --hostname
- "{{ etcd_lxc_hostname }}"
- --unprivileged
- '1'
- --cores
- '1'
- --memory
- "{{ etcd_lxc_memory }}"
- --swap
- '0'
- --rootfs
- "{{ etcd_lxc_storage }}:{{ etcd_lxc_disk_gb }}"
- --net0
- "name=eth0,bridge={{ etcd_lxc_bridge }},ip={{ etcd_lxc_address }},gw={{ etcd_lxc_gateway }},type=veth"
- --nameserver
- 192.168.10.5
- --searchdomain
- ad.ddupan.top
- --ssh-public-keys
- /run/shared-etcd-bootstrap.pub
- --onboot
- '1'
- --tags
- ansible;shared-etcd
changed_when: true
always:
- name: 删除暂存公钥
ansible.builtin.file:
path: /run/shared-etcd-bootstrap.pub
state: absent
- name: 读取容器配置
ansible.builtin.command:
argv: [pct, config, "{{ etcd_lxc_vmid }}"]
changed_when: false
register: etcd_lxc_config
when: not ansible_check_mode or etcd_lxc_existing | length > 0
- name: 配置漂移先报错,不直接改运行中的网络/资源
ansible.builtin.assert:
that:
- >-
('ip=' ~ etcd_lxc_address ~ ',') in etcd_lxc_config.stdout or
('ip=' ~ etcd_lxc_address ~ '\n') in etcd_lxc_config.stdout
- "('bridge=' ~ etcd_lxc_bridge ~ ',') in etcd_lxc_config.stdout"
- "'unprivileged: 1' in etcd_lxc_config.stdout"
- "('memory: ' ~ etcd_lxc_memory) in etcd_lxc_config.stdout"
- "('rootfs: ' ~ etcd_lxc_storage ~ ':') in etcd_lxc_config.stdout"
when: etcd_lxc_config is not skipped
- name: 读取容器运行状态
ansible.builtin.command:
argv: [pct, status, "{{ etcd_lxc_vmid }}"]
changed_when: false
register: etcd_lxc_status
when: not ansible_check_mode
- name: 启动容器
ansible.builtin.command:
argv: [pct, start, "{{ etcd_lxc_vmid }}"]
changed_when: true
when: not ansible_check_mode and 'running' not in etcd_lxc_status.stdout
- name: 通过可信宿主机取得容器 SSH 公钥
ansible.builtin.command:
argv: [pct, exec, "{{ etcd_lxc_vmid }}", --, cat, /etc/ssh/ssh_host_ed25519_key.pub]
register: etcd_lxc_hostkey
changed_when: false
retries: 12
delay: 5
until: etcd_lxc_hostkey.rc == 0
when: not ansible_check_mode
- name: 保存经宿主机验证的 SSH host key
ansible.builtin.known_hosts:
name: "{{ etcd_lxc_address.split('/')[0] }}"
key: "{{ etcd_lxc_address.split('/')[0] }} {{ etcd_lxc_hostkey.stdout }}"
delegate_to: localhost
become: false
when: not ansible_check_mode
@@ -1,90 +0,0 @@
---
# 首次部署承载调整,逐节点迁移;普通 lxc.yml 不自动移动磁盘。
- name: 逐个将新 etcd 容器迁入声明的 SSD 池
hosts: etcd_pve
become: true
gather_facts: false
serial: 1
any_errors_fatal: true
vars:
etcd_move_health_command:
- /opt/homelab-etcd/etcdctl
- --endpoints=https://192.168.10.127:2379,https://10.60.0.20:2379,https://10.60.0.21:2379
- --cacert=/etc/homelab-etcd/ca.crt
- --cert=/etc/homelab-etcd/admin.crt
- --key=/etc/homelab-etcd/admin.key
- endpoint
- health
tasks:
- name: 核对容器配置
ansible.builtin.command:
argv: [pct, config, "{{ etcd_lxc_vmid }}"]
register: etcd_move_config
changed_when: false
check_mode: false
- name: 限定本项目新建容器与允许的源池
ansible.builtin.assert:
that:
- etcd_lxc_vmid in [150, 151]
- "('hostname: ' ~ etcd_lxc_hostname) in etcd_move_config.stdout"
- "'shared-etcd' in etcd_move_config.stdout"
- "'rootfs: local-lvm:' in etcd_move_config.stdout or 'rootfs: pve-rg:' in etcd_move_config.stdout"
- etcd_lxc_storage == 'pve-rg'
- name: 验证迁移前全部端点健康
ansible.builtin.command:
argv: "{{ etcd_move_health_command }}"
delegate_to: localhost
changed_when: false
check_mode: false
- name: 迁移当前尚在本地池的根卷
when: "'rootfs: local-lvm:' in etcd_move_config.stdout and not ansible_check_mode"
block:
- name: 创建已验证的集群快照
ansible.builtin.command:
argv: [systemctl, start, homelab-etcd-snapshot.service]
delegate_to: localhost
changed_when: true
- name: 正常关闭一个容器
ansible.builtin.command:
argv: [pct, shutdown, "{{ etcd_lxc_vmid }}", --timeout, '60']
changed_when: true
- name: 复制成功后移除该新建容器的原卷
ansible.builtin.command:
argv:
- pct
- move-volume
- "{{ etcd_lxc_vmid }}"
- rootfs
- "{{ etcd_lxc_storage }}"
- --delete
- '1'
- --bwlimit
- '32768'
changed_when: true
always:
- name: 核对容器运行状态
ansible.builtin.command:
argv: [pct, status, "{{ etcd_lxc_vmid }}"]
register: etcd_move_status
changed_when: false
- name: 重新启动容器
ansible.builtin.command:
argv: [pct, start, "{{ etcd_lxc_vmid }}"]
when: "'running' not in etcd_move_status.stdout"
changed_when: true
- name: 等待当前成员回归
ansible.builtin.command:
argv: "{{ etcd_move_health_command }}"
delegate_to: localhost
changed_when: false
register: etcd_move_health
retries: 18
delay: 5
until: etcd_move_health.rc == 0
when: not ansible_check_mode
@@ -1,4 +0,0 @@
---
collections:
- name: community.crypto
version: 3.2.1
@@ -1,17 +0,0 @@
---
etcd_version: 3.7.2
etcd_archive_checksum: sha256:3a3679bc51a4ee9d30bccea1da7cd4fe62c6fc1d2ca1255068d2c53bf3026135
etcd_install_dir: /opt/homelab-etcd
etcd_config_dir: /etc/homelab-etcd
etcd_data_dir: /var/lib/homelab-etcd
etcd_cluster_token: homelab-shared-etcd-v1
etcd_client_port: 2379
etcd_peer_port: 2380
etcd_metrics_port: 2381
etcd_quota_bytes: 268435456
etcd_memory_high: 256M
etcd_memory_max: 384M
etcd_certificate_ttl: 1440h
etcd_renew_before: +14d
etcd_snapshot_dir: /var/backups/homelab-etcd
etcd_snapshot_keep: 3
@@ -1,93 +0,0 @@
---
- name: 在成员本地生成私钥
community.crypto.openssl_privatekey:
path: "{{ etcd_config_dir }}/{{ etcd_cert.name }}.key"
type: ECC
curve: secp256r1
owner: root
group: "{{ etcd_cert.group }}"
mode: '0640'
- name: 本地生成 CSR
community.crypto.openssl_csr:
path: "{{ etcd_config_dir }}/{{ etcd_cert.name }}.csr"
privatekey_path: "{{ etcd_config_dir }}/{{ etcd_cert.name }}.key"
common_name: "{{ etcd_cert.cn | default(omit, true) }}"
subject_alt_name: >-
{{ ['IP:' ~ etcd_address, 'DNS:' ~ inventory_hostname] if etcd_cert.name in ['server', 'peer']
else (['DNS:' ~ inventory_hostname] if etcd_cert.name == 'gateway' else []) }}
use_common_name_for_san: false
extended_key_usage: "{{ etcd_cert.eku }}"
key_usage: [digitalSignature]
mode: '0644'
register: etcd_csr_state
- name: 检查已有证书
ansible.builtin.stat:
path: "{{ etcd_config_dir }}/{{ etcd_cert.name }}.crt"
register: etcd_cert_file
- name: 检查续签窗口
community.crypto.x509_certificate_info:
path: "{{ etcd_config_dir }}/{{ etcd_cert.name }}.crt"
valid_at:
renewal: "{{ etcd_renew_before }}"
register: etcd_cert_info
when: etcd_cert_file.stat.exists
- name: 通过中央 CA 签发需更新的证书
when: >-
not etcd_cert_file.stat.exists or etcd_csr_state is changed or
not (etcd_cert_info.valid_at.renewal | default(false))
block:
- name: 仅在确需签发时要求 Bao 凭据
ansible.builtin.assert:
that: etcd_bao_token | length > 0
no_log: true
- name: 读取 CSR 公共内容
ansible.builtin.slurp:
src: "{{ etcd_config_dir }}/{{ etcd_cert.name }}.csr"
register: etcd_csr
- name: 控制端提交 Bao 签名请求
ansible.builtin.uri:
url: "{{ etcd_bao_url }}/v1/{{ etcd_bao_pki_mount }}/sign/{{ etcd_cert.role }}"
method: POST
headers:
X-Vault-Token: "{{ etcd_bao_token }}"
body_format: json
body:
csr: "{{ etcd_csr.content | b64decode }}"
ttl: "{{ etcd_certificate_ttl }}"
status_code: 200
delegate_to: localhost
become: false
register: etcd_signed
no_log: true
when: not ansible_check_mode
- name: 保存签发的证书链
ansible.builtin.copy:
content: |
{{ etcd_signed.json.data.certificate }}
{{ etcd_signed.json.data.ca_chain | join('\n') }}
dest: "{{ etcd_config_dir }}/{{ etcd_cert.name }}.crt"
owner: root
group: "{{ etcd_cert.group }}"
mode: '0644'
when: not ansible_check_mode
- name: 保存中央 CA 信任链
ansible.builtin.copy:
content: |
{{ etcd_signed.json.data.ca_chain | join('\n') }}
dest: "{{ etcd_config_dir }}/ca.crt"
owner: root
group: homelab-etcd
mode: '0644'
when: not ansible_check_mode
- name: 标记证书已更新
ansible.builtin.set_fact:
etcd_certificates_changed: true
@@ -1,3 +0,0 @@
---
# 只加载 role defaults,供运维入口复用。
[]
@@ -1,134 +0,0 @@
---
- name: 验证拓扑和签发配置
ansible.builtin.assert:
that:
- groups['etcd'] | length == 3
- groups['etcd'] | map('extract', hostvars, 'etcd_address') | unique | length == 3
- ansible_facts['architecture'] == 'x86_64'
- etcd_bao_url is match('^https://')
- etcd_archive_checksum is match('^sha256:[a-f0-9]{64}$')
no_log: true
- name: 安装证书处理依赖
ansible.builtin.package:
name: [python3-cryptography, openssl]
state: present
- name: 创建独立 etcd 组
ansible.builtin.group:
name: homelab-etcd
system: true
- name: 创建独立 etcd 用户
ansible.builtin.user:
name: homelab-etcd
group: homelab-etcd
system: true
shell: /usr/sbin/nologin
create_home: false
- name: 创建受管目录
ansible.builtin.file:
path: "{{ item.path }}"
state: directory
owner: "{{ item.owner }}"
group: homelab-etcd
mode: "{{ item.mode }}"
loop:
- {path: "{{ etcd_config_dir }}", owner: root, mode: '0750'}
- {path: "{{ etcd_data_dir }}", owner: homelab-etcd, mode: '0700'}
- {path: "{{ etcd_install_dir }}", owner: root, mode: '0755'}
- name: 下载固定版本及校验归档
ansible.builtin.get_url:
url: >-
https://github.com/etcd-io/etcd/releases/download/v{{ etcd_version }}/etcd-v{{ etcd_version }}-linux-amd64.tar.gz
dest: "{{ etcd_install_dir }}/etcd-v{{ etcd_version }}.tar.gz"
checksum: "{{ etcd_archive_checksum }}"
mode: '0644'
register: etcd_download
retries: 3
delay: 5
until: etcd_download is succeeded
- name: 展开固定版本
ansible.builtin.unarchive:
src: "{{ etcd_install_dir }}/etcd-v{{ etcd_version }}.tar.gz"
dest: "{{ etcd_install_dir }}"
remote_src: true
creates: "{{ etcd_install_dir }}/etcd-v{{ etcd_version }}-linux-amd64/etcd"
- name: 安装版本链接
ansible.builtin.file:
src: "{{ etcd_install_dir }}/etcd-v{{ etcd_version }}-linux-amd64/{{ item }}"
dest: "{{ etcd_install_dir }}/{{ item }}"
state: link
loop: [etcd, etcdctl, etcdutl]
register: etcd_binary_links
- name: 签发成员与管理员证书
ansible.builtin.include_tasks: certificate.yml
loop:
- {name: server, role: homelab-etcd-server, cn: "{{ inventory_hostname }}", eku: [serverAuth], group: homelab-etcd}
- {name: peer, role: homelab-etcd-peer, cn: homelab-etcd-peer, eku: [serverAuth, clientAuth], group: homelab-etcd}
- {name: gateway, role: homelab-etcd-gateway, cn: "", eku: [clientAuth], group: homelab-etcd}
- {name: admin, role: homelab-etcd-admin, cn: root, eku: [clientAuth], group: root}
loop_control:
loop_var: etcd_cert
- name: 写入独立 etcd 配置
ansible.builtin.template:
src: etcd.yml.j2
dest: "{{ etcd_config_dir }}/etcd.yml"
owner: root
group: homelab-etcd
mode: '0640'
register: etcd_config_file
- name: 写入独立 systemd unit
ansible.builtin.template:
src: homelab-etcd.service.j2
dest: /etc/systemd/system/homelab-etcd.service
mode: '0644'
register: etcd_unit
# 跨失败重跑记录激活状态,防止上一轮写文件后中断导致漏掉必要重启。
- name: 计算受管文件校验和
ansible.builtin.stat:
path: "{{ item }}"
checksum_algorithm: sha256
loop:
- "{{ etcd_config_dir }}/etcd.yml"
- "{{ etcd_config_dir }}/server.crt"
- "{{ etcd_config_dir }}/peer.crt"
- "{{ etcd_config_dir }}/gateway.crt"
- "{{ etcd_config_dir }}/ca.crt"
- /etc/systemd/system/homelab-etcd.service
register: etcd_managed_files
- name: 计算期望激活指纹
ansible.builtin.set_fact:
etcd_config_fingerprint: >-
{{ ((etcd_managed_files.results | map(attribute='stat.checksum') | list | join(':'))
~ ':' ~ etcd_version) | hash('sha256') }}
when: not ansible_check_mode
- name: 检查已激活指纹
ansible.builtin.stat:
path: "{{ etcd_config_dir }}/activated.sha256"
register: etcd_activated_file
- name: 读取已激活指纹
ansible.builtin.slurp:
src: "{{ etcd_config_dir }}/activated.sha256"
register: etcd_activated
when: etcd_activated_file.stat.exists
- name: 判断是否需要滚动激活
ansible.builtin.set_fact:
etcd_config_changed: >-
{{ not etcd_activated_file.stat.exists or
(etcd_activated.content | default('') | b64decode | trim) != etcd_config_fingerprint | default('check-mode') }}
- name: 管理本地快照任务
ansible.builtin.import_tasks: snapshot.yml
@@ -1,26 +0,0 @@
---
- name: 创建仅 root 可访问的快照目录
ansible.builtin.file:
path: "{{ etcd_snapshot_dir }}"
state: directory
owner: root
group: root
mode: '0700'
- name: 写入快照脚本和 systemd 任务
ansible.builtin.template:
src: "{{ item.src }}"
dest: "{{ item.dest }}"
mode: "{{ item.mode }}"
loop:
- {src: snapshot.sh.j2, dest: "{{ etcd_install_dir }}/snapshot", mode: '0700'}
- {src: snapshot.service.j2, dest: /etc/systemd/system/homelab-etcd-snapshot.service, mode: '0644'}
- {src: snapshot.timer.j2, dest: /etc/systemd/system/homelab-etcd-snapshot.timer, mode: '0644'}
register: etcd_snapshot_units
- name: 启用本地快照计划
ansible.builtin.systemd_service:
name: homelab-etcd-snapshot.timer
daemon_reload: "{{ etcd_snapshot_units is changed }}"
enabled: true
state: started
@@ -1,33 +0,0 @@
# Ansible 管理;与 k3s、数据库生命周期独立。
name: {{ inventory_hostname | to_json }}
data-dir: {{ etcd_data_dir | to_json }}
listen-client-urls: https://{{ etcd_address }}:{{ etcd_client_port }}
advertise-client-urls: https://{{ etcd_address }}:{{ etcd_client_port }}
listen-peer-urls: https://{{ etcd_address }}:{{ etcd_peer_port }}
initial-advertise-peer-urls: https://{{ etcd_address }}:{{ etcd_peer_port }}
initial-cluster: "{% for member in groups['etcd'] %}{{ member }}=https://{{ hostvars[member].etcd_address }}:{{ etcd_peer_port }}{{ ',' if not loop.last else '' }}{% endfor %}"
initial-cluster-token: {{ etcd_cluster_token | to_json }}
initial-cluster-state: new
# 已存在的数据目录优先;成员替换必须走单独 runbook,不删除数据重建。
client-transport-security:
cert-file: {{ etcd_config_dir }}/server.crt
key-file: {{ etcd_config_dir }}/server.key
client-cert-file: {{ etcd_config_dir }}/gateway.crt
client-key-file: {{ etcd_config_dir }}/gateway.key
trusted-ca-file: {{ etcd_config_dir }}/ca.crt
client-cert-auth: true
peer-transport-security:
cert-file: {{ etcd_config_dir }}/peer.crt
key-file: {{ etcd_config_dir }}/peer.key
trusted-ca-file: {{ etcd_config_dir }}/ca.crt
client-cert-auth: true
allowed-cn: [homelab-etcd-peer]
# 独立 metrics listener 仅提供指标/健康,不开放 KV API;只绑定受管内网地址。
listen-metrics-urls: {{ etcd_metrics_urls | default("http://127.0.0.1:" ~ etcd_metrics_port) | to_json }}
quota-backend-bytes: {{ etcd_quota_bytes }}
auto-compaction-mode: periodic
auto-compaction-retention: '1h'
heartbeat-interval: 100
election-timeout: 1000
logger: zap
log-level: info
@@ -1,23 +0,0 @@
[Unit]
Description=Homelab shared etcd
Wants=network-online.target
After=network-online.target
[Service]
User=homelab-etcd
Group=homelab-etcd
ExecStart={{ etcd_install_dir }}/etcd --config-file={{ etcd_config_dir }}/etcd.yml
Restart=on-failure
RestartSec=5
TimeoutStopSec=60
MemoryHigh={{ etcd_memory_high }}
MemoryMax={{ etcd_memory_max }}
UMask=0077
NoNewPrivileges=true
ProtectSystem=strict
ProtectHome=true
PrivateTmp=true
ReadWritePaths={{ etcd_data_dir }}
[Install]
WantedBy=multi-user.target
@@ -1,11 +0,0 @@
[Unit]
Description=Snapshot homelab shared etcd
After=homelab-etcd.service
[Service]
Type=oneshot
ExecStart={{ etcd_install_dir }}/snapshot
User=root
UMask=0077
TimeoutStartSec=300
Nice=10
@@ -1,20 +0,0 @@
#!/bin/bash
set -euo pipefail
umask 077
export ETCDCTL_ENDPOINTS="https://{{ etcd_address }}:{{ etcd_client_port }}"
export ETCDCTL_CACERT="{{ etcd_config_dir }}/ca.crt"
export ETCDCTL_CERT="{{ etcd_config_dir }}/admin.crt"
export ETCDCTL_KEY="{{ etcd_config_dir }}/admin.key"
repo="{{ etcd_snapshot_dir }}"
exec 9>"$repo/.lock"
flock -n 9 || exit 0
output="$repo/$(date -u +%Y%m%dT%H%M%SZ).db"
trap 'rm -f "$output.partial" "$output.partial.part"' EXIT
{{ etcd_install_dir }}/etcdctl snapshot save "$output.partial"
{{ etcd_install_dir }}/etcdutl snapshot status "$output.partial" >/dev/null
mv "$output.partial" "$output"
# 只有新快照成功且验证可读才清理旧备份。目录仅由此任务管理。
mapfile -t snapshots < <(find "$repo" -maxdepth 1 -type f -name '????????T??????Z.db' -printf '%f\n' | sort -r)
for old in "${snapshots[@]:{{ etcd_snapshot_keep }}}"; do
rm -- "$repo/$old"
done
@@ -1,10 +0,0 @@
[Unit]
Description=Daily homelab etcd snapshot
[Timer]
OnCalendar=*-*-* 03:20:00 UTC
RandomizedDelaySec=300
Persistent=true
[Install]
WantedBy=timers.target
-66
View File
@@ -1,66 +0,0 @@
---
# 先配置全体,再滚动启动:首次集群形成不能在第一个节点等待 quorum。
- name: 配置 shared etcd 成员
hosts: etcd
become: true
roles:
- shared_etcd
- name: 滚动启动并验证 shared etcd
hosts: etcd
become: true
serial: 1
pre_tasks:
- name: 加载共享默认参数
ansible.builtin.import_role:
name: shared_etcd
tasks_from: context
tasks:
- name: 启动已配置的成员
ansible.builtin.systemd_service:
name: homelab-etcd
enabled: true
daemon_reload: true
state: "{{ 'restarted' if etcd_config_changed | bool else 'started' }}"
- name: 等待本机客户端端口
ansible.builtin.wait_for:
host: "{{ etcd_address }}"
port: "{{ etcd_client_port }}"
timeout: 60
when: not ansible_check_mode
- name: 已有集群每次激活后等待本成员恢复 quorum 通信
ansible.builtin.command:
argv:
- "{{ etcd_install_dir }}/etcdctl"
- --endpoints=https://{{ etcd_address }}:{{ etcd_client_port }}
- --cacert={{ etcd_config_dir }}/ca.crt
- --cert={{ etcd_config_dir }}/admin.crt
- --key={{ etcd_config_dir }}/admin.key
- endpoint
- health
changed_when: false
register: etcd_member_health
retries: 12
delay: 5
until: etcd_member_health.rc == 0
when: etcd_activated_file.stat.exists and not ansible_check_mode
- name: 核对全部成员
ansible.builtin.import_playbook: verify.yml
- name: 记录已成功激活的配置
hosts: etcd
become: true
pre_tasks:
- name: 加载共享默认参数
ansible.builtin.import_role:
name: shared_etcd
tasks_from: context
tasks:
- name: 写入激活指纹(全体健康检查通过后)
ansible.builtin.copy:
content: "{{ etcd_config_fingerprint }}\n"
dest: "{{ etcd_config_dir }}/activated.sha256"
mode: '0644'
when: not ansible_check_mode
@@ -1,170 +0,0 @@
---
- name: 验证消费者范围
ansible.builtin.assert:
that:
- etcd_consumer.name is match('^[a-z][a-z0-9-]+$')
- etcd_consumer.name != 'root'
- etcd_consumer.prefix is match('^/homelab/[a-zA-Z0-9/_-]+/$')
- etcd_consumer.prefix | length > 10
- name: 读取已有用户和角色
ansible.builtin.command:
argv: ["{{ etcd_install_dir }}/etcdctl", --write-out=json, "{{ item }}", list]
loop: [user, role]
register: etcd_identities
changed_when: false
check_mode: false
# 404 data 也可能代表被删除/销毁的旧秘密;只有 metadata 不存在才允许生成。
- name: 控制端读取 Bao 秘密元数据
ansible.builtin.uri:
url: "{{ etcd_bao_url }}/v1/{{ etcd_bao_kv_mount }}/metadata/{{ etcd_bao_secret_base }}/{{ etcd_consumer.name }}"
headers:
X-Vault-Token: "{{ etcd_bao_token }}"
status_code: [200, 404]
delegate_to: localhost
become: false
no_log: true
register: etcd_secret_metadata
check_mode: false
- name: 禁止已有用户丢失秘密后自动换密码
ansible.builtin.assert:
that: >-
etcd_secret_metadata.status == 200 or
etcd_consumer.name not in ((etcd_identities.results[0].stdout | from_json).users | default([], true))
fail_msg: etcd 用户已存在但 Bao 秘密缺失;需恢复原秘密或执行显式轮换。
- name: 首次创建随机秘密且禁止覆盖已有版本
ansible.builtin.uri:
url: "{{ etcd_bao_url }}/v1/{{ etcd_bao_kv_mount }}/data/{{ etcd_bao_secret_base }}/{{ etcd_consumer.name }}"
method: POST
headers:
X-Vault-Token: "{{ etcd_bao_token }}"
body_format: json
body:
options: {cas: 0}
data:
username: "{{ etcd_consumer.name }}"
password: "{{ lookup('ansible.builtin.password', '/dev/null', length=48, chars=['ascii_letters', 'digits']) }}"
prefix: "{{ etcd_consumer.prefix }}"
status_code: 200
delegate_to: localhost
become: false
no_log: true
changed_when: true
when:
- etcd_secret_metadata.status == 404
- not ansible_check_mode
- name: 控制端读取既有秘密
ansible.builtin.uri:
url: "{{ etcd_bao_url }}/v1/{{ etcd_bao_kv_mount }}/data/{{ etcd_bao_secret_base }}/{{ etcd_consumer.name }}"
headers:
X-Vault-Token: "{{ etcd_bao_token }}"
status_code: 200
delegate_to: localhost
become: false
register: etcd_consumer_secret
no_log: true
when: etcd_secret_metadata.status == 200 or not ansible_check_mode
check_mode: false
- name: 核对已保存秘密归属
ansible.builtin.assert:
that:
- etcd_consumer_secret.json.data.data.username == etcd_consumer.name
- etcd_consumer_secret.json.data.data.prefix == etcd_consumer.prefix
- etcd_consumer_secret.json.data.data.password | length >= 32
no_log: true
when: etcd_secret_metadata.status == 200 or not ansible_check_mode
- name: 新建消费者用户(密码只通过 stdin 传递)
ansible.builtin.command:
argv: ["{{ etcd_install_dir }}/etcdctl", user, add, "{{ etcd_consumer.name }}", --interactive=false]
stdin: "{{ etcd_consumer_secret.json.data.data.password }}"
no_log: true
changed_when: true
when:
- etcd_consumer.name not in ((etcd_identities.results[0].stdout | from_json).users | default([], true))
- not ansible_check_mode
- name: 新建消费者角色
ansible.builtin.command:
argv: ["{{ etcd_install_dir }}/etcdctl", role, add, "{{ etcd_consumer.name }}"]
changed_when: true
when: etcd_consumer.name not in ((etcd_identities.results[1].stdout | from_json).roles | default([], true))
- name: 读取角色权限
ansible.builtin.command:
argv: ["{{ etcd_install_dir }}/etcdctl", --write-out=json, role, get, "{{ etcd_consumer.name }}"]
register: etcd_role_state
changed_when: false
when: >-
not ansible_check_mode or
etcd_consumer.name in ((etcd_identities.results[1].stdout | from_json).roles | default([], true))
- name: 拒绝不符合声明的既有权限(不自动扩大或删除)
ansible.builtin.assert:
that:
- >-
((etcd_role_state.stdout | from_json).perm | default([], true)) in
[[], [{'permType': 2, 'key': etcd_consumer.prefix | b64encode,
'range_end': (etcd_consumer.prefix[:-1] ~ '0') | b64encode}]]
fail_msg: 既有角色权限与声明不同,请显式审查权限迁移。
when: etcd_role_state is not skipped
- name: 授予唯一 prefix 读写权限
ansible.builtin.command:
argv:
- "{{ etcd_install_dir }}/etcdctl"
- role
- grant-permission
- "{{ etcd_consumer.name }}"
- readwrite
- "{{ etcd_consumer.prefix }}"
- --prefix=true
changed_when: true
when:
- etcd_role_state is not skipped
- ((etcd_role_state.stdout | from_json).perm | default([], true)) | length == 0
- name: 读取用户角色
ansible.builtin.command:
argv: ["{{ etcd_install_dir }}/etcdctl", --write-out=json, user, get, "{{ etcd_consumer.name }}"]
register: etcd_user_state
changed_when: false
when: >-
not ansible_check_mode or
etcd_consumer.name in ((etcd_identities.results[0].stdout | from_json).users | default([], true))
- name: 拒绝消费者已有额外角色
ansible.builtin.assert:
that: >-
((etcd_user_state.stdout | from_json).roles | default([], true))
| difference([etcd_consumer.name]) | length == 0
when: etcd_user_state is not skipped
- name: 绑定消费者角色
ansible.builtin.command:
argv: ["{{ etcd_install_dir }}/etcdctl", user, grant-role, "{{ etcd_consumer.name }}", "{{ etcd_consumer.name }}"]
changed_when: true
when:
- etcd_user_state is not skipped
- etcd_consumer.name not in ((etcd_user_state.stdout | from_json).roles | default([], true))
# 使用 gateway 的真实密码登录来核对 Bao 与 etcd 一致性,不能只看用户已存在。
- name: 验证消费者密码可经 gateway 登录
ansible.builtin.uri:
url: "https://{{ etcd_address }}:{{ etcd_client_port }}/v3/auth/authenticate"
method: POST
client_cert: "{{ etcd_config_dir }}/gateway.crt"
client_key: "{{ etcd_config_dir }}/gateway.key"
ca_path: "{{ etcd_config_dir }}/ca.crt"
body_format: json
body:
name: "{{ etcd_consumer.name }}"
password: "{{ etcd_consumer_secret.json.data.data.password }}"
status_code: 200
no_log: true
when: not ansible_check_mode
-29
View File
@@ -1,29 +0,0 @@
---
- name: 验证 shared etcd 全部端点
hosts: etcd[0]
become: true
gather_facts: false
pre_tasks:
- name: 加载共享默认参数
ansible.builtin.import_role:
name: shared_etcd
tasks_from: context
tasks:
- name: 通过管理员 mTLS 检查所有端点健康
ansible.builtin.command:
argv:
- "{{ etcd_install_dir }}/etcdctl"
- >-
--endpoints={{ groups['etcd'] | map('extract', hostvars, 'etcd_address')
| map('regex_replace', '^(.*)$', 'https://\1:' ~ etcd_client_port) | join(',') }}
- --cacert={{ etcd_config_dir }}/ca.crt
- --cert={{ etcd_config_dir }}/admin.crt
- --key={{ etcd_config_dir }}/admin.key
- endpoint
- health
changed_when: false
register: etcd_health
retries: 12
delay: 5
until: etcd_health.rc == 0
when: not ansible_check_mode
-30
View File
@@ -1,30 +0,0 @@
#!/usr/bin/python3
"""仅由受控调用者捕获 stdout;不得手动运行以免在终端输出短期 token。"""
import json
import ssl
import sys
import urllib.request
def main():
context = ssl.create_default_context()
context.load_cert_chain('/etc/homelab-etcd/peer.crt', '/etc/homelab-etcd/peer.key')
request = urllib.request.Request(
'https://bao.ad.ddupan.top:8200/v1/auth/homelab-etcd-renewal/login',
data=json.dumps({'name': 'etcd-laptop'}).encode(),
headers={'Content-Type': 'application/json'}, method='POST',
)
with urllib.request.urlopen(request, context=context, timeout=30) as response:
token = json.load(response)['auth']['client_token']
if not isinstance(token, str) or not token:
raise ValueError('empty token')
sys.stdout.write(token)
if __name__ == '__main__':
try:
main()
except Exception:
# Bao 响应和异常对象可能带敏感内容,不写入 journal。
sys.stderr.write('Bao certificate login failed\n')
sys.exit(1)
-51
View File
@@ -1,51 +0,0 @@
#!/usr/bin/python3
"""每日一次,机器证书换短期 token;使用现有串行 Ansible 收敛,不常驻 agent。"""
import fcntl
import os
from pathlib import Path
import subprocess
import sys
import urllib.request
ROOT = Path('/opt/homelab-etcd-controller')
def main():
with open('/var/lib/homelab-etcd-controller/renew.lock', 'a') as lock:
try:
fcntl.flock(lock, fcntl.LOCK_EX | fcntl.LOCK_NB)
except BlockingIOError:
return 0
login = subprocess.run(
['sudo', '-n', '/usr/bin/python3', str(ROOT / 'login.py')],
capture_output=True, text=True, timeout=40,
)
if login.returncode or not login.stdout.strip():
print('Bao 机器证书登录失败;保留现有证书与运行中的 etcd。', file=sys.stderr)
return 1
token = login.stdout.strip()
env = dict(os.environ, BAO_TOKEN=token)
try:
for play in ['verify.yml', 'site.yml']:
result = subprocess.run(
['/home/panxiao81/.local/bin/ansible-playbook', play],
cwd=ROOT / 'ansible', env=env, timeout=750,
)
if result.returncode:
return result.returncode
Path('/var/lib/homelab-etcd-controller/last-success').touch()
return 0
finally:
request = urllib.request.Request(
'https://bao.ad.ddupan.top:8200/v1/auth/token/revoke-self',
data=b'{}', headers={'X-Vault-Token': token}, method='POST',
)
try:
with urllib.request.urlopen(request, timeout=15):
pass
except Exception:
print('短期 token 撤销未确认,将由 TTL 自动失效。', file=sys.stderr)
if __name__ == '__main__':
raise SystemExit(main())
-43
View File
@@ -1,43 +0,0 @@
#!/usr/bin/env python3
"""从当前 Bao 登录会话向子进程传递凭据,不写临时秘密文件或输出秘密。"""
import json
import os
from pathlib import Path
import subprocess
import sys
ROOT = Path(__file__).resolve().parent
def main():
if len(sys.argv) < 3 or sys.argv[1] not in ('terraform', 'ansible'):
raise SystemExit('用法:python3 run.py terraform <args> | ansible <playbook> [args]')
env = dict(os.environ)
env.setdefault('BAO_ADDR', 'https://bao.ad.ddupan.top:8200')
token = env.get('BAO_TOKEN') or env.get('VAULT_TOKEN')
if not token:
token = Path('~/.vault-token').expanduser().read_text().strip()
env['BAO_TOKEN'] = env['VAULT_TOKEN'] = token
if sys.argv[1] == 'terraform':
response = subprocess.run(
['bao', 'kv', 'get', '-format=json', 'kv/k8s/seaweedfs-s3'],
env=env, capture_output=True, text=True,
)
if response.returncode:
raise SystemExit('读取 tfstate 受限身份失败;请检查 Bao 登录和授权。')
config = json.loads(response.stdout)['data']['data']['seaweedfs_s3_config']
config = json.loads(config) if isinstance(config, str) else config
identities = [i for i in config['identities'] if i['name'] == 'terraform']
if len(identities) != 1 or len(identities[0]['credentials']) != 1:
raise SystemExit('tfstate 身份不唯一,拒绝猜测凭据。')
credential = identities[0]['credentials'][0]
env['AWS_ACCESS_KEY_ID'] = credential['accessKey']
env['AWS_SECRET_ACCESS_KEY'] = credential['secretKey']
command, cwd = ['terraform', *sys.argv[2:]], ROOT / 'terraform'
else:
command, cwd = ['ansible-playbook', *sys.argv[2:]], ROOT / 'ansible'
raise SystemExit(subprocess.run(command, cwd=cwd, env=env).returncode)
if __name__ == '__main__':
main()
-22
View File
@@ -1,22 +0,0 @@
# This file is maintained automatically by "terraform init".
# Manual edits may be lost in future updates.
provider "registry.terraform.io/hashicorp/vault" {
version = "4.8.0"
constraints = "~> 4.0"
hashes = [
"h1:aHqgWQhDBMeZO9iUKwJYMlh4q+xNMUlMIcjRbF4d02Y=",
"zh:269ab13433f67684012ae7e15876532b0312f5d0d2002a9cf9febb1279ce5ea6",
"zh:4babc95bf0c40eb85005db1dc2ca403c46be4a71dd3e409db3711a56f7a5ca0e",
"zh:78d5eefdd9e494defcb3c68d282b8f96630502cac21d1ea161f53cfe9bb483b3",
"zh:86e27c1c625ecc24446a11eeffc3ac319b36c2b4e51251db8579256a0dbcf136",
"zh:a32f31da94824009e26b077374440b52098aecb93c92ff55dc3d31dd37c4ea25",
"zh:be0a18c6c0425518bab4fbffd82078b82036a88503b5d76064de551c9f646cbf",
"zh:be5a77fdfd36863ebeec79cd12b1d13322ffad6821d157a0b279789fa06b5937",
"zh:be8317d142a3caad74c7d936039ae27076a1b2b8312ef5208e2871a5f525977c",
"zh:c94a84895a3d9954b80e983eed4603330a5cdbbd8eef5b3c99278c2d1402ef3c",
"zh:de1fb712784dd8415f011ca5346a34f87fab6046c730557615247e511dbc7d98",
"zh:e3eafae7da550f86cae395d6660b2a0e93ec8d2b0e0e5ef982ec762e961fc952",
"zh:ff35fb1ab6add288f0f368981e56f780b50405accd1937131cba1137999c8d83",
]
}
-15
View File
@@ -1,15 +0,0 @@
# 与既有服务复用受限 tfstate 身份,使用独立对象与原生锁;不复用 Bao 的 state。
terraform {
backend "s3" {
bucket = "tfstate"
key = "etcd/terraform.tfstate"
endpoints = { s3 = "https://s3.ad.ddupan.top" }
region = "us-east-1"
use_path_style = true
skip_credentials_validation = true
skip_metadata_api_check = true
skip_region_validation = true
skip_requesting_account_id = true
use_lockfile = true
}
}
-93
View File
@@ -1,93 +0,0 @@
terraform {
required_version = ">= 1.10"
required_providers {
vault = {
source = "hashicorp/vault"
version = "~> 4.0"
}
}
}
provider "vault" {
address = var.bao_address
}
variable "bao_address" {
type = string
default = "https://bao.ad.ddupan.top:8200"
}
variable "pki_mount" {
type = string
default = "pki"
}
variable "kv_mount" {
type = string
default = "kv"
}
variable "member_names" {
type = list(string)
default = ["etcd-laptop", "etcd-pve1", "etcd-pve2"]
}
variable "consumer_names" {
type = set(string)
default = ["patroni-pg-prod"]
}
# 仅管理既有 PKI 下的新 role/policy,不纳管 CA 私钥、mount 或秘密值。
locals {
certificate_roles = {
server = { names = var.member_names, server = true, client = false, ips = true }
peer = { names = concat(var.member_names, ["homelab-etcd-peer"]), server = true, client = true, ips = true }
admin = { names = ["root"], server = false, client = true, ips = false }
gateway = { names = var.member_names, server = false, client = true, ips = false }
client = { names = [for name in var.consumer_names : "etcd-${name}"], server = false, client = true, ips = false }
}
}
resource "vault_pki_secret_backend_role" "etcd" {
for_each = local.certificate_roles
backend = var.pki_mount
name = "homelab-etcd-${each.key}"
allowed_domains = each.value.names
allow_bare_domains = true
allow_subdomains = false
allow_glob_domains = false
allow_any_name = false
allow_localhost = false
allow_wildcard_certificates = false
allow_ip_sans = each.value.ips
server_flag = each.value.server
client_flag = each.value.client
key_type = "ec"
key_bits = 256
ttl = 5184000
max_ttl = 5184000
require_cn = !contains(["gateway", "client"], each.key)
use_csr_common_name = true
use_csr_sans = true
}
resource "vault_policy" "etcd_provisioner" {
name = "homelab-etcd-provisioner"
policy = <<-EOT
path "${var.pki_mount}/sign/homelab-etcd-*" {
capabilities = ["update"]
}
path "${var.kv_mount}/metadata/infra/etcd/consumers/*" {
capabilities = ["read"]
}
path "${var.kv_mount}/data/infra/etcd/consumers/*" {
capabilities = ["create", "read", "update"]
}
EOT
}
# 身份绑定沿用既有控制端认证方式,创建 policy 不自动授权任何身份。
resource "vault_policy" "etcd_consumer" {
for_each = var.consumer_names
name = "homelab-etcd-${each.key}"
policy = <<-EOT
path "${var.kv_mount}/data/infra/etcd/consumers/${each.key}" {
capabilities = ["read"]
}
EOT
}
-28
View File
@@ -1,28 +0,0 @@
# 独立挂载:不接管全局认证,不保存长期 token,也不授权 KV 消费者秘密读取。
data "vault_generic_secret" "etcd_ca" {
path = "${var.pki_mount}/cert/ca"
}
resource "vault_auth_backend" "etcd_renewal" {
type = "cert"
path = "homelab-etcd-renewal"
}
resource "vault_policy" "etcd_renewal" {
name = "homelab-etcd-renewal"
policy = join("\n", concat([
for kind in ["server", "peer", "admin", "gateway"] :
"path \"${var.pki_mount}/sign/homelab-etcd-${kind}\" { capabilities = [\"update\"] }"
], ["path \"auth/token/revoke-self\" { capabilities = [\"update\"] }"]))
}
resource "vault_cert_auth_backend_role" "etcd_renewal" {
backend = vault_auth_backend.etcd_renewal.path
name = "etcd-laptop"
certificate = data.vault_generic_secret.etcd_ca.data["certificate"]
allowed_dns_sans = ["etcd-laptop"]
token_policies = [vault_policy.etcd_renewal.name]
token_no_default_policy = true
token_ttl = 600
token_max_ttl = 900
}
-233
View File
@@ -1,233 +0,0 @@
#!/usr/bin/env python3
"""临时三成员 mTLS/RBAC 集成测试;仅绑定 loopback,不访问生产 Bao。"""
import base64
import json
import os
from pathlib import Path
import re
import shutil
import ssl
import subprocess
import tempfile
import threading
import time
import urllib.error
import urllib.request
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
import jinja2
import yaml
ROOT = Path(__file__).resolve().parents[1]
BIN = Path(os.environ.get('ETCD_TEST_BIN', '/tmp/etcd-v3.7.2-linux-amd64'))
def run(argv, **kwargs):
return subprocess.run([str(x) for x in argv], capture_output=True, text=True, check=True, **kwargs)
class FakeBao(BaseHTTPRequestHandler):
records = {}
versions = {}
writes = 0
deny = False
def log_message(self, *_):
pass
def respond(self, status, body):
self.send_response(status)
self.send_header('Content-Type', 'application/json')
self.end_headers()
self.wfile.write(json.dumps(body).encode())
def do_GET(self):
if self.deny:
return self.respond(403, {'errors': ['permission denied']})
name = self.path.rsplit('/', 1)[-1]
if '/metadata/' in self.path and name in self.versions:
return self.respond(200, {'data': {'current_version': self.versions[name]}})
if '/data/' in self.path and name in self.records:
return self.respond(200, {'data': {'data': self.records[name]}})
self.respond(404, {'errors': []})
def do_POST(self):
body = json.loads(self.rfile.read(int(self.headers['Content-Length'])))
name = self.path.rsplit('/', 1)[-1]
if body['options']['cas'] != 0 or name in self.versions:
return self.respond(400, {'errors': ['CAS mismatch']})
self.records[name] = body['data']
self.versions[name] = 1
type(self).writes += 1
self.respond(200, {'data': {'version': 1}})
def main():
for key in ('HTTP_PROXY', 'HTTPS_PROXY', 'ALL_PROXY', 'http_proxy', 'https_proxy', 'all_proxy'):
os.environ.pop(key, None)
os.environ['NO_PROXY'] = '*'
processes = []
handles = []
server = None
with tempfile.TemporaryDirectory(prefix='shared-etcd-test-') as work:
w = Path(work)
w.chmod(0o700)
try:
run(['openssl', 'req', '-x509', '-newkey', 'rsa:2048', '-nodes', '-keyout', w/'ca.key',
'-out', w/'ca.crt', '-days', '1', '-subj', '/CN=isolated-test-ca'])
def cert(name, cn, eku, ip=None):
run(['openssl', 'req', '-new', '-newkey', 'rsa:2048', '-nodes', '-keyout', w/f'{name}.key',
'-out', w/f'{name}.csr', '-subj', f'/CN={cn}' if cn else '/'])
ext = w/f'{name}.ext'
ext.write_text(f'extendedKeyUsage={eku}\n' + (f'subjectAltName=IP:{ip},IP:127.0.0.1\n' if ip else ''))
run(['openssl', 'x509', '-req', '-in', w/f'{name}.csr', '-CA', w/'ca.crt',
'-CAkey', w/'ca.key', '-CAcreateserial', '-out', w/f'{name}.crt', '-days', '1', '-extfile', ext])
cert('admin', 'root', 'clientAuth')
cert('client', '', 'clientAuth')
cert('gateway', '', 'clientAuth')
env = jinja2.Environment(loader=jinja2.FileSystemLoader(ROOT/'ansible/roles/shared_etcd/templates'), undefined=jinja2.StrictUndefined)
env.filters['to_json'] = json.dumps
defaults = yaml.safe_load((ROOT/'ansible/roles/shared_etcd/defaults/main.yml').read_text())
names = ['test1', 'test2', 'test3']
hosts = {name: {'etcd_address': f'127.0.0.{i+2}'} for i, name in enumerate(names)}
for name in names:
d = w/name
d.mkdir()
for kind, cn, eku in [('server', name, 'serverAuth'), ('peer', 'homelab-etcd-peer', 'serverAuth,clientAuth')]:
cert(f'{name}-{kind}', cn, eku, hosts[name]['etcd_address'])
for suffix in ['crt', 'key']:
shutil.copy(w/f'{name}-{kind}.{suffix}', d/f'{kind}.{suffix}')
shutil.copy(w/'ca.crt', d/'ca.crt')
for suffix in ['crt', 'key']:
shutil.copy(w/f'gateway.{suffix}', d/f'gateway.{suffix}')
values = dict(defaults, inventory_hostname=name, groups={'etcd': names}, hostvars=hosts,
etcd_address=hosts[name]['etcd_address'], etcd_config_dir=str(d), etcd_data_dir=str(d/'data'),
etcd_client_port=22379, etcd_peer_port=22380, etcd_metrics_port=0)
config = env.get_template('etcd.yml.j2').render(**values)
# 三成员共享进程命名空间,metrics 用独立 loopback IP。
config = config.replace('http://127.0.0.1:0', f"http://{hosts[name]['etcd_address']}:22381")
(d/'config.yml').write_text(config)
handle = (d/'etcd.log').open('w')
handles.append(handle)
processes.append(subprocess.Popen([str(BIN/'etcd'), '--config-file='+str(d/'config.yml')], stdout=handle, stderr=handle))
base_env = dict(os.environ, ETCDCTL_ENDPOINTS='https://127.0.0.2:22379', ETCDCTL_CACERT=str(w/'ca.crt'),
ETCDCTL_CERT=str(w/'admin.crt'), ETCDCTL_KEY=str(w/'admin.key'), ETCDCTL_DIAL_TIMEOUT='2s', ETCDCTL_COMMAND_TIMEOUT='3s')
def ctl(*args, input=None, env=None):
return run([BIN/'etcdctl', *args], input=input, env=env or base_env)
for attempt in range(5):
try:
ctl('endpoint', 'health')
break
except subprocess.CalledProcessError:
if any(p.poll() is not None for p in processes):
for name in names:
print((w/name/'etcd.log').read_text()[-4000:])
raise RuntimeError('test etcd exited during startup')
time.sleep(0.2)
else:
print((w/'test1'/'etcd.log').read_text()[-5000:])
raise RuntimeError('test quorum did not form')
server = ThreadingHTTPServer(('127.0.0.1', 0), FakeBao)
threading.Thread(target=server.serve_forever, daemon=True).start()
inventory = w/'hosts.yml'
inventory.write_text(yaml.safe_dump({'all': {'children': {'etcd': {'hosts': {'test1': {'ansible_connection': 'local'}}}}}}))
extra = {
'ansible_become': False, 'etcd_address': '127.0.0.2', 'etcd_client_port': 22379,
'etcd_install_dir': str(BIN), 'etcd_config_dir': str(w), 'etcd_bootstrap_auth': True,
'etcd_bao_url': f'http://127.0.0.1:{server.server_port}', 'etcd_bao_token': 'isolated-test-token',
'etcd_bao_kv_mount': 'kv', 'etcd_bao_secret_base': 'infra/etcd/consumers',
'etcd_consumers': [{'name': 'patroni-pg-prod', 'prefix': '/homelab/patroni/pg-prod/'}],
}
(w/'extra.json').write_text(json.dumps(extra))
ansible_env = dict(os.environ, ANSIBLE_LOCAL_TEMP=str(w/'ansible-tmp'), ANSIBLE_NOCOLOR='1',
ANSIBLE_ROLES_PATH=str(ROOT/'ansible/roles'))
def play(name, fail=False):
r = subprocess.run(['ansible-playbook', '-i', str(inventory), str(ROOT/'ansible'/name), '-e', '@'+str(w/'extra.json')],
capture_output=True, text=True, env=ansible_env)
(w/(name+'.log')).write_text(r.stdout+r.stderr)
if fail:
assert r.returncode != 0, 'expected a fail-closed playbook error'
elif r.returncode:
# Tasks use no_log for secret data; retain useful task/line diagnostic.
print(r.stdout[-5000:]);print(r.stderr[-2000:])
raise RuntimeError(name+' failed')
return r.stdout
play('bootstrap-auth.yml')
assert re.search(r'changed=0\s', play('bootstrap-auth.yml'))
play('consumers.yml')
secret = FakeBao.records['patroni-pg-prod']['password']
assert len(secret) == 48
assert re.search(r'changed=0\s', play('consumers.yml'))
assert FakeBao.writes == 1
role = json.loads(ctl('--write-out=json', 'role', 'get', 'patroni-pg-prod').stdout)
assert role['perm'][0]['key'] == base64.b64encode(b'/homelab/patroni/pg-prod/').decode()
client_env = dict(base_env, ETCDCTL_CERT=str(w/'client.crt'), ETCDCTL_KEY=str(w/'client.key'),
ETCDCTL_USER='patroni-pg-prod', ETCDCTL_PASSWORD=secret)
ctl('put', '/homelab/patroni/pg-prod/test', 'ok', env=client_env)
try:
ctl('put', '/homelab/other/test', 'denied', env=client_env)
raise AssertionError('cross-prefix write succeeded')
except subprocess.CalledProcessError:
pass
# 模拟 Patroni 的 gateway 协议,证明 mTLS + username/password 可组合使用。
ctx = ssl.create_default_context(cafile=str(w/'ca.crt'))
ctx.load_cert_chain(w/'client.crt', w/'client.key')
body = json.dumps({'name':'patroni-pg-prod', 'password':secret}).encode()
req = urllib.request.Request('https://127.0.0.2:22379/v3/auth/authenticate', data=body, headers={'Content-Type':'application/json'})
with urllib.request.urlopen(req, context=ctx) as response:
token = json.load(response)['token']
gateway_body = json.dumps({'key': base64.b64encode(b'/homelab/patroni/pg-prod/gateway').decode(),
'value': base64.b64encode(b'ok').decode()}).encode()
req = urllib.request.Request('https://127.0.0.2:22379/v3/kv/put', data=gateway_body,
headers={'Content-Type': 'application/json', 'Authorization': token})
with urllib.request.urlopen(req, context=ctx) as response:
assert response.status == 200
# 对已有用户,秘密值被意外覆盖也必须失败,而不是悄悄改 etcd 密码。
FakeBao.records['patroni-pg-prod']['password'] = 'x' * 48
play('consumers.yml', fail=True)
FakeBao.records['patroni-pg-prod']['password'] = secret
FakeBao.deny = True
play('consumers.yml', fail=True)
FakeBao.deny = False
saved = FakeBao.records.pop('patroni-pg-prod')
play('consumers.yml', fail=True) # metadata 存在、data 已删除
FakeBao.versions.clear()
play('consumers.yml', fail=True) # 用户存在但 metadata 丢失
assert FakeBao.writes == 1
FakeBao.records['patroni-pg-prod'] = saved
FakeBao.versions['patroni-pg-prod'] = 1
ctl('snapshot', 'save', str(w/'snapshot.db'))
snapshot = json.loads(run([BIN/'etcdutl', '--write-out=json', 'snapshot', 'status', w/'snapshot.db']).stdout)
assert snapshot['totalKey'] > 0
run([BIN/'etcdutl', 'snapshot', 'restore', w/'snapshot.db', '--data-dir='+str(w/'restored')])
rss = []
for process in processes:
match = re.search(r'^VmRSS:\s+(\d+)', Path(f'/proc/{process.pid}/status').read_text(), re.M)
rss.append(round(int(match[1]) / 1024, 1))
processes[2].terminate();processes[2].wait(timeout=10)
ctl('put', '/homelab/patroni/pg-prod/after-member-loss', 'ok', env=client_env)
print('PASS: three-member mTLS, auth bootstrap/idempotence, Bao create-once/fail-closed, prefix isolation, gateway auth, snapshot/restore, one-member loss')
print('Idle test member RSS MiB (not a production capacity result):', rss)
finally:
if server:
server.shutdown();server.server_close()
for process in processes:
if process.poll() is None:
process.terminate()
try: process.wait(timeout=10)
except subprocess.TimeoutExpired: process.kill();process.wait()
for handle in handles:
handle.close()
if __name__ == '__main__':
main()

Some files were not shown because too many files have changed in this diff Show More