diff --git a/CLAUDE.md b/CLAUDE.md index ed74093..41a27d8 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -150,6 +150,11 @@ recovered, so `.vault_pass.gpg` is the authoritative recovery path. all-clear. Use `git check-ignore --no-index` and `git rm --cached` to actually remove it. - **A `.tfplan` is a zip containing a full `tfstate`.** It walks straight past `*.tfstate` ignore rules. Ignore `*.tfplan` everywhere. +- **SPIRE CLI JSON can be an array of response blocks.** `spire-agent api fetch jwt + -output json` in 1.15.3 returns blocks containing `svids` and `bundles`. Capture stdout + privately and type-check before extracting fields; `list(response)` prints full tokens + when the response is already an array. Never inspect credential payloads by printing + their containers, and never put fetched JWTs in command arguments or Pod logs. - **Quoting does not survive two ssh hops.** `ssh pve1 "ssh pve3 'cmd | qm monitor 103'"` loses the inner quotes — ssh re-joins argv with spaces, so the pipeline splits and the tail runs on the **jump host**. It fails silently if you discard stderr: a `screendump` diff --git a/apps/seaweedfs/README.md b/apps/seaweedfs/README.md index ebba835..98e3b9b 100644 --- a/apps/seaweedfs/README.md +++ b/apps/seaweedfs/README.md @@ -11,7 +11,9 @@ | `helm.sh` | Installs or upgrades the SeaweedFS release. | **Install** -1. Set real S3 access and secret keys in `values.yaml`. +1. 在 OpenBao `kv/k8s/seaweedfs-s3` 维护基础 S3 配置;zot 凭据单独以 + `kv/k8s/zot-s3` 为唯一来源。ESO 合成为 `seaweedfs-s3-config`,详见下文。 + 不要把真实 AK/SK 放进 `values.yaml`。 2. Apply the manifests: ```bash bash ~/services/apps/seaweedfs/helm.sh @@ -31,4 +33,22 @@ **Notes** - The chart manages master, volume, filer, S3, and admin components. -- The chart-managed S3 secret uses the current AK/SK pair for the admin user. +- The filer uses the ESO-managed `seaweedfs-s3-config` Secret for static S3 identities. + +## zot 制品存储 + +`zot` bucket 专用于 [zot Registry](../zot/README.md),OCI 数据位于 `registry/` +前缀。静态身份 `zot` 只有该 bucket 的 Read/Write/List/Tagging 权限,凭据唯一来源为 +Bao `kv/k8s/zot-s3` 的 `access_key` / `secret_key`,同时供 zot consumer 和 +SeaweedFS 服务端使用。 + +[ExternalSecret 模板](../../platform/external-secrets/externalsecrets.yaml) 保留 +`kv/k8s/seaweedfs-s3` 的原有身份及其他配置,再追加 zot 身份与限定 bucket 的权限。 +基础配置当前版本不保存 zot AK/SK;旧 KV 版本历史仍保留。新增其他身份时使用 +KV compare-and-set 保留已有内容,不覆盖 Terraform 或其他应用的 AK/SK。 +不要直接编辑生成的 Kubernetes Secret。该 ExternalSecret 已单独应用到集群, +目前仍未加入 ESO 的 Flux Kustomization,遵循该组件现有 ownership 边界。 + +运行版本 `4.22` 可在 Secret volume 更新后向 filer/内嵌 S3 的 `weed` 进程发送 +SIGHUP,重新加载静态配置,无需重启共享 S3 服务。本次接入没有启用 SeaweedFS +OIDC/STS;SPIRE 认证发生在 zot 的客户端入口。 diff --git a/apps/zot/README.md b/apps/zot/README.md new file mode 100644 index 0000000..cf4bcbb --- /dev/null +++ b/apps/zot/README.md @@ -0,0 +1,123 @@ +# zot OCI Registry + +内网入口为 `https://zot.ad.ddupan.top`。使用官方 Helm chart `0.1.124`,运行 +zot `v2.1.21`,镜像固定到官方 linux/amd64 digest。 + +## 存储与凭据 + +制品、manifest 和 OCI layout 保存在现有 SeaweedFS 的 `zot` bucket,前缀为 +`registry/`,S3 endpoint 为 `https://s3.ad.ddupan.top`。**不创建 PVC**;chart 的 +`/var/lib/registry` 是 `emptyDir`,仅用于运行时本地工作数据。 + +首期单副本,关闭跨仓库 dedupe,不额外部署 Redis/DynamoDB 缓存。保留 zot GC, +暂不配置自动删除已发布版本的 retention policy。增加副本、启用 dedupe 或搜索等 +扩展前,需要重新检查共享元数据与缓存的持久化要求。 + +凭据链路: + +```text +OpenBao kv/k8s/seaweedfs-s3 + → 原有 S3 身份及基础配置 ─┐ + ├→ ESO 模板 → seaweedfs/seaweedfs-s3-config +OpenBao kv/k8s/zot-s3 ────┘ → 完整 s3.config + └→ ESO → zot/zot-s3 → zot 的 AWS_ACCESS_KEY_ID / AWS_SECRET_ACCESS_KEY + +``` + +`kv/k8s/zot-s3` 是 zot AK/SK 的唯一维护来源。基础配置保留原有身份及其他字段, +不再保存 zot 凭据副本;[SeaweedFS ExternalSecret](../../platform/external-secrets/externalsecrets.yaml) +使用 ESO v2 模板追加 zot 身份。两个 Kubernetes Secret 都是自动生成的消费副本, +不手工编辑。Bao 的旧版本历史保留,回滚基础配置时模板也会替换其中的旧 zot 身份。 + +专用 S3 身份只有 `Read:zot`、`Write:zot`、`List:zot`、`Tagging:zot`,不能读取 +Terraform 的 `tfstate` bucket。`kv/k8s/zot-s3` 的字段是 `access_key` 和 +`secret_key`。AK/SK 不进入 Git、Helm values 或 CI;这里仍是静态 S3 凭据,尚未 +接入 SPIRE/STS。 + +本次归一没有轮换密钥,生成的完整配置与归一前语义一致。当前模板只有一组 zot +凭据,尚未实现新旧密钥重叠轮换。后续轮换只修改 `kv/k8s/zot-s3`,但仍需协调 +两个 ExternalSecret 同步:确认 SeaweedFS Secret volume 更新后向 filer 的 +`weed` 进程发送 SIGHUP,再确认 zot Secret 更新并重启 zot(环境变量不会热更新)。 +两端异步更新期间可能短暂认证失败;需要无中断轮换时先扩展模板支持新旧凭据重叠。 + +## SPIRE 认证和授权 + +| 参数 | 值 | +|---|---| +| issuer | `https://spire-oidc.ad.ddupan.top` | +| JWT audience | `zot` | +| subject | `spiffe://ddupan.top/` 下的 workload SPIFFE ID | +| token endpoint | `https://zot.ad.ddupan.top/zot/auth/token` | +| 当前权限 | 受信身份可以读取所有仓库;没有常驻写入或删除授权 | + +zot 通过已配置的 issuer discovery/JWKS 验证 JWT-SVID,再以 `sub` 作为授权身份。 +不接受任意 issuer,不关闭 TLS/issuer 验证。新的 Kata CI 负责取得并更新自己的 +JWT-SVID;确认其身份命名后,再添加针对具体 repository 的 `create`/`update` +授权,不能把整个 trust domain 都授予写权限。 + +现阶段拉取也需要 JWT-SVID。原定内网匿名拉取尚未启用:zot `v2.1.21` 的 +OIDC Bearer middleware 会在授权阶段之前拒绝无 token 请求,单独增加 +`anonymousPolicy` 无法解决。匿名读取与 SPIRE 写入共存需后续单独验证方案。 + +已有 SPIRE 身份的进程可以通过 Workload API 获取 `aud=zot` 的 JWT-SVID,然后 +通过 `docker login` 或 `crane auth login` 的 `--password-stdin` 交给 Registry。 +用户名可以使用 `zot`,实际权限取自已验证 JWT 的身份。使用独立、权限为 `0700` +的临时 `DOCKER_CONFIG`,结束后删除;不要开启 shell tracing,不要打印 token, +不要把 token 放进命令参数。token 接口不会延长 SVID 有效期。 + +## 部署与网络 + +- 官方 chart 管理 Deployment、Service、ConfigMap 和 HTTPRoute。 +- `persistence: false`,Service 为 ClusterIP,TLS 由已有 Envoy Gateway 的 + `https` listener 与内网通配符证书终止。 +- 仅配置 Samba AD 内网 DNS;不创建公网 DNS 或 Cloudflare Tunnel route。 +- NetworkPolicy 只允许现有 Envoy Gateway 数据面访问 zot 的 5000 端口。 +- HTTPRoute 只暴露 `/v2/` 和 `/zot/auth/token`,不暴露内部健康检查或管理端点。 +- namespace 使用 restricted PodSecurity,容器非 root、只读根文件系统。 + +首次已按用户授权从本地执行 `kubectl apply -k apps/zot`,由集群 Helm controller +安装。`clusters/homelab/apps/zot.yaml` 是 GitOps composition;对应文件合并进入 +Flux 跟踪分支后,才由根 Kustomization 持续管理,不能把未提交的本地部署写成 +已完成 Git 接管。 + +检查与渲染: + +```bash +helm template zot --repo https://zotregistry.dev/helm-charts \ + --version 0.1.124 --namespace zot -f apps/zot/values.yaml --skip-tests +sudo k3s kubectl -n zot get helmrelease,pods,externalsecret,httproute +sudo k3s kubectl -n zot get pvc +``` + +上游 chart 的 Helm test Pod 不满足本 namespace 的 restricted 策略,也没有 +SPIRE 凭据,因此不运行默认 `helm test`;使用下述真实身份验收。 + +## 验收与恢复 + +验收使用独立临时 Pod,通过 SPIFFE CSI socket 和真实 Workload API 取得 JWT-SVID, +没有修改现有 runner。仅在初始化 `verification/smoke:spire-s3` 测试镜像时临时 +授予该测试身份针对该仓库的写权限;完成后必须撤回 HelmRelease override,并删除 +临时 Pod、ServiceAccount 与 ClusterSPIFFEID。 + +验收项目:有效 SVID + crane pull、manifest digest 一致、错误 audience、错误 +signature、过期 token、无凭据写入、跨仓库写入、只读身份写入和删除拒绝;另外检查 +Pod 重建后镜像仍可拉取,以及 S3 身份不能访问 `tfstate`。 + +2026-09-14 已完成上述验收:HelmRelease Ready、HTTPRoute Accepted/ResolvedRefs, +DNS 第二次 Ansible check 为 `changed=0`;一分钟真实 JWT-SVID 到期后返回 401。 +临时写权限已移除。测试镜像可供后续 CI 验证拉取: + +```text +zot.ad.ddupan.top/verification/smoke:spire-s3 +sha256:b8d3b977a1235022759470903dab4a46b7cf8107958624f1f76a323eabe37c5e +``` + +它是仅含验证文本的 OCI 测试镜像,没有可执行入口,不用于运行服务。 + +Registry 恢复需要完整的 SeaweedFS bucket 数据、Bao 专用凭据和此目录配置。 +zot 的临时目录不是制品备份。独立异机/离线备份尚未在本次部署中建立;不能把同一 +SeaweedFS 内的数据副本当作独立灾备。重装 zot 不得删除 `zot` bucket。 + +参考:[官方 Kubernetes 安装](https://zotregistry.dev/v2.1.21/install-guides/install-guide-k8s/)、 +[S3 存储](https://zotregistry.dev/v2.1.21/articles/storage/)、 +[OIDC workload identity](https://github.com/project-zot/zot/blob/v2.1.21/examples/README-OIDC-WORKLOAD-IDENTITY.md)。 diff --git a/apps/zot/external-secret.yaml b/apps/zot/external-secret.yaml new file mode 100644 index 0000000..fb0a0d6 --- /dev/null +++ b/apps/zot/external-secret.yaml @@ -0,0 +1,22 @@ +apiVersion: external-secrets.io/v1 +kind: ExternalSecret +metadata: + name: zot-s3 + namespace: zot +spec: + refreshInterval: 1h + secretStoreRef: + kind: ClusterSecretStore + name: openbao + target: + name: zot-s3 + creationPolicy: Owner + data: + - secretKey: access_key + remoteRef: + key: k8s/zot-s3 + property: access_key + - secretKey: secret_key + remoteRef: + key: k8s/zot-s3 + property: secret_key diff --git a/apps/zot/helmrelease.yaml b/apps/zot/helmrelease.yaml new file mode 100644 index 0000000..08d1d31 --- /dev/null +++ b/apps/zot/helmrelease.yaml @@ -0,0 +1,30 @@ +apiVersion: helm.toolkit.fluxcd.io/v2 +kind: HelmRelease +metadata: + name: zot + namespace: zot +spec: + chart: + spec: + chart: zot + version: 0.1.124 + interval: 1h + sourceRef: + kind: HelmRepository + name: zot + releaseName: zot + interval: 30m + timeout: 5m + driftDetection: + mode: enabled + install: + strategy: + name: RetryOnFailure + retryInterval: 5m + upgrade: + strategy: + name: RetryOnFailure + retryInterval: 5m + valuesFrom: + - kind: ConfigMap + name: zot-values diff --git a/apps/zot/helmrepository.yaml b/apps/zot/helmrepository.yaml new file mode 100644 index 0000000..95eda44 --- /dev/null +++ b/apps/zot/helmrepository.yaml @@ -0,0 +1,8 @@ +apiVersion: source.toolkit.fluxcd.io/v1 +kind: HelmRepository +metadata: + name: zot + namespace: zot +spec: + interval: 1h + url: https://zotregistry.dev/helm-charts diff --git a/apps/zot/kustomization.yaml b/apps/zot/kustomization.yaml new file mode 100644 index 0000000..bb776de --- /dev/null +++ b/apps/zot/kustomization.yaml @@ -0,0 +1,18 @@ +apiVersion: kustomize.config.k8s.io/v1beta1 +kind: Kustomization +resources: + - namespace.yaml + - serviceaccount.yaml + - external-secret.yaml + - helmrepository.yaml + - helmrelease.yaml + - networkpolicy.yaml +generatorOptions: + disableNameSuffixHash: true + labels: + reconcile.fluxcd.io/watch: Enabled +configMapGenerator: + - name: zot-values + namespace: zot + files: + - values.yaml=values.yaml diff --git a/apps/zot/namespace.yaml b/apps/zot/namespace.yaml new file mode 100644 index 0000000..a33613e --- /dev/null +++ b/apps/zot/namespace.yaml @@ -0,0 +1,7 @@ +apiVersion: v1 +kind: Namespace +metadata: + name: zot + labels: + pod-security.kubernetes.io/enforce: restricted + pod-security.kubernetes.io/enforce-version: v1.36 diff --git a/apps/zot/networkpolicy.yaml b/apps/zot/networkpolicy.yaml new file mode 100644 index 0000000..f73c69b --- /dev/null +++ b/apps/zot/networkpolicy.yaml @@ -0,0 +1,22 @@ +apiVersion: networking.k8s.io/v1 +kind: NetworkPolicy +metadata: + name: zot-ingress + namespace: zot +spec: + podSelector: + matchLabels: + app.kubernetes.io/name: zot + policyTypes: [Ingress] + ingress: + - from: + - namespaceSelector: + matchLabels: + kubernetes.io/metadata.name: envoy-gateway-system + podSelector: + matchLabels: + gateway.envoyproxy.io/owning-gateway-name: eg + gateway.envoyproxy.io/owning-gateway-namespace: envoy-gateway-system + ports: + - protocol: TCP + port: 5000 diff --git a/apps/zot/serviceaccount.yaml b/apps/zot/serviceaccount.yaml new file mode 100644 index 0000000..898debf --- /dev/null +++ b/apps/zot/serviceaccount.yaml @@ -0,0 +1,6 @@ +apiVersion: v1 +kind: ServiceAccount +metadata: + name: zot + namespace: zot +automountServiceAccountToken: false diff --git a/apps/zot/values.yaml b/apps/zot/values.yaml new file mode 100644 index 0000000..06bb2aa --- /dev/null +++ b/apps/zot/values.yaml @@ -0,0 +1,147 @@ +# 官方 chart 0.1.124 / zot v2.1.21;制品与 manifests 保存在 SeaweedFS S3。 +# persistence=false 仅保留 chart 的 emptyDir,不创建 PVC。 +# 首期关闭跨仓库 dedupe,不额外引入 Redis/DynamoDB 持久缓存。 +replicaCount: 1 +image: + repository: ghcr.io/project-zot/zot + tag: v2.1.21@sha256:8258443838e95989c13c891f78a02bc1c391b5a00591ffef24cb8c17cde28038 +persistence: false +strategy: + type: Recreate +serviceAccount: + create: false + name: zot +service: + type: ClusterIP + port: 5000 +mountConfig: true +mountSecret: false +secretFiles: {} +configFiles: + config.json: | + { + "distSpecVersion": "1.1.1", + "storage": { + "rootDirectory": "/var/lib/registry", + "dedupe": false, + "gc": true, + "gcDelay": "24h", + "gcInterval": "24h", + "storageDriver": { + "name": "s3", + "region": "us-east-1", + "regionendpoint": "https://s3.ad.ddupan.top", + "bucket": "zot", + "rootdirectory": "/registry", + "secure": true, + "skipverify": false, + "forcepathstyle": true + } + }, + "http": { + "address": "0.0.0.0", + "port": "5000", + "externalUrl": "https://zot.ad.ddupan.top", + "compat": [ + "docker2s2" + ], + "auth": { + "bearer": { + "realm": "https://zot.ad.ddupan.top/zot/auth/token", + "service": "zot.ad.ddupan.top", + "oidc": [ + { + "issuer": "https://spire-oidc.ad.ddupan.top", + "audiences": [ + "zot" + ], + "claimMapping": { + "username": "claims.sub", + "validations": [ + { + "expression": "claims.sub.startsWith('spiffe://ddupan.top/')", + "message": "SPIFFE trust domain mismatch" + } + ] + } + } + ] + } + }, + "accessControl": { + "repositories": { + "**": { + "defaultPolicy": [ + "read" + ] + } + } + } + }, + "log": { + "level": "info" + } + } +env: +- name: AWS_ACCESS_KEY_ID + valueFrom: + secretKeyRef: + name: zot-s3 + key: access_key +- name: AWS_SECRET_ACCESS_KEY + valueFrom: + secretKeyRef: + name: zot-s3 + key: secret_key +- name: AWS_EC2_METADATA_DISABLED + value: 'true' +podSecurityContext: + runAsNonRoot: true + runAsUser: 10001 + runAsGroup: 10001 + fsGroup: 10001 + seccompProfile: + type: RuntimeDefault +securityContext: + allowPrivilegeEscalation: false + readOnlyRootFilesystem: true + capabilities: + drop: + - ALL +resources: + requests: + cpu: 100m + memory: 128Mi + limits: + cpu: '1' + memory: 512Mi +extraVolumes: +- name: tmp + emptyDir: + sizeLimit: 128Mi +extraVolumeMounts: +- name: tmp + mountPath: /tmp +startupProbe: + initialDelaySeconds: 5 + periodSeconds: 5 + failureThreshold: 60 +httproute: + enabled: true + parentRefs: + - name: eg + namespace: envoy-gateway-system + sectionName: https + hostnames: + - zot.ad.ddupan.top + rules: + - matches: + - path: + type: PathPrefix + value: /v2/ + - path: + type: Exact + value: /zot/auth/token + timeouts: + request: 900s + backendRequest: 900s diff --git a/clusters/homelab/apps/zot.yaml b/clusters/homelab/apps/zot.yaml new file mode 100644 index 0000000..d20a12a --- /dev/null +++ b/clusters/homelab/apps/zot.yaml @@ -0,0 +1,18 @@ +apiVersion: kustomize.toolkit.fluxcd.io/v1 +kind: Kustomization +metadata: + name: zot + namespace: flux-system +spec: + dependsOn: + - name: envoy-gateway + - name: external-secrets + - name: spire + interval: 10m + path: ./apps/zot + prune: false + sourceRef: + kind: GitRepository + name: flux-system + timeout: 5m + wait: true diff --git a/clusters/homelab/kustomization.yaml b/clusters/homelab/kustomization.yaml index f6268f5..5da14e8 100644 --- a/clusters/homelab/kustomization.yaml +++ b/clusters/homelab/kustomization.yaml @@ -12,3 +12,4 @@ resources: - apps/openebs.yaml - apps/spire.yaml - apps/observability.yaml + - apps/zot.yaml diff --git a/infrastructure/dns/records.yml b/infrastructure/dns/records.yml index 11a3375..71303d1 100644 --- a/infrastructure/dns/records.yml +++ b/infrastructure/dns/records.yml @@ -14,6 +14,7 @@ homelab_dns: - { zone: ad.ddupan.top, name: netbox, type: A, values: [192.168.10.127] } - { zone: ad.ddupan.top, name: s3, type: A, values: [192.168.10.127] } - { zone: ad.ddupan.top, name: spire-oidc, type: A, values: [192.168.10.127] } + - { zone: ad.ddupan.top, name: zot, type: A, values: [192.168.10.127] } split_horizon: # LAN and pod resolvers should eventually render the same set from here. diff --git a/platform/external-secrets/externalsecrets.yaml b/platform/external-secrets/externalsecrets.yaml index 0c16aa6..d12571b 100644 --- a/platform/external-secrets/externalsecrets.yaml +++ b/platform/external-secrets/externalsecrets.yaml @@ -101,6 +101,7 @@ spec: # The chart normally GENERATES seaweedfs-s3-secret from s3.credentials. We point # filer.s3.existingConfigSecret at this one instead, so the chart stops rendering # credentials from values entirely. +# zot 的 AK/SK 只保存在 k8s/zot-s3;在此组装服务端配置,不在基础配置中维护副本。 apiVersion: external-secrets.io/v1 kind: ExternalSecret metadata: @@ -114,6 +115,38 @@ spec: target: name: seaweedfs-s3-config creationPolicy: Owner - dataFrom: - - extract: + template: + engineVersion: v2 + mergePolicy: Replace + data: + seaweedfs_s3_config: |- + {{- $config := mustFromJson .baseConfig -}} + {{- if not (kindIs "slice" $config.identities) -}} + {{- fail "base S3 configuration must contain an identities array" -}} + {{- end -}} + {{- if or (eq .zotAccessKey "") (eq .zotSecretKey "") -}} + {{- fail "zot S3 credentials must not be empty" -}} + {{- end -}} + {{- $identities := list -}} + {{- range $config.identities -}} + {{- if ne .name "zot" -}} + {{- $identities = append $identities . -}} + {{- end -}} + {{- end -}} + {{- $credential := dict "accessKey" .zotAccessKey "secretKey" .zotSecretKey -}} + {{- $zot := dict "name" "zot" "credentials" (list $credential) "actions" (list "Read:zot" "Write:zot" "List:zot" "Tagging:zot") -}} + {{- $_ := set $config "identities" (append $identities $zot) -}} + {{- mustToJson $config -}} + data: + - secretKey: baseConfig + remoteRef: key: k8s/seaweedfs-s3 + property: seaweedfs_s3_config + - secretKey: zotAccessKey + remoteRef: + key: k8s/zot-s3 + property: access_key + - secretKey: zotSecretKey + remoteRef: + key: k8s/zot-s3 + property: secret_key