Establish clean homelab infrastructure baseline

Reorganize the brownfield repository, remove retired and generated artifacts, harden ignore rules, and record the GitOps/IaC redesign.

docs/cicd.md

@@ -0,0 +1,185 @@
+# CI/CD — what we are building
+
+Status: **design, partly built.** Stage 1 is live. The substrate is undecided.
+Started 2026-07-28.
+
+## Goal
+
+Every infrastructure change goes through a pipeline that tests it, **and that
+pipeline is the only path to production.**
+
+The second half is the point. Tests you can bypass with `helm upgrade` are not
+controls, they are suggestions. The cluster already shows what that looks like:
+`authelia` is on revision **14** and `e5renew` on revision **28**, all hand-run.
+Nothing is wrong with those releases — but nothing proves anything about them
+either, and two of the thirteen deployed Helm releases (`e5renew`, `rustfs`)
+have no representation in this repo at all.
+
+So the aim is not "add CI". It is to make the tested path the *only* path, and
+to make drift between this repo and reality visible when it happens.
+
+## Current state
+
+| | |
+|---|---|
+| git | History since 2026-07-28. Four commits, **no remote yet** |
+| stage 1 | Live and green — `yamllint`, `ansible-lint`, `terraform fmt`/`validate`. Configs tuned against a real run |
+| gitea | 1.25.5, Actions **enabled**, `DEFAULT_ACTIONS_URL=github`. **No runner deployed**, so nothing executes |
+| ansible | 33 roles across `infrastructure/proxmox/`, `infrastructure/samba-ad/`, `infrastructure/openbao/` |
+| terraform | 4 roots, **local state**, each with **different interactive auth** (`bao login -method=oidc`, `az login`) |
+| k8s | ~13 Helm releases, all deployed by hand |
+| secrets | 4 config files are gitignored because they embed live secrets, so their contents are **not** version controlled |
+
+## The four stages
+
+**Stage 1 — static. Built.**
+`yamllint`, `ansible-lint`, `terraform fmt -check` / `validate -backend=false`.
+No cluster, no credentials, no mutation, so it is safe on every push. It already
+found a real defect: `infrastructure/proxmox/ansible/` had no `requirements.yml` at all, so a
+fresh checkout could not reproduce its collections.
+
+**Stage 2 — server-side dry-run.**
+`kubectl apply --dry-run=server`. This is the one that is usually skipped and
+matters most here: the cluster runs Gateway API, cert-manager, the
+VictoriaMetrics operator and Envoy `SecurityPolicy` CRDs. Offline schema
+validation cannot see any of them; server-side dry-run validates against the
+real CRDs and admission webhooks. Needs cluster credentials, so it lives in a
+separate workflow from stage 1 — a credential problem there must never be able
+to block the static gate.
+
+**Stage 3 — ephemeral integration.**
+- **Ansible: Molecule.** Its default scenario asserts *idempotence*, which is
+  already this repo's written acceptance test ("a second run must report
+  `changed=0`"). The standard exists; Molecule just makes something other than a
+  human enforce it.
+- **Kubernetes: a throwaway `k3d` cluster** — apply everything, assert it
+  converges. Catches ordering and cross-resource problems a dry-run cannot.
+
+Realistic coverage: roughly a third of the 33 roles. `vyos_router` (no Python
+interpreter), `pve_cluster`, `pve_linstor` and the Windows-join roles need real
+hardware or nested VMs on Proxmox. Those stay `--check --diff` against live.
+
+**Stage 4 — deploy and drift.**
+Flux reconciles Kubernetes from git. Terraform and Ansible get **scheduled drift
+detection**, not auto-apply: `terraform plan -detailed-exitcode` and
+`ansible-playbook --check --diff` on a timer, alerting into the VictoriaMetrics
+stack that already exists. That is where most of the value is and almost none of
+the risk.
+
+## What enforces what
+
+There are only two enforcement mechanisms available, and this lab has both:
+
+- **Kubernetes → reconciliation.** Flux is the only writer. A hand-run
+  `helm upgrade` gets reverted on the next interval. Enforcement is free.
+- **Ansible and Terraform → credentials.** No reconciler exists for them and one
+  should not be invented; nothing stops someone SSHing to a PVE node and running
+  `pveum` by hand. The lever is **OpenBao's SSH CA**: if humans hold no standing
+  SSH access and the pipeline is the only identity that can get a signed cert,
+  the pipeline becomes the only path by credential control. Same shape for
+  Terraform — keep the AppRole and the Azure service principal only in CI.
+
+Credential lockdown comes **last**. It is the disruptive step and is only worth
+doing once the pipeline is trustworthy. A documented break-glass path must
+survive it: the playbooks still have to run by hand from the laptop during an
+incident, or enforcement locks you out exactly when you need in.
+
+## Substrate — open
+
+**Requirement:** a light always-on orchestrator with ephemeral workers. Not a VM
+or LXC parked idle waiting for work. Something must always listen — that is
+inherent to event-driven CI — but it should be a controller, not a pet.
+
+| option | verdict |
+|---|---|
+| **Woodpecker CI** | **Front-runner.** Kubernetes backend runs each step as a standalone Pod; first-class Gitea (OAuth2 + auto-created webhooks); server + agent, both light; stable. Costs a different pipeline syntax, which for shell-step lint jobs is ~20 lines |
+| `act_runner` (Gitea native) | Docker or host execution **only** — no Kubernetes executor. Confirmed in source: `labels.go` has just `SchemeDocker`/`SchemeHost`, and `run_context.go` branches only to `startHostEnvironment` or `startJobContainer`. Cheapest (one pod) but leaves a persistent worker |
+| Gitea ARC | Real pod-per-job operator, but **Enterprise Edition only** |
+| GARM + `garm-provider-k8s` | Right shape. GARM supports Gitea from 1.24, but latest is **v0.2.0-beta1** and the provider documents GitHub runners only — the Gitea pairing is unverified |
+| Write a runner from zero | The protocol (`actions-proto-go`) is approachable; reimplementing execution is not — `act_runner` delegates that to a vendored `nektos/act`. A shim spawning one-shot pods is blocked on ephemeral registration (`go-apps/gitea/gitea#32461`) |
+
+**Where runners run** is a separate axis. Argument for Proxmox: pve1–3 have clean
+egress, while the k3s node carries `openvpn-client@naist`, whose 58 split-tunnel
+routes blackhole Cloudflare, Fastly and Microsoft ranges whenever the tunnel dies
+— and pods inherit the host routing table. CI on the laptop will fail
+mid-build in confusing ways every time that happens. Argument against: no GARM
+Proxmox provider exists, so ephemeral Proxmox VMs would mean writing one.
+
+## Prerequisites
+
+Blocking, in order:
+
+1. **Externalise secrets.** `apps/gitea/gitea-values.yaml`, `apps/authelia/values.yaml`,
+   `infrastructure/cloudflared/cloudflared.yaml` and `litellm-gateway/docker-compose.yml` are
+   gitignored because they embed live credentials. GitOps requires the opposite —
+   a reconciler can only apply what is in git. OpenBao + External Secrets is the
+   natural fit; `openbao_bootstrap/tasks/auth_kubernetes.yml` already exists.
+2. **A git remote.** Gitea is the obvious host, but it runs *on* the cluster it
+   would deploy, so keep an off-cluster mirror for disaster recovery.
+3. **Terraform: shared state.** All four roots use local `*.tfstate`. Any
+   pipeline that applies needs shared state — the S3-compatible SeaweedFS/rustfs
+   already on the cluster is a candidate, with the caveat that cluster state
+   living on the cluster is the same circularity as hosting git there.
+
+**Not a blocker: non-interactive auth.** An earlier draft of this document
+called `bao login -method=oidc` and `az login` hard blockers that would widen the
+blast radius. That was wrong on both counts.
+
+- **OpenBao** — the Kubernetes auth backend is **already bootstrapped**
+  (`infrastructure/openbao/ansible/roles/openbao_bootstrap/tasks/auth_kubernetes.yml`). A CI pod
+  authenticates with its ServiceAccount JWT and stores **no credential at all**.
+  AppRole is the equivalent for anything running off-cluster.
+- **Azure** — the provider supports service principal with a client secret,
+  service principal with a **client certificate**, managed identity, and **OIDC
+  workload identity federation** (`use_oidc = true` with `oidc_token_file_path`,
+  which explicitly covers generic OIDC providers, not just GitHub/ADO).
+  HashiCorp's own guidance is to use a service principal or MSI *specifically*
+  when running non-interactively in CI, and reserve the Azure CLI for local runs.
+
+It also **narrows** privilege rather than widening it. The current path needs an
+`az login` as Global Admin / Privileged Role Admin — because that is what
+creating app registrations and granting admin consent requires. A scoped service
+principal managing steady state is dramatically less privileged than the human
+identity in use today. Same for OpenBao: a per-root AppRole or Kubernetes role
+can be scoped to exactly the paths that root owns, whereas an interactive OIDC
+admin login is not.
+
+The blast-radius concern in CLAUDE.md is about **merging the roots**, not about
+machine identities. Per-root scoped credentials preserve that separation and
+reduce privilege at the same time.
+
+The honest caveat is a split, not a blocker: app-registration creation and admin
+consent are genuinely high-privilege **bootstrap** operations and should stay
+manual. CI should hold an identity that manages steady state only. The cleanest
+assembly with what already exists: keep the Azure SP certificate in OpenBao, and
+let the CI pod fetch it via Kubernetes auth — so nothing long-lived is stored in
+the CI system itself.
+
+## Non-goals
+
+- **High availability.** One node. Nothing to fail over to. GitOps here buys
+  drift elimination and rebuild-from-scratch, not uptime.
+- **Autoscaling.** Nothing to scale across.
+- **Untrusted-workload isolation.** One author. This changes the day it stops
+  being true, and the runner's trust model must be revisited then.
+- **Terraform or Ansible under a Kubernetes reconciler.** More machinery on the
+  single node that is also NFS server, libvirt host and netboot appliance makes
+  recovery harder, not easier.
+- **Merging the Terraform roots.** Already considered and rejected; see CLAUDE.md.
+
+## Sequencing
+
+1. Externalise secrets — unblocks everything, valuable on its own
+2. Capture `e5renew` and `rustfs` into the repo (`helm get values`)
+3. Git remote
+4. Pick the substrate; stand it up in an isolated namespace
+5. Stage 2, then stage 3 on **one** container-friendly role first
+6. Flux on one low-stakes namespace (`http-echo` or `marker`)
+7. Drift detection for Terraform and Ansible — scoped machine identities
+   (OpenBao Kubernetes auth / AppRole, a steady-state Azure SP) land here, and
+   are a privilege *reduction* on the admin logins used today
+8. Credential lockdown — remove standing human access, with break-glass
+   documented. Last, and only once the pipeline has earned trust
+
+Steps 1, 2 and 7 are worth doing even if the CI substrate is never chosen — they
+are just "the repo should describe reality".

docs/homelab-gitops-redesign.md

@@ -0,0 +1,229 @@
+# Homelab GitOps and IaC redesign
+
+Status: **design; no infrastructure changes have been applied.**
+
+Started 2026-09-09. This is the durable record of the redesign discussion. It
+separates observations, decisions and open work so an assumption cannot silently
+become a destructive migration.
+
+## Goals
+
+- Make Git the source of infrastructure intent and the normal path to production.
+- Reconcile Kubernetes continuously; gate Terraform and Ansible with plans and approval.
+- Adopt existing infrastructure before changing it; avoid recreating VMs or disks.
+- Keep recovery possible when k3s, Gitea, OpenBao or the home WAN is unavailable.
+- Keep OCI within Always Free allowances except deliberately tiny Object Storage use.
+
+## Observed state
+
+### Local host
+
+The laptop is the single-node k3s host, NFS server and libvirt host. These
+persistent libvirt domains were observed on 2026-09-09:
+
+| domain | state | CPU | memory | root disk | network |
+|---|---|---:|---:|---|---|
+| `dc1` | running | 2 | 2 GiB | `data/vm/dc1` zvol | bridge `br0` |
+| `winadmin` | running | 4 | 6 GiB | `data/vm/winadmin` zvol | bridge `br0` |
+| `bao1` | running | 1 | 1 GiB | `data/vm/bao1` zvol | bridge `br0` |
+| `win2k25` | stopped | 2 | 4 GiB | `data/vm/win2k25` zvol | network `network` |
+
+All four have autostart disabled. The intended values for `dc1`, `winadmin`
+and `bao1` mostly agree with live configuration. Current Ansible roles stop
+managing a VM once its domain exists. That avoids recreation but cannot report
+or reconcile later CPU, memory, NIC or boot drift.
+
+### Kubernetes and delivery
+
+- Kubernetes is a single-node k3s cluster.
+- Helm releases and manifests have historically been applied by hand.
+- No Flux or Argo CD installation was found during the initial audit.
+- Repository history records External Secrets Operator 2.8.0 as deployed. Five
+  `ExternalSecret` resources cover Authelia, Gitea, Cloudflared and SeaweedFS.
+  All five reported `SecretSynced=True` during a live check on 2026-09-09.
+- The repository has no configured Git remote yet.
+- Terraform roots remain per-service and must not be merged.
+- The k3s node was `Ready` on 2026-09-09. The Snap-packaged `kubectl` could
+  not start because the user systemd session was degraded; `k3s kubectl` with the
+  same user kubeconfig reached the API successfully. This is a local client issue,
+  not evidence that the cluster is down.
+- `docs/cicd.md` remains useful but some inventory statements are stale.
+
+### OCI
+
+Read-only OCI discovery found:
+
+- home region `ap-osaka-1`;
+- bucket `oci-k8s-free-tier-tfstate` in the root compartment;
+- one approximately 25 KiB object, `terraform.tfstate`, modified 2026-08-15;
+- state format 4, Terraform 1.15.8, serial 249;
+- state for a VM, VCN, public subnet, Internet Gateway, route table, security
+  list and free-tier quota;
+- Object Storage versioning is not enabled;
+- the source Terraform root was not found locally.
+
+The state is likely authoritative. Its contents are sensitive and must not be
+printed or committed. Restore its configuration to a zero-change plan before
+considering a public-to-private subnet redesign.
+
+## Agreed architecture
+
+### Control planes
+
+| scope | controller | application model |
+|---|---|---|
+| Kubernetes | Flux | continuous pull reconciliation after merge |
+| Terraform | CI | plan; explicit approval before apply |
+| Ansible | CI | check/diff; explicit approval before execution |
+| Backstage | none | read-only portal and PR authoring |
+
+Flux was selected over Argo CD for a small cluster whose primary interface
+should remain Git and Kubernetes resources. Backstage supplies the unified UI
+without becoming another deployment controller.
+
+### CI execution
+
+Most CI runs as ephemeral Kubernetes Pods through Woodpecker's Kubernetes
+backend: lint, formatting, manifest rendering, unit tests and ordinary plans.
+
+Only explicitly labelled jobs needing privilege, nested virtualization,
+amd64-only software or isolation from k3s use an ephemeral Proxmox VM. IaC owns
+the PVE template, pool, permissions, network and quotas; the scheduler owns the
+short-lived clone/start/run/destroy lifecycle. Ephemeral workers do not enter
+Terraform state. OCI A1 may later run OCI plans and external checks.
+
+### Ingress
+
+Envoy Gateway becomes the sole in-cluster L7 routing and policy point:
+
+```text
+public client -> Cloudflare edge -> Tunnel -> Envoy -> HTTPRoute -> Service
+LAN/tailnet  -> split DNS ----------------> Envoy -> HTTPRoute -> Service
+```
+
+Tunnel hostnames currently point directly at Services. Migrate them one by one
+only after the matching listener, certificate, HTTPRoute and authorization
+policy have been verified. Preserve Host and TLS SNI. Remove the global
+`no_tls_verify` after origin verification works. Keep an explicit public
+hostname allow-list rather than a wildcard tunnel rule.
+
+### Naming and DNS ownership
+
+AD remains `ad.ddupan.top`; moving it back to the apex would make Samba and
+Cloudflare competing authorities.
+
+| namespace/data | owner |
+|---|---|
+| AD SRV, Kerberos, LDAP and member records | Samba AD DNS |
+| infrastructure host A/PTR intent | NetBox, reconciled into Samba |
+| application names under `ddupan.top` | Git service declaration |
+| public application answers | Cloudflare DNS |
+| private answers for the same names | Blocky |
+| Kubernetes service discovery | CoreDNS |
+
+Machine and service identity are intentionally distinct: `pve1.ad.ddupan.top`
+names one host, while `git.ddupan.top` names a movable service. Split-horizon
+DNS necessarily returns two answers; the goal is one human-authored service
+intent that drives Cloudflare, Blocky, Gateway API and Backstage.
+
+After validation, Blocky should serve LAN and Tailscale clients. CoreDNS should
+forward `ad.ddupan.top` to Samba and application split-DNS queries to Blocky
+instead of keeping one template per hostname. Keep the router as secondary DNS
+so laptop failure degrades to the public route rather than a household outage.
+
+Samba stays an AD-specific authority. Its current tasks are add-only; a future
+reconciler may update and delete only an explicit owned set and must never purge
+the zone or touch Samba-generated records.
+
+### Certificates
+
+Browser-facing internal services use Let's Encrypt with Cloudflare DNS-01. A
+private DNS answer does not prevent public ACME DNS validation. OpenBao remains
+the CA for LDAPS, database TLS, mTLS, SSH and machine identities.
+
+PVE UI/API must not depend on k3s. Keep direct break-glass endpoints at
+`pve1/2/3.ad.ddupan.top:8006` and change only the optional pveproxy certificate
+from OpenBao ACME to Let's Encrypt DNS-01. Never replace PVE's cluster-internal
+`pve-ssl.pem`. Use a dedicated DNS-only Cloudflare token.
+
+### OCI recovery island
+
+The home has no public IP; Tailscale is the management network. The minimal OCI
+shape therefore needs no public VM address or Load Balancer:
+
+```text
+home -> Tailscale -> private OCI A1 VM
+                         |-> NAT Gateway for outbound connectivity
+                         `-> Service Gateway -> Object Storage
+```
+
+The VM may hold a read-only Git mirror, external probes, recovery docs and
+backup verification, but is not a dependency of healthy home services. OCI LB
+and free MySQL remain optional for a later interactive backup Forgejo portal.
+
+### Terraform state
+
+- Recover the OCI root before changing backend or network architecture.
+- Never initialize an empty replacement with the same state name.
+- Take an encrypted independent backup before enabling bucket versioning.
+- Test two concurrent operations before relying on OCI S3 `use_lockfile`.
+- Gitea 1.27 provides a Terraform State Registry through Terraform HTTP backend,
+  including locking and state-version history. Confirm the running Gitea version
+  and test backup/restore before adopting it.
+- Prefer Gitea State Registry for local roots once validated. Keep the OCI recovery
+  root in OCI Object Storage initially so cloud recovery does not depend on the
+  home cluster or Gitea.
+- Existing SeaweedFS state remains authoritative until each root is migrated
+  deliberately; retain encrypted off-site state copies throughout migration.
+
+### Secret delivery
+
+OpenBao is the authority for workload secrets and External Secrets Operator is
+the Kubernetes delivery mechanism. Git contains `ClusterSecretStore` and
+`ExternalSecret` intent only; ESO materializes ordinary Kubernetes Secrets.
+
+This path already exists: ESO uses its ServiceAccount JWT, the OpenBao role is
+bound to that service account and namespace, and policy access is read-only under
+`kv/k8s/*`. Before expanding IaC automation, re-verify live reconciliation,
+OpenBao snapshot recovery and application behaviour during an OpenBao outage,
+then migrate any remaining manually managed Secrets one service at a time.
+
+OpenBao bootstrap, unseal/recovery material and backup credentials require an
+offline break-glass path. ESO-generated Secrets are projections, not backups.
+
+## Implementation phases
+
+1. Preserve OCI state and sanitized libvirt/Helm evidence.
+2. Re-verify the existing OpenBao/ESO delivery and recovery path, inventory
+   remaining manually managed Secrets, then migrate them incrementally.
+3. Configure Gitea remote plus a one-way off-site mirror. Confirm whether the
+   running Gitea supports the 1.27 Terraform State Registry.
+4. Deploy Woodpecker, then bootstrap Flux on `http-echo` or `marker` without
+   enabling prune until live ownership is audited.
+5. Move Tunnel origins to Envoy and consolidate split DNS through Blocky.
+6. Deploy Backstage read-only with Catalog, Kubernetes, Flux and TechDocs.
+7. Reconstruct the OCI root to a zero-change plan and add libvirt drift reports.
+8. Add dynamic PVE VM workers only when a real job requires one, with TTL cleanup
+   and hard concurrency/resource limits first.
+
+## Open decisions
+
+- Primary Git remote and off-site mirror location.
+- Running Gitea version, State Registry availability and independent backup path.
+- Whether OCI Object Storage passes the concurrent lockfile test.
+- Whether the OCI VM should later move from its current public subnet.
+- Schema/generator for the Git-owned service declaration.
+- Which live Helm releases are absent from or differ from Git.
+- Whether each local libvirt VM should autostart.
+- Which first job genuinely requires a dynamic Proxmox VM.
+
+## Safety invariants
+
+- Imported Terraform roots require a zero-change plan before apply.
+- Existing zvols, VM boot disks and OCI boot volumes are protected from deletion.
+- No runner image or VM template contains a long-lived credential.
+- OpenBao recovery material has an offline break-glass copy; ESO output is never
+  treated as the only secret backup.
+- Flux deploys Kubernetes; Backstage observes and opens pull requests.
+- PVE, OpenBao and Git retain recovery paths independent of k3s.
+- No credential or Terraform state content is committed or copied into an issue.

docs/superpowers/RETIRED.md

@@ -0,0 +1,29 @@
+# docs/superpowers — RETIRED 2026-07-28
+
+The `plans/` + `specs/` workflow is no longer used. Its one piece of work is
+**done and live**, so nothing here is pending.
+
+**What it covered:** the 2026-04-18 migration of shared PostgreSQL to
+CloudNativePG on OpenEBS ZFS —
+`specs/2026-04-18-postgresql-cnpg-zfs-migration-design.md` (design) and
+`plans/2026-04-18-postgresql-cnpg-zfs-migration.md` (task-by-task plan).
+
+**Outcome: shipped.** `kubectl get cluster -n shared-db` shows
+`shared-postgresql`, 1 instance, *"Cluster in healthy state"*, 101 days old, with
+`shared-postgresql-1` Running. The authoritative manifests live in
+`../../shared-postgresql/` (`cloudnativepg-cluster.yaml`,
+`shared-postgresql-service.yaml`, `serviceaccount.yaml`), and the dump/restore
+/cutover/rollback runbook is `../../shared-postgresql/migration.md`.
+
+**Why retired:** the format carried agent-workflow scaffolding ("REQUIRED
+SUB-SKILL", checkbox task lists) that only ever suited one migration. Design
+documents now live directly in `docs/` — see `../cicd.md` — and the split that
+matters is:
+
+- `CHANGELOG.md` — what changed, for humans
+- `CLAUDE.md` — traps and procedures, for agents
+- `docs/*.md` — design docs for work not yet built
+- `<service>/README.md` — how a service actually works
+
+The two files are kept for the migration rationale, which is still the best
+record of why CNPG and ZFS were chosen. Nothing reads them automatically.

docs/superpowers/plans/2026-04-18-postgresql-cnpg-zfs-migration.md

@@ -0,0 +1,193 @@
+# PostgreSQL CloudNativePG Migration Implementation Plan
+
+> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
+
+**Goal:** Replace the legacy shared PostgreSQL deployment with a single-instance CloudNativePG cluster on OpenEBS ZFS while preserving the existing `shared-postgresql` service name for the apps that already depend on it.
+
+**Architecture:** Keep the old PostgreSQL deployment alive until the new CNPG cluster is ready, then migrate data with a logical dump/restore, swap the service endpoints, and retire the Helm-based deployment only after validation passes. The new cluster stays single-instance because there is only one worker node.
+
+**Tech Stack:** Kubernetes manifests, CloudNativePG operator, OpenEBS ZFS storage, `kubectl`, PostgreSQL client tools.
+
+---
+
+## File Map
+
+- `shared-postgresql/cloudnativepg-cluster.yaml`: new authoritative CNPG cluster manifest.
+- `shared-postgresql/tailscale-loadbalancer.yaml`: keep the existing external service, but retarget it to the CNPG primary during cutover.
+- `shared-postgresql/shared-postgresql-values.yaml`: legacy Helm values to delete after the rollback window closes.
+- `shared-postgresql/shared-postgresql-init.sql`: reusable SQL used during migration to recreate app databases and roles.
+- `shared-postgresql/migration.md`: runbook for dump, restore, cutover, and rollback.
+
+### Task 1: Add the CNPG cluster manifest
+
+**Files:**
+- Create: `shared-postgresql/cloudnativepg-cluster.yaml`
+
+- [ ] **Step 1: Confirm the OpenEBS ZFS storage class name**
+
+Run: `kubectl get storageclass`
+
+Expected: one OpenEBS-backed ZFS class is available and can be copied into `shared-postgresql/cloudnativepg-cluster.yaml` before apply.
+
+- [ ] **Step 2: Write the cluster manifest**
+
+```yaml
+apiVersion: postgresql.cnpg.io/v1
+kind: Cluster
+metadata:
+  name: shared-postgresql
+  namespace: shared-db
+spec:
+  instances: 1
+  enableSuperuserAccess: true
+  storage:
+    size: 10Gi
+  bootstrap:
+    initdb:
+      database: shared_data
+      owner: sharedadmin
+```
+
+Add the `storage.storageClass` line using the exact OpenEBS ZFS class name discovered in Step 1.
+
+- [ ] **Step 3: Verify the manifest is structurally valid**
+
+Run: `kubectl apply --dry-run=server -f shared-postgresql/cloudnativepg-cluster.yaml`
+
+Expected: the API server accepts the `Cluster` object once the CNPG CRD is installed.
+
+- [ ] **Step 4: Apply the cluster manifest in the staging namespace**
+
+Run: `kubectl apply -f shared-postgresql/cloudnativepg-cluster.yaml`
+
+Expected: the CNPG cluster is created and starts provisioning a single primary pod.
+
+- [ ] **Step 5: Wait for the cluster to become ready**
+
+Run: `kubectl wait -n shared-db --for=condition=Ready cluster/shared-postgresql --timeout=15m`
+
+Expected: the cluster reaches a ready state before any data migration begins.
+
+### Task 2: Write the migration runbook
+
+**Files:**
+- Create: `shared-postgresql/migration.md`
+
+- [ ] **Step 1: Document the source and target endpoints**
+
+```text
+Source service: shared-postgresql.shared-db.svc.cluster.local:5432
+Target rw service: shared-postgresql-rw.shared-db.svc.cluster.local:5432
+```
+
+- [ ] **Step 2: Document the dump and restore commands**
+
+```text
+Pause writes from the dependent apps before taking the dump so the logical backup is consistent.
+```
+
+```bash
+PGPASSWORD="$OLD_SHAREDADMIN_PASSWORD" pg_dump -h shared-postgresql.shared-db.svc.cluster.local -U sharedadmin -Fc -d shared_data -f shared_data.dump
+PGPASSWORD="$OLD_GITEA_PASSWORD" pg_dump -h shared-postgresql.shared-db.svc.cluster.local -U gitea -Fc -d gitea -f gitea.dump
+PGPASSWORD="$OLD_CASDOOR_PASSWORD" pg_dump -h shared-postgresql.shared-db.svc.cluster.local -U casdoor -Fc -d casdoor -f casdoor.dump
+
+PGPASSWORD="$NEW_SHAREDADMIN_PASSWORD" psql -h shared-postgresql-rw.shared-db.svc.cluster.local -U sharedadmin -d postgres -f shared-postgresql/shared-postgresql-init.sql
+PGPASSWORD="$NEW_SHAREDADMIN_PASSWORD" pg_restore -h shared-postgresql-rw.shared-db.svc.cluster.local -U sharedadmin -d shared_data shared_data.dump
+PGPASSWORD="$NEW_GITEA_PASSWORD" pg_restore -h shared-postgresql-rw.shared-db.svc.cluster.local -U gitea -d gitea gitea.dump
+PGPASSWORD="$NEW_CASDOOR_PASSWORD" pg_restore -h shared-postgresql-rw.shared-db.svc.cluster.local -U casdoor -d casdoor casdoor.dump
+```
+
+- [ ] **Step 3: Document the rollback decision point**
+
+```text
+If restore or validation fails, keep the old Helm deployment active and do not delete its Service or PVC.
+```
+
+- [ ] **Step 4: Review the runbook for exact cutover order**
+
+Expected: the document explains that the old service stays live until data restore and app checks pass.
+
+### Task 3: Cut over services to CNPG
+
+**Files:**
+- Modify: `shared-postgresql/tailscale-loadbalancer.yaml`
+- Create: `shared-postgresql/shared-postgresql-service.yaml`
+
+- [ ] **Step 1: Add the compatibility ClusterIP service**
+
+```yaml
+apiVersion: v1
+kind: Service
+metadata:
+  name: shared-postgresql
+  namespace: shared-db
+spec:
+  type: ClusterIP
+  selector:
+    cnpg.io/cluster: shared-postgresql
+    cnpg.io/instanceRole: primary
+  ports:
+    - name: postgres
+      port: 5432
+      targetPort: 5432
+```
+
+- [ ] **Step 2: Update the Tailscale LoadBalancer selector**
+
+```yaml
+apiVersion: v1
+kind: Service
+metadata:
+  name: shared-postgresql-tailscale
+  namespace: shared-db
+spec:
+  type: LoadBalancer
+  loadBalancerClass: tailscale
+  ports:
+    - name: tcp-postgresql
+      port: 5432
+      protocol: TCP
+      targetPort: 5432
+  selector:
+    cnpg.io/cluster: shared-postgresql
+    cnpg.io/instanceRole: primary
+```
+
+- [ ] **Step 3: Apply the compatibility service only after the Helm release is removed**
+
+Run: `kubectl apply -f shared-postgresql/shared-postgresql-service.yaml -f shared-postgresql/tailscale-loadbalancer.yaml`
+
+Expected: the original `shared-postgresql` hostname resolves to the CNPG primary and Tailscale reaches the same pod.
+
+- [ ] **Step 4: Validate application connectivity**
+
+Run: `PGPASSWORD="$NEW_SHAREDADMIN_PASSWORD" psql -h shared-postgresql.shared-db.svc.cluster.local -U sharedadmin -d shared_data -c 'select 1;'`
+
+Expected: the compatibility Service resolves and accepts a real PostgreSQL login after the swap.
+
+### Task 4: Remove the legacy Helm deployment
+
+**Files:**
+- Delete: `shared-postgresql/shared-postgresql-values.yaml`
+
+- [ ] **Step 1: Confirm all dependent apps are using the CNPG-backed Service**
+
+Run: `kubectl get endpoints -n shared-db shared-postgresql`
+
+Expected: the endpoints point at the CNPG pod, not the old Helm chart pod.
+
+- [ ] **Step 2: Remove the old Helm values file after the rollback window**
+
+Run: `git rm shared-postgresql/shared-postgresql-values.yaml`
+
+Expected: the repository no longer advertises the retired deployment path once cutover is stable.
+
+- [ ] **Step 3: Keep the bootstrap SQL file as the migration reference**
+
+Expected: `shared-postgresql/shared-postgresql-init.sql` remains in the tree until the migration is fully complete and documented.
+
+- [ ] **Step 4: Re-run the manifest validation pass**
+
+Run: `kubectl apply --dry-run=server -f shared-postgresql/`
+
+Expected: the remaining manifest set is clean and consistent.

docs/superpowers/specs/2026-04-18-postgresql-cnpg-zfs-migration-design.md

@@ -0,0 +1,77 @@
+# PostgreSQL Migration Design
+
+## Goal
+Move the current shared PostgreSQL deployment to CloudNativePG and back it with the existing OpenEBS ZFS storage layer.
+
+The target state is a single PostgreSQL cluster managed by CloudNativePG, running one instance on the only worker node, with one shared database service for the apps that already use `shared-postgresql`.
+
+## Current State
+- PostgreSQL is currently deployed from chart values in `shared-postgresql/shared-postgresql-values.yaml`.
+- Storage currently uses `local-path`.
+- Apps such as Gitea and Casdoor point at the shared PostgreSQL service.
+- There is an init SQL file for bootstrap data in `shared-postgresql/shared-postgresql-init.sql`.
+
+## Target State
+- CloudNativePG manages the PostgreSQL lifecycle.
+- Storage comes from the OpenEBS-provided ZFS-backed `StorageClass`.
+- The cluster runs with `instances: 1` because there is only one worker node.
+- Existing app databases remain in the same PostgreSQL cluster.
+- A compatibility Kubernetes `Service` preserves the existing shared database hostname or provides an equivalent stable alias.
+
+## Non-Goals
+- High availability across multiple nodes.
+- Automatic failover.
+- Database sharding or splitting apps into separate clusters.
+- Changing application-level schemas unless required by migration.
+
+## Approach
+1. Provision or verify the OpenEBS ZFS `StorageClass` suitable for PostgreSQL PVCs.
+2. Deploy CloudNativePG and create one cluster with a single instance.
+3. Create the required role and database layout for the shared apps.
+4. Migrate data from the current PostgreSQL instance with a logical dump and restore.
+5. Cut over workloads to the new service.
+6. Validate app logins, migrations, and basic read/write behavior.
+
+## Migration Plan
+### Phase 1: Storage
+- Confirm the OpenEBS ZFS components are installed and healthy.
+- Create a dedicated PostgreSQL `StorageClass` for CNPG.
+- Use a single PVC for the primary instance.
+
+### Phase 2: Database Deployment
+- Install CloudNativePG.
+- Create a `Cluster` manifest with one instance.
+- Set resource requests and limits close to the current PostgreSQL footprint.
+- Mount any required bootstrap SQL through CloudNativePG-supported init methods.
+
+### Phase 3: Data Migration
+- Quiesce writes on the current PostgreSQL instance.
+- Take a logical backup of the existing databases and roles, including globals such as users and grants.
+- Restore into the new CloudNativePG cluster.
+- Recreate any app-specific users, grants, and schemas that are not captured automatically.
+
+### Phase 4: Cutover
+- Point app secrets or connection settings at the new CNPG service.
+- Restart or roll the dependent workloads.
+- Verify each app can connect and operate normally.
+
+### Phase 5: Cleanup
+- Keep the old deployment available until validation completes.
+- Remove the old PostgreSQL deployment only after the new cluster is confirmed healthy.
+
+## Risks
+- OpenEBS ZFS must be healthy before PostgreSQL migration starts.
+- Single-node deployment means node failure still causes downtime.
+- Logical restore can miss permissions or extension details if the old instance has custom setup.
+- Existing app passwords should be preserved or rotated carefully during cutover.
+
+## Validation
+- CNPG cluster reports healthy.
+- PVC is bound on the ZFS-backed storage class.
+- Existing apps can connect using the shared PostgreSQL service.
+- Basic read/write checks succeed for each app database.
+- Restarting the PostgreSQL pod does not lose data.
+
+## Rollback
+- If restore or validation fails, keep the current PostgreSQL deployment active and point apps back to the old service.
+- Do not delete the old PVC or deployment until the new cluster passes validation.