homelab-infra/docs/homelab-gitops-redesign.md

# Homelab GitOps and IaC redesign

Status: **implementation in progress; CI、Flux bootstrap、漂移修复与受控 prune 已验证。**

Started 2026-09-09. This is the durable record of the redesign discussion. It
separates observations, decisions and open work so an assumption cannot silently
become a destructive migration.

## Goals

- Make Git the source of infrastructure intent and the normal path to production.
- Reconcile Kubernetes continuously; gate Terraform and Ansible with plans and approval.
- Adopt existing infrastructure before changing it; avoid recreating VMs or disks.
- Keep recovery possible when k3s, Gitea, OpenBao or the home WAN is unavailable.
- Keep OCI within Always Free allowances except deliberately tiny Object Storage use.

## Observed state

### Local host

The laptop is the single-node k3s host, NFS server and libvirt host. These
persistent libvirt domains were observed on 2026-09-09:

| domain | state | CPU | memory | root disk | network |
|---|---|---:|---:|---|---|
| `dc1` | running | 2 | 2 GiB | `data/vm/dc1` zvol | bridge `br0` |
| `winadmin` | running | 4 | 6 GiB | `data/vm/winadmin` zvol | bridge `br0` |
| `bao1` | running | 1 | 1 GiB | `data/vm/bao1` zvol | bridge `br0` |
| `win2k25` | stopped | 2 | 4 GiB | `data/vm/win2k25` zvol | network `network` |

All four have autostart disabled. The intended values for `dc1`, `winadmin`
and `bao1` mostly agree with live configuration. Current Ansible roles stop
managing a VM once its domain exists. That avoids recreation but cannot report
or reconcile later CPU, memory, NIC or boot drift.

### Kubernetes and delivery

- Kubernetes is a single-node k3s `v1.36.4+k3s1` cluster.
- Helm releases and manifests have historically been applied by hand and are now
  being adopted by Flux one release at a time.
- Flux `v2.9.5` is live. Its internal Gitea source and root Kustomization are
  Ready; `http-echo` proved automatic deployment, drift repair and scoped prune.
- Repository history records External Secrets Operator 2.8.0 as deployed. Five
  `ExternalSecret` resources cover Authelia, Gitea, Cloudflared and SeaweedFS.
  All five reported `SecretSynced=True` during a live check on 2026-09-09.
- Gitea Actions now has one instance-scoped runner in namespace `gitea-actions`.
  Its Helm release is deployed, its Pod is `2/2 Running`, its identity PVC is
  bound, and a sixth `ExternalSecret` delivers the registration token from OpenBao.
- The repository is hosted at `panxiao81/homelab-infra` on the local Gitea
  instance. A one-way off-site mirror is still missing.
- Terraform roots remain per-service and must not be merged.
- The k3s node was `Ready` on 2026-09-09. The Snap-packaged `kubectl` could
  not start because the user systemd session was degraded; `k3s kubectl` with the
  same user kubeconfig reached the API successfully. This is a local client issue,
  not evidence that the cluster is down.
- `docs/cicd.md` remains useful but some inventory statements are stale.

### OCI

Read-only OCI discovery found:

- home region `ap-osaka-1`;
- bucket `oci-k8s-free-tier-tfstate` in the root compartment;
- one approximately 25 KiB object, `terraform.tfstate`, modified 2026-08-15;
- state format 4, Terraform 1.15.8, serial 249;
- state for a VM, VCN, public subnet, Internet Gateway, route table, security
  list and free-tier quota;
- Object Storage versioning is not enabled;
- the source Terraform root was not found locally.

The state is likely authoritative. Its contents are sensitive and must not be
printed or committed. Restore its configuration to a zero-change plan before
considering a public-to-private subnet redesign.

## Agreed architecture

### Control planes

| scope | controller | application model |
|---|---|---|
| Kubernetes | Flux | continuous pull reconciliation after merge |
| Terraform | CI | plan; explicit approval before apply |
| Ansible | CI | check/diff; explicit approval before execution |
| Backstage | none | read-only portal and PR authoring |

Flux was selected over Argo CD for a small cluster whose primary interface
should remain Git and Kubernetes resources. Backstage supplies the unified UI
without becoming another deployment controller.

### CI execution

Most CI uses Gitea Actions for its GitHub Actions compatibility. A persistent
Gitea Runner StatefulSet runs in Kubernetes with Docker-in-Docker and capacity
four; individual job containers are created dynamically. The chart requires a
privileged DinD container in both modes, and rootlesskit is blocked by the node's
AppArmor unprivileged-userns policy, so regular DinD is used instead of weakening
that host-wide policy. The runner is instance-scoped and accepts trusted
repositories and workflows only.

Only explicitly labelled jobs needing privilege, nested virtualization,
amd64-only software or isolation from k3s use an ephemeral Proxmox VM. IaC owns
the PVE template, pool, permissions, network and quotas; the scheduler owns the
short-lived clone/start/run/destroy lifecycle. Ephemeral workers do not enter
Terraform state. OCI A1 may later run OCI plans and external checks.

### Ingress

Envoy Gateway becomes the sole in-cluster L7 routing and policy point:

```text
public client -> Cloudflare edge -> Tunnel -> Envoy -> HTTPRoute -> Service
LAN/tailnet  -> split DNS ----------------> Envoy -> HTTPRoute -> Service
```

Tunnel hostnames currently point directly at Services. Migrate them one by one
only after the matching listener, certificate, HTTPRoute and authorization
policy have been verified. Preserve Host and TLS SNI. Remove the global
`no_tls_verify` after origin verification works. Keep an explicit public
hostname allow-list rather than a wildcard tunnel rule.

### Naming and DNS ownership

AD remains `ad.ddupan.top`; moving it back to the apex would make Samba and
Cloudflare competing authorities.

| namespace/data | owner |
|---|---|
| AD SRV, Kerberos, LDAP and member records | Samba AD DNS |
| infrastructure host A/PTR intent | NetBox, reconciled into Samba |
| application names under `ddupan.top` | Git service declaration |
| public application answers | Cloudflare DNS |
| private answers for the same names | Blocky |
| Kubernetes service discovery | CoreDNS |

Machine and service identity are intentionally distinct: `pve1.ad.ddupan.top`
names one host, while `git.ddupan.top` names a movable service. Split-horizon
DNS necessarily returns two answers; the goal is one human-authored service
intent that drives Cloudflare, Blocky, Gateway API and Backstage.

After validation, Blocky should serve LAN and Tailscale clients. CoreDNS should
forward `ad.ddupan.top` to Samba and application split-DNS queries to Blocky
instead of keeping one template per hostname. Keep the router as secondary DNS
so laptop failure degrades to the public route rather than a household outage.

Samba stays an AD-specific authority. Its current tasks are add-only; a future
reconciler may update and delete only an explicit owned set and must never purge
the zone or touch Samba-generated records.

### Certificates

Browser-facing internal services use Let's Encrypt with Cloudflare DNS-01. A
private DNS answer does not prevent public ACME DNS validation. OpenBao remains
the CA for LDAPS, database TLS, mTLS, SSH and machine identities.

PVE UI/API must not depend on k3s. Keep direct break-glass endpoints at
`pve1/2/3.ad.ddupan.top:8006` and change only the optional pveproxy certificate
from OpenBao ACME to Let's Encrypt DNS-01. Never replace PVE's cluster-internal
`pve-ssl.pem`. Use a dedicated DNS-only Cloudflare token.

### OCI recovery island

The home has no public IP; Tailscale is the management network. The minimal OCI
shape therefore needs no public VM address or Load Balancer:

```text
home -> Tailscale -> private OCI A1 VM
                         |-> NAT Gateway for outbound connectivity
                         `-> Service Gateway -> Object Storage
```

The VM may hold a read-only Git mirror, external probes, recovery docs and
backup verification, but is not a dependency of healthy home services. OCI LB
and free MySQL remain optional for a later interactive backup Forgejo portal.

### Terraform state

- Recover the OCI root before changing backend or network architecture.
- Never initialize an empty replacement with the same state name.
- Take an encrypted independent backup before enabling bucket versioning.
- Test two concurrent operations before relying on OCI S3 `use_lockfile`.
- Gitea 1.27 provides a Terraform State Registry through Terraform HTTP backend,
  including locking and state-version history. Confirm the running Gitea version
  and test backup/restore before adopting it.
- Prefer Gitea State Registry for local roots once validated. Keep the OCI recovery
  root in OCI Object Storage initially so cloud recovery does not depend on the
  home cluster or Gitea.
- Existing SeaweedFS state remains authoritative until each root is migrated
  deliberately; retain encrypted off-site state copies throughout migration.

### Secret delivery

OpenBao is the authority for workload secrets and External Secrets Operator is
the Kubernetes delivery mechanism. Git contains `ClusterSecretStore` and
`ExternalSecret` intent only; ESO materializes ordinary Kubernetes Secrets.

This path already exists: ESO uses its ServiceAccount JWT, the OpenBao role is
bound to that service account and namespace, and policy access is read-only under
`kv/k8s/*`. Before expanding IaC automation, re-verify live reconciliation,
OpenBao snapshot recovery and application behaviour during an OpenBao outage,
then migrate any remaining manually managed Secrets one service at a time.

OpenBao bootstrap, unseal/recovery material and backup credentials require an
offline break-glass path. ESO-generated Secrets are projections, not backups.

## Implementation phases

1. **In progress:** sanitized libvirt/Helm evidence is recorded; the independent
   encrypted OCI state backup is still missing.
2. **In progress:** live OpenBao/ESO delivery is verified; verify recovery and inventory
   remaining manually managed Secrets, then migrate them incrementally.
3. **In progress:** the Gitea remote exists; add a one-way off-site mirror and
   revisit the Terraform State Registry after upgrading beyond Gitea 1.25.5.
4. **Complete:** the reviewed Gitea Runner and Stage 1 CI are live. Flux deploys
   `http-echo`; automatic deployment, replica drift repair and scoped deletion
   were verified. Root prune remains disabled for brownfield safety.
5. **In progress:** Gitea Actions and Gitea are managed by Flux. Adopt External
   Secrets Operator next with its existing chart `2.8.0` and repository values;
   register the suspended release first, then activate it in a separate PR after
   proving the fixed render matches Helm's stored manifest.
6. Move Tunnel origins to Envoy and consolidate split DNS through Blocky.
7. Deploy Backstage read-only with Catalog, Kubernetes, Flux and TechDocs.
8. Reconstruct the OCI root to a zero-change plan and add libvirt drift reports.
9. Add dynamic PVE VM workers only when a real job requires one, with TTL cleanup
   and hard concurrency/resource limits first.

## Open decisions

- Primary Git remote and off-site mirror location.
- Running Gitea version, State Registry availability and independent backup path.
- Whether OCI Object Storage passes the concurrent lockfile test.
- Whether the OCI VM should later move from its current public subnet.
- Schema/generator for the Git-owned service declaration.
- Migration order for Helm releases after the `gitea-actions` adoption.
- Whether each local libvirt VM should autostart.
- Which first job genuinely requires a dynamic Proxmox VM.

## Safety invariants

- Imported Terraform roots require a zero-change plan before apply.
- Existing zvols, VM boot disks and OCI boot volumes are protected from deletion.
- No runner image or VM template contains a long-lived credential.
- OpenBao recovery material has an offline break-glass copy; ESO output is never
  treated as the only secret backup.
- Flux deploys Kubernetes; Backstage observes and opens pull requests.
- PVE, OpenBao and Git retain recovery paths independent of k3s.
- No credential or Terraform state content is committed or copied into an issue.