homelab-infra/docs/cicd.md

CI/CD — what we are building

Status: design, partly built. Stage 1 is live. The substrate is undecided. Started 2026-07-28.

Goal

Every infrastructure change goes through a pipeline that tests it, and that pipeline is the only path to production.

The second half is the point. Tests you can bypass with helm upgrade are not controls, they are suggestions. The cluster already shows what that looks like: authelia is on revision 14 and e5renew on revision 28, all hand-run. Nothing is wrong with those releases — but nothing proves anything about them either, and two of the thirteen deployed Helm releases (e5renew, rustfs) have no representation in this repo at all.

So the aim is not "add CI". It is to make the tested path the only path, and to make drift between this repo and reality visible when it happens.

Current state

git	History since 2026-07-28. Four commits, no remote yet
stage 1	Live and green — yamllint, ansible-lint, terraform fmt/validate. Configs tuned against a real run
gitea	1.25.5, Actions enabled, DEFAULT_ACTIONS_URL=github. No runner deployed, so nothing executes
ansible	33 roles across infrastructure/proxmox/, infrastructure/samba-ad/, infrastructure/openbao/
terraform	4 roots, local state, each with different interactive auth (bao login -method=oidc, az login)
k8s	~13 Helm releases, all deployed by hand
secrets	4 config files are gitignored because they embed live secrets, so their contents are not version controlled

The four stages

Stage 1 — static. Built. yamllint, ansible-lint, terraform fmt -check / validate -backend=false. No cluster, no credentials, no mutation, so it is safe on every push. It already found a real defect: infrastructure/proxmox/ansible/ had no requirements.yml at all, so a fresh checkout could not reproduce its collections.

Stage 2 — server-side dry-run. kubectl apply --dry-run=server. This is the one that is usually skipped and matters most here: the cluster runs Gateway API, cert-manager, the VictoriaMetrics operator and Envoy SecurityPolicy CRDs. Offline schema validation cannot see any of them; server-side dry-run validates against the real CRDs and admission webhooks. Needs cluster credentials, so it lives in a separate workflow from stage 1 — a credential problem there must never be able to block the static gate.

Stage 3 — ephemeral integration.

• Ansible: Molecule. Its default scenario asserts idempotence, which is already this repo's written acceptance test ("a second run must report changed=0"). The standard exists; Molecule just makes something other than a human enforce it.
• Kubernetes: a throwaway k3d cluster — apply everything, assert it converges. Catches ordering and cross-resource problems a dry-run cannot.

Realistic coverage: roughly a third of the 33 roles. vyos_router (no Python interpreter), pve_cluster, pve_linstor and the Windows-join roles need real hardware or nested VMs on Proxmox. Those stay --check --diff against live.

Stage 4 — deploy and drift. Flux reconciles Kubernetes from git. Terraform and Ansible get scheduled drift detection, not auto-apply: terraform plan -detailed-exitcode and ansible-playbook --check --diff on a timer, alerting into the VictoriaMetrics stack that already exists. That is where most of the value is and almost none of the risk.

What enforces what

There are only two enforcement mechanisms available, and this lab has both:

• Kubernetes → reconciliation. Flux is the only writer. A hand-run helm upgrade gets reverted on the next interval. Enforcement is free.
• Ansible and Terraform → credentials. No reconciler exists for them and one should not be invented; nothing stops someone SSHing to a PVE node and running pveum by hand. The lever is OpenBao's SSH CA: if humans hold no standing SSH access and the pipeline is the only identity that can get a signed cert, the pipeline becomes the only path by credential control. Same shape for Terraform — keep the AppRole and the Azure service principal only in CI.

Credential lockdown comes last. It is the disruptive step and is only worth doing once the pipeline is trustworthy. A documented break-glass path must survive it: the playbooks still have to run by hand from the laptop during an incident, or enforcement locks you out exactly when you need in.

Substrate — open

Requirement: a light always-on orchestrator with ephemeral workers. Not a VM or LXC parked idle waiting for work. Something must always listen — that is inherent to event-driven CI — but it should be a controller, not a pet.

option	verdict
Woodpecker CI	Front-runner. Kubernetes backend runs each step as a standalone Pod; first-class Gitea (OAuth2 + auto-created webhooks); server + agent, both light; stable. Costs a different pipeline syntax, which for shell-step lint jobs is ~20 lines
act_runner (Gitea native)	Docker or host execution only — no Kubernetes executor. Confirmed in source: labels.go has just SchemeDocker/SchemeHost, and run_context.go branches only to startHostEnvironment or startJobContainer. Cheapest (one pod) but leaves a persistent worker
Gitea ARC	Real pod-per-job operator, but Enterprise Edition only
GARM + garm-provider-k8s	Right shape. GARM supports Gitea from 1.24, but latest is v0.2.0-beta1 and the provider documents GitHub runners only — the Gitea pairing is unverified
Write a runner from zero	The protocol (actions-proto-go) is approachable; reimplementing execution is not — act_runner delegates that to a vendored nektos/act. A shim spawning one-shot pods is blocked on ephemeral registration (go-apps/gitea/gitea#32461)

Where runners run is a separate axis. Argument for Proxmox: pve1–3 have clean egress, while the k3s node carries openvpn-client@naist, whose 58 split-tunnel routes blackhole Cloudflare, Fastly and Microsoft ranges whenever the tunnel dies — and pods inherit the host routing table. CI on the laptop will fail mid-build in confusing ways every time that happens. Argument against: no GARM Proxmox provider exists, so ephemeral Proxmox VMs would mean writing one.

Prerequisites

Blocking, in order:

1. Externalise secrets. apps/gitea/gitea-values.yaml, apps/authelia/values.yaml, infrastructure/cloudflared/cloudflared.yaml and litellm-gateway/docker-compose.yml are gitignored because they embed live credentials. GitOps requires the opposite — a reconciler can only apply what is in git. OpenBao + External Secrets is the natural fit; openbao_bootstrap/tasks/auth_kubernetes.yml already exists.
2. A git remote. Gitea is the obvious host, but it runs on the cluster it would deploy, so keep an off-cluster mirror for disaster recovery.
3. Terraform: shared state. All four roots use local *.tfstate. Any pipeline that applies needs shared state — the S3-compatible SeaweedFS/rustfs already on the cluster is a candidate, with the caveat that cluster state living on the cluster is the same circularity as hosting git there.

Not a blocker: non-interactive auth. An earlier draft of this document called bao login -method=oidc and az login hard blockers that would widen the blast radius. That was wrong on both counts.

• OpenBao — the Kubernetes auth backend is already bootstrapped (infrastructure/openbao/ansible/roles/openbao_bootstrap/tasks/auth_kubernetes.yml). A CI pod authenticates with its ServiceAccount JWT and stores no credential at all. AppRole is the equivalent for anything running off-cluster.
• Azure — the provider supports service principal with a client secret, service principal with a client certificate, managed identity, and OIDC workload identity federation (use_oidc = true with oidc_token_file_path, which explicitly covers generic OIDC providers, not just GitHub/ADO). HashiCorp's own guidance is to use a service principal or MSI specifically when running non-interactively in CI, and reserve the Azure CLI for local runs.

It also narrows privilege rather than widening it. The current path needs an az login as Global Admin / Privileged Role Admin — because that is what creating app registrations and granting admin consent requires. A scoped service principal managing steady state is dramatically less privileged than the human identity in use today. Same for OpenBao: a per-root AppRole or Kubernetes role can be scoped to exactly the paths that root owns, whereas an interactive OIDC admin login is not.

The blast-radius concern in CLAUDE.md is about merging the roots, not about machine identities. Per-root scoped credentials preserve that separation and reduce privilege at the same time.

The honest caveat is a split, not a blocker: app-registration creation and admin consent are genuinely high-privilege bootstrap operations and should stay manual. CI should hold an identity that manages steady state only. The cleanest assembly with what already exists: keep the Azure SP certificate in OpenBao, and let the CI pod fetch it via Kubernetes auth — so nothing long-lived is stored in the CI system itself.

Non-goals

• High availability. One node. Nothing to fail over to. GitOps here buys drift elimination and rebuild-from-scratch, not uptime.
• Autoscaling. Nothing to scale across.
• Untrusted-workload isolation. One author. This changes the day it stops being true, and the runner's trust model must be revisited then.
• Terraform or Ansible under a Kubernetes reconciler. More machinery on the single node that is also NFS server, libvirt host and netboot appliance makes recovery harder, not easier.
• Merging the Terraform roots. Already considered and rejected; see CLAUDE.md.

Sequencing

1. Externalise secrets — unblocks everything, valuable on its own
2. Capture e5renew and rustfs into the repo (helm get values)
3. Git remote
4. Pick the substrate; stand it up in an isolated namespace
5. Stage 2, then stage 3 on one container-friendly role first
6. Flux on one low-stakes namespace (http-echo or marker)
7. Drift detection for Terraform and Ansible — scoped machine identities (OpenBao Kubernetes auth / AppRole, a steady-state Azure SP) land here, and are a privilege reduction on the admin logins used today
8. Credential lockdown — remove standing human access, with break-glass documented. Last, and only once the pipeline has earned trust

Steps 1, 2 and 7 are worth doing even if the CI substrate is never chosen — they are just "the repo should describe reality".