Establish clean homelab infrastructure baseline

Reorganize the brownfield repository, remove retired and generated artifacts, harden ignore rules, and record the GitOps/IaC redesign.
2026-09-09 16:47:20 +00:00
commit 88a02ababa
418 changed files with 50579 additions and 0 deletions
@@ -0,0 +1,2 @@
+# Real Cloudflare API token — keep secret.example.yaml as the committed template.
+secret.yaml
@@ -0,0 +1,113 @@
+# cert-manager — certificate issuance
+
+Gives LAN services on `ad.ddupan.top` real, auto-renewing X.509 certs. Added 2026-07-25
+so NetBox could be served at `https://netbox.ad.ddupan.top` without a hand-issued cert.
+
+- Chart: `jetstack/cert-manager` **v1.21.0**, namespace `cert-manager`
+- Two `ClusterIssuer`s, because the homelab has two ACME sources
+
+| issuer | CA | challenge | use it when |
+|---|---|---|---|
+| `letsencrypt` | Let's Encrypt (public) | DNS-01 via Cloudflare | browsers must trust it with no CA install |
+| `bao-acme` | OpenBao internal PKI (`../../infrastructure/openbao`) | HTTP-01 via the shared gateway | no WAN dependency, or the name must stay out of CT logs |
+
+Both were verified `Ready=True` (ACME account registered) on creation.
+
+## The wildcard, and why
+
+`certificate-wildcard-ad.yaml` issues **one** `*.ad.ddupan.top` cert into
+`envoy-gateway-system/wildcard-ad-ddupan-top-tls`, which the shared Gateway's `https`
+listener serves for every LAN service (`../envoy-gateway/gateway.yaml`).
+
+Consequences worth understanding:
+
+- **Adding a service costs no certificate work.** An `HTTPRoute` plus an A record in
+  `samba_ad_extra_a_records` (`../../infrastructure/samba-ad`) is the whole job. No Gateway edit.
+- **Internal hostnames stay out of Certificate Transparency logs.** Every Let's Encrypt
+  issuance is published publicly; per-host certs would make the internal estate
+  enumerable by anyone. One wildcard entry leaks one name.
+- The Secret **must** live in `envoy-gateway-system` (the Gateway's namespace): a listener may only reference a
+  Secret in the Gateway's own namespace unless a `ReferenceGrant` exists.
+- `*.ad.ddupan.top` matches one label only, so the apex `ad.ddupan.top` is listed as an
+  explicit second SAN.
+
+## Two traps that will cost you an hour each
+
+**1. DNS-01 self-check must not use the cluster resolver.**
+cert-manager polls authoritative nameservers for its `_acme-challenge` TXT record before
+asking the CA to validate. In-cluster, `ad.ddupan.top` is routed straight to the Samba AD
+DC (`../k3s/coredns-custom.yaml`), which is authoritative internally and knows nothing
+about a TXT record written into the **public** Cloudflare zone — so the self-check spins
+forever while the record is plainly there. Fixed in `values.yaml`:
+
+```yaml
+dns01RecursiveNameservers: "1.1.1.1:53,8.8.8.8:53"
+dns01RecursiveNameserversOnly: true
+```
+
+**2. DNS-01, not HTTP-01, for Let's Encrypt here.** `ad.ddupan.top` resolves only on the
+LAN, so LE cannot reach the host to validate. DNS-01 needs no inbound reachability and is
+the only challenge that can issue a wildcard.
+
+## Deploy
+
+```bash
+helm repo add jetstack https://charts.jetstack.io && helm repo update jetstack
+helm upgrade --install cert-manager jetstack/cert-manager --version v1.21.0 \
+  -n cert-manager --create-namespace -f cert-manager/values.yaml
+
+cp cert-manager/secret.example.yaml cert-manager/secret.yaml   # then fill in the token
+kubectl apply -f cert-manager/secret.yaml
+kubectl apply -f cert-manager/clusterissuer-letsencrypt.yaml \
+               -f cert-manager/clusterissuer-bao-acme.yaml
+kubectl apply -f cert-manager/certificate-wildcard-ad.yaml
+```
+
+The Cloudflare token is the same one `cloudflared/terraform` uses. Extract it with:
+
+```bash
+sed -nE 's/^[[:space:]]*cloudflare_api_token[[:space:]]*=[[:space:]]*"([^"]*)".*/\1/p' \
+  cloudflared/terraform/terraform.tfvars
+```
+
+> A naive `cut -d= -f2` also swallows the trailing comment on that line and yields a
+> 126-character "token" that fails with `Invalid format for Authorization header`.
+
+The Secret must be in the **cert-manager** namespace — a ClusterIssuer resolves solver
+Secret refs where cert-manager runs, not where the Certificate lives.
+
+## ⚠ Token scope
+
+That token carries **Account·Cloudflare Tunnel:Edit** as well as Zone·DNS:Edit, so
+anything able to read the Secret could rewrite tunnel routing, not just DNS. Reusing it
+was a deliberate call to avoid blocking; minting a token scoped to Zone·DNS:Edit on
+`ddupan.top` and swapping it in is a clean, self-contained follow-up.
+
+## Verify
+
+```bash
+kubectl get clusterissuer                                   # both Ready=True
+kubectl -n envoy-gateway-system get certificate              # wildcard Ready=True
+kubectl -n envoy-gateway-system get order,challenge          # empty once issued
+
+kubectl -n envoy-gateway-system get secret wildcard-ad-ddupan-top-tls \
+  -o jsonpath='{.data.tls\.crt}' | base64 -d | openssl x509 -noout -subject -issuer -enddate
+```
+
+Renewal is automatic at 30 days remaining (`renewBefore: 720h`), a deliberately wide
+window so a flaky WAN has many chances to retry.
+
+## Using the internal CA instead
+
+Point a Certificate at `bao-acme` when CT-log exposure or WAN dependence matters:
+
+```yaml
+issuerRef:
+  name: bao-acme
+  kind: ClusterIssuer
+```
+
+Issuance is capped by `default_directory_policy = role:bao-server`
+(`../../infrastructure/openbao/terraform/pki.tf`), which permits `ad.ddupan.top` subdomains only. Clients
+need the internal CA in their trust store — already true for the PVE nodes, the DC and
+Authelia, generally **not** for a browser.
@@ -0,0 +1,54 @@
+# Cert for the PUBLIC Authelia hostname, so it can also be served on the LAN.
+#
+# WHY this exists at all, when ../envoy-gateway/gateway.yaml says adding a service
+# needs no cert work: that promise holds only for `*.ad.ddupan.top`. Authelia is
+# reached at `auth.ddupan.top` — a different zone (Cloudflare is authoritative for
+# ddupan.top; the DC is authoritative only for ad.ddupan.top) and one label
+# shallower, so the wildcard cannot cover it.
+#
+# WHY serve a public name internally (split-horizon) rather than introduce an
+# internal alias: the OIDC issuer, every registered redirect_uri, and the session
+# cookie domain are all `auth.ddupan.top`. Changing the name Gitea talks to would
+# mean re-registering every client. Resolving the SAME name to the LAN changes
+# nothing Authelia knows about itself.
+#
+# WHAT BROKE WITHOUT IT (2026-07-28): auth.ddupan.top resolves to Cloudflare proxy
+# IPs (104.21.6.55 / 172.67.154.245). TCP/443 to both fails from this network,
+# persistently, while other Cloudflare IPs (104.16.132.229) connect fine. Gitea's
+# chart runs `gitea admin auth update-oauth` in an INIT container, which fetches
+# the discovery URL on every pod start — so Gitea CrashLoopBackOff'd on any
+# restart, and server-side token exchange timed out. Routing the name to the LAN
+# removes the public internet from an entirely in-cluster conversation. See
+# CLAUDE.md: "Internal name resolution must never depend on the WAN."
+#
+# CT-log note: ../cert-manager/certificate-wildcard-ad.yaml deliberately uses a
+# wildcard to keep internal hostnames out of Certificate Transparency logs. That
+# reasoning does not apply here — auth.ddupan.top is already public in CT via the
+# Cloudflare-facing cert, so naming it costs nothing.
+#
+# Lives in envoy-gateway-system because a Gateway listener may only reference a
+# Secret in the Gateway's own namespace.
+---
+apiVersion: cert-manager.io/v1
+kind: Certificate
+metadata:
+  name: auth-ddupan-top
+  namespace: envoy-gateway-system
+spec:
+  secretName: auth-ddupan-top-tls
+  issuerRef:
+    name: letsencrypt
+    kind: ClusterIssuer
+    group: cert-manager.io
+  commonName: "auth.ddupan.top"
+  dnsNames:
+    - "auth.ddupan.top"
+  # DNS-01 via Cloudflare (the letsencrypt ClusterIssuer's solver). Cloudflare stays
+  # authoritative for ddupan.top, so the challenge resolves publicly even though the
+  # A record we serve internally points at the LAN.
+  duration: 2160h    # 90d — Let's Encrypt maximum
+  renewBefore: 720h  # 30d
+  privateKey:
+    algorithm: ECDSA
+    size: 256
+    rotationPolicy: Always
@@ -0,0 +1,44 @@
+# Cert for the PUBLIC Gitea hostname, so it can also be served on the LAN.
+#
+# Same reasoning as certificate-auth-ddupan.yaml: `*.ad.ddupan.top` cannot cover
+# `git.ddupan.top` — different zone (Cloudflare is authoritative for ddupan.top,
+# the DC only for ad.ddupan.top) and one label shallower.
+#
+# WHY serve the public name internally rather than introduce git.ad.ddupan.top:
+# the remote URL ends up in every clone, every CI checkout, and every existing
+# working copy. Split-horizon on the SAME name means none of that has to change,
+# and a laptop that leaves the LAN still reaches Gitea through the tunnel with the
+# identical URL.
+#
+# WHAT IT AVOIDS: without this, `git push` goes laptop -> Cloudflare -> tunnel ->
+# back into the cluster the laptop is hosting. On 2026-07-28 that path was
+# blackholed for hours by a dead VPN tunnel, and the repo is exactly what you need
+# during an incident.
+#
+# CT-log note: the wildcard in certificate-wildcard-ad.yaml exists to keep internal
+# hostnames out of Certificate Transparency. That does not apply here —
+# git.ddupan.top is already public via the Cloudflare-facing cert.
+---
+apiVersion: cert-manager.io/v1
+kind: Certificate
+metadata:
+  name: git-ddupan-top
+  namespace: envoy-gateway-system
+spec:
+  secretName: git-ddupan-top-tls
+  issuerRef:
+    name: letsencrypt
+    kind: ClusterIssuer
+    group: cert-manager.io
+  commonName: "git.ddupan.top"
+  dnsNames:
+    - "git.ddupan.top"
+  # DNS-01 via Cloudflare, which stays authoritative for the zone — so the ACME
+  # challenge resolves publicly even though the A record we serve on the LAN
+  # points at the gateway.
+  duration: 2160h    # 90d — Let's Encrypt maximum
+  renewBefore: 720h  # 30d
+  privateKey:
+    algorithm: ECDSA
+    size: 256
+    rotationPolicy: Always
@@ -0,0 +1,41 @@
+# One wildcard cert serving every LAN service on the shared Contour gateway.
+#
+# WHY A WILDCARD rather than a cert per service:
+#   1. Certificate Transparency. Per-host LE certs publish every internal hostname
+#      to public CT logs, making the whole internal estate enumerable. A single
+#      *.ad.ddupan.top entry leaks one name and hides the rest.
+#   2. The Gateway's HTTPS listener needs exactly one certificateRef to cover all
+#      hostnames; a new service then needs only an HTTPRoute + a DNS A record, with
+#      no cert work and no Gateway edit at all.
+#   3. Fewer ACME orders against Let's Encrypt rate limits.
+#
+# Lives in envoy-gateway-system because a Gateway listener may only reference a Secret
+# in the Gateway's OWN namespace (cross-namespace refs need a ReferenceGrant).
+# Was projectcontour until the gateway moved to Envoy Gateway — see ../envoy-gateway.
+#
+# NOTE the bare apex `ad.ddupan.top` is listed as well: a wildcard covers
+# one label only, so `*.ad.ddupan.top` does NOT match `ad.ddupan.top` itself.
+---
+apiVersion: cert-manager.io/v1
+kind: Certificate
+metadata:
+  name: wildcard-ad-ddupan-top
+  namespace: envoy-gateway-system
+spec:
+  secretName: wildcard-ad-ddupan-top-tls
+  issuerRef:
+    name: letsencrypt
+    kind: ClusterIssuer
+    group: cert-manager.io
+  commonName: "*.ad.ddupan.top"
+  dnsNames:
+    - "*.ad.ddupan.top"
+    - "ad.ddupan.top"
+  # Renew with 30 days to spare. The WAN is unreliable, so leave a wide window for
+  # retries rather than the default cutting it fine.
+  duration: 2160h    # 90d — Let's Encrypt maximum
+  renewBefore: 720h  # 30d
+  privateKey:
+    algorithm: ECDSA
+    size: 256
+    rotationPolicy: Always
@@ -0,0 +1,43 @@
+# OpenBao's internal PKI over ACME (../../infrastructure/openbao/terraform/pki.tf).
+#
+# WHEN TO PREFER THIS OVER letsencrypt:
+#   * the name must never appear in a public Certificate Transparency log
+#   * issuance/renewal must not depend on the WAN (see netbox/CONTEXT.md §6)
+#   * it is a non-web service (LDAPS, Postgres, syslog) where "browser trusts it
+#     out of the box" buys nothing and the internal CA is already distributed
+#
+# COST: clients must trust the ddupan.top internal CA. Already true for the PVE
+# nodes (pve_ca_trust), Authelia, and the DC — generally NOT true of a fresh browser.
+#
+# Verified 2026-07-25:
+#   * directory live, "externalAccountRequired": false  -> no EAB stanza needed,
+#     matching acme_eab_policy = "not-required" in openbao/terraform
+#   * bao.ad.ddupan.top:8200 serves a REAL Let's Encrypt cert (issuer CN=YE1), so
+#     cert-manager validates it against public roots — no spec.acme.caBundle required
+#   * issuance is capped by default_directory_policy = role:bao-server, which permits
+#     subdomains of ad.ddupan.top only
+---
+apiVersion: cert-manager.io/v1
+kind: ClusterIssuer
+metadata:
+  name: bao-acme
+spec:
+  acme:
+    server: https://bao.ad.ddupan.top:8200/v1/pki/acme/directory
+    # OpenBao ignores the contact address, but ACME clients must send something.
+    email: [email protected]
+    privateKeySecretRef:
+      name: bao-acme-account-key
+    solvers:
+      # http-01, not dns01: bao resolves ad.ddupan.top and can reach LAN hosts
+      # directly (noted as verified in openbao/terraform/pki.tf), so it can fetch
+      # the challenge over the LAN with no public exposure. cert-manager creates a
+      # temporary HTTPRoute on the shared Contour gateway to answer it.
+      - http01:
+          gatewayHTTPRoute:
+            parentRefs:
+              - name: contour-gateway
+                namespace: projectcontour
+                kind: Gateway
+                group: gateway.networking.k8s.io
+                sectionName: http     # the plaintext :80 listener
@@ -0,0 +1,37 @@
+# Let's Encrypt via DNS-01 (Cloudflare).
+#
+# WHY DNS-01 and not HTTP-01: ad.ddupan.top names resolve ONLY on the LAN, so
+# Let's Encrypt cannot reach http://<host>/.well-known/... to validate. DNS-01
+# proves control of the name by writing a TXT record into the PUBLIC ddupan.top
+# Cloudflare zone, which needs no inbound reachability at all. It is also the only
+# challenge type that can issue a WILDCARD.
+#
+# CT-LOG NOTE: every LE-issued name is published to Certificate Transparency logs.
+# Issuing the single wildcard *.ad.ddupan.top (see certificate-wildcard-ad.yaml)
+# means only that one entry appears — individual internal hostnames stay private.
+# For anything that must not appear at all, use the bao-acme issuer instead.
+---
+apiVersion: cert-manager.io/v1
+kind: ClusterIssuer
+metadata:
+  name: letsencrypt
+spec:
+  acme:
+    server: https://acme-v02.api.letsencrypt.org/directory
+    email: [email protected]
+    # Account key. cert-manager creates this; losing it just means a new account.
+    privateKeySecretRef:
+      name: letsencrypt-account-key
+    solvers:
+      - dns01:
+          cloudflare:
+            # Token is reused from cloudflared/terraform (see cert-manager/README.md).
+            # ⚠ It also carries Account·Cloudflare Tunnel:Edit, so anything able to
+            # read this Secret can rewrite tunnel routing, not just DNS. Narrowing it
+            # to Zone·DNS:Edit is a worthwhile follow-up.
+            apiTokenSecretRef:
+              name: cloudflare-api-token
+              key: api-token
+        selector:
+          dnsZones:
+            - ddupan.top
@@ -0,0 +1,20 @@
+# Template for cert-manager/secret.yaml (gitignored). Copy, fill in, apply.
+#
+# Same token as cloudflared/terraform/terraform.tfvars — extract it with:
+#   sed -nE 's/^[[:space:]]*cloudflare_api_token[[:space:]]*=[[:space:]]*"([^"]*)".*/\1/p' \
+#     cloudflared/terraform/terraform.tfvars
+# (a naive `cut -d= -f2` also swallows the trailing comment on that line)
+#
+# Must live in the cert-manager namespace: a ClusterIssuer resolves solver Secret
+# refs in the namespace where cert-manager runs, not where the Certificate is.
+#
+# ⚠ This token also carries Account·Cloudflare Tunnel:Edit. Replacing it with one
+# scoped to Zone·DNS:Edit on ddupan.top would shrink the blast radius to DNS.
+apiVersion: v1
+kind: Secret
+metadata:
+  name: cloudflare-api-token
+  namespace: cert-manager
+type: Opaque
+stringData:
+  api-token: "REPLACE_WITH_CLOUDFLARE_API_TOKEN"
@@ -0,0 +1,55 @@
+# cert-manager — X.509 issuance for the cluster.
+#
+# Chart: jetstack/cert-manager v1.21.0  (repo: https://charts.jetstack.io)
+#
+# Exists so LAN services on ad.ddupan.top get real, auto-renewing certs instead of
+# hand-issued ones. The homelab has TWO ACME sources and both are wired up as
+# ClusterIssuers (see clusterissuer-*.yaml):
+#   letsencrypt  — public CA, DNS-01 via Cloudflare. Browser-trusted with no CA
+#                  install. Used for the *.ad.ddupan.top wildcard.
+#   bao-acme     — OpenBao's internal PKI (../../infrastructure/openbao). No WAN dependency and
+#                  nothing published to Certificate Transparency logs.
+
+# CRDs are part of the release so `helm uninstall` is a clean removal and there is
+# no separate kubectl-apply step to forget.
+crds:
+  enabled: true
+  keep: true   # don't let an accidental uninstall garbage-collect live Certificates
+
+# Single-node k3s: one of everything, modest requests. The laptop runs the whole
+# homelab (see netbox/CONTEXT.md §6).
+replicaCount: 1
+resources:
+  requests:
+    cpu: 10m
+    memory: 64Mi
+  limits:
+    memory: 192Mi
+
+webhook:
+  replicaCount: 1
+  resources:
+    requests:
+      cpu: 10m
+      memory: 32Mi
+    limits:
+      memory: 128Mi
+
+cainjector:
+  replicaCount: 1
+  resources:
+    requests:
+      cpu: 10m
+      memory: 96Mi
+    limits:
+      memory: 256Mi
+
+# ⚠ DNS-01 self-check: cert-manager polls authoritative NS for the _acme-challenge
+# TXT record before telling the CA to validate. By default it asks the cluster's
+# resolver, which for ad.ddupan.top is CoreDNS -> the Samba AD DC (k3s/coredns-custom.yaml).
+# The DC is authoritative for ad.ddupan.top internally and knows nothing about the
+# TXT record we just wrote into the PUBLIC Cloudflare zone, so the self-check would
+# spin forever. Forcing public recursive resolvers makes the check see what the CA
+# will see.
+dns01RecursiveNameservers: "1.1.1.1:53,8.8.8.8:53"
+dns01RecursiveNameserversOnly: true