Establish clean homelab infrastructure baseline

Reorganize the brownfield repository, remove retired and generated artifacts, harden ignore rules, and record the GitOps/IaC redesign.
2026-09-09 16:47:20 +00:00
commit 88a02ababa
418 changed files with 50579 additions and 0 deletions
@@ -0,0 +1,104 @@
+# Blocky — LAN resolver, ad-blocker, split-horizon DNS
+
+**Status: DEPLOYED and verified 2026-07-28. NOT yet the LAN resolver** — DHCP still
+hands out the DC/router pair, so only clients querying 192.168.10.127 explicitly
+use it. Flip DHCP to adopt it; `docker compose down` to abandon it.
+
+Solves three things at once:
+
+1. **Split-horizon.** `git.ddupan.top`, `auth.ddupan.top` and `obj.ddupan.top` are
+   public names that resolve to Cloudflare. On the LAN they should resolve to the
+   Envoy gateway so traffic never leaves the network to reach a service hosted on
+   it. There is currently **nowhere** to put such a record: the DC is
+   authoritative only for `ad.ddupan.top`, and the NEC IX has no static-host
+   feature (`show dns` offers only `fqdn-database`).
+2. **Ad-blocking**, which the DC cannot do.
+3. **Query visibility.** There is none today. On 2026-07-28 a dead VPN tunnel was
+   mis-diagnosed as a bad ISP for hours; a query log would have shortened that.
+
+## Why here, and why compose
+
+Only the laptop is always-on, so PVE is out despite being better isolated.
+
+Within the laptop, this is a **compose stack rather than a k3s Deployment** on
+purpose: DNS is the most foundational service on the network, and putting it
+inside the cluster means a bad upgrade or a CrashLoop takes name resolution with
+it. The cluster broke twice on 2026-07-28. Compose starts earlier in boot and has
+fewer dependencies. k3s CoreDNS is untouched and keeps serving pods.
+
+## Before deploying — the one step with no workaround
+
+**The NEC IX is the DHCP server** (`netboot/dnsmasq.conf` is proxyDHCP only —
+`dhcp-range=…,proxy` — so it assigns nothing). Clients get their resolver list
+from the router, so pointing them at Blocky means changing the DHCP DNS option
+there. Confirm that is editable before relying on any of this.
+
+**Keep the router as a secondary resolver.** Today, if the laptop dies, clients
+fall back to the router and the internet still works for the household. If Blocky
+becomes the *only* resolver, a laptop reboot is a household-wide DNS outage.
+
+The trade-off, stated plainly: clients that fall through to the secondary get
+**no ad-blocking and no split-horizon** — they resolve `git.ddupan.top` to
+Cloudflare and reach Gitea over the tunnel. That fails *open*, which is the right
+direction for a household. Choose consistency over resilience only deliberately.
+
+## Deploy
+
+> Two bugs only surfaced on first run, both now fixed — see the notes at the
+> bottom. Staging validated the syntax but could not have caught either.
+
+```bash
+mkdir -p logs
+docker compose up -d
+docker compose logs -f blocky        # expect "listening" + denylist counts
+```
+
+Verify **before** touching DHCP — nothing depends on it until then:
+
+```bash
+dig @192.168.10.127 git.ddupan.top   +short   # -> 192.168.10.127 (split-horizon)
+dig @192.168.10.127 dc1.ad.ddupan.top +short  # -> 192.168.10.5   (conditional -> DC)
+dig @192.168.10.127 github.com        +short  # -> real answer    (upstream)
+dig @192.168.10.127 doubleclick.net   +short  # -> NXDOMAIN       (blocked)
+curl -s http://192.168.10.127:4000/metrics | head
+```
+
+Only once all five behave should the DHCP option change.
+
+## Rollback
+
+Blocky holds no state that matters. Revert the DHCP DNS option and
+`docker compose down`; clients return to the DC/router pair on next lease. Keep
+the old option value written down before changing it.
+
+## Notes
+
+- **Upstream is the router (`192.168.10.1`), not DoH.** `cloudflare-dns.com` sits
+  in `104.21/16` and `172.67/16` — precisely the ranges the NAIST VPN's
+  split-tunnel routes swallow when the tunnel dies. A DoH upstream there would
+  fail in the same silent way that cost hours on 2026-07-28.
+- **AAAA is filtered for the custom names** (`filterUnmappedTypes` defaults true).
+  Deliberate: the laptop's only global IPv6 belongs to `tun0`, so a AAAA answer
+  would push LAN traffic into the VPN. Matches `../../platform/k3s/coredns-custom.yaml`.
+- `../../platform/k3s/coredns-custom.yaml` still handles the same names for **pods**. The two
+  are independent and must be kept in sync — a name added here usually wants
+  adding there too.
+- Metrics are scraped by `../../platform/observability/metrics/scrapes/blocky.yaml`.
+
+## Bugs found on first deploy
+
+Both were in the staged config and invisible to `docker compose config`, yamllint
+and a server-side dry-run. Worth recording because the same shape recurs:
+
+1. **`ports.dns: 192.168.10.127:53` in `config.yml`.** Those are the *container's*
+   listen addresses, and that IP does not exist inside a bridge-networked
+   container — Blocky exited with `cannot assign requested address`. Host-side
+   restriction belongs in the compose `ports:` mapping; the container listens on
+   all interfaces. Loud failure, quick fix.
+2. **Query log silently did nothing.** The image runs as uid 100, `./logs` is
+   created as uid 1000, so every query logged `fileQueryLogWriter: permission
+   denied` while DNS itself worked perfectly. This is the dangerous one: the
+   service looked healthy, answered correctly, and passed its healthcheck while
+   the feature it was deployed for produced nothing. Fixed with `user: "1000:1000"`.
+
+The second is the reason to check the *feature*, not just the process state.
@@ -0,0 +1,54 @@
+# Blocky — LAN DNS. STAGED, NOT DEPLOYED. See README.md.
+#
+# WHY compose on the laptop and NOT a k3s Deployment, given everything else here
+# is Kubernetes:
+#   * Only the laptop is always-on, so the PVE nodes are not an option.
+#   * DNS is the most foundational service on the network. Running it inside k3s
+#     means a CrashLoopBackOff or a bad `helm upgrade` takes LAN name resolution
+#     with it — and the cluster broke twice on 2026-07-28 alone.
+#   * A compose unit starts earlier in boot and has fewer moving parts than
+#     kubelet -> CNI -> CoreDNS -> Deployment.
+# k3s CoreDNS is unaffected and keeps doing its pod-only job.
+services:
+  blocky:
+    image: spx01/blocky:v0.26
+    container_name: blocky
+    restart: unless-stopped
+
+    # Bridge networking with an EXPLICIT host IP, not network_mode: host. The
+    # laptop already has :53 bound on 192.168.100.1, 192.168.122.1 and the
+    # systemd-resolved stub; host networking plus a 0.0.0.0 bind would collide.
+    # Binding br0 only also means the libvirt networks keep their own resolvers.
+    ports:
+      - "192.168.10.127:53:53/udp"
+      - "192.168.10.127:53:53/tcp"
+      - "192.168.10.127:4000:4000/tcp"   # REST API + /metrics
+
+    # The image runs as uid 100 by default, which cannot write to ./logs (created
+    # as the invoking user, uid 1000) — Blocky then logs
+    # "fileQueryLogWriter: permission denied" every query and silently keeps no
+    # query log at all. Running as the directory owner is more reproducible than a
+    # chown, because a fresh clone creates ./logs as uid 1000 anyway.
+    user: "1000:1000"
+
+    volumes:
+      - ./config.yml:/app/config.yml:ro
+      - ./logs:/logs
+
+    environment:
+      TZ: Asia/Tokyo
+
+    # Blocky answers its own health check. If DNS stops resolving, restart rather
+    # than sit there accepting queries it cannot serve.
+    healthcheck:
+      test: ["CMD", "/app/blocky", "healthcheck"]
+      interval: 30s
+      timeout: 5s
+      retries: 3
+      start_period: 30s
+
+    logging:
+      driver: json-file
+      options:
+        max-size: "10m"
+        max-file: "3"
@@ -0,0 +1,109 @@
+# Blocky — LAN resolver, ad-blocker and split-horizon DNS.
+#
+# DEPLOYED 2026-07-28 and verified, but NOT yet the LAN resolver — clients still
+# get the DC/router pair from DHCP. Making it the resolver needs a DHCP change on
+# the NEC IX; see README.md. Until then only clients that query 192.168.10.127
+# explicitly are affected, so this is safely reversible.
+
+ports:
+  # These are the CONTAINER's listen addresses, so they must be unqualified —
+  # 192.168.10.127 does not exist inside a bridge-networked container, and Blocky
+  # exits with "cannot assign requested address" if you put it here.
+  #
+  # Restricting to the LAN address is done on the HOST side, by the explicit
+  # 192.168.10.127:53:53 mapping in compose.yaml. That matters because the laptop
+  # already has :53 bound on 192.168.100.1, 192.168.122.1 (libvirt bridges) and
+  # 127.0.0.53/54 (the resolved stub) — a plain 53:53 mapping would collide.
+  dns: 53
+  # REST API + Prometheus metrics. Not :80, which Envoy already holds.
+  http: 4000
+
+upstreams:
+  # strict = try the group in order rather than racing them. One upstream here,
+  # so the practical effect is "no surprises".
+  strategy: strict
+  groups:
+    default:
+      # The NEC IX, deliberately. NOT a DoH/DoT resolver at Cloudflare:
+      # cloudflare-dns.com lives in 104.21/16 and 172.67/16, exactly the ranges
+      # the NAIST VPN's 58 split-tunnel routes swallow when the tunnel dies. That
+      # would make DNS fail completely in the same silent way that cost hours on
+      # 2026-07-28. Plain UDP to the router keeps working when the tunnel does not.
+      - 192.168.10.1
+
+conditional:
+  # Queries for the AD zone go straight to the DC, which is authoritative. This
+  # replaces the "DC first, router second" resolver ordering that clients use today.
+  mapping:
+    ad.ddupan.top: 192.168.10.5
+    # Reverse lookups for LAN hosts — the DC holds the reverse zone.
+    10.168.192.in-addr.arpa: 192.168.10.5
+
+customDNS:
+  customTTL: 1h
+  # Split-horizon. These names are PUBLIC (Cloudflare is authoritative for
+  # ddupan.top) and resolve to Cloudflare from outside, which is correct. On the
+  # LAN they must resolve to the Envoy gateway instead, so traffic never leaves
+  # the network to reach a service hosted on it.
+  #
+  # Each has a real Let's Encrypt cert for the exact name on the gateway, so TLS
+  # verifies identically inside and out and no client config differs.
+  #
+  # filterUnmappedTypes defaults to true, which returns an empty answer for AAAA.
+  # That is deliberate and matches what k3s CoreDNS does for the same names — the
+  # laptop's only global IPv6 belongs to tun0, so a AAAA answer would send LAN
+  # traffic into the VPN. See CLAUDE.md.
+  mapping:
+    git.ddupan.top: 192.168.10.127
+    auth.ddupan.top: 192.168.10.127
+    obj.ddupan.top: 192.168.10.127
+
+blocking:
+  denylists:
+    ads:
+      - https://raw.githubusercontent.com/StevenBlack/hosts/master/hosts
+      - https://s3.amazonaws.com/lists.disconnect.me/simple_ad.txt
+  clientGroupsBlock:
+    default:
+      - ads
+  # nxDomain rather than zeroIp: some clients retry forever against 0.0.0.0,
+  # whereas NXDOMAIN is an unambiguous "stop asking".
+  blockType: nxDomain
+  loading:
+    # The WAN is unreliable. Do not fail startup because a denylist could not be
+    # fetched — start with what is cached and refresh later. A resolver that
+    # refuses to boot without the internet is a worse outcome than stale lists.
+    strategy: fast
+    refreshPeriod: 24h
+    downloads:
+      timeout: 60s
+      attempts: 5
+      cooldown: 10s
+
+caching:
+  # serve-stale equivalent: keep answering from cache while upstream is
+  # unreachable. Same reasoning as the CoreDNS `serve_stale` note in
+  # ../../platform/k3s/coredns-custom.yaml — WAN blips must not become resolution failures.
+  minTime: 5m
+  maxTime: 30m
+  maxItemsCount: 0
+  prefetching: true
+  prefetchExpires: 2h
+  prefetchThreshold: 5
+  cacheTimeNegative: 30s
+
+prometheus:
+  enable: true
+  path: /metrics
+
+# queryLog gives the DNS visibility that does not exist today. Sizing this at 7
+# days on purpose: long enough to answer "what was resolving when X broke",
+# short enough not to grow unbounded on the laptop's disk.
+queryLog:
+  type: csv
+  target: /logs
+  logRetentionDays: 7
+
+log:
+  level: info
+  format: text