Establish clean homelab infrastructure baseline

Reorganize the brownfield repository, remove retired and generated artifacts, harden ignore rules, and record the GitOps/IaC redesign.

infrastructure/proxmox/.gitignore

@@ -0,0 +1,8 @@
+# Answer files embed the root password hash + host network config — keep out of git.
+# They're regenerated from the eventual Ansible role / vault.
+pxe/answer/*.toml
+pxe/secrets.env
+
+# Build scratch (large PXE artifacts live on the netboot host, not here)
+pxe/build/
+vyos/credentials.yml

infrastructure/proxmox/README-ha.md

@@ -0,0 +1,94 @@
+# Proxmox HA — and the watchdog that makes it real
+
+HA was enabled for **one** guest on 2026-07-26. The cluster's posture is otherwise
+unchanged: no HA, guests disposable.
+
+| | |
+|---|---|
+| resource | `vm:100` (`vyos-rtr`), `state started`, `max_restart 3`, `max_relocate 2` |
+| why | it is the gateway for **both** SDN VNets and an OSPF speaker; losing it takes `labnet` + `retronet` offline (the main LAN is unaffected — the NEC IX is its gateway) |
+| storage | `pve-rg` (DRBD, place-count 2): `pve1` + `pve2` hold **UpToDate diskful** replicas, `pve3` attaches **diskless** — so it can start on any node |
+| codified | `ansible/roles/pve_ha` + `ansible/ha.yml` (separate from `site.yml`, like `cluster.yml`) |
+
+**What this buys, precisely:** crash *restart* on another node, roughly **1–3 minutes**
+of gateway downtime. It is **not** seamless failover. Real gateway HA is a second VyOS
+with VRRP owning `10.60.0.1`/`10.61.0.1`; that was considered and deferred.
+
+**What it costs:** adding any HA resource **arms fencing cluster-wide**. A node that
+loses quorum now self-reboots. Corosync here runs a **single ring** on the flat 1G LAN,
+so a network blip is now a reboot rather than a shrug.
+
+Also set at the same time: `qm set 100 --agent 0`. PVE had `agent: enabled=1` while
+VyOS 2025.11 ships **no `qemu-ga` binary and no apt sources** — the virtio channel was
+wired but nothing could answer, so snapshots could not filesystem-freeze and no guest
+IPs were reported. Installing it would mean adding Debian repos to the only router, and
+would be wiped by the next `add system image`.
+
+## ⚠ The watchdog is still `softdog` — finish this
+
+`softdog` is a **software** watchdog: a kernel timer. **It cannot fire when the kernel
+itself is frozen**, which is exactly the failure this cluster has actually had (pve2's
+Raven Ridge idle freeze). So today's HA covers clean crashes, power loss and network
+partition — but *not* the freeze case that motivated it.
+
+Hardware watchdogs are present and verified (2026-07-26):
+
+| node | module | timeout |
+|---|---|---|
+| pve1 (Intel i3-6100U) | `iTCO_wdt` | 30s |
+| pve2 / pve3 (Ryzen 2400GE) | `sp5100_tco` | 60s |
+
+`ansible/ha.yml` already writes the config (`WATCHDOG_MODULE` + a `blacklist softdog`
+in `/etc/modprobe.d/pve-ha-watchdog.conf`). **Both are needed**: `watchdog-mux` opens
+`/dev/watchdog`, which belongs to whichever watchdog registered *first*, and PVE loads
+softdog at boot — so setting `WATCHDOG_MODULE` alone leaves the hardware module sitting
+unused as `watchdog1`.
+
+It takes effect on **the next reboot of each node**. The play deliberately does not
+reboot: a surprise rolling reboot of all three nodes is not something an idempotent
+baseline run should do.
+
+### Live swap, without rebooting (do it in this order)
+
+The danger is stopping `watchdog-mux` while an LRM holds the watchdog — that self-fences
+the node. Parking the resource first removes every active LRM, which makes the rest safe.
+
+```bash
+# 1. park the resource; all LRMs go idle and release their watchdogs
+ha-manager set vm:100 --state ignored
+# wait until NO line says "watchdog active" — vm:100 keeps running throughout
+ha-manager status
+
+# 2. per node (pve3, pve2, pve1 — least critical first):
+systemctl stop watchdog-mux
+rmmod softdog
+modprobe -r iTCO_wdt 2>/dev/null; modprobe iTCO_wdt      # pve1
+# modprobe -r sp5100_tco; modprobe sp5100_tco            # pve2 / pve3
+cat /sys/class/watchdog/watchdog0/identity                # must be the HW one now
+systemctl start watchdog-mux                              # Restart=no: start it explicitly
+ls -l /proc/$(pidof watchdog-mux)/fd | grep watchdog      # must hold /dev/watchdog
+
+# 3. hand the resource back
+ha-manager set vm:100 --state started
+ha-manager status
+```
+
+If a step fails, leave `vm:100` in `ignored` and fix it — the VM keeps running and
+nothing gets fenced. `ignored` means HA does not touch the guest, not that it stops.
+
+## Verify
+
+```bash
+ha-manager status                    # "fencing armed", "service vm:100 (pveN, started)"
+ha-manager config
+qm config 100 | grep -E 'agent|onboot'
+linstor r l | grep pm-                # replica placement
+```
+
+## Undo
+
+```bash
+ha-manager remove vm:100             # disarms fencing once no resources remain
+rm /etc/modprobe.d/pve-ha-watchdog.conf
+# and revert WATCHDOG_MODULE in /etc/default/pve-ha-manager
+```

infrastructure/proxmox/ansible/ansible.cfg

@@ -0,0 +1,18 @@
+[defaults]
+inventory = inventory/hosts.yml
+roles_path = roles
+host_key_checking = False
+callback_result_format = yaml
+nocows = True
+
+# Every host here is a Proxmox node reached as root, so become is unnecessary
+# (unlike samba-ad/, whose inventory is mixed Linux + Windows).
+# Encrypted group_vars/all/vault.yml are COMMITTED; the password is not.
+# Relative to this file, so it resolves for any checkout location.
+vault_password_file = ../../../.vault_pass
+
+[ssh_connection]
+# The homelab LAN and the home uplink are both flaky; keep connections warm and
+# retry rather than failing a long play halfway through.
+pipelining = True
+retries = 3

infrastructure/proxmox/ansible/auth.yml

@@ -0,0 +1,10 @@
+---
+# AD realm + user/group sync. Kept separate from site.yml because it writes
+# cluster-wide config (/etc/pve/domains.cfg) and needs the bind password.
+#
+#   ansible-playbook auth.yml -e @../../samba-ad/ansible/group_vars/all/vault.yml
+- name: Proxmox AD authentication realm
+  hosts: pve
+  gather_facts: false
+  roles:
+    - pve_auth

infrastructure/proxmox/ansible/cluster.yml

@@ -0,0 +1,16 @@
+---
+# Form the "homelab" cluster. SEPARATE from site.yml on purpose: site.yml is
+# safely re-runnable baseline config, whereas cluster formation is effectively
+# one-way (undoing a join means reinstalling the node). Run it deliberately.
+#
+#   ansible-playbook cluster.yml
+#
+# serial:1 with the primary first — a joining node needs the primary quorate,
+# and joining all three at once is how you get a split-brain corosync config.
+- name: Form the Proxmox cluster
+  hosts: pve
+  gather_facts: true
+  serial: 1
+  order: sorted            # pve1 (the primary) first
+  roles:
+    - pve_cluster

infrastructure/proxmox/ansible/group_vars/pve.yml

@@ -0,0 +1,50 @@
+---
+ansible_user: root
+ansible_python_interpreter: /usr/bin/python3
+
+# Internal domain. NOTE: ad.ddupan.top, NOT lab.ddupan.top -- the installer
+# answer files originally wrote "lab" and it was corrected post-install on
+# 2026-07-25. This is the Samba AD realm (see services/samba-ad/).
+pve_domain: ad.ddupan.top
+
+# ── apt / repositories ──────────────────────────────────────────────────
+# Debian codename PVE 9.2 is built on. Derived at runtime, but pinned here as a
+# fallback so a failed fact-gather cannot silently point apt at the wrong suite.
+pve_suite: trixie
+
+# The enterprise repos 401 without a subscription and make every apt run noisy.
+pve_disable_enterprise_repo: true
+pve_enable_no_subscription_repo: true
+
+# Ceph is deliberately NOT used on this cluster -- it was tried on HDD OSDs and
+# was far too slow. LINSTOR/DRBD replaces it, so the Ceph repo is disabled.
+pve_disable_ceph_repo: true
+
+# Strip the "No valid subscription" web-UI dialog.
+pve_remove_subscription_nag: true
+
+# Run a full dist-upgrade. Off by default: the home uplink is unstable and a
+# half-applied upgrade across a live cluster is worse than a stale one. Enable
+# deliberately:  -e pve_dist_upgrade=true
+pve_dist_upgrade: false
+
+# ── outbound mail ───────────────────────────────────────────────────────
+# These nodes cannot hand mail to M365 directly from a residential IP, so all
+# mail relays through the k3s smtp-relay on the laptop. See services/smtp-relay/.
+pve_mail_relayhost: "[192.168.10.127]:25"
+# M365 authenticates as this mailbox and REFUSES to send as anything else
+# (5.7.60 SendAsDenied), so every local sender is rewritten to it.
+pve_mail_from: [email protected]
+
+# ── internal PKI ────────────────────────────────────────────────────────
+# OpenBao (192.168.10.8) is the homelab root-of-trust. This endpoint is
+# unauthenticated by design, so nodes can (re)fetch the CA without a token.
+pve_internal_ca_url: https://bao.ad.ddupan.top:8200/v1/pki/ca/pem
+
+# ── DNS ─────────────────────────────────────────────────────────────────
+# See roles/pve_dns for WHY the DC must come first.
+pve_dns_search: ad.ddupan.top
+pve_nameservers:
+  - 192.168.10.5    # Samba AD DC: internal zone + external forwarding
+  - 192.168.10.1    # router: external only, NO internal zone
+pve_dns_probe_name: bao.ad.ddupan.top

infrastructure/proxmox/ansible/ha.yml

@@ -0,0 +1,16 @@
+---
+# Proxmox HA. SEPARATE from site.yml deliberately, like cluster.yml: registering an
+# HA resource arms fencing CLUSTER-WIDE, so a node that loses quorum will self-reboot.
+# That is a behaviour change for a cluster whose documented posture is "no HA", and it
+# should be run on purpose rather than as a side effect of a baseline run.
+#
+#   ansible-playbook ha.yml
+#   ansible-playbook ha.yml --tags watchdog   # watchdog config only
+#
+# Preconditions (all verified 2026-07-26): cluster quorate, guest disk on replicated
+# storage (pve-rg / DRBD place-count 2), and `onboot: 1` on the guest.
+- name: Proxmox HA
+  hosts: pve
+  gather_facts: true
+  roles:
+    - pve_ha

infrastructure/proxmox/ansible/inventory/host_vars/pve1.yml

@@ -0,0 +1,3 @@
+# Hardware watchdog for HA fencing. Verified present 2026-07-26 (iTCO_wdt).
+# softdog cannot fence a frozen kernel; see roles/pve_ha.
+pve_ha_watchdog_module: "iTCO_wdt"

infrastructure/proxmox/ansible/inventory/host_vars/pve2.yml

@@ -0,0 +1,3 @@
+# Hardware watchdog for HA fencing. Verified present 2026-07-26 (sp5100_tco).
+# softdog cannot fence a frozen kernel; see roles/pve_ha.
+pve_ha_watchdog_module: "sp5100_tco"

infrastructure/proxmox/ansible/inventory/host_vars/pve3.yml

@@ -0,0 +1,3 @@
+# Hardware watchdog for HA fencing. Verified present 2026-07-26 (sp5100_tco).
+# softdog cannot fence a frozen kernel; see roles/pve_ha.
+pve_ha_watchdog_module: "sp5100_tco"

infrastructure/proxmox/ansible/inventory/hosts.yml

@@ -0,0 +1,57 @@
+---
+# 3-node Proxmox VE 9.2 cluster "homelab". All were netboot auto-installed on
+# 2026-07-25 (see ../pxe/ and the project memory for the PXE recipe).
+#
+# IPs are all BELOW the router's DHCP pool (192.168.10.10-250) on purpose.
+# .5 = samba AD DC (dc1, libvirt VM on the laptop), .8 = reserved for OpenBao,
+# .127 = laptop ("core" node: NFS + netboot.xyz + k3s + smtp-relay).
+all:
+  children:
+    pve:
+      hosts:
+        pve1:
+          ansible_host: 192.168.10.4
+          # Intel NUC i3-6100, 16G. Most RAM of the three -> LINSTOR controller.
+          # Its UEFI cannot unpack a large initramfs, so it needs the special
+          # PXE path in ../pxe/ if it is ever rebuilt.
+          pve_install_disk: sdb
+          pve_bulk_disk: sda
+
+        pve2:
+          ansible_host: 192.168.10.7
+          # ThinkCentre, Ryzen 2400GE (Raven Ridge), 8G. THIS is the node that
+          # randomly froze. Kernel workaround below; also set BIOS
+          # "Power Supply Idle Control" -> "Typical Current Idle" (the more
+          # reliable half of the fix, and it can only be done physically).
+          pve_kernel_cmdline_extra: "idle=nomwait processor.max_cstate=1"
+          pve_install_disk: nvme0n1
+          pve_bulk_disk: sda
+
+        pve3:
+          ansible_host: 192.168.10.9
+          # Same Raven Ridge silicon as pve2 but has never frozen. Deliberately
+          # left WITHOUT the C-state workaround: processor.max_cstate=1 blocks
+          # deeper idle states and costs power, so it is not applied pre-emptively.
+          # If pve3 ever freezes, set pve_kernel_cmdline_extra here too.
+          pve_install_disk: nvme0n1
+          pve_bulk_disk: sda
+
+    # Network devices driven over network_cli (NOT ssh/python) — they have no
+    # Python interpreter, so normal modules do not apply.
+    network:
+      hosts:
+        vyos-rtr:
+          ansible_host: 192.168.10.2
+          ansible_user: vyos
+          ansible_connection: ansible.netcommon.network_cli
+          ansible_network_os: vyos.vyos.vyos
+          # Key auth; the vyos password is in ../vyos/credentials.yml (gitignored).
+          ansible_ssh_private_key_file: ~/.ssh/id_ed25519
+
+    # Lab VMs on the cluster (normal ssh/python hosts, unlike the `network` group).
+    labvms:
+      hosts:
+        retrolab:
+          ansible_host: 10.60.0.10
+          ansible_user: panxiao81
+          ansible_become: true

infrastructure/proxmox/ansible/linstor.yml

@@ -0,0 +1,11 @@
+---
+# LINSTOR/DRBD storage. Separate from site.yml because it partitions and wipes
+# disks. The HDD wipe additionally requires -e pve_linstor_wipe_hdd=true.
+#
+#   ansible-playbook linstor.yml --tags linstor_pkgs      # packages only
+#   ansible-playbook linstor.yml -e pve_linstor_wipe_hdd=true
+- name: LINSTOR hyperconverged storage
+  hosts: pve
+  gather_facts: true
+  roles:
+    - pve_linstor

infrastructure/proxmox/ansible/requirements.yml

@@ -0,0 +1,15 @@
+---
+# Install with: ansible-galaxy collection install -r requirements.yml
+#
+# This file did not exist until ansible-lint flagged unresolvable modules: the
+# other two projects (samba-ad/, openbao/) declared their collections but this
+# one relied on whatever happened to be in the operator's environment. A fresh
+# checkout or a CI runner could not reproduce it.
+collections:
+  - name: ansible.netcommon     # network_cli connection for VyOS — it has no Python
+                                # interpreter, so the usual ssh+python path cannot work
+  - name: vyos.vyos             # vyos_config; see CLAUDE.md on preferring explicit
+                                # `set` lines over the resource modules, which lag upstream
+  - name: ansible.posix         # mount, authorized_key
+  - name: community.general     # proxmox_kvm (guest lifecycle). NOTE: deb822_repository
+                                # is ansible.builtin, NOT community.general — see CLAUDE.md

infrastructure/proxmox/ansible/retrolab.yml

@@ -0,0 +1,41 @@
+---
+# 86Box host: desktop + RDP. AD join is NOT here — that uses the shared
+# `ad_sssd_join` role in ../../samba-ad/ansible (join-sssd.yml), so every
+# Linux host joins the domain the same way.
+- name: Retro lab desktop
+  hosts: retrolab
+  gather_facts: true
+
+  environment:
+    # DO NOT REMOVE. This host holds long-lived interactive desktop sessions.
+    # Ubuntu's needrestart hooks apt and, in non-interactive mode, RESTARTS any
+    # service linked against an upgraded library. On 2026-07-25 an unrelated
+    # `apt install patchelf libcap2-bin` pulled in a libcap2 upgrade, which made
+    # needrestart restart xrdp-sesman. sesman came back with an empty session
+    # table and could no longer reattach the running :10 session, stranding a
+    # live 86Box VM: every reconnect created a NEW session, and xfce4-session
+    # refuses to start twice for one user, so each new session died instantly.
+    # `l` = list what would need restarting, restart nothing.
+    NEEDRESTART_MODE: l
+
+  vars:
+    # ISO library, served from the laptop (the same export PVE mounts as its
+    # `laptop` storage). Mounted READ-ONLY: it is where 86Box picks CD images
+    # from, and nothing on the retro side should ever write to the library.
+    nfs_mounts:
+      - src: "192.168.10.127:/mnt/pool/proxmox/template/iso"
+        path: /mnt/iso
+
+    # The bridge 86Box attaches its TAP interfaces to. net1 (retronet, VLAN
+    # 110) deliberately has no IP on the host — see roles/tap_bridge/defaults.
+    tap_bridge_name: br-retro
+    tap_bridge_member: enp6s19
+    tap_bridge_member_mac: "bc:24:11:9f:63:ed"
+    retro_86box_tap_bridge: br-retro
+
+  roles:
+    - guest_base
+    - retro_desktop
+    - nfs_mounts
+    - tap_bridge
+    - retro_86box

infrastructure/proxmox/ansible/roles/guest_base/tasks/main.yml

@@ -0,0 +1,61 @@
+---
+# Baseline every lab VM needs to be manageable. Cloud-init installs these on a
+# FRESH VM; this role covers VMs that already exist (cloud-init's package module
+# runs once per instance) and repairs any that failed a WAN blip on first boot.
+
+# The WAN drops mid-transaction often enough that dpkg gets left interrupted
+# ("dpkg was interrupted, you must manually run 'sudo dpkg --configure -a'"),
+# after which EVERY later apt run fails. Repair it rather than fail the play.
+- name: Detect an interrupted dpkg
+  ansible.builtin.command:
+    cmd: dpkg --audit
+  register: _dpkg_audit
+  changed_when: false
+  failed_when: false
+  check_mode: false
+
+- name: Repair interrupted dpkg
+  ansible.builtin.command:
+    cmd: dpkg --configure -a
+  when: (_dpkg_audit.stdout | default('') | trim) | length > 0
+  changed_when: true
+
+
+# Repair dpkg if a previous apt run was cut short. NOTE the usual cause is an
+# operator stopping/rebooting the VM while cloud-init or apt is still running --
+# not a network fault. Check `cloud-init status` before power-cycling a fresh VM.
+- name: Detect an interrupted dpkg
+  ansible.builtin.command:
+    cmd: dpkg --audit
+  register: _dpkg_audit
+  changed_when: false
+  failed_when: false
+  check_mode: false
+
+- name: Repair interrupted dpkg
+  ansible.builtin.command:
+    cmd: dpkg --configure -a
+  when: (_dpkg_audit.stdout | default('') | trim) | length > 0
+  changed_when: true
+
+- name: Install guest baseline packages
+  ansible.builtin.apt:
+    name:
+      # Gives PVE the guest's IPs, clean shutdown, and fsfreeze for snapshots.
+      # Without it `qm agent <id> ping` fails and PVE is blind inside the guest.
+      - qemu-guest-agent
+      - openssh-server
+    state: present
+    update_cache: true
+    cache_valid_time: 3600
+  register: _gb
+  retries: 3
+  delay: 15
+  until: _gb is succeeded
+
+- name: Enable the guest agent and sshd
+  ansible.builtin.systemd_service:
+    name: "{{ item }}"
+    enabled: true
+    state: started
+  loop: [qemu-guest-agent, ssh]

infrastructure/proxmox/ansible/roles/nfs_mounts/defaults/main.yml

@@ -0,0 +1,16 @@
+---
+# Generic NFS client mounts. Shareable: pass a list, get idempotent mounts.
+#
+# Each entry: { src, path, opts (optional), state (optional) }
+nfs_mounts: []
+
+# Defaults chosen for read-mostly media (ISO libraries), NOT for VM disks:
+#   ro       - the share is a library; nothing here should write to it. Also
+#              the only real protection available, because the server exports
+#              it to <world> with no_root_squash.
+#   soft     - a hard mount wedges every process that touches the path when the
+#              server or the WAN-adjacent link blips, including the desktop
+#              file manager. Soft returns EIO instead of hanging forever.
+#   _netdev  - do not attempt the mount before the network is up.
+#   nofail   - a missing NFS server must never stop this host from booting.
+nfs_mounts_default_opts: "ro,soft,timeo=100,retrans=3,_netdev,nofail,noatime"

infrastructure/proxmox/ansible/roles/nfs_mounts/tasks/main.yml

@@ -0,0 +1,43 @@
+---
+- name: Install the NFS client
+  ansible.builtin.apt:
+    name: nfs-common
+    state: present
+    update_cache: true
+    cache_valid_time: 3600
+  register: _nfsc
+  retries: 3
+  delay: 15
+  until: _nfsc is succeeded
+
+- name: Create the mount points
+  ansible.builtin.file:
+    path: "{{ item.path }}"
+    state: directory
+    mode: "0755"
+  loop: "{{ nfs_mounts }}"
+  loop_control:
+    label: "{{ item.path }}"
+
+- name: Mount the NFS shares
+  ansible.posix.mount:
+    src: "{{ item.src }}"
+    path: "{{ item.path }}"
+    fstype: nfs
+    opts: "{{ item.opts | default(nfs_mounts_default_opts) }}"
+    state: "{{ item.state | default('mounted') }}"
+  loop: "{{ nfs_mounts }}"
+  loop_control:
+    label: "{{ item.src }} -> {{ item.path }}"
+
+- name: Verify each share is actually readable
+  # `mount` reporting success is not proof: a stale handle or a squashed uid
+  # shows up only on the first read.
+  ansible.builtin.command:
+    cmd: "ls {{ item.path }}"
+  loop: "{{ nfs_mounts }}"
+  loop_control:
+    label: "{{ item.path }}"
+  register: _nfs_ls
+  changed_when: false
+  when: (item.state | default('mounted')) == 'mounted'

infrastructure/proxmox/ansible/roles/pve_acme/defaults/main.yml

@@ -0,0 +1,61 @@
+---
+# pve_acme role defaults.
+#
+# Gives pveproxy (the :8006 web UI) a real certificate from the homelab's own CA,
+# using PVE's BUILT-IN ACME client rather than a bolted-on lego/certbot. PVE then
+# owns the whole lifecycle itself: it writes the cert, restarts pveproxy, and
+# renews daily via pve-daily-update.timer. Nothing extra to install or babysit.
+
+# --- Which certificate this role touches -------------------------------------
+# ONLY /etc/pve/local/pveproxy-ssl.pem -- the optional override pveproxy serves.
+#
+# It must NEVER be confused with /etc/pve/local/pve-ssl.pem, which is signed by
+# the per-cluster "PVE Cluster Manager CA" and is what the nodes use to
+# authenticate each other for the cluster API, live migration and replication.
+# PVE owns that one and rotates it itself; replacing it breaks the cluster.
+# When pveproxy-ssl.pem is absent, pveproxy falls back to pve-ssl.pem -- which is
+# exactly the untrusted-cert warning this role exists to remove.
+pve_acme_cert_file: /etc/pve/local/pveproxy-ssl.pem
+
+# --- CA / directory ----------------------------------------------------------
+# OpenBao's internal PKI, via the ROLE-SCOPED ACME directory (same endpoint shape
+# as samba-ad's samba_ad_acme role). The role scope matters for security: the
+# unscoped /v1/pki/acme/directory would fall back to whatever
+# default_directory_policy is set to, whereas this URL pins issuance to
+# bao-server, whose allowed_domains caps it at subdomains of ad.ddupan.top.
+#
+# WHY the internal CA and not Let's Encrypt (decided 2026-07-26):
+#   * renewal must not depend on the WAN -- the uplink drops at random and the
+#     hypervisor management plane is the last thing that should need the internet
+#   * per-node names pve1/2/3.ad.ddupan.top would otherwise be published to public
+#     Certificate Transparency logs, which is precisely what the wildcard
+#     convention in services/cert-manager/ exists to avoid
+#   * it keeps the Cloudflare DNS token off all three hypervisors
+# COST: browsers must trust "ddupan.top Internal CA". The nodes themselves already
+# do (role pve_ca_trust); your workstation needs it installed once.
+pve_acme_directory: "https://bao.ad.ddupan.top:8200/v1/pki/roles/bao-server/acme/directory"
+
+# ACME account name. Lives in /etc/pve/priv/acme/<name>, which is on the pmxcfs --
+# so it is CLUSTER-WIDE and only ever registered once, not once per node.
+pve_acme_account: default
+
+# OpenBao ignores the contact address, but the ACME protocol requires one.
+pve_acme_email: [email protected]
+
+# --- Challenge ---------------------------------------------------------------
+# "standalone" is PVE's built-in http-01 plugin: it starts a throwaway listener on
+# port 80 for the duration of the challenge. That works here -- and would NOT work
+# against Let's Encrypt -- because bao is on the same flat LAN, resolves
+# ad.ddupan.top via the DC, and fetches the challenge directly. Nothing is exposed
+# to the internet and no port forward is involved.
+#
+# Requires port 80 to be free on the node. Verified 2026-07-26: PVE listens on
+# 8006/3128/111 but nothing on 80, and the PVE firewall is disabled cluster-wide.
+pve_acme_plugin: standalone
+
+# The name to certify. Per-node, unlike the account and plugin config.
+pve_acme_domain: "{{ inventory_hostname }}.{{ pve_domain }}"
+
+# The CA's subject, used to decide whether an existing pveproxy-ssl.pem already
+# came from us or is a leftover that should be replaced.
+pve_acme_issuer_cn: "ddupan.top Internal CA"

infrastructure/proxmox/ansible/roles/pve_acme/tasks/main.yml

@@ -0,0 +1,115 @@
+---
+# Point PVE's built-in ACME client at OpenBao's internal PKI and get a real cert
+# onto pveproxy. See defaults/main.yml for WHY the internal CA over Let's Encrypt.
+#
+# Everything here is idempotent: a second run reports changed=0. Renewal is NOT
+# our job -- pve-daily-update.timer runs `pvenode acme cert renew` once a day and
+# PVE reissues when the cert is inside 30 days of expiry.
+
+# --- Preconditions -----------------------------------------------------------
+# The standalone plugin binds :80 for the challenge. If something else holds it,
+# the order fails deep inside pvenode with a confusing error, so check up front.
+- name: Check that port 80 is free for the http-01 challenge
+  ansible.builtin.command: ss -lnt 'sport = :80'
+  register: _port80
+  changed_when: false
+  # Read-only, and every later condition depends on it -- so it must still run
+  # under --check, or the whole role errors out on an undefined register.
+  check_mode: false
+
+- name: Fail early if port 80 is occupied
+  ansible.builtin.assert:
+    that:
+      - "':80' not in _port80.stdout"
+    fail_msg: >-
+      Something is already listening on port 80; the ACME standalone plugin cannot
+      bind it and the order would fail. Free the port or switch to a dns-01 plugin.
+    quiet: true
+
+# --- ACME account (cluster-wide, registered once) ----------------------------
+# /etc/pve/priv/acme/ is on the pmxcfs, so the account is shared by all three
+# nodes. Guarding on the file -- rather than run_once -- is deliberate: this play
+# uses serial: 1, where each host is its own batch and run_once would therefore
+# fire on EVERY host, re-registering the account three times.
+- name: Check whether the ACME account already exists
+  ansible.builtin.stat:
+    path: "/etc/pve/priv/acme/{{ pve_acme_account }}"
+  register: _acme_account
+
+- name: Register the ACME account against OpenBao
+  # Non-interactive only because bao's directory advertises no termsOfService
+  # (verified 2026-07-26: meta contains just externalAccountRequired=false).
+  # pvenode prompts for ToS acceptance when a CA does publish one, and there is
+  # no --accept-tos flag to suppress it -- so a CA change here can hang the play.
+  ansible.builtin.command:
+    argv:
+      - pvenode
+      - acme
+      - account
+      - register
+      - "{{ pve_acme_account }}"
+      - "{{ pve_acme_email }}"
+      - --directory
+      - "{{ pve_acme_directory }}"
+  when: not _acme_account.stat.exists
+  changed_when: true
+
+# --- Per-node domain config --------------------------------------------------
+# Written to /etc/pve/nodes/<node>/config as an `acmedomain0:` line. That file
+# does not exist until the first `pvenode config set`, hence the default('').
+- name: Read the node config
+  ansible.builtin.slurp:
+    src: "/etc/pve/nodes/{{ inventory_hostname }}/config"
+  register: _node_cfg
+  failed_when: false
+
+- name: Configure the ACME domain for this node
+  ansible.builtin.command:
+    argv:
+      - pvenode
+      - config
+      - set
+      - "--acmedomain0"
+      - "{{ pve_acme_domain }},plugin={{ pve_acme_plugin }}"
+  vars:
+    _want: "acmedomain0: {{ pve_acme_domain }},plugin={{ pve_acme_plugin }}"
+  when: _want not in (_node_cfg.content | default('') | b64decode)
+  changed_when: true
+
+# --- Certificate -------------------------------------------------------------
+# Order only when there is no usable cert already. Checking the ISSUER rather
+# than mere existence means a leftover self-signed or previously-Let's-Encrypted
+# pveproxy-ssl.pem gets replaced, while our own cert is left alone for PVE's
+# renewal timer to manage.
+- name: Inspect the current pveproxy certificate
+  ansible.builtin.command: "openssl x509 -noout -issuer -subject -in {{ pve_acme_cert_file }}"
+  register: _current_cert
+  changed_when: false
+  failed_when: false
+  check_mode: false   # read-only; the order task's condition depends on it
+
+- name: Order the certificate from OpenBao
+  # --force overwrites an existing pveproxy-ssl.pem, which is what we want once
+  # the check above has decided the current one is wrong or missing. It does NOT
+  # touch pve-ssl.pem, so the cluster's internal trust is unaffected.
+  #
+  # KNOWN FAILURE MODE: the bao-server PKI role pins key_type=rsa/key_bits=2048.
+  # If a future PVE generates an EC CSR, finalize is rejected by OpenBao with a
+  # key-type error (the same trap documented in samba-ad's samba_ad_acme role).
+  # The fix is on the bao side -- widen the role -- not here.
+  ansible.builtin.command: pvenode acme cert order --force
+  when: >-
+    _current_cert.rc != 0
+    or pve_acme_issuer_cn not in _current_cert.stdout
+    or pve_acme_domain not in _current_cert.stdout
+  changed_when: true
+
+# --- Renewal -----------------------------------------------------------------
+# This timer is what keeps the cert alive; without it the cert simply expires in
+# place. Enabled by default on PVE, asserted here so the guarantee is explicit
+# rather than assumed.
+- name: Ensure the daily renewal timer is enabled
+  ansible.builtin.systemd:
+    name: pve-daily-update.timer
+    state: started
+    enabled: true

infrastructure/proxmox/ansible/roles/pve_auth/defaults/main.yml

@@ -0,0 +1,73 @@
+---
+# Authenticate PVE users against Samba AD over verified LDAPS.
+#
+# WHY the AD realm and not OIDC: PVE's `ad`/`ldap` realms support a real user +
+# group SYNC (pveum realm sync), so permissions can be granted to a group BEFORE
+# anyone logs in, and username+password works for the API/CLI (Terraform,
+# pvesh). PVE's OIDC realm only creates users on first browser login and cannot
+# do non-interactive auth. Authelia OIDC can be added later as an EXTRA realm
+# for convenient SSO; it is not a replacement.
+
+pve_auth_realm: ad                     # users appear as <user>@ad
+pve_auth_domain: ad.ddupan.top
+
+# MUST be the hostname, NEVER 192.168.10.5: the DC's LDAPS cert is issued by
+# OpenBao ACME with a DNS SAN only (no IP SAN), so verification fails by IP with
+# "IP address mismatch". Verified 2026-07-25.
+pve_auth_server1: dc1.ad.ddupan.top
+pve_auth_port: 636
+pve_auth_mode: ldaps                   # `--secure` is DEPRECATED in favour of this
+
+# The whole point of trusting the internal CA on these nodes (role pve_ca_trust):
+# PVE defaults --verify to 0, i.e. it does NOT check the DC's certificate, which
+# makes the directory bind trivially MITM-able on a flat LAN. capath's default
+# (/etc/ssl/certs) already contains the hashed internal CA.
+pve_auth_verify: 1
+pve_auth_capath: /etc/ssl/certs
+
+pve_auth_base_dn: "DC=ad,DC=ddupan,DC=top"
+pve_auth_bind_dn: "CN=svc-pve,CN=Users,DC=ad,DC=ddupan,DC=top"
+# Read-only bind account created by samba-ad (samba_ad_service_accounts).
+pve_auth_bind_password: "{{ vault_pve_bind_password | default('') }}"
+
+# AD logon name. Without this PVE would try the default LDAP `uid`, which AD
+# does not populate.
+pve_auth_user_attr: sAMAccountName
+# AD stores users AND groups under CN=Users by default (not an OU).
+pve_auth_group_dn: "CN=Users,DC=ad,DC=ddupan,DC=top"
+pve_auth_group_classes: group
+pve_auth_user_classes: user
+
+# Only sync real, ENABLED people — not service or system accounts.
+# The userAccountControl bit-AND rule (1.2.840.113556.1.4.803 := 2) excludes
+# DISABLED accounts, which is what keeps AD's built-in `Guest` and `krbtgt` out.
+# Filtering on name alone let both through on the first sync.
+pve_auth_filter: >-
+  (&(objectCategory=person)(objectClass=user)(!(userAccountControl:1.2.840.113556.1.4.803:=2))(!(sAMAccountName=svc-*)))
+
+# Only groups this cluster actually uses for RBAC. `(objectClass=group)` pulls in
+# every builtin AD group: PVE rejects most outright ("group name 'Domain Users-ad'
+# contains invalid characters" — spaces are illegal in PVE group IDs) and imports
+# the rest as clutter (DnsAdmins-ad, DnsUpdateProxy-ad). Convention: name any
+# group PVE should see `pve-*`.
+pve_auth_group_filter: "(&(objectClass=group)(cn=pve-*))"
+pve_auth_sync_attributes: "email=mail,firstname=givenName,lastname=sn"
+
+# AD logon names are case-insensitive; PVE defaults to case-sensitive, which
+# means Panxiao81 and panxiao81 would become two different PVE users.
+pve_auth_case_sensitive: 0
+
+# remove-vanished: 'none' on purpose. Anything stronger lets a transient LDAP
+# hiccup delete users, their properties, or their ACLs from a live cluster.
+pve_auth_sync_defaults: "scope=both,enable-new=1,remove-vanished=none"
+
+# --- scheduled sync ---
+pve_auth_sync_job: pve-ad-sync
+pve_auth_sync_schedule: "*-*-* 04:11:00"
+
+# --- RBAC ---
+# PVE RENAMES synced groups to "<name>-<realm>", so AD's pve-admins becomes
+# pve-admins-ad. Granting the ACL to "pve-admins" would silently match nothing.
+pve_auth_admin_group: "pve-admins-{{ pve_auth_realm }}"
+pve_auth_admin_role: Administrator
+pve_auth_admin_path: /

infrastructure/proxmox/ansible/roles/pve_auth/tasks/main.yml

@@ -0,0 +1,201 @@
+---
+# Realm config lives in /etc/pve/domains.cfg, which is REPLICATED cluster-wide,
+# so every task here runs once against a single node. Doing it per-host would
+# just have three nodes racing to write the same file.
+
+- name: Require the bind password
+  ansible.builtin.assert:
+    that: pve_auth_bind_password | length > 0
+    fail_msg: >-
+      pve_auth_bind_password is empty. It comes from vault_pve_bind_password in
+      samba-ad/ansible/group_vars/all/vault.yml — pass it via -e or a vars file.
+    quiet: true
+  run_once: true
+
+- name: Check whether the realm already exists
+  ansible.builtin.command:
+    cmd: "pveum realm list --output-format json"
+  register: _realms
+  changed_when: false
+  check_mode: false
+  run_once: true
+
+- name: Decide create vs update
+  ansible.builtin.set_fact:
+    _realm_exists: "{{ pve_auth_realm in (_realms.stdout | from_json | map(attribute='realm') | list) }}"
+  run_once: true
+
+# --- shared option set, so create and update cannot drift apart -------------
+# `--type` is deliberately NOT in here: a realm's type is immutable, and
+# `pveum realm modify --type ad` fails with "Unknown option: type". It is passed
+# only on the create path below.
+- name: Build the realm option string
+  ansible.builtin.set_fact:
+    _realm_opts: >-
+      --domain {{ pve_auth_domain }}
+      --server1 {{ pve_auth_server1 }}
+      --port {{ pve_auth_port }}
+      --mode {{ pve_auth_mode }}
+      --verify {{ pve_auth_verify }}
+      --capath {{ pve_auth_capath }}
+      --base_dn '{{ pve_auth_base_dn }}'
+      --bind_dn '{{ pve_auth_bind_dn }}'
+      --user_attr {{ pve_auth_user_attr }}
+      --user_classes '{{ pve_auth_user_classes }}'
+      --group_dn '{{ pve_auth_group_dn }}'
+      --group_classes '{{ pve_auth_group_classes }}'
+      --filter '{{ pve_auth_filter }}'
+      --group_filter '{{ pve_auth_group_filter }}'
+      --sync_attributes '{{ pve_auth_sync_attributes }}'
+      --sync-defaults-options '{{ pve_auth_sync_defaults }}'
+      --case-sensitive {{ pve_auth_case_sensitive }}
+      --comment 'Samba AD (dc1) over verified LDAPS'
+  run_once: true
+
+- name: Create the realm
+  # --check-connection makes PVE actually bind before saving, so a wrong DN,
+  # password, or an untrusted certificate fails HERE instead of silently
+  # producing a realm nobody can log in to.
+  ansible.builtin.shell:
+    cmd: >-
+      pveum realm add {{ pve_auth_realm }} --type ad {{ _realm_opts }}
+      --password '{{ pve_auth_bind_password }}'
+      --check-connection 1
+  when: not _realm_exists
+  run_once: true
+  # no_log hides the bind password, but it also hides WHY a failure happened.
+  # Keep it on (the password is on the command line) and rely on
+  # --check-connection plus the manual `pveum realm modify` path for diagnosis.
+  no_log: true
+
+- name: Read the current realm config
+  ansible.builtin.command:
+    cmd: "pvesh get /access/domains/{{ pve_auth_realm }} --output-format json"
+  register: _realm_cur
+  changed_when: false
+  check_mode: false
+  when: _realm_exists
+  run_once: true
+
+- name: Detect realm drift
+  # pveum has no diff mode, so compare the fields we manage. Without this the
+  # role reported "changed" on every single run, which makes real drift invisible.
+  ansible.builtin.set_fact:
+    _realm_drift: "{{ _realm_exists and (
+        (_realm_cur.stdout | from_json).get('server1') != pve_auth_server1 or
+        (_realm_cur.stdout | from_json).get('base_dn') != pve_auth_base_dn or
+        (_realm_cur.stdout | from_json).get('bind_dn') != pve_auth_bind_dn or
+        (_realm_cur.stdout | from_json).get('mode') != pve_auth_mode or
+        (_realm_cur.stdout | from_json).get('verify') | default(0) | int != pve_auth_verify | int or
+        (_realm_cur.stdout | from_json).get('filter') != pve_auth_filter | trim or
+        (_realm_cur.stdout | from_json).get('group_filter') != pve_auth_group_filter or
+        (_realm_cur.stdout | from_json).get('user_attr') != pve_auth_user_attr or
+        (_realm_cur.stdout | from_json).get('sync_attributes') != pve_auth_sync_attributes
+      ) }}"
+  run_once: true
+
+- name: Update the realm
+  ansible.builtin.shell:
+    cmd: >-
+      pveum realm modify {{ pve_auth_realm }} {{ _realm_opts }}
+      --password '{{ pve_auth_bind_password }}'
+      --check-connection 1
+  when: _realm_drift | default(false)
+  run_once: true
+  no_log: true
+
+- name: Show the resulting realm
+  ansible.builtin.command:
+    cmd: "pveum realm list --output-format json"
+  register: _realm_after
+  changed_when: false
+  run_once: true
+
+- name: Report it
+  ansible.builtin.debug:
+    msg: "{{ _realm_after.stdout | from_json | selectattr('realm', 'equalto', pve_auth_realm) | list }}"
+  run_once: true
+
+# --- initial sync ----------------------------------------------------------
+- name: Dry-run the sync first
+  # Proves the bind and filters work, and shows what WOULD be imported, without
+  # writing to user.cfg.
+  ansible.builtin.command:
+    cmd: "pveum realm sync {{ pve_auth_realm }} --scope both --dry-run 1"
+  register: _sync_dry
+  changed_when: false
+  run_once: true
+
+- name: Show what the sync would import
+  ansible.builtin.debug:
+    msg: "{{ _sync_dry.stderr_lines | default([]) + _sync_dry.stdout_lines | default([]) }}"
+  run_once: true
+
+- name: Sync users and groups
+  # Always reports "changed": this is a reconcile ACTION against a live
+  # directory, not a declared state, and pveum gives no no-op signal to key off.
+  # The realm config and ACL above ARE drift-detected, so a re-run showing
+  # changed=1 means "sync ran", not "something was wrong".
+  ansible.builtin.command:
+    cmd: "pveum realm sync {{ pve_auth_realm }} --scope both --enable-new 1 --remove-vanished none"
+  register: _sync
+  changed_when: true
+  run_once: true
+
+# --- scheduled sync --------------------------------------------------------
+- name: List existing realm-sync jobs
+  ansible.builtin.command:
+    cmd: "pvesh get /cluster/jobs/realm-sync --output-format json"
+  register: _jobs
+  changed_when: false
+  check_mode: false
+  run_once: true
+
+- name: Create the scheduled sync job
+  ansible.builtin.command:
+    cmd: >-
+      pvesh create /cluster/jobs/realm-sync/{{ pve_auth_sync_job }}
+      --realm {{ pve_auth_realm }}
+      --schedule '{{ pve_auth_sync_schedule }}'
+      --scope both --enable-new 1 --remove-vanished none --enabled 1
+      --comment 'Nightly AD user/group sync'
+  when: pve_auth_sync_job not in (_jobs.stdout | from_json | map(attribute='id') | list)
+  run_once: true
+
+# --- RBAC ------------------------------------------------------------------
+- name: Read current ACLs
+  ansible.builtin.command:
+    cmd: "pveum acl list --output-format json"
+  register: _acl_cur
+  changed_when: false
+  check_mode: false
+  run_once: true
+
+- name: Grant the admin group its role
+  # NOTE the group name: PVE appends "-<realm>" to every synced group, so AD's
+  # `pve-admins` is `pve-admins-ad` here. Granting to the AD name matches nothing.
+  ansible.builtin.command:
+    cmd: >-
+      pveum acl modify {{ pve_auth_admin_path }}
+      --group {{ pve_auth_admin_group }} --role {{ pve_auth_admin_role }}
+  when: >-
+    (_acl_cur.stdout | from_json
+      | selectattr('path', 'equalto', pve_auth_admin_path)
+      | selectattr('ugid', 'equalto', pve_auth_admin_group)
+      | selectattr('roleid', 'equalto', pve_auth_admin_role) | list | length) == 0
+  run_once: true
+
+- name: Report users, groups and ACLs
+  ansible.builtin.shell:
+    cmd: |
+      echo "--- users ---";  pveum user list  --output-format json | python3 -c "import json,sys;[print('   ',u['userid']) for u in json.load(sys.stdin)]"
+      echo "--- groups ---"; pveum group list --output-format json | python3 -c "import json,sys;[print('   ',g['groupid'], g.get('users','')) for g in json.load(sys.stdin)]"
+      echo "--- acls ---";   pveum acl list   --output-format json | python3 -c "import json,sys;[print('   ',a['path'],a.get('ugid'),a.get('roleid')) for a in json.load(sys.stdin)]"
+  register: _final
+  changed_when: false
+  run_once: true
+
+- name: Show it
+  ansible.builtin.debug:
+    msg: "{{ _final.stdout_lines }}"
+  run_once: true

infrastructure/proxmox/ansible/roles/pve_ca_trust/handlers/main.yml

@@ -0,0 +1,5 @@
+---
+- name: Update CA certificates
+  ansible.builtin.command:
+    cmd: update-ca-certificates
+  changed_when: true

infrastructure/proxmox/ansible/roles/pve_ca_trust/tasks/main.yml

@@ -0,0 +1,38 @@
+---
+# Trust the homelab's internal CA (OpenBao pki/, "ddupan.top Internal CA").
+#
+# Needed so the PVE nodes can verify LDAPS against the Samba AD DC with
+# verify=1 instead of disabling verification -- an unverified directory bind is
+# trivially MITM-able on a flat LAN, and the whole point of running our own CA
+# is not having to do that.
+#
+# The CA is pulled from bao's UNAUTHENTICATED /v1/pki/ca/pem endpoint (same
+# approach as roles/openbao_ssh_ca_trust in services/openbao): no token needed,
+# and a CA rotation is picked up simply by re-running this.
+
+- name: Fetch the internal CA from OpenBao
+  ansible.builtin.uri:
+    url: "{{ pve_internal_ca_url }}"
+    return_content: true
+    # bao serves a real Let's Encrypt cert (openbao_acme role), so normal
+    # verification works here -- do NOT relax this.
+    validate_certs: true
+  register: _bao_ca
+  changed_when: false
+  retries: 3
+  delay: 10
+  until: _bao_ca is succeeded
+
+- name: Sanity-check that we actually got a CA certificate
+  ansible.builtin.assert:
+    that:
+      - "'BEGIN CERTIFICATE' in _bao_ca.content"
+    fail_msg: "OpenBao did not return a PEM certificate -- refusing to install it as a trust anchor."
+    quiet: true
+
+- name: Install the internal CA into the system trust store
+  ansible.builtin.copy:
+    dest: /usr/local/share/ca-certificates/ddupan-internal-ca.crt
+    content: "{{ _bao_ca.content }}"
+    mode: "0644"
+  notify: Update CA certificates

infrastructure/proxmox/ansible/roles/pve_cluster/defaults/main.yml

@@ -0,0 +1,11 @@
+---
+pve_cluster_name: homelab
+
+# The node that runs `pvecm create`; everyone else joins it. pve1 is also the
+# intended LINSTOR controller (most RAM of the three).
+pve_cluster_primary: pve1
+
+# Corosync link address per node. Single flat 1G LAN on one unmanaged switch, so
+# there is exactly one ring. A dedicated corosync ring would need a second NIC
+# (mini-PCIe) plus another switch — noted as a future upgrade, not done here.
+pve_cluster_link0: "{{ ansible_host }}"

infrastructure/proxmox/ansible/roles/pve_cluster/tasks/main.yml

@@ -0,0 +1,98 @@
+---
+# Form the PVE cluster. THIS IS THE ONE STEP IN THIS REPO THAT IS NOT EASILY
+# REVERSIBLE: undoing a join effectively means reinstalling the node. Everything
+# guards on "already clustered" so re-runs are safe.
+
+- name: Detect existing cluster membership
+  ansible.builtin.stat:
+    path: /etc/pve/corosync.conf
+  register: _corosync
+  changed_when: false
+
+- name: Record membership
+  ansible.builtin.set_fact:
+    _in_cluster: "{{ _corosync.stat.exists }}"
+
+# ── SSH trust, needed for a non-interactive `pvecm add` ───────────────────
+# `pvecm add --use_ssh` authenticates over SSH from the JOINING node to the
+# primary. Without pre-shared keys it prompts for the root password, which
+# cannot be automated cleanly.
+- name: Ensure root has an SSH keypair
+  ansible.builtin.user:
+    name: root
+    generate_ssh_key: true
+    ssh_key_type: ed25519
+    ssh_key_file: .ssh/id_ed25519
+  when: not _in_cluster
+  check_mode: false          # --check would otherwise leave the next task nothing to read
+
+- name: Read this node's root public key
+  ansible.builtin.slurp:
+    src: /root/.ssh/id_ed25519.pub
+  register: _rootpub
+  when: not _in_cluster
+  check_mode: false
+
+- name: Authorise THIS node's root key on the primary
+  # Each joining node pushes its OWN key. Do NOT loop over groups['pve'] reading
+  # hostvars[item]._rootpub: with serial:1 the later nodes have not run yet, so
+  # their facts are undefined and their keys would silently never be installed —
+  # `pvecm add --use_ssh` would then sit waiting for a password.
+  ansible.posix.authorized_key:
+    user: root
+    key: "{{ _rootpub.content | b64decode }}"
+    state: present
+  delegate_to: "{{ pve_cluster_primary }}"
+  when:
+    - not _in_cluster
+    - inventory_hostname != pve_cluster_primary
+
+- name: Pre-seed the primary's host key so SSH does not prompt
+  ansible.builtin.known_hosts:
+    path: /root/.ssh/known_hosts
+    name: "{{ hostvars[pve_cluster_primary].ansible_host }}"
+    key: "{{ lookup('pipe', 'ssh-keyscan -t ed25519 ' + hostvars[pve_cluster_primary].ansible_host + ' 2>/dev/null') }}"
+    state: present
+  when:
+    - not _in_cluster
+    - inventory_hostname != pve_cluster_primary
+
+# ── create / join ─────────────────────────────────────────────────────────
+- name: Create the cluster on the primary
+  ansible.builtin.command:
+    cmd: "pvecm create {{ pve_cluster_name }} --link0 {{ pve_cluster_link0 }}"
+  when:
+    - not _in_cluster
+    - inventory_hostname == pve_cluster_primary
+
+- name: Wait for the primary to report quorum before anyone joins
+  ansible.builtin.command:
+    cmd: pvecm status
+  register: _primary_q
+  until: _primary_q.stdout is search('Quorate:\s+Yes')
+  retries: 12
+  delay: 5
+  changed_when: false
+  # No cluster exists during --check (the create is skipped), so there is nothing
+  # to wait for; skip rather than fail the dry run.
+  when:
+    - inventory_hostname == pve_cluster_primary
+    - not ansible_check_mode
+
+- name: Join the cluster
+  # --use_ssh avoids the interactive API-ticket password prompt.
+  ansible.builtin.command:
+    cmd: "pvecm add {{ hostvars[pve_cluster_primary].ansible_host }} --link0 {{ pve_cluster_link0 }} --use_ssh"
+  when:
+    - not _in_cluster
+    - inventory_hostname != pve_cluster_primary
+
+- name: Wait for this node to be quorate
+  ansible.builtin.command:
+    cmd: pvecm status
+  register: _q
+  until: _q.stdout is search('Quorate:\s+Yes')
+  retries: 24
+  delay: 5
+  changed_when: false
+  when: not ansible_check_mode

infrastructure/proxmox/ansible/roles/pve_dns/tasks/main.yml

@@ -0,0 +1,55 @@
+---
+# Resolver configuration.
+#
+# ORDER MATTERS AND IS NOT ARBITRARY:
+#   .5  = Samba AD DC (dc1) -- AUTHORITATIVE for ad.ddupan.top AND forwards
+#         external queries onward. Resolves both internal and public names.
+#   .1  = LAN router -- resolves public names ONLY. Internal ad.ddupan.top
+#         lookups return EMPTY here (verified 2026-07-25: `dig @192.168.10.1
+#         bao.ad.ddupan.top` -> nothing, while @192.168.10.5 -> 192.168.10.8).
+#
+# The installer left these nodes pointing at .1 alone, which silently broke
+# every internal name -- including the OpenBao CA fetch and, later, the AD
+# realm. .1 is kept as a SECOND entry purely so public DNS survives the DC
+# being down (dc1 is a VM on the laptop); internal names correctly fail then.
+
+- name: Configure resolv.conf
+  ansible.builtin.copy:
+    dest: /etc/resolv.conf
+    mode: "0644"
+    content: |
+      # Managed by Ansible (services/proxmox/ansible, role pve_dns).
+      search {{ pve_dns_search }}
+      {% for ns in pve_nameservers %}
+      nameserver {{ ns }}
+      {% endfor %}
+
+- name: Verify an internal name now resolves
+  # Guards against a regression that would otherwise only surface much later as
+  # a confusing failure in an unrelated role.
+  ansible.builtin.command:
+    cmd: getent hosts {{ pve_dns_probe_name }}
+  register: _dns_probe
+  changed_when: false
+  failed_when: _dns_probe.rc != 0
+  retries: 3
+  delay: 5
+  until: _dns_probe is succeeded
+
+# ── address-family preference ─────────────────────────────────────────────
+# These nodes are IPv4-ONLY: no global IPv6 address, no default IPv6 route
+# (verified 2026-07-25). But the DC returns AAAA records and glibc hands those
+# out first, so anything resolving a dual-stack name tries a dead IPv6 path.
+# That is exactly how `apt update` against packages.linbit.com stalled: it
+# resolved to 2a01:4f8:1c1c:6ab9::1 and hung, while IPv4 answered fine.
+#
+# Prefer IPv4 system-wide until this LAN actually has IPv6 egress.
+- name: Prefer IPv4 over IPv6 in glibc resolution
+  ansible.builtin.copy:
+    dest: /etc/gai.conf
+    mode: "0644"
+    content: |
+      # Managed by Ansible (services/proxmox/ansible, role pve_dns).
+      # Raise the precedence of IPv4-mapped addresses above native IPv6 so
+      # getaddrinfo() returns A records first on this IPv4-only network.
+      precedence ::ffff:0:0/96  100

infrastructure/proxmox/ansible/roles/pve_floppy/defaults/main.yml

@@ -0,0 +1,26 @@
+---
+# Web UI for the floppy drives PVE cannot express (see files/floppy-ui.py).
+#
+# ONE instance serves the cluster: every action goes through `pvesh`, which
+# proxies to whichever node owns the VM, so this does not need to run on the
+# node the guest happens to live on -- and keeps working after a migration.
+
+#
+# LOGIN is PVE's own: the app posts credentials to /access/ticket, so it accepts
+# whatever realms the cluster has -- `pam` for node-local accounts and `ad` for
+# Samba AD over verified LDAPS (role pve_auth) -- and stores no bind credential.
+# A session additionally needs Sys.Modify on / in PVE's ACL, which in practice
+# means the pve-admins-ad group. It serves TLS with the node's own ACME cert, so
+# reach it BY HOSTNAME: https://pve1.ad.ddupan.top:8088 (CN/SAN is the FQDN, and
+# there is no IP SAN -- the same trap as the DC's LDAPS cert).
+
+pve_floppy_listen: "0.0.0.0"
+pve_floppy_port: 8088
+# The shared ISO storage (`laptop`), mounted at the same path on every node.
+pve_floppy_image_dir: /mnt/pve/laptop/template/iso
+
+# The retro modem switchboard's phonebook (role retro_modem). It lives on
+# pmxcfs so this UI, running on pve1, edits the same file the switchboard reads
+# on pve3 -- no API and no IPC between them: the file IS the interface, and the
+# switchboard re-reads it on the next dial.
+pve_floppy_phonebook: /etc/pve/retro-phonebook

infrastructure/proxmox/ansible/roles/pve_floppy/files/floppy-ui.py

@@ -0,0 +1,711 @@
+#!/usr/bin/env python3
+"""Floppy drives for Proxmox VMs, which PVE itself cannot express.
+
+There is no floppy in the PVE UI *or* the VM config schema, and PVE starts QEMU
+with -nodefaults, so a retro guest has no A: at all unless one is smuggled in
+through the `args` field. This is the UI for that.
+
+Two facts shape the whole design:
+
+  * `args` is root@pam-ONLY. The check in PVE::API2::Qemu is a literal
+    `$authuser eq 'root@pam'`, and an API token's authuser is `root@pam!name`,
+    so no token can ever set it. Hence: run on a node, shell out to `pvesh` as
+    root. A network API client would need the root PASSWORD, which is worse.
+  * `pvesh` proxies to whichever node owns the VM. So ONE instance on ONE node
+    serves the whole cluster and keeps working when a VM migrates.
+
+Changing `args` only takes effect on the next QEMU process, i.e. a real stop +
+start; a guest-initiated reboot reuses the same process. Swapping the *medium*
+of an existing drive is live, via the monitor. The UI reflects that split, and
+deliberately has no stop/start buttons -- the PVE UI already has those.
+
+LOGIN: no PAM code and no LDAP code here. The node already authenticates against
+both -- realm `pam` for local accounts and realm `ad` for Samba AD over VERIFIED
+LDAPS (roles/pve_auth) -- so this posts the credentials to PVE's own
+/access/ticket and believes the answer. That also means it inherits the realm
+list, the LDAPS certificate verification, and the account lockouts for free, and
+holds no bind DN or password of its own.
+
+Then AUTHORISATION, which is the half that matters: authenticating merely proves
+you are someone in AD. Inserting a host file into a VM is a host-level action, so
+a session additionally needs Sys.Modify on `/` in PVE's own ACL -- i.e. the same
+`pve-admins-ad` group that already administers the cluster.
+"""
+
+import argparse
+import base64
+import concurrent.futures
+import hmac
+import html
+import json
+import os
+import re
+import secrets
+import ssl
+import subprocess
+import time
+import urllib.error
+import urllib.parse
+import urllib.request
+from http.cookies import SimpleCookie
+from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
+
+# Floppy media. .vfd is what the NT4 driver disks ship as; .ima/.flp/.dsk show
+# up in retro archives. .iso is deliberately absent -- CD-ROMs PVE handles.
+SUFFIXES = (".img", ".ima", ".vfd", ".flp", ".dsk")
+
+# `args` is one shell-ish string and HMP `change` takes an unquoted path, so a
+# filename with whitespace cannot be expressed in either without quoting rules
+# that differ between the two. Such images are hidden rather than half-supported.
+# ponytail: if a spaced filename ever matters, shlex.quote for args + rename for HMP.
+FLOPPY_ARG = re.compile(r"-drive\s+if=floppy[^\s]*")
+
+# pvedaemon, the API pveproxy itself proxies to. Used instead of `pvesh create
+# /access/ticket` so the password never appears in a process argv, and instead of
+# pveproxy:8006 so there is no TLS-to-self certificate dance. It is bound to
+# loopback by PVE, so plain HTTP here does not put anything on the wire.
+PVEDAEMON = "http://127.0.0.1:85/api2/json"
+
+# What a logged-in user must additionally HAVE. Attaching a floppy points a VM at
+# an arbitrary file on the host, so audit-level access is not enough.
+REQUIRED_PRIV = "Sys.Modify"
+
+# Sessions are signed with a key generated at startup: a restart logs everyone
+# out, which for a homelab tool is a feature, not a limitation to engineer away.
+SESSION_KEY = secrets.token_bytes(32)
+SESSION_TTL = 8 * 3600
+
+
+def pvesh(*args, check=True):
+    """Run pvesh. Returns stdout. Raises RuntimeError with PVE's own message."""
+    p = subprocess.run(
+        ["pvesh", *args], capture_output=True, text=True, timeout=60
+    )
+    if check and p.returncode != 0:
+        raise RuntimeError((p.stderr or p.stdout).strip() or f"pvesh {args[1]} failed")
+    return p.stdout
+
+
+def pvesh_json(*args):
+    return json.loads(pvesh(*args, "--output-format", "json"))
+
+
+_REALMS = []
+
+
+def realms():
+    """Whatever PVE is configured with -- typically pam (local) and ad (LDAPS).
+
+    Cached for the life of the process: this is on the LOGIN page, so without it
+    every unauthenticated hit paid 1.9s for a `pvesh` to list something that
+    changes when someone adds an auth domain -- i.e. never, and a restart picks
+    it up.
+    """
+    if not _REALMS:
+        _REALMS.extend(sorted(d["realm"] for d in pvesh_json("get", "/access/domains")))
+    return _REALMS
+
+
+def pve_authenticate(userid, password):
+    """True if PVE accepts these credentials for this realm. No PAM/LDAP here."""
+    data = urllib.parse.urlencode({"username": userid, "password": password}).encode()
+    try:
+        urllib.request.urlopen(f"{PVEDAEMON}/access/ticket", data=data, timeout=20).read()
+        return True
+    except urllib.error.HTTPError:
+        return False  # 401 for a bad password, and for a disabled/expired account
+    except urllib.error.URLError as e:
+        # pvedaemon down is an outage, not a wrong password. Saying so avoids an
+        # hour of retyping a password that was right all along.
+        raise RuntimeError(f"PVE API unreachable: {e.reason}") from None
+
+
+def pve_authorized(userid):
+    """True if PVE's own ACL gives this user REQUIRED_PRIV on the whole tree."""
+    perms = pvesh_json("get", "/access/permissions", "--userid", userid)
+    return bool(perms.get("/", {}).get(REQUIRED_PRIV))
+
+
+def _mac(msg):
+    return hmac.new(SESSION_KEY, msg, "sha256").digest()
+
+
+def session_new(userid):
+    # The signature is HEX, not the raw digest: the parts are joined with '|'
+    # and split back with rsplit, and 32 random bytes contain 0x7C ('|') about
+    # 12% of the time -- which split the token inside its own signature and made
+    # roughly one login in eight bounce straight back to the login page.
+    msg = f"{userid}|{int(time.time()) + SESSION_TTL}".encode()
+    return base64.urlsafe_b64encode(msg + b"|" + _mac(msg).hex().encode()).decode()
+
+
+def session_user(cookie):
+    """The userid a cookie proves, or None. Constant-time, expiry enforced."""
+    try:
+        raw = base64.urlsafe_b64decode(cookie.encode())
+        msg, sig = raw.rsplit(b"|", 1)
+        if not hmac.compare_digest(sig, _mac(msg).hex().encode()):
+            return None
+        userid, exp = msg.decode().rsplit("|", 1)
+        return userid if int(exp) > time.time() else None
+    except Exception:
+        return None
+
+
+def csrf_token(userid):
+    return base64.urlsafe_b64encode(_mac(b"csrf|" + userid.encode())).decode()
+
+
+def floppy_spec(path):
+    return f"-drive if=floppy,format=raw,file={path}"
+
+
+def args_with_floppy(args, path):
+    """Add or replace the floppy drive in an existing `args` string."""
+    spec = floppy_spec(path)
+    if FLOPPY_ARG.search(args or ""):
+        return FLOPPY_ARG.sub(spec, args, count=1).strip()
+    return f"{args} {spec}".strip() if args else spec
+
+
+def args_without_floppy(args):
+    return " ".join(FLOPPY_ARG.sub("", args or "").split())
+
+
+def floppy_in_args(args):
+    """The image `args` will boot with, '' for a drive with no file, None for none."""
+    m = FLOPPY_ARG.search(args or "")
+    if not m:
+        return None
+    f = re.search(r"file=([^,\s]+)", m.group(0))
+    return f.group(1) if f else ""
+
+
+def read_conf(path):
+    """Parse a PVE VM config off pmxcfs.
+
+    Snapshots are appended as [name] sections after the live config, so stop at
+    the first one. Split on the FIRST colon only -- an `args` value is full of
+    them (file=/mnt/pve/...).
+    """
+    conf = {}
+    try:
+        f = open(path)
+    except OSError:
+        return conf
+    with f:
+        for line in f:
+            if line.startswith("["):
+                break
+            key, sep, val = line.partition(":")
+            if sep:
+                conf[key.strip()] = val.strip()
+    return conf
+
+
+def inserted_medium(node, vmid):
+    """What is in the drive RIGHT NOW, which is not what `args` says after a swap.
+
+    The single most expensive thing this app does: 1.9s for a VM on this node,
+    3.6s when pveproxy has to forward it to another one. Only ask for VMs that
+    actually have a floppy drive, and only when the cache has expired.
+    """
+    try:
+        out = pvesh("create", f"/nodes/{node}/qemu/{vmid}/monitor", "--command", "info block")
+    except RuntimeError:
+        return None
+    m = re.search(r"^floppy0[^:]*:\s*(.*)$", out, re.M)
+    if not m:
+        return None
+    val = m.group(1).strip()
+    if val.startswith("[") or not val:
+        return ""  # [not inserted]
+    return val.split(" (")[0]
+
+
+# A page load costs several pvesh round trips (a monitor query forwarded to
+# another node is 3.6s on its own), and clicking around re-pays them every time.
+# The TTL is deliberately longer than a human's click interval -- at 5s every
+# click still missed. Staleness is bounded to someone else running `qm` by hand,
+# because every action here clears the cache, so what you just did is never what
+# you see stale.
+CACHE_TTL = 30.0
+_CACHE = {"at": 0.0, "rows": None}
+
+
+def vms(fresh=False):
+    """The VM table, read off pmxcfs instead of asked for one VM at a time.
+
+    Every `pvesh` costs ~1.9s (Perl startup plus a cluster round trip), so the
+    old shape -- one call for the list, then one per VM for its config, then one
+    per running VM for its monitor -- made a four-guest page take ~16 seconds.
+    /etc/pve is that same data replicated to every node, at file-read speed, so
+    only the monitor round trips are left and those run in parallel.
+    """
+    if not fresh and _CACHE["rows"] and time.time() - _CACHE["at"] < CACHE_TTL:
+        return _CACHE["rows"]
+    ids = json.load(open("/etc/pve/.vmlist"))["ids"]
+    out = []
+    for vmid, meta in ids.items():
+        if meta.get("type") != "qemu":
+            continue
+        conf = read_conf(f"/etc/pve/nodes/{meta['node']}/qemu-server/{vmid}.conf")
+        args = conf.get("args", "")
+        out.append({
+            "vmid": int(vmid),
+            "name": conf.get("name", ""),
+            "node": meta["node"],
+            "args": args,
+            "configured": floppy_in_args(args),
+        })
+    # Everything remaining is a pvesh round trip, so overlap them: run status
+    # alongside the monitors, and ask the monitor ONLY about VMs that have a
+    # floppy drive -- "what is in the drive" is meaningless for the others, and
+    # it was two thirds of the calls here.
+    with concurrent.futures.ThreadPoolExecutor(max_workers=8) as pool:
+        status = pool.submit(pvesh_json, "get", "/cluster/resources", "--type", "vm")
+        media = {
+            vm["vmid"]: pool.submit(inserted_medium, vm["node"], vm["vmid"])
+            for vm in out
+            if vm["configured"] is not None
+        }
+        running = {
+            r["vmid"]: r.get("status") == "running"
+            for r in status.result()
+            if r.get("type") == "qemu"
+        }
+    for vm in out:
+        vm["running"] = running.get(vm["vmid"], False)
+        vm["inserted"] = media[vm["vmid"]].result() if vm["vmid"] in media else None
+    rows = sorted(out, key=lambda v: v["vmid"])
+    _CACHE.update(at=time.time(), rows=rows)
+    return rows
+
+
+def images(image_dir):
+    return sorted(
+        f
+        for f in os.listdir(image_dir)
+        if f.lower().endswith(SUFFIXES) and not re.search(r"\s", f)
+    )
+
+
+# Targets this UI is willing to write. It is an allowlist, not a blocklist,
+# because a phonebook entry is a thing the modem ACTS ON: `ppp` and `ssh:` make
+# it spawn a process, so a free-form target would be remote command execution
+# wearing a phone number. Anything new must be added here deliberately.
+PHONEBOOK_TARGET = re.compile(
+    r"""^( ppp                                  # hand the line to pppd
+         | vm:\d+                               # another line of the switchboard
+         | ssh:[\w.@-]+(:\d+)?                  # ssh -tt to a host
+         | (telnet:)?[\w.-]+:\d+                # raw TCP, or telnet with IAC handling
+         )$""",
+    re.VERBOSE,
+)
+
+
+def phonebook_read(path):
+    try:
+        return open(path).read()
+    except OSError:
+        return ""
+
+
+def phonebook_check(text):
+    """Return an error string, or None if every line is a comment or valid."""
+    for n, line in enumerate(text.splitlines(), 1):
+        body = line.split("#", 1)[0].split()
+        if not body:
+            continue
+        if len(body) != 2:
+            return f"line {n}: expected '<number> <target>'"
+        if not body[0].isdigit():
+            return f"line {n}: {body[0]!r} is not a phone number"
+        if not PHONEBOOK_TARGET.match(body[1]):
+            return f"line {n}: target {body[1]!r} is not one of ppp / vm:N / ssh:host / host:port / telnet:host:port"
+    return None
+
+
+def phonebook_write(path, text):
+    """Write via a temp file in the same directory, so a failed write cannot
+    leave the modem reading half a phonebook."""
+    text = text.replace("\r\n", "\n")
+    if not text.endswith("\n"):
+        text += "\n"
+    tmp = path + ".new"
+    with open(tmp, "w") as f:
+        f.write(text)
+    os.replace(tmp, path)
+
+
+PAGE = """<!doctype html><meta charset=utf-8><title>PVE floppy</title>
+<style>
+ body{{font:14px/1.5 system-ui,sans-serif;margin:2rem;max-width:60rem}}
+ table{{border-collapse:collapse;width:100%}}
+ td,th{{border-bottom:1px solid #ccc;padding:.5rem;text-align:left;vertical-align:top}}
+ .msg{{padding:.6rem;background:#eef;border-left:3px solid #66c;margin-bottom:1rem}}
+ .err{{background:#fee;border-color:#c66}}
+ .note{{color:#666;font-size:.85em}}
+ code{{background:#f4f4f4;padding:0 .2em}}
+</style>
+<h1>Floppy drives</h1>
+<p class=note>{user} &mdash;
+<form method=post action=/logout style=display:inline>
+<input type=hidden name=csrf value="{csrf}"><button>log out</button></form></p>
+<p class=note>PVE has no floppy in its UI or config schema; these live in
+<code>args</code>. Attaching or detaching a drive takes effect on the next
+<b>stop + start</b> (not a guest reboot). Inserting into an existing drive is live.</p>
+{msg}
+<table><tr><th>VM<th>drive in args<th>in the drive now<th>action</tr>
+{rows}
+</table>
+<p class=note>Images: {image_dir} &mdash; names with spaces are hidden (see the source).</p>
+
+<h1>Modem phonebook</h1>
+<p class=note>{phonebook_path} &mdash; on pmxcfs, so every node sees it, and the
+switchboard re-reads it on the next dial. No restart, and a call in progress
+survives the edit.</p>
+<form method=post action=/phonebook>
+<input type=hidden name=csrf value="{csrf}">
+<textarea name=text rows=10 style="width:100%;font-family:monospace">{phonebook}</textarea>
+<br><button>Save phonebook</button>
+</form>
+"""
+
+ROW = """<tr>
+<td>{vmid} {name}<br><span class=note>{node}, {state}</span>
+<td>{configured}
+<td>{inserted}
+<td><form method=post>
+ <input type=hidden name=vmid value="{vmid}">
+ <input type=hidden name=csrf value="{csrf}">
+ <select name=image>{options}</select><br>
+ {buttons}
+</form>
+</tr>
+"""
+
+LOGIN = """<!doctype html><meta charset=utf-8><title>PVE floppy</title>
+<style>
+ body{{font:14px/1.5 system-ui,sans-serif;margin:4rem auto;max-width:22rem}}
+ input,select,button{{width:100%;padding:.4rem;margin:.2rem 0;box-sizing:border-box}}
+ .msg{{padding:.6rem;background:#fee;border-left:3px solid #c66}}
+ .note{{color:#666;font-size:.85em}}
+</style>
+<h1>Floppy drives</h1>
+{msg}
+<form method=post action=/login>
+ <label>User<input name=user autofocus></label>
+ <label>Password<input name=password type=password></label>
+ <label>Realm<select name=realm>{realms}</select></label>
+ <button>Log in</button>
+</form>
+<p class=note>Proxmox accounts. <code>pam</code> is local to the node,
+<code>ad</code> is Samba AD over LDAPS. Needs {priv} on / in PVE.</p>
+"""
+
+
+def render_login(msg=""):
+    opts = "".join(f"<option>{html.escape(r)}</option>" for r in realms())
+    banner = f'<p class=msg>{html.escape(msg)}</p>' if msg else ""
+    return LOGIN.format(msg=banner, realms=opts, priv=REQUIRED_PRIV)
+
+
+def render(image_dir, user, msg="", err=False):
+    opts = "".join(f"<option>{html.escape(i)}</option>" for i in images(image_dir))
+    rows = []
+    for vm in vms():
+        if vm["configured"] is None:
+            buttons = '<button name=action value=attach>Attach drive</button>'
+            configured = "<span class=note>none</span>"
+        else:
+            buttons = (
+                '<button name=action value=attach>Set in args</button> '
+                '<button name=action value=detach>Detach drive</button>'
+            )
+            configured = html.escape(os.path.basename(vm["configured"]) or "(empty)")
+        if vm["running"] and vm["inserted"] is not None:
+            buttons += (
+                ' <button name=action value=insert>Insert now</button>'
+                ' <button name=action value=eject>Eject</button>'
+            )
+        if vm["inserted"] is None:
+            inserted = "<span class=note>no drive</span>"
+        else:
+            inserted = html.escape(os.path.basename(vm["inserted"]) or "(empty)")
+        rows.append(
+            ROW.format(
+                vmid=vm["vmid"],
+                name=html.escape(vm["name"]),
+                node=html.escape(vm["node"]),
+                state="running" if vm["running"] else "stopped",
+                configured=configured,
+                inserted=inserted,
+                options=opts,
+                buttons=buttons,
+                csrf=csrf_token(user),
+            )
+        )
+    banner = (
+        f'<p class="msg{" err" if err else ""}">{html.escape(msg)}</p>' if msg else ""
+    )
+    return PAGE.format(
+        phonebook=html.escape(phonebook_read(Handler.phonebook)),
+        phonebook_path=html.escape(Handler.phonebook),
+        msg=banner,
+        rows="".join(rows),
+        image_dir=html.escape(image_dir),
+        user=html.escape(user),
+        csrf=csrf_token(user),
+    )
+
+
+def act(image_dir, action, vmid, image):
+    """Perform one action. Every input is re-validated against live state.
+
+    fresh=True on purpose: acting on a cached view of the cluster is exactly the
+    case the cache must not cover, and the redirect afterwards has to show the
+    result, not the state from before the click.
+    """
+    _CACHE["rows"] = None
+    vm = next((v for v in vms(fresh=True) if str(v["vmid"]) == str(vmid)), None)
+    if not vm:
+        raise RuntimeError(f"no such VM: {vmid}")
+    path = None
+    if action in ("attach", "insert"):
+        if image not in images(image_dir):  # no path traversal, no arbitrary host file
+            raise RuntimeError(f"unknown image: {image}")
+        path = os.path.join(image_dir, image)
+    base = f"/nodes/{vm['node']}/qemu/{vm['vmid']}"
+
+    if action == "attach":
+        pvesh("set", f"{base}/config", "--args", args_with_floppy(vm["args"], path))
+        return f"{vmid}: args now carry {image}. Stop and start the VM to get the drive."
+    if action == "detach":
+        rest = args_without_floppy(vm["args"])
+        if rest:
+            pvesh("set", f"{base}/config", "--args", rest)
+        else:
+            pvesh("set", f"{base}/config", "--delete", "args")
+        return f"{vmid}: floppy removed from args. Takes effect on stop + start."
+    if action == "insert":
+        pvesh("create", f"{base}/monitor", "--command", f"change floppy0 {path}")
+        return f"{vmid}: inserted {image}."
+    if action == "eject":
+        pvesh("create", f"{base}/monitor", "--command", "eject floppy0")
+        return f"{vmid}: ejected."
+    raise RuntimeError(f"unknown action: {action}")
+
+
+class Handler(BaseHTTPRequestHandler):
+    image_dir = "/mnt/pve/laptop/template/iso"
+    secure = False  # set when serving TLS, so the cookie can demand it
+
+    phonebook = "/etc/pve/retro-phonebook"
+
+    def _send(self, code, body, headers=()):
+        data = body.encode()
+        self.send_response(code)
+        self.send_header("Content-Type", "text/html; charset=utf-8")
+        self.send_header("Content-Length", str(len(data)))
+        for k, v in headers:
+            self.send_header(k, v)
+        self.end_headers()
+        self.wfile.write(data)
+
+    def _redirect(self, msg="", err=False):
+        q = urllib.parse.urlencode({"msg": msg, **({"err": "1"} if err else {})})
+        self._send(303, "", [("Location", f"/?{q}" if msg else "/")])
+
+    def _user(self):
+        cookie = SimpleCookie(self.headers.get("Cookie", "")).get("floppy")
+        return session_user(cookie.value) if cookie else None
+
+    def _form(self):
+        n = int(self.headers.get("Content-Length", 0))
+        return urllib.parse.parse_qs(self.rfile.read(n).decode())
+
+    def do_GET(self):
+        if self.path.split("?")[0] != "/":
+            return self._send(404, "not found")
+        user = self._user()
+        q = urllib.parse.parse_qs(urllib.parse.urlparse(self.path).query)
+        try:
+            page = (
+                render(self.image_dir, user, q.get("msg", [""])[0], "err" in q)
+                if user
+                else render_login(q.get("msg", [""])[0])
+            )
+        except Exception as e:  # a broken pvesh must show up, not 500 silently
+            page = f"<h1>Floppy drives</h1><p class=msg>pvesh failed: {html.escape(str(e))}</p>"
+        self._send(200, page)
+
+    def do_POST(self):
+        path = self.path.split("?")[0]
+        form = self._form()
+
+        if path == "/login":
+            user = form.get("user", [""])[0].strip()
+            # Everyone types the name they use everywhere else, `me@ad`, which
+            # would become `me@ad@ad` and fail as "no such user". A sAMAccountName
+            # cannot contain '@', so dropping a realm suffix is unambiguous.
+            user = user.split("@")[0]
+            realm = form.get("realm", [""])[0]
+            if realm not in realms() or not user:
+                return self._send(200, render_login("pick a valid realm"))
+            userid = f"{user}@{realm}"
+            try:
+                ok = pve_authenticate(userid, form.get("password", [""])[0])
+            except RuntimeError as e:
+                return self._send(200, render_login(str(e)))
+            if not ok:
+                # One message for both cases on purpose: a distinct "no such
+                # user" would enumerate the directory for anyone who can reach this.
+                return self._send(200, render_login("login failed"))
+            if not pve_authorized(userid):
+                return self._send(200, render_login(f"{userid} lacks {REQUIRED_PRIV} on /"))
+            cookie = (
+                f"floppy={session_new(userid)}; Path=/; HttpOnly; SameSite=Strict"
+                + ("; Secure" if self.secure else "")
+            )
+            return self._send(303, "", [("Location", "/"), ("Set-Cookie", cookie)])
+
+        user = self._user()
+        if not user:
+            return self._send(200, render_login("session expired"))
+        # SameSite=Strict already blocks cross-site posts in current browsers;
+        # the token is what covers the ones that do not implement it.
+        if not hmac.compare_digest(form.get("csrf", [""])[0], csrf_token(user)):
+            return self._send(400, "bad csrf token")
+
+        if path == "/phonebook":
+            text = form.get("text", [""])[0]
+            bad = phonebook_check(text)
+            if bad:
+                return self._redirect(bad, err=True)
+            try:
+                phonebook_write(self.phonebook, text)
+            except OSError as e:
+                return self._redirect(f"could not write phonebook: {e}", err=True)
+            print(f"{user} saved phonebook", flush=True)
+            return self._redirect("phonebook saved")
+
+        if path == "/logout":
+            return self._send(
+                303, "", [("Location", "/"), ("Set-Cookie", "floppy=; Path=/; Max-Age=0")]
+            )
+        if path != "/":
+            return self._send(404, "not found")
+
+        action = form.get("action", [""])[0]
+        vmid = form.get("vmid", [""])[0]
+        try:
+            msg = act(self.image_dir, action, vmid, form.get("image", [""])[0])
+            _CACHE["rows"] = None  # the next render must show what just happened
+            # Who did what to which VM belongs in the journal -- this app hands
+            # out an action nothing else in PVE records.
+            print(f"{user} {action} vm {vmid}: {msg}", flush=True)
+            self._redirect(msg)
+        except Exception as e:
+            print(f"{user} {action} vm {vmid} FAILED: {e}", flush=True)
+            self._redirect(str(e), err=True)
+
+    def log_message(self, fmt, *a):
+        pass  # journald already has the systemd unit's own noise
+
+
+def selftest():
+    # sessions: a valid one round-trips, a tampered or expired one does not
+    s = session_new("me@ad")
+    assert session_user(s) == "me@ad"
+    # Repeated on purpose: a raw-digest signature only broke ~12% of the time,
+    # so one round trip passed the test and still logged people out at random.
+    for _ in range(300):
+        assert session_user(session_new("me@ad")) == "me@ad", "flaky session token"
+    assert session_user(s[:-4] + "AAAA") is None
+    assert session_user("garbage") is None
+    global SESSION_TTL
+    SESSION_TTL, old = -1, SESSION_TTL
+    assert session_user(session_new("me@ad")) is None, "expiry not enforced"
+    SESSION_TTL = old
+    assert csrf_token("me@ad") != csrf_token("you@ad")
+
+    a = "-drive if=floppy,format=raw,file=/iso/a.img"
+    assert floppy_in_args("") is None
+    assert floppy_in_args(a) == "/iso/a.img"
+    assert args_with_floppy("", "/iso/a.img") == a
+    # replaces in place, does not duplicate, and leaves unrelated args alone
+    mixed = f"-cpu foo {a} -boot order=a"
+    swapped = args_with_floppy(mixed, "/iso/b.img")
+    assert swapped == "-cpu foo -drive if=floppy,format=raw,file=/iso/b.img -boot order=a", swapped
+    assert args_without_floppy(mixed) == "-cpu foo -boot order=a"
+    assert args_without_floppy(a) == ""
+    assert floppy_in_args("-drive if=floppy") == ""
+
+    # phonebook: the allowlist is the security boundary, so test what it REFUSES
+    assert phonebook_check("# just a comment\n5551212 ppp\n5551102 vm:102\n") is None
+    assert phonebook_check("5552323 telnet:bbs.example.com:23\n") is None
+    assert phonebook_check("5551000 ssh:[email protected]:22\n") is None
+    assert phonebook_check("5551212 192.168.10.127:6060  # trailing comment\n") is None
+    assert phonebook_check("5551212 exec:/bin/sh\n"), "exec: must be refused"
+    assert phonebook_check("5551212 ppp; rm -rf /\n"), "shell metacharacters must be refused"
+    assert phonebook_check("notanumber ppp\n"), "non-numeric number must be refused"
+    assert phonebook_check("5551212\n"), "a target is required"
+
+    # config parsing: args is full of colons, and snapshots follow the live config
+    import tempfile
+    conf = tempfile.NamedTemporaryFile("w", suffix=".conf", delete=False)
+    conf.write(
+        "name: retro-pdc\n"
+        "args: -drive if=floppy,format=raw,file=/mnt/pve/laptop/template/iso/a.img\n"
+        "cores: 1\n"
+        "[snap1]\n"
+        "args: -drive if=floppy,format=raw,file=/WRONG.img\n"
+    )
+    conf.close()
+    c = read_conf(conf.name)
+    assert c["name"] == "retro-pdc"
+    assert c["args"].endswith("/iso/a.img"), c["args"]      # colons survived
+    assert floppy_in_args(c["args"]).endswith("/iso/a.img")
+    assert c["cores"] == "1"
+    os.unlink(conf.name)
+    assert read_conf("/nonexistent/vm.conf") == {}          # missing is empty, not fatal
+    print("ok")
+
+
+if __name__ == "__main__":
+    ap = argparse.ArgumentParser(description=__doc__)
+    ap.add_argument("--listen", default="0.0.0.0")
+    ap.add_argument("--port", type=int, default=8088)
+    ap.add_argument("--image-dir", default=Handler.image_dir)
+    ap.add_argument("--phonebook", default=Handler.phonebook,
+                    help="modem phonebook; on pmxcfs so every node sees it")
+    # The node's own ACME certificate (role pve_acme), the one pveproxy serves.
+    ap.add_argument("--cert", default="/etc/pve/local/pveproxy-ssl.pem")
+    ap.add_argument("--key", default="/etc/pve/local/pveproxy-ssl.key")
+    ap.add_argument(
+        "--insecure",
+        action="store_true",
+        help="serve plain HTTP. There is a password form on this app, so this "
+        "puts credentials on the LAN in clear -- hence opt-in, not fallback.",
+    )
+    ap.add_argument("--selftest", action="store_true")
+    o = ap.parse_args()
+    if o.selftest:
+        selftest()
+        raise SystemExit(0)
+
+    Handler.image_dir = o.image_dir
+    Handler.phonebook = o.phonebook
+    srv = ThreadingHTTPServer((o.listen, o.port), Handler)
+    have_cert = os.path.exists(o.cert) and os.path.exists(o.key)
+    if not have_cert and not o.insecure:
+        # Fail closed: a login form on cleartext HTTP is worse than no service.
+        raise SystemExit(f"no certificate at {o.cert} -- pass --insecure to serve anyway")
+    if have_cert:
+        ctx = ssl.SSLContext(ssl.PROTOCOL_TLS_SERVER)
+        ctx.load_cert_chain(o.cert, o.key)
+        srv.socket = ctx.wrap_socket(srv.socket, server_side=True)
+        Handler.secure = True
+    print(f"listening on {'https' if Handler.secure else 'http'}://{o.listen}:{o.port}", flush=True)
+    srv.serve_forever()

infrastructure/proxmox/ansible/roles/pve_floppy/handlers/main.yml

@@ -0,0 +1,6 @@
+---
+- name: Restart pve-floppy-ui
+  ansible.builtin.systemd_service:
+    name: pve-floppy-ui
+    state: restarted
+    daemon_reload: true

infrastructure/proxmox/ansible/roles/pve_floppy/tasks/main.yml

@@ -0,0 +1,31 @@
+---
+# Deploy on ONE node only -- see defaults for why one instance covers the cluster.
+
+- name: Install the floppy UI
+  ansible.builtin.copy:
+    src: floppy-ui.py
+    dest: /usr/local/bin/pve-floppy-ui
+    mode: "0755"
+  notify: Restart pve-floppy-ui
+
+- name: Self-check the args parsing
+  # The one piece of real logic in the app: it rewrites the `args` string of a
+  # live VM, so a bad edit there corrupts a VM config. Cheap to assert, and it
+  # also proves the node's python can run the file at all.
+  ansible.builtin.command:
+    cmd: /usr/local/bin/pve-floppy-ui --selftest
+  changed_when: false
+
+- name: Install the unit
+  ansible.builtin.template:
+    src: pve-floppy-ui.service.j2
+    dest: /etc/systemd/system/pve-floppy-ui.service
+    mode: "0644"
+  notify: Restart pve-floppy-ui
+
+- name: Enable and start it
+  ansible.builtin.systemd_service:
+    name: pve-floppy-ui
+    state: started
+    enabled: true
+    daemon_reload: true

infrastructure/proxmox/ansible/roles/pve_floppy/templates/pve-floppy-ui.service.j2

@@ -0,0 +1,18 @@
+[Unit]
+Description=Floppy drive manager for PVE VMs
+Documentation=file:///usr/local/bin/pve-floppy-ui
+# pvesh talks to pveproxy, which needs the cluster fs mounted.
+After=pve-cluster.service pveproxy.service
+Wants=pve-cluster.service
+
+[Service]
+# Runs as root deliberately: the VM `args` option is root@pam-only, so there is
+# no lesser identity that can do this job -- not even an API token. Root is also
+# what can read the pveproxy TLS key it serves with; the app refuses to start
+# without a certificate rather than put a login form on cleartext HTTP.
+ExecStart=/usr/local/bin/pve-floppy-ui --listen {{ pve_floppy_listen }} --port {{ pve_floppy_port }} --image-dir {{ pve_floppy_image_dir }} --phonebook {{ pve_floppy_phonebook }}
+Restart=on-failure
+RestartSec=5
+
+[Install]
+WantedBy=multi-user.target

infrastructure/proxmox/ansible/roles/pve_ha/defaults/main.yml

@@ -0,0 +1,28 @@
+---
+# HA-managed guests. DELIBERATELY SHORT: the cluster's stated posture is "no HA,
+# guests are disposable" (see ../../../CLAUDE.md). Anything listed here is an
+# exception that has earned it, and the reason belongs in the comment.
+pve_ha_resources:
+  # vyos-rtr. It is the gateway for BOTH SDN VNets and an OSPF speaker, so losing
+  # it takes labnet+retronet offline (the main LAN is unaffected — the NEC IX is
+  # its gateway). Disk is on pve-rg (DRBD, place-count 2), so it can start on any
+  # node; pve3 would attach diskless.
+  # NOTE this buys crash-RESTART (~1-3 min outage), not seamless failover. Real
+  # gateway HA would be a second VyOS with VRRP.
+  - sid: "vm:100"
+    state: started
+    max_restart: 3
+    max_relocate: 2
+
+# --- Watchdog ---------------------------------------------------------------
+# ⚠ WHY THIS MATTERS: adding ANY HA resource arms fencing cluster-wide. PVE's
+# default is `softdog`, a SOFTWARE watchdog — a kernel timer, which CANNOT fire if
+# the kernel itself is frozen. That is precisely the failure this cluster has
+# actually seen (pve2's Raven Ridge idle freeze). A hardware watchdog is
+# independent silicon and fires regardless.
+#
+# Verified available 2026-07-26:
+#   pve1 (Intel i3-6100U) -> iTCO_wdt      (timeout 30s)
+#   pve2/pve3 (Ryzen 2400GE) -> sp5100_tco (timeout 60s)
+# Set per-host in inventory host_vars; empty string = leave PVE's softdog default.
+pve_ha_watchdog_module: ""

infrastructure/proxmox/ansible/roles/pve_ha/handlers/main.yml

@@ -0,0 +1,10 @@
+---
+# Deliberately does NOT reboot. Swapping the watchdog under an armed cluster is a
+# manual, ordered operation (see ../../../README-ha.md); a surprise rolling reboot
+# of all three nodes is exactly what you do not want an idempotent play to do.
+- name: reboot required for watchdog
+  ansible.builtin.debug:
+    msg: >-
+      Watchdog config changed on {{ inventory_hostname }}. It takes effect on the
+      NEXT REBOOT of this node. Until then fencing still uses softdog, which cannot
+      fence a frozen kernel. See proxmox/README-ha.md for the live-swap procedure.

infrastructure/proxmox/ansible/roles/pve_ha/tasks/main.yml

@@ -0,0 +1,76 @@
+---
+# Proxmox HA. Read roles/pve_ha/defaults/main.yml first — it explains why the
+# resource list is short and why the watchdog choice is not cosmetic.
+
+# --- Watchdog ---------------------------------------------------------------
+# Done BEFORE registering resources: adding a resource arms fencing, and arming
+# fencing on a software watchdog is the weakest configuration.
+- name: Select the hardware watchdog module
+  ansible.builtin.lineinfile:
+    path: /etc/default/pve-ha-manager
+    regexp: '^#?\s*WATCHDOG_MODULE='
+    line: "WATCHDOG_MODULE={{ pve_ha_watchdog_module }}"
+    create: false
+  when: pve_ha_watchdog_module | length > 0
+  notify: reboot required for watchdog
+  tags: [ha, watchdog]
+
+# watchdog-mux opens /dev/watchdog, which belongs to whichever watchdog registered
+# FIRST. PVE loads softdog at boot, so it wins and the hardware module ends up as
+# an unused watchdog1. Blacklisting softdog is what actually makes the hardware
+# one take effect — setting WATCHDOG_MODULE alone does nothing.
+- name: Blacklist softdog so the hardware watchdog claims /dev/watchdog
+  ansible.builtin.copy:
+    dest: /etc/modprobe.d/pve-ha-watchdog.conf
+    mode: "0644"
+    content: |
+      # Managed by ansible (roles/pve_ha). See that role for the reasoning.
+      # softdog cannot fence a frozen kernel; blacklisting it lets
+      # {{ pve_ha_watchdog_module }} register as watchdog0 and own /dev/watchdog.
+      blacklist softdog
+  when: pve_ha_watchdog_module | length > 0
+  notify: reboot required for watchdog
+  tags: [ha, watchdog]
+
+# ⚠ NOT DONE LIVE ON PURPOSE. Swapping the watchdog on a running node means
+# stopping watchdog-mux and unloading softdog while fencing is armed — get the
+# order wrong and the node self-fences (reboots). The safe live procedure is
+# documented in ../../README-ha.md; otherwise it simply takes effect on the next
+# reboot, which is why the handler only WARNS rather than rebooting anything.
+
+# --- Resources ---------------------------------------------------------------
+- name: Read current HA resources
+  ansible.builtin.command: ha-manager status
+  register: pve_ha_status
+  changed_when: false
+  failed_when: false
+  run_once: true
+  tags: [ha]
+
+- name: Register HA resources
+  ansible.builtin.command: >-
+    ha-manager add {{ item.sid }}
+    --state {{ item.state }}
+    --max_restart {{ item.max_restart }}
+    --max_relocate {{ item.max_relocate }}
+  loop: "{{ pve_ha_resources }}"
+  loop_control:
+    label: "{{ item.sid }}"
+  # `ha-manager add` errors if the resource already exists, so gate on the status
+  # output. This is what keeps a second run at changed=0.
+  when: "'service ' ~ item.sid not in (pve_ha_status.stdout | default(''))"
+  run_once: true
+  tags: [ha]
+
+- name: Verify HA is armed and the resource is known
+  ansible.builtin.command: ha-manager status
+  register: pve_ha_verify
+  changed_when: false
+  run_once: true
+  tags: [ha]
+
+- name: Show HA state
+  ansible.builtin.debug:
+    msg: "{{ pve_ha_verify.stdout_lines }}"
+  run_once: true
+  tags: [ha]

infrastructure/proxmox/ansible/roles/pve_kernel_params/handlers/main.yml

@@ -0,0 +1,7 @@
+---
+- name: Update grub
+  # Rewrites /boot/grub/grub.cfg. Takes effect on the NEXT boot only -- this
+  # role intentionally never reboots a node on its own.
+  ansible.builtin.command:
+    cmd: update-grub
+  changed_when: true

infrastructure/proxmox/ansible/roles/pve_kernel_params/tasks/main.yml

@@ -0,0 +1,33 @@
+---
+# Per-host kernel cmdline additions, driven by `pve_kernel_cmdline_extra` in the
+# inventory. Currently only pve2 sets it (Raven Ridge random-freeze workaround);
+# pve3 is identical silicon but is deliberately left alone -- see inventory.
+#
+# This is a no-op on hosts that do not define the variable.
+
+- name: Set GRUB_CMDLINE_LINUX_DEFAULT
+  ansible.builtin.lineinfile:
+    path: /etc/default/grub
+    regexp: '^GRUB_CMDLINE_LINUX_DEFAULT='
+    line: 'GRUB_CMDLINE_LINUX_DEFAULT="quiet {{ pve_kernel_cmdline_extra }}"'
+    backup: true
+  when: pve_kernel_cmdline_extra is defined and pve_kernel_cmdline_extra | length > 0
+  notify: Update grub
+
+- name: Check which extra params are actually active
+  # /proc/cmdline is the only honest source: editing grub without running
+  # update-grub, or rebooting, silently leaves the fix inactive.
+  ansible.builtin.command:
+    cmd: cat /proc/cmdline
+  register: _cmdline
+  changed_when: false
+
+- name: Warn when a configured param is not yet live
+  ansible.builtin.debug:
+    msg: >-
+      {{ inventory_hostname }}: "{{ item }}" is configured but NOT active in
+      /proc/cmdline -- a reboot is required for it to take effect.
+  loop: "{{ pve_kernel_cmdline_extra.split() }}"
+  when:
+    - pve_kernel_cmdline_extra is defined and pve_kernel_cmdline_extra | length > 0
+    - item not in _cmdline.stdout

infrastructure/proxmox/ansible/roles/pve_linstor/defaults/main.yml

@@ -0,0 +1,46 @@
+---
+# LINSTOR/DRBD replacing Ceph. Ceph was tried on HDD OSDs + SSD DB and was far
+# too slow; LINSTOR wins here through DATA LOCALITY — a VM reads from the copy on
+# its own node instead of pulling every block across a single flat 1G LAN.
+
+# --- LINBIT public repo (free, no subscription) ---
+pve_linstor_repo_suite: proxmox-9
+pve_linstor_repo_url: "http://packages.linbit.com/public/"
+pve_linstor_repo_key_url: "https://packages.linbit.com/package-signing-pubkey.asc"
+pve_linstor_keyring: /usr/share/keyrings/linbit-keyring.gpg
+
+# --- Roles ---
+# Controller on pve1: most RAM (16G vs 8G). Single controller, no HA — this
+# cluster deliberately holds nothing critical.
+pve_linstor_controller: pve1
+
+# --- Storage pools ---
+# SSD = VM disks (fast). Built from the space the installer left free beyond the
+# 60G `pve` VG: ~107G on pve1's SATA SSD, ~178G on the ThinkCentres' NVMe.
+pve_linstor_ssd_vg: vg_ssd
+pve_linstor_ssd_pool: ssd
+
+# HDD = bulk. Whole 1TB spindle. pve2/pve3 still carry STALE CEPH VGs here that
+# must be wiped first; pve1's was already cleared.
+pve_linstor_hdd_vg: vg_hdd
+pve_linstor_hdd_pool: hdd
+pve_linstor_hdd_disk: /dev/sda
+
+# --- Resource groups ---
+# place-count 2 = two replicas across three nodes: survives losing any one node
+# while costing 2x rather than 3x capacity.
+#
+# ONE PER STORAGE POOL. A resource group binds to a single pool, so without an
+# hdd group the ~2.7TB of spinning disk is provisioned in LINSTOR but completely
+# unreachable from Proxmox — pools exist, nothing can allocate from them.
+pve_linstor_place_count: 2
+pve_linstor_resource_groups:
+  - name: pve-rg            # fast: VM/CT root disks
+    pool: "{{ pve_linstor_ssd_pool }}"
+    content: images,rootdir
+  - name: pve-rg-hdd        # bulk: large/cold volumes on the 1TB spindles
+    pool: "{{ pve_linstor_hdd_pool }}"
+    content: images,rootdir
+
+# Safety: wiping a disk is destructive and irreversible. Must be set explicitly.
+pve_linstor_wipe_hdd: false

infrastructure/proxmox/ansible/roles/pve_linstor/tasks/linstor.yml

@@ -0,0 +1,168 @@
+---
+# LINSTOR cluster: controller on one node, satellites everywhere, storage pools,
+# resource group, and the PVE storage entry.
+#
+# All linstor CLI calls run ONCE against the controller — LINSTOR is itself a
+# cluster-wide database, so repeating them per-host would just race.
+
+- name: Enable and start the satellite on every node
+  ansible.builtin.systemd_service:
+    name: linstor-satellite
+    enabled: true
+    state: started
+
+- name: Enable and start the controller
+  ansible.builtin.systemd_service:
+    name: linstor-controller
+    enabled: true
+    state: started
+  when: inventory_hostname == pve_linstor_controller
+
+- name: Point the client at the controller
+  # Without this, `linstor` talks to localhost and fails on the satellites.
+  ansible.builtin.copy:
+    dest: /etc/linstor/linstor-client.conf
+    mode: "0644"
+    content: |
+      # Managed by Ansible (role pve_linstor).
+      [global]
+      controllers=linstor://{{ hostvars[pve_linstor_controller].ansible_host }}
+
+- name: Wait for the controller API
+  ansible.builtin.command:
+    cmd: linstor node list
+  register: _lin_ready
+  until: _lin_ready.rc == 0
+  retries: 24
+  delay: 5
+  changed_when: false
+  run_once: true
+  delegate_to: "{{ pve_linstor_controller }}"
+
+# ── nodes ─────────────────────────────────────────────────────────────────
+- name: Register each node with LINSTOR
+  ansible.builtin.command:
+    cmd: >-
+      linstor node create {{ item }} {{ hostvars[item].ansible_host }}
+      --node-type {{ 'Combined' if item == pve_linstor_controller else 'Satellite' }}
+  loop: "{{ groups['pve'] }}"
+  register: _node_create
+  changed_when: "'successfully' in (_node_create.stdout | default('') | lower)"
+  failed_when:
+    - _node_create.rc != 0
+    - "'already exists' not in (_node_create.stdout | default('') + _node_create.stderr | default('')) | lower"
+  run_once: true
+  delegate_to: "{{ pve_linstor_controller }}"
+
+- name: Wait for all nodes to come ONLINE
+  ansible.builtin.shell:
+    cmd: |
+      set -o pipefail
+      linstor -m --output-version v1 node list \
+        | python3 -c "import json,sys; print(all(n['connection_status']=='ONLINE' for n in json.load(sys.stdin)[0]))"
+    executable: /bin/bash
+  register: _nodes_online
+  until: "'True' in _nodes_online.stdout"
+  retries: 24
+  delay: 5
+  changed_when: false
+  run_once: true
+  delegate_to: "{{ pve_linstor_controller }}"
+
+# ── storage pools ─────────────────────────────────────────────────────────
+- name: Create the storage pools
+  ansible.builtin.command:
+    cmd: >-
+      linstor storage-pool create lvm {{ item.0 }} {{ item.1.pool }} {{ item.1.vg }}
+  loop: "{{ groups['pve'] | product([
+              {'pool': pve_linstor_ssd_pool, 'vg': pve_linstor_ssd_vg},
+              {'pool': pve_linstor_hdd_pool, 'vg': pve_linstor_hdd_vg}]) | list }}"
+  register: _sp
+  changed_when: "'successfully' in (_sp.stdout | default('') | lower)"
+  failed_when:
+    - _sp.rc != 0
+    - "'already exists' not in (_sp.stdout | default('') + _sp.stderr | default('')) | lower"
+  run_once: true
+  delegate_to: "{{ pve_linstor_controller }}"
+
+# ── resource groups ───────────────────────────────────────────────────────
+# One per storage pool — a resource group binds to exactly one pool, so the hdd
+# pool needs its own or that capacity is unusable from Proxmox.
+- name: Create the resource groups
+  ansible.builtin.command:
+    cmd: >-
+      linstor resource-group create {{ item.name }}
+      --storage-pool {{ item.pool }}
+      --place-count {{ item.place_count | default(pve_linstor_place_count) }}
+  loop: "{{ pve_linstor_resource_groups }}"
+  loop_control:
+    label: "{{ item.name }} -> {{ item.pool }}"
+  register: _rg
+  changed_when: "'successfully' in (_rg.stdout | default('') | lower)"
+  failed_when:
+    - _rg.rc != 0
+    - "'already exists' not in (_rg.stdout | default('') + _rg.stderr | default('')) | lower"
+  run_once: true
+  delegate_to: "{{ pve_linstor_controller }}"
+
+- name: Read existing volume-group definitions
+  # `linstor volume-group create` is NOT idempotent: every call APPENDS another
+  # volume number. Running the play three times left pve-rg with VlmNrs 0,1,2, so
+  # PVE then failed every disk create with "has 3 Volume groups, but only 1 sizes
+  # were provided". Check before creating.
+  ansible.builtin.shell:
+    # Count table rows. NOTE the leading char class: linstor draws its tables
+    # with BOX-DRAWING pipes (U+2502), not ASCII '|', so a '^\\|' pattern never
+    # matches and the count is always 0 -- which silently appends a NEW volume
+    # group on every run.
+    cmd: "linstor volume-group list {{ item.name }} 2>/dev/null | grep -cE '^.[[:space:]]*[0-9]+[[:space:]]' || true"
+  loop: "{{ pve_linstor_resource_groups }}"
+  loop_control:
+    label: "{{ item.name }}"
+  register: _vg_count
+  changed_when: false
+  check_mode: false
+  run_once: true
+  delegate_to: "{{ pve_linstor_controller }}"
+
+- name: Create a volume group where none exists
+  ansible.builtin.command:
+    cmd: "linstor volume-group create {{ item.item.name }}"
+  loop: "{{ _vg_count.results }}"
+  loop_control:
+    label: "{{ item.item.name }} (has {{ item.stdout | default('?') | trim }})"
+  when: (item.stdout | default('0') | trim | int) == 0
+  run_once: true
+  delegate_to: "{{ pve_linstor_controller }}"
+
+# ── PVE storage entries ───────────────────────────────────────────────────
+- name: Register each resource group as PVE storage
+  ansible.builtin.command:
+    cmd: >-
+      pvesm add drbd {{ item.name }}
+      --content {{ item.content | default('images,rootdir') }}
+      --controller {{ hostvars[pve_linstor_controller].ansible_host }}
+      --resourcegroup {{ item.name }}
+  loop: "{{ pve_linstor_resource_groups }}"
+  loop_control:
+    label: "{{ item.name }}"
+  register: _pvesm
+  changed_when: _pvesm.rc == 0
+  failed_when:
+    - _pvesm.rc != 0
+    - "'already defined' not in (_pvesm.stderr | default('') + _pvesm.stdout | default(''))"
+  run_once: true
+  delegate_to: "{{ pve_linstor_controller }}"
+
+- name: Report LINSTOR state
+  ansible.builtin.shell:
+    cmd: "linstor storage-pool list; echo; linstor resource-group list; echo; pvesm status"
+  register: _lin_state
+  changed_when: false
+  run_once: true
+  delegate_to: "{{ pve_linstor_controller }}"
+
+- name: Show it
+  ansible.builtin.debug:
+    msg: "{{ _lin_state.stdout_lines }}"
+  run_once: true

infrastructure/proxmox/ansible/roles/pve_linstor/tasks/main.yml

@@ -0,0 +1,15 @@
+---
+# LINSTOR/DRBD hyperconverged storage. Split into stages so the destructive part
+# (disk wiping) is separately gated and cannot run by accident.
+
+- name: Install packages
+  ansible.builtin.import_tasks: packages.yml
+  tags: [linstor_pkgs]
+
+- name: Prepare backing storage (LVM)
+  ansible.builtin.import_tasks: storage.yml
+  tags: [linstor_storage]
+
+- name: Configure the LINSTOR cluster
+  ansible.builtin.import_tasks: linstor.yml
+  tags: [linstor_config]

infrastructure/proxmox/ansible/roles/pve_linstor/tasks/packages.yml

@@ -0,0 +1,112 @@
+---
+# LINBIT's public repo. Free and needs no subscription — the subscription only
+# buys support and their prebuilt kernel modules; drbd-dkms builds locally.
+
+- name: Install the LINBIT signing key
+  ansible.builtin.get_url:
+    url: "{{ pve_linstor_repo_key_url }}"
+    dest: /tmp/linbit-pubkey.asc
+    mode: "0644"
+  retries: 3
+  delay: 10
+
+- name: Convert the key to a keyring
+  ansible.builtin.shell:
+    cmd: "gpg --dearmor < /tmp/linbit-pubkey.asc > {{ pve_linstor_keyring }}"
+    creates: "{{ pve_linstor_keyring }}"
+
+- name: Add the LINBIT repository
+  ansible.builtin.deb822_repository:
+    name: linbit
+    types: [deb]
+    uris: "{{ pve_linstor_repo_url }}"
+    suites: "{{ pve_linstor_repo_suite }}"
+    components: [drbd-9]
+    signed_by: "{{ pve_linstor_keyring }}"
+    enabled: true
+    state: present
+  register: _linbit_repo
+
+- name: Update apt cache
+  ansible.builtin.apt:
+    update_cache: true
+  register: _apt
+  retries: 3
+  delay: 15
+  until: _apt is succeeded
+
+- name: Install DRBD + LINSTOR packages
+  # drbd-dkms COMPILES a kernel module against the running kernel, so the
+  # headers must match. This is the step most likely to fail on a flaky uplink,
+  # hence the retries.
+  ansible.builtin.apt:
+    name:
+      - "pve-headers-{{ ansible_facts['kernel'] }}"
+      - drbd-dkms
+      - drbd-utils
+      - linstor-controller
+      - linstor-satellite
+      - linstor-client
+      - linstor-proxmox
+    state: present
+  register: _linstor_pkgs
+  retries: 2
+  delay: 30
+  until: _linstor_pkgs is succeeded
+
+- name: Load the DRBD module
+  ansible.builtin.command:
+    cmd: modprobe drbd
+  changed_when: false
+
+- name: Check which DRBD version is actually LOADED
+  ansible.builtin.shell:
+    cmd: "cat /proc/drbd 2>/dev/null | head -1"
+  register: _drbd_ver
+  changed_when: false
+
+# The kernel ships an IN-TREE drbd 8.4. If anything loaded it before drbd-dkms
+# was installed, modprobe is a no-op and the running module stays 8.4 even
+# though dkms built 9.x correctly (modinfo will happily report 9.x from
+# .../updates/dkms/drbd.ko). LINSTOR needs 9. Swap it live when nothing is using
+# it; refuse and demand a reboot when something is.
+- name: Check the module refcount before swapping
+  ansible.builtin.shell:
+    cmd: "lsmod | awk '$1==\"drbd\" {print $3}' | head -1"
+  register: _drbd_refs
+  changed_when: false
+  when: "'version: 9' not in _drbd_ver.stdout"
+
+- name: Refuse to swap a DRBD module that is in use
+  ansible.builtin.fail:
+    msg: >-
+      In-tree DRBD {{ _drbd_ver.stdout }} is loaded and IN USE
+      (refcount {{ _drbd_refs.stdout | default('?') }}). Reboot
+      {{ inventory_hostname }} to pick up the dkms-built DRBD 9.
+  when:
+    - "'version: 9' not in _drbd_ver.stdout"
+    - (_drbd_refs.stdout | default('0') | trim | int) > 0
+
+- name: Swap the in-tree DRBD 8.4 for the dkms-built DRBD 9
+  ansible.builtin.shell:
+    cmd: |
+      set -e
+      modprobe -r drbd_transport_tcp 2>/dev/null || true
+      modprobe -r drbd
+      depmod -a
+      modprobe drbd
+  when:
+    - "'version: 9' not in _drbd_ver.stdout"
+    - (_drbd_refs.stdout | default('0') | trim | int) == 0
+
+- name: Re-read the DRBD version after the swap
+  ansible.builtin.shell:
+    cmd: "cat /proc/drbd 2>/dev/null | head -1"
+  register: _drbd_ver
+  changed_when: false
+
+- name: Assert DRBD 9
+  ansible.builtin.assert:
+    that: "'version: 9' in _drbd_ver.stdout"
+    fail_msg: "Expected DRBD 9, got: {{ _drbd_ver.stdout }}"
+    quiet: true

infrastructure/proxmox/ansible/roles/pve_linstor/tasks/storage.yml

@@ -0,0 +1,176 @@
+---
+# LVM volume groups that back the LINSTOR pools.
+#
+# DESTRUCTIVE. Everything that destroys data is gated behind
+# pve_linstor_wipe_hdd and guarded by an explicit "is this really free?" check.
+
+# ── SSD pool: the space the installer left beyond the 60G `pve` VG ─────────
+- name: Find the SSD holding the pve VG
+  ansible.builtin.shell:
+    cmd: |
+      set -o pipefail
+      pvs --noheadings -o pv_name --select 'vg_name=pve' | tr -d ' ' | head -1
+    executable: /bin/bash
+  register: _pve_pv
+  changed_when: false
+  check_mode: false          # read-only discovery; must run in --check
+
+- name: Derive the parent disk holding the pve PV
+  # DO NOT use `lsblk -no PKNAME <pv>`: without --nodeps lsblk also lists the LVs
+  # stacked on the partition, whose PKNAME is the PARTITION itself, so `head -1`
+  # can return e.g. "nvme0n1p3" instead of "nvme0n1". That mistake made sgdisk
+  # write a GPT INTO the LVM PV holding root on pve2/pve3 and destroyed their
+  # `pve` VG metadata (2026-07-25). sysfs is unambiguous: the parent of
+  # /sys/class/block/<part> IS the disk.
+  ansible.builtin.shell:
+    cmd: |
+      set -o pipefail
+      PV="{{ _pve_pv.stdout | trim }}"
+      PART="$(basename "$PV")"
+      DISK="$(basename "$(readlink -f "/sys/class/block/${PART}/..")")"
+      echo "/dev/${DISK}"
+    executable: /bin/bash
+  register: _ssd_disk
+  changed_when: false
+  check_mode: false          # read-only discovery; must run in --check
+
+- name: Refuse to proceed unless that really is a whole disk
+  # Last line of defence: sgdisk against a partition is destructive, so verify
+  # the derived device is TYPE=disk and not a partition before touching it.
+  ansible.builtin.shell:
+    cmd: "lsblk -dno TYPE {{ _ssd_disk.stdout | trim }}"
+  register: _ssd_type
+  changed_when: false
+  check_mode: false          # read-only discovery; must run in --check
+
+- name: Assert it is a disk
+  ansible.builtin.assert:
+    that: "(_ssd_type.stdout | trim) == 'disk'"
+    fail_msg: >-
+      Derived SSD device {{ _ssd_disk.stdout | trim }} is TYPE
+      '{{ _ssd_type.stdout | trim }}', not 'disk'. Refusing to partition it —
+      running sgdisk against a partition destroys whatever is on it.
+    quiet: true
+
+- name: Check whether the SSD VG already exists
+  ansible.builtin.command:
+    cmd: "vgs {{ pve_linstor_ssd_vg }}"
+  register: _ssd_vg
+  changed_when: false
+  failed_when: false
+
+- name: Create a partition from the SSD's free tail
+  # sgdisk -N uses ALL remaining free space for the next partition number.
+  # Only touches unallocated space; the existing pve partitions are untouched.
+  ansible.builtin.command:
+    cmd: "sgdisk -N 0 -t 0:8e00 -c 0:linstor-ssd {{ _ssd_disk.stdout | trim }}"
+  when: _ssd_vg.rc != 0
+  register: _ssd_part
+
+- name: Re-read the partition table
+  # partx (util-linux) rather than partprobe: `parted` is NOT installed on a
+  # stock PVE 9 node, so partprobe fails with "No such file or directory".
+  ansible.builtin.shell:
+    cmd: |
+      set -e
+      partx -u "{{ _ssd_disk.stdout | trim }}"
+      udevadm settle
+    executable: /bin/bash
+  when: _ssd_part is changed
+  changed_when: true
+
+- name: Find the new SSD partition
+  ansible.builtin.shell:
+    cmd: |
+      set -o pipefail
+      lsblk -rno NAME,PARTLABEL "{{ _ssd_disk.stdout | trim }}" \
+        | awk '$2=="linstor-ssd" {print "/dev/"$1}' | head -1
+    executable: /bin/bash
+  register: _ssd_partdev
+  changed_when: false
+
+- name: Create the SSD volume group
+  ansible.builtin.command:
+    cmd: "vgcreate {{ pve_linstor_ssd_vg }} {{ _ssd_partdev.stdout | trim }}"
+  when:
+    - _ssd_vg.rc != 0
+    - (_ssd_partdev.stdout | trim) | length > 0
+
+# ── HDD pool: the whole 1TB spindle ───────────────────────────────────────
+- name: Confirm the HDD target is a whole disk
+  # Same guard as the SSD path. wipefs/vgcreate against a partition by mistake is
+  # how the pve VG on pve2/pve3 got destroyed on 2026-07-25; assert the device
+  # type rather than trusting the variable.
+  ansible.builtin.shell:
+    cmd: "lsblk -dno TYPE {{ pve_linstor_hdd_disk }}"
+  register: _hdd_type
+  changed_when: false
+  check_mode: false          # read-only discovery; must run in --check
+
+- name: Assert the HDD target is a disk
+  ansible.builtin.assert:
+    that: "(_hdd_type.stdout | trim) == 'disk'"
+    fail_msg: >-
+      {{ pve_linstor_hdd_disk }} is TYPE '{{ _hdd_type.stdout | trim }}', not
+      'disk'. Refusing to wipe it.
+    quiet: true
+
+- name: Inspect the HDD's current volume group
+  ansible.builtin.shell:
+    cmd: |
+      set -o pipefail
+      pvs --noheadings -o vg_name {{ pve_linstor_hdd_disk }} 2>/dev/null | tr -d ' '
+    executable: /bin/bash
+  register: _hdd_vg
+  changed_when: false
+  check_mode: false          # read-only discovery; must run in --check
+  failed_when: false
+
+- name: Refuse to wipe a HDD carrying anything other than a stale Ceph VG
+  # The only VGs we expect here are the leftover ceph-<uuid> ones from the old
+  # cluster. Anything else means this disk is not what we think it is.
+  ansible.builtin.fail:
+    msg: >-
+      {{ pve_linstor_hdd_disk }} on {{ inventory_hostname }} holds VG
+      '{{ _hdd_vg.stdout | trim }}', which is not a stale ceph-* VG.
+      Refusing to wipe. Inspect it by hand.
+  when:
+    - (_hdd_vg.stdout | trim) | length > 0
+    - not (_hdd_vg.stdout | trim).startswith('ceph-')
+    - (_hdd_vg.stdout | trim) != pve_linstor_hdd_vg
+
+- name: Remove the stale Ceph volume group
+  ansible.builtin.command:
+    cmd: "vgremove -f {{ _hdd_vg.stdout | trim }}"
+  when:
+    - pve_linstor_wipe_hdd | bool
+    - (_hdd_vg.stdout | trim).startswith('ceph-')
+
+- name: Wipe the HDD's signatures
+  ansible.builtin.command:
+    cmd: "wipefs -a {{ pve_linstor_hdd_disk }}"
+  when:
+    - pve_linstor_wipe_hdd | bool
+    - (_hdd_vg.stdout | trim).startswith('ceph-')
+
+- name: Check whether the HDD VG already exists
+  ansible.builtin.command:
+    cmd: "vgs {{ pve_linstor_hdd_vg }}"
+  register: _hdd_vg_now
+  changed_when: false
+  failed_when: false
+
+- name: Create the HDD volume group
+  ansible.builtin.command:
+    cmd: "vgcreate {{ pve_linstor_hdd_vg }} {{ pve_linstor_hdd_disk }}"
+  when: _hdd_vg_now.rc != 0
+
+- name: Report the resulting volume groups
+  ansible.builtin.command:
+    cmd: "vgs --noheadings -o vg_name,vg_size,vg_free"
+  register: _vgs
+  changed_when: false
+
+- name: Show them
+  ansible.builtin.debug:
+    msg: "{{ _vgs.stdout_lines }}"

infrastructure/proxmox/ansible/roles/pve_mail_relay/handlers/main.yml

@@ -0,0 +1,5 @@
+---
+- name: Reload postfix
+  ansible.builtin.systemd_service:
+    name: postfix
+    state: reloaded

infrastructure/proxmox/ansible/roles/pve_mail_relay/tasks/main.yml

@@ -0,0 +1,56 @@
+---
+# Route all node mail (PVE alerts, smartd, cron) through the k3s smtp-relay on
+# the laptop, which authenticates to M365 with OAuth2/XOAUTH2.
+# See services/smtp-relay/ for the relay side.
+#
+# Applied by hand on 2026-07-25; codified here so a rebuild does not lose it.
+#
+# Each setting is READ first and only written when it differs, so re-runs report
+# no change. `check_mode: false` on the reads lets --check make a real
+# comparison instead of skipping (postconf reads nothing but config).
+
+- name: Read current postfix settings
+  ansible.builtin.command:
+    cmd: postconf -h relayhost sender_canonical_classes sender_canonical_maps
+  register: _pf
+  changed_when: false
+  check_mode: false
+
+- name: Point postfix at the smtp-relay
+  # [brackets] mean "this is the literal host", suppressing the MX lookup that
+  # would otherwise be attempted for a bare address.
+  ansible.builtin.command:
+    cmd: postconf -e "relayhost = {{ pve_mail_relayhost }}"
+  when: (_pf.stdout_lines[0] | default('') | trim) != pve_mail_relayhost
+  notify: Reload postfix
+
+- name: Install the sender rewrite map
+  # M365 authenticates as {{ pve_mail_from }} and rejects any other From with
+  # 5.7.60 SendAsDenied. Node mail is generated as root@<fqdn>, so every local
+  # sender must be rewritten. In-cluster apps (Authelia, Gitea) never hit this
+  # because they already send as noreply@.
+  ansible.builtin.copy:
+    dest: /etc/postfix/sender_canonical
+    mode: "0644"
+    content: |
+      # Managed by Ansible (role pve_mail_relay).
+      /.+/	{{ pve_mail_from }}
+  notify: Reload postfix
+
+- name: Enable sender rewriting for the envelope
+  ansible.builtin.command:
+    cmd: postconf -e "sender_canonical_classes = envelope_sender, header_sender"
+  when: (_pf.stdout_lines[1] | default('') | trim) != 'envelope_sender, header_sender'
+  notify: Reload postfix
+
+- name: Point postfix at the sender rewrite map
+  ansible.builtin.command:
+    cmd: postconf -e "sender_canonical_maps = regexp:/etc/postfix/sender_canonical"
+  when: (_pf.stdout_lines[2] | default('') | trim) != 'regexp:/etc/postfix/sender_canonical'
+  notify: Reload postfix
+
+- name: Ensure postfix is enabled and running
+  ansible.builtin.systemd_service:
+    name: postfix
+    state: started
+    enabled: true

infrastructure/proxmox/ansible/roles/pve_network/defaults/main.yml

@@ -0,0 +1,30 @@
+---
+# Jumbo frames on the LAN.
+#
+# VERIFIED 2026-07-25: the unmanaged switch passes 8972-byte payloads with DF set
+# between all three nodes, so jumbo works despite there being no managed switch.
+#
+# WHY: (1) DRBD/LINSTOR replicates every write across this single 1G link —
+# larger frames mean fewer interrupts and better throughput on the storage path.
+# (2) It gives VXLAN room for its ~50-byte overhead, so SDN guests keep a normal
+# 1500 MTU instead of being cut to 1450 (retro OSes handle PMTUD badly).
+#
+# SAFETY on a mixed-MTU segment (router/DC/laptop remain 1500): TCP exchanges MSS
+# in the SYN, so each side sends no more than the other advertised — verified,
+# 1472B pings to all three still pass. The residual risk is only large UDP to a
+# 1500-MTU host; there is none here (corosync uses its own netmtu 1500, DNS is
+# small, NFS is TCP).
+pve_network_mtu: 9000
+
+# Make vmbr0 VLAN-aware so PVE SDN "vlan" zones can hang tagged VNets off it.
+#
+# VERIFIED on this LAN 2026-07-25: the unmanaged switch forwards 802.1Q-tagged
+# frames untouched (tested VLAN 100 between nodes at both 1472B and 8972B). Dumb
+# switches forward on MAC only — the tag is opaque payload to them.
+#
+# So VLAN zones beat VXLAN here: native forwarding, no encapsulation overhead, no
+# MTU maths. Trade-off: a dumb switch enforces no isolation, so VLANs give
+# SEGMENTATION, not security — anything on the LAN could inject tagged frames.
+# Fine for lab networks; do not treat a VLAN here as a security boundary.
+pve_network_vlan_aware: true
+pve_network_bridge_vids: "2-4094"

infrastructure/proxmox/ansible/roles/pve_network/tasks/main.yml

@@ -0,0 +1,66 @@
+---
+- name: Find the bridge port behind vmbr0
+  # Differs per node (pve1 eno1, pve2/pve3 enp1s0f0), so detect rather than hardcode.
+  ansible.builtin.shell:
+    cmd: "ip -br link show master vmbr0 | awk '{print $1}' | head -1"
+  register: _phys
+  changed_when: false
+  check_mode: false
+
+- name: Set MTU persistently on the physical port
+  ansible.builtin.lineinfile:
+    path: /etc/network/interfaces
+    insertafter: "^iface {{ _phys.stdout | trim }} inet manual"
+    line: "\tmtu {{ pve_network_mtu }}"
+    regexp: "^\tmtu "
+    firstmatch: true
+    backup: true
+  register: _phys_mtu
+
+- name: Set MTU persistently on vmbr0
+  ansible.builtin.blockinfile:
+    path: /etc/network/interfaces
+    insertafter: "^\tbridge-fd 0"
+    marker: "#{mark} ANSIBLE pve_network mtu"
+    block: "\tmtu {{ pve_network_mtu }}"
+  register: _br_mtu
+
+- name: Make vmbr0 VLAN-aware
+  ansible.builtin.blockinfile:
+    path: /etc/network/interfaces
+    insertafter: "^\tbridge-fd 0"
+    marker: "#{mark} ANSIBLE pve_network vlan-aware"
+    # Double-quoted so \t is a YAML ESCAPE: a literal tab cannot be used
+    # for indentation inside a block scalar -- YAML rejects tabs outright.
+    block: "\tbridge-vlan-aware yes\n\tbridge-vids {{ pve_network_bridge_vids }}"
+  when: pve_network_vlan_aware | bool
+  register: _vlan_aware
+
+- name: Reload networking if the bridge definition changed
+  # ifreload applies in place; it does NOT drop the management IP for a simple
+  # vlan-aware flip, but this is still done serially (site.yml runs serial:1) so
+  # a mistake cannot take all three nodes at once.
+  ansible.builtin.command:
+    cmd: ifreload -a
+  when: _vlan_aware is changed
+  changed_when: true
+
+- name: Apply at runtime too (no reboot needed)
+  ansible.builtin.shell:
+    cmd: |
+      ip link set {{ _phys.stdout | trim }} mtu {{ pve_network_mtu }}
+      ip link set vmbr0 mtu {{ pve_network_mtu }}
+  when: _phys_mtu is changed or _br_mtu is changed
+  changed_when: true
+
+- name: Verify the MTU is live
+  ansible.builtin.shell:
+    cmd: "ip link show vmbr0 | grep -o 'mtu [0-9]*'"
+  register: _mtu_now
+  changed_when: false
+
+- name: Assert it took
+  ansible.builtin.assert:
+    that: "pve_network_mtu | string in _mtu_now.stdout"
+    fail_msg: "vmbr0 MTU is {{ _mtu_now.stdout }}, expected {{ pve_network_mtu }}"
+    quiet: true

infrastructure/proxmox/ansible/roles/pve_nfs/defaults/main.yml

@@ -0,0 +1,18 @@
+---
+# NFS storage from the laptop ("core" node, 192.168.10.127), which is
+# deliberately NOT a cluster member — it holds the ZFS pool, serves NFS, and runs
+# netboot.xyz + k3s. See the project memory.
+#
+# This carries ISOs, container templates, backups and snippets. VM/CT DISKS live
+# on LINSTOR (pve-rg / pve-rg-hdd), NOT here: an NFS-backed disk would make every
+# guest depend on the laptop being up, which is exactly the coupling the
+# hyperconverged storage exists to avoid.
+pve_nfs_storage_id: laptop
+pve_nfs_server: 192.168.10.127
+pve_nfs_export: /mnt/pool/proxmox
+# Matches the directories that already exist in the export (dump/template/
+# snippets/import). No `images`/`rootdir` — see above.
+pve_nfs_content: "backup,iso,vztmpl,snippets,import"
+# The export is published by ZFS `sharenfs` (see /etc/exports.d/zfs.exports on
+# the laptop), not /etc/exports — change it with `zfs set sharenfs=...`.
+pve_nfs_options: ""

infrastructure/proxmox/ansible/roles/pve_nfs/tasks/main.yml

@@ -0,0 +1,47 @@
+---
+# Storage config lives in /etc/pve/storage.cfg, which is replicated cluster-wide,
+# so this runs once against a single node.
+
+- name: Check whether the storage is already defined
+  ansible.builtin.command:
+    cmd: "pvesm status --storage {{ pve_nfs_storage_id }}"
+  register: _nfs_exists
+  changed_when: false
+  failed_when: false
+  check_mode: false
+  run_once: true
+
+- name: Add the NFS storage
+  ansible.builtin.command:
+    cmd: >-
+      pvesm add nfs {{ pve_nfs_storage_id }}
+      --server {{ pve_nfs_server }}
+      --export {{ pve_nfs_export }}
+      --content {{ pve_nfs_content }}
+      {% if pve_nfs_options %}--options {{ pve_nfs_options }}{% endif %}
+  when: _nfs_exists.rc != 0
+  run_once: true
+
+- name: Wait for it to come active
+  # NFS storage is only usable once every node has mounted it; PVE mounts lazily.
+  ansible.builtin.command:
+    cmd: "pvesm status --storage {{ pve_nfs_storage_id }}"
+  register: _nfs_status
+  until: "'active' in (_nfs_status.stdout | default(''))"
+  retries: 12
+  delay: 5
+  changed_when: false
+  run_once: true
+
+- name: Verify every node can actually see it
+  # A cluster-wide storage entry that only ONE node can mount is a silent trap:
+  # migrations and backups fail on the others. Check per-node, not just once.
+  ansible.builtin.command:
+    cmd: "pvesm status --storage {{ pve_nfs_storage_id }}"
+  register: _nfs_node
+  changed_when: false
+  failed_when: "'active' not in (_nfs_node.stdout | default(''))"
+
+- name: Report
+  ansible.builtin.debug:
+    msg: "{{ inventory_hostname }}: {{ _nfs_node.stdout_lines | select('search', pve_nfs_storage_id) | list }}"

infrastructure/proxmox/ansible/roles/pve_post_install/defaults/main.yml

@@ -0,0 +1,4 @@
+---
+# Path to the web-UI JS carrying the subscription check. Kept here (not inline)
+# because it appears in both the patch task and the apt hook.
+_pve_nag_file: /usr/share/javascript/proxmox-widget-toolkit/proxmoxlib.js

infrastructure/proxmox/ansible/roles/pve_post_install/handlers/main.yml

@@ -0,0 +1,15 @@
+---
+- name: Refresh apt cache
+  ansible.builtin.apt:
+    update_cache: true
+  register: _h_apt
+  retries: 3
+  delay: 15
+  until: _h_apt is succeeded
+
+- name: Restart pveproxy
+  # Serves the web UI; picks up the patched proxmoxlib.js. Does NOT interrupt
+  # running guests.
+  ansible.builtin.systemd_service:
+    name: pveproxy
+    state: restarted

infrastructure/proxmox/ansible/roles/pve_post_install/tasks/main.yml

@@ -0,0 +1,174 @@
+---
+# Declarative equivalent of the community "post-pve-install.sh" script.
+# Written as a role rather than piping curl into bash as root, so it is
+# idempotent, reviewable, and survives a node rebuild.
+
+- name: Confirm we are actually on a Proxmox node
+  ansible.builtin.stat:
+    path: /usr/bin/pveversion
+  register: _pveversion
+  changed_when: false
+
+- name: Fail early on a non-PVE host
+  ansible.builtin.fail:
+    msg: "{{ inventory_hostname }} has no /usr/bin/pveversion -- this role only targets PVE nodes."
+  when: not _pveversion.stat.exists
+
+- name: Force apt to use IPv4
+  # MUST come before any apt operation. These nodes have NO IPv6 egress (no
+  # global v6 address, no default v6 route) but DNS still returns AAAA records,
+  # so apt tries a dead IPv6 path first and stalls per-mirror. This is what hung
+  # `apt update` against packages.linbit.com (resolved to 2a01:4f8:1c1c:6ab9::1).
+  # Complements the gai.conf precedence set by pve_dns.
+  ansible.builtin.copy:
+    dest: /etc/apt/apt.conf.d/99force-ipv4
+    mode: "0644"
+    content: |
+      // Managed by Ansible (role pve_post_install). Nodes have no IPv6 egress.
+      Acquire::ForceIPv4 "true";
+
+# ── repositories ────────────────────────────────────────────────────────
+# PVE 9 / Debian 13 uses deb822 (.sources) files, NOT the old one-line .list
+# format, so these use deb822_repository rather than apt_repository.
+#
+# NOTE the FQCN: deb822_repository has been promoted INTO ansible-core, so it is
+# `ansible.builtin.deb822_repository`. It is NOT in community.general (which
+# ships only apt_repo, for openSUSE) -- reaching for the community.general name
+# fails with "couldn't resolve module/action".
+
+- name: Disable the pve-enterprise repository
+  # 401s on every apt update without a subscription.
+  ansible.builtin.deb822_repository:
+    name: pve-enterprise
+    types: [deb]
+    uris: https://enterprise.proxmox.com/debian/pve
+    suites: ["{{ ansible_facts['distribution_release'] | default(pve_suite) }}"]
+    components: [pve-enterprise]
+    signed_by: /usr/share/keyrings/proxmox-archive-keyring.gpg
+    enabled: false
+    state: present
+  when: pve_disable_enterprise_repo
+  notify: Refresh apt cache
+
+- name: Disable the enterprise Ceph repository
+  # This cluster uses LINSTOR/DRBD, not Ceph -- Ceph on HDD OSDs was tried and
+  # was far too slow. Leaving this enabled only produces 401s.
+  ansible.builtin.deb822_repository:
+    name: ceph
+    types: [deb]
+    uris: https://enterprise.proxmox.com/debian/ceph-squid
+    suites: ["{{ ansible_facts['distribution_release'] | default(pve_suite) }}"]
+    components: [enterprise]
+    signed_by: /usr/share/keyrings/proxmox-archive-keyring.gpg
+    enabled: false
+    state: present
+  when: pve_disable_ceph_repo
+  notify: Refresh apt cache
+
+- name: Enable the pve-no-subscription repository
+  ansible.builtin.deb822_repository:
+    name: pve-no-subscription
+    types: [deb]
+    uris: http://download.proxmox.com/debian/pve
+    suites: ["{{ ansible_facts['distribution_release'] | default(pve_suite) }}"]
+    components: [pve-no-subscription]
+    signed_by: /usr/share/keyrings/proxmox-archive-keyring.gpg
+    enabled: true
+    state: present
+  when: pve_enable_no_subscription_repo
+  notify: Refresh apt cache
+
+- name: Apply repository changes now
+  ansible.builtin.meta: flush_handlers
+
+# ── subscription nag ────────────────────────────────────────────────────
+# The web UI shows a "No valid subscription" modal on every login. The check is
+# a single JS expression; forcing it false removes the dialog.
+
+- name: Back up proxmoxlib.js before patching
+  ansible.builtin.copy:
+    src: "{{ _pve_nag_file }}"
+    dest: "{{ _pve_nag_file }}.orig"
+    remote_src: true
+    force: false          # never clobber an existing pristine backup
+    mode: "0644"
+  when: pve_remove_subscription_nag
+
+- name: Remove the subscription nag dialog
+  # In PVE 9 the expression is `res.data.status.toLowerCase() !== 'active'`.
+  # The widely-copied sed matches only `data.status...` and so leaves `res.false`
+  # behind -- broken-looking JS that happens to work only because `res.false`
+  # evaluates to undefined (falsy). Match the optional `res.` prefix too so the
+  # result is a clean `false` instead of relying on that accident.
+  ansible.builtin.replace:
+    path: "{{ _pve_nag_file }}"
+    regexp: "(?:res\\.)?data\\.status\\.toLowerCase\\(\\) !== 'active'"
+    replace: "false"
+  when: pve_remove_subscription_nag
+  notify: Restart pveproxy
+
+- name: Install the nag re-patch helper script
+  # The sed lives in a script, NOT inlined in the apt hook: apt.conf's own
+  # quoting cannot carry a regex full of backslashes and quotes, and an invalid
+  # file there breaks EVERY apt invocation ("Extra junk after value") -- which
+  # is a much worse failure than the nag itself.
+  ansible.builtin.copy:
+    dest: /usr/local/sbin/pve-remove-nag.sh
+    mode: "0755"
+    content: |
+      #!/bin/sh
+      # Managed by Ansible (services/proxmox/ansible, role pve_post_install).
+      # Re-strips the "No valid subscription" dialog, which a
+      # proxmox-widget-toolkit upgrade restores.
+      # The (res\.)? prefix matters: without it this leaves `res.false`.
+      F={{ _pve_nag_file }}
+      [ -s "$F" ] || exit 0
+      sed -i -E "s/(res\.)?data\.status\.toLowerCase\(\) !== 'active'/false/g" "$F"
+      exit 0
+  when: pve_remove_subscription_nag
+
+- name: Re-apply the nag patch after any proxmox-widget-toolkit upgrade
+  # An apt upgrade reinstates the original file, so without this hook the nag
+  # silently returns. This is what makes the change durable rather than one-shot.
+  # `|| true` so a failure here can never block a package operation.
+  ansible.builtin.copy:
+    dest: /etc/apt/apt.conf.d/no-nag-script
+    mode: "0644"
+    content: |
+      // Managed by Ansible (services/proxmox/ansible, role pve_post_install).
+      DPkg::Post-Invoke { "/usr/local/sbin/pve-remove-nag.sh || true"; };
+  when: pve_remove_subscription_nag
+
+# ── updates ─────────────────────────────────────────────────────────────
+
+- name: Update the apt cache
+  ansible.builtin.apt:
+    update_cache: true
+    cache_valid_time: 0
+  register: _apt_update
+  # The home uplink drops out regularly (observed mid-session), so a single
+  # transient failure must not abort the play.
+  retries: 3
+  delay: 15
+  until: _apt_update is succeeded
+
+- name: Perform a full dist-upgrade
+  ansible.builtin.apt:
+    upgrade: dist
+    autoremove: true
+  register: _apt_upgrade
+  retries: 2
+  delay: 30
+  until: _apt_upgrade is succeeded
+  when: pve_dist_upgrade
+
+- name: Report whether a reboot is required
+  ansible.builtin.stat:
+    path: /var/run/reboot-required
+  register: _reboot_required
+  changed_when: false
+
+- name: Show reboot notice
+  ansible.builtin.debug:
+    msg: "{{ inventory_hostname }} requires a reboot (kernel or core library updated)."
+  when: _reboot_required.stat.exists

infrastructure/proxmox/ansible/roles/pve_sdn/defaults/main.yml

@@ -0,0 +1,29 @@
+---
+# PVE SDN — VLAN zone on vmbr0.
+#
+# WHY VLAN AND NOT VXLAN: the unmanaged switch forwards 802.1Q-tagged frames
+# untouched (dumb switches forward on MAC; the tag is opaque payload). VERIFIED
+# on this LAN 2026-07-25 — a VLAN-100 ping between nodes passed at both 1472B and
+# 8972B. So VLAN gives cross-node L2 with NO encapsulation, no overhead, and no
+# MTU arithmetic. VXLAN would work too (jumbo leaves room for its ~50 bytes) but
+# buys nothing here and costs CPU.
+#
+# ⚠️ A dumb switch enforces nothing, so these VLANs are SEGMENTATION, not
+# security: anything on the LAN could inject tagged frames. Do not treat a VNet
+# here as an isolation boundary for anything that matters.
+#
+# NOTE: a VLAN zone is pure L2 — no gateway, no SNAT, no internet. That is the
+# point for lab/retro-OS networks. If a VNet later needs routing or internet
+# egress, that is an EVPN zone (frr is already installed) or a router VM.
+pve_sdn_zone: lab
+pve_sdn_bridge: vmbr0
+# Guests keep a standard 1500 MTU; the 9000 underlay carries it comfortably.
+pve_sdn_mtu: 1500
+
+pve_sdn_vnets:
+  - name: labnet          # general isolated lab L2
+    tag: 100
+    alias: "Isolated lab network (no gateway)"
+  - name: retronet        # retro OSes: keep ancient stacks off the real LAN
+    tag: 110
+    alias: "Retro OS network (no gateway, no internet)"

infrastructure/proxmox/ansible/roles/pve_sdn/tasks/main.yml

@@ -0,0 +1,60 @@
+---
+# SDN config is cluster-wide (/etc/pve/sdn/), so everything runs once.
+
+- name: Read existing zones
+  ansible.builtin.command:
+    cmd: "pvesh get /cluster/sdn/zones --output-format json"
+  register: _zones
+  changed_when: false
+  check_mode: false
+  run_once: true
+
+- name: Create the VLAN zone
+  ansible.builtin.command:
+    cmd: >-
+      pvesh create /cluster/sdn/zones --zone {{ pve_sdn_zone }} --type vlan
+      --bridge {{ pve_sdn_bridge }} --mtu {{ pve_sdn_mtu }}
+      --nodes {{ groups['pve'] | join(',') }}
+  when: pve_sdn_zone not in (_zones.stdout | from_json | map(attribute='zone') | list)
+  run_once: true
+
+- name: Read existing vnets
+  ansible.builtin.command:
+    cmd: "pvesh get /cluster/sdn/vnets --output-format json"
+  register: _vnets
+  changed_when: false
+  check_mode: false
+  run_once: true
+
+- name: Create the VNets
+  ansible.builtin.command:
+    cmd: >-
+      pvesh create /cluster/sdn/vnets --vnet {{ item.name }}
+      --zone {{ pve_sdn_zone }} --tag {{ item.tag }}
+      --alias '{{ item.alias }}'
+  loop: "{{ pve_sdn_vnets }}"
+  loop_control:
+    label: "{{ item.name }} (vlan {{ item.tag }})"
+  when: item.name not in (_vnets.stdout | from_json | map(attribute='vnet') | list)
+  run_once: true
+
+- name: Apply the SDN configuration
+  # SDN changes stay PENDING until applied; without this the VNet bridges are
+  # never actually created on the nodes.
+  ansible.builtin.command:
+    cmd: "pvesh set /cluster/sdn"
+  register: _apply
+  changed_when: true
+  run_once: true
+
+- name: Report
+  ansible.builtin.shell:
+    cmd: "pvesh get /cluster/sdn/vnets --output-format json | python3 -c \"import json,sys;[print('   ',v['vnet'],'vlan',v.get('tag'),'zone',v.get('zone')) for v in json.load(sys.stdin)]\""
+  register: _rep
+  changed_when: false
+  run_once: true
+
+- name: Show it
+  ansible.builtin.debug:
+    msg: "{{ _rep.stdout_lines }}"
+  run_once: true

infrastructure/proxmox/ansible/roles/pve_vm/defaults/main.yml

@@ -0,0 +1,77 @@
+---
+# Declarative-ish VM definitions. Uses `qm` over SSH rather than
+# community.general.proxmox_kvm: that module needs `proxmoxer` plus API-token
+# auth, while every other role here already drives pvesh/qm on the node. Staying
+# consistent beats adding a dependency for one role.
+#
+# Creation is guarded by `qm status <vmid>`, so re-runs never touch an existing
+# VM. Changing a definition here does NOT retro-fit a live VM — adjust it with
+# `qm set` deliberately, or destroy and recreate.
+
+pve_vm_node: pve1          # where new VMs are created
+
+# Anything a VM needs to be MANAGEABLE, installed by cloud-init on first boot —
+# before Ansible ever connects. qemu-guest-agent especially: without it PVE
+# cannot see guest IPs, do a clean shutdown, or freeze the fs for snapshots.
+pve_vm_base_packages:
+  - qemu-guest-agent
+  - openssh-server
+  - python3            # Ansible needs an interpreter
+pve_vm_ssh_key: "{{ lookup('file', '~/.ssh/id_ed25519.pub') }}"
+# PVE storage with `snippets` content enabled (see role pve_nfs).
+pve_vm_snippet_storage: laptop
+pve_vm_snippet_path: /mnt/pve/laptop/snippets
+
+pve_vms:
+  - vmid: 100
+    name: vyos-rtr
+    description: "VyOS router: OSPF peer to the NEC IX, gateway for the SDN VNets."
+    cores: 2
+    memory: 2048
+    disk: "pve-rg:8"
+    # Headless by design: UEFI + serial console, NO emulated VGA. VyOS's own
+    # kernel cmdline carries console=ttyS0, and the installer was answered with
+    # console=Serial, so `qm terminal 100` stays available out-of-band.
+    bios: ovmf
+    machine: q35
+    vga: serial0
+    serial: true
+    nets:
+      - { id: 0, bridge: vmbr0 }      # LAN / OSPF
+      - { id: 1, bridge: labnet }
+      - { id: 2, bridge: retronet }
+    onboot: 1
+    iso: "laptop:iso/vyos-2025.11-generic-amd64.iso"
+
+  - vmid: 101
+    name: retrolab
+    description: "86Box host. RDP in when you want it. net1 = retronet (no IP), for the emulated NICs."
+    # ⚠️ BELONGS ON pve2/pve3, NOT on the pve1 that `pve_vm_node` creates it on.
+    # 86Box is a single-threaded recompiler and pve1 is an i3-6100U (2c/4t,
+    # 2.3 GHz) that also carries vyos-rtr — emulation stuttered and the emulated
+    # Sound Blaster glitched. The Ryzen 5 PRO 2400GE nodes (4c/8t, 3.2 GHz) do
+    # not. Moved to pve3 on 2026-07-28. If this VM is ever recreated here,
+    # migrate it back off pve1 afterwards; `cpu: host` (kept, it is most of the
+    # win) makes that an OFFLINE migration, because pve1 is Intel and pve2/pve3
+    # are AMD.
+    cores: 4
+    memory: 4096
+    disk: "pve-rg:40"
+    bios: ovmf
+    machine: q35
+    vga: virtio             # a desktop, unlike the router
+    serial: true
+    # MACs are PINNED. Destroying and recreating a VM otherwise generates new
+    # ones, so its DHCP reservation stops matching and its address moves -- which
+    # is exactly what happened on the first rebuild.
+    nets:
+      - { id: 0, bridge: labnet, mac: "BC:24:11:68:A0:51" }   # DHCP-reserved to 10.60.0.10
+      - { id: 1, bridge: retronet, mac: "BC:24:11:9F:63:ED" } # no IP - handed to 86Box
+    onboot: 0
+    cloudinit: true
+    ciuser: panxiao81
+    # ⚠️ Do NOT use `qm importdisk` for this image. It reported success
+    # ("transferred 3.5 GiB", exit 0) but left ZEROS where the partition table
+    # belongs, producing a silently unbootable disk. Write it straight to the
+    # DRBD device instead (see tasks/image.yml).
+    image: "/mnt/pve/laptop/import/noble-server-cloudimg-amd64.img.raw"

infrastructure/proxmox/ansible/roles/pve_vm/tasks/image.yml

@@ -0,0 +1,67 @@
+---
+# Write a base image onto a freshly created VM disk.
+#
+# ⚠️ Deliberately NOT `qm importdisk`. On this LINSTOR/DRBD storage it reported
+# success ("transferred 3.5 GiB", exit 0) but left ZEROS where the partition
+# table belongs, producing a silently unbootable disk. `qemu-img convert`
+# straight to the DRBD device works and takes ~40s. (Verified 2026-07-25.)
+#
+# Also note: the source is named *.img.raw but is actually QCOW2 — always let
+# qemu detect the format rather than trusting the extension.
+
+- name: Resolve the DRBD device backing the VM disk
+  ansible.builtin.shell:
+    cmd: |
+      set -o pipefail
+      # scsi0: <storage>:<volume>,opts...  -> we want <volume>, i.e. field 3
+      vol=$(qm config {{ item.vmid }} | sed -n 's/^scsi0: *[^:]*:\([^,]*\).*/\1/p')
+      # PVE's volume is pm-xxxx_<vmid>, but the LINSTOR RESOURCE is just
+      # pm-xxxx -- strip the _<vmid> suffix or nothing ever matches.
+      res=${vol%_*}
+      linstor resource list-volumes 2>/dev/null \
+        | awk -v v="$res" -v n="$(hostname)" '$0 ~ v && $0 ~ n {print}' \
+        | grep -oE '/dev/drbd[0-9]+' | head -1
+    executable: /bin/bash
+  register: _drbd_dev
+  changed_when: false
+  delegate_to: "{{ pve_vm_node }}"
+
+- name: Fail if the device could not be resolved
+  ansible.builtin.fail:
+    msg: "Could not find the DRBD device for VM {{ item.vmid }} scsi0 — refusing to write an image blind."
+  delegate_to: "{{ pve_vm_node }}"
+  when: (_drbd_dev.stdout | trim) | length == 0
+
+- name: Check whether the disk already has a partition table
+  # Idempotency guard: never overwrite a disk that already looks installed.
+  ansible.builtin.shell:
+    cmd: |
+      python3 -c "
+      import sys
+      try:
+          d=open('{{ _drbd_dev.stdout | trim }}','rb').read(512)
+          sys.stdout.write('yes' if d[510:512]==b'\x55\xaa' else 'no')
+      except Exception:
+          sys.stdout.write('unreadable')"
+    executable: /bin/bash
+  register: _has_mbr
+  changed_when: false
+  delegate_to: "{{ pve_vm_node }}"
+
+- name: Write the base image
+  # DRBD must be Primary to accept writes; reading a Secondary returns ZEROS,
+  # which is its own excellent way to misdiagnose an empty disk.
+  ansible.builtin.shell:
+    cmd: |
+      set -e
+      drbdadm primary {{ _res }} 2>/dev/null || true
+      qemu-img convert -O raw "{{ item.image }}" "{{ _dev }}"
+      sync
+      drbdadm secondary {{ _res }} 2>/dev/null || true
+    executable: /bin/bash
+  vars:
+    _dev: "{{ _drbd_dev.stdout | trim }}"
+    _res: "{{ _drbd_dev.stdout | trim | regex_replace('.*drbd', '') }}"
+  when: _has_mbr.stdout is not search('yes')
+  changed_when: true
+  delegate_to: "{{ pve_vm_node }}"

infrastructure/proxmox/ansible/roles/pve_vm/tasks/main.yml

@@ -0,0 +1,89 @@
+---
+# Create VMs that do not exist yet. Idempotent by existence check only — this
+# role does NOT reconcile the config of a live VM (see defaults for why).
+
+- name: Check which VMs already exist
+  ansible.builtin.command:
+    cmd: "qm status {{ item.vmid }}"
+  loop: "{{ pve_vms }}"
+  loop_control:
+    label: "{{ item.vmid }} {{ item.name }}"
+  register: _vm_exists
+  changed_when: false
+  failed_when: false
+  check_mode: false
+  run_once: true
+  delegate_to: "{{ pve_vm_node }}"
+
+- name: Render cloud-init user-data for VMs that use it
+  # Written to the NFS `snippets` dir so every node can read it. Rendered even
+  # for existing VMs so the file stays in sync with the definition — but see the
+  # template header: cloud-init only APPLIES it on first boot.
+  ansible.builtin.template:
+    src: cloudinit-user.yml.j2
+    dest: "{{ pve_vm_snippet_path }}/{{ item.name }}-user.yml"
+    mode: "0644"
+  loop: "{{ pve_vms | selectattr('cloudinit', 'defined') | selectattr('cloudinit') | list }}"
+  loop_control:
+    label: "{{ item.name }}"
+  vars:
+    vm: "{{ item }}"
+  run_once: true
+  delegate_to: "{{ pve_vm_node }}"
+
+
+- name: Create missing VMs
+  ansible.builtin.command:
+    cmd: >-
+      qm create {{ item.item.vmid }}
+      --name {{ item.item.name }}
+      --description '{{ item.item.description }}'
+      --machine {{ item.item.machine | default('q35') }}
+      --bios {{ item.item.bios | default('ovmf') }}
+      --memory {{ item.item.memory }} --cores {{ item.item.cores }} --cpu host
+      --scsihw virtio-scsi-single
+      --scsi0 {{ item.item.disk }},discard=on,ssd=1
+      --efidisk0 {{ item.item.disk.split(':')[0] }}:1,efitype=4m,pre-enrolled-keys=0
+      --vga {{ item.item.vga | default('virtio') }}
+      {% if item.item.serial | default(false) %}--serial0 socket{% endif %}
+      {% for n in item.item.nets %}--net{{ n.id }} virtio{% if n.mac is defined %}={{ n.mac }}{% endif %},bridge={{ n.bridge }} {% endfor %}
+      {% if item.item.iso is defined %}--ide2 {{ item.item.iso }},media=cdrom --boot order=ide2;scsi0{% else %}--boot order=scsi0{% endif %}
+      {% if item.item.cloudinit | default(false) %}--ide2 {{ item.item.disk.split(':')[0] }}:cloudinit --ipconfig0 ip=dhcp --ciuser {{ item.item.ciuser }}{% endif %}
+      --agent enabled=1 --onboot {{ item.item.onboot | default(0) }} --ostype l26
+  loop: "{{ _vm_exists.results }}"
+  loop_control:
+    label: "{{ item.item.vmid }} {{ item.item.name }}"
+  when: item.rc != 0
+  run_once: true
+  delegate_to: "{{ pve_vm_node }}"
+
+- name: Point those VMs at their user-data
+  ansible.builtin.command:
+    cmd: >-
+      qm set {{ item.vmid }} --cicustom user={{ pve_vm_snippet_storage }}:snippets/{{ item.name }}-user.yml
+  loop: "{{ pve_vms | selectattr('cloudinit', 'defined') | selectattr('cloudinit') | list }}"
+  loop_control:
+    label: "{{ item.name }}"
+  register: _cicustom
+  changed_when: "'update VM' in (_cicustom.stdout | default(''))"
+  run_once: true
+  delegate_to: "{{ pve_vm_node }}"
+
+- name: Write base images onto newly created disks
+  ansible.builtin.include_tasks: image.yml
+  # ALL image-backed VMs, not just newly created ones: image.yml decides by
+  # checking for an MBR on the disk. Gating on "was just created" meant a failed
+  # image write could never be repaired by re-running.
+  loop: "{{ pve_vms | selectattr('image', 'defined') | list }}"
+  loop_control:
+    label: "{{ item.name }}"
+  run_once: true
+
+- name: Report
+  ansible.builtin.debug:
+    msg: >-
+      {{ _vm_exists.results
+         | map(attribute='item')
+         | zip(_vm_exists.results | map(attribute='rc'))
+         | map('join', ' rc=') | list }}
+  run_once: true

infrastructure/proxmox/ansible/roles/pve_vm/templates/cloudinit-user.yml.j2

@@ -0,0 +1,45 @@
+#cloud-config
+# {{ ansible_managed }}
+# Full cloud-init user-data for {{ vm.name }} (vmid {{ vm.vmid }}).
+#
+# PVE's built-in cloud-init options only cover user/password/keys/network. That
+# left qemu-guest-agent uninstalled and bootstrap state fragile. Anything a VM
+# needs to be MANAGEABLE should happen here, on first boot, before Ansible ever
+# connects.
+#
+# NOTE: cloud-init runs `packages`/`runcmd` ONCE per instance. Adding entries
+# here does not retro-fit an already-provisioned VM — that needs Ansible (or a
+# `cloud-init clean` + reboot).
+
+hostname: {{ vm.name }}
+manage_etc_hosts: true
+
+users:
+  - name: {{ vm.ciuser }}
+    groups: [adm, sudo]
+    shell: /bin/bash
+    sudo: "ALL=(ALL) NOPASSWD:ALL"
+    lock_passwd: false
+    ssh_authorized_keys:
+      - "{{ pve_vm_ssh_key }}"
+
+package_update: true
+# Deliberately NO dist-upgrade at first boot. cloud-init has no retry, and on
+# this WAN `apt-get dist-upgrade` exits 100 the moment the link blips — which
+# aborts the whole package module, so qemu-guest-agent never installs and the VM
+# comes up unmanageable. Keep first boot minimal; Ansible does the rest WITH
+# retries. (Observed 2026-07-25.)
+package_upgrade: false
+packages:
+{% for p in vm.cloudinit_packages | default(pve_vm_base_packages) %}
+  - {{ p }}
+{% endfor %}
+
+runcmd:
+  # qemu-guest-agent gives PVE the guest's IPs, clean shutdown, and fsfreeze for
+  # snapshots. Without it `qm agent` fails and PVE cannot see inside the guest.
+  - [systemctl, enable, --now, qemu-guest-agent]
+  - [systemctl, enable, --now, ssh]
+
+# Keep the host keys stable across reboots so known_hosts does not churn.
+ssh_deletekeys: false

infrastructure/proxmox/ansible/roles/retro_86box/defaults/main.yml

@@ -0,0 +1,76 @@
+---
+# 86Box + Avalonia86, installed as AppImages.
+#
+# WHY AppImage: 86Box is not in the Ubuntu archive. The alternatives were a
+# Flatpak (adds a ~1-2GB runtime, and its sandbox complicates letting the
+# manager pass per-VM config paths) or building from source (large Qt6/SDL2
+# toolchain, manual updates). Upstream publishes official Linux AppImages for
+# BOTH 86Box and Avalonia86, so this is a single file each, no runtime, no
+# sandbox, and the manager can point straight at the binary.
+
+retro_86box_dir: /opt/86box
+
+# Pinned. Bump deliberately — a silent upstream change to the emulator is not
+# something you want arriving with an unrelated play run.
+retro_86box_version: "6.0"
+retro_86box_build: "b9001"
+retro_86box_url: "https://github.com/86Box/86Box/releases/download/v{{ retro_86box_version }}/86Box-Linux-x86_64-{{ retro_86box_build }}.AppImage"
+
+retro_86box_mgr_version: "1.5.1"
+retro_86box_mgr_url: "https://github.com/notBald/Avalonia86/releases/download/v{{ retro_86box_mgr_version }}/Avalonia-86-for-Linux-x64-{{ retro_86box_mgr_version }}.AppImage"
+
+# ROMs are NOT bundled with 86Box (licensing). Without them no machine will
+# boot — the emulator starts and then cannot find a system BIOS.
+retro_86box_roms_url: "https://github.com/86Box/roms/archive/refs/tags/v{{ retro_86box_version }}.tar.gz"
+
+# AppImages need FUSE. Ubuntu 24.04 does not ship libfuse2 by default; without
+# it an AppImage fails with "dlopen(): error loading libfuse.so.2".
+# patchelf + setcap are needed to make TAP networking work (see below).
+retro_86box_packages:
+  - libfuse2t64
+  - patchelf
+  - libcap2-bin
+
+# ---------------------------------------------------------------------------
+# TAP networking: why the AppImage is extracted instead of run as-is
+# ---------------------------------------------------------------------------
+# 86Box's TAP backend creates the tap device itself and enslaves it to a
+# bridge (verified in src/network/net_tap.c):
+#     ifr.ifr_flags = IFF_TAP | IFF_NO_PI;  ioctl(fd, TUNSETIFF, &ifr);
+#     ioctl_or_fail(sock, SIOCBRADDBR, &ifr_bridge);   /* create bridge  */
+#     ioctl_or_fail(sock, SIOCBRADDIF, &ifr_bridge);   /* enslave the tap */
+# so it needs CAP_NET_ADMIN (upstream's own error text says as much). There is
+# no "pre-create the tap and hand it over" path -- the setting is a BRIDGE
+# name, not a tap name.
+#
+# File capabilities cannot be applied to the AppImage: the real ELF lives
+# inside a read-only squashfs mounted through FUSE with nosuid, where the
+# kernel ignores file caps entirely. So the AppImage is extracted once and the
+# extracted binary carries the caps.
+#
+# That in turn breaks the AppImage's own library loading, because a file with
+# capabilities runs in secure-execution mode (AT_SECURE=1) and glibc drops
+# LD_LIBRARY_PATH -- which is exactly how AppRun points the binary at its
+# bundled Qt. Two patches fix it, and BOTH are load-bearing:
+#   1. PT_INTERP is the RELATIVE path "lib64/ld-linux-x86-64.so.2"; it must be
+#      made absolute or exec fails with "required file not found".
+#   2. The rpath must be DT_RPATH (--force-rpath), NOT the modern DT_RUNPATH.
+#      RUNPATH is not inherited by transitive dependencies, and 8 of the
+#      bundled libraries (libicuuc, libpcre2-16, libFLAC, ...) are needed by
+#      Qt itself rather than by 86Box, so with RUNPATH they are "not found".
+retro_86box_app_dir: "{{ retro_86box_dir }}/app"
+retro_86box_bin: "{{ retro_86box_app_dir }}/usr/local/bin/86Box"
+retro_86box_wrapper: /usr/local/bin/86box
+retro_86box_interp: /lib64/ld-linux-x86-64.so.2
+retro_86box_caps: "CAP_NET_RAW,CAP_NET_ADMIN=eip"
+
+# Taken from the AppImage's own AppRun.env (APPDIR_LIBRARY_PATH), made
+# absolute. Keep in sync if a future 86Box release changes its bundle layout.
+retro_86box_rpath_dirs:
+  - "{{ retro_86box_app_dir }}/lib/x86_64-linux-gnu"
+  - "{{ retro_86box_app_dir }}/usr/lib"
+  - "{{ retro_86box_app_dir }}/usr/lib/x86_64-linux-gnu"
+  - "{{ retro_86box_app_dir }}/lib/x86_64"
+
+# The bridge 86Box should attach its taps to. Must match tap_bridge_name.
+retro_86box_tap_bridge: br-retro

infrastructure/proxmox/ansible/roles/retro_86box/tasks/main.yml

@@ -0,0 +1,127 @@
+---
+- name: Install AppImage runtime dependency
+  ansible.builtin.apt:
+    name: "{{ retro_86box_packages }}"
+    state: present
+    update_cache: true
+    cache_valid_time: 3600
+  register: _fuse
+  retries: 3
+  delay: 15
+  until: _fuse is succeeded
+
+- name: Create the 86Box directory
+  ansible.builtin.file:
+    path: "{{ retro_86box_dir }}"
+    state: directory
+    mode: "0755"
+
+- name: Download 86Box and Avalonia86
+  # The WAN drops out; get_url resumes rather than restarting from zero.
+  ansible.builtin.get_url:
+    url: "{{ item.url }}"
+    dest: "{{ retro_86box_dir }}/{{ item.name }}"
+    mode: "0755"
+  loop:
+    - { name: "86Box.AppImage", url: "{{ retro_86box_url }}" }
+    - { name: "Avalonia86.AppImage", url: "{{ retro_86box_mgr_url }}" }
+  loop_control:
+    label: "{{ item.name }}"
+  register: _dl
+  retries: 3
+  delay: 20
+  until: _dl is succeeded
+
+- name: Check whether ROMs are already extracted
+  ansible.builtin.stat:
+    path: "{{ retro_86box_dir }}/roms/machines"
+  register: _roms
+
+- name: Create the roms directory
+  ansible.builtin.file:
+    path: "{{ retro_86box_dir }}/roms"
+    state: directory
+    mode: "0755"
+
+- name: Download and extract the ROM set
+  # Extract INTO roms/, stripping the tarball's own roms-<ver>/ wrapper.
+  # Extracting to the parent with --strip-components=1 dumps machines/, floppy/,
+  # hdd/ ... loose next to the binaries, which is NOT the layout 86Box and
+  # Avalonia86 expect (they want everything under roms/).
+  ansible.builtin.unarchive:
+    src: "{{ retro_86box_roms_url }}"
+    dest: "{{ retro_86box_dir }}/roms"
+    remote_src: true
+    extra_opts: [--strip-components=1]
+    creates: "{{ retro_86box_dir }}/roms/machines"
+  when: not _roms.stat.exists
+  register: _romdl
+  retries: 3
+  delay: 20
+  until: _romdl is succeeded
+
+- name: Extract the AppImage and grant it CAP_NET_ADMIN for TAP networking
+  ansible.builtin.include_tasks: privileged.yml
+
+- name: Add desktop launchers
+  ansible.builtin.copy:
+    dest: "/usr/share/applications/{{ item.file }}"
+    mode: "0644"
+    content: |
+      [Desktop Entry]
+      Type=Application
+      Name={{ item.name }}
+      Exec={{ item.bin }}
+      Icon=computer
+      Categories=System;Emulator;
+      Terminal=false
+  loop:
+    - file: "avalonia86.desktop"
+      name: "Avalonia86 (86Box manager)"
+      bin: "{{ retro_86box_dir }}/Avalonia86.AppImage"
+    # NOT the .AppImage: only the extracted binary carries CAP_NET_ADMIN, so
+    # launching the AppImage would silently lose TAP networking.
+    - file: "86box.desktop"
+      name: "86Box"
+      bin: "{{ retro_86box_wrapper }}"
+  loop_control:
+    label: "{{ item.name }}"
+
+- name: Verify 86Box actually runs
+  # Runs the CAPABILITY-BEARING binary, which is the one that must work.
+  # QT_QPA_PLATFORM=offscreen is REQUIRED: 86Box is a Qt GUI app and still
+  # initialises a display for --help, so it dies headless with
+  # "could not connect to display / Could not load the Qt platform plugin xcb".
+  # This also proves the patchelf work: under secure-execution mode (which
+  # file capabilities trigger) glibc drops LD_LIBRARY_PATH, so if the RPATH or
+  # the interpreter were wrong this step fails with "required file not found".
+  ansible.builtin.command:
+    cmd: "{{ retro_86box_wrapper }} --help"
+  environment:
+    QT_QPA_PLATFORM: offscreen
+  register: _ver
+  changed_when: false
+  failed_when: "'86box' not in (_ver.stdout + _ver.stderr) | lower and 'usage' not in (_ver.stdout + _ver.stderr) | lower"
+
+- name: Read back the capabilities actually on the binary
+  ansible.builtin.command:
+    cmd: "getcap {{ retro_86box_bin }}"
+  register: _capcheck
+  changed_when: false
+
+- name: Verify the capabilities survived
+  ansible.builtin.assert:
+    that:
+      - "'cap_net_admin' in _capcheck.stdout"
+      - "'cap_net_raw' in _capcheck.stdout"
+    fail_msg: "86Box has no CAP_NET_ADMIN; TAP networking will fail to allocate a tap device"
+    success_msg: "{{ _capcheck.stdout | trim }}"
+
+- name: Report
+  ansible.builtin.debug:
+    msg:
+      - "86Box:  {{ retro_86box_wrapper }} -> {{ retro_86box_bin }}"
+      - "caps:   {{ _capcheck.stdout | trim }}"
+      - "roms:   {{ retro_86box_dir }}/roms"
+      - "ISOs:   /mnt/iso (read-only)"
+      - "TAP:    set each NIC to 'TAP' with bridge '{{ retro_86box_tap_bridge }}'"

infrastructure/proxmox/ansible/roles/retro_86box/tasks/privileged.yml

@@ -0,0 +1,102 @@
+---
+# Extract the AppImage and give the real binary CAP_NET_ADMIN so TAP
+# networking works. See the long comment in defaults/main.yml for why every
+# step here is necessary.
+
+- name: Read the extracted build id
+  # The extracted tree is a derived artifact; it must be rebuilt whenever the
+  # pinned AppImage build changes, or a version bump would leave the old
+  # binary in place while the AppImage next to it says otherwise.
+  ansible.builtin.slurp:
+    src: "{{ retro_86box_app_dir }}/.build-id"
+  register: _bid
+  failed_when: false
+  changed_when: false
+
+- name: Decide whether the AppImage must be re-extracted
+  ansible.builtin.set_fact:
+    _86box_extract: >-
+      {{ _bid.content is not defined
+         or (_bid.content | b64decode | trim) != retro_86box_build }}
+
+- name: Extract the AppImage
+  # --appimage-extract always writes ./squashfs-root in the CWD and refuses to
+  # target a directory, hence the extract-then-move.
+  ansible.builtin.shell:
+    cmd: |
+      set -e
+      rm -rf "{{ retro_86box_app_dir }}" "{{ retro_86box_dir }}/squashfs-root"
+      cd "{{ retro_86box_dir }}"
+      ./86Box.AppImage --appimage-extract >/dev/null
+      mv squashfs-root "{{ retro_86box_app_dir }}"
+  when: _86box_extract
+  changed_when: true
+
+- name: Read the current ELF interpreter
+  ansible.builtin.command:
+    cmd: "patchelf --print-interpreter {{ retro_86box_bin }}"
+  register: _interp
+  changed_when: false
+
+- name: Make the ELF interpreter absolute
+  ansible.builtin.command:
+    cmd: "patchelf --set-interpreter {{ retro_86box_interp }} {{ retro_86box_bin }}"
+  when: _interp.stdout | trim != retro_86box_interp
+  changed_when: true
+
+- name: Read the current rpath
+  ansible.builtin.command:
+    cmd: "patchelf --print-rpath {{ retro_86box_bin }}"
+  register: _rpath
+  changed_when: false
+
+- name: Bake the bundled library directories in as DT_RPATH
+  ansible.builtin.command:
+    cmd: >-
+      patchelf --force-rpath --set-rpath
+      "{{ retro_86box_rpath_dirs | join(':') }}" {{ retro_86box_bin }}
+  when: _rpath.stdout | trim != (retro_86box_rpath_dirs | join(':'))
+  changed_when: true
+
+- name: Verify no library is left unresolved
+  # Must pass BEFORE setcap is worth doing -- a capability binary that cannot
+  # load its libraries fails with a misleading "required file not found".
+  ansible.builtin.shell:
+    cmd: "ldd {{ retro_86box_bin }} 2>&1 | grep -c 'not found' || true"
+  register: _missing
+  changed_when: false
+  failed_when: (_missing.stdout | trim | int) != 0
+
+- name: Read the current capabilities
+  ansible.builtin.command:
+    cmd: "getcap {{ retro_86box_bin }}"
+  register: _caps
+  changed_when: false
+
+- name: Grant CAP_NET_ADMIN and CAP_NET_RAW
+  # NOTE: must run AFTER patchelf. patchelf rewrites the file and drops the
+  # security.capability xattr, so setting caps first silently loses them.
+  ansible.builtin.command:
+    cmd: "setcap '{{ retro_86box_caps }}' {{ retro_86box_bin }}"
+  when: "'cap_net_admin' not in _caps.stdout or 'cap_net_raw' not in _caps.stdout"
+  changed_when: true
+
+- name: Record the extracted build id
+  ansible.builtin.copy:
+    content: "{{ retro_86box_build }}\n"
+    dest: "{{ retro_86box_app_dir }}/.build-id"
+    mode: "0644"
+
+- name: Install the launcher wrapper
+  ansible.builtin.copy:
+    dest: "{{ retro_86box_wrapper }}"
+    mode: "0755"
+    content: |
+      #!/bin/sh
+      # Managed by Ansible (roles/retro_86box).
+      #
+      # Runs the EXTRACTED 86Box, not the AppImage: only the extracted binary
+      # can carry CAP_NET_ADMIN, which 86Box needs to create its tap device
+      # and enslave it to {{ retro_86box_tap_bridge }}. Point Avalonia86 at
+      # this path, not at the .AppImage.
+      exec {{ retro_86box_bin }} "$@"

infrastructure/proxmox/ansible/roles/retro_desktop/defaults/main.yml

@@ -0,0 +1,42 @@
+---
+# Remote desktop for the 86Box host.
+#
+# RDP, not VNC — because 86Box needs SOUND and VNC carries no audio. xrdp
+# redirects audio; that is the whole reason for this choice.
+#
+# xrdp destroys a session only on explicit LOGOUT, not on disconnect, so
+# emulator instances survive closing the client. That is why 86Box is left as a
+# normal desktop app rather than forced into a systemd unit: several GUI
+# instances under a manager is a desktop, and pretending otherwise fights the
+# tooling for no benefit.
+
+retro_desktop_packages:
+  - xfce4                 # light, and xrdp-friendly
+  - xfce4-terminal
+  - dbus-x11
+  - xrdp
+  - xorgxrdp
+
+# Audio over RDP — the whole reason this box uses RDP rather than VNC.
+#
+# Ubuntu 24.04 is PipeWire-based, and it ships a PIPEWIRE-native xrdp module
+# (`pipewire-module-xrdp`), so the old pulseaudio-module-xrdp build dance is not
+# needed. It currently arrives as a transitive dependency of xrdp — listed here
+# EXPLICITLY so a future dependency change cannot silently kill sound.
+retro_desktop_audio: true
+retro_desktop_audio_packages:
+  - pipewire-module-xrdp   # provides /etc/xdg/autostart/pipewire-xrdp.desktop
+  - pipewire-pulse
+  - pulseaudio-utils       # paplay/pactl, for testing inside a session
+
+# Channels that must be enabled in xrdp.ini for audio to reach the client.
+retro_desktop_audio_channels: [rdpsnd, drdynvc]
+
+# What to rename xorgxrdp's pointer to, so 86Box can capture the mouse.
+#
+# This string is not cosmetic: it is matched CHARACTER FOR CHARACTER against
+# 86Box's allow-list of pointer devices that report absolute coordinates
+# (src/qt/xinput2_mouse.cpp, xinput2_get_xtest_pointer()). Change it and mouse
+# capture breaks again. The full reasoning is in tasks/main.yml.
+retro_desktop_xrdp_pointer_name: "TigerVNC pointer"
+retro_desktop_xrdp_xorg_conf: /etc/X11/xrdp/xorg.conf

infrastructure/proxmox/ansible/roles/retro_desktop/tasks/main.yml

@@ -0,0 +1,183 @@
+---
+- name: Install the desktop and xrdp
+  ansible.builtin.apt:
+    name: "{{ retro_desktop_packages }}"
+    state: present
+    update_cache: true
+    cache_valid_time: 3600
+  register: _pkgs
+  retries: 3
+  delay: 15
+  until: _pkgs is succeeded          # the WAN drops out; do not fail a long run on one blip
+
+- name: Install audio support
+  ansible.builtin.apt:
+    name: "{{ retro_desktop_audio_packages }}"
+    state: present
+  register: _apkgs
+  retries: 3
+  delay: 15
+  until: _apkgs is succeeded
+  when: retro_desktop_audio | bool
+
+- name: Use XFCE for xrdp sessions
+  ansible.builtin.copy:
+    dest: /etc/skel/.xsession
+    content: "xfce4-session\n"
+    mode: "0644"
+
+- name: Give the existing user an .xsession too
+  ansible.builtin.copy:
+    dest: "/home/{{ ansible_user }}/.xsession"
+    content: "xfce4-session\n"
+    owner: "{{ ansible_user }}"
+    group: "{{ ansible_user }}"
+    mode: "0644"
+
+# ⚠️ NO WINDOW MANAGER after login? It is a saved session, and it is
+# self-perpetuating. xfce4-session restores exactly the client list in
+# ~/.cache/sessions/xfce4-session-<host>:<display>, so once xfwm4 is missing
+# from that list every later login is WM-less too — no title bars, no
+# Applications menu, hence no way to log out and no way to fix it from inside.
+# Seen 2026-07-28: the file listed only xfsettingsd, xfce4-panel, Thunar and
+# xfdesktop. The distro's failsafe session does NOT cover this — it only runs
+# when you pick Failsafe at the greeter, which xrdp never offers.
+#
+# How xfwm4 falls out of the list: any xfwm4 started outside the session
+# manager. `xfwm4 --replace` over SSH has no SESSION_MANAGER in its environment
+# and says so ("Failed to connect to session manager"), so the next save does
+# not record it. That is what happened here — the recovery caused the relapse.
+#
+# Recovery, from SSH:
+#   DISPLAY=:10 XAUTHORITY=~/.Xauthority xfwm4 --replace &
+#   rm ~/.cache/sessions/xfce4-session-*
+# Prevention, applied on this host by hand rather than by a task:
+#   xfconf-query -c xfce4-session -p /general/SaveOnExit -n -t bool -s false
+# Deliberately NOT automated: the system-wide xfce4-session.xml also defines
+# the failsafe session, so a role would have to either template the whole
+# distro file or add python3-lxml for the xml module, and the per-user file it
+# would otherwise write is rewritten by xfconfd at every logout — a permanently
+# "changed" task guarding against an operator mistake.
+
+# ---------------------------------------------------------------------------
+# Mouse capture in 86Box: rename xorgxrdp's pointer device
+# ---------------------------------------------------------------------------
+# xorgxrdp's pointer LIES about itself. xrdpmouse/rdpMouse.c labels its two
+# valuators REL_X/REL_Y — so XInput2 advertises "Mode: relative" with the dummy
+# range -1..-1 — and then posts every event as
+#     xf86PostMotionEvent(device, TRUE /* is_absolute */, 0, 2, x, y)
+# i.e. ABSOLUTE screen coordinates down a nominally RELATIVE axis. RDP itself
+# carries no relative motion, so it has nothing else to send.
+#
+# 86Box's X11 input backend (src/qt/xinput2_mouse.cpp) reads XI_RawMotion and
+# branches on exactly that axis mode: for a "relative" axis it feeds the raw
+# valuator straight in as a movement delta. Under xrdp that "delta" is an
+# absolute pixel coordinate, so the instant you click to capture, the emulated
+# pointer is thrown into a corner and stays pinned — which presents as the
+# mouse not being captured at all.
+#
+# Upstream already handles this failure mode, just not for us: VNC servers
+# inject through XTEST, which tells the same lie, so 86Box keeps an allow-list
+# of pointer device NAMES it treats as absolute — "TigerVNC pointer" and
+# "Virtual core XTEST pointer" — and those take a correct absolute->delta path.
+# An X input device is named after its xorg.conf Identifier, so renaming xrdp's
+# pointer onto that list is the whole fix. Verified safe on this host: the
+# identifier occurs only in the two lines this task rewrites (ServerLayout and
+# the InputDevice section), and xrdp binds the device by DRIVER ("xrdpmouse"),
+# never by identifier.
+#
+# Known limit, not fixable here: motion still stops when the CLIENT's cursor
+# hits the edge of the remote screen, because there is no relative-motion event
+# to send. xrdp PR #3091 (TS_RELPOINTER) is still an unmerged draft as of
+# 0.9.24 / xorgxrdp 0.9.19. Escaping that needs a different transport, not a
+# different setting.
+#
+# sesman starts a fresh Xorg per login, so nothing needs restarting — but a
+# session that is already running keeps the old device until the user LOGS OUT.
+# Disconnecting is not enough (that is the point of this host's setup).
+- name: Present the xrdp pointer to 86Box as the absolute device it really is
+  ansible.builtin.replace:
+    path: "{{ retro_desktop_xrdp_xorg_conf }}"
+    regexp: '"xrdpMouse"'
+    replace: '"{{ retro_desktop_xrdp_pointer_name }}"'
+
+# Stop needrestart from ever restarting xrdp-sesman. It has stranded a live
+# desktop twice now: 2026-07-25 (our own `apt install` pulled libcap2) and
+# 2026-07-28 (unattended-upgrades pulled libc6 at 06:28). Both times sesman came
+# back with an EMPTY session table, so it could no longer reattach the running
+# display — every reconnect started a new one, xfce4-session refused to run
+# twice for the same user, and each new session died in about a second. The
+# desktop and its 86Box VM keep running, just permanently unreachable.
+#
+# `NEEDRESTART_MODE: l` in retrolab.yml only covers playbook runs. It does
+# nothing about unattended-upgrades, which is what actually did it the second
+# time — so the exemption has to live on the host.
+#
+# This is not a novel idea: needrestart.conf already ships `override_rc`
+# entries pinning gdm, sddm, xdm and friends to 0, because restarting the thing
+# that owns your login sessions costs more than it fixes. xrdp-sesman is the
+# same class of service and simply is not on that list.
+#
+# The trade, stated plainly: sesman keeps running against the old libc until
+# the host is rebooted. That is the intended outcome — a stale session manager
+# that works beats a fresh one that has orphaned every session.
+- name: Never let needrestart bounce the xrdp session manager
+  ansible.builtin.copy:
+    dest: /etc/needrestart/conf.d/50-xrdp.conf
+    # conf.d is parsed after needrestart.conf and merges into the hash, so this
+    # ADDS a key rather than replacing the shipped defaults. Assigning
+    # $nrconf{override_rc} wholesale here would silently drop them.
+    content: |
+      # Managed by Ansible (proxmox/ansible/roles/retro_desktop). See that role
+      # for why: restarting sesman empties its session table and strands every
+      # live xrdp desktop until the user logs out.
+      $nrconf{override_rc}{qr(^xrdp)} = 0;
+    owner: root
+    group: root
+    mode: "0644"
+
+- name: Enable xrdp
+  ansible.builtin.systemd_service:
+    name: "{{ item }}"
+    enabled: true
+    state: started
+  loop: [xrdp, xrdp-sesman]
+
+# Audio silently not working is the classic xrdp complaint; assert the pieces
+# rather than discover it mid-session.
+- name: Read the xrdp channel configuration
+  ansible.builtin.command:
+    cmd: sed -n '/^\[Channels\]/,/^\[/p' /etc/xrdp/xrdp.ini
+  register: _chan
+  changed_when: false
+  # Reads nothing and writes nothing, but `command` is skipped under --check by
+  # default, which left `_chan.stdout` empty and made the assert below fail
+  # every dry run with a false "audio is broken".
+  check_mode: false
+  when: retro_desktop_audio | bool
+
+- name: Assert the audio channels are enabled
+  ansible.builtin.assert:
+    that: "'{{ item }}=true' in _chan.stdout"
+    fail_msg: "xrdp channel {{ item }} is not enabled — audio will not reach the client."
+    quiet: true
+  loop: "{{ retro_desktop_audio_channels }}"
+  when: retro_desktop_audio | bool
+
+- name: Assert the PipeWire xrdp module is present
+  ansible.builtin.stat:
+    path: /etc/xdg/autostart/pipewire-xrdp.desktop
+  register: _pwx
+  when: retro_desktop_audio | bool
+
+- name: Fail if the audio module is missing
+  ansible.builtin.assert:
+    that: _pwx.stat.exists
+    fail_msg: "pipewire-module-xrdp is not installed; xrdp sessions will have no sound."
+    quiet: true
+  when: retro_desktop_audio | bool
+
+- name: Confirm xrdp is listening on 3389
+  ansible.builtin.wait_for:
+    port: 3389
+    timeout: 30

infrastructure/proxmox/ansible/roles/retro_modem/files/atmodem.py

@@ -0,0 +1,864 @@
+#!/usr/bin/env python3
+"""Hayes AT modem emulator with a socket DTE side.
+
+The emulated machine's serial port connects here (unix socket or TCP). Dialling
+either opens a TCP connection (BBS) or hands the raw line to pppd -- i.e. an ISP
+terminal server, which is what a period PC actually dialled.
+
+It also answers. --line gives the modem a phone number: an inbound TCP connection
+rings the DTE, and the guest picks up with ATA (or automatically, once it has set
+S0). That is the half NT4's Remote Access Server needs to *receive* calls, and it
+is what lets one retro guest dial another.
+
+Why not tcpser: its only socket DTE transport is ip232, which doubles 0xFF and
+steals FF 00 / FF 01 to carry DTR. Every PPP frame starts FF 03, so ip232 eats
+the link. This is 8-bit clean, and speaking unix sockets natively removes the
+socat + pty sandwich (PVE's `-serial0 socket` plugs straight in).
+
+One process is one modem on one line, because that is what a modem is. An ISP's
+T1 into a rack of them is N processes on N ports; a single number in front of the
+rack is a hunt group, which is a dispatcher this does not have.
+
+No modem control lines: a QEMU socket chardev carries none, so DCD reads as
+permanently asserted in the guest and the NO CARRIER result code is the only
+carrier signal available. Guests set &C1 (DCD follows carrier) and &D2 (drop DTR
+to hang up) anyway; neither can be honoured. Terminal software parses the result
+code and copes. PPP cannot see the drop and falls back to LCP echo timeouts.
+
+  qm set 102 -serial0 socket    # QEMU LISTENS on the socket, so we connect to it
+  atmodem.py --connect /var/run/qemu-server/102.serial0 --phonebook pb.txt \
+             --line 6102 --pppd 'pppd notty 10.62.0.1:10.62.0.2 require-pap lock'
+"""
+
+import argparse
+import asyncio
+import logging
+import os
+import re
+import shlex
+import socket
+import tempfile
+import time
+
+# Hayes guard time: the silence required either side of +++ so that binary
+# payloads containing +++ do not drop the link.
+GUARD = 1.0
+RING = 4.0  # seconds between RING results, as on a real line
+RINGS = 8  # then the caller has given up
+
+log = logging.getLogger("atmodem")
+
+CODES = {"OK": 0, "CONNECT": 1, "RING": 2, "NO CARRIER": 3, "ERROR": 4, "BUSY": 7}
+
+
+def digits(s):
+    """Dialers emit ATDT9,1-555-1212 and friends; only the digits identify it."""
+    return "".join(c for c in s if c.isdigit())
+
+
+def load_phonebook(path):
+    """86Box's own format on purpose -- <number><whitespace><target>, one per
+    line -- so a single file serves both emulators. Target is host:port, or the
+    literal 'ppp' to hand the call to pppd."""
+    book = {}
+    if not path:
+        return book
+    with open(path) as f:
+        for line in f:
+            parts = line.split("#", 1)[0].split()
+            if len(parts) >= 2:
+                book[digits(parts[0])] = parts[1]
+    return book
+
+
+class Phonebook:
+    """A phonebook that re-reads itself when the file changes.
+
+    Drop-in for the dict it replaces (both answer .get). Without this, editing a
+    number means restarting the modem, which drops whatever call is up -- the
+    single most annoying thing about operating this, and three lines to fix.
+    """
+
+    def __init__(self, path):
+        self.path, self.mtime, self.book = path, None, {}
+
+    def get(self, number):
+        try:
+            mtime = os.stat(self.path).st_mtime if self.path else None
+        except OSError:
+            mtime = None  # deleted mid-flight: keep serving the last good copy
+        if mtime is not None and mtime != self.mtime:
+            self.mtime, self.book = mtime, load_phonebook(self.path)
+            log.info("phonebook reloaded: %d numbers", len(self.book))
+        return self.book.get(number)
+
+
+def parse(body):
+    """Split an AT command line into (cmd, arg) pairs.
+
+    The one subtlety worth code: extended commands are &X / %X / \\X, so a naive
+    search for 'D' fires on the &D2 in every dialer's init string and tries to
+    dial "2". Consume the prefix first. D itself swallows the rest of the line,
+    per Hayes.
+    """
+    out, i = [], 0
+    while i < len(body):
+        c = body[i]
+        if c in " \t":
+            i += 1
+            continue
+        prefix = ""
+        if c in "&%\\":
+            prefix, i = c, i + 1
+            if i >= len(body):
+                break
+            c = body[i]
+        c, i = c.upper(), i + 1
+        if not prefix and c == "D":
+            out.append(("D", body[i:].strip()))
+            break
+        if not prefix and c == "S":
+            # Sn=v is the one command whose register number matters (S0 is
+            # auto-answer), so keep them apart: ("S0", "1"), not ("S", "1").
+            m = re.match(r"(\d+)\s*=\s*(\d+)", body[i:])
+            if m:
+                i += m.end()
+                out.append(("S" + m.group(1), m.group(2)))
+                continue
+        m = re.match(r"(\d*)\s*(?:=\s*(\d+))?", body[i:])
+        i += m.end()
+        out.append((prefix + c, m.group(2) if m.group(2) is not None else m.group(1)))
+    return out
+
+
+class Telnet:
+    """A minimal telnet client, wrapping a peer stream with read/write/drain.
+
+    A real telnetd opens with `IAC WILL ECHO, IAC WILL SGA`; raw-piped, those
+    six bytes land on the guest's screen as garbage before the login prompt, and
+    an un-doubled 0xFF corrupts any 8-bit transfer. Opt in per phonebook entry
+    ('telnet:host[:port]') -- it MUST stay off for PPP and guest-to-guest links,
+    where 0xFF is data, which is the same toggle 86Box gets wrong by default.
+    """
+
+    IAC, SE, SB, WILL, WONT, DO, DONT = 255, 240, 250, 251, 252, 253, 254
+    AGREE = (1, 3)  # ECHO and SUPPRESS-GO-AHEAD: what a dumb terminal wants
+
+    def __init__(self, reader, writer):
+        self.r, self.w = reader, writer
+        self.iac = self.sb = self.sb_iac = False
+        self.verb = 0
+
+    def _filter(self, data):
+        """Split a chunk into (data for the guest, negotiation for the host)."""
+        out, reply = bytearray(), bytearray()
+        for b in data:
+            if self.sb:  # skip subnegotiation payload until IAC SE
+                if self.sb_iac and b == self.SE:
+                    self.sb = False
+                self.sb_iac = b == self.IAC and not self.sb_iac
+            elif self.verb:  # this byte is the option the verb applies to
+                if self.verb == self.WILL:
+                    yes = self.DO if b in self.AGREE else self.DONT
+                    reply += bytes((self.IAC, yes, b))
+                elif self.verb == self.DO:
+                    reply += bytes((self.IAC, self.WONT, b))  # we offer nothing
+                self.verb = 0  # WONT/DONT need no answer
+            elif self.iac:
+                self.iac = False
+                if b == self.IAC:
+                    out.append(self.IAC)  # doubled 0xFF is literal data
+                elif b in (self.WILL, self.WONT, self.DO, self.DONT):
+                    self.verb = b
+                elif b == self.SB:
+                    self.sb, self.sb_iac = True, False
+            elif b == self.IAC:
+                self.iac = True
+            else:
+                out.append(b)
+        return bytes(out), bytes(reply)
+
+    async def read(self, n):
+        while True:  # a chunk of pure negotiation yields nothing; go round again
+            data = await self.r.read(n)
+            if not data:
+                return b""
+            out, reply = self._filter(data)
+            if reply:
+                self.w.write(reply)
+                await self.w.drain()
+            if out:
+                return out
+
+    def write(self, data):
+        self.w.write(data.replace(b"\xff", b"\xff\xff"))
+
+    async def drain(self):
+        await self.w.drain()
+
+
+class Modem:
+    def __init__(self, dte_r, dte_w, book, pppd, guard=GUARD, ring=RING, hook=None):
+        self.dte_r, self.dte_w = dte_r, dte_w
+        self.book, self.pppd, self.guard = book, pppd, guard
+        self.ring_gap = ring  # not self.ring -- that name is the method below
+        self.hook = hook  # called whenever the line goes on/off hook
+        self.echo, self.verbose = True, True
+        self.s = {}  # S registers; only S0 (auto-answer) is honoured
+        self.staged = ""  # digits from ATD...; dialled in stages
+        self.pushback = b""  # bytes after a "+++ATH"-style escape
+        self.board = None  # set when this modem is one line of a switchboard
+        self.peer = None  # (reader, writer, close) while off-hook
+        self.incoming = None  # same, while the phone is ringing
+        self.autoanswer = asyncio.Event()
+
+    # -- DTE plumbing ------------------------------------------------------
+
+    async def reply(self, text):
+        # V1 is verbose words, V0 is the numeric code. Both are in the wild.
+        log.info("DCE< %s", text)
+        out = f"\r\n{text}\r\n" if self.verbose else f"{CODES[text]}\r"
+        self.dte_w.write(out.encode())
+        await self.dte_w.drain()
+
+    async def _read_byte(self):
+        """One byte from the DTE, but interruptible by auto-answer.
+
+        Without the race, S0 auto-answer could never fire: between calls the
+        modem sits blocked here, and a guest that has set S0 sends nothing at
+        all -- it just waits for CONNECT. Returns None to mean "answer now".
+        """
+        if self.pushback:  # command that arrived glued to a +++ escape
+            b, self.pushback = self.pushback[:1], self.pushback[1:]
+            return b
+        read = asyncio.ensure_future(self.dte_r.read(1))
+        ring = asyncio.ensure_future(self.autoanswer.wait())
+        done, pending = await asyncio.wait(
+            {read, ring}, return_when=asyncio.FIRST_COMPLETED)
+        for task in pending:
+            task.cancel()  # safe: unread bytes stay in the StreamReader buffer
+        if read in done:
+            return read.result()  # a real byte wins; the event survives for next time
+        self.autoanswer.clear()
+        return None
+
+    async def read_command(self):
+        """Collect one CR-terminated line, honouring echo and backspace."""
+        buf = bytearray()
+        while True:
+            b = await self._read_byte()
+            if b is None:
+                return "ATA"
+            if not b:
+                return None
+            if self.echo:
+                self.dte_w.write(b)
+                await self.dte_w.drain()
+            if b in (b"\r", b"\n"):
+                if buf:
+                    return buf.decode("latin-1")
+                buf.clear()
+            elif b == b"\x08":
+                buf[-1:] = b""
+            else:
+                buf += b
+
+    # -- command mode ------------------------------------------------------
+
+    async def command(self, line):
+        log.info("DTE> %s", line)
+        if not line[:2].upper() == "AT":
+            return await self.reply("ERROR")
+        for cmd, arg in parse(line[2:]):
+            if cmd == "D":
+                return await self.dial(arg)
+            if cmd == "A":
+                return await self.answer()
+            if cmd == "H":
+                await self.hangup()
+            elif cmd == "O" and self.peer:
+                return await self.online()
+            elif cmd == "E":
+                self.echo = arg != "0"
+            elif cmd == "V":
+                self.verbose = arg != "0"
+            elif cmd.startswith("S") and cmd[1:].isdigit():
+                self.s[int(cmd[1:])] = int(arg or 0)
+            # Everything else (Z, &F, &C1, &D2, S0=0, X4, ...) is accepted and
+            # ignored on purpose: answering OK to unknown setup commands is what
+            # makes an emulated modem work with dialers you have never seen.
+        await self.reply("OK")
+
+    async def dial(self, num):
+        # A trailing ';' means "dial, then return to command state" -- Windows
+        # TAPI opens every call with a bare `ATDT;` and only then sends the
+        # digits, so answering NO CARRIER here kills the call before it starts.
+        # Accumulate the staged digits and answer OK, as a real modem does.
+        num = num.rstrip()
+        if num.endswith(";"):
+            self.staged += num[:-1]
+            return await self.reply("OK")
+        num, self.staged = self.staged + num, ""
+
+        # D takes an optional dial modifier -- T(one) or P(ulse). Strip exactly
+        # one, never lstrip(): dialling a host called "telnet.example.com" must
+        # keep its 't'. Harmless for phonebook lookups, which go through
+        # digits(), but it lands inside the hostname when dialling an address.
+        if num[:1] in ("T", "P", "t", "p"):
+            num = num[1:].strip()
+
+        # Exact-digit lookup, same as 86Box: an outside-line prefix (ATDT9,555...)
+        # is part of the number and will miss. Turn the prefix off in the dialer.
+        target = self.book.get(digits(num)) or (num if ":" in num else None)
+        if not target:
+            return await self.reply("NO CARRIER")
+        try:
+            if target.startswith("vm:"):
+                # Internal call to another line of this switchboard: no TCP hop,
+                # no per-line port, and the callee really rings rather than
+                # being handed raw bytes.
+                name = target[len("vm:"):]
+                if self.board is None or name not in self.board.lines:
+                    return await self.reply("NO CARRIER")
+                got = await self.board.call(name)
+                if got is None:
+                    return await self.reply("BUSY")
+                # ponytail: CONNECT lands as soon as the callee starts RINGing,
+                # not when it answers -- parity with the TCP path, where the
+                # kernel completes connect() before anyone picks up. Data just
+                # buffers until the callee's ATA. Wait on an answer event if a
+                # guest ever objects.
+                self.peer = (got[0], got[1], got[1].close)
+            elif target.startswith("ssh:"):
+                # Same subprocess shape as ppp. Deliberately NOT a generic
+                # "exec:" target: a phonebook is the kind of file that ends up
+                # editable by something other than root, and arbitrary argv in
+                # it would be remote root execution wearing a phone number.
+                dest = target[len("ssh:"):]
+                host, sep, port = dest.rpartition(":")
+                cmd = ["ssh", "-tt"] + (["-p", port] if sep else []) + [host if sep else dest]
+                proc = await asyncio.create_subprocess_exec(
+                    *cmd,
+                    stdin=asyncio.subprocess.PIPE,
+                    stdout=asyncio.subprocess.PIPE,
+                )
+                self.peer = (proc.stdout, proc.stdin, proc.kill)
+            elif target == "ppp":
+                proc = await asyncio.create_subprocess_exec(
+                    *self.pppd,
+                    stdin=asyncio.subprocess.PIPE,
+                    stdout=asyncio.subprocess.PIPE,
+                )
+                self.peer = (proc.stdout, proc.stdin, proc.kill)
+            elif target.startswith("telnet:"):
+                addr = target[len("telnet:"):]
+                host, sep, port = addr.rpartition(":")
+                if not sep:  # "telnet:bbs.example.com" -- the default port is the point
+                    host, port = addr, "23"
+                r, w = await asyncio.open_connection(host, int(port))
+                tn = Telnet(r, w)
+                self.peer = (tn, tn, w.close)
+            else:
+                host, _, port = target.rpartition(":")
+                r, w = await asyncio.open_connection(host, int(port))
+                self.peer = (r, w, w.close)
+        except ConnectionRefusedError:
+            # The far end is engaged: a busy line closes its listener, so the
+            # refusal at connect() IS the busy tone. Must be caught before
+            # OSError, which it subclasses.
+            return await self.reply("BUSY")
+        except (OSError, ValueError):
+            return await self.reply("NO CARRIER")
+        await self._hook()
+        await self.reply("CONNECT")
+        await self.online()
+
+    async def _hook(self):
+        """Tell the owner whether the line is engaged. A ringing line counts as
+        engaged, same as a real one -- you do not get two calls on one pair."""
+        if self.hook:
+            await self.hook()
+
+    async def ring(self, r, w):
+        """An inbound call. Ring the DTE until it answers or the caller tires."""
+        call = self.incoming = (r, w, w.close)
+        await self._hook()
+        for n in range(1, RINGS + 1):
+            if self.incoming is not call:
+                return  # answered, or hung up on
+            await self.reply("RING")
+            if self.s.get(0) and n >= self.s[0]:
+                self.autoanswer.set()
+                return
+            await asyncio.sleep(self.ring_gap)
+        if self.incoming is call:  # nobody picked up
+            self.incoming = None
+            w.close()
+            await self._hook()
+
+    async def answer(self):
+        if not self.incoming:
+            return await self.reply("NO CARRIER")
+        self.peer, self.incoming = self.incoming, None
+        await self.reply("CONNECT")
+        await self.online()
+
+    async def hangup(self):
+        # ATH while ringing rejects the call, which is also how a real one behaves.
+        for slot in ("peer", "incoming"):
+            call = getattr(self, slot)
+            if call:
+                try:
+                    call[2]()
+                except (OSError, ProcessLookupError):
+                    pass
+                setattr(self, slot, None)
+        await self._hook()
+
+    # -- data mode ---------------------------------------------------------
+
+    async def online(self):
+        """Pump both directions until +++ escapes or the far end drops.
+
+        Returns to command mode with the call still up on escape (ATO resumes,
+        ATH hangs up), exactly as a real modem does.
+        """
+        peer_r, peer_w, _ = self.peer
+        up = asyncio.ensure_future(self._dte_to_peer(peer_w))
+        down = asyncio.ensure_future(self._peer_to_dte(peer_r))
+        # Whichever direction ends first ends the call: waiting only on the DTE
+        # side left the modem pumping into a dead peer after NO CARRIER, so it
+        # never came back to command mode and swallowed every later command.
+        done, pending = await asyncio.wait(
+            {up, down}, return_when=asyncio.FIRST_COMPLETED)
+        for task in pending:
+            task.cancel()
+        if up in done and up.result():  # +++ : the call stays up, ATO resumes it
+            await self.reply("OK")
+
+    async def _dte_to_peer(self, peer_w):
+        """Guest -> far end. True if the guest escaped with +++."""
+        held, last = b"", time.monotonic()
+        while True:
+            try:
+                data = await asyncio.wait_for(
+                    self.dte_r.read(4096), self.guard if held else None)
+            except asyncio.TimeoutError:
+                return True  # trailing guard elapsed: escaped
+            if not data:
+                return False
+            if held:  # +++ was followed by traffic, so it was just data
+                peer_w.write(held)
+                held = b""
+            if data.startswith(b"+++") and time.monotonic() - last >= self.guard:
+                # NT4's RAS sends the escape and the command as ONE write --
+                # "+++ATH" -- so matching only a bare b"+++" forwarded the whole
+                # thing to the far end and the line could never be hung up.
+                # A bare +++ still needs its trailing guard time; +++ followed by
+                # a command is unambiguous, so escape at once and hand the rest
+                # to the command reader.
+                # ponytail: still assumes +++ arrives in one read. A dialer that
+                # dribbles it byte-by-byte needs a per-byte timer instead.
+                if len(data) == 3:
+                    held = data
+                    continue
+                self.pushback = data[3:]
+                return True
+            try:
+                peer_w.write(data)
+                await peer_w.drain()
+            except (ConnectionError, BrokenPipeError):
+                return False
+            last = time.monotonic()
+
+    async def _peer_to_dte(self, peer_r):
+        """Far end -> guest, until carrier drops."""
+        while True:
+            data = await peer_r.read(4096)
+            if not data:
+                break
+            self.dte_w.write(data)
+            await self.dte_w.drain()
+        await self.hangup()
+        await self.reply("NO CARRIER")
+
+    async def run(self):
+        while True:
+            line = await self.read_command()
+            if line is None:
+                return await self.hangup()
+            await self.command(line)
+
+
+class Switchboard:
+    """The lines this process owns, so a call between two of them never leaves it.
+
+    Dialling a VM cannot be "just another target type": the answering guest needs
+    a MODEM to hear RING and reply ATA, so wiring the caller straight to its
+    serial socket would hand RAS raw bytes and it would never pick up. Only a
+    process holding both ends can ring one on behalf of the other -- which is
+    also where the hunt group and a real busy signal come from.
+    """
+
+    def __init__(self):
+        self.lines = {}  # name -> Modem, while attached
+
+    def free(self, name):
+        modem = self.lines.get(name)
+        return modem is not None and not modem.peer and not modem.incoming
+
+    async def call(self, name):
+        """Ring line `name`. Returns the caller's end, or None if it is engaged.
+
+        A socketpair is the whole implementation: each modem gets one end and
+        every path below -- the pump, 8-bit cleanliness, +++, NO CARRIER -- is
+        the same code that carries an external call.
+        """
+        if not self.free(name):
+            return None
+        left, right = socket.socketpair()
+        caller_r, caller_w = await asyncio.open_connection(sock=left)
+        callee_r, callee_w = await asyncio.open_connection(sock=right)
+        asyncio.ensure_future(self.lines[name].ring(callee_r, callee_w))
+        return caller_r, caller_w
+
+
+async def connect_dte(target):
+    """Attach to a DTE that is listening: a unix socket path, or host:port."""
+    if ":" in target:
+        host, _, port = target.rpartition(":")
+        return await asyncio.open_connection(host, int(port))
+    return await asyncio.open_unix_connection(target)
+
+
+async def serve(book, pppd, listen=None, connect=None, line=None, guard=GUARD,
+                ring=RING, board=None, name=None):
+    """Bring up one modem and, if --line was given, its phone line.
+
+    Returns (dte_server_or_None, phone), where phone["port"] is the line's real
+    port once it has been bound. With a `board`, the modem registers itself so
+    other lines can ring it without going out over TCP.
+    """
+    state = {}
+    phone = {"port": None if line is None else int(line), "srv": None}
+
+    async def call(r, w):
+        modem = state.get("modem")
+        if modem is None or modem.peer or modem.incoming:
+            return w.close()  # lost a race with hook(); refuse politely
+        await modem.ring(r, w)
+
+    async def hook():
+        """Listen only while a modem is attached and on-hook.
+
+        This is the busy signal. Closing the listener means a second caller is
+        refused by the kernel at connect(), before anything can pretend the call
+        went through -- whereas accepting and then closing would make the caller
+        report CONNECT and immediately NO CARRIER. A ringing line is engaged too.
+        """
+        if phone["port"] is None:
+            return
+        modem = state.get("modem")
+        free = modem is not None and not modem.peer and not modem.incoming
+        if free and not phone["srv"]:
+            # 0.0.0.0, not "": an unspecified host binds one socket per family,
+            # and with port 0 each gets a *different* ephemeral port. v4-only is
+            # honest here -- retronet has no IPv6 and the laptop's is broken.
+            phone["srv"] = await asyncio.start_server(call, "0.0.0.0", phone["port"])
+            phone["port"] = phone["srv"].sockets[0].getsockname()[1]
+        elif not free and phone["srv"]:
+            phone["srv"].close()
+            phone["srv"] = None
+
+    async def attach(r, w):
+        modem = Modem(r, w, book, pppd, guard, ring, hook)
+        modem.board = board
+        state["modem"] = modem
+        if board is not None:
+            board.lines[name] = modem
+        await hook()
+        try:
+            await modem.run()
+        finally:
+            state.pop("modem", None)
+            if board is not None:
+                board.lines.pop(name, None)
+            await hook()
+            w.close()
+
+    if connect:
+        # PVE's `-serial0 socket` leaves QEMU listening, so for a VM we are the
+        # client. 86Box and plain TCP want the opposite; hence both modes.
+        async def keep_attached():
+            """Reattach forever: a guest reboot takes the chardev peer with it,
+            and a switchboard that needs restarting after every VM reboot is not
+            a service."""
+            while True:
+                try:
+                    r, w = await connect_dte(connect)
+                except OSError as exc:
+                    log.warning("line %s: %s", name or connect, exc)
+                    await asyncio.sleep(5)
+                    continue
+                log.info("line %s attached", name or connect)
+                await attach(r, w)
+                log.info("line %s dropped, reattaching", name or connect)
+                await asyncio.sleep(1)
+
+        asyncio.ensure_future(keep_attached())
+        return None, phone
+
+    if ":" in listen:
+        host, _, port = listen.rpartition(":")
+        dte = await asyncio.start_server(attach, host or "127.0.0.1", int(port))
+    else:
+        dte = await asyncio.start_unix_server(attach, listen)
+    return dte, phone
+
+
+async def selftest():
+    """Covers the three things that are easy to get wrong: the link must be
+    8-bit clean (what ip232 fails, and the reason this file exists), &D2 must
+    not read as a dial command, and an inbound call must ring and be answerable
+    both ways -- ATA, and S0 auto-answer."""
+    # start_server only schedules the callback if it is a coroutine FUNCTION --
+    # a lambda returning a coroutine is silently dropped. Pass _echo itself.
+    echo = await asyncio.start_server(_echo, "127.0.0.1", 0)
+    port = echo.sockets[0].getsockname()[1]
+    dead = _closed_port()  # nothing listening: dialling it must give BUSY
+    book = {"5551212": f"127.0.0.1:{port}", "5559999": f"127.0.0.1:{dead}"}
+    dte, phone = await serve(book, [], listen="127.0.0.1:0", line=0,
+                             guard=0.05, ring=0.05)
+    r, w = await asyncio.open_connection(*dte.sockets[0].getsockname()[:2])
+
+    # unknown setup commands must not be mistaken for a dial (&D2!)
+    assert parse("&F&C1&D2S0=0") == [("&F", ""), ("&C", "1"), ("&D", "2"), ("S0", "0")]
+    assert parse("DT9,1-555-1212") == [("D", "T9,1-555-1212")]
+    assert digits("9,1-555-1212") == "915551212"
+
+    w.write(b"AT&F&C1&D2S0=0\r")
+    await w.drain()
+    assert b"OK" in await r.readuntil(b"OK\r\n")
+    line = phone["port"]  # only bound once a modem attached, which OK just proved
+    assert line, "line never opened"
+
+    w.write(b"ATDT555-1212\r")
+    await w.drain()
+    assert b"CONNECT" in await r.readuntil(b"CONNECT\r\n")
+
+    payload = bytes(range(256)) * 4  # 0xFF and FF 03 included, deliberately
+    w.write(payload)
+    await w.drain()
+    assert await r.readexactly(len(payload)) == payload, "link is not 8-bit clean"
+
+    await asyncio.sleep(0.2)  # leading guard
+    w.write(b"+++")
+    await w.drain()
+    assert b"OK" in await r.readuntil(b"OK\r\n"), "+++ did not escape"
+    w.write(b"ATH\r")
+    await w.drain()
+    assert b"OK" in await r.readuntil(b"OK\r\n")
+
+    # the real Win98 dial sequence: init string, then a staged dial
+    w.write(b"ATE0V1&C1&D2S0=0\r")
+    await w.drain()
+    assert b"OK" in await r.readuntil(b"OK\r\n"), "Win98 init string rejected"
+    w.write(b"ATDT;\r")           # TAPI opens the call with no digits at all
+    await w.drain()
+    assert b"OK" in await r.readuntil(b"OK\r\n"), "ATDT; must be OK, not NO CARRIER"
+    w.write(b"ATDT555-1212\r")    # ...and only then dials
+    await w.drain()
+    assert b"CONNECT" in await r.readuntil(b"CONNECT\r\n")
+    await asyncio.sleep(0.2)      # data mode now: escape before ATH is a command
+    w.write(b"+++")
+    await w.drain()
+    assert b"OK" in await r.readuntil(b"OK\r\n")
+    w.write(b"ATH\r")
+    await w.drain()
+    assert b"OK" in await r.readuntil(b"OK\r\n")
+
+    # "+++ATH" in one write is how NT4 RAS hangs up: escape, then the command
+    w.write(b"ATDT555-1212\r")
+    await w.drain()
+    assert b"CONNECT" in await r.readuntil(b"CONNECT\r\n")
+    await asyncio.sleep(0.2)
+    w.write(b"+++ATH\r")
+    await w.drain()
+    assert b"OK" in await r.readuntil(b"OK\r\n"), "+++ATH did not escape"
+    assert b"OK" in await r.readuntil(b"OK\r\n"), "ATH after +++ was not obeyed"
+
+    # an address dialled straight, with no phonebook entry: the T modifier
+    # must not end up inside the hostname
+    w.write(f"ATDT127.0.0.1:{port}\r".encode())
+    await w.drain()
+    assert b"CONNECT" in await r.readuntil(b"CONNECT\r\n"), "direct address dial failed"
+    await asyncio.sleep(0.2)
+    w.write(b"+++ATH\r")
+    await w.drain()
+    assert b"OK" in await r.readuntil(b"OK\r\n")
+    assert b"OK" in await r.readuntil(b"OK\r\n")
+
+    # telnet: mode must swallow the IAC handshake and answer it, so the guest
+    # sees only the prompt -- and must double 0xFF on the way out
+    seen = []
+
+    async def fake_telnetd(tr, tw):
+        tw.write(b"\xff\xfb\x01\xff\xfb\x03login: ")  # WILL ECHO, WILL SGA
+        await tw.drain()
+        while (d := await tr.read(100)):
+            seen.append(d)
+
+    td = await asyncio.start_server(fake_telnetd, "127.0.0.1", 0)
+    book["5552323"] = f"telnet:127.0.0.1:{td.sockets[0].getsockname()[1]}"
+    w.write(b"ATDT5552323\r")
+    await w.drain()
+    assert b"CONNECT" in await r.readuntil(b"CONNECT\r\n")
+    assert await r.readexactly(7) == b"login: ", "IAC negotiation reached the guest"
+    w.write(b"\xff\x01")               # guest sends a literal 0xFF
+    await w.drain()
+    await asyncio.sleep(0.3)
+    got = b"".join(seen)
+    assert got.startswith(b"\xff\xfd\x01\xff\xfd\x03"), f"no DO ECHO / DO SGA: {got}"
+    assert got.endswith(b"\xff\xff\x01"), f"0xFF was not doubled: {got}"
+    await asyncio.sleep(0.2)
+    w.write(b"+++ATH\r")
+    await w.drain()
+    assert b"OK" in await r.readuntil(b"OK\r\n")
+    assert b"OK" in await r.readuntil(b"OK\r\n")
+
+    # dialling a line with nobody on it is BUSY, not NO CARRIER
+    w.write(b"ATDT5559999\r")
+    await w.drain()
+    assert b"BUSY" in await r.readuntil(b"BUSY\r\n")
+
+    # inbound, answered by hand -- NT4 RAS / TAPI issue ATA on RING
+    cr, cw = await asyncio.open_connection("127.0.0.1", line)
+    assert b"RING" in await r.readuntil(b"RING\r\n")
+
+    # ...and while it rings, the line is engaged: the second caller is refused
+    # by the kernel, so its modem reports BUSY instead of a phantom CONNECT
+    try:
+        await asyncio.open_connection("127.0.0.1", line)
+        raise AssertionError("second caller was not given a busy line")
+    except ConnectionRefusedError:
+        pass
+
+    w.write(b"ATA\r")
+    await w.drain()
+    assert b"CONNECT" in await r.readuntil(b"CONNECT\r\n")
+    cw.write(b"\xff\x03hello")  # a PPP-shaped frame, inbound this time
+    await cw.drain()
+    assert await r.readexactly(7) == b"\xff\x03hello"
+    cw.close()
+    assert b"NO CARRIER" in await r.readuntil(b"NO CARRIER\r\n")
+
+    # inbound, auto-answered: the guest sets S0 and then says nothing at all
+    w.write(b"ATS0=2\r")
+    await w.drain()
+    assert b"OK" in await r.readuntil(b"OK\r\n")
+    cr, cw = await asyncio.open_connection("127.0.0.1", line)
+    assert b"CONNECT" in await r.readuntil(b"CONNECT\r\n"), "S0 did not auto-answer"
+    # switchboard: two lines in one process, an internal call between them
+    tmp = tempfile.mkdtemp()
+    ends = {}
+
+    async def fake_qemu(nm):  # stands in for QEMU's listening serial socket
+        async def cb(qr, qw):
+            ends[nm] = (qr, qw)
+        await asyncio.start_unix_server(cb, f"{tmp}/{nm}.sock")
+
+    await fake_qemu("A")
+    await fake_qemu("B")
+    board, book2 = Switchboard(), {"5551111": "vm:B", "5552222": "vm:NOPE"}
+    for nm in ("A", "B"):
+        await serve(book2, [], connect=f"{tmp}/{nm}.sock", board=board, name=nm,
+                    guard=0.05, ring=0.05)
+    for _ in range(60):
+        if len(board.lines) == 2 and len(ends) == 2:
+            break
+        await asyncio.sleep(0.05)
+    assert len(board.lines) == 2, f"lines did not attach: {board.lines}"
+    ar, aw = ends["A"]
+    br, bw = ends["B"]
+
+    aw.write(b"ATDT5551111\r")
+    await aw.drain()
+    assert b"RING" in await br.readuntil(b"RING\r\n"), "callee never rang"
+    bw.write(b"ATA\r")
+    await bw.drain()
+    assert b"CONNECT" in await ar.readuntil(b"CONNECT\r\n")
+    assert b"CONNECT" in await br.readuntil(b"CONNECT\r\n")
+    both = bytes(range(256))  # the internal hop must be 8-bit clean too
+    aw.write(both)
+    await aw.drain()
+    assert await br.readexactly(256) == both, "internal call is not 8-bit clean"
+
+    # an unknown line is NO CARRIER; an engaged one is BUSY
+    await asyncio.sleep(0.2)
+    bw.write(b"+++")
+    await bw.drain()
+    assert b"OK" in await br.readuntil(b"OK\r\n")
+    bw.write(b"ATDT5552222\r")
+    await bw.drain()
+    assert b"NO CARRIER" in await br.readuntil(b"NO CARRIER\r\n"), "unknown line"
+    bw.write(b"ATDT5551111\r")   # line B calling... a line that is engaged (itself)
+    await bw.drain()
+    assert b"BUSY" in await br.readuntil(b"BUSY\r\n"), "engaged line was not BUSY"
+
+    print("ok")
+
+
+def _closed_port():
+    """A port the OS just handed back, so nothing is listening on it."""
+    with socket.socket() as s:
+        s.bind(("127.0.0.1", 0))
+        return s.getsockname()[1]
+
+
+async def _echo(r, w):
+    while (data := await r.read(4096)):
+        w.write(data)
+        await w.drain()
+
+
+def main():
+    ap = argparse.ArgumentParser(description=__doc__)
+    dte = ap.add_mutually_exclusive_group()
+    dte.add_argument("--listen", help="unix socket path, or host:port, to listen on")
+    dte.add_argument("--connect", help="unix socket path, or host:port, to dial into "
+                                       "(PVE's -serial0 socket leaves QEMU listening)")
+    ap.add_argument("--phonebook", help="86Box-format number->target map")
+    ap.add_argument("--pppd", default="", help="command run for a 'ppp' target")
+    ap.add_argument("--line", help="TCP port to accept incoming calls on")
+    ap.add_argument("--vm", action="append", default=[], metavar="VMID[:PORT]",
+                    help="switchboard line on a PVE VM's serial0 socket; repeatable. "
+                         "PORT accepts calls from off-box (86Box on another host); "
+                         "other lines reach it as the phonebook target vm:VMID")
+    ap.add_argument("--debug", action="store_true", help="log the AT conversation")
+    ap.add_argument("--selftest", action="store_true")
+    a = ap.parse_args()
+    logging.basicConfig(level=logging.INFO if a.debug else logging.WARNING,
+                        format="%(asctime)s %(message)s", datefmt="%H:%M:%S")
+    if a.selftest:
+        return asyncio.run(selftest())
+    if not (a.listen or a.connect or a.vm):
+        ap.error("one of --listen, --connect or --vm is required")
+    asyncio.run(_run(a))
+
+
+async def _run(a):
+    book, pppd = Phonebook(a.phonebook), shlex.split(a.pppd)
+    if a.vm:
+        board = Switchboard()
+        for spec in a.vm:
+            vmid, _, port = spec.partition(":")
+            await serve(book, pppd, board=board, name=vmid, line=port or None,
+                        connect=f"/var/run/qemu-server/{vmid}.serial0")
+            log.info("line %s%s", vmid, f" answering on :{port}" if port else "")
+    if a.listen or a.connect:
+        await serve(book, pppd, listen=a.listen, connect=a.connect, line=a.line)
+    await asyncio.Event().wait()
+
+
+if __name__ == "__main__":
+    main()

infrastructure/proxmox/ansible/roles/tap_bridge/defaults/main.yml

@@ -0,0 +1,23 @@
+---
+# A bare Linux bridge with one physical port and NO IP address, used as the
+# attach point for emulator TAP interfaces (86Box, PCem, qemu, ...).
+#
+# WHY a bridge at all: 86Box's TAP backend creates its own tap device
+# (TUNSETIFF) and then enslaves it to a bridge you name (SIOCBRADDIF). It does
+# NOT bridge to a raw NIC, so the NIC has to already be in a bridge for the
+# emulated guests to reach the wire.
+#
+# WHY no IP on the bridge: the host deliberately does not sit on the retro
+# segment. Guests use the VyOS SDN gateway (10.61.0.1) directly. Giving the
+# host an address here would put an up-to-date Linux box inside the retro
+# broadcast domain for no reason, and would make the retro guests able to
+# reach this host's services.
+tap_bridge_name: br0
+
+# Matched by MAC, not by name: the kernel's predictable-interface name depends
+# on PCI slot ordering, which moves if the VM's NIC layout changes. The MACs
+# are pinned in pve_vm defaults precisely so they can be matched on.
+tap_bridge_member: ""
+tap_bridge_member_mac: ""
+
+tap_bridge_netplan_file: /etc/netplan/60-tap-bridge.yaml

infrastructure/proxmox/ansible/roles/tap_bridge/handlers/main.yml

@@ -0,0 +1,9 @@
+---
+- name: Apply netplan
+  # `netplan apply` re-applies EVERY netplan file, including the cloud-init one
+  # that owns the management NIC this play is connected over. It does not
+  # normally bounce an unchanged interface, but if a run ever hangs here that
+  # is the reason -- check the console via `qm terminal 101`.
+  ansible.builtin.command:
+    cmd: netplan apply
+  changed_when: true

infrastructure/proxmox/ansible/roles/tap_bridge/tasks/main.yml

@@ -0,0 +1,59 @@
+---
+- name: Check the bridge role is configured
+  ansible.builtin.assert:
+    that:
+      - tap_bridge_member | length > 0
+      - tap_bridge_member_mac | length > 0
+    fail_msg: "tap_bridge_member and tap_bridge_member_mac must be set"
+
+- name: Install bridge utilities
+  # bridge-utils is not needed by netplan (it uses netlink), but brctl is the
+  # quickest way to see which taps an emulator has attached while debugging.
+  ansible.builtin.apt:
+    name: bridge-utils
+    state: present
+    update_cache: true
+    cache_valid_time: 3600
+  register: _brutils
+  retries: 3
+  delay: 15
+  until: _brutils is succeeded
+
+- name: Configure the bridge in netplan
+  ansible.builtin.template:
+    src: netplan.yaml.j2
+    dest: "{{ tap_bridge_netplan_file }}"
+    # 0600: netplan warns loudly about world-readable config and ignores such
+    # files in newer releases.
+    mode: "0600"
+    owner: root
+    group: root
+  notify: Apply netplan
+
+- name: Flush handlers so the bridge exists before it is verified
+  ansible.builtin.meta: flush_handlers
+
+- name: Read back the bridge state
+  ansible.builtin.command:
+    cmd: "ip -br link show {{ tap_bridge_name }}"
+  register: _br
+  changed_when: false
+  failed_when: false
+
+- name: Read back the enslaved port
+  # `master <bridge>` in `ip link` output is the authoritative proof that the
+  # NIC is actually in the bridge -- netplan reporting success is not.
+  ansible.builtin.shell:
+    cmd: "ip -o link show {{ tap_bridge_member }} | grep -o 'master [^ ]*' || true"
+  register: _member
+  changed_when: false
+
+- name: Verify the bridge is up with the port enslaved
+  ansible.builtin.assert:
+    that:
+      - _br.rc == 0
+      - "'master ' ~ tap_bridge_name in _member.stdout"
+    fail_msg: >-
+      Bridge {{ tap_bridge_name }} is not correctly set up.
+      bridge: {{ _br.stdout | default('absent') }} / port: {{ _member.stdout | default('none') }}
+    success_msg: "{{ tap_bridge_name }} up, {{ tap_bridge_member }} enslaved"

infrastructure/proxmox/ansible/roles/tap_bridge/templates/netplan.yaml.j2

@@ -0,0 +1,33 @@
+# {{ ansible_managed }}
+# Bridge for emulator TAP interfaces. Separate from 50-cloud-init.yaml on
+# purpose: cloud-init rewrites its own file, and it only claims the primary
+# NIC (matched by MAC), so this file can own the second NIC without conflict.
+network:
+  version: 2
+  ethernets:
+    {{ tap_bridge_member }}:
+      match:
+        macaddress: "{{ tap_bridge_member_mac }}"
+      set-name: {{ tap_bridge_member }}
+      dhcp4: false
+      dhcp6: false
+      accept-ra: false
+      # optional: do not let systemd-networkd-wait-online block boot for a port
+      # that has no L3 config and may have nothing plugged behind it.
+      optional: true
+  bridges:
+    {{ tap_bridge_name }}:
+      interfaces: [{{ tap_bridge_member }}]
+      dhcp4: false
+      dhcp6: false
+      accept-ra: false
+      optional: true
+      parameters:
+        # STP off + zero forward delay: taps appear and disappear every time an
+        # emulated machine is powered on. With STP the bridge would hold each
+        # new port in learning state for ~15s and silently drop the guest's
+        # first packets, which looks exactly like a broken guest TCP/IP stack.
+        # Safe here because this bridge has a single uplink -- no loop is
+        # possible.
+        stp: false
+        forward-delay: 0

infrastructure/proxmox/ansible/roles/vyos_router/defaults/main.yml

@@ -0,0 +1,73 @@
+---
+# VyOS router VM (VM 100) — gateway for the PVE SDN VNets, OSPF peer to the NEC IX.
+#
+# HOW THIS IS MANAGED — and why not the obvious way:
+# We drive `vyos.vyos.vyos_config` with explicit `set` lines from a template,
+# NOT the collection's resource modules (vyos_interfaces / vyos_ospfv2 /
+# vyos_firewall_rules). Reason, verified 2026-07-25 on this box:
+#   * VyOS 2025.11 REJECTS the old syntax: `set firewall name X ...` ->
+#     "Configuration path: firewall [name] is not valid"
+#   * vyos.vyos 6.0.0 still EMITS `firewall name` (grepped the module_utils)
+# So the resource modules would generate config this release cannot parse. On a
+# Stream/tech-preview build the safe move is to own the syntax ourselves;
+# vyos_config still gives idempotency by diffing against the running config.
+
+vyos_router_hostname: vyos-rtr
+
+# LAN leg — OSPF adjacency with the NEC IX (and the laptop, which is also an
+# OSPF speaker on this segment).
+vyos_lan_interface: eth0
+vyos_lan_address: "192.168.10.2/24"
+vyos_lan_gateway: "192.168.10.1"
+
+vyos_nameserver: "192.168.10.5"
+vyos_dhcp_nameserver: "192.168.10.5"     # the Samba DC — see the flaky-WAN notes
+
+# SDN legs. `vnet` is the PVE VNet the NIC is attached to; `passive` keeps OSPF
+# from trying to form adjacencies with guests on these segments.
+# `dhcp.subnet_id` MUST be unique across the whole dhcp-server config (VyOS 1.4+
+# requires it explicitly). Ranges deliberately start at .100 so .2-.99 stay free
+# for anything that wants a static address inside a VNet.
+vyos_sdn_interfaces:
+  - iface: eth1
+    vnet: labnet
+    address: "10.60.0.1/24"
+    network: "10.60.0.0/24"
+    description: "labnet VLAN100 gateway"
+    dhcp:
+      subnet_id: 1
+      start: "10.60.0.100"
+      stop: "10.60.0.200"
+      domain: "ad.ddupan.top"
+      # Reservations sit BELOW the .100 pool start so they never collide with it.
+      # Keyed on the VM's pinned MAC (see proxmox/ansible/roles/pve_vm).
+      reservations:
+        - { name: retrolab, mac: "bc:24:11:68:a0:51", address: "10.60.0.10" }
+  - iface: eth2
+    vnet: retronet
+    address: "10.61.0.1/24"
+    network: "10.61.0.0/24"
+    description: "retronet VLAN110 gateway"
+    dhcp:
+      subnet_id: 2
+      start: "10.61.0.100"
+      stop: "10.61.0.200"
+      domain: "ad.ddupan.top"
+      # Retro Windows (9x/NT/2000) resolves names via NetBIOS, not DNS, so the
+      # segment needs a WINS server -- this is the whole reason retronet exists.
+      # Was 192.168.10.5 (the Samba DC); now retro-pdc, which is the PDC of the
+      # RETRONET domain, so browser elections and domain logons resolve on-segment
+      # and retronet keeps no dependency on the production DC.
+      # PRECONDITION: the WINS service must actually be installed and running on
+      # retro-pdc -- an unanswering wins-server option is worse than the old one.
+      wins: "10.61.0.5"
+
+# OSPF. The SDN subnets are declared as INTRA-AREA networks, deliberately.
+# Do NOT switch this to `redistribute connected`: that advertises every connected
+# interface (so any future NIC leaks automatically) and injects E2 routes whose
+# metric does not accumulate path cost. Verified on the IX: the same prefix went
+# from `O E2 ... [110/20]` to `O ... [110/2]` after this change.
+vyos_ospf_router_id: "192.168.10.2"
+vyos_ospf_area: "0"
+vyos_ospf_networks:
+  - "192.168.10.0/24"

infrastructure/proxmox/ansible/roles/vyos_router/tasks/main.yml

@@ -0,0 +1,69 @@
+---
+# Apply the desired VyOS configuration. `vyos_config` compares the rendered
+# `set` lines against the running config and issues only the differences, so
+# re-runs are no-ops. Supports --check and --diff.
+
+- name: Render the desired configuration
+  ansible.builtin.set_fact:
+    _vyos_lines: >-
+      {{ lookup('template', 'vyos.conf.j2').splitlines()
+         | map('trim') | reject('equalto', '') | list }}
+
+- name: Apply configuration
+  vyos.vyos.vyos_config:
+    lines: "{{ _vyos_lines }}"
+    # Persist to config.boot; without this the config is lost on reboot.
+    save: true
+    # Pull the PRE-change running config back to the control host. VyOS also
+    # keeps its own commit revisions (`show system commit`, `rollback N`), but
+    # those are only reachable if the box is still reachable -- which is exactly
+    # what a bad change takes away.
+    backup: true
+    backup_options:
+      dir_path: "{{ playbook_dir }}/../vyos/backups"
+      filename: "config.boot"
+    comment: "ansible {{ lookup('pipe', 'date -u +%Y-%m-%dT%H:%M:%SZ') }}"
+  register: _vyos_cfg
+  # `backup: true` fetches the running config every run, which the module counts
+  # as a change. Report changed ONLY when commands were actually issued,
+  # otherwise real drift is indistinguishable from a routine backup.
+  changed_when: (_vyos_cfg.commands | default([]) | length) > 0
+
+- name: Show what changed
+  ansible.builtin.debug:
+    msg: "{{ _vyos_cfg.commands | default(['(no changes)']) }}"
+
+# ── post-deploy verification ──────────────────────────────────────────────
+# Proving the config was WRITTEN is not the same as proving the network still
+# WORKS. These assert operational state, which is the part a bad change breaks.
+- name: Collect operational state
+  vyos.vyos.vyos_command:
+    commands:
+      - show ip ospf neighbor
+      - show interfaces
+  register: _vyos_state
+  changed_when: false
+
+- name: Assert OSPF adjacency with the upstream router is Full
+  ansible.builtin.assert:
+    that: "'Full' in _vyos_state.stdout[0] and vyos_lan_gateway in _vyos_state.stdout[0]"
+    fail_msg: >-
+      No Full OSPF adjacency with {{ vyos_lan_gateway }}. The SDN subnets are
+      NOT being advertised, so nothing can reach them.
+      Neighbors seen:\n{{ _vyos_state.stdout[0] }}
+    success_msg: "OSPF adjacency with {{ vyos_lan_gateway }} is Full"
+
+- name: Assert each SDN gateway address is actually live
+  ansible.builtin.assert:
+    that: "item.address in _vyos_state.stdout[1]"
+    fail_msg: >-
+      {{ item.iface }} ({{ item.description }}) is missing {{ item.address }} --
+      guests on that VNet have no gateway.
+    quiet: true
+  loop: "{{ vyos_sdn_interfaces }}"
+  loop_control:
+    label: "{{ item.iface }} {{ item.address }}"
+
+- name: Report
+  ansible.builtin.debug:
+    msg: "{{ _vyos_state.stdout[0].splitlines() | select('search', 'Full') | list }}"

infrastructure/proxmox/ansible/roles/vyos_router/templates/vyos.conf.j2

@@ -0,0 +1,50 @@
+{# Desired VyOS config as `set` lines. vyos_config diffs these against the
+   running config, so only differences are applied. Written for VyOS 2025.11
+   syntax explicitly -- see defaults/main.yml for why we do not use the
+   collection's resource modules. #}
+set system host-name {{ vyos_router_hostname }}
+set system name-server {{ vyos_nameserver }}
+
+{# --- LAN leg: OSPF peer with the NEC IX --- #}
+set interfaces ethernet {{ vyos_lan_interface }} address {{ vyos_lan_address }}
+set interfaces ethernet {{ vyos_lan_interface }} description 'LAN / OSPF to NEC IX'
+
+{# --- SDN legs: gateways for the PVE VNets --- #}
+{% for i in vyos_sdn_interfaces %}
+set interfaces ethernet {{ i.iface }} address {{ i.address }}
+set interfaces ethernet {{ i.iface }} description '{{ i.description }}'
+{% endfor %}
+
+{# --- default route out; OSPF carries the rest --- #}
+set protocols static route 0.0.0.0/0 next-hop {{ vyos_lan_gateway }}
+
+{# --- OSPF: intra-area, NOT redistribute connected --- #}
+set protocols ospf parameters router-id {{ vyos_ospf_router_id }}
+{% for n in vyos_ospf_networks %}
+set protocols ospf area {{ vyos_ospf_area }} network {{ n }}
+{% endfor %}
+{% for i in vyos_sdn_interfaces %}
+set protocols ospf area {{ vyos_ospf_area }} network {{ i.network }}
+{# passive: advertise the subnet, but never try to peer with guests on it #}
+set protocols ospf interface {{ i.iface }} passive
+{% endfor %}
+
+{# --- management --- #}
+set service ssh port 22
+
+{# --- DHCP for the SDN VNets --- #}
+{% for i in vyos_sdn_interfaces if i.dhcp is defined %}
+set service dhcp-server shared-network-name {{ i.vnet | upper }} subnet {{ i.network }} subnet-id {{ i.dhcp.subnet_id }}
+set service dhcp-server shared-network-name {{ i.vnet | upper }} subnet {{ i.network }} option default-router {{ i.address.split('/')[0] }}
+set service dhcp-server shared-network-name {{ i.vnet | upper }} subnet {{ i.network }} option name-server {{ vyos_dhcp_nameserver }}
+set service dhcp-server shared-network-name {{ i.vnet | upper }} subnet {{ i.network }} option domain-name {{ i.dhcp.domain }}
+{% if i.dhcp.wins is defined %}
+set service dhcp-server shared-network-name {{ i.vnet | upper }} subnet {{ i.network }} option wins-server {{ i.dhcp.wins }}
+{% endif %}
+set service dhcp-server shared-network-name {{ i.vnet | upper }} subnet {{ i.network }} range 0 start {{ i.dhcp.start }}
+set service dhcp-server shared-network-name {{ i.vnet | upper }} subnet {{ i.network }} range 0 stop {{ i.dhcp.stop }}
+{% for r in i.dhcp.reservations | default([]) %}
+set service dhcp-server shared-network-name {{ i.vnet | upper }} subnet {{ i.network }} static-mapping {{ r.name }} mac {{ r.mac }}
+set service dhcp-server shared-network-name {{ i.vnet | upper }} subnet {{ i.network }} static-mapping {{ r.name }} ip-address {{ r.address }}
+{% endfor %}
+{% endfor %}

infrastructure/proxmox/ansible/sdn.yml

@@ -0,0 +1,8 @@
+---
+# PVE SDN (VLAN zone). Separate from site.yml: it defines guest networking, and
+# applying SDN reloads network config on every node.
+- name: Proxmox SDN
+  hosts: pve
+  gather_facts: false
+  roles:
+    - pve_sdn

infrastructure/proxmox/ansible/site.yml

@@ -0,0 +1,41 @@
+---
+# Baseline configuration for the 3-node Proxmox cluster.
+#
+#   ansible-playbook site.yml --check      # dry run first
+#   ansible-playbook site.yml
+#   ansible-playbook site.yml --limit pve2
+#   ansible-playbook site.yml -e pve_dist_upgrade=true
+#
+# Cluster formation and LINSTOR are deliberately NOT here yet -- this playbook
+# is safe to re-run against nodes at any time.
+- name: Baseline Proxmox nodes
+  hosts: pve
+  gather_facts: true
+  # One node at a time. These are cluster members; a mistake applied in parallel
+  # to all three is a mistake with no healthy node left to compare against.
+  serial: 1
+  roles:
+    - pve_network
+    - pve_dns
+    - pve_post_install
+    - pve_ca_trust
+    # Must follow pve_ca_trust: the node has to trust the internal CA before a
+    # cert issued by it is any use, and the CA fetch is the cheaper thing to fail.
+    - role: pve_acme
+      # Tagged so cert work can be re-run or audited on its own:
+      #   ansible-playbook site.yml --tags acme
+      tags: [acme]
+    - pve_mail_relay
+    - pve_kernel_params
+    - pve_nfs
+
+# Deliberately its own play, on ONE node: the role must not be part of the
+# serial:1 loop above, which would put three instances of the same UI on the LAN.
+# pvesh proxies to whichever node owns a VM, so one is enough for the cluster.
+#   ansible-playbook site.yml --tags floppy
+- name: Floppy drive UI
+  hosts: pve1
+  gather_facts: false
+  tags: [floppy]
+  roles:
+    - pve_floppy

infrastructure/proxmox/ansible/vms.yml

@@ -0,0 +1,8 @@
+---
+# VM definitions. Creation only — this does not reconcile live VMs.
+#   ansible-playbook vms.yml
+- name: Proxmox VMs
+  hosts: pve1
+  gather_facts: false
+  roles:
+    - pve_vm

infrastructure/proxmox/ansible/vyos.yml

@@ -0,0 +1,9 @@
+---
+# VyOS router configuration.
+#   ansible-playbook vyos.yml --check --diff
+#   ansible-playbook vyos.yml
+- name: VyOS router
+  hosts: vyos-rtr
+  gather_facts: false
+  roles:
+    - vyos_router

infrastructure/proxmox/lvm-recovery/UUIDS.txt

@@ -0,0 +1,2 @@
+  pve2: W7gP56-VTY2-UaCN-OgmN-yT1c-jiRg-PPzQVd
+  pve3: Jt95bI-RHQ4-WnBF-TCqB-lj6y-OUEG-6g7I3n

infrastructure/proxmox/lvm-recovery/pve2-archive/ceph-167b988c-7c90-4898-baf3-d619675acbb2_00000-280520128.vg

@@ -0,0 +1,61 @@
+# Generated by LVM2 version 2.03.31(2) (2025-02-27): Sat Jul 25 04:25:33 2026
+
+contents = "Text Format Volume Group"
+version = 1
+
+description = "Created *before* executing '/sbin/vgs --separator : --noheadings --units b --unbuffered --nosuffix --options vg_name,vg_size,vg_free,lv_count'"
+
+creation_host = "pve2"	# Linux pve2 7.0.2-6-pve #1 SMP PREEMPT_DYNAMIC PMX 7.0.2-6 (2026-05-20T08:55Z) x86_64
+creation_time = 1784921133	# Sat Jul 25 04:25:33 2026
+
+ceph-167b988c-7c90-4898-baf3-d619675acbb2 {
+	id = "mqsI8e-TFKD-erja-LLrp-z9GS-4D4D-s1EftS"
+	seqno = 5
+	format = "lvm2"			# informational
+	status = ["RESIZEABLE", "READ", "WRITE"]
+	flags = []
+	extent_size = 8192		# 4 Megabytes
+	max_lv = 0
+	max_pv = 0
+	metadata_copies = 0
+
+	physical_volumes {
+
+		pv0 {
+			id = "icuiQG-cNJ4-VkNz-wk5Y-ilek-Awhh-LpMiMR"
+			device = "/dev/sda"	# Hint only
+
+			status = ["ALLOCATABLE"]
+			flags = []
+			dev_size = 1953525168	# 931.513 Gigabytes
+			pe_start = 2048
+			pe_count = 238467	# 931.512 Gigabytes
+		}
+	}
+
+	logical_volumes {
+
+		osd-block-cd522829-87d2-4c5a-b7f1-6954d1550a72 {
+			id = "kMPnPl-AbSn-mnnH-KlXK-xY9R-SIh7-ZEVPE8"
+			status = ["READ", "WRITE", "VISIBLE"]
+			flags = []
+			tags = ["ceph.osd_fsid=cd522829-87d2-4c5a-b7f1-6954d1550a72", "ceph.osd_id=1", "ceph.cluster_fsid=dfea42a0-6089-4a79-ba81-cae3356ac41f", "ceph.cluster_name=ceph", "ceph.crush_device_class=", "ceph.osdspec_affinity=", "ceph.block_device=/dev/ceph-167b988c-7c90-4898-baf3-d619675acbb2/osd-block-cd522829-87d2-4c5a-b7f1-6954d1550a72", "ceph.block_uuid=kMPnPl-AbSn-mnnH-KlXK-xY9R-SIh7-ZEVPE8", "ceph.cephx_lockbox_secret=", "ceph.encrypted=0", "ceph.with_tpm=0", "ceph.vdo=0", "ceph.type=block", "ceph.db_uuid=TGfD0i-n7OW-3Bjp-z4k6-FGSI-D6PO-VfBS1c", "ceph.db_device=/dev/ceph-fa6cef5f-3822-4252-acea-627d43747837/osd-db-0b54061e-dce6-4be7-8be9-295d2a8383be"]
+			creation_time = 1766233980	# 2025-12-20 21:33:00 +0900
+			creation_host = "pve2"
+			segment_count = 1
+
+			segment1 {
+				start_extent = 0
+				extent_count = 238467	# 931.512 Gigabytes
+
+				type = "striped"
+				stripe_count = 1	# linear
+
+				stripes = [
+					"pv0", 0
+				]
+			}
+		}
+	}
+
+}

infrastructure/proxmox/lvm-recovery/pve2-archive/pve_00000-1403618575.vg

@@ -0,0 +1,166 @@
+# Generated by LVM2 version 2.03.31(2) (2025-02-27): Sat Jul 25 04:25:33 2026
+
+contents = "Text Format Volume Group"
+version = 1
+
+description = "Created *before* executing '/sbin/vgs --separator : --noheadings --units b --unbuffered --nosuffix --options vg_name,vg_size,vg_free,lv_count'"
+
+creation_host = "pve2"	# Linux pve2 7.0.2-6-pve #1 SMP PREEMPT_DYNAMIC PMX 7.0.2-6 (2026-05-20T08:55Z) x86_64
+creation_time = 1784921133	# Sat Jul 25 04:25:33 2026
+
+pve {
+	id = "Vv2rZ1-WkZM-Qvpx-4dL2-1hRp-xe8c-ey2qJU"
+	seqno = 7
+	format = "lvm2"			# informational
+	status = ["RESIZEABLE", "READ", "WRITE"]
+	flags = []
+	extent_size = 8192		# 4 Megabytes
+	max_lv = 0
+	max_pv = 0
+	metadata_copies = 0
+
+	physical_volumes {
+
+		pv0 {
+			id = "W7gP56-VTY2-UaCN-OgmN-yT1c-jiRg-PPzQVd"
+			device = "/dev/nvme0n1p3"	# Hint only
+
+			status = ["ALLOCATABLE"]
+			flags = []
+			dev_size = 123729921	# 58.999 Gigabytes
+			pe_start = 2048
+			pe_count = 15103	# 58.9961 Gigabytes
+		}
+	}
+
+	logical_volumes {
+
+		swap {
+			id = "Vq63er-hRX6-tseB-Wl2j-sqeE-3aBd-FaOXuc"
+			status = ["READ", "WRITE", "VISIBLE"]
+			flags = []
+			creation_time = 1784920960	# 2026-07-25 04:22:40 +0900
+			creation_host = "proxmox"
+			segment_count = 1
+
+			segment1 {
+				start_extent = 0
+				extent_count = 1856	# 7.25 Gigabytes
+
+				type = "striped"
+				stripe_count = 1	# linear
+
+				stripes = [
+					"pv0", 0
+				]
+			}
+		}
+
+		root {
+			id = "BsbZiD-RkOH-oZrv-xS8B-qjzX-JtXn-2jAxwu"
+			status = ["READ", "WRITE", "VISIBLE"]
+			flags = []
+			creation_time = 1784920960	# 2026-07-25 04:22:40 +0900
+			creation_host = "proxmox"
+			segment_count = 1
+
+			segment1 {
+				start_extent = 0
+				extent_count = 6383	# 24.9336 Gigabytes
+
+				type = "striped"
+				stripe_count = 1	# linear
+
+				stripes = [
+					"pv0", 1856
+				]
+			}
+		}
+
+		data {
+			id = "dd2Ugb-nXPl-8f87-N7pq-D9wq-Llcn-cb5hg7"
+			status = ["READ", "WRITE", "VISIBLE"]
+			flags = []
+			creation_time = 1784920960	# 2026-07-25 04:22:40 +0900
+			creation_host = "proxmox"
+			segment_count = 1
+
+			segment1 {
+				start_extent = 0
+				extent_count = 4495	# 17.5586 Gigabytes
+
+				type = "thin-pool"
+				metadata = "data_tmeta"
+				pool = "data_tdata"
+				transaction_id = 0
+				chunk_size = 128	# 64 Kilobytes
+				discards = "passdown"
+				zero_new_blocks = 1
+			}
+		}
+
+		data_tmeta {
+			id = "jnAVce-fi8p-7Nkn-fJ1v-e942-5vdy-20wRWp"
+			status = ["READ", "WRITE"]
+			flags = []
+			creation_time = 1784920960	# 2026-07-25 04:22:40 +0900
+			creation_host = "proxmox"
+			segment_count = 1
+
+			segment1 {
+				start_extent = 0
+				extent_count = 256	# 1024 Megabytes
+
+				type = "striped"
+				stripe_count = 1	# linear
+
+				stripes = [
+					"pv0", 12734
+				]
+			}
+		}
+
+		lvol0_pmspare {
+			id = "igVgep-HKOE-dKLU-K3x0-qtTh-ckMr-oFh9ia"
+			status = ["READ", "WRITE"]
+			flags = []
+			creation_time = 1784920962	# 2026-07-25 04:22:42 +0900
+			creation_host = "proxmox"
+			segment_count = 1
+
+			segment1 {
+				start_extent = 0
+				extent_count = 256	# 1024 Megabytes
+
+				type = "striped"
+				stripe_count = 1	# linear
+
+				stripes = [
+					"pv0", 12990
+				]
+			}
+		}
+
+		data_tdata {
+			id = "hvHPTe-d2xI-s9tE-5soa-b27A-Wwx4-E7j7Gq"
+			status = ["READ", "WRITE"]
+			flags = []
+			creation_time = 1784920964	# 2026-07-25 04:22:44 +0900
+			creation_host = "proxmox"
+			segment_count = 1
+
+			segment1 {
+				start_extent = 0
+				extent_count = 4495	# 17.5586 Gigabytes
+
+				type = "striped"
+				stripe_count = 1	# linear
+
+				stripes = [
+					"pv0", 8239
+				]
+			}
+		}
+	}
+
+}

infrastructure/proxmox/lvm-recovery/pve2-pve-metadata

@@ -0,0 +1,166 @@
+# Generated by LVM2 version 2.03.31(2) (2025-02-27): Sat Jul 25 04:25:33 2026
+
+contents = "Text Format Volume Group"
+version = 1
+
+description = "Created *after* executing '/sbin/vgs --separator : --noheadings --units b --unbuffered --nosuffix --options vg_name,vg_size,vg_free,lv_count'"
+
+creation_host = "pve2"	# Linux pve2 7.0.2-6-pve #1 SMP PREEMPT_DYNAMIC PMX 7.0.2-6 (2026-05-20T08:55Z) x86_64
+creation_time = 1784921133	# Sat Jul 25 04:25:33 2026
+
+pve {
+	id = "Vv2rZ1-WkZM-Qvpx-4dL2-1hRp-xe8c-ey2qJU"
+	seqno = 7
+	format = "lvm2"			# informational
+	status = ["RESIZEABLE", "READ", "WRITE"]
+	flags = []
+	extent_size = 8192		# 4 Megabytes
+	max_lv = 0
+	max_pv = 0
+	metadata_copies = 0
+
+	physical_volumes {
+
+		pv0 {
+			id = "W7gP56-VTY2-UaCN-OgmN-yT1c-jiRg-PPzQVd"
+			device = "/dev/nvme0n1p3"	# Hint only
+
+			status = ["ALLOCATABLE"]
+			flags = []
+			dev_size = 123729921	# 58.999 Gigabytes
+			pe_start = 2048
+			pe_count = 15103	# 58.9961 Gigabytes
+		}
+	}
+
+	logical_volumes {
+
+		swap {
+			id = "Vq63er-hRX6-tseB-Wl2j-sqeE-3aBd-FaOXuc"
+			status = ["READ", "WRITE", "VISIBLE"]
+			flags = []
+			creation_time = 1784920960	# 2026-07-25 04:22:40 +0900
+			creation_host = "proxmox"
+			segment_count = 1
+
+			segment1 {
+				start_extent = 0
+				extent_count = 1856	# 7.25 Gigabytes
+
+				type = "striped"
+				stripe_count = 1	# linear
+
+				stripes = [
+					"pv0", 0
+				]
+			}
+		}
+
+		root {
+			id = "BsbZiD-RkOH-oZrv-xS8B-qjzX-JtXn-2jAxwu"
+			status = ["READ", "WRITE", "VISIBLE"]
+			flags = []
+			creation_time = 1784920960	# 2026-07-25 04:22:40 +0900
+			creation_host = "proxmox"
+			segment_count = 1
+
+			segment1 {
+				start_extent = 0
+				extent_count = 6383	# 24.9336 Gigabytes
+
+				type = "striped"
+				stripe_count = 1	# linear
+
+				stripes = [
+					"pv0", 1856
+				]
+			}
+		}
+
+		data {
+			id = "dd2Ugb-nXPl-8f87-N7pq-D9wq-Llcn-cb5hg7"
+			status = ["READ", "WRITE", "VISIBLE"]
+			flags = []
+			creation_time = 1784920960	# 2026-07-25 04:22:40 +0900
+			creation_host = "proxmox"
+			segment_count = 1
+
+			segment1 {
+				start_extent = 0
+				extent_count = 4495	# 17.5586 Gigabytes
+
+				type = "thin-pool"
+				metadata = "data_tmeta"
+				pool = "data_tdata"
+				transaction_id = 0
+				chunk_size = 128	# 64 Kilobytes
+				discards = "passdown"
+				zero_new_blocks = 1
+			}
+		}
+
+		data_tmeta {
+			id = "jnAVce-fi8p-7Nkn-fJ1v-e942-5vdy-20wRWp"
+			status = ["READ", "WRITE"]
+			flags = []
+			creation_time = 1784920960	# 2026-07-25 04:22:40 +0900
+			creation_host = "proxmox"
+			segment_count = 1
+
+			segment1 {
+				start_extent = 0
+				extent_count = 256	# 1024 Megabytes
+
+				type = "striped"
+				stripe_count = 1	# linear
+
+				stripes = [
+					"pv0", 12734
+				]
+			}
+		}
+
+		lvol0_pmspare {
+			id = "igVgep-HKOE-dKLU-K3x0-qtTh-ckMr-oFh9ia"
+			status = ["READ", "WRITE"]
+			flags = []
+			creation_time = 1784920962	# 2026-07-25 04:22:42 +0900
+			creation_host = "proxmox"
+			segment_count = 1
+
+			segment1 {
+				start_extent = 0
+				extent_count = 256	# 1024 Megabytes
+
+				type = "striped"
+				stripe_count = 1	# linear
+
+				stripes = [
+					"pv0", 12990
+				]
+			}
+		}
+
+		data_tdata {
+			id = "hvHPTe-d2xI-s9tE-5soa-b27A-Wwx4-E7j7Gq"
+			status = ["READ", "WRITE"]
+			flags = []
+			creation_time = 1784920964	# 2026-07-25 04:22:44 +0900
+			creation_host = "proxmox"
+			segment_count = 1
+
+			segment1 {
+				start_extent = 0
+				extent_count = 4495	# 17.5586 Gigabytes
+
+				type = "striped"
+				stripe_count = 1	# linear
+
+				stripes = [
+					"pv0", 8239
+				]
+			}
+		}
+	}
+
+}

infrastructure/proxmox/lvm-recovery/pve3-archive/ceph-c7860953-e6b0-401c-81a8-9271c4920b45_00000-395409748.vg

@@ -0,0 +1,61 @@
+# Generated by LVM2 version 2.03.31(2) (2025-02-27): Sat Jul 25 04:15:39 2026
+
+contents = "Text Format Volume Group"
+version = 1
+
+description = "Created *before* executing '/sbin/vgs --separator : --noheadings --units b --unbuffered --nosuffix --options vg_name,vg_size,vg_free,lv_count'"
+
+creation_host = "pve3"	# Linux pve3 7.0.2-6-pve #1 SMP PREEMPT_DYNAMIC PMX 7.0.2-6 (2026-05-20T08:55Z) x86_64
+creation_time = 1784920539	# Sat Jul 25 04:15:39 2026
+
+ceph-c7860953-e6b0-401c-81a8-9271c4920b45 {
+	id = "Khv4bE-npJw-cNXq-OcjO-ltkd-pZuu-ohJNOF"
+	seqno = 5
+	format = "lvm2"			# informational
+	status = ["RESIZEABLE", "READ", "WRITE"]
+	flags = []
+	extent_size = 8192		# 4 Megabytes
+	max_lv = 0
+	max_pv = 0
+	metadata_copies = 0
+
+	physical_volumes {
+
+		pv0 {
+			id = "dF1pQ9-Twe9-Akdq-F0Vy-7uZv-mptx-n1JUw1"
+			device = "/dev/sda"	# Hint only
+
+			status = ["ALLOCATABLE"]
+			flags = []
+			dev_size = 1953525168	# 931.513 Gigabytes
+			pe_start = 2048
+			pe_count = 238467	# 931.512 Gigabytes
+		}
+	}
+
+	logical_volumes {
+
+		osd-block-ac905dac-06ab-4cc9-b0d7-56bbf4650f8e {
+			id = "70eZh0-oi4f-58Bt-jLne-xG4k-c3oQ-eTbdGt"
+			status = ["READ", "WRITE", "VISIBLE"]
+			flags = []
+			tags = ["ceph.osd_fsid=ac905dac-06ab-4cc9-b0d7-56bbf4650f8e", "ceph.osd_id=2", "ceph.cluster_fsid=dfea42a0-6089-4a79-ba81-cae3356ac41f", "ceph.cluster_name=ceph", "ceph.crush_device_class=", "ceph.osdspec_affinity=", "ceph.block_device=/dev/ceph-c7860953-e6b0-401c-81a8-9271c4920b45/osd-block-ac905dac-06ab-4cc9-b0d7-56bbf4650f8e", "ceph.block_uuid=70eZh0-oi4f-58Bt-jLne-xG4k-c3oQ-eTbdGt", "ceph.cephx_lockbox_secret=", "ceph.encrypted=0", "ceph.with_tpm=0", "ceph.vdo=0", "ceph.type=block", "ceph.db_uuid=5cUIlD-8uct-5hV5-BdOO-asiY-Bprs-KLFb5G", "ceph.db_device=/dev/ceph-7f98c2d0-9331-4563-b796-a9d01f92a90e/osd-db-9c90a0dd-921a-4e3e-9c02-f8ef3967e406"]
+			creation_time = 1766234030	# 2025-12-20 21:33:50 +0900
+			creation_host = "pve3"
+			segment_count = 1
+
+			segment1 {
+				start_extent = 0
+				extent_count = 238467	# 931.512 Gigabytes
+
+				type = "striped"
+				stripe_count = 1	# linear
+
+				stripes = [
+					"pv0", 0
+				]
+			}
+		}
+	}
+
+}

infrastructure/proxmox/lvm-recovery/pve3-archive/pve_00000-1126721422.vg

@@ -0,0 +1,166 @@
+# Generated by LVM2 version 2.03.31(2) (2025-02-27): Sat Jul 25 04:15:39 2026
+
+contents = "Text Format Volume Group"
+version = 1
+
+description = "Created *before* executing '/sbin/vgs --separator : --noheadings --units b --unbuffered --nosuffix --options vg_name,vg_size,vg_free,lv_count'"
+
+creation_host = "pve3"	# Linux pve3 7.0.2-6-pve #1 SMP PREEMPT_DYNAMIC PMX 7.0.2-6 (2026-05-20T08:55Z) x86_64
+creation_time = 1784920539	# Sat Jul 25 04:15:39 2026
+
+pve {
+	id = "oH8tdA-8NxD-JQNp-mtRv-22q8-z8FD-6Oe48B"
+	seqno = 7
+	format = "lvm2"			# informational
+	status = ["RESIZEABLE", "READ", "WRITE"]
+	flags = []
+	extent_size = 8192		# 4 Megabytes
+	max_lv = 0
+	max_pv = 0
+	metadata_copies = 0
+
+	physical_volumes {
+
+		pv0 {
+			id = "Jt95bI-RHQ4-WnBF-TCqB-lj6y-OUEG-6g7I3n"
+			device = "/dev/nvme0n1p3"	# Hint only
+
+			status = ["ALLOCATABLE"]
+			flags = []
+			dev_size = 123729921	# 58.999 Gigabytes
+			pe_start = 2048
+			pe_count = 15103	# 58.9961 Gigabytes
+		}
+	}
+
+	logical_volumes {
+
+		swap {
+			id = "X0Gqtt-SZD8-hER8-sOTF-NqMz-8ypg-6EPuhW"
+			status = ["READ", "WRITE", "VISIBLE"]
+			flags = []
+			creation_time = 1784920369	# 2026-07-25 04:12:49 +0900
+			creation_host = "proxmox"
+			segment_count = 1
+
+			segment1 {
+				start_extent = 0
+				extent_count = 1856	# 7.25 Gigabytes
+
+				type = "striped"
+				stripe_count = 1	# linear
+
+				stripes = [
+					"pv0", 0
+				]
+			}
+		}
+
+		root {
+			id = "gmtDqF-k0YD-eB74-gbFn-UP63-LAgK-j2JdHB"
+			status = ["READ", "WRITE", "VISIBLE"]
+			flags = []
+			creation_time = 1784920369	# 2026-07-25 04:12:49 +0900
+			creation_host = "proxmox"
+			segment_count = 1
+
+			segment1 {
+				start_extent = 0
+				extent_count = 6383	# 24.9336 Gigabytes
+
+				type = "striped"
+				stripe_count = 1	# linear
+
+				stripes = [
+					"pv0", 1856
+				]
+			}
+		}
+
+		data {
+			id = "09Pvtu-BcsK-23mB-ouOp-D5cc-0epZ-RW4UVV"
+			status = ["READ", "WRITE", "VISIBLE"]
+			flags = []
+			creation_time = 1784920369	# 2026-07-25 04:12:49 +0900
+			creation_host = "proxmox"
+			segment_count = 1
+
+			segment1 {
+				start_extent = 0
+				extent_count = 4495	# 17.5586 Gigabytes
+
+				type = "thin-pool"
+				metadata = "data_tmeta"
+				pool = "data_tdata"
+				transaction_id = 0
+				chunk_size = 128	# 64 Kilobytes
+				discards = "passdown"
+				zero_new_blocks = 1
+			}
+		}
+
+		data_tmeta {
+			id = "lhM9mI-1ZGn-sMfR-AjCq-j2D6-Plgi-04f8Dr"
+			status = ["READ", "WRITE"]
+			flags = []
+			creation_time = 1784920369	# 2026-07-25 04:12:49 +0900
+			creation_host = "proxmox"
+			segment_count = 1
+
+			segment1 {
+				start_extent = 0
+				extent_count = 256	# 1024 Megabytes
+
+				type = "striped"
+				stripe_count = 1	# linear
+
+				stripes = [
+					"pv0", 12734
+				]
+			}
+		}
+
+		lvol0_pmspare {
+			id = "apmoWT-rcHL-f3y8-1X4c-cMUy-pvEk-NGm3he"
+			status = ["READ", "WRITE"]
+			flags = []
+			creation_time = 1784920372	# 2026-07-25 04:12:52 +0900
+			creation_host = "proxmox"
+			segment_count = 1
+
+			segment1 {
+				start_extent = 0
+				extent_count = 256	# 1024 Megabytes
+
+				type = "striped"
+				stripe_count = 1	# linear
+
+				stripes = [
+					"pv0", 12990
+				]
+			}
+		}
+
+		data_tdata {
+			id = "zij0Es-lKe7-KnEZ-0yom-I2R4-sIL1-nTAMbT"
+			status = ["READ", "WRITE"]
+			flags = []
+			creation_time = 1784920374	# 2026-07-25 04:12:54 +0900
+			creation_host = "proxmox"
+			segment_count = 1
+
+			segment1 {
+				start_extent = 0
+				extent_count = 4495	# 17.5586 Gigabytes
+
+				type = "striped"
+				stripe_count = 1	# linear
+
+				stripes = [
+					"pv0", 8239
+				]
+			}
+		}
+	}
+
+}

infrastructure/proxmox/lvm-recovery/pve3-pve-metadata

@@ -0,0 +1,166 @@
+# Generated by LVM2 version 2.03.31(2) (2025-02-27): Sat Jul 25 04:15:39 2026
+
+contents = "Text Format Volume Group"
+version = 1
+
+description = "Created *after* executing '/sbin/vgs --separator : --noheadings --units b --unbuffered --nosuffix --options vg_name,vg_size,vg_free,lv_count'"
+
+creation_host = "pve3"	# Linux pve3 7.0.2-6-pve #1 SMP PREEMPT_DYNAMIC PMX 7.0.2-6 (2026-05-20T08:55Z) x86_64
+creation_time = 1784920539	# Sat Jul 25 04:15:39 2026
+
+pve {
+	id = "oH8tdA-8NxD-JQNp-mtRv-22q8-z8FD-6Oe48B"
+	seqno = 7
+	format = "lvm2"			# informational
+	status = ["RESIZEABLE", "READ", "WRITE"]
+	flags = []
+	extent_size = 8192		# 4 Megabytes
+	max_lv = 0
+	max_pv = 0
+	metadata_copies = 0
+
+	physical_volumes {
+
+		pv0 {
+			id = "Jt95bI-RHQ4-WnBF-TCqB-lj6y-OUEG-6g7I3n"
+			device = "/dev/nvme0n1p3"	# Hint only
+
+			status = ["ALLOCATABLE"]
+			flags = []
+			dev_size = 123729921	# 58.999 Gigabytes
+			pe_start = 2048
+			pe_count = 15103	# 58.9961 Gigabytes
+		}
+	}
+
+	logical_volumes {
+
+		swap {
+			id = "X0Gqtt-SZD8-hER8-sOTF-NqMz-8ypg-6EPuhW"
+			status = ["READ", "WRITE", "VISIBLE"]
+			flags = []
+			creation_time = 1784920369	# 2026-07-25 04:12:49 +0900
+			creation_host = "proxmox"
+			segment_count = 1
+
+			segment1 {
+				start_extent = 0
+				extent_count = 1856	# 7.25 Gigabytes
+
+				type = "striped"
+				stripe_count = 1	# linear
+
+				stripes = [
+					"pv0", 0
+				]
+			}
+		}
+
+		root {
+			id = "gmtDqF-k0YD-eB74-gbFn-UP63-LAgK-j2JdHB"
+			status = ["READ", "WRITE", "VISIBLE"]
+			flags = []
+			creation_time = 1784920369	# 2026-07-25 04:12:49 +0900
+			creation_host = "proxmox"
+			segment_count = 1
+
+			segment1 {
+				start_extent = 0
+				extent_count = 6383	# 24.9336 Gigabytes
+
+				type = "striped"
+				stripe_count = 1	# linear
+
+				stripes = [
+					"pv0", 1856
+				]
+			}
+		}
+
+		data {
+			id = "09Pvtu-BcsK-23mB-ouOp-D5cc-0epZ-RW4UVV"
+			status = ["READ", "WRITE", "VISIBLE"]
+			flags = []
+			creation_time = 1784920369	# 2026-07-25 04:12:49 +0900
+			creation_host = "proxmox"
+			segment_count = 1
+
+			segment1 {
+				start_extent = 0
+				extent_count = 4495	# 17.5586 Gigabytes
+
+				type = "thin-pool"
+				metadata = "data_tmeta"
+				pool = "data_tdata"
+				transaction_id = 0
+				chunk_size = 128	# 64 Kilobytes
+				discards = "passdown"
+				zero_new_blocks = 1
+			}
+		}
+
+		data_tmeta {
+			id = "lhM9mI-1ZGn-sMfR-AjCq-j2D6-Plgi-04f8Dr"
+			status = ["READ", "WRITE"]
+			flags = []
+			creation_time = 1784920369	# 2026-07-25 04:12:49 +0900
+			creation_host = "proxmox"
+			segment_count = 1
+
+			segment1 {
+				start_extent = 0
+				extent_count = 256	# 1024 Megabytes
+
+				type = "striped"
+				stripe_count = 1	# linear
+
+				stripes = [
+					"pv0", 12734
+				]
+			}
+		}
+
+		lvol0_pmspare {
+			id = "apmoWT-rcHL-f3y8-1X4c-cMUy-pvEk-NGm3he"
+			status = ["READ", "WRITE"]
+			flags = []
+			creation_time = 1784920372	# 2026-07-25 04:12:52 +0900
+			creation_host = "proxmox"
+			segment_count = 1
+
+			segment1 {
+				start_extent = 0
+				extent_count = 256	# 1024 Megabytes
+
+				type = "striped"
+				stripe_count = 1	# linear
+
+				stripes = [
+					"pv0", 12990
+				]
+			}
+		}
+
+		data_tdata {
+			id = "zij0Es-lKe7-KnEZ-0yom-I2R4-sIL1-nTAMbT"
+			status = ["READ", "WRITE"]
+			flags = []
+			creation_time = 1784920374	# 2026-07-25 04:12:54 +0900
+			creation_host = "proxmox"
+			segment_count = 1
+
+			segment1 {
+				start_extent = 0
+				extent_count = 4495	# 17.5586 Gigabytes
+
+				type = "striped"
+				stripe_count = 1	# linear
+
+				stripes = [
+					"pv0", 8239
+				]
+			}
+		}
+	}
+
+}

infrastructure/proxmox/vyos/backups/config.boot

@@ -0,0 +1,67 @@
+set interfaces ethernet eth0 address '192.168.10.2/24'
+set interfaces ethernet eth0 description 'LAN / OSPF to NEC IX'
+set interfaces ethernet eth0 hw-id 'bc:24:11:d6:4c:ac'
+set interfaces ethernet eth0 offload gro
+set interfaces ethernet eth0 offload gso
+set interfaces ethernet eth0 offload sg
+set interfaces ethernet eth0 offload tso
+set interfaces ethernet eth1 address '10.60.0.1/24'
+set interfaces ethernet eth1 description 'labnet VLAN100 gateway'
+set interfaces ethernet eth1 hw-id 'bc:24:11:1b:d3:e9'
+set interfaces ethernet eth1 offload gro
+set interfaces ethernet eth1 offload gso
+set interfaces ethernet eth1 offload sg
+set interfaces ethernet eth1 offload tso
+set interfaces ethernet eth2 address '10.61.0.1/24'
+set interfaces ethernet eth2 description 'retronet VLAN110 gateway'
+set interfaces ethernet eth2 hw-id 'bc:24:11:bd:3c:7a'
+set interfaces ethernet eth2 offload gro
+set interfaces ethernet eth2 offload gso
+set interfaces ethernet eth2 offload sg
+set interfaces ethernet eth2 offload tso
+set interfaces loopback lo
+set protocols ospf area 0 network '192.168.10.0/24'
+set protocols ospf area 0 network '10.60.0.0/24'
+set protocols ospf area 0 network '10.61.0.0/24'
+set protocols ospf interface eth1 passive
+set protocols ospf interface eth2 passive
+set protocols ospf parameters router-id '192.168.10.2'
+set protocols static route 0.0.0.0/0 next-hop 192.168.10.1
+set service dhcp-server shared-network-name LABNET subnet 10.60.0.0/24 option default-router '10.60.0.1'
+set service dhcp-server shared-network-name LABNET subnet 10.60.0.0/24 option domain-name 'ad.ddupan.top'
+set service dhcp-server shared-network-name LABNET subnet 10.60.0.0/24 option name-server '192.168.10.5'
+set service dhcp-server shared-network-name LABNET subnet 10.60.0.0/24 range 0 start '10.60.0.100'
+set service dhcp-server shared-network-name LABNET subnet 10.60.0.0/24 range 0 stop '10.60.0.200'
+set service dhcp-server shared-network-name LABNET subnet 10.60.0.0/24 static-mapping retrolab ip-address '10.60.0.10'
+set service dhcp-server shared-network-name LABNET subnet 10.60.0.0/24 static-mapping retrolab mac 'bc:24:11:68:a0:51'
+set service dhcp-server shared-network-name LABNET subnet 10.60.0.0/24 subnet-id '1'
+set service dhcp-server shared-network-name RETRONET subnet 10.61.0.0/24 option default-router '10.61.0.1'
+set service dhcp-server shared-network-name RETRONET subnet 10.61.0.0/24 option domain-name 'ad.ddupan.top'
+set service dhcp-server shared-network-name RETRONET subnet 10.61.0.0/24 option name-server '192.168.10.5'
+set service dhcp-server shared-network-name RETRONET subnet 10.61.0.0/24 option wins-server '10.61.0.5'
+set service dhcp-server shared-network-name RETRONET subnet 10.61.0.0/24 range 0 start '10.61.0.100'
+set service dhcp-server shared-network-name RETRONET subnet 10.61.0.0/24 range 0 stop '10.61.0.200'
+set service dhcp-server shared-network-name RETRONET subnet 10.61.0.0/24 subnet-id '2'
+set service ntp allow-client address '127.0.0.0/8'
+set service ntp allow-client address '169.254.0.0/16'
+set service ntp allow-client address '10.0.0.0/8'
+set service ntp allow-client address '172.16.0.0/12'
+set service ntp allow-client address '192.168.0.0/16'
+set service ntp allow-client address '::1/128'
+set service ntp allow-client address 'fe80::/10'
+set service ntp allow-client address 'fc00::/7'
+set service ntp server time1.vyos.net
+set service ntp server time2.vyos.net
+set service ntp server time3.vyos.net
+set service ssh port '22'
+set system config-management commit-revisions '100'
+set system console device ttyS0 speed '115200'
+set system host-name 'vyos-rtr'
+set system login user vyos authentication encrypted-password '$6$rounds=656000$LrCooxAFgQ99.OkP$a5G8BnDrtoHl/8ZujgVIJr0z9aP.LZL35A1W8SXMnEDxT7eafT9Z3eoUD9jHBseIVuaqK2QpTMNg1YthCdDo..'
+set system login user vyos authentication plaintext-password ''
+set system login user vyos authentication public-keys laptop key 'AAAAC3NzaC1lZDI1NTE5AAAAIOLvzIxZhVRd9wEFWR/uCOx7b4HQEdPDiZd8LCN7Hics'
+set system login user vyos authentication public-keys laptop type 'ssh-ed25519'
+set system name-server '192.168.10.5'
+set system option reboot-on-upgrade-failure '5'
+set system syslog local facility all level 'info'
+set system syslog local facility local7 level 'debug'