# OpenBao

Self-hosted secrets management + internal CA (SSH cert authority, dynamic secrets,
KV, PKI, transit) — the root-of-trust for AI agents and internal TLS.

## Design decisions

- **OpenBao, not Vault** — covers every engine we need (SSH CA, KV, dynamic Postgres
  creds, transit, PKI) under an OSI license. Vault only if you later need an engine
  OpenBao dropped in its fork.
- **On a dedicated host, NOT in k8s** — a secrets/CA root-of-trust must outlive the
  cluster it serves (no circular dependency) and sit outside its blast radius.
  `k8s-auth` still reaches back into the cluster for in-cluster workloads.
- **Tailscale + LAN only, never the Cloudflare tunnel** — machine clients authenticate
  with tokens/JWTs (not browser SSO), and the recovery service must not depend on an
  external SaaS to be reachable. Pods use the LAN IP; humans + the laptop agent use
  Tailscale.
- **Integrated Raft storage** — no external DB.

## Layout

```
ansible/
  create-bao-vm.yml            # create the host as a tiny libvirt VM (cloud-init)
  provision-openbao.yml        # deploy play (install + Raft + TLS + systemd)
  inventory/hosts.yml          # the bao host (LAN IP)
  group_vars/all/vars.yml      # version/checksum, addresses, firewall, auto-unseal
  group_vars/all/vault.example.yml
  roles/bao_vm/                # 1 vCPU / 1 GB / 10 GB Ubuntu VM on br0 (ZFS zvol)
  roles/openbao/               # install, TLS bootstrap cert, config, systemd unit
```

`provision-openbao.yml` **only deploys the daemon**. It does not initialize, unseal, or
configure any secrets engines/auth methods — that is a separate bootstrap play (later).

```
terraform/                     # OpenBao's API-level CONFIGURATION (see below)
  mounts.tf  pki.tf  ssh.tf  auth.tf  policies.tf
  auth-spire.tf                # SPIFFE JWT-SVID -> short-lived Bao tokens
  imports.tf                   # adopts the already-running instance into state
  policies/*.hcl               # policy bodies, kept diffable
```

## Ownership: Terraform vs Ansible

**Terraform manages OpenBao itself. Ansible manages the machine under it, plus the
few things Terraform must not own.**

| Terraform (`terraform/`) | Ansible (`ansible/`) |
|---|---|
| secrets engine mounts (`kv`, `pki`, `ssh-client-signer`) | the daemon, Raft, TLS files, systemd |
| PKI role, issuing/CRL URLs, cluster paths, **ACME** | `bao operator init` / unseal (manual, PGP-wrapped) |
| SSH signing role (`ai-agent`) | **PKI root CA + SSH CA signing key** |
| OIDC/Kubernetes auth mount、config 与 role | OIDC **client secret** (`auth/oidc/config`) |
| all policies | snapshot token + script + systemd timer |
|  | host-level CA trust distribution (`openbao_ssh_ca_trust`) |

Why the exceptions stay in Ansible:

- **Root-of-trust key material** (PKI root CA, SSH CA signing key). A Terraform
  resource treats drift as *regenerate*, which would silently invalidate every
  issued certificate and every `TrustedUserCAKeys` line on every host. Generated
  once, never reconciled.
- **The OIDC client secret.** Terraform cannot read it back from the API, so
  managing it would put the plaintext into `terraform.tfstate` **and** produce a
  permanent diff. It lives in `group_vars/all/vault.yml` (ansible-vault).
- **Snapshot token + timer.** A systemd unit on a host, and a token that would
  otherwise land in state.

The switch is `openbao_config_managed_by_terraform` (default `true`) in
`roles/openbao_bootstrap/defaults/main.yml`. It gates every task Terraform now
owns. Set it `false` only to bootstrap without Terraform — leaving it `false`
against a Terraform-managed instance makes the two overwrite each other on
alternate runs.

**Fresh-install order** (the one ordering constraint this split creates — CA
material needs its mount to exist first):

1. `ansible-playbook provision-openbao.yml` — daemon, TLS, systemd
2. `bao operator init` + unseal — manual, PGP-wrapped to the YubiKey
3. `terraform apply` — mounts, roles, policies, ACME
4. `ansible-playbook bootstrap-openbao.yml` — root CA, SSH CA key, OIDC secret, snapshots

For the **existing** instance, `terraform/imports.tf` adopts what is already
running; the plan must read *"N to import, 0 to destroy"*. A proposed destroy or
replace of `vault_mount.pki` would take the root CA with it — fix the HCL, never
apply through it.

Terraform authenticates from the ambient CLI session (`bao login -method=oidc`,
then `VAULT_ADDR`/`VAULT_TOKEN`), mirroring how `smtp-relay/terraform` uses
`az login`. It targets OpenBao via the **`hashicorp/vault`** provider, because the
native `openbao/openbao` provider is published only to the OpenTofu registry and
cannot be resolved by the HashiCorp `terraform` CLI.

Sandbox 集群使用独立的 `auth/kubernetes-sandbox`。其 API 地址、公开 Kubernetes CA、
ESO role 与只读 `kv/k8s/opensandbox-api` policy 全部由 Terraform 管理；CA 位于
`terraform/certs/sandbox-kubernetes-ca.crt`。集群重建并轮换 CA 后，先更新该文件并
apply，再让 Flux 恢复 ESO reconciliation。该 backend 不保存 reviewer JWT，而是使用
ESO 的短期登录 JWT 执行 TokenReview。该组资源已于 2026-09-18 apply，随后复验 plan
为 zero-diff。

## DNS

`bao.ad.ddupan.top` is an **internal-only** name — not a public Cloudflare record and
not behind the tunnel. Add the A record on the Samba DC (authoritative for
`ad.ddupan.top`), same pattern as the KMS record:

```bash
# on the DC, or with -U administrator%<pass>:
samba-tool dns add 192.168.10.5 ad.ddupan.top bao A 192.168.10.8 -U administrator
```

For remote/off-LAN clients, resolution rides your existing setup: the DC (`192.168.10.5`)
is reachable over the tailnet via the `192.168.10.0/24` subnet route, so point
`ad.ddupan.top` at it in Tailscale **split-DNS** (as you already do for other internal
names). LAN clients that use the DC as resolver get it directly.

## Create the host

A dedicated minimal VM — NOT on the DC or the k8s host — keeps the root-of-trust out of
the blast radius of what it protects. bao is tiny, so this costs almost nothing.

```bash
cd ansible
ansible-galaxy collection install -r requirements.yml
# set openbao_lan_ip in group_vars/all/vars.yml (also the VM's static IP)
ansible-playbook create-bao-vm.yml           # runs on the libvirt host (localhost)
```

## Deploy

```bash
# inventory/hosts.yml bao1 IP must match openbao_lan_ip
ansible-playbook provision-openbao.yml --ask-vault-pass
ansible-playbook provision-openbao.yml --tags verify      # smoke tests
```

After the first run the node is **uninitialized + sealed** — expected.

## Initialize (once, by hand — PGP-wrapped to your YubiKey)

Encrypt the unseal keys + root token to your YubiKey's GPG public key so they are
never printed in plaintext. Add a **backup offline GPG key** as a second share so a
lost/dead YubiKey doesn't make the data unrecoverable.

```bash
gpg --export <YUBIKEY_KEYID> | base64 > yubikey.pub.b64
gpg --export <BACKUP_KEYID>  | base64 > backup.pub.b64

BAO_ADDR=https://127.0.0.1:8200 BAO_SKIP_VERIFY=true \
  bao operator init -key-shares=2 -key-threshold=1 \
    -pgp-keys="yubikey.pub.b64,backup.pub.b64" \
    -root-token-pgp-key="yubikey.pub.b64"

# unseal (decrypt a share — touch the YubiKey):
echo "<encrypted-key-b64>" | base64 -d | gpg -dq | xargs bao operator unseal
```

With transit auto-unseal (`openbao_auto_unseal: true`) there are no unseal keys — use
`-recovery-pgp-keys` instead, and unsealing becomes automatic on restart.

## Bootstrap (engines, auth, policies)

Once initialized + unsealed, configure the bao side. Authenticate with the root token
(decrypt it, then export), and run the bootstrap play:

```bash
echo "<encrypted-root-token-b64>" | base64 -d | gpg -dq   # touch YubiKey
export BAO_TOKEN=<plaintext-root-token>
ansible-playbook bootstrap-openbao.yml --ask-vault-pass
ansible-playbook bootstrap-openbao.yml --tags verify
```

It enables **KV v2**, the **SSH CA** (`ssh-client-signer` + `ai-agent` role), **PKI**,
**OIDC auth** (the Authelia `openbao` client → `admin` policy for the
`vault-admins` AD group), the **ai-agent-ssh** policy, and a **Raft snapshot** timer.
Idempotent — safe to re-run. Selective runs via tags: `kv,ssh_ca,oidc,pki,k8s,policies,snapshots`.
Debug a step with `-e openbao_no_log=false`.

Off by default (need extra inputs, enable when ready):
- **Kubernetes auth** — `openbao_enable_k8s_auth: true` + reviewer JWT/CA (for in-cluster agents like hermes).
- **PKI listener cert** — `openbao_pki_replace_listener_cert: true` swaps the self-signed
  cert for a PKI-issued one (clients must then trust the PKI root CA; a Shamir node
  re-seals on the restart).

Afterwards, create a scoped admin path (OIDC login) and **revoke the root token**:
`bao token revoke -self`.

## Publicly-trusted TLS (ACME DNS-01)

Replace the self-signed listener cert with a **Let's Encrypt** cert so clients drop
`BAO_SKIP_VERIFY`. `ad.ddupan.top` is split-horizon — the DC serves it internally, but
it is **not delegated** in public DNS, so Cloudflare answers `*.ad.ddupan.top`
authoritatively. lego writes a **transient** `_acme-challenge.bao.ad.ddupan.top` TXT into
the Cloudflare `ddupan.top` zone, LE validates, and issues for the **internal** name —
no permanent record, no IP leak. Renewal reloads bao via **SIGHUP** (no restart/reseal).

By default it **reuses the Cloudflare token already managed for the tunnel**
(`cloudflared/terraform/terraform.tfvars` — it has `Zone:DNS:Edit` on ddupan.top), so
there's nothing new to store. To use a dedicated least-privilege token instead, set
`vault_openbao_cf_dns_token` in `vault.yml`.

```bash
ansible-playbook acme-openbao.yml --ask-vault-pass
```

Test against LE **staging** first to avoid rate limits: set `openbao_acme_server:
https://acme-staging-v02.api.letsencrypt.org/directory`, run, confirm, then clear it and
re-run for a real cert (`rm -rf /etc/openbao/acme` on the host between the two to reset).

Note: the LE cert covers the **hostname only** (`bao.ad.ddupan.top`), not the IP — so
after this, use `BAO_ADDR=https://bao.ad.ddupan.top:8200` (no skip-verify), not the IP.

## Using it

Kubernetes workload 不接收长期 `BAO_TOKEN`：它通过 SPIRE Workload API 获取
JWT-SVID，再经 `auth/jwt-spire/login` 换取短期、最小权限 token。完整接入流程、
manifest、exchange 脚本、安全要求和排障方法见
[`../../platform/spire/RUNBOOK.md`](../../platform/spire/RUNBOOK.md)。

```bash
# human: log in via Authelia (2FA)
bao login -method=oidc            # browser → auth.ddupan.top

# agent: mint a 5-min SSH cert for a target, then connect
ssh-keygen -t ed25519 -f /run/agent/id -N ''
bao write -field=signed_key ssh-client-signer/sign/ai-agent \
  public_key=@/run/agent/id.pub valid_principals=<node> > /run/agent/id-cert.pub
ssh -i /run/agent/id <user>@<node>
```

Each no-root target trusts the CA via one line in `~/.ssh/authorized_keys` (the play
prints it): `cert-authority,principals="<node>",restrict,pty <ca-pubkey>`.

## Operational notes

- **Restart re-seals** a Shamir node (needs manual unseal). Harmless before first init;
  automatic re-unseal with transit auto-unseal.
- **Back up** Raft snapshots off-box once initialized: `bao operator raft snapshot save`
  (belongs in the bootstrap play — it needs a token).
- The listener uses a **self-signed bootstrap cert**; replace it with a cert issued by
  OpenBao's own PKI engine in the bootstrap play.