Establish clean homelab infrastructure baseline
lint / yaml (push) Has been cancelled
lint / ansible (push) Has been cancelled
lint / terraform (push) Has been cancelled

Reorganize the brownfield repository, remove retired and generated artifacts, harden ignore rules, and record the GitOps/IaC redesign.
This commit is contained in:
2026-09-09 16:47:20 +00:00
commit 88a02ababa
418 changed files with 50579 additions and 0 deletions
+4
View File
@@ -0,0 +1,4 @@
# Never commit the real credentials or minted tokens
secret.yaml
tokens/
.noreply-password
+136
View File
@@ -0,0 +1,136 @@
# SMTP relay (Postfix + sasl-xoauth2 → Microsoft 365)
One internal SMTP endpoint that in-cluster apps use for outbound mail. It authenticates
to Exchange Online with **OAuth2 (XOAUTH2)** via [`mauroreggio/postfix-365`], which bundles
[`sasl-xoauth2`] — tokens are refreshed **inside the SASL layer**, no sidecar/cron. Basic-auth
SMTP (app passwords) is being retired by Microsoft; this is the modern replacement.
```
Authelia / Gitea / … ──plain SMTP :25 (in-cluster, no auth)──▶ smtp-relay ──587 STARTTLS + XOAUTH2──▶ smtp.office365.com
```
Reach it at: `smtp-relay.smtp-relay.svc.cluster.local:25`. Sends **as** `[email protected]`.
[`mauroreggio/postfix-365`]: https://github.com/mauroreggio/postfix-365
[`sasl-xoauth2`]: https://github.com/tarickb/sasl-xoauth2
---
## Prerequisites (Microsoft 365 / Entra — mostly interactive, one-time)
1. **`ddupan.top` is a verified domain** in the tenant (Admin center → Settings → Domains,
Status = Healthy). If not, add it and complete the TXT verification (DNS is Cloudflare —
can be done in `../../../infrastructure/cloudflared/terraform/`).
2. **Mailbox `[email protected]`** exists with an **Exchange Online license**.
3. **Authenticated SMTP enabled** on it: Admin center → Users → that user → Mail →
*Manage email apps* → tick **Authenticated SMTP**. (OAuth won't work without this.)
## Step 1 — Entra app registration (as code)
```bash
cd terraform
az login # account that can create app regs AND grant admin consent (Global Admin)
terraform init
terraform apply # creates the app, delegated Graph SMTP.Send + admin consent, a client secret
```
Grab the three values for the k8s secret:
```bash
terraform output -raw client_id
terraform output -raw tenant_id
```
> This is a **PUBLIC** client (device-code flow) — there is **no client secret**. Leave
> `CLIENT_SECRET` empty in `secret.yaml`. Presenting a secret makes Entra reject the token
> refresh with `AADSTS700025 "Client is public..."`.
## Step 2 — Deploy the relay
```bash
cd ..
cp secret.example.yaml secret.yaml # paste CLIENT_ID / CLIENT_SECRET / TENANT_ID
kubectl apply -f namespace.yaml
kubectl apply -f secret.yaml -f pvc.yaml
kubectl apply -f deployment.yaml -f service.yaml
kubectl -n smtp-relay rollout status deploy/smtp-relay
```
At this point Postfix runs but has **no token yet**, so relaying fails until step 3.
## Step 3 — Bootstrap the token (one-time, interactive device-code)
The image's `sasl-xoauth2-tool` needs the `msal` Python module, which isn't bundled.
Install it ephemerally (only needed for this one mint; the C++ SASL plugin refreshes
without it):
```bash
POD=$(kubectl get pod -n smtp-relay -l app=smtp-relay -o name | head -1 | cut -d/ -f2)
kubectl exec -n smtp-relay $POD -- sh -c 'python3 -m ensurepip >/dev/null 2>&1; python3 -m pip install -q msal'
```
Mint the token (env vars come from the pod's secret; `CLIENT_SECRET` is empty → public flow):
```bash
kubectl exec -n smtp-relay -it deploy/smtp-relay -- sh -c \
'sasl-xoauth2-tool get-token outlook /etc/tokens/[email protected] \
--client-id="$CLIENT_ID" --tenant="$TENANT_ID" --client-secret="$CLIENT_SECRET" --use-device-flow'
```
It prints a URL + code — open <https://microsoft.com/devicelogin> and **sign in as the SENDER
mailbox `[email protected]`** (NOT yourself/the admin — a token minted for the wrong user gives
`535 5.7.3`). If prompted for a client secret, press Enter. Then fix ownership so Postfix can
read/rewrite it:
```bash
kubectl exec -n smtp-relay $POD -- chown postfix:postfix /etc/tokens/[email protected]
```
Verify identity if unsure: decode the token and check `upn` == `[email protected]`. The token
persists on the PVC; sasl-xoauth2 refreshes it automatically thereafter.
## Step 4 — Test
```bash
kubectl exec -n smtp-relay $POD -- sh -c \
'echo "Subject: relay test\n\nhello" | sendmail -f [email protected] [email protected]'
kubectl exec -n smtp-relay $POD -- tail -n 40 /var/log/maillog # look for "status=sent"
```
## Step 5 — Point apps at it
- **Authelia** — replace the filesystem notifier with SMTP in `../authelia/values.yaml`:
address `smtp://smtp-relay.smtp-relay.svc.cluster.local:25`, sender `[email protected]`,
`disable_require_tls: true` (plain in-cluster hop). 2FA enrollment codes then go to real email.
- Any future app: same address, **From = `[email protected]`** (O365 rejects other senders
with `5.7.60` unless a send-as alias is configured in Exchange).
## Deliverability (keep mail out of Junk)
- **SPF / MX / DMARC** for `ddupan.top` already exist (M365 domain setup).
- **DKIM**: ✅ **enabled 2026-07-28** (`Enabled: True`, `Status: Valid`). CNAMEs are in
`../../infrastructure/cloudflared/terraform/` (`selector1/2._domainkey`); signing was turned on with
`scripts/enable-dkim.ps1` then `scripts/enable-dkim-finish.ps1`.
- ⚠️ **The CNAME target is NXDOMAIN until signing is enabled.** Microsoft creates the
tenant host (`<tenant>.d-v1.dkim.mail.microsoft`) only at enable time, so a correct
CNAME looks broken beforehand and `Get-DkimSigningConfig` reports `CnameMissing`.
**Do not go hunting for the "real" CNAME value** — run step 2 and re-check DNS.
- ⚠️ **Both scripts deadlock if run via `!` or with output redirected to a file** — the
device code never becomes visible. Run under a PTY:
`DOTNET_SYSTEM_NET_DISABLEIPV6=1 script -qfc "pwsh -NoProfile -File scripts/enable-dkim.ps1" /tmp/dkim.log`
The `DISABLEIPV6` is required on the laptop — see the IPv6 trap in the root `CLAUDE.md`;
without it `Connect-ExchangeOnline` hangs in `SYN-SENT` with no output at all.
- **Still soft**: SPF is `~all` and DMARC is `p=none`. Harden to `-all` / `p=quarantine`
once aggregate reports confirm DKIM passes — not before, or you quarantine your own mail.
## Notes / gotchas
- **PUBLIC client, no secret** — see Step 1. `CLIENT_SECRET` stays empty.
- **Sign in as `noreply@` (the sender), not the admin**, during the Step 3 device login — a
token minted for the wrong identity fails with `535 5.7.3`.
- **`msal` is ephemeral** — reinstall it in the pod (Step 3) before any re-mint; it's gone after
a restart but only the one-time mint needs it.
- **Token PVC is writable state, not in Git.** The refresh token rotates; it lives only on
the PVC. Back it up if you want to avoid re-running step 3.
- **Refresh-token longevity**: Azure AD refresh tokens renew on use but can expire under
Conditional Access / long idle — if relaying suddenly fails auth, re-run step 3.
+81
View File
@@ -0,0 +1,81 @@
apiVersion: apps/v1
kind: Deployment
metadata:
name: smtp-relay
namespace: smtp-relay
labels:
app: smtp-relay
spec:
replicas: 1
strategy:
type: Recreate # single writer on the token PVC (RWO)
selector:
matchLabels:
app: smtp-relay
template:
metadata:
labels:
app: smtp-relay
spec:
containers:
- name: postfix
# Postfix + sasl-xoauth2 (OAuth2/XOAUTH2 to M365). Refreshes tokens in the
# SASL layer — no sidecar. https://github.com/mauroreggio/postfix-365
image: ghcr.io/mauroreggio/postfix-365:1.0.0
envFrom:
- secretRef:
name: smtp-relay-secret # CLIENT_ID, CLIENT_SECRET, TENANT_ID
env:
- name: TIMEZONE
value: 'Asia/Shanghai'
- name: HOSTNAME
value: 'smtp-relay.ddupan.top' # HELO name
- name: DOMAIN_NAME
value: 'ddupan.top'
# Submitters trusted without SMTP AUTH. k3s pod + service CIDRs, plus the
# three Proxmox nodes by /32 so they can relay system mail (PVE alerts,
# smartd, cron) to M365 — they have no other way off a residential IP.
# Deliberately /32s, NOT 192.168.10.0/24: everything else on the LAN still
# hits `defer_unauth_destination`, so this stays a closed relay.
# Exposed to those nodes via service-lan.yaml (LoadBalancer :25).
- name: MY_NETWORK
value: '10.42.0.0/16, 10.43.0.0/16, 192.168.10.4/32, 192.168.10.7/32, 192.168.10.9/32'
- name: DISABLE_SMTP_AUTH_ON_PORT_25
value: 'true'
- name: MESSAGE_SIZE_LIMIT
value: '26214400' # 25 MiB
# The M365 mailbox we authenticate + send AS (device-code refresh token
# lives at /etc/tokens/<AUTH_USER> on the PVC).
- name: AUTH_USER
value: '[email protected]'
- name: RELAY_HOST
value: 'smtp.office365.com'
- name: RELAY_HOST_PORT
value: '587'
ports:
- name: smtp
containerPort: 25
volumeMounts:
# Writable + persistent: sasl-xoauth2 rewrites the token file on refresh.
- name: tokens
mountPath: /etc/tokens
readinessProbe:
tcpSocket:
port: 25
initialDelaySeconds: 10
periodSeconds: 15
livenessProbe:
tcpSocket:
port: 25
initialDelaySeconds: 30
periodSeconds: 30
resources:
requests:
cpu: 10m
memory: 64Mi
limits:
memory: 256Mi
volumes:
- name: tokens
persistentVolumeClaim:
claimName: smtp-relay-tokens
+4
View File
@@ -0,0 +1,4 @@
apiVersion: v1
kind: Namespace
metadata:
name: smtp-relay
+15
View File
@@ -0,0 +1,15 @@
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: smtp-relay-tokens
namespace: smtp-relay
labels:
app: smtp-relay
spec:
# Holds /etc/tokens/<AUTH_USER> — the OAuth refresh/access token that sasl-xoauth2
# rewrites as it refreshes. Must persist across restarts, so NOT a Secret mount.
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 64Mi
@@ -0,0 +1,18 @@
# DKIM step 2 of 2: enable signing, once the CNAME targets from enable-dkim.ps1 are
# published and resolving. Validates the CNAMEs and turns on DKIM signing.
#
# ! pwsh ~/services/apps/smtp-relay/scripts/enable-dkim-finish.ps1
param(
[string]$Domain = 'ddupan.top'
)
Import-Module ExchangeOnlineManagement
Connect-ExchangeOnline -Device -ShowBanner:$false
try {
Set-DkimSigningConfig -Identity $Domain -Enabled $true -ErrorAction Stop
Write-Host "DKIM enabled for $Domain."
} catch {
Write-Host "Enable failed: $($_.Exception.Message)"
Write-Host "If it mentions CNAME records, they aren't visible to Exchange yet — wait and re-run."
}
Get-DkimSigningConfig -Identity $Domain | Format-List Name, Enabled, Status, Selector1CNAME, Selector2CNAME
Disconnect-ExchangeOnline -Confirm:$false
+31
View File
@@ -0,0 +1,31 @@
# DKIM step 1 of 2: create the signing config (disabled) and PRINT the exact CNAME
# targets. The 2025 CNAME format includes a per-tenant character only Exchange knows,
# so we must read Selector1CNAME/Selector2CNAME from here, publish them, THEN enable.
#
# ! pwsh ~/services/apps/smtp-relay/scripts/enable-dkim.ps1
# Sign in as a tenant admin. Paste the CNAME values back so DNS can be updated.
param(
[string]$Domain = 'ddupan.top'
)
Import-Module ExchangeOnlineManagement
Connect-ExchangeOnline -Device -ShowBanner:$false
# Create directly (disabled). Don't pre-check with Get — on a missing domain it only
# WARNS (not errors), which defeats try/catch. Catch the "already exists" case instead.
try {
New-DkimSigningConfig -DomainName $Domain -KeySize 2048 -Enabled $false -ErrorAction Stop | Out-Null
Write-Host "Created DKIM config for $Domain (disabled)."
} catch {
if ("$($_.Exception.Message)" -match 'already exist') {
Write-Host "DKIM config already exists — continuing."
} else {
Write-Host "New-DkimSigningConfig failed: $($_.Exception.Message)"
Disconnect-ExchangeOnline -Confirm:$false
return
}
}
Write-Host "`n== PUBLISH THESE EXACT CNAME TARGETS (paste them back) =="
Get-DkimSigningConfig -Identity $Domain | Format-List Name, Enabled, Status, Selector1CNAME, Selector2CNAME
Write-Host "After the CNAMEs are updated + resolving, run scripts/enable-dkim-finish.ps1 to enable."
Disconnect-ExchangeOnline -Confirm:$false
@@ -0,0 +1,19 @@
# Enables Authenticated SMTP (SMTP AUTH) on [email protected] so OAuth SMTP works.
# Run it yourself (interactive admin device-login):
# ! pwsh ~/services/apps/smtp-relay/scripts/enable-smtp-auth.ps1
# At the prompt, open https://microsoft.com/devicelogin and sign in as a tenant ADMIN.
param(
[string]$Mailbox = '[email protected]'
)
Import-Module ExchangeOnlineManagement
Connect-ExchangeOnline -Device -ShowBanner:$false
$mbx = Get-CASMailbox -Identity $Mailbox -ErrorAction SilentlyContinue
if (-not $mbx) {
Write-Host "Mailbox '$Mailbox' isn't provisioned yet (licensing can take a few minutes). Wait and re-run."
} else {
Set-CASMailbox -Identity $Mailbox -SmtpClientAuthenticationDisabled:$false
Write-Host "== Authenticated SMTP status =="
Get-CASMailbox -Identity $Mailbox | Format-List Name, SmtpClientAuthenticationDisabled
}
Disconnect-ExchangeOnline -Confirm:$false
+19
View File
@@ -0,0 +1,19 @@
# Copy to secret.yaml (gitignored) and fill in — or let the Terraform in ./terraform/
# emit these values (it creates the Entra app registration + client secret).
#
# CLIENT_ID = Application (client) ID of the Entra app registration
# CLIENT_SECRET = a client secret VALUE for that app (used by sasl-xoauth2 to refresh)
# TENANT_ID = your Entra tenant (directory) ID
#
# NOTE: these three are NOT the whole story — the relay also needs a device-code
# refresh token minted once into the PVC (see README "Bootstrap the token").
apiVersion: v1
kind: Secret
metadata:
name: smtp-relay-secret
namespace: smtp-relay
type: Opaque
stringData:
CLIENT_ID: 'CHANGE-ME'
CLIENT_SECRET: 'CHANGE-ME'
TENANT_ID: 'CHANGE-ME'
+40
View File
@@ -0,0 +1,40 @@
# LAN-facing SMTP endpoint for hosts OUTSIDE the k3s cluster — specifically the
# three Proxmox nodes (pve1 .4, pve2 .7, pve3 .9), which relay their system mail
# (PVE notifications, smartd, cron) through here to M365.
#
# Kept SEPARATE from service.yaml on purpose: that ClusterIP service is what
# Authelia and Gitea address by DNS name (smtp-relay.smtp-relay.svc.cluster.local),
# and it must not change shape.
#
# k3s servicelb (klipper) host-binds :25 on the laptop (192.168.10.127).
#
# CRITICAL — why loadBalancerSourceRanges is the ONLY real access control here:
# klipper SNATs incoming connections, so the relay sees every LAN client as
# `_gateway[10.42.0.1]` rather than its true address. 10.42.0.1 is inside the
# pod CIDR that MY_NETWORK already trusts, which means postfix's IP-based
# `permit_mynetworks` CANNOT distinguish a Proxmox node from any other LAN host —
# per-node /32 entries in MY_NETWORK are decorative. Without the source ranges
# below this service is an OPEN RELAY to the whole LAN (verified: an untrusted
# host got a 220 banner and would have been permitted to relay).
# So: restrict at the LB. Do not remove this block.
apiVersion: v1
kind: Service
metadata:
name: smtp-relay-lan
namespace: smtp-relay
labels:
app: smtp-relay
spec:
type: LoadBalancer
selector:
app: smtp-relay
# Only the three Proxmox nodes may even open a TCP connection to :25.
loadBalancerSourceRanges:
- 192.168.10.4/32 # pve1
- 192.168.10.7/32 # pve2
- 192.168.10.9/32 # pve3
ports:
- name: smtp
port: 25
targetPort: 25
protocol: TCP
+15
View File
@@ -0,0 +1,15 @@
apiVersion: v1
kind: Service
metadata:
name: smtp-relay
namespace: smtp-relay
labels:
app: smtp-relay
spec:
selector:
app: smtp-relay
ports:
# In-cluster apps send here: smtp-relay.smtp-relay.svc.cluster.local:25
- name: smtp
port: 25
targetPort: 25
+5
View File
@@ -0,0 +1,5 @@
*.tfstate
*.tfstate.*
.terraform/
terraform.tfvars
crash.log
+22
View File
@@ -0,0 +1,22 @@
# This file is maintained automatically by "terraform init".
# Manual edits may be lost in future updates.
provider "registry.terraform.io/hashicorp/azuread" {
version = "3.9.0"
constraints = "~> 3.0"
hashes = [
"h1:+ZknnMPMLJ1dIVqxto9ZWoakX4ljsek5cmajhUfEwN4=",
"zh:1c3e89cf19118fc07d7b04257251fc9897e722c16e0a0df7b07fcd261f8c12e7",
"zh:39b11a075e4baa4f6ed5c72a8427013d50f43eecc1a7603b73bccf80f952f758",
"zh:41484c196c943b39411f561e70a308bd2a71da18155bfec7381ba0bd61361d34",
"zh:42068e5da223494beea5f7fcb9057c308cbfa92f96e53c50083e2639216479d8",
"zh:464d7da44682443a4b64bfdaf3d0eb53011c6e1471f244f6354c4d5bca18edce",
"zh:49f597ea3fac39931ff91e55afd5b5cc91e449920a03716f82509d588aaab708",
"zh:6092c376accfc50b555b7a0cd56b76c09abc3d65ac9dd5069063d6f9f1e76d3b",
"zh:65326a9f3ac0783c16e05c16422d191f0a926b8d021fd5303c1fdf8dc42f16e9",
"zh:784214ed809347d74562bb38194c0cef57831eaa621ba3b7cdd3fe7a7a76d844",
"zh:b4233f9bc791adc7d6643507fa5b47360a21125763a072d953586151cacb65f9",
"zh:c4ecdd995ff99b7e362e087c45f080816bcf097da5be257c87b912210e45dd3e",
"zh:f0122771f71cb98248e70cdd6c2ccd3bffb34e79d19897fa28b785c86b2312ed",
]
}
+40
View File
@@ -0,0 +1,40 @@
# Remote state in SeaweedFS S3, on the LAN.
#
# WHY remote at all: local state means the only copy lives on this laptop, which is
# also the k3s node, the NFS server and the libvirt host — i.e. the single point of
# failure. It also cannot be locked, so two concurrent applies silently corrupt it.
#
# WHY s3.ad.ddupan.top and NOT obj.ddupan.top: the public name resolves to
# Cloudflare and hairpins through the WAN. On 2026-07-28 that path was blackholed
# for hours by a dead VPN tunnel. State must be reachable when the WAN is not —
# it is what you need DURING an incident. See ../../seaweedfs/httproute-s3.yaml.
#
# CREDENTIALS are not in this file. Export them before running terraform:
# export AWS_ACCESS_KEY_ID=$(bao kv get -field=... kv/k8s/seaweedfs-s3) # see README
# export AWS_SECRET_ACCESS_KEY=...
# The `terraform` S3 identity is scoped to this bucket only — it deliberately
# cannot create buckets or read anything else in the store.
terraform {
backend "s3" {
bucket = "tfstate"
key = "smtp-relay/terraform.tfstate"
endpoints = {
s3 = "https://s3.ad.ddupan.top"
}
# SeaweedFS is not AWS: it has no regions, no IAM, no metadata service and no
# account IDs, so every AWS-specific validation has to be skipped or the
# provider fails before it ever talks to the endpoint.
region = "us-east-1"
use_path_style = true
skip_credentials_validation = true
skip_metadata_api_check = true
skip_region_validation = true
skip_requesting_account_id = true
# Native S3 locking (Terraform >= 1.10; this repo runs 1.15). Writes a
# .tflock object alongside the state — no DynamoDB table needed.
use_lockfile = true
}
}
+50
View File
@@ -0,0 +1,50 @@
# Entra app registration for the Postfix sasl-xoauth2 relay.
# Delegated Microsoft Graph SMTP.Send + admin consent + a client secret. The relay
# still needs a one-time device-code login to mint the refresh token (see ../README.md).
data "azuread_client_config" "current" {}
# Microsoft Graph well-known IDs, so we don't hardcode the SMTP.Send permission UUID.
data "azuread_application_published_app_ids" "well_known" {}
resource "azuread_service_principal" "msgraph" {
client_id = data.azuread_application_published_app_ids.well_known.result["MicrosoftGraph"]
use_existing = true
}
resource "azuread_application" "smtp_relay" {
display_name = var.app_display_name
sign_in_audience = "AzureADMyOrg"
# Enables "Allow public client flows" so the device-code flow works, while we still
# keep a client secret for confidential refresh.
fallback_public_client_enabled = true
public_client {
redirect_uris = ["https://login.microsoftonline.com/common/oauth2/nativeclient"]
}
required_resource_access {
resource_app_id = data.azuread_application_published_app_ids.well_known.result["MicrosoftGraph"]
resource_access {
id = azuread_service_principal.msgraph.oauth2_permission_scope_ids["SMTP.Send"]
type = "Scope" # delegated
}
}
}
resource "azuread_service_principal" "smtp_relay" {
client_id = azuread_application.smtp_relay.client_id
}
# NOTE: no client secret. This is a PUBLIC client (device-code delegated flow); the
# refresh token is the credential. Presenting a secret makes Entra reject the refresh
# with AADSTS700025 ("Client is public..."). CLIENT_SECRET in the k8s secret is empty.
# Org-wide admin consent for the delegated SMTP.Send scope (no per-user consent prompt).
resource "azuread_service_principal_delegated_permission_grant" "smtp_send" {
service_principal_object_id = azuread_service_principal.smtp_relay.object_id
resource_service_principal_object_id = azuread_service_principal.msgraph.object_id
claim_values = ["SMTP.Send"]
}
+15
View File
@@ -0,0 +1,15 @@
# Feed these into smtp-relay/secret.yaml (CLIENT_ID / CLIENT_SECRET / TENANT_ID).
# terraform output -raw client_id
# terraform output -raw tenant_id
# terraform output -raw client_secret
output "client_id" {
value = azuread_application.smtp_relay.client_id
description = "CLIENT_ID for the relay secret."
}
output "tenant_id" {
value = data.azuread_client_config.current.tenant_id
description = "TENANT_ID for the relay secret."
}
+6
View File
@@ -0,0 +1,6 @@
variable "app_display_name" {
type = string
default = "smtp-relay-sasl-xoauth2"
description = "Display name of the Entra app registration for the SMTP relay."
}
+14
View File
@@ -0,0 +1,14 @@
terraform {
required_version = ">= 1.5"
required_providers {
azuread = {
source = "hashicorp/azuread"
version = "~> 3.0"
}
}
}
# Authenticates via Azure CLI by default: run `az login` as an account that can
# create app registrations AND grant admin consent (Global Admin / Privileged Role
# Admin) before `terraform apply`.
provider "azuread" {}