homelab-infra/platform/observability/README.md

observability

Metrics, logs, and traces for the cluster — one VictoriaMetrics-ecosystem stack, operator-driven, running in the monitoring namespace. Replaces the standalone Docker Compose stack in ../../apps/victoriametrics/.

Status — DEPLOYED (2026-07-10)

Live and verified in the monitoring namespace (operator chart 0.66.2): metrics (data queryable), logs (pods ingesting), traces (VTSingle CRD, verified via Jaeger API), Grafana (3 datasources healthy, VM dashboard loaded, Tailscale ingress grafana.tail7e769.ts.net, Authelia OIDC client registered). All PVCs on localpv-zfs-ceph. Note the vmagent hot-reload RBAC fix (metrics/reload-rbac.yaml, see metrics/README-reload.md) and that VL/VT CRDs are served under .../v1.

Still host-side (not yet done): deploy docker-hosts/compose.yaml on the Docker hosts and logs/vlogs-ingress.yaml to push their logs.

Decision

Pillar	Choice	Why
Metrics	VictoriaMetrics (VMSingle)	already in use; keep it
Logs	VictoriaLogs (VLSingle)	same vendor/operator, tiny footprint, LogsQL
Traces	VictoriaTraces + OTel Collector	no extra storage deps, OTLP in, Jaeger query API
Collect	vmagent (metrics), vlagent (logs), OTel Collector (traces)	native to each backend
Visualize	Grafana, one instance, 3 datasources	single pane
Manage	victoria-metrics-operator	VMSingle/VMAgent/VMAlert/VMAlertmanager/VMRule and VLSingle as CRDs
Expose	Tailscale ingress (private) + Authelia OIDC	admin tool: private + SSO

Grafana's Prometheus-operator converter is on, so any chart shipping a ServiceMonitor/PodMonitor/PrometheusRule is scraped automatically.

Architecture

 Grafana (grafana.<tailnet>.ts.net, Authelia OIDC)
   ├── VictoriaMetrics datasource → VMSingle  ← vmagent (k8s SD, node-exporter, kubelet/cAdvisor, KSM)
   ├── VictoriaLogs   datasource → VLSingle   ← vlagent DaemonSet (pod logs)
   └── Jaeger         datasource → VictoriaTraces ← OTel Collector (OTLP 4317/4318) ← apps
 VMAlert (VMRules) → VMAlertmanager

Layout

observability/
  namespace.yaml            monitoring namespace
  operator/                 victoria-metrics-operator (Helm)
  metrics/
    vmsingle|vmagent|vmalert|vmalertmanager.yaml   core CRs
    rules/                  VMRules ported from ../../apps/victoriametrics/rules
    exporters/              node-exporter (+VMNodeScrape), kube-state-metrics (Helm)
    scrapes/               kubelet + cAdvisor VMNodeScrapes
  logs/                     VLSingle CR + victoria-logs-collector (Helm)
  traces/                   victoria-traces-single (Helm) + OTel Collector
  grafana/                  Grafana (Helm) values, OIDC secret, install script
  docker-hosts/             per-Docker-host sidecar (cAdvisor + node-exporter + Vector)

Docker hosts

The Compose services on Docker hosts (../../apps/vlmcsd, ../../apps/ps3netsrv, ../../apps/netboot, ... and the legacy ../../apps/victoriametrics stack) are monitored too:

• Metrics — pull, nothing exposed. Each host runs docker-hosts/compose.yaml (cAdvisor :8080 + node-exporter :9100). The cluster's vmagent scrapes their LAN IPs via metrics/scrapes/docker-hosts.yaml (VMStaticScrape). VMSingle stays private.
• Logs — push over the tailnet. A Vector container per host tails the Docker socket and ships to VictoriaLogs' Elasticsearch-bulk endpoint, exposed privately as vlogs.<tailnet>.ts.net by logs/vlogs-ingress.yaml.

Deploy:

# cluster side (once): expose VictoriaLogs ingest + register the scrape
kubectl apply -f logs/vlogs-ingress.yaml -f metrics/scrapes/docker-hosts.yaml

# on each Docker host:
cd docker-hosts
cp .env.example .env        # set HOST_LABEL and confirm VLOGS_ENDPOINT (tailnet FQDN)
docker compose up -d
# then add the host's IP to metrics/scrapes/docker-hosts.yaml and re-apply

Note: cAdvisor uses host port 8080 — remap it (and the scrape target) if taken.

Deploy order

# 1. Operator (installs CRDs) + namespace
cd operator && ./helm.sh && cd ..

# 2. Metrics: core CRs, rules, scrapes, exporters
kubectl apply -f metrics/vmsingle.yaml -f metrics/vmagent.yaml \
              -f metrics/vmalert.yaml -f metrics/vmalertmanager.yaml
kubectl apply -f metrics/rules/ -f metrics/scrapes/
kubectl apply -f metrics/exporters/node-exporter.yaml
metrics/exporters/kube-state-metrics.sh    # needs Prometheus CRDs; see the script

# 3. Logs
cd logs && ./helm.sh && cd ..

# 4. Traces
cd traces && ./helm.sh && cd ..

# 5. Grafana (create the OIDC secret first — see grafana/oidc-secret.yaml)
cd grafana && ./helm.sh && cd ..

Before it works — required edits

• OIDC secret pair. Generate once: authelia crypto hash generate pbkdf2 --variant sha512 --random --random.length 72
  • plaintext → the grafana-oidc Secret (create imperatively, don't commit)
  • hash → authelia/values.yaml grafana client client_secret, then helm upgrade Authelia.
• Tailnet FQDN. This scaffold assumes grafana.tail7e769.ts.net (your existing tailnet, per seaweedfs-admin). If the operator assigns a different name, update it in grafana/values.yaml (root_url) and the Authelia grafana redirect_uris.
• Chart value names. logs/collector-values.yaml and traces/traces-values.yaml note a helm show values check — verify the keys against the installed chart versions.
• Pin chart + image versions after first install (all helm.sh scripts note this; otel-collector.yaml pins the collector image — bump to current).

Follow-ups

• Dashboards: the VictoriaMetrics single-node board is migrated to grafana/dashboards/victoriametrics.json; grafana/dashboards/apply.sh loads it via the Grafana sidecar. (The old vmagent.json/vmalert.json were empty dirs Docker auto-created, not real boards.) Add the canonical ones by dropping their JSON in grafana/dashboards/ and re-running the script — vmagent (gnetId 12683), vmalert (14950), node-exporter (1860), cAdvisor now that those exporters exist.
• Wire real Alertmanager receivers in metrics/vmalertmanager.yaml (email via ../../apps/smtp-relay/), replacing the ported blackhole.
• Once parity is confirmed, decommission the Compose stack: ../../apps/victoriametrics/ (docker compose down), and retire that folder.
• Add app instrumentation: point OTEL_EXPORTER_OTLP_ENDPOINT at otel-collector.monitoring.svc:4317.