homelab-infra/platform/observability

observability

Metrics, logs, and traces for the cluster — one VictoriaMetrics-ecosystem stack, operator-driven, running in the monitoring namespace. Replaces the standalone Docker Compose stack in ../../apps/victoriametrics/.

Grafana 当前入口（2026-09-16）

统一使用 https://grafana.ad.ddupan.top，A 记录指向 192.168.10.127，由共享 Envoy Gateway 的 https listener 与 *.ad.ddupan.top 证书提供 TLS。 Grafana root_url 和 Authelia 回调均使用此域名；旧专用 Tailscale Ingress 已停用。 远程客户端仍需有到 LAN 的路由及已配置的内网 DNS 转发。

内存看板：/d/homelab-memory；采集配置及 AppArmor 规则见 主机与进程内存采集。

Grafana 由 Flux 管理；修改 values 后通过 Git 合并触发 HelmRelease，避免现场 Helm 修改被漂移检测回滚。Authelia 当前不由 Flux 管理，需单独 Helm upgrade 并先结构比较 live values。迁移时先添加 DNS 和回调，再切换 Grafana。Recreate 策略会短暂中断访问， 但保留原 PVC、用户、看板和数据源。回滚需同时恢复 root_url、回调和 Ingress 配置。

初始部署记录（2026-07-10）

Live and verified in the monitoring namespace (operator chart 0.66.2): metrics (data queryable), logs (pods ingesting), traces (VTSingle CRD, verified via Jaeger API), Grafana (3 datasources healthy, VM dashboard loaded, Tailscale ingress grafana.tail7e769.ts.net, Authelia OIDC client registered). All PVCs on localpv-zfs-ceph. Note the vmagent hot-reload RBAC fix (metrics/reload-rbac.yaml, see metrics/README-reload.md) and that VL/VT CRDs are served under .../v1.

Flux 接管状态

可观测性栈使用一个 observability Flux Kustomization，按依赖顺序逐步加入 Operator、 Metrics、Logs、Traces 和 Grafana，避免多个 reconciler 争夺共享 namespace、Helm source 或 VictoriaMetrics CRD。brownfield 迁移期间保持 prune: false。

VictoriaMetrics Operator 已固定 chart 0.66.2 / app v0.73.1 完成分阶段 Flux HelmRelease 接管，保存 values 与 operator/values.yaml 一致。阶段一确认 Helm revision 保持为 2、Operator Pod UID 未变、24 个 CRD 及 6 个核心 VM/VL/VT CR 未变化；随后 通过独立 PR 解除暂停。

其余已部署资源也已统一完成接管：Metrics CR、rules、scrapes、node-exporter，Logs 的 VLSingle 与固定 0.3.6 的 Collector，以及 Traces 的 VTSingle、OTel Collector，最后是 固定 10.5.15 的 Grafana 与 dashboard ConfigMap。阶段一确认 11 个 Pod UID、4 个 PVC、 6 个核心 CR、17 个 rule/scrape 对象及 dashboard 哈希均未变化，Collector 和 Grafana Helm revision 均保持为 1；随后在同一独立 PR 中解除两个 HelmRelease 的暂停。

首次 Grafana Helm reconcile 暴露了本地 ZFS RWO 卷限制：chart 默认 RollingUpdate 会先创建新 Pod，但 CSI 拒绝在旧 Pod 仍挂载 PVC 时再次 mount，使 upgrade 卡在 pending-upgrade。grafana/values.yaml 因此显式使用 deploymentStrategy.type: Recreate。 Grafana 升级会有一次短暂停机，但旧 Pod 会先退出，新 Pod 才挂载同一 PVC；不要改回 RollingUpdate，除非存储改为真正支持并发挂载的 RWX。

以下内容明确不属于本批接管：尚未部署的 kube-state-metrics、vlogs-ingress.yaml，以及 仅作 standalone chart 参考的 traces-values.yaml。grafana/oidc-secret.yaml 是不含真实 值的占位模板；live grafana-oidc Secret 继续只被 HelmRelease 引用，尚未由 OpenBao/ESO 纳管。

Still host-side (not yet done): deploy docker-hosts/compose.yaml on the Docker hosts and logs/vlogs-ingress.yaml to push their logs.

Decision

Pillar	Choice	Why
Metrics	VictoriaMetrics (VMSingle)	already in use; keep it
Logs	VictoriaLogs (VLSingle)	same vendor/operator, tiny footprint, LogsQL
Traces	VictoriaTraces + OTel Collector	no extra storage deps, OTLP in, Jaeger query API
Collect	vmagent (metrics), vlagent (logs), OTel Collector (traces)	native to each backend
Visualize	Grafana, one instance, 3 datasources	single pane
Manage	victoria-metrics-operator	VMSingle/VMAgent/VMAlert/VMAlertmanager/VMRule and VLSingle as CRDs
Expose	Tailscale ingress (private) + Authelia OIDC	admin tool: private + SSO

VictoriaMetrics Operator 的 Prometheus converter 已启用；官方 prometheus-operator-crds chart 由 operator/ 一并管理。因此应用 chart 可以原生 声明 ServiceMonitor、PodMonitor 或 PrometheusRule，再由 converter 转换为对应 VM 资源，不需要每个应用额外维护一份 VM*Scrape。

Architecture

 Grafana (grafana.<tailnet>.ts.net, Authelia OIDC)
   ├── VictoriaMetrics datasource → VMSingle  ← vmagent (k8s SD, node-exporter, kubelet/cAdvisor, KSM)
   ├── VictoriaLogs   datasource → VLSingle   ← vlagent DaemonSet (pod logs)
   └── Jaeger         datasource → VictoriaTraces ← OTel Collector (OTLP 4317/4318) ← apps
 VMAlert (VMRules) → VMAlertmanager

Layout

observability/
  namespace.yaml            monitoring namespace
  operator/                 victoria-metrics-operator (Helm)
  metrics/
    vmsingle|vmagent|vmalert|vmalertmanager.yaml   core CRs
    rules/                  VMRules ported from ../../apps/victoriametrics/rules
    exporters/              node-exporter (+VMNodeScrape), kube-state-metrics (Helm)
    scrapes/               kubelet + cAdvisor VMNodeScrapes
  logs/                     VLSingle CR + victoria-logs-collector (Helm)
  traces/                   victoria-traces-single (Helm) + OTel Collector
  grafana/                  Grafana (Helm) values, OIDC secret, install script
  docker-hosts/             per-Docker-host sidecar (cAdvisor + node-exporter + Vector)

Docker hosts

The Compose services on Docker hosts (../../apps/vlmcsd, ../../apps/ps3netsrv, ../../apps/netboot, ... and the legacy ../../apps/victoriametrics stack) are monitored too:

• Metrics — pull, nothing exposed. Each host runs docker-hosts/compose.yaml (cAdvisor :8080 + node-exporter :9100). The cluster's vmagent scrapes their LAN IPs via metrics/scrapes/docker-hosts.yaml (VMStaticScrape). VMSingle stays private.
• Logs — push over the tailnet. A Vector container per host tails the Docker socket and ships to VictoriaLogs' Elasticsearch-bulk endpoint, exposed privately as vlogs.<tailnet>.ts.net by logs/vlogs-ingress.yaml.

Deploy:

# cluster side (once): expose VictoriaLogs ingest + register the scrape
kubectl apply -f logs/vlogs-ingress.yaml -f metrics/scrapes/docker-hosts.yaml

# on each Docker host:
cd docker-hosts
cp .env.example .env        # set HOST_LABEL and confirm VLOGS_ENDPOINT (tailnet FQDN)
docker compose up -d
# then add the host's IP to metrics/scrapes/docker-hosts.yaml and re-apply

Note: cAdvisor uses host port 8080 — remap it (and the scrape target) if taken.

Deploy order

# 1. Operator (installs CRDs) + namespace
cd operator && ./helm.sh && cd ..

# 2. Metrics: core CRs, rules, scrapes, exporters
kubectl apply -f metrics/vmsingle.yaml -f metrics/vmagent.yaml \
              -f metrics/vmalert.yaml -f metrics/vmalertmanager.yaml
kubectl apply -f metrics/rules/ -f metrics/scrapes/
kubectl apply -f metrics/exporters/node-exporter.yaml
metrics/exporters/kube-state-metrics.sh    # needs Prometheus CRDs; see the script

# 3. Logs
cd logs && ./helm.sh && cd ..

# 4. Traces
cd traces && ./helm.sh && cd ..

# 5. Grafana (create the OIDC secret first — see grafana/oidc-secret.yaml)
cd grafana && ./helm.sh && cd ..

Before it works — required edits

• OIDC secret pair. Generate once: authelia crypto hash generate pbkdf2 --variant sha512 --random --random.length 72
  • plaintext → the grafana-oidc Secret (create imperatively, don't commit)
  • hash → authelia/values.yaml grafana client client_secret, then helm upgrade Authelia.
• Tailnet FQDN. This scaffold assumes grafana.tail7e769.ts.net (your existing tailnet, per seaweedfs-admin). If the operator assigns a different name, update it in grafana/values.yaml (root_url) and the Authelia grafana redirect_uris.
• Chart value names. logs/collector-values.yaml and traces/traces-values.yaml note a helm show values check — verify the keys against the installed chart versions.
• Pin chart + image versions after first install (all helm.sh scripts note this; otel-collector.yaml pins the collector image — bump to current).

Follow-ups

• Dashboards: the VictoriaMetrics single-node board is migrated to grafana/dashboards/victoriametrics.json; grafana/dashboards/apply.sh loads it via the Grafana sidecar. (The old vmagent.json/vmalert.json were empty dirs Docker auto-created, not real boards.) Add the canonical ones by dropping their JSON in grafana/dashboards/ and re-running the script — vmagent (gnetId 12683), vmalert (14950), node-exporter (1860), cAdvisor now that those exporters exist.
• Wire real Alertmanager receivers in metrics/vmalertmanager.yaml (email via ../../apps/smtp-relay/), replacing the ported blackhole.
• Once parity is confirmed, decommission the Compose stack: ../../apps/victoriametrics/ (docker compose down), and retire that folder.
• Add app instrumentation: point OTEL_EXPORTER_OTLP_ENDPOINT at otel-collector.monitoring.svc:4317.