167 lines
8.4 KiB
Markdown
167 lines
8.4 KiB
Markdown
# observability
|
||
|
||
Metrics, logs, and traces for the cluster — one VictoriaMetrics-ecosystem stack,
|
||
operator-driven, running in the `monitoring` namespace. Replaces the standalone
|
||
Docker Compose stack in `../../apps/victoriametrics/`.
|
||
|
||
## Status — DEPLOYED (2026-07-10)
|
||
|
||
Live and verified in the `monitoring` namespace (operator chart 0.66.2):
|
||
metrics (data queryable), logs (pods ingesting), traces (VTSingle CRD, verified via
|
||
Jaeger API), Grafana (3 datasources healthy, VM dashboard loaded, Tailscale ingress
|
||
`grafana.tail7e769.ts.net`, Authelia OIDC client registered). All PVCs on
|
||
`localpv-zfs-ceph`. Note the vmagent hot-reload RBAC fix (`metrics/reload-rbac.yaml`,
|
||
see `metrics/README-reload.md`) and that VL/VT CRDs are served under `.../v1`.
|
||
|
||
### Flux 接管状态
|
||
|
||
可观测性栈使用一个 `observability` Flux Kustomization，按依赖顺序逐步加入 Operator、
|
||
Metrics、Logs、Traces 和 Grafana，避免多个 reconciler 争夺共享 namespace、Helm source
|
||
或 VictoriaMetrics CRD。brownfield 迁移期间保持 `prune: false`。
|
||
|
||
VictoriaMetrics Operator 已固定 chart `0.66.2` / app `v0.73.1` 完成分阶段 Flux
|
||
HelmRelease 接管，保存 values 与 `operator/values.yaml` 一致。阶段一确认 Helm revision
|
||
保持为 2、Operator Pod UID 未变、24 个 CRD 及 6 个核心 VM/VL/VT CR 未变化；随后
|
||
通过独立 PR 解除暂停。
|
||
|
||
其余已部署资源也已统一完成接管：Metrics CR、rules、scrapes、node-exporter，Logs 的
|
||
VLSingle 与固定 `0.3.6` 的 Collector，以及 Traces 的 VTSingle、OTel Collector，最后是
|
||
固定 `10.5.15` 的 Grafana 与 dashboard ConfigMap。阶段一确认 11 个 Pod UID、4 个 PVC、
|
||
6 个核心 CR、17 个 rule/scrape 对象及 dashboard 哈希均未变化，Collector 和 Grafana
|
||
Helm revision 均保持为 1；随后在同一独立 PR 中解除两个 HelmRelease 的暂停。
|
||
|
||
首次 Grafana Helm reconcile 暴露了本地 ZFS RWO 卷限制：chart 默认 `RollingUpdate`
|
||
会先创建新 Pod，但 CSI 拒绝在旧 Pod 仍挂载 PVC 时再次 mount，使 upgrade 卡在
|
||
`pending-upgrade`。`grafana/values.yaml` 因此显式使用 `deploymentStrategy.type: Recreate`。
|
||
Grafana 升级会有一次短暂停机，但旧 Pod 会先退出，新 Pod 才挂载同一 PVC；不要改回
|
||
RollingUpdate，除非存储改为真正支持并发挂载的 RWX。
|
||
|
||
以下内容明确不属于本批接管：尚未部署的 kube-state-metrics、`vlogs-ingress.yaml`，以及
|
||
仅作 standalone chart 参考的 `traces-values.yaml`。`grafana/oidc-secret.yaml` 是不含真实
|
||
值的占位模板；live `grafana-oidc` Secret 继续只被 HelmRelease 引用，尚未由 OpenBao/ESO
|
||
纳管。
|
||
|
||
Still host-side (not yet done): deploy `docker-hosts/compose.yaml` on the Docker
|
||
hosts and `logs/vlogs-ingress.yaml` to push their logs.
|
||
|
||
## Decision
|
||
|
||
| Pillar | Choice | Why |
|
||
| -------- | ----------------------------- | --- |
|
||
| Metrics | **VictoriaMetrics** (VMSingle)| already in use; keep it |
|
||
| Logs | **VictoriaLogs** (VLSingle) | same vendor/operator, tiny footprint, LogsQL |
|
||
| Traces | **VictoriaTraces** + OTel Collector | no extra storage deps, OTLP in, Jaeger query API |
|
||
| Collect | vmagent (metrics), vlagent (logs), OTel Collector (traces) | native to each backend |
|
||
| Visualize| **Grafana**, one instance, 3 datasources | single pane |
|
||
| Manage | **victoria-metrics-operator** | VMSingle/VMAgent/VMAlert/VMAlertmanager/VMRule **and** VLSingle as CRDs |
|
||
| Expose | **Tailscale ingress** (private) + **Authelia OIDC** | admin tool: private + SSO |
|
||
|
||
Grafana's Prometheus-operator converter is on, so any chart shipping a
|
||
`ServiceMonitor`/`PodMonitor`/`PrometheusRule` is scraped automatically.
|
||
|
||
## Architecture
|
||
|
||
```
|
||
Grafana (grafana.<tailnet>.ts.net, Authelia OIDC)
|
||
├── VictoriaMetrics datasource → VMSingle ← vmagent (k8s SD, node-exporter, kubelet/cAdvisor, KSM)
|
||
├── VictoriaLogs datasource → VLSingle ← vlagent DaemonSet (pod logs)
|
||
└── Jaeger datasource → VictoriaTraces ← OTel Collector (OTLP 4317/4318) ← apps
|
||
VMAlert (VMRules) → VMAlertmanager
|
||
```
|
||
|
||
## Layout
|
||
|
||
```
|
||
observability/
|
||
namespace.yaml monitoring namespace
|
||
operator/ victoria-metrics-operator (Helm)
|
||
metrics/
|
||
vmsingle|vmagent|vmalert|vmalertmanager.yaml core CRs
|
||
rules/ VMRules ported from ../../apps/victoriametrics/rules
|
||
exporters/ node-exporter (+VMNodeScrape), kube-state-metrics (Helm)
|
||
scrapes/ kubelet + cAdvisor VMNodeScrapes
|
||
logs/ VLSingle CR + victoria-logs-collector (Helm)
|
||
traces/ victoria-traces-single (Helm) + OTel Collector
|
||
grafana/ Grafana (Helm) values, OIDC secret, install script
|
||
docker-hosts/ per-Docker-host sidecar (cAdvisor + node-exporter + Vector)
|
||
```
|
||
|
||
## Docker hosts
|
||
|
||
The Compose services on Docker hosts (`../../apps/vlmcsd`, `../../apps/ps3netsrv`, `../../apps/netboot`, ...
|
||
and the legacy `../../apps/victoriametrics` stack) are monitored too:
|
||
|
||
- **Metrics — pull, nothing exposed.** Each host runs `docker-hosts/compose.yaml`
|
||
(cAdvisor `:8080` + node-exporter `:9100`). The cluster's vmagent scrapes their LAN
|
||
IPs via `metrics/scrapes/docker-hosts.yaml` (VMStaticScrape). VMSingle stays private.
|
||
- **Logs — push over the tailnet.** A **Vector** container per host tails the Docker
|
||
socket and ships to VictoriaLogs' Elasticsearch-bulk endpoint, exposed privately as
|
||
`vlogs.<tailnet>.ts.net` by `logs/vlogs-ingress.yaml`.
|
||
|
||
Deploy:
|
||
|
||
```bash
|
||
# cluster side (once): expose VictoriaLogs ingest + register the scrape
|
||
kubectl apply -f logs/vlogs-ingress.yaml -f metrics/scrapes/docker-hosts.yaml
|
||
|
||
# on each Docker host:
|
||
cd docker-hosts
|
||
cp .env.example .env # set HOST_LABEL and confirm VLOGS_ENDPOINT (tailnet FQDN)
|
||
docker compose up -d
|
||
# then add the host's IP to metrics/scrapes/docker-hosts.yaml and re-apply
|
||
```
|
||
|
||
Note: cAdvisor uses host port `8080` — remap it (and the scrape target) if taken.
|
||
|
||
## Deploy order
|
||
|
||
```bash
|
||
# 1. Operator (installs CRDs) + namespace
|
||
cd operator && ./helm.sh && cd ..
|
||
|
||
# 2. Metrics: core CRs, rules, scrapes, exporters
|
||
kubectl apply -f metrics/vmsingle.yaml -f metrics/vmagent.yaml \
|
||
-f metrics/vmalert.yaml -f metrics/vmalertmanager.yaml
|
||
kubectl apply -f metrics/rules/ -f metrics/scrapes/
|
||
kubectl apply -f metrics/exporters/node-exporter.yaml
|
||
metrics/exporters/kube-state-metrics.sh # needs Prometheus CRDs; see the script
|
||
|
||
# 3. Logs
|
||
cd logs && ./helm.sh && cd ..
|
||
|
||
# 4. Traces
|
||
cd traces && ./helm.sh && cd ..
|
||
|
||
# 5. Grafana (create the OIDC secret first — see grafana/oidc-secret.yaml)
|
||
cd grafana && ./helm.sh && cd ..
|
||
```
|
||
|
||
## Before it works — required edits
|
||
|
||
- **OIDC secret pair.** Generate once:
|
||
`authelia crypto hash generate pbkdf2 --variant sha512 --random --random.length 72`
|
||
- plaintext → the `grafana-oidc` Secret (create imperatively, don't commit)
|
||
- hash → `authelia/values.yaml` grafana client `client_secret`, then `helm upgrade` Authelia.
|
||
- **Tailnet FQDN.** This scaffold assumes `grafana.tail7e769.ts.net` (your existing tailnet,
|
||
per `seaweedfs-admin`). If the operator assigns a different name, update it in
|
||
`grafana/values.yaml` (`root_url`) **and** the Authelia grafana `redirect_uris`.
|
||
- **Chart value names.** `logs/collector-values.yaml` and `traces/traces-values.yaml` note a
|
||
`helm show values` check — verify the keys against the installed chart versions.
|
||
- **Pin chart + image versions** after first install (all `helm.sh` scripts note this;
|
||
`otel-collector.yaml` pins the collector image — bump to current).
|
||
|
||
## Follow-ups
|
||
|
||
- Dashboards: the VictoriaMetrics single-node board is migrated to
|
||
`grafana/dashboards/victoriametrics.json`; `grafana/dashboards/apply.sh` loads it via
|
||
the Grafana sidecar. (The old `vmagent.json`/`vmalert.json` were empty dirs Docker
|
||
auto-created, not real boards.) Add the canonical ones by dropping their JSON in
|
||
`grafana/dashboards/` and re-running the script — vmagent (gnetId 12683),
|
||
vmalert (14950), node-exporter (1860), cAdvisor now that those exporters exist.
|
||
- Wire real Alertmanager receivers in `metrics/vmalertmanager.yaml` (email via
|
||
`../../apps/smtp-relay/`), replacing the ported `blackhole`.
|
||
- Once parity is confirmed, decommission the Compose stack: `../../apps/victoriametrics/`
|
||
(`docker compose down`), and retire that folder.
|
||
- Add app instrumentation: point `OTEL_EXPORTER_OTLP_ENDPOINT` at
|
||
`otel-collector.monitoring.svc:4317`.
|