150 lines
7.1 KiB
Markdown
150 lines
7.1 KiB
Markdown
# observability
|
|
|
|
Metrics, logs, and traces for the cluster — one VictoriaMetrics-ecosystem stack,
|
|
operator-driven, running in the `monitoring` namespace. Replaces the standalone
|
|
Docker Compose stack in `../../apps/victoriametrics/`.
|
|
|
|
## Status — DEPLOYED (2026-07-10)
|
|
|
|
Live and verified in the `monitoring` namespace (operator chart 0.66.2):
|
|
metrics (data queryable), logs (pods ingesting), traces (VTSingle CRD, verified via
|
|
Jaeger API), Grafana (3 datasources healthy, VM dashboard loaded, Tailscale ingress
|
|
`grafana.tail7e769.ts.net`, Authelia OIDC client registered). All PVCs on
|
|
`localpv-zfs-ceph`. Note the vmagent hot-reload RBAC fix (`metrics/reload-rbac.yaml`,
|
|
see `metrics/README-reload.md`) and that VL/VT CRDs are served under `.../v1`.
|
|
|
|
### Flux 接管状态
|
|
|
|
可观测性栈使用一个 `observability` Flux Kustomization，按依赖顺序逐步加入 Operator、
|
|
Metrics、Logs、Traces 和 Grafana，避免多个 reconciler 争夺共享 namespace、Helm source
|
|
或 VictoriaMetrics CRD。brownfield 迁移期间保持 `prune: false`。
|
|
|
|
VictoriaMetrics Operator 已固定 chart `0.66.2` / app `v0.73.1` 完成分阶段 Flux
|
|
HelmRelease 接管，保存 values 与 `operator/values.yaml` 一致。阶段一确认 Helm revision
|
|
保持为 2、Operator Pod UID 未变、24 个 CRD 及 6 个核心 VM/VL/VT CR 未变化；随后
|
|
通过独立 PR 解除暂停。Metrics、Logs、Traces 与 Grafana 仍需逐项接管。
|
|
|
|
Still host-side (not yet done): deploy `docker-hosts/compose.yaml` on the Docker
|
|
hosts and `logs/vlogs-ingress.yaml` to push their logs.
|
|
|
|
## Decision
|
|
|
|
| Pillar | Choice | Why |
|
|
| -------- | ----------------------------- | --- |
|
|
| Metrics | **VictoriaMetrics** (VMSingle)| already in use; keep it |
|
|
| Logs | **VictoriaLogs** (VLSingle) | same vendor/operator, tiny footprint, LogsQL |
|
|
| Traces | **VictoriaTraces** + OTel Collector | no extra storage deps, OTLP in, Jaeger query API |
|
|
| Collect | vmagent (metrics), vlagent (logs), OTel Collector (traces) | native to each backend |
|
|
| Visualize| **Grafana**, one instance, 3 datasources | single pane |
|
|
| Manage | **victoria-metrics-operator** | VMSingle/VMAgent/VMAlert/VMAlertmanager/VMRule **and** VLSingle as CRDs |
|
|
| Expose | **Tailscale ingress** (private) + **Authelia OIDC** | admin tool: private + SSO |
|
|
|
|
Grafana's Prometheus-operator converter is on, so any chart shipping a
|
|
`ServiceMonitor`/`PodMonitor`/`PrometheusRule` is scraped automatically.
|
|
|
|
## Architecture
|
|
|
|
```
|
|
Grafana (grafana.<tailnet>.ts.net, Authelia OIDC)
|
|
├── VictoriaMetrics datasource → VMSingle ← vmagent (k8s SD, node-exporter, kubelet/cAdvisor, KSM)
|
|
├── VictoriaLogs datasource → VLSingle ← vlagent DaemonSet (pod logs)
|
|
└── Jaeger datasource → VictoriaTraces ← OTel Collector (OTLP 4317/4318) ← apps
|
|
VMAlert (VMRules) → VMAlertmanager
|
|
```
|
|
|
|
## Layout
|
|
|
|
```
|
|
observability/
|
|
namespace.yaml monitoring namespace
|
|
operator/ victoria-metrics-operator (Helm)
|
|
metrics/
|
|
vmsingle|vmagent|vmalert|vmalertmanager.yaml core CRs
|
|
rules/ VMRules ported from ../../apps/victoriametrics/rules
|
|
exporters/ node-exporter (+VMNodeScrape), kube-state-metrics (Helm)
|
|
scrapes/ kubelet + cAdvisor VMNodeScrapes
|
|
logs/ VLSingle CR + victoria-logs-collector (Helm)
|
|
traces/ victoria-traces-single (Helm) + OTel Collector
|
|
grafana/ Grafana (Helm) values, OIDC secret, install script
|
|
docker-hosts/ per-Docker-host sidecar (cAdvisor + node-exporter + Vector)
|
|
```
|
|
|
|
## Docker hosts
|
|
|
|
The Compose services on Docker hosts (`../../apps/vlmcsd`, `../../apps/ps3netsrv`, `../../apps/netboot`, ...
|
|
and the legacy `../../apps/victoriametrics` stack) are monitored too:
|
|
|
|
- **Metrics — pull, nothing exposed.** Each host runs `docker-hosts/compose.yaml`
|
|
(cAdvisor `:8080` + node-exporter `:9100`). The cluster's vmagent scrapes their LAN
|
|
IPs via `metrics/scrapes/docker-hosts.yaml` (VMStaticScrape). VMSingle stays private.
|
|
- **Logs — push over the tailnet.** A **Vector** container per host tails the Docker
|
|
socket and ships to VictoriaLogs' Elasticsearch-bulk endpoint, exposed privately as
|
|
`vlogs.<tailnet>.ts.net` by `logs/vlogs-ingress.yaml`.
|
|
|
|
Deploy:
|
|
|
|
```bash
|
|
# cluster side (once): expose VictoriaLogs ingest + register the scrape
|
|
kubectl apply -f logs/vlogs-ingress.yaml -f metrics/scrapes/docker-hosts.yaml
|
|
|
|
# on each Docker host:
|
|
cd docker-hosts
|
|
cp .env.example .env # set HOST_LABEL and confirm VLOGS_ENDPOINT (tailnet FQDN)
|
|
docker compose up -d
|
|
# then add the host's IP to metrics/scrapes/docker-hosts.yaml and re-apply
|
|
```
|
|
|
|
Note: cAdvisor uses host port `8080` — remap it (and the scrape target) if taken.
|
|
|
|
## Deploy order
|
|
|
|
```bash
|
|
# 1. Operator (installs CRDs) + namespace
|
|
cd operator && ./helm.sh && cd ..
|
|
|
|
# 2. Metrics: core CRs, rules, scrapes, exporters
|
|
kubectl apply -f metrics/vmsingle.yaml -f metrics/vmagent.yaml \
|
|
-f metrics/vmalert.yaml -f metrics/vmalertmanager.yaml
|
|
kubectl apply -f metrics/rules/ -f metrics/scrapes/
|
|
kubectl apply -f metrics/exporters/node-exporter.yaml
|
|
metrics/exporters/kube-state-metrics.sh # needs Prometheus CRDs; see the script
|
|
|
|
# 3. Logs
|
|
cd logs && ./helm.sh && cd ..
|
|
|
|
# 4. Traces
|
|
cd traces && ./helm.sh && cd ..
|
|
|
|
# 5. Grafana (create the OIDC secret first — see grafana/oidc-secret.yaml)
|
|
cd grafana && ./helm.sh && cd ..
|
|
```
|
|
|
|
## Before it works — required edits
|
|
|
|
- **OIDC secret pair.** Generate once:
|
|
`authelia crypto hash generate pbkdf2 --variant sha512 --random --random.length 72`
|
|
- plaintext → the `grafana-oidc` Secret (create imperatively, don't commit)
|
|
- hash → `authelia/values.yaml` grafana client `client_secret`, then `helm upgrade` Authelia.
|
|
- **Tailnet FQDN.** This scaffold assumes `grafana.tail7e769.ts.net` (your existing tailnet,
|
|
per `seaweedfs-admin`). If the operator assigns a different name, update it in
|
|
`grafana/values.yaml` (`root_url`) **and** the Authelia grafana `redirect_uris`.
|
|
- **Chart value names.** `logs/collector-values.yaml` and `traces/traces-values.yaml` note a
|
|
`helm show values` check — verify the keys against the installed chart versions.
|
|
- **Pin chart + image versions** after first install (all `helm.sh` scripts note this;
|
|
`otel-collector.yaml` pins the collector image — bump to current).
|
|
|
|
## Follow-ups
|
|
|
|
- Dashboards: the VictoriaMetrics single-node board is migrated to
|
|
`grafana/dashboards/victoriametrics.json`; `grafana/dashboards/apply.sh` loads it via
|
|
the Grafana sidecar. (The old `vmagent.json`/`vmalert.json` were empty dirs Docker
|
|
auto-created, not real boards.) Add the canonical ones by dropping their JSON in
|
|
`grafana/dashboards/` and re-running the script — vmagent (gnetId 12683),
|
|
vmalert (14950), node-exporter (1860), cAdvisor now that those exporters exist.
|
|
- Wire real Alertmanager receivers in `metrics/vmalertmanager.yaml` (email via
|
|
`../../apps/smtp-relay/`), replacing the ported `blackhole`.
|
|
- Once parity is confirmed, decommission the Compose stack: `../../apps/victoriametrics/`
|
|
(`docker compose down`), and retire that folder.
|
|
- Add app instrumentation: point `OTEL_EXPORTER_OTLP_ENDPOINT` at
|
|
`otel-collector.monitoring.svc:4317`.
|