Establish clean homelab infrastructure baseline

Reorganize the brownfield repository, remove retired and generated artifacts, harden ignore rules, and record the GitOps/IaC redesign.

platform/observability/README.md

@@ -0,0 +1,138 @@
+# observability
+
+Metrics, logs, and traces for the cluster — one VictoriaMetrics-ecosystem stack,
+operator-driven, running in the `monitoring` namespace. Replaces the standalone
+Docker Compose stack in `../../apps/victoriametrics/`.
+
+## Status — DEPLOYED (2026-07-10)
+
+Live and verified in the `monitoring` namespace (operator chart 0.66.2):
+metrics (data queryable), logs (pods ingesting), traces (VTSingle CRD, verified via
+Jaeger API), Grafana (3 datasources healthy, VM dashboard loaded, Tailscale ingress
+`grafana.tail7e769.ts.net`, Authelia OIDC client registered). All PVCs on
+`localpv-zfs-ceph`. Note the vmagent hot-reload RBAC fix (`metrics/reload-rbac.yaml`,
+see `metrics/README-reload.md`) and that VL/VT CRDs are served under `.../v1`.
+
+Still host-side (not yet done): deploy `docker-hosts/compose.yaml` on the Docker
+hosts and `logs/vlogs-ingress.yaml` to push their logs.
+
+## Decision
+
+| Pillar   | Choice                        | Why |
+| -------- | ----------------------------- | --- |
+| Metrics  | **VictoriaMetrics** (VMSingle)| already in use; keep it |
+| Logs     | **VictoriaLogs** (VLSingle)   | same vendor/operator, tiny footprint, LogsQL |
+| Traces   | **VictoriaTraces** + OTel Collector | no extra storage deps, OTLP in, Jaeger query API |
+| Collect  | vmagent (metrics), vlagent (logs), OTel Collector (traces) | native to each backend |
+| Visualize| **Grafana**, one instance, 3 datasources | single pane |
+| Manage   | **victoria-metrics-operator** | VMSingle/VMAgent/VMAlert/VMAlertmanager/VMRule **and** VLSingle as CRDs |
+| Expose   | **Tailscale ingress** (private) + **Authelia OIDC** | admin tool: private + SSO |
+
+Grafana's Prometheus-operator converter is on, so any chart shipping a
+`ServiceMonitor`/`PodMonitor`/`PrometheusRule` is scraped automatically.
+
+## Architecture
+
+```
+ Grafana (grafana.<tailnet>.ts.net, Authelia OIDC)
+   ├── VictoriaMetrics datasource → VMSingle  ← vmagent (k8s SD, node-exporter, kubelet/cAdvisor, KSM)
+   ├── VictoriaLogs   datasource → VLSingle   ← vlagent DaemonSet (pod logs)
+   └── Jaeger         datasource → VictoriaTraces ← OTel Collector (OTLP 4317/4318) ← apps
+ VMAlert (VMRules) → VMAlertmanager
+```
+
+## Layout
+
+```
+observability/
+  namespace.yaml            monitoring namespace
+  operator/                 victoria-metrics-operator (Helm)
+  metrics/
+    vmsingle|vmagent|vmalert|vmalertmanager.yaml   core CRs
+    rules/                  VMRules ported from ../../apps/victoriametrics/rules
+    exporters/              node-exporter (+VMNodeScrape), kube-state-metrics (Helm)
+    scrapes/               kubelet + cAdvisor VMNodeScrapes
+  logs/                     VLSingle CR + victoria-logs-collector (Helm)
+  traces/                   victoria-traces-single (Helm) + OTel Collector
+  grafana/                  Grafana (Helm) values, OIDC secret, install script
+  docker-hosts/             per-Docker-host sidecar (cAdvisor + node-exporter + Vector)
+```
+
+## Docker hosts
+
+The Compose services on Docker hosts (`../../apps/vlmcsd`, `../../apps/ps3netsrv`, `../../apps/netboot`, ...
+and the legacy `../../apps/victoriametrics` stack) are monitored too:
+
+- **Metrics — pull, nothing exposed.** Each host runs `docker-hosts/compose.yaml`
+  (cAdvisor `:8080` + node-exporter `:9100`). The cluster's vmagent scrapes their LAN
+  IPs via `metrics/scrapes/docker-hosts.yaml` (VMStaticScrape). VMSingle stays private.
+- **Logs — push over the tailnet.** A **Vector** container per host tails the Docker
+  socket and ships to VictoriaLogs' Elasticsearch-bulk endpoint, exposed privately as
+  `vlogs.<tailnet>.ts.net` by `logs/vlogs-ingress.yaml`.
+
+Deploy:
+
+```bash
+# cluster side (once): expose VictoriaLogs ingest + register the scrape
+kubectl apply -f logs/vlogs-ingress.yaml -f metrics/scrapes/docker-hosts.yaml
+
+# on each Docker host:
+cd docker-hosts
+cp .env.example .env        # set HOST_LABEL and confirm VLOGS_ENDPOINT (tailnet FQDN)
+docker compose up -d
+# then add the host's IP to metrics/scrapes/docker-hosts.yaml and re-apply
+```
+
+Note: cAdvisor uses host port `8080` — remap it (and the scrape target) if taken.
+
+## Deploy order
+
+```bash
+# 1. Operator (installs CRDs) + namespace
+cd operator && ./helm.sh && cd ..
+
+# 2. Metrics: core CRs, rules, scrapes, exporters
+kubectl apply -f metrics/vmsingle.yaml -f metrics/vmagent.yaml \
+              -f metrics/vmalert.yaml -f metrics/vmalertmanager.yaml
+kubectl apply -f metrics/rules/ -f metrics/scrapes/
+kubectl apply -f metrics/exporters/node-exporter.yaml
+metrics/exporters/kube-state-metrics.sh    # needs Prometheus CRDs; see the script
+
+# 3. Logs
+cd logs && ./helm.sh && cd ..
+
+# 4. Traces
+cd traces && ./helm.sh && cd ..
+
+# 5. Grafana (create the OIDC secret first — see grafana/oidc-secret.yaml)
+cd grafana && ./helm.sh && cd ..
+```
+
+## Before it works — required edits
+
+- **OIDC secret pair.** Generate once:
+  `authelia crypto hash generate pbkdf2 --variant sha512 --random --random.length 72`
+  - plaintext → the `grafana-oidc` Secret (create imperatively, don't commit)
+  - hash → `authelia/values.yaml` grafana client `client_secret`, then `helm upgrade` Authelia.
+- **Tailnet FQDN.** This scaffold assumes `grafana.tail7e769.ts.net` (your existing tailnet,
+  per `seaweedfs-admin`). If the operator assigns a different name, update it in
+  `grafana/values.yaml` (`root_url`) **and** the Authelia grafana `redirect_uris`.
+- **Chart value names.** `logs/collector-values.yaml` and `traces/traces-values.yaml` note a
+  `helm show values` check — verify the keys against the installed chart versions.
+- **Pin chart + image versions** after first install (all `helm.sh` scripts note this;
+  `otel-collector.yaml` pins the collector image — bump to current).
+
+## Follow-ups
+
+- Dashboards: the VictoriaMetrics single-node board is migrated to
+  `grafana/dashboards/victoriametrics.json`; `grafana/dashboards/apply.sh` loads it via
+  the Grafana sidecar. (The old `vmagent.json`/`vmalert.json` were empty dirs Docker
+  auto-created, not real boards.) Add the canonical ones by dropping their JSON in
+  `grafana/dashboards/` and re-running the script — vmagent (gnetId 12683),
+  vmalert (14950), node-exporter (1860), cAdvisor now that those exporters exist.
+- Wire real Alertmanager receivers in `metrics/vmalertmanager.yaml` (email via
+  `../../apps/smtp-relay/`), replacing the ported `blackhole`.
+- Once parity is confirmed, decommission the Compose stack: `../../apps/victoriametrics/`
+  (`docker compose down`), and retire that folder.
+- Add app instrumentation: point `OTEL_EXPORTER_OTLP_ENDPOINT` at
+  `otel-collector.monitoring.svc:4317`.

platform/observability/docker-hosts/.env.example

@@ -0,0 +1,9 @@
+# Copy to .env and adjust per host. Do not commit the real .env.
+
+# A stable name for THIS Docker host; becomes the `host` log stream label and the
+# `host` label on its scraped metrics (keep it consistent with the VMStaticScrape).
+HOST_LABEL=docker-01
+
+# VictoriaLogs Elasticsearch-bulk ingest endpoint, reached over the tailnet
+# (exposed by ../logs/vlogs-ingress.yaml). Confirm the tailnet FQDN after applying.
+VLOGS_ENDPOINT=https://vlogs.tail7e769.ts.net/insert/elasticsearch/

platform/observability/docker-hosts/compose.yaml

@@ -0,0 +1,50 @@
+# Per-Docker-host monitoring sidecar. Deploy this compose on every Docker host you
+# want visible in the observability stack. It exposes host + container metrics for the
+# cluster's vmagent to PULL over the LAN, and pushes container logs to VictoriaLogs.
+#
+#   cp .env.example .env   # set HOST_LABEL + VLOGS_ENDPOINT
+#   docker compose up -d
+#
+# Then add this host's IP to ../metrics/scrapes/docker-hosts.yaml (VMStaticScrape).
+services:
+  # Host metrics (CPU/mem/disk/net/fs). host networking -> listens on <hostIP>:9100.
+  node-exporter:
+    image: quay.io/prometheus/node-exporter:v1.9.1
+    command:
+      - --path.rootfs=/host
+    network_mode: host
+    pid: host
+    volumes:
+      - /:/host:ro,rslave
+    restart: unless-stopped
+
+  # Container metrics. Published on <hostIP>:8080. If 8080 is taken on this host,
+  # remap the left side (e.g. "8098:8080") and update the VMStaticScrape target.
+  cadvisor:
+    image: gcr.io/cadvisor/cadvisor:v0.49.1
+    command:
+      - --housekeeping_interval=30s
+      - --docker_only=true
+    ports:
+      - "8080:8080"
+    volumes:
+      - /:/rootfs:ro
+      - /var/run:/var/run:ro
+      - /sys:/sys:ro
+      - /var/lib/docker/:/var/lib/docker:ro
+      - /dev/disk/:/dev/disk:ro
+    devices:
+      - /dev/kmsg
+    privileged: true
+    restart: unless-stopped
+
+  # Ships every container's logs to VictoriaLogs (pushed over the tailnet).
+  vector:
+    image: timberio/vector:0.44.0-debian
+    volumes:
+      - /var/run/docker.sock:/var/run/docker.sock:ro
+      - ./vector.yaml:/etc/vector/vector.yaml:ro
+    environment:
+      VLOGS_ENDPOINT: ${VLOGS_ENDPOINT}
+      HOST_LABEL: ${HOST_LABEL}
+    restart: unless-stopped

platform/observability/docker-hosts/vector.yaml

@@ -0,0 +1,30 @@
+# Vector: tail all Docker container logs on this host and ship them to VictoriaLogs
+# via the Elasticsearch bulk API (the VM-recommended Vector sink). Endpoint + host
+# label come from the environment (see .env / compose.yaml).
+sources:
+  docker:
+    type: docker_logs
+
+transforms:
+  enrich:
+    type: remap
+    inputs: [docker]
+    source: |
+      .host = get_env_var!("HOST_LABEL")
+
+sinks:
+  vlogs:
+    type: elasticsearch
+    inputs: [enrich]
+    endpoints:
+      - "${VLOGS_ENDPOINT}"      # e.g. https://vlogs.tail7e769.ts.net/insert/elasticsearch/
+    mode: bulk
+    api_version: v8
+    compression: gzip
+    healthcheck:
+      enabled: false
+    query:
+      _msg_field: message
+      _time_field: timestamp
+      # Group log streams by host + container so LogsQL stream filters work well.
+      _stream_fields: host,container_name

platform/observability/grafana/dashboards/apply.sh

@@ -0,0 +1,23 @@
+#!/usr/bin/env bash
+# Load every *.json in this folder into Grafana as a sidecar dashboard ConfigMap.
+# The Grafana chart's dashboard sidecar (sidecar.dashboards.enabled) watches for
+# ConfigMaps labeled grafana_dashboard=1 in the monitoring namespace and imports them.
+# Idempotent — re-run after adding/updating a dashboard JSON.
+#
+# The VictoriaMetrics board uses a `$ds` datasource variable that resolves to the
+# default datasource (VictoriaMetrics), so no per-panel rewiring is needed.
+set -euo pipefail
+cd "$(dirname "$0")"
+
+NS=monitoring
+for f in *.json; do
+  [ -e "$f" ] || continue
+  name="grafana-dashboard-$(basename "$f" .json)"
+  echo "applying $name from $f"
+  kubectl create configmap "$name" \
+    --namespace "$NS" \
+    --from-file="$f" \
+    --dry-run=client -o yaml \
+  | kubectl label --local -f - grafana_dashboard=1 --dry-run=client -o yaml \
+  | kubectl apply -f -
+done

platform/observability/grafana/helm.sh

@@ -0,0 +1,22 @@
+#!/usr/bin/env bash
+# Deploy Grafana. Create the OIDC client secret first (see oidc-secret.yaml header),
+# then install the chart. Requires the metrics/logs/traces backends to exist so the
+# provisioned datasources resolve.
+set -euo pipefail
+
+# 1) OIDC client secret (edit oidc-secret.yaml, or create it imperatively — preferred).
+kubectl apply -f oidc-secret.yaml
+
+helm repo add grafana https://grafana.github.io/helm-charts
+helm repo update grafana
+
+# Pin --version after the first install (helm search repo grafana/grafana --versions).
+helm upgrade --install grafana grafana/grafana \
+  --namespace monitoring \
+  --values values.yaml \
+  --wait
+
+# The tailnet FQDN, once the tailscale operator assigns it:
+#   kubectl -n monitoring get ingress grafana -o jsonpath='{.status.loadBalancer.ingress[0].hostname}'
+# Break-glass admin password:
+#   kubectl -n monitoring get secret grafana -o jsonpath='{.data.admin-password}' | base64 -d

platform/observability/grafana/oidc-secret.yaml

@@ -0,0 +1,18 @@
+# Plaintext OIDC client secret Grafana presents to Authelia. Authelia stores only the
+# pbkdf2-sha512 HASH of this same value (see authelia/values.yaml grafana client).
+#
+# Generate a matching pair:
+#   authelia crypto hash generate pbkdf2 --variant sha512 --random --random.length 72
+# Put the "Random Password" (plaintext) below; put the "Digest" (hash) in Authelia.
+#
+# Do NOT commit the real secret. Create it out-of-band instead, e.g.:
+#   kubectl -n monitoring create secret generic grafana-oidc \
+#     --from-literal=client_secret='<plaintext>'
+apiVersion: v1
+kind: Secret
+metadata:
+  name: grafana-oidc
+  namespace: monitoring
+type: Opaque
+stringData:
+  client_secret: "REPLACE_ME_WITH_PLAINTEXT_OIDC_SECRET"

platform/observability/grafana/values.yaml

@@ -0,0 +1,98 @@
+# Grafana — single pane over metrics (VictoriaMetrics), logs (VictoriaLogs) and
+# traces (VictoriaTraces via Jaeger API). Exposed privately on the tailnet
+# (grafana.tail7e769.ts.net) and authenticated via Authelia OIDC. Local admin is
+# break-glass only.
+#
+# Chart: grafana/grafana   (repo: https://grafana.github.io/helm-charts)
+
+# --- VictoriaLogs needs its Grafana datasource plugin ---
+plugins:
+  - victoriametrics-logs-datasource
+
+# --- Dashboard sidecar: auto-loads any ConfigMap labeled grafana_dashboard=1 in the
+# namespace. Migrated boards live in ./dashboards and are applied by ./dashboards/apply.sh ---
+sidecar:
+  dashboards:
+    enabled: true
+    label: grafana_dashboard
+    labelValue: "1"
+    folderAnnotation: grafana_folder
+    provider:
+      foldersFromFilesStructure: true
+
+# --- Provisioned datasources ---
+datasources:
+  datasources.yaml:
+    apiVersion: 1
+    datasources:
+      - name: VictoriaMetrics
+        type: prometheus
+        access: proxy
+        url: http://vmsingle-main.monitoring.svc:8428
+        isDefault: true
+        jsonData:
+          prometheusType: Prometheus
+      - name: VictoriaLogs
+        type: victoriametrics-logs-datasource
+        access: proxy
+        url: http://vlsingle-main.monitoring.svc:9428
+      - name: VictoriaTraces
+        type: jaeger
+        access: proxy
+        # VictoriaTraces (VTSingle CR) exposes a Jaeger-compatible query API under /select/jaeger.
+        url: http://vtsingle-main.monitoring.svc:10428/select/jaeger
+
+# --- Persistence on OpenEBS ZFS ---
+persistence:
+  enabled: true
+  storageClassName: localpv-zfs-ceph
+  size: 5Gi
+
+# --- Private exposure via the Tailscale ingress (like seaweedfs-admin) ---
+# The tailscale operator provisions grafana.<tailnet>.ts.net and a TLS cert.
+ingress:
+  enabled: true
+  ingressClassName: tailscale
+  hosts:
+    - grafana
+  tls:
+    - hosts:
+        - grafana
+
+# --- OIDC via Authelia (AD groups -> Grafana roles) ---
+# client_secret is injected from the grafana-oidc Secret (see oidc-secret.yaml),
+# which overrides any value in grafana.ini.
+envValueFrom:
+  GF_AUTH_GENERIC_OAUTH_CLIENT_SECRET:
+    secretKeyRef:
+      name: grafana-oidc
+      key: client_secret
+
+grafana.ini:
+  server:
+    root_url: "https://grafana.tail7e769.ts.net"   # must match the tailnet FQDN + Authelia redirect_uri
+  auth:
+    # Keep the local admin login available as break-glass; don't force OIDC-only.
+    disable_login_form: false
+    oauth_auto_login: false
+  auth.generic_oauth:
+    enabled: true
+    name: Authelia
+    client_id: grafana
+    scopes: "openid profile email groups"
+    auth_url: "https://auth.ddupan.top/api/oidc/authorization"
+    token_url: "https://auth.ddupan.top/api/oidc/token"
+    api_url: "https://auth.ddupan.top/api/oidc/userinfo"
+    login_attribute_path: preferred_username
+    name_attribute_path: name
+    email_attribute_path: email
+    groups_attribute_path: groups
+    # AD "Enterprise Admins" -> full Grafana server admin; "Domain Admins" -> org Admin;
+    # everyone else who can authenticate -> Viewer. Tune group names to taste.
+    role_attribute_path: "contains(groups[*], 'Enterprise Admins') && 'GrafanaAdmin' || contains(groups[*], 'Domain Admins') && 'Admin' || 'Viewer'"
+    allow_assign_grafana_admin: true
+    role_attribute_strict: false
+    use_pkce: true
+
+# Dashboards are loaded by the sidecar (above) from ConfigMaps created by
+# ./dashboards/apply.sh. Drop more JSON into ./dashboards and re-run that script.

platform/observability/logs/collector-values.yaml

@@ -0,0 +1,22 @@
+# victoria-logs-collector — deploys vlagent as a DaemonSet that discovers and tails
+# every pod's container logs on each node and ships them to VictoriaLogs. Buffers to
+# a local volume when VictoriaLogs is unavailable.
+#
+# NOTE: confirm key names against the installed chart before applying:
+#   helm show values vm/victoria-logs-collector
+# (chart value layout occasionally shifts between versions.)
+
+remoteWrite:
+  - url: "http://vlsingle-main.monitoring.svc:9428"
+
+# Tolerate control-plane taints so logs are collected from every node.
+tolerations:
+  - operator: Exists
+
+resources:
+  requests:
+    cpu: 50m
+    memory: 128Mi
+  limits:
+    cpu: 500m
+    memory: 512Mi

platform/observability/logs/helm.sh

@@ -0,0 +1,15 @@
+#!/usr/bin/env bash
+# Deploy the pod-log collector (vlagent DaemonSet) that ships container logs to the
+# VLSingle instance created by ./vlsingle.yaml. Apply vlsingle.yaml first.
+set -euo pipefail
+
+kubectl apply -f vlsingle.yaml
+
+helm repo add vm https://victoriametrics.github.io/helm-charts/
+helm repo update vm
+
+# Pin --version after the first install (helm search repo vm/victoria-logs-collector --versions).
+helm upgrade --install victoria-logs-collector vm/victoria-logs-collector \
+  --namespace monitoring \
+  --values collector-values.yaml \
+  --wait

platform/observability/logs/vlogs-ingress.yaml

@@ -0,0 +1,24 @@
+# Private tailnet exposure of the VictoriaLogs INGEST endpoint so off-cluster log
+# shippers (Vector on Docker hosts) can push to it. Provisions vlogs.<tailnet>.ts.net.
+# Ingestion is unauthenticated — acceptable on the private tailnet; do not expose publicly.
+apiVersion: networking.k8s.io/v1
+kind: Ingress
+metadata:
+  name: victorialogs
+  namespace: monitoring
+spec:
+  ingressClassName: tailscale
+  rules:
+    - host: vlogs
+      http:
+        paths:
+          - path: /
+            pathType: Prefix
+            backend:
+              service:
+                name: vlsingle-main
+                port:
+                  number: 9428
+  tls:
+    - hosts:
+        - vlogs

platform/observability/logs/vlsingle.yaml

@@ -0,0 +1,25 @@
+# Single-node VictoriaLogs, managed by the same VM operator as the metrics stack.
+# Service exposed by the operator: vlsingle-main.monitoring.svc:9428
+# Query via Grafana (VictoriaLogs datasource) with LogsQL.
+# NOTE: the VictoriaLogs/Traces CRDs are served under .../v1 (metrics CRDs use v1beta1).
+apiVersion: operator.victoriametrics.com/v1
+kind: VLSingle
+metadata:
+  name: main
+  namespace: monitoring
+spec:
+  retentionPeriod: "30d"
+  storage:
+    accessModes:
+      - ReadWriteOnce
+    storageClassName: localpv-zfs-ceph
+    resources:
+      requests:
+        storage: 30Gi
+  resources:
+    requests:
+      cpu: 100m
+      memory: 256Mi
+    limits:
+      cpu: "1"
+      memory: 2Gi

platform/observability/metrics/README-reload.md

@@ -0,0 +1,41 @@
+# vmagent scrape-config hot-reload on k3s
+
+## Symptom
+New/changed `VMServiceScrape`/`VMNodeScrape`/`VMStaticScrape`/... are written by the
+operator into the `vmagent-main` config Secret, but the running vmagent keeps serving
+its **startup** config — the new targets never appear until vmagent is restarted.
+
+## Cause
+vmagent's config-reloader (`victoriametrics/operator:config-reloader-*`) watches the
+config Secret with a client-go informer, which does `list` **and** `watch`. The
+operator-generated Role `monitoring:monitoring:vmagent-main` grants `secrets: [get, watch]`
+but **not `list`**, so on k3s the informer errors with:
+
+```
+cannot list resource "secrets" in API group "" in the namespace "monitoring"
+```
+
+and no reload is ever triggered.
+
+## Fix options
+
+**A) Grant `list` on secrets (proper hot-reload).** Additive, namespace-scoped Role
+(vmagent already has get/watch; this adds only `list`). Apply `reload-rbac.yaml`:
+
+```bash
+kubectl apply -f metrics/reload-rbac.yaml
+```
+
+Marginal exposure: lets the vmagent SA enumerate Secrets in the `monitoring` namespace
+only (it can already read them individually). This is what the operator is expected to
+grant; the generated Role simply omits `list` on this version.
+
+**B) No RBAC change — restart on change.** Leave RBAC as-is and restart vmagent after
+applying scrape-config changes:
+
+```bash
+kubectl -n monitoring rollout restart deploy/vmagent-main
+```
+
+Fine for a homelab where scrape configs change rarely; the tradeoff is a manual step
+(and a ~10s gap) each time.

platform/observability/metrics/exporters/kube-state-metrics.sh

@@ -0,0 +1,20 @@
+#!/usr/bin/env bash
+# kube-state-metrics — metrics about k8s objects (deployments, pods, nodes, PVCs, ...).
+# Its chart ships a ServiceMonitor; the VM operator's Prometheus converter turns that
+# into a VMServiceScrape automatically, so no hand-written scrape CR is needed here.
+#
+# Prereq: the Prometheus-operator CRDs (ServiceMonitor, ...) must exist in the cluster
+# for `serviceMonitor.enabled` to apply. If they don't:
+#   kubectl apply --server-side -f \
+#     https://github.com/prometheus-operator/prometheus-operator/releases/latest/download/bundle.yaml \
+#     # (or just the monitoring.coreos.com CRDs)
+# Alternatively, disable the ServiceMonitor below and add a VMServiceScrape by hand.
+set -euo pipefail
+
+helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
+helm repo update prometheus-community
+
+helm upgrade --install kube-state-metrics prometheus-community/kube-state-metrics \
+  --namespace monitoring \
+  --set prometheus.monitor.enabled=true \
+  --wait

platform/observability/metrics/exporters/node-exporter.yaml

@@ -0,0 +1,67 @@
+# Host metrics (CPU/mem/disk/net/filesystem) for every node. node-exporter has no
+# ServiceMonitor of its own here, so we scrape it directly with a VMNodeScrape:
+# vmagent discovers each Node and scrapes its InternalIP:9100.
+apiVersion: apps/v1
+kind: DaemonSet
+metadata:
+  name: node-exporter
+  namespace: monitoring
+  labels:
+    app.kubernetes.io/name: node-exporter
+spec:
+  selector:
+    matchLabels:
+      app.kubernetes.io/name: node-exporter
+  template:
+    metadata:
+      labels:
+        app.kubernetes.io/name: node-exporter
+    spec:
+      hostNetwork: true          # bind 9100 on the node so VMNodeScrape can reach it
+      hostPID: true
+      tolerations:
+        - operator: Exists       # run on control-plane / tainted nodes too
+      containers:
+        - name: node-exporter
+          image: quay.io/prometheus/node-exporter:v1.9.1
+          args:
+            - --path.procfs=/host/proc
+            - --path.sysfs=/host/sys
+            - --path.rootfs=/host/root
+            - --collector.filesystem.mount-points-exclude=^/(dev|proc|sys|var/lib/docker/.+|var/lib/kubelet/.+)($|/)
+          ports:
+            - name: metrics
+              containerPort: 9100
+              hostPort: 9100
+          resources:
+            requests:
+              cpu: 25m
+              memory: 32Mi
+            limits:
+              cpu: 200m
+              memory: 128Mi
+          volumeMounts:
+            - { name: proc, mountPath: /host/proc, readOnly: true }
+            - { name: sys, mountPath: /host/sys, readOnly: true }
+            - { name: root, mountPath: /host/root, readOnly: true, mountPropagation: HostToContainer }
+      volumes:
+        - { name: proc, hostPath: { path: /proc } }
+        - { name: sys, hostPath: { path: /sys } }
+        - { name: root, hostPath: { path: / } }
+---
+apiVersion: operator.victoriametrics.com/v1beta1
+kind: VMNodeScrape
+metadata:
+  name: node-exporter
+  namespace: monitoring
+  labels:
+    app.kubernetes.io/part-of: victoria-metrics
+spec:
+  port: "9100"
+  scheme: http
+  interval: 30s
+  relabelConfigs:
+    - action: labelmap
+      regex: __meta_kubernetes_node_label_(.+)
+    - targetLabel: job
+      replacement: node-exporter

platform/observability/metrics/reload-rbac.yaml

@@ -0,0 +1,29 @@
+# Enables vmagent scrape-config hot-reload. The operator-generated Role grants the
+# vmagent-main SA secrets [get,watch] but omits [list], which its config-reloader's
+# informer requires — so scrape-config changes stall until vmagent restarts.
+# This supplementary, namespace-scoped Role covers secrets get/list/watch so the
+# reloader works regardless of the operator Role. Scope: Secrets in `monitoring` only.
+# See README-reload.md for background.
+apiVersion: rbac.authorization.k8s.io/v1
+kind: Role
+metadata:
+  name: vmagent-main-secret-lister
+  namespace: monitoring
+rules:
+  - apiGroups: [""]
+    resources: ["secrets"]
+    verbs: ["get", "list", "watch"]
+---
+apiVersion: rbac.authorization.k8s.io/v1
+kind: RoleBinding
+metadata:
+  name: vmagent-main-secret-lister
+  namespace: monitoring
+roleRef:
+  apiGroup: rbac.authorization.k8s.io
+  kind: Role
+  name: vmagent-main-secret-lister
+subjects:
+  - kind: ServiceAccount
+    name: vmagent-main
+    namespace: monitoring

platform/observability/metrics/rules/vm-health.yaml

@@ -0,0 +1,127 @@
+# Ported from ../../../../apps/victoriametrics/rules/alerts-health.yml
+apiVersion: operator.victoriametrics.com/v1beta1
+kind: VMRule
+metadata:
+  name: vm-health
+  namespace: monitoring
+  labels:
+    app.kubernetes.io/part-of: victoria-metrics
+spec:
+  groups:
+    - name: vm-health
+      rules:
+        - alert: TooManyRestarts
+          expr: changes(process_start_time_seconds{job=~".*(victoriametrics|vmselect|vminsert|vmstorage|vmagent|vmalert|vmsingle|vmalertmanager|vmauth).*"}[15m]) > 2
+          labels:
+            severity: critical
+          annotations:
+            summary: "{{ $labels.job }} too many restarts (instance {{ $labels.instance }})"
+            description: >
+              Job {{ $labels.job }} (instance {{ $labels.instance }}) has restarted more than twice in the last 15 minutes.
+              It might be crashlooping.
+        - alert: ServiceDown
+          expr: up{job=~".*(victoriametrics|vmselect|vminsert|vmstorage|vmagent|vmalert|vmsingle|vmalertmanager|vmauth).*"} == 0
+          for: 2m
+          labels:
+            severity: critical
+          annotations:
+            summary: "Service {{ $labels.job }} is down on {{ $labels.instance }}"
+            description: "{{ $labels.instance }} of job {{ $labels.job }} has been down for more than 2 minutes."
+        - alert: ProcessNearFDLimits
+          expr: (process_max_fds - process_open_fds) < 100
+          for: 5m
+          labels:
+            severity: critical
+          annotations:
+            summary: "Number of free file descriptors is less than 100 for \"{{ $labels.job }}\"(\"{{ $labels.instance }}\") for the last 5m"
+            description: |
+              Exhausting OS file descriptors limit can cause severe degradation of the process.
+              Consider to increase the limit as fast as possible.
+        - alert: TooHighMemoryUsage
+          expr: (min_over_time(process_resident_memory_anon_bytes[10m]) / vm_available_memory_bytes) > 0.8
+          for: 5m
+          labels:
+            severity: critical
+          annotations:
+            summary: "It is more than 80% of memory used by \"{{ $labels.job }}\"(\"{{ $labels.instance }}\")"
+            description: |
+              Too high memory usage may result into multiple issues such as OOMs or degraded performance.
+              Consider to either increase available memory or decrease the load on the process.
+        - alert: TooHighCPUUsage
+          expr: rate(process_cpu_seconds_total[5m]) / process_cpu_cores_available > 0.9
+          for: 5m
+          labels:
+            severity: critical
+          annotations:
+            summary: "More than 90% of CPU is used by \"{{ $labels.job }}\"(\"{{ $labels.instance }}\") during the last 5m"
+            description: >
+              Too high CPU usage may be a sign of insufficient resources and make process unstable.
+              Consider to either increase available CPU resources or decrease the load on the process.
+        - alert: TooHighGoroutineSchedulingLatency
+          expr: histogram_quantile(0.99, sum(rate(go_sched_latencies_seconds_bucket{job=~".*(victoriametrics|vmselect|vminsert|vmstorage|vmagent|vmalert|vmsingle|vmalertmanager|vmauth).*"}[5m])) by (le, job, instance)) > 0.1
+          for: 15m
+          labels:
+            severity: critical
+          annotations:
+            summary: "\"{{ $labels.job }}\"(\"{{ $labels.instance }}\") has insufficient CPU resources for >15m"
+            description: >
+              Go runtime is unable to schedule goroutines execution in acceptable time. This is usually a sign of
+              insufficient CPU resources or CPU throttling.
+        - alert: TooManyLogs
+          expr: sum(increase(vm_log_messages_total{level="error"}[5m])) without (app_version, location) > 0
+          for: 15m
+          labels:
+            severity: warning
+          annotations:
+            summary: "Too many logs printed for job \"{{ $labels.job }}\" ({{ $labels.instance }})"
+            description: >
+              Logging rate for job \"{{ $labels.job }}\" ({{ $labels.instance }}) is {{ $value }} for last 15m.
+              Worth to check logs for specific error messages.
+        - alert: TooManyTSIDMisses
+          expr: increase(vm_missing_tsids_for_metric_id_total[5m]) > 0
+          for: 15m
+          labels:
+            severity: critical
+          annotations:
+            summary: "Unexpected TSID misses for job \"{{ $labels.job }}\" ({{ $labels.instance }}) for the last 15 minutes"
+            description: |
+              Unexpected TSID misses for \"{{ $labels.job }}\" ({{ $labels.instance }}) for the last 15 minutes.
+              If this happens after unclean shutdown of VictoriaMetrics process (via \"kill -9\", OOM or power off),
+              then this is OK - the alert must go away in a few minutes after the restart.
+        - alert: ConcurrentInsertsHitTheLimit
+          expr: avg_over_time(vm_concurrent_insert_current[1m]) >= vm_concurrent_insert_capacity
+          for: 15m
+          labels:
+            severity: warning
+          annotations:
+            summary: "{{ $labels.job }} on instance {{ $labels.instance }} is constantly hitting concurrent inserts limit"
+            description: |
+              The limit of concurrent inserts on instance {{ $labels.instance }} depends on the number of CPUs.
+              Usually, when component constantly hits the limit it is likely the component is overloaded and requires more CPU.
+        - alert: IndexDBRecordsDrop
+          expr: increase(vm_indexdb_items_dropped_total[5m]) > 0
+          labels:
+            severity: critical
+          annotations:
+            summary: "IndexDB skipped registering items during data ingestion with reason={{ $labels.reason }}."
+            description: |
+              VictoriaMetrics could skip registering new timeseries during ingestion if they fail the validation process.
+        - alert: RowsRejectedOnIngestion
+          expr: rate(vm_rows_ignored_total[5m]) > 0
+          for: 15m
+          labels:
+            severity: warning
+          annotations:
+            summary: "Some rows are rejected on \"{{ $labels.instance }}\" on ingestion attempt"
+            description: "Ingested rows on instance \"{{ $labels.instance }}\" are rejected due to the
+              following reason: \"{{ $labels.reason }}\""
+        - alert: TooHighQueryLoad
+          expr: increase(vm_concurrent_select_limit_timeout_total[5m]) > 0
+          for: 15m
+          labels:
+            severity: warning
+          annotations:
+            summary: "Read queries fail with timeout for {{ $labels.job }} on instance {{ $labels.instance }}"
+            description: |
+              Instance {{ $labels.instance }} ({{ $labels.job }}) is failing to serve read queries during last 15m.
+              Possible solutions: reduce the query load; increase compute resources; adjust search concurrency limits.

platform/observability/metrics/rules/vmagent.yaml

@@ -0,0 +1,137 @@
+# Ported from ../../../../apps/victoriametrics/rules/alerts-vmagent.yml
+apiVersion: operator.victoriametrics.com/v1beta1
+kind: VMRule
+metadata:
+  name: vmagent
+  namespace: monitoring
+  labels:
+    app.kubernetes.io/part-of: victoria-metrics
+spec:
+  groups:
+    - name: vmagent
+      interval: 30s
+      concurrency: 2
+      rules:
+        - alert: PersistentQueueIsDroppingData
+          expr: sum(increase(vm_persistentqueue_bytes_dropped_total[5m])) without (path) > 0
+          for: 10m
+          labels:
+            severity: critical
+          annotations:
+            summary: "Instance {{ $labels.instance }} is dropping data from persistent queue"
+            description: "Vmagent dropped {{ $value | humanize1024 }} from persistent queue
+                on instance {{ $labels.instance }} for the last 10m."
+        - alert: RejectedRemoteWriteDataBlocksAreDropped
+          expr: sum(increase(vmagent_remotewrite_packets_dropped_total[5m])) without (url) > 0
+          for: 15m
+          labels:
+            severity: warning
+          annotations:
+            summary: "Vmagent is dropping data blocks that are rejected by remote storage"
+            description: "Job \"{{ $labels.job }}\" on instance {{ $labels.instance }} drops the rejected by
+              remote-write server data blocks. Check the logs to find the reason for rejects."
+        - alert: TooManyScrapeErrors
+          expr: increase(vm_promscrape_scrapes_failed_total[5m]) > 0
+          for: 15m
+          labels:
+            severity: warning
+          annotations:
+            summary: "Vmagent fails to scrape one or more targets"
+            description: "Job \"{{ $labels.job }}\" on instance {{ $labels.instance }} fails to scrape targets for last 15m"
+        - alert: ScrapePoolHasNoTargets
+          expr: sum(vm_promscrape_scrape_pool_targets) without (status, instance, pod) == 0
+          for: 30m
+          labels:
+            severity: warning
+          annotations:
+            summary: "Vmagent has scrape_pool with 0 configured/discovered targets"
+            description: "Vmagent \"{{ $labels.job }}\" has scrape_pool \"{{ $labels.scrape_job }}\"
+              with 0 discovered targets. It is likely a misconfiguration."
+        - alert: TooManyWriteErrors
+          expr: |
+            (sum(increase(vm_ingestserver_request_errors_total[5m])) without (name,net,type)
+            +
+            sum(increase(vmagent_http_request_errors_total[5m])) without (path,protocol)) > 0
+          for: 15m
+          labels:
+            severity: warning
+          annotations:
+            summary: "Vmagent responds with too many errors on data ingestion protocols"
+            description: "Job \"{{ $labels.job }}\" on instance {{ $labels.instance }} responds with errors to write requests for last 15m."
+        - alert: TooManyRemoteWriteErrors
+          expr: rate(vmagent_remotewrite_retries_count_total[5m]) > 0
+          for: 15m
+          labels:
+            severity: warning
+          annotations:
+            summary: "Job \"{{ $labels.job }}\" on instance {{ $labels.instance }} fails to push to remote storage"
+            description: "Vmagent fails to push data via remote write protocol to destination \"{{ $labels.url }}\".
+              Ensure that destination is up and reachable."
+        - alert: RemoteWriteConnectionIsSaturated
+          expr: |
+            (
+             rate(vmagent_remotewrite_send_duration_seconds_total[5m])
+             /
+             vmagent_remotewrite_queues
+            ) > 0.9
+          for: 15m
+          labels:
+            severity: warning
+          annotations:
+            summary: "Remote write connection from \"{{ $labels.job }}\" (instance {{ $labels.instance }}) to {{ $labels.url }} is saturated"
+            description: "The remote write connection between vmagent and destination \"{{ $labels.url }}\"
+              is saturated by more than 90%. Increase -remoteWrite.queues or check the destination's capacity."
+        - alert: PersistentQueueForWritesIsSaturated
+          expr: rate(vm_persistentqueue_write_duration_seconds_total[5m]) > 0.9
+          for: 15m
+          labels:
+            severity: warning
+          annotations:
+            summary: "Persistent queue writes for instance {{ $labels.instance }} are saturated"
+            description: "Persistent queue writes are saturated by more than 90%. Reduce load or improve disk throughput."
+        - alert: PersistentQueueForReadsIsSaturated
+          expr: rate(vm_persistentqueue_read_duration_seconds_total[5m]) > 0.9
+          for: 15m
+          labels:
+            severity: warning
+          annotations:
+            summary: "Persistent queue reads for instance {{ $labels.instance }} are saturated"
+            description: "Persistent queue reads are saturated by more than 90%. Reduce load or improve disk throughput."
+        - alert: SeriesLimitHourReached
+          expr: (vmagent_hourly_series_limit_current_series / vmagent_hourly_series_limit_max_series) > 0.9
+          labels:
+            severity: critical
+          annotations:
+            summary: "Instance {{ $labels.instance }} reached 90% of the hourly series limit"
+            description: "Max series limit set via -remoteWrite.maxHourlySeries is close to the max value."
+        - alert: SeriesLimitDayReached
+          expr: (vmagent_daily_series_limit_current_series / vmagent_daily_series_limit_max_series) > 0.9
+          labels:
+            severity: critical
+          annotations:
+            summary: "Instance {{ $labels.instance }} reached 90% of the daily series limit"
+            description: "Max series limit set via -remoteWrite.maxDailySeries is close to the max value."
+        - alert: ConfigurationReloadFailure
+          expr: |
+            vm_promscrape_config_last_reload_successful != 1
+            or
+            vmagent_relabel_config_last_reload_successful != 1
+          labels:
+            severity: warning
+          annotations:
+            summary: "Configuration reload failed for vmagent instance {{ $labels.instance }}"
+            description: "Configuration hot-reload failed for vmagent on instance {{ $labels.instance }}. Check the logs."
+        - alert: StreamAggrFlushTimeout
+          expr: increase(vm_streamaggr_flush_timeouts_total[5m]) > 0
+          labels:
+            severity: warning
+          annotations:
+            summary: "Streaming aggregation at \"{{ $labels.job }}\" (instance {{ $labels.instance }}) can't keep up."
+            description: "Stream aggregation can't finish within the configured interval and may produce incorrect results."
+        - alert: StreamAggrDedupFlushTimeout
+          expr: increase(vm_streamaggr_dedup_flush_timeouts_total[5m]) > 0
+          labels:
+            severity: warning
+          annotations:
+            summary: "Deduplication \"{{ $labels.job }}\" (instance {{ $labels.instance }}) can't keep up."
+            description: "Deduplication can't finish within the configured interval and may produce incorrect results."

platform/observability/metrics/rules/vmalert.yaml

@@ -0,0 +1,77 @@
+# Ported from ../../../../apps/victoriametrics/rules/alerts-vmalert.yml
+apiVersion: operator.victoriametrics.com/v1beta1
+kind: VMRule
+metadata:
+  name: vmalert
+  namespace: monitoring
+  labels:
+    app.kubernetes.io/part-of: victoria-metrics
+spec:
+  groups:
+    - name: vmalert
+      interval: 30s
+      rules:
+        - alert: ConfigurationReloadFailure
+          expr: vmalert_config_last_reload_successful != 1
+          labels:
+            severity: warning
+          annotations:
+            summary: "Configuration reload failed for vmalert instance {{ $labels.instance }}"
+            description: "Configuration hot-reload failed for vmalert on instance {{ $labels.instance }}. Check the logs."
+        - alert: AlertingRulesError
+          expr: sum(increase(vmalert_alerting_rules_errors_total[5m])) without(id) > 0
+          for: 5m
+          labels:
+            severity: warning
+          annotations:
+            summary: "Alerting rules are failing for vmalert instance {{ $labels.instance }}"
+            description: "Alerting rules execution is failing for group \"{{ $labels.group }}\" in file \"{{ $labels.file }}\". Check the logs."
+        - alert: RecordingRulesError
+          expr: sum(increase(vmalert_recording_rules_errors_total[5m])) without(id) > 0
+          for: 5m
+          labels:
+            severity: warning
+          annotations:
+            summary: "Recording rules are failing for vmalert instance {{ $labels.instance }}"
+            description: "Recording rules execution is failing for group \"{{ $labels.group }}\" in file \"{{ $labels.file }}\". Check the logs."
+        - alert: RecordingRulesNoData
+          expr: sum(vmalert_recording_rules_last_evaluation_samples) without(id) < 1
+          for: 30m
+          labels:
+            severity: info
+          annotations:
+            summary: "Recording rule ({{ $labels.group }}) produces no data"
+            description: "Recording rule from group \"{{ $labels.group }}\" in file \"{{ $labels.file }}\"
+              produces 0 samples over the last 30min. Possible misconfiguration or incorrect query."
+        - alert: TooManyMissedIterations
+          expr: increase(vmalert_iteration_missed_total[5m]) > 0
+          for: 15m
+          labels:
+            severity: warning
+          annotations:
+            summary: "vmalert instance {{ $labels.instance }} is missing rules evaluations"
+            description: "Group evaluation takes longer than the configured interval. Increase interval or concurrency."
+        - alert: RemoteWriteErrors
+          expr: increase(vmalert_remotewrite_errors_total[5m]) > 0
+          for: 15m
+          labels:
+            severity: warning
+          annotations:
+            summary: "vmalert instance {{ $labels.instance }} is failing to push metrics to remote write URL"
+            description: "vmalert is failing to push alerting/recording rule metrics to remote write. Check the logs."
+        - alert: RemoteWriteDroppingData
+          expr: increase(vmalert_remotewrite_dropped_rows_total[5m]) > 0
+          for: 5m
+          labels:
+            severity: critical
+          annotations:
+            summary: "vmalert instance {{ $labels.instance }} is dropping data sent to remote write URL"
+            description: "vmalert is dropping alerting/recording rule results. This may cause gaps in state. Check the logs."
+        - alert: AlertmanagerErrors
+          expr: increase(vmalert_alerts_send_errors_total[5m]) > 0
+          for: 15m
+          labels:
+            severity: warning
+          annotations:
+            summary: "vmalert instance {{ $labels.instance }} is failing to send notifications to Alertmanager"
+            description: "vmalert is failing to send alert notifications to \"{{ $labels.addr }}\". Check the logs."

platform/observability/metrics/rules/vmsingle.yaml

@@ -0,0 +1,127 @@
+# Ported from ../../../../apps/victoriametrics/rules/alerts.yml
+apiVersion: operator.victoriametrics.com/v1beta1
+kind: VMRule
+metadata:
+  name: vmsingle
+  namespace: monitoring
+  labels:
+    app.kubernetes.io/part-of: victoria-metrics
+spec:
+  groups:
+    - name: vmsingle
+      interval: 30s
+      concurrency: 2
+      rules:
+        - alert: DiskRunsOutOfSpaceIn3Days
+          expr: |
+            sum(vm_free_disk_space_bytes) without(path) /
+            (
+              (rate(vm_rows_added_to_storage_total[1d]) - sum(rate(vm_deduplicated_samples_total[1d])) without(type)) * (
+                sum(vm_data_size_bytes{type!~"indexdb.*"}) without(type) /
+                sum(vm_rows{type!~"indexdb.*"}) without(type)
+              )
+              +
+              rate(vm_new_timeseries_created_total[1d]) * (
+                sum(vm_data_size_bytes{type="indexdb/file"}) without(type)/
+                sum(vm_rows{type="indexdb/file"}) without(type)
+              )
+            ) < 3 * 24 * 3600 > 0
+          for: 30m
+          labels:
+            severity: critical
+          annotations:
+            summary: "Instance {{ $labels.instance }} will run out of disk space soon"
+            description: "Taking into account current ingestion rate, free disk space will be enough only
+              for {{ $value | humanizeDuration }} on instance {{ $labels.instance }}.\n
+              Consider to limit the ingestion rate, decrease retention or scale the disk space if possible."
+        - alert: NodeBecomesReadonlyIn3Days
+          expr: |
+            sum(vm_free_disk_space_bytes - vm_free_disk_space_limit_bytes) without(path) /
+            (
+                (rate(vm_rows_added_to_storage_total[1d]) - sum(rate(vm_deduplicated_samples_total[1d])) without(type)) * (
+                  sum(vm_data_size_bytes{type!~"indexdb.*"}) without(type) /
+                  sum(vm_rows{type!~"indexdb.*"}) without(type)
+                )
+                +
+                rate(vm_new_timeseries_created_total[1d]) * (
+                  sum(vm_data_size_bytes{type="indexdb/file"}) without(type) /
+                  sum(vm_rows{type="indexdb/file"}) without(type)
+                )
+            ) < 3 * 24 * 3600 > 0
+          for: 30m
+          labels:
+            severity: warning
+          annotations:
+            summary: "Instance {{ $labels.instance }} will become read-only in 3 days"
+            description: "Taking into account current ingestion rate and free disk space
+                instance {{ $labels.instance }} is writable for {{ $value | humanizeDuration }}.\n
+                Consider to limit the ingestion rate, decrease retention or scale the disk space up if possible."
+        - alert: DiskRunsOutOfSpace
+          expr: |
+            sum(vm_data_size_bytes) by(job, instance) /
+            (
+             sum(vm_free_disk_space_bytes) by(job, instance) +
+             sum(vm_data_size_bytes) by(job, instance)
+            ) > 0.8
+          for: 30m
+          labels:
+            severity: critical
+          annotations:
+            summary: "Instance {{ $labels.instance }} (job={{ $labels.job }}) will run out of disk space soon"
+            description: "Disk utilisation on instance {{ $labels.instance }} is more than 80%.\n
+              Having less than 20% of free disk space could cripple merge processes and overall performance.
+              Consider to limit the ingestion rate, decrease retention or scale the disk space if possible."
+        - alert: RequestErrorsToAPI
+          expr: increase(vm_http_request_errors_total[5m]) > 0
+          for: 15m
+          labels:
+            severity: warning
+          annotations:
+            summary: "Too many errors served for path {{ $labels.path }} (instance {{ $labels.instance }})"
+            description: "Requests to path {{ $labels.path }} are receiving errors.
+              Please verify if clients are sending correct requests."
+        - alert: TooHighChurnRate
+          expr: |
+            (
+               sum(rate(vm_new_timeseries_created_total[5m])) by(instance)
+               /
+               sum(rate(vm_rows_inserted_total[5m])) by(instance)
+             ) > 0.1
+          for: 15m
+          labels:
+            severity: warning
+          annotations:
+            summary: "Churn rate is more than 10% on \"{{ $labels.instance }}\" for the last 15m"
+            description: "VM constantly creates new time series on \"{{ $labels.instance }}\".\n
+              This effect is known as Churn Rate.\n
+              High Churn Rate is tightly connected with database performance and may
+              result in unexpected OOM's or slow queries."
+        - alert: TooHighChurnRate24h
+          expr: |
+            sum(increase(vm_new_timeseries_created_total[24h])) by(instance)
+            >
+            (sum(vm_cache_entries{type="storage/hour_metric_ids"}) by(instance) * 3)
+          for: 15m
+          labels:
+            severity: warning
+          annotations:
+            summary: "Too high number of new series on \"{{ $labels.instance }}\" created over last 24h"
+            description: "The number of created new time series over last 24h is 3x times higher than
+              current number of active series on \"{{ $labels.instance }}\".\n
+              This effect is known as Churn Rate.\n
+              High Churn Rate is tightly connected with database performance and may
+              result in unexpected OOM's or slow queries."
+        - alert: TooHighSlowInsertsRate
+          expr: |
+            (
+               sum(rate(vm_slow_row_inserts_total[5m])) by(instance)
+               /
+               sum(rate(vm_rows_inserted_total[5m])) by(instance)
+             ) > 0.05
+          for: 15m
+          labels:
+            severity: warning
+          annotations:
+            summary: "Percentage of slow inserts is more than 5% on \"{{ $labels.instance }}\" for the last 15m"
+            description: "High rate of slow inserts on \"{{ $labels.instance }}\" may be a sign of resource exhaustion
+              for the current load. It is likely more RAM is needed for optimal handling of the current number of active time series."

platform/observability/metrics/scrapes/blocky.yaml

@@ -0,0 +1,30 @@
+# Blocky DNS metrics. Same VMStaticScrape pattern as docker-hosts.yaml, because
+# Blocky runs as a compose stack on the laptop rather than in the cluster
+# (see ../../../../apps/blocky/README.md for why DNS deliberately lives outside k3s).
+#
+# ⚠ Blocky is STAGED, not deployed — this target will be DOWN until it is started.
+# That is expected; do not go hunting for a broken exporter.
+#
+# Worth alerting on once it is live, because there is currently no DNS visibility
+# at all:
+#   blocky_error_total                     — upstream failures
+#   blocky_denylist_cache_entries          — collapses to 0 if a list fetch fails
+#   blocky_cache_hits_total / _misses_total — cache effectiveness
+#   blocky_query_total{response_type=...}  — BLOCKED vs CACHED vs RESOLVED split
+---
+apiVersion: operator.victoriametrics.com/v1beta1
+kind: VMStaticScrape
+metadata:
+  name: blocky
+  namespace: monitoring
+  labels:
+    app.kubernetes.io/part-of: victoria-metrics
+spec:
+  jobName: blocky
+  targetEndpoints:
+    - targets:
+        - "192.168.10.127:4000"
+      path: /metrics
+      labels:
+        job: blocky
+        host: docker-01

platform/observability/metrics/scrapes/docker-hosts.yaml

@@ -0,0 +1,29 @@
+# Pull metrics from Docker hosts running ../../docker-hosts/compose.yaml.
+# vmagent scrapes each host's node-exporter (:9100) and cAdvisor (:8080) over the LAN.
+# Add one target block per host; keep the `host` label in sync with its HOST_LABEL.
+apiVersion: operator.victoriametrics.com/v1beta1
+kind: VMStaticScrape
+metadata:
+  name: docker-hosts
+  namespace: monitoring
+  labels:
+    app.kubernetes.io/part-of: victoria-metrics
+spec:
+  jobName: docker-hosts
+  targetEndpoints:
+    # --- docker-01 (192.168.10.127) ---
+    - targets:
+        - "192.168.10.127:9100"
+      labels:
+        job: node-exporter
+        host: docker-01
+    - targets:
+        - "192.168.10.127:8080"
+      labels:
+        job: cadvisor
+        host: docker-01
+    # --- add more hosts below, mirroring the two blocks above ---
+    # - targets: ["192.168.10.x:9100"]
+    #   labels: { job: node-exporter, host: docker-02 }
+    # - targets: ["192.168.10.x:8080"]
+    #   labels: { job: cadvisor, host: docker-02 }

platform/observability/metrics/scrapes/kubelet.yaml

@@ -0,0 +1,47 @@
+# Kubelet + cAdvisor metrics (per-node pod/container CPU, memory, network, fs).
+# Scraped over the kubelet's authenticated https port using the vmagent pod's
+# ServiceAccount token. Pattern from the operator docs (VMNodeScrape / cadvisor).
+apiVersion: operator.victoriametrics.com/v1beta1
+kind: VMNodeScrape
+metadata:
+  name: kubelet
+  namespace: monitoring
+  labels:
+    app.kubernetes.io/part-of: victoria-metrics
+spec:
+  scheme: https
+  honorLabels: true
+  honorTimestamps: false
+  interval: 30s
+  tlsConfig:
+    insecureSkipVerify: true
+    caFile: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
+  bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token
+  relabelConfigs:
+    - action: labelmap
+      regex: __meta_kubernetes_node_label_(.+)
+    - targetLabel: job
+      replacement: kubelet
+---
+apiVersion: operator.victoriametrics.com/v1beta1
+kind: VMNodeScrape
+metadata:
+  name: cadvisor
+  namespace: monitoring
+  labels:
+    app.kubernetes.io/part-of: victoria-metrics
+spec:
+  scheme: https
+  honorLabels: true
+  honorTimestamps: false
+  interval: 30s
+  path: /metrics/cadvisor
+  tlsConfig:
+    insecureSkipVerify: true
+    caFile: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
+  bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token
+  relabelConfigs:
+    - action: labelmap
+      regex: __meta_kubernetes_node_label_(.+)
+    - targetLabel: job
+      replacement: cadvisor

platform/observability/metrics/vmagent.yaml

@@ -0,0 +1,21 @@
+# Scrape agent. selectAllByDefault picks up every VMServiceScrape / VMPodScrape /
+# VMNodeScrape / VMStaticScrape in ALL namespaces, so app teams add their own
+# scrape CRs and vmagent discovers them automatically. Writes to VMSingle.
+apiVersion: operator.victoriametrics.com/v1beta1
+kind: VMAgent
+metadata:
+  name: main
+  namespace: monitoring
+spec:
+  replicaCount: 1
+  selectAllByDefault: true
+  scrapeInterval: 30s
+  remoteWrite:
+    - url: http://vmsingle-main.monitoring.svc:8428/api/v1/write
+  resources:
+    requests:
+      cpu: 100m
+      memory: 256Mi
+    limits:
+      cpu: "1"
+      memory: 1Gi

platform/observability/metrics/vmalert.yaml

@@ -0,0 +1,27 @@
+# Rule evaluator. selectAllByDefault picks up every VMRule in all namespaces
+# (see ./rules). Reads from and writes alert/recording state to VMSingle;
+# sends firing alerts to VMAlertmanager.
+apiVersion: operator.victoriametrics.com/v1beta1
+kind: VMAlert
+metadata:
+  name: main
+  namespace: monitoring
+spec:
+  replicaCount: 1
+  selectAllByDefault: true
+  evaluationInterval: 30s
+  datasource:
+    url: http://vmsingle-main.monitoring.svc:8428
+  remoteWrite:
+    url: http://vmsingle-main.monitoring.svc:8428
+  remoteRead:
+    url: http://vmsingle-main.monitoring.svc:8428
+  notifier:
+    url: http://vmalertmanager-main.monitoring.svc:9093
+  resources:
+    requests:
+      cpu: 50m
+      memory: 128Mi
+    limits:
+      cpu: 500m
+      memory: 512Mi

platform/observability/metrics/vmalertmanager.yaml

@@ -0,0 +1,30 @@
+# Alert router/notifier. Ported 1:1 from ../../../apps/victoriametrics/alertmanager.yaml,
+# which currently blackholes everything. Wire real receivers here (email via the
+# in-cluster smtp-relay, or a webhook) when you want notifications.
+apiVersion: operator.victoriametrics.com/v1beta1
+kind: VMAlertmanager
+metadata:
+  name: main
+  namespace: monitoring
+spec:
+  replicaCount: 1
+  configRawYaml: |
+    route:
+      receiver: blackhole
+    receivers:
+      - name: blackhole
+    # Example email receiver via the in-cluster Postfix relay (smtp-relay/):
+    # receivers:
+    #   - name: email
+    #     email_configs:
+    #       - to: '[email protected]'
+    #         from: 'Alertmanager <[email protected]>'
+    #         smarthost: 'smtp-relay.smtp-relay.svc.cluster.local:25'
+    #         require_tls: false
+  resources:
+    requests:
+      cpu: 25m
+      memory: 64Mi
+    limits:
+      cpu: 200m
+      memory: 256Mi

platform/observability/metrics/vmsingle.yaml

@@ -0,0 +1,26 @@
+# Single-node metrics TSDB. Replaces the Compose `victoriametrics` service.
+# Service exposed by the operator: vmsingle-main.monitoring.svc:8428
+apiVersion: operator.victoriametrics.com/v1beta1
+kind: VMSingle
+metadata:
+  name: main
+  namespace: monitoring
+spec:
+  retentionPeriod: "6"          # months
+  removePvcAfterDelete: false   # keep data if the CR is deleted
+  storage:
+    accessModes:
+      - ReadWriteOnce
+    storageClassName: localpv-zfs-ceph   # OpenEBS ZFS (data/ceph pool)
+    resources:
+      requests:
+        storage: 50Gi
+  extraArgs:
+    dedup.minScrapeInterval: 30s
+  resources:
+    requests:
+      cpu: 250m
+      memory: 512Mi
+    limits:
+      cpu: "2"
+      memory: 4Gi

platform/observability/namespace.yaml

@@ -0,0 +1,6 @@
+apiVersion: v1
+kind: Namespace
+metadata:
+  name: monitoring
+  labels:
+    app.kubernetes.io/part-of: observability

platform/observability/operator/helm.sh

@@ -0,0 +1,20 @@
+#!/usr/bin/env bash
+# Install / upgrade the VictoriaMetrics operator into the monitoring namespace.
+# The operator owns the VM* + VLSingle CRDs that everything else in this folder relies on.
+set -euo pipefail
+
+kubectl apply -f ../namespace.yaml
+
+helm repo add vm https://victoriametrics.github.io/helm-charts/
+helm repo update vm
+
+# Pin the chart version after the first install for reproducibility:
+#   helm search repo vm/victoria-metrics-operator --versions | head
+# then add:  --version <x.y.z>
+helm upgrade --install vm-operator vm/victoria-metrics-operator \
+  --namespace monitoring \
+  --values values.yaml \
+  --wait
+
+# CRDs land cluster-wide; verify:
+#   kubectl get crd | grep victoriametrics

platform/observability/operator/values.yaml

@@ -0,0 +1,38 @@
+# victoria-metrics-operator — manages the VM* CRDs (VMSingle, VMAgent, VMAlert,
+# VMAlertmanager, VMRule, VMServiceScrape, VMNodeScrape, ...) AND VLSingle (VictoriaLogs).
+# One operator reconciles the whole metrics + logs stack declaratively.
+#
+# Install: ./helm.sh    (chart: vm/victoria-metrics-operator)
+
+# Let the chart install/upgrade the CRDs.
+crds:
+  plain: true
+  cleanup:
+    enabled: false        # keep CRDs (and thus CRs) if the operator is uninstalled
+
+# Watch every namespace so app-owned VMServiceScrape/VMPodScrape/VMRule are picked up.
+operator:
+  # convert legacy Prometheus-operator CRs too, harmless if none exist
+  disable_prometheus_converter: false
+
+# Pin VictoriaMetrics' native config-reloader (this is already the operator default).
+# NOTE: this reloader watches the config Secret via a client-go informer (list+watch).
+# The operator-generated vmagent Role grants secrets [get,watch] but NOT [list], so on
+# k3s the informer fails and scrape-config hot-reload stalls until vmagent is restarted.
+# See ../metrics/README-reload.md for the fix options.
+env:
+  - name: VM_USECUSTOMCONFIGRELOADER
+    value: "true"
+
+# The operator only reconciles CRs; the actual components (vmsingle, vmagent, ...) are
+# defined as CRs under ../metrics and ../logs. Keep the operator itself lean.
+resources:
+  requests:
+    cpu: 50m
+    memory: 150Mi
+  limits:
+    cpu: 500m
+    memory: 500Mi
+
+serviceMonitor:
+  enabled: true          # let the operator self-scrape via a VMServiceScrape

platform/observability/traces/helm.sh

@@ -0,0 +1,13 @@
+#!/usr/bin/env bash
+# Deploy tracing: VictoriaTraces (VTSingle CR, operator-managed) + the OpenTelemetry
+# Collector that fronts it. Apps send OTLP to otel-collector.monitoring.svc:4317 (gRPC)
+# / :4318 (HTTP); the collector batches and forwards to VictoriaTraces.
+#
+# NOTE: we use the operator's VTSingle CRD (traces/vtsingle.yaml) rather than the
+# victoria-traces-single Helm chart — one operator manages metrics, logs, and traces.
+# traces-values.yaml is kept only as a reference for the standalone-chart alternative.
+set -euo pipefail
+
+kubectl apply -f vtsingle.yaml
+kubectl -n monitoring wait --for=jsonpath='{.status.updateStatus}'=operational vtsingle/main --timeout=120s || true
+kubectl apply -f otel-collector.yaml

platform/observability/traces/otel-collector.yaml

@@ -0,0 +1,117 @@
+# OpenTelemetry Collector — OTLP ingress point for app traces. Receives OTLP over
+# gRPC (4317) and HTTP (4318), batches, and exports to VictoriaTraces' OTLP endpoint.
+# Point your apps' OTEL_EXPORTER_OTLP_ENDPOINT at otel-collector.monitoring.svc:4317.
+apiVersion: v1
+kind: ServiceAccount
+metadata:
+  name: otel-collector
+  namespace: monitoring
+---
+apiVersion: v1
+kind: ConfigMap
+metadata:
+  name: otel-collector-config
+  namespace: monitoring
+data:
+  config.yaml: |
+    receivers:
+      otlp:
+        protocols:
+          grpc:
+            endpoint: 0.0.0.0:4317
+          http:
+            endpoint: 0.0.0.0:4318
+    processors:
+      memory_limiter:
+        check_interval: 5s
+        limit_percentage: 80
+        spike_limit_percentage: 25
+      batch:
+        timeout: 5s
+    exporters:
+      otlphttp/victoriatraces:
+        # VictoriaTraces (VTSingle CR) OTLP-over-HTTP ingestion endpoint.
+        traces_endpoint: http://vtsingle-main.monitoring.svc:10428/insert/opentelemetry/v1/traces
+        tls:
+          insecure: true
+    service:
+      pipelines:
+        traces:
+          receivers: [otlp]
+          processors: [memory_limiter, batch]
+          exporters: [otlphttp/victoriatraces]
+      telemetry:
+        metrics:
+          address: 0.0.0.0:8888
+---
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: otel-collector
+  namespace: monitoring
+  labels:
+    app.kubernetes.io/name: otel-collector
+spec:
+  replicas: 1
+  selector:
+    matchLabels:
+      app.kubernetes.io/name: otel-collector
+  template:
+    metadata:
+      labels:
+        app.kubernetes.io/name: otel-collector
+    spec:
+      serviceAccountName: otel-collector
+      containers:
+        - name: otel-collector
+          # Pin to a current release; check https://github.com/open-telemetry/opentelemetry-collector-releases
+          image: otel/opentelemetry-collector-contrib:0.119.0
+          args: ["--config=/etc/otel/config.yaml"]
+          ports:
+            - { name: otlp-grpc, containerPort: 4317 }
+            - { name: otlp-http, containerPort: 4318 }
+            - { name: metrics, containerPort: 8888 }
+          volumeMounts:
+            - name: config
+              mountPath: /etc/otel
+          resources:
+            requests:
+              cpu: 50m
+              memory: 128Mi
+            limits:
+              cpu: 500m
+              memory: 512Mi
+      volumes:
+        - name: config
+          configMap:
+            name: otel-collector-config
+---
+apiVersion: v1
+kind: Service
+metadata:
+  name: otel-collector
+  namespace: monitoring
+  labels:
+    app.kubernetes.io/name: otel-collector
+spec:
+  selector:
+    app.kubernetes.io/name: otel-collector
+  ports:
+    - { name: otlp-grpc, port: 4317, targetPort: 4317 }
+    - { name: otlp-http, port: 4318, targetPort: 4318 }
+    - { name: metrics, port: 8888, targetPort: 8888 }
+---
+# Self-scrape the collector's own telemetry into VictoriaMetrics.
+apiVersion: operator.victoriametrics.com/v1beta1
+kind: VMServiceScrape
+metadata:
+  name: otel-collector
+  namespace: monitoring
+  labels:
+    app.kubernetes.io/part-of: victoria-metrics
+spec:
+  selector:
+    matchLabels:
+      app.kubernetes.io/name: otel-collector
+  endpoints:
+    - port: metrics

platform/observability/traces/traces-values.yaml

@@ -0,0 +1,21 @@
+# victoria-traces-single — single-node distributed tracing backend. No external
+# storage deps. Ingests OTLP (from the OTel Collector) and exposes a Jaeger Query
+# API that Grafana reads via a Jaeger datasource.
+# Service: victoria-traces-single-server.monitoring.svc:10428
+#
+# NOTE: confirm key names against the installed chart before applying:
+#   helm show values vm/victoria-traces-single
+
+server:
+  retentionPeriod: 30d
+  persistentVolume:
+    enabled: true
+    storageClassName: localpv-zfs-ceph
+    size: 20Gi
+  resources:
+    requests:
+      cpu: 100m
+      memory: 256Mi
+    limits:
+      cpu: "1"
+      memory: 2Gi

platform/observability/traces/vtsingle.yaml

@@ -0,0 +1,25 @@
+# Single-node VictoriaTraces, managed by the VM operator (no separate Helm chart needed
+# — the operator ships the VTSingle CRD). Ingests OTLP from the OTel Collector and
+# exposes a Jaeger-compatible query API that Grafana reads.
+# Service: vtsingle-main.monitoring.svc:10428
+apiVersion: operator.victoriametrics.com/v1
+kind: VTSingle
+metadata:
+  name: main
+  namespace: monitoring
+spec:
+  retentionPeriod: "30d"
+  storage:
+    accessModes:
+      - ReadWriteOnce
+    storageClassName: localpv-zfs-ceph
+    resources:
+      requests:
+        storage: 20Gi
+  resources:
+    requests:
+      cpu: 100m
+      memory: 256Mi
+    limits:
+      cpu: "1"
+      memory: 2Gi