Establish clean homelab infrastructure baseline
Reorganize the brownfield repository, remove retired and generated artifacts, harden ignore rules, and record the GitOps/IaC redesign.
This commit is contained in:
@@ -0,0 +1,2 @@
|
||||
# Real Cloudflare API token — keep secret.example.yaml as the committed template.
|
||||
secret.yaml
|
||||
@@ -0,0 +1,113 @@
|
||||
# cert-manager — certificate issuance
|
||||
|
||||
Gives LAN services on `ad.ddupan.top` real, auto-renewing X.509 certs. Added 2026-07-25
|
||||
so NetBox could be served at `https://netbox.ad.ddupan.top` without a hand-issued cert.
|
||||
|
||||
- Chart: `jetstack/cert-manager` **v1.21.0**, namespace `cert-manager`
|
||||
- Two `ClusterIssuer`s, because the homelab has two ACME sources
|
||||
|
||||
| issuer | CA | challenge | use it when |
|
||||
|---|---|---|---|
|
||||
| `letsencrypt` | Let's Encrypt (public) | DNS-01 via Cloudflare | browsers must trust it with no CA install |
|
||||
| `bao-acme` | OpenBao internal PKI (`../../infrastructure/openbao`) | HTTP-01 via the shared gateway | no WAN dependency, or the name must stay out of CT logs |
|
||||
|
||||
Both were verified `Ready=True` (ACME account registered) on creation.
|
||||
|
||||
## The wildcard, and why
|
||||
|
||||
`certificate-wildcard-ad.yaml` issues **one** `*.ad.ddupan.top` cert into
|
||||
`envoy-gateway-system/wildcard-ad-ddupan-top-tls`, which the shared Gateway's `https`
|
||||
listener serves for every LAN service (`../envoy-gateway/gateway.yaml`).
|
||||
|
||||
Consequences worth understanding:
|
||||
|
||||
- **Adding a service costs no certificate work.** An `HTTPRoute` plus an A record in
|
||||
`samba_ad_extra_a_records` (`../../infrastructure/samba-ad`) is the whole job. No Gateway edit.
|
||||
- **Internal hostnames stay out of Certificate Transparency logs.** Every Let's Encrypt
|
||||
issuance is published publicly; per-host certs would make the internal estate
|
||||
enumerable by anyone. One wildcard entry leaks one name.
|
||||
- The Secret **must** live in `envoy-gateway-system` (the Gateway's namespace): a listener may only reference a
|
||||
Secret in the Gateway's own namespace unless a `ReferenceGrant` exists.
|
||||
- `*.ad.ddupan.top` matches one label only, so the apex `ad.ddupan.top` is listed as an
|
||||
explicit second SAN.
|
||||
|
||||
## Two traps that will cost you an hour each
|
||||
|
||||
**1. DNS-01 self-check must not use the cluster resolver.**
|
||||
cert-manager polls authoritative nameservers for its `_acme-challenge` TXT record before
|
||||
asking the CA to validate. In-cluster, `ad.ddupan.top` is routed straight to the Samba AD
|
||||
DC (`../k3s/coredns-custom.yaml`), which is authoritative internally and knows nothing
|
||||
about a TXT record written into the **public** Cloudflare zone — so the self-check spins
|
||||
forever while the record is plainly there. Fixed in `values.yaml`:
|
||||
|
||||
```yaml
|
||||
dns01RecursiveNameservers: "1.1.1.1:53,8.8.8.8:53"
|
||||
dns01RecursiveNameserversOnly: true
|
||||
```
|
||||
|
||||
**2. DNS-01, not HTTP-01, for Let's Encrypt here.** `ad.ddupan.top` resolves only on the
|
||||
LAN, so LE cannot reach the host to validate. DNS-01 needs no inbound reachability and is
|
||||
the only challenge that can issue a wildcard.
|
||||
|
||||
## Deploy
|
||||
|
||||
```bash
|
||||
helm repo add jetstack https://charts.jetstack.io && helm repo update jetstack
|
||||
helm upgrade --install cert-manager jetstack/cert-manager --version v1.21.0 \
|
||||
-n cert-manager --create-namespace -f cert-manager/values.yaml
|
||||
|
||||
cp cert-manager/secret.example.yaml cert-manager/secret.yaml # then fill in the token
|
||||
kubectl apply -f cert-manager/secret.yaml
|
||||
kubectl apply -f cert-manager/clusterissuer-letsencrypt.yaml \
|
||||
-f cert-manager/clusterissuer-bao-acme.yaml
|
||||
kubectl apply -f cert-manager/certificate-wildcard-ad.yaml
|
||||
```
|
||||
|
||||
The Cloudflare token is the same one `cloudflared/terraform` uses. Extract it with:
|
||||
|
||||
```bash
|
||||
sed -nE 's/^[[:space:]]*cloudflare_api_token[[:space:]]*=[[:space:]]*"([^"]*)".*/\1/p' \
|
||||
cloudflared/terraform/terraform.tfvars
|
||||
```
|
||||
|
||||
> A naive `cut -d= -f2` also swallows the trailing comment on that line and yields a
|
||||
> 126-character "token" that fails with `Invalid format for Authorization header`.
|
||||
|
||||
The Secret must be in the **cert-manager** namespace — a ClusterIssuer resolves solver
|
||||
Secret refs where cert-manager runs, not where the Certificate lives.
|
||||
|
||||
## ⚠ Token scope
|
||||
|
||||
That token carries **Account·Cloudflare Tunnel:Edit** as well as Zone·DNS:Edit, so
|
||||
anything able to read the Secret could rewrite tunnel routing, not just DNS. Reusing it
|
||||
was a deliberate call to avoid blocking; minting a token scoped to Zone·DNS:Edit on
|
||||
`ddupan.top` and swapping it in is a clean, self-contained follow-up.
|
||||
|
||||
## Verify
|
||||
|
||||
```bash
|
||||
kubectl get clusterissuer # both Ready=True
|
||||
kubectl -n envoy-gateway-system get certificate # wildcard Ready=True
|
||||
kubectl -n envoy-gateway-system get order,challenge # empty once issued
|
||||
|
||||
kubectl -n envoy-gateway-system get secret wildcard-ad-ddupan-top-tls \
|
||||
-o jsonpath='{.data.tls\.crt}' | base64 -d | openssl x509 -noout -subject -issuer -enddate
|
||||
```
|
||||
|
||||
Renewal is automatic at 30 days remaining (`renewBefore: 720h`), a deliberately wide
|
||||
window so a flaky WAN has many chances to retry.
|
||||
|
||||
## Using the internal CA instead
|
||||
|
||||
Point a Certificate at `bao-acme` when CT-log exposure or WAN dependence matters:
|
||||
|
||||
```yaml
|
||||
issuerRef:
|
||||
name: bao-acme
|
||||
kind: ClusterIssuer
|
||||
```
|
||||
|
||||
Issuance is capped by `default_directory_policy = role:bao-server`
|
||||
(`../../infrastructure/openbao/terraform/pki.tf`), which permits `ad.ddupan.top` subdomains only. Clients
|
||||
need the internal CA in their trust store — already true for the PVE nodes, the DC and
|
||||
Authelia, generally **not** for a browser.
|
||||
@@ -0,0 +1,54 @@
|
||||
# Cert for the PUBLIC Authelia hostname, so it can also be served on the LAN.
|
||||
#
|
||||
# WHY this exists at all, when ../envoy-gateway/gateway.yaml says adding a service
|
||||
# needs no cert work: that promise holds only for `*.ad.ddupan.top`. Authelia is
|
||||
# reached at `auth.ddupan.top` — a different zone (Cloudflare is authoritative for
|
||||
# ddupan.top; the DC is authoritative only for ad.ddupan.top) and one label
|
||||
# shallower, so the wildcard cannot cover it.
|
||||
#
|
||||
# WHY serve a public name internally (split-horizon) rather than introduce an
|
||||
# internal alias: the OIDC issuer, every registered redirect_uri, and the session
|
||||
# cookie domain are all `auth.ddupan.top`. Changing the name Gitea talks to would
|
||||
# mean re-registering every client. Resolving the SAME name to the LAN changes
|
||||
# nothing Authelia knows about itself.
|
||||
#
|
||||
# WHAT BROKE WITHOUT IT (2026-07-28): auth.ddupan.top resolves to Cloudflare proxy
|
||||
# IPs (104.21.6.55 / 172.67.154.245). TCP/443 to both fails from this network,
|
||||
# persistently, while other Cloudflare IPs (104.16.132.229) connect fine. Gitea's
|
||||
# chart runs `gitea admin auth update-oauth` in an INIT container, which fetches
|
||||
# the discovery URL on every pod start — so Gitea CrashLoopBackOff'd on any
|
||||
# restart, and server-side token exchange timed out. Routing the name to the LAN
|
||||
# removes the public internet from an entirely in-cluster conversation. See
|
||||
# CLAUDE.md: "Internal name resolution must never depend on the WAN."
|
||||
#
|
||||
# CT-log note: ../cert-manager/certificate-wildcard-ad.yaml deliberately uses a
|
||||
# wildcard to keep internal hostnames out of Certificate Transparency logs. That
|
||||
# reasoning does not apply here — auth.ddupan.top is already public in CT via the
|
||||
# Cloudflare-facing cert, so naming it costs nothing.
|
||||
#
|
||||
# Lives in envoy-gateway-system because a Gateway listener may only reference a
|
||||
# Secret in the Gateway's own namespace.
|
||||
---
|
||||
apiVersion: cert-manager.io/v1
|
||||
kind: Certificate
|
||||
metadata:
|
||||
name: auth-ddupan-top
|
||||
namespace: envoy-gateway-system
|
||||
spec:
|
||||
secretName: auth-ddupan-top-tls
|
||||
issuerRef:
|
||||
name: letsencrypt
|
||||
kind: ClusterIssuer
|
||||
group: cert-manager.io
|
||||
commonName: "auth.ddupan.top"
|
||||
dnsNames:
|
||||
- "auth.ddupan.top"
|
||||
# DNS-01 via Cloudflare (the letsencrypt ClusterIssuer's solver). Cloudflare stays
|
||||
# authoritative for ddupan.top, so the challenge resolves publicly even though the
|
||||
# A record we serve internally points at the LAN.
|
||||
duration: 2160h # 90d — Let's Encrypt maximum
|
||||
renewBefore: 720h # 30d
|
||||
privateKey:
|
||||
algorithm: ECDSA
|
||||
size: 256
|
||||
rotationPolicy: Always
|
||||
@@ -0,0 +1,44 @@
|
||||
# Cert for the PUBLIC Gitea hostname, so it can also be served on the LAN.
|
||||
#
|
||||
# Same reasoning as certificate-auth-ddupan.yaml: `*.ad.ddupan.top` cannot cover
|
||||
# `git.ddupan.top` — different zone (Cloudflare is authoritative for ddupan.top,
|
||||
# the DC only for ad.ddupan.top) and one label shallower.
|
||||
#
|
||||
# WHY serve the public name internally rather than introduce git.ad.ddupan.top:
|
||||
# the remote URL ends up in every clone, every CI checkout, and every existing
|
||||
# working copy. Split-horizon on the SAME name means none of that has to change,
|
||||
# and a laptop that leaves the LAN still reaches Gitea through the tunnel with the
|
||||
# identical URL.
|
||||
#
|
||||
# WHAT IT AVOIDS: without this, `git push` goes laptop -> Cloudflare -> tunnel ->
|
||||
# back into the cluster the laptop is hosting. On 2026-07-28 that path was
|
||||
# blackholed for hours by a dead VPN tunnel, and the repo is exactly what you need
|
||||
# during an incident.
|
||||
#
|
||||
# CT-log note: the wildcard in certificate-wildcard-ad.yaml exists to keep internal
|
||||
# hostnames out of Certificate Transparency. That does not apply here —
|
||||
# git.ddupan.top is already public via the Cloudflare-facing cert.
|
||||
---
|
||||
apiVersion: cert-manager.io/v1
|
||||
kind: Certificate
|
||||
metadata:
|
||||
name: git-ddupan-top
|
||||
namespace: envoy-gateway-system
|
||||
spec:
|
||||
secretName: git-ddupan-top-tls
|
||||
issuerRef:
|
||||
name: letsencrypt
|
||||
kind: ClusterIssuer
|
||||
group: cert-manager.io
|
||||
commonName: "git.ddupan.top"
|
||||
dnsNames:
|
||||
- "git.ddupan.top"
|
||||
# DNS-01 via Cloudflare, which stays authoritative for the zone — so the ACME
|
||||
# challenge resolves publicly even though the A record we serve on the LAN
|
||||
# points at the gateway.
|
||||
duration: 2160h # 90d — Let's Encrypt maximum
|
||||
renewBefore: 720h # 30d
|
||||
privateKey:
|
||||
algorithm: ECDSA
|
||||
size: 256
|
||||
rotationPolicy: Always
|
||||
@@ -0,0 +1,41 @@
|
||||
# One wildcard cert serving every LAN service on the shared Contour gateway.
|
||||
#
|
||||
# WHY A WILDCARD rather than a cert per service:
|
||||
# 1. Certificate Transparency. Per-host LE certs publish every internal hostname
|
||||
# to public CT logs, making the whole internal estate enumerable. A single
|
||||
# *.ad.ddupan.top entry leaks one name and hides the rest.
|
||||
# 2. The Gateway's HTTPS listener needs exactly one certificateRef to cover all
|
||||
# hostnames; a new service then needs only an HTTPRoute + a DNS A record, with
|
||||
# no cert work and no Gateway edit at all.
|
||||
# 3. Fewer ACME orders against Let's Encrypt rate limits.
|
||||
#
|
||||
# Lives in envoy-gateway-system because a Gateway listener may only reference a Secret
|
||||
# in the Gateway's OWN namespace (cross-namespace refs need a ReferenceGrant).
|
||||
# Was projectcontour until the gateway moved to Envoy Gateway — see ../envoy-gateway.
|
||||
#
|
||||
# NOTE the bare apex `ad.ddupan.top` is listed as well: a wildcard covers
|
||||
# one label only, so `*.ad.ddupan.top` does NOT match `ad.ddupan.top` itself.
|
||||
---
|
||||
apiVersion: cert-manager.io/v1
|
||||
kind: Certificate
|
||||
metadata:
|
||||
name: wildcard-ad-ddupan-top
|
||||
namespace: envoy-gateway-system
|
||||
spec:
|
||||
secretName: wildcard-ad-ddupan-top-tls
|
||||
issuerRef:
|
||||
name: letsencrypt
|
||||
kind: ClusterIssuer
|
||||
group: cert-manager.io
|
||||
commonName: "*.ad.ddupan.top"
|
||||
dnsNames:
|
||||
- "*.ad.ddupan.top"
|
||||
- "ad.ddupan.top"
|
||||
# Renew with 30 days to spare. The WAN is unreliable, so leave a wide window for
|
||||
# retries rather than the default cutting it fine.
|
||||
duration: 2160h # 90d — Let's Encrypt maximum
|
||||
renewBefore: 720h # 30d
|
||||
privateKey:
|
||||
algorithm: ECDSA
|
||||
size: 256
|
||||
rotationPolicy: Always
|
||||
@@ -0,0 +1,43 @@
|
||||
# OpenBao's internal PKI over ACME (../../infrastructure/openbao/terraform/pki.tf).
|
||||
#
|
||||
# WHEN TO PREFER THIS OVER letsencrypt:
|
||||
# * the name must never appear in a public Certificate Transparency log
|
||||
# * issuance/renewal must not depend on the WAN (see netbox/CONTEXT.md §6)
|
||||
# * it is a non-web service (LDAPS, Postgres, syslog) where "browser trusts it
|
||||
# out of the box" buys nothing and the internal CA is already distributed
|
||||
#
|
||||
# COST: clients must trust the ddupan.top internal CA. Already true for the PVE
|
||||
# nodes (pve_ca_trust), Authelia, and the DC — generally NOT true of a fresh browser.
|
||||
#
|
||||
# Verified 2026-07-25:
|
||||
# * directory live, "externalAccountRequired": false -> no EAB stanza needed,
|
||||
# matching acme_eab_policy = "not-required" in openbao/terraform
|
||||
# * bao.ad.ddupan.top:8200 serves a REAL Let's Encrypt cert (issuer CN=YE1), so
|
||||
# cert-manager validates it against public roots — no spec.acme.caBundle required
|
||||
# * issuance is capped by default_directory_policy = role:bao-server, which permits
|
||||
# subdomains of ad.ddupan.top only
|
||||
---
|
||||
apiVersion: cert-manager.io/v1
|
||||
kind: ClusterIssuer
|
||||
metadata:
|
||||
name: bao-acme
|
||||
spec:
|
||||
acme:
|
||||
server: https://bao.ad.ddupan.top:8200/v1/pki/acme/directory
|
||||
# OpenBao ignores the contact address, but ACME clients must send something.
|
||||
email: [email protected]
|
||||
privateKeySecretRef:
|
||||
name: bao-acme-account-key
|
||||
solvers:
|
||||
# http-01, not dns01: bao resolves ad.ddupan.top and can reach LAN hosts
|
||||
# directly (noted as verified in openbao/terraform/pki.tf), so it can fetch
|
||||
# the challenge over the LAN with no public exposure. cert-manager creates a
|
||||
# temporary HTTPRoute on the shared Contour gateway to answer it.
|
||||
- http01:
|
||||
gatewayHTTPRoute:
|
||||
parentRefs:
|
||||
- name: contour-gateway
|
||||
namespace: projectcontour
|
||||
kind: Gateway
|
||||
group: gateway.networking.k8s.io
|
||||
sectionName: http # the plaintext :80 listener
|
||||
@@ -0,0 +1,37 @@
|
||||
# Let's Encrypt via DNS-01 (Cloudflare).
|
||||
#
|
||||
# WHY DNS-01 and not HTTP-01: ad.ddupan.top names resolve ONLY on the LAN, so
|
||||
# Let's Encrypt cannot reach http://<host>/.well-known/... to validate. DNS-01
|
||||
# proves control of the name by writing a TXT record into the PUBLIC ddupan.top
|
||||
# Cloudflare zone, which needs no inbound reachability at all. It is also the only
|
||||
# challenge type that can issue a WILDCARD.
|
||||
#
|
||||
# CT-LOG NOTE: every LE-issued name is published to Certificate Transparency logs.
|
||||
# Issuing the single wildcard *.ad.ddupan.top (see certificate-wildcard-ad.yaml)
|
||||
# means only that one entry appears — individual internal hostnames stay private.
|
||||
# For anything that must not appear at all, use the bao-acme issuer instead.
|
||||
---
|
||||
apiVersion: cert-manager.io/v1
|
||||
kind: ClusterIssuer
|
||||
metadata:
|
||||
name: letsencrypt
|
||||
spec:
|
||||
acme:
|
||||
server: https://acme-v02.api.letsencrypt.org/directory
|
||||
email: [email protected]
|
||||
# Account key. cert-manager creates this; losing it just means a new account.
|
||||
privateKeySecretRef:
|
||||
name: letsencrypt-account-key
|
||||
solvers:
|
||||
- dns01:
|
||||
cloudflare:
|
||||
# Token is reused from cloudflared/terraform (see cert-manager/README.md).
|
||||
# ⚠ It also carries Account·Cloudflare Tunnel:Edit, so anything able to
|
||||
# read this Secret can rewrite tunnel routing, not just DNS. Narrowing it
|
||||
# to Zone·DNS:Edit is a worthwhile follow-up.
|
||||
apiTokenSecretRef:
|
||||
name: cloudflare-api-token
|
||||
key: api-token
|
||||
selector:
|
||||
dnsZones:
|
||||
- ddupan.top
|
||||
@@ -0,0 +1,20 @@
|
||||
# Template for cert-manager/secret.yaml (gitignored). Copy, fill in, apply.
|
||||
#
|
||||
# Same token as cloudflared/terraform/terraform.tfvars — extract it with:
|
||||
# sed -nE 's/^[[:space:]]*cloudflare_api_token[[:space:]]*=[[:space:]]*"([^"]*)".*/\1/p' \
|
||||
# cloudflared/terraform/terraform.tfvars
|
||||
# (a naive `cut -d= -f2` also swallows the trailing comment on that line)
|
||||
#
|
||||
# Must live in the cert-manager namespace: a ClusterIssuer resolves solver Secret
|
||||
# refs in the namespace where cert-manager runs, not where the Certificate is.
|
||||
#
|
||||
# ⚠ This token also carries Account·Cloudflare Tunnel:Edit. Replacing it with one
|
||||
# scoped to Zone·DNS:Edit on ddupan.top would shrink the blast radius to DNS.
|
||||
apiVersion: v1
|
||||
kind: Secret
|
||||
metadata:
|
||||
name: cloudflare-api-token
|
||||
namespace: cert-manager
|
||||
type: Opaque
|
||||
stringData:
|
||||
api-token: "REPLACE_WITH_CLOUDFLARE_API_TOKEN"
|
||||
@@ -0,0 +1,55 @@
|
||||
# cert-manager — X.509 issuance for the cluster.
|
||||
#
|
||||
# Chart: jetstack/cert-manager v1.21.0 (repo: https://charts.jetstack.io)
|
||||
#
|
||||
# Exists so LAN services on ad.ddupan.top get real, auto-renewing certs instead of
|
||||
# hand-issued ones. The homelab has TWO ACME sources and both are wired up as
|
||||
# ClusterIssuers (see clusterissuer-*.yaml):
|
||||
# letsencrypt — public CA, DNS-01 via Cloudflare. Browser-trusted with no CA
|
||||
# install. Used for the *.ad.ddupan.top wildcard.
|
||||
# bao-acme — OpenBao's internal PKI (../../infrastructure/openbao). No WAN dependency and
|
||||
# nothing published to Certificate Transparency logs.
|
||||
|
||||
# CRDs are part of the release so `helm uninstall` is a clean removal and there is
|
||||
# no separate kubectl-apply step to forget.
|
||||
crds:
|
||||
enabled: true
|
||||
keep: true # don't let an accidental uninstall garbage-collect live Certificates
|
||||
|
||||
# Single-node k3s: one of everything, modest requests. The laptop runs the whole
|
||||
# homelab (see netbox/CONTEXT.md §6).
|
||||
replicaCount: 1
|
||||
resources:
|
||||
requests:
|
||||
cpu: 10m
|
||||
memory: 64Mi
|
||||
limits:
|
||||
memory: 192Mi
|
||||
|
||||
webhook:
|
||||
replicaCount: 1
|
||||
resources:
|
||||
requests:
|
||||
cpu: 10m
|
||||
memory: 32Mi
|
||||
limits:
|
||||
memory: 128Mi
|
||||
|
||||
cainjector:
|
||||
replicaCount: 1
|
||||
resources:
|
||||
requests:
|
||||
cpu: 10m
|
||||
memory: 96Mi
|
||||
limits:
|
||||
memory: 256Mi
|
||||
|
||||
# ⚠ DNS-01 self-check: cert-manager polls authoritative NS for the _acme-challenge
|
||||
# TXT record before telling the CA to validate. By default it asks the cluster's
|
||||
# resolver, which for ad.ddupan.top is CoreDNS -> the Samba AD DC (k3s/coredns-custom.yaml).
|
||||
# The DC is authoritative for ad.ddupan.top internally and knows nothing about the
|
||||
# TXT record we just wrote into the PUBLIC Cloudflare zone, so the self-check would
|
||||
# spin forever. Forcing public recursive resolvers makes the check see what the CA
|
||||
# will see.
|
||||
dns01RecursiveNameservers: "1.1.1.1:53,8.8.8.8:53"
|
||||
dns01RecursiveNameserversOnly: true
|
||||
@@ -0,0 +1,118 @@
|
||||
# Envoy Gateway — LAN ingress + Authelia forward-auth
|
||||
|
||||
The cluster's HTTP entry point for services on `ad.ddupan.top`, and the enforcement
|
||||
point for Authelia authentication. Envoy's LoadBalancer holds **192.168.10.127** (k3s
|
||||
ServiceLB); routing is by `Host` header.
|
||||
|
||||
- Chart: `oci://docker.io/envoyproxy/gateway-helm` **v1.5.6**, ns `envoy-gateway-system`
|
||||
- `gateway.yaml` — `GatewayClass eg` + `Gateway eg` (`:80` plaintext, `:443` wildcard TLS)
|
||||
- Gateway API **v1.3.0**
|
||||
- Replaced Contour on 2026-07-25 (see [Why not Contour](#why-not-contour))
|
||||
|
||||
## Adding a service
|
||||
|
||||
1. an `HTTPRoute` with `parentRefs` → `eg` / `envoy-gateway-system`, `sectionName: https`
|
||||
2. an A record in `samba_ad_extra_a_records` → `192.168.10.127` (`../../infrastructure/samba-ad`), then
|
||||
`ansible-playbook provision-dc.yml --tags dns`
|
||||
3. **optionally** a `SecurityPolicy` for Authelia forward-auth — see `../../apps/netbox`
|
||||
|
||||
No certificate work: the `:443` listener already serves the `*.ad.ddupan.top` wildcard
|
||||
from `../cert-manager`. Worked example: `../../apps/netbox`.
|
||||
|
||||
## Why not Contour
|
||||
|
||||
Contour worked as an ingress, but supports **only the gRPC** Envoy ext_authz protocol —
|
||||
*"Only the Envoy GRPC authorization protocol will be supported"* (Contour 1.33 docs).
|
||||
Authelia implements the **HTTP** ExtAuthz filter. The two cannot meet, so Authelia
|
||||
forward-auth was impossible.
|
||||
|
||||
That mattered because **NetBox has no SSO group→role mapping** — its group/superuser
|
||||
mapping is LDAP-only, and the OIDC pipeline can only assign one static group. Without
|
||||
forward-auth the options were "promote every user by hand" or "drop SSO and use LDAP,
|
||||
losing 2FA". Envoy Gateway's `SecurityPolicy.extAuth.http` removes the dilemma.
|
||||
|
||||
Envoy Gateway was chosen over Traefik because everything already built — Gateway,
|
||||
HTTPRoute, wildcard cert, DNS — is Gateway API, so only the GatewayClass and controller
|
||||
changed. Traefik would have meant reverting to `IngressRoute` + `Middleware`.
|
||||
|
||||
## Forward-auth: how it fits together
|
||||
|
||||
```
|
||||
browser ──▶ Envoy ──(ext_authz HTTP)──▶ Authelia ──▶ 200 + Remote-* headers
|
||||
│ └──▶ 401 ⇒ 302 to auth.ddupan.top
|
||||
└──▶ upstream app (headers attached)
|
||||
```
|
||||
|
||||
Pieces, each in the directory that owns it:
|
||||
|
||||
| where | what |
|
||||
|---|---|
|
||||
| `../../apps/netbox/securitypolicy.yaml` | `SecurityPolicy` targeting the app's HTTPRoute |
|
||||
| `../../apps/authelia/referencegrant-extauth.yaml` | lets a SecurityPolicy in another namespace reference the Authelia Service |
|
||||
| `../../apps/authelia/values.yaml` | `server.endpoints.authz.ext-authz` + an `access_control` rule |
|
||||
| `../../apps/netbox/networkpolicy.yaml` | stops anything bypassing Envoy to reach the app directly |
|
||||
|
||||
### Three details that cost time
|
||||
|
||||
- **`headersToBackend` lives under `extAuth.http`, not `extAuth`.** One level up the API
|
||||
rejects it: `unknown field "spec.extAuth.headersToBackend"`.
|
||||
- **`backendRefs.port` is the SERVICE port, not the container port.** The Authelia chart
|
||||
publishes `80 → targetPort http (9091)`; using `9091` fails with
|
||||
`TCP Port 9091 not found on service authelia/authelia`.
|
||||
- **Authelia's session cookie is scoped to `ddupan.top`**, which already covers
|
||||
`*.ad.ddupan.top` — so SSO works across both without touching the cookie config.
|
||||
|
||||
### ⚠ Trust boundary
|
||||
|
||||
Apps behind forward-auth trust `Remote-*` headers. Two things make that safe and **both**
|
||||
must remain true:
|
||||
|
||||
1. `headersToBackend` **overrides** any client-supplied value ("coexisting headers will be
|
||||
overridden"), so a spoofed `Remote-User` cannot survive the hop through Envoy.
|
||||
2. A `NetworkPolicy` per app restricts pod ingress to `envoy-gateway-system`, so nothing
|
||||
in-cluster can bypass Envoy. k3s enforces NetworkPolicy (kube-router), so this is real.
|
||||
|
||||
Verified by test: a pod in `default` sending `Remote-User: admin` to the app Service gets
|
||||
**connection refused**, while the same request from `envoy-gateway-system` is served.
|
||||
Also set `failOpen: false` — if Authelia is down, refuse traffic rather than admit
|
||||
unauthenticated requests to an app whose auth model is "trust the header".
|
||||
|
||||
## Install
|
||||
|
||||
```bash
|
||||
# The Gateway API CRDs may already be owned by another tool's field manager (Contour's
|
||||
# quickstart used client-side apply), which makes Helm fail with
|
||||
# "conflict with kubectl-client-side-apply: .spec.versions".
|
||||
# Transfer ownership WITHOUT deleting the CRDs (deleting them would delete every
|
||||
# Gateway and HTTPRoute):
|
||||
helm pull oci://docker.io/envoyproxy/gateway-helm --version v1.5.6 --untar
|
||||
kubectl apply --server-side --force-conflicts --field-manager=helm \
|
||||
-f gateway-helm/crds/gatewayapi-crds.yaml
|
||||
|
||||
helm upgrade --install envoy-gateway oci://docker.io/envoyproxy/gateway-helm \
|
||||
--version v1.5.6 -n envoy-gateway-system --create-namespace
|
||||
|
||||
kubectl apply -f envoy-gateway/gateway.yaml
|
||||
```
|
||||
|
||||
Only one LoadBalancer can hold `:80/:443` — k3s ServiceLB uses hostPorts, so a second
|
||||
one's `svclb-*` DaemonSet sits at `0/1` and the Gateway stays `PROGRAMMED: False` until
|
||||
the previous controller's Service is gone.
|
||||
|
||||
## Verify
|
||||
|
||||
```bash
|
||||
kubectl -n envoy-gateway-system get gateway eg # PROGRAMMED=True, ADDRESS 192.168.10.127
|
||||
kubectl -n envoy-gateway-system get pods # envoy-... 2/2
|
||||
kubectl -n <ns> get securitypolicy # Accepted=True
|
||||
curl -s -o /dev/null -w '%{http_code}\n' http://192.168.10.127/ # 404 = serving, no route matched
|
||||
```
|
||||
|
||||
`ss -lntp` shows **nothing** on `:80/:443` even when healthy — ServiceLB forwards via
|
||||
iptables/NodePort rather than binding. Test with `curl`, not `ss`.
|
||||
|
||||
## Outstanding
|
||||
|
||||
Contour source manifests were removed from the clean baseline. Live leftovers may
|
||||
still exist and must be inventoried before deletion; do not infer live state from
|
||||
the archive cleanup. The legacy Git history retains the retired manifests.
|
||||
@@ -0,0 +1,116 @@
|
||||
# LAN ingress for k3s services on ad.ddupan.top.
|
||||
#
|
||||
# Replaced Contour 2026-07-25. Contour was dropped for one concrete reason: it
|
||||
# supports ONLY the gRPC Envoy ext_authz protocol ("Only the Envoy GRPC authorization
|
||||
# protocol will be supported" — Contour 1.33 docs), while Authelia implements the
|
||||
# HTTP ExtAuthz filter. That made Authelia forward-auth impossible, which in turn made
|
||||
# AD-group -> role mapping impossible for apps like NetBox whose SSO support has no
|
||||
# group mapping. Envoy Gateway's SecurityPolicy supports extAuth.http, so it works.
|
||||
#
|
||||
# Everything else carried over unchanged: the Gateway API objects, the wildcard cert
|
||||
# from ../cert-manager, and the DNS A records in ../../infrastructure/samba-ad.
|
||||
---
|
||||
apiVersion: gateway.networking.k8s.io/v1
|
||||
kind: GatewayClass
|
||||
metadata:
|
||||
name: eg
|
||||
spec:
|
||||
controllerName: gateway.envoyproxy.io/gatewayclass-controller
|
||||
---
|
||||
apiVersion: gateway.networking.k8s.io/v1
|
||||
kind: Gateway
|
||||
metadata:
|
||||
name: eg
|
||||
namespace: envoy-gateway-system
|
||||
spec:
|
||||
gatewayClassName: eg
|
||||
listeners:
|
||||
# Plaintext :80 — ACME http-01 (the bao-acme ClusterIssuer solves through here)
|
||||
# and HTTP->HTTPS redirects. Never serve anything sensitive on it.
|
||||
- name: http
|
||||
protocol: HTTP
|
||||
port: 80
|
||||
allowedRoutes:
|
||||
namespaces:
|
||||
from: All
|
||||
kinds:
|
||||
- group: gateway.networking.k8s.io
|
||||
kind: HTTPRoute
|
||||
|
||||
# HTTPS :443. One wildcard covers every LAN service, so adding a service is an
|
||||
# HTTPRoute + a DNS A record — no cert work and no edit here.
|
||||
#
|
||||
# ⚠ The cert Secret MUST live in this Gateway's namespace (a listener may only
|
||||
# reference a Secret in its own namespace without a ReferenceGrant), which is why
|
||||
# cert-manager/certificate-wildcard-ad.yaml issues into envoy-gateway-system.
|
||||
#
|
||||
# ⚠ Never point this at a Secret the gateway controller generates for its own
|
||||
# internal use. Doing exactly that with Contour's `contourcert-*` xDS secret killed
|
||||
# its dataplane for 141 days — see the legacy Git history.
|
||||
- name: https
|
||||
protocol: HTTPS
|
||||
port: 443
|
||||
hostname: "*.ad.ddupan.top"
|
||||
tls:
|
||||
mode: Terminate
|
||||
certificateRefs:
|
||||
- kind: Secret
|
||||
name: wildcard-ad-ddupan-top-tls
|
||||
allowedRoutes:
|
||||
namespaces:
|
||||
from: All
|
||||
kinds:
|
||||
- group: gateway.networking.k8s.io
|
||||
kind: HTTPRoute
|
||||
|
||||
# HTTPS :443 for auth.ddupan.top specifically.
|
||||
#
|
||||
# WHY a second listener rather than another hostname on the one above: a
|
||||
# listener carries exactly one hostname, and `*.ad.ddupan.top` cannot match
|
||||
# `auth.ddupan.top` — different zone (Cloudflare is authoritative for
|
||||
# ddupan.top, the DC only for ad.ddupan.top) and one label shallower. Envoy
|
||||
# selects between them by SNI, so both coexist on :443 cleanly.
|
||||
#
|
||||
# This exists so in-cluster clients reach Authelia over the LAN instead of
|
||||
# hairpinning through Cloudflare proxy IPs that are unroutable from this
|
||||
# network. Full reasoning in ../cert-manager/certificate-auth-ddupan.yaml;
|
||||
# the route is ../../apps/authelia/httproute.yaml.
|
||||
- name: https-auth
|
||||
protocol: HTTPS
|
||||
port: 443
|
||||
hostname: "auth.ddupan.top"
|
||||
tls:
|
||||
mode: Terminate
|
||||
certificateRefs:
|
||||
- kind: Secret
|
||||
name: auth-ddupan-top-tls
|
||||
allowedRoutes:
|
||||
namespaces:
|
||||
from: All
|
||||
kinds:
|
||||
- group: gateway.networking.k8s.io
|
||||
kind: HTTPRoute
|
||||
|
||||
# HTTPS :443 for git.ddupan.top. Third listener for the same reason as the
|
||||
# second: one hostname per listener, and the *.ad.ddupan.top wildcard cannot
|
||||
# match a name in the ddupan.top zone. Envoy picks between all three by SNI.
|
||||
#
|
||||
# Exists so `git push` stays on the LAN instead of going out to Cloudflare and
|
||||
# back down the tunnel into this same cluster. The repo is what you need during
|
||||
# an incident, so it must not depend on the WAN. Cert:
|
||||
# ../cert-manager/certificate-git-ddupan.yaml; route: ../../apps/gitea/httproute.yaml.
|
||||
- name: https-git
|
||||
protocol: HTTPS
|
||||
port: 443
|
||||
hostname: "git.ddupan.top"
|
||||
tls:
|
||||
mode: Terminate
|
||||
certificateRefs:
|
||||
- kind: Secret
|
||||
name: git-ddupan-top-tls
|
||||
allowedRoutes:
|
||||
namespaces:
|
||||
from: All
|
||||
kinds:
|
||||
- group: gateway.networking.k8s.io
|
||||
kind: HTTPRoute
|
||||
@@ -0,0 +1,43 @@
|
||||
# The one store every namespace reads from.
|
||||
#
|
||||
# Cluster-scoped on purpose: authelia, gitea and cloudflared all consume it, and a
|
||||
# per-namespace SecretStore would mean duplicating the OpenBao connection details
|
||||
# three times.
|
||||
#
|
||||
# AUTH: no credential is stored anywhere. ESO presents its own ServiceAccount JWT,
|
||||
# OpenBao validates it against the cluster's TokenReview API, and hands back a
|
||||
# short-lived token scoped by the `external-secrets` role. The reviewer JWT that
|
||||
# makes that possible lives on the bao host, configured by
|
||||
# ../../infrastructure/openbao/ansible/roles/openbao_bootstrap/tasks/auth_kubernetes.yml — it is key
|
||||
# material, which is why Ansible owns it and Terraform does not.
|
||||
#
|
||||
# TLS: bao presents a Let's Encrypt cert for bao.ad.ddupan.top, so no caBundle or
|
||||
# caProvider is needed — verified from inside a pod (HTTP 200, ssl_verify_result 0).
|
||||
# ⚠ Address it by HOSTNAME, never 192.168.10.8: the cert carries a DNS SAN only,
|
||||
# so connecting by IP fails verification. Same trap as dc1's LDAPS cert.
|
||||
---
|
||||
apiVersion: external-secrets.io/v1
|
||||
kind: ClusterSecretStore
|
||||
metadata:
|
||||
name: openbao
|
||||
spec:
|
||||
provider:
|
||||
vault:
|
||||
# OpenBao is Vault-API compatible; ESO's vault provider drives it unchanged.
|
||||
server: 'https://bao.ad.ddupan.top:8200'
|
||||
# Mount path of the KV engine, from ../../infrastructure/openbao/terraform/mounts.tf.
|
||||
path: 'kv'
|
||||
version: 'v2'
|
||||
auth:
|
||||
kubernetes:
|
||||
mountPath: 'kubernetes'
|
||||
role: 'external-secrets'
|
||||
serviceAccountRef:
|
||||
name: 'external-secrets'
|
||||
# namespace is MANDATORY on a ClusterSecretStore (it has no namespace
|
||||
# of its own to resolve the reference against).
|
||||
namespace: 'external-secrets'
|
||||
# NOTE: Vault 1.21+ requires an `audiences: ['vault']` entry here, and
|
||||
# the bao role must declare a matching audience. OpenBao 2.6.1 does not,
|
||||
# so it is omitted — if auth ever starts failing with an audience
|
||||
# mismatch after an upgrade, this is the first thing to add.
|
||||
@@ -0,0 +1,119 @@
|
||||
# One ExternalSecret per Kubernetes Secret that was previously created by hand.
|
||||
#
|
||||
# These REPLACE the hand-applied secret.yaml files in ../../apps/authelia, ../../apps/gitea and
|
||||
# ../../infrastructure/cloudflared. Those files stay gitignored as the break-glass copy — if OpenBao
|
||||
# is down during a rebuild you can still `kubectl apply -f <svc>/secret.yaml` and
|
||||
# carry on. ESO will reconcile back over it once bao returns.
|
||||
#
|
||||
# ⚠ ESO takes OWNERSHIP of the target Secret. The existing hand-created Secrets have
|
||||
# the same names on purpose, so ESO adopts rather than duplicates them — but that
|
||||
# also means a wrong key name here will OVERWRITE a working Secret with an empty or
|
||||
# partial one. Check `kubectl get externalsecret -A` reports SecretSynced before
|
||||
# trusting it, and remember Authelia and Gitea only read their secrets at startup.
|
||||
#
|
||||
# refreshInterval is deliberately long. Nothing here rotates on its own yet, and a
|
||||
# short interval only adds load plus a wider window to notice a bad sync. When the
|
||||
# OpenBao PostgreSQL secrets engine lands (see ../docs/cicd.md) the rotating
|
||||
# credentials will want a shorter interval AND something to restart the consumer.
|
||||
---
|
||||
apiVersion: external-secrets.io/v1
|
||||
kind: ExternalSecret
|
||||
metadata:
|
||||
name: authelia-secrets
|
||||
namespace: authelia
|
||||
spec:
|
||||
refreshInterval: 1h
|
||||
secretStoreRef:
|
||||
name: openbao
|
||||
kind: ClusterSecretStore
|
||||
target:
|
||||
name: authelia-secrets
|
||||
creationPolicy: Owner
|
||||
# extract pulls EVERY key at the path, so the six key names live in OpenBao
|
||||
# rather than being restated here. They must match the `path:` values in
|
||||
# ../../apps/authelia/values.yaml exactly.
|
||||
dataFrom:
|
||||
- extract:
|
||||
key: k8s/authelia
|
||||
---
|
||||
# Separate Secret, not a seventh key in authelia-secrets — the chart projects the
|
||||
# existingSecret volume with an explicit items: list of only the six keys it
|
||||
# generates, so an extra key would be stored but never mounted. That mistake took
|
||||
# SSO down on 2026-07-28.
|
||||
apiVersion: external-secrets.io/v1
|
||||
kind: ExternalSecret
|
||||
metadata:
|
||||
name: authelia-oidc-jwks
|
||||
namespace: authelia
|
||||
spec:
|
||||
refreshInterval: 1h
|
||||
secretStoreRef:
|
||||
name: openbao
|
||||
kind: ClusterSecretStore
|
||||
target:
|
||||
name: authelia-oidc-jwks
|
||||
creationPolicy: Owner
|
||||
dataFrom:
|
||||
- extract:
|
||||
key: k8s/authelia-oidc-jwks
|
||||
---
|
||||
apiVersion: external-secrets.io/v1
|
||||
kind: ExternalSecret
|
||||
metadata:
|
||||
name: gitea-db
|
||||
namespace: gitea
|
||||
spec:
|
||||
refreshInterval: 1h
|
||||
secretStoreRef:
|
||||
name: openbao
|
||||
kind: ClusterSecretStore
|
||||
target:
|
||||
name: gitea-db
|
||||
creationPolicy: Owner
|
||||
dataFrom:
|
||||
- extract:
|
||||
key: k8s/gitea
|
||||
---
|
||||
apiVersion: external-secrets.io/v1
|
||||
kind: ExternalSecret
|
||||
metadata:
|
||||
name: cloudflared-tunnel
|
||||
namespace: cloudflared
|
||||
spec:
|
||||
refreshInterval: 1h
|
||||
secretStoreRef:
|
||||
name: openbao
|
||||
kind: ClusterSecretStore
|
||||
target:
|
||||
name: cloudflared-tunnel
|
||||
creationPolicy: Owner
|
||||
dataFrom:
|
||||
- extract:
|
||||
key: k8s/cloudflared
|
||||
---
|
||||
# SeaweedFS S3 identities, including the least-privilege `terraform` identity that
|
||||
# holds Terraform remote state.
|
||||
#
|
||||
# ⚠ These credentials were previously INLINE in ../../apps/seaweedfs/values.yaml and went
|
||||
# into git with the initial commit. Externalising stops the bleeding; the leaked
|
||||
# anvAdmin key still needs rotating separately, since it is in history.
|
||||
#
|
||||
# The chart normally GENERATES seaweedfs-s3-secret from s3.credentials. We point
|
||||
# filer.s3.existingConfigSecret at this one instead, so the chart stops rendering
|
||||
# credentials from values entirely.
|
||||
apiVersion: external-secrets.io/v1
|
||||
kind: ExternalSecret
|
||||
metadata:
|
||||
name: seaweedfs-s3-config
|
||||
namespace: seaweedfs
|
||||
spec:
|
||||
refreshInterval: 1h
|
||||
secretStoreRef:
|
||||
name: openbao
|
||||
kind: ClusterSecretStore
|
||||
target:
|
||||
name: seaweedfs-s3-config
|
||||
creationPolicy: Owner
|
||||
dataFrom:
|
||||
- extract:
|
||||
key: k8s/seaweedfs-s3
|
||||
@@ -0,0 +1,43 @@
|
||||
# External Secrets Operator — pulls secret material from OpenBao into Kubernetes
|
||||
# Secrets, so the Secrets themselves become declarative instead of hand-created.
|
||||
#
|
||||
# Install:
|
||||
# helm upgrade --install external-secrets external-secrets/external-secrets \
|
||||
# -n external-secrets --create-namespace -f values.yaml
|
||||
#
|
||||
# WHY this and not SOPS: OpenBao is already the secrets store and the internal CA
|
||||
# here, and its Kubernetes auth backend is already bootstrapped
|
||||
# (../../infrastructure/openbao/ansible/roles/openbao_bootstrap/tasks/auth_kubernetes.yml), so a pod
|
||||
# authenticates with its own ServiceAccount JWT and NOTHING long-lived is stored
|
||||
# in the cluster. SOPS would mean managing an age key and committing ciphertext.
|
||||
#
|
||||
# Single node, so one replica of each component. The webhook and cert-controller
|
||||
# are not optional — the CRDs use conversion/validating webhooks.
|
||||
replicaCount: 1
|
||||
|
||||
webhook:
|
||||
replicaCount: 1
|
||||
resources:
|
||||
requests: {cpu: 10m, memory: 32Mi}
|
||||
limits: {memory: 128Mi}
|
||||
|
||||
certController:
|
||||
replicaCount: 1
|
||||
resources:
|
||||
requests: {cpu: 10m, memory: 32Mi}
|
||||
limits: {memory: 128Mi}
|
||||
|
||||
resources:
|
||||
requests: {cpu: 10m, memory: 64Mi}
|
||||
limits: {memory: 256Mi}
|
||||
|
||||
# The controller's own ServiceAccount is what the ClusterSecretStore presents to
|
||||
# OpenBao, so its name is part of the contract with the bao Kubernetes auth role
|
||||
# (bound_service_account_names). Pinned rather than left to the chart's default.
|
||||
serviceAccount:
|
||||
create: true
|
||||
name: external-secrets
|
||||
|
||||
# ClusterSecretStore is cluster-scoped; leaving this on lets one store serve every
|
||||
# namespace, which is the point here (authelia, gitea and cloudflared all consume it).
|
||||
installCRDs: true
|
||||
@@ -0,0 +1,24 @@
|
||||
.:53 {
|
||||
errors
|
||||
health
|
||||
ready
|
||||
kubernetes cluster.local in-addr.arpa ip6.arpa {
|
||||
pods insecure
|
||||
fallthrough in-addr.arpa ip6.arpa
|
||||
}
|
||||
hosts /etc/coredns/NodeHosts {
|
||||
ttl 60
|
||||
reload 15s
|
||||
fallthrough
|
||||
}
|
||||
prometheus :9153
|
||||
cache 30 {
|
||||
serve_stale 1h immediate
|
||||
}
|
||||
loop
|
||||
reload
|
||||
loadbalance
|
||||
import /etc/coredns/custom/*.override
|
||||
forward . /etc/resolv.conf
|
||||
}
|
||||
import /etc/coredns/custom/*.server
|
||||
@@ -0,0 +1,128 @@
|
||||
# Route internal AD-zone lookups straight to the Samba DC instead of the LAN
|
||||
# router, for every pod in the cluster.
|
||||
#
|
||||
# WHY: k3s CoreDNS forwards to the node's /etc/resolv.conf, which lists the
|
||||
# router (192.168.10.1) — and that resolver flaps. When it hangs, CoreDNS's
|
||||
# forward plugin blocks and even CLUSTER-INTERNAL lookups
|
||||
# (*.svc.cluster.local) start timing out, which is how Authelia ended up in
|
||||
# CrashLoopBackOff: its startup check resolves both dc1.ad.ddupan.top and
|
||||
# smtp-relay.smtp-relay.svc.cluster.local, and a strict startup check turns a
|
||||
# transient DNS blip into a restart loop.
|
||||
#
|
||||
# dc1 is AUTHORITATIVE for ad.ddupan.top, so sending that zone directly to it
|
||||
# removes the router from the path entirely — no forwarding, no upstream
|
||||
# dependency, no flap.
|
||||
#
|
||||
# k3s picks this up via `import /etc/coredns/custom/*.server` in its Corefile.
|
||||
# The `reload` plugin applies it without restarting CoreDNS.
|
||||
apiVersion: v1
|
||||
kind: ConfigMap
|
||||
metadata:
|
||||
name: coredns-custom
|
||||
namespace: kube-system
|
||||
data:
|
||||
ad-ddupan-top.server: |
|
||||
ad.ddupan.top:53 {
|
||||
errors
|
||||
cache 30
|
||||
forward . 192.168.10.5 {
|
||||
policy sequential
|
||||
}
|
||||
}
|
||||
# ── stop search-domain permutations from hanging on the router ──────────
|
||||
# Pods get `options ndots:5` plus the NODE's search list, which still carries
|
||||
# `lab.ddupan.top` (the RETIRED domain) and tailscale's MagicDNS suffix. A name
|
||||
# like dc1.ad.ddupan.top has only 3 dots, so glibc tries EVERY suffix first:
|
||||
# dc1.ad.ddupan.top.tail7e769.ts.net
|
||||
# dc1.ad.ddupan.top.lab.ddupan.top
|
||||
# Both are forwarded upstream to the flaky router and time out, so the client
|
||||
# gives up before the bare name is ever tried. That is what put Authelia in
|
||||
# CrashLoopBackOff -- and it hit the cluster-internal smtp-relay name too
|
||||
# (4 dots < ndots:5).
|
||||
#
|
||||
# Answer these locally with an instant NXDOMAIN. Neither zone should ever
|
||||
# resolve from inside the cluster.
|
||||
dead-search-domains.server: |
|
||||
lab.ddupan.top:53 {
|
||||
errors
|
||||
template ANY ANY {
|
||||
rcode NXDOMAIN
|
||||
}
|
||||
}
|
||||
tail7e769.ts.net:53 {
|
||||
errors
|
||||
template ANY ANY {
|
||||
rcode NXDOMAIN
|
||||
}
|
||||
}
|
||||
# ── split-horizon for the Authelia hostname ─────────────────────────────
|
||||
# Cloudflare is authoritative for ddupan.top (the DC only holds ad.ddupan.top),
|
||||
# so auth.ddupan.top publicly resolves to Cloudflare proxy IPs — 104.21.6.55 and
|
||||
# 172.67.154.245. TCP/443 to BOTH fails from this network, persistently, while
|
||||
# other Cloudflare IPs (104.16.132.229) connect fine. So every in-cluster
|
||||
# consumer of Authelia was hairpinning out to an internet path that does not
|
||||
# work, to reach a Service sitting in the same cluster.
|
||||
#
|
||||
# Concretely (2026-07-28): the Gitea chart runs `gitea admin auth update-oauth`
|
||||
# in an INIT container, which FETCHES the OIDC discovery URL on every pod start.
|
||||
# It timed out, so Gitea CrashLoopBackOff'd on any restart, and server-side
|
||||
# token exchange failed for logins. This was latent — any restart would do it.
|
||||
#
|
||||
# Answer with the Envoy Gateway LAN address instead. The gateway terminates TLS
|
||||
# with a real Let's Encrypt cert for this exact name
|
||||
# (../cert-manager/certificate-auth-ddupan.yaml) and routes to the authelia
|
||||
# Service (../../apps/authelia/httproute.yaml), so the name, issuer, redirect URIs and
|
||||
# cookie domain are all unchanged — only the path stops leaving the LAN.
|
||||
#
|
||||
# A `template` block (not `hosts`) so it answers only A/AAAA-shaped queries and
|
||||
# returns NOERROR/no-data rather than NXDOMAIN for anything else.
|
||||
auth-ddupan-top.server: |
|
||||
auth.ddupan.top:53 {
|
||||
errors
|
||||
template IN A {
|
||||
answer "{{ .Name }} 60 IN A 192.168.10.127"
|
||||
}
|
||||
template IN AAAA {
|
||||
rcode NOERROR
|
||||
}
|
||||
}
|
||||
# ── split-horizon for Gitea ─────────────────────────────────────────────
|
||||
# Same shape and same reason as auth.ddupan.top above: git.ddupan.top publicly
|
||||
# resolves to Cloudflare, so an in-cluster client cloning from Gitea would leave
|
||||
# the LAN, cross the WAN and come back down the tunnel into this same cluster.
|
||||
#
|
||||
# This matters for CI: an Actions runner checking out the repo, and anything
|
||||
# else that clones from inside the cluster. Served by the `https-git` listener
|
||||
# with a real LE cert for the name (../cert-manager/certificate-git-ddupan.yaml),
|
||||
# so the clone URL is identical inside and outside — no remote needs rewriting.
|
||||
#
|
||||
# ⚠ This only covers PODS. LAN clients resolve via the DC, which forwards
|
||||
# ddupan.top to the router — and the NEC IX has no static-host/proxy-DNS
|
||||
# feature (`show dns` offers only fqdn-database). The DC therefore needs its own
|
||||
# `git.ddupan.top` zone for laptops and PVE nodes to get the LAN answer.
|
||||
git-ddupan-top.server: |
|
||||
git.ddupan.top:53 {
|
||||
errors
|
||||
template IN A {
|
||||
answer "{{ .Name }} 60 IN A 192.168.10.127"
|
||||
}
|
||||
template IN AAAA {
|
||||
rcode NOERROR
|
||||
}
|
||||
}
|
||||
---
|
||||
# NOTE: `serve_stale` could NOT be added here.
|
||||
#
|
||||
# The cache plugin already exists in k3s's own Corefile (`cache 30`), and CoreDNS
|
||||
# rejects a duplicate plugin in the same server block — so the *.override import
|
||||
# cannot carry it. It was applied by patching the `coredns` ConfigMap directly;
|
||||
# the intended Corefile is kept alongside as `Corefile.desired`.
|
||||
#
|
||||
# ⚠️ A k3s restart/upgrade re-applies its bundled manifest and will REVERT that
|
||||
# patch. Re-apply from Corefile.desired if external name resolution starts
|
||||
# failing hard during WAN outages again.
|
||||
#
|
||||
# WHY: the ISP drops the WAN at random and it cannot be changed. serve_stale
|
||||
# keeps answering with expired entries while the upstream is unreachable, so
|
||||
# previously-resolved external names (smtp.office365.com, package mirrors) keep
|
||||
# working through a blip instead of hanging.
|
||||
@@ -0,0 +1,138 @@
|
||||
# observability
|
||||
|
||||
Metrics, logs, and traces for the cluster — one VictoriaMetrics-ecosystem stack,
|
||||
operator-driven, running in the `monitoring` namespace. Replaces the standalone
|
||||
Docker Compose stack in `../../apps/victoriametrics/`.
|
||||
|
||||
## Status — DEPLOYED (2026-07-10)
|
||||
|
||||
Live and verified in the `monitoring` namespace (operator chart 0.66.2):
|
||||
metrics (data queryable), logs (pods ingesting), traces (VTSingle CRD, verified via
|
||||
Jaeger API), Grafana (3 datasources healthy, VM dashboard loaded, Tailscale ingress
|
||||
`grafana.tail7e769.ts.net`, Authelia OIDC client registered). All PVCs on
|
||||
`localpv-zfs-ceph`. Note the vmagent hot-reload RBAC fix (`metrics/reload-rbac.yaml`,
|
||||
see `metrics/README-reload.md`) and that VL/VT CRDs are served under `.../v1`.
|
||||
|
||||
Still host-side (not yet done): deploy `docker-hosts/compose.yaml` on the Docker
|
||||
hosts and `logs/vlogs-ingress.yaml` to push their logs.
|
||||
|
||||
## Decision
|
||||
|
||||
| Pillar | Choice | Why |
|
||||
| -------- | ----------------------------- | --- |
|
||||
| Metrics | **VictoriaMetrics** (VMSingle)| already in use; keep it |
|
||||
| Logs | **VictoriaLogs** (VLSingle) | same vendor/operator, tiny footprint, LogsQL |
|
||||
| Traces | **VictoriaTraces** + OTel Collector | no extra storage deps, OTLP in, Jaeger query API |
|
||||
| Collect | vmagent (metrics), vlagent (logs), OTel Collector (traces) | native to each backend |
|
||||
| Visualize| **Grafana**, one instance, 3 datasources | single pane |
|
||||
| Manage | **victoria-metrics-operator** | VMSingle/VMAgent/VMAlert/VMAlertmanager/VMRule **and** VLSingle as CRDs |
|
||||
| Expose | **Tailscale ingress** (private) + **Authelia OIDC** | admin tool: private + SSO |
|
||||
|
||||
Grafana's Prometheus-operator converter is on, so any chart shipping a
|
||||
`ServiceMonitor`/`PodMonitor`/`PrometheusRule` is scraped automatically.
|
||||
|
||||
## Architecture
|
||||
|
||||
```
|
||||
Grafana (grafana.<tailnet>.ts.net, Authelia OIDC)
|
||||
├── VictoriaMetrics datasource → VMSingle ← vmagent (k8s SD, node-exporter, kubelet/cAdvisor, KSM)
|
||||
├── VictoriaLogs datasource → VLSingle ← vlagent DaemonSet (pod logs)
|
||||
└── Jaeger datasource → VictoriaTraces ← OTel Collector (OTLP 4317/4318) ← apps
|
||||
VMAlert (VMRules) → VMAlertmanager
|
||||
```
|
||||
|
||||
## Layout
|
||||
|
||||
```
|
||||
observability/
|
||||
namespace.yaml monitoring namespace
|
||||
operator/ victoria-metrics-operator (Helm)
|
||||
metrics/
|
||||
vmsingle|vmagent|vmalert|vmalertmanager.yaml core CRs
|
||||
rules/ VMRules ported from ../../apps/victoriametrics/rules
|
||||
exporters/ node-exporter (+VMNodeScrape), kube-state-metrics (Helm)
|
||||
scrapes/ kubelet + cAdvisor VMNodeScrapes
|
||||
logs/ VLSingle CR + victoria-logs-collector (Helm)
|
||||
traces/ victoria-traces-single (Helm) + OTel Collector
|
||||
grafana/ Grafana (Helm) values, OIDC secret, install script
|
||||
docker-hosts/ per-Docker-host sidecar (cAdvisor + node-exporter + Vector)
|
||||
```
|
||||
|
||||
## Docker hosts
|
||||
|
||||
The Compose services on Docker hosts (`../../apps/vlmcsd`, `../../apps/ps3netsrv`, `../../apps/netboot`, ...
|
||||
and the legacy `../../apps/victoriametrics` stack) are monitored too:
|
||||
|
||||
- **Metrics — pull, nothing exposed.** Each host runs `docker-hosts/compose.yaml`
|
||||
(cAdvisor `:8080` + node-exporter `:9100`). The cluster's vmagent scrapes their LAN
|
||||
IPs via `metrics/scrapes/docker-hosts.yaml` (VMStaticScrape). VMSingle stays private.
|
||||
- **Logs — push over the tailnet.** A **Vector** container per host tails the Docker
|
||||
socket and ships to VictoriaLogs' Elasticsearch-bulk endpoint, exposed privately as
|
||||
`vlogs.<tailnet>.ts.net` by `logs/vlogs-ingress.yaml`.
|
||||
|
||||
Deploy:
|
||||
|
||||
```bash
|
||||
# cluster side (once): expose VictoriaLogs ingest + register the scrape
|
||||
kubectl apply -f logs/vlogs-ingress.yaml -f metrics/scrapes/docker-hosts.yaml
|
||||
|
||||
# on each Docker host:
|
||||
cd docker-hosts
|
||||
cp .env.example .env # set HOST_LABEL and confirm VLOGS_ENDPOINT (tailnet FQDN)
|
||||
docker compose up -d
|
||||
# then add the host's IP to metrics/scrapes/docker-hosts.yaml and re-apply
|
||||
```
|
||||
|
||||
Note: cAdvisor uses host port `8080` — remap it (and the scrape target) if taken.
|
||||
|
||||
## Deploy order
|
||||
|
||||
```bash
|
||||
# 1. Operator (installs CRDs) + namespace
|
||||
cd operator && ./helm.sh && cd ..
|
||||
|
||||
# 2. Metrics: core CRs, rules, scrapes, exporters
|
||||
kubectl apply -f metrics/vmsingle.yaml -f metrics/vmagent.yaml \
|
||||
-f metrics/vmalert.yaml -f metrics/vmalertmanager.yaml
|
||||
kubectl apply -f metrics/rules/ -f metrics/scrapes/
|
||||
kubectl apply -f metrics/exporters/node-exporter.yaml
|
||||
metrics/exporters/kube-state-metrics.sh # needs Prometheus CRDs; see the script
|
||||
|
||||
# 3. Logs
|
||||
cd logs && ./helm.sh && cd ..
|
||||
|
||||
# 4. Traces
|
||||
cd traces && ./helm.sh && cd ..
|
||||
|
||||
# 5. Grafana (create the OIDC secret first — see grafana/oidc-secret.yaml)
|
||||
cd grafana && ./helm.sh && cd ..
|
||||
```
|
||||
|
||||
## Before it works — required edits
|
||||
|
||||
- **OIDC secret pair.** Generate once:
|
||||
`authelia crypto hash generate pbkdf2 --variant sha512 --random --random.length 72`
|
||||
- plaintext → the `grafana-oidc` Secret (create imperatively, don't commit)
|
||||
- hash → `authelia/values.yaml` grafana client `client_secret`, then `helm upgrade` Authelia.
|
||||
- **Tailnet FQDN.** This scaffold assumes `grafana.tail7e769.ts.net` (your existing tailnet,
|
||||
per `seaweedfs-admin`). If the operator assigns a different name, update it in
|
||||
`grafana/values.yaml` (`root_url`) **and** the Authelia grafana `redirect_uris`.
|
||||
- **Chart value names.** `logs/collector-values.yaml` and `traces/traces-values.yaml` note a
|
||||
`helm show values` check — verify the keys against the installed chart versions.
|
||||
- **Pin chart + image versions** after first install (all `helm.sh` scripts note this;
|
||||
`otel-collector.yaml` pins the collector image — bump to current).
|
||||
|
||||
## Follow-ups
|
||||
|
||||
- Dashboards: the VictoriaMetrics single-node board is migrated to
|
||||
`grafana/dashboards/victoriametrics.json`; `grafana/dashboards/apply.sh` loads it via
|
||||
the Grafana sidecar. (The old `vmagent.json`/`vmalert.json` were empty dirs Docker
|
||||
auto-created, not real boards.) Add the canonical ones by dropping their JSON in
|
||||
`grafana/dashboards/` and re-running the script — vmagent (gnetId 12683),
|
||||
vmalert (14950), node-exporter (1860), cAdvisor now that those exporters exist.
|
||||
- Wire real Alertmanager receivers in `metrics/vmalertmanager.yaml` (email via
|
||||
`../../apps/smtp-relay/`), replacing the ported `blackhole`.
|
||||
- Once parity is confirmed, decommission the Compose stack: `../../apps/victoriametrics/`
|
||||
(`docker compose down`), and retire that folder.
|
||||
- Add app instrumentation: point `OTEL_EXPORTER_OTLP_ENDPOINT` at
|
||||
`otel-collector.monitoring.svc:4317`.
|
||||
@@ -0,0 +1,9 @@
|
||||
# Copy to .env and adjust per host. Do not commit the real .env.
|
||||
|
||||
# A stable name for THIS Docker host; becomes the `host` log stream label and the
|
||||
# `host` label on its scraped metrics (keep it consistent with the VMStaticScrape).
|
||||
HOST_LABEL=docker-01
|
||||
|
||||
# VictoriaLogs Elasticsearch-bulk ingest endpoint, reached over the tailnet
|
||||
# (exposed by ../logs/vlogs-ingress.yaml). Confirm the tailnet FQDN after applying.
|
||||
VLOGS_ENDPOINT=https://vlogs.tail7e769.ts.net/insert/elasticsearch/
|
||||
@@ -0,0 +1,50 @@
|
||||
# Per-Docker-host monitoring sidecar. Deploy this compose on every Docker host you
|
||||
# want visible in the observability stack. It exposes host + container metrics for the
|
||||
# cluster's vmagent to PULL over the LAN, and pushes container logs to VictoriaLogs.
|
||||
#
|
||||
# cp .env.example .env # set HOST_LABEL + VLOGS_ENDPOINT
|
||||
# docker compose up -d
|
||||
#
|
||||
# Then add this host's IP to ../metrics/scrapes/docker-hosts.yaml (VMStaticScrape).
|
||||
services:
|
||||
# Host metrics (CPU/mem/disk/net/fs). host networking -> listens on <hostIP>:9100.
|
||||
node-exporter:
|
||||
image: quay.io/prometheus/node-exporter:v1.9.1
|
||||
command:
|
||||
- --path.rootfs=/host
|
||||
network_mode: host
|
||||
pid: host
|
||||
volumes:
|
||||
- /:/host:ro,rslave
|
||||
restart: unless-stopped
|
||||
|
||||
# Container metrics. Published on <hostIP>:8080. If 8080 is taken on this host,
|
||||
# remap the left side (e.g. "8098:8080") and update the VMStaticScrape target.
|
||||
cadvisor:
|
||||
image: gcr.io/cadvisor/cadvisor:v0.49.1
|
||||
command:
|
||||
- --housekeeping_interval=30s
|
||||
- --docker_only=true
|
||||
ports:
|
||||
- "8080:8080"
|
||||
volumes:
|
||||
- /:/rootfs:ro
|
||||
- /var/run:/var/run:ro
|
||||
- /sys:/sys:ro
|
||||
- /var/lib/docker/:/var/lib/docker:ro
|
||||
- /dev/disk/:/dev/disk:ro
|
||||
devices:
|
||||
- /dev/kmsg
|
||||
privileged: true
|
||||
restart: unless-stopped
|
||||
|
||||
# Ships every container's logs to VictoriaLogs (pushed over the tailnet).
|
||||
vector:
|
||||
image: timberio/vector:0.44.0-debian
|
||||
volumes:
|
||||
- /var/run/docker.sock:/var/run/docker.sock:ro
|
||||
- ./vector.yaml:/etc/vector/vector.yaml:ro
|
||||
environment:
|
||||
VLOGS_ENDPOINT: ${VLOGS_ENDPOINT}
|
||||
HOST_LABEL: ${HOST_LABEL}
|
||||
restart: unless-stopped
|
||||
@@ -0,0 +1,30 @@
|
||||
# Vector: tail all Docker container logs on this host and ship them to VictoriaLogs
|
||||
# via the Elasticsearch bulk API (the VM-recommended Vector sink). Endpoint + host
|
||||
# label come from the environment (see .env / compose.yaml).
|
||||
sources:
|
||||
docker:
|
||||
type: docker_logs
|
||||
|
||||
transforms:
|
||||
enrich:
|
||||
type: remap
|
||||
inputs: [docker]
|
||||
source: |
|
||||
.host = get_env_var!("HOST_LABEL")
|
||||
|
||||
sinks:
|
||||
vlogs:
|
||||
type: elasticsearch
|
||||
inputs: [enrich]
|
||||
endpoints:
|
||||
- "${VLOGS_ENDPOINT}" # e.g. https://vlogs.tail7e769.ts.net/insert/elasticsearch/
|
||||
mode: bulk
|
||||
api_version: v8
|
||||
compression: gzip
|
||||
healthcheck:
|
||||
enabled: false
|
||||
query:
|
||||
_msg_field: message
|
||||
_time_field: timestamp
|
||||
# Group log streams by host + container so LogsQL stream filters work well.
|
||||
_stream_fields: host,container_name
|
||||
+23
@@ -0,0 +1,23 @@
|
||||
#!/usr/bin/env bash
|
||||
# Load every *.json in this folder into Grafana as a sidecar dashboard ConfigMap.
|
||||
# The Grafana chart's dashboard sidecar (sidecar.dashboards.enabled) watches for
|
||||
# ConfigMaps labeled grafana_dashboard=1 in the monitoring namespace and imports them.
|
||||
# Idempotent — re-run after adding/updating a dashboard JSON.
|
||||
#
|
||||
# The VictoriaMetrics board uses a `$ds` datasource variable that resolves to the
|
||||
# default datasource (VictoriaMetrics), so no per-panel rewiring is needed.
|
||||
set -euo pipefail
|
||||
cd "$(dirname "$0")"
|
||||
|
||||
NS=monitoring
|
||||
for f in *.json; do
|
||||
[ -e "$f" ] || continue
|
||||
name="grafana-dashboard-$(basename "$f" .json)"
|
||||
echo "applying $name from $f"
|
||||
kubectl create configmap "$name" \
|
||||
--namespace "$NS" \
|
||||
--from-file="$f" \
|
||||
--dry-run=client -o yaml \
|
||||
| kubectl label --local -f - grafana_dashboard=1 --dry-run=client -o yaml \
|
||||
| kubectl apply -f -
|
||||
done
|
||||
File diff suppressed because it is too large
Load Diff
Executable
+22
@@ -0,0 +1,22 @@
|
||||
#!/usr/bin/env bash
|
||||
# Deploy Grafana. Create the OIDC client secret first (see oidc-secret.yaml header),
|
||||
# then install the chart. Requires the metrics/logs/traces backends to exist so the
|
||||
# provisioned datasources resolve.
|
||||
set -euo pipefail
|
||||
|
||||
# 1) OIDC client secret (edit oidc-secret.yaml, or create it imperatively — preferred).
|
||||
kubectl apply -f oidc-secret.yaml
|
||||
|
||||
helm repo add grafana https://grafana.github.io/helm-charts
|
||||
helm repo update grafana
|
||||
|
||||
# Pin --version after the first install (helm search repo grafana/grafana --versions).
|
||||
helm upgrade --install grafana grafana/grafana \
|
||||
--namespace monitoring \
|
||||
--values values.yaml \
|
||||
--wait
|
||||
|
||||
# The tailnet FQDN, once the tailscale operator assigns it:
|
||||
# kubectl -n monitoring get ingress grafana -o jsonpath='{.status.loadBalancer.ingress[0].hostname}'
|
||||
# Break-glass admin password:
|
||||
# kubectl -n monitoring get secret grafana -o jsonpath='{.data.admin-password}' | base64 -d
|
||||
@@ -0,0 +1,18 @@
|
||||
# Plaintext OIDC client secret Grafana presents to Authelia. Authelia stores only the
|
||||
# pbkdf2-sha512 HASH of this same value (see authelia/values.yaml grafana client).
|
||||
#
|
||||
# Generate a matching pair:
|
||||
# authelia crypto hash generate pbkdf2 --variant sha512 --random --random.length 72
|
||||
# Put the "Random Password" (plaintext) below; put the "Digest" (hash) in Authelia.
|
||||
#
|
||||
# Do NOT commit the real secret. Create it out-of-band instead, e.g.:
|
||||
# kubectl -n monitoring create secret generic grafana-oidc \
|
||||
# --from-literal=client_secret='<plaintext>'
|
||||
apiVersion: v1
|
||||
kind: Secret
|
||||
metadata:
|
||||
name: grafana-oidc
|
||||
namespace: monitoring
|
||||
type: Opaque
|
||||
stringData:
|
||||
client_secret: "REPLACE_ME_WITH_PLAINTEXT_OIDC_SECRET"
|
||||
@@ -0,0 +1,98 @@
|
||||
# Grafana — single pane over metrics (VictoriaMetrics), logs (VictoriaLogs) and
|
||||
# traces (VictoriaTraces via Jaeger API). Exposed privately on the tailnet
|
||||
# (grafana.tail7e769.ts.net) and authenticated via Authelia OIDC. Local admin is
|
||||
# break-glass only.
|
||||
#
|
||||
# Chart: grafana/grafana (repo: https://grafana.github.io/helm-charts)
|
||||
|
||||
# --- VictoriaLogs needs its Grafana datasource plugin ---
|
||||
plugins:
|
||||
- victoriametrics-logs-datasource
|
||||
|
||||
# --- Dashboard sidecar: auto-loads any ConfigMap labeled grafana_dashboard=1 in the
|
||||
# namespace. Migrated boards live in ./dashboards and are applied by ./dashboards/apply.sh ---
|
||||
sidecar:
|
||||
dashboards:
|
||||
enabled: true
|
||||
label: grafana_dashboard
|
||||
labelValue: "1"
|
||||
folderAnnotation: grafana_folder
|
||||
provider:
|
||||
foldersFromFilesStructure: true
|
||||
|
||||
# --- Provisioned datasources ---
|
||||
datasources:
|
||||
datasources.yaml:
|
||||
apiVersion: 1
|
||||
datasources:
|
||||
- name: VictoriaMetrics
|
||||
type: prometheus
|
||||
access: proxy
|
||||
url: http://vmsingle-main.monitoring.svc:8428
|
||||
isDefault: true
|
||||
jsonData:
|
||||
prometheusType: Prometheus
|
||||
- name: VictoriaLogs
|
||||
type: victoriametrics-logs-datasource
|
||||
access: proxy
|
||||
url: http://vlsingle-main.monitoring.svc:9428
|
||||
- name: VictoriaTraces
|
||||
type: jaeger
|
||||
access: proxy
|
||||
# VictoriaTraces (VTSingle CR) exposes a Jaeger-compatible query API under /select/jaeger.
|
||||
url: http://vtsingle-main.monitoring.svc:10428/select/jaeger
|
||||
|
||||
# --- Persistence on OpenEBS ZFS ---
|
||||
persistence:
|
||||
enabled: true
|
||||
storageClassName: localpv-zfs-ceph
|
||||
size: 5Gi
|
||||
|
||||
# --- Private exposure via the Tailscale ingress (like seaweedfs-admin) ---
|
||||
# The tailscale operator provisions grafana.<tailnet>.ts.net and a TLS cert.
|
||||
ingress:
|
||||
enabled: true
|
||||
ingressClassName: tailscale
|
||||
hosts:
|
||||
- grafana
|
||||
tls:
|
||||
- hosts:
|
||||
- grafana
|
||||
|
||||
# --- OIDC via Authelia (AD groups -> Grafana roles) ---
|
||||
# client_secret is injected from the grafana-oidc Secret (see oidc-secret.yaml),
|
||||
# which overrides any value in grafana.ini.
|
||||
envValueFrom:
|
||||
GF_AUTH_GENERIC_OAUTH_CLIENT_SECRET:
|
||||
secretKeyRef:
|
||||
name: grafana-oidc
|
||||
key: client_secret
|
||||
|
||||
grafana.ini:
|
||||
server:
|
||||
root_url: "https://grafana.tail7e769.ts.net" # must match the tailnet FQDN + Authelia redirect_uri
|
||||
auth:
|
||||
# Keep the local admin login available as break-glass; don't force OIDC-only.
|
||||
disable_login_form: false
|
||||
oauth_auto_login: false
|
||||
auth.generic_oauth:
|
||||
enabled: true
|
||||
name: Authelia
|
||||
client_id: grafana
|
||||
scopes: "openid profile email groups"
|
||||
auth_url: "https://auth.ddupan.top/api/oidc/authorization"
|
||||
token_url: "https://auth.ddupan.top/api/oidc/token"
|
||||
api_url: "https://auth.ddupan.top/api/oidc/userinfo"
|
||||
login_attribute_path: preferred_username
|
||||
name_attribute_path: name
|
||||
email_attribute_path: email
|
||||
groups_attribute_path: groups
|
||||
# AD "Enterprise Admins" -> full Grafana server admin; "Domain Admins" -> org Admin;
|
||||
# everyone else who can authenticate -> Viewer. Tune group names to taste.
|
||||
role_attribute_path: "contains(groups[*], 'Enterprise Admins') && 'GrafanaAdmin' || contains(groups[*], 'Domain Admins') && 'Admin' || 'Viewer'"
|
||||
allow_assign_grafana_admin: true
|
||||
role_attribute_strict: false
|
||||
use_pkce: true
|
||||
|
||||
# Dashboards are loaded by the sidecar (above) from ConfigMaps created by
|
||||
# ./dashboards/apply.sh. Drop more JSON into ./dashboards and re-run that script.
|
||||
@@ -0,0 +1,22 @@
|
||||
# victoria-logs-collector — deploys vlagent as a DaemonSet that discovers and tails
|
||||
# every pod's container logs on each node and ships them to VictoriaLogs. Buffers to
|
||||
# a local volume when VictoriaLogs is unavailable.
|
||||
#
|
||||
# NOTE: confirm key names against the installed chart before applying:
|
||||
# helm show values vm/victoria-logs-collector
|
||||
# (chart value layout occasionally shifts between versions.)
|
||||
|
||||
remoteWrite:
|
||||
- url: "http://vlsingle-main.monitoring.svc:9428"
|
||||
|
||||
# Tolerate control-plane taints so logs are collected from every node.
|
||||
tolerations:
|
||||
- operator: Exists
|
||||
|
||||
resources:
|
||||
requests:
|
||||
cpu: 50m
|
||||
memory: 128Mi
|
||||
limits:
|
||||
cpu: 500m
|
||||
memory: 512Mi
|
||||
Executable
+15
@@ -0,0 +1,15 @@
|
||||
#!/usr/bin/env bash
|
||||
# Deploy the pod-log collector (vlagent DaemonSet) that ships container logs to the
|
||||
# VLSingle instance created by ./vlsingle.yaml. Apply vlsingle.yaml first.
|
||||
set -euo pipefail
|
||||
|
||||
kubectl apply -f vlsingle.yaml
|
||||
|
||||
helm repo add vm https://victoriametrics.github.io/helm-charts/
|
||||
helm repo update vm
|
||||
|
||||
# Pin --version after the first install (helm search repo vm/victoria-logs-collector --versions).
|
||||
helm upgrade --install victoria-logs-collector vm/victoria-logs-collector \
|
||||
--namespace monitoring \
|
||||
--values collector-values.yaml \
|
||||
--wait
|
||||
@@ -0,0 +1,24 @@
|
||||
# Private tailnet exposure of the VictoriaLogs INGEST endpoint so off-cluster log
|
||||
# shippers (Vector on Docker hosts) can push to it. Provisions vlogs.<tailnet>.ts.net.
|
||||
# Ingestion is unauthenticated — acceptable on the private tailnet; do not expose publicly.
|
||||
apiVersion: networking.k8s.io/v1
|
||||
kind: Ingress
|
||||
metadata:
|
||||
name: victorialogs
|
||||
namespace: monitoring
|
||||
spec:
|
||||
ingressClassName: tailscale
|
||||
rules:
|
||||
- host: vlogs
|
||||
http:
|
||||
paths:
|
||||
- path: /
|
||||
pathType: Prefix
|
||||
backend:
|
||||
service:
|
||||
name: vlsingle-main
|
||||
port:
|
||||
number: 9428
|
||||
tls:
|
||||
- hosts:
|
||||
- vlogs
|
||||
@@ -0,0 +1,25 @@
|
||||
# Single-node VictoriaLogs, managed by the same VM operator as the metrics stack.
|
||||
# Service exposed by the operator: vlsingle-main.monitoring.svc:9428
|
||||
# Query via Grafana (VictoriaLogs datasource) with LogsQL.
|
||||
# NOTE: the VictoriaLogs/Traces CRDs are served under .../v1 (metrics CRDs use v1beta1).
|
||||
apiVersion: operator.victoriametrics.com/v1
|
||||
kind: VLSingle
|
||||
metadata:
|
||||
name: main
|
||||
namespace: monitoring
|
||||
spec:
|
||||
retentionPeriod: "30d"
|
||||
storage:
|
||||
accessModes:
|
||||
- ReadWriteOnce
|
||||
storageClassName: localpv-zfs-ceph
|
||||
resources:
|
||||
requests:
|
||||
storage: 30Gi
|
||||
resources:
|
||||
requests:
|
||||
cpu: 100m
|
||||
memory: 256Mi
|
||||
limits:
|
||||
cpu: "1"
|
||||
memory: 2Gi
|
||||
@@ -0,0 +1,41 @@
|
||||
# vmagent scrape-config hot-reload on k3s
|
||||
|
||||
## Symptom
|
||||
New/changed `VMServiceScrape`/`VMNodeScrape`/`VMStaticScrape`/... are written by the
|
||||
operator into the `vmagent-main` config Secret, but the running vmagent keeps serving
|
||||
its **startup** config — the new targets never appear until vmagent is restarted.
|
||||
|
||||
## Cause
|
||||
vmagent's config-reloader (`victoriametrics/operator:config-reloader-*`) watches the
|
||||
config Secret with a client-go informer, which does `list` **and** `watch`. The
|
||||
operator-generated Role `monitoring:monitoring:vmagent-main` grants `secrets: [get, watch]`
|
||||
but **not `list`**, so on k3s the informer errors with:
|
||||
|
||||
```
|
||||
cannot list resource "secrets" in API group "" in the namespace "monitoring"
|
||||
```
|
||||
|
||||
and no reload is ever triggered.
|
||||
|
||||
## Fix options
|
||||
|
||||
**A) Grant `list` on secrets (proper hot-reload).** Additive, namespace-scoped Role
|
||||
(vmagent already has get/watch; this adds only `list`). Apply `reload-rbac.yaml`:
|
||||
|
||||
```bash
|
||||
kubectl apply -f metrics/reload-rbac.yaml
|
||||
```
|
||||
|
||||
Marginal exposure: lets the vmagent SA enumerate Secrets in the `monitoring` namespace
|
||||
only (it can already read them individually). This is what the operator is expected to
|
||||
grant; the generated Role simply omits `list` on this version.
|
||||
|
||||
**B) No RBAC change — restart on change.** Leave RBAC as-is and restart vmagent after
|
||||
applying scrape-config changes:
|
||||
|
||||
```bash
|
||||
kubectl -n monitoring rollout restart deploy/vmagent-main
|
||||
```
|
||||
|
||||
Fine for a homelab where scrape configs change rarely; the tradeoff is a manual step
|
||||
(and a ~10s gap) each time.
|
||||
@@ -0,0 +1,20 @@
|
||||
#!/usr/bin/env bash
|
||||
# kube-state-metrics — metrics about k8s objects (deployments, pods, nodes, PVCs, ...).
|
||||
# Its chart ships a ServiceMonitor; the VM operator's Prometheus converter turns that
|
||||
# into a VMServiceScrape automatically, so no hand-written scrape CR is needed here.
|
||||
#
|
||||
# Prereq: the Prometheus-operator CRDs (ServiceMonitor, ...) must exist in the cluster
|
||||
# for `serviceMonitor.enabled` to apply. If they don't:
|
||||
# kubectl apply --server-side -f \
|
||||
# https://github.com/prometheus-operator/prometheus-operator/releases/latest/download/bundle.yaml \
|
||||
# # (or just the monitoring.coreos.com CRDs)
|
||||
# Alternatively, disable the ServiceMonitor below and add a VMServiceScrape by hand.
|
||||
set -euo pipefail
|
||||
|
||||
helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
|
||||
helm repo update prometheus-community
|
||||
|
||||
helm upgrade --install kube-state-metrics prometheus-community/kube-state-metrics \
|
||||
--namespace monitoring \
|
||||
--set prometheus.monitor.enabled=true \
|
||||
--wait
|
||||
@@ -0,0 +1,67 @@
|
||||
# Host metrics (CPU/mem/disk/net/filesystem) for every node. node-exporter has no
|
||||
# ServiceMonitor of its own here, so we scrape it directly with a VMNodeScrape:
|
||||
# vmagent discovers each Node and scrapes its InternalIP:9100.
|
||||
apiVersion: apps/v1
|
||||
kind: DaemonSet
|
||||
metadata:
|
||||
name: node-exporter
|
||||
namespace: monitoring
|
||||
labels:
|
||||
app.kubernetes.io/name: node-exporter
|
||||
spec:
|
||||
selector:
|
||||
matchLabels:
|
||||
app.kubernetes.io/name: node-exporter
|
||||
template:
|
||||
metadata:
|
||||
labels:
|
||||
app.kubernetes.io/name: node-exporter
|
||||
spec:
|
||||
hostNetwork: true # bind 9100 on the node so VMNodeScrape can reach it
|
||||
hostPID: true
|
||||
tolerations:
|
||||
- operator: Exists # run on control-plane / tainted nodes too
|
||||
containers:
|
||||
- name: node-exporter
|
||||
image: quay.io/prometheus/node-exporter:v1.9.1
|
||||
args:
|
||||
- --path.procfs=/host/proc
|
||||
- --path.sysfs=/host/sys
|
||||
- --path.rootfs=/host/root
|
||||
- --collector.filesystem.mount-points-exclude=^/(dev|proc|sys|var/lib/docker/.+|var/lib/kubelet/.+)($|/)
|
||||
ports:
|
||||
- name: metrics
|
||||
containerPort: 9100
|
||||
hostPort: 9100
|
||||
resources:
|
||||
requests:
|
||||
cpu: 25m
|
||||
memory: 32Mi
|
||||
limits:
|
||||
cpu: 200m
|
||||
memory: 128Mi
|
||||
volumeMounts:
|
||||
- { name: proc, mountPath: /host/proc, readOnly: true }
|
||||
- { name: sys, mountPath: /host/sys, readOnly: true }
|
||||
- { name: root, mountPath: /host/root, readOnly: true, mountPropagation: HostToContainer }
|
||||
volumes:
|
||||
- { name: proc, hostPath: { path: /proc } }
|
||||
- { name: sys, hostPath: { path: /sys } }
|
||||
- { name: root, hostPath: { path: / } }
|
||||
---
|
||||
apiVersion: operator.victoriametrics.com/v1beta1
|
||||
kind: VMNodeScrape
|
||||
metadata:
|
||||
name: node-exporter
|
||||
namespace: monitoring
|
||||
labels:
|
||||
app.kubernetes.io/part-of: victoria-metrics
|
||||
spec:
|
||||
port: "9100"
|
||||
scheme: http
|
||||
interval: 30s
|
||||
relabelConfigs:
|
||||
- action: labelmap
|
||||
regex: __meta_kubernetes_node_label_(.+)
|
||||
- targetLabel: job
|
||||
replacement: node-exporter
|
||||
@@ -0,0 +1,29 @@
|
||||
# Enables vmagent scrape-config hot-reload. The operator-generated Role grants the
|
||||
# vmagent-main SA secrets [get,watch] but omits [list], which its config-reloader's
|
||||
# informer requires — so scrape-config changes stall until vmagent restarts.
|
||||
# This supplementary, namespace-scoped Role covers secrets get/list/watch so the
|
||||
# reloader works regardless of the operator Role. Scope: Secrets in `monitoring` only.
|
||||
# See README-reload.md for background.
|
||||
apiVersion: rbac.authorization.k8s.io/v1
|
||||
kind: Role
|
||||
metadata:
|
||||
name: vmagent-main-secret-lister
|
||||
namespace: monitoring
|
||||
rules:
|
||||
- apiGroups: [""]
|
||||
resources: ["secrets"]
|
||||
verbs: ["get", "list", "watch"]
|
||||
---
|
||||
apiVersion: rbac.authorization.k8s.io/v1
|
||||
kind: RoleBinding
|
||||
metadata:
|
||||
name: vmagent-main-secret-lister
|
||||
namespace: monitoring
|
||||
roleRef:
|
||||
apiGroup: rbac.authorization.k8s.io
|
||||
kind: Role
|
||||
name: vmagent-main-secret-lister
|
||||
subjects:
|
||||
- kind: ServiceAccount
|
||||
name: vmagent-main
|
||||
namespace: monitoring
|
||||
@@ -0,0 +1,127 @@
|
||||
# Ported from ../../../../apps/victoriametrics/rules/alerts-health.yml
|
||||
apiVersion: operator.victoriametrics.com/v1beta1
|
||||
kind: VMRule
|
||||
metadata:
|
||||
name: vm-health
|
||||
namespace: monitoring
|
||||
labels:
|
||||
app.kubernetes.io/part-of: victoria-metrics
|
||||
spec:
|
||||
groups:
|
||||
- name: vm-health
|
||||
rules:
|
||||
- alert: TooManyRestarts
|
||||
expr: changes(process_start_time_seconds{job=~".*(victoriametrics|vmselect|vminsert|vmstorage|vmagent|vmalert|vmsingle|vmalertmanager|vmauth).*"}[15m]) > 2
|
||||
labels:
|
||||
severity: critical
|
||||
annotations:
|
||||
summary: "{{ $labels.job }} too many restarts (instance {{ $labels.instance }})"
|
||||
description: >
|
||||
Job {{ $labels.job }} (instance {{ $labels.instance }}) has restarted more than twice in the last 15 minutes.
|
||||
It might be crashlooping.
|
||||
- alert: ServiceDown
|
||||
expr: up{job=~".*(victoriametrics|vmselect|vminsert|vmstorage|vmagent|vmalert|vmsingle|vmalertmanager|vmauth).*"} == 0
|
||||
for: 2m
|
||||
labels:
|
||||
severity: critical
|
||||
annotations:
|
||||
summary: "Service {{ $labels.job }} is down on {{ $labels.instance }}"
|
||||
description: "{{ $labels.instance }} of job {{ $labels.job }} has been down for more than 2 minutes."
|
||||
- alert: ProcessNearFDLimits
|
||||
expr: (process_max_fds - process_open_fds) < 100
|
||||
for: 5m
|
||||
labels:
|
||||
severity: critical
|
||||
annotations:
|
||||
summary: "Number of free file descriptors is less than 100 for \"{{ $labels.job }}\"(\"{{ $labels.instance }}\") for the last 5m"
|
||||
description: |
|
||||
Exhausting OS file descriptors limit can cause severe degradation of the process.
|
||||
Consider to increase the limit as fast as possible.
|
||||
- alert: TooHighMemoryUsage
|
||||
expr: (min_over_time(process_resident_memory_anon_bytes[10m]) / vm_available_memory_bytes) > 0.8
|
||||
for: 5m
|
||||
labels:
|
||||
severity: critical
|
||||
annotations:
|
||||
summary: "It is more than 80% of memory used by \"{{ $labels.job }}\"(\"{{ $labels.instance }}\")"
|
||||
description: |
|
||||
Too high memory usage may result into multiple issues such as OOMs or degraded performance.
|
||||
Consider to either increase available memory or decrease the load on the process.
|
||||
- alert: TooHighCPUUsage
|
||||
expr: rate(process_cpu_seconds_total[5m]) / process_cpu_cores_available > 0.9
|
||||
for: 5m
|
||||
labels:
|
||||
severity: critical
|
||||
annotations:
|
||||
summary: "More than 90% of CPU is used by \"{{ $labels.job }}\"(\"{{ $labels.instance }}\") during the last 5m"
|
||||
description: >
|
||||
Too high CPU usage may be a sign of insufficient resources and make process unstable.
|
||||
Consider to either increase available CPU resources or decrease the load on the process.
|
||||
- alert: TooHighGoroutineSchedulingLatency
|
||||
expr: histogram_quantile(0.99, sum(rate(go_sched_latencies_seconds_bucket{job=~".*(victoriametrics|vmselect|vminsert|vmstorage|vmagent|vmalert|vmsingle|vmalertmanager|vmauth).*"}[5m])) by (le, job, instance)) > 0.1
|
||||
for: 15m
|
||||
labels:
|
||||
severity: critical
|
||||
annotations:
|
||||
summary: "\"{{ $labels.job }}\"(\"{{ $labels.instance }}\") has insufficient CPU resources for >15m"
|
||||
description: >
|
||||
Go runtime is unable to schedule goroutines execution in acceptable time. This is usually a sign of
|
||||
insufficient CPU resources or CPU throttling.
|
||||
- alert: TooManyLogs
|
||||
expr: sum(increase(vm_log_messages_total{level="error"}[5m])) without (app_version, location) > 0
|
||||
for: 15m
|
||||
labels:
|
||||
severity: warning
|
||||
annotations:
|
||||
summary: "Too many logs printed for job \"{{ $labels.job }}\" ({{ $labels.instance }})"
|
||||
description: >
|
||||
Logging rate for job \"{{ $labels.job }}\" ({{ $labels.instance }}) is {{ $value }} for last 15m.
|
||||
Worth to check logs for specific error messages.
|
||||
- alert: TooManyTSIDMisses
|
||||
expr: increase(vm_missing_tsids_for_metric_id_total[5m]) > 0
|
||||
for: 15m
|
||||
labels:
|
||||
severity: critical
|
||||
annotations:
|
||||
summary: "Unexpected TSID misses for job \"{{ $labels.job }}\" ({{ $labels.instance }}) for the last 15 minutes"
|
||||
description: |
|
||||
Unexpected TSID misses for \"{{ $labels.job }}\" ({{ $labels.instance }}) for the last 15 minutes.
|
||||
If this happens after unclean shutdown of VictoriaMetrics process (via \"kill -9\", OOM or power off),
|
||||
then this is OK - the alert must go away in a few minutes after the restart.
|
||||
- alert: ConcurrentInsertsHitTheLimit
|
||||
expr: avg_over_time(vm_concurrent_insert_current[1m]) >= vm_concurrent_insert_capacity
|
||||
for: 15m
|
||||
labels:
|
||||
severity: warning
|
||||
annotations:
|
||||
summary: "{{ $labels.job }} on instance {{ $labels.instance }} is constantly hitting concurrent inserts limit"
|
||||
description: |
|
||||
The limit of concurrent inserts on instance {{ $labels.instance }} depends on the number of CPUs.
|
||||
Usually, when component constantly hits the limit it is likely the component is overloaded and requires more CPU.
|
||||
- alert: IndexDBRecordsDrop
|
||||
expr: increase(vm_indexdb_items_dropped_total[5m]) > 0
|
||||
labels:
|
||||
severity: critical
|
||||
annotations:
|
||||
summary: "IndexDB skipped registering items during data ingestion with reason={{ $labels.reason }}."
|
||||
description: |
|
||||
VictoriaMetrics could skip registering new timeseries during ingestion if they fail the validation process.
|
||||
- alert: RowsRejectedOnIngestion
|
||||
expr: rate(vm_rows_ignored_total[5m]) > 0
|
||||
for: 15m
|
||||
labels:
|
||||
severity: warning
|
||||
annotations:
|
||||
summary: "Some rows are rejected on \"{{ $labels.instance }}\" on ingestion attempt"
|
||||
description: "Ingested rows on instance \"{{ $labels.instance }}\" are rejected due to the
|
||||
following reason: \"{{ $labels.reason }}\""
|
||||
- alert: TooHighQueryLoad
|
||||
expr: increase(vm_concurrent_select_limit_timeout_total[5m]) > 0
|
||||
for: 15m
|
||||
labels:
|
||||
severity: warning
|
||||
annotations:
|
||||
summary: "Read queries fail with timeout for {{ $labels.job }} on instance {{ $labels.instance }}"
|
||||
description: |
|
||||
Instance {{ $labels.instance }} ({{ $labels.job }}) is failing to serve read queries during last 15m.
|
||||
Possible solutions: reduce the query load; increase compute resources; adjust search concurrency limits.
|
||||
@@ -0,0 +1,137 @@
|
||||
# Ported from ../../../../apps/victoriametrics/rules/alerts-vmagent.yml
|
||||
apiVersion: operator.victoriametrics.com/v1beta1
|
||||
kind: VMRule
|
||||
metadata:
|
||||
name: vmagent
|
||||
namespace: monitoring
|
||||
labels:
|
||||
app.kubernetes.io/part-of: victoria-metrics
|
||||
spec:
|
||||
groups:
|
||||
- name: vmagent
|
||||
interval: 30s
|
||||
concurrency: 2
|
||||
rules:
|
||||
- alert: PersistentQueueIsDroppingData
|
||||
expr: sum(increase(vm_persistentqueue_bytes_dropped_total[5m])) without (path) > 0
|
||||
for: 10m
|
||||
labels:
|
||||
severity: critical
|
||||
annotations:
|
||||
summary: "Instance {{ $labels.instance }} is dropping data from persistent queue"
|
||||
description: "Vmagent dropped {{ $value | humanize1024 }} from persistent queue
|
||||
on instance {{ $labels.instance }} for the last 10m."
|
||||
- alert: RejectedRemoteWriteDataBlocksAreDropped
|
||||
expr: sum(increase(vmagent_remotewrite_packets_dropped_total[5m])) without (url) > 0
|
||||
for: 15m
|
||||
labels:
|
||||
severity: warning
|
||||
annotations:
|
||||
summary: "Vmagent is dropping data blocks that are rejected by remote storage"
|
||||
description: "Job \"{{ $labels.job }}\" on instance {{ $labels.instance }} drops the rejected by
|
||||
remote-write server data blocks. Check the logs to find the reason for rejects."
|
||||
- alert: TooManyScrapeErrors
|
||||
expr: increase(vm_promscrape_scrapes_failed_total[5m]) > 0
|
||||
for: 15m
|
||||
labels:
|
||||
severity: warning
|
||||
annotations:
|
||||
summary: "Vmagent fails to scrape one or more targets"
|
||||
description: "Job \"{{ $labels.job }}\" on instance {{ $labels.instance }} fails to scrape targets for last 15m"
|
||||
- alert: ScrapePoolHasNoTargets
|
||||
expr: sum(vm_promscrape_scrape_pool_targets) without (status, instance, pod) == 0
|
||||
for: 30m
|
||||
labels:
|
||||
severity: warning
|
||||
annotations:
|
||||
summary: "Vmagent has scrape_pool with 0 configured/discovered targets"
|
||||
description: "Vmagent \"{{ $labels.job }}\" has scrape_pool \"{{ $labels.scrape_job }}\"
|
||||
with 0 discovered targets. It is likely a misconfiguration."
|
||||
- alert: TooManyWriteErrors
|
||||
expr: |
|
||||
(sum(increase(vm_ingestserver_request_errors_total[5m])) without (name,net,type)
|
||||
+
|
||||
sum(increase(vmagent_http_request_errors_total[5m])) without (path,protocol)) > 0
|
||||
for: 15m
|
||||
labels:
|
||||
severity: warning
|
||||
annotations:
|
||||
summary: "Vmagent responds with too many errors on data ingestion protocols"
|
||||
description: "Job \"{{ $labels.job }}\" on instance {{ $labels.instance }} responds with errors to write requests for last 15m."
|
||||
- alert: TooManyRemoteWriteErrors
|
||||
expr: rate(vmagent_remotewrite_retries_count_total[5m]) > 0
|
||||
for: 15m
|
||||
labels:
|
||||
severity: warning
|
||||
annotations:
|
||||
summary: "Job \"{{ $labels.job }}\" on instance {{ $labels.instance }} fails to push to remote storage"
|
||||
description: "Vmagent fails to push data via remote write protocol to destination \"{{ $labels.url }}\".
|
||||
Ensure that destination is up and reachable."
|
||||
- alert: RemoteWriteConnectionIsSaturated
|
||||
expr: |
|
||||
(
|
||||
rate(vmagent_remotewrite_send_duration_seconds_total[5m])
|
||||
/
|
||||
vmagent_remotewrite_queues
|
||||
) > 0.9
|
||||
for: 15m
|
||||
labels:
|
||||
severity: warning
|
||||
annotations:
|
||||
summary: "Remote write connection from \"{{ $labels.job }}\" (instance {{ $labels.instance }}) to {{ $labels.url }} is saturated"
|
||||
description: "The remote write connection between vmagent and destination \"{{ $labels.url }}\"
|
||||
is saturated by more than 90%. Increase -remoteWrite.queues or check the destination's capacity."
|
||||
- alert: PersistentQueueForWritesIsSaturated
|
||||
expr: rate(vm_persistentqueue_write_duration_seconds_total[5m]) > 0.9
|
||||
for: 15m
|
||||
labels:
|
||||
severity: warning
|
||||
annotations:
|
||||
summary: "Persistent queue writes for instance {{ $labels.instance }} are saturated"
|
||||
description: "Persistent queue writes are saturated by more than 90%. Reduce load or improve disk throughput."
|
||||
- alert: PersistentQueueForReadsIsSaturated
|
||||
expr: rate(vm_persistentqueue_read_duration_seconds_total[5m]) > 0.9
|
||||
for: 15m
|
||||
labels:
|
||||
severity: warning
|
||||
annotations:
|
||||
summary: "Persistent queue reads for instance {{ $labels.instance }} are saturated"
|
||||
description: "Persistent queue reads are saturated by more than 90%. Reduce load or improve disk throughput."
|
||||
- alert: SeriesLimitHourReached
|
||||
expr: (vmagent_hourly_series_limit_current_series / vmagent_hourly_series_limit_max_series) > 0.9
|
||||
labels:
|
||||
severity: critical
|
||||
annotations:
|
||||
summary: "Instance {{ $labels.instance }} reached 90% of the hourly series limit"
|
||||
description: "Max series limit set via -remoteWrite.maxHourlySeries is close to the max value."
|
||||
- alert: SeriesLimitDayReached
|
||||
expr: (vmagent_daily_series_limit_current_series / vmagent_daily_series_limit_max_series) > 0.9
|
||||
labels:
|
||||
severity: critical
|
||||
annotations:
|
||||
summary: "Instance {{ $labels.instance }} reached 90% of the daily series limit"
|
||||
description: "Max series limit set via -remoteWrite.maxDailySeries is close to the max value."
|
||||
- alert: ConfigurationReloadFailure
|
||||
expr: |
|
||||
vm_promscrape_config_last_reload_successful != 1
|
||||
or
|
||||
vmagent_relabel_config_last_reload_successful != 1
|
||||
labels:
|
||||
severity: warning
|
||||
annotations:
|
||||
summary: "Configuration reload failed for vmagent instance {{ $labels.instance }}"
|
||||
description: "Configuration hot-reload failed for vmagent on instance {{ $labels.instance }}. Check the logs."
|
||||
- alert: StreamAggrFlushTimeout
|
||||
expr: increase(vm_streamaggr_flush_timeouts_total[5m]) > 0
|
||||
labels:
|
||||
severity: warning
|
||||
annotations:
|
||||
summary: "Streaming aggregation at \"{{ $labels.job }}\" (instance {{ $labels.instance }}) can't keep up."
|
||||
description: "Stream aggregation can't finish within the configured interval and may produce incorrect results."
|
||||
- alert: StreamAggrDedupFlushTimeout
|
||||
expr: increase(vm_streamaggr_dedup_flush_timeouts_total[5m]) > 0
|
||||
labels:
|
||||
severity: warning
|
||||
annotations:
|
||||
summary: "Deduplication \"{{ $labels.job }}\" (instance {{ $labels.instance }}) can't keep up."
|
||||
description: "Deduplication can't finish within the configured interval and may produce incorrect results."
|
||||
@@ -0,0 +1,77 @@
|
||||
# Ported from ../../../../apps/victoriametrics/rules/alerts-vmalert.yml
|
||||
apiVersion: operator.victoriametrics.com/v1beta1
|
||||
kind: VMRule
|
||||
metadata:
|
||||
name: vmalert
|
||||
namespace: monitoring
|
||||
labels:
|
||||
app.kubernetes.io/part-of: victoria-metrics
|
||||
spec:
|
||||
groups:
|
||||
- name: vmalert
|
||||
interval: 30s
|
||||
rules:
|
||||
- alert: ConfigurationReloadFailure
|
||||
expr: vmalert_config_last_reload_successful != 1
|
||||
labels:
|
||||
severity: warning
|
||||
annotations:
|
||||
summary: "Configuration reload failed for vmalert instance {{ $labels.instance }}"
|
||||
description: "Configuration hot-reload failed for vmalert on instance {{ $labels.instance }}. Check the logs."
|
||||
- alert: AlertingRulesError
|
||||
expr: sum(increase(vmalert_alerting_rules_errors_total[5m])) without(id) > 0
|
||||
for: 5m
|
||||
labels:
|
||||
severity: warning
|
||||
annotations:
|
||||
summary: "Alerting rules are failing for vmalert instance {{ $labels.instance }}"
|
||||
description: "Alerting rules execution is failing for group \"{{ $labels.group }}\" in file \"{{ $labels.file }}\". Check the logs."
|
||||
- alert: RecordingRulesError
|
||||
expr: sum(increase(vmalert_recording_rules_errors_total[5m])) without(id) > 0
|
||||
for: 5m
|
||||
labels:
|
||||
severity: warning
|
||||
annotations:
|
||||
summary: "Recording rules are failing for vmalert instance {{ $labels.instance }}"
|
||||
description: "Recording rules execution is failing for group \"{{ $labels.group }}\" in file \"{{ $labels.file }}\". Check the logs."
|
||||
- alert: RecordingRulesNoData
|
||||
expr: sum(vmalert_recording_rules_last_evaluation_samples) without(id) < 1
|
||||
for: 30m
|
||||
labels:
|
||||
severity: info
|
||||
annotations:
|
||||
summary: "Recording rule ({{ $labels.group }}) produces no data"
|
||||
description: "Recording rule from group \"{{ $labels.group }}\" in file \"{{ $labels.file }}\"
|
||||
produces 0 samples over the last 30min. Possible misconfiguration or incorrect query."
|
||||
- alert: TooManyMissedIterations
|
||||
expr: increase(vmalert_iteration_missed_total[5m]) > 0
|
||||
for: 15m
|
||||
labels:
|
||||
severity: warning
|
||||
annotations:
|
||||
summary: "vmalert instance {{ $labels.instance }} is missing rules evaluations"
|
||||
description: "Group evaluation takes longer than the configured interval. Increase interval or concurrency."
|
||||
- alert: RemoteWriteErrors
|
||||
expr: increase(vmalert_remotewrite_errors_total[5m]) > 0
|
||||
for: 15m
|
||||
labels:
|
||||
severity: warning
|
||||
annotations:
|
||||
summary: "vmalert instance {{ $labels.instance }} is failing to push metrics to remote write URL"
|
||||
description: "vmalert is failing to push alerting/recording rule metrics to remote write. Check the logs."
|
||||
- alert: RemoteWriteDroppingData
|
||||
expr: increase(vmalert_remotewrite_dropped_rows_total[5m]) > 0
|
||||
for: 5m
|
||||
labels:
|
||||
severity: critical
|
||||
annotations:
|
||||
summary: "vmalert instance {{ $labels.instance }} is dropping data sent to remote write URL"
|
||||
description: "vmalert is dropping alerting/recording rule results. This may cause gaps in state. Check the logs."
|
||||
- alert: AlertmanagerErrors
|
||||
expr: increase(vmalert_alerts_send_errors_total[5m]) > 0
|
||||
for: 15m
|
||||
labels:
|
||||
severity: warning
|
||||
annotations:
|
||||
summary: "vmalert instance {{ $labels.instance }} is failing to send notifications to Alertmanager"
|
||||
description: "vmalert is failing to send alert notifications to \"{{ $labels.addr }}\". Check the logs."
|
||||
@@ -0,0 +1,127 @@
|
||||
# Ported from ../../../../apps/victoriametrics/rules/alerts.yml
|
||||
apiVersion: operator.victoriametrics.com/v1beta1
|
||||
kind: VMRule
|
||||
metadata:
|
||||
name: vmsingle
|
||||
namespace: monitoring
|
||||
labels:
|
||||
app.kubernetes.io/part-of: victoria-metrics
|
||||
spec:
|
||||
groups:
|
||||
- name: vmsingle
|
||||
interval: 30s
|
||||
concurrency: 2
|
||||
rules:
|
||||
- alert: DiskRunsOutOfSpaceIn3Days
|
||||
expr: |
|
||||
sum(vm_free_disk_space_bytes) without(path) /
|
||||
(
|
||||
(rate(vm_rows_added_to_storage_total[1d]) - sum(rate(vm_deduplicated_samples_total[1d])) without(type)) * (
|
||||
sum(vm_data_size_bytes{type!~"indexdb.*"}) without(type) /
|
||||
sum(vm_rows{type!~"indexdb.*"}) without(type)
|
||||
)
|
||||
+
|
||||
rate(vm_new_timeseries_created_total[1d]) * (
|
||||
sum(vm_data_size_bytes{type="indexdb/file"}) without(type)/
|
||||
sum(vm_rows{type="indexdb/file"}) without(type)
|
||||
)
|
||||
) < 3 * 24 * 3600 > 0
|
||||
for: 30m
|
||||
labels:
|
||||
severity: critical
|
||||
annotations:
|
||||
summary: "Instance {{ $labels.instance }} will run out of disk space soon"
|
||||
description: "Taking into account current ingestion rate, free disk space will be enough only
|
||||
for {{ $value | humanizeDuration }} on instance {{ $labels.instance }}.\n
|
||||
Consider to limit the ingestion rate, decrease retention or scale the disk space if possible."
|
||||
- alert: NodeBecomesReadonlyIn3Days
|
||||
expr: |
|
||||
sum(vm_free_disk_space_bytes - vm_free_disk_space_limit_bytes) without(path) /
|
||||
(
|
||||
(rate(vm_rows_added_to_storage_total[1d]) - sum(rate(vm_deduplicated_samples_total[1d])) without(type)) * (
|
||||
sum(vm_data_size_bytes{type!~"indexdb.*"}) without(type) /
|
||||
sum(vm_rows{type!~"indexdb.*"}) without(type)
|
||||
)
|
||||
+
|
||||
rate(vm_new_timeseries_created_total[1d]) * (
|
||||
sum(vm_data_size_bytes{type="indexdb/file"}) without(type) /
|
||||
sum(vm_rows{type="indexdb/file"}) without(type)
|
||||
)
|
||||
) < 3 * 24 * 3600 > 0
|
||||
for: 30m
|
||||
labels:
|
||||
severity: warning
|
||||
annotations:
|
||||
summary: "Instance {{ $labels.instance }} will become read-only in 3 days"
|
||||
description: "Taking into account current ingestion rate and free disk space
|
||||
instance {{ $labels.instance }} is writable for {{ $value | humanizeDuration }}.\n
|
||||
Consider to limit the ingestion rate, decrease retention or scale the disk space up if possible."
|
||||
- alert: DiskRunsOutOfSpace
|
||||
expr: |
|
||||
sum(vm_data_size_bytes) by(job, instance) /
|
||||
(
|
||||
sum(vm_free_disk_space_bytes) by(job, instance) +
|
||||
sum(vm_data_size_bytes) by(job, instance)
|
||||
) > 0.8
|
||||
for: 30m
|
||||
labels:
|
||||
severity: critical
|
||||
annotations:
|
||||
summary: "Instance {{ $labels.instance }} (job={{ $labels.job }}) will run out of disk space soon"
|
||||
description: "Disk utilisation on instance {{ $labels.instance }} is more than 80%.\n
|
||||
Having less than 20% of free disk space could cripple merge processes and overall performance.
|
||||
Consider to limit the ingestion rate, decrease retention or scale the disk space if possible."
|
||||
- alert: RequestErrorsToAPI
|
||||
expr: increase(vm_http_request_errors_total[5m]) > 0
|
||||
for: 15m
|
||||
labels:
|
||||
severity: warning
|
||||
annotations:
|
||||
summary: "Too many errors served for path {{ $labels.path }} (instance {{ $labels.instance }})"
|
||||
description: "Requests to path {{ $labels.path }} are receiving errors.
|
||||
Please verify if clients are sending correct requests."
|
||||
- alert: TooHighChurnRate
|
||||
expr: |
|
||||
(
|
||||
sum(rate(vm_new_timeseries_created_total[5m])) by(instance)
|
||||
/
|
||||
sum(rate(vm_rows_inserted_total[5m])) by(instance)
|
||||
) > 0.1
|
||||
for: 15m
|
||||
labels:
|
||||
severity: warning
|
||||
annotations:
|
||||
summary: "Churn rate is more than 10% on \"{{ $labels.instance }}\" for the last 15m"
|
||||
description: "VM constantly creates new time series on \"{{ $labels.instance }}\".\n
|
||||
This effect is known as Churn Rate.\n
|
||||
High Churn Rate is tightly connected with database performance and may
|
||||
result in unexpected OOM's or slow queries."
|
||||
- alert: TooHighChurnRate24h
|
||||
expr: |
|
||||
sum(increase(vm_new_timeseries_created_total[24h])) by(instance)
|
||||
>
|
||||
(sum(vm_cache_entries{type="storage/hour_metric_ids"}) by(instance) * 3)
|
||||
for: 15m
|
||||
labels:
|
||||
severity: warning
|
||||
annotations:
|
||||
summary: "Too high number of new series on \"{{ $labels.instance }}\" created over last 24h"
|
||||
description: "The number of created new time series over last 24h is 3x times higher than
|
||||
current number of active series on \"{{ $labels.instance }}\".\n
|
||||
This effect is known as Churn Rate.\n
|
||||
High Churn Rate is tightly connected with database performance and may
|
||||
result in unexpected OOM's or slow queries."
|
||||
- alert: TooHighSlowInsertsRate
|
||||
expr: |
|
||||
(
|
||||
sum(rate(vm_slow_row_inserts_total[5m])) by(instance)
|
||||
/
|
||||
sum(rate(vm_rows_inserted_total[5m])) by(instance)
|
||||
) > 0.05
|
||||
for: 15m
|
||||
labels:
|
||||
severity: warning
|
||||
annotations:
|
||||
summary: "Percentage of slow inserts is more than 5% on \"{{ $labels.instance }}\" for the last 15m"
|
||||
description: "High rate of slow inserts on \"{{ $labels.instance }}\" may be a sign of resource exhaustion
|
||||
for the current load. It is likely more RAM is needed for optimal handling of the current number of active time series."
|
||||
@@ -0,0 +1,30 @@
|
||||
# Blocky DNS metrics. Same VMStaticScrape pattern as docker-hosts.yaml, because
|
||||
# Blocky runs as a compose stack on the laptop rather than in the cluster
|
||||
# (see ../../../../apps/blocky/README.md for why DNS deliberately lives outside k3s).
|
||||
#
|
||||
# ⚠ Blocky is STAGED, not deployed — this target will be DOWN until it is started.
|
||||
# That is expected; do not go hunting for a broken exporter.
|
||||
#
|
||||
# Worth alerting on once it is live, because there is currently no DNS visibility
|
||||
# at all:
|
||||
# blocky_error_total — upstream failures
|
||||
# blocky_denylist_cache_entries — collapses to 0 if a list fetch fails
|
||||
# blocky_cache_hits_total / _misses_total — cache effectiveness
|
||||
# blocky_query_total{response_type=...} — BLOCKED vs CACHED vs RESOLVED split
|
||||
---
|
||||
apiVersion: operator.victoriametrics.com/v1beta1
|
||||
kind: VMStaticScrape
|
||||
metadata:
|
||||
name: blocky
|
||||
namespace: monitoring
|
||||
labels:
|
||||
app.kubernetes.io/part-of: victoria-metrics
|
||||
spec:
|
||||
jobName: blocky
|
||||
targetEndpoints:
|
||||
- targets:
|
||||
- "192.168.10.127:4000"
|
||||
path: /metrics
|
||||
labels:
|
||||
job: blocky
|
||||
host: docker-01
|
||||
@@ -0,0 +1,29 @@
|
||||
# Pull metrics from Docker hosts running ../../docker-hosts/compose.yaml.
|
||||
# vmagent scrapes each host's node-exporter (:9100) and cAdvisor (:8080) over the LAN.
|
||||
# Add one target block per host; keep the `host` label in sync with its HOST_LABEL.
|
||||
apiVersion: operator.victoriametrics.com/v1beta1
|
||||
kind: VMStaticScrape
|
||||
metadata:
|
||||
name: docker-hosts
|
||||
namespace: monitoring
|
||||
labels:
|
||||
app.kubernetes.io/part-of: victoria-metrics
|
||||
spec:
|
||||
jobName: docker-hosts
|
||||
targetEndpoints:
|
||||
# --- docker-01 (192.168.10.127) ---
|
||||
- targets:
|
||||
- "192.168.10.127:9100"
|
||||
labels:
|
||||
job: node-exporter
|
||||
host: docker-01
|
||||
- targets:
|
||||
- "192.168.10.127:8080"
|
||||
labels:
|
||||
job: cadvisor
|
||||
host: docker-01
|
||||
# --- add more hosts below, mirroring the two blocks above ---
|
||||
# - targets: ["192.168.10.x:9100"]
|
||||
# labels: { job: node-exporter, host: docker-02 }
|
||||
# - targets: ["192.168.10.x:8080"]
|
||||
# labels: { job: cadvisor, host: docker-02 }
|
||||
@@ -0,0 +1,47 @@
|
||||
# Kubelet + cAdvisor metrics (per-node pod/container CPU, memory, network, fs).
|
||||
# Scraped over the kubelet's authenticated https port using the vmagent pod's
|
||||
# ServiceAccount token. Pattern from the operator docs (VMNodeScrape / cadvisor).
|
||||
apiVersion: operator.victoriametrics.com/v1beta1
|
||||
kind: VMNodeScrape
|
||||
metadata:
|
||||
name: kubelet
|
||||
namespace: monitoring
|
||||
labels:
|
||||
app.kubernetes.io/part-of: victoria-metrics
|
||||
spec:
|
||||
scheme: https
|
||||
honorLabels: true
|
||||
honorTimestamps: false
|
||||
interval: 30s
|
||||
tlsConfig:
|
||||
insecureSkipVerify: true
|
||||
caFile: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
|
||||
bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token
|
||||
relabelConfigs:
|
||||
- action: labelmap
|
||||
regex: __meta_kubernetes_node_label_(.+)
|
||||
- targetLabel: job
|
||||
replacement: kubelet
|
||||
---
|
||||
apiVersion: operator.victoriametrics.com/v1beta1
|
||||
kind: VMNodeScrape
|
||||
metadata:
|
||||
name: cadvisor
|
||||
namespace: monitoring
|
||||
labels:
|
||||
app.kubernetes.io/part-of: victoria-metrics
|
||||
spec:
|
||||
scheme: https
|
||||
honorLabels: true
|
||||
honorTimestamps: false
|
||||
interval: 30s
|
||||
path: /metrics/cadvisor
|
||||
tlsConfig:
|
||||
insecureSkipVerify: true
|
||||
caFile: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
|
||||
bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token
|
||||
relabelConfigs:
|
||||
- action: labelmap
|
||||
regex: __meta_kubernetes_node_label_(.+)
|
||||
- targetLabel: job
|
||||
replacement: cadvisor
|
||||
@@ -0,0 +1,21 @@
|
||||
# Scrape agent. selectAllByDefault picks up every VMServiceScrape / VMPodScrape /
|
||||
# VMNodeScrape / VMStaticScrape in ALL namespaces, so app teams add their own
|
||||
# scrape CRs and vmagent discovers them automatically. Writes to VMSingle.
|
||||
apiVersion: operator.victoriametrics.com/v1beta1
|
||||
kind: VMAgent
|
||||
metadata:
|
||||
name: main
|
||||
namespace: monitoring
|
||||
spec:
|
||||
replicaCount: 1
|
||||
selectAllByDefault: true
|
||||
scrapeInterval: 30s
|
||||
remoteWrite:
|
||||
- url: http://vmsingle-main.monitoring.svc:8428/api/v1/write
|
||||
resources:
|
||||
requests:
|
||||
cpu: 100m
|
||||
memory: 256Mi
|
||||
limits:
|
||||
cpu: "1"
|
||||
memory: 1Gi
|
||||
@@ -0,0 +1,27 @@
|
||||
# Rule evaluator. selectAllByDefault picks up every VMRule in all namespaces
|
||||
# (see ./rules). Reads from and writes alert/recording state to VMSingle;
|
||||
# sends firing alerts to VMAlertmanager.
|
||||
apiVersion: operator.victoriametrics.com/v1beta1
|
||||
kind: VMAlert
|
||||
metadata:
|
||||
name: main
|
||||
namespace: monitoring
|
||||
spec:
|
||||
replicaCount: 1
|
||||
selectAllByDefault: true
|
||||
evaluationInterval: 30s
|
||||
datasource:
|
||||
url: http://vmsingle-main.monitoring.svc:8428
|
||||
remoteWrite:
|
||||
url: http://vmsingle-main.monitoring.svc:8428
|
||||
remoteRead:
|
||||
url: http://vmsingle-main.monitoring.svc:8428
|
||||
notifier:
|
||||
url: http://vmalertmanager-main.monitoring.svc:9093
|
||||
resources:
|
||||
requests:
|
||||
cpu: 50m
|
||||
memory: 128Mi
|
||||
limits:
|
||||
cpu: 500m
|
||||
memory: 512Mi
|
||||
@@ -0,0 +1,30 @@
|
||||
# Alert router/notifier. Ported 1:1 from ../../../apps/victoriametrics/alertmanager.yaml,
|
||||
# which currently blackholes everything. Wire real receivers here (email via the
|
||||
# in-cluster smtp-relay, or a webhook) when you want notifications.
|
||||
apiVersion: operator.victoriametrics.com/v1beta1
|
||||
kind: VMAlertmanager
|
||||
metadata:
|
||||
name: main
|
||||
namespace: monitoring
|
||||
spec:
|
||||
replicaCount: 1
|
||||
configRawYaml: |
|
||||
route:
|
||||
receiver: blackhole
|
||||
receivers:
|
||||
- name: blackhole
|
||||
# Example email receiver via the in-cluster Postfix relay (smtp-relay/):
|
||||
# receivers:
|
||||
# - name: email
|
||||
# email_configs:
|
||||
# - to: '[email protected]'
|
||||
# from: 'Alertmanager <[email protected]>'
|
||||
# smarthost: 'smtp-relay.smtp-relay.svc.cluster.local:25'
|
||||
# require_tls: false
|
||||
resources:
|
||||
requests:
|
||||
cpu: 25m
|
||||
memory: 64Mi
|
||||
limits:
|
||||
cpu: 200m
|
||||
memory: 256Mi
|
||||
@@ -0,0 +1,26 @@
|
||||
# Single-node metrics TSDB. Replaces the Compose `victoriametrics` service.
|
||||
# Service exposed by the operator: vmsingle-main.monitoring.svc:8428
|
||||
apiVersion: operator.victoriametrics.com/v1beta1
|
||||
kind: VMSingle
|
||||
metadata:
|
||||
name: main
|
||||
namespace: monitoring
|
||||
spec:
|
||||
retentionPeriod: "6" # months
|
||||
removePvcAfterDelete: false # keep data if the CR is deleted
|
||||
storage:
|
||||
accessModes:
|
||||
- ReadWriteOnce
|
||||
storageClassName: localpv-zfs-ceph # OpenEBS ZFS (data/ceph pool)
|
||||
resources:
|
||||
requests:
|
||||
storage: 50Gi
|
||||
extraArgs:
|
||||
dedup.minScrapeInterval: 30s
|
||||
resources:
|
||||
requests:
|
||||
cpu: 250m
|
||||
memory: 512Mi
|
||||
limits:
|
||||
cpu: "2"
|
||||
memory: 4Gi
|
||||
@@ -0,0 +1,6 @@
|
||||
apiVersion: v1
|
||||
kind: Namespace
|
||||
metadata:
|
||||
name: monitoring
|
||||
labels:
|
||||
app.kubernetes.io/part-of: observability
|
||||
Executable
+20
@@ -0,0 +1,20 @@
|
||||
#!/usr/bin/env bash
|
||||
# Install / upgrade the VictoriaMetrics operator into the monitoring namespace.
|
||||
# The operator owns the VM* + VLSingle CRDs that everything else in this folder relies on.
|
||||
set -euo pipefail
|
||||
|
||||
kubectl apply -f ../namespace.yaml
|
||||
|
||||
helm repo add vm https://victoriametrics.github.io/helm-charts/
|
||||
helm repo update vm
|
||||
|
||||
# Pin the chart version after the first install for reproducibility:
|
||||
# helm search repo vm/victoria-metrics-operator --versions | head
|
||||
# then add: --version <x.y.z>
|
||||
helm upgrade --install vm-operator vm/victoria-metrics-operator \
|
||||
--namespace monitoring \
|
||||
--values values.yaml \
|
||||
--wait
|
||||
|
||||
# CRDs land cluster-wide; verify:
|
||||
# kubectl get crd | grep victoriametrics
|
||||
@@ -0,0 +1,38 @@
|
||||
# victoria-metrics-operator — manages the VM* CRDs (VMSingle, VMAgent, VMAlert,
|
||||
# VMAlertmanager, VMRule, VMServiceScrape, VMNodeScrape, ...) AND VLSingle (VictoriaLogs).
|
||||
# One operator reconciles the whole metrics + logs stack declaratively.
|
||||
#
|
||||
# Install: ./helm.sh (chart: vm/victoria-metrics-operator)
|
||||
|
||||
# Let the chart install/upgrade the CRDs.
|
||||
crds:
|
||||
plain: true
|
||||
cleanup:
|
||||
enabled: false # keep CRDs (and thus CRs) if the operator is uninstalled
|
||||
|
||||
# Watch every namespace so app-owned VMServiceScrape/VMPodScrape/VMRule are picked up.
|
||||
operator:
|
||||
# convert legacy Prometheus-operator CRs too, harmless if none exist
|
||||
disable_prometheus_converter: false
|
||||
|
||||
# Pin VictoriaMetrics' native config-reloader (this is already the operator default).
|
||||
# NOTE: this reloader watches the config Secret via a client-go informer (list+watch).
|
||||
# The operator-generated vmagent Role grants secrets [get,watch] but NOT [list], so on
|
||||
# k3s the informer fails and scrape-config hot-reload stalls until vmagent is restarted.
|
||||
# See ../metrics/README-reload.md for the fix options.
|
||||
env:
|
||||
- name: VM_USECUSTOMCONFIGRELOADER
|
||||
value: "true"
|
||||
|
||||
# The operator only reconciles CRs; the actual components (vmsingle, vmagent, ...) are
|
||||
# defined as CRs under ../metrics and ../logs. Keep the operator itself lean.
|
||||
resources:
|
||||
requests:
|
||||
cpu: 50m
|
||||
memory: 150Mi
|
||||
limits:
|
||||
cpu: 500m
|
||||
memory: 500Mi
|
||||
|
||||
serviceMonitor:
|
||||
enabled: true # let the operator self-scrape via a VMServiceScrape
|
||||
Executable
+13
@@ -0,0 +1,13 @@
|
||||
#!/usr/bin/env bash
|
||||
# Deploy tracing: VictoriaTraces (VTSingle CR, operator-managed) + the OpenTelemetry
|
||||
# Collector that fronts it. Apps send OTLP to otel-collector.monitoring.svc:4317 (gRPC)
|
||||
# / :4318 (HTTP); the collector batches and forwards to VictoriaTraces.
|
||||
#
|
||||
# NOTE: we use the operator's VTSingle CRD (traces/vtsingle.yaml) rather than the
|
||||
# victoria-traces-single Helm chart — one operator manages metrics, logs, and traces.
|
||||
# traces-values.yaml is kept only as a reference for the standalone-chart alternative.
|
||||
set -euo pipefail
|
||||
|
||||
kubectl apply -f vtsingle.yaml
|
||||
kubectl -n monitoring wait --for=jsonpath='{.status.updateStatus}'=operational vtsingle/main --timeout=120s || true
|
||||
kubectl apply -f otel-collector.yaml
|
||||
@@ -0,0 +1,117 @@
|
||||
# OpenTelemetry Collector — OTLP ingress point for app traces. Receives OTLP over
|
||||
# gRPC (4317) and HTTP (4318), batches, and exports to VictoriaTraces' OTLP endpoint.
|
||||
# Point your apps' OTEL_EXPORTER_OTLP_ENDPOINT at otel-collector.monitoring.svc:4317.
|
||||
apiVersion: v1
|
||||
kind: ServiceAccount
|
||||
metadata:
|
||||
name: otel-collector
|
||||
namespace: monitoring
|
||||
---
|
||||
apiVersion: v1
|
||||
kind: ConfigMap
|
||||
metadata:
|
||||
name: otel-collector-config
|
||||
namespace: monitoring
|
||||
data:
|
||||
config.yaml: |
|
||||
receivers:
|
||||
otlp:
|
||||
protocols:
|
||||
grpc:
|
||||
endpoint: 0.0.0.0:4317
|
||||
http:
|
||||
endpoint: 0.0.0.0:4318
|
||||
processors:
|
||||
memory_limiter:
|
||||
check_interval: 5s
|
||||
limit_percentage: 80
|
||||
spike_limit_percentage: 25
|
||||
batch:
|
||||
timeout: 5s
|
||||
exporters:
|
||||
otlphttp/victoriatraces:
|
||||
# VictoriaTraces (VTSingle CR) OTLP-over-HTTP ingestion endpoint.
|
||||
traces_endpoint: http://vtsingle-main.monitoring.svc:10428/insert/opentelemetry/v1/traces
|
||||
tls:
|
||||
insecure: true
|
||||
service:
|
||||
pipelines:
|
||||
traces:
|
||||
receivers: [otlp]
|
||||
processors: [memory_limiter, batch]
|
||||
exporters: [otlphttp/victoriatraces]
|
||||
telemetry:
|
||||
metrics:
|
||||
address: 0.0.0.0:8888
|
||||
---
|
||||
apiVersion: apps/v1
|
||||
kind: Deployment
|
||||
metadata:
|
||||
name: otel-collector
|
||||
namespace: monitoring
|
||||
labels:
|
||||
app.kubernetes.io/name: otel-collector
|
||||
spec:
|
||||
replicas: 1
|
||||
selector:
|
||||
matchLabels:
|
||||
app.kubernetes.io/name: otel-collector
|
||||
template:
|
||||
metadata:
|
||||
labels:
|
||||
app.kubernetes.io/name: otel-collector
|
||||
spec:
|
||||
serviceAccountName: otel-collector
|
||||
containers:
|
||||
- name: otel-collector
|
||||
# Pin to a current release; check https://github.com/open-telemetry/opentelemetry-collector-releases
|
||||
image: otel/opentelemetry-collector-contrib:0.119.0
|
||||
args: ["--config=/etc/otel/config.yaml"]
|
||||
ports:
|
||||
- { name: otlp-grpc, containerPort: 4317 }
|
||||
- { name: otlp-http, containerPort: 4318 }
|
||||
- { name: metrics, containerPort: 8888 }
|
||||
volumeMounts:
|
||||
- name: config
|
||||
mountPath: /etc/otel
|
||||
resources:
|
||||
requests:
|
||||
cpu: 50m
|
||||
memory: 128Mi
|
||||
limits:
|
||||
cpu: 500m
|
||||
memory: 512Mi
|
||||
volumes:
|
||||
- name: config
|
||||
configMap:
|
||||
name: otel-collector-config
|
||||
---
|
||||
apiVersion: v1
|
||||
kind: Service
|
||||
metadata:
|
||||
name: otel-collector
|
||||
namespace: monitoring
|
||||
labels:
|
||||
app.kubernetes.io/name: otel-collector
|
||||
spec:
|
||||
selector:
|
||||
app.kubernetes.io/name: otel-collector
|
||||
ports:
|
||||
- { name: otlp-grpc, port: 4317, targetPort: 4317 }
|
||||
- { name: otlp-http, port: 4318, targetPort: 4318 }
|
||||
- { name: metrics, port: 8888, targetPort: 8888 }
|
||||
---
|
||||
# Self-scrape the collector's own telemetry into VictoriaMetrics.
|
||||
apiVersion: operator.victoriametrics.com/v1beta1
|
||||
kind: VMServiceScrape
|
||||
metadata:
|
||||
name: otel-collector
|
||||
namespace: monitoring
|
||||
labels:
|
||||
app.kubernetes.io/part-of: victoria-metrics
|
||||
spec:
|
||||
selector:
|
||||
matchLabels:
|
||||
app.kubernetes.io/name: otel-collector
|
||||
endpoints:
|
||||
- port: metrics
|
||||
@@ -0,0 +1,21 @@
|
||||
# victoria-traces-single — single-node distributed tracing backend. No external
|
||||
# storage deps. Ingests OTLP (from the OTel Collector) and exposes a Jaeger Query
|
||||
# API that Grafana reads via a Jaeger datasource.
|
||||
# Service: victoria-traces-single-server.monitoring.svc:10428
|
||||
#
|
||||
# NOTE: confirm key names against the installed chart before applying:
|
||||
# helm show values vm/victoria-traces-single
|
||||
|
||||
server:
|
||||
retentionPeriod: 30d
|
||||
persistentVolume:
|
||||
enabled: true
|
||||
storageClassName: localpv-zfs-ceph
|
||||
size: 20Gi
|
||||
resources:
|
||||
requests:
|
||||
cpu: 100m
|
||||
memory: 256Mi
|
||||
limits:
|
||||
cpu: "1"
|
||||
memory: 2Gi
|
||||
@@ -0,0 +1,25 @@
|
||||
# Single-node VictoriaTraces, managed by the VM operator (no separate Helm chart needed
|
||||
# — the operator ships the VTSingle CRD). Ingests OTLP from the OTel Collector and
|
||||
# exposes a Jaeger-compatible query API that Grafana reads.
|
||||
# Service: vtsingle-main.monitoring.svc:10428
|
||||
apiVersion: operator.victoriametrics.com/v1
|
||||
kind: VTSingle
|
||||
metadata:
|
||||
name: main
|
||||
namespace: monitoring
|
||||
spec:
|
||||
retentionPeriod: "30d"
|
||||
storage:
|
||||
accessModes:
|
||||
- ReadWriteOnce
|
||||
storageClassName: localpv-zfs-ceph
|
||||
resources:
|
||||
requests:
|
||||
storage: 20Gi
|
||||
resources:
|
||||
requests:
|
||||
cpu: 100m
|
||||
memory: 256Mi
|
||||
limits:
|
||||
cpu: "1"
|
||||
memory: 2Gi
|
||||
@@ -0,0 +1,22 @@
|
||||
# ZFS-backed StorageClass for openebs zfs-localpv (provisioner zfs.csi.openebs.io).
|
||||
# Standalone resource — NOT part of the openebs Helm release. Backs PVCs on ZFS pool
|
||||
# `data/ceph`. Apply with: kubectl apply -f storageclasses.yaml
|
||||
apiVersion: storage.k8s.io/v1
|
||||
kind: StorageClass
|
||||
metadata:
|
||||
name: localpv-zfs-ceph
|
||||
provisioner: zfs.csi.openebs.io
|
||||
reclaimPolicy: Delete
|
||||
volumeBindingMode: WaitForFirstConsumer
|
||||
allowVolumeExpansion: true
|
||||
parameters:
|
||||
poolname: data/ceph # ZFS dataset the volumes are carved from
|
||||
csi.storage.k8s.io/fstype: xfs
|
||||
compression: lz4
|
||||
dedup: "off"
|
||||
atime: "off"
|
||||
relatime: "on"
|
||||
recordsize: 128k
|
||||
volblocksize: 4k
|
||||
thinprovision: "yes"
|
||||
xattr: sa
|
||||
@@ -0,0 +1,26 @@
|
||||
# openebs umbrella chart (openebs/openebs) values — single-node homelab.
|
||||
# Install/upgrade:
|
||||
# helm upgrade --install openebs openebs/openebs --version 4.4.0 -n openebs -f values.yaml
|
||||
|
||||
# Local ZFS provisioner ON — backs the cluster's PVCs (StorageClass localpv-zfs-ceph on
|
||||
# pool data/ceph, see storageclasses.yaml). LVM + rawfile OFF (unused — no LVM
|
||||
# StorageClasses/PVs/volumes). Replicated (Mayastor) OFF. Pinned explicitly so a future
|
||||
# chart-default change can't silently flip storage engines.
|
||||
engines:
|
||||
local:
|
||||
zfs:
|
||||
enabled: true
|
||||
lvm:
|
||||
enabled: false
|
||||
rawfile:
|
||||
enabled: false
|
||||
replicated:
|
||||
mayastor:
|
||||
enabled: false
|
||||
|
||||
# The bundled Loki (logging) defaults to 3 SingleBinary replicas with one-pod-per-node
|
||||
# anti-affinity. On this single node, replicas 2 & 3 are permanently Pending. Filesystem-
|
||||
# backed SingleBinary Loki should be 1 replica anyway.
|
||||
loki:
|
||||
singleBinary:
|
||||
replicas: 1
|
||||
Reference in New Issue
Block a user