diff --git a/CHANGELOG.md b/CHANGELOG.md index 7535f4a..7a4d612 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -26,12 +26,12 @@ What changed in this homelab, when, and why. Newest first. | secrets | Recorded that ESO 2.8.0, five ExternalSecrets and the scoped OpenBao Kubernetes-auth path already exist; the next gate is live recovery testing and migration of any remaining manual Secrets | | Terraform | Recorded Gitea 1.27 State Registry as the preferred candidate for local roots after version and recovery testing; the OCI recovery root remains in OCI Object Storage to avoid a home-control-plane dependency loop | | cleanup | Removed the retired NapCat tree, the Contour and Kanidm archive trees, and seven generated Terraform plan files before establishing the clean Git baseline; plans may embed complete state and remain globally ignored | -| cleanup | Removed the retired NapCat tree and seven generated Terraform plan files before establishing the clean Git baseline; plans may embed complete state and remain globally ignored | +| CI | Added a review-first Gitea Actions runner bootstrap: official actions chart 0.1.1, pinned runner 2.3.0, one persistent Kubernetes runner with capacity four and rootless DinD, plus an ESO reference to a repository-scoped registration token in OpenBao. It is not deployed until the PR is merged | `Carried forward`: re-verify OpenBao/ESO recovery and remaining Secret inventory; configure a Git remote and off-site mirror; confirm the running Gitea version; take an encrypted independent OCI state copy before enabling bucket versioning; -reconstruct the missing root to a zero-change plan; bootstrap Woodpecker and Flux on a low-risk +reconstruct the missing root to a zero-change plan; bootstrap Gitea Actions and Flux on a low-risk service; then move Tunnel origins to Envoy one hostname at a time. ## 2026-08-15 diff --git a/docs/cicd.md b/docs/cicd.md index a7deb79..6c35f3a 100644 --- a/docs/cicd.md +++ b/docs/cicd.md @@ -84,19 +84,22 @@ doing once the pipeline is trustworthy. A documented break-glass path must survive it: the playbooks still have to run by hand from the laptop during an incident, or enforcement locks you out exactly when you need in. -## Substrate — open +## Substrate — decided -**Requirement:** a light always-on orchestrator with ephemeral workers. Not a VM -or LXC parked idle waiting for work. Something must always listen — that is -inherent to event-driven CI — but it should be a controller, not a pet. +Gitea Actions is the CI control plane. It integrates directly with repository +permissions and status checks and preserves GitHub Actions workflow syntax. -| option | verdict | -|---|---| -| **Woodpecker CI** | **Front-runner.** Kubernetes backend runs each step as a standalone Pod; first-class Gitea (OAuth2 + auto-created webhooks); server + agent, both light; stable. Costs a different pipeline syntax, which for shell-step lint jobs is ~20 lines | -| `act_runner` (Gitea native) | Docker or host execution **only** — no Kubernetes executor. Confirmed in source: `labels.go` has just `SchemeDocker`/`SchemeHost`, and `run_context.go` branches only to `startHostEnvironment` or `startJobContainer`. Cheapest (one pod) but leaves a persistent worker | -| Gitea ARC | Real pod-per-job operator, but **Enterprise Edition only** | -| GARM + `garm-provider-k8s` | Right shape. GARM supports Gitea from 1.24, but latest is **v0.2.0-beta1** and the provider documents GitHub runners only — the Gitea pairing is unverified | -| Write a runner from zero | The protocol (`actions-proto-go`) is approachable; reimplementing execution is not — `act_runner` delegates that to a vendored `nektos/act`. A shim spawning one-shot pods is blocked on ephemeral registration (`go-apps/gitea/gitea#32461`) | +The bootstrap worker is the official Gitea Runner chart in Kubernetes: one +persistent StatefulSet Pod, rootless Docker-in-Docker and capacity four. Job +containers are dynamic, while the runner and its Docker daemon remain resident. +Rootless DinD still needs a privileged Pod to establish its user namespace, so +the runner is repository-scoped and restricted to trusted workflows. See +`platform/gitea-runner/`. + +This is not native pod-per-job execution. If stronger isolation becomes useful, +the runner's ephemeral registration and Gitea `workflow_job` webhook can later +drive a small controller that creates one runner Pod per job. That complexity is +not required for the bootstrap. **Where runners run** is a separate axis. Argument for Proxmox: pve1–3 have clean egress, while the k3s node carries `openvpn-client@naist`, whose 58 split-tunnel @@ -107,19 +110,16 @@ Proxmox provider exists, so ephemeral Proxmox VMs would mean writing one. ## Prerequisites -Blocking, in order: +Bootstrap dependencies and current status: -1. **Externalise secrets.** `apps/gitea/gitea-values.yaml`, `apps/authelia/values.yaml`, - `infrastructure/cloudflared/cloudflared.yaml` and `litellm-gateway/docker-compose.yml` are - gitignored because they embed live credentials. GitOps requires the opposite — - a reconciler can only apply what is in git. OpenBao + External Secrets is the - natural fit; `openbao_bootstrap/tasks/auth_kubernetes.yml` already exists. -2. **A git remote.** Gitea is the obvious host, but it runs *on* the cluster it - would deploy, so keep an off-cluster mirror for disaster recovery. -3. **Terraform: shared state.** All four roots use local `*.tfstate`. Any - pipeline that applies needs shared state — the S3-compatible SeaweedFS/rustfs - already on the cluster is a candidate, with the caveat that cluster state - living on the cluster is the same circularity as hosting git there. +1. **Secret delivery exists.** OpenBao and External Secrets Operator already + synchronize five Secrets. Add the repository-scoped runner registration token + at `kv/k8s/gitea-runner`; Git contains only its `ExternalSecret` reference. +2. **Git remote exists.** `homelab-infra` is hosted in Gitea. An off-cluster + read-only mirror remains required for disaster recovery. +3. **Terraform state is mixed.** Local roots currently use SeaweedFS S3. Gitea + 1.27 State Registry is the preferred future backend for local roots, but the + running Gitea is 1.25.5. OCI recovery state remains in OCI Object Storage. **Not a blocker: non-interactive auth.** An earlier draft of this document called `bao login -method=oidc` and `az login` hard blockers that would widen the diff --git a/docs/homelab-gitops-redesign.md b/docs/homelab-gitops-redesign.md index aee06e3..29f84b1 100644 --- a/docs/homelab-gitops-redesign.md +++ b/docs/homelab-gitops-redesign.md @@ -83,8 +83,11 @@ without becoming another deployment controller. ### CI execution -Most CI runs as ephemeral Kubernetes Pods through Woodpecker's Kubernetes -backend: lint, formatting, manifest rendering, unit tests and ordinary plans. +Most CI uses Gitea Actions for its GitHub Actions compatibility. A persistent +Gitea Runner StatefulSet runs in Kubernetes with rootless Docker-in-Docker and +capacity four; individual job containers are created dynamically. Rootless DinD +still requires a privileged Pod, so the runner is repository-scoped and accepts +trusted workflows only. Only explicitly labelled jobs needing privilege, nested virtualization, amd64-only software or isolation from k3s use an ephemeral Proxmox VM. IaC owns @@ -198,8 +201,8 @@ offline break-glass path. ESO-generated Secrets are projections, not backups. remaining manually managed Secrets, then migrate them incrementally. 3. Configure Gitea remote plus a one-way off-site mirror. Confirm whether the running Gitea supports the 1.27 Terraform State Registry. -4. Deploy Woodpecker, then bootstrap Flux on `http-echo` or `marker` without - enabling prune until live ownership is audited. +4. Manually deploy the reviewed Gitea Runner bootstrap, then bootstrap Flux on + `http-echo` or `marker` without enabling prune until live ownership is audited. 5. Move Tunnel origins to Envoy and consolidate split DNS through Blocky. 6. Deploy Backstage read-only with Catalog, Kubernetes, Flux and TechDocs. 7. Reconstruct the OCI root to a zero-change plan and add libvirt drift reports. diff --git a/platform/gitea-runner/README.md b/platform/gitea-runner/README.md new file mode 100644 index 0000000..2d1e7c1 --- /dev/null +++ b/platform/gitea-runner/README.md @@ -0,0 +1,63 @@ +# Gitea Actions runner + +This is the bootstrap runner for Gitea Actions. One persistent runner Pod accepts +up to four jobs; each job runs in a dynamically created container inside a +rootless Docker-in-Docker daemon. Rootless DinD still requires a privileged Pod +to create its user namespace, so this runner is restricted to this repository +and trusted workflows. + +The runner registration token is authoritative in OpenBao at +`kv/k8s/gitea-runner`. External Secrets Operator projects its `token` property to +the `gitea-runner-token` Secret. Never put the token in this directory or a Helm +command line. + +## Review-first bootstrap + +This is a one-time manual deployment because Flux is not installed yet: + +1. Merge the reviewed PR. +2. Create a repository-scoped runner registration token in Gitea. +3. Store it as the `token` property at `kv/k8s/gitea-runner` without exposing it + in shell history: + + ```bash + read -rsp 'Runner token: ' runner_token + printf '%s' "$runner_token" | bao kv put kv/k8s/gitea-runner token=- + unset runner_token + ``` + +4. From the updated `main`, create the namespace and ExternalSecret, then wait + for `SecretSynced=True`: + + ```bash + KUBECONFIG="$HOME/.kube/config" k3s kubectl apply \ + -f platform/gitea-runner/namespace.yaml + KUBECONFIG="$HOME/.kube/config" k3s kubectl apply \ + -f platform/gitea-runner/external-secret.yaml + KUBECONFIG="$HOME/.kube/config" k3s kubectl wait \ + --namespace gitea-actions \ + --for=condition=Ready externalsecret/gitea-runner-token \ + --timeout=60s + ``` + +5. Install chart `actions` version `0.1.1` from + `https://dl.gitea.com/charts/` with this `values.yaml`: + + ```bash + helm repo add gitea-charts https://dl.gitea.com/charts/ + helm repo update gitea-charts + helm upgrade --install gitea-actions gitea-charts/actions \ + --namespace gitea-actions \ + --version 0.1.1 \ + --values platform/gitea-runner/values.yaml \ + --wait --timeout 10m + ``` + +6. Confirm the runner is online, then re-run the queued lint workflow. + +Do not deploy from an unmerged feature branch. Do not use `--set` for the token. + +The 1 GiB PVC preserves `.runner` identity. Docker image layers are ephemeral; +the Pod has a 20 GiB ephemeral-storage limit. Terraform apply jobs must use a +workflow concurrency group because runner capacity does not serialize access to +a shared state. diff --git a/platform/gitea-runner/external-secret.yaml b/platform/gitea-runner/external-secret.yaml new file mode 100644 index 0000000..2a914a0 --- /dev/null +++ b/platform/gitea-runner/external-secret.yaml @@ -0,0 +1,18 @@ +apiVersion: external-secrets.io/v1 +kind: ExternalSecret +metadata: + name: gitea-runner-token + namespace: gitea-actions +spec: + refreshInterval: 1h + secretStoreRef: + kind: ClusterSecretStore + name: openbao + target: + creationPolicy: Owner + name: gitea-runner-token + data: + - secretKey: token + remoteRef: + key: k8s/gitea-runner + property: token diff --git a/platform/gitea-runner/namespace.yaml b/platform/gitea-runner/namespace.yaml new file mode 100644 index 0000000..e31ad23 --- /dev/null +++ b/platform/gitea-runner/namespace.yaml @@ -0,0 +1,4 @@ +apiVersion: v1 +kind: Namespace +metadata: + name: gitea-actions diff --git a/platform/gitea-runner/values.yaml b/platform/gitea-runner/values.yaml new file mode 100644 index 0000000..5c1079a --- /dev/null +++ b/platform/gitea-runner/values.yaml @@ -0,0 +1,57 @@ +enabled: true + +giteaRootURL: http://gitea-http.gitea.svc.cluster.local:3000 +existingSecret: gitea-runner-token +existingSecretKey: token + +statefulset: + replicas: 1 + timezone: Etc/UTC + securityContext: + fsGroup: 1000 + # Chart 0.1.1 applies this block to both runner and DinD containers. + resources: + requests: + cpu: 250m + memory: 512Mi + ephemeral-storage: 2Gi + limits: + cpu: "4" + memory: 6Gi + ephemeral-storage: 20Gi + persistence: + size: 1Gi + + runner: + registry: docker.io + repository: gitea/runner + tag: 2.3.0 + pullPolicy: IfNotPresent + config: | + log: + level: info + runner: + file: .runner + capacity: 4 + timeout: 3h + shutdown_timeout: 3h + labels: + - self-hosted:docker://docker.gitea.com/runner-images:ubuntu-latest + cache: + enabled: false + container: + require_docker: true + docker_timeout: 300s + + dind: + rootless: true + uid: 1000 + registry: docker.io + repository: docker + tag: 29.7.1-dind-rootless + pullPolicy: IfNotPresent + extraEnvs: + - name: DOCKERD_ROOTLESS_ROOTLESSKIT_NET + value: slirp4netns + - name: DOCKERD_ROOTLESS_ROOTLESSKIT_MTU + value: "65520"