归档：Kata / microVM runner 实验（2026-09-17 工作区快照）

从旧工作区 chore/recover-old-workspace 清理时保存，内容与 2026-09-17 stash@{0} 快照中的版本一致；未合并、未在 main 上使用，仅作参考，不开 PR。被 gitignore 的 tfstate 与凭据文件不在此分支，仍留在本地工作区。 Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-10-01 17:30:48 +00:00
parent 69d3476a6b
commit 55bb5be8b6
39 changed files with 1891 additions and 0 deletions
@@ -0,0 +1,72 @@
+# kata-lab
+
+`kata-lab` 是位于 Proxmox `pve2` 的可销毁验证环境，用来验证 nested KVM、k3s、
+Kata Containers，以及后续一 job 一 Kata Pod 的 Gitea Runner。它不是 dev/stg
+长期服务，也不承载持久数据。
+
+> 状态：VMID 147 已于 2026-09-14 销毁，Terraform state 为空。目录保留首次验证结果
+> 与可复现配置；后续验证转向跨 PVE 节点的轻量 LXC worker 方案。
+
+## Terraform bootstrap
+
+1. 将 Proxmox API token 写入被 Git 忽略且权限为 `0600` 的
+   `terraform/credentials.auto.tfvars`。
+2. 取得内部 CA，并仅为当前 Terraform 进程设置 Go TLS trust：
+
+   ```bash
+   curl -fsSL https://bao.ad.ddupan.top:8200/v1/pki/ca/pem \
+     -o /tmp/kata-lab-ddupan-ca.pem
+   export SSL_CERT_FILE=/tmp/kata-lab-ddupan-ca.pem
+   ```
+
+3. 初始化并审阅 plan：
+
+   ```bash
+   cd infrastructure/kata-lab/terraform
+   terraform init
+   terraform plan
+   ```
+
+Terraform 使用 `laptop:import/noble-server-cloudimg-amd64.qcow2`（40 GiB
+virtual size），在
+`pve-rg-hdd` 创建 VM root disk，并把 cloud-init snippet 放到 `laptop`
+datastore。VM root disk 设为 41 GiB，以容纳 PVE import 的块对齐增量。VM 使用
+DHCP、`cpu.type = host`、4 vCPU 和 4 GiB 内存；cloud-init
+安装 `qemu-guest-agent`、单节点 k3s，并记录 `/dev/kvm` 检查结果。
+
+## Kata Containers
+
+Ansible 使用 Kata Containers 4.1.0 官方 `kata-deploy` chart。配置明确选择 k3s，
+只安装 `qemu-runtime-rs`，不安装额外 snapshotter，并采用 `job` 模式，安装结束后
+不保留 privileged DaemonSet。默认 `RuntimeClass` 名为 `kata`。
+
+Terraform 输出的 DHCP 地址变化时，先更新 `ansible/inventory/hosts.yml`，然后运行：
+
+```bash
+cd infrastructure/kata-lab/ansible
+ansible-playbook kata.yml
+ansible-playbook verify.yml
+```
+
+`verify.yml` 首先创建一个 `runtimeClassName: kata` 的短生命周期 Pod，并确认 Pod 内
+看到的 Kata guest kernel 与 k3s node 的宿主 kernel 不同。随后它在另一个 Kata Pod
+内启动 privileged `dockerd` sidecar，让普通 Docker client 容器通过共享网络命名空间
+的 `127.0.0.1:2375` 运行一个嵌套容器。这对应后续 Gitea Runner 的目标形态：runner
+直接执行 job，只在需要 Docker API 时连接同 Pod 的 daemon，而不把 Docker executor
+作为 job 的外层。Kata 下共享 volume 中的 Unix socket 文件虽然双方可见，但不能跨
+容器连接，因此不能在这里沿用普通 Pod 常见的 `/var/run/docker.sock` 方案；loopback
+端口没有通过 Service 暴露到 Pod 外。
+验证 Pod 会保留以供排查；再次运行时会先替换它们。
+
+2026-09-13 的首次验证结果：Kata guest kernel 为 `6.18.35`，node host kernel 为
+`6.8.0-90-generic`；sidecar daemon 成功运行 `busybox:1.37` 嵌套容器。Docker 27.5.1
+在当前 Kata guest 中无法挂载 overlay2，自动回退到 `vfs`；功能验证通过，但正式用于
+CI 前需要解决或接受其镜像层复制与磁盘性能开销。
+
+不要提交 `credentials.auto.tfvars`、Terraform state 或 saved plan。销毁前确认目标
+仍然是 VMID 147 / `kata-lab`：
+
+```bash
+terraform plan -destroy
+terraform destroy
+```
@@ -0,0 +1,10 @@
+[defaults]
+inventory = inventory/hosts.yml
+host_key_checking = True
+interpreter_python = auto_silent
+retry_files_enabled = False
+local_tmp = /tmp/ansible-kata-lab-local
+remote_tmp = /tmp/ansible-kata-lab-remote
+
+[ssh_connection]
+pipelining = True
@@ -0,0 +1,8 @@
+---
+all:
+  hosts:
+    kata-lab:
+      ansible_host: 192.168.10.13
+      ansible_user: ansible
+  vars:
+    ansible_become: true
@@ -0,0 +1,88 @@
+---
+- name: Install Kata Containers in the disposable k3s lab
+  hosts: kata-lab
+  gather_facts: false
+  vars:
+    kata_version: "4.1.0"
+    kata_chart_url: >-
+      https://github.com/kata-containers/kata-containers/releases/download/{{ kata_version }}/kata-deploy-{{ kata_version }}.tgz
+  tasks:
+    - name: Check nested KVM is usable
+      ansible.builtin.stat:
+        path: /dev/kvm
+      register: kata_kvm
+
+    - name: Require nested KVM
+      ansible.builtin.assert:
+        that:
+          - kata_kvm.stat.exists
+          - kata_kvm.stat.ischr
+        fail_msg: /dev/kvm is unavailable; Kata QEMU cannot start efficiently
+
+    - name: Install Kata through the k3s Helm controller
+      ansible.builtin.copy:
+        dest: /var/lib/rancher/k3s/server/manifests/kata-deploy.yaml
+        owner: root
+        group: root
+        mode: "0644"
+        content: |
+          apiVersion: helm.cattle.io/v1
+          kind: HelmChart
+          metadata:
+            name: kata-deploy
+            namespace: kube-system
+          spec:
+            chart: {{ kata_chart_url }}
+            targetNamespace: kata-system
+            createNamespace: true
+            timeout: 15m
+            failurePolicy: abort
+            valuesContent: |-
+              deploymentMode: job
+              k8sDistribution: k3s
+              snapshotter:
+                setup: []
+              shims:
+                disableAll: true
+                qemu-runtime-rs:
+                  enabled: true
+              runtimeClasses:
+                enabled: true
+                createDefault: true
+              node-feature-discovery:
+                enabled: false
+
+    - name: Wait for the Helm installation job to appear
+      ansible.builtin.command:
+        cmd: k3s kubectl get job/helm-install-kata-deploy -n kube-system
+      register: kata_helm_job
+      retries: 60
+      delay: 2
+      until: kata_helm_job.rc == 0
+      changed_when: false
+
+    - name: Wait for the Helm installation job to finish
+      ansible.builtin.command:
+        cmd: >-
+          k3s kubectl wait --for=condition=complete
+          job/helm-install-kata-deploy -n kube-system --timeout=20m
+      changed_when: false
+
+    - name: Wait for Kata node installation jobs
+      ansible.builtin.shell:
+        cmd: |
+          set -euo pipefail
+          jobs=$(k3s kubectl get jobs -n kata-system -l app.kubernetes.io/instance=kata-deploy -o name)
+          test -n "${jobs}"
+          k3s kubectl wait --for=condition=complete -n kata-system $jobs --timeout=20m
+        executable: /bin/bash
+      changed_when: false
+
+    - name: Wait for the default Kata RuntimeClass
+      ansible.builtin.command:
+        cmd: k3s kubectl get runtimeclass kata
+      register: kata_runtime_class
+      retries: 30
+      delay: 2
+      until: kata_runtime_class.rc == 0
+      changed_when: false
@@ -0,0 +1,125 @@
+---
+- name: Verify Kata isolation with a disposable Pod
+  hosts: kata-lab
+  gather_facts: false
+  tasks:
+    - name: Remove an earlier smoke Pod
+      ansible.builtin.command:
+        cmd: k3s kubectl delete pod kata-smoke --ignore-not-found --wait=true
+      changed_when: false
+
+    - name: Create the Kata smoke Pod
+      ansible.builtin.shell:
+        cmd: |
+          set -o pipefail
+          k3s kubectl apply -f - <<'EOF'
+          apiVersion: v1
+          kind: Pod
+          metadata:
+            name: kata-smoke
+          spec:
+            runtimeClassName: kata
+            restartPolicy: Never
+            containers:
+              - name: smoke
+                image: docker.io/library/busybox:1.37
+                command:
+                  - sh
+                  - -c
+                  - 'printf "guest-kernel="; uname -r; test -c /dev/kvm && exit 1 || true'
+          EOF
+        executable: /bin/bash
+      changed_when: true
+
+    - name: Wait for the Kata Pod to finish
+      ansible.builtin.command:
+        cmd: k3s kubectl wait --for=jsonpath='{.status.phase}'=Succeeded pod/kata-smoke --timeout=10m
+      changed_when: false
+
+    - name: Read the guest kernel version
+      ansible.builtin.command:
+        cmd: k3s kubectl logs kata-smoke
+      register: kata_smoke_log
+      changed_when: false
+
+    - name: Read the host kernel version
+      ansible.builtin.command:
+        cmd: uname -r
+      register: kata_host_kernel
+      changed_when: false
+
+    - name: Require a distinct guest kernel
+      ansible.builtin.assert:
+        that:
+          - kata_smoke_log.stdout is match('^guest-kernel=.+')
+          - kata_smoke_log.stdout | regex_replace('^guest-kernel=', '') != kata_host_kernel.stdout
+        success_msg: >-
+          Kata guest {{ kata_smoke_log.stdout }} differs from host-kernel={{ kata_host_kernel.stdout }}
+
+    - name: Remove an earlier Docker sidecar smoke Pod
+      ansible.builtin.command:
+        cmd: k3s kubectl delete pod kata-docker-smoke --ignore-not-found --wait=true
+      changed_when: false
+
+    - name: Create a Kata Pod with a Docker daemon sidecar
+      ansible.builtin.shell:
+        cmd: |
+          set -o pipefail
+          k3s kubectl apply -f - <<'EOF'
+          apiVersion: v1
+          kind: Pod
+          metadata:
+            name: kata-docker-smoke
+          spec:
+            runtimeClassName: kata
+            restartPolicy: Never
+            volumes:
+              - name: docker-run
+                emptyDir: {}
+            containers:
+              - name: dockerd
+                image: docker.io/library/docker:27-dind
+                securityContext:
+                  privileged: true
+                env:
+                  - name: DOCKER_TLS_CERTDIR
+                    value: ""
+                volumeMounts:
+                  - name: docker-run
+                    mountPath: /var/run
+              - name: client
+                image: docker.io/library/docker:27-cli
+                env:
+                  - name: DOCKER_HOST
+                    value: tcp://127.0.0.1:2375
+                volumeMounts:
+                  - name: docker-run
+                    mountPath: /var/run
+                command:
+                  - sh
+                  - -c
+                  - |
+                    until docker info >/dev/null 2>&1; do sleep 1; done
+                    docker run --rm docker.io/library/busybox:1.37 echo nested-docker-ok
+          EOF
+        executable: /bin/bash
+      changed_when: true
+
+    - name: Wait for the Docker client to finish
+      ansible.builtin.command:
+        cmd: >-
+          k3s kubectl wait --for=jsonpath='{.status.containerStatuses[?(@.name=="client")].state.terminated.exitCode}'=0
+          pod/kata-docker-smoke --timeout=10m
+      changed_when: false
+
+    - name: Read the Docker client result
+      ansible.builtin.command:
+        cmd: k3s kubectl logs kata-docker-smoke -c client
+      register: kata_docker_smoke_log
+      changed_when: false
+
+    - name: Require Docker to run a nested container
+      ansible.builtin.assert:
+        that:
+          - "'nested-docker-ok' in kata_docker_smoke_log.stdout"
+        success_msg: Docker daemon sidecar completed a nested container inside the Kata VM
@@ -0,0 +1,24 @@
+# This file is maintained automatically by "terraform init".
+# Manual edits may be lost in future updates.
+
+provider "registry.terraform.io/bpg/proxmox" {
+  version     = "0.111.1"
+  constraints = "0.111.1"
+  hashes = [
+    "h1:ML2D3UUZTM99yrll/EBXj7wBYMb8xmQgomqFNybEoxY=",
+    "zh:18fb7c31a08dde6bffa1a4d4a211e604d6d17eec7092fd59331b3db3c6f3742c",
+    "zh:1cd60761538289d4dd2a1086b3ae62a7b0bdd4b1a2f824e9a44e243413168dba",
+    "zh:2eb76f6fc8299b6820ff678c8252332cc3366e226b5ae2e61748fd2449c1ed92",
+    "zh:45e6f7ebd0bf48911d37060359a4f359b5743b3092e985295733990e406d0416",
+    "zh:4aa8ba912eae37975d2e983394d173e595ca34fc76b5bf220b37d0e99d76e98c",
+    "zh:58e0789923103a77d502a0a9fc3eb920625e8eb935ec2d4ac0d006aebd1d186c",
+    "zh:6df8aa85fb8865915537e946c19b02538ad188018a629759c213c6f03730f642",
+    "zh:6ed47bc00d0913a1d0880618fa1376115e9edab6b4a658c081061a7f0e4ca360",
+    "zh:c5b10ff4f33df7e4c29e8f1127d49845b561b37b57517e844fb0954d7923d65e",
+    "zh:d016510e14b738499f0db9d9b3aafe82fc6877fb4ab4e9f831fb68a8d70a1385",
+    "zh:d941f394069bbf24351b363da1c64383f487067aaee0a84f9b96476d4912e212",
+    "zh:ddf271dbc2632ae8ffa8de3972f243ee47d260cb2ac90aa784f2746d98e21a0f",
+    "zh:ed0caa3501c42f611b7e9622c9b1df69fd85dc25a3cd88d3076381829688cd62",
+    "zh:f26e0763dbe6a6b2195c94b44696f2110f7f55433dc142839be16b9697fa5597",
+  ]
+}
@@ -0,0 +1,34 @@
+#cloud-config
+hostname: ${hostname}
+manage_etc_hosts: true
+timezone: Etc/UTC
+
+users:
+  - default
+  - name: ansible
+    groups: [adm, sudo]
+    shell: /bin/bash
+    lock_passwd: true
+    sudo: ALL=(ALL) NOPASSWD:ALL
+    ssh_authorized_keys:
+      - ${ssh_public_key}
+
+package_update: true
+package_upgrade: false
+packages:
+  - ca-certificates
+  - curl
+  - jq
+  - qemu-guest-agent
+
+runcmd:
+  - [systemctl, enable, --now, qemu-guest-agent]
+  - [modprobe, kvm_amd]
+  - [sh, -c, 'test -c /dev/kvm && echo available > /var/lib/cloud/nested-kvm.status || echo unavailable > /var/lib/cloud/nested-kvm.status']
+  - [sh, -c, 'curl -sfL https://get.k3s.io -o /tmp/install-k3s.sh']
+  - [chmod, '0755', /tmp/install-k3s.sh]
+  - [sh, -c, 'INSTALL_K3S_VERSION="${k3s_version}" INSTALL_K3S_EXEC="server --disable=traefik --write-kubeconfig-mode=0644" /tmp/install-k3s.sh']
+  - [sh, -c, 'kubectl wait --for=condition=Ready node/${hostname} --timeout=180s']
+  - [touch, /var/lib/cloud/kata-lab-bootstrap.done]
+
+final_message: "kata-lab bootstrap completed after $UPTIME seconds"
@@ -0,0 +1,3 @@
+# Copy this file to credentials.auto.tfvars and replace the placeholder.
+# Format: user@realm!token-id=token-secret
+proxmox_api_token = "REPLACE_ME"
@@ -0,0 +1,86 @@
+locals {
+  ssh_public_key = trimspace(file(pathexpand(var.ssh_public_key_file)))
+  cloud_init = templatefile("${path.module}/cloud-init.yaml.tftpl", {
+    hostname       = var.vm_name
+    ssh_public_key = local.ssh_public_key
+    k3s_version    = var.k3s_version
+  })
+}
+
+data "proxmox_file" "ubuntu_noble" {
+  content_type = "import"
+  datastore_id = var.snippet_datastore_id
+  node_name    = var.node_name
+  file_name    = "noble-server-cloudimg-amd64.qcow2"
+}
+
+resource "proxmox_virtual_environment_file" "cloud_init" {
+  content_type = "snippets"
+  datastore_id = var.snippet_datastore_id
+  node_name    = var.node_name
+
+  source_raw {
+    data      = local.cloud_init
+    file_name = "${var.vm_name}-cloud-init.yaml"
+  }
+}
+
+resource "proxmox_virtual_environment_vm" "kata_lab" {
+  name        = var.vm_name
+  description = "Disposable single-node k3s and Kata Containers validation environment."
+  tags        = ["terraform", "disposable", "kata"]
+
+  node_name = var.node_name
+  vm_id     = var.vm_id
+
+  started         = true
+  on_boot         = false
+  stop_on_destroy = true
+
+  agent {
+    enabled = true
+    timeout = "15m"
+  }
+
+  cpu {
+    cores = var.vm_cores
+    type  = "host"
+  }
+
+  memory {
+    dedicated = var.vm_memory_mb
+  }
+
+  operating_system {
+    type = "l26"
+  }
+
+  scsi_hardware = "virtio-scsi-single"
+
+  disk {
+    datastore_id = var.disk_datastore_id
+    import_from  = data.proxmox_file.ubuntu_noble.id
+    interface    = "scsi0"
+    iothread     = true
+    discard      = "on"
+    size         = var.vm_disk_gb
+  }
+
+  initialization {
+    datastore_id      = var.disk_datastore_id
+    user_data_file_id = proxmox_virtual_environment_file.cloud_init.id
+
+    ip_config {
+      ipv4 {
+        address = "dhcp"
+      }
+    }
+  }
+
+  network_device {
+    bridge = var.network_bridge
+    model  = "virtio"
+  }
+
+  serial_device {}
+}
@@ -0,0 +1,12 @@
+output "vm_id" {
+  value = proxmox_virtual_environment_vm.kata_lab.vm_id
+}
+
+output "vm_ipv4_addresses" {
+  description = "QEMU guest agent 报告的地址；忽略 loopback 和 CNI 地址后再用于 SSH。"
+  value       = proxmox_virtual_environment_vm.kata_lab.ipv4_addresses
+}
+
+output "ssh_user" {
+  value = "ansible"
+}
@@ -0,0 +1,17 @@
+provider "proxmox" {
+  endpoint  = var.proxmox_endpoint
+  api_token = var.proxmox_api_token
+
+  # Snippet uploads use SSH. The provider deliberately does not read
+  # ~/.ssh/config, so map the PVE node explicitly and use the current agent.
+  ssh {
+    agent       = false
+    username    = "root"
+    private_key = file(pathexpand(var.proxmox_ssh_private_key_file))
+
+    node {
+      name    = "pve2"
+      address = "192.168.10.7"
+    }
+  }
+}
@@ -0,0 +1,83 @@
+variable "proxmox_endpoint" {
+  description = "Proxmox VE API endpoint，不包含 /api2/json。"
+  type        = string
+  default     = "https://pve1.ad.ddupan.top:8006/"
+}
+
+variable "proxmox_api_token" {
+  description = "Proxmox API token，格式为 user@realm!token-id=secret。"
+  type        = string
+  sensitive   = true
+}
+
+variable "node_name" {
+  description = "承载 disposable kata-lab VM 的 PVE 节点。"
+  type        = string
+  default     = "pve2"
+}
+
+variable "vm_id" {
+  description = "kata-lab VMID。"
+  type        = number
+  default     = 147
+}
+
+variable "vm_name" {
+  description = "kata-lab VM 名称和 guest hostname。"
+  type        = string
+  default     = "kata-lab"
+}
+
+variable "vm_memory_mb" {
+  description = "VM 固定内存；pve2 只有约 7.4 GiB 可见内存。"
+  type        = number
+  default     = 4096
+}
+
+variable "vm_cores" {
+  description = "VM vCPU 数量。"
+  type        = number
+  default     = 4
+}
+
+variable "vm_disk_gb" {
+  description = "VM root disk 大小。"
+  type        = number
+  default     = 41
+}
+
+variable "disk_datastore_id" {
+  description = "VM root disk 所在 datastore。"
+  type        = string
+  default     = "pve-rg-hdd"
+}
+
+variable "snippet_datastore_id" {
+  description = "启用 snippets 的共享 datastore。"
+  type        = string
+  default     = "laptop"
+}
+
+variable "network_bridge" {
+  description = "连接 kata-lab VM 的 PVE bridge。"
+  type        = string
+  default     = "vmbr0"
+}
+
+variable "ssh_public_key_file" {
+  description = "注入 cloud-init 用户的 SSH 公钥路径。"
+  type        = string
+  default     = "~/.ssh/id_ed25519.pub"
+}
+
+variable "proxmox_ssh_private_key_file" {
+  description = "provider 上传 snippet 时登录 PVE 节点使用的私钥路径。"
+  type        = string
+  default     = "~/.ssh/id_ed25519"
+}
+
+variable "k3s_version" {
+  description = "可选的固定 k3s 版本；空值使用 stable channel。"
+  type        = string
+  default     = ""
+}
@@ -0,0 +1,10 @@
+terraform {
+  required_version = ">= 1.10.0"
+
+  required_providers {
+    proxmox = {
+      source  = "bpg/proxmox"
+      version = "0.111.1"
+    }
+  }
+}