Compare commits
10
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
bb73a48695
|
||
|
|
99ef62bfa0
|
||
|
|
33cbd23861 | ||
|
|
0dfdbd46c8
|
||
|
|
21aabe162f
|
||
|
|
cbfcb10143 | ||
|
|
3ec528623a
|
||
|
|
11669ea81a | ||
|
|
83eb87bcec
|
||
|
|
75798f4c89 |
@@ -0,0 +1,19 @@
|
|||||||
|
name: dynamic Pod smoke test
|
||||||
|
|
||||||
|
on:
|
||||||
|
workflow_dispatch:
|
||||||
|
|
||||||
|
jobs:
|
||||||
|
pod-smoke:
|
||||||
|
name: pod-smoke
|
||||||
|
runs-on: [self-hosted, pod]
|
||||||
|
timeout-minutes: 5
|
||||||
|
steps:
|
||||||
|
- name: Verify disposable host environment
|
||||||
|
run: |
|
||||||
|
test -S /run/spire/agent-sockets/spire-agent.sock
|
||||||
|
/opt/spire/bin/spire-agent api fetch jwt \
|
||||||
|
-audience ci-smoke \
|
||||||
|
-socketPath /run/spire/agent-sockets/spire-agent.sock \
|
||||||
|
>/dev/null
|
||||||
|
test "$(id -u)" = 0
|
||||||
@@ -25,13 +25,19 @@ runs-on: [self-hosted, vm]
|
|||||||
- `guest-runner`:在 guest 中领取一次性 runner registration token,注册 ephemeral
|
- `guest-runner`:在 guest 中领取一次性 runner registration token,注册 ephemeral
|
||||||
runner,执行一个 job 后关机。
|
runner,执行一个 job 后关机。
|
||||||
- `pod-worker`:在 Kubernetes 中创建一次性 privileged Pod;Pod 内的 workflow 使用
|
- `pod-worker`:在 Kubernetes 中创建一次性 privileged Pod;Pod 内的 workflow 使用
|
||||||
host executor,Docker、BuildKit 和 kind 等工具由 pipeline 按需 setup。
|
host executor,Docker、BuildKit 和 kind 等工具由 pipeline 按需 setup。Runner 固定在
|
||||||
|
支持原生 job hooks 的 3.x 版本,在 workflow 第一步前等待实际任务对应的 SVID。
|
||||||
- `jwt-broker`:早期共享 Kubernetes runner 的过渡实验;目标架构不部署它,每个
|
- `jwt-broker`:早期共享 Kubernetes runner 的过渡实验;目标架构不部署它,每个
|
||||||
动态 Pod 或 VM 直接取得自己的 SPIFFE 身份。
|
动态 Pod 或 VM 直接取得自己的 SPIFFE 身份。
|
||||||
|
|
||||||
消息流使用一个 `WorkQueuePolicy` stream。相同 runner label 的所有 worker 共享同一
|
消息流使用一个 `WorkQueuePolicy` stream。相同 runner label 的所有 worker 共享同一
|
||||||
durable consumer;扩容只需要增加 worker 或提高单机 capacity。
|
durable consumer;扩容只需要增加 worker 或提高单机 capacity。
|
||||||
|
|
||||||
|
当前 webhook → NATS 流程是用于尽快验证 Pod/VM 生命周期的 bootstrap 实现,不是
|
||||||
|
长期调度接口。长期目标是让 controller 作为兼容 Gitea Runner 协议的调度器直接注册、
|
||||||
|
声明 labels、领取 task,并把已领取 task 交给 Pod/VM executor;路线与迁移边界见
|
||||||
|
[`docs/runner-protocol-roadmap.md`](docs/runner-protocol-roadmap.md)。
|
||||||
|
|
||||||
## 开发
|
## 开发
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
|
|||||||
@@ -7,9 +7,11 @@ COPY src ./src
|
|||||||
RUN python -m venv /venv && /venv/bin/pip install --no-cache-dir .
|
RUN python -m venv /venv && /venv/bin/pip install --no-cache-dir .
|
||||||
|
|
||||||
FROM python:3.12.11-alpine3.22
|
FROM python:3.12.11-alpine3.22
|
||||||
RUN addgroup -S -g 65532 runner && adduser -S -D -H -u 65532 -G runner runner
|
RUN addgroup -S -g 65532 runner \
|
||||||
|
&& adduser -S -D -H -u 65532 -G runner runner \
|
||||||
|
&& install -d -o 65532 -g 65532 /var/run/secrets/kubernetes.io/serviceaccount
|
||||||
COPY --from=build /venv /venv
|
COPY --from=build /venv /venv
|
||||||
COPY --from=spire /opt/spire/bin/spire-agent /opt/spire/bin/spire-agent
|
COPY --from=spire /opt/spire/bin/spire-agent /opt/spire/bin/spire-agent
|
||||||
USER 65532:65532
|
USER 65532:65532
|
||||||
EXPOSE 8787
|
EXPOSE 8787
|
||||||
ENTRYPOINT ["/venv/bin/gitea-microvm-controller"]
|
ENTRYPOINT ["/venv/bin/gitea-dynamic-runner-controller"]
|
||||||
|
|||||||
@@ -1,6 +1,6 @@
|
|||||||
FROM ghcr.io/spiffe/spire-agent:1.15.3@sha256:41b0dcd8b258a69db9e2768292a060766fb76fd866e4bc925849981ea1b825ff AS spire
|
FROM ghcr.io/spiffe/spire-agent:1.15.3@sha256:41b0dcd8b258a69db9e2768292a060766fb76fd866e4bc925849981ea1b825ff AS spire
|
||||||
|
|
||||||
FROM docker.io/gitea/runner:2@sha256:66d80966792e621c9761c47919644198d35fd1c297e9a01e69ed3c1ae37db0c7
|
FROM docker.io/gitea/runner:3.5.0@sha256:66b7da94dc7dcadb2e076bec6928221336a9a637196399281c4b766fe1288242
|
||||||
USER root
|
USER root
|
||||||
COPY --from=spire /opt/spire/bin/spire-agent /opt/spire/bin/spire-agent
|
COPY --from=spire /opt/spire/bin/spire-agent /opt/spire/bin/spire-agent
|
||||||
COPY config/runner.yaml /etc/gitea-runner/config.yaml
|
COPY config/runner.yaml /etc/gitea-runner/config.yaml
|
||||||
|
|||||||
@@ -0,0 +1,59 @@
|
|||||||
|
# Gitea Runner 协议调度器路线
|
||||||
|
|
||||||
|
## 目标
|
||||||
|
|
||||||
|
长期形态不依赖 `workflow_job` webhook 发现工作。controller 本身作为 Gitea Runner
|
||||||
|
协议客户端注册,并声明 `self-hosted`、`pod` 和 `vm` labels;它只在后端存在可用容量
|
||||||
|
时领取 task,然后将该 task 交给一个一次性 Pod 或 microVM 执行。
|
||||||
|
|
||||||
|
```text
|
||||||
|
Gitea RunnerService
|
||||||
|
│ Register / Declare / FetchTask
|
||||||
|
▼
|
||||||
|
dynamic-runner scheduler
|
||||||
|
│ 已领取的 task + lease
|
||||||
|
├── Pod executor
|
||||||
|
└── microVM executor
|
||||||
|
│ logs / state / result
|
||||||
|
└──────────────────────► Gitea
|
||||||
|
```
|
||||||
|
|
||||||
|
这与“收到 webhook 后临时注册另一个 act_runner”不同。`FetchTask` 已经完成任务分配,
|
||||||
|
不能再期待 Gitea 把同一个 task 分配给随后启动的 runner。协议调度器必须让 executor
|
||||||
|
执行已经领取的 task,并继续完成日志、状态、心跳、取消和最终结果上报。
|
||||||
|
|
||||||
|
## 设计约束
|
||||||
|
|
||||||
|
- 对 workflow 的接口保持 `[self-hosted, pod]` 和 `[self-hosted, vm]` 不变。
|
||||||
|
- scheduler 在没有对应 backend 容量时不领取 task,避免本地形成不可控积压。
|
||||||
|
- 每个 executor 只执行一个 task,完成后销毁。
|
||||||
|
- SPIFFE 身份从实际领取的 task 的 repository 和 job name 派生,不需要 queued 与
|
||||||
|
in-progress webhook 的二阶段关联。
|
||||||
|
- scheduler 的 runner registration credential 不进入 executor;executor 只得到执行
|
||||||
|
当前 task 所需的短期 lease/capability。
|
||||||
|
- task ACK、心跳和结果必须能够跨 scheduler 重启恢复;NATS 可以继续作为内部 handoff,
|
||||||
|
但不是 Gitea 任务事实来源。
|
||||||
|
- Pod 与 VM 共享 task/executor 协议,只有环境创建和销毁实现不同。
|
||||||
|
|
||||||
|
## 实现顺序
|
||||||
|
|
||||||
|
1. 固定当前 Gitea 版本所使用的 RunnerService protobuf 与 act_runner 版本,记录兼容
|
||||||
|
范围并建立协议契约测试。
|
||||||
|
2. 实现只注册、Declare labels 和容量感知 FetchTask 的 scheduler spike,暂不执行
|
||||||
|
task。
|
||||||
|
3. 从 act_runner 提取或复用 task 执行与日志上报能力,定义 scheduler 到 executor 的
|
||||||
|
单任务协议。
|
||||||
|
4. 首先接入 Pod executor,验证成功、失败、取消、超时和 scheduler 重启。
|
||||||
|
5. 接入 microVM executor,并复用同一 task 协议和身份派生逻辑。
|
||||||
|
6. 双轨运行并验证后,移除 webhook receiver、临时 runner 注册和 identity binding
|
||||||
|
subject。
|
||||||
|
|
||||||
|
## Bootstrap 实现的退出条件
|
||||||
|
|
||||||
|
只有同时满足以下条件才能删除 webhook 路径:
|
||||||
|
|
||||||
|
- scheduler 能通过 RunnerService 稳定领取并执行 Pod/VM task;
|
||||||
|
- Gitea UI 中的实时日志、取消、超时和结论与官方 runner 行为一致;
|
||||||
|
- scheduler 重启不会丢失已领取 task,也不会重复执行;
|
||||||
|
- SPIFFE 身份只来自实际领取 task;
|
||||||
|
- 同一套 workflow 无需修改 `runs-on` 即可从 bootstrap 迁移。
|
||||||
+7
-7
@@ -3,9 +3,9 @@ requires = ["setuptools>=75"]
|
|||||||
build-backend = "setuptools.build_meta"
|
build-backend = "setuptools.build_meta"
|
||||||
|
|
||||||
[project]
|
[project]
|
||||||
name = "gitea-microvm-runner"
|
name = "gitea-dynamic-runner"
|
||||||
version = "0.2.0"
|
version = "0.3.0"
|
||||||
description = "On-demand Cloud Hypervisor runners for Gitea Actions"
|
description = "On-demand Pod and microVM execution environments for Gitea Actions"
|
||||||
requires-python = ">=3.11"
|
requires-python = ">=3.11"
|
||||||
dependencies = ["aiohttp==3.12.15", "nats-py==2.11.0"]
|
dependencies = ["aiohttp==3.12.15", "nats-py==2.11.0"]
|
||||||
|
|
||||||
@@ -13,10 +13,10 @@ dependencies = ["aiohttp==3.12.15", "nats-py==2.11.0"]
|
|||||||
test = ["pytest==8.4.2", "pytest-asyncio==1.2.0"]
|
test = ["pytest==8.4.2", "pytest-asyncio==1.2.0"]
|
||||||
|
|
||||||
[project.scripts]
|
[project.scripts]
|
||||||
gitea-microvm-controller = "gitea_microvm_runner.controller:main"
|
gitea-dynamic-runner-controller = "gitea_dynamic_runner.controller:main"
|
||||||
gitea-microvm-worker = "gitea_microvm_runner.worker:cli"
|
gitea-dynamic-runner-pod-worker = "gitea_dynamic_runner.pod_worker:cli"
|
||||||
gitea-pod-worker = "gitea_microvm_runner.pod_worker:cli"
|
gitea-dynamic-runner-vm-worker = "gitea_dynamic_runner.worker:cli"
|
||||||
gitea-spire-jwt-broker = "gitea_microvm_runner.jwt_broker:main"
|
gitea-spire-jwt-broker = "gitea_dynamic_runner.jwt_broker:main"
|
||||||
|
|
||||||
[tool.pytest.ini_options]
|
[tool.pytest.ini_options]
|
||||||
asyncio_mode = "auto"
|
asyncio_mode = "auto"
|
||||||
|
|||||||
@@ -0,0 +1 @@
|
|||||||
|
"""Dynamic Pod and microVM execution environments for Gitea Actions."""
|
||||||
@@ -1 +0,0 @@
|
|||||||
"""Gitea microVM runner controller and worker."""
|
|
||||||
@@ -7,7 +7,7 @@ Requires=microvm-runner-network.service
|
|||||||
[Service]
|
[Service]
|
||||||
Type=simple
|
Type=simple
|
||||||
EnvironmentFile=-/etc/microvm-runner/worker.env
|
EnvironmentFile=-/etc/microvm-runner/worker.env
|
||||||
ExecStart=/opt/gitea-microvm-runner/venv/bin/gitea-microvm-worker
|
ExecStart=/opt/gitea-dynamic-runner/venv/bin/gitea-dynamic-runner-vm-worker
|
||||||
Restart=on-failure
|
Restart=on-failure
|
||||||
RestartSec=5s
|
RestartSec=5s
|
||||||
SupplementaryGroups=kvm
|
SupplementaryGroups=kvm
|
||||||
|
|||||||
@@ -4,8 +4,8 @@ import json
|
|||||||
|
|
||||||
import pytest
|
import pytest
|
||||||
|
|
||||||
from gitea_microvm_runner import controller
|
from gitea_dynamic_runner import controller
|
||||||
from gitea_microvm_runner.models import IdentityBinding, RunnerRequest
|
from gitea_dynamic_runner.models import IdentityBinding, RunnerRequest
|
||||||
|
|
||||||
|
|
||||||
def queued_job(**overrides):
|
def queued_job(**overrides):
|
||||||
|
|||||||
@@ -1,6 +1,6 @@
|
|||||||
import pytest
|
import pytest
|
||||||
|
|
||||||
from gitea_microvm_runner.jwt_broker import extract_svid
|
from gitea_dynamic_runner.jwt_broker import extract_svid
|
||||||
|
|
||||||
|
|
||||||
def test_extract_svid_from_spire_json():
|
def test_extract_svid_from_spire_json():
|
||||||
|
|||||||
@@ -1,7 +1,7 @@
|
|||||||
import pytest
|
import pytest
|
||||||
|
|
||||||
from gitea_microvm_runner.models import RunnerRequest
|
from gitea_dynamic_runner.models import RunnerRequest
|
||||||
from gitea_microvm_runner import pod_worker
|
from gitea_dynamic_runner import pod_worker
|
||||||
|
|
||||||
|
|
||||||
def request() -> RunnerRequest:
|
def request() -> RunnerRequest:
|
||||||
|
|||||||
Reference in New Issue
Block a user