> For AI agents: the complete documentation index is available at /llms.txt, the full documentation bundle is available at /llms-full.txt.
# CI runner roles
The ARM64 CI fleet uses three disjoint scheduling roles. The desired allocation
is recorded in `devops/ci-runner-roles.json`.
Control workflows select their role alongside `self-hosted`, `Linux`, `ARM64` and
`pharos-ubuntu-24.04`. General and infrastructure workflows select portable pools
described below, preserving the ARM role allocation.
Control workers deliberately omit `pharos-ci`, fencing them from older workflows
that selected the whole CI fleet.
| Role label | Desired workers | Purpose |
| -------------------------- | -------------------------------- | ------------------------------------------------------------- |
| `pharos-ci-control` | `builder-m3-04` | Affected-check selection and final CI gates |
| `pharos-ci-general` | `builder-m3-03`, `k3s-m3-01` | Verification, compatibility and other general ARM64 jobs |
| `pharos-ci-infrastructure` | `builder-m3-01`, `builder-m3-02` | Real persistence, broker, assembled images and browser checks |
This keeps ordinary verification from consuming the infrastructure workers
and keeps short control jobs from joining the test queue. Each runner still
executes one Actions job at a time. Role labels affect subsequent assignments;
existing jobs finish normally. Older workflow revisions using broad labels can
still consume general and infrastructure workers while those runs drain.
## ARM preference and AMD64 fallback
Verification and compatibility use `pharos-ci-portable-general`; infrastructure
uses `pharos-ci-portable-infrastructure`. Both require `self-hosted`, `Linux` and
`pharos-ci`. ARM workers join the portable pool corresponding to their existing
role. AMD64 workers with `pharos-ci` join both pools and must have the general and
infrastructure prerequisites below, including at least 12 GiB and 4 CPUs for
infrastructure. Their Linux distribution can differ from the ARM workers' Ubuntu
image. Each worker runs one job at a time.
Before enabling these workflow selectors, preview and apply the pool labels:
```sh
python3 scripts/ci_runner_pools.py
python3 scripts/ci_runner_pools.py --apply
```
Repeat after registering workers or changing their roles. The command preserves
other labels and current jobs, excludes control workers, and reads back each
change. It excludes workers marked `maintenance` or `maintenance-*`, even if a
legacy `pharos-ci` label remains. It changes scheduling labels only; provisioning
owns capacity and tools.
The original ARM role-management command continues to manage only that fleet.
The native Metallo host inventory and pinned provisioning requirements live in
`devops/github-runners/native`.
That workflow registers new workers in maintenance, qualifies the service
account's environment and fixtures, then enables scheduling. Update that inventory
when adding or replacing hosts; adding GitHub labels alone does not qualify one.
The scope job uses its ordinary read-only Actions token to inspect current job
activity. A configured ARM worker that finished a job within 15 minutes, has no
observed current assignment and has no competing queued work for its role
contributes one preferred slot. Each slot is used at most once within the current
workflow's selection. The selector spends at most 15 seconds reading up to 12
active runs plus five recent completed runs. Missing permissions, incomplete
responses, larger queues and unavailable APIs all select the shared pool.
No runner-administration token enters the workflow.
With a preferred slot, the job adds `ARM64` to its portable pool selector.
Otherwise either architecture can take the job. This is a best-effort observation,
not a reservation: another workflow can claim a slot first, and a worker can go
offline after selection. An ARM-selected job then waits for an ARM worker; the
selector does not migrate jobs already queued. A long-idle ARM worker may also
lack recent evidence and be considered through the shared pool.
The core success-key namespace is `pharos-ci-v3-…-linux-…`. A successful portable
check on either architecture can satisfy identical source inputs. The record
also includes its actual runner OS, architecture and name for diagnostics.
These records contain test evidence; compiled outputs, Python environments,
native packages and Docker layers remain in their workers' separate caches.
Desktop and release packaging retain their explicit platform selectors.
## Inspect and apply
Use Python 3 and authenticated `gh` with repository runner-management access.
Existing SSH aliases `builder-m3` and `k3s-m3` reach the owning `vasilis` account.
Its rootless Podman containers are `pharos-actions-NN`; its user systemd units
have the same name with `.service`. The host Docker daemon and host toolchain
are separate from the worker environment.
```sh
python3 scripts/ci_runners.py inventory
python3 scripts/ci_runners.py inventory --probe
python3 scripts/ci_runners.py validate --plan devops/ci-runner-roles.json
python3 scripts/ci_runners.py apply --plan devops/ci-runner-roles.json
```
Inventory reads every page of GitHub registrations. Optional probes execute
read-only commands as the container's `runner` user and report registration
identity, Ubuntu/ARM64, cgroup limits, current usage, host capacity and required
tools. No fixture containers, builds, dependency installations or service
restarts are performed. Credentials and raw failed-command output are withheld.
Validation requires every active eligible worker and every existing role-labelled
worker to be represented exactly once. It checks online status and base labels,
the container's matching registration, a writable tool cache and these minimum
limits:
| Role | Memory limit | CPU limit | Prerequisites |
| -------------- | ------------ | --------- | ----------------------------------------------------------------------------------------------- |
| Control | 2 GiB | 1 | Git, gh, Python, Node, tar and gzip |
| General | 6 GiB | 2 | Control tools, uv, kubectl, Helm and OpenSSL |
| Infrastructure | 12 GiB | 4 | General tools, Docker daemon, Compose, Buildx and non-interactive sudo for browser dependencies |
These are scheduling requirements for the current lanes. Workflows install the
pinned application toolchains. A prerequisite probe does not qualify actual
PostgreSQL, broker, Docker builds or Chromium behaviour; run the appropriate
[infrastructure lanes](/reference/TESTING.md#infrastructure-checks-on-the-dev-box) after
changing a worker image or resource profile.
Validation also checks that the selected workers' combined limits fit each host's
CPU count and leave at least 2 GiB outside worker memory limits. Current available
memory is diagnostic because busy jobs and other host services change it. The
check cannot reserve capacity for unrelated host workloads.
Apply repeats resource probes and rechecks registrations and labels before changing
anything. Control assignment removes `pharos-ci` before adding the control role.
Reassignment from control restores `pharos-ci` before removing the control role,
after validating the destination's resource requirements. Obsolete role labels
are removed individually; platform, host and operator labels are preserved.
Each result is verified, including the control worker's absence of `pharos-ci`.
A partial API failure reports completed workers; rerun validation and apply to
reconcile the recorded plan. Busy workers remain running throughout. Inventory
and probes continue to include isolated control workers.
## Capacity and maintenance
The 3 November 2026 inventory before the capacity reallocation found:
| Worker | Existing memory / CPUs | Planned memory / CPUs |
| --------------- | ---------------------- | --------------------- |
| `builder-m3-01` | 12 GiB / 4 | 12 GiB / 4 |
| `builder-m3-02` | 6 GiB / 4 | 12 GiB / 4 |
| `builder-m3-03` | 6 GiB / 4 | 6 GiB / 4 |
| `builder-m3-04` | 6 GiB / 2 | 2 GiB / 1 |
| `k3s-m3-01` | 6 GiB / 4 | 6 GiB / 4 |
Each host has about 34.8 GiB RAM and 14 CPUs. Builder limits total 30 GiB and
14 CPUs before reallocation, and 32 GiB and 13 CPUs afterwards. The final profile
leaves about 2.8 GiB outside the builder worker limits. The two remaining k3s
registrations are offline and lack `pharos-ci`; the plan leaves them out.
Re-run inventory for current status and capacity.
The committed plan describes the target allocation. The second infrastructure
worker and reduced control profile require the staged rollout below; recording
the plan does not apply resource changes. Validation rejects `builder-m3-02`
while it retains its 6 GiB limit.
## Staged capacity rollout
The inspected builder host uses rootless Podman 5.8.9, systemd delegation,
cgroup v2 and crun. Its worker units use `podman run --rm`, so both the live
container and the persistent user unit require matching resource limits.
Podman updates individual resource controls; the resource, unit and GitHub
changes are separate operations. Stop and inspect after any failed step.
1. Publish the workflow selectors that allow an isolated control worker.
Resolve older runs that still require its old selector. Use a temporary,
complete plan with the existing allocation (`builder-m3-02` remains general)
to validate and apply control isolation. This removes only the control
worker's generic eligibility while its current job finishes.
2. Wait for GitHub to report `builder-m3-04` idle and confirm its container has
no `Runner.Worker` process. Read `memory.current`, `memory.stat`,
`memory.events`, `memory.max`, `memory.swap.max` and `cpu.max`. Confirm that
idle anonymous and kernel memory leave room for the control task within
2 GiB. File cache is reclaimable; a historical peak from full verification
does not measure the control task's requirement.
3. Back up `~/.config/systemd/user/pharos-actions-04.service` in a private
directory with mode `0700` and backup mode `0600`. Change only its resource
arguments from `--cpus=2 --memory=6g --memory-swap=6g` to
`--cpus=1 --memory=2g --memory-swap=2g`. Verify the expected old arguments
before an atomic file replacement, preserve the unit's permissions and run
`systemctl --user daemon-reload`. This leaves the running container intact
and makes an unexpected future recreation use the smaller allocation.
4. While control remains idle, apply its live reductions on `builder-m3`:
```sh
podman update --cpus=1 pharos-actions-04
podman update --memory=2g --memory-swap=2g pharos-actions-04
```
Read back `cpu.max` as `100000 100000`, `memory.max` as `2147483648`,
and `memory.swap.max` as `0`. Verify Podman's resource configuration and
persistent unit arguments agree. Check for any new OOM events before proceeding.
5. Back up and edit `pharos-actions-02.service` using the same precautions.
Change only `--memory=6g --memory-swap=6g` to
`--memory=12g --memory-swap=12g`; its CPU limit remains 4. Reload the units,
then increase the live container's limit:
```sh
podman update --memory=12g --memory-swap=12g pharos-actions-02
```
This increase preserves its current job. Verify `memory.max` is
`12884901888`, `memory.swap.max` remains `0` and `cpu.max` remains
`400000 100000`. Equal memory and combined memory-plus-swap arguments
preserve the existing zero-swap allocation.
6. Once both resource changes are verified, validate and apply the committed
plan. Role reassignment preserves `builder-m3-02`'s current job and affects
subsequent assignments. Keep `builder-m3-01`'s host-specific and generic
labels so previously queued infrastructure jobs remain eligible there.
7. Qualify a new scope and final gate on the 2 GiB control worker, comparing
its OOM counters before and after. Run the full infrastructure lane on a
worker with the new profile. Record actual job results and current limits.
Resource probes alone do not establish fixture qualification.
Runner provisioning and image maintenance are developed separately in PR #981.
That historical profile describes three 6 GiB workers per host; the live fleet
has a fourth builder worker and a larger `builder-m3-01`. Reconcile those profiles
before provisioning or rebuilding. Role assignment here changes only GitHub
labels and never registers, restarts, replaces or resizes a worker.
## Warm tools and dependencies
The active ARM workers already have persistent, separate Podman volumes for
`/home/runner`, `/opt/hostedtoolcache` and `/var/lib/docker`. Native Metallo
workers retain separate homes and tool caches on the hosts in the
native inventory, with one Docker daemon per host. The GitHub runner on
`metallo-vasilis-2` has been retired.
These stores serve every checkout on the same worker. Runner recreation must
preserve the volumes; the provisioning work in PR #981 owns their creation.
CI uses the following retained stores:
| Content | Location and validation |
| -------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Node, Go, uv and kubectl | Actions tool cache, using the workflow's pinned setup actions and versions |
| pnpm packages | `$RUNNER_TOOL_CACHE/pharos/pnpm/$RUNNER_NAME`, with pnpm's versioned store beneath it; frozen installs check the checked-out lockfile |
| Python packages | Main CI: `$RUNNER_TOOL_CACHE/pharos-uv/$RUNNER_NAME`; authentication: `$RUNNER_TOOL_CACHE/pharos/uv/$RUNNER_NAME` |
| Go modules and compiled packages | Native `go env GOMODCACHE GOCACHE`; each service retains its own `go.mod` and `go.sum` |
| Chromium | Playwright's native user cache, with the version selected by the root package lock |
| Ansible collections | `~/.cache/pharos/foundation-collections` and `~/.cache/pharos/kubespray-2.32.0/collections`; the existing helpers check pinned versions and Kubespray source provenance |
| Fixture images and Docker build layers | Each worker's Docker store; normal builds still use the current checkout and Docker's input checks |
Run the existing **CI** workflow with **warm_runners** enabled to populate these
stores after provisioning or dependency updates:
```sh
gh workflow run ci.yml --ref chore/ci-improvements \
-f warm_runners=true -f publish_dev_images=false
```
After merge, select the maintained default branch instead. This mode calls
`.github/workflows/ci-warm.yml` and visits the four ARM test workers and five
native Metallo workers by name. Tests compare the native matrix with the maintained
host inventory. The control, retired and VPN-maintenance workers are excluded. At most two warming jobs
run concurrently, and each waits in its worker's normal Actions queue. Existing
jobs finish normally. Repeated warming requests never cancel active warming.
General workers fetch both pnpm workspaces, all verification Python environments,
Go dependencies, generator dependencies and pinned Ansible collections.
Infrastructure workers fetch their application dependencies and missing fixture
and base images. Authentication workers fetch both frontend workspaces, their
Python and Go dependencies, Chromium and authentication fixture images.
Node 22 and 24 are prepared where the worker's lanes need both versions.
Package prefetching uses disposable manifests and lockfiles, disables pnpm
install scripts, skips local Python packages, and verifies that source inputs and
locks remain unchanged. Locked third-party Python dependencies are built into
cached wheels when no compatible wheel is available. It records the actual cache
paths and timings. pnpm's store is resolved from each normal workspace and bound
explicitly into temporary staging, so a separate `/tmp` filesystem cannot select
an unused store. Warming and test jobs select the same persistent store before
pnpm setup; the setup action replaces its own `~/setup-pnpm` directory on each job.
Frozen installs remain in normal jobs and fetch any newly required dependencies.
Browser warming preserves other installed browser versions. Image warming
derives references from the checked-out fixture configuration and Dockerfiles,
downloads missing native images serially, and verifies their presence.
It never publishes application images or starts fixture services.
Warming has separate run concurrency and a separate check name. It produces no
successful test records and cannot satisfy `ci-gate`. A successful maintenance
run establishes that the downloads and tools are ready; source and real fixture
qualification remain separate.
Package and collection warming require at least 5 GiB free space; image warming
requires 10 GiB and rechecks space between pulls. These commands leave cache
retention and cleanup to maintenance. Keep each worker's Docker daemon and
package stores private to its runner account; sharing mutable working trees or
a Docker data directory between workers would couple otherwise isolated jobs.
Offline checks:
```sh
python3 -m unittest discover -s scripts -p test_ci_runners.py
```
Page source
Source snapshot · Revision `f928834c8`.