#CI runner roles

The ARM64 CI fleet uses three disjoint scheduling roles. The desired allocation is recorded in devops/ci-runner-roles.json. Control workflows select their role alongside self-hosted, Linux, ARM64 and pharos-ubuntu-24.04. General and infrastructure workflows select portable pools described below, preserving the ARM role allocation. Control workers deliberately omit pharos-ci, fencing them from older workflows that selected the whole CI fleet.

This keeps ordinary verification from consuming the infrastructure workers and keeps short control jobs from joining the test queue. Each runner still executes one Actions job at a time. Role labels affect subsequent assignments; existing jobs finish normally. Older workflow revisions using broad labels can still consume general and infrastructure workers while those runs drain.

#ARM preference and AMD64 fallback

Verification and compatibility use pharos-ci-portable-general; infrastructure uses pharos-ci-portable-infrastructure. Both require self-hosted, Linux and pharos-ci. ARM workers join the portable pool corresponding to their existing role. AMD64 workers with pharos-ci join both pools and must have the general and infrastructure prerequisites below, including at least 12 GiB and 4 CPUs for infrastructure. Their Linux distribution can differ from the ARM workers' Ubuntu image. Each worker runs one job at a time.

Before enabling these workflow selectors, preview and apply the pool labels:

python3 scripts/ci_runner_pools.py
python3 scripts/ci_runner_pools.py --apply

Repeat after registering workers or changing their roles. The command preserves other labels and current jobs, excludes control workers, and reads back each change. It excludes workers marked maintenance or maintenance-*, even if a legacy pharos-ci label remains. It changes scheduling labels only; provisioning owns capacity and tools. The original ARM role-management command continues to manage only that fleet.

The native Metallo host inventory and pinned provisioning requirements live in devops/github-runners/native. That workflow registers new workers in maintenance, qualifies the service account's environment and fixtures, then enables scheduling. Update that inventory when adding or replacing hosts; adding GitHub labels alone does not qualify one.

The scope job uses its ordinary read-only Actions token to inspect current job activity. A configured ARM worker that finished a job within 15 minutes, has no observed current assignment and has no competing queued work for its role contributes one preferred slot. Each slot is used at most once within the current workflow's selection. The selector spends at most 15 seconds reading up to 12 active runs plus five recent completed runs. Missing permissions, incomplete responses, larger queues and unavailable APIs all select the shared pool. No runner-administration token enters the workflow.

With a preferred slot, the job adds ARM64 to its portable pool selector. Otherwise either architecture can take the job. This is a best-effort observation, not a reservation: another workflow can claim a slot first, and a worker can go offline after selection. An ARM-selected job then waits for an ARM worker; the selector does not migrate jobs already queued. A long-idle ARM worker may also lack recent evidence and be considered through the shared pool.

The core success-key namespace is pharos-ci-v3-…-linux-…. A successful portable check on either architecture can satisfy identical source inputs. The record also includes its actual runner OS, architecture and name for diagnostics. These records contain test evidence; compiled outputs, Python environments, native packages and Docker layers remain in their workers' separate caches. Desktop and release packaging retain their explicit platform selectors.

#Inspect and apply

Use Python 3 and authenticated gh with repository runner-management access. Existing SSH aliases builder-m3 and k3s-m3 reach the owning vasilis account. Its rootless Podman containers are pharos-actions-NN; its user systemd units have the same name with .service. The host Docker daemon and host toolchain are separate from the worker environment.

python3 scripts/ci_runners.py inventory
python3 scripts/ci_runners.py inventory --probe
python3 scripts/ci_runners.py validate --plan devops/ci-runner-roles.json
python3 scripts/ci_runners.py apply --plan devops/ci-runner-roles.json

Inventory reads every page of GitHub registrations. Optional probes execute read-only commands as the container's runner user and report registration identity, Ubuntu/ARM64, cgroup limits, current usage, host capacity and required tools. No fixture containers, builds, dependency installations or service restarts are performed. Credentials and raw failed-command output are withheld.

Validation requires every active eligible worker and every existing role-labelled worker to be represented exactly once. It checks online status and base labels, the container's matching registration, a writable tool cache and these minimum limits:

These are scheduling requirements for the current lanes. Workflows install the pinned application toolchains. A prerequisite probe does not qualify actual PostgreSQL, broker, Docker builds or Chromium behaviour; run the appropriate infrastructure lanes after changing a worker image or resource profile.

Validation also checks that the selected workers' combined limits fit each host's CPU count and leave at least 2 GiB outside worker memory limits. Current available memory is diagnostic because busy jobs and other host services change it. The check cannot reserve capacity for unrelated host workloads.

Apply repeats resource probes and rechecks registrations and labels before changing anything. Control assignment removes pharos-ci before adding the control role. Reassignment from control restores pharos-ci before removing the control role, after validating the destination's resource requirements. Obsolete role labels are removed individually; platform, host and operator labels are preserved. Each result is verified, including the control worker's absence of pharos-ci. A partial API failure reports completed workers; rerun validation and apply to reconcile the recorded plan. Busy workers remain running throughout. Inventory and probes continue to include isolated control workers.

#Capacity and maintenance

The 3 November 2026 inventory before the capacity reallocation found:

Each host has about 34.8 GiB RAM and 14 CPUs. Builder limits total 30 GiB and 14 CPUs before reallocation, and 32 GiB and 13 CPUs afterwards. The final profile leaves about 2.8 GiB outside the builder worker limits. The two remaining k3s registrations are offline and lack pharos-ci; the plan leaves them out. Re-run inventory for current status and capacity.

The committed plan describes the target allocation. The second infrastructure worker and reduced control profile require the staged rollout below; recording the plan does not apply resource changes. Validation rejects builder-m3-02 while it retains its 6 GiB limit.

#Staged capacity rollout

The inspected builder host uses rootless Podman 5.8.9, systemd delegation, cgroup v2 and crun. Its worker units use podman run --rm, so both the live container and the persistent user unit require matching resource limits. Podman updates individual resource controls; the resource, unit and GitHub changes are separate operations. Stop and inspect after any failed step.

1. Publish the workflow selectors that allow an isolated control worker. Resolve older runs that still require its old selector. Use a temporary, complete plan with the existing allocation (builder-m3-02 remains general) to validate and apply control isolation. This removes only the control worker's generic eligibility while its current job finishes.

2. Wait for GitHub to report builder-m3-04 idle and confirm its container has no Runner.Worker process. Read memory.current, memory.stat, memory.events, memory.max, memory.swap.max and cpu.max. Confirm that idle anonymous and kernel memory leave room for the control task within 2 GiB. File cache is reclaimable; a historical peak from full verification does not measure the control task's requirement.

3. Back up ~/.config/systemd/user/pharos-actions-04.service in a private directory with mode 0700 and backup mode 0600. Change only its resource arguments from --cpus=2 --memory=6g --memory-swap=6g to --cpus=1 --memory=2g --memory-swap=2g. Verify the expected old arguments before an atomic file replacement, preserve the unit's permissions and run systemctl --user daemon-reload. This leaves the running container intact and makes an unexpected future recreation use the smaller allocation.

4. While control remains idle, apply its live reductions on builder-m3:

   podman update --cpus=1 pharos-actions-04
   podman update --memory=2g --memory-swap=2g pharos-actions-04

   Read back cpu.max as 100000 100000, memory.max as 2147483648, and memory.swap.max as 0. Verify Podman's resource configuration and persistent unit arguments agree. Check for any new OOM events before proceeding.

5. Back up and edit pharos-actions-02.service using the same precautions. Change only --memory=6g --memory-swap=6g to --memory=12g --memory-swap=12g; its CPU limit remains 4. Reload the units, then increase the live container's limit:

   podman update --memory=12g --memory-swap=12g pharos-actions-02

   This increase preserves its current job. Verify memory.max is 12884901888, memory.swap.max remains 0 and cpu.max remains 400000 100000. Equal memory and combined memory-plus-swap arguments preserve the existing zero-swap allocation.

6. Once both resource changes are verified, validate and apply the committed plan. Role reassignment preserves builder-m3-02's current job and affects subsequent assignments. Keep builder-m3-01's host-specific and generic labels so previously queued infrastructure jobs remain eligible there.

7. Qualify a new scope and final gate on the 2 GiB control worker, comparing its OOM counters before and after. Run the full infrastructure lane on a worker with the new profile. Record actual job results and current limits. Resource probes alone do not establish fixture qualification.

Runner provisioning and image maintenance are developed separately in PR #981. That historical profile describes three 6 GiB workers per host; the live fleet has a fourth builder worker and a larger builder-m3-01. Reconcile those profiles before provisioning or rebuilding. Role assignment here changes only GitHub labels and never registers, restarts, replaces or resizes a worker.

#Warm tools and dependencies

The active ARM workers already have persistent, separate Podman volumes for /home/runner, /opt/hostedtoolcache and /var/lib/docker. Native Metallo workers retain separate homes and tool caches on the hosts in the native inventory, with one Docker daemon per host. The GitHub runner on metallo-vasilis-2 has been retired. These stores serve every checkout on the same worker. Runner recreation must preserve the volumes; the provisioning work in PR #981 owns their creation.

CI uses the following retained stores:

Run the existing CI workflow with warm_runners enabled to populate these stores after provisioning or dependency updates:

gh workflow run ci.yml --ref chore/ci-improvements \
  -f warm_runners=true -f publish_dev_images=false

After merge, select the maintained default branch instead. This mode calls .github/workflows/ci-warm.yml and visits the four ARM test workers and five native Metallo workers by name. Tests compare the native matrix with the maintained host inventory. The control, retired and VPN-maintenance workers are excluded. At most two warming jobs run concurrently, and each waits in its worker's normal Actions queue. Existing jobs finish normally. Repeated warming requests never cancel active warming.

General workers fetch both pnpm workspaces, all verification Python environments, Go dependencies, generator dependencies and pinned Ansible collections. Infrastructure workers fetch their application dependencies and missing fixture and base images. Authentication workers fetch both frontend workspaces, their Python and Go dependencies, Chromium and authentication fixture images. Node 22 and 24 are prepared where the worker's lanes need both versions.

Package prefetching uses disposable manifests and lockfiles, disables pnpm install scripts, skips local Python packages, and verifies that source inputs and locks remain unchanged. Locked third-party Python dependencies are built into cached wheels when no compatible wheel is available. It records the actual cache paths and timings. pnpm's store is resolved from each normal workspace and bound explicitly into temporary staging, so a separate /tmp filesystem cannot select an unused store. Warming and test jobs select the same persistent store before pnpm setup; the setup action replaces its own ~/setup-pnpm directory on each job. Frozen installs remain in normal jobs and fetch any newly required dependencies. Browser warming preserves other installed browser versions. Image warming derives references from the checked-out fixture configuration and Dockerfiles, downloads missing native images serially, and verifies their presence. It never publishes application images or starts fixture services.

Warming has separate run concurrency and a separate check name. It produces no successful test records and cannot satisfy ci-gate. A successful maintenance run establishes that the downloads and tools are ready; source and real fixture qualification remain separate.

Package and collection warming require at least 5 GiB free space; image warming requires 10 GiB and rechecks space between pulls. These commands leave cache retention and cleanup to maintenance. Keep each worker's Docker daemon and package stores private to its runner account; sharing mutable working trees or a Docker data directory between workers would couple otherwise isolated jobs.

Offline checks:

python3 -m unittest discover -s scripts -p test_ci_runners.py