# Native Debian runners

[`inventory.json`](inventory.json) is the host-to-worker inventory. Each worker
has its own account, home, checkout and Actions tool cache.

| Host | Address | `metallo-runners-` suffixes |
| --- | --- | --- |
| `metallo-vasilis-4` | `10.0.1.40` | `01`, `02`, `05`, `06`, `09` |

The active fleet has five workers. Hosts `metallo-vasilis-5` through `-10` were
withdrawn after loaded operation exposed substantial storage waits and longer CI
jobs. Review shared storage performance before expanding again; passing environment
qualification alone does not establish useful capacity under load.

`metallo-vasilis-2` remains the SSH jump host; its GitHub runner is retired.
`10.0.1.41` is also retired from this fleet. The Rocky Linux workers on
[`platform-pharos-dev`](../platform-pharos-dev.md) have a separate maintenance
procedure and are outside this Debian inventory.

## Provision

Use the frozen DevOps environment and select exactly one host. SSH credentials
stay on the administrator workstation; replace the example key path as needed.

```sh
uv run --project devops --frozen ansible-playbook \
  -i devops/github-runners/native/inventory.json \
  devops/github-runners/native/provision.yml \
  --limit metallo-vasilis-4 --private-key ~/.ssh/ntua-dev-vdanias
```

[`toolchain.yml`](toolchain.yml) pins the runner archive, Python 3.13, uv, Helm,
Docker Engine, Compose and Buildx. Downloaded runner, uv and Helm archives have
committed checksums; Docker packages come from its signed repository. The host
also receives `make`, GCC/G++, native PostgreSQL, TLS roots and Chromium libraries.
The Python installation lives under `/opt/pharos-python`; `/usr/bin/python3`
remains Debian's interpreter. Every worker's `.path` puts the managed Python and
PostgreSQL binaries first.

The playbook requires Debian 12 AMD64, checks the actual hostname, and checks host
CPU and memory against its worker count. It refuses running runner processes,
active Docker containers and hosts with Kubernetes services before package changes.
When migrating cloud-init's `docker.io`, it stops Docker, its socket and containerd
before replacing the packages, then reloads and starts the installed units.
It prepares accounts and service overrides without starting runners or obtaining
GitHub credentials. Repeated provisioning preserves runner registration files,
auto-updated runner binaries, checkouts and caches.

For an existing host, first quarantine every worker on that host:

```sh
python3 devops/github-runners/native.py quarantine \
  --host metallo-vasilis-4 --identity ~/.ssh/ntua-dev-vdanias
```

Quarantine removes the generic and portable scheduling labels while preserving
operator labels and current jobs. Wait for the jobs to finish, then stop that
host's `actions.runner.pharos-ai-factory-pharos-platform.metallo-runners-*.service`
units before provisioning. Inspect and clean up only known abandoned CI fixtures
if containers remain. Do not use a blanket Docker prune on shared state.

## Add one worker to a qualified host

Declare the new number in `inventory.json` and the warming workflow, then use
`add-worker.yml` while the existing workers continue their jobs:

```sh
uv run --project devops --frozen ansible-playbook \
  -i devops/github-runners/native/inventory.json \
  devops/github-runners/native/add-worker.yml \
  --limit metallo-vasilis-4 --private-key ~/.ssh/ntua-dev-vdanias \
  -e '{"new_runner_number":"09"}'
python3 devops/github-runners/native.py register \
  --host metallo-vasilis-4 --runner 09 --identity ~/.ssh/ntua-dev-vdanias
python3 devops/github-runners/native.py qualify \
  --host metallo-vasilis-4 --runner 09 --identity ~/.ssh/ntua-dev-vdanias
```

The example prepares worker `09` when that account is absent. Addition requires
the current root-owned startup preflight, an unused account without a
registration or service, and disk space for another populated workspace. It
downloads a verified runner archive and prepares only the selected account,
directory and service override. It does not update shared packages or restart
existing services. An interrupted preparation can be retried before registration;
after registration, use `native.py` to reconcile and qualify that worker.

[`capacity.yml`](capacity.yml) budgets three vCPUs and 5 GiB RAM per worker,
plus 2 GiB host memory. Both playbooks validate the complete intended inventory.
Addition also budgets 9 GiB for the new workspace/cache while preserving an
18 GiB free-space reserve. These admission budgets support the approved
five-worker allocation on the 16-vCPU, 31-GiB host; they are not enforced service
quotas. A sixth worker exceeds the CPU budget. Review loaded CPU, memory and
I/O measurements and disk growth before changing this policy again.

## Disk retention

Full provisioning enables Docker's native build-cache collection with an 8 GB
retention target and local container logs capped at three 10 MB files per container.
These settings apply to the host's default Docker builder and newly created
containers. They preserve unrelated daemon settings and require an idle host.

Every new worker receives a post-job hook. Below 25 GiB free, it removes only
that completed job's checkout, after its ordinary and post-job steps finish.
It preserves sibling runners, Actions tool caches, package caches and Docker
resources. It rejects redirected paths, foreign ownership and root execution.
The hook has a two-minute timeout; cleanup errors produce a warning without
changing the completed job's result. Qualification tests deletion and sibling
preservation using a disposable checkout.

If free space remains below the 18 GiB reserve, quarantine and drain that host
before removing stale package/browser caches or unused Docker data. Inspect
`df -h`, account-level `du` and `docker system df` first. Restore at least 25 GiB
free, then qualify before admission. The startup preflight still rejects less
than 10 GiB free. Do not run a shared Docker prune while other workers have jobs.
Older hosts acquire this policy during their next drained full provision;
adding a worker installs the hook for the new account without restarting others.

To add retention without replacing packages or existing readiness checks, quarantine
and drain the host as above, stop its runner services, and stop only identified
abandoned CI containers. Then apply:

```sh
uv run --project devops --frozen ansible-playbook \
  -i devops/github-runners/native/inventory.json \
  devops/github-runners/native/retention.yml \
  --limit metallo-vasilis-4 --private-key ~/.ssh/ntua-dev-vdanias
```

This playbook requires stopped runners and no running containers. It installs
the hook, adds a service override and restarts Docker to activate retention.
Runner services remain stopped. Validate the hook with a disposable checkout,
start and qualify the workers, then restore their scheduling labels.

## Register and qualify

Run these from the administrator workstation with authenticated `gh` access to
manage this repository's runners:

```sh
python3 devops/github-runners/native.py register \
  --host metallo-vasilis-4 --identity ~/.ssh/ntua-dev-vdanias
python3 devops/github-runners/native.py qualify \
  --host metallo-vasilis-4 --identity ~/.ssh/ntua-dev-vdanias
```

Repeat for each selected host in the inventory. Add `--runner 05`, for example, to select one
declared worker on the chosen host.

Registration checks the whole selection before requesting credentials. Existing
local and GitHub registrations must agree on ID, name and repository. New workers
receive only `maintenance-tools`, so they cannot take the repository's normal CI
jobs. Tokens pass through SSH stdin and are never stored in Ansible variables,
unit files or the repository. An interrupted service installation can be retried.

Qualification requires online, idle, quarantined workers. It tests the service
account's actual PATH, Python type-parameter syntax, TLS, SQLite, writable caches,
at least 10 GiB free space, compilers, Helm, Compose and PostgreSQL. Its fixtures
exercise a C build through `make`, Helm rendering, Compose build/up/exec/run/down,
native PostgreSQL and headless Chromium. Resource names are unique to the
invocation; cleanup removes only those resources. Logs are private under ignored
`devops/github-runners/logs/`.

Only after every selected worker passes does qualification recheck registration
identity and online/idle status, then enable `pharos-ci` and both portable pools.
A final completion marker is required; an early successful shell exit is rejected.
The script is buffered before execution so child commands cannot consume later
checks from the SSH input stream.
A failed check keeps the selection quarantined. Other maintenance reasons must be
resolved explicitly. The pool-label reconciler also excludes maintenance-marked
workers even when a legacy `pharos-ci` label remains.

The systemd `ExecStartPre` check runs before every listener start and rejects a
broken environment. To run that same read-only check while jobs are active:

```sh
python3 devops/github-runners/native.py check \
  --host metallo-vasilis-4 --identity ~/.ssh/ntua-dev-vdanias
```

These native workers share host CPU, memory and one Docker daemon. Separate
accounts isolate workspaces and caches; they do not reserve memory or provide a
security boundary against Docker access. Size the host for the intended
concurrent jobs and inspect real resource usage before increasing the count.
Passing the environment qualification does not replace the repository's full
PostgreSQL, broker, browser, UAT or deployment lanes.

## Validate changes

```sh
python3 -m unittest discover -s scripts -p 'test_*github_runners.py'
python3 -m unittest discover -s scripts -p test_native_runner_preflight.py
python3 -m unittest discover -s scripts -p test_ci_runner_pools.py
uv run --project devops --frozen ansible-playbook \
  -i devops/github-runners/native/inventory.json \
  devops/github-runners/native/provision.yml --limit metallo-vasilis-4 --syntax-check
```

Run `pnpm verify` and qualify the changed environment on an idle, quarantined
worker before rollout. A syntax check does not establish that a fresh host was
provisioned successfully.