# DD — Site Runtime context (5D)

Current accepted-work authority and retained member account bindings are described in the
[Site Runtime README](../../../contexts/site_runtime/README.md). The migration and
qualification history below records the earlier personal-member provider lanes.

[Common design](dd.md) · [Completion checklist](adr.md) ·
[Delivery recovery](runtime-recovery.md) · [Native execution](kubernetes-jobs.md).

## Scope and ownership

Site Runtime is a bounded context inside the Metallo package, deployed at the
Site. It owns ordered local command processing, recovery of native effects,
current member-access checks and repository/workspace rules. Kubernetes Jobs,
application Deployments and KServe InferenceServices use the shared worker and
ports. Slurm implements the same executor/recovery boundary under 6A–6C; its
adaptor baseline and remaining production owners are described in
[Reliable Slurm job execution](slurm-jobs.md). This does not qualify the native
Icarus path.

| Owner | Authoritative facts |
| --- | --- |
| Execution Core | Runs, child Attempts, lifecycle state, retry decisions and completion events. |
| Site Coordination | Ordered environment command streams, command identities, recorded results and server acknowledgement progress. |
| Site Runtime | Local delivery checkpoint, exact pending result, native recovery records and private workspace identity. |
| Provisioning and permission providers | Member/Project account bindings, current access and repository credential grants, exposed through explicit ports. |

Run, Attempt, member, Project and command IDs retain their existing meaning.
Runtime correlates each command and result to those identities; a native Job,
controller, Pod or PVC UID is an opaque provider binding. Local recovery facts
cannot independently advance a Run or rewrite Site Coordination's command stream.

## Package boundaries

Paths below are relative to `services/metallo`.

| Package | Responsibility |
| --- | --- |
| `contexts/site_runtime/domain` | Checkpoint and failure values; member binding, permission and start identity values; repository request/grant values; repository destination and private-read rules. |
| `contexts/site_runtime/application` | Ordered worker, access orchestration, family routing, ports, explicit batch/service provider bundles and the one-time installation decision. |
| `contexts/site_runtime/adaptors` | Site HTTPS client, rotating credentials, SQLite delivery/native records, Kubernetes execution/access/workspaces, Git/LFS fetching, configured image resolution and the installation record ConfigMap. |
| `adaptors/site_protocol` | Shared Site wire encoding, decoding and validation used by outer context adaptors. |
| `services/site_runtime.py` | Compose the common worker with its transport, journal, codec and supplied native executor. |
| `services/kubernetes_runtime.py` | Compose the Kubernetes executors, installed provider bindings and persistent journal paths; run the installation hook and the Site identity check. |
| `services/runtime_liveness.py` | Fail the liveness probe when the worker's heartbeat is missing or stale. |

`StartIdentity` belongs beside the other member-access values in
`domain/access_model.py`. Repository request and grant values live in
`domain/repositories.py`; provider and workspace protocols live in
`application/repositories.py`. This keeps domain rules independent of application
orchestration. Kubernetes manifest construction and native identity validation
remain adaptors because their rules depend on the provider's API.

Domain/application imports point inward, using context-local values and the
existing provider-independent foundation. They do not import generated wire
models, HTTP clients, SQLite, Kubernetes or another context's implementation.
Generated contract classes and their validation stay in outer adaptors. The
shared Site protocol package prevents Runtime and Site Coordination adaptors from
depending on each other's internal codecs. Runtime's codec adaptor correlates
commands, results and native references. The worker and journal retain their
existing recovery-error handling.

Executable startup and environment/file configuration stay in `services/`.
Composition explicitly supplies native executors and provider ports. Adding a
context package does not activate another process in the central Metallo service.

## Application ports and flows

| Ports | Required behaviour |
| --- | --- |
| `SiteControl`, `RuntimeCodec` | Pull numbered commands, validate their identity and acknowledge a correlated result using the existing Site protocol. |
| `RuntimeJournal` | Lock, restore and atomically update local checkpoint state; persist a command before execution and the exact result before acknowledgement. |
| `CommandExecutor` | Execute a new command or reconcile an uncertain effect through `recover`, using stable command/Attempt identity. |
| `MemberBindings`, `RuntimePermissions`, `NativeWorkloads` | Resolve a correlated native binding, check current access and submit under the approved member identity. |
| `BatchImages`, `BatchRecords` | Supply the configured execution image and durable native recovery records. |
| `RepositoryGrants`, `RepositoryAccess`, `Workspaces` | Authorise scoped Git/LFS fetch and write access, retain a normal checkout, manage temporary credentials and prepare/verify/read the correct private workspace. |
| `ServiceData` | Read private logs/files or proxy a service endpoint after current permission and exact native identity checks. |
| `InstallationRecord` | Record, outside the journals' volume, that this installation initialised them; keep an existing record. |

The worker retains the 4C ordering and recovery rules. It saves each command
before invoking a native executor, saves the exact result before sending it, and
clears pending state only after an exact acknowledgement. A lost response replays
the saved result. An uncertain native action invokes recovery; unexplained local,
server or native state stops progress.

The access workflow checks the command's member, Project and account reference
against a fresh trusted binding and current permission before each new native
start. Native adaptors enforce impersonation, exact identity checks and UID-bound
cleanup. Repository and private-read rules retain the 5C ownership, current
permission and path checks. The configured family and saved `serviceKind` retain
the existing executor selection; a replay cannot change the selected service kind.

## Persistence and compatibility

The context registry declares Site Runtime's persistence as `sqlite`. Runtime
does not receive a platform PostgreSQL database, migration history, runtime role,
database credentials or PostgreSQL aggregate storage. Execution Core and Site
Coordination retain their separate private PostgreSQL databases and transactions.

The existing Runtime PVC holds the same three private files:

| File | Existing recovery facts |
| --- | --- |
| `delivery.sqlite` | Environment/authority binding, acknowledged sequence, pending command and exact result, including their integrity hashes. |
| `jobs.sqlite` | Batch preparation/submission, saved native identity and private workspace records. |
| `services.sqlite` | Application/KServe preparation, specification, native identity and recovery/cleanup progress. |

This is a package and ownership refactor. Keep each journal's schema version,
stored representation, configured location, locking and restoration checks
unchanged. Existing files reopen without a reset or conversion. Initialisation
remains explicit; normal restart must not replace a missing or inconsistent
journal. The installation hook initialises only a volume with no journals and no
installation record, then records it; complete journals that predate the record
are recorded as they are. Once recorded, missing or partial journals stop the hook
until they are restored together. The installed PVC and
single-writer requirements are unchanged.

The Site HTTP routes, OpenAPI/generated execution contracts, command operations,
identities, sequencing, acknowledgement and exact-result replay rules remain
unchanged. No new operation, public endpoint, browser subscription or provider
connection is introduced. The fixed platform image, repository helper, private
workspaces and native access policy retain their existing rules, with the native
qualification fixes described below.

## Fixes found during native qualification

The batch harness now installs the existing workspace permissions and admission
policy as well as Job permissions. The role is limited to workspace operations;
it does not grant application or KServe control.

Real preparation also exposed a timing loop: Kubernetes' default 30-second helper
shutdown consumed the whole cleanup wait. New trusted helper Pods use a five-second
grace period, followed by a separate 30-second foreground-deletion allowance.
Recovery also accepts the previous 30-second helper grace while retaining all
other identity and specification checks. Readiness still requires credential
revocation and complete helper removal. Neither SQLite representation nor the
Site wire contract changes.

Kubernetes omits a readiness probe's `initialDelaySeconds` when it is zero.
Recovery now accepts that exact omitted default when comparing a Deployment,
InferenceService or service Pod with its saved specification. Nonzero delays,
other missing fields and changed native identities still fail recovery. The
rendered specification and existing journal records keep the same format.

The pinned KServe 0.16.0 installation requires the `Standard` deployment-mode
name for these manifests. New inference preparations and the isolated installer
use that value, following the [upstream rename](https://github.com/kserve/kserve/releases/tag/v0.16.0).
Existing saved `RawDeployment` specifications and endpoint records remain readable
and retain their recorded value; they are not silently migrated. Their other
specification and identity checks remain strict. A server-side dry run of the
corrected manifest passed under the member identity before rerunning the full lane.

The endpoint admission policy permits the configured KServe controller's
`attempt-<Attempt ID>-predictor` Service with exactly one selector,
`app=isvc.<Service name>`. It requires an InferenceService controller owner
reference with the matching Attempt name, API version and nonempty UID. Existing
member/Project identity checks, member selectors and private ClusterIP restrictions
remain enforced; other callers cannot use the controller exception.

Private inference checks accept only the pinned runtime's exact argument prefix:
`--model_name=<saved InferenceService name>`, optionally followed by
`--backend=huggingface`, before the complete unchanged saved model arguments.
Changed names, paths, backends, extra arguments and postfixes are rejected.
The original Pod still has to match its controller's template and UID lineage.

The isolated test installer uses dotted KServe hostnames through
`{{ .Name }}.{{ .Namespace }}.{{ .IngressDomain }}` and selects
`--backend=huggingface` in its namespaced ServingRuntime for the tiny CPU
classification fixture. These are qualification settings; they do not change
shared installation configuration or select a production model/backend.

The batch quota fixture now reads the saved preparation outcome before submitting
its start. Previously it rebuilt an acknowledged preparation command with the same
ID and a new timestamp, correctly triggering the store's conflicting-retry check.
Production command idempotency remains unchanged.

## Required verification and evidence

Historical evidence in 4B/4C and 5A–5C retains its dated scope. It does not by
itself verify the moved packages or the installed Runtime after this change.
Record new results here and update 5D's checkboxes only after the relevant checks
have completed.

- [x] Verify context registration, inward imports, packaged entry points and
  rejection of Runtime/provider code outside the intended boundary; run the
  repository architecture checks and `pnpm verify`.
- [x] Run Runtime worker, access, routing, repository and native recovery unit
  tests, including real SQLite reopen, corruption, locking and rollback cases.
  Identify transport, native executor and external-provider test doubles.
- [x] Run the real PostgreSQL/HTTPS/SQLite integration lane with production
  codecs, saved Runs, ordered commands, exact result replay and Runtime restarts.
  Record any doubled identity authority, native effects and provider lookups.
- [x] Qualify the installed Kubernetes Runtime and native recovery separately
  for Jobs and both service kinds, including Runtime replacement and retained
  native identities/journals. State the real Kubernetes/KServe dependencies and
  remaining provisioning, permission and repository-provider doubles.

Real Slurm execution and recovery remain in 6A–6C. Live provider connections,
browser/UAT and other deployment lanes retain their existing scope and must be
reported explicitly when not exercised; package reorganisation supplies no new
qualification evidence for them.

### Verification recorded on 2026-10-15

| Check | Result and scope |
| --- | --- |
| `pnpm generate:contracts` and drift checks | Passed; this refactor changes no wire schemas or generated transport models. |
| `pnpm verify` | Full run passed, including architecture checks, editable/wheel import checks, mypy on 195 files and 1,217 Metallo tests. Four Git/LFS cases were skipped in the tools image and passed separately in the preparation image below. Evidence: `/tmp/metallo-runtime-context-verify-final.log`. |
| Real Git/LFS helper tests | All four passed: immutable checkout, replay, size limit and symlink rejection. HTTPS Git/LFS is real; authorisation and credential issuance use test doubles. |
| `pnpm test:integration` | Passed: 163 Metallo infrastructure cases, real PostgreSQL/RabbitMQ, all service images and the browser lane. Native execution and external providers remain doubled in the common integration lane. |
| Shared dev-box protocol lane | Passed on real PostgreSQL, HTTPS, Kubernetes TokenReview and SQLite on a PVC. Covered delivery/replay, Runtime replacement, supervisor recovery/fencing and native member-access probes. Lifecycle executors and provider interfaces use test doubles in this lane. |
| Native recovery regressions | Ruff, mypy and 119 focused Job/service/workspace tests passed after the native readiness-default fix. A further 71 service/private-access cases passed with the KServe per-field comparison and mode fixes, including recovery from a saved legacy mode. |
| Native batch dev-box lane | Passed with real Jobs, private PVCs, exact Git/LFS checkout, outputs/logs, Runtime Pod replacement, retained Job identity, quota blocking/release, failed jobs and cancellation after native grant removal. Provisioning, platform permissions and credential issuance use test doubles. |

### Native service qualification

The fresh isolated service run passed its four recorded phases on 15 October
2026: three logical workloads, one real prediction and four native admission
denials. PostgreSQL, Site HTTPS, TokenReview, SQLite, Kubernetes, KServe and HTTPS
Git/LFS were real. Provisioning bindings, current platform permissions and
repository grant issuance/revocation remained explicit test doubles.

The local test orchestrator was deliberately paused before private checks while
a supplementary automated restart exercised both healthy services. Remote
workloads continued unchanged; no product or configuration repairs were made.
The original private checks and cancellation/termination phases then completed.
The permanent harness now includes this restart probe.

- Both service kinds prepared private repository PVCs, ran `workspace-init` in
  the serving Pod and reached healthy saved Run state. The application returned
  its expected response and the real Hugging Face CPU model returned a prediction.
  The same lane also completed a real batch Run.
- The first Runtime replacement occurred after creating the application
  Deployment but before recording its native result. Recovery retained its UID
  and the journal PVC without creating a duplicate application.
- A second replacement occurred with both services healthy. The new Runtime
  became Ready; the Deployment, InferenceService and journal PVC kept their UIDs,
  both services remained healthy, and no duplicate native services appeared.
  This verifies restart of an existing inference service. Recovery from an
  uncertain KServe creation remains outside this evidence.
- After that replacement, both services passed private file/log and endpoint
  checks. Other-member access and escaping paths were denied. Native admission
  rejected foreign workspace mounts, a changed container UID and mismatched
  application/inference Pod ownership. The permission double also denied revoked
  inference access.
- Removing the member's native create grants was followed by successful
  cancellation and terminal Run state for both kinds. The final check found no
  remaining application Deployments or InferenceServices.

| Qualified component | Recorded version or immutable identity |
| --- | --- |
| Runtime image | `pharos-metallo/identity:services-qualified-c58faad65dee` |
| Runtime image ID | `sha256:77a0c0b6a808d1a28c51c90fe7eb631cb3a4b97b70feda684b67ae20505d89cd` |
| Fixed platform image | `docker.io/pharos-metallo/batch@sha256:0deced726793b1a218c0e0be639fb4530878c26e9ee2c186e3041cc2bc3237d5` |
| KServe | `0.16.0` |
| Hugging Face serving image | `kserve/huggingfaceserver:v0.16.0` |
| Hugging Face serving image digest | `sha256:6c5ebbc2b660da504476cf18db0a1f64ee80417459ce0bf36fadbf767a62f52f` |
| CPU model fixture | `hf-internal-testing/tiny-random-BertForSequenceClassification` |
| Model revision | `4d5a3af95422fc0ebf732796c685c586b6d7420b` |

This completes the native recovery checks for the 5D context refactor and the
5C native-service private-data checkbox. The remaining native checks were qualified
on 23 October 2026 (below); real-provider connections retain their open ADR checks. Automatic cleanup removed the owned namespaces and all
23 owned KServe global resources. All three successful lanes and the earlier
failed/diagnostic lanes were cleaned; private reports were preserved. The final
full verification rerun also passed after adding the permanent healthy-service
restart regression. Sanitised evidence: `/tmp/runtime-context-native-summary.json`.

### Native qualification on 23 October 2026

Uninterrupted runs of `make metallo-site-api-services-test` and
`make metallo-site-api-jobs-test` passed on a dev box (Kubernetes `v1.34.11+k3s1`)
and removed their namespaces, admission policies and the 23 KServe global
resources afterwards. PostgreSQL, Site HTTPS, TokenReview, SQLite, Kubernetes,
KServe `v0.16.0` (`kserve/huggingfaceserver:v0.16.0`) and HTTPS Git/LFS were real.
Provisioning bindings, current platform permissions and repository grant
issuance/revocation remained explicit test doubles. The fixed platform image was
`docker.io/pharos-metallo/batch@sha256:0deced726793b1a218c0e0be639fb4530878c26e9ee2c186e3041cc2bc3237d5`.

- **Uncertain creates:** the Runtime was replaced after each kind's first real
  create and before it saved the result. Both replacements adopted the existing
  Deployment and InferenceService: their UIDs and the journal PVC were unchanged
  and no duplicate service appeared.
- **Native access (5A):** server-side dry runs with the Runtime's own credential
  checked both kinds in the member's Project and in a second Project of the same
  member. 36 requests were refused: creates without a member or with an
  administrator, default-account, privileged-group or KServe-controller identity,
  the member's account in another Project, services claiming another Project or
  member, patches by the Runtime or the member, Runtime deletes without the exact
  UID, member deletes, another Project's services, Secrets, Pods and exec, role
  grants, workload tokens, namespaces, admission changes, cluster-wide Secrets and
  nodes. Six controls were accepted: the bound member's own create in each Project
  and the Runtime's exact-UID cleanup. After the member's grants were removed, its
  creates were refused while its other Project and the Runtime's cleanup were not.
- **Quota (5B):** a count quota at the Project's current usage refused both kinds'
  creates. Each start failed with `native-request-denied`, the Runs kept their only
  Attempt without a native reference, and the supervisor retried. After the quota
  was removed, both services started and became healthy.
- **Failure (5B):** an application that exits and a model server whose model files
  were removed both reached the saved Run state `failed` with one completion event:
  the Deployment at `ProgressDeadlineExceeded`, the InferenceService at
  `BlockedByFailedLoad`/`FailedToLoad` (`ModelLoadFailed`, exit code 1). Failed
  services remain until retention cleanup.
- **Revocation (5B):** all four running services were cancelled and terminated
  after the member's native create grants were removed.
- **Batch data (5C):** after the first member's Job finished, direct requests as a
  second member of the Project and as the Runtime without a member were refused 14
  times: mounting the first member's workspace or credential, acting as that member,
  bare Pods, reading its logs, workspace claim or credential, exec into its Pod,
  Jobs without a member, cloning or binding its volume, and naming a workspace or
  credential for another member. The second member's own Job and a fresh workspace
  claim were accepted. The rest of the batch lane passed unchanged.

Two lane faults were corrected before these runs. Probe manifests had been
narrowed with a copying helper, so they kept the original identity; unit tests now
check them. The batch quota case's 90-second wall time also counted the time the
quota blocked its Pod, so its release could exceed the deadline; that case then
allowed 300 seconds, until the wall time moved to the Pod (#764).

### Installed dev-box Runtime on 23 October 2026

The installation commit, then based on main `c2ed661e8`, was delivered to the
user's dev box (Kubernetes `v1.34.11+k3s1`); `pharos-site-runtime` became Synced
and Healthy at that commit.

- The cluster built the Metallo and fixed batch images. The Deployment runs
  `registry.pharos-build.svc.cluster.local:5000/metallo@sha256:7832458c4685615809dc785b10a21dc2fbdf224e6124a0e7204e285f9d8554d6`
  and `BATCH_IMAGE` is
  `registry.pharos-build.svc.cluster.local:5000/batch@sha256:91700eb55ed731862839639cb06afb04f5eb599e47313107766379e2118eb770`.
  The Deployment stays at zero replicas with its liveness probe.
- The installation hook initialised the journal volume and recorded
  `runtime-installation` (authority `https://site-api.pharos.svc:8443`, 23:31:47 UTC);
  Argo deleted the completed hook.
- The bootstrap installed the eight admission policies and their bindings. Each
  was type-checked without warnings and denies only in namespaces labelled
  `execution.pharos.dev/access-policy: pharos-site-runtime`.
- A one-off Job built from the installed Deployment's Pod template ran `--check`.
  The Site API authenticated its projected token as
  `system:serviceaccount:pharos-site-runtime:runtime-k8s`, authorised the `k8s`
  stream, and the read-only pull returned no commands.

The node's network policy controller admits a new Pod about four seconds after it
starts, so a new Runtime Pod's first connections to the Site API are refused. The
serving worker retries them. `--check` exited on the first refusal, so the check
above waited 15 seconds before it ran; `--check` now retries a failed connection
for up to a minute.

### Kubernetes Job lane on the dev box, 28 October 2026

`make metallo-site-api-jobs-test` passed all 16 phases on the same dev box
(`v1.34.11+k3s1`) and removed its namespace. Two Runs of one CPU each started
together under a quota with room for one: one ran, the other waited queued and then
finished with its full wall time, as the [capacity rule](../../resource-site/design.md)
requires (#764).

Before this, no Run got past workspace staging there. The checkout ran Git without
`/usr/local/bin` on its path, where the preparation image pins Git LFS, so
`git lfs install` failed before any network access. The Project namespace's network
policy also admits a new helper Pod a second or more after it starts. The checkout
now runs Git with that path and waits, within its deadline, until the repository
host accepts a connection. The lane had failed the same way on governance without
the wall-time change.