For AI agents: the complete documentation index is available at /llms.txt, the full documentation bundle is available at /llms-full.txt, and this page is available as Markdown at /reference/services/metallo/contexts/site_runtime/README.md.

Site Runtime

The optional Run serving listener accepts the API's dedicated TLS transport credential. Each request resolves current viewer authority or explicit public access from the Run owner and validates the saved Attempt, native account, service, endpoint and Pod lineage. Public access uses the accepted workload's retained account without inventing a viewer identity. Open connections revalidate every five seconds and close on expiry, revocation, replacement or unavailable authority.

Metallo owns Site-local command delivery, native recovery, private workspaces and execution access. Runtime retains private SQLite journals; it has no platform PostgreSQL credentials. Its domain and application remain provider-independent. The context design describes delivery, receipts and native recovery states.

For accepted execution, Runtime queries Site Coordination's authenticated /v1/site/executions/authorise interface. The owner returns the saved Run/Attempt material, current Project authority and Resource Provisioning's confirmed member account binding. The material digest, canonical IDs and account reference must agree. Success is used for the immediate effect and is never retained as a reusable permission verdict. Repository grants use the separate Run-scoped Site API routes.

The Slurm Runtime composes the same way: it passes no separate personal-access ports and takes the binding, the current authority, the Run-scoped repository grants and each per-action member token from those Site API routes over its own scoped bearer. Preparation and start supply the complete Run/Attempt identity and immutable material digest to the authority query. Its workspaces are member-owned directories with no filesystem ACLs. The Runtime writes credentials into its own read-only hand-off; each member staging job creates its temporary Git configuration beside the repository directory and removes it on exit. The configured REPOSITORY_CA_FILE, when present, supplies the repository provider's CA to both staging jobs. Before entering its service sandbox, the installer runs --initialise-workspaces as the Runtime user. This creates or validates only RUNTIME_WORKSPACE_ROOT/.runtime, owned by that user and pharos-users with mode 0711; it needs no Site credentials or network connection.

The exact accepted member account is retained across personal membership removal. Each new dispatch or stage-in credential issue still needs accepted-work authority; Project closure and changed management generation refuse it. Existing native observation and UID-bound cancellation or cleanup continue. Personal-access provider ports remain for isolated legacy lanes.

If endpoint creation is refused after a native service was created, Runtime deletes that exact native UID and confirms cleanup before reporting the refusal. Replaying the original command retains its recorded reason. A later command for the same Attempt returns native-start-compensated; Execution Core records a terminal rejection. A definite refusal before native creation can still be retried.

Temporary authority or Kubernetes Secret refresh failures defer the refresh without asking Source Control to revoke the running Attempt's grant. Source Control owns any issued renewal until the same request recovers it. Expiry-driven rotation can still invalidate the previous token before the Secret update has completed. A definite authority denial and normal execution cleanup revoke the grant.

Every ExecutionDeferred names a stable code from domain/deferrals.py, and the Runtime logs that code with its own message. The platform explains each code to members with fixed text. scripts/check_architecture.py refuses a deferral without a catalogued code. Any other error is logged as runtime-error, without its text.

When the pending command defers, the worker also reports the code to POST /v1/site/commands/progress, once per command and code. The report is best effort: the command stays pending whether or not it arrives, and a failed report is sent again with the next deferral. When a repository grant fails and the cleanup revocation also has to wait, the grant's own code is the one reported; a definite denial still waits for the revocation before it is acknowledged.

Observations of an unfinished workload can say why it has not run yet, with a code from domain/native_waits.py and never provider text: Kubernetes Pods that cannot be scheduled, whose image cannot be pulled, whose container cannot be created or keeps crashing, and pending Slurm jobs by their state_reason. A service's reason comes from the same Pods its health check below reads, so a Pod that cannot be trusted gives no reason, and private reads keep refusing it.

A Deployment or InferenceService fails when its conditions say so, or when one of its own Pods (found by its labels and owner chain) has a container or init container that is not ready and has restarted more than SERVICE_RESTART_LIMIT times (default 3) since the service last stayed healthy for ten minutes (#1054). Kubernetes normally retains cumulative restart counts, so Runtime saves each container's count in the private journal once the service has been healthy, without a restart, for ten minutes, the period after which the kubelet resets its own restart back-off, and counts only later restarts. A node restart can reset a container's count. Runtime detects decreases against the last observed counts and saves a zero baseline for that container before judging its health, so the restart limit still applies after Runtime replacement. Other containers retain their baselines. A service that is ready for a while between crashes therefore still fails. Without that check a crash-looping service would wait for the rollout's progress deadline, and one that crashes after becoming healthy would never fail. The failed record keeps the container, its new restarts and its last exit code until the observation can carry a reason (#1053). Observing checks only that a Pod belongs to the service, not its exact template, so Pods a controller rolls after a KServe or serving-runtime upgrade still count. A Pod whose ownership or status cannot be confirmed is left out, and without Pods the observation uses the native object's conditions alone: a Pod never fails an observation, while private reads keep every check.

A failed Deployment or InferenceService can leave native Pods retrying. Runtime records a failing cleanup state, deletes the exact service and endpoint UIDs, and confirms their absence before reporting terminal failure. Lost deletion responses and interrupted cleanup resume from the private journal. Private health reads do not trigger cleanup. Runs already reported as terminal by older code need separate operator recovery because Execution Core has stopped their supervision.

Terminated and failed Kubernetes services report cleanupComplete only after their native service and endpoint have gone, temporary helper Jobs have finished deleting, repository grants are revoked and the ephemeral Attempt PVC has gone. Workspace disposal records its pending state before deletion and a completed tombstone after confirmation; retries and Runtime replacement cannot recreate that Attempt's workspace. Cancelled batch Jobs, abandoned preparations and permanently refused inputs dispose of their helper and workspace by the same path, once no native Job remains. A batch Job that finishes during cancellation retains its natural terminal outcome in the journal; Runtime reports it after native deletion and workspace disposal complete, including after restart. Native deletions use exact UIDs and independently enforced ownership policies. Execution Core can then publish the accepted-work release needed for account withdrawal.

Completed batch Jobs retain their workspaces for authorised output and log reads. Their retention cleanup remains separate and can still consume Project storage quota. Existing terminal service Runs recorded without cleanup confirmation still need operator recovery because their supervisor has already stopped.

Workspace PVC creation distinguishes a definite Kubernetes admission or quota refusal from an uncertain response. A refused create records native-request-denied through the normal command outcome, allowing subsequent commands, including cancellation, to cross the ordered Runtime queue. Conflicts, connection loss, authentication failures and server errors retain the pending command for recovery. The workspace intent remains journalled before creation and a recovered PVC must still match its saved identity.

Kubernetes admission independently compares the submitted native material with the owner's saved payload and reads the current native account UID. The admission process has its own Site identity and read-only native grants. Restricted CEL policies continue to protect Pods and KServe controller work. Production composition lives under services/; installation details are in the native admission profile.

Private reads use a separate owner-issued permit. read_output, read_logs, read_file and predict accept read_permit_id alongside the actual requesting Party and Project. Production checks the permit's exact Run, Attempt, viewer, read kind, path and byte limit before reading and before returning data. A completed batch Run can be read by another currently authorised Project member; the original initiator's personal account is unnecessary. Service files, logs and prediction require the native service to remain active. Its ephemeral workspace is discarded after termination, so no service output read is available afterwards. These reads use the current viewer's confirmed member identity and leave the accepted dispatch deadline unchanged. A viewer with no account in the Run's allocation is refused. This covers Kubernetes only. Slurm log reads use the separately activated dedicated spool. Other Slurm output files remain owned by the initiator's POSIX account, so reading those files as the viewer still needs its own decision (#932).

Private inference reads accept the installed KServe controller's INFERENCE_SERVICE_NAME environment addition only when its value is the exact saved InferenceService name. Duplicate, altered and unexpected additions remain invalid, alongside the existing controller UID and Pod template checks.

GPU inference uses the class catalogue's vLLM runtime and its OpenAI routes: GET /v1/models and completed POST /v1/chat/completions, /v1/completions or /v1/embeddings requests. Prediction bodies must name the saved model and cannot request streaming. Duplicate JSON keys, non-finite values, other routes and CPU runtime argument additions are refused. Current permission, exact native identity and byte limits still apply before and after the request. This private Runtime method requires a separate authorised browser serving path; deployment qualification is described in the GPU staging procedure.

Active service file reads execute the bounded, symlink-refusing reader in the verified service Pod, with its immutable image's Python interpreter. Native identity and current permission are checked before and after the read. This avoids mounting the live workspace in another Pod, which can change its SELinux label and revoke the service's file access. The reader selects the fixed /usr/local/bin/python or /usr/bin/python3 path for the managed and upstream KServe images; workload PATH and Python environment overrides cannot select the interpreter or its imports.

The batch read-only workspace helper carries the opaque permit ID. Admission confirms it through /v1/site/executions/authorise-read and admits only the expected read-only helper without repository credentials. The viewer's own account creates the helper and admission requires that caller, so another member's account in the namespace cannot use the permit. Launch, preparation and mutable native objects cannot use a read permit. The owner rechecks current permission on every use; an expired permit requires a new authenticated request. See Execution Core's private read contract.

Repository grants carry separate stage-in and stage-out identities. Stage-in requires current continuation authority and the original deadline. Stage-out is limited to the exact saved repository of an outstanding submitted Attempt, so output recovery can cross that deadline and a policy cancellation. It cannot admit native compute. Repository ownership is checked on every issue; completed Attempts cannot renew. Kubernetes cleanup revokes both grant identities before acknowledging, and Slurm records each staging obligation in its private journal. GitLab's date-based expiry bounds a recovery token to the next UTC midnight; normal cleanup and Source Control reconciliation revoke it earlier.

A Slurm submission whose outcome is uncertain is never repeated: the Attempt waits until its job is visible. A connection slurmrestd refused carried nothing, so that Attempt goes back to prepared and submits again once slurmrestd answers. slurmrestd answers a job that slurmctld refused with HTTP 500 and the controller's error_number, such as 2014 when no node has what the job asks for. Only an error saying the job does not fit the Site's nodes, partition, time limit or GRES (DEFINITE_REFUSALS in slurm_access.py) is a definite refusal, slurm-error-<number>. Execution Core ends the Run when Slurm refuses stage-in at start; when it refuses a later stage, the Attempt fails after its cleanup. Errors that turn the job away for now, before slurmctld creates it, such as 2050 while the jobs already submitted fill MaxSubmitJobs, send the stage back to prepared, and it is submitted again (slurm-submission-limited). Communication errors and any other 500 stay uncertain.

A Slurm compute job asks for GPUs through the Site's class-to-GRES map, for example {"gh200": "gpu:nvidia_gh200_144g_hbm3e"}, as tres_per_node, the field Slurm fills from --gres. The class's GRES is saved when the Attempt is accepted, so a later change to the map never changes that Attempt's compute job. Stage jobs never get a GPU and never read the map. The Site's max_wall_seconds bounds the compute job only. A class the map lacks, the legacy gpu class included, is refused as unsupported-slurm-accelerator-class, and a longer request as slurm-wall-time-above-limit. Both refusals come before the workspace is touched, and Execution Core ends the Run with their reason. The executors take both settings as arguments; the production composition does not pass them yet.

The member-token owner's four permanent consent refusals end the Attempt. Runtime saves the fixed refusal before cleanup, so a restart or a later command cannot submit another stage. Before stage-in, it returns a command rejection once cleanup succeeds. Between stages, it returns a failed observation with failureCode and cleanupComplete; already completed native jobs retain their recorded identities. Repository revocation and workspace cleanup remain retryable. Issuer text, tokens and sealed values never enter the refusal or journal. Temporary token-owner failures defer the same stage without producing a terminal outcome.

Private Run log commands

The outer LogCommandExecutor handles queued-batch and reconciled-service log reads before the native lifecycle executor. It checks the current viewer permit, exact saved Run/Attempt/material/native reference, caps the operation at ten seconds and the encoded upload at 5 MiB, then uploads before acknowledgement. The source-message budget is at most 4 MiB. A private expiring SQLite upload cache preserves identical bytes across recovery and requires fresh permission before replay. It carries no execution authority and may be regenerated.

Kubernetes uses its existing bounded Pod readers. Slurm uses the root-managed dedicated spool; it never reads member scratch. Completed Slurm jobs require sealed root-owned output and a matching root receipt. Legacy jobs have no such receipt and fail closed. This implements active Kubernetes reads and retained Slurm reads; a final snapshot taken before failed-service cleanup remains separate work.

Failed Slurm stages record a bounded failure before cleanup: stage-in, compute or stage-out, the exact scheduler state and job ID, and each available process or derived-step exit code and signal. A scheduler failure with exit code zero remains a failure with zero recorded; missing values remain absent. Runtime retains these facts through cleanup retries, cancellation and replacement, without rereading expired accounting history. If cancellation first discovers a failed stage, or a job fails while cancellation is pending, Runtime records the failure before cleanup. It preserves the cancellation acknowledgement; a job reported as CANCELLED stays cancelled. Scheduler messages, stdout, stderr, credentials and environment values never enter this failure record.

New Attempts save diagnostic script version 1. Their scripts print a fixed stage/exit summary, retain the original exit status through Git configuration cleanup, and preserve a child signal through the timestamp wrapper. Failed pinned fetches retain the last 4 KiB of stderr in their existing private log. Older Attempts retain their original generated script bytes when a prepared stage is retried.

The first-run bootstrap still opens its private log only after the member root and Attempt have been validated. Before that, the bounded scheduler failure record provides the stage and available exit evidence. Capturing earlier stdout and stderr requires the administrator-installed spool; shared allocation directories provide no safe substitute. The spool and central log-read activation remain operator work. Existing failed Runs and discarded output cannot be reconstructed by deploying this change.

Page source

Source snapshot ยท Revision f928834c8.