# DD — Staged Slurm batch execution [Site Runtime context](site-runtime-context.md) · Icarus access in [#163](https://github.com/pharos-ai-factory/pharos-platform/issues/163) · run path in [#165](https://github.com/pharos-ai-factory/pharos-platform/issues/165). ## Scope This change adds the production-shaped Site Runtime adaptor for one `queued-batch` Attempt on Slurm. The Attempt is backed by three dependent native jobs: `stage-in`, `compute` and `stage-out`. It implements `queued-batch.prepare`, `.start`, `.observe` and `.cancel` through the existing `CommandExecutor` boundary. The common delivery worker, journal, Site codec, lifecycle supervision and wire contracts are unchanged. The profile uses one `main_repo` and packages installed on the Icarus nodes. It does not run a container image. Multiple repository layouts, input mounts, arrays, distributed jobs, accelerators, checkpointing, experiment tracking, a supplied `artifactRef` and ephemeral-storage demand are rejected before the workspace owner starts any staging effect. `stage-in` checks out the source's exact saved commit on the configured staging partition and runs `git lfs pull`. It keeps an attached branch for stage-out; moving or deleting that branch after acceptance cannot change the input. Older commands without a saved commit retain their branch checkout for compatibility. A remote refusal of the saved object stops staging with `repository-commit-missing` in the stage diagnostic, while transport failures remain distinct. The current observation contract reports the native stage as failed without carrying its reason. `compute` runs on the offline partition after `afterok:`. `stage-out` returns to the staging partition, commits changes, pushes LFS objects and pushes `HEAD` after `afterok:`. Repository access is an explicit `RepositoryGrants` port and private-file installation remains a `SlurmWorkspaces` responsibility; production connections remain open. Usage reconciliation remains outside this change. ## Accepted-work deadline and recovery Preparation and start retain the command's `acceptedUntil` separately in the private Attempt record. Native names, material hashes, workspace receipts and provider references keep their original meaning. Later commands cannot remove or change a recorded deadline. A saved legacy preparation may adopt its supplied deadline once; work accepted before this evidence existed retains the documented legacy behaviour. Malformed saved deadlines stop recovery. The staged executor and retained single-job executor save preparation intent before calling the workspace owner. They check the current clock before preparation and fresh native submission, including after owner calls and journal writes. The Slurm HTTP adaptor checks again after JWT/JWKS verification, immediately before its submit request. Repository grant intent is recorded before issuance so interrupted staging can revoke its exact grant and remove private access files during recovery. Expiry refuses stage-in and compute that have not been submitted. A stage-in already submitted may finish; if it completes after expiry, the Attempt finishes as failed without submitting compute. Workspace and credential cleanup must finish before a terminal refusal or observation is reported. Failed cleanup remains pending and is retried from the saved record. A submitted or uncertain native job keeps its identity and is recovered and observed after expiry. An empty lookup does not prove non-submission and never permits cleanup or another submission. Already submitted compute may finish and create its stage-out job after expiry. Stage-out, cancellation and cleanup retain their existing authority requirements; expiry alone does not cancel native work. ## Deployment boundary Icarus-cloud and Icarus-hpc are separate installations. Icarus-cloud owns the Pharos platform, the Kubernetes Site Runtime, the Site API and the GitLab endpoint with its Git LFS storage. Icarus-hpc is the independently administered Slurm cluster: it owns OpenLDAP, SSSD/sshd, Slurm, the upstream Waldur Site agent, the SSH access agent and the native Slurm Site Runtime. It holds no platform PostgreSQL database, does not run the platform's Kubernetes workloads and exposes no inbound port to Icarus-cloud. All connections from Icarus-hpc to Icarus-cloud are outbound HTTPS. The native Slurm Runtime pulls commands from the Site API and obtains repository grants for the configured Icarus-cloud GitLab endpoint. `stage-in` and `stage-out` therefore use that endpoint and its LFS storage; the later Kubernetes step runs in Icarus-cloud and checks out the exact commit pushed by `stage-out`. The single-host `icarus-hpc` laboratory co-locates disposable versions of these cloud services to qualify the native boundary, but that co-location is not a production dependency or deployment assumption. ## Member binding and replacement The generic binding path is already shared with the Kubernetes executors. Before each new Attempt, `MemberAccess` resolves `memberId`, `projectId` and `accountReference`, checks current permission, and the Slurm executor saves the resulting `MemberBinding` with the Attempt. Preparation and start reject a changed binding. The member JWT authenticates each bounded native Slurm operation but is not the authority for those binding facts. The remaining delta is the versioned Beto Site binding from #765/#785 and its production adaptor. The laboratory still supplies binding and permission doubles; the production composition remains fail-closed. The versioned contract must carry, with source revision and generation: - the canonical member and Project identities; - the opaque `accountReference` returned by Project provisioning; - the Site, the active `site_username` and the native Slurm account; - whether the binding is active, replaced or retired. The Slurm Runtime validates the binding against the member JWT's `site_username` before a new Attempt. It records the exact generation and source revision with the Attempt and reports the applied values in its result. A replaced or retired binding cannot start a new Attempt, even when a later binding reuses the same username or Slurm account name. A saved in-flight Attempt continues from its exact binding and native identities unless an accepted revocation policy cancels it. Observation and cleanup use those saved values rather than re-resolving mutable access. Cancellation still needs a native credential after the binding is retired. A member delegation that is revoked with the binding cannot call `scancel`; whether Slurm needs a separate narrowly scoped cancel and cleanup authority remains part of the #755 access-revocation decision. No new Slurm executor boundary is required. The existing `MemberBindings`, `RuntimePermissions` and `MemberCredentialDelegation` ports are the right Runtime boundary. A Slurm-specific binding model should not duplicate the Beto projection; it should extend the shared binding value and validate its Slurm projection. ## Provider decisions A step resolves its repository input when that step prepares, not when the Pipeline starts. The later Kubernetes step therefore checks out the commit pushed by the Slurm step. #767 records the exact input for that step, which the Runtime fetches even if the branch advances before stage-in. Future outputs are resolved when their step starts. The mixed lane is the Slurm half of #771. It enters through the same API request, launch processing and Site Runtime paths as the Kubernetes half, rather than calling internal executors directly. Its owner ports are labelled doubles in `dx/` and are replaceable as the real providers land. The production composition remains fail-closed. The native Runtime authenticates to the installed Site endpoint with the `psb_` command grant from #744. A grant is read afresh on every request, names exactly one stream environment and must be unexpired and unrevoked. Negative tests cover expired, revoked, wrong-scope and wrong-environment credentials. Successful native use closes the Slurm-host portion of the open 4B verification. Execution Core no longer meters usage. A terminal Slurm Attempt needs the provider's native job identity and terminal observation only. A definite submission refusal is stored as `slurm-error-` when the provider returns one bounded code, or `native-request-denied` otherwise. Until #766 defines a terminal rejected outcome, Execution Core retries the start until the Run's dispatch deadline or cancellation; provider text is not copied into the reason. ## Identity and credentials `MemberBinding.scope` is the native Slurm account and `MemberBinding.subject` is the member's `site_username`. `accountReference` remains the canonical provisioning reference; Runtime never derives an account or Unix user from it. `MemberCredentialDelegation` is a separate explicit port. It supplies a freshly renewed ordinary member access token for each bounded native operation and returns the correlated member and `site_username`. Runtime does not use the technical Site Runtime token as a member credential. The production owner is not connected and composition fails closed. Before every native Slurm request, the adaptor verifies the compact JWT signature and claims with the configured issuer JWKS. Under the direct-token policy in [ADR 0035](../../../../../docs/decisions/0035-local-aai-laboratory.md), it requires the exact issuer, configured login client's `azp`, `site_username`, `typ=Bearer`, the `site_username` scope, a nonempty subject, at most 3600 seconds of validity and no conflicting native username claims. Staging omits `aud`; if present it must name only the login client. No exchange or additional client is needed. `nbf` is checked when the issuer supplies it; Keycloak access tokens without `nbf` remain admissible. The token then travels only as `X-SLURM-USER-TOKEN`; `X-SLURM-USER-NAME` must equal the verified binding subject. Admitted JWKS keys are cached for at most five minutes using the verifier's injected clock. An unknown key causes one refresh; an expired cache is refreshed before reuse, so a removed key does not remain trusted indefinitely. No member token enters a command, SQLite record, outcome, event, exception, job environment or workload script, and it is not passed to the workspace owner. `MemberCredential` redacts its bearer value from `repr`. Git credentials are issued just in time for each staging job through `RepositoryGrants`. The production workspace puts the username and token in the stage job's own Slurm environment when the Runtime submits it, which Slurm shows only to the job's owner and its administrators, and writes only a runtime-owned askpass helper and the repository CA to disk; the saved job never carries them (#974). The laboratory's workspace double still hands them over as files in its own credential directory. The member's stage-in job creates the Attempt and workspace itself with `umask 077`, so Runtime does not traverse or modify the private workspace. The Git token never appears in the script or durable record. ## Native job identity | Field | Recorded value | | --- | --- | | Names | `pharos--in`, `pharos-`, `pharos--out` | | Account | trusted `MemberBinding.scope` | | User | trusted `MemberBinding.subject` | | Comment | `pharos:` | | Dependencies | `compute` after `stage-in`; `stage-out` after `compute` | | Partitions | staging for stage-in/stage-out; configured offline partition for compute | | Resources | compute uses the recorded demand; staging uses the saved Site configuration described below | | Retry | `requeue: false`; Execution Core owns new Attempts | The saved SQLite record contains the material identity, trusted binding, staged workspace receipt, exact request and state for every submitted stage, rejection evidence, native job IDs, repository-release progress and the exact cancellation result. Each stage writes to an Attempt-scoped log beside the private workspace. Restored records re-render every stage from the saved material, binding, partition and specification, so scripts, resources, environment and identities must match byte-for-byte. Exact lookup also requires the requested and returned native job IDs to agree. The record contains no member or repository credential. ### Staging settings and reported state The native Runtime reads `SLURM_STAGING_CPUS` (default `1`), `SLURM_STAGING_MEMORY_MIB` (default `512`) and `SLURM_STAGING_TIME_MINUTES` (default `10`). Each must be an integer from 1 to 2147483647. Operators should choose values that fit the Site's staging partition, transfer sizes and LFS throughput. Slurm's partition and allocation limits still apply. Preparation saves all three limits and `SLURM_STAGING_PARTITION` in the private Attempt journal. Both Git/LFS jobs use that saved intent, including stage-out created hours later. A configuration change affects new Attempts only. Recovery validates existing native job specifications against the saved staging intent and adopts uncertain submissions without changing their resources or submitting twice. Old journals retain the former 1 CPU, 512 MiB and 10-minute limits. Their partition comes from any recorded staging job; a legacy preparation with no native intent records the current partition once before submission. ### Site GPU classes and wall-time limit `SLURM_GPU_GRES` maps an accelerator class to the GRES the compute partition offers, as `CLASS=GRES` pairs separated by commas (`gh200=gpu:nvidia_gh200_144g_hbm3e`), and `SLURM_MAX_WALL_SECONDS` is the longest wall time one compute job may ask for. Empty or absent means the Site offers no GPUs and sets no limit; a malformed value stops the Runtime before it opens a journal. The installer writes both from `site.env` (`NATIVE_SLURM_GPU_GRES` and `NATIVE_SLURM_MAX_WALL_SECONDS`). The class must agree with the action's `gpuClass` and the allocation's `gpu.`; a mapped class becomes the compute job's `tres_per_node`, stage jobs never get a GPU, and a Run above the limit or with an unmapped class ends with its reason instead of retrying. New Attempts report `queued` during stage-in and while compute is pending. They report `running` once compute starts and throughout stage-out. The Runtime saves its first running observation before returning it; later native requeues retain `running`, as Execution Core does not permit a backwards transition. Old active journals may already have published running during stage-in, so recovery retains that conservative state. Untouched legacy preparations use the new reporting rule. ## Recovery Preparation first invokes the workspace owner without the Slurm member token, then saves the Attempt and all three `prepared` native-stage placeholders. The workspace owner must durably own its staging intent and idempotent recovery. Start records the `stage-in` intent and `submitting` phase before the POST. Repeated lookup and recovery search `slurmctld` and then `slurmdbd` by the exact name, comment, account and member. Several matches or an unavailable job collection stop progress. A missing job after an uncertain submission is not proof that the POST had no effect, so Runtime never resubmits it automatically. A lost response after a successful POST is adopted only after the exact native identity is visible. The returned job ID is saved before confirmation. If cancellation arrives before that ID is saved, Runtime searches by the same exact identity before answering. It does not infer absence from an uncertain submission. Observation checks each saved job ID, name, account, member, comment and state. A completed predecessor permits the next stage's durable submission; a restart after completed `stage-in` therefore resumes with `compute` without duplicating the clone. A failed or cancelled stage makes the whole native Attempt terminal. Cancellation records the whole-chain intent, stops submitted jobs from newest to oldest and never launches a later stage. It persists the exact `accepted`, `already-terminal` or `not-found` result before returning it and replays that result after restart. A 404 plus an empty accounting page is not treated as proven absence: absence remains deferred for Site-owned recovery. Repository access for each staging job is revoked after its successful execution or terminal failure. The release is recorded per stage so restart does not repeat it. Workspace cleanup after a terminal or cancelled Attempt is also durable; an unavailable owner defers completion rather than losing the cleanup intent. All requests use no proxy, no redirect and no administrator or mux fallback. Plain HTTP is accepted only for an explicitly enabled loopback `slurmrestd`. ## Packages and composition | Path | Responsibility | | --- | --- | | `contexts/site_runtime/application/slurm.py` | Slurm provider, credential, workspace, repository and native ports. | | `contexts/site_runtime/adaptors/slurm_chain.py` | Three-stage Attempt phases, recovery and cleanup. | | `contexts/site_runtime/adaptors/slurm_jobs.py` | Retained single-job baseline. | | `contexts/site_runtime/adaptors/slurm_job_model.py` | Restricted request/script rendering, identity and state checks. | | `contexts/site_runtime/adaptors/slurm_access.py` | OIDC JWT verification and `slurmrestd` HTTPS access. | | `services/slurm_runtime.py` | Worker composition and paired `delivery.sqlite`/`jobs.sqlite` installation. | Composition reads `RUNTIME_STATE_DIRECTORY`, `SITE_API_URL`, `SITE_API_CREDENTIAL_FILE`, `SITE_API_CA_FILE`, `SLURM_REST_URL`, `SLURM_REST_ALLOW_INSECURE_LOOPBACK`, `SLURM_PARTITION`, `SLURM_STAGING_PARTITION`, `SLURM_GPU_GRES`, `SLURM_MAX_WALL_SECONDS`, `SLURM_JWT_ISSUER`, `SLURM_JWT_AUTHORIZED_PARTY` and `SLURM_CA_FILE`; staging grants no `aud`, so there is no audience input. The native Site credential is a private `psb_` file for the separate Site API `/v1/site/` edge route, read afresh for every request and never retained in a durable message. Member bindings, current permission, credential delegation, repository grants and workspace staging arrive through explicit owner ports. Their production connections are not implemented in this change, so startup fails closed. ## Verification On 24 October 2026, the focused staged-chain lane passed 59 tests. They cover the three exact native jobs and partitions, `afterok` dependencies, restart between stage-in and compute, uncertain submission reconciliation without resubmission, cancellation during compute, repository release and workspace cleanup, durable rejection replay, credential retry and credential-free records. The retained single-job lane remains green. Ruff and strict mypy passed for the service. The earlier baseline recorded repository-wide `pnpm verify` and `pnpm test:integration` passes before the staged change. They will be rerun before hand-off. Real member consent and credential delegation, real GitLab staging, native cancellation recovery, UAT, Minikube/k3s and deployment were not run. Usage reconciliation remains outside this change. Mocked provider and native adaptors do not complete the live mixed-Pipeline lane.