# DD — Slurm run path on Icarus Status: Proposal for discussion. Icarus provides option B below, a Slurm partition with Internet access for data staging, and no container runtime (23 October 2026). The laboratory implements the fallback without external fetches or container images. Work item: [#165](https://github.com/pharos-ai-factory/pharos-platform/issues/165), run a repository action on Kubernetes or Slurm through Site Runtime. Feature scope: [ADR](adr.md) Task 6 (6B, 6C) and Task 4 for Slurm. Related documents: - [Slurm Site access](slurm-site-access.md) ([#163](https://github.com/pharos-ai-factory/pharos-platform/issues/163)): how each job runs with the member's credentials; - [Icarus HPC accounts](../project-provisioning/icarus-hpc-accounts.md) ([#164](https://github.com/pharos-ai-factory/pharos-platform/issues/164)): the accounts and associations a Run relies on. A Run executes a Workload at an exact repository commit. Repositories may hold very large Git LFS content, such as hundreds of gigabytes of model weights. A typical job reads the repository, changes part of it, then commits and pushes the result. **Network facts, to be confirmed by the Icarus administrators.** Icarus login nodes have SSH access and outbound internet; compute nodes have neither. GitLab and its LFS object storage (S3 on Ceph RGW) run in the same data centre as Icarus, and both are reachable from outside, because members push and pull from their own workstations. | Reader | What we ask | | --- | --- | | Icarus administrators | Choose option A or B; answer the storage questions | | Platform developers | Review, then implement under 6B, 6C and Task 4 | ## Summary - **Data moves directly** between Icarus and GitLab or S3 inside the data centre, and **only the delta returns**: Git LFS uploads only new objects. - **Code runs as the member.** Wherever the network allows, one Slurm job of the member does the whole Run: clone, LFS download, work, commit and push. - **Site Runtime orchestrates.** It submits and observes jobs with member tokens and issues Run-bound repository credentials. It never executes repository content. - **Icarus provides B:** a staging partition with Internet access, while compute nodes stay offline. Option A was the proposal. The fallback, where Runtime stages data itself, needs ACLs, which Icarus Lustre lacks, so it serves only the laboratory. ## Git LFS facts From the GitLab deployment role (`legacy/devops/ansible/roles/deploy_gitlab`): | Setting | Value | Consequence | | --- | --- | --- | | Object storage | Ceph RGW, bucket `git-lfs` | LFS content lives in the same data centre | | `proxy_download` | `false` | Clients download LFS objects **directly from S3** through pre-signed URLs from the LFS batch API | | `direct_upload` | `true` | Clients upload through GitLab (Workhorse), which writes to S3 | | `max_file_size` | 50 GiB | Each LFS file (for example a model shard) must stay below 50 GiB | Git LFS identifies objects by SHA-256, so a push uploads only objects the server lacks. It has no binary delta inside a file: a changed 5 GB shard uploads 5 GB, and a fine-tuning that rewrites every shard uploads all of them, while adapters upload little. The S3 endpoint that GitLab hands to clients must be reachable from wherever the download happens; this holds for workstations and must hold for Icarus. ## Data movement options | | What Icarus provides | Who moves data | Return path | | --- | --- | --- | --- | | **A. Outbound access from compute nodes** (proposed) | Egress TCP 443 from compute nodes to the GitLab and S3 hostnames only, by firewall rule or by a forward proxy on the login node | The member's job | `git push` from the job | | **B. Transfer partition** | A Slurm partition on a node with outbound access (the login node or a data-transfer node) | Stage-in and stage-out jobs of the member | `git push` from the stage-out job | | **Fallback: Runtime staging** | Nothing beyond ACLs on scratch | Runtime, as `pharos-runtime` | Runtime reads one exported archive and uploads it | ### A. Outbound access from compute nodes (proposed) Text for the Icarus administrators: > From the Icarus compute-node subnet, allow **outbound** TCP 443 only to > `gitlab.`, to the S3 endpoint that GitLab uses for LFS (`s3.`) and, > for Workloads with container images, to the existing image registry > (`registry.`), at their data-centre addresses. All other outbound traffic > stays closed, and no inbound connection is opened. If compute nodes cannot be routed, a forward proxy on the login node (for example Squid) with the same allowlist works: jobs set `https_proxy`, which Git and Git LFS honour, provided compute nodes reach the login node internally. One job of the member does the whole Run. It clones the exact commit, downloads LFS objects straight from S3, works in its own directory, commits on a Run branch and pushes only the new objects. There is no staging, no ACL handover and no Runtime quota. User code can reach only these Pharos services, and all require authentication. ### B. Transfer partition Runtime submits three jobs of the member with the same token: stage-in on the transfer partition (clone, LFS download), the compute job (offline, `afterok`), and stage-out on the transfer partition (commit, push, `afterok`). Every file belongs to the member from the start. Icarus must accept jobs on the login node or provide a data-transfer node, with small limits on CPU, memory and `MaxTime`. ### Fallback: Runtime staging For small repositories at a Site without A or B. Icarus provides B, and its Lustre has no ACLs, so this path is not available there: 1. Runtime creates `/scratch/pharos-stage//{in,out}` with mode `0700`, clones the exact commit with LFS into `in/` and grants the member read access by ACL. It never executes repository content: no hooks (`core.hooksPath=/dev/null`), no submodules, `protocol.file.allow=never`, `lfs.url` fixed to GitLab (see Credentials), and a size limit on LFS objects. 2. The job copies the input into its working directory and writes `outputs.tar` there. It grants `pharos-runtime` read access to that file and execute-only access (`u:pharos-runtime:--x`) to each directory above it, so Runtime can open that one known path but list nothing. 3. Runtime opens it with `O_NOFOLLOW|O_NONBLOCK` and accepts it only if it is a regular file owned by the member's UID, with `st_nlink == 1`. The member can still append to their file, so Runtime enforces the size limit while copying it through a byte-limited reader into a file of its own, hashing the copy. It uploads that copy without extracting it, then removes the Attempt directory. The ACL entries name one user: other members gain nothing, and the member's directories stay `0700` for everyone else. The ownership and link checks stop a member from presenting another member's file as their own output. Staged input counts against the `pharos-runtime` quota, so Runtime limits its size and deletes it early. ### Optional per-member LFS cache The job may set `lfs.storage` to a directory in the member's own scratch space, so a later Run of the same member on the same repository downloads only missing objects. The cache counts against the member's quota. A cache shared per Project is not proposed. ## Stages and commands Every Site command is numbered, durable and acknowledged with its recorded outcome (Task 4). `prepare` stages data for the fallback and does nothing for A and B. `start`, `observe` and `usage` call `slurmrestd` with a fresh member token each time; with B, `start` submits the three dependent jobs. `stage-out` is a command before `usage` for the fallback. Each step finds before it acts. `start` looks up the job by a name derived from the Attempt before submitting, after recording a native submission intent, so an uncertain response is never retried blindly. Recovery and late observation read slurmdbd, which stores the Attempt in the job comment (`AccountingStoreFlags=job_comment`); an uncertain submission with no native record needs operator investigation. Stage-out keeps a recovery receipt before cleaning up, and the control plane accepts an export only for the current stage-out command. Icarus has no Apptainer or other container runtime (23 October 2026): jobs use the packages installed on its nodes. Workloads that name a container image are therefore not offered on Icarus. ## A Run, end to end (option A) ```mermaid sequenceDiagram autonumber actor U as Member participant APP as Pharos web and Client API participant CP as Control plane and Site API participant GL as gitlab-adaptor participant SR as pharos-runtime (login node) participant RD as slurmrestd participant J as Member job (compute node) participant G as GitLab and S3 U->>APP: Run Workload at commit X APP-->>U: 202 Accepted SR->>CP: Pull "start", which carries identities only SR->>CP: Request the credentials for this command CP->>GL: Run-bound repository credential GL-->>CP: Token CP-->>SR: Repository token and a fresh member token, in this response only SR->>RD: Submit with X-SLURM-USER-TOKEN, account pharos_a_#lt;allocation#gt; RD->>J: Job runs as the member J->>G: Clone commit X, download LFS objects from S3 J->>J: Work in the member's directory J->>G: Commit on branch pharos/runs/#lt;run#gt;, push new objects only loop Until the job ends SR->>RD: Observe with a fresh member token end SR->>CP: Acknowledge with state and usage CP->>GL: Revoke the Run-bound credential ``` ## Credentials and the Runtime host - **Repository credential.** `gitlab-adaptor` issues a Run-bound token for the Workload's repository: `read_repository`, plus `write_repository` when the Workload pushes. GitLab cannot limit a token to one branch, so until the Run ends, Workload code can push to any unprotected branch or tag of that repository. We accept this: it never exceeds the member's own developer access, and protected branches and tags stay closed. The Run pushes to `pharos/runs/`. Token expiry has day granularity, so the token is revoked explicitly when the Run ends. This revises the #165 criterion of a read-only token revoked after the fetch, for Workloads that push. - **Credential scope.** Git LFS honours a repository's `.lfsconfig`, which could point the LFS batch API, and with it the credential, at another host. The job therefore sets `lfs.url` to the GitLab LFS endpoint in the repository's own Git configuration, which takes precedence, and the askpass helper answers only for the GitLab host. - **Where it goes.** With A and B, the token reaches the member's job as a `0600` file in the member's directory. With the fallback, only `pharos-runtime` holds it. It is never a member's personal access token, and the Slurm JWT never enters a job. - **Runtime's own configuration.** The systemd unit runs as `pharos-runtime` with an `EnvironmentFile` owned by it with mode `0600` (`/etc/pharos-icarus-runtime.env` in the laboratory): the Site identifier, the Site credential, the Site API address and the recovery state path. It holds no member token and no administrator credential. - **Unit hardening** (laboratory): `NoNewPrivileges`, an empty `CapabilityBoundingSet`, `ProtectSystem=strict`, `ProtectHome`, `PrivateTmp`, `PrivateDevices`, `RestrictSUIDSGID`, `ProtectKernelTunables`, `ProtectKernelModules`, `ProtectKernelLogs`, `ProtectControlGroups`, `ProtectClock`, `ProtectHostname`, `RestrictNamespaces`, `RestrictRealtime`, `LockPersonality`, `RestrictAddressFamilies=AF_INET AF_INET6 AF_UNIX`, `SystemCallArchitectures=native`, and write access only to staging and its state. ## Files, storage and member access - **Working directory.** `/scratch/pharos///`, below the shared [`/scratch/pharos`](../project-provisioning/icarus-hpc-accounts.md#home-and-scratch-directories). The member root `/scratch/pharos/` is trusted: the site's root `Prolog` creates it with `mkdir` when missing — never with `install -d` or another call that would follow a symlink inserted after the guard — and only then hands it to the member with `chown`, because the sticky `1770` parent lets nobody else replace the entry it just made. An entry that already exists is accepted only as a real directory owned by the member with mode `0700`; anything else is left untouched. The job never creates or reuses that root; it checks that it is a real directory (not a symlink) owned by the member with mode `0700`, and exits before touching anything that fails the check. It then creates the attempt directory with `mkdir -m 700` **without `-p`**, so an existing path is refused rather than accepted. The attempt directory is owned by the member, mode `0700`, and counted against the member's quota. Another member cannot create the victim's predictable path under the `1770` parent, and cannot traverse the trusted root even if they know the path. - **Sharing with the Project.** A Run never writes below a path shared with other members: its working directory is the private `/scratch/pharos///` above. Project-shared scratch is a separate, site-owned path, `/scratch/pharos/shared/` (`root:`, mode `3770`), which the member uses by hand and no job creates or requires; the mode check above covers only the trusted member root, so the two contracts do not conflict. Icarus has not created that path yet, so it stays deferred ([accounts](../project-provisioning/icarus-hpc-accounts.md#home-and-scratch-directories)). - **Members with SSH.** If Icarus grants interactive login to Pharos accounts, a member signs in with the same account their jobs use, sees their working directories, and sees and may cancel their jobs with `squeue --me`, `sacct` and `scancel`. Pharos treats a native cancellation as such. A local Icarus account is a different UID and cannot read these files. - **Retention.** Working directories remain until a cleanup policy or the Icarus scratch purge, 3 months after the Project ends, removes them. The pushed commits and uploaded results are the durable copy. - **Storage at Icarus** (23 October 2026): Lustre, mounted on login, compute and staging nodes, without ACLs; HOME 25 GB per member; one shared `/scratch/pharos` with a Pharos quota. With B, `pharos-runtime` stages nothing and needs no quota. ## Limits and usage - Slurm enforces the Project's limits through `pharos_a_` and the `pharos` ceiling. Runtime translates the authorised Project into `job.account`; the member sees the backend's reason when a job is rejected. - Usage is reported per Project from Slurm accounting, for successful and failed jobs, against the existing allocation. No per-Attempt Waldur resources are created. - Status, logs and outputs reach the member through the control plane, subject to current access checks. ## Metrics while a job runs Compute nodes cannot reach experiment tracking unless Icarus adds it to the allowlist. For the review, the job writes metrics to a local `file:` store, which is pushed or exported with the results. Live metrics need the tracking host in the allowlist or a login-node proxy. ## Trust | If compromised | Can | Cannot | | --- | --- | --- | | `pharos-runtime` | Act as members whose tokens pass through it, for minutes; use the current Runs' repository credentials; read files granted to it | Act without a token; write LDAP or Slurm accounting; read members' homes or working directories | | A member's job | Use its own Run-bound credential, including pushes to unprotected branches and tags of its repository; reach only the allowlisted hosts (A) | Read other members' files; push to protected branches; obtain the member's Slurm JWT | ## Verification | # | Check | Status | | --- | --- | --- | | A1 | A job clones a large LFS repository, changes one object and pushes; only the new object is uploaded | Not tested | | A2 | With the allowlist, the job reaches GitLab, S3 and the registry and nothing else | Not tested | | A3 | A repository `.lfsconfig` pointing elsewhere receives no request and no credential | Not tested | | B1 | Stage-in, compute and stage-out run as the member with `afterok` dependencies | Not tested | | F1 | Fallback: input readable only by its member; output accepted only as a regular file owned by the member with one link | The laboratory passes the equivalent with a runtime-owned output file; the member-owned variant is not tested | | F2 | Another member cannot read a known output path | Denied peer access passed in the laboratory audit | | F3 | A second UID pre-creates `/scratch/pharos/` (directory or symlink) before the victim's first job; the job refuses the root and writes nothing | Not tested; the two-UID reproduction of 29 October accepted the squatted path before this contract | | F4 | An existing `/scratch/pharos//` is not reused: the job exits without writing into it | Not tested | | F5 | The trusted root is a real directory owned by the member with mode `0700`; a root with another owner, mode or a symlink parent is refused | Not tested | | F6 | A second UID races the Prolog's check with a symlink for `/scratch/pharos/`; the Prolog creates and `chown`s nothing through the link, the target keeps its owner and mode, and the job refuses the root | Not tested; the two-UID reproduction of 29 October changed an unrelated directory's owner and mode with the `install -d` recipe | | F7 | Once Icarus creates the sharing path, a member of the allocation group writes under `/scratch/pharos/shared/` and a peer outside the group neither traverses nor reads it; a Run still refuses a trusted root that is not `0700`, so the shared path is never used as one | Deferred with the path; not tested | | R1 | Repeated `start` and lost responses create no duplicate job; recovery uses accounting | Passed in the laboratory tests | | R2 | Usage per Project for successful and failed jobs | Successful jobs audited in the laboratory; failed jobs not tested | The laboratory lanes cover two members in two Projects, permission withdrawal, native accounting, denied peer and foreign-Project access, and credential-free logs. The laboratory currently has Runtime pre-create the output file; the fallback above moves that file into the member's directory so its size counts against the member. ## Decisions needed | # | Decision | Proposal | Affects | | --- | --- | --- | --- | | 1 | How compute nodes reach repository data | B, as Icarus provides: stage-in and stage-out jobs on the staging partition | #165, Icarus | | 2 | Where results go | Commits on `pharos/runs/` through `git push`; small results as Run artefacts. The write token also reaches other unprotected branches; accepted, revising the #165 criterion | #165 | | 3 | Per-member LFS cache | Optional, in the member's scratch space | #165 | | 4 | Metrics during a job | Local store pushed with the results; live metrics if Icarus allowlists tracking | #165, Icarus | **Requests to the Icarus administrators:** answered on 23 October 2026: B, the storage questions, `pharos-runtime` without `sudo`, and no container runtime. Still needed: the staging partition's name and limits, and its egress to GitLab and S3. **Platform work:** a Slurm connector over `slurmrestd` with the member token; the stage-in and stage-out jobs on the staging partition; Run-bound repository credentials from `gitlab-adaptor` with explicit revocation; jobs found by name before `start` submits; and the member carried in every command that starts or cancels work. ## Appendix — sketches ```bash # Option A: the member's job. TOKEN_FILE is the Run-bound credential (0600). set -eu; umask 077 ROOT=/scratch/pharos/$USER # The site's root Prolog creates ROOT once. Never accept a squatted or linked path. [ -d "$ROOT" ] && [ ! -L "$ROOT" ] && [ "$(stat -c %u -- "$ROOT")" = "$(id -u)" ] \ && [ "$(stat -c %a -- "$ROOT")" = 700 ] || { echo "untrusted scratch root" >&2; exit 70; } WORK=$ROOT/$ATTEMPT mkdir -m 700 -- "$WORK" || { echo "attempt directory already exists" >&2; exit 70; } cd "$WORK" export GIT_ASKPASS=$PHAROS_ASKPASS # reads TOKEN_FILE, only for the GitLab host git init -q repo && cd repo && git config core.hooksPath /dev/null git config lfs.url "$REPO_URL/info/lfs" # overrides any .lfsconfig in the repository git remote add origin "$REPO_URL" && git fetch --depth 1 origin "$COMMIT" GIT_LFS_SKIP_SMUDGE=1 git checkout --detach "$COMMIT" git lfs pull # objects come straight from S3 "$@" # the Workload command git switch -c "pharos/runs/$RUN" && git add -A if ! git diff --cached --quiet; then # a Run without changes pushes nothing git -c user.name="$PHAROS_AUTHOR_NAME" -c user.email="$PHAROS_AUTHOR_EMAIL" commit -q -m "Run $RUN" git push origin "pharos/runs/$RUN" # uploads only new LFS objects fi ``` ```bash # Submit as the member through slurmrestd (data parser version supported by Icarus). curl -s -H "X-SLURM-USER-TOKEN: $T" -H 'Content-Type: application/json' \ "http://localhost:6820/slurm/v0.0.42/job/submit" -d '{ "script": "#!/bin/bash\nid; hostname", "job": {"name": "pharos-probe", "account": "pharos_a_", "current_working_directory": "/scratch/pharos/", "environment": ["PATH=/usr/bin:/bin"]}}' | jq '.job_id, .errors' ``` With a username claim configured, `slurmrestd` may not need `X-SLURM-USER-NAME`; check the Icarus version. The laboratory sends both.