# Native Slurm Run logs This is a separately reviewed Site activation procedure. The normal installer does not change Slurm, directory identities, POSIX allocation or filesystem retention policy. ## Authority and storage Execution Core issues a sixty-second, current-viewer permit for one saved Run Attempt. Runtime rechecks the permit before reading and before uploading. The upload route and the browser read also check current authority. Removing a viewer's Project permission prevents another read, including a cached upload. The initiating member's departure does not by itself remove another current Project reader's `runs.output.view` permission. Runtime reads only a dedicated spool. It gains no sudo, ACL, LDAP, member-home or private-scratch permission. A dedicated supplementary group contains only the Runtime service account. It must be outside the member and Project GID ranges. Each directory is root-owned, group-owned by that reader group and mode `0751`. Each active output file is owned by its job's numeric member UID, group-owned by the reader group and mode `0640`. Members can open their own file but cannot create, rename or replace directory entries. Other members cannot read it. The directory hierarchy is: ```text ////-.log ``` The companion `.json` is a root-owned `0640` receipt. It records the exact Run, Attempt, material, stage, job ID, username, account and original numeric UID. Slurm accounting does not always retain the numeric UID; this receipt supplies it without resolving a potentially recreated directory account. The root helper cross-checks the native job's actual combined stdout/stderr path, name, material comment, account and user before preparing or retaining anything. Runtime cross-checks the same fields against its saved submission. Every path component is opened with no-follow descriptors. File types, owner, group, mode and single hard-link count are checked; changes fail closed. Existing legacy Runs without the saved spool configuration and receipt remain unavailable: after the reader's permission is checked, Runtime answers them with one fixed line saying the Run started before this Site collected Run logs, and reads nothing. After the scheduler confirms the exact job is terminal, root seals its output by changing the owner to root. The original numeric UID remains in the receipt. Runtime requires a sealed root-owned file for a terminal job. This removes the original member's direct filesystem access to retained output. ## Required identity-retention safeguard Before activation, the Site must enforce that a numeric UID cannot be allocated to another identity while its active or unsealed output exists. A scheduler Epilog and a periodic seal pass reduce the completion window; neither alone prevents a concurrent UID release and reassignment. The current Waldur POSIX implementation supports withheld identity rows through `recyclable=False`, but normal account deletion can release recyclable values. The Site operator must verify and install an owner-side reservation or ordered identity-removal safeguard before enabling this spool. A written convention or the offering user's currently pinned UID is insufficient. If that safeguard is absent, leave both Runtime spool variables and central log dispatch disabled. The opt-in Waldur pool safeguard is available in [PR #1162](https://github.com/pharos-ai-factory/pharos-platform/pull/1162). It reserves released UIDs and protects their ledger rows. Its provider guide requires the exact offering-to-pool check and a complete patched API, worker and beat rollout before this spool can be enabled. ## Reviewed installation steps `sudo ./pharos-hpc run-logs` performs steps 2, 3, 5 and 6 below from `site.env` (`NATIVE_RUN_LOG_*`) and prints the Prolog and Epilog lines for step 4 and the order of steps 7 and 8. Step 1 and the identity retention above remain the Site operator's decisions. 1. Choose the shared spool path, exclusive reader group/GID, approved member UID range and retention interval. Verify the filesystem's no-follow and POSIX permission behaviour from every compute and Runtime host. Ancestors of the spool must be root-owned and not group/other writable. 2. Install `run_log_spool.py` from the reviewed release to a root-owned, non-writable-by-Runtime executable location. Create the exclusive reader group and add only `pharos-runtime`; the normal service primary group stays unchanged. 3. Run `initialise` as root. It creates only the final root directory and checks existing ancestors. It does not repair unexpected permissions. 4. Extend the Site's existing root Slurm Prolog for Pharos jobs to call `prepare`. Supply the exact native job ID, installed `scontrol` and `sacct` executables, root, group and approved UID bounds. Preserve other Site Prolog work. A rejected preparation must prevent that job from starting; follow the Site's established failure handling rather than weakening ownership checks. 5. Install a root finalisation hook and periodic catch-up pass using `seal`. Slurm may still report `COMPLETING` during Epilog, so the catch-up pass is required. Only confirmed terminal jobs are sealed. 6. Install a root timer for `sweep`, with the chosen positive retention seconds. It first seals confirmed terminal files and deletes exact log/receipt pairs only after the retention interval measured from scheduler completion. Missing accounting, identity differences and unsafe permissions leave the files for operator investigation. Empty canonical directories may remain. 7. Configure Runtime `SLURM_LOG_SPOOL` and `SLURM_LOG_GROUP_ID` together, then restart the reviewed Runtime with its journals and credentials preserved. New Attempts save these settings; changing them does not relocate old output. 8. After Runtime and the Site API are upgraded, enable central `RUN_LOG_READ_ENVIRONMENTS=slurm` (or `k8s,slurm` after both are qualified) on the request host and execution worker. The default is disabled. Older Runtimes cannot decode read-log commands, so upgrade before enabling issuance. Example root invocation, with Site-approved values supplied by the operator: ```sh /usr/local/libexec/pharos-run-log-spool initialise \ --root /var/lib/pharos-run-logs --group pharos-log-reader \ --member-uid-min 5000000 --member-uid-max 5499999 /usr/local/libexec/pharos-run-log-spool prepare \ --root /var/lib/pharos-run-logs --group pharos-log-reader \ --member-uid-min 5000000 --member-uid-max 5499999 \ --job-id "$SLURM_JOB_ID" \ --scontrol /opt/pharos-icarus-hpc/slurm/bin/scontrol \ --sacct /opt/pharos-icarus-hpc/slurm/bin/sacct /usr/local/libexec/pharos-run-log-spool sweep \ --root /var/lib/pharos-run-logs --group pharos-log-reader \ --member-uid-min 5000000 --member-uid-max 5499999 \ --retention-seconds 604800 \ --scontrol /opt/pharos-icarus-hpc/slurm/bin/scontrol \ --sacct /opt/pharos-icarus-hpc/slurm/bin/sacct ``` Seven days above is the CPU qualification setting, not a production default. Set `SLURM_CONF` where the installed clients require it. Retention commands hold no Runtime or central platform credential. ## Qualification Use owned bounded CPU jobs with timestamped stdout and stderr, successful and failed completion, and enough output to exercise truncation. Confirm current Project readers can fetch both terminal and active logs, unrelated members cannot, revocation clears previously rendered text, and a replay preserves the same upload bytes. Verify another numeric Unix identity cannot read active files and the original UID cannot read sealed files. Before rollout, run the dedicated root-only disposable filesystem test `services/metallo/tests/infrastructure/slurm_log_spool.py`; it creates and removes only its unique directory below `/var/lib`. It never edits users, groups, Slurm or shared spool state. Central PostgreSQL and contract tests are separate from this permission test. A live M18-23 result requires both source qualification and the installed browser journey.