# Icarus-hpc laboratory This laboratory turns a [Metallo dev box](../../common/README.md) host into the Icarus-hpc Site of [DD-0055](../../../docs/dd/0055-hpc-ssh-key-provisioning.md) section 7: OpenLDAP, SSSD and sshd, native Slurm, the upstream Waldur Site agent and the [SSH access agent](../../../services/metallo/deploy/ssh-access-agent/README.md). Everything installs under `/opt/pharos-icarus-hpc` and as systemd services on the host; the platform's namespaces and data are left as they are. The laboratory is a qualification fixture, not the production topology. In production, Icarus-cloud owns the Pharos platform, Kubernetes Runtime, Site API and GitLab/LFS endpoint, while Icarus-hpc is the separately administered standalone Slurm cluster. The Slurm Runtime and Pharos agents run on Icarus-hpc and connect outbound over HTTPS; Icarus-hpc needs no platform database, Kubernetes workload or inbound platform connection. The laboratory co-locates disposable cloud services on one dev box so that it can exercise the same native Git/LFS and JWT paths. ## Install The dev box must already run the `waldur` and `pharos-site` profiles, so that `waldur.pharos` and `site.pharos` are served through the edge on port 8443, and the workstation needs the browser connection that `make deploy` starts for `waldur.localtest.me` (`make devbox-forward` opens it again). The controller fetches the pinned Slurm source, because dev box hosts may not reach SchedMD; the access agent gets its own pinned CPython 3.13, since Debian 12 ships 3.11. Select the dedicated credentials described in the [development guide](../../common/GUIDE.md#dedicated-ssh-credentials) when the controller runs in the tools environment. ```sh export METALLO_SSH_DIR=/absolute/path/to/dedicated/ssh make metallo-icarus TARGET=dev- SSH= ``` `up.py` runs these steps, each idempotent: 1. Builds the Metallo wheel, the Site agent's account-name plugin (`services/hpc-site-agent-plugin`) and Metallo's hash-pinned requirements from this checkout. 2. Signs alice and bob in to Waldur through the local AAI with `dx/common/scripts/service-login.sh`. Waldur identifies each by the AAI subject and stores their `site_username` claim (`pharosalice`, `pharosbob`). 3. Bootstraps Waldur through `waldur shell` in the mastermind pod (`waldur_bootstrap.py`): the provider and its POSIX ID pools (`5000000–5499999` for users and `5500000–5549999` for their primary groups; the agent gives its project groups `5550000–5599999`, so the two allocators never share a range), the **HPC Icarus** Slurm offering, the existing **Pharos** organisation selected by the deployed Project owner's organisation UUID, **Project HPC** (alice, bob) and **Project HPC qualification** (alice) with their orders, and the non-staff `pharos-icarus-site-agent` as offering manager **and as owner of the provider organisation**. Ownership is not administrative convenience: Waldur 8.1.3 rejects a usage report with a date from anyone who holds no role on the provider, and the agent always sends one, so reports failed with HTTP 400 until it owned the provider. That organisation serves this single offering, as the Cloud Site Agent owns its own. The agent stays a coordinator of `pharos` in Slurm and never becomes Waldur staff. Existing `70001–70999` laboratory identities are moved once into the approved ranges. The migration record survives a retry so the directory is repaired even after Waldur has already committed the new IDs. Staff authority stays in Waldur's process; only the agent's token returns. Waldur allocates an identity when it creates an offering user, so an offering user that already exists without one — `make waldur-seed` ordering `hpc-icarus` before this step — receives an identity from the pool here, and a rerun adds nothing. The rest of the approved group range, `5550000–5599999`, belongs to the Site agent, which gives each project group its GID there; the two allocators never share a range, and the directory keeps one `^55[0-9]{5}$` constraint over both. 4. Issues native Site grants for the access agent (`ssh-access@hpc`) and the native Runtime (`commands@hpc`, environment `slurm`) and applies the digest-only `site-api-grants` Secret. 5. Installs the AAI certificate authority and the current signing keys (JWKS) that the Runtime verifies member tokens with. No member token is captured here: a member's offline consent arrives through a real login and is sealed in Metallo's Member Credentials context, which refreshes and revokes it, and the Runtime asks the Site API for one access token per native Slurm operation. 6. Runs the Ansible playbook in `ansible/` against the host, which installs the account-name plugin into the pinned upstream Site agent. The offering's policy is `service_provider`: Metallo — not Waldur and not the agent — writes each offering user's username from the Party's `site` binding, the agent provisions exactly that name, and it pins the UID the directory gave on the offering user (`set_posix_attributes`). The directory's UID/GID ranges are the Waldur POSIX pool's, so a pinned UID is always inside the pool. Before the Site agent starts, Ansible applies only Waldur's recorded legacy renumbering to the named users, personal groups and `pharos-members`; memberships and SSH keys are preserved. 7. Waits until Waldur membership has produced the `pharosalice` and `pharosbob` directory accounts. An offering user whose name Metallo has not written yet stays `ERROR_CREATING` with the reason in the agent's log, and the next cycle picks it up: no account is ever created under a generated name. Credentials stay in the ignored `.local/icarus-hpc//private`, mode 0700, and reach the host only as private files. None is printed or passed in argv. The bootstrap reuses the seeded Project HPC and preserves existing roles. It uses the seed's `project-hpc` allocation name, so running `make seed` after the bootstrap reuses the allocation or its pending order. It adds alice for the two-Project qualification journey. It does not create a second consumer organisation or change Project A, Project B or Project Cloud. `up.py --settings ` accepts a JSON `members` mapping from dedicated HPC Project names to Waldur usernames; the organisation always comes from `beto-project-source`. The three general seeded Project names are rejected. Qualification uses the recorded organisation, offering and Project UUIDs from `waldur.json`, including custom names, and resolves actual allocation backend IDs. An older laboratory's `Pharos research` data remains for separate administrative review and cleanup. ## Site layout | Concern | Owner | Notes | | --- | --- | --- | | Accounts, POSIX attributes, groups under `ou=pharos` | `pharos-agent` (upstream Site agent) | Binds as `cn=pharos-agent,ou=services` | | `sshPublicKey` values under `ou=people,ou=pharos` | `pharos-ssh-access` (access agent) | SASL EXTERNAL over `ldapi`, mapped to `cn=pharos-ssh-access,ou=services` | | Reads for NSS and sshd | SSSD | `cn=sssd-reader,ou=services`, read-only | | Slurm associations under `pharos` | `pharos-agent` | Slurm 24.11.5, built from the checksummed source | | `pharos_default`, members' `DefaultAccount` | the host | A leaf under `pharos` with `MaxSubmitJobs=0` | | Local accounts (`icaruslocal`) | the host | Keep `authorized_keys`; Pharos never writes them | The directory listens only on `ldap://127.0.0.1/` and `ldapi:///`, matching the Icarus-hpc decision that there is no LDAPS endpoint. SSSD and the upstream Site agent bind over loopback LDAP; the access agent writes over `ldapi` using its Unix identity without a password. The directory administrator password is removed after configuration. Debian's startup-only TLS attributes remain harmless in `cn=config`, with no TLS listener bound. The access agent's Site bearer comes from `LoadCredential=`. sshd reads keys for the `pharos-members` group only from `sss_ssh_authorizedkeys`. The drop-in is validated with `sshd -t` and removed if validation fails. SSSD caches entries and keys for 90 seconds (`icarus_hpc_cache_seconds`), inside the 120-second window accepted by the Icarus-hpc administrators. The remaining 30 seconds cover agent polling and acknowledgements; the e2e lane fails if the end-to-end submit or revoke journey exceeds the accepted window. ### Node-hour allocations and usage An allocation is bought in **node-hours**, following the CSCS Alps model for the GH200 partition `gh200`: each node has 144 Grace cores, 2 GPUs and about 960 GB of memory, and the call limits a job to 8 nodes, 72 hours and 72 node-hours through the `standard` QoS. The offering's `node` component is limit-billed with period `total` and `unit_factor` 60, so an order of *n* node-hours becomes `GrpTRESMins=node=60n` on the member's allocation account and the limit is the order's whole lifetime, not a monthly allowance. The allocation has an end date and unused hours are lost; more hours are a new order, never a raised limit. Slurm's TRES `node` counts each allocated node as one, so a job on two nodes consumes two node-hours per hour, and a job that uses part of a node is charged the whole node, as in the Alps call. The value is always integral, so no TRES weights or `billing` are needed; `billing` would truncate the decimals (`job_mgr.c:9480` stores a double in a `uint64_t`). `waldur_bootstrap.py` renames the `cpu` component an earlier installation made and updates it to this contract with `update_or_create`; a repeated bootstrap leaves the same record. The Site agent runs `report` alongside `order_process` and `membership_sync`, every minute rather than the upstream default of 30, so Waldur sees usage while a job is still running. Slurm keeps that usage without returning it: `slurm.conf` sets `PriorityDecayHalfLife=0` with `PriorityUsageResetPeriod=NONE`, and `PriorityCalcPeriod=1` charges a running job each minute. The Site agent gives every member the `DefaultAccount` `pharos_default`. That account is a leaf, so its `MaxSubmitJobs=0` limits only its own user associations, and `AccountingStorageEnforce=limits` refuses jobs there. A member therefore submits with `sbatch -A pharos_a_`; a job without `-A` is refused instead of charged to an arbitrary allocation, and a member who left every Project keeps only that association. A limit on the root `pharos` would pass to every allocation below it. ## End-to-end lane This lane uses the API's local operator credential and reads its member identity from the development owner fixture. Its recorded results qualify Site behaviour with those fixture inputs. For canonical owner installations, follow the [real member qualification](../../../gitops/manifests/pharos/base/ssh-access/README.md#real-member-qualification) with an AAI-authenticated member, Beto ownership and retained AUP acceptance. ```sh METALLO_SSH_DIR=/absolute/path/to/dedicated/ssh \ python3 dx/metallo/icarus-hpc/e2e.py --target dev- --ssh ``` `e2e.py` runs the DD-0055 section 8 scenarios on the host through `e2e_site.py`: membership to directory account, a key submitted before and after provisioning, `ssh`, an access agent restart, revocation, termination of an allocation (its account stays without users, `sbatch` on it is refused), and leaving one and then both Projects (login stays, `sbatch` is refused, also without `-A`). The API's CLI credential and the test keys stay on the host; credential-free evidence goes to `.local/icarus-hpc//evidence`. A run takes about 20 minutes and leaves the member in both Projects without keys; it removes the terminated allocation's empty account as the site's administrator would. On `dev-ikons` (23 October 2026) all eight scenarios passed. Over four runs a submitted key reached a working `ssh` login in 55–64 s and a revoked key was refused in 12–32 s, within the 120-second propagation window; the final run measured 60.4 s and 22.1 s respectively. The API reported the key `deployed` or `revoked` once the window had passed. ## Slurm Runs from the platform The laboratory runs the Slurm Runtime that the HPC installer runs on Icarus, `python -m metallo.services.slurm_runtime` with its production owners, as the `pharos-slurm-runtime` unit. Accepted work, the member's binding, repository grants and each member token come from the dev box's Site API, so a `slurm-job` action that a member launches from the platform reaches it, for any repository. The host gets the site's side of the contract: - `/scratch/pharos` is `root:pharos-users` mode `1770`, and `pharos-runtime` and the members are in `pharos-users`. The Runtime writes only its `.runtime` box there. `up.py` adds the bootstrap members once the Site agent has created their accounts. A member added later joins with `sudo gpasswd -a pharos-users`. - A root Prolog, `/etc/slurm/pharos-prolog.sh`, creates `/scratch/pharos/` as the member's own `0700` directory before their first job runs. - The compute partition allows 72 hours, the `standard` QoS limit on Icarus; a `slurm-job` asks for 24 hours unless its action sets a timeout. - `REPOSITORY_CA_FILE` holds the development CA, since the dev box GitLab answers through the same edge. The mixed Pipeline lane below runs its own Runtime with laboratory owner doubles, and it refuses to start while this unit runs: stop it first with `sudo systemctl stop pharos-slurm-runtime`, and start it again afterwards. ## Mixed Slurm-to-Kubernetes Pipeline After the laboratory installation, prepare the dedicated SSH credentials described in the [development guide](../../common/GUIDE.md#dedicated-ssh-credentials) and run: ```sh make metallo-pipeline-mixed-test TARGET=dev- SSH= ``` `SSH` selects the lane's SSH target, which it needs for the HPC host's port-forwards. Without it the lane builds its namespace and then stops. The lane's defaults name the original two-VM laboratory. On a dev box that is its own HPC host, export these settings for the run and for `make metallo-pipeline-clean`: | Setting | Value on such a dev box | | --- | --- | | `METALLO_MIXED_HANDOFF` | a private directory holding the local AAI's CA as `aai-ca.crt` | | `METALLO_MIXED_CLOUD_IP`, `METALLO_MIXED_HPC_IP` | the host's address, for both | | `METALLO_MIXED_CLOUD_NAME`, `METALLO_MIXED_HPC_NAME` | the host's name | | `METALLO_MIXED_HPC_UUID` | the host's `/sys/class/dmi/id/product_uuid` | | `METALLO_MIXED_HPC_PYTHON` | `/opt/pharos-ssh-access-agent/python/bin/python3.13` | | `METALLO_MIXED_HPC_JUMP` | empty: no jump host | | `METALLO_MIXED_HPC_KNOWN_HOSTS`, `METALLO_MIXED_HPC_IDENTITY` | `known_hosts` and `id_ed25519` of the dedicated SSH credentials | | `METALLO_MIXED_SUBJECT` | the member's Site username, such as `pharosalice` | | `METALLO_MIXED_ACCOUNT` | that member's Slurm account in `sacctmgr show assoc` | The disposable Pipeline has two ordered steps. The first is a native Icarus Slurm job that writes `icarus-handoff.bin` and its SHA-256 file, then stage-out commits and pushes both through Git LFS. The second is a Kubernetes batch job that clones the same branch, pulls LFS, and fails unless the exact payload and checksum match. The controller reaches the disposable Site API and GitLab bridge through private loopback port-forwards on the Icarus host. It records credential-free evidence in that target's `pipeline` evidence directory and removes host tokens, temporary host aliases, port-forwards and journal state after the run. Keep a failed owned installation for inspection with `METALLO_KEEP_PIPELINE=1`; inspect or clean it with the printed `metallo-pipeline-inspect` or `metallo-pipeline-clean` command. ### Member tokens and the coordinator The Runtime never holds a member's refresh token. The member's own login writes the consent into the consent store, and the Runtime asks the Site API for one access token per native action through the provider configuration key `siteApi`: ```json {"siteApi": {"url": "https://site-api.example", "credentialFile": "/opt/pharos-slurm-runtime/private/site-token", "caFile": "/opt/pharos-slurm-runtime/private/site-ca.crt"}} ``` Submission carries the member's own token, so the job runs as the member. Native monitoring, accounting and cancellation use the Runtime's own Slurm identity. The role creates `pharos-runtime` with `MaxSubmitJobs=0` and `add coordinator account=pharos names=pharos-runtime`, and the unit puts the laboratory Slurm client on its `PATH`, so those three actions run as that Unix user through munge. There is no coordinator token anywhere: the Runtime defers monitoring rather than falling back to a member token when the client or the grant is missing. ### Capturing a member consent `member_consent.py` performs a real login of the laboratory member at the local AAI: the ordinary authorisation-code flow with PKCE, `offline_access` and the consent prompt, in the pinned Playwright image. It then posts that login's offline consent to the Site API's consent route with the shared intake key, where the API hands each sign-in. The route is HTTPS, so a forward to it names the CA and the certificate name to check: ```sh MEMBER_CREDENTIALS_INTAKE_KEY=... python3 dx/metallo/icarus-hpc/member_consent.py \ --target dev- --ssh --member \ --intake https://127.0.0.1:/member-credentials/v1/consents \ --intake-ca site-api-ca.crt --intake-name site-api..svc ``` The refresh token lives only in a 0600 file beside the laboratory state, which the script deletes before it exits, and in the request body, which travels over the SSH session's standard input. Nothing is printed except the route's status word. Run it again for the same member to replace the earlier consent; the store revokes the previous offline session at the issuer. The mixed Pipeline qualification does this itself: it prepares the consent store and the login client's binding for its disposable Site API, captures one consent from a real login, and captures a second login while the Pipeline's Runs are executing, so the later native actions submit with the replacement consent. ## Known limitations ### Allocation limits are never raised The laboratory never raises the limit of an existing allocation. More hours are a new order, which creates a new allocation and a new Slurm account `pharos_a_…`. The agent stays coordinator of `pharos` and receives no further rights. An order that asks for a higher `GrpTRESMins` on an existing allocation ends `erred`, and the resource is shown as `Erred`. Waldur applies an order's limits only when the agent reports it complete (`src/waldur_mastermind/marketplace/callbacks.py`, `resource_update_succeeded`), so the allocation keeps its old limit and its running jobs continue. HomePort still offers the update and Waldur accepts the order: Waldur 8.1.3 has no offering or component option that forbids a limit change on an existing resource. `plugin_options["enable_resource_limit_change_requests"]` only gates the request workflow for users who lack the limits permission, and is off by default; `can_update_limits` is a property of the plugin type (`marketplace_site_agent/apps.py`), not of an offering; and `supports_downscaling` is about reductions, not raises. The order therefore fails at Slurm, not in Waldur. Slurm 24.11.5 refuses the increase itself. In `src/plugins/accounting_storage/mysql/as_mysql_assoc.c` (around line 1764) a coordinator's change calls `assoc_mgr_check_assoc_lim_incr(mod_assoc)`, where `mod_assoc->id` is the association being changed; the new limit is compared with that account's own current limit, not with `pharos`. In `src/common/assoc_mgr.c` (`_find_tres_incr`) the comparison is skipped only when the current or the new limit is `NO_VAL` or `INFINITE`. That is why the error message "Coordinators can not increase … above the parent limit" is misleading, and why a ceiling on `pharos` changes nothing: a coordinator may set a first limit or lower one, but may not raise one. What the member does: order a new allocation for the extra hours, and submit with `sbatch --account` naming the new account. The old allocation keeps its own limit until it is used up or terminated. What the administrator does: grant no rights and do not edit the limit by hand, because Slurm refuses the increase for whoever asks. Tell the member to order a new allocation. Then clear the failed resource: as the offering's manager, use HomePort's "Set resource state to OK" action, or `POST /api/marketplace-resources//set_state_ok/`. Waldur allows it from `Erred`, the allocation keeps its old limit, and usage reporting and membership stay as they were. Options we rejected: - `AdminLevel=Operator` on the agent. It worked on the dev box, but it covers the whole Slurm database, not this laboratory's associations. - `sudo` or a helper on icarus-hpc. The Pharos agent would take rights on infrastructure that Icarus administrators run. - A two-step raise: set `GrpTRESMins=-1` and then the new value. According to the Slurm code the second change would pass the coordinator check, but it exploits a gap in Slurm and defeats the restriction's intent. We neither implement nor test it. ### Other limitations - Termination keeps the Slurm account. As coordinator of `pharos`, the Site agent may remove users and change limits, but `sacctmgr remove account` needs operator level. With `soft_delete` it cancels the account's jobs, removes its users and zeroes its limits. The empty account stays under `pharos` until a site administrator removes it. - `waldur-site-agent` 1.0.8rc9 is a pre-release. It is the first release with `account_source: waldur`. - The upstream Site agent supports only a simple bind, so it uses unencrypted loopback LDAP on the directory host. The listener is not exposed off the host. - OpenLDAP does not check attribute ACLs when an entry is added, so an add by the Site agent could include `sshPublicKey`. The access agent's snapshot pass reports values it does not manage; see the access agent README. ### Slurm staging resources The laboratory exposes `icarus_hpc_slurm_staging_cpus` (default `1`), `icarus_hpc_slurm_staging_memory_mib` (default `512`) and `icarus_hpc_slurm_staging_time_minutes` (default `10`) as positive integer Ansible variables. They populate the Runtime's `SLURM_STAGING_CPUS`, `SLURM_STAGING_MEMORY_MIB` and `SLURM_STAGING_TIME_MINUTES` environment settings; the time value also sets the staging partition's `MaxTime`. Choose CPU and memory requests within the laboratory node's available capacity. Validation happens before host changes. Attempts prepared before a configuration change retain their saved limits and partition for both stage-in and stage-out. See the [Slurm Runtime guide](../../../services/metallo/docs/epics/instantiate-and-run/slurm-jobs.md#staging-settings-and-reported-state) for recovery and legacy journal behaviour. ### LDAP entry identity qualification `entry_identity_probe.py` is a lab-only probe for an account deleted and recreated with its POSIX UID unchanged. Run `sweep` and `observe` inside the Metallo pod, and `launch` and `retained` inside the request-host pod, with the source revision under review on `PYTHONPATH`. It uses the pods' existing Secret mounts and refuses configurations whose source or repository instance is not `devbox`. The qualification needs a real Beto member, its established Slurm Site binding, a live offering resource and a real repository under the Project's `repos` group. The probe creates a Workload through its owner with current Beto capability checks, accepts a launch with the real confirmed account, and waits for the deployed execution worker to store the Run. `retained --run ` exercises the production retained-account reader over that saved Run. No owner doubles or fabricated projection rows are used. Keep the baseline UID reporter separate from the fixed reporter; never run both against the same allocation. Use `observe --old-identity ` before a lab administrator deletes the entry. It reads the account rows until the previous identity is absent and the replacement is ready, recording wall time and the lab kernel's monotonic clock. Keep the normal agent and Metallo polling periods. Compare file ownership, confirm that `retained --expect rejected` refuses the old Run with the accepted-identity mismatch, and that a new launch accepts the replacement. This checks admission and retained identity; native job execution is a separate lane. The devbox must enable the chart's existing `pharos.metallo.enabled` ingress setting when qualifying HPC naming or resource metadata. It admits only `pharos` pods labelled `app: metallo` to the Waldur API; existing LLM-agent rules remain independent. An HTTP probe from the host alone does not establish connectivity from the Metallo pod.