For AI agents: the complete documentation index is available at /llms.txt, the full documentation bundle is available at /llms-full.txt, and this page is available as Markdown at /reference/docs/decisions/0059-icarus-hpc-provisioning-run-installation.md.

0059 — Production Icarus-hpc provisioning and run installation

Status: Proposed. Date: 25 October 2026. Amended 1 November 2026 (decision 3 and the required Site-agent mode set; see Amendment). Implementation and remaining qualification are in DD-0059. Stacked on ADR/DD-0055 and the verified Slurm Runtime work in #778 (native Slurm Runtime delivery).

Context

Icarus-cloud and Icarus-hpc are separate installations. Icarus-cloud owns the Pharos platform, Site API, Kubernetes Runtime and GitLab/LFS endpoint. The browser platform and Site API use separate public hostnames, backend services, identities and bearer scopes. The independently administered Icarus-hpc site owns OpenLDAP, SSSD/sshd, Slurm, the upstream Waldur Site agent, the SSH access agent and the native Slurm Site Runtime. Icarus-hpc exposes no inbound platform port and needs no platform database, broker or Kubernetes credentials.

The existing dx/metallo/icarus-hpc laboratory is a disposable qualification fixture. It creates a complete Debian host, fixture identities, a development CA, test providers and a compiled Slurm installation. Those choices are not a safe production installer. The earlier installation drafts and their ADR/DD numbers are historical proposals, not accepted decisions.

PR #643 records the Icarus agreements: OpenLDAP without LDAPS, Pharos entries below ou=pharos, member UIDs 5000000–5499999, allocation GIDs 5500000–5599999, Lustre homes, shared /scratch/pharos, a staging/compute/staging job chain, and pharos-agent as coordinator rather than Slurm administrator. The production installer must preserve those boundaries while leaving native policy changes under the Icarus administrators' review.

Decision

  1. One administrator-run installation has four explicit phases. Preflight, prepare/install, validate and activate are separate. Installation writes a versioned release and never starts it implicitly. Activation requires an explicit component set and accepted external-contract evidence.
  2. The production input is one verified release and a small site handoff. Platform maintainers publish one versioned pharos-hpc-site-vMAJOR.MINOR.PATCH.tar.gz archive tied to one reviewed source revision. It carries the thin install.sh entry point, exact installer sources and controller lock files, the Metallo wheel, complete hashed requirements for all three components, release manifest/checksums and public examples. The HPC administrator receives that archive plus a separate public profile/inventory/installer configuration and a private credential directory. The operator does not build the wheel, derive dependency hashes or reconstruct a source revision. The installer accepts either a clean Git checkout at the recorded revision or an extracted archive whose per-file installer provenance matches it. Symlinked inputs, changed artifacts, unknown phases and inline secrets are refused. A release declares index or wheelhouse dependency mode and its target architecture. Index mode uses one approved HTTPS package index; wheelhouse mode supplies every required wheel and uses --no-index --find-links. Both use complete transitive hashes, binary-only, no-dependency isolation. Python 3.13 remains an explicit controller/target host prerequisite and is never replaced by the archive.
  3. Central bootstrap and HPC-local policy remain separate. Platform administrators create the Waldur provider, offering and its node component, POSIX pools, the provider token and the Site API grants. The token belongs to a CUSTOMER.OWNER of the provider organisation or a CUSTOMER.MANAGER of its service provider, because Waldur 8.1.3 accepts a dated usage report only with RESOURCE.SET_USAGE on one of those two sources; one scoped offering manager is not enough. Either role reaches every offering the provider sells, so it is acceptable only while the provider sells this single HPC offering (amended 1 November 2026, see the amendment below). Icarus administrators review and apply LDAP schema, ACLs, SSSD/sshd, Slurm accounting/admission, homes, quotas, scratch and network prerequisites. Native agents use the separate Site API origin as their base; clients append the /v1/site/commands/* and /v1/site/ssh-access/* paths. The edge routes that hostname and prefix to the Site API, while browser API and web requests retain separate hosts, backends and credentials. Native requests use scoped bearers and do not enter a browser OIDC flow. Installing an agent unit does not prove any of those operations.
  4. Three unprivileged identities have disjoint authority. pharos-agent owns restricted LDAP account/POSIX/group changes and the Pharos Slurm association tree as coordinator. pharos-ssh-access writes only member sshPublicKey values. pharos-runtime owns only its journals, bounded native operations and the workspace contract. None has sudo, root, a shell or another component's token. Administrator privilege exists only for the installer and approved native policy. The profile names the two LDAP connections separately. This installer supports only ldapi with SASL EXTERNAL and no password for the access agent; its owner adaptor's separately credentialed TCP mode is not exposed until that complete input contract is implemented. The pinned upstream Site agent retains a restricted loopback ldap:// simple bind. A local socket alone does not provide the upstream agent with SASL EXTERNAL.
  5. Reusable service packaging owns the units. The installer reuses the SSH access and native Runtime packaging under services/metallo/deploy/ and renders the upstream Site agent configuration. It must not import or install the laboratory or copy fixture providers into production.
  6. Upgrade and rollback preserve state. Releases are immutable directories and stable component links switch only during activation. Repeat installation keeps identities, homes, scratch, jobs, credentials and paired journals. Rollback reactivates a retained hash-verified release and never deletes application state. Activation and rollback persist selected prior link, service and informational success-record state, validate after the link switch and restore that selected state before reporting failure. Active-release success is recorded only after the selected services pass health checks. Installation, validation, activation and rollback are the same tested entry point and the same site configuration file; there is no second orchestration layer.
  7. Activation is fail-closed on real owner contracts. Production activation requires accepted revisions for central bootstrap, Beto synchronisation, member/Site/Project bindings, current Project/Run/cancel authority, member credential delegation, repository grants and the member-owned no-ACL workspace implementation. The Runtime also runs --check-owners and refuses to start while any production owner adaptor is unconnected.
  8. Lustre workspaces do not depend on filesystem ACLs. Production must not reuse the laboratory's setfacl hand-off. The 23 October answers in #643 selected option B: member-owned stage-in, compute and stage-out jobs use an Internet-enabled staging partition while compute nodes remain offline. Jobs create private workspaces and own their files. Runtime orchestrates rather than traversing or deleting member-private workspace content. The laboratory implements that member-created workspace and keeps its narrow setfacl hand-off only for the runtime-owned credential directory the member's stage-in job reads; production must not reuse it. Production gives the member's stage job the Run-scoped repository credential in that job's own Slurm environment, set when the Runtime submits the job, and never in a file: Slurm shows a job's environment only to the job's owner and to its administrators, and the job runs as the member. The Runtime saves and compares the job without those two values, so no credential enters the job script or the Runtime's journals. Into /scratch/pharos/.runtime, a runtime-owned directory without list permission, it writes only what is not secret, the askpass helper that hands Git the two values from the environment and the repository's CA, once per Attempt under a name derived from the Attempt and its material, and it deletes them when the stage releases the grant. Those files carry the pharos-users group instead of a filesystem ACL, so the one grant a site makes is usermod -aG pharos-users pharos-runtime, never a step per member, and no sudo, broad impersonation or world-writable scratch is added. An earlier version kept the credential itself in those group-readable files and accepted that a member who learnt an Attempt's path could read it. The path is not secret, because the Attempt ID and material digest it derives from appear in the job's name and comment, which any Slurm user can list, so that weakness was closed (#974). Credential delivery follows ADR-0035: the member's own access token, refreshed from one offline consent, submits every native action. Runtime monitors, accounts and cancels with its own Slurm coordinator role, which is a distinct identity and never a substitute for the member's token. That identity holds no token of any kind: the Runtime runs squeue, sacct and scancel as its own Unix user and reaches the site through munge, so nothing has to be issued, stored or refreshed for a machine identity. The Runtime fetches the member's token per native action from POST /v1/site/commands/{commandId}/member-token on the Site API, under the same /v1/site/commands/ scope as its command stream; the path names the unacknowledged command, the body names its Attempt, and the Site is the one its own grant proves. That keeps the token route inside the command prefix, so a Site API can scope one prefix for commands and their tokens without a third scope.
  9. Membership removal is not automatic Run cancellation. Existing jobs continuing after their initiator leaves remains danias's proposal in #755, not a settled contract. A departed member loses inspection/control/new execution as agreed; another currently authorised Project member may inspect or cancel subject to explicit permissions and audited actor identity. Observation, cancellation, stage-out, later steps, retry, repository credentials and workspace/results access each need an agreed authority. Revoked personal credentials are never retained and technical-user impersonation is not a workaround. A coordinator role may observe, account and cancel for policy reasons without any member token; submission and stage-out still require the acting member's own current token, so a replacement at login is what restores a Run whose offline consent has expired. The laboratory realm now ends an offline session 86400 seconds after consent, as GRNET staging does (ADR-0035), so no qualification may assume a wider window.

Amendment (1 November 2026)

Two recorded boundaries changed with the production installer's Waldur offering contract, and this amendment replaces the earlier text rather than leaving it implied.

  1. The Site-agent token is a provider token, not one scoped offering manager. Waldur 8.1.3 accepts a dated usage report only from a user holding RESOURCE.SET_USAGE on the offering's organisation or on that organisation's service provider (PROVIDER_USAGE_SOURCES in waldur_mastermind/marketplace/serializers.py). A token scoped to a single OFFERING.MANAGER holds the permission on the offering, so every report is refused with HTTP 400. The installer therefore requires CUSTOMER.OWNER on the provider organisation or CUSTOMER.MANAGER on its service provider, and reads the agent's own roles from /users/me/ because a service-provider manager cannot read the organisation's team. Both roles also create and delete every offering the provider sells and approve its orders. That wider authority is acceptable only while the provider sells this single HPC offering; selling another offering requires a separate review or a narrower deployment. CUSTOMER.OWNER is the role to assign: this deployment disables customer_manager, so the API refuses to grant the other accepted role and it is available only from waldur shell.
  2. report is a required Site-agent mode, not an owner decision. The node component is limit-billed with period total, so consumption reaches Waldur only through the agent's periodic report. validate rejects a profile whose site_agent_modes omits report; event_process remains an explicit input. validate does not compare the component's unit_factor: Waldur treats it as metadata, the pinned loader does not send it, and the conversion runs in the Site agent from its own component configuration.

Consequences

The installer can be developed and rehearsed before the production owners are ready, but an installed unit is not production activation or qualification. The first release should install the three service environments, private configuration, systemd units and runbooks, validate host prerequisites and remain paused. Runtime and mixed Pipeline claims stay blocked on #755, #765, #766, the Beto synchronisation policy, delegation/repository owners and the member-owned no-ACL workspace implementation.

The exact release boundary adds an artifact assembly step. In return, an HPC host cannot silently install a different wheel, requirements set, interpreter, LDAP endpoint or source revision. State-preserving upgrades need disk space for retained releases and journals, but avoid deleting the known-good path during an upgrade. The bundle builder and extracted-source verifier are implemented on the implementation branch. Publishing that builder's output through a named GitHub release or signed OCI artefact remains platform-maintainer work; the existing platform OCI release does not by itself contain or publish this HPC bundle.

Open questions

  • Final Icarus host OS, architecture and approved Python 3.13 runtime installation.
  • Production Site-agent modes and their event/checkpoint/recovery settings.
  • Native job_submit.lua, coordinator rights and definite rejection codes on the installed Slurm version.
  • The staging partition name and limits, staging egress to GitLab/S3, and the agent host's approved outbound connectivity. These are the concrete outstanding Icarus inputs recorded after the 23 October answers.
  • The distributed owner's accepted commit for the separate Site API hostname, its route-priority and timeout policy, and equivalent staging hostname/TLS evidence.
  • Our member-owned no-ACL workspace, credential rendezvous, cleanup and output-retention implementation and native Lustre qualification.
  • Missing-progress timeout, maximum access-removal delay, permission names and separate cancel/cleanup authority under #755/#766.
Page source

Source snapshot · Revision f928834c8.