#DD-0059 — Production Icarus-hpc provisioning and run installation

Related ADR: ADR-0059.

Status: Draft. Deterministic installation and gate implementation is in review; production activation and native qualification are outstanding. The installer is plain bash since 3 November 2026 (§1–§4, §8); the earlier Ansible role is removed.

Date: 2026-10-25. Amended 2026-11-03.

#1. Installation boundary

The operator entry point is pharos-hpc, one plain-bash script with three sourced libraries (lib/common.sh, lib/check.sh, lib/install.sh, lib/service.sh) that ships inside the release archive. It runs as root on the login node; Python runs only the agents, inside their own virtual environments. There is no Ansible controller, inventory, become or PyPI-fetched tooling between the administrator and the host, because the HPC administrators who operate this site read and support the installer themselves.

sudo ./pharos-hpc check                           # read-only
sudo ./pharos-hpc install                         # users and one release
sudo ./pharos-hpc configure [TOKEN_DIRECTORY]     # configuration, secrets, offering components
sudo ./pharos-hpc activate site-agent ssh-access  # link, units, start, health
sudo ./pharos-hpc status | rollback [RELEASE_ID] | stop

The operator-facing procedure is the HPC Site runbook.

#2. Inputs, configuration and exact artifacts

One file, /etc/pharos-hpc/site.env, carries every site value as KEY=value. The installer parses it with a strict KEY=value reader: it is never sourced, values are never substituted, duplicate keys and unknown syntax are rejected, and no name arrives from the calling environment. It replaces the former public profile, the site-install file and the hpc_site inventory. site.env.example carries every key with a comment.

The real file must use URLs without embedded credentials, keep plaintext LDAP and slurmrestd on host loopback, use different compute/staging partitions and allocate unique, non-system service UIDs/GIDs outside approved member ranges. CENTRAL_OFFERING_UUID is 32 lowercase hex characters. CENTRAL_SITE_API_URL is an origin only: the agents append /v1/site/commands/* and /v1/site/ssh-access/* themselves, and the edge routes the host and prefix to the Site API backend. Browser API and Site API remain separate services with different hosts and credentials.

ACTIVATION_COMPONENTS is validated against the release manifest's components: line. The Release does not ship pharos-slurm-runtime; a component the manifest does not list cannot be activated, so the Runtime is absent rather than half-configured until its owner connections land.

Private inputs are files below /etc/pharos-hpc/secrets, never values in the configuration and never command arguments:

configure accepts the directory the central step writes and copies those files unchanged, setting the modes above. It publishes the two CA files again as /etc/pharos-hpc/site-agent-ca.crt and /etc/pharos-hpc/site-api-ca.crt, because the agents run unprivileged and cannot read the root-only secrets directory. Changing a token is one configure run and needs no new release.

Platform maintainers produce one archive with scripts/hpc_site_release.py; HPC administrators do not build the wheel, derive requirements or reconstruct a source checkout. The archive contains pharos-hpc, lib/, units/, site.env.example, README.md, the Metallo wheel, the hash-pinned requirements files named by the manifest, the release manifest and SHA256SUMS. check, install, activate and rollback run sha256sum --check over the bundle and over the installed release; install records a fingerprint of the manifest and site.env and refuses to reuse a release ID with different inputs. Official archives are Sigstore-signed by the release workflow.

The release states dependency_mode: index or dependency_mode: wheelhouse. Index mode requires an approved HTTPS PYTHON_INDEX_URL; wheelhouse mode carries every locked wheel and installs with --no-index --find-links. Both modes run pip isolated with --require-hashes, --no-deps and --only-binary=:all:. Neither mode supplies the target Python 3.13 interpreter, OS packages or LDAP/Slurm.

#3. Component layout and privilege matrix

The stable paths are symlinks through /opt/pharos-hpc/current, which points at one immutable /opt/pharos-hpc/releases/<release-id>/ directory. Both component links live inside /opt/pharos-hpc, so a host that still carries an older /opt/pharos-ssh-access-agent directory is never written into. Code lives there; configuration, secrets and public trust stay in /etc/pharos-hpc, so a token change does not create a release and a rollback does not touch configuration.

The unit files are final files, not templates: they read EnvironmentFile=/etc/pharos-hpc/<component>.env and take the access bearer through LoadCredential=. They name the fixed accounts pharos-agent and pharos-ssh-access; the site chooses UIDs and GIDs. Units have NoNewPrivileges, an empty capability set and no root user, and the Site agent owns a private StateDirectory. Neither receives platform PostgreSQL, RabbitMQ, Kubernetes, LDAP administrator or Slurm administrator authority.

SITE_AGENT_MODES selects order_process, membership_sync and report. report is required rather than optional: the node component is limit-billed with period total, so the periodic report is the only path by which consumption reaches Waldur, and check rejects a configuration without it. No mode is inferred from a running host.

#4. Activation, health and recovery

Activation validates the installed release's SHA256SUMS, requires configure to have run, records the resolved current target as previous, points current at the release, installs the release's unit files, then starts and enables the selected units. Success is recorded in /var/lib/pharos-hpc/active-release only after every selected unit reports active and:

• the offering reads Active in Waldur through the agent's own token and CA;
• the Site API answers over TLS against the configured CA.

Both probes use curl without --location, so a redirect never carries the scoped bearer to another host. A failing step restores previous, re-installs its unit files, restarts the same units and reports that the earlier release is active again, all from the two symlinks and /var/lib/pharos-hpc/active-release; no transaction journal or separate helper is involved. The operation never deletes retained releases, identities, journals, workspaces or Slurm jobs.

#5. Icarus native prerequisites

These are reviewed administrator operations and are not hidden installer tests:

1. Record and approve the actual host OS, architecture, NSS and Python 3.13+ interpreter. Debian 12's system Python alone is not sufficient for Metallo.
2. Preserve existing Slurm and OpenLDAP installations and unrelated identities. Validate UID/GID collisions before creating service identities. The approved Icarus numbers are split by owner: member UIDs 5000000–5499999, members' primary GIDs 5500000–5549999 from Waldur's POSIX pool, and project-group GIDs 5550000–5599999 from the Site agent. The installer writes only the project-group range and check refuses it if it overlaps the members' primary GID range.
3. Apply the agreed ou=pharos schema and one-writer-per-attribute ACLs. This installer's access agent uses ldapi with SASL EXTERNAL; its owner package can also construct an explicitly credentialed TCP connection, but the installer does not yet accept that separate DN, password and CA input contract. The upstream agent keeps its documented loopback simple bind.
4. Configure SSSD/sshd key lookup, cache/propagation policy and the 120-second access window. Key revocation does not terminate an already established SSH session.
5. Configure Slurm accounting, pharos-agent and pharos-runtime coordinator rights, admission rules that prevent service submissions, member JWT verification and explicit definite rejection codes. The coordinator identity lets the Runtime observe, account and cancel without a member token; members' own tokens submit. It reaches the site with the Slurm client over munge, so the Runtime host needs munge and the Slurm binary directory on the unit's path rather than a credential.
6. Configure Lustre homes, quotas, scratch retention, staging/compute partitions and outbound GitLab/LFS/S3/CA reachability. The 23 October answers in #643 already selected option B: member-owned stage-in and stage-out jobs use an Internet-enabled staging partition while compute nodes remain offline. Still to be supplied are its concrete partition name/limits and confirmation of staging egress. Do not reopen the movement profile or ask for filesystem ACLs.
7. Run a deliberate canary Project/member. Readiness checks never create production accounts or jobs as an undocumented side effect.

site.env carries NATIVE_SSH_ACCESS_LDAP_URI and NATIVE_SITE_AGENT_LDAP_URI separately. The installer permits only ldapi:// for the access agent, which selects SASL EXTERNAL and rejects a bind DN, password or CA; authority is the exact Unix peer identity. TCP LDAP/LDAPS remains a capability of the separately packaged adaptor, not a selectable installer setting until its complete credential contract is implemented and qualified. The pinned upstream waldur_site_agent_ldap==1.0.7 does not support that authentication path. Its schema requires bind_dn and bind_password, and LdapClient._connect() passes them to ldap3.Connection without an authentication or SASL option. With the pinned ldap3==2.9.1, that selects SIMPLE when a user is present. Its URI must therefore remain loopback ldap://; ldapi:// alone would change transport but not request SASL EXTERNAL. Removing the password requires an upstream change and a separate compatibility qualification.

The upstream LDAP username backend is authoritative only when the dedicated offering has username_generation_policy=service_provider and service_provider_can_create_offering_user=true. check reads those fields through the same provider credential and Unix identity as the Site agent. An identity_claim offering is rejected before activation because Waldur, not the agent, would own username generation and the expected POSIX/Slurm member would never be created. The generic provider-plan route is not a health signal for this non-staff identity; use the nested public-offering plan route recorded in the operator runbook.

#6. Member-owned no-ACL workspace implementation

The Icarus answers recorded by #643 are the accepted Site design, not an open ACL or staging decision. Lustre has no filesystem ACLs. Icarus supplies option B: a separate Internet-enabled staging partition, offline compute nodes, and stage-in, compute and stage-out jobs that all run as the member. The member's job creates its workspace with umask 077, mode 0700, and owns its files from the start. The Runtime orchestrates Slurm and does not clone, stage or clean repository files as pharos-runtime.

The laboratory now pre-provisions /scratch/pharos/<member> as a member-owned mode-0700 root and has the member's stage-in job create the Attempt and workspace with umask 077; it does not use setfacl for either. Stage logs sit beside that pre-existing root, because Slurm otherwise creates a missing Attempt parent before the member script and defeats the private mode contract. The Runtime keeps only the planned path and cannot read or write the member-private workspace. The temporary Git credential hand-off still uses a narrow setfacl double under /scratch/pharos/.runtime, a runtime-owned mode-0711 root. That is a qualification limitation, not a reason to request ACLs at Icarus or to reproduce its cross-UID inspection/deletion model in production. The remaining implementation is:

• stage-in, compute and stage-out run under the member identity and cannot read another member's workspace;
• a just-in-time repository credential is delivered to the member's job through an owning credential broker, revoked after use, and is never retained by the Runtime;
• the Runtime verifies only the workspace facts that the Run protocol needs and does not require broad traversal of member-private files;
• working-directory retention and deletion follow the agreed member ownership, cleanup policy and Site scratch purge; the laboratory Runtime removes only its credential hand-off and never traverses or deletes the member Attempt, with any separately scoped cancellation or cleanup authority audited;
• no production composition imports the dx/ provider double, grants sudo to a service identity or makes /scratch/pharos globally readable or writable.

Runtime activation stays blocked until the temporary credential delivery/revocation owner is connected and the complete member-owned flow is tested against native Lustre. The profile's accepted workspace-contract marker is a gate, not evidence by itself. Native Lustre validation is our outstanding implementation qualification; the Site architecture and ACL facts are already settled.

#7. Continuity after member removal

The following matrix records what must be decided, not a claim that it is implemented:

Membership removal is not automatic Run cancellation. Existing jobs continuing after creator departure is danias's current proposal, not a final contract. project.view, runs.view, runs.output.view and runs.cancel are suggested names only.

The same matrix covers a member whose offline Session ends while the Run is still executing, which is a different cause and a different answer. Observation, accounting and cancellation do not use the member's token at all: they run as the Site's Slurm coordinator over munge, so a Run keeps being watched and can be cancelled. Stage-out and every later step that must run as the member do need a live consent; the Site API answers with one fixed reason and the Attempt must then end with a terminal outcome rather than wait for ever.

The periodic consent refresh that keeps an idle Run inside the issuer's idle window is implemented in the Member Credentials owner (#879): one worker refreshes every active consent of a member who still holds Project execution authority once its idle deadline falls inside the next two refresh intervals, under the same per-member rotation lock a native action takes, and keeps no access token. It never refreshes a revoked, superseded, lost or expired consent, nor one of a member who left the Project, and it cannot move a deadline past the issuer's maximum lifespan, which the issuer reports. The terminal Attempt outcome for a Session that can no longer be used, and the notice that asks the member to sign in again, remain tracked as one issue against #766 and #636. The offline window itself is ADR-0035.

#8. Verification and deferred lanes

scripts/test_hpc_site_installation.py runs the installer against a temporary root (PHAROS_HPC_ROOT) with recording shims for useradd, groupadd, getent, systemctl, systemd-analyze, curl and the venv's console scripts. It covers a missing secret before any change, an unchanged check, the offline release and its fingerprint, configuration and token adoption, both LDAP mechanisms, the offering component reconciliation, activation and the current/previous links, rollback, a failed activation that restores the predecessor's unit files, a tampered release, an unavailable component, and probes that never follow a redirect. scripts/test_hpc_site_release.py covers the archive contents. shellcheck -x and the release build run in pnpm verify; scripts/shellcheck.py fetches the pinned release (0.11.0, SHA-256 checked) into the tool cache on first use, so the check runs on the ARM64 runners and on a machine that never installed ShellCheck.

The laboratory lane installs the candidate bundle on the icarus-hpc development box over a host that already carries the Phase 1 LDAP, Slurm and NSS prerequisites, then runs its e2e script (account creation, key deployment, submit and revoke inside 120 seconds), a rollback and an activation that is made to fail. The laboratory is restored afterwards with its own installer. That lane proves the script against real systemd, real accounts and real LDAP; it does not qualify the production host.

The Site API edge qualification must exercise the separate staging hostname and the equivalent devbox endpoint before staging cut-over:

• /v1/site/commands/* and /v1/site/ssh-access/* reach the Site API on their dedicated hostname with the configured route-specific timeouts and body limits;
• missing, malformed, expired and revoked native grants, wrong route scopes and cross-Site attempts return the expected authentication failures rather than a web response or browser-login redirect;
• valid Runtime and SSH access grants complete native pull and acknowledge round trips through hostname- and CA-verified TLS;
• the existing site.pharos.<dev-domain> alias is reached through the established devbox edge/tunnel, with its certificate SAN and public CA verified even when the test caller resolves the host to loopback;
• site.staging.pharosfactory.ai passes equivalent DNS, TLS, SNI, routing and authentication checks once the distributed owner and DNS owner publish it.

The agent's native psb_ grant is the authentication control. TLS trust proves the server endpoint only and is never treated as member, Runtime or Site authority.

Pipeline evidence from #793 remains owner-provided chart-GitLab executor evidence, and exact commit pinning from #787 is a prerequisite for matching later-step inputs. Neither lane closes the owner ports above, and neither may be reported from the older #778 mixed run without rerunning its exact integrated revision and inputs.

Not established by those checks: separate-host edge deployment in a live distributed environment, real Icarus OS/Python facts, LDAP/SSSD propagation, Waldur/LDAP/Slurm order provisioning, native direct member access-token delivery, real GitLab/LFS stage-in/out, no-ACL Lustre isolation, member removal continuity, systemd production activation, PostgreSQL/broker integration, browser/UAT, cloud cutover or rollback on a real HPC host. Those remain explicit qualification lanes and require owner participation.

#9. Provider role and offering component ownership

Added 1 November 2026 with the production installer's offering contract; this section amends §2 and §3 above.

Waldur 8.1.3 accepts a dated usage report only from a user holding RESOURCE.SET_USAGE on the offering's organisation or on that organisation's service provider (PROVIDER_USAGE_SOURCES in waldur_mastermind/marketplace/serializers.py). The token therefore belongs to a CUSTOMER.OWNER of the provider organisation or to a CUSTOMER.MANAGER of its service provider, not to one scoped offering manager, which holds the permission on the offering and is refused with HTTP 400. The contract reads the agent's own roles from /users/me/ because CUSTOMER.MANAGER is granted on the service provider and the organisation's team read returns 404 for it.

The role to assign is CUSTOMER.OWNER on the provider organisation, as the laboratory bootstrap does. This deployment disables customer_manager (disabledRoles in gitops/manifests/waldur/base/values.yaml), so the Waldur API refuses to grant it ("Role is not active") and it can only be granted from waldur shell. The contract keeps accepting it — Waldur grants the permission on either source — but an administrator who follows this document does not need it.

Both accepted roles reach every offering the provider sells, including creating and deleting offerings and approving orders. That is acceptable only while the provider sells this single HPC offering; a provider that later sells another offering must move this agent to a narrower deployment or accept the wider authority explicitly.

configure also owns the offering's node component: it runs the upstream waldur_site_load_components entry point against the installed Site-agent configuration, so an offering that predates this contract gains the component on its next upgrade. The step is idempotent, creates no plan and leaves the component's limit_amount unset, because each order carries its own quantity. The pinned 1.0.8rc9 loader does not send unit_factor, so the component it creates reads Waldur's default. Waldur defines unit_factor as metadata outside any Mastermind computation, and the conversion that produces node-hours runs in the Site agent from its own backend_components.node.unit_factor: 60.