> For AI agents: the complete documentation index is available at /llms.txt, the full documentation bundle is available at /llms-full.txt. # DD-0059 — Production Icarus-hpc provisioning and run installation **Related ADR:** [ADR-0059](/reference/docs/decisions/0059-icarus-hpc-provisioning-run-installation.md). **Status:** Draft. Deterministic installation and gate implementation is in review; production activation and native qualification are outstanding. The installer is plain bash since 3 November 2026 (§1–§4, §8); the earlier Ansible role is removed. **Date:** 2026-10-25. Amended 2026-11-03. ## 1. Installation boundary The operator entry point is `pharos-hpc`, one plain-bash script with three sourced libraries (`lib/common.sh`, `lib/check.sh`, `lib/install.sh`, `lib/service.sh`) that ships inside the release archive. It runs as root on the login node; Python runs only the agents, inside their own virtual environments. There is no Ansible controller, inventory, `become` or PyPI-fetched tooling between the administrator and the host, because the HPC administrators who operate this site read and support the installer themselves. ```sh sudo ./pharos-hpc check # read-only sudo ./pharos-hpc install # users and one release sudo ./pharos-hpc configure [TOKEN_DIRECTORY] # configuration, secrets, offering components sudo ./pharos-hpc activate site-agent ssh-access # link, units, start, health sudo ./pharos-hpc status | rollback [RELEASE_ID] | stop ``` | Command | Effect | | ----------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `check` | Validate the release checksums, `site.env`, the private inputs, the host Python and OS, the unit files and the read-only Waldur offering contract. Changes nothing. | | `install` | Create the two service identities without shells or homes and stage one immutable release with offline environments. Start nothing. | | `configure` | Write the agents' configuration from `site.env`, adopt the token files unchanged, publish the public CA files, and reconcile the offering's components with the Site agent's configuration. | | `activate` | Validate the installed release again, record `previous`, point `current` at it, install its unit files, start the selected units and require health. A failure restores the recorded predecessor. | | `rollback` | The same path with `previous`, or with a named installed release. | | `stop` | Stop and disable this site's agents; keep releases, configuration and secrets. | The operator-facing procedure is the HPC Site runbook. ## 2. Inputs, configuration and exact artifacts One file, `/etc/pharos-hpc/site.env`, carries every site value as `KEY=value`. The installer parses it with a strict `KEY=value` reader: it is never sourced, values are never substituted, duplicate keys and unknown syntax are rejected, and no name arrives from the calling environment. It replaces the former public profile, the site-install file and the `hpc_site` inventory. `site.env.example` carries every key with a comment. The real file must use URLs without embedded credentials, keep plaintext LDAP and `slurmrestd` on host loopback, use different compute/staging partitions and allocate unique, non-system service UIDs/GIDs outside approved member ranges. `CENTRAL_OFFERING_UUID` is 32 lowercase hex characters. `CENTRAL_SITE_API_URL` is an origin only: the agents append `/v1/site/commands/*` and `/v1/site/ssh-access/*` themselves, and the edge routes the host and prefix to the Site API backend. Browser API and Site API remain separate services with different hosts and credentials. `ACTIVATION_COMPONENTS` is validated against the release manifest's `components:` line. The Release does not ship `pharos-slurm-runtime`; a component the manifest does not list cannot be activated, so the Runtime is absent rather than half-configured until its owner connections land. Private inputs are files below `/etc/pharos-hpc/secrets`, never values in the configuration and never command arguments: | File | Mode | Authority | | -------------------------------------- | ----------------------------- | -------------------------------------------------------------------------------------------- | | `site-agent-token` | 0400 `root` | `CUSTOMER.OWNER` on the provider organisation, or `CUSTOMER.MANAGER` on its service provider | | `site-agent-ldap-password` | 0400 `root` | Restricted `cn=pharos-agent` bind | | `ssh-access-site-token` | 0640 `root:pharos-ssh-access` | SSH access Site API routes only | | `site-agent-ca.crt`, `site-api-ca.crt` | 0644 | Public trust for Waldur and the Site API edge | `configure` accepts the directory the central step writes and copies those files unchanged, setting the modes above. It publishes the two CA files again as `/etc/pharos-hpc/site-agent-ca.crt` and `/etc/pharos-hpc/site-api-ca.crt`, because the agents run unprivileged and cannot read the root-only secrets directory. Changing a token is one `configure` run and needs no new release. Platform maintainers produce one archive with `scripts/hpc_site_release.py`; HPC administrators do not build the wheel, derive requirements or reconstruct a source checkout. The archive contains `pharos-hpc`, `lib/`, `units/`, `site.env.example`, `README.md`, the Metallo wheel, the hash-pinned requirements files named by the manifest, the release manifest and `SHA256SUMS`. `check`, `install`, `activate` and `rollback` run `sha256sum --check` over the bundle and over the installed release; `install` records a fingerprint of the manifest and `site.env` and refuses to reuse a release ID with different inputs. Official archives are Sigstore-signed by the release workflow. The release states `dependency_mode: index` or `dependency_mode: wheelhouse`. Index mode requires an approved HTTPS `PYTHON_INDEX_URL`; wheelhouse mode carries every locked wheel and installs with `--no-index --find-links`. Both modes run pip isolated with `--require-hashes`, `--no-deps` and `--only-binary=:all:`. Neither mode supplies the target Python 3.13 interpreter, OS packages or LDAP/Slurm. ## 3. Component layout and privilege matrix | Component | Stable path | Unit | Allowed authority | | -------------------------- | ---------------------------- | -------------------------- | ------------------------------------------------------------------------------------------------------------------- | | Upstream Waldur Site agent | `/opt/pharos-hpc/site-agent` | `waldur-site-agent@` | Outbound Waldur HTTPS; restricted LDAP account/POSIX/group writes; Slurm coordinator operations for the Pharos tree | | SSH access agent | `/opt/pharos-hpc/ssh-access` | `pharos-ssh-access-agent` | Outbound Site API HTTPS; `sshPublicKey` writes and identity reads only | | Native Slurm Runtime | `/opt/pharos-slurm-runtime` | `pharos-slurm-runtime` | Not part of this release; see §6 | The stable paths are symlinks through `/opt/pharos-hpc/current`, which points at one immutable `/opt/pharos-hpc/releases//` directory. Both component links live inside `/opt/pharos-hpc`, so a host that still carries an older `/opt/pharos-ssh-access-agent` directory is never written into. Code lives there; configuration, secrets and public trust stay in `/etc/pharos-hpc`, so a token change does not create a release and a rollback does not touch configuration. The unit files are final files, not templates: they read `EnvironmentFile=/etc/pharos-hpc/.env` and take the access bearer through `LoadCredential=`. They name the fixed accounts `pharos-agent` and `pharos-ssh-access`; the site chooses UIDs and GIDs. Units have `NoNewPrivileges`, an empty capability set and no root user, and the Site agent owns a private `StateDirectory`. Neither receives platform PostgreSQL, RabbitMQ, Kubernetes, LDAP administrator or Slurm administrator authority. `SITE_AGENT_MODES` selects `order_process`, `membership_sync` and `report`. `report` is required rather than optional: the `node` component is limit-billed with period `total`, so the periodic report is the only path by which consumption reaches Waldur, and `check` rejects a configuration without it. No mode is inferred from a running host. ## 4. Activation, health and recovery Activation validates the installed release's `SHA256SUMS`, requires `configure` to have run, records the resolved `current` target as `previous`, points `current` at the release, installs the release's unit files, then starts and enables the selected units. Success is recorded in `/var/lib/pharos-hpc/active-release` only after every selected unit reports `active` and: - the offering reads `Active` in Waldur through the agent's own token and CA; - the Site API answers over TLS against the configured CA. Both probes use `curl` without `--location`, so a redirect never carries the scoped bearer to another host. A failing step restores `previous`, re-installs its unit files, restarts the same units and reports that the earlier release is active again, all from the two symlinks and `/var/lib/pharos-hpc/active-release`; no transaction journal or separate helper is involved. The operation never deletes retained releases, identities, journals, workspaces or Slurm jobs. ## 5. Icarus native prerequisites These are reviewed administrator operations and are not hidden installer tests: 1. Record and approve the actual host OS, architecture, NSS and Python 3.13+ interpreter. Debian 12's system Python alone is not sufficient for Metallo. 2. Preserve existing Slurm and OpenLDAP installations and unrelated identities. Validate UID/GID collisions before creating service identities. The approved Icarus numbers are split by owner: member UIDs `5000000–5499999`, members' primary GIDs `5500000–5549999` from Waldur's POSIX pool, and project-group GIDs `5550000–5599999` from the Site agent. The installer writes only the project-group range and `check` refuses it if it overlaps the members' primary GID range. 3. Apply the agreed `ou=pharos` schema and one-writer-per-attribute ACLs. This installer's access agent uses `ldapi` with SASL EXTERNAL; its owner package can also construct an explicitly credentialed TCP connection, but the installer does not yet accept that separate DN, password and CA input contract. The upstream agent keeps its documented loopback simple bind. 4. Configure SSSD/sshd key lookup, cache/propagation policy and the 120-second access window. Key revocation does not terminate an already established SSH session. 5. Configure Slurm accounting, `pharos-agent` and `pharos-runtime` coordinator rights, admission rules that prevent service submissions, member JWT verification and explicit definite rejection codes. The coordinator identity lets the Runtime observe, account and cancel without a member token; members' own tokens submit. It reaches the site with the Slurm client over munge, so the Runtime host needs munge and the Slurm binary directory on the unit's path rather than a credential. 6. Configure Lustre homes, quotas, scratch retention, staging/compute partitions and outbound GitLab/LFS/S3/CA reachability. The 23 October answers in #643 already selected option B: member-owned stage-in and stage-out jobs use an Internet-enabled staging partition while compute nodes remain offline. Still to be supplied are its concrete partition name/limits and confirmation of staging egress. Do not reopen the movement profile or ask for filesystem ACLs. 7. Run a deliberate canary Project/member. Readiness checks never create production accounts or jobs as an undocumented side effect. `site.env` carries `NATIVE_SSH_ACCESS_LDAP_URI` and `NATIVE_SITE_AGENT_LDAP_URI` separately. The installer permits only `ldapi://` for the access agent, which selects SASL EXTERNAL and rejects a bind DN, password or CA; authority is the exact Unix peer identity. TCP LDAP/LDAPS remains a capability of the separately packaged adaptor, not a selectable installer setting until its complete credential contract is implemented and qualified. The pinned upstream `waldur_site_agent_ldap==1.0.7` does not support that authentication path. Its schema requires `bind_dn` and `bind_password`, and `LdapClient._connect()` passes them to `ldap3.Connection` without an authentication or SASL option. With the pinned `ldap3==2.9.1`, that selects SIMPLE when a user is present. Its URI must therefore remain loopback `ldap://`; `ldapi://` alone would change transport but not request SASL EXTERNAL. Removing the password requires an upstream change and a separate compatibility qualification. The upstream LDAP username backend is authoritative only when the dedicated offering has `username_generation_policy=service_provider` and `service_provider_can_create_offering_user=true`. `check` reads those fields through the same provider credential and Unix identity as the Site agent. An `identity_claim` offering is rejected before activation because Waldur, not the agent, would own username generation and the expected POSIX/Slurm member would never be created. The generic provider-plan route is not a health signal for this non-staff identity; use the nested public-offering plan route recorded in the operator runbook. ## 6. Member-owned no-ACL workspace implementation The Icarus answers recorded by #643 are the accepted Site design, not an open ACL or staging decision. Lustre has no filesystem ACLs. Icarus supplies option B: a separate Internet-enabled staging partition, offline compute nodes, and stage-in, compute and stage-out jobs that all run as the member. The member's job creates its workspace with `umask 077`, mode `0700`, and owns its files from the start. The Runtime orchestrates Slurm and does not clone, stage or clean repository files as `pharos-runtime`. The laboratory now pre-provisions `/scratch/pharos/` as a member-owned mode-`0700` root and has the member's stage-in job create the Attempt and workspace with `umask 077`; it does not use `setfacl` for either. Stage logs sit beside that pre-existing root, because Slurm otherwise creates a missing Attempt parent before the member script and defeats the private mode contract. The Runtime keeps only the planned path and cannot read or write the member-private workspace. The temporary Git credential hand-off still uses a narrow `setfacl` double under `/scratch/pharos/.runtime`, a runtime-owned mode-`0711` root. That is a qualification limitation, not a reason to request ACLs at Icarus or to reproduce its cross-UID inspection/deletion model in production. The remaining implementation is: - stage-in, compute and stage-out run under the member identity and cannot read another member's workspace; - a just-in-time repository credential is delivered to the member's job through an owning credential broker, revoked after use, and is never retained by the Runtime; - the Runtime verifies only the workspace facts that the Run protocol needs and does not require broad traversal of member-private files; - working-directory retention and deletion follow the agreed member ownership, cleanup policy and Site scratch purge; the laboratory Runtime removes only its credential hand-off and never traverses or deletes the member Attempt, with any separately scoped cancellation or cleanup authority audited; - no production composition imports the `dx/` provider double, grants sudo to a service identity or makes `/scratch/pharos` globally readable or writable. Runtime activation stays blocked until the temporary credential delivery/revocation owner is connected and the complete member-owned flow is tested against native Lustre. The profile's accepted workspace-contract marker is a gate, not evidence by itself. Native Lustre validation is our outstanding implementation qualification; the Site architecture and ACL facts are already settled. ## 7. Continuity after member removal The following matrix records what must be decided, not a claim that it is implemented: | Operation | Proposed outcome after initiator leaves | Required owner input | | -------------------------------- | ---------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------- | | Observe submitted Slurm job | May continue; the Runtime reads it as Slurm coordinator | Coordinator role, saved native identity and Project/Run observer authority | | Cancel/cleanup | May be performed by a current authorised member or operator; policy cancellation uses the coordinator role | Separate scoped cancel/cleanup authority and actor audit | | Stage-out or later Pipeline step | Does not continue without the member's own current token | Current Project execution authority and a live offline consent | | Retry | Refused without current authority | Fresh binding and fresh bounded credential | | Repository credential | Its grant follows Run/repository policy, never a revoked personal token | Explicit issuance/revocation and continuity contract | | Workspace/log/output access | Departed member loses access; current members follow agreed permissions | Workspace/results permission boundary and audit actor | Membership removal is not automatic Run cancellation. Existing jobs continuing after creator departure is danias's current proposal, not a final contract. `project.view`, `runs.view`, `runs.output.view` and `runs.cancel` are suggested names only. The same matrix covers a member whose offline Session ends while the Run is still executing, which is a different cause and a different answer. Observation, accounting and cancellation do not use the member's token at all: they run as the Site's Slurm coordinator over munge, so a Run keeps being watched and can be cancelled. Stage-out and every later step that must run as the member do need a live consent; the Site API answers with one fixed reason and the Attempt must then end with a terminal outcome rather than wait for ever. The periodic consent refresh that keeps an idle Run inside the issuer's idle window is implemented in the Member Credentials owner (#879): one worker refreshes every `active` consent of a member who still holds Project execution authority once its idle deadline falls inside the next two refresh intervals, under the same per-member rotation lock a native action takes, and keeps no access token. It never refreshes a `revoked`, `superseded`, `lost` or `expired` consent, nor one of a member who left the Project, and it cannot move a deadline past the issuer's maximum lifespan, which the issuer reports. The terminal Attempt outcome for a Session that can no longer be used, and the notice that asks the member to sign in again, remain tracked as one issue against #766 and #636. The offline window itself is ADR-0035. ## 8. Verification and deferred lanes `scripts/test_hpc_site_installation.py` runs the installer against a temporary root (`PHAROS_HPC_ROOT`) with recording shims for `useradd`, `groupadd`, `getent`, `systemctl`, `systemd-analyze`, `curl` and the venv's console scripts. It covers a missing secret before any change, an unchanged `check`, the offline release and its fingerprint, configuration and token adoption, both LDAP mechanisms, the offering component reconciliation, activation and the `current`/`previous` links, rollback, a failed activation that restores the predecessor's unit files, a tampered release, an unavailable component, and probes that never follow a redirect. `scripts/test_hpc_site_release.py` covers the archive contents. `shellcheck -x` and the release build run in `pnpm verify`; `scripts/shellcheck.py` fetches the pinned release (0.11.0, SHA-256 checked) into the tool cache on first use, so the check runs on the ARM64 runners and on a machine that never installed ShellCheck. The laboratory lane installs the candidate bundle on the `icarus-hpc` development box over a host that already carries the Phase 1 LDAP, Slurm and NSS prerequisites, then runs its e2e script (account creation, key deployment, submit and revoke inside 120 seconds), a rollback and an activation that is made to fail. The laboratory is restored afterwards with its own installer. That lane proves the script against real systemd, real accounts and real LDAP; it does not qualify the production host. The Site API edge qualification must exercise the separate staging hostname and the equivalent devbox endpoint before staging cut-over: - `/v1/site/commands/*` and `/v1/site/ssh-access/*` reach the Site API on their dedicated hostname with the configured route-specific timeouts and body limits; - missing, malformed, expired and revoked native grants, wrong route scopes and cross-Site attempts return the expected authentication failures rather than a web response or browser-login redirect; - valid Runtime and SSH access grants complete native pull and acknowledge round trips through hostname- and CA-verified TLS; - the existing `site.pharos.` alias is reached through the established devbox edge/tunnel, with its certificate SAN and public CA verified even when the test caller resolves the host to loopback; - `site.staging.pharosfactory.ai` passes equivalent DNS, TLS, SNI, routing and authentication checks once the distributed owner and DNS owner publish it. The agent's native `psb_` grant is the authentication control. TLS trust proves the server endpoint only and is never treated as member, Runtime or Site authority. Pipeline evidence from #793 remains owner-provided chart-GitLab executor evidence, and exact commit pinning from #787 is a prerequisite for matching later-step inputs. Neither lane closes the owner ports above, and neither may be reported from the older \#778 mixed run without rerunning its exact integrated revision and inputs. Not established by those checks: separate-host edge deployment in a live distributed environment, real Icarus OS/Python facts, LDAP/SSSD propagation, Waldur/LDAP/Slurm order provisioning, native direct member access-token delivery, real GitLab/LFS stage-in/out, no-ACL Lustre isolation, member removal continuity, systemd production activation, PostgreSQL/broker integration, browser/UAT, cloud cutover or rollback on a real HPC host. Those remain explicit qualification lanes and require owner participation. ## 9. Provider role and offering component ownership Added 1 November 2026 with the production installer's offering contract; this section amends §2 and §3 above. Waldur 8.1.3 accepts a dated usage report only from a user holding `RESOURCE.SET_USAGE` on the offering's organisation or on that organisation's service provider (`PROVIDER_USAGE_SOURCES` in `waldur_mastermind/marketplace/serializers.py`). The token therefore belongs to a `CUSTOMER.OWNER` of the provider organisation or to a `CUSTOMER.MANAGER` of its service provider, not to one scoped offering manager, which holds the permission on the offering and is refused with HTTP `400`. The contract reads the agent's own roles from `/users/me/` because `CUSTOMER.MANAGER` is granted on the service provider and the organisation's team read returns `404` for it. **The role to assign is `CUSTOMER.OWNER` on the provider organisation**, as the laboratory bootstrap does. This deployment disables `customer_manager` (`disabledRoles` in `gitops/manifests/waldur/base/values.yaml`), so the Waldur API refuses to grant it ("Role is not active") and it can only be granted from `waldur shell`. The contract keeps accepting it — Waldur grants the permission on either source — but an administrator who follows this document does not need it. Both accepted roles reach every offering the provider sells, including creating and deleting offerings and approving orders. That is acceptable **only while the provider sells this single HPC offering**; a provider that later sells another offering must move this agent to a narrower deployment or accept the wider authority explicitly. `configure` also owns the offering's `node` component: it runs the upstream `waldur_site_load_components` entry point against the installed Site-agent configuration, so an offering that predates this contract gains the component on its next upgrade. The step is idempotent, creates no plan and leaves the component's `limit_amount` unset, because each order carries its own quantity. The pinned 1.0.8rc9 loader does not send `unit_factor`, so the component it creates reads Waldur's default. Waldur defines `unit_factor` as metadata outside any Mastermind computation, and the conversion that produces node-hours runs in the Site agent from its own `backend_components.node.unit_factor: 60`.
Page source Source snapshot · Revision `f928834c8`.