---
title: 'DD-0059: Production Icarus-hpc provisioning and run installation'
---

# DD-0059 — Production Icarus-hpc provisioning and run installation

**Related ADR:** [ADR-0059](../decisions/0059-icarus-hpc-provisioning-run-installation.md).

**Status:** Draft. Deterministic installation and gate implementation is in review;
production activation and native qualification are outstanding. The installer is plain
bash since 3 November 2026 (§1–§4, §8); the earlier Ansible role is removed.

**Date:** 2026-10-25. Amended 2026-11-03.
## 1. Installation boundary

The operator entry point is `pharos-hpc`, one plain-bash script with three sourced libraries
(`lib/common.sh`, `lib/check.sh`, `lib/install.sh`, `lib/service.sh`) that ships inside the
release archive. It runs as root on the login node; Python runs only the agents, inside their
own virtual environments. There is no Ansible controller, inventory, `become` or PyPI-fetched
tooling between the administrator and the host, because the HPC administrators who operate this
site read and support the installer themselves.

```sh
sudo ./pharos-hpc check                           # read-only
sudo ./pharos-hpc install                         # users and one release
sudo ./pharos-hpc configure [TOKEN_DIRECTORY]     # configuration, secrets, offering components
sudo ./pharos-hpc activate site-agent ssh-access  # link, units, start, health
sudo ./pharos-hpc status | rollback [RELEASE_ID] | stop
```

| Command | Effect |
| --- | --- |
| `check` | Validate the release checksums, `site.env`, the private inputs, the host Python and OS, the unit files and the read-only Waldur offering contract. Changes nothing. |
| `install` | Create the two service identities without shells or homes and stage one immutable release with offline environments. Start nothing. |
| `configure` | Write the agents' configuration from `site.env`, adopt the token files unchanged, publish the public CA files, and reconcile the offering's components with the Site agent's configuration. |
| `activate` | Validate the installed release again, record `previous`, point `current` at it, install its unit files, start the selected units and require health. A failure restores the recorded predecessor. |
| `rollback` | The same path with `previous`, or with a named installed release. |
| `stop` | Stop and disable this site's agents; keep releases, configuration and secrets. |

The operator-facing procedure is the [HPC Site runbook](../../devops/hpc/site/README.md).

## 2. Inputs, configuration and exact artifacts

One file, `/etc/pharos-hpc/site.env`, carries every site value as `KEY=value`. The installer
parses it with a strict `KEY=value` reader: it is never sourced, values are never substituted,
duplicate keys and unknown syntax are rejected, and no name arrives from the calling
environment. It replaces the former public profile, the site-install file and the `hpc_site`
inventory. `site.env.example` carries every key with a comment.

The real file must use URLs without embedded credentials, keep plaintext LDAP and `slurmrestd`
on host loopback, use different compute/staging partitions and allocate unique, non-system
service UIDs/GIDs outside approved member ranges. `CENTRAL_OFFERING_UUID` is 32 lowercase hex
characters. `CENTRAL_SITE_API_URL` is an origin only: the agents append `/v1/site/commands/*`
and `/v1/site/ssh-access/*` themselves, and the edge routes the host and prefix to the Site API
backend. Browser API and Site API remain separate services with different hosts and
credentials.

`ACTIVATION_COMPONENTS` is validated against the release manifest's `components:` line. The
Release does not ship `pharos-slurm-runtime`; a component the manifest does not list cannot be
activated, so the Runtime is absent rather than half-configured until its owner connections
land.

Private inputs are files below `/etc/pharos-hpc/secrets`, never values in the configuration and
never command arguments:

| File | Mode | Authority |
| --- | --- | --- |
| `site-agent-token` | 0400 `root` | `CUSTOMER.OWNER` on the provider organisation, or `CUSTOMER.MANAGER` on its service provider |
| `site-agent-ldap-password` | 0400 `root` | Restricted `cn=pharos-agent` bind |
| `ssh-access-site-token` | 0640 `root:pharos-ssh-access` | SSH access Site API routes only |
| `site-agent-ca.crt`, `site-api-ca.crt` | 0644 | Public trust for Waldur and the Site API edge |

`configure` accepts the directory the central step writes and copies those files unchanged,
setting the modes above. It publishes the two CA files again as `/etc/pharos-hpc/site-agent-ca.crt`
and `/etc/pharos-hpc/site-api-ca.crt`, because the agents run unprivileged and cannot read the
root-only secrets directory. Changing a token is one `configure` run and needs no new release.

Platform maintainers produce one archive with `scripts/hpc_site_release.py`; HPC administrators
do not build the wheel, derive requirements or reconstruct a source checkout. The archive
contains `pharos-hpc`, `lib/`, `units/`, `site.env.example`, `README.md`, the Metallo wheel, the
hash-pinned requirements files named by the manifest, the release manifest and `SHA256SUMS`.
`check`, `install`, `activate` and `rollback` run `sha256sum --check` over the bundle and over
the installed release; `install` records a fingerprint of the manifest and `site.env` and
refuses to reuse a release ID with different inputs. Official archives are Sigstore-signed by
the release workflow.

The release states `dependency_mode: index` or `dependency_mode: wheelhouse`. Index mode
requires an approved HTTPS `PYTHON_INDEX_URL`; wheelhouse mode carries every locked wheel and
installs with `--no-index --find-links`. Both modes run pip isolated with `--require-hashes`,
`--no-deps` and `--only-binary=:all:`. Neither mode supplies the target Python 3.13
interpreter, OS packages or LDAP/Slurm.

## 3. Component layout and privilege matrix

| Component | Stable path | Unit | Allowed authority |
| --- | --- | --- | --- |
| Upstream Waldur Site agent | `/opt/pharos-hpc/site-agent` | `waldur-site-agent@<mode>` | Outbound Waldur HTTPS; restricted LDAP account/POSIX/group writes; Slurm coordinator operations for the Pharos tree |
| SSH access agent | `/opt/pharos-hpc/ssh-access` | `pharos-ssh-access-agent` | Outbound Site API HTTPS; `sshPublicKey` writes and identity reads only |
| Native Slurm Runtime | `/opt/pharos-slurm-runtime` | `pharos-slurm-runtime` | Not part of this release; see §6 |

The stable paths are symlinks through `/opt/pharos-hpc/current`, which points at one immutable
`/opt/pharos-hpc/releases/<release-id>/` directory. Both component links live inside
`/opt/pharos-hpc`, so a host that still carries an older `/opt/pharos-ssh-access-agent` directory
is never written into. Code lives there; configuration, secrets and public trust stay in
`/etc/pharos-hpc`, so a token change does not create a release and a rollback does not touch
configuration.

The unit files are final files, not templates: they read
`EnvironmentFile=/etc/pharos-hpc/<component>.env` and take the access bearer through
`LoadCredential=`. They name the fixed accounts `pharos-agent` and `pharos-ssh-access`; the
site chooses UIDs and GIDs. Units have `NoNewPrivileges`, an empty capability set and no root
user, and the Site agent owns a private `StateDirectory`. Neither receives platform
PostgreSQL, RabbitMQ, Kubernetes, LDAP administrator or Slurm administrator authority.

`SITE_AGENT_MODES` selects `order_process`, `membership_sync` and `report`. `report` is required
rather than optional: the `node` component is limit-billed with period `total`, so the periodic
report is the only path by which consumption reaches Waldur, and `check` rejects a configuration
without it. No mode is inferred from a running host.

## 4. Activation, health and recovery

Activation validates the installed release's `SHA256SUMS`, requires `configure` to have run,
records the resolved `current` target as `previous`, points `current` at the release, installs
the release's unit files, then starts and enables the selected units. Success is recorded in
`/var/lib/pharos-hpc/active-release` only after every selected unit reports `active` and:

- the offering reads `Active` in Waldur through the agent's own token and CA;
- the Site API answers over TLS against the configured CA.

Both probes use `curl` without `--location`, so a redirect never carries the scoped bearer to
another host. A failing step restores `previous`, re-installs its unit files, restarts the same
units and reports that the earlier release is active again, all from the two symlinks and
`/var/lib/pharos-hpc/active-release`; no transaction journal or separate helper is involved.
The operation never deletes retained releases, identities, journals, workspaces or Slurm jobs.

## 5. Icarus native prerequisites

These are reviewed administrator operations and are not hidden installer tests:

1. Record and approve the actual host OS, architecture, NSS and Python 3.13+
   interpreter. Debian 12's system Python alone is not sufficient for Metallo.
2. Preserve existing Slurm and OpenLDAP installations and unrelated identities.
   Validate UID/GID collisions before creating service identities. The approved Icarus
   numbers are split by owner: member UIDs `5000000–5499999`, members' primary GIDs
   `5500000–5549999` from Waldur's POSIX pool, and project-group GIDs
   `5550000–5599999` from the Site agent. The installer writes only the project-group
   range and `check` refuses it if it overlaps the members' primary GID range.
3. Apply the agreed `ou=pharos` schema and one-writer-per-attribute ACLs. This
   installer's access agent uses `ldapi` with SASL EXTERNAL; its owner package can
   also construct an explicitly credentialed TCP connection, but the installer does
   not yet accept that separate DN, password and CA input contract. The upstream
   agent keeps its documented loopback simple bind.
4. Configure SSSD/sshd key lookup, cache/propagation policy and the 120-second access
   window. Key revocation does not terminate an already established SSH session.
5. Configure Slurm accounting, `pharos-agent` and `pharos-runtime` coordinator
   rights, admission rules that prevent service submissions, member JWT verification
   and explicit definite rejection codes. The coordinator identity lets the Runtime
   observe, account and cancel without a member token; members' own tokens submit.
   It reaches the site with the Slurm client over munge, so the Runtime host needs
   munge and the Slurm binary directory on the unit's path rather than a credential.
6. Configure Lustre homes, quotas, scratch retention, staging/compute partitions and
   outbound GitLab/LFS/S3/CA reachability. The 23 October answers in #643 already
   selected option B: member-owned stage-in and stage-out jobs use an
   Internet-enabled staging partition while compute nodes remain offline. Still to
   be supplied are its concrete partition name/limits and confirmation of staging
   egress. Do not reopen the movement profile or ask for filesystem ACLs.
7. Run a deliberate canary Project/member. Readiness checks never create production
   accounts or jobs as an undocumented side effect.

`site.env` carries `NATIVE_SSH_ACCESS_LDAP_URI` and `NATIVE_SITE_AGENT_LDAP_URI`
separately. The installer permits only `ldapi://` for the access agent, which selects SASL
EXTERNAL and rejects a bind DN, password or CA; authority is the exact Unix peer
identity. TCP LDAP/LDAPS remains a capability of the separately packaged adaptor,
not a selectable installer setting until its complete credential contract is
implemented and qualified. The pinned upstream
`waldur_site_agent_ldap==1.0.7` does not support that authentication path. Its
schema requires `bind_dn` and `bind_password`, and `LdapClient._connect()` passes
them to `ldap3.Connection` without an authentication or SASL option. With the
pinned `ldap3==2.9.1`, that selects SIMPLE when a user is present. Its URI must
therefore remain loopback `ldap://`; `ldapi://` alone would change transport but
not request SASL EXTERNAL. Removing the password requires an upstream change and
   a separate compatibility qualification.

The upstream LDAP username backend is authoritative only when the dedicated
offering has `username_generation_policy=service_provider` and
`service_provider_can_create_offering_user=true`. `check` reads those fields
through the same provider credential and Unix identity as the Site agent. An `identity_claim` offering is rejected before activation because Waldur,
not the agent, would own username generation and the expected POSIX/Slurm member
would never be created. The generic provider-plan route is not a health signal for
this non-staff identity; use the nested public-offering plan route recorded in the
operator runbook.

## 6. Member-owned no-ACL workspace implementation

The Icarus answers recorded by #643 are the accepted Site design, not an open ACL
or staging decision. Lustre has no filesystem ACLs. Icarus supplies option B: a
separate Internet-enabled staging partition, offline compute nodes, and stage-in,
compute and stage-out jobs that all run as the member. The member's job creates its
workspace with `umask 077`, mode `0700`, and owns its files from the start. The
Runtime orchestrates Slurm and does not clone, stage or clean repository files as
`pharos-runtime`.

The laboratory now pre-provisions `/scratch/pharos/<member>` as a member-owned
mode-`0700` root and has the member's stage-in job create the Attempt and workspace
with `umask 077`; it does not use `setfacl` for either. Stage logs sit beside that
pre-existing root, because Slurm otherwise creates a missing Attempt parent before the
member script and defeats the private mode contract. The Runtime keeps only the planned
path and cannot read or write the member-private workspace. The temporary Git
credential hand-off still uses a narrow `setfacl` double under
`/scratch/pharos/.runtime`, a runtime-owned mode-`0711` root. That is a qualification
limitation, not a reason to request ACLs at Icarus or to reproduce its cross-UID
inspection/deletion model in production. The remaining implementation is:

- stage-in, compute and stage-out run under the member identity and cannot read
  another member's workspace;
- a just-in-time repository credential is delivered to the member's job through an
  owning credential broker, revoked after use, and is never retained by the Runtime;
- the Runtime verifies only the workspace facts that the Run protocol needs and does
  not require broad traversal of member-private files;
- working-directory retention and deletion follow the agreed member ownership,
  cleanup policy and Site scratch purge; the laboratory Runtime removes only its
  credential hand-off and never traverses or deletes the member Attempt, with any
  separately scoped cancellation or cleanup authority audited;
- no production composition imports the `dx/` provider double, grants sudo to a
  service identity or makes `/scratch/pharos` globally readable or writable.

Runtime activation stays blocked until the temporary credential delivery/revocation
owner is connected and the complete member-owned flow is tested against native
Lustre. The profile's accepted workspace-contract marker is a gate, not evidence by
itself. Native Lustre validation is our outstanding implementation qualification;
the Site architecture and ACL facts are already settled.

## 7. Continuity after member removal

The following matrix records what must be decided, not a claim that it is implemented:

| Operation | Proposed outcome after initiator leaves | Required owner input |
| --- | --- | --- |
| Observe submitted Slurm job | May continue; the Runtime reads it as Slurm coordinator | Coordinator role, saved native identity and Project/Run observer authority |
| Cancel/cleanup | May be performed by a current authorised member or operator; policy cancellation uses the coordinator role | Separate scoped cancel/cleanup authority and actor audit |
| Stage-out or later Pipeline step | Does not continue without the member's own current token | Current Project execution authority and a live offline consent |
| Retry | Refused without current authority | Fresh binding and fresh bounded credential |
| Repository credential | Its grant follows Run/repository policy, never a revoked personal token | Explicit issuance/revocation and continuity contract |
| Workspace/log/output access | Departed member loses access; current members follow agreed permissions | Workspace/results permission boundary and audit actor |

Membership removal is not automatic Run cancellation. Existing jobs continuing after
creator departure is danias's current proposal, not a final contract. `project.view`,
`runs.view`, `runs.output.view` and `runs.cancel` are suggested names only.

The same matrix covers a member whose offline Session ends while the Run is still
executing, which is a different cause and a different answer. Observation, accounting
and cancellation do not use the member's token at all: they run as the Site's Slurm
coordinator over munge, so a Run keeps being watched and can be cancelled. Stage-out
and every later step that must run as the member do need a live consent; the Site API
answers with one fixed reason and the Attempt must then end with a terminal outcome
rather than wait for ever.

The periodic consent refresh that keeps an idle Run inside the issuer's idle window is
implemented in the Member Credentials owner (#879): one worker refreshes every `active`
consent of a member who still holds Project execution authority once its idle deadline
falls inside the next two refresh intervals, under the same per-member rotation lock a
native action takes, and keeps no access token. It never refreshes a `revoked`,
`superseded`, `lost` or `expired` consent, nor one of a member who left the Project, and
it cannot move a deadline past the issuer's maximum lifespan, which the issuer reports.
The terminal Attempt outcome for a Session that can no longer be used, and the notice
that asks the member to sign in again, remain tracked as one issue against #766 and #636.
The offline window itself is ADR-0035.

## 8. Verification and deferred lanes

`scripts/test_hpc_site_installation.py` runs the installer against a temporary root
(`PHAROS_HPC_ROOT`) with recording shims for `useradd`, `groupadd`, `getent`, `systemctl`,
`systemd-analyze`, `curl` and the venv's console scripts. It covers a missing secret before
any change, an unchanged `check`, the offline release and its fingerprint, configuration and
token adoption, both LDAP mechanisms, the offering component reconciliation, activation and
the `current`/`previous` links, rollback, a failed activation that restores the predecessor's
unit files, a tampered release, an unavailable component, and probes that never follow a
redirect. `scripts/test_hpc_site_release.py` covers the archive contents. `shellcheck -x` and
the release build run in `pnpm verify`; `scripts/shellcheck.py` fetches the pinned release
(0.11.0, SHA-256 checked) into the tool cache on first use, so the check runs on the ARM64
runners and on a machine that never installed ShellCheck.

The laboratory lane installs the candidate bundle on the `icarus-hpc` development box over a
host that already carries the Phase 1 LDAP, Slurm and NSS prerequisites, then runs its e2e
script (account creation, key deployment, submit and revoke inside 120 seconds), a rollback
and an activation that is made to fail. The laboratory is restored afterwards with its own
installer. That lane proves the script against real systemd, real accounts and real LDAP; it
does not qualify the production host.

The Site API edge qualification must exercise the separate staging hostname and the
equivalent devbox endpoint before staging cut-over:

- `/v1/site/commands/*` and `/v1/site/ssh-access/*` reach the Site API on their
  dedicated hostname with the configured route-specific timeouts and body limits;
- missing, malformed, expired and revoked native grants, wrong route scopes and
  cross-Site attempts return the expected authentication failures rather than a web
  response or browser-login redirect;
- valid Runtime and SSH access grants complete native pull and acknowledge round
  trips through hostname- and CA-verified TLS;
- the existing `site.pharos.<dev-domain>` alias is reached through the established
  devbox edge/tunnel, with its certificate SAN and public CA verified even when the
  test caller resolves the host to loopback;
- `site.staging.pharosfactory.ai` passes equivalent DNS, TLS, SNI, routing and
  authentication checks once the distributed owner and DNS owner publish it.

The agent's native `psb_` grant is the authentication control. TLS trust proves the
server endpoint only and is never treated as member, Runtime or Site authority.

Pipeline evidence from #793 remains owner-provided chart-GitLab executor evidence,
and exact commit pinning from #787 is a prerequisite for matching later-step inputs.
Neither lane closes the owner ports above, and neither may be reported from the older
#778 mixed run without rerunning its exact integrated revision and inputs.

Not established by those checks: separate-host edge deployment in a live distributed
environment, real Icarus OS/Python facts, LDAP/SSSD propagation,
Waldur/LDAP/Slurm order provisioning, native direct member access-token delivery, real GitLab/LFS
stage-in/out, no-ACL Lustre isolation, member removal continuity, systemd production
activation, PostgreSQL/broker integration, browser/UAT, cloud cutover or rollback on a
real HPC host. Those remain explicit qualification lanes and require owner
participation.

## 9. Provider role and offering component ownership

Added 1 November 2026 with the production installer's offering contract; this section
amends §2 and §3 above.

Waldur 8.1.3 accepts a dated usage report only from a user holding
`RESOURCE.SET_USAGE` on the offering's organisation or on that organisation's service
provider (`PROVIDER_USAGE_SOURCES` in `waldur_mastermind/marketplace/serializers.py`).
The token therefore belongs to a `CUSTOMER.OWNER` of the provider organisation or to a
`CUSTOMER.MANAGER` of its service provider, not to one scoped offering manager, which
holds the permission on the offering and is refused with HTTP `400`. The contract
reads the agent's own roles from `/users/me/` because `CUSTOMER.MANAGER` is granted on
the service provider and the organisation's team read returns `404` for it.

**The role to assign is `CUSTOMER.OWNER` on the provider organisation**, as the
laboratory bootstrap does. This deployment disables `customer_manager`
(`disabledRoles` in `gitops/manifests/waldur/base/values.yaml`), so the Waldur API
refuses to grant it ("Role is not active") and it can only be granted from
`waldur shell`. The contract keeps accepting it — Waldur grants the permission on
either source — but an administrator who follows this document does not need it.

Both accepted roles reach every offering the provider sells, including creating and
deleting offerings and approving orders. That is acceptable **only while the provider
sells this single HPC offering**; a provider that later sells another offering must
move this agent to a narrower deployment or accept the wider authority explicitly.

`configure` also owns the offering's `node` component: it runs the upstream
`waldur_site_load_components` entry point against the installed Site-agent
configuration, so an offering that predates this contract gains the component on its
next upgrade. The step is idempotent, creates no plan and leaves the component's
`limit_amount` unset, because each order carries its own quantity. The pinned 1.0.8rc9
loader does not send `unit_factor`, so the component it creates reads Waldur's
default. Waldur defines `unit_factor` as metadata outside any Mastermind
computation, and the conversion that produces node-hours runs in the Site agent from
its own `backend_components.node.unit_factor: 60`.