--- title: 'DD-0055: Member SSH access to Icarus-hpc' --- # DD-0055 — Member SSH access to Icarus-hpc **Related ADR:** [ADR-0055](../decisions/0055-hpc-ssh-key-provisioning.md). **Status:** Draft for review. The dev-ikons laboratory and e2e lane are qualified; Icarus-hpc qualification remains outstanding. **Date:** 2026-10-21. Terminology (Pharos platform, Icarus-cloud, Icarus-hpc, Site agent, access agent, Site Runtime) is defined in ADR-0055. ## 1. Components and accounts ```mermaid flowchart TB subgraph CLOUD["Icarus-cloud"] API["API"] -->|"RabbitMQ"| SA["Metallo ssh_access"] SA --- DB[("pharos_ssh_access
PostgreSQL")] SITEAPI["Metallo Site API
site.<domain>/v1/site/"] --> SA WALDUR["Waldur 8.1.3"] end subgraph HOST["Icarus-hpc service host"] AGENT["pharos-agent
Site agent"] ACCESS["pharos-ssh-access
access agent"] RT["pharos-runtime
Site Runtime"] end subgraph NATIVE["Icarus-hpc services"] LDAP[("OpenLDAP ou=pharos")] SLURM["slurmctld / slurmdbd"] LOGIN["login nodes
SSSD → sshd"] end AGENT -->|"Waldur token"| WALDUR ACCESS -->|"bearer: ssh-access@icarus"| SITEAPI RT -->|"bearer: commands@icarus"| SITEAPI AGENT -->|"cn=pharos-agent
entries, POSIX, groups"| LDAP AGENT -->|"coordinator of pharos"| SLURM ACCESS -->|"cn=pharos-ssh-access
sshPublicKey only"| LDAP LDAP --> LOGIN ``` | Unix user / unit | Writes | Credentials | No authority over | | --- | --- | --- | --- | | `pharos-agent` / Site agent units | LDAP entries, POSIX attributes and groups under `ou=pharos`; Slurm associations as coordinator of `pharos` | Waldur offering token; `cn=pharos-agent` bind | `sshPublicKey` | | `pharos-ssh-access` / `pharos-ssh-access-agent` | `sshPublicKey` values under `ou=pharos` | native Site API bearer for `ssh-access@`; `cn=pharos-ssh-access` bind | entries, POSIX attributes, groups, Slurm, Waldur | | `pharos-runtime` / Site Runtime | nothing in LDAP | native Site API bearer for commands | LDAP, SSH access routes | All connections open from Icarus-hpc. Icarus-hpc exposes no inbound port to Pharos. ## 2. Flows ### 2.1 Account provisioning (upstream Waldur and Site agent) Verified against Mastermind 8.1.3 [`marketplace/handlers.py`](https://github.com/waldur/waldur-mastermind/blob/8.1.3/src/waldur_mastermind/marketplace/handlers.py) and the Site agent [`ldap` plugin](https://github.com/waldur/waldur-site-agent/tree/main/plugins/ldap). ```mermaid sequenceDiagram actor PM as Project manager participant W as Waldur participant AG as pharos-agent participant L as OpenLDAP participant S as Slurm PM->>W: Order (Create) for HPC Icarus W->>W: Order PENDING_PROVIDER or EXECUTING W->>W: offering user per Project member
username = site_username, POSIX IDs from pool, state OK AG->>W: order_process AG->>S: create Slurm account under pharos AG->>W: Resource OK, backend_id AG->>W: membership_sync / event_process AG->>L: create entry (posixAccount, ldapPublicKey, ...) AG->>S: add user association PM->>W: remove Project role AG->>S: remove user association Note over L: entry stays enabled
(no offering-user auto-deletion) ``` 1. A Project gets a Resource of the **HPC Icarus** offering through an Order (Create). When the Order reaches `PENDING_PROVIDER` or `EXECUTING`, Waldur creates an offering user for every Project member. A member who later receives a Project role in that Project also gets an offering user. Both require the offering's `service_provider_can_create_offering_user` plugin option. 2. With the `service_provider` username policy, Metallo writes the username the member holds at the Site (the Party's `site` binding) onto the offering user, and POSIX IDs come from the provider's pool. Waldur does not derive the name from a claim, and the Site agent never generates one. 3. The Site agent's `order_process` mode creates the Resource's Slurm account under `pharos`. Its `membership_sync` or `event_process` mode lets the account-name plugin's LDAP backend create the entry (with `ldapPublicKey` in `user_object_classes`) under that name and pin the UID it got on the offering user (`set_posix_attributes`), and the `slurm` plugin add the user's association. 4. Losing a Project role removes that association. Offering-user auto-deletion is **not** enabled, so the offering user is not released and the LDAP entry stays enabled (ADR-0055 decision 7). The upstream `on_departure` setting has only `disable` and `delete`; confirm in the lab that no release occurs without auto-deletion. `OK` in Waldur therefore does not prove that the LDAP entry exists. The access agent tolerates that interval (§5.3). ### 2.2 Submitting and revoking a key ```mermaid sequenceDiagram actor M as Member participant API as API participant SA as ssh_access participant AA as pharos-ssh-access participant L as OpenLDAP participant LN as login node M->>API: POST /keys API->>SA: SubmitMemberKey (actor from session) SA->>SA: gates, key checks, duplicate check
pending_deployment, generation n SA-->>M: 201 key record loop until acknowledged AA->>SA: pull (Site from credential) SA-->>AA: member, username, uidNumber, key n present AA->>L: search uid, check uidNumber alt entry missing or mismatched AA->>AA: nothing written, retry later else entry matches AA->>L: MODIFY_ADD sshPublicKey AA->>L: read back AA->>SA: acknowledge applied (key, n) end end SA->>SA: after propagation window: deployed M->>LN: ssh LN->>L: sss_ssh_authorizedkeys (cached) M->>API: POST /keys/{id}/revoke API->>SA: RevokeMemberKey SA->>SA: pending_revocation, generation n+1 AA->>SA: pull AA->>L: MODIFY_DELETE sshPublicKey, read back AA->>SA: acknowledge removed (key, n+1) SA->>SA: after propagation window: revoked ``` | Event | Pharos (`ssh_access`) | Icarus-hpc | | --- | --- | --- | | Member leaves every HPC Project | Gates report `not_eligible`; existing keys stay | Slurm association removed; login still works | | AUP change | Durable bulk revocation of affected keys | Keys removed as in a member revocation | | Party suspended (`disabled` or authentication-ineligible) | Durably revoke existing keys; preserve the Party and Site bindings | Remove managed keys from LDAP | | Party reinstated | Allow explicit resubmission after current AUP, Project and account checks; never reinstall keys automatically | Install only a newly authorised key generation | | Site binding retired for `security` | Durably revoke keys authorised through that binding; retain its retirement record | Remove those managed keys from LDAP | Suspension is reversible. Site-binding retirement is terminal, as defined by [Party Governance](../../services/beto/contexts/party_governance/README.md#domain-authority). Suspending a Party preserves its Site bindings. Reinstating the Party does not reverse a binding's retirement or reinstate old SSH authorisations. Each submitted key records the member authority version and Site-binding ID that authorised it. A later authority version invalidates the earlier grant even if Metallo first observes the reinstated state after an interruption. Beto owner events wake this check; a periodic sweep recovers missed wake-ups and restarts. Confirmed withdrawal can revoke keys independently of AUP availability. An unavailable owner query defers the decision. Removal follows the existing Site acknowledgement and propagation-window rules. It prevents subsequent authentication with the managed key; it does not terminate SSH sessions that are already open. Member-initiated revocation remains ungated by HPC eligibility, AUP or account readiness; the API still requires an authenticated Party. Automatic withdrawal handles suspended users who cannot sign in. ## 3. `ssh_access` context (Metallo) ### 3.1 Aggregate and states `MemberSshAccess` is one aggregate per `(member_id, site)`. Its children are the member's keys. One root per member makes per-member fingerprint uniqueness and "one submission per member at a time" atomic with the root version check. Children have no separate write repository. | Field | Notes | | --- | --- | | `member_id` | Canonical member ID (Beto PartyId once available); never email or username | | `site` | Configured Site ID, e.g. `icarus` | | `target` | Last known Site identity: `username`, `uid_number`, source revision; opaque to the domain | | `keys[]` | `key_id` (stable UUID), `name`, `value` (exact line), `fingerprint` (SHA256), `algorithm`, `bits`, `state`, `comment`, `generation`, `submitted_at`, `deployed_at`, `rejected_at`, `revoked_at`, `applied_evidence`, `access_grant` (member authority version and Site-binding ID) | | `revision` | Root version for optimistic concurrency and projection ordering | ```mermaid stateDiagram-v2 [*] --> pending_deployment: submit pending_deployment --> deployed: applied + window pending_deployment --> rejected: permanent refusal pending_deployment --> pending_revocation: revoke, acknowledgement unknown deployed --> pending_revocation: revoke or AUP change pending_revocation --> revoked: removed + window revoked --> pending_deployment: resubmit (same key_id) rejected --> [*]: delete revoked --> [*]: delete ``` Every transition replaces or clears `comment`. Each submission, resubmission or revocation increments the key's `generation`; Site results for an older generation change nothing. A missing acknowledgement never proves that an add did not reach LDAP: revocation waits for a `removed` result and the propagation window before it reports `revoked`, so delete and resubmit are unavailable until then. Repeated revoke or resubmit returns the key as is, per #678. ### 3.2 Commands and queries | Operation | Kind | Rules | | --- | --- | --- | | `GetMemberHpcAccess` | query | Read model with every applicable disabled reason in #678 order; never fails on gates | | `ListMemberKeys`, `GetMemberKey` | query | Caller's keys only; deleted key is not found | | `ValidateMemberKey` | query | Same gates and checks as submission; records nothing; no submission lock | | `SubmitMemberKey` | command | Gates, then key checks, then duplicate check in the root transaction | | `RevokeMemberKey` | command | Never gated; idempotent per #678 | | `ResubmitMemberKey` | command | Gates first; same `key_id`; stored line rechecked | | `DeleteMemberKey` | command | Only `rejected` or `revoked` | | `PullSiteKeyWork`, `GetSiteKeySnapshot` | Site query | Authorised Site only (§4) | | `RecordSiteKeyResults` | Site command | Generation-fenced; idempotent per `(key_id, generation, outcome)` | | `CompletePropagation` | internal | Owner worker moves applied keys to `deployed`/`revoked` after the window | | `RevokeForAccessChange` | internal | Revoke observed key generations whose member authority or Site binding was withdrawn | | `RevokeForAupChange` | internal | One command per affected root, scheduled durably | Key checks are one shared implementation used by validate, submit and resubmit: public key only (private material rejected without echoing it), no control characters except tab, `ssh-ed25519` or `ssh-rsa` of at least 3072 bits, server-side fingerprint/algorithm/bits, per-member limit (10 proposed). ### 3.3 Ports | Port | Owner integration | Future source evolution | | --- | --- | --- | | `MemberSiteIdentityPort` → username, UID, revision | Beto Site-binding projection combined with Resource Provisioning's query of verified Waldur offering-user account reports | An account owner's versioned facts may feed a local projection | | `HpcEligibilityPort` → HPC Projects | Beto Project/membership/authority projections and current `project.view` query, combined with a usable HPC allocation at the requested Site | Resource events may replace the allocation query | | `AupPort` → accepted and current versions | Retained Themelio's private query for the platform AUP, using a separate SSH reader credential | The eventual policy owner supplies versioned acceptance and required-policy events | | `Clock`, `IdGenerator` | standard | — | Projections built from Beto events are owned by `ssh_access`, stored in its database with receipts, and fenced by source revision. Eight durable subscriptions and checkpoint queries supply initial loading and repair. The canonical Party is the API actor and browser principal. Fixtures remain an explicit development option. Numeric UID and AUP acceptance use owner queries during this transition. Resource Provisioning already reconciles Waldur offering users against Beto Site bindings; its fresh account reports supply the numeric UID. Retained Themelio is the current platform AUP authority. Waldur Terms of Service remain separate. Missing or stale authority defers new access. A periodic worker rechecks held keys and durably revokes those whose member has not accepted the current AUP. Confirmed absence of a Party or current policy binding is distinct from an owner outage: the required policy version remains readable independently of that binding, so absence cannot leave revocation retrying indefinitely. Revocation commands identify the observed key generations, allowing a later resubmission to be revoked again while protecting newer generations from delayed work. Future events require an authoritative owner, ordered revisions, explicit absence semantics and snapshot repair before replacing these adaptors. Implementation details and qualification boundaries are in the [SSH Access context](../../services/metallo/contexts/ssh_access/README.md#owner-inputs). ### 3.4 Persistence Private PostgreSQL database `pharos_ssh_access`, restricted runtime role, migrations applied by administration (`scripts/migrate.py`), following [Site Coordination](../../services/metallo/contexts/site_coordination/README.md)'s aggregate and receipt pattern: - root and children (`pharos.aggregates` kind `member_ssh_access`, child table for keys); - command receipts and recorded outcomes; - a pending Site work index: `(site, member_id, root revision)` for roots with unacknowledged generations, updated in the root transaction; - Site result receipts keyed by `(key_id, generation)`; - Beto event receipts and projections. Browser projection bytes and one principal-targeted publication intent per actor commit with every root revision. ## 4. Site protocol Served by the existing Metallo Site API listener ([`site_api.py`](../../services/metallo/services/site_api.py)) on dedicated routes, declared in a new HTTP contract next to [`site-coordination.v1.yaml`](../../contracts/http/site-coordination.v1.yaml). | Route | Body | Reply | | --- | --- | --- | | `POST /v1/site/ssh-access/pull` | `limit`, optional page token | Members with pending work: `memberRef`, `username`, `uidNumber`, `revision`, keys with `keyId`, `generation`, `desired` (`present`/`absent`), `fingerprint`, `publicKey` for `present` | | `POST /v1/site/ssh-access/snapshot` | page token | Complete desired state for every member of the Site, revision-pinned, for drift reconciliation | | `POST /v1/site/ssh-access/acknowledge` | `memberRef`, `revision`, results: `keyId`, `generation`, `outcome` (`applied`, `removed`, `rejected`), `reason`, `observedAt` | `recorded`, `replayed` or `stale` per result | ```mermaid flowchart LR REQ["request + bearer"] --> V{"native verifier:
digest, expiry,
revocation"} V -->|"invalid"| E401["401"] V --> G{"grant covers
route scope?"} G -->|"no"| E403["403"] G -->|"ssh-access@site"| H["ssh_access handler
Site from grant"] G -->|"commands@site"| C["site_coordination handler"] ``` - The Site is derived from the credential, never from the body. `memberRef` is an opaque per-Site reference, not the canonical member ID. - Pull is idempotent and stateless for the agent: pending work stays pending until acknowledged. The agent needs no cursor or local journal. - Authentication uses a native bearer: at least 256 random bits, digest and grant stored on the Site API side, grant bound to principal, Site, route scope, expiry and revocation, verified on every request. The verifier is shared with the native Site Runtime. A Runtime credential is refused on SSH routes and an access-agent credential on command routes. - Error mapping follows [Secure Site communication](../../services/metallo/docs/epics/instantiate-and-run/secure-site-communication.md): `401`, `403`, `409`, `422`, `503`, with redacted bodies. - Icarus-cloud exposes the listener on a separate Site API host, for example `https://site.staging.pharosfactory.ai`. The client appends `/v1/site/ssh-access/*`; the edge routes that hostname and prefix to the Site API. Browser API and Site API remain separate hosts and backends with separate identities and credentials. Native agents use scoped bearers and require no OIDC redirect or callback registration. ## 5. Access agent ### 5.1 Packaging and configuration A Site-side composition under `services/metallo/services/` with its adaptor in the `ssh_access` context, packaged for systemd. Unit `pharos-ssh-access-agent`, user `pharos-ssh-access`, `NoNewPrivileges=yes`, `ProtectSystem=strict`, `PrivateTmp=yes`, credentials through `LoadCredential=`. Configuration: Site API URL and CA file; bearer credential file (read on each request); LDAP URI (`ldapi://` with SASL EXTERNAL on the directory host, or `ldaps://` with CA, bind DN and password file); people base `ou=people,ou=pharos,…`; pull interval; snapshot interval; backoff bounds. For `ldapi`, the directory maps the service user's exact Unix peer credentials to the restricted access-agent DN; no LDAP password is loaded. See the [agent packaging guide](../../services/metallo/deploy/ssh-access-agent/README.md). ### 5.2 LDAP operations Only these operations exist; there is no generic command execution. - Search `uid=` under the people base, attributes `uid`, `uidNumber`, `entryUUID`, `entryCSN`, `objectClass`, `sshPublicKey`. - `MODIFY_ADD sshPublicKey: ` includes `MODIFY_ADD objectClass: ldapPublicKey` when the class is absent. Both changes are one atomic modify. - `MODIFY_DELETE sshPublicKey: ` includes deletion of that class when a fresh read shows this is the last value. Both changes are one atomic modify. - Every modify has a critical RFC 4528 assertion for `uidNumber`, `entryUUID` and the freshly read `entryCSN`. Concurrent changes retry from a fresh read; duplicate attribute/class results do not prove that the atomic key change was applied. Both `MUST sshPublicKey` and `MAY sshPublicKey` schemas are supported. - Read back the same entry. Values are matched by fingerprint. The agent removes only values whose fingerprint Pharos holds for that member; it never removes or rewrites other values. ### 5.3 Algorithm ```mermaid flowchart TD P["pull or snapshot page"] --> M["next member"] M --> F{"entry uid=username
found once?"} F -->|"no"| R["leave pending
retry with backoff"] F -->|"yes"| U{"uidNumber matches?"} U -->|"no"| A["write nothing
operational alert"] U -->|"yes"| K["apply desired presence
per key fingerprint"] K --> B["read back"] B --> OK{"matches desired?"} OK -->|"yes"| ACK["acknowledge applied / removed"] OK -->|"permanent LDAP refusal"| REJ["acknowledge rejected + reason"] OK -->|"LDAP or network failure"| R ``` Treat "already present" and "already absent" as success. The snapshot pass repeats the same steps for every member, which recovers a crash between an LDAP change and its acknowledgement and any drift in managed values. ### 5.4 Propagation window LDAP read-back proves the directory state, not the login path. `ssh_access` keeps an applied key in `pending_deployment` (or a removed key in `pending_revocation`) until `observedAt + propagation_window`. The window is configuration, set from the lab measurement and the Icarus-hpc SSSD settings; the SSSD default `entry_cache_timeout` is 5400 s. The Icarus-hpc administrators accepted a 120 s window on 23 October 2026. A revocation that remains pending beyond its alert budget raises an operational alert. ## 6–10. Browser surface, laboratory and delivery The browser API and its live projection (§6), the `metallo-ikons-1` laboratory (§7), verification (§8), delivery (§9) and the open questions (§10) are in [DD-0055 — browser surface, laboratory and delivery](0055-hpc-ssh-key-provisioning-browser-and-operations.md). Section numbers continue there unchanged.