# 0055 — Member SSH access to Icarus-hpc

Status: Proposed. Date: 21 October 2026. The implementation is in
[DD-0055](../dd/0055-hpc-ssh-key-provisioning.md).
Deciders: ikons; review by Metallo owners (dkatsimpris) and the Icarus-hpc
administrators (jimsiak).

This decision builds on the September prototype
([#137](https://github.com/pharos-ai-factory/pharos-platform/issues/137), Approach C).
It keeps the prototype's fingerprint identity, lock ordering, withdrawal handling
and compensation for an add that lands after its withdrawal. It moves key ownership
from the Waldur offering user to the member, in a Metallo context, and delivers
changes to Icarus-hpc through the Site API instead of a CSV queue.

## Terminology

| Term | Meaning |
| --- | --- |
| **Pharos platform** | Our platform: this repository's services and contracts. |
| **Icarus-cloud** | The Pharos platform installation that serves Icarus. |
| **Icarus-hpc** | The isolated HPC infrastructure, administered by the NTUA HPC administrators. |
| **Site agent** | Upstream `waldur-site-agent` with the `ldap` and `slurm` plugins, run as `pharos-agent` on Icarus-hpc. |
| **Access agent** | `pharos-ssh-access-agent`, run as `pharos-ssh-access` on Icarus-hpc. It changes member SSH public keys in LDAP. |
| **Site Runtime** | The existing Metallo Runtime, run as `pharos-runtime` on Icarus-hpc. |

```mermaid
flowchart LR
    subgraph HPC["Icarus-hpc (NTUA administered)"]
        direction TB
        SLURM["Slurm"]
        AGENT["pharos-agent<br/>Site agent"]
        LDAP[("OpenLDAP<br/>ou=pharos")]
        ACCESS["pharos-ssh-access<br/>access agent"]
        RT["pharos-runtime<br/>Site Runtime"]
        SSHD["SSSD → sshd<br/>login nodes"]
        AGENT -->|"associations"| SLURM
        AGENT -->|"entries, POSIX, groups"| LDAP
        ACCESS -->|"keys + ldapPublicKey class"| LDAP
        LDAP --> SSHD
    end
    subgraph CLOUD["Icarus-cloud (Pharos platform)"]
        direction TB
        WALDUR["Waldur"]
        SITEAPI["Metallo Site API"]
        SA["Metallo ssh_access"]
        API["API"]
        WEB["Web"]
        BETO["Beto<br/>Party, Project"]
        SITEAPI --> SA
        WEB --> API --> SA
        BETO -.->|"events, after<br/>Beto merges"| SA
    end
    AGENT ==>|"HTTPS"| WALDUR
    ACCESS ==>|"HTTPS"| SITEAPI
    RT ==>|"HTTPS"| SITEAPI
```

Thick arrows are outbound HTTPS connections from Icarus-hpc; each agent uses its
own credential.

## Context

[GitHub #678](https://github.com/pharos-ai-factory/pharos-platform/issues/678)
defines member-facing SSH key routes: gates, a stable key ID across resubmission,
per-member fingerprint uniqueness and the states `pending_deployment`, `deployed`,
`rejected`, `pending_revocation` and `revoked`. Keys belong to the member, not to a
Project or Waldur offering. Each member has one native account on Icarus-hpc.

Upstream Waldur 8.1.3 offers per-user key storage (`/api/keys/`) and a provider key
listing, but no per-key state, comment or result reporting. Writing a key for a
member needs that member's own Waldur credential. Key events reach only staff,
support or the member.

The September prototype is a Waldur extension. A member files a request to add or
remove a key, the HPC side approves or rejects it, and the extension keeps the list
of approved keys for each offering user.

#678 was written afterwards and describes the same lifecycle from the key's point
of view. A pending add request corresponds to `pending_deployment`, an approved add
to `deployed` and a rejected add to `rejected`. A pending remove request corresponds
to `pending_revocation`, and an approved remove to `revoked`.

#678 also adds a few things. A key keeps one ID for its whole life, so a revoked key
can be resubmitted without pasting it again; in the prototype each request has its
own ID. #678 checks the AUP and the HPC username before accepting a key, and lets the
page validate a key without submitting it.

The main difference is scope. The prototype attaches keys to an offering user,
that is, to one member in one offering. SSH keys are personal, so this decision
attaches them to the member, across all their Projects and offerings.

On Icarus-hpc, the Site agent's account-name plugin creates each member's POSIX entry
under `ou=pharos` under the username Metallo wrote from the Party's `site` binding, and
pins the UID it got on the offering user. The offering's policy is `service_provider`;
Waldur itself does not derive the name from a claim. The LDAP schema is `openssh-lpk`.
The Icarus account model is proposed in
[PR #643](https://github.com/pharos-ai-factory/pharos-platform/pull/643).

The Icarus-hpc administrators confirmed on 21 October 2026:

- `authorized_keys` is used only for specific local users, so LDAP is the only key
  source for Pharos members;
- their own key-sync script writes elsewhere, not under `ou=pharos`, and Pharos UIDs
  come from a separate range;
- when a Waldur project ends they remove the Slurm association; the user can still
  log in with SSH but cannot submit jobs;
- the access agent can run on the same host as the Site agent; outbound HTTPS from
  that host to Icarus-cloud is permitted;
- there is no site-specific key policy beyond LDAP and Waldur defaults;
- SSSD propagation time is not yet known.

On 23 October 2026 they also confirmed:

- Pharos UIDs come from 5000000–5499999 and allocation groups from 5500000–5599999;
- the directory is OpenLDAP without LDAPS;
- no GRNET users are synchronised into the Icarus LDAP, so the Site agent creates
  every Pharos entry under `ou=pharos`;
- the 120-second propagation window is acceptable;
- they will apply the directory, service-account and Slurm setup that Pharos specifies.

## Decision

1. **Metallo owns member SSH keys.** A new Metallo context, `ssh_access`, owns each
   member's keys for a Site and their #678 lifecycle in its own PostgreSQL database
   with dedicated credentials. Waldur does not store these keys. No Waldur
   extension is installed.
2. **The Site agent creates accounts; the access agent writes keys.** The upstream
   Site agent remains the only writer of LDAP entries, POSIX attributes and groups.
   The access agent never creates, renames, disables or deletes entries. It adds or
   removes individual `sshPublicKey` values. It adds the `ldapPublicKey` auxiliary
   class atomically with the first key and removes it atomically with the last
   key, supporting directories where the class requires `sshPublicKey` (`MUST`).
3. **The LDAP server enforces one writer per attribute.** The access agent binds as
   `cn=pharos-ssh-access` with write access to `sshPublicKey` and only the
   `ldapPublicKey` value of `objectClass` under `ou=pharos`, plus read access to
   identity and concurrency attributes. All other class values remain read-only
   to that bind. The Site agent's bind DN loses write access to `sshPublicKey`;
   account creation must omit `ldapPublicKey` when the schema requires a key.
4. **The two agents run as separate service identities on the same host.** The
   access agent has its own Unix user and systemd unit. It has no Slurm, Waldur,
   root or sudo authority, and no platform database credential.
5. **The access agent reaches Icarus-cloud only by outbound HTTPS.** It pulls work
   from, and acknowledges results to, the existing Metallo Site API on dedicated
   `/v1/site/ssh-access/*` routes on the separate Site API host. The edge routes
   that hostname and prefix to the Site API while browser API and web traffic keep
   separate hosts, backends and credentials. It authenticates with its own native
   service credential: a high-entropy bearer whose digest and grant the Site API
   retains. Its grant covers only the SSH access routes for its Site. The Site
   Runtime's grant does not cover them, and the same verifier serves the native Site
   Runtime.
6. **The access agent converges per member.** Work is level-triggered: for each
   member it applies the desired key set to the exact entry identified by `uid` and
   `uidNumber`. Each modification carries a critical RFC 4528 assertion for both
   `uidNumber` and the entry's immutable `entryUUID`, so an account replacement at
   the same DN cannot receive the previous account's key. Read-back verifies the
   same identity. A fresh read and an `entryCSN` assertion also fence concurrent
   key/class changes; contention retries rather than rejecting a valid key. A missing entry leaves keys pending and retried; it is not a
   rejection. The agent reconciles after uncertain outcomes.
7. **Departure follows Icarus-hpc practice.** Leaving every HPC Project removes
   the Slurm association through the Site agent; the LDAP entry and its keys remain,
   so the member can still retrieve data. New submissions are refused with
   `not_eligible`. Keys are removed only by member revocation or by an AUP change.
8. **Beto inputs arrive through Metallo-owned ports.** `ssh_access` reads member
   Site identity and HPC eligibility through its own ports. Until Beto's Party and
   Project integration events
   ([`feat/beto-party-project-governance`](https://github.com/pharos-ai-factory/pharos-platform/tree/feat/beto-party-project-governance))
   are on `main`, configured fixture adaptors supply them. Subsequent adaptors
   consume `PartyProviderBindingEstablished`, `PartyProviderBindingRetired`,
   `ProjectChanged` and `ProjectAccessChanged`.
9. **The browser contract follows #678 within current platform rules.** Routes are
   declared in the API's OpenAPI contract and reach `ssh_access` through typed
   command and query bindings. Browser updates use a typed Protobuf projection and
   Centrifugo once the page is mounted. There is no browser polling, no NextAuth
   route placement and no email matching.

## Consequences

**Positive:** one authority for keys and their states, held per member; Waldur and
the Site agent need no Pharos-specific code; native authority on Icarus-hpc is limited to the key attribute and one auxiliary class value
and enforced by the LDAP server; ordering between account creation and key
submission does not matter; the native credential verifier and Site API route also
serve the native Site Runtime.

**Negative:** keys added in Waldur Homeport are ignored on Icarus-hpc; a new
PostgreSQL context and a Site-side process must be operated; `deployed` and
`revoked` depend on the SSSD cache: the administrators accepted a 120-second window,
which the laboratory measured but Icarus-hpc has not yet qualified.

**Neutral:** the access agent and Site API routes can ship in separate PRs behind the
same lifecycle; Beto adaptors replace fixtures without changing the context.

## Open questions

- Context name: `ssh_access` is proposed; `credentials` was also considered but
  overlaps with service credentials and delegated member JWTs.
- Source of the AUP gate: the retained Themelio AUP or the Waldur offering's Terms
  of Service.
- The access agent uses `ldapi://` with SASL EXTERNAL on the directory host,
  mapped from its Unix identity to its restricted LDAP identity without a password.
  The upstream Site agent supports simple bind only and uses loopback `ldap://`
  on Icarus-hpc. Confirm that listener and the service-user mapping with jimsiak;
  the access agent still refuses non-loopback cleartext LDAP.
- The Site agent's bind DN ACL (pending jimsiak). No existing entry needs
  `ldapPublicKey`, since no GRNET users are synchronised.
- Per-member key limit (10 proposed) and FIDO (`sk-ssh-ed25519`) keys.

## Related documents

- [DD-0055](../dd/0055-hpc-ssh-key-provisioning.md).
- [GitHub #678](https://github.com/pharos-ai-factory/pharos-platform/issues/678),
  [#137](https://github.com/pharos-ai-factory/pharos-platform/issues/137),
  [#141](https://github.com/pharos-ai-factory/pharos-platform/issues/141).
- [PR #643: Icarus HPC accounts, member execution and run path](https://github.com/pharos-ai-factory/pharos-platform/pull/643).
- [Secure Site communication](../../services/metallo/docs/epics/instantiate-and-run/secure-site-communication.md).
- [Waldur Site agent LDAP plugin](https://github.com/waldur/waldur-site-agent/blob/main/plugins/ldap/README.md).