# SSH Access SSH Access is the Metallo context that owns members' SSH public keys for each Site and their [#678](https://github.com/pharos-ai-factory/pharos-platform/issues/678) lifecycle. Waldur does not store these keys. The access agent on the Site applies them to LDAP through the Site API. See [ADR-0055](../../../../docs/decisions/0055-hpc-ssh-key-provisioning.md) and [DD-0055](../../../../docs/dd/0055-hpc-ssh-key-provisioning.md). ## Aggregate `domain.access.MemberSshAccess` is one root per member and Site. Its identity is derived from both, and it doubles as the opaque `memberRef` the Site sees. Its children are the member's keys (`domain.key.SshKey`). One root makes per-member fingerprint uniqueness, the key limit (10) and serial submissions atomic with the root lock and version. The root also records the member's last known Site account (`username`, `uidNumber`, source revision); an older identity revision never replaces a newer one. A different username or UID requires newer evidence and acknowledged removal of all existing keys. Validation and submission enforce the same account check. Canonical submissions also record an `AccessGrant` on each key: the Party's authentication authority version and the exact Site-binding ID. The submission transaction rechecks these against the owner projection under its locks. Private snapshots read older keys without this field; the canonical withdrawal worker revokes such unproven grants once, so the member must explicitly resubmit. HTTP and browser projection shapes are unchanged. Each key keeps one `id` for its whole life. Its states follow #678: `pending_deployment`, `deployed`, `rejected`, `pending_revocation` and `revoked`. Every submission, resubmission and revocation issues a new `generation` and asks the Site for the key's presence (`present` while pending or deployed, otherwise `absent`). Site results for an older generation are `stale` and change nothing; a repeated result for the current generation is `replayed`; a different result for it, or one that contradicts the desired presence, is a `site_result_conflict`. An `applied` or `removed` result proves the LDAP state only. The key stays pending until `max(observedAt, generation issued) + propagation window`, then becomes `deployed` or `revoked`. `rejected` is final and carries the Site's reason. Revoking a pending key that has no `applied` result also keeps it in `pending_revocation`. A missing acknowledgement does not prove that an add never reached LDAP, so the next generation asks the Site for absence. Only a `removed` result and the propagation window make it `revoked`; delete and resubmit remain unavailable until then. ## Operations | Operation | Handler | Rules | | --- | --- | --- | | `GetMemberHpcAccess` | `MemberKeyOperations.access` | Every disabled reason in #678 order; never fails on gates | | `ListMemberKeys`, `GetMemberKey` | `.keys`, `.key` | Caller's root only; a deleted key is `key_not_found` | | `ValidateMemberKey` | `.validate` | Gates, key checks, duplicate and limit; records nothing | | `SubmitMemberKey` | `.submit` | Same checks; the key ID derives from the command ID | | `RevokeMemberKey` | `.revoke` | Never gated; a repeat returns the key as is | | `ResubmitMemberKey` | `.resubmit` | Gates first, then the stored line is checked again | | `DeleteMemberKey` | `.delete` | Only `rejected` or `revoked`; never gated | | `PullSiteKeyWork`, `GetSiteKeySnapshot` | `SiteKeyOperations.pull`, `.snapshot` | Configured Site only; paginated by root | | `RecordSiteKeyResults` | `SiteKeyOperations.record` | Per-result `recorded`, `replayed` or `stale` | | `CompletePropagation` | `PropagationCompletion` | Due roots from the work index | | `RevokeForAupChange` | `AupRevocation` | One command per affected root, AUP version and observed key generations | | `RevokeForAccessChange` | `AccessRevocation` | Withdraw observed generations after suspension, authority change or security retirement | Gate failures raise `HpcAccessGated` with every reason that holds: `not_eligible`, `aup_not_accepted`, `aup_outdated`, `hpc_username_missing`. Key checks run in the order private key (`key_private`, never echoed), format (`key_malformed`), type (`key_type_unsupported`), strength (`key_too_weak`), then the name (`key_name_invalid`). Gate and key check failures are raised before the transaction and record nothing. Duplicates (`key_duplicate`, with the existing record), the limit (`key_limit_reached`) and state conflicts (`key_state_conflict`) are recorded rejections that change nothing. #678's OpenAPI attachment is not available, so the key check, name and limit codes are named here; the browser contract confirms or renames them. `MemberSiteIdentityPort`, `HpcEligibilityPort` and `AupPort` supply the gates; answers that do not match the requested member and Site raise `InputsUnavailable` for retry. `Clock` and `IdGenerator` supply time and identities (`adaptors.system`). `SshAccessSettings` holds the Site, propagation window, key limit and login nodes. ## Persistence Private database `pharos_ssh_access` with a restricted runtime role. One `pharos.aggregates` row of kind `member_ssh_access` holds the root snapshot. The same transaction writes the command receipt and: - `pharos.ssh_access_keys`: one row per key, unique per root and fingerprint; - `pharos.ssh_access_work`: whether the root has unacknowledged generations or present keys, and when its earliest propagation window started; - `pharos.ssh_access_site_results`: one immutable row per recorded `(key_id, generation)`. A database guard rejects child writes outside their root's command transaction or at another root version. The adaptor checks every row against the locked snapshot before a decision; a mismatch is `CorruptState`. Each change also records the browser projection's realtime publication; the aggregate publishes no integration events. ## Transport and composition `adaptors.messaging` registers the API's [SSH access requests](../../../../contracts/ssh_access/README.md) with the shared `RequestRegistry` in `metallo.adaptors.requests`. Commands answer with their recorded outcome, whose reply commits in the same transaction. `ActorMembers` maps the authenticated actor to a member; an unmapped actor gets `member_not_found`. `metallo.services.ssh_access` composes the request consumers, reply and realtime relays, propagation, owner projection and policy workers in the `ssh-access` Deployment. The normal composition uses the owner adaptors below. `adaptors.fixture_owners` remains available when a development test explicitly sets `SSH_ACCESS_FIXTURE_FILE`; it refuses to load outside development configuration. Installed manifests use canonical owners, enforced by deployment checks. ## Owner inputs `adaptors.owner_views.SshAccessView` keeps this context's own Party, provider binding, Project, membership and Project authority tables. Migrations `0002` and `0003` add these beside the existing key aggregate. Eight durable Beto subscriptions use `SSH_ACCESS_VIEWS_BROKER_URL`, the `pharos_ssh_access_views` identity. Consumers start before checkpoint synchronisation; snapshots repair missed events and the view becomes ready only after matching Beto's checkpoints. Repair includes Parties referenced by saved key roots after they leave Projects. `adaptors.canonical_owners` interprets the API actor as the canonical Party ID. A fresh Beto Party snapshot repairs authentication and binding gaps on member requests; a newer event already saved takes precedence. Site results publish to that same Party principal. The Site agent needs only Metallo's Site API and LDAP. | Input | Authority and read | | --- | --- | | Site username | The unique established Beto `site` binding at the configured instance | | Numeric UID | Resource Provisioning's `HpcAccessQueries.account`, through the service-level `SshAccessResources` adaptor | | Eligible HPC Projects | Scoped Beto membership and current `project.view` decision, combined with Resource Provisioning's usable allocation at the configured HPC offering | | AUP | Retained Themelio's private policy query, authenticated with the separate `SSH_ACCESS_POLICY_KEY` | Both supported Beto membership roles hold `project.view`. A Project needs a matching `ready` or `changing` allocation; other offerings, Sites, pending, suspended and retired resources grant no new key access. Resource evidence expires after fifteen minutes. The numeric UID comes from the existing HPC offering-user reconciliation, which checks Waldur's username against the canonical Site binding. Resource Provisioning stores it separately from the LDAP entry UUID; the numeric value is required explicitly, including after an upgrade. SSH Access requires the exact Party and binding, a ready account and a positive POSIX UID; conflicting reports defer the operation. The report timestamp supplies the identity evidence revision. These are Resource Provisioning owner queries, composed with its own database credentials. `adaptors.legacy_policy` queries accepted and required versions of the retained platform AUP. Waldur Terms of Service are a separate policy. Acceptance replies must name the requested Party and its current GitLab binding and carry recent evidence. TLS verifies the private authority; redirects and inherited proxies are refused. An unavailable or malformed answer, including a broken HTTP response, defers new submissions and resubmissions. A confirmed missing Party or current GitLab binding means no current acceptance can be established. The reader then fetches the required version through the binding-independent private policy GET, returning no accepted version. The same rule applies when the policy POST confirms that the exact requested binding is no longer current. Failed owner queries remain unavailable inputs; they never become evidence of absence or a fabricated policy version. The policy worker checks roots holding present keys every minute. When acceptance differs from the current version, it issues the existing deterministic revocation command per root, AUP version and observed key generations. A later resubmission can therefore be revoked if confirmed binding loss means current acceptance can no longer be proved, even when the required version remains unchanged. A delayed command affects only its observed generations. The durable work index supports retry after an outage or restart; one unavailable member does not block the rest. Existing keys remain discoverable and removable when Project membership or account evidence disappears. Membership loss alone does not revoke existing keys. The suspension and reinstatement policy is defined in [DD-0055 ยง2.2](../../../../docs/dd/0055-hpc-ssh-key-provisioning.md#22-submitting-and-revoking-a-key). `AccessRevocation` compares each present key's grant with current Party authority and security-retired Site bindings. A changed authentication generation withdraws the old grant even when suspension and reinstatement both happened during an interruption. Reinstatement leaves the old key revoked until the member explicitly resubmits after normal gate checks. Retirement never changes the stored account target: removal still addresses the original LDAP entry. Committed Party and binding events wake the access worker. Its one-minute repair sweep uses the existing durable key index; unavailable owner queries defer that member without blocking others. This worker does not query AUP, so a policy outage cannot prevent a confirmed security withdrawal. Generation-specific commands and Site acknowledgements protect newer key submissions from delayed work. The API still requires an authenticated Party for member commands; automatic withdrawal handles disabled members who cannot use that route. UID and AUP events can replace these query adaptors when their authoritative contexts publish suitable contracts, including initial-load and repair queries. The gates and Site protocol retain their current ports. ## Browser projection `adaptors.projection` encodes `pharos.realtime.v1.MemberSshAccessProjection`: the member, the Site and the visible keys with the #678 fields except the key value. The PostgreSQL boundary records it with every committed root revision, including Site results and propagation completion. `ActorMembers` supplies one principal-targeted publication per actor; a member with no actor has no browser audience. The projection relay sends each exact persisted envelope through the restricted realtime gateway. The browser reads the owner's `ProjectionSnapshot`, applies only newer revisions per root and reconciles once after unrecoverable history. ## Site protocol `adaptors.site_http` serves `POST /v1/site/ssh-access/pull`, `snapshot` and `acknowledge` from [the Site contract](../../../../contracts/http/site-ssh-access.v1.yaml) on the Metallo Site API listener. `adaptors.native_callers` accepts only native bearers whose grant has the `ssh-access` scope; the Site comes from that grant, and an unconfigured Site is refused. Pull and snapshot pages use route- and Site-bound page tokens; `memberRef` is the root identity. Bodies are strict JSON, duplicate fields and nulls are refused, and every reply is checked against the generated contract models before it is sent. Errors are redacted: `401` credentials, `403` scope or Site, `409` conflicting results, `422` invalid input or member references, `503` unavailable credentials or storage. ## Access agent `application.agent.AccessAgent` converges one member at a time on the Site host. It finds exactly one entry with `uid=` under the people base; a missing entry leaves the work pending, because Waldur marks the account `OK` before the Site agent creates it. A different `uidNumber` blocks every write and raises an operational alert. For each key it adds the exact line or deletes the values with that fingerprint, reads the entry back and acknowledges only what the read-back proves. Already present and already absent count as success. A permanent refusal of an add is acknowledged as `rejected` with the directory's reason; a refused removal stays pending with an alert. The pull pass acknowledges every converged key, which recovers a crash between an LDAP change and its acknowledgement. The snapshot pass repairs drift, acknowledges only keys it changed and alerts on `sshPublicKey` values Pharos does not hold. A removal after a recorded rejection replays, since nothing remains. `adaptors.ldap_directory` implements only search, value add, value delete and read-back with `ldap3`. Every modification carries a critical RFC 4528 assertion for the entry's `uidNumber`, immutable `entryUUID` and freshly read `entryCSN`. The first key and `ldapPublicKey` class are added in one modify; the last key and class are removed in one modify. Every value change reads again, including successive removals from a stale caller entry. Contention retries from fresh state; result 20 alone is not proof that a compound modify stored a key. This supports both MUST and MAY schemas. The bind may write only that class value, not other classes or POSIX attributes. Read-back verifies the same identity, so a replacement account at the same DN cannot receive the old key. It uses SASL EXTERNAL over a local `ldapi` socket without a password, or simple bind over verified LDAPS; cleartext LDAP is restricted to loopback. `adaptors.site_client` calls the Site routes over verified HTTPS without proxies or redirects and validates every reply against the contract. `metallo.services.ssh_access_agent` composes them; packaging and directory ACLs are in [`deploy/ssh-access-agent`](../../deploy/ssh-access-agent/README.md). ## Status The context, browser/API contracts, Site protocol, access agent, Beto projections, resource queries and retained AUP reader/revocation worker are implemented. An installation must qualify its own AAI Site binding, offering, policy reader and real member journey before activation; see the [deployment guide](../../../../gitops/manifests/pharos/base/ssh-access/README.md). Fixture qualification and live deployment qualification are separate evidence. ## Verification - Unit: `services/metallo/tests/ssh_access` covers the #678 transition table, key validation, gate order, generation fencing, the propagation window, deleted keys awaiting removal, UID changes, policy transport failures and validate/submit parity. Resource Provisioning's `test_hpc_access.py` covers offering scope, freshness, account correlation and conflicting reports. - PostgreSQL: `services/metallo/tests/infrastructure/ssh_access.py`, run by `pnpm test:integration`, covers concurrent submissions, duplicates in every state, lost-response replay, resubmission, stale and replayed Site results, AUP revocation, rollback, corrupt indexes and the restricted role. The same lane drives the HTTP routes through the API, RabbitMQ and the `ssh-access` image. - Beto/Metallo: the `cross-service` integration group runs `tests/infrastructure/ssh_access_owner.py` with real Beto, PostgreSQL and broker identities. It checks event delivery, checkpoint rebuild, pending authority, membership retirement and canonical Site bindings. A labelled policy HTTP fixture and directory double exercise submission and AUP removal through the production Site routes and agent codecs. - API contract: `services/api/adaptors/http/ssh_access_test.go` checks every route and error code against the bundled OpenAPI document, the gated `403`, validate/submit parity, origin, content type, command identity and preservation of the owner's projection bytes. - Browser projection: `tests/ssh_access/test_projection.py` covers snapshot bytes, deleted keys and per-actor audiences; `e2e/ssh-access-realtime.spec.ts` covers first read, recovered history and one reconciliation after history loss. - Site API: `tests/ssh_access/test_site_http.py` and `tests/test_site_credentials.py` cover grant separation in both directions, expired, revoked and unknown bearers, the Site taken from the grant, strict bodies and page tokens. The integration lane pulls and acknowledges through the deployed `site-api` image with issued grants. - Access agent: `tests/ssh_access/test_agent.py` with directory and Site doubles; `scripts/ssh_access_ldap_integration.py` (also run by `pnpm test:integration`) runs the agent against OpenLDAP with openssh-lpk and both ACLs: ACL enforcement in both directions, missing entry, `uidNumber` mismatch, unmanaged values, a crash after modify and drift repair. The focused `uv run --project devops --frozen python scripts/ssh_access_must_integration.py` lane uses disposable `osixia/openldap:1.5.0`, replaces its openssh-lpk class with `MUST sshPublicKey` and reads the schema back before testing. It applies the actual ACL template and checks first/last key changes, stale entries, concurrent writers, and result 50 for other class values and attributes. The container is removed on exit.