# Site API process This Deployment runs `python -m metallo.services.site_api`: Site Coordination's HTTPS API through which Site Runtimes pull their ordered commands and return results ([4B design](../../../../../services/metallo/docs/epics/instantiate-and-run/secure-site-communication.md)). It listens on port 8443 behind the `site-api` Service. Every overlay runs one replica; the Metallo workstation loop (`metallo-devbox`) keeps it at zero replicas because it issues neither the certificate nor the TokenReview grant. The process receives separate context-owned PostgreSQL bindings from `site-api-credentials`. Site Coordination serves command delivery and SSH Access serves key propagation; when execution authority is enabled, Execution Core supplies accepted Run material, Resource Provisioning supplies the exact retained native member binding, and Source Control owns repository grants. Each adaptor uses its owner's restricted database credentials. Kubernetes mounts no default ServiceAccount token. Instead, a projected token with the API server's audience and a 600-second lifetime lets the process create TokenReviews. A Runtime's token identifies an exact ServiceAccount subject, and `SITE_API_GRANTS_JSON` maps that subject to the `k8s` or `slurm` stream. The mounted `SITE_COORDINATION_VIEWS_BROKER_URL` provides a separate query-only Beto caller for checking current `runs.output.view` before serving a private read. It has its own reply family and cannot consume Execution Core’s projection queues. Beto enrols `site_coordination` for the same Project source scope. `SITE_API_AUTHORITY_GRANTS_JSON` separately maps the native admission account to its environment. The maps must be disjoint: admission may read accepted material, and cannot pull or acknowledge commands or issue repository credentials. ## Enable accepted Run authority The base keeps `SITE_API_EXECUTION_AUTHORITY_ENABLED=false`, and the execution worker and Pipeline coordinator at zero replicas. The enabling overlay sets that flag to `true` after recovery and provides `SOURCE_CONTROL_GITLAB_URL`, `SOURCE_CONTROL_GITLAB_INSTANCE` and `SOURCE_CONTROL_GITLAB_ROOT_GROUP`, matching the Source Control installation. A dev box's overlay and a site's `distributed` overlay both enable it. On a site the root adds the GitLab URL from the edge's GitLab origin and the instance from the API authentication binding, and delivery writes the tokens (`devops/platform/scripts/platform_deploy/source_control.py`). The Site API receives the read-only GitLab token and public CA, plus a separate `SOURCE_CONTROL_RUN_ISSUER_TOKEN`. Provision this PAT for a dedicated external, non-administrator GitLab service user with permission to issue project access tokens for the managed repositories. The process verifies that user at startup and before issuance. The credentials Job leaves this external PAT as `pending` until enrolment, and preserves its installed value on subsequent reconciliation. The group-managing Source Control token and provider administrator credentials do not enter this process. `SOURCE_CONTROL_RUN_GRANT_SEALING_KEY` is a separately generated 32-byte secret, written as 64 hex characters, used to seal durable per-Run tokens. Preserve it across replacement Pods and recovery. Its mount and the issuer PAT are limited to Site API; workers and Runtimes receive neither. Stage-in repository tokens have the provider's date-based expiry, bounded by the accepted work deadline; a deadline before the provider's next supported expiry refuses issuance. Stage-out recovery can issue a separate token for the latest submitted, non-terminal Attempt after that deadline. It remains bound to the saved repository input and expires at the next provider-supported UTC date. Cleanup revokes both grant purposes before reporting the Attempt as final. Site API egress reaches PostgreSQL, scoped RabbitMQ owner calls and HTTPS, including the configured GitLab origin, and the Kubernetes API for TokenReview. Source Control and Resource Provisioning use their own enrolled Beto broker callers. The dev-box registration adds those callers to the same existing Project source scope without rotating provider credentials, and publishes only that scope to Metallo. ## Installation inputs | Input | Dev box (Argo) | Local Minikube and k3s | | --- | --- | --- | | `site-api-credentials` | `pharos-credentials` Job, from `devops/secret-keys.json` | `scripts/environment.py` `provision_secrets` | | `site-api-tls` and `site-api-trust` | `pharos-credentials` Job | `scripts/environment.py` `provision_site_api_tls` | | TokenReview grant | `argocd/bootstrap/site-api-review.yaml`, applied by `make deploy` | `pharos/site-api-review`, included by the overlay | | `site-api-grants` (native bearers) | `scripts/site_credentials.py`, as administration | the same | | `pharos-site-api-trust` (edge) | the `edge` role of `make deploy`, from `site-api-trust` | not installed | Argo cannot create cluster-scoped rights for a service (decision 0041), so after this change the dev box needs one `make deploy` before the Site API can authenticate Runtimes; `make reconcile` alone does not apply the bootstrap. `scripts/site_api_tls.py` issues the certificate once per namespace. A private CA signs one server certificate for `site-api`, `site-api.`, `site-api..svc` and `site-api..svc.cluster.local`, then its key is discarded. The CA is published in the `site-api-trust` ConfigMap. Existing material is never replaced. To rotate, delete both `site-api-tls` and `site-api-trust`, re-run the issuer and update every Runtime's trust. ## Connect a Runtime The default grant admits the Kubernetes Site Runtime installed as ServiceAccount `runtime-k8s` in namespace `pharos-site-runtime`. For other Runtimes, replace `SITE_API_GRANTS_JSON` in the target overlay; a Slurm Runtime needs the same kind of explicit subject. Configure the Runtime with: - `SITE_API_URL=https://site-api..svc:8443`; - a `site-api-trust` ConfigMap in its own namespace, copied from this namespace; - its projected token with audience `site-control`, as in the Runtime manifests. The ingress policy admits Pods labelled `app: runtime` or `app: execution-admission` from any namespace on port 8443, and the platform gateway. The Pods' own namespaces must allow that egress. Each subject still needs its explicit Runtime or query-only grant. ## Sites outside the cluster The edge catalogue publishes the same listener at `site.pharos.` (profile `pharos-site`) for Sites that only open outbound HTTPS, such as the Icarus SSH access agent and native Runtimes. The public prefixes are `/v1/site/commands/`, `/v1/site/ssh-access/`, `/v1/site/executions/` and `/v1/site/repositories/`. Execution authorisation and repository grants use the Runtime's scoped native bearer and the accepted Run's current authority. The shared gateway terminates the public TLS and opens a new TLS connection to port 8443. It verifies the server certificate against `pharos-site-api-trust`, the `site-api-trust` CA that the `edge` role publishes in `istio-system`, and never disables hostname checking. After the certificate is first issued or rotated, run `make deploy` once so that the gateway trusts it. These callers present native bearers, checked against `site-api-grants` (`grants.json`): a digest-only store that administration writes with `scripts/site_credentials.py`. The Secret is optional; without it every native bearer is refused as unavailable. Native callers use scoped bearer grants and do not perform browser OIDC. Configure `SITE_API_URL` and `SSH_ACCESS_SITE_API_URL` as the public base origin, without `/v1/site/`; clients append their protocol paths. ## One Site per listener `SITE_API_SITE_ID` names the one Site this listener serves, and every native grant (`@`) is checked against it before any storage read; a grant for another Site answers the shared `site_mismatch`. The setting belongs to the Site API, not to one of its route families, so the command routes, the SSH access routes and the member credentials route of the AAI work all read it. A listener that is configured with native grants but no `SITE_API_SITE_ID` refuses to start: the check is never silently skipped. A listener that serves only Kubernetes identities may omit it, because a projected ServiceAccount token carries no Site claim. `ssh-access-settings` supplies `SITE_API_SITE_ID`, `SSH_ACCESS_SITE` and the propagation window, so the Site API and the `ssh-access` process share one Site. When both variables are present they must name the same Site, otherwise the Site API does not start. The 4B qualification lane renders this base without that ConfigMap and sets `SITE_API_SITE_ID` itself (see `dx/metallo/scripts/metallo_dx/site_api_resources.py`). With `SSH_ACCESS_SITE` set, the listener also serves the SSH access pull, snapshot and acknowledgement routes. The installation issues one grant per native service and Site: `ssh-access@` with the `ssh-access` scope, and `commands@` with the `commands` scope and its explicit `slurm` or `k8s` environment. Keep them separate. They map to different Unix service identities and route families, and separate grants allow independent expiry and revocation without coupling SSH key delivery to Slurm command delivery. The supported rotation procedure is deliberately non-overlapping because `site_credentials.py issue` rejects a second active grant for the same principal. Revoke first, publish and restart/reconcile the Site API, prove the old bearer is rejected, then issue and publish the replacement. Projected Secret updates are eventually consistent; use the owning deployment restart/reconcile procedure and verify both the new store and the expected `401` before handing over a replacement. Never run two active grants for one principal or claim zero-downtime rotation. `/healthz` is served over HTTPS for the readiness and liveness probes. The API holds no state of its own: replacing its Pod keeps every command and result in PostgreSQL, and Runtimes retry identical results after a lost response. ## Member offline consents The Site API also serves the Runtime's per-action member token, and the login service hands each offline consent to it. Four values arrive from the administration reconciliation's scoped `member-credentials` projection — the login client's issuer, client ID, client secret and the AAI CA — and three from `site-api-credentials`: `MEMBER_CREDENTIALS_DATABASE_URL`, `MEMBER_CREDENTIALS_INTAKE_KEY` and `MEMBER_CREDENTIALS_SEAL_KEY`. Without an intake key the listener refuses to start with a partial consent configuration rather than serving a route that cannot answer; the Database URL and seal key are then required together. `MEMBER_CREDENTIALS_INTAKE_KEY` is shared with the API, which posts the hand-off after a successful login. The Site API executes `devops/aai/identity_config.py`'s client registration: the login client needs `offline_access` as an optional scope and `consentRequired: true`, which `.local_aai` already declares for `pharos-control-plane`. The store keeps one active consent per member, seals the refresh token, rotates it before returning an access token and revokes the previous consent when the member signs in again. The listener also runs the sweep that refreshes idle consents and retries revocations. It refreshes only with `SITE_API_EXECUTION_AUTHORITY_ENABLED=true`, which connects the members' Project execution authority; until then it only retries revocations. See [Member Credentials](../../../../../services/metallo/contexts/member_credentials/README.md).