# 4B — Secure Site communication

Site Runtime reads its commands from Site Coordination over HTTPS and sends back
results. Site Coordination authenticates every request and selects the stream
from the Runtime's verified identity. The two streams are `k8s` and `slurm`.

## Flow

1. Kubernetes mounts a projected ServiceAccount token in the Runtime Pod, with
   audience `site-control` and a 600-second lifetime. Kubelet rotates it.
2. The Runtime client reads the current token, checks its subject, audience and
   validity times, and sends it as `Authorization: Bearer …` over verified HTTPS.
3. Metallo submits the token to the Kubernetes TokenReview API. Its own projected
   reviewer credential permits only `create` on `tokenreviews`.
4. A successful TokenReview identifies an exact ServiceAccount subject. Metallo's
   configuration maps that subject to `k8s` or `slurm`.
5. `POST /v1/site/commands/pull` reads that stream from Site Coordination's private
   PostgreSQL database. The body accepts `afterSequence` (default 0) and `limit` (default 50).
6. The Runtime sends a typed result to `POST /v1/site/commands/acknowledge`.
   Site Coordination checks the caller's environment, command and execution
   identity, then commits the result and acknowledgement progress together.
7. Repeating a committed result returns `replayed`. Conflicting results or sequence
   gaps are rejected. A pull alone does not acknowledge a command.
8. While the next command waits, the Runtime may say why with a catalogued code at
   `POST /v1/site/commands/progress`. The command stays pending; the report is best
   effort and is refused once the command is acknowledged (#1053).

The credential identifies the Runtime process. Member permission checks and native
Kubernetes impersonation or Slurm delegation remain separate checks on execution.
A Runtime never receives platform database credentials.

## Implementation

| File | Responsibility |
| --- | --- |
| `contracts/http/site-coordination.v1.yaml` | Two HTTPS operations, strict request/reply schemas and bearer authentication. |
| `contexts/site_coordination/application/site_api.py` | Authenticate through a port, derive environment, call the storage port. |
| `contexts/site_coordination/adaptors/site_http.py` | Validate HTTP, invoke handlers, return bounded and redacted errors. |
| `contexts/site_coordination/adaptors/kubernetes_identity.py` | Verify current identity through Kubernetes TokenReview and configured subject grants. |
| `contexts/site_runtime/adaptors/site_runtime_identity.py` | Read and check the rotating projected token on each request. |
| `contexts/site_runtime/adaptors/site_runtime_client.py` | Verified HTTPS calls using production transport models. |
| `services/site_api.py` | Compose authentication, restricted database access and the HTTPS listener. |

Paths other than the contract are relative to `services/metallo`.
The existing 4A stream aggregate, commands, outcomes and receipts remain the storage
boundary. This subtask adds no tables and does not change their transaction rules.

## Configuration and failure handling

The Metallo image runs the Site listener with
`python -m metallo.services.site_api`, using its installed virtual environment.
It listens on HTTPS port 8443. The normal internal Metallo HTTP entry point keeps
its existing operator authentication. Every installation runs the listener as the
`site-api` Deployment ([installation](../../../../../gitops/manifests/pharos/base/site-api/README.md))
with these settings:

- `APP_ENV=development` and `SITE_COORDINATION_DATABASE_URL_FILE` for its restricted database role.
- `SITE_API_TLS_CERT_FILE` and `SITE_API_TLS_KEY_FILE` for the server certificate and key.
- `SITE_API_KUBERNETES_URL`, `SITE_API_KUBERNETES_CA_FILE` and `SITE_API_REVIEWER_TOKEN_FILE` for TokenReview.
- `SITE_API_GRANTS_JSON`, for example `{"system:serviceaccount:runtime:k8s":"k8s"}`.

The installation issues the server certificate once per namespace with
`scripts/site_api_tls.py` and publishes its CA as the `site-api-trust` ConfigMap.
The reviewer credential is a projected 600-second token for the API server. The only
cluster-scoped right is `create` on `tokenreviews`, granted by the dev box's Argo
bootstrap or by the local and k3s overlays. NetworkPolicies admit PostgreSQL and the
API server outbound, and Runtime Pods inbound on 8443. The dev-box qualification
lanes install these GitOps resources, changing only the namespace, image, grants and
the label of the lane's database.

The Runtime needs the Site API HTTPS address, its CA, its own projected token file,
expected ServiceAccount subject and configured environment. Certificate and
hostname verification are required. Redirects and implicit HTTP proxies are disabled.
Both the Runtime token and the reviewer's token are read afresh for each request.

The public Site API has its own hostname: `https://site.staging.pharosfactory.ai` in
staging and the existing catalogued `site.pharos.<dev-domain>` route in the dev box.
Clients configure only that origin and append `/v1/site/commands/*` or
`/v1/site/ssh-access/*`. The edge routes that hostname and prefixes to the separate
Site API backend. Missing or invalid native grants must produce the Site API's
`401`/`403` response, never a web page or a browser-login redirect. Native agents
receive public CA trust only; they do not receive the edge key and do not disable
hostname or SNI verification. They also require no OIDC redirect or callback.

Invalid credentials return `401`, forbidden environments `403`, conflicting
results or sequence gaps `409`, invalid payloads `422`, and unavailable identity
or database connections `503`. Authentication failure or unavailable authority
prevents database access. Error replies omit tokens, provider responses and SQL.

Site-host qualification covers deployed route-specific timeouts and body limits,
missing/wrong/expired/revoked/cross-scope credentials, native pull/acknowledge round
trips, and hostname/TLS validation. The dev-box lane uses the established reachable
edge through `site.pharos.<dev-domain>` while retaining the certificate SAN and
public CA. The staging lane repeats the same checks on
`site.staging.pharosfactory.ai` after the distributed route owner and DNS owner
publish it.

Runtime progress journalling and recovery are implemented under 4C. The caller can
retry an identical outcome after a lost response. A command acknowledgement records
handling of that instruction, rather than completion of the whole workload.

## Native Site credentials

A Site without Kubernetes presents a native bearer: `psb_` and 256 random bits.
`SITE_API_NATIVE_GRANTS_FILE` names a digest-only grant store that the listener
reads on every request. A grant names its principal (`commands@<site>` or
`ssh-access@<site>`), Site, route scope, stream environment for command grants,
expiry and revocation. The listener serves one Site, `SITE_API_SITE_ID`, and every
native grant is checked against it before any storage read; a grant for another Site
answers the shared `site_mismatch` (`403`). With native grants configured but no
`SITE_API_SITE_ID` the listener refuses to start, so the check is never absent;
Kubernetes-only listeners need no Site, since their tokens name none. Command routes
accept only `commands` grants and SSH access
routes only `ssh-access` grants (`403` otherwise); unknown, expired and revoked
bearers return `401`. `scripts/site_credentials.py` issues and revokes grants as an
administration operation and writes each bearer once to a private file for the
Site host. `NativeSiteBearer` reads that file on every Runtime request. The production
installer treats each release as immutable and supports credential or trust rotation
through a new verified release and controlled activation/restart, not by editing a
retained release in place. The SSH access agent receives its bearer through systemd
`LoadCredential=`, which snapshots the source at service start, so that service also
requires activation/restart for rotation. Kubernetes TokenReview remains available
when `SITE_API_KUBERNETES_URL` is configured; at least one verifier is required.

## Architecture-reset baseline

This adapts the pull/acknowledge handlers, identity-verifier port, Kubernetes
TokenReview verifier and projected-token HTTPS client from architecture-reset
`48e60e542`. The destination becomes the configured environment instead of `siteId`.
The existing Python 4A store retains its pure pull query and atomic result recording.
Python ports stay independent of FastAPI, Kubernetes and generated wire classes.

## Normal installation

The [installation](../../../../../gitops/manifests/pharos/base/site-api/README.md)
describes the `site-api` Deployment, its inputs in each profile and how a Runtime
connects. Verified on 2026-10-21:

- `pnpm verify` passed, including **1,847 Metallo tests**. New checks render the
  local, k3s, dev-box and workstation-loop profiles; reject Kubernetes tokens in other
  workloads, long-lived or foreign-audience reviewer tokens and foreign or readable
  certificate Secrets; complete strict Python TLS handshakes for every Service name;
  and issue the certificate once from the credentials Job and the local launcher.
  The `database-admin` image (OpenSSL 3.5.9) issued it as the Job's read-only user.
- `pnpm test:integration` passed **284 Metallo cases**. Its Compose stack omits only
  the Site API, which needs TokenReview.
- `make metallo-site-api-test` passed all **57 Jobs** on the dev box with the installed
  GitOps ServiceAccount, Service, Deployment and NetworkPolicies, changed only in
  namespace, image, grants and database label, and a certificate from the installation
  issuer. It delivered both environments' commands, rejected untrusted and wrong-audience
  tokens, replaced the API Pod twice with exact result replay and Runtime recovery,
  returned `503` while the TokenReview grant was removed, and saved **16 commands and
  16 outcomes**, 8 acknowledged per environment. The namespace and grant were removed.
- `make metallo-site-api-jobs-test` passed all **14 phases** with the same resources:
  real Jobs, Runtime Pod replacement, quota blocking and release, native failure and
  cancellation. Two earlier runs failed on native timing while the Site API served
  every command: the supervisor's 150-second wait ended as a replaced Runtime reported
  a finished Job, and a quota-released Job exceeded its 90-second wall time after about
  a minute of Pod-creation backoff.
- `make metallo-pipeline-test` passed with the hosted coordinator, launch and
  supervision loops and Runtime: success, `actions.yaml`, failure and cancellation
  pipelines, coordinator and Runtime Pod replacement, and private application and
  KServe services with a real prediction.
- `make metallo-site-api-services-test` passed: an actual Deployment and KServe
  inference, same-Pod init, restart recovery, private access and revocation cleanup.
  Every lane removed its namespaces and TokenReview grants.

The isolated lanes ran on `metallo-dkatsimpris-1`, because the dev box with the Argo
installation was unreachable. The bootstrap grant, the in-cluster credentials Job and
the Argo rollout remain unexercised there; 8D verifies them.

## Verification evidence

Verified on 2026-10-12:

- `pnpm verify` passed, including **705 Metallo tests**, strict Mypy, Ruff,
  generated Site contract drift checks, Go checks, Web build, Kubernetes renders
  and infrastructure checks. The controller suite passed **146 tests**.
- `pnpm test:integration` passed **119 PostgreSQL cases**, including three new
  HTTPS tests for delivery, replay after restart and atomic rollback. Broker,
  service-image and browser checks also passed. Identity authority is doubled in
  the local HTTPS tests.
- The isolated dev-box lane passed **nine probe phases** against real PostgreSQL,
  TLS and Kubernetes TokenReview. It delivered all six operations in each
  environment, saved **12 commands and 12 outcomes**, and advanced each stream
  to acknowledgement 6. It checked pagination, duplicate/conflicting results,
  missing/invalid/untrusted tokens, wrong audiences, forbidden environments,
  unknown TLS trust, database network isolation and replay after API restart.
- Removing the test API's TokenReview permission made command pulls return `503`.
  After restoring the permission, PostgreSQL inspection confirmed that the recorded
  commands and outcomes remained unchanged.
- The first cluster attempt exposed a missing key-usage extension in the disposable
  CA. The generator was corrected and now verifies its certificate chain, hostname
  and server purpose with strict OpenSSL rules before deploying. TLS checks remain enabled.

Evidence was recorded under
`.local/metallo/identity-tests/dev-dkats/site-api/metallo-site-api-dev-dkats-85c708f83f54/report.json`.
Both the initial failed namespace and the successful namespace were removed,
along with their temporary TokenReview ClusterRoles and bindings. The installed
application and its databases were not changed.

The cluster lane uses real PostgreSQL, TLS and Kubernetes TokenReview. It simulates
the supervisor and native Runtime handling. A second Kubernetes ServiceAccount
exercises the `slurm` stream, so this does not establish a real Slurm-host credential
connection. Native execution and installed deployment rollout retain their own
verification steps.