# Platform service exposure The A1/A2 foundation has one service catalogue, one host edge owner and a pinned Istio/CNI gateway. `make deploy` runs Kubernetes → storage → mesh → Argo bootstrap → edge → Argo sync → storage-check → route-check. The development Argo ownership design is in [ADR-0039](../../../docs/decisions/0039-dev-box-argocd-ownership.md) and [DD-0039](../../../docs/dd/0039-dev-box-argocd-ownership.md); both retain wider review as a separate boundary from the owned VM qualification. AAI lives in `dx/common/local_aai` and reaches the host edge through this catalogue. The Icarus-hpc laboratory uses the platform's existing `site.pharos` and `waldur.pharos` routes rather than owning separate edge routes. The edge role lives beside `storage`. A2 adds the `mesh` stage, catalogue routes through the shared gateway and route checks to `make deploy`; no provider is reinstalled. The rationale and scope are in [ADR-0038](../../../docs/decisions/0038-platform-edge-catalogue.md). The implementation, migration and evidence boundaries are in [DD-0038](../../../docs/dd/0038-platform-edge-catalogue.md). This guide contains the operator workflow; development qualification leaves wider review/approval open. ## Catalogue and routes `base/services.yaml` contains stable host/path, backend, namespace, gateway, transport and access contracts. `overlays/devbox/values.yaml` supplies generic dev values, profiles and the loopback gateway binding. Neither contains credentials or a VM address. Ansible validates them before changing the host and installs the route chart in `gitops/manifests/routes/chart` with the same value files that its Argo Application consumes. ```mermaid flowchart LR Browser[Browser via SSH tunnel] -->|HTTPS 8443| Edge[Host Nginx] Runtime[Native Runtime] -->|HTTPS 8443| Edge Edge -->|HTTP/1.1 or HTTP/2, loopback 30088| Gateway[Istio gateway] Gateway -->|HTTP and WebSocket| Web[Pharos web] Gateway -->|Reviewed HTTP paths| API[Go API] Gateway -->|HTTP and WebSocket| Pharos[Pharos Platform] Gateway -->|Verified TLS| Site[Site API] Gateway -->|HTTP| AAI[AAI] Gateway -->|HTTP and WebSocket| Services[GitLab, Waldur, themelio-compat, Argo CD] Gateway -->|Pinned TLS| Ceph[Ceph dashboard] ``` | Service | Host prefix | Path | Namespace/service | Previous NodePort | |---|---|---|---|---| | Pharos web | `web.pharos` | `/` | `pharos/web:8080` | No public NodePort | | Pharos API | `api.pharos` | `/api/v1/`, `/auth/`, `/healthz` | `pharos/api:8080` | No public NodePort | | Pharos Platform | `platform.pharos` | `/` | `pharos/web:8080` | No public NodePort | | Site commands | `site.pharos` | `/v1/site/commands/` | `pharos/site-api:8443` over TLS | No public NodePort | | Site SSH access | `site.pharos` | `/v1/site/ssh-access/` | `pharos/site-api:8443` over TLS | No public NodePort | | Site execution authorisation | `site.pharos` | `/v1/site/executions/` | `pharos/site-api:8443` over TLS | No public NodePort | | Site repository grants | `site.pharos` | `/v1/site/repositories/` | `pharos/site-api:8443` over TLS | No public NodePort | | AAI | `pharos` | `/auth/` | `pharos-aai/keycloak:8080` | `30080` | | Ceph dashboard | `ceph.pharos` | `/` | `rook-ceph/rook-ceph-mgr-dashboard:8443` | No standalone gateway | | Argo CD | `argo.pharos` | `/` | `argocd/argocd-server:80` | No public NodePort | | GitLab | `gitlab.pharos` | `/` | `gitlab/gitlab-webservice-default:8181` | No public NodePort | | Waldur | `waldur.pharos` | `/` | `waldur/waldur-homeport:80` | No public NodePort | | Waldur API | `waldur.pharos` | `/api/`, `/api-auth/`, `/static/`, `/health-check/` | `waldur/waldur-mastermind-api:80` | No public NodePort | | Waldur admin (management) | `waldur.pharos` | `/admin/` | `waldur/waldur-mastermind-api:80` | No public NodePort | | Chat (Open WebUI) | `open-webui.pharos` | `/` | `llm/open-webui:80` | No public NodePort | | LLM API models | `llm-api.pharos` | `/v1/models` | `llm/litellm:4000` | No public NodePort | | LLM API chat completions | `llm-api.pharos` | `/v1/chat/completions` | `llm/litellm:4000` | No public NodePort | | LLM API completions | `llm-api.pharos` | `/v1/completions` | `llm/litellm:4000` | No public NodePort | | LLM API embeddings | `llm-api.pharos` | `/v1/embeddings` | `llm/litellm:4000` | No public NodePort | | LLM API responses | `llm-api.pharos` | `/v1/responses` | `llm/litellm:4000` | No public NodePort | | LLM API messages | `llm-api.pharos` | `/v1/messages` | `llm/litellm:4000` | No public NodePort | | Zammad (`PROFILES=zammad`) | `zammad.pharos`, `zammad.localtest.me` | `/` | `zammad/zammad-nginx:8080` | No public NodePort | The `llm-chat` and `llm-api` profiles publish shared LLM access as two separate names, each one public origin: chat and the API. The API routes are an allowlist of the member inference paths a personal key needs: `/v1/models`, `/v1/chat/completions`, `/v1/completions`, `/v1/embeddings` and `/v1/responses`. LiteLLM serves its management routes (`/key`, `/team`, `/user`, `/model`, `/ui`) and other `/v1` endpoints (model info, files, batches, fine-tuning, assistants, threads, vector stores) on the same port, so each path is an exact prefix and the edge answers 404 for everything else; Metallo reaches the management routes inside the cluster, and nothing else is routed to them. Open WebUI keeps its own administration behind the application's roles, because the chat application and its administration share one origin. The `llm` chart owns the ingress policies that admit the gateway, as the `pharos` chart does; the routes chart writes only the gateway's side, so the routes Application gains no rights over `llm`. A public site approves an origin for each profile it activates (`llm-chat` for `open-webui.pharos`, `llm-api` for `llm-api.pharos`), and registers the chat client `pharos-open-webui` in the external AAI file. Kubeflow remains pending: its development binding is disabled and public-edge policy rejects the profile. Once enabled with its profile, the development route serves `kubeflow.localtest.me`, including `/oauth2/callback`, with a matching certificate. The `kubeflow-core` Application owns the proxy's ingress policy; the routes chart admits gateway egress to `oauth2-proxy` on port 4180 within its existing namespace permissions. Route checks probe `/` over HTTP while retaining WebSocket forwarding. The route declares its trusted identity headers. Edge Nginx removes client values on every path of the Kubeflow host before the gateway's external authorisation check, including login and WebSocket paths. The proxy supplies `site_username`; its pinned version omits the identity header for a missing or empty claim. The dashboard sidecar requires that header together with an admitted gateway principal. This check runs on the next hop, after external authorisation. Route-stage header removal would also discard the identity returned by the proxy. Hosts use suffix `127.0.0.1.sslip.io`, with HTTPS port 8443. The issuer remains `https://pharos.127.0.0.1.sslip.io:8443/auth/realms/pharos`. Active routes use the loopback gateway NodePort named by the environment policy (`gateway_node_port`); a fresh installation publishes 30088, and a retained installation names the port its gateway Service already serves (Icarus keeps 30080). Previous NodePorts remain for rollback. The platform web has the canonical browser host `platform.localtest.me`. Its same-origin `/api/v1/`, `/auth/` and `/connection/websocket` requests pass through the web service. `api.localtest.me` reaches the Go API directly on `/api/v1/`, `/auth/` and `/healthz`; unmatched paths remain closed at the edge, including internal authentication and diagnostics endpoints. The edge reaches that NodePort from a cluster node (by default over loopback; a site whose kube-proxy binds node ports only on node addresses names its own address in `edge_mesh_address`), so kube-proxy forwards the request with the edge node as its source. Under the default Calico pool the node's traffic leaves through the VXLAN tunnel, so the gateway sees the node's *tunnel* address rather than its underlay address; an IPIP pool behaves the same way through its own tunnel. Delivery reads the `platform-edge` hosts, resolves each to its pool's tunnel address (or to the node address when the pool does not encapsulate) and passes them to `pharos-gateway-isolation` as `edge.sources`, which admits only those addresses on port 8080. Without a resolved source the policy keeps `ingress: []`, and a missing tunnel annotation stops delivery instead of guessing. Fresh Kubespray and k3s foundations bind NodePorts to loopback. Keep the default `edge_mesh_address: 127.0.0.1` there. For the retained Icarus gateway, set `edge_mesh_address: '{{ ansible_facts.default_ipv4.address }}'` on the reviewed `platform-edge` inventory hosts whose gateway NodePort listens on that address. Use an explicit host address instead if kube-proxy selects another interface. The public edge role preserves this inventory binding on every run. `vxlanMode` and `ipipMode` may be `CrossSubnet`, which encapsulates only towards another subnet: the same pool then carries the tunnel address across subnets and the node's own address between nodes inside one. Each edge host therefore contributes both of its exact sources, because the gateway pod the request lands on is not fixed. An unencapsulated pool alongside an `Always` pool also retains both sources. Disabled pools remain relevant: disabling IPAM allocation preserves existing pod addresses and their routes during migration. An unknown mode or both encapsulation types at once stops delivery because the resolver cannot select one tunnel annotation. The `platform-edge` group names cluster nodes; Kubernetes never installs it, and the delivery re-reads it on every run, so adding or removing a host is a route change rather than a foundation change. The group is not part of the recorded foundation identity, and an existing installation, including Icarus, adopts it through `make reconcile` without a reset. An edge host that is not a cluster node, or that lacks the annotation of the pool in use, stops the delivery that names it. The platform web has the canonical browser host `platform.localtest.me`. Its same-origin `/api/v1/`, `/auth/` and `/connection/websocket` requests pass through the web service. `api.localtest.me` reaches the Go API directly on `/api/v1/`, `/auth/` and `/healthz`; unmatched paths remain closed at the edge, including internal authentication and diagnostics endpoints. Retained Themelio declares the canonical browser host `themelio.localtest.me`. Its scoped VirtualService and gateway policy serve the existing web and Git routes, with a matching certificate SAN. The old host preserves Git/LFS routes and redirects other requests to the canonical HTTPS address. The web uses its existing `AUTH_URL`, `THEMELIO_URL` and `SERVICE_PUBLIC_HOST` settings; application source stays unchanged. The dev-box edge also serves this public host on port 443 for repository clients. Other service names, including management hosts, are refused on that listener. Cluster DNS resolves the short name to the node, and takeover exports that node address to native and Docker workloads. Workstation browsers use the local relay. AAI registers the exact short callback. Private replacement client registries must include `https://themelio.localtest.me/api/auth/callback/openid_connect` for `pharos-control-plane` as well. Workstation port 443 is a loopback-only byte relay to the managed local 8443 SSH connection. Its privileged bootstrap binds sockets, drops supplementary groups/GID/UID and transfers descriptors to the developer's process before exiting. The relay handles no TLS keys and runs no SSH commands. It has bounded concurrent connections and no traffic logging. See [activation and lifecycle](../../../dx/common/GUIDE.md#short-service-addresses). The explicit setup command reads only the public CA through authenticated SSH, validates its signature, validity and the live HTTPS leaf, then offers OS trust. macOS trust is scoped to SSL in the user's keychain; Windows uses the user's root store; Linux uses system CA trust and NSS server-certificate trust. The command checks the resulting trust and reports cancellation or failure. Ordinary URL listing and service opening never install a certificate or request approval. A route may set `body: {limit, buffering}`; GitLab's unlimited, unbuffered body carries Git and LFS. The paired NetworkPolicy admits the gateway on the backend pod's own port, which differs from the Service port for Argo and Waldur. Pharos web, Pharos Platform, Site API, AAI, storage, Argo, GitLab, Waldur, themelio-compat and the two LLM hosts are the default dev-box profiles; zammad is optional (`make deploy PROFILES=zammad`). Disabled entries are never rendered. Explicit `routing.mode: nodeports` retains the AAI's browser rollback and rejects every other entry and every laboratory route. Application namespaces have `mesh: false`; this first phase installs the gateway plane without application injection/restart. Schema/semantic checks reject credentials, inventory fields, unsafe syntax, duplicate host/path pairs, unknown namespaces/gateways and unreviewed backends. An HTTPS backend names its reviewed `tls` trust: a public-CA Secret in the gateway namespace, the SNI and, where the leaf carries one, the verified name. The `edge` role publishes `pharos-ceph-trust` and `pharos-site-api-trust`. The generated chart schema also enforces the exposure contracts when Helm is called directly. Regenerate it with `scripts/generate_platform_route_schema.py`. Browser host bindings belong to the selected environment's `browser_hosts` map, keyed by the catalogue's logical host. The dev-box values retain the established short names, callbacks and Git/LFS forwarding. Public-certificate environments select only their explicit `hosts`: development aliases, redirects and loopback listeners cannot enter that plan. Python resolution and the Helm schema enforce the same boundary; public Nginx uses the selected hosts' existing certificates. A new surface needs owner review, AAI redirect URIs where applicable, negative fixtures and live qualification. Short browser hosts also cover the Pharos web and API, GitLab, Waldur, Ceph and Argo CD. GitLab and Waldur generate canonical short links and AAI callbacks; their older API/Git routes still proxy to the same backends. The dev-box port-443 listener serves public aliases only and denies management paths even from loopback. Ceph, Argo and Waldur admin remain available through the operator SSH connection on local port 8443, with the workstation relay providing local port 443. The Site API deliberately has no browser alias. Development qualification uses the catalogued `site.pharos.127.0.0.1.sslip.io` host over the development CA; production uses the environment's distinct `site.` certificate. Neither host falls through to the Pharos web `/` route, and a Site request never causes a browser-login redirect. The gateway verifies the Site API certificate using `site-api.pharos.svc` as both SNI and subject alternative name. ## Mesh and trust Istio 1.30.2 charts have committed checksums and images have manifest digests. The controller caches a verified Istioctl. CNI chains onto Flannel. K3s 1.34.11 uses `/var/lib/pharos-metallo/k3s/data/cni` for binaries and `/var/lib/pharos-metallo/k3s/agent/etc/cni/net.d` for configuration. The preflight checks the actual containerd search path: `data/current/bin` can leave a ready CNI agent while new pods cannot find its plugin. Host inotify capacity retains at least 1024 instances/524288 watches, without lowering larger existing values. Nginx retains the public host/scheme, overwrites forwarded client addresses and uses HTTP/1.1 for HTTP/WebSocket proxying, HTTP/2 for gRPC. The gateway trusts one hop: the `edge-trusted-hop` EnvoyFilter sets `xff_num_trusted_hops: 1` on its HTTP connection managers, so Envoy keeps the `X-Forwarded-Proto` the edge terminated instead of replacing it with the plain-HTTP scheme of the NodePort connection it serves, and still derives the client address from `X-Forwarded-For`. Without it every application behind the gateway built http URLs: Waldur sent `redirect_uri=http://…` to the AAI, which the provider refuses, and GitLab failed its CSRF origin check. Policies declare explicit hosts; specific paths precede `/`. WebSocket request timeout is omitted. Paired NetworkPolicies use namespace, pod and port selectors; the live new-pod lane proves direct gateway/backend access is denied. Empty management CIDRs admit loopback/SSH clients only. Forged forwarding headers cannot bypass the edge. Ceph's existing self-signed leaf has no SAN: its exact fingerprint and CA signature are checked by a filter scoped to that one cluster. SDS trust Secrets contain public certificates only, never private keys. Istio access logs omit queries and credentials. ### Workload and operator access The host also binds a workload listener to `127.0.0.1:8444`. It uses the same validated catalogue, certificate and public routes. Management hosts and paths return HTTP 403 even for loopback clients; explicit denials keep Waldur's `/admin` and `/admin/` out of its public `/` route. Unknown hosts, mismatched Host/SNI and ambiguous path encodings are rejected. Laboratory routes retain their declared public or management policy. GitLab's complete public origin accepts encoded slashes in its group, repository and file identifiers, preserving the raw request path for provider authorisation. Double encoding, backslashes and null bytes remain refused there; other origins also refuse encoded slashes. For HTTP/2 connection coalescing, a mismatched Host/SNI receives HTTP 421 so the browser can retry with the correct SNI. HTTP/1 mismatches receive HTTP 403; management restrictions still apply after a retry. The optional [profile-specific public-alias exception](../../../docs/decisions/0038-platform-edge-catalogue.md#profile-specific-http2-connection-sharing-23-october-2026) permits reviewed HTTP/2 origin pairs while keeping management requests denied. Its shared default is empty. Takeover forwards local `8443` to this workload listener, preserving the public issuer, URLs and port. Native, Docker Desktop, Colima and Linux Engine use the same policy. Before taking a lease or starting applications, the controller checks public sign-in readiness, certificate trust and management denial over the verified SSH connection. It refuses an older dev box without the listener and never falls back to the operator listener. An owned legacy forward is replaced; an unrelated process on local `8443` requires its owner to stop it. To upgrade, run `make handback`, stop any `make devbox-forward` terminal, run `make deploy`, then run `make takeover`. `make deploy` reconciles the Ansible-owned edge; `make reconcile` alone does not install this listener. Operator access remains on the host's original `8443` listener, with management CIDRs and application authentication. For browser administration, run `make handback` followed by `make devbox-forward`, then open the usual Argo CD, Ceph or Waldur administration URL. Close that forward before the next takeover. Management links return 403 while the workload forward owns local `8443`. The separate route closes the management access accidentally provided by the workload tunnel. An operator who opens another management forward may expose it to local processes and to containers through their host gateway. Native code runs with the developer's filesystem permissions, and tools with Docker socket access control that Docker host. These workflows require trusted code and a dedicated runner; the listener does not provide a workstation sandbox. Native host processes remain outside NetworkPolicy protection; the separate Icarus isolation change owns job firewall rules. Shared-environment access, Let's Encrypt, SMTP, injection and production deployment remain later work. ## Reconciliation and rollback The AAI and laboratory commands invoke the shared edge. Before adoption Helm refreshes routes; afterwards the recorded Argo writer does so. The edge requires a ready mesh before switching traffic. A chart source checksum in Helm values detects template changes while its version is unchanged. Rebuild the tools through `make dx-tools` after source changes. The first reconciliation saves the old `/etc/nginx/conf.d/pharos-local-aai.conf`, include directory and TLS directory to `/opt/pharos-edge/rollback`, private to root, retaining the snapshot on retries. The canonical Nginx file path stays the same; the file contains the resolved routes and no include. The CA remains byte-for-byte identical; a new leaf is issued for changed SANs, invalid material or approaching expiry. A broken CA stops reconciliation. The complete Nginx configuration is validated before reload and the preceding configuration restored on failure. A renewed leaf or changed route records `/opt/pharos-edge/reload-pending` before validation; it is removed only after a successful reload, so a retry after a failed validation still reloads. Unchanged runs with nothing pending issue no certificate and reload no edge. The unused Debian default HTTP site is removed. The live A2 backup is `/opt/pharos-edge/istio-a2-20261012`, with preceding Nginx/include/TLS/Runtime configuration and a qualified A2 copy. Freeze provider commands before rollback. Restore the preceding Nginx file/include, run `nginx -t`, reload Nginx, restore Runtime configuration and restart that service. Retained NodePorts restore the preceding path. Reconcile A2 to return to the gateway. Rollback/restoration were exercised; keep the backup private. ## Laboratory routes The catalogue, its chart and Argo's routes name no laboratory, which `scripts/check_architecture.py` enforces. A laboratory keeps a routes file beside its code, in this schema's `extension` form, and its playbook passes it to the edge as `edge_extension`. The edge serves it only on hosts and a namespace that no catalogue entry uses, and only through the gateway. After a validated reload, the edge registers the file in `/opt/pharos-edge/extensions` on the host, so every later run, `make deploy` included, keeps serving it. Delete the file and rerun `make deploy` to withdraw a laboratory's hosts. The laboratory applies its own gateway objects, labelled `pharos.lab/route`; the route-writer fence and Argo do not manage them. See [DD-0038](../../../docs/dd/0038-platform-edge-catalogue.md#laboratory-routes). ## Verification Argo CD 3.5.5/chart 10.10.3 use verified chart/package checksums and image digests. Bootstrap and self-management consume the same committed values. `make deploy` delivers the exact committed checkout to the private Git transport before root/ child sync; commit tracked changes first. No GitHub credential is needed on the VM. The `platform` project excludes personal application namespaces. API cache views separate the private Argo namespace from route kinds without reading backend Secrets. Bootstrap owns project policy, API bindings, transport and scoped route rights. Use `https://argo.pharos.127.0.0.1.sslip.io:8443` through the existing SSH tunnel. It requires Argo authentication. The initial administrator credential stays in `argocd/argocd-initial-admin-secret`; retrieve it only through operator tooling. Anonymous API access returns 401 and foreign-pod management access is denied. The wrapper preserves existing upstream NetworkPolicy names while removing open ingress. A default-deny policy alone cannot counter an additive allow policy. The versioned `istio-system/pharos-platform-ownership` record fences Helm before adoption. Root, `argocd` and `pharos-platform-routes` must each be Synced/Healthy at the delivered commit. The chart retains route names, paired policies and Helm history. Ansible still refreshes public trust and resolves the authoritative Ceph leaf pin; it does not reconcile the adopted chart in parallel. A provider command from a different route source fails and needs delivery through its owner checkout. A retirement opens `routes: restoring` only to delete a withdrawn route, and records the ownership it replaced in `routes_previous`, so a run killed between its two writes is resumed by the next one and the fence closes. A window with no recorded predecessor is refused rather than reopened or closed on a guess. Namespaces, PersistentVolumes, PersistentVolumeClaims and CustomResourceDefinitions are never retired. The manual recovery below opens its own window without a recorded predecessor: the predecessor a delivery left behind names Argo ownership that recovery is replacing with the Helm chart, so a recovery killed half way is finished by running it again, never by resuming a delivery's window. For recovery run `scripts/recover_platform_gitops.py --target dev-NAME` inside the tools image with the same shared locks as Make. It suspends root/children, returns routes to Helm without replacement, and verifies UIDs. `make deploy` restores adoption. `--component argocd` explicitly restores an interrupted Argo bootstrap writer while retaining the route fence. Never substitute Application existence for readiness or uninstall releases to recover ownership. DD-0039 has the recovery and qualification boundaries. ```sh pnpm verify pnpm check:large-file-review pnpm test:integration uv run --project devops --frozen python scripts/check_platform_mesh.py uv run --project devops --frozen python scripts/check_platform_gitops.py uv run --project devops --frozen python scripts/platform_edge_smoke.py ``` On the bound dev box, capture `scripts/qualify_platform_mesh.py --output ...` before/after deployment and pass `--compare ...` afterwards. It records only identities/hashes, never secret values. Use the controller or the checkout owner's private target state. `scripts/probe_platform_mesh_network.py` creates one restricted, tokenless pod with public CA trust in an owned namespace, proves direct access denial/public issuer access/management denial, then removes it. Run the existing AAI baseline/identity lanes, and the Icarus laboratory's browser/lifecycle/native/log lanes against its private state through its own hosts. They cover real Slurm execution, offline/revocation and restart recovery. Reconciliations run serially: public Make locks differ from ad hoc host tooling. Fresh-VM/reboot, manual operator UAT, Minikube and production qualification are separate evidence. A render or ready CNI agent cannot establish live behaviour. ## Staging search indexing The staging environment selects `edge.block_indexing: true`. The host edge then adds `X-Robots-Tag: noindex, nofollow` to every service response, including errors, and serves `/robots.txt` as plain text with `User-agent: *` and an empty `Disallow`. This lets search engines crawl pages to discover the exclusion and remove existing listings. `Disallow: /` would prevent them from reading the header; see [Google's indexing guidance](https://developers.google.com/search/docs/crawling-indexing/block-indexing). Other environments retain their existing indexing behaviour by default. Reconcile the Ansible-owned edge to apply this policy; an application image update alone does not change it. Public host bindings and valid certificates remain required. The staging policy leaves delivery and the public edge disabled. This setting applies when the environment is configured and the edge is reconciled. ## Static documentation The edge also serves the versioned Pharos Platform documentation directly from static files. Its separate [documentation task](../../../devops/ansible/roles/edge/tasks/documentation.yml) owns the two documentation hosts, their leaf certificates and static release pointer. It adds no Kubernetes backend or application credentials. Application routes continue to use the catalogue and gateway described above. See [documentation delivery](../../../docs/README.md#deploy-documentation) for `docs.staging.pharosfactory.ai`, `docs.localtest.me`, generic dev-box setup, artifact verification and rollback. A later platform edge delivery preserves the separate documentation configuration and its existing certificate owner.