# 0038 — Platform service catalogue and shared host edge

**Status:** Accepted for the authorised development deployment; platform-wide adoption and PR review remain pending.

**Date:** 2026-10-12. Extended for A2 on 2026-10-13. Laboratory routes leave the
catalogue on 2026-10-14; see [Laboratory routes](#laboratory-routes).

**Deciders:** ikons for the development scope. Wider platform review remains pending.

**Related DD:** [DD-0038 — Platform edge and mesh](../dd/0038-platform-edge-catalogue.md).

## Context

AAI and the Icarus laboratory currently write separate pieces of the same host
Nginx configuration. New providers need one exposure contract before Istio and
Argo CD join `make deploy`. This is technical platform administration. It adds
no bounded context, aggregate, product route or subscription. The owned fact is
the committed service exposure configuration; resolution is a query and edge
reconciliation is an administration command.

### Current state and problems

Before A1, AAI owned the host TLS/Nginx configuration and Icarus wrote an include
into the same server. Browser routes used separate loopback NodePorts, and
the native Runtime bypassed the shared edge through the Site NodePort. This
split configuration ownership and made the proposed shared platform-service
deployment depend on provider-specific edge writers.

A1 introduced a common catalogue and edge owner in #667, on top of #659's AAI
relocation. A2 in #668 deploys the pinned mesh and routes on the owned
development VM. The development evidence establishes neither production adoption
nor native job firewall isolation.

## Considered option

The user selected one catalogue, one host edge and an Ansible-installed Istio
sidecar/CNI gateway before Argo CD. A1 retains NodePort forwarding for the
ownership refactor; A2 moves traffic through the shared gateway. Application
injection remains a later phase. The records below describe that instructed
solution rather than introducing additional alternatives.

## Decision

Use the common catalogue and host edge as the exposure contract. Install the
mesh through Ansible after storage, preserve the deployed issuer/CA, and route
the enabled development services through the common gateway. Keep application
contexts, data authority and browser subscriptions with their existing owners.

**See also:** [DD-0038 — Implementation, migration and qualification](../dd/0038-platform-edge-catalogue.md).

### A1 decision record

`gitops/manifests/routes/base/services.yaml` holds stable host/path, backend,
gateway, namespace and access declarations. `environments/devbox.yaml` holds
environment values and transitional NodePort bindings. Neither contains
credentials or a VM's address. JSON Schemas reject unknown fields; additional
validation rejects duplicate host/path pairs, unknown namespaces or gateways,
unreviewed backend surfaces and unexpected NodePorts. Duplicate YAML keys fail
rather than replacing an access declaration silently.

The `edge` Ansible role is the sole writer of host Nginx configuration.
The AAI command and `make deploy` invoke it through their playbooks, and a
laboratory's playbook through its own (see [Laboratory routes](#laboratory-routes)).
The role keeps the existing issuer hostname, port 8443 and `/auth/` prefix, and the
AAI's loopback NodePort 30080 for rollback.
The dev-box application's separate container browser router keeps its owner.

The original CA remains in `/opt/pharos-local-aai/tls` and is never regenerated
when it exists. An incomplete, mismatched or expired CA fails and needs operator
restoration. Only the server leaf changes when SANs change or expiry approaches;
its SANs include the existing issuer, the catalogue hosts, a wildcard below
`pharos.127.0.0.1.sslip.io` and loopback. Keys stay on the host with mode 0600.
The first reconciliation keeps a private pre-migration Nginx/TLS backup. Nginx
validates its complete configuration before reload; a validation failure restores
the preceding configuration. A pending reload is kept on the host until nginx
reloads, so a retry after a failed run cannot keep serving a superseded leaf.
Unchanged runs with nothing pending do not issue certificates or reload.

A1 supports `dev-ca` and existing HTTP browser routes. The Site gRPC endpoint and
Ceph dashboard are catalogued but explicitly disabled at the edge. Their current
machine and optional standalone endpoints remain unchanged. No namespace gains
sidecar injection. Let's Encrypt, SMTP streams, shared-environment CIDRs and
Istio routes need their own implementation and qualification.

A2 extends this decision with pinned Istio, CNI, the common gateway, host-scoped
policies, a route chart and NetworkPolicy edges. The intended deployment order
is Kubernetes → storage → mesh → edge → checks; `make deploy` still installs and
checks Kubernetes/storage during A1. Argo CD bootstrap follows A2 in its own PR
and decision. Production TLS, realm authority and provider ownership remain with
their existing owners.

## A2 — Qualified development mesh, 13 October 2026

Ansible owns Istio before Argo CD, as in the legacy deployment. Version 1.30.2
supports the dev box's Kubernetes 1.34. Charts have committed SHA-256 checksums,
images have manifest digests, and the controller caches a checksum-verified
Istioctl. The shared gateway is HTTP on loopback NodePort 30088. Nginx terminates
HTTPS on the original 8443 listener and uses HTTP/1.1 for HTTP/WebSocket
upstreams and HTTP/2 for gRPC. It overwrites forwarded client addresses; the
gateway trusts exactly one proxy hop.

Native sidecars and the chained CNI are configured, but no existing application
namespace receives injection or an application restart. The runtime containerd
configuration on K3s 1.34.11 names `/var/lib/pharos-metallo/k3s/data/cni` as its
binary directory, and `/var/lib/pharos-metallo/k3s/agent/etc/cni/net.d` as its
configuration directory. A ready CNI agent alone is insufficient evidence: a
new-pod network lane must prove the plugin runs. The host retains at least 1024
inotify instances and 524288 watches, without lowering higher existing values.

The same Helm route chart consumes the catalogue/environment values. Its
generated schema also rejects unreviewed backends when Ansible is bypassed.
Policies target explicit hosts; a host's more specific routes precede `/`.
WebSocket request timeouts are omitted because the installed CRD rejects an
explicit zero duration. Gateway egress and backend ingress use paired namespace,
pod and port selectors. The gateway admits host edge traffic, while a foreign
pod cannot enter it or the protected backend services directly. Node processes
remain outside NetworkPolicy protection; native job firewall isolation belongs
to the separate Icarus isolation change.

The gateway receives public CA certificates only, through scoped SDS trust
Secrets. The Ceph dashboard's existing self-signed leaf has no SAN. A filter
scoped to its exact backend cluster verifies the authoritative SHA-256 leaf
fingerprint and CA signature, disabling only automatic hostname matching for
that cluster. No private CA key or certificate rotation is involved.

The first mesh deployment was interrupted before readiness completed and then
reconciled from its persisted Helm releases. Live qualification covers source
verification, catalogue rejection, actual pod NetworkPolicy enforcement,
AAI identity/baseline, Icarus browser and native Slurm, offline/revocation/restart
recovery, existing identity/material preservation, repeat deployment and edge
rollback. This qualifies the owned development VM; fresh-VM/reboot, manual
operator UAT, Minikube and production deployment remain separate evidence.

Istio access logs omit all query parameters and credentials. A new service,
gateway policy, TLS trust or injection phase requires matching negative fixtures
and live qualification. Chart source changes carry a checksum in Helm values so
reconciliation notices template changes even while the chart version is fixed.

## Consequences

**Positive:** One catalogue and edge owner provide consistent URLs and reviewed
host/path authority. Preserving CA, data and previous endpoints supports an
incremental migration with rollback. The same route chart can be handed to a
future GitOps owner under an explicit ownership contract.

**Negative:** The common gateway becomes a shared availability boundary.
Operators must maintain compatible pinned Istio, CNI, K3s and Nginx inputs.
Ceph's existing certificate requires a scoped Envoy filter whose compatibility
must be requalified on upgrade. New-pod and protocol testing are necessary;
rendering and readiness alone missed real deployment failures.

**Neutral:** Development services remain outside application sidecar injection.
The old NodePorts remain available for rollback. Ceph/mesh/host provisioning
retain Ansible ownership; the later Argo decision must define route adoption.

### Migration

Migration is required for the existing AAI/Icarus edge and Runtime binding. A1
consolidates the writers; A2 switches routes and the native client's endpoint
while retaining trust and durable state. Concrete migration, verification and
rollback steps are in DD-0038. The authorised `dev-ikons` migration is complete;
other environments and wider adoption remain pending.

## Development Argo handover

[ADR-0039](0039-dev-box-argocd-ownership.md) and its companion DD implement the
subsequent development route handover. After the recorded adoption, Argo owns
the existing route resources and Ansible retains public trust renewal, mesh,
storage, host edge and native bindings. Admission fences older Helm writers.
This changes the development route writer without changing names, data or trust
authority; wider stack review and merge remain pending.

## Laboratory routes

Amended on 2026-10-14. The catalogue, its chart and Argo's routes belong to the
platform alone. A laboratory ([decision 0040](0040-platform-layout.md), decision 3)
brings its own routes file, in the catalogue's schema without profiles or
gateways, and its playbook passes it to the `edge` role:

- **Validation:** the edge accepts a laboratory's routes only on hosts no catalogue
  service uses, under the laboratory's own namespace, and only through the gateway;
  the node-port rollback mode serves no laboratory.
- **Persistence:** after a validated reload the edge registers the file in
  `/opt/pharos-edge/extensions` on the host. Every later edge run, `make deploy`
  included, keeps serving the registered hosts, so the platform neither names nor
  discovers a laboratory. Removing the file and rerunning the edge withdraws them.
- **Gateway objects:** the laboratory renders and applies its own Gateway,
  VirtualServices, AuthorizationPolicies, DestinationRules and paired
  NetworkPolicies. They carry the label `pharos.lab/route`, not the platform's
  route label, so the route-writer fence and Argo leave them alone.
- **Enforcement:** `scripts/check_architecture.py` rejects any laboratory name in
  the platform's Ansible, GitOps, edge and GitOps tooling.

The Icarus laboratory moves accordingly to `icarus.pharos` for the member's
browser and `icarus-site.pharos` for its Site Runtime, which reaches Site gRPC and
the `/site/` endpoints through that one management host
([decision 0036](0036-icarus-runtime-laboratory.md)). `pharos` keeps only the AAI,
and `site.pharos` is reserved and catalogued for the platform's own Site API. Its
public hostname is distinct from the Pharos browser hostname; the native protocol
prefixes remain `/v1/site/commands/` and `/v1/site/ssh-access/`.

## Workload tunnel restriction, 19 October 2026

SSH forwards arrive from loopback, so forwarding the operator edge to a takeover
workload also granted its loopback management access. The edge now has a
dedicated TLS listener on `127.0.0.1:8444`, generated from the same validated
catalogue. Public routes retain their host/path contracts; management and unknown
routes fail closed. The original listener retains operator access and its CIDR
checks. Workstation takeover keeps its public URLs on local `8443` while
forwarding only to the workload listener, and verifies the policy before
starting applications. Older targets require `make deploy`; there is no fallback.

This separates the traffic provided by the takeover controller. It does not
isolate native code from the developer account, Docker socket holders from
their host, or workloads from additional operator forwards opened on that host.
The [route guide](../../gitops/manifests/routes/README.md#workload-and-operator-access)
records upgrade and operator access steps.

## Profile-specific HTTP/2 connection sharing, 23 October 2026

The ARM64 single-drive profile opts into an explicit list of public browser
aliases that may share an HTTP/2 TLS connection. Browsers can reuse connections
when the address and certificate cover multiple origins; strict Host/SNI equality
returned 421 for legitimate requests between those aliases. The initial HTTP/1.1
workaround is replaced by this scoped policy.

The edge rejects configured names outside the catalogue's public browser aliases.
Both SNI and request host must belong to the explicit list, and the request must
use HTTP/2. Raw Host validation, path guards, certificate trust and per-route
access remain enforced. Cross-host requests to management paths are denied even
on the operator listener. Matching-SNI operator access retains its existing CIDR
checks. Unknown, management and unlisted hosts cannot use the exception in either
direction. HTTP/1 requests retain strict matching.

The shared default is an empty list. Other dev-box configurations and laboratory
routes retain their existing policy; new catalogue aliases need explicit review
before joining a profile's list. Real TLS fixtures exercise both configurations,
all approved origin pairs, workload and operator listeners, forged headers and
encoded management paths. The [profile guide](../../dx/arm64-singledrive/README.md)
records live qualification separately.

## Open work

- Complete review and merge of #668.
- Complete review of Argo bootstrap and ownership in ADR/DD-0039 and its stacked
  implementation, then add later provider/secrets foundations separately.
- Qualify fresh-VM/reboot, manual operator UAT, Minikube and production rollout.
- Implement native host job firewall isolation and later application injection
  in their separately scoped changes.

This supersedes decision 0035's ownership of the host edge and replaces the
laboratory Nginx include. It changes neither laboratory data nor identity policy.
The role consolidates behaviour from the current local AAI/Icarus templates;
the archived `legacy/` files are unchanged. The legacy `ssl-reverse-proxy` is
design reference only; its bastion implementation is not copied in A1.