For AI agents: the complete documentation index is available at /llms.txt, the full documentation bundle is available at /llms-full.txt, and this page is available as Markdown at /reference/docs/decisions/0038-platform-edge-catalogue.md.

0038 — Platform service catalogue and shared host edge

Status: Accepted for the authorised development deployment; platform-wide adoption and PR review remain pending.

Date: 2026-10-12. Extended for A2 on 2026-10-13. Laboratory routes leave the catalogue on 2026-10-14; see Laboratory routes.

Deciders: ikons for the development scope. Wider platform review remains pending.

Related DD: DD-0038 — Platform edge and mesh.

Context

AAI and the Icarus laboratory currently write separate pieces of the same host Nginx configuration. New providers need one exposure contract before Istio and Argo CD join make deploy. This is technical platform administration. It adds no bounded context, aggregate, product route or subscription. The owned fact is the committed service exposure configuration; resolution is a query and edge reconciliation is an administration command.

Current state and problems

Before A1, AAI owned the host TLS/Nginx configuration and Icarus wrote an include into the same server. Browser routes used separate loopback NodePorts, and the native Runtime bypassed the shared edge through the Site NodePort. This split configuration ownership and made the proposed shared platform-service deployment depend on provider-specific edge writers.

A1 introduced a common catalogue and edge owner in #667, on top of #659's AAI relocation. A2 in #668 deploys the pinned mesh and routes on the owned development VM. The development evidence establishes neither production adoption nor native job firewall isolation.

Considered option

The user selected one catalogue, one host edge and an Ansible-installed Istio sidecar/CNI gateway before Argo CD. A1 retains NodePort forwarding for the ownership refactor; A2 moves traffic through the shared gateway. Application injection remains a later phase. The records below describe that instructed solution rather than introducing additional alternatives.

Decision

Use the common catalogue and host edge as the exposure contract. Install the mesh through Ansible after storage, preserve the deployed issuer/CA, and route the enabled development services through the common gateway. Keep application contexts, data authority and browser subscriptions with their existing owners.

See also: DD-0038 — Implementation, migration and qualification.

A1 decision record

gitops/manifests/routes/base/services.yaml holds stable host/path, backend, gateway, namespace and access declarations. environments/devbox.yaml holds environment values and transitional NodePort bindings. Neither contains credentials or a VM's address. JSON Schemas reject unknown fields; additional validation rejects duplicate host/path pairs, unknown namespaces or gateways, unreviewed backend surfaces and unexpected NodePorts. Duplicate YAML keys fail rather than replacing an access declaration silently.

The edge Ansible role is the sole writer of host Nginx configuration. The AAI command and make deploy invoke it through their playbooks, and a laboratory's playbook through its own (see Laboratory routes). The role keeps the existing issuer hostname, port 8443 and /auth/ prefix, and the AAI's loopback NodePort 30080 for rollback. The dev-box application's separate container browser router keeps its owner.

The original CA remains in /opt/pharos-local-aai/tls and is never regenerated when it exists. An incomplete, mismatched or expired CA fails and needs operator restoration. Only the server leaf changes when SANs change or expiry approaches; its SANs include the existing issuer, the catalogue hosts, a wildcard below pharos.127.0.0.1.sslip.io and loopback. Keys stay on the host with mode 0600. The first reconciliation keeps a private pre-migration Nginx/TLS backup. Nginx validates its complete configuration before reload; a validation failure restores the preceding configuration. A pending reload is kept on the host until nginx reloads, so a retry after a failed run cannot keep serving a superseded leaf. Unchanged runs with nothing pending do not issue certificates or reload.

A1 supports dev-ca and existing HTTP browser routes. The Site gRPC endpoint and Ceph dashboard are catalogued but explicitly disabled at the edge. Their current machine and optional standalone endpoints remain unchanged. No namespace gains sidecar injection. Let's Encrypt, SMTP streams, shared-environment CIDRs and Istio routes need their own implementation and qualification.

A2 extends this decision with pinned Istio, CNI, the common gateway, host-scoped policies, a route chart and NetworkPolicy edges. The intended deployment order is Kubernetes → storage → mesh → edge → checks; make deploy still installs and checks Kubernetes/storage during A1. Argo CD bootstrap follows A2 in its own PR and decision. Production TLS, realm authority and provider ownership remain with their existing owners.

A2 — Qualified development mesh, 13 October 2026

Ansible owns Istio before Argo CD, as in the legacy deployment. Version 1.30.2 supports the dev box's Kubernetes 1.34. Charts have committed SHA-256 checksums, images have manifest digests, and the controller caches a checksum-verified Istioctl. The shared gateway is HTTP on loopback NodePort 30088. Nginx terminates HTTPS on the original 8443 listener and uses HTTP/1.1 for HTTP/WebSocket upstreams and HTTP/2 for gRPC. It overwrites forwarded client addresses; the gateway trusts exactly one proxy hop.

Native sidecars and the chained CNI are configured, but no existing application namespace receives injection or an application restart. The runtime containerd configuration on K3s 1.34.11 names /var/lib/pharos-metallo/k3s/data/cni as its binary directory, and /var/lib/pharos-metallo/k3s/agent/etc/cni/net.d as its configuration directory. A ready CNI agent alone is insufficient evidence: a new-pod network lane must prove the plugin runs. The host retains at least 1024 inotify instances and 524288 watches, without lowering higher existing values.

The same Helm route chart consumes the catalogue/environment values. Its generated schema also rejects unreviewed backends when Ansible is bypassed. Policies target explicit hosts; a host's more specific routes precede /. WebSocket request timeouts are omitted because the installed CRD rejects an explicit zero duration. Gateway egress and backend ingress use paired namespace, pod and port selectors. The gateway admits host edge traffic, while a foreign pod cannot enter it or the protected backend services directly. Node processes remain outside NetworkPolicy protection; native job firewall isolation belongs to the separate Icarus isolation change.

The gateway receives public CA certificates only, through scoped SDS trust Secrets. The Ceph dashboard's existing self-signed leaf has no SAN. A filter scoped to its exact backend cluster verifies the authoritative SHA-256 leaf fingerprint and CA signature, disabling only automatic hostname matching for that cluster. No private CA key or certificate rotation is involved.

The first mesh deployment was interrupted before readiness completed and then reconciled from its persisted Helm releases. Live qualification covers source verification, catalogue rejection, actual pod NetworkPolicy enforcement, AAI identity/baseline, Icarus browser and native Slurm, offline/revocation/restart recovery, existing identity/material preservation, repeat deployment and edge rollback. This qualifies the owned development VM; fresh-VM/reboot, manual operator UAT, Minikube and production deployment remain separate evidence.

Istio access logs omit all query parameters and credentials. A new service, gateway policy, TLS trust or injection phase requires matching negative fixtures and live qualification. Chart source changes carry a checksum in Helm values so reconciliation notices template changes even while the chart version is fixed.

Consequences

Positive: One catalogue and edge owner provide consistent URLs and reviewed host/path authority. Preserving CA, data and previous endpoints supports an incremental migration with rollback. The same route chart can be handed to a future GitOps owner under an explicit ownership contract.

Negative: The common gateway becomes a shared availability boundary. Operators must maintain compatible pinned Istio, CNI, K3s and Nginx inputs. Ceph's existing certificate requires a scoped Envoy filter whose compatibility must be requalified on upgrade. New-pod and protocol testing are necessary; rendering and readiness alone missed real deployment failures.

Neutral: Development services remain outside application sidecar injection. The old NodePorts remain available for rollback. Ceph/mesh/host provisioning retain Ansible ownership; the later Argo decision must define route adoption.

Migration

Migration is required for the existing AAI/Icarus edge and Runtime binding. A1 consolidates the writers; A2 switches routes and the native client's endpoint while retaining trust and durable state. Concrete migration, verification and rollback steps are in DD-0038. The authorised dev-ikons migration is complete; other environments and wider adoption remain pending.

Development Argo handover

ADR-0039 and its companion DD implement the subsequent development route handover. After the recorded adoption, Argo owns the existing route resources and Ansible retains public trust renewal, mesh, storage, host edge and native bindings. Admission fences older Helm writers. This changes the development route writer without changing names, data or trust authority; wider stack review and merge remain pending.

Laboratory routes

Amended on 2026-10-14. The catalogue, its chart and Argo's routes belong to the platform alone. A laboratory (decision 0040, decision 3) brings its own routes file, in the catalogue's schema without profiles or gateways, and its playbook passes it to the edge role:

  • Validation: the edge accepts a laboratory's routes only on hosts no catalogue service uses, under the laboratory's own namespace, and only through the gateway; the node-port rollback mode serves no laboratory.
  • Persistence: after a validated reload the edge registers the file in /opt/pharos-edge/extensions on the host. Every later edge run, make deploy included, keeps serving the registered hosts, so the platform neither names nor discovers a laboratory. Removing the file and rerunning the edge withdraws them.
  • Gateway objects: the laboratory renders and applies its own Gateway, VirtualServices, AuthorizationPolicies, DestinationRules and paired NetworkPolicies. They carry the label pharos.lab/route, not the platform's route label, so the route-writer fence and Argo leave them alone.
  • Enforcement: scripts/check_architecture.py rejects any laboratory name in the platform's Ansible, GitOps, edge and GitOps tooling.

The Icarus laboratory moves accordingly to icarus.pharos for the member's browser and icarus-site.pharos for its Site Runtime, which reaches Site gRPC and the /site/ endpoints through that one management host (decision 0036). pharos keeps only the AAI, and site.pharos is reserved and catalogued for the platform's own Site API. Its public hostname is distinct from the Pharos browser hostname; the native protocol prefixes remain /v1/site/commands/ and /v1/site/ssh-access/.

Workload tunnel restriction, 19 October 2026

SSH forwards arrive from loopback, so forwarding the operator edge to a takeover workload also granted its loopback management access. The edge now has a dedicated TLS listener on 127.0.0.1:8444, generated from the same validated catalogue. Public routes retain their host/path contracts; management and unknown routes fail closed. The original listener retains operator access and its CIDR checks. Workstation takeover keeps its public URLs on local 8443 while forwarding only to the workload listener, and verifies the policy before starting applications. Older targets require make deploy; there is no fallback.

This separates the traffic provided by the takeover controller. It does not isolate native code from the developer account, Docker socket holders from their host, or workloads from additional operator forwards opened on that host. The route guide records upgrade and operator access steps.

Profile-specific HTTP/2 connection sharing, 23 October 2026

The ARM64 single-drive profile opts into an explicit list of public browser aliases that may share an HTTP/2 TLS connection. Browsers can reuse connections when the address and certificate cover multiple origins; strict Host/SNI equality returned 421 for legitimate requests between those aliases. The initial HTTP/1.1 workaround is replaced by this scoped policy.

The edge rejects configured names outside the catalogue's public browser aliases. Both SNI and request host must belong to the explicit list, and the request must use HTTP/2. Raw Host validation, path guards, certificate trust and per-route access remain enforced. Cross-host requests to management paths are denied even on the operator listener. Matching-SNI operator access retains its existing CIDR checks. Unknown, management and unlisted hosts cannot use the exception in either direction. HTTP/1 requests retain strict matching.

The shared default is an empty list. Other dev-box configurations and laboratory routes retain their existing policy; new catalogue aliases need explicit review before joining a profile's list. Real TLS fixtures exercise both configurations, all approved origin pairs, workload and operator listeners, forged headers and encoded management paths. The profile guide records live qualification separately.

Open work

  • Complete review and merge of #668.
  • Complete review of Argo bootstrap and ownership in ADR/DD-0039 and its stacked implementation, then add later provider/secrets foundations separately.
  • Qualify fresh-VM/reboot, manual operator UAT, Minikube and production rollout.
  • Implement native host job firewall isolation and later application injection in their separately scoped changes.

This supersedes decision 0035's ownership of the host edge and replaces the laboratory Nginx include. It changes neither laboratory data nor identity policy. The role consolidates behaviour from the current local AAI/Icarus templates; the archived legacy/ files are unchanged. The legacy ssl-reverse-proxy is design reference only; its bastion implementation is not copied in A1.

Page source

Source snapshot · Revision f928834c8.