> For AI agents: the complete documentation index is available at /llms.txt, the full documentation bundle is available at /llms-full.txt. # DD-0038 — Platform edge and mesh **Related ADR:** [ADR-0038 — Platform service catalogue and shared host edge](/reference/docs/decisions/0038-platform-edge-catalogue.md). **Status:** Draft for review. The A1/A2 implementation and owned development deployment are qualified; wider deployment and approval remain pending. **Date:** 2026-10-13. **Author:** Implementation/documentation prepared for ikons; platform review pending. ## Scope and authority This is platform administration, with the committed exposure configuration as its owned fact. Resolving routes is a query; reconciling the platform is an administration command. It introduces no product bounded context, aggregate, public business API, event catalogue or browser subscription. Application data and identity policy remain with their existing owners. The development stages are Kubernetes → storage → mesh → edge → Argo → `edge-final` → storage-check → route-check. Ansible owns Ceph, mesh installation and the host edge. The first `edge` pass publishes host routing before Argo has created the Site API's internal `site-api-trust` ConfigMap; `edge-final` then publishes the public CA as `pharos-site-api-trust` in `istio-system` before any gateway route is qualified. This ordering is required on both first install and reconcile. A later Argo decision must specify adoption of route resources before changing writers. The personal application controller retains its existing namespace/resources. ## ADR-derived technical notes The user instructed gateway-only migration first, without application injection or restart. The existing hostname, HTTPS port 8443 and development CA remain the browser and native-client trust contract. The shared gateway is HTTP on loopback NodePort 30088; old NodePorts remain for rollback. Live qualification established the actual K3s CNI binary search path as `/var/lib/pharos-metallo/k3s/data/cni`, not `data/current/bin`. The preflight reads containerd's configuration and checks that path. Host inotify needs at least 1024 instances and 524288 watches while retaining higher existing values. The installed CRD rejects explicit zero WebSocket duration, so the route omits the request timeout. Nginx uses HTTP/1.1 for HTTP/WebSocket upstreams and HTTP/2 for gRPC. A fingerprint-scoped TLS filter handles the existing Ceph leaf without SANs; replacing the certificate or disabling CA verification is not required. ## Configuration and source map | Responsibility | Source | | --------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Exposure contracts | Service catalogue | | Generic development values | Environment values | | Validation and route resolution | Catalogue adaptor | | Chart/tool/image pins | Istio pins | | Mesh installation and runtime preflight | Mesh playbook, component values | | Routes and policies | Route chart | | Host edge and public trust | Edge role | | Icarus-hpc host services | Icarus-hpc laboratory (uses the platform's existing Site API route) | | Deployment stages/checks | Controller, mesh checks | The catalogue contains stable contracts; private target inventory contains the VM binding and administrator API trust. Schema and semantic validation reject duplicate paths, unknown namespaces/gateways, unreviewed backends, credentials and unsafe syntax. The generated Helm schema enforces the same reviewed surface when Helm is invoked directly. Its generator is drift-checked by verify. The chart creates one host-specific VirtualService with specific paths before `/`, gateway ALLOW policies with explicit hosts, scoped backend TLS rules and paired namespace/pod/port NetworkPolicies. A chart source checksum is included in recorded Helm values. Ansible passes merged catalogue/environment values so unchanged reconciliation works without an additional Helm diff plugin. ## Traffic, trust and identity ```mermaid flowchart LR Browser[Browser through SSH tunnel] -->|HTTPS 8443| Edge[Host Nginx] Runtime[Native Runtime] -->|gRPC TLS 8443| Edge Edge -->|HTTP/1.1 or HTTP/2, loopback 30088| Gateway[Istio gateway] Gateway -->|HTTP / WebSocket| Services[AAI and laboratory browser services] Gateway -->|Verified TLS| Site[Laboratory Site gRPC API] Gateway -->|Pinned TLS| Ceph[Ceph dashboard] ``` Nginx retains the public Host/scheme and overwrites forwarded client addresses. The gateway trusts one proxy hop. Empty management CIDRs mean loopback/SSH management access. Forged forwarding headers cannot bypass the host restriction. The original CA/key stay on the host. Leaf renewal preserves that CA and occurs only for changed SANs, invalid leaf material or approaching expiry. Since 2026-10-14 the laboratory's Site Runtime uses `icarus-site.pharos.127.0.0.1.sslip.io:8443` with its existing token and durable cursor; the laboratory's own DestinationRule verifies the unchanged Site leaf and its identity (ADR-0038, Laboratory routes). SDS trust Secrets contain public CA certificates only. Ceph's filter matches only its exact backend cluster and verifies both CA signature and authoritative leaf hash. Query parameters and credentials are omitted from gateway logs. The CNI and native-sidecar configuration prepare a later injection phase; application namespaces remain outside injection. Kubernetes NetworkPolicy restricts pod traffic. Native host processes require the separate Icarus firewall design and are not qualified as isolated by this deployment. ## Laboratory routes A laboratory's routes file has `version`, `name`, `namespace` and `routes`; each route has the catalogue's host, path, access, transport, backend service, port, protocol, WebSocket, timeout and optional body fields (`#/$defs/extension` in the schema, which the chart's generated schema leaves out). 1. The laboratory's playbook passes the file as `edge_extension`. 2. The edge reads every file in `/opt/pharos-edge/extensions` on the host; the caller's file replaces the registered file of the same name. 3. `catalogue.py --extensions -` validates the set: the schema, a unique name, a namespace no catalogue route uses, hosts that are neither `pharos` nor any catalogue service's (whatever its profile or binding), hosts shared with no other laboratory, unique host+path pairs, and the gateway routing mode. 4. Each route reaches the gateway NodePort under the name `-`, and its host joins the certificate SANs. 5. After the validated reload, the edge writes the caller's file to the registry (`0644`, root), so later runs from any checkout keep serving it. The laboratory renders the gateway side from the same file and applies it with its workloads: one Gateway and one VirtualService per host in `istio-system`, an ALLOW AuthorizationPolicy per route (management routes admit loopback and the management CIDRs only), a verifying DestinationRule per HTTPS backend, a gateway egress NetworkPolicy, and an ingress NetworkPolicy per backend in its namespace. None carries `pharos.platform/route`. The laboratory probes its routes through `devbox.mesh.probe`, as `make deploy` probes the catalogue's. The platform catalogue also reserves the distinct `site.pharos` host for the installed Site API. It exposes only `/v1/site/commands/` and `/v1/site/ssh-access/`, verifies the HTTPS backend with the internal Site API trust and SNI `site-api.pharos.svc`, and has no browser-login callback. The `pharos` Application owns the Site API and Pharos web ingress policies in its namespace; the route chart must not create duplicate policies there. The browser host and Site API host are deliberately separate. Browsers and cloud-internal gateway traffic use their own TLS authorities: the public edge certificate is handed to external native callers, while `pharos-site-api-trust` remains cloud-internal. Native callers use scoped `commands@` or `ssh-access@` bearers on the exact protocol prefixes; they never perform browser OIDC. The credential and TLS handover procedure is in the Site API operator README. ## Migration and recovery 1. Bind the selected development target and verify its saved node identity. Freeze the previous AAI/Icarus writers during migration. 2. Capture issuer, CA, existing pod/PVC/Secret identities and material hashes with the qualification tool. Retain a private host backup of edge/include, TLS and Runtime configuration; record no secret value in evidence. 3. Install the pinned base, control plane, CNI and gateway. Check runtime CNI paths and watcher capacity, then readiness of every component. 4. Reconcile public trust, catalogue routes and paired policies before updating the host edge. Validate Nginx before reload. Retarget an installed Runtime only after the validated edge is active. 5. Run route, new-pod network and existing AAI/Icarus protocol lanes. Compare retained identities/material, and repeat deployment to prove no mesh/edge changes for unchanged inputs. 6. Exercise rollback to the preceding edge/Runtime binding and restoration of the qualified gateway path. Retain old NodePorts until a separate removal change is qualified. Helm releases persist progress across an interrupted mesh stage. Retry reuses those releases and pinned inputs. A failed route release uses atomic rollback; a failed Nginx validation restores the preceding configuration before reload. A renewed leaf, changed route or removed default site records a pending reload, and a changed Runtime binding a pending restart, on the host; each is cleared only after the reload or restart succeeds, so a retry after a failed run still applies it. Certificate reconciliation preserves an existing CA and fails on corrupt or incomplete CA authority rather than silently replacing it. The qualified VM backup is `/opt/pharos-edge/istio-a2-20261012`, root-only. Restore the previous Nginx/include, validate/reload, restore the original Runtime configuration and restart that service. Reconcile A2 to return to gateway routing. The operations guide carries operator commands and the qualification tools. ## Qualification and limits Live qualification below used A2 code `e4ebfbae8`, before #668 was rebased onto main. The rebase replaces the reload/restart handlers with the pending markers above; the rebased head needs the live lanes repeated. On 25 October 2026 the disposable `metallo-lfsqual-1` first installation reproduced the ordering defect: the native Site route returned HTTP 503 because `pharos-site-api-trust` was absent when the initial edge pass rendered the gateway, while the internal `site-api-trust` existed in `pharos`. The `edge-final` regression test covers first install and reconcile ordering. A subsequent deployment must verify the trust ConfigMap, exact separate host, TLS SNI and native authentication before the route is recorded as ready; the failed first run alone is not authentication evidence. | Evidence boundary | Result | | --------------------------------------------------------------------------------------------- | ------------------------------------- | | Source, generated drift, large files, syntax/lint, chart and rejection fixtures | Passed | | Real isolated PostgreSQL/HTTPS, broker, service images, sessions and browser | Passed | | Owned k3s VM: public make deploy, storage and route checks | Passed | | Interrupted mesh retry and unchanged mesh/edge reconciliation | Passed | | New restricted/tokenless pod: CNI, direct service denial, public issuer, forged-header denial | Passed | | AAI baseline/identity; Icarus browser, real Slurm, lifecycle, native and log security | Passed | | Original issuer/CA, node, application pod/PVC/Secret identity/material preservation | Passed | | Edge/Runtime rollback and restoration | Passed | | Fresh-VM/reboot, manual operator UAT, Minikube, production deployment | Not exercised | | Native job firewall isolation, application injection, Argo hand-over | Separate implementation/qualification | The linked PR contains the review-facing validation summary. The local operator evidence is indexed by `C:\\md-notes\\pharos-istio-a2-evidence-20261013\\summary.json`; private credentials and VM backups are not repository artefacts. Development qualification does not establish production readiness or wider stakeholder approval of the DD.
Page source Source snapshot · Revision `f928834c8`.