#DD-0038 — Platform edge and mesh

Related ADR: ADR-0038 — Platform service catalogue and shared host edge.

Status: Draft for review. The A1/A2 implementation and owned development deployment are qualified; wider deployment and approval remain pending.

Date: 2026-10-13.

Author: Implementation/documentation prepared for ikons; platform review pending.

#Scope and authority

This is platform administration, with the committed exposure configuration as its owned fact. Resolving routes is a query; reconciling the platform is an administration command. It introduces no product bounded context, aggregate, public business API, event catalogue or browser subscription. Application data and identity policy remain with their existing owners.

The development stages are Kubernetes → storage → mesh → edge → Argo → edge-final → storage-check → route-check. Ansible owns Ceph, mesh installation and the host edge. The first edge pass publishes host routing before Argo has created the Site API's internal site-api-trust ConfigMap; edge-final then publishes the public CA as pharos-site-api-trust in istio-system before any gateway route is qualified. This ordering is required on both first install and reconcile. A later Argo decision must specify adoption of route resources before changing writers. The personal application controller retains its existing namespace/resources.

#ADR-derived technical notes

The user instructed gateway-only migration first, without application injection or restart. The existing hostname, HTTPS port 8443 and development CA remain the browser and native-client trust contract. The shared gateway is HTTP on loopback NodePort 30088; old NodePorts remain for rollback.

Live qualification established the actual K3s CNI binary search path as /var/lib/pharos-metallo/k3s/data/cni, not data/current/bin. The preflight reads containerd's configuration and checks that path. Host inotify needs at least 1024 instances and 524288 watches while retaining higher existing values.

The installed CRD rejects explicit zero WebSocket duration, so the route omits the request timeout. Nginx uses HTTP/1.1 for HTTP/WebSocket upstreams and HTTP/2 for gRPC. A fingerprint-scoped TLS filter handles the existing Ceph leaf without SANs; replacing the certificate or disabling CA verification is not required.

#Configuration and source map

The catalogue contains stable contracts; private target inventory contains the VM binding and administrator API trust. Schema and semantic validation reject duplicate paths, unknown namespaces/gateways, unreviewed backends, credentials and unsafe syntax. The generated Helm schema enforces the same reviewed surface when Helm is invoked directly. Its generator is drift-checked by verify.

The chart creates one host-specific VirtualService with specific paths before /, gateway ALLOW policies with explicit hosts, scoped backend TLS rules and paired namespace/pod/port NetworkPolicies. A chart source checksum is included in recorded Helm values. Ansible passes merged catalogue/environment values so unchanged reconciliation works without an additional Helm diff plugin.

#Traffic, trust and identity

Nginx retains the public Host/scheme and overwrites forwarded client addresses. The gateway trusts one proxy hop. Empty management CIDRs mean loopback/SSH management access. Forged forwarding headers cannot bypass the host restriction.

The original CA/key stay on the host. Leaf renewal preserves that CA and occurs only for changed SANs, invalid leaf material or approaching expiry. Since 2026-10-14 the laboratory's Site Runtime uses icarus-site.pharos.127.0.0.1.sslip.io:8443 with its existing token and durable cursor; the laboratory's own DestinationRule verifies the unchanged Site leaf and its identity (ADR-0038, Laboratory routes). SDS trust Secrets contain public CA certificates only. Ceph's filter matches only its exact backend cluster and verifies both CA signature and authoritative leaf hash. Query parameters and credentials are omitted from gateway logs.

The CNI and native-sidecar configuration prepare a later injection phase; application namespaces remain outside injection. Kubernetes NetworkPolicy restricts pod traffic. Native host processes require the separate Icarus firewall design and are not qualified as isolated by this deployment.

#Laboratory routes

A laboratory's routes file has version, name, namespace and routes; each route has the catalogue's host, path, access, transport, backend service, port, protocol, WebSocket, timeout and optional body fields (#/$defs/extension in the schema, which the chart's generated schema leaves out).

1. The laboratory's playbook passes the file as edge_extension.
2. The edge reads every file in /opt/pharos-edge/extensions on the host; the caller's file replaces the registered file of the same name.
3. catalogue.py --extensions - validates the set: the schema, a unique name, a namespace no catalogue route uses, hosts that are neither pharos nor any catalogue service's (whatever its profile or binding), hosts shared with no other laboratory, unique host+path pairs, and the gateway routing mode.
4. Each route reaches the gateway NodePort under the name <laboratory>-<route>, and its host joins the certificate SANs.
5. After the validated reload, the edge writes the caller's file to the registry (0644, root), so later runs from any checkout keep serving it.

The laboratory renders the gateway side from the same file and applies it with its workloads: one Gateway and one VirtualService per host in istio-system, an ALLOW AuthorizationPolicy per route (management routes admit loopback and the management CIDRs only), a verifying DestinationRule per HTTPS backend, a gateway egress NetworkPolicy, and an ingress NetworkPolicy per backend in its namespace. None carries pharos.platform/route. The laboratory probes its routes through devbox.mesh.probe, as make deploy probes the catalogue's.

The platform catalogue also reserves the distinct site.pharos host for the installed Site API. It exposes only /v1/site/commands/ and /v1/site/ssh-access/, verifies the HTTPS backend with the internal Site API trust and SNI site-api.pharos.svc, and has no browser-login callback. The pharos Application owns the Site API and Pharos web ingress policies in its namespace; the route chart must not create duplicate policies there.

The browser host and Site API host are deliberately separate. Browsers and cloud-internal gateway traffic use their own TLS authorities: the public edge certificate is handed to external native callers, while pharos-site-api-trust remains cloud-internal. Native callers use scoped commands@<site> or ssh-access@<site> bearers on the exact protocol prefixes; they never perform browser OIDC. The credential and TLS handover procedure is in the Site API operator README.

#Migration and recovery

1. Bind the selected development target and verify its saved node identity. Freeze the previous AAI/Icarus writers during migration.
2. Capture issuer, CA, existing pod/PVC/Secret identities and material hashes with the qualification tool. Retain a private host backup of edge/include, TLS and Runtime configuration; record no secret value in evidence.
3. Install the pinned base, control plane, CNI and gateway. Check runtime CNI paths and watcher capacity, then readiness of every component.
4. Reconcile public trust, catalogue routes and paired policies before updating the host edge. Validate Nginx before reload. Retarget an installed Runtime only after the validated edge is active.
5. Run route, new-pod network and existing AAI/Icarus protocol lanes. Compare retained identities/material, and repeat deployment to prove no mesh/edge changes for unchanged inputs.
6. Exercise rollback to the preceding edge/Runtime binding and restoration of the qualified gateway path. Retain old NodePorts until a separate removal change is qualified.

Helm releases persist progress across an interrupted mesh stage. Retry reuses those releases and pinned inputs. A failed route release uses atomic rollback; a failed Nginx validation restores the preceding configuration before reload. A renewed leaf, changed route or removed default site records a pending reload, and a changed Runtime binding a pending restart, on the host; each is cleared only after the reload or restart succeeds, so a retry after a failed run still applies it. Certificate reconciliation preserves an existing CA and fails on corrupt or incomplete CA authority rather than silently replacing it.

The qualified VM backup is /opt/pharos-edge/istio-a2-20261012, root-only. Restore the previous Nginx/include, validate/reload, restore the original Runtime configuration and restart that service. Reconcile A2 to return to gateway routing. The operations guide carries operator commands and the qualification tools.

#Qualification and limits

Live qualification below used A2 code e4ebfbae8, before #668 was rebased onto main. The rebase replaces the reload/restart handlers with the pending markers above; the rebased head needs the live lanes repeated.

On 25 October 2026 the disposable metallo-lfsqual-1 first installation reproduced the ordering defect: the native Site route returned HTTP 503 because pharos-site-api-trust was absent when the initial edge pass rendered the gateway, while the internal site-api-trust existed in pharos. The edge-final regression test covers first install and reconcile ordering. A subsequent deployment must verify the trust ConfigMap, exact separate host, TLS SNI and native authentication before the route is recorded as ready; the failed first run alone is not authentication evidence.

The linked PR contains the review-facing validation summary. The local operator evidence is indexed by C:\\md-notes\\pharos-istio-a2-evidence-20261013\\summary.json; private credentials and VM backups are not repository artefacts. Development qualification does not establish production readiness or wider stakeholder approval of the DD.

Page source

Source snapshot · Revision f928834c8.