# DD-0058 — Complete distributed services composition

Related decision: [ADR-0058](../decisions/0058-distributed-services-composition.md).
Status: Implementation in progress; deterministic evidence is recorded below.
Date: 24 October 2026.

## Composition and ownership

`platform_gitops/distributed.py` defines one closed child set and one path per
Application. The root overlay `gitops/apps/overlays/distributed-services` keeps
the shared provider Applications, removes development AAI and selects a
distributed manifest overlay for routes, GitLab, Waldur, LLM, Pharos, Site Runtime
and retained Themelio. The `icarus-cloud-root-app.yaml` entry selects that same
composition.

Shared Pharos provider manifests live in `pharos/overlays/common`; the development
overlay still adds Mailpit, local aliases and seeds. Shared Site Runtime trust and
configuration live in `pharos-site-runtime/overlays/common`. Neither complete
composition copies development configuration.

`themelio-compat` remains in the closed child set on sync wave 4. Its Application
is not optional, its public origin is required, and its manifest overlay retains
the web, GitLab actor and Kubernetes actor. The compatibility layer is therefore
available while any legacy route remains to be migrated. Provider tokens are
administrator-delivered and expire after 30 days; delivery rotates them when
fewer than seven days remain and restarts only the compatibility consumers.

## Identity and route boundary

`platform_deploy/identity.py` loads the external-AAI file privately, validates the
shared provider contract, fingerprints its canonical content and renders the
GitLab, Waldur and Themelio consumer Secrets. An explicit public CA is required
because all three consumers mount it as their trust bundle. Discovery reads only
public metadata.

`platform_deploy/public_edge.py` maps explicit service origins to the existing
closed route catalogue. Delivery adds `activeProfiles`, `sourceChecksum` and
`cephCertificateHash` after catalogue validation; the values are passed directly
to the route Application. Storage trust is read from Ceph and applied only when
the storage profile is active. The browser and Site API have separate catalogue
hosts. HTTPS origins are normalised to lowercase hostnames without the default
`:443` before injection into the API and Centrifugo allowlists. The browser origin
is injected only into web-facing origin settings; the
Site API origin enables the two exact native prefixes. The gateway uses internal
SNI and subject-alternative-name verification for `site-api.pharos.svc`, and the
Pharos Application owns both backend ingress policies without duplicate
route-chart policies in its namespace.

The plan fails before writing state when private AAI input is absent or invalid.
Delivery re-loads and verifies the identity fingerprint before opening the
cluster API or building images. It delivers OIDC consumer Secrets before applying
the root, then waits for exact child revisions, healthy operations and required
compatibility tokens.

The public gateway route to `site-api.pharos.svc` depends on trust and policy
objects owned by the Pharos child. Service delivery must therefore complete the
Pharos Application before the final edge reconciliation and route qualification;
an edge rendered only against the pre-services foundation is incomplete. The
personal dev-box path records this as `edge-final` after Argo. The equivalent
distributed delivery uses the shared `platform_edge/site_trust.py` publisher
after all exact-revision Applications are ready. It copies only `ca.crt` from
`pharos/site-api-trust` into `istio-system/pharos-site-api-trust`, validates the
certificate and refuses to record the delivered revision or success receipt if
the CA is absent, invalid or cannot be published. The initial personal edge pass
may omit a not-yet-created CA; its post-Argo pass uses the same idempotent
publisher. Backend ingress and gateway egress policies remain part of the
exact-revision Application readiness and separate route qualification.

## Verification and deferred activation

Deterministic tests cover the exact nine-child set, removal of development AAI,
mandatory retained Themelio, exact child paths and revisions, route value
completeness, external identity fingerprinting, private-file permissions,
bounded errors, token rotation through stdin and real Kustomize renders. The route
and distributed-service render checks run in `pnpm verify`.

The disposable lab and final staging are distinct evidence lanes. The combined
qualification candidate `test/distributed-lab-800-744-lfs` assembles #800
(separate browser/Site API origins and first-install trust), #744 (native Site
API and gateway policy ownership) and #789 (GitLab LFS tuning) into
`metallo-lfsqual-1`. Its lab origins use the catalogue's separate development
Site host; this does not approve a public staging name, DNS record or certificate.
The candidate is not a reviewed deployable revision and must not be applied to
the real Icarus-cloud.

Final staging uses the separate `site.staging.pharosfactory.ai` and
`platform.staging.pharosfactory.ai` hosts. Native Site callers use only
`/v1/site/commands/` and `/v1/site/ssh-access/`, with public edge CA verification
and scoped bearers. The gateway independently verifies
`site-api.pharos.svc` against the internal CA and SNI. DNS/TLS readiness,
external AAI, the real nine-node topology and native HPC/service integration
remain separate gates. A green Argo status or a single-node lab result does not
qualify those gates.