# DD-0051 — Distributed inventory setup

Related decision: [ADR-0051](../decisions/0051-distributed-inventory-setup.md).
Status: Implementation in progress; evidence below distinguishes code from live
qualification. Date: 20 October 2026.

## Ownership and input boundary

This is technical administration, with no business context, aggregate, command API
or browser subscription. Native inventory authors topology; committed policy
selects supported behaviour. Private state records source/input revisions,
machine IDs, Kubernetes namespace/node UIDs, Ceph identity and completed layers.

`platform_deploy/inventory.py` uses Ansible's inventory, variable and template
resolution. It preserves native memberships and derives modern Kubespray aliases.
Only cluster/etcd members enter host discovery and installation. Vault credentials
remain in Ansible's configured secret facility. Public plans project explicit IP,
label, taint and device fields; connection passwords/keys and arbitrary hostvars
never enter output, hashes, images or Argo values.

`plan.py` performs raw SSH reads and Kubernetes/Ceph GET/status queries. It creates
only private local evidence; it never installs remote modules or writes target
resources. Offline plans omit observations and cannot authorise a mutation.
Mixed personal and environment selections fail before transport.

Existing-cluster observations record the API server and every kubelet version and
refuse a kubelet newer than the API server or more than three minor versions older.
This detects unsupported component skew before a cutover. It does not upgrade the
foundation; control-plane alignment remains a separate maintenance operation.

`platform_deploy/runner.py` selects the administration machine.
`DEPLOY_RUNNER=<ssh-alias>`, the variable the personal workflow already uses, is
recorded in the workstation's private `.local/deployments/<ENV>/runner.json`
before dispatch and reused; another runner, or local execution, is refused while
that binding exists. `deploy_runner.py` runs on the runner as an unprivileged
account. It receives a Git bundle into its owned
`~/.local/share/pharos-deployments/<ENV>/checkout`, verifies the commit, takes the
environment lock and runs `deploy.py` with runner paths. The private log
`delivery.log` survives a lost SSH session. Runner selection, checkout, lock and
log mechanisms (`runner_dispatch.py`, `runner_checkout.py`, `runner_logging.py`)
are shared with the personal SSH runner; the personal state layout and its
`.local` symlink stay in DX. Only selection and explicit runner paths travel;
workstation `PHAROS_*`, `TARGET` and `SSH` settings do not reach the runner's
command.

The administration machine can use direct API access or an explicit native SSH
binding. SSH forwarding is process-scoped, loopback-only and retains certificate
verification against the original API address. Forwarding and connection sharing
are disabled. The original private kubeconfig remains unchanged.

## Delivery stages

| Stage | Implementation | Qualification boundary |
| --- | --- | --- |
| F1 inventory/plan | Native loader, derived groups, closed policies, private records and mixed-target guards | Nine-node/switch fixtures and three real lab hosts; refresh the real NTUA inventory before claiming its nine-node match. |
| F2 delivery | Closed foundation layout and complete services layout, exact source bundles and shared Argo gates; shared in-cluster builders with explicit placement | Offline positive/negative renders; live self-management, repeat and registry checks pending. Full composition detail is in [DD-0058](0058-distributed-services-composition.md). |
| F3 fresh foundation | Isolated pinned Kubespray tools, stable-disk guard, replication 3, native Calico/kubelet paths, controller-only API credentials | Fresh three-VM Debian 12 run in progress. Debian 13 and ARM/GPU installation are not claimed. |
| F4 site preparation | Public hostname/certificate mode, retained management policy, separate verified SMTP TLS process | Real disposable Debian HTTPS survives SMTP DNS failures. Actual DNS/TLS/sender approval and live application edge activation belong to the reviewed complete composition. |
| F5 cutover | Read-only identity capture, offline bounded rehearsal and operational runbook | Fixtures reject changed UIDs, foreign writers and retained storage. Live resource capture and execution follow the approved site window. |
| F6 qualification | Cross-node RBD/CephFS/S3, Calico denial and registry pull probes in an owned disposable namespace | Live cold-cluster, repeat and probe results pending. Full platform activation/recovery is recorded once under DD-0058. |

## Recovery and authority

The controller takes an environment lock and freezes committed source in a private
detached checkout. Long-running installers read that checkout. Retrying a pending
foundation checks its original material inputs and machine identities; a foreign
occupied host is refused. Persist completed layers after their checks. A ready
foundation may reapply its other layers with changed public inputs while its
recorded topology holds: nodes, groups, mode, Kubernetes version and CNI. It first
records the new inputs as pending, so an interrupted run retries them. Topology
changes, upgrades and removing an installed layer stay separate operations; a
record without a topology fingerprint cannot be reconfigured. Kubernetes and Ceph
identity changes stop subsequent writes. Existing-foundation mode never
uses the fresh-host installer as a repair or adoption shortcut.

Argo receives one selected root at an exact commit. Render its complete child set
before applying it. Unknown Applications, paths, repositories or namespaces fail
closed. Preserve the previous ready delivery until source validation/builds pass;
record the new delivery only after exact-source, hook and health gates. Argo starts
and retries its own sync operations. Keep provider administration in its existing
Jobs and all consumer credentials scoped under ADR-0048.

Service-owned edge trust is a delivery dependency, not a pre-Argo foundation
input. The Site API's internal `site-api-trust` is created by the Pharos child,
after which the public CA can be published as `pharos-site-api-trust` for the
gateway. The personal DX sequence records this as a second `edge-final` stage;
a distributed first install or reconcile must preserve the same causal order
before route probes or readiness claims. The first-install failure and regression
are owned by [DD-0038](0038-platform-edge-catalogue.md) and #800 (separate
browser/Site API origins and first-install trust).

Qualification writes only owned disposable resources. Namespace deletion includes
the created UID as a precondition. Failed probes remain for inspection; they do
not grant permission to delete unrelated claims or workloads.

## Site reference and deferred inputs

The local `pharos.devops` CSLab reference uses individual `ph-*` public names,
per-hostname `/etc/letsencrypt/live/<hostname>/fullchain.pem` and `privkey.pem`,
and `relay.grnet.gr:587`. The NTUA reference contains corresponding `nc-*` names.
Committed Icarus policy records these as explicit site bindings. Actual registration
bindings, certificate ownership/renewal, relay permission and From address remain
private site inputs. Reuse existing GRNET client bindings where verified; this
change does not create clients or assume the development issuer is production AAI.

The local NTUA inventory currently describes five Kubernetes hosts; the dated
live assessment described nine. Do not replace the live topology with that older
file. Actual SSH/API/Ceph identities and disk mappings must be refreshed before
the [cutover](../../devops/runbooks/icarus-cloud-cutover.md). Retain unselected
CNPG/Valkey, registry, monitoring and Kubeflow consumers throughout.

GPU infrastructure uses an explicit split described in the
[GPU Operator guide](../gpu-operator.md). Host drivers, the Container Toolkit,
containerd's NVIDIA runtime fragment and reboot/health checks belong to a selected
GPU-host Ansible profile. The cluster-side operator, CRDs, ClusterPolicy and
operands belong to an opt-in exact-revision Argo component. The component is not
rendered for the dev-box and is suspended during existing-release adoption, so a
normal delivery cannot silently upgrade Icarus' live `26.3.3` release. The vendored
`26.7.2` chart is a supported candidate, not evidence of a completed live upgrade.
Debian 13 L40 and GH200 64KiB fresh-install qualification remains outstanding.

## Evidence

The main integration preserves short-address dev-box access while moving browser
bindings into the development environment values. Public render regressions check
the exact host/certificate set, absence of development redirects and one HTTPS
listener. The foundation composition excludes the subsequently added `llm` child.
Make selection tests retain personal defaults and reject explicit mixed inputs.
The shared runner's optional bounded private stdin frame supports the provider
selection protocol proposed in #712 without moving its policy or credentials into
Git, release packages or log output. The default transfer remains an ordinary Git
bundle. Provider activation and #727's additional platform surface remain owned by
those changes; their composition must be requalified when integrated.

Deterministic checks and live results will be recorded here after completion.
The current private evidence directory is
`.local/evidence/icaruscloudlab/`; deployment records live outside the checkout
under `~/.local/state/pharos/deployments/icaruscloudlab/`.

Manual UAT, personal Minikube/k3s, production PostgreSQL/broker/identity, actual
mail delivery, Debian 13 and GH200 remain separate lanes. No Icarus data deletion,
service replacement or mail transmission is part of this implementation session.