#DD-0039 — Development Argo CD bootstrap and ownership

Related ADR: ADR-0039.

Status: Draft for review. The authorised dev-ikons implementation is qualified; wider adoption, review and production approval remain pending.

Date: 2026-10-13.

#Scope and responsibilities

This is infrastructure configuration and reconciliation, with delivered source and recorded ownership as its facts. Configuration resolution is an administration query; bootstrap, sync and recovery are administration commands. There is no new business context, aggregate, command API, event or subscription. The first root selects only Argo and routes. Metallo's business registries remain unchanged.

Ansible retains Kubernetes, storage, mesh, host edge, public trust, native bindings, Git transport, API bindings, project policy and cross-namespace route RBAC. The personal controller retains its application resources. Argo owns its installation and the explicitly adopted routes after handover. The stack is based on #668 and includes the merged #659 AAI bindings.

#Root Application

Following decision 0040, the root is a Kustomize Application, gitops/devbox-root-app.yaml, over gitops/apps/overlays/devbox. The child Applications in gitops/apps/base name the placeholder revision delivered-commit-required, so an unpatched child cannot track a branch. make deploy applies the root with the exact delivered commit and two Kustomize patches: one sets every child's targetRevision, the other sets the routes' Helm valuesObject (profiles, chart checksum, Ceph leaf hash). The controller validates both inputs before applying, and reconcile.children renders the root as Argo does for checks and tests. Optional services will be Kustomize components selected on the same root. The earlier Helm root chart rendered the same two children; only their chart paths moved.

#Source and supply chain

Argo CD 3.5.5/chart 10.10.3 were checked against upstream support and Kubernetes compatibility on 13 October. The upstream chart is vendored unchanged and verified by SHA-256. The wrapper and Ansible consume the same values file. Argo, Redis and Git images use manifest digests. The Alpine Git image omits the daemon; the matching signed upstream Alpine package is vendored separately, checksum-verified, delivered as a public ConfigMap and extracted by a restricted init container. Runtime needs no package download. The pins record upstream URLs and the package's GPL provenance.

The controller rejects modified tracked source before deployment, then creates a Git bundle from the exact checkout commit. No filesystem archive includes private state. The tokenless, non-root Git pod imports it over administrator kubectl exec; only upload-pack is exposed to the repo-server. Delivered revisions retain refs for recovery, and the controller checks the imported SHA. Roots and children all pin that SHA. GitHub credentials never enter the VM; push is not needed to test.

The linked-worktree metadata mount is the narrow change from #657/3584995ee: repository metadata is mounted read-only at its real path. Public storage/app/dev command names are unchanged. No archived documentation framework is revived.

#Cache and privilege boundaries

Two verified API bindings point to the same K3s cluster. Self-management uses https://kubernetes.default.svc:443 and caches only argocd; route management uses https://kubernetes.default.svc.cluster.local and caches only route kinds in explicit namespaces. The implicit in-cluster binding is disabled. Both use public Kubernetes CA trust and an exec credential reading the caller's rotating projected token. There is no copied administrator certificate, key or bearer token. The explicit root port avoids normalising that binding to the service IP.

The engine assumes uniform kind permissions across a binding's cached namespaces. Separating these views avoids granting backend Secret/configuration access to make its cache work. Controller RBAC admits only required names/kinds and scoped route writes. Bootstrap owns the project/default-project policy; the default has no grants. Admission rejects Argo ServiceAccount edits to those projects. Named Argo roles have only the bind/escalate permissions needed for self-management.

The Argo UI uses local Argo authentication through the existing development CA, SSH/loopback management edge and gateway. Anonymous API access returns 401; a foreign pod and forged forwarding header cannot reach the management route. The upstream chart creates a server policy admitting all ingress. The wrapper replaces all four upstream policy names with narrow rules, preserving their UIDs; no prune/delete is used. Redis/repository peers are explicit and local. Git has no API token or outbound network access. Host-job firewall isolation remains the separate Icarus follow-up; pod policy is not evidence for native isolation.

#Adoption and reconciliation

make deploy runs Kubernetes → storage → mesh → Argo bootstrap → edge → Argo sync → storage-check → route-check. Bootstrap creates the Argo backend before the edge renders its paired policy. The edge always renews public trust independently.

1. Check the target's saved node UID and disk, lock the selected target/cluster, and validate committed source/pins before mutation.
2. Capture issuer/CA, original pod/PVC/Secret identities/material hashes, and a separate private VM backup. Preserve the A2 backup and all application state.
3. Bootstrap/recover only the recorded Argo writer. Maintain administrator-owned policy, Git transport, API bindings and scoped cross-namespace rights.
4. Resolve profiles, chart source checksum and Ceph's authoritative SHA-256 leaf fingerprint. Render the same catalogue/environment and record the exact adoption diff/resource UIDs. Retain the original Helm values privately.
5. Write adopting and the delivered revision/inputs to the ownership ConfigMap with Kubernetes resourceVersion checks before starting root reconciliation. Admission now admits only the Argo controller for labelled route resources.
6. Sync without prune, replacement or namespace creation. Require root, argocd and pharos-platform-routes to be Synced, Healthy and at the expected revision, with successful operation state. Retire only routes the catalogue withdrew, as described below. Verify unchanged adopted UIDs, then record argo/argo. A repeat at the same revision reuses ownership and health gates.

A catalogue withdrawal is the one deliberate deletion, and every Application has one: retire(cluster, application, commit) reads that Application's own withdrawal and deletes exactly that set, so a child that must never prune can still stop rendering an object. Argo keeps no delete verb and the routes Application keeps prune: false, so the routes' removal belongs to the deploy that already writes the window: admission admits administrators only while ownership records restoring. Nothing else pays for that bookkeeping — a Namespace, a PersistentVolume and a PersistentVolumeClaim are never retired here at all, because withdrawing one destroys more than the object the catalogue stopped rendering. Delivery retires an object only when all of these hold: it carries this Application's tracking id (the argocd.argoproj.io/tracking-id annotation, or the app.kubernetes.io/instance label when Argo tracks by label), a catalogue route also carries pharos.platform/route=true, Argo's current comparison marks it for pruning (requiresPruning) or a successful operation at this revision reports its prune as PruneSkipped, and the delivered catalogue is applied at the delivered revision with every desired resource Synced and present, only the withdrawn ones out of sync. A failed operation's resource list is not evidence, because a selective sync names the resources it applies and the ones it prunes in one list. That covers both a superseding route and an explicit catalogue removal, and an aggregate Healthy is not enough: Argo's health skips resources a catalogue has not applied yet. The aggregate OutOfSync Argo reports for a resource that needs pruning is itself not a blocker: the desired resources are validated individually and the withdrawn entries account for the aggregate, so an upgrade whose only difference is the withdrawal still retires it — including when Argo starts no operation at all, because there is nothing left to sync and the comparison is then the only record of the withdrawal. Delivery records restoring, deletes each verified object with its own credentials, returns to the recorded state, and starts one clean sync at the delivered revision when Argo still records the denied targeted prune as its last operation; every step is idempotent and logged. The recorded source is compared with Argo's normalised form (model.compared_source), which drops JSON null members such as distributed domain: null. An object that fails a condition fails the gate instead of being removed. This is how a box that already stopped in adopting — the split of site-api into site-api-commands and site-api-ssh-access is the first case — recovers on the next make deploy without a reset or a manual kubectl.

The edge splits trust renewal from chart reconciliation. While Argo owns routes, provider commands can update profiles/trust only from the recorded route source; a different checkout must deliver its source with make deploy. Platform deploy collects dynamic inputs before the Argo stage. Ceph renewal preserves authoritative pinning and CA validation; a mismatched transition fails closed until exact sync completes. No second Helm writer or TLS-validation bypass is introduced.

#Interruption and recovery

Application existence is not readiness. adopting survives interruption and retry continues Argo. It never causes an automatic parallel Helm upgrade. A failed sync at an unchanged revision is explicitly retryable through the deploy controller. The admission fence also rejects the previous A2 Helm chart/writer.

scripts/recover_platform_gitops.py --target dev-ikons suspends the root before its children, waits for quiescence, records restoring, clears only Argo tracking and upgrades the same route release with its private prior values. It compares UIDs before recording Ansible ownership. Root/children remain suspended until make deploy restores adoption. Helm history, namespace/PVC/Secret identities, trust and application data are retained. There is no uninstall or destructive prune.

For an interrupted/unhealthy Argo installation, --component argocd explicitly suspends the same graph and restores the bootstrap writer while keeping the route fence. The role requires quiescence before Helm recovery of an existing Application. It resumes the same committed chart/values; a successful handover skips Helm. Root resume transfers only intentionally suspended fields on the reviewed root.

Run recovery inside the usual tools image with the shared /metallo-locks volume; host-local locks do not serialise against Make. The development backup is /opt/pharos-edge/argocd-20261013, root-only. Private controller snapshots/diffs remain under .local; no raw logs or credentials belong in Git/PR/shared notes.

#Qualification and remaining boundaries

The implementation is tested on the existing dev-ikons VM. Source and live qualification cover pinned rendering, rejection fixtures, syntax/lint, actual public deploy/repeat, preserved issuer/CA/node/application identities, exact SHA health, drift, interrupted bootstrap/adoption recovery, old-writer denial and rollback/restore. The final foreign-pod probe distinguishes the upstream additive allow-all policy from correct isolation. Automated Argo login and existing AAI/ Icarus browser, lifecycle, native, revocation, log and Site/Ceph paths are exercised.

pnpm verify, large-file review and isolated PostgreSQL/broker/images/sessions/ browser integration are separate evidence boundaries. The local operator summary records current PR HEAD and deployed code SHA independently; documentation/test commits do not imply a different live source revision.

Fresh-VM/reboot, manual operator UAT, Minikube, shared/staging/GPU and production rollouts are not exercised. Later provider/secrets foundations, GitHub App access, Image Updater, application injection and native firewall isolation need their own changes and qualification. Development evidence does not establish wider approval.

Page source

Source snapshot · Revision f928834c8.