---
title: 'DD-0039: Development Argo CD Bootstrap and Ownership'
---

# DD-0039 — Development Argo CD bootstrap and ownership

**Related ADR:** [ADR-0039](../decisions/0039-dev-box-argocd-ownership.md).

**Status:** Draft for review. The authorised `dev-ikons` implementation is
qualified; wider adoption, review and production approval remain pending.

**Date:** 2026-10-13.

## Scope and responsibilities

This is infrastructure configuration and reconciliation, with delivered source
and recorded ownership as its facts. Configuration resolution is an administration
query; bootstrap, sync and recovery are administration commands. There is no new
business context, aggregate, command API, event or subscription. The first root
selects only Argo and routes. Metallo's business registries remain unchanged.

Ansible retains Kubernetes, storage, mesh, host edge, public trust, native bindings,
Git transport, API bindings, project policy and cross-namespace route RBAC. The
personal controller retains its application resources. Argo owns its installation
and the explicitly adopted routes after handover. The stack is based on #668
and includes the merged #659 AAI bindings.

## Root Application

Following [decision 0040](../decisions/0040-platform-layout.md), the root is a Kustomize
Application, `gitops/devbox-root-app.yaml`, over `gitops/apps/overlays/devbox`. The
child Applications in `gitops/apps/base` name the placeholder revision
`delivered-commit-required`, so an unpatched child cannot track a branch. `make deploy`
applies the root with the exact delivered commit and two Kustomize patches: one sets
every child's `targetRevision`, the other sets the routes' Helm `valuesObject`
(profiles, chart checksum, Ceph leaf hash). The controller validates both inputs before
applying, and `reconcile.children` renders the root as Argo does for checks and tests.
Optional services will be Kustomize components selected on the same root. The earlier
Helm root chart rendered the same two children; only their chart paths moved.

## Source and supply chain

| Responsibility | Source |
|---|---|
| Chart, image and daemon pins | [Pins](../../gitops/manifests/argocd/pins.yaml) |
| Shared bootstrap/self-management values | [Base](../../gitops/manifests/argocd/base/values.yaml) and [dev-box](../../gitops/manifests/argocd/overlays/devbox/values.yaml) values |
| Verified upstream chart and local policy wrapper | [Chart](../../gitops/manifests/argocd/chart/Chart.yaml) |
| Root and selected child Applications | [Dev-box root](../../gitops/devbox-root-app.yaml), [child Applications](../../gitops/apps/base/), [dev-box overlay](../../gitops/apps/overlays/devbox/kustomization.yaml) |
| Repository/project/namespace/kind policy | [Project](../../gitops/manifests/argocd/bootstrap/project.yaml) |
| Persistent upload-pack transport | [Git deployment](../../gitops/manifests/argocd/bootstrap/git.yaml) |
| Writer/project admission fences | [Policies](../../gitops/manifests/argocd/bootstrap/admission.yaml) |
| Bootstrap and administrator-owned configuration | [Role](../../devops/ansible/roles/argocd/tasks/main.yml) |
| Exact source, state, gates and recovery | [Adaptors](../../devops/platform/scripts/platform_gitops/model.py), [controller](../../dx/common/scripts/devbox/argocd.py), [recovery](../../scripts/recover_platform_gitops.py) |

Argo CD 3.5.5/chart 10.10.3 were checked against upstream support and Kubernetes
compatibility on 13 October. The upstream chart is vendored unchanged and verified
by SHA-256. The wrapper and Ansible consume the same values file. Argo, Redis and
Git images use manifest digests. The Alpine Git image omits the daemon; the matching
signed upstream Alpine package is vendored separately, checksum-verified, delivered
as a public ConfigMap and extracted by a restricted init container. Runtime needs
no package download. The pins record upstream URLs and the package's GPL provenance.

The controller rejects modified tracked source before deployment, then creates a
Git bundle from the exact checkout commit. No filesystem archive includes private
state. The tokenless, non-root Git pod imports it over administrator `kubectl exec`;
only upload-pack is exposed to the repo-server. Delivered revisions retain refs
for recovery, and the controller checks the imported SHA. Roots and children all
pin that SHA. GitHub credentials never enter the VM; push is not needed to test.

The linked-worktree metadata mount is the narrow change from #657/`3584995ee`:
repository metadata is mounted read-only at its real path. Public storage/app/dev
command names are unchanged. No archived documentation framework is revived.

## Cache and privilege boundaries

```mermaid
flowchart LR
  Checkout[Committed checkout] -->|Git bundle| Git[Private Git transport]
  Bootstrap[Ansible bootstrap] --> Argo[Argo CD]
  Git --> Root[Root at delivered commit]
  Root --> Self[Argo self-management]
  Root --> Routes[Existing catalogue routes]
  Trust[Ansible public trust renewal] --> Gateway[Shared gateway]
  Routes --> Gateway
```

Two verified API bindings point to the same K3s cluster. Self-management uses
`https://kubernetes.default.svc:443` and caches only `argocd`; route management
uses `https://kubernetes.default.svc.cluster.local` and caches only route kinds
in explicit namespaces. The implicit in-cluster binding is disabled. Both use
public Kubernetes CA trust and an exec credential reading the caller's rotating
projected token. There is no copied administrator certificate, key or bearer token.
The explicit root port avoids normalising that binding to the service IP.

The engine assumes uniform kind permissions across a binding's cached namespaces.
Separating these views avoids granting backend Secret/configuration access to
make its cache work. Controller RBAC admits only required names/kinds and scoped
route writes. Bootstrap owns the project/default-project policy; the default has
no grants. Admission rejects Argo ServiceAccount edits to those projects. Named
Argo roles have only the bind/escalate permissions needed for self-management.

The Argo UI uses local Argo authentication through the existing development CA,
SSH/loopback management edge and gateway. Anonymous API access returns 401; a
foreign pod and forged forwarding header cannot reach the management route.
The upstream chart creates a server policy admitting all ingress. The wrapper
replaces all four upstream policy names with narrow rules, preserving their UIDs;
no prune/delete is used. Redis/repository peers are explicit and local. Git has
no API token or outbound network access. Host-job firewall isolation remains the
separate Icarus follow-up; pod policy is not evidence for native isolation.

## Adoption and reconciliation

`make deploy` runs Kubernetes → storage → mesh → Argo bootstrap → edge → Argo sync
→ storage-check → route-check. Bootstrap creates the Argo backend before the edge
renders its paired policy. The edge always renews public trust independently.

1. Check the target's saved node UID and disk, lock the selected target/cluster,
   and validate committed source/pins before mutation.
2. Capture issuer/CA, original pod/PVC/Secret identities/material hashes, and a
   separate private VM backup. Preserve the A2 backup and all application state.
3. Bootstrap/recover only the recorded Argo writer. Maintain administrator-owned
   policy, Git transport, API bindings and scoped cross-namespace rights.
4. Resolve profiles, chart source checksum and Ceph's authoritative SHA-256 leaf
   fingerprint. Render the same catalogue/environment and record the exact
   adoption diff/resource UIDs. Retain the original Helm values privately.
5. Write `adopting` and the delivered revision/inputs to the ownership ConfigMap
   with Kubernetes resourceVersion checks before starting root reconciliation.
   Admission now admits only the Argo controller for labelled route resources.
6. Sync without prune, replacement or namespace creation. Require root, `argocd`
   and `pharos-platform-routes` to be Synced, Healthy and at the expected revision,
   with successful operation state. Retire only routes the catalogue withdrew, as
   described below. Verify unchanged adopted UIDs, then record `argo/argo`. A
   repeat at the same revision reuses ownership and health gates.

A catalogue withdrawal is the one deliberate deletion, and every Application has
one: `retire(cluster, application, commit)` reads that Application's own
withdrawal and deletes exactly that set, so a child that must never prune can
still stop rendering an object. Argo keeps no `delete` verb and the routes
Application keeps `prune: false`, so the routes' removal belongs to the deploy
that already writes the window: admission admits administrators only while
ownership records `restoring`. Nothing else pays for that bookkeeping — a
Namespace, a PersistentVolume and a PersistentVolumeClaim are never retired here
at all, because withdrawing one destroys more than the object the catalogue
stopped rendering.
Delivery retires an object only when all of these hold: it carries this
Application's tracking id (the `argocd.argoproj.io/tracking-id` annotation, or
the `app.kubernetes.io/instance` label when Argo tracks by label), a catalogue
route also carries `pharos.platform/route=true`, Argo's current comparison marks
it for pruning (`requiresPruning`) or a successful operation at this revision
reports its prune as `PruneSkipped`, and the delivered catalogue is applied at
the delivered revision with every desired resource Synced and present, only the
withdrawn ones out of sync. A failed operation's resource list is not evidence,
because a selective sync names the resources it applies and the ones it prunes
in one list. That covers both a superseding route and an explicit catalogue
removal, and an aggregate Healthy is not enough: Argo's health skips resources a
catalogue has not applied yet.
The aggregate `OutOfSync` Argo reports for a resource that needs pruning is
itself not a blocker: the desired resources are validated individually and the
withdrawn entries account for the aggregate, so an upgrade whose only difference
is the withdrawal still retires it — including when Argo starts no operation at
all, because there is nothing left to sync and the comparison is then the only
record of the withdrawal.
Delivery records `restoring`, deletes each verified object with its own
credentials, returns to the recorded state, and starts one clean sync at the delivered revision
when Argo still records the denied targeted prune as its last operation; every step
is idempotent and logged. The recorded source is compared with Argo's normalised
form (`model.compared_source`), which drops JSON null members such as distributed
`domain: null`. An object that fails a
condition fails the gate instead of being removed. This is how a box that already
stopped in `adopting` — the split of `site-api` into `site-api-commands` and
`site-api-ssh-access` is the first case — recovers on the next `make deploy`
without a reset or a manual `kubectl`.

The edge splits trust renewal from chart reconciliation. While Argo owns routes,
provider commands can update profiles/trust only from the recorded route source;
a different checkout must deliver its source with `make deploy`. Platform deploy
collects dynamic inputs before the Argo stage. Ceph renewal preserves authoritative
pinning and CA validation; a mismatched transition fails closed until exact sync
completes. No second Helm writer or TLS-validation bypass is introduced.

## Interruption and recovery

Application existence is not readiness. `adopting` survives interruption and retry
continues Argo. It never causes an automatic parallel Helm upgrade. A failed sync
at an unchanged revision is explicitly retryable through the deploy controller.
The admission fence also rejects the previous A2 Helm chart/writer.

`scripts/recover_platform_gitops.py --target dev-ikons` suspends the root before
its children, waits for quiescence, records `restoring`, clears only Argo tracking
and upgrades the same route release with its private prior values. It compares
UIDs before recording Ansible ownership. Root/children remain suspended until
`make deploy` restores adoption. Helm history, namespace/PVC/Secret identities,
trust and application data are retained. There is no uninstall or destructive prune.

For an interrupted/unhealthy Argo installation, `--component argocd` explicitly
suspends the same graph and restores the bootstrap writer while keeping the route
fence. The role requires quiescence before Helm recovery of an existing Application.
It resumes the same committed chart/values; a successful handover skips Helm.
Root resume transfers only intentionally suspended fields on the reviewed root.

Run recovery inside the usual tools image with the shared `/metallo-locks` volume;
host-local locks do not serialise against Make. The development backup is
`/opt/pharos-edge/argocd-20261013`, root-only. Private controller snapshots/diffs
remain under `.local`; no raw logs or credentials belong in Git/PR/shared notes.

## Qualification and remaining boundaries

The implementation is tested on the existing `dev-ikons` VM. Source and live
qualification cover pinned rendering, rejection fixtures, syntax/lint, actual
public deploy/repeat, preserved issuer/CA/node/application identities, exact SHA
health, drift, interrupted bootstrap/adoption recovery, old-writer denial and
rollback/restore. The final foreign-pod probe distinguishes the upstream additive
allow-all policy from correct isolation. Automated Argo login and existing AAI/
Icarus browser, lifecycle, native, revocation, log and Site/Ceph paths are exercised.

`pnpm verify`, large-file review and isolated PostgreSQL/broker/images/sessions/
browser integration are separate evidence boundaries. The local operator summary
records current PR HEAD and deployed code SHA independently; documentation/test
commits do not imply a different live source revision.

Fresh-VM/reboot, manual operator UAT, Minikube, shared/staging/GPU and production
rollouts are not exercised. Later provider/secrets foundations, GitHub App access,
Image Updater, application injection and native firewall isolation need their own
changes and qualification. Development evidence does not establish wider approval.