#0045 — The development loop: deploy, reconcile and takeover

Status: Proposed. Implements stage 6 of 0040; the interfaces are in DD-0045. Date: 14 October 2026.

#Context

Since decisions 0041 and 0043, Argo CD installs GitLab, Waldur and the retained Themelio on a dev box from the exact commit make deploy delivers. The new platform still runs outside Argo CD, as metallo-devbox, and make deploy builds images on the workstation. #626 added workstation runtimes that replace selected workloads with local processes, but against its own deployment, and its takeover refused workloads that Argo CD manages. In the legacy layout, a commit was enough for Argo CD to roll out a change, and the checkout was mounted into Minikube for reload.

#Decision

1. Three commands, one owner each.

   • make deploy installs the cluster layers and public trust, then delivers AAI, shared providers and consumers through Argo like make reconcile. The previous provider layout is converted once using a durable cleanup/ installation checkpoint (0048). Existing application data is discarded during conversion; subsequent deploys preserve the new data. make reconcile requires that conversion to be ready.
   • make reconcile delivers a commit and builds only the images it changes. When all changes are desktop or Markdown files, it compares the catalogue's image fingerprints and checks the existing applications at their exact deployed revision. An unchanged, healthy deployment returns without source transfer or Argo sync, preserving its deployment receipt. Unknown inputs, runtime/configuration changes and unhealthy applications use normal delivery. Active takeover, maintenance and changed ownership still prevent this shortcut.
   • Takeover runs uncommitted edits on the workstation.

   Tilt sends saved files to takeover and offers make reconcile and make deploy.

2. The new platform is the Argo CD service pharos in the pharos namespace, installed on every dev box from gitops/manifests/pharos/overlays/devbox. It replaces metallo-devbox, and dev-box data starts afresh. Until the new Themelio replaces themelio-compat:

   • themelio-compat is installed on every dev box beside it, not by profile (amending 0043);
   • the platform's web has no edge route.

   When the new Themelio replaces it, themelio-compat becomes optional again. The namespace administers itself, so nothing generated leaves the cluster:

   • one Job creates its credentials once and never changes an existing value;
   • after PostgreSQL is ready, another Job creates each context's role and database and applies its migrations.

   Both Jobs run the database-admin image, and no runtime workload receives the PostgreSQL administrator password. overlays/metallo-devbox keeps the Metallo controller's own loop working until its takeover moves to pharos.

3. Images are built in the cluster from the delivered commit. One catalogue, gitops/manifests/images.yaml, lists each image's build and every input. Each image has a fingerprint of its inputs. BuildKit Jobs build changed images from the in-cluster Git server, and push them to an in-cluster registry that k3s pulls from. Only after every build succeeds does the deployment receipt name the new digests, which the root passes to each Application. Nothing is uploaded from the workstation, and no image is named by tag.

4. Takeover keeps #626's workstation runtimes (RUNTIME=native|docker, native by default, on macOS ARM64, Linux and WSL2), with the following from #626:

   • automatic selection of affected workloads from Git-owned runtime inputs;
   • Python and Next.js reload, and Go rebuilds;
   • the session journal and guardian;
   • scoped launch environments;
   • browser access through loopback forwards;
   • hand-back when the source is restored.

   It compares the working tree with the deployment receipt, using the catalogue's fingerprints.

5. Takeover borrows Argo CD Applications through a lease instead of refusing them. While the lease is held, a deny sync window with manualSync: false stops automated and manual syncs of the leased Applications only. Their sync policy does not change, so the root has nothing to restore. make reconcile and make deploy refuse to deliver while the lease is held. The takeover renews the lease; when renewal stops, the window ends and self-heal restores the workloads.

6. One developer per dev box. A target has at most one lease, covering every Application the takeover touches.

#Differences from the legacy GitOps

The layout follows legacy/gitops, as 0040 decided:

• Applications are in apps/base, with the path of each set by apps/overlays/<environment>;
• Kustomize services are in manifests/<service>/{base,overlays}.

It differs in the following ways, each owned by a decision:

• The root names an exact commit and image digests. The controller patches it for each delivery (0039), instead of every Application following a branch.
• Images come from in-cluster builds (decision 3), instead of CI builds that Argo CD Image Updater picks by dev-<sha> tag.
• Routes come from the central catalogue of 0038 until stage 7 of 0040 moves them to each service.
• The services project is restricted to its namespaces and kinds. The root does not prune, and the bootstrap owns the namespaces (0041).
• waldur-site-agent-cloud is not installed yet.

#Consequences

Positive: a commit reaches the cluster with one command and builds only what it changes; uncommitted edits reload on the workstation against the real GitLab, Waldur, AAI and data; Argo CD owns every service; a lost workstation cannot leave a dev box frozen.

Negative: the takeover controller needs bootstrap administrator access to manage the lease's sync windows. The existing project-writer fence already permits that identity and still denies Argo CD service accounts. The in-cluster registry and BuildKit cache use Ceph volume space and node CPU while building. Takeover still covers only workloads with a workstation runtime, so retained actor changes go through make reconcile. Legacy CPU execution is deferred until the new backend supports it; this delivery does not qualify the legacy executor.

Neutral: #626's own provider, mesh, TLS and deployment code and its ADR numbers are not carried over; its workstation runtime decisions continue here. The Minikube path (pnpm dev:up) stays separate.

#Workstation implementation

make takeover, make handback, make takeover-status and make takeover-logs run outside the tools container, with a separate private journal and API tunnel. make takeover-setup prepares the pinned native daemon installer. Docker takeover uses a containerised daemon, scoped read-only configuration and source mounts. One traffic manager serves pharos and themelio-compat; returning an individual service preserves the shared connection. Final hand-back removes queued agents before releasing the lease. The lease lasts 12 hours and renews every five; an expired holder must reacquire it.

The root make tilt uses the same workstation commands, automatically invoking takeover for saved source and offering manual hand-back and delivery actions. It leaves service selection, reload and lease renewal to the controller. The developer guide describes these commands. Removal of the older metallo-devbox loop remains a follow-up. The guide records live qualification evidence and the remaining platform checks before wider rollout.

On every workstation OS, root deploy and reconcile transfer the selected commit as a Git bundle to an owned Linux runner checkout. The runner invokes the same pinned deployment tools and in-cluster builds. Existing private deployment state can be reused through an explicit REMOTE_ROOT binding; source checkouts are preserved. Only the public browser CA and generated AAI test-user passwords return to private workstation state. This keeps native takeover and delivery available without local Docker. Setup uses the same runner selection: explicit runner, existing saved binding, then the target SSH destination. The internal Linux deployment entry point remains the runner backend; it does not dispatch another SSH delivery.

An explicit BUILD_MODE=runner builds the same committed inputs on the selected Linux delivery runner. Native platform selection, catalogue fingerprints, immutable receipts and Argo ownership remain shared. The runner exports OCI content, transfers it over the dedicated SSH connection and publishes it to the target's persistent registry before recording success. A local containerd cache alone is insufficient evidence for reuse. Optional provider catalogues pin public upstream source revisions and their release-preparation recipes; only the selected target receives those provider digest overrides.

K3S_MODE=connect binds the existing cluster's node, CA and machine identities without reinstalling K3s. Its registry and mesh gateway use private ClusterIP services, preserving other applications' NodePorts. A registry-specific containerd mirror is merged without replacing unrelated registry settings. Personal ingress coexistence remains in the opt-in profile. Ordinary dedicated dev boxes retain their loopback NodePorts and shared image catalogue.

The shared Argo routes Application forwards retained Themelio Git/LFS URLs to GitLab and admits frontend WebSockets. /api/ retains precedence for Themelio; Git forwarding removes frontend cookies and identity headers while preserving Git credentials for GitLab's authorisation checks.

#Pharos DX desktop

The Tauri 2 tray panel and Workspace dashboard consume the existing workstation controller through an allowlisted local subprocess protocol. Root .env supplies workstation defaults. Opening the app queries status; explicit takeover and hand-back actions use the same lease, engine binding and recovery as Make. Compatibility checks compare the working tree to the deployment receipt, blocking unknown or delivery-only changes. Status events contain no credentials or raw logs. The Rust host emits local IPC events; the webview does not poll the cluster. Windows uses a native window with the controller in a selected WSL2 distribution. The app's release lifecycle is independent of platform deployment: dx-v…-preview.N tags publish universal macOS preview DMGs through GitHub Actions. Signing, notarisation and automatic updates remain follow-up work before wider rollout.

Development provider sharing is updated by decision 0048. Consumer database identities and broker scopes remain separate.

The desktop and Make share a dev-box lifecycle controller. A reviewed operation runs in a detached workstation process that retains the checkout lock and writes private progress events and its outcome. CLI and desktop observers can reconnect from that checkout without starting another operation. Remote delivery and Argo retain their existing ownership; the journal adds no reconciliation authority. Setup/reset are technical host-administration commands with expiring, commit-bound review plans. Reset binds the SSH machine, saved single-node cluster, disk, ownership record and application/storage UIDs; each deletion carries a UID precondition. Once all root/child writers are suspended, reset records route restoring and Argo bootstrap ownership so existing admission guards permit namespace cleanup and reinstallation. Delivery restores Argo ownership after adoption. An ownership maintenance record fences takeover and unrelated delivery until the exact-commit health gate succeeds. Reset keeps the cluster foundation and uses normal storage finalisers to delete reviewed application data before reinstalling. Failures retain private logs and maintenance state for reviewed recovery. Progress is streamed from deployment stages through native local events to both windows.

#On-demand operator credentials

The dev box remains authoritative for generated service passwords. A separate read-only SSH command retrieves one allowlisted password after verifying the selected target. Listing entries returns metadata only. This operator facility uses the developer's existing SSH and sudo access; it creates no password vault, master password or additional cluster credential writer.

The CLI offers an explicit terminal reveal or clipboard copy. Pharos DX uses separate native commands for listing, Copy and Reveal, outside its diagnostic and activity channels. Copy writes through the native clipboard without returning the password to the webview. Reveal returns one value to transient state in the requesting Workspace window, with timed and navigation-based clearing. Neither path stores passwords in settings, status snapshots, events or log files. The allowlist excludes private SSH keys, signing keys and Kubernetes credentials.

Clipboard access and an explicit reveal expose the selected value to the developer's session. Clipboard history, terminal recording and other software running as that user remain outside this boundary. Initial administrator passwords are labelled as such and may cease to work after a manual change; retrieval never rotates or resets them. See the credential access guide.

Page source

Source snapshot · Revision f928834c8.