# 0045 — The development loop: deploy, reconcile and takeover

Status: Proposed. Implements stage 6 of [0040](0040-platform-layout.md); the
interfaces are in [DD-0045](../dd/0045-development-loop.md).
Date: 14 October 2026.

## Context

Since [decisions 0041](0041-gitlab-waldur-services.md) and
[0043](0043-themelio-compat-service.md), Argo CD installs GitLab, Waldur and the
retained Themelio on a dev box from the exact commit `make deploy` delivers. The
new platform still runs outside Argo CD, as `metallo-devbox`, and `make deploy`
builds images on the workstation. #626 added workstation runtimes that replace
selected workloads with local processes, but against its own deployment, and its
takeover refused workloads that Argo CD manages. In the legacy layout, a commit was
enough for Argo CD to roll out a change, and the checkout was mounted into Minikube
for reload.

## Decision

1. **Three commands, one owner each.**
   - `make deploy` installs the cluster layers and public trust, then
     delivers AAI, shared providers and consumers through Argo like `make reconcile`.
     The previous provider layout is converted once using a durable cleanup/
     installation checkpoint ([0048](0048-shared-development-infrastructure.md)).
     Existing application data is discarded during conversion; subsequent deploys
     preserve the new data. `make reconcile` requires that conversion to be ready.
   - `make reconcile` delivers a commit and builds only the images it changes.
     When all changes are desktop or Markdown files, it compares the catalogue's
     image fingerprints and checks the existing applications at their exact
     deployed revision. An unchanged, healthy deployment returns without source
     transfer or Argo sync, preserving its deployment receipt. Unknown inputs,
     runtime/configuration changes and unhealthy applications use normal delivery.
     Active takeover, maintenance and changed ownership still prevent this shortcut.
   - Takeover runs uncommitted edits on the workstation.

   Tilt sends saved files to takeover and offers `make reconcile` and `make deploy`.
2. **The new platform is the Argo CD service `pharos`** in the `pharos` namespace,
   installed on every dev box from `gitops/manifests/pharos/overlays/devbox`. It
   replaces `metallo-devbox`, and dev-box data starts afresh. Until the new Themelio
   replaces `themelio-compat`:
   - `themelio-compat` is installed on every dev box beside it, not by profile
     (amending [0043](0043-themelio-compat-service.md));
   - the platform's web has no edge route.

   When the new Themelio replaces it, `themelio-compat` becomes optional again.
   The namespace administers itself, so nothing generated leaves the cluster:
   - one Job creates its credentials once and never changes an existing value;
   - after PostgreSQL is ready, another Job creates each context's role and
     database and applies its migrations.

   Both Jobs run the `database-admin` image, and no runtime workload receives the
   PostgreSQL administrator password. `overlays/metallo-devbox` keeps the Metallo
   controller's own loop working until its takeover moves to `pharos`.
3. **Images are built in the cluster from the delivered commit.** One catalogue,
   `gitops/manifests/images.yaml`, lists each image's build and every input. Each
   image has a fingerprint of its inputs. BuildKit Jobs build changed images from
   the in-cluster Git server, and push them to an in-cluster registry that k3s
   pulls from. Only
   after every build succeeds does the deployment receipt name the new digests,
   which the root passes to each Application. Nothing is uploaded from the
   workstation, and no image is named by tag.
4. **Takeover keeps #626's workstation runtimes** (`RUNTIME=native|docker`, native
   by default, on macOS ARM64, Linux and WSL2), with the following from #626:
   - automatic selection of affected workloads from Git-owned runtime inputs;
   - Python and Next.js reload, and Go rebuilds;
   - the session journal and guardian;
   - scoped launch environments;
   - browser access through loopback forwards;
   - hand-back when the source is restored.

   It compares the working tree with the deployment receipt, using the catalogue's
   fingerprints.
5. **Takeover borrows Argo CD Applications through a lease** instead of refusing
   them. While the lease is held, a deny sync window with `manualSync: false` stops
   automated and manual syncs of the leased Applications only. Their sync policy
   does not change, so the root has nothing to restore. `make reconcile` and
   `make deploy` refuse to deliver while the lease is held. The takeover renews the
   lease; when renewal stops, the window ends and self-heal restores the workloads.
6. **One developer per dev box.** A target has at most one lease, covering every
   Application the takeover touches.

## Differences from the legacy GitOps

The layout follows `legacy/gitops`, as [0040](0040-platform-layout.md) decided:
- Applications are in `apps/base`, with the path of each set by
  `apps/overlays/<environment>`;
- Kustomize services are in `manifests/<service>/{base,overlays}`.

It differs in the following ways, each owned by a decision:

- **The root names an exact commit and image digests.** The controller patches it
  for each delivery ([0039](0039-dev-box-argocd-ownership.md)), instead of every
  Application following a branch.
- **Images come from in-cluster builds** (decision 3), instead of CI builds that
  Argo CD Image Updater picks by `dev-<sha>` tag.
- **Routes come from the central catalogue** of
  [0038](0038-platform-edge-catalogue.md) until stage 7 of 0040 moves them to each
  service.
- **The services project is restricted** to its namespaces and kinds. The root
  does not prune, and the bootstrap owns the namespaces
  ([0041](0041-gitlab-waldur-services.md)).
- **`waldur-site-agent-cloud` is not installed** yet.

## Consequences

**Positive:** a commit reaches the cluster with one command and builds only what
it changes; uncommitted edits reload on the workstation against the real GitLab,
Waldur, AAI and data; Argo CD owns every service; a lost workstation cannot leave
a dev box frozen.

**Negative:** the takeover controller needs bootstrap administrator access to
manage the lease's sync windows. The existing project-writer fence already permits
that identity and still denies Argo CD service accounts. The in-cluster registry and BuildKit cache use Ceph volume
space and node CPU while building. Takeover still covers only workloads with a
workstation runtime, so retained actor changes go through `make reconcile`.
Legacy CPU execution is deferred until the new backend supports it; this delivery
does not qualify the legacy executor.

**Neutral:** #626's own provider, mesh, TLS and deployment code and its ADR
numbers are not carried over; its workstation runtime decisions continue here.
The Minikube path (`pnpm dev:up`) stays separate.

## Workstation implementation

`make takeover`, `make handback`, `make takeover-status` and `make takeover-logs`
run outside the tools container, with a separate private journal and API tunnel.
`make takeover-setup` prepares the pinned native daemon installer. Docker takeover
uses a containerised daemon, scoped read-only configuration and source mounts.
One traffic manager serves `pharos` and `themelio-compat`; returning an individual
service preserves the shared connection. Final hand-back removes queued agents
before releasing the lease. The lease lasts 12 hours and renews every five;
an expired holder must reacquire it.

The root `make tilt` uses the same workstation commands, automatically invoking
takeover for saved source and offering manual hand-back and delivery actions.
It leaves service selection, reload and lease renewal to the controller.
The [developer guide](../development.md) describes these commands. Removal of
the older `metallo-devbox` loop remains a follow-up. The guide records live
qualification evidence and the remaining platform checks before wider rollout.

On every workstation OS, root `deploy` and `reconcile` transfer the selected commit as a Git
bundle to an owned Linux runner checkout. The runner invokes the same pinned
deployment tools and in-cluster builds. Existing private deployment state can
be reused through an explicit `REMOTE_ROOT` binding; source checkouts are
preserved. Only the public browser CA and generated AAI test-user passwords
return to private workstation state. This keeps native takeover and delivery
available without local Docker. Setup uses the same runner selection: explicit
runner, existing saved binding, then the target SSH destination. The internal
Linux deployment entry point remains the runner backend; it does not dispatch
another SSH delivery.

An explicit `BUILD_MODE=runner` builds the same committed inputs on the selected
Linux delivery runner. Native platform selection, catalogue fingerprints,
immutable receipts and Argo ownership remain shared. The runner exports OCI
content, transfers it over the dedicated SSH connection and publishes it to the
target's persistent registry before recording success. A local containerd cache
alone is insufficient evidence for reuse. Optional provider catalogues pin public
upstream source revisions and their release-preparation recipes; only the selected
target receives those provider digest overrides.

`K3S_MODE=connect` binds the existing cluster's node, CA and machine identities
without reinstalling K3s. Its registry and mesh gateway use private ClusterIP
services, preserving other applications' NodePorts. A registry-specific containerd
mirror is merged without replacing unrelated registry settings. Personal ingress
coexistence remains in the opt-in profile. Ordinary dedicated dev boxes retain
their loopback NodePorts and shared image catalogue.

The shared Argo routes Application forwards retained Themelio Git/LFS URLs to
GitLab and admits frontend WebSockets. `/api/` retains precedence for Themelio;
Git forwarding removes frontend cookies and identity headers while preserving
Git credentials for GitLab's authorisation checks.

## Pharos DX desktop

The Tauri 2 tray panel and Workspace dashboard consume the existing workstation
controller through an allowlisted local subprocess protocol. Root `.env` supplies
workstation defaults. Opening the app queries status; explicit takeover and
hand-back actions use the same lease, engine binding and recovery as Make.
Compatibility checks compare the working tree to the deployment receipt, blocking
unknown or delivery-only changes. Status events contain no credentials or raw logs.
The Rust host emits local IPC events; the webview does not poll the cluster.
Windows uses a native window with the controller in a selected WSL2 distribution.
The app's release lifecycle is independent of platform deployment: `dx-v…-preview.N`
tags publish universal macOS preview DMGs through GitHub Actions. Signing,
notarisation and automatic updates remain follow-up work before wider rollout.

Development provider sharing is updated by [decision 0048](0048-shared-development-infrastructure.md). Consumer database identities and broker scopes remain separate.

The desktop and Make share a dev-box lifecycle controller. A reviewed operation
runs in a detached workstation process that retains the checkout lock and writes
private progress events and its outcome. CLI and desktop observers can reconnect
from that checkout without starting another operation. Remote delivery and Argo
retain their existing ownership; the journal adds no reconciliation authority.
Setup/reset are
technical host-administration commands with expiring, commit-bound review plans.
Reset binds the SSH machine, saved single-node cluster, disk, ownership record
and application/storage UIDs; each deletion carries a UID precondition. Once
all root/child writers are suspended, reset records route `restoring` and Argo
`bootstrap` ownership so existing admission guards permit namespace cleanup and
reinstallation. Delivery restores Argo ownership after adoption. An
ownership maintenance record fences takeover and unrelated delivery until the
exact-commit health gate succeeds. Reset keeps the cluster foundation and uses
normal storage finalisers to delete reviewed application data before reinstalling.
Failures retain private logs and maintenance state for reviewed recovery. Progress
is streamed from deployment stages through native local events to both windows.

### On-demand operator credentials

The dev box remains authoritative for generated service passwords. A separate
read-only SSH command retrieves one allowlisted password after verifying the
selected target. Listing entries returns metadata only. This operator facility
uses the developer's existing SSH and sudo access; it creates no password vault,
master password or additional cluster credential writer.

The CLI offers an explicit terminal reveal or clipboard copy. Pharos DX uses
separate native commands for listing, Copy and Reveal, outside its diagnostic and
activity channels. Copy writes through the native clipboard without returning the
password to the webview. Reveal returns one value to transient state in the
requesting Workspace window, with timed and navigation-based clearing. Neither
path stores passwords in settings, status snapshots, events or log files.
The allowlist excludes private SSH keys, signing keys and Kubernetes credentials.

Clipboard access and an explicit reveal expose the selected value to the
developer's session. Clipboard history, terminal recording and other software
running as that user remain outside this boundary. Initial administrator
passwords are labelled as such and may cease to work after a manual change;
retrieval never rotates or resets them. See the
[credential access guide](../../dx/common/CREDENTIALS.md).