---
title: 'DD-0045: Development Loop Interface: Image Catalogue, Deployment Receipt and Takeover Lease'
---

# DD-0045 — Development loop interface: image catalogue, deployment receipt and takeover lease

**Related ADR:** [ADR-0045](../decisions/0045-development-loop.md).

**Status:** Draft for review. The lease, mode, expiry and catalogue decisions were
reviewed with danias before implementation; dev-box qualification follows each
pull request.

**Date:** 2026-10-14.

## Scope

Three commands share one dev box:

| Command | Input | Result |
|---|---|---|
| `make deploy` | The committed checkout, the target's profiles | Cluster layers (Ansible), AAI, provider tokens, then the same delivery as `make reconcile` |
| `make reconcile` | A commit | The commit delivered to the in-cluster Git server, the images it changes built from that commit, Argo CD synced to it |
| takeover | The local checkout with uncommitted edits | Selected workloads run on the workstation (native or Docker runtime) against the dev box, under a lease |

On macOS, `deploy` and `reconcile` send the exact Git commit as a bundle to an
SSH Linux runner. The runner uses the existing Linux tools and the same receipt,
image and lease protocols. Its owned checkout is separate from operator source;
an explicit `REMOTE_ROOT` can reuse an existing installation's private state.

`make reconcile` and takeover share the same delivery interfaces. They agree on three
interfaces, described here: the image catalogue, the deployment receipt and the
takeover lease. The workstation watcher follows saved files directly. The root
`make tilt` adds a dashboard: saved files invoke takeover, while hand-back,
delivery, status and log snapshots remain explicit actions.

## Image catalogue

One file, `gitops/manifests/images.yaml`, lists every image the platform builds. It
replaces `devops/parallel/build-inventory.json` from #626 and the inputs that
`dx/common/scripts/devbox/source.py` and `themelio_compat.py` encode today.
It sits in `gitops/manifests`, since only root Applications, `apps` and `manifests`
may appear at the GitOps root.

```yaml
platform: linux/amd64
images:
  api:
    image: pharos/api                 # as the manifests name it
    application: pharos
    context: .
    dockerfile: services/api/Dockerfile
    inputs: [services/api, contracts, .dockerignore]
  realtime-gateway:
    image: pharos/realtime-gateway
    application: pharos
    context: .
    dockerfile: services/api/Dockerfile
    args: {APP: realtime-gateway}
    inputs: [services/api, contracts, .dockerignore]
  compat-web:
    image: docker.io/pharos/themelio-compat-web
    application: themelio-compat
    context: compat/themelio
    dockerfile: compat/themelio/Dockerfile
    target: web
    inputs: [compat/themelio]           # the Dockerfile copies its whole context
  # ... themelio, beto, metallo, web, compat-gitlab-root-actor, compat-k8s-root-actor
```

- **`inputs`** are the paths whose content decides the image, including shared
  modules, contracts and lockfiles. The Dockerfile, `args`, `target` and
  `platform` are always inputs. Test files and documentation are excluded by the
  existing `runtime_path` rule.
- **The fingerprint** of an image is the SHA-256 of its inputs' paths, modes and
  contents, plus the Dockerfile, `args`, `target` and `platform`. It is computed
  from a Git tree for `make reconcile` and from the working tree for takeover, with
  the same function.
- **A check** (`check_platform_gitops.py`) fails when a Dockerfile `COPY`s a path
  that is not an input, when an input does not exist, or when an image named by
  `gitops/manifests` is not in the catalogue.
- **`application`** names the Argo CD Application whose workloads use the image;
  the lease and the receipt use it.

## Deployment receipt

`make reconcile` and `make deploy` write one receipt per delivery, and takeover
reads the last one as its baseline.

- **In the cluster:** the existing `pharos-platform-ownership` ConfigMap gains the
  delivered commit (already its `revision`) and the image map. That map is part of
  the root's inputs, so the root, not a patch on a child, passes each digest to its
  Application's `kustomize.images`, as #679 does for `themelio-compat`.
- **Locally:** `.local/metallo/<target>/receipt.json` repeats it with each image's
  fingerprint, so takeover can compare without a cluster round trip:

```json
{"schema": 1, "commit": "<40 hex>", "delivered": "2026-10-14T12:00:00Z",
 "images": {"compat-web": {"fingerprint": "<64 hex>",
                           "reference": "registry.pharos-build.svc.cluster.local:5000/compat-web@sha256:<64 hex>"}}}
```

- **Order:** the receipt and the root's inputs change only after every changed
  image has built and been pushed. A failed build leaves the previous delivery in
  place.
- **Takeover's comparison:** an image whose working-tree fingerprint equals the
  receipt's runs in the cluster. One that differs is a candidate for takeover if it
  supports a runtime there, or needs `make reconcile` if not (actors,
  migrations).

## Builds in the cluster

- A BuildKit Job per changed image in the `pharos-build` namespace. Its context is
  the delivered commit, fetched from the in-cluster Git server, so the build uses
  the commit that was delivered and nothing is uploaded from the workstation.
- An in-cluster registry in the same namespace, on a Ceph block volume, which also
  holds the BuildKit cache. k3s pulls from it through a `registries.yaml` mirror
  the `k3s` role installs, over the cluster network only.
- Requests and limits are set per Job; builds run one at a time by default.
- Images are named by digest in the root's inputs and never by tag.

## The `pharos` service

`gitops/apps/base/pharos.yaml` is installed on every dev box on wave 3, in the
bootstrap's `pharos` namespace; `apps/overlays/devbox` sets its path to
`gitops/manifests/pharos/overlays/devbox`. The overlay removes the base's Namespace
and the Minikube Ingress, so the web has no edge route until the new Themelio
replaces `themelio-compat`. It adds the namespace's administration, in sync waves:

| Wave | Resources |
|---|---|
| −4 | The credentials Job's account and Role (Secrets `get`, `create`, `patch`); the namespace-wide `dns` and `default-deny` policies; the Jobs' own egress and PostgreSQL's ingress from the provision Job |
| −3 | Sync hook `pharos-credentials` (`scripts/cluster_credentials.py`) |
| −2 | PostgreSQL's StatefulSet, Service and ServiceAccount |
| −1 | Sync hook `pharos-database-provision` (`scripts/provision_contexts.py --secrets`) |
| 0 | The workloads |

- **Credentials.** Every key in `devops/secret-keys.json` has one value, kept in
  `pharos-generated`, and each `<owner>-credentials` Secret receives only its own
  keys, so shared keys agree. A key the inventory gains is added; an existing
  value is never changed, and a Secret that differs from the record is refused. A
  lost record is recovered from the workloads' Secrets. The Job prints names only.
- **Databases.** The provision Job waits for PostgreSQL, then creates each
  context's role and database and applies its migrations from the bindings in
  `metallo-credentials`. It alone receives the administrator password.
- **`database-admin` image.** PostgreSQL's client with Python, the inventory, the
  context registry, the migrations and the scripts, running as UID 70.
  `check_platform_gitops.py` requires every registered context's binding in the
  Job and the inventory, its migrations in the image's inputs, and every
  namespace-wide policy before the hooks.
- **Transition.** `overlays/metallo-devbox` installs the same base in
  `metallo-devbox` for the Metallo controller's own loop, until its takeover moves
  to `pharos`.

## Takeover lease

A takeover borrows the Applications whose workloads it replaces. While it holds
them, Argo CD syncs neither those Applications automatically nor by hand. Every
other Application, and the shared GitLab, Waldur, AAI and edge, keep syncing.

**Mode: one developer per dev box.** A target has at most one lease. It covers
every Application the takeover touches (`pharos`, `themelio-compat` or both). A
second checkout is refused rather than given its own lease.

### The lease is a sync window

The lease has two parts:

- **A record** in the ownership ConfigMap: the holder (checkout and workstation),
  the Applications, the receipt's commit, when it was taken and when it expires.
- **A deny sync window** on the Applications' AppProject, naming only the leased
  Applications, with `manualSync: false`. A deny window stops automated syncs,
  self-heal included, and with `manualSync: false` also manual ones, for those
  Applications only.

The Applications' own sync policy does not change. The root therefore has nothing
to restore while the lease is held, and Argo CD shows the leased Applications as
OutOfSync without reverting them.

The AppProjects belong to the Ansible bootstrap and the project-writer admission
fence. The controller uses the existing bootstrap administrator identity, which
the fence already permits; Argo CD service accounts remain denied. It preserves
other windows and adds or removes only its own deny window with
`manualSync: false`, matching the Applications in the ownership record.

### Expiry

- A window has a start (`schedule`, a cron expression for the minute it begins)
  and a `duration`. The lease's window starts when it is acquired and lasts the
  lease's time to live, 12 hours by default.
- While the takeover runs, it renews the lease every 5 minutes, moving the
  record's expiry and the window's `duration` forward.
- If the workstation stops renewing (closed laptop, lost connection, crash), the
  window ends on its own. Argo CD then syncs the Applications again, and self-heal
  restores their workloads from the receipt's commit without any command.
- The next command for the target (takeover, `make reconcile` or `make deploy`)
  removes the expired record and window and continues. A later takeover from the
  same checkout starts a new lease.

### Acquire

1. Refuse if an unexpired lease, an unfinished `make deploy` or `make reconcile`,
   or a different checkout already owns the target.
2. Write the record, then add the window.
3. Wait for any operation in progress on the leased Applications to finish, or
   end it if it is for a superseded revision, as the health wait does today.
4. Only then replace the workloads.

### While held

`make reconcile` and `make deploy` refuse to deliver while an unexpired lease
exists. The holder hands back first.

### Hand-back

1. Stop the local runtime.
2. Detach replacements and restore container aliases. Queue agent removal until
   every local runtime has stopped; namespace-scoped uninstall can then reconnect
   without disrupting another service. Journal each cleanup before proceeding.
3. Remove the window and then the record. Argo CD syncs the Applications to the
   receipt's commit, and self-heal restores their workloads.
4. Wait until the Applications are Synced and Healthy at that commit.

### Interruption and recovery

- The record and a local journal mark each step, so a retried acquire or
  hand-back continues from the last completed step.
- A window without a record, or a record without a window, is completed or removed
  by the next command for the target.
- Qualification includes an interrupted acquire, an interrupted hand-back, a
  dropped connection, a sleeping workstation and an expired lease.

## Decisions from review

- Manual sync is blocked only for the Applications under takeover, through the
  deny sync window; Argo CD's RBAC does not change.
- One developer per dev box, so one lease per target.
- A lease expires when its holder stops renewing it.
- The catalogue's inputs as listed above.