--- title: "DD-0019: Container Image Tag Management — Implementation" --- # DD-0019: Container Image Tag Management — Implementation **Related ADR:** [ADR-0019: Container Image Tag Management for GitOps Deployments](/adr/0019-container-image-tag-management) **Status:** Draft **Date:** 2026-06-29 **Authors:** TBD ## ADR-derived technical notes These notes capture implementation detail surfaced while scoping ADR-0019, kept for the chosen direction (Option B direct-to-Application + Option R1 retention). Stale per-option notes (the rejected Options A/D git write-back, and the incorrect "GitLab Container Registry cleanup policy") have been removed. - **Re-enabling immutable tags:** the per-commit tags already existed and were commented out in `themelio.config.yaml` (commit `e2b0ecaa`): `dev-${GIT_COMMIT}`, `staging-${GIT_COMMIT}`, `prod-${GIT_COMMIT}`. `GIT_COMMIT` is **already short** — `scripts/build.sh` sets it via `git rev-parse --short HEAD` — and tags are expanded with **`envsubst`**, which does **not** support a bash substring (`${GIT_COMMIT:0:8}`). So re-enabling is simply uncommenting the existing lines; no substring expression. - **Pinning standardization:** all services converge on a **hardcoded tag** in `deployment.yaml`. No git-side kustomize `images:` block is needed — Image Updater's `spec.source.kustomize.images` override matches by image name and replaces the tag regardless. `waldur-site-agent-cloud`'s staging overlay (which previously used an `images:` block) was converted to the same hardcoded-tag patch as its dev/prod overlays. - **ArgoCD specifics:** root apps are `gitops/-root-app.yaml` with `automated: { prune: true, selfHeal: true }`, `repoURL` the same repo, `path: gitops/apps/overlays/`, `targetRevision: HEAD`. The app-of-apps renders the child Applications from git, which is why `selfHeal` would otherwise scrub a direct image override (see Design §4). - **No git write-back / no same-repo loop:** because the chosen write-back is direct-to-Application, nothing is committed to git, so the same-repo CI-loop concern (Q2) and a scoped bot token are **not** needed. - **Registry is `twuni/docker-registry` (plain CNCF Distribution):** deployed by `registry-values.yml.j2`. It has **no native "keep last N" retention** — that capability is Harbor/GitLab-registry only. Today the template sets neither `persistence.deleteEnabled` (DELETE API returns 405) nor `garbageCollect.enabled` (no GC CronJob), so old images accumulate. Distribution can only delete by digest and bulk-`--delete-untagged`; bounded retention therefore requires an external prune over enumerable **tags** (Design §5). - **`newest-build` on Distribution:** Image Updater's `newest-build` strategy ranks tags by image build timestamp, read from each image's config blob; this works on plain Distribution (no Harbor API needed). ## Overview This document specifies how Themelio deploys freshly built service images without manual `kubectl`, and how it bounds registry growth. CI tags each build with an immutable `-` tag and pushes it to the `twuni/docker-registry` registry. ArgoCD Image Updater watches the registry with the `newest-build` strategy and writes the selected tag **directly onto the live ArgoCD Application** (`write-back-method: argocd`); an `ignoreDifferences` carve-out on the parent root-app lets `selfHeal` manage every manifest field from git **except** the image version, which Image Updater owns. **All environments auto-update.** A daily Kubernetes `CronJob` prunes the registry to the newest 5 per service and the chart's GC reclaims the freed blobs; the prune emits structured JSON log lines that a **Loki-backed Grafana dashboard** renders as the active + retained tags per image (no Prometheus/Pushgateway component). ## Goals and Non-Goals ### Goals - A CI-pushed build reaches its environment automatically, with no manual rollout. - Every manifest field except the image version is reproducible from git and self-healed. - Registry storage is bounded to the newest 5 tags per service, deep enough for a 4-build rollback. - Operators can see, in Grafana, the active tag and the 4 retained predecessors per image. ### Non-Goals - Git-reproducibility or `git revert` rollback **of the image version** (explicitly waived in the ADR; image rollback is a re-point via Image Updater/ArgoCD). - Migrating to a registry with native retention (Harbor/GitLab) — rejected as Option R3. - Per-service (single-image) CI builds — removed from ADR-0019 scope. - Any data migration — this is purely deploy-wiring. ## Design ### 1. Build & tag (CI) Re-enable the commented immutable tags in `themelio.config.yaml` so each build pushes both: - the mutable env tag (`dev`/`staging`/`prod`) — kept for convenience, **not** what drives deployment, and - the immutable `-` tag — the one Image Updater selects. `GIT_COMMIT` is already short (`git rev-parse --short HEAD` in `scripts/build.sh`) and tags are expanded by `envsubst`, so the tags are just uncommented as-is — no substring expression. No change to `scripts/build.sh`'s service loop. *(Implemented.)* ### 2. Pinning standardization No git-side `images:` block is needed. Image Updater (`write-back-method: argocd`) writes the resolved tag into the Application's `spec.source.kustomize.images`, and the kustomize image transformer **matches by image name** — so it overrides the tag in each `deployment.yaml`/`job-migrate.yaml` regardless of the hardcoded `:` value. The themelio overlays therefore keep their hardcoded tags as the git baseline; Image Updater overrides them at the Application level. `waldur-site-agent-cloud`'s staging overlay was converted from its `images:` block to the same hardcoded-tag patch (matching its dev/prod overlays), so **all services now standardize on hardcoded tags**. ### 3. Image Updater (direct-to-Application) The controller is installed in the `argocd` namespace from its upstream manifest by the `argocd-deploy` role (`devops/ansible/roles/argocd-deploy/tasks/main.yml`), alongside a `registries.conf` ConfigMap + basic-auth secret pointing at the pharos registries (reusing the `k8s` htpasswd user — pull-only). The annotations live **per-env in `gitops/apps/overlays//kustomization.yaml`**, not `apps/base`, because the registry path differs per env (`dev.registry…/test` vs `staging.registry…/themelio`). They patch the single multi-image `themelio` Application: ```yaml argocd-image-updater.argoproj.io/write-back-method: argocd argocd-image-updater.argoproj.io/image-list: >- web=/web, gitlab-root-actor=/gitlab-root-actor, k8s-root-actor=/k8s-root-actor argocd-image-updater.argoproj.io/.update-strategy: newest-build argocd-image-updater.argoproj.io/.allow-tags: regexp:^-[0-9a-f]+$ ``` `write-back-method: argocd` writes the resolved image into the live `themelio` Application's `spec.source.kustomize.images` — no git commit. All envs auto-update. *(Implemented for dev + staging.)* ### 4. selfHeal carve-out (Q4) The child Applications are rendered from git by the parent `gitops/-root-app.yaml` (`selfHeal: true`), which would scrub the override. Add `ignoreDifferences` on the **parent** for the child Application resource: ```yaml ignoreDifferences: - group: argoproj.io kind: Application jsonPointers: - /spec/source/kustomize/images ``` Result: git owns and self-heals every field except the image version. ### 5. Retention prune (Option R1) Enable on the registry (in `registry-values.yml.j2`): ```yaml persistence: deleteEnabled: true # required for the DELETE API used by the prune garbageCollect: enabled: true # chart GC CronJob deleteUntagged: true schedule: "0 1 * * *" ``` Add a daily `CronJob` in the `registry` namespace, deployed by the `image-registry` role (`devops/ansible/roles/image-registry/templates/registry-prune.yml.j2`) (ConfigMap script + CronJob). It uses **`regctl`** (not raw curl) with the registry's basic-auth creds (`registry-auth` secret, `ci` user). Per service it: 1. `regctl repo ls` → repositories; `regctl tag ls` → tags. 2. Skip the protected mutable tags (`dev`/`staging`/`prod`/`latest`); rank the rest newest-first by image build timestamp (`regctl image config`). 3. Keep the newest 5; `regctl tag rm` (DELETE manifest) for the rest. 4. Emit one structured **JSON log line per service** (`event=registry_prune`, `service`, `active`, `kept`, `pruned`, counts). The prune only untags; the chart GC CronJob reclaims the blobs (offline GC does not need `deleteEnabled`, but the prune's `tag rm` does — hence `image_registry_delete_enabled: true`). ### 6. Observability (Loki + Grafana) **No Prometheus/Pushgateway component.** The prune CronJob's JSON log lines are shipped to the existing **Loki** stack. A Grafana dashboard (`registry-image-tags.json`), provisioned as a ConfigMap via the `grafana_dashboard` sidecar label, renders: - a **table** of the latest prune per service (`active` tag + `kept` retained set) via `{namespace="registry", app="registry-prune"} | json`, and - a **logs** panel of recent prune events. ## Data Model **Tag scheme:** `/:-` where `` is `git rev-parse --short HEAD`; immutable. Mutable `` tags coexist but are protected from pruning. **Prune log line** (one per service per run, JSON): `event="registry_prune"`, `service`, `total`, `kept_count`, `pruned_count`, `active` (newest kept tag), `kept` (space-separated retained set), `pruned`, `protected`. This is the dashboard's data source — there is no Prometheus metric. ## API Design Registry access is via `regctl` over the Distribution HTTP API V2 (Basic auth): - `regctl repo ls` / `regctl tag ls` — enumerate repos and tags. - `regctl image config` — read the build timestamp for ranking. - `regctl tag rm` → `DELETE /v2//manifests/` — untag (requires `deleteEnabled: true`; returns `405` otherwise). ## Testing Strategy - **Tag build:** a pipeline run pushes `-`; confirm it appears in `tags/list`. - **Image Updater:** push a newer tag; confirm the child Application's `spec.source.kustomize.images` and the running Deployment update without a git commit. - **selfHeal:** force a parent sync; confirm the image override survives and a change to any other field is reverted to git. - **Prune dry-run:** run with deletes disabled/logged-only; assert it keeps exactly 5 per service and never selects a mutable tag. - **Guard:** assert a run with only `dev`/`staging`/`prod` present deletes nothing. - **Dashboard:** after a run, the Grafana Loki table shows the active tag + retained set per image. - **Acceptance:** registry size stops growing across N>5 builds; rollback to any of the last 4 tags succeeds via Image Updater. ## Deployment & Rollout Applied through the existing Ansible roles (no new bespoke pipeline): 1. `image-registry` role — `deleteEnabled` + `garbageCollect.*` values, and the prune `CronJob` (§5). 2. Re-enable the immutable tags in `themelio.config.yaml` (§1). 3. Overlays keep their hardcoded image tags; no kustomize `images:` block (§2). 4. `argocd-deploy` role installs Image Updater + `registries.conf`; the apps overlays carry the per-env annotations; the root-apps carry `ignoreDifferences` (§3–4). 5. `prometheus-grafana-deploy` role auto-loads the Loki dashboard JSON (§6). 6. Verify per the Testing Strategy. **All envs auto-update** once wired. **Rollback of this change:** revert the overlay/values commits; Image Updater and the prune CronJob are additive and can be scaled to zero / suspended without affecting running workloads. ## Open Questions 1. ~~**Promotion model (ADR Q3):** dev auto vs staging/prod gated?~~ **Resolved:** all environments auto-update to the newest matching `-` tag. 2. ~~**Pushgateway vs Loki-only?**~~ **Resolved:** Loki-only — the dashboard is built from LogQL over the prune job's JSON logs; no Prometheus metric or Pushgateway. 3. **Active-tag source:** the dashboard infers "active" from the newest registry tag in the prune log. If the *actually-deployed* image (read from the ArgoCD Application) is wanted instead, that needs a separate source. 4. **prod environment:** there is **no `prod` themelio overlay** in `gitops/manifests/themelio/overlays/` today, so dev + staging are wired but prod is not. Confirm whether prod is deployed via this gitops path before wiring it. 5. **Validation:** none of this has been run against a cluster — Image Updater's `registries.conf` credential format, the `ignoreDifferences` behaviour under the app-of-apps, and the dashboard's LogQL/transformations all need live verification. ## References - [ADR-0019: Container Image Tag Management for GitOps Deployments](/adr/0019-container-image-tag-management) - `themelio.config.yaml` — image build/tag definitions - `registry-values.yml.j2` — registry Helm values - [ArgoCD Image Updater](https://argocd-image-updater.readthedocs.io/) — strategies and write-back methods - [Distribution registry HTTP API V2](https://distribution.github.io/distribution/spec/api/)