For AI agents: the complete documentation index is available at /llms.txt, the full documentation bundle is available at /llms-full.txt, and this page is available as Markdown at /dd/0019-container-image-tag-management.md.

Themelio compatibility reference. This page describes the retained Themelio implementation or its historical design. Check the parallel-operation guide before applying it to the current platform.

DD-0019: Container Image Tag Management — Implementation

Related ADR: ADR-0019: Container Image Tag Management for GitOps Deployments

Status: Draft

Date: 2026-06-29

Authors: TBD

ADR-derived technical notes

These notes capture implementation detail surfaced while scoping ADR-0019, kept for the chosen direction (Option B direct-to-Application + Option R1 retention). Stale per-option notes (the rejected Options A/D git write-back, and the incorrect "GitLab Container Registry cleanup policy") have been removed.

  • Re-enabling immutable tags: the per-commit tags already existed and were commented out in themelio.config.yaml (commit e2b0ecaa): dev-${GIT_COMMIT}, staging-${GIT_COMMIT}, prod-${GIT_COMMIT}. GIT_COMMIT is already short — scripts/build.sh sets it via git rev-parse --short HEAD — and tags are expanded with envsubst, which does not support a bash substring (${GIT_COMMIT:0:8}). So re-enabling is simply uncommenting the existing lines; no substring expression.
  • Pinning standardization: all services converge on a hardcoded tag in deployment.yaml. No git-side kustomize images: block is needed — Image Updater's spec.source.kustomize.images override matches by image name and replaces the tag regardless. waldur-site-agent-cloud's staging overlay (which previously used an images: block) was converted to the same hardcoded-tag patch as its dev/prod overlays.
  • ArgoCD specifics: root apps are gitops/<env>-root-app.yaml with automated: { prune: true, selfHeal: true }, repoURL the same repo, path: gitops/apps/overlays/<env>, targetRevision: HEAD. The app-of-apps renders the child Applications from git, which is why selfHeal would otherwise scrub a direct image override (see Design §4).
  • No git write-back / no same-repo loop: because the chosen write-back is direct-to-Application, nothing is committed to git, so the same-repo CI-loop concern (Q2) and a scoped bot token are not needed.
  • Registry is twuni/docker-registry (plain CNCF Distribution): deployed by registry-values.yml.j2. It has no native "keep last N" retention — that capability is Harbor/GitLab-registry only. Today the template sets neither persistence.deleteEnabled (DELETE API returns 405) nor garbageCollect.enabled (no GC CronJob), so old images accumulate. Distribution can only delete by digest and bulk---delete-untagged; bounded retention therefore requires an external prune over enumerable tags (Design §5).
  • newest-build on Distribution: Image Updater's newest-build strategy ranks tags by image build timestamp, read from each image's config blob; this works on plain Distribution (no Harbor API needed).

Overview

This document specifies how Themelio deploys freshly built service images without manual kubectl, and how it bounds registry growth. CI tags each build with an immutable <env>-<shortsha> tag and pushes it to the twuni/docker-registry registry. ArgoCD Image Updater watches the registry with the newest-build strategy and writes the selected tag directly onto the live ArgoCD Application (write-back-method: argocd); an ignoreDifferences carve-out on the parent root-app lets selfHeal manage every manifest field from git except the image version, which Image Updater owns. All environments auto-update. A daily Kubernetes CronJob prunes the registry to the newest 5 per service and the chart's GC reclaims the freed blobs; the prune emits structured JSON log lines that a Loki-backed Grafana dashboard renders as the active + retained tags per image (no Prometheus/Pushgateway component).

Goals and Non-Goals

Goals

  • A CI-pushed build reaches its environment automatically, with no manual rollout.
  • Every manifest field except the image version is reproducible from git and self-healed.
  • Registry storage is bounded to the newest 5 tags per service, deep enough for a 4-build rollback.
  • Operators can see, in Grafana, the active tag and the 4 retained predecessors per image.

Non-Goals

  • Git-reproducibility or git revert rollback of the image version (explicitly waived in the ADR; image rollback is a re-point via Image Updater/ArgoCD).
  • Migrating to a registry with native retention (Harbor/GitLab) — rejected as Option R3.
  • Per-service (single-image) CI builds — removed from ADR-0019 scope.
  • Any data migration — this is purely deploy-wiring.

Design

1. Build & tag (CI)

Re-enable the commented immutable tags in themelio.config.yaml so each build pushes both:

  • the mutable env tag (dev/staging/prod) — kept for convenience, not what drives deployment, and
  • the immutable <env>-<shortsha> tag — the one Image Updater selects.

GIT_COMMIT is already short (git rev-parse --short HEAD in scripts/build.sh) and tags are expanded by envsubst, so the tags are just uncommented as-is — no substring expression. No change to scripts/build.sh's service loop. (Implemented.)

2. Pinning standardization

No git-side images: block is needed. Image Updater (write-back-method: argocd) writes the resolved tag into the Application's spec.source.kustomize.images, and the kustomize image transformer matches by image name — so it overrides the tag in each deployment.yaml/job-migrate.yaml regardless of the hardcoded :<env> value. The themelio overlays therefore keep their hardcoded tags as the git baseline; Image Updater overrides them at the Application level. waldur-site-agent-cloud's staging overlay was converted from its images: block to the same hardcoded-tag patch (matching its dev/prod overlays), so all services now standardize on hardcoded tags.

3. Image Updater (direct-to-Application)

The controller is installed in the argocd namespace from its upstream manifest by the argocd-deploy role (devops/ansible/roles/argocd-deploy/tasks/main.yml), alongside a registries.conf ConfigMap + basic-auth secret pointing at the pharos registries (reusing the k8s htpasswd user — pull-only).

The annotations live per-env in gitops/apps/overlays/<env>/kustomization.yaml, not apps/base, because the registry path differs per env (dev.registry…/test vs staging.registry…/themelio). They patch the single multi-image themelio Application:

argocd-image-updater.argoproj.io/write-back-method: argocd
argocd-image-updater.argoproj.io/image-list: >-
  web=<registry>/web, gitlab-root-actor=<registry>/gitlab-root-actor, k8s-root-actor=<registry>/k8s-root-actor
argocd-image-updater.argoproj.io/<svc>.update-strategy: newest-build
argocd-image-updater.argoproj.io/<svc>.allow-tags: regexp:^<env>-[0-9a-f]+$

write-back-method: argocd writes the resolved image into the live themelio Application's spec.source.kustomize.images — no git commit. All envs auto-update. (Implemented for dev + staging.)

4. selfHeal carve-out (Q4)

The child Applications are rendered from git by the parent gitops/<env>-root-app.yaml (selfHeal: true), which would scrub the override. Add ignoreDifferences on the parent for the child Application resource:

ignoreDifferences:
  - group: argoproj.io
    kind: Application
    jsonPointers:
      - /spec/source/kustomize/images

Result: git owns and self-heals every field except the image version.

5. Retention prune (Option R1)

Enable on the registry (in registry-values.yml.j2):

persistence:
  deleteEnabled: true        # required for the DELETE API used by the prune
garbageCollect:
  enabled: true              # chart GC CronJob
  deleteUntagged: true
  schedule: "0 1 * * *"

Add a daily CronJob in the registry namespace, deployed by the image-registry role (devops/ansible/roles/image-registry/templates/registry-prune.yml.j2) (ConfigMap script + CronJob). It uses regctl (not raw curl) with the registry's basic-auth creds (registry-auth secret, ci user). Per service it:

  1. regctl repo ls → repositories; regctl tag ls → tags.
  2. Skip the protected mutable tags (dev/staging/prod/latest); rank the rest newest-first by image build timestamp (regctl image config).
  3. Keep the newest 5; regctl tag rm (DELETE manifest) for the rest.
  4. Emit one structured JSON log line per service (event=registry_prune, service, active, kept, pruned, counts).

The prune only untags; the chart GC CronJob reclaims the blobs (offline GC does not need deleteEnabled, but the prune's tag rm does — hence image_registry_delete_enabled: true).

6. Observability (Loki + Grafana)

No Prometheus/Pushgateway component. The prune CronJob's JSON log lines are shipped to the existing Loki stack. A Grafana dashboard (registry-image-tags.json), provisioned as a ConfigMap via the grafana_dashboard sidecar label, renders:

  • a table of the latest prune per service (active tag + kept retained set) via {namespace="registry", app="registry-prune"} | json, and
  • a logs panel of recent prune events.

Data Model

Tag scheme: <registry>/<name>:<env>-<shortsha> where <shortsha> is git rev-parse --short HEAD; immutable. Mutable <env> tags coexist but are protected from pruning.

Prune log line (one per service per run, JSON): event="registry_prune", service, total, kept_count, pruned_count, active (newest kept tag), kept (space-separated retained set), pruned, protected. This is the dashboard's data source — there is no Prometheus metric.

API Design

Registry access is via regctl over the Distribution HTTP API V2 (Basic auth):

  • regctl repo ls / regctl tag ls — enumerate repos and tags.
  • regctl image config — read the build timestamp for ranking.
  • regctl tag rm → DELETE /v2/<name>/manifests/<digest> — untag (requires deleteEnabled: true; returns 405 otherwise).

Testing Strategy

  • Tag build: a pipeline run pushes <env>-<shortsha>; confirm it appears in tags/list.
  • Image Updater: push a newer tag; confirm the child Application's spec.source.kustomize.images and the running Deployment update without a git commit.
  • selfHeal: force a parent sync; confirm the image override survives and a change to any other field is reverted to git.
  • Prune dry-run: run with deletes disabled/logged-only; assert it keeps exactly 5 per service and never selects a mutable tag.
  • Guard: assert a run with only dev/staging/prod present deletes nothing.
  • Dashboard: after a run, the Grafana Loki table shows the active tag + retained set per image.
  • Acceptance: registry size stops growing across N>5 builds; rollback to any of the last 4 tags succeeds via Image Updater.

Deployment & Rollout

Applied through the existing Ansible roles (no new bespoke pipeline):

  1. image-registry role — deleteEnabled + garbageCollect.* values, and the prune CronJob (§5).
  2. Re-enable the immutable tags in themelio.config.yaml (§1).
  3. Overlays keep their hardcoded image tags; no kustomize images: block (§2).
  4. argocd-deploy role installs Image Updater + registries.conf; the apps overlays carry the per-env annotations; the root-apps carry ignoreDifferences (§3–4).
  5. prometheus-grafana-deploy role auto-loads the Loki dashboard JSON (§6).
  6. Verify per the Testing Strategy. All envs auto-update once wired.

Rollback of this change: revert the overlay/values commits; Image Updater and the prune CronJob are additive and can be scaled to zero / suspended without affecting running workloads.

Open Questions

  1. Promotion model (ADR Q3): dev auto vs staging/prod gated? Resolved: all environments auto-update to the newest matching <env>-<sha> tag.
  2. Pushgateway vs Loki-only? Resolved: Loki-only — the dashboard is built from LogQL over the prune job's JSON logs; no Prometheus metric or Pushgateway.
  3. Active-tag source: the dashboard infers "active" from the newest registry tag in the prune log. If the actually-deployed image (read from the ArgoCD Application) is wanted instead, that needs a separate source.
  4. prod environment: there is no prod themelio overlay in gitops/manifests/themelio/overlays/ today, so dev + staging are wired but prod is not. Confirm whether prod is deployed via this gitops path before wiring it.
  5. Validation: none of this has been run against a cluster — Image Updater's registries.conf credential format, the ignoreDifferences behaviour under the app-of-apps, and the dashboard's LogQL/transformations all need live verification.

References

Page source

Source snapshot · Revision f928834c8.