> For AI agents: the complete documentation index is available at /llms.txt, the full documentation bundle is available at /llms-full.txt. # ADR-0019: Container Image Tag Management for GitOps Deployments **Status:** Draft **Date:** 2026-06-26 **Deciders:** Athanasios Katsadas, TBD ## Context Themelio's service images (`web`, `gitlab-root-actor`, `k8s-root-actor`, `waldur-root-actor`, `mlflow-ui`, and others) are built and pushed by GitLab CI, and deployed by ArgoCD. ArgoCD automatically syncs the cluster based on the repo status. How images are tagged determines whether a freshly built image actually reaches the cluster, whether the running version is reproducible from git, and how a bad deploy is rolled back. This ADR exists to settle the image-tagging and deployment synchronization status without relying on manual steps. The question surfaced when we were researching how a db-migrations job would rerun on new changes. It is now tracked by the ops issue _Image tagging mechanism_, which additionally calls for bounded image retention in the registry (so per-build tags do not accumulate). ### Current State - Images are built from `themelio.config.yaml` via `scripts/build.sh `, invoked in the GitLab CI `deploy`/`build` stages (`.gitlab-ci.yml`). - Each environment pushes a **mutable, per-environment tag**: `dev`, `staging`, `latest`, `prod`. An immutable per-commit tag (`dev-${GIT_COMMIT}`, `staging-${GIT_COMMIT}`, `prod-${GIT_COMMIT}`) was previously also pushed but was **commented out** in `themelio.config.yaml` (commit `e2b0ecaa`), so only mutable tags are produced today. - ArgoCD tracks this same repository (`repoURL: https://gitlab.grnet.gr/themelio/themelio.git`, `path: gitops/...`, `targetRevision: HEAD`) ,i.e. application manifests and the gitops manifests live in **one repo**. - Image references are pinned **inconsistently**: the `themelio` overlays hardcode the tag directly in each `deployment.yaml` (e.g. `image: …/web:staging`, `imagePullPolicy: Always`), while `waldur-site-agent-cloud` pins via a kustomize `images:` block (`newName`/`newTag`). - `scripts/build.sh ` builds the services listed for that environment in `themelio.config.yaml`. The registry is a `twuni/docker-registry` (plain CNCF Distribution) deployment (`registry-values.yml.j2`); its Helm values set neither `persistence.deleteEnabled` (so manifest deletes are disabled) nor `garbageCollect.enabled`, so the registry has **no retention/garbage-collection policy** for old images. ### Current Problems - **ArgoCD reconciles git, not the registry.** With a mutable tag, pushing a new `:dev` image does not change anything in git, so ArgoCD detects no drift and never restarts the deployment. New image pushes are only applied after manual deployment rollout restart. - **The running version is not reproducible from git.** Because the manifest only ever says `:dev`/`:staging`, git cannot answer "which build is running in this environment?", and two replicas pulling `:dev` at different times can run different code. - **No clean rollback.** Reverting a deploy is not a \`git revert', it requires rebuilding or repushing a tag. - **Vanilla ArgoCD cannot auto-detect newer images by tag pattern.** Kustomize/ArgoCD resolve a fixed reference, selecting "the newest tag matching a wildcard" is not a built-in capability and requires ArgoCD Image Updater (Option B). CI write-back (Option A) avoids detection entirely by writing the exact tag at build time. - **Immutable per-build tags accumulate** in the registry unless a retention/garbage-collection policy is added. ### Requirements 1. A build pushed by CI must reach the target environment reliably, without manual `kubectl` intervention. 2. The image running in each environment must be reproducible from git state alone. 3. Rolling back to a previous build should be a git operation. 4. The mechanism must fit ArgoCD automated sync + `selfHeal` and the single-repo gitops layout. 5. The image-pinning mechanism should be consistent across all services. 6. The registry must not grow unbounded: old images are garbage-collected on a retention window (issue suggests keeping the last 5 per service) that is still deep enough to roll back. ## Decision Drivers - **Rollout reliability** — a build must deploy without manual steps. - **Provenance & reproducibility** — git is the source of truth for what runs. - **Rollback** — revert by reverting a commit. - **Operational simplicity** — minimize new in-cluster components and credentials. - **Readability** — image references should be intelligible to operators where possible. - **Fit with existing setup** — single-repo gitops, CI that already builds and pushes, ArgoCD auto-sync. - **Registry hygiene** — bounded retention, per-build tags must be garbage-collected, yet kept deep enough to roll back. ## Considered Options This decision splits into two largely separable concerns, presented in turn below. **Registry retention** (how old images are bounded on the current `twuni/docker-registry` plain-Distribution registry) is addressed first. **Image tagging & deployment sync** (how a built image is referenced and reaches the cluster) is left open for now — the tagging Options A–C are retained unchanged pending a later decision. > **Interdependency.** On plain Distribution the registry API lists _tags_ but cannot enumerate _untagged_ manifests, so retention can only prune images that are reachable as a distinct tag. The retention options below therefore assume immutable per-build tags; a mutable-only or digest-pinned tagging scheme (Option C, or Option B's `digest` strategy) leaves old images either unbounded or only bulk-deletable via `--delete-untagged`, with no "keep last N". ### Part 1: Registry retention #### Option R1: Scheduled tag prune (Kubernetes CronJob) + chart GC Enable deletes and GC on the current registry — set `persistence.deleteEnabled: true` and `garbageCollect.enabled: true` (with `deleteUntagged: true` and a `schedule`) in the `twuni/docker-registry` Helm values (`registry-values.yml.j2`) — then run the prune as a dedicated Kubernetes `CronJob` (once daily) in the `registry` namespace rather than in CI. On each run it lists tags via `GET /v2//tags/list`, keeps the newest N (the issue suggests 5) per service, and `DELETE`s the manifests of older per-build tags, with a guard that never deletes the mutable `dev`/`staging`/`prod` tags. It authenticates with the in-cluster registry credentials (the `registry-auth` secret). The prune selects _what_ to remove; the chart's GC CronJob then reclaims the blob storage. Both run in-cluster on independent schedules. **Observability.** Each run should emit metrics — tags kept / pruned / protected per service, and blobs and bytes reclaimed by GC — into the existing Prometheus + Grafana stack so a Grafana dashboard shows what every run changed over time. Because a daily CronJob is too short-lived to be scraped directly, the job pushes these metrics to a **Prometheus Pushgateway**, which Prometheus then scrapes. The human-readable per-run log lines (e.g. "pruned `dev-e2bcd14`, `dev-a17f902`; active tags: …") are additionally surfaced through the existing Loki stack for log-level inspection. To make the **retained tags themselves** visible (not just counts), the prune job also pushes one gauge series per still-present tag — e.g. `registry_image_tag{service, env, tag, position}`, where `position=0` is the active (newest) tag and `position=1..4` are the four retained predecessors, and the value carries the tag's build timestamp. Replacing the Pushgateway group on each run keeps the series set bounded to exactly the five tags that currently exist per service (stale SHAs drop off automatically). A Grafana **table panel** over `registry_image_tag` then shows, per image, the current active tag plus the previous four retained tags ordered by recency. The authoritative active tag may alternatively be read from the deployed ArgoCD Application image rather than inferred as the newest registry tag. **Pros:** - Stays on the existing registry; prune and GC both run in-cluster on schedules — no registry-host access, and no coupling to the CI pipeline. - Bounded storage with an explicit, auditable rollback window (depth = N). - The prune runs regardless of build activity and reuses the cluster's registry credentials, so CI needs no delete rights. **Cons:** - Custom prune logic to maintain (a CronJob image/script); the "never delete mutable tags" guard is safety-critical. - Requires immutable, enumerable per-build tags, so it couples to the tagging decision. - Deletes must first be enabled (`persistence.deleteEnabled` currently defaults to false). - Pruning is periodic (daily), so per-build tags created between runs persist until the next pass. - Time-series dashboarding needs a **Prometheus Pushgateway** (one new component), since a daily CronJob cannot be scraped directly. #### Option R2: Chart garbage-collection only Enable only `garbageCollect.enabled` + `deleteUntagged: true` and prune no tags. Distribution GC reclaims blobs no longer referenced by any manifest and (with `--delete-untagged`) manifests no longer referenced by a tag. **Pros:** - Trivial — a few Helm values, no CI code, no delete-guard to maintain. **Cons:** - Not real retention: distinct per-build tags accumulate forever (GC never touches a tagged manifest). Only useful if no immutable per-build tags are pushed — i.e. paired with mutable-only tagging, which forgoes `git revert` rollback. ### Part 2: Image tagging & deployment sync _Left open pending a later decision; the options below are unchanged._ ### Option A: Immutable per-commit tag + CI writes the tag back to git CI builds `web:-`, pushes it, then updates the gitops manifest (e.g. `kustomize edit set image` or `yq`) to reference the new tag and commits the change back to the repo. ArgoCD sees the git diff and syncs, rolling the Deployment. **Pros:** - Fully declarative; git is the source of truth and every deploy is a commit (provenance + `git revert` rollback). - Reliable rollout via the existing auto-sync, no new in-cluster component. - Reuses what CI already knows, meaning the exact tag it just pushed . **Cons:** - CI needs write access to the repo (a bot token). - Because gitops is the **same repo**, the bot commit must not trigger its own pipeline (skip-ci / path rules). - Requires standardizing every service on a single pinning mechanism (kustomize `images:` block). **Retention impact:** immutable per-build tags are distinct and enumerable via `tags/list`, so bounded retention is straightforward — Option R1 (keep last N per service) yields a rollback window of depth N. ### Option B: ArgoCD Image Updater Run the ArgoCD Image Updater controller (a long-running Deployment in the `argocd` namespace) with per-Application annotations describing an image allowlist, a tag-selection strategy, and a write-back method (git or direct-to-Application). **Chosen variant (see Decision): direct-to-Application.** CI builds an immutable per-commit tag `web:-` and pushes it to the registry. Image Updater watches the registry with the `newest-build` strategy, selects the newest matching tag, and writes the image override **directly onto the live ArgoCD Application** (`write-back-method: argocd`) — it does **not** commit to git. The git manifests keep only a static mutable placeholder tag (e.g. `:dev`) that is never what actually runs. Because the Applications are defined in git via `gitops/-root-app.yaml` with `automated.selfHeal: true`, the parent app-of-apps would otherwise reconcile the child Application back to its git-defined image and undo the override; this is resolved by adding `ignoreDifferences` on the parent for the child Application's `/spec/source/kustomize/images`, so git tracks and self-heals the whole manifest **except the image version**, which Image Updater owns. **Pros:** - Closes the registry-watch gap without a CI commit step. - No CI repo-write credentials and no same-repo CI-loop handling (Q2), since nothing is committed to git. - Maintains flow within argocd. **Cons:** - A new controller to operate and secure. - Polling latency. - **Git is not the source of truth:** the running build lives only in the Application's live spec; the repo holds an unused static tag, so "which build is in `staging`?" is not answerable from git. - **No `git revert` rollback:** rollback is performed through ArgoCD/Image Updater against a still-present registry tag, not git. **Retention impact:** the registry still accumulates an immutable `-` tag per build (needed for `newest-build` selection), and these are enumerable via `tags/list`, so bounded retention works via Option R1 (keep last N per service). ### Option C: Keep mutable tags, force a rollout from CI Keep `:dev`/`:staging` tags and have CI trigger a rollout after push — `kubectl rollout restart`, `argocd app actions run … restart`, or bumping a pod-template annotation. With `imagePullPolicy: Always`, the restart re-pulls the mutable tag. **Pros:** - Smallest change from today, no immutable tags or image-updater. **Cons:** - Not GitOps, git no longer reflects what is running; no provenance and no `git revert` rollback. - Requires cluster API access from CI. - "Which commit is in staging?" remains unanswerable; replicas can still diverge. - A stopgap rather than a model. **Retention impact:** no immutable per-build references and no `git revert` rollback, so there is nothing to retain — Option R2 (GC only) suffices to reclaim orphaned blobs. ## Comparison Matrix | Driver | A: per-commit tag + write-back | B: Image Updater | C: mutable + forced rollout | | ------------------------------------- | ------------------------------ | ------------------------------- | --------------------------- | | Reliable rollout | ✅ | ✅ | ✅ | | Git is source of truth / reproducible | ✅ | ✅ (git write-back) / ❌ (direct) | ❌ | | Rollback via `git revert` | ✅ | ✅ (git write-back) | ❌ | | New in-cluster component / creds | none | controller + registry creds | cluster access from CI | | Human-readable reference | ✅ (`env-sha`) | ✅ | ✅ | | CI repo-write needed | yes | only for git write-back | no | ## Decision We will adopt **Option B (ArgoCD Image Updater) with the direct-to-Application write-back method**. CI builds an immutable per-commit image tag `-` and pushes it to the registry; Image Updater, using the `newest-build` strategy, detects the newest tag and reconciles it **directly onto the live ArgoCD Applications**, without committing to git. Git remains the source of truth for the whole manifest **except the image version**. ArgoCD reconciles and self-heals everything from git as normal; only the image field is owned by Image Updater. This boundary is drawn with `ignoreDifferences` on the parent root-app for the child Application's `/spec/source/kustomize/images`, so the parent's `selfHeal` does not scrub the override. We explicitly accept the trade-off that this carve-out implies: the **image version** alone is not reproducible from git (it lives in the child Application's live spec, not the repo) and is **not `git revert`-able** — rollback of the image is performed via Image Updater/ArgoCD against a registry tag that is still present. Everything else in the manifest stays git-tracked and self-healed. This knowingly narrows Requirements 2 and 3 to "all manifest fields except the image version". For registry retention we will pair this with **Option R1** (scheduled CronJob tag prune + chart GC): the per-commit `-` tags remain enumerable in the registry, and the retention window sets how far back a roll-forward/rollback to an older tag can reach. **See also:** [DD-0019: Implementation details](/dd/0019-container-image-tag-management.md) ## Consequences These follow from the chosen option — Option B (Image Updater) with direct-to-Application write-back, immutable `-` tags, the `ignoreDifferences` image carve-out, and Option R1 retention. **Positive:** - Builds roll out reliably through Image Updater + ArgoCD auto-sync, with no manual `kubectl`, and no CI repo-write credentials or same-repo CI loop (Q2) since nothing is committed to git. - Every manifest field **except the image version** stays reproducible from git and self-healed. - Registry storage stays bounded via the R1 prune + chart GC, with a rollback window of depth N. **Negative:** - The **image version** is not reproducible from git and not `git revert`-able; image rollback is a re-point via Image Updater/ArgoCD to an older registry tag, only as far back as the R1 retention window allows. - A new in-cluster controller (Image Updater) and a registry pull credential must be operated and secured. - The setup depends on an `ignoreDifferences` carve-out on the parent root-app; if it is mis-scoped, `selfHeal` either scrubs the image override or stops healing real drift on other fields. - Image Updater polls the registry, so rollout has detection latency rather than being push-driven. **Neutral:** - All services would converge on one image-pinning mechanism — a **hardcoded tag** in `deployment.yaml` — replacing the current mix of hardcoded tags and `images:` blocks. No git-side `images:` block is needed: Image Updater overrides the tag by image name at the Application level. - The per-environment mutable tags may still be pushed for convenience but are not what drives deployment. - An immutable `-` tag is (re-)enabled in `themelio.config.yaml`. ### Migration Migration of the deploy wiring is required once an option is chosen: re-enable an immutable tag in `themelio.config.yaml`, keep each overlay's hardcoded image tag (no kustomize `images:` block needed), and add the write-back (or controller) path. There is **no data migration** — this changes only how images are referenced and rolled out. Concrete steps belong in DD-0019. ## Open Questions ### Q1: Tag scheme Per-commit tag (`-`) or semver/release tags? Trades readability against immutability strength. ### Q2: Same-repo write-back loop How does an automated manifest commit avoid triggering its own CI pipeline, given gitops and code share one repository? ### Q3: Promotion model Does `dev` auto-deploy on merge while `staging`/`prod` require a manual or gated promotion, and how is that expressed? ### Q4: selfHeal interaction and rollback ~~How do automated `selfHeal` and the rollback story interact with the chosen write-back path?~~ **Resolved:** with direct-to-Application write-back, the parent app-of-apps (`selfHeal: true`) would scrub Image Updater's override on the child Application. We add `ignoreDifferences` on the parent for the child Application's `/spec/source/kustomize/images`, so `selfHeal` reconciles every field from git **except the image version**, which Image Updater owns. Image rollback is therefore not a `git revert` but a re-point via Image Updater/ArgoCD to an older registry tag (which must still be within the Option R1 retention window). ### Q5: Registry retention / garbage collection Keep the last N images per service (the issue suggests 5)? Enforce via the registry's own retention/cleanup policy or a CI prune step? The window must be at least as deep as the desired rollback range, since `git revert` only restores a reference whose image still exists in the registry. ## References - GitLab issue _ops: Image tagging mechanism_ (Athanasios Katsadas) — the ops ticket tracking this decision. - `themelio.config.yaml` — image build/tag definitions - `.gitlab-ci.yml` — CI pipeline stages - ArgoCD root applications: `gitops/-root-app.yaml` - [ArgoCD Image Updater](https://argocd-image-updater.readthedocs.io/) - [Kustomize image transformer (`images:`)](https://kubectl.docs.kubernetes.io/references/kustomize/kustomization/images/)
Page source Source snapshot · Revision `f928834c8`.