#ADR-0019: Container Image Tag Management for GitOps Deployments

Status: Draft

Date: 2026-06-26

Deciders: Athanasios Katsadas, TBD

#Context

Themelio's service images (web, gitlab-root-actor, k8s-root-actor, waldur-root-actor, mlflow-ui, and others) are built and pushed by GitLab CI, and deployed by ArgoCD. ArgoCD automatically syncs the cluster based on the repo status. How images are tagged determines whether a freshly built image actually reaches the cluster, whether the running version is reproducible from git, and how a bad deploy is rolled back. This ADR exists to settle the image-tagging and deployment synchronization status without relying on manual steps. The question surfaced when we were researching how a db-migrations job would rerun on new changes. It is now tracked by the ops issue Image tagging mechanism, which additionally calls for bounded image retention in the registry (so per-build tags do not accumulate).

#Current State

• Images are built from themelio.config.yaml via scripts/build.sh <env>, invoked in the GitLab CI deploy/build stages (.gitlab-ci.yml).
• Each environment pushes a mutable, per-environment tag: dev, staging, latest, prod. An immutable per-commit tag (dev-${GIT_COMMIT}, staging-${GIT_COMMIT}, prod-${GIT_COMMIT}) was previously also pushed but was commented out in themelio.config.yaml (commit e2b0ecaa), so only mutable tags are produced today.
• ArgoCD tracks this same repository (repoURL: https://gitlab.grnet.gr/themelio/themelio.git, path: gitops/..., targetRevision: HEAD) ,i.e. application manifests and the gitops manifests live in one repo.
• Image references are pinned inconsistently: the themelio overlays hardcode the tag directly in each deployment.yaml (e.g. image: …/web:staging, imagePullPolicy: Always), while waldur-site-agent-cloud pins via a kustomize images: block (newName/newTag).
• scripts/build.sh <environment> builds the services listed for that environment in themelio.config.yaml. The registry is a twuni/docker-registry (plain CNCF Distribution) deployment (registry-values.yml.j2); its Helm values set neither persistence.deleteEnabled (so manifest deletes are disabled) nor garbageCollect.enabled, so the registry has no retention/garbage-collection policy for old images.

#Current Problems

• ArgoCD reconciles git, not the registry. With a mutable tag, pushing a new :dev image does not change anything in git, so ArgoCD detects no drift and never restarts the deployment. New image pushes are only applied after manual deployment rollout restart.
• The running version is not reproducible from git. Because the manifest only ever says :dev/:staging, git cannot answer "which build is running in this environment?", and two replicas pulling :dev at different times can run different code.
• No clean rollback. Reverting a deploy is not a `git revert', it requires rebuilding or repushing a tag.
• Vanilla ArgoCD cannot auto-detect newer images by tag pattern. Kustomize/ArgoCD resolve a fixed reference, selecting "the newest tag matching a wildcard" is not a built-in capability and requires ArgoCD Image Updater (Option B). CI write-back (Option A) avoids detection entirely by writing the exact tag at build time.
• Immutable per-build tags accumulate in the registry unless a retention/garbage-collection policy is added.

#Requirements

1. A build pushed by CI must reach the target environment reliably, without manual kubectl intervention.
2. The image running in each environment must be reproducible from git state alone.
3. Rolling back to a previous build should be a git operation.
4. The mechanism must fit ArgoCD automated sync + selfHeal and the single-repo gitops layout.
5. The image-pinning mechanism should be consistent across all services.
6. The registry must not grow unbounded: old images are garbage-collected on a retention window (issue suggests keeping the last 5 per service) that is still deep enough to roll back.

#Decision Drivers

• Rollout reliability — a build must deploy without manual steps.
• Provenance & reproducibility — git is the source of truth for what runs.
• Rollback — revert by reverting a commit.
• Operational simplicity — minimize new in-cluster components and credentials.
• Readability — image references should be intelligible to operators where possible.
• Fit with existing setup — single-repo gitops, CI that already builds and pushes, ArgoCD auto-sync.
• Registry hygiene — bounded retention, per-build tags must be garbage-collected, yet kept deep enough to roll back.

#Considered Options

This decision splits into two largely separable concerns, presented in turn below. Registry retention (how old images are bounded on the current twuni/docker-registry plain-Distribution registry) is addressed first. Image tagging & deployment sync (how a built image is referenced and reaches the cluster) is left open for now — the tagging Options A–C are retained unchanged pending a later decision.

Interdependency. On plain Distribution the registry API lists tags but cannot enumerate untagged manifests, so retention can only prune images that are reachable as a distinct tag. The retention options below therefore assume immutable per-build tags; a mutable-only or digest-pinned tagging scheme (Option C, or Option B's digest strategy) leaves old images either unbounded or only bulk-deletable via --delete-untagged, with no "keep last N".

#Part 1: Registry retention

#Option R1: Scheduled tag prune (Kubernetes CronJob) + chart GC

Enable deletes and GC on the current registry — set persistence.deleteEnabled: true and garbageCollect.enabled: true (with deleteUntagged: true and a schedule) in the twuni/docker-registry Helm values (registry-values.yml.j2) — then run the prune as a dedicated Kubernetes CronJob (once daily) in the registry namespace rather than in CI. On each run it lists tags via GET /v2/<name>/tags/list, keeps the newest N (the issue suggests 5) per service, and DELETEs the manifests of older per-build tags, with a guard that never deletes the mutable dev/staging/prod tags. It authenticates with the in-cluster registry credentials (the registry-auth secret). The prune selects what to remove; the chart's GC CronJob then reclaims the blob storage. Both run in-cluster on independent schedules.

Observability. Each run should emit metrics — tags kept / pruned / protected per service, and blobs and bytes reclaimed by GC — into the existing Prometheus + Grafana stack so a Grafana dashboard shows what every run changed over time. Because a daily CronJob is too short-lived to be scraped directly, the job pushes these metrics to a Prometheus Pushgateway, which Prometheus then scrapes. The human-readable per-run log lines (e.g. "pruned dev-e2bcd14, dev-a17f902; active tags: …") are additionally surfaced through the existing Loki stack for log-level inspection.

To make the retained tags themselves visible (not just counts), the prune job also pushes one gauge series per still-present tag — e.g. registry_image_tag{service, env, tag, position}, where position=0 is the active (newest) tag and position=1..4 are the four retained predecessors, and the value carries the tag's build timestamp. Replacing the Pushgateway group on each run keeps the series set bounded to exactly the five tags that currently exist per service (stale SHAs drop off automatically). A Grafana table panel over registry_image_tag then shows, per image, the current active tag plus the previous four retained tags ordered by recency. The authoritative active tag may alternatively be read from the deployed ArgoCD Application image rather than inferred as the newest registry tag.

Pros:

• Stays on the existing registry; prune and GC both run in-cluster on schedules — no registry-host access, and no coupling to the CI pipeline.
• Bounded storage with an explicit, auditable rollback window (depth = N).
• The prune runs regardless of build activity and reuses the cluster's registry credentials, so CI needs no delete rights.

Cons:

• Custom prune logic to maintain (a CronJob image/script); the "never delete mutable tags" guard is safety-critical.
• Requires immutable, enumerable per-build tags, so it couples to the tagging decision.
• Deletes must first be enabled (persistence.deleteEnabled currently defaults to false).
• Pruning is periodic (daily), so per-build tags created between runs persist until the next pass.
• Time-series dashboarding needs a Prometheus Pushgateway (one new component), since a daily CronJob cannot be scraped directly.

#Option R2: Chart garbage-collection only

Enable only garbageCollect.enabled + deleteUntagged: true and prune no tags. Distribution GC reclaims blobs no longer referenced by any manifest and (with --delete-untagged) manifests no longer referenced by a tag.

Pros:

• Trivial — a few Helm values, no CI code, no delete-guard to maintain.

Cons:

• Not real retention: distinct per-build tags accumulate forever (GC never touches a tagged manifest). Only useful if no immutable per-build tags are pushed — i.e. paired with mutable-only tagging, which forgoes git revert rollback.

#Part 2: Image tagging & deployment sync

Left open pending a later decision; the options below are unchanged.

#Option A: Immutable per-commit tag + CI writes the tag back to git

CI builds web:<env>-<shortsha>, pushes it, then updates the gitops manifest (e.g. kustomize edit set image or yq) to reference the new tag and commits the change back to the repo. ArgoCD sees the git diff and syncs, rolling the Deployment.

Pros:

• Fully declarative; git is the source of truth and every deploy is a commit (provenance + git revert rollback).
• Reliable rollout via the existing auto-sync, no new in-cluster component.
• Reuses what CI already knows, meaning the exact tag it just pushed .

Cons:

• CI needs write access to the repo (a bot token).
• Because gitops is the same repo, the bot commit must not trigger its own pipeline (skip-ci / path rules).
• Requires standardizing every service on a single pinning mechanism (kustomize images: block).

Retention impact: immutable per-build tags are distinct and enumerable via tags/list, so bounded retention is straightforward — Option R1 (keep last N per service) yields a rollback window of depth N.

#Option B: ArgoCD Image Updater

Run the ArgoCD Image Updater controller (a long-running Deployment in the argocd namespace) with per-Application annotations describing an image allowlist, a tag-selection strategy, and a write-back method (git or direct-to-Application).

Chosen variant (see Decision): direct-to-Application. CI builds an immutable per-commit tag web:<env>-<shortsha> and pushes it to the registry. Image Updater watches the registry with the newest-build strategy, selects the newest matching tag, and writes the image override directly onto the live ArgoCD Application (write-back-method: argocd) — it does not commit to git. The git manifests keep only a static mutable placeholder tag (e.g. :dev) that is never what actually runs. Because the Applications are defined in git via gitops/<env>-root-app.yaml with automated.selfHeal: true, the parent app-of-apps would otherwise reconcile the child Application back to its git-defined image and undo the override; this is resolved by adding ignoreDifferences on the parent for the child Application's /spec/source/kustomize/images, so git tracks and self-heals the whole manifest except the image version, which Image Updater owns.

Pros:

• Closes the registry-watch gap without a CI commit step.
• No CI repo-write credentials and no same-repo CI-loop handling (Q2), since nothing is committed to git.
• Maintains flow within argocd.

Cons:

• A new controller to operate and secure.
• Polling latency.
• Git is not the source of truth: the running build lives only in the Application's live spec; the repo holds an unused static tag, so "which build is in staging?" is not answerable from git.
• No git revert rollback: rollback is performed through ArgoCD/Image Updater against a still-present registry tag, not git.

Retention impact: the registry still accumulates an immutable <env>-<shortsha> tag per build (needed for newest-build selection), and these are enumerable via tags/list, so bounded retention works via Option R1 (keep last N per service).

#Option C: Keep mutable tags, force a rollout from CI

Keep :dev/:staging tags and have CI trigger a rollout after push — kubectl rollout restart, argocd app actions run … restart, or bumping a pod-template annotation. With imagePullPolicy: Always, the restart re-pulls the mutable tag.

Pros:

• Smallest change from today, no immutable tags or image-updater.

Cons:

• Not GitOps, git no longer reflects what is running; no provenance and no git revert rollback.
• Requires cluster API access from CI.
• "Which commit is in staging?" remains unanswerable; replicas can still diverge.
• A stopgap rather than a model.

Retention impact: no immutable per-build references and no git revert rollback, so there is nothing to retain — Option R2 (GC only) suffices to reclaim orphaned blobs.

#Comparison Matrix

#Decision

We will adopt Option B (ArgoCD Image Updater) with the direct-to-Application write-back method. CI builds an immutable per-commit image tag <env>-<shortsha> and pushes it to the registry; Image Updater, using the newest-build strategy, detects the newest tag and reconciles it directly onto the live ArgoCD Applications, without committing to git.

Git remains the source of truth for the whole manifest except the image version. ArgoCD reconciles and self-heals everything from git as normal; only the image field is owned by Image Updater. This boundary is drawn with ignoreDifferences on the parent root-app for the child Application's /spec/source/kustomize/images, so the parent's selfHeal does not scrub the override.

We explicitly accept the trade-off that this carve-out implies: the image version alone is not reproducible from git (it lives in the child Application's live spec, not the repo) and is not git revert-able — rollback of the image is performed via Image Updater/ArgoCD against a registry tag that is still present. Everything else in the manifest stays git-tracked and self-healed. This knowingly narrows Requirements 2 and 3 to "all manifest fields except the image version".

For registry retention we will pair this with Option R1 (scheduled CronJob tag prune + chart GC): the per-commit <env>-<shortsha> tags remain enumerable in the registry, and the retention window sets how far back a roll-forward/rollback to an older tag can reach.

See also: DD-0019: Implementation details

#Consequences

These follow from the chosen option — Option B (Image Updater) with direct-to-Application write-back, immutable <env>-<shortsha> tags, the ignoreDifferences image carve-out, and Option R1 retention.

Positive:

• Builds roll out reliably through Image Updater + ArgoCD auto-sync, with no manual kubectl, and no CI repo-write credentials or same-repo CI loop (Q2) since nothing is committed to git.
• Every manifest field except the image version stays reproducible from git and self-healed.
• Registry storage stays bounded via the R1 prune + chart GC, with a rollback window of depth N.

Negative:

• The image version is not reproducible from git and not git revert-able; image rollback is a re-point via Image Updater/ArgoCD to an older registry tag, only as far back as the R1 retention window allows.
• A new in-cluster controller (Image Updater) and a registry pull credential must be operated and secured.
• The setup depends on an ignoreDifferences carve-out on the parent root-app; if it is mis-scoped, selfHeal either scrubs the image override or stops healing real drift on other fields.
• Image Updater polls the registry, so rollout has detection latency rather than being push-driven.

Neutral:

• All services would converge on one image-pinning mechanism — a hardcoded tag in deployment.yaml — replacing the current mix of hardcoded tags and images: blocks. No git-side images: block is needed: Image Updater overrides the tag by image name at the Application level.
• The per-environment mutable tags may still be pushed for convenience but are not what drives deployment.
• An immutable <env>-<shortsha> tag is (re-)enabled in themelio.config.yaml.

#Migration

Migration of the deploy wiring is required once an option is chosen: re-enable an immutable tag in themelio.config.yaml, keep each overlay's hardcoded image tag (no kustomize images: block needed), and add the write-back (or controller) path. There is no data migration — this changes only how images are referenced and rolled out. Concrete steps belong in DD-0019.

#Open Questions

#Q1: Tag scheme

Per-commit tag (<env>-<shortsha>) or semver/release tags? Trades readability against immutability strength.

#Q2: Same-repo write-back loop

How does an automated manifest commit avoid triggering its own CI pipeline, given gitops and code share one repository?

#Q3: Promotion model

Does dev auto-deploy on merge while staging/prod require a manual or gated promotion, and how is that expressed?

#Q4: selfHeal interaction and rollback

How do automated selfHeal and the rollback story interact with the chosen write-back path? Resolved: with direct-to-Application write-back, the parent app-of-apps (selfHeal: true) would scrub Image Updater's override on the child Application. We add ignoreDifferences on the parent for the child Application's /spec/source/kustomize/images, so selfHeal reconciles every field from git except the image version, which Image Updater owns. Image rollback is therefore not a git revert but a re-point via Image Updater/ArgoCD to an older registry tag (which must still be within the Option R1 retention window).

#Q5: Registry retention / garbage collection

Keep the last N images per service (the issue suggests 5)? Enforce via the registry's own retention/cleanup policy or a CI prune step? The window must be at least as deep as the desired rollback range, since git revert only restores a reference whose image still exists in the registry.

#References

• GitLab issue ops: Image tagging mechanism (Athanasios Katsadas) — the ops ticket tracking this decision.
• themelio.config.yaml — image build/tag definitions
• .gitlab-ci.yml — CI pipeline stages
• ArgoCD root applications: gitops/<env>-root-app.yaml
• ArgoCD Image Updater
• Kustomize image transformer (images:)

Page source

Source snapshot · Revision f928834c8.