For AI agents: the complete documentation index is available at /llms.txt, the full documentation bundle is available at /llms-full.txt, and this page is available as Markdown at /adr/0017-shared-themelio-database-decoupled-migrations.md.

Themelio compatibility reference. This page describes the retained Themelio implementation or its historical design. Check the parallel-operation guide before applying it to the current platform.

ADR-0017: Shared Themelio Database with Decoupled Schema Migrations

Status: Proposed

Date: 2026-06-08

Deciders: TBD

Relates to:

Context

The Themelio platform runs a shared CloudNativePG (CNPG) PostgreSQL cluster in the databases namespace (see ADR-0016). We are introducing a new themelio database (owner themelio) on that cluster to hold durable job and deployment state — the persistent store decided in ADR-0012, replacing the previous Valkey/JSON-file approach.

This ADR settles two questions about that database before it is built: how it should be owned, and how its schema should be applied. The naive default — letting the single service that first needs the data own the schema, embed the migration code, and run migrations in-process on every boot — is simple but couples the database to one service and ties schema application to the application lifecycle. We want to decide deliberately rather than inherit that default.

Requirements

  1. The database must be consumable by more than one service over time, without forcing reads to proxy through a single owning service.
  2. Schema application must be correct regardless of how many application replicas run.
  3. A migration interrupted mid-run (OOM, node death, SIGKILL) must recover without manual operator intervention.
  4. The long-running serving process should not hold schema-altering (DDL) rights.
  5. Migration outcome must be observable and must gate the rollout — applications must never serve against an un-migrated schema.

Decision Drivers

  • Shared consumption — multiple services should be able to read/write the database without proxying through one owner.
  • Multi-replica safety — correctness must not depend on running exactly one replica.
  • Deploy-wide ordering — the schema must be current before new application pods serve traffic.
  • Observable failure surface — migration success/failure should be a discrete, inspectable object, not a line buried in application logs.
  • Least privilege / decoupled DDL — only the migrator should hold schema-altering rights.
  • Fit with existing conventions — reuse the shared CNPG cluster and the established ArgoCD PreSync hook-Job pattern.

Considered Options

Option A: In-process migrations, schema owned by the consuming service

Embed the schema and migration code inside the service that first needs the data, and apply the schema with an in-process migrate.latest() call on every application boot, relying on a database-level migration lock to serialize concurrent runners.

Pros:

  • Simplest possible wiring — one code path owns connection, schema, and serving.
  • No additional build artifact or deploy step.

Cons:

  • The schema is owned by, and only reachable through, one service; other services must proxy through it.
  • Not multi-replica safe: a typical migration lock makes a second concurrent runner throw rather than wait, producing crash-restart noise on rollout.
  • A migrator killed mid-run can leave a stuck lock requiring manual recovery.
  • The serving container carries DDL rights and migration code on its hot path.
  • Boot is coupled to a migration round-trip; readiness is not simply "can connect + serve."

Option B: Shared platform database + PreSync migration Job from a dedicated image (chosen)

Treat themelio as shared platform infrastructure, keep the schema and migration code in a dedicated migration component that ships its own image, and apply migrations once per deploy via an ArgoCD PreSync hook Job. Application services connect to the database at runtime over DATABASE_URL and carry no migration code.

Pros:

  • The database is shareable: schema ownership is not trapped inside a service.
  • Multi-replica safe by construction — exactly one migrator runs per deploy.
  • The Job clears any stale lock on start, so a killed migrator self-heals on the next run.
  • Migration is a discrete, observable object that blocks promotion on failure.
  • The serving container holds no DDL rights and no migration code; startup is fast.
  • Reuses the existing waldur PreSync hook-Job precedent.

Cons:

  • A new build artifact (image) and a new Job manifest to maintain.
  • The migration image must be kept version-aligned with the schema.
  • The local source-mode overlay needs a non-image migration path.
  • One more moving part in the deploy.

Decision

We will adopt Option B, as three coupled sub-decisions:

  1. Treat themelio as a shared platform database. We will model it as platform infrastructure that multiple services may connect to directly over DATABASE_URL, not as a store private to a single service. No second consumer connects at launch; this decision fixes ownership up front so that future consumers (for example, web reading job and deployment data directly) are not forced to proxy through one service later.

  2. Run schema migrations once per deploy via an ArgoCD PreSync hook Job, rather than in-process on application boot. The PreSync hook must complete successfully before the new application ReplicaSet rolls, giving a deploy-wide ordering guarantee — applications are never scheduled against an un-migrated schema — and a discrete, observable failure surface that blocks promotion of the new version on failure. This mirrors the existing gitops/manifests/waldur-site-agent-cloud/base/job-load-components.yaml hook-Job precedent. Because exactly one migrator runs per deploy, the Job can clear any stale migration lock on start, avoiding any manual-unlock failure mode.

  3. Own the schema and migration code in a dedicated migration component that ships its own dedicated image, outside any consuming service. The PreSync Job runs this image; the long-running services carry no migration code or DDL rights. Because the code is not service-private, this component is the natural single owner of the shared schema, and the database it migrates can be consumed by any service. This is the end goal. As a fast temporary solution, the PreSync Job will reuse an existing service image and invoke only its migrate entrypoint (the serving process runs the same image and never migrates on boot), deferring the build of a separate image; the implementation and the extraction trigger are specified in DD-0017 (see Open Questions below).

See also: DD-0017: Implementation details

Consequences

Positive:

  • Multi-replica safe by construction; running N replicas needs no migration-related changes.
  • No crash-restart noise on rollout — application pods just connect and serve.
  • A failed migration blocks the rollout cleanly; the previous version keeps serving.
  • Migration status is a discrete, re-runnable Job object, observable in kubectl and the Argo UI.
  • The serving container runs with least privilege — no DDL rights, no migration code.
  • Fast, simple application startup; readiness is "can connect + serve."
  • The schema is reusable by future services without proxying through one owner.

Negative:

  • A new Job manifest must be maintained.
  • Until the migration code is extracted into a dedicated component/image (the end goal), it remains co-located in the serving image — invoked as a separate entrypoint and never on boot, but not yet physically isolated.
  • The local source-mode overlay (which runs from a hostPath rather than a built image) needs an equivalent non-image migration path.
  • The deploy has one more moving part.

Neutral:

  • Services connect to the shared database at runtime via DATABASE_URL.
  • A dependency-free Knex config can be retained for ad-hoc local CLI runs.

Migration

No migration is required. The themelio database is new and starts empty; the initial migration establishes the first schema. There is no existing infrastructure or data to migrate. The concrete wiring steps for the migration component and PreSync Job live in DD-0017.

Open Questions

Q1: When do we extract the migration code into a dedicated component with its own image?

The end goal (decision #3) is a dedicated migration component shipping its own image. The fast temporary solution reuses an existing service image and runs only its migrate entrypoint. The trigger and steps for the extraction are tracked in DD-0017; this question stays open until that extraction is scheduled.

References

Page source

Source snapshot · Revision f928834c8.