#DD-0041 — GitLab and Waldur as Argo CD services

Related ADR: ADR-0041.

Status: Draft for review. Qualified on the authorised dev-ikons dev box; wider adoption, staging and production remain pending.

Date: 2026-10-14. Waldur pin updated to 8.1.4 on 2026-11-07.

#Scope and responsibilities

This is infrastructure configuration, stage 3 of decision 0040. There is no new business context, aggregate, command API, event or subscription.

#Applications and project

gitops/apps/base/{gitlab,waldur}.yaml install wrapper charts from gitops/manifests/<service>/chart. Each vendors its upstream chart by SHA-256 (GitLab 10.5.1, Waldur 8.1.4) and pins every image by digest. Both run on wave 3. They ignore the defaults Kubernetes fills into StatefulSet claim templates. Each reads gitops/manifests/<service>/base/values.yaml, and the environment's overlay in gitops/apps/overlays/<env> adds overlays/<env>/values.yaml: on dev boxes the edge names and port, one replica of each component, sizes and the seed claim. Argo CD's own values follow the same layout.

The services project allows only the two namespaces on a third cluster binding (https://kubernetes.default.svc.cluster.local:443), no cluster-scoped kinds and an explicit list of namespaced kinds. The controller's per-namespace Role adds delete only for hook kinds and create on pods/exec for Waldur's bootstrap. The project-writer admission fence covers services, and scripts/check_platform_gitops.py rejects a wider project.

GitLab migrations keep the upstream Job and ConfigMap lifecycle. Changing chart configuration gives Jobs a new name. A preceding Argo Sync hook retires obsolete, completed Jobs bearing this application's tracking ID, using a UID-checked TTL update. The chart supplies its desired Job name through the upstream helper; that Job, active, failed, foreign and hook-managed Jobs are preserved. This keeps configuration updates deployable with automatic pruning still disabled.

#Dependencies and generated credentials

The services use shared PostgreSQL 18.6 with separate databases and roles, while GitLab retains Redis 7.2 and Waldur uses shared RabbitMQ 4.1 with an isolated vhost. The providers use official images on ceph-block. A Sync hook at wave -3 creates each Secret once and never rewrites it; StatefulSets follow at wave -1. Volumes use fsGroupChangePolicy: OnRootMismatch, so later mounts keep RabbitMQ's Erlang cookie and PostgreSQL's data directory private. RabbitMQ also repairs a cookie left group-readable by an earlier mount. GitLab's LFS, artifacts, uploads, packages and backups use Ceph RGW buckets claimed per type; a hook composes their connection Secrets and the backup s3cfg.

#Sign-in and routes

clients.json registers pharos-gitlab and pharos-tools with {service_url:<name>} redirects. After the AAI stage, make deploy renders gitops/manifests/<service>/aai-client.yaml into a Secret in the service namespace, with the client and the AAI's public CA. It writes only into an existing namespace.

• GitLab uses OmniAuth OpenID Connect with PKCE, trusts the CA through customCAs, and builds its URLs with GITLAB_PORT=8443.
• Waldur 8.1 completes OIDC on the server, at /api-auth/keycloak/complete/. Its chart imports providers in the migration init container, which cannot be given the CA, and OIDC discovery verifies TLS. The chart therefore imports no provider. The PostSync bootstrap pipes the provider from waldur-oidc over stdin into waldur import_auth_social /dev/stdin in the API container, which trusts the CA, then creates the staff user and the Pharos customer.

The catalogue routes gitlab.pharos and waldur.pharos through the shared gateway. Waldur's /api/, /api-auth/, /static/ and /health-check/ go to the API and /admin/ is management-only. A route may set body.limit and switch off request buffering; GitLab uses an unlimited, unbuffered body for Git and LFS. The paired NetworkPolicies admit the gateway on each backend pod's own port (8080 for Waldur) and fail to render without a pod label.

Inside the cluster, CoreDNS resolves pharos.<domain> and its subdomains to the node, so services reach the issuer and each other through the edge on 8443. Isolation inside the namespaces is ingress-only.

#Waldur dev-box seed

Waldur's first installation runs 911 migrations, which took about 48 minutes on dev-ikons. The vendored chart has a documented three-line patch making the API, beat and worker Deployment deadlines configurable, with its original 2400-second default. Dev boxes allow 4800 seconds inside the 5400-second delivery gate, so a checkout without a seed can finish in one deploy. Both the original and patched archive hashes are recorded in the pins; rendered checks enforce the dev-box deadlines. With pharos.seed.claim set (only in the dev-box values):

1. Before Argo, make deploy creates the claim waldur-seed (1 GiB) and copies .local/metallo/seeds/waldur-<chart>-<digest>.sql.gz into it through a restricted transfer pod, verifying its SHA-256. The chart's pharos.seed helper and devbox/seeds.py derive the same name; the GitOps check enforces it.
2. The PostgreSQL Service and StatefulSet run in wave -1, alongside the restore hook, so it can connect on an empty cluster. The hook restores the seed only into a database without django_migrations, in one transaction. initdb then applies only what the seed lacks.
3. A PostSync hook at wave -1, before the bootstrap, captures a seed when the claim has none. It refuses unless every column named like a secret, password, token, key or credential is empty outside the identity provider table, and it leaves that table's data out. The dump has no owners or privileges.
4. After Argo, make deploy saves a newly captured seed back to .local/.

Restored data has no users or encrypted values, so each installation keeps its own generated keys. The compatibility token writer waits for successful provider PostSync completion, including seed capture and bootstrap, before adding its staff user and token.

#Interruption and recovery

• Superseded operations. An automated retry of an earlier revision keeps the Application's operation until its retries run out. While waiting for health, the controller ends an operation for another revision once the child names the delivered one; auto-sync then starts the delivered revision.
• Broken StatefulSet pods. With OrderedReady, a crash-looping pod is not replaced by a new revision. Delete the pod to apply the fix (forced rollback).
• Waldur without a seed. The first make deploy waits for cold migrations, then captures the seed and bootstraps Waldur. Deployment deadlines and the outer health gate remain bounded; an unhealthy installation still fails.
• Stale routes. Routes do not prune. Back up and delete an obsolete VirtualService under the restoring ownership state.

#Qualification and remaining boundaries

On dev-ikons:

• Full deploy: every stage and all 13 route probes pass, and a repeat run changes only the known Rook repository task.
• Fresh Waldur installation from the seed: after volumes, workloads and generated Secrets were deleted, make deploy completed in 13 minutes. The restore took 28 seconds and initdb 3 minutes.
• Sign-in: alice and bob sign in to GitLab and Waldur through the AAI in Chromium, which trusts only the AAI CA. Each service's API reports the signed-in user.
• GitLab: dx/common/scripts/qualify_gitlab.py passes. A maintainer pushes a 30 MiB LFS object and a developer restores its checksum. A developer cannot push to the protected default branch but can push a branch, and cannot delete the project. Outsider and anonymous clones are refused. The script's Git runs without credential helpers, and it revokes its root token afterwards.
• Isolation: a restricted pod in a new namespace cannot reach GitLab, Waldur, their databases, Redis or RabbitMQ directly. The public Waldur route works, and /admin/ is refused even with a forged forwarding header.
• Seed guard: on a scratch database restored from the seed, the guard passed with a fixture identity provider, whose data the dump left out. The live database, which has users, was refused.

Not exercised: a cold capture end to end (the seed came from a hand-taken dump of the same state), egress isolation, fresh-VM and reboot qualification, staging and production. The GitLab client is a new registration for any external federation.

Page source

Source snapshot · Revision f928834c8.