--- title: 'DD-0041: GitLab and Waldur as Argo CD Services' --- # DD-0041 — GitLab and Waldur as Argo CD services **Related ADR:** [ADR-0041](../decisions/0041-gitlab-waldur-services.md). **Status:** Draft for review. Qualified on the authorised `dev-ikons` dev box; wider adoption, staging and production remain pending. **Date:** 2026-10-14. Waldur pin updated to 8.1.4 on 2026-11-07. ## Scope and responsibilities This is infrastructure configuration, stage 3 of [decision 0040](../decisions/0040-platform-layout.md). There is no new business context, aggregate, command API, event or subscription. | Owner | Responsibility | |---|---| | Ansible bootstrap | `gitlab` and `waldur` namespaces and Pod Security labels, the `services` AppProject, its cluster binding and the controller's Roles, route RBAC and the in-cluster DNS override | | Argo CD | Each service's chart, its databases, broker or cache, generated Secrets and bootstrap hooks | | `make deploy` | AAI client Secrets, the Waldur seed claim and file, and the delivered revision | ## Applications and project `gitops/apps/base/{gitlab,waldur}.yaml` install wrapper charts from `gitops/manifests//chart`. Each vendors its upstream chart by SHA-256 (GitLab 10.5.1, Waldur 8.1.4) and pins every image by digest. Both run on wave 3. They ignore the defaults Kubernetes fills into StatefulSet claim templates. Each reads `gitops/manifests//base/values.yaml`, and the environment's overlay in `gitops/apps/overlays/` adds `overlays//values.yaml`: on dev boxes the edge names and port, one replica of each component, sizes and the seed claim. Argo CD's own values follow the same layout. The `services` project allows only the two namespaces on a third cluster binding (`https://kubernetes.default.svc.cluster.local:443`), no cluster-scoped kinds and an explicit list of namespaced kinds. The controller's per-namespace Role adds `delete` only for hook kinds and `create` on `pods/exec` for Waldur's bootstrap. The project-writer admission fence covers `services`, and `scripts/check_platform_gitops.py` rejects a wider project. GitLab migrations keep the upstream Job and ConfigMap lifecycle. Changing chart configuration gives Jobs a new name. A preceding Argo Sync hook retires obsolete, completed Jobs bearing this application's tracking ID, using a UID-checked TTL update. The chart supplies its desired Job name through the upstream helper; that Job, active, failed, foreign and hook-managed Jobs are preserved. This keeps configuration updates deployable with automatic pruning still disabled. ## Dependencies and generated credentials The services use shared PostgreSQL 18.6 with separate databases and roles, while GitLab retains Redis 7.2 and Waldur uses shared RabbitMQ 4.1 with an isolated vhost. The providers use official images on `ceph-block`. A Sync hook at wave -3 creates each Secret once and never rewrites it; StatefulSets follow at wave -1. Volumes use `fsGroupChangePolicy: OnRootMismatch`, so later mounts keep RabbitMQ's Erlang cookie and PostgreSQL's data directory private. RabbitMQ also repairs a cookie left group-readable by an earlier mount. GitLab's LFS, artifacts, uploads, packages and backups use Ceph RGW buckets claimed per type; a hook composes their connection Secrets and the backup `s3cfg`. ## Sign-in and routes `clients.json` registers `pharos-gitlab` and `pharos-tools` with `{service_url:}` redirects. After the AAI stage, `make deploy` renders `gitops/manifests//aai-client.yaml` into a Secret in the service namespace, with the client and the AAI's public CA. It writes only into an existing namespace. - **GitLab** uses OmniAuth OpenID Connect with PKCE, trusts the CA through `customCAs`, and builds its URLs with `GITLAB_PORT=8443`. - **Waldur 8.1** completes OIDC on the server, at `/api-auth/keycloak/complete/`. Its chart imports providers in the migration init container, which cannot be given the CA, and OIDC discovery verifies TLS. The chart therefore imports no provider. The PostSync bootstrap pipes the provider from `waldur-oidc` over stdin into `waldur import_auth_social /dev/stdin` in the API container, which trusts the CA, then creates the staff user and the `Pharos` customer. The catalogue routes `gitlab.pharos` and `waldur.pharos` through the shared gateway. Waldur's `/api/`, `/api-auth/`, `/static/` and `/health-check/` go to the API and `/admin/` is management-only. A route may set `body.limit` and switch off request buffering; GitLab uses an unlimited, unbuffered body for Git and LFS. The paired NetworkPolicies admit the gateway on each backend pod's own port (8080 for Waldur) and fail to render without a pod label. Inside the cluster, CoreDNS resolves `pharos.` and its subdomains to the node, so services reach the issuer and each other through the edge on 8443. Isolation inside the namespaces is ingress-only. ## Waldur dev-box seed Waldur's first installation runs 911 migrations, which took about 48 minutes on `dev-ikons`. The vendored chart has a documented three-line patch making the API, beat and worker Deployment deadlines configurable, with its original 2400-second default. Dev boxes allow 4800 seconds inside the 5400-second delivery gate, so a checkout without a seed can finish in one deploy. Both the original and patched archive hashes are recorded in the pins; rendered checks enforce the dev-box deadlines. With `pharos.seed.claim` set (only in the dev-box values): 1. Before Argo, `make deploy` creates the claim `waldur-seed` (1 GiB) and copies `.local/metallo/seeds/waldur--.sql.gz` into it through a restricted transfer pod, verifying its SHA-256. The chart's `pharos.seed` helper and `devbox/seeds.py` derive the same name; the GitOps check enforces it. 2. The PostgreSQL Service and StatefulSet run in wave -1, alongside the restore hook, so it can connect on an empty cluster. The hook restores the seed only into a database without `django_migrations`, in one transaction. `initdb` then applies only what the seed lacks. 3. A PostSync hook at wave -1, before the bootstrap, captures a seed when the claim has none. It refuses unless every column named like a secret, password, token, key or credential is empty outside the identity provider table, and it leaves that table's data out. The dump has no owners or privileges. 4. After Argo, `make deploy` saves a newly captured seed back to `.local/`. Restored data has no users or encrypted values, so each installation keeps its own generated keys. The compatibility token writer waits for successful provider PostSync completion, including seed capture and bootstrap, before adding its staff user and token. ## Interruption and recovery - **Superseded operations.** An automated retry of an earlier revision keeps the Application's operation until its retries run out. While waiting for health, the controller ends an operation for another revision once the child names the delivered one; auto-sync then starts the delivered revision. - **Broken StatefulSet pods.** With `OrderedReady`, a crash-looping pod is not replaced by a new revision. Delete the pod to apply the fix (forced rollback). - **Waldur without a seed.** The first `make deploy` waits for cold migrations, then captures the seed and bootstraps Waldur. Deployment deadlines and the outer health gate remain bounded; an unhealthy installation still fails. - **Stale routes.** Routes do not prune. Back up and delete an obsolete VirtualService under the `restoring` ownership state. ## Qualification and remaining boundaries On `dev-ikons`: - **Full deploy:** every stage and all 13 route probes pass, and a repeat run changes only the known Rook repository task. - **Fresh Waldur installation from the seed:** after volumes, workloads and generated Secrets were deleted, `make deploy` completed in 13 minutes. The restore took 28 seconds and `initdb` 3 minutes. - **Sign-in:** alice and bob sign in to GitLab and Waldur through the AAI in Chromium, which trusts only the AAI CA. Each service's API reports the signed-in user. - **GitLab:** `dx/common/scripts/qualify_gitlab.py` passes. A maintainer pushes a 30 MiB LFS object and a developer restores its checksum. A developer cannot push to the protected default branch but can push a branch, and cannot delete the project. Outsider and anonymous clones are refused. The script's Git runs without credential helpers, and it revokes its root token afterwards. - **Isolation:** a restricted pod in a new namespace cannot reach GitLab, Waldur, their databases, Redis or RabbitMQ directly. The public Waldur route works, and `/admin/` is refused even with a forged forwarding header. - **Seed guard:** on a scratch database restored from the seed, the guard passed with a fixture identity provider, whose data the dump left out. The live database, which has users, was refused. Not exercised: a cold capture end to end (the seed came from a hand-taken dump of the same state), egress isolation, fresh-VM and reboot qualification, staging and production. The GitLab client is a new registration for any external federation.