# GitLab and retained AUP authentication deployment The API-auth overlays compose the Go API, Beto Party Governance, retained Themelio, the GitLab actor and a separate compatibility worker. The existing GitLab-backed AUP reader/writer remains authoritative. Waldur identity setup and binding-based invitation acceptance are a separate integration; their pending state does not block GitLab or AUP. ## Selection and delivery The dev-box GitOps root always selects `pharos/overlays/devbox-api-auth` and `themelio-compat/overlays/devbox-api-auth` together. NextAuth and the legacy Party bridge are retired. Distributed environments enrol the same components through their own reviewed policy, as described below. Qualify this commit with the commands below and confirm the deployed AAI and GitLab identities, then use the normal `make deploy TARGET=dev-NAME` workflow. Deployment requires the infrastructure, AAI, GitLab and Themelio profiles and no active takeover. Argo delivers the same commit and image digests to both Applications. Existing NextAuth sessions require a fresh login. Unenrolled base and distributed compositions must gain the API transport, credentials and provider-worker configuration before upgrading to these images; the runtime refuses missing configuration. This change does not qualify those installations. The Recreate strategy coordinates writer replacement; expect a brief interruption. These four Deployments use Argo's `Replace=true` sync option so the update clears Kubernetes' existing rolling-update defaults. This updates the Deployment object in place; the cutover does not use forced deletion. The rendered strategy also retains an explicit `rollingUpdate: null`. The AAI bootstrap registers the exact callbacks `https://themelio.localtest.me/auth/oidc/callback` and `https://platform.localtest.me/auth/oidc/callback` for `pharos-control-plane`. Private client registries must preserve both callbacks and the complete AAI profile scopes. The API selects a callback from the login's exact origin and each public host receives host-only login and session cookies. Both hosts share the same session authority and retained compatibility workflow. The actor explicitly selects GitLab's `openid_connect` provider. Its issuer comes from the same AAI source as the API. Exact issuer/subject proof establishes the account association; email does not link an existing account. Account setup establishes the federated identity and provider binding independently of repository namespaces. Source Control provisions category groups beneath each canonical Project UUID under its configured root. See the retained web's [repository layout and rollout prerequisites](../../../../compat/themelio/services/web/api-spec/project-repositories.md). ## Distributed staging The distributed root reads `authentication` from the committed environment policy. The default `mode: legacy` retains the unenrolled distributed composition. The current images require API authentication, so qualify and enrol an existing installation before upgrading it. API mode selects `pharos/overlays/distributed-api-auth` and `themelio-compat/overlays/distributed-api-auth` together at the same source revision. It supplies the exact callback/origin set, both provider instance identities and the common account scope to the API, administration Job, retained web, worker and provider actors. Beto's enrolments and provider-read grants are projected from the same recorded identities. The delivery gate rejects a child with missing bindings or the wrong overlay. Independently rendered overlays have empty required bindings and cannot inherit the development identities. Add the following to the selected environment policy after verifying the existing provider bindings. The angle-bracket values are required operator inputs; they are rejected by validation until replaced: ```yaml authentication: mode: api provider_instances: gitlab: '' waldur: '' account_scope: platform edge: origins: pharos: https://platform.staging.pharosfactory.ai themelio-compat: https://themelio.staging.pharosfactory.ai # Preserve the environment's other reviewed origins. ``` Provider instance IDs identify provider installations in persisted bindings and signed work. Confirm them in the existing Beto enrolments and retained provider configuration; they are independent of DNS names and OIDC client IDs. A fresh installation may enrol new stable names explicitly. Distributed policy rejects `devbox`. Existing records under that identity need a reviewed migration before selection. Keep `account_scope: platform` only if it matches the enrolled scope. The root derives these exact `pharos-control-plane` callbacks: - `https://platform.staging.pharosfactory.ai/auth/oidc/callback` - `https://themelio.staging.pharosfactory.ai/auth/oidc/callback` Confirm both with the external AAI provider before the cutover. The retained legacy callback is `/api/auth/callback/openid_connect`; enrolling that callback alone does not enable API mode. The current images retire that flow; coordinate its removal from the provider registration with the installation upgrade. The existing private external-AAI input remains the source of issuer, client credentials and trust; the expected Icarus issuer is `https://staging.aai.pharos.grnet.gr/auth/realms/pharos`. No credential enters this public policy. The normal Ansible Argo bootstrap already applies `bootstrap/api-auth.yaml`. Verify those named slots and grants on an adopted installation before delivery. The administration Job stores the new installation's provider scopes in `pharos/api-auth-material`. Repeated delivery keeps every signing/session key and scope. Older records without a scope field permanently denote the historical devbox identities. A changed instance or account scope stops reconciliation before writes and needs explicit migration; deleting the authority Secret is not a recovery procedure. At the staging base `8c55b7b7e`, the committed Icarus policy still uses the older Kubernetes pin and public origins. Retain the reviewed Icarus foundation, gateway, GPU, DNS and forwarding-header fixes when integrating this change. This change does not reinstall the foundation or reproduce the integration branch. The public edge on this base serves the `pharos` origin through the web's same-origin proxy and keeps direct API aliases disabled; the newer Icarus browser-surface mapping must retain that property when the branches are combined. Keep the operator-code browser surface within the approved management network until the API-mode login is qualified. Exercise first/returning AAI login on both hosts, Project visibility and authorisation, AUP, provider identity binding, logout/replay, expiry and interrupted worker recovery. Check that the operator-code session endpoint is closed and private auth routes remain unreachable publicly. Argo's Synced/Healthy state alone does not establish these results. Existing sessions need a fresh login. Use the rollback procedure below if qualification fails, preserving authority and pending work. ## Routes and capabilities The existing public route still targets `themelio-compat/web` port 3000. Its transport container sends `/auth/*` to the API over verified private TLS and `/api/v1/*` to the public Go API with the session cookie, public Origin and command headers preserved. Retained server adaptors also reach these canonical contracts through the private TLS API listener. Both routes use the API's ordinary session authorisation. Retained pages and other `/api/*` routes reach Next.js on loopback port 3001. `/internal` and `/internal/*` are refused publicly, including normalised encoded paths. The existing platform `/auth/` issuer route remains with Keycloak. Existing Git/LFS routing remains with the platform gateway. The dev-box overlay additionally selects the `development-hmr` component for the exact `/_next/webpack-hmr` WebSocket path during frontend takeover. It forwards only that upgrade to Next.js and retains the canonical public forwarding headers. The shared API-auth component alone and legacy overlays retain their existing routes. The ARM64 profile inherits the same development route. | Caller | Private destination | Capability | | --- | --- | --- | | Retained web | `https://api-auth.pharos.svc.cluster.local` | Canonical session validation and binding reads | | API | `https://themelio-auth.themelio-compat.svc.cluster.local` | Durable work dispatch and independent AUP reads | | Compatibility worker | `https://api-auth.pharos.svc.cluster.local` | Evidence, current session fences and signed binding hand-off | | Compatibility worker | `https://gitlab-auth.themelio-compat.svc.cluster.local:8443` | Closed, authenticated account and token operations | | Retained web | `https://gitlab-auth.themelio-compat.svc.cluster.local` | Existing private actor operations, including AUP | The actor's two Services ports lead to separate TLS listeners: 8443 exposes the worker protocol; 8444 retains the web's actor operations. NetworkPolicies allow the worker only its closed listener. The broad retained namespace ingress rule excludes auth workloads, so it cannot reopen the AUP writer to the worker. Retained server-side requests use loopback Next.js directly and carry the same canonical session cookie through its normal middleware. The worker has its own tokenless ServiceAccount, restricted Valkey credentials, scoped service capabilities and an Ed25519 binding signer. It receives no OIDC client secret, GitLab administrator token or database credential. The existing GitLab administrator token remains with the retained root actor. The worker image contains its compiled bundle and production Node runtime. Loopback health probes report storage/startup readiness; termination stops claims, drains the active operation and closes storage within the 180-second grace period. ## Authority material and recovery The administrator bootstrap creates named empty Secret slots and grants the `pharos/api-auth-credentials` Job access to those slots, the published `themelio-oidc` source and `get`/`patch` on the `pharos/beto` Deployment. Its runtime image is the administration image; application Pods receive no Kubernetes token or authority-record access. The Job creates `pharos/api-auth-material` once. It retains the session derivation key, separate service capabilities, separate API/worker signing keys and a private TLS CA. Each runtime Secret is a scoped projection. Beto receives public attester keys and provider read scopes. Site binding enrolment remains a separate owner configuration; this slice retains the complete AAI profile. The AAI CA and private transport CA form the runtime trust bundles. No CA private key or provider administrator credential reaches the API, Beto or worker. A repeated reconciliation preserves values. Partial delivery reuses the same record. Missing authority with existing projections, mismatched scope material or a changed issuer/client identity stops delivery and requires explicit recovery. Back up the authority Secret using the operator's normal protected backup process. Restoring credentials from generated placeholders would invalidate live sessions and outstanding signed work. Beto reads Site binding enrolments at startup. After publishing its projection, the Job records their digest in Beto's `pharos.ai/party-site-bindings` Pod annotation, which rolls existing Pods when the value changes. Every retry checks the annotation, including when a previous attempt saved the Secret and failed before the rollout. Argo preserves this one Job-owned field during sync. Repeating an unchanged reconciliation leaves the Pod template unchanged; a first installation reads the projection when Beto starts. Run `make hpc-site-central` before the deployment that should activate the HPC Site, so the Job can read `WALDUR_HPC_SITE` from `hpc-offering-settings`. If those settings arrive later, rerun the Pharos sync to publish the enrolment and roll Beto. Existing installations must receive the updated bootstrap Role before the hook runs. An authority record created before the isolated Waldur actor is extended once with its account and invitation capabilities and TLS leaf. Existing session, signing, service capabilities and TLS private keys remain byte-for-byte unchanged; certificates retain the normal renewal policy. Any other change to the record still requires explicit recovery. TLS leaves renew within 30 days of expiry with unchanged private keys and CA. The source-owned OIDC secret and public AAI trust can be refreshed. After changing source credentials or renewing certificates, run the administration reconciliation and coordinate restarts of API, retained web, actor and worker: their environment and Node trust are loaded at startup. Routine delivery never rotates session or signing authority. CA replacement and issuer migration require an explicit plan. For a planned rollback, complete logout and provider-token cleanup while the API is running, then restore a qualified coordinated release. Preserve API session and signing authority, Party identities and durable compatibility work. Never flush Valkey or regenerate keys to recover provisioning. There is no runtime switch that restores NextAuth in this release. ## Qualification Install the frozen root and retained pnpm workspaces. The assembled lane needs Go, native PostgreSQL tools (`initdb`, `pg_ctl`, `psql`), OpenSSL, Docker and Playwright Chromium. RabbitMQ and the production-pinned Valkey run in disposable Docker containers; Valkey retains its production ACL. Install the browser with `pnpm exec playwright install chromium`. The proxy lane needs Docker and the frozen `devops` Python environment. Both lanes use disposable resources. ```sh pnpm verify pnpm test:authentication:e2e uv run --project devops --frozen python scripts/authentication_proxy_smoke.py ``` The assembled lane starts the production API, Beto, Next.js, GitLab actor and worker with real PostgreSQL, RabbitMQ and durable Valkey. Chromium exercises first signup, paused account setup, explicit resume, identical-command retry after an uncertain response, retained AUP acceptance and normal pages. HTTP assertions cover returning users, current AUP gates in both endpoint families, private-host checks, callback replay and logout cleanup. OIDC and GitLab upstreams are fixtures. The proxy lane exercises the committed Nginx configuration with real certificates and separate provider stand-ins. Neither lane establishes live provider or Kubernetes NetworkPolicy behaviour. Before cutover, exercise the selected overlay on the intended dev box: first and returning AAI login, all configured GitLab namespaces, AUP acceptance/version change, retained pages, Git/HTTPS/LFS, logout, expiry and provider interruption. Probe public denial of private routes, foreign-Pod access denial, Pod replacement and worker recovery. Record live provider UAT, k3s/Minikube and deployment results separately. Waldur provisioning and invitations await their coordinated changes. ### Live Keycloak and GitLab journey Use a personal dev box and a disposable identity. The provider helper creates a unique Keycloak user through an administration Job, copies no administrator credential to the workstation and leaves shared users and client configuration unchanged. It prints the private state directory and fixture username. Keep those values for verification and cleanup. ```sh python3 scripts/authentication_live_provider.py prepare --target dev-NAME --ssh HOST ``` Account preparation requires no installation roots or personal groups. The `ensure-roots` helper remains available only for qualifying older installations; it is not a prerequisite for this journey. Run the retained journey against its canonical HTTPS origin and exact deployed issuer: ```sh NODE_EXTRA_CA_CERTS=.local/local-aai/dev-NAME/ca.crt \ node scripts/authentication_live_browser.mjs \ --credentials-file DIR/credentials.json \ --origin https://themelio.localtest.me \ --issuer https://pharos.127.0.0.1.sslip.io:8443/auth/realms/pharos \ --report-file DIR/browser-report.json --loopback ``` Qualify the Pharos web with the same disposable identity. This checks its own host-local callback and session cookie, the generated Project request, rendered Project list and logout: ```sh NODE_EXTRA_CA_CERTS=.local/local-aai/dev-NAME/ca.crt \ node scripts/platform_authentication_live_browser.mjs \ --credentials-file DIR/credentials.json \ --origin https://platform.localtest.me \ --issuer https://pharos.127.0.0.1.sslip.io:8443/auth/realms/pharos \ --report-file DIR/platform-browser-report.json --loopback ``` Chromium must trust the dev-box CA. Alternatively, `--spki-file FILE` accepts explicit certificate-key pins after independent CA and hostname verification of both endpoints. It does not disable certificate verification globally. The report must be a new file; credentials and reports are private, and session cookies stay in memory. `NODE_EXTRA_CA_CERTS` points to the dev-box CA verified independently over SSH. It enables TLS verification for emergency HTTP session cleanup if a browser step fails; browser trust and scoped SPKI verification remain unchanged. Emergency cleanup is recorded separately from the real Sign Out checks. The retained journey exercises first login, policy gates in both endpoint families, the retained AUP button, GitLab readiness, retained pages and the canonical Project directory through all six retained route aliases. It then checks an independent returning session, logout, rejection of a revoked session and callback replay, and survival of the other session. Bounded status reads belong to the qualification driver; the product adds no polling. `--existing-user` explicitly skips the first-time acceptance assertion when repeating a journey after acceptance. The Pharos web journey accepts either a populated or empty Project directory and checks that the rendered count matches the API response. After both sessions log out, independently inspect provider state. Use the deployed policy version: ```sh python3 scripts/authentication_live_provider.py verify-gitlab \ --state-dir DIR --aup-version VERSION --expected-active-tokens 0 python3 scripts/authentication_live_provider.py cleanup \ --state-dir DIR --confirm-fixture FIXTURE_USERNAME ``` Verification checks the exact federated subject, absence of personal category groups, the AUP attribute and token revocation through GitLab's toolbox. Cleanup verifies recorded ownership and absence of foreign contents before using provider deletion services. If GitLab schedules asynchronous deletion, repeat cleanup after it completes. Cleanup removes the owned Keycloak identity, administration resources and local password file. Canonical Party records and audit history remain intact.