#0047 — Dev-box consolidation: ownership, Argo AAI and shared data

Status: proposed

Date: 16 October 2026. Source requirements: the development hand-off of 15 October 2026, inspected against main at 297a7d50b6dc6d2e034d7f4ec97f9d0598ee609a.

#Context and proposal status

This record brings the requested three-PR hand-off into the repository for architecture discussion. It retains the implementation requirements and acceptance criteria; it does not declare the remaining work implemented or accepted. PR #694 implements the first part. AAI work is in progress; the shared-data conversion and seeded cold-installation acceptance remain outstanding.

The observations below describe the original baseline, not immutable live state. Recheck the target, branch and overlapping PRs before implementation. Destructive test authorisation is limited to the named personal targets; this proposal grants no authority to change another developer's VM or staging/production installation.

#Selected integration

The consolidation is implemented in PR #697, retaining its dx/common tooling and per-service DX layout. PR #696 remains the separate Project feature and lands first. PR #694's guarded ownership logic is incorporated with attribution; its original proposal commit retains its author. Adapted ownership, Argo AAI and shared-data contributions credit ikons as co-author.

Argo owns shared infrastructure and AAI after foundation bootstrap. Their Sync Jobs provide retryable administration, ordered by the root's sync waves. Argo starts every operation; the controller applies the root and observes revision, health and operation outcome, as amended in decision 0048. The desktop displays those observations. AAI source, custom mapper and credentials delivery are under devops/aai, with workstation exports and probes under dx/common/local_aai. PostgreSQL 18 conversion and its live acceptance remain the next implementation step.

The historical hand-off below records the original three-PR proposal. Its devops/tools destination is superseded by the selected DX layout. Current shared infrastructure uses decision 0048; Project communication uses 0046.

#Proposed outcome and development scope

Deliver three reviewable PRs, in this order:

1. User-owned development state and relocation of the controller/tools.
2. Argo-managed AAI and a working Keycloak Account Console.
3. A common PostgreSQL 18 data layer and consolidated broker/cache deployment.

The user explicitly permits discarding old application data on these development installations. An upgrade must support a clean application installation on the existing cluster; database migration and restoration of old business data are not required. Routine subsequent deployment must preserve the new data.

The user authorises destructive tests and recreation of metallo-ikons-2 and creation of metallo-ikons-3 for cold-installation tests. Preserve the existing installation on -2 until its in-place conversion has been qualified. The Icarus laboratory remains independently installed and is not reset by platform deployment. Do not make unsolicited changes to other developers' VMs.

The primary acceptance condition is:

After updating to the implemented release and returning any active takeover, an existing developer runs the ordinary make deploy TARGET=dev-NAME and receives the new installation. They do not need a separate reset command, manual kubectl deletions, a fresh target name or a replacement VM.

#Read before implementation

Follow root AGENTS.md, ARCHITECTURE.md, DDD.md, CLIENT-SUBSCRIPTIONS.md, TESTING.md and CONTRIBUTING.md. Read decisions 0035, 0039, 0040, 0041 and 0045 and their associated guides/DDs. Read any nearer instructions in files added since this hand-off.

These are technical platform administration changes. The owned facts are target binding, delivered revision, deployment layout, resource ownership and transition progress. Commands install, reconcile, convert and repair an owned target; queries inspect its status and evidence. No new business context or aggregate is introduced. Keep Metallo's context boundaries and empty delivery registries intact.

Do not edit legacy/; it is an archived reference with integrity checks. Update the active architecture, developer guides, changelog, decisions and enforceable security/architecture fixtures as appropriate. Check main and all open PRs before allocating any new decision number.

#Explicit delta from ADR-0040

Read this against ADR-0040, which is currently Proposed. This hand-off authorises the described development work; it does not by itself change the ADR's wider approval status. Update the decision and linked guides in the implementing PRs so their stated behaviour matches code.

The following requirements are additions to ADR-0040, rather than previously implemented stages that can be assumed to work:

• One shared PostgreSQL 18 server across all listed services, with private database/role authority and a compatible GitLab/chart version.
• Retirement of the separate Pharos RabbitMQ deployment in favour of one physical broker. The baseline Pharos binding was inactive; the API/Beto Project queries proposed in #696 must retain their topology, users and permissions on the shared broker if integrated. Keep Waldur and Pharos scopes isolated; any later Metallo binding is separately enabled.
• Explicit session/history/GitLab/Argo cache profiles and their isolation checks. Sharing the sessions instance intentionally changes the existing per-service deployment; preserve the separate history instance required by client delivery.
• Automatic, one-time conversion of already installed dev-boxes by ordinary make deploy, with old data disposal, exact obsolete-resource cleanup, durable retry progress and preservation on later deploys. This is not specified by the original layout ADR and must be implemented and qualified, not inferred.
• User-owned generated state and repair of existing root-owned state on local and SSH-runner paths, including interrupted execution and private state symlinks.
• Explicit existing-stack and seeded cold-stack acceptance on the authorised VMs. Waldur seeding itself already exists under ADR-0041; adapting and proving it against shared PostgreSQL 18 is the new work, not inventing another seed system.

Update ADR-0041's per-service stateful dependency wording, ADR-0045's delivery behaviour and architecture/client-subscription guides where the implementation changes their guarantees. Keep the transition implementation small and specific to this layout change rather than introducing a general deployment framework.

#Observed starting point

The following were inspected for the original hand-off, not promised as immutable live state. Reinspect before changing resources.

All seven observed Argo Applications were Synced/Healthy. This did not establish that the Account Console worked. A Chromium login accepted the test user's credentials, then /auth/realms/pharos/account/ returned HTTP 401 and displayed "Something went wrong". The account-console clients in pharos and pharos-baseline had no attached client scopes. Its observed access token lacked sub; the master realm had the normal scope attachments.

There were 6,486 root-owned descendants under this checkout's .local, primarily metallo/tools-home and metallo/dev-developers. Reading target configuration and traversing istio-cache already failed for the host user. The existing local_aai.common.restore_ownership repairs only selected state and is insufficient.

Key entry points before relocation:

• devops/metallo/tools/run.sh, Dockerfile.tools, Makefile.
• devops/metallo/scripts/deliver.py and scripts/devbox/delivery_runner.py.
• devops/metallo/scripts/devbox/deploy.py, argocd.py and seeds.py.
• devops/metallo/local_aai/{up,generate,identity_config,delivery,common}.py.
• gitops/apps/{base,overlays} and gitops/manifests/{pharos,gitlab,waldur,themelio-compat}.
• docs/development.md, devops/metallo/GUIDE.md, docs/decisions/0040-platform-layout.md.

#PR 1: user-owned state and tools relocation

Suggested title: fix(devops): make development tools state user-owned.

• Move the generic dev-box controller and tools image to devops/tools, completing stage 9. Keep Metallo-specific development responsibilities clearly owned.
• Update imports, root discovery, Make targets, image build inputs, test discovery, source delivery, documentation and Icarus consumers of moved generic utilities.
• Run ordinary filesystem-writing tools with the calling host user's UID/GID. Limit privilege to operations that require it. Account for Docker socket access, shared locks, SSH, caches, restrictive umasks and writable container home paths.
• Fix both the Linux launcher and macOS-to-Linux SSH delivery runner; the latter currently invokes the entire runner using sudo. Recover the original caller's identity rather than inferring ownership from an accidentally root-owned checkout.
• Make ordinary make deploy repair recognised pre-existing administrator-owned deployment state early enough to read its configuration and acquire its locks. Retain target/SSH/disk bindings and necessary credentials. Repair only verified owned paths; do not follow state-tree symlinks into SSH keys or other directories.
• Handle linked worktrees and the runner's .local symlink. Preserve private file modes. A recursive permission relaxation is not an ownership fix.
• Ensure a cached tools image cannot keep executing the old root-writing launcher. Repeated use, failed commands and interruptions must leave usable user-owned state.

Qualification: seeded old root-owned state, real Docker binds, host and SSH-runner delivery, linked state, failure/retry, takeover and hand-back. Scan all state paths written by these commands, including caches and seeds, after success and failure.

#PR 2: Argo-managed AAI and Account Console fix

Suggested title: feat(aai): manage Keycloak through Argo CD.

• Add the AAI Application and manifests under the existing GitOps layout. Move Keycloak lifecycle, bootstrap, realm/client/user reconciliation and client-secret delivery into explicit, ordered administration operations managed through Argo.
• Include the custom Keycloak image/mapper in the normal committed-source build and exact-revision delivery path. Remove the independent imperative AAI installer.
• Make make reconcile deliver AAI source/configuration changes through Argo. Retain existing convenience commands where useful as wrappers for the new owner.
• Correct built-in Account Console scope/mapper configuration in both realms, including existing realms. Verify subject, account audience/roles and required account endpoints rather than stopping after fixing the first 401.
• Preserve the intentional ordinary-client token/userinfo claim shapes, resource exchange restrictions and administrator-only attribute editing.
• Keep the issuer URLs and public development CA stable. Host edge/trust remains bootstrap-owned unless a separate reviewed change intentionally moves it.
• Keep administrator credentials in administration Jobs, outside the Keycloak runtime and consumer services. Repeated sync must not rotate client/user secrets.
• Extend Argo namespace/RBAC/admission/NetworkPolicy boundaries precisely. Client secret delivery must not grant broad cross-namespace Secret administration.
• Remove the old writer as ownership transfers. First adoption and interrupted adoption must be recoverable through ordinary make deploy.
• Keep the current AAI PostgreSQL deployment for this intermediate PR if needed; PR 3 moves it onto the shared server. Icarus consumes the same public AAI contract.

Qualification: Account Console login and authenticated account endpoints in both realms, identity/baseline probes, GitLab/Waldur/Themelio login, repeated sync with stable secrets, first adoption and interrupted deployment. Run applicable Icarus identity checks without making the laboratory a platform installation stage.

#PR 3: shared PostgreSQL 18 and data dependency cleanup

Suggested title: feat(devbox): consolidate the application data layer.

• Add an Argo-owned shared PostgreSQL 18 service, delivered before consumers and their database/bootstrap/migration Jobs. Resolve first-sync ordering across Applications and readiness explicitly; numbered waves alone are not evidence.
• Retain separate application/context databases, migration authority and restricted runtime roles. The observed databases are pharos_execution_core, pharos_site_coordination, themelio, keycloak, gitlabhq_production, waldur and celery_results. Recheck their required extensions and privileges.
• Runtime Site-local SQLite journals remain private. Site Runtime receives no platform database or administrator credentials.
• Move consumer connection configuration, Secrets, administration Jobs and network policies onto the shared server. Reconcile stored URLs as well as manifest values.
• Remove superseded per-service PostgreSQL resources and provisioning code.
• Keep one RabbitMQ instance serving Waldur, with its required management/STOMP plugins. Retire the separate Pharos instance, preserving any required API/Beto Project query bindings from #696 on the shared broker with separate vhosts, credentials and permissions. Update administration and runtime NetworkPolicies for cross-namespace access. Remove only obsolete bootstrap dependencies. Do not activate new Metallo event subscriptions as scaffolding. Future Metallo delivery can use the same instance with a separate vhost and credentials.
• Keep the current recommendation explicit: one persistent sessions Valkey shared by Pharos/Themelio, separate disposable realtime-history Valkey, and independent GitLab and Argo Redis instances. A different cache topology needs a separate decision. Preserve the existing durability and revocation requirements.
• Before sharing sessions, replace Themelio's flushDb() cache-clear behaviour with owner-scoped clearing and restrict its commands/key/channel access. Logical database numbers alone do not establish isolation. Verify denied foreign access.

#PostgreSQL and GitLab versions

At the original baseline GitLab used 18.4.6, chart 9.4.6, with PostgreSQL 17.6. Pharos and AAI used 18.0; Waldur and Themelio used 17.6. The legacy shared CNPG role defaulted to PostgreSQL 18.1. The target is one PostgreSQL 18 server for every application.

Recheck official compatibility and select exact supported GitLab/chart/database pins, including patch versions and image digests. The official current matrix inspected for the original hand-off listed PostgreSQL up to 17 for GitLab 18.x and up to 18 for GitLab 19.x. Do not assume the existing 18.4.6 has qualified PostgreSQL 18 support, or that a successful legacy configuration establishes it. Include the necessary GitLab/chart update and its validation in this PR.

Primary reference: https://docs.gitlab.com/install/requirements/#supported-versions

#Mandatory existing-installation behaviour of make deploy

Implement this behaviour inside the supported deployment path, not just in a runbook. Both local Linux and SSH-runner delivery must use it.

1. Resolve the existing target and validate development environment, saved cluster identity, disk binding and ownership. Refuse active takeover using the existing lease rules. Repair legacy state ownership before a permission failure blocks this preflight. Keep the saved target and binding usable through tools relocation.
2. Detect the previous layout from recognised resources and ownership records. Persist an explicit technical transition version/progress record. Scope cleanup to known platform application resources in this target; avoid label-only broad deletion and any staging/production target.
3. Fence old Argo desired state and the imperative AAI writer before deleting or replacing their resources. No active writer may recreate the old dependencies. Stop application writers/workers as necessary while databases are reset.
4. On the one-time layout transition, discard the superseded application data and remove the exact obsolete workloads, Services, Jobs and PVCs that conflict with the new installation. Handle GitLab's existing database repositories and credentials consistently if its database is reset; test an operational service. Preserve cluster, Ceph infrastructure, public CA/edge and independent laboratory resources. Platform reset must not wipe the entire data disk.
5. Install/provision the new dependencies, restore the qualified Waldur seed, initialise consumers and complete their administration hooks. Reconcile stored credentials/configuration consistently; credentials must not point to removed DBs.
6. Require every selected Application to be Synced/Healthy at the delivered commit, all required hooks complete and the relevant functional checks successful before marking the new layout ready. A partial failure must retain retry information.
7. Repeat ordinary make deploy after failures, including after individual cleanup operations. Resume the transition without a second writer, broken target binding, duplicate bootstrap or unconditional deletion of newly provisioned data.
8. On subsequent successful deploys, skip destructive conversion. Demonstrate that data created after conversion, new PVC identities and credentials survive.

Current Applications use prune: false; removing manifests does not retire the old live resources. StatefulSet PVCs also need explicit treatment. Implement exact cleanup/ownership transfer instead of assuming Argo will infer it. Repeated PostSync hooks must never reset a healthy database.

Document expected downtime and that old application data is discarded once. Other developers should only need to update the repository, hand back takeover if active and run their existing make deploy TARGET=.... Preserve first-install support for SSH=... STORAGE_DEVICE=/dev/vdb and saved bindings on later deploys.

#Live qualification targets and Waldur seed

#Existing stack: metallo-ikons-2

The existing qualification VM is metallo-ikons-2, with saved platform target dev-developers. Resolve its private SSH/gateway inventory before use; addresses can change after recreation. Use configured access and sudo -n k3s kubectl where appropriate. Private connection settings remain outside this record.

Before changing it, capture non-secret inventory, owner records, old resource UIDs and the delivered revision. Test conversion of the actual old installation using ordinary make deploy, then repeat unchanged deployment and an interrupted/retried transition. Do not erase this baseline before proving conversion works.

#Cold stack: metallo-ikons-3

Read the separate VM administration repository's AGENTS.md and README.md before using its tooling. Creation of metallo-ikons-3 is authorised. Select only VM index 3 and suppress access email (SKIP_ACCESS_EMAIL=1); keep administration repository paths and private inventory outside the platform source.

Check inventory and resource existence first. Keep private OpenStack credentials on the administration host. Use the selected VM's actual fresh data disk and a distinct platform target, for example dev-cold-3. Configure/verify its SSH alias and host identity, then qualify a first installation and unchanged second deploy. Record inventory changes separately from the platform PRs. No access email is needed or authorised by this hand-off.

The user has also authorised destroying/recreating -2 for tests. The repository's remove-user.sh ikons deletes all that user's VMs, removes the user entry and can affect access. It is not a single-VM reset command. Any recreation must select only the authorised VM and its exact volumes/port, leaving -1 and other users alone.

#Seeded cold setup is required

Waldur's seed mechanism is already implemented and previously qualified. Use it for this cold test; do not spend another full first-install migration cycle as the default test path. At inspection the local file was:

.local/metallo/seeds/waldur-8.1.2-4c82b15d9042.sql.gz

It was 227,590 bytes. Rediscover it if state moves. seeds.settings() derives the name from the chart version and Waldur image digest; its value must agree with the chart's seed helper. Existing seeds.offer/collect moves it via the waldur-seed PVC, and the restore hook precedes normal migrations.

Preserve and adapt this path to the shared PostgreSQL 18 server. Verify checksum, gzip integrity, expected schema/version, absence of credentials and successful restore with the chosen database/client versions. Do not copy a populated developer database as a seed. If the application's schema/image changes, produce and qualify a matching credential-free seed rather than relabelling the old one.

Pre-position the qualified seed in the actual deployment runner's private state before the cold test. Source bundles exclude ignored .local files; local seed existence does not mean a macOS SSH runner has it. Preserve the supported private seed-transfer mechanism and make missing/mismatched seeds visible in diagnostics. Do not commit private state or credentials. Evidence must show the restore hook actually restored the seed; silent fallback to full migrations does not qualify the requested seeded cold-installation lane.

#Tests, evidence and completion

For each PR run pnpm verify, pnpm check:large-file-review and the applicable real infrastructure lanes. Adapt relocated commands and test discovery. Add focused regressions for the new ownership, transition, retry and isolation rules. Keep handwritten files below 450 lines or register justified reviews.

Required final evidence includes:

• Existing-stack conversion, interruption/retry and unchanged repeat on -2.
• Seeded cold installation and unchanged repeat on -3.
• PostgreSQL provisioning, extensions/migrations and denied foreign/runtime-admin access; real broker/session/realtime integration checks.
• Account Console in both realms, OIDC identity/baseline probes and retained GitLab/Waldur/Themelio login; applicable Icarus identity regressions.
• GitLab Git/LFS and background work, Waldur API/workers, session logout/revocation and realtime recovery with the selected data services.
• Network-policy probes, exact-revision Argo health/hooks and ownership boundaries.
• No root-owned generated host state after successful, failed and repeated delivery, including the SSH runner, caches, seed state and supported takeover runtimes.

Record commits, target/VM identities, layout version, seed digest/restore result, test results and remaining limitations without secrets. Separate automated browser checks from manual UAT. Explicitly report skipped PostgreSQL, browser, UAT, Minikube/k3s, deployment and provider lanes; do not report a skipped lane as a pass.

The original investigation's pnpm verify and large-file review passed. Live k3s inspection reproduced the Account Console failure. That baseline evidence does not qualify the implementation described here. Record subsequent results in each implementing PR; do not infer completion from this proposal.

#Discussion with overlapping proposals

PR #697 proposes workstation defaults, Colima, DX relocation, a desktop application, shared infrastructure and guided setup/reset. Its overlapping infrastructure decision uses PostgreSQL 17.6 and requires reset before installing the new layout. This proposal requires PostgreSQL 18 and one-time conversion through ordinary deploy. These alternatives need agreement before the overlapping implementations land.

PR #696 adds API/Beto Project queries through RabbitMQ. The broker requirement above updates the hand-off's original observation that Pharos' broker binding was inactive; it preserves that work without adding Metallo event subscriptions.

Agree one controller/tooling directory map, reuse qualified ownership and provider-isolation work, and preserve public commands and private target bindings. Argo remains the application owner after explicit bootstrap/adoption; ordinary deploy must not reintroduce a recurring independent infrastructure writer. Desktop/setup UX should call the supported commands rather than introduce another application lifecycle owner. Keep this transition specific to the selected layout, without adding a general deployment framework.

Neither overlapping PR's ADR 0046 is accepted or renumbered by this record. The earlier decisions retain their current status; implementing PRs must update their affected guarantees when the proposed changes are agreed.