#Testing

#Choosing checks

Select hand-off checks from the complete change, including uncommitted edits.

• Documentation- and instruction-only changes: run pnpm verify --group policy and git diff --check. The policy group includes architecture, large-file review and frozen archive checks. Review changed links and confirm documented commands against their scripts or CLI definitions. Search for tests that consume the changed documentation and run those tests when applicable.
• Code, contracts, schemas, dependencies, tests, generators or deployment changes: run the full pnpm verify suite and every applicable infrastructure lane. This also applies to mixed changes and executable fixtures embedded in documentation.

For documentation changes that introduce or alter an operational command, validate that command at the smallest appropriate boundary using its owning guide. Any required infrastructure checks follow the dev-box workflow below. A prose-only edit needs no unrelated service build or live deployment run.

Report the checks run and the reason for any omissions. For the documentation lane, explicitly state that full verification and infrastructure lanes were omitted because the diff contains only documentation or agent instructions. This hand-off policy leaves automated CI selection unchanged.

Provider-selection regressions in test_aai_provider.py cover private-file permissions, SSH framing, discovery trust, credential rotation, stale reviews and preservation of local identity when switching back. The DX browser lane covers the switch in both colour schemes; hosted AAI registrations and end-to-end login remain a separate environment-dependent check.

#Ad hoc platform acceptance

The Platform Acceptance Suite uses Playwright Test, TypeScript and Gherkin against the retained Themelio experience. It provides selectable user scenarios, local HTML reports and explicit coverage gaps. Run it ad hoc against a prepared deployment; it is not part of CI or a release gate. ADR 0068 records the choice.

For combined qualification of pending PRs, use the disposable integration branch. Its manifest records the selected source commits; full CI runs against an immutable snapshot. Live acceptance still requires that revision to be deployed to the selected dev box.

#Infrastructure checks on the dev box

Run the selected verification and focused unit checks on the workstation. For development and agent-led checks that need PostgreSQL, RabbitMQ, Valkey, Kubernetes or provider services, use the configured personal dev box. An unavailable workstation Docker daemon or missing local service is a reason to move the lane to the dev box.

CI retains its configured isolated runners. pnpm verify renders charts with Helm 3: a Helm 4 on PATH is passed over for the pinned release, which scripts/platform_helm.py fetches once into .local/platform-tools.

1. Resolve TARGET, SSH and DEPLOY_RUNNER from the checkout's .env, explicit overrides and saved target settings. Follow the workstation settings and dev-box prerequisites. Inspect the saved runner binding in .local/metallo-delivery/<target>/runner.json; an existing target may use a separate build runner. Check SSH access and the tools required by the selected lane before running it.
2. Select the existing lane below or in the owning context's README. Cluster and live-provider qualification commands address the selected dev box through their documented target/SSH options. Browser drivers may run on the workstation against the dev box's authenticated endpoints.
3. Run generic pnpm infrastructure suites over SSH in a separate checkout on the configured runner, falling back to the dev box when no separate runner is configured. Use that host's Docker Engine and native tools. These commands execute where invoked; TARGET, SSH and DEPLOY_RUNNER do not redirect them. Keep bind-mounted files on the same host as Docker. Preserve the deployment runner's checkout and private state.
4. Test the exact source revision under review. Transfer any required uncommitted changes and record them with the base commit. Exclude workstation .env, credentials, generated private state and tool caches from source transfers. Use the repository's pinned toolchains and frozen dependencies; the CI workflow records fixture prerequisites, including native PostgreSQL and Chromium for assembled authentication.
5. Run fixture suites as the normal runner or dev-box user. Keep their databases, broker scopes, containers and namespaces disposable and isolated from the installed platform. Use each lane's owned cleanup or documented retention option.

Before reporting a required infrastructure lane as blocked, check the configured runner and dev box and attempt the lane where its prerequisites are available. Resolve missing target details with the user. Record the target, tested revision, command, result and any concrete access or prerequisite failure. Report omitted PostgreSQL, broker, browser, UAT, Minikube/k3s and deployment checks explicitly. Disposable fixture results and live deployment qualification remain separate evidence.

#Verification lanes

The experimental MCP guide describes its OpenAPI parity, Go SDK and OAuth/AAI fixtures. pnpm verify runs the source checks; python3 scripts/mcp_broker_integration.py runs the broker's PKCE/consent/refresh journey and concurrent state consumption against restricted PostgreSQL on the configured runner. After pnpm --filter @pharos/mcp-broker build, node scripts/mcp_broker_browser.mjs uses Chromium to exercise the real broker callback and consent form through PKCE code exchange with disposable AAI/Party fixtures; the authentication workflow runs this lane.

pnpm changelog:check validates all pending changelog fragment names and content without Git history or service dependencies. The always-run policy group includes this check. scripts/test_changelog.py exercises real Towncrier assembly in temporary Git repositories, including preview, history preservation, consumed fragments and duplicate release rejection. See the fragment guide.

Distributed inventory/render/authority fixtures run in pnpm verify. They also reject an incomplete distributed child set, development AAI and missing external identity. Route-value tests cover the active profiles and checksums needed by the complete composition. Existing-cluster checks record API server/kubelet versions and reject component skew outside Kubernetes' supported direction. GPU Operator checks verify both upstream and installed NVIDIA archives, the reproducible single-member RBAC reduction, the disabled host-management boundaries, the manual-until-reviewed Argo policy and the absence of the optional Application from the dev-box root. The live adoption evidence, including both GPU profiles, is recorded in the GPU Operator ownership guide. Run uv run --project devops --frozen python scripts/gpu_admission_smoke.py on a Linux Docker host with kubectl, Helm and OpenSSL to qualify the chart hook RBAC and CRD admission. It starts a private API server, policy-status and role-aggregation controllers, and etcd in disposable containers. It checks admission compilation before RBAC, permitted GPU/Argo writes, denied foreign CRD writes, arbitrary-role delegation and policy revocation, plus recovery through cluster-admin without system:masters. Aggregated-role cases cover denied rule and membership-label changes, native aggregation and metadata-only Argo synchronisation. It then removes its containers and network. It never reads an existing kubeconfig. After changing the pinned GPU chart, review the reductions in devops/platform/scripts/platform_gitops/gpu_chart.py, run uv run --project devops --frozen python scripts/prepare_gpu_chart.py, then run uv run --project devops --frozen python scripts/generate_gpu_admission.py and review the exact role/binding constraints. Accelerator classes have one source, gitops/manifests/gpu-acceleration/classes.yaml: scripts/generate_gpu_offering.py --check proves the agent's offering components match it, and scripts/prepare_gpu_dra.py --check proves the vendored DRA chart's DeviceClass objects match it. dx/common/tests/test_gpu_acceleration.py also checks that no class shares a resource, that no rendered DeviceClass aliases nvidia.com/gpu, that only openst-gpu05 advertises whole cards, and that no node is served by both the DRA driver and the device plugin. The 0051 guide describes disposable Kubespray, cross-node storage/registry and network-denial qualification. Run scripts/distributed_edge_smoke.py in the frozen devops environment for real Debian HTTPS/SMTP DNS-failure isolation. Existing-site capture remains read-only; adoption fixtures exercise the full capture, approval, retirement and adoption sequence, stale reviews, identity changes and interrupted authority writes. Live identity, mail delivery and site cutover follow the complete composition.

scripts/test_platform_release.py checks deterministic OCI package bytes, complete image evidence, forward-only semantic tags, checksummed GitHub upload bytes and fenced Git promotion. scripts/test_hpc_site_release.py also checks the HPC archive name, explicit target architecture, artifact hashes and wheelhouse inventory. The release guide separates those checks from live publication, signature and Argo qualification.

Pharos DX tests cover diagnostic privacy, scoped credential reads, saved settings, source comparisons, Docker binding and blocked actions. Architecture checks reject webview polling, remote/clipboard access and credentials in shared state. pnpm verify builds the desktop frontend; native Rust tests, Clippy and installer builds can be run manually through .github/workflows/dx-desktop.yml; automatic pull-request and main-branch desktop checks are temporarily disabled. Playwright exercises a mocked IPC boundary. See desktop validation and credential checks for live lanes. Release regressions reject stable/malformed tags, mismatched package/lockfile versions, incomplete platform sets, altered installers and unsafe inputs. Tagged releases run native checks on all three OSs and UI tests on Linux before assembling all four installers and publishing their checksums together.

#Temporary fast CI

From 4 November 2026, automatic CI runs only the policy group (architecture, large-file review, frozen archive and changelog validation) and the CI selection, execution and temporary-mode regressions. These checks run in the scope job using Python's standard library, without dependency installation or remote success lookups. Both scope and ci-gate have a two-minute limit.

The full verification, infrastructure, retained Themelio and authentication jobs are temporarily skipped on pull requests and branch pushes, including reruns. The gate requires successful fast checks and explicit skips for all paused jobs. Paused groups produce no reusable success records.

Re-enable automatic full CI manually during the week beginning 9 November by setting env.CI_FAST_ONLY to 'false' in .github/workflows/ci.yml. There is no automatic expiry. Full CI remains available through manual source runs throughout the pause; local pnpm verify and infrastructure commands retain their full suites.

Draft pull requests skip automatic CI, including the fast checks. Marking a PR ready for review starts the selected checks; later commits and target branch changes run them again. Converting it back to draft cancels its current automatic run. Skipped draft jobs use a separate check name from ci-gate. Title and description edits leave current source checks running.

To run every CI group while a PR is a draft, open Actions → CI → Run workflow, select its source branch and leave warm_runners and publish_dev_images disabled. The equivalent command is:

gh workflow run ci.yml --ref <source-branch>

Manual dispatch qualifies the selected branch revision. Automatic PR runs qualify the merge result with the target branch. Each manual source run forces every verification, infrastructure and retained Themelio group, plus fresh authentication. Use Run workflow for this override; rerunning a skipped draft PR event retains its draft guard. Pushes to main and fresh continue to run CI automatically.

#Full CI

Full CI selects service verification, real infrastructure and retained Themelio by independent groups. scripts/ci_inputs.py declares each complete input set; scripts/ci_scope.py hashes the checked-out Git tree, including paths, executable modes, dependencies and the CI implementation. Unknown source areas invalidate all groups. Policy checks always run; repository tooling and DX retain broad inputs because their tests inspect source and documentation across the repository.

A group reuses success only when an immutable GitHub artifact identifies a successful CI run and job in this repository and its fingerprints match the independently read source tree. Pull-request evidence also proves the tested merge revision from the checkout action's own log. This permits reuse between branches without relying on GitHub's branch-scoped dependency caches. Missing, expired, failed, cancelled, ambiguous or inaccessible evidence causes fresh execution. Records expire after seven days and renew weekly. Manual dispatches and reruns that include selection force every group. Job summaries show selected groups, source runs and execution times. pnpm verify and pnpm test:integration still run their full suites by default; repeated --group options select named groups for diagnosis, and the documentation lane selects only the verification policy group. Integration groups share one disposable fixture per invocation, and browser/image checks start their heavier containers only when selected.

Every direct CI job uses a custom Linux runner. Verification, compatibility and infrastructure use portable pools with a best-effort ARM64 preference and AMD64 fallback. Their success keys describe portable Linux checks; records include the actual runner architecture for diagnostics. Native dependency and build caches remain local to each worker. ARM64 workers have disjoint roles: pharos-ci-control for selection and the gate, pharos-ci-general for source and compatibility checks, and pharos-ci-infrastructure for real fixtures. The infrastructure workers need at least 12 GiB each. The 2 GiB control worker omits the generic pharos-ci label, so older verification workflows cannot consume its reserved capacity. Runner roles documents validation, resource allocation, portable pool membership and assignment. Scheduling uses read-only job activity; uncertain availability selects the shared pool. Persistent workers use local uv and pnpm stores with frozen dependency installs, avoiding remote archive transfers for those stores. Go module and build caches remain local. Keep runner prerequisites and pinned tool versions aligned. The CI workflow's manual warm_runners mode preloads the active test workers' tools, locked dependencies, Chromium and missing fixture images through their normal job queues. Its downloads and cache checks produce no test success records. See runner cache maintenance. Ansible syntax checks use the same persistent, pinned collections as deployment tools, so checkout cleanup no longer discards those downloads.

The reusable .github/workflows/authentication.yml selects the custom AMD64 runner pool for assembled login/AUP and private-TLS qualification. Every source invocation requires both lanes afresh, independently of reusable source checks. The runner needs Docker, native PostgreSQL tools and Chromium system libraries. Frontend builds run before fixture URLs or credentials exist. A private, runner-scoped cache retains only Next compiler data, keyed by source, toolchain and build environment, with four-entry and 2 GiB limits. Both builds still execute; each invocation creates fresh databases, containers and browser sessions.

ci-gate accepts successful selected jobs and explicitly skipped reusable groups; unexpected skips, failed selection and failed fresh authentication fail the gate. Cancelling a workflow cancels its gate, allowing replacement work to proceed. PR title/body edits use a separate concurrency group and skip source checks with a separate metadata context; changing the target branch runs qualification.

Desktop packaging remains manually dispatched, with full native packages on each manual invocation. Linux image, desktop and platform release jobs use the dedicated AMD64 runner on metallo-vasilis-2; macOS and Windows use GitHub-hosted runners. Extend the input sets and scripts/test_ci_*.py when a check gains a dependency.

dx/common/tests/test_devbox_setup.py covers reviewed scope, reset confirmation and maintenance. test_setup_sessions.py kills real observer processes and reconnects to one fixture deployment, covering completion, failure, locks and target isolation. Runner-log tests cover disconnects, exit status, locks and IPC privacy. Incremental delivery tests cover ready layouts, stale reviews and Argo completion. Layout checkpoint tests allow kernel renumbering for a serial-backed by-id disk; target, machine, node, serial, size and owner must match. Other disks match exactly. UI tests cover commit/target review, cancellation, dirty source, active leases, expired reviews, and deployment progress, alongside real event-driven progress, reset review, changed destinations and warning wrapping at narrow tray widths. Live qualification requires a disposable personal dev box: review reset, reinstall, repeat setup, then qualify takeover against the resulting shared infrastructure. test_unchanged_delivery.py checks desktop/documentation-only delivery, catalogue fingerprints, deleted/renamed/executable inputs, missing history, application health, ownership changes and maintenance exclusion. The shortcut preserves the deployed commit and performs no transfer or cluster writes. UI checks cover this result and Windows-only WSL settings in both colour schemes.

dx/common/tests/test_workstation_env.py checks optional root .env loading, Make overrides, saved TARGET/SSH, nested targets and retained Docker endpoint/engine identity. It also checks that macOS engines can reuse the existing forwards without binding to a VM's bridge address, while Linux Engine retains its host-side bridge path. test_launcher.py uses a disposable Unix socket and a stub Docker CLI to verify custom socket mounts and reject remote or missing sockets without a live engine. Full Docker takeover requires live qualification. Colima passed reload, browser, connection-loss, retry and hand-back on 16 October 2026; see the profile and source baseline in the developer guide.

test_state_ownership.py and the launcher/runner tests cover identity, guarded legacy-state repair, linked state, retained modes and access ordering. test_ssh_credentials.py and test_ssh_bootstrap.py reject unsafe inputs and check trusted remote preparation, interrupted/concurrent retries and refusal before reset. On Linux Docker, run ownership_check.py, ssh_credentials_check.py and ssh_bootstrap_check.py under dx/common/tests/ with uv run --project devops --frozen python for caller/socket access, file-only mounts, legacy cleanup and same-host/separate-runner SSH. The SSH check builds from private (0600) source; PHAROS_SSH_TEST_ROOT selects its Linux fixture path. test_ssh_policy.py and test_ssh_tunnel.py resolve hostile forwarding defaults through real OpenSSH. test_browser_trust_tls.py uses real loopback TLS with an isolated hashed CA directory and no CA bundle. It covers lazy trust loading, installation and reuse, and rejection of missing trust, wrong hostnames and changed certificates; Windows and NSS store checks are stubbed without changing workstation trust. Run uv run --project devops --frozen python dx/common/tests/browser_access_check.py --ssh VM --ca PATH to qualify public HTTPS, management denial, reverse-forward suppression, reuse and reconnect on macOS/Linux/WSL2. dx/common/tests/test_takeover.py checks workload selection, delivery-only changes, namespace-qualified dependencies, scoped native launch environments, retained Next.js preparation, process ownership and recovery that preserves other services. test_takeover_delivery.py checks that make deploy and make reconcile hand back a takeover and resume it only after a successful delivery. test_lease.py checks fenced ownership, renewal, expiry and retry; an expired holder cannot renew. Live takeover qualification must also exercise reload, AAI browser login, multiple services, interrupted hand-back and loss of the connection/guardian against the owned dev box, for each supported runtime. Workstation state must be usable beside inaccessible administrator deployment state, with private directory and file permissions. test_web_readiness.py uses a real HTTP server with a green health endpoint and cold entry pages. Native and Docker readiness must warm the retained dashboard and sign-in page, accepting authentication redirects without following them. The browser login helper allows 90 seconds for navigation through cold pages. The local AAI bootstrap regression verifies that internal network edges precede administration jobs even when the gateway already isolates Keycloak ingress. Provider-token tests also defer delivery until the first root sync creates its GitLab and Waldur Applications. They require successful completion of provider PostSync hooks before writing credential-bearing users or tokens. Explicit sync regressions preserve the Application's bounded retry policy after a transient child startup failure. Rendered Waldur seed checks reject missing database Services and StatefulSets or ones ordered after the restore hook, including on an empty cluster. With Tilt 0.37.8 installed, qualify the root dashboard's initial takeover, source save/revert, manual hand-back and log/status actions. Deploy and reconcile must never run automatically on startup or a source save.

The reusable k3s acceptance command is uv run --project devops --frozen python dx/common/scripts/qualify_takeover.py --target dev-NAME --ssh VM --runtime native --browser; repeat with --runtime docker. It checks all seven runtimes, repeated preparation and reload, both frontend HMR connections without navigation, retained AAI login, catalogue, logout/session replay, connection-loss shutdown, retry and Argo hand-back. Use a returning test account whose policy the user has already accepted. This checks canonical session cookies and local revocation; it never accepts policy terms or provider invitations. The dev-box API-auth overlay supplies the retained frontend's public HMR route on both architectures. test_devbox_auth_hmr.py checks its exact route and read-only mount, with unchanged legacy and non-dev-box API-auth rendering. The real authentication proxy smoke lane checks WebSocket frame delivery and rejects private, foreign-host and unrelated upgrade routes. test_runtime_auth_files.py and test_proxy_takeover.py cover exact credential inventories, unchanged default profiles and guarded proxy Service recovery. test_docker_auth_hosts.py checks the optional API-auth issuer's exact mounted snapshot, public hostname routing, rotation and credential privacy, while preserving profiles without API-auth and direct issuer URLs. test_docker_relay.py checks the Linux Engine relay's reverse-only forwards to the takeover's own public-only tunnel, its isolated SSH options, literal state paths, pinned host key, reuse, ownership and hand-back cleanup. test_takeover_reconcile.py checks that the relay starts after the journal and that a Telepresence session on a former relay is replaced. test_file_device.py covers file-device directory permissions, immutable udev rules and stable-path registration. See the prerequisites and commands. dx/common/tests/docker_credentials_check.py separately exercises real Docker file binds across repeated exports and rotation. dx/common/tests/docker_relay_check.py runs the relay with real Docker and OpenSSH: another container on the default bridge reaches its two reverse forwards, and it refuses other listeners, local forwards and sessions. dx/common/scripts/qualify_gitlab.py --target dev-NAME --git-origin themelio-compat checks the advertised clone origin, including LFS and role restrictions. These checks intentionally exclude legacy CPU execution, which is deferred to the new backend. They do not establish immediate GitLab impersonation-token rejection after logout.

Use the smallest sufficient boundary. Pure policy belongs in deterministic native unit tests. PostgreSQL transactions, broker acknowledgement, browser delivery and runtime deployment need real dependencies. Do not report a skipped lane as a pass.

pnpm check:go runs the shared API and Beto checks: formatting, architecture, Go vet, pinned Staticcheck and errcheck. pnpm check:api and pnpm check:beto select one service. The Docker Go toolchain uses /cache/staticcheck for its Staticcheck cache, alongside the existing module and build caches, including when it runs as the developer's UID. pnpm verify runs the shared checks and Themelio's frozen Python dependencies, Ruff, strict Mypy, pytest and wheel checks. Negative fixtures prove that the shared Go checks reject unformatted source, invalid calls and unchecked errors.

pnpm test:project:postgres runs Beto's command foundation, Party and Project against a private native PostgreSQL cluster; initdb, pg_ctl and psql must be on PATH. Restricted owner roles qualify atomic receipts, snapshots, publications, observation leases, nullable unresolved members, rollback and replay with race checks. The lane removes its cluster after execution. The shared beto infrastructure group runs the cleanup SQL with psql inside its disposable PostgreSQL container; its host needs no PostgreSQL client. Native PostgreSQL lanes use their native client.

pnpm test:project:broker checks assembled Beto, current Party authority, scoped queries, independent public observers, removal and replay against disposable RabbitMQ and PostgreSQL. pnpm test:project:source checks verified TLS/STOMP with native Waldur headers and restricted credentials against another disposable broker. The migration record records coverage and live checks.

python3 scripts/ssh_access_ldap_integration.py builds a disposable OpenLDAP container with the openssh-lpk schema and the ou=pharos service ACLs, then runs the SSH access agent against it with test Site doubles. It requires a running Docker engine; pnpm test:integration includes it. It does not establish SSSD, sshd or Site API behaviour, which the Icarus-hpc laboratory qualifies.

check:beto additionally runs historical wire/HTTP compatibility against adb80196ea1c, and Go architecture tests enforce command capability boundaries. Generator fixtures compile evolved input/response models and command bindings; negative fixtures cover presence, ownership, subscriptions and compatibility.

Beto also checks generated-contract drift. Its contract checks compare all files in the selected output families, including removed outputs. pnpm generate:beto --category requests regenerates its query bindings independently. See the adaptor map.

With Go installed, run the API and Beto isolation tests without RabbitMQ or Docker from the repository root:

PHAROS_GO_PROJECT=services/api python3 scripts/go.py test -race -count=1 ./adaptors/http ./adaptors/messaging
PHAROS_GO_PROJECT=services/beto python3 scripts/go.py test -race -count=1 ./contexts/project/... ./adaptors/messaging/requests

python3 scripts/project_broker_integration.py uses disposable RabbitMQ and native PostgreSQL/Go for API-to-Beto delivery, caller isolation, signed Party registration and publication. It removes its owned resources. Beto provisions the scoped Metallo callers and queues; the receiving projections remain in issues 757 and 780.

python3 scripts/api_native_sessions.py checks canonical sessions, current authority, logout races and revocation against restricted Valkey 8.1.3 with AOF. python3 scripts/themelio_party_integration.py checks durable account work, credential recovery and logout against native Valkey. Live AAI, browser login/AUP and deployment require separate qualification.

pnpm test:authentication:e2e qualifies API-owned signup, GitLab account work, onboarding recovery and retained AUP through the production service entrypoints and Chromium, with real PostgreSQL, RabbitMQ and durable Valkey. OIDC and GitLab upstreams are fixtures. Chromium uses the production platform web for login, Project reads and logout, and same-origin browser API calls for onboarding resume and AUP acceptance. The unrelated SSH Sites panel receives an unavailable fixture response because this lane does not start Metallo; SSH access and full-stack realtime remain covered by their dedicated lanes. A second, independently provisioned fixture runs the retained Themelio browser journey: sign-in, explicit account recovery, retry with the original command identity after a lost response, AUP acceptance, server-rendered Project data, dashboard access and logout replay. Both fixtures run the HTTP authority and returning-login checks. uv run --project devops --frozen python scripts/authentication_proxy_smoke.py separately checks the deployed proxies with real private TLS. See the prerequisites and live cutover checks. The live Keycloak/GitLab journey uses a disposable provider identity, the deployed browser endpoints and independent GitLab state checks, followed by guarded fixture cleanup. API's HTTP tests inject a fake query client and validate the legacy JSON response against OpenAPI. Its messaging tests feed synthetic deliveries into reply correlation and failure handling. Beto's tests run the real Protobuf decoder, query registry, handler and response encoder with an injected read repository. These test each service independently; they do not emulate a RabbitMQ server or connect both services. The real broker lane checks permissions, routing, acknowledgements, redelivery and the API-to-Beto round trip. The focused python3 scripts/beto_broker_integration.py lane uses disposable RabbitMQ and native Go to check overlapping API connections, reconnect and caller queue isolation. It activates the query_fixture test owner from scripts/fixtures/ beside Project in a temporary tree, generates its request Go there and builds Beto with a fixture module, so one Beto process answers both owners' queries. One API client asks both owners; Beto-side tests refuse foreign-owner, wildcard and undeclared-command access. Its fixture credentials and container are removed afterwards; it requires a running Docker engine and the native Go toolchain. scripts/test_beto_second_owner.py runs in pnpm verify and proves generation, drift checks and provisioning for the same owner without RabbitMQ.

pnpm verify checks architecture ownership and inward imports, fault-injection fixtures, large-file review, Go formatting/race tests/vet/build for both Go modules, frozen Python dependencies/Ruff/strict Mypy/pytest for Metallo and Themelio, Next.js types/build, both Kubernetes overlay renders, workload credential/configuration checks, pinned CI actions and the k3s role's Ansible syntax check with pinned collections. The archived main tree is verified byte-for-byte, including executable modes.

Themelio's technical fixtures use the same disposable PostgreSQL instance as the other lanes, with their own logical database and runtime credentials. Its immutable delivery migrations are checked separately from Beto's. Real broker tests exercise two overlapping callers, connection replacement, durable replies and actor-scoped command replay. The production context, command, query and subscription registries remain empty. Native takeover launcher checks cover the Python factory; live k3s and Colima qualification belongs to the shared infrastructure work in PR #697.

Metallo launcher contract tests run the host's /bin/bash on macOS and Linux. They cover optional repository mounts, paths with spaces and resolved symlinks. Docker calls are stubbed; the tests require /var/run/docker.sock to exist as a socket and report a skip when it is absent. The supported workstation and SSH runner workflows are described in the development guide.

The Metallo package check builds and extracts its wheel, then imports the Site Runtime worker, native adaptors and process entry point independently of the editable checkout. It also verifies that the repository preparation helper is packaged. Context registration tests keep Site Runtime’s SQLite journals out of platform PostgreSQL provisioning and migration enrolment.

The Go and kubectl setup actions use Node.js 24. Persistent self-hosted CI workers reuse their local Go module and build caches; setup-go archive caching is disabled to avoid restoring over existing read-only module files.

pnpm test:integration uses an isolated Docker project, generated from the Kubernetes workloads. Ports, credentials and technical fixture identities are unique; its database is never the development database. Cleanup removes only that owned project and the pharos-test/* images it tagged. --keep retains both for diagnosis. The built database-admin image bootstraps every context database twice, with the provision Job's command, so each owner's migrations are rendered from the packaged files. The Execution Core lane uses its real private database and runtime role. It reloads saved Runs and tests all six 1C messages with production codecs; provisioning, permissions and GitLab credential mocks exist only in tests. Launch acceptance adds Workload lookup mocks and real decision/receipt persistence, concurrent retries and rollback. Launch tests verify that saving creates no browser-notification rows and that a missing projection registration still fails and rolls back. Planning tests reload accepted launches, create Runs, encode commands, exercise concurrent retries and roll back failed writes; Workload/provisioning/permission lookups remain mocked. Launch processing adds the real PostgreSQL claim loop, atomic acceptance hand-off, fenced Run completion, restart recovery, retry timing, deadline checks and competing-worker tests. The Site Coordination lane uses a separate private database and restricted role. It carries commands built from saved Runs through ordered storage, paginated reads, simulated Runtime results and result reload, using production codecs. It verifies concurrent sequence allocation, replay/conflicts, gaps, environment and database isolation, corrupt state and transaction rollback. Runtime and supervisor callers are test fixtures. The Site API PostgreSQL lane adds real HTTPS delivery, result replay after API restart and rollback, using a test-only identity authority. Runtime recovery extends this lane with the production worker and real SQLite: resend exact outcomes after lost requests/responses, recover uncertain native effects through a test executor, and reject unexplained progress. SQLite unit tests cover corruption, atomic rollback and competing processes. Execution lifecycle cases build commands from saved Runs, execute them through the common Runtime, then apply recorded results through Execution Core's owner port. All three families run in both environments with real PostgreSQL, HTTPS and SQLite. Native execution, provisioning and permission authorities are test doubles. The cases preserve preparation bindings, reload each state change, replay results, reject competing versions and verify rollback when projection encoding fails. The generic technical fixture stays outside product contexts. Reliable supervision adds the production polling loop, fenced leases, durable command recovery, command and Attempt retries and an atomic completion event defined under Metallo's own contracts. Both environments complete directly from their native observations. PostgreSQL cases cover competing workers, stale snapshots, lease expiry during writes, immutable evidence, migration enrolment and transaction rollback. Metallo's request lanes exercise the transport with a test-only owner, then the metallo-requests composition with a caller holding the API's broker role: running an action through real Workload preparation and launch acceptance, its retry and parameter errors, Run listing and reads, a stop and the actions list. The repository directory, actions.yaml and permission owners are test doubles.

The same suite then runs scripts/metallo_beto_integration.py with Beto's real composition and Metallo's enrolled callers, checkpoint synchroniser and Run access adaptor. It checks all ten current-person capabilities, owner authority fields, pending evidence and renewal, revocation before local delivery, repair and rebuild. Source acquisition and identity proofs use fixtures. This lane uses only the suite's disposable databases and broker, before service images claim their queues. See the Run access checks for the boundary with live-provider and native execution qualification.

make metallo-access-test TARGET=dev-NAME builds the selected cluster's native image and runs the real Beto composition and Metallo access consumers against isolated PostgreSQL and RabbitMQ. It checks exact capabilities, checkpoint reconciliation, deleted membership repair, equal-revision Party/binding repair, pending-source deferral, renewal, denial before local retirement delivery, the retirement event's application before checkpoint repair, and rebuilding afterwards. It also checks independent authority revisions and expiry, refusal during recovery, Source Control's complete owner snapshots, and Resource Provisioning's Site roster from Beto-published facts. Pending authority must reach Execution Core and Resource Provisioning through their dedicated consumers before checkpoint repair. Provider acquisition and identity verification use labelled fixtures. Separate PostgreSQL recovery probes use query doubles and scoped authority fault notices. This lane does not qualify live Waldur, browser access or accepted-work execution. Installed workers remain unchanged. A successful run removes its namespace; KEEP_ACCESS=1 retains it. Inspect or clean a retained run with make metallo-access-inspect or make metallo-access-clean, the same TARGET, and its printed ACCESS_NAMESPACE. The private target state contains the report and test log.

make metallo-identity-test TARGET=dev-NAME SSH=user@host KEEP_IDENTITY=1 runs the Execution Core, Site Coordination and Pipeline Coordination unit and PostgreSQL cases inside the actual dev-box cluster. It creates an owned temporary namespace with separate PostgreSQL and a test Job; the installed application's database and configuration are unchanged. Provisioning, permissions and credentials remain mocked only in test tooling. The output prints guarded inspection and cleanup commands. A successful run establishes the internal 1B/1C/3A/3B/3C/3D/4A cluster lane: saved Run identity, all six execution messages, permission and credential decisions, Project resource lookup launch acceptance, environment-based planning, durable launch processing and ordered command/result storage. Workload revision lookup is also mocked. PostgreSQL and message codecs are real; external connections use test doubles. Browser/native launch integration remains a separate lane. Pipeline cases exercise internal start/read/ cancellation through production adaptors, snapshot and receipt reload, concurrent versions, corrupt-state guards and atomic pending-work rollback. The Pipeline's Workload and permission ports use test-only providers. The SSH target validates the installed dev box through its owned controller namespace or its root-owned normal-installation marker, then pins the single node identity. Completion of this lane qualifies 8A records, while step execution remains 8B–8D.

make metallo-site-api-test TARGET=dev-NAME SSH=user@host KEEP_SITE_API=1 runs the Site HTTPS API and Runtime client in another isolated namespace. It uses real PostgreSQL, TLS, projected ServiceAccount tokens and Kubernetes TokenReview. The API is the installed GitOps Site API, changed only in namespace, image, grants and database label, with a certificate from the installations' issuer; the Jobs, services and pipeline lanes reuse it. scripts/test_site_api_deployment.py checks every profile's rendering. The API receives only its restricted context database credential; Runtime probe Jobs receive their own identity token and TLS trust. The controller owns and cleans up the temporary TokenReview RBAC. Full native lifecycle execution remains simulated; the Slurm environment is exercised with a second Kubernetes ServiceAccount. Each environment also has a private SQLite journal on a PVC. Replacement Runtime Jobs exercise process crashes before sending saved results and after server commit, followed by an API restart and exact result replay. A durable test executor detects repeated simulated native effects. This verifies the common recovery path; actual native recovery belongs to the Kubernetes/Slurm adaptor tests. The same lane then exercises preparation, submission, active and terminal observations for jobs, sessions and services. Test callers invoke the production lifecycle application between Runtime phases and reload its saved Run. The recovery and lifecycle cases check 38 command/result pairs. Automatic supervision adds six Runs across both environments, bringing the lane to 66 command/result pairs. It kills a supervisor after sending a saved command, replaces its expired claim and verifies exact command reuse. Final checks require terminal Run state and typed completion events at the same aggregate version. Provisioning, permissions, input materialisation, retry admission and native actions use test doubles.

The lane then runs three Kubernetes member-access probes and six real Jobs in two additional disposable Project namespaces. It uses the committed RBAC/admission profile and a projected Runtime credential to verify member starts, direct-request denial, Project-specific revocation and Job-count quota rejection. Provisioning bindings and platform permission responses remain mocked. The administrator installs and revokes grants outside the Runtime Pods. Owned namespaces and admission policies are cleaned on success; failures retain them for inspection through the same Site API cleanup command. Live provider revocation remains an integration task.

make metallo-site-api-jobs-test TARGET=dev-NAME SSH=your-alias KEEP_SITE_API=1 qualifies the permanent Kubernetes Runtime Deployment in a separate isolated installation. It runs the production supervisor through saved Run completion, replaces a Runtime Pod after real Job creation but before saving its result, and checks the unchanged Job UID/PVC and absence of duplicate Jobs. Further probes check native failure without retries, CPU quota blocking/release and cancellation after member grant removal. Direct requests with the Runtime credential, as a second member of the Project and without a member, cannot reach the first member's workspace, credential, Pod or logs. PostgreSQL, HTTPS, TokenReview, SQLite, Kubernetes Jobs/RBAC/admission and production codecs are real. Provisioning, platform permissions and immutable source/artifact resolution are explicit test doubles. Run completion has no metering provider dependency. Reports use the existing private Site API evidence directory and cleanup commands.

make metallo-site-api-services-test TARGET=dev-NAME SSH=your-alias extends this isolated installation with actual Deployments and KServe/Hugging Face prediction. The lane reuses the ready managed KServe installation on current dev boxes and excludes its controllers, CRDs and runtime from cleanup. On older, unoccupied AMD64 clusters the pinned disposable installer owns these resources. An incompatible existing installation is preserved and reported. It exercises same-Pod init commands, private repository PVCs, native recovery, revocation and authorised application/prediction/file/log reads. It replaces the Runtime after each kind's first real create and before its result is saved. Server-side dry runs with the Runtime credential check cross-member and cross-Project denial in a second Project. A count quota blocks both kinds' creates until it is released. A crashing application, an unloadable model and an application that crashes after becoming healthy, and is ready again for a while after each restart, must each fail within 420 seconds, under the Deployment progress deadline, because their containers pass the Runtime's restart limit (#1054); the report records how long each took. Terminal failure requires their native workloads and endpoints to have been removed. PostgreSQL, Site HTTPS, TokenReview, SQLite, Git/LFS transport and Kubernetes/KServe are real; provisioning, platform permissions and repository-grant issuance remain test doubles. Record manual resumption separately from a successful uninterrupted qualification. The installer selects native ARM64 or AMD64 serving images. The same lane starts Gradio and Streamlit with the image's offline wheels and checks cancellation while PVC-count quota remains exhausted. Image sources and dependency maintenance are in the workload image guide. Reserve at least 3.5 schedulable CPU cores and 6 GiB memory beyond the cluster's existing requests. The lane releases the initial services before the failure pair and waits for failure cleanup before the quota pair, limiting concurrent service requests to three cores without changing their resource contracts.

make metallo-pipeline-test TARGET=dev-NAME SSH=your-dev-box runs the isolated Kubernetes pipeline lane through the hosted production coordinator, launch and supervision loops, Site API and Runtime. It uses real PostgreSQL, RabbitMQ and the installed GitLab, then exercises Jobs, application/KServe readiness and prediction, failure, cancellation and process replacement. Owner-provider doubles and the test-only GitLab TLS bridge are recorded separately. Scenarios, retained evidence and cleanup.

Bind-mounted secret and configuration directories have explicit 0755 modes and their files are read-only (0444) so container users can access them under restrictive host umasks. The enclosing disposable project stays private (0700) and its credential configuration stays 0600. The deterministic mount regression runs in the pinned devops Python environment as part of pnpm verify. Runtime images must also make copied Python source and Nginx configuration readable by their non-root users when a CI checkout has 0700 directories and 0600 files. Image qualification must exercise those restrictive build-context permissions.

The PostgreSQL lanes exercise rollback after encoding failure, state/receipt/ outbox atomicity, stable retry outcomes, conflicting input, concurrent expected versions, consumer receipt conflicts, single-root write guards, fenced dispatch, immutable evidence and denied runtime DDL. The Python lane also proves atomic realtime bytes and reclaimed-lease fencing. The broker lane proves actual typed Protobuf transport, mandatory confirmed publication, retry convergence and denied runtime topology mutation. All six images are built and started from the same Kubernetes workload configuration, including the technical realtime gateway.

Publication retry checks cover persisted exponential deadlines across worker replacement, jitter bounds, stale-lease rejection, unchanged bytes, reply expiry and fresh work progressing beside delayed records. Go and Python worker tests exercise cancellation and sustained connection failures without sleeping. Real RabbitMQ checks confirm that an unroutable mandatory publication leaves its channel usable for the next routable message. The policy is recorded in ADR 0046.

Chromium runs in the pinned Playwright image with the real API, Valkey and Centrifugo. It proves two independent authenticated windows, a live Protobuf connection, revocation across windows sharing one session, reconnection for another valid session, and mobile layout. A browser-routed AAI fixture separately checks the top-level login return, canonical session read, generated member Project pagination, empty membership, unavailable authority, expiry and logout without polling. Real PostgreSQL and broker tests qualify Party isolation, pagination and retired-binding denial. pnpm test:browser targets an already running local stack. It needs the private .local/dev.env access code, or explicit BASE_URL and OPERATOR_PASSWORD environment values.

Session writes retain appendfsync always. The API allows up to ten seconds for a Valkey reply, including its durable acknowledgement on a dev box's Ceph-backed volume; dial and socket-write limits remain three seconds. The wire regression delays a transaction acknowledgement past the previous three-second read limit. Browser checks assert login's HTTP success before waiting for realtime, with a 15-second assertion and 90-second scenario budget. These bounds allow slow storage without treating an unacknowledged write as success.

pnpm generate:contracts reproduces Go/Python envelope bindings, Python type stubs, strict Python Site execution/coordination models, the assembled Go OpenAPI document, the browser's Orval clients and migration checksum metadata. pnpm check:http compares the complete generated browser tree, including MSW handlers, Faker factories and typed fixtures. CI checks generated drift. Add the smallest test that distinguishes a failure from correct behaviour; in-memory policy evidence cannot establish transactional or recovery guarantees.

Minikube/k3s rollout, NetworkPolicy enforcement, PVC recovery, operator UAT, remote CI, provider integration and production deployment are separate evidence. Report them explicitly until exercised. Execution Core has internal behaviour examples; its user-facing launch and native execution UAT remain future work. Future contexts must add owner-native business examples, transaction/broker regressions and their browser recovery scenarios before hand-off.

Development GitOps adds scripts/check_platform_gitops.py, exact source/health and ownership rejection fixtures, and strict catalogue renders to verification. dx/common/tests/test_argo_health.py executes the deployed Lua health rule through the pinned test-only Lua 5.1 runtime and compares its readiness with the CLI and setup agent, including error conditions, recovery and stale source comparisons. On the owned k3s VM, qualify first adoption, unchanged public deploy, drift, interruption/retry and explicit rollback/restore. Check each selected Application at its delivered commit and compare retained identities/hashes. Extend scripts/probe_platform_mesh_network.py when adding a management backend: rendered default-deny cannot establish isolation when another policy admits the same pod. The Argo server's upstream allow-all policy is replaced under the same resource name; foreign-pod and forged-header checks cover the server and Git transport. Development qualification and automated browser login do not replace manual UAT, fresh-VM/reboot, Minikube, shared/staging or production deployment evidence.

uv run --project devops --frozen python scripts/gitops_hooks_smoke.py checks setup hook delivery with the pinned Argo controller and repository server on a Linux Docker host. It uses the disposable API fixture, garbage collection, Redis and a temporary Git repository. Sync and PostSync ConfigMap receipts demonstrate fresh setup, skipped hook-only changes, recovery at a new revision, changed inputs at the same commit and unchanged repeat delivery. It removes its containers and network afterwards and reads no existing kubeconfig. This qualifies hook orchestration; provider bootstraps and fresh dev-box installation remain separate lanes.

GitLab and Waldur (DD-0041) are qualified on the dev box. METALLO_TARGET=dev-NAME dx/common/scripts/service-login.sh gitlab|waldur [user] signs an AAI test user in through Chromium, trusting only the AAI CA, and checks the service's API reports that user. METALLO_SSH=<vm> python3 dx/common/scripts/qualify_gitlab.py --target dev-NAME checks roles, HTTPS Git and a 30 MiB LFS round trip with throwaway users, which it removes, and runs Git without credential helpers. METALLO_SSH=<vm> python3 scripts/qualify_waldur.py --target dev-NAME signs in as throwaway users through Waldur's REST API and checks the two Project roles, invitations, membership events and the limits of a worker's user, then deletes what it created; it changes no Waldur setting. It also signs in as the local administrator with the generated password and checks the account. --recovery and --rotation also change that password, sync the Waldur Application and check that the generated or the new password works (about two minutes each); if either stops half-way it puts the generated password back, which test_qualify_waldur.py checks with test doubles and no dev box. The network probe covers both services and Waldur's management-only /admin/. check_platform_gitops.py checks that the Waldur seed Jobs and make deploy name the same seed file, and that the Waldur AAI client identifies a person by the subject. METALLO_SSH=<vm> python3 scripts/qualify_gitlab_admin.py --target dev-NAME signs in as the GitLab local administrator (root) with the chart's password and checks the account. --recovery, --sync and --lockout change that password or lock the account, and set it back at the end. Source Control's GitLab slice (DD) is qualified on the dev box with four scripts, each run as METALLO_SSH=<vm> uv run --project services/metallo --frozen python scripts/NAME --target dev-NAME. They set up what installation administration leaves (an administrator-made root group and a plain worker that owns it, with a one-day token), run the production worker, reconcile and handlers, and remove everything they create (pharos-recon-*). qualify_source_control_groups.py covers the worker and reconcile: what a plain worker may not do, groups, roles, repeats, removals, hand-made changes, renames and awkward names. qualify_source_control_handlers.py runs the handlers over in-memory stores and a real GitLab: duplicate events, a lost answer and a restart, stale evidence, disabled Parties, retired bindings and closure. qualify_source_control_creation.py checks repository creation roles and scoped Source Control deletion, including Maintainer denial, Project mismatch and safe retries. qualify_source_control_layout.py asks the real GitLab for the kind subgroups of a new Project group: which exist, whether a repeat adds any, whether one deleted by hand comes back, who may create repositories and subgroups in them, and whether levels changed by hand are put back. qualify_source_control_edges.py covers a deleted group, 101 members, a revoked and renewed token, a member who already owns the root group, and two concurrent runs; it makes 101 users, so its clean-up is slow. scripts/cleanup_source_control_qualification.py --target dev-NAME removes whatever an interrupted run left. scripts/test_qualify_gitlab_admin.py runs the script's Rails runner locally against a stand-in gitlab-rails and checks that its temporary file is private, unique and removed after a failure, that the runner's failure is reported without its output, and that --target is required. METALLO_SSH=<vm> uv run --project services/metallo --frozen python scripts/qualify_source_control_repositories.py --target dev-NAME builds a throwaway two-Project layout in GitLab and reads it through Source Control's GitLab adaptor with a read-only group access token: canonical IDs by path and clone URL, refs and digests, a moved branch, rename, transfer, deletion and refused writes. It waits through passing GitLab outages on a slow dev box and removes its groups and tokens through the toolbox. METALLO_SSH=<vm> python3 scripts/qualify_waldur_accounts.py --target dev-NAME checks that Waldur finds a person by the subject of the sign-in, with a simulated Pharos writer and simulated Keycloak claims, and how its invitations behave; it stops, with a note, when the installed provider does not use the subject. METALLO_SSH=<vm> python3 scripts/qualify_llm_gateway_keys.py --target dev-NAME runs the reconciler, issuer and usage keys that the Metallo pod mounts against the real LiteLLM and checks that each reaches its own routes and is refused every route that would mint a key, create a user or a team, change a team or undo a block. It prints only statuses. METALLO_SSH=<vm> python3 scripts/qualify_llm_acceptance.py --target dev-NAME is the shared LLM access acceptance run (epic subtask 6B). It orders two throwaway allocations and takes the dev-box's disposable test users through no Project, one, two and a leaving member: chat is used by signing in through the AAI at the public chat host, the personal key is the one LLM Access mails and is used at the public API host. It checks what each person sees and may do, and that a member who leaves loses chat and the key, and removes its allocations and the users' Open WebUI accounts, also after a failure. It drives the Site Agent's member routes, so chat membership must be site-agent. The exhausted-allowance case waits for strict allowance enforcement (5A) and is reported as skipped. dx/common/tests/test_qualify_llm_acceptance.py runs the scenario against a stand-in platform, checks that leaks of chat or key access fail it, that clean-up always runs and that the report names no session or key. METALLO_SSH=<vm> python3 scripts/qualify_site_agent.py --target dev-NAME qualifies the cloud offering and the Waldur Site Agent (project provisioning 5.1): a throwaway Project's member orders a resource and staff approve it, and the script checks the Kubeflow Profile namespace, its quota and Site Runtime set-up, member accounts and their bounded access, limit changes, usage, the storage quota, the agent's single Waldur role, the agent's admission policies (server-side dry runs as the agent, including Runtime's grants or another member's impersonation bound to a member account, and Profiles outside its shape or another owner's, for which it briefly creates a foreign Profile), an agent restart while an Order is pending, and termination. It needs the installed Site Runtime (its account pharos-site-runtime/runtime-k8s), stops without it and, also after a failure or an interrupt, terminates its resources and deletes its organisation, Project and users. A run takes about fifteen minutes, as the agent polls Waldur every minute and reports usage every five. dx/metallo/tests/test_qualify_site_agent.py checks with test doubles that the clean-up always runs and cannot be interrupted, that --target is required and that an admission refusal is told apart from a missing permission.

pnpm check:large-file-review enforces the large-file policy and rejects missing, duplicate and obsolete entries in docs/large-file-review.json. scripts/test_large_file_review.py covers the size boundary, automatic exemptions and register validation. The legacy archive retains its separate byte, mode, inventory and relocation-history checks.

make devbox-storage-test TARGET=dev-NAME is the dev-box storage lane, and make deploy runs it after every install. It writes and reads a token through a ceph-block volume, a shared ceph-filesystem volume and S3 on a ceph-bucket bucket, in an owned namespace with restricted Pod security, then deletes it (KEEP_STORAGE_PROBE=1 keeps it). dx/common/tests/test_storage.py covers the disk binding, the controller inventory, the stages of make deploy, that a first deploy without its disk or VM changes nothing, that setup and Keycloak wait for storage, the playbook's disk guard and that the copied role keeps the final deployment's defaults.

The local AAI has live identity probes against its deployed Keycloak. The platform edge has negative catalogue fixtures and real OpenSSL tests for CA preservation and certificate retries. A real Ansible run checks that a renewal whose validation or reload fails is reloaded by the retry, and that nothing pending causes no reload. uv run --project devops --frozen python scripts/platform_edge_smoke.py checks the rendered Nginx routes with real TLS and a simulated gateway in a disposable container, for the platform's hosts and a fixture laboratory's own routes, including management and unknown-host denial. Live AAI/Icarus migration remains a separate lane. uv run --project devops --frozen python scripts/kubeflow_auth_smoke.py runs the pending Kubeflow route through rendered Nginx, pinned oauth2-proxy and pinned Istio Envoy containers. It signs local OIDC sessions with missing, empty and issued site_username claims, checks forged identity headers on login and WebSocket paths and all dev listeners, and exercises the dashboard's gateway-principal and identity requirements over mTLS. Both checks run in CI's infra-edge group. The fixture uses static Envoy listeners for the reviewed policies; live GRNET, Kubeflow browser journeys and Kubernetes deployment remain separate lanes. scripts/check_platform_mesh.py and the generated chart schema enforce known backends and host-scoped policies, including when Helm is invoked directly. scripts/qualify_platform_mesh.py compares the issuer, CA, node and existing pod/PVC/Secret identities and material without retaining secret values. scripts/probe_platform_mesh_network.py runs a new restricted, tokenless Pod in an owned temporary namespace: direct gateway/backend access must fail, the public issuer must work, and forged forwarding headers cannot admit management access. The namespace is removed afterwards. Run these on the qualified dev box alongside the Icarus-hpc and mixed Pipeline lanes. dx/common/tests/test_local_aai.py covers per-target binding, the controller's credential refresh and SSH configuration, the staging attribute and claim shape, and the client registry: its validation, exchange only for declared resources, and the deployment variables written under the vault's names.

The Icarus-hpc laboratory has one live DD-0055 e2e lane for membership, directory keys, SSH, revocation, termination and Project departure. make metallo-pipeline-mixed-test TARGET=dev-NAME SSH=your-dev-box adds the real mixed Slurm-to-Kubernetes lane with Git LFS evidence. Run both against the owned dev box and retain evidence without credentials; mocked provider tests do not qualify the live run path.

Shared dev-box infrastructure: scripts/check_shared_infrastructure.py renders the provider and consumer manifests; test_shared_infrastructure.py includes negative credential, management-port, broker-scope and Valkey command/prefix cases. Incomplete records must fail before generation or writes. Tests require session-worker ZRANGE and reject consumers changing published Secrets. On a dev box, change a copy and sync pharos-infra: its recorded value returns and the consumer keeps working. Qualify PostgreSQL cross-database denial, Valkey cross-prefix/admin denial and Waldur event-vhost lifecycle on real providers before fresh k3s and takeover tests. GitLab database initialisation has focused interruption and retention regressions in dx/common/tests/test_gitlab_database_initialisation.py. Run uv run --project devops --frozen python scripts/check_gitlab_database_initialisation.py on the configured Linux runner for disposable PostgreSQL checks of one-use creation, retry, data preservation and refusal before credential changes. The GitLab recovery guide separates these fixtures from operator-led live reconciliation.

Themelio's empty composition tests verify its authenticated operational surface, absence of business endpoints and external connections, partial-startup cleanup, and worker draining before resource disposal. Contract tests reject unregistered messages, requests and browser projections. The wheel check imports its executable composition and delivery adaptors independently of the editable source. pnpm generate:contracts reproduces its service-local wire types, plain shared models, response encoders and empty registries.

pnpm test:integration runs Themelio's technical PostgreSQL and RabbitMQ fixtures after the older service lanes. It applies the checksummed delivery extension twice to the disposable fixture and checks rollback on encoding failure, stable retries, conflicting input, concurrent expected versions, exact command replies, realtime lease fencing, denied runtime DDL/topology mutation and typed broker retry delivery. No fixture is a registered product context. Image and takeover qualification must also exercise Themelio's Python runtime with restrictive build-context permissions.

#Documentation site

Run pnpm docs:check to assemble authoritative sources, extract code examples, build the static site and check internal navigation and build metadata. Run pnpm docs:test:browser after the build for browser navigation, compatibility notices and diagram rendering. These checks use local static files and require no platform databases or deployment. See the site guide.