# DD — Reliable launch processing (3D)

[Common design](dd.md) · [Tasks and completion status](adr.md).

## Purpose and baseline

An accepted launch must reach a recorded planning result even if a worker crashes,
a dependency is unavailable or two workers pick up work at the same time.

Use architecture-reset `48e60e542` as the baseline. Its `RunLaunchWorker` polls
PostgreSQL and runs beside Run supervision in the `run-supervisor-worker` process.
RabbitMQ carries lifecycle events in that process. Pending launches come from PostgreSQL.
The Python implementation keeps this launch-processing approach and adds a lease
check around the actual Run transaction.

## Flow

1. Launch acceptance saves its decision and receipt. For an accepted launch, the
   same transaction also inserts one pending work record in `pharos.launch_processing`.
2. The launch worker polls for due work. PostgreSQL locks and claims one record
   with a worker name, unique token, generation and expiry. Other workers skip it.
3. The worker calls the existing Run planner. It loads the accepted launch and
   exact Workload revision, checks current permission, selects the ready Project
   resource in `k8s` or `slurm`, and checks its applied limits.
4. Provider queries finish before the database transaction. The writer then checks
   that the claim is still current, including its expiry. It saves the Run, child
   Attempt, command receipt, existing typed Run projection and completed work
   result together. It checks the lease again before completion.
5. The work record points to the saved Run. Planning rejection instead records its
   reason and creates no Run. Repeated acceptance never reopens completed work.

The same worker handles Kubernetes and Slurm planning. The saved environment
continues through the existing execution messages to later Site delivery.

## Recovery rules

| Situation | Result |
| --- | --- |
| Worker stops after claiming | Another worker can claim it after the lease expires. |
| Workload, permission or provisioning lookup is unavailable | Save a retry time and a short error code, then release the claim. |
| Resource is still being provisioned | Keep the launch pending and try again later. |
| Permission is denied or the Workload exceeds limits | Record a terminal planning rejection. |
| Old worker resumes after expiry or replacement | Reject its Run write, completion and retry updates. |
| Lease expires while saving | Roll back the Run, receipt, projection and completion together. |
| Process stops after commit | The completed work record prevents another planning attempt. |
| A Run already exists from an earlier direct planner | Validate its launch, member, Project, Workload revision and environment, then record that Run as the result. |
| Stored state is corrupt | Retain pending work with a retry time and diagnostic code for repair. |

A launch has deterministic Run, initial Attempt and planning-command identifiers.
Each new claim uses the current database time for planning. Temporary failures
leave no command receipt, so later attempts can check deadlines afresh. A terminal
receipt and processing result commit together. The writer also checks the deadline
after provider queries, so a slow lookup cannot approve an expired launch.

## Storage and code

The accepted Launch remains an immutable business decision. Its original response
and retry key remain valid throughout processing. The work record is operational
delivery state, with status `pending`, `placed`, `rejected` or `failed`, next due
time, lease, attempt count, Run reference, outcome and error code. It is not a
second aggregate. Only the Run aggregate changes during planning completion.

Migration `0002_launch_processing.sql` creates this table and its due-work index,
grants the restricted Execution Core role data access, and schedules existing
accepted launches. Runtime startup does not apply migrations.

- `application/launch_processing.py` defines the worker and processing port.
- `adaptors/launch_requests.py` inserts pending work in the acceptance transaction.
- `adaptors/launch_processing.py` claims work, fences Run writes and records retries
  and results using the existing aggregate writer.
- `adaptors.postgres.launch_worker(...)` supplies the database and explicit Workload,
  permission and provisioning ports.
- `services/launch_worker.py` runs the polling loop and stops through a supplied event.

The composition requires real owner implementations to enable an installed worker.
Test doubles live in the test suite. The [hosted process](#hosted-process) builds
the available connections. Site commands, Run supervision and native execution
continue in Tasks 4, 5 and 6.

## Hosted process

`services/execution_worker.py` runs this loop and the 4E Run Supervisor in one
process, the `execution-worker` Deployment
([installation](../../../../../gitops/manifests/pharos/base/execution-worker/README.md)).

- `create_app()` is the production entry point. It reads Workload revisions from
  Workload Portfolio and sends commands through Site Coordination's store.
- The current Project permission, Project resources, member accounts, retry
  admission and saved input materialisation come from owners outside Metallo
  that are not connected yet. `adaptors/external_owners.py` names each one. Startup
  stops with `OwnerNotConnected` before opening a database, so every profile
  installs zero replicas.
- `compose(owners, intervals)` opens the three restricted databases, resolves the
  owner ports, then starts both loops. Shutdown drains both loops before closing
  the pools; a failed startup closes what it opened. Claims use the Pod UID.
- The dev-box pipeline lane runs the same composition through
  `tests/infrastructure/pipeline_native_worker.py`, with labelled doubles and
  shorter intervals, using the installed process's uvicorn command.

The Pipeline Coordinator and the Kubernetes Runtime follow the same rule: their
production entries build owner adaptors in code, and their lanes pass doubles to
the same composition. Verified on 2026-10-22:

- `pnpm verify` passed, including **1,856 Metallo tests**: both processes drain
  before closing pools, clean up partial startup and owner failures, and refuse
  before opening a database in production. It renders every profile and rejects
  configured owner factories in composition roots and GitOps manifests.
- `pnpm test:integration` passed **284 Metallo cases**. The coordinator ran through
  `compose(...)` against real PostgreSQL and RabbitMQ.
- `make metallo-pipeline-test` passed all **12 phases** with the coordinator,
  execution worker and Runtime started from their test entries: success,
  `actions.yaml`, failure and cancellation pipelines, coordinator and Runtime Pod
  replacement, and private application and KServe services with a real prediction.
- `make metallo-site-api-jobs-test` passed all **14 phases** and
  `make metallo-site-api-services-test` all **4 phases** with the Runtime's new entry.

The lanes ran in temporary namespaces on `metallo-dkatsimpris-1`, which were then
removed. Mocked owners: the saved-Pipeline Workload fixture, provisioned resources
and accounts, current permissions and repository credential issuance. The paused
GitOps Deployment itself was rendered and checked, not rolled out.

## Verification evidence

Verified on 11 October 2026:

- `pnpm verify` passed, including architecture checks, generated-model drift,
  Ruff, strict Mypy, **389 Metallo unit tests**, builds and deployment renders.
- The full disposable integration suite passed with **95 PostgreSQL cases**,
  RabbitMQ, service startup and the browser/session test.
- The isolated dev-box Job passed **465 cases**, comprising 371 Execution Core
  unit tests and 94 PostgreSQL cases. This includes the 3D worker's complete
  accepted-launch to saved-Run flow for Kubernetes and Slurm, production
  start/cancel message encoding, competing workers, retries, abandoned claims,
  lease expiry, slow lookups, existing-Run recovery and atomic rollback.

**Real:** PostgreSQL, restricted runtime role, migrations, worker loop, receipts,
Run/Attempt persistence and production codecs. **Mocked:** Workload queries,
Project permission, provisioning and source credentials. **Pending:** real owner
connections, installed worker hosting and native execution.

The namespace was `metallo-identity-dev-dkats-191d697bf7a6`, with its own database
and test Job. It was removed after verification. The test image was
`pharos-metallo/identity:191d697bf7a6`. A slow VPN upload was replaced with an
incremental image transfer, reusing unchanged pinned dependencies and verifying
all 115 source files against the checkout. The ordinary test image also built
successfully locally. An initial registry DNS failure in the wider integration
suite was resolved by rerunning it successfully.

Local evidence files are `/tmp/metallo-3d-verify.log`,
`/tmp/metallo-3d-integration-retry.log`, `/tmp/metallo-3d-devbox-verified.log` and
`/tmp/metallo-3d-incremental-report.json`. The harness's retained report is under
`.local/metallo/identity-tests/dev-dkats/identity/metallo-identity-dev-dkats-191d697bf7a6/report.json`.

## Remaining integration

Connect the real permission and provisioning ports, enable the installed worker
and verify restart recovery there. Native execution follows through the later
runtime tasks. Browser launch notifications remain in Milestone 2.