# DD — Pipeline repository hand-offs and recovery (8C)

[Common design](dd.md) · [Tasks and completion status](adr.md)

## Repository hand-off

1. Execute the saved actions in order, using their existing repository selections
   and commands.
2. The training action fetches the commit it was pinned to at acceptance into
   its private workspace. Its script or post-command commits and pushes model
   files with Git/LFS.
3. Wait for training's successful final Run state before launching inference.
4. Inference is pinned when its step starts, to the commit its branch names then,
   and fetches that commit when preparing. Its repository therefore includes the
   weights pushed by training.

Saved Workload revisions remain immutable. Their repository references contain a
repository ID and optional branch. An omitted branch selects the remote default
branch. Runtime fetches it when preparing the action and retains a normal Git
checkout with its remote, working branch and Git commit identity. Temporary
repository-scoped `write_repository` credentials remain usable by action commands.
The Runtime mounts them in a separate per-Attempt Kubernetes Secret, refreshes
them while work remains active and revokes them after execution or cancellation.
It also revokes failed preparation and definite submission rejection. The Pod and
SQLite record only the Secret reference and UID. Raw tokens stay outside durable
Pipeline records, execution messages, workspace files, logs and image layers.

Pipeline Coordination owns ordering and Run references. Runtime owns repository
preparation and Git credential handling. The existing `actions.yaml` parser is
connected in 8E; backend tests create saved Workloads and Pipeline steps directly.
The same repository workflow supports Kubernetes and Slurm without a new action
input/output format.

## Recovery and cancellation

Stable per-step launch identities recover uncertain launch responses. Execution
Core continues to own each Run, its Attempts and retry policy. A failed final
step prevents later steps from launching. PostgreSQL leases and root versions
prevent stale coordinators from advancing the Pipeline.

An accepted Pipeline cancellation stops new launches. The coordinator visits
every submitted step, including a ready service, and records an Execution Core
launch stop before cancelling an existing Run. The immutable
`pharos.launch_stops` fence serialises with launch acceptance and both direct and
leased planning, so delayed work cannot create a cancelled Run.

Existing Runs use a trusted continuation of the accepted cancellation. It retains
the original requester after access revocation. New member requests still check
current permission. Keep coordinator work pending until all known Runs have final
owner evidence or their absent launches are durably stopped. Final cancellation
and its pending-work outcome commit in the same Pipeline transaction.

A new cancellation request on an already succeeded, failed or cancelled Pipeline
remains a no-op. A service outliving a finished Pipeline can be cancelled through
its Run. All live services participate in an already accepted Pipeline cancellation.

## Saved and wire compatibility

Repository references now contain `repositoryId` and optional `branch`, replacing
commit-pinned source fields in Workload/Run records and Site messages. Pipeline
steps also drop the earlier input/output binding fields. These are stored and
wire-shape changes; the strict current decoders reject the previous shapes.

The coordinator's provider Deployment remains paused and the implementation has
used disposable data. Local fixtures are recreated for this correction; no
production-data migration is supplied. Regenerate transport models and deploy
matching platform/Runtime code together.

## Verification and remaining connections

Current corrective verification on 20 October 2026:

| Boundary | Evidence |
| --- | --- |
| Repository checks | `pnpm verify` passed with 1,699 Metallo tests, Ruff, strict Mypy over 268 source files, packaging and deployment checks |
| Transport generation | Pinned generation reproduced the checked models without drift |
| PostgreSQL | 267 applicable cases passed with production adaptors, migrations, receipts and restricted runtime roles |
| Disposable integration | `pnpm test:integration` passed with 272 Metallo PostgreSQL cases, real RabbitMQ, application images and existing browser checks |
| Isolated dev-box PostgreSQL | 1,458 cases passed; the owned namespace was removed |
| Git/LFS transport | Real HTTPS checkout, push and latest-branch tests passed; permission and credential issuance use explicit test providers |
| Native Kubernetes Jobs | Resumed qualification passed: Git/LFS push followed by a later Job reading the latest weights, Runtime replacement/UID recovery, private access, six credential/volume admission denials, failure, quota block/release and cancellation after native access removal |
| Native application and inference | Resumed qualification passed with real Deployments and KServe: prediction, private files/logs, both profiles' credential/volume denials, same-Pod initialisation, Runtime replacement with retained native/PVC identities and cancellation after native access removal |

The native Job lane resumed after a 45-second fixture wait expired following
reader preparation. It continued the same durable commands and production Runtime
with a 120-second test wait; this is a resumed qualification. Its owned namespace,
roles and admission policies were removed after the persisted report passed.

The service lane resumed after fixing prediction queries' Git-credential side
effects. A rebuilt production Runtime image replaced only the owned test Runtime
Deployment, preserving the healthy application, InferenceService and SQLite PVC
identities. Private reads and cancellation then passed. This was a controlled
image upgrade and resumed qualification. The owned namespaces, KServe CRDs and
webhooks, admission policies and RBAC were removed afterwards.

The complete native Kubernetes pipeline and installed GitLab transport are
recorded under [8D](pipeline-verification.md). Production credential issuance,
refresh/revocation, Slurm/mixed environments and the enabled permanent coordinator
remain open. Permission and grant providers in these native lanes are explicit
test doubles. This closes 8C's implementation and scoped verification.