# DD — Pipeline records and internal operations (8A)

[Common design](dd.md) · [Tasks and completion status](adr.md).

Pipeline Coordination saves an ordered list of Workload revisions for one member
and Project. Each step will become a separate Run through the existing execution
path. This subtask saves and validates that intent. The coordinator that launches
and advances the steps is described in [8B](pipeline-coordinator.md).

## Flow

1. Validate a `StartPipelineCommand` with the Pipeline identity, initiating member,
   Project and ordered steps. Each step references an exact saved Workload revision.
2. Check the member's current Project permission. Read each referenced revision
   through the Workload owner port and check its Project, family and service kind. Check permission again after those reads.
3. Save the Pipeline snapshot, version, command outcome/receipt and initial pending
   coordinator work in one PostgreSQL transaction. The Pipeline and all its child
   steps are saved together.
4. Read the saved Pipeline only after checking the requesting member's current
   Project access.
5. Accept a cancellation request by saving its requester, reason and time on the
   Pipeline and waking its pending work in the same transaction. Preserve the
   initiating identity and first request. Later coordination cancels its Runs.

```mermaid
flowchart TD
    C["Internal start command"] --> H["Pipeline start handler"]
    H --> V["Validate ordered Pipeline steps"]
    V --> A["Check current Project access"]
    A --> W["Check exact Workload revisions"]
    W --> R["Recheck current access"]
    R --> P[("Pipeline PostgreSQL<br/>Snapshot, receipt and pending work")]
    P --> Q["8B coordinator claims the saved work"]
    Q --> L["Existing launch path creates each step's Run"]
```

The coordinator and step launches in the diagram are subsequent work. The same
Pipeline definition supports Kubernetes, Slurm and mixed environments.

## Ownership and saved data

`pipeline_coordination` is a Metallo bounded context with its own PostgreSQL
database and restricted runtime role. `Pipeline` is the aggregate root. Its ordered
steps are children and have no independent write repository. Execution Core owns
the Runs and Attempts created for those steps.

| Saved value | Meaning |
| --- | --- |
| Pipeline `id`, `memberId`, `projectId` | Canonical UUIDs for the Pipeline, initiating member and Project |
| `createdAt`, `version`, `status` | Explicit creation time, aggregate version and current record state |
| Ordered `steps` | Canonical step IDs, Workload IDs and exact positive revision numbers |
| Step `environment`, `family`, `serviceKind` | `k8s` or `slurm`, lifecycle family and the required service discriminator |
| Step `advanceWhen` | Batch success, or service health with an endpoint |
| Optional `cancellation` | First accepted request ID, requesting member, reason and timestamp |

A new Pipeline and its steps start as `waiting`. Cancellation changes the root to
`cancellation_requested`; the child steps remain waiting at this stage. Step Run
references and execution progress are defined in [8B](pipeline-coordinator.md).

For example, training commits and pushes model files to its repository in its
script or post-command. Inference runs next, pinned when it starts to the commit
its selected branch names then. The saved step order controls that sequence. Repository
selection and commands belong to the Workload, as described in [8C](pipeline-outputs.md).

## Validation and advancement conditions

The domain checks canonical identifiers, exact revision numbers, immutable child
records and strict snapshot fields. It rejects empty pipelines, duplicate step IDs
and unsupported advancement conditions.

The application asks `WorkloadRevisions.get_revision(workload_id, revision)` for
that exact revision. Its Project, identity, family and service kind must match the
step. Missing or mismatched Workloads are recordable business rejections;
unavailable or malformed owner evidence fails for retry.

- `queued-batch` supports `k8s` and `slurm`, with
  `advanceWhen: {state: succeeded, requireEndpoint: false}`.
- `reconciled-service` supports `k8s` with `serviceKind: application` or
  `huggingface-inference`, and
  `advanceWhen: {state: healthy, requireEndpoint: true}`.

These conditions are saved definitions. In 8B the coordinator evaluates Run
completion or a service's healthy observation and usable endpoint. A command
acknowledgement alone cannot advance the Pipeline.

## Commands, ports and permissions

`PipelineOperations` handles `StartPipelineCommand`, `ReadPipelineQuery` and
`CancelPipelineCommand`. It receives the shared `CommandPort`/`QueryPort` plus
context-local `PipelineAuthority` and `WorkloadRevisions` ports. The
`adaptors.postgres.operations(database, authority, workloads)` factory supplies
the production persistence adaptors. Real permission and Workload adaptors are
required explicitly when those owners are integrated. Current tests supply doubles.

Under the agreed two-role policy, `project-manager` and `developer` members can
start, read and request cancellation of Pipelines in their Project. Start and
cancellation checks `pipelines.cancel`, starts `pipelines.start`, and reads
`pipelines.view` (#755). The
requester can differ from the initiating member. The authority response must
match the requested member, Project and capability and contain a boolean verdict.
Denied access is rejected. Unavailable or uncorrelated authority fails for retry.

Access is checked afresh before receipt replay. A previously accepted Pipeline
retains its saved Workload intent, including if a Workload owner later removes a
revision or becomes unavailable. That historical acceptance does not authorise
later native execution. Each step's launch still uses the existing current-access
and resource checks. Accepted cancellation remains saved for later coordination.

## Persistence and recovery

| Table in the Pipeline database | What it records |
| --- | --- |
| `pharos.aggregates` | One `kind = pipeline` snapshot containing the root and all child steps, with its aggregate version |
| `pharos.command_receipts` | Command identity, material-input fingerprint and original outcome |
| `pharos.pipeline_work` | One pending/completed coordinator work row per Pipeline, its matching root version and availability/update times |

The shared PostgreSQL writer checks command receipts and expected versions before
applying the prepared decision. Start expects version zero. Cancellation supplies
the expected Pipeline version. Start material includes identities, creation time
and the full ordered definition, including environments and advancement conditions. Cancellation
fingerprints the target, requester, reason, time and expected version.
An exact replay returns its original outcome. Changed input under the same command
identity conflicts. A later cancellation preserves the first request without
incrementing the version again.

An accepted start inserts pending work atomically with its snapshot and receipt.
The first cancellation wakes that same row as pending at the new Pipeline version.
A receipt replay, no-op or rejection preserves its existing status and timing.
There is no independent step write and no cross-context database access.

Owner calls happen outside the transaction. Expected Workload rejections receive
receipts. Infrastructure failures and corrupt snapshots propagate without recording
a terminal business outcome.
Restoration checks the snapshot against its queried ID and stored version.
Invalid identities, ordering, progress or cancellation metadata raise `CorruptState`.
Command writes also check that coordinator work exists at the saved root version.

The 8A migration adds `pharos.pipeline_work` to the baseline schema.
The additive [8B migration](pipeline-coordinator.md#records-and-recovery) adds
leases and the step launch correlation index.
The writer explicitly disables browser projections for the `pipeline` aggregate.
Browser updates are connected in 8F. Pipeline persistence uses its restricted
context role.
A failed transaction leaves no partial Pipeline, receipt or pending work.

The removal of earlier Pipeline binding fields changes saved definitions and
command material. See the [compatibility note](pipeline-outputs.md#saved-and-wire-compatibility).

## Verification

Historical 8A verification, completed on 16 October 2026. The repository workflow
correction is verified separately in [8C](pipeline-outputs.md#verification-and-remaining-connections).

| Boundary | Evidence |
| --- | --- |
| Record and application rules | All 276 Pipeline domain/application cases passed |
| PostgreSQL and production adaptors | All 36 Pipeline cases passed with restricted runtime credentials: reload, receipts/conflicts, stale versions, concurrent requests, corruption and atomic rollback |
| Repository checks | `pnpm verify` passed with 1,558 Metallo tests, strict Mypy over 227 source files, package checks, frontend build and deployment renders |
| Disposable integration | `pnpm test:integration` passed with 221 Metallo PostgreSQL cases, real RabbitMQ, all application images and the existing platform browser check |
| Isolated dev-box backend flow | 1,314 cases passed, including all Pipeline cases, with real PostgreSQL and production adaptors. The owned namespace was removed |
| Project permissions and exact Workload lookup | Explicit test-only providers. Real owner connections remain open |
| Coordinator and native execution | Implementation and qualification follow in 8B–8D |

The [persistence cases](../../../tests/infrastructure/pipeline.py) and
[failure/rollback cases](../../../tests/infrastructure/pipeline_failures.py)
exercise internal start, reload, read and cancellation. Run the local lane with
`uv run --project devops --frozen pnpm test:integration`, and the dev-box lane with
`make metallo-identity-test TARGET=dev-NAME SSH=your-dev-box`. Context migration
metadata regeneration produced no drift.

This closes 8A's records and internal operations. Step launching, Run completion
delivery, repository hand-offs and native pipelines are covered in 8B–8D. Public action parsing,
API contracts and Pipeline browser flows are connected in 8E–8F. Minikube and manual
UAT were not run for this internal backend subtask.