#Pipeline Coordination

Pipeline Coordination is the Metallo context for executing ordered steps from saved Workload revisions. The Pipeline is its aggregate root and its steps are children, saved and changed together.

The domain record identifies the member and Project, the ordered steps, each step's immutable Workload revision, k8s or slurm environment and advancement condition. The Workload selects repositories and branches; each step is pinned to their commits when it starts and fetches those when preparing. Its internal operations start, read and request cancellation of a Pipeline through explicit ports.

Execution Core continues to own Runs and Attempts. Pipeline Coordination requests step launches through Execution Core's published interfaces and records its own progress and Run references. Workload lookup and Project permission checks use explicit owner ports. Required cross-aggregate work uses a durable hand-off, with each owner committing its own transaction.

#Registration

The registry enrols pipeline_coordination with the shared PostgreSQL bootstrap, which provisions its private database and restricted runtime role when explicitly run. The context uses the baseline schema plus its 0001_pipeline_work.sql migration.

The PostgreSQL adaptor uses this database for internal Pipeline operations. Coordinator execution and its dedicated process are implemented in 8B.

#Domain records

domain.pipeline.Pipeline is the aggregate root. Its frozen child steps preserve order and reference canonical Workload IDs with positive exact revision numbers. Every new Pipeline and step starts as waiting. The member and Project belong to the root. Each step records the referenced Workload's family and, for a service, its serviceKind.

Step order determines execution order. Training can commit and push its model files using its existing script or post-command. Inference then fetches the latest contents of its selected repository branch. Duplicate step IDs and incompatible advancement conditions are rejected.

A batch step saves advanceWhen: {state: succeeded, requireEndpoint: false}. Application and Hugging Face service steps save advanceWhen: {state: healthy, requireEndpoint: true}. Batch steps support both environments; services use k8s, matching current execution policy. The coordinator applies these conditions to Run observations and the service endpoint, with final owner evidence required for batch completion.

to_mapping() returns detached JSON-compatible data. from_mapping() validates incoming data, while restore() treats invalid saved data as CorruptState. The application checks Workload ownership, revision, family, service kind and saved execution profile through its owner port before accepting a new Pipeline.

#Public requests

application.member_pipelines.MemberPipelines starts a repository branch's actions, reads a Pipeline with its steps' action names and Runs, and asks one to stop. adaptors.requests maps those typed requests from the API and the internal accepted-Pipelines query. metallo-requests hosts both. See Public requests.

#Internal operations

application.handlers.PipelineOperations receives the shared CommandPort and QueryPort, plus explicit PipelineAuthority and WorkloadRevisions ports. adaptors.postgres.operations(database, authority, workloads, lifetime=...) supplies production PostgreSQL adaptors. lifetime is required: the composition passes the deployment's accepted_work_lifetime(), the same ACCEPTED_WORK_LIFETIME_DAYS launches use. Permission and Workload providers remain explicit dependencies; current verification supplies test-only implementations of those two ports.

• start(StartPipelineCommand) validates the initial record and checks each exact Workload revision against its Project and execution profile. The complete initial record is the command's material input. The persistence port owns receipt replay, conflicting retries and expected-version checks.
• read(ReadPipelineQuery) loads and validates the saved record, checks the requesting member's current access and returns the immutable Pipeline.
• cancel(CancelPipelineCommand) records the first cancellation request and its requester, reason and time. The root advances to cancellation_requested while preserving its initiating identity and waiting child steps. A later request preserves the first request. The coordinator stops further launches. The coordinator continues accepted cancellation through Execution Core, including live services and delayed launches.

Under the agreed two-role policy, project-manager and developer members can start, read and request cancellation of Pipelines in their Project. Start consults pipelines.start, cancellation pipelines.cancel and reads pipelines.view (#755), through their own context-local authority port, including when the requester differs from the initiating member. Access is checked afresh before receipt replay; start checks it again after Workload lookup. Denied access is rejected and unavailable or uncorrelated authority fails for retry.

Owner lookups happen before the aggregate transaction. A missing or mismatched Workload produces a recordable business rejection. Unavailable or malformed Workload evidence fails for retry. The receipt gate preserves an already recorded outcome when a lookup later fails. An accepted Pipeline replay reuses its saved intent rather than looking up its Workloads again. Current authorisation remains required. Real owner adaptors will implement these same ports.

A start saves the evidence Beto's allowing pipelines.start answer carried, together with the accepted-work deadline that the declared steps share. An allow without evidence defers the start. AcceptedSteps.confirm returns that saved acceptance with the confirmed step, and Execution Core saves it on the step's Launch (#766). Pipelines started before the record carry none. adaptors.postgres.accepted_pipelines(database) lists an item for each Site of every accepted Pipeline, required until it is final and then released as its completion published it, for the Site owner's initial load and repair (#765, #766). The start commits pipeline_coordination.pipeline.accepted_work_changed.v1 for each Site, and the coordinator completing the work releases them one past the completed version; the coordinator process relays them with PIPELINE_COORDINATION_BROKER_URL. Enrolled Metallo callers read it as the internal query ListAcceptedPipelines, which the coordinator process answers with PIPELINE_COORDINATION_REQUESTS_BROKER_URL (internal queries). Once the account owner is connected (PipelineAccounts), a start saves the member's exact account at every one of those Sites and skips none: a Site without one refuses the start, and an unclear answer leaves it pending. A step's Launch keeps the one at its Site. The change event, pipeline_coordination.pipeline.accepted_work_changed.v1, is published once its consumer (#785) subscribes. See the accepted-work design.

#Repository action pipelines

application.repository_pipelines.RepositoryPipelines.start runs every action of a repository branch as one Pipeline. Its RepositoryActionWorkloads port prepares each action's Workload revision in actions.yaml order through Workload Portfolio; the steps then start through the internal Pipeline start. The service-level adaptors.pipeline_actions.repository_pipelines(..., lifetime=...) composes it and requires the deployment's accepted-work lifetime. Design and verification.

#Persistence

adaptors.postgres composes the shared PostgreSQL Commands and Queries adaptors. pharos.aggregates holds one kind = pipeline snapshot with its child steps. pharos.command_receipts holds command fingerprints and recorded outcomes. The context migration adds pharos.pipeline_work for pending coordinator work. Start saves all three together. The first cancellation wakes work at the new root version in the same transaction. Replay, rejection and no-op preserve its status and schedule. Missing or mismatched work, snapshot identity or versions fail as CorruptState before command replay.

The runtime role can read and update its own work table and has the baseline aggregate/receipt grants. Schema administration remains separate. Pipeline records are explicitly registered without a browser projection. The coordinator adds fenced work claiming, a launch correlation index and step progress to these saved records. Coordinator design and evidence.

The context migration also adds Pipeline Coordination's copy of Beto's Project memberships, Party states and provider bindings (pharos.project_views, project_members, party_views, party_bindings, project_access_invalidations, view_state, view_checkpoints), applied from Beto's integration events with consumer receipts and loaded from Beto's checkpoints. The coordinator runs its eight consumers and synchroniser with PIPELINE_COORDINATION_VIEWS_BROKER_URL. PartyProjectAuthority decides the Pipeline permission from it by Beto's rule once it is loaded (#757), and defers unless Beto's latest Project authority (pharos.project_authority, migration 0004) is ready, unexpired and describes the Project revision it holds (#798). Current-person operations confirm allowing decisions with Beto (#755). The coordinator continues accepted steps through application.accepted_steps.AcceptedSteps, which confirms their exact saved scope, launching progress, order and cancellation state for Execution Core. Personal suspension preserves that accepted scope. Inputs are resolved when each step starts, so it can consume outputs from its predecessor; future native jobs are created only when their step becomes eligible. Design, accepted-work policy.

See the 8A design for the complete internal flow and verification boundaries.

A confirmed account supplied at start also saves its allocation ID, account reference and native UID at each Site. Step admission carries those fields into Execution Core unchanged; replay keeps the original evidence. Existing snapshots with only the binding and generation remain readable and retain the weaker native-identity check; follow the queued-work rollout inventory to bound that compatibility period. The request host composes public Pipeline starts through this account owner.

#Verification

Registration verified on 16 October 2026:

• uv run --project devops --frozen pnpm verify passed, including context ownership, editable/wheel imports, credential upgrade and runtime-isolation regressions, strict Mypy, all 1,282 Metallo tests, frontend build and deployment renders.
• Regenerating context migration metadata produced no drift.
• A disposable PostgreSQL 18 instance ran production credential configuration and context bootstrap twice. The Pipeline runtime role connected to its own database with the correct context identity and empty context migration history. Real DDL and connections to both other context databases were denied. The disposable container and storage were removed afterwards.

This evidence covers context registration. Pipeline operations and coordinator execution are verified in their subsequent implementation stages.

#Domain verification

On 16 October 2026, all 124 tests in tests/pipeline_coordination/test_pipeline.py passed. They cover ordered Kubernetes/Slurm steps, both service kinds, exact identities and revisions, immutable records, advancement conditions and corrupt snapshots. uv run --project devops --frozen pnpm verify also passed, including all 1,406 Metallo tests, strict Mypy over 210 source files and installed-wheel imports.

These are deterministic domain tests without external providers. PostgreSQL persistence, browser/UAT and live-cluster execution are later verification stages.

#Application verification

On 16 October 2026, all 276 Pipeline tests passed, including 114 application tests and 38 added cancellation-domain tests. They cover current and revoked access, missing, mismatched and unavailable Workloads, exact revisions, receipt replay and conflicts, stale versions, first-request preservation and corrupt-state guards.

uv run --project devops --frozen pnpm verify passed, including all 1,558 Metallo tests, strict Mypy over 223 source files, installed-wheel imports, frontend build and deployment renders. Permission, Workload and storage providers in these new tests are test-only doubles. Real PostgreSQL persistence and rollback, real owner connections, browser/UAT and live-cluster execution were not run for this internal handler stage; they remain separate verification work.

#Persistence verification

Completed on 16 October 2026:

• All 36 Pipeline PostgreSQL cases passed using the production adaptors and a restricted runtime role. They cover start/read/cancellation reload, identities, ordered mixed environments and both service kinds, receipt replay/conflicts, current-access failures, concurrent commands, corrupt state and atomic rollback after the snapshot, receipt and pending work have all been written.
• uv run --project devops --frozen pnpm verify passed, including 1,558 Metallo tests, strict Mypy over 227 source files, installed-wheel imports and renders.
• uv run --project devops --frozen pnpm test:integration passed with 221 Metallo PostgreSQL cases, actual RabbitMQ, all application images and the existing platform's Chromium session/realtime check. Pipeline browser integration is 8E–8F.
• make metallo-identity-test TARGET=dev-NAME SSH=your-dev-box passed in an owned namespace on the dev box: 1,314 cases, including the same 36 Pipeline database cases and all 276 Pipeline domain/application cases. Production migrations and restricted-role PostgreSQL were real. The namespace was removed afterwards.
• Context migration generation reproduced the checked metadata without drift.

Persistence cases and failure/rollback cases are the reproducible evidence for 8A. Workload revisions and Project permissions use explicit test-only providers. Real owner connections, coordinator/native pipelines, permanent coordinator deployment, Minikube and manual UAT remain later verification. PostgreSQL, permission and Workload connections are not conflated.

#Coordinator implementation (8B)

The coordinator saves one deterministic launch identity per step before using Execution Core's acceptance/planning path. PostgreSQL stores child progress, leases and correlation with the Pipeline. Run completion is delivered through the existing typed outbox and RabbitMQ, with atomic consumer receipts and progress. Service readiness and submission recovery use Execution Core owner queries.

The dedicated startup and GitOps resources are included. The Deployment installs paused until real Workload and permission providers are configured. Flow, recovery, verification and remaining real-provider checks.

On 20 October 2026, all 364 Pipeline policy/handler cases and 72 combined Pipeline PostgreSQL/RabbitMQ cases passed. pnpm verify passed with 1,671 Metallo tests and strict Mypy over 251 source files. The disposable repository integration lane and pinned generation/drift check passed. The isolated dev-box PostgreSQL lane passed 1,444 cases and removed its owned namespace. External Workload, permission, provisioning and native responses remain test doubles. Real-provider, native pipeline and enabled permanent-deployment qualification remains under 8D.

#Repository hand-offs and cancellation (8C)

Each step is pinned, when it starts, to the commit its selected branch names then. Training scripts or post-commands commit and push model files with scoped Git/LFS write credentials. Inference, pinned after that push, fetches those files when its action prepares. Pipeline Coordination preserves order and saved Workload references; Runtime owns the repository checkout and credential lifecycle.

Accepted cancellation fences delayed launches and planning, cancels live Runs including services and waits for final Execution Core evidence.

Design, corrective verification and remaining real-provider connections.

Current verification and remaining real-provider connections are recorded in the linked design document.

#Accepted Project work

The Pipeline stores its original Beto authority revision, management generation, acceptance time, bounded dispatch deadline and immutable definition digest in its snapshot. Its internal step hand-off reads the persisted launching step and its recorded launch identity. Execution Core accepts only that correlated definition and keeps the original Pipeline deadline. A member leaving therefore does not block already accepted steps; current-person requests still require their exact Project permission. Repository branches are resolved when each step launches, so a later step can consume a previous step's pushed results.

The coordinator checks Project and saved Site binding fences before advancing an accepted Pipeline. It records a policy cancellation with a stable request identity before stopping any step. The existing cancellation sweep fences delayed launches and waits for each Execution Core owner result. Closure and security cancellation therefore survive process restarts and prevent later steps from starting. Owner outages defer advancement; saved cancellations remain recoverable.

Page source

Source snapshot · Revision f928834c8.