# DD — Kubernetes pipeline verification (8D)

[Common design](dd.md) · [Tasks and completion status](adr.md)

## Scope

Exercise a saved Pipeline through the production coordinator, launch acceptance,
launch processing, Run supervision, Site Coordination and Kubernetes Runtime.
PostgreSQL, RabbitMQ, HTTPS, projected identity, SQLite/PVC journals, Jobs,
application Deployments and KServe are real. Workload, Project permission and
provisioning ports use explicit qualification providers until their owner
integrations are available.

The qualification uses the dev box's installed GitLab with temporary repositories and
repository-scoped write credentials. Administrative bootstrap remains in the
workstation tooling. Runtime and user workloads receive only their scoped test
credentials. Automatic production token issuance and revocation remain a separate
provider connection.

The provider probe finds the dev box's GitLab: the GitLab chart that Argo CD
installs in `gitlab` (Rails through `gitlab-toolbox`, Git and LFS through the
Workhorse Service on 8181), or the older Omnibus installation in
`pharos-local-providers`. It creates private source and denied repositories and a
source-only `write_repository` project token. Both serve internal HTTP and deny
other ingress, so an owned test bridge, admitted by its own network grant,
provides verified HTTPS to Runtime and rewrites Git LFS URLs only for those
repositories. Its final hop to GitLab remains private HTTP. The bridge has no provider token and changes no shared GitLab setting.
This qualifies real GitLab transport and token scope, rather than production
HTTPS rollout or the future automatic credential-provider connection.

## Scenarios

1. Start an ordered training Job, application service, inference service and
   verification Job through the internal Pipeline handler. Training reads its
   repository dataset, changes model files and commits/pushes through Git/LFS.
   Later steps fetch the latest selected branch and verify those published files.
2. Match each finished Run to its published completion event and coordinator
   receipt through the real outbox and RabbitMQ subscription. Services advance
   after saved health and a usable endpoint. Invoke both private endpoints and
   read the training files from their exact Attempt volumes.
3. Replace the coordinator and Runtime during execution. Compare saved launch,
   Run/Attempt, native UID and journal-PVC identities to prove recovery preserves
   existing work. Repeat service prediction and private reads after Runtime
   replacement. Require exactly one Run per submitted step launch.
4. Fail a native batch and confirm later steps stay unlaunched.
5. Cancel an active Pipeline containing a ready service and unfinished work.
   Wait for cancellation cleanup and final owner evidence. Confirm existing
   native resources stop and later steps do not start.
6. Run all repository actions in order ([8E](repository-pipelines.md)). The
   seeded branch's `actions.yaml` is read from the installed GitLab, its
   application and inference actions become Workload Portfolio Workloads, and one
   Pipeline starts over them. Both services become healthy with endpoints and answer
   private requests. The coordinator and execution worker read these Workloads from
   Workload Portfolio through its restricted role.

## Installation and evidence

The harness installs in uniquely named, owned namespaces, using the committed
Runtime and admission templates and the production coordinator composition.
Temporary process manifests host the coordinator and production launch/supervision
loops alongside disposable PostgreSQL and RabbitMQ stores. Its provider doubles are test modules and are not fallback
production integrations.

This isolated installation qualifies the backend flow. Normal GitOps installation
with real owner connections, Slurm and mixed-environment Pipelines have separate
unchecked ADR items.

Run the lane with:

```bash
make metallo-pipeline-test TARGET=dev-NAME SSH=your-dev-box
```

Use the existing dedicated SSH credentials described by the development guide.
Reports and scoped temporary provider records are private under the selected
target's `pipeline/<namespace>/` directory. Successful runs remove their owned
GitLab access, network grant, KServe resources and Kubernetes namespaces. Failed
runs retain evidence. Use `make metallo-pipeline-inspect` or
`make metallo-pipeline-clean` with the printed `PIPELINE_NAMESPACE`; cleanup checks
ownership and can recover provider objects after a lost bootstrap response.

## Verification on 20 October 2026

| Check | Result |
| --- | --- |
| Repository verification | `pnpm verify` passed, including 1,716 Metallo tests, strict Mypy, packaging, contract drift, Go and frontend/deployment checks |
| Disposable integration | `pnpm test:integration` passed, including 272 PostgreSQL cases, real RabbitMQ, service images, sessions and browser regressions |
| Native success Pipeline | Training Job, application, Hugging Face inference and final verification Job passed. Each later action read the training commit through real Git/LFS |
| Delivery and identity | Both successful batches, the failed batch and both cancelled Runs had exact published completion events and matching coordinator receipts. Each submitted step launch had exactly one Run and one Attempt |
| Recovery and private reads | Coordinator/Runtime replacement preserved native and PVC identities. Application prediction, inference prediction and private training-file reads passed before and after Runtime replacement |
| Failure and cancellation | Native batch failure left the later step unlaunched. Pipeline cancellation stopped its ready service and running batch, preserved the unstarted step and waited for both owner-final records and completion receipts |
| Installed GitLab | GitLab 18.4.0 accepted source checkout and Git/LFS push. Its project token was denied read and write access to a second existing private repository |
| Cleanup | Temporary GitLab access and repositories, provider network grant, KServe resources and owned namespaces were removed |

An initial qualification exposed a premature cancellation assertion in the test:
accepted cancellation preceded asynchronous Run cleanup. The verifier now waits
for cleanup and matching delivery evidence. The corrected lane passed from a
fresh image and namespace without manual continuation or an image upgrade.

On 21 October 2026 the same lane passed again with scenario 6. `actions.yaml` from
the installed GitLab started one Pipeline over Workload Portfolio Workloads. Its real
application Deployment and KServe InferenceService became healthy and answered
private requests. All earlier scenarios and cleanup passed unchanged.

The private native report is under the selected target's
`pipeline/metallo-site-api-dev-dkats-9129b86e/report.json` evidence directory.
Workload ownership, provisioned resource/account mappings, current platform
permissions and automatic credential issuance/renewal remain qualification
providers. Real-provider revocation, normal GitOps installation, Slurm and mixed
pipelines remain unchecked in the ADR.