# DD — Instantiate and run workloads

Status: Initial proposal.

This is an initial technical suggestion. Each person working on this feature should review it and revise it as needed before implementing their part.

Feature scope, tasks and PR subtasks: [ADR](adr.md).

The delivery scope uses **`queued-batch`** for Kubernetes/Slurm jobs and
**`reconciled-service`** for Kubernetes applications and Hugging Face inference.
The internal `intent.serviceKind` selects `application` or `huggingface-inference`.
Native service implementation and verification are organised under 5A–5C, using
architecture-reset `48e60e542` as the baseline with the adaptations below.
Notebooks and interactive sessions are outside this epic. Existing session
contracts and technical tests remain shared coverage.

Runtime execution has no per-Run metering command or collector. The supervisor
commits terminal state and its completion event together. Provisioning owns
Project quota reporting through the Waldur Site Agents.

## Proposed technical implementation

- **Python Metallo:** Workload Portfolio stores Workloads created from repository actions. Execution Core stores Runs, Attempts and execution results. Site Coordination stores commands and results. These platform contexts have private PostgreSQL databases. Site Runtime is a separate bounded context with private Site-local SQLite journals; it has no platform database.
- **Client API and workers:** the Go API sends repository-action requests to Workload Portfolio and Run requests to Execution Core. PostgreSQL saves changes and responses together. RabbitMQ delivers recorded events. Launch and supervisor workers claim work with leases and reject stale claims.
- **Site communication:** generate request and response models from OpenAPI and send them over HTTPS/JSON. Site Runtime pulls numbered commands and acknowledges results. Both sides save progress for recovery.
- **Native execution:** one Python Site Runtime per Site uses shared application ports with Kubernetes or Slurm adaptors under `contexts/site_runtime`. Startup and wiring stay in `services/`. It has Site-scoped credentials, without platform database or Waldur API access. [Context boundary](site-runtime-context.md).
- **Member delegation:** Kubernetes uses member service-account impersonation, restricted RBAC and admission controls. Slurm uses a member JWT through local slurmrestd, as proposed in [Slurm Site access](slurm-site-access.md).
- **Results:** native status and execution results return through Site Coordination to Execution Core. Milestone 2 (7A) connects Next.js views to authorised typed Protobuf updates through Centrifugo. Logs and outputs require current access checks.

## Workload families and serviceKind

Milestone 1 consumes saved Workload revisions. The Milestone 2 authoring adaptor
maps external `actions.yaml` types into the internal model:

| Authoring action type | Environment | Workload family | `intent.serviceKind` | Native execution |
| --- | --- | --- | --- | --- |
| `slurm-job` | `slurm` | `queued-batch` | Not applicable | Slurm job |
| `k8s-job` | `k8s` | `queued-batch` | Not applicable | Kubernetes Job |
| `k8s-deployment` | `k8s` | `reconciled-service` | `application` | Kubernetes Deployment, including Spaces |
| `k8s-inferenceservice` | `k8s` | `reconciled-service` | `huggingface-inference` | KServe InferenceService |

The execution environment comes only from the action type. Workload Portfolio
derives it when it creates, reuses or revises the action's Workload and passes it
with the prepared revision to the launch and to Pipeline steps. `actions.yaml` has
no environment field, and the public requests and responses carry none.

The `k8s-job` mapping uses the existing Kubernetes batch Job path. Its supported
options are described in the [Workload Portfolio guide](../../../contexts/workload_portfolio/README.md#repository-actions).
Family describes the lifecycle. Both service kinds stay active while healthy and
use the existing service operations and state rules. `serviceKind` selects the
native implementation within `k8s`.

```yaml
intent:
  family: reconciled-service
  serviceKind: application  # or huggingface-inference
```

The internal schema requires `serviceKind`, a `repository` reference, `preCommands`
(an empty list is valid), and the common service policies. Planning, Run restoration
and command codecs validate the kind and preserve it in the immutable execution
specification. Runtime selects the native implementation from that saved intent.

Repository fetching reuses the trusted helper Pod: Git/LFS goes directly to the
private Attempt PVC. Repository-scoped write credentials remain available to the
action commands and are revoked when execution ends or is cancelled.
For **both service kinds**, `preCommands` run in an **init container in the same
Pod as the main programme**. It shares the PVC and must succeed before the main
container starts. A normal main-container restart reuses completed initialisation;
a replacement Pod runs its init container again. Commands must tolerate that repetition.

Application init and main containers use the fixed platform executor image. The
init container prepares the private Python environment; the main container reuses
it and runs `runtime.command` from the repository. Any post-command included there
runs after the programme exits. Hugging Face init uses the platform image, while
KServe chooses the server image from its installed runtime through
`modelFormat: huggingface`. Its Workload supplies `model` settings and
`runtime.environment`, without an artifact image or server command. The two
containers share files, not installed system packages. Post-serving hooks remain
future lifecycle work.

Architecture-reset's action converter maps both Kubernetes action types to
`reconciled-service`; its Runtime routes by Environment and family to a configured
connector. We retain the family and add `serviceKind` because our `k8s` destination
contains both profiles. KServe-managed Hugging Face image selection also adapts
the reset branch's explicit artifact-image resolver. Implementation and the
separate mocked, PostgreSQL and native evidence are recorded under 5A–5C. Real
provider connections remain open. Earlier completion claims retain their recorded scope.

## Document organisation

Each subtask's detailed design, implementation notes and verification evidence live
in a separate Markdown file linked below. Add a file and a link here when designing
or implementing each later subtask. Record real and mocked verification separately,
including what remains to be tested. Track completion checkboxes in the [ADR](adr.md).

## 1A — Kubernetes Site access

Kubernetes member delegation, Runtime permissions, current access and isolation.
The Kubernetes design is documented. The Slurm design is proposed in
[Slurm Site access](slurm-site-access.md) and awaits agreement under ADR subtask 1A.

[Design and required evidence](site-access.md).

## 1B — Member and Project identity

Member and Project identity on Runs and Attempts, persistence and fresh account lookup.
Identity and persistence are tested with real PostgreSQL. The provisioning lookup uses mocks.

[Design, identity diagrams and verification](member-project-identity.md).

## Local Party and Project views (#757)

The member and Project facts Metallo's contexts need, what Beto publishes on the
governance branch, the decisions taken in #754 and #755, and the context-owned views,
their consumers and their load from Beto's checkpoints. A loaded view decides by Beto's
rule, provider bindings included; every allowing current-person operation still asks
Beto for current evidence at decision time.

[Needs, coverage, decisions and remaining gaps](party-project-views.md).

## Run access with current Project permissions (#766)

Reads, outputs, launch, manual retry and cancellation ask Beto Project for their
exact `runs.*` capability. Workloads and Pipelines use their own explicit
capabilities. The note defines the decision-time owner check and bounded authority
that continues accepted work after its initiator leaves.

[Capabilities, evidence, continuation and reads](run-access.md).

## 1C — Execution messages and permission contracts

Execution messages, current permissions and delayed credentials. Batch uses
`queued-batch`; both service kinds use `reconciled-service`. Extending the
contracts/codecs now preserve `serviceKind` as part of the 5B extension. Existing session
messages remain shared contract coverage.
The contract implementation and required local/dev-box verification are complete.
Real provider connections and native execution remain in their integration tasks.

[Design, implementation and verification](execution-messages.md).

## 1D — Repository action and Run API messages

Browser requests reach Metallo through the Go API as typed RabbitMQ queries and
commands. Metallo validates and dispatches them to owner handlers; command replies
commit with their deciding transaction and are relayed after commit.

[Request transport, durable replies and verification](request-transport.md).


## 2D — Reliable Workload creation

Workload Portfolio saves Project-owned Workloads with immutable, numbered
revisions. Internal operations check current Project access and save each
snapshot with its receipt. Owner adaptors answer the Workload ports used by 3B,
3C and 8A. Running a repository action validates it, converts it into a
specification and creates, reuses or revises that action's one Workload.

[Ownership, operations, owner adaptors and verification](workload-creation.md).


## 3A — Project resource lookup

The query for a Project's provisioned resources, environments, readiness and limits.
Internal implementation and mock/dev-box verification are complete. Real provisioning remains open.

[Design, query response and verification](project-resource-lookup.md).

## 3B — Check and record launch requests

Status: Internal implementation and isolated dev-box verification complete, with real PostgreSQL and mocked external owners.

Execution Core checks current Project permission and the saved Workload's Project,
active state, version and revision. It stores the initiating member, request and
accepted/rejected decision in PostgreSQL. Launch notification contracts and recording
belong to Milestone 2 (7A). Planning follows separately.

Implementation, retry rules, storage and verification evidence are in the
[launch acceptance DD](launch-acceptance.md). Real Workload and permission
connections remain open in the ADR; completed mocked/internal steps are separate.

## 3C — Admission and placement

The accepted launch supplies `environment: k8s | slurm`. Execution Core reads its
exact Workload revision and the Project's provisioned resources, checks readiness
and applied limits, and records a Run with its immutable specification and target.
Implementation, replay rules and separate real/mocked verification evidence are
in the [planning DD](run-planning.md).

## 3D — Reliable launch processing

The launch worker claims accepted requests from PostgreSQL and calls the planner.
Leases reject stale workers. Run persistence and processing completion commit
atomically, while temporary failures retain a durable retry time. Internal and
isolated dev-box verification passed with real PostgreSQL and mocked external owners.

[Worker, recovery rules and verification](launch-processing.md).

## 4A — Ordered commands and recorded results

Site Coordination owns a private database containing ordered commands and their
results. Kubernetes and Slurm have separate streams. Acknowledgements advance
only after the next command's result is recorded atomically. Local and isolated
dev-box verification passed with real PostgreSQL and production codecs. Supervisor
and Runtime callers were simulated, with external owners mocked.

[Storage, protocol rules and verification](ordered-commands.md).

## 4B — Secure Site communication

Runtime callers use HTTPS and rotating credentials to pull their environment's
commands and report results. The platform verifies each identity through Kubernetes
TokenReview before accessing its Site Coordination store. Local and isolated
dev-box verification passed with real PostgreSQL, TLS and Kubernetes identities.
Native execution was simulated; real Slurm-host identity and installed endpoint
verification remain open.

[HTTP operations, credentials and verification](secure-site-communication.md).

## 4C — Runtime recovery

The shared Site Runtime worker saves pending commands and exact results in a local
SQLite journal. Restarts replay saved results or ask the native executor to recover
an uncertain action before advancing. Site Coordination keeps authoritative
commands and results in PostgreSQL. Local and dev-box tests passed with real
PostgreSQL, HTTPS, SQLite, persistent volumes and Kubernetes identities. Native
execution uses test doubles; real native submission recovery remains in 5B/6B.

[Worker, storage and verification](runtime-recovery.md).

## 4D — Execution lifecycle

Build preparation, submission and observation commands from the saved Run.
Apply Site Coordination's recorded results to the matching Run and Attempt, with
version checks and atomic persistence. Command acknowledgement and workload state
remain separate. The common flow uses the same interfaces for Kubernetes and Slurm.
Local and isolated dev-box verification passed with real PostgreSQL, HTTPS, SQLite
and Kubernetes identities. Native executors and external provider lookups use test doubles.

[Lifecycle rules, storage and verification](execution-lifecycle.md).

## 4E — Reliable supervision

The Run Supervisor polls durable PostgreSQL work and drives the 4D lifecycle.
It persists commands before dispatch, recovers interrupted work with fenced
leases, schedules retries and commits final Run state and a completion event together. The completion contract lives inside Metallo. The pipeline coordinator
consumes it through RabbitMQ and records its own receipt and progress atomically.

[Polling decision, recovery rules and verification](reliable-supervision.md).

## 5A — Kubernetes member permissions

Site Runtime resolves a trusted member binding and checks current permission before
native submission. Kubernetes enforces Project-scoped member impersonation, RBAC
and admission restrictions, including after membership revocation. The Job profile
has isolated native evidence. Service member impersonation, namespaced RBAC,
admission and scoped recovery/cleanup are implemented for Deployments and KServe,
with provider/native doubles verified. Their native qualification is recorded
separately under 5A.

[Access flow, native policy and verification](kubernetes-member-permissions.md).

## 5B — Reliable Kubernetes Job and service execution

The permanent Runtime Deployment uses the shared worker and real Kubernetes
executor. It saves native preparation and identity on its PVC, recovers the same
Job after restart, respects quotas and leaves batch retries to new Attempts.

Application Deployment and KServe executors apply, recover, observe and terminate
the saved service. Planning, Run restoration and command codecs preserve
`intent.serviceKind`. Runtime routes `queued-batch` to the Job executor and
`reconciled-service` by that discriminator to the appropriate service executor.
The shared worker journals delivery on its PVC alongside `jobs.sqlite` and
`services.sqlite`. Native service evidence and external-provider checks are
recorded separately in 5B.

[Execution, recovery, deployment and verification](kubernetes-jobs.md).

## 5C — Kubernetes member data isolation

For batch jobs, a preparation Pod downloads the latest code and dataset branch contents directly into an
Attempt's private PVC. The execution Pod mounts it and runs the saved command using
the fixed platform image. Output/log reads check current member access and exact
native identities. The installation includes the scoped workspace permissions;
current native and external-provider qualification is recorded in 5C.

Application and inference services reuse repository/PVC preparation, then execute
`preCommands` in an init container inside their serving Pod. Applications run the
saved command using the fixed platform image. Hugging Face uses KServe's installed
runtime image to serve the prepared model files. Private file/log reads and the
application/prediction proxy verify native ownership and current permission.
Mocked-provider checks pass; native evidence and real-provider connections retain
separate checkboxes under 5C.

[Repository preparation, private reads and verification](member-data-isolation.md).

## 5D — Site Runtime context

Site Runtime owns ordered local processing, recovery, current member access and
repository/workspace rules under `contexts/site_runtime`. Domain values and rules,
application workflows and ports, and technology-specific adaptors have separate
layers. The shared worker supports native executors through the same ports;
process startup and wiring remain in `services/`.

Execution Core retains Run/Attempt authority and Site Coordination retains the
command stream and recorded results. Runtime keeps the existing delivery and
native SQLite journals without a platform PostgreSQL database. Command contracts,
journal formats and configured storage locations remain unchanged. Refactor
verification is tracked separately from the historical evidence above.

[Ownership, ports, persistence and required verification](site-runtime-context.md).

## Run logs (#1056)

Status: proposal for review; nothing is implemented. A member's read of a Run's logs travels from the Site by a read
command and a separate upload route that the Runtime calls outward, bound to the command and to the read permit. The Run
page's log panel stays as it is and polls a logs route every 5 seconds for now, which needs one narrow exception to the
no-polling rule. The note sets the limits and timeouts that keep a slow read from holding up launches and cancellations,
what the logs contain, a bounded final copy of a failed service's logs kept for seven days, and the points that stay open: whether the copy
is one per Attempt or one per member, and how Slurm files are read.

[Transport, limits, contents and open points](run-logs.md).

## 7C — Member cancellation

Execution Core saves the first accepted cancellation request on the Run and wakes
its Supervisor atomically. Site Coordination fences starts that were never
appended; uncertain submissions retain their identity until recovered. The existing
Runtime cancellation is followed by observation of the actual native state.

[Request handling, delivery races and verification](run-cancellation.md).

## 8A — Pipeline records and interfaces

Pipeline Coordination owns an ordered Pipeline and its child steps. Internal
operations validate current Project access, exact Workload revisions and execution profiles,
then save the Pipeline, receipt and initial coordinator work atomically. The
coordinator launches and advances the steps as described in 8B.

[Records, internal operations, persistence and verification](pipeline-records.md).


## 8B — Pipeline coordinator

The coordinator saves each step's stable launch identity, submits it through
Execution Core and follows its Run. Batch completion arrives through RabbitMQ.
Services advance when the owner query confirms health and an endpoint. PostgreSQL
leases, receipts and saved progress recover interruptions and duplicate delivery.

[Flow, persistence, deployment and verification](pipeline-coordinator.md).


## 8C — Repository hand-offs and recovery

Training commands commit and push model files through Git/LFS. Each following
action is pinned, when it starts, to the commit its selected branch names then,
so it reads what the step before it pushed. Recover
these hand-offs with stable identities. Accepted cancellation fences delayed
launches and stops active Runs, including ready services.

[Repository hand-offs, cancellation, recovery and verification](pipeline-outputs.md).

## 8D — Kubernetes pipeline verification

The isolated dev-box lane runs saved Pipelines through the hosted coordinator,
launch processing, supervision, Site API and Runtime. It qualifies real GitLab
repository hand-offs, Kubernetes Jobs, application services and KServe, keeping
owner-provider doubles separate from native evidence.

[Installation, scenarios and verification](pipeline-verification.md).

## Pinned Run inputs (#767)

A Run records the exact commit of every repository it uses: the action's
repository at the commit `actions.yaml` was read from, the others at acceptance,
and a Pipeline step when it starts. A retry fetches the same commits, and a Site
that cannot prepare them rejects the Attempt for good.

[Inventory, decisions, contracts and verification](pinned-run-inputs.md).

## 8E — Repository actions and pipeline API

"Run all in order" prepares every action of a branch through Workload Portfolio,
in `actions.yaml` order, and starts one Pipeline over those exact revisions.
Public start, read and stop requests call it through the API.

[Flow, ownership, limits and verification](repository-pipelines.md).