# Execution Core — internal launch and execution planning

The stateless `ServiceAccess` query authorises current `runs.output.view` members
against the latest active Kubernetes service Attempt. Its public query requires
explicit publication in the accepted execution material and current accepted-work
continuation authority. Neither query creates a durable grant. The API and Runtime
use these queries for
[Run services](../../../../docs/decisions/0069-project-private-run-services.md);
bounded file/log/prediction permits retain their existing protocol.

This slice records a planned Run and its child Attempts with the initiating
Pharos member and Project, then resolves their provisioned account for the unique
Site. The Run owns the consistency boundary; Attempts have no separate write
repository.

## Baseline and provenance

The literal `legacy/` baseline supplies the existing workflow and provisioning
behaviour. `legacy/services/k8s-root-actor/src/routes/actions.ts` carries the
initiating GitLab user and Project through repository actions.
`legacy/services/waldur-k8s-site-agent/waldur_site_agent_k8s/client.py` provisions
member service accounts and namespace role bindings. Their provider identifiers
remain opaque bindings rather than canonical Pharos IDs.

The prior Metallo implementation supplies the Run record construction reused
here. Its saved source is the archive
`services/metallo/docs/reference/recovered-work.tar.gz` on
`temp/metallo-recovery`, commit `e34e33cfc`. Relevant archive members are:

- `files/metallo/contexts/execution_core/domain/planned_run.py`.
- `files/metallo/contexts/execution_core/application/commands.py`.
- `files/metallo/services/metallo/tests/unit/test_planned_run.py` and
  `files/metallo/services/metallo/tests/fixtures/typescript-planned-runs.json`.

The adapted Run retains member, Project, Workload/revision, required inputs and
session/service policies. Fixed `k8s` and `slurm` destinations replace the baseline
Environment/Site UUID pair; an opaque provisioned resource reference replaces the
old entitlement/resource-pool selection. Planning copies the exact execution
specification onto the Run and checks applied resource limits.
Command arguments retain their exact whitespace, including the final newline
of a YAML block script. Blank, non-string and NUL-containing arguments are rejected.
The Site execution contract uses a separate `CommandArgument` scalar for this
material. Field names and stored command bytes are unchanged; deploy the updated
platform and Site Runtime before submitting scripts that older consumers reject.

## Adaptation to the current skeleton

Runs and Attempts use canonical UUIDs and explicit `memberId`/`projectId` fields.
The recovered launch DTO still exposes `requestedByPartyId`; its planning input
now carries that identity as `memberId`. New Attempts inherit both identifiers
from their Run.

The recovered Run started at version two after an earlier launch transition.
This slice creates the planned aggregate directly at version one. Each subsequent
Attempt command advances the Run version through the foundation transaction.
Persistence uses the context-private `pharos_execution_core` database and current
aggregate snapshots, receipts and atomic typed messages. Every changed Run records
its `RunProjection` for the Run's canonical Project audience in the same
transaction. The restricted gateway derives the Project channel and preserves the
saved bytes. Browser delivery remains fail-closed until Party Governance and
Project publish their authority queries and the API adaptor can obtain the
principal's current authorised Project audiences. The old shared `platform.*`
repositories are replaced by the current PostgreSQL ports.

Restoration rejects invalid canonical identifiers and children whose member or
Project differs from their Run as corrupt saved authority. Malformed new command
evidence and Attempt timestamps produce recorded typed rejections instead.

## Internal usage

Compose `adaptors.postgres.execution_core(database, provisioning)` with an
Execution Core `Database` and an explicit implementation of the
`ProvisioningAccounts` port. `accounts_for(member_id, project_id)` returns
correlated, opaque account references; Site is implicit in configuration.

- `domain.planned_run.record_planned_run(...)` constructs records from supplied
  accepted evidence, explicit IDs and time.
- `ExecutionCore.store_planned_run(StorePlannedRunCommand(...))` records the Run
  and initial Attempt atomically; `get(run_id)` reads their saved state.
- `add_attempt(AddAttemptCommand(...))` adds an identity-inheriting child with an
  expected Run version. The later retry worker must invoke the 1C permission gate before starting.
- `resolve_account(run_id)` performs a fresh read-only provisioning lookup and
  distinguishes missing, ambiguous, mismatched and unavailable associations.

## Execution messages and current access

1C adapts architecture-reset `48e60e542` execution/workload contracts and the
Python Site protocol in recovered Metallo `e34e33cfc`. The six operations are
`queued-batch.start/cancel`, `interactive-session.start/terminate` and
`reconciled-service.apply/terminate`. See the [design and evidence](../../docs/epics/instantiate-and-run/dd.md#1c--execution-messages-and-permission-contracts).

Compose `application.message_construction.ExecutionMessages` with the existing
query port (`ExecutionCore.queries`), explicit provisioning and authority ports,
and the configured canonical Site ID. `start(...)` derives identity, resources,
sources and saved policies from the Run and rejects conflicting preparation.
`StartPreparation` is supplied by a trusted owner worker from the immutable
Workload revision and validated preparation results. The full executable intent
is not in the Run snapshot; querying that revision remains later integration.
Native references are correlated to the complete identity and Site.

`cancel(...)` checks the requesting member's current Project capability, retains
the initiator separately and does no account lookup. `PolicyCancellation` is an
internal trusted-worker capability, never a browser-selected bypass.
`before_start(...)` checks the saved execution, account binding and expiry
immediately before every native start, including delays and automatic retries.
Accepted work uses its recorded authority after personal Project suspension;
new launches and manual retries require the requester's current permission. The later
Site Coordination worker must also fence the lifecycle and reconcile existing
native effects; message construction does not establish delivery idempotency.

`SourceFetchAccess.for_fetch(...)` issues a Run/Attempt-bound, repository-scoped
`write_repository` grant for the saved accepted work and verified binding. Source references
select a repository and optional branch; a planned Run pins each to the commit it
was accepted with (`application/source_versions.py`, [design](../../docs/epics/instantiate-and-run/pinned-run-inputs.md)).
Runtime fetches that commit when the action prepares, retains a normal Git
checkout on the branch and keeps authorised write credentials available to action
commands until execution finishes or is cancelled.
Credentials remain outside durable messages. Real provider integration must
recover failed revocation durably.

Use `adaptors.execution_codec.encode/decode` at the transport boundary. Generated
strict models reject unknown, missing, null and malformed values; application
DTOs remain independent of Pydantic. Generate and check drift using
`uv run --project services/metallo --frozen python scripts/generate_site_execution.py`
(with `--check` for verification), or `pnpm generate:contracts`.

Provisioning, authority and credential doubles exist only in tests. The disposable
PostgreSQL and dev-box lanes reload saved Runs and exercise production codecs.
This slice adds no HTTP launch API, browser flow or executable worker. Live
provisioning, permission/credential providers and Kubernetes/Slurm execution
still require their own evidence.


## Local Party and Project view

Execution Core keeps its own copy of Beto's Project memberships, Party states and provider
bindings (`pharos.project_views`, `project_members`, `party_views`, `party_bindings`,
`project_access_invalidations`, `view_state`, `view_checkpoints`), applied from Beto's
Project, access, authentication and binding events by the shared consume loop with consumer
receipts. An access change newer than the saved snapshot records its revision for the
affected Parties, who stay denied until a complete snapshot reaches it, so a late older
snapshot cannot restore them. Beto's latest Project authority (`pharos.project_authority`)
is kept per Project without receipts; a decision defers unless it is ready, unexpired and
describes the Project revision the view holds (#798). The execution worker runs the eight
consumers with `EXECUTION_CORE_VIEWS_BROKER_URL`, and the synchroniser loads and repairs the
copy from Beto's checkpoints for `PROJECT_SOURCE_SCOPE`; the view answers only once it
matches them.
`PartyProjectAuthority` decides `runs.view`, `runs.output.view`, `runs.launch`,
`runs.retry` and `runs.cancel` by Beto's rule: an active Project, an
executing role, an enabled and eligible Party, and an established provider binding at the
membership's recorded revision; an unknown Party or binding defers the decision. The
execution worker confirms every allowing current-person decision with Beto's
`CheckProjectCapability` through the same identity (#755). Accepted planning,
automatic retries and native dispatch do not re-check the initiator after removal.
Production continuation still requires exact accepted-work evidence and the
native retention/release contract in #765/#766; retaining a member's native
identity must not retain their personal Project access. Runtime-only operation
authority and independent management authority remain activation requirements.
[Design](../../docs/epics/instantiate-and-run/party-project-views.md),
[run access](../../docs/epics/instantiate-and-run/run-access.md).

## Project resource lookup

Compose `application.resource_lookup.ProjectResourceLookup(provisioning)` with
an explicit `ProvisioningResources` implementation. `resources_for(project_id)`
returns all execution resources for the canonical Project ID, including resources
on both configured Sites. Each resource carries its opaque reference, readiness,
confirmed limits and environment. The query validates correlation and copies the
response into an immutable result. Every call reads the owner again.

An unknown Project raises `project_not_found`; a known Project may return no
resources. Invalid or mismatched responses are rejected, while
`ResourceLookupUnavailable` remains a retryable failure. A successful read does
not grant launch permission or reserve resources. Planning checks the requested environment and resource; 3A introduces no persistence or delivery work.

Provisioning owns the real query over its records and provider-field mapping.
Only test doubles are currently available. See the [3A design and evidence](../../docs/epics/instantiate-and-run/dd.md#3a--project-resource-lookup).


## Launch acceptance

Compose `adaptors.postgres.launch_acceptance(database, workloads, authority)`
with explicit Workload Portfolio and Project permission ports. Call
`request(RequestRunLaunchCommand(...), at=...)` to check and save the request, or
`get(request_id)` to read its recorded decision internally. `planning` means accepted
for later planning; a rejected result includes its reason. Accepted
`planning_input()` carries the member and Project into the 1B planning boundary.

The original launch is reused for concurrent deliveries sharing its Workload and
retry key. Changed intent conflicts. PostgreSQL stores the decision and receipt
atomically. Launch browser notifications are deferred to Milestone 2 (7A), with
an explicit `None` projection registration in the shared writer. Context migration
`0001_launch_retry.sql` adds the launch retry and delivery uniqueness indexes. Owner lookups happen outside the transaction, with test-only
doubles until real integrations exist. See the
[3B DD](../../docs/epics/instantiate-and-run/launch-acceptance.md) for rules and evidence.

A new person launch saves the evidence Beto's allowing answer carried: Project
revision, management generation, authority revision and `valid_until`. It also
saves its absolute accepted-work deadline: the acceptance time plus the
deployment's `ACCEPTED_WORK_LIFETIME_DAYS` (14 by default), or the request's earlier
`deadline`. An allow without that evidence defers the launch. A replay returns the
original record, and planning copies it onto the Run. A Site limit waits for #765,
and continuation enforcement follows in #813. See the
[accepted-work design](../../docs/epics/instantiate-and-run/accepted-work.md) (#766).

`LaunchAcceptance.request_accepted_pipeline` is the internal continuation path
for a saved Pipeline step. It confirms the exact ready step through Pipeline
Coordination's published query and stores its owner revision and provenance with
the launch. It preserves the accepted Workload revision after personal suspension
or later Workload revision/retirement. Member-facing launch requests retain their
current permission checks and cannot supply this provenance. Already proven
launches recover with their original identity and evidence; older snapshots
without Pipeline provenance remain readable and cancellable, but cannot be
adopted as accepted continuation. See the
[coordinator recovery rules](../../docs/epics/instantiate-and-run/pipeline-coordinator.md#records-and-recovery).

`adaptors.postgres.accepted_work(database)` lists every accepted launch and whether
it still needs its initiator's native account, for the Site owner's initial load and
repair (#765): required from acceptance until planning refused it or the Site has
finished with its Run, then released at the revision its release was published at,
so a repair receives a release whose event it missed (#766). It is paged by launch
ID, with the deadline, a revision, the management generation and the member's
`site` binding. Enrolled Metallo callers
read it as the internal query `ListAcceptedWork`, answered in `metallo-requests`
([internal queries](../../docs/epics/instantiate-and-run/request-transport.md#internal-queries)).
A final Run stays required until the Site confirms its cleanup: every Attempt the
Site ran must report `cleanupComplete` with its terminal observation
(`application/accepted_work.py`, `cleanup_confirmed`). An unknown cleanup keeps
the account. Acceptance saves the member's exact Site, binding, generation, allocation,
account reference and confirmed native UID. Planning keeps that allocation and subsequent
checks refuse a replacement UID. Missing, stale, incomplete or superseded-generation reports
prevent admission until confirmation; a fresh complete report without the binding establishes absence
for that allocation. If no matching account is confirmed, this also reports `account_unconfirmed`.
An account awaiting Site confirmation (`AccountAwaitingSite`) saves nothing on a new launch:
the request host answers `account_unconfirmed` with a fixed message, and the same request can
be sent again. An owner outage answers `temporarily_unavailable` (#1119).
Resource Provisioning documents the [age setting, credential scope and activation prerequisites](../resource_provisioning/README.md#report-freshness-and-activation).

Older snapshots without native identity remain readable and retain the binding/generation
check, so a pre-change queued Run or Pipeline may use a recreated native UID. At rollout,
inventory these items and record their latest saved accepted-work deadline as this risk's
end date. Existing deadlines remain unchanged; the fourteen-day default applies to new
acceptances. Drain or cancel items without a deadline before enabling workers. A calendar
end date depends on that deployment inventory. Expiry prevents new starts and preserves
the existing cleanup requirements for prepared work.

The change events,
`execution_core.run_launch.accepted_work_changed.v1` and
`execution_core.run.accepted_work_changed.v1`, commit with acceptance, planning's
Run or refusal, a stop before planning, and the completion of a Run whose cleanup
the Site confirmed, and reach Resource Provisioning (#837) through the pipeline
coordinator's relay.

The Site's permanent member-consent refusal rejects an unstarted Attempt. If
stage-in or compute has already finished, its failed observation includes a fixed
`failureCode` and confirms cleanup. Neither outcome starts another automatic
Attempt. The same Run transaction commits completion and a typed sign-in-again
notice; the browser displays one notice per Run to its initiating member.
Owner timeouts and unavailable services remain pending and produce no notice.

Succeeded and failed Runs also appear in the initiating member's completion
notifications, with an authorised canonical Run link. New completion facts can
send a minimal email through the configured relay. The
[completion delivery contract](completion-notifications.md) defines current
authority, browser recovery, migration and uncertain SMTP outcomes.

Continuation stops at that deadline. Planning refuses a launch past it. An
automatic retry is not created after it. `before_start` refuses a new native
start, fetch or source credential after it with `accepted_work_expired`. Start and
preparation commands carry it to the Site as `acceptedUntil`, and the Runtime
creates nothing new at or after it, while existing work keeps running. Expired
work ends as rejected (`accepted-work-expired`) instead of retrying. A
Pipeline's declared steps share the Pipeline's saved acceptance and deadline, and
a step after it is refused. Legacy commands without the deadline are replaced only
after Site Coordination fences their old transport identity; appended commands
retain their exact bytes for recovery. Prepared work waits for confirmed Site
cleanup before its terminal expiry is published. The same three points stop once
the Project is no longer active or its management generation changed
(`accepted_work_revoked`). Known closure, an increased management generation or
retirement of the exact accepted Site binding is a permanent fence, including
while source refresh is pending. Unknown or older evidence defers new work.
The supervisor saves a policy cancellation before contacting the Site, preserves
in-flight command identities and resumes cancellation after restart. Prepared
work waits for the Site's cleanup acknowledgement; submitted work waits for native
terminal evidence. Ordinary membership removal preserves accepted work's binding,
native UID and original deadline. Resource Provisioning's confirmed-account reader
serves both admission and retained execution; current membership is checked only
for new admission.

When the Site refuses a preparation or start that the supervisor retries, the
current Attempt keeps the Site's code as its waiting reason (`Attempt.waiting`),
with the time it began. A repeated code saves nothing, and any progress of the
Attempt clears it. Run reads return the code as `status_reason`, and a fixed
explanation from `application/run_waits.py` as `status_message`; the Site's own
text never reaches members. A refused observation leaves the Attempt unchanged,
because the observation carries the Attempt revision that a new reason would
advance.

While a preparation or start has no result yet, the supervisor also reads Site
Coordination's progress for it: the Runtime's deferral code, or
`site-queue-busy` when an earlier command for the same Site is the one waiting.
An observation of an unfinished workload can carry a native reason, which is
saved with the observation and keeps the time it began while it repeats. Every
Runtime and native code has its own text in `application/run_waits.py`.

A failed batch observation may also carry bounded native failure facts after
cleanup. The Run persists them with its Attempt and returns a fixed summary in
the existing `status_reason` and `status_message` fields: failed stage, opaque
provider job/state and available exit/signal values. Process and derived-step
results remain separate, including a zero process exit alongside a scheduler
failure. The summary contains no provider error message or job output; the
existing authorised Run-log path carries stdout and stderr.


## Member runs and actions

`adaptors.postgres.action_launches(...)` and `repository_runs(...)` compose the
member-facing use cases behind the [runtime endpoints](../../docs/epics/instantiate-and-run/runtime-endpoints.md).
`ActionLaunches.launch` prepares a repository action's Workload through the
`ActionWorkloads` port and requests its launch, named by the command ID.
`RepositoryRuns` lists and reads a repository's or namespace's jobs and deployments
and stops one with `StopRunCommand`, whose arrival time is not material input. Reading
or stopping a Run also works at its repository's last known path after a rename or
deletion, or while the provider cannot answer, through the directory's `describe`. The
`RepositoryDirectory` port resolves Themelio's repository paths and `WorkloadNames`
names each Run's action. `metallo-requests` composes `RepositoryDirectory`, and the
execution worker `SourceVersions`, over Source Control
(`metallo.adaptors.source_control_owners`, #796). `adaptors.requests` maps the typed API requests, and
`adaptors.run_reads` lists saved Runs using the `0007_run_listing.sql` indexes.

The authorised detail query adds `application.run_configuration`'s immutable
configuration: the saved Workload ID and revision, execution environment, named
parameter evidence, requested and admitted resource demand, source repository
selections and commit pins, and wall-time limit. It reads only the Run snapshot;
it performs no later Workload or repository query. Commands, environment values
and provider account references are excluded. Named parameters preserve legacy
unknown, recorded empty and deliberately hidden values separately. Pending
launches have no planned configuration, and list/realtime projections omit it.
The [runtime endpoint contract](../../docs/epics/instantiate-and-run/runtime-endpoints.md#reading-runs)
defines the HTTP and retained UI behaviour.


## Private Run logs

`RepositoryRunLogs` resolves the authorised saved Run and latest Attempt, checks
`runs.output.view`, and returns only that viewer's unexpired copy. A separate
`request_run_logs` command saves an immutable sixty-second private-read permit
with the native target. Its command identity, target, byte limit and expiry are
stable on retry. Queries create no work.

The execution worker scans those durable permit intents and rechecks permission
before handing them to Site Coordination. The Site stream admits at most four
pending log reads; normal execution commands are unaffected. Uploaded bytes live
in the context-owned expiring copy table, separate from Run state, events and
receipts. Copies older than ten seconds request a refresh. The browser's log
panel polls every five seconds under the existing exception.

`RUN_LOG_READ_ENVIRONMENTS` defaults to empty on both compositions. Upgrade
Runtime before enabling `k8s` or `slurm`. A failed service's final copy, which the
Runtime reads before it removes the service, waits in `pharos.final_run_log_copies`,
one row per Attempt, for seven days, and answers the logs route once the service has
ended. See the [Run logs design](../../docs/epics/instantiate-and-run/run-logs.md) for scope.

## Planning a saved launch

Compose `adaptors.postgres.run_planning(database, workloads, authority, provisioning)`
with the exact Workload-revision query, Project permission port and provisioning
resource query. `plan(PlanRunCommand(...))` reads the accepted launch, checks its
chosen environment and applied limits, and persists one Run and initial Attempt.
External owners remain test-only doubles until their real connections are supplied.
See the [planning DD](../../docs/epics/instantiate-and-run/run-planning.md) for
admission, saved specifications, deterministic IDs and verification evidence.


## Processing accepted launches

Compose `adaptors.postgres.launch_worker(database, workloads, authority, provisioning,
owner=...)` with explicit owner implementations. `run_once()` claims due launches
and invokes planning. `services.launch_worker.run_launch_worker(worker, stop)` runs
the stoppable polling loop. The processing query exposes pending work, retries and
the saved Run or rejection. Acceptance and its work record commit together. The
Run and processing completion also commit together, guarded by the current lease.
See the [worker DD](../../docs/epics/instantiate-and-run/launch-processing.md).
The accepted launch is the authority for this immutable processing step; new
manual retry and cancellation requests use their requester's current capability.