> For AI agents: the complete documentation index is available at /llms.txt, the full documentation bundle is available at /llms-full.txt. # Execution Core — internal launch and execution planning The stateless `ServiceAccess` query authorises current `runs.output.view` members against the latest active Kubernetes service Attempt. Its public query requires explicit publication in the accepted execution material and current accepted-work continuation authority. Neither query creates a durable grant. The API and Runtime use these queries for [Run services](/reference/docs/decisions/0069-project-private-run-services.md); bounded file/log/prediction permits retain their existing protocol. This slice records a planned Run and its child Attempts with the initiating Pharos member and Project, then resolves their provisioned account for the unique Site. The Run owns the consistency boundary; Attempts have no separate write repository. ## Baseline and provenance The literal `legacy/` baseline supplies the existing workflow and provisioning behaviour. `legacy/services/k8s-root-actor/src/routes/actions.ts` carries the initiating GitLab user and Project through repository actions. `legacy/services/waldur-k8s-site-agent/waldur_site_agent_k8s/client.py` provisions member service accounts and namespace role bindings. Their provider identifiers remain opaque bindings rather than canonical Pharos IDs. The prior Metallo implementation supplies the Run record construction reused here. Its saved source is the archive `services/metallo/docs/reference/recovered-work.tar.gz` on `temp/metallo-recovery`, commit `e34e33cfc`. Relevant archive members are: - `files/metallo/contexts/execution_core/domain/planned_run.py`. - `files/metallo/contexts/execution_core/application/commands.py`. - `files/metallo/services/metallo/tests/unit/test_planned_run.py` and `files/metallo/services/metallo/tests/fixtures/typescript-planned-runs.json`. The adapted Run retains member, Project, Workload/revision, required inputs and session/service policies. Fixed `k8s` and `slurm` destinations replace the baseline Environment/Site UUID pair; an opaque provisioned resource reference replaces the old entitlement/resource-pool selection. Planning copies the exact execution specification onto the Run and checks applied resource limits. Command arguments retain their exact whitespace, including the final newline of a YAML block script. Blank, non-string and NUL-containing arguments are rejected. The Site execution contract uses a separate `CommandArgument` scalar for this material. Field names and stored command bytes are unchanged; deploy the updated platform and Site Runtime before submitting scripts that older consumers reject. ## Adaptation to the current skeleton Runs and Attempts use canonical UUIDs and explicit `memberId`/`projectId` fields. The recovered launch DTO still exposes `requestedByPartyId`; its planning input now carries that identity as `memberId`. New Attempts inherit both identifiers from their Run. The recovered Run started at version two after an earlier launch transition. This slice creates the planned aggregate directly at version one. Each subsequent Attempt command advances the Run version through the foundation transaction. Persistence uses the context-private `pharos_execution_core` database and current aggregate snapshots, receipts and atomic typed messages. Every changed Run records its `RunProjection` for the Run's canonical Project audience in the same transaction. The restricted gateway derives the Project channel and preserves the saved bytes. Browser delivery remains fail-closed until Party Governance and Project publish their authority queries and the API adaptor can obtain the principal's current authorised Project audiences. The old shared `platform.*` repositories are replaced by the current PostgreSQL ports. Restoration rejects invalid canonical identifiers and children whose member or Project differs from their Run as corrupt saved authority. Malformed new command evidence and Attempt timestamps produce recorded typed rejections instead. ## Internal usage Compose `adaptors.postgres.execution_core(database, provisioning)` with an Execution Core `Database` and an explicit implementation of the `ProvisioningAccounts` port. `accounts_for(member_id, project_id)` returns correlated, opaque account references; Site is implicit in configuration. - `domain.planned_run.record_planned_run(...)` constructs records from supplied accepted evidence, explicit IDs and time. - `ExecutionCore.store_planned_run(StorePlannedRunCommand(...))` records the Run and initial Attempt atomically; `get(run_id)` reads their saved state. - `add_attempt(AddAttemptCommand(...))` adds an identity-inheriting child with an expected Run version. The later retry worker must invoke the 1C permission gate before starting. - `resolve_account(run_id)` performs a fresh read-only provisioning lookup and distinguishes missing, ambiguous, mismatched and unavailable associations. ## Execution messages and current access 1C adapts architecture-reset `48e60e542` execution/workload contracts and the Python Site protocol in recovered Metallo `e34e33cfc`. The six operations are `queued-batch.start/cancel`, `interactive-session.start/terminate` and `reconciled-service.apply/terminate`. See the design and evidence. Compose `application.message_construction.ExecutionMessages` with the existing query port (`ExecutionCore.queries`), explicit provisioning and authority ports, and the configured canonical Site ID. `start(...)` derives identity, resources, sources and saved policies from the Run and rejects conflicting preparation. `StartPreparation` is supplied by a trusted owner worker from the immutable Workload revision and validated preparation results. The full executable intent is not in the Run snapshot; querying that revision remains later integration. Native references are correlated to the complete identity and Site. `cancel(...)` checks the requesting member's current Project capability, retains the initiator separately and does no account lookup. `PolicyCancellation` is an internal trusted-worker capability, never a browser-selected bypass. `before_start(...)` checks the saved execution, account binding and expiry immediately before every native start, including delays and automatic retries. Accepted work uses its recorded authority after personal Project suspension; new launches and manual retries require the requester's current permission. The later Site Coordination worker must also fence the lifecycle and reconcile existing native effects; message construction does not establish delivery idempotency. `SourceFetchAccess.for_fetch(...)` issues a Run/Attempt-bound, repository-scoped `write_repository` grant for the saved accepted work and verified binding. Source references select a repository and optional branch; a planned Run pins each to the commit it was accepted with (`application/source_versions.py`, design). Runtime fetches that commit when the action prepares, retains a normal Git checkout on the branch and keeps authorised write credentials available to action commands until execution finishes or is cancelled. Credentials remain outside durable messages. Real provider integration must recover failed revocation durably. Use `adaptors.execution_codec.encode/decode` at the transport boundary. Generated strict models reject unknown, missing, null and malformed values; application DTOs remain independent of Pydantic. Generate and check drift using `uv run --project services/metallo --frozen python scripts/generate_site_execution.py` (with `--check` for verification), or `pnpm generate:contracts`. Provisioning, authority and credential doubles exist only in tests. The disposable PostgreSQL and dev-box lanes reload saved Runs and exercise production codecs. This slice adds no HTTP launch API, browser flow or executable worker. Live provisioning, permission/credential providers and Kubernetes/Slurm execution still require their own evidence. ## Local Party and Project view Execution Core keeps its own copy of Beto's Project memberships, Party states and provider bindings (`pharos.project_views`, `project_members`, `party_views`, `party_bindings`, `project_access_invalidations`, `view_state`, `view_checkpoints`), applied from Beto's Project, access, authentication and binding events by the shared consume loop with consumer receipts. An access change newer than the saved snapshot records its revision for the affected Parties, who stay denied until a complete snapshot reaches it, so a late older snapshot cannot restore them. Beto's latest Project authority (`pharos.project_authority`) is kept per Project without receipts; a decision defers unless it is ready, unexpired and describes the Project revision the view holds (#798). The execution worker runs the eight consumers with `EXECUTION_CORE_VIEWS_BROKER_URL`, and the synchroniser loads and repairs the copy from Beto's checkpoints for `PROJECT_SOURCE_SCOPE`; the view answers only once it matches them. `PartyProjectAuthority` decides `runs.view`, `runs.output.view`, `runs.launch`, `runs.retry` and `runs.cancel` by Beto's rule: an active Project, an executing role, an enabled and eligible Party, and an established provider binding at the membership's recorded revision; an unknown Party or binding defers the decision. The execution worker confirms every allowing current-person decision with Beto's `CheckProjectCapability` through the same identity (#755). Accepted planning, automatic retries and native dispatch do not re-check the initiator after removal. Production continuation still requires exact accepted-work evidence and the native retention/release contract in #765/#766; retaining a member's native identity must not retain their personal Project access. Runtime-only operation authority and independent management authority remain activation requirements. Design, run access. ## Project resource lookup Compose `application.resource_lookup.ProjectResourceLookup(provisioning)` with an explicit `ProvisioningResources` implementation. `resources_for(project_id)` returns all execution resources for the canonical Project ID, including resources on both configured Sites. Each resource carries its opaque reference, readiness, confirmed limits and environment. The query validates correlation and copies the response into an immutable result. Every call reads the owner again. An unknown Project raises `project_not_found`; a known Project may return no resources. Invalid or mismatched responses are rejected, while `ResourceLookupUnavailable` remains a retryable failure. A successful read does not grant launch permission or reserve resources. Planning checks the requested environment and resource; 3A introduces no persistence or delivery work. Provisioning owns the real query over its records and provider-field mapping. Only test doubles are currently available. See the 3A design and evidence. ## Launch acceptance Compose `adaptors.postgres.launch_acceptance(database, workloads, authority)` with explicit Workload Portfolio and Project permission ports. Call `request(RequestRunLaunchCommand(...), at=...)` to check and save the request, or `get(request_id)` to read its recorded decision internally. `planning` means accepted for later planning; a rejected result includes its reason. Accepted `planning_input()` carries the member and Project into the 1B planning boundary. The original launch is reused for concurrent deliveries sharing its Workload and retry key. Changed intent conflicts. PostgreSQL stores the decision and receipt atomically. Launch browser notifications are deferred to Milestone 2 (7A), with an explicit `None` projection registration in the shared writer. Context migration `0001_launch_retry.sql` adds the launch retry and delivery uniqueness indexes. Owner lookups happen outside the transaction, with test-only doubles until real integrations exist. See the 3B DD for rules and evidence. A new person launch saves the evidence Beto's allowing answer carried: Project revision, management generation, authority revision and `valid_until`. It also saves its absolute accepted-work deadline: the acceptance time plus the deployment's `ACCEPTED_WORK_LIFETIME_DAYS` (14 by default), or the request's earlier `deadline`. An allow without that evidence defers the launch. A replay returns the original record, and planning copies it onto the Run. A Site limit waits for #765, and continuation enforcement follows in #813. See the accepted-work design (#766). `LaunchAcceptance.request_accepted_pipeline` is the internal continuation path for a saved Pipeline step. It confirms the exact ready step through Pipeline Coordination's published query and stores its owner revision and provenance with the launch. It preserves the accepted Workload revision after personal suspension or later Workload revision/retirement. Member-facing launch requests retain their current permission checks and cannot supply this provenance. Already proven launches recover with their original identity and evidence; older snapshots without Pipeline provenance remain readable and cancellable, but cannot be adopted as accepted continuation. See the coordinator recovery rules. `adaptors.postgres.accepted_work(database)` lists every accepted launch and whether it still needs its initiator's native account, for the Site owner's initial load and repair (#765): required from acceptance until planning refused it or the Site has finished with its Run, then released at the revision its release was published at, so a repair receives a release whose event it missed (#766). It is paged by launch ID, with the deadline, a revision, the management generation and the member's `site` binding. Enrolled Metallo callers read it as the internal query `ListAcceptedWork`, answered in `metallo-requests` (internal queries). A final Run stays required until the Site confirms its cleanup: every Attempt the Site ran must report `cleanupComplete` with its terminal observation (`application/accepted_work.py`, `cleanup_confirmed`). An unknown cleanup keeps the account. Acceptance saves the member's exact Site, binding, generation, allocation, account reference and confirmed native UID. Planning keeps that allocation and subsequent checks refuse a replacement UID. Missing, stale, incomplete or superseded-generation reports prevent admission until confirmation; a fresh complete report without the binding establishes absence for that allocation. If no matching account is confirmed, this also reports `account_unconfirmed`. An account awaiting Site confirmation (`AccountAwaitingSite`) saves nothing on a new launch: the request host answers `account_unconfirmed` with a fixed message, and the same request can be sent again. An owner outage answers `temporarily_unavailable` (#1119). Resource Provisioning documents the [age setting, credential scope and activation prerequisites](/reference/services/metallo/contexts/resource_provisioning/README.md#report-freshness-and-activation). Older snapshots without native identity remain readable and retain the binding/generation check, so a pre-change queued Run or Pipeline may use a recreated native UID. At rollout, inventory these items and record their latest saved accepted-work deadline as this risk's end date. Existing deadlines remain unchanged; the fourteen-day default applies to new acceptances. Drain or cancel items without a deadline before enabling workers. A calendar end date depends on that deployment inventory. Expiry prevents new starts and preserves the existing cleanup requirements for prepared work. The change events, `execution_core.run_launch.accepted_work_changed.v1` and `execution_core.run.accepted_work_changed.v1`, commit with acceptance, planning's Run or refusal, a stop before planning, and the completion of a Run whose cleanup the Site confirmed, and reach Resource Provisioning (#837) through the pipeline coordinator's relay. The Site's permanent member-consent refusal rejects an unstarted Attempt. If stage-in or compute has already finished, its failed observation includes a fixed `failureCode` and confirms cleanup. Neither outcome starts another automatic Attempt. The same Run transaction commits completion and a typed sign-in-again notice; the browser displays one notice per Run to its initiating member. Owner timeouts and unavailable services remain pending and produce no notice. Succeeded and failed Runs also appear in the initiating member's completion notifications, with an authorised canonical Run link. New completion facts can send a minimal email through the configured relay. The completion delivery contract defines current authority, browser recovery, migration and uncertain SMTP outcomes. Continuation stops at that deadline. Planning refuses a launch past it. An automatic retry is not created after it. `before_start` refuses a new native start, fetch or source credential after it with `accepted_work_expired`. Start and preparation commands carry it to the Site as `acceptedUntil`, and the Runtime creates nothing new at or after it, while existing work keeps running. Expired work ends as rejected (`accepted-work-expired`) instead of retrying. A Pipeline's declared steps share the Pipeline's saved acceptance and deadline, and a step after it is refused. Legacy commands without the deadline are replaced only after Site Coordination fences their old transport identity; appended commands retain their exact bytes for recovery. Prepared work waits for confirmed Site cleanup before its terminal expiry is published. The same three points stop once the Project is no longer active or its management generation changed (`accepted_work_revoked`). Known closure, an increased management generation or retirement of the exact accepted Site binding is a permanent fence, including while source refresh is pending. Unknown or older evidence defers new work. The supervisor saves a policy cancellation before contacting the Site, preserves in-flight command identities and resumes cancellation after restart. Prepared work waits for the Site's cleanup acknowledgement; submitted work waits for native terminal evidence. Ordinary membership removal preserves accepted work's binding, native UID and original deadline. Resource Provisioning's confirmed-account reader serves both admission and retained execution; current membership is checked only for new admission. When the Site refuses a preparation or start that the supervisor retries, the current Attempt keeps the Site's code as its waiting reason (`Attempt.waiting`), with the time it began. A repeated code saves nothing, and any progress of the Attempt clears it. Run reads return the code as `status_reason`, and a fixed explanation from `application/run_waits.py` as `status_message`; the Site's own text never reaches members. A refused observation leaves the Attempt unchanged, because the observation carries the Attempt revision that a new reason would advance. While a preparation or start has no result yet, the supervisor also reads Site Coordination's progress for it: the Runtime's deferral code, or `site-queue-busy` when an earlier command for the same Site is the one waiting. An observation of an unfinished workload can carry a native reason, which is saved with the observation and keeps the time it began while it repeats. Every Runtime and native code has its own text in `application/run_waits.py`. A failed batch observation may also carry bounded native failure facts after cleanup. The Run persists them with its Attempt and returns a fixed summary in the existing `status_reason` and `status_message` fields: failed stage, opaque provider job/state and available exit/signal values. Process and derived-step results remain separate, including a zero process exit alongside a scheduler failure. The summary contains no provider error message or job output; the existing authorised Run-log path carries stdout and stderr. ## Member runs and actions `adaptors.postgres.action_launches(...)` and `repository_runs(...)` compose the member-facing use cases behind the runtime endpoints. `ActionLaunches.launch` prepares a repository action's Workload through the `ActionWorkloads` port and requests its launch, named by the command ID. `RepositoryRuns` lists and reads a repository's or namespace's jobs and deployments and stops one with `StopRunCommand`, whose arrival time is not material input. Reading or stopping a Run also works at its repository's last known path after a rename or deletion, or while the provider cannot answer, through the directory's `describe`. The `RepositoryDirectory` port resolves Themelio's repository paths and `WorkloadNames` names each Run's action. `metallo-requests` composes `RepositoryDirectory`, and the execution worker `SourceVersions`, over Source Control (`metallo.adaptors.source_control_owners`, #796). `adaptors.requests` maps the typed API requests, and `adaptors.run_reads` lists saved Runs using the `0007_run_listing.sql` indexes. The authorised detail query adds `application.run_configuration`'s immutable configuration: the saved Workload ID and revision, execution environment, named parameter evidence, requested and admitted resource demand, source repository selections and commit pins, and wall-time limit. It reads only the Run snapshot; it performs no later Workload or repository query. Commands, environment values and provider account references are excluded. Named parameters preserve legacy unknown, recorded empty and deliberately hidden values separately. Pending launches have no planned configuration, and list/realtime projections omit it. The runtime endpoint contract defines the HTTP and retained UI behaviour. ## Private Run logs `RepositoryRunLogs` resolves the authorised saved Run and latest Attempt, checks `runs.output.view`, and returns only that viewer's unexpired copy. A separate `request_run_logs` command saves an immutable sixty-second private-read permit with the native target. Its command identity, target, byte limit and expiry are stable on retry. Queries create no work. The execution worker scans those durable permit intents and rechecks permission before handing them to Site Coordination. The Site stream admits at most four pending log reads; normal execution commands are unaffected. Uploaded bytes live in the context-owned expiring copy table, separate from Run state, events and receipts. Copies older than ten seconds request a refresh. The browser's log panel polls every five seconds under the existing exception. `RUN_LOG_READ_ENVIRONMENTS` defaults to empty on both compositions. Upgrade Runtime before enabling `k8s` or `slurm`. A failed service's final copy, which the Runtime reads before it removes the service, waits in `pharos.final_run_log_copies`, one row per Attempt, for seven days, and answers the logs route once the service has ended. See the Run logs design for scope. ## Planning a saved launch Compose `adaptors.postgres.run_planning(database, workloads, authority, provisioning)` with the exact Workload-revision query, Project permission port and provisioning resource query. `plan(PlanRunCommand(...))` reads the accepted launch, checks its chosen environment and applied limits, and persists one Run and initial Attempt. External owners remain test-only doubles until their real connections are supplied. See the planning DD for admission, saved specifications, deterministic IDs and verification evidence. ## Processing accepted launches Compose `adaptors.postgres.launch_worker(database, workloads, authority, provisioning, owner=...)` with explicit owner implementations. `run_once()` claims due launches and invokes planning. `services.launch_worker.run_launch_worker(worker, stop)` runs the stoppable polling loop. The processing query exposes pending work, retries and the saved Run or rejection. Acceptance and its work record commit together. The Run and processing completion also commit together, guarded by the current lease. See the worker DD. The accepted launch is the authority for this immutable processing step; new manual retry and cancellation requests use their requester's current capability.
Page source Source snapshot · Revision `f928834c8`.