# DD — Accepted work: evidence, continuation and native retention (#766)

[Run access](run-access.md) · [Common design](dd.md) · [Tasks and completion status](adr.md) ·
[Site accounts #765](https://github.com/pharos-ai-factory/pharos-platform/issues/765) ·
[Journey #771](https://github.com/pharos-ai-factory/pharos-platform/issues/771)

Status: proposed. Steps 1–5 are implemented: the query, its RabbitMQ contracts,
saved accounts and the change events, which Resource Provisioning consumes (#837), including decision 4's management-generation check. Decisions 3 and 6 are agreed
([configured deadline and expiry](run-access.md#configured-deadline-and-expiry);
decision 6 with the Site account owners on #766).

A person's decision accepts work. The work then runs under that recorded decision,
not under the person's current access, so it can finish after the person leaves
the Pharos Project. [Run access](run-access.md) states the policy. This note
defines the mechanism:
- the record each accepting context saves;
- how continuation uses it;
- how a Pipeline's later steps launch without a new person decision;
- the facts that tell the Site owner when a retained native account can go.

## Decisions

1. **Evidence comes from Beto's allowing answer.** A launch, manual retry or
   Pipeline start already asks Beto's `CheckProjectCapability` at decision time.
   Its allowing answer carries the Project revision, management generation,
   authority revision and `valid_until`. The accepting context saves these values,
   together with the decided person, capability and time, as `AcceptanceEvidence`.
   An allowing answer without that evidence, or one already past `valid_until`,
   defers the work instead of accepting it.
2. **The record lives on the accepting aggregate.** The Launch (Execution Core)
   and the Pipeline (Pipeline Coordination) save the evidence with their decision.
   Planning copies the Launch's evidence onto the Run. A replay returns the
   original record unchanged, so it can renew neither `valid_until` nor the deadline.
3. **One deadline per accepted work (agreed on #766).** Save an absolute deadline at
   acceptance. The deployment sets the maximum lifetime with
   `ACCEPTED_WORK_LIFETIME_DAYS`: whole days from 1 to 365, default and initial value
   14, which covers the twelve-day training case. The request's own earlier
   `deadline` wins; a Site limit would too, but no Site publishes one yet (#765). A
   Pipeline's steps share its original deadline. Configuration changes affect new
   acceptances; replay, retries and credential renewal preserve the saved deadline.
   Pipeline starts take the same setting: their production factories require it, so
   none falls back to the 14-day constant.

   Expiry stops new compute dispatch, later steps and automatic retries. It does
   not itself terminate running native work. Scoped observation, stage-out,
   authorised cancellation and cleanup continue until their obligations finish,
   after which the retained account can be released. Fresh work needs a new
   authorised person decision.
4. **Continuation uses the record.** #799 already stopped re-checking the
   initiator for planning, native dispatch, Run-scoped source credentials and
   automatic retries. These check the saved deadline instead: planning, automatic
   retries and `before_start` stop at it. The deadline also travels to the Site as
   `acceptedUntil` on every start and preparation command. The Runtime checks it
   against its own clock right before it creates anything new, so a command queued
   before the deadline cannot create a workspace, Job or service after it. Anything
   that already exists keeps being recovered, observed, cancelled and cleaned up.
   Expired work ends instead of retrying: an Attempt that nothing reached the Site
   for ends as rejected (`accepted-work-expired`) and publishes its completion. A
   prepared Attempt still sends its start; the Site refuses it, which shows that
   nothing started and releases the preparation's credentials. A command saved
   before `acceptedUntil` existed is fenced through Site Coordination before it is
   replaced with a new transport command carrying the original deadline. The Run,
   Attempt and prepared inputs keep their identity. After expiry, an unsent
   preparation ends locally; an already prepared Attempt still waits for the Site
   to confirm its cleanup. Commands already appended retain their exact bytes and
   their outcome is awaited. Runtime checks repeat after provider calls and before
   native creation. Interrupted preparation and failed credential revocation remain
   recoverable; a terminal refusal is returned only after cleanup succeeds.
   Planning, automatic retries and `before_start` also stop once the Project is no
   longer active or its management generation differs from the one the work was
   accepted under (`accepted_work_revoked`). Execution Core reads both from its own
   Project view, and only once Beto's latest authority for the Project is current;
   until then the work waits, so a view that fell behind cannot let it through. An
   owner that cannot answer this check never lets accepted work through. The Site
   cannot tell that a Project closed, so a revoked Attempt sends nothing more: it
   ends as rejected (`accepted-work-revoked`) and publishes its completion, and a
   start already saved ends once the Site proves it was never delivered, otherwise
   its outcome is awaited. Running work is not cancelled here; that is the
   management-authority fence of decision 9. A person decision (launch, manual
   retry, cancellation) still needs that person's current capability.
5. **A Pipeline step is not a new person request.** #799's
   `request_accepted_pipeline` confirms the exact saved step through Pipeline
   Coordination's `AcceptedSteps` query: Project, member, Workload revision,
   environment, stable launch identity, step order and no cancellation. The
   confirmation also carries the Pipeline's saved acceptance, and the step's Launch
   saves it, so the step shares the Pipeline's evidence and deadline. A missing or
   uncorrelated answer stays retryable, and a caller-supplied claim never grants
   continuation. A Pipeline started before the record launches its steps without one.
6. **Requirements are published per accepted work item (agreed on #766).**
   Execution Core keeps one item per accepted launch, from acceptance through its
   Run. Pipeline Coordination keeps one per Site of an accepted Pipeline's steps,
   with a work ID derived from the Pipeline and the Site. An item carries:
   - its work ID and a revision;
   - canonical Party, Project and one Site;
   - the Project's management generation the work was accepted under;
   - the member's `site` binding and its generation;
   - the deadline;
   - the state, `required` or `released`;
   - for a Run, its Run ID.

   Acceptance saves the binding with the evidence, from the account owner planning
   uses (`ProvisionedAccount`). Once that owner is connected, every new launch
   needs the member's exact account at its Site: the Site, the Party's `site`
   binding and its generation. A member without one usable account with all three
   is refused (`account_binding_missing` when the owner names no exact binding),
   and an owner outage leaves the request pending. A Pipeline start saves one at
   every Site its steps use and skips none, and a step's Launch keeps the one at
   its Site. Work saved before the owner was connected carries none and is handled
   separately. The Site owner (#765) keeps an account while any item for that
   Party, Project, Site and binding generation is `required`. A Pipeline stays
   `required` from acceptance until it is final, so a gap between steps holds the
   account. No operation list or cleanup deadline is needed.

   Items publish `AcceptedWorkChanged`
   ([contract](../../../contracts/events/proto/pharos/execution/v1/accepted_work.proto))
   under one event name per aggregate:
   `execution_core.run_launch.accepted_work_changed.v1`,
   `execution_core.run.accepted_work_changed.v1` and
   `pipeline_coordination.pipeline.accepted_work_changed.v1`.
   Each commits with the transition that makes it: a launch's acceptance, its
   planning refusal or a stop before planning, the planned Run at its first version,
   a Pipeline's start, and the coordinator completing it. A refusal or stop changes
   no Launch, so its release names the Launch at its current version. A Pipeline's
   Sites are released one past the version the coordinator completed, after every
   revision the query could have listed, so the Site owner never ignores the release
   as a repeat. The pipeline coordinator relays both owners' events. Metallo grants
   their publishers only because Resource Provisioning subscribes to them (#837).
7. **Release is final, and needs evidence.** A launch's item is `released` when
   planning refused it, as nothing reached the Site. A Run keeps its item after it
   is final, whether it finished or an authorised cancellation ended it, until the
   Site confirms that its stage-out and cleanup went through. A batch Attempt's
   terminal observation carries `cleanupComplete` only once the Kubernetes Runtime
   has revoked its repository credentials or the Slurm Runtime has released
   repository access and cleaned the workspace; a failed cleanup holds the terminal
   observation back and is retried. When every Attempt the Site ran confirmed it,
   the Run's completion publishes the release in the same transaction, one past the
   Run's version, and the query lists it as released from then on. A Site that does
   not confirm the cleanup keeps the account. An Attempt without a submission has
   nothing native: the supervisor ends it only once the Site proved its start was
   never delivered, or after the Site refused that start and cleaned up. So a Run
   that ended before any submission (cancelled early, refused at preparation or
   start, or stopped by a fence) releases on completion. A
   Pipeline's item is released when the coordinator completed it: terminal, with
   any accepted cancellation complete; each step's Run keeps its own item. The
   deadline alone releases nothing. The revision is 1 while a launch waits for
   planning, then one more than its Run's version; for a Pipeline it is the
   Pipeline's version. Once released, an item cannot be required again, and a
   consumer ignores any older revision.
8. **A query serves initial load and repair.** Each context answers a paged query of
   its accepted work, required or released, in ID order
   (`application/accepted_work.py` in both), over RabbitMQ as a versioned Protobuf
   [internal query](request-transport.md#internal-queries): Execution Core's
   `ListAcceptedWork` from the `metallo-requests` process, and Pipeline
   Coordination's `ListAcceptedPipelines` from the `pipeline-coordinator` process.
   A released item stays listed at the revision its release was published at: the
   consumer keeps a hold that is merely absent, so only this explicit evidence
   repairs a release whose event it missed (#766).
   An item has the event's fields; a Pipeline item also names its Pipeline. Only a
   Metallo context enrolled as a caller may ask, and the Site account owner enrols
   with its consumer (#837). Durable event queues must exist before recovery begins.
   Every page carries the SHA-256 hash and item count of the owner's complete
   inventory, read in the same PostgreSQL statement snapshot as that page. The consumer
   requires identical proof across pages, hashes its assembled items, and compares both
   owners once more before completing. An acceptance behind the cursor invalidates the
   pass even while its event waits in the outbox. Missing proof, changed content or an
   incomplete pass keeps recovery pending.

   The fingerprint is SHA-256 over a JSON array of lexically sorted item JSON strings;
   each item uses the event payload's camel-case fields, sorted keys, ASCII escaping and
   compact separators. Optional absent fields are omitted. Each owner keeps its items
   with the stamp of the rows they were built from (`accepted_work_items` and
   `accepted_pipeline_items`, #933). A page rebuilds only the items whose rows moved,
   then reads the page, the count and the fingerprint over the saved items in one
   statement, so it no longer restores the whole history. Pages rebuild in turn under
   an advisory lock, so a page that read older rows cannot save over a newer item.
   Finding the moved rows still joins every launch or Pipeline on each page; it
   restores nothing, but it grows with the history. The stamp starts with the
   items' version, which is raised whenever their fields or encoding change, so every
   item is rebuilt then. The first page after the upgrade builds them all, so nothing
   is backfilled. The saved item is the fingerprint's own encoding, so the hash is the
   one the consumer computes.

   This proves agreement at the comparison, with live events continuing afterwards.
   It does not make separate owner queues globally ordered or authorise native account
   removal. Recovery preserves existing local items absent from a pass; only explicit
   release evidence removes a hold. Missing historical releases therefore retain an
   account conservatively. The Site-facing list of retained accounts is answered only
   while a complete pass stands, and is unavailable before the first pass after a start
   and while a later pass runs again or fails.
9. **Fences.** A Project closure, a change of management generation or an
   explicit binding retirement fences the record: dispatch stops, and durable
   cancellation and cleanup are recorded under management authority. The Beto
   owner commands for these are prerequisites (#755). Ordinary suspension leaves
   the record intact.

## Flow

1. **Launch.** A launch request passes the current capability check (a Beto allow
   with evidence). The Launch is saved with its evidence, deadline and the member's
   account binding at its Site, and publishes `required`.
2. **Planning.** Planning creates the Run with the copied evidence and binding.
   The Run publishes the same item, `required`, at its next revision.
3. **Execution.** Attempts, automatic retries and every native dispatch check the
   deadline and the management generation.
4. **Completion.** A final Run stays `required`. It publishes `released` only
   once Execution Core records that the Site finished its stage-out and cleanup.
5. **Pipelines.** The Pipeline is saved with its evidence and the member's account
   at each Site of its steps on start, and each Site stays `required` until the
   Pipeline is terminal. Each step's Launch is accepted under the Pipeline's
   evidence and the account at its Site.

## Implementation order

1. Carry Beto's evidence through `project_access` and the context permission ports.
2. Save it on the Launch and copy it onto the Run; save it on the Pipeline.
3. Accept a Pipeline step under its Pipeline's evidence.
4. Check the deadline when planning, retrying and dispatching.
5. Add the query of the accepted work, answered over RabbitMQ, and save the
   member's account at acceptance; publish the change events and enrol the query
   caller with their consumer (#785).
6. Add the fences and the management-generation check, once Beto's owner commands
   and the authority stream exist.

## Verification

Done for step 1 and the Launch and Run part of step 2
(`tests/execution_core/test_acceptance_evidence.py`, `tests/test_beto_queries.py`,
`tests/test_party_project_authority.py`). These cover:
- Beto's generation and authority revision being decoded;
- the evidence reaching the Execution Core permission;
- the saved record and its deadline, the request's earlier deadline and the
  deployment setting (`tests/test_request_host_service.py` covers an invalid one
  stopping startup);
- a replay that keeps the original record;
- an allow without evidence, or with expired evidence, deferring and saving nothing;
- planning copying the record onto the Run;
- restoring records saved before this change;
- refusing corrupt or foreign evidence.

Done for step 3 (`tests/test_accepted_steps.py`, `tests/infrastructure/pipeline.py`,
with #799's own step-confirmation cases). These cover:
- a Pipeline start saving its evidence and deadline, and an allow without evidence
  deferring it;
- the owner's confirmation carrying the Pipeline's acceptance;
- a declared step's Launch saving that acceptance while the initiator's own access
  is refused, and a replay keeping it;
- refusal after the Pipeline's deadline, saving nothing;
- a Pipeline started before the record launching its steps without a deadline;
- acceptance for another decision, member or Project being refused, and an owner
  answer carrying it staying retryable;
- the real PostgreSQL lane saving and reloading the Pipeline's evidence.

Done for step 4 (`tests/execution_core/test_accepted_continuation.py`). These cover:
- planning refusing a launch past its deadline;
- an automatic retry stopping there;
- `before_start` refusing a new native start, fetch or source credential after it;
- work saved before the record continuing as before;
- planning, automatic retries and `before_start` stopping once the Project closed or
  changed its management generation, and waiting while the view cannot vouch for it
  (also `tests/test_party_project_authority.py`, and the PostgreSQL view lane with
  real Project and authority events);
- revoked work ending as rejected without sending anything more, whether or not it
  was prepared; a saved start withdrawn, or awaited when the Site may hold it; and a
  saved start waiting while the Project's state cannot be vouched for;
- start and preparation commands carrying `acceptedUntil`, through the codec;
- expired work that nothing reached the Site for ending as rejected with its
  completion published; a prepared Attempt's start being sent and the Site's
  refusal ending it; a command saved without the deadline being withdrawn, or
  awaited when the Site may hold it;
- replacing an unexpired legacy command only after its old transport identity is
  fenced, including recovery when saving the replacement fails;
- retaining prepared work until the Site confirms cleanup after an expired legacy
  start is withdrawn.

`tests/test_runtime_accepted_deadline.py` covers the Runtime: a preparation, Job
or service past the deadline creates nothing, the refused start releases its
workspace, and a Job or service created before the deadline keeps recovering
and running. The cases also cross the deadline during provider calls, recover
interrupted preparation through its separate workspace journal, retry failed
cleanup, preserve uncertain submissions and renew credentials for existing work.

The Slurm regressions (`tests/test_slurm_accepted_deadline.py` and
`tests/test_slurm_chain_deadline.py`) apply the same deadline to both executors,
including a stage-in finishing after expiry, retained compute and stage-out,
interrupted preparation and grant issuance, failed cleanup and uncertain native
submissions. The HTTP test crosses expiry during real JWT/JWKS verification and
checks that no submit request is sent. See [Slurm deadline recovery](slurm-jobs.md#accepted-work-deadline-and-recovery)
for the saved-state and legacy compatibility rules.

Fixtures accept work at fixed past dates, while some lanes plan, dispatch or launch
steps by the real clock. Those lanes use a long test lifetime, and the production
default stays 14 days.

Done for the query of step 5 (`tests/execution_core/test_accepted_work.py`,
`tests/pipeline_coordination/test_accepted_work.py`,
`tests/infrastructure/accepted_work.py`). These cover:
- a launch required at revision 1, then held by its Run at one more than the Run's
  version, and still required with its exact binding after the Run finished or an
  authorised cancellation ended it while the Site's cleanup was delayed;
- a launch planning refused or a stop came before, listed as released;
- a Pipeline required on every Site of its steps until the coordinator completes it,
  then listed as released one past its version;
- a release the consumer missed reaching it from the inventory
  (`tests/infrastructure/accepted_work_recovery.py`);
- paging in ID order, the page bounds and cursor, and inconsistent unfinished work
  refused as corrupt, while damaged finished work is reported and left out, so the
  consumer keeps its hold and one historical row never stops a repair pass.

Done for the query's RabbitMQ contracts (`tests/test_accepted_work_queries.py`,
`tests/test_request_host_service.py`, `tests/test_pipeline_coordinator_service.py`,
`scripts/test_metallo_broker.py`). These cover:
- a complete pass through both owners' handlers and contracts, page by page, giving
  each item as the event carries it;
- an owner refusal, an outage, an uncorrelated reply, a released or malformed item,
  a Pipeline item naming another Pipeline and a page that breaks its cursor each
  failing the pass;
- the owner refusing an invalid cursor or page size;
- the request worker answering an enrolled caller on its own route, and
  dead-lettering an internal query from the API, another user, another family or
  without a broker-validated user;
- the internal queues and owner grants, an enrolled caller's exact permissions, the
  API's role covering no internal query, and invalid enrolments refused;
- both processes registering their handlers and draining the new worker.

Real RabbitMQ qualification of a pass waits for the consumer's enrolment (#837).

Done for reduced limits (#766): planning refuses a queued launch over the limits the
Site enforces now, though it was accepted under larger ones
(`tests/execution_core/test_planning_requirements.py`), and the execution worker's
retry admission (`adaptors/execution_provisioning.py`) refuses an automatic retry the
same way, needs the Run's own allocation ready at its Site, and stops once the
Project closed (`tests/test_execution_provisioning.py`).

Done for step 5's saved accounts and event contract
(`tests/execution_core/test_launch_accounts.py`,
`tests/pipeline_coordination/test_start_accounts.py`, `tests/test_accepted_steps.py`,
`tests/test_pipeline_accounts.py`, `tests/test_accepted_work_events.py`,
`tests/execution_core/test_lifecycle_application.py` and the PostgreSQL lane above).
These cover:
- a launch saving the member's exact account at its Site, and a replay keeping
  the saved account without asking again;
- a missing, ambiguous, invalid or unbound account refusing the launch or
  Pipeline start, an owner answering no account for a Site leaving the start
  pending, and an owner outage leaving either for a retry;
- a Pipeline saving the account at every Site of its steps, a saved Pipeline
  missing one being refused, and a step keeping the one at its Site;
- a Run resolving the account at its own Site when the owner names Sites;
- items carrying the management generation and the binding, one per Pipeline Site;
- each event round-tripping, an item about another aggregate, Site or revision
  being refused, and malformed or unknown fields refused both ways.

Done for step 5's events (`tests/execution_core/test_accepted_work.py`,
`tests/pipeline_coordination/test_accepted_work.py`, `tests/test_accepted_work_events.py`,
`tests/test_pipeline_coordinator_service.py`, and on real PostgreSQL and RabbitMQ
`tests/infrastructure/accepted_work.py`, `pipeline_cancellation.py` and
`accepted_work_delivery.py`). These cover:
- acceptance, a planned Run, a planning refusal and a stop before planning each
  committing the item the query lists next, and a final Run releasing nothing;
- a Pipeline's start committing each Site, and its completed cancellation releasing
  them one past the completed version, with the stopped step's launch released;
- a Pipeline release at the version the query could still list being refused;
- a refused launch reaching Resource Provisioning through Execution Core's relay,
  RabbitMQ and its consumers, held and then released with its account.

The production workers stay paused through #771.