# DD — Resource and Site facts for execution (#763, #764, #765)

Status: proposed for review; #764's allocation core is implemented, #765 is design. Ownership decision: danias.
Admission and Site Runtime: dimitriskatsimpris. HPC and SSH: ikons. Lead: TsigerTheSecond. Part of #750; depends on
#751, #754 and #755. Implementation: [the context README](../../contexts/resource_provisioning/README.md).

## The gap

Execution knows a Run's member and Project by canonical Pharos IDs. To start it, it also needs the Project's
allocation at a Site, the limits applied to it, and the member's native account there. Those facts come from Waldur
and the Site agents under provider IDs, and `adaptors/external_owners.py` refuses `project_resources()`,
`member_accounts()` and `member_bindings()` at start-up.

## Owner

The [context catalogue](../../../../docs/context-catalogue.md) lists **Resource Provisioning (working name)** as the
proposed owner of execution allocations, applied limits, readiness and provisioned native account/access facts, with
its service open and Metallo proposed. This design settles it as the Metallo context `resource_provisioning`, with
its own PostgreSQL database and credentials. The name and service await danias's decision.

| Concern | Owner |
|---|---|
| Orders, prices, approval | Waldur marketplace |
| Party identity and verified external accounts | Beto Party Governance (#754) |
| Canonical Projects, membership, Project ↔ provider Project | Beto Project (#754) |
| **Allocations, lifecycle, configured and applied limits, readiness** | **Resource Provisioning** (#764) |
| **Native account bindings and applied access** | **Resource Provisioning** (#765) |
| Fulfilling resources at a Site (namespace, quota, accounts) | The Site's agent (cloud: #742), reporting what it applied |
| Site-local delivery, journal and enforcement | Site Runtime |

LLM Access is the precedent: it owns the LLM offering's allocation in Metallo, and the catalogue keeps general compute
admission separate. Execution Core must not carry provider mappings, Site Runtime has no platform database, and Beto
owns identity rather than Site resources, so none of them fits.

## Allocations (#764) — implemented

| Fact | Source | Held as |
|---|---|---|
| Allocation: canonical Project, resource, Site | The Site agent's report of a Waldur resource; the Project through Beto's binding | `Allocation`, ID derived from the provider resource |
| Lifecycle | Waldur state and paused flag, plus whether the Site applied limits and the Project resolved | `pending` with reasons, `ready`, `changing`, `suspended`, `retired` |
| Configured limits | Waldur's resource limits, in canonical units | `configured` |
| Applied limits | What the Site enforces (for `k8s`, the namespace quota), read back by the agent | `applied`, absent until confirmed |
| Provider binding | Waldur resource, offering and Project; the namespace | Opaque fields |
| Current usage | The Site; reported to Waldur for accounting | Not held: diagnostics only, never an admission guarantee |

Decisions taken while implementing:

- **The agent reports; Metallo reads no provider.** As the LLM Site Agent does for LLM Access, the Site agent posts
  each resource after its cycle to `/resource-provisioning/<site>/reports`
  ([contract](../../../../contracts/http/resource-provisioning.v1.yaml)). Each Site's prefix has its own key, so the
  Site comes from the credential, not the message. Metallo holds no Waldur or cluster credentials. On the dev box the
  key is generated in `pharos`; no account of the agent's namespace may read it, so an operator copies it once.
- **Two counters.** `revision` changes only with a material fact. A newer unchanged report renews only the time the
  allocation was last seen, beside it, with no command receipt or new version, so freshness never looks like a change
  and the store grows only with real changes. Older reports are `stale_report`.
- **Freshness.** Execution Core's adaptor treats an allocation not observed for fifteen minutes as not ready: stale
  evidence is pending, never a cached success.
- **A limit change in progress stays usable.** `changing` keeps the previous applied limits in force.
- **A local view of Beto's Projects.** As #754 asks, this context keeps its own view of the Project bindings,
  built from Beto's `project.ProjectChanged` events, with a receipt per event and revision ordering. Until a report's
  Project is in the view, its allocation is `pending` with `project_unbound`; the next report resolves it. Nothing
  guesses a Project from a name. The provider instance is configured to match the one Beto's events name. The
  broker subscription is not composed, as Metallo is not enrolled for Beto's events; tests feed real encoded events.
- **Execution Core reads through a service-level owner adaptor** (`adaptors/provisioning_resources.py`), as it reads
  Workloads. Opening this context's database in the execution worker belongs with #766, which connects the other
  owner ports of that worker; until then it stays paused by the permission placeholder.
- **`AllocationChanged` is defined, not yet published.** The broker grants a publisher only for declared consumers and
  an unrouted event would stay in the outbox. Publication starts with the first subscriber, which loads the complete
  paginated snapshot first and then applies newer revisions (#754's handover).
- **Units.** CPU millicores, memory and storage bytes, accelerators `gpu.<class>` in devices. Planning compares whole
  cores, so millicores round down, which is exact for whole-number demands. Persistent volume storage has no planning
  limit yet.

### Capacity rule (agreed with dimitriskatsimpris; `slurm` for ikons to confirm)

- **The Site's quota decides.** There is no platform reservation. On `k8s` the API server checks the namespace quota
  for every Pod atomically, so concurrent launches cannot oversubscribe it.
- **Planning refuses only a Run larger than the whole allocation.**
- **A Run that does not fit now waits.** Its Pod is refused, its Job retries, and Site Runtime shows the Run as queued
  until there is room. It is never denied for lack of room and never partially started.
- **`slurm`.** The scheduler queues jobs over the limit in the same way; ikons to confirm.
- **The Run's time limit is on its Pod.** It used to be on the Job, whose clock starts when the Job is created, so
  waiting used up the Run's time. The Pod's clock starts when the Run does.
- **Concurrent admission test** (the Kubernetes Job dev-box lane). A quota of N cores and two Runs that each fit alone
  but not together, started at once:
  1. exactly one gets a Pod, and the quota is never exceeded;
  2. the other stays queued, not failed;
  3. when the first finishes, the second runs with its full time.

  It passed on the k3s dev box on 28 October 2026
  ([evidence](../epics/instantiate-and-run/site-runtime-context.md#kubernetes-job-lane-on-the-dev-box-28-october-2026)).

## Native accounts (#765) — design

| Binding | Creates | Observes and proves | Retires |
|---|---|---|---|
| Party ↔ Waldur user | Beto Party Governance | Beto | Beto |
| Project ↔ Waldur Project | Beto Project | Beto | Beto |
| (Party, Project, allocation) → native account | The Site agent, from the desired set | Resource Provisioning, from the agent's report | Resource Provisioning; the agent removes the account |
| Runtime member binding | Resource Provisioning (desired) | Site Runtime (applied, in its journal) | Same path |

An identity association never implies a native account, and neither is inferred from an email or a matching
username. A member without a verified Party binding gets no account and no Runtime binding. Each binding has a
generation: removing a member retires it, adding the member again creates a new one, and Runtime refuses a retired
generation even if the account name is reused. Bindings reach Runtime through the authenticated Site protocol into its
private SQLite journal, which its `MemberBindings` port reads; Runtime reports applied revisions back.

The agent reports what it found, and Resource Provisioning keeps it apart from Beto's association. After each cycle
the agent posts the complete set of member accounts in a resource's native scope to
`POST /resource-provisioning/<site>/accounts`, each named by what it wrote on the account: the Party, the account's
name `member-<Party>` (the account Site Runtime acts as; a report that names it otherwise is refused), the Site's
immutable UID of that account (so an account made again under the same name is
another one), the Party's `site` binding and the member generation, as `ready` (the account and all its Roles and bindings exist) or
`incomplete` (an interrupted provisioning). The set replaces the allocation's last one in `native_accounts`: an account
no longer listed becomes `absent`, an account of an older generation stays as evidence, and a report older than the
saved one is refused as stale, so a recreated account cannot revive an old grant. The UID is part of an account's key:
one made again under the same name, binding and generation is a new row and the old row stays `absent`, and the read of
the ready accounts returns the UID, so a reader can compare it with the account Runtime is about to act as. A confirmation only informs. It
grants nothing, removes nothing, and what an unconfirmed account means for a launch is Execution Core's decision (#766).

A binding says an account exists with the template's permissions; permission to act is Beto's access evidence
(#755). Site Runtime checks the binding, then current permission, before each native effect
(`site_runtime/application/access.py`). A removed member's access ends within #755's bound; the agent's later removal
of the account is clean-up, not the security boundary.

The cloud agent (#742) moves from its stand-in to canonical IDs: accounts `member-<PartyId>` and the namespace's
`project-id` label the canonical Project. Existing namespaces need their accounts replaced and their label changed;
the agent's admission guard forbids changing `project-id`, so the migration needs a one-time allowance from a Waldur
UUID to the canonical value, or recreation.

### How an account is found

The agent sees a Waldur username and a Waldur resource. Execution sees a Party and a Project. Four facts join them,
each with its own owner:

| Step | Fact | Source | Held in |
|---|---|---|---|
| 1 | The offering username of a resource user is the Party's `site_username` | Waldur, through the agent | Not held |
| 2 | That username belongs to a Party: the Party's `site` provider binding (provider `site`, the Site's environment as instance, the username as external ID) | Beto Party Governance, `PartyProviderBinding*` | A Party binding view |
| 3 | The Party is a verified member of the Project | Beto Project, `ProjectChanged` memberships | The Project view (implemented) |
| 4 | The Project has an allocation at the Site | The agent's report | Allocations (implemented) |
| Result | Account `member-<PartyId>` and a binding with its generation | This context | Derived, never stored separately |

The membership in `ProjectChanged` names the Party's binding with the Project's provider (Waldur), which is not the
`site` binding of step 2, so the Project view alone cannot give the agent a Party for a username.

Decisions:

- **The agent asks this context.** It does not read Beto. It resolves a username and its provider Project through one
  route of the Site contract, which returns the canonical Project, the Party, the binding's ID and revision and its
  generation, or that the binding is retired. This answers question 4 with desired facts delivered from here.
- **Views come from #757's mechanism.** Party and Project facts are kept as each execution context keeps them, in this
  context's own database, loaded from Beto's checkpoints and kept current by its events. This context's copy and its
  Project lookup are built; the Project view built here stays as the interim until Beto enrols this context's views
  caller, which answers question 2.
- **A person's account serves a person's action.** Work that outlives its initiator uses a Project- or Run-scoped Site
  identity, not `member-<PartyId>` (#799). That identity is open with the Runtime owner; this context's member
  accounts do not change for it.

## States

| Case | Outcome |
|---|---|
| Ordered, native resources not ready | `pending` (`provisioning`, `limits_not_applied`) |
| Project not yet bound in Beto | `pending` (`project_unbound`) until a report resolves it |
| Limits changed, not yet applied | `changing`; planning uses the previous applied limits |
| Provider paused | `suspended`; planning refuses Runs with `resource_suspended` while the pause is current |
| Terminated | `retired`, kept as a tombstone; later non-retiring reports are refused |
| No report for fifteen minutes, or the store unreachable | Not ready, or `ResourceLookupUnavailable`; never cached success |
| Duplicate or older report | Unchanged, or `stale_report` |
| Member added before Party verification (#765) | No account, Run deferred, until the binding arrives |
| Member removed (#765) | Access withdrawn by #755's evidence; binding retired with its generation |

## Current Waldur writers

To confirm before replacing any operation; each path keeps working until its replacement passes real-service checks.

| Writer | Writes in Waldur |
|---|---|
| Retained Themelio account writer | Users created ahead of sign-in |
| Waldur sign-in (`user_claim: sub`) | Users at first sign-in |
| Beto Project source | Nothing; reads Projects and memberships |
| Cloud Site Agent (#742) | Resource state, backend IDs, limits, usage, offering users |
| LLM Site Agent (#749) | The same for the LLM offering |
| Administrators and the marketplace | Organisations, offerings, orders, approvals |

## Open questions

1. The service assignment and final name (danias).
2. Whether this context keeps its own Party and Project views or reuses #757's (dimitriskatsimpris). Proposed:
   reuse its mechanism, in this context's own database (see "How an account is found").
3. The capacity rule for `slurm` (ikons); `k8s` is agreed (see "Capacity rule").
4. How the Site agent learns canonical IDs: a scoped Beto query, or desired bindings delivered from here. Proposed:
   delivered from here, by one route of the Site contract.
8. The Site identity for accepted work that outlives its initiator (#799, #765), with the Runtime owner.
5. The `project-id` migration under the agent's admission guard.
6. Several ready resources in one environment: forbid, or a selection rule (`resource_ambiguous` today).
7. The agent's polling bound for account removal against #755's revocation bound.

## Verification

| Evidence | Result |
|---|---|
| Domain, handler, event, route and Execution Core adaptor tests | Passing (`services/metallo/tests/resource_provisioning`) |
| Real PostgreSQL: receipts, versions, per-Project and paginated reads, concurrent reports, corruption | Passing on a dev box (`tests/infrastructure/resource_provisioning.py`, `pnpm test:integration`) |
| Project view from encoded Beto events: receipts, ordering, rebinding, conflicts | Passing, including real PostgreSQL |
| Cloud Site agent reports: configured and quota-enforced limits, termination, failures | Passing on the agent's fake cluster; separate change on #742's plugin |
| The agent's reporting code on a dev box's live Cloud resources, read-only, into a local intake on throwaway PostgreSQL | Passing: real Waldur state and quotas mapped and accepted; `project_unbound`, then `ready` after a Beto-shaped `ProjectChanged`; renewal keeps the revision |
| The deployed agent reporting to the deployed Metallo (network policies, mounted key) | Not run; due with the next dev-box deployment of the integrating branch |
| Broker subscription to Beto, publication, #765 | Not implemented |