# Design note — Subtask 3B: order fulfilment

Status: proposal for review. It revises the [DD](dd.md) where it says so and leaves the rest as written.

## Scope of this change

This note, and the core it describes in `contexts/llm_access`: the domain, the application, PostgreSQL persistence
through the generic aggregate store, and delivery to the LiteLLM gateway. The context is registered and activated.
Each change records its state, version, command receipt and one private reconcile intent
(`llm_access.allocation.reconcile.v1`) in one transaction; the aggregate has no browser projection yet. A scoped
broker identity and the declared queue carry the intent to a consumer that calls the gateway, using only the team
routes, and then records the result on the allocation. The consumer is optional: it exists only when the gateway URL
and key are both configured. There is no route and no Open WebUI adaptor yet.

## Flow

The project provisioning slides of the Pharos LLM talk (9 October 2026) show the validated flow:

```text
Project team -- order (models, virtual budget, limits) --> Waldur
Pharos admin -- approves (never automatic for this offering) --> Waldur
Site Agent   -- pulls the approved order --------------------> Waldur
Site Agent   -- one idempotent call --> Reconciler (the only writer)
Reconciler   -- team, budget, models, limits, project key --> LiteLLM
Reconciler   -- group and model access ---------------------> Open WebUI
Site Agent   -- order done --> Waldur
```

The Reconciler is this DD's "scoped management adaptor". The root actor of the earlier design is not carried over:
this context holds the decision, and the adaptors hold narrow credentials for one provider each.

## Bounded context and aggregate

`llm_access`, owned by Metallo. One aggregate: `ProjectAllocation`, one per provider resource bound to a Project.

- Canonical ID: independent of Waldur. The Waldur resource and order stay opaque bindings (`resourceRef`, `appliedOrder`).
- State: `active` or `blocked`. Termination blocks and never deletes, so spend history survives.
- Entitlement: sorted served model names, `maxBudget` (virtual units, exact decimal text), `budgetDuration`
  (always `1mo`, one calendar month in UTC), optional tokens and requests per minute.
- `version`, the applied order and its time, for idempotency and ordering.

## Rules

| Order | Outcome |
| --- | --- |
| State other than `executing` (pending, rejected, cancelled, done, erred) | Rejected `order_not_approved`: no access |
| Model not in the catalogue, no model, a missing or malformed budget, duration or limit | Rejected, nothing stored |
| `create`, no allocation | Active allocation, version 1 |
| `update` | The whole entitlement is replaced, version + 1 |
| `terminate` | Blocked, version + 1; a second terminate changes nothing. Reconciling it blocks every key of the allocation that has a gateway token and then the team, and the team is blocked before any change to its limits |
| The order already applied (redelivery) | Unchanged, no new intent |
| Older than the applied order, or a different order at the same moment | Rejected `stale_order` |
| Other resource, Project or Project alias than the allocation | Rejected `allocation_mismatch` |
| `create` twice, `update` or `terminate` without allocation, change after terminate | Rejected |

A budget is an exact plain amount: digits without a leading zero, at most six decimals, below a billion units, kept as the
text it arrived as. It has no exponent, sign, space or separator, so two requests with the same meaning cannot differ in
their material input. The budget and its duration have no default: an order that omits one is rejected, never read as a
zero budget.

### Waldur's order types

Waldur's order types are `create`, `update-limits`, `switch-plan` and `terminate`, and an order other than `create`
does not carry the offering's `models` attribute. The core takes a different shape on purpose, so the Site Agent has a
single mapping to make:

| Waldur order | Core order | Fields come from |
| --- | --- | --- |
| `create` | `create` | the order's attributes (`models`) and limits |
| `update-limits`, `switch-plan` | `update` | **the resource's full current entitlement**, not the order's delta: the models the resource already holds, with the limits and plan the order sets |
| `terminate` | `terminate` | the resource and Project only; no budget is needed |

An `update` replaces the entitlement as a whole, so the Site Agent must send the resource's complete entitlement on every
`update`. A missing model list, budget or duration is a rejection, which is why the order has no defaults for them. The
order's time is Waldur's, and an order at or before the applied order's time is stale, so the Site Agent must give each
order of a resource a strictly later time. The Project alias is part of the allocation: an order that names another alias
is rejected, because the alias names the gateway team and the chat group.

The Site Agent forwards only approved orders, and the core refuses everything else too.
`executing` is the Waldur order state the Site Agent works on; to be confirmed against the Waldur version in use.

## Messages with the Site Agent (subtask 1A)

The contract is `contracts/http/llm-access.v1.yaml`, plain HTTP and JSON like the other Metallo contracts.

- **Order** (`POST /llm-access/orders`): a request ID and the evidence of one order. The command ID is derived from the
  request ID and the order's material input, so a retry of the same content returns the saved result, while a request
  whose evidence changed (an order that has moved from pending to executing) is a new command and not a conflict with
  an earlier refusal. An order that is not being fulfilled is refused before any command and leaves nothing behind. Any
  other refusal is saved with its request, so the agent chooses a new request ID after one. The message never carries a
  canonical identity.
- **Allocation:** derived from the Waldur resource (`derived_id("llm_access.allocation", resourceRef)`), so every order
  for a resource, retried or not, reaches the same allocation. The agent records the returned `allocationId` as the
  resource's backend ID.
- **Result:** `accepted` or `rejected`, with the rejection code and message. `accepted` means saved, not effective, and
  carries the `entitlementVersion` the order resulted in (or that already held it, for a replay or a second termination
  of a blocked allocation).
- **Status** (`GET /llm-access/allocations/{allocationId}`): the state, the applied order, the entitlement and
  reconciled versions, and `ready`. The agent marks an order done once `reconciledVersion` reaches the
  `entitlementVersion` of its result, so a Waldur order never completes before the gateway has caught up and the
  order IDs need no ordering.
- **Strictness:** unknown fields, nulls, wrong types, non-canonical IDs, amounts with an exponent or more than six
  decimals, and out-of-range limits are refused before anything is read. A refusal never echoes the message.

The routes are mounted only when `LLM_ACCESS_INTAKE_KEY` is set, which the devbox overlay does together with the model
list. That key guards `/llm-access` alone: the service key does not open it and the intake key opens no other route, so
the agent cannot read anything outside its own messages. Responses: 200 accepted, 422 refused with the reason, 400 for a
malformed message (never echoed, duplicate JSON keys and deep nesting refused, the body read only up to 64 KiB), 401
without the key, 404 for an unknown allocation, and 503 without detail when the store fails. The served model names
come from `LLM_ACCESS_MODELS`; a platform check keeps the devbox list equal to the models the gateway serves and the
Waldur offering offers.

## Key records (subtask 1B)

A key record (`AccessKey`, its own aggregate) says which member a personal key belongs to (a Project key, the one
key a Project's chat connection uses, has no member), which allocation (so
which Project) it spends, and which models it may use. It holds no secret: the gateway key is identified by the
alias `pharos-key-v2-<key id>`, and the record's `gateway_key()` is the desired gateway key for the issue step (4B).

- The Project is the allocation's and the caller cannot choose it. The models are a non-empty subset of the
  allocation's. A terminated allocation gets no new keys. Issuing again with the same details returns the saved
  record, and the same key ID with other details is refused.
- What a key may use is its models that the allocation still includes, and nothing once the key is revoked or the
  allocation terminated. `check` answers a request with a reason: `key_revoked`, `allocation_mismatch`,
  `allocation_blocked`, `project_mismatch` or `model_not_permitted`.
- A key with nothing left to use is described as **blocked**, never with an empty model list: the gateway reads an
  empty list as every model of the team (measured).
- The key's projection names the key, the allocation, the state and the models, and leaves out the member.
- `keys_of_allocation` lists every key of an allocation, revoked ones included, for the termination step above.

Measured on the clean test VM (LiteLLM v1.96.0), with a key bound to a team and one model:

| Attempt | Result |
| --- | --- |
| A model the key was not given, though the team has it | 403, "key not allowed to access model" |
| `team_id`, `metadata.team_id`, `user` or tags in the request body | Accepted and ignored; all spend stayed on the key's team |
| `/key/generate` for another team, `/team/update`, `/key/update` (to widen its models), `/spend/logs` | 401 |
| `/team/info` of another team, `/key/list` | 403 |
| A model removed from the team afterwards | The key loses it (403, "team not allowed to access model") |
| A blocked key | 401 at once; the team's other keys keep working |

So the gateway holds the Project and the model list for a key once it is created with its team and a non-empty
model list. This subtask records them and decides what to ask for; issuing the gateway key is 4B.

## Key issuing credential (measured, clean test VM, LiteLLM v1.96.0)

Both the Project key that chat uses and personal keys need `/key/generate`, which cannot go to the Reconciler: a key
that may generate can mint unrestricted keys. Team admins are an enterprise feature, so the issuer is an ordinary
internal user, restricted three ways.

| Attempt | Result |
| --- | --- |
| A plain member generating for a team that grants members nothing | 401, "does not have permissions" |
| The same member where the team grants `/key/generate` (set in the Reconciler's own `/team/new` call) | A key for that team is made |
| A key for another team, or for another user | 400, "not assigned to team" |
| `allowed_routes` on a key | 403, only administrators |
| A personal key with no team | Made, but it takes the user's model list, which names no real model, so it can reach nothing |
| A key naming a model outside the team, or a budget above it | Made, but the team's own caps apply at request time (403 for the model, 429 once the team's budget is gone) |
| A key made without `key_type` | It can itself generate keys, so a leaked Project key could mint more |
| A key made with `key_type: llm_api` | It reaches its model and cannot generate, read or list keys or read its team |
| The issuer listing keys | Only its own, not another team's |
| The issuer updating a team, creating users or registering models | Refused |

So the adaptor always sends `llm_api`, the issuer has no user-level budget or rate limit, and its key is limited to
`/key/generate`. The issuer user is a member of every team, added by the Reconciler when it creates the team.

## Chat access in Open WebUI (measured, clean test VM, Open WebUI 0.11.0)

Connection-level access grants in 0.11.0 gate tool servers and terminals, not model connections, so the grant is per
model. What was measured with an admin API key limited to `/openai/config`, `/api/v1/groups` and
`/api/v1/models/model/access`:

| Attempt | Result |
| --- | --- |
| Read the connection config, list groups | Allowed |
| Read users, the admin config, add a user, delete a model entry, run a chat, export config, read its own key | 403 |
| A connection with the Project's key and prefix, before any model grant | Neither a member nor an outsider sees its model |
| A group grant on `alias.model`, made with the scoped key | The member sees it and can chat on it; the outsider does not see it and gets "Model not found" on both chat paths |
| The member's chat | Spent the Project's gateway allowance |
| The scoped key's config read | Returns every connection's key, the Project key included |
| Removing the connection through the whole-list update | The model leaves the member's list |

So the Open WebUI credential is a different, larger-exposure credential from the gateway's, and a model with no grant is
admin-only, which makes the order of the steps the safety. Clearing a grant with an empty list hides the model from the member
(measured on the live run of the adaptor).

**Project key.** The connection uses the Project's key, which is a key record of kind `project` with no member. The gateway
shows a key's secret once, so a connection that is new or has been lost always needs a fresh key. The chat step therefore
asks for a key only when it is about to create a connection, retires every earlier Project key (revokes the record and
blocks the token) and issues the next one, with the key ID derived from the allocation and a generation number. The key
asks the gateway for no model list, which a key reads as the team's models, so a changed model list needs no key change.
The Reconciler runs chat after the gateway in every reconcile, and a failure in either raises, so nothing is recorded as
reconciled and the delivery is retried. A conflict (a group of that name that belongs to something else, a prefix used for
another gateway) cannot be fixed by retrying, so it is retried until it dead-letters and shows in the diagnostics.

## Personal keys for members (subtask 4B)

A key is made when a member joins an allocation (the decision text: "all project members get access automatically"). The
Site Agent tells Metallo with `POST /llm-access/allocations/{id}/members` (a request ID, the member's Waldur reference and
their email). The member's ID is derived from the reference, and the key's ID from the allocation, the member and a
generation, so the call is idempotent and a member who is removed and added again gets the next generation.

**The secret has to be kept until it is mailed.** The gateway shows it once, and the ADR says to retry a failed delivery
without issuing a duplicate key, which needs the secret again. Two options were weighed (October 2026):

| Option | Outcome |
| --- | --- |
| Seal it until delivered (chosen) | The secret, the token and the address are encrypted with AES-256-GCM, bound to the delivery, into a pending delivery record that is never projected or published, and erased when the key is sent or the delivery is given up. A retry sends the same key. It needs the `cryptography` dependency and a sealing key delivered like any other credential. |
| Block and reissue on failure | No stored secret, but every mail failure creates and blocks a gateway key and a member can receive two emails of which only the second works |

The order of the steps is the safety. The record is made, then a random caller-supplied key is sealed in a private provision
and its hashed token is bound to the key record **before** LiteLLM is called. A provider timeout or process interruption
therefore replays the exact key; an alias response is accepted only for such a saved replay. After the secret is sealed into
the delivery, the provision is settled and its sealed value erased. Failure to save the delivery leaves the provision for
retry. A first-call alias collision is refused, and the versioned `pharos-key-v2-` aliases cannot collide with an uncertain
key from the earlier server-generated flow. Two identical member requests are serialised as an additional local guard.

A personal key names its models, the allocation's at the time it was made. A model added to the allocation later does not
reach existing keys until they are rotated (subtask 4D). The email address lives only inside the sealed value and goes with it.

## Mailing keys (subtask 4C)

Metallo does not run a mail server and needs none of its own: it sends through an SMTP account on a relay the platform may use,
with a dedicated account whose password is a vault-managed secret (the design document asks for exactly that). Which relay, port
and sender address to use is a deployment decision for the team. In tests a small fake SMTP receiver stands in, and for a demo
a local mail catcher such as Mailpit shows the email arriving.

- **What is sent.** One email per key: the secret, the Project, the service address a client should use and how to set it up,
  and a plain warning to keep it private and to ask the Project's administrator to revoke it if it is exposed. No links other
  than the service address, and no tracking.
- **Retry without a second key.** A failed send is counted and keeps the sealed secret; the next attempt sends the same key. The
  delay grows (1, 3, 7, 15, 30 and 60 minutes after the key was made, then one more hour for each further failure) and is computed
  from when the delivery began and how many attempts failed, so a restart never loses it.
- **Time limit.** After 72 hours (configurable up to 30 days) a delivery that has not gone out is given up: the key is stopped at
  the gateway through the Reconciler's block route, the record is revoked and the sealed value erased. A key that its owner never
  received would be a credential nobody could have, so it must not stay alive. The Site Agent's next membership pass then makes a
  fresh one.
- **At least once.** If the mail went out but recording that failed, the same key is mailed again. The email is identical, so the
  member holds one key, never two. Exactly-once would need the mail server's help and is not attempted.
- **Nothing sensitive leaks.** TLS is required with the certificate checked (STARTTLS, or TLS from the start), and the login is sent
  only after TLS. Unencrypted SMTP is refused unless the server is on a loopback address. smtplib's own error text can carry the
  recipient, so a failure is reported by its class alone without the original traceback, and the diagnostics record only that class
  and the key's ID.

## Revoking a member's keys (subtask 4D, the keys part)

`POST /llm-access/allocations/{allocationId}/members/revoke` takes a request ID and the member's Waldur reference, and is what the
Site Agent calls when Waldur's membership sync reports that a member left a Project. It exists with personal keys, behind the same
intake key.

- **What it does.** For every personal key the member holds on that allocation: the key is stopped at the gateway through the
  Reconciler's `/key/block` (immediate, unlike the team's block), the record is revoked, and a delivery still pending is ended as
  abandoned, which erases the sealed secret and the address, so a key the member no longer may have is never mailed. A pending
  provision has its token recorded before the provider call and is therefore stoppable too.
- **Order.** The gateway is stopped first. If the process fails after the record is revoked, nothing would look for the live key
  again; failing after the block leaves an active record that the repeated request finishes.
- **Safe to repeat.** A key already revoked is only finished off, a member who holds no key is accepted with no key IDs, and the
  answer lists the keys this request stopped. The route shares the lock of the one that adds members, so a member added and removed at
  once never ends with a live key. A member who is added again later gets the next generation of key; the old one stays stopped.
- **What it does not touch.** The Project's other members, the Project key, other allocations and the member's keys in them. A
  member's access to one Project is independent of their access to another.
- **Chat access.** It follows the Project's Open WebUI group, which LLM Access writes (subtask 4A), so revoking a member takes them
  out of the group as well. Open WebUI reads group membership on each request, so the removal ends access in a session that is
  already open. When Beto decides membership (`LLM_ACCESS_CHAT_MEMBERSHIP=beto`), a member Beto removes has their keys stopped
  through the same revocation before the group is changed, so the two ends cannot disagree.
- **What it does not do yet.** A bounded expiry rule for sessions is not built, and neither is proof against a real Waldur Project
  of removal ending chat and keys together (#769).

### Rotating a member's key (subtask 4D, rotation)

**Decision (October 2026): two steps. The next key is issued and mailed, and the key it replaces is stopped only after that mail
has gone out**, so a member is never left without a working key and a failed or lost mail never costs them the one they have.

- **Trigger.** `POST /llm-access/allocations/{allocationId}/members/rotate`, with the same request and answer as adding a member,
  behind the same intake key. It is called on purpose (an administrator or the Site Agent); nothing rotates on its own yet, so a
  change of the allocation's models does not replace keys.
- **Step one, the route.** The member's next generation of key is created at the gateway and its secret and address are sealed in a
  pending delivery, exactly as for a new member. The current key is untouched. A member with no live key is refused (422).
  Asking again while the next key is still on its way returns that key and makes no third one.
- **Step two, the mailer.** There is no separate link between the two keys: when a personal key's mail has gone out, the member's
  other live personal keys on that allocation are the ones it replaces, and they are stopped at the gateway and then revoked, before
  the delivery is settled as sent. If stopping or settling fails, the next pass mails the same key again and tries again, so the old
  key is never left live once the new one was sent. A key sent for the first time has nothing to replace.
- **If the next key is never mailed.** After the delivery limit it is abandoned like any key, and the current key stays live.
- **Not built.** Rotation on a model change, and a Waldur action that calls the route (Waldur has no member-key event); the plugin
  does not call it.

## The allowance period (subtask 5B): the gateway's own calendar-month reset

**Decision (October 2026): the allowance period is a platform policy of one calendar month in UTC, and the gateway's own
`1mo` reset restores it.** An order carries `budgetDuration` `1mo` and nothing else; a number of days is refused (`30d`,
`1d`, `2mo`, an empty value), because the period is not something an order chooses. The Reconciler creates and compares
each team with `budget_duration` `1mo`. Measured on the clean test VM (LiteLLM v1.96.0), with teams created at 01:13 UTC
on 11 October:

| Duration | Reset time the gateway assigns |
| --- | --- |
| `1mo` | 00:00:00 UTC on the first of next month (2026-11-01) |
| `30d` | The same instant, 2026-11-01: not a rolling window on this version, but not something to rely on across versions |
| `1d`, `1h`, `2m` | The next UTC midnight, the next hour, the next even minute |
| none | No reset |

- Updating a team from `30d` to `1mo` changes nothing (both resolve to the same instant), and changing only the budget keeps
  the reset time.
- **The reset is not instant.** A team on a `2m` window with a reset time of 01:14:00 was still refused at 01:19:58 and was
  served again, with its spend back at zero and its next reset time moved on, at 01:20:14: 374 seconds after the reset time.
  The gateway applies resets on its own schedule, which looks like roughly every ten minutes. **A Project that has used its
  allowance stays blocked for up to about ten minutes into the new month**, and usage in that gap is added to the old month's
  spend and then wiped by the reset, so enforcement gives a Project a little more than a month's worth. This is the gateway's
  behaviour and is accepted for now.
- **Reporting is not affected.** Usage reporting (5C) reads the immutable per-day figures, so a request after midnight is
  counted in the new month whatever the reset job has done.
- The first month is a partial month: a team created mid-month resets at the next boundary and starts with a full allowance.
- Not measured: more than one gateway replica. A reset moves the reset time on, so a second reset in the same window should
  not happen, but that has not been shown. Subtask 5B stays open for the gap above, not for the boundary, which is exact.

## Allowance enforcement (subtask 5A): not strict for now

**Decision (October 2026): keep the gateway's own, non-strict enforcement and document it.** A Project's allowance is a
soft cap. What the gateway does, measured on the clean test VM (LiteLLM v1.96.0):

| Situation | Measured |
| --- | --- |
| Requests one after another against a team budget of 0.0004 | Five were accepted before the first refusal and the team ended at 0.00046: the check runs before a request, with the cost known only afterwards, so the request that crosses the cap is still served |
| A budget smaller than one request (0.00002) | The first request is accepted, the next three are refused with HTTP 429 |
| Ten simultaneous requests against the same budget | Four accepted, six refused, 0.000368 spent in that run, so no overshoot there; one run is not proof |
| A refused or failed request | No spend, and it counts as failed in the usage figures |
| A stream the client cancelled | Billed for the tokens produced, not the tokens requested |
| `/team/info` spend | Lags the gateway's own count by the spend-write interval, so it must not be trusted just after a request |

**What this means for the allowance.** A Project can spend past its allowance by the requests that were already admitted
when the spend crossed the cap: at most one request's cost for sequential use, more for a burst, bounded by the team's
`tpm_limit` and `rpm_limit` over the spend-write interval. The size of that bound has not been measured and is not a
guarantee. What stops spending is unchanged: once the persisted spend is over the cap every further request is refused
(429) until the allowance resets, and the overshoot is visible in the usage figures (5C), so it is reported to Waldur,
not hidden. Nothing here lets a Project reach a model it was not given or spend on another Project's allowance.

**What a strict guarantee would take, not done.** The design document proposes admission with cost reservation and
settlement: reserve the worst-case cost of a request when it is admitted (input size times the input price plus the
largest answer times the output price), release the reservation and record the real cost when it ends, and expire a
reservation whose request was cancelled or lost. That is custom code inside the gateway, a new deployment artifact, a
shared store if the gateway has more than one replica, and tests for streaming and cancellation. Until it is built the ADR's
"prevent spending beyond the approved allowance" is **not met strictly**, and subtask 5A stays open. The cheaper levers
that exist today are the team's `tpm_limit` and `rpm_limit` (already carried by the order), which shrink the overshoot but
do not remove it.

## Usage reporting (subtask 5C, measured on the clean test VM, LiteLLM v1.96.0)

The Site Agent's report mode asks Metallo for an allocation's cumulative usage for a calendar month and posts it to Waldur,
so Metallo holds no Waldur credential (the earlier stack worked the same way through its root actor). What the gateway
offered a reader, after generating real traffic (three served requests, one refused model, one cancelled stream, and a
tiny-budget team that was refused after its first request):

| Route | Result |
| --- | --- |
| `/spend/logs?team_id=` | **Did not filter**: it returned all 269 entries of every team (118 failures, 151 successes). Its entries have fields for messages, responses and the request, which were empty only because this gateway does not store prompts. Rejected. |
| `/team/daily/activity?team_ids=` | Exactly the team's figures per UTC day: spend 0.000171, 121 input and 25 output tokens, 146 in total, 5 requests of which 4 served and 1 failed. The spend equals the team's own spend. No prompt content. |
| A cancelled stream | Counts as a served request, for the tokens it produced |
| A refused request | Counts as failed, with no spend |
| A key limited to `/team/daily/activity` | Reads it and was refused every other route; it can read any team's figures |

So the feed is the daily activity route, and a calendar month is an exact sum of whole UTC days. The report is cumulative
for the month, so duplicate delivery, interruption and restart cannot change a figure, and a request that finishes after
midnight belongs to the month it finishes in. The gateway's own budget is not used for reporting, because it resets on its own
schedule. The figures arrive a little after the requests, so the month just ended should be reported once more after it
closes.

## Membership input

This section records the shared membership semantics for the personal keys and for the chat access that follows them
(subtasks 4A and 4D, tracked in GitLab #162), and what changes when canonical membership comes from the Beto Project
service under #164 and not from the Waldur membership sync.

**Today.** The Waldur Site Agent's membership sync is the only trigger. A member added to a Project's resource reaches
`POST .../members` and a member removed reaches `POST .../members/revoke`, each with a request ID derived from the
allocation and the member, so a repeat is safe. The member is identified by the Waldur reference, an opaque binding from
which Metallo derives the member's ID (`member_id_of`). Two properties matter to a canonical feed:

- A removal stops **every** personal key the member holds on that allocation at the moment it is processed. Nothing in
  the request says which membership it ends. The order of changes is therefore Waldur's: the sync applies them one after
  another, and a removal that were processed after a later re-add would revoke the key issued for the later membership.
- The `generation` in a key's ID counts the keys the member has held on the allocation (a rotation adds one too). It
  makes key IDs stable and distinct. It is not a membership revision and carries no order of memberships.

**What stays.** Key blocking (the key first, the team after), the abandonment of a pending delivery with its sealed secret,
and the retry and idempotence rules do not depend on where a membership change comes from, and a canonical consumer
should call the same use cases.

**What a canonical consumer must add**, and its tests with it, when it is connected:

1. *Identity mapping.* Canonical Project and Party IDs resolve to the allocation (derived from the Waldur resource of the
   Project) and to the Waldur member reference that the keys are bound to, through the verified Party and provider
   bindings, never through a name or an address.
2. *Membership revisions.* Every add and removal carries the revision of the membership it belongs to. A key records the
   revision it was issued under, a removal revokes only keys issued under that revision or an older one, and an add for a
   revision older than the member's latest removal is ignored. A delayed or repeated event then cannot revoke a later
   membership's key or give a removed member a new one. Until this exists, only an in-order feed such as the sync is safe.
3. *Initial state and missed events.* On connection, and after any gap, the consumer reads the Project's current members
   and compares them with the live keys: a key whose member is gone is revoked, and a current member without a key gets
   one. This is the same read, compare, change shape as the gateway reconcile and the recovery sweep.
4. *Freshness.* A key is issued only while the consumer's view of the membership is recent enough, and a stale view stops
   issuing but never stops revoking. The bound is to be agreed in #162, with the same rule for chat access.

Delayed, out-of-order and duplicated canonical events need their own tests; the existing rejoining, retry and idempotence
tests of the members and revocation routes are the base to extend. Nothing here is implemented yet.

## Membership-aware sign-in (subtask 4A)

**Decision (October 2026, following ADR-0020): Waldur is authoritative for membership and Metallo writes it directly to
the allocation's Open WebUI group.** The Site Agent remains the only membership feed. The sign-in provider is
authentication only: in production it is the GRNET realm, which Pharos cannot write to, so no group is kept there.

When the Site Agent adds a member, Metallo records a `chat_member` (allocation, member, address, whether the person
belongs, whether Open WebUI agrees) and, once the person is known to Open WebUI, adds them to the group. When it revokes
the member, the keys are stopped first and then the person is removed from the group. A person Open WebUI does not
know yet (no first sign-in, no group yet) stays pending and the recovery sweep retries every interval; a person who
leaves before ever signing in is simply done. A failure raises, so the request is retried too. The key is recorded
first, and a group with the allocation's name that does not carry its description is a conflict, never changed.

**Beto decides (#769, `LLM_ACCESS_CHAT_MEMBERSHIP=beto`).** Beto owns Project membership and access policy, so the group
follows Beto's membership instead of the Site Agent's member routes. LLM Access keeps its own copy of Beto's Project and
Party facts and takes Beto's rule from it. The Site Agent names each person with their sign-in subject, address and Waldur
user UUID, which pairs them with Beto's membership: Beto records a Project's members by that UUID, and the allocation is
the Project whose Waldur ID it carries. The sweep makes the group follow what the rule admits, so a removal Beto has
applied takes the person out at the next look and a replayed add cannot put them back. The default stays `site-agent`
until Beto's Waldur acquisition is active in an environment.

The person is bound by the sign-in subject, not by a name: the Site Agent's reference is the account's subject at the
platform AAI, and Open WebUI records it for the account that signed in. Metallo adds the account whose recorded subject
and address are both the member's, so a lookalike with the same address or name is never added. The record keeps the
add requests already taken into account; replaying one after the member left, or an older add still in flight, changes
nothing, and only a new request puts the member back. A revision from the Site Agent would also order requests that
carry different IDs; that is part of the move to published access facts (#769, #755).

Open WebUI is an OIDC client of the platform AAI (`pharos-open-webui`), with group management and creation off, so a
sign-in neither copies a claim onto groups nor removes anyone from one. Membership is read per request, so a removal
ends access in a session that is already open; the bounded expiry rule for sessions, the canonical Beto identity and
the applied-state tracking of grant and removal across components are #769. The chat key gains `/api/v1/users`
to find a person by address and read the subject Open WebUI recorded for the account (the search route does not return
it, `GET /api/v1/users/{id}` does). Open WebUI's allowlist is by path and not by method, so this also reaches the other
user routes of an admin key; the route list is a platform check for that reason.
The browser callback is `https://open-webui.pharos.127.0.0.1.sslip.io:8443/oauth/oidc/callback`. That name is not
published yet (subtask 6A), so a real browser sign-in has not been run.

## Desired state

The allocation derives two documents. Adaptors compare them with the provider and change only what differs.

| LiteLLM team | Source |
| --- | --- |
| `team_id` | the allocation's canonical ID (not the Waldur resource) |
| `team_alias` | the Project alias and the first eight characters of the allocation ID (see below) |
| `models`, `max_budget`, `budget_duration`, `tpm_limit`, `rpm_limit` | the entitlement |
| `blocked` | state is `blocked` |
| `metadata` | `pharos_project_id` and the Waldur resource, for traceability |

| Open WebUI | Source |
| --- | --- |
| group | the allocation's name: the Project alias and the first eight characters of the allocation ID |
| models | `<allocation name>.<served name>` for each ordered model |
| enabled | state is `active` |

A Project can hold more than one allocation, one for each Waldur resource, so the Project alias alone would give two of
them the same group and team name: the last to reconcile would win, and ending one would disable the group the other
still pays for. The allocation therefore carries its own name (the Project alias followed by the first eight characters
of its ID), which keeps the readable part and is unique and stable. Two allocations never share a group, a team or a
model prefix.

Personal keys per member are subtask 4B. The project-level key of the earlier design is not part of this slice.

## Idempotency and recovery

- The command ID is the receipt: a retried delivery returns the recorded outcome.
- Redelivering the applied order, or a repeated terminate, changes nothing and publishes nothing.
- A change publishes one durable reconcile intent (`llm_access.allocation.reconcile.v1`, allocation ID and version, which must equal the envelope's aggregate and version).
  Provider calls run from that intent outside the transaction, and record their progress, so an interrupted run
  resumes without duplicating a team. Each adaptor call is read, compare, update.
- Reconciling one allocation is serialised: the whole sequence (read the allocation, call the gateway, record the
  result) runs while holding a session-level PostgreSQL advisory lock on the allocation, taken outside the aggregate
  transaction and bounded to 120 seconds, and the allocation is read again once the lock is held. The broker consumer,
  the recovery sweep and any replica can therefore run at once without an older call landing after a newer one while
  the allocation records the newer as reconciled.
- The gateway credential is never sent to a redirect target: the adaptor does not follow redirects, and any 3xx is a
  failure to retry. A missing team is recognised only by the phrase LiteLLM v1.96.0 uses for it (HTTP 404, "Team not
  found"), pinned by a fixture captured from a running gateway, so an upgrade that rewords it fails a test.
- Orders for one allocation are ordered by their order time, so an old delivery cannot undo a newer one.
- Recovery (subtask 3C): broker retries last only seconds, so a longer gateway outage ends in the dead-letter
  queue. A periodic sweep in the service finds allocations whose reconciled version is behind their entitlement
  version and runs the same reconciler for them, under its own consumer receipt keyed by allocation and version.
  A completed attempt is recorded once and never repeated, a failed one records nothing and is tried on the next
  pass, and one failing allocation does not stop the others. An allocation changed within the grace period is
  left to the broker path. The sweep runs only when the gateway is configured, like the consumer. It needs no
  broker, so it also recovers from a lost message.
  Key creation has its own crash-safe hand-off: the intended secret is sealed and its token is durable before the
  provider call. Termination blocks tokens from both key records and pending provisions, treats LiteLLM's exact
  `Key not found` response as the safe pre-call case, blocks the team, and then erases every pending provision.

## Credentials and open questions

1. **Scoped credentials.** Application containers hold no administrator credentials. Checked on the clean test VM
   (LiteLLM v1.96.0): a key for a `proxy_admin` user with `allowed_routes` limited to team and key routes could
   manage teams and keys and was refused (HTTP 403) on model registration, user creation, spend logs, config
   updates and inference. **But it escalates:** with `/key/generate` allowed it minted keys for the admin user with
   no route limit, and those keys could read spend logs and list keys. A key that may mint keys is a master key.
   The design therefore changes:
   - The Reconciler's LiteLLM credential gets the team routes (`/team/new`, `/team/update`, `/team/info`,
     `/team/block`, `/team/unblock`) and `/key/block`, and no other `/key/*` route. A key that may block keys
     could not unblock, generate, update, delete, rotate, list or read keys, create users, read spend logs or
     register models (measured). It can block any key whose hashed token it holds, which is why the token is kept
     in the key record and never exposed. There is deliberately no `/key/unblock`: ending an allocation and
     revoking a key are final, and the credential must not be able to undo them.
   - Personal key issue (4B) is a separate credential and design. Candidates to test: a non-admin team admin who can
     mint keys for one team only, and LiteLLM's key-generation settings.
   Open WebUI has no scoped admin; it needs an answer before its adaptor.
2. **Binding.** Something maps the Waldur resource to the allocation ID before this command runs. Deterministic
   derivation from the resource is simple but ties the ID to the provider; a stored binding costs a lookup.
3. **Order state.** Confirm that `executing` is what the Site Agent delivers, and what it expects back.
4. **Intake.** Whether the Site Agent calls an HTTP endpoint on Metallo or publishes a message (subtask 1A).
5. **Budget period.** ~~The order form says 30 days; subtask 5B asks for calendar-month resets.~~ **Resolved (October 2026):** the
   period is one calendar month in UTC, written `1mo`, and the gateway's own reset restores it (see the allowance period section).
6. **Result.** The "order done" or "failed" report to Waldur is part of 1A's result format.

## Gateway behaviour measured (clean test VM, LiteLLM v1.96.0)

These shape the adaptors and subtasks 5A and 5C.

- Over budget the gateway answers HTTP 429 with "Budget has been exceeded". The check runs before a request, so
  the last accepted request can overshoot: five requests were accepted against a budget of 0.0004 and the team
  ended at 0.00046. Ten simultaneous requests gave four accepted, six refused and 0.000368 spent in that run, so no
  overshoot there, but one run is not proof and 5A needs a deterministic test.
- Refused and rejected requests are written to the spend log as `failure` rows with no spend. Usage reporting (5C)
  must filter on status.
- `/team/info` shows spend from the database, which lags the gateway's own count by the batch write interval
  (it showed 0.000184 while the gateway counted 0.00046). Reconciliation and reporting must not trust it just
  after a request.
- A streamed request cancelled by the client was billed for the 4 tokens produced, not the 400 requested.
- A request for a model outside the team's models is refused with 403 and costs nothing.
- **Blocking a team does not stop a key at once.** A key already in use kept being served after `/team/block`
  and was refused only after about 60 seconds. A key created after the block, or never used, was refused
  immediately, and `/team/update` with `blocked` behaved the same. Blocking each key of the team (`/key/block`)
  stopped a warm key at once. So termination must block the team **and every key of it**, which needs the key
  records (subtask 1B). The Reconciler now blocks every recorded or pending-provision token before the team, including
  revoked keys. A failure raises, so nothing is recorded as reconciled and the broker or sweep retries. A provision's
  token exists before its provider call; LiteLLM's exact `Key not found` response is safe and any other failure retries.
  The live check of the adaptor in the first gateway PR confirmed the team's `blocked` flag, not that traffic was
  refused.

## Next changes

1. Membership-aware SSO and chat revocation (4A and the remaining part of 4D).
2. Strict concurrent allowance enforcement and acceptance on target GPU hardware (5A and 2A).
3. Public endpoints and operational acceptance (6A–6C).