---
title: "ADR-0020: LLM Access Membership Source of Truth"
---

# ADR-0020: LLM Access Membership Source of Truth

**Status:** Draft

**Date:** 2026-06-30

**Deciders:** Themelio developers (confirmed in session, 2026-06-30)

**Relates to:**
- [ADR-0005 (AAI → Waldur User Synchronization)](0005-aai-waldur-user-sync.md)
- [ADR-0010 (Project-Scoped LLM Gateway, Token Quotas, and API Access)](0010-project-scoped-llm-gateway-token-quotas-api-access.md)
- [ADR-0011 (Human Web UI and Agent Client Access)](0011-human-web-ui-and-agent-client-access.md)

## Context

The LLM access platform needs one authoritative system for user-to-project membership so that:

- Open WebUI group membership stays in sync with who has been allocated access,
- LiteLLM project routing, keys, and budget limits stay aligned with the same
  Waldur project allocation,
- membership addition and revocation can be exercised end to end through a single control plane.

Two candidate sources of truth were considered during design: Keycloak and Waldur.

### Current State

The implementation state as of the decision date:

- `llm-access-root-actor` is a privileged Python service that reconciles LiteLLM teams, budgets, model allowlists, project API keys, and Open WebUI groups and connections.
- The root actor exposes `/waldur/add_project_members`, `/waldur/remove_project_members`, and `/waldur/list_project_members` endpoints that manage Open WebUI group membership directly.
- `waldur-llm-access-site-agent` implements `add_users_to_resource`, `remove_users_from_resource`, and `list_resource_users` via those root actor endpoints.
- A `membership_sync` deployment of the site agent is provisioned alongside `order_process` and `report`.
- A live probe against `staging.aai.pharos.grnet.gr` (the production-like external Keycloak / Pharos AAI stack) confirmed that the external realm's group list for production users is empty — it is a brokered federation identity facade, not a local-group authorization store.
- A read-only probe against the production-shape Waldur instance (`dev.waldur.pharos.grnet.gr`) confirmed that projects such as Buffermusic, DyaniaHealth, and OpenCosmos store human membership in Waldur project membership, while the project resource is keyed by `backend_id == project_slug`.

### Current Problems

Before the decision was finalized, the Proposed ADR text favored Keycloak-authoritative membership. The problems with that model are:

- The external production Keycloak realm is a brokered federation IdP (MyAccessID → NTUA Shibboleth → Keycloak `id` realm). Themelio does not control that shared realm and cannot write per-project groups to it.
- Even if a locally controlled Keycloak realm is introduced, managing group membership in Keycloak in parallel with Waldur creates two editable sources of the same authorization decision with no reconciliation bridge.
- Waldur already drives the full resource lifecycle (create, update, delete, limits, usage). Forking membership into a separate system produces a split control plane for the same resource.
- The existing `waldur-k8s-site-agent` already resolves membership through site agent callbacks to a privileged root actor. The pattern is proven in the same codebase.

## Decision Drivers

- Single control plane for all Waldur-allocated LLM-access resources, including who can use them.
- Consistency with the existing `waldur-k8s-site-agent` pattern for membership reconciliation.
- Avoidance of dependency on a shared external Keycloak realm that Themelio does not control.
- Ability to prove revocation end to end in the existing local test stack without a separate Keycloak admin path.

## Considered Options

### Option A: Waldur authoritative (adopted)

Waldur is the single source of truth for who has access to which LLM-access project. The
`membership_sync` backend in `waldur-llm-access-site-agent` drives periodic reconciliation.
Membership changes are written to Open WebUI groups via the `llm-access-root-actor`. No
Keycloak membership writes are performed.

**Pros:**
- Single control plane for the full resource lifecycle including membership.
- Consistent with the `waldur-k8s-site-agent` pattern already in the repo.
- No dependency on writing to the external Keycloak / AAI realm, which we do not control.
- Membership revocation is testable immediately in the local stack.

**Cons:**
- Open WebUI does not derive membership from OIDC token claims (`project_ids`); it derives it
  from group membership managed by the root actor. The local Keycloak group-to-claim path
  is no longer the production authorization channel.
- Users not yet registered in Open WebUI (i.e., have not yet logged in) are returned as
  `not_found` by the root actor and retried on the next `membership_sync` cycle.

### Option B: Keycloak authoritative (rejected)

Keycloak project groups drive `project_ids` claims, which Open WebUI synchronizes on login.
Waldur would have remained authoritative only for resources and budgets, not membership.

This option was rejected because the production external Keycloak realm is a brokered
federation IdP that Themelio does not control. Writing per-project groups there is not a
viable production path.

## Decision

We will use **Waldur as the authoritative system for user-to-project membership** for
LLM access (Option A).

Specifically:

- The `waldur-llm-access-site-agent` `membership_sync` deployment reconciles Waldur
  project membership into Open WebUI groups via the `llm-access-root-actor`.
- The root actor's `/waldur/add_project_members`, `/waldur/remove_project_members`, and
  `/waldur/list_project_members` endpoints own the Open WebUI group member list.
- The external Keycloak / AAI layer remains the **authentication** authority only. It is
  not written to for per-project authorization state.
- The local Keycloak `project_ids` claim path remains in the local dev stack for testing
  the OIDC login flow, but it is not the production authorization channel and
  must not be treated as evidence that Waldur membership sync is working.

**See also:** [DD-0020: Implementation details](../content/dd/0020-llm-access-membership-source-of-truth.md)

### Update (October 2026): Beto decides Project membership

The platform now has canonical owners for this fact, so the decision above is narrowed:

- **Beto's Project and Party Governance contexts own Project membership and access policy.** Waldur is still where a
  Project's membership comes from, but Beto acquires it, gives it canonical Party and Project identities and publishes it
  as integration events and checkpoints. LLM Access keeps its own copy of those facts, like every Metallo context that
  decides access (#780), and takes Beto's rule from it. It does not read Waldur for membership.
- **LLM Access writes the Open WebUI group**, as the one writer of that target, and records what it has applied. The root
  actor is not part of this path. The group follows Beto's membership: a removal Beto has applied takes the person out at
  the next look, and a replayed message from the Site Agent cannot put them back.
- **The Site Agent names the people requesting personal keys**, with each member's sign-in subject, address and Waldur
  user UUID. The UUID pairs them with the membership Beto publishes. In `beto` mode, issuance, retries and rotation require
  Beto's loaded authority to admit that member before any key is created or resumed. Unknown or pending authority defers
  the request without issuing a key; a refused or unpaired member receives a business refusal (#884).
- The authentication layer is unchanged: AAI says who a person is, and no group is written to it.

Until Beto's Waldur acquisition is active in an environment, `LLM_ACCESS_CHAT_MEMBERSHIP=site-agent` keeps the Site Agent
deciding chat membership, as before, and `beto` switches to Beto's membership (#769).

### Beto cutover

Existing installations must inventory and repair member links before switching to `beto`. A held Project refuses
unpaired keys, including keys created by an earlier no-chat installation with no member record. This revokes the key
at the gateway and abandons a pending delivery; a later membership refresh cannot restore that key.

Complete this sequence for a switch that preserves authorised members' existing keys:

1. Deploy the version that persists member links in both authority modes to every issuing process, with
   `LLM_ACCESS_CHAT_MEMBERSHIP=site-agent` still selected. Drain old processes before proceeding. Open WebUI can remain
   disabled throughout this preparation.
2. Run [the read-only preflight](../../services/metallo/contexts/llm_access/adaptors/beto_cutover.sql) against the
   LLM Access database using its scoped connection. It returns `unlinked_personal_keys`, `unpaired_personal_keys`
   and `unpaired_wanted_members`. The first two count all active personal keys, including incomplete provisions and
   members never wanted in chat; their join includes both allocation and member ID. A missing, blank or unusable
   Waldur user UUID counts as unpaired. Project keys and revoked keys are excluded.
3. Refresh existing active owners through the authenticated Site Agent member-add route while still in `site-agent`
   mode. Use authoritative Site Agent/Waldur data: the original member reference/sign-in subject, current address,
   correct Waldur user UUID and a fresh request ID. An unchanged member reference reuses the existing personal key,
   gateway token and delivery while saving the missing link. Retry incomplete provisioning through the same request.
   The derived `memberId` cannot recover the original subject or Waldur UUID; never invent either binding. Owners
   whose identity cannot be recovered must be resolved before a continuity-preserving switch.
4. Confirm Beto has acquired the Projects and loaded the required Project, Party and binding facts. Pause member-add
   and rotation intake and drain in-flight issuance, then run the preflight again. All three counts must be zero.
   Keep issuance paused during the mode change so an old process or an add without a UUID cannot introduce a new
   blocker after the check. This is an operator preflight, rather than a global Beto startup dependency.
5. Switch to `beto`, resume intake and verify that admitted owners retain the same key IDs and that refused members'
   keys are blocked. Pending chat work can recover separately when Open WebUI is available.

The runtime recovery rules remain:

- An allocation whose Project the view does not hold yet is left as it is, with a warning from the sweep, so selecting the
  mode before Beto has acquired the Project does not empty its group. A Project the view holds that names nobody refuses
  them, which is the fail-closed case.
- Every member refused by a held Project loses their personal keys, stopped at the gateway before they leave the group.
  This includes someone never admitted to chat, someone without a usable Waldur user UUID and a key with no member link.
  Reconciliation repairs old keys independently of the group's applied state and erases pending deliveries after
  revocation. A replayed add cannot issue a replacement while Beto refuses the member. An incomplete view or missing
  Project continues to preserve existing membership while preventing new issuance.
- The sweep stops those keys through the same member use cases and the same lock as the member routes, so the two never
  issue and stop a key at the same time. The lock is a named database advisory lock under a re-entrant thread lock, because
  a rolling update runs the new pod's sweep while the old pod still serves the routes.
- Each record keeps the Open WebUI account the group was last told about, so a member whose address or subject changed, or
  who leaves, is removed from that account and not only from the one the current address and subject find. When the new
  address and subject have no account yet, the earlier account leaves the group at once and only the add waits for the
  sign-in: a subject can change who the person is, so access does not continue on the old account meanwhile. For a change
  of address alone, the same person is out of the group until they sign in with the new address.

## Consequences

**Positive:**
- Membership add, update, and revocation are driven entirely from Waldur, closing the
  authorization lifecycle loop within the existing operational control plane.
- The existing site-agent pattern (`add_users_to_resource` / `remove_users_from_resource`)
  works without requiring a Keycloak admin client or a shared external realm.
- Local stack smoke tests can cover end-to-end membership flows in the same environment as
  other lifecycle operations.

**Negative:**
- Users must have logged in to Open WebUI at least once before the root actor can resolve
  their email to an Open WebUI user ID. First-login provisioning is outside the membership
  sync path; those users are retried on the next sync cycle.
- Open WebUI group membership is now managed externally (root actor), not via Open WebUI's
  own admin UI. Administrators should not edit group membership directly in Open WebUI.
- The chat key's allowlist includes `/api/v1/users`. Open WebUI scopes a key by path, not by method, so the key can also
  change a role and delete a user. This is accepted: the key is already an administrator's and can read every
  connection's secret, and the adaptor only searches, reads one account and changes group members.
- The `project_ids` OIDC claim path (used in the local Keycloak dev stack) does not carry
  membership to Open WebUI in production; only the Waldur-driven group assignment does.

**Neutral:**
- External AAI / Pharos Keycloak remains the authentication authority and continues to
  issue user identity tokens. Nothing changes on the authn side.
- The local Keycloak group-to-claim path is kept for local dev smoke testing and can be
  removed later if it adds maintenance cost without value.
- Local validation of this ADR must seed users, projects, and resources in Waldur.
  Keycloak-only fixtures are insufficient to prove the real membership flow.

### Migration

The Beto cutover requires the [member-link preflight and backfill](#beto-cutover) above. Existing aggregate schemas
already support the links, so no database schema migration is needed. Correct Open WebUI group membership alone
does not establish the identity link required for personal-key authority.

## Failure Behavior

1. If a user's Waldur project membership is removed, the next `membership_sync` cycle
   removes them from the corresponding Open WebUI group. Access to project-qualified models
   through Open WebUI stops at the next session check.
2. Users not yet registered in Open WebUI (no login yet) are returned as `not_found` by the
   root actor and remain in the `not_found` list until they first log in and the next sync
   cycle runs.
3. LiteLLM enforcement remains independent: the project-restricted API key is revoked or
   blocked at the LiteLLM level even if Open WebUI state is temporarily stale.
4. If the root actor is unavailable during a membership sync, the site agent logs the error
   and retries on the next cycle; no membership state is corrupted.

## Tests

The required proof for this ADR:

1. A user added to a Waldur project appears in the Open WebUI group after the next
   `membership_sync` cycle.
2. A user removed from a Waldur project is removed from the Open WebUI group after the
   next `membership_sync` cycle.
3. A user not yet registered in Open WebUI is listed as `not_found` and is retried on
   the following cycle once they log in.
4. LiteLLM still rejects cross-project API key use independently of UI membership state.

## References

- External Keycloak probe findings (session 2026-06-30): brokered federation realm,
  empty groups list for production users, `project_ids` claim absent from default
  `userinfo` response — confirming that the external realm is authn-only.
- Production-shape Waldur probe findings (session 2026-06-30): Buffermusic,
  DyaniaHealth, and OpenCosmos expose project membership and resource linkage in
  Waldur itself, with project resources keyed by `backend_id == project_slug`.
- `services/llm-access-root-actor/llm_access_root_actor/service.py` — `add_project_members`,
  `remove_project_members`, `list_project_members` implementations.
- `services/waldur-llm-access-site-agent/waldur_site_agent_llm_access/backend.py` —
  `add_users_to_resource`, `remove_users_from_resource`.
- `gitops/manifests/llm-access/base/waldur-llm-access-site-agent/deployment-membership-sync.yaml`