> For AI agents: the complete documentation index is available at /llms.txt, the full documentation bundle is available at /llms-full.txt.

> **Themelio compatibility reference.** This page describes the retained Themelio implementation or its historical design. Check the [parallel-operation guide](/compatibility/index.md) before applying it to the current platform.

# ADR-0020: LLM Access Membership Source of Truth

**Status:** Draft

**Date:** 2026-06-30

**Deciders:** Themelio developers (confirmed in session, 2026-06-30)

**Relates to:**

- [ADR-0005 (AAI → Waldur User Synchronization)](/adr/0005-aai-waldur-user-sync.md)
- [ADR-0010 (Project-Scoped LLM Gateway, Token Quotas, and API Access)](/adr/0010-project-scoped-llm-gateway-token-quotas-api-access.md)
- [ADR-0011 (Human Web UI and Agent Client Access)](/adr/0011-human-web-ui-and-agent-client-access.md)

## Context

The LLM access platform needs one authoritative system for user-to-project membership so that:

- Open WebUI group membership stays in sync with who has been allocated access,
- LiteLLM project routing, keys, and budget limits stay aligned with the same
  Waldur project allocation,
- membership addition and revocation can be exercised end to end through a single control plane.

Two candidate sources of truth were considered during design: Keycloak and Waldur.

### Current State

The implementation state as of the decision date:

- `llm-access-root-actor` is a privileged Python service that reconciles LiteLLM teams, budgets, model allowlists, project API keys, and Open WebUI groups and connections.
- The root actor exposes `/waldur/add_project_members`, `/waldur/remove_project_members`, and `/waldur/list_project_members` endpoints that manage Open WebUI group membership directly.
- `waldur-llm-access-site-agent` implements `add_users_to_resource`, `remove_users_from_resource`, and `list_resource_users` via those root actor endpoints.
- A `membership_sync` deployment of the site agent is provisioned alongside `order_process` and `report`.
- A live probe against `staging.aai.pharos.grnet.gr` (the production-like external Keycloak / Pharos AAI stack) confirmed that the external realm's group list for production users is empty — it is a brokered federation identity facade, not a local-group authorization store.
- A read-only probe against the production-shape Waldur instance (`dev.waldur.pharos.grnet.gr`) confirmed that projects such as Buffermusic, DyaniaHealth, and OpenCosmos store human membership in Waldur project membership, while the project resource is keyed by `backend_id == project_slug`.

### Current Problems

Before the decision was finalized, the Proposed ADR text favored Keycloak-authoritative membership. The problems with that model are:

- The external production Keycloak realm is a brokered federation IdP (MyAccessID → NTUA Shibboleth → Keycloak `id` realm). Themelio does not control that shared realm and cannot write per-project groups to it.
- Even if a locally controlled Keycloak realm is introduced, managing group membership in Keycloak in parallel with Waldur creates two editable sources of the same authorization decision with no reconciliation bridge.
- Waldur already drives the full resource lifecycle (create, update, delete, limits, usage). Forking membership into a separate system produces a split control plane for the same resource.
- The existing `waldur-k8s-site-agent` already resolves membership through site agent callbacks to a privileged root actor. The pattern is proven in the same codebase.

## Decision Drivers

- Single control plane for all Waldur-allocated LLM-access resources, including who can use them.
- Consistency with the existing `waldur-k8s-site-agent` pattern for membership reconciliation.
- Avoidance of dependency on a shared external Keycloak realm that Themelio does not control.
- Ability to prove revocation end to end in the existing local test stack without a separate Keycloak admin path.

## Considered Options

### Option A: Waldur authoritative (adopted)

Waldur is the single source of truth for who has access to which LLM-access project. The
`membership_sync` backend in `waldur-llm-access-site-agent` drives periodic reconciliation.
Membership changes are written to Open WebUI groups via the `llm-access-root-actor`. No
Keycloak membership writes are performed.

**Pros:**

- Single control plane for the full resource lifecycle including membership.
- Consistent with the `waldur-k8s-site-agent` pattern already in the repo.
- No dependency on writing to the external Keycloak / AAI realm, which we do not control.
- Membership revocation is testable immediately in the local stack.

**Cons:**

- Open WebUI does not derive membership from OIDC token claims (`project_ids`); it derives it
  from group membership managed by the root actor. The local Keycloak group-to-claim path
  is no longer the production authorization channel.
- Users not yet registered in Open WebUI (i.e., have not yet logged in) are returned as
  `not_found` by the root actor and retried on the next `membership_sync` cycle.

### Option B: Keycloak authoritative (rejected)

Keycloak project groups drive `project_ids` claims, which Open WebUI synchronizes on login.
Waldur would have remained authoritative only for resources and budgets, not membership.

This option was rejected because the production external Keycloak realm is a brokered
federation IdP that Themelio does not control. Writing per-project groups there is not a
viable production path.

## Decision

We will use **Waldur as the authoritative system for user-to-project membership** for
LLM access (Option A).

Specifically:

- The `waldur-llm-access-site-agent` `membership_sync` deployment reconciles Waldur
  project membership into Open WebUI groups via the `llm-access-root-actor`.
- The root actor's `/waldur/add_project_members`, `/waldur/remove_project_members`, and
  `/waldur/list_project_members` endpoints own the Open WebUI group member list.
- The external Keycloak / AAI layer remains the **authentication** authority only. It is
  not written to for per-project authorization state.
- The local Keycloak `project_ids` claim path remains in the local dev stack for testing
  the OIDC login flow, but it is not the production authorization channel and
  must not be treated as evidence that Waldur membership sync is working.

**See also:** [DD-0020: Implementation details](/dd/0020-llm-access-membership-source-of-truth.md)

## Consequences

**Positive:**

- Membership add, update, and revocation are driven entirely from Waldur, closing the
  authorization lifecycle loop within the existing operational control plane.
- The existing site-agent pattern (`add_users_to_resource` / `remove_users_from_resource`)
  works without requiring a Keycloak admin client or a shared external realm.
- Local stack smoke tests can cover end-to-end membership flows in the same environment as
  other lifecycle operations.

**Negative:**

- Users must have logged in to Open WebUI at least once before the root actor can resolve
  their email to an Open WebUI user ID. First-login provisioning is outside the membership
  sync path; those users are retried on the next sync cycle.
- Open WebUI group membership is now managed externally (root actor), not via Open WebUI's
  own admin UI. Administrators should not edit group membership directly in Open WebUI.
- The `project_ids` OIDC claim path (used in the local Keycloak dev stack) does not carry
  membership to Open WebUI in production; only the Waldur-driven group assignment does.

**Neutral:**

- External AAI / Pharos Keycloak remains the authentication authority and continues to
  issue user identity tokens. Nothing changes on the authn side.
- The local Keycloak group-to-claim path is kept for local dev smoke testing and can be
  removed later if it adds maintenance cost without value.
- Local validation of this ADR must seed users, projects, and resources in Waldur.
  Keycloak-only fixtures are insufficient to prove the real membership flow.

### Migration

No migration of existing membership state is required. The membership sync deployment will
reconcile the current Waldur project membership into Open WebUI on its first run. Users
who are already in the correct Open WebUI groups (from the local bootstrap or manual setup)
will not be re-added unnecessarily.

## Failure Behavior

1. If a user's Waldur project membership is removed, the next `membership_sync` cycle
   removes them from the corresponding Open WebUI group. Access to project-qualified models
   through Open WebUI stops at the next session check.
2. Users not yet registered in Open WebUI (no login yet) are returned as `not_found` by the
   root actor and remain in the `not_found` list until they first log in and the next sync
   cycle runs.
3. LiteLLM enforcement remains independent: the project-restricted API key is revoked or
   blocked at the LiteLLM level even if Open WebUI state is temporarily stale.
4. If the root actor is unavailable during a membership sync, the site agent logs the error
   and retries on the next cycle; no membership state is corrupted.

## Tests

The required proof for this ADR:

1. A user added to a Waldur project appears in the Open WebUI group after the next
   `membership_sync` cycle.
2. A user removed from a Waldur project is removed from the Open WebUI group after the
   next `membership_sync` cycle.
3. A user not yet registered in Open WebUI is listed as `not_found` and is retried on
   the following cycle once they log in.
4. LiteLLM still rejects cross-project API key use independently of UI membership state.

## References

- External Keycloak probe findings (session 2026-06-30): brokered federation realm,
  empty groups list for production users, `project_ids` claim absent from default
  `userinfo` response — confirming that the external realm is authn-only.
- Production-shape Waldur probe findings (session 2026-06-30): Buffermusic,
  DyaniaHealth, and OpenCosmos expose project membership and resource linkage in
  Waldur itself, with project resources keyed by `backend_id == project_slug`.
- `services/llm-access-root-actor/llm_access_root_actor/service.py` — `add_project_members`,
  `remove_project_members`, `list_project_members` implementations.
- `services/waldur-llm-access-site-agent/waldur_site_agent_llm_access/backend.py` —
  `add_users_to_resource`, `remove_users_from_resource`.
- `gitops/manifests/llm-access/base/waldur-llm-access-site-agent/deployment-membership-sync.yaml`

<details class="page-source"><summary>Page source</summary>

<a href="/source/docs/content/adr/0020-llm-access-membership-source-of-truth.md.txt">Source snapshot</a> · Revision `f928834c8`.

</details>