---
title: "DD-0020: LLM Access Membership Source of Truth — Implementation"
---

# DD-0020: LLM Access Membership Source of Truth — Implementation

**Related ADR:** [ADR-0020: LLM Access Membership Source of Truth](/adr/0020-llm-access-membership-source-of-truth)

**Status:** Draft

**Date:** 2026-06-30

**Authors:** TBD

## ADR-derived technical notes

### Implementation branches

The implementation described in this DD lives in the following stacked `local/*` branches,
none of which have been merged to `main` yet:

| Branch | Key content |
|---|---|
| `local/llm-access-openwebui` | `llm-access-root-actor` service, `waldur-llm-access-site-agent`, gitops manifests, `waldur-llm-access-offering` Ansible role |
| `local/llm-access-waldur-seed` | Local Waldur AI offering seed fixtures; idempotent local stack integration |

:::warning
All branches have been tested on a local Minikube cluster in **CPU mode only**.
Membership sync and Open WebUI group management are exercised against local instances
of Waldur, LiteLLM, and Open WebUI. Staging and production validation is pending.
:::

### `llm-access` namespace topology

All services run in the dedicated `llm-access` Kubernetes namespace (separate from the
`litellm`, `open-webui`, `waldur`, and `inference` namespaces). The namespace is declared
in `gitops/manifests/llm-access/base/namespace.yaml` and managed by an ArgoCD Application.

```
llm-access namespace
├── Deployment: llm-access-root-actor          (FastAPI, port 8000)
│     ServiceAccount: llm-access-root-actor
│     Role/RoleBinding: read+write llm-access/* Secrets
│
├── Deployment: waldur-llm-access-agent-order-process   (waldur_site_agent -m order_process)
├── Deployment: waldur-llm-access-agent-report          (waldur_site_agent -m report)
└── Deployment: waldur-llm-access-agent-membership-sync (waldur_site_agent -m membership_sync)
      ServiceAccount: waldur-llm-access-site-agent (shared by all three)
      Secret: waldur-llm-access-site-agent-config   (shared by all three)
```

All four deployments are declared in
`gitops/manifests/llm-access/base/{llm-access-root-actor,waldur-llm-access-site-agent}/`.

### `llm-access-root-actor` service

**Image**: `llm-access-root-actor` (Python/FastAPI/uvicorn), port 8000.

**Authentication**: all `/waldur/*` endpoints require `Authorization: Bearer <token>`.
The token is set from the `LLM_ACCESS_ROOT_ACTOR_INTERNAL_TOKEN` env var (sourced from
the `llm-access-root-actor-secrets` Kubernetes Secret). The health endpoint `/-/health`
is unauthenticated.

**Environment** (from gitops base Deployment):

| Variable | Value |
|---|---|
| `LITELLM_BASE_URL` | `http://litellm.litellm.svc.cluster.local:4000` |
| `LITELLM_MASTER_KEY` | from Secret `llm-access-root-actor-secrets` |
| `OPENWEBUI_BASE_URL` | `http://open-webui.open-webui.svc.cluster.local` |
| `OPENWEBUI_OPENAI_BASE_URL` | `http://litellm.litellm.svc.cluster.local:4000/v1` |
| `OPENWEBUI_ADMIN_EMAIL` / `OPENWEBUI_ADMIN_PASSWORD` | from Secret |
| `PROJECT_KEY_SECRET_BACKEND` | `kubernetes` |
| `PROJECT_KEY_SECRET_NAMESPACE` | `llm-access` |
| `PROJECT_KEY_SECRET_NAME_PREFIX` | `llm-access-openwebui-key-` |

**RBAC**: the `llm-access-root-actor` ServiceAccount has a Role in the `llm-access` namespace
granting `get`, `create`, `update`, `delete` on `secrets` — required to store/retrieve
per-project LiteLLM virtual keys as Kubernetes Secrets.

**Endpoints** (all under `/waldur/`, all require Bearer token except `/-/health`):

| Method | Path | Service method |
|---|---|---|
| GET | `/-/health` | `health()` — checks LiteLLM ping, Open WebUI ping, secret store |
| POST | `/waldur/project_create` | `reconcile_project()` |
| POST | `/waldur/project_update` | `reconcile_project_update()` |
| POST | `/waldur/project_delete` | `delete_project()` |
| POST | `/waldur/rotate_project_key` | `rotate_project_key()` |
| POST | `/waldur/report_usage` | `report_usage()` |
| POST | `/waldur/add_project_members` | `add_project_members()` |
| POST | `/waldur/remove_project_members` | `remove_project_members()` |
| GET | `/waldur/list_project_members` | `list_project_members()` |

**Key storage**: per-project LiteLLM virtual keys are stored as Kubernetes Secrets with
name `llm-access-openwebui-key-<resource_backend_id>` in the `llm-access` namespace.
The `secret_store.KubernetesSecretStore` class manages read/write/delete operations.

### `waldur-llm-access-site-agent` deployments

Three deployments share the same image (`waldur-llm-access-site-agent`) and the same
config Secret (`waldur-llm-access-site-agent-config`, mounted at `/etc/waldur/`). They
are differentiated only by the `-m` mode flag passed to `waldur_site_agent`:

| Deployment | Mode | Period | Root-actor endpoints called |
|---|---|---|---|
| `waldur-llm-access-agent-order-process` | `order_process` | every 2 min | `project_create`, `project_update`, `project_delete` |
| `waldur-llm-access-agent-report` | `report` | (scheduler default) | `report_usage` |
| `waldur-llm-access-agent-membership-sync` | `membership_sync` | every 5 min | `add_project_members`, `remove_project_members`, `list_project_members` |

The config Secret is rendered by the `waldur-llm-access-offering` Ansible role from a
Jinja2 template. It includes the Waldur API URL, site-agent service token, offering UUID,
root-actor URL and token, `available_models` list (derived from `inference.yml`), default
`budget_duration`, and the LLM access backend component schema.

The `waldur-llm-access-offering` role restarts all three deployments automatically when
the config Secret content changes (detected via `register: waldur_llm_access_site_agent_config_secret_result`).

### Membership sync data flow

```
Waldur project membership (source of truth)
  ↓  every 5 min
waldur-llm-access-agent-membership-sync
  (waldur_site_agent -m membership_sync)
  ↓  POST /waldur/add_project_members
  ↓  POST /waldur/remove_project_members
llm-access-root-actor
  ↓  GET /api/v1/users/ (resolve email → Open WebUI user ID)
Open WebUI group membership
  (group name == resource_backend_id)
```

### `not_found` retry

When a user has not yet logged in to Open WebUI, `GET /api/v1/users/` returns no match.
The root actor returns that email in the `not_found` list. The site agent returns only
successfully-resolved emails to Waldur as "added"; Waldur retries `add_users_to_resource`
on the next membership_sync cycle (every 5 minutes). No retry state is stored.

### LiteLLM enforcement is independent of Open WebUI membership

Removing a Waldur project membership removes the user from the Open WebUI group (blocking
UI model access). The project-scoped LiteLLM team budget and virtual key remain active
until the Waldur resource itself is terminated (which triggers `project_delete` via the
`order_process` agent). This means LiteLLM API key revocation is a resource lifecycle
event, not a membership event.

### Local Keycloak `project_ids` claim path (local dev only)

The local dev stack seeds Keycloak groups (`project-a`, `project-b`) and configures a
custom `project_ids` OIDC claim for Open WebUI. This is a **local-only bootstrap shim**
to test the OIDC login flow; it is not the production authorization channel. Group
membership in production is set exclusively by the root actor via `membership_sync`.

### Ansible local seed role

`devops/ansible/roles/waldur-llm-access-local-seed/` creates local Waldur fixtures
(4 users, 2 projects, 1 offering, 4 offering-users, 2 resources) needed to exercise the
full membership sync path in the local Minikube stack. It is idempotent and is called by
the `local-setup`, `local-up-full`, and `waldur-local-patches` plays.
The seed was verified live on `themelio-local`: second reconcile completes with `changed=0`,
4 active offering-users, membership shape correct (project-a/b members, owui-none excluded).

<!-- All other DD sections are intentionally left for a later session. -->