> For AI agents: the complete documentation index is available at /llms.txt, the full documentation bundle is available at /llms-full.txt. > **Themelio compatibility reference.** This page describes the retained Themelio implementation or its historical design. Check the [parallel-operation guide](/compatibility/index.md) before applying it to the current platform. # DD-0020: LLM Access Membership Source of Truth — Implementation **Related ADR:** [ADR-0020: LLM Access Membership Source of Truth](/adr/0020-llm-access-membership-source-of-truth.md) **Status:** Draft **Date:** 2026-06-30 **Authors:** TBD ## ADR-derived technical notes ### Implementation branches The implementation described in this DD lives in the following stacked `local/*` branches, none of which have been merged to `main` yet: | Branch | Key content | | ------------------------------ | ---------------------------------------------------------------------------------------------------------------------------- | | `local/llm-access-openwebui` | `llm-access-root-actor` service, `waldur-llm-access-site-agent`, gitops manifests, `waldur-llm-access-offering` Ansible role | | `local/llm-access-waldur-seed` | Local Waldur AI offering seed fixtures; idempotent local stack integration | :::warning All branches have been tested on a local Minikube cluster in **CPU mode only**. Membership sync and Open WebUI group management are exercised against local instances of Waldur, LiteLLM, and Open WebUI. Staging and production validation is pending. ::: ### `llm-access` namespace topology All services run in the dedicated `llm-access` Kubernetes namespace (separate from the `litellm`, `open-webui`, `waldur`, and `inference` namespaces). The namespace is declared in `gitops/manifests/llm-access/base/namespace.yaml` and managed by an ArgoCD Application. ``` llm-access namespace ├── Deployment: llm-access-root-actor (FastAPI, port 8000) │ ServiceAccount: llm-access-root-actor │ Role/RoleBinding: read+write llm-access/* Secrets │ ├── Deployment: waldur-llm-access-agent-order-process (waldur_site_agent -m order_process) ├── Deployment: waldur-llm-access-agent-report (waldur_site_agent -m report) └── Deployment: waldur-llm-access-agent-membership-sync (waldur_site_agent -m membership_sync) ServiceAccount: waldur-llm-access-site-agent (shared by all three) Secret: waldur-llm-access-site-agent-config (shared by all three) ``` All four deployments are declared in `gitops/manifests/llm-access/base/{llm-access-root-actor,waldur-llm-access-site-agent}/`. ### `llm-access-root-actor` service **Image**: `llm-access-root-actor` (Python/FastAPI/uvicorn), port 8000. **Authentication**: all `/waldur/*` endpoints require `Authorization: Bearer `. The token is set from the `LLM_ACCESS_ROOT_ACTOR_INTERNAL_TOKEN` env var (sourced from the `llm-access-root-actor-secrets` Kubernetes Secret). The health endpoint `/-/health` is unauthenticated. **Environment** (from gitops base Deployment): | Variable | Value | | ---------------------------------------------------- | -------------------------------------------------- | | `LITELLM_BASE_URL` | `http://litellm.litellm.svc.cluster.local:4000` | | `LITELLM_MASTER_KEY` | from Secret `llm-access-root-actor-secrets` | | `OPENWEBUI_BASE_URL` | `http://open-webui.open-webui.svc.cluster.local` | | `OPENWEBUI_OPENAI_BASE_URL` | `http://litellm.litellm.svc.cluster.local:4000/v1` | | `OPENWEBUI_ADMIN_EMAIL` / `OPENWEBUI_ADMIN_PASSWORD` | from Secret | | `PROJECT_KEY_SECRET_BACKEND` | `kubernetes` | | `PROJECT_KEY_SECRET_NAMESPACE` | `llm-access` | | `PROJECT_KEY_SECRET_NAME_PREFIX` | `llm-access-openwebui-key-` | **RBAC**: the `llm-access-root-actor` ServiceAccount has a Role in the `llm-access` namespace granting `get`, `create`, `update`, `delete` on `secrets` — required to store/retrieve per-project LiteLLM virtual keys as Kubernetes Secrets. **Endpoints** (all under `/waldur/`, all require Bearer token except `/-/health`): | Method | Path | Service method | | ------ | -------------------------------- | --------------------------------------------------------------- | | GET | `/-/health` | `health()` — checks LiteLLM ping, Open WebUI ping, secret store | | POST | `/waldur/project_create` | `reconcile_project()` | | POST | `/waldur/project_update` | `reconcile_project_update()` | | POST | `/waldur/project_delete` | `delete_project()` | | POST | `/waldur/rotate_project_key` | `rotate_project_key()` | | POST | `/waldur/report_usage` | `report_usage()` | | POST | `/waldur/add_project_members` | `add_project_members()` | | POST | `/waldur/remove_project_members` | `remove_project_members()` | | GET | `/waldur/list_project_members` | `list_project_members()` | **Key storage**: per-project LiteLLM virtual keys are stored as Kubernetes Secrets with name `llm-access-openwebui-key-` in the `llm-access` namespace. The `secret_store.KubernetesSecretStore` class manages read/write/delete operations. ### `waldur-llm-access-site-agent` deployments Three deployments share the same image (`waldur-llm-access-site-agent`) and the same config Secret (`waldur-llm-access-site-agent-config`, mounted at `/etc/waldur/`). They are differentiated only by the `-m` mode flag passed to `waldur_site_agent`: | Deployment | Mode | Period | Root-actor endpoints called | | ----------------------------------------- | ----------------- | ------------------- | ----------------------------------------------------------------------- | | `waldur-llm-access-agent-order-process` | `order_process` | every 2 min | `project_create`, `project_update`, `project_delete` | | `waldur-llm-access-agent-report` | `report` | (scheduler default) | `report_usage` | | `waldur-llm-access-agent-membership-sync` | `membership_sync` | every 5 min | `add_project_members`, `remove_project_members`, `list_project_members` | The config Secret is rendered by the `waldur-llm-access-offering` Ansible role from a Jinja2 template. It includes the Waldur API URL, site-agent service token, offering UUID, root-actor URL and token, `available_models` list (derived from `inference.yml`), default `budget_duration`, and the LLM access backend component schema. The `waldur-llm-access-offering` role restarts all three deployments automatically when the config Secret content changes (detected via `register: waldur_llm_access_site_agent_config_secret_result`). ### Membership sync data flow ``` Waldur project membership (source of truth) ↓ every 5 min waldur-llm-access-agent-membership-sync (waldur_site_agent -m membership_sync) ↓ POST /waldur/add_project_members ↓ POST /waldur/remove_project_members llm-access-root-actor ↓ GET /api/v1/users/ (resolve email → Open WebUI user ID) Open WebUI group membership (group name == resource_backend_id) ``` ### `not_found` retry When a user has not yet logged in to Open WebUI, `GET /api/v1/users/` returns no match. The root actor returns that email in the `not_found` list. The site agent returns only successfully-resolved emails to Waldur as "added"; Waldur retries `add_users_to_resource` on the next membership_sync cycle (every 5 minutes). No retry state is stored. ### LiteLLM enforcement is independent of Open WebUI membership Removing a Waldur project membership removes the user from the Open WebUI group (blocking UI model access). The project-scoped LiteLLM team budget and virtual key remain active until the Waldur resource itself is terminated (which triggers `project_delete` via the `order_process` agent). This means LiteLLM API key revocation is a resource lifecycle event, not a membership event. ### Local Keycloak `project_ids` claim path (local dev only) The local dev stack seeds Keycloak groups (`project-a`, `project-b`) and configures a custom `project_ids` OIDC claim for Open WebUI. This is a **local-only bootstrap shim** to test the OIDC login flow; it is not the production authorization channel. Group membership in production is set exclusively by the root actor via `membership_sync`. ### Ansible local seed role `devops/ansible/roles/waldur-llm-access-local-seed/` creates local Waldur fixtures (4 users, 2 projects, 1 offering, 4 offering-users, 2 resources) needed to exercise the full membership sync path in the local Minikube stack. It is idempotent and is called by the `local-setup`, `local-up-full`, and `waldur-local-patches` plays. The seed was verified live on `themelio-local`: second reconcile completes with `changed=0`, 4 active offering-users, membership shape correct (project-a/b members, owui-none excluded).
Page source Source snapshot · Revision `f928834c8`.