Themelio compatibility reference. This page describes the retained Themelio implementation or its historical design. Check the parallel-operation guide before applying it to the current platform.

#DD-0020: LLM Access Membership Source of Truth — Implementation

Related ADR: ADR-0020: LLM Access Membership Source of Truth

Status: Draft

Date: 2026-06-30

Authors: TBD

#ADR-derived technical notes

#Implementation branches

The implementation described in this DD lives in the following stacked local/* branches, none of which have been merged to main yet:

#llm-access namespace topology

All services run in the dedicated llm-access Kubernetes namespace (separate from the litellm, open-webui, waldur, and inference namespaces). The namespace is declared in gitops/manifests/llm-access/base/namespace.yaml and managed by an ArgoCD Application.

llm-access namespace
├── Deployment: llm-access-root-actor          (FastAPI, port 8000)
│     ServiceAccount: llm-access-root-actor
│     Role/RoleBinding: read+write llm-access/* Secrets
│
├── Deployment: waldur-llm-access-agent-order-process   (waldur_site_agent -m order_process)
├── Deployment: waldur-llm-access-agent-report          (waldur_site_agent -m report)
└── Deployment: waldur-llm-access-agent-membership-sync (waldur_site_agent -m membership_sync)
      ServiceAccount: waldur-llm-access-site-agent (shared by all three)
      Secret: waldur-llm-access-site-agent-config   (shared by all three)

All four deployments are declared in gitops/manifests/llm-access/base/{llm-access-root-actor,waldur-llm-access-site-agent}/.

#llm-access-root-actor service

Image: llm-access-root-actor (Python/FastAPI/uvicorn), port 8000.

Authentication: all /waldur/* endpoints require Authorization: Bearer <token>. The token is set from the LLM_ACCESS_ROOT_ACTOR_INTERNAL_TOKEN env var (sourced from the llm-access-root-actor-secrets Kubernetes Secret). The health endpoint /-/health is unauthenticated.

Environment (from gitops base Deployment):

RBAC: the llm-access-root-actor ServiceAccount has a Role in the llm-access namespace granting get, create, update, delete on secrets — required to store/retrieve per-project LiteLLM virtual keys as Kubernetes Secrets.

Endpoints (all under /waldur/, all require Bearer token except /-/health):

Key storage: per-project LiteLLM virtual keys are stored as Kubernetes Secrets with name llm-access-openwebui-key-<resource_backend_id> in the llm-access namespace. The secret_store.KubernetesSecretStore class manages read/write/delete operations.

#waldur-llm-access-site-agent deployments

Three deployments share the same image (waldur-llm-access-site-agent) and the same config Secret (waldur-llm-access-site-agent-config, mounted at /etc/waldur/). They are differentiated only by the -m mode flag passed to waldur_site_agent:

The config Secret is rendered by the waldur-llm-access-offering Ansible role from a Jinja2 template. It includes the Waldur API URL, site-agent service token, offering UUID, root-actor URL and token, available_models list (derived from inference.yml), default budget_duration, and the LLM access backend component schema.

The waldur-llm-access-offering role restarts all three deployments automatically when the config Secret content changes (detected via register: waldur_llm_access_site_agent_config_secret_result).

#Membership sync data flow

Waldur project membership (source of truth)
  ↓  every 5 min
waldur-llm-access-agent-membership-sync
  (waldur_site_agent -m membership_sync)
  ↓  POST /waldur/add_project_members
  ↓  POST /waldur/remove_project_members
llm-access-root-actor
  ↓  GET /api/v1/users/ (resolve email → Open WebUI user ID)
Open WebUI group membership
  (group name == resource_backend_id)

#not_found retry

When a user has not yet logged in to Open WebUI, GET /api/v1/users/ returns no match. The root actor returns that email in the not_found list. The site agent returns only successfully-resolved emails to Waldur as "added"; Waldur retries add_users_to_resource on the next membership_sync cycle (every 5 minutes). No retry state is stored.

#LiteLLM enforcement is independent of Open WebUI membership

Removing a Waldur project membership removes the user from the Open WebUI group (blocking UI model access). The project-scoped LiteLLM team budget and virtual key remain active until the Waldur resource itself is terminated (which triggers project_delete via the order_process agent). This means LiteLLM API key revocation is a resource lifecycle event, not a membership event.

#Local Keycloak project_ids claim path (local dev only)

The local dev stack seeds Keycloak groups (project-a, project-b) and configures a custom project_ids OIDC claim for Open WebUI. This is a local-only bootstrap shim to test the OIDC login flow; it is not the production authorization channel. Group membership in production is set exclusively by the root actor via membership_sync.

#Ansible local seed role

devops/ansible/roles/waldur-llm-access-local-seed/ creates local Waldur fixtures (4 users, 2 projects, 1 offering, 4 offering-users, 2 resources) needed to exercise the full membership sync path in the local Minikube stack. It is idempotent and is called by the local-setup, local-up-full, and waldur-local-patches plays. The seed was verified live on themelio-local: second reconcile completes with changed=0, 4 active offering-users, membership shape correct (project-a/b members, owui-none excluded).