Themelio compatibility reference. This page describes the retained Themelio implementation or its historical design. Check the parallel-operation guide before applying it to the current platform.

#DD-0010: LLM Gateway and Project Token Quotas — Implementation

Related ADR: ADR-0010: LLM Gateway and Project Token Quotas

Status: Draft

Date: 2026-06-30

Authors: TBD

#Implementation branches

The implementation described in this DD lives in the following stacked local/* branches, none of which have been merged to main yet:

The branches are stacked on top of each other in the order above; all diverge from origin/main by approximately 110 commits total across the full LLM access stack.

#LiteLLM deployment

LiteLLM is deployed by the litellm-deploy Ansible role into the litellm namespace. The role renders two Kubernetes resources from Jinja2 templates and applies them directly (kubectl apply); there is no Helm chart.

Image: ghcr.io/berriai/litellm:main-stable

Resources:

#config.yaml structure

general_settings:
  master_key: "<vault_litellm_master_key>"
  database_url: "postgresql://litellm:<password>@pg-rw.databases.svc.cluster.local:5432/litellm"
  store_model_in_db: true
  default_internal_user_params:
    user_role: "internal_user"   # SSO users can chat; avoids internal_user_viewer lockout

litellm_settings:
  drop_params: true     # silently drop unsupported model params
  num_retries: 0
  request_timeout: 900  # 15 min for long inference

model_list:
  - model_name: "<served_name>"
    litellm_params:
      model: "hosted_vllm/<served_name>"
      api_base: "http://<model-name>-openai.inference.svc.cluster.local:8000/v1"
      api_key: "dummy"
    model_info:
      mode: "chat"
      input_cost_per_token: 0.0
      output_cost_per_token: 0.0

store_model_in_db: true means LiteLLM persists teams, keys, and spend logs to the shared CNPG PostgreSQL cluster (pg-rw.databases.svc.cluster.local). The database is provisioned by the postgres Ansible role; the litellm user and database are added via the declarative CNPG managed.roles patch.

#Model list from inference.yml

The model_list section is generated from the inference_models catalog (devops/ansible/inventory/<env>/group_vars/inference.yml). Each catalog entry maps to:

The api_base points to the stable alias Service created by inference-deploy (see DD-0009). This decouples the LiteLLM config from the LLMInferenceService pod lifecycle; LiteLLM can start before the model is ready without failing.

After any catalog change both make inference and make litellm must be re-run: make inference applies new LLMInferenceService resources; make litellm regenerates and applies the ConfigMap and restarts the LiteLLM pod.

#LiteLLM SSO (admin UI)

LiteLLM's admin Proxy UI supports OIDC login via a generic OIDC provider. Configured by the litellm-keycloak branch using the same shared Keycloak client as other services:

#Project provisioning (via llm-access-root-actor)

All LiteLLM team and key state is managed by llm-access-root-actor via the LiteLLM admin API (authenticated with the master key). Waldur lifecycle events are translated to LiteLLM API calls by the order_process site-agent deployment.

#LiteLLM Team

One Team per Waldur project resource:

Teams are created idempotently: the root actor reads the current team state via GET /team/info, compares against desired, and calls POST /team/update only if a field differs. On resource termination, the team is blocked (blocked: true) rather than deleted, preserving spend history.

#LiteLLM Virtual Key

One virtual key per project, stored as a Kubernetes Secret:

The key is created via POST /key/generate. Rotation is triggered by POST /waldur/rotate_project_key on the root actor, which generates a new key and deletes the old one. The Open WebUI connection for the project is updated with the new key in the same call.

#Spend reporting

The report site-agent deployment calls POST /waldur/report_usage on the root actor periodically. The root actor queries GET /spend/logs on LiteLLM for the project team and aggregates:

The report is posted to Waldur's usage reporting endpoint.

#Budget enforcement mechanics

LiteLLM enforces budget at request admission time using the last-known spend value in its PostgreSQL database. The check is:

current_spend + estimated_request_cost <= max_budget

If the budget is exceeded, LiteLLM returns HTTP 429. This is a soft check: spend is recorded after the response, so there is a small window for overage if multiple requests arrive simultaneously before the spend is persisted. This is acceptable for MVP.

Cost accounting uses per-model input_cost_per_token / output_cost_per_token from the model_info config. On-premises models with zero cost configured still track token counts but never trigger the budget limit (spend stays at 0).