---
title: "DD-0010: LLM Gateway and Project Token Quotas — Implementation"
---

# DD-0010: LLM Gateway and Project Token Quotas — Implementation

**Related ADR:** [ADR-0010: LLM Gateway and Project Token Quotas](/adr/0010-project-scoped-llm-gateway-token-quotas-api-access)

**Status:** Draft

**Date:** 2026-06-30

**Authors:** TBD

## Implementation branches

The implementation described in this DD lives in the following stacked `local/*` branches,
none of which have been merged to `main` yet:

| Branch | Key content |
|---|---|
| `local/litellm-gateway` | `litellm-deploy` Ansible role; LiteLLM ConfigMap + Deployment; model registration from `inference.yml` |
| `local/litellm-keycloak` | LiteLLM admin UI SSO via Keycloak (generic OIDC) |
| `local/llm-access-openwebui` | `llm-access-root-actor`: LiteLLM team/key provisioning endpoints; Open WebUI connection management |
| `local/llm-access-waldur-seed` | Waldur AI offering seed fixtures; local stack integration |

The branches are stacked on top of each other in the order above; all diverge from `origin/main`
by approximately 110 commits total across the full LLM access stack.

:::warning
All branches have been tested on a local Minikube cluster in **CPU mode only** (`gpu_per_node: 0`,
`vllm/vllm-openai-cpu` image). GPU enforcement, multi-replica LiteLLM, and production cost-based
budget accounting require validation on a cluster with NVIDIA GPUs before staging/production use.
:::

## LiteLLM deployment

LiteLLM is deployed by the `litellm-deploy` Ansible role into the `litellm` namespace.
The role renders two Kubernetes resources from Jinja2 templates and applies them directly
(`kubectl apply`); there is no Helm chart.

| Resource | Name | Purpose |
|---|---|---|
| ConfigMap | `litellm-config` | LiteLLM `config.yaml` (model list, DB URL, master key, settings) |
| Deployment | `litellm` | Single replica; mounts ConfigMap as `/app/config.yaml`; port 4000 |
| Service | `litellm` | ClusterIP, port 4000; exposed via Istio for external access |

**Image:** `ghcr.io/berriai/litellm:main-stable`

**Resources:**

| | CPU | Memory |
|---|---|---|
| Request | 200 m | 512 Mi |
| Limit | 1 000 m | 2 Gi |

### `config.yaml` structure

```yaml
general_settings:
  master_key: "<vault_litellm_master_key>"
  database_url: "postgresql://litellm:<password>@pg-rw.databases.svc.cluster.local:5432/litellm"
  store_model_in_db: true
  default_internal_user_params:
    user_role: "internal_user"   # SSO users can chat; avoids internal_user_viewer lockout

litellm_settings:
  drop_params: true     # silently drop unsupported model params
  num_retries: 0
  request_timeout: 900  # 15 min for long inference

model_list:
  - model_name: "<served_name>"
    litellm_params:
      model: "hosted_vllm/<served_name>"
      api_base: "http://<model-name>-openai.inference.svc.cluster.local:8000/v1"
      api_key: "dummy"
    model_info:
      mode: "chat"
      input_cost_per_token: 0.0
      output_cost_per_token: 0.0
```

`store_model_in_db: true` means LiteLLM persists teams, keys, and spend logs to the
shared CNPG PostgreSQL cluster (`pg-rw.databases.svc.cluster.local`). The database is
provisioned by the `postgres` Ansible role; the `litellm` user and database are added via
the declarative CNPG `managed.roles` patch.

### Model list from `inference.yml`

The `model_list` section is generated from the `inference_models` catalog
(`devops/ansible/inventory/<env>/group_vars/inference.yml`). Each catalog entry maps to:

| `inference.yml` field | `config.yaml` field | Notes |
|---|---|---|
| `m.served_name` | `model_name` and `hosted_vllm/<served_name>` | Client-visible model ID |
| `m.name` | `api_base` hostname prefix | `<m.name>-openai.<namespace>.svc.cluster.local:8000/v1` |
| `m.litellm_mode` | `model_info.mode` | Default: `chat` |
| `m.litellm_model_info.*` | `model_info.*` | Includes `input_cost_per_token`, `output_cost_per_token` |
| `m.litellm_max_tokens` | `litellm_params.max_tokens` | Optional hard cap |

The `api_base` points to the stable alias Service created by `inference-deploy`
(see [DD-0009](/dd/0009-shared-llm-inference-backend)). This decouples the LiteLLM
config from the `LLMInferenceService` pod lifecycle; LiteLLM can start before the
model is ready without failing.

After any catalog change both `make inference` and `make litellm` must be re-run:
`make inference` applies new `LLMInferenceService` resources; `make litellm` regenerates
and applies the ConfigMap and restarts the LiteLLM pod.

### LiteLLM SSO (admin UI)

LiteLLM's admin Proxy UI supports OIDC login via a generic OIDC provider. Configured
by the `litellm-keycloak` branch using the same shared Keycloak client as other services:

| Variable | Value |
|---|---|
| `GENERIC_CLIENT_ID` | shared OAuth2 client ID |
| `GENERIC_CLIENT_SECRET` | from vault |
| `GENERIC_AUTHORIZATION_ENDPOINT` | `<issuer>/protocol/openid-connect/auth` |
| `GENERIC_TOKEN_ENDPOINT` | internal: `http://keycloak.<ns>.svc.cluster.local/realms/<realm>/...` (local) or issuer URL (prod) |
| `PROXY_BASE_URL` | `https://<litellm_host>` (for correct redirect URI) |

:::note
LiteLLM's built-in Proxy UI SSO is limited to **5 users** in the open-source tier.
This is why Open WebUI (ADR-0011) is the human-facing interface for end users; LiteLLM
admin UI is reserved for platform operators.
:::

## Project provisioning (via `llm-access-root-actor`)

All LiteLLM team and key state is managed by `llm-access-root-actor` via the LiteLLM
admin API (authenticated with the master key). Waldur lifecycle events are translated
to LiteLLM API calls by the `order_process` site-agent deployment.

### LiteLLM Team

One Team per Waldur project resource:

| LiteLLM field | Source | Example |
|---|---|---|
| `team_id` | `resource.backend_id` (Waldur) | `"buffermusic"` |
| `team_alias` | `project.name` | `"Buffer Music Project"` |
| `models` | `resource.available_models` (from offering) | `["qwen2.5-0.5b"]` |
| `max_budget` | `resource.max_budget` (offering attribute) | `100.0` |
| `budget_duration` | `resource.budget_duration` (offering attribute) | `"30d"` |
| `tpm_limit` | `resource.tpm_limit` (optional) | `null` |
| `rpm_limit` | `resource.rpm_limit` (optional) | `null` |
| `metadata.waldur_project_uuid` | Waldur project UUID | for traceability |
| `metadata.waldur_resource_uuid` | Waldur resource UUID | for traceability |

Teams are created idempotently: the root actor reads the current team state via
`GET /team/info`, compares against desired, and calls `POST /team/update` only if a
field differs. On resource termination, the team is blocked (`blocked: true`) rather
than deleted, preserving spend history.

### LiteLLM Virtual Key

One virtual key per project, stored as a Kubernetes Secret:

| Attribute | Value |
|---|---|
| K8s Secret name | `llm-access-openwebui-key-<resource_backend_id>` in `llm-access` namespace |
| `key_alias` | `"<resource_backend_id>-key"` |
| LiteLLM `team_id` | `resource_backend_id` (same as Team above) |
| Models scope | inherits from Team |

The key is created via `POST /key/generate`. Rotation is triggered by
`POST /waldur/rotate_project_key` on the root actor, which generates a new key and
deletes the old one. The Open WebUI connection for the project is updated with the new
key in the same call.

### Spend reporting

The `report` site-agent deployment calls `POST /waldur/report_usage` on the root actor
periodically. The root actor queries `GET /spend/logs` on LiteLLM for the project team
and aggregates:

| Waldur field | LiteLLM source |
|---|---|
| `total_spend` | sum of `spend` over log entries for `team_id` |
| `total_requests` | count of log entries |
| `input_tokens` | sum of `prompt_tokens` |
| `output_tokens` | sum of `completion_tokens` |
| `total_tokens` | sum of `total_tokens` |

The report is posted to Waldur's usage reporting endpoint.

## Budget enforcement mechanics

LiteLLM enforces budget at request admission time using the last-known spend value
in its PostgreSQL database. The check is:

```
current_spend + estimated_request_cost <= max_budget
```

If the budget is exceeded, LiteLLM returns HTTP 429. This is a soft check: spend is
recorded after the response, so there is a small window for overage if multiple requests
arrive simultaneously before the spend is persisted. This is acceptable for MVP.

Cost accounting uses per-model `input_cost_per_token` / `output_cost_per_token` from
the `model_info` config. On-premises models with zero cost configured still track token
counts but never trigger the budget limit (spend stays at 0).

<!-- Additional sections (multi-replica semantics, budget reset mechanics, key rotation sequence diagram) are intentionally deferred to a later session. -->