For AI agents: the complete documentation index is available at /llms.txt, the full documentation bundle is available at /llms-full.txt, and this page is available as Markdown at /reference/docs/decisions/0010-project-scoped-llm-gateway-token-quotas-api-access.md.

ADR-0010: LLM Gateway and Project Token Quotas

Status: Draft

Date: 2026-05-22

Deciders: Platform developers (confirmed 2026-06-30)

Relates to:

Context

The platform already uses Waldur to manage project-level resource allocations such as CPU, RAM, GPU, and storage. These quotas are infrastructure quotas: a team receives capacity and consumes it inside its project environment.

The shared LLM backend introduced in ADR-0009 requires a different allocation model. Users should be able to request access to platform-operated shared LLMs during onboarding, and the platform should grant project-scoped token budgets rather than dedicated GPU resources.

The desired behavior is similar to common LLM platforms:

  • A project receives a shared spend budget.
  • All users in that project consume from the same budget.
  • Budgets reset on a configured duration (e.g., 30d).
  • Users and project owners can see what they have consumed.
  • The platform can enforce hard limits at request time.
  • API clients and web UI clients use the same governed access path.

This ADR decides the governed access layer: how authenticated users and projects are mapped to gateway credentials, how spend quotas are enforced, how usage is reported, and what model surfaces are exposed. The human-facing web UI and agent client UX are handled in ADR-0011.

Decision

The platform uses LiteLLM as the governed LLM gateway and quota-enforcement layer for shared LLM access. The human-facing chat interface is a separate decision, made in ADR-0011.

Waldur remains the source of truth for project entitlements. LiteLLM is the runtime enforcement point for spend/request/throughput quotas on the shared LLM API. Every client — human web UI or API/agent client — connects to LiteLLM via a per-project virtual key managed by the llm-access-root-actor service; how that key reaches human users is decided in ADR-0011.

The mapping is:

Platform/Waldur conceptLiteLLM concept
Waldur projectLiteLLM Team (team_id = resource backend ID)
Project spend entitlementmax_budget (USD) + budget_duration (e.g., 30d)
Optional throughput limittpm_limit (tokens/min), rpm_limit (requests/min)
Project model allowlistLiteLLM Team models list
Project service keyLiteLLM Virtual Key scoped to the team
Shared model catalogvLLM backend models registered in LiteLLM
Per-token cost configlitellm_model_info.input_cost_per_token / output_cost_per_token in inference.yml
Project usage dashboardWaldur usage report driven by LiteLLM spend logs

See also: DD-0010: Implementation details

Chosen Architecture

Runtime request path

  1. The client — human web UI or API/agent client, see ADR-0011 — presents a per-project LiteLLM virtual key to LiteLLM's OpenAI-compatible endpoint (POST /v1/chat/completions).
  2. LiteLLM checks the team's max_budget and optional tpm_limit/rpm_limit.
  3. If within limits, LiteLLM routes to the vLLM backend from ADR-0009.
  4. LiteLLM records spend from the response and decrements the team budget.

LiteLLM enforces the same team budget and model allowlist regardless of which client path (UI or API) presented the key.

Model catalog

The list of available models and their per-token cost configuration is defined in devops/ansible/inventory/<env>/group_vars/inference.yml. This file is the single source of truth consumed by both the inference-deploy role (which creates KServe LLMInferenceService resources) and the litellm-deploy role (which registers each served model in LiteLLM with its input_cost_per_token / output_cost_per_token). On-premises models served by the platform can be configured with zero cost, in which case spend accounting tracks token counts but the max_budget limit has no financial meaning. Per-environment overlays (dev/, staging/, prod/) provide environment- specific model selections and real cost figures for externally-hosted models.

Provisioning path

  1. User requests AI access through the Waldur/Platform allocation workflow (AI offering).
  2. Waldur approves the resource and calls the waldur-llm-access-site-agent.
  3. The site agent calls llm-access-root-actor which idempotently reconciles the LiteLLM Team (budget, model allowlist, metadata) and a LiteLLM Virtual Key scoped to the team. Client-facing provisioning (chat UI connection, group ACLs) is decided in ADR-0011.
  4. On resource termination, the root actor blocks the team and revokes the key.
  5. The site-agent and root-actor architecture that connects this stack to Waldur, including project membership sync, is being decided in ADR-0020.

Provider-Compatible API Surfaces

LiteLLM natively exposes an OpenAI-compatible endpoint, which covers the main access patterns:

  • GET /v1/models — project-filtered model list
  • POST /v1/chat/completions — streaming and non-streaming
  • Tool/function calling (when the backend model supports it)
  • Standard OpenAI rate-limit headers
  • Anthropic models are proxied by LiteLLM through the same OpenAI-compatible surface; no separate Anthropic facade is required at this layer.

This same OpenAI-compatible endpoint is what any human-facing chat interface must connect to; which interface to deploy and why is decided in ADR-0011.

Quota Enforcement and Visibility

Spend quotas are enforced at the LiteLLM Team level:

  • max_budget (float, USD): hard limit per budget_duration window.
  • budget_duration (string, e.g., 30d): rolling reset period.
  • tpm_limit / rpm_limit (optional): secondary throughput guards.

LiteLLM blocks new requests when current_spend + request_cost would exceed max_budget. Budget is tracked against spend (USD) rather than raw token counts; the conversion depends on the model pricing configured in LiteLLM.

Usage is reported to Waldur by the report_usage endpoint of the root actor, which queries LiteLLM spend logs per project and returns aggregated total_spend, total_requests, input_tokens, output_tokens, and total_tokens.

Note

Multi-window quotas (5h / 1w / 1M simultaneously) are not implemented in the current single-budget model. Supporting multiple concurrent windows would require either LiteLLM multi-budget support or a sidecar quota layer. This is a known limitation for MVP.

Considered Options

Option A: LiteLLM as the governed gateway (adopted)

LiteLLM is the governed gateway and quota-enforcement layer. The llm-access-root-actor provisions and reconciles Teams, budgets, and virtual keys via the waldur-llm-access-site-agent.

Pros:

  • LiteLLM already implements the full OpenAI-compatible surface, multi-provider routing, virtual keys, team budgets, and spend tracking — no custom extension required.
  • The root-actor pattern is already proven in this codebase (waldur-k8s-site-agent delegates GitLab membership to gitlab-root-actor); the same reconciliation model applies.
  • Idempotent provisioning from Waldur with no dependency on external Bifrost infra.

Cons:

  • Budget granularity is cost-based (USD), not raw-token-based. Token visibility requires querying LiteLLM spend logs.
  • Single budget window per team at MVP; multi-window quota requires future work.
  • Per-user spend attribution is visible in LiteLLM logs but not surfaced to end users in the current UI.

Option B: Bifrost with custom extension (considered, not adopted)

Use Bifrost as the LLM gateway. Bifrost Teams map to Waldur projects and its rate-limit hierarchy would enforce token quotas. The Bifrost schema would need extension to support simultaneous multi-window rate limits per team (mirroring the existing multi-budget pattern), since only one RateLimitID is currently supported per team.

A source inspection confirmed that Bifrost already evaluates multiple rate limits from the hierarchy if collected; the missing piece was schema-level multi-limit support per owner. However, the required Bifrost schema changes (add owner fields to TableRateLimit, update Team/VK API types, update collectRateLimitsFromHierarchy) represent a non-trivial upstream contribution or maintained fork.

Pros:

  • Designed from the ground up as a governed LLM gateway with token-level rate-limit semantics.
  • Multi-window enforcement would be native once the schema extension is merged.

Cons:

  • Requires maintaining a Bifrost fork or upstream contribution with uncertain timeline.
  • LiteLLM already delivers the required MVP functionality without modification.

Decision: Rejected in favor of Option A for the MVP. Option B remains viable if multi-window token quotas become a hard requirement that LiteLLM cannot satisfy.

Option C: Waldur-only accounting (rejected)

Waldur stores token allocations and the platform periodically reconciles usage. No runtime gateway enforcement.

Cons: No request-time enforcement; users can exceed limits until reconciliation catches up.

Option D: vLLM/KServe-level rate limiting (rejected)

Rely on model-serving runtime or Kubernetes-level controls.

Cons: Not project-aware by default; does not integrate with Waldur project entitlements.

Consequences

Positive:

  • Token/spend quotas are a first-class platform allocation type alongside CPU/RAM/GPU/storage.
  • Shared LLM access is governed at the project level, matching Waldur's project model.
  • Project members share a common budget pool, matching research team access patterns.
  • No custom gateway code: LiteLLM handles provider routing, key enforcement, and spend tracking.
  • Usage reports flow from LiteLLM spend logs → root actor → Waldur, closing the accounting loop.

Negative:

  • Budget granularity is cost-based (USD), not token-based. Token counts are visible in logs but the enforcement unit is spend.
  • Only one budget window per team at MVP; multi-window (5h/1w/1M) is not yet implemented.
  • LiteLLM's spend accounting is per-process in-memory first, persisted to its database. Hard enforcement across multiple LiteLLM replicas requires careful counter semantics (same constraint as was identified for Bifrost in the source inspection).

Neutral:

  • LiteLLM exposes an OpenAI-compatible endpoint; no separate Anthropic facade is required at this layer since LiteLLM proxies Anthropic models through the same surface.
  • The llm-access-root-actor follows the same privileged-actor pattern already used by gitlab-root-actor (called by waldur-k8s-site-agent) and k8s-root-actor (called directly by services/web).
  • Waldur remains the source of truth for entitlements and budgets; LiteLLM state is always reconcilable from Waldur. Project membership is a separate concern, covered by ADR-0020.

Migration

No migration of existing state is required. The provisioning is idempotent and will reconcile from whatever Waldur project state exists at first run.

Open Questions

1. Quota source of truth: Should token quotas be native Waldur offering attributes, marketplace plan components, or Themelio-specific metadata? Resolved: Waldur offering attributes. max_budget and budget_duration are set as offering component plan attributes and passed through the site-agent to the root actor at resource creation/update time.

2. Reservation model: Should the gateway reserve a maximum token estimate before the request or only block when current usage is already exhausted? Resolved: Post-request accounting. LiteLLM records actual spend after the response. Requests that would exceed the budget are blocked at admission using the last-known spend value. No pre-request reservation is implemented at MVP.

3. Prompt vs. completion tokens: Are limits based on total tokens only, or should prompt/completion be visible separately? Resolved: Both are tracked in LiteLLM spend logs (prompt_tokens, completion_tokens, total_tokens) and reported to Waldur by the root actor. Enforcement is cost-based (USD), not token-count-based.

4. Per-user attribution: Should project members have individual soft limits or only usage visibility? Resolved: Project-level only at MVP. LiteLLM logs tag the virtual key (project-scoped) but not the individual end user. Per-user attribution can be derived from LiteLLM logs if needed later.

5. Model-specific quotas: Should limits be global per project, per model class, or weighted by model cost? Resolved: Global per-project budget (cost-weighted by LiteLLM model pricing). Model-specific budgets are not configured at MVP.

6. Billing semantics: Are tokens an allocation-only quota, a future chargeback unit, or both? Resolved: Allocation-only quota at MVP. Spend data is reported to Waldur for visibility but not used for chargeback.

7. Error format: What error body should clients receive when quota is exceeded? Resolved: LiteLLM's standard HTTP 429 response body is returned as-is. No custom Themelio wrapping at MVP.

8. Upstream strategy: Should Themelio contribute multi-rate-limit support upstream to Bifrost, maintain a fork, or wrap with a sidecar? Resolved: Neither — LiteLLM was selected instead of Bifrost. The Bifrost extension strategy is no longer on the critical path.

9. Privacy: What usage metadata can be shown to project owners without exposing other users' prompts? Resolved: Only aggregated spend, request count, and token totals are reported to Waldur. Prompt and completion content remains in LiteLLM logs only.

10. Calendar alignment: Should weekly/monthly windows reset at calendar boundaries or use rolling windows from allocation time? Resolved: Rolling window from the last reset, driven by LiteLLM's budget_duration string (e.g., 30d). Calendar-aligned resets are not implemented at MVP.

11. Responses API: Is /v1/responses required for the Codex clients Themelio wants to support, or is Chat Completions enough for MVP? Resolved: A separate responses-proxy service translates /v1/responses to Chat Completions for Codex CLI clients. This is outside the scope of the LLM gateway layer and is handled at the client integration layer.

12. Anthropic facade: Does LiteLLM natively cover the required Anthropic Messages compatibility, or do we need a Themelio adapter? Resolved: LiteLLM proxies Anthropic models through its standard OpenAI-compatible surface. No separate Anthropic facade is required at the gateway layer.

References

Page source

Source snapshot · Revision f928834c8.