For AI agents: the complete documentation index is available at /llms.txt, the full documentation bundle is available at /llms-full.txt, and this page is available as Markdown at /adr/0011-human-web-ui-and-agent-client-access.md.

Themelio compatibility reference. This page describes the retained Themelio implementation or its historical design. Check the parallel-operation guide before applying it to the current platform.

ADR-0011: Chat UI and Client Access for Shared LLM Services

Status: Draft

Date: 2026-05-22

Deciders: Themelio developers (confirmed 2026-06-30)

Relates to:

Context

ADR-0009 introduces a shared LLM inference backend based on KServe LLMInferenceService and vLLM. ADR-0010 introduces LiteLLM as the governed gateway that maps Waldur projects to LiteLLM Teams, enforces spend budgets, and exposes an OpenAI-compatible API surface.

If Themelio offers only API access, the service works well for SDK clients and coding assistants but is incomplete for many human users. Users also expect a browser-based chat experience: sign in with SSO, use an allowed model, chat with the model, and consume the project budget. The client interaction layer must not bypass or duplicate LiteLLM's budget enforcement.

The non-negotiable architectural rule:

All user-facing clients — web UI, CLI clients, IDE extensions, SDKs, and agents — must call LiteLLM (ADR-0010). Direct access from user clients to vLLM/KServe model endpoints is not allowed.

This means the human web UI must also route through LiteLLM using a project-scoped virtual key. Budget enforcement is performed by LiteLLM regardless of the access path (UI or API).

Considered Options

Option A: LibreChat (considered, not adopted)

Deploy LibreChat behind Themelio SSO. LibreChat supports custom OpenAI-compatible endpoints and has stronger multi-provider and agent/MCP workflow features.

Cons:

  • Additional persistence layer (MongoDB) to operate.
  • No public API for automating per-project connection setup.
  • Agent/MCP features require additional security review before enabling.

Decision: Not adopted at MVP. Remains a candidate if agent/MCP workflows become a hard requirement.

Option B: Custom Themelio frontend (deferred)

Build a dedicated frontend for shared LLM access.

Pros: Full control over quota display, project selection, Themelio branding. Cons: Highest implementation cost; delays service availability.

Decision: Long-term option. Not adopted for MVP.

Option C: Open WebUI (adopted)

Deploy Open WebUI as the browser-based chat UI. Configure it to connect to LiteLLM using per-project virtual keys managed by llm-access-root-actor.

Pros:

  • Self-hosted, full OIDC SSO support without a user count cap.
  • Group management API enables automated per-project access control.
  • Single Helm chart, straightforward operational model.
  • No direct vLLM access — all requests go through LiteLLM.

Cons:

  • Open WebUI admin edits (groups, connections) are overwritten by root-actor reconciliation.
  • Per-project quota display is not surfaced natively in Open WebUI; visible via Waldur usage reports.

Decision: Accepted.

Option D: API-only access (rejected)

Expose only LiteLLM virtual keys with API documentation. No web UI.

Decision: Rejected as the production state. Remains valid as an interim milestone before Open WebUI is deployed.

Why Open WebUI over LibreChat and LiteLLM's built-in UI

CriterionLiteLLM built-in UILibreChatOpen WebUI (chosen)
OIDC SSO without user capNo — limited to 5 users in OSS tierYesYes
OpenAI-compatible backendNativeYesYes
Per-project model ACLsNot applicable (admin UI only)LimitedGroup-scoped model access grants
Admin API for automationNative (used by root actor)NoYes (/openai/config, /api/v1/groups/)
Operational simplicitySingle binaryAdditional persistence (MongoDB)Single Helm chart

LiteLLM's built-in Proxy UI enforces a 5-user limit for SSO/OIDC login in the open-source tier; API access (virtual keys) has no such restriction. Open WebUI avoids this limit, supports full OIDC SSO with no user cap, and exposes a group management API that llm-access-root-actor uses to automate per-project membership without manual admin intervention.

Decision

Themelio uses Open WebUI as the browser-based chat interface and LiteLLM as the unified enforcement and routing layer for all client access paths.

  • Open WebUI provides human-facing SSO chat with per-project model visibility and group-based access control. It connects to LiteLLM via project-scoped virtual keys — it does not call vLLM directly.
  • LiteLLM enforces budget and throughput limits for every request regardless of whether it originates from Open WebUI or a direct API client.
  • API clients (SDK, Codex CLI, Claude Code) call LiteLLM directly with a project virtual key.

See also: DD-0011: Implementation details

Architecture

Human user request path

  1. The user authenticates to Open WebUI via OIDC (Keycloak / AAI).
  2. Open WebUI resolves the user's group membership and shows project-prefixed models (e.g., projectid.model-name).
  3. The user selects a model; Open WebUI routes the request to LiteLLM using the per-project virtual key stored in that project's connection entry.
  4. LiteLLM checks the team's max_budget and optional tpm_limit/rpm_limit.
  5. If within limits, LiteLLM routes to the vLLM backend from ADR-0009.
  6. LiteLLM records spend and decrements the team budget.

API user request path

API users receive the per-project LiteLLM virtual key (via the Waldur offering / ADR-0010 key distribution). They call LiteLLM's OpenAI-compatible endpoint directly. The same budget and model allowlist checks apply.

Per-project connection provisioning

llm-access-root-actor reconciles both the LiteLLM side (team, key) and the Open WebUI side (connection entry, group, model ACLs) in a single idempotent call per Waldur project lifecycle event. See DD-0011 for the connection data model.

API Surface

LiteLLM's OpenAI-compatible endpoint covers both UI and API access paths:

  • GET /v1/models — project-filtered model list (filtered by team allowlist)
  • POST /v1/chat/completions — streaming and non-streaming
  • Tool/function calling where the backend model supports it
  • Standard OpenAI rate-limit headers
  • Anthropic models proxied through the same surface

For Codex CLI clients that require the Responses API (POST /v1/responses), a separate responses-proxy service translates to Chat Completions. This is outside the LiteLLM gateway layer and is not part of this ADR.

Consequences

Positive:

  • Human users get browser-based shared LLM access via OIDC SSO without installing any developer tools.
  • API clients, Open WebUI users, and coding tools all consume the same project budget through the same LiteLLM enforcement point.
  • Budget enforcement is centralized in LiteLLM regardless of access path; Open WebUI cannot bypass it.
  • Per-project model visibility is enforced through Open WebUI group ACLs (group name = resource_backend_id).

Negative:

  • Open WebUI group membership and connection state are externally managed; admin edits in the Open WebUI UI are overwritten on the next reconciliation cycle.
  • Project-level quota visibility is available in Waldur and LiteLLM spend logs, but not surfaced in the Open WebUI chat UI.
  • Chat history is stored in Open WebUI's SQLite/PVC; retention and deletion policy must be explicitly addressed before production.

Neutral:

  • File upload, terminals, and pipelines are disabled at deployment (ENABLE_DIRECT_CONNECTIONS: false).
  • Open WebUI connects to LiteLLM; it does not need direct access to vLLM or the inference namespace.

Open Questions

1. Should the web UI be embedded in Themelio or deployed as a separate service with Themelio branding? Resolved: Deployed as a separate service in its own open-webui namespace. No Themelio branding customization at MVP.

2. How should users choose the active Waldur project in the web UI? Resolved: Project selection is implicit via model naming. Each project's models are prefixed <resource_backend_id>.<model>. Users see only models accessible to their Open WebUI groups (one group per Waldur project they are a member of).

3. Should each project receive a shared API key hidden behind the UI, or should the UI call the gateway using the user's identity? Resolved: Shared per-project virtual key stored in a Kubernetes Secret and used by the Open WebUI connection entry. The key is scoped to the LiteLLM Team (project), not the individual user.

4. Should conversation history be per-user, per-project, both, or disabled for MVP? Resolved: Per-user in Open WebUI's local SQLite persistence (PVC-backed). No cross-project isolation of conversation history at MVP.

5. Can Open WebUI be configured to display quota without forking? Resolved: No native quota display in Open WebUI. Budget visibility is available via Waldur usage reports and LiteLLM admin spend logs. Not surfaced in the chat UI at MVP.

6. Should file upload and RAG features be enabled in MVP? Resolved: Disabled at deployment (ENABLE_DIRECT_CONNECTIONS: false, tika.enabled: false, terminals.enabled: false).

7. Which external clients are officially supported versus best-effort? Resolved: Officially supported: Open WebUI (human UI) + OpenAI SDK clients via LiteLLM API. Best-effort: Codex CLI (via responses-proxy), Claude Code (direct LiteLLM virtual key with ANTHROPIC_BASE_URL pointing to LiteLLM).

8. What model capabilities are required for coding assistant support? Resolved: Determined per model via inference.yml catalog fields (enable_auto_tool_choice, tool_call_parser, max_model_len).

9. How will Themelio document setup for Codex, Claude Code, and generic SDK clients? Resolved: Deferred to operations documentation. The mechanism is: set base URL to the LiteLLM external endpoint and use the project virtual key as the API key.

10. Should Themelio provide client-side helper scripts for quota status bars? Resolved: Not implemented at MVP. LiteLLM's standard rate-limit headers are returned on every response.

References

Page source

Source snapshot · Revision f928834c8.