For AI agents: the complete documentation index is available at /llms.txt, the full documentation bundle is available at /llms-full.txt, and this page is available as Markdown at /dd/0009-shared-llm-inference-backend.md.

Themelio compatibility reference. This page describes the retained Themelio implementation or its historical design. Check the parallel-operation guide before applying it to the current platform.

DD-0009: Shared LLM Inference Backend — Implementation

Related ADR: ADR-0009: Shared LLM Inference Backend

Status: Draft

Date: 2026-06-30

Authors: TBD

ADR-derived technical notes

Implementation branches

The implementation described in this DD lives in the following stacked local/* branches, none of which have been merged to main yet:

BranchKey content
local/litellm-gatewayLiteLLM gateway, litellm-deploy role, initial inference.yml catalog
local/inference-multinodeLWS multi-node vLLM path in inference-deploy role
local/kserve-llmisvcMigration to LLMInferenceService (v1alpha2), removal of legacy InferenceService path
local/llm-access-openwebuiwaldur-llm-access-offering role wires inference.yml served names into the Waldur AI offering
local/llm-access-waldur-seedLocal Waldur seed fixtures; budget-enforcement e2e test fixture; per-resource tpm_limit/rpm_limit fix
local/local-test-identitiesGeneralizes local Keycloak test-identity seeding (local-test-identities role) beyond LLM access (tip of the stacked chain)
Warning

All branches have been tested on a local Minikube cluster in CPU mode only (gpu_per_node: 0, vllm/vllm-openai-cpu image). GPU profiles (Profile A, B, C) and multi-node LWS setups require validation on a cluster with NVIDIA GPUs before being used in staging or production.

inference.yml — single source of truth for three consumers

devops/ansible/inventory/<env>/group_vars/inference.yml (and its per-environment overlays) is consumed by three Ansible roles:

RoleWhat it readsWhat it produces
inference-deployinference_models (name, hf_repo, topology, extra_args …)KServe LLMInferenceService + PVC + alias Service per model
litellm-deployinference_models (served_name, litellm_model_info, litellm_max_tokens …)LiteLLM model registration with cost/token metadata
waldur-llm-access-offeringinference_models | map(attribute='served_name') | listavailable_models field in the Waldur AI offering site-agent config — controls which model IDs users may request

local-test-identities (formerly llm-access-local-bootstrap) does not read inference_models at all: it only seeds Keycloak groups/users for local SSO/OIDC testing (llm_access_projects/themelio_local_test_users in all.yml), generically — not specific to LLM access. Project-to-model wiring (LiteLLM teams, project API keys, Open WebUI connections) is owned exclusively by the Waldur-driven path (waldur-llm-access-local-seed → site agent → llm-access-root-actor) so there is a single source of truth for it instead of two roles racing to write the same LiteLLM team and k8s secret.

The consequence is: adding or removing a model from inference.yml must be followed by re-running the three dependent make targets (inference-deploy, litellm-deploy, and waldur-llm-access-offering). The waldur-llm-access-offering role updates the site-agent config secret and restarts the three waldur-llm-access-agent-* deployments automatically when the secret changes.

KServe LLMInferenceService (not generic InferenceService)

The implementation uses the LLMInferenceService CRD (serving.kserve.io/v1alpha2), a dedicated KServe API for LLM workloads, distinct from the older InferenceService (v1beta1). Installation requires two extra Kustomize components beyond the base KServe controller:

  • kserve-llmisvc-crd — installs the CRD
  • kserve-llmisvc-resources — installs the llmisvc controller manager

These are applied via make kserve before make inference-deploy.

Stable alias Service for LiteLLM routing

The llmisvc controller already creates its own Service for the LLMInferenceService, so this is an intentional second Service, not a replacement for a missing one. The inference-deploy role additionally creates a stable Kubernetes Service <model-name>-openai in the inference namespace alongside each LLMInferenceService, with a selector the role controls explicitly: app.kubernetes.io/name: <model-name> plus kserve.io/component: workload (the label the llmisvc controller sets on data-path pods only, excluding any prefill/decode split pods that may appear in multi-role setups). For multi-node (LWS) models, the selector additionally pins leaderworkerset.sigs.k8s.io/worker-index: "0" so the Service only ever targets the leader pod's HTTP server on port 8000 — worker pods run vLLM in --headless mode and do not serve HTTP, so routing to one by accident would break the request. LiteLLM routes to <model-name>-openai.inference.svc.cluster.local:8000. Decoupling this explicitly-selected alias Service from whatever Service the llmisvc controller manages internally means LiteLLM's routing target is defined once, by this role, and does not depend on how the controller names or reshapes its own Service across versions or reconciliation cycles.

Multi-node LWS topology (Profile C)

When node_count > 1, llmisvc.yml.j2 renders a worker: section in the LLMInferenceService spec. The LWS controller creates a LeaderWorkerSet:

  • Leader pod (from template:): full vLLM HTTP server on port 8000; receives --nnodes=$(LWS_GROUP_SIZE), --node-rank=0, --master-addr=$(LWS_LEADER_ADDRESS), --pipeline-parallel-size=$(LWS_GROUP_SIZE) from LWS environment variables
  • Worker pods (from worker:): same vLLM command plus --headless — pipeline parallel stage, no HTTP server, same resource requests/limits as the leader

Role validation: tensor_parallel_size × pipeline_parallel_size = node_count × gpu_per_node fails fast at Ansible task level before any Kubernetes resource is applied.

The LWS operator (v0.6.0) is installed only when any catalog entry has node_count > 1.

Model download and caching

A git-lfs download Job populates the model PVC on first deploy. A ConfigMap marker (<model-name>-download-done) prevents re-download on subsequent reconciliations. PVCs use ReadWriteOnce; model files are mounted read-only by leader and worker pods.

CPU mode (local dev only)

gpu_per_node: 0 removes nvidia.com/gpu resource requests/limits and uses the CPU-specific image from inference.yml (docker.io/vllm/vllm-openai-cpu:v0.23.0-x86_64 in the local overlay). This is a control-plane smoke path only, not a production profile.

Page source

Source snapshot · Revision f928834c8.