> For AI agents: the complete documentation index is available at /llms.txt, the full documentation bundle is available at /llms-full.txt. > **Themelio compatibility reference.** This page describes the retained Themelio implementation or its historical design. Check the [parallel-operation guide](/compatibility/index.md) before applying it to the current platform. # DD-0009: Shared LLM Inference Backend — Implementation **Related ADR:** [ADR-0009: Shared LLM Inference Backend](/adr/0009-shared-llm-inference-backend.md) **Status:** Draft **Date:** 2026-06-30 **Authors:** TBD ## ADR-derived technical notes ### Implementation branches The implementation described in this DD lives in the following stacked `local/*` branches, none of which have been merged to `main` yet: | Branch | Key content | | ------------------------------ | ---------------------------------------------------------------------------------------------------------------------------- | | `local/litellm-gateway` | LiteLLM gateway, `litellm-deploy` role, initial `inference.yml` catalog | | `local/inference-multinode` | LWS multi-node vLLM path in `inference-deploy` role | | `local/kserve-llmisvc` | Migration to `LLMInferenceService` (v1alpha2), removal of legacy InferenceService path | | `local/llm-access-openwebui` | `waldur-llm-access-offering` role wires `inference.yml` served names into the Waldur AI offering | | `local/llm-access-waldur-seed` | Local Waldur seed fixtures; budget-enforcement e2e test fixture; per-resource `tpm_limit`/`rpm_limit` fix | | `local/local-test-identities` | Generalizes local Keycloak test-identity seeding (`local-test-identities` role) beyond LLM access (tip of the stacked chain) | :::warning All branches have been tested on a local Minikube cluster in **CPU mode only** (`gpu_per_node: 0`, `vllm/vllm-openai-cpu` image). GPU profiles (Profile A, B, C) and multi-node LWS setups require validation on a cluster with NVIDIA GPUs before being used in staging or production. ::: ### `inference.yml` — single source of truth for three consumers `devops/ansible/inventory//group_vars/inference.yml` (and its per-environment overlays) is consumed by **three Ansible roles**: | Role | What it reads | What it produces | | ---------------------------- | -------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------- | | `inference-deploy` | `inference_models` (name, hf_repo, topology, extra_args …) | KServe `LLMInferenceService` + PVC + alias Service per model | | `litellm-deploy` | `inference_models` (served_name, litellm_model_info, litellm_max_tokens …) | LiteLLM model registration with cost/token metadata | | `waldur-llm-access-offering` | `inference_models \| map(attribute='served_name') \| list` | `available_models` field in the Waldur AI offering site-agent config — controls which model IDs users may request | `local-test-identities` (formerly `llm-access-local-bootstrap`) does not read `inference_models` at all: it only seeds Keycloak groups/users for local SSO/OIDC testing (`llm_access_projects`/`themelio_local_test_users` in `all.yml`), generically — not specific to LLM access. Project-to-model wiring (LiteLLM teams, project API keys, Open WebUI connections) is owned exclusively by the Waldur-driven path (`waldur-llm-access-local-seed` → site agent → `llm-access-root-actor`) so there is a single source of truth for it instead of two roles racing to write the same LiteLLM team and k8s secret. The consequence is: **adding or removing a model from `inference.yml` must be followed by re-running the three dependent make targets** (`inference-deploy`, `litellm-deploy`, and `waldur-llm-access-offering`). The `waldur-llm-access-offering` role updates the site-agent config secret and restarts the three `waldur-llm-access-agent-*` deployments automatically when the secret changes. ### KServe LLMInferenceService (not generic InferenceService) The implementation uses the `LLMInferenceService` CRD (`serving.kserve.io/v1alpha2`), a dedicated KServe API for LLM workloads, distinct from the older `InferenceService` (v1beta1). Installation requires two extra Kustomize components beyond the base KServe controller: - `kserve-llmisvc-crd` — installs the CRD - `kserve-llmisvc-resources` — installs the llmisvc controller manager These are applied via `make kserve` before `make inference-deploy`. ### Stable alias Service for LiteLLM routing The `llmisvc` controller already creates its own Service for the `LLMInferenceService`, so this is an intentional second Service, not a replacement for a missing one. The `inference-deploy` role additionally creates a stable Kubernetes Service `-openai` in the `inference` namespace alongside each LLMInferenceService, with a selector the role controls explicitly: `app.kubernetes.io/name: ` plus `kserve.io/component: workload` (the label the llmisvc controller sets on data-path pods only, excluding any prefill/decode split pods that may appear in multi-role setups). For multi-node (LWS) models, the selector additionally pins `leaderworkerset.sigs.k8s.io/worker-index: "0"` so the Service only ever targets the **leader** pod's HTTP server on port 8000 — worker pods run vLLM in `--headless` mode and do not serve HTTP, so routing to one by accident would break the request. LiteLLM routes to `-openai.inference.svc.cluster.local:8000`. Decoupling this explicitly-selected alias Service from whatever Service the llmisvc controller manages internally means LiteLLM's routing target is defined once, by this role, and does not depend on how the controller names or reshapes its own Service across versions or reconciliation cycles. ### Multi-node LWS topology (Profile C) When `node_count > 1`, `llmisvc.yml.j2` renders a `worker:` section in the LLMInferenceService spec. The LWS controller creates a LeaderWorkerSet: - **Leader pod** (from `template:`): full vLLM HTTP server on port 8000; receives `--nnodes=$(LWS_GROUP_SIZE)`, `--node-rank=0`, `--master-addr=$(LWS_LEADER_ADDRESS)`, `--pipeline-parallel-size=$(LWS_GROUP_SIZE)` from LWS environment variables - **Worker pods** (from `worker:`): same vLLM command plus `--headless` — pipeline parallel stage, no HTTP server, same resource requests/limits as the leader Role validation: `tensor_parallel_size × pipeline_parallel_size = node_count × gpu_per_node` fails fast at Ansible task level before any Kubernetes resource is applied. The LWS operator (v0.6.0) is installed only when any catalog entry has `node_count > 1`. ### Model download and caching A git-lfs download Job populates the model PVC on first deploy. A ConfigMap marker (`-download-done`) prevents re-download on subsequent reconciliations. PVCs use `ReadWriteOnce`; model files are mounted read-only by leader and worker pods. ### CPU mode (local dev only) `gpu_per_node: 0` removes `nvidia.com/gpu` resource requests/limits and uses the CPU-specific image from `inference.yml` (`docker.io/vllm/vllm-openai-cpu:v0.23.0-x86_64` in the local overlay). This is a control-plane smoke path only, not a production profile.
Page source Source snapshot · Revision `f928834c8`.