---
title: "ADR-0009: Shared LLM Inference Backend"
---

# ADR-0009: Shared LLM Inference Backend

**Status:** Draft

**Date:** 2026-05-22

**Deciders:** Platform developers (confirmed 2026-06-30)

**Relates to:**
- [ADR-0000 (Friends & Family MVP)](/adr/0000-mvp-friends-and-family)
- [ADR-0003 (Waldur Proposal Workflow)](/adr/0003-waldur-proposal-workflow)
- [ADR-0005 (AAI → Waldur User Synchronization)](/adr/0005-aai-waldur-user-sync)
- [ADR-0010 (LLM Gateway and Project Token Quotas)](/adr/0010-project-scoped-llm-gateway-token-quotas-api-access)
- [ADR-0011 (Chat UI and Client Access for Shared LLM Services)](/adr/0011-human-web-ui-and-agent-client-access)

## Context

The platform currently provisions project-owned infrastructure quotas through Waldur. Teams receive CPU, RAM, GPU, and storage allocations and use them inside their project environment. The current inference implementation is still close to this infrastructure model: a simple Kubernetes inference pod runs on a GPU node, and the NVIDIA Kubernetes device plugin exposes GPU resources to workloads.

The platform also needs a second access mode for popular open-source LLMs: **shared managed inference**. In this mode, users do not receive dedicated GPU infrastructure. Instead, the platform operates a shared model-serving pool and users consume project-scoped LLM service capacity through the governed access layer described in [ADR-0010](/adr/0010-project-scoped-llm-gateway-token-quotas-api-access).

This ADR is intentionally limited to the **backend inference plane**: where model servers run, how GPUs are allocated, how large models are sharded across GPUs/nodes, how model-serving topologies are represented, and how the backend is exposed internally to the governed LLM gateway. It does not decide user-facing API token management, billing/quota semantics, or web UI behavior. Those are covered by [ADR-0010](/adr/0010-project-scoped-llm-gateway-token-quotas-api-access) and [ADR-0011](/adr/0011-human-web-ui-and-agent-client-access).

The intended backend uses KServe as the Kubernetes model-serving abstraction and vLLM as the primary generative inference runtime for open-source LLMs. vLLM is treated as an internal model endpoint, not a public user-facing API. Public access always goes through the governed gateway.

This ADR must also account for models that do **not** fit on a single GPU. vLLM supports distributed inference through tensor parallelism (`tensor_parallel_size`) and pipeline parallelism (`pipeline_parallel_size`). For multi-node serving, the LeaderWorkerSet (LWS) operator coordinates a leader pod (full vLLM HTTP server) with one or more worker pods (headless vLLM pipeline stages).

**See also:** [DD-0009: Implementation details](/dd/0009-shared-llm-inference-backend)

## Decision

The platform operates a **shared LLM inference backend** using:

- **KServe `LLMInferenceService`** (CRD `serving.kserve.io/v1alpha2`) as the Kubernetes model-serving abstraction.
- **vLLM** (`vllm/vllm-openai`) as the inference runtime for generative open-source LLMs.
- **LeaderWorkerSet (LWS) operator** for multi-node inference, triggered by the `worker:` spec in `LLMInferenceService`.
- **Internal-only vLLM endpoints** reachable only by the governed LLM gateway (ADR-0010), not directly by end users.
- **Explicit model serving profiles** for single-GPU, single-node multi-GPU, and multi-node multi-GPU deployments.
- **A stable headless alias Service** (`<model-name>-openai`) created by Ansible, used by the governed gateway for routing (ADR-0010).

The model catalog is defined in `devops/ansible/inventory/<env>/group_vars/inference.yml` and is shared with the governed gateway layer: `inference-deploy` (this ADR's role, creating KServe resources) reads the same file as the gateway's model-registration role (ADR-0010), and `waldur-llm-access-offering` (available model names for the Waldur AI offering) also reads it. `local-test-identities` (a generic local SSO/OIDC test-identity seeder, not LLM-specific) only seeds local Keycloak fixtures and does not read the model catalog; project-to-model wiring is owned exclusively by the Waldur-driven path (see DD-0009). Users never interact with the backend directly; all traffic goes through the governed gateway (ADR-0010).

```mermaid
flowchart LR
    U[User / Web UI / Agent client] --> GW[Governed LLM gateway]
    GW -->|internal alias Service| KS[KServe LLMInferenceService]
    KS --> V[vLLM leader pod]
    V -.->|LWS pipeline\nmultiprocess| W[vLLM worker pods\n--headless]
    V --> G[Shared GPU pool]
    W --> G

    MC[inference.yml\nmodel catalog] --> KS
    MC --> GW
```

## Model Serving Profiles

Every model in the catalog has a **serving profile** defining runtime topology, resource class, and operational constraints. The profile is set by the `node_count` and `gpu_per_node` fields in the catalog entry. Each entry also declares per-token USD cost fields consumed by the governed gateway for spend accounting and project budget enforcement (ADR-0010). On-premises models can be set to zero cost, in which case spend is tracked but never triggers the budget limit.

The role validates: `tensor_parallel_size × pipeline_parallel_size = node_count × gpu_per_node`. Mis-configurations fail at deployment time before any Kubernetes resources are applied.

### Profile A: Single-GPU vLLM

- `node_count: 1`, `gpu_per_node: 1` (or more GPUs with `tensor_parallel_size`)
- `LLMInferenceService` without `worker:` spec → llmisvc controller creates a single Deployment
- MVP profile for the first shared models

### Profile B: Single-node multi-GPU vLLM

- `node_count: 1`, `tensor_parallel_size > 1`, multiple GPUs on one node
- `LLMInferenceService` without `worker:` spec, but `--tensor-parallel-size` passed to vLLM
- Requires homogeneous GPU type and high intra-node GPU bandwidth

### Profile C: Multi-node multi-GPU via LWS

- `node_count > 1` triggers the LWS path: `LLMInferenceService` with a `worker:` section
- **Leader pod** (the `template:` container): full vLLM HTTP server; receives `--nnodes`, `--node-rank=0`, `--master-addr` from LWS env vars (`LWS_GROUP_SIZE`, `LWS_WORKER_INDEX`, `LWS_LEADER_ADDRESS`)
- **Worker pods** (the `worker:` container): vLLM in `--headless` mode (pipeline parallel stage, no HTTP listener)
- `pipeline_parallel_size = node_count` enforced by the role; validated before apply
- Requires a ReadWriteOnce PVC per pod (one PVC per model replica, shared via `ReadWriteMany` only if the storage class supports it), NCCL-ready networking
- No autoscaling for multi-node setups; topology changes require redeployment

### Profile CPU (local/dev only)

- `gpu_per_node: 0` removes `nvidia.com/gpu` resource requests
- Uses `vllm/vllm-openai-cpu` image (configured per environment in `inference.yml`)
- Only for local Minikube dev; not a production serving profile

## Backend Requirements

1. Support a curated set of popular open-source LLMs.
2. Keep shared model endpoints internal to the platform.
3. Route all user-facing access through the governed gateway (ADR-0010).
4. Support streaming and tool/function-calling-capable models where needed.
5. Support single-GPU, single-node multi-GPU, and multi-node multi-GPU model profiles.
6. Treat topology as model catalog metadata rather than a per-request user option.
7. Avoid exposing GPU orchestration details to shared LLM users.
8. Preserve dedicated project GPU provisioning for teams that need custom models or exclusive resources.

## Considered Options

### Option A: Continue with dedicated GPU pods only

Every project receives GPU quota and runs its own inference pod.

**Decision:** Rejected for shared LLM access. Keep this model for custom/dedicated workloads.

### Option B: Expose vLLM/KServe endpoints directly to users

Users receive API tokens that call vLLM/KServe directly.

**Decision:** Rejected. vLLM/KServe endpoints remain internal. Quota enforcement, project authorization, audit, and policy must be centralized in the gateway.

### Option C: KServe LLMInferenceService + vLLM behind governed gateway (adopted)

Use KServe `LLMInferenceService` internally. Expose only the governed gateway (ADR-0010) to users and clients. LWS handles multi-node pipeline parallelism.

**Pros:**
- Separates serving from access governance; backend can evolve independently.
- LLMInferenceService provides a dedicated LLM-optimized CRD (readiness probes, LWS wiring, config presets).
- Keeps centralized authorization, quota enforcement, audit, and usage reporting in the governed gateway (ADR-0010).
- Fits existing Kubernetes/Istio operational model.

**Cons:**
- Adds a gateway dependency on every inference request path.
- Multi-node model serving has operational restrictions (no autoscaling, NCCL dependency, topology-as-redeployment).
- LLMInferenceService (v1alpha2) is newer/less battle-tested than the generic InferenceService.

**Decision:** Accepted.

### Option D: Custom vLLM deployment without KServe

Deploy vLLM directly with custom Helm/Kubernetes manifests.

**Decision:** Not selected for the platform default. May be used for exceptional models if LLMInferenceService cannot express the required topology.

## Consequences

**Positive:**
- The platform gains a managed shared LLM backend without giving every user direct GPU access.
- Model serving topology is explicit, declarative, and reviewable in `inference.yml`.
- Larger models can be supported with vLLM tensor/pipeline parallelism where the cluster topology allows it.
- The public access layer (the governed gateway, ADR-0010) can evolve independently of the backend serving layer.

**Negative:**
- Multi-node vLLM serving with LWS requires operational discipline: homogeneous GPU types, NCCL-ready networking, no autoscaling, disruptive redeployment for topology changes.
- LLMInferenceService (v1alpha2) is a newer CRD with a smaller operational track record than generic InferenceService.
- Backend capacity planning must account for KV cache, context length, concurrency, and token throughput, not only model weight size.

**Neutral:**
- CPU mode (local only) is a control-plane smoke path, not a production serving profile.
- The `inference-deploy` role and the gateway's model-registration role (ADR-0010) are separate make targets but share the same `inference.yml` catalog, so both must be re-run after any model catalog change.

### Migration

No migration from existing infrastructure is required. The inference-deploy role is idempotent; removed catalog entries are pruned automatically (LLMInferenceService deleted; PVC kept unless `inference_prune_data=true`).

Removing a model from `inference.yml` also removes anything that model contributed on the Waldur side (e.g. its entry in the AI offering's model list/pricing). Waldur commonly refuses to delete an offering component while it is still referenced by an offering or plan, even if that component has no active order using it. When pruning is blocked this way, use `make remove-stale-waldur-component COMPONENT=<key>` (on `main`; not yet ported to this branch) rather than working around it by hand.

## Open Questions

~~1. Which open-source models are in the initial shared catalog?~~
**Resolved:** The local catalog includes `qwen2.5-0.5b-llmisvc` and `qwen2.5-coder-0.5b-llmisvc` (CPU-mode smoke-test models). Production model selection is defined per environment in `inference.yml`.

~~2. What is the minimum context length and tool-calling support required for coding-assistant clients?~~
**Resolved:** `max_model_len` and `enable_auto_tool_choice` / `tool_call_parser` are per-model fields in the catalog. Resolved per model at deployment time.

~~3. What GPU types and interconnect assumptions are available in the managed bare-metal cluster?~~
**Resolved:** Deferred to environment-specific `inference.yml` entries and infrastructure planning.

~~4. Should shared models use a central model cache, PVC-backed storage, or preloaded node-local caches?~~
**Resolved:** PVC-backed storage per model. A git-lfs download Job populates the PVC on first deploy; a ConfigMap marker prevents re-download. PVCs are retained after model removal unless `inference_prune_data=true`.

~~5. What rollout strategy should be used for large multi-node models when spare GPUs are unavailable?~~
**Resolved:** Treat Profile C models as explicitly approved platform services. Rollout requires sufficient spare GPU capacity; no autoscaling for multi-node setups. Strategy is a redeployment (operator deletes and recreates the LWS leader+worker group).

~~6. How will model upgrades be coordinated with gateway model aliases and client-visible model names?~~
**Resolved:** `served_name` in the catalog decouples the internal resource name from the model ID visible to clients and the gateway (ADR-0010). Catalog changes re-run both `inference-deploy` and the gateway's model-registration role.

~~7. What metrics will define capacity saturation?~~
**Resolved:** GPU utilization, queue length, and vLLM's built-in `/metrics` endpoint (Prometheus format) are the primary signals. Thresholds defined per environment.

~~8. Which models, if any, require custom deployment outside KServe?~~
**Resolved:** None identified at MVP. Option D (custom deployment) remains available for exceptional cases.

## References

- `devops/ansible/inventory/<env>/group_vars/inference.yml` — model catalog (single source of truth for deployed models and serving profiles)
- `devops/ansible/roles/inference-deploy/` — Ansible role that applies KServe LLMInferenceService resources
- `devops/ansible/roles/inference-deploy/templates/llmisvc.yml.j2` — Jinja2 template for `LLMInferenceService` manifests
- [KServe LLMInferenceService](https://kserve.github.io/website/docs/model-serving/generative-inference/llminferenceservice/)
- [vLLM: Parallelism and Scaling](https://docs.vllm.ai/en/latest/serving/parallelism_scaling/)
- [Kubernetes SIGs LeaderWorkerSet (LWS)](https://github.com/kubernetes-sigs/lws)
- [NVIDIA Kubernetes Device Plugin](https://github.com/NVIDIA/k8s-device-plugin)