For AI agents: the complete documentation index is available at /llms.txt, the full documentation bundle is available at /llms-full.txt, and this page is available as Markdown at /adr/0009-shared-llm-inference-backend.md.

Themelio compatibility reference. This page describes the retained Themelio implementation or its historical design. Check the parallel-operation guide before applying it to the current platform.

ADR-0009: Shared LLM Inference Backend

Status: Draft

Date: 2026-05-22

Deciders: Platform developers (confirmed 2026-06-30)

Relates to:

Context

The platform currently provisions project-owned infrastructure quotas through Waldur. Teams receive CPU, RAM, GPU, and storage allocations and use them inside their project environment. The current inference implementation is still close to this infrastructure model: a simple Kubernetes inference pod runs on a GPU node, and the NVIDIA Kubernetes device plugin exposes GPU resources to workloads.

The platform also needs a second access mode for popular open-source LLMs: shared managed inference. In this mode, users do not receive dedicated GPU infrastructure. Instead, the platform operates a shared model-serving pool and users consume project-scoped LLM service capacity through the governed access layer described in ADR-0010.

This ADR is intentionally limited to the backend inference plane: where model servers run, how GPUs are allocated, how large models are sharded across GPUs/nodes, how model-serving topologies are represented, and how the backend is exposed internally to the governed LLM gateway. It does not decide user-facing API token management, billing/quota semantics, or web UI behavior. Those are covered by ADR-0010 and ADR-0011.

The intended backend uses KServe as the Kubernetes model-serving abstraction and vLLM as the primary generative inference runtime for open-source LLMs. vLLM is treated as an internal model endpoint, not a public user-facing API. Public access always goes through the governed gateway.

This ADR must also account for models that do not fit on a single GPU. vLLM supports distributed inference through tensor parallelism (tensor_parallel_size) and pipeline parallelism (pipeline_parallel_size). For multi-node serving, the LeaderWorkerSet (LWS) operator coordinates a leader pod (full vLLM HTTP server) with one or more worker pods (headless vLLM pipeline stages).

See also: DD-0009: Implementation details

Decision

The platform operates a shared LLM inference backend using:

  • KServe LLMInferenceService (CRD serving.kserve.io/v1alpha2) as the Kubernetes model-serving abstraction.
  • vLLM (vllm/vllm-openai) as the inference runtime for generative open-source LLMs.
  • LeaderWorkerSet (LWS) operator for multi-node inference, triggered by the worker: spec in LLMInferenceService.
  • Internal-only vLLM endpoints reachable only by the governed LLM gateway (ADR-0010), not directly by end users.
  • Explicit model serving profiles for single-GPU, single-node multi-GPU, and multi-node multi-GPU deployments.
  • A stable headless alias Service (<model-name>-openai) created by Ansible, used by the governed gateway for routing (ADR-0010).

The model catalog is defined in devops/ansible/inventory/<env>/group_vars/inference.yml and is shared with the governed gateway layer: inference-deploy (this ADR's role, creating KServe resources) reads the same file as the gateway's model-registration role (ADR-0010), and waldur-llm-access-offering (available model names for the Waldur AI offering) also reads it. local-test-identities (a generic local SSO/OIDC test-identity seeder, not LLM-specific) only seeds local Keycloak fixtures and does not read the model catalog; project-to-model wiring is owned exclusively by the Waldur-driven path (see DD-0009). Users never interact with the backend directly; all traffic goes through the governed gateway (ADR-0010).

Model Serving Profiles

Every model in the catalog has a serving profile defining runtime topology, resource class, and operational constraints. The profile is set by the node_count and gpu_per_node fields in the catalog entry. Each entry also declares per-token USD cost fields consumed by the governed gateway for spend accounting and project budget enforcement (ADR-0010). On-premises models can be set to zero cost, in which case spend is tracked but never triggers the budget limit.

The role validates: tensor_parallel_size × pipeline_parallel_size = node_count × gpu_per_node. Mis-configurations fail at deployment time before any Kubernetes resources are applied.

Profile A: Single-GPU vLLM

  • node_count: 1, gpu_per_node: 1 (or more GPUs with tensor_parallel_size)
  • LLMInferenceService without worker: spec → llmisvc controller creates a single Deployment
  • MVP profile for the first shared models

Profile B: Single-node multi-GPU vLLM

  • node_count: 1, tensor_parallel_size > 1, multiple GPUs on one node
  • LLMInferenceService without worker: spec, but --tensor-parallel-size passed to vLLM
  • Requires homogeneous GPU type and high intra-node GPU bandwidth

Profile C: Multi-node multi-GPU via LWS

  • node_count > 1 triggers the LWS path: LLMInferenceService with a worker: section
  • Leader pod (the template: container): full vLLM HTTP server; receives --nnodes, --node-rank=0, --master-addr from LWS env vars (LWS_GROUP_SIZE, LWS_WORKER_INDEX, LWS_LEADER_ADDRESS)
  • Worker pods (the worker: container): vLLM in --headless mode (pipeline parallel stage, no HTTP listener)
  • pipeline_parallel_size = node_count enforced by the role; validated before apply
  • Requires a ReadWriteOnce PVC per pod (one PVC per model replica, shared via ReadWriteMany only if the storage class supports it), NCCL-ready networking
  • No autoscaling for multi-node setups; topology changes require redeployment

Profile CPU (local/dev only)

  • gpu_per_node: 0 removes nvidia.com/gpu resource requests
  • Uses vllm/vllm-openai-cpu image (configured per environment in inference.yml)
  • Only for local Minikube dev; not a production serving profile

Backend Requirements

  1. Support a curated set of popular open-source LLMs.
  2. Keep shared model endpoints internal to the platform.
  3. Route all user-facing access through the governed gateway (ADR-0010).
  4. Support streaming and tool/function-calling-capable models where needed.
  5. Support single-GPU, single-node multi-GPU, and multi-node multi-GPU model profiles.
  6. Treat topology as model catalog metadata rather than a per-request user option.
  7. Avoid exposing GPU orchestration details to shared LLM users.
  8. Preserve dedicated project GPU provisioning for teams that need custom models or exclusive resources.

Considered Options

Option A: Continue with dedicated GPU pods only

Every project receives GPU quota and runs its own inference pod.

Decision: Rejected for shared LLM access. Keep this model for custom/dedicated workloads.

Option B: Expose vLLM/KServe endpoints directly to users

Users receive API tokens that call vLLM/KServe directly.

Decision: Rejected. vLLM/KServe endpoints remain internal. Quota enforcement, project authorization, audit, and policy must be centralized in the gateway.

Option C: KServe LLMInferenceService + vLLM behind governed gateway (adopted)

Use KServe LLMInferenceService internally. Expose only the governed gateway (ADR-0010) to users and clients. LWS handles multi-node pipeline parallelism.

Pros:

  • Separates serving from access governance; backend can evolve independently.
  • LLMInferenceService provides a dedicated LLM-optimized CRD (readiness probes, LWS wiring, config presets).
  • Keeps centralized authorization, quota enforcement, audit, and usage reporting in the governed gateway (ADR-0010).
  • Fits existing Kubernetes/Istio operational model.

Cons:

  • Adds a gateway dependency on every inference request path.
  • Multi-node model serving has operational restrictions (no autoscaling, NCCL dependency, topology-as-redeployment).
  • LLMInferenceService (v1alpha2) is newer/less battle-tested than the generic InferenceService.

Decision: Accepted.

Option D: Custom vLLM deployment without KServe

Deploy vLLM directly with custom Helm/Kubernetes manifests.

Decision: Not selected for the platform default. May be used for exceptional models if LLMInferenceService cannot express the required topology.

Consequences

Positive:

  • The platform gains a managed shared LLM backend without giving every user direct GPU access.
  • Model serving topology is explicit, declarative, and reviewable in inference.yml.
  • Larger models can be supported with vLLM tensor/pipeline parallelism where the cluster topology allows it.
  • The public access layer (the governed gateway, ADR-0010) can evolve independently of the backend serving layer.

Negative:

  • Multi-node vLLM serving with LWS requires operational discipline: homogeneous GPU types, NCCL-ready networking, no autoscaling, disruptive redeployment for topology changes.
  • LLMInferenceService (v1alpha2) is a newer CRD with a smaller operational track record than generic InferenceService.
  • Backend capacity planning must account for KV cache, context length, concurrency, and token throughput, not only model weight size.

Neutral:

  • CPU mode (local only) is a control-plane smoke path, not a production serving profile.
  • The inference-deploy role and the gateway's model-registration role (ADR-0010) are separate make targets but share the same inference.yml catalog, so both must be re-run after any model catalog change.

Migration

No migration from existing infrastructure is required. The inference-deploy role is idempotent; removed catalog entries are pruned automatically (LLMInferenceService deleted; PVC kept unless inference_prune_data=true).

Removing a model from inference.yml also removes anything that model contributed on the Waldur side (e.g. its entry in the AI offering's model list/pricing). Waldur commonly refuses to delete an offering component while it is still referenced by an offering or plan, even if that component has no active order using it. When pruning is blocked this way, use make remove-stale-waldur-component COMPONENT=<key> (on main; not yet ported to this branch) rather than working around it by hand.

Open Questions

1. Which open-source models are in the initial shared catalog? Resolved: The local catalog includes qwen2.5-0.5b-llmisvc and qwen2.5-coder-0.5b-llmisvc (CPU-mode smoke-test models). Production model selection is defined per environment in inference.yml.

2. What is the minimum context length and tool-calling support required for coding-assistant clients? Resolved: max_model_len and enable_auto_tool_choice / tool_call_parser are per-model fields in the catalog. Resolved per model at deployment time.

3. What GPU types and interconnect assumptions are available in the managed bare-metal cluster? Resolved: Deferred to environment-specific inference.yml entries and infrastructure planning.

4. Should shared models use a central model cache, PVC-backed storage, or preloaded node-local caches? Resolved: PVC-backed storage per model. A git-lfs download Job populates the PVC on first deploy; a ConfigMap marker prevents re-download. PVCs are retained after model removal unless inference_prune_data=true.

5. What rollout strategy should be used for large multi-node models when spare GPUs are unavailable? Resolved: Treat Profile C models as explicitly approved platform services. Rollout requires sufficient spare GPU capacity; no autoscaling for multi-node setups. Strategy is a redeployment (operator deletes and recreates the LWS leader+worker group).

6. How will model upgrades be coordinated with gateway model aliases and client-visible model names? Resolved: served_name in the catalog decouples the internal resource name from the model ID visible to clients and the gateway (ADR-0010). Catalog changes re-run both inference-deploy and the gateway's model-registration role.

7. What metrics will define capacity saturation? Resolved: GPU utilization, queue length, and vLLM's built-in /metrics endpoint (Prometheus format) are the primary signals. Thresholds defined per environment.

8. Which models, if any, require custom deployment outside KServe? Resolved: None identified at MVP. Option D (custom deployment) remains available for exceptional cases.

References

Page source

Source snapshot · Revision f928834c8.