# Shared LLM serving

Ansible roles for the shared LLM backend: model serving (KServe and vLLM), the
LiteLLM gateway and Open WebUI chat. They are the starting point for the shared
LLM access epic
([ADR](../../services/metallo/docs/epics/shared-llm-access/adr.md), subtasks 1C,
2A, 2B and 2C).

**Provenance.** This logic is imported from the CSLab `pharos.devops` fork
(`feature/llm-inference-stack-gb10`, `02c6bf1f`), where it was built and qualified
on the NTUA GB10 cluster. We import it here and build on top of it. The roles are
byte-identical to that source. Later changes happen in this repository, not in
the fork.

The platform will ultimately be deployed elsewhere, so nothing here is specific
to one cluster beyond the environment's own inventory and catalogue. GB10 is only
the reference environment these roles were proven on; see
[Reference environment](#reference-environment).

## What is here

| Path | Purpose |
| --- | --- |
| `ansible/roles/inference-deploy` | Serves each catalogue model as a KServe `LLMInferenceService` on vLLM or TEI. Handles weight download into a PVC (`git-lfs` or `hf`), multi-node serving with LeaderWorkerSet and the per-rank checkpoint split (`files/split_pp.py`). |
| `ansible/roles/litellm-deploy` | Deploys LiteLLM and registers every enabled catalogue model behind one OpenAI-compatible endpoint, passing through each entry's `litellm_model_info`. |
| `ansible/roles/open-webui-deploy` | Deploys Open WebUI against the LiteLLM endpoint. |
| `ansible/roles/inference-deploy/container/kimi-k3-sm121` | Sources for the arm64 Kimi-K3 serving image, referenced by digest in the catalogue. |
| `ansible/inventory/cslab/group_vars/inference.yml` | The model catalogue: the single source of truth read by both `inference-deploy` and `litellm-deploy`. |
| `ansible/inventory/cslab.example/` | Example private inventory: hosts, DNS names, NodePorts and vault keys, with placeholders only. |
| `ansible/playbooks/{inference,litellm,open-webui}.yml` | Entry points. |

## Catalogue

### The catalogue in a GitOps environment

An environment delivered by `make deploy` does not run these playbooks. Its catalogue is the `inference:`
block of its environment policy, `devops/platform/environments/<env>.yml` (Icarus:
`devops/platform/environments/icarus-cloud.yml`), and there it is the only place a model is declared.
`scripts/render_llm_inference.py` renders that one catalogue into every reader, so no two of them can
drift and no overlay may carry a copy of a name:

| Reader | Rendered file | How it reaches the deployment |
| --- | --- | --- |
| Serving | `gitops/manifests/llm-inference/models/*.yaml`, `presets.yaml`, `values.yaml` | the `llm-inference` Application |
| Gateway | `llm-inference/values.yaml`, `llm.models` | the LiteLLM release |
| Order gate | `llm-inference/metallo-env.yaml` | the delivery adds it as a kustomize patch of the `pharos` Application, which sets `LLM_ACCESS_MODELS` on the Metallo Deployment |
| Offering | `llm-inference/waldur-values.yaml` | the delivery adds it as a value file of the `waldur` Application (`pharos.llmOffering.models` and `modelPricing`) |
| Site Agent | none | reads `LLM_ACCESS_MODELS` off the running Metallo Deployment, so it offers exactly what the order gate accepts |

Every served entry states `orderable: true` or `orderable: false`: a project orders chat models, not
embeddings. The token prices come from each entry's `litellm_model_info`, read once (`token_prices`) and
rendered into both places that quote them: Waldur's offering (`modelPricing`, the order form) and the
gateway's `llm.models` (`modelInfo.input_cost_per_token` and `output_cost_per_token`, which the chart writes
into LiteLLM's `model_info` — the numbers LiteLLM charges a project's budget). An orderable entry without prices is rendered at zero on the order
form, which says "not priced yet", not "free", and the gateway is given no cost fields at all, so the zero
never becomes a charge. Setting one side of the pair and not the other is an error the renderer refuses, for
both readers; the one exception is a TEI embedding, which has no output side and prices its input alone.

The default monthly allowance is the environment's too: `inference.budget.max_budget_eur` renders the
offering's `plan.quotas.max_budget`, and the offering's `max_budget` limit component takes the same number
as its `default_limit` (the reconciliation in `llm-offering.md`). An environment that names no budget
renders no plan, so the chart's own default applies.

`scripts/check_platform_gitops.py` holds the four readers to the catalogue (`catalogue_errors`, prices
included: the offering and the gateway must quote the catalogue's numbers exactly) and rejects any overlay
that copies a model name or price again (`copied_catalogue_errors`). The reference catalogue
below, `ansible/inventory/cslab`, is now only the GB10 laboratory's: nothing in a GitOps environment reads it.

### The dev box's catalogue

A dev box has a catalogue too, in the same place and the same shape:
`devops/platform/environments/devbox.yml`, `kind: devbox`. It names no delivery, foundation or edge, and
`make deploy ENV=devbox` refuses it with that reason; a dev box is delivered by `make deploy` with no `ENV`.
The renderer reads it exactly as it reads a site's:

| Reader | Rendered file | How it reaches the dev box |
| --- | --- | --- |
| Gateway and CPU server | `gitops/manifests/llm/overlays/devbox/catalogue.yaml` | the dev-box root adds it as the last value file of the `llm` release |
| Order gate | `gitops/manifests/pharos/overlays/devbox/metallo-env.yaml` | the pharos dev-box overlay names it as a patch file |
| Offering | `gitops/manifests/waldur/overlays/devbox/catalogue.yaml` | the dev-box root adds it as the last value file of the `waldur` release |

An entry with `runtime: cpu` is the `llm` chart's own llama.cpp server: the renderer writes `llm.cpuModel`
and the gateway entry that points at it, and the chart runs one CPU server, so a second such entry is
refused. `dx/common/waldur-seed.yaml` orders the model this catalogue marks orderable, and
`seed_catalogue_errors` checks it.

`inventory/cslab` is the reference environment's catalogue: the three models
enabled there today, with their entries and every shared setting unchanged from
the source. A different deployment adds its own catalogue beside it and points
`inference_catalogue_file` at it.

| Model | Serving | Image |
| --- | --- | --- |
| `qwen2-5-3b-llmisvc` | vLLM, one node | `vllm/vllm-openai:v0.26.0` (role default) |
| `kimi-k3` | vLLM on LeaderWorkerSet, sixteen nodes, DSpark draft model | `vllm-kimi-k3-sm121@sha256:9e61a05b…` from the internal registry |
| `qwen3-embedding-0-6b` | TEI, one GPU time-slice | `text-embeddings-inference@sha256:f1e03109…` |

The tokens are priced per model in `litellm_model_info`, and the same numbers reach
both the offering and the gateway. For Icarus, decision 1C agreed
`2.1e-07`/`8.5e-07` (Mistral Small 4), `7.0e-08`/`2.6e-07` (Krikri) and
`2.0e-08` (the embedder's input; an embedding has no output tokens, so its
`output_cost_per_token` is omitted rather than set to zero). These are the prices
the live deployment charges, and moving the catalogue into GitOps keeps them;
turning them into zeros would make every model free and stop a project's
allowance from running down. Setting the numbers in the catalogue is enough on
both delivery paths — the GitOps renderer passes them to the offering and to
LiteLLM's `model_info`, and the Ansible role rendered the same fields.

A model that generates tokens must set both halves of the pair: a half pair, or a
negative price, is refused when the catalogue renders. Only a TEI embedding may
price its input alone.

The CPU-only dev-box catalogue sets virtual test prices
(1 and 2 virtual EUR per million input and output tokens) so that spend and the
Waldur order form can be exercised on a test cluster. They are not agreed prices.

The embedding model supports Open WebUI retrieval and is not an orderable chat
model. The source catalogue has seventeen further disabled entries, which are not
carried over.

### CPU-only catalogue

`inventory/cslab.example/group_vars/inference-cpu.yml` is a second catalogue for a
cluster without GPUs, such as a clean test VM. It has one entry, `qwen2-5-0-5b-cpu`
(Qwen2.5-0.5B-Instruct), and is not loaded by default:

```bash
ansible-playbook -i <private inventory> playbooks/inference.yml \
  -e inference_catalogue_file=../inventory/cslab.example/group_vars/inference-cpu.yml
```

Pass the same file to the `litellm` and `open-webui` playbooks, so the gateway lists the
model that is served. It serves through the same KServe, vLLM and LiteLLM path as the
GPU catalogue, with the settings the CPU backend needs: the `vllm-openai-cpu` image
(x86_64 only, so not for GB10), `gpu_per_node: 0` (no `nvidia.com/gpu` request),
`float32` (no bf16 kernel on CPUs without AVX512-BF16), `--enforce-eager` and a small
batch size. It is a control-plane and accounting smoke path and says nothing about GPU
capacity. The reference catalogue above is unchanged.

## Private inventory

The playbooks read DNS names and credentials from an inventory kept outside
version control. Copy `ansible/inventory/cslab.example/` somewhere private, fill
it in, and pass it with `-i`.

Variables the roles expect, from reading them:

- `dns.litellm`, `dns.open_webui`, `dns.registry`.
- `litellm_gateway_http_nodeport` and `open_webui_gateway_http_nodeport`, the
  NodePorts of the two Services. There is no role default.
- `vault_litellm_master_key`, `vault_litellm_ui_password`,
  `vault_open_webui_admin_password`, `vault_open_webui_litellm_key`,
  `vault_postgres_passwords.litellm` and `.open_webui`.
- `shared_oauth2_issuer`. `open-webui-deploy` has no default for it and fails
  with an undefined variable when it is missing, even without SSO. Set it to the
  OIDC issuer URL to enable SSO, or to `""` to disable SSO and use the local
  admin login. With SSO on, also set `shared_oauth2_client_id` and
  `shared_oauth2_client_secret`.
- `open_webui_db_enabled: true` and
  `open_webui_default_openai_key: "{{ vault_open_webui_litellm_key }}"`. The role
  defaults are `false` and a placeholder, so without them Open WebUI keeps its
  data in SQLite on its volume and lists no models. The reference environment
  sets both. `vault_open_webui_litellm_key` is also the value `litellm-deploy`
  creates the `open-webui` virtual key from, so keep it in the vault: LiteLLM
  stores keys only as hashes and the value cannot be read back.
- Optional: `databases_namespace` (default `databases`), `deployment_environment`
  (only in messages) and `nodeport_ingress_address`.

The roles fall back to development defaults for some credentials when a vault
value is absent. Treat that as a development convenience and set every value on
any shared deployment.

## Running the playbooks

Run `ansible-playbook` from `devops/ansible`. The roles path is set by the
`ansible.cfg` there, so from any other directory the roles are not found.

```sh
cd devops && uv sync --frozen
cd ansible
uv --project .. run ansible-galaxy collection install -r requirements.yml
uv --project .. run ansible-playbook -i /path/to/private/inventory/hosts.yml \
  playbooks/litellm.yml
```

The controller needs the collections in `requirements.yml`, including
`kubernetes.core`, which all three roles use. The host that
`kube-master-primary` names needs `kubectl` with a working kubeconfig, `helm`
(for Open WebUI) and the Python `kubernetes` and `yaml` modules (Debian packages
`python3-kubernetes` and `python3-yaml`), because the roles run their
`kubernetes.core` tasks on that host. For a catalogue other than the reference
one, add `-e inference_catalogue_file=/path/to/catalogue.yml`.

## Assumptions about the target cluster

- PostgreSQL is a CloudNativePG cluster that exists beforehand; nothing here
  provisions it. See [Database contract](#database-contract).
- KServe 0.19 with its llmisvc controller, Istio, cert-manager, an internal image
  registry, ReadWriteMany storage, and the NVIDIA GPU operator on arm64 nodes.
  The CPU-only catalogue needs none of the last three: no internal registry, no
  ReadWriteMany storage and no GPU operator.

## Database contract

`litellm-deploy` and `open-webui-deploy` do not create their databases. They wait
for CloudNativePG `Database` objects and stop with an error when one is missing.
In the reference environment the `postgres` role of `pharos.devops` creates them
(`make ENV=<env> postgres`), from an entry list in the environment's `all.yml`.
This repository does not contain that role, so any other deployment must provide
the same objects, or change the roles to provision their own.

Expected objects, all in the `databases` namespace (`databases_namespace`) on a
CNPG `Cluster` named `pg` whose read-write service is `pg-rw`:

| Consumer | `Database` object | PostgreSQL database | Owner role | Credentials secret | Extensions |
| --- | --- | --- | --- | --- | --- |
| LiteLLM | `litellm` | `litellm` | `litellm` | `litellm-db-secret` | none |
| Open WebUI | `open-webui` | `open_webui` | `open_webui` | `open-webui-db-secret` | `vector` (pgvector, for RAG embeddings) |

Each secret is `kubernetes.io/basic-auth` with `username` and `password`. The
passwords must equal `vault_postgres_passwords.litellm` and
`vault_postgres_passwords.open_webui`, because the roles build their connection
URLs from the vault values. A matching `Database` object looks like this:

```yaml
apiVersion: postgresql.cnpg.io/v1
kind: Database
metadata: {name: open-webui, namespace: databases}
spec:
  name: open_webui
  owner: open_webui
  cluster: {name: pg}
  extensions: [{name: vector, ensure: present}]
```

The owner roles must also be listed in the cluster's `spec.managed.roles`, which
the source `postgres` role rewrites as a whole. Open WebUI uses this database
only when `open_webui_db_enabled` is true; the role default is `false`, and the
reference environment sets it to `true`.

## Reference environment

The roles were qualified on the NTUA GB10 cluster (arm64, Blackwell, unified
memory), which runs these three models today and keeps its ArgoCD applications on
`pharos.devops`. This repository adds source only and changes nothing there.
Hardware, measurements and the Kimi-K3 findings are recorded verbatim in
[`gb10-inference-notes.md`](gb10-inference-notes.md).

## Verification

`scripts/check_ansible.py`, which `pnpm verify` runs, syntax-checks the three
playbooks against the example inventory and needs no cluster. Behaviour was
qualified on the reference cluster in `pharos.devops`. Running these roles from
this repository has not been exercised.