# DD-0044 — The shared LLM stack as the `llm` service

Related ADR: [0044](../decisions/0044-llm-stack-service.md). Status: draft for
review. Date: 20 October 2026.

## Scope and responsibilities

This is infrastructure configuration under [decision 0040](../decisions/0040-platform-layout.md).
There is no new business context, aggregate, command, event or subscription.

| Owner | Responsibility |
|---|---|
| Ansible bootstrap | The `llm` namespace and its Pod Security labels, the `services` project's destination, the controller's Role, the services cluster binding |
| Argo CD | The `llm` chart: generated Secrets, LiteLLM, Open WebUI, the dev-box model and the namespace's NetworkPolicies |

## Application

`gitops/apps/base/llm.yaml` installs `gitops/manifests/llm/chart` on wave 3 with
`base/values.yaml`, `pins.yaml` and the environment's `overlays/<env>/values.yaml`.
Sync options match GitLab and Waldur.

## Chart

| Object | Notes |
|---|---|
| `llm-secrets` hook (wave -3) | Creates `llm-litellm` and `llm-open-webui` once, with only values this namespace generates; the Role it runs under can only read and create Secrets |
| `llm-webui-key` hook (PostSync) | Registers the chat's gateway key (`llm-open-webui/openai-api-key`) as an inference-only virtual key, inside the gateway pod; its account `llm-gateway-keys` may read that one Secret and exec in the gateway pod |
| `llm-postgresql` Service | An `ExternalName` to the shared PostgreSQL in `pharos-infra`; `pharos-infra` creates the `litellm` and `open_webui` roles and databases and their Secret (decision 0048) |
| `litellm` | Deployment and Service on port 4000; routes from `llm.models`, written to a ConfigMap whose checksum restarts the pod; a route changes only through GitOps. The same ConfigMap carries the `tool_order` callback, which the gateway loads only when the catalogue flags a model with `strictToolOrder`: that model's Responses history gets each tool result next to its call, whatever order the client replayed |
| `llm-open-webui-settings` hook (PostSync, wave 1) | Reconciles the admin settings a site opts into through Open WebUI's own routes, after the chat's gateway key exists: web search, the embedding model and function calling. It reads each form, merges its own fields and writes only what changed, and writes the embedding key on every sync. A site that names no settings renders no hook |
| `open-webui` | Deployment and Service on port 80; its own database, a 2 Gi volume for uploads, the gateway as its only model connection, reached with its own virtual key |
| `llm-cpu-model` | A dev box's llama.cpp server for `qwen2.5-0.5b-cpu`; weights download on first start into a scratch volume on the node's disk |
| NetworkPolicies | Default-deny ingress; the chat may reach the gateway and only the gateway may reach the model; egress is not restricted |

Web search is a per-chat choice and a site opts into it with `llm.openWebui.settings.webSearch`. The query leaves the
cluster for the search engine the setting names (DuckDuckGo by default), which the platform holds no key for and cannot
audit; the retrieved page is then read by the same chat. A member who does not turn it on for a chat sends nothing to
an external engine.

## Images

`gitops/manifests/llm/pins.yaml` is the one place the images are written, each by digest: LiteLLM v1.96.0, Open
WebUI 0.11.0, the kubectl image Waldur uses, and llama.cpp's server
image. The check fails if the chart renders another image or one without a digest.

## Access

No route is published. To reach the stack from a workstation:

```sh
kubectl --kubeconfig <target kubeconfig> -n llm port-forward svc/litellm 4000
kubectl --kubeconfig <target kubeconfig> -n llm port-forward svc/open-webui 8080:80
```

The master key and the administrator password are in the `llm-litellm` and
`llm-open-webui` Secrets.

## Not covered

Routes and sign-in through the AAI, KServe and cert-manager, the shared CloudNativePG
cluster, and the Metallo reconciler's own credentials.