> For AI agents: the complete documentation index is available at /llms.txt, the full documentation bundle is available at /llms-full.txt. # DD-0044 — The shared LLM stack as the `llm` service Related ADR: [0044](/reference/docs/decisions/0044-llm-stack-service.md). Status: draft for review. Date: 20 October 2026. ## Scope and responsibilities This is infrastructure configuration under [decision 0040](/reference/docs/decisions/0040-platform-layout.md). There is no new business context, aggregate, command, event or subscription. | Owner | Responsibility | | ----------------- | ------------------------------------------------------------------------------------------------------------------------------------------ | | Ansible bootstrap | The `llm` namespace and its Pod Security labels, the `services` project's destination, the controller's Role, the services cluster binding | | Argo CD | The `llm` chart: generated Secrets, LiteLLM, Open WebUI, the dev-box model and the namespace's NetworkPolicies | ## Application `gitops/apps/base/llm.yaml` installs `gitops/manifests/llm/chart` on wave 3 with `base/values.yaml`, `pins.yaml` and the environment's `overlays//values.yaml`. Sync options match GitLab and Waldur. ## Chart | Object | Notes | | ------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `llm-secrets` hook (wave -3) | Creates `llm-litellm` and `llm-open-webui` once, with only values this namespace generates; the Role it runs under can only read and create Secrets | | `llm-webui-key` hook (PostSync) | Registers the chat's gateway key (`llm-open-webui/openai-api-key`) as an inference-only virtual key, inside the gateway pod; its account `llm-gateway-keys` may read that one Secret and exec in the gateway pod | | `llm-postgresql` Service | An `ExternalName` to the shared PostgreSQL in `pharos-infra`; `pharos-infra` creates the `litellm` and `open_webui` roles and databases and their Secret (decision 0048) | | `litellm` | Deployment and Service on port 4000; routes from `llm.models`, written to a ConfigMap whose checksum restarts the pod; a route changes only through GitOps. The same ConfigMap carries the `tool_order` callback, which the gateway loads only when the catalogue flags a model with `strictToolOrder`: that model's Responses history gets each tool result next to its call, whatever order the client replayed | | `llm-open-webui-settings` hook (PostSync, wave 1) | Reconciles the admin settings a site opts into through Open WebUI's own routes, after the chat's gateway key exists: web search, the embedding model and function calling. It reads each form, merges its own fields and writes only what changed, and writes the embedding key on every sync. A site that names no settings renders no hook | | `open-webui` | Deployment and Service on port 80; its own database, a 2 Gi volume for uploads, the gateway as its only model connection, reached with its own virtual key | | `llm-cpu-model` | A dev box's llama.cpp server for `qwen2.5-0.5b-cpu`; weights download on first start into a scratch volume on the node's disk | | NetworkPolicies | Default-deny ingress; the chat may reach the gateway and only the gateway may reach the model; egress is not restricted | Web search is a per-chat choice and a site opts into it with `llm.openWebui.settings.webSearch`. The query leaves the cluster for the search engine the setting names (DuckDuckGo by default), which the platform holds no key for and cannot audit; the retrieved page is then read by the same chat. A member who does not turn it on for a chat sends nothing to an external engine. ## Images `gitops/manifests/llm/pins.yaml` is the one place the images are written, each by digest: LiteLLM v1.96.0, Open WebUI 0.11.0, the kubectl image Waldur uses, and llama.cpp's server image. The check fails if the chart renders another image or one without a digest. ## Access No route is published. To reach the stack from a workstation: ```sh kubectl --kubeconfig -n llm port-forward svc/litellm 4000 kubectl --kubeconfig -n llm port-forward svc/open-webui 8080:80 ``` The master key and the administrator password are in the `llm-litellm` and `llm-open-webui` Secrets. ## Not covered Routes and sign-in through the AAI, KServe and cert-manager, the shared CloudNativePG cluster, and the Metallo reconciler's own credentials.
Page source Source snapshot · Revision `f928834c8`.