> For AI agents: the complete documentation index is available at /llms.txt, the full documentation bundle is available at /llms-full.txt.
# 0044 — The shared LLM stack as the `llm` Argo CD service
Status: Proposed. Applies [0040](/reference/docs/decisions/0040-platform-layout.md) to the shared LLM access
epic; the implementation is in [DD-0044](/reference/docs/dd/0044-llm-stack-service.md).
Date: 20 October 2026.
## Context
Shared LLM access gives each Project personal API keys, a chat and a monthly token
allowance (ADRs 0009, 0010, 0011 and 0020). Its gateway is LiteLLM and its chat is
Open WebUI. The first roles for them (`inference-deploy`, `litellm-deploy` and
`open-webui-deploy`) were written for the NTUA GB10 cluster and assume a shared
CloudNativePG cluster, KServe, cert-manager and a private inventory. A dev box has
none of these, and [0040](/reference/docs/decisions/0040-platform-layout.md) says Argo CD installs every service.
## Decision
1. **The gateway and the chat are one Argo CD service, `llm`,** in the `llm`
namespace under the `services` project: `gitops/apps/base/llm.yaml` and the
wrapper chart `gitops/manifests/llm/chart`. It runs on sync wave 3 like GitLab and
Waldur. The namespace, its Pod Security labels and the controller's Role stay in
the Ansible bootstrap, and the project gains no kind.
2. **PostgreSQL is the shared development server** ([decision 0048](/reference/docs/decisions/0048-shared-development-infrastructure.md)),
as for GitLab and Waldur: it holds the `litellm` and `open_webui` databases, each with
its own role and generated password, written by `pharos-infra` to the Secret
`llm-postgresql`. The service has no PostgreSQL of its own, and the shared
CloudNativePG cluster of the GB10 roles is not needed.
3. **Credentials are generated in the namespace** by a Sync hook that creates each
Secret once and never rewrites it: the gateway's master key and UI password, and
the chat's secret key, administrator password and gateway key. Nothing the hook
writes is a copy of another credential: the database passwords stay in the Secret
`pharos-infra` publishes and reach each pod through its environment. The chat's
gateway key is a virtual key of its own, limited to the model routes and registered
by a hook inside the gateway pod, so the master key reaches no other pod
(AGENTS.md: administrator credentials never enter runtime application containers).
4. **Every image is pinned by digest** in `gitops/manifests/llm/pins.yaml`, which the
Application reads as a values file so the images are written once;
`scripts/check_platform_gitops.py` renders the chart and rejects an unpinned or
unlisted image and any kind the project does not allow.
5. **A dev box serves one small model on the CPU** through llama.cpp's
OpenAI-compatible server, so the gateway and the chat can be tested end to end.
Production serving stays KServe. The platform serving composition
now supplies KServe and its certificate prerequisite. The Ansible roles stay as
the GB10 reference.
6. **No route is published yet.** The service is reached through a port-forward until
the catalogue routes `llm` and `chat` (a later change), and its sign-in through
the shared AAI follows with that route. Until then the gateway uses its generated
master key and the chat its generated administrator account. Only the gateway may
reach the model, and only the chat may reach the gateway; no pod may reach every
other.
## Consequences
**Positive:** the stack installs and updates with every other dev-box service and is
reproducible from the repository; the GB10 path is untouched.
**Negative:** two ways to install the stack exist until the GB10 cluster moves to
Argo CD, and the dev-box model is not the production runtime.
**Neutral:** Metallo's `llm_access` context reconciles teams, keys and chat access
through the gateway and chat APIs, whichever way they were installed.
Page source
Source snapshot · Revision `f928834c8`.