# 0044 — The shared LLM stack as the `llm` Argo CD service

Status: Proposed. Applies [0040](0040-platform-layout.md) to the shared LLM access
epic; the implementation is in [DD-0044](../dd/0044-llm-stack-service.md).
Date: 20 October 2026.

## Context

Shared LLM access gives each Project personal API keys, a chat and a monthly token
allowance (ADRs 0009, 0010, 0011 and 0020). Its gateway is LiteLLM and its chat is
Open WebUI. The first roles for them (`inference-deploy`, `litellm-deploy` and
`open-webui-deploy`) were written for the NTUA GB10 cluster and assume a shared
CloudNativePG cluster, KServe, cert-manager and a private inventory. A dev box has
none of these, and [0040](0040-platform-layout.md) says Argo CD installs every service.

## Decision

1. **The gateway and the chat are one Argo CD service, `llm`,** in the `llm`
   namespace under the `services` project: `gitops/apps/base/llm.yaml` and the
   wrapper chart `gitops/manifests/llm/chart`. It runs on sync wave 3 like GitLab and
   Waldur. The namespace, its Pod Security labels and the controller's Role stay in
   the Ansible bootstrap, and the project gains no kind.
2. **PostgreSQL is the shared development server** ([decision 0048](0048-shared-development-infrastructure.md)),
   as for GitLab and Waldur: it holds the `litellm` and `open_webui` databases, each with
   its own role and generated password, written by `pharos-infra` to the Secret
   `llm-postgresql`. The service has no PostgreSQL of its own, and the shared
   CloudNativePG cluster of the GB10 roles is not needed.
3. **Credentials are generated in the namespace** by a Sync hook that creates each
   Secret once and never rewrites it: the gateway's master key and UI password, and
   the chat's secret key, administrator password and gateway key. Nothing the hook
   writes is a copy of another credential: the database passwords stay in the Secret
   `pharos-infra` publishes and reach each pod through its environment. The chat's
   gateway key is a virtual key of its own, limited to the model routes and registered
   by a hook inside the gateway pod, so the master key reaches no other pod
   (AGENTS.md: administrator credentials never enter runtime application containers).
4. **Every image is pinned by digest** in `gitops/manifests/llm/pins.yaml`, which the
   Application reads as a values file so the images are written once;
   `scripts/check_platform_gitops.py` renders the chart and rejects an unpinned or
   unlisted image and any kind the project does not allow.
5. **A dev box serves one small model on the CPU** through llama.cpp's
   OpenAI-compatible server, so the gateway and the chat can be tested end to end.
   Production serving stays KServe. The [platform serving composition](../../gitops/manifests/kserve/README.md)
   now supplies KServe and its certificate prerequisite. The Ansible roles stay as
   the GB10 reference.
6. **No route is published yet.** The service is reached through a port-forward until
   the catalogue routes `llm` and `chat` (a later change), and its sign-in through
   the shared AAI follows with that route. Until then the gateway uses its generated
   master key and the chat its generated administrator account. Only the gateway may
   reach the model, and only the chat may reach the gateway; no pod may reach every
   other.

## Consequences

**Positive:** the stack installs and updates with every other dev-box service and is
reproducible from the repository; the GB10 path is untouched.

**Negative:** two ways to install the stack exist until the GB10 cluster moves to
Argo CD, and the dev-box model is not the production runtime.

**Neutral:** Metallo's `llm_access` context reconciles teams, keys and chat access
through the gateway and chat APIs, whichever way they were installed.