For AI agents: the complete documentation index is available at /llms.txt, the full documentation bundle is available at /llms-full.txt, and this page is available as Markdown at /reference/docs/decisions/0044-llm-stack-service.md.

0044 — The shared LLM stack as the llm Argo CD service

Status: Proposed. Applies 0040 to the shared LLM access epic; the implementation is in DD-0044. Date: 20 October 2026.

Context

Shared LLM access gives each Project personal API keys, a chat and a monthly token allowance (ADRs 0009, 0010, 0011 and 0020). Its gateway is LiteLLM and its chat is Open WebUI. The first roles for them (inference-deploy, litellm-deploy and open-webui-deploy) were written for the NTUA GB10 cluster and assume a shared CloudNativePG cluster, KServe, cert-manager and a private inventory. A dev box has none of these, and 0040 says Argo CD installs every service.

Decision

  1. The gateway and the chat are one Argo CD service, llm, in the llm namespace under the services project: gitops/apps/base/llm.yaml and the wrapper chart gitops/manifests/llm/chart. It runs on sync wave 3 like GitLab and Waldur. The namespace, its Pod Security labels and the controller's Role stay in the Ansible bootstrap, and the project gains no kind.
  2. PostgreSQL is the shared development server (decision 0048), as for GitLab and Waldur: it holds the litellm and open_webui databases, each with its own role and generated password, written by pharos-infra to the Secret llm-postgresql. The service has no PostgreSQL of its own, and the shared CloudNativePG cluster of the GB10 roles is not needed.
  3. Credentials are generated in the namespace by a Sync hook that creates each Secret once and never rewrites it: the gateway's master key and UI password, and the chat's secret key, administrator password and gateway key. Nothing the hook writes is a copy of another credential: the database passwords stay in the Secret pharos-infra publishes and reach each pod through its environment. The chat's gateway key is a virtual key of its own, limited to the model routes and registered by a hook inside the gateway pod, so the master key reaches no other pod (AGENTS.md: administrator credentials never enter runtime application containers).
  4. Every image is pinned by digest in gitops/manifests/llm/pins.yaml, which the Application reads as a values file so the images are written once; scripts/check_platform_gitops.py renders the chart and rejects an unpinned or unlisted image and any kind the project does not allow.
  5. A dev box serves one small model on the CPU through llama.cpp's OpenAI-compatible server, so the gateway and the chat can be tested end to end. Production serving stays KServe. The platform serving composition now supplies KServe and its certificate prerequisite. The Ansible roles stay as the GB10 reference.
  6. No route is published yet. The service is reached through a port-forward until the catalogue routes llm and chat (a later change), and its sign-in through the shared AAI follows with that route. Until then the gateway uses its generated master key and the chat its generated administrator account. Only the gateway may reach the model, and only the chat may reach the gateway; no pod may reach every other.

Consequences

Positive: the stack installs and updates with every other dev-box service and is reproducible from the repository; the GB10 path is untouched.

Negative: two ways to install the stack exist until the GB10 cluster moves to Argo CD, and the dev-box model is not the production runtime.

Neutral: Metallo's llm_access context reconciles teams, keys and chat access through the gateway and chat APIs, whichever way they were installed.

Page source

Source snapshot · Revision f928834c8.