For AI agents: the complete documentation index is available at /llms.txt, the full documentation bundle is available at /llms-full.txt, and this page is available as Markdown at /reference/services/metallo/contexts/llm_access/README.md.

LLM Access — order fulfilment core (3B, first slice)

This folder holds the pure core of subtask 3B: how one approved Waldur order becomes a Project's allocation (models, budget, limits) and the desired gateway and chat state derived from it.

The context is registered and activated: the Metallo service opens its private database and relays its outbox, the workload mounts its restricted credentials, and allocations persist through the generic aggregate store (state, version, command receipts and the reconcile intent in one transaction). The reconcile intent, llm_access.allocation.reconcile.v1, is an event private to this context; the aggregate has no browser projection, because nothing reads one, and a shared realtime contract is added when a browser feature needs it. A snapshot whose version differs from its row is refused as corrupt.

Delivery runs as outbox relay, then the broker queue ref.llm_access.gateway, then the consumer, then the Reconciler, then LiteLLM. The consumer is optional by configuration:

  • With neither LLM_ACCESS_GATEWAY_URL nor LLM_ACCESS_GATEWAY_KEY set there is no consumer. The relay still publishes and the queue waits. This is the base deployment's setting, which has no LiteLLM; the devbox overlay, which runs the llm service, sets both.
  • With both set the service consumes. With only one set it refuses to start.
  • The key is a LiteLLM key limited to /team/new, /team/update, /team/info, /team/block, /team/unblock and /key/block, created by a LiteLLM administrator outside the container (never the master key, and never a key that may call /key/generate, which can mint unrestricted keys). /key/block is the only key route and there is no /key/unblock, so the credential cannot undo a block. Measured: with it the key could not unblock, generate, update, delete, rotate, list or read keys, create users, read spend logs or register models. Its user has the proxy admin role, because LiteLLM refuses a team update that raises max_budget from any other key and an order may raise the allowance; the route list is what limits it. Ending an allocation does not depend on a change to its limits being accepted: its keys are blocked first, and the team is blocked before its limits are updated. Mount it as the file LLM_ACCESS_GATEWAY_KEY_FILE; it is read on each call, so replacing the file rotates it. A deployment that sets them also has to allow the Metallo pod network egress to the gateway, which the base policies do not; the devbox overlay does both.
  • A gateway failure is retried by the broker a few times, a second apart, and then the message goes to ref.llm_access.gateway.dead, where it waits for the explicit replay (python -m metallo.services.replay llm_access.gateway). Recording the allocation as reconciled happens only after the gateway call succeeds.

A recovery sweep (adaptors/sweep.py) starts with the consumer and reconciles allocations whose gateway state is behind their entitlement, which also covers a dead-lettered message.

The Site Agent intake (contracts/http/llm-access.v1.yaml) is optional by configuration too:

  • With LLM_ACCESS_INTAKE_KEY unset there are no routes. With it set, POST /llm-access/orders and GET /llm-access/allocations/{allocationId} are mounted, and the key must resolve like any other secret (LLM_ACCESS_INTAKE_KEY_FILE is preferred).
  • The key guards only /llm-access. The service key does not open it and the intake key opens nothing else, so the agent that holds it cannot read sessions or Execution Core. It must be at least 32 characters and differ from the service key.
  • LLM_ACCESS_MODELS, a comma-separated list of the served model names, is then required: orders may only name models in it. The base deployment sets neither the key nor the list; the devbox overlay, which runs the llm service, sets both, and a platform check keeps the list equal to the models the gateway serves.
  • The deployment has to allow the agent's network access to Metallo, which the base policies do not.
  • projectAlias preserves the Waldur Project slug's case. Aliases accept up to 63 ASCII letters, digits and internal hyphens. Stored allocations, gateway team names and chat prefixes retain that spelling. Allocation identity comes from the resource UUID; Project aliases can coincide. An applied alias is immutable, including its case. Existing lower-case snapshots retain their representation. Older Metallo versions cannot restore allocations created with mixed-case aliases.

Key records (domain/key.py, application/keys.py, adaptors/keys.py) are a second aggregate of this context: a personal key's member, allocation and models, with no secret. They publish no event yet. Nothing issues gateway keys until subtask 4B, which records the gateway's hashed token on the record. Ending an allocation blocks every key that has a token, revoked ones included, and then its team. The token can block the key, so it is never in a projection, a route or an error.

Key issuing (domain/provision.py, application/issuing.py, adaptors/issuer.py) creates a recoverable gateway key. It chooses the key locally, saves it sealed in a private provision and binds its hashed token before calling LiteLLM. An interrupted call therefore replays the exact key, while termination can block it whether creation completed or not. The caller settles the provision, erasing the sealed value, after it has durably kept or consumed the secret.

  • LLM_ACCESS_ISSUER_KEY is the key of one ordinary internal gateway user, never a proxy admin. A gateway administrator creates the user once, with a model list that names no real model (that is what bounds any personal key it makes) and no budget or rate limit (every Project's keys count toward it), and creates its key with allowed_routes: ["/key/generate"], which only an administrator can set. Mount it as LLM_ACCESS_ISSUER_KEY_FILE.
  • LLM_ACCESS_ISSUER_USER is that user's ID. The Reconciler adds it to every new team as a plain member and grants members /key/generate in the same team call, so it needs no extra route. A team created before this is set needs the user added by an administrator. With the issuer key set, the gateway URL and the issuer user must be set too.
  • Measured: that user can generate keys only for its own teams, cannot name another user or set allowed_routes, and a key it makes with key_type llm_api (always sent) cannot generate, read or list keys. A model list or budget above the team's is accepted but capped by the team at request time. The caller-supplied key and replay behaviour were qualified against LiteLLM v1.96.0. LLM_ACCESS_SEAL_KEY protects both provisions and pending mail deliveries.
  • A leaked issuer credential can create extra keys inside any Project's team, capped by that team and blockable by blocking the team. The Reconciler cannot find such keys, because no record knows them.

Chat access (application/chat.py, adaptors/openwebui.py) gives a Project's members the models its allocation entitles in Open WebUI and removes them when it ends. The Reconciler runs it as the last step of each reconcile, after the gateway, and only when chat is configured:

  • LLM_ACCESS_CHAT_URL is Open WebUI as Metallo reaches it, LLM_ACCESS_CHAT_KEY the admin API key described below (mount LLM_ACCESS_CHAT_KEY_FILE), and LLM_ACCESS_CHAT_GATEWAY_URL the gateway's address as Open WebUI reaches it, the one that goes into every Project's connection. All three or none. Chat needs the gateway and key issuing to be configured, or the service refuses to start. The base policies do not let Metallo reach Open WebUI.

  • A Project key is a key record with no member. An interrupted creation resumes from its sealed provision without making another key. Once issuing returns, the provision is erased; if a later chat step fails, the next attempt retires that recorded key and issues the next. The key asks for no model list, so it follows the team's current models.

  • Per allocation: a group in Open WebUI named for the allocation (the Project alias and the first eight characters of its ID, which is also the model prefix; its description binds it to the allocation, and a group with that name that belongs to something else is a conflict, not overwritten), a connection to the gateway with the Project's key and that name as prefix, and a grant to the group on each name.model. Open WebUI shows a model only to users a grant names and keeps an ungranted model admin-only (measured), so access is added connection first, grants second, and removed grants first, connection second.

  • Group membership follows the member routes (4A, ADR-0020). Adding a member records a chat_member that keeps their address and puts them into the allocation's Open WebUI group; revoking them takes them out. The sign-in provider only says who a person is: Metallo writes no group to its realm. Open WebUI knows a person by their address only after their first sign-in, so a member who has not signed in yet stays pending and the recovery sweep retries until they have; a failure raises and the request is retried. Group membership is read per request, so a removal ends access in a session that is already open. Group management from the sign-in claim is off, so a sign-in never changes groups. Without chat configured, keys are still issued and the member record persists in both authority modes. Group changes wait until chat is configured, retaining the last applied account so that a pending removal can then finish. In beto mode the recovery sweep remains active for key authority. Who belongs is decided in one of two ways, by LLM_ACCESS_CHAT_MEMBERSHIP. site-agent (the default) lets the Site Agent's member routes decide, as above. beto lets Beto's membership decide (#769): LLM Access keeps its own copy of Beto's Project and Party facts (LLM_ACCESS_VIEWS_BROKER_URL is its enrolled identity, and PROJECT_SOURCE_SCOPE the source it loads for), the member routes only record who the Site Agent named (sign-in subject, address and Waldur user UUID), and the sweep makes the group follow the people Beto's rule admits to the allocation's Project. An allocation is the Project whose Waldur ID it carries, and a named person is in it when Beto's membership holds their Waldur user UUID. A person Beto names that the Site Agent has not named, or one without a Waldur user UUID, is never added. A Project the loaded view does not hold is left as it is, with a warning from the sweep, and a held Project that does not name a person refuses them. The view must have finished loading before anything changes. A person Beto takes away after admitting them also has their personal keys stopped at the gateway and revoked, before they leave the group. Every refused member is checked for old keys, including someone never admitted to chat or without a usable Waldur user UUID. This repairs keys issued before the membership check existed; pending deliveries are abandoned and their sealed secrets erased. The sweep also discovers allocations through their personal keys, covering older issuance that failed before saving its first member record. Once the view holds the Project, personal keys with no member record are unpaired and revoked. The member directory is rechecked under the key-issuance lock before this repair, so concurrent authorised issuance keeps its newly saved link. An incomplete view or an unheld Project still preserves existing keys. Before switching an existing installation from site-agent to beto, complete the member-link preflight and backfill. It inventories every active personal key, including keys whose provisioning has not finished. Missing member links or usable Waldur UUIDs block a switch that preserves authorised members' existing keys. The account is the one whose sign-in subject Open WebUI recorded as the member's Waldur reference (the platform AAI issues it as the subject) and whose address is the member's: an account with the same name or address is never used. The record also keeps the Open WebUI account the group was last told about: when the member's address or subject moves to an account that has not signed in yet, the earlier account leaves the group at once and only the add waits. The record remembers the add requests it has taken into account, so an add replayed after the member left, or an older one still in flight, cannot put them back; only a new request does.

  • The credential is an Open WebUI admin API key limited to four path prefixes: /openai/config, /api/v1/groups, /api/v1/models/model/access and /api/v1/users (to find a signed-in account by address and read the subject it signed in with; the allowlist is by path, so it also covers the other user routes of an admin key). Set it up on the Open WebUI side with ENABLE_API_KEYS, endpoint restrictions on and those prefixes as the allowed endpoints (the Open WebUI role has variables for them; once saved they live in its database, so an existing install changes them through /api/v1/auths/admin/config). With it the key could not read users or the admin config, add users, delete models, chat or read its own key (measured). It can read every connection's key through the config, so it is held by this adaptor alone and is never the gateway's credential.

  • The connection list is replaced as a whole and its settings are keyed by position, so it is read, changed and written under a lock, and removing one connection renumbers the others.

Usage reporting (application/usage.py, adaptors/usage.py) answers GET /llm-access/allocations/{id}/usage?period=YYYY-MM for the Site Agent's report mode, which posts the figures to Waldur. Metallo holds no Waldur credential:

  • The month is summed from the gateway's per-UTC-day figures for the allocation's team (/team/daily/activity), never from the gateway's own budget, which resets on its own schedule, so a request that finishes after midnight is counted in its own month. The figures are cumulative: asking again, or twice, gives the same or a larger answer, which makes duplicate delivery, interruption and restart harmless. A later month is refused, an earlier one is allowed, and a month before the allocation existed is zero.
  • Waldur components: spend is the priced one (TOTAL_ACCOUNT_USAGE); requests (served requests, a cancelled stream counts for what it produced), inputTokens, outputTokens and totalTokens are metered. failedRequests costs nothing and is not reported. The agent should report the month just ended once more after it closes, because the last figures arrive a little late.
  • The route exists only with LLM_ACCESS_USAGE_KEY (mount LLM_ACCESS_USAGE_KEY_FILE), the gateway URL and the intake. The key is a gateway key limited to the one route: a gateway administrator creates a proxy-admin user's key with allowed_routes: ["/team/daily/activity"]. Measured: it reads any team's figures and was refused everything else. It reads no prompts. The raw /spend/logs route was rejected because its team filter did not filter (it returned every team's entries) and its entries have fields for messages and responses.

Member budget view (application/member_budgets.py) answers the API's typed ListProjectBudgets query. GET /api/v1/projects/{projectId}/llm-budgets requires a current Party session and Beto's project.view permission. The owner checks access before and after the read, resolves the canonical Project's enrolled Waldur binding, and returns all matching allocations with their revisions.

The product decision for M18-34 is the existing monetary budget in EUR. Configured Pharos model prices and offering components use virtual EUR; the default development prices are documented in LLM serving. The view reports the approved budget, measured spend and remaining amount for the current UTC calendar month. Remaining budget stops at zero when spend exceeds the allowance; the measured spend remains unchanged. TPM and RPM are separate optional rate limits. Measured token counts describe usage and are not a monthly token allowance.

Missing measurement configuration or a gateway failure produces null usage and remaining amounts while retaining the approved budget. A successful empty usage report produces measured zero. The measurement timestamp and next monthly reset are explicit; recent requests can arrive after the snapshot. The retained web reads on entry and on the member's explicit refresh, without background polling.

The Metallo process runs this query consumer with its dedicated LLM_ACCESS_REQUESTS_BROKER_URL, existing context database, scoped usage key and Beto caller. The API receives neither gateway credentials nor provider bindings. Enable it only with an enrolled Waldur PROJECT_SOURCE_SCOPE. External model prices denominated in another currency must be converted into the configured euro accounting prices before use; this view does not relabel dollar prices.

Dashboard chat entry (application/member_chat.py) answers the typed GetProjectChatAccess query at GET /api/v1/projects/{projectId}/llm-chat-access. It checks current project.view and runs.launch authority, then reuses the chat membership policy over the caller's established provider accounts in the context-owned Beto view. The local membership revision must match the current Project revision. Access and binding checks run again before returning the result. An incomplete view is retryable; an inaccessible Project remains missing.

The result contains canonical Project/allocation IDs, revisions and permitted Project-prefixed model names. Each allocation is ready, preparing, blocked or unavailable. Ready means its entitlement is reconciled and this member's current chat account has its requested membership applied. An eligible member can open an active allocation while it is preparing: first OIDC sign-in creates the Open WebUI account the existing recovery sweep needs. This read writes no membership or provider grant. Blocked allocations disclose no model grants or entry link. Current Project access and the existing provider revocation workflow remain authoritative for access from an already open chat session.

LLM_ACCESS_CHAT_PUBLIC_URL is the reviewed public HTTPS origin, with no path, query, fragment or credentials. GitOps derives it from the development edge catalogue or the distributed environment's llm-chat origin. The owner adaptor uses Open WebUI's /oauth/oidc/login entry; the enrolled callback remains /oauth/oidc/callback. LLM_ACCESS_CHAT_URL retains its internal provider purpose. An unpublished chat has no entry link. The dashboard reads on entry, explicit refresh and click, without polling. Metering outages do not affect this query.

Key deliveries (domain/delivery.py, application/deliveries.py, adaptors/sealing.py) are the foundation for sending a personal key to its owner. The gateway shows a key's secret once, so until the key is mailed it is kept in a pending delivery, sealed (AES-256-GCM, bound to that delivery) together with its token and the owner's address. The record is internal: it is never projected to the browser and publishes no event, and the sealed value is erased as soon as the delivery is settled as sent or abandoned. A pending delivery cannot lose its secret and a settled one cannot hold one; saved state that breaks either rule is refused as corrupt. Nothing creates a delivery yet: the route that does comes in the next change.

Personal keys (application/member_keys.py, domain/delivery.py, adaptors/sealing.py) give a member of a Project their own key. POST /llm-access/allocations/{id}/members takes a request ID, the member's Waldur reference (a binding the service never rewrites; it is the offering username, which a federated deployment issues as the AAI address and which may carry an @; the member's ID and the key's ID are derived from it) and their email address:

  • In beto membership mode, application/member_admission.py checks the allocation's Project and the member's Waldur binding against Beto's loaded authority before creating or resuming a key, returning an existing key or rotating it. A refusal returns member_not_authorised; an incomplete view, missing Project or pending authority remains retryable and issues nothing. A retry or rotation may use the member's already recorded binding when the request omits it. The check applies even when Open WebUI is not configured. The member link is saved before creating or resuming a key through a record-only operation: it contacts no chat provider and reconciles no other member. site-agent mode retains its existing authority and records the member through the request-aware operation after the key and delivery are durable, including when chat is disabled.
  • The key record takes the allocation's models and Project. Before LiteLLM is called, a random key is sealed into a provision and its hashed token is recorded. After LiteLLM confirms that exact key, its secret, token and address are sealed into the pending delivery and the provision is erased. Nothing in the clear is stored, projected or published.
  • A retry never creates a second key: it resumes the provision or delivery. Failure to save the delivery leaves the same sealed provision for retry; an alias collision on the first provider call is refused rather than mistaken for replay. Two identical requests at once are serialised by one lock that the routes, the recovery sweep and the chat share: a re-entrant thread lock under a named database advisory lock, so the second pod of a rolling update waits for the first.
  • Optional chat synchronisation follows the durable key and delivery. A provider outage, conflicting group or another member's failed group change may make the request retry, while the issued key and sealed delivery remain recoverable. The chat record retains pending work and a retry reuses the same key and delivery.
  • LLM_ACCESS_SEAL_KEY (mount LLM_ACCESS_SEAL_KEY_FILE) is 32 random bytes as 64 hex characters or base64. It is a different secret from every other credential: losing it loses pending deliveries and leaking it exposes them. The route exists only with it, the intake, the gateway and key issuing; the service refuses to start with the sealing key but without them. Without it a deployment that names the gateway still starts — orders, usage and the chat group do not need it — but it records seal-key-missing for llm_access/member_keys in /diagnostics and logs it, because it then issues no personal key and stops none for a member Beto takes away. A restart cannot bring a missing key back; restoring the secret can.
  • The key is then mailed (subtask 4C, below). A delivery stays pending until it is.

Member revocation (application/member_revocation.py) takes a leaving member's keys away. POST /llm-access/allocations/{id}/members/revoke takes a request ID and the member's Waldur reference. Each personal key the member holds on the allocation is stopped at the gateway first, then revoked, and a delivery never mailed is ended and its sealed secret and address erased. It is safe to repeat, shares the lock of the route that adds members and touches nothing else (other members, the Project key, other allocations). Ending chat access is not built.

Member rotation (MemberKeys.rotate_member, KeyMailing) replaces a member's key in two steps. POST /llm-access/allocations/{id}/members/rotate issues the next generation key and seals it for mailing, leaving the current key live; asking again meanwhile returns the same next key. When that key's mail has gone out, the mailer stops and revokes the member's other live keys on the allocation before settling the delivery, so the member always has a working key. An unmailed next key is abandoned after the delivery limit and the current key stays.

Mailing keys (application/mailing.py, adaptors/mailer.py, adaptors/mailing.py) sends each pending key to its owner from a dedicated platform account. Metallo does not run a mail server: it needs an SMTP account on one the platform may use, and nothing starts without it. LLM_ACCESS_SMTP_HOST turns it on, and then these are required or the service refuses to start:

  • LLM_ACCESS_SMTP_FROM, the sender address, and LLM_ACCESS_PUBLIC_API_URL, the gateway address a member's client uses, which the email shows. Personal keys (the sealing key, the gateway and key issuing) must be configured too, because mailing sends what they keep.
  • LLM_ACCESS_SMTP_PORT (587), LLM_ACCESS_SMTP_TLS (starttls, the default, or ssl; none is accepted only for a loopback mail catcher), and LLM_ACCESS_SMTP_USERNAME with LLM_ACCESS_SMTP_PASSWORD (mount LLM_ACCESS_SMTP_PASSWORD_FILE), which go together. The password is read at every send, so rotating it needs no restart. The server's certificate is checked.
  • LLM_ACCESS_DELIVERY_TTL_HOURS (72) and LLM_ACCESS_MAIL_INTERVAL_SECONDS (30).

The dev-box root derives the emailed base from its reviewed llm-api route and appends /v1; it enables the loopback Mailpit sender only while that profile is selected. Distributed delivery requires smtp.public_api to match the reviewed edge.origins.llm-api and sends the same /v1 base. LLM_ACCESS_GATEWAY_URL retains the internal address used for provider operations.

A pass mails every pending key that is due and settles it as sent, which erases the sealed secret and address. A failure is counted and keeps the secret, so the retry sends the same key; the delays grow (1, 3, 7, 15, 30 and 60 minutes after the key was made, then one more hour for each further failure). A key still pending after its time limit is given up: the key is stopped at the gateway, its record is revoked and its secret is erased, so no key is left that its owner never received. The member's next request then makes a new one. If the mail went out but settling failed, the same key is mailed again, never a second key. A failure is reported by its class alone: the address, the secret and the mail server's own text are never stored or logged.

Personal key self-service

Settings → LLM Keys selects a canonical Project and allocation. The member can request an emailed personal key, replace it, revoke it and explicitly refresh its status. The token settings page links to this entry. Key material is delivered by the existing mail worker; the page, API, request receipts and browser messages contain no secret, gateway token, address or provider account.

PersonalAccess is the member/Project/allocation request aggregate in LLM Access's private database. application/personal_access.py records its command identity, allocation version, reserved key identities and a digest of the existing trusted member delivery binding. The snapshot and command reply commit atomically. A retry preserves the original material input; a new request while one is pending reuses that pending work. Recovery finishes the reserved key rather than selecting another generation after a lost acknowledgement. MemberKeys and KeyMailing retain issuance, sealed provision/delivery, rotation and revocation authority.

The member query and command use fresh Beto Project visibility and execution authority with the enrolled Party/provider-account view. Issuance also requires a settled, active allocation and the trusted member binding already supplied to LLM Access. Missing delivery details are reported as preparing; the browser cannot submit an email or provider identity. Usage measurement is independent of access. The HTTP definitions are in keys.paths.json. The existing LLM_ACCESS_REQUESTS_BROKER_URL enrols query and command consumers, the durable command-reply relay and recovery; no new credential is required.

Valid pending phases are requested (gateway work unacknowledged), mail_pending (key prepared, email unacknowledged) and revoking (gateway revocation unacknowledged). mail_sent means SMTP acceptance, not inbox receipt. revoked acknowledges the recorded key revocations; denied records withdrawn authority, changed material scope or an unusable reserved key. Provider outages retain the pending request. The owner retries every five seconds; the browser only reads on entry or Refresh.

Acceptance, issuance, revocation and mailing share the existing member-key lock across threads and service replicas. The mail worker reloads each scanned delivery and key under that lock and checks current membership before opening its sealed payload. Self-service deliveries also check current canonical authority and the accepted allocation/contact binding. A recorded revocation fences later mail and stops its exact keys; replay cannot revoke a subsequently issued key. Revocation preserves chat membership. Replacing retires the old key after SMTP accepts the new email. An explicit revocation or denied request also settles any sealed provision.

test_personal_keys.py covers recovery, retries, scope/authority changes and secret absence. llm_personal_requests.py, included in pnpm test:integration -- --group metallo, checks private PostgreSQL receipts and RabbitMQ replay. The browser fixture is pnpm exec playwright test --config acceptance/browser/personal-keys.config.ts. Live qualification additionally requires an owned trusted member binding, its Project/allocation and authorised mailbox access; the earlier Site Agent acceptance helper requires adaptation for Beto membership mode. Fixture results do not establish live gateway request or revocation behaviour.

The allowance period is one calendar month (UTC), restored by the gateway's own 1mo reset: an order carries 1mo and anything else is refused. The reset happens at 00:00 UTC on the first of the month but is applied by the gateway up to about ten minutes late (measured), so a Project that ran out stays blocked that long into the new month. Details are in the design note (subtask 5B).

Allowance enforcement is not strict. The gateway checks a team's budget before a request and learns the cost after it, so the request that crosses the cap is still served and a burst can go further. It is documented, with the measurements, in the design note (subtask 5A), and the overshoot shows in the usage figures. Subtask 5A stays open until admission with cost reservation exists.

  • domain/ has no technology imports: Order evidence, the ProjectAllocation aggregate, and fulfil, the decision function.
  • adaptors/ composes the aggregate store (postgres.py) and encodes and decodes the reconcile event's payload (projection.py), refusing unknown fields and anything the encoder would refuse; the codec ties the payload to the envelope's allocation and version.
  • application/ has the command and LlmAccess.apply_order, built on the foundation command port. It hands a durable reconcile intent on for each change.

The rules, the open questions and the next steps are in the design note.

For members

A key from the welcome mail works with any OpenAI-compatible client. The address in that mail is the LLM service's public origin, written as <llm-api-host> below: the host alone, with no /v1 and no trailing slash. Two coding agents need settings beyond the key, because they send more than plain chat messages.

Codex CLI

~/.codex/config.toml:

model = "mistral-small-4"
model_provider = "pharos"
model_reasoning_effort = "high"
model_reasoning_summary = "none"

[model_providers.pharos]
name = "Pharos"
base_url = "https://<llm-api-host>/v1"
wire_api = "responses"
env_key = "PHAROS_API_KEY"

model_reasoning_summary = "none" matters: a client that asks the model for a reasoning summary sends reasoning: {effort, summary}, and the gateway forwards that object as reasoning_effort, which vLLM refuses (Input should be 'none', 'minimal', 'low', …). Codex 0.164.0-alpha.11.1 does this by default; 0.163.0, the version the service was measured with, does not.

PHAROS_API_KEY holds the key from the welcome mail. Without a catalogue entry for the model, Codex falls back to its own defaults and warns that the model is unknown; the entry below removes the warning and the wrong context size, and is written as model_catalog_json = "/path/to/catalogue.json" in the same file.

{
  "mistral-small-4": {
    "context_window": 32768,
    "max_context_window": 32768,
    "auto_compact_token_limit": 26000,
    "input_modalities": ["text", "image"],
    "default_reasoning_level": "high",
    "apply_patch_tool_type": null
  }
}

Codex starts each request with about 7,000 tokens of instructions, so the remaining window suits small tasks: a file to write, a bug to fix, a command to run.

input_modalities says what the model takes. mistral-small-4 serves its vision tower, so it is ["text", "image"]: an image is sent as an input_image content part of input and the gateway carries it to the model through the entry's chat-completions path. A model the catalogue does not mark as image-capable takes ["text"] only, and Codex refuses to attach an image to it instead of sending one that cannot be read. The entry is the member's own file, so a member with a text-only model keeps ["text"].

Claude Code

export ANTHROPIC_BASE_URL="https://<llm-api-host>"
export ANTHROPIC_AUTH_TOKEN="<key from the welcome mail>"
export ANTHROPIC_MODEL="mistral-small-4"
export ANTHROPIC_SMALL_FAST_MODEL="mistral-small-4"
export ANTHROPIC_DEFAULT_HAIKU_MODEL="mistral-small-4"
export ANTHROPIC_DEFAULT_SONNET_MODEL="mistral-small-4"
export ANTHROPIC_DEFAULT_OPUS_MODEL="mistral-small-4"
export CLAUDE_CODE_MAX_CONTEXT_TOKENS=32768
export CLAUDE_CODE_AUTO_COMPACT_WINDOW=26000
export CLAUDE_CODE_MAX_OUTPUT_TOKENS=4096
export CLAUDE_CODE_DISABLE_UNKNOWN_MODEL_WINDOW_ENFORCEMENT=1

Claude Code calls /v1/messages on that origin, which the gateway serves in the Anthropic format. Every model variable names the same entry, so no request is routed to a model the key cannot reach.

An image is an image content block — a base64 source or a url — on a user message; the gateway's Anthropic adapter turns it into the OpenAI image_url part before the model is called, and /v1/messages needs no extra setting for it. Claude Code attaches an image itself when the model is marked as taking one.

Images

Whether a model takes an image is decided once, in the environment catalogue (supports_vision). Three readers use that one answer: the gateway reports it in model_info (/model/info; /v1/models stays the plain OpenAI listing), Open WebUI is told it per project model, and the members' Codex catalogue documents it. /v1/chat/completions, /v1/responses and /v1/messages all carry an image to a model the catalogue marks; a text-only entry refuses it with the upstream's own error, because the gateway does not inspect the content.

The number of images follows the serving engine, not this platform: no catalogue entry sets vLLM's --limit-mm-per-prompt, so its default applies — one image per prompt. A request with more is refused by vLLM and the member sees that error through the gateway. A site that needs a different number sets it in that entry's extra_args. Open WebUI compresses an uploaded image only when a compression width is configured, and refuses a file larger than its own limit (unset here), so the size a member hits first is the model's own.

Page source

Source snapshot · Revision f928834c8.