> For AI agents: the complete documentation index is available at /llms.txt, the full documentation bundle is available at /llms-full.txt. > **Themelio compatibility reference.** This page describes the retained Themelio implementation or its historical design. Check the [parallel-operation guide](/compatibility/index.md) before applying it to the current platform. # ADR-0018: Queued Status and Over-Capacity Handling for Managed Workloads **Status:** Proposed **Date:** 2026-07-02 **Deciders:** TBD ## Context `k8s-root-actor` manages three workload kinds — k8s Deployments, KServe InferenceServices, and SLURM jobs — each tracked by a lifecycle status (see [ADR-0012](/adr/0012-job-deployment-management-system.md) and [DD-0012](/dd/0012-job-deployment-management-system.md)). Until now the only "not yet running" status was `pending`, which conflated two situations: 1. **The record was just created**, before the k8s resources were applied or the SLURM job submitted. 2. **The workload is submitted but cannot start yet** because the resources it asked for are currently in use by others — a transient wait that self-heals. Worse, an **over-capacity request** — asking for more CPU/GPU/memory than the cluster or SLURM partition can _ever_ provide — was not handled at all: - A k8s pod requesting e.g. 40 GPUs when the largest node has 8 lands `Pending` with `PodScheduled=False, reason=Unschedulable`. The pod's init script never runs, so no notification callback fires. The existing readiness watcher only detects Waldur _quota_ blocks (`ReplicaFailure`), not scheduling shortfalls, so the resource sat in `pending` indefinitely with no explanation. - A SLURM job requesting more than the partition can offer stays `PENDING`. The submit script polled only the `sacct` _state_, never the pending _reason_, so again nothing surfaced. Operators and users could not distinguish "waiting for the cluster to free up" from "this will never run", and the second case looked identical to a healthy `pending`. ## Decision Introduce a single new transient lifecycle status, **`queued`**, meaning "submitted, waiting for resources to be freed", for all three workload kinds. An **over-capacity request** is surfaced as the existing terminal **`failed`** status with a descriptive message (e.g. _"requested 40 GPUs, but the largest node provides 8"_) — no separate terminal status is added. Detection is done **both** before submission and at runtime, split by where each capability is reachable: | Kind | Pre-submission over-capacity check | Runtime detection | | --------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------- | | k8s Deployment / InferenceService | At the action POST: query node allocatable via `kubectl get nodes`. A GPU request larger than the max single-node allocatable is recorded as `failed` before the manifest is applied. | Readiness watcher detects an `Unschedulable` pod → `queued`, or `failed` when the request exceeds node capacity. | | SLURM job | Inside the launcher pod, before `sbatch`: `sinfo` partition limits. A request no node can satisfy fails fast. | The poll loop reads the `PENDING` reason (`squeue`/`scontrol`) → `queued`. | `queued` is an **active** status (it still occupies the workload's slug, so the `(slug) WHERE active` uniqueness still holds) and is transient — the readiness watcher keeps watching until the workload runs or fails. ### Rationale - **One new status, not two.** A distinct terminal `rejected` status was considered and rejected: over-capacity is a failure like any other apply-time failure, and reusing `failed` (with a clear message) keeps the state machine small and existing consumers unchanged. - **Over-capacity → `failed` with a record, not a bare HTTP error.** The row is created and then marked `failed` (with a log line), so the attempt is visible in history exactly like a runtime failure, rather than vanishing as a 4xx. - **Both detection paths.** The pre-submission check gives fast, clear feedback for the obvious cases; the runtime check is the backstop for conditions that only appear at schedule time (and for capacity that could not be read up front). - **Split by capability.** `k8s-root-actor` can query k8s node capacity directly, but has no SLURM access — `sinfo`/`squeue` are only reachable from inside the launcher pod, so the SLURM checks live in the submit script. ## Consequences - Users and operators can distinguish a transient resource wait (`queued`) from a request that will never run (`failed`), each with a human-readable reason. - Downstream consumers that render workload status (notably the `web` service) must handle the new `queued` value; see the DD's downstream note. - Capacity queries are best-effort: when node/partition capacity cannot be read, the pre-submission gate is skipped and the runtime path keeps the workload `queued` rather than wrongly failing it. ## Related - [ADR-0012: Job and Deployment Management System](/adr/0012-job-deployment-management-system.md) - [ADR-0007: GPU Slicing and Quota Enforcement](/adr/0007-gpu-slicing-and-quota-enforcement.md) - [DD-0018: Queued Status and Over-Capacity Handling — Implementation](/dd/0018-queued-status-and-over-capacity-handling.md)
Page source Source snapshot · Revision `f928834c8`.