For AI agents: the complete documentation index is available at /llms.txt, the full documentation bundle is available at /llms-full.txt, and this page is available as Markdown at /reference/docs/decisions/0018-queued-status-and-over-capacity-handling.md.

ADR-0018: Queued Status and Over-Capacity Handling for Managed Workloads

Status: Proposed

Date: 2026-07-02

Deciders: TBD

Context

k8s-root-actor manages three workload kinds — k8s Deployments, KServe InferenceServices, and SLURM jobs — each tracked by a lifecycle status (see ADR-0012 and DD-0012). Until now the only "not yet running" status was pending, which conflated two situations:

  1. The record was just created, before the k8s resources were applied or the SLURM job submitted.
  2. The workload is submitted but cannot start yet because the resources it asked for are currently in use by others — a transient wait that self-heals.

Worse, an over-capacity request — asking for more CPU/GPU/memory than the cluster or SLURM partition can ever provide — was not handled at all:

  • A k8s pod requesting e.g. 40 GPUs when the largest node has 8 lands Pending with PodScheduled=False, reason=Unschedulable. The pod's init script never runs, so no notification callback fires. The existing readiness watcher only detects Waldur quota blocks (ReplicaFailure), not scheduling shortfalls, so the resource sat in pending indefinitely with no explanation.
  • A SLURM job requesting more than the partition can offer stays PENDING. The submit script polled only the sacct state, never the pending reason, so again nothing surfaced.

Operators and users could not distinguish "waiting for the cluster to free up" from "this will never run", and the second case looked identical to a healthy pending.

Decision

Introduce a single new transient lifecycle status, queued, meaning "submitted, waiting for resources to be freed", for all three workload kinds. An over-capacity request is surfaced as the existing terminal failed status with a descriptive message (e.g. "requested 40 GPUs, but the largest node provides 8") — no separate terminal status is added.

Detection is done both before submission and at runtime, split by where each capability is reachable:

KindPre-submission over-capacity checkRuntime detection
k8s Deployment / InferenceServiceAt the action POST: query node allocatable via kubectl get nodes. A GPU request larger than the max single-node allocatable is recorded as failed before the manifest is applied.Readiness watcher detects an Unschedulable pod → queued, or failed when the request exceeds node capacity.
SLURM jobInside the launcher pod, before sbatch: sinfo partition limits. A request no node can satisfy fails fast.The poll loop reads the PENDING reason (squeue/scontrol) → queued.

queued is an active status (it still occupies the workload's slug, so the (slug) WHERE active uniqueness still holds) and is transient — the readiness watcher keeps watching until the workload runs or fails.

Rationale

  • One new status, not two. A distinct terminal rejected status was considered and rejected: over-capacity is a failure like any other apply-time failure, and reusing failed (with a clear message) keeps the state machine small and existing consumers unchanged.
  • Over-capacity → failed with a record, not a bare HTTP error. The row is created and then marked failed (with a log line), so the attempt is visible in history exactly like a runtime failure, rather than vanishing as a 4xx.
  • Both detection paths. The pre-submission check gives fast, clear feedback for the obvious cases; the runtime check is the backstop for conditions that only appear at schedule time (and for capacity that could not be read up front).
  • Split by capability. k8s-root-actor can query k8s node capacity directly, but has no SLURM access — sinfo/squeue are only reachable from inside the launcher pod, so the SLURM checks live in the submit script.

Consequences

  • Users and operators can distinguish a transient resource wait (queued) from a request that will never run (failed), each with a human-readable reason.
  • Downstream consumers that render workload status (notably the web service) must handle the new queued value; see the DD's downstream note.
  • Capacity queries are best-effort: when node/partition capacity cannot be read, the pre-submission gate is skipped and the runtime path keeps the workload queued rather than wrongly failing it.
Page source

Source snapshot · Revision f928834c8.