---
title: "ADR-0018: Queued Status and Over-Capacity Handling for Managed Workloads"
---

# ADR-0018: Queued Status and Over-Capacity Handling for Managed Workloads

**Status:** Proposed

**Date:** 2026-07-02

**Deciders:** TBD

## Context

`k8s-root-actor` manages three workload kinds — k8s Deployments, KServe
InferenceServices, and SLURM jobs — each tracked by a lifecycle status
(see [ADR-0012](/adr/0012-job-deployment-management-system) and
[DD-0012](/dd/0012-job-deployment-management-system)). Until now the only
"not yet running" status was `pending`, which conflated two situations:

1. **The record was just created**, before the k8s resources were applied or the
   SLURM job submitted.
2. **The workload is submitted but cannot start yet** because the resources it
   asked for are currently in use by others — a transient wait that self-heals.

Worse, an **over-capacity request** — asking for more CPU/GPU/memory than the
cluster or SLURM partition can *ever* provide — was not handled at all:

- A k8s pod requesting e.g. 40 GPUs when the largest node has 8 lands `Pending`
  with `PodScheduled=False, reason=Unschedulable`. The pod's init script never
  runs, so no notification callback fires. The existing readiness watcher only
  detects Waldur *quota* blocks (`ReplicaFailure`), not scheduling shortfalls, so
  the resource sat in `pending` indefinitely with no explanation.
- A SLURM job requesting more than the partition can offer stays `PENDING`. The
  submit script polled only the `sacct` *state*, never the pending *reason*, so
  again nothing surfaced.

Operators and users could not distinguish "waiting for the cluster to free up"
from "this will never run", and the second case looked identical to a healthy
`pending`.

## Decision

Introduce a single new transient lifecycle status, **`queued`**, meaning
"submitted, waiting for resources to be freed", for all three workload kinds. An
**over-capacity request** is surfaced as the existing terminal **`failed`** status
with a descriptive message (e.g. *"requested 40 GPUs, but the largest node
provides 8"*) — no separate terminal status is added.

Detection is done **both** before submission and at runtime, split by where each
capability is reachable:

| Kind | Pre-submission over-capacity check | Runtime detection |
|------|-----------------------------------|-------------------|
| k8s Deployment / InferenceService | At the action POST: query node allocatable via `kubectl get nodes`. A GPU request larger than the max single-node allocatable is recorded as `failed` before the manifest is applied. | Readiness watcher detects an `Unschedulable` pod → `queued`, or `failed` when the request exceeds node capacity. |
| SLURM job | Inside the launcher pod, before `sbatch`: `sinfo` partition limits. A request no node can satisfy fails fast. | The poll loop reads the `PENDING` reason (`squeue`/`scontrol`) → `queued`. |

`queued` is an **active** status (it still occupies the workload's slug, so the
`(slug) WHERE active` uniqueness still holds) and is transient — the readiness
watcher keeps watching until the workload runs or fails.

### Rationale

- **One new status, not two.** A distinct terminal `rejected` status was
  considered and rejected: over-capacity is a failure like any other apply-time
  failure, and reusing `failed` (with a clear message) keeps the state machine
  small and existing consumers unchanged.
- **Over-capacity → `failed` with a record, not a bare HTTP error.** The row is
  created and then marked `failed` (with a log line), so the attempt is visible
  in history exactly like a runtime failure, rather than vanishing as a 4xx.
- **Both detection paths.** The pre-submission check gives fast, clear feedback
  for the obvious cases; the runtime check is the backstop for conditions that
  only appear at schedule time (and for capacity that could not be read up front).
- **Split by capability.** `k8s-root-actor` can query k8s node capacity directly,
  but has no SLURM access — `sinfo`/`squeue` are only reachable from inside the
  launcher pod, so the SLURM checks live in the submit script.

## Consequences

- Users and operators can distinguish a transient resource wait (`queued`) from a
  request that will never run (`failed`), each with a human-readable reason.
- Downstream consumers that render workload status (notably the `web` service)
  must handle the new `queued` value; see the DD's downstream note.
- Capacity queries are best-effort: when node/partition capacity cannot be read,
  the pre-submission gate is skipped and the runtime path keeps the workload
  `queued` rather than wrongly failing it.

## Related

- [ADR-0012: Job and Deployment Management System](/adr/0012-job-deployment-management-system)
- [ADR-0007: GPU Slicing and Quota Enforcement](/adr/0007-gpu-slicing-and-quota-enforcement)
- [DD-0018: Queued Status and Over-Capacity Handling — Implementation](/dd/0018-queued-status-and-over-capacity-handling)