---
title: "DD-0018: Queued Status and Over-Capacity Handling — Implementation"
---

# DD-0018: Queued Status and Over-Capacity Handling — Implementation

**Related ADR:** [ADR-0018: Queued Status and Over-Capacity Handling for Managed Workloads](/adr/0018-queued-status-and-over-capacity-handling)

**Status:** Draft

**Date:** 2026-07-02

**Authors:** TBD

## Overview

This document specifies how `k8s-root-actor` implements the transient `queued`
status and over-capacity → `failed` handling decided in ADR-0018, across k8s
Deployments, KServe InferenceServices, and SLURM jobs.

## 1. Status model (`src/types/db.ts`)

`queued` is added to both `JOB_STATUSES` and `DEPLOYMENT_STATUSES`, positioned
after `pending`. It is **active**: it is absent from `INACTIVE_JOB_STATUSES` /
`INACTIVE_DEPLOYMENT_STATUSES`, so `isActive*Status('queued')` is `true` and the
row keeps its slug under the `(slug) WHERE active` partial-unique index. It is
also absent from the `*_SETTLED_STATUSES` lists in `src/routes/actions.ts`, so the
readiness watcher keeps watching a queued workload until it runs or fails.

`mapSourceStatus()` gains two rules (mirroring the existing `endsWith('_FAILED')`
convention):

- `SLURM_JOB_SUBMITTED` → `queued` (previously `running`; a freshly submitted job
  is not yet running). The subsequent `RUNNING` poll (`SLURM_JOB_STATUS`) moves it
  to `running`.
- any `*_QUEUED` token → `queued` (level `warning`). This single rule covers both
  `SLURM_JOB_QUEUED` and the k8s `POD_QUEUED` token.

No DB migration is required: the `status` columns are plain text with no `CHECK`
constraint.

## 2. Token vocabulary added

| Token | Emitted by | Maps to |
|-------|-----------|---------|
| `SLURM_JOB_QUEUED` | SLURM submit script (poll loop, `PENDING`) | `queued` |
| `SLURM_JOB_SUBMISSION_FAILED` | SLURM submit script (`precheck_capacity`, pre-`sbatch`) | `failed` (via `*_FAILED`) |
| `POD_QUEUED` | readiness watcher (unschedulable, transient) | `queued` |
| `POD_SCHEDULING_FAILED` | readiness watcher (request exceeds node capacity) | `failed` (via `*_FAILED`) |
| `POD_QUOTA_EXCEEDED_FAILED` | readiness watcher (deployment/ISVC blocked by namespace `ResourceQuota`) | `failed` (via `*_FAILED`) |
| `CAPACITY_EXCEEDED` | action POST (pre-submission gate, log only) | — (status set directly to `failed`) |

## 3. SLURM detection (`src/templates/slurm-job/submit-script.sh.hbs`)

Both checks run inside the launcher pod, where `sinfo`/`squeue` are reachable.

- **Pre-submission gate** (`precheck_capacity`, before `sbatch`): reads CPUs (`%c`),
  memory (`%m`, MB), and GPUs (parsed from the `%G` GRES column) **per node** for the
  target partition (`ml` for GPU jobs, else the default partition), then checks
  feasibility per node as a conjunction — a node counts only if it satisfies the CPU,
  RAM, and GPU request all at once. If no single node satisfies the whole request
  (including the case where each resource fits some node but none fits all) it emits
  `SLURM_JOB_SUBMISSION_FAILED` and exits (maps to `failed`). Best-effort: skipped
  silently if `sinfo` is unavailable/unparseable.
- **Runtime reason** (poll loop): when `sacct` reports `PENDING`, `get_slurm_reason`
  reads `squeue -h -j <id> -o '%r'` and emits `SLURM_JOB_QUEUED` with the reason,
  deduped by reason. Other states keep emitting `SLURM_JOB_STATUS`.

## 4. k8s detection (deployments + inference)

Both share the readiness watcher (`kind: 'deployment'`) and the `actions.ts`
dispatch; the InferenceService watch targets the KServe predictor Deployment.

**(a) Pre-submission gate** — `validateGpuCapacity()` in `src/routes/actions.ts`,
run after the row is created and before the manifest is applied. It compares the
GPU request against `kubectlGetMaxNodeGpuAllocatable(gpuLabel)` (max
`status.allocatable[<gpuLabel>]` across nodes; `gpuLabel` is the k8s resource
name). On over-capacity the row is marked `failed` via `markDeploymentFailed()`, a
`CAPACITY_EXCEEDED` log is appended, and the apply is skipped. The scope is GPU
requests — the primary scarce resource; CPU/memory over-capacity is left to the
runtime backstop. `kubectlGetMaxNodeGpuAllocatable` returns `null` (not `0`) when
capacity can't be read or no node advertises the resource, so the gate is skipped
rather than wrongly failing a request.

**(b) Runtime backstop** — `watchK8sReadiness()` in
`src/lib/action-readiness-watch.ts`, covering two runtime failure modes:

- **Quota block** — a pod rejected at admission by the namespace `ResourceQuota`
  (`exceeded quota: waldur-quota`), surfaced via the Deployment's `ReplicaFailure`
  condition. Because the quota is a fixed policy cap that waiting won't lift, this
  is failed immediately (no grace period): for a deployment/ISVC the watcher emits
  `POD_QUOTA_EXCEEDED_FAILED` (→ `failed`) and stops. (Jobs have no `onStatusEvent`
  channel, so for them a block is only logged as `POD_CREATION_BLOCKED`.)
- **Unschedulable pod** — `kubectlGetPodSchedulingFailure()` (resolve the
  Deployment's pod selector → find a pod with `PodScheduled=False,
  reason=Unschedulable`). Classified via the injected `gpuRequest`: a request
  exceeding max node capacity emits `POD_SCHEDULING_FAILED` (→ `failed`, watcher
  stops); otherwise `POD_QUEUED` (→ `queued`, keeps watching).

The status-bearing tokens are routed through the injected `onStatusEvent`, which
the caller wires to `applyDeploymentNotificationWithTeardown()`.

**Shared teardown** — `applyDeploymentNotificationWithTeardown()` in
`src/services/deployments.ts` records the notification and, if it flipped an active
deployment to an inactive status (e.g. `failed`), tears down the orphaned k8s
resources in the background. It is used by **both** the HTTP notification route
(`src/routes/deployments.ts`) and the readiness watcher, so a `failed` verdict from
either path behaves identically (extracted from the route's previous inline logic).

## 5. Status flow

```
pending → queued → preparing → running     k8s: capacity busy, then schedules
pending → preparing → queued → running     SLURM: setup+pre-cmds, then PENDING, then runs
pending → queued → failed                  over-capacity detected at runtime
pending → preparing → failed               k8s: quota block detected at runtime (fail-fast + teardown)
pending → failed                           over-capacity caught pre-submission
```

Jobs also share the `preparing` status: the SLURM init/repo/pre-command tokens map
to `preparing` (previously `running`), so a job is never `running` until SLURM
reports it executing (`SLURM_JOB_STATUS`). Because that setup runs on the login node
before `sbatch`, `preparing` precedes `queued` for jobs (the reverse of k8s, where
scheduling precedes the init container). The capacity gate runs before any setup, so
an over-capacity job goes `pending → failed` without entering `preparing`.

## 6. Downstream note

The `web` service renders workload status and may keep its own status list/label
map. It must be updated to label `queued` (and not fall through to a default).
This is tracked separately and is out of scope for the `k8s-root-actor` change.

## 7. Verification

- `pnpm type-check` and `pnpm build` in `services/k8s-root-actor`.
- `bash -n` on the rendered SLURM submit script confirms the shell additions parse.
- Static read-through of `actions.ts` confirms over-capacity creates the row then
  `failed` (record + `CAPACITY_EXCEEDED` log preserved) and skips the apply, and
  that the watcher's `failed` verdict triggers teardown via the shared helper.

_No unit-test suite exists in `k8s-root-actor` (no test framework is configured);
verification is via the type-checker, the build, and the shell syntax check above._