> For AI agents: the complete documentation index is available at /llms.txt, the full documentation bundle is available at /llms-full.txt. > **Themelio compatibility reference.** This page describes the retained Themelio implementation or its historical design. Check the [parallel-operation guide](/compatibility/index.md) before applying it to the current platform. # DD-0018: Queued Status and Over-Capacity Handling — Implementation **Related ADR:** [ADR-0018: Queued Status and Over-Capacity Handling for Managed Workloads](/adr/0018-queued-status-and-over-capacity-handling.md) **Status:** Draft **Date:** 2026-07-02 **Authors:** TBD ## Overview This document specifies how `k8s-root-actor` implements the transient `queued` status and over-capacity → `failed` handling decided in ADR-0018, across k8s Deployments, KServe InferenceServices, and SLURM jobs. ## 1. Status model (`src/types/db.ts`) `queued` is added to both `JOB_STATUSES` and `DEPLOYMENT_STATUSES`, positioned after `pending`. It is **active**: it is absent from `INACTIVE_JOB_STATUSES` / `INACTIVE_DEPLOYMENT_STATUSES`, so `isActive*Status('queued')` is `true` and the row keeps its slug under the `(slug) WHERE active` partial-unique index. It is also absent from the `*_SETTLED_STATUSES` lists in `src/routes/actions.ts`, so the readiness watcher keeps watching a queued workload until it runs or fails. `mapSourceStatus()` gains two rules (mirroring the existing `endsWith('_FAILED')` convention): - `SLURM_JOB_SUBMITTED` → `queued` (previously `running`; a freshly submitted job is not yet running). The subsequent `RUNNING` poll (`SLURM_JOB_STATUS`) moves it to `running`. - any `*_QUEUED` token → `queued` (level `warning`). This single rule covers both `SLURM_JOB_QUEUED` and the k8s `POD_QUEUED` token. No DB migration is required: the `status` columns are plain text with no `CHECK` constraint. ## 2. Token vocabulary added | Token | Emitted by | Maps to | | ----------------------------- | ------------------------------------------------------------------------ | ----------------------------------- | | `SLURM_JOB_QUEUED` | SLURM submit script (poll loop, `PENDING`) | `queued` | | `SLURM_JOB_SUBMISSION_FAILED` | SLURM submit script (`precheck_capacity`, pre-`sbatch`) | `failed` (via `*_FAILED`) | | `POD_QUEUED` | readiness watcher (unschedulable, transient) | `queued` | | `POD_SCHEDULING_FAILED` | readiness watcher (request exceeds node capacity) | `failed` (via `*_FAILED`) | | `POD_QUOTA_EXCEEDED_FAILED` | readiness watcher (deployment/ISVC blocked by namespace `ResourceQuota`) | `failed` (via `*_FAILED`) | | `CAPACITY_EXCEEDED` | action POST (pre-submission gate, log only) | — (status set directly to `failed`) | ## 3. SLURM detection (`src/templates/slurm-job/submit-script.sh.hbs`) Both checks run inside the launcher pod, where `sinfo`/`squeue` are reachable. - **Pre-submission gate** (`precheck_capacity`, before `sbatch`): reads CPUs (`%c`), memory (`%m`, MB), and GPUs (parsed from the `%G` GRES column) **per node** for the target partition (`ml` for GPU jobs, else the default partition), then checks feasibility per node as a conjunction — a node counts only if it satisfies the CPU, RAM, and GPU request all at once. If no single node satisfies the whole request (including the case where each resource fits some node but none fits all) it emits `SLURM_JOB_SUBMISSION_FAILED` and exits (maps to `failed`). Best-effort: skipped silently if `sinfo` is unavailable/unparseable. - **Runtime reason** (poll loop): when `sacct` reports `PENDING`, `get_slurm_reason` reads `squeue -h -j -o '%r'` and emits `SLURM_JOB_QUEUED` with the reason, deduped by reason. Other states keep emitting `SLURM_JOB_STATUS`. ## 4. k8s detection (deployments + inference) Both share the readiness watcher (`kind: 'deployment'`) and the `actions.ts` dispatch; the InferenceService watch targets the KServe predictor Deployment. **(a) Pre-submission gate** — `validateGpuCapacity()` in `src/routes/actions.ts`, run after the row is created and before the manifest is applied. It compares the GPU request against `kubectlGetMaxNodeGpuAllocatable(gpuLabel)` (max `status.allocatable[]` across nodes; `gpuLabel` is the k8s resource name). On over-capacity the row is marked `failed` via `markDeploymentFailed()`, a `CAPACITY_EXCEEDED` log is appended, and the apply is skipped. The scope is GPU requests — the primary scarce resource; CPU/memory over-capacity is left to the runtime backstop. `kubectlGetMaxNodeGpuAllocatable` returns `null` (not `0`) when capacity can't be read or no node advertises the resource, so the gate is skipped rather than wrongly failing a request. **(b) Runtime backstop** — `watchK8sReadiness()` in `src/lib/action-readiness-watch.ts`, covering two runtime failure modes: - **Quota block** — a pod rejected at admission by the namespace `ResourceQuota` (`exceeded quota: waldur-quota`), surfaced via the Deployment's `ReplicaFailure` condition. Because the quota is a fixed policy cap that waiting won't lift, this is failed immediately (no grace period): for a deployment/ISVC the watcher emits `POD_QUOTA_EXCEEDED_FAILED` (→ `failed`) and stops. (Jobs have no `onStatusEvent` channel, so for them a block is only logged as `POD_CREATION_BLOCKED`.) - **Unschedulable pod** — `kubectlGetPodSchedulingFailure()` (resolve the Deployment's pod selector → find a pod with `PodScheduled=False, reason=Unschedulable`). Classified via the injected `gpuRequest`: a request exceeding max node capacity emits `POD_SCHEDULING_FAILED` (→ `failed`, watcher stops); otherwise `POD_QUEUED` (→ `queued`, keeps watching). The status-bearing tokens are routed through the injected `onStatusEvent`, which the caller wires to `applyDeploymentNotificationWithTeardown()`. **Shared teardown** — `applyDeploymentNotificationWithTeardown()` in `src/services/deployments.ts` records the notification and, if it flipped an active deployment to an inactive status (e.g. `failed`), tears down the orphaned k8s resources in the background. It is used by **both** the HTTP notification route (`src/routes/deployments.ts`) and the readiness watcher, so a `failed` verdict from either path behaves identically (extracted from the route's previous inline logic). ## 5. Status flow ``` pending → queued → preparing → running k8s: capacity busy, then schedules pending → preparing → queued → running SLURM: setup+pre-cmds, then PENDING, then runs pending → queued → failed over-capacity detected at runtime pending → preparing → failed k8s: quota block detected at runtime (fail-fast + teardown) pending → failed over-capacity caught pre-submission ``` Jobs also share the `preparing` status: the SLURM init/repo/pre-command tokens map to `preparing` (previously `running`), so a job is never `running` until SLURM reports it executing (`SLURM_JOB_STATUS`). Because that setup runs on the login node before `sbatch`, `preparing` precedes `queued` for jobs (the reverse of k8s, where scheduling precedes the init container). The capacity gate runs before any setup, so an over-capacity job goes `pending → failed` without entering `preparing`. ## 6. Downstream note The `web` service renders workload status and may keep its own status list/label map. It must be updated to label `queued` (and not fall through to a default). This is tracked separately and is out of scope for the `k8s-root-actor` change. ## 7. Verification - `pnpm type-check` and `pnpm build` in `services/k8s-root-actor`. - `bash -n` on the rendered SLURM submit script confirms the shell additions parse. - Static read-through of `actions.ts` confirms over-capacity creates the row then `failed` (record + `CAPACITY_EXCEEDED` log preserved) and skips the apply, and that the watcher's `failed` verdict triggers teardown via the shared helper. _No unit-test suite exists in `k8s-root-actor` (no test framework is configured); verification is via the type-checker, the build, and the shell syntax check above._
Page source Source snapshot · Revision `f928834c8`.