Themelio compatibility reference. This page describes the retained Themelio implementation or its historical design. Check the parallel-operation guide before applying it to the current platform.

#DD-0018: Queued Status and Over-Capacity Handling — Implementation

Related ADR: ADR-0018: Queued Status and Over-Capacity Handling for Managed Workloads

Status: Draft

Date: 2026-07-02

Authors: TBD

#Overview

This document specifies how k8s-root-actor implements the transient queued status and over-capacity → failed handling decided in ADR-0018, across k8s Deployments, KServe InferenceServices, and SLURM jobs.

#1. Status model (src/types/db.ts)

queued is added to both JOB_STATUSES and DEPLOYMENT_STATUSES, positioned after pending. It is active: it is absent from INACTIVE_JOB_STATUSES / INACTIVE_DEPLOYMENT_STATUSES, so isActive*Status('queued') is true and the row keeps its slug under the (slug) WHERE active partial-unique index. It is also absent from the *_SETTLED_STATUSES lists in src/routes/actions.ts, so the readiness watcher keeps watching a queued workload until it runs or fails.

mapSourceStatus() gains two rules (mirroring the existing endsWith('_FAILED') convention):

• SLURM_JOB_SUBMITTED → queued (previously running; a freshly submitted job is not yet running). The subsequent RUNNING poll (SLURM_JOB_STATUS) moves it to running.
• any *_QUEUED token → queued (level warning). This single rule covers both SLURM_JOB_QUEUED and the k8s POD_QUEUED token.

No DB migration is required: the status columns are plain text with no CHECK constraint.

#2. Token vocabulary added

#3. SLURM detection (src/templates/slurm-job/submit-script.sh.hbs)

Both checks run inside the launcher pod, where sinfo/squeue are reachable.

• Pre-submission gate (precheck_capacity, before sbatch): reads CPUs (%c), memory (%m, MB), and GPUs (parsed from the %G GRES column) per node for the target partition (ml for GPU jobs, else the default partition), then checks feasibility per node as a conjunction — a node counts only if it satisfies the CPU, RAM, and GPU request all at once. If no single node satisfies the whole request (including the case where each resource fits some node but none fits all) it emits SLURM_JOB_SUBMISSION_FAILED and exits (maps to failed). Best-effort: skipped silently if sinfo is unavailable/unparseable.
• Runtime reason (poll loop): when sacct reports PENDING, get_slurm_reason reads squeue -h -j <id> -o '%r' and emits SLURM_JOB_QUEUED with the reason, deduped by reason. Other states keep emitting SLURM_JOB_STATUS.

#4. k8s detection (deployments + inference)

Both share the readiness watcher (kind: 'deployment') and the actions.ts dispatch; the InferenceService watch targets the KServe predictor Deployment.

(a) Pre-submission gate — validateGpuCapacity() in src/routes/actions.ts, run after the row is created and before the manifest is applied. It compares the GPU request against kubectlGetMaxNodeGpuAllocatable(gpuLabel) (max status.allocatable[<gpuLabel>] across nodes; gpuLabel is the k8s resource name). On over-capacity the row is marked failed via markDeploymentFailed(), a CAPACITY_EXCEEDED log is appended, and the apply is skipped. The scope is GPU requests — the primary scarce resource; CPU/memory over-capacity is left to the runtime backstop. kubectlGetMaxNodeGpuAllocatable returns null (not 0) when capacity can't be read or no node advertises the resource, so the gate is skipped rather than wrongly failing a request.

(b) Runtime backstop — watchK8sReadiness() in src/lib/action-readiness-watch.ts, covering two runtime failure modes:

• Quota block — a pod rejected at admission by the namespace ResourceQuota (exceeded quota: waldur-quota), surfaced via the Deployment's ReplicaFailure condition. Because the quota is a fixed policy cap that waiting won't lift, this is failed immediately (no grace period): for a deployment/ISVC the watcher emits POD_QUOTA_EXCEEDED_FAILED (→ failed) and stops. (Jobs have no onStatusEvent channel, so for them a block is only logged as POD_CREATION_BLOCKED.)
• Unschedulable pod — kubectlGetPodSchedulingFailure() (resolve the Deployment's pod selector → find a pod with PodScheduled=False, reason=Unschedulable). Classified via the injected gpuRequest: a request exceeding max node capacity emits POD_SCHEDULING_FAILED (→ failed, watcher stops); otherwise POD_QUEUED (→ queued, keeps watching).

The status-bearing tokens are routed through the injected onStatusEvent, which the caller wires to applyDeploymentNotificationWithTeardown().

Shared teardown — applyDeploymentNotificationWithTeardown() in src/services/deployments.ts records the notification and, if it flipped an active deployment to an inactive status (e.g. failed), tears down the orphaned k8s resources in the background. It is used by both the HTTP notification route (src/routes/deployments.ts) and the readiness watcher, so a failed verdict from either path behaves identically (extracted from the route's previous inline logic).

#5. Status flow

pending → queued → preparing → running     k8s: capacity busy, then schedules
pending → preparing → queued → running     SLURM: setup+pre-cmds, then PENDING, then runs
pending → queued → failed                  over-capacity detected at runtime
pending → preparing → failed               k8s: quota block detected at runtime (fail-fast + teardown)
pending → failed                           over-capacity caught pre-submission

Jobs also share the preparing status: the SLURM init/repo/pre-command tokens map to preparing (previously running), so a job is never running until SLURM reports it executing (SLURM_JOB_STATUS). Because that setup runs on the login node before sbatch, preparing precedes queued for jobs (the reverse of k8s, where scheduling precedes the init container). The capacity gate runs before any setup, so an over-capacity job goes pending → failed without entering preparing.

#6. Downstream note

The web service renders workload status and may keep its own status list/label map. It must be updated to label queued (and not fall through to a default). This is tracked separately and is out of scope for the k8s-root-actor change.

#7. Verification

• pnpm type-check and pnpm build in services/k8s-root-actor.
• bash -n on the rendered SLURM submit script confirms the shell additions parse.
• Static read-through of actions.ts confirms over-capacity creates the row then failed (record + CAPACITY_EXCEEDED log preserved) and skips the apply, and that the watcher's failed verdict triggers teardown via the shared helper.

No unit-test suite exists in k8s-root-actor (no test framework is configured); verification is via the type-checker, the build, and the shell syntax check above.