Themelio compatibility reference. This page describes the retained Themelio implementation or its historical design. Check the parallel-operation guide before applying it to the current platform.

#DD-0023: DRA GPU Extended Resource Implementation

Related ADR: ADR-0023: DRA GPU Allocation with Extended Resource Names

Status: Draft

Date: 2026-07-09

Authors: TBD

#ADR-derived technical notes

These notes scope the implementation of ADR-0023. The decision is to adopt NVIDIA DRA through Kubernetes extended resource names, without changing the Themelio action API or introducing explicit ResourceClaim objects into user-facing workflows.

• Config shape stays unchanged: THEMELIO_GPU_LABELS_CONFIG remains a YAML document with a top-level gpu_labels map. No gpu_resources schema is introduced by this DD.
• Correctness comes from labels: each gpu_labels key is the Kubernetes extended resource name rendered into workload resources.requests and resources.limits.
• Target labels are differentiated: use product-specific names such as nvidia.com/a10g, nvidia.com/a100, nvidia.com/a100-mig-1g.5gb, and nvidia.com/a100-mig-3g.20gb instead of collapsing multiple products into nvidia.com/gpu.
• Known mismatch to fix: gpu_a10g currently maps inconsistently: the web mapping uses nvidia.com/gpu, while the Waldur site-agent template uses requests.nvidiax.com/gpu. The target implementation must use one corrected catalog label, nvidia.com/a10g, and its quota key requests.nvidia.com/a10g.
• Cluster prerequisite: the current Kubespray defaults in this repository resolve to Kubernetes 1.32.8, which is too old for the ADR-0023 DRA extended resource path. DRA rollout therefore requires a Kubernetes upgrade before cluster enablement.

#Overview

Themelio keeps the existing action contract:

options:
  gpus: 1
  gpuLabel: nvidia.com/a100

k8s-root-actor continues to render gpuLabel into the workload container's Kubernetes resource requests and limits. DRA remains an infrastructure-backed implementation detail: DRA DeviceClass objects advertise matching extendedResourceName values, and Kubernetes translates extended resource requests into DRA allocation internally when DRAExtendedResource is available.

The implementation work is therefore a label-alignment effort across the existing config consumers, not a new GPU catalog schema and not a new workload rendering model.

#Goals and Non-Goals

#Goals

• Preserve THEMELIO_GPU_LABELS_CONFIG as the deployed config surface.
• Preserve the top-level YAML key gpu_labels.
• Preserve options.gpus plus options.gpuLabel in actions.
• Preserve workload GPU allocation through Kubernetes resources.requests and resources.limits.
• Align Themelio, Waldur, Kubeflow, and DRA DeviceClass extended resource names around the same corrected GPU labels.
• Replace generic or incorrect GPU labels with differentiated extended resource names where Themelio exposes distinct GPU offerings.

#Non-Goals

• Introducing gpu_resources or another richer catalog schema.
• Exposing ResourceClaim or ResourceClaimTemplate fields to users.
• Reworking Slurm GPU allocation; Slurm actions continue to use --gres=gpu.
• Designing DRA CEL selectors in this DD without validating actual NVIDIA ResourceSlice attributes in a target cluster.

#Design

#1. GPU label configuration

The generated ConfigMap data remains:

gpu_labels:
  nvidia.com/a100:
    node_selector:
      accelerator: "nvidia-a100"
    mig_enabled: "false"

  nvidia.com/a100-mig-1g.5gb:
    node_selector:
      accelerator: "nvidia-a100"
    mig_enabled: "true"

The implementation updates only the map keys and metadata values. Existing parsers should continue to require:

• a non-empty top-level gpu_labels map,
• a non-empty node_selector map for each entry, and
• string mig_enabled metadata.

The minimum target label set is:

The A10G mapping must not keep nvidiax.com. That value is treated as a typo or legacy mismatch, not as a migration alias.

#2. Application consumers

k8s-root-actor remains the authority for action execution:

• GPU_LABELS_CONFIG is parsed from THEMELIO_GPU_LABELS_CONFIG.
• options.gpuLabel is valid only when it is a key of gpu_labels.
• options.gpuLabel is required when options.gpus is greater than zero for k8s-deployment and k8s-inferenceservice.
• Deployment and InferenceService templates keep rendering the selected label in both resources.requests and resources.limits.
• The node selector still comes from the selected gpu_labels entry.

The web service must use the same corrected labels when showing the copyable action GPU label for Waldur GPU components. Component mapping should be:

OpenAPI and product documentation must describe the corrected labels and avoid a stale enum that lists only nvidia.com/gpu, nvidia.com/mig-1g.5gb, and nvidia.com/mig-3g.20gb.

#3. Infrastructure consumers

Ansible remains responsible for generating and applying the config files used by Themelio, Waldur, and Kubeflow:

• themelio_gpu_labels defines the gpu_labels ConfigMap consumed by web and k8s-root-actor.
• The Waldur site-agent quota mapping uses requests.<gpuLabel> for each GPU component.
• Kubeflow vendor configuration uses the same label as limitsKey and a user-facing name that matches the offering.

The DRA infrastructure must advertise matching extended resource names. For each Themelio GPU label, the corresponding DRA DeviceClass must set spec.extendedResourceName to that exact label.

DeviceClass selectors must be validated against actual NVIDIA DRA ResourceSlice attributes in the target cluster before rollout. This DD does not prescribe unverified CEL attribute names.

#4. DRA rollout prerequisites

DRA rollout is blocked until the cluster supports extended resource allocation by DRA.

Target Kubernetes version:

• Prefer Kubernetes 1.36+, where DRAExtendedResource is beta and enabled by default.
• Kubernetes 1.34 or 1.35 may be used only if DynamicResourceAllocation and DRAExtendedResource are explicitly enabled on the required control-plane components and kubelets.
• Kubernetes 1.32.x is not acceptable for this rollout.

GPU node and operator requirements:

• Label every DRA-capable GPU node with nvidia.com/dra-kubelet-plugin=true.
• Configure the NVIDIA GPU Operator for DRA-backed GPU allocation, including disabling the traditional NVIDIA device plugin when switching a node pool to DRA allocation.
• Install the NVIDIA DRA driver with GPU allocation enabled and the kubelet plugin node selector set to nvidia.com/dra-kubelet-plugin=true.
• For MIG-enabled A100 nodes, apply MIG configuration before validating DRA ResourceSlice and DeviceClass behavior; if the DRA kubelet plugin was already running during MIG changes, restart it before validation.

#Testing Strategy

#Unit and template tests

• Config parser accepts the unchanged gpu_labels structure with corrected labels.
• Config parser rejects missing or empty gpu_labels, empty node_selector, and unsupported gpuLabel values.
• k8s-root-actor validates updated valid labels and rejects stale or invalid labels.
• Deployment and InferenceService rendering tests confirm the selected gpuLabel appears in both resources.requests and resources.limits.
• Web tests confirm Waldur GPU components expose the corrected action gpuLabel, including gpu_a10g -> nvidia.com/a10g.
• Ansible/template tests confirm Waldur quota mapping and Kubeflow vendors use the same corrected labels.

#Cluster validation

Before enabling the corrected labels for users, validate the target cluster:

kubectl get deviceclass
kubectl get resourceslice -o wide
kubectl describe deviceclass <deviceclass-name>
kubectl describe node -l nvidia.com/dra-kubelet-plugin=true

Then run one Themelio k8s-deployment or k8s-inferenceservice action for each enabled GPU label and confirm:

• the generated pod requests the corrected extended resource name,
• quota accounting uses requests.<gpuLabel>,
• the workload schedules and receives GPU access,
• Kubeflow displays the same label where notebook GPU selection is enabled, and
• no Themelio workflow exposes or requires explicit ResourceClaim objects.

#Deployment and Rollout

1. Upgrade Kubernetes to a version that supports the desired DRAExtendedResource behavior; use 1.36+ as the default target.
2. Deploy or update NVIDIA GPU Operator and NVIDIA DRA driver configuration.
3. Label DRA-capable GPU nodes with nvidia.com/dra-kubelet-plugin=true.
4. Create or update DRA DeviceClass resources so their extendedResourceName values match the gpu_labels keys.
5. Update Ansible GPU label defaults, Waldur quota mapping, Kubeflow vendors, web GPU label mapping, OpenAPI, and actions-yaml documentation in one coordinated change.
6. Apply Themelio configuration and restart GPU-label consumers when the ConfigMap changes.
7. Validate with the test plan above before announcing the corrected labels to users.

Rollback is to restore the previous gpu_labels, Waldur quota mapping, Kubeflow vendors, and web copy-label mapping, then restart Themelio GPU-label consumers. Rollback does not require changing user action schema because the schema is unchanged.

#Open Questions

• Which target Kubernetes version will the first DRA-enabled Themelio cluster use: 1.36+, or 1.34/1.35 with explicit feature gates?
• Which exact DRA DeviceClass objects and CEL selectors should represent A10G, A100, and A100 MIG offerings after inspecting real NVIDIA ResourceSlice attributes?
• Should old labels such as nvidia.com/gpu remain temporarily enabled for existing actions, or should the migration require users to update action files before running new GPU workloads?

#References

• ADR-0023: DRA GPU Allocation with Extended Resource Names
• Kubernetes: Dynamic Resource Allocation
• Kubernetes: Feature Gates
• NVIDIA GPU Operator: DRA Driver for NVIDIA GPUs
• DRA Driver for NVIDIA GPUs