---
title: "DD-0023: DRA GPU Extended Resource Implementation"
---

# DD-0023: DRA GPU Extended Resource Implementation

**Related ADR:** [ADR-0023: DRA GPU Allocation with Extended Resource Names](/adr/0023-dra-gpu-extended-resource-names)

**Status:** Draft

**Date:** 2026-07-09

**Authors:** TBD

## ADR-derived technical notes

These notes scope the implementation of ADR-0023. The decision is to adopt
NVIDIA DRA through Kubernetes extended resource names, without changing the
Themelio action API or introducing explicit `ResourceClaim` objects into
user-facing workflows.

- **Config shape stays unchanged:** `THEMELIO_GPU_LABELS_CONFIG` remains a YAML
  document with a top-level `gpu_labels` map. No `gpu_resources` schema is
  introduced by this DD.
- **Correctness comes from labels:** each `gpu_labels` key is the Kubernetes
  extended resource name rendered into workload `resources.requests` and
  `resources.limits`.
- **Target labels are differentiated:** use product-specific names such as
  `nvidia.com/a10g`, `nvidia.com/a100`, `nvidia.com/a100-mig-1g.5gb`, and
  `nvidia.com/a100-mig-3g.20gb` instead of collapsing multiple products into
  `nvidia.com/gpu`.
- **Known mismatch to fix:** `gpu_a10g` currently maps inconsistently: the web
  mapping uses `nvidia.com/gpu`, while the Waldur site-agent template uses
  `requests.nvidiax.com/gpu`. The target implementation must use one corrected
  catalog label, `nvidia.com/a10g`, and its quota key `requests.nvidia.com/a10g`.
- **Cluster prerequisite:** the current Kubespray defaults in this repository
  resolve to Kubernetes `1.32.8`, which is too old for the ADR-0023 DRA extended
  resource path. DRA rollout therefore requires a Kubernetes upgrade before
  cluster enablement.

## Overview

Themelio keeps the existing action contract:

```yaml
options:
  gpus: 1
  gpuLabel: nvidia.com/a100
```

`k8s-root-actor` continues to render `gpuLabel` into the workload container's
Kubernetes resource requests and limits. DRA remains an infrastructure-backed
implementation detail: DRA `DeviceClass` objects advertise matching
`extendedResourceName` values, and Kubernetes translates extended resource
requests into DRA allocation internally when `DRAExtendedResource` is available.

The implementation work is therefore a label-alignment effort across the
existing config consumers, not a new GPU catalog schema and not a new workload
rendering model.

## Goals and Non-Goals

### Goals

- Preserve `THEMELIO_GPU_LABELS_CONFIG` as the deployed config surface.
- Preserve the top-level YAML key `gpu_labels`.
- Preserve `options.gpus` plus `options.gpuLabel` in actions.
- Preserve workload GPU allocation through Kubernetes `resources.requests` and
  `resources.limits`.
- Align Themelio, Waldur, Kubeflow, and DRA `DeviceClass` extended resource
  names around the same corrected GPU labels.
- Replace generic or incorrect GPU labels with differentiated extended resource
  names where Themelio exposes distinct GPU offerings.

### Non-Goals

- Introducing `gpu_resources` or another richer catalog schema.
- Exposing `ResourceClaim` or `ResourceClaimTemplate` fields to users.
- Reworking Slurm GPU allocation; Slurm actions continue to use `--gres=gpu`.
- Designing DRA CEL selectors in this DD without validating actual NVIDIA
  `ResourceSlice` attributes in a target cluster.

## Design

### 1. GPU label configuration

The generated ConfigMap data remains:

```yaml
gpu_labels:
  nvidia.com/a100:
    node_selector:
      accelerator: "nvidia-a100"
    mig_enabled: "false"

  nvidia.com/a100-mig-1g.5gb:
    node_selector:
      accelerator: "nvidia-a100"
    mig_enabled: "true"
```

The implementation updates only the map keys and metadata values. Existing
parsers should continue to require:

- a non-empty top-level `gpu_labels` map,
- a non-empty `node_selector` map for each entry, and
- string `mig_enabled` metadata.

The minimum target label set is:

| GPU offering | `gpuLabel` | Waldur quota key | Kubeflow `limitsKey` |
|---|---|---|---|
| NVIDIA A10G | `nvidia.com/a10g` | `requests.nvidia.com/a10g` | `nvidia.com/a10g` |
| NVIDIA A100 | `nvidia.com/a100` | `requests.nvidia.com/a100` | `nvidia.com/a100` |
| NVIDIA A100 MIG 1g.5gb | `nvidia.com/a100-mig-1g.5gb` | `requests.nvidia.com/a100-mig-1g.5gb` | `nvidia.com/a100-mig-1g.5gb` |
| NVIDIA A100 MIG 3g.20gb | `nvidia.com/a100-mig-3g.20gb` | `requests.nvidia.com/a100-mig-3g.20gb` | `nvidia.com/a100-mig-3g.20gb` |

The A10G mapping must not keep `nvidiax.com`. That value is treated as a typo or
legacy mismatch, not as a migration alias.

### 2. Application consumers

`k8s-root-actor` remains the authority for action execution:

- `GPU_LABELS_CONFIG` is parsed from `THEMELIO_GPU_LABELS_CONFIG`.
- `options.gpuLabel` is valid only when it is a key of `gpu_labels`.
- `options.gpuLabel` is required when `options.gpus` is greater than zero for
  `k8s-deployment` and `k8s-inferenceservice`.
- Deployment and InferenceService templates keep rendering the selected label in
  both `resources.requests` and `resources.limits`.
- The node selector still comes from the selected `gpu_labels` entry.

The web service must use the same corrected labels when showing the copyable
action GPU label for Waldur GPU components. Component mapping should be:

| Waldur component | Action `gpuLabel` |
|---|---|
| `gpu_a10g` | `nvidia.com/a10g` |
| `gpu_a100` | `nvidia.com/a100` |
| `gpu_a100_mig_1g_5gb` | `nvidia.com/a100-mig-1g.5gb` |
| `gpu_a100_mig_3g_20gb` | `nvidia.com/a100-mig-3g.20gb` |

OpenAPI and product documentation must describe the corrected labels and avoid a
stale enum that lists only `nvidia.com/gpu`, `nvidia.com/mig-1g.5gb`, and
`nvidia.com/mig-3g.20gb`.

### 3. Infrastructure consumers

Ansible remains responsible for generating and applying the config files used by
Themelio, Waldur, and Kubeflow:

- `themelio_gpu_labels` defines the `gpu_labels` ConfigMap consumed by web and
  `k8s-root-actor`.
- The Waldur site-agent quota mapping uses `requests.<gpuLabel>` for each GPU
  component.
- Kubeflow vendor configuration uses the same label as `limitsKey` and a
  user-facing name that matches the offering.

The DRA infrastructure must advertise matching extended resource names. For each
Themelio GPU label, the corresponding DRA `DeviceClass` must set
`spec.extendedResourceName` to that exact label.

DeviceClass selectors must be validated against actual NVIDIA DRA
`ResourceSlice` attributes in the target cluster before rollout. This DD does
not prescribe unverified CEL attribute names.

### 4. DRA rollout prerequisites

DRA rollout is blocked until the cluster supports extended resource allocation
by DRA.

Target Kubernetes version:

- Prefer Kubernetes `1.36+`, where `DRAExtendedResource` is beta and enabled by
  default.
- Kubernetes `1.34` or `1.35` may be used only if `DynamicResourceAllocation`
  and `DRAExtendedResource` are explicitly enabled on the required control-plane
  components and kubelets.
- Kubernetes `1.32.x` is not acceptable for this rollout.

GPU node and operator requirements:

- Label every DRA-capable GPU node with
  `nvidia.com/dra-kubelet-plugin=true`.
- Configure the NVIDIA GPU Operator for DRA-backed GPU allocation, including
  disabling the traditional NVIDIA device plugin when switching a node pool to
  DRA allocation.
- Install the NVIDIA DRA driver with GPU allocation enabled and the kubelet
  plugin node selector set to `nvidia.com/dra-kubelet-plugin=true`.
- For MIG-enabled A100 nodes, apply MIG configuration before validating DRA
  `ResourceSlice` and DeviceClass behavior; if the DRA kubelet plugin was
  already running during MIG changes, restart it before validation.

## Testing Strategy

### Unit and template tests

- Config parser accepts the unchanged `gpu_labels` structure with corrected
  labels.
- Config parser rejects missing or empty `gpu_labels`, empty `node_selector`, and
  unsupported `gpuLabel` values.
- `k8s-root-actor` validates updated valid labels and rejects stale or invalid
  labels.
- Deployment and InferenceService rendering tests confirm the selected
  `gpuLabel` appears in both `resources.requests` and `resources.limits`.
- Web tests confirm Waldur GPU components expose the corrected action
  `gpuLabel`, including `gpu_a10g -> nvidia.com/a10g`.
- Ansible/template tests confirm Waldur quota mapping and Kubeflow vendors use
  the same corrected labels.

### Cluster validation

Before enabling the corrected labels for users, validate the target cluster:

```bash
kubectl get deviceclass
kubectl get resourceslice -o wide
kubectl describe deviceclass <deviceclass-name>
kubectl describe node -l nvidia.com/dra-kubelet-plugin=true
```

Then run one Themelio `k8s-deployment` or `k8s-inferenceservice` action for each
enabled GPU label and confirm:

- the generated pod requests the corrected extended resource name,
- quota accounting uses `requests.<gpuLabel>`,
- the workload schedules and receives GPU access,
- Kubeflow displays the same label where notebook GPU selection is enabled, and
- no Themelio workflow exposes or requires explicit `ResourceClaim` objects.

## Deployment and Rollout

1. Upgrade Kubernetes to a version that supports the desired
   `DRAExtendedResource` behavior; use `1.36+` as the default target.
2. Deploy or update NVIDIA GPU Operator and NVIDIA DRA driver configuration.
3. Label DRA-capable GPU nodes with `nvidia.com/dra-kubelet-plugin=true`.
4. Create or update DRA `DeviceClass` resources so their
   `extendedResourceName` values match the `gpu_labels` keys.
5. Update Ansible GPU label defaults, Waldur quota mapping, Kubeflow vendors,
   web GPU label mapping, OpenAPI, and actions-yaml documentation in one
   coordinated change.
6. Apply Themelio configuration and restart GPU-label consumers when the
   ConfigMap changes.
7. Validate with the test plan above before announcing the corrected labels to
   users.

Rollback is to restore the previous `gpu_labels`, Waldur quota mapping, Kubeflow
vendors, and web copy-label mapping, then restart Themelio GPU-label consumers.
Rollback does not require changing user action schema because the schema is
unchanged.

## Open Questions

- Which target Kubernetes version will the first DRA-enabled Themelio cluster
  use: `1.36+`, or `1.34`/`1.35` with explicit feature gates?
- Which exact DRA `DeviceClass` objects and CEL selectors should represent A10G,
  A100, and A100 MIG offerings after inspecting real NVIDIA `ResourceSlice`
  attributes?
- Should old labels such as `nvidia.com/gpu` remain temporarily enabled for
  existing actions, or should the migration require users to update action
  files before running new GPU workloads?

## References

- [ADR-0023: DRA GPU Allocation with Extended Resource Names](/adr/0023-dra-gpu-extended-resource-names)
- [Kubernetes: Dynamic Resource Allocation](https://kubernetes.io/docs/concepts/scheduling-eviction/dynamic-resource-allocation/)
- [Kubernetes: Feature Gates](https://kubernetes.io/docs/reference/command-line-tools-reference/feature-gates/)
- [NVIDIA GPU Operator: DRA Driver for NVIDIA GPUs](https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/latest/dra-intro-install.html)
- [DRA Driver for NVIDIA GPUs](https://dra-driver-nvidia-gpu.sigs.k8s.io/docs/)