---
title: "ADR-0023: DRA GPU Allocation with Extended Resource Names"
---

# ADR-0023: DRA GPU Allocation with Extended Resource Names

**Status:** Accepted

**Date:** 2026-07-06

**Deciders:** Themelio developers

**Relates to:** [ADR-0007 (GPU Slicing and Quota Enforcement)](/adr/0007-gpu-slicing-and-quota-enforcement) - decided the GPU slicing and quota model; this ADR decides how Themelio should consume NVIDIA DRA while preserving that model.

## Context

Themelio currently exposes GPU capacity to Kubernetes workloads through extended resource names such as `nvidia.com/gpu`, `nvidia.com/mig-1g.5gb`, and `nvidia.com/mig-3g.20gb`. Those names are visible in user action configuration, Kubernetes workload manifests, Waldur quota mapping, and Kubeflow notebook GPU vendor configuration.

NVIDIA now provides a DRA Driver for NVIDIA GPUs. NVIDIA documents DRA GPU allocation as a replacement for the traditional NVIDIA Kubernetes Device Plugin allocation path, while Kubernetes also provides a `DRAExtendedResource` feature that lets extended resource requests be translated into DRA-backed `ResourceClaims`. Themelio wants to migrate to DRA so that GPU allocation can distinguish GPU products and slice profiles cleanly, rather than treating dissimilar devices as the same generic `nvidia.com/gpu` resource. Themelio therefore needs to decide whether adopting DRA should also change the user-facing and integration-facing GPU allocation shape.

This ADR does not adopt the discarded `gpu_classes` catalog prototype from the `gpu-quota-labels-fix` branch. That implementation does not work and is not part of the target architecture.

### Current State

The `k8s-root-actor` action API accepts `options.gpus` and `options.gpuLabel`. The `gpuLabel` value is a Kubernetes extended resource key, and the service validates it against `THEMELIO_GPU_LABELS_CONFIG` before rendering a Kubernetes manifest.

For `k8s-deployment` and `k8s-inferenceservice` actions, `k8s-root-actor` renders the selected `gpuLabel` into container `resources.requests` and `resources.limits`. The same config also carries a node selector for the selected GPU class. Slurm actions use `--gres=gpu` and do not use this Kubernetes `gpuLabel` path.

Waldur quota accounting is mapped to Kubernetes `ResourceQuota` request keys such as `requests.nvidia.com/gpu`, `requests.nvidia.com/mig-1g.5gb`, and `requests.nvidia.com/mig-3g.20gb`. Kubeflow notebook GPU options use matching `limitsKey` values.

Terraform and Ansible carry node-level GPU intent. Terraform emits labels such as `node.kubernetes.io/instance-type=gpu`, `nvidia.com/gpu=true`, `accelerator=nvidia-a100`, `mig_enabled`, and `mig_profile`. The Ansible GPU role installs the NVIDIA GPU Operator, applies GPU node labels and taints, and uses NVIDIA MIG Manager labels such as `nvidia.com/mig.config=<profile>` when MIG is enabled.

### Current Problems

Moving directly to explicit `ResourceClaim` or `ResourceClaimTemplate` objects would force Themelio to change several stable integration points at once: action input shape, manifest rendering, Waldur quota mapping, Kubeflow GPU selection, and operational troubleshooting.

The current system already treats extended resource names as the common contract between workloads, quotas, and UI choices. Replacing that contract would create a larger migration than is required to adopt DRA-backed GPU allocation.

The current generic whole-GPU resource name does not describe which physical GPU product is being requested. That makes it harder to expose A10G, A100, and future GPU classes as distinct quota and scheduling choices. DRA-backed `DeviceClass` definitions with `extendedResourceName` give Themelio a way to publish differentiated GPU resource names while keeping the same workload request pattern.

Themelio still needs DRA driver deployment and DRA-specific node selection, but those changes should not require users or higher-level Themelio services to understand `ResourceClaim` lifecycle details.

### Requirements

1. Preserve the user-facing `options.gpus` plus `options.gpuLabel` workflow.
2. Preserve Kubernetes resource `requests` and `limits` as the workload allocation surface.
3. Preserve Waldur and Kubeflow integration through the same extended resource names.
4. Avoid introducing a second GPU allocation model into Themelio application code.
5. Allow the underlying GPU allocation implementation to move from the NVIDIA device plugin path to the NVIDIA DRA driver path.
6. Represent distinct GPU products and MIG slice profiles as distinct catalog entries and extended resource names.

## Decision Drivers

- **Integration stability:** Keep the existing action API, quota keys, and Kubeflow GPU options stable.
- **Operational simplicity:** Avoid making operators reason about both explicit `ResourceClaims` and extended resource names for the same Themelio workflows.
- **DRA compatibility:** Use Kubernetes and NVIDIA-supported DRA behavior rather than direct node status patching or custom resource-name translation.
- **Migration size:** Limit the rollout to infrastructure and cluster configuration changes where possible.
- **GPU differentiation:** Make the requested GPU product or slice profile explicit in Themelio's catalog, quotas, and workload resource name.

## Considered Options

### Option A: Switch entirely to ResourceClaims

Themelio would expose DRA directly. Workloads would use `ResourceClaim` or `ResourceClaimTemplate` objects, and `k8s-root-actor` would render pod `spec.resourceClaims` instead of GPU resource `requests` and `limits`.

**Pros:**
- Uses the native DRA API directly.
- Gives future access to DRA-specific selectors, allocation status, and device attributes.
- Avoids relying on the extended resource compatibility path.

**Cons:**
- Requires a new Themelio workload rendering model.
- Requires Waldur quota and reporting behavior to be redesigned around claims instead of `requests.<resource>`.
- Requires Kubeflow GPU selection to be reworked or bypassed.
- Exposes a more complex allocation lifecycle than Themelio needs for current GPU workflows.

### Option B: Use DRA extendedResourceName and keep the current workflow

Themelio keeps using Kubernetes extended resource requests in workload manifests. DRA `DeviceClass` objects expose `extendedResourceName` values that match Themelio's GPU catalog resource keys. During migration, those keys can include existing names such as `nvidia.com/gpu`, `nvidia.com/mig-1g.5gb`, and `nvidia.com/mig-3g.20gb`, and can move toward differentiated names such as `nvidia.com/a10g`, `nvidia.com/a100`, and `nvidia.com/a100-mig-1g.5gb`.

With `DRAExtendedResource` enabled, the scheduler translates those extended resource requests into DRA-backed allocation internally. Users and Themelio services continue to use `options.gpus`, `options.gpuLabel`, Kubernetes resource `requests` and `limits`, Waldur quota keys, and Kubeflow `limitsKey` values.

**Pros:**
- Preserves the current user-facing and service-facing contract.
- Keeps Waldur quota accounting and Kubeflow GPU selection aligned with current resource names.
- Lets Themelio adopt the NVIDIA DRA driver without introducing explicit `ResourceClaim` objects into application workflows.
- Reduces migration risk by changing the infrastructure allocation backend rather than every consumer.

**Cons:**
- Does not expose the full DRA claim model to Themelio users or actions.
- Depends on the cluster supporting `DRAExtendedResource`.
- Requires DRA `DeviceClass` configuration to stay aligned with the existing extended resource names.

### Option C: Use ResourceClaims where possible and extended resources where switching would require hacky solutions

Themelio would support both explicit DRA claims and extended resource requests. New or simple workflows could use `ResourceClaims`, while existing integrations such as Waldur, Kubeflow, or action manifests could stay on extended resource names until they can be changed cleanly.

**Pros:**
- Allows gradual adoption of DRA-native capabilities.
- Avoids forcing every integration to move at once.
- Keeps an escape hatch for workloads that need explicit DRA semantics.

**Cons:**
- Creates two allocation models for the same GPU capacity.
- Makes quota accounting and user documentation harder because some workloads are claim-based and others are request-based.
- Increases testing and operational burden.
- Risks inconsistent behavior between Themelio workflows.

## Comparison Matrix

| Option | Integration stability | Operational simplicity | DRA-native expressiveness | Migration size |
|--------|-----------------------|------------------------|---------------------------|----------------|
| A: ResourceClaims entirely | Low | Medium | High | High |
| B: extendedResourceName only | High | High | Medium | Low |
| C: Hybrid | Medium | Low | High | Medium |

## Decision

We will use **Option B: DRA extendedResourceName and keep the current workflow**.

DRA `DeviceClass` objects must expose extended resource names matching the resource keys in Themelio's GPU catalog. During the initial migration this includes the resource keys Themelio already uses, such as `nvidia.com/gpu`, `nvidia.com/mig-1g.5gb`, and `nvidia.com/mig-3g.20gb`; the target catalog should use differentiated resource names where Themelio needs distinct offerings, such as `nvidia.com/a10g`, `nvidia.com/a100`, and `nvidia.com/a100-mig-1g.5gb`. Workloads created by Themelio continue to request GPUs through container resource `requests` and `limits`; Themelio will not expose `ResourceClaim` or `ResourceClaimTemplate` objects in the user-facing action API.

The user-facing action shape remains `options.gpus` plus `options.gpuLabel`. Waldur quota mapping and Kubeflow GPU vendor configuration continue to use the same extended resource names. Terraform and Ansible keep carrying node intent labels for physical GPU placement and MIG selection where those labels are still needed.

Future DRA deployment work must enable or verify `DRAExtendedResource` support and label DRA-capable GPU nodes with `nvidia.com/dra-kubelet-plugin=true`, using that label as the NVIDIA DRA kubelet plugin node selector.

### GPU Catalog Structure

The GPU catalog is the source of truth for Themelio GPU offerings. It is not the discarded `gpu_classes` implementation from the abandoned branch; it is the target structure that keeps DRA `DeviceClass` configuration, Kubernetes resource names, Waldur quotas, Kubeflow labels, and action validation aligned.

The catalog should be keyed by the Kubernetes extended resource name that workloads request:

```yaml
gpu_resources:
  nvidia.com/a100:
    display_name: "NVIDIA A100"
    product: "a100"
    partition: "whole"
    extended_resource_name: "nvidia.com/a100"
    dra_device_class: "gpu.nvidia.com"
    waldur_key: "gpu_a100"
    measured_unit: "GPU"
    node_selector:
      accelerator: "nvidia-a100"
    default: false

  nvidia.com/a100-mig-1g.5gb:
    display_name: "NVIDIA A100 MIG 1g.5gb"
    product: "a100"
    partition: "mig"
    mig_profile: "1g.5gb"
    extended_resource_name: "nvidia.com/a100-mig-1g.5gb"
    dra_device_class: "mig.nvidia.com"
    waldur_key: "gpu_a100_mig_1g_5gb"
    measured_unit: "Slice"
    node_selector:
      accelerator: "nvidia-a100"
    default: true
```

Each entry must define the resource name rendered into workload `requests` and `limits`, the DRA `DeviceClass` that advertises that name, user-facing display metadata, quota metadata, and any placement metadata Themelio still needs. The catalog may include migration aliases for old names, but new Themelio offerings should use differentiated resource names instead of collapsing different GPU products into `nvidia.com/gpu`.

## Consequences

**Positive:**
- Themelio can adopt the NVIDIA DRA driver without changing the action API, actions.yaml GPU shape, Waldur quota keys, or Kubeflow GPU labels.
- Operators have one GPU allocation contract to reason about: extended resource names.
- Existing quota and over-capacity behavior based on Kubernetes resource requests remains applicable.
- GPU products and MIG profiles can be exposed as separate catalog offerings instead of being hidden behind a generic whole-GPU resource.

**Negative:**
- Themelio does not expose DRA-native claim selectors or claim lifecycle controls to users.
- Cluster correctness now depends on DRA `DeviceClass` `extendedResourceName` configuration matching Themelio's existing resource keys.
- The DRA extended resource compatibility path must be available and enabled in the Kubernetes control plane and kubelets.

**Neutral:**
- The underlying GPU allocation implementation changes, but Themelio service interfaces do not.
- MIG profile selection remains node configuration managed by infrastructure automation, not a per-action claim selector.
- The discarded `gpu_classes` catalog implementation is not part of this decision.
- Existing resource names can remain temporarily during migration, but the target catalog should converge on differentiated names where users and quotas need that distinction.

### Migration

Migration is required operationally. The cluster must install and configure the NVIDIA DRA driver, configure DRA `DeviceClass` objects with catalog-defined extended resource names, enable or verify `DRAExtendedResource`, and label eligible GPU nodes with `nvidia.com/dra-kubelet-plugin=true`.

No user-facing API migration is required. Existing actions using `options.gpus` and `options.gpuLabel` continue to render Kubernetes resource `requests` and `limits`. Where Themelio introduces differentiated resource names, the migration updates the allowed `gpuLabel` values, Waldur quota keys, Kubeflow GPU labels, and DRA `DeviceClass` `extendedResourceName` values together through the GPU catalog.

## References

- [NVIDIA GPU Operator: DRA Driver for NVIDIA GPUs](https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/latest/dra-intro-install.html)
- [Kubernetes: Dynamic Resource Allocation](https://kubernetes.io/docs/concepts/scheduling-eviction/dynamic-resource-allocation/)
- [ADR-0007: GPU Slicing and Quota Enforcement](/adr/0007-gpu-slicing-and-quota-enforcement)