Themelio compatibility reference. This page describes the retained Themelio implementation or its historical design. Check the parallel-operation guide before applying it to the current platform.

#ADR-0023: DRA GPU Allocation with Extended Resource Names

Status: Accepted

Date: 2026-07-06

Deciders: Themelio developers

Relates to: ADR-0007 (GPU Slicing and Quota Enforcement) - decided the GPU slicing and quota model; this ADR decides how Themelio should consume NVIDIA DRA while preserving that model.

#Context

Themelio currently exposes GPU capacity to Kubernetes workloads through extended resource names such as nvidia.com/gpu, nvidia.com/mig-1g.5gb, and nvidia.com/mig-3g.20gb. Those names are visible in user action configuration, Kubernetes workload manifests, Waldur quota mapping, and Kubeflow notebook GPU vendor configuration.

NVIDIA now provides a DRA Driver for NVIDIA GPUs. NVIDIA documents DRA GPU allocation as a replacement for the traditional NVIDIA Kubernetes Device Plugin allocation path, while Kubernetes also provides a DRAExtendedResource feature that lets extended resource requests be translated into DRA-backed ResourceClaims. Themelio wants to migrate to DRA so that GPU allocation can distinguish GPU products and slice profiles cleanly, rather than treating dissimilar devices as the same generic nvidia.com/gpu resource. Themelio therefore needs to decide whether adopting DRA should also change the user-facing and integration-facing GPU allocation shape.

This ADR does not adopt the discarded gpu_classes catalog prototype from the gpu-quota-labels-fix branch. That implementation does not work and is not part of the target architecture.

#Current State

The k8s-root-actor action API accepts options.gpus and options.gpuLabel. The gpuLabel value is a Kubernetes extended resource key, and the service validates it against THEMELIO_GPU_LABELS_CONFIG before rendering a Kubernetes manifest.

For k8s-deployment and k8s-inferenceservice actions, k8s-root-actor renders the selected gpuLabel into container resources.requests and resources.limits. The same config also carries a node selector for the selected GPU class. Slurm actions use --gres=gpu and do not use this Kubernetes gpuLabel path.

Waldur quota accounting is mapped to Kubernetes ResourceQuota request keys such as requests.nvidia.com/gpu, requests.nvidia.com/mig-1g.5gb, and requests.nvidia.com/mig-3g.20gb. Kubeflow notebook GPU options use matching limitsKey values.

Terraform and Ansible carry node-level GPU intent. Terraform emits labels such as node.kubernetes.io/instance-type=gpu, nvidia.com/gpu=true, accelerator=nvidia-a100, mig_enabled, and mig_profile. The Ansible GPU role installs the NVIDIA GPU Operator, applies GPU node labels and taints, and uses NVIDIA MIG Manager labels such as nvidia.com/mig.config=<profile> when MIG is enabled.

#Current Problems

Moving directly to explicit ResourceClaim or ResourceClaimTemplate objects would force Themelio to change several stable integration points at once: action input shape, manifest rendering, Waldur quota mapping, Kubeflow GPU selection, and operational troubleshooting.

The current system already treats extended resource names as the common contract between workloads, quotas, and UI choices. Replacing that contract would create a larger migration than is required to adopt DRA-backed GPU allocation.

The current generic whole-GPU resource name does not describe which physical GPU product is being requested. That makes it harder to expose A10G, A100, and future GPU classes as distinct quota and scheduling choices. DRA-backed DeviceClass definitions with extendedResourceName give Themelio a way to publish differentiated GPU resource names while keeping the same workload request pattern.

Themelio still needs DRA driver deployment and DRA-specific node selection, but those changes should not require users or higher-level Themelio services to understand ResourceClaim lifecycle details.

#Requirements

1. Preserve the user-facing options.gpus plus options.gpuLabel workflow.
2. Preserve Kubernetes resource requests and limits as the workload allocation surface.
3. Preserve Waldur and Kubeflow integration through the same extended resource names.
4. Avoid introducing a second GPU allocation model into Themelio application code.
5. Allow the underlying GPU allocation implementation to move from the NVIDIA device plugin path to the NVIDIA DRA driver path.
6. Represent distinct GPU products and MIG slice profiles as distinct catalog entries and extended resource names.

#Decision Drivers

• Integration stability: Keep the existing action API, quota keys, and Kubeflow GPU options stable.
• Operational simplicity: Avoid making operators reason about both explicit ResourceClaims and extended resource names for the same Themelio workflows.
• DRA compatibility: Use Kubernetes and NVIDIA-supported DRA behavior rather than direct node status patching or custom resource-name translation.
• Migration size: Limit the rollout to infrastructure and cluster configuration changes where possible.
• GPU differentiation: Make the requested GPU product or slice profile explicit in Themelio's catalog, quotas, and workload resource name.

#Considered Options

#Option A: Switch entirely to ResourceClaims

Themelio would expose DRA directly. Workloads would use ResourceClaim or ResourceClaimTemplate objects, and k8s-root-actor would render pod spec.resourceClaims instead of GPU resource requests and limits.

Pros:

• Uses the native DRA API directly.
• Gives future access to DRA-specific selectors, allocation status, and device attributes.
• Avoids relying on the extended resource compatibility path.

Cons:

• Requires a new Themelio workload rendering model.
• Requires Waldur quota and reporting behavior to be redesigned around claims instead of requests.<resource>.
• Requires Kubeflow GPU selection to be reworked or bypassed.
• Exposes a more complex allocation lifecycle than Themelio needs for current GPU workflows.

#Option B: Use DRA extendedResourceName and keep the current workflow

Themelio keeps using Kubernetes extended resource requests in workload manifests. DRA DeviceClass objects expose extendedResourceName values that match Themelio's GPU catalog resource keys. During migration, those keys can include existing names such as nvidia.com/gpu, nvidia.com/mig-1g.5gb, and nvidia.com/mig-3g.20gb, and can move toward differentiated names such as nvidia.com/a10g, nvidia.com/a100, and nvidia.com/a100-mig-1g.5gb.

With DRAExtendedResource enabled, the scheduler translates those extended resource requests into DRA-backed allocation internally. Users and Themelio services continue to use options.gpus, options.gpuLabel, Kubernetes resource requests and limits, Waldur quota keys, and Kubeflow limitsKey values.

Pros:

• Preserves the current user-facing and service-facing contract.
• Keeps Waldur quota accounting and Kubeflow GPU selection aligned with current resource names.
• Lets Themelio adopt the NVIDIA DRA driver without introducing explicit ResourceClaim objects into application workflows.
• Reduces migration risk by changing the infrastructure allocation backend rather than every consumer.

Cons:

• Does not expose the full DRA claim model to Themelio users or actions.
• Depends on the cluster supporting DRAExtendedResource.
• Requires DRA DeviceClass configuration to stay aligned with the existing extended resource names.

#Option C: Use ResourceClaims where possible and extended resources where switching would require hacky solutions

Themelio would support both explicit DRA claims and extended resource requests. New or simple workflows could use ResourceClaims, while existing integrations such as Waldur, Kubeflow, or action manifests could stay on extended resource names until they can be changed cleanly.

Pros:

• Allows gradual adoption of DRA-native capabilities.
• Avoids forcing every integration to move at once.
• Keeps an escape hatch for workloads that need explicit DRA semantics.

Cons:

• Creates two allocation models for the same GPU capacity.
• Makes quota accounting and user documentation harder because some workloads are claim-based and others are request-based.
• Increases testing and operational burden.
• Risks inconsistent behavior between Themelio workflows.

#Comparison Matrix

#Decision

We will use Option B: DRA extendedResourceName and keep the current workflow.

DRA DeviceClass objects must expose extended resource names matching the resource keys in Themelio's GPU catalog. During the initial migration this includes the resource keys Themelio already uses, such as nvidia.com/gpu, nvidia.com/mig-1g.5gb, and nvidia.com/mig-3g.20gb; the target catalog should use differentiated resource names where Themelio needs distinct offerings, such as nvidia.com/a10g, nvidia.com/a100, and nvidia.com/a100-mig-1g.5gb. Workloads created by Themelio continue to request GPUs through container resource requests and limits; Themelio will not expose ResourceClaim or ResourceClaimTemplate objects in the user-facing action API.

The user-facing action shape remains options.gpus plus options.gpuLabel. Waldur quota mapping and Kubeflow GPU vendor configuration continue to use the same extended resource names. Terraform and Ansible keep carrying node intent labels for physical GPU placement and MIG selection where those labels are still needed.

Future DRA deployment work must enable or verify DRAExtendedResource support and label DRA-capable GPU nodes with nvidia.com/dra-kubelet-plugin=true, using that label as the NVIDIA DRA kubelet plugin node selector.

#GPU Catalog Structure

The GPU catalog is the source of truth for Themelio GPU offerings. It is not the discarded gpu_classes implementation from the abandoned branch; it is the target structure that keeps DRA DeviceClass configuration, Kubernetes resource names, Waldur quotas, Kubeflow labels, and action validation aligned.

The catalog should be keyed by the Kubernetes extended resource name that workloads request:

gpu_resources:
  nvidia.com/a100:
    display_name: "NVIDIA A100"
    product: "a100"
    partition: "whole"
    extended_resource_name: "nvidia.com/a100"
    dra_device_class: "gpu.nvidia.com"
    waldur_key: "gpu_a100"
    measured_unit: "GPU"
    node_selector:
      accelerator: "nvidia-a100"
    default: false

  nvidia.com/a100-mig-1g.5gb:
    display_name: "NVIDIA A100 MIG 1g.5gb"
    product: "a100"
    partition: "mig"
    mig_profile: "1g.5gb"
    extended_resource_name: "nvidia.com/a100-mig-1g.5gb"
    dra_device_class: "mig.nvidia.com"
    waldur_key: "gpu_a100_mig_1g_5gb"
    measured_unit: "Slice"
    node_selector:
      accelerator: "nvidia-a100"
    default: true

Each entry must define the resource name rendered into workload requests and limits, the DRA DeviceClass that advertises that name, user-facing display metadata, quota metadata, and any placement metadata Themelio still needs. The catalog may include migration aliases for old names, but new Themelio offerings should use differentiated resource names instead of collapsing different GPU products into nvidia.com/gpu.

#Consequences

Positive:

• Themelio can adopt the NVIDIA DRA driver without changing the action API, actions.yaml GPU shape, Waldur quota keys, or Kubeflow GPU labels.
• Operators have one GPU allocation contract to reason about: extended resource names.
• Existing quota and over-capacity behavior based on Kubernetes resource requests remains applicable.
• GPU products and MIG profiles can be exposed as separate catalog offerings instead of being hidden behind a generic whole-GPU resource.

Negative:

• Themelio does not expose DRA-native claim selectors or claim lifecycle controls to users.
• Cluster correctness now depends on DRA DeviceClass extendedResourceName configuration matching Themelio's existing resource keys.
• The DRA extended resource compatibility path must be available and enabled in the Kubernetes control plane and kubelets.

Neutral:

• The underlying GPU allocation implementation changes, but Themelio service interfaces do not.
• MIG profile selection remains node configuration managed by infrastructure automation, not a per-action claim selector.
• The discarded gpu_classes catalog implementation is not part of this decision.
• Existing resource names can remain temporarily during migration, but the target catalog should converge on differentiated names where users and quotas need that distinction.

#Migration

Migration is required operationally. The cluster must install and configure the NVIDIA DRA driver, configure DRA DeviceClass objects with catalog-defined extended resource names, enable or verify DRAExtendedResource, and label eligible GPU nodes with nvidia.com/dra-kubelet-plugin=true.

No user-facing API migration is required. Existing actions using options.gpus and options.gpuLabel continue to render Kubernetes resource requests and limits. Where Themelio introduces differentiated resource names, the migration updates the allowed gpuLabel values, Waldur quota keys, Kubeflow GPU labels, and DRA DeviceClass extendedResourceName values together through the GPU catalog.

#References

• NVIDIA GPU Operator: DRA Driver for NVIDIA GPUs
• Kubernetes: Dynamic Resource Allocation
• ADR-0007: GPU Slicing and Quota Enforcement