> For AI agents: the complete documentation index is available at /llms.txt, the full documentation bundle is available at /llms-full.txt. > **Themelio compatibility reference.** This page describes the retained Themelio implementation or its historical design. Check the [parallel-operation guide](/compatibility/index.md) before applying it to the current platform. # DD-0023: DRA GPU Extended Resource Implementation **Related ADR:** [ADR-0023: DRA GPU Allocation with Extended Resource Names](/adr/0023-dra-gpu-extended-resource-names.md) **Status:** Draft **Date:** 2026-07-09 **Authors:** TBD ## ADR-derived technical notes These notes scope the implementation of ADR-0023. The decision is to adopt NVIDIA DRA through Kubernetes extended resource names, without changing the Themelio action API or introducing explicit `ResourceClaim` objects into user-facing workflows. - **Config shape stays unchanged:** `THEMELIO_GPU_LABELS_CONFIG` remains a YAML document with a top-level `gpu_labels` map. No `gpu_resources` schema is introduced by this DD. - **Correctness comes from labels:** each `gpu_labels` key is the Kubernetes extended resource name rendered into workload `resources.requests` and `resources.limits`. - **Target labels are differentiated:** use product-specific names such as `nvidia.com/a10g`, `nvidia.com/a100`, `nvidia.com/a100-mig-1g.5gb`, and `nvidia.com/a100-mig-3g.20gb` instead of collapsing multiple products into `nvidia.com/gpu`. - **Known mismatch to fix:** `gpu_a10g` currently maps inconsistently: the web mapping uses `nvidia.com/gpu`, while the Waldur site-agent template uses `requests.nvidiax.com/gpu`. The target implementation must use one corrected catalog label, `nvidia.com/a10g`, and its quota key `requests.nvidia.com/a10g`. - **Cluster prerequisite:** the current Kubespray defaults in this repository resolve to Kubernetes `1.32.8`, which is too old for the ADR-0023 DRA extended resource path. DRA rollout therefore requires a Kubernetes upgrade before cluster enablement. ## Overview Themelio keeps the existing action contract: ```yaml options: gpus: 1 gpuLabel: nvidia.com/a100 ``` `k8s-root-actor` continues to render `gpuLabel` into the workload container's Kubernetes resource requests and limits. DRA remains an infrastructure-backed implementation detail: DRA `DeviceClass` objects advertise matching `extendedResourceName` values, and Kubernetes translates extended resource requests into DRA allocation internally when `DRAExtendedResource` is available. The implementation work is therefore a label-alignment effort across the existing config consumers, not a new GPU catalog schema and not a new workload rendering model. ## Goals and Non-Goals ### Goals - Preserve `THEMELIO_GPU_LABELS_CONFIG` as the deployed config surface. - Preserve the top-level YAML key `gpu_labels`. - Preserve `options.gpus` plus `options.gpuLabel` in actions. - Preserve workload GPU allocation through Kubernetes `resources.requests` and `resources.limits`. - Align Themelio, Waldur, Kubeflow, and DRA `DeviceClass` extended resource names around the same corrected GPU labels. - Replace generic or incorrect GPU labels with differentiated extended resource names where Themelio exposes distinct GPU offerings. ### Non-Goals - Introducing `gpu_resources` or another richer catalog schema. - Exposing `ResourceClaim` or `ResourceClaimTemplate` fields to users. - Reworking Slurm GPU allocation; Slurm actions continue to use `--gres=gpu`. - Designing DRA CEL selectors in this DD without validating actual NVIDIA `ResourceSlice` attributes in a target cluster. ## Design ### 1. GPU label configuration The generated ConfigMap data remains: ```yaml gpu_labels: nvidia.com/a100: node_selector: accelerator: "nvidia-a100" mig_enabled: "false" nvidia.com/a100-mig-1g.5gb: node_selector: accelerator: "nvidia-a100" mig_enabled: "true" ``` The implementation updates only the map keys and metadata values. Existing parsers should continue to require: - a non-empty top-level `gpu_labels` map, - a non-empty `node_selector` map for each entry, and - string `mig_enabled` metadata. The minimum target label set is: | GPU offering | `gpuLabel` | Waldur quota key | Kubeflow `limitsKey` | | ----------------------- | ----------------------------- | -------------------------------------- | ----------------------------- | | NVIDIA A10G | `nvidia.com/a10g` | `requests.nvidia.com/a10g` | `nvidia.com/a10g` | | NVIDIA A100 | `nvidia.com/a100` | `requests.nvidia.com/a100` | `nvidia.com/a100` | | NVIDIA A100 MIG 1g.5gb | `nvidia.com/a100-mig-1g.5gb` | `requests.nvidia.com/a100-mig-1g.5gb` | `nvidia.com/a100-mig-1g.5gb` | | NVIDIA A100 MIG 3g.20gb | `nvidia.com/a100-mig-3g.20gb` | `requests.nvidia.com/a100-mig-3g.20gb` | `nvidia.com/a100-mig-3g.20gb` | The A10G mapping must not keep `nvidiax.com`. That value is treated as a typo or legacy mismatch, not as a migration alias. ### 2. Application consumers `k8s-root-actor` remains the authority for action execution: - `GPU_LABELS_CONFIG` is parsed from `THEMELIO_GPU_LABELS_CONFIG`. - `options.gpuLabel` is valid only when it is a key of `gpu_labels`. - `options.gpuLabel` is required when `options.gpus` is greater than zero for `k8s-deployment` and `k8s-inferenceservice`. - Deployment and InferenceService templates keep rendering the selected label in both `resources.requests` and `resources.limits`. - The node selector still comes from the selected `gpu_labels` entry. The web service must use the same corrected labels when showing the copyable action GPU label for Waldur GPU components. Component mapping should be: | Waldur component | Action `gpuLabel` | | ---------------------- | ----------------------------- | | `gpu_a10g` | `nvidia.com/a10g` | | `gpu_a100` | `nvidia.com/a100` | | `gpu_a100_mig_1g_5gb` | `nvidia.com/a100-mig-1g.5gb` | | `gpu_a100_mig_3g_20gb` | `nvidia.com/a100-mig-3g.20gb` | OpenAPI and product documentation must describe the corrected labels and avoid a stale enum that lists only `nvidia.com/gpu`, `nvidia.com/mig-1g.5gb`, and `nvidia.com/mig-3g.20gb`. ### 3. Infrastructure consumers Ansible remains responsible for generating and applying the config files used by Themelio, Waldur, and Kubeflow: - `themelio_gpu_labels` defines the `gpu_labels` ConfigMap consumed by web and `k8s-root-actor`. - The Waldur site-agent quota mapping uses `requests.` for each GPU component. - Kubeflow vendor configuration uses the same label as `limitsKey` and a user-facing name that matches the offering. The DRA infrastructure must advertise matching extended resource names. For each Themelio GPU label, the corresponding DRA `DeviceClass` must set `spec.extendedResourceName` to that exact label. DeviceClass selectors must be validated against actual NVIDIA DRA `ResourceSlice` attributes in the target cluster before rollout. This DD does not prescribe unverified CEL attribute names. ### 4. DRA rollout prerequisites DRA rollout is blocked until the cluster supports extended resource allocation by DRA. Target Kubernetes version: - Prefer Kubernetes `1.36+`, where `DRAExtendedResource` is beta and enabled by default. - Kubernetes `1.34` or `1.35` may be used only if `DynamicResourceAllocation` and `DRAExtendedResource` are explicitly enabled on the required control-plane components and kubelets. - Kubernetes `1.32.x` is not acceptable for this rollout. GPU node and operator requirements: - Label every DRA-capable GPU node with `nvidia.com/dra-kubelet-plugin=true`. - Configure the NVIDIA GPU Operator for DRA-backed GPU allocation, including disabling the traditional NVIDIA device plugin when switching a node pool to DRA allocation. - Install the NVIDIA DRA driver with GPU allocation enabled and the kubelet plugin node selector set to `nvidia.com/dra-kubelet-plugin=true`. - For MIG-enabled A100 nodes, apply MIG configuration before validating DRA `ResourceSlice` and DeviceClass behavior; if the DRA kubelet plugin was already running during MIG changes, restart it before validation. ## Testing Strategy ### Unit and template tests - Config parser accepts the unchanged `gpu_labels` structure with corrected labels. - Config parser rejects missing or empty `gpu_labels`, empty `node_selector`, and unsupported `gpuLabel` values. - `k8s-root-actor` validates updated valid labels and rejects stale or invalid labels. - Deployment and InferenceService rendering tests confirm the selected `gpuLabel` appears in both `resources.requests` and `resources.limits`. - Web tests confirm Waldur GPU components expose the corrected action `gpuLabel`, including `gpu_a10g -> nvidia.com/a10g`. - Ansible/template tests confirm Waldur quota mapping and Kubeflow vendors use the same corrected labels. ### Cluster validation Before enabling the corrected labels for users, validate the target cluster: ```bash kubectl get deviceclass kubectl get resourceslice -o wide kubectl describe deviceclass kubectl describe node -l nvidia.com/dra-kubelet-plugin=true ``` Then run one Themelio `k8s-deployment` or `k8s-inferenceservice` action for each enabled GPU label and confirm: - the generated pod requests the corrected extended resource name, - quota accounting uses `requests.`, - the workload schedules and receives GPU access, - Kubeflow displays the same label where notebook GPU selection is enabled, and - no Themelio workflow exposes or requires explicit `ResourceClaim` objects. ## Deployment and Rollout 1. Upgrade Kubernetes to a version that supports the desired `DRAExtendedResource` behavior; use `1.36+` as the default target. 2. Deploy or update NVIDIA GPU Operator and NVIDIA DRA driver configuration. 3. Label DRA-capable GPU nodes with `nvidia.com/dra-kubelet-plugin=true`. 4. Create or update DRA `DeviceClass` resources so their `extendedResourceName` values match the `gpu_labels` keys. 5. Update Ansible GPU label defaults, Waldur quota mapping, Kubeflow vendors, web GPU label mapping, OpenAPI, and actions-yaml documentation in one coordinated change. 6. Apply Themelio configuration and restart GPU-label consumers when the ConfigMap changes. 7. Validate with the test plan above before announcing the corrected labels to users. Rollback is to restore the previous `gpu_labels`, Waldur quota mapping, Kubeflow vendors, and web copy-label mapping, then restart Themelio GPU-label consumers. Rollback does not require changing user action schema because the schema is unchanged. ## Open Questions - Which target Kubernetes version will the first DRA-enabled Themelio cluster use: `1.36+`, or `1.34`/`1.35` with explicit feature gates? - Which exact DRA `DeviceClass` objects and CEL selectors should represent A10G, A100, and A100 MIG offerings after inspecting real NVIDIA `ResourceSlice` attributes? - Should old labels such as `nvidia.com/gpu` remain temporarily enabled for existing actions, or should the migration require users to update action files before running new GPU workloads? ## References - [ADR-0023: DRA GPU Allocation with Extended Resource Names](/adr/0023-dra-gpu-extended-resource-names.md) - [Kubernetes: Dynamic Resource Allocation](https://kubernetes.io/docs/concepts/scheduling-eviction/dynamic-resource-allocation/) - [Kubernetes: Feature Gates](https://kubernetes.io/docs/reference/command-line-tools-reference/feature-gates/) - [NVIDIA GPU Operator: DRA Driver for NVIDIA GPUs](https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/latest/dra-intro-install.html) - [DRA Driver for NVIDIA GPUs](https://dra-driver-nvidia-gpu.sigs.k8s.io/docs/)
Page source Source snapshot ยท Revision `f928834c8`.