#NVIDIA GPU Operator ownership

Status: live in Icarus Cloud. On 26 October 2026 Argo CD adopted the existing installation at source revision 808f1a31f87712282892be97116579f77517f950 and applied GPU Operator v26.7.2. This page records the ownership boundary and the live adoption evidence. It does not claim a fresh Debian 13 installation.

#Decision

The deployment uses two owners because they operate on different state:

This follows ADR-0040: host prerequisites are foundation state, while cluster workloads have an exact-revision GitOps owner. An Ansible play must target an explicit GPU-host group. It must not run against the dev-box or derive GPU work from all inventory hosts. The GPU Operator Application is an opt-in Kustomize component and is absent from the dev-box rendering.

#Pinned installation source

The committed chart is v26.7.2, the same release devops/platform/profiles/host-profiles.yml pins for the environment and gitops/manifests/gpu-operator/pins.yaml names with its archive, tag object and source commit. Its official archive, source tag object, dereferenced commit and NFD dependency are recorded in pins.yaml. The component lives at gitops/apps/components/gpu-operator. A small Pharos wrapper chart installs the derived, checksum-pinned archive from chart/charts/. The original NVIDIA archive is retained under upstream/, outside Helm's dependency directory. Both identities are recorded in pins.yaml. scripts/prepare_gpu_chart.py derives the installed archive deterministically: only gpu-operator/templates/clusterrole.yaml changes. Two exact rules are reduced to get/list/watch: CRDs and admission policies/bindings. Every other member's bytes and metadata are preserved. Verification rejects source-rule drift, archive drift, DRA GPUCluster or NVIDIADriver enablement, disabled ClusterPolicy and operator.cleanupCRD. The upgrade hook retains its reviewed CRD-write grant; Argo owns CRD lifecycle.

This reduction applies to the conventional ClusterPolicy profile. In the pinned source controller, CRD permissions are read-only; its CRD lookup uses Get. The conventional resource manager has no CRD handler; generic CRD reconciliation and cleanup belong to the inactive GPUCluster/NVIDIADriver state managers. Verification keeps driver.nvidiaDriverCRD.enabled=false. The source's admission policy manifests belong to the DRA operand. The separate GPUCluster reconciler requires a GPUCluster object before synchronising those states. Bootstrap checks that no such object exists before installing this profile. Enabling DRA or CRD cleanup requires a fresh permission and workload review.

To reproduce the installed source and its generated delegation policy:

uv run --project devops --frozen python scripts/prepare_gpu_chart.py
uv run --project devops --frozen python scripts/generate_gpu_admission.py

The reductions prevent the operator from withdrawing admission guards or granting CRD-write permissions to another account. Kubernetes does not run API-based ValidatingAdmissionPolicy against admission policies or bindings, so revocation is prevented through RBAC. These permission changes are qualified against a disposable Kubernetes API. GPU workloads and a live GPU rollout with the derived archive remain unqualified; the adoption evidence below describes the earlier upstream installation.

The values preserve host ownership and implement the reviewed allocation below:

• driver.enabled=false and toolkit.enabled=false: Ansible owns host software.
• driver.version is empty, so the chart omits it from the ClusterPolicy: nothing installs a driver from the cluster, and the two host profiles run different releases. The version that is actually running lives in devops/platform/profiles/host-profiles.yml, and scripts/test_gpu_host_verification.py fails if the operator values ever name it. The 595.91.07 the live ClusterPolicy used to declare was only a declaration: with driver.enabled=false the operator installs no driver, so it described nothing the node ran.
• mig.strategy=mixed and migManager.enabled=true: the manager applies the one geometry each GH200 node is labelled with, and leaves unlabelled nodes alone.
• The device plugin stays enabled and serves the L40 hosts only; its per-node configurations are described under the reviewed allocation.
• CDI is enabled and NRI is disabled.
• The conventional ClusterPolicy is installed; DRA GPUCluster is not.
• NVIDIA is not the default containerd runtime, timeSlicing is still never set, and no RuntimeClass or taint is derived from the allocation.

The Application remains manually synchronised by omitting Argo's automated sync policy altogether. The reviewed live operation ran one explicit sync at the exact source revision above. Ordinary delivery therefore records the desired object but does not silently change the GPU stack. Enabling automated sync remains a separate reviewed operation.

The Application uses Argo's server-side diff because the live Argo v2.12.6 controller predates Kubernetes v1.35 status fields. Without it, comparison fails on the API server's own Deployment schema before a sync can be reviewed.

The committed icarus-cloud environment selects the component, while the common and disposable-lab policies do not. Normal delivery creates the manual child Application and excludes it from the ready-set wait. The dev-box overlay never selects it.

The child also carries pharos.platform/deferred-sync: "true". The platform's Application health rule treats a deferred child as settled unless it is being operated on, because pharos-platform otherwise stays Progressing — and holds every later sync wave — while the manual child is OutOfSync/Missing. The child still reports its own running operation and its error conditions, so the explicit GPU operation keeps its evidence.

The destination is https://kubernetes.default.svc:443, the self-management alias the platform Argo already registers. cluster.inClusterEnabled is false, so https://kubernetes.default.svc is not a cluster Argo can reach; the alias also registers the gpu-operator namespace and caches cluster-scoped resources. The bootstrap grants the application controller, by name, the chart's ClusterRoles, ClusterRoleBindings, namespaced Roles and bindings, eight NVIDIA/NFD CRDs and the cluster-policy ClusterPolicy. Broad CRD access is read-only for cache discovery. Named role grants include escalate and bind; a generated admission policy requires the exact reviewed rules, roleRefs and subjects for all 14 chart RBAC objects and rejects aggregation. A separate actor-based policy limits Argo and the upgrade hook to the GPU CRDs, preserving Argo's own three CRDs. Because the API server's escalation check carries no object name, escalate itself stays unnamed, so a third generated policy bounds the actor instead of the chart: the application controller may create or change only the cluster roles the platform's own grants review — the names the pinned GPU charts render and the names the kubeflow, site-agent, gpu-operator and gpu-dra bootstraps reserve — it may bind only those, and no cluster role it writes may aggregate — except the five Kubeflow roles the composition renders with an aggregationRule (kubeflow-admin, kubeflow-edit, kubeflow-view and the two poddefaults-webhook-kubeflow-poddefaults-* roles), whose selectors are pinned to the rendered ones. Argo creates these roles with empty rules and preserves their existing rules on update, including during server-side apply. Kubernetes' aggregation controller owns subsequent rule changes; its requests remain outside the Argo policy. This prevents direct permission changes taking effect before aggregation reconciles. ClusterRole labels are pinned to their rendered values, so Argo cannot add another role's permissions by changing membership of Kubeflow, built-in or external aggregates. Argo's configured annotation-based tracking remains supported. Those names are read from the grants, and the aggregated selectors from the composition render, when the policy is generated, so a role nobody reviewed has no entry. Bootstrap installs both policies and their Deny bindings, waits for compilation and rejects type warnings before granting controller rights. The existing writer policy also admits the upgrade hook and the garbage collector for its reviewed delete path.

The bootstrap's own objects are named pharos-gpu-operator-writer, outside the GPU-owned names. Restricted identities cannot change the admission objects; an administrator with cluster-admin rights can recover them without membership of system:masters. The separate original GPU writer fence retains its existing administrator-group recovery rules. scripts/test_gpu_operator.py checks the rendered names and bootstrap ordering; scripts/gpu_admission_smoke.py tests actual chart permissions, direct and delegated CRD denials, policy revocation and administrator recovery on an isolated API server. Its aggregation cases reject direct Argo rule changes through create, replace, patch and server-side apply, and check that native aggregation and metadata-only Argo synchronisation still work.

The gpu-operator and gpu-dra namespaces are bootstrap prerequisites. Argo does not create namespaces or use CreateNamespace=true. A selected GPU environment must create them through its reviewed bootstrap path before syncing the components.

#Reviewed accelerator allocation

Decided on 1 November 2026 and narrowed on 2 November 2026. The four GH200 nodes move to DRA; the L40 hosts keep the device plugin, because the device plugin v0.20.1 this release pins cannot rename an extended resource and those hosts keep driver 550.163.01. Both remain ordinary delivery: the labels are written by platform_deploy/accelerators.py, and nothing in this allocation is set by hand.

mps-4 stays the reviewed sharing of openst-gpu06: four nvidia.com/gpu.shared shares of one card. The device plugin starts the MPS control daemon by looking nvidia-cuda-mps-control up on its own PATH; the plugin image does not carry the binary, and the container runtime only injects what the host driver provides. While the host driver package lacks it, the daemon exits with executable file not found and the pod CrashLoops, so the environment declares whole-gpu for that host. The driver package belongs to the foundation (gpu-host role); once it ships the binary, the one line in devops/platform/environments/icarus-cloud.yml returns to mps-4.

The geometry is per device, not per node, and that is the reviewed design rather than a stopgap. Only GPU 0 is sliced: GPU 1 of a GH200 NVL2 takes no MIG geometry on this platform. Against driver 610.43.02 MIG on GPU 1 fails with a GSP INSUFFICIENT_RESOURCES, followed by NV_ERR_TIMEOUT in MIGMGR_SET_GPU_INSTANCES, and the trial driver 580 fails the same way while GPU 0 succeeds on both (see icarus-deployment-state.md §26–§29). The environment policy therefore names the geometry as {device, profile, slices} and platform_gitops/gpu_mig.py validates it: the device must be GPU 0, the profile must be one the class map publishes, and the slice count must fill the device. The same module writes the operator chart's mig-configs document, so the key delivery labels a node with and the entry the MIG manager applies cannot drift apart. A node that returns to whole GPUs is reset with all-disabled, which is the chart's default.

A sliced host publishes exactly one whole GPU beside its slices, and that GPU is neither shared nor usable together with the slices. The NVIDIA driver provides no NCCL or peer-to-peer path between MIG instances, or between a MIG instance and a GPU without MIG, so a workload placed on cluster-node-22, cluster-node-23 or cluster-node-24 can never hold both of the host's GPUs at once. No class expresses such a pair, and the offering does not invent one: a job that needs two GPUs across a NVLink pair needs an unsliced host. The only such GH200 host in this layout is cluster-node-21, reserved for shared inference; no two-card GH200 pair is available to ordinary user workloads. Outside that reservation, the pool publishes three whole GH200 GPUs, fourteen 1g.18gb slices and two 3g.72gb slices. The rejected alternative was decided against on 2 November 2026: the pinned dra-driver-nvidia-gpu 0.5.1 carries no GpuConfig resource at all (its CRDs are the two ComputeDomain ones), so declarative GpuConfig.sharing with TimeSlicing or MPS is not available in this release, and its only MPS path is the experimental MPSSupport feature gate with a control daemon. No "GH200 shared" class is offered.

cluster-node-21 keeps both cards whole for the shared Mistral model, and delivery taints it pharos.platform/inference=true:NoSchedule. The DRA driver's own daemonset tolerates that taint; the inference workloads must tolerate it too. Without the taint a notebook or a Run asking for a whole GH200 could be placed on the node the shared model owns, because the gh200 class covers every whole GH200 DRA publishes. The 2g.36gb class is not offered while no node cuts that profile.

openst-gpu06 advertises four MPS clients per card, so eight shares of about 11 GiB. MPS caps each client's pinned device memory at totalMemory/replicas, which time-slicing does not; the control daemon sets every card to EXCLUSIVE_PROCESS, and the plugin fails any request for more than one share (failRequestsGreaterThanOne: true). The MPS control daemon must be Ready before the plugin advertises the resource: the plugin retries until it is, so a delivery that finishes before the daemon is healthy is not a success, and the DaemonSet can sit in CrashLoopBackOff until then.

A DeviceClass publishes its extended resource only while the DRAExtendedResource gate is enabled in the kube-apiserver, the kube-scheduler and the kubelet. Delivery reads the API server's /metrics and each DRA node's kubelet /metrics before it fences a node, and refuses to continue when either reports the gate off: a Site without the gate keeps the device plugin instead of losing its GPUs.

Nothing else names an accelerator resource. gitops/manifests/gpu-acceleration/classes.yaml maps each class to its resource and binds it to the offering component, the quota key and the DRA DeviceClass; scripts/generate_gpu_offering.py writes the agent's components from it, and scripts/prepare_gpu_dra.py writes the chart's DeviceClass objects from it. Two classes may not share a resource, and a class the map does not contain is refused at Site Agent startup. The DRA chart no longer ships its own gpu.nvidia.com DeviceClass, so nothing aliases nvidia.com/gpu to whatever whole GPU DRA happens to discover; whole L40 cards are exactly the two the openst-gpu05 device plugin advertises.

#MPS on the L40 hosts

MPS needs two things this release does not get for free, and a delivery that misses either one leaves openst-gpu06 without nvidia.com/gpu.shared:

1. The MPS binaries of the host driver. The MPS control daemon runs exec.Command("nvidia-cuda-mps-control", "-d") with a plain PATH lookup, and the node image the plugin runs (nvcr.io/nvidia/k8s-device-plugin:v0.20.1) does not carry that binary: the NVIDIA Container Toolkit injects it from the host driver install, like nvidia-smi. On 2 November 2026 the L40 hosts had driver 550.163.01-2 installed without the nvidia-cuda-mps package, so the generated CDI spec listed /usr/bin/nvidia-smi and nothing else, and the daemon crash-looped with exec: "nvidia-cuda-mps-control": executable file not found in $PATH. The package is nvidia-cuda-mps=550.163.01-2 (Debian, same version as nvidia-driver), it provides /usr/bin/nvidia-cuda-mps-control and /usr/bin/nvidia-cuda-mps-server, and it belongs to the host profile, which this repository records as host-package-unowned-in-pharos-platform: it is installed on the host, never from a cluster manifest. Verify before a delivery enables MPS:

   dpkg -s nvidia-cuda-mps | head -3
   command -v nvidia-cuda-mps-control nvidia-cuda-mps-server
   grep -c nvidia-cuda-mps /var/run/cdi/nvidia.yaml

2. A device plugin that does not run the mixed MIG strategy. The plugin stops at startup with error starting plugins: unable to validate flags: using --mig-strategy=mixed is not supported with MPS, and the operator copies the ClusterPolicy's mig.strategy into the plugin. That strategy stays mixed for the GPU feature discovery of the MIG-carrying GH200 nodes, so base/values.yaml overrides it on the plugin alone through devicePlugin.env. This holds because the plugin runs only on the L40 hosts: the GH200 nodes are fenced off with nvidia.com/gpu.deploy.device-plugin=false and served by DRA, so the plugin never has to expose a MIG device. mps_errors in scripts/gpu_checks.py refuses any configuration that shares devices with MPS without the override, without devicePlugin.mps.root, or without the per-client isolation, and scripts/test_gpu_operator.py applies it to both the reviewed values and the rendered ClusterPolicy.

A Foundation that installs nvidia-cuda-mps on the L40 hosts, and delivery that waits for the MPS control daemon to be Ready before the plugin advertises the shares, are both prerequisites for re-enabling nvidia.com/gpu.shared; until then the node stays on whole-gpu.

#Migrating a GH200 node

A MIG geometry needs the card idle, and the driver may need a reset or a reboot before it applies, so delivery cordons and drains the node, writes the labels, waits for nvidia.com/mig.config.state=success, restarts the DRA kubelet plugin, which discovers slices only at startup, and uncordons the node only after that restart. The restart is recorded as pharos.platform/dra-rediscovery=pending in the same write as the geometry, and removed after it: a delivery that dies in between leaves the node cordoned with the marker, and the next delivery completes the restart before it releases the node instead of leaving a plugin that still names destroyed slices. The same order holds when a node keeps DRA and returns to whole GPUs. A node that reports no success in time stays cordoned. The shared models run on cluster-node-21, which has no MIG label and is therefore left whole.

A whole-GPU node that changes allocation owner — the device plugin taking it back, or DRA taking it — is drained first and released only when the new plugin publishes the cards and the old one has stopped: the device plugin advertises nvidia.com/gpu in the node's allocatable, the DRA driver publishes ResourceSlices, and the pods the previous plugin placed keep the devices they were given. A plugin that never publishes leaves the node cordoned for the next delivery to resume.

Rolling back restores the previous labels through the environment policy: a node leaving DRA loses pharos.platform/gpu-dra, regains nvidia.com/gpu.deploy.device-plugin, and the MPS and MIG labels are removed. Reconfiguring MIG back to whole GPUs drains the node again.

#Live adoption on 26 October 2026

The pre-change private snapshot recorded the operator 26.3.3 installation, ClusterPolicy, RuntimeClasses, Helm release metadata and workload state. The reviewed Argo sync started at 2026-10-26T11:28:21Z, completed at 2026-10-26T11:28:38Z, and reported Succeeded with dryRun=false. The Application is Synced/Healthy at revision 808f1a31f87712282892be97116579f77517f950.

The GFD, device-plugin, DCGM and validator DaemonSets each reached 6/6 Ready and 6/6 updated; the NFD worker reached 9/9. ClusterPolicy cluster-policy reached Ready=True, generation 4. The cluster UID 3ccee890-dcd6-4ffc-8f56-e7f43dcbf9a2, GPU namespace UID d2b28791-a4ee-45dc-8f59-adc5a0cedfe1, ClusterPolicy UID 24867769-9785-440d-9f6b-349fb93c2aad and RuntimeClass UIDs were preserved.

At adoption the live values retained driver.enabled=false, toolkit.enabled=false, migManager.enabled=false, mig.strategy=mixed, CDI enabled and NRI disabled. No Driver or GPUCluster custom resource exists. No default RuntimeClass, taint, label, MIG allocation, DRA driver, time-slicing profile, driver, toolkit or containerd configuration was changed by that adoption; the allocation that follows was reviewed and delivered separately.

Argo renders and applies the Helm chart without updating Helm storage. The legacy Helm release metadata therefore still reports chart/app v26.3.3, revision 3; the profile records that release as legacy_helm_release with the role forensic-only, so the profile, the pins, the chart and this page name v26.7.2 as the one release that runs and that the cutover installs. That record is retained for forensics and rollback reference, not as live desired state. Running an independent helm upgrade or helm rollback would create a second owner and is not a supported maintenance path while Argo owns these resources.

Live Argo v2.12.6 exposed three compatibility requirements now committed in the Application source: use a registered in-cluster destination (https://kubernetes.default.svc:443), express a manual sync by omitting automated, and enable ServerSideDiff=true because local typed comparison cannot decode the Kubernetes v1.35 Deployment status field .status.terminatingReplicas.

#Version and support boundaries

Sources checked on 26 October 2026:

• NVIDIA GPU Operator platform support
• NVIDIA GPU Operator release notes

26.7.x is the supported current line in those documents. 26.3.x is marked deprecated and 25.10.x and earlier are end of support. NVIDIA validates Kubernetes 1.33–1.37 on listed bare-metal operating systems and versions; the exact Debian 13 combinations at Icarus are not listed in NVIDIA's validated OS table. GH200 and L40 are supported GPU products, but that does not establish a supported fresh Debian 13/GH200-64KiB/L40 installation recipe.

The completed operator upgrade path is 26.3.3 to 26.7.2. It cannot be used as evidence for a driver, toolkit, containerd, MIG, DRA or fresh-OS transition.

#Acceptance

The adoption has current evidence for all acceptance gates:

1. The cluster, GPU namespace, ClusterPolicy and RuntimeClass identities were preserved, and the Argo operation completed successfully at the exact source revision.
2. The operator, NFD, GFD, device plugin, DCGM Exporter and validator are Ready.
3. All six GPU nodes still advertise two nvidia.com/gpu devices, for 12 total; labels, taints, RuntimeClasses and conventional device allocation are retained.
4. Native CUDA vectorAdd succeeded after the operator change on one L40 and one GH200. The jobs ran one at a time, with one GPU, bounded CPU/memory, a deadline, zero retries, the normal scheduler, runtimeClassName: nvidia, no privileged mode, no host mount and no service-account token. The temporary PriorityClass was owner-labelled, globalDefault=false, value -1000, and non-preempting.
5. Ceph remained HEALTH_OK with FSID d75dc3d0-ce56-4226-970b-c1f2977cec5f, 76 OSDs, quorum a,b,c and 265/265 PGs active+clean; 26/26 PVCs remained Bound. There were no Pending pods and no non-Ready containers in Running pods. The seven Failed pods were the same retained registry-prune and old Rook exporter failures observed before the operator change.

Temporary namespace UID 317e6645-6181-48e6-86cd-a05c789c45b9 and PriorityClass UID 94703e08-36c8-4ef6-8d78-bf9670d9b461 were removed with API UID preconditions after confirming their only Pod references were the two owned acceptance Pods. Both returned HTTP 404 afterwards.

#Recovery and rollback

Private pre- and post-change snapshots are held in 0700 administrator directories with 0600 payloads. The pre-change archive SHA-256 is f56f245e063ea0f5126690f36bc69e9af00039748ab592eb063113ba14a78f85; the verified post-change archive SHA-256 is 4eb462a2f2e89d68a114fb511837c6ff5def34c181dafd4b58f143f538c94143. They include the Argo Application, Helm history/values/manifest and release secrets, NVIDIA CRDs/CRs, ClusterPolicy, RuntimeClasses, GPU namespace and logs. They are not published in Git.

A routine downgrade is not defined. Helm rollback is not authoritative because Argo applies the chart without updating Helm release storage. Recovering the operator requires a separately reviewed GitOps diff and sync, informed by the private snapshots. Deleting the GPU namespace or NVIDIA CRDs is destructive, removes cluster-wide API state, and is not a rollback. Changes to host driver, toolkit, containerd, MIG or DRA are outside this boundary.

The host layer is owned by the gpu-host Ansible role, driven by devops/platform/profiles/host-profiles.yml. Each profile declares where its driver comes from:

• L40 (apt-package): nvidia-driver, nvidia-cuda-mps and nvidia-container-toolkit are installed at pinned apt specs from the Debian repositories the host already has, then held.
• GH200 (preinstalled-package): the driver is not built into the kernel. It is nvidia.ko inside linux-modules-6.14.0-1015-nvidia-64k 6.14.0-1015.15, which ships with the vendor image alongside linux-image-unsigned-6.14.0-1015-nvidia-64k 6.14.0-1015.15; both were installed by dpkg with no apt history entry and no repository configured on the node. The role therefore verifies those exact versions with dpkg-query, verifies the module path and kernel release with modinfo, and holds the packages; it never installs them.
• A locally built driver against the pinned kernel (local-build) is described by the profile vocabulary and validated, but refused by the role until a reviewed build task exists.

No automatic upgrade path exists on those nodes: no unattended-upgrades, no DKMS and no pins or holds before this role. The role adds the holds and the blocklist.

Two host details the role reproduces as they are, rather than as they are usually documented:

• The CDI specification lives under /var/run/cdi, where the running toolkit publishes it; the profiles name that path and there is no /etc/cdi/nvidia.yaml on these nodes.
• /etc/containerd/conf.d/99-nvidia.toml carries the containerd 2.x runtime section and a legacy io.containerd.grpc.v1.cri block left by the earlier setup. The role verifies the file against the profile by default and accepts it when the declared runtime is present; rewriting it drops the legacy block, so rendering and the containerd restart that applies it are explicit and belong to a maintenance window.

The remaining ownership gap is fresh-OS reproducibility, and on GH200 it is a provenance gap: the kernel and modules packages that carry the driver have no reviewed source, so the repository can pin and verify them but cannot recreate them from a repository.

Negative tests reject a floating chart, a changed source commit or archive, a wrapper dependency that differs from the reviewed release, an automatic adoption sync, an enabled driver or toolkit, a device-plugin configuration outside the reviewed set, time-slicing insertion, a DRA DeviceClass that aliases nvidia.com/gpu, and accidental selection on the dev-box.