# NVIDIA GPU Operator ownership Status: live in Icarus Cloud. On 26 October 2026 Argo CD adopted the existing installation at source revision `808f1a31f87712282892be97116579f77517f950` and applied GPU Operator `v26.7.2`. This page records the ownership boundary and the live adoption evidence. It does not claim a fresh Debian 13 installation. ## Decision The deployment uses two owners because they operate on different state: | State | Owner | Selected by | | --- | --- | --- | | NVIDIA kernel driver and headers | `gpu-host` Ansible role | An inventory `gpu_host` group only | | NVIDIA Container Toolkit and containerd NVIDIA runtime fragment | `gpu-host` Ansible role | The same `gpu_host` group only | | Host GPU/CDI health gate | `gpu-host` Ansible role | The same `gpu_host` group only | | GPU Operator, CRDs and cluster policy | Argo CD Application | An explicit environment component | | Device plugin (L40 hosts), GFD, DCGM Exporter and validator | GPU Operator | The reviewed ClusterPolicy | | MIG Manager (GH200 hosts) | GPU Operator | The geometry delivery labels per node | | DRA driver and the GH200 `DeviceClass` objects | A second Argo CD Application, `gpu-dra` | The same environment component list | | Accelerator class names, pods and quota keys | The class map under `gitops/manifests/gpu-acceleration` | Regenerated, never hand-written | This follows [ADR-0040](decisions/0040-platform-layout.md): host prerequisites are foundation state, while cluster workloads have an exact-revision GitOps owner. An Ansible play must target an explicit GPU-host group. It must not run against the dev-box or derive GPU work from all inventory hosts. The GPU Operator Application is an opt-in Kustomize component and is absent from the dev-box rendering. ## Pinned installation source The committed chart is `v26.7.2`, the same release `devops/platform/profiles/host-profiles.yml` pins for the environment and `gitops/manifests/gpu-operator/pins.yaml` names with its archive, tag object and source commit. Its official archive, source tag object, dereferenced commit and NFD dependency are recorded in [`pins.yaml`](../gitops/manifests/gpu-operator/pins.yaml). The component lives at [`gitops/apps/components/gpu-operator`](../gitops/apps/components/gpu-operator). A small Pharos wrapper chart installs the derived, checksum-pinned archive from `chart/charts/`. The original NVIDIA archive is retained under `upstream/`, outside Helm's dependency directory. Both identities are recorded in `pins.yaml`. `scripts/prepare_gpu_chart.py` derives the installed archive deterministically: only `gpu-operator/templates/clusterrole.yaml` changes. Two exact rules are reduced to `get/list/watch`: CRDs and admission policies/bindings. Every other member's bytes and metadata are preserved. Verification rejects source-rule drift, archive drift, DRA GPUCluster or NVIDIADriver enablement, disabled ClusterPolicy and `operator.cleanupCRD`. The upgrade hook retains its reviewed CRD-write grant; Argo owns CRD lifecycle. This reduction applies to the conventional ClusterPolicy profile. In the pinned [source controller](https://github.com/NVIDIA/gpu-operator/blob/5a9c8a8b3f5bac7fb06ad3c2a05771fceb8b4832/controllers/clusterpolicy_controller.go#L87), CRD permissions are read-only; its [CRD lookup](https://github.com/NVIDIA/gpu-operator/blob/5a9c8a8b3f5bac7fb06ad3c2a05771fceb8b4832/controllers/object_controls.go#L4988) uses `Get`. The conventional resource manager has no CRD handler; generic CRD reconciliation and cleanup belong to the inactive GPUCluster/NVIDIADriver state managers. Verification keeps `driver.nvidiaDriverCRD.enabled=false`. The source's admission policy manifests belong to the [DRA operand](https://github.com/NVIDIA/gpu-operator/blob/5a9c8a8b3f5bac7fb06ad3c2a05771fceb8b4832/manifests/state-dra-driver/0700_resourceslices-policy.yaml). The separate [GPUCluster reconciler](https://github.com/NVIDIA/gpu-operator/blob/5a9c8a8b3f5bac7fb06ad3c2a05771fceb8b4832/controllers/gpucluster_controller.go#L80) requires a GPUCluster object before synchronising those states. Bootstrap checks that no such object exists before installing this profile. Enabling DRA or CRD cleanup requires a fresh permission and workload review. To reproduce the installed source and its generated delegation policy: ```sh uv run --project devops --frozen python scripts/prepare_gpu_chart.py uv run --project devops --frozen python scripts/generate_gpu_admission.py ``` The reductions prevent the operator from withdrawing admission guards or granting CRD-write permissions to another account. Kubernetes does not run API-based [ValidatingAdmissionPolicy against admission policies or bindings](https://kubernetes.io/docs/reference/kubernetes-api/admissionregistration/validating-admission-policy-v1/), so revocation is prevented through RBAC. These permission changes are qualified against a disposable Kubernetes API. GPU workloads and a live GPU rollout with the derived archive remain unqualified; the adoption evidence below describes the earlier upstream installation. The values preserve host ownership and implement the reviewed allocation below: - `driver.enabled=false` and `toolkit.enabled=false`: Ansible owns host software. - `driver.version` is empty, so the chart omits it from the ClusterPolicy: nothing installs a driver from the cluster, and the two host profiles run different releases. The version that is actually running lives in `devops/platform/profiles/host-profiles.yml`, and `scripts/test_gpu_host_verification.py` fails if the operator values ever name it. The `595.91.07` the live ClusterPolicy used to declare was only a declaration: with `driver.enabled=false` the operator installs no driver, so it described nothing the node ran. - `mig.strategy=mixed` and `migManager.enabled=true`: the manager applies the one geometry each GH200 node is labelled with, and leaves unlabelled nodes alone. - The device plugin stays enabled and serves the L40 hosts only; its per-node configurations are described under [the reviewed allocation](#reviewed-accelerator-allocation). - CDI is enabled and NRI is disabled. - The conventional ClusterPolicy is installed; DRA GPUCluster is not. - NVIDIA is not the default containerd runtime, `timeSlicing` is still never set, and no `RuntimeClass` or taint is derived from the allocation. The Application remains manually synchronised by omitting Argo's `automated` sync policy altogether. The reviewed live operation ran one explicit sync at the exact source revision above. Ordinary delivery therefore records the desired object but does not silently change the GPU stack. Enabling automated sync remains a separate reviewed operation. The Application uses Argo's server-side diff because the live Argo `v2.12.6` controller predates Kubernetes `v1.35` status fields. Without it, comparison fails on the API server's own Deployment schema before a sync can be reviewed. The committed `icarus-cloud` environment selects the component, while the common and disposable-lab policies do not. Normal delivery creates the manual child Application and excludes it from the ready-set wait. The dev-box overlay never selects it. The child also carries `pharos.platform/deferred-sync: "true"`. The platform's Application health rule treats a deferred child as settled unless it is being operated on, because `pharos-platform` otherwise stays `Progressing` — and holds every later sync wave — while the manual child is `OutOfSync`/`Missing`. The child still reports its own running operation and its error conditions, so the explicit GPU operation keeps its evidence. The destination is `https://kubernetes.default.svc:443`, the self-management alias the platform Argo already registers. `cluster.inClusterEnabled` is false, so `https://kubernetes.default.svc` is not a cluster Argo can reach; the alias also registers the `gpu-operator` namespace and caches cluster-scoped resources. The bootstrap grants the application controller, by name, the chart's ClusterRoles, ClusterRoleBindings, namespaced Roles and bindings, eight NVIDIA/NFD CRDs and the `cluster-policy` ClusterPolicy. Broad CRD access is read-only for cache discovery. Named role grants include `escalate` and `bind`; a generated admission policy requires the exact reviewed rules, roleRefs and subjects for all 14 chart RBAC objects and rejects aggregation. A separate actor-based policy limits Argo and the upgrade hook to the GPU CRDs, preserving Argo's own three CRDs. Because the API server's escalation check carries no object name, `escalate` itself stays unnamed, so a third generated policy bounds the actor instead of the chart: the application controller may create or change only the cluster roles the platform's own grants review — the names the pinned GPU charts render and the names the `kubeflow`, `site-agent`, `gpu-operator` and `gpu-dra` bootstraps reserve — it may bind only those, and no cluster role it writes may aggregate — except the five Kubeflow roles the composition renders with an `aggregationRule` (`kubeflow-admin`, `kubeflow-edit`, `kubeflow-view` and the two `poddefaults-webhook-kubeflow-poddefaults-*` roles), whose selectors are pinned to the rendered ones. Argo creates these roles with empty rules and preserves their existing rules on update, including during server-side apply. Kubernetes' aggregation controller owns subsequent rule changes; its requests remain outside the Argo policy. This prevents direct permission changes taking effect before aggregation reconciles. ClusterRole labels are pinned to their rendered values, so Argo cannot add another role's permissions by changing membership of Kubeflow, built-in or external aggregates. Argo's configured annotation-based tracking remains supported. Those names are read from the grants, and the aggregated selectors from the composition render, when the policy is generated, so a role nobody reviewed has no entry. Bootstrap installs both policies and their Deny bindings, waits for compilation and rejects type warnings before granting controller rights. The existing writer policy also admits the upgrade hook and the garbage collector for its reviewed delete path. The bootstrap's own objects are named `pharos-gpu-operator-writer`, outside the GPU-owned names. Restricted identities cannot change the admission objects; an administrator with cluster-admin rights can recover them without membership of `system:masters`. The separate original GPU writer fence retains its existing administrator-group recovery rules. `scripts/test_gpu_operator.py` checks the rendered names and bootstrap ordering; `scripts/gpu_admission_smoke.py` tests actual chart permissions, direct and delegated CRD denials, policy revocation and administrator recovery on an isolated API server. Its aggregation cases reject direct Argo rule changes through create, replace, patch and server-side apply, and check that native aggregation and metadata-only Argo synchronisation still work. The `gpu-operator` and `gpu-dra` namespaces are bootstrap prerequisites. Argo does not create namespaces or use `CreateNamespace=true`. A selected GPU environment must create them through its reviewed bootstrap path before syncing the components. ## Reviewed accelerator allocation Decided on 1 November 2026 and narrowed on 2 November 2026. The four GH200 nodes move to DRA; the L40 hosts keep the device plugin, because the device plugin `v0.20.1` this release pins cannot rename an extended resource and those hosts keep driver `550.163.01`. Both remain ordinary delivery: the labels are written by `platform_deploy/accelerators.py`, and nothing in this allocation is set by hand. | Hosts | Served by | Advertised resource | Delivery label | | --- | --- | --- | --- | | `cluster-node-21` | DRA driver, both GPUs whole, reserved for inference | `nvidia.com/gh200` | `pharos.platform/gpu-dra=true`, `pharos.platform/inference=true:NoSchedule` | | `cluster-node-22` | DRA driver, GPU 0 as `1g.18gb` × 7, GPU 1 whole | `nvidia.com/gh200-mig-1g.18gb`, `nvidia.com/gh200` | `pharos.platform/gpu-dra=true`, `nvidia.com/mig.config=gpu0-1g.18gb-x7` | | `cluster-node-23` | DRA driver, GPU 0 as `1g.18gb` × 7, GPU 1 whole | `nvidia.com/gh200-mig-1g.18gb`, `nvidia.com/gh200` | `pharos.platform/gpu-dra=true`, `nvidia.com/mig.config=gpu0-1g.18gb-x7` | | `cluster-node-24` | DRA driver, GPU 0 as `3g.72gb` × 2, GPU 1 whole | `nvidia.com/gh200-mig-3g.72gb`, `nvidia.com/gh200` | the same, `nvidia.com/mig.config=gpu0-3g.72gb-x2` | | `openst-gpu05` | Device plugin, `whole-gpu` | `nvidia.com/gpu` | `nvidia.com/device-plugin.config=whole-gpu` | | `openst-gpu06` | Device plugin, `whole-gpu` until the host ships the MPS binaries; `mps-4` is the reviewed sharing | `nvidia.com/gpu`; `nvidia.com/gpu.shared` with `mps-4` | `nvidia.com/device-plugin.config=whole-gpu` | `mps-4` stays the reviewed sharing of `openst-gpu06`: four `nvidia.com/gpu.shared` shares of one card. The device plugin starts the MPS control daemon by looking `nvidia-cuda-mps-control` up on its own `PATH`; the plugin image does not carry the binary, and the container runtime only injects what the host driver provides. While the host driver package lacks it, the daemon exits with `executable file not found` and the pod CrashLoops, so the environment declares `whole-gpu` for that host. The driver package belongs to the foundation (`gpu-host` role); once it ships the binary, the one line in `devops/platform/environments/icarus-cloud.yml` returns to `mps-4`. The geometry is per device, not per node, and that is the reviewed design rather than a stopgap. Only GPU 0 is sliced: GPU 1 of a GH200 NVL2 takes no MIG geometry on this platform. Against driver `610.43.02` MIG on GPU 1 fails with a GSP `INSUFFICIENT_RESOURCES`, followed by `NV_ERR_TIMEOUT` in `MIGMGR_SET_GPU_INSTANCES`, and the trial driver `580` fails the same way while GPU 0 succeeds on both (see `icarus-deployment-state.md` §26–§29). The environment policy therefore names the geometry as `{device, profile, slices}` and `platform_gitops/gpu_mig.py` validates it: the device must be GPU 0, the profile must be one the class map publishes, and the slice count must fill the device. The same module writes the operator chart's `mig-configs` document, so the key delivery labels a node with and the entry the MIG manager applies cannot drift apart. A node that returns to whole GPUs is reset with `all-disabled`, which is the chart's default. A sliced host publishes exactly one whole GPU beside its slices, and that GPU is neither shared nor usable together with the slices. The NVIDIA driver provides no NCCL or peer-to-peer path between MIG instances, or between a MIG instance and a GPU without MIG, so a workload placed on `cluster-node-22`, `cluster-node-23` or `cluster-node-24` can never hold both of the host's GPUs at once. No class expresses such a pair, and the offering does not invent one: a job that needs two GPUs across a NVLink pair needs an unsliced host. The only such GH200 host in this layout is `cluster-node-21`, reserved for shared inference; no two-card GH200 pair is available to ordinary user workloads. Outside that reservation, the pool publishes three whole GH200 GPUs, fourteen `1g.18gb` slices and two `3g.72gb` slices. The rejected alternative was decided against on 2 November 2026: the pinned `dra-driver-nvidia-gpu` `0.5.1` carries no `GpuConfig` resource at all (its CRDs are the two ComputeDomain ones), so declarative `GpuConfig.sharing` with `TimeSlicing` or `MPS` is not available in this release, and its only MPS path is the experimental `MPSSupport` feature gate with a control daemon. No "GH200 shared" class is offered. `cluster-node-21` keeps both cards whole for the shared Mistral model, and delivery taints it `pharos.platform/inference=true:NoSchedule`. The DRA driver's own daemonset tolerates that taint; the inference workloads must tolerate it too. Without the taint a notebook or a Run asking for a whole GH200 could be placed on the node the shared model owns, because the `gh200` class covers every whole GH200 DRA publishes. The `2g.36gb` class is not offered while no node cuts that profile. `openst-gpu06` advertises four MPS clients per card, so eight shares of about 11 GiB. MPS caps each client's pinned device memory at `totalMemory/replicas`, which time-slicing does not; the control daemon sets every card to `EXCLUSIVE_PROCESS`, and the plugin fails any request for more than one share (`failRequestsGreaterThanOne: true`). The MPS control daemon must be Ready before the plugin advertises the resource: the plugin retries until it is, so a delivery that finishes before the daemon is healthy is not a success, and the DaemonSet can sit in `CrashLoopBackOff` until then. A `DeviceClass` publishes its extended resource only while the `DRAExtendedResource` gate is enabled in the kube-apiserver, the kube-scheduler and the kubelet. Delivery reads the API server's `/metrics` and each DRA node's kubelet `/metrics` before it fences a node, and refuses to continue when either reports the gate off: a Site without the gate keeps the device plugin instead of losing its GPUs. Nothing else names an accelerator resource. `gitops/manifests/gpu-acceleration/classes.yaml` maps each class to its resource and binds it to the offering component, the quota key and the DRA `DeviceClass`; `scripts/generate_gpu_offering.py` writes the agent's components from it, and `scripts/prepare_gpu_dra.py` writes the chart's `DeviceClass` objects from it. Two classes may not share a resource, and a class the map does not contain is refused at Site Agent startup. The DRA chart no longer ships its own `gpu.nvidia.com` `DeviceClass`, so nothing aliases `nvidia.com/gpu` to whatever whole GPU DRA happens to discover; whole L40 cards are exactly the two the `openst-gpu05` device plugin advertises. ### MPS on the L40 hosts MPS needs two things this release does not get for free, and a delivery that misses either one leaves `openst-gpu06` without `nvidia.com/gpu.shared`: 1. **The MPS binaries of the host driver.** The MPS control daemon runs `exec.Command("nvidia-cuda-mps-control", "-d")` with a plain `PATH` lookup, and the node image the plugin runs (`nvcr.io/nvidia/k8s-device-plugin:v0.20.1`) does not carry that binary: the NVIDIA Container Toolkit injects it from the host driver install, like `nvidia-smi`. On 2 November 2026 the L40 hosts had driver `550.163.01-2` installed without the `nvidia-cuda-mps` package, so the generated CDI spec listed `/usr/bin/nvidia-smi` and nothing else, and the daemon crash-looped with `exec: "nvidia-cuda-mps-control": executable file not found in $PATH`. The package is `nvidia-cuda-mps=550.163.01-2` (Debian, same version as `nvidia-driver`), it provides `/usr/bin/nvidia-cuda-mps-control` and `/usr/bin/nvidia-cuda-mps-server`, and it belongs to the host profile, which this repository records as `host-package-unowned-in-pharos-platform`: it is installed on the host, never from a cluster manifest. Verify before a delivery enables MPS: ``` dpkg -s nvidia-cuda-mps | head -3 command -v nvidia-cuda-mps-control nvidia-cuda-mps-server grep -c nvidia-cuda-mps /var/run/cdi/nvidia.yaml ``` 2. **A device plugin that does not run the `mixed` MIG strategy.** The plugin stops at startup with `error starting plugins: unable to validate flags: using --mig-strategy=mixed is not supported with MPS`, and the operator copies the ClusterPolicy's `mig.strategy` into the plugin. That strategy stays `mixed` for the GPU feature discovery of the MIG-carrying GH200 nodes, so `base/values.yaml` overrides it on the plugin alone through `devicePlugin.env`. This holds because the plugin runs only on the L40 hosts: the GH200 nodes are fenced off with `nvidia.com/gpu.deploy.device-plugin=false` and served by DRA, so the plugin never has to expose a MIG device. `mps_errors` in `scripts/gpu_checks.py` refuses any configuration that shares devices with MPS without the override, without `devicePlugin.mps.root`, or without the per-client isolation, and `scripts/test_gpu_operator.py` applies it to both the reviewed values and the rendered ClusterPolicy. A Foundation that installs `nvidia-cuda-mps` on the L40 hosts, and delivery that waits for the MPS control daemon to be Ready before the plugin advertises the shares, are both prerequisites for re-enabling `nvidia.com/gpu.shared`; until then the node stays on `whole-gpu`. ### Migrating a GH200 node A MIG geometry needs the card idle, and the driver may need a reset or a reboot before it applies, so delivery cordons and drains the node, writes the labels, waits for `nvidia.com/mig.config.state=success`, restarts the DRA kubelet plugin, which discovers slices only at startup, and uncordons the node only after that restart. The restart is recorded as `pharos.platform/dra-rediscovery=pending` in the same write as the geometry, and removed after it: a delivery that dies in between leaves the node cordoned with the marker, and the next delivery completes the restart before it releases the node instead of leaving a plugin that still names destroyed slices. The same order holds when a node keeps DRA and returns to whole GPUs. A node that reports no success in time stays cordoned. The shared models run on `cluster-node-21`, which has no MIG label and is therefore left whole. A whole-GPU node that changes allocation owner — the device plugin taking it back, or DRA taking it — is drained first and released only when the new plugin publishes the cards and the old one has stopped: the device plugin advertises `nvidia.com/gpu` in the node's allocatable, the DRA driver publishes `ResourceSlices`, and the pods the previous plugin placed keep the devices they were given. A plugin that never publishes leaves the node cordoned for the next delivery to resume. Rolling back restores the previous labels through the environment policy: a node leaving DRA loses `pharos.platform/gpu-dra`, regains `nvidia.com/gpu.deploy.device-plugin`, and the MPS and MIG labels are removed. Reconfiguring MIG back to whole GPUs drains the node again. ## Live adoption on 26 October 2026 The pre-change private snapshot recorded the operator `26.3.3` installation, ClusterPolicy, RuntimeClasses, Helm release metadata and workload state. The reviewed Argo sync started at `2026-10-26T11:28:21Z`, completed at `2026-10-26T11:28:38Z`, and reported `Succeeded` with `dryRun=false`. The Application is `Synced/Healthy` at revision `808f1a31f87712282892be97116579f77517f950`. | Component | Before | Live after adoption | | --- | --- | --- | | GPU Operator | `v26.3.3` | `v26.7.2` | | Node Feature Discovery | `v0.18.3` | `v0.19.0` | | Device plugin / GFD | `v0.19.3` | `v0.20.1` | | DCGM Exporter | `4.5.3-4.8.2-distroless` | `4.6.1-4.8.4-distroless` | | Operator validator | Previous release | `v26.7.2` | The GFD, device-plugin, DCGM and validator DaemonSets each reached 6/6 Ready and 6/6 updated; the NFD worker reached 9/9. ClusterPolicy `cluster-policy` reached `Ready=True`, generation 4. The cluster UID `3ccee890-dcd6-4ffc-8f56-e7f43dcbf9a2`, GPU namespace UID `d2b28791-a4ee-45dc-8f59-adc5a0cedfe1`, ClusterPolicy UID `24867769-9785-440d-9f6b-349fb93c2aad` and RuntimeClass UIDs were preserved. At adoption the live values retained `driver.enabled=false`, `toolkit.enabled=false`, `migManager.enabled=false`, `mig.strategy=mixed`, CDI enabled and NRI disabled. No `Driver` or `GPUCluster` custom resource exists. No default RuntimeClass, taint, label, MIG allocation, DRA driver, time-slicing profile, driver, toolkit or containerd configuration was changed by that adoption; the allocation that follows was reviewed and delivered separately. Argo renders and applies the Helm chart without updating Helm storage. The legacy Helm release metadata therefore still reports chart/app `v26.3.3`, revision 3; the profile records that release as `legacy_helm_release` with the role `forensic-only`, so the profile, the pins, the chart and this page name `v26.7.2` as the one release that runs and that the cutover installs. That record is retained for forensics and rollback reference, not as live desired state. Running an independent `helm upgrade` or `helm rollback` would create a second owner and is not a supported maintenance path while Argo owns these resources. Live Argo `v2.12.6` exposed three compatibility requirements now committed in the Application source: use a registered in-cluster destination (`https://kubernetes.default.svc:443`), express a manual sync by omitting `automated`, and enable `ServerSideDiff=true` because local typed comparison cannot decode the Kubernetes `v1.35` Deployment status field `.status.terminatingReplicas`. ## Version and support boundaries Sources checked on 26 October 2026: - [NVIDIA GPU Operator platform support](https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/latest/platform-support.html) - [NVIDIA GPU Operator release notes](https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/latest/release-notes.html) `26.7.x` is the supported current line in those documents. `26.3.x` is marked deprecated and `25.10.x` and earlier are end of support. NVIDIA validates Kubernetes `1.33`–`1.37` on listed bare-metal operating systems and versions; the exact Debian 13 combinations at Icarus are not listed in NVIDIA's validated OS table. GH200 and L40 are supported GPU products, but that does not establish a supported fresh Debian 13/GH200-64KiB/L40 installation recipe. | Profile | Live adoption evidence | Fresh-install claim | | --- | --- | --- | | AMD64 L40 | Debian 13, kernel `6.12.95+deb13-amd64`, 4KiB pages, driver `550.163.01`, toolkit `1.19.1`, containerd `2.2.1` | Not qualified | | ARM64 GH200 | Debian 13, kernel `6.14.0-1015-nvidia-64k` `6.14.0-1015.15`, 64KiB pages, driver `610.43.02` as `nvidia.ko` in `linux-modules-6.14.0-1015-nvidia-64k`, toolkit `1.19.1`, containerd `2.2.1` | Not qualified; the driver packages come from the vendor image, not a repository | The completed operator upgrade path is `26.3.3` to `26.7.2`. It cannot be used as evidence for a driver, toolkit, containerd, MIG, DRA or fresh-OS transition. ## Acceptance The adoption has current evidence for all acceptance gates: 1. The cluster, GPU namespace, ClusterPolicy and RuntimeClass identities were preserved, and the Argo operation completed successfully at the exact source revision. 2. The operator, NFD, GFD, device plugin, DCGM Exporter and validator are Ready. 3. All six GPU nodes still advertise two `nvidia.com/gpu` devices, for 12 total; labels, taints, RuntimeClasses and conventional device allocation are retained. 4. Native CUDA `vectorAdd` succeeded after the operator change on one L40 and one GH200. The jobs ran one at a time, with one GPU, bounded CPU/memory, a deadline, zero retries, the normal scheduler, `runtimeClassName: nvidia`, no privileged mode, no host mount and no service-account token. The temporary PriorityClass was owner-labelled, `globalDefault=false`, value `-1000`, and non-preempting. 5. Ceph remained `HEALTH_OK` with FSID `d75dc3d0-ce56-4226-970b-c1f2977cec5f`, 76 OSDs, quorum `a,b,c` and 265/265 PGs `active+clean`; 26/26 PVCs remained Bound. There were no Pending pods and no non-Ready containers in Running pods. The seven Failed pods were the same retained registry-prune and old Rook exporter failures observed before the operator change. | Profile | Job / Pod UID | Immutable image | Result | | --- | --- | --- | --- | | AMD64 L40 on `openst-gpu05` | `e0057168-779e-4755-b3e0-99b0ad60a31b` / `4375b63d-42ae-4c36-8739-70658f5e4433` | `nvcr.io/nvidia/k8s/cuda-sample@sha256:9855f4c8500addf185360474184b9efdaf4384284779aa9173dcd70164a4ae6f` | `Test PASSED`; driver `550.163.01`, containerd `2.2.1`, kernel `6.12.95+deb13-amd64`, 4 KiB pages | | ARM64 GH200 on `cluster-node-21` | `6a02b9ce-63da-4483-980a-721384fddb68` / `f5527554-36cf-4e9d-936f-f9a6b6f37aae` | `nvcr.io/nvidia/k8s/cuda-sample@sha256:a54fdceac3bc2a8d177f07db942defc2f7237e18d07fca2ff00718ba5aee4940` | `Test PASSED`; driver `610.43.02`, containerd `2.2.1`, kernel `6.14.0-1015-nvidia-64k`, 64 KiB pages | Temporary namespace UID `317e6645-6181-48e6-86cd-a05c789c45b9` and PriorityClass UID `94703e08-36c8-4ef6-8d78-bf9670d9b461` were removed with API UID preconditions after confirming their only Pod references were the two owned acceptance Pods. Both returned HTTP 404 afterwards. ## Recovery and rollback Private pre- and post-change snapshots are held in `0700` administrator directories with `0600` payloads. The pre-change archive SHA-256 is `f56f245e063ea0f5126690f36bc69e9af00039748ab592eb063113ba14a78f85`; the verified post-change archive SHA-256 is `4eb462a2f2e89d68a114fb511837c6ff5def34c181dafd4b58f143f538c94143`. They include the Argo Application, Helm history/values/manifest and release secrets, NVIDIA CRDs/CRs, ClusterPolicy, RuntimeClasses, GPU namespace and logs. They are not published in Git. A routine downgrade is not defined. Helm rollback is not authoritative because Argo applies the chart without updating Helm release storage. Recovering the operator requires a separately reviewed GitOps diff and sync, informed by the private snapshots. Deleting the GPU namespace or NVIDIA CRDs is destructive, removes cluster-wide API state, and is not a rollback. Changes to host driver, toolkit, containerd, MIG or DRA are outside this boundary. The host layer is owned by the `gpu-host` Ansible role, driven by `devops/platform/profiles/host-profiles.yml`. Each profile declares where its driver comes from: - L40 (`apt-package`): `nvidia-driver`, `nvidia-cuda-mps` and `nvidia-container-toolkit` are installed at pinned apt specs from the Debian repositories the host already has, then held. - GH200 (`preinstalled-package`): the driver is **not** built into the kernel. It is `nvidia.ko` inside `linux-modules-6.14.0-1015-nvidia-64k` `6.14.0-1015.15`, which ships with the vendor image alongside `linux-image-unsigned-6.14.0-1015-nvidia-64k` `6.14.0-1015.15`; both were installed by `dpkg` with no apt history entry and no repository configured on the node. The role therefore verifies those exact versions with `dpkg-query`, verifies the module path and kernel release with `modinfo`, and holds the packages; it never installs them. - A locally built driver against the pinned kernel (`local-build`) is described by the profile vocabulary and validated, but refused by the role until a reviewed build task exists. No automatic upgrade path exists on those nodes: no `unattended-upgrades`, no DKMS and no pins or holds before this role. The role adds the holds and the blocklist. Two host details the role reproduces as they are, rather than as they are usually documented: - The CDI specification lives under `/var/run/cdi`, where the running toolkit publishes it; the profiles name that path and there is no `/etc/cdi/nvidia.yaml` on these nodes. - `/etc/containerd/conf.d/99-nvidia.toml` carries the containerd 2.x runtime section **and** a legacy `io.containerd.grpc.v1.cri` block left by the earlier setup. The role verifies the file against the profile by default and accepts it when the declared runtime is present; rewriting it drops the legacy block, so rendering and the containerd restart that applies it are explicit and belong to a maintenance window. The remaining ownership gap is fresh-OS reproducibility, and on GH200 it is a provenance gap: the kernel and modules packages that carry the driver have no reviewed source, so the repository can pin and verify them but cannot recreate them from a repository. Negative tests reject a floating chart, a changed source commit or archive, a wrapper dependency that differs from the reviewed release, an automatic adoption sync, an enabled driver or toolkit, a device-plugin configuration outside the reviewed set, time-slicing insertion, a DRA `DeviceClass` that aliases `nvidia.com/gpu`, and accidental selection on the dev-box.