> For AI agents: the complete documentation index is available at /llms.txt, the full documentation bundle is available at /llms-full.txt.
# NVIDIA GPU Operator ownership
Status: live in Icarus Cloud. On 26 October 2026 Argo CD adopted the existing
installation at source revision `808f1a31f87712282892be97116579f77517f950` and
applied GPU Operator `v26.7.2`. This page records the ownership boundary and the
live adoption evidence. It does not claim a fresh Debian 13 installation.
## Decision
The deployment uses two owners because they operate on different state:
| State | Owner | Selected by |
| --------------------------------------------------------------- | ------------------------------------------------------- | ------------------------------------- |
| NVIDIA kernel driver and headers | `gpu-host` Ansible role | An inventory `gpu_host` group only |
| NVIDIA Container Toolkit and containerd NVIDIA runtime fragment | `gpu-host` Ansible role | The same `gpu_host` group only |
| Host GPU/CDI health gate | `gpu-host` Ansible role | The same `gpu_host` group only |
| GPU Operator, CRDs and cluster policy | Argo CD Application | An explicit environment component |
| Device plugin (L40 hosts), GFD, DCGM Exporter and validator | GPU Operator | The reviewed ClusterPolicy |
| MIG Manager (GH200 hosts) | GPU Operator | The geometry delivery labels per node |
| DRA driver and the GH200 `DeviceClass` objects | A second Argo CD Application, `gpu-dra` | The same environment component list |
| Accelerator class names, pods and quota keys | The class map under `gitops/manifests/gpu-acceleration` | Regenerated, never hand-written |
This follows [ADR-0040](/reference/docs/decisions/0040-platform-layout.md): host prerequisites are
foundation state, while cluster workloads have an exact-revision GitOps owner. An
Ansible play must target an explicit GPU-host group. It must not run against the
dev-box or derive GPU work from all inventory hosts. The GPU Operator Application
is an opt-in Kustomize component and is absent from the dev-box rendering.
## Pinned installation source
The committed chart is `v26.7.2`, the same release `devops/platform/profiles/host-profiles.yml`
pins for the environment and `gitops/manifests/gpu-operator/pins.yaml` names with its
archive, tag object and source commit. Its official archive, source tag object,
dereferenced commit and NFD dependency are recorded in
`pins.yaml`. The component lives at
`gitops/apps/components/gpu-operator`.
A small Pharos wrapper chart installs the derived, checksum-pinned archive from
`chart/charts/`. The original NVIDIA archive is retained under `upstream/`, outside
Helm's dependency directory. Both identities are recorded in `pins.yaml`.
`scripts/prepare_gpu_chart.py` derives the installed archive deterministically:
only `gpu-operator/templates/clusterrole.yaml` changes. Two exact rules are reduced
to `get/list/watch`: CRDs and admission policies/bindings. Every other member's
bytes and metadata are preserved. Verification rejects source-rule drift, archive
drift, DRA GPUCluster or NVIDIADriver enablement, disabled ClusterPolicy and
`operator.cleanupCRD`.
The upgrade hook retains its reviewed CRD-write grant; Argo owns CRD lifecycle.
This reduction applies to the conventional ClusterPolicy profile. In the pinned
[source controller](https://github.com/NVIDIA/gpu-operator/blob/5a9c8a8b3f5bac7fb06ad3c2a05771fceb8b4832/controllers/clusterpolicy_controller.go#L87),
CRD permissions are read-only; its
[CRD lookup](https://github.com/NVIDIA/gpu-operator/blob/5a9c8a8b3f5bac7fb06ad3c2a05771fceb8b4832/controllers/object_controls.go#L4988)
uses `Get`. The conventional resource manager has no CRD handler; generic CRD
reconciliation and cleanup belong to the inactive GPUCluster/NVIDIADriver state
managers. Verification keeps `driver.nvidiaDriverCRD.enabled=false`. The source's
admission policy manifests belong to the
[DRA operand](https://github.com/NVIDIA/gpu-operator/blob/5a9c8a8b3f5bac7fb06ad3c2a05771fceb8b4832/manifests/state-dra-driver/0700_resourceslices-policy.yaml).
The separate [GPUCluster reconciler](https://github.com/NVIDIA/gpu-operator/blob/5a9c8a8b3f5bac7fb06ad3c2a05771fceb8b4832/controllers/gpucluster_controller.go#L80)
requires a GPUCluster object before synchronising those states. Bootstrap checks
that no such object exists before installing this profile. Enabling DRA or CRD
cleanup requires a fresh permission and workload review.
To reproduce the installed source and its generated delegation policy:
```sh
uv run --project devops --frozen python scripts/prepare_gpu_chart.py
uv run --project devops --frozen python scripts/generate_gpu_admission.py
```
The reductions prevent the operator from withdrawing admission guards or granting
CRD-write permissions to another account. Kubernetes does not run API-based
[ValidatingAdmissionPolicy against admission policies or bindings](https://kubernetes.io/docs/reference/kubernetes-api/admissionregistration/validating-admission-policy-v1/),
so revocation is prevented through RBAC. These permission changes are qualified
against a disposable Kubernetes API. GPU workloads and a live GPU rollout with
the derived archive remain unqualified; the adoption evidence below describes
the earlier upstream installation.
The values preserve host ownership and implement the reviewed allocation below:
- `driver.enabled=false` and `toolkit.enabled=false`: Ansible owns host software.
- `driver.version` is empty, so the chart omits it from the ClusterPolicy: nothing installs a
driver from the cluster, and the two host profiles run different releases. The version that is
actually running lives in `devops/platform/profiles/host-profiles.yml`, and
`scripts/test_gpu_host_verification.py` fails if the operator values ever name it. The
`595.91.07` the live ClusterPolicy used to declare was only a declaration: with
`driver.enabled=false` the operator installs no driver, so it described nothing the node ran.
- `mig.strategy=mixed` and `migManager.enabled=true`: the manager applies the one geometry each
GH200 node is labelled with, and leaves unlabelled nodes alone.
- The device plugin stays enabled and serves the L40 hosts only; its per-node configurations are
described under [the reviewed allocation](#reviewed-accelerator-allocation).
- CDI is enabled and NRI is disabled.
- The conventional ClusterPolicy is installed; DRA GPUCluster is not.
- NVIDIA is not the default containerd runtime, `timeSlicing` is still never set, and no
`RuntimeClass` or taint is derived from the allocation.
The Application remains manually synchronised by omitting Argo's `automated` sync
policy altogether. The reviewed live operation ran one explicit sync at the exact
source revision above. Ordinary delivery therefore records the desired object but
does not silently change the GPU stack. Enabling automated sync remains a separate
reviewed operation.
The Application uses Argo's server-side diff because the live Argo `v2.12.6`
controller predates Kubernetes `v1.35` status fields. Without it, comparison fails
on the API server's own Deployment schema before a sync can be reviewed.
The committed `icarus-cloud` environment selects the component, while the common
and disposable-lab policies do not. Normal delivery creates the manual child
Application and excludes it from the ready-set wait. The dev-box overlay never
selects it.
The child also carries `pharos.platform/deferred-sync: "true"`. The platform's
Application health rule treats a deferred child as settled unless it is being
operated on, because `pharos-platform` otherwise stays `Progressing` — and holds
every later sync wave — while the manual child is `OutOfSync`/`Missing`. The child
still reports its own running operation and its error conditions, so the explicit
GPU operation keeps its evidence.
The destination is `https://kubernetes.default.svc:443`, the self-management
alias the platform Argo already registers. `cluster.inClusterEnabled` is false, so
`https://kubernetes.default.svc` is not a cluster Argo can reach; the alias also
registers the `gpu-operator` namespace and caches cluster-scoped resources. The
bootstrap grants the application controller, by name, the chart's ClusterRoles,
ClusterRoleBindings, namespaced Roles and bindings, eight NVIDIA/NFD CRDs and the
`cluster-policy` ClusterPolicy. Broad CRD access is read-only for cache discovery.
Named role grants include `escalate` and `bind`; a generated admission policy
requires the exact reviewed rules, roleRefs and subjects for all 14 chart RBAC
objects and rejects aggregation. A separate actor-based policy limits Argo and
the upgrade hook to the GPU CRDs, preserving Argo's own three CRDs. Because the
API server's escalation check carries no object name, `escalate` itself stays
unnamed, so a third generated policy bounds the actor instead of the chart: the
application controller may create or change only the cluster roles the platform's
own grants review — the names the pinned GPU charts render and the names the
`kubeflow`, `site-agent`, `gpu-operator` and `gpu-dra` bootstraps reserve — it may
bind only those, and no cluster role it writes may aggregate — except the five
Kubeflow roles the composition renders with an `aggregationRule`
(`kubeflow-admin`, `kubeflow-edit`, `kubeflow-view` and the two
`poddefaults-webhook-kubeflow-poddefaults-*` roles), whose selectors are pinned to
the rendered ones. Argo creates these roles with empty rules and preserves their
existing rules on update, including during server-side apply. Kubernetes' aggregation
controller owns subsequent rule changes; its requests remain outside the Argo policy.
This prevents direct permission changes taking effect before aggregation reconciles.
ClusterRole labels are pinned to their rendered values, so Argo cannot add another
role's permissions by changing membership of Kubeflow, built-in or external aggregates.
Argo's configured annotation-based tracking remains supported.
Those names are read from the grants, and the aggregated selectors from the composition
render, when the policy is generated, so a role nobody reviewed has no entry. Bootstrap
installs both policies and their Deny bindings, waits for compilation and rejects
type warnings before granting controller rights. The existing writer policy also
admits the upgrade hook and the garbage collector for its reviewed delete path.
The bootstrap's own objects are named `pharos-gpu-operator-writer`, outside the
GPU-owned names. Restricted identities cannot change the admission objects;
an administrator with cluster-admin rights can recover them without membership
of `system:masters`. The separate original GPU writer fence retains its existing
administrator-group recovery rules. `scripts/test_gpu_operator.py` checks the
rendered names and bootstrap ordering; `scripts/gpu_admission_smoke.py` tests
actual chart permissions, direct and delegated CRD denials, policy revocation
and administrator recovery on an isolated API server. Its aggregation cases reject
direct Argo rule changes through create, replace, patch and server-side apply, and
check that native aggregation and metadata-only Argo synchronisation still work.
The `gpu-operator` and `gpu-dra` namespaces are bootstrap prerequisites. Argo does
not create namespaces or use `CreateNamespace=true`. A selected GPU environment
must create them through its reviewed bootstrap path before syncing the components.
## Reviewed accelerator allocation
Decided on 1 November 2026 and narrowed on 2 November 2026. The four GH200 nodes
move to DRA; the L40 hosts keep the device plugin, because the device plugin
`v0.20.1` this release pins cannot rename an extended resource and those hosts keep
driver `550.163.01`. Both remain ordinary delivery: the labels are written by
`platform_deploy/accelerators.py`, and nothing in this allocation is set by hand.
| Hosts | Served by | Advertised resource | Delivery label |
| ----------------- | ------------------------------------------------------------------------------------------------- | ------------------------------------------------------ | --------------------------------------------------------------------------- |
| `cluster-node-21` | DRA driver, both GPUs whole, reserved for inference | `nvidia.com/gh200` | `pharos.platform/gpu-dra=true`, `pharos.platform/inference=true:NoSchedule` |
| `cluster-node-22` | DRA driver, GPU 0 as `1g.18gb` × 7, GPU 1 whole | `nvidia.com/gh200-mig-1g.18gb`, `nvidia.com/gh200` | `pharos.platform/gpu-dra=true`, `nvidia.com/mig.config=gpu0-1g.18gb-x7` |
| `cluster-node-23` | DRA driver, GPU 0 as `1g.18gb` × 7, GPU 1 whole | `nvidia.com/gh200-mig-1g.18gb`, `nvidia.com/gh200` | `pharos.platform/gpu-dra=true`, `nvidia.com/mig.config=gpu0-1g.18gb-x7` |
| `cluster-node-24` | DRA driver, GPU 0 as `3g.72gb` × 2, GPU 1 whole | `nvidia.com/gh200-mig-3g.72gb`, `nvidia.com/gh200` | the same, `nvidia.com/mig.config=gpu0-3g.72gb-x2` |
| `openst-gpu05` | Device plugin, `whole-gpu` | `nvidia.com/gpu` | `nvidia.com/device-plugin.config=whole-gpu` |
| `openst-gpu06` | Device plugin, `whole-gpu` until the host ships the MPS binaries; `mps-4` is the reviewed sharing | `nvidia.com/gpu`; `nvidia.com/gpu.shared` with `mps-4` | `nvidia.com/device-plugin.config=whole-gpu` |
`mps-4` stays the reviewed sharing of `openst-gpu06`: four `nvidia.com/gpu.shared`
shares of one card. The device plugin starts the MPS control daemon by looking
`nvidia-cuda-mps-control` up on its own `PATH`; the plugin image does not carry
the binary, and the container runtime only injects what the host driver provides.
While the host driver package lacks it, the daemon exits with
`executable file not found` and the pod CrashLoops, so the environment declares
`whole-gpu` for that host. The driver package belongs to the foundation (`gpu-host`
role); once it ships the binary, the one line in
`devops/platform/environments/icarus-cloud.yml` returns to `mps-4`.
The geometry is per device, not per node, and that is the reviewed design rather
than a stopgap. Only GPU 0 is sliced: GPU 1 of a GH200 NVL2 takes no MIG geometry
on this platform. Against driver `610.43.02` MIG on GPU 1 fails with a GSP
`INSUFFICIENT_RESOURCES`, followed by `NV_ERR_TIMEOUT` in
`MIGMGR_SET_GPU_INSTANCES`, and the trial driver `580` fails the same way while
GPU 0 succeeds on both (see `icarus-deployment-state.md` §26–§29). The environment
policy therefore names the geometry as `{device, profile, slices}` and
`platform_gitops/gpu_mig.py` validates it: the device must be GPU 0, the profile
must be one the class map publishes, and the slice count must fill the device. The
same module writes the operator chart's `mig-configs` document, so the key
delivery labels a node with and the entry the MIG manager applies cannot drift
apart. A node that returns to whole GPUs is reset with `all-disabled`, which is
the chart's default.
A sliced host publishes exactly one whole GPU beside its slices, and that GPU is
neither shared nor usable together with the slices. The NVIDIA driver provides no
NCCL or peer-to-peer path between MIG instances, or between a MIG instance and a
GPU without MIG, so a workload placed on `cluster-node-22`, `cluster-node-23` or `cluster-node-24`
can never hold both of the host's GPUs at once. No class expresses such a pair,
and the offering does not invent one: a job that needs two GPUs across a
NVLink pair needs an unsliced host. The only such GH200 host in this layout is
`cluster-node-21`, reserved for shared inference; no two-card GH200 pair is
available to ordinary user workloads. Outside that reservation, the pool publishes
three whole GH200 GPUs, fourteen `1g.18gb` slices and two `3g.72gb` slices.
The rejected alternative was decided against on 2 November 2026: the pinned
`dra-driver-nvidia-gpu` `0.5.1` carries no `GpuConfig` resource at all (its CRDs
are the two ComputeDomain ones), so declarative `GpuConfig.sharing` with
`TimeSlicing` or `MPS` is not available in this release, and its only MPS path is
the experimental `MPSSupport` feature gate with a control daemon. No "GH200
shared" class is offered.
`cluster-node-21` keeps both cards whole for the shared Mistral model, and delivery
taints it `pharos.platform/inference=true:NoSchedule`. The DRA driver's own daemonset
tolerates that taint; the inference workloads must tolerate it too. Without the
taint a notebook or a Run asking for a whole GH200 could be placed on the node the
shared model owns, because the `gh200` class covers every whole GH200 DRA
publishes. The `2g.36gb` class is not offered while no node cuts that profile.
`openst-gpu06` advertises four MPS clients per card, so eight shares of about
11 GiB. MPS caps each client's pinned device memory at `totalMemory/replicas`, which
time-slicing does not; the control daemon sets every card to `EXCLUSIVE_PROCESS`,
and the plugin fails any request for more than one share
(`failRequestsGreaterThanOne: true`). The MPS control daemon must be Ready before
the plugin advertises the resource: the plugin retries until it is, so a delivery
that finishes before the daemon is healthy is not a success, and the DaemonSet can
sit in `CrashLoopBackOff` until then.
A `DeviceClass` publishes its extended resource only while the `DRAExtendedResource`
gate is enabled in the kube-apiserver, the kube-scheduler and the kubelet. Delivery
reads the API server's `/metrics` and each DRA node's kubelet `/metrics` before it
fences a node, and refuses to continue when either reports the gate off: a Site
without the gate keeps the device plugin instead of losing its GPUs.
Nothing else names an accelerator resource. `gitops/manifests/gpu-acceleration/classes.yaml`
maps each class to its resource and binds it to the offering component, the quota
key and the DRA `DeviceClass`; `scripts/generate_gpu_offering.py` writes the agent's
components from it, and `scripts/prepare_gpu_dra.py` writes the chart's
`DeviceClass` objects from it. Two classes may not share a resource, and a class the
map does not contain is refused at Site Agent startup. The DRA chart no longer
ships its own `gpu.nvidia.com` `DeviceClass`, so nothing aliases `nvidia.com/gpu`
to whatever whole GPU DRA happens to discover; whole L40 cards are exactly the
two the `openst-gpu05` device plugin advertises.
### MPS on the L40 hosts
MPS needs two things this release does not get for free, and a delivery that misses either one leaves
`openst-gpu06` without `nvidia.com/gpu.shared`:
1. **The MPS binaries of the host driver.** The MPS control daemon runs
`exec.Command("nvidia-cuda-mps-control", "-d")` with a plain `PATH` lookup, and the node image the plugin
runs (`nvcr.io/nvidia/k8s-device-plugin:v0.20.1`) does not carry that binary: the NVIDIA Container Toolkit
injects it from the host driver install, like `nvidia-smi`. On 2 November 2026 the L40 hosts had driver
`550.163.01-2` installed without the `nvidia-cuda-mps` package, so the generated CDI spec listed
`/usr/bin/nvidia-smi` and nothing else, and the daemon crash-looped with
`exec: "nvidia-cuda-mps-control": executable file not found in $PATH`. The package is
`nvidia-cuda-mps=550.163.01-2` (Debian, same version as `nvidia-driver`), it provides
`/usr/bin/nvidia-cuda-mps-control` and `/usr/bin/nvidia-cuda-mps-server`, and it belongs to the host
profile, which this repository records as `host-package-unowned-in-pharos-platform`: it is installed on
the host, never from a cluster manifest. Verify before a delivery enables MPS:
```
dpkg -s nvidia-cuda-mps | head -3
command -v nvidia-cuda-mps-control nvidia-cuda-mps-server
grep -c nvidia-cuda-mps /var/run/cdi/nvidia.yaml
```
2. **A device plugin that does not run the `mixed` MIG strategy.** The plugin stops at startup with
`error starting plugins: unable to validate flags: using --mig-strategy=mixed is not supported with MPS`,
and the operator copies the ClusterPolicy's `mig.strategy` into the plugin. That strategy stays `mixed`
for the GPU feature discovery of the MIG-carrying GH200 nodes, so `base/values.yaml` overrides it on the
plugin alone through `devicePlugin.env`. This holds because the plugin runs only on the L40 hosts: the
GH200 nodes are fenced off with `nvidia.com/gpu.deploy.device-plugin=false` and served by DRA, so the
plugin never has to expose a MIG device. `mps_errors` in `scripts/gpu_checks.py` refuses any
configuration that shares devices with MPS without the override, without `devicePlugin.mps.root`, or
without the per-client isolation, and `scripts/test_gpu_operator.py` applies it to both the reviewed
values and the rendered ClusterPolicy.
A Foundation that installs `nvidia-cuda-mps` on the L40 hosts, and delivery that waits for the MPS control
daemon to be Ready before the plugin advertises the shares, are both prerequisites for re-enabling
`nvidia.com/gpu.shared`; until then the node stays on `whole-gpu`.
### Migrating a GH200 node
A MIG geometry needs the card idle, and the driver may need a reset or a reboot
before it applies, so delivery cordons and drains the node, writes the labels, waits
for `nvidia.com/mig.config.state=success`, restarts the DRA kubelet plugin, which
discovers slices only at startup, and uncordons the node only after that restart.
The restart is recorded as `pharos.platform/dra-rediscovery=pending` in the same
write as the geometry, and removed after it: a delivery that dies in between leaves
the node cordoned with the marker, and the next delivery completes the restart before
it releases the node instead of leaving a plugin that still names destroyed slices.
The same order holds when a node keeps DRA and returns to whole GPUs. A node that
reports no success in time stays cordoned. The shared models run on
`cluster-node-21`, which has no MIG label and is therefore left whole.
A whole-GPU node that changes allocation owner — the device plugin taking it back,
or DRA taking it — is drained first and released only when the new plugin publishes
the cards and the old one has stopped: the device plugin advertises `nvidia.com/gpu`
in the node's allocatable, the DRA driver publishes `ResourceSlices`, and the pods
the previous plugin placed keep the devices they were given. A plugin that never
publishes leaves the node cordoned for the next delivery to resume.
Rolling back restores the previous labels through the environment policy: a node
leaving DRA loses `pharos.platform/gpu-dra`, regains
`nvidia.com/gpu.deploy.device-plugin`, and the MPS and MIG labels are removed.
Reconfiguring MIG back to whole GPUs drains the node again.
## Live adoption on 26 October 2026
The pre-change private snapshot recorded the operator `26.3.3` installation,
ClusterPolicy, RuntimeClasses, Helm release metadata and workload state. The
reviewed Argo sync started at `2026-10-26T11:28:21Z`, completed at
`2026-10-26T11:28:38Z`, and reported `Succeeded` with `dryRun=false`. The
Application is `Synced/Healthy` at revision `808f1a31f87712282892be97116579f77517f950`.
| Component | Before | Live after adoption |
| ---------------------- | ------------------------ | ------------------------ |
| GPU Operator | `v26.3.3` | `v26.7.2` |
| Node Feature Discovery | `v0.18.3` | `v0.19.0` |
| Device plugin / GFD | `v0.19.3` | `v0.20.1` |
| DCGM Exporter | `4.5.3-4.8.2-distroless` | `4.6.1-4.8.4-distroless` |
| Operator validator | Previous release | `v26.7.2` |
The GFD, device-plugin, DCGM and validator DaemonSets each reached 6/6 Ready and
6/6 updated; the NFD worker reached 9/9. ClusterPolicy `cluster-policy` reached
`Ready=True`, generation 4. The cluster UID
`3ccee890-dcd6-4ffc-8f56-e7f43dcbf9a2`, GPU namespace UID
`d2b28791-a4ee-45dc-8f59-adc5a0cedfe1`, ClusterPolicy UID
`24867769-9785-440d-9f6b-349fb93c2aad` and RuntimeClass UIDs were preserved.
At adoption the live values retained `driver.enabled=false`, `toolkit.enabled=false`,
`migManager.enabled=false`, `mig.strategy=mixed`, CDI enabled and NRI disabled.
No `Driver` or `GPUCluster` custom resource exists. No default RuntimeClass,
taint, label, MIG allocation, DRA driver, time-slicing profile, driver, toolkit or
containerd configuration was changed by that adoption; the allocation that
follows was reviewed and delivered separately.
Argo renders and applies the Helm chart without updating Helm storage. The legacy
Helm release metadata therefore still reports chart/app `v26.3.3`, revision 3; the
profile records that release as `legacy_helm_release` with the role `forensic-only`,
so the profile, the pins, the chart and this page name `v26.7.2` as the one release
that runs and that the cutover installs.
That record is retained for forensics and rollback reference, not as live desired
state. Running an independent `helm upgrade` or `helm rollback` would create a
second owner and is not a supported maintenance path while Argo owns these
resources.
Live Argo `v2.12.6` exposed three compatibility requirements now committed in the
Application source: use a registered in-cluster destination
(`https://kubernetes.default.svc:443`), express a manual sync by omitting `automated`, and enable
`ServerSideDiff=true` because local typed comparison cannot decode the Kubernetes
`v1.35` Deployment status field `.status.terminatingReplicas`.
## Version and support boundaries
Sources checked on 26 October 2026:
- [NVIDIA GPU Operator platform support](https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/latest/platform-support.html)
- [NVIDIA GPU Operator release notes](https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/latest/release-notes.html)
`26.7.x` is the supported current line in those documents. `26.3.x` is marked
deprecated and `25.10.x` and earlier are end of support. NVIDIA validates
Kubernetes `1.33`–`1.37` on listed bare-metal operating systems and versions; the
exact Debian 13 combinations at Icarus are not listed in NVIDIA's validated OS
table. GH200 and L40 are supported GPU products, but that does not establish a
supported fresh Debian 13/GH200-64KiB/L40 installation recipe.
| Profile | Live adoption evidence | Fresh-install claim |
| ----------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------- |
| AMD64 L40 | Debian 13, kernel `6.12.95+deb13-amd64`, 4KiB pages, driver `550.163.01`, toolkit `1.19.1`, containerd `2.2.1` | Not qualified |
| ARM64 GH200 | Debian 13, kernel `6.14.0-1015-nvidia-64k` `6.14.0-1015.15`, 64KiB pages, driver `610.43.02` as `nvidia.ko` in `linux-modules-6.14.0-1015-nvidia-64k`, toolkit `1.19.1`, containerd `2.2.1` | Not qualified; the driver packages come from the vendor image, not a repository |
The completed operator upgrade path is `26.3.3` to `26.7.2`. It cannot be used as
evidence for a driver, toolkit, containerd, MIG, DRA or fresh-OS transition.
## Acceptance
The adoption has current evidence for all acceptance gates:
1. The cluster, GPU namespace, ClusterPolicy and RuntimeClass identities were
preserved, and the Argo operation completed successfully at the exact source
revision.
2. The operator, NFD, GFD, device plugin, DCGM Exporter and validator are Ready.
3. All six GPU nodes still advertise two `nvidia.com/gpu` devices, for 12 total;
labels, taints, RuntimeClasses and conventional device allocation are retained.
4. Native CUDA `vectorAdd` succeeded after the operator change on one L40 and one
GH200. The jobs ran one at a time, with one GPU, bounded CPU/memory, a deadline,
zero retries, the normal scheduler, `runtimeClassName: nvidia`, no privileged
mode, no host mount and no service-account token. The temporary PriorityClass
was owner-labelled, `globalDefault=false`, value `-1000`, and non-preempting.
5. Ceph remained `HEALTH_OK` with FSID
`d75dc3d0-ce56-4226-970b-c1f2977cec5f`, 76 OSDs, quorum `a,b,c` and 265/265 PGs
`active+clean`; 26/26 PVCs remained Bound. There were no Pending pods and no
non-Ready containers in Running pods. The seven Failed pods were the same
retained registry-prune and old Rook exporter failures observed before the
operator change.
| Profile | Job / Pod UID | Immutable image | Result |
| -------------------------------- | ------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------- |
| AMD64 L40 on `openst-gpu05` | `e0057168-779e-4755-b3e0-99b0ad60a31b` / `4375b63d-42ae-4c36-8739-70658f5e4433` | `nvcr.io/nvidia/k8s/cuda-sample@sha256:9855f4c8500addf185360474184b9efdaf4384284779aa9173dcd70164a4ae6f` | `Test PASSED`; driver `550.163.01`, containerd `2.2.1`, kernel `6.12.95+deb13-amd64`, 4 KiB pages |
| ARM64 GH200 on `cluster-node-21` | `6a02b9ce-63da-4483-980a-721384fddb68` / `f5527554-36cf-4e9d-936f-f9a6b6f37aae` | `nvcr.io/nvidia/k8s/cuda-sample@sha256:a54fdceac3bc2a8d177f07db942defc2f7237e18d07fca2ff00718ba5aee4940` | `Test PASSED`; driver `610.43.02`, containerd `2.2.1`, kernel `6.14.0-1015-nvidia-64k`, 64 KiB pages |
Temporary namespace UID `317e6645-6181-48e6-86cd-a05c789c45b9` and
PriorityClass UID `94703e08-36c8-4ef6-8d78-bf9670d9b461` were removed with API
UID preconditions after confirming their only Pod references were the two owned
acceptance Pods. Both returned HTTP 404 afterwards.
## Recovery and rollback
Private pre- and post-change snapshots are held in `0700` administrator
directories with `0600` payloads. The pre-change archive SHA-256 is
`f56f245e063ea0f5126690f36bc69e9af00039748ab592eb063113ba14a78f85`; the verified
post-change archive SHA-256 is
`4eb462a2f2e89d68a114fb511837c6ff5def34c181dafd4b58f143f538c94143`.
They include the Argo Application, Helm history/values/manifest and release
secrets, NVIDIA CRDs/CRs, ClusterPolicy, RuntimeClasses, GPU namespace and logs.
They are not published in Git.
A routine downgrade is not defined. Helm rollback is not authoritative because
Argo applies the chart without updating Helm release storage. Recovering the
operator requires a separately reviewed GitOps diff and sync, informed by the
private snapshots. Deleting the GPU namespace or NVIDIA CRDs is destructive,
removes cluster-wide API state, and is not a rollback. Changes to host driver,
toolkit, containerd, MIG or DRA are outside this boundary.
The host layer is owned by the `gpu-host` Ansible role, driven by
`devops/platform/profiles/host-profiles.yml`. Each profile declares where its driver
comes from:
- L40 (`apt-package`): `nvidia-driver`, `nvidia-cuda-mps` and
`nvidia-container-toolkit` are installed at pinned apt specs from the Debian
repositories the host already has, then held.
- GH200 (`preinstalled-package`): the driver is **not** built into the kernel. It is
`nvidia.ko` inside `linux-modules-6.14.0-1015-nvidia-64k` `6.14.0-1015.15`, which
ships with the vendor image alongside `linux-image-unsigned-6.14.0-1015-nvidia-64k`
`6.14.0-1015.15`; both were installed by `dpkg` with no apt history entry and no
repository configured on the node. The role therefore verifies those exact versions
with `dpkg-query`, verifies the module path and kernel release with `modinfo`, and
holds the packages; it never installs them.
- A locally built driver against the pinned kernel (`local-build`) is described by the
profile vocabulary and validated, but refused by the role until a reviewed build task
exists.
No automatic upgrade path exists on those nodes: no `unattended-upgrades`, no DKMS and
no pins or holds before this role. The role adds the holds and the blocklist.
Two host details the role reproduces as they are, rather than as they are usually documented:
- The CDI specification lives under `/var/run/cdi`, where the running toolkit publishes it; the
profiles name that path and there is no `/etc/cdi/nvidia.yaml` on these nodes.
- `/etc/containerd/conf.d/99-nvidia.toml` carries the containerd 2.x runtime section **and** a
legacy `io.containerd.grpc.v1.cri` block left by the earlier setup. The role verifies the file
against the profile by default and accepts it when the declared runtime is present; rewriting it
drops the legacy block, so rendering and the containerd restart that applies it are explicit and
belong to a maintenance window.
The remaining ownership gap is fresh-OS reproducibility, and on GH200 it is a
provenance gap: the kernel and modules packages that carry the driver have no reviewed
source, so the repository can pin and verify them but cannot recreate them from a
repository.
Negative tests reject a floating chart, a changed source commit or archive, a
wrapper dependency that differs from the reviewed release, an automatic adoption
sync, an enabled driver or toolkit, a device-plugin configuration outside the
reviewed set, time-slicing insertion, a DRA `DeviceClass` that aliases
`nvidia.com/gpu`, and accidental selection on the dev-box.
Page source
Source snapshot · Revision `f928834c8`.