For some reason, the ubuntu24.04 daemonset is selecting a 22.04 binary driver image (reopens #722)

[doctorpangloss]

Reopens #722

Describe the bug

Name:                 nvidia-driver-daemonset-6.8.0-59-generic-ubuntu24.04-lzpsx
Namespace:            gpu-operator
...
helm.sh/chart=gpu-operator-v25.3.0
...
Controlled By:  DaemonSet/nvidia-driver-daemonset-6.8.0-59-generic-ubuntu24.04
...
Containers:
  nvidia-driver-ctr:
    Container ID:
    Image:         nvcr.io/nvidia/driver:570-6.8.0-59-generic-ubuntu22.04

Well, that's wrong.

To Reproduce
Install Ubuntu 24.04 on your cluster. Observe the wrong driver is selected..

Expected behavior
You have the 24.04 images. Why would the daemonset be deployed with the wrong image? A bug is why.

Environment (please provide the following information):

• GPU Operator Version: 25.3.0
• OS: Ubuntu24.04
• Kernel Version: 6.8.0-59-generic
• Container Runtime Version: containerd 1.7.22
• Kubernetes Distro and Version: k0s 1.30

[doctorpangloss]

kubectl logs pod/nvidia-driver-daemonset-q42pc -n gpu-operator --follow
DRIVER_ARCH is x86_64
Creating directory NVIDIA-Linux-x86_64-570.124.06
Verifying archive integrity... OK
Uncompressing NVIDIA Accelerated Graphics Driver for Linux-x86_64 570.124.06.......................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................

WARNING: Unable to determine the default X library path. The path /tmp/null/lib will be used, but this path was not detected in the ldconfig(8) cache, and no directory exists at this path, so it is likely that libraries installed there will not be found by the loader.


WARNING: You specified the '--no-kernel-modules' command line option, nvidia-installer will not install any kernel modules as part of this driver installation, and it will not remove existing NVIDIA kernel modules not part of an earlier NVIDIA driver installation.  Please ensure that NVIDIA kernel modules matching this driver version are installed separately.


WARNING: This NVIDIA driver package includes Vulkan components, but no Vulkan ICD loader was detected on this system. The NVIDIA Vulkan ICD will not function without the loader. Most distributions package the Vulkan loader; try installing the "vulkan-loader", "vulkan-icd-loader", or "libvulkan1" package.


========== NVIDIA Software Installer ==========

Starting installation of NVIDIA driver version 570.124.06 for Linux kernel version 6.8.0-60-generic

Stopping NVIDIA persistence daemon...
Unloading NVIDIA driver kernel modules...
Unmounting NVIDIA driver rootfs...
Checking NVIDIA driver packages...
Updating the package cache...
Resolving Linux kernel version...
Could not resolve Linux kernel version
Stopping NVIDIA persistence daemon...
Unloading NVIDIA driver kernel modules...
Unmounting NVIDIA driver rootfs...

Issue persists, what is going on? Why is this giving me a 22 image on 24?

[peterezzo]

i have this same issue, the ds pod shows 24.04 but the driver-ctr container is still 22.04

Events:
  Type     Reason     Age                  From               Message
  ----     ------     ----                 ----               -------
  Normal   Pulled     25m                  kubelet            Container image "nvcr.io/nvidia/cloud-native/k8s-driver-manager:v0.8.0" already present on machine
  Normal   Created    25m                  kubelet            Created container: k8s-driver-manager
  Normal   Started    25m                  kubelet            Started container k8s-driver-manager
  Normal   Scheduled  24m                  default-scheduler  Successfully assigned gpu-operator/nvidia-driver-daemonset-6.8.0-60-generic-ubuntu24.04-6h794 to gpu5
  Normal   Pulling    23m (x4 over 25m)    kubelet            Pulling image "nvcr.io/nvidia/driver:570-6.8.0-60-generic-ubuntu22.04"
  Warning  Failed     23m (x4 over 25m)    kubelet            Failed to pull image "nvcr.io/nvidia/driver:570-6.8.0-60-generic-ubuntu22.04": rpc error: code = NotFound desc = failed to pull and unpack image "nvcr.io/nvidia/driver:570-6.8.0-60-generic-ubuntu22.04": failed to resolve reference "nvcr.io/nvidia/driver:570-6.8.0-60-generic-ubuntu22.04": nvcr.io/nvidia/driver:570-6.8.0-60-generic-ubuntu22.04: not found
  Warning  Failed     23m (x4 over 25m)    kubelet            Error: ErrImagePull
  Warning  Failed     23m (x5 over 24m)    kubelet            Error: ImagePullBackOff
  Normal   BackOff    16s (x105 over 24m)  kubelet            Back-off pulling image "nvcr.io/nvidia/driver:570-6.8.0-60-generic-ubuntu22.04"