Skip to content

driver: request gpu when creating container builder #3063

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 1 commit into from
Mar 13, 2025

Conversation

crazy-max
Copy link
Member

With CDI support added in BuildKit moby/buildkit#4056, we need to request GPUs when creating a container builder.

This is necessary on WSL for Nvidia GPUs because there is no linux drivers available that we could install in BuildKit container during OnDemand setup. Mounting /usr/lib/wsl works https://github.com/docker/buildx/compare/master...crazy-max:buildx:driver-ctn-mount-wsl-lib?expand=1 but this is not desired and could be error prone if a remote context/builder is used. So instead we can make a device request when creating the container builder.

Can be tested by creating a BuildKit container with ubuntu base:

$ EXPORT_BASE=ubuntu PLATFORMS=linux/amd64,linux/arm64 ./hack/images ubuntu crazymax/buildkit push

You can use crazymax/buildkit:ubuntu for testing on WSL with Nvidia GPU:

$ docker buildx create --name testgpu --bootstrap --driver-opt "image=crazymax/buildkit:ubuntu"
#1 [internal] booting buildkit
#1 pulling image crazymax/buildkit:ubuntu
#1 pulling image crazymax/buildkit:ubuntu 5.3s done
#1 creating container buildx_buildkit_testgpu0
#1 creating container buildx_buildkit_testgpu0 1.9s done
#1 DONE 7.2s
testgpu
$ docker exec -it buildx_buildkit_testgpu0 nvidia-smi
Thu Mar 13 15:41:39 2025       
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 570.124.06             Driver Version: 572.70         CUDA Version: 12.8     |
|-----------------------------------------+------------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  NVIDIA GeForce RTX 3060 Ti     On  |   00000000:2A:00.0  On |                  N/A |
|  0%   55C    P8             23W /  200W |    2955MiB /   8192MiB |     13%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+

+-----------------------------------------------------------------------------------------+
| Processes:                                                                              |
|  GPU   GI   CI              PID   Type   Process name                        GPU Memory |
|        ID   ID                                                               Usage      |
|=========================================================================================|
|    0   N/A  N/A              23      G   /Xwayland                             N/A      |
|    0   N/A  N/A              25      G   /Xwayland                             N/A      |
|    0   N/A  N/A              35      G   /Xwayland                             N/A      |
+-----------------------------------------------------------------------------------------+

}
useInit := true
hc := &container.HostConfig{
Privileged: true,
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I guess Privileged not really required for this test.

@crazy-max crazy-max force-pushed the driver-ctn-gpu-request branch from 0cebbba to c17ea70 Compare March 13, 2025 16:22
@crazy-max crazy-max force-pushed the driver-ctn-gpu-request branch from c17ea70 to 3375782 Compare March 13, 2025 16:44
@tonistiigi tonistiigi merged commit 97f1d47 into docker:master Mar 13, 2025
138 checks passed
// hasGPUCapability checks if docker daemon has GPU capability. We need to run
// a dummy container with GPU device to check if the daemon has this capability
// because there is no API to check it yet.
func (d *Driver) hasGPUCapability(ctx context.Context, image string, gpus []container.DeviceRequest) bool {
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Time this func takes:

  • ~15ms on MacOS (no GPU)
  • ~230ms on WSL (with GPU)
  • ~200ms on WSL (no GPU)
  • ~200ms on Windows (with GPU)
  • ~160ms on Windows (no GPU)

@crazy-max crazy-max deleted the driver-ctn-gpu-request branch March 13, 2025 17:00
@crazy-max
Copy link
Member Author

Added follow-up label to have an API for this capability on Moby side

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants