Support GPUs

[tmc]

What would you like to be added: GPU support.

Why is this needed: Development of AI/ML models in kind.

https://docs.docker.com/config/containers/resource_constraints/

Related/previous issues:

• Does kind support gpu accelerated containers? #2450
• add gpus support in docker provider #1886
• Add CDI support to --devices  docker/cli#3864

[aojea]

it looks hard to generalize https://docs.docker.com/config/containers/resource_constraints/#access-an-nvidia-gpu , it seems you need some prerequisites and also run some nvidia code

[tmc]

it looks hard to generalize https://docs.docker.com/config/containers/resource_constraints/#access-an-nvidia-gpu , it seems you need some prerequisites and also run some nvidia code

Yes, put setting out a path for even making it possible (with some elbow grease) would be appreciated.

[aojea]

how this is supposed to be used, a kind cluster with gpus ... asking from the ignorance

[BenTheElder]

I think we're still lacking a portable standard for this and a use case that can't be solved with extraMount / the existing --privileged

[samos123]

Could you consider allowing this to be added for now?

There is a lot of value to end-users that don't want to fork Kind just to get GPU support. I was going through this blog post and feel like lacking a portable standard isn't a strong enough reason to have to fork. A fork is even less portable.

[aojea]

https://github.com/jacobtomlinson/kind/pull/1/files

it does not seem as hard to maintain

[BenTheElder]

Is passing through all GPUs expected to remain desirable?

we do need some limited abstraction that we can retarget to podman and maybe nerdctl in the future.

a bool that simply passes all GPUs is one possible abstraction, it's not clear if it's a good one or if podman can support it

As per our contributing guide some specific proposal must be discussed and we should be looking at podman as well, that PR doesn't appear to.

[BenTheElder]

If we have to investigate podman etc ourselves this will remain on the backlog below pressing bug fixes like #3223.

If anyone is interested in helping move this along then please outline the approach to portability. Not just between runtimes but of the configurations across hosts.

The --gpu all mode in docker looks reasonable enough. It's better than hardcoding devices into config. But it seems somewhat unlikely to satisfy all users, and it doesn't look like podman has an equivilant at a glance so at the very least we need to address that with an error in the podman driver and config doc comment ...

happy to consider a specific plan that looks into these. Otherwise we'll write one later, tricky fixes haven't left much time to research this particular low-demand feature.

[samos123]

Got it. Makes sense. Thanks for sharing the details again of what's holding it back. I will try to make some time for this and come up with a proposal that also covers how other runtimes should be handled.

[psigen]

I am also being affected by this: using GPUs is a particularly common use-case when developing AI/ML workloads, so this is a limitation for using kind to locally test workloads destined for e.g. GKE. I suspect some users may have simply decided they cannot use kind at all due to the limitation, which might skew visible measures of demand.

FWIW: --gpu all mode in docker would work fine for all of my use-cases, I don't need this support to extend to podman. Here is a thread that links to several of ongoing podman discussions around GPU support: containers/podman#11088 .

I would be happy with the intermediate state where we can enable --gpu all mode in docker and simply document that this does not work in podman. While other users might have more specific needs, having some functionality is a strictly better state for those users as well.

[samos123]

podman and nercdtl seems to support the same all kind of behavior:

podman run --rm --device nvidia.com/gpu=all ubuntu nvidia-smi -L
nerdctl run -it --rm --gpus all nvidia/cuda:9.0-base nvidia-smi

sources:

• https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/install-guide.html
• https://github.com/containerd/nerdctl/blob/main/docs/gpu.md

Is passing through all GPUs expected to remain desirable?
I think we should allow "all" and in future support "gpu[0-8]" for passing an individual GPU but majority of people would simply use "all".

@BenTheElder what's the way to submit a proposal? You got an example of a specific format that should be used?

Are you fine of the proposal of only supporting "all" mode initially for both podman and docker? when users ask for support specific GPUs then that can be added later. In the implementation I will ensure it's using a string instead of bool such that eventually people could specify gpu0 instead of all

[BenTheElder]

I am also being affected by this: using GPUs is a particularly common use-case when developing AI/ML workloads, so this is a limitation for using kind to locally test workloads destined for e.g. GKE. I suspect some users may have simply decided they cannot use kind at all due to the limitation, which might skew visible measures of demand.

I mean this was not feasible earlier in the project, the underlying support in docker/containerd was not there in a way that would work reasonably with kind.

FWIW: --gpu all mode in docker would work fine for all of my use-cases, I don't need this support to extend to podman.

Do you all use this with a single node/cluster and just share all to it?

What does usage look like?

Here is a thread that links to several of ongoing podman discussions around GPU support: containers/podman#11088 .

Thanks, but what we actually need is to identify what options are supportable, if I get a bit more time I'll go search/read but I'd appreciate if someone else interested in this feature outlined that.

I would be happy with the intermediate state where we can enable --gpu all mode in docker and simply document that this does not work in podman. While other users might have more specific needs, having some functionality is a strictly better state for those users as well.

I get that, I'd just like someone to look at what is potentially doable with podman before we settle on the API, adding fields is cheap, changing them requires a revision and I'd rather not do that because nobody was willing to look at podman.

See also:

• https://kind.sigs.k8s.io/docs/design/principles/
• https://kind.sigs.k8s.io/docs/contributing/project-scope/

KIND separates configuration (exporting ports, enabling kubernetes feature gates ...) from environment (HTTP_PROXY ...) as much as possible. Cluster configs are meant to be controlled by something like make e2e-test in projects using kind, such that they just work on different contributor machines. Ideally they should support all node runtimes. If it's impossible to do that that's new territory for the project.

If we have to go that route we should make it fail with a helpful error. But we should evaluate if we even have to do that.

I took a quick look at nerdctl for other reasons recently and it seems to have a compatible --gpu flag so at least only podman should need further investigation.

@BenTheElder what's the way to submit a proposal? You got an example of a specific format that should be used?

Just an issue discussion outlining how this will work.
https://kind.sigs.k8s.io/docs/contributing/getting-started/

Are you fine of the proposal of only supporting "all" mode initially for both podman and docker? when users ask for support specific GPUs then that can be added later. In the implementation I will ensure it's using a string instead of bool such that eventually people could specify gpu0 instead of all

Given feedback that all is fine for current interested users so far, we can probably just do a bool and we can change it in the next API revision.

But nobody has answered if this field is reasonably implementable with podman and what that would look like.