Intel ARC Support

[linus378]

I was wondering if Intel ARC Gpu's work with this. Could not read anything about this here.

[linus378]

Also i do wonder if this could support 2 gpu's so you don't have too offload anything into RAM. Such as a arc a770 and a rx 6600.

[dan9070]

It doesn't have support for OneAPI or OpenVINO currently from my knowledge, as I myself own an Intel Arc GPU.

[mmccool]

It doesn't, unfortunately. I really wish it did though as I have a dual A770 system myself (and these cards have a lot of VRAM for the price, and also good low-precision AI accelerators, etc). For now I'm running on CPU which is, of course, horribly slow.

However, one issue is that Intel's support for pytorch on its GPUs needs a special version based on pytorch 1.10 (see https://www.intel.com/content/www/us/en/developer/articles/technical/introducing-intel-extension-for-pytorch-for-gpus.html) but this system uses pytorch 2.0.0. As soon as Intel gpu support for pytorch 2.0.0 comes out though I'm hoping support can be extended in this system (if I can find time maybe I'll even be able to contribute some patches). For CPU pytorch 2.0.0 is already supported: https://intel.github.io/intel-extension-for-pytorch/latest/tutorials/releases.html

In the meantime, it would be great if the readme could at least be updated to say WHAT GPUs are supported.

BTW The one-click installer also fails if you don't have an NVIDIA GPU, even if you select "None". I had to go the git clone route.

[mmccool]

Multi-GPU support for multiple Intel GPUs would, of course, also be nice. MultiGPU is supported for other cards, should not (in theory) be a problem. I personally don't really care about mixing GPUs from different vendors, though :)

A bonus would be the ability to use Intel integrated graphics, although they have limited VRAM capabilities, but maybe good enough for some simple things.

[rattlecanblack]

Would love to see this as well, with the power and amount of VRAM the arc is a great little card for those of us that do more compute stuff than gaming, especially considering the price.

[miraged3]

Intel has released torch 2.0 support for arc gpus. https://github.com/intel/intel-extension-for-pytorch/releases/tag/v2.0.110%2Bxpu

[itlackey]

Does the release of pytorch 2 support move things forward for Arc support?

[oobabooga]

I have created a pinned thread for Intel Arc discussion and welcome you to move the discussion there: #3761

To my knowledge, llama-cpp-python should work with GPU acceleration on Intel Arc as long as you compile it with CLBLAST. See https://github.com/oobabooga/text-generation-webui#amd-metal-intel-arc-and-cpus-without-avcx2

[itlackey]

You rock! Thank you for all the hard work on this project!

[abhilash1910]

@oobabooga Intel Arc GPU support is in the pipeline ; the support integration would be started in 2-3 weeks time (by myself) . There are some other items in the pipeline at Intel which we are covering - and we plan to add this to our GPU soon.

[oobabooga]

@abhilash1910 thanks for the info. For XPU inference on transformers, is it currently enough to do

model.to(torch.device('xpu'))

or similar, like here?

Does any special pytorch import command have to be made?

[itlackey]

I found this while researching how this all works.

https://intel.github.io/intel-extension-for-pytorch/xpu/latest/tutorials/examples.html

It looks like there shouldn't be much to change, but I'm new to LLM/AI development. So I may be missing something.

[oobabooga]

Thanks @itlackey. I guess it should be a few changed lines (for the transformers loader):

1. model = model.to("xpu") in modules.models.huggingface_loader
2. return input_ids.to(torch.device('xpu')) in modules.text-generation.encode.

It would be nice if someone could test this.

[itlackey]

I'll have time in a few days and will give it a shot. We may also need to make some changes to the installer and/or docker image to load the Intel libs and driver and recompile llama.cpp to get xpu to work.
I was able to do this with a docker image for FastChat and llama.cpp. We should be able to do the same for textgen.