Error during Intel loadBinary: Triton Error [ZE]: 0x78000011

[Stonepia]

Describe the bug

When running on the latest PyTorch nightly with Triton, I got the following error. This error happens before, but we don't have a concrete result. So I re-created one.

Error during Intel loadBinary: Triton Error [ZE]: 0x78000011

Traceback (most recent call last):
  File "F:\ao\reproducer.py", line 143, in <module>
    int_scaled_matmul_kernel(a, b, scales1, c, config)
  File "F:\ao\reproducer.py", line 102, in int_scaled_matmul_kernel
    scaled_matmul_kernel_with_block_pointers[grid](
  File "F:\miniforge\envs\nightly\lib\site-packages\triton\runtime\jit.py", line 336, in <lambda>
    return lambda *args, **kwargs: self.run(grid=grid, warmup=False, *args, **kwargs)
  File "F:\miniforge\envs\nightly\lib\site-packages\triton\runtime\jit.py", line 585, in run
    kernel.run(grid_0, grid_1, grid_2, stream, kernel.function, kernel.packed_metadata,
  File "F:\miniforge\envs\nightly\lib\site-packages\triton\compiler\compiler.py", line 417, in __getattribute__
    self._init_handles()
  File "F:\miniforge\envs\nightly\lib\site-packages\triton\compiler\compiler.py", line 412, in _init_handles
    self.module, self.function, self.n_regs, self.n_spills = driver.active.utils.load_binary(
  File "F:\miniforge\envs\nightly\lib\site-packages\triton\backends\intel\driver.py", line 208, in load_binary
    return self.shared_library.load_binary(args)
RuntimeError: Triton Error [ZE]: 0x78000011

Attached the reproducer.py.txt (you might need to change it to .py) and mlir_dump.log for your convenience.

mlir_dump.log

reproducer.py.txt

Environment details

OS: Windows
Platform: BMG
Driver version: 32.0.101.6734
Level Zero: Included in the driver, should be under "C:\Program Files\LevelZeroSDK\1.21.6"
igc version: 32.0.101.6734

torch version:'2.8.0.dev20250408+xpu'
Triton version: 3.3.0+git0bcc8265 (commit: 0bcc826)

Other info that may help:

> (Get-Item C:\Windows\system32\ze_loader.dll).VersionInfo.ProductVersion
1.20.6
> cl --version
Microsoft (R) C/C++ Optimizing Compiler Version 19.43.34809 for x64

[anmyachev]

Hi @pbchekin, @kwasd, could you update driver upto 32.0.101.6734 version at least on one runner with BMG+WIN? Research into this issue is currently blocked by this.

Note: this problem is not reproduced on either main branch or 0bcc826, regardless of pip flow or flow with DLE. The problem also does not reproduce if only L0 SDK is updated in CI to version 1.21.6.

@Stonepia thanks for the detailed issue!

I had to explicitly specify xpu device for tensors to run your reproducer . How did your reproducer work without this?

[pbchekin]

Hi @pbchekin, @kwasd, could you update driver upto 32.0.101.6734 version at least on one runner with BMG+WIN? Research into this issue is currently blocked by this.

Done, all Windows runners have been updated to 32.0.101.6734.

> (Get-WmiObject Win32_VideoController | where {$_.VideoProcessor -like "*Intel*" }).DriverVersion
32.0.101.6734

[anmyachev]

Hi @pbchekin, @kwasd, could you update driver upto 32.0.101.6734 version at least on one runner with BMG+WIN? Research into this issue is currently blocked by this.

Done, all Windows runners have been updated to 32.0.101.6734.

> (Get-WmiObject Win32_VideoController | where {$_.VideoProcessor -like "*Intel*" }).DriverVersion
32.0.101.6734

Many thanks!

[Stonepia]

I had to explicitly specify xpu device for tensors to run your reproducer . How did your reproducer work without this?

Hi @anmyachev sorry for my mistake. I forgot to add to('xpu') for the reproducer. And the 0x78000011 happens before the actually run, thus I missed it.
I have changed the reproducer a little bit to add to('xpu') on it.

[anmyachev]

Binary reproducer for driver team is in progress. Example of spv file with the same problem is below, for some reason the problem is not observed with the particular reproducer of this issue, however we have similar problems after updating the driver in our CI to 6734 version:

Note: remove .txt from filename
invalid_kernel_name_on_bmg.spv.txt