OpenZFS 2.0.x and Xeon Phi incompatibility

[thedarave]

System information

Type	Version/Name
Distribution Name	Ubuntu
Distribution Version	21.04
Linux Kernel	generic
Architecture	amd64
ZFS Version	2.0.2-1ubuntu5
SPL Version	2.0.2-1ubuntu5

Describe the problem you're observing

I tried moving one of my ASRockRack 2U4N-F/X200/X200D6HM Xeon Phi nodes to OpenZFS 2.0 just for the hell of it this weekend, and found that no matter what I used, it crashed in the same place: insmod zcommon.ko. This included both native Ubuntu builds from 21.04 (2.0.2), and PPA repos for 20.04 (2.0.4). Here's the crash message I see:

Apr 24 19:38:45 ubuntu kernel: [   41.985812] spl: loading out-of-tree module taints kernel.
Apr 24 19:38:45 ubuntu kernel: [   42.006663] znvpair: module license 'CDDL' taints kernel.
Apr 24 19:38:45 ubuntu kernel: [   42.006663] znvpair: module license 'CDDL' taints kernel.
Apr 24 19:38:45 ubuntu kernel: [   42.006677] Disabling lock debugging due to kernel taint
Apr 24 19:38:45 ubuntu kernel: [   42.052754] invalid opcode: 0000 [#1] SMP PTI
Apr 24 19:38:45 ubuntu kernel: [   42.052790] CPU: 13 PID: 3842 Comm: modprobe Tainted: P           O      5.11.0-16-generic #17-Ubuntu
Apr 24 19:38:45 ubuntu kernel: [   42.052801] Hardware name: ASRockRack 2U4N-F/X200/X200D6HM, BIOS L1.02 05/11/2018
Apr 24 19:38:45 ubuntu kernel: [   42.052808] RIP: 0010:fletcher_4_avx512bw_byteswap+0x4e/0xa0 [zcommon]
Apr 24 19:38:45 ubuntu kernel: [   42.052885] Code: 62 d1 fe 48 6f 4d 01 62 d1 fe 48 6f 55 02 62 d1 fe 48 6f 5d 03 62 f1 fe 48 6f 2d 3d 1a 00 00 4c 39 e3 73 2d 62 f2 7d 48 35 23 <62> f2 5d 48 00 e5 62 f1 fd 48 d4 c4 62 f1 f5 48 d4 c8 62 f1 ed 48
Apr 24 19:38:45 ubuntu kernel: [   42.052896] RSP: 0018:ffffb0fae15fbad8 EFLAGS: 00010087
Apr 24 19:38:45 ubuntu kernel: [   42.052908] RAX: 0000000000000000 RBX: ffff9b23987c0000 RCX: 0000000000000000
Apr 24 19:38:45 ubuntu kernel: [   42.052915] RDX: 00000000ffffffff RSI: ffff9b23987c0000 RDI: ffff9b2395258000
Apr 24 19:38:45 ubuntu kernel: [   42.052922] RBP: ffffb0fae15fbaf0 R08: ffffffffc074cb80 R09: ffffb0fae15fbb00
Apr 24 19:38:45 ubuntu kernel: [   42.052929] R10: c1c9042535540000 R11: 0a288a716f260000 R12: ffff9b23987e0000
Apr 24 19:38:45 ubuntu kernel: [   42.052936] R13: ffffb0fae15fbb00 R14: 0000000000020000 R15: 00000009666478d7
Apr 24 19:38:45 ubuntu kernel: [   42.052944] FS:  00007f2aee236580(0000) GS:ffff9b323c140000(0000) knlGS:0000000000000000
Apr 24 19:38:45 ubuntu kernel: [   42.052953] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Apr 24 19:38:45 ubuntu kernel: [   42.052960] CR2: 0000556446932000 CR3: 000000014e5f2000 CR4: 00000000001506e0
Apr 24 19:38:45 ubuntu kernel: [   42.052968] Call Trace:
Apr 24 19:38:45 ubuntu kernel: [   42.052983]  fletcher_4_byteswap_impl+0x58/0x90 [zcommon]
Apr 24 19:38:45 ubuntu kernel: [   42.053023]  ? __kmalloc_node+0x22f/0x2b0
Apr 24 19:38:45 ubuntu kernel: [   42.053044]  ? kernel_init_free_pages+0x4a/0x60
Apr 24 19:38:45 ubuntu kernel: [   42.053056]  fletcher_4_byteswap+0x63/0xa0 [zcommon]
Apr 24 19:38:45 ubuntu kernel: [   42.053089]  ? fletcher_4_byteswap_impl+0x90/0x90 [zcommon]
Apr 24 19:38:45 ubuntu kernel: [   42.053121]  fletcher_4_benchmark_impl.constprop.0+0x9f/0x1b0 [zcommon]
Apr 24 19:38:45 ubuntu kernel: [   42.053154]  ? kvmalloc+0xe/0x10 [spl]
Apr 24 19:38:45 ubuntu kernel: [   42.053238]  fletcher_4_benchmark+0xa8/0xd0 [zcommon]
Apr 24 19:38:45 ubuntu kernel: [   42.053272]  fletcher_4_init+0xe/0x90 [zcommon]
Apr 24 19:38:45 ubuntu kernel: [   42.053306]  zcommon_init+0x1c/0x1000 [zcommon]
Apr 24 19:38:45 ubuntu kernel: [   42.053338]  ? 0xffffffffc06d2000
Apr 24 19:38:45 ubuntu kernel: [   42.053347]  do_one_initcall+0x48/0x1d0
Apr 24 19:38:45 ubuntu kernel: [   42.053365]  ? kmem_cache_alloc_trace+0xf6/0x200
Apr 24 19:38:45 ubuntu kernel: [   42.053376]  ? do_init_module+0x28/0x290
Apr 24 19:38:45 ubuntu kernel: [   42.053390]  do_init_module+0x62/0x290
Apr 24 19:38:45 ubuntu kernel: [   42.053400]  load_module+0x6fd/0x780
Apr 24 19:38:45 ubuntu kernel: [   42.053411]  __do_sys_finit_module+0xc2/0x120
Apr 24 19:38:45 ubuntu kernel: [   42.053424]  __x64_sys_finit_module+0x1a/0x20
Apr 24 19:38:45 ubuntu kernel: [   42.053434]  do_syscall_64+0x38/0x90
Apr 24 19:38:45 ubuntu kernel: [   42.053456]  entry_SYSCALL_64_after_hwframe+0x44/0xa9
Apr 24 19:38:45 ubuntu kernel: [   42.053471] RIP: 0033:0x7f2aee371f6d
Apr 24 19:38:45 ubuntu kernel: [   42.053482] Code: 28 0d 00 0f 05 eb a9 66 0f 1f 44 00 00 f3 0f 1e fa 48 89 f8 48 89 f7 48 89 d6 48 89 ca 4d 89 c2 4d 89 c8 4c 8b 4c 24 08 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d cb de 0c 00 f7 d8 64 89 01 48
Apr 24 19:38:45 ubuntu kernel: [   42.053491] RSP: 002b:00007ffe87e697c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000139
Apr 24 19:38:45 ubuntu kernel: [   42.053502] RAX: ffffffffffffffda RBX: 0000556446932440 RCX: 00007f2aee371f6d
Apr 24 19:38:45 ubuntu kernel: [   42.053509] RDX: 0000000000000000 RSI: 00005564461f6260 RDI: 0000000000000005
Apr 24 19:38:45 ubuntu kernel: [   42.053515] RBP: 0000000000040000 R08: 0000000000000000 R09: 0000556446932440
Apr 24 19:38:45 ubuntu kernel: [   42.053522] R10: 0000000000000005 R11: 0000000000000246 R12: 00005564461f6260
Apr 24 19:38:45 ubuntu kernel: [   42.053529] R13: 0000000000000000 R14: 0000556446932400 R15: 0000556446932440
Apr 24 19:38:45 ubuntu kernel: [   42.053539] Modules linked in: zcommon(PO+) znvpair(PO) spl(O) intel_rapl_msr mei_me mei intel_rapl_common sb_edac x86_pkg_temp_thermal intel_powerclamp coretemp rapl ipmi_ssif intel_cstate efi_pstore acpi_ipmi ipmi_si ipmi_devintf ipmi_msghandler acpi_pad mac_hid sch_fq_codel msr parport_pc ppdev lp parport ip_tables x_tables autofs4 overlay nls_iso8859_1 dm_mirror dm_region_hash dm_log uas usb_storage crct10dif_pclmul ast crc32_pclmul ghash_clmulni_intel drm_vram_helper drm_ttm_helper aesni_intel ttm crypto_simd cryptd drm_kms_helper glue_helper syscopyarea sysfillrect sysimgblt fb_sys_fops igb cec rc_core ahci i2c_i801 dca xhci_pci drm lpc_ich i2c_smbus libahci xhci_pci_renesas i2c_algo_bit wmi
Apr 24 19:38:45 ubuntu kernel: [   42.053739] ---[ end trace 0846c9c00d259779 ]---
Apr 24 19:38:45 ubuntu kernel: [   42.269868] RIP: 0010:fletcher_4_avx512bw_byteswap+0x4e/0xa0 [zcommon]
Apr 24 19:38:45 ubuntu kernel: [   42.269910] Code: 62 d1 fe 48 6f 4d 01 62 d1 fe 48 6f 55 02 62 d1 fe 48 6f 5d 03 62 f1 fe 48 6f 2d 3d 1a 00 00 4c 39 e3 73 2d 62 f2 7d 48 35 23 <62> f2 5d 48 00 e5 62 f1 fd 48 d4 c4 62 f1 f5 48 d4 c8 62 f1 ed 48
Apr 24 19:38:45 ubuntu kernel: [   42.269920] RSP: 0018:ffffb0fae15fbad8 EFLAGS: 00010087
Apr 24 19:38:45 ubuntu kernel: [   42.269930] RAX: 0000000000000000 RBX: ffff9b23987c0000 RCX: 0000000000000000
Apr 24 19:38:45 ubuntu kernel: [   42.269936] RDX: 00000000ffffffff RSI: ffff9b23987c0000 RDI: ffff9b2395258000
Apr 24 19:38:45 ubuntu kernel: [   42.269943] RBP: ffffb0fae15fbaf0 R08: ffffffffc074cb80 R09: ffffb0fae15fbb00
Apr 24 19:38:45 ubuntu kernel: [   42.269949] R10: c1c9042535540000 R11: 0a288a716f260000 R12: ffff9b23987e0000
Apr 24 19:38:45 ubuntu kernel: [   42.269956] R13: ffffb0fae15fbb00 R14: 0000000000020000 R15: 00000009666478d7
Apr 24 19:38:45 ubuntu kernel: [   42.269963] FS:  00007f2aee236580(0000) GS:ffff9b323c140000(0000) knlGS:0000000000000000
Apr 24 19:38:45 ubuntu kernel: [   42.269970] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Apr 24 19:38:45 ubuntu kernel: [   42.269977] CR2: 0000556446932000 CR3: 000000014e5f2000 CR4: 00000000001506e0

If I am reading this right, it looks like it is crashing in fletcher_4_avx512.c trying to utilize the biteswap AVX instruction. This instruction doesn't exist in the AVX512 instruction set on the Xeon Phi. ( https://en.wikipedia.org/wiki/AVX-512#CPUs_with_AVX-512 ) As you'll note, it does support the AVX512 Foundation instructions and Conflict Detection instructions, but nothing else "common" beyond that.
Here's the top /proc/cpuinfo ID in case it is useful:

processor : 0
vendor_id : GenuineIntel
cpu family : 6
model : 87
model name : Intel(R) Xeon Phi(TM) CPU 7210 @ 1.30GHz
stepping : 1
microcode : 0x1b6
cpu MHz : 1000.000
cache size : 1024 KB
physical id : 0
siblings : 256
core id : 0
cpu cores : 64
apicid : 0
initial apicid : 0
fpu : yes
fpu_exception : yes
cpuid level : 13
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes64 monitor ds_cpl est tm2 ssse3 fma cx16 xtpr pdcm sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch ring3mwait cpuid_fault epb pti ibrs ibpb fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms avx512f rdseed adx avx512pf avx512er avx512cd xsaveopt dtherm ida arat pln pts
bugs : cpu_meltdown spectre_v1 spectre_v2 mds msbds_only
bogomips : 2599.80
clflush size : 64
cache_alignment : 64
address sizes : 46 bits physical, 48 bits virtual
power management:

Describe how to reproduce the problem

Execute

insmod zcommon.ko

while running on a Xeon Phi 72xx or on a system that has foundation instructions only support for AVX512.

Include any warning/errors/backtraces from the system logs

[rdolbeau]

Apr 24 19:38:45 ubuntu kernel: [ 42.052808] RIP: 0010:fletcher_4_avx512bw_byteswap+0x4e/0xa0 [zcommon]

That's the AVX512BW-only function, which indeed cannot work on KNL.

Apr 24 19:38:45 ubuntu kernel: [ 42.053238] fletcher_4_benchmark+0xa8/0xd0 [zcommon]

It seems when the offending function is compiled-in, the benchmark
function will test it and cause the crash.

It should have it's own fletcher_4_avx512bw_valid, but it is using
fletcher_4_avx512f_valid (at least that's part of the problem - I haven't checked if the
benchmarking code calls the .valid function).

@thedarave Can you try what happens with https://github.com/rdolbeau/zfs/tree/fletcher_avx512bw_fix ?

[thedarave]

root@testbed:~# zpool --version
zfs-2.0.4-0york2~20.04
zfs-kmod-2.1.99-1
root@testbed:~# zfs --version
zfs-2.0.4-0york2~20.04
zfs-kmod-2.1.99-1
root@testbed:~#

This gets past the modprobe phase.

Any system level tests that you'd recommend performing or like to see performed?

[rdolbeau]

@thedarave Maybe check the bench numbers in /proc/spl/kstat/zfs/fletcher_4_bench, avx512bw should not be there and avx512[f] should be - and likely be the best (the important thing is, it wasn't disabled). This should be a fairly safe patch, as it just disable the offending function on machines where it won't work.

[thedarave]

root@testbed:~# cat /proc/spl/kstat/zfs/fletcher_4_bench
0 0 0x01 -1 0 17057433023 659609206667
implementation   native         byteswap
scalar           1150427360     934320930
superscalar      1589134352     1172990399
superscalar4     1575950499     1074879373
sse2             2685810484     1358745276
ssse3            2697870676     1086540822
avx2             5159734109     1152542223
avx512f          5653798041     3839492645
fastest          avx512f        avx512f
root@testbed:~#

As requested.

[rdolbeau]

@thedarave Thanks! LGTM. And sorry for having introduced the bug in the first place.

[thedarave]

@rdolbeau No worries! I'm sort of an edge case anyway. Going to close this out unless someone else has something they want me to test.