[55529.312046] VERIFY(avl_find(tree, new_node, &where) == NULL) failed [55529.312054] PANIC at avl.c:641:avl_add() [55529.312056] Showing stack for process 31083 [55529.312062] CPU: 2 PID: 31083 Comm: z_livelist_dest Tainted: P OE 5.4.0-64-generic #72~18.04.1-Ubuntu [55529.312064] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 0.0.0 02/06/2015 [55529.312065] Call Trace: [55529.312139] dump_stack+0x6d/0x95 [55529.312167] spl_dumpstack+0x29/0x30 [spl] [55529.312176] spl_panic+0xc8/0x110 [spl] [55529.312205] ? ___slab_alloc+0x2a0/0x590 [55529.312215] ? spl_kmem_alloc+0xdc/0x140 [spl] [55529.312220] ? __slab_alloc+0x20/0x40 [55529.312229] ? _cond_resched+0x19/0x40 [55529.312232] ? __kmalloc_node+0x319/0x330 [55529.312614] ? dsl_deadlist_load_cache+0x240/0x240 [zfs] [55529.312625] avl_add+0xa1/0xb0 [zavl] [55529.312704] dsl_livelist_iterate+0x16b/0x1d0 [zfs] [55529.312756] bpobj_iterate_blkptrs+0x155/0x360 [zfs] [55529.312830] ? dsl_deadlist_load_cache+0x240/0x240 [zfs] [55529.312878] bpobj_iterate_impl+0x302/0x4a0 [zfs] [55529.312948] ? dsl_deadlist_load_cache+0x240/0x240 [zfs] [55529.313007] bpobj_iterate_nofree+0x16/0x20 [zfs] [55529.313094] dsl_process_sub_livelist+0x70/0xb0 [zfs] [55529.313179] ? dsl_deadlist_cache_compare+0x20/0x20 [zfs] [55529.313375] spa_livelist_delete_cb+0x199/0x310 [zfs] [55529.313384] ? __schedule+0x125/0x720 [55529.313518] zthr_procedure+0x144/0x160 [zfs] [55529.313626] ? zrl_is_locked+0x20/0x20 [zfs] [55529.313636] thread_generic_wrapper+0x74/0x90 [spl] [55529.313664] kthread+0x121/0x140 [55529.313673] ? __thread_exit+0x20/0x20 [spl] [55529.313677] ? kthread_park+0x90/0x90 [55529.313684] ret_from_fork+0x35/0x40 [55704.757618] INFO: task z_livelist_dest:31083 blocked for more than 120 seconds. [55704.757622] Tainted: P OE 5.4.0-64-generic #72~18.04.1-Ubuntu [55704.757623] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [55704.757624] z_livelist_dest D 0 31083 2 0x80004000 [55704.757626] Call Trace: [55704.757632] __schedule+0x293/0x720 [55704.757635] schedule+0x33/0xa0 [55704.757641] spl_panic+0xfa/0x110 [spl] [55704.757644] ? ___slab_alloc+0x2a0/0x590 [55704.757646] ? spl_kmem_alloc+0xdc/0x140 [spl] [55704.757647] ? __slab_alloc+0x20/0x40 [55704.757648] ? _cond_resched+0x19/0x40 [55704.757649] ? __kmalloc_node+0x319/0x330 [55704.757693] ? dsl_deadlist_load_cache+0x240/0x240 [zfs] [55704.757695] avl_add+0xa1/0xb0 [zavl] [55704.757725] dsl_livelist_iterate+0x16b/0x1d0 [zfs] [55704.757745] bpobj_iterate_blkptrs+0x155/0x360 [zfs] [55704.757773] ? dsl_deadlist_load_cache+0x240/0x240 [zfs] [55704.757792] bpobj_iterate_impl+0x302/0x4a0 [zfs] [55704.757819] ? dsl_deadlist_load_cache+0x240/0x240 [zfs] [55704.757838] bpobj_iterate_nofree+0x16/0x20 [zfs] [55704.757863] dsl_process_sub_livelist+0x70/0xb0 [zfs] [55704.757889] ? dsl_deadlist_cache_compare+0x20/0x20 [zfs] [55704.757922] spa_livelist_delete_cb+0x199/0x310 [zfs] [55704.757924] ? __schedule+0x125/0x720 [55704.757959] zthr_procedure+0x144/0x160 [zfs] [55704.757994] ? zrl_is_locked+0x20/0x20 [zfs] [55704.757997] thread_generic_wrapper+0x74/0x90 [spl] [55704.758000] kthread+0x121/0x140 [55704.758003] ? __thread_exit+0x20/0x20 [spl] [55704.758004] ? kthread_park+0x90/0x90 [55704.758006] ret_from_fork+0x35/0x40 [55825.589907] INFO: task z_livelist_dest:31083 blocked for more than 241 seconds. [55825.589911] Tainted: P OE 5.4.0-64-generic #72~18.04.1-Ubuntu [55825.589912] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [55825.589913] z_livelist_dest D 0 31083 2 0x80004000 [55825.589915] Call Trace: [55825.589921] __schedule+0x293/0x720 [55825.589925] schedule+0x33/0xa0 [55825.589932] spl_panic+0xfa/0x110 [spl] [55825.589936] ? ___slab_alloc+0x2a0/0x590 [55825.589939] ? spl_kmem_alloc+0xdc/0x140 [spl] [55825.589940] ? __slab_alloc+0x20/0x40 [55825.589942] ? _cond_resched+0x19/0x40 [55825.589943] ? __kmalloc_node+0x319/0x330 [55825.590048] ? dsl_deadlist_load_cache+0x240/0x240 [zfs] [55825.590050] avl_add+0xa1/0xb0 [zavl] [55825.590125] dsl_livelist_iterate+0x16b/0x1d0 [zfs] [55825.590199] bpobj_iterate_blkptrs+0x155/0x360 [zfs] [55825.590254] ? dsl_deadlist_load_cache+0x240/0x240 [zfs] [55825.590277] bpobj_iterate_impl+0x302/0x4a0 [zfs] [55825.590306] ? dsl_deadlist_load_cache+0x240/0x240 [zfs] [55825.590327] bpobj_iterate_nofree+0x16/0x20 [zfs] [55825.590355] dsl_process_sub_livelist+0x70/0xb0 [zfs] [55825.590382] ? dsl_deadlist_cache_compare+0x20/0x20 [zfs] [55825.590416] spa_livelist_delete_cb+0x199/0x310 [zfs] [55825.590418] ? __schedule+0x125/0x720 [55825.590453] zthr_procedure+0x144/0x160 [zfs] [55825.590491] ? zrl_is_locked+0x20/0x20 [zfs] [55825.590495] thread_generic_wrapper+0x74/0x90 [spl] [55825.590500] kthread+0x121/0x140 [55825.590502] ? __thread_exit+0x20/0x20 [spl] [55825.590504] ? kthread_park+0x90/0x90 [55825.590507] ret_from_fork+0x35/0x40 [55946.427629] INFO: task z_livelist_dest:31083 blocked for more than 362 seconds. [55946.427634] Tainted: P OE 5.4.0-64-generic #72~18.04.1-Ubuntu [55946.427635] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [55946.427636] z_livelist_dest D 0 31083 2 0x80004000 [55946.427638] Call Trace: [55946.427647] __schedule+0x293/0x720 [55946.427652] schedule+0x33/0xa0 [55946.427660] spl_panic+0xfa/0x110 [spl] [55946.427664] ? ___slab_alloc+0x2a0/0x590 [55946.427668] ? spl_kmem_alloc+0xdc/0x140 [spl] [55946.427671] ? __slab_alloc+0x20/0x40 [55946.427673] ? _cond_resched+0x19/0x40 [55946.427674] ? __kmalloc_node+0x319/0x330 [55946.427782] ? dsl_deadlist_load_cache+0x240/0x240 [zfs] [55946.427788] avl_add+0xa1/0xb0 [zavl] [55946.427843] dsl_livelist_iterate+0x16b/0x1d0 [zfs] [55946.427886] bpobj_iterate_blkptrs+0x155/0x360 [zfs] [55946.427934] ? dsl_deadlist_load_cache+0x240/0x240 [zfs] [55946.427975] bpobj_iterate_impl+0x302/0x4a0 [zfs] [55946.428019] ? dsl_deadlist_load_cache+0x240/0x240 [zfs] [55946.428059] bpobj_iterate_nofree+0x16/0x20 [zfs] [55946.428109] dsl_process_sub_livelist+0x70/0xb0 [zfs] [55946.428167] ? dsl_deadlist_cache_compare+0x20/0x20 [zfs] [55946.428233] spa_livelist_delete_cb+0x199/0x310 [zfs] [55946.428238] ? __schedule+0x125/0x720 [55946.428306] zthr_procedure+0x144/0x160 [zfs] [55946.428377] ? zrl_is_locked+0x20/0x20 [zfs] [55946.428399] thread_generic_wrapper+0x74/0x90 [spl] [55946.428404] kthread+0x121/0x140 [55946.428409] ? __thread_exit+0x20/0x20 [spl] [55946.428411] ? kthread_park+0x90/0x90 [55946.428414] ret_from_fork+0x35/0x40 [56067.254550] INFO: task z_livelist_dest:31083 blocked for more than 483 seconds. [56067.254553] Tainted: P OE 5.4.0-64-generic #72~18.04.1-Ubuntu [56067.254554] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [56067.254555] z_livelist_dest D 0 31083 2 0x80004000 [56067.254556] Call Trace: [56067.254562] __schedule+0x293/0x720 [56067.254568] schedule+0x33/0xa0 [56067.254574] spl_panic+0xfa/0x110 [spl] [56067.254576] ? ___slab_alloc+0x2a0/0x590 [56067.254579] ? spl_kmem_alloc+0xdc/0x140 [spl] [56067.254580] ? __slab_alloc+0x20/0x40 [56067.254581] ? _cond_resched+0x19/0x40 [56067.254582] ? __kmalloc_node+0x319/0x330 [56067.254627] ? dsl_deadlist_load_cache+0x240/0x240 [zfs] [56067.254629] avl_add+0xa1/0xb0 [zavl] [56067.254659] dsl_livelist_iterate+0x16b/0x1d0 [zfs] [56067.254682] bpobj_iterate_blkptrs+0x155/0x360 [zfs] [56067.254711] ? dsl_deadlist_load_cache+0x240/0x240 [zfs] [56067.254731] bpobj_iterate_impl+0x302/0x4a0 [zfs] [56067.254759] ? dsl_deadlist_load_cache+0x240/0x240 [zfs] [56067.254779] bpobj_iterate_nofree+0x16/0x20 [zfs] [56067.254806] dsl_process_sub_livelist+0x70/0xb0 [zfs] [56067.254832] ? dsl_deadlist_cache_compare+0x20/0x20 [zfs] [56067.254864] spa_livelist_delete_cb+0x199/0x310 [zfs] [56067.254865] ? __schedule+0x125/0x720 [56067.254900] zthr_procedure+0x144/0x160 [zfs] [56067.254934] ? zrl_is_locked+0x20/0x20 [zfs] [56067.254938] thread_generic_wrapper+0x74/0x90 [spl] [56067.254941] kthread+0x121/0x140 [56067.254943] ? __thread_exit+0x20/0x20 [spl] [56067.254944] ? kthread_park+0x90/0x90 [56067.254946] ret_from_fork+0x35/0x40 [56188.086793] INFO: task z_livelist_dest:31083 blocked for more than 604 seconds. [56188.086797] Tainted: P OE 5.4.0-64-generic #72~18.04.1-Ubuntu [56188.086798] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [56188.086799] z_livelist_dest D 0 31083 2 0x80004000 [56188.086801] Call Trace: [56188.086808] __schedule+0x293/0x720 [56188.086812] schedule+0x33/0xa0 [56188.086819] spl_panic+0xfa/0x110 [spl] [56188.086822] ? ___slab_alloc+0x2a0/0x590 [56188.086825] ? spl_kmem_alloc+0xdc/0x140 [spl] [56188.086826] ? __slab_alloc+0x20/0x40 [56188.086828] ? _cond_resched+0x19/0x40 [56188.086829] ? __kmalloc_node+0x319/0x330 [56188.086895] ? dsl_deadlist_load_cache+0x240/0x240 [zfs] [56188.086897] avl_add+0xa1/0xb0 [zavl] [56188.086927] dsl_livelist_iterate+0x16b/0x1d0 [zfs] [56188.086947] bpobj_iterate_blkptrs+0x155/0x360 [zfs] [56188.086976] ? dsl_deadlist_load_cache+0x240/0x240 [zfs] [56188.086995] bpobj_iterate_impl+0x302/0x4a0 [zfs] [56188.087024] ? dsl_deadlist_load_cache+0x240/0x240 [zfs] [56188.087043] bpobj_iterate_nofree+0x16/0x20 [zfs] [56188.087071] dsl_process_sub_livelist+0x70/0xb0 [zfs] [56188.087100] ? dsl_deadlist_cache_compare+0x20/0x20 [zfs] [56188.087138] spa_livelist_delete_cb+0x199/0x310 [zfs] [56188.087139] ? __schedule+0x125/0x720 [56188.087177] zthr_procedure+0x144/0x160 [zfs] [56188.087240] ? zrl_is_locked+0x20/0x20 [zfs] [56188.087245] thread_generic_wrapper+0x74/0x90 [spl] [56188.087248] kthread+0x121/0x140 [56188.087251] ? __thread_exit+0x20/0x20 [spl] [56188.087253] ? kthread_park+0x90/0x90 [56188.087255] ret_from_fork+0x35/0x40