BUG: workqueue lockup - pool cpus=0 node=0 flags=0x0 nice=0 stuck for 44s! Showing busy workqueues and worker pools: workqueue events: flags=0x100 pwq 2: cpus=0 node=0 flags=0x0 nice=0 active=6 refcnt=7 in-flight: 3791:nsim_fib_event_work nsim_fib_event_work pending: psi_avgs_work, vmstat_shepherd, psi_avgs_work, drm_fb_helper_damage_work pwq 6: cpus=1 node=0 flags=0x0 nice=0 active=2 refcnt=3 pending: 2*nsim_dev_hwstats_traffic_work workqueue events_unbound: flags=0x2 pwq 8: cpus=0-1 flags=0x6 nice=0 active=1 refcnt=2 pending: flush_memcg_stats_dwork workqueue events_unbound: flags=0x2 pwq 8: cpus=0-1 flags=0x6 nice=0 active=2 refcnt=3 in-flight: 4801:linkwatch_event pending: nsim_dev_trap_report_work pwq 8: cpus=0-1 flags=0x6 nice=0 active=1 refcnt=2 in-flight: 3699:nsim_dev_trap_report_work workqueue events_power_efficient: flags=0x182 pwq 8: cpus=0-1 flags=0x6 nice=0 active=2 refcnt=3 in-flight: 26:gc_worker pending: fb_flashcursor pwq 8: cpus=0-1 flags=0x6 nice=0 active=1 refcnt=2 in-flight: 1545:wg_ratelimiter_gc_entries workqueue mm_percpu_wq: flags=0x8 pwq 2: cpus=0 node=0 flags=0x0 nice=0 active=1 refcnt=2 pending: vmstat_update pwq 6: cpus=1 node=0 flags=0x0 nice=0 active=1 refcnt=2 pending: vmstat_update workqueue ipv6_addrconf: flags=0x6000a pwq 8: cpus=0-1 flags=0x6 nice=0 active=1 refcnt=7 in-flight: 2074(RESCUER):addrconf_verify_work inactive: 2*addrconf_verify_work workqueue wg-kex-wg0: flags=0x6 pwq 8: cpus=0-1 flags=0x6 nice=0 active=2 refcnt=3 in-flight: 814:wg_packet_handshake_send_worker pending: wg_packet_handshake_send_worker workqueue wg-kex-wg1: flags=0x6 pwq 8: cpus=0-1 flags=0x6 nice=0 active=2 refcnt=3 pending: 2*wg_packet_handshake_send_worker workqueue wg-kex-wg2: flags=0x6 pwq 8: cpus=0-1 flags=0x6 nice=0 active=2 refcnt=3 in-flight: 1565:wg_packet_handshake_send_worker pending: wg_packet_handshake_send_worker workqueue wg-kex-wg0: flags=0x124 pwq 2: cpus=0 node=0 flags=0x0 nice=0 active=2 refcnt=3 in-flight: 3321:wg_packet_handshake_receive_worker pending: wg_packet_handshake_receive_worker pwq 6: cpus=1 node=0 flags=0x0 nice=0 active=2 refcnt=3 in-flight: 3774:wg_packet_handshake_receive_worker wg_packet_handshake_receive_worker workqueue wg-kex-wg0: flags=0x6 pwq 8: cpus=0-1 flags=0x6 nice=0 active=1 refcnt=2 in-flight: 1554:wg_packet_handshake_send_worker workqueue wg-kex-wg1: flags=0x124 pwq 2: cpus=0 node=0 flags=0x0 nice=0 active=1 refcnt=2 in-flight: 784:wg_packet_handshake_receive_worker pwq 6: cpus=1 node=0 flags=0x0 nice=0 active=2 refcnt=3 in-flight: 3322:wg_packet_handshake_receive_worker wg_packet_handshake_receive_worker workqueue wg-kex-wg1: flags=0x6 pwq 8: cpus=0-1 flags=0x6 nice=0 active=2 refcnt=3 in-flight: 12:wg_packet_handshake_send_worker pending: wg_packet_handshake_send_worker workqueue wg-kex-wg2: flags=0x124 pwq 2: cpus=0 node=0 flags=0x0 nice=0 active=2 refcnt=3 in-flight: 10:wg_packet_handshake_receive_worker wg_packet_handshake_receive_worker pwq 6: cpus=1 node=0 flags=0x0 nice=0 active=2 refcnt=3 in-flight: 785:wg_packet_handshake_receive_worker wg_packet_handshake_receive_worker workqueue wg-kex-wg2: flags=0x6 pwq 8: cpus=0-1 flags=0x6 nice=0 active=2 refcnt=3 in-flight: 1006:wg_packet_handshake_send_worker pending: wg_packet_handshake_send_worker pool 2: cpus=0 node=0 flags=0x0 nice=0 hung=44s workers=6 idle: 9 3900 pool 6: cpus=1 node=0 flags=0x0 nice=0 hung=1s workers=7 idle: 3477 3475 4981 4871 pool 8: cpus=0-1 flags=0x6 nice=0 hung=0s workers=10 manager: 4556 Showing backtraces of running workers in stalled CPU-bound worker pools: pool 2: task:kworker/0:3 state:R running task stack:0 pid:3321 tgid:3321 ppid:2 task_flags:0x4208060 flags:0x00000010 Workqueue: wg-kex-wg0 wg_packet_handshake_receive_worker Call trace: __switch_to+0x208/0x4f0 arch/arm64/kernel/process.c:742 (T) context_switch kernel/sched/core.c:5260 [inline] __schedule+0xcfc/0x2fec kernel/sched/core.c:6867 preempt_schedule_common kernel/sched/core.c:7051 [inline] preempt_schedule+0xf4/0x254 kernel/sched/core.c:7075 __raw_spin_unlock_irqrestore include/linux/spinlock_api_smp.h:152 [inline] _raw_spin_unlock_irqrestore+0x94/0xac kernel/locking/spinlock.c:194 __debug_check_no_obj_freed lib/debugobjects.c:1108 [inline] debug_check_no_obj_freed+0x278/0x500 lib/debugobjects.c:1129 free_pages_prepare mm/page_alloc.c:1440 [inline] __free_frozen_pages+0x374/0x121c mm/page_alloc.c:2973 free_frozen_pages+0x14/0x20 mm/page_alloc.c:3011 __free_slab+0xe8/0x1ac mm/slub.c:3317 free_slab+0x30/0x104 mm/slub.c:3340 discard_slab mm/slub.c:3346 [inline] __put_partials+0x15c/0x1f0 mm/slub.c:3886 put_cpu_partial+0x1c8/0x364 mm/slub.c:3961 __slab_free+0x270/0x348 mm/slub.c:5952 do_slab_free mm/slub.c:6618 [inline] ___cache_free+0xc4/0xd8 mm/slub.c:6737 qlink_free mm/kasan/quarantine.c:163 [inline] qlist_free_all+0x78/0x130 mm/kasan/quarantine.c:179 kasan_quarantine_reduce+0x148/0x19c mm/kasan/quarantine.c:286 __kasan_kmalloc+0x8c/0xbc mm/kasan/common.c:406 kasan_kmalloc include/linux/kasan.h:263 [inline] __kmalloc_cache_noprof+0x2c0/0x710 mm/slub.c:5776 kmalloc_noprof include/linux/slab.h:957 [inline] kzalloc_noprof include/linux/slab.h:1094 [inline] keypair_create drivers/net/wireguard/noise.c:100 [inline] wg_noise_handshake_begin_session+0xe0/0xa64 drivers/net/wireguard/noise.c:827 wg_packet_send_handshake_response+0x1a8/0x250 drivers/net/wireguard/send.c:96 wg_receive_handshake_packet+0x1b8/0x810 drivers/net/wireguard/receive.c:154 wg_packet_handshake_receive_worker+0xd8/0x314 drivers/net/wireguard/receive.c:213 process_one_work+0x7cc/0x18d4 kernel/workqueue.c:3257 process_scheduled_works kernel/workqueue.c:3340 [inline] worker_thread+0x734/0xb84 kernel/workqueue.c:3421 kthread+0x348/0x5fc kernel/kthread.c:463 ret_from_fork+0x10/0x20 arch/arm64/kernel/entry.S:860 pool 2: task:kworker/0:5 state:R running task stack:0 pid:3791 tgid:3791 ppid:2 task_flags:0x4208060 flags:0x00000010 Workqueue: events nsim_fib_event_work Call trace: __switch_to+0x208/0x4f0 arch/arm64/kernel/process.c:742 (T) context_switch kernel/sched/core.c:5260 [inline] __schedule+0xcfc/0x2fec kernel/sched/core.c:6867 preempt_schedule_common kernel/sched/core.c:7051 [inline] preempt_schedule+0xf4/0x254 kernel/sched/core.c:7075 __raw_spin_unlock_irqrestore include/linux/spinlock_api_smp.h:152 [inline] _raw_spin_unlock_irqrestore+0x94/0xac kernel/locking/spinlock.c:194 spin_unlock_irqrestore include/linux/spinlock.h:406 [inline] __put_partials+0x108/0x1f0 mm/slub.c:3879 put_cpu_partial+0x1c8/0x364 mm/slub.c:3961 __slab_free+0x270/0x348 mm/slub.c:5952 do_slab_free mm/slub.c:6618 [inline] ___cache_free+0xc4/0xd8 mm/slub.c:6737 qlink_free mm/kasan/quarantine.c:163 [inline] qlist_free_all+0x78/0x130 mm/kasan/quarantine.c:179 kasan_quarantine_reduce+0x148/0x19c mm/kasan/quarantine.c:286 __kasan_slab_alloc+0x60/0x90 mm/kasan/common.c:350 kasan_slab_alloc include/linux/kasan.h:253 [inline] slab_post_alloc_hook mm/slub.c:4953 [inline] slab_alloc_node mm/slub.c:5263 [inline] __kmalloc_cache_noprof+0x260/0x710 mm/slub.c:5771 kmalloc_noprof include/linux/slab.h:957 [inline] kzalloc_noprof include/linux/slab.h:1094 [inline] nsim_fib6_rt_nh_add+0x4c/0x220 drivers/net/netdevsim/fib.c:500 nsim_fib6_rt_create drivers/net/netdevsim/fib.c:562 [inline] nsim_fib6_rt_insert drivers/net/netdevsim/fib.c:752 [inline] nsim_fib6_event drivers/net/netdevsim/fib.c:856 [inline] nsim_fib_event drivers/net/netdevsim/fib.c:889 [inline] nsim_fib_event_work+0x9f8/0x4038 drivers/net/netdevsim/fib.c:1493 process_one_work+0x7cc/0x18d4 kernel/workqueue.c:3257 process_scheduled_works kernel/workqueue.c:3340 [inline] worker_thread+0x734/0xb84 kernel/workqueue.c:3421 kthread+0x348/0x5fc kernel/kthread.c:463 ret_from_fork+0x10/0x20 arch/arm64/kernel/entry.S:860 pool 2: task:kworker/0:1 state:R running task stack:0 pid:10 tgid:10 ppid:2 task_flags:0x4208060 flags:0x00000010 Workqueue: wg-kex-wg2 wg_packet_handshake_receive_worker Call trace: __switch_to+0x208/0x4f0 arch/arm64/kernel/process.c:742 (T) context_switch kernel/sched/core.c:5260 [inline] __schedule+0xcfc/0x2fec kernel/sched/core.c:6867 preempt_schedule_common kernel/sched/core.c:7051 [inline] preempt_schedule+0xf4/0x254 kernel/sched/core.c:7075 __local_bh_enable_ip+0x384/0x4a4 kernel/softirq.c:457 local_bh_enable include/linux/bottom_half.h:33 [inline] rcu_read_unlock_bh include/linux/rcupdate.h:936 [inline] mod_peer_timer drivers/net/wireguard/timers.c:38 [inline] wg_timers_any_authenticated_packet_traversal+0x188/0x264 drivers/net/wireguard/timers.c:218 wg_packet_send_handshake_response+0x1c0/0x250 drivers/net/wireguard/send.c:99 wg_receive_handshake_packet+0x1b8/0x810 drivers/net/wireguard/receive.c:154 wg_packet_handshake_receive_worker+0xd8/0x314 drivers/net/wireguard/receive.c:213 process_one_work+0x7cc/0x18d4 kernel/workqueue.c:3257 process_scheduled_works kernel/workqueue.c:3340 [inline] worker_thread+0x734/0xb84 kernel/workqueue.c:3421 kthread+0x348/0x5fc kernel/kthread.c:463 ret_from_fork+0x10/0x20 arch/arm64/kernel/entry.S:860 pool 2: task:kworker/0:2 state:R running task stack:0 pid:784 tgid:784 ppid:2 task_flags:0x4208060 flags:0x00000010 Workqueue: wg-kex-wg1 wg_packet_handshake_receive_worker Call trace: __switch_to+0x208/0x4f0 arch/arm64/kernel/process.c:742 (T) context_switch kernel/sched/core.c:5260 [inline] __schedule+0xcfc/0x2fec kernel/sched/core.c:6867 preempt_schedule_irq+0x80/0x1a0 kernel/sched/core.c:7194 raw_irqentry_exit_cond_resched kernel/entry/common.c:173 [inline] raw_irqentry_exit_cond_resched+0x40/0x44 kernel/entry/common.c:165 irqentry_exit+0x18c/0x474 kernel/entry/common.c:216 exit_to_kernel_mode+0x10/0x1c arch/arm64/kernel/entry-common.c:58 __el1_irq arch/arm64/kernel/entry-common.c:500 [inline] el1_interrupt+0x4c/0x60 arch/arm64/kernel/entry-common.c:510 el1h_64_irq_handler+0x18/0x24 arch/arm64/kernel/entry-common.c:515 el1h_64_irq+0x6c/0x70 arch/arm64/kernel/entry.S:592 memcpy+0x58/0x9c mm/kasan/shadow.c:75 (P) fsquare_fsquare_times lib/crypto/curve25519-hacl64.c:285 [inline] addanddouble_fmonty lib/crypto/curve25519-hacl64.c:521 [inline] ladder_smallloop_cmult_small_loop_step lib/crypto/curve25519-hacl64.c:550 [inline] ladder_smallloop_cmult_small_loop_double_step lib/crypto/curve25519-hacl64.c:560 [inline] ladder_smallloop_cmult_small_loop lib/crypto/curve25519-hacl64.c:570 [inline] ladder_bigloop_cmult_big_loop lib/crypto/curve25519-hacl64.c:583 [inline] ladder_cmult.constprop.0+0xb70/0x33d0 lib/crypto/curve25519-hacl64.c:597 curve25519_generic+0x2b0/0x3cf0 lib/crypto/curve25519-hacl64.c:779 curve25519_arch lib/crypto/curve25519.c:29 [inline] curve25519+0x18/0x44 lib/crypto/curve25519.c:44 mix_dh+0x90/0x11c drivers/net/wireguard/noise.c:411 wg_noise_handshake_create_response+0x2e4/0x4d4 drivers/net/wireguard/noise.c:702 wg_packet_send_handshake_response+0xc4/0x250 drivers/net/wireguard/send.c:94 wg_receive_handshake_packet+0x1b8/0x810 drivers/net/wireguard/receive.c:154 wg_packet_handshake_receive_worker+0xd8/0x314 drivers/net/wireguard/receive.c:213 process_one_work+0x7cc/0x18d4 kernel/workqueue.c:3257 process_scheduled_works kernel/workqueue.c:3340 [inline] worker_thread+0x734/0xb84 kernel/workqueue.c:3421 kthread+0x348/0x5fc kernel/kthread.c:463 ret_from_fork+0x10/0x20 arch/arm64/kernel/entry.S:860 BUG: workqueue lockup - pool cpus=0 node=0 flags=0x0 nice=0 stuck for 74s! Showing busy workqueues and worker pools: workqueue events: flags=0x100 pwq 2: cpus=0 node=0 flags=0x0 nice=0 active=7 refcnt=8 in-flight: 3791:nsim_fib_event_work nsim_fib_event_work pending: psi_avgs_work, vmstat_shepherd, psi_avgs_work, drm_fb_helper_damage_work, psi_avgs_work pwq 6: cpus=1 node=0 flags=0x0 nice=0 active=2 refcnt=3 pending: 2*nsim_dev_hwstats_traffic_work workqueue events_unbound: flags=0x2 pwq 8: cpus=0-1 flags=0x6 nice=0 active=1 refcnt=2 pending: flush_memcg_stats_dwork workqueue events_unbound: flags=0x2 pwq 8: cpus=0-1 flags=0x6 nice=0 active=3 refcnt=4 in-flight: 4801:linkwatch_event pending: 2*nsim_dev_trap_report_work pwq 8: cpus=0-1 flags=0x6 nice=0 active=1 refcnt=2 pending: crng_reseed workqueue events_power_efficient: flags=0x182 pwq 8: cpus=0-1 flags=0x6 nice=0 active=2 refcnt=3 in-flight: 3699:neigh_periodic_work ,12:gc_worker pwq 8: cpus=0-1 flags=0x6 nice=0 active=3 refcnt=4 in-flight: 1545:wg_ratelimiter_gc_entries pending: neigh_periodic_work, fb_flashcursor workqueue mm_percpu_wq: flags=0x8 pwq 2: cpus=0 node=0 flags=0x0 nice=0 active=1 refcnt=2 pending: vmstat_update pwq 6: cpus=1 node=0 flags=0x0 nice=0 active=1 refcnt=2 pending: vmstat_update workqueue ipv6_addrconf: flags=0x6000a pwq 8: cpus=0-1 flags=0x6 nice=0 active=1 refcnt=7 in-flight: 2074(RESCUER):addrconf_verify_work inactive: 2*addrconf_verify_work workqueue wg-kex-wg1: flags=0x6 pwq 8: cpus=0-1 flags=0x6 nice=0 active=1 refcnt=2 in-flight: 26:wg_packet_handshake_send_worker pwq 8: cpus=0-1 flags=0x6 nice=0 active=1 refcnt=2 pending: wg_packet_handshake_send_worker workqueue wg-kex-wg2: flags=0x6 pwq 8: cpus=0-1 flags=0x6 nice=0 active=2 refcnt=3 in-flight: 1554:wg_packet_handshake_send_worker pending: wg_packet_handshake_send_worker workqueue wg-kex-wg0: flags=0x124 pwq 2: cpus=0 node=0 flags=0x0 nice=0 active=1 refcnt=2 pending: wg_packet_handshake_receive_worker workqueue wg-kex-wg0: flags=0x6 pwq 8: cpus=0-1 flags=0x6 nice=0 active=2 refcnt=3 in-flight: 814:wg_packet_handshake_send_worker pending: wg_packet_handshake_send_worker workqueue wg-kex-wg1: flags=0x124 pwq 2: cpus=0 node=0 flags=0x0 nice=0 active=2 refcnt=3 in-flight: 784:wg_packet_handshake_receive_worker pending: wg_packet_handshake_receive_worker pwq 6: cpus=1 node=0 flags=0x0 nice=0 active=2 refcnt=3 in-flight: 3322:wg_packet_handshake_receive_worker wg_packet_handshake_receive_worker workqueue wg-kex-wg1: flags=0x6 pwq 8: cpus=0-1 flags=0x6 nice=0 active=1 refcnt=2 pending: wg_packet_handshake_send_worker pwq 8: cpus=0-1 flags=0x6 nice=0 active=1 refcnt=2 pending: wg_packet_handshake_send_worker workqueue wg-kex-wg2: flags=0x6 pwq 8: cpus=0-1 flags=0x6 nice=0 active=1 refcnt=2 pending: wg_packet_handshake_send_worker pwq 8: cpus=0-1 flags=0x6 nice=0 active=2 refcnt=3 in-flight: 1565:wg_packet_handshake_send_worker ,1006:wg_packet_handshake_send_worker pool 2: cpus=0 node=0 flags=0x0 nice=0 hung=74s workers=6 idle: 10 3321 9 3900 pool 6: cpus=1 node=0 flags=0x0 nice=0 hung=0s workers=7 idle: 3477 785 3774 3475 4981 4871 pool 8: cpus=0-1 flags=0x6 nice=0 hung=2s workers=10 manager: 4556 Showing backtraces of running workers in stalled CPU-bound worker pools: pool 2: task:kworker/0:5 state:R running task stack:0 pid:3791 tgid:3791 ppid:2 task_flags:0x4208060 flags:0x00000010 Workqueue: events nsim_fib_event_work Call trace: __switch_to+0x208/0x4f0 arch/arm64/kernel/process.c:742 (T) context_switch kernel/sched/core.c:5260 [inline] __schedule+0xcfc/0x2fec kernel/sched/core.c:6867 preempt_schedule_irq+0x80/0x1a0 kernel/sched/core.c:7194 raw_irqentry_exit_cond_resched kernel/entry/common.c:173 [inline] raw_irqentry_exit_cond_resched+0x40/0x44 kernel/entry/common.c:165 irqentry_exit+0x18c/0x474 kernel/entry/common.c:216 exit_to_kernel_mode+0x10/0x1c arch/arm64/kernel/entry-common.c:58 __el1_irq arch/arm64/kernel/entry-common.c:500 [inline] el1_interrupt+0x4c/0x60 arch/arm64/kernel/entry-common.c:510 el1h_64_irq_handler+0x18/0x24 arch/arm64/kernel/entry-common.c:515 el1h_64_irq+0x6c/0x70 arch/arm64/kernel/entry.S:592 kasan_get_free_meta+0x0/0x18 mm/kasan/generic.c:473 (P)