BUG: workqueue lockup - pool cpus=1 node=0 flags=0x0 nice=0 stuck for 59s! Showing busy workqueues and worker pools: workqueue events: flags=0x100 pwq 2: cpus=0 node=0 flags=0x2 nice=0 active=4 refcnt=5 pending: 2*psi_avgs_work, vmstat_shepherd, nsim_dev_hwstats_traffic_work pwq 6: cpus=1 node=0 flags=0x0 nice=0 active=5 refcnt=6 in-flight: 5224:nsim_fib_event_work for 59s pending: nsim_dev_hwstats_traffic_work, drm_fb_helper_damage_work, nsim_fib_event_work, psi_avgs_work workqueue events_unbound: flags=0x2 pwq 8: cpus=0-1 flags=0x6 nice=0 active=2 refcnt=3 pending: nsim_dev_trap_report_work, flush_memcg_stats_dwork pwq 8: cpus=0-1 flags=0x6 nice=0 active=2 refcnt=3 in-flight: 766:linkwatch_event for 134s pending: nsim_dev_trap_report_work workqueue events_power_efficient: flags=0x182 pwq 8: cpus=0-1 flags=0x6 nice=0 active=2 refcnt=3 in-flight: 3352:wg_ratelimiter_gc_entries for 40s pending: fb_flashcursor pwq 8: cpus=0-1 flags=0x6 nice=0 active=2 refcnt=3 pending: neigh_periodic_work, gc_worker workqueue mm_percpu_wq: flags=0x108 pwq 2: cpus=0 node=0 flags=0x2 nice=0 active=1 refcnt=4 MAYDAY pending: vmstat_update pwq 6: cpus=1 node=0 flags=0x0 nice=0 active=1 refcnt=2 pending: vmstat_update workqueue kblockd: flags=0x118 pwq 3: cpus=0 node=0 flags=0x0 nice=-20 active=1 refcnt=2 pending: blk_mq_timeout_work workqueue ipv6_addrconf: flags=0x6000a pwq 8: cpus=0-1 flags=0x6 nice=0 active=1 refcnt=6 in-flight: 4860:addrconf_verify_work for 73s inactive: 2*addrconf_verify_work workqueue wg-kex-wg0: flags=0x124 pwq 2: cpus=0 node=0 flags=0x2 nice=0 active=2 refcnt=3 in-flight: 9:wg_packet_handshake_receive_worker for 199s wg_packet_handshake_receive_worker pwq 6: cpus=1 node=0 flags=0x0 nice=0 active=2 refcnt=3 in-flight: 5215:wg_packet_handshake_receive_worker for 165s wg_packet_handshake_receive_worker workqueue wg-kex-wg0: flags=0x6 pwq 8: cpus=0-1 flags=0x6 nice=0 active=1 refcnt=2 in-flight: 12:wg_packet_handshake_send_worker for 20s workqueue wg-kex-wg0: flags=0x124 pwq 2: cpus=0 node=0 flags=0x2 nice=0 active=2 refcnt=3 in-flight: 4041:wg_packet_handshake_receive_worker for 191s wg_packet_handshake_receive_worker pwq 6: cpus=1 node=0 flags=0x0 nice=0 active=2 refcnt=3 in-flight: 5067:wg_packet_handshake_receive_worker for 235s wg_packet_handshake_receive_worker workqueue wg-kex-wg0: flags=0x6 pwq 8: cpus=0-1 flags=0x6 nice=0 active=1 refcnt=2 in-flight: 3443:wg_packet_handshake_send_worker for 4s workqueue wg-kex-wg1: flags=0x124 pwq 2: cpus=0 node=0 flags=0x2 nice=0 active=1 refcnt=2 in-flight: 5208:wg_packet_handshake_receive_worker for 38s pwq 6: cpus=1 node=0 flags=0x0 nice=0 active=1 refcnt=2 pending: wg_packet_handshake_receive_worker workqueue wg-kex-wg1: flags=0x6 pwq 8: cpus=0-1 flags=0x6 nice=0 active=1 refcnt=2 pending: wg_packet_handshake_send_worker workqueue wg-kex-wg1: flags=0x124 pwq 2: cpus=0 node=0 flags=0x2 nice=0 active=1 refcnt=2 pending: wg_packet_handshake_receive_worker pwq 6: cpus=1 node=0 flags=0x0 nice=0 active=1 refcnt=2 pending: wg_packet_handshake_receive_worker workqueue wg-kex-wg1: flags=0x6 pwq 8: cpus=0-1 flags=0x6 nice=0 active=1 refcnt=2 pending: wg_packet_handshake_send_worker pwq 8: cpus=0-1 flags=0x6 nice=0 active=1 refcnt=2 pending: wg_packet_handshake_send_worker workqueue wg-kex-wg2: flags=0x124 pwq 2: cpus=0 node=0 flags=0x2 nice=0 active=1 refcnt=2 in-flight: 53:wg_packet_handshake_receive_worker for 36s pwq 6: cpus=1 node=0 flags=0x0 nice=0 active=1 refcnt=2 pending: wg_packet_handshake_receive_worker workqueue wg-kex-wg2: flags=0x6 pwq 8: cpus=0-1 flags=0x6 nice=0 active=2 refcnt=3 in-flight: 4861:wg_packet_handshake_send_worker for 1s ,3373:wg_packet_handshake_send_worker for 22s workqueue wg-kex-wg2: flags=0x124 pwq 2: cpus=0 node=0 flags=0x2 nice=0 active=1 refcnt=2 in-flight: 3513:wg_packet_handshake_receive_worker for 25s pwq 6: cpus=1 node=0 flags=0x0 nice=0 active=1 refcnt=2 pending: wg_packet_handshake_receive_worker workqueue wg-kex-wg2: flags=0x6 pwq 8: cpus=0-1 flags=0x6 nice=0 active=1 refcnt=2 pending: wg_packet_handshake_send_worker pwq 8: cpus=0-1 flags=0x6 nice=0 active=1 refcnt=2 in-flight: 3371:wg_packet_handshake_send_worker for 15s pool 2: cpus=0 node=0 flags=0x2 nice=0 hung=3s workers=6 manager: 5307 pool 6: cpus=1 node=0 flags=0x0 nice=0 hung=59s workers=9 idle: 3354 778 5308 23 5210 3512 pool 8: cpus=0-1 flags=0x6 nice=0 hung=1s workers=9 manager: 3533 Showing backtraces of busy workers in stalled worker pools: pool 6: task:kworker/1:1 state:R running task stack:0 pid:5067 tgid:5067 ppid:2 task_flags:0x4208060 flags:0x00000010 Workqueue: wg-kex-wg0 wg_packet_handshake_receive_worker Call trace: __switch_to+0x20c/0x520 arch/arm64/kernel/process.c:810 (T) context_switch kernel/sched/core.c:5387 [inline] __schedule+0xd3c/0x3494 kernel/sched/core.c:7188 preempt_schedule_common kernel/sched/core.c:7369 [inline] preempt_schedule+0xf0/0x250 kernel/sched/core.c:7393 __free_frozen_pages+0x8cc/0x1080 mm/page_alloc.c:2973 free_frozen_pages+0x14/0x20 mm/page_alloc.c:2981 __free_slab+0x114/0x27c mm/slub.c:3541 free_slab+0x34/0xf4 mm/slub.c:3575 discard_slab mm/slub.c:3581 [inline] __slab_free+0x2e8/0x3f4 mm/slub.c:5608 ___cache_free+0x78/0x8c mm/slub.c:6318 qlink_free mm/kasan/quarantine.c:163 [inline] qlist_free_all+0x7c/0x138 mm/kasan/quarantine.c:179 kasan_quarantine_reduce+0x148/0x1c0 mm/kasan/quarantine.c:286 __kasan_slab_alloc+0x60/0x9c mm/kasan/common.c:350 kasan_slab_alloc include/linux/kasan.h:253 [inline] slab_post_alloc_hook mm/slub.c:4569 [inline] slab_alloc_node mm/slub.c:4898 [inline] __kmalloc_cache_noprof+0x240/0x6a0 mm/slub.c:5410 kmalloc_noprof include/linux/slab.h:950 [inline] kzalloc_noprof include/linux/slab.h:1188 [inline] keypair_create drivers/net/wireguard/noise.c:100 [inline] wg_noise_handshake_begin_session+0xe0/0xa40 drivers/net/wireguard/noise.c:827 wg_packet_send_handshake_response+0x1bc/0x260 drivers/net/wireguard/send.c:96 wg_receive_handshake_packet+0x19c/0x81c drivers/net/wireguard/receive.c:154 wg_packet_handshake_receive_worker+0xd8/0x320 drivers/net/wireguard/receive.c:213 process_one_work+0x810/0x1b1c kernel/workqueue.c:3302 process_scheduled_works kernel/workqueue.c:3385 [inline] worker_thread+0x42c/0xc90 kernel/workqueue.c:3466 kthread+0x2f0/0x388 kernel/kthread.c:436 ret_from_fork+0x10/0x20 arch/arm64/kernel/entry.S:858 pool 6: task:kworker/1:7 state:R running task stack:0 pid:5224 tgid:5224 ppid:2 task_flags:0x4208060 flags:0x00000010 Workqueue: events nsim_fib_event_work Call trace: __switch_to+0x20c/0x520 arch/arm64/kernel/process.c:810 (T) context_switch kernel/sched/core.c:5387 [inline] __schedule+0xd3c/0x3494 kernel/sched/core.c:7188 preempt_schedule_common kernel/sched/core.c:7369 [inline] preempt_schedule+0xf0/0x250 kernel/sched/core.c:7393 __raw_spin_unlock_irqrestore include/linux/spinlock_api_smp.h:179 [inline] _raw_spin_unlock_irqrestore+0x94/0xb0 kernel/locking/spinlock.c:198 spin_unlock_irqrestore include/linux/spinlock.h:408 [inline] __slab_free+0x228/0x3f4 mm/slub.c:5593 ___cache_free+0x78/0x8c mm/slub.c:6318 qlink_free mm/kasan/quarantine.c:163 [inline] qlist_free_all+0x7c/0x138 mm/kasan/quarantine.c:179 kasan_quarantine_reduce+0x148/0x1c0 mm/kasan/quarantine.c:286 __kasan_slab_alloc+0x60/0x9c mm/kasan/common.c:350 kasan_slab_alloc include/linux/kasan.h:253 [inline] slab_post_alloc_hook mm/slub.c:4569 [inline] slab_alloc_node mm/slub.c:4898 [inline] __kmalloc_cache_noprof+0x240/0x6a0 mm/slub.c:5410 kmalloc_noprof include/linux/slab.h:950 [inline] kzalloc_noprof include/linux/slab.h:1188 [inline] nsim_fib6_rt_nh_add+0x4c/0x220 drivers/net/netdevsim/fib.c:500 nsim_fib6_rt_create drivers/net/netdevsim/fib.c:562 [inline] nsim_fib6_rt_insert drivers/net/netdevsim/fib.c:752 [inline] nsim_fib6_event drivers/net/netdevsim/fib.c:856 [inline] nsim_fib_event drivers/net/netdevsim/fib.c:889 [inline] nsim_fib_event_work+0x4ec/0x3b9c drivers/net/netdevsim/fib.c:1493 process_one_work+0x810/0x1b1c kernel/workqueue.c:3302 process_scheduled_works kernel/workqueue.c:3385 [inline] worker_thread+0x42c/0xc90 kernel/workqueue.c:3466 kthread+0x2f0/0x388 kernel/kthread.c:436 ret_from_fork+0x10/0x20 arch/arm64/kernel/entry.S:858 pool 6: task:kworker/1:6 state:R running task stack:0 pid:5215 tgid:5215 ppid:2 task_flags:0x4208060 flags:0x00000010 Workqueue: wg-kex-wg0 wg_packet_handshake_receive_worker Call trace: __switch_to+0x20c/0x520 arch/arm64/kernel/process.c:810 (T) context_switch kernel/sched/core.c:5387 [inline] __schedule+0xd3c/0x3494 kernel/sched/core.c:7188 preempt_schedule_common kernel/sched/core.c:7369 [inline] preempt_schedule+0xf0/0x250 kernel/sched/core.c:7393 __local_bh_enable_ip+0x370/0x4a0 kernel/softirq.c:457 local_bh_enable include/linux/bottom_half.h:33 [inline] wg_receive_handshake_packet+0x1ec/0x81c drivers/net/wireguard/receive.c:199 wg_packet_handshake_receive_worker+0xd8/0x320 drivers/net/wireguard/receive.c:213 process_one_work+0x810/0x1b1c kernel/workqueue.c:3302 process_scheduled_works kernel/workqueue.c:3385 [inline] worker_thread+0x42c/0xc90 kernel/workqueue.c:3466 kthread+0x2f0/0x388 kernel/kthread.c:436 ret_from_fork+0x10/0x20 arch/arm64/kernel/entry.S:858