BUG: workqueue lockup - pool cpus=0 node=0 flags=0x2 nice=0 stuck for 440s!
Showing busy workqueues and worker pools:
workqueue events: flags=0x0
  pwq 2: cpus=0 node=0 flags=0x2 nice=0 active=7 refcnt=8
    pending: 2*nsim_dev_hwstats_traffic_work, psi_avgs_work, free_obj_work, vmstat_shepherd, 2*psi_avgs_work
workqueue events_unbound: flags=0x2
  pwq 9: cpus=0-1 node=0 flags=0x6 nice=0 active=2 refcnt=3
    in-flight: 3227:nsim_dev_trap_report_work
    pending: flush_memcg_stats_dwork
  pwq 10: cpus=0-1 node=0 flags=0x6 nice=0 active=7 refcnt=8
    pending: toggle_allocation_gate, crng_reseed, idle_cull_fn, nsim_dev_trap_report_work, 3*macvlan_process_broadcast
workqueue events_power_efficient: flags=0x80
  pwq 2: cpus=0 node=0 flags=0x2 nice=0 active=4 refcnt=5
    pending: gc_worker, neigh_periodic_work, neigh_managed_work, check_lifetime
  pwq 6: cpus=1 node=0 flags=0x2 nice=0 active=3 refcnt=4
    in-flight: 4178:wg_ratelimiter_gc_entries ,4202:neigh_managed_work
    pending: neigh_periodic_work
workqueue mm_percpu_wq: flags=0x8
  pwq 6: cpus=1 node=0 flags=0x2 nice=0 active=1 refcnt=2
    pending: vmstat_update
workqueue writeback: flags=0x4a
  pwq 9: cpus=0-1 node=0 flags=0x6 nice=0 active=1 refcnt=2
    in-flight: 28:wb_workfn
workqueue wg-kex-wg0: flags=0x24
  pwq 2: cpus=0 node=0 flags=0x2 nice=0 active=1 refcnt=2
    pending: wg_packet_handshake_receive_worker
  pwq 6: cpus=1 node=0 flags=0x2 nice=0 active=2 refcnt=3
    in-flight: 3868:wg_packet_handshake_receive_worker wg_packet_handshake_receive_worker
workqueue wg-kex-wg0: flags=0x6
  pwq 9: cpus=0-1 node=0 flags=0x6 nice=0 active=3 refcnt=4
    in-flight: 3563:wg_queued_expired_zero_key_material ,4173:wg_packet_handshake_send_worker wg_packet_handshake_send_worker
  pwq 10: cpus=0-1 node=0 flags=0x6 nice=0 active=1 refcnt=2
    in-flight: 3286:wg_packet_handshake_send_worker
workqueue wg-kex-wg0: flags=0x24
  pwq 2: cpus=0 node=0 flags=0x2 nice=0 active=1 refcnt=2
    pending: wg_packet_handshake_receive_worker
  pwq 6: cpus=1 node=0 flags=0x2 nice=0 active=2 refcnt=3
    in-flight: 4183:wg_packet_handshake_receive_worker wg_packet_handshake_receive_worker
workqueue wg-kex-wg0: flags=0x6
  pwq 10: cpus=0-1 node=0 flags=0x6 nice=0 active=3 refcnt=4
    in-flight: 66:wg_packet_handshake_send_worker
    pending: 2*wg_packet_handshake_send_worker
workqueue wg-kex-wg1: flags=0x24
  pwq 2: cpus=0 node=0 flags=0x2 nice=0 active=2 refcnt=3
    in-flight: 3831:wg_packet_handshake_receive_worker
    pending: wg_packet_handshake_receive_worker
  pwq 6: cpus=1 node=0 flags=0x2 nice=0 active=2 refcnt=3
    in-flight: 4162:wg_packet_handshake_receive_worker wg_packet_handshake_receive_worker
workqueue wg-kex-wg1: flags=0x6
  pwq 10: cpus=0-1 node=0 flags=0x6 nice=0 active=2 refcnt=3
    pending: 2*wg_packet_handshake_send_worker
workqueue wg-kex-wg1: flags=0x24
  pwq 2: cpus=0 node=0 flags=0x2 nice=0 active=1 refcnt=2
    pending: wg_packet_handshake_receive_worker
  pwq 6: cpus=1 node=0 flags=0x2 nice=0 active=2 refcnt=3
    in-flight: 4127:wg_packet_handshake_receive_worker wg_packet_handshake_receive_worker
workqueue wg-kex-wg1: flags=0x6
  pwq 10: cpus=0-1 node=0 flags=0x6 nice=0 active=3 refcnt=4
    in-flight: 62:wg_packet_handshake_send_worker
    pending: 2*wg_packet_handshake_send_worker
workqueue wg-kex-wg2: flags=0x24
  pwq 2: cpus=0 node=0 flags=0x2 nice=0 active=2 refcnt=3
    in-flight: 9:wg_packet_handshake_receive_worker
    pending: wg_packet_handshake_receive_worker
  pwq 6: cpus=1 node=0 flags=0x2 nice=0 active=2 refcnt=3
    in-flight: 24:wg_packet_handshake_receive_worker wg_packet_handshake_receive_worker
workqueue wg-kex-wg2: flags=0x6
  pwq 10: cpus=0-1 node=0 flags=0x6 nice=0 active=2 refcnt=3
    pending: 2*wg_packet_handshake_send_worker
workqueue wg-kex-wg2: flags=0x24
  pwq 2: cpus=0 node=0 flags=0x2 nice=0 active=2 refcnt=3
    in-flight: 4010:wg_packet_handshake_receive_worker
    pending: wg_packet_handshake_receive_worker
  pwq 6: cpus=1 node=0 flags=0x2 nice=0 active=2 refcnt=3
    in-flight: 879:wg_packet_handshake_receive_worker wg_packet_handshake_receive_worker
workqueue wg-kex-wg2: flags=0x6
  pwq 9: cpus=0-1 node=0 flags=0x6 nice=0 active=1 refcnt=2
    in-flight: 144:wg_packet_handshake_send_worker
  pwq 10: cpus=0-1 node=0 flags=0x6 nice=0 active=1 refcnt=2
    pending: wg_packet_handshake_send_worker
pool 2: cpus=0 node=0 flags=0x2 nice=0 hung=440s workers=4 manager: 3039
pool 6: cpus=1 node=0 flags=0x2 nice=0 hung=3s workers=14 manager: 3925 idle: 4210 4126 3924 4190 3814
pool 9: cpus=0-1 node=0 flags=0x6 nice=0 hung=1s workers=6 manager: 3356
pool 10: cpus=0-1 node=0 flags=0x6 nice=0 hung=31s workers=4 manager: 50
Showing backtraces of running workers in stalled CPU-bound worker pools:
pool 2:
task:kworker/0:1     state:R  running task     stack:0     pid:3831  tgid:3831  ppid:2      task_flags:0x4208060 flags:0x00000000
Workqueue: wg-kex-wg1 wg_packet_handshake_receive_worker
Call Trace:
[<ffffffff863ce5c0>] context_switch kernel/sched/core.c:5357 [inline]
[<ffffffff863ce5c0>] __schedule+0x1042/0x4106 kernel/sched/core.c:6961
[<ffffffff863d2158>] preempt_schedule_irq+0x7a/0x1d2 kernel/sched/core.c:7288
[<ffffffff863c513a>] raw_irqentry_exit_cond_resched kernel/entry/common.c:154 [inline]
[<ffffffff863c513a>] raw_irqentry_exit_cond_resched kernel/entry/common.c:146 [inline]
[<ffffffff863c513a>] irqentry_exit+0x132/0x18c kernel/entry/common.c:197
[<ffffffff863c3e9a>] do_irq+0x34/0x48 arch/riscv/kernel/traps.c:402
[<ffffffff863ed574>] handle_exception+0x124/0x152 arch/riscv/kernel/entry.S:186
[<ffffffff80a7ae1c>] arch_local_irq_restore arch/riscv/include/asm/irqflags.h:51 [inline]
[<ffffffff80a7ae1c>] slab_update_freelist mm/slub.c:773 [inline]
[<ffffffff80a7ae1c>] __slab_free+0xb4/0x32a mm/slub.c:4532
[<ffffffff80a7ffe8>] do_slab_free mm/slub.c:4632 [inline]
[<ffffffff80a7ffe8>] ___cache_free+0x1a2/0x1dc mm/slub.c:4738
[<ffffffff80b21d9e>] qlink_free mm/kasan/quarantine.c:163 [inline]
[<ffffffff80b21d9e>] qlist_free_all+0x76/0x168 mm/kasan/quarantine.c:179
[<ffffffff80b2244c>] kasan_quarantine_reduce+0x158/0x1ba mm/kasan/quarantine.c:286
[<ffffffff80b1d87e>] __kasan_kmalloc+0x80/0xa6 mm/kasan/common.c:396
[<ffffffff80a78aea>] kasan_kmalloc include/linux/kasan.h:260 [inline]
[<ffffffff80a78aea>] __kmalloc_cache_noprof+0x15c/0x3d6 mm/slub.c:4396
[<ffffffff83294d70>] kmalloc_noprof include/linux/slab.h:905 [inline]
[<ffffffff83294d70>] kzalloc_noprof include/linux/slab.h:1039 [inline]
[<ffffffff83294d70>] keypair_create drivers/net/wireguard/noise.c:100 [inline]
[<ffffffff83294d70>] wg_noise_handshake_begin_session+0xfe/0xdaa drivers/net/wireguard/noise.c:827
[<ffffffff8329eafa>] wg_packet_send_handshake_response+0x218/0x2cc drivers/net/wireguard/send.c:96
[<ffffffff832a2352>] wg_receive_handshake_packet+0x244/0xaa0 drivers/net/wireguard/receive.c:154
[<ffffffff832a43be>] wg_packet_handshake_receive_worker+0x170/0x384 drivers/net/wireguard/receive.c:213
[<ffffffff801c7a44>] process_one_work+0x96a/0x1f32 kernel/workqueue.c:3236
[<ffffffff801cab54>] process_scheduled_works kernel/workqueue.c:3319 [inline]
[<ffffffff801cab54>] worker_thread+0x5ce/0xde8 kernel/workqueue.c:3400
[<ffffffff801e843c>] kthread+0x39c/0x7d4 kernel/kthread.c:463
[<ffffffff800699c4>] ret_from_fork_kernel+0x2a/0xbb4 arch/riscv/kernel/process.c:214
[<ffffffff863ed6b6>] ret_from_fork_kernel_asm+0x16/0x18 arch/riscv/kernel/entry.S:327
pool 2:
task:kworker/0:0     state:R  running task     stack:0     pid:9     tgid:9     ppid:2      task_flags:0x4208060 flags:0x00000000
Workqueue: wg-kex-wg2 wg_packet_handshake_receive_worker
Call Trace:
[<ffffffff863ce5c0>] context_switch kernel/sched/core.c:5357 [inline]
[<ffffffff863ce5c0>] __schedule+0x1042/0x4106 kernel/sched/core.c:6961
[<ffffffff863d2158>] preempt_schedule_irq+0x7a/0x1d2 kernel/sched/core.c:7288
[<ffffffff863c513a>] raw_irqentry_exit_cond_resched kernel/entry/common.c:154 [inline]
[<ffffffff863c513a>] raw_irqentry_exit_cond_resched kernel/entry/common.c:146 [inline]
[<ffffffff863c513a>] irqentry_exit+0x132/0x18c kernel/entry/common.c:197
[<ffffffff863c3e9a>] do_irq+0x34/0x48 arch/riscv/kernel/traps.c:402
[<ffffffff863ed574>] handle_exception+0x124/0x152 arch/riscv/kernel/entry.S:186
[<ffffffff80a7a9b0>] arch_local_irq_restore arch/riscv/include/asm/irqflags.h:51 [inline]
[<ffffffff80a7a9b0>] put_cpu_partial+0x170/0x296 mm/slub.c:3290
[<ffffffff80a7af4a>] __slab_free+0x1e2/0x32a mm/slub.c:4550
[<ffffffff80a7ffe8>] do_slab_free mm/slub.c:4632 [inline]
[<ffffffff80a7ffe8>] ___cache_free+0x1a2/0x1dc mm/slub.c:4738
[<ffffffff80b21d9e>] qlink_free mm/kasan/quarantine.c:163 [inline]
[<ffffffff80b21d9e>] qlist_free_all+0x76/0x168 mm/kasan/quarantine.c:179
[<ffffffff80b2244c>] kasan_quarantine_reduce+0x158/0x1ba mm/kasan/quarantine.c:286
[<ffffffff80b1d87e>] __kasan_kmalloc+0x80/0xa6 mm/kasan/common.c:396
[<ffffffff80a78aea>] kasan_kmalloc include/linux/kasan.h:260 [inline]
[<ffffffff80a78aea>] __kmalloc_cache_noprof+0x15c/0x3d6 mm/slub.c:4396
[<ffffffff83294d70>] kmalloc_noprof include/linux/slab.h:905 [inline]
[<ffffffff83294d70>] kzalloc_noprof include/linux/slab.h:1039 [inline]
[<ffffffff83294d70>] keypair_create drivers/net/wireguard/noise.c:100 [inline]
[<ffffffff83294d70>] wg_noise_handshake_begin_session+0xfe/0xdaa drivers/net/wireguard/noise.c:827
[<ffffffff8329eafa>] wg_packet_send_handshake_response+0x218/0x2cc drivers/net/wireguard/send.c:96
[<ffffffff832a2352>] wg_receive_handshake_packet+0x244/0xaa0 drivers/net/wireguard/receive.c:154
[<ffffffff832a43be>] wg_packet_handshake_receive_worker+0x170/0x384 drivers/net/wireguard/receive.c:213
[<ffffffff801c7a44>] process_one_work+0x96a/0x1f32 kernel/workqueue.c:3236
[<ffffffff801cab54>] process_scheduled_works kernel/workqueue.c:3319 [inline]
[<ffffffff801cab54>] worker_thread+0x5ce/0xde8 kernel/workqueue.c:3400
[<ffffffff801e843c>] kthread+0x39c/0x7d4 kernel/kthread.c:463
[<ffffffff800699c4>] ret_from_fork_kernel+0x2a/0xbb4 arch/riscv/kernel/process.c:214
[<ffffffff863ed6b6>] ret_from_fork_kernel_asm+0x16/0x18 arch/riscv/kernel/entry.S:327