watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/1:3:20255]
Modules linked in:
irq event stamp: 104395
hardirqs last  enabled at (104394): [<ffff8000857d18b0>] irqentry_exit_to_kernel_mode_after_preempt include/linux/irq-entry-common.h:515 [inline]
hardirqs last  enabled at (104394): [<ffff8000857d18b0>] arm64_exit_to_kernel_mode+0x50/0xc0 arch/arm64/kernel/entry-common.c:62
hardirqs last disabled at (104395): [<ffff8000857d5578>] __el1_irq arch/arm64/kernel/entry-common.c:493 [inline]
hardirqs last disabled at (104395): [<ffff8000857d5578>] el1_interrupt+0x28/0x60 arch/arm64/kernel/entry-common.c:509
softirqs last  enabled at (99688): [<ffff8000828b6d94>] spin_unlock_bh include/linux/spinlock.h:396 [inline]
softirqs last  enabled at (99688): [<ffff8000828b6d94>] ptr_ring_consume_bh include/linux/ptr_ring.h:377 [inline]
softirqs last  enabled at (99688): [<ffff8000828b6d94>] wg_packet_decrypt_worker+0x248/0x3a0 drivers/net/wireguard/receive.c:499
softirqs last disabled at (99692): [<ffff8000828b6bb4>] spin_lock_bh include/linux/spinlock.h:348 [inline]
softirqs last disabled at (99692): [<ffff8000828b6bb4>] ptr_ring_consume_bh include/linux/ptr_ring.h:375 [inline]
softirqs last disabled at (99692): [<ffff8000828b6bb4>] wg_packet_decrypt_worker+0x68/0x3a0 drivers/net/wireguard/receive.c:499
CPU: 1 UID: 0 PID: 20255 Comm: kworker/1:3 Tainted: G             L      syzkaller #0 PREEMPT 
Tainted: [L]=SOFTLOCKUP
Hardware name: linux,dummy-virt (DT)
Workqueue: wg-crypt-wg0 wg_packet_decrypt_worker
pstate: 20000005 (nzCv daif -PAN -UAO -TCO -DIT -SSBS BTYPE=--)
pc : memory_is_poisoned_n mm/kasan/generic.c:141 [inline]
pc : memory_is_poisoned mm/kasan/generic.c:172 [inline]
pc : check_region_inline mm/kasan/generic.c:191 [inline]
pc : kasan_check_range+0x38/0x1b4 mm/kasan/generic.c:200
lr : __kasan_check_read+0x20/0x2c mm/kasan/shadow.c:31
sp : ffff8000a19478c0
x29: ffff8000a19478c0 x28: ffff80008766bed0 x27: 0000000000000000
x26: 1fffe00003374a09 x25: 1fffe00003374a0b x24: ffff000019ba5058
x23: 0000000000000000 x22: 1ffff00014328f20 x21: 1ffff00014328f3c
x20: 0000000000000000 x19: ffff000019ba5048 x18: 0000000000000000
x17: 0000000000000000 x16: 0000000000000001 x15: 0000000012224261
x14: 0000000000000001 x13: ffff800089e7c8d8 x12: ffff600003374a0a
x11: 1fffe00003374a09 x10: ffff600003374a09 x9 : dfff800000000000
x8 : ffff000019ba5048 x7 : 0000000000000001 x6 : ffff600003374a09
x5 : ffff000019ba5048 x4 : ffff000000000000 x3 : ffff8000857fe9b0
x2 : 0000000000000000 x1 : 0000000000000001 x0 : 0000000000000001
Call trace:
 check_region_inline mm/kasan/generic.c:188 [inline] (P)
 kasan_check_range+0x38/0x1b4 mm/kasan/generic.c:200 (P)
 queued_spin_lock_slowpath+0x610/0xf80 kernel/locking/qspinlock.c:197
 queued_spin_lock include/asm-generic/qspinlock.h:114 [inline]
 do_raw_spin_lock+0x258/0x308 kernel/locking/spinlock_debug.c:116
 __raw_spin_lock_bh include/linux/spinlock_api_smp.h:151 [inline]
 _raw_spin_lock_bh+0x58/0x68 kernel/locking/spinlock.c:182
 spin_lock_bh include/linux/spinlock.h:348 [inline]
 ptr_ring_consume_bh include/linux/ptr_ring.h:375 [inline]
 wg_packet_decrypt_worker+0x68/0x3a0 drivers/net/wireguard/receive.c:499
 process_one_work+0x810/0x1b1c kernel/workqueue.c:3302
 process_scheduled_works kernel/workqueue.c:3385 [inline]
 worker_thread+0x42c/0xc90 kernel/workqueue.c:3466
 kthread+0x2f0/0x388 kernel/kthread.c:436
 ret_from_fork+0x10/0x20 arch/arm64/kernel/entry.S:858
Sending NMI from CPU 1 to CPUs 0:
NMI backtrace for cpu 0
CPU: 0 UID: 0 PID: 19280 Comm: kworker/0:7 Tainted: G             L      syzkaller #0 PREEMPT 
Tainted: [L]=SOFTLOCKUP
Hardware name: linux,dummy-virt (DT)
Workqueue: wg-crypt-wg0 wg_packet_decrypt_worker
pstate: 10000005 (nzcV daif -PAN -UAO -TCO -DIT -SSBS BTYPE=--)
pc : __daif_local_irq_restore arch/arm64/include/asm/irqflags.h:176 [inline]
pc : arch_local_irq_restore arch/arm64/include/asm/irqflags.h:195 [inline]
pc : lock_release+0x39c/0x424 kernel/locking/lockdep.c:5892
lr : lockdep_recursion_finish kernel/locking/lockdep.c:470 [inline]
lr : lock_release+0x37c/0x424 kernel/locking/lockdep.c:5891
sp : ffff8000a1eb7a40
x29: ffff8000a1eb7a50 x28: ffff00003f7678f0 x27: ffff000019ba50c0
x26: ffff600003374a19 x25: 1fffe00003374a18 x24: 0000000000000000
x23: ffff000012db5640 x22: ffff000012db61c0 x21: ffff8000828b6c88
x20: ffff000019ba5060 x19: ffff800087612208 x18: 0000000000000000
x17: 0000000000000000 x16: 0000000000000000 x15: ffff80008766f6e8
x14: 1ffff00010ecdedd x13: ffff800089e7c8d8 x12: ffff7000143d6f41
x11: 1ffff000143d6f40 x10: ffff7000143d6f40 x9 : dfff800000000000
x8 : 0000000000000003 x7 : 0000000000000001 x6 : 00000000000003e0
x5 : 0000000000000000 x4 : 0000000000000000 x3 : 1fffe000025b6ac9
x2 : 0000000000000000 x1 : ffff7fffe29a8000 x0 : 00000000000000c0
Call trace:
 __daif_local_irq_restore arch/arm64/include/asm/irqflags.h:175 [inline] (P)
 arch_local_irq_restore arch/arm64/include/asm/irqflags.h:195 [inline] (P)
 lock_release+0x39c/0x424 kernel/locking/lockdep.c:5892 (P)
 __raw_spin_unlock_bh include/linux/spinlock_api_smp.h:194 [inline]
 _raw_spin_unlock_bh+0x28/0x54 kernel/locking/spinlock.c:214
 spin_unlock_bh include/linux/spinlock.h:396 [inline]
 ptr_ring_consume_bh include/linux/ptr_ring.h:377 [inline]
 wg_packet_decrypt_worker+0x13c/0x3a0 drivers/net/wireguard/receive.c:499
 process_one_work+0x810/0x1b1c kernel/workqueue.c:3302
 process_scheduled_works kernel/workqueue.c:3385 [inline]
 worker_thread+0x42c/0xc90 kernel/workqueue.c:3466
 kthread+0x2f0/0x388 kernel/kthread.c:436
 ret_from_fork+0x10/0x20 arch/arm64/kernel/entry.S:858
BUG: workqueue lockup - pool cpus=0 node=0 flags=0x0 nice=0 stuck for 33s!
Showing busy workqueues and worker pools:
workqueue events: flags=0x100
  pwq 2: cpus=0 node=0 flags=0x0 nice=0 active=4 refcnt=5
    pending: nsim_dev_hwstats_traffic_work, 2*psi_avgs_work, vmstat_shepherd
  pwq 6: cpus=1 node=0 flags=0x0 nice=0 active=10 refcnt=11
    pending: 9*nsim_dev_hwstats_traffic_work, psi_avgs_work
workqueue events_unbound: flags=0x2
  pwq 8: cpus=0-1 flags=0x4 nice=0 active=1 refcnt=2
    in-flight: 16479:call_usermodehelper_exec_work for 42s
workqueue events_unbound: flags=0x2
  pwq 8: cpus=0-1 flags=0x4 nice=0 active=5 refcnt=6
    pending: 5*nsim_dev_trap_report_work
  pwq 8: cpus=0-1 flags=0x4 nice=0 active=5 refcnt=6
    pending: 5*nsim_dev_trap_report_work
workqueue events_power_efficient: flags=0x182
  pwq 8: cpus=0-1 flags=0x4 nice=0 active=2 refcnt=3
    pending: neigh_managed_work, wg_ratelimiter_gc_entries
  pwq 8: cpus=0-1 flags=0x4 nice=0 active=4 refcnt=5
    pending: neigh_periodic_work, gc_worker, neigh_periodic_work, neigh_managed_work
workqueue mm_percpu_wq: flags=0x108
  pwq 2: cpus=0 node=0 flags=0x0 nice=0 active=1 refcnt=2
    pending: vmstat_update
  pwq 6: cpus=1 node=0 flags=0x0 nice=0 active=1 refcnt=2
    pending: vmstat_update
workqueue writeback: flags=0x4a
  pwq 8: cpus=0-1 flags=0x4 nice=0 active=1 refcnt=2
    in-flight: 10400:wb_workfn for 33s
workqueue kblockd: flags=0x118
  pwq 7: cpus=1 node=0 flags=0x0 nice=-20 active=2 refcnt=3
    pending: blk_mq_timeout_work, blk_mq_requeue_work
workqueue ipv6_addrconf: flags=0x6000a
  pwq 8: cpus=0-1 flags=0x4 nice=0 active=1 refcnt=6
    pending: addrconf_verify_work
    inactive: 2*addrconf_verify_work
workqueue wg-kex-wg0: flags=0x6
  pwq 8: cpus=0-1 flags=0x4 nice=0 active=1 refcnt=2
    pending: wg_packet_handshake_send_worker
workqueue wg-crypt-wg0: flags=0x128
  pwq 2: cpus=0 node=0 flags=0x0 nice=0 active=1 refcnt=2
    in-flight: 19280:wg_packet_decrypt_worker for 33s
  pwq 6: cpus=1 node=0 flags=0x0 nice=0 active=2 refcnt=3
    in-flight: 20255:wg_packet_decrypt_worker for 33s
    pending: wg_packet_encrypt_worker
workqueue wg-crypt-wg1: flags=0x128
  pwq 6: cpus=1 node=0 flags=0x0 nice=0 active=1 refcnt=2
    pending: wg_packet_encrypt_worker
workqueue wg-kex-wg2: flags=0x6
  pwq 8: cpus=0-1 flags=0x4 nice=0 active=1 refcnt=2
    pending: wg_packet_handshake_send_worker
  pwq 8: cpus=0-1 flags=0x4 nice=0 active=1 refcnt=2
    pending: wg_packet_handshake_send_worker
workqueue wg-crypt-wg2: flags=0x128
  pwq 6: cpus=1 node=0 flags=0x0 nice=0 active=1 refcnt=2
    pending: wg_packet_encrypt_worker
workqueue wg-kex-wg0: flags=0x6
  pwq 8: cpus=0-1 flags=0x4 nice=0 active=1 refcnt=2
    pending: wg_packet_handshake_send_worker
  pwq 8: cpus=0-1 flags=0x4 nice=0 active=1 refcnt=2
    pending: wg_packet_handshake_send_worker
workqueue wg-kex-wg1: flags=0x6
  pwq 8: cpus=0-1 flags=0x4 nice=0 active=1 refcnt=2
    pending: wg_packet_handshake_send_worker
  pwq 8: cpus=0-1 flags=0x4 nice=0 active=1 refcnt=2
    pending: wg_packet_handshake_send_worker
workqueue wg-kex-wg2: flags=0x6
  pwq 8: cpus=0-1 flags=0x4 nice=0 active=2 refcnt=3
    pending: 2*wg_packet_handshake_send_worker
workqueue wg-kex-wg0: flags=0x6
  pwq 8: cpus=0-1 flags=0x4 nice=0 active=3 refcnt=4
    in-flight: 15424:wg_packet_handshake_send_worker for 33s ,20263:wg_packet_handshake_send_worker for 33s
    pending: wg_packet_handshake_send_worker
workqueue wg-kex-wg1: flags=0x6
  pwq 8: cpus=0-1 flags=0x4 nice=0 active=1 refcnt=2
    pending: wg_packet_handshake_send_worker
  pwq 8: cpus=0-1 flags=0x4 nice=0 active=1 refcnt=2
    pending: wg_packet_handshake_send_worker
workqueue wg-kex-wg2: flags=0x6
  pwq 8: cpus=0-1 flags=0x4 nice=0 active=1 refcnt=2
    pending: wg_packet_handshake_send_worker
  pwq 8: cpus=0-1 flags=0x4 nice=0 active=2 refcnt=3
    in-flight: 17220:wg_packet_handshake_send_worker for 33s
    pending: wg_packet_handshake_send_worker
workqueue wg-kex-wg0: flags=0x6
  pwq 8: cpus=0-1 flags=0x4 nice=0 active=1 refcnt=2
    pending: wg_packet_handshake_send_worker
workqueue wg-crypt-wg0: flags=0x128
  pwq 2: cpus=0 node=0 flags=0x0 nice=0 active=1 refcnt=2
    pending: wg_packet_decrypt_worker
  pwq 6: cpus=1 node=0 flags=0x0 nice=0 active=1 refcnt=2
    pending: wg_packet_encrypt_worker
workqueue wg-kex-wg1: flags=0x6
  pwq 8: cpus=0-1 flags=0x4 nice=0 active=1 refcnt=2
    pending: wg_packet_handshake_send_worker
workqueue wg-crypt-wg1: flags=0x128
  pwq 2: cpus=0 node=0 flags=0x0 nice=0 active=1 refcnt=2
    pending: wg_packet_encrypt_worker
  pwq 6: cpus=1 node=0 flags=0x0 nice=0 active=1 refcnt=2
    pending: wg_packet_encrypt_worker
workqueue wg-kex-wg2: flags=0x6
  pwq 8: cpus=0-1 flags=0x4 nice=0 active=1 refcnt=2
    pending: wg_packet_handshake_send_worker
workqueue wg-crypt-wg2: flags=0x128
  pwq 6: cpus=1 node=0 flags=0x0 nice=0 active=1 refcnt=2
    pending: wg_packet_encrypt_worker
workqueue wg-crypt-wg0: flags=0x128
  pwq 2: cpus=0 node=0 flags=0x0 nice=0 active=1 refcnt=2
    pending: wg_packet_encrypt_worker
  pwq 6: cpus=1 node=0 flags=0x0 nice=0 active=2 refcnt=3
    pending: wg_packet_decrypt_worker, wg_packet_encrypt_worker
workqueue wg-crypt-wg1: flags=0x128
  pwq 2: cpus=0 node=0 flags=0x0 nice=0 active=1 refcnt=2
    pending: wg_packet_encrypt_worker
  pwq 6: cpus=1 node=0 flags=0x0 nice=0 active=1 refcnt=2
    pending: wg_packet_encrypt_worker
workqueue wg-crypt-wg2: flags=0x128
  pwq 2: cpus=0 node=0 flags=0x0 nice=0 active=1 refcnt=2
    pending: wg_packet_encrypt_worker
  pwq 6: cpus=1 node=0 flags=0x0 nice=0 active=1 refcnt=2
    pending: wg_packet_encrypt_worker
workqueue krds_cp_wq#1/0: flags=0x20002
  pwq 8: cpus=0-1 flags=0x4 nice=0 active=1 refcnt=4
    pending: rds_connect_worker
workqueue krds_cp_wq#2/0: flags=0x20002
  pwq 8: cpus=0-1 flags=0x4 nice=0 active=1 refcnt=4
    pending: rds_connect_worker
pool 2: cpus=0 node=0 flags=0x0 nice=0 hung=33s workers=7 idle: 20317 20625 17368 18803 20184 16642
pool 6: cpus=1 node=0 flags=0x0 nice=0 hung=33s workers=6 idle: 20316 15747 16513 18710 19225
pool 8: cpus=0-1 flags=0x4 nice=0 hung=0s workers=13 idle: 15591 8052 14588 7215 11940 15432 19121
Showing backtraces of busy workers in stalled worker pools:
pool 2:
task:kworker/0:7     state:R  running task     stack:0     pid:19280 tgid:19280 ppid:2      task_flags:0x4208060 flags:0x00000012
Workqueue: wg-crypt-wg0 wg_packet_decrypt_worker
Call trace:
 show_stack+0x18/0x24 arch/arm64/kernel/stacktrace.c:499 (C)
 sched_show_task kernel/sched/core.c:8139 [inline]
 sched_show_task+0x454/0x5e0 kernel/sched/core.c:8114
 show_cpu_pool_busy_workers kernel/workqueue.c:7674 [inline]
 show_cpu_pools_busy_workers kernel/workqueue.c:7693 [inline]
 wq_watchdog_timer_fn+0x840/0x870 kernel/workqueue.c:7809
 call_timer_fn+0x1b8/0xaa0 kernel/time/timer.c:1748
 expire_timers kernel/time/timer.c:1799 [inline]
 __run_timers+0x8bc/0xb30 kernel/time/timer.c:2374
 __run_timer_base kernel/time/timer.c:2386 [inline]
 __run_timer_base kernel/time/timer.c:2378 [inline]
 run_timer_base+0x110/0x180 kernel/time/timer.c:2395
 run_timer_softirq+0x24/0x50 kernel/time/timer.c:2406
 handle_softirqs+0x2dc/0x1050 kernel/softirq.c:622
 __do_softirq+0x14/0x20 kernel/softirq.c:656
 ____do_softirq+0x10/0x20 arch/arm64/kernel/irq.c:68
 call_on_irq_stack+0x30/0x48 arch/arm64/kernel/entry.S:889
 do_softirq_own_stack+0x1c/0x40 arch/arm64/kernel/irq.c:73
 do_softirq kernel/softirq.c:523 [inline]
 do_softirq+0x12c/0x15c kernel/softirq.c:510
 __local_bh_enable_ip+0x408/0x4a0 kernel/softirq.c:450
 __raw_spin_unlock_bh include/linux/spinlock_api_smp.h:196 [inline]
 _raw_spin_unlock_bh+0x44/0x54 kernel/locking/spinlock.c:214
 spin_unlock_bh include/linux/spinlock.h:396 [inline]
 ptr_ring_consume_bh include/linux/ptr_ring.h:377 [inline]
 wg_packet_decrypt_worker+0x13c/0x3a0 drivers/net/wireguard/receive.c:499
 process_one_work+0x810/0x1b1c kernel/workqueue.c:3302
 process_scheduled_works kernel/workqueue.c:3385 [inline]
 worker_thread+0x42c/0xc90 kernel/workqueue.c:3466
 kthread+0x2f0/0x388 kernel/kthread.c:436
 ret_from_fork+0x10/0x20 arch/arm64/kernel/entry.S:858