BUG: workqueue lockup - pool cpus=0 node=0 flags=0x0 nice=0 stuck for 41s! Showing busy workqueues and worker pools: workqueue events: flags=0x100 pwq 2: cpus=0 node=0 flags=0x0 nice=0 active=14 refcnt=15 pending: 6*nsim_dev_hwstats_traffic_work, psi_avgs_work, key_garbage_collector, vmstat_shepherd, psi_avgs_work, 3*ovs_dp_masks_rebalance, free_obj_work pwq 6: cpus=1 node=0 flags=0x0 nice=0 active=3 refcnt=4 pending: 3*nsim_dev_hwstats_traffic_work workqueue events_unbound: flags=0x2 pwq 8: cpus=0-1 flags=0x4 nice=0 active=9 refcnt=10 in-flight: 2688:cfg80211_wiphy_work for 80s cfg80211_wiphy_work ,932:cfg80211_wiphy_work for 72s cfg80211_wiphy_work ,3712:linkwatch_event for 93s linkwatch_event ,6114:cfg80211_wiphy_work for 84s cfg80211_wiphy_work ,6115:cfg80211_wiphy_work for 80s cfg80211_wiphy_work pwq 8: cpus=0-1 flags=0x4 nice=0 active=22 refcnt=23 in-flight: 12:nsim_dev_trap_report_work for 4s ,5535:cfg80211_wiphy_work for 75s cfg80211_wiphy_work ,58:cfg80211_wiphy_work for 76s cfg80211_wiphy_work ,13:nsim_dev_trap_report_work for 14s ,54:cfg80211_wiphy_work for 81s cfg80211_wiphy_work ,40:nsim_dev_trap_report_work for 4s ,6116:cfg80211_wiphy_work for 81s cfg80211_wiphy_work ,5281:nsim_dev_trap_report_work for 3s pending: crng_reseed, nsim_dev_trap_report_work, toggle_allocation_gate, 5*nsim_dev_trap_report_work, flush_memcg_stats_dwork workqueue events_power_efficient: flags=0x82 pwq 8: cpus=0-1 flags=0x4 nice=0 active=5 refcnt=6 in-flight: 8996:neigh_periodic_work for 70s pending: fb_flashcursor, gc_worker, hash_mac4_gc, wg_ratelimiter_gc_entries workqueue events_dfl_long: flags=0x2 pwq 8: cpus=0-1 flags=0x4 nice=0 active=9 refcnt=10 in-flight: 10407:defense_work_handler for 4s pending: 8*defense_work_handler workqueue netns: flags=0x6000a pwq 8: cpus=0-1 flags=0x4 nice=0 active=1 refcnt=4 in-flight: 5537:cleanup_net for 97s workqueue mm_percpu_wq: flags=0x108 pwq 2: cpus=0 node=0 flags=0x0 nice=0 active=1 refcnt=2 pending: vmstat_update workqueue writeback: flags=0x4a pwq 8: cpus=0-1 flags=0x4 nice=0 active=1 refcnt=2 in-flight: 1207:wb_workfn for 73s workqueue kblockd: flags=0x118 pwq 7: cpus=1 node=0 flags=0x0 nice=-20 active=1 refcnt=2 pending: blk_mq_timeout_work workqueue ipv6_addrconf: flags=0x6000a pwq 8: cpus=0-1 flags=0x4 nice=0 active=1 refcnt=7 MAYDAY in-flight: 2823(RESCUER):addrconf_verify_work for 35s inactive: addrconf_verify_work workqueue bat_events: flags=0x6000a pwq 8: cpus=0-1 flags=0x4 nice=0 active=1 refcnt=56 MAYDAY in-flight: 2855(RESCUER):batadv_tt_purge for 10s inactive: batadv_tt_purge, batadv_dat_purge, batadv_bla_periodic_work, batadv_dat_purge, batadv_bla_periodic_work, 3*batadv_tt_purge, batadv_mcast_mla_update, 2*batadv_bla_periodic_work, batadv_dat_purge, batadv_bla_periodic_work, 2*batadv_dat_purge, batadv_bla_periodic_work, batadv_dat_purge, 6*batadv_mcast_mla_update, batadv_iv_send_outstanding_bat_ogm_packet, batadv_purge_orig, batadv_iv_send_outstanding_bat_ogm_packet, 3*batadv_purge_orig, 6*batadv_iv_send_outstanding_bat_ogm_packet, 2*batadv_purge_orig, batadv_iv_send_outstanding_bat_ogm_packet, batadv_purge_orig, 7*batadv_iv_send_outstanding_bat_ogm_packet, batadv_tt_purge, batadv_bla_periodic_work, batadv_dat_purge, batadv_tt_purge workqueue hci6: flags=0x20012 pwq 9: cpus=0-1 node=0 flags=0x4 nice=-20 active=1 refcnt=4 in-flight: 4722:hci_cmd_sync_work for 66s workqueue wg-kex-wg0: flags=0x6 pwq 8: cpus=0-1 flags=0x4 nice=0 active=1 refcnt=2 pending: wg_packet_handshake_send_worker workqueue wg-crypt-wg0: flags=0x128 pwq 2: cpus=0 node=0 flags=0x0 nice=0 active=2 refcnt=3 pending: wg_packet_encrypt_worker, wg_packet_tx_worker workqueue wg-kex-wg1: flags=0x6 pwq 8: cpus=0-1 flags=0x4 nice=0 active=2 refcnt=3 pending: 2*wg_packet_handshake_send_worker workqueue wg-crypt-wg1: flags=0x128 pwq 2: cpus=0 node=0 flags=0x0 nice=0 active=2 refcnt=3 pending: wg_packet_encrypt_worker, wg_packet_tx_worker workqueue wg-kex-wg2: flags=0x6 pwq 8: cpus=0-1 flags=0x4 nice=0 active=1 refcnt=2 in-flight: 6113:wg_packet_handshake_send_worker for 2s workqueue wg-crypt-wg2: flags=0x128 pwq 2: cpus=0 node=0 flags=0x0 nice=0 active=1 refcnt=2 pending: wg_packet_encrypt_worker workqueue hci1: flags=0x20012 pwq 9: cpus=0-1 node=0 flags=0x4 nice=-20 active=1 refcnt=4 in-flight: 4724:hci_cmd_sync_work for 40s workqueue wg-kex-wg0: flags=0x124 pwq 2: cpus=0 node=0 flags=0x0 nice=0 active=1 refcnt=2 pending: wg_packet_handshake_receive_worker pwq 6: cpus=1 node=0 flags=0x0 nice=0 active=1 refcnt=2 in-flight: 6551:wg_packet_handshake_receive_worker for 3s workqueue wg-crypt-wg0: flags=0x128 pwq 2: cpus=0 node=0 flags=0x0 nice=0 active=4 refcnt=5 in-flight: 6117:wg_packet_decrypt_worker for 62s pending: wg_packet_decrypt_worker, wg_packet_encrypt_worker, wg_packet_tx_worker workqueue wg-kex-wg0: flags=0x124 pwq 2: cpus=0 node=0 flags=0x0 nice=0 active=1 refcnt=2 pending: wg_packet_handshake_receive_worker pwq 6: cpus=1 node=0 flags=0x0 nice=0 active=1 refcnt=2 in-flight: 4811:wg_packet_handshake_receive_worker for 4s workqueue wg-crypt-wg0: flags=0x128 pwq 2: cpus=0 node=0 flags=0x0 nice=0 active=3 refcnt=4 pending: wg_packet_decrypt_worker, wg_packet_encrypt_worker, wg_packet_tx_worker workqueue wg-crypt-wg1: flags=0x128 pwq 2: cpus=0 node=0 flags=0x0 nice=0 active=4 refcnt=5 in-flight: 4819:wg_packet_encrypt_worker for 41s pending: wg_packet_tx_worker, wg_packet_decrypt_worker, wg_packet_encrypt_worker workqueue wg-crypt-wg2: flags=0x128 pwq 2: cpus=0 node=0 flags=0x0 nice=0 active=2 refcnt=3 pending: wg_packet_encrypt_worker, wg_packet_tx_worker workqueue wg-crypt-wg1: flags=0x128 pwq 2: cpus=0 node=0 flags=0x0 nice=0 active=4 refcnt=5 in-flight: 9:wg_packet_decrypt_worker for 62s pending: wg_packet_decrypt_worker, wg_packet_encrypt_worker, wg_packet_tx_worker workqueue wg-kex-wg2: flags=0x6 pwq 8: cpus=0-1 flags=0x4 nice=0 active=1 refcnt=2 pending: wg_packet_handshake_send_worker workqueue wg-crypt-wg2: flags=0x128 pwq 2: cpus=0 node=0 flags=0x0 nice=0 active=3 refcnt=4 pending: wg_packet_tx_worker, wg_packet_encrypt_worker, wg_packet_decrypt_worker workqueue krds_cp_wq#0/0: flags=0x20002 pwq 8: cpus=0-1 flags=0x4 nice=0 active=1 refcnt=4 pending: rds_connect_worker workqueue krds_cp_wq#1/0: flags=0x20002 pwq 8: cpus=0-1 flags=0x4 nice=0 active=1 refcnt=4 pending: rds_send_worker workqueue krds_cp_wq#1/4: flags=0x20002 pwq 8: cpus=0-1 flags=0x4 nice=0 active=1 refcnt=4 pending: rds_send_worker workqueue krds_cp_wq#1/5: flags=0x20002 pwq 8: cpus=0-1 flags=0x4 nice=0 active=1 refcnt=4 pending: rds_send_worker workqueue krds_cp_wq#1/7: flags=0x20002 pwq 8: cpus=0-1 flags=0x4 nice=0 active=1 refcnt=4 pending: rds_send_worker workqueue krds_cp_wq#2/0: flags=0x20002 pwq 8: cpus=0-1 flags=0x4 nice=0 active=1 refcnt=4 pending: rds_send_worker workqueue krds_cp_wq#2/1: flags=0x20002 pwq 8: cpus=0-1 flags=0x4 nice=0 active=1 refcnt=4 pending: rds_send_worker workqueue krds_cp_wq#2/6: flags=0x20002 pwq 8: cpus=0-1 flags=0x4 nice=0 active=1 refcnt=4 pending: rds_send_worker workqueue krds_cp_wq#2/7: flags=0x20002 pwq 8: cpus=0-1 flags=0x4 nice=0 active=1 refcnt=4 pending: rds_send_worker workqueue wg-kex-wg0: flags=0x124 pwq 2: cpus=0 node=0 flags=0x0 nice=0 active=1 refcnt=2 pending: wg_packet_handshake_receive_worker workqueue wg-kex-wg0: flags=0x6 pwq 8: cpus=0-1 flags=0x4 nice=0 active=1 refcnt=2 pending: wg_packet_handshake_send_worker workqueue wg-crypt-wg0: flags=0x128 pwq 2: cpus=0 node=0 flags=0x0 nice=0 active=4 refcnt=5 in-flight: 10:wg_packet_tx_worker for 41s pending: wg_packet_decrypt_worker, wg_packet_encrypt_worker, wg_packet_tx_worker workqueue wg-crypt-wg1: flags=0x128 pwq 2: cpus=0 node=0 flags=0x0 nice=0 active=3 refcnt=4 pending: wg_packet_decrypt_worker, wg_packet_tx_worker, wg_packet_encrypt_worker workqueue wg-kex-wg2: flags=0x124 pwq 2: cpus=0 node=0 flags=0x0 nice=0 active=1 refcnt=2 pending: wg_packet_handshake_receive_worker pwq 6: cpus=1 node=0 flags=0x0 nice=0 active=1 refcnt=2 in-flight: 4774:wg_packet_handshake_receive_worker for 10s workqueue wg-crypt-wg2: flags=0x128 pwq 2: cpus=0 node=0 flags=0x0 nice=0 active=3 refcnt=4 pending: wg_packet_decrypt_worker, wg_packet_encrypt_worker, wg_packet_tx_worker pool 2: cpus=0 node=0 flags=0x0 nice=0 hung=41s workers=11 idle: 6392 4816 6420 4796 6540 865 4797 pool 6: cpus=1 node=0 flags=0x0 nice=0 hung=1s workers=8 idle: 1500 4469 4804 26 24 pool 8: cpus=0-1 flags=0x4 nice=0 hung=2s workers=19 idle: 10408 pool 9: cpus=0-1 node=0 flags=0x4 nice=-20 hung=0s workers=6 idle: 50 4714 4720 4711 Showing backtraces of busy workers in stalled worker pools: pool 2: task:kworker/0:7 state:R running task stack:0 pid:6117 tgid:6117 ppid:2 task_flags:0x4208060 flags:0x00000010 Workqueue: wg-crypt-wg0 wg_packet_decrypt_worker Call trace: __switch_to+0x2b4/0x6e4 arch/arm64/kernel/process.c:775 (T) context_switch kernel/sched/core.c:5533 [inline] __schedule+0x135c/0x2b00 kernel/sched/core.c:7284 preempt_schedule_common+0xd4/0x190 kernel/sched/core.c:7463 preempt_schedule kernel/sched/core.c:7487 [inline] class_preempt_destructor include/linux/preempt.h:496 [inline] try_to_wake_up+0x5f8/0xdc4 kernel/sched/core.c:4445 wake_up_process+0x18/0x24 kernel/sched/core.c:4570 process_one_work kernel/workqueue.c:3364 [inline] process_scheduled_works+0x744/0x1250 kernel/workqueue.c:3479 worker_thread+0x798/0xbd0 kernel/workqueue.c:3560 kthread+0x304/0x3d4 kernel/kthread.c:436 ret_from_fork+0x10/0x20 arch/arm64/kernel/entry.S:854 pool 2: task:kworker/0:6 state:R running task stack:0 pid:4819 tgid:4819 ppid:2 task_flags:0x4208060 flags:0x00000010 Workqueue: wg-crypt-wg1 wg_packet_encrypt_worker Call trace: __switch_to+0x2b4/0x6e4 arch/arm64/kernel/process.c:775 (T) context_switch kernel/sched/core.c:5533 [inline] __schedule+0x135c/0x2b00 kernel/sched/core.c:7284 preempt_schedule_common+0xd4/0x190 kernel/sched/core.c:7463 preempt_schedule+0x60/0x78 kernel/sched/core.c:7487 __local_bh_enable_ip+0x16c/0x234 kernel/softirq.c:480 __raw_spin_unlock_bh include/linux/spinlock_api_smp.h:237 [inline] _raw_spin_unlock_bh+0x3c/0x4c kernel/locking/spinlock.c:245 spin_unlock_bh include/linux/spinlock.h:407 [inline] ptr_ring_consume_bh include/linux/ptr_ring.h:399 [inline] wg_packet_encrypt_worker+0x120c/0x1290 drivers/net/wireguard/send.c:293 process_one_work kernel/workqueue.c:3396 [inline] process_scheduled_works+0x91c/0x1250 kernel/workqueue.c:3479 worker_thread+0x798/0xbd0 kernel/workqueue.c:3560 kthread+0x304/0x3d4 kernel/kthread.c:436 ret_from_fork+0x10/0x20 arch/arm64/kernel/entry.S:854 pool 2: task:kworker/0:0 state:R running task stack:0 pid:9 tgid:9 ppid:2 task_flags:0x4208060 flags:0x20000010 Workqueue: wg-crypt-wg1 wg_packet_decrypt_worker Call trace: __switch_to+0x2b4/0x6e4 arch/arm64/kernel/process.c:775 (T) context_switch kernel/sched/core.c:5533 [inline] __schedule+0x135c/0x2b00 kernel/sched/core.c:7284 preempt_schedule_common+0xd4/0x190 kernel/sched/core.c:7463 preempt_schedule+0x60/0x78 kernel/sched/core.c:7487 __local_bh_enable_ip+0x16c/0x234 kernel/softirq.c:480 local_bh_enable include/linux/bottom_half.h:33 [inline] put_cpu_fpsimd_context arch/arm64/kernel/fpsimd.c:251 [inline] kernel_neon_begin+0x29c/0x400 arch/arm64/kernel/fpsimd.c:1924 class_ksimd_constructor arch/arm64/include/asm/simd.h:48 [inline] poly1305_blocks+0x110/0x204 lib/crypto/arm64/poly1305.h:30 poly1305_update+0x168/0x39c lib/crypto/poly1305.c:59 chacha20poly1305_crypt_sg_inplace+0x888/0xc30 lib/crypto/chacha20poly1305.c:304 chacha20poly1305_decrypt_sg_inplace+0x6c/0x94 lib/crypto/chacha20poly1305.c:353 decrypt_packet drivers/net/wireguard/receive.c:278 [inline] wg_packet_decrypt_worker+0x490/0xa00 drivers/net/wireguard/receive.c:501 process_one_work kernel/workqueue.c:3396 [inline] process_scheduled_works+0x91c/0x1250 kernel/workqueue.c:3479 worker_thread+0x798/0xbd0 kernel/workqueue.c:3560 kthread+0x304/0x3d4 kernel/kthread.c:436 ret_from_fork+0x10/0x20 arch/arm64/kernel/entry.S:854 pool 2: task:kworker/0:1 state:R running task stack:0 pid:10 tgid:10 ppid:2 task_flags:0x4208060 flags:0x00000010 Workqueue: wg-crypt-wg0 wg_packet_tx_worker Call trace: __switch_to+0x2b4/0x6e4 arch/arm64/kernel/process.c:775 (T) context_switch kernel/sched/core.c:5533 [inline] __schedule+0x135c/0x2b00 kernel/sched/core.c:7284 preempt_schedule_common+0xd4/0x190 kernel/sched/core.c:7463 preempt_schedule+0x60/0x78 kernel/sched/core.c:7487 __local_bh_enable_ip+0x16c/0x234 kernel/softirq.c:480 __raw_read_unlock_bh include/linux/rwlock_api_smp.h:283 [inline] _raw_read_unlock_bh+0x3c/0x4c kernel/locking/spinlock.c:319 wg_socket_send_skb_to_peer+0x150/0x19c drivers/net/wireguard/socket.c:183 wg_packet_create_data_done drivers/net/wireguard/send.c:251 [inline] wg_packet_tx_worker+0x188/0x75c drivers/net/wireguard/send.c:276 process_one_work kernel/workqueue.c:3396 [inline] process_scheduled_works+0x91c/0x1250 kernel/workqueue.c:3479 worker_thread+0x798/0xbd0 kernel/workqueue.c:3560 kthread+0x304/0x3d4 kernel/kthread.c:436 ret_from_fork+0x10/0x20 arch/arm64/kernel/entry.S:854