BUG: workqueue lockup - pool cpus=1 node=0 flags=0x0 nice=0 stuck for 42s! Showing busy workqueues and worker pools: workqueue events: flags=0x0 pwq 6: cpus=1 node=0 flags=0x0 nice=0 active=17 refcnt=18 in-flight: 8283:reg_todo pending: 5*nsim_dev_hwstats_traffic_work, drm_fb_helper_damage_work, 4*psi_avgs_work, 3*ovs_dp_masks_rebalance, psi_avgs_work, free_obj_work, ovs_dp_masks_rebalance workqueue events_long: flags=0x0 pwq 6: cpus=1 node=0 flags=0x0 nice=0 active=5 refcnt=6 pending: 5*defense_work_handler workqueue events_unbound: flags=0x2 pwq 8: cpus=0-1 flags=0x6 nice=0 active=7 refcnt=8 in-flight: 14:nsim_dev_trap_report_work ,12:nsim_dev_trap_report_work ,41:toggle_allocation_gate ,8402:nsim_dev_trap_report_work ,4618:nsim_dev_trap_report_work pending: 2*nsim_dev_trap_report_work pwq 8: cpus=0-1 flags=0x6 nice=0 active=4 refcnt=5 in-flight: 1925:cfg80211_wiphy_work cfg80211_wiphy_work ,42:cfg80211_wiphy_work cfg80211_wiphy_work workqueue events_power_efficient: flags=0x82 pwq 8: cpus=0-1 flags=0x6 nice=0 active=4 refcnt=5 in-flight: 2233:gc_worker ,8404:reg_check_chans_work ,4767:neigh_periodic_work pending: wg_ratelimiter_gc_entries workqueue kvfree_rcu_reclaim: flags=0xa pwq 8: cpus=0-1 flags=0x6 nice=0 active=1 refcnt=2 pending: kfree_rcu_monitor workqueue mm_percpu_wq: flags=0x8 pwq 6: cpus=1 node=0 flags=0x0 nice=0 active=1 refcnt=2 pending: vmstat_update workqueue writeback: flags=0x4a pwq 8: cpus=0-1 flags=0x6 nice=0 active=1 refcnt=2 in-flight: 4840:wb_workfn workqueue usb_hub_wq: flags=0x4 pwq 6: cpus=1 node=0 flags=0x0 nice=0 active=2 refcnt=3 in-flight: 2334:hub_event hub_event workqueue mld: flags=0x40008 pwq 6: cpus=1 node=0 flags=0x0 nice=0 active=1 refcnt=26 in-flight: 24:mld_dad_work inactive: mld_ifc_work, mld_dad_work, 4*mld_ifc_work, mld_dad_work, mld_ifc_work, 3*mld_dad_work, 2*mld_ifc_work, mld_dad_work, mld_ifc_work, 3*mld_dad_work, 3*mld_ifc_work, 3*mld_dad_work workqueue ipv6_addrconf: flags=0x6000a pwq 8: cpus=0-1 flags=0x6 nice=0 active=1 refcnt=74 in-flight: 213:addrconf_dad_work inactive: 69*addrconf_dad_work, addrconf_verify_work workqueue bat_events: flags=0x6000a pwq 8: cpus=0-1 flags=0x6 nice=0 active=1 refcnt=43 in-flight: 4280(RESCUER):batadv_dat_purge inactive: batadv_bla_periodic_work, 3*batadv_tt_purge, batadv_dat_purge, batadv_bla_periodic_work, batadv_dat_purge, batadv_bla_periodic_work, batadv_dat_purge, batadv_bla_periodic_work, batadv_nc_worker, batadv_mcast_mla_update, 4*batadv_nc_worker, 4*batadv_mcast_mla_update, 2*batadv_iv_send_outstanding_bat_ogm_packet, batadv_purge_orig, batadv_iv_send_outstanding_bat_ogm_packet, 3*batadv_purge_orig, 8*batadv_iv_send_outstanding_bat_ogm_packet, batadv_purge_orig, 2*batadv_tt_purge workqueue hci0: flags=0x20012 pwq 9: cpus=0-1 node=0 flags=0x4 nice=-20 active=1 refcnt=4 in-flight: 52:hci_cmd_sync_work workqueue hci1: flags=0x20012 pwq 9: cpus=0-1 node=0 flags=0x4 nice=-20 active=1 refcnt=4 in-flight: 6546:hci_cmd_sync_work workqueue hci3: flags=0x20012 pwq 9: cpus=0-1 node=0 flags=0x4 nice=-20 active=1 refcnt=4 in-flight: 6549:hci_cmd_sync_work workqueue wg-kex-wg0: flags=0x24 pwq 2: cpus=0 node=0 flags=0x0 nice=0 active=1 refcnt=2 in-flight: 6608:wg_packet_handshake_receive_worker pwq 6: cpus=1 node=0 flags=0x0 nice=0 active=1 refcnt=2 pending: wg_packet_handshake_receive_worker workqueue wg-crypt-wg0: flags=0x28 pwq 6: cpus=1 node=0 flags=0x0 nice=0 active=3 refcnt=4 pending: wg_packet_decrypt_worker, wg_packet_tx_worker, wg_packet_encrypt_worker workqueue wg-kex-wg1: flags=0x24 pwq 6: cpus=1 node=0 flags=0x0 nice=0 active=1 refcnt=2 pending: wg_packet_handshake_receive_worker workqueue wg-crypt-wg1: flags=0x28 pwq 6: cpus=1 node=0 flags=0x0 nice=0 active=3 refcnt=4 pending: wg_packet_decrypt_worker, wg_packet_encrypt_worker, wg_packet_tx_worker workqueue wg-crypt-wg1: flags=0x28 pwq 6: cpus=1 node=0 flags=0x0 nice=0 active=2 refcnt=3 pending: wg_packet_tx_worker, wg_packet_encrypt_worker workqueue wg-kex-wg2: flags=0x24 pwq 6: cpus=1 node=0 flags=0x0 nice=0 active=1 refcnt=2 pending: wg_packet_handshake_receive_worker workqueue wg-crypt-wg2: flags=0x28 pwq 6: cpus=1 node=0 flags=0x0 nice=0 active=2 refcnt=3 pending: wg_packet_tx_worker, wg_packet_decrypt_worker workqueue wg-crypt-wg2: flags=0x28 pwq 6: cpus=1 node=0 flags=0x0 nice=0 active=1 refcnt=2 pending: wg_packet_encrypt_worker workqueue wg-kex-wg0: flags=0x6 pwq 8: cpus=0-1 flags=0x6 nice=0 active=1 refcnt=2 pending: wg_packet_handshake_send_worker workqueue wg-crypt-wg0: flags=0x28 pwq 6: cpus=1 node=0 flags=0x0 nice=0 active=2 refcnt=3 pending: wg_packet_encrypt_worker, wg_packet_tx_worker workqueue wg-crypt-wg1: flags=0x28 pwq 6: cpus=1 node=0 flags=0x0 nice=0 active=2 refcnt=3 pending: wg_packet_tx_worker, wg_packet_encrypt_worker workqueue wg-crypt-wg2: flags=0x28 pwq 6: cpus=1 node=0 flags=0x0 nice=0 active=2 refcnt=3 in-flight: 6607:wg_packet_tx_worker pending: wg_packet_encrypt_worker workqueue wg-kex-wg0: flags=0x24 pwq 2: cpus=0 node=0 flags=0x0 nice=0 active=1 refcnt=2 in-flight: 6617:wg_packet_handshake_receive_worker pwq 6: cpus=1 node=0 flags=0x0 nice=0 active=1 refcnt=2 pending: wg_packet_handshake_receive_worker workqueue wg-crypt-wg0: flags=0x28 pwq 6: cpus=1 node=0 flags=0x0 nice=0 active=2 refcnt=3 pending: wg_packet_decrypt_worker, wg_packet_encrypt_worker workqueue wg-kex-wg1: flags=0x24 pwq 6: cpus=1 node=0 flags=0x0 nice=0 active=1 refcnt=2 pending: wg_packet_handshake_receive_worker workqueue wg-crypt-wg1: flags=0x28 pwq 6: cpus=1 node=0 flags=0x0 nice=0 active=2 refcnt=3 pending: wg_packet_decrypt_worker, wg_packet_encrypt_worker workqueue wg-kex-wg2: flags=0x24 pwq 2: cpus=0 node=0 flags=0x0 nice=0 active=2 refcnt=3 in-flight: 6545:wg_packet_handshake_receive_worker wg_packet_handshake_receive_worker pwq 6: cpus=1 node=0 flags=0x0 nice=0 active=2 refcnt=3 in-flight: 8207:wg_packet_handshake_receive_worker pending: wg_packet_handshake_receive_worker workqueue wg-crypt-wg2: flags=0x28 pwq 6: cpus=1 node=0 flags=0x0 nice=0 active=2 refcnt=3 pending: wg_packet_decrypt_worker, wg_packet_encrypt_worker pool 2: cpus=0 node=0 flags=0x0 nice=0 hung=0s workers=9 idle: 9 10 6600 6149 6686 6618 pool 6: cpus=1 node=0 flags=0x0 nice=0 hung=42s workers=8 idle: 6654 8292 26 pool 8: cpus=0-1 flags=0x6 nice=0 hung=0s workers=13 manager: 8405 pool 9: cpus=0-1 node=0 flags=0x4 nice=-20 hung=0s workers=7 idle: 6554 6148 6556 6555 Showing backtraces of running workers in stalled CPU-bound worker pools: pool 6: task:kworker/1:3 state:R running task stack:0 pid:6607 tgid:6607 ppid:2 task_flags:0x4208060 flags:0x00000010 Workqueue: wg-crypt-wg2 wg_packet_tx_worker Call trace: __switch_to+0x418/0x87c arch/arm64/kernel/process.c:741 (T) context_switch kernel/sched/core.c:5357 [inline] __schedule+0x13b0/0x2864 kernel/sched/core.c:6961 preempt_schedule_common+0xe0/0x194 kernel/sched/core.c:7145 preempt_schedule+0x60/0x80 kernel/sched/core.c:7169 __local_bh_enable_ip+0x20c/0x35c kernel/softirq.c:414 local_bh_enable+0x28/0x34 include/linux/bottom_half.h:33 rcu_read_unlock_bh include/linux/rcupdate.h:910 [inline] keep_key_fresh drivers/net/wireguard/send.c:135 [inline] wg_packet_create_data_done drivers/net/wireguard/send.c:259 [inline] wg_packet_tx_worker+0x634/0x6c8 drivers/net/wireguard/send.c:276 process_one_work+0x7e8/0x155c kernel/workqueue.c:3236 process_scheduled_works kernel/workqueue.c:3319 [inline] worker_thread+0x958/0xed8 kernel/workqueue.c:3400 kthread+0x5fc/0x75c kernel/kthread.c:463 ret_from_fork+0x10/0x20 arch/arm64/kernel/entry.S:844 pool 6: task:kworker/1:2 state:R running task stack:0 pid:2334 tgid:2334 ppid:2 task_flags:0x4208060 flags:0x00000010 Workqueue: usb_hub_wq hub_event Call trace: __switch_to+0x418/0x87c arch/arm64/kernel/process.c:741 (T) context_switch kernel/sched/core.c:5357 [inline] __schedule+0x13b0/0x2864 kernel/sched/core.c:6961 preempt_schedule_common+0xe0/0x194 kernel/sched/core.c:7145 preempt_schedule+0x60/0x80 kernel/sched/core.c:7169 __free_frozen_pages+0x648/0xcac mm/page_alloc.c:2924 free_frozen_pages+0x14/0x20 mm/page_alloc.c:2933 __free_slab+0x104/0x1c8 mm/slub.c:2729 free_slab+0x24/0x120 mm/slub.c:2752 discard_slab mm/slub.c:2758 [inline] __put_partials+0x194/0x228 mm/slub.c:3223 put_cpu_partial+0x164/0x1c0 mm/slub.c:3298 __slab_free+0x1a0/0x25c mm/slub.c:4565 do_slab_free mm/slub.c:4647 [inline] ___cache_free+0x150/0x170 mm/slub.c:4753 qlink_free+0x48/0x94 mm/kasan/quarantine.c:163 qlist_free_all+0x44/0xb8 mm/kasan/quarantine.c:179 kasan_quarantine_reduce+0x118/0x124 mm/kasan/quarantine.c:286 __kasan_slab_alloc+0x2c/0x88 mm/kasan/common.c:340 kasan_slab_alloc include/linux/kasan.h:250 [inline] slab_post_alloc_hook mm/slub.c:4191 [inline] slab_alloc_node mm/slub.c:4240 [inline] kmem_cache_alloc_node_noprof+0x240/0x3f4 mm/slub.c:4292 kmalloc_reserve+0xc4/0x268 net/core/skbuff.c:578 __alloc_skb+0x170/0x2ec net/core/skbuff.c:669 alloc_skb include/linux/skbuff.h:1336 [inline] alloc_uevent_skb+0x78/0x1e0 lib/kobject_uevent.c:289 uevent_net_broadcast_tagged lib/kobject_uevent.c:352 [inline] kobject_uevent_net_broadcast+0x15c/0x548 lib/kobject_uevent.c:413 kobject_uevent_env+0x528/0x884 lib/kobject_uevent.c:608 kobject_uevent+0x2c/0x3c lib/kobject_uevent.c:656 __kobject_del+0xd8/0x2d4 lib/kobject.c:601 kobject_cleanup lib/kobject.c:680 [inline] kobject_release lib/kobject.c:720 [inline] kref_put include/linux/kref.h:65 [inline] kobject_put+0x2c8/0x438 lib/kobject.c:737 netdev_queue_update_kobjects+0x4f4/0x5e0 net/core/net-sysfs.c:2071 remove_queue_kobjects net/core/net-sysfs.c:2168 [inline] netdev_unregister_kobject+0x100/0x3bc net/core/net-sysfs.c:2323 unregister_netdevice_many_notify+0x1770/0x1de0 net/core/dev.c:12201 unregister_netdevice_many net/core/dev.c:12229 [inline] unregister_netdevice_queue net/core/dev.c:12073 [inline] unregister_netdevice include/linux/netdevice.h:3385 [inline] unregister_netdev+0x15c/0x1f8 net/core/dev.c:12247 unregister_candev+0x20/0x30 drivers/net/can/dev/dev.c:580 peak_usb_create_dev drivers/net/can/usb/peak_usb/pcan_usb_core.c:1026 [inline] peak_usb_probe+0xd10/0xe1c drivers/net/can/usb/peak_usb/pcan_usb_core.c:1089 usb_probe_interface+0x5a4/0xaac drivers/usb/core/driver.c:396 call_driver_probe drivers/base/dd.c:-1 [inline] really_probe+0x3b4/0x944 drivers/base/dd.c:659 __driver_probe_device+0x180/0x2d4 drivers/base/dd.c:801 driver_probe_device+0x78/0x330 drivers/base/dd.c:831 __device_attach_driver+0x290/0x4e0 drivers/base/dd.c:959 bus_for_each_drv+0x220/0x2b4 drivers/base/bus.c:462 __device_attach+0x26c/0x388 drivers/base/dd.c:1031 device_initial_probe+0x24/0x34 drivers/base/dd.c:1080 bus_probe_device+0x178/0x240 drivers/base/bus.c:537 device_add+0x71c/0xa60 drivers/base/core.c:3689 usb_set_configuration+0x1640/0x1bac drivers/usb/core/message.c:2210 usb_generic_driver_probe+0x8c/0x144 drivers/usb/core/generic.c:250 usb_probe_device+0x1a4/0x348 drivers/usb/core/driver.c:291 call_driver_probe drivers/base/dd.c:-1 [inline] really_probe+0x3b4/0x944 drivers/base/dd.c:659 __driver_probe_device+0x180/0x2d4 drivers/base/dd.c:801 driver_probe_device+0x78/0x330 drivers/base/dd.c:831 __device_attach_driver+0x290/0x4e0 drivers/base/dd.c:959 bus_for_each_drv+0x220/0x2b4 drivers/base/bus.c:462 __device_attach+0x26c/0x388 drivers/base/dd.c:1031 device_initial_probe+0x24/0x34 drivers/base/dd.c:1080 bus_probe_device+0x178/0x240 drivers/base/bus.c:537 device_add+0x71c/0xa60 drivers/base/core.c:3689 usb_new_device+0x7f0/0x123c drivers/usb/core/hub.c:2694 hub_port_connect drivers/usb/core/hub.c:5566 [inline] hub_port_connect_change drivers/usb/core/hub.c:5706 [inline] port_event drivers/usb/core/hub.c:5870 [inline] hub_event+0x211c/0x3c78 drivers/usb/core/hub.c:5952 process_one_work+0x7e8/0x155c kernel/workqueue.c:3236 process_scheduled_works kernel/workqueue.c:3319 [inline] worker_thread+0x958/0xed8 kernel/workqueue.c:3400 kthread+0x5fc/0x75c kernel/kthread.c:463 ret_from_fork+0x10/0x20 arch/arm64/kernel/entry.S:844 pool 6: task:kworker/1:5 state:R running task stack:0 pid:8207 tgid:8207 ppid:2 task_flags:0x4208060 flags:0x00000010 Workqueue: wg-kex-wg2 wg_packet_handshake_receive_worker Call trace: __switch_to+0x418/0x87c arch/arm64/kernel/process.c:741 (T) context_switch kernel/sched/core.c:5357 [inline] __schedule+0x13b0/0x2864 kernel/sched/core.c:6961 preempt_schedule_common+0xe0/0x194 kernel/sched/core.c:7145 preempt_schedule+0x60/0x80 kernel/sched/core.c:7169 __local_bh_enable_ip+0x20c/0x35c kernel/softirq.c:414 local_bh_enable+0x28/0x34 include/linux/bottom_half.h:33 rcu_read_unlock_bh include/linux/rcupdate.h:910 [inline] mod_peer_timer+0x1e0/0x220 drivers/net/wireguard/timers.c:38 wg_timers_session_derived+0x5c/0x6c drivers/net/wireguard/timers.c:208 wg_packet_send_handshake_response+0x10c/0x24c drivers/net/wireguard/send.c:98 wg_receive_handshake_packet drivers/net/wireguard/receive.c:154 [inline] wg_packet_handshake_receive_worker+0x4fc/0xc20 drivers/net/wireguard/receive.c:213 process_one_work+0x7e8/0x155c kernel/workqueue.c:3236 process_scheduled_works kernel/workqueue.c:3319 [inline] worker_thread+0x958/0xed8 kernel/workqueue.c:3400 kthread+0x5fc/0x75c kernel/kthread.c:463 ret_from_fork+0x10/0x20 arch/arm64/kernel/entry.S:844 pool 6: task:kworker/1:0 state:R running task stack:0 pid:24 tgid:24 ppid:2 task_flags:0x4208060 flags:0x00000010 Workqueue: mld mld_dad_work Call trace: __switch_to+0x418/0x87c arch/arm64/kernel/process.c:741 (T) context_switch kernel/sched/core.c:5357 [inline] __schedule+0x13b0/0x2864 kernel/sched/core.c:6961 preempt_schedule_common+0xe0/0x194 kernel/sched/core.c:7145 preempt_schedule+0x60/0x80 kernel/sched/core.c:7169 __local_bh_enable_ip+0x20c/0x35c kernel/softirq.c:414 __raw_read_unlock_bh include/linux/rwlock_api_smp.h:257 [inline] _raw_read_unlock_bh+0x3c/0x4c kernel/locking/spinlock.c:284 ipv6_get_lladdr+0x26c/0x3a4 net/ipv6/addrconf.c:1942 mld_newpack+0x340/0xa28 net/ipv6/mcast.c:1794 add_grhead+0x6c/0x25c net/ipv6/mcast.c:1891 add_grec+0x1120/0x13c8 net/ipv6/mcast.c:2030 mld_send_initial_cr+0x27c/0x534 net/ipv6/mcast.c:2273 mld_dad_work+0x40/0x418 net/ipv6/mcast.c:2299 process_one_work+0x7e8/0x155c kernel/workqueue.c:3236 process_scheduled_works kernel/workqueue.c:3319 [inline] worker_thread+0x958/0xed8 kernel/workqueue.c:3400 kthread+0x5fc/0x75c kernel/kthread.c:463 ret_from_fork+0x10/0x20 arch/arm64/kernel/entry.S:844