BUG: workqueue lockup - pool cpus=0 node=0 flags=0x0 nice=0 stuck for 163s! Showing busy workqueues and worker pools: workqueue events: flags=0x100 pwq 2: cpus=0 node=0 flags=0x0 nice=0 active=18 refcnt=19 in-flight: 5837:nsim_fib_event_work for 163s pending: 6*nsim_dev_hwstats_traffic_work, psi_avgs_work, vmstat_shepherd, 3*ovs_dp_masks_rebalance, 3*psi_avgs_work, ovs_dp_masks_rebalance, nsim_fib_event_work, rht_deferred_worker pwq 6: cpus=1 node=0 flags=0x0 nice=0 active=5 refcnt=6 in-flight: 1229:switchdev_deferred_process_work for 13s pending: 4*nsim_dev_hwstats_traffic_work workqueue events_long: flags=0x100 pwq 2: cpus=0 node=0 flags=0x0 nice=0 active=7 refcnt=8 pending: 4*defense_work_handler, 3*br_multicast_gc_work workqueue events_unbound: flags=0x2 pwq 8: cpus=0-1 flags=0x4 nice=0 active=18 refcnt=19 in-flight: 28996:nsim_dev_trap_report_work for 1s ,6257:cfg80211_wiphy_work for 0s ,28999:nsim_dev_trap_report_work for 0s ,6273:cfg80211_wiphy_work for 0s pending: 8*nsim_dev_trap_report_work, toggle_allocation_gate, 5*cfg80211_wiphy_work workqueue events_freezable: flags=0x104 pwq 2: cpus=0 node=0 flags=0x0 nice=0 active=1 refcnt=2 pending: update_balloon_stats_func workqueue events_power_efficient: flags=0x180 pwq 2: cpus=0 node=0 flags=0x0 nice=0 active=7 refcnt=8 pending: check_lifetime, do_cache_clean, check_lifetime, hash_ipportip4_gc, 3*check_lifetime pwq 6: cpus=1 node=0 flags=0x0 nice=0 active=1 refcnt=2 in-flight: 6683:gc_worker for 0s workqueue rcu_gp: flags=0x108 pwq 2: cpus=0 node=0 flags=0x0 nice=0 active=3 refcnt=4 pending: 3*srcu_invoke_callbacks workqueue mm_percpu_wq: flags=0x108 pwq 2: cpus=0 node=0 flags=0x0 nice=0 active=1 refcnt=2 pending: vmstat_update workqueue kblockd: flags=0x18 pwq 3: cpus=0 node=0 flags=0x0 nice=-20 active=1 refcnt=2 pending: blk_mq_timeout_work workqueue ipv6_addrconf: flags=0x6000a pwq 8: cpus=0-1 flags=0x4 nice=0 active=1 refcnt=8 in-flight: 28993:addrconf_dad_work for 12s inactive: 4*addrconf_verify_work workqueue bat_events: flags=0x6000a pwq 8: cpus=0-1 flags=0x4 nice=0 active=1 refcnt=33 pending: batadv_mcast_mla_update inactive: 4*batadv_mcast_mla_update, 3*batadv_iv_send_outstanding_bat_ogm_packet, 5*batadv_purge_orig, 3*batadv_iv_send_outstanding_bat_ogm_packet, batadv_tt_purge, 2*batadv_iv_send_outstanding_bat_ogm_packet, batadv_tt_purge, 4*batadv_iv_send_outstanding_bat_ogm_packet, batadv_bla_periodic_work, 2*batadv_dat_purge, batadv_bla_periodic_work, batadv_dat_purge, batadv_tt_purge workqueue wg-crypt-wg0: flags=0x128 pwq 2: cpus=0 node=0 flags=0x0 nice=0 active=2 refcnt=3 pending: wg_packet_encrypt_worker, wg_packet_tx_worker workqueue wg-crypt-wg1: flags=0x128 pwq 2: cpus=0 node=0 flags=0x0 nice=0 active=2 refcnt=3 pending: wg_packet_encrypt_worker, wg_packet_tx_worker workqueue wg-crypt-wg2: flags=0x128 pwq 2: cpus=0 node=0 flags=0x0 nice=0 active=2 refcnt=3 pending: wg_packet_encrypt_worker, wg_packet_tx_worker workqueue wg-kex-wg0: flags=0x124 pwq 2: cpus=0 node=0 flags=0x0 nice=0 active=1 refcnt=2 pending: wg_packet_handshake_receive_worker workqueue wg-crypt-wg0: flags=0x128 pwq 2: cpus=0 node=0 flags=0x0 nice=0 active=3 refcnt=4 pending: wg_packet_encrypt_worker, wg_packet_tx_worker, wg_packet_decrypt_worker workqueue wg-kex-wg1: flags=0x124 pwq 2: cpus=0 node=0 flags=0x0 nice=0 active=1 refcnt=2 pending: wg_packet_handshake_receive_worker workqueue wg-kex-wg1: flags=0x6 pwq 8: cpus=0-1 flags=0x4 nice=0 active=1 refcnt=2 pending: wg_packet_handshake_send_worker workqueue wg-crypt-wg1: flags=0x128 pwq 2: cpus=0 node=0 flags=0x0 nice=0 active=3 refcnt=4 pending: wg_packet_decrypt_worker, wg_packet_encrypt_worker, wg_packet_tx_worker workqueue wg-kex-wg2: flags=0x124 pwq 2: cpus=0 node=0 flags=0x0 nice=0 active=1 refcnt=2 pending: wg_packet_handshake_receive_worker workqueue wg-crypt-wg2: flags=0x128 pwq 2: cpus=0 node=0 flags=0x0 nice=0 active=2 refcnt=3 pending: wg_packet_encrypt_worker, wg_packet_tx_worker workqueue wg-kex-wg0: flags=0x124 pwq 2: cpus=0 node=0 flags=0x0 nice=0 active=1 refcnt=2 pending: wg_packet_handshake_receive_worker workqueue wg-crypt-wg0: flags=0x128 pwq 2: cpus=0 node=0 flags=0x0 nice=0 active=3 refcnt=4 pending: wg_packet_encrypt_worker, wg_packet_tx_worker, wg_packet_decrypt_worker workqueue wg-kex-wg1: flags=0x124 pwq 2: cpus=0 node=0 flags=0x0 nice=0 active=1 refcnt=2 pending: wg_packet_handshake_receive_worker workqueue wg-crypt-wg1: flags=0x128 pwq 2: cpus=0 node=0 flags=0x0 nice=0 active=3 refcnt=4 pending: wg_packet_decrypt_worker, wg_packet_encrypt_worker, wg_packet_tx_worker workqueue wg-crypt-wg0: flags=0x128 pwq 2: cpus=0 node=0 flags=0x0 nice=0 active=2 refcnt=3 pending: wg_packet_encrypt_worker, wg_packet_tx_worker workqueue wg-kex-wg2: flags=0x124 pwq 2: cpus=0 node=0 flags=0x0 nice=0 active=1 refcnt=2 pending: wg_packet_handshake_receive_worker workqueue wg-crypt-wg2: flags=0x128 pwq 2: cpus=0 node=0 flags=0x0 nice=0 active=2 refcnt=3 pending: wg_packet_encrypt_worker, wg_packet_tx_worker workqueue wg-crypt-wg1: flags=0x128 pwq 2: cpus=0 node=0 flags=0x0 nice=0 active=2 refcnt=3 pending: wg_packet_encrypt_worker, wg_packet_tx_worker workqueue wg-crypt-wg2: flags=0x128 pwq 2: cpus=0 node=0 flags=0x0 nice=0 active=2 refcnt=3 pending: wg_packet_encrypt_worker, wg_packet_tx_worker workqueue wg-kex-wg0: flags=0x124 pwq 2: cpus=0 node=0 flags=0x0 nice=0 active=1 refcnt=2 pending: wg_packet_handshake_receive_worker workqueue wg-crypt-wg0: flags=0x128 pwq 2: cpus=0 node=0 flags=0x0 nice=0 active=3 refcnt=4 pending: wg_packet_encrypt_worker, wg_packet_tx_worker, wg_packet_decrypt_worker workqueue wg-kex-wg2: flags=0x124 pwq 2: cpus=0 node=0 flags=0x0 nice=0 active=1 refcnt=2 pending: wg_packet_handshake_receive_worker workqueue wg-crypt-wg2: flags=0x128 pwq 2: cpus=0 node=0 flags=0x0 nice=0 active=2 refcnt=3 pending: wg_packet_encrypt_worker, wg_packet_tx_worker pool 2: cpus=0 node=0 flags=0x0 nice=0 hung=164s workers=9 idle: 5916 797 29136 5890 6770 17955 9 10 pool 6: cpus=1 node=0 flags=0x0 nice=0 hung=1s workers=7 idle: 5944 796 6684 18802 24 pool 8: cpus=0-1 flags=0x4 nice=0 hung=0s workers=33 idle: 28984 28991 28985 28998 28990 28997 28994 29001 28981 28995 28988 36 28986 86 28980 6269 6965 29000 1105 12 6271 28975 28977 28976 1148 29002 Showing backtraces of busy workers in stalled worker pools: pool 2: task:kworker/0:3 state:R running task stack:24416 pid:5837 tgid:5837 ppid:2 task_flags:0x4208060 flags:0x00080000 Workqueue: events nsim_fib_event_work Call Trace: context_switch kernel/sched/core.c:5298 [inline] __schedule+0x15dd/0x52d0 kernel/sched/core.c:6911 __schedule_loop kernel/sched/core.c:6993 [inline] schedule+0x164/0x360 kernel/sched/core.c:7008 schedule_timeout+0x158/0x2c0 kernel/time/sleep_timeout.c:99 schedule_timeout_uninterruptible kernel/time/sleep_timeout.c:158 [inline] msleep+0xa1/0x100 kernel/time/sleep_timeout.c:318 nsim_fib6_rt_replace drivers/net/netdevsim/fib.c:727 [inline] nsim_fib6_rt_insert drivers/net/netdevsim/fib.c:761 [inline] nsim_fib6_event drivers/net/netdevsim/fib.c:856 [inline] nsim_fib_event+0x8b01/0xa410 drivers/net/netdevsim/fib.c:889 nsim_fib_event_work+0x254/0x3d0 drivers/net/netdevsim/fib.c:1493 process_one_work kernel/workqueue.c:3276 [inline] process_scheduled_works+0xb6e/0x18c0 kernel/workqueue.c:3359 worker_thread+0xa53/0xfc0 kernel/workqueue.c:3440 kthread+0x388/0x470 kernel/kthread.c:436 ret_from_fork+0x51e/0xb90 arch/x86/kernel/process.c:158 ret_from_fork_asm+0x1a/0x30 arch/x86/entry/entry_64.S:245