syzbot


INFO: task hung in worker_thread (5)

Status: upstream: reported C repro on 2024/09/30 12:46
Subsystems: kernel
[Documentation on labels]
Reported-by: syzbot+c27cd2da6c2dc003b4a1@syzkaller.appspotmail.com
First crash: 374d, last: 35d
Cause bisection: introduced by (bisect log) :
commit d15121be7485655129101f3960ae6add40204463
Author: Paolo Abeni <pabeni@redhat.com>
Date: Mon May 8 06:17:44 2023 +0000

  Revert "softirq: Let ksoftirqd do its job"

Crash: no output from test machine (log)
Repro: C syz .config
  
Fix bisection: fixed by (bisect log) :
commit 66951e4860d3c688bfa550ea4a19635b57e00eca
Author: Peter Zijlstra <peterz@infradead.org>
Date: Mon Jan 13 12:50:11 2025 +0000

  sched/fair: Fix update_cfs_group() vs DELAY_DEQUEUE

  
Discussions (1)
Title Replies (including bot) Last reply
[syzbot] [kernel?] INFO: task hung in worker_thread (5) 0 (2) 2025/05/16 13:12
Similar bugs (6)
Kernel Title Repro Cause bisect Fix bisect Count Last Reported Patched Status
linux-4.19 INFO: task hung in worker_thread 1 1224d 1224d 0/1 auto-closed as invalid on 2022/05/08 06:12
upstream INFO: task hung in worker_thread (3) serial 1 817d 817d 0/28 auto-obsoleted due to no activity on 2023/05/20 16:01
upstream INFO: task hung in worker_thread (4) kernel 1 502d 502d 0/28 auto-obsoleted due to no activity on 2024/03/31 02:07
upstream INFO: task hung in worker_thread (2) fs 1 1577d 1577d 0/28 auto-closed as invalid on 2021/05/17 11:26
linux-5.15 INFO: task hung in worker_thread 1 193d 193d 0/3 auto-obsoleted due to no activity on 2025/02/12 21:25
upstream INFO: task hung in worker_thread fs 1 1743d 1743d 0/28 auto-closed as invalid on 2020/11/06 03:23
Last patch testing requests (2)
Created Duration User Patch Repo Result
2025/03/30 01:36 35m retest repro upstream report log
2024/11/16 06:51 25m retest repro upstream report log
Fix bisection attempts (4)
Created Duration User Patch Repo Result
2025/05/16 03:30 9h40m bisect fix upstream OK (1) job log
2025/04/11 13:59 3h04m bisect fix upstream OK (0) job log log
2025/01/18 20:55 4h38m bisect fix upstream OK (0) job log log
2024/12/19 13:14 3h01m bisect fix upstream OK (0) job log log

Sample crash report:
INFO: task kworker/1:9:7428 blocked for more than 144 seconds.
      Not tainted 6.14.0-rc2-syzkaller #0
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
task:kworker/1:9     state:I stack:20240 pid:7428  tgid:7428  ppid:2      task_flags:0x4208060 flags:0x00004000
Workqueue:  0x0 (events)
Call Trace:
 <TASK>
 context_switch kernel/sched/core.c:5377 [inline]
 __schedule+0x190e/0x4c90 kernel/sched/core.c:6764
 __schedule_loop kernel/sched/core.c:6841 [inline]
 schedule+0x14b/0x320 kernel/sched/core.c:6856
 worker_thread+0xa30/0xd30 kernel/workqueue.c:3413
 kthread+0x7a9/0x920 kernel/kthread.c:464
 ret_from_fork+0x4b/0x80 arch/x86/kernel/process.c:148
 ret_from_fork_asm+0x1a/0x30 arch/x86/entry/entry_64.S:244
 </TASK>

Showing all locks held in the system:
1 lock held by kthreadd/2:
3 locks held by kworker/1:0/25:
1 lock held by khungtaskd/30:
 #0: ffffffff8e9387e0 (rcu_read_lock){....}-{1:3}, at: rcu_lock_acquire include/linux/rcupdate.h:337 [inline]
 #0: ffffffff8e9387e0 (rcu_read_lock){....}-{1:3}, at: rcu_read_lock include/linux/rcupdate.h:849 [inline]
 #0: ffffffff8e9387e0 (rcu_read_lock){....}-{1:3}, at: debug_show_all_locks+0x55/0x2a0 kernel/locking/lockdep.c:6746
2 locks held by kworker/0:2/1203:
2 locks held by syslogd/5189:
1 lock held by klogd/5196:
5 locks held by udevd/5207:
2 locks held by dhcpcd/5501:
2 locks held by getty/5593:
 #0: ffff88814e7e00a0 (&tty->ldisc_sem){++++}-{0:0}, at: tty_ldisc_ref_wait+0x25/0x70 drivers/tty/tty_ldisc.c:243
 #1: ffffc90002fde2f0 (&ldata->atomic_read_lock){+.+.}-{4:4}, at: n_tty_read+0x6a6/0x1e00 drivers/tty/n_tty.c:2211
3 locks held by syz-executor/5819:
5 locks held by syz-executor/14552:
1 lock held by kworker/u8:15/16040:
 #0: ffff8880b863e958 (&rq->__lock){-.-.}-{2:2}, at: raw_spin_rq_lock_nested+0x2a/0x140 kernel/sched/core.c:598
3 locks held by kworker/u8:16/19699:
7 locks held by kworker/u8:20/19704:
2 locks held by syz-executor/20347:
2 locks held by syz.8.2757/23083:
 #0: ffffffff8fcc0b48 (rtnl_mutex){+.+.}-{4:4}, at: tun_detach drivers/net/tun.c:698 [inline]
 #0: ffffffff8fcc0b48 (rtnl_mutex){+.+.}-{4:4}, at: tun_chr_close+0x3b/0x1b0 drivers/net/tun.c:3517
 #1: ffffffff8e93dcb8 (rcu_state.exp_mutex){+.+.}-{4:4}, at: exp_funnel_lock kernel/rcu/tree_exp.h:302 [inline]
 #1: ffffffff8e93dcb8 (rcu_state.exp_mutex){+.+.}-{4:4}, at: synchronize_rcu_expedited+0x381/0x830 kernel/rcu/tree_exp.h:996
3 locks held by syz-executor/23234:

=============================================

NMI backtrace for cpu 1
CPU: 1 UID: 0 PID: 30 Comm: khungtaskd Not tainted 6.14.0-rc2-syzkaller #0
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 12/27/2024
Call Trace:
 <TASK>
 __dump_stack lib/dump_stack.c:94 [inline]
 dump_stack_lvl+0x241/0x360 lib/dump_stack.c:120
 nmi_cpu_backtrace+0x49c/0x4d0 lib/nmi_backtrace.c:113
 nmi_trigger_cpumask_backtrace+0x198/0x320 lib/nmi_backtrace.c:62
 trigger_all_cpu_backtrace include/linux/nmi.h:162 [inline]
 check_hung_uninterruptible_tasks kernel/hung_task.c:236 [inline]
 watchdog+0x1058/0x10a0 kernel/hung_task.c:399
 kthread+0x7a9/0x920 kernel/kthread.c:464
 ret_from_fork+0x4b/0x80 arch/x86/kernel/process.c:148
 ret_from_fork_asm+0x1a/0x30 arch/x86/entry/entry_64.S:244
 </TASK>
Sending NMI from CPU 1 to CPUs 0:
NMI backtrace for cpu 0
CPU: 0 UID: 0 PID: 20347 Comm: syz-executor Not tainted 6.14.0-rc2-syzkaller #0
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 12/27/2024
RIP: 0010:preempt_count_add+0x34/0x190 kernel/sched/core.c:5851
Code: fb 48 c7 c0 80 09 40 9a 48 c1 e8 03 49 bf 00 00 00 00 00 fc ff df 42 0f b6 04 38 84 c0 0f 85 ed 00 00 00 83 3d 7c 8d b2 18 00 <75> 07 65 8b 05 7b 5a 76 7e 65 01 1d 74 5a 76 7e 48 c7 c0 80 09 40
RSP: 0018:ffffc900058bf730 EFLAGS: 00000246
RAX: 0000000000000004 RBX: 0000000000000001 RCX: ffffffff9a400903
RDX: dffffc0000000000 RSI: ffffffff823a6244 RDI: 0000000000000001
RBP: ffffc900058bf858 R08: 0000000000000009 R09: ffffc900058bf910
R10: ffffc900058bf870 R11: ffffffff81ab0e90 R12: dffffc0000000000
R13: ffffc900058bf820 R14: ffffffff823a6244 R15: dffffc0000000000
FS:  000055557d3db500(0000) GS:ffff8880b8600000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f86290d8740 CR3: 0000000022782000 CR4: 00000000003526f0
DR0: 0000000000000007 DR1: 0000000000000002 DR2: 0000000000000008
DR3: 1000000100000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
 <NMI>
 </NMI>
 <TASK>
 unwind_next_frame+0xb0/0x22d0 arch/x86/kernel/unwind_orc.c:479
 arch_stack_walk+0x11c/0x150 arch/x86/kernel/stacktrace.c:25
 stack_trace_save+0x118/0x1d0 kernel/stacktrace.c:122
 save_stack+0xfb/0x1f0 mm/page_owner.c:156
 __reset_page_owner+0x76/0x430 mm/page_owner.c:297
 reset_page_owner include/linux/page_owner.h:25 [inline]
 free_pages_prepare mm/page_alloc.c:1127 [inline]
 free_frozen_pages+0xe0d/0x10e0 mm/page_alloc.c:2660
 discard_slab mm/slub.c:2684 [inline]
 __put_partials+0x160/0x1c0 mm/slub.c:3153
 put_cpu_partial+0x17c/0x250 mm/slub.c:3228
 __slab_free+0x290/0x380 mm/slub.c:4479
 qlink_free mm/kasan/quarantine.c:163 [inline]
 qlist_free_all+0x9a/0x140 mm/kasan/quarantine.c:179
 kasan_quarantine_reduce+0x14f/0x170 mm/kasan/quarantine.c:286
 __kasan_slab_alloc+0x23/0x80 mm/kasan/common.c:329
 kasan_slab_alloc include/linux/kasan.h:250 [inline]
 slab_post_alloc_hook mm/slub.c:4115 [inline]
 slab_alloc_node mm/slub.c:4164 [inline]
 kmem_cache_alloc_noprof+0x1d9/0x380 mm/slub.c:4171
 getname_flags+0xb7/0x540 fs/namei.c:139
 user_path_at+0x24/0x60 fs/namei.c:3071
 ksys_umount fs/namespace.c:2071 [inline]
 __do_sys_umount fs/namespace.c:2079 [inline]
 __se_sys_umount fs/namespace.c:2077 [inline]
 __x64_sys_umount+0xf1/0x170 fs/namespace.c:2077
 do_syscall_x64 arch/x86/entry/common.c:52 [inline]
 do_syscall_64+0xf3/0x230 arch/x86/entry/common.c:83
 entry_SYSCALL_64_after_hwframe+0x77/0x7f
RIP: 0033:0x7f84f8f8e117
Code: a8 ff ff ff f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 44 00 00 31 f6 e9 09 00 00 00 66 0f 1f 84 00 00 00 00 00 b8 a6 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 01 c3 48 c7 c2 a8 ff ff ff f7 d8 64 89 02 b8
RSP: 002b:00007f84f92cecb8 EFLAGS: 00000246 ORIG_RAX: 00000000000000a6
RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f84f8f8e117
RDX: 0000000000000000 RSI: 0000000000000009 RDI: 00007f84f92ced70
RBP: 00007f84f92ced70 R08: 0000000000000000 R09: 0000000000000000
R10: 00000000ffffffff R11: 0000000000000246 R12: 00007f84f92cfe00
R13: 00007f84f900e08c R14: 00000000000b3379 R15: 00007f84f92cfe40
 </TASK>

Crashes (44):
Time Kernel Commit Syzkaller Config Log Report Syz repro C repro VM info Assets (help?) Manager Title
2025/02/10 23:06 upstream a64dcfb451e2 43f51a00 .config console log report info [disk image] [vmlinux] [kernel image] ci-upstream-kasan-gce INFO: task hung in worker_thread
2025/02/02 02:28 upstream 69e858e0b8b2 0dff8567 .config console log report info [disk image] [vmlinux] [kernel image] ci-upstream-kasan-gce-root INFO: task hung in worker_thread
2025/01/27 03:31 upstream c2da8b3f914f 9fbd772e .config console log report info [disk image] [vmlinux] [kernel image] ci-upstream-kasan-gce-smack-root INFO: task hung in worker_thread
2024/11/16 21:42 upstream e8bdb3c8be08 cfe3a04a .config console log report info [disk image] [vmlinux] [kernel image] ci-upstream-kasan-gce INFO: task hung in worker_thread
2024/10/31 04:54 upstream 4236f913808c fb888278 .config console log report info [disk image] [vmlinux] [kernel image] ci-upstream-kasan-gce-smack-root INFO: task hung in worker_thread
2024/10/07 11:08 upstream 8cf0b93919e1 d7906eff .config console log report info [disk image] [vmlinux] [kernel image] ci-upstream-kasan-gce-root INFO: task hung in worker_thread
2024/10/05 16:15 upstream 27cc6fdf7201 d7906eff .config console log report info [disk image] [vmlinux] [kernel image] ci-upstream-kasan-gce-root INFO: task hung in worker_thread
2024/09/28 01:19 upstream e477dba5442c 2b1784d6 .config console log report info [disk image] [vmlinux] [kernel image] ci-upstream-kasan-gce-smack-root INFO: task hung in worker_thread
2024/09/26 12:37 upstream aa486552a110 0d19f247 .config console log report syz / log C [disk image] [vmlinux] [kernel image] ci-upstream-kasan-gce INFO: task hung in worker_thread
2024/09/20 14:21 upstream a430d95c5efa 6f888b75 .config console log report info [disk image] [vmlinux] [kernel image] ci-upstream-kasan-gce-selinux-root INFO: task hung in worker_thread
2024/08/11 02:38 upstream 5189dafa4cf9 6f4edef4 .config console log report info [disk image] [vmlinux] [kernel image] ci-upstream-kasan-gce-root INFO: task hung in worker_thread
2024/07/13 05:22 upstream 528dd46d0fc3 eaeb5c15 .config console log report info [disk image] [vmlinux] [kernel image] ci-upstream-kasan-gce-selinux-root INFO: task hung in worker_thread
2024/07/09 08:21 upstream 4376e966ecb7 bc23a442 .config console log report info [disk image] [vmlinux] [kernel image] ci-upstream-kasan-gce-root INFO: task hung in worker_thread
2024/07/05 15:04 upstream 661e504db04c 2a40360c .config console log report info [disk image] [vmlinux] [kernel image] ci-upstream-kasan-gce-root INFO: task hung in worker_thread
2024/06/24 11:05 upstream f2661062f16b edc5149a .config console log report info [disk image] [vmlinux] [kernel image] ci-upstream-kasan-gce-smack-root INFO: task hung in worker_thread
2024/06/24 05:24 upstream 7c16f0a4ed1c edc5149a .config console log report info [disk image] [vmlinux] [kernel image] ci-upstream-kasan-gce-root INFO: task hung in worker_thread
2024/06/15 14:24 upstream 2ccbdf43d5e7 f429ab00 .config console log report info [disk image] [vmlinux] [kernel image] ci-upstream-kasan-gce-root INFO: task hung in worker_thread
2024/06/14 12:39 upstream 2ccbdf43d5e7 a9616ff5 .config console log report info [disk image] [vmlinux] [kernel image] ci-upstream-kasan-gce-smack-root INFO: task hung in worker_thread
2024/06/09 18:43 upstream 771ed66105de 82c05ab8 .config console log report info [disk image] [vmlinux] [kernel image] ci-upstream-kasan-gce-smack-root INFO: task hung in worker_thread
2024/06/05 13:50 upstream 32f88d65f01b e1e2c66e .config console log report info [disk image] [vmlinux] [kernel image] ci-upstream-kasan-gce-smack-root INFO: task hung in worker_thread
2024/05/30 04:05 upstream 4a4be1ad3a6e 34889ee3 .config console log report info [disk image] [vmlinux] [kernel image] ci-upstream-kasan-gce-root INFO: task hung in worker_thread
2024/05/28 18:27 upstream 2bfcfd584ff5 34889ee3 .config console log report info [disk image] [vmlinux] [kernel image] ci-upstream-kasan-gce-smack-root INFO: task hung in worker_thread
2024/05/25 07:19 upstream 0b32d436c015 a10a183e .config console log report info [disk image] [vmlinux] [kernel image] ci-upstream-kasan-gce-root INFO: task hung in worker_thread
2024/10/08 16:41 upstream 87d6aab2389e 402f1df0 .config console log report info [disk image] [vmlinux] [kernel image] ci-upstream-kasan-gce-386 INFO: task hung in worker_thread
2024/07/21 13:36 upstream 2c9b3512402e b88348e9 .config console log report info [disk image] [vmlinux] [kernel image] ci-upstream-kasan-gce-386 INFO: task hung in worker_thread
2025/02/12 14:26 net e589adf5b70c b27c2402 .config console log report info [disk image] [vmlinux] [kernel image] ci-upstream-net-this-kasan-gce INFO: task hung in worker_thread
2025/02/05 12:20 net a1300691aed9 5896748e .config console log report info [disk image] [vmlinux] [kernel image] ci-upstream-net-this-kasan-gce INFO: task hung in worker_thread
2025/01/26 06:12 net 15a901361ec3 9fbd772e .config console log report info [disk image] [vmlinux] [kernel image] ci-upstream-net-this-kasan-gce INFO: task hung in worker_thread
2024/11/19 12:17 net 4262bacb748f 571351cb .config console log report info [disk image] [vmlinux] [kernel image] ci-upstream-net-this-kasan-gce INFO: task hung in worker_thread
2024/10/18 01:50 net cb560795c8c2 666f77ed .config console log report info [disk image] [vmlinux] [kernel image] ci-upstream-net-this-kasan-gce INFO: task hung in worker_thread
2024/07/11 10:13 net 76a0a3f9cc2f c699c2eb .config console log report info [disk image] [vmlinux] [kernel image] ci-upstream-net-this-kasan-gce INFO: task hung in worker_thread
2024/06/20 02:19 net be27b8965297 41b7e219 .config console log report info [disk image] [vmlinux] [kernel image] ci-upstream-net-this-kasan-gce INFO: task hung in worker_thread
2025/03/11 13:44 net-next 40587f749df2 16256247 .config console log report info [disk image] [vmlinux] [kernel image] ci-upstream-net-kasan-gce INFO: task hung in worker_thread
2025/02/01 01:07 net-next c2933b2befe2 aa47157c .config console log report info [disk image] [vmlinux] [kernel image] ci-upstream-net-kasan-gce INFO: task hung in worker_thread
2025/01/26 14:05 net-next 0ad9617c78ac 9fbd772e .config console log report info [disk image] [vmlinux] [kernel image] ci-upstream-net-kasan-gce INFO: task hung in worker_thread
2025/01/23 01:01 net-next 0ad9617c78ac 25e17fd3 .config console log report info [disk image] [vmlinux] [kernel image] ci-upstream-net-kasan-gce INFO: task hung in worker_thread
2024/10/18 18:18 net-next 6d858708d465 cd6fc0a3 .config console log report info [disk image] [vmlinux] [kernel image] ci-upstream-net-kasan-gce INFO: task hung in worker_thread
2024/06/12 12:52 net-next 91579c93a9b2 f815599d .config console log report info [disk image] [vmlinux] [kernel image] ci2-upstream-net-next-test-gce INFO: task hung in worker_thread
2024/11/02 06:34 linux-next c88416ba074a f00eed24 .config console log report info [disk image] [vmlinux] [kernel image] ci-upstream-linux-next-kasan-gce-root INFO: task hung in worker_thread
2024/08/02 20:01 linux-next 931a3b3bccc9 53683cf2 .config console log report info [disk image] [vmlinux] [kernel image] ci-upstream-linux-next-kasan-gce-root INFO: task hung in worker_thread
2024/05/27 16:35 linux-next 3689b0ef08b7 a10a183e .config console log report info [disk image] [vmlinux] [kernel image] ci-upstream-linux-next-kasan-gce-root INFO: task hung in worker_thread
2024/05/17 15:46 linux-next c75962170e49 a12e99e7 .config console log report info [disk image] [vmlinux] [kernel image] ci-upstream-linux-next-kasan-gce-root INFO: task hung in worker_thread
2024/05/13 12:20 linux-next 6ba6c795dc73 9026e142 .config console log report info [disk image] [vmlinux] [kernel image] ci-upstream-linux-next-kasan-gce-root INFO: task hung in worker_thread
2024/05/07 19:08 linux-next 93a39e476608 cb2dcc0e .config console log report info [disk image] [vmlinux] [kernel image] ci-upstream-linux-next-kasan-gce-root INFO: task hung in worker_thread
* Struck through repros no longer work on HEAD.