INFO: rcu detected stall in cleanup_mnt (3)

Status: auto-obsoleted due to no activity on 2023/08/29 03:36
Subsystems: block fs
[Documentation on labels]
First crash: 1196d, last: 1186d
▶ ▼ Similar bugs (9)
Kernel Title Rank 🛈 Repro Cause bisect Fix bisect Count Last Reported Patched Status
linux-5.15 INFO: rcu detected stall in cleanup_mnt 1 1 1175d 1175d 0/3 auto-obsoleted due to no activity on 2023/09/19 04:00
upstream INFO: rcu detected stall in cleanup_mnt (2) block 1 1 1711d 1711d 0/29 closed as invalid on 2022/02/08 10:10
linux-6.1 INFO: rcu detected stall in cleanup_mnt (2) 1 1 804d 804d 0/3 auto-obsoleted due to no activity on 2024/09/23 19:34
linux-6.1 INFO: rcu detected stall in cleanup_mnt (3) 1 1 438d 438d 0/3 auto-obsoleted due to no activity on 2025/09/24 18:34
upstream INFO: rcu detected stall in cleanup_mnt (5) f2fs 1 12 226d 352d 0/29 auto-obsoleted due to no activity on 2026/04/14 22:36
linux-6.1 INFO: rcu detected stall in cleanup_mnt 1 1 1120d 1120d 0/3 auto-obsoleted due to no activity on 2023/11/13 00:06
upstream INFO: rcu detected stall in cleanup_mnt exfat 1 1 2242d 2242d 0/29 auto-closed as invalid on 2020/09/06 14:28
upstream INFO: rcu detected stall in cleanup_mnt (4) f2fs mm 1 21 451d 808d 0/29 auto-obsoleted due to no activity on 2025/09/02 03:29
linux-5.15 INFO: rcu detected stall in cleanup_mnt (2) origin:lts-only 1 syz, C done 21 13d 748d 0/3 upstream: reported C repro on 2024/08/10 23:43

Sample crash report:
rcu: INFO: rcu_preempt detected stalls on CPUs/tasks:
rcu: 	0-...!: (1 GPs behind) idle=15bc/1/0x4000000000000000 softirq=61719/61720 fqs=1
rcu: 	(detected by 1, t=10504 jiffies, g=76513, q=50 ncpus=2)
Sending NMI from CPU 1 to CPUs 0:
NMI backtrace for cpu 0
CPU: 0 PID: 5031 Comm: syz-executor.4 Not tainted 6.4.0-rc4-syzkaller-00047-gafead42fdfca #0
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 05/25/2023
RIP: 0010:__lock_release kernel/locking/lockdep.c:5363 [inline]
RIP: 0010:lock_release+0x183/0x670 kernel/locking/lockdep.c:5725
Code: 33 e9 a9 08 48 89 da 83 e3 07 48 b8 00 00 00 00 00 fc ff df 48 c1 ea 03 c7 44 24 40 01 00 00 00 83 c3 03 65 ff 05 1d a2 9c 7e <0f> b6 04 02 65 4c 8b 3c 25 c0 bb 03 00 38 c3 7c 08 84 c0 0f 85 43
RSP: 0018:ffffc90000007c78 EFLAGS: 00000002
RAX: dffffc0000000000 RBX: 0000000000000003 RCX: ffffffff8165e685
RDX: 1ffffffff1cf561e RSI: 0000000000010003 RDI: 0000000000000000
RBP: 1ffff92000000f91 R08: 0000000000000001 R09: ffffffff8e7a7bd7
R10: fffffbfff1cf4f7a R11: 0000000000000001 R12: ffffffff91fbad48
R13: ffffffff8a4ee140 R14: ffff8880283f1dc0 R15: ffff88807c41b340
FS:  0000555556709400(0000) GS:ffff8880b9800000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00005555561b8708 CR3: 0000000033805000 CR4: 0000000000350ef0
Call Trace:
 <NMI>
 </NMI>
 <IRQ>
 __raw_spin_unlock_irqrestore include/linux/spinlock_api_smp.h:149 [inline]
 _raw_spin_unlock_irqrestore+0x1a/0x70 kernel/locking/spinlock.c:194
 debug_object_deactivate lib/debugobjects.c:796 [inline]
 debug_object_deactivate+0x268/0x300 lib/debugobjects.c:762
 debug_hrtimer_deactivate kernel/time/hrtimer.c:425 [inline]
 debug_deactivate kernel/time/hrtimer.c:481 [inline]
 __run_hrtimer kernel/time/hrtimer.c:1653 [inline]
 __hrtimer_run_queues+0x3f3/0xbe0 kernel/time/hrtimer.c:1749
 hrtimer_interrupt+0x320/0x7b0 kernel/time/hrtimer.c:1811
 local_apic_timer_interrupt arch/x86/kernel/apic/apic.c:1095 [inline]
 __sysvec_apic_timer_interrupt+0x14a/0x430 arch/x86/kernel/apic/apic.c:1112
 sysvec_apic_timer_interrupt+0x92/0xc0 arch/x86/kernel/apic/apic.c:1106
 </IRQ>
 <TASK>
 asm_sysvec_apic_timer_interrupt+0x1a/0x20 arch/x86/include/asm/idtentry.h:645
RIP: 0010:__raw_spin_unlock_irq include/linux/spinlock_api_smp.h:160 [inline]
RIP: 0010:_raw_spin_unlock_irq+0x29/0x50 kernel/locking/spinlock.c:202
Code: 90 f3 0f 1e fa 55 48 8b 74 24 08 48 89 fd 48 83 c7 18 e8 8a 9d 52 f7 48 89 ef e8 b2 0b 53 f7 e8 3d 97 77 f7 fb bf 01 00 00 00 <e8> 32 e1 44 f7 65 8b 05 53 73 f0 75 85 c0 74 02 5d c3 e8 20 41 ed
RSP: 0018:ffffc9000424f3f0 EFLAGS: 00000202
RAX: 00000000007d7813 RBX: ffff88801ede4d00 RCX: 1ffffffff22b61ae
RDX: 0000000000000000 RSI: 0000000000000001 RDI: 0000000000000001
RBP: ffff88801ed1e8c0 R08: 0000000000000001 R09: ffffffff91528e6f
R10: 0000000000000001 R11: 0000000000000001 R12: 0000000000000000
R13: ffff88807b5be660 R14: ffff88801ed1e800 R15: 0000000000000001
 spin_unlock_irq include/linux/spinlock.h:400 [inline]
 loop_queue_work drivers/block/loop.c:898 [inline]
 loop_queue_rq+0x627/0x1260 drivers/block/loop.c:1855
 __blk_mq_issue_directly+0xd1/0x260 block/blk-mq.c:2572
 blk_mq_try_issue_directly+0x187/0x360 block/blk-mq.c:2631
 blk_mq_submit_bio+0x1671/0x1f50 block/blk-mq.c:2989
 __submit_bio+0xfc/0x310 block/blk-core.c:594
 __submit_bio_noacct_mq block/blk-core.c:673 [inline]
 submit_bio_noacct_nocheck+0x7f9/0xb40 block/blk-core.c:702
 submit_bio_noacct+0x945/0x19f0 block/blk-core.c:801
 __submit_merged_bio+0x2fc/0x630 fs/f2fs/data.c:578
 __f2fs_submit_merged_write fs/f2fs/data.c:666 [inline]
 __submit_merged_write_cond+0x185/0x3c0 fs/f2fs/data.c:688
 f2fs_sync_node_pages+0x1ec7/0x21c0 fs/f2fs/node.c:2051
 block_operations+0x8ae/0xe80 fs/f2fs/checkpoint.c:1272
 f2fs_write_checkpoint+0x5fa/0x4b40 fs/f2fs/checkpoint.c:1651
 kill_f2fs_super+0x32c/0x3c0 fs/f2fs/super.c:4696
 deactivate_locked_super+0x98/0x160 fs/super.c:331
 deactivate_super+0xb1/0xd0 fs/super.c:362
 cleanup_mnt+0x2ae/0x3d0 fs/namespace.c:1177
 task_work_run+0x16f/0x270 kernel/task_work.c:179
 resume_user_mode_work include/linux/resume_user_mode.h:49 [inline]
 exit_to_user_mode_loop kernel/entry/common.c:171 [inline]
 exit_to_user_mode_prepare+0x210/0x240 kernel/entry/common.c:204
 __syscall_exit_to_user_mode_work kernel/entry/common.c:286 [inline]
 syscall_exit_to_user_mode+0x1d/0x50 kernel/entry/common.c:297
 do_syscall_64+0x46/0xb0 arch/x86/entry/common.c:86
 entry_SYSCALL_64_after_hwframe+0x63/0xcd
RIP: 0033:0x7f3a7048d5d7
Code: ff ff ff f7 d8 64 89 01 48 83 c8 ff c3 66 0f 1f 44 00 00 31 f6 e9 09 00 00 00 66 0f 1f 84 00 00 00 00 00 b8 a6 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 c7 c1 b8 ff ff ff f7 d8 64 89 01 48
RSP: 002b:00007ffe98198058 EFLAGS: 00000246 ORIG_RAX: 00000000000000a6
RAX: 0000000000000000 RBX: 0000000000000000 RCX: 00007f3a7048d5d7
RDX: 00007ffe9819812b RSI: 000000000000000a RDI: 00007ffe98198120
RBP: 00007ffe98198120 R08: 00000000ffffffff R09: 00007ffe98197ef0
R10: 000055555670a893 R11: 0000000000000246 R12: 00007f3a704e6cdc
R13: 00007ffe981991e0 R14: 000055555670a810 R15: 00007ffe98199220
 </TASK>
rcu: rcu_preempt kthread timer wakeup didn't happen for 10498 jiffies! g76513 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x402
rcu: 	Possible timer handling issue on cpu=1 timer-softirq=45761
rcu: rcu_preempt kthread starved for 10499 jiffies! g76513 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x402 ->cpu=1
rcu: 	Unless rcu_preempt kthread gets sufficient CPU time, OOM is now expected behavior.
rcu: RCU grace-period kthread stack dump:
task:rcu_preempt     state:I stack:28408 pid:16    ppid:2      flags:0x00004000
Call Trace:
 <TASK>
 context_switch kernel/sched/core.c:5343 [inline]
 __schedule+0xc9a/0x5880 kernel/sched/core.c:6669
 schedule+0xde/0x1a0 kernel/sched/core.c:6745
 schedule_timeout+0x14e/0x2b0 kernel/time/timer.c:2167
 rcu_gp_fqs_loop+0x190/0x910 kernel/rcu/tree.c:1609
 rcu_gp_kthread+0x23a/0x360 kernel/rcu/tree.c:1808
 kthread+0x344/0x440 kernel/kthread.c:379
 ret_from_fork+0x1f/0x30 arch/x86/entry/entry_64.S:308
 </TASK>
rcu: Stack dump where RCU GP kthread last ran:
CPU: 1 PID: 17655 Comm: kworker/u4:48 Not tainted 6.4.0-rc4-syzkaller-00047-gafead42fdfca #0
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 05/25/2023
Workqueue: events_unbound toggle_allocation_gate
RIP: 0010:csd_lock_wait kernel/smp.c:285 [inline]
RIP: 0010:smp_call_function_many_cond+0x684/0x15a0 kernel/smp.c:828
Code: 0b 00 85 ed 74 48 48 8b 44 24 08 49 89 c4 83 e0 07 49 c1 ec 03 48 89 c5 4d 01 f4 83 c5 03 e8 b3 40 0b 00 f3 90 41 0f b6 04 24 <40> 38 c5 7c 08 84 c0 0f 85 ad 0c 00 00 8b 43 08 31 ff 83 e0 01 41
RSP: 0018:ffffc9000b357948 EFLAGS: 00000293
RAX: 0000000000000000 RBX: ffff8880b98451e0 RCX: 0000000000000000
RDX: ffff8880282e1dc0 RSI: ffffffff8179036d RDI: 0000000000000005
RBP: 0000000000000003 R08: 0000000000000005 R09: 0000000000000000
R10: 0000000000000001 R11: 0000000000000001 R12: ffffed1017308a3d
R13: 0000000000000000 R14: dffffc0000000000 R15: 0000000000000001
FS:  0000000000000000(0000) GS:ffff8880b9900000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f42421a8000 CR3: 000000000c571000 CR4: 0000000000350ee0
Call Trace:
 <IRQ>
 </IRQ>
 <TASK>
 on_each_cpu_cond_mask+0x5a/0xa0 kernel/smp.c:996
 on_each_cpu include/linux/smp.h:71 [inline]
 text_poke_sync arch/x86/kernel/alternative.c:1770 [inline]
 text_poke_bp_batch+0x237/0x770 arch/x86/kernel/alternative.c:1970
 text_poke_flush arch/x86/kernel/alternative.c:2161 [inline]
 text_poke_flush arch/x86/kernel/alternative.c:2158 [inline]
 text_poke_finish+0x1a/0x30 arch/x86/kernel/alternative.c:2168
 arch_jump_label_transform_apply+0x17/0x30 arch/x86/kernel/jump_label.c:146
 jump_label_update+0x32f/0x410 kernel/jump_label.c:829
 static_key_enable_cpuslocked+0x1b5/0x270 kernel/jump_label.c:205
 static_key_enable+0x1a/0x20 kernel/jump_label.c:218
 toggle_allocation_gate mm/kfence/core.c:831 [inline]
 toggle_allocation_gate+0xf8/0x230 mm/kfence/core.c:823
 process_one_work+0x99a/0x15e0 kernel/workqueue.c:2405
 worker_thread+0x67d/0x10c0 kernel/workqueue.c:2552
 kthread+0x344/0x440 kernel/kthread.c:379
 ret_from_fork+0x1f/0x30 arch/x86/entry/entry_64.S:308
 </TASK>

Crashes (2):
Time Kernel Commit Syzkaller Config Log Report Syz repro C repro VM info Assets (help?) Manager Title
2023/05/31 03:26 upstream afead42fdfca 09898419 .config console log report info [disk image] [vmlinux] [kernel image] ci-upstream-kasan-gce-root INFO: rcu detected stall in cleanup_mnt
2023/05/20 16:19 upstream d635f6cc934b 4bce1a3e .config console log report info ci-upstream-kasan-gce-smack-root INFO: rcu detected stall in cleanup_mnt
* Struck through repros no longer work on HEAD.