INFO: rcu detected stall in br_hello_timer_expired

Status: fixed on 2019/10/09 10:54
Subsystems: net
[Documentation on labels]
Fix commit: d4d6ec6dac07 sch_hhf: ensure quantum and hhf_non_hh_weight are non-zero
First crash: 2518d, last: 2513d
▶ ▼ Similar bugs (1)
Kernel Title Rank 🛈 Repro Cause bisect Fix bisect Count Last Reported Patched Status
linux-4.14 INFO: rcu detected stall in br_hello_timer_expired 1 1 2374d 2374d 0/1 auto-closed as invalid on 2020/05/31 22:08

Sample crash report:
rcu: INFO: rcu_preempt self-detected stall on CPU
rcu: 	0-....: (1 GPs behind) idle=82a/1/0x4000000000000004 softirq=213360/213361 fqs=3235 
	(t=10500 jiffies g=378649 q=3561)
rcu: rcu_preempt kthread starved for 3934 jiffies! g378649 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x402 ->cpu=1
rcu: RCU grace-period kthread stack dump:
rcu_preempt     I29160    10      2 0x80004000
Call Trace:
 context_switch kernel/sched/core.c:3254 [inline]
 __schedule+0x755/0x1580 kernel/sched/core.c:3880
 schedule+0xd9/0x260 kernel/sched/core.c:3947
 schedule_timeout+0x486/0xc50 kernel/time/timer.c:1807
 rcu_gp_fqs_loop kernel/rcu/tree.c:1611 [inline]
 rcu_gp_kthread+0x9b2/0x18c0 kernel/rcu/tree.c:1768
 kthread+0x361/0x430 kernel/kthread.c:255
 ret_from_fork+0x24/0x30 arch/x86/entry/entry_64.S:352
NMI backtrace for cpu 0
CPU: 0 PID: 5108 Comm: syz-executor.1 Not tainted 5.3.0-rc8+ #0
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 01/01/2011
Call Trace:
 <IRQ>
 __dump_stack lib/dump_stack.c:77 [inline]
 dump_stack+0x172/0x1f0 lib/dump_stack.c:113
 nmi_cpu_backtrace.cold+0x70/0xb2 lib/nmi_backtrace.c:101
 nmi_trigger_cpumask_backtrace+0x23b/0x28b lib/nmi_backtrace.c:62
 arch_trigger_cpumask_backtrace+0x14/0x20 arch/x86/kernel/apic/hw_nmi.c:38
 trigger_single_cpu_backtrace include/linux/nmi.h:164 [inline]
 rcu_dump_cpu_stacks+0x183/0x1cf kernel/rcu/tree_stall.h:254
 print_cpu_stall kernel/rcu/tree_stall.h:455 [inline]
 check_cpu_stall kernel/rcu/tree_stall.h:529 [inline]
 rcu_pending kernel/rcu/tree.c:2736 [inline]
 rcu_sched_clock_irq.cold+0x4dd/0xc13 kernel/rcu/tree.c:2183
 update_process_times+0x32/0x80 kernel/time/timer.c:1639
 tick_sched_handle+0xa2/0x190 kernel/time/tick-sched.c:167
 tick_sched_timer+0x53/0x140 kernel/time/tick-sched.c:1296
 __run_hrtimer kernel/time/hrtimer.c:1389 [inline]
 __hrtimer_run_queues+0x364/0xe40 kernel/time/hrtimer.c:1451
 hrtimer_interrupt+0x314/0x770 kernel/time/hrtimer.c:1509
 local_apic_timer_interrupt arch/x86/kernel/apic/apic.c:1110 [inline]
 smp_apic_timer_interrupt+0x160/0x610 arch/x86/kernel/apic/apic.c:1135
 apic_timer_interrupt+0xf/0x20 arch/x86/entry/entry_64.S:830
RIP: 0010:__sanitizer_cov_trace_const_cmp4+0x0/0x20 kernel/kcov.c:188
Code: f6 fe ff ff 5d c3 0f 1f 40 00 55 0f b7 d6 0f b7 f7 bf 03 00 00 00 48 89 e5 48 8b 4d 08 e8 d8 fe ff ff 5d c3 66 0f 1f 44 00 00 <55> 89 f2 89 fe bf 05 00 00 00 48 89 e5 48 8b 4d 08 e8 ba fe ff ff
RSP: 0018:ffff8880ae8096f0 EFLAGS: 00000246 ORIG_RAX: ffffffffffffff13
RAX: 0000000000000000 RBX: ffff8880a7ff8bf8 RCX: ffffffff85c660f9
RDX: 0000000000000100 RSI: 0000000000000000 RDI: 0000000000000000
RBP: ffff8880ae809748 R08: ffff888095dbe3c0 R09: 0000000000000000
R10: fffffbfff134afaf R11: ffff888095dbe3c0 R12: dffffc0000000000
R13: ffff8880a7ff8900 R14: ffff8880a7ff8c90 R15: 0000000000000000
 dequeue_skb net/sched/sch_generic.c:258 [inline]
 qdisc_restart net/sched/sch_generic.c:361 [inline]
 __qdisc_run+0x1e7/0x19d0 net/sched/sch_generic.c:379
 __dev_xmit_skb net/core/dev.c:3533 [inline]
 __dev_queue_xmit+0x16f1/0x3650 net/core/dev.c:3838
 dev_queue_xmit+0x18/0x20 net/core/dev.c:3902
 br_send_bpdu_finish net/bridge/br_stp_bpdu.c:32 [inline]
 NF_HOOK include/linux/netfilter.h:305 [inline]
 br_send_bpdu.isra.0.constprop.0+0x5ce/0xa70 net/bridge/br_stp_bpdu.c:59
 br_send_config_bpdu+0x68c/0x7a0 net/bridge/br_stp_bpdu.c:120
 br_transmit_config.part.0+0x517/0x780 net/bridge/br_stp.c:203
 br_transmit_config net/bridge/br_stp.c:364 [inline]
 br_config_bpdu_generation+0x1d2/0x230 net/bridge/br_stp.c:362
 br_hello_timer_expired+0xab/0x190 net/bridge/br_stp_timer.c:37
 call_timer_fn+0x1ac/0x780 kernel/time/timer.c:1322
 expire_timers kernel/time/timer.c:1366 [inline]
 __run_timers kernel/time/timer.c:1685 [inline]
 __run_timers kernel/time/timer.c:1653 [inline]
 run_timer_softirq+0x697/0x17a0 kernel/time/timer.c:1698
 __do_softirq+0x262/0x98c kernel/softirq.c:292
 invoke_softirq kernel/softirq.c:373 [inline]
 irq_exit+0x19b/0x1e0 kernel/softirq.c:413
 exiting_irq arch/x86/include/asm/apic.h:537 [inline]
 smp_apic_timer_interrupt+0x1a3/0x610 arch/x86/kernel/apic/apic.c:1137
 apic_timer_interrupt+0xf/0x20 arch/x86/entry/entry_64.S:830
 </IRQ>
RIP: 0010:__raw_spin_unlock_irq include/linux/spinlock_api_smp.h:169 [inline]
RIP: 0010:_raw_spin_unlock_irq+0x54/0x90 kernel/locking/spinlock.c:199
Code: c0 60 f4 d2 88 48 ba 00 00 00 00 00 fc ff df 48 c1 e8 03 80 3c 10 00 75 33 48 83 3d 75 ce 94 01 00 74 20 fb 66 0f 1f 44 00 00 <bf> 01 00 00 00 e8 d2 9b 10 fa 65 8b 05 03 d8 c3 78 85 c0 74 06 41
RSP: 0018:ffff8881fd1b7980 EFLAGS: 00000282 ORIG_RAX: ffffffffffffff13
RAX: 1ffffffff11a5e8c RBX: ffff888095dbe3c0 RCX: 1ffffffff134b5ee
RDX: dffffc0000000000 RSI: ffffffff8177f18e RDI: ffffffff873e25c8
RBP: ffff8881fd1b7988 R08: ffff888095dbe3c0 R09: ffffed1015d06ad1
R10: ffffed1015d06ad0 R11: ffff8880ae835683 R12: ffff8880ae835680
R13: ffff88802f0e8200 R14: 0000000000000000 R15: 0000000000000001
 finish_lock_switch kernel/sched/core.c:3004 [inline]
 finish_task_switch+0x147/0x720 kernel/sched/core.c:3104
 context_switch kernel/sched/core.c:3257 [inline]
 __schedule+0x75d/0x1580 kernel/sched/core.c:3880
 schedule+0xd9/0x260 kernel/sched/core.c:3947
 pipe_wait+0x11c/0x1f0 fs/pipe.c:117
 pipe_write+0x5fa/0xf40 fs/pipe.c:497
 call_write_iter include/linux/fs.h:1870 [inline]
 new_sync_write+0x4d3/0x770 fs/read_write.c:483
 __vfs_write+0xe1/0x110 fs/read_write.c:496
 vfs_write+0x268/0x5d0 fs/read_write.c:558
 ksys_write+0x14f/0x290 fs/read_write.c:611
 __do_sys_write fs/read_write.c:623 [inline]
 __se_sys_write fs/read_write.c:620 [inline]
 __x64_sys_write+0x73/0xb0 fs/read_write.c:620
 do_syscall_64+0xfd/0x6a0 arch/x86/entry/common.c:296
 entry_SYSCALL_64_after_hwframe+0x49/0xbe
RIP: 0033:0x4598e9
Code: fd b7 fb ff c3 66 2e 0f 1f 84 00 00 00 00 00 66 90 48 89 f8 48 89 f7 48 89 d6 48 89 ca 4d 89 c2 4d 89 c8 4c 8b 4c 24 08 0f 05 <48> 3d 01 f0 ff ff 0f 83 cb b7 fb ff c3 66 2e 0f 1f 84 00 00 00 00
RSP: 002b:00007f8b36c2dc78 EFLAGS: 00000246 ORIG_RAX: 0000000000000001
RAX: ffffffffffffffda RBX: 0000000000000003 RCX: 00000000004598e9
RDX: 0000000041395527 RSI: 0000000020000340 RDI: 0000000000000005
RBP: 000000000075bf20 R08: 0000000000000000 R09: 0000000000000000
R10: 0000000000000000 R11: 0000000000000246 R12: 00007f8b36c2e6d4
R13: 00000000004c5e50 R14: 00000000004e0380 R15: 00000000ffffffff

Crashes (4):
Time Kernel Commit Syzkaller Config Log Report Syz repro C repro VM info Assets (help?) Manager Title
2019/09/13 08:03 upstream 505a8ec7e11a 40fa42bc .config console log report ci-upstream-kasan-gce
2019/09/16 02:59 net-next-old a3d3c74da49c 32d59357 .config console log report ci-upstream-net-kasan-gce
2019/09/14 09:14 net-next-old 1ba569fc2250 32d59357 .config console log report ci-upstream-net-kasan-gce
2019/09/10 17:29 net-next-old db63864786c7 a60cb4cd .config console log report ci-upstream-net-kasan-gce
* Struck through repros no longer work on HEAD.