INFO: rcu detected stall in kvm_vcpu

Kernel	Title	Rank 🛈	Repro	Cause bisect	Fix bisect	Count	Last	Reported	Patched	Status
upstream	INFO: rcu detected stall in kvm_vcpu_release (2) fs	1				1	442d	442d	0/29	auto-obsoleted due to no activity on 2024/09/05 03:45
linux-5.15	INFO: rcu detected stall in kvm_vcpu_release	1				1	455d	455d	0/3	auto-obsoleted due to no activity on 2024/09/02 00:50

rcu: INFO: rcu_preempt self-detected stall on CPU rcu: 0-...!: (10499 ticks this GP) idle=372/1/0x4000000000000002 softirq=218219/218219 fqs=0 rcu: (t=10500 jiffies g=264473 q=294) rcu: rcu_preempt kthread starved for 10500 jiffies! g264473 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x402 ->cpu=1 rcu: RCU grace-period kthread stack dump: rcu_preempt I29168 10 2 0x80000000 Call Trace: context_switch kernel/sched/core.c:2877 [inline] __schedule+0x813/0x1cc0 kernel/sched/core.c:3518 schedule+0x92/0x180 kernel/sched/core.c:3562 schedule_timeout+0x4db/0xfd0 kernel/time/timer.c:1803 rcu_gp_fqs_loop kernel/rcu/tree.c:1971 [inline] rcu_gp_kthread+0x962/0x17b0 kernel/rcu/tree.c:2128 kthread+0x357/0x430 kernel/kthread.c:253 ret_from_fork+0x3a/0x50 arch/x86/entry/entry_64.S:352 NMI backtrace for cpu 0 CPU: 0 PID: 16439 Comm: syz-executor.1 Not tainted 5.1.0-rc6+ #87 Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 01/01/2011 Call Trace: <IRQ> __dump_stack lib/dump_stack.c:77 [inline] dump_stack+0x172/0x1f0 lib/dump_stack.c:113 nmi_cpu_backtrace.cold+0x63/0xa4 lib/nmi_backtrace.c:101 nmi_trigger_cpumask_backtrace+0x1be/0x236 lib/nmi_backtrace.c:62 arch_trigger_cpumask_backtrace+0x14/0x20 arch/x86/kernel/apic/hw_nmi.c:38 trigger_single_cpu_backtrace include/linux/nmi.h:164 [inline] rcu_dump_cpu_stacks+0x183/0x1cf kernel/rcu/tree.c:1223 print_cpu_stall kernel/rcu/tree.c:1360 [inline] check_cpu_stall kernel/rcu/tree.c:1434 [inline] rcu_pending kernel/rcu/tree.c:3103 [inline] rcu_sched_clock_irq.cold+0x500/0xa4a kernel/rcu/tree.c:2544 update_process_times+0x32/0x80 kernel/time/timer.c:1635 tick_sched_handle+0xa2/0x190 kernel/time/tick-sched.c:161 tick_sched_timer+0x47/0x130 kernel/time/tick-sched.c:1271 __run_hrtimer kernel/time/hrtimer.c:1389 [inline] __hrtimer_run_queues+0x33e/0xde0 kernel/time/hrtimer.c:1451 hrtimer_interrupt+0x314/0x770 kernel/time/hrtimer.c:1509 local_apic_timer_interrupt arch/x86/kernel/apic/apic.c:1035 [inline] smp_apic_timer_interrupt+0x120/0x570 arch/x86/kernel/apic/apic.c:1060 apic_timer_interrupt+0xf/0x20 arch/x86/entry/entry_64.S:807 </IRQ> RIP: 0010:__read_once_size include/linux/compiler.h:193 [inline] RIP: 0010:csd_lock_wait kernel/smp.c:108 [inline] RIP: 0010:smp_call_function_single+0x139/0x420 kernel/smp.c:302 Code: 00 e8 0b 88 0a 00 48 8b 4c 24 08 48 8b 54 24 10 48 8d 74 24 40 8b 7c 24 1c e8 23 fa ff ff 41 89 c5 eb 07 e8 e9 87 0a 00 f3 90 <44> 8b 64 24 58 31 ff 41 83 e4 01 44 89 e6 e8 54 89 0a 00 45 85 e4 RSP: 0018:ffff888090d6f860 EFLAGS: 00000293 ORIG_RAX: ffffffffffffff13 RAX: ffff8880a5226000 RBX: 1ffff110121adf10 RCX: ffffffff8166051c RDX: 0000000000000000 RSI: ffffffff81660507 RDI: 0000000000000005 RBP: ffff888090d6f938 R08: ffff8880a5226000 R09: ffffed1015d25be9 R10: ffffed1015d25be8 R11: ffff8880ae92df47 R12: 0000000000000001 R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000000 smp_call_function_many+0x750/0x8c0 kernel/smp.c:434 smp_call_function+0x42/0x90 kernel/smp.c:492 on_each_cpu+0x31/0x200 kernel/smp.c:602 text_poke_bp+0xde/0x19b arch/x86/kernel/alternative.c:809 __jump_label_transform+0x263/0x330 arch/x86/kernel/jump_label.c:91 arch_jump_label_transform+0x2b/0x40 arch/x86/kernel/jump_label.c:99 __jump_label_update+0x16a/0x210 kernel/jump_label.c:389 jump_label_update kernel/jump_label.c:752 [inline] jump_label_update+0x1ce/0x3d0 kernel/jump_label.c:731 __static_key_slow_dec_cpuslocked+0x61/0x150 kernel/jump_label.c:228 __static_key_slow_dec kernel/jump_label.c:238 [inline] static_key_slow_dec+0x60/0xa0 kernel/jump_label.c:252 kvm_arch_vcpu_uninit+0x1b0/0x1f0 arch/x86/kvm/x86.c:9127 kvm_vcpu_uninit+0x44/0x90 arch/x86/kvm/../../../virt/kvm/kvm_main.c:340 vmx_free_vcpu+0x16d/0x280 arch/x86/kvm/vmx/vmx.c:6551 kvm_arch_vcpu_free arch/x86/kvm/x86.c:8741 [inline] kvm_free_vcpus arch/x86/kvm/x86.c:9194 [inline] kvm_arch_destroy_vm+0x2ce/0x580 arch/x86/kvm/x86.c:9291 kvm_destroy_vm arch/x86/kvm/../../../virt/kvm/kvm_main.c:749 [inline] kvm_put_kvm+0x55b/0xc70 arch/x86/kvm/../../../virt/kvm/kvm_main.c:770 kvm_vcpu_release+0x7b/0xa0 arch/x86/kvm/../../../virt/kvm/kvm_main.c:2500 __fput+0x2e5/0x8d0 fs/file_table.c:278 ____fput+0x16/0x20 fs/file_table.c:309 task_work_run+0x14a/0x1c0 kernel/task_work.c:113 tracehook_notify_resume include/linux/tracehook.h:188 [inline] exit_to_usermode_loop+0x273/0x2c0 arch/x86/entry/common.c:166 prepare_exit_to_usermode arch/x86/entry/common.c:197 [inline] syscall_return_slowpath arch/x86/entry/common.c:268 [inline] do_syscall_64+0x52d/0x610 arch/x86/entry/common.c:293 entry_SYSCALL_64_after_hwframe+0x49/0xbe RIP: 0033:0x412b61 Code: 75 14 b8 03 00 00 00 0f 05 48 3d 01 f0 ff ff 0f 83 e4 1a 00 00 c3 48 83 ec 08 e8 0a fc ff ff 48 89 04 24 b8 03 00 00 00 0f 05 <48> 8b 3c 24 48 89 c2 e8 53 fc ff ff 48 89 d0 48 83 c4 08 48 3d 01 RSP: 002b:00007fff7aefba10 EFLAGS: 00000293 ORIG_RAX: 0000000000000003 RAX: 0000000000000000 RBX: 0000000000000007 RCX: 0000000000412b61 RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000006 RBP: 0000000000000001 R08: 00000000b334236b R09: 00000000b334236f R10: 00007fff7aefbaf0 R11: 0000000000000293 R12: 00000000007410b0 R13: 00000000001634f6 R14: 0000000000163523 R15: 000000000073bf0c Sending NMI from CPU 0 to CPUs 1: NMI backtrace for cpu 1 CPU: 1 PID: 5026 Comm: kworker/1:0 Not tainted 5.1.0-rc6+ #87 Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 01/01/2011 Workqueue: rcu_gp process_srcu RIP: 0010:cpu_relax arch/x86/include/asm/processor.h:666 [inline] RIP: 0010:virt_spin_lock arch/x86/include/asm/qspinlock.h:84 [inline] RIP: 0010:native_queued_spin_lock_slowpath+0x132/0x9f0 kernel/locking/qspinlock.c:334 Code: 00 00 00 48 8b 45 d0 65 48 33 04 25 28 00 00 00 0f 85 37 07 00 00 48 81 c4 98 00 00 00 5b 41 5c 41 5d 41 5e 41 5f 5d c3 f3 90 <e9> 73 ff ff ff 8b 45 98 4c 8d 65 d8 3d 00 01 00 00 0f 84 e5 00 00 RSP: 0018:ffff8880ae907c78 EFLAGS: 00000002 RAX: 0000000000000000 RBX: ffffffff889a8200 RCX: ffffffff81583507 RDX: 0000000000000000 RSI: 0000000000000004 RDI: ffffffff889a8200 RBP: ffff8880ae907d38 R08: 1ffffffff1135040 R09: fffffbfff1135041 R10: fffffbfff1135040 R11: ffffffff889a8203 R12: 0000000000000001 R13: 0000000000000003 R14: fffffbfff1135040 R15: 0000000000000001 FS: 0000000000000000(0000) GS:ffff8880ae900000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 0000000000000000 CR3: 000000008b621000 CR4: 00000000001426e0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 Call Trace: <IRQ> pv_queued_spin_lock_slowpath arch/x86/include/asm/paravirt.h:654 [inline] queued_spin_lock_slowpath arch/x86/include/asm/qspinlock.h:50 [inline] queued_spin_lock include/asm-generic/qspinlock.h:90 [inline] do_raw_spin_lock+0x20e/0x2e0 kernel/locking/spinlock_debug.c:113 __raw_spin_lock_irqsave include/linux/spinlock_api_smp.h:117 [inline] _raw_spin_lock_irqsave+0x9d/0xcd kernel/locking/spinlock.c:152 rcu_report_qs_rdp kernel/rcu/tree.c:2283 [inline] rcu_check_quiescent_state kernel/rcu/tree.c:2346 [inline] rcu_core+0x1d8/0x13a0 kernel/rcu/tree.c:2754 __do_softirq+0x266/0x95a kernel/softirq.c:293 invoke_softirq kernel/softirq.c:374 [inline] irq_exit+0x180/0x1d0 kernel/softirq.c:414 exiting_irq arch/x86/include/asm/apic.h:536 [inline] smp_apic_timer_interrupt+0x14a/0x570 arch/x86/kernel/apic/apic.c:1062 apic_timer_interrupt+0xf/0x20 arch/x86/entry/entry_64.S:807 </IRQ> RIP: 0010:preempt_schedule_irq+0xab/0x140 kernel/sched/core.c:3745 Code: 00 00 e8 08 5a 3e fa e8 c3 60 66 fa 4c 89 f0 48 c1 e8 03 80 3c 18 00 75 6c 48 83 3d 6e fe 82 01 00 74 5e fb 66 0f 1f 44 00 00 <bf> 01 00 00 00 e8 cb d5 ff ff 48 c7 c0 78 97 92 88 48 c1 e8 03 80 RSP: 0018:ffff888051c6fb08 EFLAGS: 00000282 ORIG_RAX: ffffffffffffff13 RAX: 1ffffffff11252f0 RBX: dffffc0000000000 RCX: 1ffff11012099171 RDX: 0000000000000000 RSI: ffff8880904c8b68 RDI: ffff8880904c8b3c RBP: ffff888051c6fb28 R08: ffff8880904c82c0 R09: ffff8880904c8b88 R10: 0000000000000000 R11: 0000000000000000 R12: ffffed1012099058 R13: ffff8880904c82c0 R14: ffffffff88929780 R15: 0000000000000000 retint_kernel+0x1b/0x2d RIP: 0010:arch_local_irq_restore arch/x86/include/asm/paravirt.h:767 [inline] RIP: 0010:queue_delayed_work_on+0x11d/0x200 kernel/workqueue.c:1678 Code: 00 fc ff df 48 c1 e8 03 80 3c 10 00 0f 85 d8 00 00 00 48 83 3d f3 b2 47 07 00 0f 84 9a 00 00 00 e8 68 a8 25 00 48 89 df 57 9d <0f> 1f 44 00 00 e8 59 a8 25 00 44 89 f8 48 83 c4 10 5b 41 5c 41 5d RSP: 0018:ffff888051c6fbe0 EFLAGS: 00000293 ORIG_RAX: ffffffffffffff02 RAX: ffff8880904c82c0 RBX: 0000000000000293 RCX: 1ffff11012099171 RDX: 0000000000000000 RSI: ffffffff814ae488 RDI: 0000000000000293 RBP: ffff888051c6fc18 R08: ffff8880904c82c0 R09: ffff8880904c8b88 R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000200 R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000001 queue_delayed_work include/linux/workqueue.h:509 [inline] srcu_reschedule+0xd7/0x130 kernel/rcu/srcutree.c:1229 process_srcu+0x125/0xe20 kernel/rcu/srcutree.c:1242 process_one_work+0x98e/0x1790 kernel/workqueue.c:2269 worker_thread+0x98/0xe40 kernel/workqueue.c:2415 kthread+0x357/0x430 kernel/kthread.c:253 ret_from_fork+0x3a/0x50 arch/x86/entry/entry_64.S:352 protocol 88fb is buggy, dev hsr_slave_0 protocol 88fb is buggy, dev hsr_slave_1 protocol 88fb is buggy, dev hsr_slave_0 protocol 88fb is buggy, dev hsr_slave_1 protocol 88fb is buggy, dev hsr_slave_0 protocol 88fb is buggy, dev hsr_slave_1