INFO: rcu detected stall in write_comp_data

Status: closed as invalid on 2019/08/23 00:21
Subsystems: fs
[Documentation on labels]
Reported-by: syzbot+de50612027760b3a87aa@syzkaller.appspotmail.com
First crash: 2707d, last: 2707d
Cause bisection: the cause commit could be any of (bisect log):
commit 2253efc850c4cf690516bbc07854eeb1077202ba
Author: Bart Van Assche <bart.vanassche@sandisk.com>
Date: Sat Oct 29 00:20:02 2016 +0000

  blk-mq: Move more code into blk_mq_direct_issue_request()

commit 52d7f1b5c2f33b5d34dc2b6af5175fb6a44999f6
Author: Bart Van Assche <bart.vanassche@sandisk.com>
Date: Sat Oct 29 00:20:32 2016 +0000

  blk-mq: Avoid that requeueing starts stopped queues

commit fd00144301d64f1742541a3c5e64cd1c51f39c55
Author: Bart Van Assche <bart.vanassche@sandisk.com>
Date: Sat Oct 29 00:19:37 2016 +0000

  blk-mq: Introduce blk_mq_queue_stopped()

commit 5d1b25c1ecabb37f8eb58c8e9dd74f77f703e5d9
Author: Bart Van Assche <bart.vanassche@sandisk.com>
Date: Sat Oct 29 00:19:15 2016 +0000

  blk-mq: Introduce blk_mq_hctx_stopped()

commit 9b7dd572cc439fa92e120290eb74d0295567c5a0
Author: Bart Van Assche <bart.vanassche@sandisk.com>
Date: Sat Oct 29 00:20:49 2016 +0000

  blk-mq: Remove blk_mq_cancel_requeue_work()

commit 6a83e74d214a47a1371cd2e6a783264fcba7d428
Author: Bart Van Assche <bart.vanassche@sandisk.com>
Date: Wed Nov 2 16:09:51 2016 +0000

  blk-mq: Introduce blk_mq_quiesce_queue()

commit bc27c01b5c46d3bfec42c96537c7a3fae0bb2cc4
Author: Bart Van Assche <bart.vanassche@sandisk.com>
Date: Sat Oct 29 00:18:48 2016 +0000

  blk-mq: Do not invoke .queue_rq() for a stopped queue

commit 2b053aca76b48e681be57b34ca3a8c2c10b275c5
Author: Bart Van Assche <bart.vanassche@sandisk.com>
Date: Sat Oct 29 00:21:41 2016 +0000

  blk-mq: Add a kick_requeue_list argument to blk_mq_requeue_request()

commit 2cefe4dbaadf83b236caab46705b4b5a4958e3b6
Author: Kent Overstreet <kent.overstreet@gmail.com>
Date: Mon Oct 31 17:59:24 2016 +0000

  block: add bio_iov_iter_get_pages()

commit 13edd5e7315a26b448c5f7f33fc7721b1e0c17ef
Author: Jens Axboe <axboe@fb.com>
Date: Tue Nov 1 16:01:35 2016 +0000

  writeback: mark background writeback as such

commit f0d33ab76cfcd696328c1330e2462b5d62e74f00
Author: Bart Van Assche <bart.vanassche@sandisk.com>
Date: Sat Oct 29 00:22:00 2016 +0000

  dm: Use BLK_MQ_S_STOPPED instead of QUEUE_FLAG_STOPPED in blk-mq code

commit 7637241e651ec36e409412869f986dd5f097735f
Author: Jens Axboe <axboe@fb.com>
Date: Tue Nov 1 16:00:38 2016 +0000

  writeback: add wbc_to_write_flags()

commit 7b17c2f7292ba1f3f98dae3f7077f9e569653276
Author: Bart Van Assche <bart.vanassche@sandisk.com>
Date: Sat Oct 29 00:22:16 2016 +0000

  dm: Fix a race condition related to stopping and starting queues

commit 1d796d6a9641fbfcd90fcfaf6fb4894a13d0304f
Author: Jens Axboe <axboe@fb.com>
Date: Tue Nov 1 15:52:57 2016 +0000

  block: add REQ_BACKGROUND

commit 3174dd33fa8316d29f238f093291ca9bb25e6f58
Author: Bart Van Assche <bart.vanassche@sandisk.com>
Date: Sat Oct 29 00:23:19 2016 +0000

  nvme: Fix a race condition related to stopping queues

commit 7281b4526cefc898d180850b54d1369f38c6b202
Author: Christoph Hellwig <hch@lst.de>
Date: Tue Nov 1 13:40:17 2016 +0000

  block: remove the CONFIG_BLOCK ifdef in blk_types.h

commit a6eaa8849f92a40f8894d6d0dcab2c16243aaf20
Author: Bart Van Assche <bart.vanassche@sandisk.com>
Date: Sat Oct 29 00:23:40 2016 +0000

  nvme: Use BLK_MQ_S_STOPPED instead of QUEUE_FLAG_STOPPED in blk-mq code

commit 46f3cc176210b2db5db89d9c0e62796cad7e3cdc
Author: Johannes Thumshirn <jthumshirn@suse.de>
Date: Thu Nov 3 09:38:54 2016 +0000

  block: drop q argument from bsg_validate_sgv4_hdr

commit be297968da22cf40c9c419df51e71ba8856a2ec2
Author: Christoph Hellwig <hch@lst.de>
Date: Tue Nov 1 13:40:16 2016 +0000

  mm: only include blk_types in swap.h if CONFIG_SWAP is enabled

commit 50d24c34403c62ad29e8b6db559d491bae20b4b7
Author: Shaohua Li <shli@fb.com>
Date: Fri Nov 4 00:03:53 2016 +0000

  block: immediately dispatch big size request

commit 9f08217120568afdfb59973a89a675e649c0096d
Author: Christoph Hellwig <hch@lst.de>
Date: Tue Nov 1 13:40:15 2016 +0000

  ceph: don't include blk_types.h in messenger.h

commit 600271d9000027c013c01be87cbb90a5a18c5c3f
Author: Shaohua Li <shli@fb.com>
Date: Fri Nov 4 00:03:54 2016 +0000

  blk-mq: immediately dispatch big size request

commit fe8ecc86fa83e6dca25b055651811dff8d80bfed
Author: Christoph Hellwig <hch@lst.de>
Date: Tue Nov 1 13:40:14 2016 +0000

  arm, arm64: don't include blk_types.h in <asm/io.h>

commit 2f8b544477e627a42e66902e948d87f86554aeca
Author: Christoph Hellwig <hch@lst.de>
Date: Tue Nov 1 13:40:13 2016 +0000

  block,fs: untangle fs.h and blk_types.h

commit d278d4a8892f13b6a9eb6102b356402f0e062324
Author: Jens Axboe <axboe@fb.com>
Date: Wed Mar 30 16:21:08 2016 +0000

  block: add code to track actual device queue depth

commit 1e3914d4cf4e14653b7917b0e965217465cb7a9c
Author: Christoph Hellwig <hch@lst.de>
Date: Tue Nov 1 13:40:12 2016 +0000

  block, fs: move submit_bio to bio.h

commit c02ebfdddbafa9a6a0f52fbd715e6bfa229af9d3
Author: Gabriel Krisman Bertazi <krisman@linux.vnet.ibm.com>
Date: Wed Sep 28 03:24:24 2016 +0000

  blk-mq: Always schedule hctx->next_cpu

commit d38499530e5f170d30f32d3841fade204e63081d
Author: Christoph Hellwig <hch@lst.de>
Date: Tue Nov 1 13:40:11 2016 +0000

  fs: decouple READ and WRITE from the block layer ops

commit feebd5687257750bb75eaf5188b93756994b8c41
Author: Christoph Hellwig <hch@lst.de>
Date: Thu Oct 27 07:04:44 2016 +0000

  pktcdvd: don't scribble over the bvec array

commit 180b0ae77d49dd63183b18609f3601430582c403
Author: Christoph Hellwig <hch@lst.de>
Date: Mon Nov 7 19:14:07 2016 +0000

  skd: use pci_alloc_irq_vectors

commit 70fd76140a6cb63262bd47b68d57b42e889c10ee
Author: Christoph Hellwig <hch@lst.de>
Date: Tue Nov 1 13:40:10 2016 +0000

  block,fs: use REQ_* flags directly

commit 8e1de26cd5db192b52819f5ee97bab308964f3ab
Author: Andy Shevchenko <andriy.shevchenko@linux.intel.com>
Date: Sat Oct 22 16:52:01 2016 +0000

  skd_main: use %*ph to dump small buffers

commit a2b809672ee6fcb4d5756ea815725b3dbaea654e
Author: Christoph Hellwig <hch@lst.de>
Date: Tue Nov 1 13:40:09 2016 +0000

  block: replace REQ_NOIDLE with REQ_IDLE

commit b57d74aff9ab92fbfb7c197c384d1adfa2827b2e
Author: Jens Axboe <axboe@fb.com>
Date: Thu Sep 1 16:20:33 2016 +0000

  writeback: track if we're sleeping on progress in balance_dirty_pages()

commit b685d3d65ac791406e0dfd8779cc9b3707fea5a3
Author: Christoph Hellwig <hch@lst.de>
Date: Tue Nov 1 13:40:08 2016 +0000

  block: treat REQ_FUA and REQ_PREFLUSH as synchronous

commit ae5b2ec8ad5e017126cd4552220f25ce8a6b92e9
Author: Jens Axboe <axboe@fb.com>
Date: Wed Nov 9 02:39:28 2016 +0000

  block: set REQ_SYNC if we clear REQ_FUA|REQ_PREFLUSH
  
▶ ▼ Discussions (1)
Title Replies (including bot) Last reply
INFO: rcu detected stall in write_comp_data 0 (1) 2019/06/24 07:27
▶ ▼ Similar bugs (2)
Kernel Title Rank 🛈 Repro Cause bisect Fix bisect Count Last Reported Patched Status
upstream INFO: rcu detected stall in write_comp_data (2) kernel 1 2 839d 843d 0/29 auto-obsoleted due to no activity on 2024/10/31 20:16
linux-4.19 INFO: rcu detected stall in write_comp_data 1 1 2398d 2398d 0/1 auto-closed as invalid on 2020/08/25 07:05

Sample crash report:
rcu: INFO: rcu_preempt detected stalls on CPUs/tasks:
	(detected by 1, t=10502 jiffies, g=7925, q=38)
rcu: All QSes seen, last rcu_preempt kthread activity 10503 (4295059525-4295049022), jiffies_till_next_fqs=1, root ->qsmask 0x0
syz-executor151 R  running task    26832  8885   8881 0x00004000
Call Trace:
 <IRQ>
 sched_show_task kernel/sched/core.c:5286 [inline]
 sched_show_task.cold+0x291/0x2fc kernel/sched/core.c:5261
 print_other_cpu_stall kernel/rcu/tree_stall.h:410 [inline]
 check_cpu_stall kernel/rcu/tree_stall.h:536 [inline]
 rcu_pending kernel/rcu/tree.c:2625 [inline]
 rcu_sched_clock_irq.cold+0xaaf/0xbfd kernel/rcu/tree.c:2161
 update_process_times+0x32/0x80 kernel/time/timer.c:1639
 tick_sched_handle+0xa2/0x190 kernel/time/tick-sched.c:167
 tick_sched_timer+0x47/0x130 kernel/time/tick-sched.c:1298
 __run_hrtimer kernel/time/hrtimer.c:1389 [inline]
 __hrtimer_run_queues+0x33b/0xdd0 kernel/time/hrtimer.c:1451
 hrtimer_interrupt+0x314/0x770 kernel/time/hrtimer.c:1509
 local_apic_timer_interrupt arch/x86/kernel/apic/apic.c:1041 [inline]
 smp_apic_timer_interrupt+0x111/0x550 arch/x86/kernel/apic/apic.c:1066
 apic_timer_interrupt+0xf/0x20 arch/x86/entry/entry_64.S:806
 </IRQ>
RIP: 0010:preempt_count arch/x86/include/asm/preempt.h:26 [inline]
RIP: 0010:check_kcov_mode kernel/kcov.c:68 [inline]
RIP: 0010:write_comp_data+0x9/0x70 kernel/kcov.c:123
Code: 12 00 00 8b 80 e4 12 00 00 48 8b 11 48 83 c2 01 48 39 d0 76 07 48 89 34 d1 48 89 11 5d c3 0f 1f 00 65 4c 8b 04 25 c0 fd 01 00 <65> 8b 05 f8 3b 91 7e a9 00 01 1f 00 75 51 41 8b 80 e0 12 00 00 83
RSP: 0018:ffff88808d9ef760 EFLAGS: 00000246 ORIG_RAX: ffffffffffffff13
RAX: dffffc0000000000 RBX: ffff88808d9ef848 RCX: ffffffff81b1c7ca
RDX: 0000000000000001 RSI: 0000000000000000 RDI: 0000000000000005
RBP: ffff88808d9ef768 R08: ffff88808421e080 R09: 0000000000000002
R10: ffffed1015d26c6f R11: ffff8880ae93637b R12: ffffffff877b6360
R13: 0000000003826ffc R14: 0000000000000001 R15: 0000000000000000
 do_iter_readv_writev+0x5ba/0x8f0 fs/read_write.c:690
 do_iter_write fs/read_write.c:970 [inline]
 do_iter_write+0x184/0x610 fs/read_write.c:951
 vfs_iter_write+0x77/0xb0 fs/read_write.c:983
 iter_file_splice_write+0x65c/0xbd0 fs/splice.c:746
 do_splice_from fs/splice.c:848 [inline]
 direct_splice_actor+0x123/0x190 fs/splice.c:1020
 splice_direct_to_actor+0x366/0x970 fs/splice.c:975
 do_splice_direct+0x1da/0x2a0 fs/splice.c:1063
 do_sendfile+0x597/0xd00 fs/read_write.c:1464
 __do_sys_sendfile64 fs/read_write.c:1519 [inline]
 __se_sys_sendfile64 fs/read_write.c:1511 [inline]
 __x64_sys_sendfile64+0x15a/0x220 fs/read_write.c:1511
 do_syscall_64+0xfd/0x680 arch/x86/entry/common.c:301
 entry_SYSCALL_64_after_hwframe+0x49/0xbe
RIP: 0033:0x4417c9
Code: e8 7c e7 ff ff 48 83 c4 18 c3 0f 1f 80 00 00 00 00 48 89 f8 48 89 f7 48 89 d6 48 89 ca 4d 89 c2 4d 89 c8 4c 8b 4c 24 08 0f 05 <48> 3d 01 f0 ff ff 0f 83 bb 07 fc ff c3 66 2e 0f 1f 84 00 00 00 00
RSP: 002b:00007ffee86956c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000028
RAX: ffffffffffffffda RBX: 00007ffee8695870 RCX: 00000000004417c9
RDX: 0000000020000000 RSI: 0000000000000003 RDI: 0000000000000003
RBP: 0000000000000000 R08: 0000000000000000 R09: 0000000000000000
R10: 00008080fffffffe R11: 0000000000000246 R12: 0000000000000000
R13: 00000000004024a0 R14: 0000000000000000 R15: 0000000000000000
rcu: rcu_preempt kthread starved for 10548 jiffies! g7925 f0x2 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=1
rcu: RCU grace-period kthread stack dump:
rcu_preempt     R  running task    29056    10      2 0x80004000
Call Trace:
 context_switch kernel/sched/core.c:2818 [inline]
 __schedule+0x7cb/0x1560 kernel/sched/core.c:3445
 schedule+0xa8/0x260 kernel/sched/core.c:3509
 schedule_timeout+0x486/0xc50 kernel/time/timer.c:1807
 rcu_gp_fqs_loop kernel/rcu/tree.c:1589 [inline]
 rcu_gp_kthread+0x9b2/0x18b0 kernel/rcu/tree.c:1746
 kthread+0x354/0x420 kernel/kthread.c:255
 ret_from_fork+0x24/0x30 arch/x86/entry/entry_64.S:352

Crashes (1):
Time Kernel Commit Syzkaller Config Log Report Syz repro C repro VM info Assets (help?) Manager Title
2019/06/22 12:59 upstream abf02e2964b3 34bf9440 .config console log report syz C ci-upstream-kasan-gce-root
* Struck through repros no longer work on HEAD.