syzbot


possible deadlock in htab_lru_map_delete_elem

Status: upstream: reported C repro on 2024/12/06 13:58
Subsystems: bpf
[Documentation on labels]
Reported-by: syzbot+0a26db48dcd6d80be6c0@syzkaller.appspotmail.com
First crash: 163d, last: 64d
Discussions (1)
Title Replies (including bot) Last reply
[syzbot] [bpf?] possible deadlock in htab_lru_map_delete_elem 1 (2) 2024/12/06 18:59
Last patch testing requests (4)
Created Duration User Patch Repo Result
2025/05/07 23:03 19m retest repro upstream OK log
2025/03/13 07:09 13m retest repro upstream report log
2025/02/26 22:23 39m retest repro upstream report log
2024/12/18 21:59 14m retest repro upstream report log

Sample crash report:
======================================================
WARNING: possible circular locking dependency detected
6.14.0-rc4-syzkaller-00073-g5394eea10651 #0 Not tainted
------------------------------------------------------
syz.0.465/7042 is trying to acquire lock:
ffff88803f6943c8 (&htab->lockdep_key#448){....}-{2:2}, at: htab_lock_bucket kernel/bpf/hashtab.c:167 [inline]
ffff88803f6943c8 (&htab->lockdep_key#448){....}-{2:2}, at: htab_lru_map_delete_elem+0x1c8/0x790 kernel/bpf/hashtab.c:1483

but task is already holding lock:
ffff88804a89d338 (&htab->lockdep_key#449){....}-{2:2}, at: htab_lock_bucket kernel/bpf/hashtab.c:167 [inline]
ffff88804a89d338 (&htab->lockdep_key#449){....}-{2:2}, at: htab_lru_map_delete_elem+0x1c8/0x790 kernel/bpf/hashtab.c:1483

which lock already depends on the new lock.


the existing dependency chain (in reverse order) is:

-> #1 (&htab->lockdep_key#449){....}-{2:2}:
       __raw_spin_lock include/linux/spinlock_api_smp.h:133 [inline]
       _raw_spin_lock+0x2e/0x40 kernel/locking/spinlock.c:154
       htab_lock_bucket kernel/bpf/hashtab.c:167 [inline]
       htab_lru_map_delete_elem+0x1c8/0x790 kernel/bpf/hashtab.c:1483
       bpf_prog_6f5f05285f674219+0x43/0x4c
       bpf_dispatcher_nop_func include/linux/bpf.h:1290 [inline]
       __bpf_prog_run include/linux/filter.h:701 [inline]
       bpf_prog_run include/linux/filter.h:708 [inline]
       __bpf_trace_run kernel/trace/bpf_trace.c:2371 [inline]
       bpf_trace_run2+0x231/0x590 kernel/trace/bpf_trace.c:2412
       __bpf_trace_contention_begin+0xca/0x110 include/trace/events/lock.h:95
       __traceiter_contention_begin+0x5a/0xa0 include/trace/events/lock.h:95
       __preempt_count_dec_and_test arch/x86/include/asm/preempt.h:94 [inline]
       class_preempt_notrace_destructor include/linux/preempt.h:481 [inline]
       __do_trace_contention_begin include/trace/events/lock.h:95 [inline]
       trace_contention_begin.constprop.0+0xf3/0x170 include/trace/events/lock.h:95
       __pv_queued_spin_lock_slowpath+0x10b/0xc90 kernel/locking/qspinlock.c:402
       pv_queued_spin_lock_slowpath arch/x86/include/asm/paravirt.h:577 [inline]
       queued_spin_lock_slowpath arch/x86/include/asm/qspinlock.h:51 [inline]
       queued_spin_lock include/asm-generic/qspinlock.h:114 [inline]
       do_raw_spin_lock+0x210/0x2c0 kernel/locking/spinlock_debug.c:116
       htab_lock_bucket kernel/bpf/hashtab.c:167 [inline]
       htab_lru_map_delete_elem+0x1c8/0x790 kernel/bpf/hashtab.c:1483
       bpf_prog_6f5f05285f674219+0x43/0x4c
       bpf_dispatcher_nop_func include/linux/bpf.h:1290 [inline]
       __bpf_prog_run include/linux/filter.h:701 [inline]
       bpf_prog_run include/linux/filter.h:708 [inline]
       __bpf_trace_run kernel/trace/bpf_trace.c:2371 [inline]
       bpf_trace_run2+0x231/0x590 kernel/trace/bpf_trace.c:2412
       __bpf_trace_contention_begin+0xca/0x110 include/trace/events/lock.h:95
       __traceiter_contention_begin+0x5a/0xa0 include/trace/events/lock.h:95
       __do_trace_contention_begin include/trace/events/lock.h:95 [inline]
       trace_contention_begin+0xd2/0x140 include/trace/events/lock.h:95
       __mutex_lock_common kernel/locking/mutex.c:587 [inline]
       __mutex_lock+0x1a8/0xb10 kernel/locking/mutex.c:730
       __set_oom_adj.isra.0+0x68/0x1120 fs/proc/base.c:1129
       oom_score_adj_write+0x1b8/0x200 fs/proc/base.c:1294
       vfs_write+0x24c/0x1150 fs/read_write.c:677
       ksys_write+0x12b/0x250 fs/read_write.c:731
       do_syscall_x64 arch/x86/entry/common.c:52 [inline]
       do_syscall_64+0xcd/0x250 arch/x86/entry/common.c:83
       entry_SYSCALL_64_after_hwframe+0x77/0x7f

-> #0 (&htab->lockdep_key#448){....}-{2:2}:
       check_prev_add kernel/locking/lockdep.c:3163 [inline]
       check_prevs_add kernel/locking/lockdep.c:3282 [inline]
       validate_chain kernel/locking/lockdep.c:3906 [inline]
       __lock_acquire+0x249e/0x3c40 kernel/locking/lockdep.c:5228
       lock_acquire.part.0+0x11b/0x380 kernel/locking/lockdep.c:5851
       __raw_spin_lock include/linux/spinlock_api_smp.h:133 [inline]
       _raw_spin_lock+0x2e/0x40 kernel/locking/spinlock.c:154
       htab_lock_bucket kernel/bpf/hashtab.c:167 [inline]
       htab_lru_map_delete_elem+0x1c8/0x790 kernel/bpf/hashtab.c:1483
       bpf_prog_6f5f05285f674219+0x43/0x4c
       bpf_dispatcher_nop_func include/linux/bpf.h:1290 [inline]
       __bpf_prog_run include/linux/filter.h:701 [inline]
       bpf_prog_run include/linux/filter.h:708 [inline]
       __bpf_trace_run kernel/trace/bpf_trace.c:2371 [inline]
       bpf_trace_run2+0x231/0x590 kernel/trace/bpf_trace.c:2412
       __bpf_trace_contention_begin+0xca/0x110 include/trace/events/lock.h:95
       __traceiter_contention_begin+0x5a/0xa0 include/trace/events/lock.h:95
       __preempt_count_dec_and_test arch/x86/include/asm/preempt.h:94 [inline]
       class_preempt_notrace_destructor include/linux/preempt.h:481 [inline]
       __do_trace_contention_begin include/trace/events/lock.h:95 [inline]
       trace_contention_begin.constprop.0+0xf3/0x170 include/trace/events/lock.h:95
       __pv_queued_spin_lock_slowpath+0x10b/0xc90 kernel/locking/qspinlock.c:402
       pv_queued_spin_lock_slowpath arch/x86/include/asm/paravirt.h:577 [inline]
       queued_spin_lock_slowpath arch/x86/include/asm/qspinlock.h:51 [inline]
       queued_spin_lock include/asm-generic/qspinlock.h:114 [inline]
       do_raw_spin_lock+0x210/0x2c0 kernel/locking/spinlock_debug.c:116
       htab_lock_bucket kernel/bpf/hashtab.c:167 [inline]
       htab_lru_map_delete_elem+0x1c8/0x790 kernel/bpf/hashtab.c:1483
       bpf_prog_6f5f05285f674219+0x43/0x4c
       bpf_dispatcher_nop_func include/linux/bpf.h:1290 [inline]
       __bpf_prog_run include/linux/filter.h:701 [inline]
       bpf_prog_run include/linux/filter.h:708 [inline]
       __bpf_trace_run kernel/trace/bpf_trace.c:2371 [inline]
       bpf_trace_run2+0x231/0x590 kernel/trace/bpf_trace.c:2412
       __bpf_trace_contention_begin+0xca/0x110 include/trace/events/lock.h:95
       __traceiter_contention_begin+0x5a/0xa0 include/trace/events/lock.h:95
       __do_trace_contention_begin include/trace/events/lock.h:95 [inline]
       trace_contention_begin+0xd2/0x140 include/trace/events/lock.h:95
       __mutex_lock_common kernel/locking/mutex.c:587 [inline]
       __mutex_lock+0x1a8/0xb10 kernel/locking/mutex.c:730
       pcpu_alloc_noprof+0xc5a/0x1680 mm/percpu.c:1782
       bpf_map_alloc_percpu+0x9a/0x4b0 kernel/bpf/syscall.c:555
       prealloc_init kernel/bpf/hashtab.c:341 [inline]
       htab_map_alloc+0x1231/0x17b0 kernel/bpf/hashtab.c:579
       map_create+0x5c5/0x1d20 kernel/bpf/syscall.c:1455
       __sys_bpf+0x4391/0x49c0 kernel/bpf/syscall.c:5777
       __do_sys_bpf kernel/bpf/syscall.c:5902 [inline]
       __se_sys_bpf kernel/bpf/syscall.c:5900 [inline]
       __x64_sys_bpf+0x78/0xc0 kernel/bpf/syscall.c:5900
       do_syscall_x64 arch/x86/entry/common.c:52 [inline]
       do_syscall_64+0xcd/0x250 arch/x86/entry/common.c:83
       entry_SYSCALL_64_after_hwframe+0x77/0x7f

other info that might help us debug this:

 Possible unsafe locking scenario:

       CPU0                    CPU1
       ----                    ----
  lock(&htab->lockdep_key#449);
                               lock(&htab->lockdep_key#448);
                               lock(&htab->lockdep_key#449);
  lock(&htab->lockdep_key#448);

 *** DEADLOCK ***

4 locks held by syz.0.465/7042:
 #0: ffffffff8e320f68 (pcpu_alloc_mutex){+.+.}-{4:4}, at: pcpu_alloc_noprof+0xc5a/0x1680 mm/percpu.c:1782
 #1: ffffffff8e1bcc80 (rcu_read_lock){....}-{1:3}, at: rcu_lock_acquire include/linux/rcupdate.h:337 [inline]
 #1: ffffffff8e1bcc80 (rcu_read_lock){....}-{1:3}, at: rcu_read_lock include/linux/rcupdate.h:849 [inline]
 #1: ffffffff8e1bcc80 (rcu_read_lock){....}-{1:3}, at: __bpf_trace_run kernel/trace/bpf_trace.c:2370 [inline]
 #1: ffffffff8e1bcc80 (rcu_read_lock){....}-{1:3}, at: bpf_trace_run2+0x1c2/0x590 kernel/trace/bpf_trace.c:2412
 #2: ffff88804a89d338 (&htab->lockdep_key#449){....}-{2:2}, at: htab_lock_bucket kernel/bpf/hashtab.c:167 [inline]
 #2: ffff88804a89d338 (&htab->lockdep_key#449){....}-{2:2}, at: htab_lru_map_delete_elem+0x1c8/0x790 kernel/bpf/hashtab.c:1483
 #3: ffffffff8e1bcc80 (rcu_read_lock){....}-{1:3}, at: rcu_lock_acquire include/linux/rcupdate.h:337 [inline]
 #3: ffffffff8e1bcc80 (rcu_read_lock){....}-{1:3}, at: rcu_read_lock include/linux/rcupdate.h:849 [inline]
 #3: ffffffff8e1bcc80 (rcu_read_lock){....}-{1:3}, at: __bpf_trace_run kernel/trace/bpf_trace.c:2370 [inline]
 #3: ffffffff8e1bcc80 (rcu_read_lock){....}-{1:3}, at: bpf_trace_run2+0x1c2/0x590 kernel/trace/bpf_trace.c:2412

stack backtrace:
CPU: 2 UID: 0 PID: 7042 Comm: syz.0.465 Not tainted 6.14.0-rc4-syzkaller-00073-g5394eea10651 #0
Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 1.16.3-debian-1.16.3-2~bpo12+1 04/01/2014
Call Trace:
 <TASK>
 __dump_stack lib/dump_stack.c:94 [inline]
 dump_stack_lvl+0x116/0x1f0 lib/dump_stack.c:120
 print_circular_bug+0x490/0x760 kernel/locking/lockdep.c:2076
 check_noncircular+0x31a/0x400 kernel/locking/lockdep.c:2208
 check_prev_add kernel/locking/lockdep.c:3163 [inline]
 check_prevs_add kernel/locking/lockdep.c:3282 [inline]
 validate_chain kernel/locking/lockdep.c:3906 [inline]
 __lock_acquire+0x249e/0x3c40 kernel/locking/lockdep.c:5228
 lock_acquire.part.0+0x11b/0x380 kernel/locking/lockdep.c:5851
 __raw_spin_lock include/linux/spinlock_api_smp.h:133 [inline]
 _raw_spin_lock+0x2e/0x40 kernel/locking/spinlock.c:154
 htab_lock_bucket kernel/bpf/hashtab.c:167 [inline]
 htab_lru_map_delete_elem+0x1c8/0x790 kernel/bpf/hashtab.c:1483
 bpf_prog_6f5f05285f674219+0x43/0x4c
 bpf_dispatcher_nop_func include/linux/bpf.h:1290 [inline]
 __bpf_prog_run include/linux/filter.h:701 [inline]
 bpf_prog_run include/linux/filter.h:708 [inline]
 __bpf_trace_run kernel/trace/bpf_trace.c:2371 [inline]
 bpf_trace_run2+0x231/0x590 kernel/trace/bpf_trace.c:2412
 __bpf_trace_contention_begin+0xca/0x110 include/trace/events/lock.h:95
 __traceiter_contention_begin+0x5a/0xa0 include/trace/events/lock.h:95
 __preempt_count_dec_and_test arch/x86/include/asm/preempt.h:94 [inline]
 class_preempt_notrace_destructor include/linux/preempt.h:481 [inline]
 __do_trace_contention_begin include/trace/events/lock.h:95 [inline]
 trace_contention_begin.constprop.0+0xf3/0x170 include/trace/events/lock.h:95
 __pv_queued_spin_lock_slowpath+0x10b/0xc90 kernel/locking/qspinlock.c:402
 pv_queued_spin_lock_slowpath arch/x86/include/asm/paravirt.h:577 [inline]
 queued_spin_lock_slowpath arch/x86/include/asm/qspinlock.h:51 [inline]
 queued_spin_lock include/asm-generic/qspinlock.h:114 [inline]
 do_raw_spin_lock+0x210/0x2c0 kernel/locking/spinlock_debug.c:116
 htab_lock_bucket kernel/bpf/hashtab.c:167 [inline]
 htab_lru_map_delete_elem+0x1c8/0x790 kernel/bpf/hashtab.c:1483
 bpf_prog_6f5f05285f674219+0x43/0x4c
 bpf_dispatcher_nop_func include/linux/bpf.h:1290 [inline]
 __bpf_prog_run include/linux/filter.h:701 [inline]
 bpf_prog_run include/linux/filter.h:708 [inline]
 __bpf_trace_run kernel/trace/bpf_trace.c:2371 [inline]
 bpf_trace_run2+0x231/0x590 kernel/trace/bpf_trace.c:2412
 __bpf_trace_contention_begin+0xca/0x110 include/trace/events/lock.h:95
 __traceiter_contention_begin+0x5a/0xa0 include/trace/events/lock.h:95
 __do_trace_contention_begin include/trace/events/lock.h:95 [inline]
 trace_contention_begin+0xd2/0x140 include/trace/events/lock.h:95
 __mutex_lock_common kernel/locking/mutex.c:587 [inline]
 __mutex_lock+0x1a8/0xb10 kernel/locking/mutex.c:730
 pcpu_alloc_noprof+0xc5a/0x1680 mm/percpu.c:1782
 bpf_map_alloc_percpu+0x9a/0x4b0 kernel/bpf/syscall.c:555
 prealloc_init kernel/bpf/hashtab.c:341 [inline]
 htab_map_alloc+0x1231/0x17b0 kernel/bpf/hashtab.c:579
 map_create+0x5c5/0x1d20 kernel/bpf/syscall.c:1455
 __sys_bpf+0x4391/0x49c0 kernel/bpf/syscall.c:5777
 __do_sys_bpf kernel/bpf/syscall.c:5902 [inline]
 __se_sys_bpf kernel/bpf/syscall.c:5900 [inline]
 __x64_sys_bpf+0x78/0xc0 kernel/bpf/syscall.c:5900
 do_syscall_x64 arch/x86/entry/common.c:52 [inline]
 do_syscall_64+0xcd/0x250 arch/x86/entry/common.c:83
 entry_SYSCALL_64_after_hwframe+0x77/0x7f
RIP: 0033:0x7f512cf8d169
Code: ff ff c3 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 40 00 48 89 f8 48 89 f7 48 89 d6 48 89 ca 4d 89 c2 4d 89 c8 4c 8b 4c 24 08 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 c7 c1 a8 ff ff ff f7 d8 64 89 01 48
RSP: 002b:00007f512de6e038 EFLAGS: 00000246 ORIG_RAX: 0000000000000141
RAX: ffffffffffffffda RBX: 00007f512d1a5fa0 RCX: 00007f512cf8d169
RDX: 0000000000000048 RSI: 00004000000009c0 RDI: 0000000000000000
RBP: 00007f512d00e2a0 R08: 0000000000000000 R09: 0000000000000000
R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000000
R13: 0000000000000001 R14: 00007f512d1a5fa0 R15: 00007ffd54adadd8
 </TASK>

Crashes (2):
Time Kernel Commit Syzkaller Config Log Report Syz repro C repro VM info Assets (help?) Manager Title
2025/02/27 07:02 upstream 5394eea10651 6a8fcbc4 .config console log report syz / log [disk image (non-bootable)] [vmlinux] [kernel image] ci-qemu-upstream possible deadlock in htab_lru_map_delete_elem
2024/12/04 21:46 upstream feffde684ac2 b50eb251 .config console log report syz / log C [disk image (non-bootable)] [vmlinux] [kernel image] ci-qemu-upstream possible deadlock in htab_lru_map_delete_elem
* Struck through repros no longer work on HEAD.