syzbot
BUG: soft lockup in wg_ratelimiter_gc_entries
Status:
moderation: reported on 2026/11/22 04:33
Subsystems:
wireguard
Labels:
prio:low
[Documentation on labels]
Reported-by: syzbot+e696759843d287a23861@syzkaller.appspotmail.com
First crash: 4h04m, last: 4h04m
✨ AI-generated root cause summary:
CPU 0 reported a 24s soft lockup inside wg_ratelimiter_gc_entries() at cond_resched() -> __might_resched() -> lock_is_held_type(), while CPU 1 was in arena_map_free() -> remove_vm_area() -> debug_check_no_obj_freed() scanning a 4GB+64KB vm_area. Inspection of drivers/net/wireguard/ratelimiter.c and lockdep irqtrace events shows wg_ratelimiter_gc_entries() has bounded iteration (table_size <= 8192), drops table_lock per bucket, and keeps IRQs and preemption enabled, with zero interrupts occurring between entry (ktime_get_coarse_with_offset, stamp 1142073) and the watchdog IRQ (stamp 1142074). Thus, no kernel infinite loop or IRQ/preemption leak is present in wg_ratelimiter_gc_entries(); the 24s elapsed time without timer interrupts indicates an inconclusive external vCPU/hypervisor stall (arm64 lacks kvm_check_and_clear_guest_paused() support), potentially exacerbated by heavy debug/sanitizer CPU load such as CPU 1's ~1M-chunk debug_check_no_obj_freed() loop under CONFIG_DEBUG_OBJECTS_FREE.
▶
▼
✨ Full AI root cause analysis
Technical Analysis
1. Crash Site & Execution State on CPU 0
The soft lockup watchdog fired on CPU 0 while kworker/u8:6 (PID 1180) was executing wg_ratelimiter_gc_entries() (drivers/net/wireguard/ratelimiter.c:55-82) on events_power_efficient (system_power_efficient_wq):
- Bounded loop and preemption:
wg_ratelimiter_init() (drivers/net/wireguard/ratelimiter.c:173-177) caps table_size at 8192 and max_entries = table_size * 8 (<= 65536). Each iteration of for (i = 0; i < table_size; ++i) acquires table_lock (non-IRQ-disabling spin_lock), scans table_v4[i] and table_v6[i], unlocks table_lock, and invokes cond_resched().
- Object lifecycle and locking: Both
wg_ratelimiter_allow() (hlist_add_head_rcu, line 149) and wg_ratelimiter_gc_entries() (entry_uninit() $\rightarrow$ hlist_del_rcu() + call_rcu(..., entry_free), lines 48-52) serialize hash-bucket modifications under table_lock, while wg_ratelimiter_uninit() calls cancel_delayed_work_sync(&gc_work) before clearing the tables and destroying entry_cache. Even if a bucket list were somehow corrupted, an infinite loop in hlist_for_each_entry_safe() would stall inside lines 64-75 with table_lock held, not at cond_resched() on line 78.
- Impossibility of a 24s
CONFIG_PREEMPT=y CPU-bound lockup here: In kernel/watchdog.c:798-826, watchdog_timer_fn() queues softlockup_fn() via stop_one_cpu_nowait() on migration/0 (SCHED_FIFO, priority 99) every sample_period. On CONFIG_PREEMPT=y, any hardware interrupt returning to wg_ratelimiter_gc_entries() at line 78 (preempt_count == 0, IRQs enabled) would immediately preempt kworker/u8:6 via arm64_exit_to_kernel_mode_preempt() (arch/arm64/kernel/entry-common.c:105-110) and run softlockup_fn(), resetting watchdog_touch_ts.
3. Evidence from irqtrace and Concurrent CPU 1 Activity (Inconclusive Trigger)
print_irqtrace_events(current) (kernel/locking/lockdep.c:4244-4265) records current->irqtrace on kworker/u8:6:
1142062 / 1142066: neigh_periodic_work() (net/core/neighbour.c:987, 1054), which also runs on system_power_efficient_wq.
1142073: ktime_get_coarse_with_offset() (kernel/time/timekeeping.c:1071 $\rightarrow$ seqcount_lockdep_reader_access(), include/linux/seqlock.h:75), called once at the start of wg_ratelimiter_gc_entries() on line 57 (ktime_get_coarse_boottime_ns()) before entering the loop.
1142074: __el1_irq (arch/arm64/kernel/entry-common.c:566), the timer interrupt in which watchdog_timer_fn() reported the 24s soft lockup.
- Because every
el1_interrupt() updates irq_events via irqentry_enter_from_kernel_mode() / irqentry_exit_to_kernel_mode_preempt() and call_rcu() also toggles IRQs, the single-step increment (1142073 $\rightarrow$ 1142074) proves zero hardware interrupts were delivered to CPU 0 (and entry_uninit() was not even called) between line 57 and line 78 of wg_ratelimiter_gc_entries().
- The 24s elapsed
running_clock() delta without intermediate timer interrupts therefore cannot be caused by wg_ratelimiter_gc_entries(). It indicates an external vCPU/hypervisor pause or host CPU starvation (Hardware name: linux,dummy-virt), where arm64's kvm_check_and_clear_guest_paused() (include/asm-generic/kvm_para.h:12) is a no-op stub returning false in watchdog_timer_fn() (kernel/watchdog.c:841), making the exact external stall trigger inconclusive from the kernel report alone.
- Concurrently, the NMI backtrace shows CPU 1 (
kworker/u8:1, PID 5013) inside bpf_map_free_deferred() $\rightarrow$ arena_map_free() (kernel/bpf/arena.c:401) $\rightarrow$ free_vm_area() $\rightarrow$ remove_vm_area() (mm/vmalloc.c:3385) $\rightarrow$ __debug_check_no_obj_freed() (lib/debugobjects.c:1141-1180). Because arena_map_alloc() reserves KERN_VM_SZ = SZ_4G + GUARD_SZ (kernel/bpf/arena.c:48, 302) and CONFIG_ARM64_4K_PAGES=y sets ODEBUG_CHUNK_SIZE = 4096 (lib/debugobjects.c:34-35), CONFIG_DEBUG_OBJECTS_FREE=y forces CPU 1 to iterate over 1,048,592 chunks (x26 = 0x7b4cc = 505,036 remaining), acquiring db->lock and scanning obj_hash on each chunk. While this does not block CPU 0 via any shared kernel lock or interrupt mask, it represents substantial debug-configuration CPU work occurring alongside the QEMU/host stall.
Sample crash report:
* Struck through repros no longer work on HEAD.