syzbot
memory leak in __pcs_replace_empty_main
Status:
upstream: reported syz repro on 2026/11/13 07:19
Subsystems:
btrfs
ntfs3
Labels:
prio:high
[Documentation on labels]
Reported-by: syzbot+527fed49e5b988882c9a@syzkaller.appspotmail.com
First crash: 4d01h, last: 4d01h
β¨ AI-generated root cause summary:
In SLUB's per-CPU sheaves (PCS) implementation (mm/slub.c), struct slab_sheaf objects allocated via __alloc_empty_sheaf() are tracked as standard heap allocations by kmemleak. When an rcu_free sheaf reaches capacity in __kfree_rcu_sheaf(), it is detached from pcs->rcu_free and queued for deferred cleanup via call_rcu(&rcu_sheaf->rcu_head, rcu_free_sheaf). While queued, the only pointer to the sheaf exists in RCU's segmented callback list (rcu_data.cblist); if any earlier callback in the singly-linked list resides in memory untracked or unscanned by kmemleak (such as struct slab allocated in struct page, SLAB_NOLEAKTRACE objects, or already-freed objects), kmemleak cannot traverse the chain. Because recycled sheaves retain their early-boot allocation timestamps, kmemleak exceeds its scan age threshold and falsely reports these in-flight sheaves as unreferenced memory leaks.
▶
▼
β¨ Full AI root cause analysis
Technical Root Cause Analysis
Involved Functions, Structs, and Files
- Files:
mm/slub.c
- Structures:
struct slab_sheaf, struct slub_percpu_sheaves, struct node_barn, struct rcu_head, struct rcu_segcblist
- Functions:
__alloc_empty_sheaf(), __kfree_rcu_sheaf(), rcu_free_sheaf(), pcs_flush_all(), __pcs_replace_full_main(), __pcs_replace_empty_main()
Technical Mechanism and Failure Progression
-
Sheaf Allocation and Tracking:
In SLUB's per-CPU sheaves (PCS) caching layer, empty sheaves (struct slab_sheaf) are allocated via __alloc_empty_sheaf():
sheaf_size = struct_size(sheaf, objects, capacity);
sheaf = kmalloc_flags(sheaf_size, gfp | __GFP_ZERO, alloc_flags, NUMA_NO_NODE);
Because standard kmalloc_flags() is used without annotating the object with kmemleak_ignore() or kmemleak_not_leak(), kmemleak registers and tracks each struct slab_sheaf as a regular kernel memory allocation.
-
Object Lifecycle and Recycling:
Sheaves allocated during early boot (evidenced by the backtraces from swapper/0, kthreadd, etc. at jiffies 4294937296β4294937453) circulate dynamically between per-CPU main/spare slots (pcs->main, pcs->spare) and the NUMA node barn (barn->sheaves_full, barn->sheaves_empty).
When needed for batching RCU frees (kfree_rcu() / kvfree_call_rcu()), an empty sheaf is assigned to pcs->rcu_free.
-
RCU Deferral in __kfree_rcu_sheaf():
In __kfree_rcu_sheaf(), when pcs->rcu_free fills up to capacity (sheaf->size == s->sheaf_capacity), it is detached:
rcu_sheaf = pcs->rcu_free;
pcs->rcu_free = NULL;
rcu_sheaf->node = numa_node_id();
...
call_rcu(&rcu_sheaf->rcu_head, rcu_free_sheaf);
At this point, all external pointers to rcu_sheaf are cleared. The only pointer referencing the sheaf is the next pointer of the preceding callback in RCU's segmented callback list (rcu_data.cblist).
-
Hex Dump Verification of In-Flight State:
All 6 reported leak instances exhibit the exact layout of a struct slab_sheaf actively queued in call_rcu():
- Offset
0x00..0x07 (rcu_head.next): Points to another struct rcu_head in the RCU callback list (at non-512-aligned offsets such as 0x...9e0, 0x...ca0, 0x...bf0, 0x...3b0).
- Offset
0x08..0x0f (rcu_head.func): Contains 0xffffffff81b40020, the kernel text address of rcu_free_sheaf.
- Offset
0x10..0x17 (sheaf->cache): Points to the associated struct kmem_cache.
- Offset
0x18..0x1b (sheaf->size): Equals 0x3c (60) or 0x20 (32), matching the cache's sheaf capacity (i.e. completely full).
- Offset
0x1c..0x1f (sheaf->node): Contains NUMA node 0.
Crucially, rcu_free_sheaf() has not yet executed for these objects. Once rcu_free_sheaf() executes, it either overwrites offset 0x00..0x0f by linking sheaf->barn_list into barn->sheaves_full (replacing rcu_head.func with a heap pointer &barn->sheaves_full), or flushes sheaf->size to 0 and links it to barn->sheaves_empty or frees it.
-
Kmemleak Pointer Chain Breakage:
kmemleak detects unreferenced objects by scanning kernel memory (stack, data/BSS sections, tracked heap objects, and per-CPU areas).
In RCU's segmented callback list, callbacks are chained singly via rcu_head.next. When any preceding callback in the chain resides within memory not scanned by kmemleakβsuch as a struct slab queued by rcu_free_slab() (which lives in struct page / mem_map, unscanned by kmemleak), an object from a SLAB_NOLEAKTRACE cache, or an object whose kmemleak_free() hook ran prior to call_rcu()βthe pointer traversal chain from rcu_data is severed.
-
Triggering False-Positive Leak Reports:
Because the struct slab_sheaf objects were originally allocated early in system boot, their allocation timestamps far exceed kmemleak's minimum age threshold. While they temporarily reside in rcu_data.cblist waiting for an RCU grace period to expire, kmemleak_scan() fails to locate a reference through the severed RCU list and incorrectly reports the in-flight struct slab_sheaf objects as leaked.
Sample crash report:
BUG: memory leak
unreferenced object 0xffff88810005f600 (size 512):
comm "swapper/0", pid 0, jiffies 4294937296
hex dump (first 32 bytes):
e0 79 f0 2b 81 88 ff ff 20 00 b4 81 ff ff ff ff .y.+.... .......
00 12 04 00 81 88 ff ff 3c 00 00 00 00 00 00 00 ........<.......
backtrace (crc 9e0404a6):
kmemleak_alloc_recursive include/linux/kmemleak.h:44 [inline]
slab_post_alloc_hook mm/slub.c:4733 [inline]
slab_alloc_node mm/slub.c:5033 [inline]
__do_kmalloc_node mm/slub.c:5530 [inline]
__kmalloc_flags_noprof+0x42c/0x570 mm/slub.c:5719
_kmalloc_flags_noprof mm/slab.h:55 [inline]
__alloc_empty_sheaf mm/slub.c:2854 [inline]
alloc_empty_sheaf mm/slub.c:2874 [inline]
__pcs_replace_empty_main+0x260/0x2e0 mm/slub.c:4806
alloc_from_pcs mm/slub.c:4887 [inline]
slab_alloc_node mm/slub.c:5021 [inline]
__kmalloc_cache_noprof+0x370/0x3f0 mm/slub.c:5676
_kmalloc_noprof include/linux/slab.h:991 [inline]
_kzalloc_noprof include/linux/slab.h:1312 [inline]
__irq_domain_alloc_fwnode+0x37/0x160 kernel/irq/irqdomain.c:97
irq_domain_alloc_named_fwnode include/linux/irqdomain.h:271 [inline]
arch_early_irq_init+0x1c/0x70 arch/x86/kernel/apic/vector.c:803
start_kernel+0x9c5/0xc10 init/main.c:1150
x86_64_start_reservations+0x24/0x30 arch/x86/kernel/head64.c:310
x86_64_start_kernel+0xce/0xd0 arch/x86/kernel/head64.c:291
common_startup_64+0x13e/0x158
BUG: memory leak
unreferenced object 0xffff8881009c5000 (size 512):
comm "kthreadd", pid 2, jiffies 4294937342
hex dump (first 32 bytes):
a0 5c 9b 2a 81 88 ff ff 20 00 b4 81 ff ff ff ff .\.*.... .......
00 12 04 00 81 88 ff ff 3c 00 00 00 00 00 00 00 ........<.......
backtrace (crc 3e15a3c6):
kmemleak_alloc_recursive include/linux/kmemleak.h:44 [inline]
slab_post_alloc_hook mm/slub.c:4733 [inline]
slab_alloc_node mm/slub.c:5033 [inline]
__do_kmalloc_node mm/slub.c:5530 [inline]
__kmalloc_flags_noprof+0x42c/0x570 mm/slub.c:5719
_kmalloc_flags_noprof mm/slab.h:55 [inline]
__alloc_empty_sheaf mm/slub.c:2854 [inline]
alloc_empty_sheaf mm/slub.c:2874 [inline]
__pcs_replace_empty_main+0x260/0x2e0 mm/slub.c:4806
alloc_from_pcs mm/slub.c:4887 [inline]
slab_alloc_node mm/slub.c:5021 [inline]
__kmalloc_cache_node_noprof+0x381/0x410 mm/slub.c:5695
_kmalloc_node_noprof include/linux/slab.h:1196 [inline]
__get_vm_area_node+0xe9/0x1f0 mm/vmalloc.c:3270
__vmalloc_node_range_noprof+0x1e1/0xd90 mm/vmalloc.c:4112
__vmalloc_node_noprof+0x71/0x90 mm/vmalloc.c:4213
alloc_thread_stack_node kernel/fork.c:359 [inline]
dup_task_struct kernel/fork.c:931 [inline]
copy_process+0x527/0x3060 kernel/fork.c:2150
kernel_clone+0x101/0x710 kernel/fork.c:2798
kernel_thread+0x80/0xb0 kernel/fork.c:2859
create_kthread kernel/kthread.c:459 [inline]
kthreadd+0x186/0x250 kernel/kthread.c:817
ret_from_fork+0x219/0x490 arch/x86/kernel/process.c:158
ret_from_fork_asm+0x1a/0x30 arch/x86/entry/entry_64.S:245
BUG: memory leak
unreferenced object 0xffff888102d99800 (size 512):
comm "kworker/u8:3", pid 235, jiffies 4294937447
hex dump (first 32 bytes):
f0 6b 9d 2a 81 88 ff ff 20 00 b4 81 ff ff ff ff .k.*.... .......
00 12 04 00 81 88 ff ff 3c 00 00 00 00 00 00 00 ........<.......
backtrace (crc 282505aa):
kmemleak_alloc_recursive include/linux/kmemleak.h:44 [inline]
slab_post_alloc_hook mm/slub.c:4733 [inline]
slab_alloc_node mm/slub.c:5033 [inline]
__do_kmalloc_node mm/slub.c:5530 [inline]
__kmalloc_flags_noprof+0x42c/0x570 mm/slub.c:5719
_kmalloc_flags_noprof mm/slab.h:55 [inline]
__alloc_empty_sheaf mm/slub.c:2854 [inline]
alloc_empty_sheaf mm/slub.c:2874 [inline]
__pcs_replace_full_main+0xf7/0x330 mm/slub.c:6094
free_to_pcs mm/slub.c:6147 [inline]
slab_free mm/slub.c:6633 [inline]
kfree+0x353/0x390 mm/slub.c:6923
call_usermodehelper_freeinfo kernel/umh.c:43 [inline]
umh_complete kernel/umh.c:57 [inline]
call_usermodehelper_exec_async+0x1c7/0x1f0 kernel/umh.c:117
ret_from_fork+0x219/0x490 arch/x86/kernel/process.c:158
ret_from_fork_asm+0x1a/0x30 arch/x86/entry/entry_64.S:245
BUG: memory leak
unreferenced object 0xffff888101943a00 (size 512):
comm "kdevtmpfs", pid 27, jiffies 4294937449
hex dump (first 32 bytes):
b0 c3 b7 0b 81 88 ff ff 20 00 b4 81 ff ff ff ff ........ .......
00 4a 04 00 81 88 ff ff 20 00 00 00 00 00 00 00 .J...... .......
backtrace (crc 6be06b44):
kmemleak_alloc_recursive include/linux/kmemleak.h:44 [inline]
slab_post_alloc_hook mm/slub.c:4733 [inline]
slab_alloc_node mm/slub.c:5033 [inline]
__do_kmalloc_node mm/slub.c:5530 [inline]
__kmalloc_flags_noprof+0x42c/0x570 mm/slub.c:5719
_kmalloc_flags_noprof mm/slab.h:55 [inline]
__alloc_empty_sheaf mm/slub.c:2854 [inline]
alloc_empty_sheaf mm/slub.c:2874 [inline]
__kfree_rcu_sheaf+0x1aa/0x2d0 mm/slub.c:6282
kfree_rcu_sheaf mm/slab_common.c:1689 [inline]
kvfree_call_rcu+0x1ef/0x3a0 mm/slab_common.c:2044
ma_free_rcu lib/maple_tree.c:173 [inline]
ma_free_rcu+0x29/0x40 lib/maple_tree.c:170
mas_topiary_node lib/maple_tree.c:1829 [inline]
mas_topiary_node lib/maple_tree.c:1817 [inline]
mas_topiary_replace+0xc49/0x1560 lib/maple_tree.c:1928
mas_wmb_replace+0x3c/0x630 lib/maple_tree.c:2114
mas_wr_split+0xcda/0x1650 lib/maple_tree.c:3502
mas_wr_store_entry+0x520/0xf60 lib/maple_tree.c:3575
mas_insert lib/maple_tree.c:3819 [inline]
mas_alloc_cyclic+0x481/0xa30 lib/maple_tree.c:3862
mtree_alloc_cyclic+0xc7/0x140 lib/maple_tree.c:5975
simple_offset_add+0x5c/0xd0 fs/libfs.c:296
shmem_mknod+0x1df/0x260 mm/shmem.c:4092
vfs_mknod+0x229/0x460 fs/namei.c:5287
handle_create drivers/base/devtmpfs.c:283 [inline]
handle drivers/base/devtmpfs.c:435 [inline]
devtmpfs_work_loop+0x292/0x64b drivers/base/devtmpfs.c:450
devtmpfsd+0x3a/0x50 drivers/base/devtmpfs.c:492
kthread+0x150/0x1a0 kernel/kthread.c:436
BUG: memory leak
unreferenced object 0xffff888102d98200 (size 512):
comm "kworker/0:1", pid 11, jiffies 4294937453
hex dump (first 32 bytes):
f0 6b 99 2a 81 88 ff ff 20 00 b4 81 ff ff ff ff .k.*.... .......
00 12 04 00 81 88 ff ff 3c 00 00 00 00 00 00 00 ........<.......
backtrace (crc 946da91f):
kmemleak_alloc_recursive include/linux/kmemleak.h:44 [inline]
slab_post_alloc_hook mm/slub.c:4733 [inline]
slab_alloc_node mm/slub.c:5033 [inline]
__do_kmalloc_node mm/slub.c:5530 [inline]
__kmalloc_flags_noprof+0x42c/0x570 mm/slub.c:5719
_kmalloc_flags_noprof mm/slab.h:55 [inline]
__alloc_empty_sheaf mm/slub.c:2854 [inline]
alloc_empty_sheaf mm/slub.c:2874 [inline]
__pcs_replace_full_main+0xf7/0x330 mm/slub.c:6094
free_to_pcs mm/slub.c:6147 [inline]
slab_free mm/slub.c:6633 [inline]
kfree+0x353/0x390 mm/slub.c:6923
vfree mm/vmalloc.c:3549 [inline]
vfree+0xa8/0x390 mm/vmalloc.c:3521
delayed_vfree_work+0x29/0x40 mm/vmalloc.c:3451
process_one_work+0x2b7/0x5c0 kernel/workqueue.c:3495
process_scheduled_works kernel/workqueue.c:3578 [inline]
worker_thread+0x255/0x4a0 kernel/workqueue.c:3659
kthread+0x150/0x1a0 kernel/kthread.c:436
ret_from_fork+0x219/0x490 arch/x86/kernel/process.c:158
ret_from_fork_asm+0x1a/0x30 arch/x86/entry/entry_64.S:245
BUG: memory leak
unreferenced object 0xffff8881024bee00 (size 512):
comm "softirq", pid 0, jiffies 4294937453
hex dump (first 32 bytes):
70 66 90 2a 81 88 ff ff 20 00 b4 81 ff ff ff ff pf.*.... .......
00 12 04 00 81 88 ff ff 3c 00 00 00 00 00 00 00 ........<.......
backtrace (crc 3d09b414):
kmemleak_alloc_recursive include/linux/kmemleak.h:44 [inline]
slab_post_alloc_hook mm/slub.c:4733 [inline]
slab_alloc_node mm/slub.c:5033 [inline]
__do_kmalloc_node mm/slub.c:5530 [inline]
__kmalloc_flags_noprof+0x42c/0x570 mm/slub.c:5719
_kmalloc_flags_noprof mm/slab.h:55 [inline]
__alloc_empty_sheaf mm/slub.c:2854 [inline]
alloc_empty_sheaf mm/slub.c:2874 [inline]
__pcs_replace_full_main+0xf7/0x330 mm/slub.c:6094
free_to_pcs mm/slub.c:6147 [inline]
slab_free mm/slub.c:6633 [inline]
kfree+0x353/0x390 mm/slub.c:6923
security_task_free+0x6e/0x130 security/security.c:2827
__put_task_struct+0x9f/0x260 kernel/fork.c:792
rcu_do_batch kernel/rcu/tree.c:2650 [inline]
rcu_core+0x269/0x720 kernel/rcu/tree.c:2919
handle_softirqs+0x114/0x330 kernel/softirq.c:669
__do_softirq kernel/softirq.c:703 [inline]
invoke_softirq kernel/softirq.c:542 [inline]
__irq_exit_rcu+0xb5/0xf0 kernel/softirq.c:786
instr_sysvec_irq_work arch/x86/kernel/irq_work.c:17 [inline]
sysvec_irq_work+0x73/0x80 arch/x86/kernel/irq_work.c:17
asm_sysvec_irq_work+0x1a/0x20 arch/x86/include/asm/idtentry.h:710
connection error: failed to recv *flatrpc.ExecutorMessageRawT: EOF
** replaying previous printk message **
* Struck through repros no longer work on HEAD.