Root Cause Analysis
1. Relevant Files, Functions, and Data Structures
- Source File:
block/bio.c
- Functions Involved:
- Data Structures & Fields:
struct bio_alloc_cache (block/bio.c:31):
struct bio *free_list: list of cached bios recycled from task context.
struct bio *free_list_irq: list of cached bios recycled from hardirq context.
unsigned int nr: number of bios currently in free_list (offset 16 / 0x10).
unsigned int nr_irq: number of bios currently in free_list_irq (offset 20 / 0x14).
2. Mechanism and Lifecycle Rules
In block/bio.c, struct bio_alloc_cache provides a per-CPU cache for allocated bios when REQ_ALLOC_CACHE is used. Because bio_put() can be called from different execution contexts:
- Task context (
in_task()) recycles bios onto cache->free_list and increments cache->nr.
- Hardirq context (
in_hardirq()) recycles bios onto cache->free_list_irq and increments cache->nr_irq.
- Any other context (such as softirq context, where
!in_task() && !in_hardirq()) cannot use the per-CPU cache and must fall back to freeing the bio via bio_free(bio).
In bio_put_percpu_cache(struct bio *bio):
static inline void bio_put_percpu_cache(struct bio *bio)
{
struct bio_alloc_cache *cache;
cache = per_cpu_ptr(bio->bi_pool->cache, get_cpu());
if (READ_ONCE(cache->nr_irq) + cache->nr > ALLOC_CACHE_MAX)
goto out_free;
if (in_task()) {
bio_uninit(bio);
bio->bi_next = cache->free_list;
/* Not necessary but helps not to iopoll already freed bios */
bio->bi_bdev = NULL;
cache->free_list = bio;
cache->nr++;
kmemleak_free(bio_slab_addr(bio));
} else if (in_hardirq()) {
lockdep_assert_irqs_disabled();
bio_uninit(bio);
bio->bi_next = cache->free_list_irq;
cache->free_list_irq = bio;
cache->nr_irq++;
kmemleak_free(bio_slab_addr(bio));
} else {
goto out_free;
}
put_cpu();
return;
out_free:
put_cpu();
bio_free(bio);
}
The function obtains the per-CPU cache via get_cpu(), which disables preemption (preempt_disable()), but does not disable local interrupts (local_irq_save()) or softirqs (local_bh_disable()).
3. How the Data Race is Triggered
-
Task Context Write:
A thread running in task context on CPU 1 (in the report, task 3788 executing an I/O completion or failure path via submit_bio_noacct -> bio_endio -> mpage_read_end_io -> bio_put) calls bio_put() on a bio with REQ_ALLOC_CACHE.
It enters bio_put_percpu_cache(), matches if (in_task()), puts the bio on cache->free_list, and performs cache->nr++ at line 834 without disabling interrupts.
-
Interrupt Interruption on the Same CPU:
While task context is executing around line 834, a hardware interrupt fires on CPU 1. When the hardware interrupt finishes, __irq_exit_rcu() invokes pending softirqs because irq_count() == HARDIRQ_OFFSET (preemption disablement via get_cpu() increments PREEMPT_MASK, but does not set SOFTIRQ_MASK, allowing softirqs to execute on the same CPU on the interrupt exit path).
-
Softirq Execution and Read Race:
Softirq processing runs blk_done_softirq (e.g., servicing SCSI completion scsi_end_request -> bio_endio -> journal_end_buffer_io_sync -> bio_endio_bh -> bio_put).
bio_put() is called for another bio that also has REQ_ALLOC_CACHE set.
bio_put() calls bio_put_percpu_cache().
-
Data Race on cache->nr:
Before checking whether the caller context can even use the cache (in_task() or in_hardirq()), line 825 unconditionally evaluates:
if (READ_ONCE(cache->nr_irq) + cache->nr > ALLOC_CACHE_MAX)
This performs an uninstrumented, plain read of cache->nr from interrupt context on CPU 1, directly racing with the write to cache->nr (cache->nr++) by the interrupted task context on the same CPU.
Additionally, softirq context cannot use the cache at all (lines 828 and 836 evaluate to false, falling through to goto out_free;), making the access to cache->nr in softirq context completely unnecessary. A similar unsynchronized read of cache->nr also occurs when bio_put_percpu_cache() is invoked from hardirq context without READ_ONCE(cache->nr).