Technical Breakdown of the Root Cause
1. Involved Functions, Structs, and Files
- Source Files:
- Key Functions:
- Key Structures and Locks:
struct scsi_device (sdev->request_queue, sdev->budget_map)struct request_queue (q->io_lockdep_map, q->mq_freeze_depth, q->mq_freeze_wq)fs_reclaim (lockdep map for memory allocation with reclaim capability)
2. Triggering Mechanism and Invariant Violation
In the block layer, freezing a request queue (blk_mq_freeze_queue()) stops new I/O requests from being processed on that queue until it is unfrozen (blk_mq_unfreeze_queue()). In lockdep, blk_freeze_acquire_lock() models this freeze window by acquiring &q->io_lockdep_map exclusively. Conversely, when submitting I/O to a block device, bio_queue_enter() acquires &q->io_lockdep_map for read (or waits on q->mq_freeze_wq in __bio_queue_enter() if q->mq_freeze_depth > 0).
In drivers/scsi/scsi_scan.c:scsi_realloc_sdev_budget_map():
if (need_free) {
blk_mq_freeze_queue(sdev->request_queue);
sb_backup = sdev->budget_map;
}
ret = sbitmap_init_node(&sdev->budget_map,
scsi_device_max_queue_depth(sdev),
new_shift, GFP_KERNEL,
sdev->request_queue->node, false, true);
if (!ret)
sbitmap_resize(&sdev->budget_map, depth);
if (need_free) {
if (ret)
sdev->budget_map = sb_backup;
else
sbitmap_free(&sb_backup);
ret = 0;
blk_mq_unfreeze_queue(sdev->request_queue);
}
When need_free is true (when reallocating the budget map due to a queue depth/shift change after ->device_configure in scsi_add_lun()), scsi_realloc_sdev_budget_map():
- Freezes
sdev->request_queue, acquiring &q->io_lockdep_map.
- Calls
sbitmap_init_node(&sdev->budget_map, ..., GFP_KERNEL, ...).
- Inside
sbitmap_init_node(), memory is allocated using kvzalloc_node(..., GFP_KERNEL, ...). Because the allocation uses GFP_KERNEL, lockdep marks might_alloc() and acquires the pseudo-lock fs_reclaim.
This creates the lock dependency:
$$&q\text{->io_lockdep_map} \longrightarrow \text{fs_reclaim}$$
3. How the Circular Locking Dependency Manifests
When system memory pressure occurs (in this case triggered by storing data into shmem/tmpfs), background reclaim or direct reclaim runs:
- The reclaim thread (
kswapd0 or direct reclaim) acquires fs_reclaim.
- To reclaim dirty pages or anonymous folios, it invokes
swap_writepage() or filesystem writeback to the swap partition or backing disk residing on that SCSI device.
- The I/O submission path enters
blk_mq_submit_bio() $\rightarrow$ bio_queue_enter(), which attempts to acquire &q->io_lockdep_map (and in real execution without lockdep, blocks waiting for q->mq_freeze_depth == 0 in __bio_queue_enter()).
This creates the inverse lock dependency:
$$\text{fs_reclaim} \longrightarrow &q\text{->io_lockdep_map}$$
Because the queue freeze was held across a GFP_KERNEL allocation, lockdep detects the circular dependency:
- Thread 1 (SCSI probe / realloc): Holds queue frozen ($&q\text{->io_lockdep_map}$) $\rightarrow$ waiting on memory allocation ($\text{fs_reclaim}$).
- Thread 2 (kswapd / reclaim): Holds $\text{fs_reclaim}$ $\rightarrow$ waiting for the queue to unfreeze ($&q\text{->io_lockdep_map}$).
If direct reclaim is invoked on the thread holding the queue frozen and direct reclaim attempts to issue writeback to that same queue, an actual deadlock occurs.
4. Remediation
The allocation of the new sbitmap should either:
- Take place before freezing the queue (e.g., allocate into a new/temporary
struct sbitmap outside the freeze section, then freeze the queue, swap in the new map, unfreeze, and free the old map), or
- Use
GFP_NOIO / memalloc_noio_save() to prohibit memory reclaim from recursing back into block I/O while the queue is frozen.