Root Cause Analysis
1. Functions, Structs, Fields, and Files Involved
- Files:
- Functions & Methods:
- Structs & Fields:
struct request_queue: mq_freeze_owner, mq_freeze_owner_depth, mq_freeze_disk_dead, mq_freeze_queue_dying, io_lockdep_map, q_lockdep_map, debugfs_mutexstruct gendisk: state (GD_DEAD)
2. Mechanism of the Bug and How It Is Triggered
Lockdep detected a potential circular locking dependency:
$$&q\to\text{debugfs_mutex} \longrightarrow &sb\to\text{s_type}\to\text{i_mutex_key} \longrightarrow \text{fs_reclaim} \longrightarrow &q\to\text{q_usage_counter(io)} \longrightarrow &q\to\text{debugfs_mutex}$$
The dependency chain is constructed as follows:
fs_reclaim $\to$ &q->q_usage_counter(io) (q->io_lockdep_map):
In block/blk-core.c:504-507, blk_alloc_queue() explicitly trains lockdep on lock ordering between memory reclaim and queue freezing via:
fs_reclaim_acquire(GFP_KERNEL);
rwsem_acquire_read(&q->io_lockdep_map, 0, 0, _RET_IP_);
rwsem_release(&q->io_lockdep_map, _RET_IP_);
fs_reclaim_release(GFP_KERNEL);
&sb->s_type->i_mutex_key $\to$ fs_reclaim:
Filesystem/debugfs operations (e.g. during debugfs_create_dir() via simple_start_creating() / lookup_one_qstr_excl()) hold the parent directory's inode i_rwsem while performing GFP_KERNEL slab allocations, which can invoke fs_reclaim.&q->debugfs_mutex $\to$ &sb->s_type->i_mutex_key:
In blk_register_queue() (block/blk-sysfs.c:967), blk_debugfs_lock(q) acquires q->debugfs_mutex before creating debugfs_create_dir(disk->disk_name, ...), acquiring the debugfs directory inode rwsem while holding q->debugfs_mutex.- The reverse edge (
&q->q_usage_counter(io) $\to$ &q->debugfs_mutex):
In wbt_init_enable_default() (block/blk-wbt.c:788-796), the kernel calls:
wbt_init(disk, rwb);
memflags = blk_debugfs_lock(q);
blk_mq_debugfs_register_rq_qos(q);
blk_debugfs_unlock(q, memflags);
Inside wbt_init() $\to$ rq_qos_add() (block/blk-rq-qos.c:341), memflags = blk_mq_freeze_queue(q) is called, which invokes blk_freeze_acquire_lock(q). Under CONFIG_LOCKDEP, this acquires q->io_lockdep_map. At line 349, rq_qos_add() calls blk_mq_unfreeze_queue(q, memflags).
However, as indicated by the log device-mapper: ioctl: device has been removed from the dev hash table. at drivers/md/dm-ioctl.c:1680, a concurrent device removal operation (dev_remove() $\to$ dm_destroy() $\to$ __dm_destroy() $\to$ blk_mark_disk_dead()) was executing against the same mapped device while table_load() was setting up the queue.
blk_mark_disk_dead() sets QUEUE_FLAG_DYING and GD_DEAD (test_and_set_bit(GD_DEAD, &disk->state)). In block/blk-mq.c and block/blk.h, lockdep state tracking for queue freezing maintains:
static inline void blk_freeze_acquire_lock(struct request_queue *q)
{
if (!q->mq_freeze_disk_dead)
rwsem_acquire(&q->io_lockdep_map, 0, 1, _RET_IP_);
if (!q->mq_freeze_queue_dying)
rwsem_acquire(&q->q_lockdep_map, 0, 1, _RET_IP_);
}
static inline void blk_unfreeze_release_lock(struct request_queue *q)
{
if (!q->mq_freeze_queue_dying)
rwsem_release(&q->q_lockdep_map, _RET_IP_);
if (!q->mq_freeze_disk_dead)
rwsem_release(&q->io_lockdep_map, _RET_IP_);
}
Because the queue state flags (mq_freeze_disk_dead, mq_freeze_queue_dying) or freeze owner tracking (mq_freeze_owner, mq_freeze_owner_depth) are stored on the shared request_queue struct, concurrent state modifications during the window between freeze and unfreeze lead to an imbalance: blk_freeze_acquire_lock() acquired the lockdep maps, but blk_unfreeze_release_lock() skipped rwsem_release(&q->io_lockdep_map) (or blk_unfreeze_check_owner() returned false).
As a result, lockdep still believed the task held &q->q_usage_counter(io) when wbt_init_enable_default() immediately reached line 794 and called blk_debugfs_lock(q) (which acquires q->debugfs_mutex), triggering the circular locking dependency splat.
3. Production Runtime Consequences & Classification Rationales
- MemoryCorruption:
false
This is a lockdep warning. There are no out-of-bounds memory accesses, use-after-free conditions, invalid frees, or memory safety violations.
- InfoLeak:
"none"
No uninitialized kernel memory or pointer values are disclosed to userspace or external observers.
- RawDenialOfService:
false
In a production kernel built without CONFIG_LOCKDEP:
- The synthetic
q->io_lockdep_map does not exist.
blk_mq_freeze_queue() increments q->mq_freeze_depth and kills the percpu refcount, while blk_mq_unfreeze_queue() cleanly decrements q->mq_freeze_depth back to zero, resurrects q->q_usage_counter, and wakes up waiters on q->mq_freeze_wq.blk_debugfs_lock() enters memalloc_noio_save(), preventing filesystem direct reclaim from entering the block layer while q->debugfs_mutex is held.- In
table_load(), when hc is discovered to be removed from the hash table, the code safely aborts, frees the pending table, and returns -ENXIO.
- No deadlock, hang, BUG, or crash occurs in production.
- IsNetCandidate:
false
The issue is entirely within the block layer core, blk-mq, blk-wbt, and Device Mapper (dm-ioctl). No networking components are involved.
- IsKVMCandidate:
false
No virtualization, hypervisor, or KVM subsystem interfaces are involved.