Root Cause Analysis

1. Overview and Key Actors

• Subsystem & Drivers: Device Mapper (drivers/md/dm-ioctl.c, drivers/md/dm.c) and Block Layer MQ/Elevator subsystem (block/elevator.c, block/blk-mq.c, block/blk.h).
• Structs & Fields:
  • struct mapped_device (holders, open_count, flags, disk, queue)
  • struct request_queue (q_usage_counter, q_lockdep_map, mq_freeze_lock, mq_freeze_depth, mq_freeze_owner, mq_freeze_owner_depth, mq_freeze_disk_dead, mq_freeze_queue_dying)
• Functions Involved:
  • table_load() (drivers/md/dm-ioctl.c)
  • dev_remove() (drivers/md/dm-ioctl.c)
  • __dm_destroy() (drivers/md/dm.c)
  • dm_setup_md_queue() (drivers/md/dm.c)
  • elevator_change() (block/elevator.c)
  • blk_freeze_set_owner(), __blk_mq_unfreeze_queue() (block/blk-mq.c)
  • blk_freeze_acquire_lock(), blk_unfreeze_release_lock() (block/blk.h)

2. Detailed Technical Mechanism & Trigger Conditions

1. Initial Setup: A mapped device is created via DM_DEV_CREATE. Userspace then initiates two concurrent/interleaved operations: DM_TABLE_LOAD (Thread A) and DM_DEV_REMOVE (Thread B).

2. Thread A (table_load):

   • Calls find_device(), which acquires a reference to the mapped device by incrementing atomic_inc(&md->holders) (note: this does not increment md->open_count).
   • For an initial table load with an immutable request-based target (such as multipath), dm_get_md_type(md) is DM_TYPE_NONE. Therefore, Thread A calls dm_setup_md_queue(md, t).
   • dm_setup_md_queue() calls add_disk(md->disk), which calls blk_register_queue().
   • blk_register_queue() invokes elevator_set_default(q) before setting QUEUE_FLAG_REGISTERED.
   • elevator_set_default() calls elevator_change(q, &ctx).
   • At line 668 in block/elevator.c:

     memflags = blk_mq_freeze_queue(q);
     

     blk_freeze_queue_start(q) calls __blk_freeze_queue_start(q, current).
   • In blk_freeze_set_owner() under q->mq_freeze_lock:
     • Since q->mq_freeze_depth == 0, it records:
       • q->mq_freeze_owner = current (Thread A)
       • q->mq_freeze_owner_depth = 1
       • q->mq_freeze_disk_dead = !blk_queue_registered(q) (which is true because QUEUE_FLAG_REGISTERED is not set yet)
       • q->mq_freeze_queue_dying = blk_queue_dying(q) (which is false because the queue is not marked dying yet)
     • q->mq_freeze_depth increments to 1, and __blk_freeze_queue_start() returns true.
   • In blk_freeze_acquire_lock(q):
     • !q->mq_freeze_disk_dead is false, so io_lockdep_map is not acquired.
     • !q->mq_freeze_queue_dying is true, so rwsem_acquire(&q->q_lockdep_map, 0, 1, _RET_IP_) is called, recording the acquisition of &q->q_usage_counter(queue) at elevator_change+0x194/0x480 block/elevator.c:668.

3. Thread B (dev_remove):

   • Concurrently, Thread B calls dev_remove().
   • It executes dm_lock_for_deletion(), which succeeds because md->open_count is 0 (Thread A only holds md->holders).
   • Thread B unlinks the mapped device from the hash tables via __hash_remove(hc) (which sets dm_set_mdptr(hc->md, NULL)).
   • Thread B calls dm_destroy(md) -> __dm_destroy(md, true).
   • In __dm_destroy() (drivers/md/dm.c:2738):

     blk_mark_disk_dead(md->disk);
     

     Defect 1: blk_mark_disk_dead(md->disk) is called before waiting for existing md->holders references to drop (while (atomic_read(&md->holders)) fsleep(1000);).
   • blk_mark_disk_dead() sets QUEUE_FLAG_DYING on the queue and calls __blk_mark_disk_dead() -> blk_queue_start_drain() -> __blk_freeze_queue_start(q, current).

4. The Unfreeze Race and Lockdep Desynchronization:

   • Back in Thread A (elevator_change()), line 683 runs:

     blk_mq_unfreeze_queue(q, memflags);
     

   • This invokes blk_mq_unfreeze_queue_nomemrestore(q):

     void blk_mq_unfreeze_queue_nomemrestore(struct request_queue *q)
     {
         if (__blk_mq_unfreeze_queue(q, false))
             blk_unfreeze_release_lock(q);
     }
     

   • In __blk_mq_unfreeze_queue(q, false):
     • It acquires q->mq_freeze_lock.
     • q->mq_freeze_depth is decremented from 1 to 0.
     • blk_unfreeze_check_owner(q) is called: q->mq_freeze_owner is cleared to NULL and q->mq_freeze_owner_depth is decremented to 0.
     • __blk_mq_unfreeze_queue() returns true.
     • Defect 2: mutex_unlock(&q->mq_freeze_lock) is released before blk_unfreeze_release_lock(q) is called.
   • Immediately upon q->mq_freeze_lock being dropped (and q->mq_freeze_depth == 0), Thread B's __blk_freeze_queue_start() enters blk_freeze_set_owner():
     • Seeing !q->mq_freeze_depth (0), Thread B sets q->mq_freeze_queue_dying = blk_queue_dying(q);, which is now true.
   • Now Thread A continues and executes blk_unfreeze_release_lock(q):

     static inline void blk_unfreeze_release_lock(struct request_queue *q)
     {
         if (!q->mq_freeze_queue_dying)
             rwsem_release(&q->q_lockdep_map, _RET_IP_);
         if (!q->mq_freeze_disk_dead)
             rwsem_release(&q->io_lockdep_map, _RET_IP_);
     }
     

   • Thread A evaluates !q->mq_freeze_queue_dying and finds it false (because Thread B overwrote q->mq_freeze_queue_dying to true).
   • Consequently, rwsem_release(&q->q_lockdep_map, _RET_IP_) is skipped!
   • In lockdep's internal bookkeeping, &q->q_usage_counter(queue) acquired at block/elevator.c:668 remains held by Thread A.

5. Return to Userspace:

   • Thread A finishes dm_setup_md_queue(), drops md_type_lock, and acquires _hash_lock.
   • Looking up the hash cell via hc = dm_get_mdptr(md), it finds hc == NULL because Thread B removed it.
   • Thread A logs "device-mapper: ioctl: device has been removed from the dev hash table.", cleans up the table, drops its reference via dm_put(md), and returns -ENXIO.
   • As Thread A exits back to userspace, lockdep detects that &q->q_usage_counter(queue) was never released, emitting: WARNING: lock held when returning to user space!.

3. Production Impact Classification

• MemoryCorruption: false The bug is a lockdep tracking desynchronization race caused by global request_queue lockdep flags being modified across thread boundaries. In production kernels without debugging sanitizers (CONFIG_LOCKDEP disabled), the rwsem_acquire/rwsem_release calls inside blk_freeze_acquire_lock and blk_unfreeze_release_lock compile to no-ops. Memory allocation, reference counting, and disk deletion proceed without memory corruption or use-after-free.
• InfoLeak: none No uninitialized data or kernel pointers are exposed to userspace.
• RawDenialOfService: false In a production kernel without CONFIG_LOCKDEP, this lockdep warning does not exist. The underlying request queue depth correctly reaches 0, Thread B waits cleanly for md->holders to drain before tearing down the gendisk and request queue via del_gendisk(), and no kernel panic, hang, deadlock, or resource leak occurs.
• IsNetCandidate: false The bug is entirely within the Device Mapper and block layer elevator/request-queue subsystem.
• IsKVMCandidate: false No virtualization or KVM subsystems are involved.