======================================================
WARNING: possible circular locking dependency detected
syzkaller #0 Not tainted
------------------------------------------------------
kworker/0:1H/53 is trying to acquire lock:
ffff888026adfd38 ((wq_completion)nvmet-wq){+.+.}-{0:0}, at: __flush_workqueue+0x14c/0x1380 kernel/workqueue.c:2812

but task is already holding lock:
ffffc90000be7d00 ((work_completion)(&queue->io_work)){+.+.}-{0:0}, at: process_one_work+0x7b4/0x1160 kernel/workqueue.c:2267

which lock already depends on the new lock.


the existing dependency chain (in reverse order) is:

-> #2 ((work_completion)(&queue->io_work)){+.+.}-{0:0}:
       __flush_work+0x122/0xac0 kernel/workqueue.c:3072
       __cancel_work_timer+0x3f4/0x560 kernel/workqueue.c:3163
       nvmet_tcp_release_queue_work+0x2ef/0xf30 drivers/nvme/target/tcp.c:1558
       process_one_work+0x8ab/0x1160 kernel/workqueue.c:2292
       worker_thread+0xaf5/0x12a0 kernel/workqueue.c:2439
       kthread+0x29d/0x330 kernel/kthread.c:376
       ret_from_fork+0x1f/0x30 arch/x86/entry/entry_64.S:295

-> #1 ((work_completion)(&queue->release_work)){+.+.}-{0:0}:
       process_one_work+0x7ec/0x1160 kernel/workqueue.c:2268
       worker_thread+0xaf5/0x12a0 kernel/workqueue.c:2439
       kthread+0x29d/0x330 kernel/kthread.c:376
       ret_from_fork+0x1f/0x30 arch/x86/entry/entry_64.S:295

-> #0 ((wq_completion)nvmet-wq){+.+.}-{0:0}:
       check_prev_add kernel/locking/lockdep.c:3162 [inline]
       check_prevs_add kernel/locking/lockdep.c:3281 [inline]
       validate_chain kernel/locking/lockdep.c:3897 [inline]
       __lock_acquire+0x3a03/0x7ea0 kernel/locking/lockdep.c:5171
       lock_acquire+0x1bc/0x4a0 kernel/locking/lockdep.c:5790
       __flush_workqueue+0x165/0x1380 kernel/workqueue.c:2812
       nvmet_tcp_install_queue+0x7e/0x380 drivers/nvme/target/tcp.c:1915
       nvmet_install_queue+0x335/0x760 drivers/nvme/target/fabrics-cmd.c:185
       nvmet_execute_admin_connect+0x332/0x7e0 drivers/nvme/target/fabrics-cmd.c:266
       nvmet_tcp_execute_request drivers/nvme/target/tcp.c:599 [inline]
       nvmet_tcp_try_recv_data drivers/nvme/target/tcp.c:1254 [inline]
       nvmet_tcp_try_recv_one drivers/nvme/target/tcp.c:1318 [inline]
       nvmet_tcp_try_recv drivers/nvme/target/tcp.c:1344 [inline]
       nvmet_tcp_io_work+0x1a65/0x88a0 drivers/nvme/target/tcp.c:1394
       process_one_work+0x8ab/0x1160 kernel/workqueue.c:2292
       worker_thread+0xaf5/0x12a0 kernel/workqueue.c:2439
       kthread+0x29d/0x330 kernel/kthread.c:376
       ret_from_fork+0x1f/0x30 arch/x86/entry/entry_64.S:295

other info that might help us debug this:

Chain exists of:
  (wq_completion)nvmet-wq --> (work_completion)(&queue->release_work) --> (work_completion)(&queue->io_work)

 Possible unsafe locking scenario:

       CPU0                    CPU1
       ----                    ----
  lock((work_completion)(&queue->io_work));
                               lock((work_completion)(&queue->release_work));
                               lock((work_completion)(&queue->io_work));
  lock((wq_completion)nvmet-wq);

 *** DEADLOCK ***

2 locks held by kworker/0:1H/53:
 #0: ffff888026ae8538 ((wq_completion)nvmet_tcp_wq){+.+.}-{0:0}, at: process_one_work+0x7b4/0x1160 kernel/workqueue.c:2267
 #1: ffffc90000be7d00 ((work_completion)(&queue->io_work)){+.+.}-{0:0}, at: process_one_work+0x7b4/0x1160 kernel/workqueue.c:2267

stack backtrace:
CPU: 0 PID: 53 Comm: kworker/0:1H Not tainted syzkaller #0
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 09/16/2026
Workqueue: nvmet_tcp_wq nvmet_tcp_io_work
Call Trace:
 <TASK>
 dump_stack_lvl+0x188/0x24e lib/dump_stack.c:106
 check_noncircular+0x2f8/0x400 kernel/locking/lockdep.c:2215
 check_prev_add kernel/locking/lockdep.c:3162 [inline]
 check_prevs_add kernel/locking/lockdep.c:3281 [inline]
 validate_chain kernel/locking/lockdep.c:3897 [inline]
 __lock_acquire+0x3a03/0x7ea0 kernel/locking/lockdep.c:5171
 lock_acquire+0x1bc/0x4a0 kernel/locking/lockdep.c:5790
 __flush_workqueue+0x165/0x1380 kernel/workqueue.c:2812
 nvmet_tcp_install_queue+0x7e/0x380 drivers/nvme/target/tcp.c:1915
 nvmet_install_queue+0x335/0x760 drivers/nvme/target/fabrics-cmd.c:185
 nvmet_execute_admin_connect+0x332/0x7e0 drivers/nvme/target/fabrics-cmd.c:266
 nvmet_tcp_execute_request drivers/nvme/target/tcp.c:599 [inline]
 nvmet_tcp_try_recv_data drivers/nvme/target/tcp.c:1254 [inline]
 nvmet_tcp_try_recv_one drivers/nvme/target/tcp.c:1318 [inline]
 nvmet_tcp_try_recv drivers/nvme/target/tcp.c:1344 [inline]
 nvmet_tcp_io_work+0x1a65/0x88a0 drivers/nvme/target/tcp.c:1394
 process_one_work+0x8ab/0x1160 kernel/workqueue.c:2292
 worker_thread+0xaf5/0x12a0 kernel/workqueue.c:2439
 kthread+0x29d/0x330 kernel/kthread.c:376
 ret_from_fork+0x1f/0x30 arch/x86/entry/entry_64.S:295
 </TASK>
nvmet: creating nvm controller 1 for subsystem testsubsys for NQN nqn.2014-08.org.nvmexpress:uuid:6378566e-9e2d-470b-a6eb-af7e929fefb6.
tcp_conntracks: 
syn=0 ack=1 fin=0 rst=0 old=3 new=3
nf_conntrack: normal packet for ffff88807340d200
tcp_conntracks: 
syn=0 ack=1 fin=0 rst=0 old=3 new=3
tcp_conntracks: 
syn=0 ack=1 fin=0 rst=0 old=3 new=3
nf_conntrack: normal packet for ffff88807340d200
tcp_conntracks: 
syn=0 ack=1 fin=0 rst=0 old=3 new=3
tcp_conntracks: 
syn=0 ack=1 fin=0 rst=0 old=3 new=3
nf_conntrack: normal packet for ffff88807340d200
tcp_conntracks: 
syn=0 ack=1 fin=0 rst=0 old=3 new=3
tcp_conntracks: 
syn=0 ack=1 fin=0 rst=0 old=3 new=3
nf_conntrack: normal packet for ffff88807340d200
tcp_conntracks: 
syn=0 ack=1 fin=0 rst=0 old=3 new=3
tcp_conntracks: 
syn=0 ack=1 fin=0 rst=0 old=3 new=3
nf_conntrack: normal packet for ffff88807340d200
tcp_conntracks: 
syn=0 ack=1 fin=0 rst=0 old=3 new=3
tcp_conntracks: 
syn=0 ack=1 fin=0 rst=0 old=3 new=3
nf_conntrack: normal packet for ffff88807340d200
tcp_conntracks: 
syn=0 ack=1 fin=0 rst=0 old=3 new=3
tcp_conntracks: 
syn=0 ack=1 fin=0 rst=0 old=3 new=3
nf_conntrack: normal packet for ffff88807340d200
tcp_conntracks: 
syn=0 ack=1 fin=0 rst=0 old=3 new=3
tcp_conntracks: 
syn=0 ack=1 fin=0 rst=0 old=3 new=3
nf_conntrack: normal packet for ffff88807340d200
tcp_conntracks: 
syn=0 ack=1 fin=0 rst=0 old=3 new=3
tcp_conntracks: 
syn=0 ack=1 fin=0 rst=0 old=3 new=3
nf_conntrack: normal packet for ffff88807340d200
tcp_conntracks: 
syn=0 ack=1 fin=0 rst=0 old=3 new=3
tcp_conntracks: 
syn=0 ack=1 fin=0 rst=0 old=3 new=3
nf_conntrack: normal packet for ffff88807340d200
tcp_conntracks: 
syn=0 ack=1 fin=0 rst=0 old=3 new=3
tcp_conntracks: 
syn=0 ack=1 fin=0 rst=0 old=3 new=3
nf_conntrack: normal packet for ffff88807340d200
tcp_conntracks: 
syn=0 ack=1 fin=0 rst=0 old=3 new=3
tcp_conntracks: 
syn=0 ack=1 fin=0 rst=0 old=3 new=3
sd 0:0:1:0: cmd ffff8880286e1f20 CDB: 0x2a
Added buffer head 170 to ffff888025f11800
sd 0:0:1:0: cmd ffff8880286e22e0 CDB: 0x2a
Added buffer head 58 to ffff888025f11800
sd 0:0:1:0: cmd ffff8880286e26a0 CDB: 0x2a
Added buffer head 55 to ffff888025f11800
sd 0:0:1:0: cmd ffff8880286e2a60 CDB: 0x2a
Added buffer head 147 to ffff888025f11800
sd 0:0:1:0: cmd ffff8880286e2e20 CDB: 0x2a
Added buffer head 166 to ffff888025f11800
sd 0:0:1:0: cmd ffff8880286e31e0 CDB: 0x2a
Added buffer head 48 to ffff888025f11800
sd 0:0:1:0: cmd ffff8880286e35a0 CDB: 0x2a
Added buffer head 173 to ffff888025f11800
sd 0:0:1:0: cmd ffff8880286e3960 CDB: 0x2a
Added buffer head 62 to ffff888025f11800
sd 0:0:1:0: cmd ffff8880286e3d20 CDB: 0x2a
Added buffer head 134 to ffff888025f11800
sd 0:0:1:0: cmd ffff8880286e40e0 CDB: 0x2a