syzbot


possible deadlock in siw_create_listen (2)

Status: upstream: reported on 2024/09/26 13:34
Subsystems: rdma
[Documentation on labels]
Reported-by: syzbot+3eb27595de9aa3cf63c3@syzkaller.appspotmail.com
First crash: 236d, last: 90d
Discussions (2)
Title Replies (including bot) Last reply
[syzbot] Monthly rdma report (Jan 2025) 0 (1) 2025/01/30 13:12
[syzbot] [rdma?] possible deadlock in siw_create_listen (2) 0 (1) 2024/09/26 13:34
Similar bugs (1)
Kernel Title Repro Cause bisect Fix bisect Count Last Reported Patched Status
upstream possible deadlock in siw_create_listen rdma C done 1320 1865d 1897d 0/28 closed as dup on 2020/03/09 17:20

Sample crash report:
lo speed is unknown, defaulting to 1000
lo speed is unknown, defaulting to 1000
iwpm_register_pid: Unable to send a nlmsg (client = 2)
======================================================
WARNING: possible circular locking dependency detected
6.14.0-rc2-syzkaller-ga64dcfb451e2 #0 Not tainted
------------------------------------------------------
syz.1.570/9837 is trying to acquire lock:
ffff0000f3684218 (sk_lock-AF_INET){+.+.}-{0:0}, at: siw_create_listen+0x164/0xe50 drivers/infiniband/sw/siw/siw_cm.c:1777

but task is already holding lock:
ffff8000923374e8 (lock#7){+.+.}-{4:4}, at: cma_add_one+0x510/0xab4 drivers/infiniband/core/cma.c:5370

which lock already depends on the new lock.


the existing dependency chain (in reverse order) is:

-> #3 (lock#7){+.+.}-{4:4}:
       __mutex_lock_common+0x1f0/0x24b8 kernel/locking/mutex.c:585
       __mutex_lock kernel/locking/mutex.c:730 [inline]
       mutex_lock_nested+0x2c/0x38 kernel/locking/mutex.c:782
       cma_init+0x2c/0x158 drivers/infiniband/core/cma.c:5454
       do_one_initcall+0x254/0xaa4 init/main.c:1257
       do_initcall_level+0x154/0x214 init/main.c:1319
       do_initcalls+0x84/0xf4 init/main.c:1335
       do_basic_setup+0x8c/0xa0 init/main.c:1354
       kernel_init_freeable+0x324/0x478 init/main.c:1568
       kernel_init+0x24/0x2a0 init/main.c:1457
       ret_from_fork+0x10/0x20 arch/arm64/kernel/entry.S:862

-> #2 (rtnl_mutex){+.+.}-{4:4}:
       __mutex_lock_common+0x1f0/0x24b8 kernel/locking/mutex.c:585
       __mutex_lock kernel/locking/mutex.c:730 [inline]
       mutex_lock_nested+0x2c/0x38 kernel/locking/mutex.c:782
       rtnl_lock+0x20/0x2c net/core/rtnetlink.c:79
       start_sync_thread+0xe0/0x24bc net/netfilter/ipvs/ip_vs_sync.c:1761
       do_ip_vs_set_ctl+0x550/0xd70 net/netfilter/ipvs/ip_vs_ctl.c:2732
       nf_setsockopt+0x270/0x290 net/netfilter/nf_sockopt.c:101
       ip_setsockopt+0x118/0x128 net/ipv4/ip_sockglue.c:1424
       tcp_setsockopt+0xcc/0xe8 net/ipv4/tcp.c:4030
       sock_common_setsockopt+0xb0/0xcc net/core/sock.c:3837
       smc_setsockopt+0x1f8/0xd0c net/smc/af_smc.c:3081
       do_sock_setsockopt+0x2a0/0x4e0 net/socket.c:2303
       __sys_setsockopt net/socket.c:2328 [inline]
       __do_sys_setsockopt net/socket.c:2334 [inline]
       __se_sys_setsockopt net/socket.c:2331 [inline]
       __arm64_sys_setsockopt+0x170/0x1e0 net/socket.c:2331
       __invoke_syscall arch/arm64/kernel/syscall.c:35 [inline]
       invoke_syscall+0x98/0x2b8 arch/arm64/kernel/syscall.c:49
       el0_svc_common+0x130/0x23c arch/arm64/kernel/syscall.c:132
       do_el0_svc+0x48/0x58 arch/arm64/kernel/syscall.c:151
       el0_svc+0x54/0x168 arch/arm64/kernel/entry-common.c:744
       el0t_64_sync_handler+0x84/0x108 arch/arm64/kernel/entry-common.c:762
       el0t_64_sync+0x198/0x19c arch/arm64/kernel/entry.S:600

-> #1 (&smc->clcsock_release_lock){+.+.}-{4:4}:
       __mutex_lock_common+0x1f0/0x24b8 kernel/locking/mutex.c:585
       __mutex_lock kernel/locking/mutex.c:730 [inline]
       mutex_lock_nested+0x2c/0x38 kernel/locking/mutex.c:782
       smc_switch_to_fallback+0x48/0xa7c net/smc/af_smc.c:903
       smc_sendmsg+0xfc/0x9f8 net/smc/af_smc.c:2781
       sock_sendmsg_nosec net/socket.c:718 [inline]
       __sock_sendmsg net/socket.c:733 [inline]
       __sys_sendto+0x360/0x4d8 net/socket.c:2187
       __do_sys_sendto net/socket.c:2194 [inline]
       __se_sys_sendto net/socket.c:2190 [inline]
       __arm64_sys_sendto+0xd8/0xf8 net/socket.c:2190
       __invoke_syscall arch/arm64/kernel/syscall.c:35 [inline]
       invoke_syscall+0x98/0x2b8 arch/arm64/kernel/syscall.c:49
       el0_svc_common+0x130/0x23c arch/arm64/kernel/syscall.c:132
       do_el0_svc+0x48/0x58 arch/arm64/kernel/syscall.c:151
       el0_svc+0x54/0x168 arch/arm64/kernel/entry-common.c:744
       el0t_64_sync_handler+0x84/0x108 arch/arm64/kernel/entry-common.c:762
       el0t_64_sync+0x198/0x19c arch/arm64/kernel/entry.S:600

-> #0 (sk_lock-AF_INET){+.+.}-{0:0}:
       check_prev_add kernel/locking/lockdep.c:3163 [inline]
       check_prevs_add kernel/locking/lockdep.c:3282 [inline]
       validate_chain kernel/locking/lockdep.c:3906 [inline]
       __lock_acquire+0x34f0/0x7904 kernel/locking/lockdep.c:5228
       lock_acquire+0x23c/0x724 kernel/locking/lockdep.c:5851
       lock_sock_nested net/core/sock.c:3645 [inline]
       lock_sock include/net/sock.h:1624 [inline]
       sock_set_reuseaddr+0x58/0x154 net/core/sock.c:788
       siw_create_listen+0x164/0xe50 drivers/infiniband/sw/siw/siw_cm.c:1777
       iw_cm_listen+0x14c/0x204 drivers/infiniband/core/iwcm.c:585
       cma_iw_listen drivers/infiniband/core/cma.c:2684 [inline]
       rdma_listen+0x8d0/0xb64 drivers/infiniband/core/cma.c:3969
       cma_listen_on_dev+0x31c/0x648 drivers/infiniband/core/cma.c:2743
       cma_add_one+0x5ec/0xab4 drivers/infiniband/core/cma.c:5373
       add_client_context+0x45c/0x7d0 drivers/infiniband/core/device.c:711
       enable_device_and_get+0x1a8/0x3e8 drivers/infiniband/core/device.c:1322
       ib_register_device+0xe84/0x110c drivers/infiniband/core/device.c:1433
       siw_device_register drivers/infiniband/sw/siw/siw_main.c:72 [inline]
       siw_newlink+0x77c/0xb84 drivers/infiniband/sw/siw/siw_main.c:431
       nldev_newlink+0x47c/0x54c drivers/infiniband/core/nldev.c:1795
       rdma_nl_rcv_skb drivers/infiniband/core/netlink.c:239 [inline]
       rdma_nl_rcv+0x5c4/0x858 drivers/infiniband/core/netlink.c:259
       netlink_unicast_kernel net/netlink/af_netlink.c:1322 [inline]
       netlink_unicast+0x668/0x8a4 net/netlink/af_netlink.c:1348
       netlink_sendmsg+0x7b4/0xa9c net/netlink/af_netlink.c:1892
       sock_sendmsg_nosec net/socket.c:718 [inline]
       __sock_sendmsg net/socket.c:733 [inline]
       ____sys_sendmsg+0x570/0x87c net/socket.c:2573
       ___sys_sendmsg net/socket.c:2627 [inline]
       __sys_sendmsg+0x238/0x304 net/socket.c:2659
       __do_sys_sendmsg net/socket.c:2664 [inline]
       __se_sys_sendmsg net/socket.c:2662 [inline]
       __arm64_sys_sendmsg+0x80/0x94 net/socket.c:2662
       __invoke_syscall arch/arm64/kernel/syscall.c:35 [inline]
       invoke_syscall+0x98/0x2b8 arch/arm64/kernel/syscall.c:49
       el0_svc_common+0x130/0x23c arch/arm64/kernel/syscall.c:132
       do_el0_svc+0x48/0x58 arch/arm64/kernel/syscall.c:151
       el0_svc+0x54/0x168 arch/arm64/kernel/entry-common.c:744
       el0t_64_sync_handler+0x84/0x108 arch/arm64/kernel/entry-common.c:762
       el0t_64_sync+0x198/0x19c arch/arm64/kernel/entry.S:600

other info that might help us debug this:

Chain exists of:
  sk_lock-AF_INET --> rtnl_mutex --> lock#7

 Possible unsafe locking scenario:

       CPU0                    CPU1
       ----                    ----
  lock(lock#7);
                               lock(rtnl_mutex);
                               lock(lock#7);
  lock(sk_lock-AF_INET);

 *** DEADLOCK ***

6 locks held by syz.1.570/9837:
 #0: ffff800097b7ec98 (&rdma_nl_types[idx].sem){.+.+}-{4:4}, at: rdma_nl_rcv_msg drivers/infiniband/core/netlink.c:164 [inline]
 #0: ffff800097b7ec98 (&rdma_nl_types[idx].sem){.+.+}-{4:4}, at: rdma_nl_rcv_skb drivers/infiniband/core/netlink.c:239 [inline]
 #0: ffff800097b7ec98 (&rdma_nl_types[idx].sem){.+.+}-{4:4}, at: rdma_nl_rcv+0x330/0x858 drivers/infiniband/core/netlink.c:259
 #1: ffff800092323850 (link_ops_rwsem){++++}-{4:4}, at: nldev_newlink+0x334/0x54c drivers/infiniband/core/nldev.c:1785
 #2: ffff800092314070 (devices_rwsem){++++}-{4:4}, at: enable_device_and_get+0x104/0x3e8 drivers/infiniband/core/device.c:1312
 #3: ffff800092314370 (clients_rwsem){++++}-{4:4}, at: enable_device_and_get+0x160/0x3e8 drivers/infiniband/core/device.c:1320
 #4: ffff0000f125c5e0 (&device->client_data_rwsem){++++}-{4:4}, at: add_client_context+0x424/0x7d0 drivers/infiniband/core/device.c:709
 #5: ffff8000923374e8 (lock#7){+.+.}-{4:4}, at: cma_add_one+0x510/0xab4 drivers/infiniband/core/cma.c:5370

stack backtrace:
CPU: 1 UID: 0 PID: 9837 Comm: syz.1.570 Not tainted 6.14.0-rc2-syzkaller-ga64dcfb451e2 #0
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 12/27/2024
Call trace:
 show_stack+0x2c/0x3c arch/arm64/kernel/stacktrace.c:466 (C)
 __dump_stack lib/dump_stack.c:94 [inline]
 dump_stack_lvl+0xe4/0x150 lib/dump_stack.c:120
 dump_stack+0x1c/0x28 lib/dump_stack.c:129
 print_circular_bug+0x154/0x1c0 kernel/locking/lockdep.c:2076
 check_noncircular+0x310/0x404 kernel/locking/lockdep.c:2208
 check_prev_add kernel/locking/lockdep.c:3163 [inline]
 check_prevs_add kernel/locking/lockdep.c:3282 [inline]
 validate_chain kernel/locking/lockdep.c:3906 [inline]
 __lock_acquire+0x34f0/0x7904 kernel/locking/lockdep.c:5228
 lock_acquire+0x23c/0x724 kernel/locking/lockdep.c:5851
 lock_sock_nested net/core/sock.c:3645 [inline]
 lock_sock include/net/sock.h:1624 [inline]
 sock_set_reuseaddr+0x58/0x154 net/core/sock.c:788
 siw_create_listen+0x164/0xe50 drivers/infiniband/sw/siw/siw_cm.c:1777
 iw_cm_listen+0x14c/0x204 drivers/infiniband/core/iwcm.c:585
 cma_iw_listen drivers/infiniband/core/cma.c:2684 [inline]
 rdma_listen+0x8d0/0xb64 drivers/infiniband/core/cma.c:3969
 cma_listen_on_dev+0x31c/0x648 drivers/infiniband/core/cma.c:2743
 cma_add_one+0x5ec/0xab4 drivers/infiniband/core/cma.c:5373
 add_client_context+0x45c/0x7d0 drivers/infiniband/core/device.c:711
 enable_device_and_get+0x1a8/0x3e8 drivers/infiniband/core/device.c:1322
 ib_register_device+0xe84/0x110c drivers/infiniband/core/device.c:1433
 siw_device_register drivers/infiniband/sw/siw/siw_main.c:72 [inline]
 siw_newlink+0x77c/0xb84 drivers/infiniband/sw/siw/siw_main.c:431
 nldev_newlink+0x47c/0x54c drivers/infiniband/core/nldev.c:1795
 rdma_nl_rcv_skb drivers/infiniband/core/netlink.c:239 [inline]
 rdma_nl_rcv+0x5c4/0x858 drivers/infiniband/core/netlink.c:259
 netlink_unicast_kernel net/netlink/af_netlink.c:1322 [inline]
 netlink_unicast+0x668/0x8a4 net/netlink/af_netlink.c:1348
 netlink_sendmsg+0x7b4/0xa9c net/netlink/af_netlink.c:1892
 sock_sendmsg_nosec net/socket.c:718 [inline]
 __sock_sendmsg net/socket.c:733 [inline]
 ____sys_sendmsg+0x570/0x87c net/socket.c:2573
 ___sys_sendmsg net/socket.c:2627 [inline]
 __sys_sendmsg+0x238/0x304 net/socket.c:2659
 __do_sys_sendmsg net/socket.c:2664 [inline]
 __se_sys_sendmsg net/socket.c:2662 [inline]
 __arm64_sys_sendmsg+0x80/0x94 net/socket.c:2662
 __invoke_syscall arch/arm64/kernel/syscall.c:35 [inline]
 invoke_syscall+0x98/0x2b8 arch/arm64/kernel/syscall.c:49
 el0_svc_common+0x130/0x23c arch/arm64/kernel/syscall.c:132
 do_el0_svc+0x48/0x58 arch/arm64/kernel/syscall.c:151
 el0_svc+0x54/0x168 arch/arm64/kernel/entry-common.c:744
 el0t_64_sync_handler+0x84/0x108 arch/arm64/kernel/entry-common.c:762
 el0t_64_sync+0x198/0x19c arch/arm64/kernel/entry.S:600
infiniband syz2: RDMA CMA: cma_listen_on_dev, error -98
lo speed is unknown, defaulting to 1000
lo speed is unknown, defaulting to 1000
lo speed is unknown, defaulting to 1000
lo speed is unknown, defaulting to 1000
lo speed is unknown, defaulting to 1000

Crashes (6):
Time Kernel Commit Syzkaller Config Log Report Syz repro C repro VM info Assets (help?) Manager Title
2025/02/15 12:46 git://git.kernel.org/pub/scm/linux/kernel/git/arm64/linux.git for-kernelci a64dcfb451e2 40a34ec9 .config console log report info [disk image] [vmlinux] [kernel image] ci-upstream-gce-arm64 possible deadlock in siw_create_listen
2025/01/28 15:41 git://git.kernel.org/pub/scm/linux/kernel/git/arm64/linux.git for-kernelci 1950a0af2d55 f5427d7c .config console log report info [disk image] [vmlinux] [kernel image] ci-upstream-gce-arm64 possible deadlock in siw_create_listen
2025/01/21 05:47 git://git.kernel.org/pub/scm/linux/kernel/git/arm64/linux.git for-kernelci 1950a0af2d55 6e87cfa2 .config console log report info [disk image] [vmlinux] [kernel image] ci-upstream-gce-arm64 possible deadlock in siw_create_listen
2024/12/15 07:16 git://git.kernel.org/pub/scm/linux/kernel/git/arm64/linux.git for-kernelci 2e7aff49b5da 7cbfbb3a .config console log report info [disk image] [vmlinux] [kernel image] ci-upstream-gce-arm64 possible deadlock in siw_create_listen
2024/12/11 05:32 git://git.kernel.org/pub/scm/linux/kernel/git/arm64/linux.git for-kernelci 146ff2c261f3 cfc402b4 .config console log report info [disk image] [vmlinux] [kernel image] ci-upstream-gce-arm64 possible deadlock in siw_create_listen
2024/09/22 13:22 git://git.kernel.org/pub/scm/linux/kernel/git/arm64/linux.git for-kernelci 5f5673607153 6f888b75 .config console log report info [disk image] [vmlinux] [kernel image] ci-upstream-gce-arm64 possible deadlock in siw_create_listen
* Struck through repros no longer work on HEAD.