possible deadlock in rds_message_put

Status: auto-obsoleted due to no activity on 2024/10/03 22:36
Bug presence: origin:upstream
Labels: missing-backport
[Documentation on labels]
Reported-by: syzbot+4f1f1d5a96ae0676cf91@syzkaller.appspotmail.com
First crash: 1202d, last: 1202d
Fix bisection: failed (error log, bisect log)
  
▶ ▼ Bug presence (3)
Date Name Commit Repro Result
2024/03/27 linux-5.15.y (ToT) 9465fef4ae35 C [report] possible deadlock in rds_wake_sk_sleep
2023/05/16 upstream (ToT) f1fcbaa18b28 C [report] possible deadlock in rds_wake_sk_sleep
2024/03/27 upstream (ToT) 7033999ecd7b C Didn't crash
▶ ▼ Similar bugs (1)
Kernel Title Rank 🛈 Repro Cause bisect Fix bisect Count Last Reported Patched Status
upstream possible deadlock in rds_message_put rds 4 syz, C done error 5 1022d 1333d 25/29 fixed on 2024/03/27 19:12
▶ ▼ Last patch testing requests (1)
Created Duration User Patch Repo Result
2024/10/03 22:16 19m retest repro linux-5.15.y no crash log
▶ ▼ Fix bisection attempts (5)
Created Duration User Patch Repo Result
2024/09/17 05:07 1m fix candidate upstream error job log
2024/07/30 15:39 1h16m fix candidate upstream error job log
2024/06/14 05:24 0m fix candidate upstream error job log
2024/05/13 23:50 3m fix candidate upstream error job log
2023/06/14 13:06 0m bisect fix linux-5.15.y error job log

Sample crash report:
======================================================
WARNING: possible circular locking dependency detected
5.15.111-syzkaller #0 Not tainted
------------------------------------------------------
syz-executor298/4087 is trying to acquire lock:
ffff0000d021d900 (&rm->m_rs_lock){....}-{2:2}, at: rds_message_purge net/rds/message.c:138 [inline]
ffff0000d021d900 (&rm->m_rs_lock){....}-{2:2}, at: rds_message_put+0x138/0xa0c net/rds/message.c:180

but task is already holding lock:
ffff0000cdc9cbf0 (&rs->rs_recv_lock){....}-{2:2}, at: rds_clear_recv_queue+0x40/0x23c net/rds/recv.c:761

which lock already depends on the new lock.


the existing dependency chain (in reverse order) is:

-> #1 (&rs->rs_recv_lock){....}-{2:2}:
       __raw_read_lock_irqsave include/linux/rwlock_api_smp.h:159 [inline]
       _raw_read_lock_irqsave+0x138/0x1e8 kernel/locking/spinlock.c:236
       rds_wake_sk_sleep+0x34/0xc8 net/rds/af_rds.c:109
       rds_send_remove_from_sock+0x1a4/0x7e0 net/rds/send.c:634
       rds_send_path_drop_acked+0x394/0x3f4 net/rds/send.c:710
       rds_tcp_write_space+0x1a8/0x5a4 net/rds/tcp_send.c:198
       tcp_new_space net/ipv4/tcp_input.c:5443 [inline]
       tcp_check_space+0x128/0x774 net/ipv4/tcp_input.c:5462
       tcp_data_snd_check net/ipv4/tcp_input.c:5471 [inline]
       tcp_rcv_established+0xac4/0x1f40 net/ipv4/tcp_input.c:5966
       tcp_v4_do_rcv+0x340/0xc70 net/ipv4/tcp_ipv4.c:1726
       sk_backlog_rcv include/net/sock.h:1057 [inline]
       __release_sock+0x1a8/0x408 net/core/sock.c:2690
       release_sock+0x68/0x270 net/core/sock.c:3231
       tcp_sock_set_cork+0x100/0x188 net/ipv4/tcp.c:3232
       rds_tcp_xmit_path_complete+0x7c/0x8c net/rds/tcp_send.c:52
       rds_send_xmit+0x19c0/0x238c net/rds/send.c:422
       rds_sendmsg+0x160c/0x1bcc net/rds/send.c:1382
       sock_sendmsg_nosec net/socket.c:704 [inline]
       sock_sendmsg net/socket.c:724 [inline]
       ____sys_sendmsg+0x584/0x870 net/socket.c:2412
       ___sys_sendmsg+0x214/0x294 net/socket.c:2466
       __sys_sendmsg net/socket.c:2495 [inline]
       __do_sys_sendmsg net/socket.c:2504 [inline]
       __se_sys_sendmsg net/socket.c:2502 [inline]
       __arm64_sys_sendmsg+0x1ac/0x25c net/socket.c:2502
       __invoke_syscall arch/arm64/kernel/syscall.c:38 [inline]
       invoke_syscall+0x98/0x2b8 arch/arm64/kernel/syscall.c:52
       el0_svc_common+0x138/0x258 arch/arm64/kernel/syscall.c:142
       do_el0_svc+0x58/0x14c arch/arm64/kernel/syscall.c:181
       el0_svc+0x7c/0x1f0 arch/arm64/kernel/entry-common.c:596
       el0t_64_sync_handler+0x84/0xe4 arch/arm64/kernel/entry-common.c:614
       el0t_64_sync+0x1a0/0x1a4 arch/arm64/kernel/entry.S:584

-> #0 (&rm->m_rs_lock){....}-{2:2}:
       check_prev_add kernel/locking/lockdep.c:3053 [inline]
       check_prevs_add kernel/locking/lockdep.c:3172 [inline]
       validate_chain kernel/locking/lockdep.c:3787 [inline]
       __lock_acquire+0x32cc/0x7620 kernel/locking/lockdep.c:5011
       lock_acquire+0x240/0x77c kernel/locking/lockdep.c:5622
       __raw_spin_lock_irqsave include/linux/spinlock_api_smp.h:110 [inline]
       _raw_spin_lock_irqsave+0xc4/0x14c kernel/locking/spinlock.c:162
       rds_message_purge net/rds/message.c:138 [inline]
       rds_message_put+0x138/0xa0c net/rds/message.c:180
       rds_loop_inc_free+0x20/0x30 net/rds/loop.c:115
       rds_inc_put+0x100/0x198 net/rds/recv.c:82
       rds_clear_recv_queue+0x1b0/0x23c net/rds/recv.c:767
       rds_release+0xbc/0x2e4 net/rds/af_rds.c:73
       __sock_release net/socket.c:649 [inline]
       sock_close+0xb8/0x1fc net/socket.c:1317
       __fput+0x30c/0x7f0 fs/file_table.c:280
       ____fput+0x20/0x30 fs/file_table.c:308
       task_work_run+0x130/0x1e4 kernel/task_work.c:164
       exit_task_work include/linux/task_work.h:32 [inline]
       do_exit+0x688/0x2134 kernel/exit.c:872
       do_group_exit+0x110/0x268 kernel/exit.c:994
       get_signal+0x634/0x1550 kernel/signal.c:2889
       do_signal arch/arm64/kernel/signal.c:890 [inline]
       do_notify_resume+0x3d0/0x32b8 arch/arm64/kernel/signal.c:943
       prepare_exit_to_user_mode arch/arm64/kernel/entry-common.c:133 [inline]
       exit_to_user_mode arch/arm64/kernel/entry-common.c:138 [inline]
       el0_svc+0xfc/0x1f0 arch/arm64/kernel/entry-common.c:597
       el0t_64_sync_handler+0x84/0xe4 arch/arm64/kernel/entry-common.c:614
       el0t_64_sync+0x1a0/0x1a4 arch/arm64/kernel/entry.S:584

other info that might help us debug this:

 Possible unsafe locking scenario:

       CPU0                    CPU1
       ----                    ----
  lock(&rs->rs_recv_lock);
                               lock(&rm->m_rs_lock);
                               lock(&rs->rs_recv_lock);
  lock(&rm->m_rs_lock);

 *** DEADLOCK ***

2 locks held by syz-executor298/4087:
 #0: ffff0000e033d610 (&sb->s_type->i_mutex_key#11){+.+.}-{3:3}, at: inode_lock include/linux/fs.h:787 [inline]
 #0: ffff0000e033d610 (&sb->s_type->i_mutex_key#11){+.+.}-{3:3}, at: __sock_release net/socket.c:648 [inline]
 #0: ffff0000e033d610 (&sb->s_type->i_mutex_key#11){+.+.}-{3:3}, at: sock_close+0x80/0x1fc net/socket.c:1317
 #1: ffff0000cdc9cbf0 (&rs->rs_recv_lock){....}-{2:2}, at: rds_clear_recv_queue+0x40/0x23c net/rds/recv.c:761

stack backtrace:
CPU: 1 PID: 4087 Comm: syz-executor298 Not tainted 5.15.111-syzkaller #0
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 04/14/2023
Call trace:
 dump_backtrace+0x0/0x530 arch/arm64/kernel/stacktrace.c:152
 show_stack+0x2c/0x3c arch/arm64/kernel/stacktrace.c:216
 __dump_stack lib/dump_stack.c:88 [inline]
 dump_stack_lvl+0x108/0x170 lib/dump_stack.c:106
 dump_stack+0x1c/0x58 lib/dump_stack.c:113
 print_circular_bug+0x150/0x1b8 kernel/locking/lockdep.c:2011
 check_noncircular+0x2cc/0x378 kernel/locking/lockdep.c:2133
 check_prev_add kernel/locking/lockdep.c:3053 [inline]
 check_prevs_add kernel/locking/lockdep.c:3172 [inline]
 validate_chain kernel/locking/lockdep.c:3787 [inline]
 __lock_acquire+0x32cc/0x7620 kernel/locking/lockdep.c:5011
 lock_acquire+0x240/0x77c kernel/locking/lockdep.c:5622
 __raw_spin_lock_irqsave include/linux/spinlock_api_smp.h:110 [inline]
 _raw_spin_lock_irqsave+0xc4/0x14c kernel/locking/spinlock.c:162
 rds_message_purge net/rds/message.c:138 [inline]
 rds_message_put+0x138/0xa0c net/rds/message.c:180
 rds_loop_inc_free+0x20/0x30 net/rds/loop.c:115
 rds_inc_put+0x100/0x198 net/rds/recv.c:82
 rds_clear_recv_queue+0x1b0/0x23c net/rds/recv.c:767
 rds_release+0xbc/0x2e4 net/rds/af_rds.c:73
 __sock_release net/socket.c:649 [inline]
 sock_close+0xb8/0x1fc net/socket.c:1317
 __fput+0x30c/0x7f0 fs/file_table.c:280
 ____fput+0x20/0x30 fs/file_table.c:308
 task_work_run+0x130/0x1e4 kernel/task_work.c:164
 exit_task_work include/linux/task_work.h:32 [inline]
 do_exit+0x688/0x2134 kernel/exit.c:872
 do_group_exit+0x110/0x268 kernel/exit.c:994
 get_signal+0x634/0x1550 kernel/signal.c:2889
 do_signal arch/arm64/kernel/signal.c:890 [inline]
 do_notify_resume+0x3d0/0x32b8 arch/arm64/kernel/signal.c:943
 prepare_exit_to_user_mode arch/arm64/kernel/entry-common.c:133 [inline]
 exit_to_user_mode arch/arm64/kernel/entry-common.c:138 [inline]
 el0_svc+0xfc/0x1f0 arch/arm64/kernel/entry-common.c:597
 el0t_64_sync_handler+0x84/0xe4 arch/arm64/kernel/entry-common.c:614
 el0t_64_sync+0x1a0/0x1a4 arch/arm64/kernel/entry.S:584

Crashes (1):
Time Kernel Commit Syzkaller Config Log Report Syz repro C repro VM info Assets (help?) Manager Title
2023/05/15 06:59 linux-5.15.y b0ece631f84a 2b9ba477 .config console log report syz C [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan-arm64 possible deadlock in rds_message_put
* Struck through repros no longer work on HEAD.