• Leon Romanovsky's avatar
    RDMA/uverbs: Fix circular locking dependency · 1ff5325c
    Leon Romanovsky authored
    Avoid circular locking dependency by calling
    to uobj_alloc_commit() outside of xrcd_tree_mutex lock.
    
    ======================================================
    WARNING: possible circular locking dependency detected
    4.15.0+ #87 Not tainted
    ------------------------------------------------------
    syzkaller401056/269 is trying to acquire lock:
     (&uverbs_dev->xrcd_tree_mutex){+.+.}, at: [<000000006c12d2cd>] uverbs_free_xrcd+0xd2/0x360
    
    but task is already holding lock:
     (&ucontext->uobjects_lock){+.+.}, at: [<00000000da010f09>] uverbs_cleanup_ucontext+0x168/0x730
    
    which lock already depends on the new lock.
    
    the existing dependency chain (in reverse order) is:
    
    -> #1 (&ucontext->uobjects_lock){+.+.}:
           __mutex_lock+0x111/0x1720
           rdma_alloc_commit_uobject+0x22c/0x600
           ib_uverbs_open_xrcd+0x61a/0xdd0
           ib_uverbs_write+0x7f9/0xef0
           __vfs_write+0x10d/0x700
           vfs_write+0x1b0/0x550
           SyS_write+0xc7/0x1a0
           entry_SYSCALL_64_fastpath+0x1e/0x8b
    
    -> #0 (&uverbs_dev->xrcd_tree_mutex){+.+.}:
           lock_acquire+0x19d/0x440
           __mutex_lock+0x111/0x1720
           uverbs_free_xrcd+0xd2/0x360
           remove_commit_idr_uobject+0x6d/0x110
           uverbs_cleanup_ucontext+0x2f0/0x730
           ib_uverbs_cleanup_ucontext.constprop.3+0x52/0x120
           ib_uverbs_close+0xf2/0x570
           __fput+0x2cd/0x8d0
           task_work_run+0xec/0x1d0
           do_exit+0x6a1/0x1520
           do_group_exit+0xe8/0x380
           SyS_exit_group+0x1e/0x20
           entry_SYSCALL_64_fastpath+0x1e/0x8b
    
    other info that might help us debug this:
    
     Possible unsafe locking scenario:
    
           CPU0                    CPU1
           ----                    ----
      lock(&ucontext->uobjects_lock);
                                   lock(&uverbs_dev->xrcd_tree_mutex);
                                   lock(&ucontext->uobjects_lock);
      lock(&uverbs_dev->xrcd_tree_mutex);
    
     *** DEADLOCK ***
    
    3 locks held by syzkaller401056/269:
     #0:  (&file->cleanup_mutex){+.+.}, at: [<00000000c9f0c252>] ib_uverbs_close+0xac/0x570
     #1:  (&ucontext->cleanup_rwsem){++++}, at: [<00000000b6994d49>] uverbs_cleanup_ucontext+0xf6/0x730
     #2:  (&ucontext->uobjects_lock){+.+.}, at: [<00000000da010f09>] uverbs_cleanup_ucontext+0x168/0x730
    
    stack backtrace:
    CPU: 0 PID: 269 Comm: syzkaller401056 Not tainted 4.15.0+ #87
    Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.7.5-0-ge51488c-20140602_164612-nilsson.home.kraxel.org 04/01/2014
    Call Trace:
     dump_stack+0xde/0x164
     ? dma_virt_map_sg+0x22c/0x22c
     ? uverbs_cleanup_ucontext+0x168/0x730
     ? console_unlock+0x502/0xbd0
     print_circular_bug.isra.24+0x35e/0x396
     ? print_circular_bug_header+0x12e/0x12e
     ? find_usage_backwards+0x30/0x30
     ? entry_SYSCALL_64_fastpath+0x1e/0x8b
     validate_chain.isra.28+0x25d1/0x40c0
     ? check_usage+0xb70/0xb70
     ? graph_lock+0x160/0x160
     ? find_usage_backwards+0x30/0x30
     ? cyc2ns_read_end+0x10/0x10
     ? print_irqtrace_events+0x280/0x280
     ? __lock_acquire+0x93d/0x1630
     __lock_acquire+0x93d/0x1630
     lock_acquire+0x19d/0x440
     ? uverbs_free_xrcd+0xd2/0x360
     __mutex_lock+0x111/0x1720
     ? uverbs_free_xrcd+0xd2/0x360
     ? uverbs_free_xrcd+0xd2/0x360
     ? __mutex_lock+0x828/0x1720
     ? mutex_lock_io_nested+0x1550/0x1550
     ? uverbs_cleanup_ucontext+0x168/0x730
     ? __lock_acquire+0x9a9/0x1630
     ? mutex_lock_io_nested+0x1550/0x1550
     ? uverbs_cleanup_ucontext+0xf6/0x730
     ? lock_contended+0x11a0/0x11a0
     ? uverbs_free_xrcd+0xd2/0x360
     uverbs_free_xrcd+0xd2/0x360
     remove_commit_idr_uobject+0x6d/0x110
     uverbs_cleanup_ucontext+0x2f0/0x730
     ? sched_clock_cpu+0x18/0x200
     ? uverbs_close_fd+0x1c0/0x1c0
     ib_uverbs_cleanup_ucontext.constprop.3+0x52/0x120
     ib_uverbs_close+0xf2/0x570
     ? ib_uverbs_remove_one+0xb50/0xb50
     ? ib_uverbs_remove_one+0xb50/0xb50
     __fput+0x2cd/0x8d0
     task_work_run+0xec/0x1d0
     do_exit+0x6a1/0x1520
     ? fsnotify_first_mark+0x220/0x220
     ? exit_notify+0x9f0/0x9f0
     ? entry_SYSCALL_64_fastpath+0x5/0x8b
     ? entry_SYSCALL_64_fastpath+0x5/0x8b
     ? trace_hardirqs_on_thunk+0x1a/0x1c
     ? time_hardirqs_on+0x27/0x670
     ? time_hardirqs_off+0x27/0x490
     ? syscall_return_slowpath+0x6c/0x460
     ? entry_SYSCALL_64_fastpath+0x5/0x8b
     do_group_exit+0xe8/0x380
     SyS_exit_group+0x1e/0x20
     entry_SYSCALL_64_fastpath+0x1e/0x8b
    RIP: 0033:0x431ce9
    
    Cc: syzkaller <syzkaller@googlegroups.com>
    Cc: <stable@vger.kernel.org> # 4.11
    Fixes: fd3c7904 ("IB/core: Change idr objects to use the new schema")
    Reported-by: default avatarNoa Osherovich <noaos@mellanox.com>
    Signed-off-by: default avatarLeon Romanovsky <leonro@mellanox.com>
    Signed-off-by: default avatarJason Gunthorpe <jgg@mellanox.com>
    1ff5325c
uverbs_cmd.c 96.9 KB