• Bart Van Assche's avatar
    target: Fix a deadlock between the XCOPY code and iSCSI session shutdown · d877d727
    Bart Van Assche authored
    Move the code for parsing an XCOPY command from the context of
    the iSCSI receiver thread to the context of the XCOPY workqueue.
    Keep the simple XCOPY checks in the context of the iSCSI receiver
    thread. Move the code for allocating and freeing struct xcopy_op
    from the code that parses an XCOPY command to its caller.
    
    This patch fixes the following deadlock:
    
    ======================================================
    [ INFO: possible circular locking dependency detected ]
    4.10.0-rc7-dbg+ #1 Not tainted
    -------------------------------------------------------
    rmdir/13321 is trying to acquire lock:
     (&sess->cmdsn_mutex){+.+.+.}, at: [<ffffffffa02cb47d>] iscsit_free_all_ooo_cmdsns+0x2d/0xb0 [iscsi_target_mod]
    
    but task is already holding lock:
     (&sb->s_type->i_mutex_key#14){++++++}, at: [<ffffffff811c6e20>] vfs_rmdir+0x50/0x140
    
    which lock already depends on the new lock.
    
    the existing dependency chain (in reverse order) is:
    -> #1 (&sb->s_type->i_mutex_key#14){++++++}:
     lock_acquire+0x71/0x90
     down_write+0x3f/0x70
     configfs_depend_item+0x3a/0xb0 [configfs]
     target_depend_item+0x13/0x20 [target_core_mod]
     target_xcopy_locate_se_dev_e4+0xdd/0x1a0 [target_core_mod]
     target_do_xcopy+0x34b/0x970 [target_core_mod]
     __target_execute_cmd+0x22/0xa0 [target_core_mod]
     target_execute_cmd+0x233/0x2c0 [target_core_mod]
     iscsit_execute_cmd+0x208/0x270 [iscsi_target_mod]
     iscsit_sequence_cmd+0x10b/0x190 [iscsi_target_mod]
     iscsit_get_rx_pdu+0x37d/0xcd0 [iscsi_target_mod]
     iscsi_target_rx_thread+0x6e/0xa0 [iscsi_target_mod]
     kthread+0x102/0x140
     ret_from_fork+0x31/0x40
    
    -> #0 (&sess->cmdsn_mutex){+.+.+.}:
     __lock_acquire+0x10e6/0x1260
     lock_acquire+0x71/0x90
     mutex_lock_nested+0x5f/0x670
     iscsit_free_all_ooo_cmdsns+0x2d/0xb0 [iscsi_target_mod]
     iscsit_close_session+0xac/0x200 [iscsi_target_mod]
     lio_tpg_close_session+0x9f/0xb0 [iscsi_target_mod]
     target_shutdown_sessions+0xc3/0xd0 [target_core_mod]
     core_tpg_del_initiator_node_acl+0x91/0x140 [target_core_mod]
     target_fabric_nacl_base_release+0x20/0x30 [target_core_mod]
     config_item_release+0x5a/0xc0 [configfs]
     config_item_put+0x1d/0x1f [configfs]
     configfs_rmdir+0x1a6/0x300 [configfs]
     vfs_rmdir+0xb7/0x140
     do_rmdir+0x1f4/0x200
     SyS_rmdir+0x11/0x20
     entry_SYSCALL_64_fastpath+0x23/0xc6
    
    other info that might help us debug this:
    
     Possible unsafe locking scenario:
           CPU0                    CPU1
           ----                    ----
      lock(&sb->s_type->i_mutex_key#14);
                                   lock(&sess->cmdsn_mutex);
                                   lock(&sb->s_type->i_mutex_key#14);
      lock(&sess->cmdsn_mutex);
    
     *** DEADLOCK ***
    
    3 locks held by rmdir/13321:
     #0:  (sb_writers#10){.+.+.+}, at: [<ffffffff811e1aff>] mnt_want_write+0x1f/0x50
     #1:  (&default_group_class[depth - 1]#2/1){+.+.+.}, at: [<ffffffff811cc8ce>] do_rmdir+0x15e/0x200
     #2:  (&sb->s_type->i_mutex_key#14){++++++}, at: [<ffffffff811c6e20>] vfs_rmdir+0x50/0x140
    
    stack backtrace:
    CPU: 2 PID: 13321 Comm: rmdir Not tainted 4.10.0-rc7-dbg+ #1
    Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.0.0-prebuilt.qemu-project.org 04/01/2014
    Call Trace:
     dump_stack+0x86/0xc3
     print_circular_bug+0x1c7/0x220
     __lock_acquire+0x10e6/0x1260
     lock_acquire+0x71/0x90
     mutex_lock_nested+0x5f/0x670
     iscsit_free_all_ooo_cmdsns+0x2d/0xb0 [iscsi_target_mod]
     iscsit_close_session+0xac/0x200 [iscsi_target_mod]
     lio_tpg_close_session+0x9f/0xb0 [iscsi_target_mod]
     target_shutdown_sessions+0xc3/0xd0 [target_core_mod]
     core_tpg_del_initiator_node_acl+0x91/0x140 [target_core_mod]
     target_fabric_nacl_base_release+0x20/0x30 [target_core_mod]
     config_item_release+0x5a/0xc0 [configfs]
     config_item_put+0x1d/0x1f [configfs]
     configfs_rmdir+0x1a6/0x300 [configfs]
     vfs_rmdir+0xb7/0x140
     do_rmdir+0x1f4/0x200
     SyS_rmdir+0x11/0x20
     entry_SYSCALL_64_fastpath+0x23/0xc6
    Signed-off-by: default avatarBart Van Assche <bart.vanassche@sandisk.com>
    Cc: Hannes Reinecke <hare@suse.com>
    Cc: Christoph Hellwig <hch@lst.de>
    Cc: Andy Grover <agrover@redhat.com>
    Cc: David Disseldorp <ddiss@suse.de>
    Signed-off-by: default avatarNicholas Bellinger <nab@linux-iscsi.org>
    d877d727
target_core_xcopy.c 30.5 KB