• Michael J. Ruhl's avatar
    IB/hfi1: Close race condition on user context disable and close · bc5add09
    Michael J. Ruhl authored
    When disabling and removing a receive context, it is possible for an
    asynchronous event (i.e IRQ) to occur.  Because of this, there is a race
    between cleaning up the context, and the context being used by the
    asynchronous event.
    
    cpu 0  (context cleanup)
        rc->ref_count-- (ref_count == 0)
        hfi1_rcd_free()
    cpu 1  (IRQ (with rcd index))
    	rcd_get_by_index()
    	lock
    	ref_count+++     <-- reference count race (WARNING)
    	return rcd
    	unlock
    cpu 0
        hfi1_free_ctxtdata() <-- incorrect free location
        lock
        remove rcd from array
        unlock
        free rcd
    
    This race will cause the following WARNING trace:
    
    WARNING: CPU: 0 PID: 175027 at include/linux/kref.h:52 hfi1_rcd_get_by_index+0x84/0xa0 [hfi1]
    CPU: 0 PID: 175027 Comm: IMB-MPI1 Kdump: loaded Tainted: G OE ------------ 3.10.0-957.el7.x86_64 #1
    Hardware name: Intel Corporation S2600KP/S2600KP, BIOS SE5C610.86B.11.01.0076.C4.111920150602 11/19/2015
    Call Trace:
      dump_stack+0x19/0x1b
      __warn+0xd8/0x100
      warn_slowpath_null+0x1d/0x20
      hfi1_rcd_get_by_index+0x84/0xa0 [hfi1]
      is_rcv_urgent_int+0x24/0x90 [hfi1]
      general_interrupt+0x1b6/0x210 [hfi1]
      __handle_irq_event_percpu+0x44/0x1c0
      handle_irq_event_percpu+0x32/0x80
      handle_irq_event+0x3c/0x60
      handle_edge_irq+0x7f/0x150
      handle_irq+0xe4/0x1a0
      do_IRQ+0x4d/0xf0
      common_interrupt+0x162/0x162
    
    The race can also lead to a use after free which could be similar to:
    
    general protection fault: 0000 1 SMP
    CPU: 71 PID: 177147 Comm: IMB-MPI1 Kdump: loaded Tainted: G W OE ------------ 3.10.0-957.el7.x86_64 #1
    Hardware name: Intel Corporation S2600KP/S2600KP, BIOS SE5C610.86B.11.01.0076.C4.111920150602 11/19/2015
    task: ffff9962a8098000 ti: ffff99717a508000 task.ti: ffff99717a508000 __kmalloc+0x94/0x230
    Call Trace:
      ? hfi1_user_sdma_process_request+0x9c8/0x1250 [hfi1]
      hfi1_user_sdma_process_request+0x9c8/0x1250 [hfi1]
      hfi1_aio_write+0xba/0x110 [hfi1]
      do_sync_readv_writev+0x7b/0xd0
      do_readv_writev+0xce/0x260
      ? handle_mm_fault+0x39d/0x9b0
      ? pick_next_task_fair+0x5f/0x1b0
      ? sched_clock_cpu+0x85/0xc0
      ? __schedule+0x13a/0x890
      vfs_writev+0x35/0x60
      SyS_writev+0x7f/0x110
      system_call_fastpath+0x22/0x27
    
    Use the appropriate kref API to verify access.
    
    Reorder context cleanup to ensure context removal before cleanup occurs
    correctly.
    
    Cc: stable@vger.kernel.org # v4.14.0+
    Fixes: f683c80c ("IB/hfi1: Resolve kernel panics by reference counting receive contexts")
    Reviewed-by: default avatarMike Marciniszyn <mike.marciniszyn@intel.com>
    Signed-off-by: default avatarMichael J. Ruhl <michael.j.ruhl@intel.com>
    Signed-off-by: default avatarDennis Dalessandro <dennis.dalessandro@intel.com>
    Signed-off-by: default avatarJason Gunthorpe <jgg@mellanox.com>
    bc5add09
hfi.h 73.3 KB