• Roland Dreier's avatar
    RDMA/cxgb3: Fix severe limit on userspace memory registration size · 273748cc
    Roland Dreier authored
    Currently, iw_cxgb3 is severely limited on the amount of userspace
    memory that can be registered in in a single memory region, which
    causes big problems for applications that expect to be able to
    register 100s of MB.
    
    The problem is that the driver uses a single kmalloc()ed buffer to
    hold the physical buffer list (PBL) for the entire memory region
    during registration, which means that 8 bytes of contiguous memory are
    required for each page of memory being registered.  For example, a 64
    MB registration will require 128 KB of contiguous memory with 4 KB
    pages, and it unlikely that such an allocation will succeed on a busy
    system.
    
    This is purely a driver problem: the temporary page list buffer is not
    needed by the hardware, so we can fix this by writing the PBL to the
    hardware in page-sized chunks rather than all at once.  We do this by
    splitting the memory registration operation up into several steps:
    
     - Allocate PBL space in adapter memory for the full registration
     - Copy PBL to adapter memory in chunks
     - Allocate STag and enable memory region
    
    This also allows several other cleanups to the __cxio_tpt_op()
    interface and related parts of the driver.
    
    This change leaves the reregister memory region and memory window
    operations broken, but they already didn't work due to other
    longstanding bugs, so fixing them will be left to a later patch.
    Signed-off-by: default avatarRoland Dreier <rolandd@cisco.com>
    273748cc
cxio_hal.h 6.83 KB