• Or Gerlitz's avatar
    mlx4: 64-byte CQE/EQE support · 08ff3235
    Or Gerlitz authored
    ConnectX-3 devices can use either 64- or 32-byte completion queue
    entries (CQEs) and event queue entries (EQEs).  Using 64-byte
    EQEs/CQEs performs better because each entry is aligned to a complete
    cacheline.  This patch queries the HCA's capabilities, and if it
    supports 64-byte CQEs and EQES the driver will configure the HW to
    work in 64-byte mode.
    
    The 32-byte vs 64-byte mode is global per HCA and not per CQ or EQ.
    
    Since this mode is global, userspace (libmlx4) must be updated to work
    with the configured CQE size, and guests using SR-IOV virtual
    functions need to know both EQE and CQE size.
    
    In case one of the 64-byte CQE/EQE capabilities is activated, the
    patch makes sure that older guest drivers that use the QUERY_DEV_FUNC
    command (e.g as done in mlx4_core of Linux 3.3..3.6) will notice that
    they need an update to be able to work with the PPF. This is done by
    changing the returned pf_context_behaviour not to be zero any more. In
    case none of these capabilities is activated that value remains zero
    and older guest drivers can run OK.
    
    The SRIOV related flow is as follows
    
    1. the PPF does the detection of the new capabilities using
       QUERY_DEV_CAP command.
    
    2. the PPF activates the new capabilities using INIT_HCA.
    
    3. the VF detects if the PPF activated the capabilities using
       QUERY_HCA, and if this is the case activates them for itself too.
    
    Note that the VF detects that it must be aware to the new PF behaviour
    using QUERY_FUNC_CAP.  Steps 1 and 2 apply also for native mode.
    
    User space notification is done through a new field introduced in
    struct mlx4_ib_ucontext which holds device capabilities for which user
    space must take action. This changes the binary interface so the ABI
    towards libmlx4 exposed through uverbs is bumped from 3 to 4 but only
    when **needed** i.e. only when the driver does use 64-byte CQEs or
    future device capabilities which must be in sync by user space. This
    practice allows to work with unmodified libmlx4 on older devices (e.g
    A0, B0) which don't support 64-byte CQEs.
    
    In order to keep existing systems functional when they update to a
    newer kernel that contains these changes in VF and userspace ABI, a
    module parameter enable_64b_cqe_eqe must be set to enable 64-byte
    mode; the default is currently false.
    Signed-off-by: default avatarEli Cohen <eli@mellanox.com>
    Signed-off-by: default avatarOr Gerlitz <ogerlitz@mellanox.com>
    Signed-off-by: default avatarRoland Dreier <roland@purestorage.com>
    08ff3235
main.c 45.8 KB