• Dragos Tatulea's avatar
    vdpa/mlx5: Pre-create hardware VQs at vdpa .dev_add time · ffb1aae4
    Dragos Tatulea authored
    Currently, hardware VQs are created right when the vdpa device gets into
    DRIVER_OK state. That is easier because most of the VQ state is known by
    then.
    
    This patch switches to creating all VQs and their associated resources
    at device creation time. The motivation is to reduce the vdpa device
    live migration downtime by moving the expensive operation of creating
    all the hardware VQs and their associated resources out of downtime on
    the destination VM.
    
    The VQs are now created in a blank state. The VQ configuration will
    happen later, on DRIVER_OK. Then the configuration will be applied when
    the VQs are moved to the Ready state.
    
    When .set_vq_ready() is called on a VQ before DRIVER_OK, special care is
    needed: now that the VQ is already created a resume_vq() will be
    triggered too early when no mr has been configured yet. Skip calling
    resume_vq() in this case, let it be handled during DRIVER_OK.
    
    For virtio-vdpa, the device configuration is done earlier during
    .vdpa_dev_add() by vdpa_register_device(). Avoid calling
    setup_vq_resources() a second time in that case.
    
    On a 64 CPU, 256 GB VM with 1 vDPA device of 16 VQps, the full VQ
    resource creation + resume time was ~370ms. Now it's down to 60 ms
    (only VQ config and resume). The measurements were done on a ConnectX6DX
    based vDPA device.
    Signed-off-by: default avatarDragos Tatulea <dtatulea@nvidia.com>
    Reviewed-by: default avatarCosmin Ratiu <cratiu@nvidia.com>
    Message-Id: <20240626-stage-vdpa-vq-precreate-v2-21-560c491078df@nvidia.com>
    Signed-off-by: default avatarMichael S. Tsirkin <mst@redhat.com>
    ffb1aae4
mlx5_vnet.c 105 KB