Remove an inability of bnxt_en driver to set eswitch to switchdev
mode without existing VFs by:

1. Allow to set switchdev mode in bnxt_dl_eswitch_mode_set() so
   representors are created only when num_vfs > 0 otherwise just
   set bp->eswitch_mode
2. Do not automatically change bp->eswitch_mode during
   bnxt_vf_reps_create() and bnxt_vf_reps_destroy() calls so
   the eswitch mode is managed only by an user by devlink.
   Just set temporarily bp->eswitch_mode to legacy to avoid
   re-opening of representors during destroy.
3. Create representors in bnxt_sriov_enable() if current eswitch
   mode is switchdev one

Tested by this sequence:
1. Set PF interface up
2. Set PF's eswitch mode to switchdev
3. Created N VFs
4. Checked that N representors were created
5. Set eswitch mode to legacy
6. Checked that representors were deleted
7. Set eswitch mode back to switchdev
8. Checked that representors exist again for VFs
9. Deleted all VFs
10. Checked that all representors were deleted as well
11. Checked that current eswitch mode is still switchdev
Signed-off-by: Ivan Vecera <ivecera@redhat.com>
Acked-by: Venkat Duvvuru <venkatkumar.duvvuru@broadcom.com>
Link: https://lore.kernel.org/r/20230411120443.126055-1-ivecera@redhat.comSigned-off-by: Paolo Abeni <pabeni@redhat.com>