Some users complain about RNR errors on the target, when heavy
high-priority tasks run on the initiator. After the investigation, we
found out that the receive WRs were exhausted, because the initiator could
not post them on time.

Receive work reqeusts are posted in chunks to reduce the number of hits to
the HCA. The WRs are posted in the receive completion handler when the
number of free receive buffers reaches the threshold. But on a high-loaded
host, receive CQEs processing can be delayed and all receive WRs will be
exhausted. In this case, the target will get an RNR error.

To avoid this, we post receive WR, as soon as possible and not in a
batch. This increases the number of hits to the HCA, but also the common
implementation in most of Linux ULPs (e.g. NVMe-oF/RDMA). As a rule of
thumb, performance improvements and heuristics are being added to the RDMA
core layer or vendors low level drivers and it's about time to align iSER
as well.

Link: https://lore.kernel.org/r/20211215135721.3662-3-mgurtovoy@nvidia.comSigned-off-by: Sergey Gorenko <sergeygo@nvidia.com>
Signed-off-by: Max Gurtovoy <mgurtovoy@nvidia.com>
Reviewed-by: Israel Rukshin <israelr@nvidia.com>
Signed-off-by: Jason Gunthorpe <jgg@nvidia.com>

No need for this dead code. This commit doesn't change any functionality
since one can still run "modprobe ib_iser pi_guard=<type>".

Link: https://lore.kernel.org/r/20211215135721.3662-2-mgurtovoy@nvidia.comSigned-off-by: Max Gurtovoy <mgurtovoy@nvidia.com>
Signed-off-by: Jason Gunthorpe <jgg@nvidia.com>

Under MAD query port, Report NDR speed when NDR is supported in the port
capability mask.

Link: https://lore.kernel.org/r/a2ab630d2a634547db9b581faa9d65da2edb9d05.1639554831.git.leonro@nvidia.comSigned-off-by: Maher Sanalla <msanalla@nvidia.com>
Reviewed-by: Michael Guralnik <michaelgur@nvidia.com>
Signed-off-by: Leon Romanovsky <leonro@nvidia.com>
Signed-off-by: Jason Gunthorpe <jgg@nvidia.com>

The API for ib_query_qp requires the driver to set cur_qp_state on return,
add the missing set.

Fixes: 67bbc055 ("RDMA/cxgb4: Add query_qp support")
Link: https://lore.kernel.org/r/20211220152530.60399-1-kamalheib1@gmail.comSigned-off-by: Kamal Heib <kamalheib1@gmail.com>
Reviewed-by: Leon Romanovsky <leonro@nvidia.com>
Signed-off-by: Jason Gunthorpe <jgg@nvidia.com>

HIP06 is no longer supported. In order to reduce unnecessary maintenance,
the code of HIP06 is removed.

Link: https://lore.kernel.org/r/20211220130558.61585-1-liangwenpeng@huawei.comSigned-off-by: Chengchang Tang <tangchengchang@huawei.com>
Signed-off-by: Wenpeng Liang <liangwenpeng@huawei.com>
Reviewed-by: Leon Romanovsky <leonro@nvidia.com>
Signed-off-by: Jason Gunthorpe <jgg@nvidia.com>

Starting from the commit 66920e1b ("rdma_rxe: Use netlink messages
to add/delete links") from the 2019, the RXE modules parameters are marked
as deprecated in favour of rdmatool. So remove the kernel code too.

Link: https://lore.kernel.org/r/c8376d7517aebe7cc851f0baaeef7b13707cf767.1641372460.git.leonro@nvidia.comSigned-off-by: Leon Romanovsky <leonro@nvidia.com>
Reviewed-by: Zhu Yanjun <zyjzyj2000@gmail.com>
Signed-off-by: Jason Gunthorpe <jgg@nvidia.com>

Several drivers used same function to initialize query MAD,
so move that function to global header file.

Link: https://lore.kernel.org/r/af6f35c590ff5ef56d0137351b8b295af0f7c13c.1641369858.git.leonro@nvidia.comSigned-off-by: Leon Romanovsky <leonro@nvidia.com>
Reviewed-by: Håkon Bugge <haakon.bugge@oracle.com>
Signed-off-by: Jason Gunthorpe <jgg@nvidia.com>

* Fix these up to always have the '+', and '|' on the continuing line
  which is the normal kernel style.
* Fix indentations correspondingly

NOTE: this patch also remove the 2 redundant plus in
IB_OPCODE_RD_FETCH_ADD and IB_OPCODE_RD_COMPARE_SWAP

Link: https://lore.kernel.org/r/20220105042605.14343-1-lizhijian@fujitsu.comSigned-off-by: Li Zhijian <lizhijian@fujitsu.com>
Reviewed-by: Christoph Hellwig <hch@lst.de>
Signed-off-by: Jason Gunthorpe <jgg@nvidia.com>

There are currently 2 ways to create a set of sysfs files for a kobj_type,
through the default_attrs field, and the default_groups field.  Move the
IB code to use default_groups field which has been the preferred way since
commit aa30f47c ("kobject: Add support for default attribute groups to
kobj_type") so that we can soon get rid of the obsolete default_attrs
field.

Link: https://lore.kernel.org/r/20220103152259.531034-1-gregkh@linuxfoundation.orgSigned-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
Reviewed-by: Leon Romanovsky <leonro@nvidia.com>
Signed-off-by: Jason Gunthorpe <jgg@nvidia.com>

mlx5_handle_error_cqe() only dump the content of the CQE which is raw hex
data, and not straighforward for debug.  Print WC status message when we
got CQE error and dump is need.

Here is an example of how the dmesg log looks like with this:

 infiniband mlx5_0: mlx5_handle_error_cqe:333:(pid 0): WC error: 10, message: remote access error
 infiniband mlx5_0: dump_cqe:272:(pid 0): dump error cqe
 00000000: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
 00000010: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
 00000020: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
 00000030: 00 00 00 00 00 00 88 13 08 03 61 b3 1e a1 42 d3

Link: https://lore.kernel.org/r/20211227123806.47530-1-dust.li@linux.alibaba.comSigned-off-by: Dust Li <dust.li@linux.alibaba.com>
Acked-by: Leon Romanovsky <leonro@nvidia.com>
Signed-off-by: Jason Gunthorpe <jgg@nvidia.com>

There is a redundant ']' in the name of opcode IB_OPCODE_RC_SEND_MIDDLE,
so just fix it.

Fixes: 8700e3e7 ("Soft RoCE driver")
Link: https://lore.kernel.org/r/20211218112320.3558770-1-cgxu519@mykernel.netSigned-off-by: Chengguang Xu <cgxu519@mykernel.net>
Acked-by: Zhu Yanjun <zyjzyj2000@gmail.com>
Reviewed-by: Bob Pearson <rpearsonhpe@gmail.com>
Signed-off-by: Jason Gunthorpe <jgg@nvidia.com>

The member variable xmit_errors can be replaced with

 rxe_counter_inc(rxe, RXE_CNT_SEND_ERR)

Link: https://lore.kernel.org/r/20211216054842.1099428-1-yanjun.zhu@linux.devSigned-off-by: Zhu Yanjun <yanjun.zhu@linux.dev>
Reviewed-by: Leon Romanovsky <leonro@nvidia.com>
Signed-off-by: Jason Gunthorpe <jgg@nvidia.com>

Return value directly instead of taking this in another redundant
variable.

Link: https://lore.kernel.org/20211215075258.442930-1-chi.minghao@zte.com.cnReported-by: Zeal Robot <zealci@zte.com.cn>
Signed-off-by: Minghao Chi <chi.minghao@zte.com.cn>
Acked-by: Zhu Yanjun <zyjzyj2000@gmail.com>
Reviewed-by: Devesh Sharma <Devesh.s.sharma@oracle.com>
Signed-off-by: Jason Gunthorpe <jgg@nvidia.com>

Return status directly from function called.

Link: https://lore.kernel.org/r/20211215055421.441375-1-chi.minghao@zte.com.cnReported-by: Zeal Robot <zealci@zte.com.cn>
Signed-off-by: Minghao Chi <chi.minghao@zte.com.cn>
Signed-off-by: Jason Gunthorpe <jgg@nvidia.com>

Saeed Mahameed says:
====================
Currently, the driver ignores the user's priority for flow steering rules
in FDB namespace. Change it and create the rule in the right priority.

It will allow to create FDB steering rules in up to 16 different
priorities.
====================
Signed-off-by: Jason Gunthorpe <jgg@nvidia.com>

* mellanox/mlx5-next:
  RDMA/mlx5: Add support to multiple priorities for FDB rules
  net/mlx5: Create more priorities for FDB bypass namespace
  net/mlx5: Refactor mlx5_get_flow_namespace
  net/mlx5: Separate FDB namespace

In preparation for FORTIFY_SOURCE performing compile-time and run-time
field bounds checking for memset(), avoid intentionally writing across
neighboring fields.

Use memset_startat() so memset() doesn't get confused about writing beyond
the destination member that is intended to be the starting point of
zeroing through the end of the struct.

Link: https://lore.kernel.org/r/20211213223331.135412-15-keescook@chromium.orgSigned-off-by: Kees Cook <keescook@chromium.org>
Signed-off-by: Jason Gunthorpe <jgg@nvidia.com>

In preparation for FORTIFY_SOURCE performing compile-time and run-time
field bounds checking for memset(), avoid intentionally writing across
neighboring fields.

Use memset_startat() so memset() doesn't get confused about writing beyond
the destination member that is intended to be the starting point of
zeroing through the end of the struct. Additionally, since everything
appears to perform a roundup (including allocation), just change the size
of the struct itself and add a build-time check to validate the expected
size.

Link: https://lore.kernel.org/r/20211213223331.135412-13-keescook@chromium.orgSigned-off-by: Kees Cook <keescook@chromium.org>
Signed-off-by: Jason Gunthorpe <jgg@nvidia.com>

In preparation for FORTIFY_SOURCE performing compile-time and run-time
field bounds checking for memset(), avoid intentionally writing across
neighboring fields.

Use memset_after() to zero the end of struct mlx5_ib_mr that should be
initialized.

Link: https://lore.kernel.org/r/20211213223331.135412-10-keescook@chromium.orgAcked-by: Leon Romanovsky <leonro@nvidia.com>
Signed-off-by: Kees Cook <keescook@chromium.org>
Signed-off-by: Jason Gunthorpe <jgg@nvidia.com>

Required due to dependencies in following patches.
Signed-off-by: Jason Gunthorpe <jgg@nvidia.com>

Currently, when cma_resolve_ib_dev() searches for a matching GID it will
stop searching after encountering the first empty GID table entry. This
behavior is wrong since neither IB nor RoCE spec enforce tightly packed
GID tables.

For example, when the matching valid GID entry exists at index N, and if a
GID entry is empty at index N-1, cma_resolve_ib_dev() will fail to find
the matching valid entry.

Fix it by making cma_resolve_ib_dev() continue searching even after
encountering missing entries.

Fixes: f17df3b0 ("RDMA/cma: Add support for AF_IB to rdma_resolve_addr()")
Link: https://lore.kernel.org/r/b7346307e3bb396c43d67d924348c6c496493991.1639055490.git.leonro@nvidia.comSigned-off-by: Avihai Horon <avihaih@nvidia.com>
Reviewed-by: Mark Zhang <markzhang@nvidia.com>
Signed-off-by: Leon Romanovsky <leonro@nvidia.com>
Signed-off-by: Jason Gunthorpe <jgg@nvidia.com>

Currently, ib_find_gid() will stop searching after encountering the first
empty GID table entry. This behavior is wrong since neither IB nor RoCE
spec enforce tightly packed GID tables.

For example, when a valid GID entry exists at index N, and if a GID entry
is empty at index N-1, ib_find_gid() will fail to find the valid entry.

Fix it by making ib_find_gid() continue searching even after encountering
missing entries.

Fixes: 5eb620c8 ("IB/core: Add helpers for uncached GID and P_Key searches")
Link: https://lore.kernel.org/r/e55d331b96cecfc2cf19803d16e7109ea966882d.1639055490.git.leonro@nvidia.comSigned-off-by: Avihai Horon <avihaih@nvidia.com>
Reviewed-by: Mark Zhang <markzhang@nvidia.com>
Signed-off-by: Leon Romanovsky <leonro@nvidia.com>
Signed-off-by: Jason Gunthorpe <jgg@nvidia.com>

Modify rdma_query_gid() to return -ENOENT for empty entries. This will
make error reporting more accurate and will be used in next patches.

Link: https://lore.kernel.org/r/1f2b65dfb4d995e74b621e3e21e7c7445d187956.1639055490.git.leonro@nvidia.comSigned-off-by: Avihai Horon <avihaih@nvidia.com>
Reviewed-by: Mark Zhang <markzhang@nvidia.com>
Signed-off-by: Leon Romanovsky <leonro@nvidia.com>
Signed-off-by: Jason Gunthorpe <jgg@nvidia.com>

The current write wqe mechanism is to write to DDR first, and then notify
the hardware through doorbell to read the data. Direct wqe is a mechanism
to fill wqe directly into the hardware. In the case of light load, the wqe
will be filled into pcie bar space of the hardware, this will reduce one
memory access operation and therefore reduce the latency. SIMD
instructions allows cpu to write the 512 bits at one time to device
memory, thus it can be used for posting direct wqe.

Add direct wqe enable switch and address mapping.

Link: https://lore.kernel.org/r/20211207124901.42123-2-liangwenpeng@huawei.comSigned-off-by: Yixing Liu <liuyixing1@huawei.com>
Signed-off-by: Wenpeng Liang <liangwenpeng@huawei.com>
Signed-off-by: Jason Gunthorpe <jgg@nvidia.com>

Currently, the driver ignores the user's priority for flow steering
rules in FDB namespace. Change it and create the rule in the right
priority.
It will allow to create FDB steering rules in up to 16 different
priorities.
Signed-off-by: Maor Gottlieb <maorg@nvidia.com>
Reviewed-by: Mark Bloch <mbloch@nvidia.com>
Signed-off-by: Saeed Mahameed <saeedm@nvidia.com>
Acked-by: Leon Romanovsky <leonro@nvidia.com>
Signed-off-by: Saeed Mahameed <saeedm@nvidia.com>

Create 16 flow steering priorities for FDB bypass users.
Signed-off-by: Maor Gottlieb <maorg@nvidia.com>
Reviewed-by: Mark Bloch <mbloch@nvidia.com>
Signed-off-by: Saeed Mahameed <saeedm@nvidia.com>
Reviewed-by: Leon Romanovsky <leonro@nvidia.com>
Signed-off-by: Saeed Mahameed <saeedm@nvidia.com>

Have all the namespace type check in the same switch case.
Signed-off-by: Maor Gottlieb <maorg@nvidia.com>
Reviewed-by: Mark Bloch <mbloch@nvidia.com>
Signed-off-by: Saeed Mahameed <saeedm@nvidia.com>
Reviewed-by: Leon Romanovsky <leonro@nvidia.com>
Signed-off-by: Saeed Mahameed <saeedm@nvidia.com>

This patch doesn't add an additional namespaces, but just separates the
naming to be used by each FDB user, bypass and kernel.
Downstream patches will actually split this up and allow to have more
than single priority for the bypass users.
Signed-off-by: Maor Gottlieb <maorg@nvidia.com>
Reviewed-by: Mark Bloch <mbloch@nvidia.com>
Signed-off-by: Saeed Mahameed <saeedm@nvidia.com>
Acked-by: Leon Romanovsky <leonro@nvidia.com>
Signed-off-by: Saeed Mahameed <saeedm@nvidia.com>

Pull USB fixes from Greg KH:
 "Here are some small USB fixes for 5.16-rc5.  They include:

   - gadget driver fixes for reported issues

   - xhci fixes for reported problems.

   - config endpoint parsing fixes for where we got bitfields wrong

  Most of these have been in linux-next, the remaining few were not, but
  got lots of local testing in my systems and in some cloud testing
  infrastructures"

* tag 'usb-5.16-rc5' of git://git.kernel.org/pub/scm/linux/kernel/git/gregkh/usb:
  usb: core: config: using bit mask instead of individual bits
  usb: core: config: fix validation of wMaxPacketValue entries
  USB: gadget: zero allocate endpoint 0 buffers
  USB: gadget: detect too-big endpoint 0 requests
  xhci: avoid race between disable slot command and host runtime suspend
  xhci: Remove CONFIG_USB_DEFAULT_PERSIST to prevent xHCI from runtime suspending
  Revert "usb: dwc3: dwc3-qcom: Enable tx-fifo-resize property by default"

Pull char/misc driver fixes from Greg KH:
 "Here are a bunch of small char/misc and other driver subsystem fixes.

  Included in here are:

   - iio driver fixes for reported problems

   - phy driver fixes for a number of reported problems

   - mhi resume bugfix for broken hardware

   - nvmem driver fix

   - rtsx driver fix for irq issues

   - fastrpc packet parsing fix

  All of these have been in linux-next for a while with no reported
  issues"

* tag 'char-misc-5.16-rc5' of git://git.kernel.org/pub/scm/linux/kernel/git/gregkh/char-misc: (33 commits)
  bus: mhi: core: Add support for forced PM resume
  iio: trigger: stm32-timer: fix MODULE_ALIAS
  misc: rtsx: Avoid mangling IRQ during runtime PM
  nvmem: eeprom: at25: fix FRAM byte_len
  misc: fastrpc: fix improper packet size calculation
  MAINTAINERS: add maintainer for Qualcomm FastRPC driver
  bus: mhi: pci_generic: Fix device recovery failed issue
  iio: adc: stm32: fix null pointer on defer_probe error
  phy: HiSilicon: Fix copy and paste bug in error handling
  dt-bindings: phy: zynqmp-psgtr: fix USB phy name
  phy: ti: omap-usb2: Fix the kernel-doc style
  phy: qualcomm: ipq806x-usb: Fix kernel-doc style
  iio: at91-sama5d2: Fix incorrect sign extension
  iio: adc: axp20x_adc: fix charging current reporting on AXP22x
  iio: gyro: adxrs290: fix data signedness
  phy: ti: tusb1210: Fix the kernel-doc warn
  phy: qualcomm: usb-hsic: Fix the kernel-doc warn
  phy: qualcomm: qmp: Add missing struct documentation
  phy: mvebu-cp110-utmi: Fix kernel-doc warns
  iio: ad7768-1: Call iio_trigger_notify_done() on error
  ...

Pull timer fixes from Thomas Gleixner:
 "Two fixes for clock chip drivers:

   - A regression fix for the Designware APB timer. A recent change to
     the error checking code transformed the error condition wrongly so
     it turned into a fail if good condition.

   - Fix a clang build fail of the ARM architected timer driver"

* tag 'timers-urgent-2021-12-12' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip:
  clocksource/drivers/arm_arch_timer: Force inlining of erratum_set_next_event_generic()
  clocksource/drivers/dw_apb_timer_of: Fix probe failure

Pull irq fixes from Thomas Gleixner:
 "A set of interrupt chip driver fixes:

   - Fix the multi vector MSI allocation on Armada 370XP

   - Do interrupt acknowledgement correctly in the aspeed-scu driver

   - Make the IPR register offset correct in the NVIC driver

   - Make redistribution table flushing correct by issueing a SYNC
     command to ensure that the invalidation command has been executed

   - Plug a device tree node reference leak in the bcm7210-l2 driver

   - Trivial fixes in the MIPS GIC and the Apple AIC drivers"

* tag 'irq-urgent-2021-12-12' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip:
  irqchip/irq-bcm7120-l2: Add put_device() after of_find_device_by_node()
  irqchip/irq-gic-v3-its.c: Force synchronisation when issuing INVALL
  irqchip/apple-aic: Mark aic_init_smp() as __init
  irqchip: nvic: Fix offset for Interrupt Priority Offsets
  irqchip/mips-gic: Use bitfield helpers
  irqchip/aspeed-scu: Replace update_bits with write_bits.
  irqchip/armada-370-xp: Fix support for Multi-MSI interrupts
  irqchip/armada-370-xp: Fix return value of armada_370_xp_msi_alloc()

Pull scheduler fix from Thomas Gleixner:
 "A single fix for the x86 scheduler topology:

  Using cluster topology on hybrid CPUs, e.g. Alder Lake, biases the
  scheduler towards the ATOM cluster as that has more total capacity.
  Use selection based on CPU priority instead"

* tag 'sched-urgent-2021-12-12' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip:
  sched,x86: Don't use cluster topology for x86 hybrid CPUs

Pull csky from Guo Ren:
 "Only one fix for csky: fix fpu config macro"

* tag 'csky-for-linus-5.16-rc5' of git://github.com/c-sky/csky-linux:
  csky: fix typo of fpu config macro

Using standard USB_EP_MAXP_MULT_MASK instead of individual bits for
extracting multiple-transactions bits from wMaxPacketSize value.
Acked-by: Alan Stern <stern@rowland.harvard.edu>
Signed-off-by: Pavel Hofman <pavel.hofman@ivitera.com>
Link: https://lore.kernel.org/r/20211210085219.16796-2-pavel.hofman@ivitera.comSigned-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>

The checks performed by commit aed9d65a ("USB: validate
wMaxPacketValue entries in endpoint descriptors") require that initial
value of the maxp variable contains both maximum packet size bits
(10..0) and multiple-transactions bits (12..11). However, the existing
code assings only the maximum packet size bits. This patch assigns all
bits of wMaxPacketSize to the variable.

Fixes: aed9d65a ("USB: validate wMaxPacketValue entries in endpoint descriptors")
Cc: stable <stable@vger.kernel.org>
Acked-by: Alan Stern <stern@rowland.harvard.edu>
Signed-off-by: Pavel Hofman <pavel.hofman@ivitera.com>
Link: https://lore.kernel.org/r/20211210085219.16796-1-pavel.hofman@ivitera.comSigned-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>

Under some conditions, USB gadget devices can show allocated buffer
contents to a host.  Fix this up by zero-allocating them so that any
extra data will all just be zeros.
Reported-by: Szymon Heidrich <szymon.heidrich@gmail.com>
Tested-by: Szymon Heidrich <szymon.heidrich@gmail.com>
Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>

Sometimes USB hosts can ask for buffers that are too large from endpoint
0, which should not be allowed.  If this happens for OUT requests, stall
the endpoint, but for IN requests, trim the request size to the endpoint
buffer size.
Co-developed-by: Szymon Heidrich <szymon.heidrich@gmail.com>
Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>

Pull SCSI fixes from James Bottomley:
 "Four fixes, all in drivers.

  Three are small and obvious, the qedi one is a bit larger but also
  pretty obvious"

* tag 'scsi-fixes' of git://git.kernel.org/pub/scm/linux/kernel/git/jejb/scsi:
  scsi: qla2xxx: Format log strings only if needed
  scsi: scsi_debug: Fix buffer size of REPORT ZONES command
  scsi: qedi: Fix cmd_cleanup_cmpl counter mismatch issue
  scsi: pm80xx: Do not call scsi_remove_host() in pm8001_alloc()

Pull xfs fix from Darrick Wong:
 "This fixes a race between a readonly remount process and other
  processes that hold a file IOLOCK on files that previously experienced
  copy on write, that could result in severe filesystem corruption if
  the filesystem is then remounted rw.

  I think this is fairly rare (since the only reliable reproducer I have
  that fits the second criteria is the experimental xfs_scrub program),
  but the race is clear, so we still need to fix this.

  Summary:

   - Fix a data corruption vector that can result from the ro remount
     process failing to clear all speculative preallocations from files
     and the rw remount process not noticing the incomplete cleanup"

* tag 'xfs-5.16-fixes-3' of git://git.kernel.org/pub/scm/fs/xfs/xfs-linux:
  xfs: remove all COW fork extents when remounting readonly