Driver fastpath employs doorbells to indicate to the device that work is
available. Each doorbell translates to a message sent to the device over
PCI. These messages are queued by the doorbell queue HW block, and handled
by the HW.

If a sufficient amount of CPU cores are sending messages at a sufficient
rate, the queue can overflow, and messages can be dropped. There are many
entities in the driver which can send doorbell messages.  When overflow
happens, a fatal HW attention is indicated, and the Doorbell HW block stops
accepting new doorbell messages until recovery procedure is done.

When overflow occurs, all doorbells are dropped. Since doorbells are
aggregatives, if more doorbells are sent nothing has to be done.  But if
the "last" doorbell is dropped, the doorbelling entity doesn’t know this
happened, and may wait forever for the device to perform the action.  The
doorbell recovery mechanism addresses just that - it sends the last
doorbell of every entity.

[mkp: fix missing brackets reported by Guenter Roeck]

Link: https://lore.kernel.org/r/20210804221412.5048-1-smalin@marvell.comCo-developed-by: Manish Rangankar <mrangankar@marvell.com>
Signed-off-by: Manish Rangankar <mrangankar@marvell.com>
Signed-off-by: Shai Malin <smalin@marvell.com>
Signed-off-by: Martin K. Petersen <martin.petersen@oracle.com>