Current code is violating the DMA Engine API by putting the submitted
requests directly into the HW queue. This causes queued transactions
to be started by another thread as soon as the first one finishes.

The DMA Engine document clearly states this.

"dmaengine_submit() will not start the DMA operation".

Move HW queuing of the requests into the issue_pending() routine
to comply with API requirements also create a new queued state for
temporarily holding the requests.

A descriptor goes through these transitions now.

free->prepared->queued->active->completed->free

as opposed to

free->prepared->active->completed->free
Signed-off-by: Sinan Kaya <okaya@codeaurora.org>
Signed-off-by: Vinod Koul <vinod.koul@intel.com>

Remove check for "len > ZYNQMP_DMA_MAX_TRANS_LEN" as its not needed.
If the length is larger, the transfer is split up into multiple parts
with the max descriptor length already.
Signed-off-by: Stefan Roese <sr@denx.de>
Cc: Kedareswara rao Appana <appanad@xilinx.com>
Cc: Vinod Koul <vinod.koul@intel.com>
Signed-off-by: Vinod Koul <vinod.koul@intel.com>

Commit edd3bdbe ("dmaengine: tegra-apb: Correct runtime-pm usage")
added pm_runtime_get/put() calls to the tegra-apb DMA system suspend
callbacks. Runtime PM is disabled during system suspend and so these
APIs cannot be used. Fix the suspend handling for the tegra-apb DMA by
moving the save and restore of the DMA register context into the
runtime PM suspend and resume callbacks, and then use the
pm_runtime_force_suspend/resume() APIs to invoke the runtime PM
callbacks during system suspend.

Fixes: edd3bdbe ("dmaengine: tegra-apb: Correct runtime-pm usage")
Signed-off-by: Jon Hunter <jonathanh@nvidia.com>
Signed-off-by: Vinod Koul <vinod.koul@intel.com>

of_device_ids are not supposed to change at runtime. All functions
working with of_device_ids provided by <linux/of.h> work with const
of_device_ids. So mark the non-const structs as const.

File size before:
   text	   data	    bss	    dec	    hex	filename
   3981	    608	      0	   4589	   11ed	drivers/dma/fsl_raid.o

File size after constify:
   text	   data	    bss	    dec	    hex	filename
   4381	    192	      0	   4573	   11dd	drivers/dma/fsl_raid.o
Signed-off-by: Arvind Yadav <arvind.yadav.cs@gmail.com>
Signed-off-by: Vinod Koul <vinod.koul@intel.com>

Parameters like maximum read/write request size and the maximum
number of active transactions are currently configured in DT/ACPI.

This patch allows a user to override these to fine tune performance
for their application.
Signed-off-by: Sinan Kaya <okaya@codeaurora.org>
Signed-off-by: Vinod Koul <vinod.koul@intel.com>

The bits of BWC, DAHTS and SAHTS in the DMA mode register must be cleared
before a new value can be or-ed in.
Signed-off-by: Thomas Breitung <thomas.breitung@izt-labs.de>
Signed-off-by: Wolfgang Ocker <weo@reccoware.de>
Signed-off-by: Vinod Koul <vinod.koul@intel.com>

Currently the help text for the MXS_DMA option is incomplete as it does
not mention MX6SX, MX6ULL and MX7D, for example.

Instead of extending this list everytime a new SoC comes out, let's
keep the text more generic.
Signed-off-by: Fabio Estevam <fabio.estevam@nxp.com>
Signed-off-by: Vinod Koul <vinod.koul@intel.com>

The functions _queue_empty(), _emit_ADDH(), _emit_NOP(), _emit_STZ()
and _emit_WFE() are not used. Delete them.
Signed-off-by: Matthias Kaehlcke <mka@chromium.org>
Signed-off-by: Vinod Koul <vinod.koul@intel.com>

The WARN_TAINT_ONCE() prints out a loud stack trace on broken BIOSes.
The systems that have this problem are several years out of support and
no longer have BIOS updates available.  The stack trace isn't necessary
and a pr_warn_once() will do.

Change WARN_TAINT_ONCE() to pr_warn_once() and taint.
Signed-off-by: Prarit Bhargava <prarit@redhat.com>
Cc: Dan Williams <dan.j.williams@intel.com>
Cc: Vinod Koul <vinod.koul@intel.com>
Cc: Duyck, Alexander H <alexander.h.duyck@intel.com>
Signed-off-by: Vinod Koul <vinod.koul@intel.com>

Currently it is not possible to select the mxs dma driver when only
mx6sx or mx7 are selected.

Extend the dependency to allow the mxs dma driver to be built whenever
ARCH_MXS or ARCH_MXC is selected.

This has the benefit to avoid having to add new entries in the
MXS_DMA Kconfig everytime a new i.MX SoC shows up and it also makes
it consistent with the other i.MX DMA engines, such as IMX_DMA and
IMX_SDMA.

While at it, also pass COMPILE_TEST for increasing the build coverage.
Acked-by: Stefan Agner <stefan@agner.ch>
Signed-off-by: Fabio Estevam <fabio.estevam@nxp.com>
Signed-off-by: Vinod Koul <vinod.koul@intel.com>

Use %zu for printing a size_t variable in order to fix the following
build warning:

drivers/dma/mxs-dma.c: In function 'mxs_dma_prep_dma_cyclic':
drivers/dma/mxs-dma.c:621:5: warning: format '%d' expects argument of type 'int', but argument 3 has type 'size_t' [-Wformat]
Reported-by: kbuild test robot <lkp@intel.com>
Signed-off-by: Fabio Estevam <fabio.estevam@nxp.com>
Signed-off-by: Vinod Koul <vinod.koul@intel.com>

This dma engine driver directly accesses page_link assuming knowledge
that should be contained only in scatterlist.h.

We replace this access with a call to sg_chain which is equivalent.
Signed-off-by: Logan Gunthorpe <logang@deltatee.com>
Signed-off-by: Stephen Bates <sbates@raithlin.com>
Acked-by: Linus Walleij <linus.walleij@linaro.org>
Cc: Dan Williams <dan.j.williams@intel.com>
Cc: Vinod Koul <vinod.koul@intel.com>
Cc: Per Förlin <per.forlin@axis.com>
Signed-off-by: Vinod Koul <vinod.koul@intel.com>

This dma engine driver directly accesses page_link assuming knowledge
that should be contained only in scatterlist.h.

We replace these with calls to sg_chain and sg_assign_page.
Signed-off-by: Logan Gunthorpe <logang@deltatee.com>
Signed-off-by: Stephen Bates <sbates@raithlin.com>
Acked-by: Linus Walleij <linus.walleij@linaro.org>
Cc: Dan Williams <dan.j.williams@intel.com>
Cc: Vinod Koul <vinod.koul@intel.com>
Cc: Per Förlin <per.forlin@axis.com>
Signed-off-by: Vinod Koul <vinod.koul@intel.com>

It is 'R-Car', not 'r-car'. No code or binding changes, only descriptive text.
Signed-off-by: Wolfram Sang <wsa+renesas@sang-engineering.com>
Acked-by: Geert Uytterhoeven <geert+renesas@glider.be>
Acked-by: Rob Herring <robh@kernel.org>
Signed-off-by: Vinod Koul <vinod.koul@intel.com>

Replace '%d' by '%zu' to fix the following compilation warning:-

drivers/dma/imx-sdma.c: In function ‘sdma_prep_dma_cyclic’:
drivers/dma/imx-sdma.c:1327:5: warning: format ‘%d’ expects argument of type ‘int’, but argument 4 has type ‘size_t’ [-Wformat=]
     channel, period_len, 0xffff);
     ^
drivers/dma/imx-sdma.c:1350:3: warning: format ‘%d’ expects argument of type ‘int’, but argument 5 has type ‘size_t’ [-Wformat=]
   dev_dbg(sdma->dev, "entry %d: count: %d dma: %#llx %s%s\n",
Signed-off-by: Arvind Yadav <arvind.yadav.cs@gmail.com>
Signed-off-by: Vinod Koul <vinod.koul@intel.com>

clk_prepare_enable() can fail here and we must check its return value.
Signed-off-by: Arvind Yadav <arvind.yadav.cs@gmail.com>
Signed-off-by: Vinod Koul <vinod.koul@intel.com>

Wire up dma_map_resource() for slave transfers, so that we can let the
PL330 use IOMMU-backed DMA mapping ops on systems with an appropriate
IOMMU and RAM above 4GB, to avoid CPU bounce buffering.
Signed-off-by: Robin Murphy <robin.murphy@arm.com>
Signed-off-by: Vinod Koul <vinod.koul@intel.com>

clk_prepare_enable() can fail here and we must check its return value.
Signed-off-by: Arvind Yadav <arvind.yadav.cs@gmail.com>
Acked-by: Viresh Kumar <viresh.kumar@linaro.org>
Acked-by: Andy Shevchenko <andriy.shevchenko@linux.intel.com>
Signed-off-by: Vinod Koul <vinod.koul@intel.com>

This switches the arbitrary shifting of hex constants in
the pl080 header to use GENMASK().
Suggested-by: Vinod Koul <vinod.koul@intel.com>
Signed-off-by: Linus Walleij <linus.walleij@linaro.org>
Signed-off-by: Vinod Koul <vinod.koul@intel.com>

After reading the specs for the Faraday Technology FTDMAC020 found
in the Gemini platform, it becomes pretty evident that this is just
another PL08x derivative, and should be handled like such by simply
extending the existing PL08x driver to handle the quirks in this
hardware.

This patch makes memcpy work and has been tested on the Gemini and
also regression-tested on the Nomadik NHK15 using dmatest with
10 threads per channel without a hinch for hours.

I have not implemented slave DMA in those codepaths, because this
device (Gemini) does not use slave DMA, and it seems like devices
using FTDMAC020 for device DMA have a slightly different register
layout so some real hardware is needed to proceed with this. I
left some FIXME etc in the code for this.

I had to do some refactorings of some helper functions, but I have
not split those into separate patches because these refactorings
do not make much sense without the increased complexity of handling
the FTDMAC020.

The DMA test would hang the platform on me on the Gemini after a
few thousand iterations, however after turning of the caches the
problem immediately disappeared and I could run the DMA engine
with 10 threads pers physical channel for days in a row without
a crash. I think there is no problem with the DMA driver: instead
it is something fishy in the FA526 cache handling code that get
pretty heavily exercised by the DMA engine and we need to go and
fix that instead.
Signed-off-by: Linus Walleij <linus.walleij@linaro.org>
Signed-off-by: Vinod Koul <vinod.koul@intel.com>

If the vendor data does not specify any signals, we do not
have to support slave DMA. Make the registration of the slave
DMA engine optional, so we can use this for the FTDMAC020
in the Gemini that only has memcpy support.
Signed-off-by: Linus Walleij <linus.walleij@linaro.org>
Signed-off-by: Vinod Koul <vinod.koul@intel.com>

This augments the PL08x bindings to include the Faraday Technology
FTDMAC020 DMA engine, as it is clearly a derivative of the PL08x
PrimeCell. Also specify that it needs the special peripheral ID
specified to work properly.

Cc: devicetree@vger.kernel.org
Acked-by: Rob Herring <robh@kernel.org>
Signed-off-by: Linus Walleij <linus.walleij@linaro.org>
Signed-off-by: Vinod Koul <vinod.koul@intel.com>

We cannot use bits from configuration registers as API between
platforms and driver like this, abstract it out to two enums
and mimic the stuff passed as device tree data.

This is done to make it possible for the driver to generate the
ccfg word on-the-fly so we can support more PL08x derivatives.
Acked-by: Olof Johansson <olof@lixom.net>
Acked-by: Arnd Bergmann <arnd@arndb.de>
Signed-off-by: Linus Walleij <linus.walleij@linaro.org>
Signed-off-by: Vinod Koul <vinod.koul@intel.com>

The documentation for the "interrupt-names" property only mentions the
per-channel interrupts, while the error interrupt has always been
mandatory, too.

Fixes: 10f5c843 ("dmaengine: rcar-dmac: Add device tree bindings
documentation")
Signed-off-by: Geert Uytterhoeven <geert+renesas@glider.be>
Reviewed-by: Niklas Söderlund <niklas.soderlund+renesas@ragnatech.se>
Signed-off-by: Vinod Koul <vinod.koul@intel.com>

This fixes a race condition where the channel resources could be freed
before the ISR had finished running resulting in a NULL pointer
reference from the ISR.

[  167.148934] Unable to handle kernel NULL pointer dereference at virtual address 00000000
[  167.157051] pgd = ffff80003c641000
[  167.160449] [00000000] *pgd=000000007c507003, *pud=000000007c4ff003, *pmd=0000000000000000
[  167.168719] Internal error: Oops: 96000046 [#1] PREEMPT SMP
[  167.174289] Modules linked in:
[  167.177348] CPU: 3 PID: 10547 Comm: dma_ioctl Not tainted 4.11.0-rc1-00001-g8d92afddc2f6633a #73
[  167.186131] Hardware name: Renesas Salvator-X board based on r8a7795 (DT)
[  167.192917] task: ffff80003a411a00 task.stack: ffff80003bcd4000
[  167.198850] PC is at rcar_dmac_chan_prep_sg+0xe0/0x400
[  167.203985] LR is at rcar_dmac_chan_prep_sg+0x48/0x400

Based of previous work by:
    Hiroyuki Yokoyama <hiroyuki.yokoyama.vx@renesas.com>.
Signed-off-by: Niklas Söderlund <niklas.soderlund+renesas@ragnatech.se>
Signed-off-by: Vinod Koul <vinod.koul@intel.com>

Implement the device_synchronize() callback which wait until a dma
channel is stopped to provide a synchronization point.

This protects the driver from multiple race conditions when terminating
and freeing resources. E.g. the completion callback still running after
device_terminate_all() has completed.
Signed-off-by: Niklas Söderlund <niklas.soderlund+renesas@ragnatech.se>
Signed-off-by: Vinod Koul <vinod.koul@intel.com>

The IRQ number is needed after probe to be able to add synchronisation
points in other places in the driver when freeing resources and to
implement a device_synchronize() callback. Store the IRQ number in the
struct rcar_dmac_chan so that it can be used later.
Signed-off-by: Niklas Söderlund <niklas.soderlund+renesas@ragnatech.se>
Signed-off-by: Vinod Koul <vinod.koul@intel.com>

Req is never null on at the point of the null check, so
remove this redundant check and just return &req->tx.

Detected by CoverityScan, CID#1436147 ("Logically dead code")
Signed-off-by: Colin Ian King <colin.king@canonical.com>
Signed-off-by: Vinod Koul <vinod.koul@intel.com>

The new driver requires both mailbox and raid support for compile
testing:

drivers/dma/built-in.o: In function `sba_remove':
edma.c:(.text+0x4414): undefined reference to `mbox_free_channel'
drivers/dma/built-in.o: In function `sba_issue_pending':
edma.c:(.text+0x46cc): undefined reference to `mbox_send_message'
drivers/dma/built-in.o: In function `sba_probe':
edma.c:(.text+0x4e60): undefined reference to `mbox_request_channel'
edma.c:(.text+0x5038): undefined reference to `mbox_free_channel'
drivers/dma/built-in.o: In function `sba_tx_status':
edma.c:(.text+0x5210): undefined reference to `mbox_client_peek_data'

drivers/dma/built-in.o: In function `sba_prep_dma_pq_req':
edma.c:(.text+0x5784): undefined reference to `raid6_gflog'
edma.c:(.text+0x5798): undefined reference to `raid6_gflog'

This rearranges the Kconfig dependencies accordingly.

Fixes: 743e1c8f ("dmaengine: Add Broadcom SBA RAID driver")
Signed-off-by: Arnd Bergmann <arnd@arndb.de>
Signed-off-by: Vinod Koul <vinod.koul@intel.com>

This patch adds the DT bindings document for newly added Broadcom
SBA RAID driver.
Acked-by: Rob Herring <robh@kernel.org>
Reviewed-by: Ray Jui <ray.jui@broadcom.com>
Reviewed-by: Scott Branden <scott.branden@broadcom.com>
Signed-off-by: Anup Patel <anup.patel@broadcom.com>
Signed-off-by: Vinod Koul <vinod.koul@intel.com>

The Broadcom stream buffer accelerator (SBA) provides offloading
capabilities for RAID operations. This SBA offload engine is
accessible via Broadcom SoC specific ring manager.

This patch adds Broadcom SBA RAID driver which provides one
DMA device with RAID capabilities using one or more Broadcom
SoC specific ring manager channels. The SBA RAID driver in its
current shape implements memcpy, xor, and pq operations.
Signed-off-by: Anup Patel <anup.patel@broadcom.com>
Reviewed-by: Ray Jui <ray.jui@broadcom.com>
Acked-by: Dan Williams <dan.j.williams@intel.com>
Signed-off-by: Vinod Koul <vinod.koul@intel.com>

The DMA_PREP_FENCE is to be used when preparing Tx descriptor if output
of Tx descriptor is to be used by next/dependent Tx descriptor.

The DMA_PREP_FENSE will not be set correctly in do_async_gen_syndrome()
when calling dma->device_prep_dma_pq() under following conditions:
1. ASYNC_TX_FENCE not set in submit->flags
2. DMA_PREP_FENCE not set in dma_flags
3. src_cnt (= (disks - 2)) is greater than dma_maxpq(dma, dma_flags)

This patch fixes DMA_PREP_FENCE usage in do_async_gen_syndrome() taking
inspiration from do_async_xor() implementation.
Signed-off-by: Anup Patel <anup.patel@broadcom.com>
Reviewed-by: Ray Jui <ray.jui@broadcom.com>
Reviewed-by: Scott Branden <scott.branden@broadcom.com>
Acked-by: Dan Williams <dan.j.williams@intel.com>
Signed-off-by: Vinod Koul <vinod.koul@intel.com>

The raid6_gfexp table represents {2}^n values for 0 <= n < 256. The
Linux async_tx framework pass values from raid6_gfexp as coefficients
for each source to prep_dma_pq() callback of DMA channel with PQ
capability. This creates problem for RAID6 offload engines (such as
Broadcom SBA) which take disk position (i.e. log of {2}) instead of
multiplicative cofficients from raid6_gfexp table.

This patch adds raid6_gflog table having log-of-2 value for any given
x such that 0 <= x < 256. For any given disk coefficient x, the
corresponding disk position is given by raid6_gflog[x]. The RAID6
offload engine driver can use this newly added raid6_gflog table to
get disk position from multiplicative coefficient.
Signed-off-by: Anup Patel <anup.patel@broadcom.com>
Reviewed-by: Scott Branden <scott.branden@broadcom.com>
Reviewed-by: Ray Jui <ray.jui@broadcom.com>
Acked-by: Shaohua Li <shli@fb.com>
Signed-off-by: Vinod Koul <vinod.koul@intel.com>