The GFNI routines in the AVX version of the ARIA implementation now use
explicit VMOVDQA instructions to load the constant input vectors, which
means they must be 16 byte aligned. So ensure that this is the case, by
dropping the section split and the incorrect .align 8 directive, and
emitting the constants into the 16-byte aligned section instead.

Note that the AVX2 version of this code deviates from this pattern, and
does not require a similar fix, given that it loads these contants as
8-byte memory operands, for which AVX2 permits any alignment.

Cc: Taehee Yoo <ap420073@gmail.com>
Fixes: 8b844753 ("crypto: x86/aria-avx - Do not use avx2 instructions")
Reported-by: syzbot+a6abcf08bad8b18fd198@syzkaller.appspotmail.com
Tested-by: syzbot+a6abcf08bad8b18fd198@syzkaller.appspotmail.com
Signed-off-by: Ard Biesheuvel <ardb@kernel.org>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>

Checking the config via ifdef incorrectly compiles out the report
functions when CRYPTO_USER is set to =m. Fix it by using IS_ENABLED()
instead.

Fixes: c0f9e01d ("crypto: api - Check CRYPTO_USER instead of NET for report")
Signed-off-by: Ondrej Mosnacek <omosnace@redhat.com>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>

CRYPTO_TFM_REQ_MAY_BACKLOG tells the crypto driver that it should
internally backlog requests until the crypto hw's queue becomes
full. At that point, crypto_engine backlogs the request and returns
-EBUSY. Calling driver such as dm-crypt then waits until the
complete() function is called with a status of -EINPROGRESS before
sending a new request.

The problem lies in the call to complete() with a value of -EINPROGRESS
that is made when a backlog item is present on the queue. The call is
done before the successful execution of the crypto request. In the case
that do_one_request() returns < 0 and the retry support is available,
the request is put back in the queue. This leads upper drivers to send
a new request even if the queue is still full.

The problem can be reproduced by doing a large dd into a crypto
dm-crypt device. This is pretty easy to see when using
Freescale CAAM crypto driver and SWIOTLB dma. Since the actual amount
of requests that can be hold in the queue is unlimited we get IOs error
and dma allocation.

The fix is to call complete with a value of -EINPROGRESS only if
the request is not enqueued back in crypto_queue. This is done
by calling complete() later in the code. In order to delay the decision,
crypto_queue is modified to correctly set the backlog pointer
when a request is enqueued back.

Fixes: 6a89f492 ("crypto: engine - support for parallel requests based on retry mechanism")
Co-developed-by: Sylvain Ouellet <souellet@genetec.com>
Signed-off-by: Sylvain Ouellet <souellet@genetec.com>
Signed-off-by: Olivier Bacon <obacon@genetec.com>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>

SS_ENCRYPTION is (0 << 7 = 0), so the test can never be true.
Use a direct comparison to SS_ENCRYPTION instead.

The same king of test is already done the same way in sun8i_ss_run_task().

Fixes: 359e893e ("crypto: sun8i-ss - rework handling of IV")
Signed-off-by: Christophe JAILLET <christophe.jaillet@wanadoo.fr>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>

Mendocino and later platform don't use the platform feature mailbox for
communication for I2C arbitration, they rely upon ringing a doorbell.

Detect the platform by the device ID of the root port and choose the
appropriate method.

Link: https://lore.kernel.org/linux-i2c/20220916131854.687371-3-jsd@semihalf.com/Signed-off-by: Mario Limonciello <mario.limonciello@amd.com>
Acked-by: Jarkko Nikula <jarkko.nikula@linux.intel.com>
Reviewed-by: Mark Hasemeyer <markhas@chromium.org>
Tested-by: Mark Hasemeyer <markhas@chromium.org>
Acked-by: Wolfram Sang <wsa@kernel.org>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>

Currently the PSP semaphore communication base address is discovered
by using an MSR that is not architecturally guaranteed for future
platforms.  Also the mailbox that is utilized for communication with
the PSP may have other consumers in the kernel, so it's better to
make all communication go through a single driver.
Signed-off-by: Mario Limonciello <mario.limonciello@amd.com>
Reviewed-by: Mark Hasemeyer <markhas@chromium.org>
Acked-by: Jarkko Nikula <jarkko.nikula@linux.intel.com>
Tested-by: Mark Hasemeyer <markhas@chromium.org>
Acked-by: Wolfram Sang <wsa@kernel.org>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>

Move Power10 feature, PPC_MODULE_FEATURE_P10, definition to be in
arch/powerpc/include/asm/cpufeature.h.
Signed-off-by: Danny Tsen <dtsen@linux.ibm.com>
Acked-by: Michael Ellerman <mpe@ellerman.id.au> (powerpc)
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>

Remove Power10 dependency in Kconfig and detect Power10 feature at runtime.
Signed-off-by: Danny Tsen <dtsen@linux.ibm.com>
Acked-by: Michael Ellerman <mpe@ellerman.id.au> (powerpc)
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>

Add some test vectors for 128-bit cmac(camellia) as found in
draft-kato-ipsec-camellia-cmac96and128-01 section 6.2.

The document also shows vectors for camellia-cmac-96, and for VK with a
length greater than 16, but I'm not sure how to express those in testmgr.

This also leaves cts(cbc(camellia)) untested, but I can't seem to find any
tests for that that I could put into testmgr.
Signed-off-by: David Howells <dhowells@redhat.com>
cc: Herbert Xu <herbert@gondor.apana.org.au>
cc: Chuck Lever <chuck.lever@oracle.com>
cc: Scott Mayhew <smayhew@redhat.com>
cc: linux-nfs@vger.kernel.org
cc: linux-crypto@vger.kernel.org
Link: https://datatracker.ietf.org/doc/pdf/draft-kato-ipsec-camellia-cmac96and128-01Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>

Allow cryptd hashes to be cloned.  The underlying hash will be cloned.
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>

The cryptd hash template was still using the obsolete cra_init/cra_exit
interface.  Make it use the modern ahash init_tfm/exit_tfm instead.
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
Reviewed-by: Simon Horman <simon.horman@corigine.com>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>

Allow hmac to be cloned.  The underlying hash can be used directly
with a reference count.
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>

This patch adds the helpers crypto_clone_ahash and crypto_clone_shash.
They are the hash-specific counterparts of crypto_clone_tfm.

This allows code paths that cannot otherwise allocate a hash tfm
object to do so.  Once a new tfm has been obtained its key could
then be changed without impacting other users.

Note that only algorithms that implement clone_tfm can be cloned.
However, all keyless hashes can be cloned by simply reusing the
tfm object.
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
Reviewed-by: Simon Horman <simon.horman@corigine.com>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>

This patch adds the helper crypto_clone_tfm.  The purpose is to
allocate a tfm object with GFP_ATOMIC.  As we cannot sleep, the
object has to be cloned from an existing tfm object.

This allows code paths that cannot otherwise allocate a crypto_tfm
object to do so.  Once a new tfm has been obtained its key could
then be changed without impacting other users.
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
Reviewed-by: Simon Horman <simon.horman@corigine.com>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>

Add a crypto_tfm_get interface to allow tfm objects to be shared.
They can still be freed in the usual way.

This should only be done with tfm objects with no keys.  You must
also not modify the tfm flags in any way once it becomes shared.
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
Reviewed-by: Simon Horman <simon.horman@corigine.com>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>

Avoid cluttering up the kallsyms symbol table with entries that should
not end up in things like backtraces, as they have undescriptive and
generated identifiers.
Signed-off-by: Ard Biesheuvel <ardb@kernel.org>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>

Avoid cluttering up the kallsyms symbol table with entries that should
not end up in things like backtraces, as they have undescriptive and
generated identifiers.
Signed-off-by: Ard Biesheuvel <ardb@kernel.org>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>

Avoid cluttering up the kallsyms symbol table with entries that should
not end up in things like backtraces, as they have undescriptive and
generated identifiers.
Signed-off-by: Ard Biesheuvel <ardb@kernel.org>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>

Prefer RIP-relative addressing where possible, which removes the need
for boot time relocation fixups.
Signed-off-by: Ard Biesheuvel <ardb@kernel.org>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>

Prefer RIP-relative addressing where possible, which removes the need
for boot time relocation fixups.
Signed-off-by: Ard Biesheuvel <ardb@kernel.org>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>

Prefer RIP-relative addressing where possible, which removes the need
for boot time relocation fixups.
Co-developed-by: Thomas Garnier <thgarnie@chromium.org>
Signed-off-by: Thomas Garnier <thgarnie@chromium.org>
Signed-off-by: Ard Biesheuvel <ardb@kernel.org>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>

Prefer RIP-relative addressing where possible, which removes the need
for boot time relocation fixups.
Signed-off-by: Ard Biesheuvel <ardb@kernel.org>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>

Prefer RIP-relative addressing where possible, which removes the need
for boot time relocation fixups.
Co-developed-by: Thomas Garnier <thgarnie@chromium.org>
Signed-off-by: Thomas Garnier <thgarnie@chromium.org>
Signed-off-by: Ard Biesheuvel <ardb@kernel.org>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>

Prefer RIP-relative addressing where possible, which removes the need
for boot time relocation fixups.
Co-developed-by: Thomas Garnier <thgarnie@chromium.org>
Signed-off-by: Thomas Garnier <thgarnie@chromium.org>
Signed-off-by: Ard Biesheuvel <ardb@kernel.org>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>

Prefer RIP-relative addressing where possible, which removes the need
for boot time relocation fixups.
Co-developed-by: Thomas Garnier <thgarnie@chromium.org>
Signed-off-by: Thomas Garnier <thgarnie@chromium.org>
Signed-off-by: Ard Biesheuvel <ardb@kernel.org>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>

Prefer RIP-relative addressing where possible, which removes the need
for boot time relocation fixups.
Signed-off-by: Ard Biesheuvel <ardb@kernel.org>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>

Prefer RIP-relative addressing where possible, which removes the need
for boot time relocation fixups. In the GCM case, we can get rid of the
oversized permutation array entirely while at it.
Signed-off-by: Ard Biesheuvel <ardb@kernel.org>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>

Prefer RIP-relative addressing where possible, which removes the need
for boot time relocation fixups.
Signed-off-by: Ard Biesheuvel <ardb@kernel.org>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>

The crypt_ctl structure must be exactly 64 bytes long to work correctly,
and it has to be a power-of-two size to allow turning the
64-bit division in crypt_phys2virt() into a shift operation, avoiding
the link failure:

ERROR: modpost: "__aeabi_uldivmod" [drivers/crypto/intel/ixp4xx/ixp4xx_crypto.ko] undefined!

The failure now shows up because the driver is available for compile
testing after the move, and a previous fix turned the more descriptive
BUILD_BUG_ON() into a link error.

Change the variably-sized dma_addr_t into the expected 'u32' type that is
needed for the hardware, and reinstate the size check for all 32-bit
architectures to simplify debugging if it hits again.

Fixes: 1bc7fdbf ("crypto: ixp4xx - Move driver to drivers/crypto/intel/ixp4xx")
Signed-off-by: Arnd Bergmann <arnd@arndb.de>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>

caam driver needs to be aware of OP-TEE f/w presence, since some things
are done differently:

1. there is no access to controller's register page (note however that
some registers are aliased in job rings' register pages)

2 Due to this, MCFGR[PS] cannot be read and driver assumes
MCFGR[PS] = b'0 - engine using 32-bit address pointers.

This is in sync with the fact that:
-all i.MX SoCs currently use MCFGR[PS] = b'0
-only i.MX OP-TEE use cases don't allow access to controller register page
Signed-off-by: Horia GeantA <horia.geanta@nxp.com>
Signed-off-by: Meenakshi Aggarwal <meenakshi.aggarwal@nxp.com>
Reviewed-by: Gaurav Jain <gaurav.jain@nxp.com>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>

Use job ring register map, in place of controller register map
to access page 0 registers, as access to the controller register
map is not permitted.
Signed-off-by: Horia GeantA <horia.geanta@nxp.com>
Signed-off-by: Gaurav Jain <gaurav.jain@nxp.com>
Signed-off-by: Meenakshi Aggarwal <meenakshi.aggarwal@nxp.com>
Reviewed-by: Varun Sethi <v.sethi@nxp.com>
Reviewed-by: Gaurav Jain <gaurav.jain@nxp.com>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>

Unlike other command registers used by the PSP, only the lower 8 bytes are
used for communication for both command and status of the command.
Suggested-by: Mark Hasemeyer <markhas@chromium.org>
Signed-off-by: Mario Limonciello <mario.limonciello@amd.com>
Reviewed-by: Mark Hasemeyer <markhas@chromium.org>
Tested-by: Mark Hasemeyer <markhas@chromium.org>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>

If the doorbell failed to ring we return -EIO, but the caller can't
determine why it failed.  Pass the reason for the failure in an
argument for caller to investigate.
Suggested-by: Mark Hasemeyer <markhas@chromium.org>
Signed-off-by: Mario Limonciello <mario.limonciello@amd.com>
Reviewed-by: Mark Hasemeyer <markhas@chromium.org>
Tested-by: Mark Hasemeyer <markhas@chromium.org>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>

This is helpful not just for debugging problems, but also for investigating
captured logs later on.
Suggested-by: Grzegorz Bernacki <gjb@semihalf.com>
Signed-off-by: Mario Limonciello <mario.limonciello@amd.com>
Tested-by: Mark Hasemeyer <markhas@chromium.org>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>

The doorbell register set used for I2C arbitration is dedicated for this
purpose and there is no need to utilize other safety checks the platform
access register set uses.
Suggested-by: Mark Hasemeyer <markhas@chromium.org>
Signed-off-by: Mario Limonciello <mario.limonciello@amd.com>
Reviewed-by: Mark Hasemeyer <markhas@chromium.org>
Tested-by: Mark Hasemeyer <markhas@chromium.org>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>

A number of platforms are emitting the error:
```ccp: unable to access the device: you might be running a broken BIOS.```

This is expected behavior as CCP is no longer accessible from the PSP's
PCIe BAR so stop trying to probe CCP for 0x1649.

Cc: stable@vger.kernel.org
Signed-off-by: Mario Limonciello <mario.limonciello@amd.com>
Acked-by: Tom Lendacky <thomas.lendacky@amd.com>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>

A number of low-level functions were exposed in crypto.h.  Move
them into algapi.h (and internal.h).
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>

The BUILD_BUG_ON preventing compilation on foreign architectures
should be disabled when we're doing compile testing.

Fixes: 1bc7fdbf ("crypto: ixp4xx - Move driver to...")
Reported-by: kernel test robot <lkp@intel.com>
Link: https://lore.kernel.org/oe-kbuild-all/202304061846.G6cpPXiQ-lkp@intel.com/Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>

HiSTB TRNG are found on some HiSilicon STB SoCs.
Signed-off-by: David Yang <mmyangfl@gmail.com>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>

Change blocksize to match the cfb(aes) generic implementation.
Signed-off-by: Ryan Wanner <Ryan.Wanner@microchip.com>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>