- 27 Oct, 2023 40 commits
-
-
Eric Biggers authored
The "ahash" API provides access to both CPU-based and hardware offload- based implementations of hash algorithms. Typically the former are implemented as "shash" algorithms under the hood, while the latter are implemented as "ahash" algorithms. The "ahash" API provides access to both. Various kernel subsystems use the ahash API because they want to support hashing hardware offload without using a separate API for it. Yet, the common case is that a crypto accelerator is not actually being used, and ahash is just wrapping a CPU-based shash algorithm. This patch optimizes the ahash API for that common case by eliminating the extra indirect call for each ahash operation on top of shash. It also fixes the double-counting of crypto stats in this scenario (though CONFIG_CRYPTO_STATS should *not* be enabled by anyone interested in performance anyway...), and it eliminates redundant checking of CRYPTO_TFM_NEED_KEY. As a bonus, it also shrinks struct crypto_ahash. Signed-off-by:
Eric Biggers <ebiggers@google.com> Signed-off-by:
Herbert Xu <herbert@gondor.apana.org.au>
-
Eric Biggers authored
Since the previous patch made crypto_shash_type visible to ahash.c, change checks for '->cra_type != &crypto_ahash_type' to '->cra_type == &crypto_shash_type'. This makes more sense and avoids having to forward-declare crypto_ahash_type. The result is still the same, since the type is either shash or ahash here. Signed-off-by:
-
Eric Biggers authored
The functions that are involved in implementing the ahash API on top of an shash algorithm belong better in ahash.c, not in shash.c where they currently are. Move them. Signed-off-by:
-
Eric Biggers authored
The function pointer crypto_ahash::init is an internal implementation detail of the ahash API that exists to help it support both ahash and shash algorithms. With an upcoming refactoring of how the ahash API supports shash algorithms, this field will be removed. Some drivers are invoking crypto_ahash::init to call into their own code, which is unnecessary and inefficient. The talitos driver is one of those drivers. Make it just call its own code directly. Signed-off-by:
-
Eric Biggers authored
The function pointer crypto_ahash::init is an internal implementation detail of the ahash API that exists to help it support both ahash and shash algorithms. With an upcoming refactoring of how the ahash API supports shash algorithms, this field will be removed. Some drivers are invoking crypto_ahash::init to call into their own code, which is unnecessary and inefficient. The chelsio driver is one of those drivers. Make it just call its own code directly. Signed-off-by:
-
Eric Biggers authored
Improve the file comment for crypto/ahash.c. Signed-off-by:
-
Eric Biggers authored
struct ahash_request_priv is unused, so remove it. Signed-off-by:
-
Eric Biggers authored
crypto_ahash_alignmask() no longer has any callers, and it always returns 0 now that neither ahash nor shash algorithms support nonzero alignmasks anymore. Therefore, remove it. Signed-off-by:
-
Eric Biggers authored
Now that the alignmask for ahash and shash algorithms is always 0, simplify crypto_gcm_create_common() accordingly. Signed-off-by:
-
Eric Biggers authored
Now that the alignmask for ahash and shash algorithms is always 0, simplify chachapoly_create() accordingly. Signed-off-by:
-
Eric Biggers authored
Now that the alignmask for ahash and shash algorithms is always 0, simplify crypto_ccm_create_common() accordingly. Signed-off-by:
-
Eric Biggers authored
Now that the alignmask for ahash and shash algorithms is always 0, crypto_ahash_alignmask() always returns 0 and will be removed. In preparation for this, stop checking crypto_ahash_alignmask() in ah6.c. Signed-off-by:
-
Eric Biggers authored
Now that the alignmask for ahash and shash algorithms is always 0, crypto_ahash_alignmask() always returns 0 and will be removed. In preparation for this, stop checking crypto_ahash_alignmask() in ah4.c. Signed-off-by:
-
Eric Biggers authored
Now that the alignmask for ahash and shash algorithms is always 0, crypto_ahash_alignmask() always returns 0 and will be removed. In preparation for this, stop checking crypto_ahash_alignmask() in testmgr. As a result of this change, test_sg_division::offset_relative_to_alignmask and testvec_config::key_offset_relative_to_alignmask no longer have any effect on ahash (or shash) algorithms. Therefore, also stop setting these flags in default_hash_testvec_configs[]. Signed-off-by:
-
Eric Biggers authored
Now that the alignmask for ahash and shash algorithms is always 0, simplify the code in authenc accordingly. Signed-off-by:
-
Eric Biggers authored
Now that the alignmask for ahash and shash algorithms is always 0, simplify the code in authenc accordingly. Signed-off-by:
-
Eric Biggers authored
Currently, the ahash API checks the alignment of all key and result buffers against the algorithm's declared alignmask, and for any unaligned buffers it falls back to manually aligned temporary buffers. This is virtually useless, however. First, since it does not apply to the message, its effect is much more limited than e.g. is the case for the alignmask for "skcipher". Second, the key and result buffers are given as virtual addresses and cannot (in general) be DMA'ed into, so drivers end up having to copy to/from them in software anyway. As a result it's easy to use memcpy() or the unaligned access helpers. The crypto_hash_walk_*() helper functions do use the alignmask to align the message. But with one exception those are only used for shash algorithms being exposed via the ahash API, not for native ahashes, and aligning the message is not required in this case, especially now that alignmask support has been removed from shash. The exception is the n2_core driver, which doesn't set an alignmask. In any case, no ahash algorithms actually set a nonzero alignmask anymore. Therefore, remove support for it from ahash. The benefit is that all the code to handle "misaligned" buffers in the ahash API goes away, reducing the overhead of the ahash API. This follows the same change that was made to shash. Signed-off-by:
-
Eric Biggers authored
The crypto API's support for alignmasks for ahash algorithms is nearly useless, as its only effect is to cause the API to align the key and result buffers. The drivers that happen to be specifying an alignmask for ahash rarely actually need it. When they do, it's easily fixable, especially considering that these buffers cannot be used for DMA. In preparation for removing alignmask support from ahash, this patch makes the stm32 driver no longer use it. This driver didn't actually rely on it; it only writes to the result buffer in stm32_hash_finish(), simply using memcpy(). And stm32_hash_setkey() does not assume any alignment for the key buffer. Signed-off-by:
-
Eric Biggers authored
The crypto API's support for alignmasks for ahash algorithms is nearly useless, as its only effect is to cause the API to align the key and result buffers. The drivers that happen to be specifying an alignmask for ahash rarely actually need it. When they do, it's easily fixable, especially considering that these buffers cannot be used for DMA. In preparation for removing alignmask support from ahash, this patch makes the starfive driver no longer use it. This driver did actually rely on it, but only for storing to the result buffer using int stores in starfive_hash_copy_hash(). This patch makes starfive_hash_copy_hash() use put_unaligned() instead. (It really should use a specific endianness, but that's an existing bug.) Signed-off-by:
-
Eric Biggers authored
The crypto API's support for alignmasks for ahash algorithms is nearly useless, as its only effect is to cause the API to align the key and result buffers. The drivers that happen to be specifying an alignmask for ahash rarely actually need it. When they do, it's easily fixable, especially considering that these buffers cannot be used for DMA. In preparation for removing alignmask support from ahash, this patch makes the rockchip driver no longer use it. This driver didn't actually rely on it; it only writes to the result buffer in rk_hash_run(), already using put_unaligned_le32(). And this driver only supports unkeyed hash algorithms, so the key buffer need not be considered. Signed-off-by:
-
Eric Biggers authored
The crypto API's support for alignmasks for ahash algorithms is nearly useless, as its only effect is to cause the API to align the key and result buffers. The drivers that happen to be specifying an alignmask for ahash rarely actually need it. When they do, it's easily fixable, especially considering that these buffers cannot be used for DMA. In preparation for removing alignmask support from ahash, this patch makes the omap-sham driver no longer use it. This driver did actually rely on it, but only for storing to the result buffer using __u32 stores in omap_sham_copy_ready_hash(). This patch makes omap_sham_copy_ready_hash() use put_unaligned() instead. (It really should use a specific endianness, but that's an existing bug.) Signed-off-by:
-
Eric Biggers authored
The crypto API's support for alignmasks for ahash algorithms is nearly useless, as its only effect is to cause the API to align the key and result buffers. The drivers that happen to be specifying an alignmask for ahash rarely actually need it. When they do, it's easily fixable, especially considering that these buffers cannot be used for DMA. In preparation for removing alignmask support from ahash, this patch makes the talitos driver no longer use it. This driver didn't actually rely on it; it only writes to the result buffer in common_nonsnoop_hash_unmap(), simply using memcpy(). And this driver's "ahash_setkey()" function does not assume any alignment for the key buffer. Signed-off-by:
-
Eric Biggers authored
The crypto API's support for alignmasks for ahash algorithms is nearly useless, as its only effect is to cause the API to align the key and result buffers. The drivers that happen to be specifying an alignmask for ahash rarely actually need it. When they do, it's easily fixable, especially considering that these buffers cannot be used for DMA. In preparation for removing alignmask support from ahash, this patch makes the s5p-sss driver no longer use it. This driver didn't actually rely on it; it only writes to the result buffer in s5p_hash_copy_result(), simply using memcpy(). And this driver only supports unkeyed hash algorithms, so the key buffer need not be considered. Signed-off-by:
-
Eric Biggers authored
The crypto API's support for alignmasks for ahash algorithms is nearly useless, as its only effect is to cause the API to align the key and result buffers. The drivers that happen to be specifying an alignmask for ahash rarely actually need it. When they do, it's easily fixable, especially considering that these buffers cannot be used for DMA. In preparation for removing alignmask support from ahash, this patch makes the mxs-dcp driver no longer use it. This driver didn't actually rely on it; it only writes to the result buffer in dcp_sha_req_to_buf(), using a bytewise copy. And this driver only supports unkeyed hash algorithms, so the key buffer need not be considered. Signed-off-by:
-
Eric Biggers authored
The crypto API's support for alignmasks for ahash algorithms is nearly useless, as its only effect is to cause the API to align the key and result buffers. The drivers that happen to be specifying an alignmask for ahash rarely actually need it. When they do, it's easily fixable, especially considering that these buffers cannot be used for DMA. In preparation for removing alignmask support from ahash, this patch makes the artpec6 driver no longer use it. This driver is unusual in that it DMAs the digest directly to the result buffer. This is broken because the crypto API provides the result buffer as an arbitrary virtual address, which might not be valid for DMA, even after the crypto API applies the alignmask. Maybe the alignmask (which this driver set only to 3) made this code work in a few more cases than it otherwise would have. But even if so, it doesn't make sense for this single driver that is broken anyway to block removal of the alignmask support. Signed-off-by:
-
Eric Biggers authored
The crypto API's support for alignmasks for ahash algorithms is nearly useless, as its only effect is to cause the API to align the key and result buffers. The drivers that happen to be specifying an alignmask for ahash rarely actually need it. When they do, it's easily fixable, especially considering that these buffers cannot be used for DMA. In preparation for removing alignmask support from ahash, this patch makes the atmel driver no longer use it. This driver didn't actually rely on it; it only writes to the result buffer in atmel_sha_copy_ready_hash(), simply using memcpy(). And this driver didn't set an alignmask for any keyed hash algorithms, so the key buffer need not be considered. Signed-off-by:
-
Eric Biggers authored
The crypto API's support for alignmasks for ahash algorithms is nearly useless, as its only effect is to cause the API to align the key and result buffers. The drivers that happen to be specifying an alignmask for ahash rarely actually need it. When they do, it's easily fixable, especially considering that these buffers cannot be used for DMA. In preparation for removing alignmask support from ahash, this patch makes the sun8i-ss driver no longer use it. This driver didn't actually rely on it; it only writes to the result buffer in sun8i_ss_hash_run(), simply using memcpy(). And sun8i_ss_hmac_setkey() does not assume any alignment for the key buffer. Signed-off-by:
Corentin Labbe <clabbe.montjoie@gmail.com> Signed-off-by:
-
Eric Biggers authored
The crypto API's support for alignmasks for ahash algorithms is nearly useless, as its only effect is to cause the API to align the key and result buffers. The drivers that happen to be specifying an alignmask for ahash rarely actually need it. When they do, it's easily fixable, especially considering that these buffers cannot be used for DMA. In preparation for removing alignmask support from ahash, this patch makes the sun8i-ce driver no longer use it. This driver didn't actually rely on it; it only writes to the result buffer in sun8i_ce_hash_run(), simply using memcpy(). And this driver only supports unkeyed hash algorithms, so the key buffer need not be considered. Signed-off-by:
-
Eric Biggers authored
The crypto API's support for alignmasks for ahash algorithms is nearly useless, as its only effect is to cause the API to align the key and result buffers. The drivers that happen to be specifying an alignmask for ahash rarely actually need it. When they do, it's easily fixable, especially considering that these buffers cannot be used for DMA. In preparation for removing alignmask support from ahash, this patch makes the sun4i-ss driver no longer use it. This driver didn't actually rely on it; it only writes to the result buffer in sun4i_hash(), already using the unaligned access helpers. And this driver only supports unkeyed hash algorithms, so the key buffer need not be considered. Signed-off-by:
-
Eric Biggers authored
crypto_shash_ctx_aligned() is no longer used, and it is useless now that shash algorithms don't support nonzero alignmasks, so remove it. Also remove crypto_tfm_ctx_aligned() which was only called by crypto_shash_ctx_aligned(). It's unlikely to be useful again, since it seems inappropriate to use cra_alignmask to represent alignment for the tfm context when it already means alignment for inputs/outputs. Signed-off-by:
-
Lukas Wunner authored
Per section 4.c. of the IETF Trust Legal Provisions, "Code Components" in IETF Documents are licensed on the terms of the BSD-3-Clause license: https://trustee.ietf.org/documents/trust-legal-provisions/tlp-5/ The term "Code Components" specifically includes ASN.1 modules: https://trustee.ietf.org/documents/trust-legal-provisions/code-components-list-3/ Add an SPDX identifier as well as a copyright notice pursuant to section 6.d. of the Trust Legal Provisions to all ASN.1 modules in the tree which are derived from IETF Documents. Section 4.d. of the Trust Legal Provisions requests that each Code Component identify the RFC from which it is taken, so link that RFC in every ASN.1 module. Signed-off-by:
Lukas Wunner <lukas@wunner.de> Signed-off-by:
-
Ashish Kalra authored
Brijesh is no longer with AMD. Signed-off-by:
Ashish Kalra <ashish.kalra@amd.com> Reviewed-by:
Michael Roth <michael.roth@amd.com> Acked-by:
Tom Lendacky <thomas.lendacky@amd.com> Signed-off-by:
-
Giovanni Cabiddu authored
If a request has the flag CRYPTO_TFM_REQ_MAY_BACKLOG set, the function qat_alg_send_message_maybacklog(), enqueues it in a backlog list if either (1) there is already at least one request in the backlog list, or (2) the HW ring is nearly full or (3) the enqueue to the HW ring fails. If an interrupt occurs right before the lock in qat_alg_backlog_req() is taken and the backlog queue is being emptied, then there is no request in the HW queues that can trigger a subsequent interrupt that can clear the backlog queue. In addition subsequent requests are enqueued to the backlog list and not sent to the hardware. Fix it by holding the lock while taking the decision if the request needs to be included in the backlog queue or not. This synchronizes the flow with the interrupt handler that drains the backlog queue. For performance reasons, the logic has been changed to try to enqueue first without holding the lock. Fixes: 38682383 ("crypto: qat - add backlog mechanism") Reported-by:
Mikulas Patocka <mpatocka@redhat.com> Closes: https://lore.kernel.org/all/af9581e2-58f9-cc19-428f-6f18f1f83d54@redhat.com/T/Signed-off-by:
Giovanni Cabiddu <giovanni.cabiddu@intel.com> Reviewed-by:
-
Giovanni Cabiddu authored
The file adf_cfg_services.h cannot be included in header files since it instantiates the structure adf_cfg_services. Move that structure to its own file and export the symbol. This does not introduce any functional change. Signed-off-by:
Damian Muszynski <damian.muszynski@intel.com> Signed-off-by:
-
Ciunas Bennett authored
Add the attribute `num_rps` to the `qat` attribute group. This returns the number of ring pairs that a single device has. This allows to know the maximum value that can be set to the attribute `rp2svc`. Signed-off-by:
Ciunas Bennett <ciunas.bennett@intel.com> Reviewed-by:
Tero Kristo <tero.kristo@linux.intel.com> Signed-off-by:
-
Ciunas Bennett authored
Add the attribute `rp2svc` to the `qat` attribute group. This provides a way for a user to query a specific ring pair for the type of service that is currently configured for. When read, the service will be returned for the defined ring pair. When written to this value will be stored as the ring pair to return the service of. Signed-off-by:
-
Ciunas Bennett authored
Add an interface for the rate limiting feature which allows to add, remove and modify a QAT SLA (Service Level Agreement). This adds a new sysfs attribute group, `qat_rl`, which can be accessed from /sys/bus/pci/devices/<BUS:DEV:FUNCTION> with the following hierarchy: |-+ qat_rl |---- id (RW) # SLA identifier |---- cir (RW) # Committed Information Rate |---- pir (RW) # Peak Information Rate |---- srv (RW) # Service to be rate limited |---- rp (RW) (HEX) # Ring pairs to be rate limited |---- cap_rem (RW) # Remaining capability for a service |---- sla_op (WO) # Allows to perform an operation on an SLA The API works by setting the appropriate RW attributes and then issuing a command through the `sla_op`. For example, to create an SLA, a user needs to input the necessary data into the attributes cir, pir, srv and rp and then write into `sla_op` the command `add` to execute the operation. The API also provides `cap_rem` attribute to get information about the remaining device capability within a certain service which is required when setting an SLA. Signed-off-by: -
Damian Muszynski authored
The Rate Limiting (RL) feature allows to control the rate of requests that can be submitted on a ring pair (RP). This allows sharing a QAT device among multiple users while ensuring a guaranteed throughput. The driver provides a mechanism that allows users to set policies, that are programmed to the device. The device is then enforcing those policies. Configuration of RL is accomplished through entities called SLAs (Service Level Agreement). Each SLA object gets a unique identifier and defines the limitations for a single service across up to four ring pairs (RPs count allocated to a single VF). The rate is determined using two fields: * CIR (Committed Information Rate), i.e., the guaranteed rate. * PIR (Peak Information Rate), i.e., the maximum rate achievable when the device has available resources. The rate values are expressed in permille scale i.e. 0-1000. Ring pair selection is achieved by providing a 64-bit mask, where each bit corresponds to one of the ring pairs. This adds an interface and logic that allow to add, update, retrieve and remove an SLA. Signed-off-by: -
Damian Muszynski authored
The QAT firmware provides a mechanism to retrieve its capabilities through the init admin interface. Add logic to retrieve the firmware capability mask from the firmware through the init/admin channel. This mask reports if the power management, telemetry and rate limiting features are supported. The fw capabilities are stored in the accel_dev structure and are used to detect if a certain feature is supported by the firmware loaded in the device. This is supported only by devices which have an admin AE. Signed-off-by:
-
Damian Muszynski authored
Some enums use the macro BIT. Include bits.h as it is missing. Signed-off-by:
-
