• Vlastimil Babka's avatar
    slab, rust: extend kmalloc() alignment guarantees to remove Rust padding · ad59baa3
    Vlastimil Babka authored
    Slab allocators have been guaranteeing natural alignment for
    power-of-two sizes since commit 59bb4798 ("mm, sl[aou]b: guarantee
    natural alignment for kmalloc(power-of-two)"), while any other sizes are
    guaranteed to be aligned only to ARCH_KMALLOC_MINALIGN bytes (although
    in practice are aligned more than that in non-debug scenarios).
    
    Rust's allocator API specifies size and alignment per allocation, which
    have to satisfy the following rules, per Alice Ryhl [1]:
    
      1. The alignment is a power of two.
      2. The size is non-zero.
      3. When you round up the size to the next multiple of the alignment,
         then it must not overflow the signed type isize / ssize_t.
    
    In order to map this to kmalloc()'s guarantees, some requested
    allocation sizes have to be padded to the next power-of-two size [2].
    For example, an allocation of size 96 and alignment of 32 will be padded
    to an allocation of size 128, because the existing kmalloc-96 bucket
    doesn't guarantee alignent above ARCH_KMALLOC_MINALIGN. Without slab
    debugging active, the layout of the kmalloc-96 slabs however naturally
    align the objects to 32 bytes, so extending the size to 128 bytes is
    wasteful.
    
    To improve the situation we can extend the kmalloc() alignment
    guarantees in a way that
    
    1) doesn't change the current slab layout (and thus does not increase
       internal fragmentation) when slab debugging is not active
    2) reduces waste in the Rust allocator use case
    3) is a superset of the current guarantee for power-of-two sizes.
    
    The extended guarantee is that alignment is at least the largest
    power-of-two divisor of the requested size. For power-of-two sizes the
    largest divisor is the size itself, but let's keep this case documented
    separately for clarity.
    
    For current kmalloc size buckets, it means kmalloc-96 will guarantee
    alignment of 32 bytes and kmalloc-196 will guarantee 64 bytes.
    
    This covers the rules 1 and 2 above of Rust's API as long as the size is
    a multiple of the alignment. The Rust layer should now only need to
    round up the size to the next multiple if it isn't, while enforcing the
    rule 3.
    
    Implementation-wise, this changes the alignment calculation in
    create_boot_cache(). While at it also do the calulation only for caches
    with the SLAB_KMALLOC flag, because the function is also used to create
    the initial kmem_cache and kmem_cache_node caches, where no alignment
    guarantee is necessary.
    
    In the Rust allocator's krealloc_aligned(), remove the code that padded
    sizes to the next power of two (suggested by Alice Ryhl) as it's no
    longer necessary with the new guarantees.
    Reported-by: default avatarAlice Ryhl <aliceryhl@google.com>
    Reported-by: default avatarBoqun Feng <boqun.feng@gmail.com>
    Link: https://lore.kernel.org/all/CAH5fLggjrbdUuT-H-5vbQfMazjRDpp2%2Bk3%3DYhPyS17ezEqxwcw@mail.gmail.com/ [1]
    Link: https://lore.kernel.org/all/CAH5fLghsZRemYUwVvhk77o6y1foqnCeDzW4WZv6ScEWna2+_jw@mail.gmail.com/ [2]
    Reviewed-by: default avatarBoqun Feng <boqun.feng@gmail.com>
    Acked-by: default avatarRoman Gushchin <roman.gushchin@linux.dev>
    Reviewed-by: default avatarAlice Ryhl <aliceryhl@google.com>
    Signed-off-by: default avatarVlastimil Babka <vbabka@suse.cz>
    ad59baa3
allocator.rs 3.01 KB