• Joonsoo Kim's avatar
    mm/slab: refill cpu cache through a new slab without holding a node lock · 213b4695
    Joonsoo Kim authored
    Until now, cache growing makes a free slab on node's slab list and then
    we can allocate free objects from it.  This necessarily requires to hold
    a node lock which is very contended.  If we refill cpu cache before
    attaching it to node's slab list, we can avoid holding a node lock as
    much as possible because this newly allocated slab is only visible to
    the current task.  This will reduce lock contention.
    
    Below is the result of concurrent allocation/free in slab allocation
    benchmark made by Christoph a long time ago.  I make the output simpler.
    The number shows cycle count during alloc/free respectively so less is
    better.
    
      * Before
      Kmalloc N*alloc N*free(32): Average=355/750
      Kmalloc N*alloc N*free(64): Average=452/812
      Kmalloc N*alloc N*free(128): Average=559/1070
      Kmalloc N*alloc N*free(256): Average=1176/980
      Kmalloc N*alloc N*free(512): Average=1939/1189
      Kmalloc N*alloc N*free(1024): Average=3521/1278
      Kmalloc N*alloc N*free(2048): Average=7152/1838
      Kmalloc N*alloc N*free(4096): Average=13438/2013
    
      * After
      Kmalloc N*alloc N*free(32): Average=248/966
      Kmalloc N*alloc N*free(64): Average=261/949
      Kmalloc N*alloc N*free(128): Average=314/1016
      Kmalloc N*alloc N*free(256): Average=741/1061
      Kmalloc N*alloc N*free(512): Average=1246/1152
      Kmalloc N*alloc N*free(1024): Average=2437/1259
      Kmalloc N*alloc N*free(2048): Average=4980/1800
      Kmalloc N*alloc N*free(4096): Average=9000/2078
    
    It shows that contention is reduced for all the object sizes and
    performance increases by 30 ~ 40%.
    Signed-off-by: default avatarJoonsoo Kim <iamjoonsoo.kim@lge.com>
    Cc: Jesper Dangaard Brouer <brouer@redhat.com>
    Cc: Christoph Lameter <cl@linux.com>
    Cc: Pekka Enberg <penberg@kernel.org>
    Cc: David Rientjes <rientjes@google.com>
    Signed-off-by: default avatarAndrew Morton <akpm@linux-foundation.org>
    Signed-off-by: default avatarLinus Torvalds <torvalds@linux-foundation.org>
    213b4695
slab.c 108 KB