• Shaohua Li's avatar
    x86: Scale up the number of TLB invalidate vectors with NR_CPUs, up to 32 · 70e4a369
    Shaohua Li authored
    Make the maxium TLB invalidate vectors depend on NR_CPUS linearly,
    with a maximum of 32 vectors.
    
    We currently only have 8 vectors for TLB invalidation and that is clearly
    inadequate. If we have a lot of CPUs, the CPUs need share the 8 vectors and
    tlbstate_lock is used to protect them. flush_tlb_page() is
    heavily used in page reclaim, which will cause a lot of lock
    contention for tlbstate_lock.
    
    Andi Kleen suggested increasing the vectors number to 32, which should be
    good for current typical systems to reduce the tlbstate_lock contention.
    
    My test system has 4 sockets and 64G memory, and 64 CPUs. My
    workload creates 64 processes. Each process mmap reads a big
    empty sparse file. The total size of the files are 2*total_mem,
    so this will cause a lot of page reclaim.
    
    Below is the result I get from perf call-graph profiling:
    
     without the patch:
     ------------------
    
        24.25%           usemem  [kernel]                                   [k] _raw_spin_lock
                         |
                         --- _raw_spin_lock
                            |
                            |--42.15%-- native_flush_tlb_others
    
     with the patch:
     ------------------
    
        14.96%           usemem  [kernel]                                   [k] _raw_spin_lock
                         |
                         --- _raw_spin_lock
                            |--13.89%-- native_flush_tlb_others
    
    So this heavily reduces the tlbstate_lock contention.
    Suggested-by: default avatarAndi Kleen <andi@firstfloor.org>
    Signed-off-by: default avatarShaohua Li <shaohua.li@intel.com>
    Cc: Eric Dumazet <eric.dumazet@gmail.com>
    Cc: Arnaldo Carvalho de Melo <acme@redhat.com>
    Cc: Frederic Weisbecker <fweisbec@gmail.com>
    Cc: Peter Zijlstra <a.p.zijlstra@chello.nl>
    LKML-Reference: <1295232727.1949.709.camel@sli10-conroe>
    Signed-off-by: default avatarIngo Molnar <mingo@elte.hu>
    70e4a369
irq_vectors.h 5.11 KB