• Pingfan Liu's avatar
    arm64/mm: save memory access in check_and_switch_context() fast switch path · c4885bbb
    Pingfan Liu authored
    On arm64, smp_processor_id() reads a per-cpu `cpu_number` variable,
    using the per-cpu offset stored in the tpidr_el1 system register. In
    some cases we generate a per-cpu address with a sequence like:
    
      cpu_ptr = &per_cpu(ptr, smp_processor_id());
    
    Which potentially incurs a cache miss for both `cpu_number` and the
    in-memory `__per_cpu_offset` array. This can be written more optimally
    as:
    
      cpu_ptr = this_cpu_ptr(ptr);
    
    Which only needs the offset from tpidr_el1, and does not need to
    load from memory.
    
    The following two test cases show a small performance improvement measured
    on a 46-cpus qualcomm machine with 5.8.0-rc4 kernel.
    
    Test 1: (about 0.3% improvement)
        #cat b.sh
        make clean && make all -j138
        #perf stat --repeat 10 --null --sync sh b.sh
    
        - before this patch
         Performance counter stats for 'sh b.sh' (10 runs):
    
                    298.62 +- 1.86 seconds time elapsed  ( +-  0.62% )
    
        - after this patch
         Performance counter stats for 'sh b.sh' (10 runs):
    
                   297.734 +- 0.954 seconds time elapsed  ( +-  0.32% )
    
    Test 2: (about 1.69% improvement)
         'perf stat -r 10 perf bench sched messaging'
            Then sum the total time of 'sched/messaging' by manual.
    
        - before this patch
          total 0.707 sec for 10 times
        - after this patch
          totol 0.695 sec for 10 times
    Signed-off-by: default avatarPingfan Liu <kernelfans@gmail.com>
    Acked-by: default avatarMark Rutland <mark.rutland@arm.com>
    Cc: Will Deacon <will@kernel.org>
    Cc: Steve Capper <steve.capper@arm.com>
    Cc: Mark Rutland <mark.rutland@arm.com>
    Cc: Vladimir Murzin <vladimir.murzin@arm.com>
    Cc: Jean-Philippe Brucker <jean-philippe@linaro.org>
    Link: https://lore.kernel.org/r/1594389852-19949-1-git-send-email-kernelfans@gmail.comSigned-off-by: default avatarCatalin Marinas <catalin.marinas@arm.com>
    c4885bbb
mmu_context.h 6.61 KB