• Rui Wang's avatar
    LoongArch: mm: Refactor TLB exception handlers · a2a84e36
    Rui Wang authored
    This patch simplifies TLB load, store and modify exception handlers:
    
    1. Reduce instructions, such as alu/csr and memory access;
    2. Execute tlb search instruction only in the fast path;
    3. Return directly from the fast path for both normal and huge pages;
    4. Re-tab the assembly for better vertical alignment.
    
    And fixes the concurrent modification issue of fast path for huge pages.
    
    This issue will occur in the following steps:
    
       CPU-1 (In TLB exception)         CPU-2 (In THP splitting)
    1: Load PMD entry (HUGE=1)
    2: Goto huge path
    3:                                  Store PMD entry (HUGE=0)
    4: Reload PMD entry (HUGE=0)
    5: Fill TLB entry (PA is incorrect)
    
    This patch also slightly improves the TLB processing performance:
    
    * Normal pages: 2.15%, Huge pages: 1.70%.
    
      #include <stdio.h>
      #include <stdlib.h>
      #include <unistd.h>
      #include <sys/mman.h>
    
      int main(int argc, char *argv[])
      {
            size_t page_size;
            size_t mem_size;
            size_t off;
            void *base;
            int flags;
            int i;
    
            if (argc < 2) {
                    fprintf(stderr, "%s MEM_SIZE [HUGE]\n", argv[0]);
                    return -1;
            }
    
            page_size = sysconf(_SC_PAGESIZE);
            flags = MAP_PRIVATE | MAP_ANONYMOUS;
            mem_size = strtoul(argv[1], NULL, 10);
            if (argc > 2)
                    flags |= MAP_HUGETLB;
    
            for (i = 0; i < 10; i++) {
                    base = mmap(NULL, mem_size, PROT_READ, flags, -1, 0);
                    if (base == MAP_FAILED) {
                            fprintf(stderr, "Map memory failed!\n");
                            return -1;
                    }
    
                    for (off = 0; off < mem_size; off += page_size)
                            *(volatile int *)(base + off);
    
                    munmap(base, mem_size);
            }
    
            return 0;
      }
    Signed-off-by: default avatarRui Wang <wangrui@loongson.cn>
    Signed-off-by: default avatarHuacai Chen <chenhuacai@loongson.cn>
    a2a84e36
tlbex.S 11.9 KB