• Ryan Roberts's avatar
    arm64/mm: implement new [get_and_]clear_full_ptes() batch APIs · 6b1e4efb
    Ryan Roberts authored
    Optimize the contpte implementation to fix some of the
    exit/munmap/dontneed performance regression introduced by the initial
    contpte commit.  Subsequent patches will solve it entirely.
    
    During exit(), munmap() or madvise(MADV_DONTNEED), mappings must be
    cleared.  Previously this was done 1 PTE at a time.  But the core-mm
    supports batched clear via the new [get_and_]clear_full_ptes() APIs.  So
    let's implement those APIs and for fully covered contpte mappings, we no
    longer need to unfold the contpte.  This significantly reduces unfolding
    operations, reducing the number of tlbis that must be issued.
    
    Link: https://lkml.kernel.org/r/20240215103205.2607016-15-ryan.roberts@arm.comSigned-off-by: default avatarRyan Roberts <ryan.roberts@arm.com>
    Tested-by: default avatarJohn Hubbard <jhubbard@nvidia.com>
    Acked-by: default avatarMark Rutland <mark.rutland@arm.com>
    Acked-by: default avatarCatalin Marinas <catalin.marinas@arm.com>
    Cc: Alistair Popple <apopple@nvidia.com>
    Cc: Andrey Ryabinin <ryabinin.a.a@gmail.com>
    Cc: Ard Biesheuvel <ardb@kernel.org>
    Cc: Barry Song <21cnbao@gmail.com>
    Cc: Borislav Petkov (AMD) <bp@alien8.de>
    Cc: Dave Hansen <dave.hansen@linux.intel.com>
    Cc: David Hildenbrand <david@redhat.com>
    Cc: "H. Peter Anvin" <hpa@zytor.com>
    Cc: Ingo Molnar <mingo@redhat.com>
    Cc: James Morse <james.morse@arm.com>
    Cc: Kefeng Wang <wangkefeng.wang@huawei.com>
    Cc: Marc Zyngier <maz@kernel.org>
    Cc: Matthew Wilcox (Oracle) <willy@infradead.org>
    Cc: Thomas Gleixner <tglx@linutronix.de>
    Cc: Will Deacon <will@kernel.org>
    Cc: Yang Shi <shy828301@gmail.com>
    Cc: Zi Yan <ziy@nvidia.com>
    Signed-off-by: default avatarAndrew Morton <akpm@linux-foundation.org>
    6b1e4efb
pgtable.h 40.9 KB