• Peter Zijlstra's avatar
    mm, locking: Rework {set,clear,mm}_tlb_flush_pending() · 8b1b436d
    Peter Zijlstra authored
    Commit:
    
      af2c1401 ("mm: numa: guarantee that tlb_flush_pending updates are visible before page table updates")
    
    added smp_mb__before_spinlock() to set_tlb_flush_pending(). I think we
    can solve the same problem without this barrier.
    
    If instead we mandate that mm_tlb_flush_pending() is used while
    holding the PTL we're guaranteed to observe prior
    set_tlb_flush_pending() instances.
    
    For this to work we need to rework migrate_misplaced_transhuge_page()
    a little and move the test up into do_huge_pmd_numa_page().
    
    NOTE: this relies on flush_tlb_range() to guarantee:
    
       (1) it ensures that prior page table updates are visible to the
           page table walker and
       (2) it ensures that subsequent memory accesses are only made
           visible after the invalidation has completed
    
    This is required for architectures that implement TRANSPARENT_HUGEPAGE
    (arc, arm, arm64, mips, powerpc, s390, sparc, x86) or otherwise use
    mm_tlb_flush_pending() in their page-table operations (arm, arm64,
    x86).
    
    This appears true for:
    
     - arm (DSB ISB before and after),
     - arm64 (DSB ISHST before, and DSB ISH after),
     - powerpc (PTESYNC before and after),
     - s390 and x86 TLB invalidate are serializing instructions
    
    But I failed to understand the situation for:
    
     - arc, mips, sparc
    
    Now SPARC64 is a wee bit special in that flush_tlb_range() is a no-op
    and it flushes the TLBs using arch_{enter,leave}_lazy_mmu_mode()
    inside the PTL. It still needs to guarantee the PTL unlock happens
    _after_ the invalidate completes.
    
    Vineet, Ralf and Dave could you guys please have a look?
    Signed-off-by: default avatarPeter Zijlstra (Intel) <peterz@infradead.org>
    Acked-by: default avatarWill Deacon <will.deacon@arm.com>
    Cc: Andrew Morton <akpm@linux-foundation.org>
    Cc: Benjamin Herrenschmidt <benh@kernel.crashing.org>
    Cc: David S. Miller <davem@davemloft.net>
    Cc: Heiko Carstens <heiko.carstens@de.ibm.com>
    Cc: Linus Torvalds <torvalds@linux-foundation.org>
    Cc: Mel Gorman <mgorman@suse.de>
    Cc: Peter Zijlstra <peterz@infradead.org>
    Cc: Ralf Baechle <ralf@linux-mips.org>
    Cc: Rik van Riel <riel@redhat.com>
    Cc: Russell King <linux@armlinux.org.uk>
    Cc: Thomas Gleixner <tglx@linutronix.de>
    Cc: Vineet Gupta <vgupta@synopsys.com>
    Signed-off-by: default avatarIngo Molnar <mingo@kernel.org>
    8b1b436d
migrate.c 51.8 KB