• Michel Lespinasse's avatar
    mm: retry page fault when blocking on disk transfer · d065bd81
    Michel Lespinasse authored
    This change reduces mmap_sem hold times that are caused by waiting for
    disk transfers when accessing file mapped VMAs.
    
    It introduces the VM_FAULT_ALLOW_RETRY flag, which indicates that the call
    site wants mmap_sem to be released if blocking on a pending disk transfer.
    In that case, filemap_fault() returns the VM_FAULT_RETRY status bit and
    do_page_fault() will then re-acquire mmap_sem and retry the page fault.
    
    It is expected that the retry will hit the same page which will now be
    cached, and thus it will complete with a low mmap_sem hold time.
    
    Tests:
    
    - microbenchmark: thread A mmaps a large file and does random read accesses
      to the mmaped area - achieves about 55 iterations/s. Thread B does
      mmap/munmap in a loop at a separate location - achieves 55 iterations/s
      before, 15000 iterations/s after.
    
    - We are seeing related effects in some applications in house, which show
      significant performance regressions when running without this change.
    
    [akpm@linux-foundation.org: fix warning & crash]
    Signed-off-by: default avatarMichel Lespinasse <walken@google.com>
    Acked-by: default avatarRik van Riel <riel@redhat.com>
    Acked-by: default avatarLinus Torvalds <torvalds@linux-foundation.org>
    Cc: Nick Piggin <nickpiggin@yahoo.com.au>
    Reviewed-by: default avatarWu Fengguang <fengguang.wu@intel.com>
    Cc: Ying Han <yinghan@google.com>
    Cc: Peter Zijlstra <a.p.zijlstra@chello.nl>
    Cc: Ingo Molnar <mingo@elte.hu>
    Cc: Thomas Gleixner <tglx@linutronix.de>
    Acked-by: default avatar"H. Peter Anvin" <hpa@zytor.com>
    Cc: <linux-arch@vger.kernel.org>
    Signed-off-by: default avatarAndrew Morton <akpm@linux-foundation.org>
    Signed-off-by: default avatarLinus Torvalds <torvalds@linux-foundation.org>
    d065bd81
filemap.c 66.5 KB