• Johannes Weiner's avatar
    mm: fadvise: avoid expensive remote LRU cache draining after FADV_DONTNEED · 4dd72b4a
    Johannes Weiner authored
    When FADV_DONTNEED cannot drop all pages in the range, it observes that
    some pages might still be on per-cpu LRU caches after recent
    instantiation and so initiates remote calls to all CPUs to flush their
    local caches.  However, in most cases, the fadvise happens from the same
    context that instantiated the pages, and any pre-LRU pages in the
    specified range are most likely sitting on the local CPU's LRU cache,
    and so in many cases this results in unnecessary remote calls, which, in
    a loaded system, can hold up the fadvise() call significantly.
    
    [ I didn't record it in the extreme case we observed at Facebook,
      unfortunately. We had a slow-to-respond system and noticed it
      lru_add_drain_all() leading the profile during fadvise calls. This
      patch came out of thinking about the code and how we commonly call
      FADV_DONTNEED.
    
      FWIW, I wrote a silly directory tree walker/searcher that recurses
      through /usr to read and FADV_DONTNEED each file it finds. On a 2
      socket 40 ht machine, over 1% is spent in lru_add_drain_all(). With
      the patch, that cost is gone; the local drain cost shows at 0.09%. ]
    
    Try to avoid the remote call by flushing the local LRU cache before even
    attempting to invalidate anything.  It's a cheap operation, and the
    local LRU cache is the most likely to hold any pre-LRU pages in the
    specified fadvise range.
    
    Link: http://lkml.kernel.org/r/20161214210017.GA1465@cmpxchg.orgSigned-off-by: default avatarJohannes Weiner <hannes@cmpxchg.org>
    Acked-by: default avatarVlastimil Babka <vbabka@suse.cz>
    Acked-by: default avatarMel Gorman <mgorman@suse.de>
    Acked-by: default avatarHillf Danton <hillf.zj@alibaba-inc.com>
    Signed-off-by: default avatarAndrew Morton <akpm@linux-foundation.org>
    Signed-off-by: default avatarLinus Torvalds <torvalds@linux-foundation.org>
    4dd72b4a
fadvise.c 4.35 KB