• Johannes Weiner's avatar
    mm: page_alloc: spill to remote nodes before waking kswapd · 3a025760
    Johannes Weiner authored
    On NUMA systems, a node may start thrashing cache or even swap anonymous
    pages while there are still free pages on remote nodes.
    
    This is a result of commits 81c0a2bb ("mm: page_alloc: fair zone
    allocator policy") and fff4068c ("mm: page_alloc: revert NUMA aspect
    of fair allocation policy").
    
    Before those changes, the allocator would first try all allowed zones,
    including those on remote nodes, before waking any kswapds.  But now,
    the allocator fastpath doubles as the fairness pass, which in turn can
    only consider the local node to prevent remote spilling based on
    exhausted fairness batches alone.  Remote nodes are only considered in
    the slowpath, after the kswapds are woken up.  But if remote nodes still
    have free memory, kswapd should not be woken to rebalance the local node
    or it may thrash cash or swap prematurely.
    
    Fix this by adding one more unfair pass over the zonelist that is
    allowed to spill to remote nodes after the local fairness pass fails but
    before entering the slowpath and waking the kswapds.
    
    This also gets rid of the GFP_THISNODE exemption from the fairness
    protocol because the unfair pass is no longer tied to kswapd, which
    GFP_THISNODE is not allowed to wake up.
    
    However, because remote spills can be more frequent now - we prefer them
    over local kswapd reclaim - the allocation batches on remote nodes could
    underflow more heavily.  When resetting the batches, use
    atomic_long_read() directly instead of zone_page_state() to calculate the
    delta as the latter filters negative counter values.
    Signed-off-by: default avatarJohannes Weiner <hannes@cmpxchg.org>
    Acked-by: default avatarRik van Riel <riel@redhat.com>
    Acked-by: default avatarMel Gorman <mgorman@suse.de>
    Cc: <stable@kernel.org>		[3.12+]
    Signed-off-by: default avatarAndrew Morton <akpm@linux-foundation.org>
    Signed-off-by: default avatarLinus Torvalds <torvalds@linux-foundation.org>
    3a025760
internal.h 11.4 KB