Commit 69b7189f authored by Vlastimil Babka's avatar Vlastimil Babka Committed by Linus Torvalds

mm, compaction: skip rechecks when lock was already held

Compaction scanners try to lock zone locks as late as possible by checking
many page or pageblock properties opportunistically without lock and
skipping them if not unsuitable.  For pages that pass the initial checks,
some properties have to be checked again safely under lock.  However, if
the lock was already held from a previous iteration in the initial checks,
the rechecks are unnecessary.

This patch therefore skips the rechecks when the lock was already held.
This is now possible to do, since we don't (potentially) drop and
reacquire the lock between the initial checks and the safe rechecks
anymore.
Signed-off-by: default avatarVlastimil Babka <vbabka@suse.cz>
Reviewed-by: default avatarZhang Yanfei <zhangyanfei@cn.fujitsu.com>
Reviewed-by: default avatarNaoya Horiguchi <n-horiguchi@ah.jp.nec.com>
Acked-by: default avatarMinchan Kim <minchan@kernel.org>
Acked-by: default avatarMel Gorman <mgorman@suse.de>
Cc: Michal Nazarewicz <mina86@mina86.com>
Cc: Naoya Horiguchi <n-horiguchi@ah.jp.nec.com>
Cc: Christoph Lameter <cl@linux.com>
Cc: Rik van Riel <riel@redhat.com>
Acked-by: default avatarDavid Rientjes <rientjes@google.com>
Signed-off-by: default avatarAndrew Morton <akpm@linux-foundation.org>
Signed-off-by: default avatarLinus Torvalds <torvalds@linux-foundation.org>
parent 8b44d279
...@@ -367,22 +367,30 @@ static unsigned long isolate_freepages_block(struct compact_control *cc, ...@@ -367,22 +367,30 @@ static unsigned long isolate_freepages_block(struct compact_control *cc,
goto isolate_fail; goto isolate_fail;
/* /*
* The zone lock must be held to isolate freepages. * If we already hold the lock, we can skip some rechecking.
* Unfortunately this is a very coarse lock and can be * Note that if we hold the lock now, checked_pageblock was
* heavily contended if there are parallel allocations * already set in some previous iteration (or strict is true),
* or parallel compactions. For async compaction do not * so it is correct to skip the suitable migration target
* spin on the lock and we acquire the lock as late as * recheck as well.
* possible.
*/ */
if (!locked) if (!locked) {
/*
* The zone lock must be held to isolate freepages.
* Unfortunately this is a very coarse lock and can be
* heavily contended if there are parallel allocations
* or parallel compactions. For async compaction do not
* spin on the lock and we acquire the lock as late as
* possible.
*/
locked = compact_trylock_irqsave(&cc->zone->lock, locked = compact_trylock_irqsave(&cc->zone->lock,
&flags, cc); &flags, cc);
if (!locked) if (!locked)
break; break;
/* Recheck this is a buddy page under lock */ /* Recheck this is a buddy page under lock */
if (!PageBuddy(page)) if (!PageBuddy(page))
goto isolate_fail; goto isolate_fail;
}
/* Found a free page, break it into order-0 pages */ /* Found a free page, break it into order-0 pages */
isolated = split_free_page(page); isolated = split_free_page(page);
...@@ -644,19 +652,20 @@ isolate_migratepages_block(struct compact_control *cc, unsigned long low_pfn, ...@@ -644,19 +652,20 @@ isolate_migratepages_block(struct compact_control *cc, unsigned long low_pfn,
page_count(page) > page_mapcount(page)) page_count(page) > page_mapcount(page))
continue; continue;
/* If the lock is not held, try to take it */ /* If we already hold the lock, we can skip some rechecking */
if (!locked) if (!locked) {
locked = compact_trylock_irqsave(&zone->lru_lock, locked = compact_trylock_irqsave(&zone->lru_lock,
&flags, cc); &flags, cc);
if (!locked) if (!locked)
break; break;
/* Recheck PageLRU and PageTransHuge under lock */ /* Recheck PageLRU and PageTransHuge under lock */
if (!PageLRU(page)) if (!PageLRU(page))
continue; continue;
if (PageTransHuge(page)) { if (PageTransHuge(page)) {
low_pfn += (1 << compound_order(page)) - 1; low_pfn += (1 << compound_order(page)) - 1;
continue; continue;
}
} }
lruvec = mem_cgroup_page_lruvec(page, zone); lruvec = mem_cgroup_page_lruvec(page, zone);
......
Markdown is supported
0%
or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment