• Zi Yan's avatar
    mm/compaction: add support for >0 order folio memory compaction. · 733aea0b
    Zi Yan authored
    Before last commit, memory compaction only migrates order-0 folios and
    skips >0 order folios.  Last commit splits all >0 order folios during
    compaction.  This commit migrates >0 order folios during compaction by
    keeping isolated free pages at their original size without splitting them
    into order-0 pages and using them directly during migration process.
    
    What is different from the prior implementation:
    1. All isolated free pages are kept in a NR_PAGE_ORDERS array of page
       lists, where each page list stores free pages in the same order.
    2. All free pages are not post_alloc_hook() processed nor buddy pages,
       although their orders are stored in first page's private like buddy
       pages.
    3. During migration, in new page allocation time (i.e., in
       compaction_alloc()), free pages are then processed by post_alloc_hook().
       When migration fails and a new page is returned (i.e., in
       compaction_free()), free pages are restored by reversing the
       post_alloc_hook() operations using newly added
       free_pages_prepare_fpi_none().
    
    Step 3 is done for a latter optimization that splitting and/or merging
    free pages during compaction becomes easier.
    
    Note: without splitting free pages, compaction can end prematurely due to
    migration will return -ENOMEM even if there is free pages.  This happens
    when no order-0 free page exist and compaction_alloc() return NULL.
    
    Link: https://lkml.kernel.org/r/20240220183220.1451315-4-zi.yan@sent.comSigned-off-by: default avatarZi Yan <ziy@nvidia.com>
    Reviewed-by: default avatarBaolin Wang <baolin.wang@linux.alibaba.com>
    Reviewed-by: default avatarVlastimil Babka <vbabka@suse.cz>
    Tested-by: default avatarBaolin Wang <baolin.wang@linux.alibaba.com>
    Tested-by: default avatarYu Zhao <yuzhao@google.com>
    Cc: Adam Manzanares <a.manzanares@samsung.com>
    Cc: David Hildenbrand <david@redhat.com>
    Cc: Huang Ying <ying.huang@intel.com>
    Cc: Johannes Weiner <hannes@cmpxchg.org>
    Cc: Kemeng Shi <shikemeng@huaweicloud.com>
    Cc: Kirill A. Shutemov <kirill.shutemov@linux.intel.com>
    Cc: Luis Chamberlain <mcgrof@kernel.org>
    Cc: Matthew Wilcox (Oracle) <willy@infradead.org>
    Cc: Mel Gorman <mgorman@techsingularity.net>
    Cc: Ryan Roberts <ryan.roberts@arm.com>
    Cc: Vishal Moola (Oracle) <vishal.moola@gmail.com>
    Cc: Vlastimil Babka <vbabka@suse.cz>
    Cc: Yin Fengwei <fengwei.yin@intel.com>
    Signed-off-by: default avatarAndrew Morton <akpm@linux-foundation.org>
    733aea0b
internal.h 39 KB