• Ryan Roberts's avatar
    mm: madvise: avoid split during MADV_PAGEOUT and MADV_COLD · 3931b871
    Ryan Roberts authored
    Rework madvise_cold_or_pageout_pte_range() to avoid splitting any large
    folio that is fully and contiguously mapped in the pageout/cold vm range. 
    This change means that large folios will be maintained all the way to swap
    storage.  This both improves performance during swap-out, by eliding the
    cost of splitting the folio, and sets us up nicely for maintaining the
    large folio when it is swapped back in (to be covered in a separate
    series).
    
    Folios that are not fully mapped in the target range are still split, but
    note that behavior is changed so that if the split fails for any reason
    (folio locked, shared, etc) we now leave it as is and move to the next pte
    in the range and continue work on the proceeding folios.  Previously any
    failure of this sort would cause the entire operation to give up and no
    folios mapped at higher addresses were paged out or made cold.  Given
    large folios are becoming more common, this old behavior would have likely
    lead to wasted opportunities.
    
    While we are at it, change the code that clears young from the ptes to use
    ptep_test_and_clear_young(), via the new mkold_ptes() batch helper
    function.  This is more efficent than get_and_clear/modify/set, especially
    for contpte mappings on arm64, where the old approach would require
    unfolding/refolding and the new approach can be done in place.
    
    Link: https://lkml.kernel.org/r/20240408183946.2991168-8-ryan.roberts@arm.comSigned-off-by: default avatarRyan Roberts <ryan.roberts@arm.com>
    Reviewed-by: default avatarBarry Song <v-songbaohua@oppo.com>
    Acked-by: default avatarDavid Hildenbrand <david@redhat.com>
    Cc: Barry Song <21cnbao@gmail.com>
    Cc: Chris Li <chrisl@kernel.org>
    Cc: Gao Xiang <xiang@kernel.org>
    Cc: "Huang, Ying" <ying.huang@intel.com>
    Cc: Kefeng Wang <wangkefeng.wang@huawei.com>
    Cc: Lance Yang <ioworker0@gmail.com>
    Cc: Matthew Wilcox (Oracle) <willy@infradead.org>
    Cc: Michal Hocko <mhocko@suse.com>
    Cc: Yang Shi <shy828301@gmail.com>
    Cc: Yu Zhao <yuzhao@google.com>
    Signed-off-by: default avatarAndrew Morton <akpm@linux-foundation.org>
    3931b871
madvise.c 39 KB