- 09 Jun, 2023 40 commits
-
-
David Hildenbrand authored
Similar to the COW selftests, also use io_uring fixed buffers to test if long-term page pinning works as expected. Link: https://lkml.kernel.org/r/20230519102723.185721-4-david@redhat.comSigned-off-by:
David Hildenbrand <david@redhat.com> Reviewed-by:
Lorenzo Stoakes <lstoakes@gmail.com> Cc: Jan Kara <jack@suse.cz> Cc: Jason Gunthorpe <jgg@nvidia.com> Cc: Jens Axboe <axboe@kernel.dk> Cc: John Hubbard <jhubbard@nvidia.com> Cc: Peter Xu <peterx@redhat.com> Cc: Shuah Khan <shuah@kernel.org> Signed-off-by:
Andrew Morton <akpm@linux-foundation.org>
-
David Hildenbrand authored
Let's add a new test for checking whether GUP long-term page pinning works as expected (R/O vs. R/W, MAP_PRIVATE vs. MAP_SHARED, GUP vs. GUP-fast). Note that COW handling with long-term R/O pinning in private mappings, and pinning of anonymous memory in general, is tested by the COW selftest. This test, therefore, focuses on page pinning in file mappings. The most interesting case is probably the "local tmpfile" case, as that will likely end up on a "real" filesystem such as ext4 or xfs, not on a virtual one like tmpfs or hugetlb where any long-term page pinning is always expected to succeed. For now, only add tests that use the "/sys/kernel/debug/gup_test" interface. We'll add tests based on liburing separately next. [akpm@linux-foundation.org: update .gitignore for gup_longterm, per Peter] Link: https://lkml.kernel.org/r/20230519102723.185721-3-david@redhat.comSigned-off-by:
-
David Hildenbrand authored
Patch series "selftests/mm: new test for FOLL_LONGTERM on file mappings". Let's add some selftests to make sure that: * R/O long-term pinning always works of file mappings * R/W long-term pinning always works in MAP_PRIVATE file mappings * R/W long-term pinning only works in MAP_SHARED mappings with special filesystems (shmem, hugetlb) and fails with other filesystems (ext4, btrfs, xfs). The tests make use of the gup_test kernel module to trigger ordinary GUP and GUP-fast, and liburing (similar to our COW selftests). Test with memfd, memfd hugetlb, tmpfile() and mkstemp(). The latter usually gives us a "real" filesystem (ext4, btrfs, xfs) where long-term pinning is expected to fail. Note that these selftests don't contain any actual reproducers for data corruptions in case R/W long-term pinning on problematic filesystems "would" work. Maybe we can later come up with a racy !FOLL_LONGTERM reproducer that can reuse an existing interface to trigger short-term pinning (I'll look into that next). On current mm/mm-unstable: # ./gup_longterm # [INFO] detected hugetlb page size: 2048 KiB # [INFO] detected hugetlb page size: 1048576 KiB TAP version 13 1..50 # [RUN] R/W longterm GUP pin in MAP_SHARED file mapping ... with memfd ok 1 Should have worked # [RUN] R/W longterm GUP pin in MAP_SHARED file mapping ... with tmpfile ok 2 Should have worked # [RUN] R/W longterm GUP pin in MAP_SHARED file mapping ... with local tmpfile ok 3 Should have failed # [RUN] R/W longterm GUP pin in MAP_SHARED file mapping ... with memfd hugetlb (2048 kB) ok 4 Should have worked # [RUN] R/W longterm GUP pin in MAP_SHARED file mapping ... with memfd hugetlb (1048576 kB) ok 5 Should have worked # [RUN] R/W longterm GUP-fast pin in MAP_SHARED file mapping ... with memfd ok 6 Should have worked # [RUN] R/W longterm GUP-fast pin in MAP_SHARED file mapping ... with tmpfile ok 7 Should have worked # [RUN] R/W longterm GUP-fast pin in MAP_SHARED file mapping ... with local tmpfile ok 8 Should have failed # [RUN] R/W longterm GUP-fast pin in MAP_SHARED file mapping ... with memfd hugetlb (2048 kB) ok 9 Should have worked # [RUN] R/W longterm GUP-fast pin in MAP_SHARED file mapping ... with memfd hugetlb (1048576 kB) ok 10 Should have worked # [RUN] R/O longterm GUP pin in MAP_SHARED file mapping ... with memfd ok 11 Should have worked # [RUN] R/O longterm GUP pin in MAP_SHARED file mapping ... with tmpfile ok 12 Should have worked # [RUN] R/O longterm GUP pin in MAP_SHARED file mapping ... with local tmpfile ok 13 Should have worked # [RUN] R/O longterm GUP pin in MAP_SHARED file mapping ... with memfd hugetlb (2048 kB) ok 14 Should have worked # [RUN] R/O longterm GUP pin in MAP_SHARED file mapping ... with memfd hugetlb (1048576 kB) ok 15 Should have worked # [RUN] R/O longterm GUP-fast pin in MAP_SHARED file mapping ... with memfd ok 16 Should have worked # [RUN] R/O longterm GUP-fast pin in MAP_SHARED file mapping ... with tmpfile ok 17 Should have worked # [RUN] R/O longterm GUP-fast pin in MAP_SHARED file mapping ... with local tmpfile ok 18 Should have worked # [RUN] R/O longterm GUP-fast pin in MAP_SHARED file mapping ... with memfd hugetlb (2048 kB) ok 19 Should have worked # [RUN] R/O longterm GUP-fast pin in MAP_SHARED file mapping ... with memfd hugetlb (1048576 kB) ok 20 Should have worked # [RUN] R/W longterm GUP pin in MAP_PRIVATE file mapping ... with memfd ok 21 Should have worked # [RUN] R/W longterm GUP pin in MAP_PRIVATE file mapping ... with tmpfile ok 22 Should have worked # [RUN] R/W longterm GUP pin in MAP_PRIVATE file mapping ... with local tmpfile ok 23 Should have worked # [RUN] R/W longterm GUP pin in MAP_PRIVATE file mapping ... with memfd hugetlb (2048 kB) ok 24 Should have worked # [RUN] R/W longterm GUP pin in MAP_PRIVATE file mapping ... with memfd hugetlb (1048576 kB) ok 25 Should have worked # [RUN] R/W longterm GUP-fast pin in MAP_PRIVATE file mapping ... with memfd ok 26 Should have worked # [RUN] R/W longterm GUP-fast pin in MAP_PRIVATE file mapping ... with tmpfile ok 27 Should have worked # [RUN] R/W longterm GUP-fast pin in MAP_PRIVATE file mapping ... with local tmpfile ok 28 Should have worked # [RUN] R/W longterm GUP-fast pin in MAP_PRIVATE file mapping ... with memfd hugetlb (2048 kB) ok 29 Should have worked # [RUN] R/W longterm GUP-fast pin in MAP_PRIVATE file mapping ... with memfd hugetlb (1048576 kB) ok 30 Should have worked # [RUN] R/O longterm GUP pin in MAP_PRIVATE file mapping ... with memfd ok 31 Should have worked # [RUN] R/O longterm GUP pin in MAP_PRIVATE file mapping ... with tmpfile ok 32 Should have worked # [RUN] R/O longterm GUP pin in MAP_PRIVATE file mapping ... with local tmpfile ok 33 Should have worked # [RUN] R/O longterm GUP pin in MAP_PRIVATE file mapping ... with memfd hugetlb (2048 kB) ok 34 Should have worked # [RUN] R/O longterm GUP pin in MAP_PRIVATE file mapping ... with memfd hugetlb (1048576 kB) ok 35 Should have worked # [RUN] R/O longterm GUP-fast pin in MAP_PRIVATE file mapping ... with memfd ok 36 Should have worked # [RUN] R/O longterm GUP-fast pin in MAP_PRIVATE file mapping ... with tmpfile ok 37 Should have worked # [RUN] R/O longterm GUP-fast pin in MAP_PRIVATE file mapping ... with local tmpfile ok 38 Should have worked # [RUN] R/O longterm GUP-fast pin in MAP_PRIVATE file mapping ... with memfd hugetlb (2048 kB) ok 39 Should have worked # [RUN] R/O longterm GUP-fast pin in MAP_PRIVATE file mapping ... with memfd hugetlb (1048576 kB) ok 40 Should have worked # [RUN] io_uring fixed buffer with MAP_SHARED file mapping ... with memfd ok 41 Should have worked # [RUN] io_uring fixed buffer with MAP_SHARED file mapping ... with tmpfile ok 42 Should have worked # [RUN] io_uring fixed buffer with MAP_SHARED file mapping ... with local tmpfile ok 43 Should have failed # [RUN] io_uring fixed buffer with MAP_SHARED file mapping ... with memfd hugetlb (2048 kB) ok 44 Should have worked # [RUN] io_uring fixed buffer with MAP_SHARED file mapping ... with memfd hugetlb (1048576 kB) ok 45 Should have worked # [RUN] io_uring fixed buffer with MAP_PRIVATE file mapping ... with memfd ok 46 Should have worked # [RUN] io_uring fixed buffer with MAP_PRIVATE file mapping ... with tmpfile ok 47 Should have worked # [RUN] io_uring fixed buffer with MAP_PRIVATE file mapping ... with local tmpfile ok 48 Should have worked # [RUN] io_uring fixed buffer with MAP_PRIVATE file mapping ... with memfd hugetlb (2048 kB) ok 49 Should have worked # [RUN] io_uring fixed buffer with MAP_PRIVATE file mapping ... with memfd hugetlb (1048576 kB) ok 50 Should have worked # Totals: pass:50 fail:0 xfail:0 xpass:0 skip:0 error:0 This patch (of 3): Let's factor detection out into vm_util, to be reused by a new test. Link: https://lkml.kernel.org/r/20230519102723.185721-1-david@redhat.com Link: https://lkml.kernel.org/r/20230519102723.185721-2-david@redhat.comSigned-off-by:
-
Johannes Weiner authored
During stress testing with higher-order allocations, a deadlock scenario was observed in compaction: One GFP_NOFS allocation was sleeping on mm/compaction.c::too_many_isolated(), while all CPUs in the system were busy with compactors spinning on buffer locks held by the sleeping GFP_NOFS allocation. Reclaim is susceptible to this same deadlock; we fixed it by granting GFP_NOFS allocations additional LRU isolation headroom, to ensure it makes forward progress while holding fs locks that other reclaimers might acquire. Do the same here. This code has been like this since compaction was initially merged, and I only managed to trigger this with out-of-tree patches that dramatically increase the contexts that do GFP_NOFS compaction. While the issue is real, it seems theoretical in nature given existing allocation sites. Worth fixing now, but no Fixes tag or stable CC. Link: https://lkml.kernel.org/r/20230519111359.40475-1-hannes@cmpxchg.orgSigned-off-by:
Johannes Weiner <hannes@cmpxchg.org> Acked-by:
Mel Gorman <mgorman@techsingularity.net> Acked-by:
Vlastimil Babka <vbabka@suse.cz> Cc: Michal Hocko <mhocko@suse.com> Signed-off-by:
-
Johannes Weiner authored
Since it only returns COMPACT_CONTINUE or COMPACT_SKIPPED now, a bool return value simplifies the callsites. Link: https://lkml.kernel.org/r/20230602151204.GD161817@cmpxchg.orgSigned-off-by:
-
Johannes Weiner authored
The watermark check in compaction_zonelist_suitable(), called from should_compact_retry(), is sandwiched between two watermark checks already: before, there are freelist attempts as part of direct reclaim and direct compaction; after, there is a last-minute freelist attempt in __alloc_pages_may_oom(). The check in compaction_zonelist_suitable() isn't necessary. Kill it. Link: https://lkml.kernel.org/r/20230519123959.77335-6-hannes@cmpxchg.orgSigned-off-by:
-
Johannes Weiner authored
Remove from all paths not reachable via /proc/sys/vm/compact_memory. Link: https://lkml.kernel.org/r/20230519123959.77335-5-hannes@cmpxchg.orgSigned-off-by:
-
Johannes Weiner authored
__compaction_suitable() is supposed to check for available migration targets. However, it also checks whether the operation was requested via /proc/sys/vm/compact_memory, and whether the original allocation request can already succeed. These don't apply to all callsites. Move the checks out to the callers, so that later patches can deal with them one by one. No functional change intended. [hannes@cmpxchg.org: fix comment, per Vlastimil] Link: https://lkml.kernel.org/r/20230602144942.GC161817@cmpxchg.org Link: https://lkml.kernel.org/r/20230519123959.77335-4-hannes@cmpxchg.orgSigned-off-by:
-
Johannes Weiner authored
The different branches for retry are unnecessarily complicated. There are really only three outcomes: progress (retry n times), skipped (retry if reclaim can help), failed (retry with higher priority). Rearrange the branches and the retry counter to make it simpler. [hannes@cmpxchg.org: restore behavior when hitting max_retries] Link: https://lkml.kernel.org/r/20230602144705.GB161817@cmpxchg.org Link: https://lkml.kernel.org/r/20230519123959.77335-3-hannes@cmpxchg.orgSigned-off-by:
-
Johannes Weiner authored
Patch series "mm: compaction: cleanups & simplifications". These compaction cleanups are split out from the huge page allocator series[1], as requested by reviewer feedback. [1] https://lore.kernel.org/linux-mm/20230418191313.268131-1-hannes@cmpxchg.org/ This patch (of 5): The compaction result helpers encode quirks that are specific to the allocator's retry logic. E.g. COMPACT_SUCCESS and COMPACT_COMPLETE actually represent failures that should be retried upon, and so on. I frequently found myself pulling up the helper implementation in order to understand and work on the retry logic. They're not quite clean abstractions; rather they split the retry logic into two locations. Remove the helpers and inline the checks. Then comment on the result interpretations directly where the decision making happens. Link: https://lkml.kernel.org/r/20230519123959.77335-1-hannes@cmpxchg.org Link: https://lkml.kernel.org/r/20230519123959.77335-2-hannes@cmpxchg.orgSigned-off-by:
-
Tom Rix authored
smatch reports mm/page_alloc.c:247:5: warning: symbol 'sysctl_lowmem_reserve_ratio' was not declared. Should it be static? This variable is only used in its defining file, so it should be static Link: https://lkml.kernel.org/r/20230518141119.927074-1-trix@redhat.comSigned-off-by:
Tom Rix <trix@redhat.com> Signed-off-by:
-
Liam R. Howlett authored
If the iterator has moved to the previous entry, then step forward one range, back to the gap. Link: https://lkml.kernel.org/r/20230518145544.1722059-36-Liam.Howlett@oracle.comSigned-off-by:
Liam R. Howlett <Liam.Howlett@oracle.com> Cc: David Binderman <dcb314@hotmail.com> Cc: Peng Zhang <zhangpeng.00@bytedance.com> Cc: Sergey Senozhatsky <senozhatsky@chromium.org> Cc: Vernon Yang <vernon2gm@gmail.com> Cc: Wei Yang <richard.weiyang@gmail.com> Signed-off-by:
-
Liam R. Howlett authored
Add functionality to the VMA iterator to advance and retreat one offset within the maple tree, regardless of the value contained. This can lead to less re-walking to find an area of interest, especially when there is nothing in that offset. Link: https://lkml.kernel.org/r/20230518145544.1722059-35-Liam.Howlett@oracle.comSigned-off-by:
-
Liam R. Howlett authored
Now that the functions have changed the limits, update the testing of the maple tree to test these new settings. Link: https://lkml.kernel.org/r/20230518145544.1722059-34-Liam.Howlett@oracle.comSigned-off-by:
-
Liam R. Howlett authored
When there is a single entry tree (range of 0-0 pointing to an entry), then ensure the limit is either 0-0 or 1-oo, depending on where the user walks. Ensure the correct node setting as well; either MAS_ROOT or MAS_NONE. Link: https://lkml.kernel.org/r/20230518145544.1722059-33-Liam.Howlett@oracle.comSigned-off-by:
-
Liam R. Howlett authored
Some users of the maple tree may want to move to the previous range regardless of the value stored there. Add this interface as well as the 'find' variant to support walking to the first value, then iterating over the previous ranges. Link: https://lkml.kernel.org/r/20230518145544.1722059-32-Liam.Howlett@oracle.comSigned-off-by:
-
Liam R. Howlett authored
Sometimes the user needs to revert to the previous slot, regardless of if it is empty or not. Add an interface to go to the previous slot. Since there can't be two consecutive NULLs in the tree, the mas_prev() function can be implemented by calling mas_prev_slot() a maximum of 2 times. Change the underlying interface to use mas_prev_slot() to align the code. Link: https://lkml.kernel.org/r/20230518145544.1722059-31-Liam.Howlett@oracle.comSigned-off-by:
-
Liam R. Howlett authored
These functions need to move for future use. Link: https://lkml.kernel.org/r/20230518145544.1722059-30-Liam.Howlett@oracle.comSigned-off-by:
-
Liam R. Howlett authored
Some users of the maple tree may want to move to the next range in the tree, even if it stores a NULL. This family of function provides that functionality by advancing one slot at a time and returning the result, while mas_contiguous() will iterate over the range and stop on encountering the first NULL. Link: https://lkml.kernel.org/r/20230518145544.1722059-29-Liam.Howlett@oracle.comSigned-off-by:
-
Liam R. Howlett authored
Sometimes, during a tree walk, the user needs the next slot regardless of if it is empty or not. Add an interface to get the next slot. Since there are no consecutive NULLs allowed in the tree, the mas_next() function can only advance two slots at most. So use the new mas_next_slot() interface to align both implementations. Use this method for mas_find() as well. Link: https://lkml.kernel.org/r/20230518145544.1722059-28-Liam.Howlett@oracle.comSigned-off-by:
-
Liam R. Howlett authored
Empty area will return -EINVAL if the search window is smaller than the requested size. Fix the test case to check for this error code. Link: https://lkml.kernel.org/r/20230518145544.1722059-27-Liam.Howlett@oracle.comSigned-off-by:
-
Liam R. Howlett authored
Since the maple tree is inclusive in range, ensure that a range of 1 (min = max) works for searching for a gap in either direction, and make sure the size is at least 1 but not larger than the delta between min and max. This commit also updates the testing. Unfortunately there isn't a way to safely update the tests and code without a test failure. Link: https://lkml.kernel.org/r/20230518145544.1722059-26-Liam.Howlett@oracle.comSigned-off-by:
Peng Zhang <zhangpeng.00@bytedance.com> Cc: David Binderman <dcb314@hotmail.com> Cc: Sergey Senozhatsky <senozhatsky@chromium.org> Cc: Vernon Yang <vernon2gm@gmail.com> Cc: Wei Yang <richard.weiyang@gmail.com> Signed-off-by:
-
Liam R. Howlett authored
Keep a reference to the node when possible with mas_prev(). This will avoid re-walking the tree. In keeping a reference to the node, keep the last/index accurate to the range being referenced. This means the limit may be within the range, but the range may extend outside of the limit. Also fix the single entry tree to respect the range (of 0), or set the node to MAS_NONE in the case of shifting beyond 0. Link: https://lkml.kernel.org/r/20230518145544.1722059-25-Liam.Howlett@oracle.comSigned-off-by:
-
Liam R. Howlett authored
Clean up the mas_next() call to try and keep a node reference when possible. This will avoid re-walking the tree in most cases. Also clean up the single entry tree handling to ensure index/last are consistent with what one would expect. (returning NULL with limit of 1-oo). Link: https://lkml.kernel.org/r/20230518145544.1722059-24-Liam.Howlett@oracle.comSigned-off-by:
-
Liam R. Howlett authored
The maple tree iterator clean up is incompatible with the way do_vmi_align_munmap() expects it to behave. Update the expected behaviour to map now since the change will work currently. Link: https://lkml.kernel.org/r/20230518145544.1722059-23-Liam.Howlett@oracle.comSigned-off-by:
-
Liam R. Howlett authored
When a dead node is detected, the depth has already been set to 1 so reset it to 0. Link: https://lkml.kernel.org/r/20230518145544.1722059-22-Liam.Howlett@oracle.comSigned-off-by:
-
Liam R. Howlett authored
mas_destroy currently checks if mas->node is MAS_START prior to calling mas_start(), but this is unnecessary as mas_start() will do nothing if the node is anything but MAS_START. Link: https://lkml.kernel.org/r/20230518145544.1722059-21-Liam.Howlett@oracle.comSigned-off-by:
-
Liam R. Howlett authored
The test functions are not needed after the module is removed, so mark them as such. Add __exit to the module removal function. Some other variables have been marked as const static as well. Link: https://lkml.kernel.org/r/20230518145544.1722059-20-Liam.Howlett@oracle.comSigned-off-by:
-
Liam R. Howlett authored
MAS_WARN_ON() will provide more information on the maple state and can be more useful for debugging. Use this version of WARN_ON() in the debugging code when storing to the tree. Update the printk to a pr_warn(), but this will only be printed when maple tree debug is enabled anyways. Making all print statements into one will keep them together on a busy terminal. Link: https://lkml.kernel.org/r/20230518145544.1722059-19-Liam.Howlett@oracle.comSigned-off-by:
Sergey Senozhatsky <senozhatsky@chromium.org> Cc: David Binderman <dcb314@hotmail.com> Cc: Peng Zhang <zhangpeng.00@bytedance.com> Cc: Vernon Yang <vernon2gm@gmail.com> Cc: Wei Yang <richard.weiyang@gmail.com> Signed-off-by:
-
Liam R. Howlett authored
Use the vma iterator in the validation code and combine the code to check the maple tree into the main validate_mm() function. Introduce a new function vma_iter_dump_tree() to dump the maple tree in hex layout. Replace all calls to validate_mm_mt() with validate_mm(). [Liam.Howlett@oracle.com: update validate_mm() to use vma iterator CONFIG flag] Link: https://lkml.kernel.org/r/20230606183538.588190-1-Liam.Howlett@oracle.com Link: https://lkml.kernel.org/r/20230518145544.1722059-18-Liam.Howlett@oracle.comSigned-off-by:
-
Liam R. Howlett authored
The test code is less useful without debug, but can still do general validations. Define mt_dump(), mas_dump() and mas_wr_dump() as a noop if debug is not enabled and document it in the test module information that more information can be obtained with another kernel config option. MT_BUG_ON() will report a failures without tree dumps, and the output will be less useful. Link: https://lkml.kernel.org/r/20230518145544.1722059-17-Liam.Howlett@oracle.comSigned-off-by:
-
Liam R. Howlett authored
Rename mte_pivots() to mas_pivots() and pass through the ma_state to set the error code to -EIO when the offset is out of range for the node type. Change the WARN_ON() to MAS_WARN_ON() to log the maple state. Link: https://lkml.kernel.org/r/20230518145544.1722059-16-Liam.Howlett@oracle.comSigned-off-by:
-
Liam R. Howlett authored
Replace the call to BUG_ON() in mas_meta_gap() with calls before the function call MAS_BUG_ON() to get more information on error condition. Link: https://lkml.kernel.org/r/20230518145544.1722059-15-Liam.Howlett@oracle.comSigned-off-by:
-
Liam R. Howlett authored
mas_store_prealloc() should never fail, but if it does due to internal tree issues then get as much debug information as possible prior to crashing the kernel. Link: https://lkml.kernel.org/r/20230518145544.1722059-14-Liam.Howlett@oracle.comSigned-off-by:
-
Liam R. Howlett authored
In the even of trying to remove data from a leaf node by use of mas_topiary_range(), log the maple state. Link: https://lkml.kernel.org/r/20230518145544.1722059-13-Liam.Howlett@oracle.comSigned-off-by:
-
Liam R. Howlett authored
Use MAS_BUG_ON() instead of MT_BUG_ON() to get the maple state information. In the unlikely event of a tree height of > 31, try to increase the probability of useful information being logged. Link: https://lkml.kernel.org/r/20230518145544.1722059-12-Liam.Howlett@oracle.comSigned-off-by:
-
Liam R. Howlett authored
Use MAS_BUG_ON() to dump the maple state and tree in the unlikely event of an issue. Link: https://lkml.kernel.org/r/20230518145544.1722059-11-Liam.Howlett@oracle.comSigned-off-by:
-
Liam R. Howlett authored
Using MT_WARN_ON() allows for the removal of if statements before logging. Using MAS_WARN_ON() will provide more information when issues are encountered. Link: https://lkml.kernel.org/r/20230518145544.1722059-10-Liam.Howlett@oracle.comSigned-off-by:
-
Liam R. Howlett authored
If RCU is enabled and the tree isn't locked, just warn the user and avoid crashing the kernel. Link: https://lkml.kernel.org/r/20230518145544.1722059-9-Liam.Howlett@oracle.comSigned-off-by:
-
Liam R. Howlett authored
Use MT_BUG_ON() to get more information when running with MAPLE_TREE_DEBUG enabled. Link: https://lkml.kernel.org/r/20230518145544.1722059-8-Liam.Howlett@oracle.comSigned-off-by:
-
