• Cyril Hrubis's avatar
    mm/mmap: check for RLIMIT_AS before unmapping · e8420a8e
    Cyril Hrubis authored
    Fix a corner case for MAP_FIXED when requested mapping length is larger
    than rlimit for virtual memory.  In such case any overlapping mappings
    are unmapped before we check for the limit and return ENOMEM.
    
    The check is moved before the loop that unmaps overlapping parts of
    existing mappings.  When we are about to hit the limit (currently mapped
    pages + len > limit) we scan for overlapping pages and check again
    accounting for them.
    
    This fixes situation when userspace program expects that the previous
    mappings are preserved after the mmap() syscall has returned with error.
    (POSIX clearly states that successfull mapping shall replace any
    previous mappings.)
    
    This corner case was found and can be tested with LTP testcase:
    
    testcases/open_posix_testsuite/conformance/interfaces/mmap/24-2.c
    
    In this case the mmap, which is clearly over current limit, unmaps
    dynamic libraries and the testcase segfaults right after returning into
    userspace.
    
    I've also looked at the second instance of the unmapping loop in the
    do_brk().  The do_brk() is called from brk() syscall and from vm_brk().
    The brk() syscall checks for overlapping mappings and bails out when
    there are any (so it can't be triggered from the brk syscall).  The
    vm_brk() is called only from binmft handlers so it shouldn't be
    triggered unless binmft handler created overlapping mappings.
    Signed-off-by: default avatarCyril Hrubis <chrubis@suse.cz>
    Reviewed-by: default avatarMel Gorman <mgorman@suse.de>
    Reviewed-by: default avatarWanpeng Li <liwanp@linux.vnet.ibm.com>
    Signed-off-by: default avatarAndrew Morton <akpm@linux-foundation.org>
    Signed-off-by: default avatarLinus Torvalds <torvalds@linux-foundation.org>
    e8420a8e
mmap.c 86.3 KB