• Mel Gorman's avatar
    hugetlbfs: kill applications that use MAP_NORESERVE with SIGBUS instead of OOM-killer · 4a6018f7
    Mel Gorman authored
    Ordinarily, application using hugetlbfs will create mappings with
    reserves.  For shared mappings, these pages are reserved before mmap()
    returns success and for private mappings, the caller process is guaranteed
    and a child process that cannot get the pages gets killed with sigbus.
    
    An application that uses MAP_NORESERVE gets no reservations and mmap()
    will always succeed at the risk the page will not be available at fault
    time.  This might be used for example on very large sparse mappings where
    the developer is confident the necessary huge pages exist to satisfy all
    faults even though the whole mapping cannot be backed by huge pages.
    Unfortunately, if an allocation does fail, VM_FAULT_OOM is returned to the
    fault handler which proceeds to trigger the OOM-killer.  This is
    unhelpful.
    
    Even without hugetlbfs mounted, a user using mmap() can trivially trigger
    the OOM-killer because VM_FAULT_OOM is returned (will provide example
    program if desired - it's a whopping 24 lines long).  It could be
    considered a DOS available to an unprivileged user.
    
    This patch alters hugetlbfs to kill a process that uses MAP_NORESERVE
    where huge pages were not available with SIGBUS instead of triggering the
    OOM killer.
    
    This change affects hugetlb_cow() as well.  I feel there is a failure case
    in there, but I didn't create one.  It would need a fairly specific target
    in terms of the faulting application and the hugepage pool size.  The
    hugetlb_no_page() path is much easier to hit but both might as well be
    closed.
    Signed-off-by: default avatarMel Gorman <mel@csn.ul.ie>
    Cc: Lee Schermerhorn <lee.schermerhorn@hp.com>
    Cc: David Rientjes <rientjes@google.com>
    Cc: Andi Kleen <andi@firstfloor.org>
    Cc: <stable@kernel.org>
    Signed-off-by: default avatarAndrew Morton <akpm@linux-foundation.org>
    Signed-off-by: default avatarLinus Torvalds <torvalds@linux-foundation.org>
    4a6018f7
hugetlb.c 72.1 KB