1. 29 Apr, 2013 40 commits
    • Cody P Schafer's avatar
    • Shaohua Li's avatar
      mm: thp: add split tail pages to shrink page list in page reclaim · 5bc7b8ac
      Shaohua Li authored
      In page reclaim, huge page is split.  split_huge_page() adds tail pages
      to LRU list.  Since we are reclaiming a huge page, it's better we
      reclaim all subpages of the huge page instead of just the head page.
      This patch adds split tail pages to shrink page list so the tail pages
      can be reclaimed soon.
      
      Before this patch, run a swap workload:
        thp_fault_alloc 3492
        thp_fault_fallback 608
        thp_collapse_alloc 6
        thp_collapse_alloc_failed 0
        thp_split 916
      
      With this patch:
        thp_fault_alloc 4085
        thp_fault_fallback 16
        thp_collapse_alloc 90
        thp_collapse_alloc_failed 0
        thp_split 1272
      
      fallback allocation is reduced a lot.
      
      [akpm@linux-foundation.org: fix CONFIG_SWAP=n build]
      Signed-off-by: default avatarShaohua Li <shli@fusionio.com>
      Acked-by: default avatarRik van Riel <riel@redhat.com>
      Acked-by: default avatarMinchan Kim <minchan@kernel.org>
      Acked-by: default avatarJohannes Weiner <hannes@cmpxchg.org>
      Reviewed-by: default avatarWanpeng Li <liwanp@linux.vnet.ibm.com>
      Cc: Andrea Arcangeli <aarcange@redhat.com>
      Cc: Hugh Dickins <hughd@google.com>
      Signed-off-by: default avatarAndrew Morton <akpm@linux-foundation.org>
      Signed-off-by: default avatarLinus Torvalds <torvalds@linux-foundation.org>
      5bc7b8ac
    • Seth Jennings's avatar
      mm: allow for outstanding swap writeback accounting · 1eec6702
      Seth Jennings authored
      To prevent flooding the swap device with writebacks, frontswap backends
      need to count and limit the number of outstanding writebacks.  The
      incrementing of the counter can be done before the call to
      __swap_writepage().  However, the caller must receive a notification
      when the writeback completes in order to decrement the counter.
      
      To achieve this functionality, this patch modifies __swap_writepage() to
      take the bio completion callback function as an argument.
      
      end_swap_bio_write(), the normal bio completion function, is also made
      non-static so that code doing the accounting can call it after the
      accounting is done.
      
      There should be no behavioural change to existing code.
      Signed-off-by: default avatarSeth Jennings <sjenning@linux.vnet.ibm.com>
      Signed-off-by: default avatarBob Liu <bob.liu@oracle.com>
      Acked-by: default avatarMinchan Kim <minchan@kernel.org>
      Reviewed-by: default avatarDan Magenheimer <dan.magenheimer@oracle.com>
      Cc: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>
      Signed-off-by: default avatarAndrew Morton <akpm@linux-foundation.org>
      Signed-off-by: default avatarLinus Torvalds <torvalds@linux-foundation.org>
      1eec6702
    • Seth Jennings's avatar
      mm: break up swap_writepage() for frontswap backends · 2f772e6c
      Seth Jennings authored
      swap_writepage() is currently where frontswap hooks into the swap write
      path to capture pages with the frontswap_store() function.  However, if
      a frontswap backend wants to "resume" the writeback of a page to the
      swap device, it can't call swap_writepage() as the page will simply
      reenter the backend.
      
      This patch separates swap_writepage() into a top and bottom half, the
      bottom half named __swap_writepage() to allow a frontswap backend, like
      zswap, to resume writeback beyond the frontswap_store() hook.
      
      __add_to_swap_cache() is also made non-static so that the page for which
      writeback is to be resumed can be added to the swap cache.
      Signed-off-by: default avatarSeth Jennings <sjenning@linux.vnet.ibm.com>
      Signed-off-by: default avatarBob Liu <bob.liu@oracle.com>
      Acked-by: default avatarMinchan Kim <minchan@kernel.org>
      Reviewed-by: default avatarDan Magenheimer <dan.magenheimer@oracle.com>
      Cc: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>
      Signed-off-by: default avatarAndrew Morton <akpm@linux-foundation.org>
      Signed-off-by: default avatarLinus Torvalds <torvalds@linux-foundation.org>
      2f772e6c
    • Cyril Hrubis's avatar
      mm/mmap: check for RLIMIT_AS before unmapping · e8420a8e
      Cyril Hrubis authored
      Fix a corner case for MAP_FIXED when requested mapping length is larger
      than rlimit for virtual memory.  In such case any overlapping mappings
      are unmapped before we check for the limit and return ENOMEM.
      
      The check is moved before the loop that unmaps overlapping parts of
      existing mappings.  When we are about to hit the limit (currently mapped
      pages + len > limit) we scan for overlapping pages and check again
      accounting for them.
      
      This fixes situation when userspace program expects that the previous
      mappings are preserved after the mmap() syscall has returned with error.
      (POSIX clearly states that successfull mapping shall replace any
      previous mappings.)
      
      This corner case was found and can be tested with LTP testcase:
      
      testcases/open_posix_testsuite/conformance/interfaces/mmap/24-2.c
      
      In this case the mmap, which is clearly over current limit, unmaps
      dynamic libraries and the testcase segfaults right after returning into
      userspace.
      
      I've also looked at the second instance of the unmapping loop in the
      do_brk().  The do_brk() is called from brk() syscall and from vm_brk().
      The brk() syscall checks for overlapping mappings and bails out when
      there are any (so it can't be triggered from the brk syscall).  The
      vm_brk() is called only from binmft handlers so it shouldn't be
      triggered unless binmft handler created overlapping mappings.
      Signed-off-by: default avatarCyril Hrubis <chrubis@suse.cz>
      Reviewed-by: default avatarMel Gorman <mgorman@suse.de>
      Reviewed-by: default avatarWanpeng Li <liwanp@linux.vnet.ibm.com>
      Signed-off-by: default avatarAndrew Morton <akpm@linux-foundation.org>
      Signed-off-by: default avatarLinus Torvalds <torvalds@linux-foundation.org>
      e8420a8e
    • Anton Vorontsov's avatar
      memcg: add memory.pressure_level events · 70ddf637
      Anton Vorontsov authored
      With this patch userland applications that want to maintain the
      interactivity/memory allocation cost can use the pressure level
      notifications.  The levels are defined like this:
      
      The "low" level means that the system is reclaiming memory for new
      allocations.  Monitoring this reclaiming activity might be useful for
      maintaining cache level.  Upon notification, the program (typically
      "Activity Manager") might analyze vmstat and act in advance (i.e.
      prematurely shutdown unimportant services).
      
      The "medium" level means that the system is experiencing medium memory
      pressure, the system might be making swap, paging out active file
      caches, etc.  Upon this event applications may decide to further analyze
      vmstat/zoneinfo/memcg or internal memory usage statistics and free any
      resources that can be easily reconstructed or re-read from a disk.
      
      The "critical" level means that the system is actively thrashing, it is
      about to out of memory (OOM) or even the in-kernel OOM killer is on its
      way to trigger.  Applications should do whatever they can to help the
      system.  It might be too late to consult with vmstat or any other
      statistics, so it's advisable to take an immediate action.
      
      The events are propagated upward until the event is handled, i.e.  the
      events are not pass-through.  Here is what this means: for example you
      have three cgroups: A->B->C.  Now you set up an event listener on
      cgroups A, B and C, and suppose group C experiences some pressure.  In
      this situation, only group C will receive the notification, i.e.  groups
      A and B will not receive it.  This is done to avoid excessive
      "broadcasting" of messages, which disturbs the system and which is
      especially bad if we are low on memory or thrashing.  So, organize the
      cgroups wisely, or propagate the events manually (or, ask us to
      implement the pass-through events, explaining why would you need them.)
      
      Performance wise, the memory pressure notifications feature itself is
      lightweight and does not require much of bookkeeping, in contrast to the
      rest of memcg features.  Unfortunately, as of current memcg
      implementation, pages accounting is an inseparable part and cannot be
      turned off.  The good news is that there are some efforts[1] to improve
      the situation; plus, implementing the same, fully API-compatible[2]
      interface for CONFIG_MEMCG=n case (e.g.  embedded) is also a viable
      option, so it will not require any changes on the userland side.
      
      [1] http://permalink.gmane.org/gmane.linux.kernel.cgroups/6291
      [2] http://lkml.org/lkml/2013/2/21/454
      
      [akpm@linux-foundation.org: coding-style fixes]
      [akpm@linux-foundation.org: fix CONFIG_CGROPUPS=n warnings]
      Signed-off-by: default avatarAnton Vorontsov <anton.vorontsov@linaro.org>
      Acked-by: default avatarKirill A. Shutemov <kirill@shutemov.name>
      Acked-by: default avatarKAMEZAWA Hiroyuki <kamezawa.hiroyu@jp.fujitsu.com>
      Cc: Tejun Heo <tj@kernel.org>
      Cc: David Rientjes <rientjes@google.com>
      Cc: Pekka Enberg <penberg@kernel.org>
      Cc: Mel Gorman <mgorman@suse.de>
      Cc: Glauber Costa <glommer@parallels.com>
      Cc: Michal Hocko <mhocko@suse.cz>
      Cc: Luiz Capitulino <lcapitulino@redhat.com>
      Cc: Greg Thelen <gthelen@google.com>
      Cc: Leonid Moiseichuk <leonid.moiseichuk@nokia.com>
      Cc: KOSAKI Motohiro <kosaki.motohiro@gmail.com>
      Cc: Minchan Kim <minchan@kernel.org>
      Cc: Bartlomiej Zolnierkiewicz <b.zolnierkie@samsung.com>
      Cc: John Stultz <john.stultz@linaro.org>
      Signed-off-by: default avatarAndrew Morton <akpm@linux-foundation.org>
      Signed-off-by: default avatarLinus Torvalds <torvalds@linux-foundation.org>
      70ddf637
    • Rasmus Villemoes's avatar
      mm: madvise: complete input validation before taking lock · 84d96d89
      Rasmus Villemoes authored
      In madvise(), there doesn't seem to be any reason for taking the
      &current->mm->mmap_sem before start and len_in have been validated.
      Incidentally, this removes the need for the out: label.
      
      [akpm@linux-foundation.org: s/out_plug/out/, per David]
      Signed-off-by: default avatarRasmus Villemoes <linux@rasmusvillemoes.dk>
      Acked-by: default avatarKOSAKI Motohiro <kosaki.motohiro@jp.fujitsu.com>
      Acked-by: default avatarDavid Rientjes <rientjes@google.com>
      Signed-off-by: default avatarAndrew Morton <akpm@linux-foundation.org>
      Signed-off-by: default avatarLinus Torvalds <torvalds@linux-foundation.org>
      84d96d89
    • David Rientjes's avatar
      mm, hotplug: avoid compiling memory hotremove functions when disabled · 4edd7cef
      David Rientjes authored
      __remove_pages() is only necessary for CONFIG_MEMORY_HOTREMOVE.  PowerPC
      pseries will return -EOPNOTSUPP if unsupported.
      
      Adding an #ifdef causes several other functions it depends on to also
      become unnecessary, which saves in .text when disabled (it's disabled in
      most defconfigs besides powerpc, including x86).  remove_memory_block()
      becomes static since it is not referenced outside of
      drivers/base/memory.c.
      
      Build tested on x86 and powerpc with CONFIG_MEMORY_HOTREMOVE both enabled
      and disabled.
      Signed-off-by: default avatarDavid Rientjes <rientjes@google.com>
      Acked-by: default avatarToshi Kani <toshi.kani@hp.com>
      Cc: Benjamin Herrenschmidt <benh@kernel.crashing.org>
      Cc: Paul Mackerras <paulus@samba.org>
      Cc: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
      Cc: Wen Congyang <wency@cn.fujitsu.com>
      Cc: Tang Chen <tangchen@cn.fujitsu.com>
      Cc: Yasuaki Ishimatsu <isimatu.yasuaki@jp.fujitsu.com>
      Signed-off-by: default avatarAndrew Morton <akpm@linux-foundation.org>
      Signed-off-by: default avatarLinus Torvalds <torvalds@linux-foundation.org>
      4edd7cef
    • Toshi Kani's avatar
      mm: change __remove_pages() to call release_mem_region_adjustable() · fe74ebb1
      Toshi Kani authored
      Change __remove_pages() to call release_mem_region_adjustable().  This
      allows a requested memory range to be released from the iomem_resource
      table even if it does not match exactly to an resource entry but still
      fits into.  The resource entries initialized at bootup usually cover the
      whole contiguous memory ranges and may not necessarily match with the
      size of memory hot-delete requests.
      
      If release_mem_region_adjustable() failed, __remove_pages() emits a
      warning message and continues to proceed as it was the case with
      release_mem_region().  release_mem_region(), which is defined to
      __release_region(), emits a warning message and returns no error since a
      void function.
      Signed-off-by: default avatarToshi Kani <toshi.kani@hp.com>
      Reviewed-by : Yasuaki Ishimatsu <isimatu.yasuaki@jp.fujitsu.com>
      Acked-by: default avatarDavid Rientjes <rientjes@google.com>
      Cc: Ram Pai <linuxram@us.ibm.com>
      Cc: T Makphaibulchoke <tmac@hp.com>
      Cc: Wen Congyang <wency@cn.fujitsu.com>
      Cc: Tang Chen <tangchen@cn.fujitsu.com>
      Cc: Jiang Liu <jiang.liu@huawei.com>
      Signed-off-by: default avatarAndrew Morton <akpm@linux-foundation.org>
      Signed-off-by: default avatarLinus Torvalds <torvalds@linux-foundation.org>
      fe74ebb1
    • Toshi Kani's avatar
      resource: add release_mem_region_adjustable() · 825f787b
      Toshi Kani authored
      Add release_mem_region_adjustable(), which releases a requested region
      from a currently busy memory resource.  This interface adjusts the
      matched memory resource accordingly even if the requested region does
      not match exactly but still fits into.
      
      This new interface is intended for memory hot-delete.  During bootup,
      memory resources are inserted from the boot descriptor table, such as
      EFI Memory Table and e820.  Each memory resource entry usually covers
      the whole contigous memory range.  Memory hot-delete request, on the
      other hand, may target to a particular range of memory resource, and its
      size can be much smaller than the whole contiguous memory.  Since the
      existing release interfaces like __release_region() require a requested
      region to be exactly matched to a resource entry, they do not allow a
      partial resource to be released.
      
      This new interface is restrictive (i.e.  release under certain
      conditions), which is consistent with other release interfaces,
      __release_region() and __release_resource().  Additional release
      conditions, such as an overlapping region to a resource entry, can be
      supported after they are confirmed as valid cases.
      
      There is no change to the existing interfaces since their restriction is
      valid for I/O resources.
      
      [akpm@linux-foundation.org: use GFP_ATOMIC under write_lock()]
      [akpm@linux-foundation.org: switch back to GFP_KERNEL, less buggily]
      [akpm@linux-foundation.org: remove unneeded and wrong kfree(), per Toshi]
      Signed-off-by: default avatarToshi Kani <toshi.kani@hp.com>
      Reviewed-by : Yasuaki Ishimatsu <isimatu.yasuaki@jp.fujitsu.com>
      Cc: David Rientjes <rientjes@google.com>
      Reviewed-by: default avatarRam Pai <linuxram@us.ibm.com>
      Cc: T Makphaibulchoke <tmac@hp.com>
      Cc: Wen Congyang <wency@cn.fujitsu.com>
      Cc: Tang Chen <tangchen@cn.fujitsu.com>
      Cc: Jiang Liu <jiang.liu@huawei.com>
      Signed-off-by: default avatarAndrew Morton <akpm@linux-foundation.org>
      Signed-off-by: default avatarLinus Torvalds <torvalds@linux-foundation.org>
      825f787b
    • Toshi Kani's avatar
      resource: add __adjust_resource() for internal use · ae8e3a91
      Toshi Kani authored
      Add __adjust_resource(), which is called by adjust_resource() internally
      after the resource_lock is held.  There is no interface change to
      adjust_resource().  This change allows other functions to call
      __adjust_resource() internally while the resource_lock is held.
      Signed-off-by: default avatarToshi Kani <toshi.kani@hp.com>
      Reviewed-by: default avatarYasuaki Ishimatsu <isimatu.yasuaki@jp.fujitsu.com>
      Acked-by: default avatarDavid Rientjes <rientjes@google.com>
      Cc: Ram Pai <linuxram@us.ibm.com>
      Cc: T Makphaibulchoke <tmac@hp.com>
      Cc: Wen Congyang <wency@cn.fujitsu.com>
      Cc: Tang Chen <tangchen@cn.fujitsu.com>
      Cc: Jiang Liu <jiang.liu@huawei.com>
      Signed-off-by: default avatarAndrew Morton <akpm@linux-foundation.org>
      Signed-off-by: default avatarLinus Torvalds <torvalds@linux-foundation.org>
      ae8e3a91
    • Srivatsa S. Bhat's avatar
      mm: rewrite the comment over migrate_pages() more comprehensibly · c73e5c9c
      Srivatsa S. Bhat authored
      The comment over migrate_pages() looks quite weird, and makes it hard to
      grasp what it is trying to say.  Rewrite it more comprehensibly.
      Signed-off-by: default avatarSrivatsa S. Bhat <srivatsa.bhat@linux.vnet.ibm.com>
      Acked-by: default avatarChristoph Lameter <cl@linux.com>
      Cc: Mel Gorman <mgorman@suse.de>
      Cc: Hugh Dickins <hughd@google.com>
      Signed-off-by: default avatarAndrew Morton <akpm@linux-foundation.org>
      Signed-off-by: default avatarLinus Torvalds <torvalds@linux-foundation.org>
      c73e5c9c
    • Minchan Kim's avatar
      THP: fix comment about memory barrier · 52f37629
      Minchan Kim authored
      Currently the memory barrier in __do_huge_pmd_anonymous_page doesn't
      work.  Because lru_cache_add_lru uses pagevec so it could miss spinlock
      easily so above rule was broken so user might see inconsistent data.
      
      I was not first person who pointed out the problem.  Mel and Peter
      pointed out a few months ago and Peter pointed out further that even
      spin_lock/unlock can't make sure of it:
      
        http://marc.info/?t=134333512700004
      
      	In particular:
      
              	*A = a;
              	LOCK
              	UNLOCK
              	*B = b;
      
      	may occur as:
      
              	LOCK, STORE *B, STORE *A, UNLOCK
      
      At last, Hugh pointed out that even we don't need memory barrier in
      there because __SetPageUpdate already have done it from Nick's commit
      0ed361de ("mm: fix PageUptodate data race") explicitly.
      
      So this patch fixes comment on THP and adds same comment for
      do_anonymous_page, too because everybody except Hugh was missing that.
      It means we need a comment about that.
      Signed-off-by: default avatarMinchan Kim <minchan@kernel.org>
      Acked-by: default avatarAndrea Arcangeli <aarcange@redhat.com>
      Acked-by: default avatarDavid Rientjes <rientjes@google.com>
      Acked-by: default avatarKAMEZAWA Hiroyuki <kamezawa.hiroyu@jp.fujitsu.com>
      Cc: Mel Gorman <mgorman@suse.de>
      Cc: Hugh Dickins <hughd@google.com>
      Cc: Peter Zijlstra <a.p.zijlstra@chello.nl>
      Signed-off-by: default avatarAndrew Morton <akpm@linux-foundation.org>
      Signed-off-by: default avatarLinus Torvalds <torvalds@linux-foundation.org>
      52f37629
    • Yijing Wang's avatar
      mm: remove CONFIG_HOTPLUG ifdefs · f1cb0879
      Yijing Wang authored
      CONFIG_HOTPLUG is going away as an option, cleanup CONFIG_HOTPLUG
      ifdefs in mm files.
      Signed-off-by: default avatarYijing Wang <wangyijing@huawei.com>
      Acked-by: default avatarGreg Kroah-Hartman <gregkh@linuxfoundation.org>
      Acked-by: default avatarRik van Riel <riel@redhat.com>
      Signed-off-by: default avatarAndrew Morton <akpm@linux-foundation.org>
      Signed-off-by: default avatarLinus Torvalds <torvalds@linux-foundation.org>
      f1cb0879
    • Michel Lespinasse's avatar
      mm/memcontrol.c: remove unnecessary ; · 573b400d
      Michel Lespinasse authored
      Just a trivial issue I stumbled on while doing something else...
      Signed-off-by: default avatarMichel Lespinasse <walken@google.com>
      Signed-off-by: default avatarAndrew Morton <akpm@linux-foundation.org>
      Signed-off-by: default avatarLinus Torvalds <torvalds@linux-foundation.org>
      573b400d
    • Andrew Shewmaker's avatar
      mm: reinititalise user and admin reserves if memory is added or removed · 1640879a
      Andrew Shewmaker authored
      Alter the admin and user reserves of the previous patches in this series
      when memory is added or removed.
      
      If memory is added and the reserves have been eliminated or increased
      above the default max, then we'll trust the admin.
      
      If memory is removed and there isn't enough free memory, then we need to
      reset the reserves.
      
      Otherwise keep the reserve set by the admin.
      
      The reserve reset code is the same as the reserve initialization code.
      
      I tested hot addition and removal by triggering it via sysfs.  The
      reserves shrunk when they were set high and memory was removed.  They
      were reset higher when memory was added again.
      
      [akpm@linux-foundation.org: use register_hotmemory_notifier()]
      [akpm@linux-foundation.org: init_user_reserve() and init_admin_reserve can no longer be __meminit]
      [fengguang.wu@intel.com: make init_reserve_notifier() static]
      [akpm@linux-foundation.org: coding-style fixes]
      Signed-off-by: default avatarAndrew Shewmaker <agshew@gmail.com>
      Signed-off-by: default avatarFengguang Wu <fengguang.wu@intel.com>
      Signed-off-by: default avatarAndrew Morton <akpm@linux-foundation.org>
      Signed-off-by: default avatarLinus Torvalds <torvalds@linux-foundation.org>
      1640879a
    • Andrew Shewmaker's avatar
      mm: replace hardcoded 3% with admin_reserve_pages knob · 4eeab4f5
      Andrew Shewmaker authored
      Add an admin_reserve_kbytes knob to allow admins to change the hardcoded
      memory reserve to something other than 3%, which may be multiple
      gigabytes on large memory systems.  Only about 8MB is necessary to
      enable recovery in the default mode, and only a few hundred MB are
      required even when overcommit is disabled.
      
      This affects OVERCOMMIT_GUESS and OVERCOMMIT_NEVER.
      
      admin_reserve_kbytes is initialized to min(3% free pages, 8MB)
      
      I arrived at 8MB by summing the RSS of sshd or login, bash, and top.
      
      Please see first patch in this series for full background, motivation,
      testing, and full changelog.
      
      [akpm@linux-foundation.org: coding-style fixes]
      [akpm@linux-foundation.org: make init_admin_reserve() static]
      Signed-off-by: default avatarAndrew Shewmaker <agshew@gmail.com>
      Signed-off-by: default avatarAndrew Morton <akpm@linux-foundation.org>
      Signed-off-by: default avatarLinus Torvalds <torvalds@linux-foundation.org>
      4eeab4f5
    • Andrew Shewmaker's avatar
      mm: limit growth of 3% hardcoded other user reserve · c9b1d098
      Andrew Shewmaker authored
      Add user_reserve_kbytes knob.
      
      Limit the growth of the memory reserved for other user processes to
      min(3% current process size, user_reserve_pages).  Only about 8MB is
      necessary to enable recovery in the default mode, and only a few hundred
      MB are required even when overcommit is disabled.
      
      user_reserve_pages defaults to min(3% free pages, 128MB)
      
      I arrived at 128MB by taking the max VSZ of sshd, login, bash, and top ...
      then adding the RSS of each.
      
      This only affects OVERCOMMIT_NEVER mode.
      
      Background
      
      1. user reserve
      
      __vm_enough_memory reserves a hardcoded 3% of the current process size for
      other applications when overcommit is disabled.  This was done so that a
      user could recover if they launched a memory hogging process.  Without the
      reserve, a user would easily run into a message such as:
      
      bash: fork: Cannot allocate memory
      
      2. admin reserve
      
      Additionally, a hardcoded 3% of free memory is reserved for root in both
      overcommit 'guess' and 'never' modes.  This was intended to prevent a
      scenario where root-cant-log-in and perform recovery operations.
      
      Note that this reserve shrinks, and doesn't guarantee a useful reserve.
      
      Motivation
      
      The two hardcoded memory reserves should be updated to account for current
      memory sizes.
      
      Also, the admin reserve would be more useful if it didn't shrink too much.
      
      When the current code was originally written, 1GB was considered
      "enterprise".  Now the 3% reserve can grow to multiple GB on large memory
      systems, and it only needs to be a few hundred MB at most to enable a user
      or admin to recover a system with an unwanted memory hogging process.
      
      I've found that reducing these reserves is especially beneficial for a
      specific type of application load:
      
       * single application system
       * one or few processes (e.g. one per core)
       * allocating all available memory
       * not initializing every page immediately
       * long running
      
      I've run scientific clusters with this sort of load.  A long running job
      sometimes failed many hours (weeks of CPU time) into a calculation.  They
      weren't initializing all of their memory immediately, and they weren't
      using calloc, so I put systems into overcommit 'never' mode.  These
      clusters run diskless and have no swap.
      
      However, with the current reserves, a user wishing to allocate as much
      memory as possible to one process may be prevented from using, for
      example, almost 2GB out of 32GB.
      
      The effect is less, but still significant when a user starts a job with
      one process per core.  I have repeatedly seen a set of processes
      requesting the same amount of memory fail because one of them could not
      allocate the amount of memory a user would expect to be able to allocate.
      For example, Message Passing Interfce (MPI) processes, one per core.  And
      it is similar for other parallel programming frameworks.
      
      Changing this reserve code will make the overcommit never mode more useful
      by allowing applications to allocate nearly all of the available memory.
      
      Also, the new admin_reserve_kbytes will be safer than the current behavior
      since the hardcoded 3% of available memory reserve can shrink to something
      useless in the case where applications have grabbed all available memory.
      
      Risks
      
      * "bash: fork: Cannot allocate memory"
      
        The downside of the first patch-- which creates a tunable user reserve
        that is only used in overcommit 'never' mode--is that an admin can set
        it so low that a user may not be able to kill their process, even if
        they already have a shell prompt.
      
        Of course, a user can get in the same predicament with the current 3%
        reserve--they just have to launch processes until 3% becomes negligible.
      
      * root-cant-log-in problem
      
        The second patch, adding the tunable rootuser_reserve_pages, allows
        the admin to shoot themselves in the foot by setting it too small.  They
        can easily get the system into a state where root-can't-log-in.
      
        However, the new admin_reserve_kbytes will be safer than the current
        behavior since the hardcoded 3% of available memory reserve can shrink
        to something useless in the case where applications have grabbed all
        available memory.
      
      Alternatives
      
       * Memory cgroups provide a more flexible way to limit application memory.
      
         Not everyone wants to set up cgroups or deal with their overhead.
      
       * We could create a fourth overcommit mode which provides smaller reserves.
      
         The size of useful reserves may be drastically different depending
         on the whether the system is embedded or enterprise.
      
       * Force users to initialize all of their memory or use calloc.
      
         Some users don't want/expect the system to overcommit when they malloc.
         Overcommit 'never' mode is for this scenario, and it should work well.
      
      The new user and admin reserve tunables are simple to use, with low
      overhead compared to cgroups.  The patches preserve current behavior where
      3% of memory is less than 128MB, except that the admin reserve doesn't
      shrink to an unusable size under pressure.  The code allows admins to tune
      for embedded and enterprise usage.
      
      FAQ
      
       * How is the root-cant-login problem addressed?
         What happens if admin_reserve_pages is set to 0?
      
         Root is free to shoot themselves in the foot by setting
         admin_reserve_kbytes too low.
      
         On x86_64, the minimum useful reserve is:
           8MB for overcommit 'guess'
         128MB for overcommit 'never'
      
         admin_reserve_pages defaults to min(3% free memory, 8MB)
      
         So, anyone switching to 'never' mode needs to adjust
         admin_reserve_pages.
      
       * How do you calculate a minimum useful reserve?
      
         A user or the admin needs enough memory to login and perform
         recovery operations, which includes, at a minimum:
      
         sshd or login + bash (or some other shell) + top (or ps, kill, etc.)
      
         For overcommit 'guess', we can sum resident set sizes (RSS)
         because we only need enough memory to handle what the recovery
         programs will typically use. On x86_64 this is about 8MB.
      
         For overcommit 'never', we can take the max of their virtual sizes (VSZ)
         and add the sum of their RSS. We use VSZ instead of RSS because mode
         forces us to ensure we can fulfill all of the requested memory allocations--
         even if the programs only use a fraction of what they ask for.
         On x86_64 this is about 128MB.
      
         When swap is enabled, reserves are useful even when they are as
         small as 10MB, regardless of overcommit mode.
      
         When both swap and overcommit are disabled, then the admin should
         tune the reserves higher to be absolutley safe. Over 230MB each
         was safest in my testing.
      
       * What happens if user_reserve_pages is set to 0?
      
         Note, this only affects overcomitt 'never' mode.
      
         Then a user will be able to allocate all available memory minus
         admin_reserve_kbytes.
      
         However, they will easily see a message such as:
      
         "bash: fork: Cannot allocate memory"
      
         And they won't be able to recover/kill their application.
         The admin should be able to recover the system if
         admin_reserve_kbytes is set appropriately.
      
       * What's the difference between overcommit 'guess' and 'never'?
      
         "Guess" allows an allocation if there are enough free + reclaimable
         pages. It has a hardcoded 3% of free pages reserved for root.
      
         "Never" allows an allocation if there is enough swap + a configurable
         percentage (default is 50) of physical RAM. It has a hardcoded 3% of
         free pages reserved for root, like "Guess" mode. It also has a
         hardcoded 3% of the current process size reserved for additional
         applications.
      
       * Why is overcommit 'guess' not suitable even when an app eventually
         writes to every page? It takes free pages, file pages, available
         swap pages, reclaimable slab pages into consideration. In other words,
         these are all pages available, then why isn't overcommit suitable?
      
         Because it only looks at the present state of the system. It
         does not take into account the memory that other applications have
         malloced, but haven't initialized yet. It overcommits the system.
      
      Test Summary
      
      There was little change in behavior in the default overcommit 'guess'
      mode with swap enabled before and after the patch. This was expected.
      
      Systems run most predictably (i.e. no oom kills) in overcommit 'never'
      mode with swap enabled. This also allowed the most memory to be allocated
      to a user application.
      
      Overcommit 'guess' mode without swap is a bad idea. It is easy to
      crash the system. None of the other tested combinations crashed.
      This matches my experience on the Roadrunner supercomputer.
      
      Without the tunable user reserve, a system in overcommit 'never' mode
      and without swap does not allow the admin to recover, although the
      admin can.
      
      With the new tunable reserves, a system in overcommit 'never' mode
      and without swap can be configured to:
      
      1. maximize user-allocatable memory, running close to the edge of
      recoverability
      
      2. maximize recoverability, sacrificing allocatable memory to
      ensure that a user cannot take down a system
      
      Test Description
      
      Fedora 18 VM - 4 x86_64 cores, 5725MB RAM, 4GB Swap
      
      System is booted into multiuser console mode, with unnecessary services
      turned off. Caches were dropped before each test.
      
      Hogs are user memtester processes that attempt to allocate all free memory
      as reported by /proc/meminfo
      
      In overcommit 'never' mode, memory_ratio=100
      
      Test Results
      
      3.9.0-rc1-mm1
      
      Overcommit | Swap | Hogs | MB Got/Wanted | OOMs | User Recovery | Admin Recovery
      ----------   ----   ----   -------------   ----   -------------   --------------
      guess        yes    1      5432/5432       no     yes             yes
      guess        yes    4      5444/5444       1      yes             yes
      guess        no     1      5302/5449       no     yes             yes
      guess        no     4      -               crash  no              no
      
      never        yes    1      5460/5460       1      yes             yes
      never        yes    4      5460/5460       1      yes             yes
      never        no     1      5218/5432       no     no              yes
      never        no     4      5203/5448       no     no              yes
      
      3.9.0-rc1-mm1-tunablereserves
      
      User and Admin Recovery show their respective reserves, if applicable.
      
      Overcommit | Swap | Hogs | MB Got/Wanted | OOMs | User Recovery | Admin Recovery
      ----------   ----   ----   -------------   ----   -------------   --------------
      guess        yes    1      5419/5419       no     - yes           8MB yes
      guess        yes    4      5436/5436       1      - yes           8MB yes
      guess        no     1      5440/5440       *      - yes           8MB yes
      guess        no     4      -               crash  - no            8MB no
      
      * process would successfully mlock, then the oom killer would pick it
      
      never        yes    1      5446/5446       no     10MB yes        20MB yes
      never        yes    4      5456/5456       no     10MB yes        20MB yes
      never        no     1      5387/5429       no     128MB no        8MB barely
      never        no     1      5323/5428       no     226MB barely    8MB barely
      never        no     1      5323/5428       no     226MB barely    8MB barely
      
      never        no     1      5359/5448       no     10MB no         10MB barely
      
      never        no     1      5323/5428       no     0MB no          10MB barely
      never        no     1      5332/5428       no     0MB no          50MB yes
      never        no     1      5293/5429       no     0MB no          90MB yes
      
      never        no     1      5001/5427       no     230MB yes       338MB yes
      never        no     4*     4998/5424       no     230MB yes       338MB yes
      
      * more memtesters were launched, able to allocate approximately another 100MB
      
      Future Work
      
       - Test larger memory systems.
      
       - Test an embedded image.
      
       - Test other architectures.
      
       - Time malloc microbenchmarks.
      
       - Would it be useful to be able to set overcommit policy for
         each memory cgroup?
      
       - Some lines are slightly above 80 chars.
         Perhaps define a macro to convert between pages and kb?
         Other places in the kernel do this.
      
      [akpm@linux-foundation.org: coding-style fixes]
      [akpm@linux-foundation.org: make init_user_reserve() static]
      Signed-off-by: default avatarAndrew Shewmaker <agshew@gmail.com>
      Signed-off-by: default avatarAndrew Morton <akpm@linux-foundation.org>
      Signed-off-by: default avatarLinus Torvalds <torvalds@linux-foundation.org>
      c9b1d098
    • Andrew Morton's avatar
      kernel/cpuset.c: use register_hotmemory_notifier() · d8f10cb3
      Andrew Morton authored
      Use the new interface, remove one ifdef.  No code size changes.
      
      We could/should have been using __meminit/__meminitdata here but there's
      now no point in doing that because all this code is elided at compile time.
      
      Cc: Li Zefan <lizefan@huawei.com>
      Cc: Tejun Heo <tj@kernel.org>
      Signed-off-by: default avatarAndrew Morton <akpm@linux-foundation.org>
      Signed-off-by: default avatarLinus Torvalds <torvalds@linux-foundation.org>
      d8f10cb3
    • Andrew Morton's avatar
      fs/proc/kcore.c: use register_hotmemory_notifier() · 3c743a7f
      Andrew Morton authored
      Saves an ifdef, no code size changes
      Signed-off-by: default avatarAndrew Morton <akpm@linux-foundation.org>
      Signed-off-by: default avatarLinus Torvalds <torvalds@linux-foundation.org>
      3c743a7f
    • Andrew Morton's avatar
      drivers/base/node.c: switch to register_hotmemory_notifier() · 6e259e7d
      Andrew Morton authored
      Squishes a warning which my change to hotplug_memory_notifier() added.
      
      I want to keep that warning, because it is punishment for failnig to check
      the hotplug_memory_notifier() return value.
      
      Cc: Greg KH <greg@kroah.com>
      Signed-off-by: default avatarAndrew Morton <akpm@linux-foundation.org>
      Signed-off-by: default avatarLinus Torvalds <torvalds@linux-foundation.org>
      6e259e7d
    • Andrew Morton's avatar
      mm/slub.c: use register_hotmemory_notifier() · 3ac38faa
      Andrew Morton authored
      Squishes a statement-with-no-effect warning, removes some ifdefs and
      shrinks .text by 2 bytes.
      
      Note that this code fails to check for blocking_notifier_chain_register()
      failures.
      
      Cc: Pekka Enberg <penberg@kernel.org>
      Signed-off-by: default avatarAndrew Morton <akpm@linux-foundation.org>
      Signed-off-by: default avatarLinus Torvalds <torvalds@linux-foundation.org>
      3ac38faa
    • Andrew Morton's avatar
      ipc/util.c: use register_hotmemory_notifier() · 8f68fa2d
      Andrew Morton authored
      Squishes a statement-with-no-effect warning, removes some ifdefs and
      shrinks .text by one byte!
      
      Note that this code fails to check for blocking_notifier_chain_register()
      failures.
      Signed-off-by: default avatarAndrew Morton <akpm@linux-foundation.org>
      Signed-off-by: default avatarLinus Torvalds <torvalds@linux-foundation.org>
      8f68fa2d
    • Andrew Morton's avatar
      include/linux/memory.h: implement register_hotmemory_notifier() · f02c6968
      Andrew Morton authored
      When CONFIG_MEMORY_HOTPLUG=n, we don't want the memory-hotplug notifier
      handlers to be included in the .o files, for space reasons.
      
      The existing hotplug_memory_notifier() tries to handle this but testing
      with gcc-4.4.4 shows that it doesn't work - the hotplug functions are
      still present in the .o files.
      
      So implement a new register_hotmemory_notifier() which is a copy of
      register_hotcpu_notifier(), and which actually works as desired.
      hotplug_memory_notifier() and register_memory_notifier() callsites
      should be converted to use this new register_hotmemory_notifier().
      
      While we're there, let's repair the existing hotplug_memory_notifier():
      it simply stomps on the register_memory_notifier() return value, so
      well-behaved code cannot check for errors.  Apparently non of the
      existing callers were well-behaved :(
      
      Cc: Andrew Shewmaker <agshew@gmail.com>
      Signed-off-by: default avatarAndrew Morton <akpm@linux-foundation.org>
      Signed-off-by: default avatarLinus Torvalds <torvalds@linux-foundation.org>
      f02c6968
    • Cody P Schafer's avatar
      powerpc/mm/numa: use setup_nr_node_ids() instead of opencoding. · f9d531b8
      Cody P Schafer authored
      [sfr@canb.auug.org.au: add missing semicolon]
      Signed-off-by: default avatarCody P Schafer <cody@linux.vnet.ibm.com>
      Cc: "H. Peter Anvin" <hpa@zytor.com>
      Cc: Ingo Molnar <mingo@redhat.com>
      Cc: Benjamin Herrenschmidt <benh@kernel.crashing.org>
      Cc: Yinghai Lu <yinghai@kernel.org>
      Signed-off-by: default avatarStephen Rothwell <sfr@canb.auug.org.au>
      Signed-off-by: default avatarAndrew Morton <akpm@linux-foundation.org>
      Signed-off-by: default avatarLinus Torvalds <torvalds@linux-foundation.org>
      f9d531b8
    • Cody P Schafer's avatar
      x86/mm/numa: use setup_nr_node_ids() instead of opencoding. · d2ad351e
      Cody P Schafer authored
      Signed-off-by: default avatarCody P Schafer <cody@linux.vnet.ibm.com>
      Cc: "H. Peter Anvin" <hpa@zytor.com>
      Cc: Ingo Molnar <mingo@redhat.com>
      Cc: Benjamin Herrenschmidt <benh@kernel.crashing.org>
      Acked-by: default avatarYinghai Lu <yinghai@kernel.org>
      Signed-off-by: default avatarAndrew Morton <akpm@linux-foundation.org>
      Signed-off-by: default avatarLinus Torvalds <torvalds@linux-foundation.org>
      d2ad351e
    • Cody P Schafer's avatar
      page_alloc: make setup_nr_node_ids() usable for arch init code · f9872caf
      Cody P Schafer authored
      powerpc and x86 were opencoding copies of setup_nr_node_ids(), which
      page_alloc provides but makes static.  Make it avaliable to the archs in
      linux/mm.h.
      Signed-off-by: default avatarCody P Schafer <cody@linux.vnet.ibm.com>
      Signed-off-by: default avatarLinus Torvalds <torvalds@linux-foundation.org>
      f9872caf
    • Russ Anderson's avatar
      mm: speedup in __early_pfn_to_nid · 7c243c71
      Russ Anderson authored
      When booting on a large memory system, the kernel spends considerable
      time in memmap_init_zone() setting up memory zones.  Analysis shows
      significant time spent in __early_pfn_to_nid().
      
      The routine memmap_init_zone() checks each PFN to verify the nid is
      valid.  __early_pfn_to_nid() sequentially scans the list of pfn ranges
      to find the right range and returns the nid.  This does not scale well.
      On a 4 TB (single rack) system there are 308 memory ranges to scan.  The
      higher the PFN the more time spent sequentially spinning through memory
      ranges.
      
      Since memmap_init_zone() increments pfn, it will almost always be
      looking for the same range as the previous pfn, so check that range
      first.  If it is in the same range, return that nid.  If not, scan the
      list as before.
      
      A 4 TB (single rack) UV1 system takes 512 seconds to get through the
      zone code.  This performance optimization reduces the time by 189
      seconds, a 36% improvement.
      
      A 2 TB (single rack) UV2 system goes from 212.7 seconds to 99.8 seconds,
      a 112.9 second (53%) reduction.
      
      [akpm@linux-foundation.org: make the statics __meminitdata]
      [akpm@linux-foundation.org: fix comment formatting]
      [akpm@linux-foundation.org: fix ia64, per yinghai]
      [akpm@linux-foundation.org: add missing semicolon, per Tony]
      Signed-off-by: default avatarRuss Anderson <rja@sgi.com>
      Cc: David Rientjes <rientjes@google.com>
      Cc: Ingo Molnar <mingo@elte.hu>
      Cc: Thomas Gleixner <tglx@linutronix.de>
      Cc: "H. Peter Anvin" <hpa@zytor.com>
      Tested-by: default avatar"Luck, Tony" <tony.luck@intel.com>
      Cc: Yinghai Lu <yinghai@kernel.org>
      Cc: Lin Feng <linfeng@cn.fujitsu.com>
      Cc: KOSAKI Motohiro <kosaki.motohiro@gmail.com>
      Signed-off-by: default avatarAndrew Morton <akpm@linux-foundation.org>
      Signed-off-by: default avatarLinus Torvalds <torvalds@linux-foundation.org>
      7c243c71
    • Jianguo Wu's avatar
    • Mel Gorman's avatar
      mm: page_alloc: avoid marking zones full prematurely after zone_reclaim() · fed2719e
      Mel Gorman authored
      The following problem was reported against a distribution kernel when
      zone_reclaim was enabled but the same problem applies to the mainline
      kernel.  The reproduction case was as follows
      
      1. Run numactl -m +0 dd if=largefile of=/dev/null
         This allocates a large number of clean pages in node 0
      
      2. numactl -N +0 memhog 0.5*Mg
         This start a memory-using application in node 0.
      
      The expected behaviour is that the clean pages get reclaimed and the
      application uses node 0 for its memory.  The observed behaviour was that
      the memory for the memhog application was allocated off-node since
      commits cd38b115 ("mm: page allocator: initialise ZLC for first zone
      eligible for zone_reclaim") and commit 76d3fbf8 ("mm: page
      allocator: reconsider zones for allocation after direct reclaim").
      
      The assumption of those patches was that it was always preferable to
      allocate quickly than stall for long periods of time and they were meant
      to take care that the zone was only marked full when necessary but an
      important case was missed.
      
      In the allocator fast path, only the low watermarks are checked.  If the
      zones free pages are between the low and min watermark then allocations
      from the allocators slow path will succeed.  However, zone_reclaim will
      only reclaim SWAP_CLUSTER_MAX or 1<<order pages.  There is no guarantee
      that this will meet the low watermark causing the zone to be marked full
      prematurely.
      
      This patch will only mark the zone full after zone_reclaim if it the min
      watermarks are checked or if page reclaim failed to make sufficient
      progress.
      
      [mhocko@suse.cz: fix alloc_flags test]
      Signed-off-by: default avatarMel Gorman <mgorman@suse.de>
      Reported-by: default avatarHedi Berriche <hedi@sgi.com>
      Tested-by: default avatarHedi Berriche <hedi@sgi.com>
      Reviewed-by: default avatarMichal Hocko <mhocko@suse.cz>
      Reviewed-by: default avatarWanpeng Li <liwanp@linux.vnet.ibm.com>
      Signed-off-by: default avatarMichal Hocko <mhocko@suse.cz>
      Signed-off-by: default avatarAndrew Morton <akpm@linux-foundation.org>
      Signed-off-by: default avatarLinus Torvalds <torvalds@linux-foundation.org>
      fed2719e
    • Johannes Weiner's avatar
      x86-64: fall back to regular page vmemmap on allocation failure · 8e2cdbcb
      Johannes Weiner authored
      Memory hotplug can happen on a machine under load, memory shortness
      and fragmentation, so huge page allocations for the vmemmap are not
      guaranteed to succeed.
      
      Try to fall back to regular pages before failing the hotplug event
      completely.
      Signed-off-by: default avatarJohannes Weiner <hannes@cmpxchg.org>
      Cc: Ben Hutchings <ben@decadent.org.uk>
      Cc: Bernhard Schmidt <Bernhard.Schmidt@lrz.de>
      Cc: Johannes Weiner <hannes@cmpxchg.org>
      Cc: Russell King <rmk@arm.linux.org.uk>
      Cc: Ingo Molnar <mingo@elte.hu>
      Cc: Thomas Gleixner <tglx@linutronix.de>
      Cc: "H. Peter Anvin" <hpa@zytor.com>
      Cc: Benjamin Herrenschmidt <benh@kernel.crashing.org>
      Cc: "Luck, Tony" <tony.luck@intel.com>
      Cc: Heiko Carstens <heiko.carstens@de.ibm.com>
      Cc: David Miller <davem@davemloft.net>
      Signed-off-by: default avatarAndrew Morton <akpm@linux-foundation.org>
      Signed-off-by: default avatarLinus Torvalds <torvalds@linux-foundation.org>
      8e2cdbcb
    • Johannes Weiner's avatar
      x86-64: use vmemmap_populate_basepages() for !pse setups · e8216da5
      Johannes Weiner authored
      We already have generic code to allocate vmemmap with regular pages, use
      it.
      Signed-off-by: default avatarJohannes Weiner <hannes@cmpxchg.org>
      Cc: Ben Hutchings <ben@decadent.org.uk>
      Cc: Bernhard Schmidt <Bernhard.Schmidt@lrz.de>
      Cc: Johannes Weiner <hannes@cmpxchg.org>
      Cc: Russell King <rmk@arm.linux.org.uk>
      Cc: Ingo Molnar <mingo@elte.hu>
      Cc: Thomas Gleixner <tglx@linutronix.de>
      Cc: "H. Peter Anvin" <hpa@zytor.com>
      Cc: Benjamin Herrenschmidt <benh@kernel.crashing.org>
      Cc: "Luck, Tony" <tony.luck@intel.com>
      Cc: Heiko Carstens <heiko.carstens@de.ibm.com>
      Cc: David Miller <davem@davemloft.net>
      Cc: Wu Fengguang <fengguang.wu@intel.com>
      Signed-off-by: default avatarAndrew Morton <akpm@linux-foundation.org>
      Signed-off-by: default avatarLinus Torvalds <torvalds@linux-foundation.org>
      e8216da5
    • Johannes Weiner's avatar
      x86-64: remove dead debugging code for !pse setups · 6c7a2ca4
      Johannes Weiner authored
      No need to maintain addr_end and p_end when they are never actually read
      anywhere on !pse setups.  Remove the dead code.
      Signed-off-by: default avatarJohannes Weiner <hannes@cmpxchg.org>
      Cc: Ben Hutchings <ben@decadent.org.uk>
      Cc: Bernhard Schmidt <Bernhard.Schmidt@lrz.de>
      Cc: Johannes Weiner <hannes@cmpxchg.org>
      Cc: Russell King <rmk@arm.linux.org.uk>
      Cc: Ingo Molnar <mingo@elte.hu>
      Cc: Thomas Gleixner <tglx@linutronix.de>
      Cc: "H. Peter Anvin" <hpa@zytor.com>
      Cc: Benjamin Herrenschmidt <benh@kernel.crashing.org>
      Cc: "Luck, Tony" <tony.luck@intel.com>
      Cc: Heiko Carstens <heiko.carstens@de.ibm.com>
      Cc: David Miller <davem@davemloft.net>
      Signed-off-by: default avatarAndrew Morton <akpm@linux-foundation.org>
      Signed-off-by: default avatarLinus Torvalds <torvalds@linux-foundation.org>
      6c7a2ca4
    • Johannes Weiner's avatar
      sparse-vmemmap: specify vmemmap population range in bytes · 0aad818b
      Johannes Weiner authored
      The sparse code, when asking the architecture to populate the vmemmap,
      specifies the section range as a starting page and a number of pages.
      
      This is an awkward interface, because none of the arch-specific code
      actually thinks of the range in terms of 'struct page' units and always
      translates it to bytes first.
      
      In addition, later patches mix huge page and regular page backing for
      the vmemmap.  For this, they need to call vmemmap_populate_basepages()
      on sub-section ranges with PAGE_SIZE and PMD_SIZE in mind.  But these
      are not necessarily multiples of the 'struct page' size and so this unit
      is too coarse.
      
      Just translate the section range into bytes once in the generic sparse
      code, then pass byte ranges down the stack.
      Signed-off-by: default avatarJohannes Weiner <hannes@cmpxchg.org>
      Cc: Ben Hutchings <ben@decadent.org.uk>
      Cc: Bernhard Schmidt <Bernhard.Schmidt@lrz.de>
      Cc: Johannes Weiner <hannes@cmpxchg.org>
      Cc: Russell King <rmk@arm.linux.org.uk>
      Cc: Ingo Molnar <mingo@elte.hu>
      Cc: Thomas Gleixner <tglx@linutronix.de>
      Cc: "H. Peter Anvin" <hpa@zytor.com>
      Cc: Benjamin Herrenschmidt <benh@kernel.crashing.org>
      Cc: "Luck, Tony" <tony.luck@intel.com>
      Cc: Heiko Carstens <heiko.carstens@de.ibm.com>
      Acked-by: default avatarDavid S. Miller <davem@davemloft.net>
      Tested-by: default avatarDavid S. Miller <davem@davemloft.net>
      Cc: Wu Fengguang <fengguang.wu@intel.com>
      Signed-off-by: default avatarAndrew Morton <akpm@linux-foundation.org>
      Signed-off-by: default avatarLinus Torvalds <torvalds@linux-foundation.org>
      0aad818b
    • Ben Hutchings's avatar
      mm: try harder to allocate vmemmap blocks · 055e4fd9
      Ben Hutchings authored
      Hot-adding memory on x86_64 normally requires huge page allocation.
      When this is done to a VM guest, it's usually because the system is
      already tight on memory, so the request tends to fail.  Try to avoid
      this by adding __GFP_REPEAT to the allocation flags.
      
      Addresses http://bugs.debian.org/699913Signed-off-by: default avatarBen Hutchings <ben@decadent.org.uk>
      Signed-off-by: default avatarJohannes Weiner <hannes@cmpxchg.org>
      Reported-by: default avatarBernhard Schmidt <Bernhard.Schmidt@lrz.de>
      Tested-by: default avatarBernhard Schmidt <Bernhard.Schmidt@lrz.de>
      Cc: Russell King <rmk@arm.linux.org.uk>
      Cc: Ingo Molnar <mingo@elte.hu>
      Cc: Thomas Gleixner <tglx@linutronix.de>
      Cc: "H. Peter Anvin" <hpa@zytor.com>
      Cc: Benjamin Herrenschmidt <benh@kernel.crashing.org>
      Cc: "Luck, Tony" <tony.luck@intel.com>
      Cc: Heiko Carstens <heiko.carstens@de.ibm.com>
      Cc: David Miller <davem@davemloft.net>
      Signed-off-by: default avatarAndrew Morton <akpm@linux-foundation.org>
      Signed-off-by: default avatarLinus Torvalds <torvalds@linux-foundation.org>
      055e4fd9
    • David Rientjes's avatar
      mm, hugetlb: include hugepages in meminfo · 949f7ec5
      David Rientjes authored
      Particularly in oom conditions, it's troublesome that hugetlb memory is
      not displayed.  All other meminfo that is emitted will not add up to
      what is expected, and there is no artifact left in the kernel log to
      show that a potentially significant amount of memory is actually
      allocated as hugepages which are not available to be reclaimed.
      
      Booting with hugepages=8192 on the command line, this memory is now
      shown in oom conditions.  For example, with echo m >
      /proc/sysrq-trigger:
      
        Node 0 hugepages_total=2048 hugepages_free=2048 hugepages_surp=0 hugepages_size=2048kB
        Node 1 hugepages_total=2048 hugepages_free=2048 hugepages_surp=0 hugepages_size=2048kB
        Node 2 hugepages_total=2048 hugepages_free=2048 hugepages_surp=0 hugepages_size=2048kB
        Node 3 hugepages_total=2048 hugepages_free=2048 hugepages_surp=0 hugepages_size=2048kB
      
      [akpm@linux-foundation.org: coding-style fixes]
      Signed-off-by: default avatarDavid Rientjes <rientjes@google.com>
      Acked-by: default avatarMichal Hocko <mhocko@suse.cz>
      Signed-off-by: default avatarAndrew Morton <akpm@linux-foundation.org>
      Signed-off-by: default avatarLinus Torvalds <torvalds@linux-foundation.org>
      949f7ec5
    • Hampson, Steven T's avatar
      mm: merging memory blocks resets mempolicy · 1444f92c
      Hampson, Steven T authored
      Using mbind to change the mempolicy to MPOL_BIND on several adjacent
      mmapped blocks may result in a reset of the mempolicy to MPOL_DEFAULT in
      vma_adjust.
      
      Test code.  Correct result is three lines containing "OK".
      
      #include <stdio.h>
      #include <unistd.h>
      #include <sys/mman.h>
      #include <numaif.h>
      #include <errno.h>
      
      /* gcc mbind_test.c -lnuma -o mbind_test -Wall */
      #define MAXNODE 4096
      
      void allocate()
      {
      	int ret;
      	int len;
      	int policy = -1;
      	unsigned char *p;
      	unsigned long mask[MAXNODE] = { 0 };
      	unsigned long retmask[MAXNODE] = { 0 };
      
      	len = getpagesize() * 0x2fc00;
      	p = mmap(NULL, len, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS,
      		 -1, 0);
      	if (p == MAP_FAILED)
      		printf("mbind err: %d\n", errno);
      
      	mask[0] = 1;
      	ret = mbind(p, len, MPOL_BIND, mask, MAXNODE, 0);
      	if (ret < 0)
      		printf("mbind err: %d %d\n", ret, errno);
      	ret = get_mempolicy(&policy, retmask, MAXNODE, p, MPOL_F_ADDR);
      	if (ret < 0)
      		printf("get_mempolicy err: %d %d\n", ret, errno);
      
      	if (policy == MPOL_BIND)
      		printf("OK\n");
      	else
      		printf("ERROR: policy is %d\n", policy);
      }
      
      int main()
      {
      	allocate();
      	allocate();
      	allocate();
      	return 0;
      }
      Signed-off-by: default avatarSteven T Hampson <steven.t.hampson@intel.com>
      Cc: Mel Gorman <mgorman@suse.de>
      Cc: KOSAKI Motohiro <kosaki.motohiro@jp.fujitsu.com>
      Cc: Rik van Riel <riel@redhat.com>
      Cc: Andi Kleen <andi@firstfloor.org>
      Signed-off-by: default avatarAndrew Morton <akpm@linux-foundation.org>
      Signed-off-by: default avatarLinus Torvalds <torvalds@linux-foundation.org>
      1444f92c
    • Catalin Marinas's avatar
      arm: set the page table freeing ceiling to TASK_SIZE · 104ad3b3
      Catalin Marinas authored
      ARM processors with LPAE enabled use 3 levels of page tables, with an
      entry in the top level (pgd) covering 1GB of virtual space.  Because of
      the branch relocation limitations on ARM, the loadable modules are
      mapped 16MB below PAGE_OFFSET, making the corresponding 1GB pgd shared
      between kernel modules and user space.
      
      If free_pgtables() is called with the default ceiling 0,
      free_pgd_range() (and subsequently called functions) also frees the page
      table shared between user space and kernel modules (which is normally
      handled by the ARM-specific pgd_free() function).  This patch changes
      defines the ARM USER_PGTABLES_CEILING to TASK_SIZE when CONFIG_ARM_LPAE
      is enabled.
      
      Note that the pgd_free() function already checks the presence of the
      shared pmd page allocated by pgd_alloc() and frees it, though with
      ceiling 0 this wasn't necessary.
      Signed-off-by: default avatarCatalin Marinas <catalin.marinas@arm.com>
      Cc: Russell King <linux@arm.linux.org.uk>
      Cc: Hugh Dickins <hughd@google.com>
      Cc: <stable@vger.kernel.org>	[3.3+]
      Signed-off-by: default avatarAndrew Morton <akpm@linux-foundation.org>
      Signed-off-by: default avatarLinus Torvalds <torvalds@linux-foundation.org>
      104ad3b3
    • Hugh Dickins's avatar
      mm: allow arch code to control the user page table ceiling · 6ee8630e
      Hugh Dickins authored
      On architectures where a pgd entry may be shared between user and kernel
      (e.g.  ARM+LPAE), freeing page tables needs a ceiling other than 0.
      This patch introduces a generic USER_PGTABLES_CEILING that arch code can
      override.  It is the responsibility of the arch code setting the ceiling
      to ensure the complete freeing of the page tables (usually in
      pgd_free()).
      
      [catalin.marinas@arm.com: commit log; shift_arg_pages(), asm-generic/pgtables.h changes]
      Signed-off-by: default avatarHugh Dickins <hughd@google.com>
      Signed-off-by: default avatarCatalin Marinas <catalin.marinas@arm.com>
      Cc: Russell King <linux@arm.linux.org.uk>
      Cc: <stable@vger.kernel.org>	[3.3+]
      Signed-off-by: default avatarAndrew Morton <akpm@linux-foundation.org>
      Signed-off-by: default avatarLinus Torvalds <torvalds@linux-foundation.org>
      6ee8630e
    • Michal Hocko's avatar
      memcg: do not check for do_swap_account in mem_cgroup_{read,write,reset} · acb6d558
      Michal Hocko authored
      Since commit 2d11085e ("memcg: do not create memsw files if swap
      accounting is disabled") memsw files are created only if memcg swap
      accounting is enabled so it doesn't make any sense to check for it
      explicitly in mem_cgroup_read(), mem_cgroup_write() and
      mem_cgroup_reset().
      Signed-off-by: default avatarMichal Hocko <mhocko@suse.cz>
      Cc: Kamezawa Hiroyuki <kamezawa.hiroyu@jp.fujitsu.com>
      Cc: Tejun Heo <tj@kernel.org>
      Acked-by: default avatarJohannes Weiner <hannes@cmpxchg.org>
      Signed-off-by: default avatarAndrew Morton <akpm@linux-foundation.org>
      Signed-off-by: default avatarLinus Torvalds <torvalds@linux-foundation.org>
      acb6d558