1. 09 Jul, 2009 2 commits
    • Tejun Heo's avatar
      linker script: unify usage of discard definition · 023bf6f1
      Tejun Heo authored
      Discarded sections in different archs share some commonality but have
      considerable differences.  This led to linker script for each arch
      implementing its own /DISCARD/ definition, which makes maintaining
      tedious and adding new entries error-prone.
      
      This patch makes all linker scripts to move discard definitions to the
      end of the linker script and use the common DISCARDS macro.  As ld
      uses the first matching section definition, archs can include default
      discarded sections by including them earlier in the linker script.
      
      ia64 is notable because it first throws away some ia64 specific
      subsections and then include the rest of the sections into the final
      image, so those sections must be discarded before the inclusion.
      
      defconfig compile tested for x86, x86-64, powerpc, powerpc64, ia64,
      alpha, sparc, sparc64 and s390.  Michal Simek tested microblaze.
      Signed-off-by: default avatarTejun Heo <tj@kernel.org>
      Acked-by: default avatarPaul Mundt <lethal@linux-sh.org>
      Acked-by: default avatarMike Frysinger <vapier@gentoo.org>
      Tested-by: default avatarMichal Simek <monstr@monstr.eu>
      Cc: linux-arch@vger.kernel.org
      Cc: Michal Simek <monstr@monstr.eu>
      Cc: microblaze-uclinux@itee.uq.edu.au
      Cc: Sam Ravnborg <sam@ravnborg.org>
      Cc: Tony Luck <tony.luck@intel.com>
      023bf6f1
    • Michal Simek's avatar
      microblaze: include EXIT_TEXT to _stext · 1dcdd091
      Michal Simek authored
      Microblaze wants to throw out EXIT_TEXT during runtime too.  This
      hasn't caused trouble till now because the linker script didn't
      discard EXIT_TEXT and it ended up in its default output section.  As
      discard definition is about to be unified, include EXIT_TEXT into
      _stext explicitly and while at it replace explicit exitcall definition
      to EXIT_CALL.
      Signed-off-by: default avatarMichal Simek <monstr@monstr.eu>
      Signed-off-by: default avatarTejun Heo <tj@kernel.org>
      1dcdd091
  2. 03 Jul, 2009 11 commits
    • Tejun Heo's avatar
      percpu: teach large page allocator about NUMA · a530b795
      Tejun Heo authored
      Large page first chunk allocator is primarily used for NUMA machines;
      however, its NUMA handling is extremely simplistic.  Regardless of
      their proximity, each cpu is put into separate large page just to
      return most of the allocated space back wasting large amount of
      vmalloc space and increasing cache footprint.
      
      This patch teachs NUMA details to large page allocator.  Given
      processor proximity information, pcpu_lpage_build_unit_map() will find
      fitting cpu -> unit mapping in which cpus in LOCAL_DISTANCE share the
      same large page and not too much virtual address space is wasted.
      
      This greatly reduces the unit and thus chunk size and wastes much less
      address space for the first chunk.  For example, on 4/4 NUMA machine,
      the original code occupied 16MB of virtual space for the first chunk
      while the new code only uses 4MB - one 2MB page for each node.
      
      [ Impact: much better space efficiency on NUMA machines ]
      Signed-off-by: default avatarTejun Heo <tj@kernel.org>
      Cc: Ingo Molnar <mingo@elte.hu>
      Cc: Jan Beulich <JBeulich@novell.com>
      Cc: Andi Kleen <andi@firstfloor.org>
      Cc: David Miller <davem@davemloft.net>
      a530b795
    • Tejun Heo's avatar
      percpu: allow non-linear / sparse cpu -> unit mapping · 2f39e637
      Tejun Heo authored
      Currently cpu and unit are always identity mapped.  To allow more
      efficient large page support on NUMA and lazy allocation for possible
      but offline cpus, cpu -> unit mapping needs to be non-linear and/or
      sparse.  This can be easily implemented by adding a cpu -> unit
      mapping array and using it whenever looking up the matching unit for a
      cpu.
      
      The only unusal conversion is in pcpu_chunk_addr_search().  The passed
      in address is unit0 based and unit0 might not be in use so it needs to
      be converted to address of an in-use unit.  This is easily done by
      adding the unit offset for the current processor.
      
      [ Impact: allows non-linear/sparse cpu -> unit mapping, no visible change yet ]
      Signed-off-by: default avatarTejun Heo <tj@kernel.org>
      Cc: Ingo Molnar <mingo@elte.hu>
      Cc: David Miller <davem@davemloft.net>
      2f39e637
    • Tejun Heo's avatar
      percpu: drop pcpu_chunk->page[] · ce3141a2
      Tejun Heo authored
      percpu core doesn't need to tack all the allocated pages.  It needs to
      know whether certain pages are populated and a way to reverse map
      address to page when freeing.  This patch drops pcpu_chunk->page[] and
      use populated bitmap and vmalloc_to_page() lookup instead.  Using
      vmalloc_to_page() exclusively is also possible but complicates first
      chunk handling, inflates cache footprint and prevents non-standard
      memory allocation for percpu memory.
      
      pcpu_chunk->page[] was used to track each page's allocation and
      allowed asymmetric population which happens during failure path;
      however, with single bitmap for all units, this is no longer possible.
      Bite the bullet and rewrite (de)populate functions so that things are
      done in clearly separated steps such that asymmetric population
      doesn't happen.  This makes the (de)population process much more
      modular and will also ease implementing non-standard memory usage in
      the future (e.g. large pages).
      
      This makes @get_page_fn parameter to pcpu_setup_first_chunk()
      unnecessary.  The parameter is dropped and all first chunk helpers are
      updated accordingly.  Please note that despite the volume most changes
      to first chunk helpers are symbol renames for variables which don't
      need to be referenced outside of the helper anymore.
      
      This change reduces memory usage and cache footprint of pcpu_chunk.
      Now only #unit_pages bits are necessary per chunk.
      
      [ Impact: reduced memory usage and cache footprint for bookkeeping ]
      Signed-off-by: default avatarTejun Heo <tj@kernel.org>
      Cc: Ingo Molnar <mingo@elte.hu>
      Cc: David Miller <davem@davemloft.net>
      ce3141a2
    • Tejun Heo's avatar
      percpu: reorder a few functions in mm/percpu.c · c8a51be4
      Tejun Heo authored
      (de)populate functions are about to be reimplemented to drop
      pcpu_chunk->page array.  Move a few functions so that the rewrite
      patch doesn't have code movement making it more difficult to read.
      
      [ Impact: code movement ]
      Signed-off-by: default avatarTejun Heo <tj@kernel.org>
      Cc: Ingo Molnar <mingo@elte.hu>
      c8a51be4
    • Tejun Heo's avatar
      percpu: simplify pcpu_setup_first_chunk() · 38a6be52
      Tejun Heo authored
      Now that all first chunk allocator helpers allocate and map the first
      chunk themselves, there's no need to have optional default alloc/map
      in pcpu_setup_first_chunk().  Drop @populate_pte_fn and only leave
      @dyn_size optional and make all other params mandatory.
      
      This makes it much easier to follow what pcpu_setup_first_chunk() is
      doing and what actual differences tweaking each parameter results in.
      
      [ Impact: drop unused code path ]
      Signed-off-by: default avatarTejun Heo <tj@kernel.org>
      Cc: Ingo Molnar <mingo@elte.hu>
      38a6be52
    • Tejun Heo's avatar
      x86,percpu: generalize lpage first chunk allocator · 8c4bfc6e
      Tejun Heo authored
      Generalize and move x86 setup_pcpu_lpage() into
      pcpu_lpage_first_chunk().  setup_pcpu_lpage() now is a simple wrapper
      around the generalized version.  Other than taking size parameters and
      using arch supplied callbacks to allocate/free/map memory,
      pcpu_lpage_first_chunk() is identical to the original implementation.
      
      This simplifies arch code and will help converting more archs to
      dynamic percpu allocator.
      
      While at it, factor out pcpu_calc_fc_sizes() which is common to
      pcpu_embed_first_chunk() and pcpu_lpage_first_chunk().
      
      [ Impact: code reorganization and generalization ]
      Signed-off-by: default avatarTejun Heo <tj@kernel.org>
      Cc: Ingo Molnar <mingo@elte.hu>
      8c4bfc6e
    • Tejun Heo's avatar
      percpu: make 4k first chunk allocator map memory · 8f05a6a6
      Tejun Heo authored
      At first, percpu first chunk was always setup page-by-page by the
      generic code.  To add other allocators, different parts of the generic
      initialization was made optional.  Now we have three allocators -
      embed, remap and 4k.  embed and remap fully handle allocation and
      mapping of the first chunk while 4k still depends on generic code for
      those.  This makes the generic alloc/map paths specifci to 4k and
      makes the code unnecessary complicated with optional generic
      behaviors.
      
      This patch makes the 4k allocator to allocate and map memory directly
      instead of depending on the generic code.  The only outside visible
      change is that now dynamic area in the first chunk is allocated
      up-front instead of on-demand.  This doesn't make any meaningful
      difference as the area is minimal (usually less than a page, just
      enough to fill the alignment) on 4k allocator.  Plus, dynamic area in
      the first chunk usually gets fully used anyway.
      
      This will allow simplification of pcpu_setpu_first_chunk() and removal
      of chunk->page array.
      
      [ Impact: no outside visible change other than up-front allocation of dyn area ]
      Signed-off-by: default avatarTejun Heo <tj@kernel.org>
      Cc: Ingo Molnar <mingo@elte.hu>
      8f05a6a6
    • Tejun Heo's avatar
      x86,percpu: generalize 4k first chunk allocator · d4b95f80
      Tejun Heo authored
      Generalize and move x86 setup_pcpu_4k() into pcpu_4k_first_chunk().
      setup_pcpu_4k() now is a simple wrapper around the generalized
      version.  Other than taking size parameters and using arch supplied
      callbacks to allocate/free memory, pcpu_4k_first_chunk() is identical
      to the original implementation.
      
      This simplifies arch code and will help converting more archs to
      dynamic percpu allocator.
      
      While at it, s/pcpu_populate_pte_fn_t/pcpu_fc_populate_pte_fn_t/ for
      consistency.
      
      [ Impact: code reorganization and generalization ]
      Signed-off-by: default avatarTejun Heo <tj@kernel.org>
      Cc: Ingo Molnar <mingo@elte.hu>
      d4b95f80
    • Tejun Heo's avatar
      percpu: drop @unit_size from embed first chunk allocator · 788e5abc
      Tejun Heo authored
      The only extra feature @unit_size provides is making dead space at the
      end of the first chunk which doesn't have any valid usecase.  Drop the
      parameter.  This will increase consistency with generalized 4k
      allocator.
      
      James Bottomley spotted missing conversion for the default
      setup_per_cpu_areas() which caused build breakage on all arcsh which
      use it.
      
      [ Impact: drop unused code path ]
      Signed-off-by: default avatarTejun Heo <tj@kernel.org>
      Cc: James Bottomley <James.Bottomley@HansenPartnership.com>
      Cc: Ingo Molnar <mingo@elte.hu>
      788e5abc
    • Tejun Heo's avatar
      x86: make pcpu_chunk_addr_search() matching stricter · 79ba6ac8
      Tejun Heo authored
      The @addr passed into pcpu_chunk_addr_search() is unit0 based address
      and thus should be matched inside unit0 area.  Currently, when it uses
      chunk size when determining whether the address falls in the first
      chunk.  Addresses in unitN where N>0 shouldn't be passed in anyway, so
      this doesn't cause any malfunction but fix it for consistency.
      
      [ Impact: mostly cleanup ]
      Signed-off-by: default avatarTejun Heo <tj@kernel.org>
      Cc: Ingo Molnar <mingo@elte.hu>
      79ba6ac8
    • Tejun Heo's avatar
      Merge branch 'master' into for-next · c43768cb
      Tejun Heo authored
      Pull linus#master to merge PER_CPU_DEF_ATTRIBUTES and alpha build fix
      changes.  As alpha in percpu tree uses 'weak' attribute instead of
      inline assembly, there's no need for __used attribute.
      
      Conflicts:
      	arch/alpha/include/asm/percpu.h
      	arch/mn10300/kernel/vmlinux.lds.S
      	include/linux/percpu-defs.h
      c43768cb
  3. 02 Jul, 2009 18 commits
  4. 01 Jul, 2009 9 commits