1. 24 Feb, 2009 7 commits
    • Tejun Heo's avatar
      percpu: give more latitude to arch specific first chunk initialization · 8d408b4b
      Tejun Heo authored
      Impact: more latitude for first percpu chunk allocation
      
      The first percpu chunk serves the kernel static percpu area and may or
      may not contain extra room for further dynamic allocation.
      Initialization of the first chunk needs to be done before normal
      memory allocation service is up, so it has its own init path -
      pcpu_setup_static().
      
      It seems archs need more latitude while initializing the first chunk
      for example to take advantage of large page mapping.  This patch makes
      the following changes to allow this.
      
      * Define PERCPU_DYNAMIC_RESERVE to give arch hint about how much space
        to reserve in the first chunk for further dynamic allocation.
      
      * Rename pcpu_setup_static() to pcpu_setup_first_chunk().
      
      * Make pcpu_setup_first_chunk() much more flexible by fetching page
        pointer by callback and adding optional @unit_size, @free_size and
        @base_addr arguments which allow archs to selectively part of chunk
        initialization to their likings.
      Signed-off-by: default avatarTejun Heo <tj@kernel.org>
      8d408b4b
    • Tejun Heo's avatar
      percpu: remove unit_size power-of-2 restriction · d9b55eeb
      Tejun Heo authored
      Impact: allow unit_size to be arbitrary multiple of PAGE_SIZE
      
      In dynamic percpu allocator, there is no reason the unit size should
      be power of two.  Remove the restriction.
      
      As non-power-of-two unit size means that empty chunks fall into the
      same slot index as lightly occupied chunks which is bad for reclaming.
      Reserve an extra slot for empty chunks.
      Signed-off-by: default avatarTejun Heo <tj@kernel.org>
      d9b55eeb
    • Tejun Heo's avatar
      x86: update populate_extra_pte() and add populate_extra_pmd() · 458a3e64
      Tejun Heo authored
      Impact: minor change to populate_extra_pte() and addition of pmd flavor
      
      Update populate_extra_pte() to return pointer to the pte_t for the
      specified address and add populate_extra_pmd() which only populates
      till the pmd and returns pointer to the pmd entry for the address.
      
      For 64bit, pud/pmd/pte fill functions are separated out from
      set_pte_vaddr[_pud]() and used for set_pte_vaddr[_pud]() and
      populate_extra_{pte|pmd}().
      Signed-off-by: default avatarTejun Heo <tj@kernel.org>
      458a3e64
    • Tejun Heo's avatar
      vmalloc: add @align to vm_area_register_early() · c0c0a293
      Tejun Heo authored
      Impact: allow larger alignment for early vmalloc area allocation
      
      Some early vmalloc users might want larger alignment, for example, for
      custom large page mapping.  Add @align to vm_area_register_early().
      While at it, drop docbook comment on non-existent @size.
      Signed-off-by: default avatarTejun Heo <tj@kernel.org>
      Cc: Nick Piggin <nickpiggin@yahoo.com.au>
      Cc: Ivan Kokshaysky <ink@jurassic.park.msu.ru>
      c0c0a293
    • Tejun Heo's avatar
      bootmem: reorder interface functions and add a missing one · 2d0aae41
      Tejun Heo authored
      Impact: cleanup and addition of missing interface wrapper
      
      The interface functions in bootmem.h was ordered in not so orderly
      manner.  Reorder them such that
      
      * functions allocating the same area group together -
        ie. alloc_bootmem group and alloc_bootmem_low group.
      
      * functions w/o node parameter come before the ones w/ node parameter.
      
      * nopanic variants are immediately below their panicky counterparts.
      
      While at it, add alloc_bootmem_pages_node_nopanic() which was missing.
      Signed-off-by: default avatarTejun Heo <tj@kernel.org>
      Cc: Johannes Weiner <hannes@saeurebad.de>
      2d0aae41
    • Tejun Heo's avatar
      bootmem: clean up arch-specific bootmem wrapping · c1329375
      Tejun Heo authored
      Impact: cleaner and consistent bootmem wrapping
      
      By setting CONFIG_HAVE_ARCH_BOOTMEM_NODE, archs can define
      arch-specific wrappers for bootmem allocation.  However, this is done
      a bit strangely in that only the high level convenience macros can be
      changed while lower level, but still exported, interface functions
      can't be wrapped.  This not only is messy but also leads to strange
      situation where alloc_bootmem() does what the arch wants it to do but
      the equivalent __alloc_bootmem() call doesn't although they should be
      able to be used interchangeably.
      
      This patch updates bootmem such that archs can override / wrap the
      backend function - alloc_bootmem_core() instead of the highlevel
      interface functions to allow simpler and consistent wrapping.  Also,
      HAVE_ARCH_BOOTMEM_NODE is renamed to HAVE_ARCH_BOOTMEM.
      Signed-off-by: default avatarTejun Heo <tj@kernel.org>
      Cc: Johannes Weiner <hannes@saeurebad.de>
      c1329375
    • Tejun Heo's avatar
      percpu: fix pcpu_chunk_struct_size · cb83b42e
      Tejun Heo authored
      Impact: fix short allocation leading to memory corruption
      
      While dropping rvalue wrapping macros around global parameters,
      pcpu_chunk_struct_size was set incorrectly resulting in shorter page
      pointer array.  Fix it.
      Signed-off-by: default avatarTejun Heo <tj@kernel.org>
      cb83b42e
  2. 21 Feb, 2009 1 commit
    • Tejun Heo's avatar
      percpu: clean up size usage · cae3aeb8
      Tejun Heo authored
      Andrew was concerned about the unit of variables named or have suffix
      size.  Every usage in percpu allocator is in bytes but make it super
      clear by adding comments.
      
      While at it, make pcpu_depopulate_chunk() take int @off and @size like
      everyone else.
      Signed-off-by: default avatarTejun Heo <tj@kernel.org>
      Cc: Andrew Morton <akpm@linux-foundation.org>
      cae3aeb8
  3. 20 Feb, 2009 10 commits
    • Tejun Heo's avatar
      x86: convert to the new dynamic percpu allocator · 11124411
      Tejun Heo authored
      Impact: use new dynamic allocator, unified access to static/dynamic
              percpu memory
      
      Convert to the new dynamic percpu allocator.
      
      * implement populate_extra_pte() for both 32 and 64
      * update setup_per_cpu_areas() to use pcpu_setup_static()
      * define __addr_to_pcpu_ptr() and __pcpu_ptr_to_addr()
      * define config HAVE_DYNAMIC_PER_CPU_AREA
      Signed-off-by: default avatarTejun Heo <tj@kernel.org>
      11124411
    • Tejun Heo's avatar
      percpu: implement new dynamic percpu allocator · fbf59bc9
      Tejun Heo authored
      Impact: new scalable dynamic percpu allocator which allows dynamic
              percpu areas to be accessed the same way as static ones
      
      Implement scalable dynamic percpu allocator which can be used for both
      static and dynamic percpu areas.  This will allow static and dynamic
      areas to share faster direct access methods.  This feature is optional
      and enabled only when CONFIG_HAVE_DYNAMIC_PER_CPU_AREA is defined by
      arch.  Please read comment on top of mm/percpu.c for details.
      Signed-off-by: default avatarTejun Heo <tj@kernel.org>
      Cc: Andrew Morton <akpm@linux-foundation.org>
      fbf59bc9
    • Tejun Heo's avatar
      vmalloc: add un/map_kernel_range_noflush() · 8fc48985
      Tejun Heo authored
      Impact: two more public map/unmap functions
      
      Implement map_kernel_range_noflush() and unmap_kernel_range_noflush().
      These functions respectively map and unmap address range in kernel VM
      area but doesn't do any vcache or tlb flushing.  These will be used by
      new percpu allocator.
      Signed-off-by: default avatarTejun Heo <tj@kernel.org>
      Cc: Nick Piggin <nickpiggin@yahoo.com.au>
      8fc48985
    • Tejun Heo's avatar
      vmalloc: implement vm_area_register_early() · f0aa6617
      Tejun Heo authored
      Impact: allow multiple early vm areas
      
      There are places where kernel VM area needs to be allocated before
      vmalloc is initialized.  This is done by allocating static vm_struct,
      initializing several fields and linking it to vmlist and later vmalloc
      initialization picking up these from vmlist.  This is currently done
      manually and if there's more than one such areas, there's no defined
      way to arbitrate who gets which address.
      
      This patch implements vm_area_register_early(), which takes vm_area
      struct with flags and size initialized, assigns address to it and puts
      it on the vmlist.  This way, multiple early vm areas can determine
      which addresses they should use.  The only current user - alpha mm
      init - is converted to use it.
      Signed-off-by: default avatarTejun Heo <tj@kernel.org>
      f0aa6617
    • Tejun Heo's avatar
      percpu: kill percpu_alloc() and friends · f2a8205c
      Tejun Heo authored
      Impact: kill unused functions
      
      percpu_alloc() and its friends never saw much action.  It was supposed
      to replace the cpu-mask unaware __alloc_percpu() but it never happened
      and in fact __percpu_alloc_mask() itself never really grew proper
      up/down handling interface either (no exported interface for
      populate/depopulate).
      
      percpu allocation is about to go through major reimplementation and
      there's no reason to carry this unused interface around.  Replace it
      with __alloc_percpu() and free_percpu().
      Signed-off-by: default avatarTejun Heo <tj@kernel.org>
      f2a8205c
    • Rusty Russell's avatar
      alloc_percpu: add align argument to __alloc_percpu. · 313e458f
      Rusty Russell authored
      This prepares for a real __alloc_percpu, by adding an alignment argument.
      Only one place uses __alloc_percpu directly, and that's for a string.
      
      tj: af_inet also uses __alloc_percpu(), update it.
      Signed-off-by: default avatarRusty Russell <rusty@rustcorp.com.au>
      Cc: Christoph Lameter <cl@linux-foundation.org>
      Cc: Jens Axboe <axboe@kernel.dk>
      313e458f
    • Rusty Russell's avatar
      alloc_percpu: change percpu_ptr to per_cpu_ptr · b36128c8
      Rusty Russell authored
      Impact: cleanup
      
      There are two allocated per-cpu accessor macros with almost identical
      spelling.  The original and far more popular is per_cpu_ptr (44
      files), so change over the other 4 files.
      
      tj: kill percpu_ptr() and update UP too
      Signed-off-by: default avatarRusty Russell <rusty@rustcorp.com.au>
      Cc: mingo@redhat.com
      Cc: lenb@kernel.org
      Cc: cpufreq@vger.kernel.org
      Signed-off-by: default avatarTejun Heo <tj@kernel.org>
      b36128c8
    • Tejun Heo's avatar
      module: reorder module pcpu related functions · 6b588c18
      Tejun Heo authored
      Impact: cleanup
      
      Move percpu_modinit() upwards.  This is to ease further changes.
      Signed-off-by: default avatarTejun Heo <tj@kernel.org>
      6b588c18
    • Tejun Heo's avatar
      vmalloc: call flush_cache_vunmap() from unmap_kernel_range() · 73426952
      Tejun Heo authored
      Impact: proper vcache flush on unmap_kernel_range()
      
      flush_cache_vunmap() should be called before pages are unmapped.  Add
      a call to it in unmap_kernel_range().
      Signed-off-by: default avatarTejun Heo <tj@kernel.org>
      73426952
    • Lai Jiangshan's avatar
      x86: use percpu data for 4k hardirq and softirq stacks · 42f8faec
      Lai Jiangshan authored
      Impact: economize memory for large NR_CPUS
      
      percpu data is setup earlier than irq, we can use percpu data
      to economize memory.
      Signed-off-by: default avatarLai Jiangshan <laijs@cn.fujitsu.com>
      Signed-off-by: default avatarTejun Heo <tj@kernel.org>
      42f8faec
  4. 12 Feb, 2009 2 commits
  5. 11 Feb, 2009 7 commits
  6. 09 Feb, 2009 13 commits
    • Tejun Heo's avatar
      x86: implement x86_32 stack protector · 60a5317f
      Tejun Heo authored
      Impact: stack protector for x86_32
      
      Implement stack protector for x86_32.  GDT entry 28 is used for it.
      It's set to point to stack_canary-20 and have the length of 24 bytes.
      CONFIG_CC_STACKPROTECTOR turns off CONFIG_X86_32_LAZY_GS and sets %gs
      to the stack canary segment on entry.  As %gs is otherwise unused by
      the kernel, the canary can be anywhere.  It's defined as a percpu
      variable.
      
      x86_32 exception handlers take register frame on stack directly as
      struct pt_regs.  With -fstack-protector turned on, gcc copies the
      whole structure after the stack canary and (of course) doesn't copy
      back on return thus losing all changed.  For now, -fno-stack-protector
      is added to all files which contain those functions.  We definitely
      need something better.
      Signed-off-by: default avatarTejun Heo <tj@kernel.org>
      Signed-off-by: default avatarIngo Molnar <mingo@elte.hu>
      60a5317f
    • Tejun Heo's avatar
      x86: make lazy %gs optional on x86_32 · ccbeed3a
      Tejun Heo authored
      Impact: pt_regs changed, lazy gs handling made optional, add slight
              overhead to SAVE_ALL, simplifies error_code path a bit
      
      On x86_32, %gs hasn't been used by kernel and handled lazily.  pt_regs
      doesn't have place for it and gs is saved/loaded only when necessary.
      In preparation for stack protector support, this patch makes lazy %gs
      handling optional by doing the followings.
      
      * Add CONFIG_X86_32_LAZY_GS and place for gs in pt_regs.
      
      * Save and restore %gs along with other registers in entry_32.S unless
        LAZY_GS.  Note that this unfortunately adds "pushl $0" on SAVE_ALL
        even when LAZY_GS.  However, it adds no overhead to common exit path
        and simplifies entry path with error code.
      
      * Define different user_gs accessors depending on LAZY_GS and add
        lazy_save_gs() and lazy_load_gs() which are noop if !LAZY_GS.  The
        lazy_*_gs() ops are used to save, load and clear %gs lazily.
      
      * Define ELF_CORE_COPY_KERNEL_REGS() which always read %gs directly.
      
      xen and lguest changes need to be verified.
      Signed-off-by: default avatarTejun Heo <tj@kernel.org>
      Cc: Jeremy Fitzhardinge <jeremy@xensource.com>
      Cc: Rusty Russell <rusty@rustcorp.com.au>
      Signed-off-by: default avatarIngo Molnar <mingo@elte.hu>
      ccbeed3a
    • Tejun Heo's avatar
      x86: add %gs accessors for x86_32 · d9a89a26
      Tejun Heo authored
      Impact: cleanup
      
      On x86_32, %gs is handled lazily.  It's not saved and restored on
      kernel entry/exit but only when necessary which usually is during task
      switch but there are few other places.  Currently, it's done by
      calling savesegment() and loadsegment() explicitly.  Define
      get_user_gs(), set_user_gs() and task_user_gs() and use them instead.
      
      While at it, clean up register access macros in signal.c.
      
      This cleans up code a bit and will help future changes.
      Signed-off-by: default avatarTejun Heo <tj@kernel.org>
      Signed-off-by: default avatarIngo Molnar <mingo@elte.hu>
      d9a89a26
    • Tejun Heo's avatar
      x86: use asm .macro instead of cpp #define in entry_32.S · f0d96110
      Tejun Heo authored
      Impact: cleanup
      
      Use .macro instead of cpp #define where approriate.  This cleans up
      code and will ease future changes.
      Signed-off-by: default avatarTejun Heo <tj@kernel.org>
      Signed-off-by: default avatarIngo Molnar <mingo@elte.hu>
      f0d96110
    • Tejun Heo's avatar
      x86: no stack protector for vdso · d627ded5
      Tejun Heo authored
      Impact: avoid crash on vsyscall
      Signed-off-by: default avatarTejun Heo <tj@kernel.org>
      Signed-off-by: default avatarIngo Molnar <mingo@elte.hu>
      d627ded5
    • Tejun Heo's avatar
      stackprotector: update make rules · 5d707e9c
      Tejun Heo authored
      Impact: no default -fno-stack-protector if stackp is enabled, cleanup
      
      Stackprotector make rules had the following problems.
      
      * cc support test and warning are scattered across makefile and
        kernel/panic.c.
      
      * -fno-stack-protector was always added regardless of configuration.
      
      Update such that cc support test and warning are contained in makefile
      and -fno-stack-protector is added iff stackp is turned off.  While at
      it, prepare for 32bit support.
      Signed-off-by: default avatarTejun Heo <tj@kernel.org>
      Signed-off-by: default avatarIngo Molnar <mingo@elte.hu>
      5d707e9c
    • Tejun Heo's avatar
      x86: stackprotector.h misc update · 76397f72
      Tejun Heo authored
      Impact: misc udpate
      
      * wrap content with CONFIG_CC_STACK_PROTECTOR so that other arch files
        can include it directly
      
      * add missing includes
      
      This will help future changes.
      Signed-off-by: default avatarTejun Heo <tj@kernel.org>
      Signed-off-by: default avatarIngo Molnar <mingo@elte.hu>
      76397f72
    • Tejun Heo's avatar
      elf: add ELF_CORE_COPY_KERNEL_REGS() · 6cd61c0b
      Tejun Heo authored
      ELF core dump is used for both user land core dump and kernel crash
      dump.  Depending on architecture, register might need to be accessed
      differently for userland and kernel.  Allow architectures to define
      ELF_CORE_COPY_KERNEL_REGS() and use different operation for kernel
      register dump.
      Signed-off-by: default avatarTejun Heo <tj@kernel.org>
      Signed-off-by: default avatarIngo Molnar <mingo@elte.hu>
      6cd61c0b
    • Ingo Molnar's avatar
      Merge branch 'x86/urgent' into core/percpu · 92e2d508
      Ingo Molnar authored
      Conflicts:
      	arch/x86/kernel/acpi/boot.c
      92e2d508
    • Ingo Molnar's avatar
      Merge branch 'x86/uaccess' into core/percpu · 5d96218b
      Ingo Molnar authored
      5d96218b
    • Tejun Heo's avatar
      x86: fix math_emu register frame access · d315760f
      Tejun Heo authored
      do_device_not_available() is the handler for #NM and it declares that
      it takes a unsigned long and calls math_emu(), which takes a long
      argument and surprisingly expects the stack frame starting at the zero
      argument would match struct math_emu_info, which isn't true regardless
      of configuration in the current code.
      
      This patch makes do_device_not_available() take struct pt_regs like
      other exception handlers and initialize struct math_emu_info with
      pointer to it and pass pointer to the math_emu_info to math_emulate()
      like normal C functions do.  This way, unless gcc makes a copy of
      struct pt_regs in do_device_not_available(), the register frame is
      correctly accessed regardless of kernel configuration or compiler
      used.
      
      This doesn't fix all math_emu problems but it at least gets it
      somewhat working.
      Signed-off-by: default avatarTejun Heo <tj@kernel.org>
      Signed-off-by: default avatarIngo Molnar <mingo@elte.hu>
      d315760f
    • Ingo Molnar's avatar
      Merge commit 'v2.6.29-rc4' into core/percpu · 249d51b5
      Ingo Molnar authored
      Conflicts:
      	arch/x86/mach-voyager/voyager_smp.c
      	arch/x86/mm/fault.c
      249d51b5
    • Tejun Heo's avatar
      x86: math_emu info cleanup · ae6af41f
      Tejun Heo authored
      Impact: cleanup
      
      * Come on, struct info?  s/struct info/struct math_emu_info/
      
      * Use struct pt_regs and kernel_vm86_regs instead of defining its own
        register frame structure.
      Signed-off-by: default avatarTejun Heo <tj@kernel.org>
      Signed-off-by: default avatarIngo Molnar <mingo@elte.hu>
      ae6af41f