1. 23 Aug, 2009 8 commits
    • Paul E. McKenney's avatar
      rcu: Merge preemptable-RCU functionality into hierarchical RCU · f41d911f
      Paul E. McKenney authored
      Create a kernel/rcutree_plugin.h file that contains definitions
      for preemptable RCU (or, under the #else branch of the #ifdef,
      empty definitions for the classic non-preemptable semantics).
      These definitions fit into plugins defined in kernel/rcutree.c
      for this purpose.
      
      This variant of preemptable RCU uses a new algorithm whose
      read-side expense is roughly that of classic hierarchical RCU
      under CONFIG_PREEMPT. This new algorithm's update-side expense
      is similar to that of classic hierarchical RCU, and, in absence
      of read-side preemption or blocking, is exactly that of classic
      hierarchical RCU.  Perhaps more important, this new algorithm
      has a much simpler implementation, saving well over 1,000 lines
      of code compared to mainline's implementation of preemptable
      RCU, which will hopefully be retired in favor of this new
      algorithm.
      
      The simplifications are obtained by maintaining per-task
      nesting state for running tasks, and using a simple
      lock-protected algorithm to handle accounting when tasks block
      within RCU read-side critical sections, making use of lessons
      learned while creating numerous user-level RCU implementations
      over the past 18 months.
      Signed-off-by: default avatarPaul E. McKenney <paulmck@linux.vnet.ibm.com>
      Cc: laijs@cn.fujitsu.com
      Cc: dipankar@in.ibm.com
      Cc: akpm@linux-foundation.org
      Cc: mathieu.desnoyers@polymtl.ca
      Cc: josht@linux.vnet.ibm.com
      Cc: dvhltc@us.ibm.com
      Cc: niv@us.ibm.com
      Cc: peterz@infradead.org
      Cc: rostedt@goodmis.org
      LKML-Reference: <12509746134003-git-send-email->
      Signed-off-by: default avatarIngo Molnar <mingo@elte.hu>
      f41d911f
    • Paul E. McKenney's avatar
      rcu: Simplify rcu_pending()/rcu_check_callbacks() API · a157229c
      Paul E. McKenney authored
      All calls from outside RCU are of the form:
      
      	if (rcu_pending(cpu))
      		rcu_check_callbacks(cpu, user);
      
      This is silly, instead we put a call to rcu_pending() in
      rcu_check_callbacks(), and then make the outside calls be to
      rcu_check_callbacks().  This cuts down on the code a bit and
      also gives the compiler a better chance of optimizing.
      Signed-off-by: default avatarPaul E. McKenney <paulmck@linux.vnet.ibm.com>
      Cc: laijs@cn.fujitsu.com
      Cc: dipankar@in.ibm.com
      Cc: akpm@linux-foundation.org
      Cc: mathieu.desnoyers@polymtl.ca
      Cc: josht@linux.vnet.ibm.com
      Cc: dvhltc@us.ibm.com
      Cc: niv@us.ibm.com
      Cc: peterz@infradead.org
      Cc: rostedt@goodmis.org
      LKML-Reference: <125097461311-git-send-email->
      Signed-off-by: default avatarIngo Molnar <mingo@elte.hu>
      a157229c
    • Paul E. McKenney's avatar
      rcu: Use debugfs_remove_recursive() simplify code. · 22f00b69
      Paul E. McKenney authored
      Suggested by Josh Triplett.
      Signed-off-by: default avatarPaul E. McKenney <paulmck@linux.vnet.ibm.com>
      Cc: laijs@cn.fujitsu.com
      Cc: dipankar@in.ibm.com
      Cc: akpm@linux-foundation.org
      Cc: mathieu.desnoyers@polymtl.ca
      Cc: josht@linux.vnet.ibm.com
      Cc: dvhltc@us.ibm.com
      Cc: niv@us.ibm.com
      Cc: peterz@infradead.org
      Cc: rostedt@goodmis.org
      Cc: "Paul E. McKenney" <paulmck@linux.vnet.ibm.com>
      LKML-Reference: <12509746132173-git-send-email->
      Signed-off-by: default avatarIngo Molnar <mingo@elte.hu>
      22f00b69
    • Paul E. McKenney's avatar
      rcu: Merge per-RCU-flavor initialization into pre-existing macro · 65cf8f86
      Paul E. McKenney authored
      Rename the RCU_DATA_PTR_INIT() macro to RCU_INIT_FLAVOR() and
      make it do the rcu_init_one() and rcu_boot_init_percpu_data()
      calls.  Merge the loop that was in the original macro with the
      loops that were in __rcu_init().
      Signed-off-by: default avatarPaul E. McKenney <paulmck@linux.vnet.ibm.com>
      Cc: laijs@cn.fujitsu.com
      Cc: dipankar@in.ibm.com
      Cc: akpm@linux-foundation.org
      Cc: mathieu.desnoyers@polymtl.ca
      Cc: josht@linux.vnet.ibm.com
      Cc: dvhltc@us.ibm.com
      Cc: niv@us.ibm.com
      Cc: peterz@infradead.org
      Cc: rostedt@goodmis.org
      LKML-Reference: <12509746133916-git-send-email->
      Signed-off-by: default avatarIngo Molnar <mingo@elte.hu>
      65cf8f86
    • Paul E. McKenney's avatar
      rcu: Fix online/offline indication for rcudata.csv trace file · 5699ed8f
      Paul E. McKenney authored
      The heading said "Online?", but the column had "Y" for offline
      CPUs and "N" for online CPUs.  Swap the "Y" and "N" to match
      the heading.
      Signed-off-by: default avatarPaul E. McKenney <paulmck@linux.vnet.ibm.com>
      Cc: laijs@cn.fujitsu.com
      Cc: dipankar@in.ibm.com
      Cc: akpm@linux-foundation.org
      Cc: mathieu.desnoyers@polymtl.ca
      Cc: josht@linux.vnet.ibm.com
      Cc: dvhltc@us.ibm.com
      Cc: niv@us.ibm.com
      Cc: peterz@infradead.org
      Cc: rostedt@goodmis.org
      LKML-Reference: <12509746132841-git-send-email->
      Signed-off-by: default avatarIngo Molnar <mingo@elte.hu>
      5699ed8f
    • Paul E. McKenney's avatar
      rcu: Consolidate sparse and lockdep declarations in include/linux/rcupdate.h · bc33f24b
      Paul E. McKenney authored
      Signed-off-by: default avatarPaul E. McKenney <paulmck@linux.vnet.ibm.com>
      Cc: laijs@cn.fujitsu.com
      Cc: dipankar@in.ibm.com
      Cc: akpm@linux-foundation.org
      Cc: mathieu.desnoyers@polymtl.ca
      Cc: josht@linux.vnet.ibm.com
      Cc: dvhltc@us.ibm.com
      Cc: niv@us.ibm.com
      Cc: peterz@infradead.org
      Cc: rostedt@goodmis.org
      LKML-Reference: <12509746132349-git-send-email->
      Signed-off-by: default avatarIngo Molnar <mingo@elte.hu>
      bc33f24b
    • Paul E. McKenney's avatar
      rcu: Renamings to increase RCU clarity · d6714c22
      Paul E. McKenney authored
      Make RCU-sched, RCU-bh, and RCU-preempt be underlying
      implementations, with "RCU" defined in terms of one of the
      three.  Update the outdated rcu_qsctr_inc() names, as these
      functions no longer increment anything.
      Signed-off-by: default avatarPaul E. McKenney <paulmck@linux.vnet.ibm.com>
      Cc: laijs@cn.fujitsu.com
      Cc: dipankar@in.ibm.com
      Cc: akpm@linux-foundation.org
      Cc: mathieu.desnoyers@polymtl.ca
      Cc: josht@linux.vnet.ibm.com
      Cc: dvhltc@us.ibm.com
      Cc: niv@us.ibm.com
      Cc: peterz@infradead.org
      Cc: rostedt@goodmis.org
      LKML-Reference: <12509746132696-git-send-email->
      Signed-off-by: default avatarIngo Molnar <mingo@elte.hu>
      d6714c22
    • Paul E. McKenney's avatar
      rcu: Move private definitions from include/linux/rcutree.h to kernel/rcutree.h · 9f77da9f
      Paul E. McKenney authored
      Some information hiding that makes it easier to merge
      preemptability into rcutree without descending into #include
      hell.
      Signed-off-by: default avatarPaul E. McKenney <paulmck@linux.vnet.ibm.com>
      Cc: laijs@cn.fujitsu.com
      Cc: dipankar@in.ibm.com
      Cc: akpm@linux-foundation.org
      Cc: mathieu.desnoyers@polymtl.ca
      Cc: josht@linux.vnet.ibm.com
      Cc: dvhltc@us.ibm.com
      Cc: niv@us.ibm.com
      Cc: peterz@infradead.org
      Cc: rostedt@goodmis.org
      LKML-Reference: <1250974613373-git-send-email->
      Signed-off-by: default avatarIngo Molnar <mingo@elte.hu>
      9f77da9f
  2. 22 Aug, 2009 1 commit
    • Paul E. McKenney's avatar
      rcu: Expunge lingering references to CONFIG_CLASSIC_RCU, optimize on !SMP · b560d8ad
      Paul E. McKenney authored
      A couple of references to CONFIG_CLASSIC_RCU have survived.
      Although these are harmless, it is past time for them to go.
      The one in hardirq.h is strictly a readability problem.
      
      The two in pagemap.h appear to disable a !SMP performance
      optimization (which this patch re-enables).
      
      This does raise the issue as to whether pagemap.h should really
      be referring to the CPU implementation.  Long term, I intend to
      make the RCU implementation driven by CONFIG_PREEMPT, at which
      point these should change from defined(CONFIG_TREE_RCU) to
      !defined(CONFIG_PREEMPT). In the meantime, is there something
      else that could be done in pagemap.h?
      Signed-off-by: default avatarPaul E. McKenney <paulmck@linux.vnet.ibm.com>
      Cc: laijs@cn.fujitsu.com
      Cc: dipankar@in.ibm.com
      Cc: akpm@linux-foundation.org
      Cc: mathieu.desnoyers@polymtl.ca
      Cc: josht@linux.vnet.ibm.com
      Cc: dvhltc@us.ibm.com
      Cc: niv@us.ibm.com
      Cc: peterz@infradead.org
      Cc: rostedt@goodmis.org
      LKML-Reference: <20090822050851.GA8414@linux.vnet.ibm.com>
      Signed-off-by: default avatarIngo Molnar <mingo@elte.hu>
      b560d8ad
  3. 19 Aug, 2009 1 commit
    • Paul E. McKenney's avatar
      rcu: Delay rcu_barrier() wait until beginning of next CPU-hotunplug operation. · 1423cc03
      Paul E. McKenney authored
      Ingo Molnar reported this lockup:
      
       [  200.380003] Hangcheck: hangcheck value past margin!
       [  248.192003] INFO: task S99local:2974 blocked for more than 120 seconds.
       [  248.194532] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
       [  248.202330] S99local      D 0000000c  6256  2974   2687 0x00000000
       [  248.208929]  9c7ebe90 00000086 6b67ef8b 0000000c 9f25a610 81a69869 00000001 820b6990
       [  248.216123]  820b6990 820b6990 9c6e4c20 9c6e4eb4 82c78990 00000000 6b993559 0000000c
       [  248.220616]  9c7ebe90 8105f22a 9c6e4eb4 9c6e4c20 00000001 9c7ebe98 9c7ebeb4 81a65cb3
       [  248.229990] Call Trace:
       [  248.234049]  [<81a69869>] ? _spin_unlock_irqrestore+0x22/0x37
       [  248.239769]  [<8105f22a>] ? prepare_to_wait+0x48/0x4e
       [  248.244796]  [<81a65cb3>] rcu_barrier_cpu_hotplug+0xaa/0xc9
       [  248.250343]  [<8105f029>] ? autoremove_wake_function+0x0/0x38
       [  248.256063]  [<81062cf2>] notifier_call_chain+0x49/0x71
       [  248.261263]  [<81062da0>] raw_notifier_call_chain+0x11/0x13
       [  248.266809]  [<81a0b475>] _cpu_down+0x272/0x288
       [  248.271316]  [<81a0b4d5>] cpu_down+0x4a/0xa2
       [  248.275563]  [<81a0c48a>] store_online+0x2a/0x5e
       [  248.280156]  [<81a0c460>] ? store_online+0x0/0x5e
       [  248.284836]  [<814ddc35>] sysdev_store+0x20/0x28
       [  248.289429]  [<8112e403>] sysfs_write_file+0xb8/0xe3
       [  248.294369]  [<8112e34b>] ? sysfs_write_file+0x0/0xe3
       [  248.299396]  [<810e4c8f>] vfs_write+0x91/0x120
       [  248.303817]  [<810e4dc1>] sys_write+0x40/0x65
       [  248.308150]  [<81002d73>] sysenter_do_call+0x12/0x28
      
      This change moves an RCU grace period delay off of the
      critical path for CPU-hotunplug operations.
      
      Since RCU callback migration is only performed on
      CPU-hotunplug operations, and since the rcu_barrier() race is
      provoked only by consecutive CPU-hotunplug operations, it is
      not necessary to delay the end of a given CPU-hotunplug
      operation.
      
      We can instead choose to delay the beginning of the next
      CPU-hotunplug operation.
      Reported-by: default avatarIngo Molnar <mingo@elte.hu>
      Signed-off-by: default avatarPaul E. McKenney <paulmck@linux.vnet.ibm.com>
      Cc: Josh Triplett <josht@linux.vnet.ibm.com>
      Cc: laijs@cn.fujitsu.com
      Cc: dipankar@in.ibm.com
      Cc: akpm@linux-foundation.org
      Cc: mathieu.desnoyers@polymtl.ca
      Cc: dvhltc@us.ibm.com
      Cc: niv@us.ibm.com
      Cc: peterz@infradead.org
      Cc: rostedt@goodmis.org
      Cc: hugh.dickins@tiscali.co.uk
      Cc: benh@kernel.crashing.org
      LKML-Reference: <20090819060614.GA14383@linux.vnet.ibm.com>
      Signed-off-by: default avatarIngo Molnar <mingo@elte.hu>
      1423cc03
  4. 15 Aug, 2009 7 commits
    • Josh Triplett's avatar
      rcu: Fix typo in rcu_irq_exit() comment header · 684ca5cc
      Josh Triplett authored
      Signed-off-by: default avatarJosh Triplett <josh@joshtriplett.org>
      Signed-off-by: default avatarPaul E. McKenney <paulmck@linux.vnet.ibm.com>
      Cc: laijs@cn.fujitsu.com
      Cc: dipankar@in.ibm.com
      Cc: josht@linux.vnet.ibm.com
      Cc: akpm@linux-foundation.org
      Cc: mathieu.desnoyers@polymtl.ca
      Cc: dvhltc@us.ibm.com
      Cc: niv@us.ibm.com
      Cc: peterz@infradead.org
      Cc: rostedt@goodmis.org
      Cc: hugh.dickins@tiscali.co.uk
      Cc: benh@kernel.crashing.org
      LKML-Reference: <1250355231169-git-send-email->
      Signed-off-by: default avatarIngo Molnar <mingo@elte.hu>
      684ca5cc
    • Paul E. McKenney's avatar
      rcu: Make rcupreempt_trace.c look at offline CPUs · b612ba80
      Paul E. McKenney authored
      Given that offline CPUs can now have non-zero counters, we need
      to dump counters for offline CPUs as well as for online CPUs.
      Signed-off-by: default avatarPaul E. McKenney <paulmck@linux.vnet.ibm.com>
      Cc: laijs@cn.fujitsu.com
      Cc: dipankar@in.ibm.com
      Cc: josht@linux.vnet.ibm.com
      Cc: akpm@linux-foundation.org
      Cc: mathieu.desnoyers@polymtl.ca
      Cc: dvhltc@us.ibm.com
      Cc: niv@us.ibm.com
      Cc: peterz@infradead.org
      Cc: rostedt@goodmis.org
      Cc: hugh.dickins@tiscali.co.uk
      Cc: benh@kernel.crashing.org
      LKML-Reference: <12503552313921-git-send-email->
      Signed-off-by: default avatarIngo Molnar <mingo@elte.hu>
      b612ba80
    • Paul E. McKenney's avatar
      rcu: Make preemptable RCU scan all CPUs when summing RCU counters · 8064d549
      Paul E. McKenney authored
      This patch eliminates the counter-moving during CPU-offline
      notifiers, eliminating potential confusion if counters are
      scanned during counter-movement process.
      
      This confusion could result in premature ending of an RCU grace
      period. For example, if there are two tasks in RCU read-side
      critical sections (so that the sum of the counters is two), and
      the counter for the CPU going offline is -2, then moving the
      count to another CPU can result in the sum momentarily
      appearing to be zero.  Since there are no memory barriers in
      either case, many more such scenarios are possible.
      
      So just don't move the counts!!!
      Signed-off-by: default avatarPaul E. McKenney <paulmck@linux.vnet.ibm.com>
      Cc: laijs@cn.fujitsu.com
      Cc: dipankar@in.ibm.com
      Cc: josht@linux.vnet.ibm.com
      Cc: akpm@linux-foundation.org
      Cc: mathieu.desnoyers@polymtl.ca
      Cc: dvhltc@us.ibm.com
      Cc: niv@us.ibm.com
      Cc: peterz@infradead.org
      Cc: rostedt@goodmis.org
      Cc: hugh.dickins@tiscali.co.uk
      Cc: benh@kernel.crashing.org
      LKML-Reference: <12503552312863-git-send-email->
      Signed-off-by: default avatarIngo Molnar <mingo@elte.hu>
      8064d549
    • Paul E. McKenney's avatar
      rcu: Simplify RCU CPU-hotplug notification · 2e597558
      Paul E. McKenney authored
      Use the new cpu_notifier() API to simplify RCU's CPU-hotplug
      notifiers, collapsing down to a single such notifier.
      
      This makes it trivial to provide the notifier-ordering
      guarantee that rcu_barrier() depends on.
      
      Also remove redundant open_softirq() calls from Hierarchical
      RCU notifier.
      Signed-off-by: default avatarPaul E. McKenney <paulmck@linux.vnet.ibm.com>
      Cc: laijs@cn.fujitsu.com
      Cc: dipankar@in.ibm.com
      Cc: josht@linux.vnet.ibm.com
      Cc: akpm@linux-foundation.org
      Cc: mathieu.desnoyers@polymtl.ca
      Cc: dvhltc@us.ibm.com
      Cc: niv@us.ibm.com
      Cc: peterz@infradead.org
      Cc: rostedt@goodmis.org
      Cc: hugh.dickins@tiscali.co.uk
      Cc: benh@kernel.crashing.org
      LKML-Reference: <12503552312510-git-send-email->
      Signed-off-by: default avatarIngo Molnar <mingo@elte.hu>
      2e597558
    • Paul E. McKenney's avatar
      cpu hotplug: Introduce cpu_notifier() to handle !HOTPLUG_CPU case · 799e64f0
      Paul E. McKenney authored
      This patch introduces a new cpu_notifier() API that is similar
      to hotcpu_notifier(), but which also notifies of CPUs coming
      online during boot in the !HOTPLUG_CPU case.
      Reported-by: default avatarIngo Molnar <mingo@elte.hu>
      Reported-by: default avatarHugh Dickins <hugh.dickins@tiscali.co.uk>
      Tested-by: default avatarHugh Dickins <hugh.dickins@tiscali.co.uk>
      Signed-off-by: default avatarPaul E. McKenney <paulmck@linux.vnet.ibm.com>
      Cc: laijs@cn.fujitsu.com
      Cc: dipankar@in.ibm.com
      Cc: josht@linux.vnet.ibm.com
      Cc: akpm@linux-foundation.org
      Cc: mathieu.desnoyers@polymtl.ca
      Cc: dvhltc@us.ibm.com
      Cc: niv@us.ibm.com
      Cc: peterz@infradead.org
      Cc: rostedt@goodmis.org
      Cc: benh@kernel.crashing.org
      LKML-Reference: <12503552312611-git-send-email->
      Signed-off-by: default avatarIngo Molnar <mingo@elte.hu>
      799e64f0
    • Paul E. McKenney's avatar
      rcu: Split hierarchical RCU initialization into boot-time and CPU-online pieces · 27569620
      Paul E. McKenney authored
      This patch divides the rcutree initialization into boot-time
      and hotplug-time components, so that the tree data structures
      are guaranteed to be fully linked at boot time regardless of
      what might happen in CPU hotplug operations.
      
      This makes RCU more resilient against CPU hotplug misbehavior
      (and vice versa), but more importantly, does a better job of
      compartmentalizing the code.
      Reported-by: default avatarIngo Molnar <mingo@elte.hu>
      Signed-off-by: default avatarPaul E. McKenney <paulmck@linux.vnet.ibm.com>
      Cc: laijs@cn.fujitsu.com
      Cc: dipankar@in.ibm.com
      Cc: josht@linux.vnet.ibm.com
      Cc: akpm@linux-foundation.org
      Cc: mathieu.desnoyers@polymtl.ca
      Cc: dvhltc@us.ibm.com
      Cc: niv@us.ibm.com
      Cc: peterz@infradead.org
      Cc: rostedt@goodmis.org
      Cc: hugh.dickins@tiscali.co.uk
      Cc: benh@kernel.crashing.org
      LKML-Reference: <1250355231152-git-send-email->
      Signed-off-by: default avatarIngo Molnar <mingo@elte.hu>
      27569620
    • Ingo Molnar's avatar
      Merge commit 'v2.6.31-rc6' into core/rcu · fa08661a
      Ingo Molnar authored
      Merge reason: the branch was on pre-rc1 .30, update to latest.
      Signed-off-by: default avatarIngo Molnar <mingo@elte.hu>
      fa08661a
  5. 13 Aug, 2009 23 commits