• Frederic Weisbecker's avatar
    rcu: Exit RCU extended QS on user preemption · 20ab65e3
    Frederic Weisbecker authored
    When exceptions or irq are about to resume userspace, if
    the task needs to be rescheduled, the arch low level code
    calls schedule() directly.
    
    If we call it, it is because we have the TIF_RESCHED flag:
    
    - It can be set after random local calls to set_need_resched()
    (RCU, drm, ...)
    
    - A wake up happened and the CPU needs preemption. This can
      happen in several ways:
    
        * Remotely: the remote waking CPU has set TIF_RESCHED and send the
          wakee an IPI to schedule the new task.
        * Remotely enqueued: the remote waking CPU sends an IPI to the target
          and the wake up is made by the target.
        * Locally: waking CPU == wakee CPU and the wakeup is done locally.
          set_need_resched() is called without IPI.
    
    In the case of local and remotely enqueued wake ups, the tick can
    be restarted when we enqueue the new task and RCU can exit the
    extended quiescent state at the same time. Then by the time we reach
    irq exit path and we call schedule, we are not in RCU user mode.
    
    But if we call schedule() only because something called set_need_resched(),
    RCU may still be in user mode when we reach schedule.
    
    Also if a wake up is done remotely, the CPU might see the TIF_RESCHED
    flag and call schedule while the IPI has not yet happen to restart the
    tick and exit RCU user mode.
    
    We need to manually protect against these corner cases.
    
    Create a new API schedule_user() that calls schedule() inside
    rcu_user_exit()-rcu_user_enter() in order to protect it. Archs
    will need to rely on it now to implement user preemption safely.
    Signed-off-by: default avatarFrederic Weisbecker <fweisbec@gmail.com>
    Cc: Alessio Igor Bogani <abogani@kernel.org>
    Cc: Andrew Morton <akpm@linux-foundation.org>
    Cc: Avi Kivity <avi@redhat.com>
    Cc: Chris Metcalf <cmetcalf@tilera.com>
    Cc: Christoph Lameter <cl@linux.com>
    Cc: Geoff Levand <geoff@infradead.org>
    Cc: Gilad Ben Yossef <gilad@benyossef.com>
    Cc: Hakan Akkan <hakanakkan@gmail.com>
    Cc: H. Peter Anvin <hpa@zytor.com>
    Cc: Ingo Molnar <mingo@kernel.org>
    Cc: Josh Triplett <josh@joshtriplett.org>
    Cc: Kevin Hilman <khilman@ti.com>
    Cc: Max Krasnyansky <maxk@qualcomm.com>
    Cc: Peter Zijlstra <peterz@infradead.org>
    Cc: Stephen Hemminger <shemminger@vyatta.com>
    Cc: Steven Rostedt <rostedt@goodmis.org>
    Cc: Sven-Thorsten Dietrich <thebigcorporation@gmail.com>
    Cc: Thomas Gleixner <tglx@linutronix.de>
    Signed-off-by: default avatarPaul E. McKenney <paulmck@linux.vnet.ibm.com>
    Reviewed-by: default avatarJosh Triplett <josh@joshtriplett.org>
    20ab65e3
core.c 205 KB