• Paul E. McKenney's avatar
    rcu: Fix rcu_barrier_callback() race condition · aa24f937
    Paul E. McKenney authored
    The rcu_barrier_callback() function does an atomic_dec_and_test(), and
    if it is the last CPU to check in, does the required wakeup.  Either way,
    it does an event trace.  Unfortunately, this is susceptible to the
    following sequence of events:
    
    o	CPU 0 invokes rcu_barrier_callback(), but atomic_dec_and_test()
    	says that it is not last.  But at this point, CPU 0 is delayed,
    	perhaps due to an NMI, SMI, or vCPU preemption.
    
    o	CPU 1 invokes rcu_barrier_callback(), and atomic_dec_and_test()
    	says that it is last.  So CPU 1 traces completion and does
    	the needed wakeup.
    
    o	The awakened rcu_barrier() function does cleanup and releases
    	rcu_state.barrier_mutex.
    
    o	Another CPU now acquires rcu_state.barrier_mutex and starts
    	another round of rcu_barrier() processing, including updating
    	rcu_state.barrier_sequence.
    
    o	CPU 0 gets its act back together and does its tracing.  Except
    	that rcu_state.barrier_sequence has already been updated, so
    	its tracing is incorrect and probably quite confusing.
    	(Wait!  Why did this CPU check in twice for one rcu_barrier()
    	invocation???)
    
    This commit therefore causes rcu_barrier_callback() to take a
    snapshot of the value of rcu_state.barrier_sequence before invoking
    atomic_dec_and_test(), thus guaranteeing that the event-trace output
    is sensible, even if the timing of the event-trace output might still
    be confusing.  (Wait!  Why did the old rcu_barrier() complete before
    all of its CPUs checked in???)  But being that this is RCU, only so much
    confusion can reasonably be eliminated.
    
    This data race was reported by KCSAN.  Not appropriate for backporting
    due to failure being unlikely and due to the mild consequences of the
    failure, namely a confusing event trace.
    Signed-off-by: default avatarPaul E. McKenney <paulmck@kernel.org>
    aa24f937
tree.c 122 KB