• Paul E. McKenney's avatar
    rcu: Switch __rcu_process_callbacks() to rcu_accelerate_cbs() · bd7af846
    Paul E. McKenney authored
    The __rcu_process_callbacks() function currently checks to see if
    the current CPU needs a grace period and also if there is any other
    reason to kick off a new grace period.  This is one of the fail-safe
    checks that has been rendered unnecessary by the changes that increase
    the accuracy of rcu_gp_cleanup()'s estimate as to whether another grace
    period is required.  Because this particular fail-safe involved acquiring
    the root rcu_node structure's ->lock, which has seen excessive contention
    in real life, this fail-safe needs to go.
    
    However, one check must remain, namely the check for newly arrived
    RCU callbacks that have not yet been associated with a grace period.
    One might hope that the checks in __note_gp_changes(), which is invoked
    indirectly from rcu_check_quiescent_state(), would suffice, but this
    function won't be invoked at all if RCU is idle.  It is therefore necessary
    to replace the fail-safe checks with a simpler check for newly arrived
    callbacks during an RCU idle period, which is exactly what this commit
    does.  This change removes the final call to rcu_start_gp(), so this
    function is removed as well.
    
    Note that lockless use of cpu_needs_another_gp() is racy, but that
    these races are harmless in this case.  If RCU really is idle, the
    values will not change, so the return value from cpu_needs_another_gp()
    will be correct.  If RCU is not idle, the resulting redundant call to
    rcu_accelerate_cbs() will be harmless, and might even have the benefit
    of reducing grace-period latency a bit.
    
    This commit also moves interrupt disabling into the "if" statement to
    improve real-time response a bit.
    Reported-by: default avatarNicholas Piggin <npiggin@gmail.com>
    Signed-off-by: default avatarPaul E. McKenney <paulmck@linux.vnet.ibm.com>
    Tested-by: default avatarNicholas Piggin <npiggin@gmail.com>
    bd7af846
tree.c 130 KB