• Paul E. McKenney's avatar
    rcu: Improve boost selection when moving tasks to root rcu_node · 5cc900cf
    Paul E. McKenney authored
    The rcu_preempt_offline_tasks() moves all tasks queued on a given leaf
    rcu_node structure to the root rcu_node, which is done when the last CPU
    corresponding the the leaf rcu_node structure goes offline.  Now that
    RCU-preempt's synchronize_rcu_expedited() implementation blocks CPU-hotplug
    operations during the initialization of each rcu_node structure's
    ->boost_tasks pointer, rcu_preempt_offline_tasks() can do a better job
    of setting the root rcu_node's ->boost_tasks pointer.
    
    The key point is that rcu_preempt_offline_tasks() runs as part of the
    CPU-hotplug process, so that a concurrent synchronize_rcu_expedited()
    is guaranteed to either have not started on the one hand (in which case
    there is no boosting on behalf of the expedited grace period) or to be
    completely initialized on the other (in which case, in the absence of
    other priority boosting, all ->boost_tasks pointers will be initialized).
    Therefore, if rcu_preempt_offline_tasks() finds that the ->boost_tasks
    pointer is equal to the ->exp_tasks pointer, it can be sure that it is
    correctly placed.
    
    In the case where there was boosting ongoing at the time that the
    synchronize_rcu_expedited() function started, different nodes might start
    boosting the tasks blocking the expedited grace period at different times.
    In this mixed case, the root node will either be boosting tasks for
    the expedited grace period already, or it will start as soon as it gets
    done boosting for the normal grace period -- but in this latter case,
    the root node's tasks needed to be boosted in any case.
    
    This commit therefore adds a check of the ->boost_tasks pointer against
    the ->exp_tasks pointer to the list that prevents updating ->boost_tasks.
    Signed-off-by: default avatarPaul E. McKenney <paul.mckenney@linaro.org>
    Signed-off-by: default avatarPaul E. McKenney <paulmck@linux.vnet.ibm.com>
    Reviewed-by: default avatarJosh Triplett <josh@joshtriplett.org>
    5cc900cf
rcutree_plugin.h 66.3 KB