Skip to content

L
linux
Commit ccc9971e authored by Mauro Carvalho Chehab's avatar Mauro Carvalho Chehab Committed by Paul E. McKenney
Browse files
Options

docs: rcu: convert some articles from html to ReST 

There are 4 RCU articles that are written on html format.

The way they are, they can't be part of the Linux Kernel
documentation body nor share the styles and pdf output.

So, convert them to ReST format.

This way, make htmldocs and make pdfdocs will produce a
documentation output that will be like the original ones, but
will be part of the Linux Kernel documentation body.

Part of the conversion was done with the help of pandoc, but
the result had some broken things that had to be manually
fixed.

Following are manual changes Mauro made when doing the automatic conversion:
Quoting from: https://lore.kernel.org/rcu/20190726154550.5eeae294@coco.lan/

> > At least the pandoc's version I used here has a bug: its conversion
> > from html to ReST on those files only start after a <body> tag - or
> > when the first quiz table starts. I only discovered that adding a
> > <body> at the beginning of the file solve this book at the last
> > conversions.
> >
> > So, for most html->ReST conversions, I manually converted the first
> > part of the document, basically stripping html paragraph tags and
> > by replacing highlights by the ReST syntax.
> >
> > Also, all the quiz tables seem to assume some javascript macro or
> > css style that would be hiding the answer part until the mouse moves
> > to it. Such macro/css was not there at the kernel tree. So, the quiz
> > answers have the same color as the background, making them invisible.
> > Even if we had such macro/css, this is not portable for pdf/LaTeX output
> > (and I'm not sure if this would work with ePub).
> >
> > So, I ended by manually doing the table conversion.
> >
> > Finally, I double-checked if the conversions ended ok, addressing any
> > issues that might have heppened.
> >
> > So, after both automatic conversion and manual fixes, I opened both the
> > html files produced by Sphinx and the original ones and compared them
> > line per line (except for the indexes, as Sphinx produces them
> > automatically), in order to see if all information from the original
> > files will be there on a format close to what we have on other ReST
> > files, fixing any pending issues if any.
Signed-off-by: default avatarMauro Carvalho Chehab <mchehab+samsung@kernel.org>
Signed-off-by: default avatarPaul E. McKenney <paulmck@kernel.org>
parent c07e6f36
Hide whitespace changes
Inline Side-by-side
Showing with 4978 additions and 6104 deletions
Documentation/RCU/Design/Data-Structures/Data-Structures.html deleted 100644 → 0
View file @ c07e6f36
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN"
"http://www.w3.org/TR/html4/loose.dtd">
<html>
<head><title>A Tour Through TREE_RCU's Data Structures [LWN.net]</title>
<meta HTTP-EQUIV="Content-Type" CONTENT="text/html; charset=iso-8859-1">
<p>December 18, 2016</p>
<p>This article was contributed by Paul E.&nbsp;McKenney</p>
<h3>Introduction</h3>
This document describes RCU's major data structures and their relationship
to each other.
<ol>
<li> <a href="#Data-Structure Relationships">
Data-Structure Relationships</a>
<li> <a href="#The rcu_state Structure">
The <tt>rcu_state</tt> Structure</a>
<li> <a href="#The rcu_node Structure">
The <tt>rcu_node</tt> Structure</a>
<li> <a href="#The rcu_segcblist Structure">
The <tt>rcu_segcblist</tt> Structure</a>
<li> <a href="#The rcu_data Structure">
The <tt>rcu_data</tt> Structure</a>
<li> <a href="#The rcu_head Structure">
The <tt>rcu_head</tt> Structure</a>
<li> <a href="#RCU-Specific Fields in the task_struct Structure">
RCU-Specific Fields in the <tt>task_struct</tt> Structure</a>
<li> <a href="#Accessor Functions">
Accessor Functions</a>
</ol>
<h3><a name="Data-Structure Relationships">Data-Structure Relationships</a></h3>
<p>RCU is for all intents and purposes a large state machine, and its
data structures maintain the state in such a way as to allow RCU readers
to execute extremely quickly, while also processing the RCU grace periods
requested by updaters in an efficient and extremely scalable fashion.
The efficiency and scalability of RCU updaters is provided primarily
by a combining tree, as shown below:
</p><p><img src="BigTreeClassicRCU.svg" alt="BigTreeClassicRCU.svg" width="30%">
</p><p>This diagram shows an enclosing <tt>rcu_state</tt> structure
containing a tree of <tt>rcu_node</tt> structures.
Each leaf node of the <tt>rcu_node</tt> tree has up to 16
<tt>rcu_data</tt> structures associated with it, so that there
are <tt>NR_CPUS</tt> number of <tt>rcu_data</tt> structures,
one for each possible CPU.
This structure is adjusted at boot time, if needed, to handle the
common case where <tt>nr_cpu_ids</tt> is much less than
<tt>NR_CPUs</tt>.
For example, a number of Linux distributions set <tt>NR_CPUs=4096</tt>,
which results in a three-level <tt>rcu_node</tt> tree.
If the actual hardware has only 16 CPUs, RCU will adjust itself
at boot time, resulting in an <tt>rcu_node</tt> tree with only a single node.
</p><p>The purpose of this combining tree is to allow per-CPU events
such as quiescent states, dyntick-idle transitions,
and CPU hotplug operations to be processed efficiently
and scalably.
Quiescent states are recorded by the per-CPU <tt>rcu_data</tt> structures,
and other events are recorded by the leaf-level <tt>rcu_node</tt>
structures.
All of these events are combined at each level of the tree until finally
grace periods are completed at the tree's root <tt>rcu_node</tt>
structure.
A grace period can be completed at the root once every CPU
(or, in the case of <tt>CONFIG_PREEMPT_RCU</tt>, task)
has passed through a quiescent state.
Once a grace period has completed, record of that fact is propagated
back down the tree.
</p><p>As can be seen from the diagram, on a 64-bit system
a two-level tree with 64 leaves can accommodate 1,024 CPUs, with a fanout
of 64 at the root and a fanout of 16 at the leaves.
<table>
<tr><th>&nbsp;</th></tr>
<tr><th align="left">Quick Quiz:</th></tr>
<tr><td>
Why isn't the fanout at the leaves also 64?
</td></tr>
<tr><th align="left">Answer:</th></tr>
<tr><td bgcolor="#ffffff"><font color="ffffff">
Because there are more types of events that affect the leaf-level
<tt>rcu_node</tt> structures than further up the tree.
Therefore, if the leaf <tt>rcu_node</tt> structures have fanout of
64, the contention on these structures' <tt>-&gt;structures</tt>
becomes excessive.
Experimentation on a wide variety of systems has shown that a fanout
of 16 works well for the leaves of the <tt>rcu_node</tt> tree.
</font>
<p><font color="ffffff">Of course, further experience with
systems having hundreds or thousands of CPUs may demonstrate
that the fanout for the non-leaf <tt>rcu_node</tt> structures
must also be reduced.
Such reduction can be easily carried out when and if it proves
necessary.
In the meantime, if you are using such a system and running into
contention problems on the non-leaf <tt>rcu_node</tt> structures,
you may use the <tt>CONFIG_RCU_FANOUT</tt> kernel configuration
parameter to reduce the non-leaf fanout as needed.
</font>
<p><font color="ffffff">Kernels built for systems with
strong NUMA characteristics might also need to adjust
<tt>CONFIG_RCU_FANOUT</tt> so that the domains of the
<tt>rcu_node</tt> structures align with hardware boundaries.
However, there has thus far been no need for this.
</font></td></tr>
<tr><td>&nbsp;</td></tr>
</table>
<p>If your system has more than 1,024 CPUs (or more than 512 CPUs on
a 32-bit system), then RCU will automatically add more levels to the
tree.
For example, if you are crazy enough to build a 64-bit system with 65,536
CPUs, RCU would configure the <tt>rcu_node</tt> tree as follows:
</p><p><img src="HugeTreeClassicRCU.svg" alt="HugeTreeClassicRCU.svg" width="50%">
</p><p>RCU currently permits up to a four-level tree, which on a 64-bit system
accommodates up to 4,194,304 CPUs, though only a mere 524,288 CPUs for
32-bit systems.
On the other hand, you can set both <tt>CONFIG_RCU_FANOUT</tt> and
<tt>CONFIG_RCU_FANOUT_LEAF</tt> to be as small as 2, which would result
in a 16-CPU test using a 4-level tree.
This can be useful for testing large-system capabilities on small test
machines.
</p><p>This multi-level combining tree allows us to get most of the
performance and scalability
benefits of partitioning, even though RCU grace-period detection is
inherently a global operation.
The trick here is that only the last CPU to report a quiescent state
into a given <tt>rcu_node</tt> structure need advance to the <tt>rcu_node</tt>
structure at the next level up the tree.
This means that at the leaf-level <tt>rcu_node</tt> structure, only
one access out of sixteen will progress up the tree.
For the internal <tt>rcu_node</tt> structures, the situation is even
more extreme: Only one access out of sixty-four will progress up
the tree.
Because the vast majority of the CPUs do not progress up the tree,
the lock contention remains roughly constant up the tree.
No matter how many CPUs there are in the system, at most 64 quiescent-state
reports per grace period will progress all the way to the root
<tt>rcu_node</tt> structure, thus ensuring that the lock contention
on that root <tt>rcu_node</tt> structure remains acceptably low.
</p><p>In effect, the combining tree acts like a big shock absorber,
keeping lock contention under control at all tree levels regardless
of the level of loading on the system.
</p><p>RCU updaters wait for normal grace periods by registering
RCU callbacks, either directly via <tt>call_rcu()</tt>
or indirectly via <tt>synchronize_rcu()</tt> and friends.
RCU callbacks are represented by <tt>rcu_head</tt> structures,
which are queued on <tt>rcu_data</tt> structures while they are
waiting for a grace period to elapse, as shown in the following figure:
</p><p><img src="BigTreePreemptRCUBHdyntickCB.svg" alt="BigTreePreemptRCUBHdyntickCB.svg" width="40%">
</p><p>This figure shows how <tt>TREE_RCU</tt>'s and
<tt>PREEMPT_RCU</tt>'s major data structures are related.
Lesser data structures will be introduced with the algorithms that
make use of them.
</p><p>Note that each of the data structures in the above figure has
its own synchronization:
<p><ol>
<li> Each <tt>rcu_state</tt> structures has a lock and a mutex,
and some fields are protected by the corresponding root
<tt>rcu_node</tt> structure's lock.
<li> Each <tt>rcu_node</tt> structure has a spinlock.
<li> The fields in <tt>rcu_data</tt> are private to the corresponding
CPU, although a few can be read and written by other CPUs.
</ol>
<p>It is important to note that different data structures can have
very different ideas about the state of RCU at any given time.
For but one example, awareness of the start or end of a given RCU
grace period propagates slowly through the data structures.
This slow propagation is absolutely necessary for RCU to have good
read-side performance.
If this balkanized implementation seems foreign to you, one useful
trick is to consider each instance of these data structures to be
a different person, each having the usual slightly different
view of reality.
</p><p>The general role of each of these data structures is as
follows:
</p><ol>
<li> <tt>rcu_state</tt>:
This structure forms the interconnection between the
<tt>rcu_node</tt> and <tt>rcu_data</tt> structures,
tracks grace periods, serves as short-term repository
for callbacks orphaned by CPU-hotplug events,
maintains <tt>rcu_barrier()</tt> state,
tracks expedited grace-period state,
and maintains state used to force quiescent states when
grace periods extend too long,
<li> <tt>rcu_node</tt>: This structure forms the combining
tree that propagates quiescent-state
information from the leaves to the root, and also propagates
grace-period information from the root to the leaves.
It provides local copies of the grace-period state in order
to allow this information to be accessed in a synchronized
manner without suffering the scalability limitations that
would otherwise be imposed by global locking.
In <tt>CONFIG_PREEMPT_RCU</tt> kernels, it manages the lists
of tasks that have blocked while in their current
RCU read-side critical section.
In <tt>CONFIG_PREEMPT_RCU</tt> with
<tt>CONFIG_RCU_BOOST</tt>, it manages the
per-<tt>rcu_node</tt> priority-boosting
kernel threads (kthreads) and state.
Finally, it records CPU-hotplug state in order to determine
which CPUs should be ignored during a given grace period.
<li> <tt>rcu_data</tt>: This per-CPU structure is the
focus of quiescent-state detection and RCU callback queuing.
It also tracks its relationship to the corresponding leaf
<tt>rcu_node</tt> structure to allow more-efficient
propagation of quiescent states up the <tt>rcu_node</tt>
combining tree.
Like the <tt>rcu_node</tt> structure, it provides a local
copy of the grace-period information to allow for-free
synchronized
access to this information from the corresponding CPU.
Finally, this structure records past dyntick-idle state
for the corresponding CPU and also tracks statistics.
<li> <tt>rcu_head</tt>:
This structure represents RCU callbacks, and is the
only structure allocated and managed by RCU users.
The <tt>rcu_head</tt> structure is normally embedded
within the RCU-protected data structure.
</ol>
<p>If all you wanted from this article was a general notion of how
RCU's data structures are related, you are done.
Otherwise, each of the following sections give more details on
the <tt>rcu_state</tt>, <tt>rcu_node</tt> and <tt>rcu_data</tt> data
structures.
<h3><a name="The rcu_state Structure">
The <tt>rcu_state</tt> Structure</a></h3>
<p>The <tt>rcu_state</tt> structure is the base structure that
represents the state of RCU in the system.
This structure forms the interconnection between the
<tt>rcu_node</tt> and <tt>rcu_data</tt> structures,
tracks grace periods, contains the lock used to
synchronize with CPU-hotplug events,
and maintains state used to force quiescent states when
grace periods extend too long,
</p><p>A few of the <tt>rcu_state</tt> structure's fields are discussed,
singly and in groups, in the following sections.
The more specialized fields are covered in the discussion of their
use.
<h5>Relationship to rcu_node and rcu_data Structures</h5>
This portion of the <tt>rcu_state</tt> structure is declared
as follows:
<pre>
1 struct rcu_node node[NUM_RCU_NODES];
2 struct rcu_node *level[NUM_RCU_LVLS + 1];
3 struct rcu_data __percpu *rda;
</pre>
<table>
<tr><th>&nbsp;</th></tr>
<tr><th align="left">Quick Quiz:</th></tr>
<tr><td>
Wait a minute!
You said that the <tt>rcu_node</tt> structures formed a tree,
but they are declared as a flat array!
What gives?
</td></tr>
<tr><th align="left">Answer:</th></tr>
<tr><td bgcolor="#ffffff"><font color="ffffff">
The tree is laid out in the array.
The first node In the array is the head, the next set of nodes in the
array are children of the head node, and so on until the last set of
nodes in the array are the leaves.
</font>
<p><font color="ffffff">See the following diagrams to see how
this works.
</font></td></tr>
<tr><td>&nbsp;</td></tr>
</table>
<p>The <tt>rcu_node</tt> tree is embedded into the
<tt>-&gt;node[]</tt> array as shown in the following figure:
</p><p><img src="TreeMapping.svg" alt="TreeMapping.svg" width="40%">
</p><p>One interesting consequence of this mapping is that a
breadth-first traversal of the tree is implemented as a simple
linear scan of the array, which is in fact what the
<tt>rcu_for_each_node_breadth_first()</tt> macro does.
This macro is used at the beginning and ends of grace periods.
</p><p>Each entry of the <tt>-&gt;level</tt> array references
the first <tt>rcu_node</tt> structure on the corresponding level
of the tree, for example, as shown below:
</p><p><img src="TreeMappingLevel.svg" alt="TreeMappingLevel.svg" width="40%">
</p><p>The zero<sup>th</sup> element of the array references the root
<tt>rcu_node</tt> structure, the first element references the
first child of the root <tt>rcu_node</tt>, and finally the second
element references the first leaf <tt>rcu_node</tt> structure.
</p><p>For whatever it is worth, if you draw the tree to be tree-shaped
rather than array-shaped, it is easy to draw a planar representation:
</p><p><img src="TreeLevel.svg" alt="TreeLevel.svg" width="60%">
</p><p>Finally, the <tt>-&gt;rda</tt> field references a per-CPU
pointer to the corresponding CPU's <tt>rcu_data</tt> structure.
</p><p>All of these fields are constant once initialization is complete,
and therefore need no protection.
<h5>Grace-Period Tracking</h5>
<p>This portion of the <tt>rcu_state</tt> structure is declared
as follows:
<pre>
1 unsigned long gp_seq;
</pre>
<p>RCU grace periods are numbered, and
the <tt>-&gt;gp_seq</tt> field contains the current grace-period
sequence number.
The bottom two bits are the state of the current grace period,
which can be zero for not yet started or one for in progress.
In other words, if the bottom two bits of <tt>-&gt;gp_seq</tt> are
zero, then RCU is idle.
Any other value in the bottom two bits indicates that something is broken.
This field is protected by the root <tt>rcu_node</tt> structure's
<tt>-&gt;lock</tt> field.
</p><p>There are <tt>-&gt;gp_seq</tt> fields
in the <tt>rcu_node</tt> and <tt>rcu_data</tt> structures
as well.
The fields in the <tt>rcu_state</tt> structure represent the
most current value, and those of the other structures are compared
in order to detect the beginnings and ends of grace periods in a distributed
fashion.
The values flow from <tt>rcu_state</tt> to <tt>rcu_node</tt>
(down the tree from the root to the leaves) to <tt>rcu_data</tt>.
<h5>Miscellaneous</h5>
<p>This portion of the <tt>rcu_state</tt> structure is declared
as follows:
<pre>
1 unsigned long gp_max;
2 char abbr;
3 char *name;
</pre>
<p>The <tt>-&gt;gp_max</tt> field tracks the duration of the longest
grace period in jiffies.
It is protected by the root <tt>rcu_node</tt>'s <tt>-&gt;lock</tt>.
<p>The <tt>-&gt;name</tt> and <tt>-&gt;abbr</tt> fields distinguish
between preemptible RCU (&ldquo;rcu_preempt&rdquo; and &ldquo;p&rdquo;)
and non-preemptible RCU (&ldquo;rcu_sched&rdquo; and &ldquo;s&rdquo;).
These fields are used for diagnostic and tracing purposes.
<h3><a name="The rcu_node Structure">
The <tt>rcu_node</tt> Structure</a></h3>
<p>The <tt>rcu_node</tt> structures form the combining
tree that propagates quiescent-state
information from the leaves to the root and also that propagates
grace-period information from the root down to the leaves.
They provides local copies of the grace-period state in order
to allow this information to be accessed in a synchronized
manner without suffering the scalability limitations that
would otherwise be imposed by global locking.
In <tt>CONFIG_PREEMPT_RCU</tt> kernels, they manage the lists
of tasks that have blocked while in their current
RCU read-side critical section.
In <tt>CONFIG_PREEMPT_RCU</tt> with
<tt>CONFIG_RCU_BOOST</tt>, they manage the
per-<tt>rcu_node</tt> priority-boosting
kernel threads (kthreads) and state.
Finally, they record CPU-hotplug state in order to determine
which CPUs should be ignored during a given grace period.
</p><p>The <tt>rcu_node</tt> structure's fields are discussed,
singly and in groups, in the following sections.
<h5>Connection to Combining Tree</h5>
<p>This portion of the <tt>rcu_node</tt> structure is declared
as follows:
<pre>
1 struct rcu_node *parent;
2 u8 level;
3 u8 grpnum;
4 unsigned long grpmask;
5 int grplo;
6 int grphi;
</pre>
<p>The <tt>-&gt;parent</tt> pointer references the <tt>rcu_node</tt>
one level up in the tree, and is <tt>NULL</tt> for the root
<tt>rcu_node</tt>.
The RCU implementation makes heavy use of this field to push quiescent
states up the tree.
The <tt>-&gt;level</tt> field gives the level in the tree, with
the root being at level zero, its children at level one, and so on.
The <tt>-&gt;grpnum</tt> field gives this node's position within
the children of its parent, so this number can range between 0 and 31
on 32-bit systems and between 0 and 63 on 64-bit systems.
The <tt>-&gt;level</tt> and <tt>-&gt;grpnum</tt> fields are
used only during initialization and for tracing.
The <tt>-&gt;grpmask</tt> field is the bitmask counterpart of
<tt>-&gt;grpnum</tt>, and therefore always has exactly one bit set.
This mask is used to clear the bit corresponding to this <tt>rcu_node</tt>
structure in its parent's bitmasks, which are described later.
Finally, the <tt>-&gt;grplo</tt> and <tt>-&gt;grphi</tt> fields
contain the lowest and highest numbered CPU served by this
<tt>rcu_node</tt> structure, respectively.
</p><p>All of these fields are constant, and thus do not require any
synchronization.
<h5>Synchronization</h5>
<p>This field of the <tt>rcu_node</tt> structure is declared
as follows:
<pre>
1 raw_spinlock_t lock;
</pre>
<p>This field is used to protect the remaining fields in this structure,
unless otherwise stated.
That said, all of the fields in this structure can be accessed without
locking for tracing purposes.
Yes, this can result in confusing traces, but better some tracing confusion
than to be heisenbugged out of existence.
<h5>Grace-Period Tracking</h5>
<p>This portion of the <tt>rcu_node</tt> structure is declared
as follows:
<pre>
1 unsigned long gp_seq;
2 unsigned long gp_seq_needed;
</pre>
<p>The <tt>rcu_node</tt> structures' <tt>-&gt;gp_seq</tt> fields are
the counterparts of the field of the same name in the <tt>rcu_state</tt>
structure.
They each may lag up to one step behind their <tt>rcu_state</tt>
counterpart.
If the bottom two bits of a given <tt>rcu_node</tt> structure's
<tt>-&gt;gp_seq</tt> field is zero, then this <tt>rcu_node</tt>
structure believes that RCU is idle.
</p><p>The <tt>&gt;gp_seq</tt> field of each <tt>rcu_node</tt>
structure is updated at the beginning and the end
of each grace period.
<p>The <tt>-&gt;gp_seq_needed</tt> fields record the
furthest-in-the-future grace period request seen by the corresponding
<tt>rcu_node</tt> structure. The request is considered fulfilled when
the value of the <tt>-&gt;gp_seq</tt> field equals or exceeds that of
the <tt>-&gt;gp_seq_needed</tt> field.
<table>
<tr><th>&nbsp;</th></tr>
<tr><th align="left">Quick Quiz:</th></tr>
<tr><td>
Suppose that this <tt>rcu_node</tt> structure doesn't see
a request for a very long time.
Won't wrapping of the <tt>-&gt;gp_seq</tt> field cause
problems?
</td></tr>
<tr><th align="left">Answer:</th></tr>
<tr><td bgcolor="#ffffff"><font color="ffffff">
No, because if the <tt>-&gt;gp_seq_needed</tt> field lags behind the
<tt>-&gt;gp_seq</tt> field, the <tt>-&gt;gp_seq_needed</tt> field
will be updated at the end of the grace period.
Modulo-arithmetic comparisons therefore will always get the
correct answer, even with wrapping.
</font></td></tr>
<tr><td>&nbsp;</td></tr>
</table>
<h5>Quiescent-State Tracking</h5>
<p>These fields manage the propagation of quiescent states up the
combining tree.
</p><p>This portion of the <tt>rcu_node</tt> structure has fields
as follows:
<pre>
1 unsigned long qsmask;
2 unsigned long expmask;
3 unsigned long qsmaskinit;
4 unsigned long expmaskinit;
</pre>
<p>The <tt>-&gt;qsmask</tt> field tracks which of this
<tt>rcu_node</tt> structure's children still need to report
quiescent states for the current normal grace period.
Such children will have a value of 1 in their corresponding bit.
Note that the leaf <tt>rcu_node</tt> structures should be
thought of as having <tt>rcu_data</tt> structures as their
children.
Similarly, the <tt>-&gt;expmask</tt> field tracks which
of this <tt>rcu_node</tt> structure's children still need to report
quiescent states for the current expedited grace period.
An expedited grace period has
the same conceptual properties as a normal grace period, but the
expedited implementation accepts extreme CPU overhead to obtain
much lower grace-period latency, for example, consuming a few
tens of microseconds worth of CPU time to reduce grace-period
duration from milliseconds to tens of microseconds.
The <tt>-&gt;qsmaskinit</tt> field tracks which of this
<tt>rcu_node</tt> structure's children cover for at least
one online CPU.
This mask is used to initialize <tt>-&gt;qsmask</tt>,
and <tt>-&gt;expmaskinit</tt> is used to initialize
<tt>-&gt;expmask</tt> and the beginning of the
normal and expedited grace periods, respectively.
<table>
<tr><th>&nbsp;</th></tr>
<tr><th align="left">Quick Quiz:</th></tr>
<tr><td>
Why are these bitmasks protected by locking?
Come on, haven't you heard of atomic instructions???
</td></tr>
<tr><th align="left">Answer:</th></tr>
<tr><td bgcolor="#ffffff"><font color="ffffff">
Lockless grace-period computation! Such a tantalizing possibility!
</font>
<p><font color="ffffff">But consider the following sequence of events:
</font>
<ol>
<li> <font color="ffffff">CPU&nbsp;0 has been in dyntick-idle
mode for quite some time.
When it wakes up, it notices that the current RCU
grace period needs it to report in, so it sets a
flag where the scheduling clock interrupt will find it.
</font><p>
<li> <font color="ffffff">Meanwhile, CPU&nbsp;1 is running
<tt>force_quiescent_state()</tt>,
and notices that CPU&nbsp;0 has been in dyntick idle mode,
which qualifies as an extended quiescent state.
</font><p>
<li> <font color="ffffff">CPU&nbsp;0's scheduling clock
interrupt fires in the
middle of an RCU read-side critical section, and notices
that the RCU core needs something, so commences RCU softirq
processing.
</font>
<p>
<li> <font color="ffffff">CPU&nbsp;0's softirq handler
executes and is just about ready
to report its quiescent state up the <tt>rcu_node</tt>
tree.
</font><p>
<li> <font color="ffffff">But CPU&nbsp;1 beats it to the punch,
completing the current
grace period and starting a new one.
</font><p>
<li> <font color="ffffff">CPU&nbsp;0 now reports its quiescent
state for the wrong
grace period.
That grace period might now end before the RCU read-side
critical section.
If that happens, disaster will ensue.
</font>
</ol>
<p><font color="ffffff">So the locking is absolutely required in
order to coordinate clearing of the bits with updating of the
grace-period sequence number in <tt>-&gt;gp_seq</tt>.
</font></td></tr>
<tr><td>&nbsp;</td></tr>
</table>
<h5>Blocked-Task Management</h5>
<p><tt>PREEMPT_RCU</tt> allows tasks to be preempted in the
midst of their RCU read-side critical sections, and these tasks
must be tracked explicitly.
The details of exactly why and how they are tracked will be covered
in a separate article on RCU read-side processing.
For now, it is enough to know that the <tt>rcu_node</tt>
structure tracks them.
<pre>
1 struct list_head blkd_tasks;
2 struct list_head *gp_tasks;
3 struct list_head *exp_tasks;
4 bool wait_blkd_tasks;
</pre>
<p>The <tt>-&gt;blkd_tasks</tt> field is a list header for
the list of blocked and preempted tasks.
As tasks undergo context switches within RCU read-side critical
sections, their <tt>task_struct</tt> structures are enqueued
(via the <tt>task_struct</tt>'s <tt>-&gt;rcu_node_entry</tt>
field) onto the head of the <tt>-&gt;blkd_tasks</tt> list for the
leaf <tt>rcu_node</tt> structure corresponding to the CPU
on which the outgoing context switch executed.
As these tasks later exit their RCU read-side critical sections,
they remove themselves from the list.
This list is therefore in reverse time order, so that if one of the tasks
is blocking the current grace period, all subsequent tasks must
also be blocking that same grace period.
Therefore, a single pointer into this list suffices to track
all tasks blocking a given grace period.
That pointer is stored in <tt>-&gt;gp_tasks</tt> for normal
grace periods and in <tt>-&gt;exp_tasks</tt> for expedited
grace periods.
These last two fields are <tt>NULL</tt> if either there is
no grace period in flight or if there are no blocked tasks
preventing that grace period from completing.
If either of these two pointers is referencing a task that
removes itself from the <tt>-&gt;blkd_tasks</tt> list,
then that task must advance the pointer to the next task on
the list, or set the pointer to <tt>NULL</tt> if there
are no subsequent tasks on the list.
</p><p>For example, suppose that tasks&nbsp;T1, T2, and&nbsp;T3 are
all hard-affinitied to the largest-numbered CPU in the system.
Then if task&nbsp;T1 blocked in an RCU read-side
critical section, then an expedited grace period started,
then task&nbsp;T2 blocked in an RCU read-side critical section,
then a normal grace period started, and finally task&nbsp;3 blocked
in an RCU read-side critical section, then the state of the
last leaf <tt>rcu_node</tt> structure's blocked-task list
would be as shown below:
</p><p><img src="blkd_task.svg" alt="blkd_task.svg" width="60%">
</p><p>Task&nbsp;T1 is blocking both grace periods, task&nbsp;T2 is
blocking only the normal grace period, and task&nbsp;T3 is blocking
neither grace period.
Note that these tasks will not remove themselves from this list
immediately upon resuming execution.
They will instead remain on the list until they execute the outermost
<tt>rcu_read_unlock()</tt> that ends their RCU read-side critical
section.
<p>
The <tt>-&gt;wait_blkd_tasks</tt> field indicates whether or not
the current grace period is waiting on a blocked task.
<h5>Sizing the <tt>rcu_node</tt> Array</h5>
<p>The <tt>rcu_node</tt> array is sized via a series of
C-preprocessor expressions as follows:
<pre>
1 #ifdef CONFIG_RCU_FANOUT
2 #define RCU_FANOUT CONFIG_RCU_FANOUT
3 #else
4 # ifdef CONFIG_64BIT
5 # define RCU_FANOUT 64
6 # else
7 # define RCU_FANOUT 32
8 # endif
9 #endif
10
11 #ifdef CONFIG_RCU_FANOUT_LEAF
12 #define RCU_FANOUT_LEAF CONFIG_RCU_FANOUT_LEAF
13 #else
14 # ifdef CONFIG_64BIT
15 # define RCU_FANOUT_LEAF 64
16 # else
17 # define RCU_FANOUT_LEAF 32
18 # endif
19 #endif
20
21 #define RCU_FANOUT_1 (RCU_FANOUT_LEAF)
22 #define RCU_FANOUT_2 (RCU_FANOUT_1 * RCU_FANOUT)
23 #define RCU_FANOUT_3 (RCU_FANOUT_2 * RCU_FANOUT)
24 #define RCU_FANOUT_4 (RCU_FANOUT_3 * RCU_FANOUT)
25
26 #if NR_CPUS &lt;= RCU_FANOUT_1
27 # define RCU_NUM_LVLS 1
28 # define NUM_RCU_LVL_0 1
29 # define NUM_RCU_NODES NUM_RCU_LVL_0
30 # define NUM_RCU_LVL_INIT { NUM_RCU_LVL_0 }
31 # define RCU_NODE_NAME_INIT { "rcu_node_0" }
32 # define RCU_FQS_NAME_INIT { "rcu_node_fqs_0" }
33 # define RCU_EXP_NAME_INIT { "rcu_node_exp_0" }
34 #elif NR_CPUS &lt;= RCU_FANOUT_2
35 # define RCU_NUM_LVLS 2
36 # define NUM_RCU_LVL_0 1
37 # define NUM_RCU_LVL_1 DIV_ROUND_UP(NR_CPUS, RCU_FANOUT_1)
38 # define NUM_RCU_NODES (NUM_RCU_LVL_0 + NUM_RCU_LVL_1)
39 # define NUM_RCU_LVL_INIT { NUM_RCU_LVL_0, NUM_RCU_LVL_1 }
40 # define RCU_NODE_NAME_INIT { "rcu_node_0", "rcu_node_1" }
41 # define RCU_FQS_NAME_INIT { "rcu_node_fqs_0", "rcu_node_fqs_1" }
42 # define RCU_EXP_NAME_INIT { "rcu_node_exp_0", "rcu_node_exp_1" }
43 #elif NR_CPUS &lt;= RCU_FANOUT_3
44 # define RCU_NUM_LVLS 3
45 # define NUM_RCU_LVL_0 1
46 # define NUM_RCU_LVL_1 DIV_ROUND_UP(NR_CPUS, RCU_FANOUT_2)
47 # define NUM_RCU_LVL_2 DIV_ROUND_UP(NR_CPUS, RCU_FANOUT_1)
48 # define NUM_RCU_NODES (NUM_RCU_LVL_0 + NUM_RCU_LVL_1 + NUM_RCU_LVL_2)
49 # define NUM_RCU_LVL_INIT { NUM_RCU_LVL_0, NUM_RCU_LVL_1, NUM_RCU_LVL_2 }
50 # define RCU_NODE_NAME_INIT { "rcu_node_0", "rcu_node_1", "rcu_node_2" }
51 # define RCU_FQS_NAME_INIT { "rcu_node_fqs_0", "rcu_node_fqs_1", "rcu_node_fqs_2" }
52 # define RCU_EXP_NAME_INIT { "rcu_node_exp_0", "rcu_node_exp_1", "rcu_node_exp_2" }
53 #elif NR_CPUS &lt;= RCU_FANOUT_4
54 # define RCU_NUM_LVLS 4
55 # define NUM_RCU_LVL_0 1
56 # define NUM_RCU_LVL_1 DIV_ROUND_UP(NR_CPUS, RCU_FANOUT_3)
57 # define NUM_RCU_LVL_2 DIV_ROUND_UP(NR_CPUS, RCU_FANOUT_2)
58 # define NUM_RCU_LVL_3 DIV_ROUND_UP(NR_CPUS, RCU_FANOUT_1)
59 # define NUM_RCU_NODES (NUM_RCU_LVL_0 + NUM_RCU_LVL_1 + NUM_RCU_LVL_2 + NUM_RCU_LVL_3)
60 # define NUM_RCU_LVL_INIT { NUM_RCU_LVL_0, NUM_RCU_LVL_1, NUM_RCU_LVL_2, NUM_RCU_LVL_3 }
61 # define RCU_NODE_NAME_INIT { "rcu_node_0", "rcu_node_1", "rcu_node_2", "rcu_node_3" }
62 # define RCU_FQS_NAME_INIT { "rcu_node_fqs_0", "rcu_node_fqs_1", "rcu_node_fqs_2", "rcu_node_fqs_3" }
63 # define RCU_EXP_NAME_INIT { "rcu_node_exp_0", "rcu_node_exp_1", "rcu_node_exp_2", "rcu_node_exp_3" }
64 #else
65 # error "CONFIG_RCU_FANOUT insufficient for NR_CPUS"
66 #endif
</pre>
<p>The maximum number of levels in the <tt>rcu_node</tt> structure
is currently limited to four, as specified by lines&nbsp;21-24
and the structure of the subsequent &ldquo;if&rdquo; statement.
For 32-bit systems, this allows 16*32*32*32=524,288 CPUs, which
should be sufficient for the next few years at least.
For 64-bit systems, 16*64*64*64=4,194,304 CPUs is allowed, which
should see us through the next decade or so.
This four-level tree also allows kernels built with
<tt>CONFIG_RCU_FANOUT=8</tt> to support up to 4096 CPUs,
which might be useful in very large systems having eight CPUs per
socket (but please note that no one has yet shown any measurable
performance degradation due to misaligned socket and <tt>rcu_node</tt>
boundaries).
In addition, building kernels with a full four levels of <tt>rcu_node</tt>
tree permits better testing of RCU's combining-tree code.
</p><p>The <tt>RCU_FANOUT</tt> symbol controls how many children
are permitted at each non-leaf level of the <tt>rcu_node</tt> tree.
If the <tt>CONFIG_RCU_FANOUT</tt> Kconfig option is not specified,
it is set based on the word size of the system, which is also
the Kconfig default.
</p><p>The <tt>RCU_FANOUT_LEAF</tt> symbol controls how many CPUs are
handled by each leaf <tt>rcu_node</tt> structure.
Experience has shown that allowing a given leaf <tt>rcu_node</tt>
structure to handle 64 CPUs, as permitted by the number of bits in
the <tt>-&gt;qsmask</tt> field on a 64-bit system, results in
excessive contention for the leaf <tt>rcu_node</tt> structures'
<tt>-&gt;lock</tt> fields.
The number of CPUs per leaf <tt>rcu_node</tt> structure is therefore
limited to 16 given the default value of <tt>CONFIG_RCU_FANOUT_LEAF</tt>.
If <tt>CONFIG_RCU_FANOUT_LEAF</tt> is unspecified, the value
selected is based on the word size of the system, just as for
<tt>CONFIG_RCU_FANOUT</tt>.
Lines&nbsp;11-19 perform this computation.
</p><p>Lines&nbsp;21-24 compute the maximum number of CPUs supported by
a single-level (which contains a single <tt>rcu_node</tt> structure),
two-level, three-level, and four-level <tt>rcu_node</tt> tree,
respectively, given the fanout specified by <tt>RCU_FANOUT</tt>
and <tt>RCU_FANOUT_LEAF</tt>.
These numbers of CPUs are retained in the
<tt>RCU_FANOUT_1</tt>,
<tt>RCU_FANOUT_2</tt>,
<tt>RCU_FANOUT_3</tt>, and
<tt>RCU_FANOUT_4</tt>
C-preprocessor variables, respectively.
</p><p>These variables are used to control the C-preprocessor <tt>#if</tt>
statement spanning lines&nbsp;26-66 that computes the number of
<tt>rcu_node</tt> structures required for each level of the tree,
as well as the number of levels required.
The number of levels is placed in the <tt>NUM_RCU_LVLS</tt>
C-preprocessor variable by lines&nbsp;27, 35, 44, and&nbsp;54.
The number of <tt>rcu_node</tt> structures for the topmost level
of the tree is always exactly one, and this value is unconditionally
placed into <tt>NUM_RCU_LVL_0</tt> by lines&nbsp;28, 36, 45, and&nbsp;55.
The rest of the levels (if any) of the <tt>rcu_node</tt> tree
are computed by dividing the maximum number of CPUs by the
fanout supported by the number of levels from the current level down,
rounding up. This computation is performed by lines&nbsp;37,
46-47, and&nbsp;56-58.
Lines&nbsp;31-33, 40-42, 50-52, and&nbsp;62-63 create initializers
for lockdep lock-class names.
Finally, lines&nbsp;64-66 produce an error if the maximum number of
CPUs is too large for the specified fanout.
<h3><a name="The rcu_segcblist Structure">
The <tt>rcu_segcblist</tt> Structure</a></h3>
The <tt>rcu_segcblist</tt> structure maintains a segmented list of
callbacks as follows:
<pre>
1 #define RCU_DONE_TAIL 0
2 #define RCU_WAIT_TAIL 1
3 #define RCU_NEXT_READY_TAIL 2
4 #define RCU_NEXT_TAIL 3
5 #define RCU_CBLIST_NSEGS 4
6
7 struct rcu_segcblist {
8 struct rcu_head *head;
9 struct rcu_head **tails[RCU_CBLIST_NSEGS];
10 unsigned long gp_seq[RCU_CBLIST_NSEGS];
11 long len;
12 long len_lazy;
13 };
</pre>
<p>
The segments are as follows:
<ol>
<li> <tt>RCU_DONE_TAIL</tt>: Callbacks whose grace periods have elapsed.
These callbacks are ready to be invoked.
<li> <tt>RCU_WAIT_TAIL</tt>: Callbacks that are waiting for the
current grace period.
Note that different CPUs can have different ideas about which
grace period is current, hence the <tt>-&gt;gp_seq</tt> field.
<li> <tt>RCU_NEXT_READY_TAIL</tt>: Callbacks waiting for the next
grace period to start.
<li> <tt>RCU_NEXT_TAIL</tt>: Callbacks that have not yet been
associated with a grace period.
</ol>
<p>
The <tt>-&gt;head</tt> pointer references the first callback or
is <tt>NULL</tt> if the list contains no callbacks (which is
<i>not</i> the same as being empty).
Each element of the <tt>-&gt;tails[]</tt> array references the
<tt>-&gt;next</tt> pointer of the last callback in the corresponding
segment of the list, or the list's <tt>-&gt;head</tt> pointer if
that segment and all previous segments are empty.
If the corresponding segment is empty but some previous segment is
not empty, then the array element is identical to its predecessor.
Older callbacks are closer to the head of the list, and new callbacks
are added at the tail.
This relationship between the <tt>-&gt;head</tt> pointer, the
<tt>-&gt;tails[]</tt> array, and the callbacks is shown in this
diagram:
</p><p><img src="nxtlist.svg" alt="nxtlist.svg" width="40%">
</p><p>In this figure, the <tt>-&gt;head</tt> pointer references the
first
RCU callback in the list.
The <tt>-&gt;tails[RCU_DONE_TAIL]</tt> array element references
the <tt>-&gt;head</tt> pointer itself, indicating that none
of the callbacks is ready to invoke.
The <tt>-&gt;tails[RCU_WAIT_TAIL]</tt> array element references callback
CB&nbsp;2's <tt>-&gt;next</tt> pointer, which indicates that
CB&nbsp;1 and CB&nbsp;2 are both waiting on the current grace period,
give or take possible disagreements about exactly which grace period
is the current one.
The <tt>-&gt;tails[RCU_NEXT_READY_TAIL]</tt> array element
references the same RCU callback that <tt>-&gt;tails[RCU_WAIT_TAIL]</tt>
does, which indicates that there are no callbacks waiting on the next
RCU grace period.
The <tt>-&gt;tails[RCU_NEXT_TAIL]</tt> array element references
CB&nbsp;4's <tt>-&gt;next</tt> pointer, indicating that all the
remaining RCU callbacks have not yet been assigned to an RCU grace
period.
Note that the <tt>-&gt;tails[RCU_NEXT_TAIL]</tt> array element
always references the last RCU callback's <tt>-&gt;next</tt> pointer
unless the callback list is empty, in which case it references
the <tt>-&gt;head</tt> pointer.
<p>
There is one additional important special case for the
<tt>-&gt;tails[RCU_NEXT_TAIL]</tt> array element: It can be <tt>NULL</tt>
when this list is <i>disabled</i>.
Lists are disabled when the corresponding CPU is offline or when
the corresponding CPU's callbacks are offloaded to a kthread,
both of which are described elsewhere.
</p><p>CPUs advance their callbacks from the
<tt>RCU_NEXT_TAIL</tt> to the <tt>RCU_NEXT_READY_TAIL</tt> to the
<tt>RCU_WAIT_TAIL</tt> to the <tt>RCU_DONE_TAIL</tt> list segments
as grace periods advance.
</p><p>The <tt>-&gt;gp_seq[]</tt> array records grace-period
numbers corresponding to the list segments.
This is what allows different CPUs to have different ideas as to
which is the current grace period while still avoiding premature
invocation of their callbacks.
In particular, this allows CPUs that go idle for extended periods
to determine which of their callbacks are ready to be invoked after
reawakening.
</p><p>The <tt>-&gt;len</tt> counter contains the number of
callbacks in <tt>-&gt;head</tt>, and the
<tt>-&gt;len_lazy</tt> contains the number of those callbacks that
are known to only free memory, and whose invocation can therefore
be safely deferred.
<p><b>Important note</b>: It is the <tt>-&gt;len</tt> field that
determines whether or not there are callbacks associated with
this <tt>rcu_segcblist</tt> structure, <i>not</i> the <tt>-&gt;head</tt>
pointer.
The reason for this is that all the ready-to-invoke callbacks
(that is, those in the <tt>RCU_DONE_TAIL</tt> segment) are extracted
all at once at callback-invocation time (<tt>rcu_do_batch</tt>), due
to which <tt>-&gt;head</tt> may be set to NULL if there are no not-done
callbacks remaining in the <tt>rcu_segcblist</tt>.
If callback invocation must be postponed, for example, because a
high-priority process just woke up on this CPU, then the remaining
callbacks are placed back on the <tt>RCU_DONE_TAIL</tt> segment and
<tt>-&gt;head</tt> once again points to the start of the segment.
In short, the head field can briefly be <tt>NULL</tt> even though the
CPU has callbacks present the entire time.
Therefore, it is not appropriate to test the <tt>-&gt;head</tt> pointer
for <tt>NULL</tt>.
<p>In contrast, the <tt>-&gt;len</tt> and <tt>-&gt;len_lazy</tt> counts
are adjusted only after the corresponding callbacks have been invoked.
This means that the <tt>-&gt;len</tt> count is zero only if
the <tt>rcu_segcblist</tt> structure really is devoid of callbacks.
Of course, off-CPU sampling of the <tt>-&gt;len</tt> count requires
careful use of appropriate synchronization, for example, memory barriers.
This synchronization can be a bit subtle, particularly in the case
of <tt>rcu_barrier()</tt>.
<h3><a name="The rcu_data Structure">
The <tt>rcu_data</tt> Structure</a></h3>
<p>The <tt>rcu_data</tt> maintains the per-CPU state for the RCU subsystem.
The fields in this structure may be accessed only from the corresponding
CPU (and from tracing) unless otherwise stated.
This structure is the
focus of quiescent-state detection and RCU callback queuing.
It also tracks its relationship to the corresponding leaf
<tt>rcu_node</tt> structure to allow more-efficient
propagation of quiescent states up the <tt>rcu_node</tt>
combining tree.
Like the <tt>rcu_node</tt> structure, it provides a local
copy of the grace-period information to allow for-free
synchronized
access to this information from the corresponding CPU.
Finally, this structure records past dyntick-idle state
for the corresponding CPU and also tracks statistics.
</p><p>The <tt>rcu_data</tt> structure's fields are discussed,
singly and in groups, in the following sections.
<h5>Connection to Other Data Structures</h5>
<p>This portion of the <tt>rcu_data</tt> structure is declared
as follows:
<pre>
1 int cpu;
2 struct rcu_node *mynode;
3 unsigned long grpmask;
4 bool beenonline;
</pre>
<p>The <tt>-&gt;cpu</tt> field contains the number of the
corresponding CPU and the <tt>-&gt;mynode</tt> field references the
corresponding <tt>rcu_node</tt> structure.
The <tt>-&gt;mynode</tt> is used to propagate quiescent states
up the combining tree.
These two fields are constant and therefore do not require synchronization.
<p>The <tt>-&gt;grpmask</tt> field indicates the bit in
the <tt>-&gt;mynode-&gt;qsmask</tt> corresponding to this
<tt>rcu_data</tt> structure, and is also used when propagating
quiescent states.
The <tt>-&gt;beenonline</tt> flag is set whenever the corresponding
CPU comes online, which means that the debugfs tracing need not dump
out any <tt>rcu_data</tt> structure for which this flag is not set.
<h5>Quiescent-State and Grace-Period Tracking</h5>
<p>This portion of the <tt>rcu_data</tt> structure is declared
as follows:
<pre>
1 unsigned long gp_seq;
2 unsigned long gp_seq_needed;
3 bool cpu_no_qs;
4 bool core_needs_qs;
5 bool gpwrap;
</pre>
<p>The <tt>-&gt;gp_seq</tt> field is the counterpart of the field of the same
name in the <tt>rcu_state</tt> and <tt>rcu_node</tt> structures. The
<tt>-&gt;gp_seq_needed</tt> field is the counterpart of the field of the same
name in the rcu_node</tt> structure.
They may each lag up to one behind their <tt>rcu_node</tt>
counterparts, but in <tt>CONFIG_NO_HZ_IDLE</tt> and
<tt>CONFIG_NO_HZ_FULL</tt> kernels can lag
arbitrarily far behind for CPUs in dyntick-idle mode (but these counters
will catch up upon exit from dyntick-idle mode).
If the lower two bits of a given <tt>rcu_data</tt> structure's
<tt>-&gt;gp_seq</tt> are zero, then this <tt>rcu_data</tt>
structure believes that RCU is idle.
<table>
<tr><th>&nbsp;</th></tr>
<tr><th align="left">Quick Quiz:</th></tr>
<tr><td>
All this replication of the grace period numbers can only cause
massive confusion.
Why not just keep a global sequence number and be done with it???
</td></tr>
<tr><th align="left">Answer:</th></tr>
<tr><td bgcolor="#ffffff"><font color="ffffff">
Because if there was only a single global sequence
numbers, there would need to be a single global lock to allow
safely accessing and updating it.
And if we are not going to have a single global lock, we need
to carefully manage the numbers on a per-node basis.
Recall from the answer to a previous Quick Quiz that the consequences
of applying a previously sampled quiescent state to the wrong
grace period are quite severe.
</font></td></tr>
<tr><td>&nbsp;</td></tr>
</table>
<p>The <tt>-&gt;cpu_no_qs</tt> flag indicates that the
CPU has not yet passed through a quiescent state,
while the <tt>-&gt;core_needs_qs</tt> flag indicates that the
RCU core needs a quiescent state from the corresponding CPU.
The <tt>-&gt;gpwrap</tt> field indicates that the corresponding
CPU has remained idle for so long that the
<tt>gp_seq</tt> counter is in danger of overflow, which
will cause the CPU to disregard the values of its counters on
its next exit from idle.
<h5>RCU Callback Handling</h5>
<p>In the absence of CPU-hotplug events, RCU callbacks are invoked by
the same CPU that registered them.
This is strictly a cache-locality optimization: callbacks can and
do get invoked on CPUs other than the one that registered them.
After all, if the CPU that registered a given callback has gone
offline before the callback can be invoked, there really is no other
choice.
</p><p>This portion of the <tt>rcu_data</tt> structure is declared
as follows:
<pre>
1 struct rcu_segcblist cblist;
2 long qlen_last_fqs_check;
3 unsigned long n_cbs_invoked;
4 unsigned long n_nocbs_invoked;
5 unsigned long n_cbs_orphaned;
6 unsigned long n_cbs_adopted;
7 unsigned long n_force_qs_snap;
8 long blimit;
</pre>
<p>The <tt>-&gt;cblist</tt> structure is the segmented callback list
described earlier.
The CPU advances the callbacks in its <tt>rcu_data</tt> structure
whenever it notices that another RCU grace period has completed.
The CPU detects the completion of an RCU grace period by noticing
that the value of its <tt>rcu_data</tt> structure's
<tt>-&gt;gp_seq</tt> field differs from that of its leaf
<tt>rcu_node</tt> structure.
Recall that each <tt>rcu_node</tt> structure's
<tt>-&gt;gp_seq</tt> field is updated at the beginnings and ends of each
grace period.
<p>
The <tt>-&gt;qlen_last_fqs_check</tt> and
<tt>-&gt;n_force_qs_snap</tt> coordinate the forcing of quiescent
states from <tt>call_rcu()</tt> and friends when callback
lists grow excessively long.
</p><p>The <tt>-&gt;n_cbs_invoked</tt>,
<tt>-&gt;n_cbs_orphaned</tt>, and <tt>-&gt;n_cbs_adopted</tt>
fields count the number of callbacks invoked,
sent to other CPUs when this CPU goes offline,
and received from other CPUs when those other CPUs go offline.
The <tt>-&gt;n_nocbs_invoked</tt> is used when the CPU's callbacks
are offloaded to a kthread.
<p>
Finally, the <tt>-&gt;blimit</tt> counter is the maximum number of
RCU callbacks that may be invoked at a given time.
<h5>Dyntick-Idle Handling</h5>
<p>This portion of the <tt>rcu_data</tt> structure is declared
as follows:
<pre>
1 int dynticks_snap;
2 unsigned long dynticks_fqs;
</pre>
The <tt>-&gt;dynticks_snap</tt> field is used to take a snapshot
of the corresponding CPU's dyntick-idle state when forcing
quiescent states, and is therefore accessed from other CPUs.
Finally, the <tt>-&gt;dynticks_fqs</tt> field is used to
count the number of times this CPU is determined to be in
dyntick-idle state, and is used for tracing and debugging purposes.
<p>
This portion of the rcu_data structure is declared as follows:
<pre>
1 long dynticks_nesting;
2 long dynticks_nmi_nesting;
3 atomic_t dynticks;
4 bool rcu_need_heavy_qs;
5 bool rcu_urgent_qs;
</pre>
<p>These fields in the rcu_data structure maintain the per-CPU dyntick-idle
state for the corresponding CPU.
The fields may be accessed only from the corresponding CPU (and from tracing)
unless otherwise stated.
<p>The <tt>-&gt;dynticks_nesting</tt> field counts the
nesting depth of process execution, so that in normal circumstances
this counter has value zero or one.
NMIs, irqs, and tracers are counted by the <tt>-&gt;dynticks_nmi_nesting</tt>
field.
Because NMIs cannot be masked, changes to this variable have to be
undertaken carefully using an algorithm provided by Andy Lutomirski.
The initial transition from idle adds one, and nested transitions
add two, so that a nesting level of five is represented by a
<tt>-&gt;dynticks_nmi_nesting</tt> value of nine.
This counter can therefore be thought of as counting the number
of reasons why this CPU cannot be permitted to enter dyntick-idle
mode, aside from process-level transitions.
<p>However, it turns out that when running in non-idle kernel context,
the Linux kernel is fully capable of entering interrupt handlers that
never exit and perhaps also vice versa.
Therefore, whenever the <tt>-&gt;dynticks_nesting</tt> field is
incremented up from zero, the <tt>-&gt;dynticks_nmi_nesting</tt> field
is set to a large positive number, and whenever the
<tt>-&gt;dynticks_nesting</tt> field is decremented down to zero,
the the <tt>-&gt;dynticks_nmi_nesting</tt> field is set to zero.
Assuming that the number of misnested interrupts is not sufficient
to overflow the counter, this approach corrects the
<tt>-&gt;dynticks_nmi_nesting</tt> field every time the corresponding
CPU enters the idle loop from process context.
</p><p>The <tt>-&gt;dynticks</tt> field counts the corresponding
CPU's transitions to and from either dyntick-idle or user mode, so
that this counter has an even value when the CPU is in dyntick-idle
mode or user mode and an odd value otherwise. The transitions to/from
user mode need to be counted for user mode adaptive-ticks support
(see timers/NO_HZ.txt).
</p><p>The <tt>-&gt;rcu_need_heavy_qs</tt> field is used
to record the fact that the RCU core code would really like to
see a quiescent state from the corresponding CPU, so much so that
it is willing to call for heavy-weight dyntick-counter operations.
This flag is checked by RCU's context-switch and <tt>cond_resched()</tt>
code, which provide a momentary idle sojourn in response.
</p><p>Finally, the <tt>-&gt;rcu_urgent_qs</tt> field is used to record
the fact that the RCU core code would really like to see a quiescent state from
the corresponding CPU, with the various other fields indicating just how badly
RCU wants this quiescent state.
This flag is checked by RCU's context-switch path
(<tt>rcu_note_context_switch</tt>) and the cond_resched code.
<table>
<tr><th>&nbsp;</th></tr>
<tr><th align="left">Quick Quiz:</th></tr>
<tr><td>
Why not simply combine the <tt>-&gt;dynticks_nesting</tt>
and <tt>-&gt;dynticks_nmi_nesting</tt> counters into a
single counter that just counts the number of reasons that
the corresponding CPU is non-idle?
</td></tr>
<tr><th align="left">Answer:</th></tr>
<tr><td bgcolor="#ffffff"><font color="ffffff">
Because this would fail in the presence of interrupts whose
handlers never return and of handlers that manage to return
from a made-up interrupt.
</font></td></tr>
<tr><td>&nbsp;</td></tr>
</table>
<p>Additional fields are present for some special-purpose
builds, and are discussed separately.
<h3><a name="The rcu_head Structure">
The <tt>rcu_head</tt> Structure</a></h3>
<p>Each <tt>rcu_head</tt> structure represents an RCU callback.
These structures are normally embedded within RCU-protected data
structures whose algorithms use asynchronous grace periods.
In contrast, when using algorithms that block waiting for RCU grace periods,
RCU users need not provide <tt>rcu_head</tt> structures.
</p><p>The <tt>rcu_head</tt> structure has fields as follows:
<pre>
1 struct rcu_head *next;
2 void (*func)(struct rcu_head *head);
</pre>
<p>The <tt>-&gt;next</tt> field is used
to link the <tt>rcu_head</tt> structures together in the
lists within the <tt>rcu_data</tt> structures.
The <tt>-&gt;func</tt> field is a pointer to the function
to be called when the callback is ready to be invoked, and
this function is passed a pointer to the <tt>rcu_head</tt>
structure.
However, <tt>kfree_rcu()</tt> uses the <tt>-&gt;func</tt>
field to record the offset of the <tt>rcu_head</tt>
structure within the enclosing RCU-protected data structure.
</p><p>Both of these fields are used internally by RCU.
From the viewpoint of RCU users, this structure is an
opaque &ldquo;cookie&rdquo;.
<table>
<tr><th>&nbsp;</th></tr>
<tr><th align="left">Quick Quiz:</th></tr>
<tr><td>
Given that the callback function <tt>-&gt;func</tt>
is passed a pointer to the <tt>rcu_head</tt> structure,
how is that function supposed to find the beginning of the
enclosing RCU-protected data structure?
</td></tr>
<tr><th align="left">Answer:</th></tr>
<tr><td bgcolor="#ffffff"><font color="ffffff">
In actual practice, there is a separate callback function per
type of RCU-protected data structure.
The callback function can therefore use the <tt>container_of()</tt>
macro in the Linux kernel (or other pointer-manipulation facilities
in other software environments) to find the beginning of the
enclosing structure.
</font></td></tr>
<tr><td>&nbsp;</td></tr>
</table>
<h3><a name="RCU-Specific Fields in the task_struct Structure">
RCU-Specific Fields in the <tt>task_struct</tt> Structure</a></h3>
<p>The <tt>CONFIG_PREEMPT_RCU</tt> implementation uses some
additional fields in the <tt>task_struct</tt> structure:
<pre>
1 #ifdef CONFIG_PREEMPT_RCU
2 int rcu_read_lock_nesting;
3 union rcu_special rcu_read_unlock_special;
4 struct list_head rcu_node_entry;
5 struct rcu_node *rcu_blocked_node;
6 #endif /* #ifdef CONFIG_PREEMPT_RCU */
7 #ifdef CONFIG_TASKS_RCU
8 unsigned long rcu_tasks_nvcsw;
9 bool rcu_tasks_holdout;
10 struct list_head rcu_tasks_holdout_list;
11 int rcu_tasks_idle_cpu;
12 #endif /* #ifdef CONFIG_TASKS_RCU */
</pre>
<p>The <tt>-&gt;rcu_read_lock_nesting</tt> field records the
nesting level for RCU read-side critical sections, and
the <tt>-&gt;rcu_read_unlock_special</tt> field is a bitmask
that records special conditions that require <tt>rcu_read_unlock()</tt>
to do additional work.
The <tt>-&gt;rcu_node_entry</tt> field is used to form lists of
tasks that have blocked within preemptible-RCU read-side critical
sections and the <tt>-&gt;rcu_blocked_node</tt> field references
the <tt>rcu_node</tt> structure whose list this task is a member of,
or <tt>NULL</tt> if it is not blocked within a preemptible-RCU
read-side critical section.
<p>The <tt>-&gt;rcu_tasks_nvcsw</tt> field tracks the number of
voluntary context switches that this task had undergone at the
beginning of the current tasks-RCU grace period,
<tt>-&gt;rcu_tasks_holdout</tt> is set if the current tasks-RCU
grace period is waiting on this task, <tt>-&gt;rcu_tasks_holdout_list</tt>
is a list element enqueuing this task on the holdout list,
and <tt>-&gt;rcu_tasks_idle_cpu</tt> tracks which CPU this
idle task is running, but only if the task is currently running,
that is, if the CPU is currently idle.
<h3><a name="Accessor Functions">
Accessor Functions</a></h3>
<p>The following listing shows the
<tt>rcu_get_root()</tt>, <tt>rcu_for_each_node_breadth_first</tt> and
<tt>rcu_for_each_leaf_node()</tt> function and macros:
<pre>
1 static struct rcu_node *rcu_get_root(struct rcu_state *rsp)
2 {
3 return &amp;rsp-&gt;node[0];
4 }
5
6 #define rcu_for_each_node_breadth_first(rsp, rnp) \
7 for ((rnp) = &amp;(rsp)-&gt;node[0]; \
8 (rnp) &lt; &amp;(rsp)-&gt;node[NUM_RCU_NODES]; (rnp)++)
9
10 #define rcu_for_each_leaf_node(rsp, rnp) \
11 for ((rnp) = (rsp)-&gt;level[NUM_RCU_LVLS - 1]; \
12 (rnp) &lt; &amp;(rsp)-&gt;node[NUM_RCU_NODES]; (rnp)++)
</pre>
<p>The <tt>rcu_get_root()</tt> simply returns a pointer to the
first element of the specified <tt>rcu_state</tt> structure's
<tt>-&gt;node[]</tt> array, which is the root <tt>rcu_node</tt>
structure.
</p><p>As noted earlier, the <tt>rcu_for_each_node_breadth_first()</tt>
macro takes advantage of the layout of the <tt>rcu_node</tt>
structures in the <tt>rcu_state</tt> structure's
<tt>-&gt;node[]</tt> array, performing a breadth-first traversal by
simply traversing the array in order.
Similarly, the <tt>rcu_for_each_leaf_node()</tt> macro traverses only
the last part of the array, thus traversing only the leaf
<tt>rcu_node</tt> structures.
<table>
<tr><th>&nbsp;</th></tr>
<tr><th align="left">Quick Quiz:</th></tr>
<tr><td>
What does
<tt>rcu_for_each_leaf_node()</tt> do if the <tt>rcu_node</tt> tree
contains only a single node?
</td></tr>
<tr><th align="left">Answer:</th></tr>
<tr><td bgcolor="#ffffff"><font color="ffffff">
In the single-node case,
<tt>rcu_for_each_leaf_node()</tt> traverses the single node.
</font></td></tr>
<tr><td>&nbsp;</td></tr>
</table>
<h3><a name="Summary">
Summary</a></h3>
So the state of RCU is represented by an <tt>rcu_state</tt> structure,
which contains a combining tree of <tt>rcu_node</tt> and
<tt>rcu_data</tt> structures.
Finally, in <tt>CONFIG_NO_HZ_IDLE</tt> kernels, each CPU's dyntick-idle
state is tracked by dynticks-related fields in the <tt>rcu_data</tt> structure.
If you made it this far, you are well prepared to read the code
walkthroughs in the other articles in this series.
<h3><a name="Acknowledgments">
Acknowledgments</a></h3>
I owe thanks to Cyrill Gorcunov, Mathieu Desnoyers, Dhaval Giani, Paul
Turner, Abhishek Srivastava, Matt Kowalczyk, and Serge Hallyn
for helping me get this document into a more human-readable state.
<h3><a name="Legal Statement">
Legal Statement</a></h3>
<p>This work represents the view of the author and does not necessarily
represent the view of IBM.
</p><p>Linux is a registered trademark of Linus Torvalds.
</p><p>Other company, product, and service names may be trademarks or
service marks of others.
</body></html>
Documentation/RCU/Design/Data-Structures/Data-Structures.rst 0 → 100644
View file @ ccc9971e
===================================================
A Tour Through TREE_RCU's Data Structures [LWN.net]
===================================================
December 18, 2016
This article was contributed by Paul E. McKenney
Introduction
============
This document describes RCU's major data structures and their relationship
to each other.
Data-Structure Relationships
============================
RCU is for all intents and purposes a large state machine, and its
data structures maintain the state in such a way as to allow RCU readers
to execute extremely quickly, while also processing the RCU grace periods
requested by updaters in an efficient and extremely scalable fashion.
The efficiency and scalability of RCU updaters is provided primarily
by a combining tree, as shown below:
.. kernel-figure:: BigTreeClassicRCU.svg
This diagram shows an enclosing ``rcu_state`` structure containing a tree
of ``rcu_node`` structures. Each leaf node of the ``rcu_node`` tree has up
to 16 ``rcu_data`` structures associated with it, so that there are
``NR_CPUS`` number of ``rcu_data`` structures, one for each possible CPU.
This structure is adjusted at boot time, if needed, to handle the common
case where ``nr_cpu_ids`` is much less than ``NR_CPUs``.
For example, a number of Linux distributions set ``NR_CPUs=4096``,
which results in a three-level ``rcu_node`` tree.
If the actual hardware has only 16 CPUs, RCU will adjust itself
at boot time, resulting in an ``rcu_node`` tree with only a single node.
The purpose of this combining tree is to allow per-CPU events
such as quiescent states, dyntick-idle transitions,
and CPU hotplug operations to be processed efficiently
and scalably.
Quiescent states are recorded by the per-CPU ``rcu_data`` structures,
and other events are recorded by the leaf-level ``rcu_node``
structures.
All of these events are combined at each level of the tree until finally
grace periods are completed at the tree's root ``rcu_node``
structure.
A grace period can be completed at the root once every CPU
(or, in the case of ``CONFIG_PREEMPT_RCU``, task)
has passed through a quiescent state.
Once a grace period has completed, record of that fact is propagated
back down the tree.
As can be seen from the diagram, on a 64-bit system
a two-level tree with 64 leaves can accommodate 1,024 CPUs, with a fanout
of 64 at the root and a fanout of 16 at the leaves.
+-----------------------------------------------------------------------+
| **Quick Quiz**: |
+-----------------------------------------------------------------------+
| Why isn't the fanout at the leaves also 64? |
+-----------------------------------------------------------------------+
| **Answer**: |
+-----------------------------------------------------------------------+
| Because there are more types of events that affect the leaf-level |
| ``rcu_node`` structures than further up the tree. Therefore, if the |
| leaf ``rcu_node`` structures have fanout of 64, the contention on |
| these structures' ``->structures`` becomes excessive. Experimentation |
| on a wide variety of systems has shown that a fanout of 16 works well |
| for the leaves of the ``rcu_node`` tree. |
| |
| Of course, further experience with systems having hundreds or |
| thousands of CPUs may demonstrate that the fanout for the non-leaf |
| ``rcu_node`` structures must also be reduced. Such reduction can be |
| easily carried out when and if it proves necessary. In the meantime, |
| if you are using such a system and running into contention problems |
| on the non-leaf ``rcu_node`` structures, you may use the |
| ``CONFIG_RCU_FANOUT`` kernel configuration parameter to reduce the |
| non-leaf fanout as needed. |
| |
| Kernels built for systems with strong NUMA characteristics might |
| also need to adjust ``CONFIG_RCU_FANOUT`` so that the domains of |
| the ``rcu_node`` structures align with hardware boundaries. |
| However, there has thus far been no need for this. |
+-----------------------------------------------------------------------+
If your system has more than 1,024 CPUs (or more than 512 CPUs on a
32-bit system), then RCU will automatically add more levels to the tree.
For example, if you are crazy enough to build a 64-bit system with
65,536 CPUs, RCU would configure the ``rcu_node`` tree as follows:
.. kernel-figure:: HugeTreeClassicRCU.svg
RCU currently permits up to a four-level tree, which on a 64-bit system
accommodates up to 4,194,304 CPUs, though only a mere 524,288 CPUs for
32-bit systems. On the other hand, you can set both
``CONFIG_RCU_FANOUT`` and ``CONFIG_RCU_FANOUT_LEAF`` to be as small as
2, which would result in a 16-CPU test using a 4-level tree. This can be
useful for testing large-system capabilities on small test machines.
This multi-level combining tree allows us to get most of the performance
and scalability benefits of partitioning, even though RCU grace-period
detection is inherently a global operation. The trick here is that only
the last CPU to report a quiescent state into a given ``rcu_node``
structure need advance to the ``rcu_node`` structure at the next level
up the tree. This means that at the leaf-level ``rcu_node`` structure,
only one access out of sixteen will progress up the tree. For the
internal ``rcu_node`` structures, the situation is even more extreme:
Only one access out of sixty-four will progress up the tree. Because the
vast majority of the CPUs do not progress up the tree, the lock
contention remains roughly constant up the tree. No matter how many CPUs
there are in the system, at most 64 quiescent-state reports per grace
period will progress all the way to the root ``rcu_node`` structure,
thus ensuring that the lock contention on that root ``rcu_node``
structure remains acceptably low.
In effect, the combining tree acts like a big shock absorber, keeping
lock contention under control at all tree levels regardless of the level
of loading on the system.
RCU updaters wait for normal grace periods by registering RCU callbacks,
either directly via ``call_rcu()`` or indirectly via
``synchronize_rcu()`` and friends. RCU callbacks are represented by
``rcu_head`` structures, which are queued on ``rcu_data`` structures
while they are waiting for a grace period to elapse, as shown in the
following figure:
.. kernel-figure:: BigTreePreemptRCUBHdyntickCB.svg
This figure shows how ``TREE_RCU``'s and ``PREEMPT_RCU``'s major data
structures are related. Lesser data structures will be introduced with
the algorithms that make use of them.
Note that each of the data structures in the above figure has its own
synchronization:
#. Each ``rcu_state`` structures has a lock and a mutex, and some fields
are protected by the corresponding root ``rcu_node`` structure's lock.
#. Each ``rcu_node`` structure has a spinlock.
#. The fields in ``rcu_data`` are private to the corresponding CPU,
although a few can be read and written by other CPUs.
It is important to note that different data structures can have very
different ideas about the state of RCU at any given time. For but one
example, awareness of the start or end of a given RCU grace period
propagates slowly through the data structures. This slow propagation is
absolutely necessary for RCU to have good read-side performance. If this
balkanized implementation seems foreign to you, one useful trick is to
consider each instance of these data structures to be a different
person, each having the usual slightly different view of reality.
The general role of each of these data structures is as follows:
#. ``rcu_state``: This structure forms the interconnection between the
``rcu_node`` and ``rcu_data`` structures, tracks grace periods,
serves as short-term repository for callbacks orphaned by CPU-hotplug
events, maintains ``rcu_barrier()`` state, tracks expedited
grace-period state, and maintains state used to force quiescent
states when grace periods extend too long,
#. ``rcu_node``: This structure forms the combining tree that propagates
quiescent-state information from the leaves to the root, and also
propagates grace-period information from the root to the leaves. It
provides local copies of the grace-period state in order to allow
this information to be accessed in a synchronized manner without
suffering the scalability limitations that would otherwise be imposed
by global locking. In ``CONFIG_PREEMPT_RCU`` kernels, it manages the
lists of tasks that have blocked while in their current RCU read-side
critical section. In ``CONFIG_PREEMPT_RCU`` with
``CONFIG_RCU_BOOST``, it manages the per-\ ``rcu_node``
priority-boosting kernel threads (kthreads) and state. Finally, it
records CPU-hotplug state in order to determine which CPUs should be
ignored during a given grace period.
#. ``rcu_data``: This per-CPU structure is the focus of quiescent-state
detection and RCU callback queuing. It also tracks its relationship
to the corresponding leaf ``rcu_node`` structure to allow
more-efficient propagation of quiescent states up the ``rcu_node``
combining tree. Like the ``rcu_node`` structure, it provides a local
copy of the grace-period information to allow for-free synchronized
access to this information from the corresponding CPU. Finally, this
structure records past dyntick-idle state for the corresponding CPU
and also tracks statistics.
#. ``rcu_head``: This structure represents RCU callbacks, and is the
only structure allocated and managed by RCU users. The ``rcu_head``
structure is normally embedded within the RCU-protected data
structure.
If all you wanted from this article was a general notion of how RCU's
data structures are related, you are done. Otherwise, each of the
following sections give more details on the ``rcu_state``, ``rcu_node``
and ``rcu_data`` data structures.
The ``rcu_state`` Structure
~~~~~~~~~~~~~~~~~~~~~~~~~~~
The ``rcu_state`` structure is the base structure that represents the
state of RCU in the system. This structure forms the interconnection
between the ``rcu_node`` and ``rcu_data`` structures, tracks grace
periods, contains the lock used to synchronize with CPU-hotplug events,
and maintains state used to force quiescent states when grace periods
extend too long,
A few of the ``rcu_state`` structure's fields are discussed, singly and
in groups, in the following sections. The more specialized fields are
covered in the discussion of their use.
Relationship to rcu_node and rcu_data Structures
''''''''''''''''''''''''''''''''''''''''''''''''
This portion of the ``rcu_state`` structure is declared as follows:
::
1 struct rcu_node node[NUM_RCU_NODES];
2 struct rcu_node *level[NUM_RCU_LVLS + 1];
3 struct rcu_data __percpu *rda;
+-----------------------------------------------------------------------+
| **Quick Quiz**: |
+-----------------------------------------------------------------------+
| Wait a minute! You said that the ``rcu_node`` structures formed a |
| tree, but they are declared as a flat array! What gives? |
+-----------------------------------------------------------------------+
| **Answer**: |
+-----------------------------------------------------------------------+
| The tree is laid out in the array. The first node In the array is the |
| head, the next set of nodes in the array are children of the head |
| node, and so on until the last set of nodes in the array are the |
| leaves. |
| See the following diagrams to see how this works. |
+-----------------------------------------------------------------------+
The ``rcu_node`` tree is embedded into the ``->node[]`` array as shown
in the following figure:
.. kernel-figure:: TreeMapping.svg
One interesting consequence of this mapping is that a breadth-first
traversal of the tree is implemented as a simple linear scan of the
array, which is in fact what the ``rcu_for_each_node_breadth_first()``
macro does. This macro is used at the beginning and ends of grace
periods.
Each entry of the ``->level`` array references the first ``rcu_node``
structure on the corresponding level of the tree, for example, as shown
below:
.. kernel-figure:: TreeMappingLevel.svg
The zero\ :sup:`th` element of the array references the root
``rcu_node`` structure, the first element references the first child of
the root ``rcu_node``, and finally the second element references the
first leaf ``rcu_node`` structure.
For whatever it is worth, if you draw the tree to be tree-shaped rather
than array-shaped, it is easy to draw a planar representation:
.. kernel-figure:: TreeLevel.svg
Finally, the ``->rda`` field references a per-CPU pointer to the
corresponding CPU's ``rcu_data`` structure.
All of these fields are constant once initialization is complete, and
therefore need no protection.
Grace-Period Tracking
'''''''''''''''''''''
This portion of the ``rcu_state`` structure is declared as follows:
::
1 unsigned long gp_seq;
RCU grace periods are numbered, and the ``->gp_seq`` field contains the
current grace-period sequence number. The bottom two bits are the state
of the current grace period, which can be zero for not yet started or
one for in progress. In other words, if the bottom two bits of
``->gp_seq`` are zero, then RCU is idle. Any other value in the bottom
two bits indicates that something is broken. This field is protected by
the root ``rcu_node`` structure's ``->lock`` field.
There are ``->gp_seq`` fields in the ``rcu_node`` and ``rcu_data``
structures as well. The fields in the ``rcu_state`` structure represent
the most current value, and those of the other structures are compared
in order to detect the beginnings and ends of grace periods in a
distributed fashion. The values flow from ``rcu_state`` to ``rcu_node``
(down the tree from the root to the leaves) to ``rcu_data``.
Miscellaneous
'''''''''''''
This portion of the ``rcu_state`` structure is declared as follows:
::
1 unsigned long gp_max;
2 char abbr;
3 char *name;
The ``->gp_max`` field tracks the duration of the longest grace period
in jiffies. It is protected by the root ``rcu_node``'s ``->lock``.
The ``->name`` and ``->abbr`` fields distinguish between preemptible RCU
(“rcu_preempt” and “p”) and non-preemptible RCU (“rcu_sched” and “s”).
These fields are used for diagnostic and tracing purposes.
The ``rcu_node`` Structure
~~~~~~~~~~~~~~~~~~~~~~~~~~
The ``rcu_node`` structures form the combining tree that propagates
quiescent-state information from the leaves to the root and also that
propagates grace-period information from the root down to the leaves.
They provides local copies of the grace-period state in order to allow
this information to be accessed in a synchronized manner without
suffering the scalability limitations that would otherwise be imposed by
global locking. In ``CONFIG_PREEMPT_RCU`` kernels, they manage the lists
of tasks that have blocked while in their current RCU read-side critical
section. In ``CONFIG_PREEMPT_RCU`` with ``CONFIG_RCU_BOOST``, they
manage the per-\ ``rcu_node`` priority-boosting kernel threads
(kthreads) and state. Finally, they record CPU-hotplug state in order to
determine which CPUs should be ignored during a given grace period.
The ``rcu_node`` structure's fields are discussed, singly and in groups,
in the following sections.
Connection to Combining Tree
''''''''''''''''''''''''''''
This portion of the ``rcu_node`` structure is declared as follows:
::
1 struct rcu_node *parent;
2 u8 level;
3 u8 grpnum;
4 unsigned long grpmask;
5 int grplo;
6 int grphi;
The ``->parent`` pointer references the ``rcu_node`` one level up in the
tree, and is ``NULL`` for the root ``rcu_node``. The RCU implementation
makes heavy use of this field to push quiescent states up the tree. The
``->level`` field gives the level in the tree, with the root being at
level zero, its children at level one, and so on. The ``->grpnum`` field
gives this node's position within the children of its parent, so this
number can range between 0 and 31 on 32-bit systems and between 0 and 63
on 64-bit systems. The ``->level`` and ``->grpnum`` fields are used only
during initialization and for tracing. The ``->grpmask`` field is the
bitmask counterpart of ``->grpnum``, and therefore always has exactly
one bit set. This mask is used to clear the bit corresponding to this
``rcu_node`` structure in its parent's bitmasks, which are described
later. Finally, the ``->grplo`` and ``->grphi`` fields contain the
lowest and highest numbered CPU served by this ``rcu_node`` structure,
respectively.
All of these fields are constant, and thus do not require any
synchronization.
Synchronization
'''''''''''''''
This field of the ``rcu_node`` structure is declared as follows:
::
1 raw_spinlock_t lock;
This field is used to protect the remaining fields in this structure,
unless otherwise stated. That said, all of the fields in this structure
can be accessed without locking for tracing purposes. Yes, this can
result in confusing traces, but better some tracing confusion than to be
heisenbugged out of existence.
.. _grace-period-tracking-1:
Grace-Period Tracking
'''''''''''''''''''''
This portion of the ``rcu_node`` structure is declared as follows:
::
1 unsigned long gp_seq;
2 unsigned long gp_seq_needed;
The ``rcu_node`` structures' ``->gp_seq`` fields are the counterparts of
the field of the same name in the ``rcu_state`` structure. They each may
lag up to one step behind their ``rcu_state`` counterpart. If the bottom
two bits of a given ``rcu_node`` structure's ``->gp_seq`` field is zero,
then this ``rcu_node`` structure believes that RCU is idle.
The ``>gp_seq`` field of each ``rcu_node`` structure is updated at the
beginning and the end of each grace period.
The ``->gp_seq_needed`` fields record the furthest-in-the-future grace
period request seen by the corresponding ``rcu_node`` structure. The
request is considered fulfilled when the value of the ``->gp_seq`` field
equals or exceeds that of the ``->gp_seq_needed`` field.
+-----------------------------------------------------------------------+
| **Quick Quiz**: |
+-----------------------------------------------------------------------+
| Suppose that this ``rcu_node`` structure doesn't see a request for a |
| very long time. Won't wrapping of the ``->gp_seq`` field cause |
| problems? |
+-----------------------------------------------------------------------+
| **Answer**: |
+-----------------------------------------------------------------------+
| No, because if the ``->gp_seq_needed`` field lags behind the |
| ``->gp_seq`` field, the ``->gp_seq_needed`` field will be updated at |
| the end of the grace period. Modulo-arithmetic comparisons therefore |
| will always get the correct answer, even with wrapping. |
+-----------------------------------------------------------------------+
Quiescent-State Tracking
''''''''''''''''''''''''
These fields manage the propagation of quiescent states up the combining
tree.
This portion of the ``rcu_node`` structure has fields as follows:
::
1 unsigned long qsmask;
2 unsigned long expmask;
3 unsigned long qsmaskinit;
4 unsigned long expmaskinit;
The ``->qsmask`` field tracks which of this ``rcu_node`` structure's
children still need to report quiescent states for the current normal
grace period. Such children will have a value of 1 in their
corresponding bit. Note that the leaf ``rcu_node`` structures should be
thought of as having ``rcu_data`` structures as their children.
Similarly, the ``->expmask`` field tracks which of this ``rcu_node``
structure's children still need to report quiescent states for the
current expedited grace period. An expedited grace period has the same
conceptual properties as a normal grace period, but the expedited
implementation accepts extreme CPU overhead to obtain much lower
grace-period latency, for example, consuming a few tens of microseconds
worth of CPU time to reduce grace-period duration from milliseconds to
tens of microseconds. The ``->qsmaskinit`` field tracks which of this
``rcu_node`` structure's children cover for at least one online CPU.
This mask is used to initialize ``->qsmask``, and ``->expmaskinit`` is
used to initialize ``->expmask`` and the beginning of the normal and
expedited grace periods, respectively.
+-----------------------------------------------------------------------+
| **Quick Quiz**: |
+-----------------------------------------------------------------------+
| Why are these bitmasks protected by locking? Come on, haven't you |
| heard of atomic instructions??? |
+-----------------------------------------------------------------------+
| **Answer**: |
+-----------------------------------------------------------------------+
| Lockless grace-period computation! Such a tantalizing possibility! |
| But consider the following sequence of events: |
| |
| #. CPU 0 has been in dyntick-idle mode for quite some time. When it |
| wakes up, it notices that the current RCU grace period needs it to |
| report in, so it sets a flag where the scheduling clock interrupt |
| will find it. |
| #. Meanwhile, CPU 1 is running ``force_quiescent_state()``, and |
| notices that CPU 0 has been in dyntick idle mode, which qualifies |
| as an extended quiescent state. |
| #. CPU 0's scheduling clock interrupt fires in the middle of an RCU |
| read-side critical section, and notices that the RCU core needs |
| something, so commences RCU softirq processing. |
| #. CPU 0's softirq handler executes and is just about ready to report |
| its quiescent state up the ``rcu_node`` tree. |
| #. But CPU 1 beats it to the punch, completing the current grace |
| period and starting a new one. |
| #. CPU 0 now reports its quiescent state for the wrong grace period. |
| That grace period might now end before the RCU read-side critical |
| section. If that happens, disaster will ensue. |
| |
| So the locking is absolutely required in order to coordinate clearing |
| of the bits with updating of the grace-period sequence number in |
| ``->gp_seq``. |
+-----------------------------------------------------------------------+
Blocked-Task Management
'''''''''''''''''''''''
``PREEMPT_RCU`` allows tasks to be preempted in the midst of their RCU
read-side critical sections, and these tasks must be tracked explicitly.
The details of exactly why and how they are tracked will be covered in a
separate article on RCU read-side processing. For now, it is enough to
know that the ``rcu_node`` structure tracks them.
::
1 struct list_head blkd_tasks;
2 struct list_head *gp_tasks;
3 struct list_head *exp_tasks;
4 bool wait_blkd_tasks;
The ``->blkd_tasks`` field is a list header for the list of blocked and
preempted tasks. As tasks undergo context switches within RCU read-side
critical sections, their ``task_struct`` structures are enqueued (via
the ``task_struct``'s ``->rcu_node_entry`` field) onto the head of the
``->blkd_tasks`` list for the leaf ``rcu_node`` structure corresponding
to the CPU on which the outgoing context switch executed. As these tasks
later exit their RCU read-side critical sections, they remove themselves
from the list. This list is therefore in reverse time order, so that if
one of the tasks is blocking the current grace period, all subsequent
tasks must also be blocking that same grace period. Therefore, a single
pointer into this list suffices to track all tasks blocking a given
grace period. That pointer is stored in ``->gp_tasks`` for normal grace
periods and in ``->exp_tasks`` for expedited grace periods. These last
two fields are ``NULL`` if either there is no grace period in flight or
if there are no blocked tasks preventing that grace period from
completing. If either of these two pointers is referencing a task that
removes itself from the ``->blkd_tasks`` list, then that task must
advance the pointer to the next task on the list, or set the pointer to
``NULL`` if there are no subsequent tasks on the list.
For example, suppose that tasks T1, T2, and T3 are all hard-affinitied
to the largest-numbered CPU in the system. Then if task T1 blocked in an
RCU read-side critical section, then an expedited grace period started,
then task T2 blocked in an RCU read-side critical section, then a normal
grace period started, and finally task 3 blocked in an RCU read-side
critical section, then the state of the last leaf ``rcu_node``
structure's blocked-task list would be as shown below:
.. kernel-figure:: blkd_task.svg
Task T1 is blocking both grace periods, task T2 is blocking only the
normal grace period, and task T3 is blocking neither grace period. Note
that these tasks will not remove themselves from this list immediately
upon resuming execution. They will instead remain on the list until they
execute the outermost ``rcu_read_unlock()`` that ends their RCU
read-side critical section.
The ``->wait_blkd_tasks`` field indicates whether or not the current
grace period is waiting on a blocked task.
Sizing the ``rcu_node`` Array
'''''''''''''''''''''''''''''
The ``rcu_node`` array is sized via a series of C-preprocessor
expressions as follows:
::
1 #ifdef CONFIG_RCU_FANOUT
2 #define RCU_FANOUT CONFIG_RCU_FANOUT
3 #else
4 # ifdef CONFIG_64BIT
5 # define RCU_FANOUT 64
6 # else
7 # define RCU_FANOUT 32
8 # endif
9 #endif
10
11 #ifdef CONFIG_RCU_FANOUT_LEAF
12 #define RCU_FANOUT_LEAF CONFIG_RCU_FANOUT_LEAF
13 #else
14 # ifdef CONFIG_64BIT
15 # define RCU_FANOUT_LEAF 64
16 # else
17 # define RCU_FANOUT_LEAF 32
18 # endif
19 #endif
20
21 #define RCU_FANOUT_1 (RCU_FANOUT_LEAF)
22 #define RCU_FANOUT_2 (RCU_FANOUT_1 * RCU_FANOUT)
23 #define RCU_FANOUT_3 (RCU_FANOUT_2 * RCU_FANOUT)
24 #define RCU_FANOUT_4 (RCU_FANOUT_3 * RCU_FANOUT)
25
26 #if NR_CPUS <= RCU_FANOUT_1
27 # define RCU_NUM_LVLS 1
28 # define NUM_RCU_LVL_0 1
29 # define NUM_RCU_NODES NUM_RCU_LVL_0
30 # define NUM_RCU_LVL_INIT { NUM_RCU_LVL_0 }
31 # define RCU_NODE_NAME_INIT { "rcu_node_0" }
32 # define RCU_FQS_NAME_INIT { "rcu_node_fqs_0" }
33 # define RCU_EXP_NAME_INIT { "rcu_node_exp_0" }
34 #elif NR_CPUS <= RCU_FANOUT_2
35 # define RCU_NUM_LVLS 2
36 # define NUM_RCU_LVL_0 1
37 # define NUM_RCU_LVL_1 DIV_ROUND_UP(NR_CPUS, RCU_FANOUT_1)
38 # define NUM_RCU_NODES (NUM_RCU_LVL_0 + NUM_RCU_LVL_1)
39 # define NUM_RCU_LVL_INIT { NUM_RCU_LVL_0, NUM_RCU_LVL_1 }
40 # define RCU_NODE_NAME_INIT { "rcu_node_0", "rcu_node_1" }
41 # define RCU_FQS_NAME_INIT { "rcu_node_fqs_0", "rcu_node_fqs_1" }
42 # define RCU_EXP_NAME_INIT { "rcu_node_exp_0", "rcu_node_exp_1" }
43 #elif NR_CPUS <= RCU_FANOUT_3
44 # define RCU_NUM_LVLS 3
45 # define NUM_RCU_LVL_0 1
46 # define NUM_RCU_LVL_1 DIV_ROUND_UP(NR_CPUS, RCU_FANOUT_2)
47 # define NUM_RCU_LVL_2 DIV_ROUND_UP(NR_CPUS, RCU_FANOUT_1)
48 # define NUM_RCU_NODES (NUM_RCU_LVL_0 + NUM_RCU_LVL_1 + NUM_RCU_LVL_2)
49 # define NUM_RCU_LVL_INIT { NUM_RCU_LVL_0, NUM_RCU_LVL_1, NUM_RCU_LVL_2 }
50 # define RCU_NODE_NAME_INIT { "rcu_node_0", "rcu_node_1", "rcu_node_2" }
51 # define RCU_FQS_NAME_INIT { "rcu_node_fqs_0", "rcu_node_fqs_1", "rcu_node_fqs_2" }
52 # define RCU_EXP_NAME_INIT { "rcu_node_exp_0", "rcu_node_exp_1", "rcu_node_exp_2" }
53 #elif NR_CPUS <= RCU_FANOUT_4
54 # define RCU_NUM_LVLS 4
55 # define NUM_RCU_LVL_0 1
56 # define NUM_RCU_LVL_1 DIV_ROUND_UP(NR_CPUS, RCU_FANOUT_3)
57 # define NUM_RCU_LVL_2 DIV_ROUND_UP(NR_CPUS, RCU_FANOUT_2)
58 # define NUM_RCU_LVL_3 DIV_ROUND_UP(NR_CPUS, RCU_FANOUT_1)
59 # define NUM_RCU_NODES (NUM_RCU_LVL_0 + NUM_RCU_LVL_1 + NUM_RCU_LVL_2 + NUM_RCU_LVL_3)
60 # define NUM_RCU_LVL_INIT { NUM_RCU_LVL_0, NUM_RCU_LVL_1, NUM_RCU_LVL_2, NUM_RCU_LVL_3 }
61 # define RCU_NODE_NAME_INIT { "rcu_node_0", "rcu_node_1", "rcu_node_2", "rcu_node_3" }
62 # define RCU_FQS_NAME_INIT { "rcu_node_fqs_0", "rcu_node_fqs_1", "rcu_node_fqs_2", "rcu_node_fqs_3" }
63 # define RCU_EXP_NAME_INIT { "rcu_node_exp_0", "rcu_node_exp_1", "rcu_node_exp_2", "rcu_node_exp_3" }
64 #else
65 # error "CONFIG_RCU_FANOUT insufficient for NR_CPUS"
66 #endif
The maximum number of levels in the ``rcu_node`` structure is currently
limited to four, as specified by lines 21-24 and the structure of the
subsequent “if” statement. For 32-bit systems, this allows
16*32*32*32=524,288 CPUs, which should be sufficient for the next few
years at least. For 64-bit systems, 16*64*64*64=4,194,304 CPUs is
allowed, which should see us through the next decade or so. This
four-level tree also allows kernels built with ``CONFIG_RCU_FANOUT=8``
to support up to 4096 CPUs, which might be useful in very large systems
having eight CPUs per socket (but please note that no one has yet shown
any measurable performance degradation due to misaligned socket and
``rcu_node`` boundaries). In addition, building kernels with a full four
levels of ``rcu_node`` tree permits better testing of RCU's
combining-tree code.
The ``RCU_FANOUT`` symbol controls how many children are permitted at
each non-leaf level of the ``rcu_node`` tree. If the
``CONFIG_RCU_FANOUT`` Kconfig option is not specified, it is set based
on the word size of the system, which is also the Kconfig default.
The ``RCU_FANOUT_LEAF`` symbol controls how many CPUs are handled by
each leaf ``rcu_node`` structure. Experience has shown that allowing a
given leaf ``rcu_node`` structure to handle 64 CPUs, as permitted by the
number of bits in the ``->qsmask`` field on a 64-bit system, results in
excessive contention for the leaf ``rcu_node`` structures' ``->lock``
fields. The number of CPUs per leaf ``rcu_node`` structure is therefore
limited to 16 given the default value of ``CONFIG_RCU_FANOUT_LEAF``. If
``CONFIG_RCU_FANOUT_LEAF`` is unspecified, the value selected is based
on the word size of the system, just as for ``CONFIG_RCU_FANOUT``.
Lines 11-19 perform this computation.
Lines 21-24 compute the maximum number of CPUs supported by a
single-level (which contains a single ``rcu_node`` structure),
two-level, three-level, and four-level ``rcu_node`` tree, respectively,
given the fanout specified by ``RCU_FANOUT`` and ``RCU_FANOUT_LEAF``.
These numbers of CPUs are retained in the ``RCU_FANOUT_1``,
``RCU_FANOUT_2``, ``RCU_FANOUT_3``, and ``RCU_FANOUT_4`` C-preprocessor
variables, respectively.
These variables are used to control the C-preprocessor ``#if`` statement
spanning lines 26-66 that computes the number of ``rcu_node`` structures
required for each level of the tree, as well as the number of levels
required. The number of levels is placed in the ``NUM_RCU_LVLS``
C-preprocessor variable by lines 27, 35, 44, and 54. The number of
``rcu_node`` structures for the topmost level of the tree is always
exactly one, and this value is unconditionally placed into
``NUM_RCU_LVL_0`` by lines 28, 36, 45, and 55. The rest of the levels
(if any) of the ``rcu_node`` tree are computed by dividing the maximum
number of CPUs by the fanout supported by the number of levels from the
current level down, rounding up. This computation is performed by
lines 37, 46-47, and 56-58. Lines 31-33, 40-42, 50-52, and 62-63 create
initializers for lockdep lock-class names. Finally, lines 64-66 produce
an error if the maximum number of CPUs is too large for the specified
fanout.
The ``rcu_segcblist`` Structure
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
The ``rcu_segcblist`` structure maintains a segmented list of callbacks
as follows:
::
1 #define RCU_DONE_TAIL 0
2 #define RCU_WAIT_TAIL 1
3 #define RCU_NEXT_READY_TAIL 2
4 #define RCU_NEXT_TAIL 3
5 #define RCU_CBLIST_NSEGS 4
6
7 struct rcu_segcblist {
8 struct rcu_head *head;
9 struct rcu_head **tails[RCU_CBLIST_NSEGS];
10 unsigned long gp_seq[RCU_CBLIST_NSEGS];
11 long len;
12 long len_lazy;
13 };
The segments are as follows:
#. ``RCU_DONE_TAIL``: Callbacks whose grace periods have elapsed. These
callbacks are ready to be invoked.
#. ``RCU_WAIT_TAIL``: Callbacks that are waiting for the current grace
period. Note that different CPUs can have different ideas about which
grace period is current, hence the ``->gp_seq`` field.
#. ``RCU_NEXT_READY_TAIL``: Callbacks waiting for the next grace period
to start.
#. ``RCU_NEXT_TAIL``: Callbacks that have not yet been associated with a
grace period.
The ``->head`` pointer references the first callback or is ``NULL`` if
the list contains no callbacks (which is *not* the same as being empty).
Each element of the ``->tails[]`` array references the ``->next``
pointer of the last callback in the corresponding segment of the list,
or the list's ``->head`` pointer if that segment and all previous
segments are empty. If the corresponding segment is empty but some
previous segment is not empty, then the array element is identical to
its predecessor. Older callbacks are closer to the head of the list, and
new callbacks are added at the tail. This relationship between the
``->head`` pointer, the ``->tails[]`` array, and the callbacks is shown
in this diagram:
.. kernel-figure:: nxtlist.svg
In this figure, the ``->head`` pointer references the first RCU callback
in the list. The ``->tails[RCU_DONE_TAIL]`` array element references the
``->head`` pointer itself, indicating that none of the callbacks is
ready to invoke. The ``->tails[RCU_WAIT_TAIL]`` array element references
callback CB 2's ``->next`` pointer, which indicates that CB 1 and CB 2
are both waiting on the current grace period, give or take possible
disagreements about exactly which grace period is the current one. The
``->tails[RCU_NEXT_READY_TAIL]`` array element references the same RCU
callback that ``->tails[RCU_WAIT_TAIL]`` does, which indicates that
there are no callbacks waiting on the next RCU grace period. The
``->tails[RCU_NEXT_TAIL]`` array element references CB 4's ``->next``
pointer, indicating that all the remaining RCU callbacks have not yet
been assigned to an RCU grace period. Note that the
``->tails[RCU_NEXT_TAIL]`` array element always references the last RCU
callback's ``->next`` pointer unless the callback list is empty, in
which case it references the ``->head`` pointer.
There is one additional important special case for the
``->tails[RCU_NEXT_TAIL]`` array element: It can be ``NULL`` when this
list is *disabled*. Lists are disabled when the corresponding CPU is
offline or when the corresponding CPU's callbacks are offloaded to a
kthread, both of which are described elsewhere.
CPUs advance their callbacks from the ``RCU_NEXT_TAIL`` to the
``RCU_NEXT_READY_TAIL`` to the ``RCU_WAIT_TAIL`` to the
``RCU_DONE_TAIL`` list segments as grace periods advance.
The ``->gp_seq[]`` array records grace-period numbers corresponding to
the list segments. This is what allows different CPUs to have different
ideas as to which is the current grace period while still avoiding
premature invocation of their callbacks. In particular, this allows CPUs
that go idle for extended periods to determine which of their callbacks
are ready to be invoked after reawakening.
The ``->len`` counter contains the number of callbacks in ``->head``,
and the ``->len_lazy`` contains the number of those callbacks that are
known to only free memory, and whose invocation can therefore be safely
deferred.
.. important::
It is the ``->len`` field that determines whether or
not there are callbacks associated with this ``rcu_segcblist``
structure, *not* the ``->head`` pointer. The reason for this is that all
the ready-to-invoke callbacks (that is, those in the ``RCU_DONE_TAIL``
segment) are extracted all at once at callback-invocation time
(``rcu_do_batch``), due to which ``->head`` may be set to NULL if there
are no not-done callbacks remaining in the ``rcu_segcblist``. If
callback invocation must be postponed, for example, because a
high-priority process just woke up on this CPU, then the remaining
callbacks are placed back on the ``RCU_DONE_TAIL`` segment and
``->head`` once again points to the start of the segment. In short, the
head field can briefly be ``NULL`` even though the CPU has callbacks
present the entire time. Therefore, it is not appropriate to test the
``->head`` pointer for ``NULL``.
In contrast, the ``->len`` and ``->len_lazy`` counts are adjusted only
after the corresponding callbacks have been invoked. This means that the
``->len`` count is zero only if the ``rcu_segcblist`` structure really
is devoid of callbacks. Of course, off-CPU sampling of the ``->len``
count requires careful use of appropriate synchronization, for example,
memory barriers. This synchronization can be a bit subtle, particularly
in the case of ``rcu_barrier()``.
The ``rcu_data`` Structure
~~~~~~~~~~~~~~~~~~~~~~~~~~
The ``rcu_data`` maintains the per-CPU state for the RCU subsystem. The
fields in this structure may be accessed only from the corresponding CPU
(and from tracing) unless otherwise stated. This structure is the focus
of quiescent-state detection and RCU callback queuing. It also tracks
its relationship to the corresponding leaf ``rcu_node`` structure to
allow more-efficient propagation of quiescent states up the ``rcu_node``
combining tree. Like the ``rcu_node`` structure, it provides a local
copy of the grace-period information to allow for-free synchronized
access to this information from the corresponding CPU. Finally, this
structure records past dyntick-idle state for the corresponding CPU and
also tracks statistics.
The ``rcu_data`` structure's fields are discussed, singly and in groups,
in the following sections.
Connection to Other Data Structures
'''''''''''''''''''''''''''''''''''
This portion of the ``rcu_data`` structure is declared as follows:
::
1 int cpu;
2 struct rcu_node *mynode;
3 unsigned long grpmask;
4 bool beenonline;
The ``->cpu`` field contains the number of the corresponding CPU and the
``->mynode`` field references the corresponding ``rcu_node`` structure.
The ``->mynode`` is used to propagate quiescent states up the combining
tree. These two fields are constant and therefore do not require
synchronization.
The ``->grpmask`` field indicates the bit in the ``->mynode->qsmask``
corresponding to this ``rcu_data`` structure, and is also used when
propagating quiescent states. The ``->beenonline`` flag is set whenever
the corresponding CPU comes online, which means that the debugfs tracing
need not dump out any ``rcu_data`` structure for which this flag is not
set.
Quiescent-State and Grace-Period Tracking
'''''''''''''''''''''''''''''''''''''''''
This portion of the ``rcu_data`` structure is declared as follows:
::
1 unsigned long gp_seq;
2 unsigned long gp_seq_needed;
3 bool cpu_no_qs;
4 bool core_needs_qs;
5 bool gpwrap;
The ``->gp_seq`` field is the counterpart of the field of the same name
in the ``rcu_state`` and ``rcu_node`` structures. The
``->gp_seq_needed`` field is the counterpart of the field of the same
name in the rcu_node structure. They may each lag up to one behind their
``rcu_node`` counterparts, but in ``CONFIG_NO_HZ_IDLE`` and
``CONFIG_NO_HZ_FULL`` kernels can lag arbitrarily far behind for CPUs in
dyntick-idle mode (but these counters will catch up upon exit from
dyntick-idle mode). If the lower two bits of a given ``rcu_data``
structure's ``->gp_seq`` are zero, then this ``rcu_data`` structure
believes that RCU is idle.
+-----------------------------------------------------------------------+
| **Quick Quiz**: |
+-----------------------------------------------------------------------+
| All this replication of the grace period numbers can only cause |
| massive confusion. Why not just keep a global sequence number and be |
| done with it??? |
+-----------------------------------------------------------------------+
| **Answer**: |
+-----------------------------------------------------------------------+
| Because if there was only a single global sequence numbers, there |
| would need to be a single global lock to allow safely accessing and |
| updating it. And if we are not going to have a single global lock, we |
| need to carefully manage the numbers on a per-node basis. Recall from |
| the answer to a previous Quick Quiz that the consequences of applying |
| a previously sampled quiescent state to the wrong grace period are |
| quite severe. |
+-----------------------------------------------------------------------+
The ``->cpu_no_qs`` flag indicates that the CPU has not yet passed
through a quiescent state, while the ``->core_needs_qs`` flag indicates
that the RCU core needs a quiescent state from the corresponding CPU.
The ``->gpwrap`` field indicates that the corresponding CPU has remained
idle for so long that the ``gp_seq`` counter is in danger of overflow,
which will cause the CPU to disregard the values of its counters on its
next exit from idle.
RCU Callback Handling
'''''''''''''''''''''
In the absence of CPU-hotplug events, RCU callbacks are invoked by the
same CPU that registered them. This is strictly a cache-locality
optimization: callbacks can and do get invoked on CPUs other than the
one that registered them. After all, if the CPU that registered a given
callback has gone offline before the callback can be invoked, there
really is no other choice.
This portion of the ``rcu_data`` structure is declared as follows:
::
1 struct rcu_segcblist cblist;
2 long qlen_last_fqs_check;
3 unsigned long n_cbs_invoked;
4 unsigned long n_nocbs_invoked;
5 unsigned long n_cbs_orphaned;
6 unsigned long n_cbs_adopted;
7 unsigned long n_force_qs_snap;
8 long blimit;
The ``->cblist`` structure is the segmented callback list described
earlier. The CPU advances the callbacks in its ``rcu_data`` structure
whenever it notices that another RCU grace period has completed. The CPU
detects the completion of an RCU grace period by noticing that the value
of its ``rcu_data`` structure's ``->gp_seq`` field differs from that of
its leaf ``rcu_node`` structure. Recall that each ``rcu_node``
structure's ``->gp_seq`` field is updated at the beginnings and ends of
each grace period.
The ``->qlen_last_fqs_check`` and ``->n_force_qs_snap`` coordinate the
forcing of quiescent states from ``call_rcu()`` and friends when
callback lists grow excessively long.
The ``->n_cbs_invoked``, ``->n_cbs_orphaned``, and ``->n_cbs_adopted``
fields count the number of callbacks invoked, sent to other CPUs when
this CPU goes offline, and received from other CPUs when those other
CPUs go offline. The ``->n_nocbs_invoked`` is used when the CPU's
callbacks are offloaded to a kthread.
Finally, the ``->blimit`` counter is the maximum number of RCU callbacks
that may be invoked at a given time.
Dyntick-Idle Handling
'''''''''''''''''''''
This portion of the ``rcu_data`` structure is declared as follows:
::
1 int dynticks_snap;
2 unsigned long dynticks_fqs;
The ``->dynticks_snap`` field is used to take a snapshot of the
corresponding CPU's dyntick-idle state when forcing quiescent states,
and is therefore accessed from other CPUs. Finally, the
``->dynticks_fqs`` field is used to count the number of times this CPU
is determined to be in dyntick-idle state, and is used for tracing and
debugging purposes.
This portion of the rcu_data structure is declared as follows:
::
1 long dynticks_nesting;
2 long dynticks_nmi_nesting;
3 atomic_t dynticks;
4 bool rcu_need_heavy_qs;
5 bool rcu_urgent_qs;
These fields in the rcu_data structure maintain the per-CPU dyntick-idle
state for the corresponding CPU. The fields may be accessed only from
the corresponding CPU (and from tracing) unless otherwise stated.
The ``->dynticks_nesting`` field counts the nesting depth of process
execution, so that in normal circumstances this counter has value zero
or one. NMIs, irqs, and tracers are counted by the
``->dynticks_nmi_nesting`` field. Because NMIs cannot be masked, changes
to this variable have to be undertaken carefully using an algorithm
provided by Andy Lutomirski. The initial transition from idle adds one,
and nested transitions add two, so that a nesting level of five is
represented by a ``->dynticks_nmi_nesting`` value of nine. This counter
can therefore be thought of as counting the number of reasons why this
CPU cannot be permitted to enter dyntick-idle mode, aside from
process-level transitions.
However, it turns out that when running in non-idle kernel context, the
Linux kernel is fully capable of entering interrupt handlers that never
exit and perhaps also vice versa. Therefore, whenever the
``->dynticks_nesting`` field is incremented up from zero, the
``->dynticks_nmi_nesting`` field is set to a large positive number, and
whenever the ``->dynticks_nesting`` field is decremented down to zero,
the the ``->dynticks_nmi_nesting`` field is set to zero. Assuming that
the number of misnested interrupts is not sufficient to overflow the
counter, this approach corrects the ``->dynticks_nmi_nesting`` field
every time the corresponding CPU enters the idle loop from process
context.
The ``->dynticks`` field counts the corresponding CPU's transitions to
and from either dyntick-idle or user mode, so that this counter has an
even value when the CPU is in dyntick-idle mode or user mode and an odd
value otherwise. The transitions to/from user mode need to be counted
for user mode adaptive-ticks support (see timers/NO_HZ.txt).
The ``->rcu_need_heavy_qs`` field is used to record the fact that the
RCU core code would really like to see a quiescent state from the
corresponding CPU, so much so that it is willing to call for
heavy-weight dyntick-counter operations. This flag is checked by RCU's
context-switch and ``cond_resched()`` code, which provide a momentary
idle sojourn in response.
Finally, the ``->rcu_urgent_qs`` field is used to record the fact that
the RCU core code would really like to see a quiescent state from the
corresponding CPU, with the various other fields indicating just how
badly RCU wants this quiescent state. This flag is checked by RCU's
context-switch path (``rcu_note_context_switch``) and the cond_resched
code.
+-----------------------------------------------------------------------+
| **Quick Quiz**: |
+-----------------------------------------------------------------------+
| Why not simply combine the ``->dynticks_nesting`` and |
| ``->dynticks_nmi_nesting`` counters into a single counter that just |
| counts the number of reasons that the corresponding CPU is non-idle? |
+-----------------------------------------------------------------------+
| **Answer**: |
+-----------------------------------------------------------------------+
| Because this would fail in the presence of interrupts whose handlers |
| never return and of handlers that manage to return from a made-up |
| interrupt. |
+-----------------------------------------------------------------------+
Additional fields are present for some special-purpose builds, and are
discussed separately.
The ``rcu_head`` Structure
~~~~~~~~~~~~~~~~~~~~~~~~~~
Each ``rcu_head`` structure represents an RCU callback. These structures
are normally embedded within RCU-protected data structures whose
algorithms use asynchronous grace periods. In contrast, when using
algorithms that block waiting for RCU grace periods, RCU users need not
provide ``rcu_head`` structures.
The ``rcu_head`` structure has fields as follows:
::
1 struct rcu_head *next;
2 void (*func)(struct rcu_head *head);
The ``->next`` field is used to link the ``rcu_head`` structures
together in the lists within the ``rcu_data`` structures. The ``->func``
field is a pointer to the function to be called when the callback is
ready to be invoked, and this function is passed a pointer to the
``rcu_head`` structure. However, ``kfree_rcu()`` uses the ``->func``
field to record the offset of the ``rcu_head`` structure within the
enclosing RCU-protected data structure.
Both of these fields are used internally by RCU. From the viewpoint of
RCU users, this structure is an opaque “cookie”.
+-----------------------------------------------------------------------+
| **Quick Quiz**: |
+-----------------------------------------------------------------------+
| Given that the callback function ``->func`` is passed a pointer to |
| the ``rcu_head`` structure, how is that function supposed to find the |
| beginning of the enclosing RCU-protected data structure? |
+-----------------------------------------------------------------------+
| **Answer**: |
+-----------------------------------------------------------------------+
| In actual practice, there is a separate callback function per type of |
| RCU-protected data structure. The callback function can therefore use |
| the ``container_of()`` macro in the Linux kernel (or other |
| pointer-manipulation facilities in other software environments) to |
| find the beginning of the enclosing structure. |
+-----------------------------------------------------------------------+
RCU-Specific Fields in the ``task_struct`` Structure
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
The ``CONFIG_PREEMPT_RCU`` implementation uses some additional fields in
the ``task_struct`` structure:
::
1 #ifdef CONFIG_PREEMPT_RCU
2 int rcu_read_lock_nesting;
3 union rcu_special rcu_read_unlock_special;
4 struct list_head rcu_node_entry;
5 struct rcu_node *rcu_blocked_node;
6 #endif /* #ifdef CONFIG_PREEMPT_RCU */
7 #ifdef CONFIG_TASKS_RCU
8 unsigned long rcu_tasks_nvcsw;
9 bool rcu_tasks_holdout;
10 struct list_head rcu_tasks_holdout_list;
11 int rcu_tasks_idle_cpu;
12 #endif /* #ifdef CONFIG_TASKS_RCU */
The ``->rcu_read_lock_nesting`` field records the nesting level for RCU
read-side critical sections, and the ``->rcu_read_unlock_special`` field
is a bitmask that records special conditions that require
``rcu_read_unlock()`` to do additional work. The ``->rcu_node_entry``
field is used to form lists of tasks that have blocked within
preemptible-RCU read-side critical sections and the
``->rcu_blocked_node`` field references the ``rcu_node`` structure whose
list this task is a member of, or ``NULL`` if it is not blocked within a
preemptible-RCU read-side critical section.
The ``->rcu_tasks_nvcsw`` field tracks the number of voluntary context
switches that this task had undergone at the beginning of the current
tasks-RCU grace period, ``->rcu_tasks_holdout`` is set if the current
tasks-RCU grace period is waiting on this task,
``->rcu_tasks_holdout_list`` is a list element enqueuing this task on
the holdout list, and ``->rcu_tasks_idle_cpu`` tracks which CPU this
idle task is running, but only if the task is currently running, that
is, if the CPU is currently idle.
Accessor Functions
~~~~~~~~~~~~~~~~~~
The following listing shows the ``rcu_get_root()``,
``rcu_for_each_node_breadth_first`` and ``rcu_for_each_leaf_node()``
function and macros:
::
1 static struct rcu_node *rcu_get_root(struct rcu_state *rsp)
2 {
3 return &rsp->node[0];
4 }
5
6 #define rcu_for_each_node_breadth_first(rsp, rnp) \
7 for ((rnp) = &(rsp)->node[0]; \
8 (rnp) < &(rsp)->node[NUM_RCU_NODES]; (rnp)++)
9
10 #define rcu_for_each_leaf_node(rsp, rnp) \
11 for ((rnp) = (rsp)->level[NUM_RCU_LVLS - 1]; \
12 (rnp) < &(rsp)->node[NUM_RCU_NODES]; (rnp)++)
The ``rcu_get_root()`` simply returns a pointer to the first element of
the specified ``rcu_state`` structure's ``->node[]`` array, which is the
root ``rcu_node`` structure.
As noted earlier, the ``rcu_for_each_node_breadth_first()`` macro takes
advantage of the layout of the ``rcu_node`` structures in the
``rcu_state`` structure's ``->node[]`` array, performing a breadth-first
traversal by simply traversing the array in order. Similarly, the
``rcu_for_each_leaf_node()`` macro traverses only the last part of the
array, thus traversing only the leaf ``rcu_node`` structures.
+-----------------------------------------------------------------------+
| **Quick Quiz**: |
+-----------------------------------------------------------------------+
| What does ``rcu_for_each_leaf_node()`` do if the ``rcu_node`` tree |
| contains only a single node? |
+-----------------------------------------------------------------------+
| **Answer**: |
+-----------------------------------------------------------------------+
| In the single-node case, ``rcu_for_each_leaf_node()`` traverses the |
| single node. |
+-----------------------------------------------------------------------+
Summary
~~~~~~~
So the state of RCU is represented by an ``rcu_state`` structure, which
contains a combining tree of ``rcu_node`` and ``rcu_data`` structures.
Finally, in ``CONFIG_NO_HZ_IDLE`` kernels, each CPU's dyntick-idle state
is tracked by dynticks-related fields in the ``rcu_data`` structure. If
you made it this far, you are well prepared to read the code
walkthroughs in the other articles in this series.
Acknowledgments
~~~~~~~~~~~~~~~
I owe thanks to Cyrill Gorcunov, Mathieu Desnoyers, Dhaval Giani, Paul
Turner, Abhishek Srivastava, Matt Kowalczyk, and Serge Hallyn for
helping me get this document into a more human-readable state.
Legal Statement
~~~~~~~~~~~~~~~
This work represents the view of the author and does not necessarily
represent the view of IBM.
Linux is a registered trademark of Linus Torvalds.
Other company, product, and service names may be trademarks or service
marks of others.
Documentation/RCU/Design/Expedited-Grace-Periods/Expedited-Grace-Periods.html deleted 100644 → 0
View file @ c07e6f36
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN"
"http://www.w3.org/TR/html4/loose.dtd">
<html>
<head><title>A Tour Through TREE_RCU's Expedited Grace Periods</title>
<meta HTTP-EQUIV="Content-Type" CONTENT="text/html; charset=iso-8859-1">
<h2>Introduction</h2>
This document describes RCU's expedited grace periods.
Unlike RCU's normal grace periods, which accept long latencies to attain
high efficiency and minimal disturbance, expedited grace periods accept
lower efficiency and significant disturbance to attain shorter latencies.
<p>
There are two flavors of RCU (RCU-preempt and RCU-sched), with an earlier
third RCU-bh flavor having been implemented in terms of the other two.
Each of the two implementations is covered in its own section.
<ol>
<li> <a href="#Expedited Grace Period Design">
Expedited Grace Period Design</a>
<li> <a href="#RCU-preempt Expedited Grace Periods">
RCU-preempt Expedited Grace Periods</a>
<li> <a href="#RCU-sched Expedited Grace Periods">
RCU-sched Expedited Grace Periods</a>
<li> <a href="#Expedited Grace Period and CPU Hotplug">
Expedited Grace Period and CPU Hotplug</a>
<li> <a href="#Expedited Grace Period Refinements">
Expedited Grace Period Refinements</a>
</ol>
<h2><a name="Expedited Grace Period Design">
Expedited Grace Period Design</a></h2>
<p>
The expedited RCU grace periods cannot be accused of being subtle,
given that they for all intents and purposes hammer every CPU that
has not yet provided a quiescent state for the current expedited
grace period.
The one saving grace is that the hammer has grown a bit smaller
over time: The old call to <tt>try_stop_cpus()</tt> has been
replaced with a set of calls to <tt>smp_call_function_single()</tt>,
each of which results in an IPI to the target CPU.
The corresponding handler function checks the CPU's state, motivating
a faster quiescent state where possible, and triggering a report
of that quiescent state.
As always for RCU, once everything has spent some time in a quiescent
state, the expedited grace period has completed.
<p>
The details of the <tt>smp_call_function_single()</tt> handler's
operation depend on the RCU flavor, as described in the following
sections.
<h2><a name="RCU-preempt Expedited Grace Periods">
RCU-preempt Expedited Grace Periods</a></h2>
<p>
<tt>CONFIG_PREEMPT=y</tt> kernels implement RCU-preempt.
The overall flow of the handling of a given CPU by an RCU-preempt
expedited grace period is shown in the following diagram:
<p><img src="ExpRCUFlow.svg" alt="ExpRCUFlow.svg" width="55%">
<p>
The solid arrows denote direct action, for example, a function call.
The dotted arrows denote indirect action, for example, an IPI
or a state that is reached after some time.
<p>
If a given CPU is offline or idle, <tt>synchronize_rcu_expedited()</tt>
will ignore it because idle and offline CPUs are already residing
in quiescent states.
Otherwise, the expedited grace period will use
<tt>smp_call_function_single()</tt> to send the CPU an IPI, which
is handled by <tt>rcu_exp_handler()</tt>.
<p>
However, because this is preemptible RCU, <tt>rcu_exp_handler()</tt>
can check to see if the CPU is currently running in an RCU read-side
critical section.
If not, the handler can immediately report a quiescent state.
Otherwise, it sets flags so that the outermost <tt>rcu_read_unlock()</tt>
invocation will provide the needed quiescent-state report.
This flag-setting avoids the previous forced preemption of all
CPUs that might have RCU read-side critical sections.
In addition, this flag-setting is done so as to avoid increasing
the overhead of the common-case fastpath through the scheduler.
<p>
Again because this is preemptible RCU, an RCU read-side critical section
can be preempted.
When that happens, RCU will enqueue the task, which will the continue to
block the current expedited grace period until it resumes and finds its
outermost <tt>rcu_read_unlock()</tt>.
The CPU will report a quiescent state just after enqueuing the task because
the CPU is no longer blocking the grace period.
It is instead the preempted task doing the blocking.
The list of blocked tasks is managed by <tt>rcu_preempt_ctxt_queue()</tt>,
which is called from <tt>rcu_preempt_note_context_switch()</tt>, which
in turn is called from <tt>rcu_note_context_switch()</tt>, which in
turn is called from the scheduler.
<table>
<tr><th>&nbsp;</th></tr>
<tr><th align="left">Quick Quiz:</th></tr>
<tr><td>
Why not just have the expedited grace period check the
state of all the CPUs?
After all, that would avoid all those real-time-unfriendly IPIs.
</td></tr>
<tr><th align="left">Answer:</th></tr>
<tr><td bgcolor="#ffffff"><font color="ffffff">
Because we want the RCU read-side critical sections to run fast,
which means no memory barriers.
Therefore, it is not possible to safely check the state from some
other CPU.
And even if it was possible to safely check the state, it would
still be necessary to IPI the CPU to safely interact with the
upcoming <tt>rcu_read_unlock()</tt> invocation, which means that
the remote state testing would not help the worst-case
latency that real-time applications care about.
<p><font color="ffffff">One way to prevent your real-time
application from getting hit with these IPIs is to
build your kernel with <tt>CONFIG_NO_HZ_FULL=y</tt>.
RCU would then perceive the CPU running your application
as being idle, and it would be able to safely detect that
state without needing to IPI the CPU.
</font></td></tr>
<tr><td>&nbsp;</td></tr>
</table>
<p>
Please note that this is just the overall flow:
Additional complications can arise due to races with CPUs going idle
or offline, among other things.
<h2><a name="RCU-sched Expedited Grace Periods">
RCU-sched Expedited Grace Periods</a></h2>
<p>
<tt>CONFIG_PREEMPT=n</tt> kernels implement RCU-sched.
The overall flow of the handling of a given CPU by an RCU-sched
expedited grace period is shown in the following diagram:
<p><img src="ExpSchedFlow.svg" alt="ExpSchedFlow.svg" width="55%">
<p>
As with RCU-preempt, RCU-sched's
<tt>synchronize_rcu_expedited()</tt> ignores offline and
idle CPUs, again because they are in remotely detectable
quiescent states.
However, because the
<tt>rcu_read_lock_sched()</tt> and <tt>rcu_read_unlock_sched()</tt>
leave no trace of their invocation, in general it is not possible to tell
whether or not the current CPU is in an RCU read-side critical section.
The best that RCU-sched's <tt>rcu_exp_handler()</tt> can do is to check
for idle, on the off-chance that the CPU went idle while the IPI
was in flight.
If the CPU is idle, then <tt>rcu_exp_handler()</tt> reports
the quiescent state.
<p> Otherwise, the handler forces a future context switch by setting the
NEED_RESCHED flag of the current task's thread flag and the CPU preempt
counter.
At the time of the context switch, the CPU reports the quiescent state.
Should the CPU go offline first, it will report the quiescent state
at that time.
<h2><a name="Expedited Grace Period and CPU Hotplug">
Expedited Grace Period and CPU Hotplug</a></h2>
<p>
The expedited nature of expedited grace periods require a much tighter
interaction with CPU hotplug operations than is required for normal
grace periods.
In addition, attempting to IPI offline CPUs will result in splats, but
failing to IPI online CPUs can result in too-short grace periods.
Neither option is acceptable in production kernels.
<p>
The interaction between expedited grace periods and CPU hotplug operations
is carried out at several levels:
<ol>
<li> The number of CPUs that have ever been online is tracked
by the <tt>rcu_state</tt> structure's <tt>-&gt;ncpus</tt>
field.
The <tt>rcu_state</tt> structure's <tt>-&gt;ncpus_snap</tt>
field tracks the number of CPUs that have ever been online
at the beginning of an RCU expedited grace period.
Note that this number never decreases, at least in the absence
of a time machine.
<li> The identities of the CPUs that have ever been online is
tracked by the <tt>rcu_node</tt> structure's
<tt>-&gt;expmaskinitnext</tt> field.
The <tt>rcu_node</tt> structure's <tt>-&gt;expmaskinit</tt>
field tracks the identities of the CPUs that were online
at least once at the beginning of the most recent RCU
expedited grace period.
The <tt>rcu_state</tt> structure's <tt>-&gt;ncpus</tt> and
<tt>-&gt;ncpus_snap</tt> fields are used to detect when
new CPUs have come online for the first time, that is,
when the <tt>rcu_node</tt> structure's <tt>-&gt;expmaskinitnext</tt>
field has changed since the beginning of the last RCU
expedited grace period, which triggers an update of each
<tt>rcu_node</tt> structure's <tt>-&gt;expmaskinit</tt>
field from its <tt>-&gt;expmaskinitnext</tt> field.
<li> Each <tt>rcu_node</tt> structure's <tt>-&gt;expmaskinit</tt>
field is used to initialize that structure's
<tt>-&gt;expmask</tt> at the beginning of each RCU
expedited grace period.
This means that only those CPUs that have been online at least
once will be considered for a given grace period.
<li> Any CPU that goes offline will clear its bit in its leaf
<tt>rcu_node</tt> structure's <tt>-&gt;qsmaskinitnext</tt>
field, so any CPU with that bit clear can safely be ignored.
However, it is possible for a CPU coming online or going offline
to have this bit set for some time while <tt>cpu_online</tt>
returns <tt>false</tt>.
<li> For each non-idle CPU that RCU believes is currently online, the grace
period invokes <tt>smp_call_function_single()</tt>.
If this succeeds, the CPU was fully online.
Failure indicates that the CPU is in the process of coming online
or going offline, in which case it is necessary to wait for a
short time period and try again.
The purpose of this wait (or series of waits, as the case may be)
is to permit a concurrent CPU-hotplug operation to complete.
<li> In the case of RCU-sched, one of the last acts of an outgoing CPU
is to invoke <tt>rcu_report_dead()</tt>, which
reports a quiescent state for that CPU.
However, this is likely paranoia-induced redundancy. <!-- @@@ -->
</ol>
<table>
<tr><th>&nbsp;</th></tr>
<tr><th align="left">Quick Quiz:</th></tr>
<tr><td>
Why all the dancing around with multiple counters and masks
tracking CPUs that were once online?
Why not just have a single set of masks tracking the currently
online CPUs and be done with it?
</td></tr>
<tr><th align="left">Answer:</th></tr>
<tr><td bgcolor="#ffffff"><font color="ffffff">
Maintaining single set of masks tracking the online CPUs <i>sounds</i>
easier, at least until you try working out all the race conditions
between grace-period initialization and CPU-hotplug operations.
For example, suppose initialization is progressing down the
tree while a CPU-offline operation is progressing up the tree.
This situation can result in bits set at the top of the tree
that have no counterparts at the bottom of the tree.
Those bits will never be cleared, which will result in
grace-period hangs.
In short, that way lies madness, to say nothing of a great many
bugs, hangs, and deadlocks.
<p><font color="ffffff">
In contrast, the current multi-mask multi-counter scheme ensures
that grace-period initialization will always see consistent masks
up and down the tree, which brings significant simplifications
over the single-mask method.
<p><font color="ffffff">
This is an instance of
<a href="http://www.cs.columbia.edu/~library/TR-repository/reports/reports-1992/cucs-039-92.ps.gz"><font color="ffffff">
deferring work in order to avoid synchronization</a>.
Lazily recording CPU-hotplug events at the beginning of the next
grace period greatly simplifies maintenance of the CPU-tracking
bitmasks in the <tt>rcu_node</tt> tree.
</font></td></tr>
<tr><td>&nbsp;</td></tr>
</table>
<h2><a name="Expedited Grace Period Refinements">
Expedited Grace Period Refinements</a></h2>
<ol>
<li> <a href="#Idle-CPU Checks">Idle-CPU checks</a>.
<li> <a href="#Batching via Sequence Counter">
Batching via sequence counter</a>.
<li> <a href="#Funnel Locking and Wait/Wakeup">
Funnel locking and wait/wakeup</a>.
<li> <a href="#Use of Workqueues">Use of Workqueues</a>.
<li> <a href="#Stall Warnings">Stall warnings</a>.
<li> <a href="#Mid-Boot Operation">Mid-boot operation</a>.
</ol>
<h3><a name="Idle-CPU Checks">Idle-CPU Checks</a></h3>
<p>
Each expedited grace period checks for idle CPUs when initially forming
the mask of CPUs to be IPIed and again just before IPIing a CPU
(both checks are carried out by <tt>sync_rcu_exp_select_cpus()</tt>).
If the CPU is idle at any time between those two times, the CPU will
not be IPIed.
Instead, the task pushing the grace period forward will include the
idle CPUs in the mask passed to <tt>rcu_report_exp_cpu_mult()</tt>.
<p>
For RCU-sched, there is an additional check:
If the IPI has interrupted the idle loop, then
<tt>rcu_exp_handler()</tt> invokes <tt>rcu_report_exp_rdp()</tt>
to report the corresponding quiescent state.
<p>
For RCU-preempt, there is no specific check for idle in the
IPI handler (<tt>rcu_exp_handler()</tt>), but because
RCU read-side critical sections are not permitted within the
idle loop, if <tt>rcu_exp_handler()</tt> sees that the CPU is within
RCU read-side critical section, the CPU cannot possibly be idle.
Otherwise, <tt>rcu_exp_handler()</tt> invokes
<tt>rcu_report_exp_rdp()</tt> to report the corresponding quiescent
state, regardless of whether or not that quiescent state was due to
the CPU being idle.
<p>
In summary, RCU expedited grace periods check for idle when building
the bitmask of CPUs that must be IPIed, just before sending each IPI,
and (either explicitly or implicitly) within the IPI handler.
<h3><a name="Batching via Sequence Counter">
Batching via Sequence Counter</a></h3>
<p>
If each grace-period request was carried out separately, expedited
grace periods would have abysmal scalability and
problematic high-load characteristics.
Because each grace-period operation can serve an unlimited number of
updates, it is important to <i>batch</i> requests, so that a single
expedited grace-period operation will cover all requests in the
corresponding batch.
<p>
This batching is controlled by a sequence counter named
<tt>-&gt;expedited_sequence</tt> in the <tt>rcu_state</tt> structure.
This counter has an odd value when there is an expedited grace period
in progress and an even value otherwise, so that dividing the counter
value by two gives the number of completed grace periods.
During any given update request, the counter must transition from
even to odd and then back to even, thus indicating that a grace
period has elapsed.
Therefore, if the initial value of the counter is <tt>s</tt>,
the updater must wait until the counter reaches at least the
value <tt>(s+3)&amp;~0x1</tt>.
This counter is managed by the following access functions:
<ol>
<li> <tt>rcu_exp_gp_seq_start()</tt>, which marks the start of
an expedited grace period.
<li> <tt>rcu_exp_gp_seq_end()</tt>, which marks the end of an
expedited grace period.
<li> <tt>rcu_exp_gp_seq_snap()</tt>, which obtains a snapshot of
the counter.
<li> <tt>rcu_exp_gp_seq_done()</tt>, which returns <tt>true</tt>
if a full expedited grace period has elapsed since the
corresponding call to <tt>rcu_exp_gp_seq_snap()</tt>.
</ol>
<p>
Again, only one request in a given batch need actually carry out
a grace-period operation, which means there must be an efficient
way to identify which of many concurrent reqeusts will initiate
the grace period, and that there be an efficient way for the
remaining requests to wait for that grace period to complete.
However, that is the topic of the next section.
<h3><a name="Funnel Locking and Wait/Wakeup">
Funnel Locking and Wait/Wakeup</a></h3>
<p>
The natural way to sort out which of a batch of updaters will initiate
the expedited grace period is to use the <tt>rcu_node</tt> combining
tree, as implemented by the <tt>exp_funnel_lock()</tt> function.
The first updater corresponding to a given grace period arriving
at a given <tt>rcu_node</tt> structure records its desired grace-period
sequence number in the <tt>-&gt;exp_seq_rq</tt> field and moves up
to the next level in the tree.
Otherwise, if the <tt>-&gt;exp_seq_rq</tt> field already contains
the sequence number for the desired grace period or some later one,
the updater blocks on one of four wait queues in the
<tt>-&gt;exp_wq[]</tt> array, using the second-from-bottom
and third-from bottom bits as an index.
An <tt>-&gt;exp_lock</tt> field in the <tt>rcu_node</tt> structure
synchronizes access to these fields.
<p>
An empty <tt>rcu_node</tt> tree is shown in the following diagram,
with the white cells representing the <tt>-&gt;exp_seq_rq</tt> field
and the red cells representing the elements of the
<tt>-&gt;exp_wq[]</tt> array.
<p><img src="Funnel0.svg" alt="Funnel0.svg" width="75%">
<p>
The next diagram shows the situation after the arrival of Task&nbsp;A
and Task&nbsp;B at the leftmost and rightmost leaf <tt>rcu_node</tt>
structures, respectively.
The current value of the <tt>rcu_state</tt> structure's
<tt>-&gt;expedited_sequence</tt> field is zero, so adding three and
clearing the bottom bit results in the value two, which both tasks
record in the <tt>-&gt;exp_seq_rq</tt> field of their respective
<tt>rcu_node</tt> structures:
<p><img src="Funnel1.svg" alt="Funnel1.svg" width="75%">
<p>
Each of Tasks&nbsp;A and&nbsp;B will move up to the root
<tt>rcu_node</tt> structure.
Suppose that Task&nbsp;A wins, recording its desired grace-period sequence
number and resulting in the state shown below:
<p><img src="Funnel2.svg" alt="Funnel2.svg" width="75%">
<p>
Task&nbsp;A now advances to initiate a new grace period, while Task&nbsp;B
moves up to the root <tt>rcu_node</tt> structure, and, seeing that
its desired sequence number is already recorded, blocks on
<tt>-&gt;exp_wq[1]</tt>.
<table>
<tr><th>&nbsp;</th></tr>
<tr><th align="left">Quick Quiz:</th></tr>
<tr><td>
Why <tt>-&gt;exp_wq[1]</tt>?
Given that the value of these tasks' desired sequence number is
two, so shouldn't they instead block on <tt>-&gt;exp_wq[2]</tt>?
</td></tr>
<tr><th align="left">Answer:</th></tr>
<tr><td bgcolor="#ffffff"><font color="ffffff">
No.
<p><font color="ffffff">
Recall that the bottom bit of the desired sequence number indicates
whether or not a grace period is currently in progress.
It is therefore necessary to shift the sequence number right one
bit position to obtain the number of the grace period.
This results in <tt>-&gt;exp_wq[1]</tt>.
</font></td></tr>
<tr><td>&nbsp;</td></tr>
</table>
<p>
If Tasks&nbsp;C and&nbsp;D also arrive at this point, they will compute the
same desired grace-period sequence number, and see that both leaf
<tt>rcu_node</tt> structures already have that value recorded.
They will therefore block on their respective <tt>rcu_node</tt>
structures' <tt>-&gt;exp_wq[1]</tt> fields, as shown below:
<p><img src="Funnel3.svg" alt="Funnel3.svg" width="75%">
<p>
Task&nbsp;A now acquires the <tt>rcu_state</tt> structure's
<tt>-&gt;exp_mutex</tt> and initiates the grace period, which
increments <tt>-&gt;expedited_sequence</tt>.
Therefore, if Tasks&nbsp;E and&nbsp;F arrive, they will compute
a desired sequence number of 4 and will record this value as
shown below:
<p><img src="Funnel4.svg" alt="Funnel4.svg" width="75%">
<p>
Tasks&nbsp;E and&nbsp;F will propagate up the <tt>rcu_node</tt>
combining tree, with Task&nbsp;F blocking on the root <tt>rcu_node</tt>
structure and Task&nbsp;E wait for Task&nbsp;A to finish so that
it can start the next grace period.
The resulting state is as shown below:
<p><img src="Funnel5.svg" alt="Funnel5.svg" width="75%">
<p>
Once the grace period completes, Task&nbsp;A
starts waking up the tasks waiting for this grace period to complete,
increments the <tt>-&gt;expedited_sequence</tt>,
acquires the <tt>-&gt;exp_wake_mutex</tt> and then releases the
<tt>-&gt;exp_mutex</tt>.
This results in the following state:
<p><img src="Funnel6.svg" alt="Funnel6.svg" width="75%">
<p>
Task&nbsp;E can then acquire <tt>-&gt;exp_mutex</tt> and increment
<tt>-&gt;expedited_sequence</tt> to the value three.
If new tasks&nbsp;G and&nbsp;H arrive and moves up the combining tree at the
same time, the state will be as follows:
<p><img src="Funnel7.svg" alt="Funnel7.svg" width="75%">
<p>
Note that three of the root <tt>rcu_node</tt> structure's
waitqueues are now occupied.
However, at some point, Task&nbsp;A will wake up the
tasks blocked on the <tt>-&gt;exp_wq</tt> waitqueues, resulting
in the following state:
<p><img src="Funnel8.svg" alt="Funnel8.svg" width="75%">
<p>
Execution will continue with Tasks&nbsp;E and&nbsp;H completing
their grace periods and carrying out their wakeups.
<table>
<tr><th>&nbsp;</th></tr>
<tr><th align="left">Quick Quiz:</th></tr>
<tr><td>
What happens if Task&nbsp;A takes so long to do its wakeups
that Task&nbsp;E's grace period completes?
</td></tr>
<tr><th align="left">Answer:</th></tr>
<tr><td bgcolor="#ffffff"><font color="ffffff">
Then Task&nbsp;E will block on the <tt>-&gt;exp_wake_mutex</tt>,
which will also prevent it from releasing <tt>-&gt;exp_mutex</tt>,
which in turn will prevent the next grace period from starting.
This last is important in preventing overflow of the
<tt>-&gt;exp_wq[]</tt> array.
</font></td></tr>
<tr><td>&nbsp;</td></tr>
</table>
<h3><a name="Use of Workqueues">Use of Workqueues</a></h3>
<p>
In earlier implementations, the task requesting the expedited
grace period also drove it to completion.
This straightforward approach had the disadvantage of needing to
account for POSIX signals sent to user tasks,
so more recent implemementations use the Linux kernel's
<a href="https://www.kernel.org/doc/Documentation/core-api/workqueue.rst">workqueues</a>.
<p>
The requesting task still does counter snapshotting and funnel-lock
processing, but the task reaching the top of the funnel lock
does a <tt>schedule_work()</tt> (from <tt>_synchronize_rcu_expedited()</tt>
so that a workqueue kthread does the actual grace-period processing.
Because workqueue kthreads do not accept POSIX signals, grace-period-wait
processing need not allow for POSIX signals.
In addition, this approach allows wakeups for the previous expedited
grace period to be overlapped with processing for the next expedited
grace period.
Because there are only four sets of waitqueues, it is necessary to
ensure that the previous grace period's wakeups complete before the
next grace period's wakeups start.
This is handled by having the <tt>-&gt;exp_mutex</tt>
guard expedited grace-period processing and the
<tt>-&gt;exp_wake_mutex</tt> guard wakeups.
The key point is that the <tt>-&gt;exp_mutex</tt> is not released
until the first wakeup is complete, which means that the
<tt>-&gt;exp_wake_mutex</tt> has already been acquired at that point.
This approach ensures that the previous grace period's wakeups can
be carried out while the current grace period is in process, but
that these wakeups will complete before the next grace period starts.
This means that only three waitqueues are required, guaranteeing that
the four that are provided are sufficient.
<h3><a name="Stall Warnings">Stall Warnings</a></h3>
<p>
Expediting grace periods does nothing to speed things up when RCU
readers take too long, and therefore expedited grace periods check
for stalls just as normal grace periods do.
<table>
<tr><th>&nbsp;</th></tr>
<tr><th align="left">Quick Quiz:</th></tr>
<tr><td>
But why not just let the normal grace-period machinery
detect the stalls, given that a given reader must block
both normal and expedited grace periods?
</td></tr>
<tr><th align="left">Answer:</th></tr>
<tr><td bgcolor="#ffffff"><font color="ffffff">
Because it is quite possible that at a given time there
is no normal grace period in progress, in which case the
normal grace period cannot emit a stall warning.
</font></td></tr>
<tr><td>&nbsp;</td></tr>
</table>
The <tt>synchronize_sched_expedited_wait()</tt> function loops waiting
for the expedited grace period to end, but with a timeout set to the
current RCU CPU stall-warning time.
If this time is exceeded, any CPUs or <tt>rcu_node</tt> structures
blocking the current grace period are printed.
Each stall warning results in another pass through the loop, but the
second and subsequent passes use longer stall times.
<h3><a name="Mid-Boot Operation">Mid-boot operation</a></h3>
<p>
The use of workqueues has the advantage that the expedited
grace-period code need not worry about POSIX signals.
Unfortunately, it has the
corresponding disadvantage that workqueues cannot be used until
they are initialized, which does not happen until some time after
the scheduler spawns the first task.
Given that there are parts of the kernel that really do want to
execute grace periods during this mid-boot &ldquo;dead zone&rdquo;,
expedited grace periods must do something else during thie time.
<p>
What they do is to fall back to the old practice of requiring that the
requesting task drive the expedited grace period, as was the case
before the use of workqueues.
However, the requesting task is only required to drive the grace period
during the mid-boot dead zone.
Before mid-boot, a synchronous grace period is a no-op.
Some time after mid-boot, workqueues are used.
<p>
Non-expedited non-SRCU synchronous grace periods must also operate
normally during mid-boot.
This is handled by causing non-expedited grace periods to take the
expedited code path during mid-boot.
<p>
The current code assumes that there are no POSIX signals during
the mid-boot dead zone.
However, if an overwhelming need for POSIX signals somehow arises,
appropriate adjustments can be made to the expedited stall-warning code.
One such adjustment would reinstate the pre-workqueue stall-warning
checks, but only during the mid-boot dead zone.
<p>
With this refinement, synchronous grace periods can now be used from
task context pretty much any time during the life of the kernel.
That is, aside from some points in the suspend, hibernate, or shutdown
code path.
<h3><a name="Summary">
Summary</a></h3>
<p>
Expedited grace periods use a sequence-number approach to promote
batching, so that a single grace-period operation can serve numerous
requests.
A funnel lock is used to efficiently identify the one task out of
a concurrent group that will request the grace period.
All members of the group will block on waitqueues provided in
the <tt>rcu_node</tt> structure.
The actual grace-period processing is carried out by a workqueue.
<p>
CPU-hotplug operations are noted lazily in order to prevent the need
for tight synchronization between expedited grace periods and
CPU-hotplug operations.
The dyntick-idle counters are used to avoid sending IPIs to idle CPUs,
at least in the common case.
RCU-preempt and RCU-sched use different IPI handlers and different
code to respond to the state changes carried out by those handlers,
but otherwise use common code.
<p>
Quiescent states are tracked using the <tt>rcu_node</tt> tree,
and once all necessary quiescent states have been reported,
all tasks waiting on this expedited grace period are awakened.
A pair of mutexes are used to allow one grace period's wakeups
to proceed concurrently with the next grace period's processing.
<p>
This combination of mechanisms allows expedited grace periods to
run reasonably efficiently.
However, for non-time-critical tasks, normal grace periods should be
used instead because their longer duration permits much higher
degrees of batching, and thus much lower per-request overheads.
</body></html>
Documentation/RCU/Design/Expedited-Grace-Periods/Expedited-Grace-Periods.rst 0 → 100644
View file @ ccc9971e
=================================================
A Tour Through TREE_RCU's Expedited Grace Periods
=================================================
Introduction
============
This document describes RCU's expedited grace periods.
Unlike RCU's normal grace periods, which accept long latencies to attain
high efficiency and minimal disturbance, expedited grace periods accept
lower efficiency and significant disturbance to attain shorter latencies.
There are two flavors of RCU (RCU-preempt and RCU-sched), with an earlier
third RCU-bh flavor having been implemented in terms of the other two.
Each of the two implementations is covered in its own section.
Expedited Grace Period Design
=============================
The expedited RCU grace periods cannot be accused of being subtle,
given that they for all intents and purposes hammer every CPU that
has not yet provided a quiescent state for the current expedited
grace period.
The one saving grace is that the hammer has grown a bit smaller
over time: The old call to ``try_stop_cpus()`` has been
replaced with a set of calls to ``smp_call_function_single()``,
each of which results in an IPI to the target CPU.
The corresponding handler function checks the CPU's state, motivating
a faster quiescent state where possible, and triggering a report
of that quiescent state.
As always for RCU, once everything has spent some time in a quiescent
state, the expedited grace period has completed.
The details of the ``smp_call_function_single()`` handler's
operation depend on the RCU flavor, as described in the following
sections.
RCU-preempt Expedited Grace Periods
===================================
``CONFIG_PREEMPT=y`` kernels implement RCU-preempt.
The overall flow of the handling of a given CPU by an RCU-preempt
expedited grace period is shown in the following diagram:
.. kernel-figure:: ExpRCUFlow.svg
The solid arrows denote direct action, for example, a function call.
The dotted arrows denote indirect action, for example, an IPI
or a state that is reached after some time.
If a given CPU is offline or idle, ``synchronize_rcu_expedited()``
will ignore it because idle and offline CPUs are already residing
in quiescent states.
Otherwise, the expedited grace period will use
``smp_call_function_single()`` to send the CPU an IPI, which
is handled by ``rcu_exp_handler()``.
However, because this is preemptible RCU, ``rcu_exp_handler()``
can check to see if the CPU is currently running in an RCU read-side
critical section.
If not, the handler can immediately report a quiescent state.
Otherwise, it sets flags so that the outermost ``rcu_read_unlock()``
invocation will provide the needed quiescent-state report.
This flag-setting avoids the previous forced preemption of all
CPUs that might have RCU read-side critical sections.
In addition, this flag-setting is done so as to avoid increasing
the overhead of the common-case fastpath through the scheduler.
Again because this is preemptible RCU, an RCU read-side critical section
can be preempted.
When that happens, RCU will enqueue the task, which will the continue to
block the current expedited grace period until it resumes and finds its
outermost ``rcu_read_unlock()``.
The CPU will report a quiescent state just after enqueuing the task because
the CPU is no longer blocking the grace period.
It is instead the preempted task doing the blocking.
The list of blocked tasks is managed by ``rcu_preempt_ctxt_queue()``,
which is called from ``rcu_preempt_note_context_switch()``, which
in turn is called from ``rcu_note_context_switch()``, which in
turn is called from the scheduler.
+-----------------------------------------------------------------------+
| **Quick Quiz**: |
+-----------------------------------------------------------------------+
| Why not just have the expedited grace period check the state of all |
| the CPUs? After all, that would avoid all those real-time-unfriendly |
| IPIs. |
+-----------------------------------------------------------------------+
| **Answer**: |
+-----------------------------------------------------------------------+
| Because we want the RCU read-side critical sections to run fast, |
| which means no memory barriers. Therefore, it is not possible to |
| safely check the state from some other CPU. And even if it was |
| possible to safely check the state, it would still be necessary to |
| IPI the CPU to safely interact with the upcoming |
| ``rcu_read_unlock()`` invocation, which means that the remote state |
| testing would not help the worst-case latency that real-time |
| applications care about. |
| |
| One way to prevent your real-time application from getting hit with |
| these IPIs is to build your kernel with ``CONFIG_NO_HZ_FULL=y``. RCU |
| would then perceive the CPU running your application as being idle, |
| and it would be able to safely detect that state without needing to |
| IPI the CPU. |
+-----------------------------------------------------------------------+
Please note that this is just the overall flow: Additional complications
can arise due to races with CPUs going idle or offline, among other
things.
RCU-sched Expedited Grace Periods
---------------------------------
``CONFIG_PREEMPT=n`` kernels implement RCU-sched. The overall flow of
the handling of a given CPU by an RCU-sched expedited grace period is
shown in the following diagram:
.. kernel-figure:: ExpSchedFlow.svg
As with RCU-preempt, RCU-sched's ``synchronize_rcu_expedited()`` ignores
offline and idle CPUs, again because they are in remotely detectable
quiescent states. However, because the ``rcu_read_lock_sched()`` and
``rcu_read_unlock_sched()`` leave no trace of their invocation, in
general it is not possible to tell whether or not the current CPU is in
an RCU read-side critical section. The best that RCU-sched's
``rcu_exp_handler()`` can do is to check for idle, on the off-chance
that the CPU went idle while the IPI was in flight. If the CPU is idle,
then ``rcu_exp_handler()`` reports the quiescent state.
Otherwise, the handler forces a future context switch by setting the
NEED_RESCHED flag of the current task's thread flag and the CPU preempt
counter. At the time of the context switch, the CPU reports the
quiescent state. Should the CPU go offline first, it will report the
quiescent state at that time.
Expedited Grace Period and CPU Hotplug
--------------------------------------
The expedited nature of expedited grace periods require a much tighter
interaction with CPU hotplug operations than is required for normal
grace periods. In addition, attempting to IPI offline CPUs will result
in splats, but failing to IPI online CPUs can result in too-short grace
periods. Neither option is acceptable in production kernels.
The interaction between expedited grace periods and CPU hotplug
operations is carried out at several levels:
#. The number of CPUs that have ever been online is tracked by the
``rcu_state`` structure's ``->ncpus`` field. The ``rcu_state``
structure's ``->ncpus_snap`` field tracks the number of CPUs that
have ever been online at the beginning of an RCU expedited grace
period. Note that this number never decreases, at least in the
absence of a time machine.
#. The identities of the CPUs that have ever been online is tracked by
the ``rcu_node`` structure's ``->expmaskinitnext`` field. The
``rcu_node`` structure's ``->expmaskinit`` field tracks the
identities of the CPUs that were online at least once at the
beginning of the most recent RCU expedited grace period. The
``rcu_state`` structure's ``->ncpus`` and ``->ncpus_snap`` fields are
used to detect when new CPUs have come online for the first time,
that is, when the ``rcu_node`` structure's ``->expmaskinitnext``
field has changed since the beginning of the last RCU expedited grace
period, which triggers an update of each ``rcu_node`` structure's
``->expmaskinit`` field from its ``->expmaskinitnext`` field.
#. Each ``rcu_node`` structure's ``->expmaskinit`` field is used to
initialize that structure's ``->expmask`` at the beginning of each
RCU expedited grace period. This means that only those CPUs that have
been online at least once will be considered for a given grace
period.
#. Any CPU that goes offline will clear its bit in its leaf ``rcu_node``
structure's ``->qsmaskinitnext`` field, so any CPU with that bit
clear can safely be ignored. However, it is possible for a CPU coming
online or going offline to have this bit set for some time while
``cpu_online`` returns ``false``.
#. For each non-idle CPU that RCU believes is currently online, the
grace period invokes ``smp_call_function_single()``. If this
succeeds, the CPU was fully online. Failure indicates that the CPU is
in the process of coming online or going offline, in which case it is
necessary to wait for a short time period and try again. The purpose
of this wait (or series of waits, as the case may be) is to permit a
concurrent CPU-hotplug operation to complete.
#. In the case of RCU-sched, one of the last acts of an outgoing CPU is
to invoke ``rcu_report_dead()``, which reports a quiescent state for
that CPU. However, this is likely paranoia-induced redundancy.
+-----------------------------------------------------------------------+
| **Quick Quiz**: |
+-----------------------------------------------------------------------+
| Why all the dancing around with multiple counters and masks tracking |
| CPUs that were once online? Why not just have a single set of masks |
| tracking the currently online CPUs and be done with it? |
+-----------------------------------------------------------------------+
| **Answer**: |
+-----------------------------------------------------------------------+
| Maintaining single set of masks tracking the online CPUs *sounds* |
| easier, at least until you try working out all the race conditions |
| between grace-period initialization and CPU-hotplug operations. For |
| example, suppose initialization is progressing down the tree while a |
| CPU-offline operation is progressing up the tree. This situation can |
| result in bits set at the top of the tree that have no counterparts |
| at the bottom of the tree. Those bits will never be cleared, which |
| will result in grace-period hangs. In short, that way lies madness, |
| to say nothing of a great many bugs, hangs, and deadlocks. |
| In contrast, the current multi-mask multi-counter scheme ensures that |
| grace-period initialization will always see consistent masks up and |
| down the tree, which brings significant simplifications over the |
| single-mask method. |
| |
| This is an instance of `deferring work in order to avoid |
| synchronization <http://www.cs.columbia.edu/~library/TR-repository/re |
| ports/reports-1992/cucs-039-92.ps.gz>`__. |
| Lazily recording CPU-hotplug events at the beginning of the next |
| grace period greatly simplifies maintenance of the CPU-tracking |
| bitmasks in the ``rcu_node`` tree. |
+-----------------------------------------------------------------------+
Expedited Grace Period Refinements
----------------------------------
Idle-CPU Checks
~~~~~~~~~~~~~~~
Each expedited grace period checks for idle CPUs when initially forming
the mask of CPUs to be IPIed and again just before IPIing a CPU (both
checks are carried out by ``sync_rcu_exp_select_cpus()``). If the CPU is
idle at any time between those two times, the CPU will not be IPIed.
Instead, the task pushing the grace period forward will include the idle
CPUs in the mask passed to ``rcu_report_exp_cpu_mult()``.
For RCU-sched, there is an additional check: If the IPI has interrupted
the idle loop, then ``rcu_exp_handler()`` invokes
``rcu_report_exp_rdp()`` to report the corresponding quiescent state.
For RCU-preempt, there is no specific check for idle in the IPI handler
(``rcu_exp_handler()``), but because RCU read-side critical sections are
not permitted within the idle loop, if ``rcu_exp_handler()`` sees that
the CPU is within RCU read-side critical section, the CPU cannot
possibly be idle. Otherwise, ``rcu_exp_handler()`` invokes
``rcu_report_exp_rdp()`` to report the corresponding quiescent state,
regardless of whether or not that quiescent state was due to the CPU
being idle.
In summary, RCU expedited grace periods check for idle when building the
bitmask of CPUs that must be IPIed, just before sending each IPI, and
(either explicitly or implicitly) within the IPI handler.
Batching via Sequence Counter
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
If each grace-period request was carried out separately, expedited grace
periods would have abysmal scalability and problematic high-load
characteristics. Because each grace-period operation can serve an
unlimited number of updates, it is important to *batch* requests, so
that a single expedited grace-period operation will cover all requests
in the corresponding batch.
This batching is controlled by a sequence counter named
``->expedited_sequence`` in the ``rcu_state`` structure. This counter
has an odd value when there is an expedited grace period in progress and
an even value otherwise, so that dividing the counter value by two gives
the number of completed grace periods. During any given update request,
the counter must transition from even to odd and then back to even, thus
indicating that a grace period has elapsed. Therefore, if the initial
value of the counter is ``s``, the updater must wait until the counter
reaches at least the value ``(s+3)&~0x1``. This counter is managed by
the following access functions:
#. ``rcu_exp_gp_seq_start()``, which marks the start of an expedited
grace period.
#. ``rcu_exp_gp_seq_end()``, which marks the end of an expedited grace
period.
#. ``rcu_exp_gp_seq_snap()``, which obtains a snapshot of the counter.
#. ``rcu_exp_gp_seq_done()``, which returns ``true`` if a full expedited
grace period has elapsed since the corresponding call to
``rcu_exp_gp_seq_snap()``.
Again, only one request in a given batch need actually carry out a
grace-period operation, which means there must be an efficient way to
identify which of many concurrent reqeusts will initiate the grace
period, and that there be an efficient way for the remaining requests to
wait for that grace period to complete. However, that is the topic of
the next section.
Funnel Locking and Wait/Wakeup
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
The natural way to sort out which of a batch of updaters will initiate
the expedited grace period is to use the ``rcu_node`` combining tree, as
implemented by the ``exp_funnel_lock()`` function. The first updater
corresponding to a given grace period arriving at a given ``rcu_node``
structure records its desired grace-period sequence number in the
``->exp_seq_rq`` field and moves up to the next level in the tree.
Otherwise, if the ``->exp_seq_rq`` field already contains the sequence
number for the desired grace period or some later one, the updater
blocks on one of four wait queues in the ``->exp_wq[]`` array, using the
second-from-bottom and third-from bottom bits as an index. An
``->exp_lock`` field in the ``rcu_node`` structure synchronizes access
to these fields.
An empty ``rcu_node`` tree is shown in the following diagram, with the
white cells representing the ``->exp_seq_rq`` field and the red cells
representing the elements of the ``->exp_wq[]`` array.
.. kernel-figure:: Funnel0.svg
The next diagram shows the situation after the arrival of Task A and
Task B at the leftmost and rightmost leaf ``rcu_node`` structures,
respectively. The current value of the ``rcu_state`` structure's
``->expedited_sequence`` field is zero, so adding three and clearing the
bottom bit results in the value two, which both tasks record in the
``->exp_seq_rq`` field of their respective ``rcu_node`` structures:
.. kernel-figure:: Funnel1.svg
Each of Tasks A and B will move up to the root ``rcu_node`` structure.
Suppose that Task A wins, recording its desired grace-period sequence
number and resulting in the state shown below:
.. kernel-figure:: Funnel2.svg
Task A now advances to initiate a new grace period, while Task B moves
up to the root ``rcu_node`` structure, and, seeing that its desired
sequence number is already recorded, blocks on ``->exp_wq[1]``.
+-----------------------------------------------------------------------+
| **Quick Quiz**: |
+-----------------------------------------------------------------------+
| Why ``->exp_wq[1]``? Given that the value of these tasks' desired |
| sequence number is two, so shouldn't they instead block on |
| ``->exp_wq[2]``? |
+-----------------------------------------------------------------------+
| **Answer**: |
+-----------------------------------------------------------------------+
| No. |
| Recall that the bottom bit of the desired sequence number indicates |
| whether or not a grace period is currently in progress. It is |
| therefore necessary to shift the sequence number right one bit |
| position to obtain the number of the grace period. This results in |
| ``->exp_wq[1]``. |
+-----------------------------------------------------------------------+
If Tasks C and D also arrive at this point, they will compute the same
desired grace-period sequence number, and see that both leaf
``rcu_node`` structures already have that value recorded. They will
therefore block on their respective ``rcu_node`` structures'
``->exp_wq[1]`` fields, as shown below:
.. kernel-figure:: Funnel3.svg
Task A now acquires the ``rcu_state`` structure's ``->exp_mutex`` and
initiates the grace period, which increments ``->expedited_sequence``.
Therefore, if Tasks E and F arrive, they will compute a desired sequence
number of 4 and will record this value as shown below:
.. kernel-figure:: Funnel4.svg
Tasks E and F will propagate up the ``rcu_node`` combining tree, with
Task F blocking on the root ``rcu_node`` structure and Task E wait for
Task A to finish so that it can start the next grace period. The
resulting state is as shown below:
.. kernel-figure:: Funnel5.svg
Once the grace period completes, Task A starts waking up the tasks
waiting for this grace period to complete, increments the
``->expedited_sequence``, acquires the ``->exp_wake_mutex`` and then
releases the ``->exp_mutex``. This results in the following state:
.. kernel-figure:: Funnel6.svg
Task E can then acquire ``->exp_mutex`` and increment
``->expedited_sequence`` to the value three. If new tasks G and H arrive
and moves up the combining tree at the same time, the state will be as
follows:
.. kernel-figure:: Funnel7.svg
Note that three of the root ``rcu_node`` structure's waitqueues are now
occupied. However, at some point, Task A will wake up the tasks blocked
on the ``->exp_wq`` waitqueues, resulting in the following state:
.. kernel-figure:: Funnel8.svg
Execution will continue with Tasks E and H completing their grace
periods and carrying out their wakeups.
+-----------------------------------------------------------------------+
| **Quick Quiz**: |
+-----------------------------------------------------------------------+
| What happens if Task A takes so long to do its wakeups that Task E's |
| grace period completes? |
+-----------------------------------------------------------------------+
| **Answer**: |
+-----------------------------------------------------------------------+
| Then Task E will block on the ``->exp_wake_mutex``, which will also |
| prevent it from releasing ``->exp_mutex``, which in turn will prevent |
| the next grace period from starting. This last is important in |
| preventing overflow of the ``->exp_wq[]`` array. |
+-----------------------------------------------------------------------+
Use of Workqueues
~~~~~~~~~~~~~~~~~
In earlier implementations, the task requesting the expedited grace
period also drove it to completion. This straightforward approach had
the disadvantage of needing to account for POSIX signals sent to user
tasks, so more recent implemementations use the Linux kernel's
`workqueues <https://www.kernel.org/doc/Documentation/core-api/workqueue.rst>`__.
The requesting task still does counter snapshotting and funnel-lock
processing, but the task reaching the top of the funnel lock does a
``schedule_work()`` (from ``_synchronize_rcu_expedited()`` so that a
workqueue kthread does the actual grace-period processing. Because
workqueue kthreads do not accept POSIX signals, grace-period-wait
processing need not allow for POSIX signals. In addition, this approach
allows wakeups for the previous expedited grace period to be overlapped
with processing for the next expedited grace period. Because there are
only four sets of waitqueues, it is necessary to ensure that the
previous grace period's wakeups complete before the next grace period's
wakeups start. This is handled by having the ``->exp_mutex`` guard
expedited grace-period processing and the ``->exp_wake_mutex`` guard
wakeups. The key point is that the ``->exp_mutex`` is not released until
the first wakeup is complete, which means that the ``->exp_wake_mutex``
has already been acquired at that point. This approach ensures that the
previous grace period's wakeups can be carried out while the current
grace period is in process, but that these wakeups will complete before
the next grace period starts. This means that only three waitqueues are
required, guaranteeing that the four that are provided are sufficient.
Stall Warnings
~~~~~~~~~~~~~~
Expediting grace periods does nothing to speed things up when RCU
readers take too long, and therefore expedited grace periods check for
stalls just as normal grace periods do.
+-----------------------------------------------------------------------+
| **Quick Quiz**: |
+-----------------------------------------------------------------------+
| But why not just let the normal grace-period machinery detect the |
| stalls, given that a given reader must block both normal and |
| expedited grace periods? |
+-----------------------------------------------------------------------+
| **Answer**: |
+-----------------------------------------------------------------------+
| Because it is quite possible that at a given time there is no normal |
| grace period in progress, in which case the normal grace period |
| cannot emit a stall warning. |
+-----------------------------------------------------------------------+
The ``synchronize_sched_expedited_wait()`` function loops waiting for
the expedited grace period to end, but with a timeout set to the current
RCU CPU stall-warning time. If this time is exceeded, any CPUs or
``rcu_node`` structures blocking the current grace period are printed.
Each stall warning results in another pass through the loop, but the
second and subsequent passes use longer stall times.
Mid-boot operation
~~~~~~~~~~~~~~~~~~
The use of workqueues has the advantage that the expedited grace-period
code need not worry about POSIX signals. Unfortunately, it has the
corresponding disadvantage that workqueues cannot be used until they are
initialized, which does not happen until some time after the scheduler
spawns the first task. Given that there are parts of the kernel that
really do want to execute grace periods during this mid-boot “dead
zone”, expedited grace periods must do something else during thie time.
What they do is to fall back to the old practice of requiring that the
requesting task drive the expedited grace period, as was the case before
the use of workqueues. However, the requesting task is only required to
drive the grace period during the mid-boot dead zone. Before mid-boot, a
synchronous grace period is a no-op. Some time after mid-boot,
workqueues are used.
Non-expedited non-SRCU synchronous grace periods must also operate
normally during mid-boot. This is handled by causing non-expedited grace
periods to take the expedited code path during mid-boot.
The current code assumes that there are no POSIX signals during the
mid-boot dead zone. However, if an overwhelming need for POSIX signals
somehow arises, appropriate adjustments can be made to the expedited
stall-warning code. One such adjustment would reinstate the
pre-workqueue stall-warning checks, but only during the mid-boot dead
zone.
With this refinement, synchronous grace periods can now be used from
task context pretty much any time during the life of the kernel. That
is, aside from some points in the suspend, hibernate, or shutdown code
path.
Summary
~~~~~~~
Expedited grace periods use a sequence-number approach to promote
batching, so that a single grace-period operation can serve numerous
requests. A funnel lock is used to efficiently identify the one task out
of a concurrent group that will request the grace period. All members of
the group will block on waitqueues provided in the ``rcu_node``
structure. The actual grace-period processing is carried out by a
workqueue.
CPU-hotplug operations are noted lazily in order to prevent the need for
tight synchronization between expedited grace periods and CPU-hotplug
operations. The dyntick-idle counters are used to avoid sending IPIs to
idle CPUs, at least in the common case. RCU-preempt and RCU-sched use
different IPI handlers and different code to respond to the state
changes carried out by those handlers, but otherwise use common code.
Quiescent states are tracked using the ``rcu_node`` tree, and once all
necessary quiescent states have been reported, all tasks waiting on this
expedited grace period are awakened. A pair of mutexes are used to allow
one grace period's wakeups to proceed concurrently with the next grace
period's processing.
This combination of mechanisms allows expedited grace periods to run
reasonably efficiently. However, for non-time-critical tasks, normal
grace periods should be used instead because their longer duration
permits much higher degrees of batching, and thus much lower per-request
overheads.
Documentation/RCU/Design/Memory-Ordering/Tree-RCU-Diagram.html deleted 100644 → 0
View file @ c07e6f36
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN"
"http://www.w3.org/TR/html4/loose.dtd">
<html>
<head><title>A Diagram of TREE_RCU's Grace-Period Memory Ordering</title>
<meta HTTP-EQUIV="Content-Type" CONTENT="text/html; charset=iso-8859-1">
<p><img src="TreeRCU-gp.svg" alt="TreeRCU-gp.svg">
</body></html>
Documentation/RCU/Design/Memory-Ordering/Tree-RCU-Memory-Ordering.html deleted 100644 → 0
View file @ c07e6f36
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN"
"http://www.w3.org/TR/html4/loose.dtd">
<html>
<head><title>A Tour Through TREE_RCU's Grace-Period Memory Ordering</title>
<meta HTTP-EQUIV="Content-Type" CONTENT="text/html; charset=iso-8859-1">
<p>August 8, 2017</p>
<p>This article was contributed by Paul E.&nbsp;McKenney</p>
<h3>Introduction</h3>
<p>This document gives a rough visual overview of how Tree RCU's
grace-period memory ordering guarantee is provided.
<ol>
<li> <a href="#What Is Tree RCU's Grace Period Memory Ordering Guarantee?">
What Is Tree RCU's Grace Period Memory Ordering Guarantee?</a>
<li> <a href="#Tree RCU Grace Period Memory Ordering Building Blocks">
Tree RCU Grace Period Memory Ordering Building Blocks</a>
<li> <a href="#Tree RCU Grace Period Memory Ordering Components">
Tree RCU Grace Period Memory Ordering Components</a>
<li> <a href="#Putting It All Together">Putting It All Together</a>
</ol>
<h3><a name="What Is Tree RCU's Grace Period Memory Ordering Guarantee?">
What Is Tree RCU's Grace Period Memory Ordering Guarantee?</a></h3>
<p>RCU grace periods provide extremely strong memory-ordering guarantees
for non-idle non-offline code.
Any code that happens after the end of a given RCU grace period is guaranteed
to see the effects of all accesses prior to the beginning of that grace
period that are within RCU read-side critical sections.
Similarly, any code that happens before the beginning of a given RCU grace
period is guaranteed to see the effects of all accesses following the end
of that grace period that are within RCU read-side critical sections.
<p>Note well that RCU-sched read-side critical sections include any region
of code for which preemption is disabled.
Given that each individual machine instruction can be thought of as
an extremely small region of preemption-disabled code, one can think of
<tt>synchronize_rcu()</tt> as <tt>smp_mb()</tt> on steroids.
<p>RCU updaters use this guarantee by splitting their updates into
two phases, one of which is executed before the grace period and
the other of which is executed after the grace period.
In the most common use case, phase one removes an element from
a linked RCU-protected data structure, and phase two frees that element.
For this to work, any readers that have witnessed state prior to the
phase-one update (in the common case, removal) must not witness state
following the phase-two update (in the common case, freeing).
<p>The RCU implementation provides this guarantee using a network
of lock-based critical sections, memory barriers, and per-CPU
processing, as is described in the following sections.
<h3><a name="Tree RCU Grace Period Memory Ordering Building Blocks">
Tree RCU Grace Period Memory Ordering Building Blocks</a></h3>
<p>The workhorse for RCU's grace-period memory ordering is the
critical section for the <tt>rcu_node</tt> structure's
<tt>-&gt;lock</tt>.
These critical sections use helper functions for lock acquisition, including
<tt>raw_spin_lock_rcu_node()</tt>,
<tt>raw_spin_lock_irq_rcu_node()</tt>, and
<tt>raw_spin_lock_irqsave_rcu_node()</tt>.
Their lock-release counterparts are
<tt>raw_spin_unlock_rcu_node()</tt>,
<tt>raw_spin_unlock_irq_rcu_node()</tt>, and
<tt>raw_spin_unlock_irqrestore_rcu_node()</tt>,
respectively.
For completeness, a
<tt>raw_spin_trylock_rcu_node()</tt>
is also provided.
The key point is that the lock-acquisition functions, including
<tt>raw_spin_trylock_rcu_node()</tt>, all invoke
<tt>smp_mb__after_unlock_lock()</tt> immediately after successful
acquisition of the lock.
<p>Therefore, for any given <tt>rcu_node</tt> structure, any access
happening before one of the above lock-release functions will be seen
by all CPUs as happening before any access happening after a later
one of the above lock-acquisition functions.
Furthermore, any access happening before one of the
above lock-release function on any given CPU will be seen by all
CPUs as happening before any access happening after a later one
of the above lock-acquisition functions executing on that same CPU,
even if the lock-release and lock-acquisition functions are operating
on different <tt>rcu_node</tt> structures.
Tree RCU uses these two ordering guarantees to form an ordering
network among all CPUs that were in any way involved in the grace
period, including any CPUs that came online or went offline during
the grace period in question.
<p>The following litmus test exhibits the ordering effects of these
lock-acquisition and lock-release functions:
<pre>
1 int x, y, z;
2
3 void task0(void)
4 {
5 raw_spin_lock_rcu_node(rnp);
6 WRITE_ONCE(x, 1);
7 r1 = READ_ONCE(y);
8 raw_spin_unlock_rcu_node(rnp);
9 }
10
11 void task1(void)
12 {
13 raw_spin_lock_rcu_node(rnp);
14 WRITE_ONCE(y, 1);
15 r2 = READ_ONCE(z);
16 raw_spin_unlock_rcu_node(rnp);
17 }
18
19 void task2(void)
20 {
21 WRITE_ONCE(z, 1);
22 smp_mb();
23 r3 = READ_ONCE(x);
24 }
25
26 WARN_ON(r1 == 0 &amp;&amp; r2 == 0 &amp;&amp; r3 == 0);
</pre>
<p>The <tt>WARN_ON()</tt> is evaluated at &ldquo;the end of time&rdquo;,
after all changes have propagated throughout the system.
Without the <tt>smp_mb__after_unlock_lock()</tt> provided by the
acquisition functions, this <tt>WARN_ON()</tt> could trigger, for example
on PowerPC.
The <tt>smp_mb__after_unlock_lock()</tt> invocations prevent this
<tt>WARN_ON()</tt> from triggering.
<p>This approach must be extended to include idle CPUs, which need
RCU's grace-period memory ordering guarantee to extend to any
RCU read-side critical sections preceding and following the current
idle sojourn.
This case is handled by calls to the strongly ordered
<tt>atomic_add_return()</tt> read-modify-write atomic operation that
is invoked within <tt>rcu_dynticks_eqs_enter()</tt> at idle-entry
time and within <tt>rcu_dynticks_eqs_exit()</tt> at idle-exit time.
The grace-period kthread invokes <tt>rcu_dynticks_snap()</tt> and
<tt>rcu_dynticks_in_eqs_since()</tt> (both of which invoke
an <tt>atomic_add_return()</tt> of zero) to detect idle CPUs.
<table>
<tr><th>&nbsp;</th></tr>
<tr><th align="left">Quick Quiz:</th></tr>
<tr><td>
But what about CPUs that remain offline for the entire
grace period?
</td></tr>
<tr><th align="left">Answer:</th></tr>
<tr><td bgcolor="#ffffff"><font color="ffffff">
Such CPUs will be offline at the beginning of the grace period,
so the grace period won't expect quiescent states from them.
Races between grace-period start and CPU-hotplug operations
are mediated by the CPU's leaf <tt>rcu_node</tt> structure's
<tt>-&gt;lock</tt> as described above.
</font></td></tr>
<tr><td>&nbsp;</td></tr>
</table>
<p>The approach must be extended to handle one final case, that
of waking a task blocked in <tt>synchronize_rcu()</tt>.
This task might be affinitied to a CPU that is not yet aware that
the grace period has ended, and thus might not yet be subject to
the grace period's memory ordering.
Therefore, there is an <tt>smp_mb()</tt> after the return from
<tt>wait_for_completion()</tt> in the <tt>synchronize_rcu()</tt>
code path.
<table>
<tr><th>&nbsp;</th></tr>
<tr><th align="left">Quick Quiz:</th></tr>
<tr><td>
What? Where???
I don't see any <tt>smp_mb()</tt> after the return from
<tt>wait_for_completion()</tt>!!!
</td></tr>
<tr><th align="left">Answer:</th></tr>
<tr><td bgcolor="#ffffff"><font color="ffffff">
That would be because I spotted the need for that
<tt>smp_mb()</tt> during the creation of this documentation,
and it is therefore unlikely to hit mainline before v4.14.
Kudos to Lance Roy, Will Deacon, Peter Zijlstra, and
Jonathan Cameron for asking questions that sensitized me
to the rather elaborate sequence of events that demonstrate
the need for this memory barrier.
</font></td></tr>
<tr><td>&nbsp;</td></tr>
</table>
<p>Tree RCU's grace--period memory-ordering guarantees rely most
heavily on the <tt>rcu_node</tt> structure's <tt>-&gt;lock</tt>
field, so much so that it is necessary to abbreviate this pattern
in the diagrams in the next section.
For example, consider the <tt>rcu_prepare_for_idle()</tt> function
shown below, which is one of several functions that enforce ordering
of newly arrived RCU callbacks against future grace periods:
<pre>
1 static void rcu_prepare_for_idle(void)
2 {
3 bool needwake;
4 struct rcu_data *rdp;
5 struct rcu_dynticks *rdtp = this_cpu_ptr(&amp;rcu_dynticks);
6 struct rcu_node *rnp;
7 struct rcu_state *rsp;
8 int tne;
9
10 if (IS_ENABLED(CONFIG_RCU_NOCB_CPU_ALL) ||
11 rcu_is_nocb_cpu(smp_processor_id()))
12 return;
13 tne = READ_ONCE(tick_nohz_active);
14 if (tne != rdtp-&gt;tick_nohz_enabled_snap) {
15 if (rcu_cpu_has_callbacks(NULL))
16 invoke_rcu_core();
17 rdtp-&gt;tick_nohz_enabled_snap = tne;
18 return;
19 }
20 if (!tne)
21 return;
22 if (rdtp-&gt;all_lazy &amp;&amp;
23 rdtp-&gt;nonlazy_posted != rdtp-&gt;nonlazy_posted_snap) {
24 rdtp-&gt;all_lazy = false;
25 rdtp-&gt;nonlazy_posted_snap = rdtp-&gt;nonlazy_posted;
26 invoke_rcu_core();
27 return;
28 }
29 if (rdtp-&gt;last_accelerate == jiffies)
30 return;
31 rdtp-&gt;last_accelerate = jiffies;
32 for_each_rcu_flavor(rsp) {
33 rdp = this_cpu_ptr(rsp-&gt;rda);
34 if (rcu_segcblist_pend_cbs(&amp;rdp-&gt;cblist))
35 continue;
36 rnp = rdp-&gt;mynode;
37 raw_spin_lock_rcu_node(rnp);
38 needwake = rcu_accelerate_cbs(rsp, rnp, rdp);
39 raw_spin_unlock_rcu_node(rnp);
40 if (needwake)
41 rcu_gp_kthread_wake(rsp);
42 }
43 }
</pre>
<p>But the only part of <tt>rcu_prepare_for_idle()</tt> that really
matters for this discussion are lines&nbsp;37&ndash;39.
We will therefore abbreviate this function as follows:
</p><p><img src="rcu_node-lock.svg" alt="rcu_node-lock.svg">
<p>The box represents the <tt>rcu_node</tt> structure's <tt>-&gt;lock</tt>
critical section, with the double line on top representing the additional
<tt>smp_mb__after_unlock_lock()</tt>.
<h3><a name="Tree RCU Grace Period Memory Ordering Components">
Tree RCU Grace Period Memory Ordering Components</a></h3>
<p>Tree RCU's grace-period memory-ordering guarantee is provided by
a number of RCU components:
<ol>
<li> <a href="#Callback Registry">Callback Registry</a>
<li> <a href="#Grace-Period Initialization">Grace-Period Initialization</a>
<li> <a href="#Self-Reported Quiescent States">
Self-Reported Quiescent States</a>
<li> <a href="#Dynamic Tick Interface">Dynamic Tick Interface</a>
<li> <a href="#CPU-Hotplug Interface">CPU-Hotplug Interface</a>
<li> <a href="Forcing Quiescent States">Forcing Quiescent States</a>
<li> <a href="Grace-Period Cleanup">Grace-Period Cleanup</a>
<li> <a href="Callback Invocation">Callback Invocation</a>
</ol>
<p>Each of the following section looks at the corresponding component
in detail.
<h4><a name="Callback Registry">Callback Registry</a></h4>
<p>If RCU's grace-period guarantee is to mean anything at all, any
access that happens before a given invocation of <tt>call_rcu()</tt>
must also happen before the corresponding grace period.
The implementation of this portion of RCU's grace period guarantee
is shown in the following figure:
</p><p><img src="TreeRCU-callback-registry.svg" alt="TreeRCU-callback-registry.svg">
<p>Because <tt>call_rcu()</tt> normally acts only on CPU-local state,
it provides no ordering guarantees, either for itself or for
phase one of the update (which again will usually be removal of
an element from an RCU-protected data structure).
It simply enqueues the <tt>rcu_head</tt> structure on a per-CPU list,
which cannot become associated with a grace period until a later
call to <tt>rcu_accelerate_cbs()</tt>, as shown in the diagram above.
<p>One set of code paths shown on the left invokes
<tt>rcu_accelerate_cbs()</tt> via
<tt>note_gp_changes()</tt>, either directly from <tt>call_rcu()</tt> (if
the current CPU is inundated with queued <tt>rcu_head</tt> structures)
or more likely from an <tt>RCU_SOFTIRQ</tt> handler.
Another code path in the middle is taken only in kernels built with
<tt>CONFIG_RCU_FAST_NO_HZ=y</tt>, which invokes
<tt>rcu_accelerate_cbs()</tt> via <tt>rcu_prepare_for_idle()</tt>.
The final code path on the right is taken only in kernels built with
<tt>CONFIG_HOTPLUG_CPU=y</tt>, which invokes
<tt>rcu_accelerate_cbs()</tt> via
<tt>rcu_advance_cbs()</tt>, <tt>rcu_migrate_callbacks</tt>,
<tt>rcutree_migrate_callbacks()</tt>, and <tt>takedown_cpu()</tt>,
which in turn is invoked on a surviving CPU after the outgoing
CPU has been completely offlined.
<p>There are a few other code paths within grace-period processing
that opportunistically invoke <tt>rcu_accelerate_cbs()</tt>.
However, either way, all of the CPU's recently queued <tt>rcu_head</tt>
structures are associated with a future grace-period number under
the protection of the CPU's lead <tt>rcu_node</tt> structure's
<tt>-&gt;lock</tt>.
In all cases, there is full ordering against any prior critical section
for that same <tt>rcu_node</tt> structure's <tt>-&gt;lock</tt>, and
also full ordering against any of the current task's or CPU's prior critical
sections for any <tt>rcu_node</tt> structure's <tt>-&gt;lock</tt>.
<p>The next section will show how this ordering ensures that any
accesses prior to the <tt>call_rcu()</tt> (particularly including phase
one of the update)
happen before the start of the corresponding grace period.
<table>
<tr><th>&nbsp;</th></tr>
<tr><th align="left">Quick Quiz:</th></tr>
<tr><td>
But what about <tt>synchronize_rcu()</tt>?
</td></tr>
<tr><th align="left">Answer:</th></tr>
<tr><td bgcolor="#ffffff"><font color="ffffff">
The <tt>synchronize_rcu()</tt> passes <tt>call_rcu()</tt>
to <tt>wait_rcu_gp()</tt>, which invokes it.
So either way, it eventually comes down to <tt>call_rcu()</tt>.
</font></td></tr>
<tr><td>&nbsp;</td></tr>
</table>
<h4><a name="Grace-Period Initialization">Grace-Period Initialization</a></h4>
<p>Grace-period initialization is carried out by
the grace-period kernel thread, which makes several passes over the
<tt>rcu_node</tt> tree within the <tt>rcu_gp_init()</tt> function.
This means that showing the full flow of ordering through the
grace-period computation will require duplicating this tree.
If you find this confusing, please note that the state of the
<tt>rcu_node</tt> changes over time, just like Heraclitus's river.
However, to keep the <tt>rcu_node</tt> river tractable, the
grace-period kernel thread's traversals are presented in multiple
parts, starting in this section with the various phases of
grace-period initialization.
<p>The first ordering-related grace-period initialization action is to
advance the <tt>rcu_state</tt> structure's <tt>-&gt;gp_seq</tt>
grace-period-number counter, as shown below:
</p><p><img src="TreeRCU-gp-init-1.svg" alt="TreeRCU-gp-init-1.svg" width="75%">
<p>The actual increment is carried out using <tt>smp_store_release()</tt>,
which helps reject false-positive RCU CPU stall detection.
Note that only the root <tt>rcu_node</tt> structure is touched.
<p>The first pass through the <tt>rcu_node</tt> tree updates bitmasks
based on CPUs having come online or gone offline since the start of
the previous grace period.
In the common case where the number of online CPUs for this <tt>rcu_node</tt>
structure has not transitioned to or from zero,
this pass will scan only the leaf <tt>rcu_node</tt> structures.
However, if the number of online CPUs for a given leaf <tt>rcu_node</tt>
structure has transitioned from zero,
<tt>rcu_init_new_rnp()</tt> will be invoked for the first incoming CPU.
Similarly, if the number of online CPUs for a given leaf <tt>rcu_node</tt>
structure has transitioned to zero,
<tt>rcu_cleanup_dead_rnp()</tt> will be invoked for the last outgoing CPU.
The diagram below shows the path of ordering if the leftmost
<tt>rcu_node</tt> structure onlines its first CPU and if the next
<tt>rcu_node</tt> structure has no online CPUs
(or, alternatively if the leftmost <tt>rcu_node</tt> structure offlines
its last CPU and if the next <tt>rcu_node</tt> structure has no online CPUs).
</p><p><img src="TreeRCU-gp-init-2.svg" alt="TreeRCU-gp-init-1.svg" width="75%">
<p>The final <tt>rcu_gp_init()</tt> pass through the <tt>rcu_node</tt>
tree traverses breadth-first, setting each <tt>rcu_node</tt> structure's
<tt>-&gt;gp_seq</tt> field to the newly advanced value from the
<tt>rcu_state</tt> structure, as shown in the following diagram.
</p><p><img src="TreeRCU-gp-init-3.svg" alt="TreeRCU-gp-init-1.svg" width="75%">
<p>This change will also cause each CPU's next call to
<tt>__note_gp_changes()</tt>
to notice that a new grace period has started, as described in the next
section.
But because the grace-period kthread started the grace period at the
root (with the advancing of the <tt>rcu_state</tt> structure's
<tt>-&gt;gp_seq</tt> field) before setting each leaf <tt>rcu_node</tt>
structure's <tt>-&gt;gp_seq</tt> field, each CPU's observation of
the start of the grace period will happen after the actual start
of the grace period.
<table>
<tr><th>&nbsp;</th></tr>
<tr><th align="left">Quick Quiz:</th></tr>
<tr><td>
But what about the CPU that started the grace period?
Why wouldn't it see the start of the grace period right when
it started that grace period?
</td></tr>
<tr><th align="left">Answer:</th></tr>
<tr><td bgcolor="#ffffff"><font color="ffffff">
In some deep philosophical and overly anthromorphized
sense, yes, the CPU starting the grace period is immediately
aware of having done so.
However, if we instead assume that RCU is not self-aware,
then even the CPU starting the grace period does not really
become aware of the start of this grace period until its
first call to <tt>__note_gp_changes()</tt>.
On the other hand, this CPU potentially gets early notification
because it invokes <tt>__note_gp_changes()</tt> during its
last <tt>rcu_gp_init()</tt> pass through its leaf
<tt>rcu_node</tt> structure.
</font></td></tr>
<tr><td>&nbsp;</td></tr>
</table>
<h4><a name="Self-Reported Quiescent States">
Self-Reported Quiescent States</a></h4>
<p>When all entities that might block the grace period have reported
quiescent states (or as described in a later section, had quiescent
states reported on their behalf), the grace period can end.
Online non-idle CPUs report their own quiescent states, as shown
in the following diagram:
</p><p><img src="TreeRCU-qs.svg" alt="TreeRCU-qs.svg" width="75%">
<p>This is for the last CPU to report a quiescent state, which signals
the end of the grace period.
Earlier quiescent states would push up the <tt>rcu_node</tt> tree
only until they encountered an <tt>rcu_node</tt> structure that
is waiting for additional quiescent states.
However, ordering is nevertheless preserved because some later quiescent
state will acquire that <tt>rcu_node</tt> structure's <tt>-&gt;lock</tt>.
<p>Any number of events can lead up to a CPU invoking
<tt>note_gp_changes</tt> (or alternatively, directly invoking
<tt>__note_gp_changes()</tt>), at which point that CPU will notice
the start of a new grace period while holding its leaf
<tt>rcu_node</tt> lock.
Therefore, all execution shown in this diagram happens after the
start of the grace period.
In addition, this CPU will consider any RCU read-side critical
section that started before the invocation of <tt>__note_gp_changes()</tt>
to have started before the grace period, and thus a critical
section that the grace period must wait on.
<table>
<tr><th>&nbsp;</th></tr>
<tr><th align="left">Quick Quiz:</th></tr>
<tr><td>
But a RCU read-side critical section might have started
after the beginning of the grace period
(the advancing of <tt>-&gt;gp_seq</tt> from earlier), so why should
the grace period wait on such a critical section?
</td></tr>
<tr><th align="left">Answer:</th></tr>
<tr><td bgcolor="#ffffff"><font color="ffffff">
It is indeed not necessary for the grace period to wait on such
a critical section.
However, it is permissible to wait on it.
And it is furthermore important to wait on it, as this
lazy approach is far more scalable than a &ldquo;big bang&rdquo;
all-at-once grace-period start could possibly be.
</font></td></tr>
<tr><td>&nbsp;</td></tr>
</table>
<p>If the CPU does a context switch, a quiescent state will be
noted by <tt>rcu_node_context_switch()</tt> on the left.
On the other hand, if the CPU takes a scheduler-clock interrupt
while executing in usermode, a quiescent state will be noted by
<tt>rcu_sched_clock_irq()</tt> on the right.
Either way, the passage through a quiescent state will be noted
in a per-CPU variable.
<p>The next time an <tt>RCU_SOFTIRQ</tt> handler executes on
this CPU (for example, after the next scheduler-clock
interrupt), <tt>rcu_core()</tt> will invoke
<tt>rcu_check_quiescent_state()</tt>, which will notice the
recorded quiescent state, and invoke
<tt>rcu_report_qs_rdp()</tt>.
If <tt>rcu_report_qs_rdp()</tt> verifies that the quiescent state
really does apply to the current grace period, it invokes
<tt>rcu_report_rnp()</tt> which traverses up the <tt>rcu_node</tt>
tree as shown at the bottom of the diagram, clearing bits from
each <tt>rcu_node</tt> structure's <tt>-&gt;qsmask</tt> field,
and propagating up the tree when the result is zero.
<p>Note that traversal passes upwards out of a given <tt>rcu_node</tt>
structure only if the current CPU is reporting the last quiescent
state for the subtree headed by that <tt>rcu_node</tt> structure.
A key point is that if a CPU's traversal stops at a given <tt>rcu_node</tt>
structure, then there will be a later traversal by another CPU
(or perhaps the same one) that proceeds upwards
from that point, and the <tt>rcu_node</tt> <tt>-&gt;lock</tt>
guarantees that the first CPU's quiescent state happens before the
remainder of the second CPU's traversal.
Applying this line of thought repeatedly shows that all CPUs'
quiescent states happen before the last CPU traverses through
the root <tt>rcu_node</tt> structure, the &ldquo;last CPU&rdquo;
being the one that clears the last bit in the root <tt>rcu_node</tt>
structure's <tt>-&gt;qsmask</tt> field.
<h4><a name="Dynamic Tick Interface">Dynamic Tick Interface</a></h4>
<p>Due to energy-efficiency considerations, RCU is forbidden from
disturbing idle CPUs.
CPUs are therefore required to notify RCU when entering or leaving idle
state, which they do via fully ordered value-returning atomic operations
on a per-CPU variable.
The ordering effects are as shown below:
</p><p><img src="TreeRCU-dyntick.svg" alt="TreeRCU-dyntick.svg" width="50%">
<p>The RCU grace-period kernel thread samples the per-CPU idleness
variable while holding the corresponding CPU's leaf <tt>rcu_node</tt>
structure's <tt>-&gt;lock</tt>.
This means that any RCU read-side critical sections that precede the
idle period (the oval near the top of the diagram above) will happen
before the end of the current grace period.
Similarly, the beginning of the current grace period will happen before
any RCU read-side critical sections that follow the
idle period (the oval near the bottom of the diagram above).
<p>Plumbing this into the full grace-period execution is described
<a href="#Forcing Quiescent States">below</a>.
<h4><a name="CPU-Hotplug Interface">CPU-Hotplug Interface</a></h4>
<p>RCU is also forbidden from disturbing offline CPUs, which might well
be powered off and removed from the system completely.
CPUs are therefore required to notify RCU of their comings and goings
as part of the corresponding CPU hotplug operations.
The ordering effects are shown below:
</p><p><img src="TreeRCU-hotplug.svg" alt="TreeRCU-hotplug.svg" width="50%">
<p>Because CPU hotplug operations are much less frequent than idle transitions,
they are heavier weight, and thus acquire the CPU's leaf <tt>rcu_node</tt>
structure's <tt>-&gt;lock</tt> and update this structure's
<tt>-&gt;qsmaskinitnext</tt>.
The RCU grace-period kernel thread samples this mask to detect CPUs
having gone offline since the beginning of this grace period.
<p>Plumbing this into the full grace-period execution is described
<a href="#Forcing Quiescent States">below</a>.
<h4><a name="Forcing Quiescent States">Forcing Quiescent States</a></h4>
<p>As noted above, idle and offline CPUs cannot report their own
quiescent states, and therefore the grace-period kernel thread
must do the reporting on their behalf.
This process is called &ldquo;forcing quiescent states&rdquo;, it is
repeated every few jiffies, and its ordering effects are shown below:
</p><p><img src="TreeRCU-gp-fqs.svg" alt="TreeRCU-gp-fqs.svg" width="100%">
<p>Each pass of quiescent state forcing is guaranteed to traverse the
leaf <tt>rcu_node</tt> structures, and if there are no new quiescent
states due to recently idled and/or offlined CPUs, then only the
leaves are traversed.
However, if there is a newly offlined CPU as illustrated on the left
or a newly idled CPU as illustrated on the right, the corresponding
quiescent state will be driven up towards the root.
As with self-reported quiescent states, the upwards driving stops
once it reaches an <tt>rcu_node</tt> structure that has quiescent
states outstanding from other CPUs.
<table>
<tr><th>&nbsp;</th></tr>
<tr><th align="left">Quick Quiz:</th></tr>
<tr><td>
The leftmost drive to root stopped before it reached
the root <tt>rcu_node</tt> structure, which means that
there are still CPUs subordinate to that structure on
which the current grace period is waiting.
Given that, how is it possible that the rightmost drive
to root ended the grace period?
</td></tr>
<tr><th align="left">Answer:</th></tr>
<tr><td bgcolor="#ffffff"><font color="ffffff">
Good analysis!
It is in fact impossible in the absence of bugs in RCU.
But this diagram is complex enough as it is, so simplicity
overrode accuracy.
You can think of it as poetic license, or you can think of
it as misdirection that is resolved in the
<a href="#Putting It All Together">stitched-together diagram</a>.
</font></td></tr>
<tr><td>&nbsp;</td></tr>
</table>
<h4><a name="Grace-Period Cleanup">Grace-Period Cleanup</a></h4>
<p>Grace-period cleanup first scans the <tt>rcu_node</tt> tree
breadth-first advancing all the <tt>-&gt;gp_seq</tt> fields, then it
advances the <tt>rcu_state</tt> structure's <tt>-&gt;gp_seq</tt> field.
The ordering effects are shown below:
</p><p><img src="TreeRCU-gp-cleanup.svg" alt="TreeRCU-gp-cleanup.svg" width="75%">
<p>As indicated by the oval at the bottom of the diagram, once
grace-period cleanup is complete, the next grace period can begin.
<table>
<tr><th>&nbsp;</th></tr>
<tr><th align="left">Quick Quiz:</th></tr>
<tr><td>
But when precisely does the grace period end?
</td></tr>
<tr><th align="left">Answer:</th></tr>
<tr><td bgcolor="#ffffff"><font color="ffffff">
There is no useful single point at which the grace period
can be said to end.
The earliest reasonable candidate is as soon as the last
CPU has reported its quiescent state, but it may be some
milliseconds before RCU becomes aware of this.
The latest reasonable candidate is once the <tt>rcu_state</tt>
structure's <tt>-&gt;gp_seq</tt> field has been updated,
but it is quite possible that some CPUs have already completed
phase two of their updates by that time.
In short, if you are going to work with RCU, you need to
learn to embrace uncertainty.
</font></td></tr>
<tr><td>&nbsp;</td></tr>
</table>
<h4><a name="Callback Invocation">Callback Invocation</a></h4>
<p>Once a given CPU's leaf <tt>rcu_node</tt> structure's
<tt>-&gt;gp_seq</tt> field has been updated, that CPU can begin
invoking its RCU callbacks that were waiting for this grace period
to end.
These callbacks are identified by <tt>rcu_advance_cbs()</tt>,
which is usually invoked by <tt>__note_gp_changes()</tt>.
As shown in the diagram below, this invocation can be triggered by
the scheduling-clock interrupt (<tt>rcu_sched_clock_irq()</tt> on
the left) or by idle entry (<tt>rcu_cleanup_after_idle()</tt> on
the right, but only for kernels build with
<tt>CONFIG_RCU_FAST_NO_HZ=y</tt>).
Either way, <tt>RCU_SOFTIRQ</tt> is raised, which results in
<tt>rcu_do_batch()</tt> invoking the callbacks, which in turn
allows those callbacks to carry out (either directly or indirectly
via wakeup) the needed phase-two processing for each update.
</p><p><img src="TreeRCU-callback-invocation.svg" alt="TreeRCU-callback-invocation.svg" width="60%">
<p>Please note that callback invocation can also be prompted by any
number of corner-case code paths, for example, when a CPU notes that
it has excessive numbers of callbacks queued.
In all cases, the CPU acquires its leaf <tt>rcu_node</tt> structure's
<tt>-&gt;lock</tt> before invoking callbacks, which preserves the
required ordering against the newly completed grace period.
<p>However, if the callback function communicates to other CPUs,
for example, doing a wakeup, then it is that function's responsibility
to maintain ordering.
For example, if the callback function wakes up a task that runs on
some other CPU, proper ordering must in place in both the callback
function and the task being awakened.
To see why this is important, consider the top half of the
<a href="#Grace-Period Cleanup">grace-period cleanup</a> diagram.
The callback might be running on a CPU corresponding to the leftmost
leaf <tt>rcu_node</tt> structure, and awaken a task that is to run on
a CPU corresponding to the rightmost leaf <tt>rcu_node</tt> structure,
and the grace-period kernel thread might not yet have reached the
rightmost leaf.
In this case, the grace period's memory ordering might not yet have
reached that CPU, so again the callback function and the awakened
task must supply proper ordering.
<h3><a name="Putting It All Together">Putting It All Together</a></h3>
<p>A stitched-together diagram is
<a href="Tree-RCU-Diagram.html">here</a>.
<h3><a name="Legal Statement">
Legal Statement</a></h3>
<p>This work represents the view of the author and does not necessarily
represent the view of IBM.
</p><p>Linux is a registered trademark of Linus Torvalds.
</p><p>Other company, product, and service names may be trademarks or
service marks of others.
</body></html>
Documentation/RCU/Design/Memory-Ordering/Tree-RCU-Memory-Ordering.rst 0 → 100644
View file @ ccc9971e
======================================================
A Tour Through TREE_RCU's Grace-Period Memory Ordering
======================================================
August 8, 2017
This article was contributed by Paul E.&nbsp;McKenney
Introduction
============
This document gives a rough visual overview of how Tree RCU's
grace-period memory ordering guarantee is provided.
What Is Tree RCU's Grace Period Memory Ordering Guarantee?
==========================================================
RCU grace periods provide extremely strong memory-ordering guarantees
for non-idle non-offline code.
Any code that happens after the end of a given RCU grace period is guaranteed
to see the effects of all accesses prior to the beginning of that grace
period that are within RCU read-side critical sections.
Similarly, any code that happens before the beginning of a given RCU grace
period is guaranteed to see the effects of all accesses following the end
of that grace period that are within RCU read-side critical sections.
Note well that RCU-sched read-side critical sections include any region
of code for which preemption is disabled.
Given that each individual machine instruction can be thought of as
an extremely small region of preemption-disabled code, one can think of
``synchronize_rcu()`` as ``smp_mb()`` on steroids.
RCU updaters use this guarantee by splitting their updates into
two phases, one of which is executed before the grace period and
the other of which is executed after the grace period.
In the most common use case, phase one removes an element from
a linked RCU-protected data structure, and phase two frees that element.
For this to work, any readers that have witnessed state prior to the
phase-one update (in the common case, removal) must not witness state
following the phase-two update (in the common case, freeing).
The RCU implementation provides this guarantee using a network
of lock-based critical sections, memory barriers, and per-CPU
processing, as is described in the following sections.
Tree RCU Grace Period Memory Ordering Building Blocks
=====================================================
The workhorse for RCU's grace-period memory ordering is the
critical section for the ``rcu_node`` structure's
``-&gt;lock``. These critical sections use helper functions for lock
acquisition, including ``raw_spin_lock_rcu_node()``,
``raw_spin_lock_irq_rcu_node()``, and ``raw_spin_lock_irqsave_rcu_node()``.
Their lock-release counterparts are ``raw_spin_unlock_rcu_node()``,
``raw_spin_unlock_irq_rcu_node()``, and
``raw_spin_unlock_irqrestore_rcu_node()``, respectively.
For completeness, a ``raw_spin_trylock_rcu_node()`` is also provided.
The key point is that the lock-acquisition functions, including
``raw_spin_trylock_rcu_node()``, all invoke ``smp_mb__after_unlock_lock()``
immediately after successful acquisition of the lock.
Therefore, for any given ``rcu_node`` structure, any access
happening before one of the above lock-release functions will be seen
by all CPUs as happening before any access happening after a later
one of the above lock-acquisition functions.
Furthermore, any access happening before one of the
above lock-release function on any given CPU will be seen by all
CPUs as happening before any access happening after a later one
of the above lock-acquisition functions executing on that same CPU,
even if the lock-release and lock-acquisition functions are operating
on different ``rcu_node`` structures.
Tree RCU uses these two ordering guarantees to form an ordering
network among all CPUs that were in any way involved in the grace
period, including any CPUs that came online or went offline during
the grace period in question.
The following litmus test exhibits the ordering effects of these
lock-acquisition and lock-release functions::
1 int x, y, z;
2
3 void task0(void)
4 {
5 raw_spin_lock_rcu_node(rnp);
6 WRITE_ONCE(x, 1);
7 r1 = READ_ONCE(y);
8 raw_spin_unlock_rcu_node(rnp);
9 }
10
11 void task1(void)
12 {
13 raw_spin_lock_rcu_node(rnp);
14 WRITE_ONCE(y, 1);
15 r2 = READ_ONCE(z);
16 raw_spin_unlock_rcu_node(rnp);
17 }
18
19 void task2(void)
20 {
21 WRITE_ONCE(z, 1);
22 smp_mb();
23 r3 = READ_ONCE(x);
24 }
25
26 WARN_ON(r1 == 0 &amp;&amp; r2 == 0 &amp;&amp; r3 == 0);
The ``WARN_ON()`` is evaluated at &ldquo;the end of time&rdquo;,
after all changes have propagated throughout the system.
Without the ``smp_mb__after_unlock_lock()`` provided by the
acquisition functions, this ``WARN_ON()`` could trigger, for example
on PowerPC.
The ``smp_mb__after_unlock_lock()`` invocations prevent this
``WARN_ON()`` from triggering.
This approach must be extended to include idle CPUs, which need
RCU's grace-period memory ordering guarantee to extend to any
RCU read-side critical sections preceding and following the current
idle sojourn.
This case is handled by calls to the strongly ordered
``atomic_add_return()`` read-modify-write atomic operation that
is invoked within ``rcu_dynticks_eqs_enter()`` at idle-entry
time and within ``rcu_dynticks_eqs_exit()`` at idle-exit time.
The grace-period kthread invokes ``rcu_dynticks_snap()`` and
``rcu_dynticks_in_eqs_since()`` (both of which invoke
an ``atomic_add_return()`` of zero) to detect idle CPUs.
+-----------------------------------------------------------------------+
| **Quick Quiz**: |
+-----------------------------------------------------------------------+
| But what about CPUs that remain offline for the entire grace period? |
+-----------------------------------------------------------------------+
| **Answer**: |
+-----------------------------------------------------------------------+
| Such CPUs will be offline at the beginning of the grace period, so |
| the grace period won't expect quiescent states from them. Races |
| between grace-period start and CPU-hotplug operations are mediated |
| by the CPU's leaf ``rcu_node`` structure's ``->lock`` as described |
| above. |
+-----------------------------------------------------------------------+
The approach must be extended to handle one final case, that of waking a
task blocked in ``synchronize_rcu()``. This task might be affinitied to
a CPU that is not yet aware that the grace period has ended, and thus
might not yet be subject to the grace period's memory ordering.
Therefore, there is an ``smp_mb()`` after the return from
``wait_for_completion()`` in the ``synchronize_rcu()`` code path.
+-----------------------------------------------------------------------+
| **Quick Quiz**: |
+-----------------------------------------------------------------------+
| What? Where??? I don't see any ``smp_mb()`` after the return from |
| ``wait_for_completion()``!!! |
+-----------------------------------------------------------------------+
| **Answer**: |
+-----------------------------------------------------------------------+
| That would be because I spotted the need for that ``smp_mb()`` during |
| the creation of this documentation, and it is therefore unlikely to |
| hit mainline before v4.14. Kudos to Lance Roy, Will Deacon, Peter |
| Zijlstra, and Jonathan Cameron for asking questions that sensitized |
| me to the rather elaborate sequence of events that demonstrate the |
| need for this memory barrier. |
+-----------------------------------------------------------------------+
Tree RCU's grace--period memory-ordering guarantees rely most heavily on
the ``rcu_node`` structure's ``->lock`` field, so much so that it is
necessary to abbreviate this pattern in the diagrams in the next
section. For example, consider the ``rcu_prepare_for_idle()`` function
shown below, which is one of several functions that enforce ordering of
newly arrived RCU callbacks against future grace periods:
::
1 static void rcu_prepare_for_idle(void)
2 {
3 bool needwake;
4 struct rcu_data *rdp;
5 struct rcu_dynticks *rdtp = this_cpu_ptr(&rcu_dynticks);
6 struct rcu_node *rnp;
7 struct rcu_state *rsp;
8 int tne;
9
10 if (IS_ENABLED(CONFIG_RCU_NOCB_CPU_ALL) ||
11 rcu_is_nocb_cpu(smp_processor_id()))
12 return;
13 tne = READ_ONCE(tick_nohz_active);
14 if (tne != rdtp->tick_nohz_enabled_snap) {
15 if (rcu_cpu_has_callbacks(NULL))
16 invoke_rcu_core();
17 rdtp->tick_nohz_enabled_snap = tne;
18 return;
19 }
20 if (!tne)
21 return;
22 if (rdtp->all_lazy &&
23 rdtp->nonlazy_posted != rdtp->nonlazy_posted_snap) {
24 rdtp->all_lazy = false;
25 rdtp->nonlazy_posted_snap = rdtp->nonlazy_posted;
26 invoke_rcu_core();
27 return;
28 }
29 if (rdtp->last_accelerate == jiffies)
30 return;
31 rdtp->last_accelerate = jiffies;
32 for_each_rcu_flavor(rsp) {
33 rdp = this_cpu_ptr(rsp->rda);
34 if (rcu_segcblist_pend_cbs(&rdp->cblist))
35 continue;
36 rnp = rdp->mynode;
37 raw_spin_lock_rcu_node(rnp);
38 needwake = rcu_accelerate_cbs(rsp, rnp, rdp);
39 raw_spin_unlock_rcu_node(rnp);
40 if (needwake)
41 rcu_gp_kthread_wake(rsp);
42 }
43 }
But the only part of ``rcu_prepare_for_idle()`` that really matters for
this discussion are lines 37–39. We will therefore abbreviate this
function as follows:
.. kernel-figure:: rcu_node-lock.svg
The box represents the ``rcu_node`` structure's ``->lock`` critical
section, with the double line on top representing the additional
``smp_mb__after_unlock_lock()``.
Tree RCU Grace Period Memory Ordering Components
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Tree RCU's grace-period memory-ordering guarantee is provided by a
number of RCU components:
#. `Callback Registry <#Callback%20Registry>`__
#. `Grace-Period Initialization <#Grace-Period%20Initialization>`__
#. `Self-Reported Quiescent
States <#Self-Reported%20Quiescent%20States>`__
#. `Dynamic Tick Interface <#Dynamic%20Tick%20Interface>`__
#. `CPU-Hotplug Interface <#CPU-Hotplug%20Interface>`__
#. `Forcing Quiescent States <Forcing%20Quiescent%20States>`__
#. `Grace-Period Cleanup <Grace-Period%20Cleanup>`__
#. `Callback Invocation <Callback%20Invocation>`__
Each of the following section looks at the corresponding component in
detail.
Callback Registry
^^^^^^^^^^^^^^^^^
If RCU's grace-period guarantee is to mean anything at all, any access
that happens before a given invocation of ``call_rcu()`` must also
happen before the corresponding grace period. The implementation of this
portion of RCU's grace period guarantee is shown in the following
figure:
.. kernel-figure:: TreeRCU-callback-registry.svg
Because ``call_rcu()`` normally acts only on CPU-local state, it
provides no ordering guarantees, either for itself or for phase one of
the update (which again will usually be removal of an element from an
RCU-protected data structure). It simply enqueues the ``rcu_head``
structure on a per-CPU list, which cannot become associated with a grace
period until a later call to ``rcu_accelerate_cbs()``, as shown in the
diagram above.
One set of code paths shown on the left invokes ``rcu_accelerate_cbs()``
via ``note_gp_changes()``, either directly from ``call_rcu()`` (if the
current CPU is inundated with queued ``rcu_head`` structures) or more
likely from an ``RCU_SOFTIRQ`` handler. Another code path in the middle
is taken only in kernels built with ``CONFIG_RCU_FAST_NO_HZ=y``, which
invokes ``rcu_accelerate_cbs()`` via ``rcu_prepare_for_idle()``. The
final code path on the right is taken only in kernels built with
``CONFIG_HOTPLUG_CPU=y``, which invokes ``rcu_accelerate_cbs()`` via
``rcu_advance_cbs()``, ``rcu_migrate_callbacks``,
``rcutree_migrate_callbacks()``, and ``takedown_cpu()``, which in turn
is invoked on a surviving CPU after the outgoing CPU has been completely
offlined.
There are a few other code paths within grace-period processing that
opportunistically invoke ``rcu_accelerate_cbs()``. However, either way,
all of the CPU's recently queued ``rcu_head`` structures are associated
with a future grace-period number under the protection of the CPU's lead
``rcu_node`` structure's ``->lock``. In all cases, there is full
ordering against any prior critical section for that same ``rcu_node``
structure's ``->lock``, and also full ordering against any of the
current task's or CPU's prior critical sections for any ``rcu_node``
structure's ``->lock``.
The next section will show how this ordering ensures that any accesses
prior to the ``call_rcu()`` (particularly including phase one of the
update) happen before the start of the corresponding grace period.
+-----------------------------------------------------------------------+
| **Quick Quiz**: |
+-----------------------------------------------------------------------+
| But what about ``synchronize_rcu()``? |
+-----------------------------------------------------------------------+
| **Answer**: |
+-----------------------------------------------------------------------+
| The ``synchronize_rcu()`` passes ``call_rcu()`` to ``wait_rcu_gp()``, |
| which invokes it. So either way, it eventually comes down to |
| ``call_rcu()``. |
+-----------------------------------------------------------------------+
Grace-Period Initialization
^^^^^^^^^^^^^^^^^^^^^^^^^^^
Grace-period initialization is carried out by the grace-period kernel
thread, which makes several passes over the ``rcu_node`` tree within the
``rcu_gp_init()`` function. This means that showing the full flow of
ordering through the grace-period computation will require duplicating
this tree. If you find this confusing, please note that the state of the
``rcu_node`` changes over time, just like Heraclitus's river. However,
to keep the ``rcu_node`` river tractable, the grace-period kernel
thread's traversals are presented in multiple parts, starting in this
section with the various phases of grace-period initialization.
The first ordering-related grace-period initialization action is to
advance the ``rcu_state`` structure's ``->gp_seq`` grace-period-number
counter, as shown below:
.. kernel-figure:: TreeRCU-gp-init-1.svg
The actual increment is carried out using ``smp_store_release()``, which
helps reject false-positive RCU CPU stall detection. Note that only the
root ``rcu_node`` structure is touched.
The first pass through the ``rcu_node`` tree updates bitmasks based on
CPUs having come online or gone offline since the start of the previous
grace period. In the common case where the number of online CPUs for
this ``rcu_node`` structure has not transitioned to or from zero, this
pass will scan only the leaf ``rcu_node`` structures. However, if the
number of online CPUs for a given leaf ``rcu_node`` structure has
transitioned from zero, ``rcu_init_new_rnp()`` will be invoked for the
first incoming CPU. Similarly, if the number of online CPUs for a given
leaf ``rcu_node`` structure has transitioned to zero,
``rcu_cleanup_dead_rnp()`` will be invoked for the last outgoing CPU.
The diagram below shows the path of ordering if the leftmost
``rcu_node`` structure onlines its first CPU and if the next
``rcu_node`` structure has no online CPUs (or, alternatively if the
leftmost ``rcu_node`` structure offlines its last CPU and if the next
``rcu_node`` structure has no online CPUs).
.. kernel-figure:: TreeRCU-gp-init-1.svg
The final ``rcu_gp_init()`` pass through the ``rcu_node`` tree traverses
breadth-first, setting each ``rcu_node`` structure's ``->gp_seq`` field
to the newly advanced value from the ``rcu_state`` structure, as shown
in the following diagram.
.. kernel-figure:: TreeRCU-gp-init-1.svg
This change will also cause each CPU's next call to
``__note_gp_changes()`` to notice that a new grace period has started,
as described in the next section. But because the grace-period kthread
started the grace period at the root (with the advancing of the
``rcu_state`` structure's ``->gp_seq`` field) before setting each leaf
``rcu_node`` structure's ``->gp_seq`` field, each CPU's observation of
the start of the grace period will happen after the actual start of the
grace period.
+-----------------------------------------------------------------------+
| **Quick Quiz**: |
+-----------------------------------------------------------------------+
| But what about the CPU that started the grace period? Why wouldn't it |
| see the start of the grace period right when it started that grace |
| period? |
+-----------------------------------------------------------------------+
| **Answer**: |
+-----------------------------------------------------------------------+
| In some deep philosophical and overly anthromorphized sense, yes, the |
| CPU starting the grace period is immediately aware of having done so. |
| However, if we instead assume that RCU is not self-aware, then even |
| the CPU starting the grace period does not really become aware of the |
| start of this grace period until its first call to |
| ``__note_gp_changes()``. On the other hand, this CPU potentially gets |
| early notification because it invokes ``__note_gp_changes()`` during |
| its last ``rcu_gp_init()`` pass through its leaf ``rcu_node`` |
| structure. |
+-----------------------------------------------------------------------+
Self-Reported Quiescent States
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
When all entities that might block the grace period have reported
quiescent states (or as described in a later section, had quiescent
states reported on their behalf), the grace period can end. Online
non-idle CPUs report their own quiescent states, as shown in the
following diagram:
.. kernel-figure:: TreeRCU-qs.svg
This is for the last CPU to report a quiescent state, which signals the
end of the grace period. Earlier quiescent states would push up the
``rcu_node`` tree only until they encountered an ``rcu_node`` structure
that is waiting for additional quiescent states. However, ordering is
nevertheless preserved because some later quiescent state will acquire
that ``rcu_node`` structure's ``->lock``.
Any number of events can lead up to a CPU invoking ``note_gp_changes``
(or alternatively, directly invoking ``__note_gp_changes()``), at which
point that CPU will notice the start of a new grace period while holding
its leaf ``rcu_node`` lock. Therefore, all execution shown in this
diagram happens after the start of the grace period. In addition, this
CPU will consider any RCU read-side critical section that started before
the invocation of ``__note_gp_changes()`` to have started before the
grace period, and thus a critical section that the grace period must
wait on.
+-----------------------------------------------------------------------+
| **Quick Quiz**: |
+-----------------------------------------------------------------------+
| But a RCU read-side critical section might have started after the |
| beginning of the grace period (the advancing of ``->gp_seq`` from |
| earlier), so why should the grace period wait on such a critical |
| section? |
+-----------------------------------------------------------------------+
| **Answer**: |
+-----------------------------------------------------------------------+
| It is indeed not necessary for the grace period to wait on such a |
| critical section. However, it is permissible to wait on it. And it is |
| furthermore important to wait on it, as this lazy approach is far |
| more scalable than a “big bang” all-at-once grace-period start could |
| possibly be. |
+-----------------------------------------------------------------------+
If the CPU does a context switch, a quiescent state will be noted by
``rcu_node_context_switch()`` on the left. On the other hand, if the CPU
takes a scheduler-clock interrupt while executing in usermode, a
quiescent state will be noted by ``rcu_sched_clock_irq()`` on the right.
Either way, the passage through a quiescent state will be noted in a
per-CPU variable.
The next time an ``RCU_SOFTIRQ`` handler executes on this CPU (for
example, after the next scheduler-clock interrupt), ``rcu_core()`` will
invoke ``rcu_check_quiescent_state()``, which will notice the recorded
quiescent state, and invoke ``rcu_report_qs_rdp()``. If
``rcu_report_qs_rdp()`` verifies that the quiescent state really does
apply to the current grace period, it invokes ``rcu_report_rnp()`` which
traverses up the ``rcu_node`` tree as shown at the bottom of the
diagram, clearing bits from each ``rcu_node`` structure's ``->qsmask``
field, and propagating up the tree when the result is zero.
Note that traversal passes upwards out of a given ``rcu_node`` structure
only if the current CPU is reporting the last quiescent state for the
subtree headed by that ``rcu_node`` structure. A key point is that if a
CPU's traversal stops at a given ``rcu_node`` structure, then there will
be a later traversal by another CPU (or perhaps the same one) that
proceeds upwards from that point, and the ``rcu_node`` ``->lock``
guarantees that the first CPU's quiescent state happens before the
remainder of the second CPU's traversal. Applying this line of thought
repeatedly shows that all CPUs' quiescent states happen before the last
CPU traverses through the root ``rcu_node`` structure, the “last CPU”
being the one that clears the last bit in the root ``rcu_node``
structure's ``->qsmask`` field.
Dynamic Tick Interface
^^^^^^^^^^^^^^^^^^^^^^
Due to energy-efficiency considerations, RCU is forbidden from
disturbing idle CPUs. CPUs are therefore required to notify RCU when
entering or leaving idle state, which they do via fully ordered
value-returning atomic operations on a per-CPU variable. The ordering
effects are as shown below:
.. kernel-figure:: TreeRCU-dyntick.svg
The RCU grace-period kernel thread samples the per-CPU idleness variable
while holding the corresponding CPU's leaf ``rcu_node`` structure's
``->lock``. This means that any RCU read-side critical sections that
precede the idle period (the oval near the top of the diagram above)
will happen before the end of the current grace period. Similarly, the
beginning of the current grace period will happen before any RCU
read-side critical sections that follow the idle period (the oval near
the bottom of the diagram above).
Plumbing this into the full grace-period execution is described
`below <#Forcing%20Quiescent%20States>`__.
CPU-Hotplug Interface
^^^^^^^^^^^^^^^^^^^^^
RCU is also forbidden from disturbing offline CPUs, which might well be
powered off and removed from the system completely. CPUs are therefore
required to notify RCU of their comings and goings as part of the
corresponding CPU hotplug operations. The ordering effects are shown
below:
.. kernel-figure:: TreeRCU-hotplug.svg
Because CPU hotplug operations are much less frequent than idle
transitions, they are heavier weight, and thus acquire the CPU's leaf
``rcu_node`` structure's ``->lock`` and update this structure's
``->qsmaskinitnext``. The RCU grace-period kernel thread samples this
mask to detect CPUs having gone offline since the beginning of this
grace period.
Plumbing this into the full grace-period execution is described
`below <#Forcing%20Quiescent%20States>`__.
Forcing Quiescent States
^^^^^^^^^^^^^^^^^^^^^^^^
As noted above, idle and offline CPUs cannot report their own quiescent
states, and therefore the grace-period kernel thread must do the
reporting on their behalf. This process is called “forcing quiescent
states”, it is repeated every few jiffies, and its ordering effects are
shown below:
.. kernel-figure:: TreeRCU-gp-fqs.svg
Each pass of quiescent state forcing is guaranteed to traverse the leaf
``rcu_node`` structures, and if there are no new quiescent states due to
recently idled and/or offlined CPUs, then only the leaves are traversed.
However, if there is a newly offlined CPU as illustrated on the left or
a newly idled CPU as illustrated on the right, the corresponding
quiescent state will be driven up towards the root. As with
self-reported quiescent states, the upwards driving stops once it
reaches an ``rcu_node`` structure that has quiescent states outstanding
from other CPUs.
+-----------------------------------------------------------------------+
| **Quick Quiz**: |
+-----------------------------------------------------------------------+
| The leftmost drive to root stopped before it reached the root |
| ``rcu_node`` structure, which means that there are still CPUs |
| subordinate to that structure on which the current grace period is |
| waiting. Given that, how is it possible that the rightmost drive to |
| root ended the grace period? |
+-----------------------------------------------------------------------+
| **Answer**: |
+-----------------------------------------------------------------------+
| Good analysis! It is in fact impossible in the absence of bugs in |
| RCU. But this diagram is complex enough as it is, so simplicity |
| overrode accuracy. You can think of it as poetic license, or you can |
| think of it as misdirection that is resolved in the |
| `stitched-together diagram <#Putting%20It%20All%20Together>`__. |
+-----------------------------------------------------------------------+
Grace-Period Cleanup
^^^^^^^^^^^^^^^^^^^^
Grace-period cleanup first scans the ``rcu_node`` tree breadth-first
advancing all the ``->gp_seq`` fields, then it advances the
``rcu_state`` structure's ``->gp_seq`` field. The ordering effects are
shown below:
.. kernel-figure:: TreeRCU-gp-cleanup.svg
As indicated by the oval at the bottom of the diagram, once grace-period
cleanup is complete, the next grace period can begin.
+-----------------------------------------------------------------------+
| **Quick Quiz**: |
+-----------------------------------------------------------------------+
| But when precisely does the grace period end? |
+-----------------------------------------------------------------------+
| **Answer**: |
+-----------------------------------------------------------------------+
| There is no useful single point at which the grace period can be said |
| to end. The earliest reasonable candidate is as soon as the last CPU |
| has reported its quiescent state, but it may be some milliseconds |
| before RCU becomes aware of this. The latest reasonable candidate is |
| once the ``rcu_state`` structure's ``->gp_seq`` field has been |
| updated, but it is quite possible that some CPUs have already |
| completed phase two of their updates by that time. In short, if you |
| are going to work with RCU, you need to learn to embrace uncertainty. |
+-----------------------------------------------------------------------+
Callback Invocation
^^^^^^^^^^^^^^^^^^^
Once a given CPU's leaf ``rcu_node`` structure's ``->gp_seq`` field has
been updated, that CPU can begin invoking its RCU callbacks that were
waiting for this grace period to end. These callbacks are identified by
``rcu_advance_cbs()``, which is usually invoked by
``__note_gp_changes()``. As shown in the diagram below, this invocation
can be triggered by the scheduling-clock interrupt
(``rcu_sched_clock_irq()`` on the left) or by idle entry
(``rcu_cleanup_after_idle()`` on the right, but only for kernels build
with ``CONFIG_RCU_FAST_NO_HZ=y``). Either way, ``RCU_SOFTIRQ`` is
raised, which results in ``rcu_do_batch()`` invoking the callbacks,
which in turn allows those callbacks to carry out (either directly or
indirectly via wakeup) the needed phase-two processing for each update.
.. kernel-figure:: TreeRCU-callback-invocation.svg
Please note that callback invocation can also be prompted by any number
of corner-case code paths, for example, when a CPU notes that it has
excessive numbers of callbacks queued. In all cases, the CPU acquires
its leaf ``rcu_node`` structure's ``->lock`` before invoking callbacks,
which preserves the required ordering against the newly completed grace
period.
However, if the callback function communicates to other CPUs, for
example, doing a wakeup, then it is that function's responsibility to
maintain ordering. For example, if the callback function wakes up a task
that runs on some other CPU, proper ordering must in place in both the
callback function and the task being awakened. To see why this is
important, consider the top half of the `grace-period
cleanup <#Grace-Period%20Cleanup>`__ diagram. The callback might be
running on a CPU corresponding to the leftmost leaf ``rcu_node``
structure, and awaken a task that is to run on a CPU corresponding to
the rightmost leaf ``rcu_node`` structure, and the grace-period kernel
thread might not yet have reached the rightmost leaf. In this case, the
grace period's memory ordering might not yet have reached that CPU, so
again the callback function and the awakened task must supply proper
ordering.
Putting It All Together
~~~~~~~~~~~~~~~~~~~~~~~
A stitched-together diagram is here:
.. kernel-figure:: TreeRCU-gp.svg
Legal Statement
~~~~~~~~~~~~~~~
This work represents the view of the author and does not necessarily
represent the view of IBM.
Linux is a registered trademark of Linus Torvalds.
Other company, product, and service names may be trademarks or service
marks of others.
Documentation/RCU/Design/Requirements/Requirements.html deleted 100644 → 0
View file @ c07e6f36
This source diff could not be displayed because it is too large. You can view the blob instead.
Documentation/RCU/Design/Requirements/Requirements.rst 0 → 100644
View file @ ccc9971e
This source diff could not be displayed because it is too large. You can view the blob instead.
Documentation/RCU/index.rst
View file @ ccc9971e
......@@ -11,6 +11,11 @@ RCU concepts
listRCU
UP
Design/Memory-Ordering/Tree-RCU-Memory-Ordering
Design/Expedited-Grace-Periods/Expedited-Grace-Periods
Design/Requirements/Requirements
Design/Data-Structures/Data-Structures
.. only:: subproject and html
Indices
......
Documentation/RCU/whatisRCU.txt
View file @ ccc9971e
......@@ -302,7 +302,7 @@ rcu_dereference()
must prohibit. The rcu_dereference_protected() variant takes
a lockdep expression to indicate which locks must be acquired
by the caller. If the indicated protection is not provided,
a lockdep splat is emitted. See RCU/Design/Requirements/Requirements.html
a lockdep splat is emitted. See Documentation/RCU/Design/Requirements/Requirements.rst
and the API's code comments for more details and example usage.
The following diagram shows how each API communicates among the
......@@ -630,7 +630,7 @@ been able to write-acquire the lock otherwise. The smp_mb__after_spinlock()
promotes synchronize_rcu() to a full memory barrier in compliance with
the "Memory-Barrier Guarantees" listed in:
Documentation/RCU/Design/Requirements/Requirements.html.
Documentation/RCU/Design/Requirements/Requirements.rst
It is possible to nest rcu_read_lock(), since reader-writer locks may
be recursively acquired. Note also that rcu_read_lock() is immune
......
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment
GitLab Nexedi Edition | About GitLab | About Nexedi | 沪ICP备2021021310号-2 | 沪ICP备2021021310号-7