Commit 026b98a9 authored by Mel Gorman's avatar Mel Gorman Committed by Peter Zijlstra

sched/numa: Adjust imb_numa_nr to a better approximation of memory channels

For a single LLC per node, a NUMA imbalance is allowed up until 25%
of CPUs sharing a node could be active. One intent of the cut-off is
to avoid an imbalance of memory channels but there is no topological
information based on active memory channels. Furthermore, there can
be differences between nodes depending on the number of populated
DIMMs.

A cut-off of 25% was arbitrary but generally worked. It does have a severe
corner cases though when an parallel workload is using 25% of all available
CPUs over-saturates memory channels. This can happen due to the initial
forking of tasks that get pulled more to one node after early wakeups
(e.g. a barrier synchronisation) that is not quickly corrected by the
load balancer. The LB may fail to act quickly as the parallel tasks are
considered to be poor migrate candidates due to locality or cache hotness.

On a range of modern Intel CPUs, 12.5% appears to be a better cut-off
assuming all memory channels are populated and is used as the new cut-off
point. A minimum of 1 is specified to allow a communicating pair to
remain local even for CPUs with low numbers of cores. For modern AMDs,
there are multiple LLCs and are not affected.
Signed-off-by: default avatarMel Gorman <mgorman@techsingularity.net>
Signed-off-by: default avatarPeter Zijlstra (Intel) <peterz@infradead.org>
Tested-by: default avatarK Prateek Nayak <kprateek.nayak@amd.com>
Link: https://lore.kernel.org/r/20220520103519.1863-5-mgorman@techsingularity.net
parent cb29a5c1
...@@ -2316,23 +2316,30 @@ build_sched_domains(const struct cpumask *cpu_map, struct sched_domain_attr *att ...@@ -2316,23 +2316,30 @@ build_sched_domains(const struct cpumask *cpu_map, struct sched_domain_attr *att
/* /*
* For a single LLC per node, allow an * For a single LLC per node, allow an
* imbalance up to 25% of the node. This is an * imbalance up to 12.5% of the node. This is
* arbitrary cutoff based on SMT-2 to balance * arbitrary cutoff based two factors -- SMT and
* between memory bandwidth and avoiding * memory channels. For SMT-2, the intent is to
* premature sharing of HT resources and SMT-4 * avoid premature sharing of HT resources but
* or SMT-8 *may* benefit from a different * SMT-4 or SMT-8 *may* benefit from a different
* cutoff. * cutoff. For memory channels, this is a very
* rough estimate of how many channels may be
* active and is based on recent CPUs with
* many cores.
* *
* For multiple LLCs, allow an imbalance * For multiple LLCs, allow an imbalance
* until multiple tasks would share an LLC * until multiple tasks would share an LLC
* on one node while LLCs on another node * on one node while LLCs on another node
* remain idle. * remain idle. This assumes that there are
* enough logical CPUs per LLC to avoid SMT
* factors and that there is a correlation
* between LLCs and memory channels.
*/ */
nr_llcs = sd->span_weight / child->span_weight; nr_llcs = sd->span_weight / child->span_weight;
if (nr_llcs == 1) if (nr_llcs == 1)
imb = sd->span_weight >> 2; imb = sd->span_weight >> 3;
else else
imb = nr_llcs; imb = nr_llcs;
imb = max(1U, imb);
sd->imb_numa_nr = imb; sd->imb_numa_nr = imb;
/* Set span based on the first NUMA domain. */ /* Set span based on the first NUMA domain. */
......
Markdown is supported
0%
or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment