Commit b30aef17 authored by Ken Chen's avatar Ken Chen Committed by Ingo Molnar

sched: Fix erroneous all_pinned logic

The scheduler load balancer has specific code to deal with cases of
unbalanced system due to lots of unmovable tasks (for example because of
hard CPU affinity). In those situation, it excludes the busiest CPU that
has pinned tasks for load balance consideration such that it can perform
second 2nd load balance pass on the rest of the system.

This all works as designed if there is only one cgroup in the system.

However, when we have multiple cgroups, this logic has false positives and
triggers multiple load balance passes despite there are actually no pinned
tasks at all.

The reason it has false positives is that the all pinned logic is deep in
the lowest function of can_migrate_task() and is too low level:

load_balance_fair() iterates each task group and calls balance_tasks() to
migrate target load. Along the way, balance_tasks() will also set a
all_pinned variable. Given that task-groups are iterated, this all_pinned
variable is essentially the status of last group in the scanning process.
Task group can have number of reasons that no load being migrated, none
due to cpu affinity. However, this status bit is being propagated back up
to the higher level load_balance(), which incorrectly think that no tasks
were moved.  It kick off the all pinned logic and start multiple passes
attempt to move load onto puller CPU.

To fix this, move the all_pinned aggregation up at the iterator level.
This ensures that the status is aggregated over all task-groups, not just
last one in the list.
Signed-off-by: default avatarKen Chen <kenchen@google.com>
Cc: stable@kernel.org
Signed-off-by: default avatarPeter Zijlstra <a.p.zijlstra@chello.nl>
Link: http://lkml.kernel.org/r/BANLkTi=ernzNawaR5tJZEsV_QVnfxqXmsQ@mail.gmail.comSigned-off-by: default avatarIngo Molnar <mingo@elte.hu>
parent b0432d8f
...@@ -2104,21 +2104,20 @@ balance_tasks(struct rq *this_rq, int this_cpu, struct rq *busiest, ...@@ -2104,21 +2104,20 @@ balance_tasks(struct rq *this_rq, int this_cpu, struct rq *busiest,
enum cpu_idle_type idle, int *all_pinned, enum cpu_idle_type idle, int *all_pinned,
int *this_best_prio, struct cfs_rq *busiest_cfs_rq) int *this_best_prio, struct cfs_rq *busiest_cfs_rq)
{ {
int loops = 0, pulled = 0, pinned = 0; int loops = 0, pulled = 0;
long rem_load_move = max_load_move; long rem_load_move = max_load_move;
struct task_struct *p, *n; struct task_struct *p, *n;
if (max_load_move == 0) if (max_load_move == 0)
goto out; goto out;
pinned = 1;
list_for_each_entry_safe(p, n, &busiest_cfs_rq->tasks, se.group_node) { list_for_each_entry_safe(p, n, &busiest_cfs_rq->tasks, se.group_node) {
if (loops++ > sysctl_sched_nr_migrate) if (loops++ > sysctl_sched_nr_migrate)
break; break;
if ((p->se.load.weight >> 1) > rem_load_move || if ((p->se.load.weight >> 1) > rem_load_move ||
!can_migrate_task(p, busiest, this_cpu, sd, idle, &pinned)) !can_migrate_task(p, busiest, this_cpu, sd, idle,
all_pinned))
continue; continue;
pull_task(busiest, p, this_rq, this_cpu); pull_task(busiest, p, this_rq, this_cpu);
...@@ -2153,9 +2152,6 @@ balance_tasks(struct rq *this_rq, int this_cpu, struct rq *busiest, ...@@ -2153,9 +2152,6 @@ balance_tasks(struct rq *this_rq, int this_cpu, struct rq *busiest,
*/ */
schedstat_add(sd, lb_gained[idle], pulled); schedstat_add(sd, lb_gained[idle], pulled);
if (all_pinned)
*all_pinned = pinned;
return max_load_move - rem_load_move; return max_load_move - rem_load_move;
} }
...@@ -3341,6 +3337,7 @@ static int load_balance(int this_cpu, struct rq *this_rq, ...@@ -3341,6 +3337,7 @@ static int load_balance(int this_cpu, struct rq *this_rq,
* still unbalanced. ld_moved simply stays zero, so it is * still unbalanced. ld_moved simply stays zero, so it is
* correctly treated as an imbalance. * correctly treated as an imbalance.
*/ */
all_pinned = 1;
local_irq_save(flags); local_irq_save(flags);
double_rq_lock(this_rq, busiest); double_rq_lock(this_rq, busiest);
ld_moved = move_tasks(this_rq, this_cpu, busiest, ld_moved = move_tasks(this_rq, this_cpu, busiest,
......
Markdown is supported
0%
or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment