Commit 45802da0 authored by Linus Torvalds's avatar Linus Torvalds

Merge branch 'sched-core-for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip

Pull scheduler updates from Ingo Molnar:
 "The main changes in this cycle were:

   - refcount conversions

   - Solve the rq->leaf_cfs_rq_list can of worms for real.

   - improve power-aware scheduling

   - add sysctl knob for Energy Aware Scheduling

   - documentation updates

   - misc other changes"

* 'sched-core-for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip: (34 commits)
  kthread: Do not use TIMER_IRQSAFE
  kthread: Convert worker lock to raw spinlock
  sched/fair: Use non-atomic cpumask_{set,clear}_cpu()
  sched/fair: Remove unused 'sd' parameter from select_idle_smt()
  sched/wait: Use freezable_schedule() when possible
  sched/fair: Prune, fix and simplify the nohz_balancer_kick() comment block
  sched/fair: Explain LLC nohz kick condition
  sched/fair: Simplify nohz_balancer_kick()
  sched/topology: Fix percpu data types in struct sd_data & struct s_data
  sched/fair: Simplify post_init_entity_util_avg() by calling it with a task_struct pointer argument
  sched/fair: Fix O(nr_cgroups) in the load balancing path
  sched/fair: Optimize update_blocked_averages()
  sched/fair: Fix insertion in rq->leaf_cfs_rq_list
  sched/fair: Add tmp_alone_branch assertion
  sched/core: Use READ_ONCE()/WRITE_ONCE() in move_queued_task()/task_rq_lock()
  sched/debug: Initialize sd_sysctl_cpus if !CONFIG_CPUMASK_OFFSTACK
  sched/pelt: Skip updating util_est when utilization is higher than CPU's capacity
  sched/fair: Update scale invariance of PELT
  sched/fair: Move the rq_of() helper function
  sched/core: Convert task_struct.stack_refcount to refcount_t
  ...
parents 203b6609 ad01423a
====================
Energy Model of CPUs
====================
1. Overview
-----------
The Energy Model (EM) framework serves as an interface between drivers knowing
the power consumed by CPUs at various performance levels, and the kernel
subsystems willing to use that information to make energy-aware decisions.
The source of the information about the power consumed by CPUs can vary greatly
from one platform to another. These power costs can be estimated using
devicetree data in some cases. In others, the firmware will know better.
Alternatively, userspace might be best positioned. And so on. In order to avoid
each and every client subsystem to re-implement support for each and every
possible source of information on its own, the EM framework intervenes as an
abstraction layer which standardizes the format of power cost tables in the
kernel, hence enabling to avoid redundant work.
The figure below depicts an example of drivers (Arm-specific here, but the
approach is applicable to any architecture) providing power costs to the EM
framework, and interested clients reading the data from it.
+---------------+ +-----------------+ +---------------+
| Thermal (IPA) | | Scheduler (EAS) | | Other |
+---------------+ +-----------------+ +---------------+
| | em_pd_energy() |
| | em_cpu_get() |
+---------+ | +---------+
| | |
v v v
+---------------------+
| Energy Model |
| Framework |
+---------------------+
^ ^ ^
| | | em_register_perf_domain()
+----------+ | +---------+
| | |
+---------------+ +---------------+ +--------------+
| cpufreq-dt | | arm_scmi | | Other |
+---------------+ +---------------+ +--------------+
^ ^ ^
| | |
+--------------+ +---------------+ +--------------+
| Device Tree | | Firmware | | ? |
+--------------+ +---------------+ +--------------+
The EM framework manages power cost tables per 'performance domain' in the
system. A performance domain is a group of CPUs whose performance is scaled
together. Performance domains generally have a 1-to-1 mapping with CPUFreq
policies. All CPUs in a performance domain are required to have the same
micro-architecture. CPUs in different performance domains can have different
micro-architectures.
2. Core APIs
------------
2.1 Config options
CONFIG_ENERGY_MODEL must be enabled to use the EM framework.
2.2 Registration of performance domains
Drivers are expected to register performance domains into the EM framework by
calling the following API:
int em_register_perf_domain(cpumask_t *span, unsigned int nr_states,
struct em_data_callback *cb);
Drivers must specify the CPUs of the performance domains using the cpumask
argument, and provide a callback function returning <frequency, power> tuples
for each capacity state. The callback function provided by the driver is free
to fetch data from any relevant location (DT, firmware, ...), and by any mean
deemed necessary. See Section 3. for an example of driver implementing this
callback, and kernel/power/energy_model.c for further documentation on this
API.
2.3 Accessing performance domains
Subsystems interested in the energy model of a CPU can retrieve it using the
em_cpu_get() API. The energy model tables are allocated once upon creation of
the performance domains, and kept in memory untouched.
The energy consumed by a performance domain can be estimated using the
em_pd_energy() API. The estimation is performed assuming that the schedutil
CPUfreq governor is in use.
More details about the above APIs can be found in include/linux/energy_model.h.
3. Example driver
-----------------
This section provides a simple example of a CPUFreq driver registering a
performance domain in the Energy Model framework using the (fake) 'foo'
protocol. The driver implements an est_power() function to be provided to the
EM framework.
-> drivers/cpufreq/foo_cpufreq.c
01 static int est_power(unsigned long *mW, unsigned long *KHz, int cpu)
02 {
03 long freq, power;
04
05 /* Use the 'foo' protocol to ceil the frequency */
06 freq = foo_get_freq_ceil(cpu, *KHz);
07 if (freq < 0);
08 return freq;
09
10 /* Estimate the power cost for the CPU at the relevant freq. */
11 power = foo_estimate_power(cpu, freq);
12 if (power < 0);
13 return power;
14
15 /* Return the values to the EM framework */
16 *mW = power;
17 *KHz = freq;
18
19 return 0;
20 }
21
22 static int foo_cpufreq_init(struct cpufreq_policy *policy)
23 {
24 struct em_data_callback em_cb = EM_DATA_CB(est_power);
25 int nr_opp, ret;
26
27 /* Do the actual CPUFreq init work ... */
28 ret = do_foo_cpufreq_init(policy);
29 if (ret)
30 return ret;
31
32 /* Find the number of OPPs for this policy */
33 nr_opp = foo_get_nr_opp(policy);
34
35 /* And register the new performance domain */
36 em_register_perf_domain(policy->cpus, nr_opp, &em_cb);
37
38 return 0;
39 }
This diff is collapsed.
...@@ -79,6 +79,7 @@ show up in /proc/sys/kernel: ...@@ -79,6 +79,7 @@ show up in /proc/sys/kernel:
- reboot-cmd [ SPARC only ] - reboot-cmd [ SPARC only ]
- rtsig-max - rtsig-max
- rtsig-nr - rtsig-nr
- sched_energy_aware
- seccomp/ ==> Documentation/userspace-api/seccomp_filter.rst - seccomp/ ==> Documentation/userspace-api/seccomp_filter.rst
- sem - sem
- sem_next_id [ sysv ipc ] - sem_next_id [ sysv ipc ]
...@@ -890,6 +891,17 @@ rtsig-nr shows the number of RT signals currently queued. ...@@ -890,6 +891,17 @@ rtsig-nr shows the number of RT signals currently queued.
============================================================== ==============================================================
sched_energy_aware:
Enables/disables Energy Aware Scheduling (EAS). EAS starts
automatically on platforms where it can run (that is,
platforms with asymmetric CPU topologies and having an Energy
Model available). If your platform happens to meet the
requirements for EAS but you do not want to use it, change
this value to 0.
==============================================================
sched_schedstats: sched_schedstats:
Enables/disables scheduler statistics. Enabling this feature Enables/disables scheduler statistics. Enabling this feature
......
...@@ -12280,14 +12280,6 @@ S: Maintained ...@@ -12280,14 +12280,6 @@ S: Maintained
F: drivers/net/ppp/pptp.c F: drivers/net/ppp/pptp.c
W: http://sourceforge.net/projects/accel-pptp W: http://sourceforge.net/projects/accel-pptp
PREEMPTIBLE KERNEL
M: Robert Love <rml@tech9.net>
L: kpreempt-tech@lists.sourceforge.net
W: https://www.kernel.org/pub/linux/kernel/people/rml/preempt-kernel
S: Supported
F: Documentation/preempt-locking.txt
F: include/linux/preempt.h
PRINTK PRINTK
M: Petr Mladek <pmladek@suse.com> M: Petr Mladek <pmladek@suse.com>
M: Sergey Senozhatsky <sergey.senozhatsky@gmail.com> M: Sergey Senozhatsky <sergey.senozhatsky@gmail.com>
...@@ -13525,6 +13517,7 @@ F: kernel/sched/ ...@@ -13525,6 +13517,7 @@ F: kernel/sched/
F: include/linux/sched.h F: include/linux/sched.h
F: include/uapi/linux/sched.h F: include/uapi/linux/sched.h
F: include/linux/wait.h F: include/linux/wait.h
F: include/linux/preempt.h
SCR24X CHIP CARD INTERFACE DRIVER SCR24X CHIP CARD INTERFACE DRIVER
M: Lubomir Rintel <lkundrak@v3.sk> M: Lubomir Rintel <lkundrak@v3.sk>
......
...@@ -1189,7 +1189,7 @@ static int de_thread(struct task_struct *tsk) ...@@ -1189,7 +1189,7 @@ static int de_thread(struct task_struct *tsk)
flush_itimer_signals(); flush_itimer_signals();
#endif #endif
if (atomic_read(&oldsighand->count) != 1) { if (refcount_read(&oldsighand->count) != 1) {
struct sighand_struct *newsighand; struct sighand_struct *newsighand;
/* /*
* This ->sighand is shared with the CLONE_SIGHAND * This ->sighand is shared with the CLONE_SIGHAND
...@@ -1199,7 +1199,7 @@ static int de_thread(struct task_struct *tsk) ...@@ -1199,7 +1199,7 @@ static int de_thread(struct task_struct *tsk)
if (!newsighand) if (!newsighand)
return -ENOMEM; return -ENOMEM;
atomic_set(&newsighand->count, 1); refcount_set(&newsighand->count, 1);
memcpy(newsighand->action, oldsighand->action, memcpy(newsighand->action, oldsighand->action,
sizeof(newsighand->action)); sizeof(newsighand->action));
......
...@@ -64,7 +64,7 @@ void task_mem(struct seq_file *m, struct mm_struct *mm) ...@@ -64,7 +64,7 @@ void task_mem(struct seq_file *m, struct mm_struct *mm)
else else
bytes += kobjsize(current->files); bytes += kobjsize(current->files);
if (current->sighand && atomic_read(&current->sighand->count) > 1) if (current->sighand && refcount_read(&current->sighand->count) > 1)
sbytes += kobjsize(current->sighand); sbytes += kobjsize(current->sighand);
else else
bytes += kobjsize(current->sighand); bytes += kobjsize(current->sighand);
......
...@@ -13,6 +13,7 @@ ...@@ -13,6 +13,7 @@
#include <linux/securebits.h> #include <linux/securebits.h>
#include <linux/seqlock.h> #include <linux/seqlock.h>
#include <linux/rbtree.h> #include <linux/rbtree.h>
#include <linux/refcount.h>
#include <linux/sched/autogroup.h> #include <linux/sched/autogroup.h>
#include <net/net_namespace.h> #include <net/net_namespace.h>
#include <linux/sched/rt.h> #include <linux/sched/rt.h>
......
...@@ -86,7 +86,7 @@ enum { ...@@ -86,7 +86,7 @@ enum {
struct kthread_worker { struct kthread_worker {
unsigned int flags; unsigned int flags;
spinlock_t lock; raw_spinlock_t lock;
struct list_head work_list; struct list_head work_list;
struct list_head delayed_work_list; struct list_head delayed_work_list;
struct task_struct *task; struct task_struct *task;
...@@ -107,7 +107,7 @@ struct kthread_delayed_work { ...@@ -107,7 +107,7 @@ struct kthread_delayed_work {
}; };
#define KTHREAD_WORKER_INIT(worker) { \ #define KTHREAD_WORKER_INIT(worker) { \
.lock = __SPIN_LOCK_UNLOCKED((worker).lock), \ .lock = __RAW_SPIN_LOCK_UNLOCKED((worker).lock), \
.work_list = LIST_HEAD_INIT((worker).work_list), \ .work_list = LIST_HEAD_INIT((worker).work_list), \
.delayed_work_list = LIST_HEAD_INIT((worker).delayed_work_list),\ .delayed_work_list = LIST_HEAD_INIT((worker).delayed_work_list),\
} }
...@@ -165,9 +165,8 @@ extern void __kthread_init_worker(struct kthread_worker *worker, ...@@ -165,9 +165,8 @@ extern void __kthread_init_worker(struct kthread_worker *worker,
#define kthread_init_delayed_work(dwork, fn) \ #define kthread_init_delayed_work(dwork, fn) \
do { \ do { \
kthread_init_work(&(dwork)->work, (fn)); \ kthread_init_work(&(dwork)->work, (fn)); \
__init_timer(&(dwork)->timer, \ timer_setup(&(dwork)->timer, \
kthread_delayed_work_timer_fn, \ kthread_delayed_work_timer_fn, 0); \
TIMER_IRQSAFE); \
} while (0) } while (0)
int kthread_worker_fn(void *worker_ptr); int kthread_worker_fn(void *worker_ptr);
......
...@@ -21,6 +21,7 @@ ...@@ -21,6 +21,7 @@
#include <linux/seccomp.h> #include <linux/seccomp.h>
#include <linux/nodemask.h> #include <linux/nodemask.h>
#include <linux/rcupdate.h> #include <linux/rcupdate.h>
#include <linux/refcount.h>
#include <linux/resource.h> #include <linux/resource.h>
#include <linux/latencytop.h> #include <linux/latencytop.h>
#include <linux/sched/prio.h> #include <linux/sched/prio.h>
...@@ -356,12 +357,6 @@ struct util_est { ...@@ -356,12 +357,6 @@ struct util_est {
* For cfs_rq, it is the aggregated load_avg of all runnable and * For cfs_rq, it is the aggregated load_avg of all runnable and
* blocked sched_entities. * blocked sched_entities.
* *
* load_avg may also take frequency scaling into account:
*
* load_avg = runnable% * scale_load_down(load) * freq%
*
* where freq% is the CPU frequency normalized to the highest frequency.
*
* [util_avg definition] * [util_avg definition]
* *
* util_avg = running% * SCHED_CAPACITY_SCALE * util_avg = running% * SCHED_CAPACITY_SCALE
...@@ -370,17 +365,14 @@ struct util_est { ...@@ -370,17 +365,14 @@ struct util_est {
* a CPU. For cfs_rq, it is the aggregated util_avg of all runnable * a CPU. For cfs_rq, it is the aggregated util_avg of all runnable
* and blocked sched_entities. * and blocked sched_entities.
* *
* util_avg may also factor frequency scaling and CPU capacity scaling: * load_avg and util_avg don't direcly factor frequency scaling and CPU
* * capacity scaling. The scaling is done through the rq_clock_pelt that
* util_avg = running% * SCHED_CAPACITY_SCALE * freq% * capacity% * is used for computing those signals (see update_rq_clock_pelt())
*
* where freq% is the same as above, and capacity% is the CPU capacity
* normalized to the greatest capacity (due to uarch differences, etc).
* *
* N.B., the above ratios (runnable%, running%, freq%, and capacity%) * N.B., the above ratios (runnable% and running%) themselves are in the
* themselves are in the range of [0, 1]. To do fixed point arithmetics, * range of [0, 1]. To do fixed point arithmetics, we therefore scale them
* we therefore scale them to as large a range as necessary. This is for * to as large a range as necessary. This is for example reflected by
* example reflected by util_avg's SCHED_CAPACITY_SCALE. * util_avg's SCHED_CAPACITY_SCALE.
* *
* [Overflow issue] * [Overflow issue]
* *
...@@ -607,7 +599,7 @@ struct task_struct { ...@@ -607,7 +599,7 @@ struct task_struct {
randomized_struct_fields_start randomized_struct_fields_start
void *stack; void *stack;
atomic_t usage; refcount_t usage;
/* Per task flags (PF_*), defined further below: */ /* Per task flags (PF_*), defined further below: */
unsigned int flags; unsigned int flags;
unsigned int ptrace; unsigned int ptrace;
...@@ -1187,7 +1179,7 @@ struct task_struct { ...@@ -1187,7 +1179,7 @@ struct task_struct {
#endif #endif
#ifdef CONFIG_THREAD_INFO_IN_TASK #ifdef CONFIG_THREAD_INFO_IN_TASK
/* A live task holds one reference: */ /* A live task holds one reference: */
atomic_t stack_refcount; refcount_t stack_refcount;
#endif #endif
#ifdef CONFIG_LIVEPATCH #ifdef CONFIG_LIVEPATCH
int patch_state; int patch_state;
...@@ -1403,7 +1395,6 @@ extern struct pid *cad_pid; ...@@ -1403,7 +1395,6 @@ extern struct pid *cad_pid;
#define PF_UMH 0x02000000 /* I'm an Usermodehelper process */ #define PF_UMH 0x02000000 /* I'm an Usermodehelper process */
#define PF_NO_SETAFFINITY 0x04000000 /* Userland is not allowed to meddle with cpus_allowed */ #define PF_NO_SETAFFINITY 0x04000000 /* Userland is not allowed to meddle with cpus_allowed */
#define PF_MCE_EARLY 0x08000000 /* Early kill for mce process policy */ #define PF_MCE_EARLY 0x08000000 /* Early kill for mce process policy */
#define PF_MUTEX_TESTER 0x20000000 /* Thread belongs to the rt mutex tester */
#define PF_FREEZER_SKIP 0x40000000 /* Freezer should not count it as freezable */ #define PF_FREEZER_SKIP 0x40000000 /* Freezer should not count it as freezable */
#define PF_SUSPEND_TASK 0x80000000 /* This thread called freeze_processes() and should not be frozen */ #define PF_SUSPEND_TASK 0x80000000 /* This thread called freeze_processes() and should not be frozen */
...@@ -1753,9 +1744,9 @@ static __always_inline bool need_resched(void) ...@@ -1753,9 +1744,9 @@ static __always_inline bool need_resched(void)
static inline unsigned int task_cpu(const struct task_struct *p) static inline unsigned int task_cpu(const struct task_struct *p)
{ {
#ifdef CONFIG_THREAD_INFO_IN_TASK #ifdef CONFIG_THREAD_INFO_IN_TASK
return p->cpu; return READ_ONCE(p->cpu);
#else #else
return task_thread_info(p)->cpu; return READ_ONCE(task_thread_info(p)->cpu);
#endif #endif
} }
......
...@@ -8,13 +8,14 @@ ...@@ -8,13 +8,14 @@
#include <linux/sched/jobctl.h> #include <linux/sched/jobctl.h>
#include <linux/sched/task.h> #include <linux/sched/task.h>
#include <linux/cred.h> #include <linux/cred.h>
#include <linux/refcount.h>
/* /*
* Types defining task->signal and task->sighand and APIs using them: * Types defining task->signal and task->sighand and APIs using them:
*/ */
struct sighand_struct { struct sighand_struct {
atomic_t count; refcount_t count;
struct k_sigaction action[_NSIG]; struct k_sigaction action[_NSIG];
spinlock_t siglock; spinlock_t siglock;
wait_queue_head_t signalfd_wqh; wait_queue_head_t signalfd_wqh;
...@@ -82,7 +83,7 @@ struct multiprocess_signals { ...@@ -82,7 +83,7 @@ struct multiprocess_signals {
* the locking of signal_struct. * the locking of signal_struct.
*/ */
struct signal_struct { struct signal_struct {
atomic_t sigcnt; refcount_t sigcnt;
atomic_t live; atomic_t live;
int nr_threads; int nr_threads;
struct list_head thread_head; struct list_head thread_head;
......
...@@ -83,4 +83,11 @@ extern int sysctl_schedstats(struct ctl_table *table, int write, ...@@ -83,4 +83,11 @@ extern int sysctl_schedstats(struct ctl_table *table, int write,
void __user *buffer, size_t *lenp, void __user *buffer, size_t *lenp,
loff_t *ppos); loff_t *ppos);
#if defined(CONFIG_ENERGY_MODEL) && defined(CONFIG_CPU_FREQ_GOV_SCHEDUTIL)
extern unsigned int sysctl_sched_energy_aware;
extern int sched_energy_aware_handler(struct ctl_table *table, int write,
void __user *buffer, size_t *lenp,
loff_t *ppos);
#endif
#endif /* _LINUX_SCHED_SYSCTL_H */ #endif /* _LINUX_SCHED_SYSCTL_H */
...@@ -88,13 +88,13 @@ extern void sched_exec(void); ...@@ -88,13 +88,13 @@ extern void sched_exec(void);
#define sched_exec() {} #define sched_exec() {}
#endif #endif
#define get_task_struct(tsk) do { atomic_inc(&(tsk)->usage); } while(0) #define get_task_struct(tsk) do { refcount_inc(&(tsk)->usage); } while(0)
extern void __put_task_struct(struct task_struct *t); extern void __put_task_struct(struct task_struct *t);
static inline void put_task_struct(struct task_struct *t) static inline void put_task_struct(struct task_struct *t)
{ {
if (atomic_dec_and_test(&t->usage)) if (refcount_dec_and_test(&t->usage))
__put_task_struct(t); __put_task_struct(t);
} }
......
...@@ -61,7 +61,7 @@ static inline unsigned long *end_of_stack(struct task_struct *p) ...@@ -61,7 +61,7 @@ static inline unsigned long *end_of_stack(struct task_struct *p)
#ifdef CONFIG_THREAD_INFO_IN_TASK #ifdef CONFIG_THREAD_INFO_IN_TASK
static inline void *try_get_task_stack(struct task_struct *tsk) static inline void *try_get_task_stack(struct task_struct *tsk)
{ {
return atomic_inc_not_zero(&tsk->stack_refcount) ? return refcount_inc_not_zero(&tsk->stack_refcount) ?
task_stack_page(tsk) : NULL; task_stack_page(tsk) : NULL;
} }
......
...@@ -176,10 +176,10 @@ typedef int (*sched_domain_flags_f)(void); ...@@ -176,10 +176,10 @@ typedef int (*sched_domain_flags_f)(void);
#define SDTL_OVERLAP 0x01 #define SDTL_OVERLAP 0x01
struct sd_data { struct sd_data {
struct sched_domain **__percpu sd; struct sched_domain *__percpu *sd;
struct sched_domain_shared **__percpu sds; struct sched_domain_shared *__percpu *sds;
struct sched_group **__percpu sg; struct sched_group *__percpu *sg;
struct sched_group_capacity **__percpu sgc; struct sched_group_capacity *__percpu *sgc;
}; };
struct sched_domain_topology_level { struct sched_domain_topology_level {
......
...@@ -308,7 +308,7 @@ do { \ ...@@ -308,7 +308,7 @@ do { \
#define __wait_event_freezable(wq_head, condition) \ #define __wait_event_freezable(wq_head, condition) \
___wait_event(wq_head, condition, TASK_INTERRUPTIBLE, 0, 0, \ ___wait_event(wq_head, condition, TASK_INTERRUPTIBLE, 0, 0, \
schedule(); try_to_freeze()) freezable_schedule())
/** /**
* wait_event_freezable - sleep (or freeze) until a condition gets true * wait_event_freezable - sleep (or freeze) until a condition gets true
...@@ -367,7 +367,7 @@ do { \ ...@@ -367,7 +367,7 @@ do { \
#define __wait_event_freezable_timeout(wq_head, condition, timeout) \ #define __wait_event_freezable_timeout(wq_head, condition, timeout) \
___wait_event(wq_head, ___wait_cond_timeout(condition), \ ___wait_event(wq_head, ___wait_cond_timeout(condition), \
TASK_INTERRUPTIBLE, 0, timeout, \ TASK_INTERRUPTIBLE, 0, timeout, \
__ret = schedule_timeout(__ret); try_to_freeze()) __ret = freezable_schedule_timeout(__ret))
/* /*
* like wait_event_timeout() -- except it uses TASK_INTERRUPTIBLE to avoid * like wait_event_timeout() -- except it uses TASK_INTERRUPTIBLE to avoid
...@@ -588,7 +588,7 @@ do { \ ...@@ -588,7 +588,7 @@ do { \
#define __wait_event_freezable_exclusive(wq, condition) \ #define __wait_event_freezable_exclusive(wq, condition) \
___wait_event(wq, condition, TASK_INTERRUPTIBLE, 1, 0, \ ___wait_event(wq, condition, TASK_INTERRUPTIBLE, 1, 0, \
schedule(); try_to_freeze()) freezable_schedule())
#define wait_event_freezable_exclusive(wq, condition) \ #define wait_event_freezable_exclusive(wq, condition) \
({ \ ({ \
......
...@@ -44,7 +44,7 @@ static struct signal_struct init_signals = { ...@@ -44,7 +44,7 @@ static struct signal_struct init_signals = {
}; };
static struct sighand_struct init_sighand = { static struct sighand_struct init_sighand = {
.count = ATOMIC_INIT(1), .count = REFCOUNT_INIT(1),
.action = { { { .sa_handler = SIG_DFL, } }, }, .action = { { { .sa_handler = SIG_DFL, } }, },
.siglock = __SPIN_LOCK_UNLOCKED(init_sighand.siglock), .siglock = __SPIN_LOCK_UNLOCKED(init_sighand.siglock),
.signalfd_wqh = __WAIT_QUEUE_HEAD_INITIALIZER(init_sighand.signalfd_wqh), .signalfd_wqh = __WAIT_QUEUE_HEAD_INITIALIZER(init_sighand.signalfd_wqh),
...@@ -61,11 +61,11 @@ struct task_struct init_task ...@@ -61,11 +61,11 @@ struct task_struct init_task
= { = {
#ifdef CONFIG_THREAD_INFO_IN_TASK #ifdef CONFIG_THREAD_INFO_IN_TASK
.thread_info = INIT_THREAD_INFO(init_task), .thread_info = INIT_THREAD_INFO(init_task),
.stack_refcount = ATOMIC_INIT(1), .stack_refcount = REFCOUNT_INIT(1),
#endif #endif
.state = 0, .state = 0,
.stack = init_stack, .stack = init_stack,
.usage = ATOMIC_INIT(2), .usage = REFCOUNT_INIT(2),
.flags = PF_KTHREAD, .flags = PF_KTHREAD,
.prio = MAX_PRIO - 20, .prio = MAX_PRIO - 20,
.static_prio = MAX_PRIO - 20, .static_prio = MAX_PRIO - 20,
......
...@@ -429,7 +429,7 @@ static void release_task_stack(struct task_struct *tsk) ...@@ -429,7 +429,7 @@ static void release_task_stack(struct task_struct *tsk)
#ifdef CONFIG_THREAD_INFO_IN_TASK #ifdef CONFIG_THREAD_INFO_IN_TASK
void put_task_stack(struct task_struct *tsk) void put_task_stack(struct task_struct *tsk)
{ {
if (atomic_dec_and_test(&tsk->stack_refcount)) if (refcount_dec_and_test(&tsk->stack_refcount))
release_task_stack(tsk); release_task_stack(tsk);
} }
#endif #endif
...@@ -447,7 +447,7 @@ void free_task(struct task_struct *tsk) ...@@ -447,7 +447,7 @@ void free_task(struct task_struct *tsk)
* If the task had a separate stack allocation, it should be gone * If the task had a separate stack allocation, it should be gone
* by now. * by now.
*/ */
WARN_ON_ONCE(atomic_read(&tsk->stack_refcount) != 0); WARN_ON_ONCE(refcount_read(&tsk->stack_refcount) != 0);
#endif #endif
rt_mutex_debug_task_free(tsk); rt_mutex_debug_task_free(tsk);
ftrace_graph_exit_task(tsk); ftrace_graph_exit_task(tsk);
...@@ -710,14 +710,14 @@ static inline void free_signal_struct(struct signal_struct *sig) ...@@ -710,14 +710,14 @@ static inline void free_signal_struct(struct signal_struct *sig)
static inline void put_signal_struct(struct signal_struct *sig) static inline void put_signal_struct(struct signal_struct *sig)
{ {
if (atomic_dec_and_test(&sig->sigcnt)) if (refcount_dec_and_test(&sig->sigcnt))
free_signal_struct(sig); free_signal_struct(sig);
} }
void __put_task_struct(struct task_struct *tsk) void __put_task_struct(struct task_struct *tsk)
{ {
WARN_ON(!tsk->exit_state); WARN_ON(!tsk->exit_state);
WARN_ON(atomic_read(&tsk->usage)); WARN_ON(refcount_read(&tsk->usage));
WARN_ON(tsk == current); WARN_ON(tsk == current);
cgroup_free(tsk); cgroup_free(tsk);
...@@ -867,7 +867,7 @@ static struct task_struct *dup_task_struct(struct task_struct *orig, int node) ...@@ -867,7 +867,7 @@ static struct task_struct *dup_task_struct(struct task_struct *orig, int node)
tsk->stack_vm_area = stack_vm_area; tsk->stack_vm_area = stack_vm_area;
#endif #endif
#ifdef CONFIG_THREAD_INFO_IN_TASK #ifdef CONFIG_THREAD_INFO_IN_TASK
atomic_set(&tsk->stack_refcount, 1); refcount_set(&tsk->stack_refcount, 1);
#endif #endif
if (err) if (err)
...@@ -896,7 +896,7 @@ static struct task_struct *dup_task_struct(struct task_struct *orig, int node) ...@@ -896,7 +896,7 @@ static struct task_struct *dup_task_struct(struct task_struct *orig, int node)
* One for us, one for whoever does the "release_task()" (usually * One for us, one for whoever does the "release_task()" (usually
* parent) * parent)
*/ */
atomic_set(&tsk->usage, 2); refcount_set(&tsk->usage, 2);
#ifdef CONFIG_BLK_DEV_IO_TRACE #ifdef CONFIG_BLK_DEV_IO_TRACE
tsk->btrace_seq = 0; tsk->btrace_seq = 0;
#endif #endif
...@@ -1463,7 +1463,7 @@ static int copy_sighand(unsigned long clone_flags, struct task_struct *tsk) ...@@ -1463,7 +1463,7 @@ static int copy_sighand(unsigned long clone_flags, struct task_struct *tsk)
struct sighand_struct *sig; struct sighand_struct *sig;
if (clone_flags & CLONE_SIGHAND) { if (clone_flags & CLONE_SIGHAND) {
atomic_inc(&current->sighand->count); refcount_inc(&current->sighand->count);
return 0; return 0;
} }
sig = kmem_cache_alloc(sighand_cachep, GFP_KERNEL); sig = kmem_cache_alloc(sighand_cachep, GFP_KERNEL);
...@@ -1471,7 +1471,7 @@ static int copy_sighand(unsigned long clone_flags, struct task_struct *tsk) ...@@ -1471,7 +1471,7 @@ static int copy_sighand(unsigned long clone_flags, struct task_struct *tsk)
if (!sig) if (!sig)
return -ENOMEM; return -ENOMEM;
atomic_set(&sig->count, 1); refcount_set(&sig->count, 1);
spin_lock_irq(&current->sighand->siglock); spin_lock_irq(&current->sighand->siglock);
memcpy(sig->action, current->sighand->action, sizeof(sig->action)); memcpy(sig->action, current->sighand->action, sizeof(sig->action));
spin_unlock_irq(&current->sighand->siglock); spin_unlock_irq(&current->sighand->siglock);
...@@ -1480,7 +1480,7 @@ static int copy_sighand(unsigned long clone_flags, struct task_struct *tsk) ...@@ -1480,7 +1480,7 @@ static int copy_sighand(unsigned long clone_flags, struct task_struct *tsk)
void __cleanup_sighand(struct sighand_struct *sighand) void __cleanup_sighand(struct sighand_struct *sighand)
{ {
if (atomic_dec_and_test(&sighand->count)) { if (refcount_dec_and_test(&sighand->count)) {
signalfd_cleanup(sighand); signalfd_cleanup(sighand);
/* /*
* sighand_cachep is SLAB_TYPESAFE_BY_RCU so we can free it * sighand_cachep is SLAB_TYPESAFE_BY_RCU so we can free it
...@@ -1527,7 +1527,7 @@ static int copy_signal(unsigned long clone_flags, struct task_struct *tsk) ...@@ -1527,7 +1527,7 @@ static int copy_signal(unsigned long clone_flags, struct task_struct *tsk)
sig->nr_threads = 1; sig->nr_threads = 1;
atomic_set(&sig->live, 1); atomic_set(&sig->live, 1);
atomic_set(&sig->sigcnt, 1); refcount_set(&sig->sigcnt, 1);
/* list_add(thread_node, thread_head) without INIT_LIST_HEAD() */ /* list_add(thread_node, thread_head) without INIT_LIST_HEAD() */
sig->thread_head = (struct list_head)LIST_HEAD_INIT(tsk->thread_node); sig->thread_head = (struct list_head)LIST_HEAD_INIT(tsk->thread_node);
...@@ -2082,7 +2082,7 @@ static __latent_entropy struct task_struct *copy_process( ...@@ -2082,7 +2082,7 @@ static __latent_entropy struct task_struct *copy_process(
} else { } else {
current->signal->nr_threads++; current->signal->nr_threads++;
atomic_inc(&current->signal->live); atomic_inc(&current->signal->live);
atomic_inc(&current->signal->sigcnt); refcount_inc(&current->signal->sigcnt);
task_join_group_stop(p); task_join_group_stop(p);
list_add_tail_rcu(&p->thread_group, list_add_tail_rcu(&p->thread_group,
&p->group_leader->thread_group); &p->group_leader->thread_group);
...@@ -2439,7 +2439,7 @@ static int check_unshare_flags(unsigned long unshare_flags) ...@@ -2439,7 +2439,7 @@ static int check_unshare_flags(unsigned long unshare_flags)
return -EINVAL; return -EINVAL;
} }
if (unshare_flags & (CLONE_SIGHAND | CLONE_VM)) { if (unshare_flags & (CLONE_SIGHAND | CLONE_VM)) {
if (atomic_read(&current->sighand->count) > 1) if (refcount_read(&current->sighand->count) > 1)
return -EINVAL; return -EINVAL;
} }
if (unshare_flags & CLONE_VM) { if (unshare_flags & CLONE_VM) {
......
...@@ -605,7 +605,7 @@ void __kthread_init_worker(struct kthread_worker *worker, ...@@ -605,7 +605,7 @@ void __kthread_init_worker(struct kthread_worker *worker,
struct lock_class_key *key) struct lock_class_key *key)
{ {
memset(worker, 0, sizeof(struct kthread_worker)); memset(worker, 0, sizeof(struct kthread_worker));
spin_lock_init(&worker->lock); raw_spin_lock_init(&worker->lock);
lockdep_set_class_and_name(&worker->lock, key, name); lockdep_set_class_and_name(&worker->lock, key, name);
INIT_LIST_HEAD(&worker->work_list); INIT_LIST_HEAD(&worker->work_list);
INIT_LIST_HEAD(&worker->delayed_work_list); INIT_LIST_HEAD(&worker->delayed_work_list);
...@@ -647,21 +647,21 @@ int kthread_worker_fn(void *worker_ptr) ...@@ -647,21 +647,21 @@ int kthread_worker_fn(void *worker_ptr)
if (kthread_should_stop()) { if (kthread_should_stop()) {
__set_current_state(TASK_RUNNING); __set_current_state(TASK_RUNNING);
spin_lock_irq(&worker->lock); raw_spin_lock_irq(&worker->lock);
worker->task = NULL; worker->task = NULL;
spin_unlock_irq(&worker->lock); raw_spin_unlock_irq(&worker->lock);
return 0; return 0;
} }
work = NULL; work = NULL;
spin_lock_irq(&worker->lock); raw_spin_lock_irq(&worker->lock);
if (!list_empty(&worker->work_list)) { if (!list_empty(&worker->work_list)) {
work = list_first_entry(&worker->work_list, work = list_first_entry(&worker->work_list,
struct kthread_work, node); struct kthread_work, node);
list_del_init(&work->node); list_del_init(&work->node);
} }
worker->current_work = work; worker->current_work = work;
spin_unlock_irq(&worker->lock); raw_spin_unlock_irq(&worker->lock);
if (work) { if (work) {
__set_current_state(TASK_RUNNING); __set_current_state(TASK_RUNNING);
...@@ -818,12 +818,12 @@ bool kthread_queue_work(struct kthread_worker *worker, ...@@ -818,12 +818,12 @@ bool kthread_queue_work(struct kthread_worker *worker,
bool ret = false; bool ret = false;
unsigned long flags; unsigned long flags;
spin_lock_irqsave(&worker->lock, flags); raw_spin_lock_irqsave(&worker->lock, flags);
if (!queuing_blocked(worker, work)) { if (!queuing_blocked(worker, work)) {
kthread_insert_work(worker, work, &worker->work_list); kthread_insert_work(worker, work, &worker->work_list);
ret = true; ret = true;
} }
spin_unlock_irqrestore(&worker->lock, flags); raw_spin_unlock_irqrestore(&worker->lock, flags);
return ret; return ret;
} }
EXPORT_SYMBOL_GPL(kthread_queue_work); EXPORT_SYMBOL_GPL(kthread_queue_work);
...@@ -841,6 +841,7 @@ void kthread_delayed_work_timer_fn(struct timer_list *t) ...@@ -841,6 +841,7 @@ void kthread_delayed_work_timer_fn(struct timer_list *t)
struct kthread_delayed_work *dwork = from_timer(dwork, t, timer); struct kthread_delayed_work *dwork = from_timer(dwork, t, timer);
struct kthread_work *work = &dwork->work; struct kthread_work *work = &dwork->work;
struct kthread_worker *worker = work->worker; struct kthread_worker *worker = work->worker;
unsigned long flags;
/* /*
* This might happen when a pending work is reinitialized. * This might happen when a pending work is reinitialized.
...@@ -849,7 +850,7 @@ void kthread_delayed_work_timer_fn(struct timer_list *t) ...@@ -849,7 +850,7 @@ void kthread_delayed_work_timer_fn(struct timer_list *t)
if (WARN_ON_ONCE(!worker)) if (WARN_ON_ONCE(!worker))
return; return;
spin_lock(&worker->lock); raw_spin_lock_irqsave(&worker->lock, flags);
/* Work must not be used with >1 worker, see kthread_queue_work(). */ /* Work must not be used with >1 worker, see kthread_queue_work(). */
WARN_ON_ONCE(work->worker != worker); WARN_ON_ONCE(work->worker != worker);
...@@ -858,7 +859,7 @@ void kthread_delayed_work_timer_fn(struct timer_list *t) ...@@ -858,7 +859,7 @@ void kthread_delayed_work_timer_fn(struct timer_list *t)
list_del_init(&work->node); list_del_init(&work->node);
kthread_insert_work(worker, work, &worker->work_list); kthread_insert_work(worker, work, &worker->work_list);
spin_unlock(&worker->lock); raw_spin_unlock_irqrestore(&worker->lock, flags);
} }
EXPORT_SYMBOL(kthread_delayed_work_timer_fn); EXPORT_SYMBOL(kthread_delayed_work_timer_fn);
...@@ -914,14 +915,14 @@ bool kthread_queue_delayed_work(struct kthread_worker *worker, ...@@ -914,14 +915,14 @@ bool kthread_queue_delayed_work(struct kthread_worker *worker,
unsigned long flags; unsigned long flags;
bool ret = false; bool ret = false;
spin_lock_irqsave(&worker->lock, flags); raw_spin_lock_irqsave(&worker->lock, flags);
if (!queuing_blocked(worker, work)) { if (!queuing_blocked(worker, work)) {
__kthread_queue_delayed_work(worker, dwork, delay); __kthread_queue_delayed_work(worker, dwork, delay);
ret = true; ret = true;
} }
spin_unlock_irqrestore(&worker->lock, flags); raw_spin_unlock_irqrestore(&worker->lock, flags);
return ret; return ret;
} }
EXPORT_SYMBOL_GPL(kthread_queue_delayed_work); EXPORT_SYMBOL_GPL(kthread_queue_delayed_work);
...@@ -957,7 +958,7 @@ void kthread_flush_work(struct kthread_work *work) ...@@ -957,7 +958,7 @@ void kthread_flush_work(struct kthread_work *work)
if (!worker) if (!worker)
return; return;
spin_lock_irq(&worker->lock); raw_spin_lock_irq(&worker->lock);
/* Work must not be used with >1 worker, see kthread_queue_work(). */ /* Work must not be used with >1 worker, see kthread_queue_work(). */
WARN_ON_ONCE(work->worker != worker); WARN_ON_ONCE(work->worker != worker);
...@@ -969,7 +970,7 @@ void kthread_flush_work(struct kthread_work *work) ...@@ -969,7 +970,7 @@ void kthread_flush_work(struct kthread_work *work)
else else
noop = true; noop = true;
spin_unlock_irq(&worker->lock); raw_spin_unlock_irq(&worker->lock);
if (!noop) if (!noop)
wait_for_completion(&fwork.done); wait_for_completion(&fwork.done);
...@@ -1002,9 +1003,9 @@ static bool __kthread_cancel_work(struct kthread_work *work, bool is_dwork, ...@@ -1002,9 +1003,9 @@ static bool __kthread_cancel_work(struct kthread_work *work, bool is_dwork,
* any queuing is blocked by setting the canceling counter. * any queuing is blocked by setting the canceling counter.
*/ */
work->canceling++; work->canceling++;
spin_unlock_irqrestore(&worker->lock, *flags); raw_spin_unlock_irqrestore(&worker->lock, *flags);
del_timer_sync(&dwork->timer); del_timer_sync(&dwork->timer);
spin_lock_irqsave(&worker->lock, *flags); raw_spin_lock_irqsave(&worker->lock, *flags);
work->canceling--; work->canceling--;
} }
...@@ -1051,7 +1052,7 @@ bool kthread_mod_delayed_work(struct kthread_worker *worker, ...@@ -1051,7 +1052,7 @@ bool kthread_mod_delayed_work(struct kthread_worker *worker,
unsigned long flags; unsigned long flags;
int ret = false; int ret = false;
spin_lock_irqsave(&worker->lock, flags); raw_spin_lock_irqsave(&worker->lock, flags);
/* Do not bother with canceling when never queued. */ /* Do not bother with canceling when never queued. */
if (!work->worker) if (!work->worker)
...@@ -1068,7 +1069,7 @@ bool kthread_mod_delayed_work(struct kthread_worker *worker, ...@@ -1068,7 +1069,7 @@ bool kthread_mod_delayed_work(struct kthread_worker *worker,
fast_queue: fast_queue:
__kthread_queue_delayed_work(worker, dwork, delay); __kthread_queue_delayed_work(worker, dwork, delay);
out: out:
spin_unlock_irqrestore(&worker->lock, flags); raw_spin_unlock_irqrestore(&worker->lock, flags);
return ret; return ret;
} }
EXPORT_SYMBOL_GPL(kthread_mod_delayed_work); EXPORT_SYMBOL_GPL(kthread_mod_delayed_work);
...@@ -1082,7 +1083,7 @@ static bool __kthread_cancel_work_sync(struct kthread_work *work, bool is_dwork) ...@@ -1082,7 +1083,7 @@ static bool __kthread_cancel_work_sync(struct kthread_work *work, bool is_dwork)
if (!worker) if (!worker)
goto out; goto out;
spin_lock_irqsave(&worker->lock, flags); raw_spin_lock_irqsave(&worker->lock, flags);
/* Work must not be used with >1 worker, see kthread_queue_work(). */ /* Work must not be used with >1 worker, see kthread_queue_work(). */
WARN_ON_ONCE(work->worker != worker); WARN_ON_ONCE(work->worker != worker);
...@@ -1096,13 +1097,13 @@ static bool __kthread_cancel_work_sync(struct kthread_work *work, bool is_dwork) ...@@ -1096,13 +1097,13 @@ static bool __kthread_cancel_work_sync(struct kthread_work *work, bool is_dwork)
* In the meantime, block any queuing by setting the canceling counter. * In the meantime, block any queuing by setting the canceling counter.
*/ */
work->canceling++; work->canceling++;
spin_unlock_irqrestore(&worker->lock, flags); raw_spin_unlock_irqrestore(&worker->lock, flags);
kthread_flush_work(work); kthread_flush_work(work);
spin_lock_irqsave(&worker->lock, flags); raw_spin_lock_irqsave(&worker->lock, flags);
work->canceling--; work->canceling--;
out_fast: out_fast:
spin_unlock_irqrestore(&worker->lock, flags); raw_spin_unlock_irqrestore(&worker->lock, flags);
out: out:
return ret; return ret;
} }
......
...@@ -107,11 +107,12 @@ struct rq *task_rq_lock(struct task_struct *p, struct rq_flags *rf) ...@@ -107,11 +107,12 @@ struct rq *task_rq_lock(struct task_struct *p, struct rq_flags *rf)
* [L] ->on_rq * [L] ->on_rq
* RELEASE (rq->lock) * RELEASE (rq->lock)
* *
* If we observe the old CPU in task_rq_lock, the acquire of * If we observe the old CPU in task_rq_lock(), the acquire of
* the old rq->lock will fully serialize against the stores. * the old rq->lock will fully serialize against the stores.
* *
* If we observe the new CPU in task_rq_lock, the acquire will * If we observe the new CPU in task_rq_lock(), the address
* pair with the WMB to ensure we must then also see migrating. * dependency headed by '[L] rq = task_rq()' and the acquire
* will pair with the WMB to ensure we then also see migrating.
*/ */
if (likely(rq == task_rq(p) && !task_on_rq_migrating(p))) { if (likely(rq == task_rq(p) && !task_on_rq_migrating(p))) {
rq_pin_lock(rq, rf); rq_pin_lock(rq, rf);
...@@ -180,6 +181,7 @@ static void update_rq_clock_task(struct rq *rq, s64 delta) ...@@ -180,6 +181,7 @@ static void update_rq_clock_task(struct rq *rq, s64 delta)
if ((irq_delta + steal) && sched_feat(NONTASK_CAPACITY)) if ((irq_delta + steal) && sched_feat(NONTASK_CAPACITY))
update_irq_load_avg(rq, irq_delta + steal); update_irq_load_avg(rq, irq_delta + steal);
#endif #endif
update_rq_clock_pelt(rq, delta);
} }
void update_rq_clock(struct rq *rq) void update_rq_clock(struct rq *rq)
...@@ -956,7 +958,7 @@ static struct rq *move_queued_task(struct rq *rq, struct rq_flags *rf, ...@@ -956,7 +958,7 @@ static struct rq *move_queued_task(struct rq *rq, struct rq_flags *rf,
{ {
lockdep_assert_held(&rq->lock); lockdep_assert_held(&rq->lock);
p->on_rq = TASK_ON_RQ_MIGRATING; WRITE_ONCE(p->on_rq, TASK_ON_RQ_MIGRATING);
dequeue_task(rq, p, DEQUEUE_NOCLOCK); dequeue_task(rq, p, DEQUEUE_NOCLOCK);
set_task_cpu(p, new_cpu); set_task_cpu(p, new_cpu);
rq_unlock(rq, rf); rq_unlock(rq, rf);
...@@ -2459,7 +2461,7 @@ void wake_up_new_task(struct task_struct *p) ...@@ -2459,7 +2461,7 @@ void wake_up_new_task(struct task_struct *p)
#endif #endif
rq = __task_rq_lock(p, &rf); rq = __task_rq_lock(p, &rf);
update_rq_clock(rq); update_rq_clock(rq);
post_init_entity_util_avg(&p->se); post_init_entity_util_avg(p);
activate_task(rq, p, ENQUEUE_NOCLOCK); activate_task(rq, p, ENQUEUE_NOCLOCK);
p->on_rq = TASK_ON_RQ_QUEUED; p->on_rq = TASK_ON_RQ_QUEUED;
......
...@@ -1767,7 +1767,7 @@ pick_next_task_dl(struct rq *rq, struct task_struct *prev, struct rq_flags *rf) ...@@ -1767,7 +1767,7 @@ pick_next_task_dl(struct rq *rq, struct task_struct *prev, struct rq_flags *rf)
deadline_queue_push_tasks(rq); deadline_queue_push_tasks(rq);
if (rq->curr->sched_class != &dl_sched_class) if (rq->curr->sched_class != &dl_sched_class)
update_dl_rq_load_avg(rq_clock_task(rq), rq, 0); update_dl_rq_load_avg(rq_clock_pelt(rq), rq, 0);
return p; return p;
} }
...@@ -1776,7 +1776,7 @@ static void put_prev_task_dl(struct rq *rq, struct task_struct *p) ...@@ -1776,7 +1776,7 @@ static void put_prev_task_dl(struct rq *rq, struct task_struct *p)
{ {
update_curr_dl(rq); update_curr_dl(rq);
update_dl_rq_load_avg(rq_clock_task(rq), rq, 1); update_dl_rq_load_avg(rq_clock_pelt(rq), rq, 1);
if (on_dl_rq(&p->dl) && p->nr_cpus_allowed > 1) if (on_dl_rq(&p->dl) && p->nr_cpus_allowed > 1)
enqueue_pushable_dl_task(rq, p); enqueue_pushable_dl_task(rq, p);
} }
...@@ -1793,7 +1793,7 @@ static void task_tick_dl(struct rq *rq, struct task_struct *p, int queued) ...@@ -1793,7 +1793,7 @@ static void task_tick_dl(struct rq *rq, struct task_struct *p, int queued)
{ {
update_curr_dl(rq); update_curr_dl(rq);
update_dl_rq_load_avg(rq_clock_task(rq), rq, 1); update_dl_rq_load_avg(rq_clock_pelt(rq), rq, 1);
/* /*
* Even when we have runtime, update_curr_dl() might have resulted in us * Even when we have runtime, update_curr_dl() might have resulted in us
* not being the leftmost task anymore. In that case NEED_RESCHED will * not being the leftmost task anymore. In that case NEED_RESCHED will
......
...@@ -315,6 +315,7 @@ void register_sched_domain_sysctl(void) ...@@ -315,6 +315,7 @@ void register_sched_domain_sysctl(void)
{ {
static struct ctl_table *cpu_entries; static struct ctl_table *cpu_entries;
static struct ctl_table **cpu_idx; static struct ctl_table **cpu_idx;
static bool init_done = false;
char buf[32]; char buf[32];
int i; int i;
...@@ -344,7 +345,10 @@ void register_sched_domain_sysctl(void) ...@@ -344,7 +345,10 @@ void register_sched_domain_sysctl(void)
if (!cpumask_available(sd_sysctl_cpus)) { if (!cpumask_available(sd_sysctl_cpus)) {
if (!alloc_cpumask_var(&sd_sysctl_cpus, GFP_KERNEL)) if (!alloc_cpumask_var(&sd_sysctl_cpus, GFP_KERNEL))
return; return;
}
if (!init_done) {
init_done = true;
/* init to possible to not have holes in @cpu_entries */ /* init to possible to not have holes in @cpu_entries */
cpumask_copy(sd_sysctl_cpus, cpu_possible_mask); cpumask_copy(sd_sysctl_cpus, cpu_possible_mask);
} }
......
This diff is collapsed.
...@@ -80,7 +80,7 @@ static int __init housekeeping_setup(char *str, enum hk_flags flags) ...@@ -80,7 +80,7 @@ static int __init housekeeping_setup(char *str, enum hk_flags flags)
cpumask_andnot(housekeeping_mask, cpumask_andnot(housekeeping_mask,
cpu_possible_mask, non_housekeeping_mask); cpu_possible_mask, non_housekeeping_mask);
if (cpumask_empty(housekeeping_mask)) if (cpumask_empty(housekeeping_mask))
cpumask_set_cpu(smp_processor_id(), housekeeping_mask); __cpumask_set_cpu(smp_processor_id(), housekeeping_mask);
} else { } else {
cpumask_var_t tmp; cpumask_var_t tmp;
......
...@@ -26,7 +26,6 @@ ...@@ -26,7 +26,6 @@
#include <linux/sched.h> #include <linux/sched.h>
#include "sched.h" #include "sched.h"
#include "sched-pelt.h"
#include "pelt.h" #include "pelt.h"
/* /*
...@@ -106,16 +105,12 @@ static u32 __accumulate_pelt_segments(u64 periods, u32 d1, u32 d3) ...@@ -106,16 +105,12 @@ static u32 __accumulate_pelt_segments(u64 periods, u32 d1, u32 d3)
* n=1 * n=1
*/ */
static __always_inline u32 static __always_inline u32
accumulate_sum(u64 delta, int cpu, struct sched_avg *sa, accumulate_sum(u64 delta, struct sched_avg *sa,
unsigned long load, unsigned long runnable, int running) unsigned long load, unsigned long runnable, int running)
{ {
unsigned long scale_freq, scale_cpu;
u32 contrib = (u32)delta; /* p == 0 -> delta < 1024 */ u32 contrib = (u32)delta; /* p == 0 -> delta < 1024 */
u64 periods; u64 periods;
scale_freq = arch_scale_freq_capacity(cpu);
scale_cpu = arch_scale_cpu_capacity(NULL, cpu);
delta += sa->period_contrib; delta += sa->period_contrib;
periods = delta / 1024; /* A period is 1024us (~1ms) */ periods = delta / 1024; /* A period is 1024us (~1ms) */
...@@ -137,13 +132,12 @@ accumulate_sum(u64 delta, int cpu, struct sched_avg *sa, ...@@ -137,13 +132,12 @@ accumulate_sum(u64 delta, int cpu, struct sched_avg *sa,
} }
sa->period_contrib = delta; sa->period_contrib = delta;
contrib = cap_scale(contrib, scale_freq);
if (load) if (load)
sa->load_sum += load * contrib; sa->load_sum += load * contrib;
if (runnable) if (runnable)
sa->runnable_load_sum += runnable * contrib; sa->runnable_load_sum += runnable * contrib;
if (running) if (running)
sa->util_sum += contrib * scale_cpu; sa->util_sum += contrib << SCHED_CAPACITY_SHIFT;
return periods; return periods;
} }
...@@ -177,7 +171,7 @@ accumulate_sum(u64 delta, int cpu, struct sched_avg *sa, ...@@ -177,7 +171,7 @@ accumulate_sum(u64 delta, int cpu, struct sched_avg *sa,
* = u_0 + u_1*y + u_2*y^2 + ... [re-labeling u_i --> u_{i+1}] * = u_0 + u_1*y + u_2*y^2 + ... [re-labeling u_i --> u_{i+1}]
*/ */
static __always_inline int static __always_inline int
___update_load_sum(u64 now, int cpu, struct sched_avg *sa, ___update_load_sum(u64 now, struct sched_avg *sa,
unsigned long load, unsigned long runnable, int running) unsigned long load, unsigned long runnable, int running)
{ {
u64 delta; u64 delta;
...@@ -221,7 +215,7 @@ ___update_load_sum(u64 now, int cpu, struct sched_avg *sa, ...@@ -221,7 +215,7 @@ ___update_load_sum(u64 now, int cpu, struct sched_avg *sa,
* Step 1: accumulate *_sum since last_update_time. If we haven't * Step 1: accumulate *_sum since last_update_time. If we haven't
* crossed period boundaries, finish. * crossed period boundaries, finish.
*/ */
if (!accumulate_sum(delta, cpu, sa, load, runnable, running)) if (!accumulate_sum(delta, sa, load, runnable, running))
return 0; return 0;
return 1; return 1;
...@@ -267,9 +261,9 @@ ___update_load_avg(struct sched_avg *sa, unsigned long load, unsigned long runna ...@@ -267,9 +261,9 @@ ___update_load_avg(struct sched_avg *sa, unsigned long load, unsigned long runna
* runnable_load_avg = \Sum se->avg.runable_load_avg * runnable_load_avg = \Sum se->avg.runable_load_avg
*/ */
int __update_load_avg_blocked_se(u64 now, int cpu, struct sched_entity *se) int __update_load_avg_blocked_se(u64 now, struct sched_entity *se)
{ {
if (___update_load_sum(now, cpu, &se->avg, 0, 0, 0)) { if (___update_load_sum(now, &se->avg, 0, 0, 0)) {
___update_load_avg(&se->avg, se_weight(se), se_runnable(se)); ___update_load_avg(&se->avg, se_weight(se), se_runnable(se));
return 1; return 1;
} }
...@@ -277,9 +271,9 @@ int __update_load_avg_blocked_se(u64 now, int cpu, struct sched_entity *se) ...@@ -277,9 +271,9 @@ int __update_load_avg_blocked_se(u64 now, int cpu, struct sched_entity *se)
return 0; return 0;
} }
int __update_load_avg_se(u64 now, int cpu, struct cfs_rq *cfs_rq, struct sched_entity *se) int __update_load_avg_se(u64 now, struct cfs_rq *cfs_rq, struct sched_entity *se)
{ {
if (___update_load_sum(now, cpu, &se->avg, !!se->on_rq, !!se->on_rq, if (___update_load_sum(now, &se->avg, !!se->on_rq, !!se->on_rq,
cfs_rq->curr == se)) { cfs_rq->curr == se)) {
___update_load_avg(&se->avg, se_weight(se), se_runnable(se)); ___update_load_avg(&se->avg, se_weight(se), se_runnable(se));
...@@ -290,9 +284,9 @@ int __update_load_avg_se(u64 now, int cpu, struct cfs_rq *cfs_rq, struct sched_e ...@@ -290,9 +284,9 @@ int __update_load_avg_se(u64 now, int cpu, struct cfs_rq *cfs_rq, struct sched_e
return 0; return 0;
} }
int __update_load_avg_cfs_rq(u64 now, int cpu, struct cfs_rq *cfs_rq) int __update_load_avg_cfs_rq(u64 now, struct cfs_rq *cfs_rq)
{ {
if (___update_load_sum(now, cpu, &cfs_rq->avg, if (___update_load_sum(now, &cfs_rq->avg,
scale_load_down(cfs_rq->load.weight), scale_load_down(cfs_rq->load.weight),
scale_load_down(cfs_rq->runnable_weight), scale_load_down(cfs_rq->runnable_weight),
cfs_rq->curr != NULL)) { cfs_rq->curr != NULL)) {
...@@ -317,7 +311,7 @@ int __update_load_avg_cfs_rq(u64 now, int cpu, struct cfs_rq *cfs_rq) ...@@ -317,7 +311,7 @@ int __update_load_avg_cfs_rq(u64 now, int cpu, struct cfs_rq *cfs_rq)
int update_rt_rq_load_avg(u64 now, struct rq *rq, int running) int update_rt_rq_load_avg(u64 now, struct rq *rq, int running)
{ {
if (___update_load_sum(now, rq->cpu, &rq->avg_rt, if (___update_load_sum(now, &rq->avg_rt,
running, running,
running, running,
running)) { running)) {
...@@ -340,7 +334,7 @@ int update_rt_rq_load_avg(u64 now, struct rq *rq, int running) ...@@ -340,7 +334,7 @@ int update_rt_rq_load_avg(u64 now, struct rq *rq, int running)
int update_dl_rq_load_avg(u64 now, struct rq *rq, int running) int update_dl_rq_load_avg(u64 now, struct rq *rq, int running)
{ {
if (___update_load_sum(now, rq->cpu, &rq->avg_dl, if (___update_load_sum(now, &rq->avg_dl,
running, running,
running, running,
running)) { running)) {
...@@ -365,22 +359,31 @@ int update_dl_rq_load_avg(u64 now, struct rq *rq, int running) ...@@ -365,22 +359,31 @@ int update_dl_rq_load_avg(u64 now, struct rq *rq, int running)
int update_irq_load_avg(struct rq *rq, u64 running) int update_irq_load_avg(struct rq *rq, u64 running)
{ {
int ret = 0; int ret = 0;
/*
* We can't use clock_pelt because irq time is not accounted in
* clock_task. Instead we directly scale the running time to
* reflect the real amount of computation
*/
running = cap_scale(running, arch_scale_freq_capacity(cpu_of(rq)));
running = cap_scale(running, arch_scale_cpu_capacity(NULL, cpu_of(rq)));
/* /*
* We know the time that has been used by interrupt since last update * We know the time that has been used by interrupt since last update
* but we don't when. Let be pessimistic and assume that interrupt has * but we don't when. Let be pessimistic and assume that interrupt has
* happened just before the update. This is not so far from reality * happened just before the update. This is not so far from reality
* because interrupt will most probably wake up task and trig an update * because interrupt will most probably wake up task and trig an update
* of rq clock during which the metric si updated. * of rq clock during which the metric is updated.
* We start to decay with normal context time and then we add the * We start to decay with normal context time and then we add the
* interrupt context time. * interrupt context time.
* We can safely remove running from rq->clock because * We can safely remove running from rq->clock because
* rq->clock += delta with delta >= running * rq->clock += delta with delta >= running
*/ */
ret = ___update_load_sum(rq->clock - running, rq->cpu, &rq->avg_irq, ret = ___update_load_sum(rq->clock - running, &rq->avg_irq,
0, 0,
0, 0,
0); 0);
ret += ___update_load_sum(rq->clock, rq->cpu, &rq->avg_irq, ret += ___update_load_sum(rq->clock, &rq->avg_irq,
1, 1,
1, 1,
1); 1);
......
#ifdef CONFIG_SMP #ifdef CONFIG_SMP
#include "sched-pelt.h"
int __update_load_avg_blocked_se(u64 now, int cpu, struct sched_entity *se); int __update_load_avg_blocked_se(u64 now, struct sched_entity *se);
int __update_load_avg_se(u64 now, int cpu, struct cfs_rq *cfs_rq, struct sched_entity *se); int __update_load_avg_se(u64 now, struct cfs_rq *cfs_rq, struct sched_entity *se);
int __update_load_avg_cfs_rq(u64 now, int cpu, struct cfs_rq *cfs_rq); int __update_load_avg_cfs_rq(u64 now, struct cfs_rq *cfs_rq);
int update_rt_rq_load_avg(u64 now, struct rq *rq, int running); int update_rt_rq_load_avg(u64 now, struct rq *rq, int running);
int update_dl_rq_load_avg(u64 now, struct rq *rq, int running); int update_dl_rq_load_avg(u64 now, struct rq *rq, int running);
...@@ -42,6 +43,101 @@ static inline void cfs_se_util_change(struct sched_avg *avg) ...@@ -42,6 +43,101 @@ static inline void cfs_se_util_change(struct sched_avg *avg)
WRITE_ONCE(avg->util_est.enqueued, enqueued); WRITE_ONCE(avg->util_est.enqueued, enqueued);
} }
/*
* The clock_pelt scales the time to reflect the effective amount of
* computation done during the running delta time but then sync back to
* clock_task when rq is idle.
*
*
* absolute time | 1| 2| 3| 4| 5| 6| 7| 8| 9|10|11|12|13|14|15|16
* @ max capacity ------******---------------******---------------
* @ half capacity ------************---------************---------
* clock pelt | 1| 2| 3| 4| 7| 8| 9| 10| 11|14|15|16
*
*/
static inline void update_rq_clock_pelt(struct rq *rq, s64 delta)
{
if (unlikely(is_idle_task(rq->curr))) {
/* The rq is idle, we can sync to clock_task */
rq->clock_pelt = rq_clock_task(rq);
return;
}
/*
* When a rq runs at a lower compute capacity, it will need
* more time to do the same amount of work than at max
* capacity. In order to be invariant, we scale the delta to
* reflect how much work has been really done.
* Running longer results in stealing idle time that will
* disturb the load signal compared to max capacity. This
* stolen idle time will be automatically reflected when the
* rq will be idle and the clock will be synced with
* rq_clock_task.
*/
/*
* Scale the elapsed time to reflect the real amount of
* computation
*/
delta = cap_scale(delta, arch_scale_cpu_capacity(NULL, cpu_of(rq)));
delta = cap_scale(delta, arch_scale_freq_capacity(cpu_of(rq)));
rq->clock_pelt += delta;
}
/*
* When rq becomes idle, we have to check if it has lost idle time
* because it was fully busy. A rq is fully used when the /Sum util_sum
* is greater or equal to:
* (LOAD_AVG_MAX - 1024 + rq->cfs.avg.period_contrib) << SCHED_CAPACITY_SHIFT;
* For optimization and computing rounding purpose, we don't take into account
* the position in the current window (period_contrib) and we use the higher
* bound of util_sum to decide.
*/
static inline void update_idle_rq_clock_pelt(struct rq *rq)
{
u32 divider = ((LOAD_AVG_MAX - 1024) << SCHED_CAPACITY_SHIFT) - LOAD_AVG_MAX;
u32 util_sum = rq->cfs.avg.util_sum;
util_sum += rq->avg_rt.util_sum;
util_sum += rq->avg_dl.util_sum;
/*
* Reflecting stolen time makes sense only if the idle
* phase would be present at max capacity. As soon as the
* utilization of a rq has reached the maximum value, it is
* considered as an always runnig rq without idle time to
* steal. This potential idle time is considered as lost in
* this case. We keep track of this lost idle time compare to
* rq's clock_task.
*/
if (util_sum >= divider)
rq->lost_idle_time += rq_clock_task(rq) - rq->clock_pelt;
}
static inline u64 rq_clock_pelt(struct rq *rq)
{
lockdep_assert_held(&rq->lock);
assert_clock_updated(rq);
return rq->clock_pelt - rq->lost_idle_time;
}
#ifdef CONFIG_CFS_BANDWIDTH
/* rq->task_clock normalized against any time this cfs_rq has spent throttled */
static inline u64 cfs_rq_clock_pelt(struct cfs_rq *cfs_rq)
{
if (unlikely(cfs_rq->throttle_count))
return cfs_rq->throttled_clock_task - cfs_rq->throttled_clock_task_time;
return rq_clock_pelt(rq_of(cfs_rq)) - cfs_rq->throttled_clock_task_time;
}
#else
static inline u64 cfs_rq_clock_pelt(struct cfs_rq *cfs_rq)
{
return rq_clock_pelt(rq_of(cfs_rq));
}
#endif
#else #else
static inline int static inline int
...@@ -67,6 +163,18 @@ update_irq_load_avg(struct rq *rq, u64 running) ...@@ -67,6 +163,18 @@ update_irq_load_avg(struct rq *rq, u64 running)
{ {
return 0; return 0;
} }
static inline u64 rq_clock_pelt(struct rq *rq)
{
return rq_clock_task(rq);
}
static inline void
update_rq_clock_pelt(struct rq *rq, s64 delta) { }
static inline void
update_idle_rq_clock_pelt(struct rq *rq) { }
#endif #endif
...@@ -1587,7 +1587,7 @@ pick_next_task_rt(struct rq *rq, struct task_struct *prev, struct rq_flags *rf) ...@@ -1587,7 +1587,7 @@ pick_next_task_rt(struct rq *rq, struct task_struct *prev, struct rq_flags *rf)
* rt task * rt task
*/ */
if (rq->curr->sched_class != &rt_sched_class) if (rq->curr->sched_class != &rt_sched_class)
update_rt_rq_load_avg(rq_clock_task(rq), rq, 0); update_rt_rq_load_avg(rq_clock_pelt(rq), rq, 0);
return p; return p;
} }
...@@ -1596,7 +1596,7 @@ static void put_prev_task_rt(struct rq *rq, struct task_struct *p) ...@@ -1596,7 +1596,7 @@ static void put_prev_task_rt(struct rq *rq, struct task_struct *p)
{ {
update_curr_rt(rq); update_curr_rt(rq);
update_rt_rq_load_avg(rq_clock_task(rq), rq, 1); update_rt_rq_load_avg(rq_clock_pelt(rq), rq, 1);
/* /*
* The previous task needs to be made eligible for pushing * The previous task needs to be made eligible for pushing
...@@ -2325,7 +2325,7 @@ static void task_tick_rt(struct rq *rq, struct task_struct *p, int queued) ...@@ -2325,7 +2325,7 @@ static void task_tick_rt(struct rq *rq, struct task_struct *p, int queued)
struct sched_rt_entity *rt_se = &p->rt; struct sched_rt_entity *rt_se = &p->rt;
update_curr_rt(rq); update_curr_rt(rq);
update_rt_rq_load_avg(rq_clock_task(rq), rq, 1); update_rt_rq_load_avg(rq_clock_pelt(rq), rq, 1);
watchdog(rq, p); watchdog(rq, p);
......
...@@ -861,7 +861,10 @@ struct rq { ...@@ -861,7 +861,10 @@ struct rq {
unsigned int clock_update_flags; unsigned int clock_update_flags;
u64 clock; u64 clock;
u64 clock_task; /* Ensure that all clocks are in the same cache line */
u64 clock_task ____cacheline_aligned;
u64 clock_pelt;
unsigned long lost_idle_time;
atomic_t nr_iowait; atomic_t nr_iowait;
...@@ -951,6 +954,22 @@ struct rq { ...@@ -951,6 +954,22 @@ struct rq {
#endif #endif
}; };
#ifdef CONFIG_FAIR_GROUP_SCHED
/* CPU runqueue to which this cfs_rq is attached */
static inline struct rq *rq_of(struct cfs_rq *cfs_rq)
{
return cfs_rq->rq;
}
#else
static inline struct rq *rq_of(struct cfs_rq *cfs_rq)
{
return container_of(cfs_rq, struct rq, cfs);
}
#endif
static inline int cpu_of(struct rq *rq) static inline int cpu_of(struct rq *rq)
{ {
#ifdef CONFIG_SMP #ifdef CONFIG_SMP
...@@ -1460,9 +1479,9 @@ static inline void __set_task_cpu(struct task_struct *p, unsigned int cpu) ...@@ -1460,9 +1479,9 @@ static inline void __set_task_cpu(struct task_struct *p, unsigned int cpu)
*/ */
smp_wmb(); smp_wmb();
#ifdef CONFIG_THREAD_INFO_IN_TASK #ifdef CONFIG_THREAD_INFO_IN_TASK
p->cpu = cpu; WRITE_ONCE(p->cpu, cpu);
#else #else
task_thread_info(p)->cpu = cpu; WRITE_ONCE(task_thread_info(p)->cpu, cpu);
#endif #endif
p->wake_cpu = cpu; p->wake_cpu = cpu;
#endif #endif
...@@ -1563,7 +1582,7 @@ static inline int task_on_rq_queued(struct task_struct *p) ...@@ -1563,7 +1582,7 @@ static inline int task_on_rq_queued(struct task_struct *p)
static inline int task_on_rq_migrating(struct task_struct *p) static inline int task_on_rq_migrating(struct task_struct *p)
{ {
return p->on_rq == TASK_ON_RQ_MIGRATING; return READ_ONCE(p->on_rq) == TASK_ON_RQ_MIGRATING;
} }
/* /*
...@@ -1781,7 +1800,7 @@ extern void init_dl_rq_bw_ratio(struct dl_rq *dl_rq); ...@@ -1781,7 +1800,7 @@ extern void init_dl_rq_bw_ratio(struct dl_rq *dl_rq);
unsigned long to_ratio(u64 period, u64 runtime); unsigned long to_ratio(u64 period, u64 runtime);
extern void init_entity_runnable_average(struct sched_entity *se); extern void init_entity_runnable_average(struct sched_entity *se);
extern void post_init_entity_util_avg(struct sched_entity *se); extern void post_init_entity_util_avg(struct task_struct *p);
#ifdef CONFIG_NO_HZ_FULL #ifdef CONFIG_NO_HZ_FULL
extern bool sched_can_stop_tick(struct rq *rq); extern bool sched_can_stop_tick(struct rq *rq);
...@@ -2211,6 +2230,13 @@ static inline void cpufreq_update_util(struct rq *rq, unsigned int flags) {} ...@@ -2211,6 +2230,13 @@ static inline void cpufreq_update_util(struct rq *rq, unsigned int flags) {}
# define arch_scale_freq_invariant() false # define arch_scale_freq_invariant() false
#endif #endif
#ifdef CONFIG_SMP
static inline unsigned long capacity_orig_of(int cpu)
{
return cpu_rq(cpu)->cpu_capacity_orig;
}
#endif
#ifdef CONFIG_CPU_FREQ_GOV_SCHEDUTIL #ifdef CONFIG_CPU_FREQ_GOV_SCHEDUTIL
/** /**
* enum schedutil_type - CPU utilization type * enum schedutil_type - CPU utilization type
...@@ -2299,11 +2325,19 @@ unsigned long scale_irq_capacity(unsigned long util, unsigned long irq, unsigned ...@@ -2299,11 +2325,19 @@ unsigned long scale_irq_capacity(unsigned long util, unsigned long irq, unsigned
#endif #endif
#if defined(CONFIG_ENERGY_MODEL) && defined(CONFIG_CPU_FREQ_GOV_SCHEDUTIL) #if defined(CONFIG_ENERGY_MODEL) && defined(CONFIG_CPU_FREQ_GOV_SCHEDUTIL)
#define perf_domain_span(pd) (to_cpumask(((pd)->em_pd->cpus))) #define perf_domain_span(pd) (to_cpumask(((pd)->em_pd->cpus)))
#else
DECLARE_STATIC_KEY_FALSE(sched_energy_present);
static inline bool sched_energy_enabled(void)
{
return static_branch_unlikely(&sched_energy_present);
}
#else /* ! (CONFIG_ENERGY_MODEL && CONFIG_CPU_FREQ_GOV_SCHEDUTIL) */
#define perf_domain_span(pd) NULL #define perf_domain_span(pd) NULL
#endif static inline bool sched_energy_enabled(void) { return false; }
#ifdef CONFIG_SMP #endif /* CONFIG_ENERGY_MODEL && CONFIG_CPU_FREQ_GOV_SCHEDUTIL */
extern struct static_key_false sched_energy_present;
#endif
...@@ -201,11 +201,37 @@ sd_parent_degenerate(struct sched_domain *sd, struct sched_domain *parent) ...@@ -201,11 +201,37 @@ sd_parent_degenerate(struct sched_domain *sd, struct sched_domain *parent)
return 1; return 1;
} }
DEFINE_STATIC_KEY_FALSE(sched_energy_present);
#if defined(CONFIG_ENERGY_MODEL) && defined(CONFIG_CPU_FREQ_GOV_SCHEDUTIL) #if defined(CONFIG_ENERGY_MODEL) && defined(CONFIG_CPU_FREQ_GOV_SCHEDUTIL)
DEFINE_STATIC_KEY_FALSE(sched_energy_present);
unsigned int sysctl_sched_energy_aware = 1;
DEFINE_MUTEX(sched_energy_mutex); DEFINE_MUTEX(sched_energy_mutex);
bool sched_energy_update; bool sched_energy_update;
#ifdef CONFIG_PROC_SYSCTL
int sched_energy_aware_handler(struct ctl_table *table, int write,
void __user *buffer, size_t *lenp, loff_t *ppos)
{
int ret, state;
if (write && !capable(CAP_SYS_ADMIN))
return -EPERM;
ret = proc_dointvec_minmax(table, write, buffer, lenp, ppos);
if (!ret && write) {
state = static_branch_unlikely(&sched_energy_present);
if (state != sysctl_sched_energy_aware) {
mutex_lock(&sched_energy_mutex);
sched_energy_update = 1;
rebuild_sched_domains();
sched_energy_update = 0;
mutex_unlock(&sched_energy_mutex);
}
}
return ret;
}
#endif
static void free_pd(struct perf_domain *pd) static void free_pd(struct perf_domain *pd)
{ {
struct perf_domain *tmp; struct perf_domain *tmp;
...@@ -322,6 +348,9 @@ static bool build_perf_domains(const struct cpumask *cpu_map) ...@@ -322,6 +348,9 @@ static bool build_perf_domains(const struct cpumask *cpu_map)
struct cpufreq_policy *policy; struct cpufreq_policy *policy;
struct cpufreq_governor *gov; struct cpufreq_governor *gov;
if (!sysctl_sched_energy_aware)
goto free;
/* EAS is enabled for asymmetric CPU capacity topologies. */ /* EAS is enabled for asymmetric CPU capacity topologies. */
if (!per_cpu(sd_asym_cpucapacity, cpu)) { if (!per_cpu(sd_asym_cpucapacity, cpu)) {
if (sched_debug()) { if (sched_debug()) {
...@@ -676,7 +705,7 @@ cpu_attach_domain(struct sched_domain *sd, struct root_domain *rd, int cpu) ...@@ -676,7 +705,7 @@ cpu_attach_domain(struct sched_domain *sd, struct root_domain *rd, int cpu)
} }
struct s_data { struct s_data {
struct sched_domain ** __percpu sd; struct sched_domain * __percpu *sd;
struct root_domain *rd; struct root_domain *rd;
}; };
......
...@@ -472,6 +472,17 @@ static struct ctl_table kern_table[] = { ...@@ -472,6 +472,17 @@ static struct ctl_table kern_table[] = {
.extra1 = &one, .extra1 = &one,
}, },
#endif #endif
#if defined(CONFIG_ENERGY_MODEL) && defined(CONFIG_CPU_FREQ_GOV_SCHEDUTIL)
{
.procname = "sched_energy_aware",
.data = &sysctl_sched_energy_aware,
.maxlen = sizeof(unsigned int),
.mode = 0644,
.proc_handler = sched_energy_aware_handler,
.extra1 = &zero,
.extra2 = &one,
},
#endif
#ifdef CONFIG_PROVE_LOCKING #ifdef CONFIG_PROVE_LOCKING
{ {
.procname = "prove_locking", .procname = "prove_locking",
......
Markdown is supported
0%
or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment