Commit eb414681 authored by Johannes Weiner's avatar Johannes Weiner Committed by Linus Torvalds

psi: pressure stall information for CPU, memory, and IO

When systems are overcommitted and resources become contended, it's hard
to tell exactly the impact this has on workload productivity, or how close
the system is to lockups and OOM kills.  In particular, when machines work
multiple jobs concurrently, the impact of overcommit in terms of latency
and throughput on the individual job can be enormous.

In order to maximize hardware utilization without sacrificing individual
job health or risk complete machine lockups, this patch implements a way
to quantify resource pressure in the system.

A kernel built with CONFIG_PSI=y creates files in /proc/pressure/ that
expose the percentage of time the system is stalled on CPU, memory, or IO,
respectively.  Stall states are aggregate versions of the per-task delay
accounting delays:

       cpu: some tasks are runnable but not executing on a CPU
       memory: tasks are reclaiming, or waiting for swapin or thrashing cache
       io: tasks are waiting for io completions

These percentages of walltime can be thought of as pressure percentages,
and they give a general sense of system health and productivity loss
incurred by resource overcommit.  They can also indicate when the system
is approaching lockup scenarios and OOMs.

To do this, psi keeps track of the task states associated with each CPU
and samples the time they spend in stall states.  Every 2 seconds, the
samples are averaged across CPUs - weighted by the CPUs' non-idle time to
eliminate artifacts from unused CPUs - and translated into percentages of
walltime.  A running average of those percentages is maintained over 10s,
1m, and 5m periods (similar to the loadaverage).

[hannes@cmpxchg.org: doc fixlet, per Randy]
  Link: http://lkml.kernel.org/r/20180828205625.GA14030@cmpxchg.org
[hannes@cmpxchg.org: code optimization]
  Link: http://lkml.kernel.org/r/20180907175015.GA8479@cmpxchg.org
[hannes@cmpxchg.org: rename psi_clock() to psi_update_work(), per Peter]
  Link: http://lkml.kernel.org/r/20180907145404.GB11088@cmpxchg.org
[hannes@cmpxchg.org: fix build]
  Link: http://lkml.kernel.org/r/20180913014222.GA2370@cmpxchg.org
Link: http://lkml.kernel.org/r/20180828172258.3185-9-hannes@cmpxchg.orgSigned-off-by: default avatarJohannes Weiner <hannes@cmpxchg.org>
Acked-by: default avatarPeter Zijlstra (Intel) <peterz@infradead.org>
Tested-by: default avatarDaniel Drake <drake@endlessm.com>
Tested-by: default avatarSuren Baghdasaryan <surenb@google.com>
Cc: Christopher Lameter <cl@linux.com>
Cc: Ingo Molnar <mingo@redhat.com>
Cc: Johannes Weiner <jweiner@fb.com>
Cc: Mike Galbraith <efault@gmx.de>
Cc: Peter Enderborg <peter.enderborg@sony.com>
Cc: Randy Dunlap <rdunlap@infradead.org>
Cc: Shakeel Butt <shakeelb@google.com>
Cc: Tejun Heo <tj@kernel.org>
Cc: Vinayak Menon <vinmenon@codeaurora.org>
Cc: Randy Dunlap <rdunlap@infradead.org>
Signed-off-by: default avatarAndrew Morton <akpm@linux-foundation.org>
Signed-off-by: default avatarLinus Torvalds <torvalds@linux-foundation.org>
parent 246b3b33
================================
PSI - Pressure Stall Information
================================
:Date: April, 2018
:Author: Johannes Weiner <hannes@cmpxchg.org>
When CPU, memory or IO devices are contended, workloads experience
latency spikes, throughput losses, and run the risk of OOM kills.
Without an accurate measure of such contention, users are forced to
either play it safe and under-utilize their hardware resources, or
roll the dice and frequently suffer the disruptions resulting from
excessive overcommit.
The psi feature identifies and quantifies the disruptions caused by
such resource crunches and the time impact it has on complex workloads
or even entire systems.
Having an accurate measure of productivity losses caused by resource
scarcity aids users in sizing workloads to hardware--or provisioning
hardware according to workload demand.
As psi aggregates this information in realtime, systems can be managed
dynamically using techniques such as load shedding, migrating jobs to
other systems or data centers, or strategically pausing or killing low
priority or restartable batch jobs.
This allows maximizing hardware utilization without sacrificing
workload health or risking major disruptions such as OOM kills.
Pressure interface
==================
Pressure information for each resource is exported through the
respective file in /proc/pressure/ -- cpu, memory, and io.
The format for CPU is as such:
some avg10=0.00 avg60=0.00 avg300=0.00 total=0
and for memory and IO:
some avg10=0.00 avg60=0.00 avg300=0.00 total=0
full avg10=0.00 avg60=0.00 avg300=0.00 total=0
The "some" line indicates the share of time in which at least some
tasks are stalled on a given resource.
The "full" line indicates the share of time in which all non-idle
tasks are stalled on a given resource simultaneously. In this state
actual CPU cycles are going to waste, and a workload that spends
extended time in this state is considered to be thrashing. This has
severe impact on performance, and it's useful to distinguish this
situation from a state where some tasks are stalled but the CPU is
still doing productive work. As such, time spent in this subset of the
stall state is tracked separately and exported in the "full" averages.
The ratios are tracked as recent trends over ten, sixty, and three
hundred second windows, which gives insight into short term events as
well as medium and long term trends. The total absolute stall time is
tracked and exported as well, to allow detection of latency spikes
which wouldn't necessarily make a dent in the time averages, or to
average trends over custom time frames.
#ifndef _LINUX_PSI_H
#define _LINUX_PSI_H
#include <linux/psi_types.h>
#include <linux/sched.h>
#ifdef CONFIG_PSI
extern bool psi_disabled;
void psi_init(void);
void psi_task_change(struct task_struct *task, int clear, int set);
void psi_memstall_tick(struct task_struct *task, int cpu);
void psi_memstall_enter(unsigned long *flags);
void psi_memstall_leave(unsigned long *flags);
#else /* CONFIG_PSI */
static inline void psi_init(void) {}
static inline void psi_memstall_enter(unsigned long *flags) {}
static inline void psi_memstall_leave(unsigned long *flags) {}
#endif /* CONFIG_PSI */
#endif /* _LINUX_PSI_H */
#ifndef _LINUX_PSI_TYPES_H
#define _LINUX_PSI_TYPES_H
#include <linux/seqlock.h>
#include <linux/types.h>
#ifdef CONFIG_PSI
/* Tracked task states */
enum psi_task_count {
NR_IOWAIT,
NR_MEMSTALL,
NR_RUNNING,
NR_PSI_TASK_COUNTS,
};
/* Task state bitmasks */
#define TSK_IOWAIT (1 << NR_IOWAIT)
#define TSK_MEMSTALL (1 << NR_MEMSTALL)
#define TSK_RUNNING (1 << NR_RUNNING)
/* Resources that workloads could be stalled on */
enum psi_res {
PSI_IO,
PSI_MEM,
PSI_CPU,
NR_PSI_RESOURCES,
};
/*
* Pressure states for each resource:
*
* SOME: Stalled tasks & working tasks
* FULL: Stalled tasks & no working tasks
*/
enum psi_states {
PSI_IO_SOME,
PSI_IO_FULL,
PSI_MEM_SOME,
PSI_MEM_FULL,
PSI_CPU_SOME,
/* Only per-CPU, to weigh the CPU in the global average: */
PSI_NONIDLE,
NR_PSI_STATES,
};
struct psi_group_cpu {
/* 1st cacheline updated by the scheduler */
/* Aggregator needs to know of concurrent changes */
seqcount_t seq ____cacheline_aligned_in_smp;
/* States of the tasks belonging to this group */
unsigned int tasks[NR_PSI_TASK_COUNTS];
/* Period time sampling buckets for each state of interest (ns) */
u32 times[NR_PSI_STATES];
/* Time of last task change in this group (rq_clock) */
u64 state_start;
/* 2nd cacheline updated by the aggregator */
/* Delta detection against the sampling buckets */
u32 times_prev[NR_PSI_STATES] ____cacheline_aligned_in_smp;
};
struct psi_group {
/* Protects data updated during an aggregation */
struct mutex stat_lock;
/* Per-cpu task state & time tracking */
struct psi_group_cpu __percpu *pcpu;
/* Periodic aggregation state */
u64 total_prev[NR_PSI_STATES - 1];
u64 last_update;
u64 next_update;
struct delayed_work clock_work;
/* Total stall times and sampled pressure averages */
u64 total[NR_PSI_STATES - 1];
unsigned long avg[NR_PSI_STATES - 1][3];
};
#else /* CONFIG_PSI */
struct psi_group { };
#endif /* CONFIG_PSI */
#endif /* _LINUX_PSI_TYPES_H */
...@@ -25,6 +25,7 @@ ...@@ -25,6 +25,7 @@
#include <linux/latencytop.h> #include <linux/latencytop.h>
#include <linux/sched/prio.h> #include <linux/sched/prio.h>
#include <linux/signal_types.h> #include <linux/signal_types.h>
#include <linux/psi_types.h>
#include <linux/mm_types_task.h> #include <linux/mm_types_task.h>
#include <linux/task_io_accounting.h> #include <linux/task_io_accounting.h>
#include <linux/rseq.h> #include <linux/rseq.h>
...@@ -706,6 +707,10 @@ struct task_struct { ...@@ -706,6 +707,10 @@ struct task_struct {
unsigned sched_contributes_to_load:1; unsigned sched_contributes_to_load:1;
unsigned sched_migrated:1; unsigned sched_migrated:1;
unsigned sched_remote_wakeup:1; unsigned sched_remote_wakeup:1;
#ifdef CONFIG_PSI
unsigned sched_psi_wake_requeue:1;
#endif
/* Force alignment to the next boundary: */ /* Force alignment to the next boundary: */
unsigned :0; unsigned :0;
...@@ -965,6 +970,10 @@ struct task_struct { ...@@ -965,6 +970,10 @@ struct task_struct {
kernel_siginfo_t *last_siginfo; kernel_siginfo_t *last_siginfo;
struct task_io_accounting ioac; struct task_io_accounting ioac;
#ifdef CONFIG_PSI
/* Pressure stall state */
unsigned int psi_flags;
#endif
#ifdef CONFIG_TASK_XACCT #ifdef CONFIG_TASK_XACCT
/* Accumulated RSS usage: */ /* Accumulated RSS usage: */
u64 acct_rss_mem1; u64 acct_rss_mem1;
...@@ -1391,6 +1400,7 @@ extern struct pid *cad_pid; ...@@ -1391,6 +1400,7 @@ extern struct pid *cad_pid;
#define PF_KTHREAD 0x00200000 /* I am a kernel thread */ #define PF_KTHREAD 0x00200000 /* I am a kernel thread */
#define PF_RANDOMIZE 0x00400000 /* Randomize virtual address space */ #define PF_RANDOMIZE 0x00400000 /* Randomize virtual address space */
#define PF_SWAPWRITE 0x00800000 /* Allowed to write to swap */ #define PF_SWAPWRITE 0x00800000 /* Allowed to write to swap */
#define PF_MEMSTALL 0x01000000 /* Stalled due to lack of memory */
#define PF_NO_SETAFFINITY 0x04000000 /* Userland is not allowed to meddle with cpus_allowed */ #define PF_NO_SETAFFINITY 0x04000000 /* Userland is not allowed to meddle with cpus_allowed */
#define PF_MCE_EARLY 0x08000000 /* Early kill for mce process policy */ #define PF_MCE_EARLY 0x08000000 /* Early kill for mce process policy */
#define PF_MUTEX_TESTER 0x20000000 /* Thread belongs to the rt mutex tester */ #define PF_MUTEX_TESTER 0x20000000 /* Thread belongs to the rt mutex tester */
......
...@@ -490,6 +490,21 @@ config TASK_IO_ACCOUNTING ...@@ -490,6 +490,21 @@ config TASK_IO_ACCOUNTING
Say N if unsure. Say N if unsure.
config PSI
bool "Pressure stall information tracking"
help
Collect metrics that indicate how overcommitted the CPU, memory,
and IO capacity are in the system.
If you say Y here, the kernel will create /proc/pressure/ with the
pressure statistics files cpu, memory, and io. These will indicate
the share of walltime in which some or all tasks in the system are
delayed due to contention of the respective resource.
For more details see Documentation/accounting/psi.txt.
Say N if unsure.
endmenu # "CPU/Task time and stats accounting" endmenu # "CPU/Task time and stats accounting"
config CPU_ISOLATION config CPU_ISOLATION
......
...@@ -1822,6 +1822,10 @@ static __latent_entropy struct task_struct *copy_process( ...@@ -1822,6 +1822,10 @@ static __latent_entropy struct task_struct *copy_process(
p->default_timer_slack_ns = current->timer_slack_ns; p->default_timer_slack_ns = current->timer_slack_ns;
#ifdef CONFIG_PSI
p->psi_flags = 0;
#endif
task_io_accounting_init(&p->ioac); task_io_accounting_init(&p->ioac);
acct_clear_integrals(p); acct_clear_integrals(p);
......
...@@ -29,3 +29,4 @@ obj-$(CONFIG_CPU_FREQ) += cpufreq.o ...@@ -29,3 +29,4 @@ obj-$(CONFIG_CPU_FREQ) += cpufreq.o
obj-$(CONFIG_CPU_FREQ_GOV_SCHEDUTIL) += cpufreq_schedutil.o obj-$(CONFIG_CPU_FREQ_GOV_SCHEDUTIL) += cpufreq_schedutil.o
obj-$(CONFIG_MEMBARRIER) += membarrier.o obj-$(CONFIG_MEMBARRIER) += membarrier.o
obj-$(CONFIG_CPU_ISOLATION) += isolation.o obj-$(CONFIG_CPU_ISOLATION) += isolation.o
obj-$(CONFIG_PSI) += psi.o
...@@ -722,8 +722,10 @@ static inline void enqueue_task(struct rq *rq, struct task_struct *p, int flags) ...@@ -722,8 +722,10 @@ static inline void enqueue_task(struct rq *rq, struct task_struct *p, int flags)
if (!(flags & ENQUEUE_NOCLOCK)) if (!(flags & ENQUEUE_NOCLOCK))
update_rq_clock(rq); update_rq_clock(rq);
if (!(flags & ENQUEUE_RESTORE)) if (!(flags & ENQUEUE_RESTORE)) {
sched_info_queued(rq, p); sched_info_queued(rq, p);
psi_enqueue(p, flags & ENQUEUE_WAKEUP);
}
p->sched_class->enqueue_task(rq, p, flags); p->sched_class->enqueue_task(rq, p, flags);
} }
...@@ -733,8 +735,10 @@ static inline void dequeue_task(struct rq *rq, struct task_struct *p, int flags) ...@@ -733,8 +735,10 @@ static inline void dequeue_task(struct rq *rq, struct task_struct *p, int flags)
if (!(flags & DEQUEUE_NOCLOCK)) if (!(flags & DEQUEUE_NOCLOCK))
update_rq_clock(rq); update_rq_clock(rq);
if (!(flags & DEQUEUE_SAVE)) if (!(flags & DEQUEUE_SAVE)) {
sched_info_dequeued(rq, p); sched_info_dequeued(rq, p);
psi_dequeue(p, flags & DEQUEUE_SLEEP);
}
p->sched_class->dequeue_task(rq, p, flags); p->sched_class->dequeue_task(rq, p, flags);
} }
...@@ -2037,6 +2041,7 @@ try_to_wake_up(struct task_struct *p, unsigned int state, int wake_flags) ...@@ -2037,6 +2041,7 @@ try_to_wake_up(struct task_struct *p, unsigned int state, int wake_flags)
cpu = select_task_rq(p, p->wake_cpu, SD_BALANCE_WAKE, wake_flags); cpu = select_task_rq(p, p->wake_cpu, SD_BALANCE_WAKE, wake_flags);
if (task_cpu(p) != cpu) { if (task_cpu(p) != cpu) {
wake_flags |= WF_MIGRATED; wake_flags |= WF_MIGRATED;
psi_ttwu_dequeue(p);
set_task_cpu(p, cpu); set_task_cpu(p, cpu);
} }
...@@ -3051,6 +3056,7 @@ void scheduler_tick(void) ...@@ -3051,6 +3056,7 @@ void scheduler_tick(void)
curr->sched_class->task_tick(rq, curr, 0); curr->sched_class->task_tick(rq, curr, 0);
cpu_load_update_active(rq); cpu_load_update_active(rq);
calc_global_load_tick(rq); calc_global_load_tick(rq);
psi_task_tick(rq);
rq_unlock(rq, &rf); rq_unlock(rq, &rf);
...@@ -6067,6 +6073,8 @@ void __init sched_init(void) ...@@ -6067,6 +6073,8 @@ void __init sched_init(void)
init_schedstats(); init_schedstats();
psi_init();
scheduler_running = 1; scheduler_running = 1;
} }
......
This diff is collapsed.
...@@ -54,6 +54,7 @@ ...@@ -54,6 +54,7 @@
#include <linux/proc_fs.h> #include <linux/proc_fs.h>
#include <linux/prefetch.h> #include <linux/prefetch.h>
#include <linux/profile.h> #include <linux/profile.h>
#include <linux/psi.h>
#include <linux/rcupdate_wait.h> #include <linux/rcupdate_wait.h>
#include <linux/security.h> #include <linux/security.h>
#include <linux/stop_machine.h> #include <linux/stop_machine.h>
...@@ -319,6 +320,7 @@ extern bool dl_cpu_busy(unsigned int cpu); ...@@ -319,6 +320,7 @@ extern bool dl_cpu_busy(unsigned int cpu);
#ifdef CONFIG_CGROUP_SCHED #ifdef CONFIG_CGROUP_SCHED
#include <linux/cgroup.h> #include <linux/cgroup.h>
#include <linux/psi.h>
struct cfs_rq; struct cfs_rq;
struct rt_rq; struct rt_rq;
......
...@@ -55,6 +55,92 @@ static inline void rq_sched_info_depart (struct rq *rq, unsigned long long delt ...@@ -55,6 +55,92 @@ static inline void rq_sched_info_depart (struct rq *rq, unsigned long long delt
# define schedstat_val_or_zero(var) 0 # define schedstat_val_or_zero(var) 0
#endif /* CONFIG_SCHEDSTATS */ #endif /* CONFIG_SCHEDSTATS */
#ifdef CONFIG_PSI
/*
* PSI tracks state that persists across sleeps, such as iowaits and
* memory stalls. As a result, it has to distinguish between sleeps,
* where a task's runnable state changes, and requeues, where a task
* and its state are being moved between CPUs and runqueues.
*/
static inline void psi_enqueue(struct task_struct *p, bool wakeup)
{
int clear = 0, set = TSK_RUNNING;
if (psi_disabled)
return;
if (!wakeup || p->sched_psi_wake_requeue) {
if (p->flags & PF_MEMSTALL)
set |= TSK_MEMSTALL;
if (p->sched_psi_wake_requeue)
p->sched_psi_wake_requeue = 0;
} else {
if (p->in_iowait)
clear |= TSK_IOWAIT;
}
psi_task_change(p, clear, set);
}
static inline void psi_dequeue(struct task_struct *p, bool sleep)
{
int clear = TSK_RUNNING, set = 0;
if (psi_disabled)
return;
if (!sleep) {
if (p->flags & PF_MEMSTALL)
clear |= TSK_MEMSTALL;
} else {
if (p->in_iowait)
set |= TSK_IOWAIT;
}
psi_task_change(p, clear, set);
}
static inline void psi_ttwu_dequeue(struct task_struct *p)
{
if (psi_disabled)
return;
/*
* Is the task being migrated during a wakeup? Make sure to
* deregister its sleep-persistent psi states from the old
* queue, and let psi_enqueue() know it has to requeue.
*/
if (unlikely(p->in_iowait || (p->flags & PF_MEMSTALL))) {
struct rq_flags rf;
struct rq *rq;
int clear = 0;
if (p->in_iowait)
clear |= TSK_IOWAIT;
if (p->flags & PF_MEMSTALL)
clear |= TSK_MEMSTALL;
rq = __task_rq_lock(p, &rf);
psi_task_change(p, clear, 0);
p->sched_psi_wake_requeue = 1;
__task_rq_unlock(rq, &rf);
}
}
static inline void psi_task_tick(struct rq *rq)
{
if (psi_disabled)
return;
if (unlikely(rq->curr->flags & PF_MEMSTALL))
psi_memstall_tick(rq->curr, cpu_of(rq));
}
#else /* CONFIG_PSI */
static inline void psi_enqueue(struct task_struct *p, bool wakeup) {}
static inline void psi_dequeue(struct task_struct *p, bool sleep) {}
static inline void psi_ttwu_dequeue(struct task_struct *p) {}
static inline void psi_task_tick(struct rq *rq) {}
#endif /* CONFIG_PSI */
#ifdef CONFIG_SCHED_INFO #ifdef CONFIG_SCHED_INFO
static inline void sched_info_reset_dequeued(struct task_struct *t) static inline void sched_info_reset_dequeued(struct task_struct *t)
{ {
......
...@@ -22,6 +22,7 @@ ...@@ -22,6 +22,7 @@
#include <linux/kthread.h> #include <linux/kthread.h>
#include <linux/freezer.h> #include <linux/freezer.h>
#include <linux/page_owner.h> #include <linux/page_owner.h>
#include <linux/psi.h>
#include "internal.h" #include "internal.h"
#ifdef CONFIG_COMPACTION #ifdef CONFIG_COMPACTION
...@@ -2068,11 +2069,15 @@ static int kcompactd(void *p) ...@@ -2068,11 +2069,15 @@ static int kcompactd(void *p)
pgdat->kcompactd_classzone_idx = pgdat->nr_zones - 1; pgdat->kcompactd_classzone_idx = pgdat->nr_zones - 1;
while (!kthread_should_stop()) { while (!kthread_should_stop()) {
unsigned long pflags;
trace_mm_compaction_kcompactd_sleep(pgdat->node_id); trace_mm_compaction_kcompactd_sleep(pgdat->node_id);
wait_event_freezable(pgdat->kcompactd_wait, wait_event_freezable(pgdat->kcompactd_wait,
kcompactd_work_requested(pgdat)); kcompactd_work_requested(pgdat));
psi_memstall_enter(&pflags);
kcompactd_do_work(pgdat); kcompactd_do_work(pgdat);
psi_memstall_leave(&pflags);
} }
return 0; return 0;
......
...@@ -37,6 +37,7 @@ ...@@ -37,6 +37,7 @@
#include <linux/shmem_fs.h> #include <linux/shmem_fs.h>
#include <linux/rmap.h> #include <linux/rmap.h>
#include <linux/delayacct.h> #include <linux/delayacct.h>
#include <linux/psi.h>
#include "internal.h" #include "internal.h"
#define CREATE_TRACE_POINTS #define CREATE_TRACE_POINTS
...@@ -1075,11 +1076,14 @@ static inline int wait_on_page_bit_common(wait_queue_head_t *q, ...@@ -1075,11 +1076,14 @@ static inline int wait_on_page_bit_common(wait_queue_head_t *q,
struct wait_page_queue wait_page; struct wait_page_queue wait_page;
wait_queue_entry_t *wait = &wait_page.wait; wait_queue_entry_t *wait = &wait_page.wait;
bool thrashing = false; bool thrashing = false;
unsigned long pflags;
int ret = 0; int ret = 0;
if (bit_nr == PG_locked && !PageSwapBacked(page) && if (bit_nr == PG_locked &&
!PageUptodate(page) && PageWorkingset(page)) { !PageUptodate(page) && PageWorkingset(page)) {
if (!PageSwapBacked(page))
delayacct_thrashing_start(); delayacct_thrashing_start();
psi_memstall_enter(&pflags);
thrashing = true; thrashing = true;
} }
...@@ -1121,8 +1125,11 @@ static inline int wait_on_page_bit_common(wait_queue_head_t *q, ...@@ -1121,8 +1125,11 @@ static inline int wait_on_page_bit_common(wait_queue_head_t *q,
finish_wait(q, wait); finish_wait(q, wait);
if (thrashing) if (thrashing) {
if (!PageSwapBacked(page))
delayacct_thrashing_end(); delayacct_thrashing_end();
psi_memstall_leave(&pflags);
}
/* /*
* A signal could leave PageWaiters set. Clearing it here if * A signal could leave PageWaiters set. Clearing it here if
......
...@@ -66,6 +66,7 @@ ...@@ -66,6 +66,7 @@
#include <linux/ftrace.h> #include <linux/ftrace.h>
#include <linux/lockdep.h> #include <linux/lockdep.h>
#include <linux/nmi.h> #include <linux/nmi.h>
#include <linux/psi.h>
#include <asm/sections.h> #include <asm/sections.h>
#include <asm/tlbflush.h> #include <asm/tlbflush.h>
...@@ -3549,15 +3550,20 @@ __alloc_pages_direct_compact(gfp_t gfp_mask, unsigned int order, ...@@ -3549,15 +3550,20 @@ __alloc_pages_direct_compact(gfp_t gfp_mask, unsigned int order,
enum compact_priority prio, enum compact_result *compact_result) enum compact_priority prio, enum compact_result *compact_result)
{ {
struct page *page; struct page *page;
unsigned long pflags;
unsigned int noreclaim_flag; unsigned int noreclaim_flag;
if (!order) if (!order)
return NULL; return NULL;
psi_memstall_enter(&pflags);
noreclaim_flag = memalloc_noreclaim_save(); noreclaim_flag = memalloc_noreclaim_save();
*compact_result = try_to_compact_pages(gfp_mask, order, alloc_flags, ac, *compact_result = try_to_compact_pages(gfp_mask, order, alloc_flags, ac,
prio); prio);
memalloc_noreclaim_restore(noreclaim_flag); memalloc_noreclaim_restore(noreclaim_flag);
psi_memstall_leave(&pflags);
if (*compact_result <= COMPACT_INACTIVE) if (*compact_result <= COMPACT_INACTIVE)
return NULL; return NULL;
...@@ -3756,11 +3762,13 @@ __perform_reclaim(gfp_t gfp_mask, unsigned int order, ...@@ -3756,11 +3762,13 @@ __perform_reclaim(gfp_t gfp_mask, unsigned int order,
struct reclaim_state reclaim_state; struct reclaim_state reclaim_state;
int progress; int progress;
unsigned int noreclaim_flag; unsigned int noreclaim_flag;
unsigned long pflags;
cond_resched(); cond_resched();
/* We now go into synchronous reclaim */ /* We now go into synchronous reclaim */
cpuset_memory_pressure_bump(); cpuset_memory_pressure_bump();
psi_memstall_enter(&pflags);
fs_reclaim_acquire(gfp_mask); fs_reclaim_acquire(gfp_mask);
noreclaim_flag = memalloc_noreclaim_save(); noreclaim_flag = memalloc_noreclaim_save();
reclaim_state.reclaimed_slab = 0; reclaim_state.reclaimed_slab = 0;
...@@ -3772,6 +3780,7 @@ __perform_reclaim(gfp_t gfp_mask, unsigned int order, ...@@ -3772,6 +3780,7 @@ __perform_reclaim(gfp_t gfp_mask, unsigned int order,
current->reclaim_state = NULL; current->reclaim_state = NULL;
memalloc_noreclaim_restore(noreclaim_flag); memalloc_noreclaim_restore(noreclaim_flag);
fs_reclaim_release(gfp_mask); fs_reclaim_release(gfp_mask);
psi_memstall_leave(&pflags);
cond_resched(); cond_resched();
......
...@@ -49,6 +49,7 @@ ...@@ -49,6 +49,7 @@
#include <linux/prefetch.h> #include <linux/prefetch.h>
#include <linux/printk.h> #include <linux/printk.h>
#include <linux/dax.h> #include <linux/dax.h>
#include <linux/psi.h>
#include <asm/tlbflush.h> #include <asm/tlbflush.h>
#include <asm/div64.h> #include <asm/div64.h>
...@@ -3305,6 +3306,7 @@ unsigned long try_to_free_mem_cgroup_pages(struct mem_cgroup *memcg, ...@@ -3305,6 +3306,7 @@ unsigned long try_to_free_mem_cgroup_pages(struct mem_cgroup *memcg,
{ {
struct zonelist *zonelist; struct zonelist *zonelist;
unsigned long nr_reclaimed; unsigned long nr_reclaimed;
unsigned long pflags;
int nid; int nid;
unsigned int noreclaim_flag; unsigned int noreclaim_flag;
struct scan_control sc = { struct scan_control sc = {
...@@ -3333,9 +3335,13 @@ unsigned long try_to_free_mem_cgroup_pages(struct mem_cgroup *memcg, ...@@ -3333,9 +3335,13 @@ unsigned long try_to_free_mem_cgroup_pages(struct mem_cgroup *memcg,
sc.gfp_mask, sc.gfp_mask,
sc.reclaim_idx); sc.reclaim_idx);
psi_memstall_enter(&pflags);
noreclaim_flag = memalloc_noreclaim_save(); noreclaim_flag = memalloc_noreclaim_save();
nr_reclaimed = do_try_to_free_pages(zonelist, &sc); nr_reclaimed = do_try_to_free_pages(zonelist, &sc);
memalloc_noreclaim_restore(noreclaim_flag); memalloc_noreclaim_restore(noreclaim_flag);
psi_memstall_leave(&pflags);
trace_mm_vmscan_memcg_reclaim_end(nr_reclaimed); trace_mm_vmscan_memcg_reclaim_end(nr_reclaimed);
...@@ -3500,6 +3506,7 @@ static int balance_pgdat(pg_data_t *pgdat, int order, int classzone_idx) ...@@ -3500,6 +3506,7 @@ static int balance_pgdat(pg_data_t *pgdat, int order, int classzone_idx)
int i; int i;
unsigned long nr_soft_reclaimed; unsigned long nr_soft_reclaimed;
unsigned long nr_soft_scanned; unsigned long nr_soft_scanned;
unsigned long pflags;
struct zone *zone; struct zone *zone;
struct scan_control sc = { struct scan_control sc = {
.gfp_mask = GFP_KERNEL, .gfp_mask = GFP_KERNEL,
...@@ -3510,6 +3517,7 @@ static int balance_pgdat(pg_data_t *pgdat, int order, int classzone_idx) ...@@ -3510,6 +3517,7 @@ static int balance_pgdat(pg_data_t *pgdat, int order, int classzone_idx)
.may_swap = 1, .may_swap = 1,
}; };
psi_memstall_enter(&pflags);
__fs_reclaim_acquire(); __fs_reclaim_acquire();
count_vm_event(PAGEOUTRUN); count_vm_event(PAGEOUTRUN);
...@@ -3611,6 +3619,7 @@ static int balance_pgdat(pg_data_t *pgdat, int order, int classzone_idx) ...@@ -3611,6 +3619,7 @@ static int balance_pgdat(pg_data_t *pgdat, int order, int classzone_idx)
out: out:
snapshot_refaults(NULL, pgdat); snapshot_refaults(NULL, pgdat);
__fs_reclaim_release(); __fs_reclaim_release();
psi_memstall_leave(&pflags);
/* /*
* Return the order kswapd stopped reclaiming at as * Return the order kswapd stopped reclaiming at as
* prepare_kswapd_sleep() takes it into account. If another caller * prepare_kswapd_sleep() takes it into account. If another caller
......
Markdown is supported
0%
or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment