the Linux scheduler is starving a number of workloads. So default
to more agressive idle-balancing. This hurts lmbench context-switching
numbers (which was the main reason we sucked at idle-balancing for
such a long time) but the lmbench numbers are fine once the system is
minimally utilized.
Signed-off-by: Ingo Molnar <mingo@elte.hu>

clean up the sleep_on() APIs:

 - do not use fastcall
 - replace fragile macro magic with proper inline functions
Signed-off-by: Ingo Molnar <mingo@elte.hu>

4 small style cleanups to sched.c: checkpatch.pl is now happy about
the totality of sched.c [ignoring false positives] - yay! ;-)
Signed-off-by: Ingo Molnar <mingo@elte.hu>

do not set softirqs to nice +19. _If_ for whatever reason
we missed to process some high-prio softirq and woke up
ksoftirqd, we should give it a fair chance to actually
get some work done, even if the system is under load.
Signed-off-by: Ingo Molnar <mingo@elte.hu>

simplify sched_rt.c's sched_find_first_bit() function: there are
only 100 RT priority levels left.
Signed-off-by: Ingo Molnar <mingo@elte.hu>

add Documentation/sched-design-CFS.txt
Signed-off-by: Ingo Molnar <mingo@elte.hu>

enable CONFIG_SCHED_DEBUG in lib/Kconfig.debug.

the runtime overhead of this option is very small.
Signed-off-by: Ingo Molnar <mingo@elte.hu>

scheduler debugging core: implement /proc/sched_debug and
/proc/<PID>/sched files for scheduler debugging.
Signed-off-by: Ingo Molnar <mingo@elte.hu>

add CFS debug sysctls: only tweakable if SCHED_DEBUG is enabled.
This allows for faster debugging of scheduler problems.
Signed-off-by: Ingo Molnar <mingo@elte.hu>

remove the old cpu-accounting field from signal_struct, now
that the code is using CFS's stats.
Signed-off-by: Ingo Molnar <mingo@elte.hu>

remove unused rq types from sched.c, now that we switched
over to CFS.
Signed-off-by: Ingo Molnar <mingo@elte.hu>

batch_task() in sched.h is now unused - remove it.
Signed-off-by: Ingo Molnar <mingo@elte.hu>

remove now-unused types/fields used by the old scheduler.
Signed-off-by: Ingo Molnar <mingo@elte.hu>

remove now unused interactivity-heuristics related defined and
types of the old scheduler.
Signed-off-by: Ingo Molnar <mingo@elte.hu>

clean up include files in sched.c, they were still old-style <asm/>.
Signed-off-by: Ingo Molnar <mingo@elte.hu>

sched_fork()/sched_exit() does not need to specify fastcall anymore,
as the x86 kernel defaults to regparm3, and no assembly code calls
these functions.
Signed-off-by: Ingo Molnar <mingo@elte.hu>

update delay-accounting to use CFS's precise stats.
Signed-off-by: Ingo Molnar <mingo@elte.hu>

make use of CFS's precise accounting to drive /proc/<pid>/stat statistics.

this code was co-authored by:

 Balbir Singh <balbir@linux.vnet.ibm.com>
 Dmitry Adamushko <dmitry.adamushko@gmail.com>
 Ingo Molnar <mingo@elte.hu>
Signed-off-by: Ingo Molnar <mingo@elte.hu>
Signed-off-by: Dmitry Adamushko <dmitry.adamushko@gmail.com>

make use of sched-clock-unstable events.
Signed-off-by: Ingo Molnar <mingo@elte.hu>

track TSC-unstable events and propagate it to the scheduler code.
Also allow sched_clock() to be used when the TSC is unstable,
the rq_clock() wrapper creates a reliable clock out of it.
Signed-off-by: Ingo Molnar <mingo@elte.hu>

apply the CFS core code.

this change switches over the scheduler core to CFS's modular
design and makes use of kernel/sched_fair/rt/idletask.c to implement
Linux's scheduling policies.

thanks to Andrew Morton and Thomas Gleixner for lots of detailed review
feedback and for fixlets.
Signed-off-by: Ingo Molnar <mingo@elte.hu>
Signed-off-by: Mike Galbraith <efault@gmx.de>
Signed-off-by: Dmitry Adamushko <dmitry.adamushko@gmail.com>
Signed-off-by: Srivatsa Vaddagiri <vatsa@linux.vnet.ibm.com>

remove the sleep-bonus interactivity code from the core scheduler.

scheduling policy is implemented in the policy modules, and CFS does
not need such type of heuristics.
Signed-off-by: Ingo Molnar <mingo@elte.hu>

remove the expired_starving() heuristics from the core scheduler.

CFS does not need it, and this did not really work well in practice
anyway, due to the rq->nr_running multiplier to STARVATION_LIMIT.
Signed-off-by: Ingo Molnar <mingo@elte.hu>

remove the sleep_type heuristics from the core scheduler - scheduling
policy is implemented in the scheduling-policy modules. (and CFS does
not use this type of sleep-type heuristics)
Signed-off-by: Ingo Molnar <mingo@elte.hu>

add the new load-calculation methods of CFS.
Signed-off-by: Ingo Molnar <mingo@elte.hu>

clean up: move __normal_prio() in head of normal_prio().

no code changed.
Signed-off-by: Ingo Molnar <mingo@elte.hu>

cleanup: move dequeue/enqueue_task() to a more logical place, to
not split up __normal_prio()/normal_prio().
Signed-off-by: Ingo Molnar <mingo@elte.hu>

move resched_task()/resched_cpu() into the 'public interfaces'
section of sched.c, for use by kernel/sched_fair/rt/idletask.c
Signed-off-by: Ingo Molnar <mingo@elte.hu>

remove the SleepAVG field from /proc/<pid>/status, as
with the removal of the sleep-average code this value
no longer makes sense.
Signed-off-by: Ingo Molnar <mingo@elte.hu>

clean up the rt priority macros, pointed out by Andrew Morton.
Signed-off-by: Ingo Molnar <mingo@elte.hu>

add the set_task_cfs_rq() abstraction needed by CONFIG_FAIR_GROUP_SCHED.

(not activated yet)
Signed-off-by: Ingo Molnar <mingo@elte.hu>

update the posix-cpu-timers code to use CFS's CPU accounting information.
Signed-off-by: Ingo Molnar <mingo@elte.hu>

add rq_clock()/__rq_clock(), a robust wrapper around sched_clock(),
used by CFS. It protects against common type of sched_clock() problems
(caused by hardware): time warps forwards and backwards.
Signed-off-by: Ingo Molnar <mingo@elte.hu>

add the CFS rq data types to sched.c.

(the old scheduler fields are still intact, they are removed
 by a later patch)
Signed-off-by: Ingo Molnar <mingo@elte.hu>

add the CFS data types to sched.h.

(the old scheduler is still fully intact.)
Signed-off-by: Ingo Molnar <mingo@elte.hu>

add kernel/sched_idletask.c - which implements the idle thread
scheduling class. This further simplifies sched.c (under CFS),
for example a number of 'if (p == rq->idle)' type of special-cases
can be removed from sched.c, and schedule() gets simpler too.
Signed-off-by: Ingo Molnar <mingo@elte.hu>

add kernel/sched_rt.c: SCHED_FIFO/SCHED_RR support. The behavior
and semantics of SCHED_FIFO/SCHED_RR tasks is unchanged.
Signed-off-by: Ingo Molnar <mingo@elte.hu>

add kernel/sched_fair.c - which implements the bulk of CFS's
behavioral changes for SCHED_OTHER tasks.

see Documentation/sched-design-CFS.txt about details.

Authors:

 Ingo Molnar <mingo@elte.hu>
 Dmitry Adamushko <dmitry.adamushko@gmail.com>
 Srivatsa Vaddagiri <vatsa@linux.vnet.ibm.com>
 Mike Galbraith <efault@gmx.de>
Signed-off-by: Ingo Molnar <mingo@elte.hu>
Signed-off-by: Mike Galbraith <efault@gmx.de>
Signed-off-by: Dmitry Adamushko <dmitry.adamushko@gmail.com>
Signed-off-by: Srivatsa Vaddagiri <vatsa@linux.vnet.ibm.com>

increase SMP-nice's resolution. This is needed by CFS to
implement SCHED_IDLE and cleaned up nice level support.

no behavioral changes.
Signed-off-by: Ingo Molnar <mingo@elte.hu>

create sched_stats.h and move sched.c schedstats code into it.
This cleans up sched.c a bit.

no code changes are caused by this patch.
Signed-off-by: Ingo Molnar <mingo@elte.hu>