Update the device tree binding documentation to include the wlf,out-mono
property that is used to specify whether each output is a mono or stereo
output.
Signed-off-by: Charles Keepax <ckeepax@opensource.wolfsonmicro.com>
Acked-by: Rob Herring <robh@kernel.org>
Signed-off-by: Lee Jones <lee.jones@linaro.org>

Add device tree bindings to support specifying outputs from the chip as
mono outputs. Whilst we are doing it change the out_mono pdata from a
bool to an int, because Sparse gets upset about using ARRAY_SIZE on
bools.
Signed-off-by: Charles Keepax <ckeepax@opensource.wolfsonmicro.com>
Signed-off-by: Lee Jones <lee.jones@linaro.org>

Allow support for the DA9063 PMIC to be modular, cfr. DA9062, which
allows to decrease size of multi-platform kernels (e.g.
multi_v7_defconfig).
Signed-off-by: Geert Uytterhoeven <geert+renesas@glider.be>
Acked-by: Steve Twiss <stwiss.opensource@diasemi.com>
Signed-off-by: Lee Jones <lee.jones@linaro.org>

Currently the driver requests the boot done and control interface IRQs
before it has requested its own IRQ line. This can cause problems on
edge triggered IRQ systems as if an edge occurs before the parent IRQ is
enabled it will be missed. Whilst we are changing the error handling
remove an unused label as well.
Signed-off-by: Charles Keepax <ckeepax@opensource.wolfsonmicro.com>
Signed-off-by: Lee Jones <lee.jones@linaro.org>

The CS47L24 and WM1831 codecs only have two GPIO lines, but are
otherwise similar to the WM8280.
Signed-off-by: Richard Fitzgerald <rf@opensource.wolfsonmicro.com>
Acked-by: Linus Walleij <linus.walleij@linaro.org>
Signed-off-by: Lee Jones <lee.jones@linaro.org>

The newly introduced as3722_i2c_suspend/resume functions are built
unconditionally, but only used when power management is enabled,
so we get a warning otherwise:

drivers/mfd/as3722.c:427:12: warning: 'as3722_i2c_suspend' defined but not used [-Wunused-function]
drivers/mfd/as3722.c:438:12: warning: 'as3722_i2c_resume' defined but not used [-Wunused-function]

This marks them both as __maybe_unused, which avoids an ugly #ifdef
and gives us best compile-time coverage. When they are unused, the
compiler will silently drop the functions from its output.
Signed-off-by: Arnd Bergmann <arnd@arndb.de>
Fixes: 35deff7e ("mfd: as3722: Handle interrupts on suspend")
Signed-off-by: Lee Jones <lee.jones@linaro.org>

These new helpers simplify implementing multi-driver modules and
properly handle failure to register one driver by unregistering all
previously registered drivers.
Signed-off-by: Thierry Reding <treding@nvidia.com>
Signed-off-by: Lee Jones <lee.jones@linaro.org>

Revision ID registers are available only on devices with
Slave IDs that are even, so don't make access to unavailable
registers.
Signed-off-by: Ivan T. Ivanov <ivan.ivanov@linaro.org>
[sboyd@codeaurora.org: Consider all slave ids that are even]
Signed-off-by: Stephen Boyd <sboyd@codeaurora.org>
Signed-off-by: Lee Jones <lee.jones@linaro.org>

wm831x_unique_id_show currently displays an interesting pattern of '0'
and '3' characters which isn't very useful (figuring out why is left
as an exercise for the reader). Presumably "buf[i]" should have been
"id[i] & 0xff".

But while there, it is much simpler to simply use %phN and do all the
formatting at once.
Signed-off-by: Rasmus Villemoes <linux@rasmusvillemoes.dk>
Acked-by: Charles Keepax <ckeepax@opensource.wolfsonmicro.com>
Signed-off-by: Lee Jones <lee.jones@linaro.org>

Currently syscon has a fixed configuration of 32 bits for register and
values widths. In some cases, it would be desirable to be able to
customize the value width.

For example, certain boards (like the ones manufactured by Technologic
Systems) have a FPGA that is memory-mapped, but its registers are only
16-bit wide.

This patch adds an optional "reg-io-width" DT binding for syscon that
allows to change the width for the data bus (i.e. val_bits). If this
property is provided, it will also set the register stride to
reg-io-width's value. If not provided, the default configuration is
used.
Signed-off-by: Damien Riegel <damien.riegel@savoirfairelinux.com>
Acked-by: Rob Herring <robh@kernel.org>
Acked-by: Arnd Bergmann <arnd@arndb.de>
Signed-off-by: Lee Jones <lee.jones@linaro.org>

Signed-off-by: Charles Keepax <ckeepax@opensource.wolfsonmicro.com>
Signed-off-by: Lee Jones <lee.jones@linaro.org>

cros_ec_cmd_xfer_spi and cros_ec_pkt_xfer_spi generally work like
this:
 - Pull CS down (active), wait a bit, then send a command
 - Wait for response (multiple requests)
 - Wait a while, pull CS up (inactive)

These operations, individually, lock the SPI bus, but there is
nothing preventing the SPI framework from interleaving messages
intended for other devices as the bus is unlocked in between.

This is a problem as the EC expects CS to be held low for the
whole duration.

Solution: Lock the SPI bus during the whole transaction, to make
sure that no other messages can be interleaved.
Signed-off-by: Nicolas Boichat <drinkcat@chromium.org>
Reviewed-by: Gwendal Grignou <gwendal@chromium.org>
Signed-off-by: Lee Jones <lee.jones@linaro.org>

The CS47L24 source is part of the ex-Wolfson "Arizona" group of
drivers. Not all cs47lxx devices are part of the Arizona driver
group so the cs47l24 is explicitly listed by its full part number.
Signed-off-by: Richard Fitzgerald <rf@opensource.wolfsonmicro.com>
Signed-off-by: Lee Jones <lee.jones@linaro.org>

This patch adds the regmap configuration tables and
core MFD handling for the CS47L24 and WM1831 codecs.

Note that compared to the other Arizona codecs, these devices
do not have an LDO1 or micsupp regulators, extcon driver, or
the DCVDD isolation control.
Signed-off-by: Richard Fitzgerald <rf@opensource.wolfsonmicro.com>
Signed-off-by: Lee Jones <lee.jones@linaro.org>

This updates the Arizona MFD device tree bindings to add the
Cirrus Logic CS47L24 and WM1831 codecs. Note that unlike all the
other codecs the DCVDD-supply and MICVDD-supply are mandatory.
Signed-off-by: Richard Fitzgerald <rf@opensource.wolfsonmicro.com>
Signed-off-by: Lee Jones <lee.jones@linaro.org>

of_match_device could return NULL, and so cause a NULL pointer
dereference later.
Signed-off-by: LABBE Corentin <clabbe.montjoie@gmail.com>
Signed-off-by: Lee Jones <lee.jones@linaro.org>

The platforms that use the AB8500 define all IRQ resources in the
device tree and they are automatically populated by matching
the .of_compatible string. These static resources are just
surplus baggage these days.
Signed-off-by: Linus Walleij <linus.walleij@linaro.org>
Signed-off-by: Lee Jones <lee.jones@linaro.org>

The da903x_chip_ops structure is never modified, so declare it
as const.

Done with the help of Coccinelle.
Signed-off-by: Julia Lawall <Julia.Lawall@lip6.fr>
Signed-off-by: Lee Jones <lee.jones@linaro.org>

For checking if a property is present or not,
use of_property_read_bool instead of of_get_property()
Signed-off-by: Saurabh Sengar <saurabh.truth@gmail.com>
Signed-off-by: Lee Jones <lee.jones@linaro.org>

The as3722 device is registered as an irqchip and the as3722-rtc interrupt
is one of it's interrupt sources. When using the as3722-rtc as a wake-up
device from suspend, the following is seen:

  PM: Syncing filesystems ... done.
  Freezing user space processes ... (elapsed 0.001 seconds) done.
  Freezing remaining freezable tasks ... (elapsed 0.001 seconds) done.
  Suspending console(s) (use no_console_suspend to debug)
  PM: suspend of devices complete after 161.119 msecs
  PM: late suspend of devices complete after 1.048 msecs
  PM: noirq suspend of devices complete after 0.756 msecs
  Disabling non-boot CPUs ...
  CPU1: shutdown
  CPU2: shutdown
  CPU3: shutdown
  Entering suspend state LP1
  Enabling non-boot CPUs ...
  CPU1 is up
  CPU2 is up
  CPU3 is up
  PM: noirq resume of devices complete after 0.487 msecs
  as3722 4-0040: Failed to read IRQ status: -16
  as3722 4-0040: Failed to read IRQ status: -16
  as3722 4-0040: Failed to read IRQ status: -16
  as3722 4-0040: Failed to read IRQ status: -16
  ...

The reason why the as3722 interrupt status cannot be read is because the
as3722 interrupt is not masked during suspend and when the as3722-rtc
interrupt occurs, to wake-up the device, the interrupt is seen before the
i2c controller has been resumed in order to read the as3722 interrupt
status.

The as3722-rtc driver sets it's interrupt as a wake-up source during
suspend, which gets propagated to the parent as3722 interrupt. However,
the as3722-rtc driver cannot disable it's interrupt during suspend
otherwise we would never be woken up and so the as3722 must disable it's
interrupt instead.

Fix this by disabling the as3722 interrupt during suspend. To ensure that
a wake-up event from the as3722 is not missing, enable the as3722 interrupt
as a wake-up source before disabling the interrupt on entering suspend.
Signed-off-by: Jon Hunter <jonathanh@nvidia.com>
Signed-off-by: Lee Jones <lee.jones@linaro.org>

The MFD part of wm8994 consists of three files wm8994-core.c,
wm8994-irq.c and wm8994-regmap.c only wm8994-core.c has a
MODULE_DESCRIPTION / LICENSE. These were clearly intended to be built
as a single module, but currently are not. This will lead to a tainted
kernel when loading modules for wm8894-irq.c and wm8994-regmap.c because
are missing a license.

This patch fixes this issue by grouping the three files together into a
single module.
Reported-by: Peter Robinson <pbrobinson@gmail.com>
Signed-off-by: Charles Keepax <ckeepax@opensource.wolfsonmicro.com>
Signed-off-by: Lee Jones <lee.jones@linaro.org>

Fix the incorrect interrupt documentation file path in binding docs.
Signed-off-by: Andrew F. Davis <afd@ti.com>
Signed-off-by: Lee Jones <lee.jones@linaro.org>

Currently S2MPSXX multifunction device is named as *-pmic,
and these MFDs also supports regulator as a one of its MFD cell which
has the same name, because current name is confusing and we want to
sort it out.

We did discussed different approaches about how the MFD and it
cells need to be named here [1].
Based in the discussion this patch rename MFD regulator name as
*-regulator instead of current *-pmic.

This patch also changes the corresponding entries in the regulator driver
to keep git-bisect happy.

[1]-> https://lkml.org/lkml/2015/10/28/417Suggested-by: Lee Jones <lee.jones@linaro.org>
Signed-off-by: Alim Akhtar <alim.akhtar@samsung.com>
Reviewed-by: Krzysztof Kozlowski <k.kozlowski@samsung.com>
Acked-by: Mark Brown <broonie@kernel.org>
Signed-off-by: Lee Jones <lee.jones@linaro.org>

Adding Intel codename Lewisburg platform device IDs for PCH.
Signed-off-by: Alexandra Yates <alexandra.yates@linux.intel.com>
Reviewed-by: Andy Shevchenko <andy.shevchenko@gmail.com>
Signed-off-by: Lee Jones <lee.jones@linaro.org>

RTC found in s2mps15 is almost same as one found on s2mps13
with few differences in RTC_UPDATE register fields, like:
1> Bit[4] and Bit[1] are reversed
   - On s2mps13
          WUDR -> bit[4], AUDR -> bit[1]
   - On s2mps15
	  WUDR -> bit[1], AUDR -> bit[4]
2> In case of s2mps13, for alarm register, need to set both
   WDUR and ADUR high, whereas for s2mps15 only set AUDR to high.
3> On s2mps15, WUDR, RUDR and AUDR functions should never be used
   at the same time.

This patch add required changes to enable s2mps15 rtc timer.
Signed-off-by: Alim Akhtar <alim.akhtar@samsung.com>
Reviewed-by: Krzysztof Kozlowski <k.kozlowski@samsung.com>
Acked-by: Alexandre Belloni <alexandre.belloni@free-electrons.com>
Signed-off-by: Lee Jones <lee.jones@linaro.org>

The S2MPS15 PMIC is similar in functionality to S2MPS11/14 PMIC. It contains
27 LDO and 10 Buck regulators and allows programming these regulators via a
I2C interface. This patch adds initial support for LDO/Buck regulators of
S2MPS15 PMIC.
Signed-off-by: Thomas Abraham <thomas.ab@samsung.com>
Signed-off-by: Alim Akhtar <alim.akhtar@samsung.com>
Reviewed-by: Krzysztof Kozlowski <k.kozlowski@samsung.com>
Acked-by: Mark Brown <broonie@kernel.org>
Signed-off-by: Lee Jones <lee.jones@linaro.org>

Add support for S2MPS15 PMIC which is similar to S2MPS11 PMIC. The S2MPS15
PMIC supports 27 LDO regulators, 10 buck regulators, RTC, three 32.768KHz
clock outputs and battery charger. This patch adds initial support for
LDO and buck regulators of S2MPS15 device.
Signed-off-by: Thomas Abraham <thomas.ab@samsung.com>
Signed-off-by: Alim Akhtar <alim.akhtar@samsung.com>
Reviewed-by: Krzysztof Kozlowski <k.kozlowski@samsung.com>
[Alim: Added s2mps15_devs like rtc and clk and related changes]
Signed-off-by: Lee Jones <lee.jones@linaro.org>

Merge slub bulk allocator updates from Andrew Morton:
 "This missed the merge window because I was waiting for some repairs to
  come in.  Nothing actually uses the bulk allocator yet and the changes
  to other code paths are pretty small.  And the net guys are waiting
  for this so they can start merging the client code"

More comments from Jesper Dangaard Brouer:
 "The kmem_cache_alloc_bulk() call, in mm/slub.c, were included in
  previous kernel.  The present version contains a bug.  Vladimir
  Davydov noticed it contained a bug, when kernel is compiled with
  CONFIG_MEMCG_KMEM (see commit 03ec0ed5: "slub: fix kmem cgroup
  bug in kmem_cache_alloc_bulk").  Plus the mem cgroup counterpart in
  kmem_cache_free_bulk() were missing (see commit 03374518 "slub:
  add missing kmem cgroup support to kmem_cache_free_bulk").

  I don't consider the fix stable-material because there are no in-tree
  users of the API.

  But with known bugs (for memcg) I cannot start using the API in the
  net-tree"

* emailed patches from Andrew Morton <akpm@linux-foundation.org>:
  slab/slub: adjust kmem_cache_alloc_bulk API
  slub: add missing kmem cgroup support to kmem_cache_free_bulk
  slub: fix kmem cgroup bug in kmem_cache_alloc_bulk
  slub: optimize bulk slowpath free by detached freelist
  slub: support for bulk free with SLUB freelists

Pull tty/serial fixes from Greg KH:
 "Here are a few small tty/serial driver fixes for 4.4-rc2 that resolve
  some reported problems.

  All have been in linux-next, full details are in the shortlog below"

* tag 'tty-4.4-rc2' of git://git.kernel.org/pub/scm/linux/kernel/git/gregkh/tty:
  serial: export fsl8250_handle_irq
  serial: 8250_mid: Add missing dependency
  tty: audit: Fix audit source
  serial: etraxfs-uart: Fix crash
  serial: fsl_lpuart: Fix earlycon support
  bcm63xx_uart: Use the device name when registering an interrupt
  tty: Fix direct use of tty buffer work
  tty: Fix tty_send_xchar() lock order inversion

Pull staging/IIO fixes from Greg KH:
 "Here are some staging and iio driver fixes for 4.4-rc2.  All of these
  are in response to issues that have been reported and have been in
  linux-next for a while"

* tag 'staging-4.4-rc2' of git://git.kernel.org/pub/scm/linux/kernel/git/gregkh/staging:
  Revert "Staging: wilc1000: coreconfigurator: Drop unneeded wrapper functions"
  iio: adc: xilinx: Fix VREFN scale
  iio: si7020: Swap data byte order
  iio: adc: vf610_adc: Fix division by zero error
  iio:ad7793: Fix ad7785 product ID
  iio: ad5064: Fix ad5629/ad5669 shift
  iio:ad5064: Make sure ad5064_i2c_write() returns 0 on success
  iio: lpc32xx_adc: fix warnings caused by enabling unprepared clock
  staging: iio: select IRQ_WORK for IIO_DUMMY_EVGEN
  vf610_adc: Fix internal temperature calculation

Pull USB fixes from Greg KH:
 "Here are a number of USB fixes and new device ids for 4.4-rc2.  All
  have been in linux-next and the details are in the shortlog"

* tag 'usb-4.4-rc2' of git://git.kernel.org/pub/scm/linux/kernel/git/gregkh/usb: (28 commits)
  usblp: do not set TASK_INTERRUPTIBLE before lock
  USB: MAINTAINERS: cxacru
  usb: kconfig: fix warning of select USB_OTG
  USB: option: add XS Stick W100-2 from 4G Systems
  xhci: Fix a race in usb2 LPM resume, blocking U3 for usb2 devices
  usb: xhci: fix checking ep busy for CFC
  xhci: Workaround to get Intel xHCI reset working more reliably
  usb: chipidea: imx: fix a possible NULL dereference
  usb: chipidea: usbmisc_imx: fix a possible NULL dereference
  usb: chipidea: otg: gadget module load and unload support
  usb: chipidea: debug: disable usb irq while role switch
  ARM: dts: imx27.dtsi: change the clock information for usb
  usb: chipidea: imx: refine clock operations to adapt for all platforms
  usb: gadget: atmel_usba_udc: Expose correct device speed
  usb: musb: enable usb_dma parameter
  usb: phy: phy-mxs-usb: fix a possible NULL dereference
  usb: dwc3: gadget: let us set lower max_speed
  usb: musb: fix tx fifo flush handling
  usb: gadget: f_loopback: fix the warning during the enumeration
  usb: dwc2: host: Fix remote wakeup when not in DWC2_L2
  ...

Pull MIPS fixes from Ralf Baechle:

 - Fix a flood of annoying build warnings

 - A number of fixes for Atheros 79xx platforms

* 'upstream' of git://git.linux-mips.org/pub/scm/ralf/upstream-linus:
  MIPS: ath79: Add a machine entry for booting OF machines
  MIPS: ath79: Fix the size of the MISC INTC registers in ar9132.dtsi
  MIPS: ath79: Fix the DDR control initialization on ar71xx and ar934x
  MIPS: Fix flood of warnings about comparsion being always true.

Pull parisc update from Helge Deller:
 "This patchset adds Huge Page and HUGETLBFS support for parisc"

Honestly, the hugepage support should have gone through in the merge
window, and is not really an rc-time fix.  But it only touches
arch/parisc, and I cannot find it in myself to care.  If one of the
three parisc users notices a breakage, I will point at Helge and make
rude farting noises.

* 'parisc-4.4-2' of git://git.kernel.org/pub/scm/linux/kernel/git/deller/parisc-linux:
  parisc: Map kernel text and data on huge pages
  parisc: Add Huge Page and HUGETLBFS support
  parisc: Use long branch to do_syscall_trace_exit
  parisc: Increase initial kernel mapping to 32MB on 64bit kernel
  parisc: Initialize the fault vector earlier in the boot process.
  parisc: Add defines for Huge page support
  parisc: Drop unused MADV_xxxK_PAGES flags from asm/mman.h
  parisc: Drop definition of start_thread_som for HP-UX SOM binaries
  parisc: Fix wrong comment regarding first pmd entry flags

Pull perf tool fixes from Thomas Gleixner:
 "A couple of fixes for perf tools:

   - Build system updates

   - Plug a memory leak in an error path of perf probe

   - Tear down probes correctly when adding fails

   - Fixes to the perf symbol handling

   - Fix ordering of event processing in buildid-list

   - Fix per DSO filtering in the histogram browser"

* 'perf-urgent-for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip:
  perf probe: Clear probe_trace_event when add_probe_trace_event() fails
  perf probe: Fix memory leaking on failure by clearing all probe_trace_events
  perf inject: Also re-pipe lost_samples event
  perf buildid-list: Requires ordered events
  perf symbols: Fix dso lookup by long name and missing buildids
  perf symbols: Allow forcing reading of non-root owned files by root
  perf hists browser: The dso can be obtained from popup_action->ms.map->dso
  perf hists browser: Fix 'd' hotkey action to filter by DSO
  perf symbols: Rebuild rbtree when adjusting symbols for kcore
  tools: Add a "make all" rule
  tools: Actually install tmon in the install rule

Pull x86 fixes from Thomas Gleixner:
 "This update contains:

   - MPX updates for handling 32bit processes

   - A fix for a long standing bug in 32bit signal frame handling
     related to FPU/XSAVE state

   - Handle get_xsave_addr() correctly in KVM

   - Fix SMAP check under paravirtualization

   - Add a comment to the static function trace entry to avoid further
     confusion about the difference to dynamic tracing"

* 'x86-urgent-for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip:
  x86/cpu: Fix SMAP check in PVOPS environments
  x86/ftrace: Add comment on static function tracing
  x86/fpu: Fix get_xsave_addr() behavior under virtualization
  x86/fpu: Fix 32-bit signal frame handling
  x86/mpx: Fix 32-bit address space calculation
  x86/mpx: Do proper get_user() when running 32-bit binaries on 64-bit kernels

Adjust kmem_cache_alloc_bulk API before we have any real users.

Adjust API to return type 'int' instead of previously type 'bool'.  This
is done to allow future extension of the bulk alloc API.

A future extension could be to allow SLUB to stop at a page boundary, when
specified by a flag, and then return the number of objects.

The advantage of this approach, would make it easier to make bulk alloc
run without local IRQs disabled.  With an approach of cmpxchg "stealing"
the entire c->freelist or page->freelist.  To avoid overshooting we would
stop processing at a slab-page boundary.  Else we always end up returning
some objects at the cost of another cmpxchg.

To keep compatible with future users of this API linking against an older
kernel when using the new flag, we need to return the number of allocated
objects with this API change.
Signed-off-by: Jesper Dangaard Brouer <brouer@redhat.com>
Cc: Vladimir Davydov <vdavydov@virtuozzo.com>
Acked-by: Christoph Lameter <cl@linux.com>
Cc: Pekka Enberg <penberg@kernel.org>
Cc: David Rientjes <rientjes@google.com>
Cc: Joonsoo Kim <iamjoonsoo.kim@lge.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>

Initial implementation missed support for kmem cgroup support in
kmem_cache_free_bulk() call, add this.

If CONFIG_MEMCG_KMEM is not enabled, the compiler should be smart enough
to not add any asm code.

Incoming bulk free objects can belong to different kmem cgroups, and
object free call can happen at a later point outside memcg context.  Thus,
we need to keep the orig kmem_cache, to correctly verify if a memcg object
match against its "root_cache" (s->memcg_params.root_cache).
Signed-off-by: Jesper Dangaard Brouer <brouer@redhat.com>
Reviewed-by: Vladimir Davydov <vdavydov@virtuozzo.com>
Cc: Christoph Lameter <cl@linux.com>
Cc: Pekka Enberg <penberg@kernel.org>
Cc: David Rientjes <rientjes@google.com>
Cc: Joonsoo Kim <iamjoonsoo.kim@lge.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>

The call slab_pre_alloc_hook() interacts with kmemgc and is not allowed to
be called several times inside the bulk alloc for loop, due to the call to
memcg_kmem_get_cache().

This would result in hitting the VM_BUG_ON in __memcg_kmem_get_cache.

As suggested by Vladimir Davydov, change slab_post_alloc_hook() to be able
to handle an array of objects.

A subtle detail is, loop iterator "i" in slab_post_alloc_hook() must have
same type (size_t) as size argument.  This helps the compiler to easier
realize that it can remove the loop, when all debug statements inside loop
evaluates to nothing.  Note, this is only an issue because the kernel is
compiled with GCC option: -fno-strict-overflow

In slab_alloc_node() the compiler inlines and optimizes the invocation of
slab_post_alloc_hook(s, flags, 1, &object) by removing the loop and access
object directly.
Signed-off-by: Jesper Dangaard Brouer <brouer@redhat.com>
Reported-by: Vladimir Davydov <vdavydov@virtuozzo.com>
Suggested-by: Vladimir Davydov <vdavydov@virtuozzo.com>
Reviewed-by: Vladimir Davydov <vdavydov@virtuozzo.com>
Cc: Christoph Lameter <cl@linux.com>
Cc: Pekka Enberg <penberg@kernel.org>
Cc: David Rientjes <rientjes@google.com>
Cc: Joonsoo Kim <iamjoonsoo.kim@lge.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>

This change focus on improving the speed of object freeing in the
"slowpath" of kmem_cache_free_bulk.

The calls slab_free (fastpath) and __slab_free (slowpath) have been
extended with support for bulk free, which amortize the overhead of
the (locked) cmpxchg_double.

To use the new bulking feature, we build what I call a detached
freelist.  The detached freelist takes advantage of three properties:

 1) the free function call owns the object that is about to be freed,
    thus writing into this memory is synchronization-free.

 2) many freelist's can co-exist side-by-side in the same slab-page
    each with a separate head pointer.

 3) it is the visibility of the head pointer that needs synchronization.

Given these properties, the brilliant part is that the detached
freelist can be constructed without any need for synchronization.  The
freelist is constructed directly in the page objects, without any
synchronization needed.  The detached freelist is allocated on the
stack of the function call kmem_cache_free_bulk.  Thus, the freelist
head pointer is not visible to other CPUs.

All objects in a SLUB freelist must belong to the same slab-page.
Thus, constructing the detached freelist is about matching objects
that belong to the same slab-page.  The bulk free array is scanned is
a progressive manor with a limited look-ahead facility.

Kmem debug support is handled in call of slab_free().

Notice kmem_cache_free_bulk no longer need to disable IRQs. This
only slowed down single free bulk with approx 3 cycles.

Performance data:
 Benchmarked[1] obj size 256 bytes on CPU i7-4790K @ 4.00GHz

SLUB fastpath single object quick reuse: 47 cycles(tsc) 11.931 ns

To get stable and comparable numbers, the kernel have been booted with
"slab_merge" (this also improve performance for larger bulk sizes).

Performance data, compared against fallback bulking:

bulk -  fallback bulk            - improvement with this patch
   1 -  62 cycles(tsc) 15.662 ns - 49 cycles(tsc) 12.407 ns- improved 21.0%
   2 -  55 cycles(tsc) 13.935 ns - 30 cycles(tsc) 7.506 ns - improved 45.5%
   3 -  53 cycles(tsc) 13.341 ns - 23 cycles(tsc) 5.865 ns - improved 56.6%
   4 -  52 cycles(tsc) 13.081 ns - 20 cycles(tsc) 5.048 ns - improved 61.5%
   8 -  50 cycles(tsc) 12.627 ns - 18 cycles(tsc) 4.659 ns - improved 64.0%
  16 -  49 cycles(tsc) 12.412 ns - 17 cycles(tsc) 4.495 ns - improved 65.3%
  30 -  49 cycles(tsc) 12.484 ns - 18 cycles(tsc) 4.533 ns - improved 63.3%
  32 -  50 cycles(tsc) 12.627 ns - 18 cycles(tsc) 4.707 ns - improved 64.0%
  34 -  96 cycles(tsc) 24.243 ns - 23 cycles(tsc) 5.976 ns - improved 76.0%
  48 -  83 cycles(tsc) 20.818 ns - 21 cycles(tsc) 5.329 ns - improved 74.7%
  64 -  74 cycles(tsc) 18.700 ns - 20 cycles(tsc) 5.127 ns - improved 73.0%
 128 -  90 cycles(tsc) 22.734 ns - 27 cycles(tsc) 6.833 ns - improved 70.0%
 158 -  99 cycles(tsc) 24.776 ns - 30 cycles(tsc) 7.583 ns - improved 69.7%
 250 - 104 cycles(tsc) 26.089 ns - 37 cycles(tsc) 9.280 ns - improved 64.4%

Performance data, compared current in-kernel bulking:

bulk - curr in-kernel  - improvement with this patch
   1 -  46 cycles(tsc) - 49 cycles(tsc) - improved (cycles:-3) -6.5%
   2 -  27 cycles(tsc) - 30 cycles(tsc) - improved (cycles:-3) -11.1%
   3 -  21 cycles(tsc) - 23 cycles(tsc) - improved (cycles:-2) -9.5%
   4 -  18 cycles(tsc) - 20 cycles(tsc) - improved (cycles:-2) -11.1%
   8 -  17 cycles(tsc) - 18 cycles(tsc) - improved (cycles:-1) -5.9%
  16 -  18 cycles(tsc) - 17 cycles(tsc) - improved (cycles: 1)  5.6%
  30 -  18 cycles(tsc) - 18 cycles(tsc) - improved (cycles: 0)  0.0%
  32 -  18 cycles(tsc) - 18 cycles(tsc) - improved (cycles: 0)  0.0%
  34 -  78 cycles(tsc) - 23 cycles(tsc) - improved (cycles:55) 70.5%
  48 -  60 cycles(tsc) - 21 cycles(tsc) - improved (cycles:39) 65.0%
  64 -  49 cycles(tsc) - 20 cycles(tsc) - improved (cycles:29) 59.2%
 128 -  69 cycles(tsc) - 27 cycles(tsc) - improved (cycles:42) 60.9%
 158 -  79 cycles(tsc) - 30 cycles(tsc) - improved (cycles:49) 62.0%
 250 -  86 cycles(tsc) - 37 cycles(tsc) - improved (cycles:49) 57.0%

Performance with normal SLUB merging is significantly slower for
larger bulking.  This is believed to (primarily) be an effect of not
having to share the per-CPU data-structures, as tuning per-CPU size
can achieve similar performance.

bulk - slab_nomerge   -  normal SLUB merge
   1 -  49 cycles(tsc) - 49 cycles(tsc) - merge slower with cycles:0
   2 -  30 cycles(tsc) - 30 cycles(tsc) - merge slower with cycles:0
   3 -  23 cycles(tsc) - 23 cycles(tsc) - merge slower with cycles:0
   4 -  20 cycles(tsc) - 20 cycles(tsc) - merge slower with cycles:0
   8 -  18 cycles(tsc) - 18 cycles(tsc) - merge slower with cycles:0
  16 -  17 cycles(tsc) - 17 cycles(tsc) - merge slower with cycles:0
  30 -  18 cycles(tsc) - 23 cycles(tsc) - merge slower with cycles:5
  32 -  18 cycles(tsc) - 22 cycles(tsc) - merge slower with cycles:4
  34 -  23 cycles(tsc) - 22 cycles(tsc) - merge slower with cycles:-1
  48 -  21 cycles(tsc) - 22 cycles(tsc) - merge slower with cycles:1
  64 -  20 cycles(tsc) - 48 cycles(tsc) - merge slower with cycles:28
 128 -  27 cycles(tsc) - 57 cycles(tsc) - merge slower with cycles:30
 158 -  30 cycles(tsc) - 59 cycles(tsc) - merge slower with cycles:29
 250 -  37 cycles(tsc) - 56 cycles(tsc) - merge slower with cycles:19

Joint work with Alexander Duyck.

[1] https://github.com/netoptimizer/prototype-kernel/blob/master/kernel/mm/slab_bulk_test01.c

[akpm@linux-foundation.org: BUG_ON -> WARN_ON;return]
Signed-off-by: Jesper Dangaard Brouer <brouer@redhat.com>
Signed-off-by: Alexander Duyck <alexander.h.duyck@redhat.com>
Acked-by: Christoph Lameter <cl@linux.com>
Cc: Pekka Enberg <penberg@kernel.org>
Cc: David Rientjes <rientjes@google.com>
Cc: Joonsoo Kim <iamjoonsoo.kim@lge.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>