This makes it easier to trace what raid5 is doing.
Signed-off-by: NeilBrown <neilb@suse.de>

handle_stripe_expansion contains:

        if (tx) {
                async_tx_ack(tx);
                dma_wait_for_async_tx(tx);
        }

which is very similar to the body of async_tx_quiesce(),
except that the later handles an error from dma_wait_for_async_tx()
(admittedly by panicing, but that decision belongs in the dma
code, not the md code).

So just us async_tx_quiesce().
Acked-by: Dan Williams <djbw@fb.com>
Reported-by: Bartlomiej Zolnierkiewicz <b.zolnierkie@samsung.com>
Signed-off-by: NeilBrown <neilb@suse.de>

If a resync is aborted cleanly, ->curr_resync is a reliable
record of where we got up to.
If there was an error it is less reliable but we always know that
->curr_resync_completed is safe.

So add a flag MD_RECOVERY_ERROR to differentiate between these cases
and set recovery_cp accordingly.
Signed-off-by: Jianpeng Ma <majianpeng@gmail.com>
Signed-off-by: NeilBrown <neilb@suse.de>

sse and avx2 stuff only exist on x86 arch, and we don't need to build
altivec on x86. And we can do that at lib/raid6/Makefile.
Proposed-by: H. Peter Anvin <hpa@zytor.com>
Signed-off-by: Yuanhan Liu <yuanhan.liu@linux.intel.com>
Reviewed-by: H. Peter Anvin <hpa@zytor.com>
Signed-off-by: Jim Kukunas <james.t.kukunas@linux.intel.com>
Signed-off-by: NeilBrown <neilb@suse.de>

Add AVX2 optimized gen_syndrom functions, which is simply based on
sse2.c written by hpa.
Signed-off-by: Yuanhan Liu <yuanhan.liu@linux.intel.com>
Reviewed-by: H. Peter Anvin <hpa@zytor.com>
Signed-off-by: Jim Kukunas <james.t.kukunas@linux.intel.com>
Signed-off-by: NeilBrown <neilb@suse.de>

Optimize RAID6 recovery functions to take advantage of
the 256-bit YMM integer instructions introduced in AVX2.

The patch was tested and benchmarked before submission.
However hardware is not yet released so benchmark numbers
cannot be reported.
Acked-by: "H. Peter Anvin" <hpa@zytor.com>
Signed-off-by: Jim Kukunas <james.t.kukunas@linux.intel.com>
Signed-off-by: NeilBrown <neilb@suse.de>

md will current only only checkpoint recovery or resync ever 1/16th
of the device size.  As devices get larger this can become a long time
an so a lot of work that might need to be duplicated after a shutdown.

So add a time-based checkpoint.  Every 5 minutes limits the amount of
duplicated effort to at most 5 minutes, and has almost zero impact on
performance.

[changelog entry re-written by NeilBrown]
Signed-off-by: Jianpeng Ma <majianpeng@gmail.com>
Signed-off-by: NeilBrown <neilb@suse.de>

In resyncing, recovery_cp only updated when resync aborted or completed.
But in md drives,many place used it to judge.So add a place to update.
Signed-off-by: Jianpeng Ma <majianpeng@gmail.com>
Signed-off-by: NeilBrown <neilb@suse.de>

Intent was unnecessarily deep.

Also change one 'switch' which has a single case element, into an
'if'.
Signed-off-by: NeilBrown <neilb@suse.de>

When we remove a device from an md array, the final removal of
the "dev-XX" sys entry is run asynchronously.
If we then re-add that device immediately before the worker thread
gets to run, we can end up trying to add the "dev-XX" sysfs entry back
before it has been removed.

So in both places where we add a device, call
  flush_workqueue(md_misc_wq);
before taking the md lock (as holding the md lock can prevent removal
to complete).
Signed-off-by: NeilBrown <neilb@suse.de>

'i' is unused.

NeilBrown <neilb@suse.de>

If the raid1 or raid10 unplug function gets called
from a make_request function (which is very possible) when
there are bios on the current->bio_list list, then it will not
be able to successfully call bitmap_unplug() and it could
need to submit more bios and wait for them to complete.
But they won't complete while current->bio_list is non-empty.

So detect that case and handle the unplugging off to another thread
just like we already do when called from within the scheduler.

RAID1 version of bug was introduced in 3.6, so that part of fix is
suitable for 3.6.y.  RAID10 part won't apply.

Cc: stable@vger.kernel.org
Reported-by: Torsten Kaiser <just.for.lkml@googlemail.com>
Reported-by: Peter Maloney <peter.maloney@brockmann-consult.de>
Signed-off-by: NeilBrown <neilb@suse.de>

When a write to a replacement device completes, we carefully
and correctly found the rdev that the write actually went to
and the blithely called rdev_dec_pending on the primary rdev,
even if this write was to the replacement.

This means that any writes to an array while a replacement
was ongoing would cause the nr_pending count for the primary
device to go negative, so it could never be removed.

This bug has been present since replacement was introduced in
3.3, so it is suitable for any -stable kernel since then.
Reported-by: "George Spelvin" <linux@horizon.com>
Cc: stable@vger.kernel.org
Signed-off-by: NeilBrown <neilb@suse.de>

When a replacement operation completes there is a small window
when the original device is marked 'faulty' and the replacement
still looks like a replacement.  The faulty should be removed and
the replacement moved in place very quickly, bit it isn't instant.

So the code write out to the array must handle the possibility that
the only working device for some slot in the replacement - but it
doesn't.  If the primary device is faulty it just gives up.  This
can lead to corruption.

So make the code more robust: if either  the primary or the
replacement is present and working, write to them.  Only when
neither are present do we give up.

This bug has been present since replacement was introduced in
3.3, so it is suitable for any -stable kernel since then.
Reported-by: "George Spelvin" <linux@horizon.com>
Cc: stable@vger.kernel.org
Signed-off-by: NeilBrown <neilb@suse.de>

commit 9e444768
    MD: raid5 avoid unnecessary zero page for trim

change raid5 to clear R5_Discard when the complete request is
handled rather than when submitting the per-device discard request.
However it did not clear R5_Discard for the parity device.

This means that if the stripe_head was reused before it expired from
the cache, the setting would be wrong and a hang would result.

Also if the R5_Uptodate bit happens to be set, R5_Discard again
won't be cleared.  But R5_Uptodate really should be clear at this point.

So make sure R5_Discard is cleared in all cases, and clear
R5_Uptodate when a 'discard' completes.
Signed-off-by: NeilBrown <neilb@suse.de>

stripe_handle.

The chunk of code in stripe_handle which responds to a
*_result value in reconstruct_state is really the completion
of some processing that happened outside of handle_stripe
(possibly asynchronously) and so should be one of the first
things done in handle_stripe().

After the next patch it will be important that it happens before
handle_stripe_clean_event(), as that will clear some dev->flags
bit that this code tests.
Signed-off-by: NeilBrown <neilb@suse.de>

blkdev_issue_discard currently assumes that the granularity
is a power of 2.  So in raid5, round the chosen number up to
avoid embarrassment.

Cc: Shaohua Li <shli@kernel.org>
Signed-off-by: NeilBrown <neilb@suse.de>

md_stop() would stop an array, but not free various attached
data structures.
For internal arrays, these are freed later in do_md_stop() or
mddev_put(), but they don't apply for dm-raid arrays.
So get md_stop() to free them, and only all it from dm-raid.
For internal arrays we now call __md_stop.
Reported-by: majianpeng <majianpeng@gmail.com>
Signed-off-by: NeilBrown <neilb@suse.de>

If read_seqretry returned true and bbp was changed, it will write
invalid address which can cause some serious problem.

This bug was introduced by commit v3.0-rc7-130-g2699b672.
So fix is suitable for 3.0.y thru 3.6.y.

Reported-by: zhuwenfeng@kedacom.com
Tested-by: zhuwenfeng@kedacom.com
Cc: stable@vger.kernel.org
Signed-off-by: Jianpeng Ma <majianpeng@gmail.com>
Signed-off-by: NeilBrown <neilb@suse.de>

This bug was introduced by commit(v3.0-rc7-126-g2230dfe).
So fix is suitable for 3.0.y thru 3.6.y.

Cc: stable@vger.kernel.org
Signed-off-by: Jianpeng Ma <majianpeng@gmail.com>
Signed-off-by: NeilBrown <neilb@suse.de>

Commit 2863b9eb didn't take into account the changes to add TRIM support to
RAID10 (commit 532a2a3f).  That is, when using dm-raid.c to create the
RAID10 arrays, there is no mddev->gendisk or mddev->queue.  The code added
to support TRIM simply assumes that mddev->queue is available without
checking.  The result is an oops any time dm-raid.c attempts to create a
RAID10 device.
Signed-off-by: Jonathan Brassow <jbrassow@redhat.com>
Signed-off-by: NeilBrown <neilb@suse.de>

setup_conf in raid1.c uses conf->raid_disks before assigning
a value.  It is used when including 'Replacement' devices.

The consequence is that assembling an array which contains a
replacement will misbehave and either not include the replacement, or
not include the device being replaced.

Though this doesn't lead directly to data corruption, it could lead to
reduced data safety.

So use mddev->raid_disks, which is initialised, instead.

Bug was introduced by commit c19d5798
      md/raid1: recognise replacements when assembling arrays.

in 3.3, so fix is suitable for 3.3.y thru 3.6.y.

Cc: stable@vger.kernel.org
Signed-off-by: NeilBrown <neilb@suse.de>

in:
fe86cdce block: do not artificially constrain max_sectors for stacking drivers

max_sectors defaults to UINT_MAX.  md faulty wasn't using
disk_stack_limits(), so inherited this large value as well.
This triggered a bug in XFS when stressed over md_faulty, when
a very large bio_alloc() failed.

That was on an older kernel, and I can't reproduce exactly the
same thing upstream, but I think the fix is appropriate in any
case.

Thanks to Mike Snitzer for pointing out the problem.
Signed-off-by: Eric Sandeen <sandeen@redhat.com>
Signed-off-by: NeilBrown <neilb@suse.de>

Merge tag 'disintegrate-raid-20121009' of git://git.infradead.org/users/dhowells/linux-headers into for-next

UAPI Disintegration 2012-10-09

Pull MIPS update from Ralf Baechle:
 "Cleanups and fixes for breakage that occured earlier during this merge
  phase.  Also a few patches that didn't make the first pull request.
  Of those is the Alchemy work that merges code for many of the SOCs and
  evaluation boards thus among other code shrinkage, reduces the number
  of MIPS defconfigs by 5."

* 'upstream' of git://git.linux-mips.org/pub/scm/ralf/upstream-linus: (22 commits)
  MIPS: SNI: Switch RM400 serial to SCCNXP driver
  MIPS: Remove unused empty_bad_pmd_table[] declaration.
  MIPS: MT: Remove kspd.
  MIPS: Malta: Fix section mismatch.
  MIPS: asm-offset.c: Delete unused irq_cpustat_t struct offsets.
  MIPS: Alchemy: Merge PB1100/1500 support into DB1000 code.
  MIPS: Alchemy: merge PB1550 support into DB1550 code
  MIPS: Alchemy: Single kernel for DB1200/1300/1550
  MIPS: Optimize TLB refill for RI/XI configurations.
  MIPS: proc: Cleanup printing of ASEs.
  MIPS: Hardwire detection of DSP ASE Rev 2 for systems, as required.
  MIPS: Add detection of DSP ASE Revision 2.
  MIPS: Optimize pgd_init and pmd_init
  MIPS: perf: Add perf functionality for BMIPS5000
  MIPS: perf: Split the Kconfig option CONFIG_MIPS_MT_SMP
  MIPS: perf: Remove unnecessary #ifdef
  MIPS: perf: Add cpu feature bit for PCI (performance counter interrupt)
  MIPS: perf: Change the "mips_perf_event" table unsupported indicator.
  MIPS: Align swapper_pg_dir to 64K for better TLB Refill code.
  vmlinux.lds.h: Allow architectures to add sections to the front of .bss
  ...

Pull module signing support from Rusty Russell:
 "module signing is the highlight, but it's an all-over David Howells frenzy..."

Hmm "Magrathea: Glacier signing key". Somebody has been reading too much HHGTTG.

* 'modules-next' of git://git.kernel.org/pub/scm/linux/kernel/git/rusty/linux: (37 commits)
  X.509: Fix indefinite length element skip error handling
  X.509: Convert some printk calls to pr_devel
  asymmetric keys: fix printk format warning
  MODSIGN: Fix 32-bit overflow in X.509 certificate validity date checking
  MODSIGN: Make mrproper should remove generated files.
  MODSIGN: Use utf8 strings in signer's name in autogenerated X.509 certs
  MODSIGN: Use the same digest for the autogen key sig as for the module sig
  MODSIGN: Sign modules during the build process
  MODSIGN: Provide a script for generating a key ID from an X.509 cert
  MODSIGN: Implement module signature checking
  MODSIGN: Provide module signing public keys to the kernel
  MODSIGN: Automatically generate module signing keys if missing
  MODSIGN: Provide Kconfig options
  MODSIGN: Provide gitignore and make clean rules for extra files
  MODSIGN: Add FIPS policy
  module: signature checking hook
  X.509: Add a crypto key parser for binary (DER) X.509 certificates
  MPILIB: Provide a function to read raw data into an MPI
  X.509: Add an ASN.1 decoder
  X.509: Add simple ASN.1 grammar compiler
  ...

The hostprogs need access to the CONFIG_* symbols found in
include/generated/autoconf.h.  But commit abbf1590 ("UAPI: Partition
the header include path sets and add uapi/ header directories") replaced
$(LINUXINCLUDE) with $(USERINCLUDE) which doesn't contain the necessary
include paths.

This has the undesirable effect of breaking the EFI boot stub because
the #ifdef CONFIG_EFI_STUB code in arch/x86/boot/tools/build.c is
never compiled.

It should also be noted that because $(USERINCLUDE) isn't exported by
the top-level Makefile it's actually empty in arch/x86/boot/Makefile.

Cc: H. Peter Anvin <hpa@zytor.com>
Cc: Ingo Molnar <mingo@kernel.org>
Acked-by: David Howells <dhowells@redhat.com>
Signed-off-by: Matt Fleming <matt.fleming@intel.com>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>

The UAPI commits forgot to test tooling builds such as tools/perf/,
and this fixes the fallout.

Manual conversion.
Signed-off-by: Ingo Molnar <mingo@kernel.org>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>

Pull ARM update from Russell King:
 "This is the final round of stuff for ARM, left until the end of the
  merge window to reduce the number of conflicts.  This set contains the
  ARM part of David Howells UAPI changes, and a fix to the ordering of
  'select' statements in ARM Kconfig files (see the appropriate commit
  for why this happened - thanks to Andrew Morton for pointing out the
  problem.)

  I've left this as long as I dare for this window to avoid conflicts,
  and I regenerated the config patch yesterday, posting it to our
  mailing list for review and testing.  I have several acks which
  include successful test reports for it.

  However, today I notice we've got new conflicts with previously unseen
  code...  though that conflict should be trivial (it's my changes vs a
  one liner.)"

* 'late-for-linus' of git://git.linaro.org/people/rmk/linux-arm:
  ARM: config: make sure that platforms are ordered by option string
  ARM: config: sort select statements alphanumerically
  UAPI: (Scripted) Disintegrate arch/arm/include/asm

Fix up fairly conflict in arch/arm/Kconfig (the select re-organization
vs recent addition of GENERIC_KERNEL_EXECVE)

Pull UAPI disintegration for include/linux/{,byteorder/}*.h from David Howells:
 "The patches contained herein do the following:

 (1) Remove kernel-only stuff in linux/ppp-comp.h from the UAPI.  I checked
     this with Paul Mackerras before I created the patch and he suggested some
     extra bits to unexport.

 (2) Remove linux/blk_types.h entirely from the UAPI as none of it is userspace
     applicable, and remove from the UAPI that part of linux/fs.h that was the
     reason for linux/blk_types.h being exported in the first place.  I
     discussed this with Jens Axboe before creating the patch.

 (3) The big patch of the series to disintegrate include/linux/*.h as a unit.
     This could be split up, though there would be collisions in moving stuff
     between the two Kbuild files when the parts are merged as that file is
     sorted alphabetically rather than being grouped by subsystem.

     Of this set of headers, 17 files have changed in the UAPI exported region
     since the 4th and only 8 since the 9th so there isn't much change in this
     area - as one might expect.

     It should be pretty obvious and straightforward if it does come to fixing
     up: stuff in __KERNEL__ guards stays where it is and stuff outside moves
     to the same file in the include/uapi/linux/ directory.

     If a new file appears then things get a bit more complicated as the
     "headers +=" line has to move to include/uapi/linux/Kbuild.  Only one new
     file has appeared since the 9th and I judge this type of event relatively
     unlikely.

 (4) A patch to disintegrate include/linux/byteorder/*.h as a unit.

  Signed-off-by: David Howells <dhowells@redhat.com>"

* tag 'disintegrate-main-20121013' of git://git.infradead.org/users/dhowells/linux-headers:
  UAPI: (Scripted) Disintegrate include/linux/byteorder
  UAPI: (Scripted) Disintegrate include/linux
  UAPI: Unexport linux/blk_types.h
  UAPI: Unexport part of linux/ppp-comp.h

Pull spi UAPI disintegration from David Howells:
 "This is to complete part of the Userspace API (UAPI) disintegration
  for which the preparatory patches were pulled recently.  After these
  patches, userspace headers will be segregated into:

        include/uapi/linux/.../foo.h

  for the userspace interface stuff, and:

        include/linux/.../foo.h

  for the strictly kernel internal stuff.
Signed-off-by: David Howells <dhowells@redhat.com>
  Acked-by: Grant Likely <grant.likely@secretlab.ca>"

* tag 'disintegrate-spi-20121009' of git://git.infradead.org/users/dhowells/linux-headers:
  UAPI: (Scripted) Disintegrate include/linux/spi

Pull OpenRISC uapi disintegration from Jonas Bonn:
 "OpenRISC UAPI disintegration work from David Howells"

* tag 'openrisc-uapi' of git://openrisc.net/jonas/linux:
  UAPI: (Scripted) Disintegrate arch/openrisc/include/asm

Pull user namespace compile fixes from Eric W Biederman:
 "This tree contains three trivial fixes.  One compiler warning, one
  thinko fix, and one build fix"

* 'for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/ebiederm/user-namespace:
  btrfs: Fix compilation with user namespace support enabled
  userns: Fix posix_acl_file_xattr_userns gid conversion
  userns: Properly print bluetooth socket uids

Pull md updates from NeilBrown:
 - "discard" support, some dm-raid improvements and other assorted bits
   and pieces.

* tag 'md-3.7' of git://neil.brown.name/md: (29 commits)
  md: refine reporting of resync/reshape delays.
  md/raid5: be careful not to resize_stripes too big.
  md: make sure manual changes to recovery checkpoint are saved.
  md/raid10: use correct limit variable
  md: writing to sync_action should clear the read-auto state.
  Subject: [PATCH] md:change resync_mismatches to atomic64_t to avoid races
  md/raid5: make sure to_read and to_write never go negative.
  md: When RAID5 is dirty, force reconstruct-write instead of read-modify-write.
  md/raid5: protect debug message against NULL derefernce.
  md/raid5: add some missing locking in handle_failed_stripe.
  MD: raid5 avoid unnecessary zero page for trim
  MD: raid5 trim support
  md/bitmap:Don't use IS_ERR to judge alloc_page().
  md/raid1: Don't release reference to device while handling read error.
  raid: replace list_for_each_continue_rcu with new interface
  add further __init annotations to crypto/xor.c
  DM RAID: Fix for "sync" directive ineffectiveness
  DM RAID: Fix comparison of index and quantity for "rebuild" parameter
  DM RAID: Add rebuild capability for RAID10
  DM RAID: Move 'rebuild' checking code to its own function
  ...

The large platform selection choice should be sorted by option string
so it's easy to find the platform you're looking for.  Fix the few
options which are out of this order.
Acked-by: Tony Lindgren <tony@atomide.com>
Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>

As suggested by Andrew Morton:

  This is a pet peeve of mine.  Any time there's a long list of items
  (header file inclusions, kconfig entries, array initalisers, etc) and
  someone wants to add a new item, they *always* go and stick it at the
  end of the list.

  Guys, don't do this.  Either put the new item into a randomly-chosen
  position or, probably better, alphanumerically sort the list.

lets sort all our select statements alphanumerically.  This commit was
created by the following perl:

while (<>) {
	while (/\\\s*$/) {
		$_ .= <>;
	}
	undef %selects if /^\s*config\s+/;
	if (/^\s+select\s+(\w+).*/) {
		if (defined($selects{$1})) {
			if ($selects{$1} eq $_) {
				print STDERR "Warning: removing duplicated $1 entry\n";
			} else {
				print STDERR "Error: $1 differently selected\n".
					"\tOld: $selects{$1}\n".
					"\tNew: $_\n";
				exit 1;
			}
		}
		$selects{$1} = $_;
		next;
	}
	if (%selects and (/^\s*$/ or /^\s+help/ or /^\s+---help---/ or
			  /^endif/ or /^endchoice/)) {
		foreach $k (sort (keys %selects)) {
			print "$selects{$k}";
		}
		undef %selects;
	}
	print;
}
if (%selects) {
	foreach $k (sort (keys %selects)) {
		print "$selects{$k}";
	}
}

It found two duplicates:

Warning: removing duplicated S5P_SETUP_MIPIPHY entry
Warning: removing duplicated HARDIRQS_SW_RESEND entry

and they are identical duplicates, hence the shrinkage in the diffstat
of two lines.

We have four testers reporting success of this change (Tony, Stephen,
Linus and Sekhar.)
Acked-by: Jason Cooper <jason@lakedaemon.net>
Acked-by: Tony Lindgren <tony@atomide.com>
Acked-by: Stephen Warren <swarren@nvidia.com>
Acked-by: Linus Walleij <linus.walleij@linaro.org>
Acked-by: Sekhar Nori <nsekhar@ti.com>
Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>

Signed-off-by: David Howells <dhowells@redhat.com>
Acked-by: Arnd Bergmann <arnd@arndb.de>
Acked-by: Thomas Gleixner <tglx@linutronix.de>
Acked-by: Michael Kerrisk <mtk.manpages@gmail.com>
Acked-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
Acked-by: Dave Jones <davej@redhat.com>

Signed-off-by: David Howells <dhowells@redhat.com>
Acked-by: Arnd Bergmann <arnd@arndb.de>
Acked-by: Thomas Gleixner <tglx@linutronix.de>
Acked-by: Michael Kerrisk <mtk.manpages@gmail.com>
Acked-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
Acked-by: Dave Jones <davej@redhat.com>