1. 13 Jul, 2007 10 commits
    • Dan Williams's avatar
      md: handle_stripe5 - add request/completion logic for async compute ops · f38e1219
      Dan Williams authored
      handle_stripe will compute a block when a backing disk has failed, or when
      it determines it can save a disk read by computing the block from all the
      other up-to-date blocks.
      
      Previously a block would be computed under the lock and subsequent logic in
      handle_stripe could use the newly up-to-date block.  With the raid5_run_ops
      implementation the compute operation is carried out a later time outside
      the lock.  To preserve the old functionality we take advantage of the
      dependency chain feature of async_tx to flag the block as R5_Wantcompute
      and then let other parts of handle_stripe operate on the block as if it
      were up-to-date.  raid5_run_ops guarantees that the block will be ready
      before it is used in another operation.
      
      However, this only works in cases where the compute and the dependent
      operation are scheduled at the same time.  If a previous call to
      handle_stripe sets the R5_Wantcompute flag there is no facility to pass the
      async_tx dependency chain across successive calls to raid5_run_ops.  The
      req_compute variable protects against this case.
      
      Changelog:
      * remove the req_compute BUG_ON
      Signed-off-by: default avatarDan Williams <dan.j.williams@intel.com>
      Acked-By: default avatarNeilBrown <neilb@suse.de>
      f38e1219
    • Dan Williams's avatar
      md: handle_stripe5 - add request/completion logic for async write ops · e33129d8
      Dan Williams authored
      After handle_stripe5 decides whether it wants to perform a
      read-modify-write, or a reconstruct write it calls
      handle_write_operations5.  A read-modify-write operation will perform an
      xor subtraction of the blocks marked with the R5_Wantprexor flag, copy the
      new data into the stripe (biodrain) and perform a postxor operation across
      all up-to-date blocks to generate the new parity.  A reconstruct write is run
      when all blocks are already up-to-date in the cache so all that is needed
      is a biodrain and postxor.
      
      On the completion path STRIPE_OP_PREXOR will be set if the operation was a
      read-modify-write.  The STRIPE_OP_BIODRAIN flag is used in the completion
      path to differentiate write-initiated postxor operations versus
      expansion-initiated postxor operations.  Completion of a write triggers i/o
      to the drives.
      
      Changelog:
      * make the 'rcw' parameter to handle_write_operations5 a simple flag, Neil Brown
      * remove test_and_set/test_and_clear BUG_ONs, Neil Brown
      Signed-off-by: default avatarDan Williams <dan.j.williams@intel.com>
      Acked-By: default avatarNeilBrown <neilb@suse.de>
      e33129d8
    • Dan Williams's avatar
      md: common infrastructure for running operations with raid5_run_ops · d84e0f10
      Dan Williams authored
      All the handle_stripe operations that are to be transitioned to use
      raid5_run_ops need a method to coherently gather work under the stripe-lock
      and hand that work off to raid5_run_ops.  The 'get_stripe_work' routine
      runs under the lock to read all the bits in sh->ops.pending that do not
      have the corresponding bit set in sh->ops.ack.  This modified 'pending'
      bitmap is then passed to raid5_run_ops for processing.
      
      The transition from 'ack' to 'completion' does not need similar protection
      as the existing release_stripe infrastructure will guarantee that
      handle_stripe will run again after a completion bit is set, and
      handle_stripe can tolerate a sh->ops.completed bit being set while the lock
      is held.
      
      A call to async_tx_issue_pending_all() is added to raid5d to kick the
      offload engines once all pending stripe operations work has been submitted.
      This enables batching of the submission and completion of operations.
      Signed-off-by: default avatarDan Williams <dan.j.williams@intel.com>
      Acked-By: default avatarNeilBrown <neilb@suse.de>
      d84e0f10
    • Dan Williams's avatar
      md: raid5_run_ops - run stripe operations outside sh->lock · 91c00924
      Dan Williams authored
      When the raid acceleration work was proposed, Neil laid out the following
      attack plan:
      
      1/ move the xor and copy operations outside spin_lock(&sh->lock)
      2/ find/implement an asynchronous offload api
      
      The raid5_run_ops routine uses the asynchronous offload api (async_tx) and
      the stripe_operations member of a stripe_head to carry out xor+copy
      operations asynchronously, outside the lock.
      
      To perform operations outside the lock a new set of state flags is needed
      to track new requests, in-flight requests, and completed requests.  In this
      new model handle_stripe is tasked with scanning the stripe_head for work,
      updating the stripe_operations structure, and finally dropping the lock and
      calling raid5_run_ops for processing.  The following flags outline the
      requests that handle_stripe can make of raid5_run_ops:
      
      STRIPE_OP_BIOFILL
       - copy data into request buffers to satisfy a read request
      STRIPE_OP_COMPUTE_BLK
       - generate a missing block in the cache from the other blocks
      STRIPE_OP_PREXOR
       - subtract existing data as part of the read-modify-write process
      STRIPE_OP_BIODRAIN
       - copy data out of request buffers to satisfy a write request
      STRIPE_OP_POSTXOR
       - recalculate parity for new data that has entered the cache
      STRIPE_OP_CHECK
       - verify that the parity is correct
      STRIPE_OP_IO
       - submit i/o to the member disks (note this was already performed outside
         the stripe lock, but it made sense to add it as an operation type
      
      The flow is:
      1/ handle_stripe sets STRIPE_OP_* in sh->ops.pending
      2/ raid5_run_ops reads sh->ops.pending, sets sh->ops.ack, and submits the
         operation to the async_tx api
      3/ async_tx triggers the completion callback routine to set
         sh->ops.complete and release the stripe
      4/ handle_stripe runs again to finish the operation and optionally submit
         new operations that were previously blocked
      
      Note this patch just defines raid5_run_ops, subsequent commits (one per
      major operation type) modify handle_stripe to take advantage of this
      routine.
      
      Changelog:
      * removed ops_complete_biodrain in favor of ops_complete_postxor and
        ops_complete_write.
      * removed the raid5_run_ops workqueue
      * call bi_end_io for reads in ops_complete_biofill, saves a call to
        handle_stripe
      * explicitly handle the 2-disk raid5 case (xor becomes memcpy), Neil Brown
      * fix race between async engines and bi_end_io call for reads, Neil Brown
      * remove unnecessary spin_lock from ops_complete_biofill
      * remove test_and_set/test_and_clear BUG_ONs, Neil Brown
      * remove explicit interrupt handling for channel switching, this feature
        was absorbed (i.e. it is now implicit) by the async_tx api
      * use return_io in ops_complete_biofill
      Signed-off-by: default avatarDan Williams <dan.j.williams@intel.com>
      Acked-By: default avatarNeilBrown <neilb@suse.de>
      91c00924
    • Dan Williams's avatar
      raid5: replace custom debug PRINTKs with standard pr_debug · 45b4233c
      Dan Williams authored
      Replaces PRINTK with pr_debug, and kills the RAID5_DEBUG definition in
      favor of the global DEBUG definition.  To get local debug messages just add
      '#define DEBUG' to the top of the file.
      Signed-off-by: default avatarDan Williams <dan.j.williams@intel.com>
      Acked-By: default avatarNeilBrown <neilb@suse.de>
      45b4233c
    • Dan Williams's avatar
      raid5: refactor handle_stripe5 and handle_stripe6 (v3) · a4456856
      Dan Williams authored
      handle_stripe5 and handle_stripe6 have very deep logic paths handling the
      various states of a stripe_head.  By introducing the 'stripe_head_state'
      and 'r6_state' objects, large portions of the logic can be moved to
      sub-routines.
      
      'struct stripe_head_state' consumes all of the automatic variables that previously
      stood alone in handle_stripe5,6.  'struct r6_state' contains the handle_stripe6
      specific variables like p_failed and q_failed.
      
      One of the nice side effects of the 'stripe_head_state' change is that it
      allows for further reductions in code duplication between raid5 and raid6.
      The following new routines are shared between raid5 and raid6:
      
      	handle_completed_write_requests
      	handle_requests_to_failed_array
      	handle_stripe_expansion
      
      Changes:
      * v2: fixed 'conf->raid_disk-1' for the raid6 'handle_stripe_expansion' path
      * v3: removed the unused 'dirty' field from struct stripe_head_state
      * v3: coalesced open coded bi_end_io routines into return_io()
      Signed-off-by: default avatarDan Williams <dan.j.williams@intel.com>
      Acked-By: default avatarNeilBrown <neilb@suse.de>
      a4456856
    • Dan Williams's avatar
      async_tx: add the async_tx api · 9bc89cd8
      Dan Williams authored
      The async_tx api provides methods for describing a chain of asynchronous
      bulk memory transfers/transforms with support for inter-transactional
      dependencies.  It is implemented as a dmaengine client that smooths over
      the details of different hardware offload engine implementations.  Code
      that is written to the api can optimize for asynchronous operation and the
      api will fit the chain of operations to the available offload resources. 
       
      	I imagine that any piece of ADMA hardware would register with the
      	'async_*' subsystem, and a call to async_X would be routed as
      	appropriate, or be run in-line. - Neil Brown
      
      async_tx exploits the capabilities of struct dma_async_tx_descriptor to
      provide an api of the following general format:
      
      struct dma_async_tx_descriptor *
      async_<operation>(..., struct dma_async_tx_descriptor *depend_tx,
      			dma_async_tx_callback cb_fn, void *cb_param)
      {
      	struct dma_chan *chan = async_tx_find_channel(depend_tx, <operation>);
      	struct dma_device *device = chan ? chan->device : NULL;
      	int int_en = cb_fn ? 1 : 0;
      	struct dma_async_tx_descriptor *tx = device ?
      		device->device_prep_dma_<operation>(chan, len, int_en) : NULL;
      
      	if (tx) { /* run <operation> asynchronously */
      		...
      		tx->tx_set_dest(addr, tx, index);
      		...
      		tx->tx_set_src(addr, tx, index);
      		...
      		async_tx_submit(chan, tx, flags, depend_tx, cb_fn, cb_param);
      	} else { /* run <operation> synchronously */
      		...
      		<operation>
      		...
      		async_tx_sync_epilog(flags, depend_tx, cb_fn, cb_param);
      	}
      
      	return tx;
      }
      
      async_tx_find_channel() returns a capable channel from its pool.  The
      channel pool is organized as a per-cpu array of channel pointers.  The
      async_tx_rebalance() routine is tasked with managing these arrays.  In the
      uniprocessor case async_tx_rebalance() tries to spread responsibility
      evenly over channels of similar capabilities.  For example if there are two
      copy+xor channels, one will handle copy operations and the other will
      handle xor.  In the SMP case async_tx_rebalance() attempts to spread the
      operations evenly over the cpus, e.g. cpu0 gets copy channel0 and xor
      channel0 while cpu1 gets copy channel 1 and xor channel 1.  When a
      dependency is specified async_tx_find_channel defaults to keeping the
      operation on the same channel.  A xor->copy->xor chain will stay on one
      channel if it supports both operation types, otherwise the transaction will
      transition between a copy and a xor resource.
      
      Currently the raid5 implementation in the MD raid456 driver has been
      converted to the async_tx api.  A driver for the offload engines on the
      Intel Xscale series of I/O processors, iop-adma, is provided in a later
      commit.  With the iop-adma driver and async_tx, raid456 is able to offload
      copy, xor, and xor-zero-sum operations to hardware engines.
       
      On iop342 tiobench showed higher throughput for sequential writes (20 - 30%
      improvement) and sequential reads to a degraded array (40 - 55%
      improvement).  For the other cases performance was roughly equal, +/- a few
      percentage points.  On a x86-smp platform the performance of the async_tx
      implementation (in synchronous mode) was also +/- a few percentage points
      of the original implementation.  According to 'top' on iop342 CPU
      utilization drops from ~50% to ~15% during a 'resync' while the speed
      according to /proc/mdstat doubles from ~25 MB/s to ~50 MB/s.
       
      The tiobench command line used for testing was: tiobench --size 2048
      --block 4096 --block 131072 --dir /mnt/raid --numruns 5
      * iop342 had 1GB of memory available
      
      Details:
      * if CONFIG_DMA_ENGINE=n the asynchronous path is compiled away by making
        async_tx_find_channel a static inline routine that always returns NULL
      * when a callback is specified for a given transaction an interrupt will
        fire at operation completion time and the callback will occur in a
        tasklet.  if the the channel does not support interrupts then a live
        polling wait will be performed
      * the api is written as a dmaengine client that requests all available
        channels
      * In support of dependencies the api implicitly schedules channel-switch
        interrupts.  The interrupt triggers the cleanup tasklet which causes
        pending operations to be scheduled on the next channel
      * Xor engines treat an xor destination address differently than a software
        xor routine.  To the software routine the destination address is an implied
        source, whereas engines treat it as a write-only destination.  This patch
        modifies the xor_blocks routine to take a an explicit destination address
        to mirror the hardware.
      
      Changelog:
      * fixed a leftover debug print
      * don't allow callbacks in async_interrupt_cond
      * fixed xor_block changes
      * fixed usage of ASYNC_TX_XOR_DROP_DEST
      * drop dma mapping methods, suggested by Chris Leech
      * printk warning fixups from Andrew Morton
      * don't use inline in C files, Adrian Bunk
      * select the API when MD is enabled
      * BUG_ON xor source counts <= 1
      * implicitly handle hardware concerns like channel switching and
        interrupts, Neil Brown
      * remove the per operation type list, and distribute operation capabilities
        evenly amongst the available channels
      * simplify async_tx_find_channel to optimize the fast path
      * introduce the channel_table_initialized flag to prevent early calls to
        the api
      * reorganize the code to mimic crypto
      * include mm.h as not all archs include it in dma-mapping.h
      * make the Kconfig options non-user visible, Adrian Bunk
      * move async_tx under crypto since it is meant as 'core' functionality, and
        the two may share algorithms in the future
      * move large inline functions into c files
      * checkpatch.pl fixes
      * gpl v2 only correction
      
      Cc: Herbert Xu <herbert@gondor.apana.org.au>
      Signed-off-by: default avatarDan Williams <dan.j.williams@intel.com>
      Acked-By: default avatarNeilBrown <neilb@suse.de>
      9bc89cd8
    • Dan Williams's avatar
      xor: make 'xor_blocks' a library routine for use with async_tx · 685784aa
      Dan Williams authored
      The async_tx api tries to use a dma engine for an operation, but will fall
      back to an optimized software routine otherwise.  Xor support is
      implemented using the raid5 xor routines.  For organizational purposes this
      routine is moved to a common area.
      
      The following fixes are also made:
      * rename xor_block => xor_blocks, suggested by Adrian Bunk
      * ensure that xor.o initializes before md.o in the built-in case
      * checkpatch.pl fixes
      * mark calibrate_xor_blocks __init, Adrian Bunk
      
      Cc: Adrian Bunk <bunk@stusta.de>
      Cc: NeilBrown <neilb@suse.de>
      Cc: Herbert Xu <herbert@gondor.apana.org.au>
      Signed-off-by: default avatarDan Williams <dan.j.williams@intel.com>
      685784aa
    • Dan Williams's avatar
      dmaengine: make clients responsible for managing channels · d379b01e
      Dan Williams authored
      The current implementation assumes that a channel will only be used by one
      client at a time.  In order to enable channel sharing the dmaengine core is
      changed to a model where clients subscribe to channel-available-events.
      Instead of tracking how many channels a client wants and how many it has
      received the core just broadcasts the available channels and lets the
      clients optionally take a reference.  The core learns about the clients'
      needs at dma_event_callback time.
      
      In support of multiple operation types, clients can specify a capability
      mask to only be notified of channels that satisfy a certain set of
      capabilities.
      
      Changelog:
      * removed DMA_TX_ARRAY_INIT, no longer needed
      * dma_client_chan_free -> dma_chan_release: switch to global reference
        counting only at device unregistration time, before it was also happening
        at client unregistration time
      * clients now return dma_state_client to dmaengine (ack, dup, nak)
      * checkpatch.pl fixes
      * fixup merge with git-ioat
      
      Cc: Chris Leech <christopher.leech@intel.com>
      Signed-off-by: default avatarShannon Nelson <shannon.nelson@intel.com>
      Signed-off-by: default avatarDan Williams <dan.j.williams@intel.com>
      Acked-by: default avatarDavid S. Miller <davem@davemloft.net>
      d379b01e
    • Dan Williams's avatar
      dmaengine: refactor dmaengine around dma_async_tx_descriptor · 7405f74b
      Dan Williams authored
      The current dmaengine interface defines mutliple routines per operation,
      i.e. dma_async_memcpy_buf_to_buf, dma_async_memcpy_buf_to_page etc.  Adding
      more operation types (xor, crc, etc) to this model would result in an
      unmanageable number of method permutations.
      
      	Are we really going to add a set of hooks for each DMA engine
      	whizbang feature?
      		- Jeff Garzik
      
      The descriptor creation process is refactored using the new common
      dma_async_tx_descriptor structure.  Instead of per driver
      do_<operation>_<dest>_to_<src> methods, drivers integrate
      dma_async_tx_descriptor into their private software descriptor and then
      define a 'prep' routine per operation.  The prep routine allocates a
      descriptor and ensures that the tx_set_src, tx_set_dest, tx_submit routines
      are valid.  Descriptor creation and submission becomes:
      
      struct dma_device *dev;
      struct dma_chan *chan;
      struct dma_async_tx_descriptor *tx;
      
      tx = dev->device_prep_dma_<operation>(chan, len, int_flag)
      tx->tx_set_src(dma_addr_t, tx, index /* for multi-source ops */)
      tx->tx_set_dest(dma_addr_t, tx, index)
      tx->tx_submit(tx)
      
      In addition to the refactoring, dma_async_tx_descriptor also lays the
      groundwork for definining cross-channel-operation dependencies, and a
      callback facility for asynchronous notification of operation completion.
      
      Changelog:
      * drop dma mapping methods, suggested by Chris Leech
      * fix ioat_dma_dependency_added, also caught by Andrew Morton
      * fix dma_sync_wait, change from Andrew Morton
      * uninline large functions, change from Andrew Morton
      * add tx->callback = NULL to dmaengine calls to interoperate with async_tx
        calls
      * hookup ioat_tx_submit
      * convert channel capabilities to a 'cpumask_t like' bitmap
      * removed DMA_TX_ARRAY_INIT, no longer needed
      * checkpatch.pl fixes
      * make set_src, set_dest, and tx_submit descriptor specific methods
      * fixup git-ioat merge
      * move group_list and phys to dma_async_tx_descriptor
      
      Cc: Jeff Garzik <jeff@garzik.org>
      Cc: Chris Leech <christopher.leech@intel.com>
      Signed-off-by: default avatarShannon Nelson <shannon.nelson@intel.com>
      Signed-off-by: default avatarDan Williams <dan.j.williams@intel.com>
      Acked-by: default avatarDavid S. Miller <davem@davemloft.net>
      7405f74b
  2. 11 Jul, 2007 8 commits
  3. 08 Jul, 2007 6 commits
  4. 07 Jul, 2007 4 commits
    • Adrian Bunk's avatar
      DLM must depend on SYSFS · 95511ad4
      Adrian Bunk authored
      The dependency of DLM on SYSFS got lost in
      commit 6ed7257b resulting in the
      following compile error with CONFIG_DLM=y, CONFIG_SYSFS=n:
      
      <--  snip  -->
      
      ...
        LD      .tmp_vmlinux1
      fs/built-in.o: In function `dlm_lockspace_init':
      /home/bunk/linux/kernel-2.6/linux-2.6.22-rc6-mm1/fs/dlm/lockspace.c:231: undefined reference to `kernel_subsys'
      fs/built-in.o: In function `configfs_init':
      /home/bunk/linux/kernel-2.6/linux-2.6.22-rc6-mm1/fs/configfs/mount.c:143: undefined reference to `kernel_subsys'
      make[1]: *** [.tmp_vmlinux1] Error 1
      
      <--  snip  -->
      Signed-off-by: default avatarAdrian Bunk <bunk@stusta.de>
      Signed-off-by: default avatarLinus Torvalds <torvalds@linux-foundation.org>
      95511ad4
    • Dave Jones's avatar
      Clean up E7520/7320/7525 quirk printk. · 38377be8
      Dave Jones authored
      The printk level in this printk is bogus, as the previous printk
      didn't have a terminating \n resulting in ..
      
      Intel E7520/7320/7525 detected.<6>Disabling irq balancing and affinity
      
      It also never printed a \n at all in the case where we didn't do
      the quirk.
      
      Change it to only make noise if it actually does something useful.
      Signed-off-by: default avatarDave Jones <davej@redhat.com>
      Signed-off-by: default avatarLinus Torvalds <torvalds@linux-foundation.org>
      38377be8
    • Adrian Bunk's avatar
      include/linux/kallsyms.h must #include <linux/errno.h> · 40e48eed
      Adrian Bunk authored
      This patch fixes the following 2.6.22 regression with CONFIG_KALLSYMS=n:
      
      <--  snip  -->
      
      ...
        CC      arch/m32r/kernel/traps.o
      In file included from /home/bunk/linux/kernel-2.6/linux-2.6.22-rc6-mm1/arch/m32r/kernel/traps.c:14:
      /home/bunk/linux/kernel-2.6/linux-2.6.22-rc6-mm1/include/linux/kallsyms.h: In function 'lookup_symbol_name':
      /home/bunk/linux/kernel-2.6/linux-2.6.22-rc6-mm1/include/linux/kallsyms.h:66: error: 'ERANGE' undeclared (first use in this function)
      /home/bunk/linux/kernel-2.6/linux-2.6.22-rc6-mm1/include/linux/kallsyms.h:66: error: (Each undeclared identifier is reported only once
      /home/bunk/linux/kernel-2.6/linux-2.6.22-rc6-mm1/include/linux/kallsyms.h:66: error: for each function it appears in.)
      /home/bunk/linux/kernel-2.6/linux-2.6.22-rc6-mm1/include/linux/kallsyms.h: In function 'lookup_symbol_attrs':
      /home/bunk/linux/kernel-2.6/linux-2.6.22-rc6-mm1/include/linux/kallsyms.h:71: error: 'ERANGE' undeclared (first use in this function)
      make[2]: *** [arch/m32r/kernel/traps.o] Error 1
      
      <--  snip  -->
      Signed-off-by: default avatarAdrian Bunk <bunk@stusta.de>
      Signed-off-by: default avatarLinus Torvalds <torvalds@linux-foundation.org>
      40e48eed
    • David Woodhouse's avatar
      Fix use-after-free oops in Bluetooth HID. · 1c39858b
      David Woodhouse authored
      When cleaning up HIDP sessions, we currently close the ACL connection
      before deregistering the input device. Closing the ACL connection
      schedules a workqueue to remove the associated objects from sysfs, but
      the input device still refers to them -- and if the workqueue happens to
      run before the input device removal, the kernel will oops when trying to
      look up PHYSDEVPATH for the removed input device.
      
      Fix this by deregistering the input device before closing the
      connections.
      Signed-off-by: default avatarDavid Woodhouse <dwmw2@infradead.org>
      Acked-by: default avatarMarcel Holtmann <marcel@holtmann.org>
      Signed-off-by: default avatarLinus Torvalds <torvalds@linux-foundation.org>
      1c39858b
  5. 06 Jul, 2007 12 commits