1. 24 Mar, 2017 15 commits
  2. 23 Mar, 2017 14 commits
    • NeilBrown's avatar
      MD: use per-cpu counter for writes_pending · 4ad23a97
      NeilBrown authored
      The 'writes_pending' counter is used to determine when the
      array is stable so that it can be marked in the superblock
      as "Clean".  Consequently it needs to be updated frequently
      but only checked for zero occasionally.  Recent changes to
      raid5 cause the count to be updated even more often - once
      per 4K rather than once per bio.  This provided
      justification for making the updates more efficient.
      
      So we replace the atomic counter a percpu-refcount.
      This can be incremented and decremented cheaply most of the
      time, and can be switched to "atomic" mode when more
      precise counting is needed.  As it is possible for multiple
      threads to want a precise count, we introduce a
      "sync_checker" counter to count the number of threads
      in "set_in_sync()", and only switch the refcount back
      to percpu mode when that is zero.
      
      We need to be careful about races between set_in_sync()
      setting ->in_sync to 1, and md_write_start() setting it
      to zero.  md_write_start() holds the rcu_read_lock()
      while checking if the refcount is in percpu mode.  If
      it is, then we know a switch to 'atomic' will not happen until
      after we call rcu_read_unlock(), in which case set_in_sync()
      will see the elevated count, and not set in_sync to 1.
      If it is not in percpu mode, we take the mddev->lock to
      ensure proper synchronization.
      
      It is no longer possible to quickly check if the count is zero, which
      we previously did to update a timer or to schedule the md_thread.
      So now we do these every time we decrement that counter, but make
      sure they are fast.
      
      mod_timer() already optimizes the case where the timeout value doesn't
      actually change.  We leverage that further by always rounding off the
      jiffies to the timeout value.  This may delay the marking of 'clean'
      slightly, but ensure we only perform atomic operation here when absolutely
      needed.
      
      md_wakeup_thread() current always calls wake_up(), even if
      THREAD_WAKEUP is already set.  That too can be optimised to avoid
      calls to wake_up().
      Signed-off-by: default avatarNeilBrown <neilb@suse.com>
      Signed-off-by: default avatarShaohua Li <shli@fb.com>
      4ad23a97
    • NeilBrown's avatar
      percpu-refcount: support synchronous switch to atomic mode. · 210f7cdc
      NeilBrown authored
      percpu_ref_switch_to_atomic_sync() schedules the switch to atomic mode, then
      waits for it to complete.
      
      Also export percpu_ref_switch_to_* so they can be used from modules.
      
      This will be used in md/raid to count the number of pending write
      requests to an array.
      We occasionally need to check if the count is zero, but most often
      we don't care.
      We always want updates to the counter to be fast, as in some cases
      we count every 4K page.
      Signed-off-by: default avatarNeilBrown <neilb@suse.com>
      Acked-by: default avatarTejun Heo <tj@kernel.org>
      Signed-off-by: default avatarShaohua Li <shli@fb.com>
      210f7cdc
    • NeilBrown's avatar
      md: close a race with setting mddev->in_sync · 55cc39f3
      NeilBrown authored
      If ->in_sync is being set just as md_write_start() is being called,
      it is possible that set_in_sync() won't see the elevated
      ->writes_pending, and md_write_start() won't see the set ->in_sync.
      
      To close this race, re-test ->writes_pending after setting ->in_sync,
      and add memory barriers to ensure the increment of ->writes_pending
      will be seen by the time of this second test, or the new ->in_sync
      will be seen by md_write_start().
      
      Add a spinlock to array_state_show() to ensure this temporary
      instability is never visible from userspace.
      Signed-off-by: default avatarNeilBrown <neilb@suse.com>
      Signed-off-by: default avatarShaohua Li <shli@fb.com>
      55cc39f3
    • NeilBrown's avatar
      md: factor out set_in_sync() · 6497709b
      NeilBrown authored
      Three separate places in md.c check if the number of active
      writes is zero and, if so, sets mddev->in_sync.
      
      There are a few differences, but there shouldn't be:
      - it is always appropriate to notify the change in
        sysfs_state, and there is no need to do this outside a
        spin-locked region.
      - we never need to check ->recovery_cp.  The state of resync
        is not relevant for whether there are any pending writes
        or not (which is what ->in_sync reports).
      
      So create set_in_sync() which does the correct tests and
      makes the correct changes, and call this in all three
      places.
      
      Any behaviour changes here a minor and cosmetic.
      Signed-off-by: default avatarNeilBrown <neilb@suse.com>
      Signed-off-by: default avatarShaohua Li <shli@fb.com>
      6497709b
    • NeilBrown's avatar
      md/raid5: don't test ->writes_pending in raid5_remove_disk · 84dd97a6
      NeilBrown authored
      This test on ->writes_pending cannot be safe as the counter
      can be incremented at any moment and cannot be locked against.
      
      Change it to test conf->active_stripes, which at least
      can be locked against.  More changes are still needed.
      
      A future patch will change ->writes_pending, and testing it here will
      be very inconvenient.
      Signed-off-by: default avatarNeilBrown <neilb@suse.com>
      Signed-off-by: default avatarShaohua Li <shli@fb.com>
      84dd97a6
    • NeilBrown's avatar
      md/raid1: stop using bi_phys_segment · 37011e3a
      NeilBrown authored
      Change to use bio->__bi_remaining to count number of r1bio attached
      to a bio.
      See precious raid10 patch for more details.
      
      Like the raid10.c patch, this fixes a bug as nr_queued and nr_pending
      used to measure different things, but were being compared.
      
      This patch fixes another bug in that nr_pending previously did not
      could write-behind requests, so behind writes could continue while
      resync was happening.  How that nr_pending counts all r1_bio,
      the resync cannot commence until the behind writes have completed.
      Signed-off-by: default avatarNeilBrown <neilb@suse.com>
      Signed-off-by: default avatarShaohua Li <shli@fb.com>
      37011e3a
    • NeilBrown's avatar
      md/raid10: stop using bi_phys_segments · fd16f2e8
      NeilBrown authored
      raid10 currently repurposes bi_phys_segments on each
      incoming bio to count how many r10bio was used to encode the
      request.
      
      We need to know when the number of attached r10bio reaches
      zero to:
      1/ call bio_endio() when all IO on the bio is finished
      2/ decrement ->nr_pending so that resync IO can proceed.
      
      Now that the bio has its own __bi_remaining counter, that
      can be used instead. We can call bio_inc_remaining to
      increment the counter and call bio_endio() every time an
      r10bio completes, rather than only when bi_phys_segments
      reaches zero.
      
      This addresses point 1, but not point 2.  bio_endio()
      doesn't (and cannot) report when the last r10bio has
      finished, so a different approach is needed.
      
      So: instead of counting bios in ->nr_pending, count r10bios.
      i.e. every time we attach a bio, increment nr_pending.
      Every time an r10bio completes, decrement nr_pending.
      
      Normally we only increment nr_pending after first checking
      that ->barrier is zero, or some other non-trivial tests and
      possible waiting.  When attaching multiple r10bios to a bio,
      we only need the tests and the waiting once.  After the
      first increment, subsequent increments can happen
      unconditionally as they are really all part of the one
      request.
      
      So introduce inc_pending() which can be used when we know
      that nr_pending is already elevated.
      
      Note that this fixes a bug.  freeze_array() contains the line
      	atomic_read(&conf->nr_pending) == conf->nr_queued+extra,
      which implies that the units for ->nr_pending, ->nr_queued and extra
      are the same.
      ->nr_queue and extra count r10_bios, but prior to this patch,
      ->nr_pending counted bios.  If a bio ever resulted in multiple
      r10_bios (due to bad blocks), freeze_array() would not work correctly.
      Now it does.
      Signed-off-by: default avatarNeilBrown <neilb@suse.com>
      Signed-off-by: default avatarShaohua Li <shli@fb.com>
      fd16f2e8
    • NeilBrown's avatar
      md/raid1, raid10: move rXbio accounting closer to allocation. · 6b6c8110
      NeilBrown authored
      When raid1 or raid10 find they will need to allocate a new
      r1bio/r10bio, in order to work around a known bad block, they
      account for the allocation well before the allocation is
      made.  This separation makes the correctness less obvious
      and requires comments.
      
      The accounting needs to be a little before: before the first
      rXbio is submitted, but that is all.
      
      So move the accounting down to where it makes more sense.
      Signed-off-by: default avatarNeilBrown <neilb@suse.com>
      Signed-off-by: default avatarShaohua Li <shli@fb.com>
      6b6c8110
    • NeilBrown's avatar
      Revert "md/raid5: limit request size according to implementation limits" · 97d53438
      NeilBrown authored
      This reverts commit e8d7c332.
      
      Now that raid5 doesn't abuse bi_phys_segments any more, we no longer
      need to impose these limits.
      Signed-off-by: default avatarNeilBrown <neilb@suse.com>
      Signed-off-by: default avatarShaohua Li <shli@fb.com>
      97d53438
    • NeilBrown's avatar
      md/raid5: remove over-loading of ->bi_phys_segments. · 0472a42b
      NeilBrown authored
      When a read request, which bypassed the cache, fails, we need to retry
      it through the cache.
      This involves attaching it to a sequence of stripe_heads, and it may not
      be possible to get all the stripe_heads we need at once.
      We do what we can, and record how far we got in ->bi_phys_segments so
      we can pick up again later.
      
      There is only ever one bio which may have a non-zero offset stored in
      ->bi_phys_segments, the one that is either active in the single thread
      which calls retry_aligned_read(), or is in conf->retry_read_aligned
      waiting for retry_aligned_read() to be called again.
      
      So we only need to store one offset value.  This can be in a local
      variable passed between remove_bio_from_retry() and
      retry_aligned_read(), or in the r5conf structure next to the
      ->retry_read_aligned pointer.
      
      Storing it there allows the last usage of ->bi_phys_segments to be
      removed from md/raid5.c.
      Signed-off-by: default avatarNeilBrown <neilb@suse.com>
      Signed-off-by: default avatarShaohua Li <shli@fb.com>
      0472a42b
    • NeilBrown's avatar
      md/raid5: use bio_inc_remaining() instead of repurposing bi_phys_segments as a counter · 016c76ac
      NeilBrown authored
      md/raid5 needs to keep track of how many stripe_heads are processing a
      bio so that it can delay calling bio_endio() until all stripe_heads
      have completed.  It currently uses 16 bits of ->bi_phys_segments for
      this purpose.
      
      16 bits is only enough for 256M requests, and it is possible for a
      single bio to be larger than this, which causes problems.  Also, the
      bio struct contains a larger counter, __bi_remaining, which has a
      purpose very similar to the purpose of our counter.  So stop using
      ->bi_phys_segments, and instead use __bi_remaining.
      
      This means we don't need to initialize the counter, as our caller
      initializes it to '1'.  It also means we can call bio_endio() directly
      as it tests this counter internally.
      Signed-off-by: default avatarNeilBrown <neilb@suse.com>
      Signed-off-by: default avatarShaohua Li <shli@fb.com>
      016c76ac
    • NeilBrown's avatar
      md/raid5: call bio_endio() directly rather than queueing for later. · bd83d0a2
      NeilBrown authored
      We currently gather bios that need to be returned into a bio_list
      and call bio_endio() on them all together.
      The original reason for this was to avoid making the calls while
      holding a spinlock.
      Locking has changed a lot since then, and that reason is no longer
      valid.
      
      So discard return_io() and various return_bi lists, and just call
      bio_endio() directly as needed.
      Signed-off-by: default avatarNeilBrown <neilb@suse.com>
      Signed-off-by: default avatarShaohua Li <shli@fb.com>
      bd83d0a2
    • NeilBrown's avatar
      md/raid5: simplfy delaying of writes while metadata is updated. · 16d997b7
      NeilBrown authored
      If a device fails during a write, we must ensure the failure is
      recorded in the metadata before the completion of the write is
      acknowleged.
      
      Commit c3cce6cd ("md/raid5: ensure device failure recorded before
      write request returns.")  added code for this, but it was
      unnecessarily complicated.  We already had similar functionality for
      handling updates to the bad-block-list, thanks to Commit de393cde
      ("md: make it easier to wait for bad blocks to be acknowledged.")
      
      So revert most of the former commit, and instead avoid collecting
      completed writes if MD_CHANGE_PENDING is set.  raid5d() will then flush
      the metadata and retry the stripe_head.
      As this change can leave a stripe_head ready for handling immediately
      after handle_active_stripes() returns, we change raid5_do_work() to
      pause when MD_CHANGE_PENDING is set, so that it doesn't spin.
      
      We check MD_CHANGE_PENDING *after* analyse_stripe() as it could be set
      asynchronously.  After analyse_stripe(), we have collected stable data
      about the state of devices, which will be used to make decisions.
      Signed-off-by: default avatarNeilBrown <neilb@suse.com>
      Signed-off-by: default avatarShaohua Li <shli@fb.com>
      16d997b7
    • NeilBrown's avatar
      md/raid5: use md_write_start to count stripes, not bios · 49728050
      NeilBrown authored
      We use md_write_start() to increase the count of pending writes, and
      md_write_end() to decrement the count.  We currently count bios
      submitted to md/raid5.  Change it count stripe_heads that a WRITE bio
      has been attached to.
      
      So now, raid5_make_request() calls md_write_start() and then
      md_write_end() to keep the count elevated during the setup of the
      request.
      
      add_stripe_bio() calls md_write_start() for each stripe_head, and the
      completion routines always call md_write_end(), instead of only
      calling it when raid5_dec_bi_active_stripes() returns 0.
      make_discard_request also calls md_write_start/end().
      
      The parallel between md_write_{start,end} and use of bi_phys_segments
      can be seen in that:
       Whenever we set bi_phys_segments to 1, we now call md_write_start.
       Whenever we increment it on non-read requests with
         raid5_inc_bi_active_stripes(), we now call md_write_start().
       Whenever we decrement bi_phys_segments on non-read requsts with
          raid5_dec_bi_active_stripes(), we now call md_write_end().
      
      This reduces our dependence on keeping a per-bio count of active
      stripes in bi_phys_segments.
      
      md_write_inc() is added which parallels md_write_start(), but requires
      that a write has already been started, and is certain never to sleep.
      This can be used inside a spinlocked region when adding to a write
      request.
      Signed-off-by: default avatarNeilBrown <neilb@suse.com>
      Signed-off-by: default avatarShaohua Li <shli@fb.com>
      49728050
  3. 16 Mar, 2017 11 commits
    • Guoqing Jiang's avatar
      md: move bitmap_destroy to the beginning of __md_stop · 48df498d
      Guoqing Jiang authored
      Since we have switched to sync way to handle METADATA_UPDATED
      msg for md-cluster, then process_metadata_update is depended
      on mddev->thread->wqueue.
      
      With the new change, clustered raid could possible hang if
      array received a METADATA_UPDATED msg after array unregistered
      mddev->thread, so we need to stop clustered raid (bitmap_destroy
      -> bitmap_free -> md_cluster_stop) earlier than unregister
      thread (mddev_detach -> md_unregister_thread).
      
      And this change should be safe for non-clustered raid since
      all writes are stopped before the destroy. Also in md_run,
      we activate the personality (pers->run()) before activating
      the bitmap (bitmap_create()). So it is pleasingly symmetric
      to stop the bitmap (bitmap_destroy()) before stopping the
      personality (__md_stop() calls pers->free()), we achieve this
      by move bitmap_destroy to the beginning of __md_stop.
      
      But we don't want to break the codes for waiting behind IO as
      Shaohua mentioned, so introduce bitmap_wait_behind_writes to
      call the codes, and call the new fun in both mddev_detach and
      bitmap_destroy, then we will not break original behind IO code
      and also fit the new condition well.
      Signed-off-by: default avatarGuoqing Jiang <gqjiang@suse.com>
      Signed-off-by: default avatarShaohua Li <shli@fb.com>
      48df498d
    • Song Liu's avatar
      md/r5cache: generate R5LOG_PAYLOAD_FLUSH · ea17481f
      Song Liu authored
      In r5c_finish_stripe_write_out(), R5LOG_PAYLOAD_FLUSH is append to
      log->current_io.
      
      Appending R5LOG_PAYLOAD_FLUSH in quiesce needs extra writes to
      journal. To simplify the logic, we just skip R5LOG_PAYLOAD_FLUSH in
      quiesce.
      
      Even R5LOG_PAYLOAD_FLUSH supports multiple stripes per payload.
      However, current implementation is one stripe per R5LOG_PAYLOAD_FLUSH,
      which is simpler.
      Signed-off-by: default avatarSong Liu <songliubraving@fb.com>
      Signed-off-by: default avatarShaohua Li <shli@fb.com>
      ea17481f
    • Song Liu's avatar
      md/r5cache: handle R5LOG_PAYLOAD_FLUSH in recovery · 2d4f4687
      Song Liu authored
      This patch adds handling of R5LOG_PAYLOAD_FLUSH in journal recovery.
      Next patch will add logic that generate R5LOG_PAYLOAD_FLUSH on flush
      finish.
      
      When R5LOG_PAYLOAD_FLUSH is seen in recovery, pending data and parity
      will be dropped from recovery. This will reduce the number of stripes
      to replay, and thus accelerate the recovery process.
      Signed-off-by: default avatarSong Liu <songliubraving@fb.com>
      Signed-off-by: default avatarShaohua Li <shli@fb.com>
      2d4f4687
    • Artur Paszkiewicz's avatar
      raid5-ppl: runtime PPL enabling or disabling · ba903a3e
      Artur Paszkiewicz authored
      Allow writing to 'consistency_policy' attribute when the array is
      active. Add a new function 'change_consistency_policy' to the
      md_personality operations structure to handle the change in the
      personality code. Values "ppl" and "resync" are accepted and
      turn PPL on and off respectively.
      
      When enabling PPL its location and size should first be set using
      'ppl_sector' and 'ppl_size' attributes and a valid PPL header should be
      written at this location on each member device.
      
      Enabling or disabling PPL is performed under a suspended array.  The
      raid5_reset_stripe_cache function frees the stripe cache and allocates
      it again in order to allocate or free the ppl_pages for the stripes in
      the stripe cache.
      Signed-off-by: default avatarArtur Paszkiewicz <artur.paszkiewicz@intel.com>
      Signed-off-by: default avatarShaohua Li <shli@fb.com>
      ba903a3e
    • Artur Paszkiewicz's avatar
      raid5-ppl: support disk hot add/remove with PPL · 6358c239
      Artur Paszkiewicz authored
      Add a function to modify the log by removing an rdev when a drive fails
      or adding when a spare/replacement is activated as a raid member.
      
      Removing a disk just clears the child log rdev pointer. No new stripes
      will be accepted for this child log in ppl_write_stripe() and running io
      units will be processed without writing PPL to the device.
      
      Adding a disk sets the child log rdev pointer and writes an empty PPL
      header.
      Signed-off-by: default avatarArtur Paszkiewicz <artur.paszkiewicz@intel.com>
      Signed-off-by: default avatarShaohua Li <shli@fb.com>
      6358c239
    • Artur Paszkiewicz's avatar
      raid5-ppl: load and recover the log · 4536bf9b
      Artur Paszkiewicz authored
      Load the log from each disk when starting the array and recover if the
      array is dirty.
      
      The initial empty PPL is written by mdadm. When loading the log we
      verify the header checksum and signature. For external metadata arrays
      the signature is verified in userspace, so here we read it from the
      header, verifying only if it matches on all disks, and use it later when
      writing PPL.
      
      In addition to the header checksum, each header entry also contains a
      checksum of its partial parity data. If the header is valid, recovery is
      performed for each entry until an invalid entry is found. If the array
      is not degraded and recovery using PPL fully succeeds, there is no need
      to resync the array because data and parity will be consistent, so in
      this case resync will be disabled.
      
      Due to compatibility with IMSM implementations on other systems, we
      can't assume that the recovery data block size is always 4K. Writes
      generated by MD raid5 don't have this issue, but when recovering PPL
      written in other environments it is possible to have entries with
      512-byte sector granularity. The recovery code takes this into account
      and also the logical sector size of the underlying drives.
      Signed-off-by: default avatarArtur Paszkiewicz <artur.paszkiewicz@intel.com>
      Signed-off-by: default avatarShaohua Li <shli@fb.com>
      4536bf9b
    • Artur Paszkiewicz's avatar
      md: add sysfs entries for PPL · 664aed04
      Artur Paszkiewicz authored
      Add 'consistency_policy' attribute for array. It indicates how the array
      maintains consistency in case of unexpected shutdown.
      
      Add 'ppl_sector' and 'ppl_size' for rdev, which describe the location
      and size of the PPL space on the device. They can't be changed for
      active members if the array is started and PPL is enabled, so in the
      setter functions only basic checks are performed. More checks are done
      in ppl_validate_rdev() when starting the log.
      
      These attributes are writable to allow enabling PPL for external
      metadata arrays and (later) to enable/disable PPL for a running array.
      Signed-off-by: default avatarArtur Paszkiewicz <artur.paszkiewicz@intel.com>
      Signed-off-by: default avatarShaohua Li <shli@fb.com>
      664aed04
    • Artur Paszkiewicz's avatar
      raid5-ppl: Partial Parity Log write logging implementation · 3418d036
      Artur Paszkiewicz authored
      Implement the calculation of partial parity for a stripe and PPL write
      logging functionality. The description of PPL is added to the
      documentation. More details can be found in the comments in raid5-ppl.c.
      
      Attach a page for holding the partial parity data to stripe_head.
      Allocate it only if mddev has the MD_HAS_PPL flag set.
      
      Partial parity is the xor of not modified data chunks of a stripe and is
      calculated as follows:
      
      - reconstruct-write case:
        xor data from all not updated disks in a stripe
      
      - read-modify-write case:
        xor old data and parity from all updated disks in a stripe
      
      Implement it using the async_tx API and integrate into raid_run_ops().
      It must be called when we still have access to old data, so do it when
      STRIPE_OP_BIODRAIN is set, but before ops_run_prexor5(). The result is
      stored into sh->ppl_page.
      
      Partial parity is not meaningful for full stripe write and is not stored
      in the log or used for recovery, so don't attempt to calculate it when
      stripe has STRIPE_FULL_WRITE.
      
      Put the PPL metadata structures to md_p.h because userspace tools
      (mdadm) will also need to read/write PPL.
      
      Warn about using PPL with enabled disk volatile write-back cache for
      now. It can be removed once disk cache flushing before writing PPL is
      implemented.
      Signed-off-by: default avatarArtur Paszkiewicz <artur.paszkiewicz@intel.com>
      Signed-off-by: default avatarShaohua Li <shli@fb.com>
      3418d036
    • Artur Paszkiewicz's avatar
      raid5: separate header for log functions · ff875738
      Artur Paszkiewicz authored
      Move raid5-cache declarations from raid5.h to raid5-log.h, add inline
      wrappers for functions which will be shared with ppl and use them in
      raid5 core instead of direct calls to raid5-cache.
      
      Remove unused parameter from r5c_cache_data(), move two duplicated
      pr_debug() calls to r5l_init_log().
      Signed-off-by: default avatarArtur Paszkiewicz <artur.paszkiewicz@intel.com>
      Signed-off-by: default avatarShaohua Li <shli@fb.com>
      ff875738
    • Artur Paszkiewicz's avatar
      md: superblock changes for PPL · ea0213e0
      Artur Paszkiewicz authored
      Include information about PPL location and size into mdp_superblock_1
      and copy it to/from rdev. Because PPL is mutually exclusive with bitmap,
      put it in place of 'bitmap_offset'. Add a new flag MD_FEATURE_PPL for
      'feature_map', analogically to MD_FEATURE_BITMAP_OFFSET. Add MD_HAS_PPL
      to mddev->flags to indicate that PPL is enabled on an array.
      Signed-off-by: default avatarArtur Paszkiewicz <artur.paszkiewicz@intel.com>
      Signed-off-by: default avatarShaohua Li <shli@fb.com>
      ea0213e0
    • Song Liu's avatar
      md/r5cache: improve recovery with read ahead page pool · effe6ee7
      Song Liu authored
      In r5cache recovery, the journal device is scanned page by page.
      Currently, we use sync_page_io() to read journal device. This is
      not efficient when we have to recovery many stripes from the journal.
      
      To improve the speed of recovery, this patch introduces a read ahead
      page pool (ra_pool) to recovery_ctx. With ra_pool, multiple consecutive
      pages are read in one IO. Then the recovery code read the journal from
      ra_pool.
      
      With ra_pool, r5l_recovery_ctx has become much bigger. Therefore,
      r5l_recovery_log() is refactored so r5l_recovery_ctx is not using
      stack space.
      Signed-off-by: default avatarSong Liu <songliubraving@fb.com>
      Signed-off-by: default avatarShaohua Li <shli@fb.com>
      effe6ee7