• Yu Kuai's avatar
    md/raid5: fix a deadlock in the case that reshape is interrupted · 868bba54
    Yu Kuai authored
    If reshape is in progress and io across reshape_position is issued, such
    io will wait for reshape to make progress(see details in the case that
    make_stripe_request() return STRIPE_SCHEDULE_AND_RETRY).
    
    It has been reported several times that if system reboot while growing
    raid5 to raid6, array assemble will hang infinitely([1, 2]). This is
    because following deadlock is triggered:
    
    1) a normal io is waiting for reshape to progress, this io can be from
       system-udevd or mdadm.
    2) while assemble, mdadm tries to suspend the array, hence
       'reconfig_mutex' is held and mddev_suspend() must wait for normal io
       to be done.
    3) daemon thread can't start reshape because 'reconfig_mutex' can't be
       held.
    
    1) and 3) is unbreakable because they're foundation design. In order to
    break 2), following is possible solutions that I can think of:
    
    a) Let mddev_suspend() fail is not a good option, because this will
       break many scenarios since mddev_suspend() doesn't fail before.
    b) Fail the io that is waiting for reshape to make progress from
       mddev_suspend().
    c) Return false for the io that is waiting for reshape to make
       progress from raid5_make_request(), and these io will wait for
       suspend to be done in md_handle_request(), where 'active_io' is
       not grabbed.
    
    c) sounds better than b), however, b) is used because it's easy and
    straightforward, and it's verified that mdadm can assemble in this case.
    On the other hand, c) breaks the logic that mddev_suspend() will wait
    for submitted io to be completely handled.
    
    Fix the problem by checking reshape in mddev_suspend(), if reshape can't
    make progress and there are still some io waiting for reshape, fail
    those io.
    
    [1] https://lore.kernel.org/all/CAFig2csUV2QiomUhj_t3dPOgV300dbQ6XtM9ygKPdXJFSH__Nw@mail.gmail.com/
    [2] https://lore.kernel.org/all/CAO2ABipzbw6QL5eNa44CQHjiVa-LTvS696Mh9QaTw+qsUKFUCw@mail.gmail.com/Reported-by: default avatarJove <jovetoo@gmail.com>
    Reported-by: default avatarDavid Gilmour <dgilmour76@gmail.com>
    Signed-off-by: default avatarYu Kuai <yukuai3@huawei.com>
    Signed-off-by: default avatarSong Liu <song@kernel.org>
    Link: https://lore.kernel.org/r/20230512015610.821290-6-yukuai1@huaweicloud.com
    868bba54
md.c 260 KB