• Pavel Begunkov's avatar
    io_uring: drain next sqe instead of shadowing · 1b4a51b6
    Pavel Begunkov authored
    There's an issue with the shadow drain logic in that we drop the
    completion lock after deciding to defer a request, then re-grab it later
    and assume that the state is still the same. In the mean time, someone
    else completing a request could have found and issued it. This can cause
    a stall in the queue, by having a shadow request inserted that nobody is
    going to drain.
    
    Additionally, if we fail allocating the shadow request, we simply ignore
    the drain.
    
    Instead of using a shadow request, defer the next request/link instead.
    This also has the following advantages:
    
    - removes semi-duplicated code
    - doesn't allocate memory for shadows
    - works better if only the head marked for drain
    - doesn't need complex synchronisation
    
    On the flip side, it removes the shadow->seq ==
    last_drain_in_in_link->seq optimization. That shouldn't be a common
    case, and can always be added back, if needed.
    
    Fixes: 4fe2c963 ("io_uring: add support for link with drain")
    Cc: Jackie Liu <liuyun01@kylinos.cn>
    Reported-by: default avatarJens Axboe <axboe@kernel.dk>
    Signed-off-by: default avatarPavel Begunkov <asml.silence@gmail.com>
    Signed-off-by: default avatarJens Axboe <axboe@kernel.dk>
    1b4a51b6
io_uring.c 117 KB