• Pavel Begunkov's avatar
    io_uring: skip request refcounting · 20e60a38
    Pavel Begunkov authored
    As submission references are gone, there is only one initial reference
    left. Instead of actually doing atomic refcounting, add a flag
    indicating whether we're going to take more refs or doing any other sync
    magic. The flag should be set before the request may get used in
    parallel.
    
    Together with the previous patch it saves 2 refcount atomics per request
    for IOPOLL and IRQ completions, and 1 atomic per req for inline
    completions, with some exceptions. In particular, currently, there are
    three cases, when the refcounting have to be enabled:
    - Polling, including apoll. Because double poll entries takes a ref.
      Might get relaxed in the near future.
    - Link timeouts, enabled for both, the timeout and the request it's
      bound to, because they work in-parallel and we need to synchronise
      to cancel one of them on completion.
    - When a request gets in io-wq, because it doesn't hold uring_lock and
      we need guarantees of submission references.
    Signed-off-by: default avatarPavel Begunkov <asml.silence@gmail.com>
    Link: https://lore.kernel.org/r/8b204b6c5f6643062270a1913d6d3a7f8f795fd9.1628705069.git.asml.silence@gmail.comSigned-off-by: default avatarJens Axboe <axboe@kernel.dk>
    20e60a38
io_uring.c 253 KB