• Xiaoguang Wang's avatar
    io_uring: fix poll_list race for SETUP_IOPOLL|SETUP_SQPOLL · bdcd3eab
    Xiaoguang Wang authored
    After making ext4 support iopoll method:
      let ext4_file_operations's iopoll method be iomap_dio_iopoll(),
    we found fio can easily hang in fio_ioring_getevents() with below fio
    job:
        rm -f testfile; sync;
        sudo fio -name=fiotest -filename=testfile -iodepth=128 -thread
    -rw=write -ioengine=io_uring  -hipri=1 -sqthread_poll=1 -direct=1
    -bs=4k -size=10G -numjobs=8 -runtime=2000 -group_reporting
    with IORING_SETUP_SQPOLL and IORING_SETUP_IOPOLL enabled.
    
    There are two issues that results in this hang, one reason is that
    when IORING_SETUP_SQPOLL and IORING_SETUP_IOPOLL are enabled, fio
    does not use io_uring_enter to get completed events, it relies on
    kernel io_sq_thread to poll for completed events.
    
    Another reason is that there is a race: when io_submit_sqes() in
    io_sq_thread() submits a batch of sqes, variable 'inflight' will
    record the number of submitted reqs, then io_sq_thread will poll for
    reqs which have been added to poll_list. But note, if some previous
    reqs have been punted to io worker, these reqs will won't be in
    poll_list timely. io_sq_thread() will only poll for a part of previous
    submitted reqs, and then find poll_list is empty, reset variable
    'inflight' to be zero. If app just waits these deferred reqs and does
    not wake up io_sq_thread again, then hang happens.
    
    For app that entirely relies on io_sq_thread to poll completed requests,
    let io_iopoll_req_issued() wake up io_sq_thread properly when adding new
    element to poll_list, and when io_sq_thread prepares to sleep, check
    whether poll_list is empty again, if not empty, continue to poll.
    Signed-off-by: default avatarXiaoguang Wang <xiaoguang.wang@linux.alibaba.com>
    Signed-off-by: default avatarJens Axboe <axboe@kernel.dk>
    bdcd3eab
io_uring.c 169 KB