Skip to content

L
linux
Switch branch/tag
  1. 05 Jul, 2020 2 commits
    • Jens Axboe's avatar
      io_uring: abstract out task work running · 4c6e277c
      Jens Axboe authored 
      Provide a helper to run task_work instead of checking and running
manually in a bunch of different spots. While doing so, also move the
task run state setting where we run the task work. Then we can move it
out of the callback helpers. This also helps ensure we only do this once
per task_work list run, not per task_work item.
Suggested-by: default avatarOleg Nesterov <oleg@redhat.com>
Signed-off-by: default avatarJens Axboe <axboe@kernel.dk>
      4c6e277c
    • Jens Axboe's avatar
      Merge branch 'io_uring-5.8' into for-5.9/io_uring · 58c6a581
      Jens Axboe authored 
      Pull in task_work changes from the 5.8 series, as we'll need to apply
the same kind of changes to other parts in the 5.9 branch.

* io_uring-5.8:
  io_uring: fix regression with always ignoring signals in io_cqring_wait()
  io_uring: use signal based task_work running
  task_work: teach task_work_add() to do signal_wake_up()
      58c6a581
  3. 04 Jul, 2020 1 commit
    • Jens Axboe's avatar
      io_uring: fix regression with always ignoring signals in io_cqring_wait() · b7db41c9
      Jens Axboe authored 
      When switching to TWA_SIGNAL for task_work notifications, we also made
any signal based condition in io_cqring_wait() return -ERESTARTSYS.
This breaks applications that rely on using signals to abort someone
waiting for events.

Check if we have a signal pending because of queued task_work, and
repeat the signal check once we've run the task_work. This provides a
reliable way of telling the two apart.

Additionally, only use TWA_SIGNAL if we are using an eventfd. If not,
we don't have the dependency situation described in the original commit,
and we can get by with just using TWA_RESUME like we previously did.

Fixes: ce593a6c ("io_uring: use signal based task_work running")
Cc: stable@vger.kernel.org # v5.7
Reported-by: default avatarAndres Freund <andres@anarazel.de>
Tested-by: default avatarAndres Freund <andres@anarazel.de>
Signed-off-by: default avatarJens Axboe <axboe@kernel.dk>
      b7db41c9
  5. 30 Jun, 2020 17 commits
  7. 29 Jun, 2020 1 commit
  9. 28 Jun, 2020 14 commits
  11. 27 Jun, 2020 1 commit
    • Randy Dunlap's avatar
      io_uring: fix function args for !CONFIG_NET · 1e16c2f9
      Randy Dunlap authored 
      Fix build errors when CONFIG_NET is not set/enabled:

../fs/io_uring.c:5472:10: error: too many arguments to function ‘io_sendmsg’
../fs/io_uring.c:5474:10: error: too many arguments to function ‘io_send’
../fs/io_uring.c:5484:10: error: too many arguments to function ‘io_recvmsg’
../fs/io_uring.c:5486:10: error: too many arguments to function ‘io_recv’
../fs/io_uring.c:5510:9: error: too many arguments to function ‘io_accept’
../fs/io_uring.c:5518:9: error: too many arguments to function ‘io_connect’
Signed-off-by: default avatarRandy Dunlap <rdunlap@infradead.org>
Cc: Jens Axboe <axboe@kernel.dk>
Cc: io-uring@vger.kernel.org
Signed-off-by: default avatarJens Axboe <axboe@kernel.dk>
      1e16c2f9
  13. 26 Jun, 2020 4 commits
    • Jens Axboe's avatar
      Merge branch 'io_uring-5.8' into for-5.9/io_uring · 2237d765
      Jens Axboe authored 
      Merge in changes that went into 5.8-rc3. GIT will silently do the
merge, but we still need a tweak on top of that since
io_complete_rw_common() was modified to take a io_comp_state pointer.
The auto-merge fails on that, and we end up with something that
doesn't compile.

* io_uring-5.8:
  io_uring: fix current->mm NULL dereference on exit
  io_uring: fix hanging iopoll in case of -EAGAIN
  io_uring: fix io_sq_thread no schedule when busy
Signed-off-by: default avatarJens Axboe <axboe@kernel.dk>
      2237d765
    • Pavel Begunkov's avatar
      io-wq: return next work from ->do_work() directly · f4db7182
      Pavel Begunkov authored 
      It's easier to return next work from ->do_work() than
having an in-out argument. Looks nicer and easier to compile.
Also, merge io_wq_assign_next() into its only user.
Signed-off-by: default avatarPavel Begunkov <asml.silence@gmail.com>
Signed-off-by: default avatarJens Axboe <axboe@kernel.dk>
      f4db7182
    • Pavel Begunkov's avatar
      io-wq: compact io-wq flags numbers · e883a79d
      Pavel Begunkov authored 
      Renumerate IO_WQ flags, so they take adjacent bits
Signed-off-by: default avatarPavel Begunkov <asml.silence@gmail.com>
Signed-off-by: default avatarJens Axboe <axboe@kernel.dk>
      e883a79d
    • Jens Axboe's avatar
      io_uring: use task_work for links if possible · c40f6379
      Jens Axboe authored 
      Currently links are always done in an async fashion, unless we catch them
inline after we successfully complete a request without having to resort
to blocking. This isn't necessarily the most efficient approach, it'd be
more ideal if we could just use the task_work handling for this.

Outside of saving an async jump, we can also do less prep work for these
kinds of requests.

Running dependent links from the task_work handler yields some nice
performance benefits. As an example, examples/link-cp from the liburing
repository uses read+write links to implement a copy operation. Without
this patch, the a cache fold 4G file read from a VM runs in about 3
seconds:

$ time examples/link-cp /data/file /dev/null

real	0m2.986s
user	0m0.051s
sys	0m2.843s

and a subsequent cache hot run looks like this:

$ time examples/link-cp /data/file /dev/null

real	0m0.898s
user	0m0.069s
sys	0m0.797s

With this patch in place, the cold case takes about 2.4 seconds:

$ time examples/link-cp /data/file /dev/null

real	0m2.400s
user	0m0.020s
sys	0m2.366s

and the cache hot case looks like this:

$ time examples/link-cp /data/file /dev/null

real	0m0.676s
user	0m0.010s
sys	0m0.665s

As expected, the (mostly) cache hot case yields the biggest improvement,
running about 25% faster with this change, while the cache cold case
yields about a 20% increase in performance. Outside of the performance
increase, we're using less CPU as well, as we're not using the async
offload threads at all for this anymore.
Signed-off-by: default avatarJens Axboe <axboe@kernel.dk>
      c40f6379
GitLab Nexedi Edition | About GitLab | About Nexedi | 沪ICP备2021021310号-2 | 沪ICP备2021021310号-7