- 25 Nov, 2022 7 commits
-
-
Dylan Yudaken authored
The only call sites which would not allow overflow are also call sites which would use the io_aux_cqe as they care about ordering. So remove this parameter from io_post_aux_cqe. Signed-off-by:
Dylan Yudaken <dylany@meta.com> Link: https://lore.kernel.org/r/20221124093559.3780686-9-dylany@meta.comSigned-off-by:
Jens Axboe <axboe@kernel.dk>
-
Dylan Yudaken authored
Add an assertion for the completion lock to io_fill_cqe_aux Signed-off-by:
-
Dylan Yudaken authored
This is only used in io_uring.c Signed-off-by:
-
Dylan Yudaken authored
Use the just introduced deferred post cqe completion state when possible in io_aux_cqe. If not possible fallback to io_post_aux_cqe. This introduces a complication because of allow_overflow. For deferred completions we cannot know without locking the completion_lock if it will overflow (and even if we locked it, another post could sneak in and cause this cqe to be in overflow). However since overflow protection is mostly a best effort defence in depth to prevent infinite loops of CQEs for poll, just checking the overflow bit is going to be good enough and will result in at most 16 (array size of deferred cqes) overflows. Suggested-by:
Pavel Begunkov <asml.silence@gmail.com> Signed-off-by:
-
Dylan Yudaken authored
Multishot ops cannot use the compl_reqs list as the request must stay in the poll list, but that means they need to run each completion without benefiting from batching. Here introduce batching infrastructure for only small (ie 16 byte) CQEs. This restriction is ok because there are no use cases posting 32 byte CQEs. In the ring keep a batch of up to 16 posted results, and flush in the same way as compl_reqs. 16 was chosen through experimentation on a microbenchmark ([1]), as well as trying not to increase the size of the ring too much. This increases the size to 1472 bytes from 1216. [1]: https://github.com/DylanZA/liburing/commit/9ac66b36bcf4477bfafeff1c5f107896b7ae31cf Run with $ make -j && ./benchmark/reg.b -s 1 -t 2000 -r 10 Gives results: baseline 8309 k/s 8 18807 k/s 16 19338 k/s 32 20134 k/s Suggested-by:
-
Dylan Yudaken authored
All failures happen under lock now, and can be deferred. To be consistent when the failure has happened after some multishot cqe has been deferred (and keep ordering), always defer failures. To make this obvious at the caller (and to help prevent a future bug) rename io_req_complete_failed to io_req_defer_failed. Signed-off-by:
-
Dylan Yudaken authored
This is required for the failure case (io_req_complete_failed) and is missing. The alternative would be to only lock in the failure path, however all of the non-error paths in io_poll_check_events that do not do not return IOU_POLL_NO_ACTION end up locking anyway. The only extraneous lock would be for the multishot poll overflowing the CQE ring, however multishot poll would probably benefit from being locked as it will allow completions to be batched. So it seems reasonable to lock always. Signed-off-by:
-
- 23 Nov, 2022 7 commits
-
-
Pavel Begunkov authored
This reverts commit 2ccc92f4 io_req_complete_post() should now behave well even in case of IOPOLL, we can remove completion_lock locking. Signed-off-by:
-
Pavel Begunkov authored
io_req_complete_post() may be used by iopoll enabled rings, grab locks in this case. That requires to pass issue_flags to propagate the locking state. Signed-off-by:
-
Pavel Begunkov authored
Inline __io_req_complete_put() into io_req_complete_post(), there are no other users. Signed-off-by:
-
Pavel Begunkov authored
Remove io_req_tw_post() and io_req_tw_post_queue(), we can use io_req_task_complete() instead. Signed-off-by:
-
Pavel Begunkov authored
Use a more generic io_req_task_complete() in timeout completion task_work instead of io_req_complete_post(). Signed-off-by:
-
Pavel Begunkov authored
A preparation patch, make sure we always hold uring_lock around io_req_complete_failed(). The only place deviating from the rule is io_cancel_defer_files(), queue a tw instead. Signed-off-by:
-
Pavel Begunkov authored
There are pieces of code that may allow iopoll to race filling cqes, temporarily add spinlocking around posting events. Cc: stable@vger.kernel.org Signed-off-by:
-
- 22 Nov, 2022 4 commits
-
-
Jens Axboe authored
__io_cq_unlock_post() is identical to io_cq_unlock_post(), and io_cqring_ev_posted() has a single caller so migth as well just inline it there. Signed-off-by: -
Jens Axboe authored
This reverts commit 7fdbc5f0. This patch dealt with a subset of the real problem, which is a potential circular dependency on the wakup path for io_uring itself. Outside of io_uring, eventfd can also trigger this (see details in 03e02acd) and so can epoll (see details in caf1aeaf). Now that we have a generic solution to this problem, get rid of the io_uring specific work-around. Signed-off-by:
-
Jens Axboe authored
Pass in EPOLL_URING_WAKE when signaling eventfd or doing poll related wakups, so that we can check for a circular event dependency between eventfd and epoll. If this flag is set when our wakeup handlers are called, then we know we have a dependency that needs to terminate multishot requests. eventfd and epoll are the only such possible dependencies. Cc: stable@vger.kernel.org # 6.0 Signed-off-by: -
Jens Axboe authored
This is identical to eventfd_signal(), but it allows the caller to pass in a mask to be used for the poll wakeup key. The use case is avoiding repeated multishot triggers if we have a dependency between eventfd and io_uring. If we setup an eventfd context and register that as the io_uring eventfd, and at the same time queue a multishot poll request for the eventfd context, then any CQE posted will repeatedly trigger the multishot request until it terminates when the CQ ring overflows. In preparation for io_uring detecting this circular dependency, add the mentioned helper so that io_uring can pass in EPOLL_URING as part of the poll wakeup key. Cc: stable@vger.kernel.org # 6.0 [axboe: fold in !CONFIG_EVENTFD fix from Zhang Qilong] Signed-off-by:
-
- 21 Nov, 2022 19 commits
-
-
Jens Axboe authored
We can have dependencies between epoll and io_uring. Consider an epoll context, identified by the epfd file descriptor, and an io_uring file descriptor identified by iofd. If we add iofd to the epfd context, and arm a multishot poll request for epfd with iofd, then the multishot poll request will repeatedly trigger and generate events until terminated by CQ ring overflow. This isn't a desired behavior. Add EPOLL_URING so that io_uring can pass it in as part of the poll wakeup key, and io_uring can check for that to detect a potential recursive invocation. Cc: stable@vger.kernel.org # 6.0 Signed-off-by: -
Pavel Begunkov authored
There is only one user of __io_req_complete_post(), inline it. Signed-off-by:
-
Pavel Begunkov authored
When the target process is dying and so task_work_add() is not allowed we push all task_work item to the fallback workqueue. Move the part responsible for moving tw items out of __io_req_task_work_add() into a separate function. Makes it a bit cleaner and gives the compiler a bit of extra info. Signed-off-by:
-
Pavel Begunkov authored
__io_req_task_work_add() is huge but marked inline, that makes compilers to generate lots of garbage. Inline the wrapper caller io_req_task_work_add() instead. before and after: text data bss dec hex filename 47347 16248 8 63603 f873 io_uring/io_uring.o text data bss dec hex filename 45303 16248 8 61559 f077 io_uring/io_uring.o Signed-off-by:
-
Lin Ma authored
Previous commit ebc11b6c ("io_uring: clean io-wq callbacks") rename io_free_work() into io_wq_free_work() for consistency. This patch also updates relevant comment to avoid misunderstanding. Fixes: ebc11b6c ("io_uring: clean io-wq callbacks") Signed-off-by:
Lin Ma <linma@zju.edu.cn> Link: https://lore.kernel.org/r/20221110122103.20120-1-linma@zju.edu.cnSigned-off-by:
-
Lin Ma authored
Previous commit 13a99017 ("io_uring: remove events caching atavisms") entirely removes the events caching optimization introduced by commit 81459350 ("io_uring: cache req->apoll->events in req->cflags"). Hence the related comment should also be removed to avoid misunderstanding. Fixes: 13a99017 ("io_uring: remove events caching atavisms") Signed-off-by:
-
Dylan Yudaken authored
With commit aa1df3a3 ("io_uring: fix CQE reordering"), there are stronger guarantees for overflow ordering. Specifically ensuring that userspace will not receive out of order receive CQEs. Therefore this is not needed any more for recv/recvmsg. Signed-off-by:
-
Dylan Yudaken authored
This is no longer needed after commit aa1df3a3 ("io_uring: fix CQE reordering"), since all reordering is now taken care of. This reverts commit cbd25748 ("io_uring: fix multishot accept ordering"). Signed-off-by:
-
Dylan Yudaken authored
Running task work when not needed can unnecessarily delay operations. Specifically IORING_SETUP_DEFER_TASKRUN tries to avoid running task work until the user requests it. Therefore do not run it in io_uring_register any more. The one catch is that io_rsrc_ref_quiesce expects it to have run in order to process all outstanding references, and so reorder it's loop to do this. Signed-off-by:
-
Xinghui Li authored
Fixes two errors: "ERROR: do not use assignment in if condition 130: FILE: io_uring/net.c:130: + if (!(issue_flags & IO_URING_F_UNLOCKED) && ERROR: do not use assignment in if condition 599: FILE: io_uring/poll.c:599: + } else if (!(issue_flags & IO_URING_F_UNLOCKED) &&" reported by checkpatch.pl in net.c and poll.c . Signed-off-by:
Xinghui Li <korantli@tencent.com> Reported-by:
kernel test robot <lkp@intel.com> Link: https://lore.kernel.org/r/20221102082503.32236-1-korantwork@gmail.com [axboe: style tweaks] Signed-off-by:
-
Pavel Begunkov authored
We can also move mm accounting to the extended callbacks. It removes a few cycles from the hot path including skipping one function call and setting io_req_task_complete as a callback directly. For user backed I/O it shouldn't make any difference taking into considering atomic mm accounting and page pinning. Signed-off-by:
-
Pavel Begunkov authored
Add custom tw and notif callbacks on top of usual bits also handling zc reporting. That moves it from the hot path. Signed-off-by:
-
Pavel Begunkov authored
io_notif_flush() is pretty simple, we can inline it. Signed-off-by:
-
Pavel Begunkov authored
Just a simple renaming patch, io_uring_tx_zerocopy_callback() is too bulky and doesn't follow usual naming style. Signed-off-by:
-
Pavel Begunkov authored
We're going to have multiple notification tw functions. In preparation for future changes default the tw callback in advance so later we can replace it with other versions. Signed-off-by:
-
Pavel Begunkov authored
io_send_zc_prep() sets up notification's rsrc_node when needed, don't unconditionally install it on notif alloc. Signed-off-by:
-
Pavel Begunkov authored
There are multiple users of io_req_task_complete() including zc notifications, but only read requests use selected buffers. As we already have an rw specific tw function, move io_put_kbuf() in there. Signed-off-by:
-
Stefan Metzmacher authored
It might be useful for applications to detect if a zero copy transfer with SEND[MSG]_ZC was actually possible or not. The application can fallback to plain SEND[MSG] in order to avoid the overhead of two cqes per request. Or it can generate a log message that could indicate to an administrator that no zero copy was possible and could explain degraded performance. Cc: stable@vger.kernel.org # 6.1 Link: https://lore.kernel.org/io-uring/fb6a7599-8a9b-15e5-9b64-6cd9d01c6ff4@gmail.com/T/#m2b0d9df94ce43b0e69e6c089bdff0ce6babbdfaaSigned-off-by:
Stefan Metzmacher <metze@samba.org> Reviewed-by:
-
Linus Torvalds authored
-
- 20 Nov, 2022 3 commits
-
-
git://git.kernel.org/pub/scm/linux/kernel/git/trace/linux-traceLinus Torvalds authored
Pull tracing/probes fixes from Steven Rostedt: - Fix possible NULL pointer dereference on trace_event_file in kprobe_event_gen_test_exit() - Fix NULL pointer dereference for trace_array in kprobe_event_gen_test_exit() - Fix memory leak of filter string for eprobes - Fix a possible memory leak in rethook_alloc() - Skip clearing aggrprobe's post_handler in kprobe-on-ftrace case which can cause a possible use-after-free - Fix warning in eprobe filter creation - Fix eprobe filter creation as it picked the wrong event for the fields * tag 'trace-probes-v6.1' of git://git.kernel.org/pub/scm/linux/kernel/git/trace/linux-trace: tracing/eprobe: Fix eprobe filter to make a filter correctly tracing/eprobe: Fix warning in filter creation kprobes: Skip clearing aggrprobe's post_handler in kprobe-on-ftrace case rethook: fix a potential memleak in rethook_alloc() tracing/eprobe: Fix memory leak of filter string tracing: kprobe: Fix potential null-ptr-deref on trace_array in kprobe_event_gen_test_exit() tracing: kprobe: Fix potential null-ptr-deref on trace_event_file in kprobe_event_gen_test_exit()
-
git://git.kernel.org/pub/scm/linux/kernel/git/trace/linux-traceLinus Torvalds authored
Pull tracing fixes from Steven Rostedt: - Fix polling to block on watermark like the reads do, as user space applications get confused when the select says read is available, and then the read blocks - Fix accounting of ring buffer dropped pages as it is what is used to determine if the buffer is empty or not - Fix memory leak in tracing_read_pipe() - Fix struct trace_array warning about being declared in parameters - Fix accounting of ftrace pages used in output at start up. - Fix allocation of dyn_ftrace pages by subtracting one from order instead of diving it by 2 - Static analyzer found a case were a pointer being used outside of a NULL check (rb_head_page_deactivate()) - Fix possible NULL pointer dereference if kstrdup() fails in ftrace_add_mod() - Fix memory leak in test_gen_synth_cmd() and test_empty_synth_event() - Fix bad pointer dereference in register_synth_event() on error path - Remove unused __bad_type_size() method - Fix possible NULL pointer dereference of entry in list 'tr->err_log' - Fix NULL pointer deference race if eprobe is called before the event setup * tag 'trace-v6.1-rc5' of git://git.kernel.org/pub/scm/linux/kernel/git/trace/linux-trace: tracing: Fix race where eprobes can be called before the event tracing: Fix potential null-pointer-access of entry in list 'tr->err_log' tracing: Remove unused __bad_type_size() method tracing: Fix wild-memory-access in register_synth_event() tracing: Fix memory leak in test_gen_synth_cmd() and test_empty_synth_event() ftrace: Fix null pointer dereference in ftrace_add_mod() ring_buffer: Do not deactivate non-existant pages ftrace: Optimize the allocation for mcount entries ftrace: Fix the possible incorrect kernel message tracing: Fix warning on variable 'struct trace_array' tracing: Fix memory leak in tracing_read_pipe() ring-buffer: Include dropped pages in counting dirty patches tracing/ring-buffer: Have polling block on watermark
-
Steven Rostedt (Google) authored
The flag that tells the event to call its triggers after reading the event is set for eprobes after the eprobe is enabled. This leads to a race where the eprobe may be triggered at the beginning of the event where the record information is NULL. The eprobe then dereferences the NULL record causing a NULL kernel pointer bug. Test for a NULL record to keep this from happening. Link: https://lore.kernel.org/linux-trace-kernel/20221116192552.1066630-1-rafaelmendsr@gmail.com/ Link: https://lore.kernel.org/linux-trace-kernel/20221117214249.2addbe10@gandalf.local.home Cc: Linux Trace Kernel <linux-trace-kernel@vger.kernel.org> Cc: Tzvetomir Stoyanov <tz.stoyanov@gmail.com> Cc: Tom Zanussi <zanussi@kernel.org> Cc: stable@vger.kernel.org Fixes: 7491e2c4 ("tracing: Add a probe that attaches to trace events") Acked-by:
Masami Hiramatsu (Google) <mhiramat@kernel.org> Reported-by:
Rafael Mendonca <rafaelmendsr@gmail.com> Signed-off-by:
Steven Rostedt (Google) <rostedt@goodmis.org>
-
