1. 04 Feb, 2021 2 commits
    • Pavel Begunkov's avatar
      io_uring: drop mm/files between task_work_submit · aec18a57
      Pavel Begunkov authored
      Since SQPOLL task can be shared and so task_work entries can be a mix of
      them, we need to drop mm and files before trying to issue next request.
      
      Cc: stable@vger.kernel.org # 5.10+
      Signed-off-by: default avatarPavel Begunkov <asml.silence@gmail.com>
      Signed-off-by: default avatarJens Axboe <axboe@kernel.dk>
      aec18a57
    • Xiaoguang Wang's avatar
      io_uring: don't modify identity's files uncess identity is cowed · d7e10d47
      Xiaoguang Wang authored
      Abaci Robot reported following panic:
      BUG: kernel NULL pointer dereference, address: 0000000000000000
      PGD 800000010ef3f067 P4D 800000010ef3f067 PUD 10d9df067 PMD 0
      Oops: 0002 [#1] SMP PTI
      CPU: 0 PID: 1869 Comm: io_wqe_worker-0 Not tainted 5.11.0-rc3+ #1
      Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011
      RIP: 0010:put_files_struct+0x1b/0x120
      Code: 24 18 c7 00 f4 ff ff ff e9 4d fd ff ff 66 90 0f 1f 44 00 00 41 57 41 56 49 89 fe 41 55 41 54 55 53 48 83 ec 08 e8 b5 6b db ff  41 ff 0e 74 13 48 83 c4 08 5b 5d 41 5c 41 5d 41 5e 41 5f e9 9c
      RSP: 0000:ffffc90002147d48 EFLAGS: 00010293
      RAX: 0000000000000000 RBX: ffff88810d9a5300 RCX: 0000000000000000
      RDX: ffff88810d87c280 RSI: ffffffff8144ba6b RDI: 0000000000000000
      RBP: 0000000000000080 R08: 0000000000000001 R09: ffffffff81431500
      R10: ffff8881001be000 R11: 0000000000000000 R12: ffff88810ac2f800
      R13: ffff88810af38a00 R14: 0000000000000000 R15: ffff8881057130c0
      FS:  0000000000000000(0000) GS:ffff88813bc00000(0000) knlGS:0000000000000000
      CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
      CR2: 0000000000000000 CR3: 000000010dbaa002 CR4: 00000000003706f0
      DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
      DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
      Call Trace:
       __io_clean_op+0x10c/0x2a0
       io_dismantle_req+0x3c7/0x600
       __io_free_req+0x34/0x280
       io_put_req+0x63/0xb0
       io_worker_handle_work+0x60e/0x830
       ? io_wqe_worker+0x135/0x520
       io_wqe_worker+0x158/0x520
       ? __kthread_parkme+0x96/0xc0
       ? io_worker_handle_work+0x830/0x830
       kthread+0x134/0x180
       ? kthread_create_worker_on_cpu+0x90/0x90
       ret_from_fork+0x1f/0x30
      Modules linked in:
      CR2: 0000000000000000
      ---[ end trace c358ca86af95b1e7 ]---
      
      I guess case below can trigger above panic: there're two threads which
      operates different io_uring ctxs and share same sqthread identity, and
      later one thread exits, io_uring_cancel_task_requests() will clear
      task->io_uring->identity->files to be NULL in sqpoll mode, then another
      ctx that uses same identity will panic.
      
      Indeed we don't need to clear task->io_uring->identity->files here,
      io_grab_identity() should handle identity->files changes well, if
      task->io_uring->identity->files is not equal to current->files,
      io_cow_identity() should handle this changes well.
      
      Cc: stable@vger.kernel.org # 5.5+
      Reported-by: default avatarAbaci Robot <abaci@linux.alibaba.com>
      Signed-off-by: default avatarXiaoguang Wang <xiaoguang.wang@linux.alibaba.com>
      Reviewed-by: default avatarPavel Begunkov <asml.silence@gmail.com>
      Signed-off-by: default avatarJens Axboe <axboe@kernel.dk>
      d7e10d47
  2. 29 Jan, 2021 1 commit
    • Pavel Begunkov's avatar
      io_uring: reinforce cancel on flush during exit · 3a7efd1a
      Pavel Begunkov authored
      What 84965ff8 ("io_uring: if we see flush on exit, cancel related tasks")
      really wants is to cancel all relevant REQ_F_INFLIGHT requests reliably.
      That can be achieved by io_uring_cancel_files(), but we'll miss it
      calling io_uring_cancel_task_requests(files=NULL) from io_uring_flush(),
      because it will go through __io_uring_cancel_task_requests().
      
      Just always call io_uring_cancel_files() during cancel, it's good enough
      for now.
      
      Cc: stable@vger.kernel.org # 5.9+
      Signed-off-by: default avatarPavel Begunkov <asml.silence@gmail.com>
      Signed-off-by: default avatarJens Axboe <axboe@kernel.dk>
      3a7efd1a
  3. 28 Jan, 2021 2 commits
    • Pavel Begunkov's avatar
      io_uring: fix sqo ownership false positive warning · 70b2c60d
      Pavel Begunkov authored
      WARNING: CPU: 0 PID: 21359 at fs/io_uring.c:9042
          io_uring_cancel_task_requests+0xe55/0x10c0 fs/io_uring.c:9042
      Call Trace:
       io_uring_flush+0x47b/0x6e0 fs/io_uring.c:9227
       filp_close+0xb4/0x170 fs/open.c:1295
       close_files fs/file.c:403 [inline]
       put_files_struct fs/file.c:418 [inline]
       put_files_struct+0x1cc/0x350 fs/file.c:415
       exit_files+0x7e/0xa0 fs/file.c:435
       do_exit+0xc22/0x2ae0 kernel/exit.c:820
       do_group_exit+0x125/0x310 kernel/exit.c:922
       get_signal+0x427/0x20f0 kernel/signal.c:2773
       arch_do_signal_or_restart+0x2a8/0x1eb0 arch/x86/kernel/signal.c:811
       handle_signal_work kernel/entry/common.c:147 [inline]
       exit_to_user_mode_loop kernel/entry/common.c:171 [inline]
       exit_to_user_mode_prepare+0x148/0x250 kernel/entry/common.c:201
       __syscall_exit_to_user_mode_work kernel/entry/common.c:291 [inline]
       syscall_exit_to_user_mode+0x19/0x50 kernel/entry/common.c:302
       entry_SYSCALL_64_after_hwframe+0x44/0xa9
      
      Now io_uring_cancel_task_requests() can be called not through file
      notes but directly, remove a WARN_ONCE() there that give us false
      positives. That check is not very important and we catch it in other
      places.
      
      Fixes: 84965ff8 ("io_uring: if we see flush on exit, cancel related tasks")
      Cc: stable@vger.kernel.org # 5.9+
      Reported-by: syzbot+3e3d9bd0c6ce9efbc3ef@syzkaller.appspotmail.com
      Signed-off-by: default avatarPavel Begunkov <asml.silence@gmail.com>
      Signed-off-by: default avatarJens Axboe <axboe@kernel.dk>
      70b2c60d
    • Pavel Begunkov's avatar
      io_uring: fix list corruption for splice file_get · f609cbb8
      Pavel Begunkov authored
      kernel BUG at lib/list_debug.c:29!
      Call Trace:
       __list_add include/linux/list.h:67 [inline]
       list_add include/linux/list.h:86 [inline]
       io_file_get+0x8cc/0xdb0 fs/io_uring.c:6466
       __io_splice_prep+0x1bc/0x530 fs/io_uring.c:3866
       io_splice_prep fs/io_uring.c:3920 [inline]
       io_req_prep+0x3546/0x4e80 fs/io_uring.c:6081
       io_queue_sqe+0x609/0x10d0 fs/io_uring.c:6628
       io_submit_sqe fs/io_uring.c:6705 [inline]
       io_submit_sqes+0x1495/0x2720 fs/io_uring.c:6953
       __do_sys_io_uring_enter+0x107d/0x1f30 fs/io_uring.c:9353
       do_syscall_64+0x2d/0x70 arch/x86/entry/common.c:46
       entry_SYSCALL_64_after_hwframe+0x44/0xa9
      
      io_file_get() may be called from splice, and so REQ_F_INFLIGHT may
      already be set.
      
      Fixes: 02a13674 ("io_uring: account io_uring internal files as REQ_F_INFLIGHT")
      Cc: stable@vger.kernel.org # 5.9+
      Reported-by: syzbot+6879187cf57845801267@syzkaller.appspotmail.com
      Signed-off-by: default avatarPavel Begunkov <asml.silence@gmail.com>
      Signed-off-by: default avatarJens Axboe <axboe@kernel.dk>
      f609cbb8
  4. 27 Jan, 2021 2 commits
    • Hao Xu's avatar
      io_uring: fix flush cqring overflow list while TASK_INTERRUPTIBLE · 6195ba09
      Hao Xu authored
      Abaci reported the follow warning:
      
      [   27.073425] do not call blocking ops when !TASK_RUNNING; state=1 set at [] prepare_to_wait_exclusive+0x3a/0xc0
      [   27.075805] WARNING: CPU: 0 PID: 951 at kernel/sched/core.c:7853 __might_sleep+0x80/0xa0
      [   27.077604] Modules linked in:
      [   27.078379] CPU: 0 PID: 951 Comm: a.out Not tainted 5.11.0-rc3+ #1
      [   27.079637] Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011
      [   27.080852] RIP: 0010:__might_sleep+0x80/0xa0
      [   27.081835] Code: 65 48 8b 04 25 80 71 01 00 48 8b 90 c0 15 00 00 48 8b 70 18 48 c7 c7 08 39 95 82 c6 05 f9 5f de 08 01 48 89 d1 e8 00 c6 fa ff  0b eb bf 41 0f b6 f5 48 c7 c7 40 23 c9 82 e8 f3 48 ec 00 eb a7
      [   27.084521] RSP: 0018:ffffc90000fe3ce8 EFLAGS: 00010286
      [   27.085350] RAX: 0000000000000000 RBX: ffffffff82956083 RCX: 0000000000000000
      [   27.086348] RDX: ffff8881057a0000 RSI: ffffffff8118cc9e RDI: ffff88813bc28570
      [   27.087598] RBP: 00000000000003a7 R08: 0000000000000001 R09: 0000000000000001
      [   27.088819] R10: ffffc90000fe3e00 R11: 00000000fffef9f0 R12: 0000000000000000
      [   27.089819] R13: 0000000000000000 R14: ffff88810576eb80 R15: ffff88810576e800
      [   27.091058] FS:  00007f7b144cf740(0000) GS:ffff88813bc00000(0000) knlGS:0000000000000000
      [   27.092775] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
      [   27.093796] CR2: 00000000022da7b8 CR3: 000000010b928002 CR4: 00000000003706f0
      [   27.094778] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
      [   27.095780] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
      [   27.097011] Call Trace:
      [   27.097685]  __mutex_lock+0x5d/0xa30
      [   27.098565]  ? prepare_to_wait_exclusive+0x71/0xc0
      [   27.099412]  ? io_cqring_overflow_flush.part.101+0x6d/0x70
      [   27.100441]  ? lockdep_hardirqs_on_prepare+0xe9/0x1c0
      [   27.101537]  ? _raw_spin_unlock_irqrestore+0x2d/0x40
      [   27.102656]  ? trace_hardirqs_on+0x46/0x110
      [   27.103459]  ? io_cqring_overflow_flush.part.101+0x6d/0x70
      [   27.104317]  io_cqring_overflow_flush.part.101+0x6d/0x70
      [   27.105113]  io_cqring_wait+0x36e/0x4d0
      [   27.105770]  ? find_held_lock+0x28/0xb0
      [   27.106370]  ? io_uring_remove_task_files+0xa0/0xa0
      [   27.107076]  __x64_sys_io_uring_enter+0x4fb/0x640
      [   27.107801]  ? rcu_read_lock_sched_held+0x59/0xa0
      [   27.108562]  ? lockdep_hardirqs_on_prepare+0xe9/0x1c0
      [   27.109684]  ? syscall_enter_from_user_mode+0x26/0x70
      [   27.110731]  do_syscall_64+0x2d/0x40
      [   27.111296]  entry_SYSCALL_64_after_hwframe+0x44/0xa9
      [   27.112056] RIP: 0033:0x7f7b13dc8239
      [   27.112663] Code: 01 00 48 81 c4 80 00 00 00 e9 f1 fe ff ff 0f 1f 00 48 89 f8 48 89 f7 48 89 d6 48 89 ca 4d 89 c2 4d 89 c8 4c 8b 4c 24 08 0f 05  3d 01 f0 ff ff 73 01 c3 48 8b 0d 27 ec 2c 00 f7 d8 64 89 01 48
      [   27.115113] RSP: 002b:00007ffd6d7f5c88 EFLAGS: 00000286 ORIG_RAX: 00000000000001aa
      [   27.116562] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f7b13dc8239
      [   27.117961] RDX: 000000000000478e RSI: 0000000000000000 RDI: 0000000000000003
      [   27.118925] RBP: 00007ffd6d7f5cb0 R08: 0000000020000040 R09: 0000000000000008
      [   27.119773] R10: 0000000000000001 R11: 0000000000000286 R12: 0000000000400480
      [   27.120614] R13: 00007ffd6d7f5d90 R14: 0000000000000000 R15: 0000000000000000
      [   27.121490] irq event stamp: 5635
      [   27.121946] hardirqs last  enabled at (5643): [] console_unlock+0x5c4/0x740
      [   27.123476] hardirqs last disabled at (5652): [] console_unlock+0x4e7/0x740
      [   27.125192] softirqs last  enabled at (5272): [] __do_softirq+0x3c5/0x5aa
      [   27.126430] softirqs last disabled at (5267): [] asm_call_irq_on_stack+0xf/0x20
      [   27.127634] ---[ end trace 289d7e28fa60f928 ]---
      
      This is caused by calling io_cqring_overflow_flush() which may sleep
      after calling prepare_to_wait_exclusive() which set task state to
      TASK_INTERRUPTIBLE
      Reported-by: default avatarAbaci <abaci@linux.alibaba.com>
      Fixes: 6c503150 ("io_uring: patch up IOPOLL overflow_flush sync")
      Reviewed-by: default avatarPavel Begunkov <asml.silence@gmail.com>
      Signed-off-by: default avatarHao Xu <haoxu@linux.alibaba.com>
      Signed-off-by: default avatarJens Axboe <axboe@kernel.dk>
      6195ba09
    • Pavel Begunkov's avatar
      io_uring: fix wqe->lock/completion_lock deadlock · 907d1df3
      Pavel Begunkov authored
      Joseph reports following deadlock:
      
      CPU0:
      ...
      io_kill_linked_timeout  // &ctx->completion_lock
      io_commit_cqring
      __io_queue_deferred
      __io_queue_async_work
      io_wq_enqueue
      io_wqe_enqueue  // &wqe->lock
      
      CPU1:
      ...
      __io_uring_files_cancel
      io_wq_cancel_cb
      io_wqe_cancel_pending_work  // &wqe->lock
      io_cancel_task_cb  // &ctx->completion_lock
      
      Only __io_queue_deferred() calls queue_async_work() while holding
      ctx->completion_lock, enqueue drained requests via io_req_task_queue()
      instead.
      
      Cc: stable@vger.kernel.org # 5.9+
      Reported-by: default avatarJoseph Qi <joseph.qi@linux.alibaba.com>
      Tested-by: default avatarJoseph Qi <joseph.qi@linux.alibaba.com>
      Signed-off-by: default avatarPavel Begunkov <asml.silence@gmail.com>
      Signed-off-by: default avatarJens Axboe <axboe@kernel.dk>
      907d1df3
  5. 26 Jan, 2021 2 commits
    • Pavel Begunkov's avatar
      io_uring: fix cancellation taking mutex while TASK_UNINTERRUPTIBLE · ca70f00b
      Pavel Begunkov authored
      do not call blocking ops when !TASK_RUNNING; state=2 set at
      	[<00000000ced9dbfc>] prepare_to_wait+0x1f4/0x3b0
      	kernel/sched/wait.c:262
      WARNING: CPU: 1 PID: 19888 at kernel/sched/core.c:7853
      	__might_sleep+0xed/0x100 kernel/sched/core.c:7848
      RIP: 0010:__might_sleep+0xed/0x100 kernel/sched/core.c:7848
      Call Trace:
       __mutex_lock_common+0xc4/0x2ef0 kernel/locking/mutex.c:935
       __mutex_lock kernel/locking/mutex.c:1103 [inline]
       mutex_lock_nested+0x1a/0x20 kernel/locking/mutex.c:1118
       io_wq_submit_work+0x39a/0x720 fs/io_uring.c:6411
       io_run_cancel fs/io-wq.c:856 [inline]
       io_wqe_cancel_pending_work fs/io-wq.c:990 [inline]
       io_wq_cancel_cb+0x614/0xcb0 fs/io-wq.c:1027
       io_uring_cancel_files fs/io_uring.c:8874 [inline]
       io_uring_cancel_task_requests fs/io_uring.c:8952 [inline]
       __io_uring_files_cancel+0x115d/0x19e0 fs/io_uring.c:9038
       io_uring_files_cancel include/linux/io_uring.h:51 [inline]
       do_exit+0x2e6/0x2490 kernel/exit.c:780
       do_group_exit+0x168/0x2d0 kernel/exit.c:922
       get_signal+0x16b5/0x2030 kernel/signal.c:2770
       arch_do_signal_or_restart+0x8e/0x6a0 arch/x86/kernel/signal.c:811
       handle_signal_work kernel/entry/common.c:147 [inline]
       exit_to_user_mode_loop kernel/entry/common.c:171 [inline]
       exit_to_user_mode_prepare+0xac/0x1e0 kernel/entry/common.c:201
       __syscall_exit_to_user_mode_work kernel/entry/common.c:291 [inline]
       syscall_exit_to_user_mode+0x48/0x190 kernel/entry/common.c:302
       entry_SYSCALL_64_after_hwframe+0x44/0xa9
      
      Rewrite io_uring_cancel_files() to mimic __io_uring_task_cancel()'s
      counting scheme, so it does all the heavy work before setting
      TASK_UNINTERRUPTIBLE.
      
      Cc: stable@vger.kernel.org # 5.9+
      Reported-by: syzbot+f655445043a26a7cfab8@syzkaller.appspotmail.com
      Signed-off-by: default avatarPavel Begunkov <asml.silence@gmail.com>
      [axboe: fix inverted task check]
      Signed-off-by: default avatarJens Axboe <axboe@kernel.dk>
      ca70f00b
    • Pavel Begunkov's avatar
      io_uring: fix __io_uring_files_cancel() with TASK_UNINTERRUPTIBLE · a1bb3cd5
      Pavel Begunkov authored
      If the tctx inflight number haven't changed because of cancellation,
      __io_uring_task_cancel() will continue leaving the task in
      TASK_UNINTERRUPTIBLE state, that's not expected by
      __io_uring_files_cancel(). Ensure we always call finish_wait() before
      retrying.
      
      Cc: stable@vger.kernel.org # 5.9+
      Signed-off-by: default avatarPavel Begunkov <asml.silence@gmail.com>
      Signed-off-by: default avatarJens Axboe <axboe@kernel.dk>
      a1bb3cd5
  6. 25 Jan, 2021 3 commits
  7. 24 Jan, 2021 28 commits