Instead of explicitly checking ->cq_wait_nr for whether there are
waiting, which is currently represented by 0, we can store there a
large value and the nr_tw will automatically filter out those cases.
Add a named constant for that and for the wake up bias value.
Signed-off-by: Pavel Begunkov <asml.silence@gmail.com>
Link: https://lore.kernel.org/r/38def30282654d980673976cd42fde9bab19b297.1705438669.git.asml.silence@gmail.comSigned-off-by: Jens Axboe <axboe@kernel.dk>

if (!first) { ... }

While it reads as do something if it's not the first entry, it does
exactly the opposite because "first" here is a pointer to the first
entry. Remove the confusion by renaming it into "head".
Signed-off-by: Pavel Begunkov <asml.silence@gmail.com>
Link: https://lore.kernel.org/r/3b8be483b52f58a524185bb88694b8a268e7e85d.1705438669.git.asml.silence@gmail.comSigned-off-by: Jens Axboe <axboe@kernel.dk>

Kill a smp_mb__after_atomic() right before wake_up, it's useless, and
add a comment explaining implicit barriers from cmpxchg and
synchronsation around ->cq_wait_nr with the waiter.
Signed-off-by: Pavel Begunkov <asml.silence@gmail.com>
Link: https://lore.kernel.org/r/3007f3c2d53c72b61de56919ef56b53158b8276f.1705438669.git.asml.silence@gmail.comSigned-off-by: Jens Axboe <axboe@kernel.dk>

The UINT_MAX work item counting bias in io_req_local_work_add() in case
of !IOU_F_TWQ_LAZY_WAKE works in a sense that we will not miss a wake up,
however it's still eerie. In particular, if we add a lazy work item
after a non-lazy one, we'll increment it and get nr_tw==0, and
subsequent adds may try to unnecessarily wake up the task, which is
though not so likely to happen in real workloads.

Half the bias, it's still large enough to be larger than any valid
->cq_wait_nr, which is limited by IORING_MAX_CQ_ENTRIES, but further
have a good enough of space before it overflows.

Fixes: 8751d154 ("io_uring: reduce scheduling due to tw")
Signed-off-by: Pavel Begunkov <asml.silence@gmail.com>
Link: https://lore.kernel.org/r/108b971e958deaf7048342930c341ba90f75d806.1705438669.git.asml.silence@gmail.comSigned-off-by: Jens Axboe <axboe@kernel.dk>

Add compat.h include to avoid a potential build issue:

io_uring/register.c:281:6: error: call to undeclared function 'in_compat_syscall'; ISO C99 and later do not support implicit function declarations [-Werror,-Wimplicit-function-declaration]

if (in_compat_syscall()) {
    ^
1 warning generated.
io_uring/register.c:282:9: error: call to undeclared function 'compat_get_bitmap'; ISO C99 and later do not support implicit function declarations [-Werror,-Wimplicit-function-declaration]
ret = compat_get_bitmap(cpumask_bits(new_mask),
      ^

Fixes: c4320315 ("io_uring/register: move io_uring_register(2) related code to register.c")
Reported-by: Manu Bretelle <chantra@meta.com>
Reviewed-by: Jeff Moyer <jmoyer@redhat.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>

For the normal read/write path, we have already locked the ring
submission side when assigning the file. This causes branch
mispredictions when we then check and try and lock again in
io_req_set_rsrc_node(). As this is a very hot path, this matters.

Add a basic helper that already assumes we already have it locked,
and use that in io_file_get_fixed().
Signed-off-by: Jens Axboe <axboe@kernel.dk>

This originally came from the aio side, and it's laid out rather oddly.
The common case here is that we either get -EIOCBQUEUED from submitting
an async request, or that we complete the request correctly with the
given number of bytes. Handling the odd internal restart error codes
is not a common operation.

Lay it out a bit more optimally that better explains the normal flow,
and switch to avoiding the indirect call completely as this is our
kiocb and we know the completion handler can only be one of two
possible variants. While at it, move it to where it belongs in the
file, with fellow end IO helpers.

Outside of being easier to read, this also reduces the text size of the
function by 24 bytes for me on arm64.
Reviewed-by: Keith Busch <kbusch@kernel.org>
Signed-off-by: Jens Axboe <axboe@kernel.dk>

A previous commit added an earlier break condition here, which is fine if
we're using non-local task_work as it'll be run on return to userspace.
However, if DEFER_TASKRUN is used, then we could be leaving local
task_work that is ready to process in the ctx list until next time that
we enter the kernel to wait for events.

Move the break condition to _after_ we have run task_work.

Cc: stable@vger.kernel.org
Fixes: 846072f1 ("io_uring: mimimise io_cqring_wait_schedule")
Signed-off-by: Jens Axboe <axboe@kernel.dk>

The tail of the provided ring buffer is shared between the kernel and
the application, but the head is private to the kernel as the
application doesn't need to see it. However, this also prevents the
application from knowing how many buffers the kernel has consumed.
Usually this is fine, as the information is inherently racy in that
the kernel could be consuming buffers continually, but for cleanup
purposes it may be relevant to know how many buffers are still left
in the ring.

Add IORING_REGISTER_PBUF_STATUS which will return status for a given
provided buffer ring. Right now it just returns the head, but space
is reserved for more information later in, if needed.

Link: https://github.com/axboe/liburing/discussions/1020Signed-off-by: Jens Axboe <axboe@kernel.dk>

If IOSQE_ASYNC is set and we fail importing an iovec for a readv or
writev request, then we leave ->bytes_done uninitialized and hence the
eventual failure CQE posted can potentially have a random res value
rather than the expected -EINVAL.

Setup ->bytes_done before potentially failing, so we have a consistent
value if we fail the request early.

Cc: stable@vger.kernel.org
Reported-by: xingwei lee <xrivendell7@gmail.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>

This is dead code after we dropped support for passing io_uring fds
over SCM_RIGHTS, get rid of it.
Signed-off-by: Jens Axboe <axboe@kernel.dk>

Since we no longer allow sending io_uring fds over SCM_RIGHTS, move to
using io_is_uring_fops() to detect whether this is a io_uring fd or not.
With that done, kill off io_uring_get_socket() as nobody calls it
anymore.

This is in preparation to yanking out the rest of the core related to
unix gc with io_uring.
Signed-off-by: Jens Axboe <axboe@kernel.dk>

Most of this code is basically self contained, move it out of the core
io_uring file to bring a bit more separation to the registration related
bits. This moves another ~10% of the code into register.c.
Signed-off-by: Jens Axboe <axboe@kernel.dk>

io_uring can currently open/close regular files or fixed/direct
descriptors. Or you can instantiate a fixed descriptor from a regular
one, and then close the regular descriptor. But you currently can't turn
a purely fixed/direct descriptor into a regular file descriptor.

IORING_OP_FIXED_FD_INSTALL adds support for installing a direct
descriptor into the normal file table, just like receiving a file
descriptor or opening a new file would do. This is all nicely abstracted
into receive_fd(), and hence adding support for this is truly trivial.

Since direct descriptors are only usable within io_uring itself, it can
be useful to turn them into real file descriptors if they ever need to
be accessed via normal syscalls. This can either be a transitory thing,
or just a permanent transition for a given direct descriptor.

By default, new fds are installed with O_CLOEXEC set. The application
can disable O_CLOEXEC by setting IORING_FIXED_FD_NO_CLOEXEC in the
sqe->install_fd_flags member.
Suggested-by: Christian Brauner <brauner@kernel.org>
Reviewed-by: Christian Brauner <brauner@kernel.org>
Signed-off-by: Jens Axboe <axboe@kernel.dk>

With io_uring_types.h we see all required definitions to inline
io_uring_cmd_get_task().
Signed-off-by: Pavel Begunkov <asml.silence@gmail.com>
Reviewed-by: Ming Lei <ming.lei@redhat.com>
Link: https://lore.kernel.org/r/aa8e317f09e651a5f3e72f8c0ad3902084c1f930.1701391955.git.asml.silence@gmail.comSigned-off-by: Jens Axboe <axboe@kernel.dk>

Now as we can easily include io_uring_types.h, move IOU_F_TWQ_LAZY_WAKE
and inline io_uring_cmd_do_in_task_lazy().
Signed-off-by: Pavel Begunkov <asml.silence@gmail.com>
Reviewed-by: Ming Lei <ming.lei@redhat.com>
Link: https://lore.kernel.org/r/2ec9fb31dd192d1c5cf26d0a2dec5657d88a8e48.1701391955.git.asml.silence@gmail.comSigned-off-by: Jens Axboe <axboe@kernel.dk>

linux/io_uring.h is slowly becoming a rubbish bin where we put
anything exposed to other subsystems. For instance, the task exit
hooks and io_uring cmd infra are completely orthogonal and don't need
each other's definitions. Start cleaning it up by splitting out all
command bits into a new header file.
Signed-off-by: Pavel Begunkov <asml.silence@gmail.com>
Link: https://lore.kernel.org/r/7ec50bae6e21f371d3850796e716917fc141225a.1701391955.git.asml.silence@gmail.comSigned-off-by: Jens Axboe <axboe@kernel.dk>

At one point in time we had an optimisation that would not spin up a
linked timeout timer when the master request successfully completes
inline (during the first nowait execution attempt). We somehow lost it,
so this patch restores it back.

Note, that it's fine using io_arm_ltimeout() after the io_issue_sqe()
completes the request because of delayed completion, but that that adds
unwanted overhead.
Reported-by: Christian Mazakas <christian.mazakas@gmail.com>
Signed-off-by: Pavel Begunkov <asml.silence@gmail.com>
Link: https://lore.kernel.org/r/8bf69c2a4beec14c565c85c86edb871ca8b8bcc8.1701390926.git.asml.silence@gmail.comSigned-off-by: Jens Axboe <axboe@kernel.dk>

IOPOLL request should never return IOU_OK, so the following iopoll
queueing check in io_issue_sqe() after getting IOU_OK doesn't make any
sense as would never turn true. Let's optimise on that and return a bit
earlier. It's also much more resilient to potential bugs from
mischieving iopoll implementations.

Cc:  <stable@vger.kernel.org>
Signed-off-by: Pavel Begunkov <asml.silence@gmail.com>
Link: https://lore.kernel.org/r/2f8690e2fa5213a2ff292fac29a7143c036cdd60.1701390926.git.asml.silence@gmail.comSigned-off-by: Jens Axboe <axboe@kernel.dk>

Merge branch 'vfs.file' of git://git.kernel.org/pub/scm/linux/kernel/git/vfs/vfs into for-6.8/io_uring

Merge vfs.file from the VFS tree to avoid conflicts with receive_fd() now
having 3 arguments rather than just 2.

* 'vfs.file' of git://git.kernel.org/pub/scm/linux/kernel/git/vfs/vfs:
  file: remove __receive_fd()
  file: stop exposing receive_fd_user()
  fs: replace f_rcuhead with f_task_work
  file: remove pointless wrapper
  file: s/close_fd_get_file()/file_close_fd()/g
  Improve __fget_files_rcu() code generation (and thus __fget_light())
  file: massage cleanup of files that failed to open

Honestly, there's little value in having a helper with and without that
int __user *ufd argument. It's just messy and doesn't really give us
anything. Just expose receive_fd() with that argument and get rid of
that helper.

Link: https://lore.kernel.org/r/20231130-vfs-files-fixes-v1-5-e73ca6f4ea83@kernel.orgReviewed-by: Jan Kara <jack@suse.cz>
Reviewed-by: Jens Axboe <axboe@kernel.dk>
Signed-off-by: Christian Brauner <brauner@kernel.org>

Not every subsystem needs to have their own specialized helper.
Just us the __receive_fd() helper.

Link: https://lore.kernel.org/r/20231130-vfs-files-fixes-v1-4-e73ca6f4ea83@kernel.orgReviewed-by: Jan Kara <jack@suse.cz>
Reviewed-by: Jens Axboe <axboe@kernel.dk>
Signed-off-by: Christian Brauner <brauner@kernel.org>

The naming is actively misleading since we switched to
SLAB_TYPESAFE_BY_RCU. rcu_head is #define callback_head. Use
callback_head directly and rename f_rcuhead to f_task_work.

Add comments in there to explain what it's used for.

Link: https://lore.kernel.org/r/20231130-vfs-files-fixes-v1-3-e73ca6f4ea83@kernel.orgReviewed-by: Jan Kara <jack@suse.cz>
Reviewed-by: Jens Axboe <axboe@kernel.dk>
Signed-off-by: Christian Brauner <brauner@kernel.org>

Only io_uring uses __close_fd_get_file(). All it does is hide
current->files but io_uring accesses files_struct directly right now
anyway so it's a bit pointless. Just rename pick_file() to
file_close_fd_locked() and let io_uring use it. Add a lockdep assert in
there that we expect the caller to hold file_lock while we're at it.

Link: https://lore.kernel.org/r/20231130-vfs-files-fixes-v1-2-e73ca6f4ea83@kernel.orgReviewed-by: Jens Axboe <axboe@kernel.dk>
Reviewed-by: Jan Kara <jack@suse.cz>
Signed-off-by: Christian Brauner <brauner@kernel.org>

That really shouldn't have "get" in there as that implies we're bumping
the reference count which we don't do at all. We used to but not anmore.
Now we're just closing the fd and pick that file from the fdtable
without bumping the reference count. Update the wrong documentation
while at it.

Link: https://lore.kernel.org/r/20231130-vfs-files-fixes-v1-1-e73ca6f4ea83@kernel.orgReviewed-by: Jan Kara <jack@suse.cz>
Reviewed-by: Jens Axboe <axboe@kernel.dk>
Signed-off-by: Christian Brauner <brauner@kernel.org>

Commit 0ede61d8 ("file: convert to SLAB_TYPESAFE_BY_RCU") caused a
performance regression as reported by the kernel test robot.

The __fget_light() function is one of those critical ones for some
loads, and the code generation was unnecessarily impacted.  Let's just
write that function to better.
Reported-by: kernel test robot <oliver.sang@intel.com>
Cc: Christian Brauner <brauner@kernel.org>
Cc: Jann Horn <jannh@google.com>
Cc: Mateusz Guzik <mjguzik@gmail.com>
Closes: https://lore.kernel.org/oe-lkp/202311201406.2022ca3f-oliver.sang@intel.comSigned-off-by: Linus Torvalds <torvalds@linux-foundation.org>
Link: https://lore.kernel.org/r/CAHk-=wiCJtLbFWNURB34b9a_R_unaH3CiMRXfkR0-iihB_z68A@mail.gmail.comSigned-off-by: Christian Brauner <brauner@kernel.org>

A file that has never gotten FMODE_OPENED will never have RCU-accessed
references, its final fput() is equivalent to file_free() and if it
doesn't have FMODE_BACKING either, it can be done from any context and
won't need task_work treatment.

Now that we have SLAB_TYPESAFE_BY_RCU we can simplify this and have
other callers benefit. All of that can be achieved easier is to make
fput() recoginze that case and call file_free() directly.

No need to introduce a special primitive for that. It also allowed
things like failing dentry_open() could benefit from that as well.
Signed-off-by: Al Viro <viro@zeniv.linux.org.uk>
[Christian Brauner <brauner@kernel.org>: massage commit message]
Link: https://lore.kernel.org/r/20231126020834.GC38156@ZenIVSigned-off-by: Christian Brauner <brauner@kernel.org>

Pull scheduler fix from Borislav Petkov:

 - Make sure tasks are thawed exactly and only once to avoid their state
   getting corrupted

* tag 'sched_urgent_for_v6.7_rc5' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip:
  freezer,sched: Do not restore saved_state of a thawed task

Pull perf event fix from Borislav Petkov:

 - Make sure perf event size validation is done on every event in the
   group

* tag 'perf_urgent_for_v6.7_rc5' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip:
  perf: Fix perf_event_validate_size()

Pull x86 fixes from Borislav Petkov:

 - Add a forgotten CPU vendor check in the AMD microcode post-loading
   callback so that the callback runs only on AMD

 - Make sure SEV-ES protocol negotiation happens only once and on the
   BSP

* tag 'x86_urgent_for_v6.7_rc5' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip:
  x86/CPU/AMD: Check vendor in the AMD microcode callback
  x86/sev: Fix kernel crash due to late update to read-only ghcb_version

Pull kvm fixes from Paolo Bonzini:
 "Generic:

   - Set .owner for various KVM file_operations so that files refcount
     the KVM module until KVM is done executing _all_ code, including
     the last few instructions of kvm_put_kvm(). And then revert the
     misguided attempt to rely on "struct kvm" refcounts to pin
     KVM-the-module.

  ARM:

   - Do not redo the mapping of vLPIs, if they have already been mapped

  s390:

   - Do not leave bits behind in PTEs

   - Properly catch page invalidations that affect the prefix of a
     nested guest

  x86:

   - When checking if a _running_ vCPU is "in-kernel", i.e. running at
     CPL0, get the CPL directly instead of relying on
     preempted_in_kernel (which is valid if and only if the vCPU was
     preempted, i.e. NOT running).

   - Fix a benign "return void" that was recently introduced.

  Selftests:

   - Makefile tweak for dependency generation

   - '-Wformat' fix"

* tag 'for-linus' of git://git.kernel.org/pub/scm/virt/kvm/kvm:
  KVM: SVM: Update EFER software model on CR0 trap for SEV-ES
  KVM: selftests: add -MP to CFLAGS
  KVM: selftests: Actually print out magic token in NX hugepages skip message
  KVM: x86: Remove 'return void' expression for 'void function'
  Revert "KVM: Prevent module exit until all VMs are freed"
  KVM: Set file_operations.owner appropriately for all such structures
  KVM: x86: Get CPL directly when checking if loaded vCPU is in kernel mode
  KVM: arm64: GICv4: Do not perform a map to a mapped vLPI
  KVM: s390/mm: Properly reset no-dat
  KVM: s390: vsie: fix wrong VIR 37 when MSO is used

Pull powerpc fix from Michael Ellerman:

 - Fix stack teardown in ftrace_no_trace, seen as crashes doing CPU
   hotplug while ftrace is active.

   Thanks to Naveen N Rao.

* tag 'powerpc-6.7-4' of git://git.kernel.org/pub/scm/linux/kernel/git/powerpc/linux:
  powerpc/ftrace: Fix stack teardown in ftrace_no_trace

Pull gpio fix from Bartosz Golaszewski:

 - fix an error path after a failed export in sysfs code

* tag 'gpio-fixes-for-v6.7-rc5' of git://git.kernel.org/pub/scm/linux/kernel/git/brgl/linux:
  gpiolib: sysfs: Fix error handling on failed export

Pull USB fixes from Greg KH:
 "Here are some small USB fixes for 6.7-rc5 to resolve some reported
  issues. Included in here are:

   - usb gadget f_hid, and uevent fix

   - xhci driver revert to resolve a much-reported issue

   - typec driver fix

  All of these have been in linux-next with no reported issues"

* tag 'usb-6.7-rc5' of git://git.kernel.org/pub/scm/linux/kernel/git/gregkh/usb:
  usb: gadget: f_hid: fix report descriptor allocation
  Revert "xhci: Loosen RPM as default policy to cover for AMD xHC 1.1"
  usb: typec: class: fix typec_altmode_put_partner to put plugs
  USB: gadget: core: adjust uevent timing on gadget unbind

Pull serial driver fixes from Greg KH:
 "Here are some small serial driver fixes for 6.7-rc4 to resolve some
  reported issues. Included in here are:

   - pl011 dma support fix

   - sc16is7xx driver fix

   - ma35d1 console index fix

   - 8250 driver fixes for small issues

  All of these have been in linux-next with no reported issues"

* tag 'tty-6.7-rc4' of git://git.kernel.org/pub/scm/linux/kernel/git/gregkh/tty:
  serial: 8250_dw: Add ACPI ID for Granite Rapids-D UART
  serial: ma35d1: Validate console index before assignment
  ARM: PL011: Fix DMA support
  serial: sc16is7xx: address RX timeout interrupt errata
  serial: 8250: 8250_omap: Clear UART_HAS_RHR_IT_DIS bit
  serial: 8250_omap: Add earlycon support for the AM654 UART controller
  serial: 8250: 8250_omap: Do not start RX DMA on THRI interrupt

Pull char / misc driver fixes from Greg KH:
 "Here are some small fixes for 6.7-rc5 for a variety of small driver
  subsystems. Included in here are:

   - debugfs revert for reported issue

   - greybus revert for reported issue

   - greybus fixup for endian build warning

   - coresight driver fixes

   - nvmem driver fixes

   - devcoredump fix

   - parport new device id

   - ndtest build fix

  All of these have ben in linux-next with no reported issues"

* tag 'char-misc-6.7-rc5' of git://git.kernel.org/pub/scm/linux/kernel/git/gregkh/char-misc:
  nvmem: Do not expect fixed layouts to grab a layout driver
  parport: Add support for Brainboxes IX/UC/PX parallel cards
  Revert "greybus: gb-beagleplay: Ensure le for values in transport"
  greybus: gb-beagleplay: Ensure le for values in transport
  greybus: BeaglePlay driver needs CRC_CCITT
  Revert "debugfs: annotate debugfs handlers vs. removal with lockdep"
  devcoredump: Send uevent once devcd is ready
  ndtest: fix typo class_regster -> class_register
  misc: mei: client.c: fix problem of return '-EOVERFLOW' in mei_cl_write
  misc: mei: client.c: return negative error code in mei_cl_write
  mei: pxp: fix mei_pxp_send_message return value
  coresight: ultrasoc-smb: Fix uninitialized before use buf_hw_base
  coresight: ultrasoc-smb: Config SMB buffer before register sink
  coresight: ultrasoc-smb: Fix sleep while close preempt in enable_smb
  Documentation: coresight: fix `make refcheckdocs` warning
  hwtracing: hisi_ptt: Don't try to attach a task
  hwtracing: hisi_ptt: Handle the interrupt in hardirq context
  hwtracing: hisi_ptt: Add dummy callback pmu::read()
  coresight: Fix crash when Perf and sysfs modes are used concurrently
  coresight: etm4x: Remove bogous __exit annotation for some functions

Merge tag 'loongarch-fixes-6.7-2' of git://git.kernel.org/pub/scm/linux/kernel/git/chenhuacai/linux-loongson

Pull LoongArch fixes from Huacai Chen:
 "Preserve syscall nr across execve(), slightly clean up drdtime(), fix
  the Clang built zboot kernel, fix a stack unwinder bug and several bpf
  jit bugs"

* tag 'loongarch-fixes-6.7-2' of git://git.kernel.org/pub/scm/linux/kernel/git/chenhuacai/linux-loongson:
  LoongArch: BPF: Fix unconditional bswap instructions
  LoongArch: BPF: Fix sign-extension mov instructions
  LoongArch: BPF: Don't sign extend function return value
  LoongArch: BPF: Don't sign extend memory load operand
  LoongArch: Preserve syscall nr across execve()
  LoongArch: Set unwind stack type to unknown rather than set error flag
  LoongArch: Slightly clean up drdtime()
  LoongArch: Apply dynamic relocations for LLD

Pull MIPS fixes from Thomas Bogendoerfer:

 - Fixes for broken Loongson firmware

 - Fix lockdep splat

 - Fix FPU states when creating kernel threads

* tag 'mips-fixes_6.7_1' of git://git.kernel.org/pub/scm/linux/kernel/git/mips/linux:
  MIPS: kernel: Clear FPU states when setting up kernel threads
  MIPS: Loongson64: Handle more memory types passed from firmware
  MIPS: Loongson64: Enable DMA noncoherent support
  MIPS: Loongson64: Reserve vgabios memory on boot
  mips/smp: Call rcutree_report_cpu_starting() earlier

Merge tag 'perf-tools-fixes-for-v6.7-2-2023-12-08' of git://git.kernel.org/pub/scm/linux/kernel/git/perf/perf-tools

Pull perf tools fixes from Namhyung Kim:
 "A random set of small bug fixes including:

   - Fix segfault on AmpereOne due to missing default metricgroup name

   - Fix segfault on `perf list --json` due to NULL pointer"

* tag 'perf-tools-fixes-for-v6.7-2-2023-12-08' of git://git.kernel.org/pub/scm/linux/kernel/git/perf/perf-tools:
  perf list: Fix JSON segfault by setting the used skip_duplicate_pmus callback
  perf vendor events arm64: AmpereOne: Add missing DefaultMetricgroupName fields
  perf metrics: Avoid segv if default metricgroup isn't set