There are only two calls in source code of io_iter_do_read(), the
function is small and pretty hot though is failed to get inlined.
Makr it as inline.
Signed-off-by: Pavel Begunkov <asml.silence@gmail.com>
Link: https://lore.kernel.org/r/25a26dae7660da73fbc2244b361b397ef43d3caf.1623634182.git.asml.silence@gmail.comSigned-off-by: Jens Axboe <axboe@kernel.dk>

Merge io_uring_cancel_sqpoll() and __io_uring_cancel() as it's easier to
have a conditional ctx traverse inside than keeping them in sync.
Signed-off-by: Pavel Begunkov <asml.silence@gmail.com>
Link: https://lore.kernel.org/r/adfe24d6dad4a3883a40eee54352b8b65ac851bb.1623634181.git.asml.silence@gmail.comSigned-off-by: Jens Axboe <axboe@kernel.dk>

tctx in submission part is always synchronised because is executed from
the task's context, so we can batch allocate tctx/task references and
store them across syscall boundaries. It avoids enough of operations,
including an atomic for getting task ref and a percpu_counter_add()
function call, which still fallback to spinlock for large batching
cases (around >=32). Should be good for SQPOLL submitting in small
portions and coming at some moment bpf submissions.
Signed-off-by: Pavel Begunkov <asml.silence@gmail.com>
Link: https://lore.kernel.org/r/14b327b973410a3eec1f702ecf650e100513aca9.1623634181.git.asml.silence@gmail.comSigned-off-by: Jens Axboe <axboe@kernel.dk>

We don't really need vmalloc for keeping tags, it's not a hot path and
is there out of convenience, so replace it with two level tables to not
litter kernel virtual memory mappings.
Signed-off-by: Pavel Begunkov <asml.silence@gmail.com>
Link: https://lore.kernel.org/r/241a3422747113a8909e7e1030eb585d4a349e0d.1623634181.git.asml.silence@gmail.comSigned-off-by: Jens Axboe <axboe@kernel.dk>

Some parts like fixed file table use 2 level tables, factor out helpers
for allocating/deallocating them as more users are to come.
Signed-off-by: Pavel Begunkov <asml.silence@gmail.com>
Link: https://lore.kernel.org/r/1709212359cd82eb416d395f86fc78431ccfc0aa.1623634181.git.asml.silence@gmail.comSigned-off-by: Jens Axboe <axboe@kernel.dk>

io_rsrc_put_work() is executed by workqueue in non-irq context, so no
need for irqsave/restore variants of spinlocking.
Signed-off-by: Pavel Begunkov <asml.silence@gmail.com>
Link: https://lore.kernel.org/r/2a7f77220735f4ad404ac885b4d73bdf42d2f836.1623634181.git.asml.silence@gmail.comSigned-off-by: Jens Axboe <axboe@kernel.dk>

Make io_rsrc_data_alloc() taking care of rsrc tags loading on
registration, so we don't need to repeat it for each new rsrc type.
Signed-off-by: Pavel Begunkov <asml.silence@gmail.com>
Link: https://lore.kernel.org/r/5609680697bd09735de10561b75edb95283459da.1623634181.git.asml.silence@gmail.comSigned-off-by: Jens Axboe <axboe@kernel.dk>

io_worker_handle_work() already takes care of the empty list case and
releases spinlock, so get rid of ugly conditional unlocking and
unconditionally call handle_work()
Signed-off-by: Pavel Begunkov <asml.silence@gmail.com>
Link: https://lore.kernel.org/r/7521e485677f381036676943e876a0afecc23017.1623634181.git.asml.silence@gmail.comSigned-off-by: Jens Axboe <axboe@kernel.dk>

io_wqe_worker()'s main loop does check IO_WQ_BIT_EXIT flag, so no need
for a second test_bit at the end as it will immediately jump to the
first check afterwards.
Signed-off-by: Pavel Begunkov <asml.silence@gmail.com>
Link: https://lore.kernel.org/r/d6af4a51c86523a527fb5417c9fbc775c4b26497.1623634181.git.asml.silence@gmail.comSigned-off-by: Jens Axboe <axboe@kernel.dk>

What at some moment was references to struct file used to control
lifetimes of task/ctx is now just internal tctx structures/nodes,
so rename outdated *task_file() routines into something more sensible.
Signed-off-by: Pavel Begunkov <asml.silence@gmail.com>
Link: https://lore.kernel.org/r/e2fbce42932154c2631ce58ffbffaa232afe18d5.1623634181.git.asml.silence@gmail.comSigned-off-by: Jens Axboe <axboe@kernel.dk>

A simple refactoring of io_iopoll_req_issued(), move in_async inside so
we don't pass it around and save on double checking it.
Signed-off-by: Pavel Begunkov <asml.silence@gmail.com>
Link: https://lore.kernel.org/r/1513bfde4f0c835be25ac69a82737ab0668d7665.1623634181.git.asml.silence@gmail.comSigned-off-by: Jens Axboe <axboe@kernel.dk>

iowq->refs is initialised to one and killed on exit, so it's not used
and we can kill it.
Signed-off-by: Pavel Begunkov <asml.silence@gmail.com>
Link: https://lore.kernel.org/r/401007393528ea7c102360e69a29b64498e15db2.1623634181.git.asml.silence@gmail.comSigned-off-by: Jens Axboe <axboe@kernel.dk>

io-wq keeps an array of pointers to struct io_wqe, allocate this array
as a part of struct io-wq, it's easier to code and saves an extra
indirection for nearly each io-wq call.
Signed-off-by: Pavel Begunkov <asml.silence@gmail.com>
Link: https://lore.kernel.org/r/1482c6a001923bbed662dc38a8a580fb08b1ed8c.1623634181.git.asml.silence@gmail.comSigned-off-by: Jens Axboe <axboe@kernel.dk>

There is a complaint against sys_io_uring_enter() blocking if it submits
stdin reads. The problem is in __io_file_supports_async(), which
sees that it's a cdev and allows it to be processed inline.

Punt char devices using generic rules of io_file_supports_async(),
including checking for presence of *_iter() versions of rw callbacks.
Apparently, it will affect most of cdevs with some exceptions like
null and zero devices.

Cc: stable@vger.kernel.org
Reported-by: Birk Hirdman <lonjil@gmail.com>
Signed-off-by: Pavel Begunkov <asml.silence@gmail.com>
Link: https://lore.kernel.org/r/d60270856b8a4560a639ef5f76e55eb563633599.1623236455.git.asml.silence@gmail.comSigned-off-by: Jens Axboe <axboe@kernel.dk>

Relax buffer registration restictions, which filters out file backed
memory, and allow shmem/memfd as they have normal anonymous pages
underneath.
Signed-off-by: Pavel Begunkov <asml.silence@gmail.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>

struct io_submit_state contains struct io_comp_state and so
locked_free_*, that renders cachelines around ->locked_free* being
invalidated on most non-inline completions, that may terrorise caches if
submissions and completions are done by different tasks.
Signed-off-by: Pavel Begunkov <asml.silence@gmail.com>
Link: https://lore.kernel.org/r/290cb5412b76892e8631978ee8ab9db0c6290dd5.1621201931.git.asml.silence@gmail.comSigned-off-by: Jens Axboe <axboe@kernel.dk>

Rename io_get_cqring() into io_get_cqe() for consistency with SQ, and
just because the old name is not as clear.
Signed-off-by: Pavel Begunkov <asml.silence@gmail.com>
Link: https://lore.kernel.org/r/a46a53e3f781de372f5632c184e61546b86515ce.1621201931.git.asml.silence@gmail.comSigned-off-by: Jens Axboe <axboe@kernel.dk>

There are two copies of cq_overflow, shared with userspace and internal
cached one. It was needed for DRAIN accounting, but now we have yet
another knob to tune the accounting, i.e. cq_extra, and we can throw
away the internal counter and just increment the one in the shared ring.

If user modifies it as so never gets the right overflow value ever
again, it's its problem, even though before we would have restored it
back by next overflow.
Signed-off-by: Pavel Begunkov <asml.silence@gmail.com>
Link: https://lore.kernel.org/r/8427965f5175dd051febc63804909861109ce859.1621201931.git.asml.silence@gmail.comSigned-off-by: Jens Axboe <axboe@kernel.dk>

No need to cache cq_mask, it's exactly cq_entries - 1, so just deduce
it to not carry it around.
Signed-off-by: Pavel Begunkov <asml.silence@gmail.com>
Link: https://lore.kernel.org/r/d439efad0503c8398451dae075e68a04362fbc8d.1621201931.git.asml.silence@gmail.comSigned-off-by: Jens Axboe <axboe@kernel.dk>

We have numbers of {sq,cq} entries cached in ctx, don't look up them in
user-shared rings as 1) it may fetch additional cacheline 2) user may
change it and so it's always error prone.
Signed-off-by: Pavel Begunkov <asml.silence@gmail.com>
Link: https://lore.kernel.org/r/745d31bc2da41283ddd0489ef784af5c8d6310e9.1621201931.git.asml.silence@gmail.comSigned-off-by: Jens Axboe <axboe@kernel.dk>

ring has two types of resource-related fields: used for request
submission, and field needed for update/registration. Reshuffle them
into these two groups for better locality and readability. The second
group is not in the hot path, so it's natural to place them somewhere in
the end. Also update an outdated comment.
Signed-off-by: Pavel Begunkov <asml.silence@gmail.com>
Link: https://lore.kernel.org/r/05b34795bb4440f4ec4510f08abd5a31830f8ca0.1621201931.git.asml.silence@gmail.comSigned-off-by: Jens Axboe <axboe@kernel.dk>

There is a bunch of scattered around ctx fields that are almost never
used, e.g. only on ring exit, plunge them to the end, better locality,
better aesthetically.
Signed-off-by: Pavel Begunkov <asml.silence@gmail.com>
Link: https://lore.kernel.org/r/782ff94b00355923eae757d58b1a47821b5b46d4.1621201931.git.asml.silence@gmail.comSigned-off-by: Jens Axboe <axboe@kernel.dk>

The main difference is in req_set_fail_links() renamed into
req_set_fail(), which now sets REQ_F_FAIL_LINK/REQ_F_FAIL flag
unconditional on whether it has been a link or not. It only matters in
io_disarm_next(), which already handles it well, and all calls to it
have a fast path checking REQ_F_LINK/HARDLINK.

It looks cleaner, and sheds binary size
   text    data     bss     dec     hex filename
  84235   12390       8   96633   17979 ./fs/io_uring.o
  84151   12414       8   96573   1793d ./fs/io_uring.o
Signed-off-by: Pavel Begunkov <asml.silence@gmail.com>
Link: https://lore.kernel.org/r/e2224154dd6e53b665ac835d29436b177872fa10.1621201931.git.asml.silence@gmail.comSigned-off-by: Jens Axboe <axboe@kernel.dk>

We don't match against files on cancellation anymore, so no need to drag
around files_struct anymore, just pass a flag telling whether only
inflight or all requests should be killed.
Signed-off-by: Pavel Begunkov <asml.silence@gmail.com>
Link: https://lore.kernel.org/r/7bfc5409a78f8e2d6b27dec3293ec2d248677348.1621201931.git.asml.silence@gmail.comSigned-off-by: Jens Axboe <axboe@kernel.dk>

Going through submission in __io_sq_thread() and still having a full SQ
is rather unexpected, so remove a check for SQ fullness and just wake up
whoever wait on sqo_sq_wait. Also skip if it doesn't do submission in
the first place, likely may to happen for SQPOLL sharing and/or IOPOLL.
Signed-off-by: Pavel Begunkov <asml.silence@gmail.com>
Link: https://lore.kernel.org/r/e2e91751e87b1a39f8d63ef884aaff578123f61e.1621201931.git.asml.silence@gmail.comSigned-off-by: Jens Axboe <axboe@kernel.dk>

As sqpoll cancel via task_work is killed, remove everything related to
park_task_work as it's not used anymore.
Signed-off-by: Pavel Begunkov <asml.silence@gmail.com>
Link: https://lore.kernel.org/r/310d8b76a2fbbf3e139373500e04ad9af7ee3dbb.1621201931.git.asml.silence@gmail.comSigned-off-by: Jens Axboe <axboe@kernel.dk>

If SQPOLL task finds a ring requesting it to continue running, no need
to set wake flag to rest of the rings as it will be cleared in a moment
anyway, so hide it in a single sqd->ctx_list loop.
Signed-off-by: Pavel Begunkov <asml.silence@gmail.com>
Link: https://lore.kernel.org/r/1ee5a696d9fd08645994c58ee147d149a8957d94.1621201931.git.asml.silence@gmail.comSigned-off-by: Jens Axboe <axboe@kernel.dk>

As sqd->state changes rarely, don't check every event one by one but
look them all at once. Add a helper function. Also don't go into event
waiting sleeping with STOP flag set.
Signed-off-by: Pavel Begunkov <asml.silence@gmail.com>
Link: https://lore.kernel.org/r/645025f95c7eeec97f88ff497785f4f1d6f3966f.1621201931.git.asml.silence@gmail.comSigned-off-by: Jens Axboe <axboe@kernel.dk>

Merge tag 'perf-tools-fixes-for-v5.13-2021-06-13' of git://git.kernel.org/pub/scm/linux/kernel/git/acme/linux

Pull perf tools fixes from Arnaldo Carvalho de Melo:

 - Correct buffer copying when peeking events

 - Sync cpufeatures/disabled-features.h header with the kernel sources

* tag 'perf-tools-fixes-for-v5.13-2021-06-13' of git://git.kernel.org/pub/scm/linux/kernel/git/acme/linux:
  tools headers cpufeatures: Sync with the kernel sources
  perf session: Correct buffer copying when peeking events

Pull NFS client bugfixes from Trond Myklebust:
 "Highlights include:

  Stable fixes:

   - Fix use-after-free in nfs4_init_client()

  Bugfixes:

   - Fix deadlock between nfs4_evict_inode() and nfs4_opendata_get_inode()

   - Fix second deadlock in nfs4_evict_inode()

   - nfs4_proc_set_acl should not change the value of NFS_CAP_UIDGID_NOMAP

   - Fix setting of the NFS_CAP_SECURITY_LABEL capability"

* tag 'nfs-for-5.13-3' of git://git.linux-nfs.org/projects/trondmy/linux-nfs:
  NFSv4: Fix second deadlock in nfs4_evict_inode()
  NFSv4: Fix deadlock between nfs4_evict_inode() and nfs4_opendata_get_inode()
  NFS: FMODE_READ and friends are C macros, not enum types
  NFS: Fix a potential NULL dereference in nfs_get_client()
  NFS: Fix use-after-free in nfs4_init_client()
  NFS: Ensure the NFS_CAP_SECURITY_LABEL capability is set when appropriate
  NFSv4: nfs4_proc_set_acl needs to restore NFS_CAP_UIDGID_NOMAP on error.

Pull SCSI fixes from James Bottomley:
 "Four reasonably small fixes to the core for scsi host allocation
  failure paths.

  The root problem is that we're not freeing the memory allocated by
  dev_set_name(), which involves a rejig of may of the free on error
  paths to do put_device() instead of kfree which, in turn, has several
  other knock on ramifications and inspection turned up a few other
  lurking bugs"

* tag 'scsi-fixes' of git://git.kernel.org/pub/scm/linux/kernel/git/jejb/scsi:
  scsi: core: Only put parent device if host state differs from SHOST_CREATED
  scsi: core: Put .shost_dev in failure path if host state changes to RUNNING
  scsi: core: Fix failure handling of scsi_add_host_with_dma()
  scsi: core: Fix error handling of scsi_host_alloc()

Pull RISC-V fixes from Palmer Dabbelt:

 - A pair of XIP fixes: one to fix alternatives, and one to turn off the
   rest of the features that require code modification

 - A fix to a type that was causing some alternatives to break

 - A build fix for BUILTIN_DTB

* tag 'riscv-for-linus-5.13-rc6' of git://git.kernel.org/pub/scm/linux/kernel/git/riscv/linux:
  riscv: Fix BUILTIN_DTB for sifive and microchip soc
  riscv: alternative: fix typo in macro name
  riscv: code patching only works on !XIP_KERNEL
  riscv: xip: support runtime trap patching

0day robot reported a 9.2% regression for will-it-scale mmap1 test
case[1], caused by commit 57efa1fe ("mm/gup: prevent gup_fast from
racing with COW during fork").

Further debug shows the regression is due to that commit changes the
offset of hot fields 'mmap_lock' inside structure 'mm_struct', thus some
cache alignment changes.

From the perf data, the contention for 'mmap_lock' is very severe and
takes around 95% cpu cycles, and it is a rw_semaphore

        struct rw_semaphore {
                atomic_long_t count;	/* 8 bytes */
                atomic_long_t owner;	/* 8 bytes */
                struct optimistic_spin_queue osq; /* spinner MCS lock */
                ...

Before commit 57efa1fe adds the 'write_protect_seq', it happens to
have a very optimal cache alignment layout, as Linus explained:

 "and before the addition of the 'write_protect_seq' field, the
  mmap_sem was at offset 120 in 'struct mm_struct'.

  Which meant that count and owner were in two different cachelines,
  and then when you have contention and spend time in
  rwsem_down_write_slowpath(), this is probably *exactly* the kind
  of layout you want.

  Because first the rwsem_write_trylock() will do a cmpxchg on the
  first cacheline (for the optimistic fast-path), and then in the
  case of contention, rwsem_down_write_slowpath() will just access
  the second cacheline.

  Which is probably just optimal for a load that spends a lot of
  time contended - new waiters touch that first cacheline, and then
  they queue themselves up on the second cacheline."

After the commit, the rw_semaphore is at offset 128, which means the
'count' and 'owner' fields are now in the same cacheline, and causes
more cache bouncing.

Currently there are 3 "#ifdef CONFIG_XXX" before 'mmap_lock' which will
affect its offset:

  CONFIG_MMU
  CONFIG_MEMBARRIER
  CONFIG_HAVE_ARCH_COMPAT_MMAP_BASES

The layout above is on 64 bits system with 0day's default kernel config
(similar to RHEL-8.3's config), in which all these 3 options are 'y'.
And the layout can vary with different kernel configs.

Relayouting a structure is usually a double-edged sword, as sometimes it
can helps one case, but hurt other cases.  For this case, one solution
is, as the newly added 'write_protect_seq' is a 4 bytes long seqcount_t
(when CONFIG_DEBUG_LOCK_ALLOC=n), placing it into an existing 4 bytes
hole in 'mm_struct' will not change other fields' alignment, while
restoring the regression.

Link: https://lore.kernel.org/lkml/20210525031636.GB7744@xsang-OptiPlex-9020/ [1]
Reported-by: kernel test robot <oliver.sang@intel.com>
Signed-off-by: Feng Tang <feng.tang@intel.com>
Reviewed-by: John Hubbard <jhubbard@nvidia.com>
Reviewed-by: Jason Gunthorpe <jgg@nvidia.com>
Cc: Peter Xu <peterx@redhat.com>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>

Pull USB fixes from Greg KH:
 "Here are a number of tiny USB fixes for 5.13-rc6.

  There are more than I would normally like, but there's been a bunch of
  people banging on the gadget and dwc3 and typec code recently for I
  think an Android release, which has resulted in a number of small
  fixes. It's nice to see companies send fixes upstream for this type of
  work, a notable change from years ago.

  Anyway, fixes in here are:

   - usb-serial device id updates

   - usb-serial cp210x driver fixes for broken firmware versions

   - typec fixes for crazy charging devices and other reported problems

   - dwc3 fixes for reported problems found

   - gadget fixes for reported problems

   - tiny xhci fixes

   - other small fixes for reported issues.

   - revert of a problem fix found by linux-next testing

  All of these have passed 0-day and linux-next testing with no reported
  problems (the revert for the found linux-next build problem included)"

* tag 'usb-5.13-rc6' of git://git.kernel.org/pub/scm/linux/kernel/git/gregkh/usb: (44 commits)
  Revert "usb: gadget: fsl: Re-enable driver for ARM SoCs"
  usb: typec: mux: Fix copy-paste mistake in typec_mux_match
  usb: typec: ucsi: Clear PPM capability data in ucsi_init() error path
  usb: gadget: fsl: Re-enable driver for ARM SoCs
  usb: typec: wcove: Use LE to CPU conversion when accessing msg->header
  USB: serial: cp210x: fix CP2102N-A01 modem control
  USB: serial: cp210x: fix alternate function for CP2102N QFN20
  usb: misc: brcmstb-usb-pinmap: check return value after calling platform_get_resource()
  usb: dwc3: ep0: fix NULL pointer exception
  usb: gadget: eem: fix wrong eem header operation
  usb: typec: intel_pmc_mux: Put ACPI device using acpi_dev_put()
  usb: typec: intel_pmc_mux: Add missed error check for devm_ioremap_resource()
  usb: typec: intel_pmc_mux: Put fwnode in error case during ->probe()
  usb: typec: tcpm: Do not finish VDM AMS for retrying Responses
  usb: fix various gadget panics on 10gbps cabling
  usb: fix various gadgets null ptr deref on 10gbps cabling.
  usb: pci-quirks: disable D3cold on xhci suspend for s2idle on AMD Renoir
  usb: f_ncm: only first packet of aggregate needs to start timer
  USB: f_ncm: ncm_bitrate (speed) is unsigned
  MAINTAINERS: usb: add entry for isp1760
  ...

Pull serial driver fix from Greg KH:
 "A single 8250_exar serial driver fix for a reported problem with a
  change that happened in 5.13-rc1.

  It has been in linux-next with no reported problems"

* tag 'tty-5.13-rc6' of git://git.kernel.org/pub/scm/linux/kernel/git/gregkh/tty:
  serial: 8250_exar: Avoid NULL pointer dereference at ->exit()

Pull staging driver fixes from Greg KH:
 "Two tiny staging driver fixes:

   - ralink-gdma driver authorship information fixed up

   - rtl8723bs driver fix for reported regression

  Both have been in linux-next for a while with no reported problems"

* tag 'staging-5.13-rc6' of git://git.kernel.org/pub/scm/linux/kernel/git/gregkh/staging:
  staging: ralink-gdma: Remove incorrect author information
  staging: rtl8723bs: Fix uninitialized variables

Merge tag 'driver-core-5.13-rc6' of git://git.kernel.org/pub/scm/linux/kernel/git/gregkh/driver-core

Pull driver core fix from Greg KH:
 "A single debugfs fix for 5.13-rc6, fixing a bug in
  debugfs_read_file_str() that showed up in 5.13-rc1.

  It has been in linux-next for a full week with no
  reported problems"

* tag 'driver-core-5.13-rc6' of git://git.kernel.org/pub/scm/linux/kernel/git/gregkh/driver-core:
  debugfs: Fix debugfs_read_file_str()

Pull char/misc driver fixes from Greg KH:
 "Here are some small misc driver fixes for 5.13-rc6 that fix some
  reported problems:

   - Tiny phy driver fixes for reported issues

   - rtsx regression for when the device suspended

   - mhi driver fix for a use-after-free

  All of these have been in linux-next for a few days with no reported
  issues"

* tag 'char-misc-5.13-rc6' of git://git.kernel.org/pub/scm/linux/kernel/git/gregkh/char-misc:
  misc: rtsx: separate aspm mode into MODE_REG and MODE_CFG
  bus: mhi: pci-generic: Fix hibernation
  bus: mhi: pci_generic: Fix possible use-after-free in mhi_pci_remove()
  bus: mhi: pci_generic: T99W175: update channel name from AT to DUN
  phy: Sparx5 Eth SerDes: check return value after calling platform_get_resource()
  phy: ralink: phy-mt7621-pci: drop 'of_match_ptr' to fix -Wunused-const-variable
  phy: ti: Fix an error code in wiz_probe()
  phy: phy-mtk-tphy: Fix some resource leaks in mtk_phy_init()
  phy: cadence: Sierra: Fix error return code in cdns_sierra_phy_probe()
  phy: usb: Fix misuse of IS_ENABLED

Pull pin control fixes from Linus Walleij:

 - Fix some documentation warnings for Allwinner

 - Fix duplicated GPIO groups on Qualcomm SDX55

 - Fix a double enablement bug in the Ralink driver

 - Fix the Qualcomm SC8180x Kconfig so the driver can be selected.

* tag 'pinctrl-v5.13-2' of git://git.kernel.org/pub/scm/linux/kernel/git/linusw/linux-pinctrl:
  pinctrl: qcom: Make it possible to select SC8180x TLMM
  pinctrl: ralink: rt2880: avoid to error in calls is pin is already enabled
  pinctrl: qcom: Fix duplication in gpio_groups
  pinctrl: aspeed: Fix minor documentation error