I didn't really want to do this, but as part of all the other changes to
the user copy loops, I've been looking at this horror.

I tried to clean it up multiple times, but every time I just found more
problems, and the way it's written, it's just too hard to fix them.

For example, the code is written to do quad-word alignment, and will use
regular byte accesses to get to that point.  That's fairly simple, but
it means that any initial 8-byte alignment will be done with cached
copies.

However, the code then is very careful to do any 4-byte _tail_ accesses
using an uncached 4-byte write, and that was claimed to be relevant in
commit a82eee74 ("x86/uaccess/64: Handle the caching of 4-byte
nocache copies properly in __copy_user_nocache()").

So if you do a 4-byte copy using that function, it carefully uses a
4-byte 'movnti' for the destination.  But if you were to do a 12-byte
copy that is 4-byte aligned, it would _not_ do a 4-byte 'movnti'
followed by a 8-byte 'movnti' to keep it all uncached.

Instead, it would align the destination to 8 bytes using a
byte-at-a-time loop, and then do a 8-byte 'movnti' for the final 8
bytes.

The main caller that cares is __copy_user_flushcache(), which knows
about this insanity, and has odd cases for it all.  But I just can't
deal with looking at this kind of "it does one case right, and another
related case entirely wrong".

And the code really wasn't fixable without hard drugs, which I try to
avoid.

So instead, rewrite it in a form that hopefully not only gets this
right, but is a bit more maintainable.  Knock wood.
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>

Every caller passes in zero, meaning they don't want any partial copy to
zero the remainder of the destination buffer.

Which is just as well, because the implementation of that function
didn't actually even look at that argument, and wasn't even aware it
existed, although some misleading comments did mention it still.

The 'zerorest' thing is a historical artifact of how "copy_from_user()"
worked, in that it would zero the rest of the kernel buffer that it
copied into.

That zeroing still exists, but it's long since been moved to generic
code, and the raw architecture-specific code doesn't do it.  See
_copy_from_user() in lib/usercopy.c for this all.

However, while __copy_user_nocache() shares some history and superficial
other similarities with copy_from_user(), it is in many ways also very
different.

In particular, while the code makes it *look* similar to the generic
user copy functions that can copy both to and from user space, and take
faults on both reads and writes as a result, __copy_user_nocache() does
no such thing at all.

__copy_user_nocache() always copies to kernel space, and will never take
a page fault on the destination.  What *can* happen, though, is that the
non-temporal stores take a machine check because one of the use cases is
for writing to stable memory, and any memory errors would then take
synchronous faults.

So __copy_user_nocache() does look a lot like copy_from_user(), but has
faulting behavior that is more akin to our old copy_in_user() (which no
longer exists, but copied from user space to user space and could fault
on both source and destination).

And it very much does not have the "zero the end of the destination
buffer", since a problem with the destination buffer is very possibly
the very source of the partial copy.

So this whole thing was just a confusing historical artifact from having
shared some code with a completely different function with completely
different use cases.
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>

So Intel introduced the FSRS ("Fast Short REP STOS") CPU capability bit,
because they seem to have done the (much simpler) REP STOS optimizations
separately and later than the REP MOVS one.

In contrast, when AMD introduced support for FSRM ("Fast Short REP
MOVS"), in the Zen 3 core, it appears to have improved the REP STOS case
at the same time, and since the FSRS bit was added by Intel later, it
doesn't show up on those AMD Zen 3 cores.

And now that we made use of FSRS for the "rep stos" conditional, that
made those AMD machines unnecessarily slower.  The Intel situation where
"rep movs" is fast, but "rep stos" isn't, is just odd.  The 'stos' case
is a lot simpler with no aliasing, no mutual alignment issues, no
complicated cases.

So this just sets FSRS automatically when FSRM is available on AMD
machines, to get back all the nice REP STOS goodness in Zen 3.
Reported-and-tested-by: Jens Axboe <axboe@kernel.dk>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>

The old 'copy_user_generic_unrolled' function was oddly implemented for
largely historical reasons: it had been largely based on the uncached
copy case, which has some other concerns.

For example, the __copy_user_nocache() function uses 'movnti' for the
destination stores, and those want the destination to be aligned.  In
contrast, the regular copy function doesn't really care, and trying to
align things only complicates matters.

Also, like the clear_user function, the copy function had some odd
handling of the repeat counts, complicating the exception handling for
no really good reason.  So as with clear_user, just write it to keep all
the byte counts in the %rcx register, exactly like the 'rep movs'
functionality that this replaces.

Unlike a real 'rep movs', we do allow for this to trash a few temporary
registers to not have to unnecessarily save/restore registers on the
stack.

And like the clearing case, rename this to what it now clearly is:
'rep_movs_alternative', and make it one coherent function, so that it
shows up as such in profiles (instead of the odd split between
"copy_user_generic_unrolled" and "copy_user_short_string", the latter of
which was not about strings at all, and which was shared with the
uncached case).
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>

The old version was oddly written to have the repeat count in multiple
registers.  So instead of taking advantage of %rax being zero, it had
some sub-counts in it.  All just for a "single word clearing" loop,
which isn't even efficient to begin with.

So get rid of those games, and just keep all the state in the same
registers we got it in (and that we should return things in).  That not
only makes this act much more like 'rep stos' (which this function is
replacing), but makes it much easier to actually do the obvious loop
unrolling.

Also rename the function from the now nonsensical 'clear_user_original'
to what it now clearly is: 'rep_stos_alternative'.

End result: if we don't have a fast 'rep stosb', at least we can have a
fast fallback for it.
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>

This does the same thing for the user copies as commit 0db7058e
("x86/clear_user: Make it faster") did for clear_user().  In other
words, it inlines the "rep movs" case when X86_FEATURE_FSRM is set,
avoiding the function call entirely.

In order to do that, it makes the calling convention for the out-of-line
case ("copy_user_generic_unrolled") match the 'rep movs' calling
convention, although it does also end up clobbering a number of
additional registers.

Also, to simplify code sharing in the low-level assembly with the
__copy_user_nocache() function (that uses the normal C calling
convention), we end up with a kind of mixed return value for the
low-level asm code: it will return the result in both %rcx (to work as
an alternative for the 'rep movs' case), _and_ in %rax (for the nocache
case).

We could avoid this by wrapping __copy_user_nocache() callers in an
inline asm, but since the cost is just an extra register copy, it's
probably not worth it.
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>

This is preparatory work for inlining the 'rep movs' case, but also a
cleanup.  The __copy_user_nocache() function was mis-used by the rdma
code to do uncached kernel copies that don't actually want user copies
at all, and as a result doesn't want the stac/clac either.
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>

The modern target to use is FSRS (Fast Short REP STOS), and the other
cases should only be used for bigger areas (ie mainly things like page
clearing).

Note! This changes the conditional for the inlining from FSRM ("fast
short rep movs") to FSRS ("fast short rep stos").

We'll have a separate fixup for AMD microarchitectures that have a good
'rep stosb' yet do not set the new Intel-specific FSRS bit (because FSRM
was there first).
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>

The modern target to use is FSRM (Fast Short REP MOVS), and the other
cases should only be used for bigger areas (ie mainly things like page
clearing).
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>

The modern target to use is FSRS (Fast Short REP STOS), and the other
cases should only be used for bigger areas (ie mainly things like page
clearing).
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>

The modern target to use is FSRM (Fast Short REP MOVS), and the other
cases should only be used for bigger areas (ie mainly things like page
copying and clearing).
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>

Pull scheduler fix from Borislav Petkov:

 - Do not pull tasks to the local scheduling group if its average load
   is higher than the average system load

* tag 'sched_urgent_for_v6.3_rc7' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip:
  sched/fair: Fix imbalance overflow

Pull x86 fix from Borislav Petkov:

 - Drop __init annotation from two rtc functions which get called after
   boot is done, in order to prevent a crash

* tag 'x86_urgent_for_v6.3_rc7' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip:
  x86/rtc: Remove __init for runtime functions

Pull powerpc fix from Michael Ellerman:

 - A fix for NUMA distance handling in the pseries SCM (pmem) driver.

   Thanks to Aneesh Kumar K.V.

* tag 'powerpc-6.3-5' of git://git.kernel.org/pub/scm/linux/kernel/git/powerpc/linux:
  powerpc/papr_scm: Update the NUMA distance table for the target node

Merge tag 'kbuild-fixes-v6.3-3' of git://git.kernel.org/pub/scm/linux/kernel/git/masahiroy/linux-kbuild

Pull Kbuild fixes from Masahiro Yamada:

 - Drop debug info from purgatory objects again

 - Document that kernel.org provides prebuilt LLVM toolchains

 - Give up handling untracked files for source package builds

 - Avoid creating corrupted cpio when KBUILD_BUILD_TIMESTAMP is given
   with a pre-epoch data.

 - Change panic_show_mem() to a macro to handle variable-length argument

 - Compress tarballs on-the-fly again

* tag 'kbuild-fixes-v6.3-3' of git://git.kernel.org/pub/scm/linux/kernel/git/masahiroy/linux-kbuild:
  kbuild: do not create intermediate *.tar for tar packages
  kbuild: do not create intermediate *.tar for source tarballs
  kbuild: merge cmd_archive_linux and cmd_archive_perf
  init/initramfs: Fix argument forwarding to panic() in panic_show_mem()
  initramfs: Check negative timestamp to prevent broken cpio archive
  kbuild: give up untracked files for source package builds
  Documentation/llvm: Add a note about prebuilt kernel.org toolchains
  purgatory: fix disabling debug info

Pull ksmbd server fix from Steve French:
 "smb311 server preauth integrity negotiate context parsing fix (check
  for out of bounds access)"

* tag '6.3-rc6-ksmbd-server-fix' of git://git.samba.org/ksmbd:
  ksmbd: avoid out of bounds access in decode_preauth_ctxt()

Commit 05e96e96 ("kbuild: use git-archive for source package
creation") split the compression as a separate step to factor out
the common build rules.

With the previous commit, we got back to the situation where source
tarballs are compressed on-the-fly.
There is no reason to keep the separate compression rules.

Generate the comressed tar packages directly.
Signed-off-by: Masahiro Yamada <masahiroy@kernel.org>
Reviewed-by: Nathan Chancellor <nathan@kernel.org>

Since commit 05e96e96 ("kbuild: use git-archive for source package
creation"), a source tarball is created in two steps; create *.tar file
then compress it. I split the compression as a separate rule because I
just thought 'git archive' supported only gzip.

For other compression algorithms, I could pipe the two commands:

  $ git archive HEAD | xz > linux.tar.xz

I read git-archive(1) carefully, and I realized GIT had provided a
more elegant way:

  $ git -c tar.tar.xz.command=xz archive -o linux.tar.xz HEAD

This commit uses 'tar.tar.*.command' configuration to specify the
compression backend so we can compress a source tarball on-the-fly.

GIT commit 767cf4579f0e ("archive: implement configurable tar filters")
is more than a decade old, so it should be available on almost all build
environments.
Signed-off-by: Masahiro Yamada <masahiroy@kernel.org>
Reviewed-by: Nathan Chancellor <nathan@kernel.org>

The two commands, cmd_archive_linux and cmd_archive_perf, are similar.
Merge them to make it easier to add more changes to the git-archive
command.
Signed-off-by: Masahiro Yamada <masahiroy@kernel.org>
Reviewed-by: Nathan Chancellor <nathan@kernel.org>

Forwarding variadic argument lists can't be done by passing a va_list
to a function with signature foo(...) (as panic() has). It ends up
interpreting the va_list itself as a single argument instead of
iterating it. printf() happily accepts it of course, leading to corrupt
output.

Convert panic_show_mem() to a macro to allow forwarding the arguments.
The function is trivial enough that it's easier than trying to introduce
a vpanic() variant.
Signed-off-by: Benjamin Gray <bgray@linux.ibm.com>
Reviewed-by: Andrew Donnellan <ajd@linux.ibm.com>
Signed-off-by: Masahiro Yamada <masahiroy@kernel.org>

Similar to commit 4c9d410f ("initramfs: Check timestamp to prevent
broken cpio archive"), except asserts that the timestamp is
non-negative. This can happen when the KBUILD_BUILD_TIMESTAMP is a value
before UNIX epoch, which may be set when making reproducible builds that
don't want to look like they use a valid date.

While support for dates before 1970 might not be supported, this is more
about preventing undetected CPIO corruption. The printf's use a minimum
length format specifier, and will happily make the field longer than 8
characters if they need to.
Signed-off-by: Benjamin Gray <bgray@linux.ibm.com>
Reviewed-by: Andrew Donnellan <ajd@linux.ibm.com>
Tested-by: Andrew Donnellan <ajd@linux.ibm.com>
Signed-off-by: Masahiro Yamada <masahiroy@kernel.org>

Pull cifs fix from Steve French:
 "Small client fix for better checking for smb311 negotiate context
  overflows, also marked for stable"

* tag '6.3-rc6-smb311-client-negcontext-fix' of git://git.samba.org/sfrench/cifs-2.6:
  cifs: fix negotiate context parsing

Pull UBI fixes from Richard Weinberger:

 - Fix failure to attach when vid_hdr offset equals the (sub)page size

 - Fix for a deadlock in UBI's worker thread

* tag 'ubifs-for-linus-6.3-rc7' of git://git.kernel.org/pub/scm/linux/kernel/git/rw/ubifs:
  ubi: Fix failure attaching when vid_hdr offset equals to (sub)page size
  ubi: Fix deadlock caused by recursively holding work_sem

smb311_decode_neg_context() doesn't properly check against SMB packet
boundaries prior to accessing individual negotiate context entries. This
is due to the length check omitting the eight byte smb2_neg_context
header, as well as incorrect decrementing of len_of_ctxts.

Fixes: 5100d8a3 ("SMB311: Improve checking of negotiate security contexts")
Reported-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Paulo Alcantara (SUSE) <pc@manguebit.com>
Signed-off-by: David Disseldorp <ddiss@suse.de>
Signed-off-by: Steve French <stfrench@microsoft.com>

Pull i2c fixes from Wolfram Sang:
 "Just two driver fixes"

* tag 'i2c-for-6.3-rc7' of git://git.kernel.org/pub/scm/linux/kernel/git/wsa/linux:
  i2c: ocores: generate stop condition after timeout in polling mode
  i2c: mchp-pci1xxxx: Update Timing registers

Pull SCSI fix from James Bottomley:
 "One small fix to SCSI Enclosure Services to fix a regression caused by
  another recent fix"

* tag 'scsi-fixes' of git://git.kernel.org/pub/scm/linux/kernel/git/jejb/scsi:
  scsi: ses: Handle enclosure with just a primary component gracefully

Pull block fix from Jens Axboe:
 "A single NVMe quirk entry addition"

* tag 'block-6.3-2023-04-14' of git://git.kernel.dk/linux:
  nvme-pci: add NVME_QUIRK_BOGUS_NID for T-FORCE Z330 SSD

Pull io_uring fix from Jens Axboe:
 "Just a small tweak to when task_work needs redirection, marked for
  stable as well"

* tag 'io_uring-6.3-2023-04-14' of git://git.kernel.dk/linux:
  io_uring: complete request via task work in case of DEFER_TASKRUN

Pull RISC-V fixes from Palmer Dabbelt:

 - A fix for a missing fence when generating the NOMMU sigreturn
   trampoline

 - A set of fixes for early DTB handling of reserved memory nodes

* tag 'riscv-for-linus-6.3-rc7' of git://git.kernel.org/pub/scm/linux/kernel/git/riscv/linux:
  riscv: No need to relocate the dtb as it lies in the fixmap region
  riscv: Do not set initial_boot_params to the linear address of the dtb
  riscv: Move early dtb mapping into the fixmap region
  riscv: add icache flush for nommu sigreturn trampoline

Pull ACPI fixes from Rafael Wysocki:
 "These add two ACPI-related quirks:

   - Add a quirk to force StorageD3Enable on AMD Picasso systems (Mario
     Limonciello)

   - Add an ACPI IRQ override quirk for ASUS ExpertBook B1502CBA (Paul
     Menzel)"

* tag 'acpi-6.3-rc7' of git://git.kernel.org/pub/scm/linux/kernel/git/rafael/linux-pm:
  ACPI: resource: Skip IRQ override on ASUS ExpertBook B1502CBA
  ACPI: x86: utils: Add Picasso to the list for forcing StorageD3Enable

Pull power management fix from Rafael Wysocki:
 "Make the amd-pstate cpufreq driver take all of the possible
  combinations of the 'old' and 'new' status values correctly while
  changing the operation mode via sysfs (Wyes Karny)"

* tag 'pm-6.3-rc7' of git://git.kernel.org/pub/scm/linux/kernel/git/rafael/linux-pm:
  amd-pstate: Fix amd_pstate mode switch

Pull thermal control fix from Rafael Wysocki:
 "Modify the Intel thermal throttling code to avoid updating unsupported
  status clearing mask bits which causes the kernel to complain about
  unchecked MSR access (Srinivas Pandruvada)"

* tag 'thermal-6.3-rc7' of git://git.kernel.org/pub/scm/linux/kernel/git/rafael/linux-pm:
  thermal: intel: Avoid updating unsupported THERM_STATUS_CLEAR mask bits

Pull sound fixes from Takashi Iwai:
 "A collection of small fixes.

  At this time, quite a few fixes for the old PCI drivers are found.
  Although they are not regression fixes, I took these as they are
  materials for stable kernels.

  In addition, a couple of regression fixes and another couple of
  HD-audio quirks are included"

* tag 'sound-6.3-rc7' of git://git.kernel.org/pub/scm/linux/kernel/git/tiwai/sound:
  ALSA: hda/hdmi: disable KAE for Intel DG2
  ALSA: hda/realtek: Add quirks for Lenovo Z13/Z16 Gen2
  ALSA: hda: patch_realtek: add quirk for Asus N7601ZM
  ALSA: firewire-tascam: add missing unwind goto in snd_tscm_stream_start_duplex()
  ALSA: emu10k1: don't create old pass-through playback device on Audigy
  ALSA: emu10k1: fix capture interrupt handler unlinking
  ALSA: hda/sigmatel: fix S/PDIF out on Intel D*45* motherboards
  ALSA: hda/sigmatel: add pin overrides for Intel DP45SG motherboard
  ALSA: i2c/cs8427: fix iec958 mixer control deactivation

Pull rdma fixes from Jason Gunthorpe:
 "We had a fairly slow cycle on the rc side this time, here are the
  accumulated fixes, mostly in drivers:

   - irdma should not generate extra completions during flushing

   - Fix several memory leaks

   - Do not get confused in irdma's iwarp mode if IPv6 is present

   - Correct a link speed calculation in mlx5

   - Increase the EQ/WQ limits on erdma as they are too small for big
     applications

   - Use the right math for erdma's inline mtt feature

   - Make erdma probing more robust to boot time ordering differences

   - Fix a KMSAN crash in CMA due to uninitialized qkey"

* tag 'for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/rdma/rdma:
  RDMA/core: Fix GID entry ref leak when create_ah fails
  RDMA/cma: Allow UD qp_type to join multicast only
  RDMA/erdma: Defer probing if netdevice can not be found
  RDMA/erdma: Inline mtt entries into WQE if supported
  RDMA/erdma: Update default EQ depth to 4096 and max_send_wr to 8192
  RDMA/erdma: Fix some typos
  IB/mlx5: Add support for 400G_8X lane speed
  RDMA/irdma: Add ipv4 check to irdma_find_listener()
  RDMA/irdma: Increase iWARP CM default rexmit count
  RDMA/irdma: Fix memory leak of PBLE objects
  RDMA/irdma: Do not generate SW completions for NOPs

Merge a quirk to force StorageD3Enable on AMD Picasso systems (Mario
Limonciello).

* acpi-x86:
  ACPI: x86: utils: Add Picasso to the list for forcing StorageD3Enable

So far io_req_complete_post() only covers DEFER_TASKRUN by completing
request via task work when the request is completed from IOWQ.

However, uring command could be completed from any context, and if io
uring is setup with DEFER_TASKRUN, the command is required to be
completed from current context, otherwise wait on IORING_ENTER_GETEVENTS
can't be wakeup, and may hang forever.

The issue can be observed on removing ublk device, but turns out it is
one generic issue for uring command & DEFER_TASKRUN, so solve it in
io_uring core code.

Fixes: e6aeb272 ("io_uring: complete all requests in task context")
Cc: stable@vger.kernel.org
Link: https://lore.kernel.org/linux-block/b3fc9991-4c53-9218-a8cc-5b4dd3952108@kernel.dk/Reported-by: Jens Axboe <axboe@kernel.dk>
Cc: Kanchan Joshi <joshi.k@samsung.com>
Signed-off-by: Ming Lei <ming.lei@redhat.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>

Pull NVMe fix from Christoph.

* 'nvme-6.3' of git://git.infradead.org/nvme:
  nvme-pci: add NVME_QUIRK_BOGUS_NID for T-FORCE Z330 SSD

Use of keep-alive (KAE) has resulted in loss of audio on some A750/770
cards as the transition from keep-alive to stream playback is not
working as expected. As there is limited benefit of the new KAE mode
on discrete cards, revert back to older silent-stream implementation
on these systems.

Cc: stable@vger.kernel.org
Fixes: 15175a4f ("ALSA: hda/hdmi: add keep-alive support for ADL-P and DG2")
Link: https://gitlab.freedesktop.org/drm/intel/-/issues/8307Signed-off-by: Kai Vehmanen <kai.vehmanen@linux.intel.com>
Link: https://lore.kernel.org/r/20230413191153.3692049-1-kai.vehmanen@linux.intel.comSigned-off-by: Takashi Iwai <tiwai@suse.de>

Added a quirk to fix the TeamGroup T-Force Cardea Zero Z330 SSDs reporting
duplicate NGUIDs.
Signed-off-by: Duy Truong <dory@dory.moe>
Cc: stable@vger.kernel.org
Signed-off-by: Christoph Hellwig <hch@lst.de>