- 11 Oct, 2015 1 commit
-
-
Borislav Petkov authored
The code under the label can normally be inline, without the jumping back and forth but the latter is an optimization. Document that. Signed-off-by:
Borislav Petkov <bp@suse.de> Acked-by:
Andy Lutomirski <luto@kernel.org> Cc: Andy Lutomirski <luto@amacapital.net> Cc: Borislav Petkov <bp@alien8.de> Cc: Brian Gerst <brgerst@gmail.com> Cc: Denys Vlasenko <dvlasenk@redhat.com> Cc: H. Peter Anvin <hpa@zytor.com> Cc: Linus Torvalds <torvalds@linux-foundation.org> Cc: Peter Zijlstra <peterz@infradead.org> Cc: Thomas Gleixner <tglx@linutronix.de> Link: http://lkml.kernel.org/r/20151009170859.GA24266@pd.tnicSigned-off-by:
Ingo Molnar <mingo@kernel.org>
-
- 09 Oct, 2015 28 commits
-
-
Andy Lutomirski authored
GCC is unable to properly optimize functions that have a very short likely case and a longer and register-heavier cold part -- it fails to sink all of the register saving and stack frame setup code into the unlikely part. Help it out with syscall_return_slowpath() by splitting it into two parts and inline the hot part. Saves 6 cycles for compat syscalls. Signed-off-by:
-
Andy Lutomirski authored
GCC is unable to properly optimize functions that have a very short likely case and a longer and register-heavier cold part -- it fails to sink all of the register saving and stack frame setup code into the unlikely part. Help it out with prepare_exit_to_usermode() by splitting it into two parts and inline the hot part. Saves 6-8 cycles for compat syscalls. Signed-off-by:
-
Andy Lutomirski authored
It generates simpler and faster code than current_thread_info(). Signed-off-by:
-
Andy Lutomirski authored
This shaves a few cycles off the slow paths. Signed-off-by:
-
Andy Lutomirski authored
We're following a 32-bit pointer, and the uaccess code isn't smart enough to figure out that the access_ok() check isn't needed. This saves about three cycles on a cache-hot fast syscall. Signed-off-by:
-
Andy Lutomirski authored
On systems that support fast syscalls, we only really care about the performance of the fast syscall path. Forcibly inline it and add a likely annotation. This saves 4-6 cycles. Signed-off-by:
-
Andy Lutomirski authored
These checks are quite slow. Disable them in non-lockdep kernels to reduce the performance hit. Signed-off-by:
-
Andy Lutomirski authored
This is slightly messy, but it eliminates an unnecessary cli;sti pair. Signed-off-by:
-
Andy Lutomirski authored
It's no longer needed. We could reinstate something like it as an optimization, which would remove two cachelines from the fast syscall entry working set. I benchmarked it, and it makes no difference whatsoever to the performance of cache-hot compat syscalls on Sandy Bridge. Signed-off-by:
-
Andy Lutomirski authored
Signed-off-by:
-
Andy Lutomirski authored
Signed-off-by:
-
Andy Lutomirski authored
syscall_exit is going away, and return tracing is just a function call now, so open-code the two non-syscall 32-bit users. While we're at it, update the big register layout comment. Signed-off-by:
-
Andy Lutomirski authored
If CS, SS and IP are as expected and FLAGS is compatible with SYSRETL, then return from fast compat syscalls (both SYSCALL and SYSENTER) using SYSRETL. Unlike native 64-bit opportunistic SYSRET, this is not invisible to user code: RCX and R8-R15 end up in a different state than shown saved in pt_regs. To compensate, we only do this when returning to the vDSO fast syscall return path. This won't interfere with syscall restart, as we won't use SYSRETL when returning to the INT80 restart instruction. Signed-off-by:
-
Andy Lutomirski authored
What, you didn't realize that SYSENTER and SYSCALL were actually the same thing? :) Unlike the old code, this actually passes the ptrace_syscall_32 test on AMD systems. Signed-off-by:
-
Andy Lutomirski authored
This handles both SYSENTER and SYSCALL. The asm glue will take care of the differences. Signed-off-by:
-
Andy Lutomirski authored
Signed-off-by:
-
Andy Lutomirski authored
System calls are really quite simple. Add a helper to call a 32-bit system call. Signed-off-by:
-
Andy Lutomirski authored
Syscalls are asmlinkage functions (on 32-bit kernels), take six args of type unsigned long, and return long. Note that uml could probably be slightly cleaned up on top of this patch. Signed-off-by:
-
Andy Lutomirski authored
The header was missing some compat declarations. Also make sys_call_ptr_t have a consistent type. Signed-off-by:
-
Andy Lutomirski authored
This is conceptually simpler. More importantly, it eliminates the PTREGSCALL and execve stubs, which were not compatible with the C ABI. This means that C code can call through the compat syscall table. The execve stubs are a bit subtle. They did two things: they cleared some registers and they forced slow-path return. Neither is necessary any more: elf_common_init clears the extra registers and start_thread calls force_iret(). Signed-off-by:
-
Andy Lutomirski authored
We now have only one code path that calls through the compat syscall table. This will make it much more pleasant to change the pt_regs vs register calling convention, which we need to do to move the call into C. Signed-off-by:
-
Andy Lutomirski authored
These audit optimizations are messy and hard to maintain. We'll get a similar effect from opportunistic sysret when fast compat system calls are re-implemented. Signed-off-by:
-
Andy Lutomirski authored
We've disabled the vDSO helpers to call them, so turn off the entries entirely (temporarily) in preparation for cleaning them up. Signed-off-by:
-
Andy Lutomirski authored
The goal is to integrate the SYSENTER and SYSCALL32 entry paths with the INT80 path. SYSENTER clobbers ESP and EIP. SYSCALL32 clobbers ECX (and, invisibly, R11). SYSRETL (long mode to compat mode) clobbers ECX and, invisibly, R11. SYSEXIT (which we only need for native 32-bit) clobbers ECX and EDX. This means that we'll need to provide ESP to the kernel in a register (I chose ECX, since it's only needed for SYSENTER) and we need to provide the args that normally live in ECX and EDX in memory. The epilogue needs to restore ECX and EDX, since user code relies on regs being preserved. We don't need to do anything special about EIP, since the kernel already knows where we are. The kernel will eventually need to know where int $0x80 lands, so add a vdso_image entry for it. The only user-visible effect of this code is that ptrace-induced changes to ECX and EDX during fast syscalls will be lost. This is already the case for the SYSENTER path. Signed-off-by:
-
Andy Lutomirski authored
Before we start calling execve in contexts that honor the full pt_regs, we need to teach it to initialize all registers. Signed-off-by:
-
Andy Lutomirski authored
Maintaining the current CFI annotations written in R'lyehian is difficult for most of us. Translate them to something a little closer to English. This will remove the CFI data for kernels built with extremely old versions of binutils. I think this is a fair tradeoff for the ability for mortals to edit the asm. Signed-off-by:
-
Andy Lutomirski authored
For the vDSO, user code wants runtime unwind info. Make sure that, if we use .cfi directives, we generate it. Signed-off-by:
-
Andy Lutomirski authored
Commit: 131484c8 ("x86/debug: Remove perpetually broken, unmaintainable dwarf annotations") removed all the manual DWARF annotations outside the vDSO. It also removed the macros we used for the manual annotations. Re-add these macros so that we can clean up the vDSO annotations. Signed-off-by:
-
- 07 Oct, 2015 11 commits
-
-
Andy Lutomirski authored
32-bit userspace will now always see the same vDSO, which is exactly what used to be the int80 vDSO. Subsequent patches will clean it up and make it support SYSENTER and SYSCALL using alternatives. Signed-off-by:
-
Andy Lutomirski authored
We eventually want to make it all the way into C code before enabling interrupts. We need to rework our flags handling slightly to delay enabling interrupts. Signed-off-by:
-
Andy Lutomirski authored
Rather than worrying about exactly where LOCKDEP_SYS_EXIT should go in the asm code, add it to prepare_exit_from_usermode() and remove all of the asm calls that are followed by prepare_exit_to_usermode(). LOCKDEP_SYS_EXIT now appears only in the syscall fast paths. Signed-off-by:
-
Andy Lutomirski authored
Clearing NT is part of the prologue, whereas loading up arg6 makes more sense to think about as part of syscall processing. Reorder them. Signed-off-by:
-
Andy Lutomirski authored
This tests assumptions about how fast syscall works wrt pt_regs and, in particular, what happens if IP is decremented by 2 during a syscall. Signed-off-by:
-
Andy Lutomirski authored
While the kernel itself doesn't use DWARF unwinding, user code expects to be able to unwind the vDSO. The vsyscall (AT_SYSINFO) entry is manually CFI-annotated, and this tests that it unwinds correctly. I tested the test by incorrectly annotating __kernel_vsyscall, and the test indeed fails if I do that. Signed-off-by:
-
Andy Lutomirski authored
This should improve code quality a bit. It also shrinks the kernel text: Before: text data bss dec filename 21828379 5194760 1277952 28301091 vmlinux After: text data bss dec filename 21827997 5194760 1277952 28300709 vmlinux ... by 382 bytes. Signed-off-by: -
Andy Lutomirski authored
GCC doesn't realize that get_user(), put_user(), and their __ variants are unlikely to fail. Tell it. I noticed this while playing with the C entry code. Before: text data bss dec filename 21828763 5194760 1277952 28301475 vmlinux.baseline After: text data bss dec filename 21828379 5194760 1277952 28301091 vmlinux.new The generated code shrunk by 384 bytes. Signed-off-by: -
Ingo Molnar authored
Signed-off-by: -
git://git.linux-nfs.org/projects/trondmy/linux-nfsLinus Torvalds authored
Pull NFS client bugfixes from Trond Myklebust: "Highlights include: Bugfixes: - Fix a use-after-free bug in the RPC/RDMA client - Fix a write performance regression - Fix up page writeback accounting - Don't try to reclaim unused state owners - Fix a NFSv4 nograce recovery hang - reset states to use open_stateid when returning delegation voluntarily - Fix a tracepoint NULL-pointer dereference" * tag 'nfs-for-4.3-3' of git://git.linux-nfs.org/projects/trondmy/linux-nfs: NFS: Fix a tracepoint NULL-pointer dereference nfs4: reset states to use open_stateid when returning delegation voluntarily NFSv4: Fix a nograce recovery hang NFSv4.1: nfs4_opendata_check_deleg needs to handle NFS4_OPEN_CLAIM_DELEG_CUR_FH NFSv4: Don't try to reclaim unused state owners NFS: Fix a write performance regression NFS: Fix up page writeback accounting xprtrdma: disconnect and flush cqs before freeing buffers -
Linus Torvalds authored
This reverts commit 998ef75d. The commit itself does not appear to be buggy per se, but it is exposing a bug in ext4 (and Ted thinks ext3 too, but we solved that by getting rid of it). It's too late in the release cycle to really worry about this, even if Dave Hansen has a patch that may actually fix the underlying ext4 problem. We can (and should) revisit this for the next release. The problem is that moving the prefaulting later now exposes a special case with partially successful writes that isn't handled correctly. And the prefaulting likely isn't normally even that much of a performance issue - it looks like at least one reason Dave saw this in his performance tests is that he also ran them on Skylake that now supports the new SMAP code, which makes the normally very cheap user space prefaulting noticeably more expensive. Bisected-and-acked-by:
Ted Ts'o <tytso@mit.edu> Analyzed-and-acked-by:
Dave Hansen <dave.hansen@linux.intel.com> Cc: Andrew Morton <akpm@linux-foundation.org> Signed-off-by:
Linus Torvalds <torvalds@linux-foundation.org>
-
