- 18 Oct, 2015 1 commit
-
-
Andy Lutomirski authored
When I rewrote entry_INT80_32, I thought that int80 was an interrupt gate. It's a trap gate. *facepalm* Thanks to Brian Gerst for pointing out that it's better to change the entry code than to change the gate type. Suggested-by:
Brian Gerst <brgerst@gmail.com> Reported-and-tested-by:
Borislav Petkov <bp@suse.de> Signed-off-by:
Andy Lutomirski <luto@kernel.org> Cc: Andy Lutomirski <luto@amacapital.net> Cc: Borislav Petkov <bp@alien8.de> Cc: Denys Vlasenko <dvlasenk@redhat.com> Cc: H. Peter Anvin <hpa@zytor.com> Cc: Linus Torvalds <torvalds@linux-foundation.org> Cc: Peter Zijlstra <peterz@infradead.org> Cc: Thomas Gleixner <tglx@linutronix.de> Fixes: 150ac78d ("x86/entry/32: Switch INT80 to the new C syscall path") Link: http://lkml.kernel.org/r/dc09d9b574a5c1dcca996847875c73f8341ce0ad.1445035014.git.luto@kernel.orgSigned-off-by:
Ingo Molnar <mingo@kernel.org>
-
- 14 Oct, 2015 3 commits
-
-
Andy Lutomirski authored
I didn't realize that um didn't include x86's asm/syscall.h. Re-add a missing typedef. Reported-by:
Richard Weinberger <richard@nod.at> Signed-off-by:
-
Andy Lutomirski authored
They are currently unused, and I don't think that anyone was ever particularly happy with them. They had the unfortunate property that they made it easy to CFI-annotate things without thinking about them -- when pushing, do you want to just update the CFA offset, or do you also want to update the saved location of the register being pushed? Suggested-by:
-
Andy Lutomirski authored
Checkpatch is really quite bad for user code like this, but it caught two legit style issues. Reported-by:
-
- 11 Oct, 2015 1 commit
-
-
Borislav Petkov authored
The code under the label can normally be inline, without the jumping back and forth but the latter is an optimization. Document that. Signed-off-by:
-
- 09 Oct, 2015 28 commits
-
-
Andy Lutomirski authored
GCC is unable to properly optimize functions that have a very short likely case and a longer and register-heavier cold part -- it fails to sink all of the register saving and stack frame setup code into the unlikely part. Help it out with syscall_return_slowpath() by splitting it into two parts and inline the hot part. Saves 6 cycles for compat syscalls. Signed-off-by:
-
Andy Lutomirski authored
GCC is unable to properly optimize functions that have a very short likely case and a longer and register-heavier cold part -- it fails to sink all of the register saving and stack frame setup code into the unlikely part. Help it out with prepare_exit_to_usermode() by splitting it into two parts and inline the hot part. Saves 6-8 cycles for compat syscalls. Signed-off-by:
-
Andy Lutomirski authored
It generates simpler and faster code than current_thread_info(). Signed-off-by:
-
Andy Lutomirski authored
This shaves a few cycles off the slow paths. Signed-off-by:
-
Andy Lutomirski authored
We're following a 32-bit pointer, and the uaccess code isn't smart enough to figure out that the access_ok() check isn't needed. This saves about three cycles on a cache-hot fast syscall. Signed-off-by:
-
Andy Lutomirski authored
On systems that support fast syscalls, we only really care about the performance of the fast syscall path. Forcibly inline it and add a likely annotation. This saves 4-6 cycles. Signed-off-by:
-
Andy Lutomirski authored
These checks are quite slow. Disable them in non-lockdep kernels to reduce the performance hit. Signed-off-by:
-
Andy Lutomirski authored
This is slightly messy, but it eliminates an unnecessary cli;sti pair. Signed-off-by:
-
Andy Lutomirski authored
It's no longer needed. We could reinstate something like it as an optimization, which would remove two cachelines from the fast syscall entry working set. I benchmarked it, and it makes no difference whatsoever to the performance of cache-hot compat syscalls on Sandy Bridge. Signed-off-by:
-
Andy Lutomirski authored
Signed-off-by:
-
Andy Lutomirski authored
Signed-off-by:
-
Andy Lutomirski authored
syscall_exit is going away, and return tracing is just a function call now, so open-code the two non-syscall 32-bit users. While we're at it, update the big register layout comment. Signed-off-by:
-
Andy Lutomirski authored
If CS, SS and IP are as expected and FLAGS is compatible with SYSRETL, then return from fast compat syscalls (both SYSCALL and SYSENTER) using SYSRETL. Unlike native 64-bit opportunistic SYSRET, this is not invisible to user code: RCX and R8-R15 end up in a different state than shown saved in pt_regs. To compensate, we only do this when returning to the vDSO fast syscall return path. This won't interfere with syscall restart, as we won't use SYSRETL when returning to the INT80 restart instruction. Signed-off-by:
-
Andy Lutomirski authored
What, you didn't realize that SYSENTER and SYSCALL were actually the same thing? :) Unlike the old code, this actually passes the ptrace_syscall_32 test on AMD systems. Signed-off-by:
-
Andy Lutomirski authored
This handles both SYSENTER and SYSCALL. The asm glue will take care of the differences. Signed-off-by:
-
Andy Lutomirski authored
Signed-off-by:
-
Andy Lutomirski authored
System calls are really quite simple. Add a helper to call a 32-bit system call. Signed-off-by:
-
Andy Lutomirski authored
Syscalls are asmlinkage functions (on 32-bit kernels), take six args of type unsigned long, and return long. Note that uml could probably be slightly cleaned up on top of this patch. Signed-off-by:
-
Andy Lutomirski authored
The header was missing some compat declarations. Also make sys_call_ptr_t have a consistent type. Signed-off-by:
-
Andy Lutomirski authored
This is conceptually simpler. More importantly, it eliminates the PTREGSCALL and execve stubs, which were not compatible with the C ABI. This means that C code can call through the compat syscall table. The execve stubs are a bit subtle. They did two things: they cleared some registers and they forced slow-path return. Neither is necessary any more: elf_common_init clears the extra registers and start_thread calls force_iret(). Signed-off-by:
-
Andy Lutomirski authored
We now have only one code path that calls through the compat syscall table. This will make it much more pleasant to change the pt_regs vs register calling convention, which we need to do to move the call into C. Signed-off-by:
-
Andy Lutomirski authored
These audit optimizations are messy and hard to maintain. We'll get a similar effect from opportunistic sysret when fast compat system calls are re-implemented. Signed-off-by:
-
Andy Lutomirski authored
We've disabled the vDSO helpers to call them, so turn off the entries entirely (temporarily) in preparation for cleaning them up. Signed-off-by:
-
Andy Lutomirski authored
The goal is to integrate the SYSENTER and SYSCALL32 entry paths with the INT80 path. SYSENTER clobbers ESP and EIP. SYSCALL32 clobbers ECX (and, invisibly, R11). SYSRETL (long mode to compat mode) clobbers ECX and, invisibly, R11. SYSEXIT (which we only need for native 32-bit) clobbers ECX and EDX. This means that we'll need to provide ESP to the kernel in a register (I chose ECX, since it's only needed for SYSENTER) and we need to provide the args that normally live in ECX and EDX in memory. The epilogue needs to restore ECX and EDX, since user code relies on regs being preserved. We don't need to do anything special about EIP, since the kernel already knows where we are. The kernel will eventually need to know where int $0x80 lands, so add a vdso_image entry for it. The only user-visible effect of this code is that ptrace-induced changes to ECX and EDX during fast syscalls will be lost. This is already the case for the SYSENTER path. Signed-off-by:
-
Andy Lutomirski authored
Before we start calling execve in contexts that honor the full pt_regs, we need to teach it to initialize all registers. Signed-off-by:
-
Andy Lutomirski authored
Maintaining the current CFI annotations written in R'lyehian is difficult for most of us. Translate them to something a little closer to English. This will remove the CFI data for kernels built with extremely old versions of binutils. I think this is a fair tradeoff for the ability for mortals to edit the asm. Signed-off-by:
-
Andy Lutomirski authored
For the vDSO, user code wants runtime unwind info. Make sure that, if we use .cfi directives, we generate it. Signed-off-by:
-
Andy Lutomirski authored
Commit: 131484c8 ("x86/debug: Remove perpetually broken, unmaintainable dwarf annotations") removed all the manual DWARF annotations outside the vDSO. It also removed the macros we used for the manual annotations. Re-add these macros so that we can clean up the vDSO annotations. Signed-off-by:
-
- 07 Oct, 2015 7 commits
-
-
Andy Lutomirski authored
32-bit userspace will now always see the same vDSO, which is exactly what used to be the int80 vDSO. Subsequent patches will clean it up and make it support SYSENTER and SYSCALL using alternatives. Signed-off-by:
-
Andy Lutomirski authored
We eventually want to make it all the way into C code before enabling interrupts. We need to rework our flags handling slightly to delay enabling interrupts. Signed-off-by:
-
Andy Lutomirski authored
Rather than worrying about exactly where LOCKDEP_SYS_EXIT should go in the asm code, add it to prepare_exit_from_usermode() and remove all of the asm calls that are followed by prepare_exit_to_usermode(). LOCKDEP_SYS_EXIT now appears only in the syscall fast paths. Signed-off-by:
-
Andy Lutomirski authored
Clearing NT is part of the prologue, whereas loading up arg6 makes more sense to think about as part of syscall processing. Reorder them. Signed-off-by:
-
Andy Lutomirski authored
This tests assumptions about how fast syscall works wrt pt_regs and, in particular, what happens if IP is decremented by 2 during a syscall. Signed-off-by:
-
Andy Lutomirski authored
While the kernel itself doesn't use DWARF unwinding, user code expects to be able to unwind the vDSO. The vsyscall (AT_SYSINFO) entry is manually CFI-annotated, and this tests that it unwinds correctly. I tested the test by incorrectly annotating __kernel_vsyscall, and the test indeed fails if I do that. Signed-off-by:
-
Andy Lutomirski authored
This should improve code quality a bit. It also shrinks the kernel text: Before: text data bss dec filename 21828379 5194760 1277952 28301091 vmlinux After: text data bss dec filename 21827997 5194760 1277952 28300709 vmlinux ... by 382 bytes. Signed-off-by:
-
