Commit 432c6bac authored by Paul Burton's avatar Paul Burton Committed by Ralf Baechle

MIPS: Use per-mm page to execute branch delay slot instructions

In some cases the kernel needs to execute an instruction from the delay
slot of an emulated branch instruction. These cases include:

  - Emulated floating point branch instructions (bc1[ft]l?) for systems
    which don't include an FPU, or upon which the kernel is run with the
    "nofpu" parameter.

  - MIPSr6 systems running binaries targeting older revisions of the
    architecture, which may include branch instructions whose encodings
    are no longer valid in MIPSr6.

Executing instructions from such delay slots is done by writing the
instruction to memory followed by a trap, as part of an "emuframe", and
executing it. This avoids the requirement of an emulator for the entire
MIPS instruction set. Prior to this patch such emuframes are written to
the user stack and executed from there.

This patch moves FP branch delay emuframes off of the user stack and
into a per-mm page. Allocating a page per-mm leaves userland with access
to only what it had access to previously, and compared to other
solutions is relatively simple.

When a thread requires a delay slot emulation, it is allocated a frame.
A thread may only have one frame allocated at any one time, since it may
only ever be executing one instruction at any one time. In order to
ensure that we can free up allocated frame later, its index is recorded
in struct thread_struct. In the typical case, after executing the delay
slot instruction we'll execute a break instruction with the BRK_MEMU
code. This traps back to the kernel & leads to a call to do_dsemulret
which frees the allocated frame & moves the user PC back to the
instruction that would have executed following the emulated branch.
In some cases the delay slot instruction may be invalid, such as a
branch, or may trigger an exception. In these cases the BRK_MEMU break
instruction will not be hit. In order to ensure that frames are freed
this patch introduces dsemul_thread_cleanup() and calls it to free any
allocated frame upon thread exit. If the instruction generated an
exception & leads to a signal being delivered to the thread, or indeed
if a signal simply happens to be delivered to the thread whilst it is
executing from the struct emuframe, then we need to take care to exit
the frame appropriately. This is done by either rolling back the user PC
to the branch or advancing it to the continuation PC prior to signal
delivery, using dsemul_thread_rollback(). If this were not done then a
sigreturn would return to the struct emuframe, and if that frame had
meanwhile been used in response to an emulated branch instruction within
the signal handler then we would execute the wrong user code.

Whilst a user could theoretically place something like a compact branch
to self in a delay slot and cause their thread to become stuck in an
infinite loop with the frame never being deallocated, this would:

  - Only affect the users single process.

  - Be architecturally invalid since there would be a branch in the
    delay slot, which is forbidden.

  - Be extremely unlikely to happen by mistake, and provide a program
    with no more ability to harm the system than a simple infinite loop
    would.

If a thread requires a delay slot emulation & no frame is available to
it (ie. the process has enough other threads that all frames are
currently in use) then the thread joins a waitqueue. It will sleep until
a frame is freed by another thread in the process.

Since we now know whether a thread has an allocated frame due to our
tracking of its index, the cookie field of struct emuframe is removed as
we can be more certain whether we have a valid frame. Since a thread may
only ever have a single frame at any given time, the epc field of struct
emuframe is also removed & the PC to continue from is instead stored in
struct thread_struct. Together these changes simplify & shrink struct
emuframe somewhat, allowing twice as many frames to fit into the page
allocated for them.

The primary benefit of this patch is that we are now free to mark the
user stack non-executable where that is possible.
Signed-off-by: default avatarPaul Burton <paul.burton@imgtec.com>
Cc: Leonid Yegoshin <leonid.yegoshin@imgtec.com>
Cc: Maciej Rozycki <maciej.rozycki@imgtec.com>
Cc: Faraz Shahbazker <faraz.shahbazker@imgtec.com>
Cc: Raghu Gandham <raghu.gandham@imgtec.com>
Cc: Matthew Fortune <matthew.fortune@imgtec.com>
Cc: linux-mips@linux-mips.org
Patchwork: https://patchwork.linux-mips.org/patch/13764/Signed-off-by: default avatarRalf Baechle <ralf@linux-mips.org>
parent 33799a6d
...@@ -64,6 +64,7 @@ config MIPS ...@@ -64,6 +64,7 @@ config MIPS
select GENERIC_TIME_VSYSCALL select GENERIC_TIME_VSYSCALL
select ARCH_CLOCKSOURCE_DATA select ARCH_CLOCKSOURCE_DATA
select HANDLE_DOMAIN_IRQ select HANDLE_DOMAIN_IRQ
select HAVE_EXIT_THREAD
menu "Machine selection" menu "Machine selection"
......
/*
* Copyright (C) 2016 Imagination Technologies
* Author: Paul Burton <paul.burton@imgtec.com>
*
* This program is free software; you can redistribute it and/or modify it
* under the terms of the GNU General Public License as published by the
* Free Software Foundation; either version 2 of the License, or (at your
* option) any later version.
*/
#ifndef __MIPS_ASM_DSEMUL_H__
#define __MIPS_ASM_DSEMUL_H__
#include <asm/break.h>
#include <asm/inst.h>
/* Break instruction with special math emu break code set */
#define BREAK_MATH(micromips) (((micromips) ? 0x7 : 0xd) | (BRK_MEMU << 16))
/* When used as a frame index, indicates the lack of a frame */
#define BD_EMUFRAME_NONE ((int)BIT(31))
struct mm_struct;
struct pt_regs;
struct task_struct;
/**
* mips_dsemul() - 'Emulate' an instruction from a branch delay slot
* @regs: User thread register context.
* @ir: The instruction to be 'emulated'.
* @branch_pc: The PC of the branch instruction.
* @cont_pc: The PC to continue at following 'emulation'.
*
* Emulate or execute an arbitrary MIPS instruction within the context of
* the current user thread. This is used primarily to handle instructions
* in the delay slots of emulated branch instructions, for example FP
* branch instructions on systems without an FPU.
*
* Return: Zero on success, negative if ir is a NOP, signal number on failure.
*/
extern int mips_dsemul(struct pt_regs *regs, mips_instruction ir,
unsigned long branch_pc, unsigned long cont_pc);
/**
* do_dsemulret() - Return from a delay slot 'emulation' frame
* @xcp: User thread register context.
*
* Call in response to the BRK_MEMU break instruction used to return to
* the kernel from branch delay slot 'emulation' frames following a call
* to mips_dsemul(). Restores the user thread PC to the value that was
* passed as the cpc parameter to mips_dsemul().
*
* Return: True if an emulation frame was returned from, else false.
*/
extern bool do_dsemulret(struct pt_regs *xcp);
/**
* dsemul_thread_cleanup() - Cleanup thread 'emulation' frame
* @tsk: The task structure associated with the thread
*
* If the thread @tsk has a branch delay slot 'emulation' frame
* allocated to it then free that frame.
*
* Return: True if a frame was freed, else false.
*/
extern bool dsemul_thread_cleanup(struct task_struct *tsk);
/**
* dsemul_thread_rollback() - Rollback from an 'emulation' frame
* @regs: User thread register context.
*
* If the current thread, whose register context is represented by @regs,
* is executing within a delay slot 'emulation' frame then exit that
* frame. The PC will be rolled back to the branch if the instruction
* that was being 'emulated' has not yet executed, or advanced to the
* continuation PC if it has.
*
* Return: True if a frame was exited, else false.
*/
extern bool dsemul_thread_rollback(struct pt_regs *regs);
/**
* dsemul_mm_cleanup() - Cleanup per-mm delay slot 'emulation' state
* @mm: The struct mm_struct to cleanup state for.
*
* Cleanup state for the given @mm, ensuring that any memory allocated
* for delay slot 'emulation' book-keeping is freed. This is to be called
* before @mm is freed in order to avoid memory leaks.
*/
extern void dsemul_mm_cleanup(struct mm_struct *mm);
#endif /* __MIPS_ASM_DSEMUL_H__ */
...@@ -24,7 +24,7 @@ ...@@ -24,7 +24,7 @@
#define _ASM_FPU_EMULATOR_H #define _ASM_FPU_EMULATOR_H
#include <linux/sched.h> #include <linux/sched.h>
#include <asm/break.h> #include <asm/dsemul.h>
#include <asm/thread_info.h> #include <asm/thread_info.h>
#include <asm/inst.h> #include <asm/inst.h>
#include <asm/local.h> #include <asm/local.h>
...@@ -60,27 +60,16 @@ do { \ ...@@ -60,27 +60,16 @@ do { \
#define MIPS_FPU_EMU_INC_STATS(M) do { } while (0) #define MIPS_FPU_EMU_INC_STATS(M) do { } while (0)
#endif /* CONFIG_DEBUG_FS */ #endif /* CONFIG_DEBUG_FS */
extern int mips_dsemul(struct pt_regs *regs, mips_instruction ir,
unsigned long cpc);
extern int do_dsemulret(struct pt_regs *xcp);
extern int fpu_emulator_cop1Handler(struct pt_regs *xcp, extern int fpu_emulator_cop1Handler(struct pt_regs *xcp,
struct mips_fpu_struct *ctx, int has_fpu, struct mips_fpu_struct *ctx, int has_fpu,
void *__user *fault_addr); void *__user *fault_addr);
int process_fpemu_return(int sig, void __user *fault_addr, int process_fpemu_return(int sig, void __user *fault_addr,
unsigned long fcr31); unsigned long fcr31);
int isBranchInstr(struct pt_regs *regs, struct mm_decoded_insn dec_insn,
unsigned long *contpc);
int mm_isBranchInstr(struct pt_regs *regs, struct mm_decoded_insn dec_insn, int mm_isBranchInstr(struct pt_regs *regs, struct mm_decoded_insn dec_insn,
unsigned long *contpc); unsigned long *contpc);
/*
* Instruction inserted following the badinst to further tag the sequence
*/
#define BD_COOKIE 0x0000bd36 /* tne $0, $0 with baggage */
/*
* Break instruction with special math emu break code set
*/
#define BREAK_MATH(micromips) (((micromips) ? 0x7 : 0xd) | (BRK_MEMU << 16))
#define SIGNALLING_NAN 0x7ff800007ff80000LL #define SIGNALLING_NAN 0x7ff800007ff80000LL
static inline void fpu_emulator_init_fpu(void) static inline void fpu_emulator_init_fpu(void)
......
...@@ -2,11 +2,20 @@ ...@@ -2,11 +2,20 @@
#define __ASM_MMU_H #define __ASM_MMU_H
#include <linux/atomic.h> #include <linux/atomic.h>
#include <linux/spinlock.h>
#include <linux/wait.h>
typedef struct { typedef struct {
unsigned long asid[NR_CPUS]; unsigned long asid[NR_CPUS];
void *vdso; void *vdso;
atomic_t fp_mode_switching; atomic_t fp_mode_switching;
/* lock to be held whilst modifying fp_bd_emupage_allocmap */
spinlock_t bd_emupage_lock;
/* bitmap tracking allocation of fp_bd_emupage */
unsigned long *bd_emupage_allocmap;
/* wait queue for threads requiring an emuframe */
wait_queue_head_t bd_emupage_queue;
} mm_context_t; } mm_context_t;
#endif /* __ASM_MMU_H */ #endif /* __ASM_MMU_H */
...@@ -16,6 +16,7 @@ ...@@ -16,6 +16,7 @@
#include <linux/smp.h> #include <linux/smp.h>
#include <linux/slab.h> #include <linux/slab.h>
#include <asm/cacheflush.h> #include <asm/cacheflush.h>
#include <asm/dsemul.h>
#include <asm/hazards.h> #include <asm/hazards.h>
#include <asm/tlbflush.h> #include <asm/tlbflush.h>
#include <asm-generic/mm_hooks.h> #include <asm-generic/mm_hooks.h>
...@@ -128,6 +129,10 @@ init_new_context(struct task_struct *tsk, struct mm_struct *mm) ...@@ -128,6 +129,10 @@ init_new_context(struct task_struct *tsk, struct mm_struct *mm)
atomic_set(&mm->context.fp_mode_switching, 0); atomic_set(&mm->context.fp_mode_switching, 0);
mm->context.bd_emupage_allocmap = NULL;
spin_lock_init(&mm->context.bd_emupage_lock);
init_waitqueue_head(&mm->context.bd_emupage_queue);
return 0; return 0;
} }
...@@ -162,6 +167,7 @@ static inline void switch_mm(struct mm_struct *prev, struct mm_struct *next, ...@@ -162,6 +167,7 @@ static inline void switch_mm(struct mm_struct *prev, struct mm_struct *next,
*/ */
static inline void destroy_context(struct mm_struct *mm) static inline void destroy_context(struct mm_struct *mm)
{ {
dsemul_mm_cleanup(mm);
} }
#define deactivate_mm(tsk, mm) do { } while (0) #define deactivate_mm(tsk, mm) do { } while (0)
......
...@@ -11,12 +11,14 @@ ...@@ -11,12 +11,14 @@
#ifndef _ASM_PROCESSOR_H #ifndef _ASM_PROCESSOR_H
#define _ASM_PROCESSOR_H #define _ASM_PROCESSOR_H
#include <linux/atomic.h>
#include <linux/cpumask.h> #include <linux/cpumask.h>
#include <linux/threads.h> #include <linux/threads.h>
#include <asm/cachectl.h> #include <asm/cachectl.h>
#include <asm/cpu.h> #include <asm/cpu.h>
#include <asm/cpu-info.h> #include <asm/cpu-info.h>
#include <asm/dsemul.h>
#include <asm/mipsregs.h> #include <asm/mipsregs.h>
#include <asm/prefetch.h> #include <asm/prefetch.h>
...@@ -78,7 +80,11 @@ extern unsigned int vced_count, vcei_count; ...@@ -78,7 +80,11 @@ extern unsigned int vced_count, vcei_count;
#endif #endif
#define STACK_TOP (TASK_SIZE & PAGE_MASK) /*
* One page above the stack is used for branch delay slot "emulation".
* See dsemul.c for details.
*/
#define STACK_TOP ((TASK_SIZE & PAGE_MASK) - PAGE_SIZE)
/* /*
* This decides where the kernel will search for a free chunk of vm * This decides where the kernel will search for a free chunk of vm
...@@ -256,6 +262,12 @@ struct thread_struct { ...@@ -256,6 +262,12 @@ struct thread_struct {
/* Saved fpu/fpu emulator stuff. */ /* Saved fpu/fpu emulator stuff. */
struct mips_fpu_struct fpu FPU_ALIGN; struct mips_fpu_struct fpu FPU_ALIGN;
/* Assigned branch delay slot 'emulation' frame */
atomic_t bd_emu_frame;
/* PC of the branch from a branch delay slot 'emulation' */
unsigned long bd_emu_branch_pc;
/* PC to continue from following a branch delay slot 'emulation' */
unsigned long bd_emu_cont_pc;
#ifdef CONFIG_MIPS_MT_FPAFF #ifdef CONFIG_MIPS_MT_FPAFF
/* Emulated instruction count */ /* Emulated instruction count */
unsigned long emulated_fp; unsigned long emulated_fp;
...@@ -323,6 +335,10 @@ struct thread_struct { ...@@ -323,6 +335,10 @@ struct thread_struct {
* FPU affinity state (null if not FPAFF) \ * FPU affinity state (null if not FPAFF) \
*/ \ */ \
FPAFF_INIT \ FPAFF_INIT \
/* Delay slot emulation */ \
.bd_emu_frame = ATOMIC_INIT(BD_EMUFRAME_NONE), \
.bd_emu_branch_pc = 0, \
.bd_emu_cont_pc = 0, \
/* \ /* \
* Saved DSP stuff \ * Saved DSP stuff \
*/ \ */ \
......
...@@ -283,7 +283,7 @@ static int jr_func(struct pt_regs *regs, u32 ir) ...@@ -283,7 +283,7 @@ static int jr_func(struct pt_regs *regs, u32 ir)
err = mipsr6_emul(regs, nir); err = mipsr6_emul(regs, nir);
if (err > 0) { if (err > 0) {
regs->cp0_epc = nepc; regs->cp0_epc = nepc;
err = mips_dsemul(regs, nir, cepc); err = mips_dsemul(regs, nir, epc, cepc);
if (err == SIGILL) if (err == SIGILL)
err = SIGEMT; err = SIGEMT;
MIPS_R2_STATS(dsemul); MIPS_R2_STATS(dsemul);
...@@ -1033,7 +1033,7 @@ int mipsr2_decoder(struct pt_regs *regs, u32 inst, unsigned long *fcr31) ...@@ -1033,7 +1033,7 @@ int mipsr2_decoder(struct pt_regs *regs, u32 inst, unsigned long *fcr31)
if (nir) { if (nir) {
err = mipsr6_emul(regs, nir); err = mipsr6_emul(regs, nir);
if (err > 0) { if (err > 0) {
err = mips_dsemul(regs, nir, cpc); err = mips_dsemul(regs, nir, epc, cpc);
if (err == SIGILL) if (err == SIGILL)
err = SIGEMT; err = SIGEMT;
MIPS_R2_STATS(dsemul); MIPS_R2_STATS(dsemul);
...@@ -1082,7 +1082,7 @@ int mipsr2_decoder(struct pt_regs *regs, u32 inst, unsigned long *fcr31) ...@@ -1082,7 +1082,7 @@ int mipsr2_decoder(struct pt_regs *regs, u32 inst, unsigned long *fcr31)
if (nir) { if (nir) {
err = mipsr6_emul(regs, nir); err = mipsr6_emul(regs, nir);
if (err > 0) { if (err > 0) {
err = mips_dsemul(regs, nir, cpc); err = mips_dsemul(regs, nir, epc, cpc);
if (err == SIGILL) if (err == SIGILL)
err = SIGEMT; err = SIGEMT;
MIPS_R2_STATS(dsemul); MIPS_R2_STATS(dsemul);
...@@ -1149,7 +1149,7 @@ int mipsr2_decoder(struct pt_regs *regs, u32 inst, unsigned long *fcr31) ...@@ -1149,7 +1149,7 @@ int mipsr2_decoder(struct pt_regs *regs, u32 inst, unsigned long *fcr31)
if (nir) { if (nir) {
err = mipsr6_emul(regs, nir); err = mipsr6_emul(regs, nir);
if (err > 0) { if (err > 0) {
err = mips_dsemul(regs, nir, cpc); err = mips_dsemul(regs, nir, epc, cpc);
if (err == SIGILL) if (err == SIGILL)
err = SIGEMT; err = SIGEMT;
MIPS_R2_STATS(dsemul); MIPS_R2_STATS(dsemul);
......
...@@ -30,6 +30,7 @@ ...@@ -30,6 +30,7 @@
#include <asm/asm.h> #include <asm/asm.h>
#include <asm/bootinfo.h> #include <asm/bootinfo.h>
#include <asm/cpu.h> #include <asm/cpu.h>
#include <asm/dsemul.h>
#include <asm/dsp.h> #include <asm/dsp.h>
#include <asm/fpu.h> #include <asm/fpu.h>
#include <asm/msa.h> #include <asm/msa.h>
...@@ -68,11 +69,22 @@ void start_thread(struct pt_regs * regs, unsigned long pc, unsigned long sp) ...@@ -68,11 +69,22 @@ void start_thread(struct pt_regs * regs, unsigned long pc, unsigned long sp)
lose_fpu(0); lose_fpu(0);
clear_thread_flag(TIF_MSA_CTX_LIVE); clear_thread_flag(TIF_MSA_CTX_LIVE);
clear_used_math(); clear_used_math();
atomic_set(&current->thread.bd_emu_frame, BD_EMUFRAME_NONE);
init_dsp(); init_dsp();
regs->cp0_epc = pc; regs->cp0_epc = pc;
regs->regs[29] = sp; regs->regs[29] = sp;
} }
void exit_thread(struct task_struct *tsk)
{
/*
* User threads may have allocated a delay slot emulation frame.
* If so, clean up that allocation.
*/
if (!(current->flags & PF_KTHREAD))
dsemul_thread_cleanup(tsk);
}
int arch_dup_task_struct(struct task_struct *dst, struct task_struct *src) int arch_dup_task_struct(struct task_struct *dst, struct task_struct *src)
{ {
/* /*
...@@ -159,6 +171,8 @@ int copy_thread(unsigned long clone_flags, unsigned long usp, ...@@ -159,6 +171,8 @@ int copy_thread(unsigned long clone_flags, unsigned long usp,
clear_tsk_thread_flag(p, TIF_FPUBOUND); clear_tsk_thread_flag(p, TIF_FPUBOUND);
#endif /* CONFIG_MIPS_MT_FPAFF */ #endif /* CONFIG_MIPS_MT_FPAFF */
atomic_set(&p->thread.bd_emu_frame, BD_EMUFRAME_NONE);
if (clone_flags & CLONE_SETTLS) if (clone_flags & CLONE_SETTLS)
ti->tp_value = regs->regs[7]; ti->tp_value = regs->regs[7];
......
...@@ -772,6 +772,14 @@ static void handle_signal(struct ksignal *ksig, struct pt_regs *regs) ...@@ -772,6 +772,14 @@ static void handle_signal(struct ksignal *ksig, struct pt_regs *regs)
struct mips_abi *abi = current->thread.abi; struct mips_abi *abi = current->thread.abi;
void *vdso = current->mm->context.vdso; void *vdso = current->mm->context.vdso;
/*
* If we were emulating a delay slot instruction, exit that frame such
* that addresses in the sigframe are as expected for userland and we
* don't have a problem if we reuse the thread's frame for an
* instruction within the signal handler.
*/
dsemul_thread_rollback(regs);
if (regs->regs[0]) { if (regs->regs[0]) {
switch(regs->regs[2]) { switch(regs->regs[2]) {
case ERESTART_RESTARTBLOCK: case ERESTART_RESTARTBLOCK:
......
...@@ -107,6 +107,16 @@ int arch_setup_additional_pages(struct linux_binprm *bprm, int uses_interp) ...@@ -107,6 +107,16 @@ int arch_setup_additional_pages(struct linux_binprm *bprm, int uses_interp)
if (down_write_killable(&mm->mmap_sem)) if (down_write_killable(&mm->mmap_sem))
return -EINTR; return -EINTR;
/* Map delay slot emulation page */
base = mmap_region(NULL, STACK_TOP, PAGE_SIZE,
VM_READ|VM_WRITE|VM_EXEC|
VM_MAYREAD|VM_MAYWRITE|VM_MAYEXEC,
0);
if (IS_ERR_VALUE(base)) {
ret = base;
goto out;
}
/* /*
* Determine total area size. This includes the VDSO data itself, the * Determine total area size. This includes the VDSO data itself, the
* data page, and the GIC user page if present. Always create a mapping * data page, and the GIC user page if present. Always create a mapping
......
...@@ -434,8 +434,8 @@ static int microMIPS32_to_MIPS32(union mips_instruction *insn_ptr) ...@@ -434,8 +434,8 @@ static int microMIPS32_to_MIPS32(union mips_instruction *insn_ptr)
* a single subroutine should be used across both * a single subroutine should be used across both
* modules. * modules.
*/ */
static int isBranchInstr(struct pt_regs *regs, struct mm_decoded_insn dec_insn, int isBranchInstr(struct pt_regs *regs, struct mm_decoded_insn dec_insn,
unsigned long *contpc) unsigned long *contpc)
{ {
union mips_instruction insn = (union mips_instruction)dec_insn.insn; union mips_instruction insn = (union mips_instruction)dec_insn.insn;
unsigned int fcr31; unsigned int fcr31;
...@@ -1268,7 +1268,7 @@ static int cop1Emulate(struct pt_regs *xcp, struct mips_fpu_struct *ctx, ...@@ -1268,7 +1268,7 @@ static int cop1Emulate(struct pt_regs *xcp, struct mips_fpu_struct *ctx,
* instruction in the dslot. * instruction in the dslot.
*/ */
sig = mips_dsemul(xcp, ir, sig = mips_dsemul(xcp, ir,
contpc); bcpc, contpc);
if (sig < 0) if (sig < 0)
break; break;
if (sig) if (sig)
...@@ -1323,7 +1323,7 @@ static int cop1Emulate(struct pt_regs *xcp, struct mips_fpu_struct *ctx, ...@@ -1323,7 +1323,7 @@ static int cop1Emulate(struct pt_regs *xcp, struct mips_fpu_struct *ctx,
* Single step the non-cp1 * Single step the non-cp1
* instruction in the dslot * instruction in the dslot
*/ */
sig = mips_dsemul(xcp, ir, contpc); sig = mips_dsemul(xcp, ir, bcpc, contpc);
if (sig < 0) if (sig < 0)
break; break;
if (sig) if (sig)
......
This diff is collapsed.
Markdown is supported
0%
or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment