1. 19 May, 2015 40 commits
    • Ingo Molnar's avatar
      x86/fpu: Factor out fpu__init_system_generic() · 7218e8b7
      Ingo Molnar authored
      Factor out the generic bits from fpu__init_system().
      
      Rename mxcsr_feature_mask_init() to fpu__init_system_mxcsr()
      to bring it in line with the rest of the nomenclature.
      Reviewed-by: default avatarBorislav Petkov <bp@alien8.de>
      Cc: Andy Lutomirski <luto@amacapital.net>
      Cc: Dave Hansen <dave.hansen@linux.intel.com>
      Cc: Fenghua Yu <fenghua.yu@intel.com>
      Cc: H. Peter Anvin <hpa@zytor.com>
      Cc: Linus Torvalds <torvalds@linux-foundation.org>
      Cc: Oleg Nesterov <oleg@redhat.com>
      Cc: Peter Zijlstra <peterz@infradead.org>
      Cc: Thomas Gleixner <tglx@linutronix.de>
      Signed-off-by: default avatarIngo Molnar <mingo@kernel.org>
      7218e8b7
    • Ingo Molnar's avatar
      x86/fpu: Factor out fpu__init_cpu_generic() · b11316ed
      Ingo Molnar authored
      Factor out the generic bits from fpu__init_cpu(), to create
      a flat sequence of per CPU initialization function calls:
      
      	fpu__init_cpu_generic();
      	fpu__init_cpu_xstate();
      	fpu__init_cpu_ctx_switch();
      Reviewed-by: default avatarBorislav Petkov <bp@alien8.de>
      Cc: Andy Lutomirski <luto@amacapital.net>
      Cc: Dave Hansen <dave.hansen@linux.intel.com>
      Cc: Fenghua Yu <fenghua.yu@intel.com>
      Cc: H. Peter Anvin <hpa@zytor.com>
      Cc: Linus Torvalds <torvalds@linux-foundation.org>
      Cc: Oleg Nesterov <oleg@redhat.com>
      Cc: Peter Zijlstra <peterz@infradead.org>
      Cc: Thomas Gleixner <tglx@linutronix.de>
      Signed-off-by: default avatarIngo Molnar <mingo@kernel.org>
      b11316ed
    • Ingo Molnar's avatar
      x86/fpu: Simplify fpu__cpu_init() · 21c4cd10
      Ingo Molnar authored
      After the latest round of cleanups, fpu__cpu_init() has become
      a simple call to fpu__init_cpu().
      
      Rename fpu__init_cpu() to fpu__cpu_init() and remove the
      extra layer.
      Reviewed-by: default avatarBorislav Petkov <bp@alien8.de>
      Cc: Andy Lutomirski <luto@amacapital.net>
      Cc: Dave Hansen <dave.hansen@linux.intel.com>
      Cc: Fenghua Yu <fenghua.yu@intel.com>
      Cc: H. Peter Anvin <hpa@zytor.com>
      Cc: Linus Torvalds <torvalds@linux-foundation.org>
      Cc: Oleg Nesterov <oleg@redhat.com>
      Cc: Peter Zijlstra <peterz@infradead.org>
      Cc: Thomas Gleixner <tglx@linutronix.de>
      Signed-off-by: default avatarIngo Molnar <mingo@kernel.org>
      21c4cd10
    • Ingo Molnar's avatar
      x86/fpu: Remove fpu__init_cpu_ctx_switch() call from fpu__init_system() · 7202ab46
      Ingo Molnar authored
      We are now doing the fpu__init_cpu_ctx_switch() call from fpu__init_cpu(),
      so there's no need to call it from fpu__init_system() anymore.
      Reviewed-by: default avatarBorislav Petkov <bp@alien8.de>
      Cc: Andy Lutomirski <luto@amacapital.net>
      Cc: Dave Hansen <dave.hansen@linux.intel.com>
      Cc: Fenghua Yu <fenghua.yu@intel.com>
      Cc: H. Peter Anvin <hpa@zytor.com>
      Cc: Linus Torvalds <torvalds@linux-foundation.org>
      Cc: Oleg Nesterov <oleg@redhat.com>
      Cc: Peter Zijlstra <peterz@infradead.org>
      Cc: Thomas Gleixner <tglx@linutronix.de>
      Signed-off-by: default avatarIngo Molnar <mingo@kernel.org>
      7202ab46
    • Ingo Molnar's avatar
      x86/fpu: Do system-wide setup from fpu__detect() · 067051cc
      Ingo Molnar authored
      fpu__cpu_init() is called on every CPU, so it is the wrong place
      to call fpu__init_system() from. Call it from fpu__detect():
      this is early CPU init code, but we already have CPU features detected,
      so we can call the system-wide FPU init code from here.
      Reviewed-by: default avatarBorislav Petkov <bp@alien8.de>
      Cc: Andy Lutomirski <luto@amacapital.net>
      Cc: Dave Hansen <dave.hansen@linux.intel.com>
      Cc: Fenghua Yu <fenghua.yu@intel.com>
      Cc: H. Peter Anvin <hpa@zytor.com>
      Cc: Linus Torvalds <torvalds@linux-foundation.org>
      Cc: Oleg Nesterov <oleg@redhat.com>
      Cc: Peter Zijlstra <peterz@infradead.org>
      Cc: Thomas Gleixner <tglx@linutronix.de>
      Signed-off-by: default avatarIngo Molnar <mingo@kernel.org>
      067051cc
    • Ingo Molnar's avatar
      x86/fpu: Call fpu__init_cpu_ctx_switch() from fpu__init_cpu() · 3960fccf
      Ingo Molnar authored
      fpu__init_cpu() is currently called from fpu__init_system(),
      which is the wrong place for it: call it from the proper high level
      per CPU init function, fpu__init_cpu().
      
      Note, we still keep the old call site as well, because it depends
      on having proper CR0::TS setup. We'll fix this in the next patch.
      Reviewed-by: default avatarBorislav Petkov <bp@alien8.de>
      Cc: Andy Lutomirski <luto@amacapital.net>
      Cc: Dave Hansen <dave.hansen@linux.intel.com>
      Cc: Fenghua Yu <fenghua.yu@intel.com>
      Cc: H. Peter Anvin <hpa@zytor.com>
      Cc: Linus Torvalds <torvalds@linux-foundation.org>
      Cc: Oleg Nesterov <oleg@redhat.com>
      Cc: Peter Zijlstra <peterz@infradead.org>
      Cc: Thomas Gleixner <tglx@linutronix.de>
      Signed-off-by: default avatarIngo Molnar <mingo@kernel.org>
      3960fccf
    • Ingo Molnar's avatar
      x86/fpu: Move the fpstate_xstate_init_size() call into fpu__init_system() · 997578b1
      Ingo Molnar authored
      The fpstate_xstate_init_size() function sets up a basic xstate_size, called
      during fpu__detect() currently.
      
      Its real dependency is to be called before fpu__init_system_xstate().
      
      So move the function call site into fpu__init_system(), to right before the
      fpu__init_system_xstate() call.
      
      Also add a once-per-boot flag to fpstate_xstate_init_size(), we'll remove
      this quirk later once we've cleaned up the init dependencies.
      
      This moves the two related functions closer to each other and makes them
      both part of the _init_system() functionality.
      
      Currently we do the fpstate_xstate_init_size()
      Reviewed-by: default avatarBorislav Petkov <bp@alien8.de>
      Cc: Andy Lutomirski <luto@amacapital.net>
      Cc: Dave Hansen <dave.hansen@linux.intel.com>
      Cc: Fenghua Yu <fenghua.yu@intel.com>
      Cc: H. Peter Anvin <hpa@zytor.com>
      Cc: Linus Torvalds <torvalds@linux-foundation.org>
      Cc: Oleg Nesterov <oleg@redhat.com>
      Cc: Peter Zijlstra <peterz@infradead.org>
      Cc: Thomas Gleixner <tglx@linutronix.de>
      Signed-off-by: default avatarIngo Molnar <mingo@kernel.org>
      997578b1
    • Ingo Molnar's avatar
      x86/fpu: Do CLTS fpu__init_system() · 530b37e4
      Ingo Molnar authored
      mxcsr_feature_mask_init() depends on TS being cleared, as it executes
      an FXSAVE instruction.
      
      After later changes we will move the TS setup into fpu__init_cpu(),
      which will interact with this - so clear the TS flag explicitly.
      Reviewed-by: default avatarBorislav Petkov <bp@alien8.de>
      Cc: Andy Lutomirski <luto@amacapital.net>
      Cc: Dave Hansen <dave.hansen@linux.intel.com>
      Cc: Fenghua Yu <fenghua.yu@intel.com>
      Cc: H. Peter Anvin <hpa@zytor.com>
      Cc: Linus Torvalds <torvalds@linux-foundation.org>
      Cc: Oleg Nesterov <oleg@redhat.com>
      Cc: Peter Zijlstra <peterz@infradead.org>
      Cc: Thomas Gleixner <tglx@linutronix.de>
      Signed-off-by: default avatarIngo Molnar <mingo@kernel.org>
      530b37e4
    • Ingo Molnar's avatar
      x86/fpu: Split fpu__ctx_switch_init() into _cpu() and _system() portions · 011545b5
      Ingo Molnar authored
      So fpu__ctx_switch_init() has two aspects: a once per bootup functionality
      that sets up a capability flag, and a per CPU functionality that sets CR0::TS.
      
      Split the function.
      
      Note that at this stage we still have duplicate calls into these methods, as
      both the _system() and the _cpu() methods are run on all CPUs, with lower
      level on_boot_cpu flags filtering out the duplicates where needed. So add
      TS flag clearing as well, to handle the aftermath of early CPU init sequences
      that might call in without having eager-fpu set - don't assume the TS flag
      is cleared.
      
      Calling each from its respective init level will happen later on.
      Reviewed-by: default avatarBorislav Petkov <bp@alien8.de>
      Cc: Andy Lutomirski <luto@amacapital.net>
      Cc: Dave Hansen <dave.hansen@linux.intel.com>
      Cc: Fenghua Yu <fenghua.yu@intel.com>
      Cc: H. Peter Anvin <hpa@zytor.com>
      Cc: Linus Torvalds <torvalds@linux-foundation.org>
      Cc: Oleg Nesterov <oleg@redhat.com>
      Cc: Peter Zijlstra <peterz@infradead.org>
      Cc: Thomas Gleixner <tglx@linutronix.de>
      Signed-off-by: default avatarIngo Molnar <mingo@kernel.org>
      011545b5
    • Ingo Molnar's avatar
      x86/fpu: Clean up eager_fpu_init() and rename it to fpu__ctx_switch_init() · 064e51e3
      Ingo Molnar authored
      It's not an xsave specific function anymore, so rename it accordingly
      and also clean it up a bit:
      
       - remove the obsolete __init_refok, as the code paths are not
         mixed anymore
      
       - rename it from eager_fpu_init() to fpu__ctx_switch_init()
      
       - remove stray 'return;'
      
       - make it static to its only user
      Reviewed-by: default avatarBorislav Petkov <bp@alien8.de>
      Cc: Andy Lutomirski <luto@amacapital.net>
      Cc: Dave Hansen <dave.hansen@linux.intel.com>
      Cc: Fenghua Yu <fenghua.yu@intel.com>
      Cc: H. Peter Anvin <hpa@zytor.com>
      Cc: Linus Torvalds <torvalds@linux-foundation.org>
      Cc: Oleg Nesterov <oleg@redhat.com>
      Cc: Peter Zijlstra <peterz@infradead.org>
      Cc: Thomas Gleixner <tglx@linutronix.de>
      Signed-off-by: default avatarIngo Molnar <mingo@kernel.org>
      064e51e3
    • Ingo Molnar's avatar
      x86/fpu: Move eager_fpu_init() to fpu/init.c · 6f5d265a
      Ingo Molnar authored
      Move eager_fpu_init() and the 'eagerfpu' boot parameter handling function
      to the generic FPU init file: it's generic FPU functionality.
      Reviewed-by: default avatarBorislav Petkov <bp@alien8.de>
      Cc: Andy Lutomirski <luto@amacapital.net>
      Cc: Dave Hansen <dave.hansen@linux.intel.com>
      Cc: Fenghua Yu <fenghua.yu@intel.com>
      Cc: H. Peter Anvin <hpa@zytor.com>
      Cc: Linus Torvalds <torvalds@linux-foundation.org>
      Cc: Oleg Nesterov <oleg@redhat.com>
      Cc: Peter Zijlstra <peterz@infradead.org>
      Cc: Thomas Gleixner <tglx@linutronix.de>
      Signed-off-by: default avatarIngo Molnar <mingo@kernel.org>
      6f5d265a
    • Ingo Molnar's avatar
      x86/fpu: Move all eager-fpu setup code to eager_fpu_init() · 89abbe01
      Ingo Molnar authored
      The FPU context switch type (lazy or eager) setup code is split into
      two places currently - move it all to eager_fpu_init().
      
      Note that the code we move will now be executed on non-xstate CPUs
      as well, but this should be safe: both xfeatures_mask and
      cpu_has_xsaveopt is 0 there.
      Reviewed-by: default avatarBorislav Petkov <bp@alien8.de>
      Cc: Andy Lutomirski <luto@amacapital.net>
      Cc: Dave Hansen <dave.hansen@linux.intel.com>
      Cc: Fenghua Yu <fenghua.yu@intel.com>
      Cc: H. Peter Anvin <hpa@zytor.com>
      Cc: Linus Torvalds <torvalds@linux-foundation.org>
      Cc: Oleg Nesterov <oleg@redhat.com>
      Cc: Peter Zijlstra <peterz@infradead.org>
      Cc: Thomas Gleixner <tglx@linutronix.de>
      Signed-off-by: default avatarIngo Molnar <mingo@kernel.org>
      89abbe01
    • Ingo Molnar's avatar
      x86/fpu: Remove setup_init_fpu_buf() call from eager_fpu_init() · a5cb56e9
      Ingo Molnar authored
      It's a pure xstate method now, no need for this duplicate call.
      Reviewed-by: default avatarBorislav Petkov <bp@alien8.de>
      Cc: Andy Lutomirski <luto@amacapital.net>
      Cc: Dave Hansen <dave.hansen@linux.intel.com>
      Cc: Fenghua Yu <fenghua.yu@intel.com>
      Cc: H. Peter Anvin <hpa@zytor.com>
      Cc: Linus Torvalds <torvalds@linux-foundation.org>
      Cc: Oleg Nesterov <oleg@redhat.com>
      Cc: Peter Zijlstra <peterz@infradead.org>
      Cc: Thomas Gleixner <tglx@linutronix.de>
      Signed-off-by: default avatarIngo Molnar <mingo@kernel.org>
      a5cb56e9
    • Ingo Molnar's avatar
      x86/fpu: Set up the legacy FPU init image from fpu__init_system() · 2507e1c0
      Ingo Molnar authored
      The legacy FPU init image is used on older CPUs who don't run xstate init.
      But the init code is called within setup_init_fpu_buf(), an xstate method.
      
      Move this legacy init out of the xstate code and put it into fpu/init.c.
      Reviewed-by: default avatarBorislav Petkov <bp@alien8.de>
      Cc: Andy Lutomirski <luto@amacapital.net>
      Cc: Dave Hansen <dave.hansen@linux.intel.com>
      Cc: Fenghua Yu <fenghua.yu@intel.com>
      Cc: H. Peter Anvin <hpa@zytor.com>
      Cc: Linus Torvalds <torvalds@linux-foundation.org>
      Cc: Oleg Nesterov <oleg@redhat.com>
      Cc: Peter Zijlstra <peterz@infradead.org>
      Cc: Thomas Gleixner <tglx@linutronix.de>
      Signed-off-by: default avatarIngo Molnar <mingo@kernel.org>
      2507e1c0
    • Ingo Molnar's avatar
      x86/fpu: Do fpu__init_system_xstate only from fpu__init_system() · 429ced50
      Ingo Molnar authored
      Only call xstate system setup routines from fpu__init_system().
      
      Likewise, don't call fpu__init_cpu_xstate() from fpu__init_system().
      Reviewed-by: default avatarBorislav Petkov <bp@alien8.de>
      Cc: Andy Lutomirski <luto@amacapital.net>
      Cc: Dave Hansen <dave.hansen@linux.intel.com>
      Cc: Fenghua Yu <fenghua.yu@intel.com>
      Cc: H. Peter Anvin <hpa@zytor.com>
      Cc: Linus Torvalds <torvalds@linux-foundation.org>
      Cc: Oleg Nesterov <oleg@redhat.com>
      Cc: Peter Zijlstra <peterz@infradead.org>
      Cc: Thomas Gleixner <tglx@linutronix.de>
      Signed-off-by: default avatarIngo Molnar <mingo@kernel.org>
      429ced50
    • Ingo Molnar's avatar
      x86/fpu: Remove xsave_init() · c42103b2
      Ingo Molnar authored
      Expand fpu__init_system_xstate() and fpu__init_cpu_xstate() calls
      into xsave_init() calls.
      
      (This will allow us to call the proper versions in higher level FPU init code
      later on.)
      
      No change in functionality.
      Reviewed-by: default avatarBorislav Petkov <bp@alien8.de>
      Cc: Andy Lutomirski <luto@amacapital.net>
      Cc: Dave Hansen <dave.hansen@linux.intel.com>
      Cc: Fenghua Yu <fenghua.yu@intel.com>
      Cc: H. Peter Anvin <hpa@zytor.com>
      Cc: Linus Torvalds <torvalds@linux-foundation.org>
      Cc: Oleg Nesterov <oleg@redhat.com>
      Cc: Peter Zijlstra <peterz@infradead.org>
      Cc: Thomas Gleixner <tglx@linutronix.de>
      Signed-off-by: default avatarIngo Molnar <mingo@kernel.org>
      c42103b2
    • Ingo Molnar's avatar
      x86/fpu: Propagate once per boot quirk into fpu__init_system_xstate() · 62db6871
      Ingo Molnar authored
      Linearize the call sequence in xsave_init():
      
      	fpu__init_system_xstate();
      	fpu__init_cpu_xstate();
      
      We do this by propagating the boot-once quirk into
      fpu__init_system_xstate(). fpu__init_cpu_xstate() is
      safe to be called multiple time.
      Reviewed-by: default avatarBorislav Petkov <bp@alien8.de>
      Cc: Andy Lutomirski <luto@amacapital.net>
      Cc: Dave Hansen <dave.hansen@linux.intel.com>
      Cc: Fenghua Yu <fenghua.yu@intel.com>
      Cc: H. Peter Anvin <hpa@zytor.com>
      Cc: Linus Torvalds <torvalds@linux-foundation.org>
      Cc: Oleg Nesterov <oleg@redhat.com>
      Cc: Peter Zijlstra <peterz@infradead.org>
      Cc: Thomas Gleixner <tglx@linutronix.de>
      Signed-off-by: default avatarIngo Molnar <mingo@kernel.org>
      62db6871
    • Ingo Molnar's avatar
      x86/fpu: Move legacy check to fpu__init_system_xstate() · e9dbfd67
      Ingo Molnar authored
      Now that legacy code can execute fpu__init_cpu_xstate() in
      xsave_init(), we can move the once per boot legacy check into
      fpu__init_system_xstate(), where it belongs.
      Reviewed-by: default avatarBorislav Petkov <bp@alien8.de>
      Cc: Andy Lutomirski <luto@amacapital.net>
      Cc: Dave Hansen <dave.hansen@linux.intel.com>
      Cc: Fenghua Yu <fenghua.yu@intel.com>
      Cc: H. Peter Anvin <hpa@zytor.com>
      Cc: Linus Torvalds <torvalds@linux-foundation.org>
      Cc: Oleg Nesterov <oleg@redhat.com>
      Cc: Peter Zijlstra <peterz@infradead.org>
      Cc: Thomas Gleixner <tglx@linutronix.de>
      Signed-off-by: default avatarIngo Molnar <mingo@kernel.org>
      e9dbfd67
    • Ingo Molnar's avatar
      x86/fpu: Move CPU capability check into fpu__init_cpu_xstate() · e84611fc
      Ingo Molnar authored
      fpu__init_system_xstate() does an FPU capability check that is better
      done in fpu__init_cpu_xstate(). This will allow us to call
      fpu__init_cpu_xstate() directly on legacy CPUs as well.
      Reviewed-by: default avatarBorislav Petkov <bp@alien8.de>
      Cc: Andy Lutomirski <luto@amacapital.net>
      Cc: Dave Hansen <dave.hansen@linux.intel.com>
      Cc: Fenghua Yu <fenghua.yu@intel.com>
      Cc: H. Peter Anvin <hpa@zytor.com>
      Cc: Linus Torvalds <torvalds@linux-foundation.org>
      Cc: Oleg Nesterov <oleg@redhat.com>
      Cc: Peter Zijlstra <peterz@infradead.org>
      Cc: Thomas Gleixner <tglx@linutronix.de>
      Signed-off-by: default avatarIngo Molnar <mingo@kernel.org>
      e84611fc
    • Ingo Molnar's avatar
      x86/fpu: Make the system/cpu init distinction clear in the xstate code as well · 55cc4678
      Ingo Molnar authored
      Rename existing xstate init functions along the system/cpu init principles:
      
      	fpu__init_system_xstate(): called once per system bootup
      	fpu__init_cpu_xstate():    called per CPU onlining
      
      Also make the fpu__init_cpu_xstate() early code invariant:
      if xfeatures_mask is not set yet then don't crash but return.
      Reviewed-by: default avatarBorislav Petkov <bp@alien8.de>
      Cc: Andy Lutomirski <luto@amacapital.net>
      Cc: Dave Hansen <dave.hansen@linux.intel.com>
      Cc: Fenghua Yu <fenghua.yu@intel.com>
      Cc: H. Peter Anvin <hpa@zytor.com>
      Cc: Linus Torvalds <torvalds@linux-foundation.org>
      Cc: Oleg Nesterov <oleg@redhat.com>
      Cc: Peter Zijlstra <peterz@infradead.org>
      Cc: Thomas Gleixner <tglx@linutronix.de>
      Signed-off-by: default avatarIngo Molnar <mingo@kernel.org>
      55cc4678
    • Ingo Molnar's avatar
      x86/fpu: Split fpu__cpu_init() into early-boot and cpu-boot parts · e35f6f14
      Ingo Molnar authored
      There are two kinds of FPU initialization sequences necessary to bring FPU
      functionality up: once per system bootup activities, such as detection,
      feature initialization, etc. of attributes that are shared by all CPUs
      in the system - and per cpu initialization sequences run when a CPU is
      brought online (either during bootup or during CPU hotplug onlining),
      such as CR0/CR4 register setting, etc.
      
      The FPU code is mixing these roles together, with no clear distinction.
      
      Start sorting this out by splitting the main FPU detection routine
      (fpu__cpu_init()) into two parts: fpu__init_system() for
      one per system init activities, and fpu__init_cpu() for the
      per CPU onlining init activities.
      
      Note that xstate_init() is called from both variants for the time being,
      because it has a dual nature as well. We'll fix that in upcoming patches.
      
      Just do the split and call it as we used to before, don't introduce any
      change in initialization behavior yet, beyond duplicate (and harmless)
      fpu__init_cpu() and xstate_init() calls - which we'll fix in later
      patches.
      Reviewed-by: default avatarBorislav Petkov <bp@alien8.de>
      Cc: Andy Lutomirski <luto@amacapital.net>
      Cc: Dave Hansen <dave.hansen@linux.intel.com>
      Cc: Fenghua Yu <fenghua.yu@intel.com>
      Cc: H. Peter Anvin <hpa@zytor.com>
      Cc: Linus Torvalds <torvalds@linux-foundation.org>
      Cc: Oleg Nesterov <oleg@redhat.com>
      Cc: Peter Zijlstra <peterz@infradead.org>
      Cc: Thomas Gleixner <tglx@linutronix.de>
      Signed-off-by: default avatarIngo Molnar <mingo@kernel.org>
      e35f6f14
    • Ingo Molnar's avatar
      x86/fpu: Remove 'init_xstate_buf' bootmem allocation · 3e5e1267
      Ingo Molnar authored
      Make init_xstate_buf allocated statically at build time.
      
      This structure's maximum size is around 1KB - and it's allocated even on
      most modern embedded x86 CPUs which strive for FPU instruction set parity
      with desktop and server CPUs, so it's not like we can save much on smaller
      systems.
      
      This removes the last bootmem allocation from the FPU init path, allowing
      it to be called earlier in the boot sequence.
      Reviewed-by: default avatarBorislav Petkov <bp@alien8.de>
      Cc: Andy Lutomirski <luto@amacapital.net>
      Cc: Dave Hansen <dave.hansen@linux.intel.com>
      Cc: Fenghua Yu <fenghua.yu@intel.com>
      Cc: H. Peter Anvin <hpa@zytor.com>
      Cc: Linus Torvalds <torvalds@linux-foundation.org>
      Cc: Oleg Nesterov <oleg@redhat.com>
      Cc: Peter Zijlstra <peterz@infradead.org>
      Cc: Thomas Gleixner <tglx@linutronix.de>
      Signed-off-by: default avatarIngo Molnar <mingo@kernel.org>
      3e5e1267
    • Ingo Molnar's avatar
      x86/fpu: Make setup_init_fpu_buf() run-once explicitly · 26b1f5d0
      Ingo Molnar authored
      Remove the dependency on the init_xstate_buf == NULL check to
      implement once-per-bootup logic in eager_fpu_init(), by making
      setup_init_fpu_buf() run once per bootup explicitly.
      
      This is in preparation to make init_xstate_buf statically
      allocated.
      
      The various boot-once quirks in the FPU init code will be removed
      in a later cleanup stage.
      Reviewed-by: default avatarBorislav Petkov <bp@alien8.de>
      Cc: Andy Lutomirski <luto@amacapital.net>
      Cc: Dave Hansen <dave.hansen@linux.intel.com>
      Cc: Fenghua Yu <fenghua.yu@intel.com>
      Cc: H. Peter Anvin <hpa@zytor.com>
      Cc: Linus Torvalds <torvalds@linux-foundation.org>
      Cc: Oleg Nesterov <oleg@redhat.com>
      Cc: Peter Zijlstra <peterz@infradead.org>
      Cc: Thomas Gleixner <tglx@linutronix.de>
      Signed-off-by: default avatarIngo Molnar <mingo@kernel.org>
      26b1f5d0
    • Ingo Molnar's avatar
      x86/fpu: Remove xsave_init() bootmem allocations · 966ece61
      Ingo Molnar authored
      There's only 8 xstate bits at the moment, and it's not like we
      can support unknown bits - so put xstate_offsets[] and
      xstate_sizes[] into static allocation.
      
      This is in preparation to be able to call the FPU init code
      earlier, when there's no bootmem available yet.
      Reviewed-by: default avatarBorislav Petkov <bp@alien8.de>
      Cc: Andy Lutomirski <luto@amacapital.net>
      Cc: Dave Hansen <dave.hansen@linux.intel.com>
      Cc: Fenghua Yu <fenghua.yu@intel.com>
      Cc: H. Peter Anvin <hpa@zytor.com>
      Cc: Linus Torvalds <torvalds@linux-foundation.org>
      Cc: Oleg Nesterov <oleg@redhat.com>
      Cc: Peter Zijlstra <peterz@infradead.org>
      Cc: Thomas Gleixner <tglx@linutronix.de>
      Signed-off-by: default avatarIngo Molnar <mingo@kernel.org>
      966ece61
    • Ingo Molnar's avatar
      x86/fpu: Remove fpstate_xstate_init_size() boot quirk · 6a133207
      Ingo Molnar authored
      fpstate_xstate_init_size() is called in fpu__cpu_init(), which is
      run on every CPU, every time they are brought online.
      
      But we want to call fpstate_xstate_init_size() only once. Move it to
      fpu__detect(), which only runs once, on the boot CPU.
      
      Also clean up the flow of fpstate_xstate_init_size() a bit, by
      removing a 'return' from the middle of the function.
      Reviewed-by: default avatarBorislav Petkov <bp@alien8.de>
      Cc: Andy Lutomirski <luto@amacapital.net>
      Cc: Dave Hansen <dave.hansen@linux.intel.com>
      Cc: Fenghua Yu <fenghua.yu@intel.com>
      Cc: H. Peter Anvin <hpa@zytor.com>
      Cc: Linus Torvalds <torvalds@linux-foundation.org>
      Cc: Oleg Nesterov <oleg@redhat.com>
      Cc: Peter Zijlstra <peterz@infradead.org>
      Cc: Thomas Gleixner <tglx@linutronix.de>
      Signed-off-by: default avatarIngo Molnar <mingo@kernel.org>
      6a133207
    • Ingo Molnar's avatar
      x86/fpu: Rename __thread_fpu_end() to fpregs_deactivate() · 66af8e27
      Ingo Molnar authored
      Propagate the 'fpu->fpregs_active' naming to the high level function that
      clears it.
      Reviewed-by: default avatarBorislav Petkov <bp@alien8.de>
      Cc: Andy Lutomirski <luto@amacapital.net>
      Cc: Dave Hansen <dave.hansen@linux.intel.com>
      Cc: Fenghua Yu <fenghua.yu@intel.com>
      Cc: H. Peter Anvin <hpa@zytor.com>
      Cc: Linus Torvalds <torvalds@linux-foundation.org>
      Cc: Oleg Nesterov <oleg@redhat.com>
      Cc: Peter Zijlstra <peterz@infradead.org>
      Cc: Thomas Gleixner <tglx@linutronix.de>
      Signed-off-by: default avatarIngo Molnar <mingo@kernel.org>
      66af8e27
    • Ingo Molnar's avatar
      x86/fpu: Rename __thread_fpu_begin() to fpregs_activate() · 232f62cd
      Ingo Molnar authored
      Propagate the 'fpu->fpregs_active' naming to the high level
      function that sets it.
      Reviewed-by: default avatarBorislav Petkov <bp@alien8.de>
      Cc: Andy Lutomirski <luto@amacapital.net>
      Cc: Dave Hansen <dave.hansen@linux.intel.com>
      Cc: Fenghua Yu <fenghua.yu@intel.com>
      Cc: H. Peter Anvin <hpa@zytor.com>
      Cc: Linus Torvalds <torvalds@linux-foundation.org>
      Cc: Oleg Nesterov <oleg@redhat.com>
      Cc: Peter Zijlstra <peterz@infradead.org>
      Cc: Thomas Gleixner <tglx@linutronix.de>
      Signed-off-by: default avatarIngo Molnar <mingo@kernel.org>
      232f62cd
    • Ingo Molnar's avatar
      x86/fpu: Rename __thread_clear_has_fpu() to __fpregs_deactivate() · 723c58e4
      Ingo Molnar authored
      Propagate the 'fpu->fpregs_active' naming to the functions that
      clears it.
      Reviewed-by: default avatarBorislav Petkov <bp@alien8.de>
      Cc: Andy Lutomirski <luto@amacapital.net>
      Cc: Dave Hansen <dave.hansen@linux.intel.com>
      Cc: Fenghua Yu <fenghua.yu@intel.com>
      Cc: H. Peter Anvin <hpa@zytor.com>
      Cc: Linus Torvalds <torvalds@linux-foundation.org>
      Cc: Oleg Nesterov <oleg@redhat.com>
      Cc: Peter Zijlstra <peterz@infradead.org>
      Cc: Thomas Gleixner <tglx@linutronix.de>
      Signed-off-by: default avatarIngo Molnar <mingo@kernel.org>
      723c58e4
    • Ingo Molnar's avatar
      x86/fpu: Rename __thread_set_has_fpu() to __fpregs_activate() · dfaea4e6
      Ingo Molnar authored
      Propagate the 'fpu->fpregs_active' naming to the functions that
      sets it.
      Reviewed-by: default avatarBorislav Petkov <bp@alien8.de>
      Cc: Andy Lutomirski <luto@amacapital.net>
      Cc: Dave Hansen <dave.hansen@linux.intel.com>
      Cc: Fenghua Yu <fenghua.yu@intel.com>
      Cc: H. Peter Anvin <hpa@zytor.com>
      Cc: Linus Torvalds <torvalds@linux-foundation.org>
      Cc: Oleg Nesterov <oleg@redhat.com>
      Cc: Peter Zijlstra <peterz@infradead.org>
      Cc: Thomas Gleixner <tglx@linutronix.de>
      Signed-off-by: default avatarIngo Molnar <mingo@kernel.org>
      dfaea4e6
    • Ingo Molnar's avatar
      x86/fpu: Rename fpu->has_fpu to fpu->fpregs_active · d5cea9b0
      Ingo Molnar authored
      So the current code uses fpu->has_cpu to determine whether a given
      user FPU context is actively loaded into the FPU's registers [*] and
      that those registers represent the task's current FPU state.
      
      But this term is not unambiguous: especially the distinction between
      fpu->has_fpu, PF_USED_MATH and fpu_fpregs_owner_ctx is not clear.
      
      Increase clarity by unambigously signalling that it's about
      hardware registers being active right now, by renaming it to
      fpu->fpregs_active.
      
      ( In later patches we'll use more of the 'fpregs' naming, which will
        make it easier to grep for as well. )
      
      [*] There's the kernel_fpu_begin()/end() primitive that also
          activates FPU hw registers as well and uses them, without
          touching the fpu->fpregs_active flag.
      Reviewed-by: default avatarBorislav Petkov <bp@alien8.de>
      Cc: Andy Lutomirski <luto@amacapital.net>
      Cc: Dave Hansen <dave.hansen@linux.intel.com>
      Cc: Fenghua Yu <fenghua.yu@intel.com>
      Cc: H. Peter Anvin <hpa@zytor.com>
      Cc: Linus Torvalds <torvalds@linux-foundation.org>
      Cc: Oleg Nesterov <oleg@redhat.com>
      Cc: Peter Zijlstra <peterz@infradead.org>
      Cc: Thomas Gleixner <tglx@linutronix.de>
      Signed-off-by: default avatarIngo Molnar <mingo@kernel.org>
      d5cea9b0
    • Ingo Molnar's avatar
      x86/fpu: Improve the __sanitize_i387_state() documentation · 73a3aeb3
      Ingo Molnar authored
      Improve the comments and add new ones, as this code isn't very obvious.
      Reviewed-by: default avatarBorislav Petkov <bp@alien8.de>
      Cc: Andy Lutomirski <luto@amacapital.net>
      Cc: Dave Hansen <dave.hansen@linux.intel.com>
      Cc: Fenghua Yu <fenghua.yu@intel.com>
      Cc: H. Peter Anvin <hpa@zytor.com>
      Cc: Linus Torvalds <torvalds@linux-foundation.org>
      Cc: Oleg Nesterov <oleg@redhat.com>
      Cc: Peter Zijlstra <peterz@infradead.org>
      Cc: Thomas Gleixner <tglx@linutronix.de>
      Signed-off-by: default avatarIngo Molnar <mingo@kernel.org>
      73a3aeb3
    • Ingo Molnar's avatar
      x86/fpu: Explain the AVX register layout in the xsave area · e783e816
      Ingo Molnar authored
      The previous explanation was rather cryptic.
      
      Also transform "u32 [64]" to the more readable "u8[256]" form.
      
      No change in implementation.
      Reviewed-by: default avatarBorislav Petkov <bp@alien8.de>
      Cc: Andy Lutomirski <luto@amacapital.net>
      Cc: Dave Hansen <dave.hansen@linux.intel.com>
      Cc: Fenghua Yu <fenghua.yu@intel.com>
      Cc: H. Peter Anvin <hpa@zytor.com>
      Cc: Linus Torvalds <torvalds@linux-foundation.org>
      Cc: Oleg Nesterov <oleg@redhat.com>
      Cc: Peter Zijlstra <peterz@infradead.org>
      Cc: Thomas Gleixner <tglx@linutronix.de>
      Signed-off-by: default avatarIngo Molnar <mingo@kernel.org>
      e783e816
    • Ingo Molnar's avatar
      x86/fpu: Rename regset FPU register accessors · 678eaf60
      Ingo Molnar authored
      Rename regset accessors to prefix them with 'regset_', because we
      want to start using the 'fpregs_active' name elsewhere.
      Reviewed-by: default avatarBorislav Petkov <bp@alien8.de>
      Cc: Andy Lutomirski <luto@amacapital.net>
      Cc: Dave Hansen <dave.hansen@linux.intel.com>
      Cc: Fenghua Yu <fenghua.yu@intel.com>
      Cc: H. Peter Anvin <hpa@zytor.com>
      Cc: Linus Torvalds <torvalds@linux-foundation.org>
      Cc: Oleg Nesterov <oleg@redhat.com>
      Cc: Peter Zijlstra <peterz@infradead.org>
      Cc: Thomas Gleixner <tglx@linutronix.de>
      Signed-off-by: default avatarIngo Molnar <mingo@kernel.org>
      678eaf60
    • Ingo Molnar's avatar
      x86/fpu: Clean up and fix MXCSR handling · 91a8c2a5
      Ingo Molnar authored
      The code has the following problems:
      
       - it uses a single global 'fx_scratch' area that multiple CPUs could
         write into simultaneously, in theory.
      
       - it wastes 512 bytes of .data for something that is only rarely used.
      
      Fix this by moving the state buffer to the stack. Note that while
      this is 512 bytes, we don't ever call this function in very deep
      callchains, so its stack usage should not be a problem.
      
      Also add comments to explain the magic 0x0000ffbf default value.
      Reviewed-by: default avatarBorislav Petkov <bp@alien8.de>
      Cc: Andy Lutomirski <luto@amacapital.net>
      Cc: Dave Hansen <dave.hansen@linux.intel.com>
      Cc: Fenghua Yu <fenghua.yu@intel.com>
      Cc: H. Peter Anvin <hpa@zytor.com>
      Cc: Linus Torvalds <torvalds@linux-foundation.org>
      Cc: Oleg Nesterov <oleg@redhat.com>
      Cc: Peter Zijlstra <peterz@infradead.org>
      Cc: Thomas Gleixner <tglx@linutronix.de>
      Signed-off-by: default avatarIngo Molnar <mingo@kernel.org>
      91a8c2a5
    • Ingo Molnar's avatar
      x86/fpu: Rename xsave.header::xstate_bv to 'xfeatures' · 400e4b20
      Ingo Molnar authored
      'xsave.header::xstate_bv' is a misnomer - what does 'bv' stand for?
      
      It probably comes from the 'XGETBV' instruction name, but I could
      not find in the Intel documentation where that abbreviation comes
      from. It could mean 'bit vector' - or something else?
      
      But how about - instead of guessing about a weird name - we named
      the field in an obvious and descriptive way that tells us exactly
      what it does?
      
      So rename it to 'xfeatures', which is a bitmask of the
      xfeatures that are fpstate_active in that context structure.
      
      Eyesore like:
      
                 fpu->state->xsave.xsave_hdr.xstate_bv |= XSTATE_FP;
      
      is now much more readable:
      
                 fpu->state->xsave.header.xfeatures |= XSTATE_FP;
      
      Which form is not just infinitely more readable, but is also
      shorter as well.
      Reviewed-by: default avatarBorislav Petkov <bp@alien8.de>
      Cc: Andy Lutomirski <luto@amacapital.net>
      Cc: Dave Hansen <dave.hansen@linux.intel.com>
      Cc: Fenghua Yu <fenghua.yu@intel.com>
      Cc: H. Peter Anvin <hpa@zytor.com>
      Cc: Linus Torvalds <torvalds@linux-foundation.org>
      Cc: Oleg Nesterov <oleg@redhat.com>
      Cc: Peter Zijlstra <peterz@infradead.org>
      Cc: Thomas Gleixner <tglx@linutronix.de>
      Signed-off-by: default avatarIngo Molnar <mingo@kernel.org>
      400e4b20
    • Ingo Molnar's avatar
      x86/fpu: Rename 'xsave_hdr' to 'header' · 3a54450b
      Ingo Molnar authored
      Code like:
      
                 fpu->state->xsave.xsave_hdr.xstate_bv |= XSTATE_FP;
      
      is an eyesore, because not only is the words 'xsave' and 'state'
      are repeated twice times (!), but also because of the 'hdr' and 'bv'
      abbreviations that are pretty meaningless at a first glance.
      
      Start cleaning this up by renaming 'xsave_hdr' to 'header'.
      Reviewed-by: default avatarBorislav Petkov <bp@alien8.de>
      Cc: Andy Lutomirski <luto@amacapital.net>
      Cc: Dave Hansen <dave.hansen@linux.intel.com>
      Cc: Fenghua Yu <fenghua.yu@intel.com>
      Cc: H. Peter Anvin <hpa@zytor.com>
      Cc: Linus Torvalds <torvalds@linux-foundation.org>
      Cc: Oleg Nesterov <oleg@redhat.com>
      Cc: Peter Zijlstra <peterz@infradead.org>
      Cc: Thomas Gleixner <tglx@linutronix.de>
      Signed-off-by: default avatarIngo Molnar <mingo@kernel.org>
      3a54450b
    • Ingo Molnar's avatar
      x86/fpu: Clean up regset functions · 8dcea8db
      Ingo Molnar authored
      Clean up various regset handlers: use the 'fpu' pointer which
      is available in most cases.
      Reviewed-by: default avatarBorislav Petkov <bp@alien8.de>
      Cc: Andy Lutomirski <luto@amacapital.net>
      Cc: Dave Hansen <dave.hansen@linux.intel.com>
      Cc: Fenghua Yu <fenghua.yu@intel.com>
      Cc: H. Peter Anvin <hpa@zytor.com>
      Cc: Linus Torvalds <torvalds@linux-foundation.org>
      Cc: Oleg Nesterov <oleg@redhat.com>
      Cc: Peter Zijlstra <peterz@infradead.org>
      Cc: Thomas Gleixner <tglx@linutronix.de>
      Signed-off-by: default avatarIngo Molnar <mingo@kernel.org>
      8dcea8db
    • Ingo Molnar's avatar
      x86/fpu: Move XCR0 manipulation to the FPU code proper · 9254aaa0
      Ingo Molnar authored
      The suspend code accesses FPU state internals, add a helper for
      it and isolate it.
      Reviewed-by: default avatarBorislav Petkov <bp@alien8.de>
      Cc: Andy Lutomirski <luto@amacapital.net>
      Cc: Dave Hansen <dave.hansen@linux.intel.com>
      Cc: Fenghua Yu <fenghua.yu@intel.com>
      Cc: H. Peter Anvin <hpa@zytor.com>
      Cc: Linus Torvalds <torvalds@linux-foundation.org>
      Cc: Oleg Nesterov <oleg@redhat.com>
      Cc: Peter Zijlstra <peterz@infradead.org>
      Cc: Thomas Gleixner <tglx@linutronix.de>
      Signed-off-by: default avatarIngo Molnar <mingo@kernel.org>
      9254aaa0
    • Ingo Molnar's avatar
      x86/fpu: Rename 'xstate_features' to 'xfeatures_nr' · 84246fe4
      Ingo Molnar authored
      The name 'xstate_features' does not tell us whether it's a bitmap
      or any other value. That it's a count of features is only obvious
      if you read the code that calculates it.
      
      Rename it to the more descriptive 'xfeatures_nr' name.
      Reviewed-by: default avatarBorislav Petkov <bp@alien8.de>
      Cc: Andy Lutomirski <luto@amacapital.net>
      Cc: Dave Hansen <dave.hansen@linux.intel.com>
      Cc: Fenghua Yu <fenghua.yu@intel.com>
      Cc: H. Peter Anvin <hpa@zytor.com>
      Cc: Linus Torvalds <torvalds@linux-foundation.org>
      Cc: Oleg Nesterov <oleg@redhat.com>
      Cc: Peter Zijlstra <peterz@infradead.org>
      Cc: Thomas Gleixner <tglx@linutronix.de>
      Signed-off-by: default avatarIngo Molnar <mingo@kernel.org>
      84246fe4
    • Ingo Molnar's avatar
      x86/fpu: Rename 'pcntxt_mask' to 'xfeatures_mask' · 614df7fb
      Ingo Molnar authored
      So the 'pcntxt_mask' is a misnomer, it's essentially meaningless to anyone
      who doesn't know what it does exactly.
      
      Name it more descriptively as 'xfeatures_mask'.
      Reviewed-by: default avatarBorislav Petkov <bp@alien8.de>
      Cc: Andy Lutomirski <luto@amacapital.net>
      Cc: Dave Hansen <dave.hansen@linux.intel.com>
      Cc: Fenghua Yu <fenghua.yu@intel.com>
      Cc: H. Peter Anvin <hpa@zytor.com>
      Cc: Linus Torvalds <torvalds@linux-foundation.org>
      Cc: Oleg Nesterov <oleg@redhat.com>
      Cc: Peter Zijlstra <peterz@infradead.org>
      Cc: Thomas Gleixner <tglx@linutronix.de>
      Signed-off-by: default avatarIngo Molnar <mingo@kernel.org>
      614df7fb