1. 03 Jun, 2009 18 commits
    • Huang Ying's avatar
      x86, mce: export MCE severities coverage via debugfs · 4611a6fa
      Huang Ying authored
      The MCE severity judgement code is data-driven, so code coverage tools
      such as gcov can not be used for measuring coverage. Instead a dedicated
      coverage mechanism is implemented.  The kernel keeps track of rules
      executed and reports them in debugfs.
      
      This is useful for increasing coverage of the mce-test testsuite.
      
      Right now it's unconditionally enabled because it's very little code.
      Signed-off-by: default avatarHuang Ying <ying.huang@intel.com>
      Signed-off-by: default avatarAndi Kleen <ak@linux.intel.com>
      Signed-off-by: default avatarHidetoshi Seto <seto.hidetoshi@jp.fujitsu.com>
      Signed-off-by: default avatarH. Peter Anvin <hpa@zytor.com>
      4611a6fa
    • Andi Kleen's avatar
      x86, mce: implement new status bits · ed7290d0
      Andi Kleen authored
      The x86 architecture recently added some new machine check status bits:
      S(ignalled) and AR (Action-Required). Signalled allows to check
      if a specific event caused an exception or was just logged through CMCI.
      AR allows the kernel to decide if an event needs immediate action
      or can be delayed or ignored.
      
      Implement support for these new status bits. mce_severity() uses
      the new bits to grade the machine check correctly and decide what
      to do. The exception handler uses AR to decide to kill or not.
      The S bit is used to separate events between the poll/CMCI handler
      and the exception handler.
      
      Classical UC always leads to panic. That was true before anyways
      because the existing CPUs always passed a PCC with it.
      
      Also corrects the rules whether to kill in user or kernel context
      and how to handle missing RIPV.
      
      The machine check handler largely uses the mce-severity grading
      engine now instead of making its own decisions. This means the logic
      is centralized in one place.  This is useful because it has to be
      evaluated multiple times.
      
      v2: Some rule fixes; Add AO events
      Fix RIPV, RIPV|EIPV order (Ying Huang)
      Fix UCNA with AR=1 message (Ying Huang)
      Add comment about panicing in m_c_p.
      Signed-off-by: default avatarAndi Kleen <ak@linux.intel.com>
      Signed-off-by: default avatarHidetoshi Seto <seto.hidetoshi@jp.fujitsu.com>
      Signed-off-by: default avatarH. Peter Anvin <hpa@zytor.com>
      ed7290d0
    • Andi Kleen's avatar
      x86, mce: print header/footer only once for multiple MCEs · 86503560
      Andi Kleen authored
      When multiple MCEs are printed print the "HARDWARE ERROR" header
      and "This is not a software error" footer only once. This
      makes the output much more compact with many CPUs.
      Signed-off-by: default avatarAndi Kleen <ak@linux.intel.com>
      Signed-off-by: default avatarHidetoshi Seto <seto.hidetoshi@jp.fujitsu.com>
      Signed-off-by: default avatarH. Peter Anvin <hpa@zytor.com>
      86503560
    • Andi Kleen's avatar
      x86, mce: default to panic timeout for machine checks · 29b0f591
      Andi Kleen authored
      Fatal machine checks can be logged to disk after boot, but only if
      the system did a warm reboot. That's unfortunately difficult with the
      default panic behaviour, which waits forever and the admin has to
      press the power button because modern systems usually miss a reset button.
      This clears the machine checks in the registers and make
      it impossible to log them.
      
      This patch changes the default for machine check panic to always
      reboot after 30s. Then the mce can be successfully logged after
      reboot.
      
      I believe this will improve machine check experience for any
      system running the X server.
      
      This is dependent on successfull boot logging of MCEs. This currently
      only works on Intel systems, on AMD there are quite a lot of systems
      around which leave junk in the machine check registers after boot,
      so it's disabled here. These systems will continue to default
      to endless waiting panic.
      
      v2: Only force panic timeout when it's shorter (H.Seto)
      v3: Only force timeout when there is no timeout
      (based on comment H.Seto)
      
      [ Fix changelog - HS ]
      Signed-off-by: default avatarAndi Kleen <ak@linux.intel.com>
      Signed-off-by: default avatarHidetoshi Seto <seto.hidetoshi@jp.fujitsu.com>
      Signed-off-by: default avatarH. Peter Anvin <hpa@zytor.com>
      29b0f591
    • Huang Ying's avatar
      x86, mce: improve mce_get_rip · 1b2797dc
      Huang Ying authored
      Assume IP on the stack is valid when either EIPV or RIPV are set.
      This influences whether the machine check exception handler decides
      to return or panic.
      
      This fixes a test case in the mce-test suite and is more compliant
      to the specification.
      
      This currently only makes a difference in a artificial testing
      scenario with the mce-test test suite.
      
      Also in addition do not force the EIPV to be valid with the exact
      register MSRs, and keep in trust the CS value on stack even if MSR
      is available.
      
      [AK: combination of patches from Huang Ying and Hidetoshi Seto, with
      new description by me]
      [add some description, no code changed - HS]
      Signed-off-by: default avatarHuang Ying <ying.huang@intel.com>
      Signed-off-by: default avatarAndi Kleen <ak@linux.intel.com>
      Signed-off-by: default avatarHidetoshi Seto <seto.hidetoshi@jp.fujitsu.com>
      Signed-off-by: default avatarH. Peter Anvin <hpa@zytor.com>
      1b2797dc
    • Andi Kleen's avatar
      x86, mce: make non Monarch panic message "Fatal machine check" too · ac960375
      Andi Kleen authored
      ... instead of "Machine check". This is for consistency with the Monarch
      panic message.
      
      Based on a report from Ying Huang.
      
      v2: But add a descriptive postfix so that the test suite can distingush.
      Signed-off-by: default avatarAndi Kleen <ak@linux.intel.com>
      Signed-off-by: default avatarHidetoshi Seto <seto.hidetoshi@jp.fujitsu.com>
      Signed-off-by: default avatarH. Peter Anvin <hpa@zytor.com>
      ac960375
    • Andi Kleen's avatar
      x86, mce: switch x86 machine check handler to Monarch election. · 3c079792
      Andi Kleen authored
      On Intel platforms machine check exceptions are always broadcast to
      all CPUs.  This patch makes the machine check handler synchronize all
      these machine checks, elect a Monarch to handle the event and collect
      the worst event from all CPUs and then process it first.
      
      This has some advantages:
      
      - When there is a truly data corrupting error the system panics as
        quickly as possible. This improves containment of corrupted
        data and makes sure the corrupted data never hits stable storage.
      
      - The panics are synchronized and do not reenter the panic code
        on multiple CPUs (which currently does not handle this well).
      
      - All the errors are reported. Currently it often happens that
        another CPU happens to do the panic first, but reports useless
        information (empty machine check) because the real error
        happened on another CPU which came in later.
        This is a big advantage on Nehalem where the 8 threads per CPU
        lead to often the wrong CPU winning the race and dumping
        useless information on a machine check.  The problem also occurs
        in a less severe form on older CPUs.
      
      - The system can detect when no CPUs detected a machine check
        and shut down the system.  This can happen when one CPU is so
        badly hung that that it cannot process a machine check anymore
        or when some external agent wants to stop the system by
        asserting the machine check pin.  This follows Intel hardware
        recommendations.
      
      - This matches the recommended error model by the CPU designers.
      
      - The events can be output in true severity order
      
      - When a panic happens on another CPU it makes sure to be actually
        be able to process the stop IPI by enabling interrupts.
      
      The code is extremly careful to handle timeouts while waiting
      for other CPUs. It can't rely on the normal timing mechanisms
      (jiffies, ktime_get) because of its asynchronous/lockless nature,
      so it uses own timeouts using ndelay() and a "SPINUNIT"
      
      The timeout is configurable. By default it waits for upto one
      second for the other CPUs.  This can be also disabled.
      
      From some informal testing AMD systems do not see to broadcast
      machine checks, so right now it's always disabled by default on
      non Intel CPUs or also on very old Intel systems.
      
      Includes fixes from Ying Huang
      Fixed a "ecception" in a comment (H.Seto)
      Moved global_nwo reset later based on suggestion from H.Seto
      v2: Avoid duplicate messages
      
      [ Impact: feature, fixes long standing problems. ]
      Signed-off-by: default avatarAndi Kleen <ak@linux.intel.com>
      Signed-off-by: default avatarHidetoshi Seto <seto.hidetoshi@jp.fujitsu.com>
      Signed-off-by: default avatarH. Peter Anvin <hpa@zytor.com>
      3c079792
    • Andi Kleen's avatar
      x86, mce: implement panic synchronization · f94b61c2
      Andi Kleen authored
      In some circumstances multiple CPUs can enter mce_panic() in parallel.
      This gives quite confused output because they will all dump the same
      machine check buffer.
      
      The other problem is that they would all panic in parallel, but not
      process each other's shutdown IPIs because interrupts are disabled.
      
      Detect this situation early on in mce_panic(). On the first CPU
      entering will do the panic, the others will just wait to be killed.
      
      For paranoia reasons in case the other CPU dies during the MCE I added
      a 5 seconds timeout. If it expires each CPU will panic on its own again.
      Signed-off-by: default avatarAndi Kleen <ak@linux.intel.com>
      Signed-off-by: default avatarHidetoshi Seto <seto.hidetoshi@jp.fujitsu.com>
      Signed-off-by: default avatarH. Peter Anvin <hpa@zytor.com>
      f94b61c2
    • Andi Kleen's avatar
      x86, mce: implement bootstrapping for machine check wakeups · ccc3c319
      Andi Kleen authored
      Machine checks support waking up the mcelog daemon quickly.
      
      The original wake up code for this was pretty ugly, relying on
      a idle notifier and a special process flag. The reason it did
      it this way is that the machine check handler is not subject
      to normal interrupt locking rules so it's not safe
      to call wake_up().  Instead it set a process flag
      and then either did the wakeup in the syscall return
      or in the idle notifier.
      
      This patch adds a new "bootstraping" method as replacement.
      
      The idea is that the handler checks if it's in a state where
      it is unsafe to call wake_up(). If it's safe it calls it directly.
      When it's not safe -- that is it interrupted in a critical
      section with interrupts disables -- it uses a new "self IPI" to trigger
      an IPI to its own CPU. This can be done safely because IPI
      triggers are atomic with some care. The IPI is raised
      once the interrupts are reenabled and can then safely call
      wake_up().
      
      When APICs are disabled the event is just queued and will be picked up
      eventually by the next polling timer. I think that's a reasonable
      compromise, since it should only happen quite rarely.
      
      Contains fixes from Ying Huang.
      
      [ solve conflict on irqinit, make it work on 32bit (entry_arch.h) - HS ]
      Signed-off-by: default avatarAndi Kleen <ak@linux.intel.com>
      Signed-off-by: default avatarHidetoshi Seto <seto.hidetoshi@jp.fujitsu.com>
      Signed-off-by: default avatarH. Peter Anvin <hpa@zytor.com>
      ccc3c319
    • Andi Kleen's avatar
      x86, mce: check early in exception handler if panic is needed · bd19a5e6
      Andi Kleen authored
      The exception handler should behave differently if the exception is
      fatal versus one that can be returned from.  In the first case it should
      never clear any registers because these need to be preserved
      for logging after the next boot. Otherwise it should clear them
      on each CPU step by step so that other CPUs sharing the same bank don't
      see duplicate events. Otherwise we risk reporting events multiple
      times on any CPUs which have shared machine check banks, which
      is a common problem on Intel Nehalem which has both SMT (two
      CPU threads sharing banks) and shared machine check banks in the uncore.
      
      Determine early in a special pass if any event requires a panic.
      This uses the mce_severity() function added earlier.
      
      This is needed for the next patch.
      
      Also fixes a problem together with an earlier patch
      that corrected events weren't logged on a fatal MCE.
      
      [ Impact: Feature ]
      Signed-off-by: default avatarAndi Kleen <ak@linux.intel.com>
      Signed-off-by: default avatarHidetoshi Seto <seto.hidetoshi@jp.fujitsu.com>
      Signed-off-by: default avatarH. Peter Anvin <hpa@zytor.com>
      bd19a5e6
    • Andi Kleen's avatar
      x86, mce: add table driven machine check grading · 817f32d0
      Andi Kleen authored
      The machine check grading (as in deciding what should be done for a given
      register value) has to be done multiple times soon and it's also getting
      more complicated.
      So it makes sense to consolidate it into a single function. To get smaller
      and more straight forward and possibly more extensible code I opted towards
      a new table driven method. The various rules are put into a table
      when is then executed by a very simple interpreter.
      
      The grading engine is in a new file mce-severity.c. I also added a private
      include file mce-internal.h, because mce.h is already a bit too cluttered.
      
      This is dead code right now, but will be used in followon patches.
      Signed-off-by: default avatarAndi Kleen <ak@linux.intel.com>
      Signed-off-by: default avatarHidetoshi Seto <seto.hidetoshi@jp.fujitsu.com>
      Signed-off-by: default avatarH. Peter Anvin <hpa@zytor.com>
      817f32d0
    • Andi Kleen's avatar
      x86, mce: remove TSC print heuristic · a0189c70
      Andi Kleen authored
      Previously mce_panic used a simple heuristic to avoid printing
      old so far unreported machine check events on a mce panic. This worked
      by comparing the TSC value at the start of the machine check handler
      with the event time stamp and only printing newer ones.
      
      This has a couple of issues, in particular on systems where the TSC
      is not fully synchronized between CPUs it could lose events or print
      old ones.
      
      It is also problematic with full system synchronization as it is
      added by the next patch.
      
      Remove the TSC heuristic and instead replace it with a simple heuristic
      to print corrected errors first and after that uncorrected errors
      and finally the worst machine check as determined by the machine
      check handler.
      
      This simplifies the code because there is no need to pass the
      original TSC value around.
      
      Contains fixes from Ying Huang
      
      [ Impact: bug fix, cleanup ]
      Signed-off-by: default avatarAndi Kleen <ak@linux.intel.com>
      Signed-off-by: default avatarHidetoshi Seto <seto.hidetoshi@jp.fujitsu.com>
      Cc: Ying Huang <ying.huang@intel.com>
      Signed-off-by: default avatarH. Peter Anvin <hpa@zytor.com>
      a0189c70
    • Andi Kleen's avatar
      x86, mce: log corrected errors when panicing · de8a84d8
      Andi Kleen authored
      Normally the machine check handler ignores corrected errors and leaves
      them to machine_check_poll(). But when panicing mcp won't run, so
      log all errors.
      
      Note: this can still miss some cases until the "early no way out"
      patch later is applied too.
      Signed-off-by: default avatarAndi Kleen <ak@linux.intel.com>
      Signed-off-by: default avatarHidetoshi Seto <seto.hidetoshi@jp.fujitsu.com>
      Signed-off-by: default avatarH. Peter Anvin <hpa@zytor.com>
      de8a84d8
    • Andi Kleen's avatar
      x86, mce: extend struct mce user interface with more information. · 8ee08347
      Andi Kleen authored
      Experience has shown that struct mce which is used to pass an machine
      check to the user space daemon currently a few limitations.  Also some
      data which is useful to print at panic level is also missing.
      
      This patch addresses most of them. The same information is also
      printed out together with mce panic.
      
      struct mce can be painlessly extended in a compatible way, the mcelog
      user space code just ignores additional fields with a warning.
      
      - It doesn't provide a wall time timestamp. There have been a few
        complaints about that. Fix that by adding a 64bit time_t
      
      - It doesn't provide the exact CPU identification. This makes
        it awkward for mcelog to decode the event correctly, especially
        when there are variations in the supported MCE codes on different
        CPU models or when mcelog is running on a different host after a panic.
        Previously the administrator had to specify the correct CPU
        when mcelog ran on a different host, but with the more variation
        in machine checks now it's better to auto detect that.
        It's also useful for more detailed analysis of CPU events.
        Pass CPUID 1.EAX and the cpu vendor (as encoded in processor.h) instead.
      
      - Socket ID and initial APIC ID are useful to report because they
        allow to identify the failing CPU in some (not all) cases.
        This is also especially useful for the panic situation.
        This addresses one of the complaints from Thomas Gleixner earlier.
      
      - The MCG capabilities MSR needs to be reported for some advanced
        error processing in mcelog
      Signed-off-by: default avatarAndi Kleen <ak@linux.intel.com>
      Signed-off-by: default avatarHidetoshi Seto <seto.hidetoshi@jp.fujitsu.com>
      Signed-off-by: default avatarH. Peter Anvin <hpa@zytor.com>
      8ee08347
    • Andi Kleen's avatar
      x86, mce: support more than 256 CPUs in struct mce · d620c67f
      Andi Kleen authored
      The old struct mce had a limitation to 256 CPUs. But x86 Linux supports
      more than that now with x2apic. Add a new field extcpu to report the
      extended number.
      Signed-off-by: default avatarAndi Kleen <ak@linux.intel.com>
      Signed-off-by: default avatarHidetoshi Seto <seto.hidetoshi@jp.fujitsu.com>
      Signed-off-by: default avatarH. Peter Anvin <hpa@zytor.com>
      d620c67f
    • Andi Kleen's avatar
      x86, mce: store record length into memory struct mce anchor · f6fb0ac0
      Andi Kleen authored
      This makes it easier for tools who want to extract the mcelog out of
      crash images or memory dumps to adapt to changing struct mce size.
      The length field replaces padding, so it's fully compatible.
      Signed-off-by: default avatarAndi Kleen <ak@linux.intel.com>
      Signed-off-by: default avatarHidetoshi Seto <seto.hidetoshi@jp.fujitsu.com>
      Signed-off-by: default avatarH. Peter Anvin <hpa@zytor.com>
      f6fb0ac0
    • Andi Kleen's avatar
      x86, mce: add MCE poll count to /proc/interrupts · ca84f696
      Andi Kleen authored
      Keep a count of the machine check polls (or CMCI events) in
      /proc/interrupts.
      
      Andi needs this for debugging, but it's also useful in general
      to see what's going in by the kernel.
      Signed-off-by: default avatarAndi Kleen <ak@linux.intel.com>
      Signed-off-by: default avatarHidetoshi Seto <seto.hidetoshi@jp.fujitsu.com>
      Signed-off-by: default avatarH. Peter Anvin <hpa@zytor.com>
      ca84f696
    • Andi Kleen's avatar
      x86, mce: add machine check exception count in /proc/interrupts · 01ca79f1
      Andi Kleen authored
      Useful for debugging, but it's also good general policy
      to have a counter for all special interrupts there. This makes it easier
      to diagnose where a CPU is spending its time.
      
      [ Impact: feature, debugging tool ]
      Signed-off-by: default avatarAndi Kleen <ak@linux.intel.com>
      Signed-off-by: default avatarHidetoshi Seto <seto.hidetoshi@jp.fujitsu.com>
      Signed-off-by: default avatarH. Peter Anvin <hpa@zytor.com>
      01ca79f1
  2. 01 Jun, 2009 13 commits
  3. 30 May, 2009 9 commits