1. 09 May, 2018 38 commits
  2. 01 May, 2018 2 commits
    • Greg Kroah-Hartman's avatar
      Linux 4.14.39 · 7d6240f0
      Greg Kroah-Hartman authored
      7d6240f0
    • Michael Neuling's avatar
      powerpc/eeh: Fix race with driver un/bind · 7fddff51
      Michael Neuling authored
      commit f0295e04 upstream.
      
      The current EEH callbacks can race with a driver unbind. This can
      result in a backtraces like this:
      
        EEH: Frozen PHB#0-PE#1fc detected
        EEH: PE location: S000009, PHB location: N/A
        CPU: 2 PID: 2312 Comm: kworker/u258:3 Not tainted 4.15.6-openpower1 #2
        Workqueue: nvme-wq nvme_reset_work [nvme]
        Call Trace:
          dump_stack+0x9c/0xd0 (unreliable)
          eeh_dev_check_failure+0x420/0x470
          eeh_check_failure+0xa0/0xa4
          nvme_reset_work+0x138/0x1414 [nvme]
          process_one_work+0x1ec/0x328
          worker_thread+0x2e4/0x3a8
          kthread+0x14c/0x154
          ret_from_kernel_thread+0x5c/0xc8
        nvme nvme1: Removing after probe failure status: -19
        <snip>
        cpu 0x23: Vector: 300 (Data Access) at [c000000ff50f3800]
            pc: c0080000089a0eb0: nvme_error_detected+0x4c/0x90 [nvme]
            lr: c000000000026564: eeh_report_error+0xe0/0x110
            sp: c000000ff50f3a80
           msr: 9000000000009033
           dar: 400
         dsisr: 40000000
          current = 0xc000000ff507c000
          paca    = 0xc00000000fdc9d80   softe: 0        irq_happened: 0x01
            pid   = 782, comm = eehd
        Linux version 4.15.6-openpower1 (smc@smc-desktop) (gcc version 6.4.0 (Buildroot 2017.11.2-00008-g4b6188e)) #2 SM                                             P Tue Feb 27 12:33:27 PST 2018
        enter ? for help
          eeh_report_error+0xe0/0x110
          eeh_pe_dev_traverse+0xc0/0xdc
          eeh_handle_normal_event+0x184/0x4c4
          eeh_handle_event+0x30/0x288
          eeh_event_handler+0x124/0x170
          kthread+0x14c/0x154
          ret_from_kernel_thread+0x5c/0xc8
      
      The first part is an EEH (on boot), the second half is the resulting
      crash. nvme probe starts the nvme_reset_work() worker thread. This
      worker thread starts touching the device which see a device error
      (EEH) and hence queues up an event in the powerpc EEH worker
      thread. nvme_reset_work() then continues and runs
      nvme_remove_dead_ctrl_work() which results in unbinding the driver
      from the device and hence releases all resources. At the same time,
      the EEH worker thread starts doing the EEH .error_detected() driver
      callback, which no longer works since the resources have been freed.
      
      This fixes the problem in the same way the generic PCIe AER code (in
      drivers/pci/pcie/aer/aerdrv_core.c) does. It makes the EEH code hold
      the device_lock() while performing the driver EEH callbacks and
      associated code. This ensures either the callbacks are no longer
      register, or if they are registered the driver will not be removed
      from underneath us.
      
      This has been broken forever. The EEH call backs were first introduced
      in 2005 (in 77bd7415) but it's not clear if a lock was needed back
      then.
      
      Fixes: 77bd7415 ("[PATCH] powerpc: PCI Error Recovery: PPC64 core recovery routines")
      Cc: stable@vger.kernel.org # v2.6.16+
      Signed-off-by: default avatarMichael Neuling <mikey@neuling.org>
      Reviewed-by: default avatarBenjamin Herrenschmidt <benh@kernel.crashing.org>
      Signed-off-by: default avatarMichael Ellerman <mpe@ellerman.id.au>
      Signed-off-by: default avatarGreg Kroah-Hartman <gregkh@linuxfoundation.org>
      7fddff51