Commit 36fafe87 authored by Omer Shpigelman's avatar Omer Shpigelman Committed by Oded Gabbay

habanalabs: don't allow hard reset with open processes

When the MMU is heavily used by the engines, unmapping might take a lot of
time due to a full MMU cache invalidation done as part of the unmap flow.
Hence we might not be able to kill all open processes before going to hard
reset the device, as it involves unmapping of all user memory.
In case of a failure in killing all open processes, we should stop the
hard reset flow as it might lead to a kernel crash - one thread (killing
of a process) is updating MMU structures that other thread (hard reset) is
freeing.
Stopping a hard reset flow leaves the device as nonoperational and the
user can then initiate a hard reset via sysfs to reinitialize the device.
Signed-off-by: default avatarOmer Shpigelman <oshpigelman@habana.ai>
Reviewed-by: default avatarOded Gabbay <oded.gabbay@gmail.com>
Signed-off-by: default avatarOded Gabbay <oded.gabbay@gmail.com>
parent 66446820
...@@ -726,7 +726,7 @@ int hl_device_resume(struct hl_device *hdev) ...@@ -726,7 +726,7 @@ int hl_device_resume(struct hl_device *hdev)
return rc; return rc;
} }
static void device_kill_open_processes(struct hl_device *hdev) static int device_kill_open_processes(struct hl_device *hdev)
{ {
u16 pending_total, pending_cnt; u16 pending_total, pending_cnt;
struct hl_fpriv *hpriv; struct hl_fpriv *hpriv;
...@@ -779,9 +779,7 @@ static void device_kill_open_processes(struct hl_device *hdev) ...@@ -779,9 +779,7 @@ static void device_kill_open_processes(struct hl_device *hdev)
ssleep(1); ssleep(1);
} }
if (!list_empty(&hdev->fpriv_list)) return list_empty(&hdev->fpriv_list) ? 0 : -EBUSY;
dev_crit(hdev->dev,
"Going to hard reset with open user contexts\n");
} }
static void device_hard_reset_pending(struct work_struct *work) static void device_hard_reset_pending(struct work_struct *work)
...@@ -908,7 +906,12 @@ int hl_device_reset(struct hl_device *hdev, bool hard_reset, ...@@ -908,7 +906,12 @@ int hl_device_reset(struct hl_device *hdev, bool hard_reset,
* process can't really exit until all its CSs are done, which * process can't really exit until all its CSs are done, which
* is what we do in cs rollback * is what we do in cs rollback
*/ */
device_kill_open_processes(hdev); rc = device_kill_open_processes(hdev);
if (rc) {
dev_crit(hdev->dev,
"Failed to kill all open processes, stopping hard reset\n");
goto out_err;
}
/* Flush the Event queue workers to make sure no other thread is /* Flush the Event queue workers to make sure no other thread is
* reading or writing to registers during the reset * reading or writing to registers during the reset
...@@ -1391,7 +1394,9 @@ void hl_device_fini(struct hl_device *hdev) ...@@ -1391,7 +1394,9 @@ void hl_device_fini(struct hl_device *hdev)
* can't really exit until all its CSs are done, which is what we * can't really exit until all its CSs are done, which is what we
* do in cs rollback * do in cs rollback
*/ */
device_kill_open_processes(hdev); rc = device_kill_open_processes(hdev);
if (rc)
dev_crit(hdev->dev, "Failed to kill all open processes\n");
hl_cb_pool_fini(hdev); hl_cb_pool_fini(hdev);
......
Markdown is supported
0%
or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment