This removes a possible race condition and crash and allows for further
improvement of the driver.
Signed-off-by: Alexandre Belloni <alexandre.belloni@free-electrons.com>