That fixes an opps when driver is repeatedly loaded and unloaded in
a tight loop.
Tested-by: Roland Kletzing <devzero@web.de>
Signed-off-by: Dmitry Torokhov <dtor@mail.ru>