Only call driver disable when device isn't already disabled, which also
handles the driver not loaded case.
Signed-off-by: Jani Nikula <ext-jani.1.nikula@nokia.com>
Signed-off-by: Tomi Valkeinen <tomi.valkeinen@nokia.com>