The framework should bring the device down before it calls the driver.
Signed-off-by: Mark Brown <broonie@opensource.wolfsonmicro.com>