Ensures that we don't interact badly with the power management framework,
especially in the cases where we're doing deferred work or we're using a
direct GPIO for these signals.
Signed-off-by: Mark Brown <broonie@opensource.wolfsonmicro.com>