Otherwise we may try to power down multiple times when the using
idle bias off and the driver is removed.
Signed-off-by: Mark Brown <broonie@opensource.wolfsonmicro.com>
Acked-by: Liam Girdwood <lrg@slimlogic.co.uk>