Ensure that the hardware is disabled prior to tearing down the modeset
support.
Signed-off-by: Russell King <rmk+kernel@armlinux.org.uk>