Normal GPIO usage does not need an additional pinmix setting. Exclusive
usage of the pin will be guaranteed by the driver.

Drop the extra pinmux settings.
Signed-off-by: Chen-Yu Tsai <wens@csie.org>
Signed-off-by: Maxime Ripard <maxime.ripard@bootlin.com>