Move pin config to appropriate places and use it only if needed.
Signed-off-by: Marek Vasut <marek.vasut@gmail.com>
Acked-By: Jonathan McDowell <noodles@earth.li>
Signed-off-by: Eric Miao <eric.y.miao@gmail.com>