Boards usually have an external pull-up on the card-detect signal, so
there's no need to add another one.

This also removes a DTC warning.
Signed-off-by: Maxime Ripard <maxime.ripard@bootlin.com>
Acked-by: Chen-Yu Tsai <wens@csie.org>