Some controllers allow a much lower frequency than 400kHz.
Keep the minimum frequency within sensible limits.
Signed-off-by: Sascha Hauer <s.hauer@pengutronix.de>