Allows removal of clk_prepare_enable to simplify the code slightly.

Tested on a TP-LINK Archer C7v2.
Signed-off-by: Rosen Penev <rosenp@gmail.com>
Link: https://patch.msgid.link/20240813170516.7301-2-rosenp@gmail.comSigned-off-by: Jakub Kicinski <kuba@kernel.org>