Make the code cleaner and avoid call clk_disable_unprepare()
Signed-off-by: Wu Bo <bo.wu@vivo.com>
Signed-off-by: Shawn Guo <shawnguo@kernel.org>