Only imx8m_clk_hw_composite_core needs to set this flag.

Fixes: a60fe746 ("clk: imx: Rework all imx_clk_hw_composite wrappers")
Signed-off-by: Alexander Stein <alexander.stein@ew.tq-group.com>
Link: https://lore.kernel.org/r/20211103123947.3222443-1-alexander.stein@ew.tq-group.comReviewed-by: Abel Vesa <abel.vesa@nxp.com>
Tested-by: Adam Ford <aford173@gmail.com> #imx8mm-beacon
Signed-off-by: Stephen Boyd <sboyd@kernel.org>