The dev_set_drvdata() is no longer needed after we used a wrapper
struct to get the data in auxiliary driver.

Cc: Xingyu Wu <xingyu.wu@starfivetech.com>
Fixes: d1aae066 ("clk: starfive: Avoid casting iomem pointers")
Signed-off-by: Hal Feng <hal.feng@starfivetech.com>
Link: https://lore.kernel.org/r/20230417074115.30786-3-hal.feng@starfivetech.comSigned-off-by: Stephen Boyd <sboyd@kernel.org>