This reverts commit b39ac542.

The issue was fixed by upstream commit 5d797111 ("clk:
tegra: Add quirk for getting CDEV1/2 clocks on Tegra20").
Signed-off-by: Sasha Levin <sashal@kernel.org>