Without this change the KSZ9477-EVB board hangs just after passing
execution flow from u-boot to Linux kernel.

This code has been copied from at91-sama5d3_xplained.dts.

Test setup: Linux 6.5-rc1
Config:     arch/arm/configs/sama5_defconfig
Toolchain:  gcc-linaro-7.3.1-2018.05-x86_64_arm-linux-gnueabi
Signed-off-by: Lukasz Majewski <lukma@denx.de>
Link: https://lore.kernel.org/r/20230712152111.3756211-1-lukma@denx.deSigned-off-by: Claudiu Beznea <claudiu.beznea@tuxon.dev>