Turn on sparse IRQs and patch-physical-to-virtual for IXP4xx
as this is required for multiplatform. Drop the PHYS_OFFSET as
we are now using P2V.

Tested and works like a charm on my systems.
Signed-off-by: Linus Walleij <linus.walleij@linaro.org>
Link: https://lore.kernel.org/r/20220211223238.648934-14-linus.walleij@linaro.orgSigned-off-by: Linus Walleij <linus.walleij@linaro.org>