This reverts commit 6c7e660a.

The commit causes breakage on several older PXA machines.
Reported-by: Mike Dunn <mikedunn@newsguy.com>
Acked-by: Haojian Zhuang <haojian.zhuang@linaro.org>
Signed-off-by: Linus Walleij <linus.walleij@linaro.org>