This reverts commit d0511b54.

After some time it was noticed that the Tegra186 among others
were experiencing problems when making this into a module.
Signed-off-by: Linus Walleij <linus.walleij@linaro.org>