This simplifies the error handling and allows us to drop the whole
chv_pinctrl_remove() function.
Signed-off-by: Mika Westerberg <mika.westerberg@linux.intel.com>
Signed-off-by: Linus Walleij <linus.walleij@linaro.org>