gpios requested with invalid numbers, or gpios requested from userspace via sysfs
should not try to be deferred on failure.

Cc: stable@kernel.org
Signed-off-by: Mathias Nyman <mathias.nyman@linux.intel.com>
Signed-off-by: Linus Walleij <linus.walleij@linaro.org>