Commit df61b366af26 ('pinctrl: core: Use delayed work for hogs')
deferred part of the registration for pinctrl devices if the pinctrl
device has hogs. This introduced a window where if the pinctrl device
with hogs was sucessfully registered, but then unregistered again
(which could be caused by parent device being probe deferred) before
the delayed work has chanced to run, then this will cause a kernel
panic to occur because:

1. The 'pctldev->p' has not yet been initialised and when unregistering
   the pinctrl device we only check to see if it is an error value, but
   now it could also be NULL.
2. The pinctrl device may not have been added to the 'pinctrldev_list'
   list and we don't check to see if it was added before removing.

Fix up the above by checking to see if the 'pctldev->p' pointer is an
error value or NULL before putting the pinctrl device and verifying
that the pinctrl device is present in 'pinctrldev_list' before removing.

Fixes: df61b366af26 ('pinctrl: core: Use delayed work for hogs')
Signed-off-by: Jon Hunter <jonathanh@nvidia.com>
Signed-off-by: Linus Walleij <linus.walleij@linaro.org>