This driver calls of_mm_gpiochip_add() to add a memory mapped gpio
chip.  So, of_mm_gpiochip_remove() should be used when removing it.

The direct call of gpiochip_remove() misses unmapping the register
and freeing the label.
Signed-off-by: Masahiro Yamada <yamada.masahiro@socionext.com>
Reviewed-by: Alexandre Courbot <acourbot@nvidia.com>
Acked-by: Tien Hock Loh <thloh@altera.com>
Signed-off-by: Linus Walleij <linus.walleij@linaro.org>