Even though the driver cannot be unloaded at the moment, it is still
good to properly free the IRQ handlers in the channel removal function.
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>