The work cancelling has had several hazards, ranging from potentially
executing work after the driver is in OFF state, to executing work after
the driver and relevant memory structures are already removed.

Fix these.
Signed-off-by: Juuso Oikarinen <juuso.oikarinen@nokia.com>
Reviewed-by: Teemu Paasikivi <ext-teemu.3.paasikivi@nokia.com>
Signed-off-by: Luciano Coelho <luciano.coelho@nokia.com>