This simplifies cleanup paths and fixes a probe time crash in the error
path when trying to cleanup mode setting before it was initialized.
Signed-off-by: Laurent Pinchart <laurent.pinchart+renesas@ideasonboard.com>