The removal cleanup code is duplicated between the different bus glues.
Move it to a central location.
Signed-off-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
Signed-off-by: Felipe Balbi <balbi@ti.com>