This reverts commit bb00788b.

Based on review comments, it was applied too soon and needs more work.
Reported-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
Link: https://lore.kernel.org/r/20231005081716.GA13853@pendragon.ideasonboard.com
Cc: Michael Grzeschik <m.grzeschik@pengutronix.de>
Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>