The original implementation removes reverse links for any input link and
assumes the presense of sink/source.
It fails when the link is a not a data link.
media_entity_remove_links when there's an ancillary link can also fail.

We only need to remove reverse links for a data link.
Signed-off-by: Yunke Cao <yunkec@google.com>
Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
Signed-off-by: Mauro Carvalho Chehab <mchehab@kernel.org>