We don't need 2 .remove callback

Fixes: 7480389f ("ASoC: twl6040: replace codec to component")
Reported-by: Julia Lawall <julia.lawall@lip6.fr>
Reported-by: kbuild test robot <fengguang.wu@intel.com>
Signed-off-by: Kuninori Morimoto <kuninori.morimoto.gx@renesas.com>
Acked-by: Peter Ujfalusi <peter.ujfalusi@ti.com>
Signed-off-by: Mark Brown <broonie@kernel.org>