The refactoring done as part of adding the core support for handling
waiting for slave transfer dropped a conditional which meant that we
started waiting for completion of all transfers, not just those that the
controller asked for.  This caused hangs and massive delays on platforms
that don't need the core delay.  Re-add the delay to fix this.

Fixes: 810923f3 (spi: Deal with slaves that return from transfer_one() unfinished)
Reported-by: Tony Lindgren <tony@atomide.com>
Tested-by: Tony Lindgren <tony@atomide.com>
Signed-off-by: Mark Brown <broonie@kernel.org>