Make sure only to copy any actual data rather than the whole buffer,
when releasing the temporary buffer used for unaligned non-isochronous
transfers.
Signed-off-by: Johan Hovold <johan@kernel.org>
Signed-off-by: Felipe Balbi <balbi@ti.com>