This reverts commit 8f2c9544.

As it breaks g_ether on my Baytrail FFRD8 device. Everything starts out
fine, but after a bit of data has been transferred it just stops
flowing.

Note that I do get a bunch of these "NOHZ: local_softirq_pending 08"
when booting the machine, but I'm not really sure if they're related
to this problem.

Cc: Felipe Balbi <balbi@ti.com>
Cc: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
Cc: linux-usb@vger.kernel.org
Cc: stable@vger.kernel.org
Signed-off-by: Ville Syrjälä <ville.syrjala@linux.intel.com>
Signed-off-by: Felipe Balbi <balbi@ti.com>