Looks like this code was only working on boards that had
the usb_l4_ick enabled in the bootloader.
Signed-off-by: Tony Lindgren <tony@atomide.com>