Sometimes the connection fail happen on renesas_usbhs.
This patch fix it up.
Signed-off-by: Kuninori Morimoto <morimoto.kuninori@renesas.com>
Signed-off-by: Greg Kroah-Hartman <gregkh@suse.de>