We shouldn't change a host-only dwc3 to gadget-only
if driver is built as gadget-only. Fix that up here.
Signed-off-by: Felipe Balbi <felipe.balbi@linux.intel.com>