No need to trigger runtime pm in driver removal, otherwise if user
disable auto suspend via sys file, runtime suspend may be entered,
which will call dwc3_core_exit() again and there will be clock disable
not balance warning:

[ 2026.820154] xhci-hcd xhci-hcd.0.auto: remove, state 4
[ 2026.825268] usb usb2: USB disconnect, device number 1
[ 2026.831017] xhci-hcd xhci-hcd.0.auto: USB bus 2 deregistered
[ 2026.836806] xhci-hcd xhci-hcd.0.auto: remove, state 4
[ 2026.842029] usb usb1: USB disconnect, device number 1
[ 2026.848029] xhci-hcd xhci-hcd.0.auto: USB bus 1 deregistered
[ 2026.865889] ------------[ cut here ]------------
[ 2026.870506] usb2_ctrl_root_clk already disabled
[ 2026.875082] WARNING: CPU: 0 PID: 731 at drivers/clk/clk.c:958
clk_core_disable+0xa0/0xa8
[ 2026.883170] Modules linked in: dwc3(-) phy_fsl_imx8mq_usb [last
unloaded: dwc3]
[ 2026.890488] CPU: 0 PID: 731 Comm: rmmod Not tainted
5.8.0-rc7-00280-g9d08cca-dirty #245
[ 2026.898489] Hardware name: NXP i.MX8MQ EVK (DT)
[ 2026.903020] pstate: 20000085 (nzCv daIf -PAN -UAO BTYPE=--)
[ 2026.908594] pc : clk_core_disable+0xa0/0xa8
[ 2026.912777] lr : clk_core_disable+0xa0/0xa8
[ 2026.916958] sp : ffff8000121b39a0
[ 2026.920271] x29: ffff8000121b39a0 x28: ffff0000b11f3700
[ 2026.925583] x27: 0000000000000000 x26: ffff0000b539c700
[ 2026.930895] x25: 000001d7e44e1232 x24: ffff0000b76fa800
[ 2026.936208] x23: ffff0000b76fa6f8 x22: ffff800008d01040
[ 2026.941520] x21: ffff0000b539ce00 x20: ffff0000b7105000
[ 2026.946832] x19: ffff0000b7105000 x18: 0000000000000010
[ 2026.952144] x17: 0000000000000001 x16: 0000000000000000
[ 2026.957456] x15: ffff0000b11f3b70 x14: ffffffffffffffff
[ 2026.962768] x13: ffff8000921b36f7 x12: ffff8000121b36ff
[ 2026.968080] x11: ffff8000119e1000 x10: ffff800011bf26d0
[ 2026.973392] x9 : 0000000000000000 x8 : ffff800011bf3000
[ 2026.978704] x7 : ffff800010695d68 x6 : 0000000000000252
[ 2026.984016] x5 : ffff0000bb9881f0 x4 : 0000000000000000
[ 2026.989327] x3 : 0000000000000027 x2 : 0000000000000023
[ 2026.994639] x1 : ac2fa471aa7cab00 x0 : 0000000000000000
[ 2026.999951] Call trace:
[ 2027.002401]  clk_core_disable+0xa0/0xa8
[ 2027.006238]  clk_core_disable_lock+0x20/0x38
[ 2027.010508]  clk_disable+0x1c/0x28
[ 2027.013911]  clk_bulk_disable+0x34/0x50
[ 2027.017758]  dwc3_core_exit+0xec/0x110 [dwc3]
[ 2027.022122]  dwc3_suspend_common+0x84/0x188 [dwc3]
[ 2027.026919]  dwc3_runtime_suspend+0x74/0x9c [dwc3]
[ 2027.031712]  pm_generic_runtime_suspend+0x28/0x40
[ 2027.036419]  genpd_runtime_suspend+0xa0/0x258
[ 2027.040777]  __rpm_callback+0x88/0x140
[ 2027.044526]  rpm_callback+0x20/0x80
[ 2027.048015]  rpm_suspend+0xd0/0x418
[ 2027.051503]  __pm_runtime_suspend+0x58/0xa0
[ 2027.055693]  dwc3_runtime_idle+0x7c/0x90 [dwc3]
[ 2027.060224]  __rpm_callback+0x88/0x140
[ 2027.063973]  rpm_idle+0x78/0x150
[ 2027.067201]  __pm_runtime_idle+0x58/0xa0
[ 2027.071130]  dwc3_remove+0x64/0xc0 [dwc3]
[ 2027.075140]  platform_drv_remove+0x28/0x48
[ 2027.079239]  device_release_driver_internal+0xf4/0x1c0
[ 2027.084377]  driver_detach+0x4c/0xd8
[ 2027.087954]  bus_remove_driver+0x54/0xa8
[ 2027.091877]  driver_unregister+0x2c/0x58
[ 2027.095799]  platform_driver_unregister+0x10/0x18
[ 2027.100509]  dwc3_driver_exit+0x14/0x1408 [dwc3]
[ 2027.105129]  __arm64_sys_delete_module+0x178/0x218
[ 2027.109922]  el0_svc_common.constprop.0+0x68/0x160
[ 2027.114714]  do_el0_svc+0x20/0x80
[ 2027.118031]  el0_sync_handler+0x88/0x190
[ 2027.121953]  el0_sync+0x140/0x180
[ 2027.125267] ---[ end trace 027f4f8189958f1f ]---
[ 2027.129976] ------------[ cut here ]------------

Fixes: fc8bb91b ("usb: dwc3: implement runtime PM")
Cc: <stable@vger.kernel.org>
Signed-off-by: Li Jun <jun.li@nxp.com>
Signed-off-by: Felipe Balbi <balbi@kernel.org>