a step towards correct hot unplug for USB devices, we need to
remove the userspace facing bits at the unplug time for correct
udev operation.
Signed-off-by: Dave Airlie <airlied@redhat.com>