If we unplug a udl device, the usb callback with deinit the
mode_config struct, however userspace will still have an open
file descriptor and a framebuffer on that device. When userspace
closes the fd, we'll oops because it'll try and look stuff up
in the object idr which we've destroyed.

This punts destroying the mode objects until release time instead.

Cc: stable@vger.kernel.org
Reviewed-by: Daniel Vetter <daniel.vetter@ffwll.ch>
Signed-off-by: Dave Airlie <airlied@redhat.com>
Link: https://patchwork.freedesktop.org/patch/msgid/20190405031715.5959-2-airlied@gmail.com