since the handle references are all tied to a file_priv, and when it disappears
all the handle refs go with it.

The fbcon ones we'd only notice on unload, but the nouveau notifier one
would would happen on reboot.

nouveau: Reported-by: Marc Dionne <marc.c.dionne@gmail.com>
nouveau: Tested-by: Marc Dionne <marc.c.dionne@gmail.com>
i915 unload: Reported-by: Keith Packard <keithp@keithp.com>
Acked-by: Ben Skeggs <bskeggs@redhat.com>
Signed-off-by: Dave Airlie <airlied@redhat.com>