Otherwise we will hit a list handling assertion when moving the object
to the inactive list.
Signed-off-by: Chris Wilson <chris@chris-wilson.co.uk>