Like for via.

v2: Actually drop the idlelock again if taken.

v3: Fixup.

v4: Fixup the "has master" vs. "is master" confusion the refactor
introduced.

v5: Drop the idlelock in the early return path.
Signed-off-by: Daniel Vetter <daniel.vetter@ffwll.ch>
Signed-off-by: Dave Airlie <airlied@redhat.com>