Not just the userspace relocs, otherwise we won't wait
for a swapped out page tables to be swapped in again.

v2: rebased on Alex current drm-fixes-3.18
Signed-off-by: Christian König <christian.koenig@amd.com>
Cc: stable@vger.kernel.org
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>