Rather than in late_init to avoid race conditions between freeing the
buffers and the initial modeset.
Reviewed-by: Christian König <christian.koenig@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>