The patch was based on my missinterpretation of the API and only
accidentally worked for me. Let's clean it out to not confuse others.

This reverts commit 3ff3f518.
Signed-off-by: Bjorn Andersson <bjorn.andersson@linaro.org>
Signed-off-by: Mark Brown <broonie@kernel.org>