This reverts commit fd23df72.

This commit broke set channel operation. Revert this and
implement it with a different approach in a separate patch.
Signed-off-by: Roger Quadros <rogerq@kernel.org>
Signed-off-by: Jakub Kicinski <kuba@kernel.org>