This reverts commit 6939a5ac.

Daniel Vetter supplied a set of fixes for all the module unload bugs he
could trigger on his machines, so let the fun recommence!