Drop the custom port driver data accessors and paranoid sanity checks.
The driver data is not cleared until the driver is unbound.
Reviewed-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
Signed-off-by: Johan Hovold <johan@kernel.org>