Drop the likewise paranoid serial structure sanity checks.

USB serial core sets the serial pointer in the port structures at
initialisation and it is never cleared, and similar for the serial
structure type.
Reviewed-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
Signed-off-by: Johan Hovold <johan@kernel.org>