Drop the helper used to retrieve the serial struct pointer from the port
structure.

Note that this helper was only used when the serial structure was
actually not needed.
Reviewed-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
Signed-off-by: Johan Hovold <johan@kernel.org>