tty->driver_data or state->port may end up being NULL in uart_close.
Make sure that we correctly clean up in this case, rather than
oopsing.