No need to set latency timeout at every open.

This also fixes an issue with the read latency being as high as 250ms
(instead of 1ms) for the first read after port probe.
Signed-off-by: Johan Hovold <jhovold@gmail.com>
Signed-off-by: Greg Kroah-Hartman <gregkh@suse.de>