This gets rid of the port semaphore, and the serialization caused by
that, and replaces it with the proper reference counting logic for
the usb serial object.