If tty_open() fails for a normal serial device, we end up doing cleanups
that should only happen for failed open of /dev/ptmx.  The results are
not pretty - devpts et.al.  end up very confused.  That's what gave
problems with ptmx.

This splits ptmx file_operations from the normal case and cleans up both
tty_open() and (new) ptmx_open().  Survived serious beating.