neonet/newlink: Fix lost conn in encoding detector

If the peers encoding is different than our encoding two different
scenarios can happen, because the handshake order is undefined (e.g.
we don't know if our handshake is received before the peer sends its
handshake):

1. Our handshake is received before peer sends its handshake, NEO/py
closes connection if it sees unexpected magic, version, etc.

2. The client already sends a handshake before it proceeds our handshake.
In this case it initally sends us it version, we can extract its encoding,
and only later, once it proceeded our handshake with the bad encoding,
it closes the connection.

Before this patch case (2) wasn't handled correctly by the automatic
encoding detection of 'DialLink'. 'DialLink' simply accepted the
different-than-expected encoding, but once the peer proceeded the nodes
handshake the peer closed the connection and the initially established
and returned link was immediately closed again. Due to this it was good
luck whether connecting with a peer different with an encoding different
from the expected one worked or didn't work (it depended on which handshake
was faster). Now 'DialLink' should reliably find the correct encoding
and return a stable link.

go/neo/neonet/newlink.go

@@ -126,6 +126,9 @@ func _handshakeClient(ctx context.Context, conn net.Conn, version uint32, encPre
 		if err != nil {
 			return err
 		}
+		if peerEnc != encPrefer {
+			return ErrEncodingMismatch
+		}
 
 		// verify version
 		if peerVer != version {
@@ -299,6 +302,7 @@ func init() {
 // handshake and wraps the connection as NodeLink.
 func DialLink(ctx context.Context, net xnet.Networker, addr string) (link *NodeLink, err error) {
 	for _, enc := range dialEncTryOrder {
+
 		peerConn, err := net.Dial(ctx, addr)
 		if err != nil {
 			return nil, err
@@ -306,10 +310,22 @@ func DialLink(ctx context.Context, net xnet.Networker, addr string) (link *NodeL
 
 		link, err = handshakeClient(ctx, peerConn, proto.Version, enc)
 
-		// NEO/py closes connection if it sees unexpected magic, version, etc.
-		// -> in such case retry with next encoding trying to autodetect and match server.
+		// If the peers encoding is different than our encoding two different
+		// scenarios can happen, because the handshake order is undefined (e.g.
+		// we don't know if our handshake is received before the peer sends its
+		// handshake):
+		//
+		//	  1. Our handshake is received before peer sends its handshake, NEO/py
+		//       closes connection if it sees unexpected magic, version, etc.
+		//
+		//	  2. The client already sends a handshake before it proceeds our handshake.
+		//       In this case it initally sends us it version, we can extract its encoding,
+		//       and only later, once it proceeded our handshake with the bad encoding,
+		//       closes the connection.
+		//
+		// -> in both cases retry with next encoding trying to autodetect and match server.
 		// -> stop trying on success, or on any other error.
-		if err == nil || !errors.Is(err, io.ErrUnexpectedEOF) {
+		if err == nil || !(errors.Is(err, io.ErrUnexpectedEOF) || errors.Is(err, ErrEncodingMismatch)) {
 			break
 		}
 	}
@@ -318,6 +334,9 @@ func DialLink(ctx context.Context, net xnet.Networker, addr string) (link *NodeL
 	return link, err
 }
 
+var ErrEncodingMismatch = errors.New("protocol encoding mismatch")
+
+
 // ListenLink starts listening on laddr for incoming connections and wraps them as NodeLink.
 //
 // The listener accepts only those connections that pass NEO protocol handshake.