1. 29 Jun, 2024 11 commits
    • Levin Zimmermann's avatar
      proto: Update 'Compression' to int to support different compression algorithms · ebad4432
      Levin Zimmermann authored
      With nexedi/neoppod@fd80cc30 NEO/py added support to encode the compression
      algorithm with the 'Compression' parameter. Before this, compression could
      only be true (= with compression) or false (= without compression). Now
      the absence of compression is encoded with 0. Any other number than 0
      encodes a compression algorithm. The mapping is currently:
      
      	1 = zlib
      
      In the future, 2 could mean zstd [1].
      
      [1] https://github.com/facebook/zstd/issues/1134
      ebad4432
    • Levin Zimmermann's avatar
      proto/msgpack: Fix {de,en}coding INVALID_{TID,OID} · ce6fe46e
      Levin Zimmermann authored
      In pre-msgpack protocol an 'INVALID_{TID,OID}' was always
      decoded as 'None' in NEO/py [1]. But in msgpack protocol
      this isn't true anymore. An `INVALID_TID` is now decoded
      as an `INVALID_TID`. And this then leads to errors later [2].
      We fix this by encoding 'INVALID_{TID,OID}' to NIL on the
      wire and by decoding NIL to 'INVALID_{TID,OID}'.
      
      ---
      
      [1] https://lab.nexedi.com/nexedi/neoppod/-/blob/6332112cba979dfd29b40fe9f98d097911fde696/neo/lib/protocol.py#L579-583
      [2] With SQLite backend we can see the following exception:
      
      Traceback (most recent call last):
        File "/home/levin/neo/neo/tests/functional/__init__.py", line 192, in start
          self.run()
        File "/home/levin/neo/neo/tests/functional/__init__.py", line 288, in run
          getattr(neo.scripts,  self.command).main()
        File "/home/levin/neo/neo/scripts/neostorage.py", line 32, in main
          app.run()
        File "/home/levin/neo/neo/storage/app.py", line 196, in run
          self._run()
        File "/home/levin/neo/neo/storage/app.py", line 227, in _run
          self.doOperation()
        File "/home/levin/neo/neo/storage/app.py", line 301, in doOperation
          poll()
        File "/home/levin/neo/neo/storage/app.py", line 145, in _poll
          self.em.poll(1)
        File "/home/levin/neo/neo/lib/event.py", line 160, in poll
          to_process.process()
        File "/home/levin/neo/neo/lib/connection.py", line 508, in process
          self._handlers.handle(self, self._queue.pop(0))
        File "/home/levin/neo/neo/lib/connection.py", line 93, in handle
          self._handle(connection, packet)
        File "/home/levin/neo/neo/lib/connection.py", line 108, in _handle
          pending[0][1].packetReceived(connection, packet)
        File "/home/levin/neo/neo/lib/handler.py", line 125, in packetReceived
          self.dispatch(*args)
        File "/home/levin/neo/neo/lib/handler.py", line 75, in dispatch
          method(conn, *args, **kw)
        File "/home/levin/neo/neo/storage/handlers/client.py", line 67, in askObject
          o = app.dm.getObject(oid, at, before)
        File "/home/levin/neo/neo/storage/database/manager.py", line 484, in getObject
          before_tid and u64(before_tid))
        File "/home/levin/neo/neo/storage/database/sqlite.py", line 336, in _getObject
          r = q(sql + ' AND tid=?', (partition, oid, tid))
      OverflowError: Python int too large to convert to SQLite INTEGER
      ce6fe46e
    • Levin Zimmermann's avatar
      go/neo/proto/RowInfo: Fix representation on the wire · f7731bbf
      Levin Zimmermann authored
      Some NEO protocol packets have the field 'RowList'. This field contains
      information about each row of a partition table. In NEO/go the information
      of each row is represented with the 'RowInfo' type [1]. This type is defined
      as a struct with the field ‘CellList’. ‘CellList’ is defined as a list of
      'CellInfo' [1] (e.g. an entry for each cell).
      
      NEO/go {en,de}codes struct types with ‘genStructHead’ (structures in
      golang are encoded as arrays in msgpack) [2]. From the 'RowList'
      definition, the msgpack decoder currently expects the following msgpack
      array structure:
      
          ArrayHeader (RowList)
      
              ArrayHeader (RowInfo)
      
                  ArrayHeader (CellList)
      
                      ArrayHeader (CellInfo)
      
                          int32 (NID)
                          enum (State)
      
      However NEO/py actually sends:
      
          ArrayHeader (RowList)
      
              ArrayHeader (CellList)
      
                  ArrayHeader (CellInfo)
      
                      int32 (NID)
                      enum (State)
      
      In other words, currently the NEO/go msgpack encoder expects one more
      nesting, which NEO/py doesn’t provide (and which also doesn’t seem to
      be necessary as the outer nesting would always only contain one
      element). We could adjust the msgpack {en,de}coder to introduce an
      exception for the 'RowInfo' type, however as the protocol definition in
      'proto.go' aims to transparently reflect the structure of the packets on
      the wire, it seems to be more appropriate to fix this straight in the
      protocol definition. This is also less error-prone as we don't have to
      fix all the different positions of the encoder, decoder & sizer and it's
      less code (particularly if 'RowInfo' doesn't stay the only case for such
      an issue).
      
      [1] https://lab.nexedi.com/kirr/neo/-/blob/1ad088c8/go/neo/proto/proto.go#L391-394
      [2] https://lab.nexedi.com/kirr/neo/-/blob/1ad088c8/go/neo/proto/protogen.go#L1770-1775
      f7731bbf
    • Levin Zimmermann's avatar
      go/neo/proto/msgpack: Fix decoding of IdTime · c8933d3d
      Levin Zimmermann authored
      The data type of IdTime is 'Optional[float]' [1], [2]. Before this patch the
      msgpack decoder could only decode 'IdTime' in case its data type was
      'float'. Now it also supports the decoding of IdTime in case it is NIL.
      
      [1] See here, the fifth argument is IdTime:
      
          https://lab.nexedi.com/nexedi/neoppod/-/blob/3ddb6663/neo/master/handlers/identification.py#L26-27
      
          This is found to be 'Optional[float]':
      
          https://lab.nexedi.com/nexedi/neoppod/-/blob/3ddb6663/neo/tests/protocol#L98
      
      [2] This seems to be true for both, pre-msgpack and post-msgpack
          protocol, because in the pre-msgpack NEO/go there is already this
          note:
      
          https://lab.nexedi.com/kirr/neo/-/blob/1ad088c8/go/neo/proto/proto.go#L352-357
      c8933d3d
    • Levin Zimmermann's avatar
      go/neo/proto/msgpack: Fix 'types.Struct' decoding · 4fff4d4c
      Levin Zimmermann authored
      In NEO/go protocol, we describe some parameters that can
      be send via NEO packages with structures. On the wire all of
      these structures are encoded as msgpack arrays [1]. Msgpack
      arrays can have 1 or more bytes as a header [2]. Therefore
      it's better to use "ReadArrayHeaderBytes" than using
      
        >>> data = data[1:]
      
      which fails in case we have array with 3 or 5 header bytes.
      
      [1] One could also declare a protocol where these parameters aren't
          send as msgpack arrays but as msgpack maps. See here for the
          msgpack map specification:
      
          https://github.com/msgpack/msgpack/blob/9aa092d6ca81/spec.md?plain=1#L338
      
      [2] https://github.com/msgpack/msgpack/blob/9aa092d6ca81/spec.md?plain=1#L315
      4fff4d4c
    • Levin Zimmermann's avatar
      go/neo/proto: Update msgcode increment logic to new protocol · 27a78346
      Levin Zimmermann authored
      In pre-msgpack protocol, the msgcode increment logic is like this:
      
      Notify 	 add +1 to next msgcode
      Request  add +0 to next msgcode   (next msg is answer & should therefore be the same)
      Answer 	 add +1 to next msgcode
      
      ('Answer' msgcode is adjusted by 'AnswerBit')
      
      In post-msgpack protocol, the logic is a bit different:
      
      Notify 	 add +1 to next msgcode
      Request  add +0 to next msgcode   (next msg is answer & should therefore be the same)
      Answer 	 add +2 to next msgcode
      
      So here we produce gaps after Request/Answer pairs.
      27a78346
    • Levin Zimmermann's avatar
      go/neo/proto: Update enum order to new protocol · 94a43764
      Levin Zimmermann authored
      The enum order has changed in the migration to the msgpack encoded
      protocol.
      
      See here for pre-msgpack order:
      
      https://lab.nexedi.com/nexedi/neoppod/-/blob/6332112cb/neo/lib/protocol.py#L62-141
      
      And here for post-msgpack order:
      
      https://lab.nexedi.com/nexedi/neoppod/-/blob/9d0bf97a1327182ac29e95d65fd9e18742c43d1f/neo/lib/protocol.py#L102-181
      
      As this order defines the encoding of the enum values, this needs to be
      strictly followed to be compatible (to follow the protocol declaration).
      94a43764
    • Levin Zimmermann's avatar
      01112210
    • Levin Zimmermann's avatar
      proto/msgpack: Fix handshake magic · e6fa7450
      Levin Zimmermann authored
      This fixes the basic NEO handshake in msgpack encoding.
      e6fa7450
    • Levin Zimmermann's avatar
      go/neo/neonet: Fix segmentation violation in case handshake fails · 00db3299
      Levin Zimmermann authored
      Because 'err' was locally assigned inside the loop of 'DialLink' [1],
      it was always 'nil' as a function return value, even when
      'handshakeClient' actually returned an error. This lead to the
      unfortunate situation that the function sometimes returned 'link=nil'
      and 'err=nil', so that the function caller tried to access 'link'
      attributes which lead to 'runtime error: invalid memory address or nil
      pointer dereference'. This patch fixes this and now the function
      correctly returns an error if the dialing fails.
      
      [1] `peerConn, err := networker.Dial(ctx, addr)`
      00db3299
    • Levin Zimmermann's avatar
      go/neo/client_test: NEO/py now only supports M encoding · 6eaf7dec
      Levin Zimmermann authored
      With the switch to the msgpack encoding, the NEO/py server
      now only supports the M encoding.
      6eaf7dec
  2. 14 Mar, 2024 1 commit
    • Julien Muchembled's avatar
      protocol: switch to msgpack for packet serialization · ccd47c48
      Julien Muchembled authored
      Not only for performance reasons (at least 3% faster) but also because of
      several ugly things in the way packets were defined:
      - packet field names, which are only documentary; for roots fields,
        they even just duplicate the packet names
      - a lot of repetitions for packet names, and even confusion between the name
        of the packet definition and the name of the actual notify/request packet
      - the need to implement field types for anything, like PByte to support new
        compression formats, since PBoolean is not enough
      
      neo/lib/protocol.py is now much smaller.
      ccd47c48
  3. 02 Feb, 2024 3 commits
    • Kirill Smelkov's avatar
      Merge branch 'master' into t · 1ad088c8
      Kirill Smelkov authored
      * master:
        go/zodb: Handle common options in zurl in generic layer
      1ad088c8
    • Kirill Smelkov's avatar
      X: Apply new URI scheme to NEO/go + some refactors and tests of URL parser · a4a9d69d
      Kirill Smelkov authored
      /reviewed-by @kirr
      /reviewed-on !4
      
      * kirr/t+new-uri:
        Revert "Y client: Adjust URI scheme to move client-specific options to fragment"
        fixup! client.go: Fix URI client option parsing for supported + unsupported options
        client.go: Fix URI client option parsing for supported + unsupported options
        fixup! client_test: Add tests for NEO URI parser
        client_test: Add tests for NEO URI parser
        fixup! client: Refactor openClientByURL for easier testing
        client: Refactor openClientByURL for easier testing
        Y go/zodb: Handle common options in zurl in generic layer
      a4a9d69d
    • Kirill Smelkov's avatar
      go/zodb: Handle common options in zurl in generic layer · f7776fc1
      Kirill Smelkov authored
      Offload drivers from handling options such as ?read-only=1 and force
      them to deal with such options only via DriverOptions, never zurl.
      
      See added comment for details.
      
      /reviewed-by @levin.zimmermann
      /reviewed-on !4
      f7776fc1
  4. 29 Jan, 2024 8 commits
  5. 22 Aug, 2023 1 commit
  6. 02 Aug, 2023 8 commits
  7. 01 Aug, 2023 1 commit
    • Kirill Smelkov's avatar
      fixup! proto.NotPrimaryMaster: Fix .Primary data type (1) · 3cb82317
      Kirill Smelkov authored
      Rerun `go generate`. As the diff in zproto-marshal.go shows changing
      NotPrimaryMaster.Primary type from NodeID to int8 actually does make a
      difference. This happens because NodeID type is based on int32 and
      changing that to int8 changes how NotPrimaryMaster structure is layed
      out in memory and on the wire.
      
      The changes to zproto-marshal.go in 5d93e434 seem to be done by hand
      and not matching the change to proto.go even though head of
      zproto-marshal.go says
      
      	// Code generated by protogen.go; DO NOT EDIT.
      3cb82317
  8. 18 Jul, 2023 7 commits
    • Levin Zimmermann's avatar
      client_test: Add nmaster={1,2} to test matrix · b2929804
      Levin Zimmermann authored
      Tests should work with both one master or more than one masters.
      b2929804
    • Levin Zimmermann's avatar
      client_test: Support test cluster /w >1 master · 6dba6409
      Levin Zimmermann authored
      Now it's possible to run client tests against a NEO cluster which
      has more than one master nodes. We need this adjustement in order
      to test NEO/go client modification in order to support more than
      one master node.
      6dba6409
    • Levin Zimmermann's avatar
      proto.NotPrimaryMaster: Fix .Primary data type · 5d93e434
      Levin Zimmermann authored
      The '.Primary' attribute of the 'NotPrimaryMaster' packet has been
      assigned to 'NodeID' data type. This is incorrect, because the data
      doesn't represent the ID of the node, but an index of the
      '.KnownMasterList' [1]. In the old protocol NEO/py therefore also
      used 'PSignedNull' instead of 'PUUID' [2]. This patches fixes the data
      type of '.Primary' and uses 'int8' instead of 'NodeID'. Technically this
      doesn't make any difference, but semantically for human beings the code
      is easier to understand now.
      
      [1] https://lab.nexedi.com/nexedi/neoppod/blob/c6453626/neo/lib/handler.py#L161
      [2] https://lab.nexedi.com/nexedi/neoppod/blob/c6453626/neo/lib/protocol.py#L716
      5d93e434
    • Levin Zimmermann's avatar
      TalkMaster: Switch master if dialed M is secondary · 22e5d1e9
      Levin Zimmermann authored
      When connecting to a master node, the client needs to try a different
      master if the initially tried one is a secondary master node. This
      statement wasn't implemented yet before this patch and therefore it was
      good luck if the initally tried master was the primary one - and the
      connection worked - or if it was a secondary master - and the client
      got stuck in re-trying the same node forever. This patch makes NEO/go
      usage with clusters of more than one master therefore much more stable.
      22e5d1e9
    • Levin Zimmermann's avatar
      Dial: Catch NotPrimaryMaster & return custom error · bbf9f440
      Levin Zimmermann authored
      After initial handshake a NEO node checks the identification of its peer
      by sending the 'RequestIdentification' packet. In case the peer is a
      secondary master it responds with 'NotPrimaryMaster'. Before this patch
      'Dial' ignored the 'NotPrimaryMaster' packet and simply returned a
      general error. Now - after this patch - 'Dial' returns an instance of
      'proto.NotPrimaryMaster' (which implements 'Error').
      This helps a caller to correctly handle the secondary-master-case, which
      otherwise is impossible to differentiate from any other error
      possibility.
      bbf9f440
    • Levin Zimmermann's avatar
      proto: Implement Error for NotPrimaryMaster · 8811f8b4
      Levin Zimmermann authored
      When a client receives 'NotPrimaryMaster' from a secondary master, the
      situation is similar to the situation when we receive an error: the
      other node tells us, don't connect with me, connect with someone else.
      Finally the peer even closes the connection.
      
      Due to this similarity in structure (& because it helps us later to
      teach NEO/go to correctly handle 'NotPrimaryMaster' with minimal
      changes), we implement 'Error' for 'proto.NotPrimaryMaster'. Now
      'NotPrimaryMaster' can be treated like an error.
      8811f8b4
    • Levin Zimmermann's avatar
      openClientByURL: Fix for >1 master (split URL host) · 2a75cdb0
      Levin Zimmermann authored
      In a NEO URI more than one master node can be specified, because a NEO
      cluster may have more masters than one. But before this patch
      'openClientByURL' always assumed that the given URL only specifies one
      master. Now the host is split into potentially > 1 master nodes. It
      therefore works now in the same way as the Python implementation [1].
      
      [1] https://lab.nexedi.com/nexedi/neoppod/blob/342168cd/neo/client/zodburi.py#L64
      2a75cdb0