1. 30 Apr, 2024 2 commits
  2. 24 Apr, 2024 6 commits
    • Levin Zimmermann's avatar
      Revert "proto/msgpack: Allow nil=false simplification" · 1791b053
      Levin Zimmermann authored
      This reverts commit d779695c.
      1791b053
    • Levin Zimmermann's avatar
      NEO/py/storage: Ensure compression is always BOOL on the wire · 0e4bb572
      Levin Zimmermann authored
      Currently it can be:
      
      - nil
      - bool
      - integer
      
      In python this isn't really a problem due to '__bool__', but in Golang
      we don't have this. Yes - it would be possible to make a parser in
      Golang to decode all of these values as either True/False, but this is
      much more complex than just adjusting python to send consistent data
      types. It's also easier to describe a protocol where one parameter is
      always 'bool' instead of having a parameter that is 'bool | int | NoneType'.
      0e4bb572
    • Levin Zimmermann's avatar
      fixup! proto/msgpack: Fix encoding INVALID_{TID,OID} · ccdfd363
      Levin Zimmermann authored
      This is the decoding part of 37dd83e9
      ccdfd363
    • Levin Zimmermann's avatar
      proto/msgpack: Allow nil=false simplification · d779695c
      Levin Zimmermann authored
      NEO/py sometimes sends 'None' for boolean values (being equal to 'False'
      due to pythons 'bool(None) == False' logic). Let's accept python based
      False values (= Nil) as equally acceptable.
      
      Without this patch we can see errors as:
      
      xtesting.go:330: load 0285cbad6e147b66:0000000000000005: returned err unexpected:
              have: neo://127.0.0.1:30002/1: load 0285cbad6e147b66:0000000000000005: 127.0.0.1:48928 - 127.0.0.1:33051 .251: decode: decode: M: AnswerObject.Compression/op: got opcode 00;  expected any from c3c2
      d779695c
    • Levin Zimmermann's avatar
      9e4dd249
    • Levin Zimmermann's avatar
      proto/msgpack: Fix encoding INVALID_{TID,OID} · 37dd83e9
      Levin Zimmermann authored
      In pre-msgpack protocol an 'INVALID_{TID,OID}' was always
      decoded as 'None' in NEO/py [1]. But in msgpack protocol
      this isn't true anymore. An `INVALID_TID` is now decoded
      as an `INVALID_TID`. And this then leads to errors later [2].
      Instead of restoring NEO/py to a pre-msgpack behaviour, we
      fix this on NEO/go side by encoding 'INVALID_{TID,OID}' to NIL
      on the wire. This is good as it makes the messages we send
      smaller.
      
      [1] https://lab.nexedi.com/nexedi/neoppod/-/blob/6332112cba979dfd29b40fe9f98d097911fde696/neo/lib/protocol.py#L579-583
      [2] With SQLite backend we can see the following exception:
      
      Traceback (most recent call last):
        File "/home/levin/neo/neo/tests/functional/__init__.py", line 192, in start
          self.run()
        File "/home/levin/neo/neo/tests/functional/__init__.py", line 288, in run
          getattr(neo.scripts,  self.command).main()
        File "/home/levin/neo/neo/scripts/neostorage.py", line 32, in main
          app.run()
        File "/home/levin/neo/neo/storage/app.py", line 196, in run
          self._run()
        File "/home/levin/neo/neo/storage/app.py", line 227, in _run
          self.doOperation()
        File "/home/levin/neo/neo/storage/app.py", line 301, in doOperation
          poll()
        File "/home/levin/neo/neo/storage/app.py", line 145, in _poll
          self.em.poll(1)
        File "/home/levin/neo/neo/lib/event.py", line 160, in poll
          to_process.process()
        File "/home/levin/neo/neo/lib/connection.py", line 508, in process
          self._handlers.handle(self, self._queue.pop(0))
        File "/home/levin/neo/neo/lib/connection.py", line 93, in handle
          self._handle(connection, packet)
        File "/home/levin/neo/neo/lib/connection.py", line 108, in _handle
          pending[0][1].packetReceived(connection, packet)
        File "/home/levin/neo/neo/lib/handler.py", line 125, in packetReceived
          self.dispatch(*args)
        File "/home/levin/neo/neo/lib/handler.py", line 75, in dispatch
          method(conn, *args, **kw)
        File "/home/levin/neo/neo/storage/handlers/client.py", line 67, in askObject
          o = app.dm.getObject(oid, at, before)
        File "/home/levin/neo/neo/storage/database/manager.py", line 484, in getObject
          before_tid and u64(before_tid))
        File "/home/levin/neo/neo/storage/database/sqlite.py", line 336, in _getObject
          r = q(sql + ' AND tid=?', (partition, oid, tid))
      OverflowError: Python int too large to convert to SQLite INTEGER
      37dd83e9
  3. 10 Apr, 2024 1 commit
  4. 09 Apr, 2024 1 commit
  5. 08 Apr, 2024 2 commits
  6. 07 Apr, 2024 4 commits
  7. 06 Apr, 2024 1 commit
  8. 19 Mar, 2024 1 commit
    • Levin Zimmermann's avatar
      fixup! go/neo/neonet: MessagePack support for link layer (draft) · d64f1baf
      Levin Zimmermann authored
      In pre-msgpack protocol, the msgcode increment logic is like this:
      
      Notify 	 add +1 to next msgcode
      Request  add +0 to next msgcode   (next msg is answer & should therefore be the same)
      Answer 	 add +1 to next msgcode
      
      ('Answer' msgcode is adjusted by 'AnswerBit')
      
      In post-msgpack protocol, the logic is a bit different:
      
      Notify 	 add +1 to next msgcode
      Request  add +0 to next msgcode   (next msg is answer & should therefore be the same)
      Answer 	 add +2 to next msgcode
      
      So here we produce gaps after Request/Answer pairs.
      
      In the current version, this patch breaks the functionality of
      the pre-msgpack protocol, because for both protocols the same
      msgcodes are used. To still support 'N' we need to use different
      msgcodes depending on if 'N' or 'M' is used.
      d64f1baf
  9. 18 Mar, 2024 1 commit
  10. 15 Mar, 2024 2 commits
  11. 14 Mar, 2024 5 commits
    • Levin Zimmermann's avatar
      . · c19abad1
      Levin Zimmermann authored
      c19abad1
    • Levin Zimmermann's avatar
      fixup! X on msgpack support · 1282b808
      Levin Zimmermann authored
      This is a fix of 1cf2712f:
      'AppendString' is the equivalent to NEO/py 'NEO' encoding, not
      'AppendBytes'.
      1282b808
    • Levin Zimmermann's avatar
      go/neo/neonet: Fix segmentation violation in case handshake fails · 9501173e
      Levin Zimmermann authored
      Because 'err' was locally assigned inside the loop of 'DialLink' [1],
      it was always 'nil' as a function return value, even when
      'handshakeClient' actually returned an error. This lead to the
      unfortunate situation that the function sometimes returned 'link=nil'
      and 'err=nil', so that the function caller tried to access 'link'
      attributes which lead to 'runtime error: invalid memory address or nil
      pointer dereference'. This patch fixes this and now the function
      correctly returns an error if the dialing fails.
      
      [1] `peerConn, err := networker.Dial(ctx, addr)`
      9501173e
    • Levin Zimmermann's avatar
      . · 5e90fc24
      Levin Zimmermann authored
      5e90fc24
    • Julien Muchembled's avatar
      protocol: switch to msgpack for packet serialization · ccd47c48
      Julien Muchembled authored
      Not only for performance reasons (at least 3% faster) but also because of
      several ugly things in the way packets were defined:
      - packet field names, which are only documentary; for roots fields,
        they even just duplicate the packet names
      - a lot of repetitions for packet names, and even confusion between the name
        of the packet definition and the name of the actual notify/request packet
      - the need to implement field types for anything, like PByte to support new
        compression formats, since PBoolean is not enough
      
      neo/lib/protocol.py is now much smaller.
      ccd47c48
  12. 02 Feb, 2024 3 commits
    • Kirill Smelkov's avatar
      Merge branch 'master' into t · 1ad088c8
      Kirill Smelkov authored
      * master:
        go/zodb: Handle common options in zurl in generic layer
      1ad088c8
    • Kirill Smelkov's avatar
      X: Apply new URI scheme to NEO/go + some refactors and tests of URL parser · a4a9d69d
      Kirill Smelkov authored
      /reviewed-by @kirr
      /reviewed-on kirr/neo!4
      
      * kirr/t+new-uri:
        Revert "Y client: Adjust URI scheme to move client-specific options to fragment"
        fixup! client.go: Fix URI client option parsing for supported + unsupported options
        client.go: Fix URI client option parsing for supported + unsupported options
        fixup! client_test: Add tests for NEO URI parser
        client_test: Add tests for NEO URI parser
        fixup! client: Refactor openClientByURL for easier testing
        client: Refactor openClientByURL for easier testing
        Y go/zodb: Handle common options in zurl in generic layer
      a4a9d69d
    • Kirill Smelkov's avatar
      go/zodb: Handle common options in zurl in generic layer · f7776fc1
      Kirill Smelkov authored
      Offload drivers from handling options such as ?read-only=1 and force
      them to deal with such options only via DriverOptions, never zurl.
      
      See added comment for details.
      
      /reviewed-by @levin.zimmermann
      /reviewed-on !4
      f7776fc1
  13. 29 Jan, 2024 8 commits
  14. 22 Aug, 2023 1 commit
  15. 02 Aug, 2023 2 commits
    • Levin Zimmermann's avatar
      X: Teach NEO/go to handle multiple master nodes · db81e0de
      Levin Zimmermann authored
      See kirr/neo!2 for discussion,
      context and details.
      
      /reviewed-by @kirr
      
      * t-with-multiple-master-nodes:
        fixup! client_test: Add nmaster={1,2} to test matrix
        fixup! client_test: Support test cluster /w >1 master
        fixup! TalkMaster: Switch master if dialed M is secondary
        fixup! Node: Add support for NEO cluster with > 1 master
        fixup! Dial: Catch NotPrimaryMaster & return custom error
        fixup! proto: Implement Error for NotPrimaryMaster
        fixup! proto.NotPrimaryMaster: Fix .Primary data type (2)
        fixup! proto.NotPrimaryMaster: Fix .Primary data type (1)
        client_test: Add nmaster={1,2} to test matrix
        client_test: Support test cluster /w >1 master
        proto.NotPrimaryMaster: Fix .Primary data type
        TalkMaster: Switch master if dialed M is secondary
        Dial: Catch NotPrimaryMaster & return custom error
        proto: Implement Error for NotPrimaryMaster
        openClientByURL: Fix for >1 master (split URL host)
        Client.URL: Fix incomplete URL if > 1 master nodes
        Node: Add support for NEO cluster with > 1 master
      db81e0de
    • Kirill Smelkov's avatar
      fixup! client_test: Add nmaster={1,2} to test matrix · 4605cba1
      Kirill Smelkov authored
      Actually do test nmaster=2 case.
      4605cba1