1. 24 Apr, 2024 3 commits
    • Levin Zimmermann's avatar
      proto/msgpack: Allow nil=false simplification · d779695c
      Levin Zimmermann authored
      NEO/py sometimes sends 'None' for boolean values (being equal to 'False'
      due to pythons 'bool(None) == False' logic). Let's accept python based
      False values (= Nil) as equally acceptable.
      
      Without this patch we can see errors as:
      
      xtesting.go:330: load 0285cbad6e147b66:0000000000000005: returned err unexpected:
              have: neo://127.0.0.1:30002/1: load 0285cbad6e147b66:0000000000000005: 127.0.0.1:48928 - 127.0.0.1:33051 .251: decode: decode: M: AnswerObject.Compression/op: got opcode 00;  expected any from c3c2
      d779695c
    • Levin Zimmermann's avatar
      9e4dd249
    • Levin Zimmermann's avatar
      proto/msgpack: Fix encoding INVALID_{TID,OID} · 37dd83e9
      Levin Zimmermann authored
      In pre-msgpack protocol an 'INVALID_{TID,OID}' was always
      decoded as 'None' in NEO/py [1]. But in msgpack protocol
      this isn't true anymore. An `INVALID_TID` is now decoded
      as an `INVALID_TID`. And this then leads to errors later [2].
      Instead of restoring NEO/py to a pre-msgpack behaviour, we
      fix this on NEO/go side by encoding 'INVALID_{TID,OID}' to NIL
      on the wire. This is good as it makes the messages we send
      smaller.
      
      [1] https://lab.nexedi.com/nexedi/neoppod/-/blob/6332112cba979dfd29b40fe9f98d097911fde696/neo/lib/protocol.py#L579-583
      [2] With SQLite backend we can see the following exception:
      
      Traceback (most recent call last):
        File "/home/levin/neo/neo/tests/functional/__init__.py", line 192, in start
          self.run()
        File "/home/levin/neo/neo/tests/functional/__init__.py", line 288, in run
          getattr(neo.scripts,  self.command).main()
        File "/home/levin/neo/neo/scripts/neostorage.py", line 32, in main
          app.run()
        File "/home/levin/neo/neo/storage/app.py", line 196, in run
          self._run()
        File "/home/levin/neo/neo/storage/app.py", line 227, in _run
          self.doOperation()
        File "/home/levin/neo/neo/storage/app.py", line 301, in doOperation
          poll()
        File "/home/levin/neo/neo/storage/app.py", line 145, in _poll
          self.em.poll(1)
        File "/home/levin/neo/neo/lib/event.py", line 160, in poll
          to_process.process()
        File "/home/levin/neo/neo/lib/connection.py", line 508, in process
          self._handlers.handle(self, self._queue.pop(0))
        File "/home/levin/neo/neo/lib/connection.py", line 93, in handle
          self._handle(connection, packet)
        File "/home/levin/neo/neo/lib/connection.py", line 108, in _handle
          pending[0][1].packetReceived(connection, packet)
        File "/home/levin/neo/neo/lib/handler.py", line 125, in packetReceived
          self.dispatch(*args)
        File "/home/levin/neo/neo/lib/handler.py", line 75, in dispatch
          method(conn, *args, **kw)
        File "/home/levin/neo/neo/storage/handlers/client.py", line 67, in askObject
          o = app.dm.getObject(oid, at, before)
        File "/home/levin/neo/neo/storage/database/manager.py", line 484, in getObject
          before_tid and u64(before_tid))
        File "/home/levin/neo/neo/storage/database/sqlite.py", line 336, in _getObject
          r = q(sql + ' AND tid=?', (partition, oid, tid))
      OverflowError: Python int too large to convert to SQLite INTEGER
      37dd83e9
  2. 10 Apr, 2024 1 commit
  3. 09 Apr, 2024 1 commit
  4. 08 Apr, 2024 2 commits
  5. 07 Apr, 2024 4 commits
  6. 06 Apr, 2024 1 commit
  7. 19 Mar, 2024 1 commit
    • Levin Zimmermann's avatar
      fixup! go/neo/neonet: MessagePack support for link layer (draft) · d64f1baf
      Levin Zimmermann authored
      In pre-msgpack protocol, the msgcode increment logic is like this:
      
      Notify 	 add +1 to next msgcode
      Request  add +0 to next msgcode   (next msg is answer & should therefore be the same)
      Answer 	 add +1 to next msgcode
      
      ('Answer' msgcode is adjusted by 'AnswerBit')
      
      In post-msgpack protocol, the logic is a bit different:
      
      Notify 	 add +1 to next msgcode
      Request  add +0 to next msgcode   (next msg is answer & should therefore be the same)
      Answer 	 add +2 to next msgcode
      
      So here we produce gaps after Request/Answer pairs.
      
      In the current version, this patch breaks the functionality of
      the pre-msgpack protocol, because for both protocols the same
      msgcodes are used. To still support 'N' we need to use different
      msgcodes depending on if 'N' or 'M' is used.
      d64f1baf
  8. 18 Mar, 2024 1 commit
  9. 15 Mar, 2024 2 commits
  10. 14 Mar, 2024 5 commits
    • Levin Zimmermann's avatar
      . · c19abad1
      Levin Zimmermann authored
      c19abad1
    • Levin Zimmermann's avatar
      fixup! X on msgpack support · 1282b808
      Levin Zimmermann authored
      This is a fix of 1cf2712f:
      'AppendString' is the equivalent to NEO/py 'NEO' encoding, not
      'AppendBytes'.
      1282b808
    • Levin Zimmermann's avatar
      go/neo/neonet: Fix segmentation violation in case handshake fails · 9501173e
      Levin Zimmermann authored
      Because 'err' was locally assigned inside the loop of 'DialLink' [1],
      it was always 'nil' as a function return value, even when
      'handshakeClient' actually returned an error. This lead to the
      unfortunate situation that the function sometimes returned 'link=nil'
      and 'err=nil', so that the function caller tried to access 'link'
      attributes which lead to 'runtime error: invalid memory address or nil
      pointer dereference'. This patch fixes this and now the function
      correctly returns an error if the dialing fails.
      
      [1] `peerConn, err := networker.Dial(ctx, addr)`
      9501173e
    • Levin Zimmermann's avatar
      . · 5e90fc24
      Levin Zimmermann authored
      5e90fc24
    • Julien Muchembled's avatar
      protocol: switch to msgpack for packet serialization · ccd47c48
      Julien Muchembled authored
      Not only for performance reasons (at least 3% faster) but also because of
      several ugly things in the way packets were defined:
      - packet field names, which are only documentary; for roots fields,
        they even just duplicate the packet names
      - a lot of repetitions for packet names, and even confusion between the name
        of the packet definition and the name of the actual notify/request packet
      - the need to implement field types for anything, like PByte to support new
        compression formats, since PBoolean is not enough
      
      neo/lib/protocol.py is now much smaller.
      ccd47c48
  11. 02 Feb, 2024 3 commits
  12. 29 Jan, 2024 8 commits
  13. 22 Aug, 2023 1 commit
  14. 02 Aug, 2023 7 commits
    • Levin Zimmermann's avatar
      X: Teach NEO/go to handle multiple master nodes · db81e0de
      Levin Zimmermann authored
      See kirr/neo!2 for discussion,
      context and details.
      
      /reviewed-by @kirr
      
      * t-with-multiple-master-nodes:
        fixup! client_test: Add nmaster={1,2} to test matrix
        fixup! client_test: Support test cluster /w >1 master
        fixup! TalkMaster: Switch master if dialed M is secondary
        fixup! Node: Add support for NEO cluster with > 1 master
        fixup! Dial: Catch NotPrimaryMaster & return custom error
        fixup! proto: Implement Error for NotPrimaryMaster
        fixup! proto.NotPrimaryMaster: Fix .Primary data type (2)
        fixup! proto.NotPrimaryMaster: Fix .Primary data type (1)
        client_test: Add nmaster={1,2} to test matrix
        client_test: Support test cluster /w >1 master
        proto.NotPrimaryMaster: Fix .Primary data type
        TalkMaster: Switch master if dialed M is secondary
        Dial: Catch NotPrimaryMaster & return custom error
        proto: Implement Error for NotPrimaryMaster
        openClientByURL: Fix for >1 master (split URL host)
        Client.URL: Fix incomplete URL if > 1 master nodes
        Node: Add support for NEO cluster with > 1 master
      db81e0de
    • Kirill Smelkov's avatar
      fixup! client_test: Add nmaster={1,2} to test matrix · 4605cba1
      Kirill Smelkov authored
      Actually do test nmaster=2 case.
      4605cba1
    • Kirill Smelkov's avatar
      fixup! client_test: Support test cluster /w >1 master · f96c8ccd
      Kirill Smelkov authored
      - show all options in error context and ran test kind
      
      - skip nmaster > 1 for NEO/go as that is currently not implemented on NEO/go server
      
      - use json for interacting with runneo.py, so that we can use
        whatever builtin type for any argument without hardcoding ad-hoc
        handling of specific arguments inside runneo.py.
      
      - adjust comments + cosmetics
      f96c8ccd
    • Kirill Smelkov's avatar
      fixup! TalkMaster: Switch master if dialed M is secondary · 3c37c0fc
      Kirill Smelkov authored
      - validate received NotPrimaryMaster
      - use Address.String() instead of printf with format that works only for ipv6
      - add some logging, comments and TODO
      3c37c0fc
    • Kirill Smelkov's avatar
      fixup! Node: Add support for NEO cluster with > 1 master · 354887e3
      Kirill Smelkov authored
      - no need to keep Node.MasterAddr anymore - the address of current PM is
        managed by TalkMaster and is provided as part of operational context
        to user functions that TalkMaster runs.
      
      - correct docstrings.
      
      - cosmetics.
      354887e3
    • Kirill Smelkov's avatar
      fixup! Dial: Catch NotPrimaryMaster & return custom error · 20f76288
      Kirill Smelkov authored
      Expect NotPrimaryMaster only if we are trying to connect to a master.
      20f76288
    • Kirill Smelkov's avatar
      fixup! proto: Implement Error for NotPrimaryMaster · 7efe5d04
      Kirill Smelkov authored
      Provide details in the error message.
      7efe5d04