- 18 Jan, 2021 1 commit
-
-
Kirill Smelkov authored
Package task provides primitives to track tasks via contexts.
-
- 15 Jan, 2021 20 commits
-
-
Kirill Smelkov authored
NEO/go server accepts preferred encoding of client from start, but NEO/py server implements either N or M encoding and disconnects client silently if handshake is not exactly what is expected. However we can teach Dial to retry with different preferred options and this way when connection to a NEO/py server, it will essentially autodetect which encoding is used and accepted by remote peer.
-
Kirill Smelkov authored
This patch adds support for serializing packet frames with M encoding on the wire. To do so it follows rules defined in nexedi/neoppod@9d0bf97a ( nexedi/neoppod!11 ) Server handshake is reworked to autodetect client's preferred encoding. Client always prefers 'N' for now.
-
Kirill Smelkov authored
-
Kirill Smelkov authored
This patch adds proto.Encoding('M') and teaches it to encode/decode messages via MessagePack by the rules defined in nexedi/neoppod@9d0bf97a ( nexedi/neoppod!11 ) It only adds support for messages serialization, without changing proto.go to match changes in e.g. enums reordering, and without adding support for MessagePack at link layer in neonet. M-encoding was only tested for NEO/go-NEO/go, and was not yet tested for NEO/go-NEO/py interoperation. There will be likely small mistakes discovered on my side that should be hopefully easy to fix step by step once we get to that phase.
-
Kirill Smelkov authored
It complements github.com/tinylib/msgp with things that neo/proto needs to encode/decode messages.
-
Kirill Smelkov authored
It is noop for 'N' encoding, but will be non-empty for MessagePack.
-
Kirill Smelkov authored
genSliceHead and genMapHead will be reused for MsgPack encoding. Changes in generated code are mostly due to move of where things are typecasted to e.g. uint32 without any change in semantic.
-
Kirill Smelkov authored
For decode-family of functions protogen.go sticks to convention to name target variable to where to assign the result as "assignto", not "path". No change in generated code.
-
Kirill Smelkov authored
Encoding specifies a way to encode/decode NEO messages and packets. Current way of how messages were encoded is called to be 'N' encoding. This patch: - adds proto.Encoding type - changes MsgEncode and MsgDecode to be methods of Encoding - renames thigs that are specific to 'N' encoding to have 'N' suffix - changes tests to run a testcase agains vector of provided encodings. That vector is currently only ['N'].
-
Kirill Smelkov authored
And provide only single top-level entry-points to encode/decode messages. As of now the entry points are just plain forwarding, but with introducing of msgpack and encodings, they will take into account through which encoding a message has to be encoded/decoded.
-
Kirill Smelkov authored
- Regrouping messages would change their message codes, thus introducing backward compatibility breakage. - The protocol itself is currently kind-of documented in separate document here: https://neo.nexedi.com/P-NEO-Protocol.Specification.2019?portal_skin=CI_slideshow
-
Kirill Smelkov authored
It is internal detail that ideally should not get outside of proto package. Don't clutter documentation of all messages with it. -> For now access message code via proto.MsgCode(msg) from outside.
-
Kirill Smelkov authored
Generating code through sizer can already give answer if a type is fixed size or not. And in practice we don't care at all if it will be a bit slower compared to previous explicit type switch. For current code it is just a small cleanup, and it will be handy to support messagepack encoding as well. No change in generated code.
-
Kirill Smelkov authored
Not used yet, but they will be needed for msgpack codec.
-
Kirill Smelkov authored
Besides proto.customCodec we will need to lookup more objects.
-
Kirill Smelkov authored
We previously used criteria that sizeof(typ.Elem()) == 1, but it would also trigger on e.g. `struct { x byte }` which we would not want to consider as byte array. -> Clarify the test to explicitly test for typ.Elem() == byte. No change in generated code.
-
Kirill Smelkov authored
Without the fix any pointer field of a struct would be silently skipped without any serialization code generated for it. With the patch default case triggers in and reports detail where such type is and in which struct/field it is used. Besides *mem.Buf - that is handled specially - we don't have pointers in proto.go, so this currently does not affect generated code at all. Noticed accidentally.
-
Kirill Smelkov authored
Without that, when lonet will be used in tests, port won't be correctly extracted from an address.
-
Kirill Smelkov authored
With upcoming msgpack encoding packets might be with data < PktHeaderSize. Adjust pktAlloc to unconditionally provide the invariant.
-
Kirill Smelkov authored
Preparatory step before we introduce support for msgpack encoding.
-
- 14 Jan, 2021 4 commits
-
-
Kirill Smelkov authored
It is not used anywhere and would have to be reworked with upcoming introduction of msgpack encoding. -> Simply remove that.
-
Kirill Smelkov authored
Just renaming.
-
Kirill Smelkov authored
This represents a neonet testing environment and in the future will be used to run tests under different protocol versions, protocol encodings, etc. For now it is noop wrapper around testing.T + t.bin that wraps []byte and will be amended to return something different depending on t's encoding.
-
Kirill Smelkov authored
Previously we were doing handshake symmetrically: both client and server were transmitting hello and receiving peer's hello simultaneously. However this does not allow server to adjust its behaviour depending on which client (protocol version, protocol encoding, ...) is connecting to it. -> Rework handshake so that client always sends its hello first, and only then the server side replies. This matches actual NEO/py behaviour: https://lab.nexedi.com/nexedi/neoppod/blob/v1.12-67-g261dd4b4/neo/lib/connector.py#L293-294 even though the "NEO protocol" states that Handshake transmissions are not ordered with respect to each other and can go in parallel. ( https://neo.nexedi.com/P-NEO-Protocol.Specification.2019?portal_skin=CI_slideshow#/9/2 ) If I recall correctly that sentence was authored by me in 2018 based on previous understanding of should-be full symmetry in-between client and server. However soon we are going to teach server sides to autodetect client encoding and adjust server to talk to client via its preferred way. This needs handshake for client and server to be differentiated. The protocol needs to be adjusted as well. However I'm not sure it is going to happen...
-
- 13 Jan, 2021 1 commit
-
-
Kirill Smelkov authored
This will be handy to test underlying error for e.g. EOF with errors.Cause or errors.Is in case that underlying error is also error wrapper.
-
- 12 Jan, 2021 1 commit
-
-
Kirill Smelkov authored
We were returning packet with tail read for next packet data. Previously decoding was forgiving, but upcoming rework for msgpack support will add check to catch packets with overlong payload, e.g. === RUN TestEmptyDB/py/!ssl I: runneo.py: /tmp/neo214694750/1 !ssl: started master(s): 127.0.0.1:30621 127.0.0.1:42266 > 127.0.0.1:30621: .1 RequestIdentification &{CLIENT ?(0)0 1 ø [] []} 127.0.0.1:42266 < 127.0.0.1:30621: (N: decode header: len(payload) != msgLen) 00 00 00 01 80 01 00 00 00 09 00 f0 00 00 01 e0 00 00 01 00 00 00 00 00 06 00 00 00 58 41 d7 ff 69 82 0a 91 25 00 00 00 03 02 00 00 00 00 e0 00 00 01 02 41 d7 ff 69 82 0a 78 ff 01 00 00 00 09 31 32 37 2e 30 2e 30 2e 31 96 f7 00 00 00 01 02 41 d7 ff 69 81 c6 65 f5 00 00 00 00 09 31 32 37 2e 30 2e 30 2e 31 77 9d f0 00 00 01 02 ff ff ff ff ff ff ff ff 00 00 00 02 00 0a 00 00 00 19 00 00 00 00 00 00 00 01 00 00 00 00 00 00 00 01 00 00 00 01 00 00 00 01 01 -> Fix is: pkt.data should be data[:pktLen], not data[:n] since n is how much we have read at all.
-
- 29 Dec, 2020 3 commits
-
-
Kirill Smelkov authored
The utility to create Networker suitable for interoperating with nodes in a NEO cluster, be it TCP, TLS/TCP, TLS/lonet, etc. See added comments for details.
-
Kirill Smelkov authored
See ConfigForP2P comment for details.
-
Kirill Smelkov authored
xnet.Networker added Close recently and is now required to be explicitly closed by the user: go123@01a77bb9
-
- 18 Dec, 2020 1 commit
-
-
Kirill Smelkov authored
See go123@515a6d14
-
- 17 Dec, 2020 1 commit
-
-
Kirill Smelkov authored
See kirr/go123@3354b401 and kirr/go123@b03d65ff The wrapping logic in LinkListener goes away because Accept from xnet now supports cancellation.
-
- 30 Nov, 2020 4 commits
-
-
Kirill Smelkov authored
It was not the case with ZEO4 server because we did not had ZEO@bf80d23d yet.
-
Kirill Smelkov authored
For ZEO it is not strictly required, but for upcoming NEO for example NEOCluster needs to shutdown gracefully, else there are processes left for e.g. storage nodes and they dump somthing as below on the terminal after tests completion: === RUN TestLoad 2020/10/21 14:33:00 zodb: FIXME: open ../zodb/storage/fs1/testdata/1.fs: raw cache is not ready for invalidations -> NoCache forced === RUN TestLoad/py I: runneo.py: /tmp/neo445013868/1: Started master(s): 127.0.0.1:24661 WARNING: This is not the recommended way to import data to NEO: you should use the Importer backend instead. NEO also does not implement IStorageRestoreable interface, which means that undo information is not preserved when using this tool: conflict resolution could happen when undoing an old transaction. Migrating from ../zodb/storage/fs1/testdata/1.fs to 127.0.0.1:24661 Migration done in 0.19877 --- PASS: TestLoad (0.75s) --- PASS: TestLoad/py (0.74s) PASS ok lab.nexedi.com/kirr/neo/go/neo 0.749s (neo) (z-dev) (g.env) kirr@deco:~/src/neo/src/lab.nexedi.com/kirr/neo/go/neo$ Traceback (most recent call last): File "/home/kirr/src/neo/src/lab.nexedi.com/kirr/neo/neo/tests/functional/__init__.py", line 182, in start getattr(neo.scripts, command).main() File "/home/kirr/src/neo/src/lab.nexedi.com/kirr/neo/neo/scripts/neostorage.py", line 66, in main app.run() File "/home/kirr/src/neo/src/lab.nexedi.com/kirr/neo/neo/storage/app.py", line 147, in run self._run() File "/home/kirr/src/neo/src/lab.nexedi.com/kirr/neo/neo/storage/app.py", line 178, in _run self.doOperation() File "/home/kirr/src/neo/src/lab.nexedi.com/kirr/neo/neo/storage/app.py", line 266, in doOperation poll() File "/home/kirr/src/neo/src/lab.nexedi.com/kirr/neo/neo/storage/app.py", line 87, in _poll self.em.poll(1) File "/home/kirr/src/neo/src/lab.nexedi.com/kirr/neo/neo/lib/event.py", line 155, in poll self._poll(blocking) File "/home/kirr/src/neo/src/lab.nexedi.com/kirr/neo/neo/lib/event.py", line 253, in _poll timeout_object.onTimeout() File "/home/kirr/src/neo/src/lab.nexedi.com/kirr/neo/neo/lib/event.py", line 259, in onTimeout on_timeout() File "/home/kirr/src/neo/src/lab.nexedi.com/kirr/neo/neo/storage/database/manager.py", line 207, in _deferredCommit self.commit() File "/home/kirr/src/neo/src/lab.nexedi.com/kirr/neo/neo/storage/database/manager.py", line 193, in commit self._commit() File "/home/kirr/src/neo/src/lab.nexedi.com/kirr/neo/neo/storage/database/sqlite.py", line 90, in _commit retry_if_locked(self.conn.commit) File "/home/kirr/src/neo/src/lab.nexedi.com/kirr/neo/neo/storage/database/sqlite.py", line 45, in retry_if_locked return f(*args) OperationalError: disk I/O error
-
Kirill Smelkov authored
And use it to verify other storages.
-
Kirill Smelkov authored
See nexedi/nxdtest@53064e71
-
- 05 Nov, 2020 4 commits
-
-
Kirill Smelkov authored
Go through zrpc and add error context to every function that can create error. This is similar to what NEO does in neonet. The reason to do this was the following obscure error from wcfs: E1105 11:32:33.295497 24639 wcfs.go:2505] zwatch zeo://:28359: EOF which, after this patch, is now E1105 12:53:15.922052 30731 wcfs.go:2510] zwatch zeo://:27024: zlink 127.0.0.1:47768 - 127.0.0.1:27024: recvPkt: EOF
-
Kirill Smelkov authored
NEO was already doing this, but let's use xio.NoEOF everywhere for uniformity.
-
Kirill Smelkov authored
If remote peer closes the link half-way of packet data - turn EOF into ErrUnexpectedEOF.
-
Kirill Smelkov authored
We will need to use those utilities for ZEO and NEO.
-