When a node tries to connect to another node it initially sends a
'RequestIdentification' packet. The other node can either reply with
'AcceptIdentification' or in case of a secondary master with
'NotPrimaryMaster'.

In the second case the message id differs from the initial requests
message id. This makes it difficult in a multi-threaded implementation
to proceed this answer: due to the different msg/connection - id the
multi-threaded implementation tries to proceed this incoming message in
a different thread, while the requesting thread waits forever for its
peers reply.

The most straightforward solution is to use the same connection - id for
both possible answers to the 'RequestIdentification' packet. This
doesn't break given NEO/py implementation and is only a small patch.
A workaround in a multi-threaded implementation on the other hand
seems to be much more complicated and time-consuming. Finally it also makes
sense semantically, because "Message IDs are used to identify response
packets" and in the given context the 'NotPrimaryMaster' *is* the de facto
response of a 'RequestIdentification'.

It's been many years we don't get 'array' objects, no idea when exactly.

Like commit 243c1a0f
("sqlite: optimize storage of metadata"), the fake changes in test
data are because we don't force upgrade for this optimization.

Starting from zodbpickle 2 its binary class does not allow users to set
arbitrary attributes and so

	binary._pack = bytes.__str__

fails with

	TypeError: can't set attributes of built-in/extension type 'zodbpickle.binary'

-> Fix it by explicitly checking for binary type on encoding instead of
setting binary._pack

See nexedi/slapos@27f574bc for pre-history.

/cc @jerome

Traceback (most recent call last):
  ...
  File ".../neo/lib/handler.py", line 75, in dispatch
    method(conn, *args, **kw)
  File ".../neo/admin/handler.py", line 174, in wrapper
    return func(self, name, *args, **kw)
  File ".../neo/admin/handler.py", line 190, in notifyMonitorInformation
    self.app.updateMonitorInformation(name, **info)
  File ".../neo/admin/app.py", line 290, in updateMonitorInformation
    self._notify(self.operational)
  File ".../neo/admin/app.py", line 315, in _notify
    body += '', name, '    ' + backup.formatSummary(upstream)[1]
  File ".../neo/admin/app.py", line 83, in formatSummary
    tid = self.ltid
AttributeError: 'Backup' object has no attribute 'ltid'

The purpose of suppress_ragged_eofs=False was to micro-optimize the
normal case: when there's no EOF.

But commit 061cd5d8 showed that this
option only turns ragged EOF into an exception. It may be easier for
alternate NEO implementations to close the SSL connection properly. Or
the performance benefit was not worth the risk to freeze a NEO process.

Testing NEO/go client wrt NEO/py server revealed a bug in NEO/py SSL
handling: proper non-ragged EOF from a peer is ignored, and so leads to
hang in infinite loop inside _SSL.receive with read_buf memory growing
indefinitely. Details are below:

NEO/py wraps raw sockets with

	ssl.wrap_socket(suppress_ragged_eofs=False)

which instructs SSL layer to convert unexpected EOF when receiving a TLS
record into SSLEOFError exception. However when remote peer properly
closes its side of the connection, socket.read() still returns b'' to
report non-ragged regular EOF:

https://github.com/python/cpython/blob/v2.7.18/Lib/ssl.py#L630-L650

The code was handling SSLEOFError but not b'' return from socket recv.
Thus after NEO/go client was disconnecting and properly closing its side
of the connection, the code started to loop indefinitely in _SSL.receive
under `while 1` with  b'' returned by self.socket.recv() appended to
read_buf again and again.

-> Fix it by detecting non-ragged EOF as well and, similarly to how
SSLEOFError is handled, converting them into self._error('recv', None).

See merge request nexedi/neoppod!17

The scenario that was described in comments was meaningless
because S1 never goes out-of-date.

For the master, the purpose of -m/--masters is to specify addresses
of other master nodes, since its own address is already known via
-b/--bind. Therefore, an empty value for -m/--masters is valid.
The user remains free to repeat the -b value in -m.

More generally, a node may choose to only specify master addresses
via -D/--dynamic-master-list, so the check that at least one master
address is specified is moved where the NodeManager is expected to be
initialized.

The time complexity of previous one was too bad. With several tens of
concurrent transactions, we saw commits take minutes to complete and
the whole application looked frozen.

This new algorithm is much simpler. Instead of asking the oldest
transaction to somewhat restart (we used the "rebase" term because
the concept was similar to what git-rebase does), the storage gives
it priority and the newest is asked to relock (this request is ignored
if vote already happened, which means there was actually no deadlock).

testLocklessWriteDuringConflictResolution was initially more complex
because Transaction.written (client) ignored KeyError (which is not the
case anymore since commit 8ef1ddba).

This is all the more important for RocksDB that it wants to keep all
transaction work in RAM.

Once we had to truncate 40% of a 1TB MyRocks DB with 24 partitions,
4 being processed in parallel. Even when committing between partitions,
MariaDB used up to 200 GB. Without the commit, 1TB RAM would not have
been enough.

The default value is quickly exceeded when truncating a DB.
Obviously, you may need a lot of RAM.

Resetting a storage node could mark all TEST log entries as being
emitted by this storage node. For example:

16:18:12.9114 S2         #0x0007 AskStoreObject                 > S1 (...)

======================================================================
FAIL: check_tid_ordering_w_commit (neo.tests.zodb.testBasic.BasicTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "ZODB/tests/BasicStorage.py", line 397, in check_tid_ordering_w_commit
    self.assertEqual(results.pop('lastTransaction'), tids[1])
  File "neo/tests/__init__.py", line 301, in assertEqual
    return super(NeoTestBase, self).assertEqual(first, second, msg=msg)
failureException: '\x03\xd8\x85H\xbffp\xbb' != '\x03\xd8\x85H\xbfs\x0b\xdd'

This requires ZODB >= 5.6.0