Pre-mortem data:
Traceback (most recent call last):
File "neo/master/app.py", line 172, in run
self._run()
File "neo/master/app.py", line 180, in _run
self.listening_conn = ListeningConnection(self, None, self.server)
File "neo/lib/connection.py", line 298, in __init__
connector.makeListeningConnection()
File "neo/lib/connector.py", line 133, in makeListeningConnection
self._error('listen', e)
File "neo/lib/connector.py", line 93, in _error
raise ConnectorException
ConnectorException
Traceback (most recent call last):
  File "neomaster", line 50, in <module>
    sys.exit(neo.scripts.neomaster.main())
  File "neo/scripts/neomaster.py", line 31, in main
    app.run()
  File "neo/master/app.py", line 175, in run
    self.log()
  File "neo/master/app.py", line 167, in log
    if self.pt is not None:
AttributeError: 'Application' object has no attribute 'pt'

This is still pack without garbage collection, and without deleting
any transaction metadata ('trans' table).

Partial pack means that the client can take a list of oids: only these
oids will be packed. No API is defined yet at IStorage level.

Storage nodes pack in background, independently from other storage
nodes, partition by partition, and calling IStorage.pack() returns
immediately (though internally, NEO does have a mechanism to wait
until it's done, which can be required for some ZODB unit tests).

This new implementation also introduces the concept of signing pack
orders. The idea is that calling IStorage.pack() only records a pack
order in the database, that can be reviewed/approved/rejected using
an UI that is left to be done. For the moment, pack orders are
automatically approved (by the master).

Internally, pack orders are stored as extra metadata of a transaction.
IOW, IStorage.pack() implies the commit of an (empty) transaction.

IStorage.pack() can be called without waiting for the previous one
to be completed. Pack orders processed in the same order as they are
requested:
- an unsigned pack order blocks the processing of any newer pack order;
- rejected pack order are ignored.

Approving a pack order also triggers pack on backup clusters.
That's the simplest way to have everything consistent.
Maybe later we could identify scenarios where it would be ok
to unsign pack orders during asynchronous replication.

The feature to check replicas is marked as experimental because it is
not aware of differences that can happen during pack operations.
_______________________________________________________________________

About concurrency within the storage node, a first implementation
extended what was done to delete partitions in background (see
previous commit). But here, the job can't be easily split in splices
that are never too big:
- it's simpler to never split the processing of an oid but this can
  freeze the application for a long time when packing an oid that was
  modified many times (e.g. 30 min for an oid with 20 millions
  historical records);
- then an attempt so that an oid can be processed in several times was
  inefficient, maybe due to a limit in RocksDB (packing the oid in the
  above example would take days during which NEO is significantly
  slower).

So background database jobs were moved to a separate thread, using a
separate connection to the underlying database. This is obviously
only useful for the MySQL backend. In order to share as much code as
possible between backends, SQLite also does the work in a separate
thread but sharing the main connection instead of opening a separate
one (so such backend would not be suited in the above example).

But deleting raw data with a secondary connection is not possible
without fsyncing too often (or transaction isolation issues...): these
deletions are deferred by recording them in a new table, which is
processed later with the main connection. This is not so bad because
the actual deletion of raw data is usually more efficient this way
(more sequential IO).

Here are a few numbers:
- without load: 10h45 (12h for the first reimplementation)
- with a load that normally takes 6h58:
  - load: 7h33 (so 8.4% slower)
  - pack: 15h36 (+4h51)

As explained above, the pack of a partition is split in 2 steps:
- the longest one (here 78% without load) should have negligible
  peformance impact on the application because the work is done in a
  separate thread with a secondary connection, and also with something
  to minimize GIL impact by prioritizing the main thread;
- the shortest one (22%) to process the deferred deletions,
  with even lower priority than replication: it tries to split
  the work in tasks that take ~10ms.

This is implemented using the same concurrency mechanism as for the
replication: the work is split in slices that should be small enough
to avoid slowing down network requests significantly.

undone_data_tid can't be equal to a TTID.

It has never been enabled and the code to drop partitions will be
changed in a way that only 'trans' may still benefit of partitioning.
We'll see in the future if we have cases where 'trans' is too big to
delete all rows (of a given partition) in a single query.

- When undoing current record, fix:
  - crash of storage nodes that don't have the undo data (non-readable cells);
  - and conflict resolution.
- Fix undo deduplication in replication when NEO deduplication is disabled.
- client: minor fixes in undo() about concurrent storage disconnections
  and PT updates.

Found by running testPruneOrphan many times. Once I even got:

  SystemError: NULL result without error in PyObject_Call

The reverts a wrong change in commit 30a02bdc
("importer: new option to write back new transactions to the source database").

It's been many years we don't get 'array' objects, no idea when exactly.

Like commit 243c1a0f
("sqlite: optimize storage of metadata"), the fake changes in test
data are because we don't force upgrade for this optimization.

Starting from zodbpickle 2 its binary class does not allow users to set
arbitrary attributes and so

	binary._pack = bytes.__str__

fails with

	TypeError: can't set attributes of built-in/extension type 'zodbpickle.binary'

-> Fix it by explicitly checking for binary type on encoding instead of
setting binary._pack

See nexedi/slapos@27f574bc for pre-history.

/cc @jerome

Traceback (most recent call last):
  ...
  File ".../neo/lib/handler.py", line 75, in dispatch
    method(conn, *args, **kw)
  File ".../neo/admin/handler.py", line 174, in wrapper
    return func(self, name, *args, **kw)
  File ".../neo/admin/handler.py", line 190, in notifyMonitorInformation
    self.app.updateMonitorInformation(name, **info)
  File ".../neo/admin/app.py", line 290, in updateMonitorInformation
    self._notify(self.operational)
  File ".../neo/admin/app.py", line 315, in _notify
    body += '', name, '    ' + backup.formatSummary(upstream)[1]
  File ".../neo/admin/app.py", line 83, in formatSummary
    tid = self.ltid
AttributeError: 'Backup' object has no attribute 'ltid'

The purpose of suppress_ragged_eofs=False was to micro-optimize the
normal case: when there's no EOF.

But commit 061cd5d8 showed that this
option only turns ragged EOF into an exception. It may be easier for
alternate NEO implementations to close the SSL connection properly. Or
the performance benefit was not worth the risk to freeze a NEO process.

Testing NEO/go client wrt NEO/py server revealed a bug in NEO/py SSL
handling: proper non-ragged EOF from a peer is ignored, and so leads to
hang in infinite loop inside _SSL.receive with read_buf memory growing
indefinitely. Details are below:

NEO/py wraps raw sockets with

	ssl.wrap_socket(suppress_ragged_eofs=False)

which instructs SSL layer to convert unexpected EOF when receiving a TLS
record into SSLEOFError exception. However when remote peer properly
closes its side of the connection, socket.read() still returns b'' to
report non-ragged regular EOF:

https://github.com/python/cpython/blob/v2.7.18/Lib/ssl.py#L630-L650

The code was handling SSLEOFError but not b'' return from socket recv.
Thus after NEO/go client was disconnecting and properly closing its side
of the connection, the code started to loop indefinitely in _SSL.receive
under `while 1` with  b'' returned by self.socket.recv() appended to
read_buf again and again.

-> Fix it by detecting non-ragged EOF as well and, similarly to how
SSLEOFError is handled, converting them into self._error('recv', None).

See merge request !17