If anything wrong happens after a transaction is locked and before the end of
onTransactionCommitted, recovery phase should be run again, so that the master
gets correct last tid.

Following patch by Vincent is an attempt to fix this:

--- a/neo/master/app.py
+++ b/neo/master/app.py
@@ -329,8 +329,8 @@ def playPrimaryRole(self):

         # recover the cluster status at startup
         try:
-            self.runManager(RecoveryManager)
             while True:
+                self.runManager(RecoveryManager)
                 self.runManager(VerificationManager)
                 try:
                     if self.backup_tid:
@@ -338,10 +338,6 @@ def playPrimaryRole(self):
                             raise RuntimeError("No upstream cluster to backup"
                                                " defined in configuration")
                         self.backup_app.provideService()
-                        # Reset connection with storages (and go through a
-                        # recovery phase) when leaving backup mode in order
-                        # to get correct last oid/tid.
-                        self.runManager(RecoveryManager)
                         continue
                     self.provideService()
                 except OperationFailure:

This should following random errors:

>   File "neo/lib/event.py", line 77, in unregister
>     self.epoll.unregister(fd)
> IOError: [Errno 2] No such file or directory

>   File "neo/tests/threaded/test.py", line 670, in testClientReconnection
>     c, = cluster.storage.nm.getClientList()
> ValueError: need more than 0 values to unpack

This implementation proper cache invalidation.

Connection to master is also made optional to load from storage nodes, as long
as partition table contains up-to-date data (which is anyway unlikely to change
when there is no master).

This is enough because on disconnection, the master already aborts all
transactions on its side.

This may help client to recover after an assertion failure.
For example, this should fix the following bug:

ERROR ZODB.Connection Couldn't load state for 0x13b6
Traceback (most recent call last):
  File "ZODB/Connection.py", line 851, in setstate
    self._setstate(obj)
  File "ZODB/Connection.py", line 901, in _setstate
    p, serial = self._storage.load(obj._p_oid, '')
  File "neo/client/Storage.py", line 85, in load
    return self.app.load(oid)[:2]
  File "neo/client/app.py", line 435, in load
    result = self._loadFromStorage(oid, tid, before_tid)
  File "neo/client/app.py", line 450, in _loadFromStorage
    for node, conn in self.cp.iterateForObject(oid, readable=True):
  File "neo/client/pool.py", line 130, in iterateForObject
    conn = getConnForNode(node)
  File "neo/client/pool.py", line 155, in getConnForNode
    conn = self._initNodeConnection(node)
  File "neo/client/pool.py", line 61, in _initNodeConnection
    connector=app.connector_handler(), dispatcher=app.dispatcher)
  File "neo/lib/connection.py", line 749, in __init__
    super(MTClientConnection, self).__init__(*args, **kwargs)
  File "neo/lib/connection.py", line 685, in __init__
    node.setConnection(self)
  File "neo/lib/node.py", line 119, in setConnection
    attributeTracker.whoSet(self, '_connection')
AssertionError

Also, saves one local variable assignment in "hit" code path.

Use a non-intrusive code profiling tool with NEO instead, like pprofile.

Also, drop extraneous parentheses in another set's creation.

Nodes are likely to be running, so filtering before sort is unlikely to
save time. Caller is likely to stop iterating after first yield connection
("load" case), so move filtering inside loop.

Also, document non-straightforward code.

Connection is more often established than not, so do a first lookup without
locking, and only acquire it if it misses. Then do a second lookup in case
another thread also established connection, and connect if it still misses.

We think there's still a bug, in ZODB.Connection or ZPublisher/Publish,
but since commit c277ed20, NEO does not
require anymore that Storage.sync() is called in order to take into account
invalidations that have already been received since the last tpc_finish.

IOW, this is not the place for such fix because it's independant to NEO.
ERP5 already ships it for ZEO (patch disabled by default).

Replace a per-call cost with a constant one.
Signed-off-by: Julien Muchembled <jm@nexedi.com>

loadBefore is more likely than other loads, as it is how repeatable-read is
implemented and the vast majority of accesses are scoped to a transaction.
So put it first in 3-branches "if" choosing between load/loadBefore/loadSerial
code paths.
next_serial is likely to not be found, because application will likely
operate on up-to-date objects. So a test is more efficient than a
try..except. Inline this test in return statement to avoid defining a local.

Next version is actually a metapackage depending on:
  persistent 4.0
  BTrees 4.0
  ZODB 4.0
  ZEO 4.0

Note that version '123dev' is considered lower than '123'

Update (only) user.

This fixes a bug causing a crash during tpc_finish phase when a storage
involved in a transaction does not receive any object, but receives at
least one CheckCurrentSerial request: no transaction was registered, and
storage would fail to lock transaction when requested by master during
tpc_finish phase.