Port from ZODB 3.2.

Stop believing the maximum oid cached in a FileStorage's .index file.

This is a critical bugfix, although the problems it addresses are
(a) rare; and, (b) not entirely fixed yet (more checkins to come).

The true max oid is found efficiently now by exploiting the recently-added
fsIndex.maxKey() method (which was, of course, added for this purpose).

Also fix that the .index file could get updated on disk when the
FileStorage was opened in read-only mode.  The code was trying to prevent
this, but missed the most obvious rewrite path.

Incidentally improved many obsolete and/or incorrect comments.

Stop believing the maximum oid cached in a FileStorage's .index file.

This is a critical bugfix, although the problems it addresses are
(a) rare; and, (b) not entirely fixed yet (more checkins to come).

The true max oid is found efficiently now by exploiting the recently-added
fsIndex.maxKey() method (which was, of course, added for this purpose).

Also fix that the .index file could get updated on disk when the
FileStorage was opened in read-only mode.  The code was trying to prevent
this, but missed the most obvious rewrite path.

Incidentally improved many obsolete and/or incorrect comments.

Port from ZODB 3.2.

Give fsIndex an efficient maxKey() implementation.

This will (in a later checkin) be used to give FileStorage an "obviously
correct" way to determine the largest oid used in an .fs.index file.

Give fsIndex an efficient maxKey() implementation.

This will (in a later checkin) be used to give FileStorage an "obviously
correct" way to determine the largest oid used in an .fs.index file.

Port from ZODB 3.2.

Note that cPickleCache.c must be recompiled else the new test will fail.

Change the exception raised when an attempt is made to add two objects to
the cache with the same oid.  The former messsage didn't make sense.

Add a test to verify that this exception does get raised, and that the
message given is the intended one.

This is the first of a series of checkins, to fix critical bugs where
ZODB can in fact raise this exception in rare, but normal, use cases.

Note that cPickleCache.c must be recompiled else the new test will fail.

Change the exception raised when an attempt is made to add two objects to
the cache with the same oid.  The former messsage didn't make sense.

Add a test to verify that this exception does get raised, and that the
message given is the intended one.

This is the first of a series of checkins, to fix critical bugs where
ZODB can in fact raise this exception in rare, but normal, use cases.

setUp():  Use a much smaller (than default) ZEO client cache.

The current default is 200MB, and at least testSerialization reads
the entire cache file into one giant string.  That grossly boosts
the process highwater mark when running the tests.  Windows reclaims
the memory after, but other platforms may not.

Cool:  testSerialization was one of the slowest tests before, but
is 100x faster now.

The current default is 200MB, and at least testSerialization reads
the entire cache file into one giant string.  That grossly boosts
the process highwater mark when running the tests.  Windows reclaims
the memory after, but other platforms may not.

Cool:  testSerialization was one of the slowest tests before, but
is 100x faster now.

Side-port changes checked in from a wrong project (Zope trunk):

r28991 | jim | 2005-01-31 10:45:46 -0500 (Mon, 31 Jan 2005) | 5 lines
...
Renamed the Zope package to Zope2

Import of the Zope package is now deprecated, but will
be supported until Zope 2.11.

r28991 | jim | 2005-01-31 10:45:46 -0500 (Mon, 31 Jan 2005) | 5 lines
...
Renamed the Zope package to Zope2

Import of the Zope package is now deprecated, but will
be supported until Zope 2.11.

Forward port from ZODB 3.2.

Connection.__init__():  Python 2.4 added a new gimmick to asyncore (a
._map attribute on asyncore.dispatcher instances) that breaks the
delicate ZEO startup dance.  Repaired that.

Connection.__init__():  Python 2.4 added a new gimmick to asyncore (a
._map attribute on asyncore.dispatcher instances) that breaks the
delicate ZEO startup dance.  Repaired that.

Port from ZODB 3.2.

Fixed several thread and asyncore races in ZEO's connection dance.

ZEO/tests/ConnectionTests.py
    The pollUp() and pollDown() methods were pure busy loops whenever
    the asyncore socket map was empty, and at least on some flavors of
    Linux that starved the other thread(s) trying to do real work.
    This grossly increased the time needed to run tests using these, and
    sometimes caused bogus "timed out" test failures.

ZEO/zrpc/client.py
ZEO/zrpc/connection.py
    Renamed class ManagedConnection to ManagedClientConnection, for clarity.

    Moved the comment block about protocol negotiation from the guts of
    ManagedClientConnection to before the Connection base class -- the
    Connection constructor can't be understood without this context.  Added
    more words about the delicate protocol negotiation dance.

    Connection class:  made this an abstract base clase.  Derived classes
    _must_ implement the handshake() method.  There was really nothing in
    common between server and client wrt what handshake() needs to do, and
    it was confusing for one of them to use the base class handshake() while
    the other replaced handshake() completely.

    Connection.__init__:  It isn't safe to register with asyncore's socket
    map before special-casing for the first (protocol handshake) message is
    set up.  Repaired that.  Also removed the pointless "optionalness" of
    the optional arguments.

    ManagedClientConnection.__init__:  Added machinery to set up correct
    (thread-safe) message queueing.  There was an unrepairable hole before,
    in the transition between "I'm queueing msgs waiting for the server
    handshake" and "I'm done queueing messages":  it was impossible to know
    whether any calls to the client's "queue a message" method were in
    progress (in other threads), so impossible to make the transition safely
    in all cases.  The client had to grow its own message_output() method,
    with a mutex protecting the transition from thread races.

    Changed zrpc-conn log messages to include "(S)" for server-side or
    "(C)" for client-side.  This is especially helpful for figuring out
    logs produced while running the test suite (the server and client
    log messages end up in the same file then).

Fixed several thread and asyncore races in ZEO's connection dance.

ZEO/tests/ConnectionTests.py
    The pollUp() and pollDown() methods were pure busy loops whenever
    the asyncore socket map was empty, and at least on some flavors of
    Linux that starved the other thread(s) trying to do real work.
    This grossly increased the time needed to run tests using these, and
    sometimes caused bogus "timed out" test failures.

ZEO/zrpc/client.py
ZEO/zrpc/connection.py
    Renamed class ManagedConnection to ManagedClientConnection, for clarity.

    Moved the comment block about protocol negotiation from the guts of
    ManagedClientConnection to before the Connection base class -- the
    Connection constructor can't be understood without this context.  Added
    more words about the delicate protocol negotiation dance.

    Connection class:  made this an abstract base clase.  Derived classes
    _must_ implement the handshake() method.  There was really nothing in
    common between server and client wrt what handshake() needs to do, and
    it was confusing for one of them to use the base class handshake() while
    the other replaced handshake() completely.

    Connection.__init__:  It isn't safe to register with asyncore's socket
    map before special-casing for the first (protocol handshake) message is
    set up.  Repaired that.  Also removed the pointless "optionalness" of
    the optional arguments.

    ManagedClientConnection.__init__:  Added machinery to set up correct
    (thread-safe) message queueing.  There was an unrepairable hole before,
    in the transition between "I'm queueing msgs waiting for the server
    handshake" and "I'm done queueing messages":  it was impossible to know
    whether any calls to the client's "queue a message" method were in
    progress (in other threads), so impossible to make the transition safely
    in all cases.  The client had to grow its own message_output() method,
    with a mutex protecting the transition from thread races.

    Changed zrpc-conn log messages to include "(S)" for server-side or
    "(C)" for client-side.  This is especially helpful for figuring out
    logs produced while running the test suite (the server and client
    log messages end up in the same file then).

Most frequently "<unknown>" classes appears for objects changed
later in the same transaction with changes of an interesting objects
so we save data records for all objects changed in the transaction
and then check them for references.

bugfixes have piled up since this was last done.

bugfixes have piled up since this was last done.

Merge rev 28529, applied from a wrong project (Zope trunk).

ZODB cache size is an integer (number of objects), not a number of bytes.

ZODB cache size is an integer (number of objects), not a number of bytes.

Merge rev 28470 from Zope3 trunk.

Typo repair checked in from a wrong project.

Typo repair checked in from a wrong project.

Repaired 3.3.1a1 release name.

Forward-port from ZODB 3.2.

Collector 1503:  excessive logging.

ClientStorage._wait_sync():  Don't log more than one "waiting for cache
verification to finish" message per 5 minutes.

Collector 1503:  excessive logging.

ClientStorage._wait_sync():  Don't log more than one "waiting for cache
verification to finish" message per 5 minutes.

Critical bugfix for invalidations during ZEO cache verification.

ClientCache.invalidate():

- This was written not to expect tid=None, which is what happens
  during cache verification, although it didn't catch the tid=None
  case and went on doing really strange things then.  Rewrote so
  that tid=None means "forget everything I think I know about
  the oid".  It was kind-of performing an invalidation wrt the
  in-memory data structures in this case, but leaving the
  invalidated data for oid looking as if it were still current on
  disk.  That was accidental, depending on what various other
  parts of the code thought "None" meant (primarily, an end_tid
  of None is generally taken here to mean "the data is current").
  
- Also changed the "and we already have version data for oid" case
  to a "forget everything I think I know about the oid" case.  The
  comments seemed to say that's what it thought it was doing, but
  the code actually left non-current revisions sitting around.  I
  really don't want to spend time trying to figure out what versions
  are supposed to do in all cases here, and nuking everything in
  this case is an obviously *safe* thing to do.

- Added more asserts, because the pre- and post-conditions here are
  delicate and non-obvious (indeed, that a new assert failed is how
  I stumbled into the invalidation problem to begin with).

FileCache.remove():  Changed to remove the key from self.key2entry.
While self.filemap keeps an in-memory map of the entire disk file,
there's no need to keep info in key2entry for keys that no longer
exist in the disk file.  I'm not sure a visible bug followed from
this (but not sure one didn't either!), but it at best wasted
memory, and violated a documented invariant.

ConnectionTests.py:  New test checkVerificationInvalidationPersists().
Before the patch, this provokes a persistent ZEO cache into
delivering stale object state.  Of course the test passes after
this patch.

stats.py:  Added a trace code meaning "invalidation hit, and the
object state was discarded".  This is what happens now for
invalidations produced by verification.  Because of MVCC, that's
really not needed *except* for the invalidations produced by
startup verification (invalidations normally just set a new
ending tid on the validity range for what was current data; but
validity-range info isn't available from verification invalidations).

ClientCache.invalidate():

- This was written not to expect tid=None, which is what happens
  during cache verification, although it didn't catch the tid=None
  case and went on doing really strange things then.  Rewrote so
  that tid=None means "forget everything I think I know about
  the oid".  It was kind-of performing an invalidation wrt the
  in-memory data structures in this case, but leaving the
  invalidated data for oid looking as if it were still current on
  disk.  That was accidental, depending on what various other
  parts of the code thought "None" meant (primarily, an end_tid
  of None is generally taken here to mean "the data is current").
  
- Also changed the "and we already have version data for oid" case
  to a "forget everything I think I know about the oid" case.  The
  comments seemed to say that's what it thought it was doing, but
  the code actually left non-current revisions sitting around.  I
  really don't want to spend time trying to figure out what versions
  are supposed to do in all cases here, and nuking everything in
  this case is an obviously *safe* thing to do.

- Added more asserts, because the pre- and post-conditions here are
  delicate and non-obvious (indeed, that a new assert failed is how
  I stumbled into the invalidation problem to begin with).

FileCache.remove():  Changed to remove the key from self.key2entry.
While self.filemap keeps an in-memory map of the entire disk file,
there's no need to keep info in key2entry for keys that no longer
exist in the disk file.  I'm not sure a visible bug followed from
this (but not sure one didn't either!), but it at best wasted
memory, and violated a documented invariant.

ConnectionTests.py:  New test checkVerificationInvalidationPersists().
Before the patch, this provokes a persistent ZEO cache into
delivering stale object state.  Of course the test passes after
this patch.

stats.py:  Added a trace code meaning "invalidation hit, and the
object state was discarded".  This is what happens now for
invalidations produced by verification.  Because of MVCC, that's
really not needed *except* for the invalidations produced by
startup verification (invalidations normally just set a new
ending tid on the validity range for what was current data; but
validity-range info isn't available from verification invalidations).

Collector 1536:  ClientStorage.__init__ ignores cache_size.

Also split test checkRollover into two tests,
checkDisconnectedCacheWorks (persistent ZEO cache large enough
to satisfy all requests) and checkDisconnectedCacheFails (cache
too small to satisfy all requests while disconnected).

Also split test checkRollover into two tests,
checkDisconnectedCacheWorks (persistent ZEO cache large enough
to satisfy all requests) and checkDisconnectedCacheFails (cache
too small to satisfy all requests while disconnected).

More cleanup, and a new problem.  Notable:

Object.serialize_header():  Stopped stuffing the version string into this
notion of "a header".  Couldn't see any reason for it, and it was
surprising.

ClientCache.invalidate():  Added some asserts to clarify intended
preconditions.  Alas, my guess that it was intended that the passed-in
tid always be greater than the current tid turned out to trigger
errors.  This gets complicated, and hasn't been resolved yet.  The
problem is that the passed-in tid can be (in fact, always is) None
during cache verification.  The code here *really* doesn't seem to
be expecting that, and the on-disk cache file is clearly left in a
wrong state (if we repopulated the cache from the disk file, the
object would not look invalidated any more, it would look current).
Dumped in a giant stack trace so I don't forget this; sent email to
Jeremy asking if he recalls what the real intent was.

Object.serialize_header():  Stopped stuffing the version string into this
notion of "a header".  Couldn't see any reason for it, and it was
surprising.

ClientCache.invalidate():  Added some asserts to clarify intended
preconditions.  Alas, my guess that it was intended that the passed-in
tid always be greater than the current tid turned out to trigger
errors.  This gets complicated, and hasn't been resolved yet.  The
problem is that the passed-in tid can be (in fact, always is) None
during cache verification.  The code here *really* doesn't seem to
be expecting that, and the on-disk cache file is clearly left in a
wrong state (if we repopulated the cache from the disk file, the
object would not look invalidated any more, it would look current).
Dumped in a giant stack trace so I don't forget this; sent email to
Jeremy asking if he recalls what the real intent was.