Forward-port from ZODB 3.2.

Collector 1503:  excessive logging.

ClientStorage._wait_sync():  Don't log more than one "waiting for cache
verification to finish" message per 5 minutes.

Collector 1503:  excessive logging.

ClientStorage._wait_sync():  Don't log more than one "waiting for cache
verification to finish" message per 5 minutes.

Critical bugfix for invalidations during ZEO cache verification.

ClientCache.invalidate():

- This was written not to expect tid=None, which is what happens
  during cache verification, although it didn't catch the tid=None
  case and went on doing really strange things then.  Rewrote so
  that tid=None means "forget everything I think I know about
  the oid".  It was kind-of performing an invalidation wrt the
  in-memory data structures in this case, but leaving the
  invalidated data for oid looking as if it were still current on
  disk.  That was accidental, depending on what various other
  parts of the code thought "None" meant (primarily, an end_tid
  of None is generally taken here to mean "the data is current").
  
- Also changed the "and we already have version data for oid" case
  to a "forget everything I think I know about the oid" case.  The
  comments seemed to say that's what it thought it was doing, but
  the code actually left non-current revisions sitting around.  I
  really don't want to spend time trying to figure out what versions
  are supposed to do in all cases here, and nuking everything in
  this case is an obviously *safe* thing to do.

- Added more asserts, because the pre- and post-conditions here are
  delicate and non-obvious (indeed, that a new assert failed is how
  I stumbled into the invalidation problem to begin with).

FileCache.remove():  Changed to remove the key from self.key2entry.
While self.filemap keeps an in-memory map of the entire disk file,
there's no need to keep info in key2entry for keys that no longer
exist in the disk file.  I'm not sure a visible bug followed from
this (but not sure one didn't either!), but it at best wasted
memory, and violated a documented invariant.

ConnectionTests.py:  New test checkVerificationInvalidationPersists().
Before the patch, this provokes a persistent ZEO cache into
delivering stale object state.  Of course the test passes after
this patch.

stats.py:  Added a trace code meaning "invalidation hit, and the
object state was discarded".  This is what happens now for
invalidations produced by verification.  Because of MVCC, that's
really not needed *except* for the invalidations produced by
startup verification (invalidations normally just set a new
ending tid on the validity range for what was current data; but
validity-range info isn't available from verification invalidations).

ClientCache.invalidate():

- This was written not to expect tid=None, which is what happens
  during cache verification, although it didn't catch the tid=None
  case and went on doing really strange things then.  Rewrote so
  that tid=None means "forget everything I think I know about
  the oid".  It was kind-of performing an invalidation wrt the
  in-memory data structures in this case, but leaving the
  invalidated data for oid looking as if it were still current on
  disk.  That was accidental, depending on what various other
  parts of the code thought "None" meant (primarily, an end_tid
  of None is generally taken here to mean "the data is current").
  
- Also changed the "and we already have version data for oid" case
  to a "forget everything I think I know about the oid" case.  The
  comments seemed to say that's what it thought it was doing, but
  the code actually left non-current revisions sitting around.  I
  really don't want to spend time trying to figure out what versions
  are supposed to do in all cases here, and nuking everything in
  this case is an obviously *safe* thing to do.

- Added more asserts, because the pre- and post-conditions here are
  delicate and non-obvious (indeed, that a new assert failed is how
  I stumbled into the invalidation problem to begin with).

FileCache.remove():  Changed to remove the key from self.key2entry.
While self.filemap keeps an in-memory map of the entire disk file,
there's no need to keep info in key2entry for keys that no longer
exist in the disk file.  I'm not sure a visible bug followed from
this (but not sure one didn't either!), but it at best wasted
memory, and violated a documented invariant.

ConnectionTests.py:  New test checkVerificationInvalidationPersists().
Before the patch, this provokes a persistent ZEO cache into
delivering stale object state.  Of course the test passes after
this patch.

stats.py:  Added a trace code meaning "invalidation hit, and the
object state was discarded".  This is what happens now for
invalidations produced by verification.  Because of MVCC, that's
really not needed *except* for the invalidations produced by
startup verification (invalidations normally just set a new
ending tid on the validity range for what was current data; but
validity-range info isn't available from verification invalidations).

Collector 1536:  ClientStorage.__init__ ignores cache_size.

Also split test checkRollover into two tests,
checkDisconnectedCacheWorks (persistent ZEO cache large enough
to satisfy all requests) and checkDisconnectedCacheFails (cache
too small to satisfy all requests while disconnected).

Also split test checkRollover into two tests,
checkDisconnectedCacheWorks (persistent ZEO cache large enough
to satisfy all requests) and checkDisconnectedCacheFails (cache
too small to satisfy all requests while disconnected).

More cleanup, and a new problem.  Notable:

Object.serialize_header():  Stopped stuffing the version string into this
notion of "a header".  Couldn't see any reason for it, and it was
surprising.

ClientCache.invalidate():  Added some asserts to clarify intended
preconditions.  Alas, my guess that it was intended that the passed-in
tid always be greater than the current tid turned out to trigger
errors.  This gets complicated, and hasn't been resolved yet.  The
problem is that the passed-in tid can be (in fact, always is) None
during cache verification.  The code here *really* doesn't seem to
be expecting that, and the on-disk cache file is clearly left in a
wrong state (if we repopulated the cache from the disk file, the
object would not look invalidated any more, it would look current).
Dumped in a giant stack trace so I don't forget this; sent email to
Jeremy asking if he recalls what the real intent was.

Object.serialize_header():  Stopped stuffing the version string into this
notion of "a header".  Couldn't see any reason for it, and it was
surprising.

ClientCache.invalidate():  Added some asserts to clarify intended
preconditions.  Alas, my guess that it was intended that the passed-in
tid always be greater than the current tid turned out to trigger
errors.  This gets complicated, and hasn't been resolved yet.  The
problem is that the passed-in tid can be (in fact, always is) None
during cache verification.  The code here *really* doesn't seem to
be expecting that, and the on-disk cache file is clearly left in a
wrong state (if we repopulated the cache from the disk file, the
object would not look invalidated any more, it would look current).
Dumped in a giant stack trace so I don't forget this; sent email to
Jeremy asking if he recalls what the real intent was.

More cleanup.  Notable:

- Object.fromFile():  renamed `header_only` arg to `skip_data`.
  Skipping the data is what it does, while there are at least 3
  distinct notions of what "a header" means in this module.
  Object has two notions of "header" all by itself (& I'm still
  not sure why Object.serialize_header() considers the version
  string to be part of "the header").

- FileCache.__init__():  log a warning if reuse=True but the
  given file path doesn't exist.  The code ignores `reuse` then
  (before, and now).  Not sure that's the best thing to do.

- Object.fromFile():  renamed `header_only` arg to `skip_data`.
  Skipping the data is what it does, while there are at least 3
  distinct notions of what "a header" means in this module.
  Object has two notions of "header" all by itself (& I'm still
  not sure why Object.serialize_header() considers the version
  string to be part of "the header").

- FileCache.__init__():  log a warning if reuse=True but the
  given file path doesn't exist.  The code ignores `reuse` then
  (before, and now).  Not sure that's the best thing to do.

More cleanup.  Notable:

- Moved the logger object to module scope.  There's virtually
  no logging of any kind done here, and I suspect that's partly
  because the logging object was clumsy to get at.

- Figured out what the code actually does when the cache size
  asked for doesn't match the actual size of a pre-existing
  cache file.  It apparently wanted to ignore the requested
  size then.  It probably shouldn't, but leaving that for later.
  Still, it left its own idea of the file size out of synch
  with the actual file size, and that was a bug.  For now,
  logged a warning when this happens, and changed the code to
  respect the actual file size.

- Moved the logger object to module scope.  There's virtually
  no logging of any kind done here, and I suspect that's partly
  because the logging object was clumsy to get at.

- Figured out what the code actually does when the cache size
  asked for doesn't match the actual size of a pre-existing
  cache file.  It apparently wanted to ignore the requested
  size then.  It probably shouldn't, but leaving that for later.
  Still, it left its own idea of the file size out of synch
  with the actual file size, and that was a bug.  For now,
  logged a warning when this happens, and changed the code to
  respect the actual file size.

More cleanup.  Notable:

ClientCache:  Removed the .tid attribute.  It was never updated; the
contained FileCache instance actually keeps track of the last tid.
This had the happy side effect of fixing bugs in testSerialization:
it was accessing .tid directly, and so was _always_ comparing None to
None.  Changing the test to use .getLastTid() instead means it's now
testing what it always intended to test.

FileCache:  Removed some unused private attributes for tracking
internal statistics.

ClientCache:  Removed the .tid attribute.  It was never updated; the
contained FileCache instance actually keeps track of the last tid.
This had the happy side effect of fixing bugs in testSerialization:
it was accessing .tid directly, and so was _always_ comparing None to
None.  Changing the test to use .getLastTid() instead means it's now
testing what it always intended to test.

FileCache:  Removed some unused private attributes for tracking
internal statistics.

More code and comment cleanups.  Notable changes:

FileCache:  removed the currentsize attribute.  It was initialized
to 0 but never updated.  Some tests referenced it, but since it was
always 0 those tests weren't getting any good from it.  Don't
recall (or never knew) what its intended purpose may have been.

FileCache.remove():  Repaired major bug.  This mistakenly stored
the "this disk block is free now" status byte into the start of
the serialized Object record's end_tid field (incorrect seek
offset).  The in-memory structures were correct then, but got out
of synch with the disk file; the latter then still claimed to have
info for a "live" object revision, but that revision was actually
dead, and the info on disk had a corrupt end_tid value.

FileCache:  removed the currentsize attribute.  It was initialized
to 0 but never updated.  Some tests referenced it, but since it was
always 0 those tests weren't getting any good from it.  Don't
recall (or never knew) what its intended purpose may have been.

FileCache.remove():  Repaired major bug.  This mistakenly stored
the "this disk block is free now" status byte into the start of
the serialized Object record's end_tid field (incorrect seek
offset).  The in-memory structures were correct then, but got out
of synch with the disk file; the latter then still claimed to have
info for a "live" object revision, but that revision was actually
dead, and the info on disk had a corrupt end_tid value.

Many small comment and code improvements.

Explicitly cast to float.  Else MSVC generated
legit warnings everywhere this macro was used,
warning about possible silent precision loss.

This adds a new IFBTree type, intended for use in indices.  Still
needs NEWS, still needs docs, and there are a pile of legitimate
compiler warnings on Windows that need fixing.

Forward port from ZODB 3.2.

Collector 1581:  fspack can blow up when .fs is corrupted

Repaired three places where fspack referenced an undefined global while
*trying* to raise CorruptedError.  Added new checkCorruptionInPack()
test to verify the correct exception gets raised.

Collector 1581:  fspack can blow up when .fs is corrupted

Repaired three places where fspack referenced an undefined global while
*trying* to raise CorruptedError.  Added new checkCorruptionInPack()
test to verify the correct exception gets raised.

type_and_adr():  Display the oid too.

Change ConflictError constructor to stop importing app objects.

When the ConflictError constructor is passed a pickle, extract
the module and class names without loading the pickle.  A ZEO
server doesn't necessarily have the implementation code for
application classes, and when that's so the attempt to raise
ConflictError was itself dying with an ImportError.

When the ConflictError constructor is passed a pickle, extract
the module and class names without loading the pickle.  A ZEO
server doesn't necessarily have the implementation code for
application classes, and when that's so the attempt to raise
ConflictError was itself dying with an ImportError.

This one doesn't exist on the 3.3 branch.

Move more get_pickle_metadata imports.

Import get_pickle_metadata from its new home.

Move get_pickle_metadata() into utils.py.

Try to make more sense of the ZODB pickle format "docs".