See the thread starting at
 http://mail.zope.org/pipermail/zope/2005-July/160433.html
for gory details.

Maintaining this is a significant expense; the code didn't
make real use of it; and every time I've tried to exploit it
I've ended up with grossly too-large simulated hit rates.
Would probably take another day to figure out exactly why that
is, and the simulated rates are "good enough" now without it.

although the toy app I wrote to generate a 500MB trace
file doesn't use MVCC in an essential way (some of the
MVCC simulation code is nevertheless exercised, since
an invalidation of current data in the presence of MVCC
always creates a cache record for the newly non-current
revision).

Still puzzling over what to do when the trace file records
a load hit but the simulated cache gets a miss.  The
old simulation code seemed to assume that a store for the same
oid would show up in the trace file next, and it could get
the info it needed about the missing object from the store
trace.  But that isn't true:  precisely because the load was
a hit in the trace file, the object isn't going to be stored
again "soon" in the trace file.

Here are some actual-vs-simulated hit rate results, for a
20MB cache, with a trace file covering about 9 million
loads, over 3 ZEO client (re)starts:

actual   simulated
------   ---------
  93.1        92.7
  79.8        79.0
  68.0        69.1
      
  81.4        81.1  overall

Since the simulated hit rates are both higher and lower
than the actual hit rates, that argues against a gross
systematic bias in the simulation (although there may
be several systematic biases in opposite directions).

MVCC.  It isn't blowing up, but it thinks the hit rate is
100% -- might have missed something there <wink>.

I think the primary hangup now is that simul.py doesn't know
anything about MVCC.  As a result, it thinks there's significatly
more free space in the cache than there really is, and that
probably accounts for it predicting significantly higher hit
rates than I actually see.

Output isn't obviously wholly insane, but isn't a reasonably
close enough match to stats.py output either yet.

struct.unpack() calls by 1/2.

Various bugfixes elsewhere.

This has survived several 100 MB of trace files
I generated over the last few days, so it's solid
now if not necessarily perfect.

Replaced simul.py with the much broader-ranging code
Jeremy and I were working on a couple years ago,
although it can't work with the current trace file
format (no real loss there -- the simul.py it's
replacing can't work with the current format either).

docs.  I'll stitch them back in from the ZConfig repo next,
via svn:externals.  Ideally, ZODB shouldn't have this as
part of its checkout tree at all, but that also has implications
for the way ZRS gets packaged, etc.

100% broken -- I can get a trace file now, and stats.py
doesn't blow up anymore.  Don't yet know whether it's
reporting sensible things, but it's not obviously insane.
simul.py doesn't know about the current cache implementation
either yet.

FileCache:  Removed the reuse= argument.  IIRC, it made some sense
when different cache schemes were first being tried, but complicated
the logic now to no good end.

Nuked the maddening log warnings about reuse=True.  They were never
helpful and were often confusing.

Added info-level log messages to record the path of the client cache
file, whether it's persistent or temporary, and if it's persistent
whether we're creating or reusing it.

Minor changes to comments.

an attempt to be compatible with both.

Bucket_rangeSearch():  De-obfuscated the calls to
Bucket_findRangeEnd(), so that they stop ignoring
the latter's error returns (a mind-bending combination
of embedded assignment nested in an UNLESS macro,
seemingly copy+paste'd so that the error occurred twice).

Collector #1846:  If an uncommitted transaction was found, fsrecover.py
fell into an infinite loop.  Fixed that, and added a new test
(testUncommittedAtEnd) to ensure this stays fixed.

I suspect BTrees/convert.py should be removed instead.

Since TmpStore.reset() had to change too to copy the index anyway,
it no longer hurts that _rollback() clears the index (as a side
effect of self._cache.invalidate(src.index)).

Allow rollback to a given savepoint multiple times.

transaction/_transaction.py
    _remove_and_invalidate_after():  Don't remove the
    argument savepoint from the savepoint stack anymore.

    abort():  abort(True) relied on the subtxn savepoint
    being invalidated as a side effect of rolling back
    to it.  Invalidate it explicitly now.

    Savepoint.rollback():  Don't invalidate self.

ZODB/Connection.py
    Connection._rollback():  Pass a temp container to
    _cache.invalidate(), because that method clears the
    container as a side effect.  Clearing the index
    in a savepoint was a disaster if the savepoint was
    used again.

    TmpStore.reset():  Make a copy of the argument index.
    An alternative would be to make all callers make
    copies, but the only caller now thinks of its `state`
    argument as being opaque.

transaction/savepoint.txt
ZODB/tests/{testSubTransaction.py, testConnectionSavepoint.txt}
    Test the new semantics, and beef up the tests.

transaction/tests/savepointsample.py
    _rollback_savepoint():  Needed to copy the data.

Primarily, _tindex should have been a dict all along, pack()
worked too hard at the end, and loadEx() is a minor variant
of load().

Ideally, load() should call loadEx() instead, but we really
don't want "an extra" Python-level call here (heavily used).

loadEx():  deleted pointless call of self._read_txn_header().

ServerStub.loadEx() comments:  these were obviously wrong in
several ways, but I don't know the full truth.  Better to
say so up front than to leave them clearly wrong, though.

The BTree minKey() and maxKey() methods gave a misleading message if no key
satisfying the constraints existed in a non-empty tree.

In some error cases when reading a FileStorage index, the
code referenced an undefined global.

Such code never made sense; it was all of the form

    t32 = 1L << 32
    if something < 0:
        something = t32 - something

which is a way to change negative integers into
gigantic positive integers greater than 4 billion, and
never made sense in context.  Possibly

        something = t32 + something

was intended, but that never made sense in context either.

In any case, `something` is invariably obtained from
struct.unpack using the "H" format code, and that never
returns a negative integer to begin with.

Clarified that the ``minKey()`` and ``maxKey()`` methods raise an exception
if no key exists satsifying the constraints.

Also improved the English in other interface docstrings.