of pack in buffered mode, then switching to unbuffered mode to copy the
tail, actually broke pack completely on Windows:  we didn't close the
buffered file handle before opening the unbuffered one, and self.gc held
on to the still-open former handle.  This prevented the caller from
renaming Data.fs to Data.fs.old (a handle on Data.fs was still open).
The cure is simply to close a handle when we stop using it.

Only open the file for unbuffered I/O after finishing the first phase
of pack.  The first phase gets its end-of-file position from the main
thread, so there's no possibility of reading a 'c' record.

Timings on Linux are inconclusive, but it seems like using buffered
I/O for the initial phase should be faster.

can still be in progress, and they're written to the same physical file via
a different file object.  Using buffered I/O in the packer creates the
possiblity for the packer to see stale data from its file object's stdio
buffers.  This is now known to happen under Linux, Gentoo, OS X, Cygwin,
and Debian.  It apparently doesn't happen under native Windows, which is
why everyone except me has been seeing the new checkPackLotsWhileWriting
test fail.

This will probably need to be backported everywhere, but first I want to
see in which new way checkPackLotsWhileWriting fails on 48 Linux boxes
overnight <wink>.

    FileStorageError:  The database has already been packed to a
    later time or no changes have been made since the last pack

exception.  Instead that message is logged (at INFO level), and
the pack attempt simply returns then (no pack is performed).

Incidentally, this should repair frequent reports of failure in the
new checkPackLotsWhileWriting test.  On non-Windows systems, it
seems that the worker thread often didn't get enough cycles to commit
a change between the main thread's attempts to run pack() (and so
the exception above got raised then).

failures in the ZEO FileStorage pack-while-writing tests, where they die
with
    CorruptedError: ... transaction with checkpoint flag set

By coincidence, the same kind of death during pack() was reported on
zodb-dev today(!).  The new checkPackLotsWhileWriting test provokes this
on my box reliably (failed each time it was run).

The problem appears to be that pack's idea of where a FileStorage
ends was obtained merely by seeking to the end of the file.  But if
a transaction is in progress, there can be an extra "not really there
yet" transaction at the end of the file, and when either buildPackIndex()
or findReachableFromFuture() bumped into a such a thing, the CorruptedError
above got raised.  This is always rare, and was exceedingly rare before
because only one pack per test was tried.  The new test tries packing
repeatedly while a thread keeps hammering the database, so is much more
likely to provoke the problem.

The fix amounts to passing FileStorage._pos to the pack code, telling
the latter directly where the legit transactions in the file end.

it, and the only value a subclass ever set it to was None.

FileStoragePacker's constructor mean.

"Yow!" here anymore <wink>.

Also remove some unused imports and a somewhat irrelevant comment.

Move two class variable definitions into __init__(), because class
variables defined in __slots__ are immutable.  This should still be a
net memory savings, because the a 8-entry dict and a 10-entry dict
both have the same allocated size.

Please make all future changes on the Zope-2_7-branch instead.

happy again.

Reported by Sidnei da Silva.

The pack() implementation would leave a corrupted storage behind if
transactions committed while packing.

2.1.

Reduces memory usage during pack by a considerable fraction with
modest 10% slowdown.