of pack in buffered mode, then switching to unbuffered mode to copy the
tail, actually broke pack completely on Windows:  we didn't close the
buffered file handle before opening the unbuffered one, and self.gc held
on to the still-open former handle.  This prevented the caller from
renaming Data.fs to Data.fs.old (a handle on Data.fs was still open).
The cure is simply to close a handle when we stop using it.

Only open the file for unbuffered I/O after finishing the first phase
of pack.  The first phase gets its end-of-file position from the main
thread, so there's no possibility of reading a 'c' record.

Timings on Linux are inconclusive, but it seems like using buffered
I/O for the initial phase should be faster.

Provide a way to shutdown the servers using an exit status.

can still be in progress, and they're written to the same physical file via
a different file object.  Using buffered I/O in the packer creates the
possiblity for the packer to see stale data from its file object's stdio
buffers.  This is now known to happen under Linux, Gentoo, OS X, Cygwin,
and Debian.  It apparently doesn't happen under native Windows, which is
why everyone except me has been seeing the new checkPackLotsWhileWriting
test fail.

This will probably need to be backported everywhere, but first I want to
see in which new way checkPackLotsWhileWriting fails on 48 Linux boxes
overnight <wink>.

(distutils mungs it anyway, so it's just relevant for developers)

    FileStorageError:  The database has already been packed to a
    later time or no changes have been made since the last pack

exception.  Instead that message is logged (at INFO level), and
the pack attempt simply returns then (no pack is performed).

Incidentally, this should repair frequent reports of failure in the
new checkPackLotsWhileWriting test.  On non-Windows systems, it
seems that the worker thread often didn't get enough cycles to commit
a change between the main thread's attempts to run pack() (and so
the exception above got raised then).

thread is interesting here, so removed the hair catering to the possiblity
of N > 1 threads.

failures in the ZEO FileStorage pack-while-writing tests, where they die
with
    CorruptedError: ... transaction with checkpoint flag set

By coincidence, the same kind of death during pack() was reported on
zodb-dev today(!).  The new checkPackLotsWhileWriting test provokes this
on my box reliably (failed each time it was run).

The problem appears to be that pack's idea of where a FileStorage
ends was obtained merely by seeking to the end of the file.  But if
a transaction is in progress, there can be an extra "not really there
yet" transaction at the end of the file, and when either buildPackIndex()
or findReachableFromFuture() bumped into a such a thing, the CorruptedError
above got raised.  This is always rare, and was exceedingly rare before
because only one pack per test was tried.  The new test tries packing
repeatedly while a thread keeps hammering the database, so is much more
likely to provoke the problem.

The fix amounts to passing FileStorage._pos to the pack code, telling
the latter directly where the legit transactions in the file end.

The only values _packt ever had were z64 and None.  The only thing _packt
was used for that appeared to make sense was to block undo while a pack
was in progress, and a bool works better for that.  tids were compared
to _packt in a few spots, but since the only values _packt could have
were in {z64, None}, the comparisons were doomed to (regardless of tid)
return "greater than" (z64) or to be undefined (None; although in recent
Pythons None is reliably less than objects of other types, so "greater
than" was the only outcome possible from these comparisons under 2.3.3).
Removed these comparisons, collapsing surrounding expressions to
equivalents taking into account that tid < self._packt was always false.

- remove old XXX comments: we have added information on schema, and
  no, we really don't want to talk about "handlers" -- they're evil

context

it, and the only value a subclass ever set it to was None.

FileStoragePacker's constructor mean.

  <key name="+">; keys need to be provided for each default value
- fix bug in converting an empty section that contained a
  <multikey name="+">
- document <default> element in reference manual and DTD
- more tests

a two-state flag, with possible values z64 or None -- eh?).

record's "status byte" are and mean.

Make _added_during_commit a regular instance variable.  Don't use
try/finally to reset it; just clear it at the start of a transaction.
XXX There was a test that needed to be removed, but it seemed to be
just a shallow test that try/finally was used.  Can't see any feature
that depends on specific of error handling: The txn is going to abort.

Remove unused _opened instance variable.
Split commit() into two smaller parts.
Get rid of extra manipulation of _creating.
Don't look for _p_serial of None; z64 is now required.
Undo local variable aliases in subtransaction methods.

Also, trivial change to pickle cache API -- get() works like dict get().

with

    CorruptedError:  ... transaction with checkpoint flag set

on my box when doing the ZEO flavor of FileStorage.  This is like the
other pack-while-writing tests, but uses only 1 client thread that does
twice as much, and packs repeatedly for so long as that thread runs.  I
usually see it fail before the fourth pack attempt.

This makes the ZEO flavors of the tests much more likely to complete
without a "thread still alive" failure.

Forcing the output even when a test passes is interesting on my box.
When ZEO is involved, a single ClientThread tends to run all the way to
completion before another ClientThread manages to sneak in -- apart from
their first transactions, it's as if we ran one thread, then started the
next after the first finished (etc).  When ZEO isn't involved, threads
take turns.  When I see failures, they involve ZEO.  It's possible that
30 seconds just isn't long enough to wait on this box.

Drop several possible tasks that didn't seem relevant,
e.g. _p_independent() tests elsewhere should cover it just fine.

It's clear in hindsight that the connection only loads non-current
revisions for objects that have been invalidated.  Thus, we never need
to track them to invalidate stale objects; they'll be invalidated
anyway.  And loadBefore() will never return a current revision,
because it would only be called if the current revision could not be
used.

an unexpectedly live thread at the end.  That's been happening forever,
but never when I stare at it, and the more I stare at those tests the more
baffled I am that it ever fails this way.  So the point here is to get
more clues when nobody is staring at it <wink>.