Collector #479: Catalog tab now correctly puts "Unknown" in the type column for cataloged objects with no meta_type metadata column

Merge with HEAD: Collector #256: Added a check in _doChangeUser to make sure passwords isn't encrypted twice.

_dorestore(): Adopt the FileStorage semantics for when the transaction
pointed to by prev_txn does not exist.  This can happen if we're
restoring to a storage where that transaction has been packed away.
In that case, treat prev_txn as a hint and ignore it. :)

_TransactionsIterator.__len__(): Give this a dummy (and lying)
implementation which is just good enough for Python 2.1.3 support.

the environment has been closed.  Fortunately, there's another way.

Add some more logging.

Catalog empty results are now always a LazyCat([]) rather than a LazyCat(SomeEmptySequencishThingFromTheIndexes)

read_only flag on the config object, and BerkeleyBase sets
self._is_read_only when this flag is true, but it leaves any
ReadOnlyError raising to BaseStorage.

function to clean up all the files that a FileStorage creates.
Sometimes it's handy to do a cleanup if you have the instance, other
times you only have the Data.fs file name.

i.e. reason first, then oid.  This is because there's always a reason
but sometimes there isn't an oid (so we can default arg the oid).

Fixed getLexicon so that it can properly raise an error when it cannot find the proper lexicon object.

* Changed logic for activating first sort algorithm to elminate bad performance with large result sets (20k+). The full sort is now faster for a larger proportion of cases. This algorithm is also skipped now if a sort limit value is passed.

* Full sort now handles sort limits where the limit is 25% or greater of the total result where N-Best performance degrades. This allows the application to always apply a sort limit up to and beyond the result set length.

* Added an "N-worst" sort handler to deal with forward sort limits (previously only reverse limits worked properly).

* Small optimizations to N-best/worst to wring out a few more CPU cycles.

it's configured when you see it in the log files.

* For unsorted sets, merged and non-merged results are identical (lazy results)

* For sorted sets, merged results are lazy and non-merged are lists of three tuples that can later be merged and sorted using mergeResults

The mergeResults function has been simplified greatly and now simply accepts a list of result sets in its first argument.

ZODB4.  We need to return the unpickled extension data.

object -- because we just care about one event we can simply use a
threading.Event object.  Specific changes include,

BerkeleyBase.py

    __init__(): Simplify the creation of the checkpointer and
    autopacker worker threads.

    close(): Replace autopacker.stop() with setting the Event object.
    This both kicks us out of the wait() and sets the thread's
    internal stop flag, so it's all we need.

    _WorkThread.init(): Take the `name' argument out of the
    constructor.  It was the only thing that 2/3rds of the subclasses
    needed to override, so just stick it in a class attribute.

    run(): Simplify to use the Event object.  Also, change _nextcheck
    to recalculate `now' after the work is done.  There's no telling
    how much time the work will take (it may not matter much in
    practice).

    stop(): Removed.

    _Checkpoint.__init__(): Removed

Full.py

    _make_autopacker(): Updated

    _Autopack.__init__(): Updated

Minimal.py

    _make_autopacker(): Updated

    _Autopack.__init__(): Removed

bogus config object.

sleep based.  We create two ends of a pipe in the parent thread, with
the child thread polling/reading one end and the parent writing to the
other.  The only thing written to this pipe is a "stop marker" -- just
a string that wakes the thread up immediately when we're closing the
storage.

The primary reason for this is to speed up shutdown so that we don't
have to wait for the next tick of the sleep counter before we'll
trigger the thread shutdown.  Makes shutting down Zope using this
storage much quicker.

Specific changes include:

BerkeleyBase.py

    SLEEP_TIME -> JOIN_TIME since it's now just the interval we'll
    wait for the thread.join to complete.

    __init__():	Create both the checkpointer thread and the autopacker
    thread, set up the pipes, and get them both rolling.  We refactor
    creation of the autopacker instance into a separate overrideable
    method since this is the one bit that's different between the two
    storages.

    _make_autopacker(): Intended to be overridden.

    close(), _doclose(): Move the thread shutdown code out of the
    lock, since we don't want to potentially deadlock the shutting
    down of the background thread.  This could happen if close() was
    entered when the autopacker thread was between lock acquisitions
    in _dopack().  Also, be sure to write to the pipe to wake the
    child threads up immediately.

    env_from_string(): Wrap the actual creation and opening of the
    environment in a try/except, so that if there's a failure, we can
    be sure to give up the file lock.

    _WorkThread class: Accept a poll object which wraps the read end
    of the pipe.  Rework run() to do the poll-with-timeout instead of
    sleep.  I don't think we strictly need to do the fd read given the
    simplicity (and mono-commandity) of the protocol, but it can't
    hurt.  The _dowork() method's signature no longer contains the
    `now' variable.

Full.py

    _make_autopacker(): Override base class to return storage specific
    _Autopack instance.

Minimal.py

    Same, but also include some code cleanup.


Also, get rid of some unnecessary imports.

Major refactor of the catalog search engine centered around optimizing sort by index operations. The resulting code greatly outperforms the previous version and uses less memory.

Exposed a new ZCatalog method "search" which has a better interface for programmatic searches. Updated documentation as well.

Implemented a sort limit option which allows you to inform the catalog that you are only interested in a certain number of results. In the common case this allows the ZCatalog machinery to use a different sorting algorithm (N-Best) which scales much better then a full sort.

Also more tightly integrated the merge option which allows you to tell the catalog that you would like raw and unsorted intermediate results returned rather than sorted and lazified results. This can be used to efficiently merge search results across multiple catalogs.

list that was being modified as we went).
[Backport from Zope3; I'm not actually sure if the list returned by
**Btree.keys() is lazy, but I assume it is.]

        # DB 4.1.24 requires that operations happening in a transaction must
        # be performed on a database that was opened in a transaction.  Since
        # we do the former, we must do the latter.  However, earlier DB
        # versions don't transactionally protect database open, so this is the
        # most portable way to write the code.

i.e. we use the DB_AUTO_COMMIT flag for the open, if it's defined.

right thing to do anyway, and should work just find for earlier
versions of the wrapper.  Specifically,

_dorecovery(), _collect_objs(), _mark(): transactionally protect
.truncate() and .consume()

_docommit(): Move the setting of the _pending flag to here from
_finish() so we can transactionally protect it more conveniently.
This does't change the semantics and the recovery code in _setupDBs()
doesn't care since the flag will already be set to COMMIT.

right thing to do anyway, and should work just find for earlier
versions of the wrapper.  Specifically,

_doabort(), _docommit(), _mark(), _sweep(), _collect_objs():
Transactionally protect .truncate() and .consume()

_docommit(): Move the setting of the _pending flag to here from
_finish() so we can transactionally protect it more conveniently.
This does't change the semantics and the recovery code in _setupDBs()
doesn't care since the flag will already be set to COMMIT.

_begin(): Transactionally protect the setting of the _pending flag to
ABORT.

I couldn't find any references to cPickle in any of the setup/build
tools, but I couldn't actually verify that the Windows build was
unaffected.

Added ZPL header to stopper test module

Python 2.2.X version