Fixed getLexicon so that it can properly raise an error when it cannot find the proper lexicon object.

* Changed logic for activating first sort algorithm to elminate bad performance with large result sets (20k+). The full sort is now faster for a larger proportion of cases. This algorithm is also skipped now if a sort limit value is passed.

* Full sort now handles sort limits where the limit is 25% or greater of the total result where N-Best performance degrades. This allows the application to always apply a sort limit up to and beyond the result set length.

* Added an "N-worst" sort handler to deal with forward sort limits (previously only reverse limits worked properly).

* Small optimizations to N-best/worst to wring out a few more CPU cycles.

it's configured when you see it in the log files.

* For unsorted sets, merged and non-merged results are identical (lazy results)

* For sorted sets, merged results are lazy and non-merged are lists of three tuples that can later be merged and sorted using mergeResults

The mergeResults function has been simplified greatly and now simply accepts a list of result sets in its first argument.

ZODB4.  We need to return the unpickled extension data.

object -- because we just care about one event we can simply use a
threading.Event object.  Specific changes include,

BerkeleyBase.py

    __init__(): Simplify the creation of the checkpointer and
    autopacker worker threads.

    close(): Replace autopacker.stop() with setting the Event object.
    This both kicks us out of the wait() and sets the thread's
    internal stop flag, so it's all we need.

    _WorkThread.init(): Take the `name' argument out of the
    constructor.  It was the only thing that 2/3rds of the subclasses
    needed to override, so just stick it in a class attribute.

    run(): Simplify to use the Event object.  Also, change _nextcheck
    to recalculate `now' after the work is done.  There's no telling
    how much time the work will take (it may not matter much in
    practice).

    stop(): Removed.

    _Checkpoint.__init__(): Removed

Full.py

    _make_autopacker(): Updated

    _Autopack.__init__(): Updated

Minimal.py

    _make_autopacker(): Updated

    _Autopack.__init__(): Removed

bogus config object.

sleep based.  We create two ends of a pipe in the parent thread, with
the child thread polling/reading one end and the parent writing to the
other.  The only thing written to this pipe is a "stop marker" -- just
a string that wakes the thread up immediately when we're closing the
storage.

The primary reason for this is to speed up shutdown so that we don't
have to wait for the next tick of the sleep counter before we'll
trigger the thread shutdown.  Makes shutting down Zope using this
storage much quicker.

Specific changes include:

BerkeleyBase.py

    SLEEP_TIME -> JOIN_TIME since it's now just the interval we'll
    wait for the thread.join to complete.

    __init__():	Create both the checkpointer thread and the autopacker
    thread, set up the pipes, and get them both rolling.  We refactor
    creation of the autopacker instance into a separate overrideable
    method since this is the one bit that's different between the two
    storages.

    _make_autopacker(): Intended to be overridden.

    close(), _doclose(): Move the thread shutdown code out of the
    lock, since we don't want to potentially deadlock the shutting
    down of the background thread.  This could happen if close() was
    entered when the autopacker thread was between lock acquisitions
    in _dopack().  Also, be sure to write to the pipe to wake the
    child threads up immediately.

    env_from_string(): Wrap the actual creation and opening of the
    environment in a try/except, so that if there's a failure, we can
    be sure to give up the file lock.

    _WorkThread class: Accept a poll object which wraps the read end
    of the pipe.  Rework run() to do the poll-with-timeout instead of
    sleep.  I don't think we strictly need to do the fd read given the
    simplicity (and mono-commandity) of the protocol, but it can't
    hurt.  The _dowork() method's signature no longer contains the
    `now' variable.

Full.py

    _make_autopacker(): Override base class to return storage specific
    _Autopack instance.

Minimal.py

    Same, but also include some code cleanup.


Also, get rid of some unnecessary imports.

Major refactor of the catalog search engine centered around optimizing sort by index operations. The resulting code greatly outperforms the previous version and uses less memory.

Exposed a new ZCatalog method "search" which has a better interface for programmatic searches. Updated documentation as well.

Implemented a sort limit option which allows you to inform the catalog that you are only interested in a certain number of results. In the common case this allows the ZCatalog machinery to use a different sorting algorithm (N-Best) which scales much better then a full sort.

Also more tightly integrated the merge option which allows you to tell the catalog that you would like raw and unsorted intermediate results returned rather than sorted and lazified results. This can be used to efficiently merge search results across multiple catalogs.

list that was being modified as we went).
[Backport from Zope3; I'm not actually sure if the list returned by
**Btree.keys() is lazy, but I assume it is.]

        # DB 4.1.24 requires that operations happening in a transaction must
        # be performed on a database that was opened in a transaction.  Since
        # we do the former, we must do the latter.  However, earlier DB
        # versions don't transactionally protect database open, so this is the
        # most portable way to write the code.

i.e. we use the DB_AUTO_COMMIT flag for the open, if it's defined.

right thing to do anyway, and should work just find for earlier
versions of the wrapper.  Specifically,

_dorecovery(), _collect_objs(), _mark(): transactionally protect
.truncate() and .consume()

_docommit(): Move the setting of the _pending flag to here from
_finish() so we can transactionally protect it more conveniently.
This does't change the semantics and the recovery code in _setupDBs()
doesn't care since the flag will already be set to COMMIT.

right thing to do anyway, and should work just find for earlier
versions of the wrapper.  Specifically,

_doabort(), _docommit(), _mark(), _sweep(), _collect_objs():
Transactionally protect .truncate() and .consume()

_docommit(): Move the setting of the _pending flag to here from
_finish() so we can transactionally protect it more conveniently.
This does't change the semantics and the recovery code in _setupDBs()
doesn't care since the flag will already be set to COMMIT.

_begin(): Transactionally protect the setting of the _pending flag to
ABORT.

I couldn't find any references to cPickle in any of the setup/build
tools, but I couldn't actually verify that the Windows build was
unaffected.

Added ZPL header to stopper test module

Python 2.2.X version

- logging an error message instead of raising an exception in case of
  unindex_object() failure

      - Collector #699: MailHosts created in 2.5 breaks in 2.6.

      - Collector #694: dtml-sendmail mailto specification replaces "To:" header.

Normalize whitespace (horizontal and vertical).

confusion caused by having the instance and the class have the same
name.

ChrisM's changes here) it is no longer necessary to call
zLOG.initialize() in order to see logging output.

least 2.2.2.)

Add -v1 to TESTOPTS, so running the entire test suite is a little less
verbose.

Stringify the object in the exception in case its a list or tuple (an inscrutable error occurs if this is not done).

_doclose() twice (runsvr.py had a bug that caused the storages to be
closed twice).