Fixes browsing ERP5 documents without activity tables available.
Of course, as most modifications trigger activity creation, no
modifications are possible. But this improves read-only access in
degraded conditions.
Also, by design a successful query must return one line, so do not test
returned value before processing it.

In the same time, remove sortable columns since ActivityTool does not offer
possibility of sorting.

When grouped messages fail, ActivityTool must distinguish 3 groups,
in order to reexecute them separately, as follows:
- first, those that succeeded
- then, those that were skipped
- at last, failed ones

Grouping methods are updated to handle partial failures, and stop doing
anything when something goes wrong.

Without this, we would have the following pathological cases.

1. Let's suppose first that skipped messages are marked as succeeded.

The problem is that each skipped message that will fail causes the reexecution
of those that didn't fail.

Exemple: A:ok B:ok C:err D:err E:err F:err
  1: A:ok, B:ok, C:err, D:skipped, E:skipped, F:skipped
  2: A:ok, B:ok, D:err, E:skipped, F:skipped
  3: A:ok, B:ok, E:err, F:skipped
  4: A:ok, B:ok, F:err
  5: A:ok, B:ok -> commit

And worst, the first failed (C) may be processable again before 5, entering
a failing loop if it is executed again in the same group as A & B.

2. Another implementation is to mark all skipped as failed.

Example:
  1: A:ok, B:ok, C:err, D:skipped, E:skipped, F:skipped
  2: A:ok, B:ok -> commit
  3: C:err, D:skipped, E:skipped, F:skipped
 >3: same as 3

=> D, E or F are never tried.

This tweaks retry delays as follows (seconds):

        ConflictError  other failures (K = 1 + retry², with retry >= 0)
before     30             15 * K
after      15             30 * K

Today, bigger delays in case of errors should not be an issue because the
quality of ERP5 has improved a lot and normal code should not rely anymore on
this.

We also don't want to lower ConflictError delay too much, because this
increase the probability of conflicts.

This will be required to improve invokeGroup, in case that a message fails.
We'd like that:
- successful messages are retried immediately
- skipped messages are retried next, and separately
- at last, failed messages are retried, also separately

Race conditions are likely to happen with CMFActivity between message tables
are automatically upgraded during bootstrap.

Most code is moved from DA patch to ZMySQLDA.

For even more refactoring between SQLDict & SQLQueue,
which now uses SQL tables with the same schema.

The action to recreate activity tables while preserving existing messages
was unsafe for 2 reasons:
- if any error happened, messages could be lost
- it relied on Message.reactivate

Which this patch, any instance created after commit d881edd1 (Aug 2010) will
upgrade successfully. For older instances, make sure you have no activity left.

For cases where 'ALTER TABLE' would not work, a better way to implement repair
functionality would be:
- one action to backup all messages in ZODB
- and another to restore them
And maybe a security so that during the backup-clear-restore sequence,
activities can't be created nor processed.

If any column is added in the future, it would still be possible to write code
that fills them by inspecting messages.

UPDATE query is exected to use the existing index on (processing_node,
priority, date) both for WHERE and ORDER BY, as is expected from
EXPLAIN-ing the equivalent SELECT:

MariaDB [erp5]> explain select uid from message_queue WHERE processing_node=0 AND date <= '2013-06-06 22:22:49' ORDER BY priority, date LIMIT 1;
+------+-------------+---------------+------+----------------------------------------------------------+-------------------------------+---------+-------+-------+--------------------------+
| id   | select_type | table         | type | possible_keys                                            | key                           | key_len | ref   | rows  | Extra                    |
+------+-------------+---------------+------+----------------------------------------------------------+-------------------------------+---------+-------+-------+--------------------------+
|    1 | SIMPLE      | message_queue | ref  | processing_node_processing,processing_node_priority_date | processing_node_priority_date | 2       | const | 26622 | Using where; Using index |
+------+-------------+---------------+------+----------------------------------------------------------+-------------------------------+---------+-------+-------+--------------------------+

If it weren't using the index for ORDER BY, "Extra" would contain
"Using filesort".

Still, UPDATE behaves differently:

  # User@Host: user[user] @  [10.0.0.3]
  # Thread_id: 1635880  Schema: erp5  QC_hit: No
  # Query_time: 2.668405  Lock_time: 2.460698  Rows_sent: 0  Rows_examined: 49263
  # Full_scan: No  Full_join: No  Tmp_table: No  Tmp_table_on_disk: No
  # Filesort: Yes  Filesort_on_disk: No  Merge_passes: 0
  SET TIMESTAMP=1370557446;
  UPDATE
    message_queue
  SET
    processing_node=12
  WHERE
    processing_node=0
    AND DATE <= '2013-06-06 22:24:04'
    ORDER BY
    priority, DATE
  LIMIT 1;

So change the UPDATE..SELECT pattern into a SELECT FOR UPDATE..UPDATE
pattern, so SELECT's correct execution plan is used.

For some reason, MIN() doesn't use the index, while ORDER BY + LIMIT does.
Also, provide a more helpful error when assertion fails.

Later, we might want to do more processing after loading, or before dumping,
accessing private Message data.

When an object is deleted, higher level code used to flush its messages (without
invoking them). However, a concurrent and very long transaction may be about to
activate such an object, without conflict. We already experienced false -3
errors that could prevent other messages to be validated.

Because there is no efficient and reliable way to flush absolutely all messages,
messages on deleted objects are now ignored and deleted without any email
notification. There's only a WARNING in logs. But for performance reasons,
there's still a flush on object deletion.

To simplify code, messages that went to -3 for other reasons, like a
non-existing method, now go to -2. In fact, this was already the case for
grouped messages.

In case that a path is recycled, it may still be possible for a message to be
executed on a wrong object (the new one), instead of being ignored (because the
activated object was deleted). So in such scenario, developer should make sure
not to delete an object that may be activated in a concurrent transaction.
If the original object has an OID at the moment it is activated, an assertion
will make sure the message is not executed on another object.

Because flush() relies on this method, it becomes also possible to do
  ob.flushActivity(method_id=('immediateReindexObject',
                              'recursiveImmediateReindexObject'))

New SQLBase._getMessageList method has a very simple ZSQLCatalog-like API,
and replaces SQL{Dict,Queue}_readMessageList DTML scripts.

'include_processing' parameter disappears (in favor of 'processing')
and default behaviour is to include messages being processed.

This often happens during reindexation of many objects:
the number of processable grouped messages often exceeds 100
and 1 extra message is reserved for nothing.

This fixes a regression caused by f02bc341
("CMFActivity: automatically reselect messages in 'processing=1' state").

This fixes a regression by commit 6fd63363.

In case of failure, reserved messages were freed before finalization.
This was useless because finalization frees them and dangerous because another
zope could start processing a message that should not be executed again.

Checked following occurrences in comments:
- "Python 2.[456]"
- "Zope 2.[891]"
- "BBB"
- "BACK"

Checked uses of:
- email, hashlib, numpy & tarfile (modules)
- ImportError
- string.Template
- suppress_events (parameter of _setObject)

Excluded:
- some forked modules (MailTemplates, PortalTransforms...)
- some i18n compatibility code

It defaults to .01 and must be in the interval ]0, 1]
If a message expands to N objects, then it costs N / group_method_cost
and activity tool stop reserving messages the sum of all costs exceeds 1.

By default, this preserves original behaviour, i.e. MAX_GROUPED_OBJECTS = 100

Value 1 is not rejected to allow generic coding
(ex: activity_kw['group_method_cost'] = 1. / packet_size)

Activities could read from the ZODB an state older than the one
that caused the activity to be created, if:
1.Zope client node (node A) processing an activity  message is different
  than the one that created the activity (node B),
2.The object cache for node A contains objects concerning the activity
  message (or its container)
3.The node A hasn't yet received the invalidation message from the ZEO
  server, for instance, if its still on the network layer (kernel buffers,
  routers in between, etc...)

The simplest fix for this issue is sending a synchronous message to the
ZEO server before the beginning of a transaction. This message will act
like a “network barrier”, making sure that any invalidation messages
sent before that point from the ZEO server are already received, and the
transaction can begin with an “updated enough” state.

Additional note from Yoshinori : This patch must be proposed to zope
developpers as soon as possible and see with them if this way is the
best.

git-svn-id: https://svn.erp5.org/repos/public/erp5/trunk@39710 20353a03-c40f-0410-a6d1-a30d3c3de9de

getTransactionalVariable has always been implemented to return a global
cache object (1 per thread): old implementation used the given context only to
have access to the portal.

So we enforce this with a warning to make sure people don't expect retrieving
1 cache object per given context.

git-svn-id: https://svn.erp5.org/repos/public/erp5/trunk@38366 20353a03-c40f-0410-a6d1-a30d3c3de9de

git-svn-id: https://svn.erp5.org/repos/public/erp5/trunk@37927 20353a03-c40f-0410-a6d1-a30d3c3de9de

git-svn-id: https://svn.erp5.org/repos/public/erp5/trunk@37686 20353a03-c40f-0410-a6d1-a30d3c3de9de

git-svn-id: https://svn.erp5.org/repos/public/erp5/trunk@37685 20353a03-c40f-0410-a6d1-a30d3c3de9de

git-svn-id: https://svn.erp5.org/repos/public/erp5/trunk@37684 20353a03-c40f-0410-a6d1-a30d3c3de9de

At most 1 message for a given serialization tag can be validated (cf [28706]).
Respect priority/date/uid when validating only 1 message (cf [34632]).

git-svn-id: https://svn.erp5.org/repos/public/erp5/trunk@34841 20353a03-c40f-0410-a6d1-a30d3c3de9de

git-svn-id: https://svn.erp5.org/repos/public/erp5/trunk@34303 20353a03-c40f-0410-a6d1-a30d3c3de9de

* Change subject from "Failed Processing Activity" to
  "Activity failed: <path>/<method>"
* The subject starts with "Pending activity already failed <failures> times:"
  if CMFActivity is going to reexecute indefinitely.
* Drop the first line of the body. It was repeating the the subject.
* Add number of failures.

git-svn-id: https://svn.erp5.org/repos/public/erp5/trunk@32880 20353a03-c40f-0410-a6d1-a30d3c3de9de

getActivityRuntimeEnvironment is changed to return an ActivityRuntimeEnvironment
instance instead of a dict (and this value is now stored in a transactional
variable, for automatic cleanup).
This object allow activities to change default behaviour of CMFActivity if an
error happens. In the future, this object could also allow executed activity to
inspect its related Message object.

In case of infinite retry, notify the user when the default limit is reached.

git-svn-id: https://svn.erp5.org/repos/public/erp5/trunk@32879 20353a03-c40f-0410-a6d1-a30d3c3de9de

 retry |  now
 ------+------
   0   |    15
   1   |    30
   2   |  1:15
   3   |  2:30
   4   |  4:15
   5   |  6:30
   6   |  9:15
   7   | 12:30
   8   | 16:15
   9   | 20:30
  ...  |  ...

For ConflictError, delay is still constant to 30 seconds.

git-svn-id: https://svn.erp5.org/repos/public/erp5/trunk@32878 20353a03-c40f-0410-a6d1-a30d3c3de9de

Now, all messages can be executed 6 times by default (or more in case of
ConflictError), regardless their initial priority.
For compatibility reasons, the 'priority' column is still increased.

git-svn-id: https://svn.erp5.org/repos/public/erp5/trunk@32877 20353a03-c40f-0410-a6d1-a30d3c3de9de