Commit 24df3ec5 authored by Vincent Pelletier's avatar Vincent Pelletier

ERP5Type.Core.Folder.recurseCallMethod: Do not change priority for re-call

Increasing priority value (hence making the activity execution priority
lower) means that all previously-spawned activities will have at least
started execution before the next batch of activities begins being
prepared. During that preparation, more activities will end, meaning there
is then unused processing node time, decreasing processing efficiency.

By making re-call activity have the same priority as spawned activities
will reduce this delay by making the re-call activity candidate for
execution at the same time as the latest activity batch, without leading
to activity queue size increase: older activities within a given priority
range still get precedence, so any older group will tend to finish before
more batches get spawned.

Measures on indexing 700k documents with 20 activity nodes:
- priority + 1: 84k documents indexed in the first 5 minutes,
  47k in the last 5 minutes. Average: 64k/5 minutes
- priority: 101k documents indexed in the first 5 minutes,
  97k in the last 5 minutes. Average: 99k/5 minutes
Peak speed (which is the initial speed) is 20% higher.
As the slowdown rate (likely coming from lower-priority activities piling
up in the message_queue table) is also much lower in the second case
(900 fewer documents indexed on each 5 minutes increment instead of 4k),
the overall average speed improves by 54%.
parent cb5f5f9c
...@@ -485,7 +485,6 @@ class FolderMixIn(ExtensionClass.Base): ...@@ -485,7 +485,6 @@ class FolderMixIn(ExtensionClass.Base):
recurse(self, 0) recurse(self, 0)
except StopIteration: except StopIteration:
activate_kw['group_method_id'] = kw['group_id'] = '' # no grouping activate_kw['group_method_id'] = kw['group_id'] = '' # no grouping
activate_kw['priority'] = 1 + activate_kw.get('priority', 1)
activate(self, 'SQLQueue', **activate_kw)._recurseCallMethod( activate(self, 'SQLQueue', **activate_kw)._recurseCallMethod(
method_id, method_args, method_kw, restricted=restricted, **kw) method_id, method_args, method_kw, restricted=restricted, **kw)
......
Markdown is supported
0%
or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment