• Vincent Pelletier's avatar
    ERP5Type.Core.Folder.recurseCallMethod: Do not change priority for re-call · 24df3ec5
    Vincent Pelletier authored
    Increasing priority value (hence making the activity execution priority
    lower) means that all previously-spawned activities will have at least
    started execution before the next batch of activities begins being
    prepared. During that preparation, more activities will end, meaning there
    is then unused processing node time, decreasing processing efficiency.
    
    By making re-call activity have the same priority as spawned activities
    will reduce this delay by making the re-call activity candidate for
    execution at the same time as the latest activity batch, without leading
    to activity queue size increase: older activities within a given priority
    range still get precedence, so any older group will tend to finish before
    more batches get spawned.
    
    Measures on indexing 700k documents with 20 activity nodes:
    - priority + 1: 84k documents indexed in the first 5 minutes,
      47k in the last 5 minutes. Average: 64k/5 minutes
    - priority: 101k documents indexed in the first 5 minutes,
      97k in the last 5 minutes. Average: 99k/5 minutes
    Peak speed (which is the initial speed) is 20% higher.
    As the slowdown rate (likely coming from lower-priority activities piling
    up in the message_queue table) is also much lower in the second case
    (900 fewer documents indexed on each 5 minutes increment instead of 4k),
    the overall average speed improves by 54%.
    24df3ec5
Folder.py 62.4 KB