• Arturo Herrero's avatar
    Propagate integrations using batching and queues · 163ad707
    Arturo Herrero authored
    This is an important change in the architecture to propagate
    integrations. We can now propagate instance-level integrations and
    templates using batching and Sidekiq queues.
    
    The problem before is the performance of the worst-case scenario, where
    if there are no matching records and the anti-join. With the new
    approach, each job in the new queues handles a batch of projects/groups;
    rather than having a single job for all of them.
    
    This is what we do right now in the complex case of propagating an
    instance-level integration:
    - Update inherited integrations,
    - Create integration for all projects without integration.
    - Create integration for all groups without integration.
    
    BEFORE:
    
                     Save integration
                            ↓
                ┌┬───────────────────────┬┐
                │| Propagate integration |│
                └┴───────────────────────┴┘
                            ↓
               Update inherited integrations
      Create integration for all projects without integration
      Create integration for all groups without integration
    
    AFTER:
    
                     Save integration
                            ↓
                ┌┬───────────────────────┬┐
                │| Propagate integration |│
                └┴───────────────────────┴┘
                   ↓                   ↓
      ┌┬─────────────────────┬┐ ┌┬───────────────────────┬┐
      │| Propagate to groups |│ │| Propagate to projects |│
      └┴─────────────────────┴┘ └┴───────────────────────┴┘
                Update inherited integrations
    163ad707
propagate_integration_group_worker_spec.rb 558 Bytes