• Sean McGivern's avatar
    Disable JobWaiter when there are many jobs · 29132e38
    Sean McGivern authored
    In WaitableWorker, when there are fewer than four jobs to process we
    simply process them without going through Sidekiq. When there are more
    jobs, we run them in Sidekiq but add a second argument for a JobWaiter
    key, and wait for `timeout` seconds to see if those jobs finished.
    
    When there are very many jobs in the batch, we will never finish the
    batch before the timeout. This has these consequences:
    
    1. The HTTP request takes longer as we're waiting for the timeout.
    2. The jobs can't be deduplicated as the second argument is a unique
       JobWaiter key.
    
    This change adds a feature flag, skip_job_waiter_for_large_batches, that
    doesn't use the JobWaiter key when the batch size is more than ten times
    the timeout, as we can process approximately ten jobs per second. By
    default the flag is off.
    29132e38
waitable_worker.rb 1.72 KB