• Andreas Brandl's avatar
    Optimize batch size for migrations automatically · da5e21b3
    Andreas Brandl authored
    This is based on what we call "time efficiency". For a single job, time
    efficiency is the ratio of total duration to interval. Ideally, this is
    close to but smaller than 1.
    
    We use exponential smoothing (EMA) to look at the last 10 jobs and their time
    efficiency. With EMA, we tolerate a spiking job better than by just
    looking at the most recent job.
    
    We also define upper and lower bounds for the batch size.
    da5e21b3
batched_migration_runner.rb 3.07 KB