• Yorick Peterse's avatar
    Steal pending merge request diff commit user jobs · b98a2d0c
    Yorick Peterse authored
    The background migration MigrateMergeRequestDiffCommitUsers processes
    batches of merge_request_diff_commits rows in batches. These migrations
    have been running for several weeks now.
    
    Unfortunately, on GitLab.com we are running into problems with the
    remaining 750 or so jobs. Some jobs take a few minutes to complete,
    while others can take over an hour to complete. The scheduling interval
    was originally just a few minutes. This lead to many jobs being thrown
    away, as they could not be picked up fast enough when there is a long
    running job. Rescheduling jobs with a greater interval hasn't proven
    successful.
    
    In this commit we add a new background migration to clean up the pending
    work. This background migration picks up a single job (in reverse
    order), processes it, then schedules the next job. This approach
    prevents jobs from conflicting with each other if they take too long.
    
    In addition, we change the existing migration to not process jobs that
    have already been processed. This way existing jobs that have yet to be
    picked up won't lead to duplicate work.
    
    Both the old and new migration _can_ process work in parallel, as they
    are different background migration classes. This shouldn't pose any
    problems. For the last few weeks we've been manually running jobs in a
    Rails console in parallel to the Sidekiq jobs, without any problems
    surfacing.
    
    See https://gitlab.com/gitlab-org/gitlab/-/issues/334394 for more
    information.
    
    Changelog: added
    b98a2d0c
20210823113259 64 Bytes