Commit 3689b9e5 authored by Yorick Peterse's avatar Yorick Peterse

Merge branch 'document-background-migration-stealing' into 'master'

Document manual cleanup after background migration stealing

See merge request gitlab-org/gitlab-ce!14477
parents 6b354370 989ccd71
......@@ -215,14 +215,29 @@ same time will ensure that both existing and new data is migrated.
In the next release we can remove the `after_commit` hooks and related code. We
will also need to add a post-deployment migration that consumes any remaining
jobs. Such a migration would look like this:
jobs and manually run on any un-migrated rows. Such a migration would look like
this:
```ruby
class ConsumeRemainingExtractServicesUrlJobs < ActiveRecord::Migration
disable_ddl_transaction!
class Service < ActiveRecord::Base
include ::EachBatch
self.table_name = 'services'
end
def up
# This must be included
Gitlab::BackgroundMigration.steal('ExtractServicesUrl')
# This should be included, but can be skipped - see below
Service.where(url: nil).each_batch(of: 50) do |batch|
range = batch.pluck('MIN(id)', 'MAX(id)').first
Gitlab::BackgroundMigration::ExtractServicesUrl.new.perform(*range)
end
end
def down
......@@ -230,6 +245,15 @@ class ConsumeRemainingExtractServicesUrlJobs < ActiveRecord::Migration
end
```
The final step runs for any un-migrated rows after all of the jobs have been
processed. This is in case a Sidekiq process running the background migrations
received SIGKILL, leading to the jobs being lost. (See
[more reliable Sidekiq queue][reliable-sidekiq] for more information.)
If the application does not depend on the data being 100% migrated (for
instance, the data is advisory, and not mission-critical), then this final step
can be skipped.
This migration will then process any jobs for the ExtractServicesUrl migration
and continue once all jobs have been processed. Once done you can safely remove
the `services.properties` column.
......@@ -254,6 +278,9 @@ for more details.
1. Make sure that background migration jobs are idempotent.
1. Make sure that tests you write are not false positives.
1. Make sure that if the data being migrated is critical and cannot be lost, the
clean-up migration also checks the final state of the data before completing.
[migrations-readme]: https://gitlab.com/gitlab-org/gitlab-ce/blob/master/spec/migrations/README.md
[issue-rspec-hooks]: https://gitlab.com/gitlab-org/gitlab-ce/issues/35351
[reliable-sidekiq]: https://gitlab.com/gitlab-org/gitlab-ce/issues/36791
Markdown is supported
0%
or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment