Commit 4be6bd5b authored by Drew Blessing's avatar Drew Blessing

Reload the schema before restoring a database backup

If a user tries to downgrade and restore after a failed upgrade,
the database may still contain newer tables. Reload the older
schema before restoring the database to avoid future upgrade
problems. Also, add a rake task to help users add migration
versions to the database so it's easier to recover from these
errors if they do occur. Fixes #13419
parent 8d3e5cf0
...@@ -2,6 +2,7 @@ Please view this file on the master branch, on stable branches it's out of date. ...@@ -2,6 +2,7 @@ Please view this file on the master branch, on stable branches it's out of date.
v 8.6.0 (unreleased) v 8.6.0 (unreleased)
- Improve the formatting for the user page bio (Connor Shea) - Improve the formatting for the user page bio (Connor Shea)
- Add option to reload the schema before restoring a database backup. !2807
v 8.5.1 v 8.5.1
- Fix group projects styles - Fix group projects styles
......
...@@ -249,6 +249,9 @@ reconfigure` after changing `gitlab-secrets.json`. ...@@ -249,6 +249,9 @@ reconfigure` after changing `gitlab-secrets.json`.
### Installation from source ### Installation from source
``` ```
# Stop processes that are connected to the database
sudo service gitlab stop
bundle exec rake gitlab:backup:restore RAILS_ENV=production bundle exec rake gitlab:backup:restore RAILS_ENV=production
``` ```
......
...@@ -15,3 +15,4 @@ Depending on the installation method and your GitLab version, there are multiple ...@@ -15,3 +15,4 @@ Depending on the installation method and your GitLab version, there are multiple
- [MySQL to PostgreSQL](mysql_to_postgresql.md) guides you through migrating your database from MySQL to PostgreSQL. - [MySQL to PostgreSQL](mysql_to_postgresql.md) guides you through migrating your database from MySQL to PostgreSQL.
- [MySQL installation guide](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/doc/install/database_mysql.md) contains additional information about configuring GitLab to work with a MySQL database. - [MySQL installation guide](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/doc/install/database_mysql.md) contains additional information about configuring GitLab to work with a MySQL database.
- [Restoring from backup after a failed upgrade](restore_after_failure.md)
# Restoring from backup after a failed upgrade
Upgrades are usually smooth and restoring from backup is a rare occurrence.
However, it's important to know how to recover when problems do arise.
## Roll back to an earlier version and restore a backup
In some cases after a failed upgrade, the fastest solution is to roll back to
the previous version you were using.
First, roll back the code or package. For source installations this involves
checking out the older version (branch or tag). For Omnibus installations this
means installing the older .deb or .rpm package. Then, restore from a backup.
Follow the instructions in the
[Backup and Restore](../raketasks/backup_restore.md#restore-a-previously-created-backup)
documentation.
## Potential problems on the next upgrade
When a rollback is necessary it can produce problems on subsequent upgrade
attempts. This is because some tables may have been added during the failed
upgrade. If these tables are still present after you restore from the
older backup it can lead to migration failures on future upgrades.
Starting in GitLab 8.5 we drop all tables prior to importing the backup to
prevent this problem. If you've restored a backup to a version prior to 8.5 you
may need to manually correct the problem next time you upgrade.
Example error:
```
== 20151103134857 CreateLfsObjects: migrating =================================
-- create_table(:lfs_objects)
rake aborted!
StandardError: An error has occurred, this and all later migrations canceled:
PG::DuplicateTable: ERROR: relation "lfs_objects" already exists
```
Copy the version from the error. In this case the version number is
`20151103134857`.
_**WARNING:** Use the following steps only if you are certain this is what you
need to do._
### GitLab 8.5+
Pass the version to a database rake task to manually mark the migration as
complete.
```
# Source install
sudo -u git -H bundle exec rake gitlab:db:mark_migration_complete[20151103134857] RAILS_ENV=production
# Omnibus install
sudo gitlab-rake gitlab:db:mark_migration_complete[20151103134857]
```
Once the migration is successfully marked, run the rake `db:migrate` task again.
You will likely have to repeat this process several times until all failed
migrations are marked complete.
### GitLab < 8.5
```
# Source install
sudo -u git -H bundle exec rails console production
# Omnibus install
sudo gitlab-rails console
```
At the rails console, type the following commands.
```
ActiveRecord::Base.connection.execute("INSERT INTO schema_migrations (version) VALUES('20151103134857')")
exit
```
Once the migration is successfully marked, run the rake `db:migrate` task again.
You will likely have to repeat this process several times until all failed
migrations are marked complete.
...@@ -22,7 +22,7 @@ namespace :gitlab do ...@@ -22,7 +22,7 @@ namespace :gitlab do
end end
# Restore backup of GitLab system # Restore backup of GitLab system
desc "GitLab | Restore a previously created backup" desc 'GitLab | Restore a previously created backup'
task restore: :environment do task restore: :environment do
warn_user_is_not_gitlab warn_user_is_not_gitlab
configure_cron_mode configure_cron_mode
...@@ -30,13 +30,32 @@ namespace :gitlab do ...@@ -30,13 +30,32 @@ namespace :gitlab do
backup = Backup::Manager.new backup = Backup::Manager.new
backup.unpack backup.unpack
Rake::Task["gitlab:backup:db:restore"].invoke unless backup.skipped?("db") unless backup.skipped?('db')
Rake::Task["gitlab:backup:repo:restore"].invoke unless backup.skipped?("repositories") warning = warning = <<-MSG.strip_heredoc
Rake::Task["gitlab:backup:uploads:restore"].invoke unless backup.skipped?("uploads") Before restoring the database we recommend removing all existing
Rake::Task["gitlab:backup:builds:restore"].invoke unless backup.skipped?("builds") tables to avoid future upgrade problems. Be aware that if you have
Rake::Task["gitlab:backup:artifacts:restore"].invoke unless backup.skipped?("artifacts") custom tables in the GitLab database these tables and all data will be
Rake::Task["gitlab:backup:lfs:restore"].invoke unless backup.skipped?("lfs") removed.
Rake::Task["gitlab:shell:setup"].invoke MSG
puts warning.red
answer = prompt('Confirm whether this task should remove all tables (yes/no): '.red, %w{yes no})
if answer == 'yes'
puts 'Removing all tables. Press `Ctrl-C` within 5 seconds to abort'.yellow
sleep(5)
# Drop all tables Load the schema to ensure we don't have any newer tables
# hanging out from a failed upgrade
$progress.puts 'Cleaning the database ... '.blue
Rake::Task['gitlab:db:drop_tables'].invoke
end
$progress.puts 'done'.green
Rake::Task['gitlab:backup:db:restore'].invoke
end
Rake::Task['gitlab:backup:repo:restore'].invoke unless backup.skipped?('repositories')
Rake::Task['gitlab:backup:uploads:restore'].invoke unless backup.skipped?('uploads')
Rake::Task['gitlab:backup:builds:restore'].invoke unless backup.skipped?('builds')
Rake::Task['gitlab:backup:artifacts:restore'].invoke unless backup.skipped?('artifacts')
Rake::Task['gitlab:backup:lfs:restore'].invoke unless backup.skipped?('lfs')
Rake::Task['gitlab:shell:setup'].invoke
backup.cleanup backup.cleanup
end end
......
namespace :gitlab do
namespace :db do
desc 'GitLab | Manually insert schema migration version'
task :mark_migration_complete, [:version] => :environment do |_, args|
unless args[:version]
puts "Must specify a migration version as an argument".red
exit 1
end
version = args[:version].to_i
if version == 0
puts "Version '#{args[:version]}' must be a non-zero integer".red
exit 1
end
sql = "INSERT INTO schema_migrations (version) VALUES (#{version})"
begin
ActiveRecord::Base.connection.execute(sql)
puts "Successfully marked '#{version}' as complete".green
rescue ActiveRecord::RecordNotUnique
puts "Migration version '#{version}' is already marked complete".yellow
end
end
desc 'Drop all tables'
task :drop_tables => :environment do
connection = ActiveRecord::Base.connection
tables = connection.tables
tables.delete 'schema_migrations'
# Truncate schema_migrations to ensure migrations re-run
connection.execute('TRUNCATE schema_migrations')
tables.each { |t| connection.execute("DROP TABLE #{t}") }
end
end
end
...@@ -3,9 +3,10 @@ require 'rake' ...@@ -3,9 +3,10 @@ require 'rake'
describe 'gitlab:app namespace rake task' do describe 'gitlab:app namespace rake task' do
before :all do before :all do
Rake.application.rake_require "tasks/gitlab/task_helpers" Rake.application.rake_require 'tasks/gitlab/task_helpers'
Rake.application.rake_require "tasks/gitlab/backup" Rake.application.rake_require 'tasks/gitlab/backup'
Rake.application.rake_require "tasks/gitlab/shell" Rake.application.rake_require 'tasks/gitlab/shell'
Rake.application.rake_require 'tasks/gitlab/db'
# empty task as env is already loaded # empty task as env is already loaded
Rake::Task.define_task :environment Rake::Task.define_task :environment
end end
...@@ -52,13 +53,14 @@ describe 'gitlab:app namespace rake task' do ...@@ -52,13 +53,14 @@ describe 'gitlab:app namespace rake task' do
it 'should invoke restoration on match' do it 'should invoke restoration on match' do
allow(YAML).to receive(:load_file). allow(YAML).to receive(:load_file).
and_return({ gitlab_version: gitlab_version }) and_return({ gitlab_version: gitlab_version })
expect(Rake::Task["gitlab:backup:db:restore"]).to receive(:invoke) expect(Rake::Task['gitlab:db:drop_tables']).to receive(:invoke)
expect(Rake::Task["gitlab:backup:repo:restore"]).to receive(:invoke) expect(Rake::Task['gitlab:backup:db:restore']).to receive(:invoke)
expect(Rake::Task["gitlab:backup:builds:restore"]).to receive(:invoke) expect(Rake::Task['gitlab:backup:repo:restore']).to receive(:invoke)
expect(Rake::Task["gitlab:backup:uploads:restore"]).to receive(:invoke) expect(Rake::Task['gitlab:backup:builds:restore']).to receive(:invoke)
expect(Rake::Task["gitlab:backup:artifacts:restore"]).to receive(:invoke) expect(Rake::Task['gitlab:backup:uploads:restore']).to receive(:invoke)
expect(Rake::Task["gitlab:backup:lfs:restore"]).to receive(:invoke) expect(Rake::Task['gitlab:backup:artifacts:restore']).to receive(:invoke)
expect(Rake::Task["gitlab:shell:setup"]).to receive(:invoke) expect(Rake::Task['gitlab:backup:lfs:restore']).to receive(:invoke)
expect(Rake::Task['gitlab:shell:setup']).to receive(:invoke)
expect { run_rake_task('gitlab:backup:restore') }.not_to raise_error expect { run_rake_task('gitlab:backup:restore') }.not_to raise_error
end end
end end
...@@ -177,17 +179,18 @@ describe 'gitlab:app namespace rake task' do ...@@ -177,17 +179,18 @@ describe 'gitlab:app namespace rake task' do
end end
it 'does not invoke repositories restore' do it 'does not invoke repositories restore' do
allow(Rake::Task["gitlab:shell:setup"]). allow(Rake::Task['gitlab:shell:setup']).
to receive(:invoke).and_return(true) to receive(:invoke).and_return(true)
allow($stdout).to receive :write allow($stdout).to receive :write
expect(Rake::Task["gitlab:backup:db:restore"]).to receive :invoke expect(Rake::Task['gitlab:db:drop_tables']).to receive :invoke
expect(Rake::Task["gitlab:backup:repo:restore"]).not_to receive :invoke expect(Rake::Task['gitlab:backup:db:restore']).to receive :invoke
expect(Rake::Task["gitlab:backup:uploads:restore"]).not_to receive :invoke expect(Rake::Task['gitlab:backup:repo:restore']).not_to receive :invoke
expect(Rake::Task["gitlab:backup:builds:restore"]).to receive :invoke expect(Rake::Task['gitlab:backup:uploads:restore']).not_to receive :invoke
expect(Rake::Task["gitlab:backup:artifacts:restore"]).to receive :invoke expect(Rake::Task['gitlab:backup:builds:restore']).to receive :invoke
expect(Rake::Task["gitlab:backup:lfs:restore"]).to receive :invoke expect(Rake::Task['gitlab:backup:artifacts:restore']).to receive :invoke
expect(Rake::Task["gitlab:shell:setup"]).to receive :invoke expect(Rake::Task['gitlab:backup:lfs:restore']).to receive :invoke
expect(Rake::Task['gitlab:shell:setup']).to receive :invoke
expect { run_rake_task('gitlab:backup:restore') }.not_to raise_error expect { run_rake_task('gitlab:backup:restore') }.not_to raise_error
end end
end end
......
Markdown is supported
0%
or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment