Skip to content

G
gitlab-ce
Commit 67b38c8d authored by Jacob Vosmaer's avatar Jacob Vosmaer
Browse files
Options

Update/improve CI migration docs for 8.0.1

parent 8ae026a6
Hide whitespace changes
Inline Side-by-side
Showing with 147 additions and 189 deletions
doc/migrate_ci_to_ce/README.md
View file @ 67b38c8d
...@@ -7,230 +7,198 @@ into the CE and EE applications. ...@@ -7,230 +7,198 @@ into the CE and EE applications.
This guide will detail the process of migrating your CI installation and data This guide will detail the process of migrating your CI installation and data
into your GitLab CE or EE installation. into your GitLab CE or EE installation.
### Before we begin We recommend that you read through the entire migration process in this
document before beginning.
**You need to have a working installation of GitLab CI version 8.0 to perform ### Overview
this migration. The older versions are not supported and will most likely break
this migration procedure.**
This migration cannot be performed online and takes a significant amount of In this document we assume you have a GitLab server and a GitLab CI server. It
time. Make sure to plan ahead. does not matter if these are the same machine.
If you are running a version of GitLab CI prior to 8.0 please follow the The migration consists of three parts: updating GitLab and GitLab CI, moving
appropriate [update guide](https://gitlab.com/gitlab-org/gitlab-ci/tree/master/doc/update/) data, and redirecting traffic.
before proceeding.
The migration is divided into four parts and covers both manual and Omnibus Please note that CI builds triggered on your GitLab server in the time between
installations: updating to 8.0 and finishing the migration will be lost.
1. [GitLab CI](#part-i-gitlab-ci) ### Before upgrading
1. [Gitlab CE (or EE)](#part-ii-gitlab-ce-or-ee)
1. [Nginx configuration](#part-iii-nginx-configuration)
1. [Finishing Up](#part-iv-finishing-up)
### Part I: GitLab CI - (1) Make sure that the backup script on both servers can connect to the database.
#### 1. Stop GitLab CI ```
# CI server
# Manual installation # Omnibus
sudo service gitlab_ci stop sudo gitlab-ci-rake backup:create
# Omnibus installation
sudo gitlab-ctl stop ci-unicorn ci-sidekiq
#### 2. Create a backup
The migration procedure modifies the structure of the CI database. If something
goes wrong, you will not be able to revert to a previous version without a
backup.
If your GitLab CI installation uses **MySQL** and your GitLab CE (or EE)
installation uses **PostgreSQL** you'll need to convert the CI database by
setting a `MYSQL_TO_POSTGRESQL` flag.
If you use the Omnibus package you most likely use **PostgreSQL** on both GitLab
CE (or EE) and CI.
You can check which database each install is using by viewing their
database configuration files:
cat /home/gitlab_ci/gitlab-ci/config/database.yml
cat /home/git/gitlab/config/database.yml
- If both applications use the same database `adapter`, create the backup with
this command:
# Manual installation
cd /home/gitlab_ci/gitlab-ci
sudo -u gitlab_ci -H bundle exec rake backup:create RAILS_ENV=production
# Omnibus installation
sudo gitlab-ci-rake backup:create
- If CI uses MySQL, and CE (or EE) uses PostgreSQL, create the backup with this
command (note the `MYSQL_TO_POSTGRESQL` flag):
# Manual installation
cd /home/gitlab_ci/gitlab-ci
sudo -u gitlab_ci -H bundle exec rake backup:create RAILS_ENV=production MYSQL_TO_POSTGRESQL=1
# Omnibus installation
sudo gitlab-ci-rake backup:create MYSQL_TO_POSTGRESQL=1
#### 3. Remove cronjob
**Note:** This step is only required for **manual installations**. Omnibus users
can [skip to the next step](#part-ii-gitlab-ce-or-ee).
# Manual installation
cd /home/gitlab_ci/gitlab-ci
sudo -u gitlab_ci -H bundle exec whenever --clear-crontab
### Part II: GitLab CE (or EE)
#### 1. Ensure GitLab is updated
Your GitLab CE or EE installation **must be version 8.0**. If it's not, follow
the [update guide](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/doc/update/7.14-to-8.0.md)
before proceeding.
If you use the Omnibus packages simply run `apt-get upgrade` to install the
latest version.
#### 2. Prevent CI usage during the migration process
As an administrator, go to **Admin Area** -> **Settings**, and under **Continuous
Integration** uncheck **Disable to prevent CI usage until rake ci:migrate is run
(8.0 only)**.
This will disable the CI integration and prevent users from creating CI projects
until the migration process is completed.
#### 3. Stop GitLab
Before you can migrate data you need to stop the GitLab service first:
# Manual installation
sudo service gitlab stop
# Omnibus installation
sudo gitlab-ctl stop unicorn sidekiq
#### 4. Create a backup # Source
cd /home/gitlab_ci/gitlab-ci
sudo -u gitlab_ci -H bundle exec rake backup:create RAILS_ENV=production
```
This migration poses a **significant risk** of breaking your GitLab ```
installation. Create a backup before proceeding: # GitLab server
# Omnibus
sudo gitlab-rake gitlab:backup:create SKIP=repositories,uploads
# Manual installation # Source
cd /home/git/gitlab cd /home/git/gitlab
sudo -u git -H bundle exec rake gitlab:backup:create RAILS_ENV=production sudo -u git -H bundle exec rake gitlab:backup:create RAILS_ENV=production SKIP=repositories,uploads
```
# Omnibus installation If this fails you need to fix it before upgrading to 8.0. Also see
sudo gitlab-rake gitlab:backup:create https://about.gitlab.com/getting-help/
It's possible to speed up backup creation by skipping repositories and uploads: - (2) Check what databases you use on your GitLab server and your CI server.
Look for the 'adapter:' line. If your CI server and your GitLab server use
the same database adapter no special care is needed. If your CI server uses
MySQL and your GitLab server uses PostgreSQL you need to pass a special option
during the 'Moving data' part. **If your CI server uses PostgreSQL and your
GitLab server uses MySQL you cannot migrate your CI data to GitLab 8.0.***
# Manual installation - (3) Decide where to store CI build traces on GitLab server. GitLab CI uses
cd /home/git/gitlab files on disk to store CI build traces. The default path for these build
sudo -u git -H bundle exec rake gitlab:backup:create RAILS_ENV=production SKIP=repositories,uploads traces is `/var/opt/gitlab/gitlab-ci/build` (Omnibus) or
`/home/git/gitlab/builds` (Source). If you are storing your repository data in
a special location, or if you are using NFS, you should make sure that you
store build traces on the same storage as your Git repositories.
# Omnibus installation ```
sudo gitlab-rake gitlab:backup:create SKIP=repositories,uploads # CI server
# Omnibus
sudo gitlab-ci-rake env:info
#### 5. Copy secret tokens from CI # Source
cd /home/gitlab_ci/gitlab-ci
sudo -u gitlab_ci -H bundle exec rake env:info RAILS_ENV=production
```
The `secrets.yml` file stores encryption keys for secure variables. ```
# GitLab server
# Omnibus
sudo gitlab-rake gitlab:env:info
- **Manual installations** need to copy the contents of GitLab CI's # Source
`config/secrets.yml` file to the same file in GitLab CE: cd /home/git/gitlab
sudo -u git -H bundle exec rake gitlab:env:info RAILS_ENV=production
```
```bash ### Upgrading
# Manual installation
sudo cp /home/gitlab_ci/gitlab-ci/config/secrets.yml /home/git/gitlab/config/secrets.yml
sudo chown git:git /home/git/gitlab/config/secrets.yml
sudo chown 0600 /home/git/gitlab/config/secrets.yml
```
- **Omnibus installations** where GitLab CI and CE (or EE) are on the same From this point on, GitLab CI will be unavailable for your end users.
server don't need to do anything further, because the secrets are stored in
`/etc/gitlab/gitlab-secrets.json`.
- **Omnibus installations** where GitLab CI is on a different server than CE (or - (1) First upgrade your GitLab server to version 8.0:
EE) will need to: https://about.gitlab.com/update/
1. On the CI server, copy the `db_key_base` value from - (2) After you update, go to the admin panel and temporarily disable CI. As
`/etc/gitlab/gitlab-secrets.json` an administrator, go to **Admin Area** -> **Settings**, and under
1. On the CE (or EE) server, add `gitlab_ci['db_key_base'] = **Continuous Integration** uncheck **Disable to prevent CI usage until rake
"VALUE_FROM_ABOVE"` to the `/etc/gitlab/gitlab.rb` file and run `sudo ci:migrate is run (8.0 only)**.
gitlab-ctl reconfigure` - (3) If you want to use custom CI settings (e.g. change where builds are
stored), please update `/etc/gitlab/gitlab.rb` (Omnibus) or
`/home/git/gitlab/config/gitlab.yml` (Source).
- (4) Now upgrade GitLab CI to version 8.0. If you are using Omnibus packages,
this may have already happened when you upgraded GitLab to 8.0.
#### 6. New configuration options for `gitlab.yml` - (5) Disable GitLab CI after upgrading to 8.0.
**Note:** This step is only required for **manual installations**. Omnibus users ```
can [skip to the next step](#7-copy-backup-from-gitlab-ci). # CI server
# Omnibus
sudo gitlab-ctl stop ci-unicorn
sudo gitlab-ctl stop ci-sidekiq
# Source
sudo service gitlab_ci stop
cd /home/gitlab_ci/gitlab-ci
sudo -u gitlab_ci -H bundle exec whenever --clear-crontab
```
There are new configuration options available for `gitlab.yml`. View them with ### Moving data
the command below and apply them manually to your current `gitlab.yml`:
git diff origin/7-14-stable:config/gitlab.yml.example origin/8-0-stable:config/gitlab.yml.example - (1) Move the database encryption key from your CI server to your GitLab server.
The new options include configuration settings for GitLab CI. ```
# CI server
# Omnibus
sudo gitlab-ci-rake backup:show_secrets
#### 7. Copy backup from GitLab CI # Source
cd /home/gitlab_ci/gitlab-ci
sudo -u gitlab_ci -H bundle exec rake backup:show_secrets RAILS_ENV=production
```
```bash - (2) Create your final CI data export. If you are converting from MySQL to
# Manual installation PostgreSQL, add ` MYSQL_TO_POSTGRESQL=1` to the end of the rake command. When
sudo cp -v /home/gitlab_ci/gitlab-ci/tmp/backups/*_gitlab_ci_backup.tar /home/git/gitlab/tmp/backups the command finishes it will print the path to your data export archive; you
sudo chown git:git /home/git/gitlab/tmp/backups/*_gitlab_ci_backup.tar will need this file later.
# Omnibus installation
sudo cp -v /var/opt/gitlab/ci-backups/*_gitlab_ci_backup.tar /var/opt/gitlab/backups/
sudo chown git:git /var/opt/gitlab/backups/*_gitlab_ci_backup.tar
``` ```
# CI server
# Omnibus
sudo gitlab-ci-rake backup:create
If moving across the servers you can use `scp`. # Source
However, this requires you to provide an authorized key or password to login to cd /home/git/gitlab
the GitLab CE (or EE) server from the CI server. You can try to use ssh-agent sudo -u git -H bundle exec rake backup:create RAILS_ENV=production
from your local machine to have that: login to your GitLab CI server using ```
`ssh -A`.
```bash - (3) Copy your CI data archive to your GitLab server. If you were running
# Manual installation GitLab and GitLab CI on the same server you can skip this step. There are
scp /home/gitlab_ci/gitlab-ci/tmp/backups/*_gitlab_ci_backup.tar root@gitlab.example.com:/home/git/gitlab/tmp/backup many ways to do this, below we use SSH agent forwarding and 'scp', which will
be easy and fast for most setups. You can also copy the data archive first from
the CI server to your laptop and then from your laptop to the GitLab server.
# Omnibus installation ```
scp /var/opt/gitlab/ci-backups/*_gitlab_ci_backup.tar root@gitlab.example.com:/var/opt/gitlab/backups/ # Start from your laptop
ssh -A ci_admin@ci_server.example
# Now on the CI server
scp /path/to/12345_gitlab_ci_backup.tar gitlab_admin@gitlab_server.example:~
``` ```
#### 8. Import GitLab CI backup - (4) Make the CI data archive discoverable for GitLab. We assume below that
you store backups in the default path, adjust the command if necessary.
Now you'll import the GitLab CI database dump that you created earlier into the ```
GitLab CE or EE database: # GitLab server
# Omnibus
sudo mv /path/to/12345_gitlab_ci_backup.tar /var/opt/gitlab/backups/
# Manual installation # Source
sudo -u git -H bundle exec rake ci:migrate RAILS_ENV=production sudo mv /path/to/12345_gitlab_ci_backup.tar /home/git/gitlab/tmp/backups/
```
# Omnibus installation - (5) Import the CI data into GitLab.
sudo gitlab-rake ci:migrate
This task will take some time. ```
# GitLab server
# Omnibus
sudo gitlab-rake ci:migrate
This migration task automatically re-enables the CI setting that you # Source
[disabled earlier](#2-prevent-ci-usage-during-the-migration-process). cd /home/git/gitlab
sudo -u git -H bundle exec rake ci:migrate RAILS_ENV=production
```
#### 9. Start GitLab - (6) Restart GitLab
You can start GitLab CE (or EE) now and see if everything is working: ```
# GitLab server
# Omnibus
sudo gitlab-ctl hup unicorn
sudo gitlab-ctl restart sidekiq
# Manual installation # Source
sudo service gitlab start sudo service gitlab reload
```
# Omnibus installation ### Redirecting traffic
sudo gitlab-ctl restart unicorn sidekiq
### Part III: Nginx configuration If you were running GitLab CI with Omnibus packages and you were using the
internal NGINX configuration your CI service should now be available both at
`ci.example.com` (the old address) and `gitlab.example.com/ci`. You are done!
This section is only required for **manual installations**. Omnibus users can If you installed GitLab CI from source we now need to configure a redirect in
[skip to the final step](#part-iv-finishing-up). NGINX so that existing CI runners can keep using the old CI server address, and
so that existing links to your CI server keep working.
#### 1. Update Nginx configuration #### 1. Update Nginx configuration
...@@ -296,16 +264,6 @@ You should also make sure that you can: ...@@ -296,16 +264,6 @@ You should also make sure that you can:
sudo /etc/init.d/nginx restart sudo /etc/init.d/nginx restart
### Part IV: Finishing Up
If everything went well you should be able to access your migrated CI install by
visiting `https://gitlab.example.com/ci/`. If you visit the old GitLab CI
address, you should be redirected to the new one.
**Enjoy!**
### Troubleshooting
#### Restore from backup #### Restore from backup
If something went wrong and you need to restore a backup, consult the [Backup If something went wrong and you need to restore a backup, consult the [Backup
......
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment
GitLab Nexedi Edition | About GitLab | About Nexedi | 沪ICP备2021021310号-2 | 沪ICP备2021021310号-7