README.md 10.4 KB
Newer Older
1
## Migrate GitLab CI to GitLab CE or EE
2

3 4 5
Beginning with version 8.0 of GitLab Community Edition (CE) and Enterprise
Edition (EE), GitLab CI is no longer its own application, but is instead built
into the CE and EE applications.
6

7 8
This guide will detail the process of migrating your CI installation and data
into your GitLab CE or EE installation.
9

10
### Before we begin
11

Kamil Trzcinski's avatar
Kamil Trzcinski committed
12
**You need to have a working installation of GitLab CI version 8.0 to perform
13 14
this migration. The older versions are not supported and will most likely break
this migration procedure.**
15

16 17
This migration cannot be performed online and takes a significant amount of
time. Make sure to plan ahead.
18

Kamil Trzcinski's avatar
Kamil Trzcinski committed
19
If you are running a version of GitLab CI prior to 8.0 please follow the
20 21
appropriate [update guide](https://gitlab.com/gitlab-org/gitlab-ci/tree/master/doc/update/)
before proceeding.
22

23 24
The migration is divided into four parts and covers both manual and Omnibus
installations:
25 26 27

1. [GitLab CI](#part-i-gitlab-ci)
1. [Gitlab CE (or EE)](#part-ii-gitlab-ce-or-ee)
28 29
1. [Nginx configuration](#part-iii-nginx-configuration)
1. [Finishing Up](#part-iv-finishing-up)
30 31 32 33

### Part I: GitLab CI

#### 1. Stop GitLab CI
34

35
    # Manual installation
36
    sudo service gitlab_ci stop
Robert Speicher's avatar
Robert Speicher committed
37

38 39
    # Omnibus installation
    sudo gitlab-ctl stop ci-unicorn ci-sidekiq
Robert Speicher's avatar
Robert Speicher committed
40

41 42 43 44
#### 2. Create a backup

The migration procedure modifies the structure of the CI database. If something
goes wrong, you will not be able to revert to a previous version without a
45
backup.
46

47 48 49
If your GitLab CI installation uses **MySQL** and your GitLab CE (or EE)
installation uses **PostgreSQL** you'll need to convert the CI database by
setting a `MYSQL_TO_POSTGRESQL` flag.
50

51 52
If you use the Omnibus package you most likely use **PostgreSQL** on both GitLab
CE (or EE) and CI.
53

54 55
You can check which database each install is using by viewing their
database configuration files:
56

Kamil Trzcinski's avatar
Kamil Trzcinski committed
57 58
    cat /home/gitlab_ci/gitlab-ci/config/database.yml
    cat /home/git/gitlab/config/database.yml
59

60 61 62
- If both applications use the same database `adapter`, create the backup with
  this command:

63 64 65
        # Manual installation
        cd /home/gitlab_ci/gitlab-ci
        sudo -u gitlab_ci -H bundle exec backup:create RAILS_ENV=production
Robert Speicher's avatar
Robert Speicher committed
66

67 68
        # Omnibus installation
        sudo gitlab-ci-rake backup:create
Robert Speicher's avatar
Robert Speicher committed
69

70 71
- If CI uses MySQL, and CE (or EE) uses PostgreSQL, create the backup with this
  command (note the `MYSQL_TO_POSTGRESQL` flag):
72

73 74 75
        # Manual installation
        cd /home/gitlab_ci/gitlab-ci
        sudo -u gitlab_ci -H bundle exec backup:create RAILS_ENV=production MYSQL_TO_POSTGRESQL=1
Robert Speicher's avatar
Robert Speicher committed
76

77 78
        # Omnibus installation
        sudo gitlab-ci-rake backup:create MYSQL_TO_POSTGRESQL=1
Robert Speicher's avatar
Robert Speicher committed
79

80 81 82 83
#### 3. Remove cronjob

**Note:** This step is only required for **manual installations**. Omnibus users
can [skip to the next step](#part-ii-gitlab-ce-or-ee).
84

85
    # Manual installation
Kamil Trzcinski's avatar
Kamil Trzcinski committed
86 87
    cd /home/gitlab_ci/gitlab-ci
    sudo -u gitlab_ci -H bundle exec whenever --clear-crontab
88 89 90 91 92 93

### Part II: GitLab CE (or EE)

#### 1. Ensure GitLab is updated

Your GitLab CE or EE installation **must be version 8.0**. If it's not, follow
94 95
the [update guide](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/doc/update/7.14-to-8.0.md)
before proceeding.
96

97 98
If you use the Omnibus packages simply run `apt-get upgrade` to install the
latest version.
99

100
#### 2. Prevent CI usage during the migration process
Kamil Trzcinski's avatar
Kamil Trzcinski committed
101

102 103 104
As an administrator, go to **Admin Area** -> **Settings**, and under **Continuous
Integration** uncheck **Disable to prevent CI usage until rake ci:migrate is run
(8.0 only)**.
Kamil Trzcinski's avatar
Kamil Trzcinski committed
105

106 107
This will disable the CI integration and prevent users from creating CI projects
until the migration process is completed.
Kamil Trzcinski's avatar
Kamil Trzcinski committed
108 109

#### 3. Stop GitLab
110 111

Before you can migrate data you need to stop the GitLab service first:
112

113
    # Manual installation
114
    sudo service gitlab stop
Robert Speicher's avatar
Robert Speicher committed
115

116 117
    # Omnibus installation
    sudo gitlab-ctl stop unicorn sidekiq
118

Kamil Trzcinski's avatar
Kamil Trzcinski committed
119
#### 4. Create a backup
120 121 122

This migration poses a **significant risk** of breaking your GitLab
installation. Create a backup before proceeding:
123

124
    # Manual installation
125 126
    cd /home/git/gitlab
    sudo -u git -H bundle exec rake gitlab:backup:create RAILS_ENV=production
Robert Speicher's avatar
Robert Speicher committed
127

128 129
    # Omnibus installation
    sudo gitlab-rake gitlab:backup:create
Robert Speicher's avatar
Robert Speicher committed
130

131
It's possible to speed up backup creation by skipping repositories and uploads:
Kamil Trzcinski's avatar
Kamil Trzcinski committed
132

133
    # Manual installation
Kamil Trzcinski's avatar
Kamil Trzcinski committed
134 135
    cd /home/git/gitlab
    sudo -u git -H bundle exec rake gitlab:backup:create RAILS_ENV=production SKIP=repositories,uploads
Robert Speicher's avatar
Robert Speicher committed
136

137 138
    # Omnibus installation
    sudo gitlab-rake gitlab:backup:create SKIP=repositories,uploads
139

Kamil Trzcinski's avatar
Kamil Trzcinski committed
140
#### 5. Copy secret tokens from CI
141 142 143

The `secrets.yml` file stores encryption keys for secure variables.

144 145
- **Manual installations** need to copy the contents of GitLab CI's
  `config/secrets.yml` file to the same file in GitLab CE:
146

147
    ```bash
148
    # Manual installation
149 150 151
    sudo cp /home/gitlab_ci/gitlab-ci/config/secrets.yml /home/git/gitlab/config/secrets.yml
    sudo chown git:git /home/git/gitlab/config/secrets.yml
    sudo chown 0600 /home/git/gitlab/config/secrets.yml
152
    ```
153

154 155 156
- **Omnibus installations** where GitLab CI and CE (or EE) are on the same
  server don't need to do anything further, because the secrets are stored in
  `/etc/gitlab/gitlab-secrets.json`.
157

158 159 160 161 162 163 164
- **Omnibus installations** where GitLab CI is on a different server than CE (or
  EE) will need to:
    1. On the CI server, copy the `db_key_base` value from
       `/etc/gitlab/gitlab-secrets.json`
    1. On the CE (or EE) server, add `gitlab_ci['db_key_base'] =
       "VALUE_FROM_ABOVE"` to the `/etc/gitlab/gitlab.rb` file and run `sudo
       gitlab-ctl reconfigure`
165

166 167
#### 6. New configuration options for `gitlab.yml`

168 169
**Note:** This step is only required for **manual installations**. Omnibus users
can [skip to the next step](#7-copy-backup-from-gitlab-ci).
170 171 172

There are new configuration options available for `gitlab.yml`. View them with
the command below and apply them manually to your current `gitlab.yml`:
173

174
    git diff origin/7-14-stable:config/gitlab.yml.example origin/8-0-stable:config/gitlab.yml.example
175

176
The new options include configuration settings for GitLab CI.
177

Kamil Trzcinski's avatar
Kamil Trzcinski committed
178
#### 7. Copy backup from GitLab CI
179

180 181 182 183
```bash
# Manual installation
sudo cp -v /home/gitlab_ci/gitlab-ci/tmp/backups/*_gitlab_ci_backup.tar /home/git/gitlab/tmp/backups
sudo chown git:git /home/git/gitlab/tmp/backups/*_gitlab_ci_backup.tar
Robert Speicher's avatar
Robert Speicher committed
184

185 186 187 188
# Omnibus installation
sudo cp -v /var/opt/gitlab/ci-backups/*_gitlab_ci_backup.tar /var/opt/gitlab/backups/
sudo chown git:git /var/opt/gitlab/backups/*_gitlab_ci_backup.tar
```
189

190 191 192
If moving across the servers you can use `scp`.
However, this requires you to provide an authorized key or password to login to
the GitLab CE (or EE) server from the CI server. You can try to use ssh-agent
193 194
from your local machine to have that: login to your GitLab CI server using
`ssh -A`.
Kamil Trzcinski's avatar
Kamil Trzcinski committed
195

196 197 198
```bash
# Manual installation
scp /home/gitlab_ci/gitlab-ci/tmp/backups/*_gitlab_ci_backup.tar root@gitlab.example.com:/home/git/gitlab/tmp/backup
Robert Speicher's avatar
Robert Speicher committed
199

200 201 202
# Omnibus installation
scp /var/opt/gitlab/ci-backups/*_gitlab_ci_backup.tar root@gitlab.example.com:/var/opt/gitlab/backups/
```
Kamil Trzcinski's avatar
Kamil Trzcinski committed
203 204

#### 8. Import GitLab CI backup
205

206 207
Now you'll import the GitLab CI database dump that you created earlier into the
GitLab CE or EE database:
208

209
    # Manual installation
Kamil Trzcinski's avatar
Kamil Trzcinski committed
210
    sudo -u git -H bundle exec rake ci:migrate RAILS_ENV=production
Robert Speicher's avatar
Robert Speicher committed
211

212 213
    # Omnibus installation
    sudo gitlab-rake ci:migrate
214

Kamil Trzcinski's avatar
Kamil Trzcinski committed
215
This task will take some time.
216

Kamil Trzcinski's avatar
Kamil Trzcinski committed
217
#### 9. Start GitLab
218

219
You can start GitLab CE (or EE) now and see if everything is working:
220

221
    # Manual installation
222
    sudo service gitlab start
Robert Speicher's avatar
Robert Speicher committed
223

224 225
    # Omnibus installation
    sudo gitlab-ctl restart unicorn sidekiq
226

227
### Part III: Nginx configuration
228

229 230
This section is only required for **manual installations**. Omnibus users can
[skip to the final step](#part-iv-finishing-up).
231

232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291
#### 1. Update Nginx configuration

To ensure that your existing CI runners are able to communicate with the
migrated installation, and that existing build triggers still work, you'll need
to update your Nginx configuration to redirect requests for the old locations to
the new ones.

Edit `/etc/nginx/sites-available/gitlab_ci` and paste:

```nginx
# GITLAB CI
server {
  listen 80 default_server;         # e.g., listen 192.168.1.1:80;
  server_name YOUR_CI_SERVER_FQDN;  # e.g., server_name source.example.com;

  access_log  /var/log/nginx/gitlab_ci_access.log;
  error_log   /var/log/nginx/gitlab_ci_error.log;

  # expose API to fix runners
  location /api {
    proxy_read_timeout    300;
    proxy_connect_timeout 300;
    proxy_redirect        off;
    proxy_set_header      X-Real-IP $remote_addr;

    # You need to specify your DNS servers that are able to resolve YOUR_GITLAB_SERVER_FQDN
    resolver 8.8.8.8 8.8.4.4;
    proxy_pass $scheme://YOUR_GITLAB_SERVER_FQDN/ci$request_uri;
  }

  # expose build endpoint to allow trigger builds
  location ~ ^/projects/\d+/build$ {
    proxy_read_timeout    300;
    proxy_connect_timeout 300;
    proxy_redirect        off;
    proxy_set_header      X-Real-IP $remote_addr;

    # You need to specify your DNS servers that are able to resolve YOUR_GITLAB_SERVER_FQDN
    resolver 8.8.8.8 8.8.4.4;
    proxy_pass $scheme://YOUR_GITLAB_SERVER_FQDN/ci$request_uri;
  }

  # redirect all other CI requests
  location / {
    return 301 $scheme://YOUR_GITLAB_SERVER_FQDN/ci$request_uri;
  }

  # adjust this to match the largest build log your runners might submit,
  # set to 0 to disable limit
  client_max_body_size 10m;
}
```

Make sure you substitute these placeholder values with your real ones:

1. `YOUR_CI_SERVER_FQDN`: The existing public-facing address of your GitLab CI
   install (e.g., `ci.gitlab.com`).
1. `YOUR_GITLAB_SERVER_FQDN`: The current public-facing address of your GitLab
   CE (or EE) install (e.g., `gitlab.com`).

292 293
**Make sure not to remove the `/ci$request_uri` part. This is required to
properly forward the requests.**
294 295 296

You should also make sure that you can:

297
1. `curl https://YOUR_GITLAB_SERVER_FQDN/` from your previous GitLab CI server.
298
1. `curl https://YOUR_CI_SERVER_FQDN/` from your GitLab CE (or EE) server.
299

300
#### 2. Check Nginx configuration
301 302 303

    sudo nginx -t

304
#### 3. Restart Nginx
305 306 307

    sudo /etc/init.d/nginx restart

308
### Part IV: Finishing Up
309

310
If everything went well you should be able to access your migrated CI install by
311 312
visiting `https://gitlab.example.com/ci/`. If you visit the old GitLab CI
address, you should be redirected to the new one.
313 314

**Enjoy!**
315 316 317 318 319 320 321

### Troubleshooting

#### Restore from backup

If something went wrong and you need to restore a backup, consult the [Backup
restoration](../raketasks/backup_restore.md) guide.