Commit 2fea2997 authored by Marcel Amirault's avatar Marcel Amirault

Merge branch 'eread/refactor-backup-restore-raketask' into 'master'

Refactor backup and restore Rake tasks page

See merge request gitlab-org/gitlab!29745
parents 0f58b134 5c7568bc
......@@ -62,6 +62,7 @@
"Gmail",
"Google",
"Grafana",
"Gzip",
"Helm",
"HipChat",
"Ingress",
......
......@@ -12,5 +12,7 @@ swap:
param: parameter
params: parameters
postgres: PostgreSQL
raketask: Rake task
raketasks: Rake tasks
self hosted: self-managed
self-hosted: self-managed
......@@ -115,6 +115,7 @@ Google
Gradle
Grafana
gravatar
Gzip
hardcode
hardcoded
hardcodes
......
......@@ -44,7 +44,7 @@ and all **secondary** nodes:
Now that the update process is complete, you may want to check whether
everything is working correctly:
1. Run the Geo raketask on all nodes, everything should be green:
1. Run the Geo Rake task on all nodes, everything should be green:
```shell
sudo gitlab-rake gitlab:geo:check
......
......@@ -76,7 +76,7 @@ Learn how to install, configure, update, and maintain your GitLab instance.
### Maintaining GitLab
- [Raketasks](../raketasks/README.md): Perform various tasks for maintenance, backups, automatic webhooks setup, and more.
- [Rake tasks](../raketasks/README.md): Perform various tasks for maintenance, backups, automatic webhooks setup, and more.
- [Backup and restore](../raketasks/backup_restore.md): Backup and restore your GitLab instance.
- [Operations](operations/index.md): Keeping GitLab up and running (clean up Redis sessions, moving repositories, Sidekiq MemoryKiller, Unicorn).
- [Restart GitLab](restart_gitlab.md): Learn how to restart GitLab and its components.
......@@ -98,7 +98,7 @@ Learn how to install, configure, update, and maintain your GitLab instance.
- [Mattermost](https://docs.gitlab.com/omnibus/gitlab-mattermost/): Integrate with [Mattermost](https://mattermost.com), an open source, private cloud workplace for web messaging.
- [PlantUML](integration/plantuml.md): Create simple diagrams in AsciiDoc and Markdown documents
created in snippets, wikis, and repos.
created in snippets, wikis, and repositories.
- [Web terminals](integration/terminal.md): Provide terminal access to your applications deployed to Kubernetes from within GitLab's CI/CD [environments](../ci/environments.md#web-terminals).
## User settings and permissions
......
......@@ -378,7 +378,7 @@ and [projects APIs](../api/projects.md).
When GitLab receives an artifacts archive, an archive metadata file is also
generated by [GitLab Workhorse](https://gitlab.com/gitlab-org/gitlab-workhorse). This metadata file describes all the entries
that are located in the artifacts archive itself.
The metadata file is in a binary format, with additional GZIP compression.
The metadata file is in a binary format, with additional Gzip compression.
GitLab does not extract the artifacts archive in order to save space, memory
and disk I/O. It instead inspects the metadata file which contains all the
......
......@@ -20,7 +20,7 @@ Although build images are built automatically via GitLab CI/CD, you can build
and tag all tooling images locally:
1. Make sure you have [Docker installed](https://docs.docker.com/install/).
1. Make sure you're on the `dockerfiles/` directory of the `gitlab-docs` repo.
1. Make sure you're in the `dockerfiles/` directory of the `gitlab-docs` repository.
1. Build the images:
```shell
......@@ -46,7 +46,7 @@ products, we need to add a
1. Check that there is a [stable branch created](https://gitlab.com/gitlab-org/charts/gitlab/-/branches)
for the new chart version. If you're unsure or can't find it, drop a line in
the `#g_delivery` channel.
1. Make sure you're on the root path of the `gitlab-docs` repo.
1. Make sure you're in the root path of the `gitlab-docs` repository.
1. Open `content/_data/chart_versions.yaml` and add the new stable branch version using the
version mapping. Note that only the `major.minor` version is needed.
1. Create a new merge request and merge it.
......@@ -61,14 +61,14 @@ this first step.
The single docs version must be created before the release merge request, but
this needs to happen when the stable branches for all products have been created.
1. Make sure you're on the root path of the `gitlab-docs` repo.
1. Make sure you're in the root path of the `gitlab-docs` repository.
1. Make sure your `master` is updated:
```shell
git pull origin master
```
1. Run the raketask to create the single version:
1. Run the Rake task to create the single version:
```shell
./bin/rake "release:single[12.0]"
......@@ -109,7 +109,7 @@ Visit `http://localhost:4000/12.0/` to see if everything works correctly.
Now it's time to create the monthly release merge request that adds the new
version and rotates the old one:
1. Make sure you're on the root path of the `gitlab-docs` repo.
1. Make sure you're in the root path of the `gitlab-docs` repository.
1. Create a branch `release-X-Y`:
```shell
......@@ -158,10 +158,10 @@ the dropdown are included in the unmerged `release-X-Y` branch.
The content of `content/_data/versions.yaml` needs to change for all online
versions:
1. Run the raketask that will create all the respective merge requests needed to
1. Run the Rake task that will create all the respective merge requests needed to
update the dropdowns and will be set to automatically be merged when their
pipelines succeed. The `release-X-Y` branch needs to be present locally,
otherwise the raketask will fail:
otherwise the Rake task will fail:
```shell
./bin/rake release:dropdowns
......
......@@ -801,7 +801,7 @@ known tool is [`pngquant`](https://pngquant.org/), which is cross-platform and
open source. Install it by visiting the official website and following the
instructions for your OS.
GitLab has a [raketask](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/tasks/pngquant.rake)
GitLab has a [Rake task](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/tasks/pngquant.rake)
that you can use to automate the process. In the root directory of your local
copy of `https://gitlab.com/gitlab-org/gitlab`, run in a terminal:
......
......@@ -20,7 +20,7 @@ The following tools are used:
1. [`gettext_i18n_rails`](https://github.com/grosser/gettext_i18n_rails): this
gem allow us to translate content from models, views and controllers. Also
it gives us access to the following raketasks:
it gives us access to the following Rake tasks:
- `rake gettext:find`: Parses almost all the files from the
Rails application looking for content that has been marked for
translation. Finally, it updates the PO files with the new content that
......@@ -30,7 +30,7 @@ The following tools are used:
1. [`gettext_i18n_rails_js`](https://github.com/webhippie/gettext_i18n_rails_js):
this gem is useful to make the translations available in JavaScript. It
provides the following raketask:
provides the following Rake task:
- `rake gettext:po_to_json`: Reads the contents from the PO files and
generates JSON files containing all the available translations.
......
......@@ -648,8 +648,8 @@ Read more on configuring an
## Backup and restore
GitLab provides [a tool to backup](../../raketasks/backup_restore.md#creating-a-backup-of-the-gitlab-system)
and restore its Git data, database, attachments, LFS objects, etc.
GitLab provides [a tool to back up](../../raketasks/backup_restore.md#back-up-gitlab)
and restore its Git data, database, attachments, LFS objects, and so on.
Some important things to know:
......@@ -675,7 +675,7 @@ For GitLab 12.1 and earlier, use `gitlab-rake gitlab:backup:create`.
### Restoring GitLab from a backup
To restore GitLab, first review the [restore documentation](../../raketasks/backup_restore.md#restore),
To restore GitLab, first review the [restore documentation](../../raketasks/backup_restore.md#restore-gitlab),
and primarily the restore prerequisites. Then, follow the steps under the
[Omnibus installations section](../../raketasks/backup_restore.md#restore-for-omnibus-gitlab-installations).
......
# Backing up and restoring GitLab
![backup banner](backup_hrz.png)
GitLab provides Rake tasks for backing up and restoring GitLab instances.
An application data backup creates an archive file that contains the database,
all repositories and all attachments.
......@@ -14,31 +14,26 @@ from one server to another is through backup restore.
In order to be able to backup and restore, you need two essential tools
installed on your system.
### Rsync
- **Rsync**: If you installed GitLab:
- Using the Omnibus package, you're all set.
- From source, make sure `rsync` is installed. For example:
If you installed GitLab:
```shell
# Debian/Ubuntu
sudo apt-get install rsync
- Using the Omnibus package, you're all set.
- From source, make sure `rsync` is installed:
# RHEL/CentOS
sudo yum install rsync
```
```shell
# Debian/Ubuntu
sudo apt-get install rsync
- **Tar**: Backup and restore tasks use `tar` under the hood to create and extract
archives. Ensure you have version 1.30 or above of `tar` available in your
system. To check the version, run:
# RHEL/CentOS
sudo yum install rsync
```shell
tar --version
```
### Tar
Backup and restore tasks use `tar` under the hood to create and extract
archives. Ensure you have version 1.30 or above of `tar` available in your
system. To check the version, run:
```shell
tar --version
```
## Backup timestamp
NOTE: **Note:**
......@@ -56,9 +51,9 @@ available.
For example, if the backup name is `1493107454_2018_04_25_10.6.4-ce_gitlab_backup.tar`,
then the timestamp is `1493107454_2018_04_25_10.6.4-ce`.
## Creating a backup of the GitLab system
## Back up GitLab
GitLab provides a simple command line interface to backup your whole instance.
GitLab provides a simple command line interface to back up your whole instance.
It backs up your:
- Database
......@@ -142,12 +137,12 @@ Deleting tmp directories...[DONE]
Deleting old backups... [SKIPPING]
```
## Storing configuration files
### Storing configuration files
A backup performed by the [raketask GitLab provides](#creating-a-backup-of-the-gitlab-system)
The [backup Rake task](#back-up-gitlab) GitLab provides
does **not** store your configuration files. The primary reason for this is that your
database contains encrypted information for two-factor authentication, the CI/CD
'secure variables', etc. Storing encrypted information along with its key in the
'secure variables', and so on. Storing encrypted information along with its key in the
same place defeats the purpose of using encryption in the first place.
CAUTION: **Warning:**
......@@ -182,11 +177,11 @@ If you use Omnibus GitLab, see some additional information
In the unlikely event that the secrets file is lost, see the
[troubleshooting section](#when-the-secrets-file-is-lost).
## Backup options
### Backup options
The command line tool GitLab provides to backup your instance can take more options.
### Backup strategy option
#### Backup strategy option
> [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/8728) in GitLab 8.17.
......@@ -214,7 +209,7 @@ sudo gitlab-backup create STRATEGY=copy
NOTE: **Note**
For GitLab 12.1 and earlier, use `gitlab-rake gitlab:backup:create`.
### Backup filename
#### Backup filename
CAUTION: **Warning:**
If you use a custom backup filename, you will not be able to
......@@ -231,7 +226,7 @@ For GitLab 12.1 and earlier, use `gitlab-rake gitlab:backup:create`.
The resulting file will then be `dump_gitlab_backup.tar`. This is useful for systems that make use of rsync and incremental backups, and will result in considerably faster transfer speeds.
### Rsyncable
#### Rsyncable
To make sure the generated archive is intelligently transferable by rsync, the `GZIP_RSYNCABLE=yes` option can be set. This will set the `--rsyncable` option to `gzip`. This is only useful in combination with setting [the Backup filename option](#backup-filename).
......@@ -244,7 +239,7 @@ sudo gitlab-backup create BACKUP=dump GZIP_RSYNCABLE=yes
NOTE: **Note**
For GitLab 12.1 and earlier, use `gitlab-rake gitlab:backup:create`.
### Excluding specific directories from the backup
#### Excluding specific directories from the backup
You can choose what should be exempt from the backup up by adding the environment variable `SKIP`.
The available options are:
......@@ -278,7 +273,7 @@ For installations from source:
sudo -u git -H bundle exec rake gitlab:backup:create SKIP=db,uploads RAILS_ENV=production
```
### Skipping tar creation
#### Skipping tar creation
The last part of creating a backup is generation of a `.tar` file containing
all the parts. In some cases (for example, if the backup is picked up by other
......@@ -303,19 +298,19 @@ For installations from source:
sudo -u git -H bundle exec rake gitlab:backup:create SKIP=tar RAILS_ENV=production
```
### Uploading backups to a remote (cloud) storage
#### Uploading backups to a remote (cloud) storage
Starting with GitLab 7.4 you can let the backup script upload the '.tar' file it creates.
It uses the [Fog library](http://fog.io/) to perform the upload.
In the example below we use Amazon S3 for storage, but Fog also lets you use
[other storage providers](http://fog.io/storage/). GitLab
[imports cloud drivers](https://gitlab.com/gitlab-org/gitlab/blob/30f5b9a5b711b46f1065baf755e413ceced5646b/Gemfile#L88)
for AWS, Google, OpenStack Swift, Rackspace and Aliyun as well. A local driver is
for AWS, Google, OpenStack Swift, Rackspace, and Aliyun as well. A local driver is
[also available](#uploading-to-locally-mounted-shares).
[Read more about using object storage with GitLab](../administration/object_storage.md).
#### Using Amazon S3
##### Using Amazon S3
For Omnibus GitLab packages:
......@@ -335,7 +330,7 @@ For Omnibus GitLab packages:
1. [Reconfigure GitLab](../administration/restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect
#### Digital Ocean Spaces
##### Digital Ocean Spaces
This example can be used for a bucket in Amsterdam (AMS3).
......@@ -360,7 +355,7 @@ usage of backup encryption. Remove or comment the line that
contains `gitlab_rails['backup_encryption']` since Digital Ocean Spaces
doesn't support encryption.
#### Other S3 Providers
##### Other S3 Providers
Not all S3 providers are fully-compatible with the Fog library. For example,
if you see `411 Length Required` errors after attempting to upload, you may
......@@ -450,7 +445,7 @@ with the name of your bucket:
}
```
#### Using Google Cloud Storage
##### Using Google Cloud Storage
If you want to use Google Cloud Storage to save backups, you'll have to create
an access key from the Google console first:
......@@ -494,7 +489,7 @@ For installations from source:
1. [Restart GitLab](../administration/restart_gitlab.md#installations-from-source) for the changes to take effect
#### Specifying a custom directory for backups
##### Specifying a custom directory for backups
Note: This option only works for remote storage. If you want to group your backups
you can pass a `DIRECTORY` environment variable:
......@@ -507,9 +502,9 @@ sudo gitlab-backup create DIRECTORY=weekly
NOTE: **Note**
For GitLab 12.1 and earlier, use `gitlab-rake gitlab:backup:create`.
### Uploading to locally mounted shares
#### Uploading to locally mounted shares
You may also send backups to a mounted share (`NFS` / `CIFS` / `SMB` / etc.) by
You may also send backups to a mounted share (for example, `NFS`,`CIFS`, or `SMB`) by
using the Fog [`Local`](https://github.com/fog/fog-local#usage) storage provider.
The directory pointed to by the `local_root` key **must** be owned by the `git`
user **when mounted** (mounting with the `uid=` of the `git` user for `CIFS` and
......@@ -559,7 +554,7 @@ For installations from source:
1. [Restart GitLab](../administration/restart_gitlab.md#installations-from-source) for the changes to take effect.
### Backup archive permissions
#### Backup archive permissions
The backup archives created by GitLab (`1393513186_2014_02_27_gitlab_backup.tar`)
will have owner/group `git`/`git` and 0600 permissions by default.
......@@ -587,7 +582,7 @@ For installations from source:
1. [Restart GitLab](../administration/restart_gitlab.md#installations-from-source) for the changes to take effect.
### Configuring cron to make daily backups
#### Configuring cron to make daily backups
CAUTION: **Warning:**
The following cron jobs do not [backup your GitLab configuration files](#storing-configuration-files)
......@@ -671,7 +666,7 @@ For installations from source:
1. [Restart GitLab](../administration/restart_gitlab.md#installations-from-source) for the changes to take effect.
## Restore
## Restore GitLab
GitLab provides a simple command line interface to restore your whole installation,
and is flexible enough to fit your needs.
......@@ -902,6 +897,8 @@ will have all your repositories, but not any other data.
## Troubleshooting
The following are possible problems you might encounter with possible solutions.
### Restoring database backup using Omnibus packages outputs warnings
If you are using backup restore procedures you might encounter the following warnings:
......@@ -1081,7 +1078,7 @@ want to run the `chown` against your custom location instead of
### Backup fails to complete with Gzip error
While running the backup, you may receive a gzip error:
While running the backup, you may receive a Gzip error:
```shell
sudo /opt/gitlab/bin/gitlab-backup create
......@@ -1095,5 +1092,5 @@ Backup failed
If this happens, check the following:
1. Confirm there is sufficient disk space for the gzip operation.
1. Confirm there is sufficient disk space for the Gzip operation.
1. If NFS is being used, check if the mount option `timeout` is set. The default is `600`, and changing this to smaller values have resulted in this error.
......@@ -14,10 +14,10 @@ authenticated web session, allowing the launching of further attacks.
The TLS Protocol CRIME Vulnerability affects systems that use data compression
over HTTPS. Your system might be vulnerable to the CRIME vulnerability if you use
SSL Compression (for example, gzip) or SPDY (which optionally uses compression).
SSL Compression (for example, Gzip) or SPDY (which optionally uses compression).
GitLab supports both gzip and [SPDY](http://nginx.org/en/docs/http/ngx_http_spdy_module.html) and mitigates the CRIME
vulnerability by deactivating gzip when HTTPS is enabled. The sources of the
GitLab supports both Gzip and [SPDY](http://nginx.org/en/docs/http/ngx_http_spdy_module.html) and mitigates the CRIME
vulnerability by deactivating Gzip when HTTPS is enabled. The sources of the
files are here:
- [Source installation NGINX file](https://gitlab.com/gitlab-org/gitlab/blob/master/lib/support/nginx/gitlab-ssl)
......
......@@ -62,14 +62,14 @@ Sometimes we need to upgrade customers from old versions of GitLab to latest, so
- Users
- Groups
- Projects
- [Backup using our Backup Rake task](../../raketasks/backup_restore.md#creating-a-backup-of-the-gitlab-system)
- [Back up using our backup Rake task](../../raketasks/backup_restore.md#back-up-gitlab)
- [Upgrade to 5.0 source using our Upgrade documentation](https://gitlab.com/gitlab-org/gitlab/blob/master/doc/update/4.2-to-5.0.md)
- [Upgrade to 5.1 source](https://gitlab.com/gitlab-org/gitlab/blob/master/doc/update/5.0-to-5.1.md)
- [Upgrade to 6.0 source](https://gitlab.com/gitlab-org/gitlab/blob/master/doc/update/5.1-to-6.0.md)
- [Upgrade to 7.14 source](https://gitlab.com/gitlab-org/gitlab/blob/master/doc/update/6.x-or-7.x-to-7.14.md)
- [Perform the MySQL to PostgreSQL migration to convert your backup](../../update/mysql_to_postgresql.md)
- [Upgrade to Omnibus 7.14](https://docs.gitlab.com/omnibus/update/README.html#upgrading-from-a-non-omnibus-installation-to-an-omnibus-installation)
- [Restore backup using our Restore Rake task](../../raketasks/backup_restore.md#restore)
- [Restore backup using our Restore Rake task](../../raketasks/backup_restore.md#restore-gitlab)
- [Upgrade to latest EE](https://about.gitlab.com/update/)
- (GitLab inc. only) Acquire and apply a license for the Enterprise Edition product, ask in #support
- Perform a downgrade from [EE to CE](../../downgrade_ee_to_ce/README.md)
......
......@@ -12,7 +12,7 @@ You can select the tag in the version dropdown in the top left corner of GitLab
### 0. Backup
It's useful to make a backup just in case things go south. Depending on the installation method, backup commands vary, see the [backing up and restoring GitLab](../raketasks/backup_restore.md#creating-a-backup-of-the-gitlab-system) documentation.
It's useful to make a backup just in case things go south. Depending on the installation method, backup commands vary. See the [backing up and restoring GitLab](../raketasks/backup_restore.md) documentation.
### 1. Stop server
......
......@@ -12,7 +12,7 @@ First, roll back the code or package. For source installations this involves
checking out the older version (branch or tag). For Omnibus installations this
means installing the older .deb or .rpm package. Then, restore from a backup.
Follow the instructions in the
[Backup and Restore](../raketasks/backup_restore.md#restore)
[Backup and Restore](../raketasks/backup_restore.md#restore-gitlab)
documentation.
## Potential problems on the next upgrade
......
......@@ -117,7 +117,10 @@ The "Move issue" button is at the bottom of the right-sidebar when viewing the i
If you have advanced technical skills you can also bulk move all the issues from one project to another in the rails console. The below script will move all the issues from one project to another that are not in status **closed**.
To access rails console run `sudo gitlab-rails console` on the GitLab server and run the below script. Please be sure to change **project**, **admin_user** and **target_project** to your values. We do also recommend [creating a backup](../../../raketasks/backup_restore.md#creating-a-backup-of-the-gitlab-system) before attempting any changes in the console.
To access rails console run `sudo gitlab-rails console` on the GitLab server and run the below
script. Please be sure to change **project**, **admin_user** and **target_project** to your values.
We do also recommend [creating a backup](../../../raketasks/backup_restore.md#back-up-gitlab) before
attempting any changes in the console.
```ruby
project = Project.find_by_full_path('full path of the project where issues are moved from')
......
Markdown is supported
0%
or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment