Skip to content

G
gitlab-ce
Commit d4134f54 authored by Suzanne Selhorn's avatar Suzanne Selhorn
Browse files
Options

Merge branch 'sunjungp-master-patch-14240' into 'master' 

Geo: Update the planned failover documentation

See merge request gitlab-org/gitlab!55747
parents e08e3da9 13f36178
Show whitespace changes
Inline Side-by-side
Showing with 83 additions and 139 deletions
doc/administration/geo/disaster_recovery/index.md
View file @ d4134f54
......@@ -38,7 +38,7 @@ order to avoid unnecessary data loss.
WARNING:
If the **primary** node goes offline, there may be data saved on the **primary** node
that has not been replicated to the **secondary** node. This data should be treated
that have not been replicated to the **secondary** node. This data should be treated
as lost if you proceed.
If an outage on the **primary** node happens, you should do everything possible to
......@@ -46,39 +46,21 @@ avoid a split-brain situation where writes can occur in two different GitLab
instances, complicating recovery efforts. So to prepare for the failover, we
must disable the **primary** node.
1. SSH into the **primary** node to stop and disable GitLab, if possible:
- If you have SSH access:
1. SSH into the **primary** node to stop and disable GitLab:
```shell
sudo gitlab-ctl stop
```
Prevent GitLab from starting up again if the server unexpectedly reboots:
1. Prevent GitLab from starting up again if the server unexpectedly reboots:
```shell
sudo systemctl disable gitlab-runsvdir
```
NOTE:
(**CentOS only**) In CentOS 6 or older, there is no easy way to prevent GitLab from being
started if the machine reboots isn't available (see [Omnibus GitLab issue #3058](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/3058)).
It may be safest to uninstall the GitLab package completely:
```shell
yum remove gitlab-ee
```
NOTE:
(**Ubuntu 14.04 LTS**) If you are using an older version of Ubuntu
or any other distribution based on the Upstart init system, you can prevent GitLab
from starting if the machine reboots by doing the following:
```shell
initctl stop gitlab-runsvvdir
echo 'manual' > /etc/init/gitlab-runsvdir.override
initctl reload-configuration
```
1. If you do not have SSH access to the **primary** node, take the machine offline and
- If you do not have SSH access to the **primary** node, take the machine offline and
prevent it from rebooting by any means at your disposal.
Since there are many ways you may prefer to accomplish this, we will avoid a
single recommendation. You may need to:
......@@ -91,17 +73,26 @@ must disable the **primary** node.
- Revoke object storage permissions from the **primary** node.
- Physically disconnect a machine.
1. If you plan to [update the primary domain DNS record](#step-4-optional-updating-the-primary-domain-dns-record),
If you plan to [update the primary domain DNS record](#step-4-optional-updating-the-primary-domain-dns-record),
you may wish to lower the TTL now to speed up propagation.
### Step 3. Promoting a **secondary** node
WARNING:
In GitLab 13.2 and 13.3, promoting a secondary node to a primary while the
secondary is paused fails. Do not pause replication before promoting a
secondary. If the node is paused, be sure to resume before promoting.
This issue has been fixed in GitLab 13.4 and later.
Note the following when promoting a secondary:
- If replication was paused on the secondary node (for example as a part of
upgrading, while you were running a version of GitLab earlier than 13.4), you
_must_ [enable the node by using the database](../replication/troubleshooting.md#message-activerecordrecordinvalid-validation-failed-enabled-geo-primary-node-cannot-be-disabled)
before proceeding.
before proceeding. If the secondary node
[has been paused](../../geo/index.md#pausing-and-resuming-replication), the promotion
performs a point-in-time recovery to the last known state.
Data that was created on the primary while the secondary was paused will be lost.
- A new **secondary** should not be added at this time. If you want to add a new
**secondary**, do this after you have completed the entire process of promoting
the **secondary** to the **primary**.
......@@ -117,62 +108,48 @@ Note the following when promoting a secondary:
sudo -i
```
1. Edit `/etc/gitlab/gitlab.rb` to reflect its new status as **primary** by
removing any lines that enabled the `geo_secondary_role`:
Users of GitLab 13.5 or later can skip this step, due to the appropriate
roles being enabled or disabled during the promotion in the following
step.
1. If you're using GitLab 13.5 and later, skip this step. If not, edit
`/etc/gitlab/gitlab.rb` and remove any of the following lines that
might be present:
```ruby
## In pre-11.5 documentation, the role was enabled as follows. Remove this line.
geo_secondary_role['enable'] = true
## In 11.5+ documentation, the role was enabled as follows. Remove this line.
roles ['geo_secondary_role']
```
1. Promote the **secondary** node to the **primary** node.
WARNING:
In GitLab 13.2 and 13.3, promoting a secondary node to a primary while the
secondary is paused fails. Do not pause replication before promoting a
secondary. If the node is paused, be sure to resume before promoting. This
issue has been fixed in GitLab 13.4 and later.
WARNING:
If the secondary node [has been paused](../../geo/index.md#pausing-and-resuming-replication), this performs
a point-in-time recovery to the last known state.
Data that was created on the primary while the secondary was paused will be lost.
NOTE:
In GitLab 13.7 and earlier, if you have a data type with zero items to sync,
this command reports `ERROR - Replication is not up-to-date` even if
replication is actually up-to-date. If replication and verification output
shows that it is complete, you can add `--skip-preflight-checks` to make the
command complete promotion. This bug was fixed in GitLab 13.8 and later.
1. Promote the **secondary** node to the **primary** node:
To promote the secondary node to primary along with preflight checks:
- To promote the secondary node to primary along with [preflight checks](planned_failover.md#preflight-checks):
```shell
gitlab-ctl promote-to-primary-node
```
If you have already run the [preflight checks](planned_failover.md#preflight-checks) separately or don't want to run them, you can skip preflight checks with:
- If you have already run the preflight checks separately or don't want to run them,
you can skip them with:
```shell
gitlab-ctl promote-to-primary-node --skip-preflight-checks
```
You can also promote the secondary node to primary **without any further confirmation**, even when preflight checks fail:
NOTE:
In GitLab 13.7 and earlier, if you have a data type with zero items to sync
and don't skip the preflight checks, promoting the secondary reports
`ERROR - Replication is not up-to-date` even if replication is actually
up-to-date. If replication and verification output
shows that it is complete, you can skip the preflight checks to make the
command complete promotion. This bug was fixed in GitLab 13.8 and later.
- To promote the secondary node to primary **without any further confirmation**,
even when preflight checks fail:
```shell
gitlab-ctl promote-to-primary-node --force
```
1. Verify you can connect to the newly promoted **primary** node using the URL used
1. Verify you can connect to the newly-promoted **primary** node using the URL used
previously for the **secondary** node.
1. If successful, the **secondary** node has now been promoted to the **primary** node.
1. If successful, the **secondary** node is now promoted to the **primary** node.
#### Promoting a **secondary** node with multiple servers
......@@ -181,17 +158,6 @@ conjunction with multiple servers, as it can only
perform changes on a **secondary** with only a single machine. Instead, you must
do this manually.
WARNING:
In GitLab 13.2 and 13.3, promoting a secondary node to a primary while the
secondary is paused fails. Do not pause replication before promoting a
secondary. If the node is paused, be sure to resume before promoting. This
issue has been fixed in GitLab 13.4 and later.
WARNING:
If the secondary node [has been paused](../../geo/index.md#pausing-and-resuming-replication), this performs
a point-in-time recovery to the last known state.
Data that was created on the primary while the secondary was paused will be lost.
1. SSH in to the database node in the **secondary** and trigger PostgreSQL to
promote to read-write:
......@@ -201,20 +167,16 @@ Data that was created on the primary while the secondary was paused will be lost
In GitLab 12.8 and earlier, see [Message: `sudo: gitlab-pg-ctl: command not found`](../replication/troubleshooting.md#message-sudo-gitlab-pg-ctl-command-not-found).
1. Edit `/etc/gitlab/gitlab.rb` on every machine in the **secondary** to
reflect its new status as **primary** by removing any lines that enabled the
`geo_secondary_role`:
1. Edit `/etc/gitlab/gitlab.rb` and remove any of the following lines that
might be present:
```ruby
## In pre-11.5 documentation, the role was enabled as follows. Remove this line.
geo_secondary_role['enable'] = true
## In 11.5+ documentation, the role was enabled as follows. Remove this line.
roles ['geo_secondary_role']
```
After making these changes [Reconfigure GitLab](../../restart_gitlab.md#omnibus-gitlab-reconfigure) each
machine so the changes take effect.
After making these changes, [reconfigure GitLab](../../restart_gitlab.md#omnibus-gitlab-reconfigure)
on each machine so the changes take effect.
1. Promote the **secondary** to **primary**. SSH into a single application
server and execute:
......@@ -223,9 +185,9 @@ Data that was created on the primary while the secondary was paused will be lost
sudo gitlab-rake geo:set_secondary_as_primary
```
1. Verify you can connect to the newly promoted **primary** using the URL used
1. Verify you can connect to the newly-promoted **primary** using the URL used
previously for the **secondary**.
1. Success! The **secondary** has now been promoted to **primary**.
1. If successful, the **secondary** node is now promoted to the **primary** node.
#### Promoting a **secondary** node with a Patroni standby cluster
......@@ -234,17 +196,6 @@ conjunction with a Patroni standby cluster, as it can only
perform changes on a **secondary** with only a single machine. Instead, you must
do this manually.
WARNING:
In GitLab 13.2 and 13.3, promoting a secondary node to a primary while the
secondary is paused fails. Do not pause replication before promoting a
secondary. If the node is paused, be sure to resume before promoting. This
issue has been fixed in GitLab 13.4 and later.
WARNING:
If the secondary node [has been paused](../../geo/index.md#pausing-and-resuming-replication), this performs
a point-in-time recovery to the last known state.
Data that was created on the primary while the secondary was paused will be lost.
1. SSH in to the Standby Leader database node in the **secondary** and trigger PostgreSQL to
promote to read-write:
......@@ -252,13 +203,10 @@ Data that was created on the primary while the secondary was paused will be lost
sudo gitlab-ctl promote-db
```
1. Edit `/etc/gitlab/gitlab.rb` on every application and Sidekiq nodes in the secondary to reflect its new status as primary by removing any lines that enabled the `geo_secondary_role`:
1. Edit `/etc/gitlab/gitlab.rb` on every application and Sidekiq nodes in the secondary to reflect its new status as primary by removing any of the following lines that might be present:
```ruby
## In pre-11.5 documentation, the role was enabled as follows. Remove this line.
geo_secondary_role['enable'] = true
## In 11.5+ documentation, the role was enabled as follows. Remove this line.
roles ['geo_secondary_role']
```
......@@ -280,9 +228,9 @@ Data that was created on the primary while the secondary was paused will be lost
sudo gitlab-rake geo:set_secondary_as_primary
```
1. Verify you can connect to the newly promoted **primary** using the URL used previously for the **secondary**.
1. Success! The **secondary** has now been promoted to **primary**.
1. Verify you can connect to the newly-promoted **primary** using the URL used
previously for the **secondary**.
1. If successful, the **secondary** node is now promoted to the **primary** node.
#### Promoting a **secondary** node with an external PostgreSQL database
......@@ -292,12 +240,12 @@ node with GitLab and the database on the same machine. As a result, a manual pro
required:
1. Promote the replica database associated with the **secondary** site. This will
set the database to read-write:
- Amazon RDS - [Promoting a Read Replica to Be a Standalone DB Instance](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_ReadRepl.html#USER_ReadRepl.Promote)
- Azure Database for PostgreSQL - [Stop replication](https://docs.microsoft.com/en-us/azure/postgresql/howto-read-replicas-portal#stop-replication)
- Other external PostgreSQL databases - save the script below in you secondary node, for example
`/tmp/geo_promote.sh`, and modify the connection parameters to match your
environment. Then, execute it to promote the replica:
set the database to read-write. The instructions vary depending on where your database is hosted:
- [Amazon RDS](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_ReadRepl.html#USER_ReadRepl.Promote)
- [Azure PostgreSQL](https://docs.microsoft.com/en-us/azure/postgresql/howto-read-replicas-portal#stop-replication)
- For other external PostgreSQL databases, save the following script in you
secondary node, for example `/tmp/geo_promote.sh`, and modify the connection
parameters to match your environment. Then, execute it to promote the replica:
```shell
#!/bin/bash
......@@ -318,14 +266,11 @@ required:
```
1. Edit `/etc/gitlab/gitlab.rb` on every node in the **secondary** site to
reflect its new status as **primary** by removing any lines that enabled the
`geo_secondary_role`:
reflect its new status as **primary** by removing any of the following
lines that might be present:
```ruby
## In GitLab 11.4 and earlier, remove this line.
geo_secondary_role['enable'] = true
## In GitLab 11.5 and later, remove this line.
roles ['geo_secondary_role']
```
......@@ -339,10 +284,9 @@ required:
sudo gitlab-rake geo:set_secondary_as_primary
```
1. Verify you can connect to the newly promoted **primary** site using the URL used
previously for the **secondary** site.
1. Success! The **secondary** site has now been promoted to **primary**.
1. Verify you can connect to the newly-promoted **primary** using the URL used
previously for the **secondary**.
1. If successful, the **secondary** node is now promoted to the **primary** node.
### Step 4. (Optional) Updating the primary domain DNS record
......@@ -443,7 +387,7 @@ and after that you also need two extra steps.
sudo -i
```
1. Edit `/etc/gitlab/gitlab.rb`
1. Edit `/etc/gitlab/gitlab.rb`:
```ruby
## Enable a Geo Primary role (if you haven't yet)
......@@ -468,7 +412,7 @@ and after that you also need two extra steps.
(For more details about these settings you can read [Configure the primary server](../setup/database.md#step-1-configure-the-primary-server))
1. Save the file and reconfigure GitLab for the database listen changes and
the replication slot changes to be applied.
the replication slot changes to be applied:
```shell
gitlab-ctl reconfigure
......
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment
GitLab Nexedi Edition | About GitLab | About Nexedi | 沪ICP备2021021310号-2 | 沪ICP备2021021310号-7