Commit e50ab739 authored by Achilleas Pipinellis's avatar Achilleas Pipinellis

Merge branch 'docs/geo-styles-and-names' into 'master'

Standardise style and nomenclature for Geo documentation

Closes #8231

See merge request gitlab-org/gitlab-ee!7878
parents ab6e5b9d 5d302102
...@@ -4,18 +4,18 @@ NOTE: **Note:** ...@@ -4,18 +4,18 @@ NOTE: **Note:**
Automatic background verification of repositories and wikis was added in Automatic background verification of repositories and wikis was added in
GitLab EE 10.6 but is enabled by default only on GitLab EE 11.1. You can GitLab EE 10.6 but is enabled by default only on GitLab EE 11.1. You can
disable or enable this feature manually by following disable or enable this feature manually by following
[these instructions][feature-flag]. [these instructions](#disabling-or-enabling-the-automatic-background-verification).
Automatic backgorund verification ensures that the transferred data matches a Automatic background verification ensures that the transferred data matches a
calculated checksum, proving that the content on the **secondary** matches that calculated checksum. If the checksum of the data on the **primary** node matches checksum of the
on the **primary**. Following a planned failover, any corrupted data may be data on the **secondary** node, the data transferred successfully. Following a planned failover,
**lost**, depending on the extent of the corruption. any corrupted data may be **lost**, depending on the extent of the corruption.
If verification fails on the **primary**, this indicates that Geo is If verification fails on the **primary** node, this indicates that Geo is
successfully replicating a corrupted object; restore it from backup or remove it successfully replicating a corrupted object; restore it from backup or remove it
it from the primary to resolve the issue. it from the **primary** node to resolve the issue.
If verification succeeds on the **primary** but fails on the **secondary**, If verification succeeds on the **primary** node but fails on the **secondary** node,
this indicates that the object was corrupted during the replication process. this indicates that the object was corrupted during the replication process.
Geo actively try to correct verification failures marking the repository to Geo actively try to correct verification failures marking the repository to
be resynced with a backoff period. If you want to reset the verification for be resynced with a backoff period. If you want to reset the verification for
...@@ -65,7 +65,7 @@ in grey, and failures in red. ...@@ -65,7 +65,7 @@ in grey, and failures in red.
![Verification status](img/verification-status-primary.png) ![Verification status](img/verification-status-primary.png)
Navigate to the **Admin Area > Geo** dashboard on the **secondary** node and expand Navigate to the **Admin Area > Geo** dashboard on the **secondary** node and expand
the **Verification information** tab for that node to view automatic verifcation the **Verification information** tab for that node to view automatic verification
status for repositories and wikis. As with checksumming, successes are shown in status for repositories and wikis. As with checksumming, successes are shown in
green, pending work in grey, and failures in red. green, pending work in grey, and failures in red.
...@@ -73,7 +73,7 @@ green, pending work in grey, and failures in red. ...@@ -73,7 +73,7 @@ green, pending work in grey, and failures in red.
## Using checksums to compare Geo nodes ## Using checksums to compare Geo nodes
To check the health of Geo secondary nodes, we use a checksum over the list of To check the health of Geo **secondary** nodes, we use a checksum over the list of
Git references and their values. The checksum includes `HEAD`, `heads`, `tags`, Git references and their values. The checksum includes `HEAD`, `heads`, `tags`,
`notes`, and GitLab-specific references to ensure true consistency. If two nodes `notes`, and GitLab-specific references to ensure true consistency. If two nodes
have the same checksum, then they definitely hold the same references. We compute have the same checksum, then they definitely hold the same references. We compute
...@@ -129,33 +129,33 @@ be resynced with a backoff period. If you want to reset them manually, this ...@@ -129,33 +129,33 @@ be resynced with a backoff period. If you want to reset them manually, this
rake task marks projects where verification has failed or the checksum mismatch rake task marks projects where verification has failed or the checksum mismatch
to be resynced without the backoff period: to be resynced without the backoff period:
#### For repositories: For repositories:
**Omnibus Installation** - Omnibus Installation
``` ```sh
sudo gitlab-rake geo:verification:repository:reset sudo gitlab-rake geo:verification:repository:reset
``` ```
**Source Installation** - Source Installation
```bash ```sh
sudo -u git -H bundle exec rake geo:verification:repository:reset RAILS_ENV=production sudo -u git -H bundle exec rake geo:verification:repository:reset RAILS_ENV=production
``` ```
#### For wikis: For wikis:
**Omnibus Installation** - Omnibus Installation
``` ```sh
sudo gitlab-rake geo:verification:wiki:reset sudo gitlab-rake geo:verification:wiki:reset
``` ```
**Source Installation** - Source Installation
```bash ```sh
sudo -u git -H bundle exec rake geo:verification:wiki:reset RAILS_ENV=production sudo -u git -H bundle exec rake geo:verification:wiki:reset RAILS_ENV=production
``` ```
## Current limitations ## Current limitations
...@@ -167,7 +167,6 @@ on both nodes, and comparing the output between them. ...@@ -167,7 +167,6 @@ on both nodes, and comparing the output between them.
Data in object storage is **not verified**, as the object store is responsible Data in object storage is **not verified**, as the object store is responsible
for ensuring the integrity of the data. for ensuring the integrity of the data.
[feature-flag]: background_verification.md#enabling-or-disabling-the-automatic-background-verification
[reset-verification]: background_verification.md#reset-verification-for-projects-where-verification-has-failed [reset-verification]: background_verification.md#reset-verification-for-projects-where-verification-has-failed
[foreground-verification]: ../../raketasks/check.md [foreground-verification]: ../../raketasks/check.md
[ee-5064]: https://gitlab.com/gitlab-org/gitlab-ee/issues/5064 [ee-5064]: https://gitlab.com/gitlab-org/gitlab-ee/issues/5064
# Bring a demoted primary node back online # Bring a demoted primary node back online
After a failover, it is possible to fail back to the demoted primary to After a failover, it is possible to fail back to the demoted **primary** node to
restore your original configuration. This process consists of two steps: restore your original configuration. This process consists of two steps:
1. Making the old primary a secondary 1. Making the old **primary** node a **secondary** node.
1. Promoting a secondary to a primary 1. Promoting a **secondary** node to a **primary** node.
> *Warning:* If you have any doubts about the consistency of the data on this node, we recommend to set up it from scratch. CAUTION: **Caution:**
If you have any doubts about the consistency of the data on this node, we recommend setting it up from scratch.
## Configure the former primary to be a secondary ## Configure the former **primary** node to be a **secondary** node
Since the former primary will be out of sync with the current primary, the first step is Since the former **primary** node will be out of sync with the current **primary** node, the first step is to bring the former **primary** node up to date. Note, deletion of data stored on disk like
to bring the former primary up to date. Note, deletion of data stored on disk like repositories and uploads will not be replayed when bringing the former **primary** node back
repositories and uploads will not be replayed when bringing the former primary in back
into sync, which may result in increased disk usage. into sync, which may result in increased disk usage.
Alternatively, you can [set up a new secondary GitLab instance][setup-geo] to avoid this. Alternatively, you can [set up a new **secondary** GitLab instance][setup-geo] to avoid this.
To bring the former primary up to date: To bring the former **primary** node up to date:
1. SSH into the former primary that has fallen behind 1. SSH into the former **primary** node that has fallen behind.
1. Make sure all the services are up: 1. Make sure all the services are up:
```bash ```sh
sudo gitlab-ctl start sudo gitlab-ctl start
``` ```
> **Note 1:** If you [disabled primary permanently][disaster-recovery-disable-primary], > **Note 1:** If you [disabled the **primary** node permanently][disaster-recovery-disable-primary],
> you need to undo those steps now. For Debian/Ubuntu you just need to run > you need to undo those steps now. For Debian/Ubuntu you just need to run
> `sudo systemctl enable gitlab-runsvdir`. For CentOS 6, you need to install > `sudo systemctl enable gitlab-runsvdir`. For CentOS 6, you need to install
> the GitLab instance from scratch and set it up as a secondary node by > the GitLab instance from scratch and set it up as a **secondary** node by
> following [Setup instructions][setup-geo]. In this case you don't need to follow the next step. > following [Setup instructions][setup-geo]. In this case, you don't need to follow the next step.
> >
> **Note 2:** If you [changed the DNS records](index.md#step-4-optional-updating-the-primary-domains-dns-record) > **Note 2:** If you [changed the DNS records](index.md#step-4-optional-updating-the-primary-domains-dns-record)
> for this node during disaster recovery procedure you may need to [block > for this node during disaster recovery procedure you may need to [block
...@@ -37,25 +37,25 @@ To bring the former primary up to date: ...@@ -37,25 +37,25 @@ To bring the former primary up to date:
> during this procedure. > during this procedure.
1. [Setup database replication][database-replication]. Note that in this 1. [Setup database replication][database-replication]. Note that in this
case, primary refers to the current primary, and secondary refers to the case, **primary** node refers to the current **primary** node, and **secondary** node refers to the
former primary. former **primary** node.
If you have lost your original primary, follow the If you have lost your original **primary** node, follow the
[setup instructions][setup-geo] to set up a new secondary. [setup instructions][setup-geo] to set up a new **secondary** node.
## Promote the secondary to primary ## Promote the **secondary** node to **primary** node
When the initial replication is complete and the primary and secondary are When the initial replication is complete and the **primary** node and **secondary** node are
closely in sync, you can do a [planned failover]. closely in sync, you can do a [planned failover].
## Restore the secondary node ## Restore the **secondary** node
If your objective is to have two nodes again, you need to bring your secondary If your objective is to have two nodes again, you need to bring your **secondary**
node back online as well by repeating the first step node back online as well by repeating the first step
([configure the former primary to be a secondary](#configure-the-former-primary-to-be-a-secondary)) ([configure the former **primary** node to be a **secondary** node](#configure-the-former-primary-node-to-be-a-secondary-node))
for the secondary node. for the **secondary** node.
[setup-geo]: ../replication/index.md#setup-instructions [setup-geo]: ../replication/index.md#setup-instructions
[database-replication]: ../replication/database.md [database-replication]: ../replication/database.md
[disaster-recovery-disable-primary]: index.md#step-2-permanently-disable-the-primary [disaster-recovery-disable-primary]: index.md#step-2-permanently-disable-the-primary-node
[planned failover]: planned_failover.md [planned failover]: planned_failover.md
...@@ -10,74 +10,74 @@ CAUTION: **Warning:** ...@@ -10,74 +10,74 @@ CAUTION: **Warning:**
Disaster recovery for multi-secondary configurations is in **Alpha**. Disaster recovery for multi-secondary configurations is in **Alpha**.
For the latest updates, check the multi-secondary [Disaster Recovery epic][gitlab-org&65]. For the latest updates, check the multi-secondary [Disaster Recovery epic][gitlab-org&65].
## Promoting secondary Geo replica in single-secondary configurations ## Promoting a **secondary** Geo node in single-secondary configurations
We don't currently provide an automated way to promote a Geo replica and do a We don't currently provide an automated way to promote a Geo replica and do a
failover, but you can do it manually if you have `root` access to the machine. failover, but you can do it manually if you have `root` access to the machine.
This process promotes a secondary Geo replica to a primary. To regain This process promotes a **secondary** Geo node to a **primary** node. To regain
geographical redundancy as quickly as possible, you should add a new secondary geographic redundancy as quickly as possible, you should add a new **secondary** node
immediately after following these instructions. immediately after following these instructions.
### Step 1. Allow replication to finish if possible ### Step 1. Allow replication to finish if possible
If the secondary is still replicating data from the primary, follow If the **secondary** node is still replicating data from the **primary** node, follow
[the planned failover docs][planned-failover] as closely as possible in [the planned failover docs][planned-failover] as closely as possible in
order to avoid unnecessary data loss. order to avoid unnecessary data loss.
### Step 2. Permanently disable the primary ### Step 2. Permanently disable the **primary** node
CAUTION: **Warning:** CAUTION: **Warning:**
If a primary goes offline, there may be data saved on the primary If the **primary** node goes offline, there may be data saved on the **primary** node
that has not been replicated to the secondary. This data should be treated that has not been replicated to the **secondary** node. This data should be treated
as lost if you proceed. as lost if you proceed.
If an outage on your primary happens, you should do everything possible to If an outage on the **primary** node happens, you should do everything possible to
avoid a split-brain situation where writes can occur in two different GitLab avoid a split-brain situation where writes can occur in two different GitLab
instances, complicating recovery efforts. So to prepare for the failover, we instances, complicating recovery efforts. So to prepare for the failover, we
must disable the primary. must disable the **primary** node.
1. SSH into your **primary** to stop and disable GitLab, if possible: 1. SSH into the **primary** node to stop and disable GitLab, if possible:
```bash ```sh
sudo gitlab-ctl stop sudo gitlab-ctl stop
``` ```
Prevent GitLab from starting up again if the server unexpectedly reboots: Prevent GitLab from starting up again if the server unexpectedly reboots:
```bash ```sh
sudo systemctl disable gitlab-runsvdir sudo systemctl disable gitlab-runsvdir
``` ```
> **CentOS only**: In CentOS 6 or older, there is no easy way to prevent GitLab from being > **CentOS only**: In CentOS 6 or older, there is no easy way to prevent GitLab from being
started if the machine reboots isn't available (see [gitlab-org/omnibus-gitlab#3058]). > started if the machine reboots isn't available (see [gitlab-org/omnibus-gitlab#3058]).
It may be safest to uninstall the GitLab package completely: > It may be safest to uninstall the GitLab package completely:
```bash ```sh
yum remove gitlab-ee yum remove gitlab-ee
``` ```
> **Ubuntu 14.04 LTS**: If you are using an older version of Ubuntu > **Ubuntu 14.04 LTS**: If you are using an older version of Ubuntu
or any other distro based on the Upstart init system, you can prevent GitLab > or any other distro based on the Upstart init system, you can prevent GitLab
from starting if the machine reboots by doing the following: > from starting if the machine reboots by doing the following:
```bash ```sh
initctl stop gitlab-runsvvdir initctl stop gitlab-runsvvdir
echo 'manual' > /etc/init/gitlab-runsvdir.override echo 'manual' > /etc/init/gitlab-runsvdir.override
initctl reload-configuration initctl reload-configuration
``` ```
1. If you do not have SSH access to your primary, take the machine offline and 1. If you do not have SSH access to the **primary** node, take the machine offline and
prevent it from rebooting by any means at your disposal. prevent it from rebooting by any means at your disposal.
Since there are many ways you may prefer to accomplish this, we will avoid a Since there are many ways you may prefer to accomplish this, we will avoid a
single recommendation. You may need to: single recommendation. You may need to:
- Reconfigure the load balancers - Reconfigure the load balancers.
- Change DNS records (e.g., point the primary DNS record to the secondary - Change DNS records (e.g., point the primary DNS record to the **secondary**
node in order to stop usage of the primary) node in order to stop usage of the **primary** node).
- Stop the virtual servers - Stop the virtual servers.
- Block traffic through a firewall - Block traffic through a firewall.
- Revoke object storage permissions from the primary - Revoke object storage permissions from the **primary** node.
- Physically disconnect a machine - Physically disconnect a machine.
1. If you plan to 1. If you plan to
[update the primary domain DNS record](#step-4-optional-updating-the-primary-domain-dns-record), [update the primary domain DNS record](#step-4-optional-updating-the-primary-domain-dns-record),
...@@ -92,9 +92,9 @@ the **secondary** to the **primary**. ...@@ -92,9 +92,9 @@ the **secondary** to the **primary**.
#### Promoting a **secondary** node running on a single machine #### Promoting a **secondary** node running on a single machine
1. SSH in to your **secondary** and login as root: 1. SSH in to your **secondary** node and login as root:
```bash ```sh
sudo -i sudo -i
``` ```
...@@ -109,15 +109,15 @@ the **secondary** to the **primary**. ...@@ -109,15 +109,15 @@ the **secondary** to the **primary**.
roles ['geo_secondary_role'] roles ['geo_secondary_role']
``` ```
1. Promote the **secondary** to **primary**. Execute: 1. Promote the **secondary** node to the **primary** node. Execute:
```bash ```sh
gitlab-ctl promote-to-primary-node gitlab-ctl promote-to-primary-node
``` ```
1. Verify you can connect to the newly promoted **primary** using the URL used 1. Verify you can connect to the newly promoted **primary** node using the URL used
previously for the **secondary**. previously for the **secondary** node.
1. Success! The **secondary** has now been promoted to **primary**. 1. If successful, the **secondary** node has now been promoted to the **primary** node.
#### Promoting a **secondary** node with HA #### Promoting a **secondary** node with HA
...@@ -161,19 +161,19 @@ do this manually. ...@@ -161,19 +161,19 @@ do this manually.
### Step 4. (Optional) Updating the primary domain DNS record ### Step 4. (Optional) Updating the primary domain DNS record
Updating the DNS records for the primary domain to point to the secondary Updating the DNS records for the primary domain to point to the **secondary** node
will prevent the need to update all references to the primary domain to the will prevent the need to update all references to the primary domain to the
secondary domain, like changing Git remotes and API URLs. secondary domain, like changing Git remotes and API URLs.
1. SSH into your **secondary** and login as root: 1. SSH into the **secondary** node and login as root:
```bash ```sh
sudo -i sudo -i
``` ```
1. Update the primary domain's DNS record. After updating the primary domain's 1. Update the primary domain's DNS record. After updating the primary domain's
DNS records to point to the secondary, edit `/etc/gitlab/gitlab.rb` on the DNS records to point to the **secondary** node, edit `/etc/gitlab/gitlab.rb` on the
secondary to reflect the new URL: **secondary** node to reflect the new URL:
```ruby ```ruby
# Change the existing external_url configuration # Change the existing external_url configuration
...@@ -184,31 +184,31 @@ secondary domain, like changing Git remotes and API URLs. ...@@ -184,31 +184,31 @@ secondary domain, like changing Git remotes and API URLs.
Changing `external_url` won't prevent access via the old secondary URL, as Changing `external_url` won't prevent access via the old secondary URL, as
long as the secondary DNS records are still intact. long as the secondary DNS records are still intact.
1. Reconfigure the secondary node for the change to take effect: 1. Reconfigure the **secondary** node for the change to take effect:
```bash ```sh
gitlab-ctl reconfigure gitlab-ctl reconfigure
``` ```
1. Execute the command below to update the newly promoted primary node URL: 1. Execute the command below to update the newly promoted **primary** node URL:
```bash ```sh
gitlab-rake geo:update_primary_node_url gitlab-rake geo:update_primary_node_url
``` ```
This command will use the changed `external_url` configuration defined This command will use the changed `external_url` configuration defined
in `/etc/gitlab/gitlab.rb`. in `/etc/gitlab/gitlab.rb`.
1. Verify you can connect to the newly promoted primary using the primary URL. 1. Verify you can connect to the newly promoted **primary** using its URL.
If you updated the DNS records for the primary domain, these changes may If you updated the DNS records for the primary domain, these changes may
not have yet propagated depending on the previous DNS records TTL. not have yet propagated depending on the previous DNS records TTL.
### Step 5. (Optional) Add secondary Geo replicas to a promoted primary ### Step 5. (Optional) Add **secondary** Geo node to a promoted **primary** node
Promoting a secondary to primary using the process above does not enable Promoting a **secondary** node to **primary** node using the process above does not enable
Geo on the new primary. Geo on the new **primary** node.
To bring a new secondary online, follow the [Geo setup instructions][setup-geo]. To bring a new **secondary** node online, follow the [Geo setup instructions][setup-geo].
### Step 6. (Optional) Removing the secondary's tracking database ### Step 6. (Optional) Removing the secondary's tracking database
...@@ -222,15 +222,15 @@ The data can be removed with the following command: ...@@ -222,15 +222,15 @@ The data can be removed with the following command:
## Promoting secondary Geo replica in multi-secondary configurations ## Promoting secondary Geo replica in multi-secondary configurations
If you have more than one secondary and you need to promote one of them we suggest you to follow If you have more than one **secondary** node and you need to promote one of them, we suggest you follow
[Promoting secondary Geo replica in single-secondary configurations](#promoting-secondary-geo-replica-in-single-secondary-configurations) [Promoting **secondary** Geo node in single-secondary configurations](#promoting-secondary-geo-node-in-single-secondary-configurations)
and after that you also need two extra steps. and after that you also need two extra steps.
### Step 1. Prepare the new primary to serve one or more secondaries ### Step 1. Prepare the new **primary** node to serve one or more **secondary** nodes
1. SSH into your new **primary** and login as root: 1. SSH into the new **primary** node and login as root:
```bash ```sh
sudo -i sudo -i
``` ```
...@@ -263,13 +263,13 @@ and after that you also need two extra steps. ...@@ -263,13 +263,13 @@ and after that you also need two extra steps.
1. Save the file and reconfigure GitLab for the database listen changes and 1. Save the file and reconfigure GitLab for the database listen changes and
the replication slot changes to be applied. the replication slot changes to be applied.
```bash ```sh
gitlab-ctl reconfigure gitlab-ctl reconfigure
``` ```
Restart PostgreSQL for its changes to take effect: Restart PostgreSQL for its changes to take effect:
```bash ```sh
gitlab-ctl restart postgresql gitlab-ctl restart postgresql
``` ```
...@@ -284,16 +284,15 @@ and after that you also need two extra steps. ...@@ -284,16 +284,15 @@ and after that you also need two extra steps.
Save the file and reconfigure GitLab: Save the file and reconfigure GitLab:
```bash ```sh
gitlab-ctl reconfigure gitlab-ctl reconfigure
``` ```
### Step 2. Initiate the replication process ### Step 2. Initiate the replication process
Now we need to make each secondary listen to changes on the new primary. To do that you need Now we need to make each **secondary** node listen to changes on the new **primary** node. To do that you need
to [initiate the replication process][initiate-the-replication-process] again but this time to [initiate the replication process][initiate-the-replication-process] again but this time
for another primary. All the old replication settings will be overwritten. for another **primary** node. All the old replication settings will be overwritten.
## Troubleshooting ## Troubleshooting
...@@ -301,11 +300,11 @@ for another primary. All the old replication settings will be overwritten. ...@@ -301,11 +300,11 @@ for another primary. All the old replication settings will be overwritten.
The setup instructions for Geo prior to 10.5 failed to replicate the The setup instructions for Geo prior to 10.5 failed to replicate the
`otp_key_base` secret, which is used to encrypt the two-factor authentication `otp_key_base` secret, which is used to encrypt the two-factor authentication
secrets stored in the database. If it differs between primary and secondary secrets stored in the database. If it differs between **primary** and **secondary**
nodes, users with two-factor authentication enabled won't be able to log in nodes, users with two-factor authentication enabled won't be able to log in
after a failover. after a failover.
If you still have access to the old primary node, you can follow the If you still have access to the old **primary** node, you can follow the
instructions in the instructions in the
[Upgrading to GitLab 10.5][updating-geo] [Upgrading to GitLab 10.5][updating-geo]
section to resolve the error. Otherwise, the secret is lost and you'll need to section to resolve the error. Otherwise, the secret is lost and you'll need to
......
...@@ -6,22 +6,22 @@ failover to migrate your GitLab instance between regions without extended ...@@ -6,22 +6,22 @@ failover to migrate your GitLab instance between regions without extended
downtime. downtime.
As replication between Geo nodes is asynchronous, a planned failover requires As replication between Geo nodes is asynchronous, a planned failover requires
a maintenance window in which updates to the **primary node** are blocked. The a maintenance window in which updates to the **primary** node are blocked. The
length of this window is determined by your replication capacity - once the length of this window is determined by your replication capacity - once the
secondary is completely in sync with the primary, the failover can occur without **secondary** node is completely synchronized with the **primary** node, the failover can occur without
data loss. data loss.
This document assumes you already have a fully configured, working Geo setup. This document assumes you already have a fully configured, working Geo setup.
Please read it and the [Disaster Recovery][disaster-recovery] failover Please read it and the [Disaster Recovery][disaster-recovery] failover
documentation in full before proceeding. Planned failover is a major operation, documentation in full before proceeding. Planned failover is a major operation,
and if performed incorrectly, there is a high risk of data loss. Consider and if performed incorrectly, there is a high risk of data loss. Consider
rehearsing the procedure until you are comfortable with the necessary steps and rehearsing the procedure until you are comfortable with the necessary steps and
have a high degree of confidence in being able to perfom them accurately. have a high degree of confidence in being able to perform them accurately.
## Not all data is automatically replicated ## Not all data is automatically replicated
If you are using any GitLab features that Geo [doesn't support][limitations], If you are using any GitLab features that Geo [doesn't support][limitations],
you must make separate provisions to ensure that the secondary node has an you must make separate provisions to ensure that the **secondary** node has an
up-to-date copy of any data associated with that feature. This may extend the up-to-date copy of any data associated with that feature. This may extend the
required scheduled maintenance period significantly. required scheduled maintenance period significantly.
...@@ -29,7 +29,7 @@ A common strategy for keeping this period as short as possible for data stored ...@@ -29,7 +29,7 @@ A common strategy for keeping this period as short as possible for data stored
in files is to use `rsync` to transfer the data. An initial `rsync` can be in files is to use `rsync` to transfer the data. An initial `rsync` can be
performed ahead of the maintenance window; subsequent `rsync`s (including a performed ahead of the maintenance window; subsequent `rsync`s (including a
final transfer inside the maintenance window) will then transfer only the final transfer inside the maintenance window) will then transfer only the
*changes* between primary and secondary. *changes* between the **primary** node and the **secondary** nodes.
Repository-centric strategies for using `rsync` effectively can be found in the Repository-centric strategies for using `rsync` effectively can be found in the
[moving repositories][moving-repositories] documentation; these strategies can [moving repositories][moving-repositories] documentation; these strategies can
...@@ -50,8 +50,8 @@ means you can decouple the replication of this data from the failover of the ...@@ -50,8 +50,8 @@ means you can decouple the replication of this data from the failover of the
GitLab service. GitLab service.
If you're already using object storage, simply verify that your **secondary** If you're already using object storage, simply verify that your **secondary**
has access to the same data as the **primary** - they must either they share the node has access to the same data as the **primary** node - they must either they share the
same object storage configuration, or the **secondary** should be configured to same object storage configuration, or the **secondary** node should be configured to
access a [geographically-replicated][os-repl] copy provided by the object store access a [geographically-replicated][os-repl] copy provided by the object store
itself. itself.
...@@ -60,21 +60,21 @@ If you have a large GitLab installation or cannot tolerate downtime, consider ...@@ -60,21 +60,21 @@ If you have a large GitLab installation or cannot tolerate downtime, consider
Doing so reduces both the length of the maintenance window, and the risk of data Doing so reduces both the length of the maintenance window, and the risk of data
loss as a result of a poorly executed planned failover. loss as a result of a poorly executed planned failover.
### Review the configuration of the secondary ### Review the configuration of each **secondary** node
Database settings are automatically replicated to the secondary, but the Database settings are automatically replicated to the **secondary** node, but the
`/etc/gitlab/gitlab.rb` file must be set up manually, and differs between `/etc/gitlab/gitlab.rb` file must be set up manually, and differs between
nodes. If features such as Mattermost, OAuth or LDAP integration are enabled nodes. If features such as Mattermost, OAuth or LDAP integration are enabled
on the primary but not the secondary, they will be lost during failover. on the **primary** node but not the **secondary** node, they will be lost during failover.
Review the `/etc/gitlab/gitlab.rb` file for both nodes and ensure the secondary Review the `/etc/gitlab/gitlab.rb` file for both nodes and ensure the **secondary** node
supports everything the primary does **before** scheduling a planned failover. supports everything the **primary** node does **before** scheduling a planned failover.
### Run system checks ### Run system checks
Run the following on both primary and secondary nodes: Run the following on both **primary** and **secondary** nodes:
``` ```sh
gitlab-rake gitlab:check gitlab-rake gitlab:check
gitlab-rake gitlab:geo:check gitlab-rake gitlab:geo:check
``` ```
...@@ -88,12 +88,12 @@ The SSH host keys and `/etc/gitlab/gitlab-secrets.json` files should be ...@@ -88,12 +88,12 @@ The SSH host keys and `/etc/gitlab/gitlab-secrets.json` files should be
identical on all nodes. Check this by running the following on all nodes and identical on all nodes. Check this by running the following on all nodes and
comparing the output: comparing the output:
``` ```sh
sudo sha256sum /etc/ssh/ssh_host* /etc/gitlab/gitlab-secrets.json sudo sha256sum /etc/ssh/ssh_host* /etc/gitlab/gitlab-secrets.json
``` ```
If any files differ, replace the content on the **secondary** with the If any files differ, replace the content on the **secondary** node with the
content on the **primary**. content from the **primary** node.
### Ensure Geo replication is up-to-date ### Ensure Geo replication is up-to-date
...@@ -117,7 +117,7 @@ You can use the [Geo status API][geo-status-api] to review failed objects and ...@@ -117,7 +117,7 @@ You can use the [Geo status API][geo-status-api] to review failed objects and
the reasons for failure. the reasons for failure.
A common cause of replication failures is the data being missing on the A common cause of replication failures is the data being missing on the
**primary** - you can resolve these failures by restoring the data from backup, **primary** node - you can resolve these failures by restoring the data from backup,
or removing references to the missing data. or removing references to the missing data.
### Verify the integrity of replicated data ### Verify the integrity of replicated data
...@@ -131,23 +131,23 @@ message. You can check under **Admin Area > Geo** to estimate how long it ...@@ -131,23 +131,23 @@ message. You can check under **Admin Area > Geo** to estimate how long it
will take to finish syncing. An example message would be: will take to finish syncing. An example message would be:
> A scheduled maintenance will take place at XX:XX UTC. We expect it to take > A scheduled maintenance will take place at XX:XX UTC. We expect it to take
less than 1 hour. > less than 1 hour.
## Prevent updates to the **primary** ## Prevent updates to the **primary** node
Until a [read-only mode][ce-19739] is implemented, updates must be prevented Until a [read-only mode][ce-19739] is implemented, updates must be prevented
from happening manually. Note that your **secondary** node still needs read-only from happening manually. Note that your **secondary** node still needs read-only
access to the primary for the duration of the maintenance window. access to the **primary** node during the maintenance window.
1. At the scheduled time, using your cloud provider or your node's firewall, block 1. At the scheduled time, using your cloud provider or your node's firewall, block
all HTTP, HTTPS and SSH traffic to/from the primary, **except** for your IP and all HTTP, HTTPS and SSH traffic to/from the **primary** node, **except** for your IP and
the secondary's IP. the **secondary** node's IP.
For instance, if your secondary originates all its traffic from `5.6.7.8` and For instance, if your **secondary** node originates all its traffic from `5.6.7.8` and
your IP is `100.0.0.1`, you might run the following commands on the server(s) your IP is `100.0.0.1`, you might run the following commands on the server(s)
making up your primary node: making up your **primary** node:
``` ```sh
sudo iptables -A INPUT -p tcp -s 5.6.7.8 --destination-port 22 -j ACCEPT sudo iptables -A INPUT -p tcp -s 5.6.7.8 --destination-port 22 -j ACCEPT
sudo iptables -A INPUT -p tcp -s 100.0.0.1 --destination-port 22 -j ACCEPT sudo iptables -A INPUT -p tcp -s 100.0.0.1 --destination-port 22 -j ACCEPT
sudo iptables -A INPUT --destination-port 22 -j REJECT sudo iptables -A INPUT --destination-port 22 -j REJECT
...@@ -162,14 +162,14 @@ access to the primary for the duration of the maintenance window. ...@@ -162,14 +162,14 @@ access to the primary for the duration of the maintenance window.
``` ```
From this point, users will be unable to view their data or make changes on the From this point, users will be unable to view their data or make changes on the
**primary** node. They will also be unable to log in to the **secondary** node, **primary** node. They will also be unable to log in to the **secondary** node.
but existing sessions will work for the remainder of the maintenance period, and However, existing sessions will work for the remainder of the maintenance period, and
public data will be accessible throughout. public data will be accessible throughout.
1. Verify the primary is blocked to HTTP traffic by visiting it in browser via 1. Verify the **primary** node is blocked to HTTP traffic by visiting it in browser via
another IP. The server should refuse connection. another IP. The server should refuse connection.
1. Verify the primary is blocked to Git over SSH traffic by attempting to pull an 1. Verify the **primary** node is blocked to Git over SSH traffic by attempting to pull an
existing Git repository with an SSH remote URL. The server should refuse existing Git repository with an SSH remote URL. The server should refuse
connection. connection.
...@@ -186,13 +186,13 @@ access to the primary for the duration of the maintenance window. ...@@ -186,13 +186,13 @@ access to the primary for the duration of the maintenance window.
1. On the **primary** node, navigate to **Admin Area > Monitoring > Background Jobs > Queues** 1. On the **primary** node, navigate to **Admin Area > Monitoring > Background Jobs > Queues**
and wait for all queues except those with `geo` in the name to drop to 0. and wait for all queues except those with `geo` in the name to drop to 0.
These queues contain work that has been submitted by your users; failing over These queues contain work that has been submitted by your users; failing over
before it is completed will cause the work to be lost! before it is completed will cause the work to be lost.
1. On the **primary** node, navigate to **Admin Area > Geo** and wait for the 1. On the **primary** node, navigate to **Admin Area > Geo** and wait for the
following conditions to be true of the **secondary** node you are failing over to: following conditions to be true of the **secondary** node you are failing over to:
* All replication meters to each 100% replicated, 0% failures - All replication meters to each 100% replicated, 0% failures.
* All verification meters reach 100% verified, 0% failures - All verification meters reach 100% verified, 0% failures.
* Database replication lag is 0ms - Database replication lag is 0ms.
* The Geo log cursor is up to date (0 events behind) - The Geo log cursor is up to date (0 events behind).
1. On the **secondary** node, navigate to **Admin Area > Monitoring > Background Jobs > Queues** 1. On the **secondary** node, navigate to **Admin Area > Monitoring > Background Jobs > Queues**
and wait for all the `geo` queues to drop to 0 queued and 0 running jobs. and wait for all the `geo` queues to drop to 0 queued and 0 running jobs.
...@@ -200,18 +200,17 @@ access to the primary for the duration of the maintenance window. ...@@ -200,18 +200,17 @@ access to the primary for the duration of the maintenance window.
to verify the integrity of CI artifacts, LFS objects and uploads in file to verify the integrity of CI artifacts, LFS objects and uploads in file
storage. storage.
At this point, your secondary will contain an up-to-date copy of everything the At this point, your **secondary** node will contain an up-to-date copy of everything the
primary has, meaning nothing will be lost when you fail over. **primary** node has, meaning nothing will be lost when you fail over.
## Promote the secondary ## Promote the **secondary** node
Finally, follow the [Disaster Recovery docs][disaster-recovery] to promote the Finally, follow the [Disaster Recovery docs][disaster-recovery] to promote the
secondary to a primary. This process will cause a brief outage on the secondary, **secondary** node to a **primary** node. This process will cause a brief outage on the **secondary** node, and users may need to log in again.
and users may need to log in again.
Once it is completed, the maintenance window is over! Your new primary will now Once it is completed, the maintenance window is over! Your new **primary** node will now
begin to diverge from the old one. If problems do arise at this point, failing begin to diverge from the old one. If problems do arise at this point, failing
back to the old primary [is possible][bring-primary-back], but likely to result back to the old **primary** node [is possible][bring-primary-back], but likely to result
in the loss of any data uploaded to the new primary in the meantime. in the loss of any data uploaded to the new primary in the meantime.
Don't forget to remove the broadcast message after failover is complete. Don't forget to remove the broadcast message after failover is complete.
......
...@@ -5,36 +5,38 @@ This is the documentation for the Omnibus GitLab packages. For installations ...@@ -5,36 +5,38 @@ This is the documentation for the Omnibus GitLab packages. For installations
from source, follow the [**Geo nodes configuration for installations from source, follow the [**Geo nodes configuration for installations
from source**][configuration-source] guide. from source**][configuration-source] guide.
## Configuring a new secondary node ## Configuring a new **secondary** node
NOTE: **Note:** NOTE: **Note:**
This is the final step in setting up a secondary Geo node. Stages of the This is the final step in setting up a **secondary** Geo node. Stages of the
setup process must be completed in the documented order. setup process must be completed in the documented order.
Before attempting the steps in this stage, [complete all prior stages][setup-geo-omnibus]. Before attempting the steps in this stage, [complete all prior stages][setup-geo-omnibus].
The basic steps of configuring a secondary node are to replicate required The basic steps of configuring a **secondary** node are to:
configurations between the primary and the secondaries; to configure a tracking
database on each secondary; and to start GitLab on the secondary node. - Replicate required configurations between the **primary** node and the **secondary** nodes.
- Configure a tracking database on each **secondary** node.
- Start GitLab on each **secondary** node.
You are encouraged to first read through all the steps before executing them You are encouraged to first read through all the steps before executing them
in your testing/production environment. in your testing/production environment.
> **Notes:** > **Notes:**
> - **Do not** setup any custom authentication in the secondary nodes, this will be > - **Do not** setup any custom authentication for the **secondary** nodes. This will be
handled by the primary node. handled by the **primary** node.
> - Any change that requires access to the **Admin Area** needs to be done in the > - Any change that requires access to the **Admin Area** needs to be done in the
primary node, as the secondary node is a read-only replica. **primary** node because the **secondary** node is a read-only replica.
### Step 1. Manually replicate secret GitLab values ### Step 1. Manually replicate secret GitLab values
GitLab stores a number of secret values in the `/etc/gitlab/gitlab-secrets.json` GitLab stores a number of secret values in the `/etc/gitlab/gitlab-secrets.json`
file which *must* match between the primary and secondary nodes. Until there is file which *must* be the same on all nodes. Until there is
a means of automatically replicating these between nodes (see issue [gitlab-org/gitlab-ee#3789]), a means of automatically replicating these between nodes (see issue [gitlab-org/gitlab-ee#3789]),
they must be manually replicated to the secondary. they must be manually replicated to the **secondary** node.
1. SSH into the **primary** node, and execute the command below: 1. SSH into the **primary** node, and execute the command below:
```bash ```sh
sudo cat /etc/gitlab/gitlab-secrets.json sudo cat /etc/gitlab/gitlab-secrets.json
``` ```
...@@ -42,20 +44,20 @@ they must be manually replicated to the secondary. ...@@ -42,20 +44,20 @@ they must be manually replicated to the secondary.
1. SSH into the **secondary** node and login as the `root` user: 1. SSH into the **secondary** node and login as the `root` user:
``` ```sh
sudo -i sudo -i
``` ```
1. Make a backup of any existing secrets: 1. Make a backup of any existing secrets:
```bash ```sh
mv /etc/gitlab/gitlab-secrets.json /etc/gitlab/gitlab-secrets.json.`date +%F` mv /etc/gitlab/gitlab-secrets.json /etc/gitlab/gitlab-secrets.json.`date +%F`
``` ```
1. Copy `/etc/gitlab/gitlab-secrets.json` from the primary to the secondary, or 1. Copy `/etc/gitlab/gitlab-secrets.json` from the **primary** node to the **secondary** node, or
copy-and-paste the file contents between nodes: copy-and-paste the file contents between nodes:
```bash ```sh
sudo editor /etc/gitlab/gitlab-secrets.json sudo editor /etc/gitlab/gitlab-secrets.json
# paste the output of the `cat` command you ran on the primary # paste the output of the `cat` command you ran on the primary
...@@ -64,57 +66,56 @@ they must be manually replicated to the secondary. ...@@ -64,57 +66,56 @@ they must be manually replicated to the secondary.
1. Ensure the file permissions are correct: 1. Ensure the file permissions are correct:
```bash ```sh
chown root:root /etc/gitlab/gitlab-secrets.json chown root:root /etc/gitlab/gitlab-secrets.json
chmod 0600 /etc/gitlab/gitlab-secrets.json chmod 0600 /etc/gitlab/gitlab-secrets.json
``` ```
1. Reconfigure the secondary node for the change to take effect: 1. Reconfigure the **secondary** node for the change to take effect:
``` ```sh
gitlab-ctl reconfigure gitlab-ctl reconfigure
gitlab-ctl restart gitlab-ctl restart
``` ```
### Step 2. Manually replicate primary SSH host keys ### Step 2. Manually replicate the **primary** node's SSH host keys
GitLab integrates with the system-installed SSH daemon, designating a user GitLab integrates with the system-installed SSH daemon, designating a user
(typically named git) through which all access requests are handled. (typically named git) through which all access requests are handled.
In a [Disaster Recovery] situation, GitLab system In a [Disaster Recovery] situation, GitLab system
administrators will promote a secondary Geo replica to a primary and they can administrators will promote a **secondary** node to the **primary** node. DNS records for the
update the DNS records for the primary domain to point to the secondary to prevent **primary** domain should also be updated to point to the new **primary** node
the need to update all references to the primary domain to the secondary domain, (previously a **secondary** node). Doing so will avoid the need to update Git remotes and API URLs.
like changing Git remotes and API URLs.
This will cause all SSH requests to the newly promoted primary node from This will cause all SSH requests to the newly promoted **primary** node to
failing due to SSH host key mismatch. To prevent this, the primary SSH host fail due to SSH host key mismatch. To prevent this, the primary SSH host
keys must be manually replicated to the secondary node. keys must be manually replicated to the **secondary** node.
1. SSH into the **secondary** node and login as the `root` user: 1. SSH into the **secondary** node and login as the `root` user:
``` ```sh
sudo -i sudo -i
``` ```
1. Make a backup of any existing SSH host keys: 1. Make a backup of any existing SSH host keys:
```bash ```sh
find /etc/ssh -iname ssh_host_* -exec cp {} {}.backup.`date +%F` \; find /etc/ssh -iname ssh_host_* -exec cp {} {}.backup.`date +%F` \;
``` ```
1. Copy OpenSSH host keys from **primary**: 1. Copy OpenSSH host keys from the **primary** node:
If you can access your primary node using the **root** user: If you can access your **primary** node using the **root** user:
```bash ```sh
# Run this from the secondary node, change `primary-node-fqdn` for the IP or FQDN of the server # Run this from the secondary node, change `primary-node-fqdn` for the IP or FQDN of the server
scp root@primary-node-fqdn:/etc/ssh/ssh_host_*_key* /etc/ssh scp root@primary-node-fqdn:/etc/ssh/ssh_host_*_key* /etc/ssh
``` ```
If you only have access through a user with **sudo** privileges: If you only have access through a user with **sudo** privileges:
```bash ```sh
# Run this from your primary node: # Run this from your primary node:
sudo tar --transform 's/.*\///g' -zcvf ~/geo-host-key.tar.gz /etc/ssh/ssh_host_*_key* sudo tar --transform 's/.*\///g' -zcvf ~/geo-host-key.tar.gz /etc/ssh/ssh_host_*_key*
...@@ -125,20 +126,20 @@ keys must be manually replicated to the secondary node. ...@@ -125,20 +126,20 @@ keys must be manually replicated to the secondary node.
1. On your **secondary** node, ensure the file permissions are correct: 1. On your **secondary** node, ensure the file permissions are correct:
```bash ```sh
chown root:root /etc/ssh/ssh_host_*_key* chown root:root /etc/ssh/ssh_host_*_key*
chmod 0600 /etc/ssh/ssh_host_*_key* chmod 0600 /etc/ssh/ssh_host_*_key*
``` ```
1. To verify key fingerprint matches, execute the following command on both nodes: 1. To verify key fingerprint matches, execute the following command on both nodes:
```bash ```sh
for file in /etc/ssh/ssh_host_*_key; do ssh-keygen -lf $file; done for file in /etc/ssh/ssh_host_*_key; do ssh-keygen -lf $file; done
``` ```
You should get an output similar to this one and they should be identical on both nodes: You should get an output similar to this one and they should be identical on both nodes:
```bash ```sh
1024 SHA256:FEZX2jQa2bcsd/fn/uxBzxhKdx4Imc4raXrHwsbtP0M root@serverhostname (DSA) 1024 SHA256:FEZX2jQa2bcsd/fn/uxBzxhKdx4Imc4raXrHwsbtP0M root@serverhostname (DSA)
256 SHA256:uw98R35Uf+fYEQ/UnJD9Br4NXUFPv7JAUln5uHlgSeY root@serverhostname (ECDSA) 256 SHA256:uw98R35Uf+fYEQ/UnJD9Br4NXUFPv7JAUln5uHlgSeY root@serverhostname (ECDSA)
256 SHA256:sqOUWcraZQKd89y/QQv/iynPTOGQxcOTIXU/LsoPmnM root@serverhostname (ED25519) 256 SHA256:sqOUWcraZQKd89y/QQv/iynPTOGQxcOTIXU/LsoPmnM root@serverhostname (ED25519)
...@@ -147,7 +148,7 @@ keys must be manually replicated to the secondary node. ...@@ -147,7 +148,7 @@ keys must be manually replicated to the secondary node.
1. Verify that you have the correct public keys for the existing private keys: 1. Verify that you have the correct public keys for the existing private keys:
```bash ```sh
# This will print the fingerprint for private keys: # This will print the fingerprint for private keys:
for file in /etc/ssh/ssh_host_*_key; do ssh-keygen -lf $file; done for file in /etc/ssh/ssh_host_*_key; do ssh-keygen -lf $file; done
...@@ -160,7 +161,7 @@ keys must be manually replicated to the secondary node. ...@@ -160,7 +161,7 @@ keys must be manually replicated to the secondary node.
1. Restart sshd on your **secondary** node: 1. Restart sshd on your **secondary** node:
```bash ```sh
# Debian or Ubuntu installations # Debian or Ubuntu installations
sudo service ssh reload sudo service ssh reload
...@@ -168,42 +169,42 @@ keys must be manually replicated to the secondary node. ...@@ -168,42 +169,42 @@ keys must be manually replicated to the secondary node.
sudo service sshd reload sudo service sshd reload
``` ```
### Step 3. Add the secondary GitLab node ### Step 3. Add the **secondary** node
1. Visit the **primary** node's **Admin Area > Geo** 1. Visit the **primary** node's **Admin Area > Geo**
(`/admin/geo_nodes`) in your browser. (`/admin/geo/nodes`) in your browser.
1. Add the secondary node by providing its full URL. **Do NOT** check the box 1. Add the **secondary** node by providing its full URL. **Do NOT** check the
'This is a primary node'. **This is a primary node** checkbox.
1. Optionally, choose which namespaces should be replicated by the 1. Optionally, choose which namespaces should be replicated by the
secondary node. Leave blank to replicate all. Read more in **secondary** node. Leave blank to replicate all. Read more in
[selective synchronization](#selective-synchronization). [selective synchronization](#selective-synchronization).
1. Click the **Add node** button. 1. Click the **Add node** button.
1. SSH into your GitLab **secondary** server and restart the services: 1. SSH into your GitLab **secondary** server and restart the services:
```bash ```sh
gitlab-ctl restart gitlab-ctl restart
``` ```
Check if there are any common issue with your Geo setup by running: Check if there are any common issue with your Geo setup by running:
```bash ```sh
gitlab-rake gitlab:geo:check gitlab-rake gitlab:geo:check
``` ```
1. SSH into your GitLab **primary** server and login as root to verify the 1. SSH into your **primary** server and login as root to verify the
secondary is reachable or there are any common issue with your Geo setup: **secondary** node is reachable or there are any common issue with your Geo setup:
```bash ```sh
gitlab-rake gitlab:geo:check gitlab-rake gitlab:geo:check
``` ```
Once added to the admin panel and restarted, the secondary will automatically start Once added to the admin panel and restarted, the **secondary** node will automatically start
replicating missing data from the primary in a process known as **backfill**. replicating missing data from the **primary** node in a process known as **backfill**.
Meanwhile, the primary node will start to notify the secondary of any changes, so Meanwhile, the **primary** node will start to notify each **secondary** node of any changes, so
that the secondary can act on those notifications immediately. that the **secondary** node can act on those notifications immediately.
Make sure the secondary instance is running and accessible. Make sure the **secondary** node is running and accessible.
You can login to the secondary node with the same credentials as used in the primary. You can login to the **secondary** node with the same credentials as used for the **primary** node.
### Step 4. Enabling Hashed Storage ### Step 4. Enabling Hashed Storage
...@@ -214,34 +215,34 @@ renames no longer require synchronization between nodes. ...@@ -214,34 +215,34 @@ renames no longer require synchronization between nodes.
(`/admin/application_settings/repository`) in your browser. (`/admin/application_settings/repository`) in your browser.
1. In the **Repository storage** section, check **Use hashed storage paths for newly created and renamed projects**. 1. In the **Repository storage** section, check **Use hashed storage paths for newly created and renamed projects**.
### Step 5. (Optional) Configuring the secondary to trust the primary ### Step 5. (Optional) Configuring the **secondary** node to trust the **primary** node
You can safely skip this step if your primary uses a CA-issued HTTPS certificate. You can safely skip this step if your **primary** node uses a CA-issued HTTPS certificate.
If your primary is using a self-signed certificate for *HTTPS* support, you will If your **primary** node is using a self-signed certificate for *HTTPS* support, you will
need to add that certificate to the secondary's trust store. Retrieve the need to add that certificate to the **secondary** node's trust store. Retrieve the
certificate from the primary and follow certificate from the **primary** node and follow
[these instructions][omnibus-ssl] [these instructions][omnibus-ssl]
on the secondary. on the **secondary** node.
### Step 6. Enable Git access over HTTP/HTTPS ### Step 6. Enable Git access over HTTP/HTTPS
Geo synchronizes repositories over HTTP/HTTPS, and therefore requires this clone Geo synchronizes repositories over HTTP/HTTPS, and therefore requires this clone
method to be enabled. Navigate to **Admin Area > Settings** method to be enabled. Navigate to **Admin Area > Settings**
(`/admin/application_settings`) on the primary node, and set (`/admin/application_settings`) on the **primary** node, and set
`Enabled Git access protocols` to `Both SSH and HTTP(S)` or `Only HTTP(S)`. `Enabled Git access protocols` to `Both SSH and HTTP(S)` or `Only HTTP(S)`.
### Step 7. Verify proper functioning of the secondary node ### Step 7. Verify proper functioning of the **secondary** node
Congratulations! Your secondary geo node is now configured! Your **secondary** node is now configured!
You can login to the secondary node with the same credentials you used on the You can login to the **secondary** node with the same credentials you used for the
primary. Visit the secondary node's **Admin Area > Geo** **primary** node. Visit the **secondary** node's **Admin Area > Geo**
(`/admin/geo_nodes`) in your browser to check if it's correctly identified as a (`/admin/geo/nodes`) in your browser to check if it's correctly identified as a
secondary Geo node and if Geo is enabled. **secondary** Geo node and if Geo is enabled.
The initial replication, or 'backfill', will probably still be in progress. You The initial replication, or 'backfill', will probably still be in progress. You
can monitor the synchronization process on each geo node from the primary can monitor the synchronization process on each geo node from the **primary**
node's Geo Nodes dashboard in your browser. node's Geo Nodes dashboard in your browser.
![Geo dashboard](img/geo_node_dashboard.png) ![Geo dashboard](img/geo_node_dashboard.png)
...@@ -251,44 +252,44 @@ If your installation isn't working properly, check the ...@@ -251,44 +252,44 @@ If your installation isn't working properly, check the
The two most obvious issues that can become apparent in the dashboard are: The two most obvious issues that can become apparent in the dashboard are:
1. Database replication not working well 1. Database replication not working well.
2. Instance to instance notification not working. In that case, it can be 1. Instance to instance notification not working. In that case, it can be
something of the following: something of the following:
- You are using a custom certificate or custom CA (see the - You are using a custom certificate or custom CA (see the
[troubleshooting document]) [troubleshooting document]).
- The instance is firewalled (check your firewall rules) - The instance is firewalled (check your firewall rules).
Please note that disabling a secondary node will stop the sync process. Please note that disabling a **secondary** node will stop the synchronization process.
Please note that if `git_data_dirs` is customized on the primary for multiple Please note that if `git_data_dirs` is customized on the **primary** node for multiple
repository shards you must duplicate the same configuration on the secondary. repository shards you must duplicate the same configuration on each **secondary** node.
Point your users to the ["Using a Geo Server" guide][using-geo]. Point your users to the ["Using a Geo Server" guide][using-geo].
Currently, this is what is synced: Currently, this is what is synced:
* Git repositories - Git repositories.
* Wikis - Wikis.
* LFS objects - LFS objects.
* Issues, merge requests, snippets, and comment attachments - Issues, merge requests, snippets, and comment attachments.
* Users, groups, and project avatars - Users, groups, and project avatars.
## Selective synchronization ## Selective synchronization
Geo supports selective synchronization, which allows admins to choose Geo supports selective synchronization, which allows admins to choose
which projects should be synchronized by secondary nodes. which projects should be synchronized by **secondary** nodes.
It is important to note that selective synchronization does not: It is important to note that selective synchronization does not:
1. Restrict permissions from secondary nodes. 1. Restrict permissions from **secondary** nodes.
1. Hide project metadata from secondary nodes. 1. Hide project metadata from **secondary** nodes.
* Since Geo currently relies on PostgreSQL replication, all project metadata - Since Geo currently relies on PostgreSQL replication, all project metadata
gets replicated to secondary nodes, but repositories that have not been gets replicated to **secondary** nodes, but repositories that have not been
selected will be empty. selected will be empty.
1. Reduce the number of events generated for the Geo event log 1. Reduce the number of events generated for the Geo event log.
* The primary generates events as long as any secondaries are present. - The **primary** node generates events as long as any **secondary** nodes are present.
Selective synchronization restrictions are implemented on the secondaries, Selective synchronization restrictions are implemented on the **secondary** nodes,
not the primary. not the **primary** node.
A subset of projects can be chosen, either by group or by storage shard. The A subset of projects can be chosen, either by group or by storage shard. The
former is ideal for replicating data belonging to a subset of users, while the former is ideal for replicating data belonging to a subset of users, while the
......
...@@ -6,37 +6,38 @@ Please consider [migrating to GitLab Omnibus install](https://docs.gitlab.com/om ...@@ -6,37 +6,38 @@ Please consider [migrating to GitLab Omnibus install](https://docs.gitlab.com/om
using the Omnibus GitLab packages, follow the using the Omnibus GitLab packages, follow the
[**Omnibus Geo nodes configuration**][configuration] guide. [**Omnibus Geo nodes configuration**][configuration] guide.
## Configuring a new secondary node ## Configuring a new **secondary** node
NOTE: **Note:** NOTE: **Note:**
This is the final step in setting up a secondary Geo node. Stages of the setup This is the final step in setting up a **secondary** node. Stages of the setup
process must be completed in the documented order. Before attempting the steps process must be completed in the documented order. Before attempting the steps
in this stage, [complete all prior stages][setup-geo-source]. in this stage, [complete all prior stages][setup-geo-source].
The basic steps of configuring a secondary node are to replicate required The basic steps of configuring a **secondary** node are to:
configurations between the primary and the secondaries; to configure a tracking
database on each secondary; and to start GitLab on the secondary node. - Replicate required configurations between the **primary** and **secondary** nodes.
- Configure a tracking database on each **secondary** node.
- Start GitLab on the **secondary** node.
You are encouraged to first read through all the steps before executing them You are encouraged to first read through all the steps before executing them
in your testing/production environment. in your testing/production environment.
NOTE: **Note:**
**Do not** set up any custom authentication on **secondary** nodes, this will be handled by the **primary** node.
NOTE: **Notes:** NOTE: **Note:**
- **Do not** setup any custom authentication in the secondary nodes, this will be **Do not** add anything in the **secondary** node's admin area (**Admin Area > Geo**). This is handled solely by the **primary** node.
handled by the primary node.
- **Do not** add anything in the secondaries Geo nodes admin area
(**Admin Area > Geo**). This is handled solely by the primary node.
### Step 1. Manually replicate secret GitLab values ### Step 1. Manually replicate secret GitLab values
GitLab stores a number of secret values in the `/home/git/gitlab/config/secrets.yml` GitLab stores a number of secret values in the `/home/git/gitlab/config/secrets.yml`
file which *must* match between the primary and secondary nodes. Until there is file which *must* match between the **primary** and **secondary** nodes. Until there is
a means of automatically replicating these between nodes (see [gitlab-org/gitlab-ee#3789]), they must a means of automatically replicating these between nodes (see [gitlab-org/gitlab-ee#3789]), they must
be manually replicated to the secondary. be manually replicated to **secondary** nodes.
1. SSH into the **primary** node, and execute the command below: 1. SSH into the **primary** node, and execute the command below:
```bash ```sh
sudo cat /home/git/gitlab/config/secrets.yml sudo cat /home/git/gitlab/config/secrets.yml
``` ```
...@@ -44,20 +45,20 @@ be manually replicated to the secondary. ...@@ -44,20 +45,20 @@ be manually replicated to the secondary.
1. SSH into the **secondary** node and login as the `git` user: 1. SSH into the **secondary** node and login as the `git` user:
```bash ```sh
sudo -i -u git sudo -i -u git
``` ```
1. Make a backup of any existing secrets: 1. Make a backup of any existing secrets:
```bash ```sh
mv /home/git/gitlab/config/secrets.yml /home/git/gitlab/config/secrets.yml.`date +%F` mv /home/git/gitlab/config/secrets.yml /home/git/gitlab/config/secrets.yml.`date +%F`
``` ```
1. Copy `/home/git/gitlab/config/secrets.yml` from the primary to the secondary, or 1. Copy `/home/git/gitlab/config/secrets.yml` from the **primary** node to the **secondary** node, or
copy-and-paste the file contents between nodes: copy-and-paste the file contents between nodes:
```bash ```sh
sudo editor /home/git/gitlab/config/secrets.yml sudo editor /home/git/gitlab/config/secrets.yml
# paste the output of the `cat` command you ran on the primary # paste the output of the `cat` command you ran on the primary
...@@ -66,65 +67,65 @@ be manually replicated to the secondary. ...@@ -66,65 +67,65 @@ be manually replicated to the secondary.
1. Ensure the file permissions are correct: 1. Ensure the file permissions are correct:
```bash ```sh
chown git:git /home/git/gitlab/config/secrets.yml chown git:git /home/git/gitlab/config/secrets.yml
chmod 0600 /home/git/gitlab/config/secrets.yml chmod 0600 /home/git/gitlab/config/secrets.yml
``` ```
1. Restart GitLab 1. Restart GitLab
```bash ```sh
service gitlab restart service gitlab restart
``` ```
Once restarted, the secondary will automatically start replicating missing data Once restarted, the **secondary** node will automatically start replicating missing data
from the primary in a process known as backfill. Meanwhile, the primary node from the **primary** node in a process known as backfill. Meanwhile, the **primary** node
will start to notify the secondary of any changes, so that the secondary can will start to notify the **secondary** node of any changes, so that the **secondary** node can
act on those notifications immediately. act on those notifications immediately.
Make sure the secondary instance is running and accessible. You can login to Make sure the **secondary** node is running and accessible. You can login to
the secondary node with the same credentials as used in the primary. the **secondary** node with the same credentials as used for the **primary** node.
### Step 2. Manually replicate primary SSH host keys ### Step 2. Manually replicate the **primary** node's SSH host keys
Read [Manually replicate primary SSH host keys][configuration-replicate-ssh] Read [Manually replicate the **primary** node's SSH host keys][configuration-replicate-ssh]
### Step 3. Add the secondary GitLab node ### Step 3. Add the **secondary** GitLab node
1. Navigate to the **primary** node's **Admin Area > Geo** 1. Navigate to the **primary** node's **Admin Area > Geo**
(`/admin/geo_nodes`) in your browser. (`/admin/geo/nodes`) in your browser.
1. Add the secondary node by providing its full URL. **Do NOT** check the box 1. Add the **secondary** node by providing its full URL. **Do NOT** check the
'This is a primary node'. **This is a primary node** checkbox.
1. Optionally, choose which namespaces should be replicated by the 1. Optionally, choose which namespaces should be replicated by the
secondary node. Leave blank to replicate all. Read more in **secondary** node. Leave blank to replicate all. Read more in
[selective synchronization](#selective-synchronization). [selective synchronization](#selective-synchronization).
1. Click the **Add node** button. 1. Click the **Add node** button.
1. SSH into your GitLab **secondary** server and restart the services: 1. SSH into your GitLab **secondary** server and restart the services:
```bash ```sh
service gitlab restart service gitlab restart
``` ```
Check if there are any common issue with your Geo setup by running: Check if there are any common issue with your Geo setup by running:
```bash ```sh
bundle exec rake gitlab:geo:check bundle exec rake gitlab:geo:check
``` ```
1. SSH into your GitLab **primary** server and login as root to verify the 1. SSH into your GitLab **primary** server and login as root to verify the
secondary is reachable or there are any common issue with your Geo setup: **secondary** node is reachable or there are any common issue with your Geo setup:
```bash ```sh
bundle exec rake gitlab:geo:check bundle exec rake gitlab:geo:check
``` ```
Once reconfigured, the secondary will automatically start Once reconfigured, the **secondary** node will automatically start
replicating missing data from the primary in a process known as backfill. replicating missing data from the **primary** node in a process known as backfill.
Meanwhile, the primary node will start to notify the secondary of any changes, so Meanwhile, the **primary** node will start to notify the **secondary** node of any changes, so
that the secondary can act on those notifications immediately. that the **secondary** node can act on those notifications immediately.
Make sure the secondary instance is running and accessible. Make sure the **secondary** node is running and accessible.
You can login to the secondary node with the same credentials as used in the primary. You can log in to the **secondary** node with the same credentials as used for the **primary** node.
### Step 4. Enabling Hashed Storage ### Step 4. Enabling Hashed Storage
...@@ -132,15 +133,15 @@ Read [Enabling Hashed Storage](configuration.md##step-4-enabling-hashed-storage) ...@@ -132,15 +133,15 @@ Read [Enabling Hashed Storage](configuration.md##step-4-enabling-hashed-storage)
### Step 5. (Optional) Configuring the secondary to trust the primary ### Step 5. (Optional) Configuring the secondary to trust the primary
You can safely skip this step if your primary uses a CA-issued HTTPS certificate. You can safely skip this step if your **primary** node uses a CA-issued HTTPS certificate.
If your primary is using a self-signed certificate for *HTTPS* support, you will If your **primary** node is using a self-signed certificate for *HTTPS* support, you will
need to add that certificate to the secondary's trust store. Retrieve the need to add that certificate to the **secondary** node's trust store. Retrieve the
certificate from the primary and follow your distribution's instructions for certificate from the **primary** node and follow your distribution's instructions for
adding it to the secondary's trust store. In Debian/Ubuntu, for example, with a adding it to the **secondary** node's trust store. In Debian/Ubuntu, for example, with a
certificate file of `primary.geo.example.com.crt`, you would follow these steps: certificate file of `primary.geo.example.com.crt`, you would follow these steps:
``` ```sh
sudo -i sudo -i
cp primary.geo.example.com.crt /usr/local/share/ca-certificates cp primary.geo.example.com.crt /usr/local/share/ca-certificates
update-ca-certificates update-ca-certificates
...@@ -150,7 +151,7 @@ update-ca-certificates ...@@ -150,7 +151,7 @@ update-ca-certificates
Geo synchronizes repositories over HTTP/HTTPS, and therefore requires this clone Geo synchronizes repositories over HTTP/HTTPS, and therefore requires this clone
method to be enabled. Navigate to **Admin Area > Settings** method to be enabled. Navigate to **Admin Area > Settings**
(`/admin/application_settings`) on the primary node, and set (`/admin/application_settings`) on the **primary** node, and set
`Enabled Git access protocols` to `Both SSH and HTTP(S)` or `Only HTTP(S)`. `Enabled Git access protocols` to `Both SSH and HTTP(S)` or `Only HTTP(S)`.
### Step 7. Verify proper functioning of the secondary node ### Step 7. Verify proper functioning of the secondary node
......
...@@ -16,7 +16,7 @@ The stages of the setup process must be completed in the documented order. ...@@ -16,7 +16,7 @@ The stages of the setup process must be completed in the documented order.
Before attempting the steps in this stage, [complete all prior stages][toc]. Before attempting the steps in this stage, [complete all prior stages][toc].
This document describes the minimal steps you have to take in order to This document describes the minimal steps you have to take in order to
replicate your primary GitLab database to a secondary node's database. You may replicate your **primary** GitLab database to a **secondary** node's database. You may
have to change some values according to your database setup, how big it is, etc. have to change some values according to your database setup, how big it is, etc.
You are encouraged to first read through all the steps before executing them You are encouraged to first read through all the steps before executing them
...@@ -24,16 +24,16 @@ in your testing/production environment. ...@@ -24,16 +24,16 @@ in your testing/production environment.
## PostgreSQL replication ## PostgreSQL replication
The GitLab primary node where the write operations happen will connect to The GitLab **primary** node where the write operations happen will connect to
the primary database server, and the secondary nodes which are read-only will the **primary** database server, and **secondary** nodes will
connect to the secondary database servers (which are also read-only). connect to their own database servers (which are also read-only).
NOTE: **Note:** NOTE: **Note:**
In database documentation you may see "primary" being referenced as "master" In database documentation, you may see "**primary**" being referenced as "master"
and "secondary" as either "slave" or "standby" server (read-only). and "**secondary**" as either "slave" or "standby" server (read-only).
We recommend using [PostgreSQL replication slots][replication-slots-article] We recommend using [PostgreSQL replication slots][replication-slots-article]
to ensure that the primary retains all the data necessary for the secondaries to to ensure that the **primary** node retains all the data necessary for the **secondary** nodes to
recover. See below for more details. recover. See below for more details.
The following guide assumes that: The following guide assumes that:
...@@ -41,12 +41,12 @@ The following guide assumes that: ...@@ -41,12 +41,12 @@ The following guide assumes that:
- You are using Omnibus and therefore you are using PostgreSQL 9.6 or later - You are using Omnibus and therefore you are using PostgreSQL 9.6 or later
which includes the [`pg_basebackup` tool][pgback] and improved which includes the [`pg_basebackup` tool][pgback] and improved
[Foreign Data Wrapper][FDW] support. [Foreign Data Wrapper][FDW] support.
- You have a primary node already set up (the GitLab server you are - You have a **primary** node already set up (the GitLab server you are
replicating from), running Omnibus' PostgreSQL (or equivalent version), and replicating from), running Omnibus' PostgreSQL (or equivalent version), and
you have a new secondary server set up with the same versions of the OS, you have a new **secondary** server set up with the same versions of the OS,
PostgreSQL, and GitLab on all nodes. PostgreSQL, and GitLab on all nodes.
- The IP of the primary server for our examples will be `1.2.3.4`, whereas the - The IP of the **primary** server for our examples will be `1.2.3.4`, whereas the
secondary's IP will be `5.6.7.8`. Note that the primary and secondary servers **secondary** node's IP will be `5.6.7.8`. Note that the **primary** and **secondary** servers
**must** be able to communicate over these addresses. More on this in the **must** be able to communicate over these addresses. More on this in the
guide below. guide below.
...@@ -54,27 +54,27 @@ CAUTION: **Warning:** ...@@ -54,27 +54,27 @@ CAUTION: **Warning:**
Geo works with streaming replication. Logical replication is not supported at this time. Geo works with streaming replication. Logical replication is not supported at this time.
There is an [issue where support is being discussed](https://gitlab.com/gitlab-org/gitlab-ee/issues/7420). There is an [issue where support is being discussed](https://gitlab.com/gitlab-org/gitlab-ee/issues/7420).
### Step 1. Configure the primary server ### Step 1. Configure the **primary** server
1. SSH into your GitLab **primary** server and login as root: 1. SSH into your GitLab **primary** server and login as root:
```bash ```sh
sudo -i sudo -i
``` ```
1. Execute the command below to define the node as primary Geo node: 1. Execute the command below to define the node as **primary** node:
```bash ```sh
gitlab-ctl set-geo-primary-node gitlab-ctl set-geo-primary-node
``` ```
This command will use your defined `external_url` in `/etc/gitlab/gitlab.rb`. This command will use your defined `external_url` in `/etc/gitlab/gitlab.rb`.
1. GitLab 10.4 and up only: Do the following to make sure the `gitlab` database user has a password defined 1. GitLab 10.4 and up only: Do the following to make sure the `gitlab` database user has a password defined:
Generate a MD5 hash of the desired password: Generate a MD5 hash of the desired password:
```bash ```sh
gitlab-ctl pg-password-md5 gitlab gitlab-ctl pg-password-md5 gitlab
# Enter password: mypassword # Enter password: mypassword
# Confirm password: mypassword # Confirm password: mypassword
...@@ -97,7 +97,7 @@ There is an [issue where support is being discussed](https://gitlab.com/gitlab-o ...@@ -97,7 +97,7 @@ There is an [issue where support is being discussed](https://gitlab.com/gitlab-o
called `gitlab_replicator`. You must set the password for this user manually. called `gitlab_replicator`. You must set the password for this user manually.
You will be prompted to enter a password: You will be prompted to enter a password:
```bash ```sh
gitlab-ctl set-replication-password gitlab-ctl set-replication-password
``` ```
...@@ -111,11 +111,11 @@ There is an [issue where support is being discussed](https://gitlab.com/gitlab-o ...@@ -111,11 +111,11 @@ There is an [issue where support is being discussed](https://gitlab.com/gitlab-o
[appropriate step](database_source.md#step-1-configure-the-primary-server) [appropriate step](database_source.md#step-1-configure-the-primary-server)
in [Geo database replication (source)](database_source.md). in [Geo database replication (source)](database_source.md).
1. Configure PostgreSQL to listen on network interfaces 1. Configure PostgreSQL to listen on network interfaces:
For security reasons, PostgreSQL does not listen on any network interfaces For security reasons, PostgreSQL does not listen on any network interfaces
by default. However, Geo requires the secondary to be able to by default. However, Geo requires the **secondary** node to be able to
connect to the primary's database. For this reason, we need the address of connect to the **primary** node's database. For this reason, we need the address of
each node. Note: For external PostgreSQL instances, see [additional instructions](external_database.md). each node. Note: For external PostgreSQL instances, see [additional instructions](external_database.md).
If you are using a cloud provider, you can lookup the addresses for each If you are using a cloud provider, you can lookup the addresses for each
...@@ -123,7 +123,7 @@ There is an [issue where support is being discussed](https://gitlab.com/gitlab-o ...@@ -123,7 +123,7 @@ There is an [issue where support is being discussed](https://gitlab.com/gitlab-o
To lookup the address of a Geo node, SSH in to the Geo node and execute: To lookup the address of a Geo node, SSH in to the Geo node and execute:
```bash ```sh
## ##
## Private address ## Private address
## ##
...@@ -138,13 +138,13 @@ There is an [issue where support is being discussed](https://gitlab.com/gitlab-o ...@@ -138,13 +138,13 @@ There is an [issue where support is being discussed](https://gitlab.com/gitlab-o
In most cases, the following addresses will be used to configure GitLab In most cases, the following addresses will be used to configure GitLab
Geo: Geo:
| Configuration | Address | | Configuration | Address |
|-----------------------------------------|---------------------------------------------| |:----------------------------------------|:------------------------------------------------------|
| `postgresql['listen_address']` | Primary's public or VPC private address | | `postgresql['listen_address']` | **Primary** node's public or VPC private address. |
| `postgresql['md5_auth_cidr_addresses']` | Secondary's public or VPC private addresses | | `postgresql['md5_auth_cidr_addresses']` | **Secondary** node's public or VPC private addresses. |
If you are using Google Cloud Platform, SoftLayer, or any other vendor that If you are using Google Cloud Platform, SoftLayer, or any other vendor that
provides a virtual private cloud (VPC) you can use the secondary's private provides a virtual private cloud (VPC) you can use the **secondary** node's private
address (corresponds to "internal address" for Google Cloud Platform) for address (corresponds to "internal address" for Google Cloud Platform) for
`postgresql['md5_auth_cidr_addresses']` and `postgresql['listen_address']`. `postgresql['md5_auth_cidr_addresses']` and `postgresql['listen_address']`.
...@@ -153,10 +153,10 @@ There is an [issue where support is being discussed](https://gitlab.com/gitlab-o ...@@ -153,10 +153,10 @@ There is an [issue where support is being discussed](https://gitlab.com/gitlab-o
documentation][pg-docs-runtime-conn] for more details. documentation][pg-docs-runtime-conn] for more details.
Depending on your network configuration, the suggested addresses may not Depending on your network configuration, the suggested addresses may not
be correct. If your primary and secondary connect over a local be correct. If your **primary** node and **secondary** nodes connect over a local
area network, or a virtual network connecting availability zones like area network, or a virtual network connecting availability zones like
[Amazon's VPC](https://aws.amazon.com/vpc/) or [Google's VPC](https://cloud.google.com/vpc/) [Amazon's VPC](https://aws.amazon.com/vpc/) or [Google's VPC](https://cloud.google.com/vpc/)
you should use the secondary's private address for `postgresql['md5_auth_cidr_addresses']`. you should use the **secondary** node's private address for `postgresql['md5_auth_cidr_addresses']`.
Edit `/etc/gitlab/gitlab.rb` and add the following, replacing the IP Edit `/etc/gitlab/gitlab.rb` and add the following, replacing the IP
addresses with addresses appropriate to your network configuration: addresses with addresses appropriate to your network configuration:
...@@ -196,7 +196,7 @@ There is an [issue where support is being discussed](https://gitlab.com/gitlab-o ...@@ -196,7 +196,7 @@ There is an [issue where support is being discussed](https://gitlab.com/gitlab-o
gitlab_rails['auto_migrate'] = false gitlab_rails['auto_migrate'] = false
``` ```
1. Optional: If you want to add another secondary, the relevant setting would look like: 1. Optional: If you want to add another **secondary** node, the relevant setting would look like:
```ruby ```ruby
postgresql['md5_auth_cidr_addresses'] = ['1.2.3.4/32', '5.6.7.8/32','9.10.11.12/32'] postgresql['md5_auth_cidr_addresses'] = ['1.2.3.4/32', '5.6.7.8/32','9.10.11.12/32']
...@@ -210,13 +210,13 @@ There is an [issue where support is being discussed](https://gitlab.com/gitlab-o ...@@ -210,13 +210,13 @@ There is an [issue where support is being discussed](https://gitlab.com/gitlab-o
1. Save the file and reconfigure GitLab for the database listen changes and 1. Save the file and reconfigure GitLab for the database listen changes and
the replication slot changes to be applied: the replication slot changes to be applied:
```bash ```sh
gitlab-ctl reconfigure gitlab-ctl reconfigure
``` ```
Restart PostgreSQL for its changes to take effect: Restart PostgreSQL for its changes to take effect:
```bash ```sh
gitlab-ctl restart postgresql gitlab-ctl restart postgresql
``` ```
...@@ -231,29 +231,29 @@ There is an [issue where support is being discussed](https://gitlab.com/gitlab-o ...@@ -231,29 +231,29 @@ There is an [issue where support is being discussed](https://gitlab.com/gitlab-o
Save the file and reconfigure GitLab: Save the file and reconfigure GitLab:
```bash ```sh
gitlab-ctl reconfigure gitlab-ctl reconfigure
``` ```
1. Now that the PostgreSQL server is set up to accept remote connections, run 1. Now that the PostgreSQL server is set up to accept remote connections, run
`netstat -plnt | grep 5432` to make sure that PostgreSQL is listening on port `netstat -plnt | grep 5432` to make sure that PostgreSQL is listening on port
`5432` to the primary server's private address. `5432` to the **primary** server's private address.
1. A certificate was automatically generated when GitLab was reconfigured. This 1. A certificate was automatically generated when GitLab was reconfigured. This
will be used automatically to protect your PostgreSQL traffic from will be used automatically to protect your PostgreSQL traffic from
eavesdroppers, but to protect against active ("man-in-the-middle") attackers, eavesdroppers, but to protect against active ("man-in-the-middle") attackers,
the secondary needs a copy of the certificate. Make a copy of the PostgreSQL the **secondary** node needs a copy of the certificate. Make a copy of the PostgreSQL
`server.crt` file on the primary node by running this command: `server.crt` file on the **primary** node by running this command:
```bash ```sh
cat ~gitlab-psql/data/server.crt cat ~gitlab-psql/data/server.crt
``` ```
Copy the output into a clipboard or into a local file. You Copy the output into a clipboard or into a local file. You
will need it when setting up the secondary! The certificate is not sensitive will need it when setting up the **secondary** node! The certificate is not sensitive
data. data.
### Step 2. Configure the secondary server ### Step 2. Configure the **secondary** server
1. SSH into your GitLab **secondary** server and login as root: 1. SSH into your GitLab **secondary** server and login as root:
...@@ -271,9 +271,9 @@ There is an [issue where support is being discussed](https://gitlab.com/gitlab-o ...@@ -271,9 +271,9 @@ There is an [issue where support is being discussed](https://gitlab.com/gitlab-o
NOTE: **Note**: NOTE: **Note**:
This step is important so we don't try to execute anything before the node is fully configured. This step is important so we don't try to execute anything before the node is fully configured.
1. [Check TCP connectivity][rake-maintenance] to the primary's PostgreSQL server: 1. [Check TCP connectivity][rake-maintenance] to the **primary** node's PostgreSQL server:
```bash ```sh
gitlab-rake gitlab:tcp_check[1.2.3.4,5432] gitlab-rake gitlab:tcp_check[1.2.3.4,5432]
``` ```
...@@ -281,44 +281,44 @@ There is an [issue where support is being discussed](https://gitlab.com/gitlab-o ...@@ -281,44 +281,44 @@ There is an [issue where support is being discussed](https://gitlab.com/gitlab-o
If this step fails, you may be using the wrong IP address, or a firewall may If this step fails, you may be using the wrong IP address, or a firewall may
be preventing access to the server. Check the IP address, paying close be preventing access to the server. Check the IP address, paying close
attention to the difference between public and private addresses and ensure attention to the difference between public and private addresses and ensure
that, if a firewall is present, the secondary is permitted to connect to the that, if a firewall is present, the **secondary** node is permitted to connect to the
primary on port 5432. **primary** node on port 5432.
1. Create a file `server.crt` in the secondary server, with the content you got on the last step of the primary setup: 1. Create a file `server.crt` in the **secondary** server, with the content you got on the last step of the **primary** node's setup:
``` ```
editor server.crt editor server.crt
``` ```
1. Set up PostgreSQL TLS verification on the secondary 1. Set up PostgreSQL TLS verification on the **secondary** node:
Install the `server.crt` file: Install the `server.crt` file:
```bash ```sh
install -D -o gitlab-psql -g gitlab-psql -m 0400 -T server.crt ~gitlab-psql/.postgresql/root.crt install -D -o gitlab-psql -g gitlab-psql -m 0400 -T server.crt ~gitlab-psql/.postgresql/root.crt
``` ```
PostgreSQL will now only recognize that exact certificate when verifying TLS PostgreSQL will now only recognize that exact certificate when verifying TLS
connections. The certificate can only be replicated by someone with access connections. The certificate can only be replicated by someone with access
to the private key, which is **only** present on the primary node. to the private key, which is **only** present on the **primary** node.
1. Test that the `gitlab-psql` user can connect to the primary's database: 1. Test that the `gitlab-psql` user can connect to the **primary** node's database:
```bash ```sh
sudo -u gitlab-psql /opt/gitlab/embedded/bin/psql --list -U gitlab_replicator -d "dbname=gitlabhq_production sslmode=verify-ca" -W -h 1.2.3.4 sudo -u gitlab-psql /opt/gitlab/embedded/bin/psql --list -U gitlab_replicator -d "dbname=gitlabhq_production sslmode=verify-ca" -W -h 1.2.3.4
``` ```
When prompted enter the password you set in the first step for the When prompted enter the password you set in the first step for the
`gitlab_replicator` user. If all worked correctly, you should see `gitlab_replicator` user. If all worked correctly, you should see
the list of primary's databases. the list of **primary** node's databases.
A failure to connect here indicates that the TLS configuration is incorrect. A failure to connect here indicates that the TLS configuration is incorrect.
Ensure that the contents of `~gitlab-psql/data/server.crt` on the primary Ensure that the contents of `~gitlab-psql/data/server.crt` on the **primary** node
match the contents of `~gitlab-psql/.postgresql/root.crt` on the secondary. match the contents of `~gitlab-psql/.postgresql/root.crt` on the **secondary** node.
1. Configure PostgreSQL to enable FDW support 1. Configure PostgreSQL to enable FDW support:
This step is similar to how we configured the primary instance. This step is similar to how we configured the **primary** instance.
We need to enable this, to enable FDW support, even if using a single node. We need to enable this, to enable FDW support, even if using a single node.
Edit `/etc/gitlab/gitlab.rb` and add the following, replacing the IP Edit `/etc/gitlab/gitlab.rb` and add the following, replacing the IP
...@@ -356,13 +356,13 @@ There is an [issue where support is being discussed](https://gitlab.com/gitlab-o ...@@ -356,13 +356,13 @@ There is an [issue where support is being discussed](https://gitlab.com/gitlab-o
1. Reconfigure GitLab for the changes to take effect: 1. Reconfigure GitLab for the changes to take effect:
```bash ```sh
gitlab-ctl reconfigure gitlab-ctl reconfigure
``` ```
1. Restart PostgreSQL for the IP change to take effect and Reconfigure again: 1. Restart PostgreSQL for the IP change to take effect and reconfigure again:
```bash ```sh
gitlab-ctl restart postgresql gitlab-ctl restart postgresql
gitlab-ctl reconfigure gitlab-ctl reconfigure
``` ```
...@@ -371,8 +371,8 @@ There is an [issue where support is being discussed](https://gitlab.com/gitlab-o ...@@ -371,8 +371,8 @@ There is an [issue where support is being discussed](https://gitlab.com/gitlab-o
### Step 3. Initiate the replication process ### Step 3. Initiate the replication process
Below we provide a script that connects the database on the secondary node to Below we provide a script that connects the database on the **secondary** node to
the database on the primary node, replicates the database, and creates the the database on the **primary** node, replicates the database, and creates the
needed files for streaming replication. needed files for streaming replication.
The directories used are the defaults that are set up in Omnibus. If you have The directories used are the defaults that are set up in Omnibus. If you have
...@@ -385,20 +385,20 @@ data before running `pg_basebackup`. ...@@ -385,20 +385,20 @@ data before running `pg_basebackup`.
1. SSH into your GitLab **secondary** server and login as root: 1. SSH into your GitLab **secondary** server and login as root:
``` ```sh
sudo -i sudo -i
``` ```
1. Choose a database-friendly name to use for your secondary to 1. Choose a database-friendly name to use for your **secondary** node to
use as the replication slot name. For example, if your domain is use as the replication slot name. For example, if your domain is
`secondary.geo.example.com`, you may use `secondary_example` as the slot `secondary.geo.example.com`, you may use `secondary_example` as the slot
name as shown in the commands below. name as shown in the commands below.
1. Execute the command below to start a backup/restore and begin the replication 1. Execute the command below to start a backup/restore and begin the replication
CAUTION: **Warning:** Each Geo secondary must have its own unique replication slot name. CAUTION: **Warning:** Each Geo **secondary** node must have its own unique replication slot name.
Using the same slot name between two secondaries will break PostgreSQL replication. Using the same slot name between two secondaries will break PostgreSQL replication.
```bash ```sh
gitlab-ctl replicate-geo-database --slot-name=secondary_example --host=1.2.3.4 gitlab-ctl replicate-geo-database --slot-name=secondary_example --host=1.2.3.4
``` ```
...@@ -419,9 +419,9 @@ data before running `pg_basebackup`. ...@@ -419,9 +419,9 @@ data before running `pg_basebackup`.
the instructions above are carefully written to ensure protection against the instructions above are carefully written to ensure protection against
both passive eavesdroppers and active "man-in-the-middle" attackers. both passive eavesdroppers and active "man-in-the-middle" attackers.
- Change the `--slot-name` to the name of the replication slot - Change the `--slot-name` to the name of the replication slot
to be used on the primary database. The script will attempt to create the to be used on the **primary** database. The script will attempt to create the
replication slot automatically if it does not exist. replication slot automatically if it does not exist.
- If you're repurposing an old server into a Geo secondary, you'll need to - If you're repurposing an old server into a Geo **secondary** node, you'll need to
add `--force` to the command line. add `--force` to the command line.
- When not in a production machine you can disable backup step if you - When not in a production machine you can disable backup step if you
really sure this is what you want by adding `--skip-backup` really sure this is what you want by adding `--skip-backup`
...@@ -433,19 +433,19 @@ The replication process is now complete. ...@@ -433,19 +433,19 @@ The replication process is now complete.
[PGBouncer](http://pgbouncer.github.io/) may be used with GitLab Geo to pool [PGBouncer](http://pgbouncer.github.io/) may be used with GitLab Geo to pool
PostgreSQL connections. We recommend using PGBouncer if you use GitLab in a PostgreSQL connections. We recommend using PGBouncer if you use GitLab in a
high-availability configuration with a cluster of nodes supporting a Geo high-availability configuration with a cluster of nodes supporting a Geo
primary and another cluster of nodes supporting a Geo secondary. For more **primary** node and another cluster of nodes supporting a Geo **secondary** node. For more
information, see the [Omnibus HA](https://docs.gitlab.com/ee/administration/high_availability/database.html#configure-using-omnibus-for-high-availability) information, see the [Omnibus HA](https://docs.gitlab.com/ee/administration/high_availability/database.html#configure-using-omnibus-for-high-availability)
documentation. documentation.
For a Geo secondary to work properly with PGBouncer in front of the database, For a Geo **secondary** node to work properly with PGBouncer in front of the database,
it will need a separate read-only user to make [PostgreSQL FDW queries][FDW] it will need a separate read-only user to make [PostgreSQL FDW queries][FDW]
work: work:
1. On the primary Geo database, enter the PostgreSQL on the console as an 1. On the **primary** Geo database, enter the PostgreSQL on the console as an
admin user. If you are using an Omnibus-managed database, log onto the primary admin user. If you are using an Omnibus-managed database, log onto the **primary**
node that is running the PostgreSQL database: node that is running the PostgreSQL database:
```bash ```sh
sudo -u gitlab-psql /opt/gitlab/embedded/bin/psql -h /var/opt/gitlab/postgresql gitlabhq_production sudo -u gitlab-psql /opt/gitlab/embedded/bin/psql -h /var/opt/gitlab/postgresql gitlabhq_production
``` ```
...@@ -465,7 +465,7 @@ work: ...@@ -465,7 +465,7 @@ work:
ALTER DEFAULT PRIVILEGES FOR USER gitlab IN SCHEMA public GRANT SELECT ON SEQUENCES TO gitlab_geo_fdw; ALTER DEFAULT PRIVILEGES FOR USER gitlab IN SCHEMA public GRANT SELECT ON SEQUENCES TO gitlab_geo_fdw;
``` ```
1. On the Geo secondary nodes, change `/etc/gitlab/gitlab.rb`: 1. On the **secondary** nodes, change `/etc/gitlab/gitlab.rb`:
``` ```
geo_postgresql['fdw_external_user'] = 'gitlab_geo_fdw' geo_postgresql['fdw_external_user'] = 'gitlab_geo_fdw'
...@@ -473,7 +473,7 @@ work: ...@@ -473,7 +473,7 @@ work:
1. Save the file and reconfigure GitLab for the changes to be applied: 1. Save the file and reconfigure GitLab for the changes to be applied:
```bash ```sh
gitlab-ctl reconfigure gitlab-ctl reconfigure
``` ```
......
...@@ -11,7 +11,7 @@ The stages of the setup process must be completed in the documented order. ...@@ -11,7 +11,7 @@ The stages of the setup process must be completed in the documented order.
Before attempting the steps in this stage, [complete all prior stages][toc]. Before attempting the steps in this stage, [complete all prior stages][toc].
This document describes the minimal steps you have to take in order to This document describes the minimal steps you have to take in order to
replicate your primary GitLab database to a secondary node's database. You may replicate your **primary** GitLab database to a **secondary** node's database. You may
have to change some values according to your database setup, how big it is, etc. have to change some values according to your database setup, how big it is, etc.
You are encouraged to first read through all the steps before executing them You are encouraged to first read through all the steps before executing them
...@@ -19,28 +19,28 @@ in your testing/production environment. ...@@ -19,28 +19,28 @@ in your testing/production environment.
## PostgreSQL replication ## PostgreSQL replication
The GitLab primary node where the write operations happen will connect to The GitLab **primary** node where the write operations happen will connect to
primary database server, and the secondary ones which are read-only will **primary** database server, and the **secondary** ones which are read-only will
connect to secondary database servers (which are read-only too). connect to **secondary** database servers (which are read-only too).
NOTE: **Note:** NOTE: **Note:**
In many databases documentation you will see "primary" being referenced as "master" In many databases' documentation, you will see "**primary**" being referenced as "master"
and "secondary" as either "slave" or "standby" server (read-only). and "**secondary**" as either "slave" or "standby" server (read-only).
We recommend using [PostgreSQL replication slots][replication-slots-article] We recommend using [PostgreSQL replication slots][replication-slots-article]
to ensure the primary retains all the data necessary for the secondaries to to ensure the **primary** node retains all the data necessary for the secondaries to
recover. See below for more details. recover. See below for more details.
The following guide assumes that: The following guide assumes that:
- You are using PostgreSQL 9.6 or later which includes the - You are using PostgreSQL 9.6 or later which includes the
[`pg_basebackup` tool][pgback] and improved [Foreign Data Wrapper][FDW] support. [`pg_basebackup` tool][pgback] and improved [Foreign Data Wrapper][FDW] support.
- You have a primary node already set up (the GitLab server you are - You have a **primary** node already set up (the GitLab server you are
replicating from), running PostgreSQL 9.6 or later, and replicating from), running PostgreSQL 9.6 or later, and
you have a new secondary server set up with the same versions of the OS, you have a new **secondary** server set up with the same versions of the OS,
PostgreSQL, and GitLab on all nodes. PostgreSQL, and GitLab on all nodes.
- The IP of the primary server for our examples will be `1.2.3.4`, whereas the - The IP of the **primary** server for our examples will be `1.2.3.4`, whereas the
secondary's IP will be `5.6.7.8`. Note that the primary and secondary servers **secondary** node's IP will be `5.6.7.8`. Note that the **primary** and **secondary** servers
**must** be able to communicate over these addresses. These IP addresses can either **must** be able to communicate over these addresses. These IP addresses can either
be public or private. be public or private.
...@@ -48,17 +48,17 @@ CAUTION: **Warning:** ...@@ -48,17 +48,17 @@ CAUTION: **Warning:**
Geo works with streaming replication. Logical replication is not supported at this time. Geo works with streaming replication. Logical replication is not supported at this time.
There is an [issue where support is being discussed](https://gitlab.com/gitlab-org/gitlab-ee/issues/7420). There is an [issue where support is being discussed](https://gitlab.com/gitlab-org/gitlab-ee/issues/7420).
### Step 1. Configure the primary server ### Step 1. Configure the **primary** server
1. SSH into your GitLab **primary** server and login as root: 1. SSH into your GitLab **primary** server and login as root:
```bash ```sh
sudo -i sudo -i
``` ```
1. Add this node as the Geo primary by running: 1. Add this node as the Geo **primary** by running:
```bash ```sh
bundle exec rake geo:set_primary_node bundle exec rake geo:set_primary_node
``` ```
...@@ -67,18 +67,18 @@ There is an [issue where support is being discussed](https://gitlab.com/gitlab-o ...@@ -67,18 +67,18 @@ There is an [issue where support is being discussed](https://gitlab.com/gitlab-o
```sql ```sql
--- Create a new user 'replicator' --- Create a new user 'replicator'
CREATE USER gitlab_replicator; CREATE USER gitlab_replicator;
--- Set/change a password and grants replication privilege --- Set/change a password and grants replication privilege
ALTER USER gitlab_replicator WITH REPLICATION ENCRYPTED PASSWORD 'replicationpasswordhere'; ALTER USER gitlab_replicator WITH REPLICATION ENCRYPTED PASSWORD 'replicationpasswordhere';
``` ```
1. Make sure your the `gitlab` database user has a password defined
```bash 1. Make sure your the `gitlab` database user has a password defined:
```sh
sudo -u postgres psql -d template1 -c "ALTER USER gitlab WITH ENCRYPTED PASSWORD 'mydatabasepassword';" sudo -u postgres psql -d template1 -c "ALTER USER gitlab WITH ENCRYPTED PASSWORD 'mydatabasepassword';"
``` ```
1. Edit the content of `database.yml` in `production:` and add the password like the exemple below: 1. Edit the content of `database.yml` in `production:` and add the password like the example below:
```yaml ```yaml
# #
...@@ -94,11 +94,11 @@ There is an [issue where support is being discussed](https://gitlab.com/gitlab-o ...@@ -94,11 +94,11 @@ There is an [issue where support is being discussed](https://gitlab.com/gitlab-o
host: /var/opt/gitlab/geo-postgresql host: /var/opt/gitlab/geo-postgresql
``` ```
1. Set up TLS support for the PostgreSQL primary server 1. Set up TLS support for the PostgreSQL **primary** server:
CAUTION: **Warning**: CAUTION: **Warning**:
Only skip this step if you **know** that PostgreSQL traffic Only skip this step if you **know** that PostgreSQL traffic
between the primary and secondary will be secured through some other between the **primary** and **secondary** nodes will be secured through some other
means, e.g., a known-safe physical network path or a site-to-site VPN that means, e.g., a known-safe physical network path or a site-to-site VPN that
you have configured. you have configured.
...@@ -109,7 +109,7 @@ There is an [issue where support is being discussed](https://gitlab.com/gitlab-o ...@@ -109,7 +109,7 @@ There is an [issue where support is being discussed](https://gitlab.com/gitlab-o
To generate a self-signed certificate and key, run this command: To generate a self-signed certificate and key, run this command:
```bash ```sh
openssl req -nodes -batch -x509 -newkey rsa:4096 -keyout server.key -out server.crt -days 3650 openssl req -nodes -batch -x509 -newkey rsa:4096 -keyout server.key -out server.crt -days 3650
``` ```
...@@ -118,7 +118,7 @@ There is an [issue where support is being discussed](https://gitlab.com/gitlab-o ...@@ -118,7 +118,7 @@ There is an [issue where support is being discussed](https://gitlab.com/gitlab-o
Copy them to the correct location for your PostgreSQL installation: Copy them to the correct location for your PostgreSQL installation:
```bash ```sh
# Copying a self-signed certificate and key # Copying a self-signed certificate and key
install -o postgres -g postgres -m 0400 -T server.crt ~postgres/9.x/main/data/server.crt install -o postgres -g postgres -m 0400 -T server.crt ~postgres/9.x/main/data/server.crt
install -o postgres -g postgres -m 0400 -T server.key ~postgres/9.x/main/data/server.key install -o postgres -g postgres -m 0400 -T server.key ~postgres/9.x/main/data/server.key
...@@ -133,7 +133,7 @@ There is an [issue where support is being discussed](https://gitlab.com/gitlab-o ...@@ -133,7 +133,7 @@ There is an [issue where support is being discussed](https://gitlab.com/gitlab-o
ssl_key_file='server.key' ssl_key_file='server.key'
``` ```
1. Edit `postgresql.conf` to configure the primary server for streaming replication 1. Edit `postgresql.conf` to configure the **primary** server for streaming replication
(for Debian/Ubuntu that would be `/etc/postgresql/9.x/main/postgresql.conf`): (for Debian/Ubuntu that would be `/etc/postgresql/9.x/main/postgresql.conf`):
``` ```
...@@ -148,12 +148,12 @@ There is an [issue where support is being discussed](https://gitlab.com/gitlab-o ...@@ -148,12 +148,12 @@ There is an [issue where support is being discussed](https://gitlab.com/gitlab-o
``` ```
NOTE: **Note**: NOTE: **Note**:
Be sure to set `max_replication_slots` to the number of Geo secondary Be sure to set `max_replication_slots` to the number of Geo **secondary**
nodes that you may potentially have (at least 1). nodes that you may potentially have (at least 1).
For security reasons, PostgreSQL by default only listens on the local For security reasons, PostgreSQL by default only listens on the local
interface (e.g. 127.0.0.1). However, Geo needs to communicate interface (e.g. 127.0.0.1). However, Geo needs to communicate
between the primary and secondary nodes over a common network, such as a between the **primary** and **secondary** nodes over a common network, such as a
corporate LAN or the public Internet. For this reason, we need to corporate LAN or the public Internet. For this reason, we need to
configure PostgreSQL to listen on more interfaces. configure PostgreSQL to listen on more interfaces.
...@@ -162,25 +162,25 @@ There is an [issue where support is being discussed](https://gitlab.com/gitlab-o ...@@ -162,25 +162,25 @@ There is an [issue where support is being discussed](https://gitlab.com/gitlab-o
documentation][pg-docs-runtime-conn] for more details. documentation][pg-docs-runtime-conn] for more details.
You may also want to edit the `wal_keep_segments` and `max_wal_senders` to You may also want to edit the `wal_keep_segments` and `max_wal_senders` to
match your database replication requirements. Consult the match your database replication requirements. Consult the
[PostgreSQL - Replication documentation][pg-docs-runtime-replication] for more information. [PostgreSQL - Replication documentation][pg-docs-runtime-replication] for more information.
1. Set the access control on the primary to allow TCP connections using the 1. Set the access control on the **primary** node to allow TCP connections using the
server's public IP and set the connection from the secondary to require a server's public IP and set the connection from the **secondary** node to require a
password. Edit `pg_hba.conf` (for Debian/Ubuntu that would be password. Edit `pg_hba.conf` (for Debian/Ubuntu that would be
`/etc/postgresql/9.x/main/pg_hba.conf`): `/etc/postgresql/9.x/main/pg_hba.conf`):
```bash ```sh
host all all 1.2.3.4/32 md5 host all all 1.2.3.4/32 md5
host replication gitlab_replicator 5.6.7.8/32 md5 host replication gitlab_replicator 5.6.7.8/32 md5
``` ```
Where `1.2.3.4` is the public IP address of the primary server, and `5.6.7.8` Where `1.2.3.4` is the public IP address of the **primary** server, and `5.6.7.8`
the public IP address of the secondary one. If you want to add another the public IP address of the **secondary** one. If you want to add another
secondary, add one more row like the replication one and change the IP secondary, add one more row like the replication one and change the IP
address: address:
```bash ```sh
host all all 1.2.3.4/32 md5 host all all 1.2.3.4/32 md5
host replication gitlab_replicator 5.6.7.8/32 md5 host replication gitlab_replicator 5.6.7.8/32 md5
host replication gitlab_replicator 11.22.33.44/32 md5 host replication gitlab_replicator 11.22.33.44/32 md5
...@@ -193,9 +193,9 @@ There is an [issue where support is being discussed](https://gitlab.com/gitlab-o ...@@ -193,9 +193,9 @@ There is an [issue where support is being discussed](https://gitlab.com/gitlab-o
`secondary.geo.example.com`, you may use `secondary_example` as the slot `secondary.geo.example.com`, you may use `secondary_example` as the slot
name. name.
1. Create the replication slot on the primary: 1. Create the replication slot on the **primary** node:
```bash ```sh
$ sudo -u postgres psql -c "SELECT * FROM pg_create_physical_replication_slot('secondary_example');" $ sudo -u postgres psql -c "SELECT * FROM pg_create_physical_replication_slot('secondary_example');"
slot_name | xlog_position slot_name | xlog_position
------------------+--------------- ------------------+---------------
...@@ -209,16 +209,15 @@ There is an [issue where support is being discussed](https://gitlab.com/gitlab-o ...@@ -209,16 +209,15 @@ There is an [issue where support is being discussed](https://gitlab.com/gitlab-o
### Step 2. Configure the secondary server ### Step 2. Configure the secondary server
Follow the first steps in ["configure the secondary server"][database-replication], Follow the first steps in ["configure the secondary server"][database-replication] and note that since you are installing from source, the username and
but note that since you are installing from source, the username and
group listed as `gitlab-psql` in those steps should be replaced by `postgres` group listed as `gitlab-psql` in those steps should be replaced by `postgres`
instead. After completing the "Test that the `gitlab-psql` user can connect to instead. After completing the "Test that the `gitlab-psql` user can connect to
the primary's database" step, continue here: the **primary** node's database" step, continue here:
1. Edit `postgresql.conf` to configure the secondary for streaming replication 1. Edit `postgresql.conf` to configure the secondary for streaming replication
(for Debian/Ubuntu that would be `/etc/postgresql/9.*/main/postgresql.conf`): (for Debian/Ubuntu that would be `/etc/postgresql/9.*/main/postgresql.conf`):
```bash ```sh
wal_level = hot_standby wal_level = hot_standby
max_wal_senders = 5 max_wal_senders = 5
checkpoint_segments = 10 checkpoint_segments = 10
...@@ -235,9 +234,9 @@ and recover automatically from some replication issues. Follow the steps below t ...@@ -235,9 +234,9 @@ and recover automatically from some replication issues. Follow the steps below t
the tracking database. the tracking database.
1. On the secondary node, run the following command to create `database_geo.yml` with the 1. On the secondary node, run the following command to create `database_geo.yml` with the
information of your secondary PostgreSQL instance: information of your secondary PostgreSQL instance:
```bash ```sh
sudo cp /home/git/gitlab/config/database_geo.yml.postgresql /home/git/gitlab/config/database_geo.yml sudo cp /home/git/gitlab/config/database_geo.yml.postgresql /home/git/gitlab/config/database_geo.yml
``` ```
...@@ -257,12 +256,11 @@ information of your secondary PostgreSQL instance: ...@@ -257,12 +256,11 @@ information of your secondary PostgreSQL instance:
host: /var/opt/gitlab/geo-postgresql host: /var/opt/gitlab/geo-postgresql
``` ```
1. Create the database `gitlabhq_geo_production` on the PostgreSQL instance of the secondary 1. Create the database `gitlabhq_geo_production` on the PostgreSQL instance of the **secondary** node.
node.
1. Set up the Geo tracking database: 1. Set up the Geo tracking database:
```bash ```sh
bundle exec rake geo:db:migrate bundle exec rake geo:db:migrate
``` ```
...@@ -270,26 +268,26 @@ node. ...@@ -270,26 +268,26 @@ node.
Save the script below in a file, ex. `/tmp/geo_fdw.sh` and modify the connection Save the script below in a file, ex. `/tmp/geo_fdw.sh` and modify the connection
params to match your environment. Execute it to set up the FDW connection. params to match your environment. Execute it to set up the FDW connection.
```bash ```sh
#!/bin/bash #!/bin/bash
# Secondary Database connection params: # Secondary Database connection params:
DB_HOST="/var/opt/gitlab/postgresql" # change to the public IP or VPC private IP if its an external server DB_HOST="/var/opt/gitlab/postgresql" # change to the public IP or VPC private IP if its an external server
DB_NAME="gitlabhq_production" DB_NAME="gitlabhq_production"
DB_USER="gitlab" DB_USER="gitlab"
DB_PORT="5432" DB_PORT="5432"
# Tracking Database connection params: # Tracking Database connection params:
GEO_DB_HOST="/var/opt/gitlab/geo-postgresql" # change to the public IP or VPC private IP if its an external server GEO_DB_HOST="/var/opt/gitlab/geo-postgresql" # change to the public IP or VPC private IP if its an external server
GEO_DB_NAME="gitlabhq_geo_production" GEO_DB_NAME="gitlabhq_geo_production"
GEO_DB_USER="gitlab_geo" GEO_DB_USER="gitlab_geo"
GEO_DB_PORT="5432" GEO_DB_PORT="5432"
query_exec () { query_exec () {
gitlab-psql -h $GEO_DB_HOST -d $GEO_DB_NAME -p $GEO_DB_PORT -c "${1}" gitlab-psql -h $GEO_DB_HOST -d $GEO_DB_NAME -p $GEO_DB_PORT -c "${1}"
} }
query_exec "CREATE EXTENSION postgres_fdw;" query_exec "CREATE EXTENSION postgres_fdw;"
query_exec "CREATE SERVER gitlab_secondary FOREIGN DATA WRAPPER postgres_fdw OPTIONS (host '${DB_HOST}', dbname '${DB_NAME}', port '${DB_PORT}');" query_exec "CREATE SERVER gitlab_secondary FOREIGN DATA WRAPPER postgres_fdw OPTIONS (host '${DB_HOST}', dbname '${DB_NAME}', port '${DB_PORT}');"
query_exec "CREATE USER MAPPING FOR ${GEO_DB_USER} SERVER gitlab_secondary OPTIONS (user '${DB_USER}');" query_exec "CREATE USER MAPPING FOR ${GEO_DB_USER} SERVER gitlab_secondary OPTIONS (user '${DB_USER}');"
...@@ -302,8 +300,8 @@ node. ...@@ -302,8 +300,8 @@ node.
### Step 3. Initiate the replication process ### Step 3. Initiate the replication process
Below we provide a script that connects the database on the secondary node to Below we provide a script that connects the database on the **secondary** node to
the database on the primary node, replicates the database, and creates the the database on the **primary** node, replicates the database, and creates the
needed files for streaming replication. needed files for streaming replication.
The directories used are the defaults for Debian/Ubuntu. If you have changed The directories used are the defaults for Debian/Ubuntu. If you have changed
...@@ -315,7 +313,7 @@ data before running `pg_basebackup`. ...@@ -315,7 +313,7 @@ data before running `pg_basebackup`.
1. SSH into your GitLab **secondary** server and login as root: 1. SSH into your GitLab **secondary** server and login as root:
```bash ```sh
sudo -i sudo -i
``` ```
...@@ -368,11 +366,11 @@ data before running `pg_basebackup`. ...@@ -368,11 +366,11 @@ data before running `pg_basebackup`.
1. Run it with: 1. Run it with:
```bash ```sh
bash /tmp/replica.sh bash /tmp/replica.sh
``` ```
When prompted, enter the IP/FQDN of the primary, and the password you set up When prompted, enter the IP/FQDN of the **primary** node, and the password you set up
for the `gitlab_replicator` user in the first step. for the `gitlab_replicator` user in the first step.
You should use `verify-ca` for the `sslmode`. You can use `disable` if you You should use `verify-ca` for the `sslmode`. You can use `disable` if you
...@@ -391,7 +389,7 @@ The replication process is now over. ...@@ -391,7 +389,7 @@ The replication process is now over.
1. First, enter the PostgreSQL console as an admin user. 1. First, enter the PostgreSQL console as an admin user.
2. Then create the read-only user: 1. Then create the read-only user:
```sql ```sql
-- NOTE: Use the password defined earlier -- NOTE: Use the password defined earlier
...@@ -407,7 +405,7 @@ The replication process is now over. ...@@ -407,7 +405,7 @@ The replication process is now over.
ALTER DEFAULT PRIVILEGES FOR USER gitlab IN SCHEMA public GRANT SELECT ON SEQUENCES TO gitlab_geo_fdw; ALTER DEFAULT PRIVILEGES FOR USER gitlab IN SCHEMA public GRANT SELECT ON SEQUENCES TO gitlab_geo_fdw;
``` ```
3. Enter the PostgreSQL console on the secondary tracking database and change the user mapping to this new user: 1. Enter the PostgreSQL console on the **secondary** tracking database and change the user mapping to this new user:
``` ```
ALTER USER MAPPING FOR gitlab_geo SERVER gitlab_secondary OPTIONS (SET user 'gitlab_geo_fdw') ALTER USER MAPPING FOR gitlab_geo SERVER gitlab_secondary OPTIONS (SET user 'gitlab_geo_fdw')
......
# Docker Registry for a secondary node # Docker Registry for a secondary node
You can set up a [Docker Registry] on your You can set up a [Docker Registry] on your
secondary Geo node that mirrors the one on the primary Geo node. **secondary** Geo node that mirrors the one on the **primary** Geo node.
## Storage support ## Storage support
CAUTION: **Warning:** CAUTION: **Warning:**
If you use [local storage][registry-storage] If you use [local storage][registry-storage]
for the Container Registry you **cannot** replicate it to the secondary Geo node. for the Container Registry you **cannot** replicate it to a **secondary** node.
Docker Registry currently supports a few types of storages. If you choose a Docker Registry currently supports a few types of storages. If you choose a
distributed storage (`azure`, `gcs`, `s3`, `swift`, or `oss`) for your Docker distributed storage (`azure`, `gcs`, `s3`, `swift`, or `oss`) for your Docker
Registry on a primary Geo node, you can use the same storage for a secondary Registry on the **primary** node, you can use the same storage for a **secondary**
Docker Registry as well. For more information, read the Docker Registry as well. For more information, read the
[Load balancing considerations][registry-load-balancing] [Load balancing considerations][registry-load-balancing]
when deploying the Registry, and how to set up the storage driver for GitLab's when deploying the Registry, and how to set up the storage driver for GitLab's
......
...@@ -7,22 +7,22 @@ The requirements are listed [on the index page](index.md#requirements-for-runnin ...@@ -7,22 +7,22 @@ The requirements are listed [on the index page](index.md#requirements-for-runnin
## Can I use Geo in a disaster recovery situation? ## Can I use Geo in a disaster recovery situation?
Yes, but there are limitations to what we replicate (see Yes, but there are limitations to what we replicate (see
[What data is replicated to a secondary node?](#what-data-is-replicated-to-a-secondary-node)). [What data is replicated to a **secondary** node?](#what-data-is-replicated-to-a-secondary-node)).
Read the documentation for [Disaster Recovery](../disaster_recovery/index.md). Read the documentation for [Disaster Recovery](../disaster_recovery/index.md).
## What data is replicated to a secondary node? ## What data is replicated to a **secondary** node?
We currently replicate project repositories, LFS objects, generated We currently replicate project repositories, LFS objects, generated
attachments / avatars and the whole database. This means user accounts, attachments / avatars and the whole database. This means user accounts,
issues, merge requests, groups, project data, etc., will be available for issues, merge requests, groups, project data, etc., will be available for
query. query.
## Can I git push to a secondary node? ## Can I git push to a **secondary** node?
Yes! Pushing directly to a **secondary** node (for both HTTP and SSH, including git-lfs) was [introduced](https://about.gitlab.com/2018/09/22/gitlab-11-3-released/) in [GitLab Premium](https://about.gitlab.com/pricing/#self-managed) 11.3. Yes! Pushing directly to a **secondary** node (for both HTTP and SSH, including git-lfs) was [introduced](https://about.gitlab.com/2018/09/22/gitlab-11-3-released/) in [GitLab Premium](https://about.gitlab.com/pricing/#self-managed) 11.3.
## How long does it take to have a commit replicated to a secondary node? ## How long does it take to have a commit replicated to a **secondary** node?
All replication operations are asynchronous and are queued to be dispatched. Therefore, it depends on a lot of All replication operations are asynchronous and are queued to be dispatched. Therefore, it depends on a lot of
factors including the amount of traffic, how big your commit is, the factors including the amount of traffic, how big your commit is, the
...@@ -30,8 +30,8 @@ connectivity between your nodes, your hardware, etc. ...@@ -30,8 +30,8 @@ connectivity between your nodes, your hardware, etc.
## What if the SSH server runs at a different port? ## What if the SSH server runs at a different port?
That's totally fine. We use HTTP(s) to fetch repository changes from the **primary** node to **secondary** nodes. That's totally fine. We use HTTP(s) to fetch repository changes from the **primary** node to all **secondary** nodes.
## Is this possible to set up a Docker Registry for a secondary node that mirrors the one on a primary node? ## Is this possible to set up a Docker Registry for a **secondary** node that mirrors the one on the **primary** node?
Yes. See [Docker Registry for a secondary Geo node](docker_registry.md). Yes. See [Docker Registry for a **secondary** node](docker_registry.md).
...@@ -10,7 +10,7 @@ described, it is possible to adapt these instructions to your needs. ...@@ -10,7 +10,7 @@ described, it is possible to adapt these instructions to your needs.
_[diagram source - gitlab employees only][diagram-source]_ _[diagram source - gitlab employees only][diagram-source]_
The topology above assumes that the primary and secondary Geo clusters The topology above assumes that the **primary** and **secondary** Geo clusters
are located in two separate locations, on their own virtual network are located in two separate locations, on their own virtual network
with private IP addresses. The network is configured such that all machines within with private IP addresses. The network is configured such that all machines within
one geographic location can communicate with each other using their private IP addresses. one geographic location can communicate with each other using their private IP addresses.
...@@ -20,14 +20,14 @@ network topology of your deployment. ...@@ -20,14 +20,14 @@ network topology of your deployment.
The only external way to access the two Geo deployments is by HTTPS at The only external way to access the two Geo deployments is by HTTPS at
`gitlab.us.example.com` and `gitlab.eu.example.com` in the example above. `gitlab.us.example.com` and `gitlab.eu.example.com` in the example above.
> **Note:** The primary and secondary Geo deployments must be able to NOTE: **Note:**
communicate to each other over HTTPS. The **primary** and **secondary** Geo deployments must be able to communicate to each other over HTTPS.
## Redis and PostgreSQL High Availability ## Redis and PostgreSQL High Availability
The primary and secondary Redis and PostgreSQL should be configured The **primary** and **secondary** Redis and PostgreSQL should be configured
for high availability. Because of the additional complexity involved for high availability. Because of the additional complexity involved
in setting up this configuration for PostgreSQL and Redis in setting up this configuration for PostgreSQL and Redis,
it is not covered by this Geo HA documentation. it is not covered by this Geo HA documentation.
For more information about setting up a highly available PostgreSQL cluster and Redis cluster using the omnibus package see the high availability documentation for For more information about setting up a highly available PostgreSQL cluster and Redis cluster using the omnibus package see the high availability documentation for
...@@ -35,7 +35,7 @@ For more information about setting up a highly available PostgreSQL cluster and ...@@ -35,7 +35,7 @@ For more information about setting up a highly available PostgreSQL cluster and
[Redis](../../high_availability/redis.md), respectively. [Redis](../../high_availability/redis.md), respectively.
NOTE: **Note:** NOTE: **Note:**
It is possible to use cloud hosted services for PostgreSQL and Redis but this is beyond the scope of this document. It is possible to use cloud hosted services for PostgreSQL and Redis, but this is beyond the scope of this document.
## Prerequisites: A working GitLab HA cluster ## Prerequisites: A working GitLab HA cluster
...@@ -189,7 +189,6 @@ following modifications: ...@@ -189,7 +189,6 @@ following modifications:
registry['uid'] = 9002 registry['uid'] = 9002
registry['gid'] = 9002 registry['gid'] = 9002
``` ```
NOTE: **Note:** NOTE: **Note:**
If you had set up PostgreSQL cluster using the omnibus package and you had set If you had set up PostgreSQL cluster using the omnibus package and you had set
up `postgresql['sql_user_password'] = 'md5 digest of secret'` setting, keep in up `postgresql['sql_user_password'] = 'md5 digest of secret'` setting, keep in
......
...@@ -30,7 +30,7 @@ Implementing Geo provides the following benefits: ...@@ -30,7 +30,7 @@ Implementing Geo provides the following benefits:
- Reduce from minutes to seconds the time taken for your distributed developers to clone and fetch large repositories and projects. - Reduce from minutes to seconds the time taken for your distributed developers to clone and fetch large repositories and projects.
- Enable all of your developers to contribute ideas and work in parallel, no matter where they are. - Enable all of your developers to contribute ideas and work in parallel, no matter where they are.
- Balance the load between your **primary** and **secondary** nodes, or offload your automated tests to the **secondary** node. - Balance the load between your **primary** and **secondary** nodes, or offload your automated tests to a **secondary** node.
In addition, it: In addition, it:
...@@ -42,8 +42,8 @@ In addition, it: ...@@ -42,8 +42,8 @@ In addition, it:
Geo provides: Geo provides:
- Read-only **secondary** nodes: Maintain one **primary** GitLab node while still enabling a read-only **secondary** node for each of your distributed teams. - Read-only **secondary** nodes: Maintain one **primary** GitLab node while still enabling read-only **secondary** nodes for each of your distributed teams.
- Authentication system hooks: The **secondary** node receives all authentication data (like user accounts and logins) from the **primary** instance. - Authentication system hooks: **Secondary** nodes receives all authentication data (like user accounts and logins) from the **primary** instance.
- An intuitive UI: **Secondary** nodes utilize the same web interface your team has grown accustomed to. In addition, there are visual notifications that block write operations and make it clear that a user is on a **secondary** node. - An intuitive UI: **Secondary** nodes utilize the same web interface your team has grown accustomed to. In addition, there are visual notifications that block write operations and make it clear that a user is on a **secondary** node.
## How it works ## How it works
...@@ -75,12 +75,12 @@ The following diagram illustrates the underlying architecture of Geo. ...@@ -75,12 +75,12 @@ The following diagram illustrates the underlying architecture of Geo.
In this diagram: In this diagram:
- There is the **primary** node and the details of one **secondary** node. - There is the **primary** node and the details of one **secondary** node.
- Writes to the database can only be performed on the **primary** node. A **secondary** node receives database - Writes to the database can only be performed on the **primary** node. A **secondary** node receives database
updates via PostgreSQL streaming replication. updates via PostgreSQL streaming replication.
- If present, the [LDAP server](#ldap) should be configured to replicate for [Disaster Recovery](../disaster_recovery/index.md) scenarios. - If present, the [LDAP server](#ldap) should be configured to replicate for [Disaster Recovery](../disaster_recovery/index.md) scenarios.
- A **secondary** node performs different type of synchronizations against the **primary** node, using a special - A **secondary** node performs different type of synchronizations against the **primary** node, using a special
authorization protected by JWT: authorization protected by JWT:
- Repositories are cloned/updated via Git over HTTPS. - Repositories are cloned/updated via Git over HTTPS.
- Attachments, LFS objects, and other files are downloaded via HTTPS using a private API endpoint. - Attachments, LFS objects, and other files are downloaded via HTTPS using a private API endpoint.
From the perspective of a user performing Git operations: From the perspective of a user performing Git operations:
...@@ -89,6 +89,7 @@ From the perspective of a user performing Git operations: ...@@ -89,6 +89,7 @@ From the perspective of a user performing Git operations:
- **Secondary** nodes are read-only but proxy Git push operations to the **primary** node. This makes **secondary** nodes appear to support push operations themselves. - **Secondary** nodes are read-only but proxy Git push operations to the **primary** node. This makes **secondary** nodes appear to support push operations themselves.
To simplify the diagram, some necessary components are omitted. Note that: To simplify the diagram, some necessary components are omitted. Note that:
- Git over SSH requires [`gitlab-shell`](https://gitlab.com/gitlab-org/gitlab-shell) and OpenSSH. - Git over SSH requires [`gitlab-shell`](https://gitlab.com/gitlab-org/gitlab-shell) and OpenSSH.
- Git over HTTPS required [`gitlab-workhorse`](https://gitlab.com/gitlab-org/gitlab-workhorse). - Git over HTTPS required [`gitlab-workhorse`](https://gitlab.com/gitlab-org/gitlab-workhorse).
...@@ -97,7 +98,7 @@ Note that a **secondary** node needs two different PostgreSQL databases: ...@@ -97,7 +98,7 @@ Note that a **secondary** node needs two different PostgreSQL databases:
- A read-only database instance that streams data from the main GitLab database. - A read-only database instance that streams data from the main GitLab database.
- [Another database instance](#geo-tracking-database) used internally by the **secondary** node to record what data has been replicated. - [Another database instance](#geo-tracking-database) used internally by the **secondary** node to record what data has been replicated.
In secondary nodes, there is an additional daemon: [Geo Log Cursor](#geo-log-cursor). In **secondary** nodes, there is an additional daemon: [Geo Log Cursor](#geo-log-cursor).
## Requirements for running Geo ## Requirements for running Geo
...@@ -115,12 +116,12 @@ The following are required to run Geo: ...@@ -115,12 +116,12 @@ The following are required to run Geo:
The following table lists basic ports that must be open between the **primary** and **secondary** nodes for Geo. The following table lists basic ports that must be open between the **primary** and **secondary** nodes for Geo.
| Primary server | Server secondary | Protocol | | **Primary** node | **Secondary** node | Protocol |
| -------------- | ---------------- | --------------- | |:-----------------|:-------------------|:-------------|
| 80 | 80 | HTTP | | 80 | 80 | HTTP |
| 443 | 443 | TCP or HTTPS | | 443 | 443 | TCP or HTTPS |
| 22 | 22 | TCP | | 22 | 22 | TCP |
| 5432 | | PostgreSQL | | 5432 | | PostgreSQL |
See the full list of ports used by GitLab in [Package defaults](https://docs.gitlab.com/omnibus/package-information/defaults.html) See the full list of ports used by GitLab in [Package defaults](https://docs.gitlab.com/omnibus/package-information/defaults.html)
...@@ -134,9 +135,12 @@ If you wish to terminate SSL at the GitLab application server instead, use TCP p ...@@ -134,9 +135,12 @@ If you wish to terminate SSL at the GitLab application server instead, use TCP p
### LDAP ### LDAP
We recommend that if you use LDAP on your **primary** node, you also set up a secondary LDAP server for the **secondary** node. Otherwise, users will not be able to perform Git operations over HTTP(s) on the **secondary** node using HTTP Basic Authentication. However, Git via SSH and personal access tokens will still work. We recommend that if you use LDAP on your **primary** node, you also set up secondary LDAP servers on each **secondary** node. Otherwise, users will not be able to perform Git operations over HTTP(s) on the **secondary** node using HTTP Basic Authentication. However, Git via SSH and personal access tokens will still work.
NOTE: **Note:**
It is possible for all **secondary** nodes to share an LDAP server, but additional latency can be an issue. Also, consider what LDAP server will be available in a [disaster recovery](../disaster_recovery/index.md) scenario if a **secondary** node is promoted to be a **primary** node.
Check with your LDAP provider for instructions on how to set up replication. For example, OpenLDAP provides [these instructions](https://www.openldap.org/doc/admin24/replication.html). Check for instructions on how to set up replication in your LDAP service. Instructions will be different depending on the software or service used. For example, OpenLDAP provides [these instructions](https://www.openldap.org/doc/admin24/replication.html).
### Geo Tracking Database ### Geo Tracking Database
...@@ -153,7 +157,7 @@ The tracking database requires the `postgres_fdw` extension. ...@@ -153,7 +157,7 @@ The tracking database requires the `postgres_fdw` extension.
This daemon: This daemon:
- Reads a log of events replicated by the **primary** node to the secondary database instance. - Reads a log of events replicated by the **primary** node to the **secondary** database instance.
- Updates the Geo Tracking Database instance with changes that need to be executed. - Updates the Geo Tracking Database instance with changes that need to be executed.
When something is marked to be updated in the tracking database instance, asynchronous jobs running on the **secondary** node will execute the required operations and update the state. When something is marked to be updated in the tracking database instance, asynchronous jobs running on the **secondary** node will execute the required operations and update the state.
......
...@@ -6,9 +6,9 @@ other compatible object storage). ...@@ -6,9 +6,9 @@ other compatible object storage).
## Configuration ## Configuration
At this time it is required that if object storage is enabled on the At this time it is required that if object storage is enabled on the
primary, it must also be enabled on the secondary. **primary** node, it must also be enabled on each **secondary** node.
The secondary nodes can use the same storage bucket as the primary, or **Secondary** nodes can use the same storage bucket as the **primary** node, or
they can use a replicated storage bucket. At this time GitLab does not they can use a replicated storage bucket. At this time GitLab does not
take care of content replication in object storage. take care of content replication in object storage.
...@@ -22,15 +22,15 @@ For user uploads, there is similar documentation to configure [upload object sto ...@@ -22,15 +22,15 @@ For user uploads, there is similar documentation to configure [upload object sto
You should enable and configure object storage on both **primary** and **secondary** You should enable and configure object storage on both **primary** and **secondary**
nodes. Migrating existing data to object storage should be performed on the nodes. Migrating existing data to object storage should be performed on the
**primary** node only; secondaries will automatically notice that the migrated **primary** node only. **Secondary** nodes will automatically notice that the migrated
files are now in object storage. files are now in object storage.
## Replication ## Replication
When using Amazon S3, you can use When using Amazon S3, you can use
[CRR](https://docs.aws.amazon.com/AmazonS3/latest/dev/crr.html) to [CRR](https://docs.aws.amazon.com/AmazonS3/latest/dev/crr.html) to
have automatic replication between the bucket used by the primary and have automatic replication between the bucket used by the **primary** node and
the bucket used by the secondary. the bucket used by **secondary** nodes.
If you are using Google Cloud Storage, consider using If you are using Google Cloud Storage, consider using
[Multi-Regional Storage](https://cloud.google.com/storage/docs/storage-classes#multi-regional). [Multi-Regional Storage](https://cloud.google.com/storage/docs/storage-classes#multi-regional).
......
...@@ -10,7 +10,7 @@ questions from [owasp.org](https://www.owasp.org). ...@@ -10,7 +10,7 @@ questions from [owasp.org](https://www.owasp.org).
### What geographic areas does the application service? ### What geographic areas does the application service?
- This varies by customer. Geo allows customers to deploy to multiple areas, - This varies by customer. Geo allows customers to deploy to multiple areas,
and they get to choose where they are. and they get to choose where they are.
- Region and node selection is entirely manual. - Region and node selection is entirely manual.
## Data Essentials ## Data Essentials
...@@ -27,70 +27,70 @@ questions from [owasp.org](https://www.owasp.org). ...@@ -27,70 +27,70 @@ questions from [owasp.org](https://www.owasp.org).
### How can the data be classified into categories according to its sensitivity? ### How can the data be classified into categories according to its sensitivity?
- GitLab’s model of sensitivity is centered around public vs. internal vs. - GitLab’s model of sensitivity is centered around public vs. internal vs.
private projects. Geo replicates them all indiscriminately. “Selective sync” private projects. Geo replicates them all indiscriminately. “Selective sync”
exists for files and repositories (but not database content), which would permit exists for files and repositories (but not database content), which would permit
only less-sensitive projects to be replicated to a secondary if desired. only less-sensitive projects to be replicated to a **secondary** node if desired.
- See also: [developing a data classification policy](https://gitlab.com/gitlab-com/security/issues/4). - See also: [developing a data classification policy](https://gitlab.com/gitlab-com/security/issues/4).
### What data backup and retention requirements have been defined for the application? ### What data backup and retention requirements have been defined for the application?
- Geo is designed to provide replication of a certain subset of the application - Geo is designed to provide replication of a certain subset of the application
data. It is part of the solution, rather than part of the problem. data. It is part of the solution, rather than part of the problem.
## End-Users ## End-Users
### Who are the application's end‐users? ### Who are the application's end‐users?
- Geo nodes (secondaries) are created in regions that are distant (in terms of - **Secondary** nodes are created in regions that are distant (in terms of
Internet latency) from the main GitLab installation (the primary). They are Internet latency) from the main GitLab installation (the **primary** node). They are
intended to be used by anyone who would ordinarily use the primary, who finds intended to be used by anyone who would ordinarily use the **primary** node, who finds
that the secondary is closer to them (in terms of Internet latency). that the **secondary** node is closer to them (in terms of Internet latency).
### How do the end‐users interact with the application? ### How do the end‐users interact with the application?
- A Geo secondary node provides all the interfaces a Geo primary node does - **Secondary** nodes provide all the interfaces a **primary** node does
(notably a HTTP/HTTPS web application, and HTTP/HTTPS or SSH git repository (notably a HTTP/HTTPS web application, and HTTP/HTTPS or SSH git repository
access), but is constrained to read-only activities. The principal use case is access), but is constrained to read-only activities. The principal use case is
envisioned to be cloning git repositories from the secondary in favor of the envisioned to be cloning git repositories from the **secondary** node in favor of the
primary, but end-users may use the GitLab web interface to view projects, **primary** node, but end-users may use the GitLab web interface to view projects,
issues, merge requests, snippets, etc. issues, merge requests, snippets, etc.
### What security expectations do the end‐users have? ### What security expectations do the end‐users have?
- The replication process must be secure. It would typically be unacceptable to - The replication process must be secure. It would typically be unacceptable to
transmit the entire database contents or all files and repositories across the transmit the entire database contents or all files and repositories across the
public Internet in plaintext, for instance. public Internet in plaintext, for instance.
- The Geo secondary must have the same access controls over its content as the - **Secondary** nodes must have the same access controls over its content as the
primary - unauthenticated users must not be able to gain access to privileged **primary** node - unauthenticated users must not be able to gain access to privileged
information on the primary by querying the secondary. information on the **primary** node by querying the **secondary** node.
- Attackers must not be able to impersonate the secondary to the primary, and - Attackers must not be able to impersonate the **secondary** node to the **primary** node, and
thus gain access to privileged information. thus gain access to privileged information.
## Administrators ## Administrators
### Who has administrative capabilities in the application? ### Who has administrative capabilities in the application?
- Nothing Geo-specific. Any user where `admin: true` is set in the database is - Nothing Geo-specific. Any user where `admin: true` is set in the database is
considered an admin with super-user privileges. considered an admin with super-user privileges.
- See also: [more granular access control](https://gitlab.com/gitlab-org/gitlab-ce/issues/32730) - See also: [more granular access control](https://gitlab.com/gitlab-org/gitlab-ce/issues/32730)
(not geo-specific) (not geo-specific)
- Much of Geo’s integration (database replication, for instance) must be - Much of Geo’s integration (database replication, for instance) must be
configured with the application, typically by system administrators. configured with the application, typically by system administrators.
### What administrative capabilities does the application offer? ### What administrative capabilities does the application offer?
- Geo secondaries may be added, modified, or removed by users with - **Secondary** nodes may be added, modified, or removed by users with
administrative access. administrative access.
- The replication process may be controlled (start/stop) via the Sidekiq - The replication process may be controlled (start/stop) via the Sidekiq
administrative controls. administrative controls.
## Network ## Network
### What details regarding routing, switching, firewalling, and load‐balancing have been defined? ### What details regarding routing, switching, firewalling, and load‐balancing have been defined?
- Geo requires the primary and secondary to be able to communicate with each - Geo requires the **primary** node and **secondary** node to be able to communicate with each
other across a TCP/IP network. In particular, the secondaries must be able to other across a TCP/IP network. In particular, the **secondary** nodes must be able to
access HTTP/HTTPS and PostgreSQL services on the primary. access HTTP/HTTPS and PostgreSQL services on the **primary** node.
### What core network devices support the application? ### What core network devices support the application?
...@@ -98,17 +98,17 @@ access HTTP/HTTPS and PostgreSQL services on the primary. ...@@ -98,17 +98,17 @@ access HTTP/HTTPS and PostgreSQL services on the primary.
### What network performance requirements exist? ### What network performance requirements exist?
- Maximum replication speeds between primary and secondary is limited by the - Maximum replication speeds between **primary** node and **secondary** node is limited by the
available bandwidth between sites. No hard requirements exist - time to complete available bandwidth between sites. No hard requirements exist - time to complete
replication (and ability to keep up with changes on the primary) is a function replication (and ability to keep up with changes on the **primary** node) is a function
of the size of the data set, tolerance for latency, and available network of the size of the data set, tolerance for latency, and available network
capacity. capacity.
### What private and public network links support the application? ### What private and public network links support the application?
- Customers choose their own networks. As sites are intended to be - Customers choose their own networks. As sites are intended to be
geographically separated, it is envisioned that replication traffic will pass geographically separated, it is envisioned that replication traffic will pass
over the public Internet in a typical deployment, but this is not a requirement. over the public Internet in a typical deployment, but this is not a requirement.
## Systems ## Systems
...@@ -121,8 +121,8 @@ over the public Internet in a typical deployment, but this is not a requirement. ...@@ -121,8 +121,8 @@ over the public Internet in a typical deployment, but this is not a requirement.
### What details regarding required OS components and lock‐down needs have been defined? ### What details regarding required OS components and lock‐down needs have been defined?
- The recommended installation method (Omnibus) packages most components itself. - The recommended installation method (Omnibus) packages most components itself.
A from-source installation method exists. Both are documented at A from-source installation method exists. Both are documented at
https://docs.gitlab.com/ee/gitlab-geo/ <https://docs.gitlab.com/ee/administration/geo/replication/index.html>
- There are significant dependencies on the system-installed OpenSSH daemon (Geo - There are significant dependencies on the system-installed OpenSSH daemon (Geo
requires users to set up custom authentication methods) and the omnibus or requires users to set up custom authentication methods) and the omnibus or
system-provided PostgreSQL daemon (it must be configured to listen on TCP, system-provided PostgreSQL daemon (it must be configured to listen on TCP,
...@@ -156,12 +156,12 @@ https://docs.gitlab.com/ee/gitlab-geo/ ...@@ -156,12 +156,12 @@ https://docs.gitlab.com/ee/gitlab-geo/
## What virtualization requirements have been defined for the application? ## What virtualization requirements have been defined for the application?
- Nothing Geo-specific, but everything in GitLab needs to have full - Nothing Geo-specific, but everything in GitLab needs to have full
functionality in such an environment. functionality in such an environment.
### What aspects of the product may or may not be hosted via the cloud computing model? ### What aspects of the product may or may not be hosted via the cloud computing model?
- GitLab is “cloud native” and this applies to Geo as much as to the rest of the - GitLab is “cloud native” and this applies to Geo as much as to the rest of the
product. Deployment in clouds is a common and supported scenario. product. Deployment in clouds is a common and supported scenario.
## If applicable, what approach(es) to cloud computing will be taken (Managed Hosting versus "Pure" Cloud, a "full machine" approach such as AWS-EC2 versus a "hosted database" approach such as AWS-RDS and Azure, etc)? ## If applicable, what approach(es) to cloud computing will be taken (Managed Hosting versus "Pure" Cloud, a "full machine" approach such as AWS-EC2 versus a "hosted database" approach such as AWS-RDS and Azure, etc)?
...@@ -184,103 +184,97 @@ product. Deployment in clouds is a common and supported scenario. ...@@ -184,103 +184,97 @@ product. Deployment in clouds is a common and supported scenario.
### How will database connection strings, encryption keys, and other sensitive components be stored, accessed, and protected from unauthorized detection? ### How will database connection strings, encryption keys, and other sensitive components be stored, accessed, and protected from unauthorized detection?
- There are some Geo-specific values. Some are shared secrets which must be - There are some Geo-specific values. Some are shared secrets which must be
securely transmitted from the primary to the secondary at setup time. Our securely transmitted from the **primary** node to the **secondary** node at setup time. Our
documentation recommends transmitting them from the primary to the system documentation recommends transmitting them from the **primary** node to the system
administrator via SSH, and then back out to the secondary in the same manner. administrator via SSH, and then back out to the **secondary** node in the same manner.
In particular, this includes the PostgreSQL replication credentials and a secret In particular, this includes the PostgreSQL replication credentials and a secret
key (`db_key_base`) which is used to decrypt certain columns in the database. key (`db_key_base`) which is used to decrypt certain columns in the database.
The `db_key_base` secret is stored unencrypted on the filesystem, in The `db_key_base` secret is stored unencrypted on the filesystem, in
`/etc/gitlab/gitlab-secrets.json`, along with a number of other secrets. There is `/etc/gitlab/gitlab-secrets.json`, along with a number of other secrets. There is
no at-rest protection for them. no at-rest protection for them.
## Data Processing ## Data Processing
### What data entry paths does the application support? ### What data entry paths does the application support?
- Data is entered via the web application exposed by GitLab itself. Some data is - Data is entered via the web application exposed by GitLab itself. Some data is
also entered using system administration commands on the GitLab servers (e.g., also entered using system administration commands on the GitLab servers (e.g.,
`gitlab-ctl set-primary-node`). `gitlab-ctl set-primary-node`).
- Secondaries also receive inputs via PostgreSQL streaming replication from the - **Secondary** nodes also receive inputs via PostgreSQL streaming replication from the **primary** node.
primary.
### What data output paths does the application support? ### What data output paths does the application support?
- Primaries output via PostgreSQL streaming replication to the secondary. - **Primary** nodes output via PostgreSQL streaming replication to the **secondary** node.
Otherwise, principally via the web application exposed by GitLab itself, and via Otherwise, principally via the web application exposed by GitLab itself, and via
SSH `git clone` operations initiated by the end-user. SSH `git clone` operations initiated by the end-user.
### How does data flow across the application's internal components? ### How does data flow across the application's internal components?
- Secondaries and primaries interact via HTTP/HTTPS (secured with JSON web - **Secondary** nodes and **primary** nodes interact via HTTP/HTTPS (secured with JSON web
tokens) and via PostgreSQL streaming replication. tokens) and via PostgreSQL streaming replication.
- Within a primary or secondary, the SSOT is the filesystem and the database - Within a **primary** node or **secondary** node, the SSOT is the filesystem and the database
(including Geo tracking database on secondary). The various internal components (including Geo tracking database on **secondary** node). The various internal components
are orchestrated to make alterations to these stores. are orchestrated to make alterations to these stores.
### What data input validation requirements have been defined? ### What data input validation requirements have been defined?
- Secondaries must have a faithful replication of the primary’s data. - **Secondary** nodes must have a faithful replication of the **primary** node’s data.
### What data does the application store and how? ### What data does the application store and how?
- Git repositories and files, tracking information related to the them, and the - Git repositories and files, tracking information related to the them, and the GitLab database contents.
GitLab database contents.
### What data is or may need to be encrypted and what key management requirements have been defined? ### What data is or may need to be encrypted and what key management requirements have been defined?
- Neither primaries or secondaries encrypt Git repository or filesystem data at - Neither **primary** nodes or **secondary** nodes encrypt Git repository or filesystem data at
rest. A subset of database columns are encrypted at rest using the `db_otp_key` rest. A subset of database columns are encrypted at rest using the `db_otp_key`.
- a static secret shared across all hosts in a GitLab deployment. - A static secret shared across all hosts in a GitLab deployment.
- In transit, data should be encrypted, although the application does permit - In transit, data should be encrypted, although the application does permit
communication to proceed unencrypted. The two main transits are the secondary’s communication to proceed unencrypted. The two main transits are the **secondary** node’s
replication process for PostgreSQL, and for git repositories/files. Both should replication process for PostgreSQL, and for git repositories/files. Both should
be protected using TLS, with the keys for that managed via Omnibus per existing be protected using TLS, with the keys for that managed via Omnibus per existing
configuration for end-user access to GitLab. configuration for end-user access to GitLab.
### What capabilities exist to detect the leakage of sensitive data? ### What capabilities exist to detect the leakage of sensitive data?
- Comprehensive system logs exist, tracking every connection to GitLab and - Comprehensive system logs exist, tracking every connection to GitLab and PostgreSQL.
PostgreSQL.
### What encryption requirements have been defined for data in transit - including transmission over WAN, LAN, SecureFTP, or publicly accessible protocols such as http: and https:? ### What encryption requirements have been defined for data in transit - including transmission over WAN, LAN, SecureFTP, or publicly accessible protocols such as http: and https:?
- Data must have the option to be encrypted in transit, and be secure against - Data must have the option to be encrypted in transit, and be secure against
both passive and active attack (e.g., MITM attacks should not be possible). both passive and active attack (e.g., MITM attacks should not be possible).
## Access ## Access
### What user privilege levels does the application support? ### What user privilege levels does the application support?
- Geo adds one type of privilege: secondaries can access a special Geo API to - Geo adds one type of privilege: **secondary** nodes can access a special Geo API to
download files over HTTP/HTTPS, and to clone repositories using HTTP/HTTPS. download files over HTTP/HTTPS, and to clone repositories using HTTP/HTTPS.
### What user identification and authentication requirements have been defined? ### What user identification and authentication requirements have been defined?
- Geo secondaries identify to Geo primaries via OAuth or JWT authentication - **Secondary** nodes identify to Geo **primary** nodes via OAuth or JWT authentication
based on the shared database (HTTP access) or a PostgreSQL replication user (for based on the shared database (HTTP access) or a PostgreSQL replication user (for
database replication). The database replication also requires IP-based access database replication). The database replication also requires IP-based access
controls to be defined. controls to be defined.
### What user authorization requirements have been defined? ### What user authorization requirements have been defined?
- Secondaries must only be able to *read* data. They are not currently able to - **Secondary** nodes must only be able to *read* data. They are not currently able to mutate data on the **primary** node.
mutate data on the primary.
### What session management requirements have been defined? ### What session management requirements have been defined?
- Geo JWTs are defined to last for only two minutes before needing to be - Geo JWTs are defined to last for only two minutes before needing to be regenerated.
regenerated.
### What access requirements have been defined for URI and Service calls? ### What access requirements have been defined for URI and Service calls?
- A Geo secondary makes many calls to the primary's API. This is how file - **Secondary** nodes make many calls to the **primary** node's API. This is how file
replication proceeds, for instance. This endpoint is only accessible with a JWT replication proceeds, for instance. This endpoint is only accessible with a JWT token.
token. - The **primary** node also makes calls to the **secondary** node to get status information.
- The primary also makes calls to the secondary to get status information.
## Application Monitoring ## Application Monitoring
### What application auditing requirements have been defined? How are audit and debug logs accessed, stored, and secured? ### What application auditing requirements have been defined? How are audit and debug logs accessed, stored, and secured?
- Structured JSON log is written to the filesystem, and can also be ingested - Structured JSON log is written to the filesystem, and can also be ingested
into a Kibana installation for further analysis. into a Kibana installation for further analysis.
# Geo Troubleshooting # Geo Troubleshooting
>**Note:** NOTE: **Note:**
This list is an attempt to document all the moving parts that can go wrong. This list is an attempt to document all the moving parts that can go wrong.
We are working into getting all this steps verified automatically in a We are working into getting all this steps verified automatically in a
rake task in the future. rake task in the future.
...@@ -9,10 +9,10 @@ Setting up Geo requires careful attention to details and sometimes it's easy to ...@@ -9,10 +9,10 @@ Setting up Geo requires careful attention to details and sometimes it's easy to
miss a step. Here is a list of questions you should ask to try to detect miss a step. Here is a list of questions you should ask to try to detect
what you need to fix (all commands and path locations are for Omnibus installs): what you need to fix (all commands and path locations are for Omnibus installs):
#### First check the health of the secondary ## First check the health of the **secondary** node
Visit the primary node's **Admin Area > Geo** (`/admin/geo_nodes`) in Visit the **primary** node's **Admin Area > Geo** (`/admin/geo/nodes`) in
your browser. We perform the following health checks on each secondary node your browser. We perform the following health checks on each **secondary** node
to help identify if something is wrong: to help identify if something is wrong:
- Is the node running? - Is the node running?
...@@ -23,30 +23,30 @@ to help identify if something is wrong: ...@@ -23,30 +23,30 @@ to help identify if something is wrong:
![Geo health check](img/geo_node_healthcheck.png) ![Geo health check](img/geo_node_healthcheck.png)
There is also an option to check the status of the secondary node by running a special rake task: There is also an option to check the status of the **secondary** node by running a special rake task:
``` ```sh
sudo gitlab-rake geo:status sudo gitlab-rake geo:status
``` ```
#### Is Postgres replication working? ## Is Postgres replication working?
#### Are my nodes pointing to the correct database instance? ### Are my nodes pointing to the correct database instance?
You should make sure your primary Geo node points to the instance with You should make sure your **primary** Geo node points to the instance with
writing permissions. writing permissions.
Any secondary nodes should point only to read-only instances. Any **secondary** nodes should point only to read-only instances.
#### Can Geo detect my current node correctly? ### Can Geo detect my current node correctly?
Geo uses the defined node from the `Admin ➔ Geo` screen, and tries to match Geo uses the defined node from the **Admin Area > Geo** screen, and tries to match
it with the value defined in the `/etc/gitlab/gitlab.rb` configuration file. it with the value defined in the `/etc/gitlab/gitlab.rb` configuration file.
The relevant line looks like: `external_url "http://gitlab.example.com"`. The relevant line looks like: `external_url "http://gitlab.example.com"`.
To check if the node on the current machine is correctly detected type: To check if the node on the current machine is correctly detected type:
```bash ```sh
sudo gitlab-rails runner "puts Gitlab::Geo.current_node.inspect" sudo gitlab-rails runner "puts Gitlab::Geo.current_node.inspect"
``` ```
...@@ -57,53 +57,52 @@ and expect something like: ...@@ -57,53 +57,52 @@ and expect something like:
``` ```
By running the command above, `primary` should be `true` when executed in By running the command above, `primary` should be `true` when executed in
the primary node, and `false` on any secondary. the **primary** node, and `false` on any **secondary** node.
#### How do I fix the message, "ERROR: replication slots can only be used if max_replication_slots > 0"? ## How do I fix the message, "ERROR: replication slots can only be used if max_replication_slots > 0"?
This means that the `max_replication_slots` PostgreSQL variable needs to This means that the `max_replication_slots` PostgreSQL variable needs to
be set on the primary database. In GitLab 9.4, we have made this setting be set on the **primary** database. In GitLab 9.4, we have made this setting
default to 1. You may need to increase this value if you have more Geo default to 1. You may need to increase this value if you have more
secondary nodes. Be sure to restart PostgreSQL for this to take **secondary** nodes. Be sure to restart PostgreSQL for this to take
effect. See the [PostgreSQL replication effect. See the [PostgreSQL replication
setup][database-pg-replication] guide for more details. setup][database-pg-replication] guide for more details.
#### How do I fix the message, "FATAL: could not start WAL streaming: ERROR: replication slot "geo_secondary_my_domain_com" does not exist"? ## How do I fix the message, "FATAL: could not start WAL streaming: ERROR: replication slot "geo_secondary_my_domain_com" does not exist"?
This occurs when PostgreSQL does not have a replication slot for the This occurs when PostgreSQL does not have a replication slot for the
secondary by that name. You may want to rerun the [replication **secondary** node by that name. You may want to rerun the [replication
process](database.md) on the secondary. process](database.md) on the **secondary** node .
#### How do I fix the message, "Command exceeded allowed execution time" when setting up replication? ## How do I fix the message, "Command exceeded allowed execution time" when setting up replication?
This may happen while [initiating the replication process][database-start-replication] on the Geo secondary, This may happen while [initiating the replication process][database-start-replication] on the **secondary** node,
and indicates that your initial dataset is too large to be replicated in the default timeout (30 minutes). and indicates that your initial dataset is too large to be replicated in the default timeout (30 minutes).
Re-run `gitlab-ctl replicate-geo-database`, but include a larger value for Re-run `gitlab-ctl replicate-geo-database`, but include a larger value for
`--backup-timeout`: `--backup-timeout`:
```bash ```sh
sudo gitlab-ctl replicate-geo-database --host=primary.geo.example.com --slot-name=secondary_geo_example_com --backup-timeout=21600 sudo gitlab-ctl replicate-geo-database --host=primary.geo.example.com --slot-name=secondary_geo_example_com --backup-timeout=21600
``` ```
This will give the initial replication up to six hours to complete, rather than This will give the initial replication up to six hours to complete, rather than
the default thirty minutes. Adjust as required for your installation. the default thirty minutes. Adjust as required for your installation.
#### How do I fix the message, "PANIC: could not write to file 'pg_xlog/xlogtemp.123': No space left on device" ## How do I fix the message, "PANIC: could not write to file 'pg_xlog/xlogtemp.123': No space left on device"
Determine if you have any unused replication slots in the primary database. This can cause large amounts of Determine if you have any unused replication slots in the **primary** database. This can cause large amounts of
log data to build up in `pg_xlog`. Removing the unused slots can reduce the amount of space used in the `pg_xlog`. log data to build up in `pg_xlog`. Removing the unused slots can reduce the amount of space used in the `pg_xlog`.
1. Start a PostgreSQL console session: 1. Start a PostgreSQL console session:
```bash ```sh
sudo gitlab-psql gitlabhq_production sudo gitlab-psql gitlabhq_production
``` ```
> Note that using `gitlab-rails dbconsole` will not work, because managing replication slots requires > Note that using `gitlab-rails dbconsole` will not work, because managing replication slots requires superuser permissions.
superuser permissions.
2. View your replication slots with 1. View your replication slots with:
```sql ```sql
SELECT * FROM pg_replication_slots; SELECT * FROM pg_replication_slots;
...@@ -111,8 +110,8 @@ log data to build up in `pg_xlog`. Removing the unused slots can reduce the amou ...@@ -111,8 +110,8 @@ log data to build up in `pg_xlog`. Removing the unused slots can reduce the amou
Slots where `active` is `f` are not active. Slots where `active` is `f` are not active.
- When this slot should be active, because you have a secondary configured using that slot, - When this slot should be active, because you have a **secondary** node configured using that slot,
log in to that secondary and check the PostgreSQL logs why the replication is not running. log in to that **secondary** node and check the PostgreSQL logs why the replication is not running.
- If you are no longer using the slot (e.g. you no longer have Geo enabled), you can remove it with in the - If you are no longer using the slot (e.g. you no longer have Geo enabled), you can remove it with in the
PostgreSQL console session: PostgreSQL console session:
...@@ -121,13 +120,13 @@ Slots where `active` is `f` are not active. ...@@ -121,13 +120,13 @@ Slots where `active` is `f` are not active.
SELECT pg_drop_replication_slot('name_of_extra_slot'); SELECT pg_drop_replication_slot('name_of_extra_slot');
``` ```
#### Very large repositories never successfully synchronize on the secondary ## Very large repositories never successfully synchronize on the **secondary** node
GitLab places a timeout on all repository clones, including project imports GitLab places a timeout on all repository clones, including project imports
and Geo synchronization operations. If a fresh `git clone` of a repository and Geo synchronization operations. If a fresh `git clone` of a repository
on the primary takes more than a few minutes, you may be affected by this. on the primary takes more than a few minutes, you may be affected by this.
To increase the timeout, add the following line to `/etc/gitlab/gitlab.rb` To increase the timeout, add the following line to `/etc/gitlab/gitlab.rb`
on the secondary: on the **secondary** node:
```ruby ```ruby
gitlab_rails['gitlab_shell_git_timeout'] = 10800 gitlab_rails['gitlab_shell_git_timeout'] = 10800
...@@ -135,16 +134,16 @@ gitlab_rails['gitlab_shell_git_timeout'] = 10800 ...@@ -135,16 +134,16 @@ gitlab_rails['gitlab_shell_git_timeout'] = 10800
Then reconfigure GitLab: Then reconfigure GitLab:
```bash ```sh
sudo gitlab-ctl reconfigure sudo gitlab-ctl reconfigure
``` ```
This will increase the timeout to three hours (10800 seconds). Choose a time This will increase the timeout to three hours (10800 seconds). Choose a time
long enough to accommodate a full clone of your largest repositories. long enough to accommodate a full clone of your largest repositories.
#### How to reset Geo secondary replication ## How to reset Geo **secondary** node replication
If you get a secondary node in a broken state and want to reset the replication state, If you get a **secondary** node in a broken state and want to reset the replication state,
to start again from scratch, there are a few steps that can help you: to start again from scratch, there are a few steps that can help you:
1. Stop Sidekiq and the Geo LogCursor 1. Stop Sidekiq and the Geo LogCursor
...@@ -155,7 +154,7 @@ to start again from scratch, there are a few steps that can help you: ...@@ -155,7 +154,7 @@ to start again from scratch, there are a few steps that can help you:
You need to send a **SIGTSTP** kill signal for the first phase and them a **SIGTERM** You need to send a **SIGTSTP** kill signal for the first phase and them a **SIGTERM**
when all jobs have finished. Otherwise just use the `gitlab-ctl stop` commands. when all jobs have finished. Otherwise just use the `gitlab-ctl stop` commands.
```bash ```sh
gitlab-ctl status sidekiq gitlab-ctl status sidekiq
# run: sidekiq: (pid 10180) <- this is the PID you will use # run: sidekiq: (pid 10180) <- this is the PID you will use
kill -TSTP 10180 # change to the correct PID kill -TSTP 10180 # change to the correct PID
...@@ -166,13 +165,13 @@ to start again from scratch, there are a few steps that can help you: ...@@ -166,13 +165,13 @@ to start again from scratch, there are a few steps that can help you:
You can watch sidekiq logs to know when sidekiq jobs processing have finished: You can watch sidekiq logs to know when sidekiq jobs processing have finished:
```bash ```sh
gitlab-ctl tail sidekiq gitlab-ctl tail sidekiq
``` ```
1. Rename repository storage folders and create new ones 1. Rename repository storage folders and create new ones
```bash ```sh
mv /var/opt/gitlab/git-data/repositories /var/opt/gitlab/git-data/repositories.old mv /var/opt/gitlab/git-data/repositories /var/opt/gitlab/git-data/repositories.old
mkdir -p /var/opt/gitlab/git-data/repositories mkdir -p /var/opt/gitlab/git-data/repositories
chown git:git /var/opt/gitlab/git-data/repositories chown git:git /var/opt/gitlab/git-data/repositories
...@@ -185,7 +184,7 @@ to start again from scratch, there are a few steps that can help you: ...@@ -185,7 +184,7 @@ to start again from scratch, there are a few steps that can help you:
1. _(Optional)_ Rename other data folders and create new ones 1. _(Optional)_ Rename other data folders and create new ones
CAUTION: **Caution**: CAUTION: **Caution**:
You may still have files on the secondary that have been removed from primary but You may still have files on the **secondary** node that have been removed from **primary** node but
removal have not been reflected. If you skip this step, they will never be removed removal have not been reflected. If you skip this step, they will never be removed
from this Geo node. from this Geo node.
...@@ -197,7 +196,7 @@ to start again from scratch, there are a few steps that can help you: ...@@ -197,7 +196,7 @@ to start again from scratch, there are a few steps that can help you:
To rename all of them: To rename all of them:
```bash ```sh
gitlab-ctl stop gitlab-ctl stop
mv /var/opt/gitlab/gitlab-rails/shared /var/opt/gitlab/gitlab-rails/shared.old mv /var/opt/gitlab/gitlab-rails/shared /var/opt/gitlab/gitlab-rails/shared.old
...@@ -210,19 +209,19 @@ to start again from scratch, there are a few steps that can help you: ...@@ -210,19 +209,19 @@ to start again from scratch, there are a few steps that can help you:
Reconfigure in order to recreate the folders and make sure permissions and ownership Reconfigure in order to recreate the folders and make sure permissions and ownership
are correctly are correctly
```bash ```sh
gitlab-ctl reconfigure gitlab-ctl reconfigure
``` ```
1. Reset the Tracking Database 1. Reset the Tracking Database
```bash ```sh
gitlab-rake geo:db:reset gitlab-rake geo:db:reset
``` ```
1. Restart previously stopped services 1. Restart previously stopped services
```bash ```sh
gitlab-ctl start gitlab-ctl start
``` ```
......
...@@ -2,14 +2,14 @@ ...@@ -2,14 +2,14 @@
## Changing the sync capacity values ## Changing the sync capacity values
In the Geo admin page (`/admin/geo_nodes`), there are several variables that In the Geo admin page (`/admin/geo/nodes`), there are several variables that
can be tuned to improve performance of Geo: can be tuned to improve performance of Geo:
* Repository sync capacity - Repository sync capacity.
* File sync capacity - File sync capacity.
Increasing these values will increase the number of jobs that are scheduled, Increasing these values will increase the number of jobs that are scheduled.
but this may not lead to a more downloads in parallel unless the number of However, this may not lead to more downloads in parallel unless the number of
available Sidekiq threads is also increased. For example, if repository sync available Sidekiq threads is also increased. For example, if repository sync
capacity is increased from 25 to 50, you may also want to increase the number capacity is increased from 25 to 50, you may also want to increase the number
of Sidekiq threads from 25 to 50. See the [Sidekiq concurrency of Sidekiq threads from 25 to 50. See the [Sidekiq concurrency
......
...@@ -8,32 +8,31 @@ different steps. ...@@ -8,32 +8,31 @@ different steps.
In order to update the Geo nodes when a new GitLab version is released, In order to update the Geo nodes when a new GitLab version is released,
all you need to do is update GitLab itself: all you need to do is update GitLab itself:
1. Log into each node (primary and secondaries) 1. Log into each node (**primary** and **secondary** nodes).
1. [Update GitLab][update] 1. [Update GitLab][update].
1. [Update tracking database on secondary node](#update-tracking-database-on-secondary-node) when 1. [Update tracking database on **secondary** node](#update-tracking-database-on-secondary-node) when
the tracking database is enabled. the tracking database is enabled.
1. [Test](#check-status-after-updating) primary and secondary nodes, and check version in each. 1. [Test](#check-status-after-updating) **primary** and **secondary** nodes, and check version in each.
## Upgrading to GitLab 10.8 ## Upgrading to GitLab 10.8
Before 10.8, broadcast messages would not propagate without flushing the cache on the secondaries. Before 10.8, broadcast messages would not propagate without flushing the cache on the **secondary** nodes. This has been fixed in 10.8, but requires one last cache flush on each **secondary** node:
This has been fixed in 10.8, but requires one last cache flush on each secondary node:
```bash ```sh
sudo gitlab-rake cache:clear sudo gitlab-rake cache:clear
``` ```
## Upgrading to GitLab 10.6 ## Upgrading to GitLab 10.6
In 10.4 we started to recommend that you define a password for database user (`gitlab`). In 10.4, we started to recommend that you define a password for database user (`gitlab`).
We now require this change as we use this password to enable the Foreign Data Wrapper, as a way to optimize We now require this change as we use this password to enable the Foreign Data Wrapper, as a way to optimize
the Geo Tracking Database. We are also improving security by disabling the use of **trust** the Geo Tracking Database. We are also improving security by disabling the use of **trust**
authentication method. authentication method.
1. **[primary]** Login to your primary node and run: 1. **[primary]** Login to your **primary** node and run:
```bash ```sh
gitlab-ctl pg-password-md5 gitlab gitlab-ctl pg-password-md5 gitlab
# Enter password: mypassword # Enter password: mypassword
# Confirm password: mypassword # Confirm password: mypassword
...@@ -58,14 +57,14 @@ authentication method. ...@@ -58,14 +57,14 @@ authentication method.
postgresql['trust_auth_cidr_addresses'] = ['127.0.0.1/32','1.2.3.4/32'] # <- Remove this postgresql['trust_auth_cidr_addresses'] = ['127.0.0.1/32','1.2.3.4/32'] # <- Remove this
``` ```
1. **[primary]** Reconfigure and restart 1. **[primary]** Reconfigure and restart:
```bash ```sh
sudo gitlab-ctl reconfigure sudo gitlab-ctl reconfigure
sudo gitlab-ctl restart sudo gitlab-ctl restart
``` ```
1. **[secondary]** Login to all secondary nodes and edit `/etc/gitlab/gitlab.rb`: 1. **[secondary]** Login to all **secondary** nodes and edit `/etc/gitlab/gitlab.rb`:
```ruby ```ruby
# Fill with the hash generated by `gitlab-ctl pg-password-md5 gitlab` # Fill with the hash generated by `gitlab-ctl pg-password-md5 gitlab`
...@@ -90,35 +89,35 @@ authentication method. ...@@ -90,35 +89,35 @@ authentication method.
postgresql['trust_auth_cidr_addresses'] = ['127.0.0.1/32','5.6.7.8/32'] # <- Remove this postgresql['trust_auth_cidr_addresses'] = ['127.0.0.1/32','5.6.7.8/32'] # <- Remove this
``` ```
1. **[secondary]** Reconfigure and restart 1. **[secondary]** Reconfigure and restart:
```bash ```sh
sudo gitlab-ctl reconfigure sudo gitlab-ctl reconfigure
sudo gitlab-ctl restart sudo gitlab-ctl restart
``` ```
## Upgrading to GitLab 10.5 ## Upgrading to GitLab 10.5
For Geo Disaster Recovery to work with minimum downtime, your Geo secondary For Geo Disaster Recovery to work with minimum downtime, your **secondary** node
should use the same set of secrets as the primary. However, setup instructions should use the same set of secrets as the **primary** node. However, setup instructions
prior to the 10.5 release only synchronized the `db_key_base` secret. prior to the 10.5 release only synchronized the `db_key_base` secret.
To rectify this error on existing installations, you should **overwrite** the To rectify this error on existing installations, you should **overwrite** the
contents of `/etc/gitlab/gitlab-secrets.json` on the secondary node with the contents of `/etc/gitlab/gitlab-secrets.json` on each **secondary** node with the
contents of `/etc/gitlab/gitlab-secrets.json` on the primary node, then run the contents of `/etc/gitlab/gitlab-secrets.json` on the **primary** node, then run the
following command on the secondary node: following command on each **secondary** node:
```bash ```sh
sudo gitlab-ctl reconfigure sudo gitlab-ctl reconfigure
``` ```
If you do not perform this step, you may find that two-factor authentication If you do not perform this step, you may find that two-factor authentication
[is broken following DR](faq.md#i-followed-the-disaster-recovery-instructions-and-now-two-factor-auth-is-broken). [is broken following DR](faq.md#i-followed-the-disaster-recovery-instructions-and-now-two-factor-auth-is-broken).
To prevent SSH requests to the newly promoted primary node from failing To prevent SSH requests to the newly promoted **primary** node from failing
due to SSH host key mismatch when updating the primary domain's DNS record due to SSH host key mismatch when updating the **primary** node domain's DNS record
you should perform the step to [Manually replicate primary SSH host keys][configuration-replicate-ssh] in each you should perform the step to [Manually replicate **primary** SSH host keys][configuration-replicate-ssh] in each
secondary node. **secondary** node.
## Upgrading to GitLab 10.4 ## Upgrading to GitLab 10.4
...@@ -145,7 +144,7 @@ file synchronization is **also** insecure in these cases! ...@@ -145,7 +144,7 @@ file synchronization is **also** insecure in these cases!
Support for TLS-secured PostgreSQL replication has been added. If you are Support for TLS-secured PostgreSQL replication has been added. If you are
currently using PostgreSQL replication across the open internet without an currently using PostgreSQL replication across the open internet without an
external means of securing the connection (e.g., a site-to-site VPN), then you external means of securing the connection (e.g., a site-to-site VPN), then you
should immediately reconfigure your primary and secondary PostgreSQL instances should immediately reconfigure your **primary** and **secondary** PostgreSQL instances
according to the [updated instructions][database]. according to the [updated instructions][database].
If you *are* securing the connections externally and wish to continue doing so, If you *are* securing the connections externally and wish to continue doing so,
...@@ -158,8 +157,8 @@ Support for replicating repositories and wikis over HTTP/HTTPS has been added. ...@@ -158,8 +157,8 @@ Support for replicating repositories and wikis over HTTP/HTTPS has been added.
Replicating over SSH has been deprecated, and support for this option will be Replicating over SSH has been deprecated, and support for this option will be
removed in a future release. removed in a future release.
To switch to HTTP/HTTPS replication, log into the primary node as an admin and visit To switch to HTTP/HTTPS replication, log into the **primary** node as an admin and visit
**Admin Area > Geo** (`/admin/geo_nodes`). For each secondary listed, **Admin Area > Geo** (`/admin/geo/nodes`). For each **secondary** node listed,
press the "Edit" button, change the "Repository cloning" setting from press the "Edit" button, change the "Repository cloning" setting from
"SSH (deprecated)" to "HTTP/HTTPS", and press "Save changes". This should take "SSH (deprecated)" to "HTTP/HTTPS", and press "Save changes". This should take
effect immediately. effect immediately.
...@@ -169,9 +168,9 @@ default setting. ...@@ -169,9 +168,9 @@ default setting.
After you've verified that HTTP/HTTPS replication is working, you should remove After you've verified that HTTP/HTTPS replication is working, you should remove
the now-unused SSH keys from your secondaries, as they may cause problems if the the now-unused SSH keys from your secondaries, as they may cause problems if the
secondary if ever promoted to a primary: **secondary** node if ever promoted to a **primary** node:
1. **[secondary]** Login to **all** your secondary nodes and run: 1. **[secondary]** Login to **all** your **secondary** nodes and run:
```ruby ```ruby
sudo -u git -H rm ~git/.ssh/id_rsa ~git/.ssh/id_rsa.pub sudo -u git -H rm ~git/.ssh/id_rsa ~git/.ssh/id_rsa.pub
...@@ -204,13 +203,13 @@ the database][ssh-fast-lookup] to avoid having to maintain consistency of the ...@@ -204,13 +203,13 @@ the database][ssh-fast-lookup] to avoid having to maintain consistency of the
`authorized_keys` file for SSH access. Failing to do this will prevent users `authorized_keys` file for SSH access. Failing to do this will prevent users
from being able to clone via SSH. from being able to clone via SSH.
Note that in older versions of Geo, attachments downloaded on the secondary Note that in older versions of Geo, attachments downloaded on the **secondary**
nodes would be saved to the wrong directory. We recommend that you do the nodes would be saved to the wrong directory. We recommend that you do the
following to clean this up. following to clean this up.
On the SECONDARY Geo nodes, run as root: On the **secondary** Geo nodes, run as root:
```bash ```sh
mv /var/opt/gitlab/gitlab-rails/working /var/opt/gitlab/gitlab-rails/working.old mv /var/opt/gitlab/gitlab-rails/working /var/opt/gitlab/gitlab-rails/working.old
mkdir /var/opt/gitlab/gitlab-rails/working mkdir /var/opt/gitlab/gitlab-rails/working
chmod 700 /var/opt/gitlab/gitlab-rails/working chmod 700 /var/opt/gitlab/gitlab-rails/working
...@@ -219,17 +218,17 @@ chown git:git /var/opt/gitlab/gitlab-rails/working ...@@ -219,17 +218,17 @@ chown git:git /var/opt/gitlab/gitlab-rails/working
You may delete `/var/opt/gitlab/gitlab-rails/working.old` any time. You may delete `/var/opt/gitlab/gitlab-rails/working.old` any time.
Once this is done, we advise restarting GitLab on the secondary nodes for the Once this is done, we advise restarting GitLab on the **secondary** nodes for the
new working directory to be used: new working directory to be used:
```bash ```sh
sudo gitlab-ctl restart sudo gitlab-ctl restart
``` ```
## Upgrading from GitLab 9.3 or older ## Upgrading from GitLab 9.3 or older
If you started running Geo on GitLab 9.3 or older, we recommend that you If you started running Geo on GitLab 9.3 or older, we recommend that you
resync your secondary PostgreSQL databases to use replication slots. If you resync your **secondary** PostgreSQL databases to use replication slots. If you
started using Geo with GitLab 9.4 or 10.x, no further action should be started using Geo with GitLab 9.4 or 10.x, no further action should be
required because replication slots are used by default. However, if you required because replication slots are used by default. However, if you
started with GitLab 9.3 and upgraded later, you should still follow the started with GitLab 9.3 and upgraded later, you should still follow the
...@@ -238,18 +237,18 @@ instructions below. ...@@ -238,18 +237,18 @@ instructions below.
When in doubt, it does not hurt to do a resync. The easiest way to do this in When in doubt, it does not hurt to do a resync. The easiest way to do this in
Omnibus is the following: Omnibus is the following:
1. Make sure you have Omnibus GitLab on the primary server 1. Make sure you have Omnibus GitLab on the **primary** server.
1. Run `gitlab-ctl reconfigure` and `gitlab-ctl restart postgresql`. This will enable replication slots on the primary database. 1. Run `gitlab-ctl reconfigure` and `gitlab-ctl restart postgresql`. This will enable replication slots on the **primary** database.
1. Check the steps about defining `postgresql['sql_user_password']`, `gitlab_rails['db_password']` 1. Check the steps about defining `postgresql['sql_user_password']`, `gitlab_rails['db_password']`.
1. Make sure `postgresql['max_replication_slots']` matches the number of secondary Geo Nodes locations 1. Make sure `postgresql['max_replication_slots']` matches the number of **secondary** Geo nodes locations.
1. Install GitLab on the secondary server. 1. Install GitLab on the **secondary** server.
1. Re-run the [database replication process][database-replication]. 1. Re-run the [database replication process][database-replication].
## Special update notes for 9.0.x ## Special update notes for 9.0.x
> **IMPORTANT**: > **IMPORTANT**:
With GitLab 9.0, the PostgreSQL version is upgraded to 9.6 and manual steps are With GitLab 9.0, the PostgreSQL version is upgraded to 9.6 and manual steps are
required in order to update the secondary nodes and keep the Streaming required in order to update the **secondary** nodes and keep the Streaming
Replication working. Downtime is required, so plan ahead. Replication working. Downtime is required, so plan ahead.
The following steps apply only if you upgrade from a 8.17 GitLab version to The following steps apply only if you upgrade from a 8.17 GitLab version to
...@@ -259,52 +258,52 @@ upgrade to 9.0+. ...@@ -259,52 +258,52 @@ upgrade to 9.0+.
--- ---
Make sure to follow the steps in the exact order as they appear below and pay Make sure to follow the steps in the exact order as they appear below and pay
extra attention in what node (primary/secondary) you execute them! Each step extra attention in what node (either **primary** or **secondary**) you execute them! Each step
is prepended with the relevant node for better clarity: is prepended with the relevant node for better clarity:
1. **[secondary]** Login to **all** your secondary nodes and stop all services: 1. **[secondary]** Login to **all** your **secondary** nodes and stop all services:
```ruby ```ruby
sudo gitlab-ctl stop sudo gitlab-ctl stop
``` ```
1. **[secondary]** Make a backup of the `recovery.conf` file on **all** 1. **[secondary]** Make a backup of the `recovery.conf` file on **all**
secondary nodes to preserve PostgreSQL's credentials: **secondary** nodes to preserve PostgreSQL's credentials:
```bash ```sh
sudo cp /var/opt/gitlab/postgresql/data/recovery.conf /var/opt/gitlab/ sudo cp /var/opt/gitlab/postgresql/data/recovery.conf /var/opt/gitlab/
``` ```
1. **[primary]** Update the primary node to GitLab 9.0 following the 1. **[primary]** Update the **primary** node to GitLab 9.0 following the
[regular update docs][update]. At the end of the update, the primary node [regular update docs][update]. At the end of the update, the **primary** node
will be running with PostgreSQL 9.6. will be running with PostgreSQL 9.6.
1. **[primary]** To prevent a de-synchronization of the repository replication, 1. **[primary]** To prevent a de-synchronization of the repository replication,
stop all services except `postgresql` as we will use it to re-initialize the stop all services except `postgresql` as we will use it to re-initialize the
secondary node's database: **secondary** node's database:
```bash ```sh
sudo gitlab-ctl stop sudo gitlab-ctl stop
sudo gitlab-ctl start postgresql sudo gitlab-ctl start postgresql
``` ```
1. **[secondary]** Run the following steps on each of the secondaries: 1. **[secondary]** Run the following steps on each of the **secondary** nodes:
1. **[secondary]** Stop all services: 1. **[secondary]** Stop all services:
```bash ```sh
sudo gitlab-ctl stop sudo gitlab-ctl stop
``` ```
1. **[secondary]** Prevent running database migrations: 1. **[secondary]** Prevent running database migrations:
```bash ```sh
sudo touch /etc/gitlab/skip-auto-migrations sudo touch /etc/gitlab/skip-auto-migrations
``` ```
1. **[secondary]** Move the old database to another directory: 1. **[secondary]** Move the old database to another directory:
```bash ```sh
sudo mv /var/opt/gitlab/postgresql{,.bak} sudo mv /var/opt/gitlab/postgresql{,.bak}
``` ```
...@@ -313,26 +312,26 @@ is prepended with the relevant node for better clarity: ...@@ -313,26 +312,26 @@ is prepended with the relevant node for better clarity:
1. **[secondary]** Make sure all services are up: 1. **[secondary]** Make sure all services are up:
```bash ```sh
sudo gitlab-ctl start sudo gitlab-ctl start
``` ```
1. **[secondary]** Reconfigure GitLab: 1. **[secondary]** Reconfigure GitLab:
```bash ```sh
sudo gitlab-ctl reconfigure sudo gitlab-ctl reconfigure
``` ```
1. **[secondary]** Run the PostgreSQL upgrade command: 1. **[secondary]** Run the PostgreSQL upgrade command:
```bash ```sh
sudo gitlab-ctl pg-upgrade sudo gitlab-ctl pg-upgrade
``` ```
1. **[secondary]** See the stored credentials for the database that you will 1. **[secondary]** See the stored credentials for the database that you will
need to re-initialize the replication: need to re-initialize the replication:
```bash ```sh
sudo grep -s primary_conninfo /var/opt/gitlab/recovery.conf sudo grep -s primary_conninfo /var/opt/gitlab/recovery.conf
``` ```
...@@ -342,28 +341,28 @@ is prepended with the relevant node for better clarity: ...@@ -342,28 +341,28 @@ is prepended with the relevant node for better clarity:
1. **[secondary]** Run the recovery script using the credentials from the 1. **[secondary]** Run the recovery script using the credentials from the
previous step: previous step:
```bash ```sh
sudo bash /tmp/replica.sh sudo bash /tmp/replica.sh
``` ```
1. **[secondary]** Reconfigure GitLab: 1. **[secondary]** Reconfigure GitLab:
```bash ```sh
sudo gitlab-ctl reconfigure sudo gitlab-ctl reconfigure
``` ```
1. **[secondary]** Start all services: 1. **[secondary]** Start all services:
```bash ```sh
sudo gitlab-ctl start sudo gitlab-ctl start
``` ```
1. **[secondary]** Repeat the steps for the rest of the secondaries. 1. **[secondary]** Repeat the steps for the remaining **secondary** nodes.
1. **[primary]** After all secondaries are updated, start all services in 1. **[primary]** After all **secondary** nodes are updated, start all services in
primary: **primary** node:
```bash ```sh
sudo gitlab-ctl start sudo gitlab-ctl start
``` ```
...@@ -374,27 +373,27 @@ everything is working correctly: ...@@ -374,27 +373,27 @@ everything is working correctly:
1. Run the Geo raketask on all nodes, everything should be green: 1. Run the Geo raketask on all nodes, everything should be green:
```bash ```sh
sudo gitlab-rake gitlab:geo:check sudo gitlab-rake gitlab:geo:check
``` ```
1. Check the primary's Geo dashboard for any errors 1. Check the **primary** node's Geo dashboard for any errors.
1. Test the data replication by pushing code to the primary and see if it 1. Test the data replication by pushing code to the **primary** node and see if it
is received by the secondaries is received by **secondary** nodes.
## Update tracking database on secondary node ## Update tracking database on **secondary** node
After updating a secondary node, you might need to run migrations on After updating a **secondary** node, you might need to run migrations on
the tracking database. The tracking database was added in GitLab 9.1, the tracking database. The tracking database was added in GitLab 9.1,
and it is required since 10.0. and it is required since 10.0.
1. Run database migrations on tracking database 1. Run database migrations on tracking database:
```bash ```sh
sudo gitlab-rake geo:db:migrate sudo gitlab-rake geo:db:migrate
``` ```
1. Repeat this step for every secondary node 1. Repeat this step for each **secondary** node.
[update]: ../../../update/README.md [update]: ../../../update/README.md
[database]: database.md [database]: database.md
......
...@@ -3,7 +3,7 @@ ...@@ -3,7 +3,7 @@
For more information about setting up GitLab Geo, read the For more information about setting up GitLab Geo, read the
[Geo documentation](../../gitlab-geo/README.md). [Geo documentation](../../gitlab-geo/README.md).
When you're done, you can navigate to **Admin area > Geo** (`/admin/geo_nodes`). When you're done, you can navigate to **Admin area > Geo** (`/admin/geo/nodes`).
## Common settings ## Common settings
......
Markdown is supported
0%
or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment