Commit 24cf02b6 authored by Achilleas Pipinellis's avatar Achilleas Pipinellis

Merge branch 'jramsay-1989-rename-geo-docs' into 'master'

Separate Disaster Recovery docs from Geo docs

Closes #1989

See merge request gitlab-org/gitlab-ee!4511
parents 45f92fc1 fa82064f
# Bring a demoted primary node back online
After a failover, it is possible to fail back to the demoted primary to
restore your original configuration. This process consists of two steps:
1. Making the old primary a secondary
1. Promoting a secondary to a primary
## Configure the former primary to be a secondary
Since the former primary will be out of sync with the current primary, the first
step is to bring the former primary up to date. There is one downside though,
some uploads and repositories that have been deleted during an idle period of a
primary node, will not be deleted from the disk but the overall sync will be
much faster. As an alternative, you can set up a
[GitLab instance from scratch](../replication/index.md#setup-instructions) to
workaround this downside.
To bring the former primary up to date:
1. SSH into the former primary that has fallen behind
1. Make sure all the services are up:
```bash
sudo gitlab-ctl start
```
NOTE: **Note:** If you [disabled the primary permanently](index.md#step-2-permanently-disable-the-primary),
you need to undo those steps now. For Debian/Ubuntu you just need to run
`sudo systemctl enable gitlab-runsvdir`. For CentOS 6, you need to install
the GitLab instance from scratch and setup it as a secondary node by
following the [setup instructions](../replication/index.md#setup-instructions).
In this case you don't need to follow the next step.
1. [Setup database replication](../replication/database.md). Note that in this
case, primary refers to the current primary, and secondary refers to the
former primary.
If you have lost your original primary, follow the
[setup instructions](../replication/index.md#setup-instructions) to set up a new secondary.
## Promote the secondary to primary
When the initial replication is complete and the primary and secondary are
closely in sync, you can do a [planned failover](planned_failover.md).
## Restore the secondary node
If your objective is to have two nodes again, you need to bring your secondary
node back online as well by repeating the first step
([configure the former primary to be a secondary](#configure-the-former-primary-to-be-a-secondary))
for the secondary node.
# Disaster Recovery
Geo replicates your database and your Git repositories. We will
support and replicate more data in the future, that will enable you to
failover with minimal effort, in a disaster situation.
See [Geo current limitations](../replication/index.md#current-limitations)
for more information.
## Promoting secondary Geo replica in single-secondary configurations
We don't currently provide an automated way to promote a Geo replica and do a
failover, but you can do it manually if you have `root` access to the machine.
This process promotes a secondary Geo replica to a primary. To regain
geographical redundancy as quickly as possible, you should add a new secondary
immediately after following these instructions.
### Step 1. Allow replication to finish if possible
If the secondary is still replicating data from the primary, follow
[the planned failover docs](planned_failover.md) as closely as possible in
order to avoid unnecessary data loss.
### Step 2. Permanently disable the primary
CAUTION: **Warning:**
If a primary goes offline, there may be data saved on the primary
that has not been replicated to the secondary. This data should be treated
as lost if you proceed.
If an outage on your primary happens, you should do everything possible to
avoid a split-brain situation where writes can occur to two different GitLab
instances, complicating recovery efforts. So to prepare for the failover, we
must disable the primary.
1. SSH into your **primary** to stop and disable GitLab, if possible:
```bash
sudo gitlab-ctl stop
```
Prevent GitLab from starting up again if the server unexpectedly reboots:
```bash
sudo systemctl disable gitlab-runsvdir
```
On some operating systems such as CentOS 6, an easy way to prevent GitLab
from being started if the machine reboots isn't available
(see [Omnibus issue #3058](https://gitlab.com/gitlab-org/omnibus-gitlab/issues/3058)).
It may be safest to uninstall the GitLab package completely:
```bash
yum remove gitlab-ee
```
1. If you do not have SSH access to your primary, take the machine offline and
prevent it from rebooting by any means at your disposal.
Since there are many ways you may prefer to accomplish this, we will avoid a
single recommendation. You may need to:
- Reconfigure the load balancers
- Change DNS records (e.g., point the primary DNS record to the secondary node in order to stop usage of the primary)
- Stop the virtual servers
- Block traffic through a firewall
- Revoke object storage permissions from the primary
- Physically disconnect a machine
### Step 3. Promoting a secondary Geo replica
1. SSH in to your **secondary** and login as root:
```bash
sudo -i
```
1. Edit `/etc/gitlab/gitlab.rb` to reflect its new status as primary by
removing the following line:
```ruby
## REMOVE THIS LINE
geo_secondary_role['enable'] = true
```
A new secondary should not be added at this time. If you want to add a new
secondary, do this after you have completed the entire process of promoting
the secondary to the primary.
1. Promote the secondary to primary. Execute:
```bash
gitlab-ctl promote-to-primary-node
```
1. Verify you can connect to the newly promoted primary using the URL used
previously for the secondary.
1. Success! The secondary has now been promoted to primary.
### Step 4. (Optional) Updating the primary domain DNS record
Updating the DNS records for the primary domain to point to the secondary
will prevent the need to update all references to the primary domain to the
secondary domain, like changing Git remotes and API URLs.
1. SSH in to your **secondary** and login as root:
```bash
sudo -i
```
1. Update the primary domain's DNS record. After updating the primary domain's
DNS records to point to the secondary, edit `/etc/gitlab/gitlab.rb` on the
secondary to reflect the new URL:
```ruby
# Change the existing external_url configuration
external_url 'https://gitlab.example.com'
```
1. Reconfigure the secondary node for the change to take effect:
```bash
gitlab-ctl reconfigure
```
1. Execute the command below to update the newly promoted primary node URL:
```bash
gitlab-rake geo:update_primary_node_url
```
This command will use the changed `external_url` configuration defined
in `/etc/gitlab/gitlab.rb`.
1. Verify you can connect to the newly promoted primary using the primary URL.
If you updated the DNS records for the primary domain, these changes may
not have yet propagated depending on the previous DNS records TTL.
### Step 5. (Optional) Add secondary Geo replicas to a promoted primary
Promoting a secondary to primary using the process above does not enable
Geo on the new primary.
To bring a new secondary online, follow the
[Geo setup instructions](../replication/index.md#setup-instructions).
## Promoting secondary Geo replica in multi-secondary configurations
CAUTION: **Caution:**
Disaster Recovery for multi-secondary configurations is in
**Alpha** development. Do not use this as your only Disaster Recovery
strategy as you may lose data.
Disaster Recovery does not yet support systems with multiple
secondary Geo replicas (e.g., one primary and two or more secondaries). We are
working on it, see [#4284](https://gitlab.com/gitlab-org/gitlab-ee/issues/4284)
for details.
## Troubleshooting
### I followed the disaster recovery instructions and now two-factor auth is broken!
The setup instructions for Geo prior to 10.5 failed to replicate the
`otp_key_base` secret, which is used to encrypt the two-factor authentication
secrets stored in the database. If it differs between primary and secondary
nodes, users with two-factor authentication enabled won't be able to log in
after a failover.
If you still have access to the old primary node, you can follow the
instructions in the
[Upgrading to GitLab 10.5](../replication/updating_the_geo_nodes.md#upgrading-to-gitlab-105)
section to resolve the error. Otherwise, the secret is lost and you'll need to
[reset two-factor authentication for all users](../../../security/two_factor_authentication.md#disabling-2fa-for-everyone).
# Disaster recovery for planned failover
A planned failover is similar to a disaster recovery scenario, except you are able
to notify users of the maintenance window, and allow data to finish replicating to
secondaries.
Please read this entire document as well as [Disaster Recovery](index.md)
before proceeding.
## Notify users of scheduled maintenance
On the primary, navigate to **Admin Area > Messages**, add a broadcast message.
You can check under **Admin Area > Geo Nodes** to estimate how long it will
take to finish syncing. An example message would be:
>
A scheduled maintenance will take place at XX:XX UTC. We expect it to take
less than 1 hour.
On the secondary, you may need to clear the cache for the broadcast message
to show up.
## Block primary traffic
At the scheduled time, using your cloud provider or your node's firewall, block
HTTP and SSH traffic to/from the primary except for your IP and the secondary's
IP.
## Allow replication to finish as much as possible
1. On the secondary, navigate to **Admin Area > Geo Nodes** and wait until all
replication progress is 100% on the secondary "Current node".
1. Navigate to **Admin Area > Monitoring > Background Jobs > Queues** and wait
until the "geo" queues drop ideally to 0.
## Promote the secondary
Finally, follow the [Disaster Recovery docs](index.md) to promote the secondary
to a primary.
# Geo configuration
>**Note:**
This is the documentation for the Omnibus GitLab packages. For installations
from source, follow the [**Geo nodes configuration for installations
from source**](configuration_source.md) guide.
## Configuring a new secondary node
>**Note:**
This is the final step in setting up a secondary Geo node. Stages of the
setup process must be completed in the documented order.
Before attempting the steps in this stage, [complete all prior stages](index.md#using-omnibus-gitlab).
The basic steps of configuring a secondary node are to replicate required
configurations between the primary and the secondaries; to configure a tracking
database on each secondary; and to start GitLab on the secondary node.
You are encouraged to first read through all the steps before executing them
in your testing/production environment.
>**Notes:**
- **Do not** setup any custom authentication in the secondary nodes, this will be
handled by the primary node.
- **Do not** add anything in the secondaries Geo nodes admin area
(**Admin Area ➔ Geo Nodes**). This is handled solely by the primary node.
### Step 1. Manually replicate secret GitLab values
GitLab stores a number of secret values in the `/etc/gitlab/gitlab-secrets.json`
file which *must* match between the primary and secondary nodes. Until there is
a means of automatically replicating these between nodes (see
[issue #3789](https://gitlab.com/gitlab-org/gitlab-ee/issues/3789)), they must
be manually replicated to the secondary.
1. SSH into the **primary** node, and execute the command below:
```bash
sudo cat /etc/gitlab/gitlab-secrets.json
```
This will display the secrets that need to be replicated, in JSON format.
1. SSH into the **secondary** node and login as the `root` user:
```
sudo -i
```
1. Make a backup of any existing secrets:
```bash
mv /etc/gitlab/gitlab-secrets.json /etc/gitlab/gitlab-secrets.json.`date +%F`
```
1. Copy `/etc/gitlab/gitlab-secrets.json` from the primary to the secondary, or
copy-and-paste the file contents between nodes:
```bash
sudo editor /etc/gitlab/gitlab-secrets.json
# paste the output of the `cat` command you ran on the primary
# save and exit
```
1. Ensure the file permissions are correct:
```bash
chown root:root /etc/gitlab/gitlab-secrets.json
chmod 0600 /etc/gitlab/gitlab-secrets.json
```
1. Reconfigure the secondary node for the change to take effect:
```
gitlab-ctl reconfigure
```
Once reconfigured, the secondary will automatically start
replicating missing data from the primary in a process known as backfill.
Meanwhile, the primary node will start to notify the secondary of any changes, so
that the secondary can act on those notifications immediately.
Make sure the secondary instance is
running and accessible. You can login to the secondary node
with the same credentials as used in the primary.
### Step 2. Manually replicate primary SSH host keys
GitLab integrates with the system-installed SSH daemon, designating a user
(typically named git) through which all access requests are handled.
In a [Disaster Recovery](../disaster_recovery/index.md) situation, GitLab system
administrators will promote a secondary Geo replica to a primary and they can
update the DNS records for the primary domain to point to the secondary to prevent
the need to update all references to the primary domain to the secondary domain,
like changing Git remotes and API URLs.
This will cause all SSH requests to the newly promoted primary node from
failing due to SSH host key mismatch. To prevent this, the primary SSH host
keys must be manually replicated to the secondary node.
1. SSH into the **secondary** node and login as the `root` user:
```
sudo -i
```
1. Make a backup of any existing SSH host keys:
```bash
find /etc/ssh -iname ssh_host_* -exec cp {} {}.backup.`date +%F` \;
```
1. SSH into the **primary** node, and execute the command below:
```bash
sudo find /etc/ssh -iname ssh_host_* -not -iname '*.pub'
```
1. For each file in that list replace the file from the primary node to
the **same** location on your **secondary** node.
1. On your **secondary** node, ensure the file permissions are correct:
```bash
chown root:root /etc/ssh/ssh_host_*
chmod 0600 /etc/ssh/ssh_host_*
```
1. Regenerate the public keys from the private keys:
```bash
find /etc/ssh -iname ssh_host_* -not -iname '*.backup*' -exec sh -c 'ssh-keygen -y -f "{}" > "{}.pub"' \;
```
1. Restart sshd:
```bash
service ssh restart
```
### Step 3. (Optional) Enabling hashed storage (from GitLab 10.0)
>**Warning**
Hashed storage is in **Beta**. It is not considered production-ready. See
[Hashed Storage](../../repository_storage_types.md) for more detail,
and for the latest updates, check
[infrastructure issue #2821](https://gitlab.com/gitlab-com/infrastructure/issues/2821).
Using hashed storage significantly improves Geo replication - project and group
renames no longer require synchronization between nodes.
1. Visit the **primary** node's **Admin Area ➔ Settings**
(`/admin/application_settings`) in your browser
1. In the `Repository Storages` section, check `Create new projects using hashed storage paths`:
![](img/hashed_storage.png)
### Step 4. (Optional) Configuring the secondary to trust the primary
You can safely skip this step if your primary uses a CA-issued HTTPS certificate.
If your primary is using a self-signed certificate for *HTTPS* support, you will
need to add that certificate to the secondary's trust store. Retrieve the
certificate from the primary and follow
[these instructions](https://docs.gitlab.com/omnibus/settings/ssl.html)
on the secondary.
### Step 5. Enable Git access over HTTP/HTTPS
Geo synchronizes repositories over HTTP/HTTPS, and therefore requires this clone
method to be enabled. Navigate to **Admin Area ➔ Settings**
(`/admin/application_settings`) on the primary node, and set
`Enabled Git access protocols` to `Both SSH and HTTP(S)` or `Only HTTP(S)`.
### Step 6. Verify proper functioning of the secondary node
Congratulations! Your secondary geo node is now configured!
You can login to the secondary node with the same credentials you used on the
primary. Visit the secondary node's **Admin Area ➔ Geo Nodes**
(`/admin/geo_nodes`) in your browser to check if it's correctly identified as a
secondary Geo node and if Geo is enabled.
The initial replication, or 'backfill', will probably still be in progress. You
can monitor the synchronization process on each geo node from the primary
node's Geo Nodes dashboard in your browser.
![Geo dashboard](img/geo_node_dashboard.png)
If your installation isn't working properly, check the
[troubleshooting document](troubleshooting.md).
The two most obvious issues that can become apparent in the dashboard are:
1. Database replication not working well
1. Instance to instance notification not working. In that case, it can be
something of the following:
- You are using a custom certificate or custom CA (see the
[troubleshooting document](troubleshooting.md))
- The instance is firewalled (check your firewall rules)
Please note that disabling a secondary node will stop the sync process.
Please note that if `git_data_dirs` is customized on the primary for multiple
repository shards you must duplicate the same configuration on the secondary.
Point your users to the ["Using a Geo Server" guide](using_a_geo_server.md).
Currently, this is what is synced:
* Git repositories
* Wikis
* LFS objects
* Issues, merge requests, snippets, and comment attachments
* Users, groups, and project avatars
## Selective synchronization
Geo supports selective synchronization, which allows admins to choose
which projects should be synchronized by secondary nodes.
It is important to note that selective synchronization does not:
1. Restrict permissions from secondary nodes.
1. Hide project metadata from secondary nodes.
* Since Geo currently relies on PostgreSQL replication, all project metadata
gets replicated to secondary nodes, but repositories that have not been
selected will be empty.
1. Reduce the number of events generated for the Geo event log
* The primary generates events as long as any secondaries are present.
Selective synchronization restrictions are implemented on the secondaries,
not the primary.
A subset of projects can be chosen, either by group or by storage shard. The
former is ideal for replicating data belonging to a subset of users, while the
latter is more suited to progressively rolling out Geo to a large GitLab
instance.
## Upgrading Geo
See the [updating the Geo nodes document](updating_the_geo_nodes.md).
## Troubleshooting
See the [troubleshooting document](troubleshooting.md).
# Geo configuration
>**Note:**
This is the documentation for installations from source. For installations
using the Omnibus GitLab packages, follow the
[**Omnibus Geo nodes configuration**](configuration.md) guide.
## Configuring a new secondary node
>**Note:**
This is the final step in setting up a secondary Geo node. Stages of the setup
process must be completed in the documented order. Before attempting the steps
in this stage, [complete all prior stages](index.md#using-gitlab-installed-from-source).
The basic steps of configuring a secondary node are to replicate required
configurations between the primary and the secondaries; to configure a tracking
database on each secondary; and to start GitLab on the secondary node.
You are encouraged to first read through all the steps before executing them
in your testing/production environment.
>**Notes:**
- **Do not** setup any custom authentication in the secondary nodes, this will be
handled by the primary node.
- **Do not** add anything in the secondaries Geo nodes admin area
(**Admin Area ➔ Geo Nodes**). This is handled solely by the primary node.
### Step 1. Manually replicate secret GitLab values
GitLab stores a number of secret values in the `/home/git/gitlab/config/secrets.yml`
file which *must* match between the primary and secondary nodes. Until there is
a means of automatically replicating these between nodes (see
[issue #3789](https://gitlab.com/gitlab-org/gitlab-ee/issues/3789)), they must
be manually replicated to the secondary.
1. SSH into the **primary** node, and execute the command below:
```bash
sudo cat /home/git/gitlab/config/secrets.yml
```
This will display the secrets that need to be replicated, in YAML format.
1. SSH into the **secondary** node and login as the `git` user:
```bash
sudo -i -u git
```
1. Make a backup of any existing secrets:
```bash
mv /home/git/gitlab/config/secrets.yml /home/git/gitlab/config/secrets.yml.`date +%F`
```
1. Copy `/home/git/gitlab/config/secrets.yml` from the primary to the secondary, or
copy-and-paste the file contents between nodes:
```bash
sudo editor /home/git/gitlab/config/secrets.yml
# paste the output of the `cat` command you ran on the primary
# save and exit
```
1. Ensure the file permissions are correct:
```bash
chown git:git /home/git/gitlab/config/secrets.yml
chmod 0600 /home/git/gitlab/config/secrets.yml
```
1. Restart GitLab for the changes to take effect:
```bash
service gitlab restart
```
Once restarted, the secondary will automatically start replicating missing data
from the primary in a process known as backfill. Meanwhile, the primary node
will start to notify the secondary of any changes, so that the secondary can
act on those notifications immediately.
Make sure the secondary instance is running and accessible. You can login to
the secondary node with the same credentials as used in the primary.
### Step 2. Manually replicate primary SSH host keys
Read [Manually replicate primary SSH host keys](configuration.md#step-2-manually-replicate-primary-ssh-host-keys)
### Step 3. (Optional) Enabling hashed storage (from GitLab 10.0)
Read [Enabling Hashed Storage](configuration.md#step-3-optional-enabling-hashed-storage-from-gitlab-10-0)
### Step 4. (Optional) Configuring the secondary to trust the primary
You can safely skip this step if your primary uses a CA-issued HTTPS certificate.
If your primary is using a self-signed certificate for *HTTPS* support, you will
need to add that certificate to the secondary's trust store. Retrieve the
certificate from the primary and follow your distribution's instructions for
adding it to the secondary's trust store. In Debian/Ubuntu, for example, with a
certificate file of `primary.geo.example.com.crt`, you would follow these steps:
```
sudo -i
cp primary.geo.example.com.crt /usr/local/share/ca-certificates
update-ca-certificates
```
### Step 5. Enable Git access over HTTP/HTTPS
Geo synchronizes repositories over HTTP/HTTPS, and therefore requires this clone
method to be enabled. Navigate to **Admin Area ➔ Settings**
(`/admin/application_settings`) on the primary node, and set
`Enabled Git access protocols` to `Both SSH and HTTP(S)` or `Only HTTP(S)`.
### Step 6. Verify proper functioning of the secondary node
Read [Verify proper functioning of the secondary node](configuration.md#step-6-verify-proper-functioning-of-the-secondary-node).
## Selective synchronization
Read [Selective synchronization](configuration.md#selective-synchronization).
## Troubleshooting
Read the [troubleshooting document](troubleshooting.md).
This diff is collapsed.
This diff is collapsed.
# Docker Registry for a secondary node
You can setup a [Docker Registry](https://docs.docker.com/registry/) on your
secondary Geo node that mirrors the one on the primary Geo node.
## Storage support
CAUTION: **Warning:**
If you use [local storage](../../container_registry.md#container-registry-storage-driver)
for the Container Registry you **cannot** replicate it to the secondary Geo node.
Docker Registry currently supports a few types of storages. If you choose a
distributed storage (`azure`, `gcs`, `s3`, `swift`, or `oss`) for your Docker
Registry on a primary Geo node, you can use the same storage for a secondary
Docker Registry as well. For more information, read the
[Load balancing considerations](https://docs.docker.com/registry/deploying/#load-balancing-considerations)
when deploying the Registry, and how to setup the storage driver for GitLab's
integrated [Container Registry](../../container_registry.md#container-registry-storage-driver).
[ee]: https://about.gitlab.com/products/
# Geo Frequently Asked Questions
## Can I use Geo in a disaster recovery situation?
Yes, but there are limitations to what we replicate (see
[What data is replicated to a secondary node?](#what-data-is-replicated-to-a-secondary-node)).
Read the documentation for [Disaster Recovery](../disaster_recovery/index.md).
## What data is replicated to a secondary node?
We currently replicate project repositories, LFS objects, generated
attachments / avatars and the whole database. This means user accounts,
issues, merge requests, groups, project data, etc., will be available for
query. We currently don't replicate artifact data (`shared/folder`).
## Can I git push to a secondary node?
No. All writing operations (this includes `git push`) must be done in your
primary node.
## How long does it take to have a commit replicated to a secondary node?
All replication operations are asynchronous and are queued to be dispatched in
a batched request every 10 minutes. Besides that, it depends on a lot of other
factors including the amount of traffic, how big your commit is, the
connectivity between your nodes, your hardware, etc.
## What if the SSH server runs at a different port?
We send the clone url from the primary server to any secondaries, so it
doesn't matter. If primary is running on port `2200`, clone url will reflect
that.
## Is this possible to set up a Docker Registry for a secondary node that mirrors the one on a primary node?
Yes. See [Docker Registry for a secondary Geo node](docker_registry.md).
# Geo High Availability
This document describes a minimal reference architecture for running Geo
in a high availability configuration. If your HA setup differs from the one
described, it is possible to adapt these instructions to your needs.
## Architecture overview
![Geo HA Diagram](../../img/high_availability/geo-ha-diagram.png)
_[diagram source - gitlab employees only](https://docs.google.com/drawings/d/1z0VlizKiLNXVVVaERFwgsIOuEgjcUqDTWPdQYsE7Z4c/edit)_
The topology above assumes that the primary and secondary Geo clusters
are located in two separate locations, on their own virtual network
with private IP addresses. The network is configured such that all machines within
one geographic location can communicate with each other using their private IP addresses.
The IP addresses given are examples and may be different depending on the
network topology of your deployment.
The only external way to access the two Geo deployments is by HTTPS at
`gitlab.us.example.com` and `gitlab.eu.example.com` in the example above.
> **Note:** The primary and secondary Geo deployments must be able to
> communicate to each other over HTTPS.
## Redis and PostgreSQL High Availability
The primary and secondary Redis and PostgreSQL should be configured
for high availability. Because of the additional complexity involved
in setting up this configuration for PostgreSQL and Redis
it is not covered by this Geo HA documentation.
The two services will instead be configured such that
they will each run on a single machine.
For more information about setting up a highly available PostgreSQL cluster and Redis cluster using the omnibus package see the high availability documentation for
[PostgreSQL](../../high_availability/database.md) and
[Redis](../../high_availability/redis.md), respectively.
From these instructions you will need the following for the examples below:
* `gitlab_rails['db_password']` for the PostgreSQL "DB password"
* `redis['password']` for the Redis "Redis password"
NOTE: **Note:**
It is possible to use cloud hosted services for PostgreSQL and Redis but this is beyond the scope of this document.
### Prerequisites
Make sure you have GitLab EE installed using the
[Omnibus package](https://about.gitlab.com/installation).
### Step 1: Configure the Geo Backend Services
On the **primary** backend servers configure the following services:
* [Redis](../../high_availability/redis.md) for high availability.
* [NFS Server](../../high_availability/nfs.md) for repository, LFS, and upload storage.
* [PostgreSQL](../../high_availability/database.md) for high availability.
On the **secondary** backend servers configure the following services:
* [Redis](../../high_availability/redis.md) for high availability.
* [NFS Server](../../high_availability/nfs.md) which will store data that is synchronized from the Geo primary.
### Step 2: Configure the Postgres services on the Geo Secondary
1. Configure the [secondary Geo PostgreSQL database](database.md)
as a read-only secondary of the primary Geo PostgreSQL database.
1. Configure the Geo tracking database on the secondary server, to do this modify `/etc/gitlab/gitlab.rb`:
```ruby
geo_postgresql['enable'] = true
geo_postgresql['listen_address'] = '10.1.4.1'
geo_postgresql['trust_auth_cidr_addresses'] = ['10.1.0.0/16']
geo_secondary['auto_migrate'] = true
geo_secondary['db_host'] = '10.1.4.1'
geo_secondary['db_password'] = 'Geo tracking DB password'
```
NOTE: **Note:**
Be sure that other non-postgresql services are disabled by setting `enable` to `false` in
the [gitlab.rb configuration](https://gitlab.com/gitlab-org/omnibus-gitlab/blob/master/files/gitlab-config-template/gitlab.rb.template).
After making these changes be sure to run `sudo gitlab-ctl reconfigure` so that they take effect.
### Step 3: Setup the LoadBalancer
In this topology there will need to be a load balancers at each geographical location
to route traffic to the application servers.
See the [Load Balancer for GitLab HA](../../high_availability/load_balancer.md)
documentation for more information.
### Step 4: Configure the Geo Frontend Application Servers
In the architecture overview there are two machines running the GitLab application
services. These services are enabled selectively in the configuration. Additionally
the addresses of the remote endpoints for PostgreSQL and Redis will need to be specified.
#### On the GitLab Primary Frontend servers
1. Edit `/etc/gitlab/gitlab.rb` and ensure the following to disable PostgreSQL and Redis from running locally.
```ruby
##
## Disable PostgreSQL on the local machine and connect to the remote
##
postgresql['enable'] = false
gitlab_rails['auto_migrate'] = false
gitlab_rails['db_host'] = '10.0.3.1'
gitlab_rails['db_password'] = 'DB password'
##
## Disable Redis on the local machine and connect to the remote
##
redis['enable'] = false
gitlab_rails['redis_host'] = '10.0.2.1'
gitlab_rails['redis_password'] = 'Redis password'
geo_primary_role['enable'] = true
```
#### On the GitLab Secondary Frontend servers
On the secondary the remote endpoint for the PostgreSQL Geo database will
be specified.
1. Edit `/etc/gitlab/gitlab.rb` and ensure the following to disable PostgreSQL and Redis from running locally. Configure the secondary to connect to the Geo tracking database.
```ruby
##
## Disable PostgreSQL on the local machine and connect to the remote
##
postgresql['enable'] = false
gitlab_rails['auto_migrate'] = false
gitlab_rails['db_host'] = '10.1.3.1'
gitlab_rails['db_password'] = 'DB password'
##
## Disable Redis on the local machine and connect to the remote
##
redis['enable'] = false
gitlab_rails['redis_host'] = '10.1.2.1'
gitlab_rails['redis_password'] = 'Redis password'
##
## Enable the geo secondary role and configure the
## geo tracking database
##
geo_secondary_role['enable'] = true
geo_secondary['db_host'] = '10.1.4.1'
geo_secondary['db_password'] = 'Geo tracking DB password'
geo_postgresql['enable'] = false
```
After making these changes [Reconfigure GitLab][] so that they take effect.
On the primary the following GitLab frontend services will be enabled:
* gitlab-pages
* gitlab-workhorse
* logrotate
* nginx
* registry
* remote-syslog
* sidekiq
* unicorn
On the secondary the following GitLab frontend services will be enabled:
* geo-logcursor
* gitlab-pages
* gitlab-workhorse
* logrotate
* nginx
* registry
* remote-syslog
* sidekiq
* unicorn
Verify these services by running `sudo gitlab-ctl status` on the frontend
application servers.
[reconfigure GitLab]: ../../restart_gitlab.md#omnibus-gitlab-reconfigure
[restart GitLab]: ../../restart_gitlab.md#omnibus-gitlab-restart
This diff is collapsed.
# Geo with Object storage
Geo can be used in combination with Object Storage (AWS S3, or
other compatible object storage).
## Configuration
At this time it is required that if object storage is enabled on the
primary, it must also be enabled on the secondary.
The secondary nodes can use the same storage bucket as the primary, or
they can use a replicated storage bucket. At this time GitLab does not
take care of content replication in object storage.
For LFS, follow the documentation to
[set up LFS object storage](../../../workflow/lfs/lfs_administration.md#setting-up-s3-compatible-object-storage).
For CI job artifacts, there is similar documentation to configure
[jobs artifact object storage](../../job_artifacts.md#using-object-storage)
Complete these steps on all nodes, primary **and** secondary.
## Replication
When using Amazon S3, you can use
[CRR](https://docs.aws.amazon.com/AmazonS3/latest/dev/crr.html) to
have automatic replication between the bucket used by the primary and
the bucket used by the secondary.
If you are using Google Cloud Storage, consider using
[Multi-Regional Storage](https://cloud.google.com/storage/docs/storage-classes#multi-regional).
Or you can use the [Storage Transfer Service](https://cloud.google.com/storage/transfer/),
although this only supports daily synchronization.
For manual synchronization, or scheduled by `cron`, please have a look at:
- [`s3cmd sync`](http://s3tools.org/s3cmd-sync)
- [`gsutil rsync`](https://cloud.google.com/storage/docs/gsutil/commands/rsync)
This diff is collapsed.
# Geo Troubleshooting
>**Note:**
This list is an attempt to document all the moving parts that can go wrong.
We are working into getting all this steps verified automatically in a
rake task in the future.
Setting up Geo requires careful attention to details and sometimes it's easy to
miss a step. Here is a list of questions you should ask to try to detect
what you need to fix (all commands and path locations are for Omnibus installs):
#### First check the health of the secondary
Visit the primary node's **Admin Area ➔ Geo Nodes** (`/admin/geo_nodes`) in
your browser. We perform the following health checks on each secondary node
to help identify if something is wrong:
- Is the node running?
- Is the node's secondary database configured for streaming replication?
- Is the node's secondary tracking database configured?
- Is the node's secondary tracking database connected?
- Is the node's secondary tracking database up-to-date?
![Geo health check](img/geo_node_healthcheck.png)
There is also an option to check the status of the secondary node by running a special rake task:
```
sudo gitlab-rake geo:status
```
#### Is Postgres replication working?
#### Are my nodes pointing to the correct database instance?
You should make sure your primary Geo node points to the instance with
writing permissions.
Any secondary nodes should point only to read-only instances.
#### Can Geo detect my current node correctly?
Geo uses the defined node from the `Admin ➔ Geo` screen, and tries to match
it with the value defined in the `/etc/gitlab/gitlab.rb` configuration file.
The relevant line looks like: `external_url "http://gitlab.example.com"`.
To check if the node on the current machine is correctly detected type:
```bash
sudo gitlab-rails runner "puts Gitlab::Geo.current_node.inspect"
```
and expect something like:
```
#<GeoNode id: 2, schema: "https", host: "gitlab.example.com", port: 443, relative_url_root: "", primary: false, ...>
```
By running the command above, `primary` should be `true` when executed in
the primary node, and `false` on any secondary.
#### How do I fix the message, "ERROR: replication slots can only be used if max_replication_slots > 0"?
This means that the `max_replication_slots` PostgreSQL variable needs to
be set on the primary database. In GitLab 9.4, we have made this setting
default to 1. You may need to increase this value if you have more Geo
secondary nodes. Be sure to restart PostgreSQL for this to take
effect. See the [PostgreSQL replication
setup](database.md#postgresql-replication) guide for more details.
#### How do I fix the message, "FATAL: could not start WAL streaming: ERROR: replication slot "geo_secondary_my_domain_com" does not exist"?
This occurs when PostgreSQL does not have a replication slot for the
secondary by that name. You may want to rerun the [replication
process](database.md) on the secondary.
#### How do I fix the message, "Command exceeded allowed execution time" when setting up replication?
This may happen while [initiating the replication process](database.md#step-4-initiate-the-replication-process) on the Geo secondary, and indicates that your
initial dataset is too large to be replicated in the default timeout (30 minutes).
Re-run `gitlab-ctl replicate-geo-database`, but include a larger value for
`--backup-timeout`:
```bash
sudo gitlab-ctl replicate-geo-database --host=primary.geo.example.com --slot-name=secondary_geo_example_com --backup-timeout=21600
```
This will give the initial replication up to six hours to complete, rather than
the default thirty minutes. Adjust as required for your installation.
#### How do I fix the message, "PANIC: could not write to file 'pg_xlog/xlogtemp.123': No space left on device"
Determine if you have any unused replication slots in the primary database. This can cause large amounts of log data to build up in `pg_xlog`.
Removing the unused slots can reduce the amount of space used in the `pg_xlog`.
1. Start a PostgreSQL console session:
```bash
sudo gitlab-psql gitlabhq_production
```
Note that using `gitlab-rails dbconsole` will not work, because managing replication slots requires superuser permissions.
2. View your replication slots with
```sql
SELECT * FROM pg_replication_slots;
```
Slots where `active` is `f` are not active.
- When this slot should be active, because you have a secondary configured using that slot,
log in to that secondary and check the PostgreSQL logs why the replication is not running.
- If you are no longer using the slot (e.g. you no longer have Geo enabled), you can remove it with in the PostgreSQL console session:
```sql
SELECT pg_drop_replication_slot('name_of_extra_slot');
```
#### Very large repositories never successfully synchronize on the secondary
GitLab places a timeout on all repository clones, including project imports
and Geo synchronization operations. If a fresh `git clone` of a repository
on the primary takes more than a few minutes, you may be affected by this.
To increase the timeout, add the following line to `/etc/gitlab/gitlab.rb`
on the secondary:
```ruby
gitlab_rails['gitlab_shell_git_timeout'] = 10800
```
Then reconfigure GitLab:
```bash
sudo gitlab-ctl reconfigure
```
This will increase the timeout to three hours (10800 seconds). Choose a time
long enough to accomodate a full clone of your largest repositories.
# Tuning Geo
## Changing the sync capacity values
In the Geo admin page (`/admin/geo_nodes`), there are several variables that
can be tuned to improve performance of Geo:
* Repository sync capacity
* File sync capacity
Increasing these values will increase the number of jobs that are scheduled,
but this may not lead to a more downloads in parallel unless the number of
available Sidekiq threads is also increased. For example, if repository sync
capacity is increased from 25 to 50, you may also want to increase the number
of Sidekiq threads from 25 to 50. See the [Sidekiq concurrency
documentation](../../operations/extra_sidekiq_processes.html#concurrency)
for more details.
This diff is collapsed.
[//]: # (Please update EE::GitLab::GeoGitAccess::GEO_SERVER_DOCS_URL if this file is moved)
# Using a Geo Server
After you set up the [database replication and configure the Geo nodes][req],
there are a few things to consider:
1. Users need an extra step to be able to fetch code from the secondary and push
to primary:
1. Clone the repository as you would normally do, but from the secondary node:
```bash
git clone git@secondary.gitlab.example.com:user/repo.git
```
1. Change the remote push URL to always push to primary, following this example:
```bash
git remote set-url --push origin git@primary.gitlab.example.com:user/repo.git
```
[req]: index.md#setup-instructions
......@@ -20,7 +20,8 @@ Learn how to install, configure, update, and maintain your GitLab instance.
- **(Starter/Premium)** [Omnibus support for log forwarding](https://docs.gitlab.com/omnibus/settings/logs.html#udp-log-shipping-gitlab-enterprise-edition-only)
- [High Availability](high_availability/README.md): Configure multiple servers for scaling or high availability.
- [High Availability on AWS](../university/high-availability/aws/README.md): Set up GitLab HA on Amazon AWS.
- **(Premium)** [GitLab Geo](../gitlab-geo/README.md): Replicate your GitLab instance to other geographical locations as a read-only fully operational version.
- **(Premium)** [Geo](geo/replication/index.md): Replicate your GitLab instance to other geographical locations as a read-only fully operational version.
- **(Premium)** [Disaster Recovery](geo/disaster_recovery/index.md): Quickly fail-over to a different site with minimal effort in a disaster situation.
- **(Premium)** [Pivotal Tile](../install/pivotal/index.md): Deploy GitLab as a pre-configured appliance using Ops Manager (BOSH) for Pivotal Cloud Foundry.
### Configuring GitLab
......
......@@ -25,14 +25,14 @@ instructions will break installations using older versions of OpenSSH, such as
those included with CentOS 6 as of September 2017. If you want to use this
feature for CentOS 6, follow [the instructions on how to build and install a custom OpenSSH package](#compiling-a-custom-version-of-openssh-for-centos-6) before continuing.
## Fast lookup is required for GitLab Geo
## Fast lookup is required for Geo
By default, GitLab manages an `authorized_keys` file, which contains all the
public SSH keys for users allowed to access GitLab. However, to maintain a
single source of truth, [Geo](../../gitlab-geo/README.md) needs to be configured to perform SSH fingerprint
lookups via database lookup.
As part of [setting up GitLab Geo](../../gitlab-geo/README.md#setup-instructions),
As part of [setting up Geo](../../gitlab-geo/README.md#setup-instructions),
you will be required to follow the steps outlined below for both the primary and
secondary nodes, but note that the `Write to "authorized keys" file` checkbox
only needs to be unchecked on the primary node since it will be reflected
......
......@@ -29,7 +29,7 @@ Any change in the URL will need to be reflected on disk (when groups / users or
projects are renamed). This can add a lot of load in big installations,
especially if using any type of network based filesystem.
For GitLab Geo in particular: Geo does work with legacy storage, but in some
For Geo in particular: Geo does work with legacy storage, but in some
edge cases due to race conditions it can lead to errors when a project is
renamed multiple times in short succession, or a project is deleted and
recreated under the same name very quickly. We expect these race events to be
......
# GitLab Geo
# Geo
Geo feature requires that we orchestrate a lot of components together.
For the Database we need to setup a streaming replication. Any operation on disk
......
This diff is collapsed.
This document was moved to [another location](using_a_geo_server.md).
This document was moved to [another location](../administration/geo/replication/using_a_geo_server.md).
## Bring a demoted primary back online
After a failover, it is possible to fail back to the demoted primary to restore your original configuration.
This process consists of two steps: making old primary a secondary and promoting secondary to a primary.
### Configure the former primary to be a secondary
Since the former primary will be out of sync with the current primary, the first
step is to bring the former primary up to date. There is one downside though, some uploads and repositories
that have been deleted during an idle period of a primary node, will not be deleted from the disk but the overall sync will be much faster. As an alternative, you can set up a [GitLab instance from scratch](https://docs.gitlab.com/ee/gitlab-geo/#setup-instructions) to workaround this downside.
1. SSH into the former primary that has fallen behind.
1. Make sure all the services are up by running the command
```bash
sudo gitlab-ctl start
```
Note: If you [disabled primary permanently](https://docs.gitlab.com/ee/gitlab-geo/disaster-recovery.html#step-2-permanently-disable-the-primary), you need to undo those steps now. For Debian/Ubuntu you just need to run `sudo systemctl enable gitlab-runsvdir`. For CentoOS 6, you need to install GitLab instance from scratch and setup it as a secondary node by following [Setup instructions](https://docs.gitlab.com/ee/gitlab-geo/#setup-instructions). In this case you don't need the step below.
1. [Setup the database replication](database.md). In this documentation, primary
refers to the current primary, and secondary refers to the former primary.
If you have lost your original primary, follow the
[setup instructions](README.md#setup-instructions) to set up a new secondary.
### Promote the secondary to primary
When initial replication is complete and the primary and secondary are closely in sync you can do a [Planned Failover](planned-failover.md)
### Restore the secondary node
If your objective is to have two nodes again, you need to bring your secondary node back online as well by repeating the first step ([Make primary a secondary](#make-primary-a-secondary)) for the secondary node.
This document was moved to [another location](../administration/geo/disaster_recovery/bring_primary_back.md).
# GitLab Geo configuration
>**Note:**
This is the documentation for the Omnibus GitLab packages. For installations
from source, follow the [**GitLab Geo nodes configuration for installations
from source**](configuration_source.md) guide.
## Configuring a new secondary node
>**Note:**
This is the final step in setting up a secondary Geo node. Stages of the
setup process must be completed in the documented order.
Before attempting the steps in this stage, [complete all prior stages](README.md#using-omnibus-gitlab).
The basic steps of configuring a secondary node are to replicate required
configurations between the primary and the secondaries; to configure a tracking
database on each secondary; and to start GitLab on the secondary node.
You are encouraged to first read through all the steps before executing them
in your testing/production environment.
>**Notes:**
- **Do not** setup any custom authentication in the secondary nodes, this will be
handled by the primary node.
- **Do not** add anything in the secondaries Geo nodes admin area
(**Admin Area ➔ Geo Nodes**). This is handled solely by the primary node.
### Step 1. Manually replicate secret GitLab values
GitLab stores a number of secret values in the `/etc/gitlab/gitlab-secrets.json`
file which *must* match between the primary and secondary nodes. Until there is
a means of automatically replicating these between nodes (see
[issue #3789](https://gitlab.com/gitlab-org/gitlab-ee/issues/3789)), they must
be manually replicated to the secondary.
1. SSH into the **primary** node, and execute the command below:
```bash
sudo cat /etc/gitlab/gitlab-secrets.json
```
This will display the secrets that need to be replicated, in JSON format.
1. SSH into the **secondary** node and login as the `root` user:
```
sudo -i
```
1. Make a backup of any existing secrets:
```bash
mv /etc/gitlab/gitlab-secrets.json /etc/gitlab/gitlab-secrets.json.`date +%F`
```
1. Copy `/etc/gitlab/gitlab-secrets.json` from the primary to the secondary, or
copy-and-paste the file contents between nodes:
```bash
sudo editor /etc/gitlab/gitlab-secrets.json
# paste the output of the `cat` command you ran on the primary
# save and exit
```
1. Ensure the file permissions are correct:
```bash
chown root:root /etc/gitlab/gitlab-secrets.json
chmod 0600 /etc/gitlab/gitlab-secrets.json
```
1. Reconfigure the secondary node for the change to take effect:
```
gitlab-ctl reconfigure
```
Once reconfigured, the secondary will automatically start
replicating missing data from the primary in a process known as backfill.
Meanwhile, the primary node will start to notify the secondary of any changes, so
that the secondary can act on those notifications immediately.
Make sure the secondary instance is
running and accessible. You can login to the secondary node
with the same credentials as used in the primary.
### Step 2. Manually replicate primary SSH host keys
GitLab integrates with the system-installed SSH daemon, designating a user
(typically named git) through which all access requests are handled.
In a [Disaster Recovery](disaster-recovery.md) situation, GitLab system
administrators will promote a secondary Geo replica to a primary and they can
update the DNS records for the primary domain to point to the secondary to prevent
the need to update all references to the primary domain to the secondary domain,
like changing Git remotes and API URLs.
This will cause all SSH requests to the newly promoted primary node from
failing due to SSH host key mismatch. To prevent this, the primary SSH host
keys must be manually replicated to the secondary node.
1. SSH into the **secondary** node and login as the `root` user:
```
sudo -i
```
1. Make a backup of any existing SSH host keys:
```bash
find /etc/ssh -iname ssh_host_* -exec cp {} {}.backup.`date +%F` \;
```
1. SSH into the **primary** node, and execute the command below:
```bash
sudo find /etc/ssh -iname ssh_host_* -not -iname '*.pub'
```
1. For each file in that list replace the file from the primary node to
the **same** location on your **secondary** node.
1. On your **secondary** node, ensure the file permissions are correct:
```bash
chown root:root /etc/ssh/ssh_host_*
chmod 0600 /etc/ssh/ssh_host_*
```
1. Regenerate the public keys from the private keys:
```bash
find /etc/ssh -iname ssh_host_* -not -iname '*.backup*' -exec sh -c 'ssh-keygen -y -f "{}" > "{}.pub"' \;
```
1. Restart sshd:
```bash
service ssh restart
```
### Step 3. (Optional) Enabling hashed storage (from GitLab 10.0)
>**Warning**
Hashed storage is in **Beta**. It is not considered production-ready. See
[Hashed Storage](../administration/repository_storage_types.md) for more detail,
and for the latest updates, check
[infrastructure issue #2821](https://gitlab.com/gitlab-com/infrastructure/issues/2821).
Using hashed storage significantly improves Geo replication - project and group
renames no longer require synchronization between nodes.
1. Visit the **primary** node's **Admin Area ➔ Settings**
(`/admin/application_settings`) in your browser
1. In the `Repository Storages` section, check `Create new projects using hashed storage paths`:
![](img/hashed-storage.png)
### Step 4. (Optional) Configuring the secondary to trust the primary
You can safely skip this step if your primary uses a CA-issued HTTPS certificate.
If your primary is using a self-signed certificate for *HTTPS* support, you will
need to add that certificate to the secondary's trust store. Retrieve the
certificate from the primary and follow
[these instructions](https://docs.gitlab.com/omnibus/settings/ssl.html)
on the secondary.
### Step 5. Enable Git access over HTTP/HTTPS
GitLab Geo synchronizes repositories over HTTP/HTTPS, and therefore requires this clone
method to be enabled. Navigate to **Admin Area ➔ Settings**
(`/admin/application_settings`) on the primary node, and set
`Enabled Git access protocols` to `Both SSH and HTTP(S)` or `Only HTTP(S)`.
### Step 6. Verify proper functioning of the secondary node
Congratulations! Your secondary geo node is now configured!
You can login to the secondary node with the same credentials you used on the
primary. Visit the secondary node's **Admin Area ➔ Geo Nodes**
(`/admin/geo_nodes`) in your browser to check if it's correctly identified as a
secondary Geo node and if Geo is enabled.
The initial replication, or 'backfill', will probably still be in progress. You
can monitor the synchronization process on each geo node from the primary
node's Geo Nodes dashboard in your browser.
![GitLab Geo dashboard](img/geo-node-dashboard.png)
If your installation isn't working properly, check the
[troubleshooting document](troubleshooting.md).
The two most obvious issues that can become apparent in the dashboard are:
1. Database replication not working well
1. Instance to instance notification not working. In that case, it can be
something of the following:
- You are using a custom certificate or custom CA (see the
[troubleshooting document](troubleshooting.md))
- The instance is firewalled (check your firewall rules)
Please note that disabling a secondary node will stop the sync process.
Please note that if `git_data_dirs` is customized on the primary for multiple
repository shards you must duplicate the same configuration on the secondary.
Point your users to the ["Using a Geo Server" guide](using_a_geo_server.md).
Currently, this is what is synced:
* Git repositories
* Wikis
* LFS objects
* Issues, merge requests, snippets, and comment attachments
* Users, groups, and project avatars
## Selective synchronization
GitLab Geo supports selective synchronization, which allows admins to choose
which projects should be synchronized by secondary nodes.
It is important to note that selective synchronization does not:
1. Restrict permissions from secondary nodes.
1. Hide project metadata from secondary nodes.
* Since Geo currently relies on PostgreSQL replication, all project metadata
gets replicated to secondary nodes, but repositories that have not been
selected will be empty.
1. Reduce the number of events generated for the Geo event log
* The primary generates events as long as any secondaries are present.
Selective synchronization restrictions are implemented on the secondaries,
not the primary.
A subset of projects can be chosen, either by group or by storage shard. The
former is ideal for replicating data belonging to a subset of users, while the
latter is more suited to progressively rolling out Geo to a large GitLab
instance.
## Upgrading Geo
See the [updating the Geo nodes document](updating_the_geo_nodes.md).
## Troubleshooting
See the [troubleshooting document](troubleshooting.md).
This document was moved to [another location](../administration/geo/replication/configuration.md).
# GitLab Geo configuration
>**Note:**
This is the documentation for installations from source. For installations
using the Omnibus GitLab packages, follow the
[**Omnibus GitLab Geo nodes configuration**](configuration.md) guide.
## Configuring a new secondary node
>**Note:**
This is the final step in setting up a secondary Geo node. Stages of the setup
process must be completed in the documented order. Before attempting the steps
in this stage, [complete all prior stages](README.md#using-gitlab-installed-from-source).
The basic steps of configuring a secondary node are to replicate required
configurations between the primary and the secondaries; to configure a tracking
database on each secondary; and to start GitLab on the secondary node.
You are encouraged to first read through all the steps before executing them
in your testing/production environment.
>**Notes:**
- **Do not** setup any custom authentication in the secondary nodes, this will be
handled by the primary node.
- **Do not** add anything in the secondaries Geo nodes admin area
(**Admin Area ➔ Geo Nodes**). This is handled solely by the primary node.
### Step 1. Manually replicate secret GitLab values
GitLab stores a number of secret values in the `/home/git/gitlab/config/secrets.yml`
file which *must* match between the primary and secondary nodes. Until there is
a means of automatically replicating these between nodes (see
[issue #3789](https://gitlab.com/gitlab-org/gitlab-ee/issues/3789)), they must
be manually replicated to the secondary.
1. SSH into the **primary** node, and execute the command below:
```bash
sudo cat /home/git/gitlab/config/secrets.yml
```
This will display the secrets that need to be replicated, in YAML format.
1. SSH into the **secondary** node and login as the `git` user:
```bash
sudo -i -u git
```
1. Make a backup of any existing secrets:
```bash
mv /home/git/gitlab/config/secrets.yml /home/git/gitlab/config/secrets.yml.`date +%F`
```
1. Copy `/home/git/gitlab/config/secrets.yml` from the primary to the secondary, or
copy-and-paste the file contents between nodes:
```bash
sudo editor /home/git/gitlab/config/secrets.yml
# paste the output of the `cat` command you ran on the primary
# save and exit
```
1. Ensure the file permissions are correct:
```bash
chown git:git /home/git/gitlab/config/secrets.yml
chmod 0600 /home/git/gitlab/config/secrets.yml
```
1. Restart GitLab for the changes to take effect:
```bash
service gitlab restart
```
Once restarted, the secondary will automatically start replicating missing data
from the primary in a process known as backfill. Meanwhile, the primary node
will start to notify the secondary of any changes, so that the secondary can
act on those notifications immediately.
Make sure the secondary instance is running and accessible. You can login to
the secondary node with the same credentials as used in the primary.
### Step 2. Manually replicate primary SSH host keys
Read [Manually replicate primary SSH host keys](configuration.md#step-2-manually-replicate-primary-ssh-host-keys)
### Step 3. (Optional) Enabling hashed storage (from GitLab 10.0)
Read [Enabling Hashed Storage](configuration.md#step-3-optional-enabling-hashed-storage-from-gitlab-10-0)
### Step 4. (Optional) Configuring the secondary to trust the primary
You can safely skip this step if your primary uses a CA-issued HTTPS certificate.
If your primary is using a self-signed certificate for *HTTPS* support, you will
need to add that certificate to the secondary's trust store. Retrieve the
certificate from the primary and follow your distribution's instructions for
adding it to the secondary's trust store. In Debian/Ubuntu, for example, with a
certificate file of `primary.geo.example.com.crt`, you would follow these steps:
```
sudo -i
cp primary.geo.example.com.crt /usr/local/share/ca-certificates
update-ca-certificates
```
### Step 5. Enable Git access over HTTP/HTTPS
GitLab Geo synchronizes repositories over HTTP/HTTPS, and therefore requires this clone
method to be enabled. Navigate to **Admin Area ➔ Settings**
(`/admin/application_settings`) on the primary node, and set
`Enabled Git access protocols` to `Both SSH and HTTP(S)` or `Only HTTP(S)`.
### Step 6. Verify proper functioning of the secondary node
Read [Verify proper functioning of the secondary node](configuration.md#step-6-verify-proper-functioning-of-the-secondary-node).
## Selective synchronization
Read [Selective synchronization](configuration.md#selective-synchronization).
## Troubleshooting
Read the [troubleshooting document](troubleshooting.md).
This document was moved to [another location](../administration/geo/replication/configuration_source.md).
This diff is collapsed.
This diff is collapsed.
# GitLab Geo Disaster Recovery
> **Note:** Disaster Recovery is in **Alpha** development. Do not use this as
> your only Disaster Recovery strategy as you may lose data.
GitLab Geo replicates your database and your Git repositories. We will
support and replicate more data in the future, that will enable you to
fail-over with minimal effort, in a disaster situation.
See [current limitations](README.md#current-limitations) for more information.
## Promoting secondary Geo replica in single-secondary configuration
We don't currently provide an automated way to promote a Geo replica and do a
fail-over, but you can do it manually if you have `root` access to the machine.
This process promotes a secondary Geo replica to a primary. To regain
geographical redundancy as quickly as possible, you should add a new secondary
immediately after following these instructions.
### Step 1. Allow replication to finish if possible
If the secondary is still replicating data from the primary, follow
[the Planned Failover doc](planned-failover.md) as closely as possible in
order to avoid unnecessary data loss.
### Step 2. Permanently disable the primary
**Warning: If a primary goes offline, there may be data saved on the primary
that has not been replicated to the secondary. This data should be treated
as lost if you proceed.**
If an outage on your primary happens, you should do everything possible to
avoid a split-brain situation where writes can occur to two different GitLab
instances, complicating recovery efforts. So to prepare for the failover, we
must disable the primary.
1. SSH into your **primary** to stop and disable GitLab, if possible.
```bash
sudo gitlab-ctl stop
```
Prevent GitLab from starting up again if the server unexpectedly reboots:
```bash
sudo systemctl disable gitlab-runsvdir
```
On some operating systems such as CentOS 6, an easy way to prevent GitLab
from being started if the machine reboots isn't available
(see [Omnibus issue #3058](https://gitlab.com/gitlab-org/omnibus-gitlab/issues/3058)).
It may be safest to uninstall the GitLab package completely:
```bash
yum remove gitlab-ee
```
1. If you do not have SSH access to your primary, take the machine offline and
prevent it from rebooting by any means at your disposal.
Since there are many ways you may prefer to accomplish this, we will avoid a
single recommendation. You may need to:
* Reconfigure load balancers
* Change DNS records (e.g. point the primary DNS record to the secondary node in order to stop usage of the primary)
* Stop virtual servers
* Block traffic through a firewall
* Revoke object storage permissions from the primary
* Physically disconnect a machine
### Step 3. Promoting a secondary Geo replica
1. SSH in to your **secondary** and login as root:
```bash
sudo -i
```
1. Edit `/etc/gitlab/gitlab.rb` to reflect its new status as primary.
Remove the following line:
```ruby
## REMOVE THIS LINE
geo_secondary_role['enable'] = true
```
A new secondary should not be added at this time. If you want to add a new
secondary, do this after you have completed the entire process of promoting
the secondary to the primary.
1. Promote the secondary to primary. Execute:
```bash
gitlab-ctl promote-to-primary-node
```
1. Verify you can connect to the newly promoted primary using the URL used
previously for the secondary.
1. Success! The secondary has now been promoted to primary.
### Step 4. (Optional) Updating the primary domain's DNS record
Updating the DNS records for the primary domain to point to the secondary
will prevent the need to update all references to the primary domain to the
secondary domain, like changing Git remotes and API URLs.
1. SSH in to your **secondary** and login as root:
```bash
sudo -i
```
1. Update the primary domain's DNS record.
After updating the primary domain's DNS records to point to the secondary,
edit `/etc/gitlab/gitlab.rb` on the the secondary to reflect the new URL:
```ruby
# Change the existing external_url configuration
external_url 'https://gitlab.example.com'
```
1. Reconfigure the secondary node for the change to take effect:
```bash
gitlab-ctl reconfigure
```
1. Execute the command below to update the newly promoted primary node URL:
```bash
gitlab-rake geo:update_primary_node_url
```
This command will use the changed `external_url` configuration defined
in `/etc/gitlab/gitlab.rb`.
1. Verify you can connect to the newly promoted primary using the primary URL.
If you updated the DNS records for the primary domain, these changes may
not have yet propagated depending on the previous DNS records TTL.
### Step 5. (Optional) Add secondary Geo replicas to a promoted primary
Promoting a secondary to primary using the process above does not enable
GitLab Geo on the new primary.
To bring a new secondary online, follow the [GitLab Geo setup instructions](
README.md#setup-instructions).
## Promoting secondary Geo replica in multi-secondary configurations
Disaster Recovery does not yet support systems with multiple
secondary Geo replicas (e.g. one primary and two or more secondaries). We are
working on it, see [#4284](https://gitlab.com/gitlab-org/gitlab-ee/issues/4284)
for details.
This document was moved to [another location](../administration/geo/disaster_recovery/index.md).
# Docker Registry for a secondary node
You can setup a [Docker Registry](https://docs.docker.com/registry/) on your
secondary Geo node that mirrors the one on the primary Geo node.
## Storage support
CAUTION: **Warning:**
If you use [local storage](../administration/container_registry.md#container-registry-storage-driver)
for the Container Registry you **cannot** replicate it to the secondary Geo node.
Docker Registry currently supports a few types of storages. If you choose a
distributed storage (`azure`, `gcs`, `s3`, `swift`, or `oss`) for your Docker
Registry on a primary Geo node, you can use the same storage for a secondary
Docker Registry as well. For more information, read the
[Load balancing considerations](https://docs.docker.com/registry/deploying/#load-balancing-considerations)
when deploying the Registry, and how to setup the storage driver for GitLab's
integrated [Container Registry](../administration/container_registry.md#container-registry-storage-driver).
[ee]: https://about.gitlab.com/products/
This document was moved to [another location](../administration/geo/replication/docker_registry.md).
# Geo Frequently Asked Questions
## Can I use Geo in a disaster recovery situation?
There are limitations to what we replicate (see
[What data is replicated to a secondary node?](#what-data-is-replicated-to-a-secondary-node)).
In an extreme data-loss situation you can make a secondary Geo into your
primary, but this is not officially supported yet.
If you still want to proceed, see our step-by-step instructions on how to
manually [promote a secondary node](disaster-recovery.md) into primary.
## I followed the disaster recovery instructions and now two-factor auth is broken!
The setup instructions for GitLab Geo prior to 10.5 failed to replicate the
`otp_key_base` secret, which used to encrypt the two-factor authentication
secrets stored in the database. If it differs between primary and secondary
nodes, users with two-factor authentication enabled won't be able to log in
after a DR failover.
If you still have access to the old primary node, you can follow the
instructions in the [Upgrading to GitLab 10.5](updating_the_geo_nodes.md#upgrading-to-gitlab-105)
section to resolve the error. Otherwise, the secret is lost and you'll need to
[reset two-factor authentication for all users](../security/two_factor_authentication.md#disabling-2fa-for-everyone).
## What data is replicated to a secondary node?
We currently replicate project repositories, LFS objects, generated
attachments / avatars and the whole database. This means user accounts,
issues, merge requests, groups, project data, etc., will be available for
query. We currently don't replicate artifact data (`shared/folder`).
## Can I git push to a secondary node?
No. All writing operations (this includes `git push`) must be done in your
primary node.
## How long does it take to have a commit replicated to a secondary node?
All replication operations are asynchronous and are queued to be dispatched in
a batched request every 10 minutes. Besides that, it depends on a lot of other
factors including the amount of traffic, how big your commit is, the
connectivity between your nodes, your hardware, etc.
## What if the SSH server runs at a different port?
We send the clone url from the primary server to any secondaries, so it
doesn't matter. If primary is running on port `2200`, clone url will reflect
that.
## Is this possible to set up a Docker Registry for a secondary node that mirrors the one on a primary node?
Yes. See [Docker Registry for a secondary Geo node](docker_registry.md).
This document was moved to [another location](../administration/geo/replication/faq.md).
# GitLab Geo High Availability
This document describes a minimal reference architecture for running GitLab Geo
in a high availability configuration. If your HA setup differs from the one
described, it is possible to adapt these instructions to your needs.
## Architecture overview
![Geo HA Diagram](../administration/img/high_availability/geo-ha-diagram.png)
_[diagram source - gitlab employees only](https://docs.google.com/drawings/d/1z0VlizKiLNXVVVaERFwgsIOuEgjcUqDTWPdQYsE7Z4c/edit)_
The topology above assumes that the primary and secondary Geo clusters
are located in two separate locations, on their own virtual network
with private IP addresses. The network is configured such that all machines within
one geographic location can communicate with each other using their private IP addresses.
The IP addresses given are examples and may be different depending on the
network topology of your deployment.
The only external way to access the two GitLab Geo deployments is by HTTPS at
`gitlab.us.example.com` and `gitlab.eu.example.com` in the example above.
NOTE: **Note:**
The primary and secondary GitLab Geo deployments must be able to communicate to each other over HTTPS.
## Redis and PostgreSQL High Availability
The primary and secondary Redis and PostgreSQL should be configured
for high availability. Because of the additional complexity involved
in setting up this configuration for PostgreSQL and Redis
it is not covered by this Geo HA documentation.
The two services will instead be configured such that
they will each run on a single machine.
For more information about setting up a highly available PostgreSQL cluster and Redis cluster using the omnibus package see the high availability documentation for
[PostgreSQL](../administration/high_availability/database.md) and
[Redis](../administration/high_availability/redis.md), respectively.
From these instructions you will need the following for the examples below:
* `gitlab_rails['db_password']` for the PostgreSQL "DB password"
* `redis['password']` for the Redis "Redis password"
NOTE: **Note:**
It is possible to use cloud hosted services for PostgreSQL and Redis but this is beyond the scope of this document.
### Prerequisites
Make sure you have GitLab EE installed using the
[Omnibus package](https://about.gitlab.com/installation).
### Step 1: Configure the Geo Backend Services
On the **primary** backend servers configure the following services:
* [Redis](../administration/high_availability/redis.md) for high availability.
* [NFS Server](../administration/high_availability/nfs.md) for repository, LFS, and upload storage.
* [PostgreSQL](../administration/high_availability/database.md) for high availability.
On the **secondary** backend servers configure the following services:
* [Redis](../administration/high_availability/redis.md) for high availability.
* [NFS Server](../administration/high_availability/nfs.md) which will store data that is synchronized from the Geo primary.
### Step 2: Configure the Postgres services on the Geo Secondary
1. Configure the [secondary Geo PostgreSQL database](../gitlab-geo/database.md)
as a read-only secondary of the primary Geo PostgreSQL database.
1. Configure the Geo tracking database on the secondary server, to do this modify `/etc/gitlab/gitlab.rb`:
```ruby
geo_postgresql['enable'] = true
geo_postgresql['listen_address'] = '10.1.4.1'
geo_postgresql['trust_auth_cidr_addresses'] = ['10.1.0.0/16']
geo_secondary['auto_migrate'] = true
geo_secondary['db_host'] = '10.1.4.1'
geo_secondary['db_password'] = 'Geo tracking DB password'
```
NOTE: **Note:**
Be sure that other non-postgresql services are disabled by setting `enable` to `false` in
the [gitlab.rb configuration](https://gitlab.com/gitlab-org/omnibus-gitlab/blob/master/files/gitlab-config-template/gitlab.rb.template).
After making these changes be sure to run `sudo gitlab-ctl reconfigure` so that they take effect.
### Step 3: Setup the LoadBalancer
In this topology there will need to be a load balancers at each geographical location
to route traffic to the application servers.
See the [Load Balancer for GitLab HA](../administration/high_availability/load_balancer.md)
documentation for more information.
### Step 4: Configure the Geo Frontend Application Servers
In the architecture overview there are two machines running the GitLab application
services. These services are enabled selectively in the configuration. Additionally
the addresses of the remote endpoints for PostgreSQL and Redis will need to be specified.
#### On the GitLab Primary Frontend servers
1. Edit `/etc/gitlab/gitlab.rb` and ensure the following to disable PostgreSQL and Redis from running locally.
```ruby
##
## Disable PostgreSQL on the local machine and connect to the remote
##
postgresql['enable'] = false
gitlab_rails['auto_migrate'] = false
gitlab_rails['db_host'] = '10.0.3.1'
gitlab_rails['db_password'] = 'DB password'
##
## Disable Redis on the local machine and connect to the remote
##
redis['enable'] = false
gitlab_rails['redis_host'] = '10.0.2.1'
gitlab_rails['redis_password'] = 'Redis password'
geo_primary_role['enable'] = true
```
#### On the GitLab Secondary Frontend servers
On the secondary the remote endpoint for the PostgreSQL Geo database will
be specified.
1. Edit `/etc/gitlab/gitlab.rb` and ensure the following to disable PostgreSQL and Redis from running locally. Configure the secondary to connect to the GitLab Geo tracking database.
```ruby
##
## Disable PostgreSQL on the local machine and connect to the remote
##
postgresql['enable'] = false
gitlab_rails['auto_migrate'] = false
gitlab_rails['db_host'] = '10.1.3.1'
gitlab_rails['db_password'] = 'DB password'
##
## Disable Redis on the local machine and connect to the remote
##
redis['enable'] = false
gitlab_rails['redis_host'] = '10.1.2.1'
gitlab_rails['redis_password'] = 'Redis password'
##
## Enable the geo secondary role and configure the
## geo tracking database
##
geo_secondary_role['enable'] = true
geo_secondary['db_host'] = '10.1.4.1'
geo_secondary['db_password'] = 'Geo tracking DB password'
geo_postgresql['enable'] = false
```
After making these changes [Reconfigure GitLab][] so that they take effect.
On the primary the following GitLab frontend services will be enabled:
* gitlab-pages
* gitlab-workhorse
* logrotate
* nginx
* registry
* remote-syslog
* sidekiq
* unicorn
On the secondary the following GitLab frontend services will be enabled:
* geo-logcursor
* gitlab-pages
* gitlab-workhorse
* logrotate
* nginx
* registry
* remote-syslog
* sidekiq
* unicorn
Verify these services by running `sudo gitlab-ctl status` on the frontend
application servers.
[reconfigure GitLab]: ../administration/restart_gitlab.md#omnibus-gitlab-reconfigure
[restart GitLab]: ../administration/restart_gitlab.md#omnibus-gitlab-restart
This document was moved to [another location](../administration/geo/replication/high_availability.md).
# GitLab Geo with Object storage
GitLab Geo can be used in combination with Object Storage (AWS S3, or
other compatible object storage).
## Configuration
At this time it is required that if object storage is enabled on the
primary, it must also be enabled on the secondary.
The secondary nodes can use the same storage bucket as the primary, or
they can use a replicated storage bucket. At this time GitLab does not
take care of content replication in object storage.
For LFS, follow the documentation to
[set up LFS object storage](../workflow/lfs/lfs_administration.md#setting-up-s3-compatible-object-storage).
For CI job artifacts, there is similar documentation to configure
[jobs artifact object storage](../administration/job_artifacts.md#using-object-storage)
Complete these steps on all nodes, primary **and** secondary.
## Replication
When using Amazon S3, you can use
[CRR](https://docs.aws.amazon.com/AmazonS3/latest/dev/crr.html) to
have automatic replication between the bucket used by the primary and
the bucket used by the secondary.
If you are using Google Cloud Storage, consider using
[Multi-Regional Storage](https://cloud.google.com/storage/docs/storage-classes#multi-regional).
Or you can use the [Storage Transfer Service](https://cloud.google.com/storage/transfer/),
although this only supports daily synchronization.
For manual synchronization, or scheduled by `cron`, please have a look at:
- [`s3cmd sync`](http://s3tools.org/s3cmd-sync)
- [`gsutil rsync`](https://cloud.google.com/storage/docs/gsutil/commands/rsync)
This document was moved to [another location](../administration/geo/replication/object_storage.md).
# GitLab Geo Planned Failover
A planned failover is similar to a disaster recovery scenario, except you are able
to notify users of the maintenance window, and allow data to finish replicating to
secondaries.
Please read this entire document as well as
[GitLab Geo Disaster Recovery](disaster-recovery.md) before proceeding.
### Notify users of scheduled maintenance
1. On the primary, in Admin Area > Messages, add a broadcast message.
Check Admin Area > Geo Nodes to estimate how long it will take to finish syncing.
```
We are doing scheduled maintenance at XX:XX UTC, expected to take less than 1 hour.
```
1. On the secondary, you may need to clear the cache for the broadcast message to show up.
### Block primary traffic
1. At the scheduled time, using your cloud provider or your node's firewall, block HTTP and SSH traffic to/from the primary except for your IP and the secondary's IP.
### Allow replication to finish as much as possible
1. On the secondary, navigate to Admin Area > Geo Nodes and wait until all replication progress is 100% on the secondary "Current node".
1. Navigate to Admin Area > Monitoring > Background Jobs > Queues and wait until the "geo" queues drop ideally to 0.
### Promote the secondary
1. Finally, follow [GitLab Geo Disaster Recovery](disaster-recovery.md) to promote the secondary to a primary.
This document was moved to [another location](../administration/geo/disaster_recovery/planned_failover.md).
This diff is collapsed.
This diff is collapsed.
# Tuning GitLab Geo
## Changing the sync capacity values
In the Geo admin page (`/admin/geo_nodes`), there are several variables that
can be tuned to improve performance of Geo:
* Repository sync capacity
* File sync capacity
Increasing these values will increase the number of jobs that are scheduled,
but this may not lead to a more downloads in parallel unless the number of
available Sidekiq threads is also increased. For example, if repository sync
capacity is increased from 25 to 50, you may also want to increase the number
of Sidekiq threads from 25 to 50. See the [Sidekiq concurrency
documentation](../administration/operations/extra_sidekiq_processes.html#concurrency)
for more details.
This document was moved to [another location](../administration/geo/replication/tuning.md).
This diff is collapsed.
[//]: # (Please update EE::GitLab::GeoGitAccess::GEO_SERVER_DOCS_URL if this file is moved)
# Using a Geo Server
After you set up the [database replication and configure the GitLab Geo nodes][req],
there are a few things to consider:
1. Users need an extra step to be able to fetch code from the secondary and push
to primary:
1. Clone the repository as you would normally do, but from the secondary node:
```bash
git clone git@secondary.gitlab.example.com:user/repo.git
```
1. Change the remote push URL to always push to primary, following this example:
```bash
git remote set-url --push origin git@primary.gitlab.example.com:user/repo.git
```
[req]: README.md#setup-instructions
This document was moved to [another location](../administration/geo/replication/using_a_geo_server.md).
This diff is collapsed.
This diff is collapsed.
Markdown is supported
0%
or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment