Commit 42654600 authored by Achilleas Pipinellis's avatar Achilleas Pipinellis

Merge branch 'tc-geo-ha-doc' into 'master'

Document Geo HA

Closes #3646

See merge request gitlab-org/gitlab-ee!3468
parents 8ad04d18 f9397c4d
---
title: Document how to set up GitLab Geo for HA
merge_request: 3468
author:
type: other
......@@ -181,6 +181,10 @@ Read through the [GitLab Geo configuration](configuration.md) documentation.
Read how to [update your Geo nodes to the latest GitLab version](updating_the_geo_nodes.md).
## Configuring GitLab Geo HA
Read through the [Geo High Availability documentation](ha.md).
## Current limitations
- You cannot push code to secondary nodes
......
......@@ -112,8 +112,10 @@ will not be able to perform all necessary configuration steps. Refer to
this example:
```bash
# Certificate and key currently used by GitLab
# - replace primary.geo.example.com with your domain
##
## Certificate and key currently used by GitLab
## - replace primary.geo.example.com with your domain
##
install -o gitlab-psql -g gitlab-psql -m 0400 -T /etc/gitlab/ssl/primary.geo.example.com.crt ~gitlab-psql/data/server.crt
install -o gitlab-psql -g gitlab-psql -m 0400 -T /etc/gitlab/ssl/primary.geo.example.com.key ~gitlab-psql/data/server.key
```
......@@ -134,8 +136,10 @@ will not be able to perform all necessary configuration steps. Refer to
to the correct location:
```
# Self-signed certificate and key
# - assumes the files are in your current working directory
##
## Self-signed certificate and key
## - assumes the files are in your current working directory
##
install -o gitlab-psql -g gitlab-psql -m 0400 -T server.crt ~gitlab-psql/data/server.crt
install -o gitlab-psql -g gitlab-psql -m 0400 -T server.key ~gitlab-psql/data/server.key
```
......@@ -166,10 +170,14 @@ will not be able to perform all necessary configuration steps. Refer to
To lookup the address of a Geo node, SSH in to the Geo node and execute:
```bash
# Private address
##
## Private address
##
ip route get 255.255.255.255 | awk '{print "Private address:", $NF; exit}'
# Public address
##
## Public address
##
echo "External address: $(curl ipinfo.io/ip)"
```
......@@ -199,23 +207,31 @@ will not be able to perform all necessary configuration steps. Refer to
```ruby
geo_primary_role['enable'] = true
# Primary address
# - replace '1.2.3.4' with the primary private address
##
## Primary address
## - replace '1.2.3.4' with the primary private address
##
postgresql['listen_address'] = '1.2.3.4'
postgresql['trust_auth_cidr_addresses'] = ['127.0.0.1/32','1.2.3.4/32']
##
# Secondary addresses
# - replace '5.6.7.8' with the secondary public address
##
postgresql['md5_auth_cidr_addresses'] = ['5.6.7.8/32']
# Replication settings
# - set this to be the number of Geo secondary nodes you have
##
## Replication settings
## - set this to be the number of Geo secondary nodes you have
##
postgresql['max_replication_slots'] = 1
# postgresql['max_wal_senders'] = 10
# postgresql['wal_keep_segments'] = 10
# Disable automatic database migrations temporarily
# (until PostgreSQL is restarted and listening on the private address)
##
## Disable automatic database migrations temporarily
## (until PostgreSQL is restarted and listening on the private address).
##
gitlab_rails['auto_migrate'] = false
```
......@@ -234,7 +250,7 @@ will not be able to perform all necessary configuration steps. Refer to
1. Save the file and reconfigure GitLab for the database listen changes and
the replication slot changes to be applied.
```bash
gitlab-ctl reconfigure
```
......@@ -322,13 +338,17 @@ because we have not yet configured the secondary server. This is the next step.
it in the right location.
```bash
# Certificate and key currently used by GitLab
##
## Certificate and key currently used by GitLab
##
mkdir -p ~gitlab-psql/.postgresql
ln -s /opt/gitlab/embedded/ssl/certs/cacert.pem ~gitlab-psql/.postgresql/root.crt
```
```bash
# Self-signed certificate and key
##
## Self-signed certificate and key
##
install -o gitlab-psql -g gitlab-psql -m 0400 -T server.crt -D ~gitlab-psql/.postgresql/root.crt
```
......@@ -336,15 +356,20 @@ because we have not yet configured the secondary server. This is the next step.
connections.
1. Test that the remote connection to the primary server works.
1. Test that the remote connection to the primary server works (as the
`gitlab-psql` user):
```
# Certificate and key currently used by GitLab, and connecting by FQDN
##
## Certificate and key currently used by GitLab, and connecting by FQDN
##
sudo -u gitlab-psql /opt/gitlab/embedded/bin/psql -U gitlab_replicator -d "dbname=gitlabhq_production sslmode=verify-full" -W -h primary.geo.example.com
```
```
# Self-signed certificate and key, or connecting by IP address
##
## Self-signed certificate and key, or connecting by IP address
##
sudo -u gitlab-psql /opt/gitlab/embedded/bin/psql -U gitlab_replicator -d "dbname=gitlabhq_production sslmode=verify-ca" -W -h 1.2.3.4
```
......@@ -417,13 +442,17 @@ data before running `pg_basebackup`.
1. Execute the command below to start a backup/restore and begin the replication:
```
# Certificate and key currently used by GitLab, and connecting by FQDN
##
## Certificate and key currently used by GitLab, and connecting by FQDN
##
gitlab-ctl replicate-geo-database --slot-name=secondary_example --host=primary.geo.example.com
```
```
# Self-signed certificate and key, or connecting by IP
gitlab-ctl replicate-geo-database --slot-name=secondary_example --sslmode=verify-ca --host=1.2.3.4
##
## Self-signed certificate and key, or connecting by IP
##
gitlab-ctl replicate-geo-database --sslmode=verify-ca --slot-name=secondary_example --host=1.2.3.4
```
If PostgreSQL is listening on a non-standard port, add `--port=` as well.
......@@ -477,14 +506,18 @@ The `geo_primary_role` makes configuration changes to `pg_hba.conf` and
`postgresql.conf` on the primary:
```
# pg_hba.conf
# GitLab Geo Primary
##
## GitLab Geo Primary
## - pg_hba.conf
##
host replication gitlab_replicator <trusted secondary IP>/32 md5
```
```
# postgresql.conf
# Geo Primary Role
##
## Geo Primary Role
## - postgresql.conf
##
sql_replication_user = gitlab_replicator
wal_level = hot_standby
max_wal_senders = 10
......@@ -499,8 +532,10 @@ on the secondary. The PostgreSQL settings for this database it adds to
the default settings:
```
# postgresql.conf
# Geo Secondary Role
##
## Geo Secondary Role
## - postgresql.conf
##
wal_level = hot_standby
max_wal_senders = 10
wal_keep_segments = 10
......
# GitLab Geo High Availability
This document describes a possible configuration on how to set up Geo
in a Highly Available environment. If your HA setup differs from the one
described in this document, you still can use the instructions and adapt them
to your needs.
## Architecture overview
![Active/Active HA Diagram](../administration/img/high_availability/active-active-diagram.png)
This documentation assumes all machines used in this HA setup can
communicate over the network using internal IP addresses.
NOTE: **Note:**
`external_url` must be the same for every machine, and `https` should be used.
## Services machine
One machine, called the Services machine will be used to run:
- NFS shares
- PostgreSQL
- Redis
- HAProxy
### Prerequisites
Make sure you have GitLab EE installed using the
[Omnibus package](https://about.gitlab.com/installation).
The following steps should be performed in the Services machine. SSH to it
and login as root:
```sh
sudo -i
```
### Step 1: Set up NFS share
1. Install the required NFS packages:
```sh
apt-get install nfs-kernel-server
```
1. Create the required directories:
```sh
mkdir -p /var/opt/gitlab/nfs/builds/ \
/var/opt/gitlab/nfs/git-data/ \
/var/opt/gitlab/nfs/shared/ \
/var/opt/gitlab/nfs/uploads/
```
1. Make the directories available through NFS, by adding this to
`/etc/exports` (see also the [NFS HA recommended options](../administration/high_availability/nfs.md#recommended-options)):
```
/var/opt/gitlab/nfs *(rw,sync,no_root_squash)
```
1. Start the NFS service:
```sh
systemctl start nfs-kernel-server.service
```
1. Apply the settings to take effect:
```sh
exportfs -a
```
### Step 2: Set up PostgreSQL server
1. Edit `/etc/gitlab/gitlab.rb` and add the following:
```ruby
postgresql['enable'] = true
##
## Replace 1.2.3.4 with the internal IP address of the current machine and
## 2.3.4.5 and 3.4.5.6 with the internal IP addresses of the machines
## running the Application server(s).
##
postgresql['listen_address'] = '1.2.3.4'
postgresql['trust_auth_cidr_addresses'] = ['1.2.3.4/32', '2.3.4.5/32', '3.4.5.6/32']
gitlab_rails['auto_migrate'] = true
gitlab_rails['db_password'] = 'DB password'
```
1. **Only for secondary nodes** Also add this to `/etc/gitlab/gitlab.rb`:
```ruby
geo_postgresql['enable'] = true
##
## Replace 1.2.3.4 with the internal IP address of the current machine and
## 2.3.4.5 and 3.4.5.6 with the internal IP addresses of the machines
## running the Application server(s).
##
geo_postgresql['listen_address'] = '1.2.3.4'
geo_postgresql['trust_auth_cidr_addresses'] = ['1.2.3.4/32', '2.3.4.5/32', '3.4.5.6/32']
geo_secondary['auto_migrate'] = true
geo_secondary['db_host'] = '1.2.3.4'
geo_secondary['db_password'] = 'Geo tracking DB password'
```
### Step 3: Set up Redis
Edit `/etc/gitlab/gitlab.rb` and add the following:
```ruby
redis['enable'] = true
redis['password'] = 'Redis password'
##
## Replace 1.2.3.4 with the internal IP address of the current machine
##
redis['bind'] = '1.2.3.4'
redis['port'] = 6379
##
## Needed because 'gitlab-ctl reconfigure' runs 'rake cache:clear:redis'
##
gitlab_rails['redis_password'] = 'Redis password'
```
### Step 4: HAProxy
We'll be using HAProxy to balance the load between the Application machines.
1. Manually stop Nginx (we will disable it in `/etc/gitlab/gitlab.rb` later):
```sh
gitlab-ctl stop nginx
```
1. Install the HAProxy package:
```sh
apt-get install haproxy
```
1. Make sure you have a single SSL `pem` file containing the
certificate and the private key.
```sh
cat /etc/ssl/cert.pem /etc/ssl/privkey.pem > /etc/ssl/aio.pem
```
1. Edit `/etc/haproxy/haproxy.cfg` and overwrite it with the following:
```
global
log 127.0.0.1 local0
log 127.0.0.1 local1 notice
maxconn 4096
user haproxy
group haproxy
daemon
defaults
log global
mode http
option httplog
option dontlognull
option forwardfor
option http-server-close
stats enable
stats uri /haproxy?stats
frontend www-http
bind *:80
reqadd X-Forwarded-Proto:\ http
default_backend www-backend
frontend www-https
bind 0.0.0.0:443 ssl crt /etc/ssl/aio.pem
reqadd X-Forwarded-Proto:\ https
default_backend www-backend
backend www-backend
redirect scheme https if !{ ssl_fc }
balance leastconn
option httpclose
option forwardfor
cookie JSESSIONID prefix
##
## Enter the IPs of your Application servers here
##
server nodeA 2.3.4.5:80 cookie A check
server nodeB 3.4.5.6:80 cookie A check
```
1. Start the HAProxy service:
```sh
service haproxy start
```
### Step 5: Apply settings
1. Edit `/etc/gitlab/gitlab.rb` and add the following:
```ruby
nginx['enable'] = false
sidekiq['enable'] = false
unicorn['enable'] = false
##
## These are optional/untested/irrelevant
##
gitaly['enable'] = false
gitlab_workhorse['enable'] = false
mailroom['enable'] = false
prometheus['enable'] = false
```
1. [Reconfigure GitLab][] for the changes to take effect.
1. Until [Omnibus#2797](https://gitlab.com/gitlab-org/omnibus-gitlab/issues/2797)
gets fixed, you will need to manually restart PostgreSQL:
```sh
gitlab-ctl restart postgresql geo-postgresql
```
### Step 6: Step up database replication
Database replication will operate between the Services machines.
Follow the [Setup the database replication](database.md) instructions
to set up.
## Application machine
Repeat these steps for every machine running `gitlab-rails`.
The following steps should be performed in the Application machine. SSH to it
and login as root:
```sh
sudo -i
```
### Step 1: Add NFS mount
1. Install the required NFS packages:
```sh
apt-get install nfs-common
```
1. Create the mount point directory:
```sh
mkdir -p /mnt/nfs
```
1. Edit `/etc/fstab` and add the following lines
(where `1.2.3.4` is the internal IP of the Services machine):
```
1.2.3.4:/var/opt/gitlab/nfs /mnt/nfs nfs defaults,nfsvers=4,soft,rsize=1048576,wsize=1048576,noatime
```
1. Mount the share:
```sh
mount -a -t nfs
```
You can check if the mount is working by checking the existence of the
directories `builds/`, `git-data/`, `shared/`, and `uploads/` in
`/mnt/nfs`:
```
ls /mnt/nfs
```
### Step 2: Configure proxied SSL
The load balancer will take care of the SSL termination, so configure nginx to
work with proxied SSL.
Follow the instructions to [configure proxied SSL](https://docs.gitlab.com/omnibus/settings/nginx.html#supporting-proxied-ssl).
### Step 3: Configure connections to the Services machine
1. Edit `/etc/gitlab/gitlab.rb` and add the following:
```ruby
##
## Use the NFS mount to store data
##
gitlab_rails['uploads_directory'] = '/mnt/nfs/uploads'
gitlab_rails['shared_path'] = '/mnt/nfs/shared'
gitlab_ci['builds_directory'] = '/mnt/nfs/builds'
git_data_dirs({
'default': {
'path': '/mnt/nfs/git-data'
}
})
high_availability['mountpoint'] = '/mnt/nfs'
##
## Disable PostgreSQL on the local machine and connect to the remote
##
postgresql['enable'] = false
gitlab_rails['auto_migrate'] = false
gitlab_rails['db_host'] = '1.2.3.4'
gitlab_rails['db_password'] = 'DB password'
##
## Disable Redis on the local machine and connect to the remote
##
redis['enable'] = false
gitlab_rails['redis_host'] = '1.2.3.4'
gitlab_rails['redis_password'] = 'Redis password'
```
1. **[Only for primary nodes]** Add the following to `/etc/gitlab/gitlab.rb`:
```ruby
geo_primary_role['enable'] = true
```
1. **[Only for secondary nodes]** Add the following to `/etc/gitlab/gitlab.rb`:
```ruby
geo_secondary_role['enable'] = true
geo_postgresql['enable'] = true # TODO set to false when https://gitlab.com/gitlab-org/omnibus-gitlab/issues/2980 is fixed
geo_secondary['auto_migrate'] = false
geo_secondary['db_host'] = '1.2.3.4'
geo_secondary['db_password'] = 'Geo tracking DB password'
```
1. Copy the database encryption key. Follow the instructions of
[Step 1. Copying the database encryption key](configuration.md#step-1-copying-the-database-encryption-key)
1. [Reconfigure GitLab][] for the changes to take effect (if you haven't done
this yet in previous step).
1. [Restart GitLab][] to start the processes with the correct connections.
## Troubleshooting
### HAProxy
You can connect to `https://example.com/haproxy?stats` to monitor the
load balancing between the Application machines.
[reconfigure GitLab]: ../administration/restart_gitlab.md#omnibus-gitlab-reconfigure
[restart GitLab]: ../administration/restart_gitlab.md#omnibus-gitlab-restart
Markdown is supported
0%
or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment