Document Geo HA

changelogs/unreleased-ee/tc-geo-ha-doc.yml

+---
+title: Document how to set up GitLab Geo for HA
+merge_request: 3468
+author:
+type: other

doc/gitlab-geo/README.md

@@ -181,6 +181,10 @@ Read through the [GitLab Geo configuration](configuration.md) documentation.
 
 Read how to [update your Geo nodes to the latest GitLab version](updating_the_geo_nodes.md).
 
+## Configuring GitLab Geo HA
+
+Read through the [Geo High Availability documentation](ha.md).
+
 ## Current limitations
 
 - You cannot push code to secondary nodes

doc/gitlab-geo/database.md

@@ -112,8 +112,10 @@ will not be able to perform all necessary configuration steps. Refer to
     this example:
 
     ```bash
-    # Certificate and key currently used by GitLab
-    # - replace primary.geo.example.com with your domain
+    ##
+    ## Certificate and key currently used by GitLab
+    ## - replace primary.geo.example.com with your domain
+    ##
     install -o gitlab-psql -g gitlab-psql -m 0400 -T /etc/gitlab/ssl/primary.geo.example.com.crt ~gitlab-psql/data/server.crt
     install -o gitlab-psql -g gitlab-psql -m 0400 -T /etc/gitlab/ssl/primary.geo.example.com.key ~gitlab-psql/data/server.key
     ```
@@ -134,8 +136,10 @@ will not be able to perform all necessary configuration steps. Refer to
     to the correct location:
 
     ```
-    # Self-signed certificate and key
-    # - assumes the files are in your current working directory
+    ##
+    ## Self-signed certificate and key
+    ## - assumes the files are in your current working directory
+    ##
     install -o gitlab-psql -g gitlab-psql -m 0400 -T server.crt ~gitlab-psql/data/server.crt
     install -o gitlab-psql -g gitlab-psql -m 0400 -T server.key ~gitlab-psql/data/server.key
     ```
@@ -166,10 +170,14 @@ will not be able to perform all necessary configuration steps. Refer to
     To lookup the address of a Geo node, SSH in to the Geo node and execute:
 
     ```bash
-    # Private address
+    ##
+    ## Private address
+    ##
     ip route get 255.255.255.255 | awk '{print "Private address:", $NF; exit}'
 
-    # Public address
+    ##
+    ## Public address
+    ##
     echo "External address: $(curl ipinfo.io/ip)"
     ```
 
@@ -199,23 +207,31 @@ will not be able to perform all necessary configuration steps. Refer to
     ```ruby
     geo_primary_role['enable'] = true
 
-    # Primary address
-    # - replace '1.2.3.4' with the primary private address
+    ##
+    ## Primary address
+    ## - replace '1.2.3.4' with the primary private address
+    ##
     postgresql['listen_address'] = '1.2.3.4'
     postgresql['trust_auth_cidr_addresses'] = ['127.0.0.1/32','1.2.3.4/32']
 
+    ##
     # Secondary addresses
     # - replace '5.6.7.8' with the secondary public address
+    ##
     postgresql['md5_auth_cidr_addresses'] = ['5.6.7.8/32']
 
-    # Replication settings
-    # - set this to be the number of Geo secondary nodes you have
+    ##
+    ## Replication settings
+    ## - set this to be the number of Geo secondary nodes you have
+    ##
     postgresql['max_replication_slots'] = 1
     # postgresql['max_wal_senders'] = 10
     # postgresql['wal_keep_segments'] = 10
 
-    # Disable automatic database migrations temporarily
-    # (until PostgreSQL is restarted and listening on the private address)
+    ##
+    ## Disable automatic database migrations temporarily
+    ## (until PostgreSQL is restarted and listening on the private address).
+    ##
     gitlab_rails['auto_migrate'] = false
     ```
 
@@ -234,7 +250,7 @@ will not be able to perform all necessary configuration steps. Refer to
 
 1. Save the file and reconfigure GitLab for the database listen changes and
    the replication slot changes to be applied.
-    
+
     ```bash
     gitlab-ctl reconfigure
     ```
@@ -322,13 +338,17 @@ because we have not yet configured the secondary server. This is the next step.
     it in the right location.
 
     ```bash
-    # Certificate and key currently used by GitLab
+    ##
+    ## Certificate and key currently used by GitLab
+    ##
     mkdir -p ~gitlab-psql/.postgresql
     ln -s /opt/gitlab/embedded/ssl/certs/cacert.pem ~gitlab-psql/.postgresql/root.crt
     ```
-    
+
     ```bash
-    # Self-signed certificate and key
+    ##
+    ## Self-signed certificate and key
+    ##
     install -o gitlab-psql -g gitlab-psql -m 0400 -T server.crt -D ~gitlab-psql/.postgresql/root.crt
     ```
 
@@ -336,15 +356,20 @@ because we have not yet configured the secondary server. This is the next step.
     connections.
 
 
-1. Test that the remote connection to the primary server works.
+1. Test that the remote connection to the primary server works (as the
+   `gitlab-psql` user):
 
     ```
-    # Certificate and key currently used by GitLab, and connecting by FQDN
+    ##
+    ## Certificate and key currently used by GitLab, and connecting by FQDN
+    ##
     sudo -u gitlab-psql /opt/gitlab/embedded/bin/psql -U gitlab_replicator -d "dbname=gitlabhq_production sslmode=verify-full" -W -h primary.geo.example.com
     ```
-    
+
     ```
-    # Self-signed certificate and key, or connecting by IP address
+    ##
+    ## Self-signed certificate and key, or connecting by IP address
+    ##
     sudo -u gitlab-psql /opt/gitlab/embedded/bin/psql -U gitlab_replicator -d "dbname=gitlabhq_production sslmode=verify-ca" -W -h 1.2.3.4
     ```
 
@@ -417,13 +442,17 @@ data before running `pg_basebackup`.
 1. Execute the command below to start a backup/restore and begin the replication:
 
     ```
-    # Certificate and key currently used by GitLab, and connecting by FQDN
+    ##
+    ## Certificate and key currently used by GitLab, and connecting by FQDN
+    ##
     gitlab-ctl replicate-geo-database --slot-name=secondary_example --host=primary.geo.example.com
     ```
-    
+
     ```
-    # Self-signed certificate and key, or connecting by IP
-    gitlab-ctl replicate-geo-database --slot-name=secondary_example --sslmode=verify-ca --host=1.2.3.4
+    ##
+    ## Self-signed certificate and key, or connecting by IP
+    ##
+    gitlab-ctl replicate-geo-database --sslmode=verify-ca --slot-name=secondary_example --host=1.2.3.4
     ```
 
     If PostgreSQL is listening on a non-standard port, add `--port=` as well.
@@ -477,14 +506,18 @@ The `geo_primary_role` makes configuration changes to `pg_hba.conf` and
 `postgresql.conf` on the primary:
 
 ```
-# pg_hba.conf
-# GitLab Geo Primary
+##
+## GitLab Geo Primary
+## - pg_hba.conf
+##
 host    replication gitlab_replicator <trusted secondary IP>/32     md5
 ```
 
 ```
-# postgresql.conf
-# Geo Primary Role
+##
+## Geo Primary Role
+## - postgresql.conf
+##
 sql_replication_user = gitlab_replicator
 wal_level = hot_standby
 max_wal_senders = 10
@@ -499,8 +532,10 @@ on the secondary. The PostgreSQL settings for this database it adds to
 the default settings:
 
 ```
-# postgresql.conf
-# Geo Secondary Role
+##
+## Geo Secondary Role
+## - postgresql.conf
+##
 wal_level = hot_standby
 max_wal_senders = 10
 wal_keep_segments = 10

doc/gitlab-geo/ha.md

+# GitLab Geo High Availability
+
+This document describes a possible configuration on how to set up Geo
+in a Highly Available environment. If your HA setup differs from the one
+described in this document, you still can use the instructions and adapt them
+to your needs.
+
+## Architecture overview
+
+![Active/Active HA Diagram](../administration/img/high_availability/active-active-diagram.png)
+
+This documentation assumes all machines used in this HA setup can
+communicate over the network using internal IP addresses.
+
+NOTE: **Note:**
+`external_url` must be the same for every machine, and `https` should be used.
+
+## Services machine
+
+One machine, called the Services machine will be used to run:
+
+- NFS shares
+- PostgreSQL
+- Redis
+- HAProxy
+
+### Prerequisites
+
+Make sure you have GitLab EE installed using the
+[Omnibus package](https://about.gitlab.com/installation).
+
+The following steps should be performed in the Services machine. SSH to it
+and login as root:
+
+```sh
+sudo -i
+```
+
+### Step 1: Set up NFS share
+
+1. Install the required NFS packages:
+
+    ```sh
+    apt-get install nfs-kernel-server
+    ```
+
+1. Create the required directories:
+
+    ```sh
+    mkdir -p /var/opt/gitlab/nfs/builds/    \
+             /var/opt/gitlab/nfs/git-data/  \
+             /var/opt/gitlab/nfs/shared/    \
+             /var/opt/gitlab/nfs/uploads/
+    ```
+
+1. Make the directories available through NFS, by adding this to
+   `/etc/exports` (see also the [NFS HA recommended options](../administration/high_availability/nfs.md#recommended-options)):
+
+    ```
+    /var/opt/gitlab/nfs *(rw,sync,no_root_squash)
+    ```
+
+1. Start the NFS service:
+
+    ```sh
+    systemctl start nfs-kernel-server.service
+    ```
+
+1. Apply the settings to take effect:
+
+    ```sh
+    exportfs -a
+    ```
+
+### Step 2: Set up PostgreSQL server
+
+1. Edit `/etc/gitlab/gitlab.rb` and add the following:
+
+    ```ruby
+    postgresql['enable'] = true
+
+    ##
+    ## Replace 1.2.3.4 with the internal IP address of the current machine and
+    ## 2.3.4.5 and 3.4.5.6 with the internal IP addresses of the machines
+    ## running the Application server(s).
+    ##
+    postgresql['listen_address'] = '1.2.3.4'
+    postgresql['trust_auth_cidr_addresses'] = ['1.2.3.4/32', '2.3.4.5/32', '3.4.5.6/32']
+
+    gitlab_rails['auto_migrate'] = true
+    gitlab_rails['db_password'] = 'DB password'
+    ```
+
+1. **Only for secondary nodes** Also add this to `/etc/gitlab/gitlab.rb`:
+
+    ```ruby
+    geo_postgresql['enable'] = true
+
+    ##
+    ## Replace 1.2.3.4 with the internal IP address of the current machine and
+    ## 2.3.4.5 and 3.4.5.6 with the internal IP addresses of the machines
+    ## running the Application server(s).
+    ##
+    geo_postgresql['listen_address'] = '1.2.3.4'
+    geo_postgresql['trust_auth_cidr_addresses'] = ['1.2.3.4/32', '2.3.4.5/32', '3.4.5.6/32']
+
+    geo_secondary['auto_migrate'] = true
+    geo_secondary['db_host'] = '1.2.3.4'
+    geo_secondary['db_password'] = 'Geo tracking DB password'
+    ```
+
+### Step 3: Set up Redis
+
+Edit `/etc/gitlab/gitlab.rb` and add the following:
+
+ ```ruby
+redis['enable'] = true
+redis['password'] = 'Redis password'
+
+##
+## Replace 1.2.3.4 with the internal IP address of the current machine
+##
+redis['bind'] = '1.2.3.4'
+redis['port'] = 6379
+
+##
+## Needed because 'gitlab-ctl reconfigure' runs 'rake cache:clear:redis'
+##
+gitlab_rails['redis_password'] = 'Redis password'
+```
+
+### Step 4: HAProxy
+
+We'll be using HAProxy to balance the load between the Application machines.
+
+1. Manually stop Nginx (we will disable it in `/etc/gitlab/gitlab.rb` later):
+
+    ```sh
+    gitlab-ctl stop nginx
+    ```
+
+1. Install the HAProxy package:
+
+    ```sh
+    apt-get install haproxy
+    ```
+
+1. Make sure you have a single SSL `pem` file containing the
+   certificate and the private key.
+
+    ```sh
+    cat /etc/ssl/cert.pem /etc/ssl/privkey.pem > /etc/ssl/aio.pem
+    ```
+
+1. Edit `/etc/haproxy/haproxy.cfg` and overwrite it with the following:
+
+    ```
+    global
+        log 127.0.0.1 local0
+        log 127.0.0.1 local1 notice
+        maxconn 4096
+        user haproxy
+        group haproxy
+        daemon
+
+    defaults
+        log global
+        mode http
+        option httplog
+        option dontlognull
+        option forwardfor
+        option http-server-close
+        stats enable
+        stats uri /haproxy?stats
+
+    frontend www-http
+        bind *:80
+        reqadd X-Forwarded-Proto:\ http
+        default_backend www-backend
+
+    frontend www-https
+        bind 0.0.0.0:443 ssl crt /etc/ssl/aio.pem
+        reqadd X-Forwarded-Proto:\ https
+        default_backend www-backend
+
+    backend www-backend
+        redirect scheme https if !{ ssl_fc }
+        balance leastconn
+        option httpclose
+        option forwardfor
+        cookie JSESSIONID prefix
+
+        ##
+        ## Enter the IPs of your Application servers here
+        ##
+        server nodeA 2.3.4.5:80 cookie A check
+        server nodeB 3.4.5.6:80 cookie A check
+    ```
+
+1. Start the HAProxy service:
+
+    ```sh
+    service haproxy start
+    ```
+
+### Step 5: Apply settings
+
+1. Edit `/etc/gitlab/gitlab.rb` and add the following:
+
+    ```ruby
+    nginx['enable'] = false
+    sidekiq['enable'] = false
+    unicorn['enable'] = false
+
+    ##
+    ## These are optional/untested/irrelevant
+    ##
+    gitaly['enable'] = false
+    gitlab_workhorse['enable'] = false
+    mailroom['enable'] = false
+    prometheus['enable'] = false
+    ```
+
+1. [Reconfigure GitLab][] for the changes to take effect.
+
+1. Until [Omnibus#2797](https://gitlab.com/gitlab-org/omnibus-gitlab/issues/2797)
+   gets fixed, you will need to manually restart PostgreSQL:
+
+    ```sh
+    gitlab-ctl restart postgresql geo-postgresql
+    ```
+
+### Step 6: Step up database replication
+
+Database replication will operate between the Services machines.
+Follow the [Setup the database replication](database.md) instructions
+to set up.
+
+## Application machine
+
+Repeat these steps for every machine running `gitlab-rails`.
+
+The following steps should be performed in the Application machine. SSH to it
+and login as root:
+
+```sh
+sudo -i
+```
+
+### Step 1: Add NFS mount
+
+1. Install the required NFS packages:
+
+    ```sh
+    apt-get install nfs-common
+    ```
+
+1. Create the mount point directory:
+
+    ```sh
+    mkdir -p /mnt/nfs
+    ```
+
+1. Edit `/etc/fstab` and add the following lines
+   (where `1.2.3.4` is the internal IP of the Services machine):
+
+    ```
+    1.2.3.4:/var/opt/gitlab/nfs /mnt/nfs nfs defaults,nfsvers=4,soft,rsize=1048576,wsize=1048576,noatime
+    ```
+
+1. Mount the share:
+
+    ```sh
+    mount -a -t nfs
+    ```
+
+You can check if the mount is working by checking the existence of the
+directories `builds/`, `git-data/`, `shared/`, and `uploads/` in
+`/mnt/nfs`:
+
+```
+ls /mnt/nfs
+```
+
+### Step 2: Configure proxied SSL
+
+The load balancer will take care of the SSL termination, so configure nginx to
+work with proxied SSL.
+
+Follow the instructions to [configure proxied SSL](https://docs.gitlab.com/omnibus/settings/nginx.html#supporting-proxied-ssl).
+
+### Step 3: Configure connections to the Services machine
+
+1. Edit `/etc/gitlab/gitlab.rb` and add the following:
+
+    ```ruby
+    ##
+    ## Use the NFS mount to store data
+    ##
+    gitlab_rails['uploads_directory'] = '/mnt/nfs/uploads'
+    gitlab_rails['shared_path'] = '/mnt/nfs/shared'
+    gitlab_ci['builds_directory'] = '/mnt/nfs/builds'
+    git_data_dirs({
+      'default': {
+        'path': '/mnt/nfs/git-data'
+      }
+    })
+
+    high_availability['mountpoint'] = '/mnt/nfs'
+
+    ##
+    ## Disable PostgreSQL on the local machine and connect to the remote
+    ##
+    postgresql['enable'] = false
+    gitlab_rails['auto_migrate'] = false
+    gitlab_rails['db_host'] = '1.2.3.4'
+    gitlab_rails['db_password'] = 'DB password'
+
+    ##
+    ## Disable Redis on the local machine and connect to the remote
+    ##
+    redis['enable'] = false
+    gitlab_rails['redis_host'] = '1.2.3.4'
+    gitlab_rails['redis_password'] = 'Redis password'
+    ```
+
+1. **[Only for primary nodes]** Add the following to `/etc/gitlab/gitlab.rb`:
+
+    ```ruby
+    geo_primary_role['enable'] = true
+    ```
+
+1. **[Only for secondary nodes]** Add the following to `/etc/gitlab/gitlab.rb`:
+
+    ```ruby
+    geo_secondary_role['enable'] = true
+
+    geo_postgresql['enable'] = true # TODO set to false when https://gitlab.com/gitlab-org/omnibus-gitlab/issues/2980 is fixed
+    geo_secondary['auto_migrate'] = false
+    geo_secondary['db_host'] = '1.2.3.4'
+    geo_secondary['db_password'] = 'Geo tracking DB password'
+    ```
+
+1. Copy the database encryption key. Follow the instructions of
+   [Step 1. Copying the database encryption key](configuration.md#step-1-copying-the-database-encryption-key)
+
+1. [Reconfigure GitLab][] for the changes to take effect (if you haven't done
+   this yet in previous step).
+
+1. [Restart GitLab][] to start the processes with the correct connections.
+
+## Troubleshooting
+
+### HAProxy
+
+You can connect to `https://example.com/haproxy?stats` to monitor the
+load balancing between the Application machines.
+
+[reconfigure GitLab]: ../administration/restart_gitlab.md#omnibus-gitlab-reconfigure
+[restart GitLab]: ../administration/restart_gitlab.md#omnibus-gitlab-restart