Skip to content

G
gitlab-ce
Commit f226961f authored by John Jarvis's avatar John Jarvis Committed by Stan Hu
Browse files
Options

Doc update for GitLab HA Geo.

parent b5df6d7c
Hide whitespace changes
Inline Side-by-side
Showing with 112 additions and 276 deletions
doc/gitlab-geo/ha.md
View file @ f226961f
# GitLab Geo High Availability # GitLab Geo High Availability
This document describes a possible configuration on how to set up Geo This document describes a minimal reference architecture for running GitLab Geo
in a Highly Available environment. If your HA setup differs from the one in a high availability configuration. If your HA setup differs from the one
described in this document, you still can use the instructions and adapt them described, it is possible to adapt these instructions to your needs.
to your needs.
## Architecture overview ## Architecture overview
![Active/Active HA Diagram](../administration/img/high_availability/active-active-diagram.png) ![Geo HA Diagram](../administration/img/high_availability/geo-ha-diagram.png)
This documentation assumes all machines used in this HA setup can _[diagram source - gitlab employees only](https://docs.google.com/drawings/d/1z0VlizKiLNXVVVaERFwgsIOuEgjcUqDTWPdQYsE7Z4c/edit)_
communicate over the network using internal IP addresses.
NOTE: **Note:** The topology above assumes that the primary and secondary Geo clusters
`external_url` must be the same for every machine, and `https` should be used. are located in two separate locations, on their own virtual network
with private IP addresses. The network is configured such that all machines within
## Services machine one geographic location can communicate with each other using their private IP addresses.
The IP addresses given are examples and may be different depending on the
One machine, called the Services machine will be used to run: network topology of your deployment.
- NFS shares
- PostgreSQL
- Redis
- HAProxy
### Prerequisites
Make sure you have GitLab EE installed using the The only external way to access the two GitLab Geo deployments is by HTTPS at
[Omnibus package](https://about.gitlab.com/installation). `gitlab.us.example.com` and `gitlab.eu.example.com` in the example above.
The following steps should be performed in the Services machine. SSH to it
and login as root:
```sh NOTE: **Note:**
sudo -i The primary and secondary GitLab Geo deployments must be able to communicate to each other over HTTPS.
```
### Step 1: Set up NFS share
1. Install the required NFS packages: ## Redis and PostgreSQL High Availability
```sh The primary and secondary Redis and PostgreSQL should be configured
apt-get install nfs-kernel-server for high availability. Because of the additional complexity involved
``` in setting up this configuration for PostgreSQL and Redis
it is not covered by this Geo HA documentation.
The two services will instead be configured such that
they will each run on a single machine.
1. Create the required directories: For more information about setting up a highly available PostgreSQL cluster and Redis cluster using the omnibus package see the high availability documentation for
[PostgreSQL](../administration/high_availability/database.md) and
[Redis](../administration/high_availability/redis.md), respectively.
```sh From these instructions you will need the following for the examples below:
mkdir -p /var/opt/gitlab/nfs/builds/ \ * `gitlab_rails['db_password']` for the PostgreSQL "DB password"
/var/opt/gitlab/nfs/git-data/ \ * `redis['password']` for the Redis "Redis password"
/var/opt/gitlab/nfs/shared/ \
/var/opt/gitlab/nfs/uploads/
```
1. Make the directories available through NFS, by adding this to NOTE: **Note:**
`/etc/exports` (see also the [NFS HA recommended options](../administration/high_availability/nfs.md#recommended-options)): It is possible to use cloud hosted services for PostgreSQL and Redis but this is beyond the scope of this document.
``` ### Prerequisites
/var/opt/gitlab/nfs *(rw,sync,no_root_squash)
```
1. Start the NFS service: Make sure you have GitLab EE installed using the
[Omnibus package](https://about.gitlab.com/installation).
```sh
systemctl start nfs-kernel-server.service
```
1. Apply the settings to take effect: ### Step 1: Configure the Geo Backend Services
```sh On the **primary** backend servers configure the following services:
exportfs -a
```
### Step 2: Set up PostgreSQL server * [Redis](../administration/high_availability/redis.md) for high availability.
* [NFS Server](../administration/high_availability/nfs.md) for repository, LFS, and upload storage.
* [PostgreSQL](../administration/high_availability/database.md) for high availability.
1. Edit `/etc/gitlab/gitlab.rb` and add the following: On the **secondary** backend servers configure the following services:
```ruby * [Redis](../administration/high_availability/redis.md) for high availability.
postgresql['enable'] = true * [NFS Server](../administration/high_availability/nfs.md) which will store data that is synchronized from the Geo primary.
## ### Step 2: Configure the Postgres services on the Geo Secondary
## Replace 1.2.3.4 with the internal IP address of the current machine and
## 2.3.4.5 and 3.4.5.6 with the internal IP addresses of the machines
## running the Application server(s).
##
postgresql['listen_address'] = '1.2.3.4'
postgresql['trust_auth_cidr_addresses'] = ['1.2.3.4/32', '2.3.4.5/32', '3.4.5.6/32']
gitlab_rails['auto_migrate'] = true 1. Configure the [secondary Geo PostgreSQL database](../gitlab-geo/database.md)
gitlab_rails['db_password'] = 'DB password' as a read-only secondary of the primary Geo PostgreSQL database.
```
1. **Only for secondary nodes** Also add this to `/etc/gitlab/gitlab.rb`: 1. Configure the Geo tracking database on the secondary server, to do this modify `/etc/gitlab/gitlab.rb`:
```ruby ```ruby
geo_postgresql['enable'] = true geo_postgresql['enable'] = true
## geo_postgresql['listen_address'] = '10.1.4.1'
## Replace 1.2.3.4 with the internal IP address of the current machine and geo_postgresql['trust_auth_cidr_addresses'] = ['10.1.0.0/16']
## 2.3.4.5 and 3.4.5.6 with the internal IP addresses of the machines
## running the Application server(s).
##
geo_postgresql['listen_address'] = '1.2.3.4'
geo_postgresql['trust_auth_cidr_addresses'] = ['1.2.3.4/32', '2.3.4.5/32', '3.4.5.6/32']
geo_secondary['auto_migrate'] = true geo_secondary['auto_migrate'] = true
geo_secondary['db_host'] = '1.2.3.4' geo_secondary['db_host'] = '10.1.4.1'
geo_secondary['db_password'] = 'Geo tracking DB password' geo_secondary['db_password'] = 'Geo tracking DB password'
``` ```
### Step 3: Set up Redis NOTE: **Note:**
Be sure that other non-postgresql services are disabled by setting `enable` to `false` in
Edit `/etc/gitlab/gitlab.rb` and add the following: the [gitlab.rb configuration](https://gitlab.com/gitlab-org/omnibus-gitlab/blob/master/files/gitlab-config-template/gitlab.rb.template).
```ruby
redis['enable'] = true
redis['password'] = 'Redis password'
##
## Replace 1.2.3.4 with the internal IP address of the current machine
##
redis['bind'] = '1.2.3.4'
redis['port'] = 6379
##
## Needed because 'gitlab-ctl reconfigure' runs 'rake cache:clear:redis'
##
gitlab_rails['redis_password'] = 'Redis password'
```
### Step 4: HAProxy
We'll be using HAProxy to balance the load between the Application machines.
1. Manually stop Nginx (we will disable it in `/etc/gitlab/gitlab.rb` later):
```sh
gitlab-ctl stop nginx
```
1. Install the HAProxy package:
```sh
apt-get install haproxy
```
1. Make sure you have a single SSL `pem` file containing the After making these changes be sure to run `sudo gitlab-ctl reconfigure` so that they take effect.
certificate and the private key.
```sh ### Step 3: Setup the LoadBalancer
cat /etc/ssl/cert.pem /etc/ssl/privkey.pem > /etc/ssl/aio.pem
```
1. Edit `/etc/haproxy/haproxy.cfg` and overwrite it with the following: In this topology there will need to be a load balancers at each geographical location
to route traffic to the application servers.
``` See the [Load Balancer for GitLab HA](../administration/high_availability/load_balancer.md)
global documentation for more information.
log 127.0.0.1 local0
log 127.0.0.1 local1 notice
maxconn 4096
user haproxy
group haproxy
daemon
defaults
log global
mode http
option httplog
option dontlognull
option forwardfor
option http-server-close
stats enable
stats uri /haproxy?stats
frontend www-http
bind *:80
reqadd X-Forwarded-Proto:\ http
default_backend www-backend
frontend www-https
bind 0.0.0.0:443 ssl crt /etc/ssl/aio.pem
reqadd X-Forwarded-Proto:\ https
default_backend www-backend
backend www-backend
redirect scheme https if !{ ssl_fc }
balance leastconn
option httpclose
option forwardfor
cookie JSESSIONID prefix
##
## Enter the IPs of your Application servers here
##
server nodeA 2.3.4.5:80 cookie A check
server nodeB 3.4.5.6:80 cookie A check
```
1. Start the HAProxy service: ### Step 4: Configure the Geo Frontend Application Servers
```sh In the architecture overview there are two machines running the GitLab application
service haproxy start services. These services are enabled selectively in the configuration. Additionally
``` the addresses of the remote endpoints for PostgreSQL and Redis will need to be specified.
### Step 5: Apply settings #### On the GitLab Primary Frontend servers
1. Edit `/etc/gitlab/gitlab.rb` and add the following: 1. Edit `/etc/gitlab/gitlab.rb` and ensure the following to disable PostgreSQL and Redis from running locally.
```ruby ```ruby
nginx['enable'] = false
sidekiq['enable'] = false
unicorn['enable'] = false
## ##
## These are optional/untested/irrelevant ## Disable PostgreSQL on the local machine and connect to the remote
## ##
gitaly['enable'] = false
gitlab_workhorse['enable'] = false
mailroom['enable'] = false
prometheus['enable'] = false
```
1. [Reconfigure GitLab][] for the changes to take effect. postgresql['enable'] = false
gitlab_rails['auto_migrate'] = false
1. Until [Omnibus#2797](https://gitlab.com/gitlab-org/omnibus-gitlab/issues/2797) gitlab_rails['db_host'] = '10.0.3.1'
gets fixed, you will need to manually restart PostgreSQL: gitlab_rails['db_password'] = 'DB password'
```sh
gitlab-ctl restart postgresql geo-postgresql
```
### Step 6: Step up database replication
Database replication will operate between the Services machines.
Follow the [Setup the database replication](database.md) instructions
to set up.
## Application machine
Repeat these steps for every machine running `gitlab-rails`.
The following steps should be performed in the Application machine. SSH to it
and login as root:
```sh
sudo -i
```
### Step 1: Add NFS mount
1. Install the required NFS packages:
```sh
apt-get install nfs-common
```
1. Create the mount point directory:
```sh
mkdir -p /mnt/nfs
```
1. Edit `/etc/fstab` and add the following lines
(where `1.2.3.4` is the internal IP of the Services machine):
``` ##
1.2.3.4:/var/opt/gitlab/nfs /mnt/nfs nfs defaults,nfsvers=4,soft,rsize=1048576,wsize=1048576,noatime ## Disable Redis on the local machine and connect to the remote
``` ##
1. Mount the share: redis['enable'] = false
gitlab_rails['redis_host'] = '10.0.2.1'
gitlab_rails['redis_password'] = 'Redis password'
```sh geo_primary_role['enable'] = true
mount -a -t nfs
``` ```
You can check if the mount is working by checking the existence of the #### On the GitLab Secondary Frontend servers
directories `builds/`, `git-data/`, `shared/`, and `uploads/` in
`/mnt/nfs`:
```
ls /mnt/nfs
```
### Step 2: Configure proxied SSL
The load balancer will take care of the SSL termination, so configure nginx to On the secondary the remote endpoint for the PostgreSQL Geo database will
work with proxied SSL. be specified.
Follow the instructions to [configure proxied SSL](https://docs.gitlab.com/omnibus/settings/nginx.html#supporting-proxied-ssl). 1. Edit `/etc/gitlab/gitlab.rb` and ensure the following to disable PostgreSQL and Redis from running locally. Configure the secondary to connect to the GitLab Geo tracking database.
### Step 3: Configure connections to the Services machine
1. Edit `/etc/gitlab/gitlab.rb` and add the following:
```ruby ```ruby
##
## Use the NFS mount to store data
##
gitlab_rails['uploads_directory'] = '/mnt/nfs/uploads'
gitlab_rails['shared_path'] = '/mnt/nfs/shared'
gitlab_ci['builds_directory'] = '/mnt/nfs/builds'
git_data_dirs({
'default': {
'path': '/mnt/nfs/git-data'
}
})
high_availability['mountpoint'] = '/mnt/nfs'
## ##
## Disable PostgreSQL on the local machine and connect to the remote ## Disable PostgreSQL on the local machine and connect to the remote
## ##
postgresql['enable'] = false postgresql['enable'] = false
gitlab_rails['auto_migrate'] = false gitlab_rails['auto_migrate'] = false
gitlab_rails['db_host'] = '1.2.3.4' gitlab_rails['db_host'] = '10.1.3.1'
gitlab_rails['db_password'] = 'DB password' gitlab_rails['db_password'] = 'DB password'
## ##
## Disable Redis on the local machine and connect to the remote ## Disable Redis on the local machine and connect to the remote
## ##
redis['enable'] = false redis['enable'] = false
gitlab_rails['redis_host'] = '1.2.3.4' gitlab_rails['redis_host'] = '10.1.2.1'
gitlab_rails['redis_password'] = 'Redis password' gitlab_rails['redis_password'] = 'Redis password'
```
1. **[Only for primary nodes]** Add the following to `/etc/gitlab/gitlab.rb`:
```ruby
geo_primary_role['enable'] = true
```
1. **[Only for secondary nodes]** Add the following to `/etc/gitlab/gitlab.rb`: ##
## Enable the geo secondary role and configure the
## geo tracking database
##
```ruby
geo_secondary_role['enable'] = true geo_secondary_role['enable'] = true
geo_secondary['db_host'] = '10.1.4.1'
geo_postgresql['enable'] = true # TODO set to false when https://gitlab.com/gitlab-org/omnibus-gitlab/issues/2980 is fixed
geo_secondary['auto_migrate'] = false
geo_secondary['db_host'] = '1.2.3.4'
geo_secondary['db_password'] = 'Geo tracking DB password' geo_secondary['db_password'] = 'Geo tracking DB password'
geo_postgresql['enable'] = false
``` ```
1. Copy the database encryption key. Follow the instructions of
[Step 1. Copying the database encryption key](configuration.md#step-1-copying-the-database-encryption-key)
1. [Reconfigure GitLab][] for the changes to take effect (if you haven't done After making these changes [Reconfigure GitLab][] so that they take effect.
this yet in previous step).
On the primary the following GitLab frontend services will be enabled:
1. [Restart GitLab][] to start the processes with the correct connections. * gitlab-pages
* gitlab-workhorse
* logrotate
* nginx
* registry
* remote-syslog
* sidekiq
* unicorn
## Troubleshooting On the secondary the following GitLab frontend services will be enabled:
### HAProxy * geo-logcursor
* gitlab-pages
* gitlab-workhorse
* logrotate
* nginx
* registry
* remote-syslog
* sidekiq
* unicorn
You can connect to `https://example.com/haproxy?stats` to monitor the Verify these services by running `sudo gitlab-ctl status` on the frontend
load balancing between the Application machines. application servers.
[reconfigure GitLab]: ../administration/restart_gitlab.md#omnibus-gitlab-reconfigure [reconfigure GitLab]: ../administration/restart_gitlab.md#omnibus-gitlab-reconfigure
[restart GitLab]: ../administration/restart_gitlab.md#omnibus-gitlab-restart [restart GitLab]: ../administration/restart_gitlab.md#omnibus-gitlab-restart
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment
GitLab Nexedi Edition | About GitLab | About Nexedi | 沪ICP备2021021310号-2 | 沪ICP备2021021310号-7