Skip to content

G
gitlab-ce
Commit cdec6546 authored by Achilleas Pipinellis's avatar Achilleas Pipinellis
Browse files
Options

Merge branch 'gy-redis-nfs-ra-updates-docs' into 'master' 

Update Redis and NFS guidance in 10k Reference Architecture docs

See merge request gitlab-org/gitlab!70602
parents 369af359 cfe40c74
Hide whitespace changes
Inline Side-by-side
Showing with 363 additions and 1119 deletions
doc/administration/reference_architectures/10k_users.md
View file @ cdec6546
...@@ -22,10 +22,8 @@ full list of reference architectures, see ...@@ -22,10 +22,8 @@ full list of reference architectures, see
| PostgreSQL<sup>1</sup> | 3 | 8 vCPU, 30 GB memory | `n1-standard-8` | `m5.2xlarge` | `D8s v3` | | PostgreSQL<sup>1</sup> | 3 | 8 vCPU, 30 GB memory | `n1-standard-8` | `m5.2xlarge` | `D8s v3` |
| PgBouncer<sup>1</sup> | 3 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | `F2s v2` | | PgBouncer<sup>1</sup> | 3 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | `F2s v2` |
| Internal load balancing node<sup>3</sup> | 1 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | `F2s v2` | | Internal load balancing node<sup>3</sup> | 1 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | `F2s v2` |
| Redis - Cache<sup>2</sup> | 3 | 4 vCPU, 15 GB memory | `n1-standard-4` | `m5.xlarge` | `D4s v3` | | Redis/Sentinel - Cache<sup>2</sup> | 3 | 4 vCPU, 15 GB memory | `n1-standard-4` | `m5.xlarge` | `D4s v3` |
| Redis - Queues / Shared State<sup>2</sup> | 3 | 4 vCPU, 15 GB memory | `n1-standard-4` | `m5.xlarge` | `D4s v3` | | Redis/Sentinel - Persistent<sup>2</sup> | 3 | 4 vCPU, 15 GB memory | `n1-standard-4` | `m5.xlarge` | `D4s v3` |
| Redis Sentinel - Cache<sup>2</sup> | 3 | 1 vCPU, 3.75 GB memory | `n1-standard-1` | `c5.large` | `A1 v2` |
| Redis Sentinel - Queues / Shared State<sup>2</sup> | 3 | 1 vCPU, 3.75 GB memory | `n1-standard-1` | `c5.large` | `A1 v2` |
| Gitaly | 3 | 16 vCPU, 60 GB memory | `n1-standard-16` | `m5.4xlarge` | `D16s v3` | | Gitaly | 3 | 16 vCPU, 60 GB memory | `n1-standard-16` | `m5.4xlarge` | `D16s v3` |
| Praefect | 3 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | `F2s v2` | | Praefect | 3 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | `F2s v2` |
| Praefect PostgreSQL<sup>1</sup> | 1+ | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | `F2s v2` | | Praefect PostgreSQL<sup>1</sup> | 1+ | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | `F2s v2` |
...@@ -33,7 +31,7 @@ full list of reference architectures, see ...@@ -33,7 +31,7 @@ full list of reference architectures, see
| GitLab Rails | 3 | 32 vCPU, 28.8 GB memory | `n1-highcpu-32` | `c5.9xlarge` | `F32s v2` | | GitLab Rails | 3 | 32 vCPU, 28.8 GB memory | `n1-highcpu-32` | `c5.9xlarge` | `F32s v2` |
| Monitoring node | 1 | 4 vCPU, 3.6 GB memory | `n1-highcpu-4` | `c5.xlarge` | `F4s v2` | | Monitoring node | 1 | 4 vCPU, 3.6 GB memory | `n1-highcpu-4` | `c5.xlarge` | `F4s v2` |
| Object storage<sup>4</sup> | n/a | n/a | n/a | n/a | n/a | | Object storage<sup>4</sup> | n/a | n/a | n/a | n/a | n/a |
| NFS server (optional, not recommended) | 1 | 4 vCPU, 3.6 GB memory | `n1-highcpu-4` | `c5.xlarge` | `F4s v2` | | NFS server (non-Gitaly) | 1 | 4 vCPU, 3.6 GB memory | `n1-highcpu-4` | `c5.xlarge` | `F4s v2` |
<!-- Disable ordered list rule https://github.com/DavidAnson/markdownlint/blob/main/doc/Rules.md#md029---ordered-list-item-prefix --> <!-- Disable ordered list rule https://github.com/DavidAnson/markdownlint/blob/main/doc/Rules.md#md029---ordered-list-item-prefix -->
<!-- markdownlint-disable MD029 --> <!-- markdownlint-disable MD029 -->
...@@ -82,11 +80,6 @@ card "Database" as database { ...@@ -82,11 +80,6 @@ card "Database" as database {
card "redis" as redis { card "redis" as redis {
collections "**Redis Persistent** x3" as redis_persistent #FF6347 collections "**Redis Persistent** x3" as redis_persistent #FF6347
collections "**Redis Cache** x3" as redis_cache #FF6347 collections "**Redis Cache** x3" as redis_cache #FF6347
collections "**Redis Persistent Sentinel** x3" as redis_persistent_sentinel #FF6347
collections "**Redis Cache Sentinel** x3"as redis_cache_sentinel #FF6347
redis_persistent <.[#FF6347]- redis_persistent_sentinel
redis_cache <.[#FF6347]- redis_cache_sentinel
} }
cloud "**Object Storage**" as object_storage #white cloud "**Object Storage**" as object_storage #white
...@@ -137,8 +130,7 @@ our [Sysbench](https://github.com/akopytov/sysbench)-based ...@@ -137,8 +130,7 @@ our [Sysbench](https://github.com/akopytov/sysbench)-based
Due to better performance and availability, for data objects (such as LFS, Due to better performance and availability, for data objects (such as LFS,
uploads, or artifacts), using an [object storage service](#configure-the-object-storage) uploads, or artifacts), using an [object storage service](#configure-the-object-storage)
is recommended instead of using NFS. Using an object storage service also is recommended.
doesn't require you to provision and maintain a node.
It's also worth noting that at this time [Praefect requires its own database server](../gitaly/praefect.md#postgresql) and It's also worth noting that at this time [Praefect requires its own database server](../gitaly/praefect.md#postgresql) and
that to achieve full High Availability a third-party PostgreSQL database solution will be required. that to achieve full High Availability a third-party PostgreSQL database solution will be required.
...@@ -169,10 +161,8 @@ To set up GitLab and its components to accommodate up to 10,000 users: ...@@ -169,10 +161,8 @@ To set up GitLab and its components to accommodate up to 10,000 users:
used for shared data objects. used for shared data objects.
1. [Configure Advanced Search](#configure-advanced-search) (optional) for faster, 1. [Configure Advanced Search](#configure-advanced-search) (optional) for faster,
more advanced code search across your entire GitLab instance. more advanced code search across your entire GitLab instance.
1. [Configure NFS](#configure-nfs-optional) (optional, and not recommended) 1. [Configure NFS](#configure-nfs)
to have shared disk storage service as an alternative to Gitaly or object to have shared disk storage service for certain GitLab operations (non Gitaly or Object Storage).
storage. You can skip this step if you're not using GitLab Pages (which
requires NFS).
The servers start on the same 10.6.0.0/24 private network range, and can The servers start on the same 10.6.0.0/24 private network range, and can
connect to each other freely on these addresses. connect to each other freely on these addresses.
...@@ -193,15 +183,9 @@ The following list includes descriptions of each server and its assigned IP: ...@@ -193,15 +183,9 @@ The following list includes descriptions of each server and its assigned IP:
- `10.6.0.51`: Redis - Cache Primary - `10.6.0.51`: Redis - Cache Primary
- `10.6.0.52`: Redis - Cache Replica 1 - `10.6.0.52`: Redis - Cache Replica 1
- `10.6.0.53`: Redis - Cache Replica 2 - `10.6.0.53`: Redis - Cache Replica 2
- `10.6.0.71`: Sentinel - Cache 1 - `10.6.0.61`: Redis - Persistent Primary
- `10.6.0.72`: Sentinel - Cache 2 - `10.6.0.62`: Redis - Persistent Replica 1
- `10.6.0.73`: Sentinel - Cache 3 - `10.6.0.63`: Redis - Persistent Replica 2
- `10.6.0.61`: Redis - Queues Primary
- `10.6.0.62`: Redis - Queues Replica 1
- `10.6.0.63`: Redis - Queues Replica 2
- `10.6.0.81`: Sentinel - Queues 1
- `10.6.0.82`: Sentinel - Queues 2
- `10.6.0.83`: Sentinel - Queues 3
- `10.6.0.91`: Gitaly 1 - `10.6.0.91`: Gitaly 1
- `10.6.0.92`: Gitaly 2 - `10.6.0.92`: Gitaly 2
- `10.6.0.93`: Gitaly 3 - `10.6.0.93`: Gitaly 3
...@@ -792,15 +776,9 @@ to be used with GitLab. The following IPs will be used as an example: ...@@ -792,15 +776,9 @@ to be used with GitLab. The following IPs will be used as an example:
- `10.6.0.51`: Redis - Cache Primary - `10.6.0.51`: Redis - Cache Primary
- `10.6.0.52`: Redis - Cache Replica 1 - `10.6.0.52`: Redis - Cache Replica 1
- `10.6.0.53`: Redis - Cache Replica 2 - `10.6.0.53`: Redis - Cache Replica 2
- `10.6.0.71`: Sentinel - Cache 1 - `10.6.0.61`: Redis - Persistent Primary
- `10.6.0.72`: Sentinel - Cache 2 - `10.6.0.62`: Redis - Persistent Replica 1
- `10.6.0.73`: Sentinel - Cache 3 - `10.6.0.63`: Redis - Persistent Replica 2
- `10.6.0.61`: Redis - Queues Primary
- `10.6.0.62`: Redis - Queues Replica 1
- `10.6.0.63`: Redis - Queues Replica 2
- `10.6.0.81`: Sentinel - Queues 1
- `10.6.0.82`: Sentinel - Queues 2
- `10.6.0.83`: Sentinel - Queues 3
### Providing your own Redis instance ### Providing your own Redis instance
...@@ -812,7 +790,7 @@ optional count argument to SPOP, which is required for [Merge Trains](../../ci/p ...@@ -812,7 +790,7 @@ optional count argument to SPOP, which is required for [Merge Trains](../../ci/p
Note the Redis node's IP address or hostname, port, and password (if required). Note the Redis node's IP address or hostname, port, and password (if required).
These will be necessary later when configuring the [GitLab application servers](#configure-gitlab-rails). These will be necessary later when configuring the [GitLab application servers](#configure-gitlab-rails).
### Configure the Redis and Sentinel Cache cluster ### Configure the Redis Cache cluster
This is the section where we install and set up the new Redis Cache instances. This is the section where we install and set up the new Redis Cache instances.
...@@ -830,8 +808,12 @@ a node and change its status from primary to replica (and vice versa). ...@@ -830,8 +808,12 @@ a node and change its status from primary to replica (and vice versa).
1. Edit `/etc/gitlab/gitlab.rb` and add the contents: 1. Edit `/etc/gitlab/gitlab.rb` and add the contents:
```ruby ```ruby
# Specify server role as 'redis_master_role' and enable Consul agent # Specify server roles as 'redis_master_role' with sentinel and the Consul agent
roles(['redis_master_role', 'consul_role']) roles ['redis_sentinel_role', 'redis_master_role', 'consul_role']
# Set IP bind address and Quorum number for Redis Sentinel service
sentinel['bind'] = '0.0.0.0'
sentinel['quorum'] = 2
# IP address pointing to a local IP that the other machines can reach to. # IP address pointing to a local IP that the other machines can reach to.
# You can also set bind to '0.0.0.0' which listen in all interfaces. # You can also set bind to '0.0.0.0' which listen in all interfaces.
...@@ -843,8 +825,19 @@ a node and change its status from primary to replica (and vice versa). ...@@ -843,8 +825,19 @@ a node and change its status from primary to replica (and vice versa).
# machines to connect to it. # machines to connect to it.
redis['port'] = 6379 redis['port'] = 6379
# Set up password authentication for Redis (use the same password in all nodes). ## Port of primary Redis server for Sentinel, uncomment to change to non default. Defaults
## to `6379`.
#redis['master_port'] = 6379
# Set up password authentication for Redis and replicas (use the same password in all nodes).
redis['password'] = 'REDIS_PRIMARY_PASSWORD_OF_FIRST_CLUSTER' redis['password'] = 'REDIS_PRIMARY_PASSWORD_OF_FIRST_CLUSTER'
redis['master_password'] = 'REDIS_PRIMARY_PASSWORD_OF_FIRST_CLUSTER'
## Must be the same in every Redis node
redis['master_name'] = 'gitlab-redis-cache'
## The IP of this primary Redis node.
redis['master_ip'] = '10.6.0.51'
# Set the Redis Cache instance as an LRU # Set the Redis Cache instance as an LRU
# 90% of available RAM in MB # 90% of available RAM in MB
...@@ -878,10 +871,6 @@ a node and change its status from primary to replica (and vice versa). ...@@ -878,10 +871,6 @@ a node and change its status from primary to replica (and vice versa).
1. [Reconfigure Omnibus GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. 1. [Reconfigure Omnibus GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect.
You can specify multiple roles, like sentinel and Redis, as:
`roles(['redis_sentinel_role', 'redis_master_role'])`. Read more about
[roles](https://docs.gitlab.com/omnibus/roles/).
#### Configure the replica Redis Cache nodes #### Configure the replica Redis Cache nodes
1. SSH in to the **replica** Redis server. 1. SSH in to the **replica** Redis server.
...@@ -889,11 +878,15 @@ You can specify multiple roles, like sentinel and Redis, as: ...@@ -889,11 +878,15 @@ You can specify multiple roles, like sentinel and Redis, as:
package of your choice. Be sure to both follow _only_ installation steps 1 and 2 package of your choice. Be sure to both follow _only_ installation steps 1 and 2
on the page, and to select the correct Omnibus GitLab package, with the same version on the page, and to select the correct Omnibus GitLab package, with the same version
and type (Community or Enterprise editions) as your current install. and type (Community or Enterprise editions) as your current install.
1. Edit `/etc/gitlab/gitlab.rb` and add the contents: 1. Edit `/etc/gitlab/gitlab.rb` and add same contents as the primary node in the previous section replacing `redis_master_node` with `redis_replica_node`:
```ruby ```ruby
# Specify server role as 'redis_replica_role' and enable Consul agent # Specify server roles as 'redis_sentinel_role' and 'redis_replica_role'
roles(['redis_replica_role', 'consul_role']) roles ['redis_sentinel_role', 'redis_replica_role', 'consul_role']
# Set IP bind address and Quorum number for Redis Sentinel service
sentinel['bind'] = '0.0.0.0'
sentinel['quorum'] = 2
# IP address pointing to a local IP that the other machines can reach to. # IP address pointing to a local IP that the other machines can reach to.
# You can also set bind to '0.0.0.0' which listen in all interfaces. # You can also set bind to '0.0.0.0' which listen in all interfaces.
...@@ -905,15 +898,19 @@ You can specify multiple roles, like sentinel and Redis, as: ...@@ -905,15 +898,19 @@ You can specify multiple roles, like sentinel and Redis, as:
# machines to connect to it. # machines to connect to it.
redis['port'] = 6379 redis['port'] = 6379
# The same password for Redis authentication you set up for the primary node. ## Port of primary Redis server for Sentinel, uncomment to change to non default. Defaults
## to `6379`.
#redis['master_port'] = 6379
# Set up password authentication for Redis and replicas (use the same password in all nodes).
redis['password'] = 'REDIS_PRIMARY_PASSWORD_OF_FIRST_CLUSTER' redis['password'] = 'REDIS_PRIMARY_PASSWORD_OF_FIRST_CLUSTER'
redis['master_password'] = 'REDIS_PRIMARY_PASSWORD_OF_FIRST_CLUSTER'
# The IP of the primary Redis node. ## Must be the same in every Redis node
redis['master_ip'] = '10.6.0.51' redis['master_name'] = 'gitlab-redis-cache'
# Port of primary Redis server, uncomment to change to non default. Defaults ## The IP of the primary Redis node.
# to `6379`. redis['master_ip'] = '10.6.0.51'
#redis['master_port'] = 6379
# Set the Redis Cache instance as an LRU # Set the Redis Cache instance as an LRU
# 90% of available RAM in MB # 90% of available RAM in MB
...@@ -949,15 +946,6 @@ You can specify multiple roles, like sentinel and Redis, as: ...@@ -949,15 +946,6 @@ You can specify multiple roles, like sentinel and Redis, as:
1. Go through the steps again for all the other replica nodes, and 1. Go through the steps again for all the other replica nodes, and
make sure to set up the IPs correctly. make sure to set up the IPs correctly.
You can specify multiple roles, like sentinel and Redis, as:
`roles(['redis_sentinel_role', 'redis_master_role'])`. Read more about
[roles](https://docs.gitlab.com/omnibus/roles/).
These values don't have to be changed again in `/etc/gitlab/gitlab.rb` after
a failover, as the nodes will be managed by the [Sentinels](#configure-the-sentinel-cache-nodes), and even after a
`gitlab-ctl reconfigure`, they will get their configuration restored by
the same Sentinels.
Advanced [configuration options](https://docs.gitlab.com/omnibus/settings/redis.html) Advanced [configuration options](https://docs.gitlab.com/omnibus/settings/redis.html)
are supported and can be added if needed. are supported and can be added if needed.
...@@ -967,133 +955,15 @@ are supported and can be added if needed. ...@@ -967,133 +955,15 @@ are supported and can be added if needed.
</a> </a>
</div> </div>
#### Configure the Sentinel Cache nodes ### Configure the Redis Persistent cluster
Now that the Redis servers are all set up, let's configure the Sentinel
servers. The following IPs will be used as an example:
- `10.6.0.71`: Sentinel - Cache 1
- `10.6.0.72`: Sentinel - Cache 2
- `10.6.0.73`: Sentinel - Cache 3
NOTE:
If you're using an external Redis Sentinel instance, be sure to exclude the
`requirepass` parameter from the Sentinel configuration. This parameter causes
clients to report `NOAUTH Authentication required.`.
[Redis Sentinel 3.2.x doesn't support password authentication](https://github.com/antirez/redis/issues/3279).
To configure the Sentinel Cache server:
1. SSH in to the server that will host Consul/Sentinel.
1. [Download and install](https://about.gitlab.com/install/) the Omnibus GitLab
package of your choice. Be sure to both follow _only_ installation steps 1 and 2
on the page, and to select the correct Omnibus GitLab package, with the same version
and type (Community or Enterprise editions) as your current install.
1. Edit `/etc/gitlab/gitlab.rb` and add the contents:
```ruby
roles(['redis_sentinel_role', 'consul_role'])
## Must be the same in every sentinel node
redis['master_name'] = 'gitlab-redis-cache'
## The same password for Redis authentication you set up for the primary node.
redis['master_password'] = 'REDIS_PRIMARY_PASSWORD_OF_FIRST_CLUSTER'
## The IP of the primary Redis node.
redis['master_ip'] = '10.6.0.51'
## Define a port so Redis can listen for TCP requests which will allow other
## machines to connect to it.
redis['port'] = 6379
## Port of primary Redis server, uncomment to change to non default. Defaults
## to `6379`.
#redis['master_port'] = 6379
## Configure Sentinel's IP
sentinel['bind'] = '10.6.0.71'
## Port that Sentinel listens on, uncomment to change to non default. Defaults
## to `26379`.
#sentinel['port'] = 26379
## Quorum must reflect the amount of voting sentinels it take to start a failover.
## Value must NOT be greater then the amount of sentinels.
##
## The quorum can be used to tune Sentinel in two ways:
## 1. If a the quorum is set to a value smaller than the majority of Sentinels
## we deploy, we are basically making Sentinel more sensible to primary failures,
## triggering a failover as soon as even just a minority of Sentinels is no longer
## able to talk with the primary.
## 1. If a quorum is set to a value greater than the majority of Sentinels, we are
## making Sentinel able to failover only when there are a very large number (larger
## than majority) of well connected Sentinels which agree about the primary being down.s
sentinel['quorum'] = 2
## Consider unresponsive server down after x amount of ms.
#sentinel['down_after_milliseconds'] = 10000
## Specifies the failover timeout in milliseconds. It is used in many ways:
##
## - The time needed to re-start a failover after a previous failover was
## already tried against the same primary by a given Sentinel, is two
## times the failover timeout.
##
## - The time needed for a replica replicating to a wrong primary according
## to a Sentinel current configuration, to be forced to replicate
## with the right primary, is exactly the failover timeout (counting since
## the moment a Sentinel detected the misconfiguration).
##
## - The time needed to cancel a failover that is already in progress but
## did not produced any configuration change (REPLICAOF NO ONE yet not
## acknowledged by the promoted replica).
##
## - The maximum time a failover in progress waits for all the replica to be
## reconfigured as replicas of the new primary. However even after this time
## the replicas will be reconfigured by the Sentinels anyway, but not with
## the exact parallel-syncs progression as specified.
#sentinel['failover_timeout'] = 60000
## Enable service discovery for Prometheus
consul['monitoring_service_discovery'] = true
## The IPs of the Consul server nodes
## You can also use FQDNs and intermix them with IPs
consul['configuration'] = {
retry_join: %w(10.6.0.11 10.6.0.12 10.6.0.13),
}
# Set the network addresses that the exporters will listen on
node_exporter['listen_address'] = '0.0.0.0:9100'
redis_exporter['listen_address'] = '0.0.0.0:9121'
# Prevent database migrations from running on upgrade automatically
gitlab_rails['auto_migrate'] = false
```
1. Copy the `/etc/gitlab/gitlab-secrets.json` file from the first Omnibus node you configured and add or replace
the file of the same name on this server. If this is the first Omnibus node you are configuring then you can skip this step.
1. [Reconfigure Omnibus GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect.
1. Go through the steps again for all the other Consul/Sentinel nodes, and
make sure you set up the correct IPs.
<div align="right">
<a type="button" class="btn btn-default" href="#setup-components">
Back to setup components <i class="fa fa-angle-double-up" aria-hidden="true"></i>
</a>
</div>
### Configure the Redis and Sentinel Queues cluster
This is the section where we install and set up the new Redis Queues instances. This is the section where we install and set up the new Redis Persistent instances.
Both the primary and replica Redis nodes need the same password defined in Both the primary and replica Redis nodes need the same password defined in
`redis['password']`. At any time during a failover, the Sentinels can reconfigure `redis['password']`. At any time during a failover, the Sentinels can reconfigure
a node and change its status from primary to replica (and vice versa). a node and change its status from primary to replica (and vice versa).
#### Configure the primary Redis Queues node #### Configure the primary Redis Persistent node
1. SSH in to the **Primary** Redis server. 1. SSH in to the **Primary** Redis server.
1. [Download and install](https://about.gitlab.com/install/) the Omnibus GitLab 1. [Download and install](https://about.gitlab.com/install/) the Omnibus GitLab
...@@ -1103,8 +973,12 @@ a node and change its status from primary to replica (and vice versa). ...@@ -1103,8 +973,12 @@ a node and change its status from primary to replica (and vice versa).
1. Edit `/etc/gitlab/gitlab.rb` and add the contents: 1. Edit `/etc/gitlab/gitlab.rb` and add the contents:
```ruby ```ruby
# Specify server role as 'redis_master_role' and enable Consul agent # Specify server roles as 'redis_master_role' with Sentinel and the Consul agent
roles(['redis_master_role', 'consul_role']) roles ['redis_sentinel_role', 'redis_master_role', 'consul_role']
# Set IP bind address and Quorum number for Redis Sentinel service
sentinel['bind'] = '0.0.0.0'
sentinel['quorum'] = 2
# IP address pointing to a local IP that the other machines can reach to. # IP address pointing to a local IP that the other machines can reach to.
# You can also set bind to '0.0.0.0' which listen in all interfaces. # You can also set bind to '0.0.0.0' which listen in all interfaces.
...@@ -1116,8 +990,19 @@ a node and change its status from primary to replica (and vice versa). ...@@ -1116,8 +990,19 @@ a node and change its status from primary to replica (and vice versa).
# machines to connect to it. # machines to connect to it.
redis['port'] = 6379 redis['port'] = 6379
# Set up password authentication for Redis (use the same password in all nodes). ## Port of primary Redis server for Sentinel, uncomment to change to non default. Defaults
redis['password'] = 'REDIS_PRIMARY_PASSWORD_OF_SECOND_CLUSTER' ## to `6379`.
#redis['master_port'] = 6379
# Set up password authentication for Redis and replicas (use the same password in all nodes).
redis['password'] = 'REDIS_PRIMARY_PASSWORD_OF_FIRST_CLUSTER'
redis['master_password'] = 'REDIS_PRIMARY_PASSWORD_OF_SECOND_CLUSTER'
## Must be the same in every Redis node
redis['master_name'] = 'gitlab-redis-persistent'
## The IP of this primary Redis node.
redis['master_ip'] = '10.6.0.61'
## Enable service discovery for Prometheus ## Enable service discovery for Prometheus
consul['monitoring_service_discovery'] = true consul['monitoring_service_discovery'] = true
...@@ -1141,13 +1026,9 @@ a node and change its status from primary to replica (and vice versa). ...@@ -1141,13 +1026,9 @@ a node and change its status from primary to replica (and vice versa).
1. [Reconfigure Omnibus GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. 1. [Reconfigure Omnibus GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect.
You can specify multiple roles, like sentinel and Redis, as: #### Configure the replica Redis Persistent nodes
`roles(['redis_sentinel_role', 'redis_master_role'])`. Read more about
[roles](https://docs.gitlab.com/omnibus/roles/).
#### Configure the replica Redis Queues nodes
1. SSH in to the **replica** Redis Queue server. 1. SSH in to the **replica** Redis Persistent server.
1. [Download and install](https://about.gitlab.com/install/) the Omnibus GitLab 1. [Download and install](https://about.gitlab.com/install/) the Omnibus GitLab
package of your choice. Be sure to both follow _only_ installation steps 1 and 2 package of your choice. Be sure to both follow _only_ installation steps 1 and 2
on the page, and to select the correct Omnibus GitLab package, with the same version on the page, and to select the correct Omnibus GitLab package, with the same version
...@@ -1155,8 +1036,12 @@ You can specify multiple roles, like sentinel and Redis, as: ...@@ -1155,8 +1036,12 @@ You can specify multiple roles, like sentinel and Redis, as:
1. Edit `/etc/gitlab/gitlab.rb` and add the contents: 1. Edit `/etc/gitlab/gitlab.rb` and add the contents:
```ruby ```ruby
# Specify server role as 'redis_replica_role' and enable Consul agent # Specify server roles as 'redis_sentinel_role' and 'redis_replica_role'
roles(['redis_replica_role', 'consul_role']) roles ['redis_sentinel_role', 'redis_replica_role', 'consul_role']
# Set IP bind address and Quorum number for Redis Sentinel service
sentinel['bind'] = '0.0.0.0'
sentinel['quorum'] = 2
# IP address pointing to a local IP that the other machines can reach to. # IP address pointing to a local IP that the other machines can reach to.
# You can also set bind to '0.0.0.0' which listen in all interfaces. # You can also set bind to '0.0.0.0' which listen in all interfaces.
...@@ -1168,16 +1053,20 @@ You can specify multiple roles, like sentinel and Redis, as: ...@@ -1168,16 +1053,20 @@ You can specify multiple roles, like sentinel and Redis, as:
# machines to connect to it. # machines to connect to it.
redis['port'] = 6379 redis['port'] = 6379
## Port of primary Redis server for Sentinel, uncomment to change to non default. Defaults
## to `6379`.
#redis['master_port'] = 6379
# The same password for Redis authentication you set up for the primary node. # The same password for Redis authentication you set up for the primary node.
redis['password'] = 'REDIS_PRIMARY_PASSWORD_OF_SECOND_CLUSTER' redis['password'] = 'REDIS_PRIMARY_PASSWORD_OF_SECOND_CLUSTER'
redis['master_password'] = 'REDIS_PRIMARY_PASSWORD_OF_SECOND_CLUSTER'
## Must be the same in every Redis node
redis['master_name'] = 'gitlab-redis-persistent'
# The IP of the primary Redis node. # The IP of the primary Redis node.
redis['master_ip'] = '10.6.0.61' redis['master_ip'] = '10.6.0.61'
# Port of primary Redis server, uncomment to change to non default. Defaults
# to `6379`.
#redis['master_port'] = 6379
## Enable service discovery for Prometheus ## Enable service discovery for Prometheus
consul['monitoring_service_discovery'] = true consul['monitoring_service_discovery'] = true
...@@ -1202,15 +1091,6 @@ You can specify multiple roles, like sentinel and Redis, as: ...@@ -1202,15 +1091,6 @@ You can specify multiple roles, like sentinel and Redis, as:
1. Go through the steps again for all the other replica nodes, and 1. Go through the steps again for all the other replica nodes, and
make sure to set up the IPs correctly. make sure to set up the IPs correctly.
You can specify multiple roles, like sentinel and Redis, as:
`roles(['redis_sentinel_role', 'redis_master_role'])`. Read more about
[roles](https://docs.gitlab.com/omnibus/roles/).
These values don't have to be changed again in `/etc/gitlab/gitlab.rb` after
a failover, as the nodes will be managed by the [Sentinels](#configure-the-sentinel-queues-nodes), and even after a
`gitlab-ctl reconfigure`, they will get their configuration restored by
the same Sentinels.
Advanced [configuration options](https://docs.gitlab.com/omnibus/settings/redis.html) Advanced [configuration options](https://docs.gitlab.com/omnibus/settings/redis.html)
are supported and can be added if needed. are supported and can be added if needed.
...@@ -1220,124 +1100,6 @@ are supported and can be added if needed. ...@@ -1220,124 +1100,6 @@ are supported and can be added if needed.
</a> </a>
</div> </div>
#### Configure the Sentinel Queues nodes
Now that the Redis servers are all set up, let's configure the Sentinel
servers. The following IPs will be used as an example:
- `10.6.0.81`: Sentinel - Queues 1
- `10.6.0.82`: Sentinel - Queues 2
- `10.6.0.83`: Sentinel - Queues 3
NOTE:
If you're using an external Redis Sentinel instance, be sure to exclude the
`requirepass` parameter from the Sentinel configuration. This parameter causes
clients to report `NOAUTH Authentication required.`.
[Redis Sentinel 3.2.x doesn't support password authentication](https://github.com/antirez/redis/issues/3279).
To configure the Sentinel Queues server:
1. SSH in to the server that will host Sentinel.
1. [Download and install](https://about.gitlab.com/install/) the Omnibus GitLab
package of your choice. Be sure to both follow _only_ installation steps 1 and 2
on the page, and to select the correct Omnibus GitLab package, with the same version
and type (Community or Enterprise editions) as your current install.
1. Edit `/etc/gitlab/gitlab.rb` and add the contents:
```ruby
roles(['redis_sentinel_role', 'consul_role'])
## Must be the same in every sentinel node
redis['master_name'] = 'gitlab-redis-persistent'
## The same password for Redis authentication you set up for the primary node.
redis['master_password'] = 'REDIS_PRIMARY_PASSWORD_OF_SECOND_CLUSTER'
## The IP of the primary Redis node.
redis['master_ip'] = '10.6.0.61'
## Define a port so Redis can listen for TCP requests which will allow other
## machines to connect to it.
redis['port'] = 6379
## Port of primary Redis server, uncomment to change to non default. Defaults
## to `6379`.
#redis['master_port'] = 6379
## Configure Sentinel's IP
sentinel['bind'] = '10.6.0.81'
## Port that Sentinel listens on, uncomment to change to non default. Defaults
## to `26379`.
#sentinel['port'] = 26379
## Quorum must reflect the amount of voting sentinels it take to start a failover.
## Value must NOT be greater then the amount of sentinels.
##
## The quorum can be used to tune Sentinel in two ways:
## 1. If a the quorum is set to a value smaller than the majority of Sentinels
## we deploy, we are basically making Sentinel more sensible to primary failures,
## triggering a failover as soon as even just a minority of Sentinels is no longer
## able to talk with the primary.
## 1. If a quorum is set to a value greater than the majority of Sentinels, we are
## making Sentinel able to failover only when there are a very large number (larger
## than majority) of well connected Sentinels which agree about the primary being down.s
sentinel['quorum'] = 2
## Consider unresponsive server down after x amount of ms.
#sentinel['down_after_milliseconds'] = 10000
## Specifies the failover timeout in milliseconds. It is used in many ways:
##
## - The time needed to re-start a failover after a previous failover was
## already tried against the same primary by a given Sentinel, is two
## times the failover timeout.
##
## - The time needed for a replica replicating to a wrong primary according
## to a Sentinel current configuration, to be forced to replicate
## with the right primary, is exactly the failover timeout (counting since
## the moment a Sentinel detected the misconfiguration).
##
## - The time needed to cancel a failover that is already in progress but
## did not produced any configuration change (REPLICAOF NO ONE yet not
## acknowledged by the promoted replica).
##
## - The maximum time a failover in progress waits for all the replica to be
## reconfigured as replicas of the new primary. However even after this time
## the replicas will be reconfigured by the Sentinels anyway, but not with
## the exact parallel-syncs progression as specified.
#sentinel['failover_timeout'] = 60000
## Enable service discovery for Prometheus
consul['monitoring_service_discovery'] = true
## The IPs of the Consul server nodes
## You can also use FQDNs and intermix them with IPs
consul['configuration'] = {
retry_join: %w(10.6.0.11 10.6.0.12 10.6.0.13),
}
# Set the network addresses that the exporters will listen on
node_exporter['listen_address'] = '0.0.0.0:9100'
redis_exporter['listen_address'] = '0.0.0.0:9121'
# Prevent database migrations from running on upgrade automatically
gitlab_rails['auto_migrate'] = false
```
1. Copy the `/etc/gitlab/gitlab-secrets.json` file from the first Omnibus node you configured and add or replace
the file of the same name on this server. If this is the first Omnibus node you are configuring then you can skip this step.
1. [Reconfigure Omnibus GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect.
1. Go through the steps again for all the other Sentinel nodes, and
make sure you set up the correct IPs.
<div align="right">
<a type="button" class="btn btn-default" href="#setup-components">
Back to setup components <i class="fa fa-angle-double-up" aria-hidden="true"></i>
</a>
</div>
## Configure Gitaly Cluster ## Configure Gitaly Cluster
[Gitaly Cluster](../gitaly/praefect.md) is a GitLab provided and recommended fault tolerant solution for storing Git repositories. [Gitaly Cluster](../gitaly/praefect.md) is a GitLab provided and recommended fault tolerant solution for storing Git repositories.
...@@ -1857,30 +1619,30 @@ To configure the Sidekiq nodes, on each one: ...@@ -1857,30 +1619,30 @@ To configure the Sidekiq nodes, on each one:
gitlab_rails['redis_cache_instance'] = 'redis://:<REDIS_PRIMARY_PASSWORD_OF_FIRST_CLUSTER>@gitlab-redis-cache' gitlab_rails['redis_cache_instance'] = 'redis://:<REDIS_PRIMARY_PASSWORD_OF_FIRST_CLUSTER>@gitlab-redis-cache'
gitlab_rails['redis_cache_sentinels'] = [ gitlab_rails['redis_cache_sentinels'] = [
{host: '10.6.0.71', port: 26379}, {host: '10.6.0.51', port: 26379},
{host: '10.6.0.72', port: 26379}, {host: '10.6.0.52', port: 26379},
{host: '10.6.0.73', port: 26379}, {host: '10.6.0.53', port: 26379},
] ]
## Second cluster that will host the queues, shared state, and actioncable ## Second cluster that will host the persistent queues, shared state, and actioncable
gitlab_rails['redis_queues_instance'] = 'redis://:<REDIS_PRIMARY_PASSWORD_OF_SECOND_CLUSTER>@gitlab-redis-persistent' gitlab_rails['redis_queues_instance'] = 'redis://:<REDIS_PRIMARY_PASSWORD_OF_SECOND_CLUSTER>@gitlab-redis-persistent'
gitlab_rails['redis_shared_state_instance'] = 'redis://:<REDIS_PRIMARY_PASSWORD_OF_SECOND_CLUSTER>@gitlab-redis-persistent' gitlab_rails['redis_shared_state_instance'] = 'redis://:<REDIS_PRIMARY_PASSWORD_OF_SECOND_CLUSTER>@gitlab-redis-persistent'
gitlab_rails['redis_actioncable_instance'] = 'redis://:<REDIS_PRIMARY_PASSWORD_OF_SECOND_CLUSTER>@gitlab-redis-persistent' gitlab_rails['redis_actioncable_instance'] = 'redis://:<REDIS_PRIMARY_PASSWORD_OF_SECOND_CLUSTER>@gitlab-redis-persistent'
gitlab_rails['redis_queues_sentinels'] = [ gitlab_rails['redis_queues_sentinels'] = [
{host: '10.6.0.81', port: 26379}, {host: '10.6.0.61', port: 26379},
{host: '10.6.0.82', port: 26379}, {host: '10.6.0.62', port: 26379},
{host: '10.6.0.83', port: 26379}, {host: '10.6.0.63', port: 26379},
] ]
gitlab_rails['redis_shared_state_sentinels'] = [ gitlab_rails['redis_shared_state_sentinels'] = [
{host: '10.6.0.81', port: 26379}, {host: '10.6.0.61', port: 26379},
{host: '10.6.0.82', port: 26379}, {host: '10.6.0.62', port: 26379},
{host: '10.6.0.83', port: 26379}, {host: '10.6.0.63', port: 26379},
] ]
gitlab_rails['redis_actioncable_sentinels'] = [ gitlab_rails['redis_actioncable_sentinels'] = [
{host: '10.6.0.81', port: 26379}, {host: '10.6.0.61', port: 26379},
{host: '10.6.0.82', port: 26379}, {host: '10.6.0.62', port: 26379},
{host: '10.6.0.83', port: 26379}, {host: '10.6.0.63', port: 26379},
] ]
# Gitaly Cluster # Gitaly Cluster
...@@ -2032,30 +1794,30 @@ On each node perform the following: ...@@ -2032,30 +1794,30 @@ On each node perform the following:
gitlab_rails['redis_cache_instance'] = 'redis://:<REDIS_PRIMARY_PASSWORD_OF_FIRST_CLUSTER>@gitlab-redis-cache' gitlab_rails['redis_cache_instance'] = 'redis://:<REDIS_PRIMARY_PASSWORD_OF_FIRST_CLUSTER>@gitlab-redis-cache'
gitlab_rails['redis_cache_sentinels'] = [ gitlab_rails['redis_cache_sentinels'] = [
{host: '10.6.0.71', port: 26379}, {host: '10.6.0.51', port: 26379},
{host: '10.6.0.72', port: 26379}, {host: '10.6.0.52', port: 26379},
{host: '10.6.0.73', port: 26379}, {host: '10.6.0.53', port: 26379},
] ]
## Second cluster that will host the queues, shared state, and actionable ## Second cluster that will host the persistent queues, shared state, and actionable
gitlab_rails['redis_queues_instance'] = 'redis://:<REDIS_PRIMARY_PASSWORD_OF_SECOND_CLUSTER>@gitlab-redis-persistent' gitlab_rails['redis_queues_instance'] = 'redis://:<REDIS_PRIMARY_PASSWORD_OF_SECOND_CLUSTER>@gitlab-redis-persistent'
gitlab_rails['redis_shared_state_instance'] = 'redis://:<REDIS_PRIMARY_PASSWORD_OF_SECOND_CLUSTER>@gitlab-redis-persistent' gitlab_rails['redis_shared_state_instance'] = 'redis://:<REDIS_PRIMARY_PASSWORD_OF_SECOND_CLUSTER>@gitlab-redis-persistent'
gitlab_rails['redis_actioncable_instance'] = 'redis://:<REDIS_PRIMARY_PASSWORD_OF_SECOND_CLUSTER>@gitlab-redis-persistent' gitlab_rails['redis_actioncable_instance'] = 'redis://:<REDIS_PRIMARY_PASSWORD_OF_SECOND_CLUSTER>@gitlab-redis-persistent'
gitlab_rails['redis_queues_sentinels'] = [ gitlab_rails['redis_queues_sentinels'] = [
{host: '10.6.0.81', port: 26379}, {host: '10.6.0.61', port: 26379},
{host: '10.6.0.82', port: 26379}, {host: '10.6.0.62', port: 26379},
{host: '10.6.0.83', port: 26379}, {host: '10.6.0.63', port: 26379},
] ]
gitlab_rails['redis_shared_state_sentinels'] = [ gitlab_rails['redis_shared_state_sentinels'] = [
{host: '10.6.0.81', port: 26379}, {host: '10.6.0.61', port: 26379},
{host: '10.6.0.82', port: 26379}, {host: '10.6.0.62', port: 26379},
{host: '10.6.0.83', port: 26379}, {host: '10.6.0.63', port: 26379},
] ]
gitlab_rails['redis_actioncable_sentinels'] = [ gitlab_rails['redis_actioncable_sentinels'] = [
{host: '10.6.0.81', port: 26379}, {host: '10.6.0.61', port: 26379},
{host: '10.6.0.82', port: 26379}, {host: '10.6.0.62', port: 26379},
{host: '10.6.0.83', port: 26379}, {host: '10.6.0.63', port: 26379},
] ]
# Set the network addresses that the exporters used for monitoring will listen on # Set the network addresses that the exporters used for monitoring will listen on
...@@ -2127,7 +1889,7 @@ On each node perform the following: ...@@ -2127,7 +1889,7 @@ On each node perform the following:
1. [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. 1. [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect.
1. If you're [using NFS](#configure-nfs-optional): 1. If you're [using NFS](#configure-nfs):
1. If necessary, install the NFS client utility packages using the following 1. If necessary, install the NFS client utility packages using the following
commands: commands:
...@@ -2271,7 +2033,7 @@ To configure the Monitoring node: ...@@ -2271,7 +2033,7 @@ To configure the Monitoring node:
## Configure the object storage ## Configure the object storage
GitLab supports using an object storage service for holding numerous types of data. GitLab supports using an object storage service for holding numerous types of data.
It's recommended over [NFS](#configure-nfs-optional) and in general it's better It's recommended over [NFS](#configure-nfs) and in general it's better
in larger setups as object storage is typically much more performant, reliable, in larger setups as object storage is typically much more performant, reliable,
and scalable. and scalable.
...@@ -2341,7 +2103,7 @@ cluster alongside your instance, read how to ...@@ -2341,7 +2103,7 @@ cluster alongside your instance, read how to
</a> </a>
</div> </div>
## Configure NFS (optional) ## Configure NFS
[Object storage](#configure-the-object-storage), along with [Gitaly](#configure-gitaly) [Object storage](#configure-the-object-storage), along with [Gitaly](#configure-gitaly)
are recommended over NFS wherever possible for improved performance. If you intend are recommended over NFS wherever possible for improved performance. If you intend
...@@ -2415,10 +2177,8 @@ services where applicable): ...@@ -2415,10 +2177,8 @@ services where applicable):
| PostgreSQL<sup>1</sup> | 3 | 8 vCPU, 30 GB memory | `n1-standard-8` | | PostgreSQL<sup>1</sup> | 3 | 8 vCPU, 30 GB memory | `n1-standard-8` |
| PgBouncer<sup>1</sup> | 3 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | | PgBouncer<sup>1</sup> | 3 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` |
| Internal load balancing node<sup>3</sup> | 1 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | | Internal load balancing node<sup>3</sup> | 1 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` |
| Redis - Cache<sup>2</sup> | 3 | 4 vCPU, 15 GB memory | `n1-standard-4` | | Redis/Sentinel - Cache<sup>2</sup> | 3 | 4 vCPU, 15 GB memory | `n1-standard-4` |
| Redis - Queues / Shared State<sup>2</sup> | 3 | 4 vCPU, 15 GB memory | `n1-standard-4` | | Redis/Sentinel - Persistent<sup>2</sup> | 3 | 4 vCPU, 15 GB memory | `n1-standard-4` |
| Redis Sentinel - Cache<sup>2</sup> | 3 | 1 vCPU, 3.75 GB memory | `n1-standard-1` |
| Redis Sentinel - Queues / Shared State<sup>2</sup> | 3 | 1 vCPU, 3.75 GB memory | `n1-standard-1` |
| Gitaly | 3 | 16 vCPU, 60 GB memory | `n1-standard-16` | | Gitaly | 3 | 16 vCPU, 60 GB memory | `n1-standard-16` |
| Praefect | 3 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | | Praefect | 3 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` |
| Praefect PostgreSQL<sup>1</sup> | 1+ | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | | Praefect PostgreSQL<sup>1</sup> | 1+ | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` |
...@@ -2474,11 +2234,6 @@ card "Database" as database { ...@@ -2474,11 +2234,6 @@ card "Database" as database {
card "redis" as redis { card "redis" as redis {
collections "**Redis Persistent** x3" as redis_persistent #FF6347 collections "**Redis Persistent** x3" as redis_persistent #FF6347
collections "**Redis Cache** x3" as redis_cache #FF6347 collections "**Redis Cache** x3" as redis_cache #FF6347
collections "**Redis Persistent Sentinel** x3" as redis_persistent_sentinel #FF6347
collections "**Redis Cache Sentinel** x3"as redis_cache_sentinel #FF6347
redis_persistent <.[#FF6347]- redis_persistent_sentinel
redis_cache <.[#FF6347]- redis_cache_sentinel
} }
cloud "**Object Storage**" as object_storage #white cloud "**Object Storage**" as object_storage #white
......
doc/administration/reference_architectures/25k_users.md
View file @ cdec6546
...@@ -22,10 +22,8 @@ full list of reference architectures, see ...@@ -22,10 +22,8 @@ full list of reference architectures, see
| PostgreSQL<sup>1</sup> | 3 | 16 vCPU, 60 GB memory | `n1-standard-16` | `m5.4xlarge` | `D16s v3` | | PostgreSQL<sup>1</sup> | 3 | 16 vCPU, 60 GB memory | `n1-standard-16` | `m5.4xlarge` | `D16s v3` |
| PgBouncer<sup>1</sup> | 3 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | `F2s v2` | | PgBouncer<sup>1</sup> | 3 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | `F2s v2` |
| Internal load balancing node<sup>3</sup> | 1 | 4 vCPU, 3.6GB memory | `n1-highcpu-4` | `c5.large` | `F2s v2` | | Internal load balancing node<sup>3</sup> | 1 | 4 vCPU, 3.6GB memory | `n1-highcpu-4` | `c5.large` | `F2s v2` |
| Redis - Cache<sup>2</sup> | 3 | 4 vCPU, 15 GB memory | `n1-standard-4` | `m5.xlarge` | `D4s v3` | | Redis/Sentinel - Cache<sup>2</sup> | 3 | 4 vCPU, 15 GB memory | `n1-standard-4` | `m5.xlarge` | `D4s v3` |
| Redis - Queues / Shared State<sup>2</sup> | 3 | 4 vCPU, 15 GB memory | `n1-standard-4` | `m5.xlarge` | `D4s v3` | | Redis/Sentinel - Persistent<sup>2</sup> | 3 | 4 vCPU, 15 GB memory | `n1-standard-4` | `m5.large` | `D4s v3` |
| Redis Sentinel - Cache<sup>2</sup> | 3 | 1 vCPU, 3.75 GB memory | `n1-standard-1` | `c5.large` | `A1 v2` |
| Redis Sentinel - Queues / Shared State<sup>2</sup>| 3 | 1 vCPU, 3.75 GB memory | `n1-standard-1` | `c5.large` | `A1 v2` |
| Gitaly | 3 | 32 vCPU, 120 GB memory | `n1-standard-32` | `m5.8xlarge` | `D32s v3` | | Gitaly | 3 | 32 vCPU, 120 GB memory | `n1-standard-32` | `m5.8xlarge` | `D32s v3` |
| Praefect | 3 | 4 vCPU, 3.6 GB memory | `n1-highcpu-4` | `c5.xlarge` | `F4s v2` | | Praefect | 3 | 4 vCPU, 3.6 GB memory | `n1-highcpu-4` | `c5.xlarge` | `F4s v2` |
| Praefect PostgreSQL<sup>1</sup> | 1+ | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | `F2s v2` | | Praefect PostgreSQL<sup>1</sup> | 1+ | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | `F2s v2` |
...@@ -33,7 +31,7 @@ full list of reference architectures, see ...@@ -33,7 +31,7 @@ full list of reference architectures, see
| GitLab Rails | 5 | 32 vCPU, 28.8 GB memory | `n1-highcpu-32` | `c5.9xlarge` | `F32s v2` | | GitLab Rails | 5 | 32 vCPU, 28.8 GB memory | `n1-highcpu-32` | `c5.9xlarge` | `F32s v2` |
| Monitoring node | 1 | 4 vCPU, 3.6 GB memory | `n1-highcpu-4` | `c5.xlarge` | `F4s v2` | | Monitoring node | 1 | 4 vCPU, 3.6 GB memory | `n1-highcpu-4` | `c5.xlarge` | `F4s v2` |
| Object storage<sup>4</sup> | n/a | n/a | n/a | n/a | n/a | | Object storage<sup>4</sup> | n/a | n/a | n/a | n/a | n/a |
| NFS server (optional, not recommended) | 1 | 4 vCPU, 3.6 GB memory | `n1-highcpu-4` | `c5.xlarge` | `F4s v2` | | NFS server (non-Gitaly) | 1 | 4 vCPU, 3.6 GB memory | `n1-highcpu-4` | `c5.xlarge` | `F4s v2` |
<!-- Disable ordered list rule https://github.com/DavidAnson/markdownlint/blob/main/doc/Rules.md#md029---ordered-list-item-prefix --> <!-- Disable ordered list rule https://github.com/DavidAnson/markdownlint/blob/main/doc/Rules.md#md029---ordered-list-item-prefix -->
<!-- markdownlint-disable MD029 --> <!-- markdownlint-disable MD029 -->
...@@ -82,11 +80,6 @@ card "Database" as database { ...@@ -82,11 +80,6 @@ card "Database" as database {
card "redis" as redis { card "redis" as redis {
collections "**Redis Persistent** x3" as redis_persistent #FF6347 collections "**Redis Persistent** x3" as redis_persistent #FF6347
collections "**Redis Cache** x3" as redis_cache #FF6347 collections "**Redis Cache** x3" as redis_cache #FF6347
collections "**Redis Persistent Sentinel** x3" as redis_persistent_sentinel #FF6347
collections "**Redis Cache Sentinel** x3"as redis_cache_sentinel #FF6347
redis_persistent <.[#FF6347]- redis_persistent_sentinel
redis_cache <.[#FF6347]- redis_cache_sentinel
} }
cloud "**Object Storage**" as object_storage #white cloud "**Object Storage**" as object_storage #white
...@@ -137,8 +130,7 @@ our [Sysbench](https://github.com/akopytov/sysbench)-based ...@@ -137,8 +130,7 @@ our [Sysbench](https://github.com/akopytov/sysbench)-based
Due to better performance and availability, for data objects (such as LFS, Due to better performance and availability, for data objects (such as LFS,
uploads, or artifacts), using an [object storage service](#configure-the-object-storage) uploads, or artifacts), using an [object storage service](#configure-the-object-storage)
is recommended instead of using NFS. Using an object storage service also is recommended.
doesn't require you to provision and maintain a node.
It's also worth noting that at this time [Praefect requires its own database server](../gitaly/praefect.md#postgresql) and It's also worth noting that at this time [Praefect requires its own database server](../gitaly/praefect.md#postgresql) and
that to achieve full High Availability a third-party PostgreSQL database solution will be required. that to achieve full High Availability a third-party PostgreSQL database solution will be required.
...@@ -169,10 +161,9 @@ To set up GitLab and its components to accommodate up to 25,000 users: ...@@ -169,10 +161,9 @@ To set up GitLab and its components to accommodate up to 25,000 users:
used for shared data objects. used for shared data objects.
1. [Configure Advanced Search](#configure-advanced-search) (optional) for faster, 1. [Configure Advanced Search](#configure-advanced-search) (optional) for faster,
more advanced code search across your entire GitLab instance. more advanced code search across your entire GitLab instance.
1. [Configure NFS](#configure-nfs-optional) (optional, and not recommended) 1. [Configure NFS](#configure-nfs)
to have shared disk storage service as an alternative to Gitaly or object to have shared disk storage service for certain GitLab operations (non
storage. You can skip this step if you're not using GitLab Pages (which Gitaly or Object Storage).
requires NFS).
The servers start on the same 10.6.0.0/24 private network range, and can The servers start on the same 10.6.0.0/24 private network range, and can
connect to each other freely on these addresses. connect to each other freely on these addresses.
...@@ -193,15 +184,9 @@ The following list includes descriptions of each server and its assigned IP: ...@@ -193,15 +184,9 @@ The following list includes descriptions of each server and its assigned IP:
- `10.6.0.51`: Redis - Cache Primary - `10.6.0.51`: Redis - Cache Primary
- `10.6.0.52`: Redis - Cache Replica 1 - `10.6.0.52`: Redis - Cache Replica 1
- `10.6.0.53`: Redis - Cache Replica 2 - `10.6.0.53`: Redis - Cache Replica 2
- `10.6.0.71`: Sentinel - Cache 1 - `10.6.0.61`: Redis - Persistent Primary
- `10.6.0.72`: Sentinel - Cache 2 - `10.6.0.62`: Redis - Persistent Replica 1
- `10.6.0.73`: Sentinel - Cache 3 - `10.6.0.63`: Redis - Persistent Replica 2
- `10.6.0.61`: Redis - Queues Primary
- `10.6.0.62`: Redis - Queues Replica 1
- `10.6.0.63`: Redis - Queues Replica 2
- `10.6.0.81`: Sentinel - Queues 1
- `10.6.0.82`: Sentinel - Queues 2
- `10.6.0.83`: Sentinel - Queues 3
- `10.6.0.91`: Gitaly 1 - `10.6.0.91`: Gitaly 1
- `10.6.0.92`: Gitaly 2 - `10.6.0.92`: Gitaly 2
- `10.6.0.93`: Gitaly 3 - `10.6.0.93`: Gitaly 3
...@@ -794,15 +779,9 @@ to be used with GitLab. The following IPs will be used as an example: ...@@ -794,15 +779,9 @@ to be used with GitLab. The following IPs will be used as an example:
- `10.6.0.51`: Redis - Cache Primary - `10.6.0.51`: Redis - Cache Primary
- `10.6.0.52`: Redis - Cache Replica 1 - `10.6.0.52`: Redis - Cache Replica 1
- `10.6.0.53`: Redis - Cache Replica 2 - `10.6.0.53`: Redis - Cache Replica 2
- `10.6.0.71`: Sentinel - Cache 1 - `10.6.0.61`: Redis - Persistent Primary
- `10.6.0.72`: Sentinel - Cache 2 - `10.6.0.62`: Redis - Persistent Replica 1
- `10.6.0.73`: Sentinel - Cache 3 - `10.6.0.63`: Redis - Persistent Replica 2
- `10.6.0.61`: Redis - Queues Primary
- `10.6.0.62`: Redis - Queues Replica 1
- `10.6.0.63`: Redis - Queues Replica 2
- `10.6.0.81`: Sentinel - Queues 1
- `10.6.0.82`: Sentinel - Queues 2
- `10.6.0.83`: Sentinel - Queues 3
### Providing your own Redis instance ### Providing your own Redis instance
...@@ -814,7 +793,7 @@ optional count argument to SPOP, which is required for [Merge Trains](../../ci/p ...@@ -814,7 +793,7 @@ optional count argument to SPOP, which is required for [Merge Trains](../../ci/p
Note the Redis node's IP address or hostname, port, and password (if required). Note the Redis node's IP address or hostname, port, and password (if required).
These will be necessary later when configuring the [GitLab application servers](#configure-gitlab-rails). These will be necessary later when configuring the [GitLab application servers](#configure-gitlab-rails).
### Configure the Redis and Sentinel Cache cluster ### Configure the Redis Cache cluster
This is the section where we install and set up the new Redis Cache instances. This is the section where we install and set up the new Redis Cache instances.
...@@ -832,10 +811,14 @@ a node and change its status from primary to replica (and vice versa). ...@@ -832,10 +811,14 @@ a node and change its status from primary to replica (and vice versa).
1. Edit `/etc/gitlab/gitlab.rb` and add the contents: 1. Edit `/etc/gitlab/gitlab.rb` and add the contents:
```ruby ```ruby
# Specify server role as 'redis_master_role' and enable Consul agent # Specify server role as 'redis_sentinel_role' 'redis_master_role'
roles(['redis_master_role', 'consul_role'] roles ['redis_sentinel_role', 'redis_master_role', 'consul_role']
# IP address pointing to a local IP that the other machines can reach to. # Set IP bind address and Quorum number for Redis Sentinel service
sentinel['bind'] = '0.0.0.0'
sentinel['quorum'] = 2
# IP address pointing to a local IP that the other machines can reach.
# You can also set bind to '0.0.0.0' which listen in all interfaces. # You can also set bind to '0.0.0.0' which listen in all interfaces.
# If you really need to bind to an external accessible IP, make # If you really need to bind to an external accessible IP, make
# sure you add extra firewall rules to prevent unauthorized access. # sure you add extra firewall rules to prevent unauthorized access.
...@@ -845,8 +828,18 @@ a node and change its status from primary to replica (and vice versa). ...@@ -845,8 +828,18 @@ a node and change its status from primary to replica (and vice versa).
# machines to connect to it. # machines to connect to it.
redis['port'] = 6379 redis['port'] = 6379
# Port of primary Redis server for Sentinel, uncomment to change to non default.
# Defaults to `6379`.
# redis['master_port'] = 6379
# Set up password authentication for Redis (use the same password in all nodes). # Set up password authentication for Redis (use the same password in all nodes).
redis['password'] = 'REDIS_PRIMARY_PASSWORD_OF_FIRST_CLUSTER' redis['master_password'] = 'REDIS_PRIMARY_PASSWORD_OF_FIRST_CLUSTER'
# Must be the same in every Redis node.
redis['master_name'] = 'gitlab-redis-cache'
# The IP of this primary Redis node.
redis['master_ip'] = '10.6.0.51'
# Set the Redis Cache instance as an LRU # Set the Redis Cache instance as an LRU
# 90% of available RAM in MB # 90% of available RAM in MB
...@@ -880,10 +873,6 @@ a node and change its status from primary to replica (and vice versa). ...@@ -880,10 +873,6 @@ a node and change its status from primary to replica (and vice versa).
1. [Reconfigure Omnibus GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. 1. [Reconfigure Omnibus GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect.
You can specify multiple roles, like sentinel and Redis, as:
`roles(['redis_sentinel_role', 'redis_master_role'])`. Read more about
[roles](https://docs.gitlab.com/omnibus/roles/).
#### Configure the replica Redis Cache nodes #### Configure the replica Redis Cache nodes
1. SSH in to the **replica** Redis server. 1. SSH in to the **replica** Redis server.
...@@ -891,14 +880,18 @@ You can specify multiple roles, like sentinel and Redis, as: ...@@ -891,14 +880,18 @@ You can specify multiple roles, like sentinel and Redis, as:
package of your choice. Be sure to both follow _only_ installation steps 1 and 2 package of your choice. Be sure to both follow _only_ installation steps 1 and 2
on the page, and to select the correct Omnibus GitLab package, with the same version on the page, and to select the correct Omnibus GitLab package, with the same version
and type (Community or Enterprise editions) as your current install. and type (Community or Enterprise editions) as your current install.
1. Edit `/etc/gitlab/gitlab.rb` and add the contents: 1. Edit `/etc/gitlab/gitlab.rb` and add the same contents as the primary node in the previous section replacing `redis_master_node` with `redis_replica_node`:
```ruby ```ruby
# Specify server role as 'redis_replica_role' and enable Consul agent # Specify server role as 'redis_replica_role' and enable Consul agent
roles(['redis_replica_role', 'consul_role'] roles ['redis_sentinel_role', 'redis_replica_role', 'consul_role']
# IP address pointing to a local IP that the other machines can reach to. # Set IP bind address and Quorum number for Redis Sentinel service
# You can also set bind to '0.0.0.0' which listen in all interfaces. sentinel['bind'] = `0.0.0.0`
sentinel['quorum'] = 2
# IP address pointing to a local IP that the other machines can reach.
# Set bind to '0.0.0.0' to listen on all interfaces.
# If you really need to bind to an external accessible IP, make # If you really need to bind to an external accessible IP, make
# sure you add extra firewall rules to prevent unauthorized access. # sure you add extra firewall rules to prevent unauthorized access.
redis['bind'] = '10.6.0.52' redis['bind'] = '10.6.0.52'
...@@ -907,16 +900,19 @@ You can specify multiple roles, like sentinel and Redis, as: ...@@ -907,16 +900,19 @@ You can specify multiple roles, like sentinel and Redis, as:
# machines to connect to it. # machines to connect to it.
redis['port'] = 6379 redis['port'] = 6379
# The same password for Redis authentication you set up for the primary node. ## Port of primary Redis server for Sentinel, uncomment to change to non default. Defaults
redis['password'] = 'REDIS_PRIMARY_PASSWORD_OF_FIRST_CLUSTER' ## to `6379`.
#redis['master_port'] = 6379
# Set up password authentication for Redis and replicas (use the same password in all nodes).
redis['master_password'] = 'REDIS_PRIMARY_PASSWORD_OF_FIRST_CLUSTER'
# Must be the same in every Redis node
redis['master_name'] = 'gitlab-redis-cache'
# The IP of the primary Redis node. # The IP of the primary Redis node.
redis['master_ip'] = '10.6.0.51' redis['master_ip'] = '10.6.0.51'
# Port of primary Redis server, uncomment to change to non default. Defaults
# to `6379`.
#redis['master_port'] = 6379
# Set the Redis Cache instance as an LRU # Set the Redis Cache instance as an LRU
# 90% of available RAM in MB # 90% of available RAM in MB
redis['maxmemory'] = '13500mb' redis['maxmemory'] = '13500mb'
...@@ -952,136 +948,7 @@ You can specify multiple roles, like sentinel and Redis, as: ...@@ -952,136 +948,7 @@ You can specify multiple roles, like sentinel and Redis, as:
1. Go through the steps again for all the other replica nodes, and 1. Go through the steps again for all the other replica nodes, and
make sure to set up the IPs correctly. make sure to set up the IPs correctly.
You can specify multiple roles, like sentinel and Redis, as: Advanced [configuration options](https://docs.gitlab.com/omnibus/settings/redis.html) are supported and can be added if needed.
`roles(['redis_sentinel_role', 'redis_master_role'])`. Read more about
[roles](https://docs.gitlab.com/omnibus/roles/).
These values don't have to be changed again in `/etc/gitlab/gitlab.rb` after
a failover, as the nodes will be managed by the [Sentinels](#configure-the-sentinel-cache-nodes), and even after a
`gitlab-ctl reconfigure`, they will get their configuration restored by
the same Sentinels.
Advanced [configuration options](https://docs.gitlab.com/omnibus/settings/redis.html)
are supported and can be added if needed.
<div align="right">
<a type="button" class="btn btn-default" href="#setup-components">
Back to setup components <i class="fa fa-angle-double-up" aria-hidden="true"></i>
</a>
</div>
#### Configure the Sentinel Cache nodes
Now that the Redis servers are all set up, let's configure the Sentinel
servers. The following IPs will be used as an example:
- `10.6.0.71`: Sentinel - Cache 1
- `10.6.0.72`: Sentinel - Cache 2
- `10.6.0.73`: Sentinel - Cache 3
NOTE:
If you're using an external Redis Sentinel instance, be sure to exclude the
`requirepass` parameter from the Sentinel configuration. This parameter causes
clients to report `NOAUTH Authentication required.`.
[Redis Sentinel 3.2.x doesn't support password authentication](https://github.com/antirez/redis/issues/3279).
To configure the Sentinel Cache server:
1. SSH in to the server that will host Consul/Sentinel.
1. [Download and install](https://about.gitlab.com/install/) the Omnibus GitLab
package of your choice. Be sure to both follow _only_ installation steps 1 and 2
on the page, and to select the correct Omnibus GitLab package, with the same version
and type (Community or Enterprise editions) as your current install.
1. Edit `/etc/gitlab/gitlab.rb` and add the contents:
```ruby
roles(['redis_sentinel_role', 'consul_role'])
## Must be the same in every sentinel node
redis['master_name'] = 'gitlab-redis-cache'
## The same password for Redis authentication you set up for the primary node.
redis['master_password'] = 'REDIS_PRIMARY_PASSWORD_OF_FIRST_CLUSTER'
## The IP of the primary Redis node.
redis['master_ip'] = '10.6.0.51'
## Define a port so Redis can listen for TCP requests which will allow other
## machines to connect to it.
redis['port'] = 6379
## Port of primary Redis server, uncomment to change to non default. Defaults
## to `6379`.
#redis['master_port'] = 6379
## Configure Sentinel's IP
sentinel['bind'] = '10.6.0.71'
## Port that Sentinel listens on, uncomment to change to non default. Defaults
## to `26379`.
#sentinel['port'] = 26379
## Quorum must reflect the amount of voting sentinels it take to start a failover.
## Value must NOT be greater then the amount of sentinels.
##
## The quorum can be used to tune Sentinel in two ways:
## 1. If a the quorum is set to a value smaller than the majority of Sentinels
## we deploy, we are basically making Sentinel more sensible to primary failures,
## triggering a failover as soon as even just a minority of Sentinels is no longer
## able to talk with the primary.
## 1. If a quorum is set to a value greater than the majority of Sentinels, we are
## making Sentinel able to failover only when there are a very large number (larger
## than majority) of well connected Sentinels which agree about the primary being down.s
sentinel['quorum'] = 2
## Consider unresponsive server down after x amount of ms.
#sentinel['down_after_milliseconds'] = 10000
## Specifies the failover timeout in milliseconds. It is used in many ways:
##
## - The time needed to re-start a failover after a previous failover was
## already tried against the same primary by a given Sentinel, is two
## times the failover timeout.
##
## - The time needed for a replica replicating to a wrong primary according
## to a Sentinel current configuration, to be forced to replicate
## with the right primary, is exactly the failover timeout (counting since
## the moment a Sentinel detected the misconfiguration).
##
## - The time needed to cancel a failover that is already in progress but
## did not produced any configuration change (REPLICAOF NO ONE yet not
## acknowledged by the promoted replica).
##
## - The maximum time a failover in progress waits for all the replica to be
## reconfigured as replicas of the new primary. However even after this time
## the replicas will be reconfigured by the Sentinels anyway, but not with
## the exact parallel-syncs progression as specified.
#sentinel['failover_timeout'] = 60000
## Enable service discovery for Prometheus
consul['monitoring_service_discovery'] = true
## The IPs of the Consul server nodes
## You can also use FQDNs and intermix them with IPs
consul['configuration'] = {
retry_join: %w(10.6.0.11 10.6.0.12 10.6.0.13),
}
# Set the network addresses that the exporters will listen on
node_exporter['listen_address'] = '0.0.0.0:9100'
redis_exporter['listen_address'] = '0.0.0.0:9121'
# Prevent database migrations from running on upgrade automatically
gitlab_rails['auto_migrate'] = false
```
1. Copy the `/etc/gitlab/gitlab-secrets.json` file from the first Omnibus node you configured and add or replace
the file of the same name on this server. If this is the first Omnibus node you are configuring then you can skip this step.
1. [Reconfigure Omnibus GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect.
1. Go through the steps again for all the other Consul/Sentinel nodes, and
make sure you set up the correct IPs.
<div align="right"> <div align="right">
<a type="button" class="btn btn-default" href="#setup-components"> <a type="button" class="btn btn-default" href="#setup-components">
...@@ -1089,15 +956,15 @@ To configure the Sentinel Cache server: ...@@ -1089,15 +956,15 @@ To configure the Sentinel Cache server:
</a> </a>
</div> </div>
### Configure the Redis and Sentinel Queues cluster ### Configure the Redis Persistent cluster
This is the section where we install and set up the new Redis Queues instances. This is the section where we install and set up the new Redis Persistent instances.
Both the primary and replica Redis nodes need the same password defined in Both the primary and replica Redis nodes need the same password defined in
`redis['password']`. At any time during a failover, the Sentinels can reconfigure `redis['password']`. At any time during a failover, the Sentinels can reconfigure
a node and change its status from primary to replica (and vice versa). a node and change its status from primary to replica (and vice versa).
#### Configure the primary Redis Queues node #### Configure the primary Redis Persistent node
1. SSH in to the **Primary** Redis server. 1. SSH in to the **Primary** Redis server.
1. [Download and install](https://about.gitlab.com/install/) the Omnibus GitLab 1. [Download and install](https://about.gitlab.com/install/) the Omnibus GitLab
...@@ -1107,8 +974,12 @@ a node and change its status from primary to replica (and vice versa). ...@@ -1107,8 +974,12 @@ a node and change its status from primary to replica (and vice versa).
1. Edit `/etc/gitlab/gitlab.rb` and add the contents: 1. Edit `/etc/gitlab/gitlab.rb` and add the contents:
```ruby ```ruby
# Specify server role as 'redis_master_role' and enable Consul agent # Specify server roles as 'redis_sentinel_role' and 'redis_master_role'
roles(['redis_master_role', 'consul_role']) roles ['redis_sentinel_role', 'redis_master_role', 'consul_role']
# Set IP bind address and Quorum number for Redis Sentinel service
sentinel['bind'] = '0.0.0.0'
sentinel['quorum'] = 2
# IP address pointing to a local IP that the other machines can reach to. # IP address pointing to a local IP that the other machines can reach to.
# You can also set bind to '0.0.0.0' which listen in all interfaces. # You can also set bind to '0.0.0.0' which listen in all interfaces.
...@@ -1120,8 +991,19 @@ a node and change its status from primary to replica (and vice versa). ...@@ -1120,8 +991,19 @@ a node and change its status from primary to replica (and vice versa).
# machines to connect to it. # machines to connect to it.
redis['port'] = 6379 redis['port'] = 6379
# Set up password authentication for Redis (use the same password in all nodes). ## Port of primary Redis server for Sentinel, uncomment to change to non default. Defaults
redis['password'] = 'REDIS_PRIMARY_PASSWORD_OF_SECOND_CLUSTER' ## to `6379`.
#redis['master_port'] = 6379
# Set up password authentication for Redis and replicas (use the same password in all nodes).
redis['password'] = 'REDIS_PRIMARY_PASSWORD_OF_FIRST_CLUSTER'
redis['master_password'] = 'REDIS_PRIMARY_PASSWORD_OF_SECOND_CLUSTER'
## Must be the same in every Redis node
redis['master_name'] = 'gitlab-redis-persistent'
## The IP of this primary Redis node.
redis['master_ip'] = '10.6.0.61'
## Enable service discovery for Prometheus ## Enable service discovery for Prometheus
consul['monitoring_service_discovery'] = true consul['monitoring_service_discovery'] = true
...@@ -1145,13 +1027,9 @@ a node and change its status from primary to replica (and vice versa). ...@@ -1145,13 +1027,9 @@ a node and change its status from primary to replica (and vice versa).
1. [Reconfigure Omnibus GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. 1. [Reconfigure Omnibus GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect.
You can specify multiple roles, like sentinel and Redis, as: #### Configure the replica Redis Persistent nodes
`roles(['redis_sentinel_role', 'redis_master_role'])`. Read more about
[roles](https://docs.gitlab.com/omnibus/roles/).
#### Configure the replica Redis Queues nodes
1. SSH in to the **replica** Redis Queue server. 1. SSH in to the **replica** Redis Persistent server.
1. [Download and install](https://about.gitlab.com/install/) the Omnibus GitLab 1. [Download and install](https://about.gitlab.com/install/) the Omnibus GitLab
package of your choice. Be sure to both follow _only_ installation steps 1 and 2 package of your choice. Be sure to both follow _only_ installation steps 1 and 2
on the page, and to select the correct Omnibus GitLab package, with the same version on the page, and to select the correct Omnibus GitLab package, with the same version
...@@ -1160,7 +1038,11 @@ You can specify multiple roles, like sentinel and Redis, as: ...@@ -1160,7 +1038,11 @@ You can specify multiple roles, like sentinel and Redis, as:
```ruby ```ruby
# Specify server role as 'redis_replica_role' and enable Consul agent # Specify server role as 'redis_replica_role' and enable Consul agent
roles(['redis_replica_role', 'consul_role']) roles ['redis_sentinel_role', 'redis_replica_role', 'consul_role']
# Set IP bind address and Quorum number for Redis Sentinel service
sentinel['bind'] = '0.0.0.0'
sentinel['quorum'] = 2
# IP address pointing to a local IP that the other machines can reach to. # IP address pointing to a local IP that the other machines can reach to.
# You can also set bind to '0.0.0.0' which listen in all interfaces. # You can also set bind to '0.0.0.0' which listen in all interfaces.
...@@ -1172,16 +1054,19 @@ You can specify multiple roles, like sentinel and Redis, as: ...@@ -1172,16 +1054,19 @@ You can specify multiple roles, like sentinel and Redis, as:
# machines to connect to it. # machines to connect to it.
redis['port'] = 6379 redis['port'] = 6379
## Port of primary Redis server for Sentinel, uncomment to change to non default. Defaults
## to `6379`.
#redis['master_port'] = 6379
# The same password for Redis authentication you set up for the primary node. # The same password for Redis authentication you set up for the primary node.
redis['password'] = 'REDIS_PRIMARY_PASSWORD_OF_SECOND_CLUSTER' redis['master_password'] = 'REDIS_PRIMARY_PASSWORD_OF_SECOND_CLUSTER'
## Must be the same in every Redis node
redis['master_name'] = 'gitlab-redis-persistent'
# The IP of the primary Redis node. # The IP of the primary Redis node.
redis['master_ip'] = '10.6.0.61' redis['master_ip'] = '10.6.0.61'
# Port of primary Redis server, uncomment to change to non default. Defaults
# to `6379`.
#redis['master_port'] = 6379
## Enable service discovery for Prometheus ## Enable service discovery for Prometheus
consul['monitoring_service_discovery'] = true consul['monitoring_service_discovery'] = true
...@@ -1211,15 +1096,6 @@ You can specify multiple roles, like sentinel and Redis, as: ...@@ -1211,15 +1096,6 @@ You can specify multiple roles, like sentinel and Redis, as:
1. Go through the steps again for all the other replica nodes, and 1. Go through the steps again for all the other replica nodes, and
make sure to set up the IPs correctly. make sure to set up the IPs correctly.
You can specify multiple roles, like sentinel and Redis, as:
`roles(['redis_sentinel_role', 'redis_master_role'])`. Read more about
[roles](https://docs.gitlab.com/omnibus/roles/).
These values don't have to be changed again in `/etc/gitlab/gitlab.rb` after
a failover, as the nodes will be managed by the [Sentinels](#configure-the-sentinel-queues-nodes), and even after a
`gitlab-ctl reconfigure`, they will get their configuration restored by
the same Sentinels.
Advanced [configuration options](https://docs.gitlab.com/omnibus/settings/redis.html) Advanced [configuration options](https://docs.gitlab.com/omnibus/settings/redis.html)
are supported and can be added if needed. are supported and can be added if needed.
...@@ -1229,136 +1105,10 @@ are supported and can be added if needed. ...@@ -1229,136 +1105,10 @@ are supported and can be added if needed.
</a> </a>
</div> </div>
#### Configure the Sentinel Queues nodes
Now that the Redis servers are all set up, let's configure the Sentinel
servers. The following IPs will be used as an example:
- `10.6.0.81`: Sentinel - Queues 1
- `10.6.0.82`: Sentinel - Queues 2
- `10.6.0.83`: Sentinel - Queues 3
NOTE:
If you're using an external Redis Sentinel instance, be sure to exclude the
`requirepass` parameter from the Sentinel configuration. This parameter causes
clients to report `NOAUTH Authentication required.`.
[Redis Sentinel 3.2.x doesn't support password authentication](https://github.com/antirez/redis/issues/3279).
To configure the Sentinel Queues server:
1. SSH in to the server that will host Sentinel.
1. [Download and install](https://about.gitlab.com/install/) the Omnibus GitLab
package of your choice. Be sure to both follow _only_ installation steps 1 and 2
on the page, and to select the correct Omnibus GitLab package, with the same version
and type (Community or Enterprise editions) as your current install.
1. Edit `/etc/gitlab/gitlab.rb` and add the contents:
```ruby
roles(['redis_sentinel_role', 'consul_role'])
## Must be the same in every sentinel node
redis['master_name'] = 'gitlab-redis-persistent'
## The same password for Redis authentication you set up for the primary node.
redis['master_password'] = 'REDIS_PRIMARY_PASSWORD_OF_SECOND_CLUSTER'
## The IP of the primary Redis node.
redis['master_ip'] = '10.6.0.61'
## Define a port so Redis can listen for TCP requests which will allow other
## machines to connect to it.
redis['port'] = 6379
## Port of primary Redis server, uncomment to change to non default. Defaults
## to `6379`.
#redis['master_port'] = 6379
## Configure Sentinel's IP
sentinel['bind'] = '10.6.0.81'
## Port that Sentinel listens on, uncomment to change to non default. Defaults
## to `26379`.
#sentinel['port'] = 26379
## Quorum must reflect the amount of voting sentinels it take to start a failover.
## Value must NOT be greater then the amount of sentinels.
##
## The quorum can be used to tune Sentinel in two ways:
## 1. If a the quorum is set to a value smaller than the majority of Sentinels
## we deploy, we are basically making Sentinel more sensible to primary failures,
## triggering a failover as soon as even just a minority of Sentinels is no longer
## able to talk with the primary.
## 1. If a quorum is set to a value greater than the majority of Sentinels, we are
## making Sentinel able to failover only when there are a very large number (larger
## than majority) of well connected Sentinels which agree about the primary being down.s
sentinel['quorum'] = 2
## Consider unresponsive server down after x amount of ms.
#sentinel['down_after_milliseconds'] = 10000
## Specifies the failover timeout in milliseconds. It is used in many ways:
##
## - The time needed to re-start a failover after a previous failover was
## already tried against the same primary by a given Sentinel, is two
## times the failover timeout.
##
## - The time needed for a replica replicating to a wrong primary according
## to a Sentinel current configuration, to be forced to replicate
## with the right primary, is exactly the failover timeout (counting since
## the moment a Sentinel detected the misconfiguration).
##
## - The time needed to cancel a failover that is already in progress but
## did not produced any configuration change (REPLICAOF NO ONE yet not
## acknowledged by the promoted replica).
##
## - The maximum time a failover in progress waits for all the replica to be
## reconfigured as replicas of the new primary. However even after this time
## the replicas will be reconfigured by the Sentinels anyway, but not with
## the exact parallel-syncs progression as specified.
#sentinel['failover_timeout'] = 60000
## Enable service discovery for Prometheus
consul['monitoring_service_discovery'] = true
## The IPs of the Consul server nodes
## You can also use FQDNs and intermix them with IPs
consul['configuration'] = {
retry_join: %w(10.6.0.11 10.6.0.12 10.6.0.13),
}
# Set the network addresses that the exporters will listen on
node_exporter['listen_address'] = '0.0.0.0:9100'
redis_exporter['listen_address'] = '0.0.0.0:9121'
# Prevent database migrations from running on upgrade automatically
gitlab_rails['auto_migrate'] = false
```
1. Copy the `/etc/gitlab/gitlab-secrets.json` file from the first Omnibus node you configured and add or replace
the file of the same name on this server. If this is the first Omnibus node you are configuring then you can skip this step.
1. To ensure database migrations are only run during reconfigure and not automatically on upgrade, run:
```shell
sudo touch /etc/gitlab/skip-auto-reconfigure
```
Only the primary GitLab application server should handle migrations.
1. [Reconfigure Omnibus GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect.
1. Go through the steps again for all the other Sentinel nodes, and
make sure you set up the correct IPs.
<div align="right">
<a type="button" class="btn btn-default" href="#setup-components">
Back to setup components <i class="fa fa-angle-double-up" aria-hidden="true"></i>
</a>
</div>
## Configure Gitaly Cluster ## Configure Gitaly Cluster
[Gitaly Cluster](../gitaly/praefect.md) is a GitLab provided and recommended fault tolerant solution for storing Git repositories. [Gitaly Cluster](../gitaly/praefect.md) is a GitLab-provided and recommended
fault tolerant solution for storing Git repositories.
In this configuration, every Git repository is stored on every Gitaly node in the cluster, with one being designated the primary, and failover occurs automatically if the primary node goes down. In this configuration, every Git repository is stored on every Gitaly node in the cluster, with one being designated the primary, and failover occurs automatically if the primary node goes down.
The recommended cluster setup includes the following components: The recommended cluster setup includes the following components:
...@@ -1875,30 +1625,30 @@ To configure the Sidekiq nodes, on each one: ...@@ -1875,30 +1625,30 @@ To configure the Sidekiq nodes, on each one:
gitlab_rails['redis_cache_instance'] = 'redis://:<REDIS_PRIMARY_PASSWORD_OF_FIRST_CLUSTER>@gitlab-redis-cache' gitlab_rails['redis_cache_instance'] = 'redis://:<REDIS_PRIMARY_PASSWORD_OF_FIRST_CLUSTER>@gitlab-redis-cache'
gitlab_rails['redis_cache_sentinels'] = [ gitlab_rails['redis_cache_sentinels'] = [
{host: '10.6.0.71', port: 26379}, {host: '10.6.0.51', port: 26379},
{host: '10.6.0.72', port: 26379}, {host: '10.6.0.52', port: 26379},
{host: '10.6.0.73', port: 26379}, {host: '10.6.0.53', port: 26379},
] ]
## Second cluster that will host the queues, shared state, and actioncable ## Second cluster that will host the persistent queues, shared state, and actioncable
gitlab_rails['redis_queues_instance'] = 'redis://:<REDIS_PRIMARY_PASSWORD_OF_SECOND_CLUSTER>@gitlab-redis-persistent' gitlab_rails['redis_queues_instance'] = 'redis://:<REDIS_PRIMARY_PASSWORD_OF_SECOND_CLUSTER>@gitlab-redis-persistent'
gitlab_rails['redis_shared_state_instance'] = 'redis://:<REDIS_PRIMARY_PASSWORD_OF_SECOND_CLUSTER>@gitlab-redis-persistent' gitlab_rails['redis_shared_state_instance'] = 'redis://:<REDIS_PRIMARY_PASSWORD_OF_SECOND_CLUSTER>@gitlab-redis-persistent'
gitlab_rails['redis_actioncable_instance'] = 'redis://:<REDIS_PRIMARY_PASSWORD_OF_SECOND_CLUSTER>@gitlab-redis-persistent' gitlab_rails['redis_actioncable_instance'] = 'redis://:<REDIS_PRIMARY_PASSWORD_OF_SECOND_CLUSTER>@gitlab-redis-persistent'
gitlab_rails['redis_queues_sentinels'] = [ gitlab_rails['redis_queues_sentinels'] = [
{host: '10.6.0.81', port: 26379}, {host: '10.6.0.61', port: 26379},
{host: '10.6.0.82', port: 26379}, {host: '10.6.0.62', port: 26379},
{host: '10.6.0.83', port: 26379}, {host: '10.6.0.63', port: 26379},
] ]
gitlab_rails['redis_shared_state_sentinels'] = [ gitlab_rails['redis_shared_state_sentinels'] = [
{host: '10.6.0.81', port: 26379}, {host: '10.6.0.61', port: 26379},
{host: '10.6.0.82', port: 26379}, {host: '10.6.0.62', port: 26379},
{host: '10.6.0.83', port: 26379}, {host: '10.6.0.63', port: 26379},
] ]
gitlab_rails['redis_actioncable_sentinels'] = [ gitlab_rails['redis_actioncable_sentinels'] = [
{host: '10.6.0.81', port: 26379}, {host: '10.6.0.61', port: 26379},
{host: '10.6.0.82', port: 26379}, {host: '10.6.0.62', port: 26379},
{host: '10.6.0.83', port: 26379}, {host: '10.6.0.63', port: 26379},
] ]
# Gitaly Cluster # Gitaly Cluster
...@@ -2052,9 +1802,9 @@ On each node perform the following: ...@@ -2052,9 +1802,9 @@ On each node perform the following:
gitlab_rails['redis_cache_instance'] = 'redis://:<REDIS_PRIMARY_PASSWORD_OF_FIRST_CLUSTER>@gitlab-redis-cache' gitlab_rails['redis_cache_instance'] = 'redis://:<REDIS_PRIMARY_PASSWORD_OF_FIRST_CLUSTER>@gitlab-redis-cache'
gitlab_rails['redis_cache_sentinels'] = [ gitlab_rails['redis_cache_sentinels'] = [
{host: '10.6.0.71', port: 26379}, {host: '10.6.0.51', port: 26379},
{host: '10.6.0.72', port: 26379}, {host: '10.6.0.52', port: 26379},
{host: '10.6.0.73', port: 26379}, {host: '10.6.0.53', port: 26379},
] ]
## Second cluster that will host the queues, shared state, and actionable ## Second cluster that will host the queues, shared state, and actionable
...@@ -2063,19 +1813,19 @@ On each node perform the following: ...@@ -2063,19 +1813,19 @@ On each node perform the following:
gitlab_rails['redis_actioncable_instance'] = 'redis://:<REDIS_PRIMARY_PASSWORD_OF_SECOND_CLUSTER>@gitlab-redis-persistent' gitlab_rails['redis_actioncable_instance'] = 'redis://:<REDIS_PRIMARY_PASSWORD_OF_SECOND_CLUSTER>@gitlab-redis-persistent'
gitlab_rails['redis_queues_sentinels'] = [ gitlab_rails['redis_queues_sentinels'] = [
{host: '10.6.0.81', port: 26379}, {host: '10.6.0.61', port: 26379},
{host: '10.6.0.82', port: 26379}, {host: '10.6.0.62', port: 26379},
{host: '10.6.0.83', port: 26379}, {host: '10.6.0.63', port: 26379},
] ]
gitlab_rails['redis_shared_state_sentinels'] = [ gitlab_rails['redis_shared_state_sentinels'] = [
{host: '10.6.0.81', port: 26379}, {host: '10.6.0.61', port: 26379},
{host: '10.6.0.82', port: 26379}, {host: '10.6.0.62', port: 26379},
{host: '10.6.0.83', port: 26379}, {host: '10.6.0.63', port: 26379},
] ]
gitlab_rails['redis_actioncable_sentinels'] = [ gitlab_rails['redis_actioncable_sentinels'] = [
{host: '10.6.0.81', port: 26379}, {host: '10.6.0.61', port: 26379},
{host: '10.6.0.82', port: 26379}, {host: '10.6.0.62', port: 26379},
{host: '10.6.0.83', port: 26379}, {host: '10.6.0.63', port: 26379},
] ]
# Set the network addresses that the exporters used for monitoring will listen on # Set the network addresses that the exporters used for monitoring will listen on
...@@ -2147,7 +1897,7 @@ On each node perform the following: ...@@ -2147,7 +1897,7 @@ On each node perform the following:
1. [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. 1. [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect.
1. If you're [using NFS](#configure-nfs-optional): 1. If you're [using NFS](#configure-nfs):
1. If necessary, install the NFS client utility packages using the following 1. If necessary, install the NFS client utility packages using the following
commands: commands:
...@@ -2289,9 +2039,9 @@ To configure the Monitoring node: ...@@ -2289,9 +2039,9 @@ To configure the Monitoring node:
## Configure the object storage ## Configure the object storage
GitLab supports using an object storage service for holding numerous types of data. GitLab supports using an object storage service for holding numerous types of data.
It's recommended over [NFS](#configure-nfs-optional) and in general it's better Object storage is also recommended over [NFS](#configure-nfs) and in general
in larger setups as object storage is typically much more performant, reliable, it's better in larger setups as object storage is typically much more performant,
and scalable. reliable, and scalable.
GitLab has been tested on a number of object storage providers: GitLab has been tested on a number of object storage providers:
...@@ -2359,7 +2109,7 @@ cluster alongside your instance, read how to ...@@ -2359,7 +2109,7 @@ cluster alongside your instance, read how to
</a> </a>
</div> </div>
## Configure NFS (optional) ## Configure NFS
[Object storage](#configure-the-object-storage), along with [Gitaly](#configure-gitaly) [Object storage](#configure-the-object-storage), along with [Gitaly](#configure-gitaly)
are recommended over NFS wherever possible for improved performance. If you intend are recommended over NFS wherever possible for improved performance. If you intend
...@@ -2427,10 +2177,8 @@ services where applicable): ...@@ -2427,10 +2177,8 @@ services where applicable):
| PostgreSQL<sup>1</sup> | 3 | 16 vCPU, 60 GB memory | `n1-standard-16` | | PostgreSQL<sup>1</sup> | 3 | 16 vCPU, 60 GB memory | `n1-standard-16` |
| PgBouncer<sup>1</sup> | 3 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | | PgBouncer<sup>1</sup> | 3 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` |
| Internal load balancing node<sup>3</sup> | 1 | 4 vCPU, 3.6GB memory | `n1-highcpu-4` | | Internal load balancing node<sup>3</sup> | 1 | 4 vCPU, 3.6GB memory | `n1-highcpu-4` |
| Redis - Cache<sup>2</sup> | 3 | 4 vCPU, 15 GB memory | `n1-standard-4` | | Redis/Sentinel - Cache<sup>2</sup> | 3 | 4 vCPU, 15 GB memory | `n1-standard-4` |
| Redis - Queues / Shared State<sup>2</sup> | 3 | 4 vCPU, 15 GB memory | `n1-standard-4` | | Redis/Sentinel - Persistent<sup>2</sup> | 3 | 4 vCPU, 15 GB memory | `n1-standard-4` |
| Redis Sentinel - Cache<sup>2</sup> | 3 | 1 vCPU, 3.75 GB memory | `n1-standard-1` |
| Redis Sentinel - Queues / Shared State<sup>2</sup> | 3 | 1 vCPU, 3.75 GB memory | `n1-standard-1` |
| Gitaly | 3 | 32 vCPU, 120 GB memory | `n1-standard-32` | | Gitaly | 3 | 32 vCPU, 120 GB memory | `n1-standard-32` |
| Praefect | 3 | 4 vCPU, 3.6 GB memory | `n1-highcpu-4` | | Praefect | 3 | 4 vCPU, 3.6 GB memory | `n1-highcpu-4` |
| Praefect PostgreSQL<sup>1</sup> | 1+ | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | | Praefect PostgreSQL<sup>1</sup> | 1+ | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` |
...@@ -2486,11 +2234,6 @@ card "Database" as database { ...@@ -2486,11 +2234,6 @@ card "Database" as database {
card "redis" as redis { card "redis" as redis {
collections "**Redis Persistent** x3" as redis_persistent #FF6347 collections "**Redis Persistent** x3" as redis_persistent #FF6347
collections "**Redis Cache** x3" as redis_cache #FF6347 collections "**Redis Cache** x3" as redis_cache #FF6347
collections "**Redis Persistent Sentinel** x3" as redis_persistent_sentinel #FF6347
collections "**Redis Cache Sentinel** x3"as redis_cache_sentinel #FF6347
redis_persistent <.[#FF6347]- redis_persistent_sentinel
redis_cache <.[#FF6347]- redis_cache_sentinel
} }
cloud "**Object Storage**" as object_storage #white cloud "**Object Storage**" as object_storage #white
......
doc/administration/reference_architectures/50k_users.md
View file @ cdec6546
...@@ -22,10 +22,8 @@ full list of reference architectures, see ...@@ -22,10 +22,8 @@ full list of reference architectures, see
| PostgreSQL<sup>1</sup> | 3 | 32 vCPU, 120 GB memory | `n1-standard-32` | `m5.8xlarge` | `D32s v3` | | PostgreSQL<sup>1</sup> | 3 | 32 vCPU, 120 GB memory | `n1-standard-32` | `m5.8xlarge` | `D32s v3` |
| PgBouncer<sup>1</sup> | 3 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | `F2s v2` | | PgBouncer<sup>1</sup> | 3 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | `F2s v2` |
| Internal load balancing node<sup>3</sup> | 1 | 8 vCPU, 7.2 GB memory | `n1-highcpu-8` | `c5.2xlarge` | `F8s v2` | | Internal load balancing node<sup>3</sup> | 1 | 8 vCPU, 7.2 GB memory | `n1-highcpu-8` | `c5.2xlarge` | `F8s v2` |
| Redis - Cache<sup>2</sup> | 3 | 4 vCPU, 15 GB memory | `n1-standard-4` | `m5.xlarge` | `D4s v3` | | Redis/Sentinel - Cache<sup>2</sup> | 3 | 4 vCPU, 15 GB memory | `n1-standard-4` | `m5.xlarge` | `D4s v3` |
| Redis - Queues / Shared State<sup>2</sup> | 3 | 4 vCPU, 15 GB memory | `n1-standard-4` | `m5.xlarge` | `D4s v3` | | Redis/Sentinel - Persistent<sup>2</sup> | 3 | 4 vCPU, 15 GB memory | `n1-standard-4` | `m5.xlarge` | `D4s v3` |
| Redis Sentinel - Cache<sup>2</sup> | 3 | 1 vCPU, 3.75 GB memory | `n1-standard-1` | `c5.large` | `A1 v2` |
| Redis Sentinel - Queues / Shared State<sup>2</sup>| 3 | 1 vCPU, 3.75 GB memory | `n1-standard-1` | `c5.large` | `A1 v2` |
| Gitaly | 3 | 64 vCPU, 240 GB memory | `n1-standard-64` | `m5.16xlarge` | `D64s v3` | | Gitaly | 3 | 64 vCPU, 240 GB memory | `n1-standard-64` | `m5.16xlarge` | `D64s v3` |
| Praefect | 3 | 4 vCPU, 3.6 GB memory | `n1-highcpu-4` | `c5.xlarge` | `F4s v2` | | Praefect | 3 | 4 vCPU, 3.6 GB memory | `n1-highcpu-4` | `c5.xlarge` | `F4s v2` |
| Praefect PostgreSQL<sup>1</sup> | 1+ | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | `F2s v2` | | Praefect PostgreSQL<sup>1</sup> | 1+ | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | `F2s v2` |
...@@ -33,7 +31,7 @@ full list of reference architectures, see ...@@ -33,7 +31,7 @@ full list of reference architectures, see
| GitLab Rails | 12 | 32 vCPU, 28.8 GB memory | `n1-highcpu-32` | `c5.9xlarge` | `F32s v2` | | GitLab Rails | 12 | 32 vCPU, 28.8 GB memory | `n1-highcpu-32` | `c5.9xlarge` | `F32s v2` |
| Monitoring node | 1 | 4 vCPU, 3.6 GB memory | `n1-highcpu-4` | `c5.xlarge` | `F4s v2` | | Monitoring node | 1 | 4 vCPU, 3.6 GB memory | `n1-highcpu-4` | `c5.xlarge` | `F4s v2` |
| Object storage<sup>4</sup> | n/a | n/a | n/a | n/a | n/a | | Object storage<sup>4</sup> | n/a | n/a | n/a | n/a | n/a |
| NFS server (optional, not recommended) | 1 | 4 vCPU, 3.6 GB memory | `n1-highcpu-4` | `c5.xlarge` | `F4s v2` | | NFS server (non-Gitaly) | 1 | 4 vCPU, 3.6 GB memory | `n1-highcpu-4` | `c5.xlarge` | `F4s v2` |
<!-- Disable ordered list rule https://github.com/DavidAnson/markdownlint/blob/main/doc/Rules.md#md029---ordered-list-item-prefix --> <!-- Disable ordered list rule https://github.com/DavidAnson/markdownlint/blob/main/doc/Rules.md#md029---ordered-list-item-prefix -->
<!-- markdownlint-disable MD029 --> <!-- markdownlint-disable MD029 -->
...@@ -82,11 +80,6 @@ card "Database" as database { ...@@ -82,11 +80,6 @@ card "Database" as database {
card "redis" as redis { card "redis" as redis {
collections "**Redis Persistent** x3" as redis_persistent #FF6347 collections "**Redis Persistent** x3" as redis_persistent #FF6347
collections "**Redis Cache** x3" as redis_cache #FF6347 collections "**Redis Cache** x3" as redis_cache #FF6347
collections "**Redis Persistent Sentinel** x3" as redis_persistent_sentinel #FF6347
collections "**Redis Cache Sentinel** x3"as redis_cache_sentinel #FF6347
redis_persistent <.[#FF6347]- redis_persistent_sentinel
redis_cache <.[#FF6347]- redis_cache_sentinel
} }
cloud "**Object Storage**" as object_storage #white cloud "**Object Storage**" as object_storage #white
...@@ -137,8 +130,7 @@ our [Sysbench](https://github.com/akopytov/sysbench)-based ...@@ -137,8 +130,7 @@ our [Sysbench](https://github.com/akopytov/sysbench)-based
Due to better performance and availability, for data objects (such as LFS, Due to better performance and availability, for data objects (such as LFS,
uploads, or artifacts), using an [object storage service](#configure-the-object-storage) uploads, or artifacts), using an [object storage service](#configure-the-object-storage)
is recommended instead of using NFS. Using an object storage service also is recommended.
doesn't require you to provision and maintain a node.
It's also worth noting that at this time [Praefect requires its own database server](../gitaly/praefect.md#postgresql) and It's also worth noting that at this time [Praefect requires its own database server](../gitaly/praefect.md#postgresql) and
that to achieve full High Availability a third-party PostgreSQL database solution will be required. that to achieve full High Availability a third-party PostgreSQL database solution will be required.
...@@ -169,10 +161,8 @@ To set up GitLab and its components to accommodate up to 50,000 users: ...@@ -169,10 +161,8 @@ To set up GitLab and its components to accommodate up to 50,000 users:
used for shared data objects. used for shared data objects.
1. [Configure Advanced Search](#configure-advanced-search) (optional) for faster, 1. [Configure Advanced Search](#configure-advanced-search) (optional) for faster,
more advanced code search across your entire GitLab instance. more advanced code search across your entire GitLab instance.
1. [Configure NFS](#configure-nfs-optional) (optional, and not recommended) 1. [Configure NFS](#configure-nfs)
to have shared disk storage service as an alternative to Gitaly or object to have shared disk storage service for certain GitLab operations (non Gitaly or Object Storage).
storage. You can skip this step if you're not using GitLab Pages (which
requires NFS).
The servers start on the same 10.6.0.0/24 private network range, and can The servers start on the same 10.6.0.0/24 private network range, and can
connect to each other freely on these addresses. connect to each other freely on these addresses.
...@@ -193,15 +183,9 @@ The following list includes descriptions of each server and its assigned IP: ...@@ -193,15 +183,9 @@ The following list includes descriptions of each server and its assigned IP:
- `10.6.0.51`: Redis - Cache Primary - `10.6.0.51`: Redis - Cache Primary
- `10.6.0.52`: Redis - Cache Replica 1 - `10.6.0.52`: Redis - Cache Replica 1
- `10.6.0.53`: Redis - Cache Replica 2 - `10.6.0.53`: Redis - Cache Replica 2
- `10.6.0.71`: Sentinel - Cache 1 - `10.6.0.61`: Redis - Persistent Primary
- `10.6.0.72`: Sentinel - Cache 2 - `10.6.0.62`: Redis - Persistent Replica 1
- `10.6.0.73`: Sentinel - Cache 3 - `10.6.0.63`: Redis - Persistent Replica 2
- `10.6.0.61`: Redis - Queues Primary
- `10.6.0.62`: Redis - Queues Replica 1
- `10.6.0.63`: Redis - Queues Replica 2
- `10.6.0.81`: Sentinel - Queues 1
- `10.6.0.82`: Sentinel - Queues 2
- `10.6.0.83`: Sentinel - Queues 3
- `10.6.0.91`: Gitaly 1 - `10.6.0.91`: Gitaly 1
- `10.6.0.92`: Gitaly 2 - `10.6.0.92`: Gitaly 2
- `10.6.0.93`: Gitaly 3 - `10.6.0.93`: Gitaly 3
...@@ -802,15 +786,9 @@ to be used with GitLab. The following IPs will be used as an example: ...@@ -802,15 +786,9 @@ to be used with GitLab. The following IPs will be used as an example:
- `10.6.0.51`: Redis - Cache Primary - `10.6.0.51`: Redis - Cache Primary
- `10.6.0.52`: Redis - Cache Replica 1 - `10.6.0.52`: Redis - Cache Replica 1
- `10.6.0.53`: Redis - Cache Replica 2 - `10.6.0.53`: Redis - Cache Replica 2
- `10.6.0.71`: Sentinel - Cache 1 - `10.6.0.61`: Redis - Persistent Primary
- `10.6.0.72`: Sentinel - Cache 2 - `10.6.0.62`: Redis - Persistent Replica 1
- `10.6.0.73`: Sentinel - Cache 3 - `10.6.0.63`: Redis - Persistent Replica 2
- `10.6.0.61`: Redis - Queues Primary
- `10.6.0.62`: Redis - Queues Replica 1
- `10.6.0.63`: Redis - Queues Replica 2
- `10.6.0.81`: Sentinel - Queues 1
- `10.6.0.82`: Sentinel - Queues 2
- `10.6.0.83`: Sentinel - Queues 3
### Providing your own Redis instance ### Providing your own Redis instance
...@@ -822,7 +800,7 @@ optional count argument to SPOP, which is required for [Merge Trains](../../ci/p ...@@ -822,7 +800,7 @@ optional count argument to SPOP, which is required for [Merge Trains](../../ci/p
Note the Redis node's IP address or hostname, port, and password (if required). Note the Redis node's IP address or hostname, port, and password (if required).
These will be necessary later when configuring the [GitLab application servers](#configure-gitlab-rails). These will be necessary later when configuring the [GitLab application servers](#configure-gitlab-rails).
### Configure the Redis and Sentinel Cache cluster ### Configure the Redis Cache cluster
This is the section where we install and set up the new Redis Cache instances. This is the section where we install and set up the new Redis Cache instances.
...@@ -840,8 +818,12 @@ a node and change its status from primary to replica (and vice versa). ...@@ -840,8 +818,12 @@ a node and change its status from primary to replica (and vice versa).
1. Edit `/etc/gitlab/gitlab.rb` and add the contents: 1. Edit `/etc/gitlab/gitlab.rb` and add the contents:
```ruby ```ruby
# Specify server role as 'redis_master_role' and enable Consul agent # Specify server role as 'redis_master_role' with Sentinel and enable Consul agent
roles(['redis_master_role', 'consul_role']) roles(['redis_sentinel_role', 'redis_master_role', 'consul_role'])
# Set IP bind address and Quorum number for Redis Sentinel service
sentinel['bind'] = '0.0.0.0'
sentinel['quorum'] = 2
# IP address pointing to a local IP that the other machines can reach to. # IP address pointing to a local IP that the other machines can reach to.
# You can also set bind to '0.0.0.0' which listen in all interfaces. # You can also set bind to '0.0.0.0' which listen in all interfaces.
...@@ -853,8 +835,19 @@ a node and change its status from primary to replica (and vice versa). ...@@ -853,8 +835,19 @@ a node and change its status from primary to replica (and vice versa).
# machines to connect to it. # machines to connect to it.
redis['port'] = 6379 redis['port'] = 6379
# Set up password authentication for Redis (use the same password in all nodes). ## Port of primary Redis server for Sentinel, uncomment to change to non default. Defaults
## to `6379`.
#redis['master_port'] = 6379
# Set up password authentication for Redis and replicas (use the same password in all nodes).
redis['password'] = 'REDIS_PRIMARY_PASSWORD_OF_FIRST_CLUSTER' redis['password'] = 'REDIS_PRIMARY_PASSWORD_OF_FIRST_CLUSTER'
redis['master_password'] = 'REDIS_PRIMARY_PASSWORD_OF_FIRST_CLUSTER'
## Must be the same in every Redis node
redis['master_name'] = 'gitlab-redis-cache'
## The IP of this primary Redis node.
redis['master_ip'] = '10.6.0.51'
# Set the Redis Cache instance as an LRU # Set the Redis Cache instance as an LRU
# 90% of available RAM in MB # 90% of available RAM in MB
...@@ -888,10 +881,6 @@ a node and change its status from primary to replica (and vice versa). ...@@ -888,10 +881,6 @@ a node and change its status from primary to replica (and vice versa).
1. [Reconfigure Omnibus GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. 1. [Reconfigure Omnibus GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect.
You can specify multiple roles, like sentinel and Redis, as:
`roles(['redis_sentinel_role', 'redis_master_role'])`. Read more about
[roles](https://docs.gitlab.com/omnibus/roles/).
#### Configure the replica Redis Cache nodes #### Configure the replica Redis Cache nodes
1. SSH in to the **replica** Redis server. 1. SSH in to the **replica** Redis server.
...@@ -899,11 +888,15 @@ You can specify multiple roles, like sentinel and Redis, as: ...@@ -899,11 +888,15 @@ You can specify multiple roles, like sentinel and Redis, as:
package of your choice. Be sure to both follow _only_ installation steps 1 and 2 package of your choice. Be sure to both follow _only_ installation steps 1 and 2
on the page, and to select the correct Omnibus GitLab package, with the same version on the page, and to select the correct Omnibus GitLab package, with the same version
and type (Community or Enterprise editions) as your current install. and type (Community or Enterprise editions) as your current install.
1. Edit `/etc/gitlab/gitlab.rb` and add the contents: 1. Edit `/etc/gitlab/gitlab.rb` and add the same contents as the priimary node in the previous section by replacing `redis_master_node` with `redis_replica_node`:
```ruby ```ruby
# Specify server role as 'redis_replica_role' and enable Consul agent # Specify server role as 'redis_replica_role' with Sentinel and enable Consul agent
roles(['redis_replica_role', 'consul_role']) roles(['roles_sentinel_role', 'redis_replica_role', 'consul_role'])
# Set IP bind address and Quorum number for Redis Sentinel service
sentinel['bind'] = '0.0.0.0'
sentinel['quorum'] = 2
# IP address pointing to a local IP that the other machines can reach to. # IP address pointing to a local IP that the other machines can reach to.
# You can also set bind to '0.0.0.0' which listen in all interfaces. # You can also set bind to '0.0.0.0' which listen in all interfaces.
...@@ -915,15 +908,19 @@ You can specify multiple roles, like sentinel and Redis, as: ...@@ -915,15 +908,19 @@ You can specify multiple roles, like sentinel and Redis, as:
# machines to connect to it. # machines to connect to it.
redis['port'] = 6379 redis['port'] = 6379
# The same password for Redis authentication you set up for the primary node. ## Port of primary Redis server for Sentinel, uncomment to change to non default. Defaults
## to `6379`.
#redis['master_port'] = 6379
# Set up password authentication for Redis and replicas (use the same password in all nodes).
redis['password'] = 'REDIS_PRIMARY_PASSWORD_OF_FIRST_CLUSTER' redis['password'] = 'REDIS_PRIMARY_PASSWORD_OF_FIRST_CLUSTER'
redis['master_password'] = 'REDIS_PRIMARY_PASSWORD_OF_FIRST_CLUSTER'
# The IP of the primary Redis node. ## Must be the same in every Redis node
redis['master_ip'] = '10.6.0.51' redis['master_name'] = 'gitlab-redis-cache'
# Port of primary Redis server, uncomment to change to non default. Defaults ## The IP of the primary Redis node.
# to `6379`. redis['master_ip'] = '10.6.0.51'
#redis['master_port'] = 6379
# Set the Redis Cache instance as an LRU # Set the Redis Cache instance as an LRU
# 90% of available RAM in MB # 90% of available RAM in MB
...@@ -952,25 +949,18 @@ You can specify multiple roles, like sentinel and Redis, as: ...@@ -952,25 +949,18 @@ You can specify multiple roles, like sentinel and Redis, as:
gitlab_rails['auto_migrate'] = false gitlab_rails['auto_migrate'] = false
``` ```
1. Copy the `/etc/gitlab/gitlab-secrets.json` file from the first Omnibus node you configured and add or replace 1. Copy the `/etc/gitlab/gitlab-secrets.json` file from the first Omnibus
the file of the same name on this server. If this is the first Omnibus node you are configuring then you can skip this step. node you configured and add or replace the file of the same name on this
server. If this is the first Omnibus node you are configuring then you
can skip this step.
1. [Reconfigure Omnibus GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. 1. [Reconfigure Omnibus GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes
to take effect.
1. Go through the steps again for all the other replica nodes, and 1. Go through the steps again for all the other replica nodes, and
make sure to set up the IPs correctly. make sure to set up the IPs correctly.
You can specify multiple roles, like sentinel and Redis, as: Advanced [configuration options](https://docs.gitlab.com/omnibus/settings/redis.html) are supported and can be added if needed.
`roles(['redis_sentinel_role', 'redis_master_role'])`. Read more about
[roles](https://docs.gitlab.com/omnibus/roles/).
These values don't have to be changed again in `/etc/gitlab/gitlab.rb` after
a failover, as the nodes will be managed by the [Sentinels](#configure-the-sentinel-cache-nodes), and even after a
`gitlab-ctl reconfigure`, they will get their configuration restored by
the same Sentinels.
Advanced [configuration options](https://docs.gitlab.com/omnibus/settings/redis.html)
are supported and can be added if needed.
<div align="right"> <div align="right">
<a type="button" class="btn btn-default" href="#setup-components"> <a type="button" class="btn btn-default" href="#setup-components">
...@@ -978,126 +968,7 @@ are supported and can be added if needed. ...@@ -978,126 +968,7 @@ are supported and can be added if needed.
</a> </a>
</div> </div>
#### Configure the Sentinel Cache nodes ### Configure the Redis Persistent cluster
Now that the Redis servers are all set up, let's configure the Sentinel
servers. The following IPs will be used as an example:
- `10.6.0.71`: Sentinel - Cache 1
- `10.6.0.72`: Sentinel - Cache 2
- `10.6.0.73`: Sentinel - Cache 3
NOTE:
If you're using an external Redis Sentinel instance, be sure to exclude the
`requirepass` parameter from the Sentinel configuration. This parameter causes
clients to report `NOAUTH Authentication required.`.
[Redis Sentinel 3.2.x doesn't support password authentication](https://github.com/antirez/redis/issues/3279).
To configure the Sentinel Cache server:
1. SSH in to the server that will host Consul/Sentinel.
1. [Download and install](https://about.gitlab.com/install/) the Omnibus GitLab
package of your choice. Be sure to both follow _only_ installation steps 1 and 2
on the page, and to select the correct Omnibus GitLab package, with the same version
and type (Community or Enterprise editions) as your current install.
1. Edit `/etc/gitlab/gitlab.rb` and add the contents:
```ruby
roles(['redis_sentinel_role', 'consul_role'])
## Must be the same in every sentinel node
redis['master_name'] = 'gitlab-redis-cache'
## The same password for Redis authentication you set up for the primary node.
redis['master_password'] = 'REDIS_PRIMARY_PASSWORD_OF_FIRST_CLUSTER'
## The IP of the primary Redis node.
redis['master_ip'] = '10.6.0.51'
## Define a port so Redis can listen for TCP requests which will allow other
## machines to connect to it.
redis['port'] = 6379
## Port of primary Redis server, uncomment to change to non default. Defaults
## to `6379`.
#redis['master_port'] = 6379
## Configure Sentinel's IP
sentinel['bind'] = '10.6.0.71'
## Port that Sentinel listens on, uncomment to change to non default. Defaults
## to `26379`.
#sentinel['port'] = 26379
## Quorum must reflect the amount of voting sentinels it take to start a failover.
## Value must NOT be greater then the amount of sentinels.
##
## The quorum can be used to tune Sentinel in two ways:
## 1. If a the quorum is set to a value smaller than the majority of Sentinels
## we deploy, we are basically making Sentinel more sensible to primary failures,
## triggering a failover as soon as even just a minority of Sentinels is no longer
## able to talk with the primary.
## 1. If a quorum is set to a value greater than the majority of Sentinels, we are
## making Sentinel able to failover only when there are a very large number (larger
## than majority) of well connected Sentinels which agree about the primary being down.s
sentinel['quorum'] = 2
## Consider unresponsive server down after x amount of ms.
#sentinel['down_after_milliseconds'] = 10000
## Specifies the failover timeout in milliseconds. It is used in many ways:
##
## - The time needed to re-start a failover after a previous failover was
## already tried against the same primary by a given Sentinel, is two
## times the failover timeout.
##
## - The time needed for a replica replicating to a wrong primary according
## to a Sentinel current configuration, to be forced to replicate
## with the right primary, is exactly the failover timeout (counting since
## the moment a Sentinel detected the misconfiguration).
##
## - The time needed to cancel a failover that is already in progress but
## did not produced any configuration change (REPLICAOF NO ONE yet not
## acknowledged by the promoted replica).
##
## - The maximum time a failover in progress waits for all the replica to be
## reconfigured as replicas of the new primary. However even after this time
## the replicas will be reconfigured by the Sentinels anyway, but not with
## the exact parallel-syncs progression as specified.
#sentinel['failover_timeout'] = 60000
## Enable service discovery for Prometheus
consul['monitoring_service_discovery'] = true
## The IPs of the Consul server nodes
## You can also use FQDNs and intermix them with IPs
consul['configuration'] = {
retry_join: %w(10.6.0.11 10.6.0.12 10.6.0.13),
}
# Set the network addresses that the exporters will listen on
node_exporter['listen_address'] = '0.0.0.0:9100'
redis_exporter['listen_address'] = '0.0.0.0:9121'
# Prevent database migrations from running on upgrade automatically
gitlab_rails['auto_migrate'] = false
```
1. Copy the `/etc/gitlab/gitlab-secrets.json` file from the first Omnibus node you configured and add or replace
the file of the same name on this server. If this is the first Omnibus node you are configuring then you can skip this step.
1. [Reconfigure Omnibus GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect.
1. Go through the steps again for all the other Consul/Sentinel nodes, and
make sure you set up the correct IPs.
<div align="right">
<a type="button" class="btn btn-default" href="#setup-components">
Back to setup components <i class="fa fa-angle-double-up" aria-hidden="true"></i>
</a>
</div>
### Configure the Redis and Sentinel Queues cluster
This is the section where we install and set up the new Redis Queues instances. This is the section where we install and set up the new Redis Queues instances.
...@@ -1105,7 +976,7 @@ Both the primary and replica Redis nodes need the same password defined in ...@@ -1105,7 +976,7 @@ Both the primary and replica Redis nodes need the same password defined in
`redis['password']`. At any time during a failover, the Sentinels can reconfigure `redis['password']`. At any time during a failover, the Sentinels can reconfigure
a node and change its status from primary to replica (and vice versa). a node and change its status from primary to replica (and vice versa).
#### Configure the primary Redis Queues node #### Configure the primary Redis Persistent node
1. SSH in to the **Primary** Redis server. 1. SSH in to the **Primary** Redis server.
1. [Download and install](https://about.gitlab.com/install/) the Omnibus GitLab 1. [Download and install](https://about.gitlab.com/install/) the Omnibus GitLab
...@@ -1115,8 +986,12 @@ a node and change its status from primary to replica (and vice versa). ...@@ -1115,8 +986,12 @@ a node and change its status from primary to replica (and vice versa).
1. Edit `/etc/gitlab/gitlab.rb` and add the contents: 1. Edit `/etc/gitlab/gitlab.rb` and add the contents:
```ruby ```ruby
# Specify server role as 'redis_master_role' and enable Consul agent # Specify server roles as 'redis_master_role' with Sentinel and enable the Consul agent
roles(['redis_master_role', 'consul_role']) roles ['redis_sentinel_role', 'redis_master_role', 'consul_role']
# Set IP bind address and Quorum number for Redis Sentinel service
sentinel['bind'] = '0.0.0.0'
sentinel['quorum'] = 2
# IP address pointing to a local IP that the other machines can reach to. # IP address pointing to a local IP that the other machines can reach to.
# You can also set bind to '0.0.0.0' which listen in all interfaces. # You can also set bind to '0.0.0.0' which listen in all interfaces.
...@@ -1128,8 +1003,19 @@ a node and change its status from primary to replica (and vice versa). ...@@ -1128,8 +1003,19 @@ a node and change its status from primary to replica (and vice versa).
# machines to connect to it. # machines to connect to it.
redis['port'] = 6379 redis['port'] = 6379
# Set up password authentication for Redis (use the same password in all nodes). ## Port of primary Redis server for Sentinel, uncomment to change to non default. Defaults
redis['password'] = 'REDIS_PRIMARY_PASSWORD_OF_SECOND_CLUSTER' ## to `6379`.
#redis['master_port'] = 6379
# Set up password authentication for Redis and replicas (use the same password in all nodes).
redis['password'] = 'REDIS_PRIMARY_PASSWORD_OF_FIRST_CLUSTER'
redis['master_password'] = 'REDIS_PRIMARY_PASSWORD_OF_SECOND_CLUSTER'
## Must be the same in every Redis node
redis['master_name'] = 'gitlab-redis-persistent'
## The IP of this primary Redis node.
redis['master_ip'] = '10.6.0.61'
## Enable service discovery for Prometheus ## Enable service discovery for Prometheus
consul['monitoring_service_discovery'] = true consul['monitoring_service_discovery'] = true
...@@ -1153,13 +1039,9 @@ a node and change its status from primary to replica (and vice versa). ...@@ -1153,13 +1039,9 @@ a node and change its status from primary to replica (and vice versa).
1. [Reconfigure Omnibus GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. 1. [Reconfigure Omnibus GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect.
You can specify multiple roles, like sentinel and Redis, as: #### Configure the replica Redis Persistent nodes
`roles ['redis_sentinel_role', 'redis_master_role']`. Read more about
[roles](https://docs.gitlab.com/omnibus/roles/).
#### Configure the replica Redis Queues nodes
1. SSH in to the **replica** Redis Queue server. 1. SSH in to the **replica** Redis Persistent server.
1. [Download and install](https://about.gitlab.com/install/) the Omnibus GitLab 1. [Download and install](https://about.gitlab.com/install/) the Omnibus GitLab
package of your choice. Be sure to both follow _only_ installation steps 1 and 2 package of your choice. Be sure to both follow _only_ installation steps 1 and 2
on the page, and to select the correct Omnibus GitLab package, with the same version on the page, and to select the correct Omnibus GitLab package, with the same version
...@@ -1167,8 +1049,12 @@ You can specify multiple roles, like sentinel and Redis, as: ...@@ -1167,8 +1049,12 @@ You can specify multiple roles, like sentinel and Redis, as:
1. Edit `/etc/gitlab/gitlab.rb` and add the contents: 1. Edit `/etc/gitlab/gitlab.rb` and add the contents:
```ruby ```ruby
# Specify server role as 'redis_replica_role' and enable Consul agent # Specify server roles as 'redis_replica_role' with Sentinel and enable Consul agent
roles(['redis_replica_role', 'consul_role']) roles ['redis_sentinel_role', 'redis_replica_role', 'consul_role']
# Set IP bind address and Quorum number for Redis Sentinel service
sentinel['bind'] = '0.0.0.0'
sentinel['quorum'] = 2
# IP address pointing to a local IP that the other machines can reach to. # IP address pointing to a local IP that the other machines can reach to.
# You can also set bind to '0.0.0.0' which listen in all interfaces. # You can also set bind to '0.0.0.0' which listen in all interfaces.
...@@ -1180,16 +1066,20 @@ You can specify multiple roles, like sentinel and Redis, as: ...@@ -1180,16 +1066,20 @@ You can specify multiple roles, like sentinel and Redis, as:
# machines to connect to it. # machines to connect to it.
redis['port'] = 6379 redis['port'] = 6379
## Port of primary Redis server for Sentinel, uncomment to change to non default. Defaults
## to `6379`.
#redis['master_port'] = 6379
# The same password for Redis authentication you set up for the primary node. # The same password for Redis authentication you set up for the primary node.
redis['password'] = 'REDIS_PRIMARY_PASSWORD_OF_SECOND_CLUSTER' redis['password'] = 'REDIS_PRIMARY_PASSWORD_OF_SECOND_CLUSTER'
redis['master_password'] = 'REDIS_PRIMARY_PASSWORD_OF_SECOND_CLUSTER'
## Must be the same in every Redis node
redis['master_name'] = 'gitlab-redis-persistent'
# The IP of the primary Redis node. # The IP of the primary Redis node.
redis['master_ip'] = '10.6.0.61' redis['master_ip'] = '10.6.0.61'
# Port of primary Redis server, uncomment to change to non default. Defaults
# to `6379`.
#redis['master_port'] = 6379
## Enable service discovery for Prometheus ## Enable service discovery for Prometheus
consul['monitoring_service_discovery'] = true consul['monitoring_service_discovery'] = true
...@@ -1215,144 +1105,7 @@ You can specify multiple roles, like sentinel and Redis, as: ...@@ -1215,144 +1105,7 @@ You can specify multiple roles, like sentinel and Redis, as:
1. Go through the steps again for all the other replica nodes, and 1. Go through the steps again for all the other replica nodes, and
make sure to set up the IPs correctly. make sure to set up the IPs correctly.
You can specify multiple roles, like sentinel and Redis, as: Advanced [configuration options](https://docs.gitlab.com/omnibus/settings/redis.html) are supported and can be added if needed.
`roles(['redis_sentinel_role', 'redis_master_role'])`. Read more about
[roles](https://docs.gitlab.com/omnibus/roles/).
These values don't have to be changed again in `/etc/gitlab/gitlab.rb` after
a failover, as the nodes will be managed by the [Sentinels](#configure-the-sentinel-queues-nodes), and even after a
`gitlab-ctl reconfigure`, they will get their configuration restored by
the same Sentinels.
Advanced [configuration options](https://docs.gitlab.com/omnibus/settings/redis.html)
are supported and can be added if needed.
<div align="right">
<a type="button" class="btn btn-default" href="#setup-components">
Back to setup components <i class="fa fa-angle-double-up" aria-hidden="true"></i>
</a>
</div>
#### Configure the Sentinel Queues nodes
Now that the Redis servers are all set up, let's configure the Sentinel
servers. The following IPs will be used as an example:
- `10.6.0.81`: Sentinel - Queues 1
- `10.6.0.82`: Sentinel - Queues 2
- `10.6.0.83`: Sentinel - Queues 3
NOTE:
If you're using an external Redis Sentinel instance, be sure to exclude the
`requirepass` parameter from the Sentinel configuration. This parameter causes
clients to report `NOAUTH Authentication required.`.
[Redis Sentinel 3.2.x doesn't support password authentication](https://github.com/antirez/redis/issues/3279).
To configure the Sentinel Queues server:
1. SSH in to the server that will host Sentinel.
1. [Download and install](https://about.gitlab.com/install/) the Omnibus GitLab
package of your choice. Be sure to both follow _only_ installation steps 1 and 2
on the page, and to select the correct Omnibus GitLab package, with the same version
and type (Community or Enterprise editions) as your current install.
1. Edit `/etc/gitlab/gitlab.rb` and add the contents:
```ruby
roles(['redis_sentinel_role', 'consul_role'])
## Must be the same in every sentinel node
redis['master_name'] = 'gitlab-redis-persistent'
## The same password for Redis authentication you set up for the primary node.
redis['master_password'] = 'REDIS_PRIMARY_PASSWORD_OF_SECOND_CLUSTER'
## The IP of the primary Redis node.
redis['master_ip'] = '10.6.0.61'
## Define a port so Redis can listen for TCP requests which will allow other
## machines to connect to it.
redis['port'] = 6379
## Port of primary Redis server, uncomment to change to non default. Defaults
## to `6379`.
#redis['master_port'] = 6379
## Configure Sentinel's IP
sentinel['bind'] = '10.6.0.81'
## Port that Sentinel listens on, uncomment to change to non default. Defaults
## to `26379`.
#sentinel['port'] = 26379
## Quorum must reflect the amount of voting sentinels it take to start a failover.
## Value must NOT be greater then the amount of sentinels.
##
## The quorum can be used to tune Sentinel in two ways:
## 1. If a the quorum is set to a value smaller than the majority of Sentinels
## we deploy, we are basically making Sentinel more sensible to primary failures,
## triggering a failover as soon as even just a minority of Sentinels is no longer
## able to talk with the primary.
## 1. If a quorum is set to a value greater than the majority of Sentinels, we are
## making Sentinel able to failover only when there are a very large number (larger
## than majority) of well connected Sentinels which agree about the primary being down.s
sentinel['quorum'] = 2
## Consider unresponsive server down after x amount of ms.
#sentinel['down_after_milliseconds'] = 10000
## Specifies the failover timeout in milliseconds. It is used in many ways:
##
## - The time needed to re-start a failover after a previous failover was
## already tried against the same primary by a given Sentinel, is two
## times the failover timeout.
##
## - The time needed for a replica replicating to a wrong primary according
## to a Sentinel current configuration, to be forced to replicate
## with the right primary, is exactly the failover timeout (counting since
## the moment a Sentinel detected the misconfiguration).
##
## - The time needed to cancel a failover that is already in progress but
## did not produced any configuration change (REPLICAOF NO ONE yet not
## acknowledged by the promoted replica).
##
## - The maximum time a failover in progress waits for all the replica to be
## reconfigured as replicas of the new primary. However even after this time
## the replicas will be reconfigured by the Sentinels anyway, but not with
## the exact parallel-syncs progression as specified.
#sentinel['failover_timeout'] = 60000
## Enable service discovery for Prometheus
consul['monitoring_service_discovery'] = true
## The IPs of the Consul server nodes
## You can also use FQDNs and intermix them with IPs
consul['configuration'] = {
retry_join: %w(10.6.0.11 10.6.0.12 10.6.0.13),
}
# Set the network addresses that the exporters will listen on
node_exporter['listen_address'] = '0.0.0.0:9100'
redis_exporter['listen_address'] = '0.0.0.0:9121'
# Prevent database migrations from running on upgrade automatically
gitlab_rails['auto_migrate'] = false
```
1. To ensure database migrations are only run during reconfigure and not automatically on upgrade, run:
```shell
sudo touch /etc/gitlab/skip-auto-reconfigure
```
Only the primary GitLab application server should handle migrations.
1. Copy the `/etc/gitlab/gitlab-secrets.json` file from the first Omnibus node you configured and add or replace
the file of the same name on this server. If this is the first Omnibus node you are configuring then you can skip this step.
1. [Reconfigure Omnibus GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect.
1. Go through the steps again for all the other Sentinel nodes, and
make sure you set up the correct IPs.
<div align="right"> <div align="right">
<a type="button" class="btn btn-default" href="#setup-components"> <a type="button" class="btn btn-default" href="#setup-components">
...@@ -1879,30 +1632,30 @@ To configure the Sidekiq nodes, on each one: ...@@ -1879,30 +1632,30 @@ To configure the Sidekiq nodes, on each one:
gitlab_rails['redis_cache_instance'] = 'redis://:<REDIS_PRIMARY_PASSWORD_OF_FIRST_CLUSTER>@gitlab-redis-cache' gitlab_rails['redis_cache_instance'] = 'redis://:<REDIS_PRIMARY_PASSWORD_OF_FIRST_CLUSTER>@gitlab-redis-cache'
gitlab_rails['redis_cache_sentinels'] = [ gitlab_rails['redis_cache_sentinels'] = [
{host: '10.6.0.71', port: 26379}, {host: '10.6.0.51', port: 26379},
{host: '10.6.0.72', port: 26379}, {host: '10.6.0.52', port: 26379},
{host: '10.6.0.73', port: 26379}, {host: '10.6.0.53', port: 26379},
] ]
## Second cluster that will host the queues, shared state, and actioncable ## Second cluster that will host the persistent queues, shared state, and actioncable
gitlab_rails['redis_queues_instance'] = 'redis://:<REDIS_PRIMARY_PASSWORD_OF_SECOND_CLUSTER>@gitlab-redis-persistent' gitlab_rails['redis_queues_instance'] = 'redis://:<REDIS_PRIMARY_PASSWORD_OF_SECOND_CLUSTER>@gitlab-redis-persistent'
gitlab_rails['redis_shared_state_instance'] = 'redis://:<REDIS_PRIMARY_PASSWORD_OF_SECOND_CLUSTER>@gitlab-redis-persistent' gitlab_rails['redis_shared_state_instance'] = 'redis://:<REDIS_PRIMARY_PASSWORD_OF_SECOND_CLUSTER>@gitlab-redis-persistent'
gitlab_rails['redis_actioncable_instance'] = 'redis://:<REDIS_PRIMARY_PASSWORD_OF_SECOND_CLUSTER>@gitlab-redis-persistent' gitlab_rails['redis_actioncable_instance'] = 'redis://:<REDIS_PRIMARY_PASSWORD_OF_SECOND_CLUSTER>@gitlab-redis-persistent'
gitlab_rails['redis_queues_sentinels'] = [ gitlab_rails['redis_queues_sentinels'] = [
{host: '10.6.0.81', port: 26379}, {host: '10.6.0.61', port: 26379},
{host: '10.6.0.82', port: 26379}, {host: '10.6.0.62', port: 26379},
{host: '10.6.0.83', port: 26379}, {host: '10.6.0.63', port: 26379},
] ]
gitlab_rails['redis_shared_state_sentinels'] = [ gitlab_rails['redis_shared_state_sentinels'] = [
{host: '10.6.0.81', port: 26379}, {host: '10.6.0.61', port: 26379},
{host: '10.6.0.82', port: 26379}, {host: '10.6.0.62', port: 26379},
{host: '10.6.0.83', port: 26379}, {host: '10.6.0.63', port: 26379},
] ]
gitlab_rails['redis_actioncable_sentinels'] = [ gitlab_rails['redis_actioncable_sentinels'] = [
{host: '10.6.0.81', port: 26379}, {host: '10.6.0.61', port: 26379},
{host: '10.6.0.82', port: 26379}, {host: '10.6.0.62', port: 26379},
{host: '10.6.0.83', port: 26379}, {host: '10.6.0.63', port: 26379},
] ]
# Gitaly # Gitaly
...@@ -2063,30 +1816,30 @@ On each node perform the following: ...@@ -2063,30 +1816,30 @@ On each node perform the following:
gitlab_rails['redis_cache_instance'] = 'redis://:<REDIS_PRIMARY_PASSWORD_OF_FIRST_CLUSTER>@gitlab-redis-cache' gitlab_rails['redis_cache_instance'] = 'redis://:<REDIS_PRIMARY_PASSWORD_OF_FIRST_CLUSTER>@gitlab-redis-cache'
gitlab_rails['redis_cache_sentinels'] = [ gitlab_rails['redis_cache_sentinels'] = [
{host: '10.6.0.71', port: 26379}, {host: '10.6.0.51', port: 26379},
{host: '10.6.0.72', port: 26379}, {host: '10.6.0.52', port: 26379},
{host: '10.6.0.73', port: 26379}, {host: '10.6.0.53', port: 26379},
] ]
## Second cluster that will host the queues, shared state, and actionable ## Second cluster that will host the persistent queues, shared state, and actionable
gitlab_rails['redis_queues_instance'] = 'redis://:<REDIS_PRIMARY_PASSWORD_OF_SECOND_CLUSTER>@gitlab-redis-persistent' gitlab_rails['redis_queues_instance'] = 'redis://:<REDIS_PRIMARY_PASSWORD_OF_SECOND_CLUSTER>@gitlab-redis-persistent'
gitlab_rails['redis_shared_state_instance'] = 'redis://:<REDIS_PRIMARY_PASSWORD_OF_SECOND_CLUSTER>@gitlab-redis-persistent' gitlab_rails['redis_shared_state_instance'] = 'redis://:<REDIS_PRIMARY_PASSWORD_OF_SECOND_CLUSTER>@gitlab-redis-persistent'
gitlab_rails['redis_actioncable_instance'] = 'redis://:<REDIS_PRIMARY_PASSWORD_OF_SECOND_CLUSTER>@gitlab-redis-persistent' gitlab_rails['redis_actioncable_instance'] = 'redis://:<REDIS_PRIMARY_PASSWORD_OF_SECOND_CLUSTER>@gitlab-redis-persistent'
gitlab_rails['redis_queues_sentinels'] = [ gitlab_rails['redis_queues_sentinels'] = [
{host: '10.6.0.81', port: 26379}, {host: '10.6.0.61', port: 26379},
{host: '10.6.0.82', port: 26379}, {host: '10.6.0.62', port: 26379},
{host: '10.6.0.83', port: 26379}, {host: '10.6.0.63', port: 26379},
] ]
gitlab_rails['redis_shared_state_sentinels'] = [ gitlab_rails['redis_shared_state_sentinels'] = [
{host: '10.6.0.81', port: 26379}, {host: '10.6.0.61', port: 26379},
{host: '10.6.0.82', port: 26379}, {host: '10.6.0.62', port: 26379},
{host: '10.6.0.83', port: 26379}, {host: '10.6.0.63', port: 26379},
] ]
gitlab_rails['redis_actioncable_sentinels'] = [ gitlab_rails['redis_actioncable_sentinels'] = [
{host: '10.6.0.81', port: 26379}, {host: '10.6.0.61', port: 26379},
{host: '10.6.0.82', port: 26379}, {host: '10.6.0.62', port: 26379},
{host: '10.6.0.83', port: 26379}, {host: '10.6.0.63', port: 26379},
] ]
# Set the network addresses that the exporters used for monitoring will listen on # Set the network addresses that the exporters used for monitoring will listen on
...@@ -2158,7 +1911,7 @@ On each node perform the following: ...@@ -2158,7 +1911,7 @@ On each node perform the following:
1. [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. 1. [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect.
1. If you're [using NFS](#configure-nfs-optional): 1. If you're [using NFS](#configure-nfs):
1. If necessary, install the NFS client utility packages using the following 1. If necessary, install the NFS client utility packages using the following
commands: commands:
...@@ -2300,7 +2053,7 @@ To configure the Monitoring node: ...@@ -2300,7 +2053,7 @@ To configure the Monitoring node:
## Configure the object storage ## Configure the object storage
GitLab supports using an object storage service for holding numerous types of data. GitLab supports using an object storage service for holding numerous types of data.
It's recommended over [NFS](#configure-nfs-optional) and in general it's better It's recommended over [NFS](#configure-nfs) and in general it's better
in larger setups as object storage is typically much more performant, reliable, in larger setups as object storage is typically much more performant, reliable,
and scalable. and scalable.
...@@ -2370,7 +2123,7 @@ cluster alongside your instance, read how to ...@@ -2370,7 +2123,7 @@ cluster alongside your instance, read how to
</a> </a>
</div> </div>
## Configure NFS (optional) ## Configure NFS
[Object storage](#configure-the-object-storage), along with [Gitaly](#configure-gitaly) [Object storage](#configure-the-object-storage), along with [Gitaly](#configure-gitaly)
are recommended over NFS wherever possible for improved performance. If you intend are recommended over NFS wherever possible for improved performance. If you intend
...@@ -2438,10 +2191,8 @@ services where applicable): ...@@ -2438,10 +2191,8 @@ services where applicable):
| PostgreSQL<sup>1</sup> | 3 | 32 vCPU, 120 GB memory | `n1-standard-32` | | PostgreSQL<sup>1</sup> | 3 | 32 vCPU, 120 GB memory | `n1-standard-32` |
| PgBouncer<sup>1</sup> | 3 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | | PgBouncer<sup>1</sup> | 3 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` |
| Internal load balancing node<sup>3</sup> | 1 | 8 vCPU, 7.2 GB memory | `n1-highcpu-8` | | Internal load balancing node<sup>3</sup> | 1 | 8 vCPU, 7.2 GB memory | `n1-highcpu-8` |
| Redis - Cache<sup>2</sup> | 3 | 4 vCPU, 15 GB memory | `n1-standard-4` | | Redis/Sentinel - Cache<sup>2</sup> | 3 | 4 vCPU, 15 GB memory | `n1-standard-4` |
| Redis - Queues / Shared State<sup>2</sup> | 3 | 4 vCPU, 15 GB memory | `n1-standard-4` | | Redis/Sentinel - Persistent<sup>2</sup> | 3 | 4 vCPU, 15 GB memory | `n1-standard-4` |
| Redis Sentinel - Cache<sup>2</sup> | 3 | 1 vCPU, 3.75 GB memory | `n1-standard-1` |
| Redis Sentinel - Queues / Shared State<sup>2</sup> | 3 | 1 vCPU, 3.75 GB memory | `n1-standard-1` |
| Gitaly | 3 | 64 vCPU, 240 GB memory | `n1-standard-64` | | Gitaly | 3 | 64 vCPU, 240 GB memory | `n1-standard-64` |
| Praefect | 3 | 4 vCPU, 3.6 GB memory | `n1-highcpu-4` | | Praefect | 3 | 4 vCPU, 3.6 GB memory | `n1-highcpu-4` |
| Praefect PostgreSQL<sup>1</sup> | 1+ | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | | Praefect PostgreSQL<sup>1</sup> | 1+ | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` |
...@@ -2497,11 +2248,6 @@ card "Database" as database { ...@@ -2497,11 +2248,6 @@ card "Database" as database {
card "redis" as redis { card "redis" as redis {
collections "**Redis Persistent** x3" as redis_persistent #FF6347 collections "**Redis Persistent** x3" as redis_persistent #FF6347
collections "**Redis Cache** x3" as redis_cache #FF6347 collections "**Redis Cache** x3" as redis_cache #FF6347
collections "**Redis Persistent Sentinel** x3" as redis_persistent_sentinel #FF6347
collections "**Redis Cache Sentinel** x3"as redis_cache_sentinel #FF6347
redis_persistent <.[#FF6347]- redis_persistent_sentinel
redis_cache <.[#FF6347]- redis_cache_sentinel
} }
cloud "**Object Storage**" as object_storage #white cloud "**Object Storage**" as object_storage #white
......
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment
GitLab Nexedi Edition | About GitLab | About Nexedi | 沪ICP备2021021310号-2 | 沪ICP备2021021310号-7