Commit 368e9d9d authored by Achilleas Pipinellis's avatar Achilleas Pipinellis

Merge branch 'brodock/update-geo-patroni-docs' into 'master'

Improve Geo documentation for Patroni Beta

See merge request gitlab-org/gitlab!62330
parents e75b8dd1 14a83343
......@@ -25,7 +25,20 @@ size.
You are encouraged to first read through all the steps before executing them
in your testing/production environment.
## PostgreSQL replication
## Single instance database replication
A single instance database replication is easier to set up and still provides the same Geo capabilities
as a clusterized alternative. It's useful for setups running on a single machine
or trying to evaluate Geo for a future clusterized installation.
A single instance can be expanded to a clusterized version using Patroni, which is recommended for a
highly available architecture.
Follow below the instructions on how to set up PostgreSQL replication as a single instance database.
Alternatively, you can look at the [Multi-node database replication](#multi-node-database-replication)
instructions on setting up replication with a Patroni cluster.
### PostgreSQL replication
The GitLab **primary** node where the write operations happen connects to
the **primary** database server, and **secondary** nodes
......@@ -48,7 +61,7 @@ WARNING:
Geo works with streaming replication. Logical replication is not supported at this time.
There is an [issue where support is being discussed](https://gitlab.com/gitlab-org/gitlab/-/issues/7420).
### Step 1. Configure the **primary** server
#### Step 1. Configure the **primary** server
1. SSH into your GitLab **primary** server and login as root:
......@@ -77,11 +90,7 @@ There is an [issue where support is being discussed](https://gitlab.com/gitlab-o
This command uses your defined `external_url` in `/etc/gitlab/gitlab.rb`.
1. GitLab 10.4 and up only: Do the following to make sure the `gitlab` database user has a password defined:
NOTE:
Until FDW settings are removed in GitLab version 14.0, avoid using single or double quotes in the
password for PostgreSQL as that leads to errors when reconfiguring.
1. Define a password for the `gitlab` database user:
Generate a MD5 hash of the desired password:
......@@ -104,17 +113,27 @@ There is an [issue where support is being discussed](https://gitlab.com/gitlab-o
gitlab_rails['db_password'] = '<your_password_here>'
```
1. Omnibus GitLab already has a [replication user](https://wiki.postgresql.org/wiki/Streaming_Replication)
called `gitlab_replicator`. You must set the password for this user manually.
You are prompted to enter a password:
1. Define a password for the database [replication user](https://wiki.postgresql.org/wiki/Streaming_Replication).
We will use the username defined in `/etc/gitlab/gitlab.rb` under the `postgresql['sql_replication_user']`
setting. The default value is `gitlab_replicator`, but if you changed it to something else, adapt
the instructions below.
Generate a MD5 hash of the desired password:
```shell
gitlab-ctl set-replication-password
gitlab-ctl pg-password-md5 gitlab_replicator
# Enter password: <your_password_here>
# Confirm password: <your_password_here>
# 950233c0dfc2f39c64cf30457c3b7f1e
```
This command also reads the `postgresql['sql_replication_user']` Omnibus
setting in case you have changed `gitlab_replicator` username to something
else.
Edit `/etc/gitlab/gitlab.rb`:
```ruby
# Fill with the hash generated by `gitlab-ctl pg-password-md5 gitlab_replicator`
postgresql['sql_replication_password'] = '<md5_hash_of_your_password>'
```
If you are using an external database not managed by Omnibus GitLab, you need
to create the replicator user and define a password to it manually:
......@@ -275,7 +294,7 @@ There is an [issue where support is being discussed](https://gitlab.com/gitlab-o
need it when setting up the **secondary** node! The certificate is not sensitive
data.
### Step 2. Configure the **secondary** server
#### Step 2. Configure the **secondary** server
1. SSH into your GitLab **secondary** server and login as root:
......@@ -376,6 +395,7 @@ There is an [issue where support is being discussed](https://gitlab.com/gitlab-o
## Database credentials password (defined previously in primary node)
## - replicate same values here as defined in primary node
##
postgresql['sql_replication_password'] = '<md5_hash_of_your_password>'
postgresql['sql_user_password'] = '<md5_hash_of_your_password>'
gitlab_rails['db_password'] = '<your_password_here>'
```
......@@ -395,7 +415,7 @@ There is an [issue where support is being discussed](https://gitlab.com/gitlab-o
gitlab-ctl restart postgresql
```
### Step 3. Initiate the replication process
#### Step 3. Initiate the replication process
Below we provide a script that connects the database on the **secondary** node to
the database on the **primary** node, replicates the database, and creates the
......@@ -461,37 +481,37 @@ data before running `pg_basebackup`.
The replication process is now complete.
## PgBouncer support (optional)
### PgBouncer support (optional)
[PgBouncer](https://www.pgbouncer.org/) may be used with GitLab Geo to pool
PostgreSQL connections. We recommend using PgBouncer if you use GitLab in a
high-availability configuration with a cluster of nodes supporting a Geo
**primary** site and two other clusters of nodes supporting a Geo **secondary** site.
One for the main database and the other for the tracking database. For more information,
PostgreSQL connections, which can improve performance even when using in a
single instance installation.
We recommend using PgBouncer if you use GitLab in a highly available
configuration with a cluster of nodes supporting a Geo **primary** site and
two other clusters of nodes supporting a Geo **secondary** site. One for the
main database and the other for the tracking database. For more information,
see [High Availability with Omnibus GitLab](../../postgresql/replication_and_failover.md).
## Patroni support
## Multi-node database replication
Support for Patroni is intended to replace `repmgr` as a
[highly available PostgreSQL solution](../../postgresql/replication_and_failover.md)
on the primary node, but it can also be used for PostgreSQL HA on a secondary
site. Similar to `repmgr`, using Patroni on a secondary node is optional.
In GitLab 14.0, Patroni replaced `repmgr` as the supported
[highly available PostgreSQL solution](../../postgresql/replication_and_failover.md).
Starting with GitLab 13.5, Patroni is available for _experimental_ use with Geo
primary and secondary sites. Due to its experimental nature, Patroni support is
subject to change without notice.
NOTE:
If you still haven't [migrated from repmgr to Patroni](#migrating-from-repmgr-to-patroni) you're highly advised to do so.
This experimental implementation has the following limitations:
### Patroni support
- Whenever `gitlab-ctl reconfigure` runs on a Patroni Leader instance, there's a
chance of the node be demoted due to the required short-time restart. To
avoid this, you can pause auto-failover by running `gitlab-ctl patroni pause`.
After a reconfigure, it resumes on its own.
Patroni is the official replication management solution for Geo. It
can be used to build a highly available cluster on the **primary** and a **secondary** Geo site.
Using Patroni on a **secondary** site is optional and you don't have to use the same amount of
nodes on each Geo site.
For instructions about how to set up Patroni on the primary site, see the
[PostgreSQL replication and failover with Omnibus GitLab](../../postgresql/replication_and_failover.md#patroni) page.
### Configuring Patroni cluster for a Geo secondary site
#### Configuring Patroni cluster for a Geo secondary site
In a Geo secondary site, the main PostgreSQL database is a read-only replica of the primary site’s PostgreSQL database.
......@@ -503,7 +523,7 @@ configuration for the secondary site. The internal load balancer provides a sing
endpoint for connecting to the Patroni cluster's leader whenever a new leader is
elected. Be sure to use [password credentials](../../postgresql/replication_and_failover.md#database-authorization-for-patroni) and other database best practices.
#### Step 1. Configure Patroni permanent replication slot on the primary site
##### Step 1. Configure Patroni permanent replication slot on the primary site
To set up database replication with Patroni on a secondary node, we need to
configure a _permanent replication slot_ on the primary node's Patroni cluster,
......@@ -526,7 +546,7 @@ Leader instance**:
retry_join: %w[CONSUL_PRIMARY1_IP CONSUL_PRIMARY2_IP CONSUL_PRIMARY3_IP]
}
repmgr['enable'] = false
roles ['patroni_role']
# You need one entry for each secondary, with a unique name following PostgreSQL slot_name constraints:
#
......@@ -539,15 +559,18 @@ Leader instance**:
patroni['use_pg_rewind'] = true
patroni['postgresql']['max_wal_senders'] = 8 # Use double of the amount of patroni/reserved slots (3 patronis + 1 reserved slot for a Geo secondary).
patroni['postgresql']['max_replication_slots'] = 8 # Use double of the amount of patroni/reserved slots (3 patronis + 1 reserved slot for a Geo secondary).
patroni['replication_password'] = 'PLAIN_TEXT_POSTGRESQL_REPLICATION_PASSWORD'
postgresql['md5_auth_cidr_addresses'] = [
'PATRONI_PRIMARY1_IP/32', 'PATRONI_PRIMARY2_IP/32', 'PATRONI_PRIMARY3_IP/32', 'PATRONI_PRIMARY_PGBOUNCER/32',
'PATRONI_SECONDARY1_IP/32', 'PATRONI_SECONDARY2_IP/32', 'PATRONI_SECONDARY3_IP/32', 'PATRONI_SECONDARY_PGBOUNCER/32' # We list all secondary instances as they can all become a Standby Leader
# We list all secondary instances as they can all become a Standby Leader
postgresql['md5_auth_cidr_addresses'] = %w[
PATRONI_PRIMARY1_IP/32 PATRONI_PRIMARY2_IP/32 PATRONI_PRIMARY3_IP/32 PATRONI_PRIMARY_PGBOUNCER/32
PATRONI_SECONDARY1_IP/32 PATRONI_SECONDARY2_IP/32 PATRONI_SECONDARY3_IP/32 PATRONI_SECONDARY_PGBOUNCER/32
]
postgresql['pgbouncer_user_password'] = 'PGBOUNCER_PASSWORD_HASH'
postgresql['sql_replication_password'] = 'POSTGRESQL_REPLICATION_PASSWORD_HASH'
postgresql['sql_user_password'] = 'POSTGRESQL_PASSWORD_HASH'
postgresql['listen_address'] = '0.0.0.0' # You can use a public or VPC address here instead
```
1. Reconfigure GitLab for the changes to take effect:
......@@ -556,7 +579,7 @@ Leader instance**:
gitlab-ctl reconfigure
```
#### Step 2. Configure the internal load balancer on the primary site
##### Step 2. Configure the internal load balancer on the primary site
To avoid reconfiguring the Standby Leader on the secondary site whenever a new
Leader is elected on the primary site, we need to set up a TCP internal load
......@@ -600,7 +623,7 @@ backend postgresql
Refer to your preferred Load Balancer's documentation for further guidance.
#### Step 3. Configure a PgBouncer node on the secondary site
##### Step 3. Configure a PgBouncer node on the secondary site
A production-ready and highly available configuration requires at least
three Consul nodes, a minimum of one PgBouncer node, but it’s recommended to have
......@@ -624,19 +647,23 @@ Follow the minimal configuration for the PgBouncer node:
roles ['pgbouncer_role']
# PgBouncer configuration
pgbouncer['admin_users'] = %w(pgbouncer gitlab-consul)
pgbouncer['users'] = {
'gitlab-consul': {
# Generate it with: `gitlab-ctl pg-password-md5 gitlab-consul`
password: 'GITLAB_CONSUL_PASSWORD_HASH'
},
'pgbouncer': {
# Generate it with: `gitlab-ctl pg-password-md5 pgbouncer`
password: 'PGBOUNCER_PASSWORD_HASH'
}
}
# Consul configuration
consul['watchers'] = %w(postgresql)
consul['configuration'] = {
retry_join: %w[CONSUL_SECONDARY1_IP CONSUL_SECONDARY2_IP CONSUL_SECONDARY3_IP]
}
consul['monitoring_service_discovery'] = true
```
......@@ -652,13 +679,13 @@ Follow the minimal configuration for the PgBouncer node:
gitlab-ctl write-pgpass --host 127.0.0.1 --database pgbouncer --user pgbouncer --hostuser gitlab-consul
```
1. Restart the PgBouncer service:
1. Reload the PgBouncer service:
```shell
gitlab-ctl restart pgbouncer
gitlab-ctl hup pgbouncer
```
#### Step 4. Configure a Standby cluster on the secondary site
##### Step 4. Configure a Standby cluster on the secondary site
NOTE:
If you are converting a secondary site to a Patroni Cluster, you must start
......@@ -676,21 +703,18 @@ For each Patroni instance on the secondary site:
1. Edit `/etc/gitlab/gitlab.rb` and add the following:
```ruby
roles ['consul_role', 'postgres_role']
roles ['consul_role', 'patroni_role']
consul['enable'] = true
consul['configuration'] = {
retry_join: %w[CONSUL_SECONDARY1_IP CONSUL_SECONDARY2_IP CONSUL_SECONDARY3_IP]
}
repmgr['enable'] = false
postgresql['md5_auth_cidr_addresses'] = [
'PATRONI_SECONDARY1_IP/32', 'PATRONI_SECONDARY2_IP/32', 'PATRONI_SECONDARY3_IP/32', 'PATRONI_SECONDARY_PGBOUNCER/32',
# Any other instance that needs access to the database as per documentation
]
patroni['enable'] = false
patroni['standby_cluster']['enable'] = true
patroni['standby_cluster']['host'] = 'INTERNAL_LOAD_BALANCER_PRIMARY_IP'
patroni['standby_cluster']['port'] = INTERNAL_LOAD_BALANCER_PRIMARY_PORT
......@@ -699,6 +723,11 @@ For each Patroni instance on the secondary site:
patroni['use_pg_rewind'] = true
patroni['postgresql']['max_wal_senders'] = 5 # A minimum of three for one replica, plus two for each additional replica
patroni['postgresql']['max_replication_slots'] = 5 # A minimum of three for one replica, plus two for each additional replica
postgresql['pgbouncer_user_password'] = 'PGBOUNCER_PASSWORD_HASH'
postgresql['sql_replication_password'] = 'POSTGRESQL_REPLICATION_PASSWORD_HASH'
postgresql['sql_user_password'] = 'POSTGRESQL_PASSWORD_HASH'
postgresql['listen_address'] = '0.0.0.0' # You can use a public or VPC address here instead
```
1. Reconfigure GitLab for the changes to take effect.
......@@ -708,28 +737,6 @@ For each Patroni instance on the secondary site:
gitlab-ctl reconfigure
```
1. Remove the PostgreSQL data directory:
WARNING:
If you are converting a secondary site to a Patroni Cluster, you must skip
this step on the PostgreSQL instance.
```shell
rm -rf /var/opt/gitlab/postgresql/data
```
1. Edit `/etc/gitlab/gitlab.rb` to enable Patroni:
```ruby
patroni['enable'] = true
```
1. Reconfigure GitLab for the changes to take effect:
```shell
gitlab-ctl reconfigure
```
### Migrating from repmgr to Patroni
1. Before migrating, it is recommended that there is no replication lag between the primary and secondary sites and that replication is paused. In GitLab 13.2 and later, you can pause and resume replication with `gitlab-ctl geo-replication-pause` and `gitlab-ctl geo-replication-resume` on a Geo secondary database node.
......
Markdown is supported
0%
or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment