Merge branch 'brodock/update-geo-patroni-docs' into 'master'

Improve Geo documentation for Patroni Beta

See merge request gitlab-org/gitlab!62330

doc/administration/geo/setup/database.md

@@ -25,7 +25,20 @@ size.
 You are encouraged to first read through all the steps before executing them
 in your testing/production environment.
 
-## PostgreSQL replication
+## Single instance database replication
+
+A single instance database replication is easier to set up and still provides the same Geo capabilities
+as a clusterized alternative. It's useful for setups running on a single machine
+or trying to evaluate Geo for a future clusterized installation.
+
+A single instance can be expanded to a clusterized version using Patroni, which is recommended for a 
+highly available architecture.
+
+Follow below the instructions on how to set up PostgreSQL replication as a single instance database. 
+Alternatively, you can look at the [Multi-node database replication](#multi-node-database-replication) 
+instructions on setting up replication with a Patroni cluster.
+
+### PostgreSQL replication
 
 The GitLab **primary** node where the write operations happen connects to
 the **primary** database server, and **secondary** nodes 
@@ -48,7 +61,7 @@ WARNING:
 Geo works with streaming replication. Logical replication is not supported at this time.
 There is an [issue where support is being discussed](https://gitlab.com/gitlab-org/gitlab/-/issues/7420).
 
-### Step 1. Configure the **primary** server
+#### Step 1. Configure the **primary** server
 
 1. SSH into your GitLab **primary** server and login as root:
 
@@ -77,11 +90,7 @@ There is an [issue where support is being discussed](https://gitlab.com/gitlab-o
 
    This command uses your defined `external_url` in `/etc/gitlab/gitlab.rb`.
 
-1. GitLab 10.4 and up only: Do the following to make sure the `gitlab` database user has a password defined:
-
-   NOTE:
-   Until FDW settings are removed in GitLab version 14.0, avoid using single or double quotes in the
-   password for PostgreSQL as that leads to errors when reconfiguring.
+1. Define a password for the `gitlab` database user:
 
    Generate a MD5 hash of the desired password:
 
@@ -103,18 +112,28 @@ There is an [issue where support is being discussed](https://gitlab.com/gitlab-o
    # must be present in all application nodes.
    gitlab_rails['db_password'] = '<your_password_here>'
    ```
+   
+1. Define a password for the database [replication user](https://wiki.postgresql.org/wiki/Streaming_Replication).
 
-1. Omnibus GitLab already has a [replication user](https://wiki.postgresql.org/wiki/Streaming_Replication)
-   called `gitlab_replicator`. You must set the password for this user manually.
-   You are prompted to enter a password:
+   We will use the username defined in `/etc/gitlab/gitlab.rb` under the `postgresql['sql_replication_user']`
+   setting. The default value is `gitlab_replicator`, but if you changed it to something else, adapt 
+   the instructions below.
+   
+   Generate a MD5 hash of the desired password:
 
    ```shell
-   gitlab-ctl set-replication-password
+   gitlab-ctl pg-password-md5 gitlab_replicator
+   # Enter password: <your_password_here>
+   # Confirm password: <your_password_here>
+   # 950233c0dfc2f39c64cf30457c3b7f1e
    ```
 
-   This command also reads the `postgresql['sql_replication_user']` Omnibus
-   setting in case you have changed `gitlab_replicator` username to something
-   else.
+   Edit `/etc/gitlab/gitlab.rb`:
+
+   ```ruby
+   # Fill with the hash generated by `gitlab-ctl pg-password-md5 gitlab_replicator`
+   postgresql['sql_replication_password'] = '<md5_hash_of_your_password>'
+   ```
 
    If you are using an external database not managed by Omnibus GitLab, you need
    to create the replicator user and define a password to it manually:
@@ -275,7 +294,7 @@ There is an [issue where support is being discussed](https://gitlab.com/gitlab-o
    need it when setting up the **secondary** node! The certificate is not sensitive
    data.
 
-### Step 2. Configure the **secondary** server
+#### Step 2. Configure the **secondary** server
 
 1. SSH into your GitLab **secondary** server and login as root:
 
@@ -376,6 +395,7 @@ There is an [issue where support is being discussed](https://gitlab.com/gitlab-o
    ## Database credentials password (defined previously in primary node)
    ## - replicate same values here as defined in primary node
    ##
+   postgresql['sql_replication_password'] = '<md5_hash_of_your_password>'
    postgresql['sql_user_password'] = '<md5_hash_of_your_password>'
    gitlab_rails['db_password'] = '<your_password_here>'
    ```
@@ -395,7 +415,7 @@ There is an [issue where support is being discussed](https://gitlab.com/gitlab-o
    gitlab-ctl restart postgresql
    ```
 
-### Step 3. Initiate the replication process
+#### Step 3. Initiate the replication process
 
 Below we provide a script that connects the database on the **secondary** node to
 the database on the **primary** node, replicates the database, and creates the
@@ -461,37 +481,37 @@ data before running `pg_basebackup`.
 
 The replication process is now complete.
 
-## PgBouncer support (optional)
+### PgBouncer support (optional)
 
 [PgBouncer](https://www.pgbouncer.org/) may be used with GitLab Geo to pool
-PostgreSQL connections. We recommend using PgBouncer if you use GitLab in a
-high-availability configuration with a cluster of nodes supporting a Geo
-**primary** site and two other clusters of nodes supporting a Geo **secondary** site.
-One for the main database and the other for the tracking database. For more information,
+PostgreSQL connections, which can improve performance even when using in a 
+single instance installation. 
+
+We recommend using PgBouncer if you use GitLab in a highly available  
+configuration with a cluster of nodes supporting a Geo **primary** site and 
+two other clusters of nodes supporting a Geo **secondary** site. One for the 
+main database and the other for the tracking database. For more information,
 see [High Availability with Omnibus GitLab](../../postgresql/replication_and_failover.md).
 
-## Patroni support
+## Multi-node database replication
 
-Support for Patroni is intended to replace `repmgr` as a
-[highly available PostgreSQL solution](../../postgresql/replication_and_failover.md)
-on the primary node, but it can also be used for PostgreSQL HA on a secondary
-site. Similar to `repmgr`, using Patroni on a secondary node is optional.
+In GitLab 14.0, Patroni replaced `repmgr` as the supported
+[highly available PostgreSQL solution](../../postgresql/replication_and_failover.md).
 
-Starting with GitLab 13.5, Patroni is available for _experimental_ use with Geo
-primary and secondary sites. Due to its experimental nature, Patroni support is
-subject to change without notice.
+NOTE:
+If you still haven't [migrated from repmgr to Patroni](#migrating-from-repmgr-to-patroni) you're highly advised to do so.
 
-This experimental implementation has the following limitations:
+### Patroni support
 
-- Whenever `gitlab-ctl reconfigure` runs on a Patroni Leader instance, there's a
-  chance of the node be demoted due to the required short-time restart. To
-  avoid this, you can pause auto-failover by running `gitlab-ctl patroni pause`.
-  After a reconfigure, it resumes on its own.
+Patroni is the official replication management solution for Geo. It
+can be used to build a highly available cluster on the **primary** and a **secondary** Geo site.
+Using Patroni on a **secondary** site is optional and you don't have to use the same amount of 
+nodes on each Geo site.
 
 For instructions about how to set up Patroni on the primary site, see the
 [PostgreSQL replication and failover with Omnibus GitLab](../../postgresql/replication_and_failover.md#patroni) page.
 
-### Configuring Patroni cluster for a Geo secondary site
+#### Configuring Patroni cluster for a Geo secondary site
 
 In a Geo secondary site, the main PostgreSQL database is a read-only replica of the primary site’s PostgreSQL database.
 
@@ -503,7 +523,7 @@ configuration for the secondary site. The internal load balancer provides a sing
 endpoint for connecting to the Patroni cluster's leader whenever a new leader is
 elected. Be sure to use [password credentials](../../postgresql/replication_and_failover.md#database-authorization-for-patroni) and other database best practices.
 
-#### Step 1. Configure Patroni permanent replication slot on the primary site
+##### Step 1. Configure Patroni permanent replication slot on the primary site
 
 To set up database replication with Patroni on a secondary node, we need to
 configure a _permanent replication slot_ on the primary node's Patroni cluster,
@@ -526,8 +546,8 @@ Leader instance**:
      retry_join: %w[CONSUL_PRIMARY1_IP CONSUL_PRIMARY2_IP CONSUL_PRIMARY3_IP]
    }
 
-   repmgr['enable'] = false
-
+   roles ['patroni_role']
+   
    # You need one entry for each secondary, with a unique name following PostgreSQL slot_name constraints:
    #
    # Configuration syntax is: 'unique_slotname' => { 'type' => 'physical' },
@@ -539,15 +559,18 @@ Leader instance**:
    patroni['use_pg_rewind'] = true
    patroni['postgresql']['max_wal_senders'] = 8 # Use double of the amount of patroni/reserved slots (3 patronis + 1 reserved slot for a Geo secondary).
    patroni['postgresql']['max_replication_slots'] = 8 # Use double of the amount of patroni/reserved slots (3 patronis + 1 reserved slot for a Geo secondary).
+   patroni['replication_password'] = 'PLAIN_TEXT_POSTGRESQL_REPLICATION_PASSWORD'
 
-   postgresql['md5_auth_cidr_addresses'] = [
-     'PATRONI_PRIMARY1_IP/32', 'PATRONI_PRIMARY2_IP/32', 'PATRONI_PRIMARY3_IP/32', 'PATRONI_PRIMARY_PGBOUNCER/32',
-     'PATRONI_SECONDARY1_IP/32', 'PATRONI_SECONDARY2_IP/32', 'PATRONI_SECONDARY3_IP/32', 'PATRONI_SECONDARY_PGBOUNCER/32' # We list all secondary instances as they can all become a Standby Leader
+   # We list all secondary instances as they can all become a Standby Leader
+   postgresql['md5_auth_cidr_addresses'] = %w[
+     PATRONI_PRIMARY1_IP/32 PATRONI_PRIMARY2_IP/32 PATRONI_PRIMARY3_IP/32 PATRONI_PRIMARY_PGBOUNCER/32
+     PATRONI_SECONDARY1_IP/32 PATRONI_SECONDARY2_IP/32 PATRONI_SECONDARY3_IP/32 PATRONI_SECONDARY_PGBOUNCER/32
    ]
 
    postgresql['pgbouncer_user_password'] = 'PGBOUNCER_PASSWORD_HASH'
    postgresql['sql_replication_password'] = 'POSTGRESQL_REPLICATION_PASSWORD_HASH'
    postgresql['sql_user_password'] = 'POSTGRESQL_PASSWORD_HASH'
+   postgresql['listen_address'] = '0.0.0.0' # You can use a public or VPC address here instead
    ```
 
 1. Reconfigure GitLab for the changes to take effect:
@@ -556,7 +579,7 @@ Leader instance**:
    gitlab-ctl reconfigure
    ```
 
-#### Step 2. Configure the internal load balancer on the primary site
+##### Step 2. Configure the internal load balancer on the primary site
 
 To avoid reconfiguring the Standby Leader on the secondary site whenever a new
 Leader is elected on the primary site, we need to set up a TCP internal load
@@ -600,7 +623,7 @@ backend postgresql
 
 Refer to your preferred Load Balancer's documentation for further guidance.
 
-#### Step 3. Configure a PgBouncer node on the secondary site
+##### Step 3. Configure a PgBouncer node on the secondary site
 
 A production-ready and highly available configuration requires at least
 three Consul nodes, a minimum of one PgBouncer node, but it’s recommended to have
@@ -624,19 +647,23 @@ Follow the minimal configuration for the PgBouncer node:
    roles ['pgbouncer_role']
 
    # PgBouncer configuration
+   pgbouncer['admin_users'] = %w(pgbouncer gitlab-consul)
    pgbouncer['users'] = {
+   'gitlab-consul': {
+      # Generate it with: `gitlab-ctl pg-password-md5 gitlab-consul`
+      password: 'GITLAB_CONSUL_PASSWORD_HASH'
+    },
      'pgbouncer': {
+       # Generate it with: `gitlab-ctl pg-password-md5 pgbouncer`
        password: 'PGBOUNCER_PASSWORD_HASH'
      }
    }
 
    # Consul configuration
    consul['watchers'] = %w(postgresql)
-
    consul['configuration'] = {
      retry_join: %w[CONSUL_SECONDARY1_IP CONSUL_SECONDARY2_IP CONSUL_SECONDARY3_IP]
    }
-
    consul['monitoring_service_discovery'] =  true
    ```
 
@@ -652,13 +679,13 @@ Follow the minimal configuration for the PgBouncer node:
    gitlab-ctl write-pgpass --host 127.0.0.1 --database pgbouncer --user pgbouncer --hostuser gitlab-consul
    ```
 
-1. Restart the PgBouncer service:
+1. Reload the PgBouncer service:
 
    ```shell
-   gitlab-ctl restart pgbouncer
+   gitlab-ctl hup pgbouncer
    ```
 
-#### Step 4. Configure a Standby cluster on the secondary site
+##### Step 4. Configure a Standby cluster on the secondary site
 
 NOTE:
 If you are converting a secondary site to a Patroni Cluster, you must start
@@ -676,21 +703,18 @@ For each Patroni instance on the secondary site:
 1. Edit `/etc/gitlab/gitlab.rb` and add the following:
 
    ```ruby
-   roles ['consul_role', 'postgres_role']
+   roles ['consul_role', 'patroni_role']
 
    consul['enable'] = true
    consul['configuration'] = {
      retry_join: %w[CONSUL_SECONDARY1_IP CONSUL_SECONDARY2_IP CONSUL_SECONDARY3_IP]
    }
 
-   repmgr['enable'] = false
-
    postgresql['md5_auth_cidr_addresses'] = [
      'PATRONI_SECONDARY1_IP/32', 'PATRONI_SECONDARY2_IP/32', 'PATRONI_SECONDARY3_IP/32', 'PATRONI_SECONDARY_PGBOUNCER/32',
      # Any other instance that needs access to the database as per documentation
    ]
 
-   patroni['enable'] = false
    patroni['standby_cluster']['enable'] = true
    patroni['standby_cluster']['host'] = 'INTERNAL_LOAD_BALANCER_PRIMARY_IP'
    patroni['standby_cluster']['port'] = INTERNAL_LOAD_BALANCER_PRIMARY_PORT
@@ -699,6 +723,11 @@ For each Patroni instance on the secondary site:
    patroni['use_pg_rewind'] = true
    patroni['postgresql']['max_wal_senders'] = 5 # A minimum of three for one replica, plus two for each additional replica
    patroni['postgresql']['max_replication_slots'] = 5 # A minimum of three for one replica, plus two for each additional replica
+   
+   postgresql['pgbouncer_user_password'] = 'PGBOUNCER_PASSWORD_HASH'
+   postgresql['sql_replication_password'] = 'POSTGRESQL_REPLICATION_PASSWORD_HASH'
+   postgresql['sql_user_password'] = 'POSTGRESQL_PASSWORD_HASH'
+   postgresql['listen_address'] = '0.0.0.0' # You can use a public or VPC address here instead
    ```
 
 1. Reconfigure GitLab for the changes to take effect.
@@ -708,28 +737,6 @@ For each Patroni instance on the secondary site:
    gitlab-ctl reconfigure
    ```
 
-1. Remove the PostgreSQL data directory:
-
-   WARNING:
-   If you are converting a secondary site to a Patroni Cluster, you must skip
-   this step on the PostgreSQL instance.
-
-   ```shell
-   rm -rf /var/opt/gitlab/postgresql/data
-   ```
-
-1. Edit `/etc/gitlab/gitlab.rb` to enable Patroni:
-
-   ```ruby
-   patroni['enable'] = true
-   ```
-
-1. Reconfigure GitLab for the changes to take effect:
-
-   ```shell
-   gitlab-ctl reconfigure
-   ```
-
 ### Migrating from repmgr to Patroni
 
 1. Before migrating, it is recommended that there is no replication lag between the primary and secondary sites and that replication is paused. In GitLab 13.2 and later, you can pause and resume replication with `gitlab-ctl geo-replication-pause` and `gitlab-ctl geo-replication-resume` on a Geo secondary database node.