Merge branch 'mk/mitigate-some-geo-ha-setup-pitfalls-docs' into 'master'

Mitigate a few Geo HA setup pitfalls in the documentation

Closes #7601

See merge request gitlab-org/gitlab-ee!8117

doc/administration/geo/disaster_recovery/index.md

@@ -83,7 +83,14 @@ must disable the primary.
    [update the primary domain DNS record](#step-4-optional-updating-the-primary-domain-dns-record),
    you may wish to lower the TTL now to speed up propagation.
 
-### Step 3. Promoting a secondary Geo replica
+### Step 3. Promoting a **secondary** node
+
+NOTE: **Note:**
+A new **secondary** should not be added at this time. If you want to add a new
+**secondary**, do this after you have completed the entire process of promoting
+the **secondary** to the **primary**.
+
+#### Promoting a **secondary** node running on a single machine
 
 1. SSH in to your **secondary** and login as root:
 
@@ -91,43 +98,66 @@ must disable the primary.
     sudo -i
     ```
 
-1. Edit `/etc/gitlab/gitlab.rb` to reflect its new status as primary by
-   removing the following line:
+1. Edit `/etc/gitlab/gitlab.rb` to reflect its new status as **primary** by
+   removing any lines that enabled the `geo_secondary_role`:
 
     ```ruby
-    ## REMOVE THIS LINE
+    ## In pre-11.5 documentation, the role was enabled as follows. Remove this line.
     geo_secondary_role['enable'] = true
-    ```
 
-    A new secondary should not be added at this time. If you want to add a new
-    secondary, do this after you have completed the entire process of promoting
-    the secondary to the primary.
+    ## In 11.5+ documentation, the role was enabled as follows. Remove this line.
+    roles ['geo_secondary_role']
+    ```
 
-1. Promote the secondary to primary. Execute:
+1. Promote the **secondary** to **primary**. Execute:
 
     ```bash
     gitlab-ctl promote-to-primary-node
     ```
 
-1. Verify you can connect to the newly promoted primary using the URL used
-   previously for the secondary.
-1. Success! The secondary has now been promoted to primary.
+1. Verify you can connect to the newly promoted **primary** using the URL used
+   previously for the **secondary**.
+1. Success! The **secondary** has now been promoted to **primary**.
 
-#### Promoting a node with HA
+#### Promoting a **secondary** node with HA
 
-The `gitlab-ctl promote-to-primary-node` command cannot be used yet in conjunction with
-High Availability or with multiple machines, as it can only perform changes on
-a single one.
+The `gitlab-ctl promote-to-primary-node` command cannot be used yet in
+conjunction with High Availability or with multiple machines, as it can only
+perform changes on a **secondary** with only a single machine. Instead, you must
+do this manually.
 
-The command above does the following changes:
+1. SSH in to the database node in the **secondary** and trigger PostgreSQL to
+   promote to read-write:
 
-- Promotes the PostgreSQL secondary to primary
-- Executes `gitlab-ctl reconfigure` to apply the changes in `/etc/gitlab/gitlab.rb`
-- Runs `gitlab-rake geo:set_secondary_as_primary`
+    ```bash
+    sudo gitlab-pg-ctl promote
+    ```
+
+1. Edit `/etc/gitlab/gitlab.rb` on every machine in the **secondary** to
+   reflect its new status as **primary** by removing any lines that enabled the
+   `geo_secondary_role`:
+
+    ```ruby
+    ## In pre-11.5 documentation, the role was enabled as follows. Remove this line.
+    geo_secondary_role['enable'] = true
+
+    ## In 11.5+ documentation, the role was enabled as follows. Remove this line.
+    roles ['geo_secondary_role']
+    ```
+
+    After making these changes [Reconfigure GitLab](../../restart_gitlab.md#omnibus-gitlab-reconfigure) each
+    machine so the changes take effect.
+
+1. Promote the **secondary** to **primary**. SSH into a single application
+   server and execute:
+
+    ```bash
+    sudo gitlab-rake geo:set_secondary_as_primary
+    ```
 
-You need to make sure all the affected machines no longer have `geo_secondary_role['enable'] = true` in
-`/etc/gitlab/gitlab.rb`, that you execute the database promotion on the required database nodes
-and you execute the `gitlab-rake geo:set_secondary_as_primary` in a machine running the application server.
+1. Verify you can connect to the newly promoted **primary** using the URL used
+   previously for the **secondary**.
+1. Success! The **secondary** has now been promoted to **primary**.
 
 ### Step 4. (Optional) Updating the primary domain DNS record
 
@@ -198,7 +228,7 @@ and after that you also need two extra steps.
 
     ```ruby
     ## Enable a Geo Primary role (if you haven't yet)
-    geo_primary_role['enable'] = true
+    roles ['geo_primary_role']
 
     ##
     # Primary and Secondary addresses

doc/administration/geo/replication/database.md

@@ -3,7 +3,7 @@
 NOTE: **Note:**
 This is the documentation for the Omnibus GitLab packages. For installations
 from source, follow the
-[**database replication for installations from source**][database-source] guide.
+[Geo database replication (source)](database_source.md) guide.
 
 NOTE: **Note:**
 If your GitLab installation uses external PostgreSQL, the Omnibus roles
@@ -101,10 +101,12 @@ The following guide assumes that:
     This command will also read the `postgresql['sql_replication_user']` Omnibus
     setting in case you have changed `gitlab_replicator` username to something
     else.
-    
+
     If you are using an external database not managed by Omnibus GitLab, you need
     to create the replicator user and define a password to it manually.
-    Check [How to create replication user][database-source-primary] documentation. 
+    For information on how to create a replication user, refer to the
+    [appropriate step](database_source.md#step-1-configure-the-primary-server)
+    in [Geo database replication (source)](database_source.md).
 
 1. Configure PostgreSQL to listen on network interfaces
 
@@ -161,14 +163,14 @@ The following guide assumes that:
     ## Geo Primary role
     ## - configure dependent flags automatically to enable Geo
     ##
-    geo_primary_role['enable'] = true
+    roles ['geo_primary_role']
 
     ##
     ## Primary address
     ## - replace '1.2.3.4' with the primary public or VPC address
     ##
     postgresql['listen_address'] = '1.2.3.4'
-    
+
     ##
     # Primary and Secondary addresses
     # - replace '1.2.3.4' with the primary public or VPC address
@@ -263,8 +265,8 @@ The following guide assumes that:
     gitlab-ctl stop sidekiq
     ```
 
-    NOTE: **Note**: 
-    This step is important so we don't try to execute anything before the node is fully configured. 
+    NOTE: **Note**:
+    This step is important so we don't try to execute anything before the node is fully configured.
 
 1. [Check TCP connectivity][rake-maintenance] to the primary's PostgreSQL server:
 
@@ -272,7 +274,7 @@ The following guide assumes that:
     gitlab-rake gitlab:tcp_check[1.2.3.4,5432]
     ```
 
-    NOTE: **Note**: 
+    NOTE: **Note**:
     If this step fails, you may be using the wrong IP address, or a firewall may
     be preventing access to the server. Check the IP address, paying close
     attention to the difference between public and private addresses and ensure
@@ -325,8 +327,8 @@ The following guide assumes that:
     ## Geo Secondary role
     ## - configure dependent flags automatically to enable Geo
     ##
-    geo_secondary_role['enable'] = true 
-  
+    roles ['geo_secondary_role']
+
     ##
     ## Secondary address
     ## - replace '5.6.7.8' with the secondary public or VPC address
@@ -336,7 +338,7 @@ The following guide assumes that:
 
     ##
     ## Database credentials password (defined previously in primary node)
-    ## - replicate same values here as defined in primary node 
+    ## - replicate same values here as defined in primary node
     ##
     postgresql['sql_user_password'] = 'fca0b89a972d69f00eb3ec98a5838484'
     gitlab_rails['db_password'] = 'mypassword'
@@ -348,7 +350,7 @@ The following guide assumes that:
     ```
 
     For external PostgreSQL instances, [see additional instructions][external postgresql].
-    If you bring a former primary back online to serve as a secondary then you also need to remove `geo_primary_role['enable'] = true`.
+    If you bring a former **primary** node back online to serve as a **secondary** node, then you also need to remove `roles ['geo_primary_role']` or `geo_primary_role['enable'] = true`.
 
 1. Reconfigure GitLab for the changes to take effect:
 
@@ -362,7 +364,7 @@ The following guide assumes that:
     gitlab-ctl restart postgresql
     gitlab-ctl reconfigure
     ```
-    
+
     This last reconfigure will provision the FDW configuration and enable it.
 
 ### Step 3. Initiate the replication process
@@ -484,7 +486,7 @@ You only need to follow the steps below if you are not using the managed
 PostgreSQL from a Omnibus GitLab package.
 
 Geo secondary nodes use a tracking database to keep track of replication
-status and recover automatically from some replication issues. 
+status and recover automatically from some replication issues.
 
 This is a separate PostgreSQL installation that can be configured to use
 FDW to connect with the secondary database for improved performance.
@@ -495,10 +497,10 @@ the instructions below:
 1. Edit `/etc/gitlab/gitlab.rb` with the connection params and credentials
 
     ```ruby
-    # note this is shared between both databases, 
+    # note this is shared between both databases,
     # make sure you define the same password in both
     gitlab_rails['db_password'] = 'mypassword'
-    
+
     geo_secondary['db_host'] = '2.3.4.5' # change to the correct public IP
     geo_secondary['db_port'] = 5431      # change to the correct port
     geo_secondary['db_fdw'] = true       # enable FDW
@@ -510,7 +512,7 @@ the instructions below:
     ```bash
     gitlab-ctl reconfigure
     ```
-    
+
 1. Run the tracking database migrations:
 
     ```bash
@@ -521,36 +523,36 @@ the instructions below:
 
     Save the script below in a file, ex. `/tmp/geo_fdw.sh` and modify the connection
     params to match your environment. Execute it to set up the FDW connection.
-    
+
     ```bash
     #!/bin/bash
-  
+
     # Secondary Database connection params:
     DB_HOST="5.6.7.8" # change to the public IP or VPC private IP
     DB_NAME="gitlabhq_production"
     DB_USER="gitlab"
     DB_PORT="5432"
-    
+
     # Tracking Database connection params:
     GEO_DB_HOST="2.3.4.5" # change to the public IP or VPC private IP
     GEO_DB_NAME="gitlabhq_geo_production"
     GEO_DB_USER="gitlab_geo"
     GEO_DB_PORT="5432"
- 
+
     query_exec () {
       gitlab-psql -h $GEO_DB_HOST -d $GEO_DB_NAME -p $GEO_DB_PORT -c "${1}"
     }
- 
+
     query_exec "CREATE EXTENSION postgres_fdw;"
     query_exec "CREATE SERVER gitlab_secondary FOREIGN DATA WRAPPER postgres_fdw OPTIONS (host '${DB_HOST}', dbname '${DB_NAME}', port '${DB_PORT}');"
     query_exec "CREATE USER MAPPING FOR ${GEO_DB_USER} SERVER gitlab_secondary OPTIONS (user '${DB_USER}');"
     query_exec "CREATE SCHEMA gitlab_secondary;"
     query_exec "GRANT USAGE ON FOREIGN SERVER gitlab_secondary TO ${GEO_DB_USER};"
     ```
-    
+
     NOTE: **Note:** The script template above uses `gitlab-psql` as it's intended to be executed from the Geo machine,
     but you can change it to `psql` and run it from any machine that has access to the database.
-    
+
 1. Restart GitLab
 
     ```bash
@@ -621,8 +623,6 @@ Read the [troubleshooting document](troubleshooting.md).
 [tracking]: database_source.md#enable-tracking-database-on-the-secondary-server
 [FDW]: https://www.postgresql.org/docs/9.6/static/postgres-fdw.html
 [toc]: index.md#using-omnibus-gitlab
-[database-source]: database_source.md
-[database-source-primary]: database_source.md#step-1-configure-the-primary-server
 [rake-maintenance]: ../../raketasks/maintenance.md
 [pg-docs-ssl]: https://www.postgresql.org/docs/9.6/static/libpq-ssl.html#LIBPQ-SSL-PROTECTION
 [pg-docs-runtime-conn]: https://www.postgresql.org/docs/9.6/static/runtime-config-connection.html