Update the Geo documentation to replicate all secrets to the secondary

changelogs/unreleased-ee/4253-dr-compliant-secrets.yml

+---
+title: Update the Geo documentation to replicate all secrets to the secondary
+merge_request: 4188
+author:
+type: fixed

doc/gitlab-geo/configuration.md

@@ -25,32 +25,49 @@ in your testing/production environment.
 - **Do not** add anything in the secondaries Geo nodes admin area
   (**Admin Area ➔ Geo Nodes**). This is handled solely by the primary node.
 
-### Step 1. Copying the database encryption key
+### Step 1. Manually replicate secret GitLab values
 
-GitLab stores a unique encryption key on disk that is used to encrypt
-sensitive data stored in the database. All secondary nodes must have the
-**exact same value** for `db_key_base` as defined on the primary node.
+GitLab stores a number of secret values in the `/etc/gitlab/gitlab-secrets.json`
+file which *must* match between the primary and secondary nodes. Until there is
+a means of automatically replicating these between nodes (see
+[issue #3789](https://gitlab.com/gitlab-org/gitlab-ee/issues/3789)), they must
+be manually replicated to the secondary.
 
-1. SSH into the **primary** node, and execute the command below
-to display the current encryption key:
+1. SSH into the **primary** node, and execute the command below:
 
     ```bash
-    sudo gitlab-rake geo:db:show_encryption_key
+    sudo cat /etc/gitlab/gitlab-secrets.json
     ```
 
-Copy the encryption key to bring it to the secondary node in the following steps.
+    This will display the secrets that need to be replicated, in JSON format.
 
-1. SSH into the **secondary** node and login as root:
+1. SSH into the **secondary** node and login as the `root` user:
 
     ```
     sudo -i
     ```
 
-1. Add the following to `/etc/gitlab/gitlab.rb`, replacing `encryption-key` with the output
-   of the previous command:
+1. Make a backup of any existing secrets:
 
-    ```ruby
-    gitlab_rails['db_key_base'] = 'encryption-key'
+    ```bash
+    mv /etc/gitlab/gitlab-secrets.json /etc/gitlab/gitlab-secrets.json.`date +%F`
+    ```
+
+1. Copy `/etc/gitlab/gitlab-secrets.json` from the primary to the secondary, or
+   copy-and-paste the file contents between nodes:
+
+    ```bash
+    sudo editor /etc/gitlab/gitlab-secrets.json
+
+    # paste the output of the `cat` command you ran on the primary
+    # save and exit
+    ```
+
+1. Ensure the file permissions are correct:
+
+    ```bash
+    chown root:root /etc/gitlab/gitlab-secrets.json
+    chmod 0600 /etc/gitlab/gitlab-secrets.json
     ```
 
 1. Reconfigure the secondary node for the change to take effect:

doc/gitlab-geo/configuration_source.md

@@ -26,30 +26,50 @@ in your testing/production environment.
 - **Do not** add anything in the secondaries Geo nodes admin area
   (**Admin Area ➔ Geo Nodes**). This is handled solely by the primary node.
 
-### Step 1. Copying the database encryption key
+### Step 1. Manually replicate secret GitLab values
 
-GitLab stores a unique encryption key on disk that is used to encrypt
-sensitive data stored in the database. All secondary nodes must have the
-**exact same value** for `db_key_base` as defined on the primary node.
+GitLab stores a number of secret values in the `/home/git/gitlab/config/secrets.yml`
+file which *must* match between the primary and secondary nodes. Until there is
+a means of automatically replicating these between nodes (see
+[issue #3789](https://gitlab.com/gitlab-org/gitlab-ee/issues/3789)), they must
+be manually replicated to the secondary.
 
-1. SSH into the **primary** node,  and execute the command below to display the
-current encryption key:
+1. SSH into the **primary** node, and execute the command below:
 
     ```bash
-    sudo -u git -H bundle exec rake geo:db:show_encryption_key RAILS_ENV=production
+    sudo cat /home/git/gitlab/config/secrets.yml
     ```
 
-Copy the encryption key to bring it to the secondary node in the following steps.
+    This will display the secrets that need to be replicated, in YAML format.
 
-1. SSH into the **secondary**, and execute the command below to open the
-`secrets.yml` file:
+1. SSH into the **secondary** node and login as the `git` user:
 
     ```bash
-    sudo -u git -H editor config/secrets.yml
+    sudo -i -u git
     ```
 
-1. Change the value of `db_key_base` to the output from the primary node.
-Then save and close the file.
+1. Make a backup of any existing secrets:
+
+    ```bash
+    mv /home/git/gitlab/config/secrets.yml /home/git/gitlab/config/secrets.yml.`date +%F`
+    ```
+
+1. Copy `/home/git/gitlab/config/secrets.yml` from the primary to the secondary, or
+   copy-and-paste the file contents between nodes:
+
+    ```bash
+    sudo editor /home/git/gitlab/config/secrets.yml
+
+    # paste the output of the `cat` command you ran on the primary
+    # save and exit
+    ```
+
+1. Ensure the file permissions are correct:
+
+    ```bash
+    chown git:git /home/git/gitlab/config/secrets.yml
+    chmod 0600 /home/git/gitlab/config/secrets.yml
+    ```
 
 1. Restart GitLab for the changes to take effect:
 

doc/gitlab-geo/faq.md

@@ -10,6 +10,19 @@ primary, but this is not officially supported yet.
 If you still want to proceed, see our step-by-step instructions on how to
 manually [promote a secondary node](disaster-recovery.md) into primary.
 
+## I followed the disaster recovery instructions and now two-factor auth is broken!
+
+The setup instructions for GitLab Geo prior to 10.5 failed to replicate the
+`otp_key_base` secret, which used to encrypt the two-factor authentication
+secrets stored in the database. If it differs between primary and secondary
+nodes, users with two-factor authentication enabled won't be able to log in
+after a DR failover.
+
+If you still have access to the old primary node, you can follow the
+instructions in the [Upgrading to GitLab 10.5](updating_the_geo_nodes.md#upgrading-to-gitlab-105)
+section to resolve the error. Otherwise, the secret is lost and you'll need to
+[reset two-factor authentication for all users](../security/two_factor_authentication.md#disabling-2fa-for-everyone).
+
 ## What data is replicated to a secondary node?
 
 We currently replicate project repositories, LFS objects, generated

doc/gitlab-geo/updating_the_geo_nodes.md

@@ -14,6 +14,28 @@ all you need to do is update GitLab itself:
    the tracking database is enabled.
 1. [Test](#check-status-after-updating) primary and secondary nodes, and check version in each.
 
+## Upgrading to GitLab 10.5
+
+For Geo Disaster Recovery to work with minimum downtime, your Geo secondary
+should use the same set of secrets as the primary. However, setup instructions
+prior to the 10.5 release only synchronized the `db_key_base` secret.
+
+To rectify this error on existing installations, you should **overwrite** the
+contents of `/etc/gitlab/gitlab-secrets.json` on the secondary node with the
+contents of `/etc/gitlab/gitlab-secrets.json` on the primary node, then run the
+following command on the secondary node:
+
+```bash
+sudo gitlab-ctl reconfigure
+```
+
+If you do not perform this step, you may find that two-factor authentication
+[is broken following DR](faq.md#i-followed-the-disaster-recovery-instructions-and-now-two-factor-auth-is-broken).
+
+## Upgrading to GitLab 10.4
+
+There are no Geo-specific steps to take!
+
 ## Upgrading to GitLab 10.3
 
 ### Support for SSH repository synchronization removed

ee/lib/tasks/geo.rake

@@ -55,11 +55,6 @@ namespace :geo do
       Gitlab::Geo::DatabaseTasks.load_seed
     end
 
-    desc 'Display database encryption key'
-    task show_encryption_key: :environment do
-      puts Rails.application.secrets.db_key_base
-    end
-
     desc 'Refresh Foreign Tables definition in Geo Secondary node'
     task refresh_foreign_tables: [:environment] do
       if Gitlab::Geo::GeoTasks.foreign_server_configured?