Merge branch '4398-reuse-primary-host-key-on-secondaries' into 'master'

Resolve "Reuse primary host key on secondaries"

Closes #4398

See merge request gitlab-org/gitlab-ee!4198

changelogs/unreleased-ee/4398-reuse-primary-host-key-on-secondaries.yml

+---
+title: Update Geo documentation to reuse the primary node SSH host key on secondary
+  node
+merge_request: 4198
+author:
+type: fixed

doc/gitlab-geo/configuration.md

@@ -85,7 +85,62 @@ Make sure the secondary instance is
 running and accessible. You can login to the secondary node
 with the same credentials as used in the primary.
 
-### Step 2. (Optional) Enabling hashed storage (from GitLab 10.0)
+### Step 2. Manually replicate primary SSH host keys
+
+GitLab integrates with the system-installed SSH daemon, designating a user
+(typically named git) through which all access requests are handled.
+
+In a [Disaster Recovery](disaster-recovery.md) situation, GitLab system
+administrators will promote a secondary Geo replica to a primary and they can
+update the DNS records for the primary domain to point to the secondary to prevent
+the need to update all references to the primary domain to the secondary domain,
+like changing Git remotes and API URLs.
+
+This will cause all SSH requests to the newly promoted primary node from
+failing due to SSH host key mismatch. To prevent this, the primary SSH host
+keys must be manually replicated to the secondary node.
+
+1. SSH into the **secondary** node and login as the `root` user:
+
+    ```
+    sudo -i
+    ```
+
+1. Make a backup of any existing SSH host keys:
+
+    ```bash
+    find /etc/ssh -iname ssh_host_* -exec mv {} {}.backup.`date +%F` \;
+    ```
+
+1. SSH into the **primary** node, and execute the command below:
+
+    ```bash
+    sudo find /etc/ssh -iname ssh_host_* -not -iname '*.pub'
+    ```
+
+1. For each file in that list copy the file from the primary node to
+   the **same** location on your **secondary** node.
+
+1. On your **secondary** node, ensure the file permissions are correct:
+
+    ```bash
+    chown root:root /etc/ssh/ssh_host_*
+    chmod 0600 /etc/ssh/ssh_host_*
+    ```
+
+1. Regenerate the public keys from the private keys:
+
+    ```bash
+    find /etc/ssh -iname ssh_host_* -not -iname '*.backup*' -exec sh -c 'ssh-keygen -y -f "{}" > "{}.pub"' \;
+    ```
+
+1. Restart sshd:
+
+    ```bash
+    service ssh restart
+    ```
+
+### Step 3. (Optional) Enabling hashed storage (from GitLab 10.0)
 
 >**Warning**
 Hashed storage is in **Alpha**. It is considered experimental and not
@@ -102,7 +157,7 @@ renames no longer require synchronization between nodes.
 
     ![](img/hashed-storage.png)
 
-### Step 3. (Optional) Configuring the secondary to trust the primary
+### Step 4. (Optional) Configuring the secondary to trust the primary
 
 You can safely skip this step if your primary uses a CA-issued HTTPS certificate.
 
@@ -112,14 +167,14 @@ certificate from the primary and follow
 [these instructions](https://docs.gitlab.com/omnibus/settings/ssl.html)
 on the secondary.
 
-### Step 4. Enable Git access over HTTP/HTTPS
+### Step 5. Enable Git access over HTTP/HTTPS
 
 GitLab Geo synchronizes repositories over HTTP/HTTPS, and therefore requires this clone
 method to be enabled. Navigate to **Admin Area ➔ Settings**
 (`/admin/application_settings`) on the primary node, and set
 `Enabled Git access protocols` to `Both SSH and HTTP(S)` or `Only HTTP(S)`.
 
-### Step 5. Verify proper functioning of the secondary node
+### Step 6. Verify proper functioning of the secondary node
 
 Congratulations! Your secondary geo node is now configured!
 

doc/gitlab-geo/configuration_source.md

@@ -77,13 +77,6 @@ be manually replicated to the secondary.
     service gitlab restart
     ```
 
-The secondary will start automatically replicating missing data from the
-primary in a process known as backfill. Meanwhile, the primary node will start
-to notify changes to the secondary, which will act on those notifications
-immediately. Make sure the secondary instance is running and accessible.
-
-### Step 2. (Optional) Enabling hashed storage
-
 Once restarted, the secondary will automatically start replicating missing data
 from the primary in a process known as backfill. Meanwhile, the primary node
 will start to notify the secondary of any changes, so that the secondary can
@@ -92,11 +85,15 @@ act on those notifications immediately.
 Make sure the secondary instance is running and accessible. You can login to
 the secondary node with the same credentials as used in the primary.
 
-### Step 2. (Optional) Enabling hashed storage (from GitLab 10.0)
+### Step 2. Manually replicate primary SSH host keys
+
+Read [Manually replicate primary SSH host keys](configuration.md#step-2-manually-replicate-primary-ssh-host-keys)
+
+### Step 3. (Optional) Enabling hashed storage (from GitLab 10.0)
 
-Read [Enabling Hashed Storage](configuration.md#step-2-optional-enabling-hashed-storage-from-gitlab-10-0)
+Read [Enabling Hashed Storage](configuration.md#step-3-optional-enabling-hashed-storage-from-gitlab-10-0)
 
-### Step 3. (Optional) Configuring the secondary to trust the primary
+### Step 4. (Optional) Configuring the secondary to trust the primary
 
 You can safely skip this step if your primary uses a CA-issued HTTPS certificate.
 
@@ -112,16 +109,16 @@ cp primary.geo.example.com.crt /usr/local/share/ca-certificates
 update-ca-certificates
 ```
 
-### Step 4. Enable Git access over HTTP/HTTPS
+### Step 5. Enable Git access over HTTP/HTTPS
 
 GitLab Geo synchronizes repositories over HTTP/HTTPS, and therefore requires this clone
 method to be enabled. Navigate to **Admin Area ➔ Settings**
 (`/admin/application_settings`) on the primary node, and set
 `Enabled Git access protocols` to `Both SSH and HTTP(S)` or `Only HTTP(S)`.
 
-### Step 5. Verify proper functioning of the secondary node
+### Step 6. Verify proper functioning of the secondary node
 
-Read [Verify proper functioning of the secondary node](configuration.md#step-5-verify-proper-functioning-of-the-secondary-node).
+Read [Verify proper functioning of the secondary node](configuration.md#step-6-verify-proper-functioning-of-the-secondary-node).
 
 
 ## Selective replication

doc/gitlab-geo/disaster-recovery.md

@@ -73,7 +73,7 @@ secondary domain, like changing Git remotes and API URLs.
 
 1. SSH in to your **secondary** and login as root:
 
-    ```
+    ```bash
     sudo -i
     ```
 
@@ -82,20 +82,20 @@ secondary domain, like changing Git remotes and API URLs.
     After updating the primary domain's DNS records to point to the secondary,
     edit `/etc/gitlab/gitlab.rb` on the the secondary to reflect the new URL:
 
-    ```
+    ```ruby
     # Change the existing external_url configuration
     external_url 'https://gitlab.example.com'
     ```
 
 1. Reconfigure the secondary node for the change to take effect:
 
-    ```
+    ```bash
     gitlab-ctl reconfigure
     ```
 
 1. Execute the command below to update the newly promoted primary node URL:
 
-    ```
+    ```bash
     gitlab-rake geo:update_primary_node_url
     ```
 

doc/gitlab-geo/updating_the_geo_nodes.md

@@ -32,6 +32,11 @@ sudo gitlab-ctl reconfigure
 If you do not perform this step, you may find that two-factor authentication
 [is broken following DR](faq.md#i-followed-the-disaster-recovery-instructions-and-now-two-factor-auth-is-broken).
 
+To prevent SSH requests to the newly promoted primary node from failing
+due to SSH host key mismatch when updating the primary domain's DNS record
+you should perform the step to [Manually replicate primary SSH host keys](configuration.md#step-2-manually-replicate-primary-ssh-host-keys) in each
+secondary node.
+
 ## Upgrading to GitLab 10.4
 
 There are no Geo-specific steps to take!
@@ -44,7 +49,7 @@ In GitLab 10.2, synchronizing secondaries over SSH was deprecated. In 10.3,
 support is removed entirely. All installations will switch to the HTTP/HTTPS
 cloning method instead. Before upgrading, ensure that all your Geo nodes are
 configured to use this method and that it works for your installation. In
-particular, ensure that [Git access over HTTP/HTTPS is enabled](configuration.md#step-4-enable-git-access-over-http-https).
+particular, ensure that [Git access over HTTP/HTTPS is enabled](configuration.md#step-5-enable-git-access-over-http-https).
 
 Synchronizing repositories over the public Internet using HTTP is insecure, so
 you should ensure that you have HTTPS configured before upgrading. Note that
@@ -52,7 +57,7 @@ file synchronization is **also** insecure in these cases!
 
 ## Upgrading to GitLab 10.2
 
-### Secure PostgreSQL replication 
+### Secure PostgreSQL replication
 
 Support for TLS-secured PostgreSQL replication has been added. If you are
 currently using PostgreSQL replication across the open internet without an