Merge branch '4395-geo-disaster-recovery' into 'master'

Resolve "`promote-to-primary-node` silently fails reconfiguration due to postgresql service not listening"

Closes #4395

See merge request gitlab-org/gitlab-ee!4097

changelogs/unreleased-ee/4395-geo-disaster-recovery.yml

+---
+title: Geo - Add a rake task to update Geo primary node URL
+merge_request: 4097
+author:
+type: fixed

doc/gitlab-geo/disaster-recovery.md

@@ -9,7 +9,7 @@ fail-over with minimal effort, in a disaster situation.
 
 See [current limitations](README.md#current-limitations) for more information.
 
-## Promoting a secondary geo replica
+### Step 1. Promoting a secondary geo replica
 
 > **Warning:** Disaster Recovery does not yet support systems with multiple
 > secondary geo replicas (e.g. one primary and two or more secondaries).
@@ -42,20 +42,6 @@ It does not enable GitLab Geo on the newly promoted primary.
     sudo -i
     ```
 
-1. Optional: Update the primary domain's DNS record.
-    
-    Updating the DNS records for the primary domain to point to the secondary
-    will prevent the need to update all references to the primary domain to the
-    secondary domain, like changing Git remotes and API URLs.
-
-    After updating the primary domain's DNS records to point to the secondary,
-    edit `/etc/gitlab/gitlab.rb` on the the secondary to reflect the new URL:
-
-    ```
-    # Change the existing external_url configuration
-    external_url 'https://gitlab.example.com'
-    ```
-
 1. Edit `/etc/gitlab/gitlab.rb` to reflect its new status as primary.
 
     Remove the following line:
@@ -79,10 +65,49 @@ It does not enable GitLab Geo on the newly promoted primary.
    previously for the secondary.
 1. Success! The secondary has now been promoted to primary.
 
+### Step 2. (Optional) Updating the primary domain's DNS record
+
+Updating the DNS records for the primary domain to point to the secondary
+will prevent the need to update all references to the primary domain to the
+secondary domain, like changing Git remotes and API URLs.
+
+1. SSH in to your **secondary** and login as root:
+
+    ```
+    sudo -i
+    ```
+
+1. Update the primary domain's DNS record.
+
+    After updating the primary domain's DNS records to point to the secondary,
+    edit `/etc/gitlab/gitlab.rb` on the the secondary to reflect the new URL:
+
+    ```
+    # Change the existing external_url configuration
+    external_url 'https://gitlab.example.com'
+    ```
+
+1. Reconfigure the secondary node for the change to take effect:
+
+    ```
+    gitlab-ctl reconfigure
+    ```
+
+1. Execute the command below to update the newly promoted primary node URL:
+
+    ```
+    gitlab-rake geo:update_primary_node_url
+    ```
+
+    This command will use the changed `external_url` configuration defined
+    in `/etc/gitlab/gitlab.rb`.
+
+1. Verify you can connect to the newly promoted primary using the primary URL.
+
     If you updated the DNS records for the primary domain, these changes may
     not have yet propagated depending on the previous DNS records TTL.
 
-## Add secondary geo replicas to a promoted primary
+### Step 3. (Optional) Add secondary geo replicas to a promoted primary
 
 Promoting a secondary to primary using the process above does not enable
 GitLab Geo on the new primary.

ee/lib/gitlab/geo/geo_tasks.rb

@@ -15,6 +15,26 @@ module Gitlab
         end
       end
 
+      def update_primary_geo_node_url
+        node = Gitlab::Geo.primary_node
+
+        unless node.present?
+          $stdout.puts 'This is not a primary node'.color(:red)
+          exit 1
+        end
+
+        $stdout.puts "Updating primary Geo node with URL #{node.url} ..."
+
+        if node.update(url: GeoNode.current_node_url)
+          puts "#{node.url} is now the primary Geo node URL".color(:green)
+          $stdout.puts "#{node.url} is now the primary Geo node URL".color(:green)
+        else
+          puts "Error saving Geo node:\n#{node.errors.full_messages.join("\n")}".color(:red)
+          $stdout.puts "Error saving Geo node:\n#{node.errors.full_messages.join("\n")}".color(:red)
+          exit 1
+        end
+      end
+
       def refresh_foreign_tables!
         sql = <<~SQL
             DROP SCHEMA IF EXISTS gitlab_secondary CASCADE;

lib/tasks/geo.rake

@@ -194,4 +194,11 @@ namespace :geo do
       current_node.update!(primary: true)
     end
   end
+
+  desc 'Update Geo primary node URL'
+  task update_primary_node_url: :environment do
+    abort GEO_LICENSE_ERROR_TEXT unless Gitlab::Geo.license_allows?
+
+    Gitlab::Geo::GeoTasks.update_primary_geo_node_url
+  end
 end

spec/tasks/geo_rake_spec.rb

 require 'rake_helper'
 
 describe 'geo rake tasks' do
+  include ::EE::GeoHelpers
+
   before do
     Rake.application.rake_require 'tasks/geo'
+    stub_licensed_features(geo: true)
   end
 
   describe 'set_primary_node task' do
     before do
-      expect(Gitlab::Geo).to receive(:license_allows?).and_return(true)
       stub_config_setting(protocol: 'https')
     end
 
@@ -26,14 +28,10 @@ describe 'geo rake tasks' do
   end
 
   describe 'set_secondary_as_primary task' do
-    include ::EE::GeoHelpers
-
     let!(:current_node) { create(:geo_node) }
     let!(:primary_node) { create(:geo_node, :primary) }
 
     before do
-      expect(Gitlab::Geo).to receive(:license_allows?).and_return(true)
-
       stub_current_geo_node(current_node)
     end
 
@@ -44,4 +42,19 @@ describe 'geo rake tasks' do
       expect(GeoNode.count).to eq(1)
     end
   end
+
+  describe 'update_primary_node_url task' do
+    let(:primary_node) { create(:geo_node, :primary, url: 'https://secondary.geo.example.com') }
+
+    before do
+      allow(GeoNode).to receive(:current_node_url).and_return('https://primary.geo.example.com')
+      stub_current_geo_node(primary_node)
+    end
+
+    it 'updates Geo primary node URL' do
+      run_rake_task('geo:update_primary_node_url')
+
+      expect(primary_node.reload.url).to eq 'https://primary.geo.example.com/'
+    end
+  end
 end