Merge branch 'eread/give-praefect-auto-migration-guidance' into 'master'

Add Praefect auto migration guidance

See merge request gitlab-org/gitlab!62973

doc/administration/gitaly/configure_gitaly.md

@@ -247,7 +247,6 @@ disable enforcement. For more information, see the documentation on configuring
    # node_exporter['enable'] = false
 
    # Prevent database connections during 'gitlab-ctl reconfigure'
-   gitlab_rails['rake_cache_clear'] = false
    gitlab_rails['auto_migrate'] = false
 
    # Configure the gitlab-shell API callback URL. Without this, `git push` will

doc/administration/gitaly/praefect.md

@@ -9,13 +9,21 @@ type: reference
 
 Configure Gitaly Cluster using either:
 
-- The Gitaly Cluster configuration instructions available as part of
-  [reference architectures](../reference_architectures/index.md) for installations for more than
-  2000 users.
-- The advanced configuration instructions that follow on this page.
+- Gitaly Cluster configuration instructions available as part of
+  [reference architectures](../reference_architectures/index.md) for installations of up to:
+  - [3000 users](../reference_architectures/3k_users.md#configure-gitaly-cluster).
+  - [5000 users](../reference_architectures/5k_users.md#configure-gitaly-cluster).
+  - [10,000 users](../reference_architectures/10k_users.md#configure-gitaly-cluster).
+  - [25,000 users](../reference_architectures/25k_users.md#configure-gitaly-cluster).
+  - [50,000 users](../reference_architectures/50k_users.md#configure-gitaly-cluster).
+- The custom configuration instructions that follow on this page.
 
 Smaller GitLab installations may need only [Gitaly itself](index.md).
 
+NOTE:
+Upgrade instructions for Omnibus GitLab installations
+[are available](https://docs.gitlab.com/omnibus/update/#gitaly-cluster).
+
 ## Requirements for configuring a Gitaly Cluster
 
 The minimum recommended configuration for a Gitaly Cluster requires:
@@ -226,8 +234,10 @@ PostgreSQL instances. Otherwise you should change the configuration parameter
 
 > [Introduced](https://gitlab.com/gitlab-org/gitaly/-/issues/2634) in GitLab 13.4, Praefect nodes can no longer be designated as `primary`.
 
-NOTE:
-If there are multiple Praefect nodes, complete these steps for **each** node.
+If there are multiple Praefect nodes:
+
+- Complete the following steps for **each** node.
+- Designate one node as the "deploy node", and configure it first.
 
 To complete this section you need a [configured PostgreSQL server](#postgresql), including:
 
@@ -262,8 +272,8 @@ application server, or a Gitaly node.
    praefect['enable'] = true
 
    # Prevent database connections during 'gitlab-ctl reconfigure'
-   gitlab_rails['rake_cache_clear'] = false
    gitlab_rails['auto_migrate'] = false
+   praefect['auto_migrate'] = false
    ```
 
 1. Configure **Praefect** to listen on network interfaces by editing
@@ -377,6 +387,30 @@ application server, or a Gitaly node.
 
 1. [Introduced](https://gitlab.com/groups/gitlab-org/-/epics/2013) in GitLab 13.1 and later, enable [distribution of reads](#distributed-reads).
 
+1. Save the changes to `/etc/gitlab/gitlab.rb` and [reconfigure
+   Praefect](../restart_gitlab.md#omnibus-gitlab-reconfigure):
+
+   ```shell
+   gitlab-ctl reconfigure
+   ```
+
+1. For:
+
+   - The "deploy node":
+     1. Enable Praefect auto-migration again by setting `praefect['auto_migrate'] = true` in
+        `/etc/gitlab/gitlab.rb`.
+     1. To ensure database migrations are only run during reconfigure and not automatically on
+        upgrade, run:
+
+        ```shell
+        sudo touch /etc/gitlab/skip-auto-reconfigure
+        ```
+
+   - The other nodes, you can leave the settings as they are. Though
+     `/etc/gitlab/skip-auto-reconfigure` isn't required, you may want to set it to prevent GitLab
+     running reconfigure automatically when running commands such as `apt-get update`. This way any
+     additional configuration changes can be done and then reconfigure can be run manually.
+
 1. Save the changes to `/etc/gitlab/gitlab.rb` and [reconfigure
    Praefect](../restart_gitlab.md#omnibus-gitlab-reconfigure):
 
@@ -602,7 +636,6 @@ documentation](configure_gitaly.md#configure-gitaly-servers).
    prometheus['enable'] = true
 
    # Prevent database connections during 'gitlab-ctl reconfigure'
-   gitlab_rails['rake_cache_clear'] = false
    gitlab_rails['auto_migrate'] = false
    ```
 
@@ -699,7 +732,7 @@ documentation](configure_gitaly.md#configure-gitaly-servers).
 
 **The steps above must be completed for each Gitaly node!**
 
-After all Gitaly nodes are configured, you can run the Praefect connection
+After all Gitaly nodes are configured, run the Praefect connection
 checker to verify Praefect can connect to all Gitaly servers in the Praefect
 configuration.