Gitaly docs revisions

doc/administration/gitaly/configure_gitaly.md

@@ -350,6 +350,10 @@ leading to `Error creating pipeline` and `Commit not found` errors, or stale dat
 As the final step, you must update Gitaly clients to switch from using local Gitaly service to use
 the Gitaly servers you just configured.
 
+NOTE:
+GitLab requires a `default` repository storage to be configured.
+[Read more about this limitation](#gitlab-requires-a-default-repository-storage).
+
 This can be risky because anything that prevents your Gitaly clients from reaching the Gitaly
 servers causes all Gitaly requests to fail. For example, any sort of network, firewall, or name
 resolution problems.
@@ -489,6 +493,18 @@ gitaly['key_path'] = "/etc/gitlab/ssl/key.pem"
 `path` can be included only for storage shards on the local Gitaly server.
 If it's excluded, default Git storage directory is used for that storage shard.
 
+### GitLab requires a default repository storage
+
+When adding Gitaly servers to an environment, you might want to replace the original `default` Gitaly service. However, you can't
+reconfigure the GitLab application servers to remove the `default` entry from `git_data_dirs` because GitLab requires a
+`git_data_dirs` entry called `default`. [Read more](https://gitlab.com/gitlab-org/gitlab/-/issues/36175) about this limitation.
+
+To work around the limitation:
+
+1. Define an additional storage location on the new Gitaly service and configure the additional storage to be `default`.
+1. In the [Admin Area](../repository_storage_paths.md#configure-where-new-repositories-are-stored), set `default` to a weight of zero
+   to prevent repositories being stored there.
+
 ### Disable Gitaly where not required (optional)
 
 If you run Gitaly [as a remote service](#run-gitaly-on-its-own-server), consider

doc/administration/gitaly/index.md

@@ -189,8 +189,7 @@ The availability objectives for Gitaly clusters assuming a single node failure a
   Writes are replicated asynchronously. Any writes that have not been replicated
   to the newly promoted primary are lost.
 
-  [Strong consistency](#strong-consistency) can be used to avoid loss in some
-  circumstances.
+  [Strong consistency](#strong-consistency) prevents loss in some circumstances.
 
 - **Recovery Time Objective (RTO):** Less than 10 seconds.
   Outages are detected by a health check run by each Praefect node every
@@ -323,19 +322,18 @@ You can [monitor distribution of reads](#monitor-gitaly-cluster) using Prometheu
 > - In GitLab 13.3, disabled unless primary-wins voting strategy is disabled.
 > - From GitLab 13.4, enabled by default.
 > - From GitLab 13.5, you must use Git v2.28.0 or higher on Gitaly nodes to enable strong consistency.
-> - From GitLab 13.6, primary-wins voting strategy and `gitaly_reference_transactions_primary_wins` feature flag were removed from the source code.
+> - From GitLab 13.6, primary-wins voting strategy and the `gitaly_reference_transactions_primary_wins` feature flag was removed.
+> - From GitLab 14.0, [Gitaly Cluster only supports strong consistency](https://gitlab.com/gitlab-org/gitaly/-/merge_requests/3575), and the `gitaly_reference_transactions` feature flag was removed.
 
-By default, Gitaly Cluster guarantees eventual consistency by replicating all writes to secondary
-Gitaly nodes after the write to the primary Gitaly node has happened.
+Gitaly Cluster writes changes synchronously to all healthy, up to date replicas.
+If a replica is outdated or unhealthy at the time of the transaction, the write is asynchronously replicated to it.
 
-Praefect can instead provide strong consistency by creating a transaction and writing changes to all
-Gitaly nodes at once.
+In GitLab 13.12 and earlier, if Gitaly Cluster wasn't configured to use strong consistency (or didn't support it yet), Gitaly Cluster guaranteed eventual consistency by replicating all writes to secondary Gitaly nodes after the write to the primary Gitaly node has occurred.
 
-If enabled, transactions are only available for a subset of RPCs. For more information, see the
+A subset of operations still use replication jobs (instead of a strong consistency transaction).
+For more information, see the
 [strong consistency epic](https://gitlab.com/groups/gitlab-org/-/epics/1189).
 
-For configuration information, see [Configure strong consistency](praefect.md#configure-strong-consistency).
-
 #### Replication factor
 
 Replication factor is the number of copies Gitaly Cluster maintains of a given repository. A higher
@@ -368,6 +366,10 @@ WARNING:
 Some [known database inconsistency issues](#known-issues) exist in Gitaly Cluster. We recommend you
 remain on your current service for now.
 
+NOTE:
+GitLab requires a `default` repository storage to be configured.
+[Read more about this limitation](configure_gitaly.md#gitlab-requires-a-default-repository-storage).
+
 ### Migrate off Gitaly Cluster
 
 If you have repositories stored on a Gitaly Cluster, but you'd like to migrate

doc/administration/gitaly/praefect.md

@@ -215,6 +215,38 @@ The database used by Praefect is now configured.
 If you see Praefect database errors after configuring PostgreSQL, see
 [troubleshooting steps](troubleshooting.md#relation-does-not-exist-errors).
 
+#### Reads distribution caching
+
+Praefect performance can be improved by additionally configuring the `database_direct`
+settings:
+
+```ruby
+praefect['database_direct_host'] = POSTGRESQL_HOST
+praefect['database_direct_port'] = 5432
+
+# Use the following to override parameters of direct database connection.
+# Comment out where the parameters are the same for both connections.
+
+praefect['database_direct_user'] = 'praefect'
+praefect['database_direct_password'] = PRAEFECT_SQL_PASSWORD
+praefect['database_direct_dbname'] = 'praefect_production'
+#praefect['database_direct_sslmode'] = '...'
+#praefect['database_direct_sslcert'] = '...'
+#praefect['database_direct_sslkey'] = '...'
+#praefect['database_direct_sslrootcert'] = '...'
+```
+
+Once configured, this connection is automatically used for the
+[SQL LISTEN](https://www.postgresql.org/docs/11/sql-listen.html) feature and
+allows Praefect to receive notifications from PostgreSQL for cache invalidation.
+
+Verify this feature is working by looking for the following log entry in the Praefect
+log:
+
+```plaintext
+reads distribution caching is enabled by configuration
+```
+
 #### Use PgBouncer
 
 To reduce PostgreSQL resource consumption, we recommend setting up and configuring
@@ -1073,30 +1105,16 @@ To get started quickly:
 Congratulations! You've configured an observable fault-tolerant Praefect
 cluster.
 
-## Configure strong consistency
-
-To enable [strong consistency](index.md#strong-consistency):
+## Strong consistency
 
-- In GitLab 13.5, you must use Git v2.28.0 or higher on Gitaly nodes to enable strong consistency.
-- In GitLab 13.4 and later, the strong consistency voting strategy has been improved and enabled by default.
-  Instead of requiring all nodes to agree, only the primary and half of the secondaries need to agree.
-- In GitLab 13.3, reference transactions are enabled by default with a primary-wins strategy.
-  This strategy causes all transactions to succeed for the primary and thus does not ensure strong consistency.
-  To enable strong consistency, disable the `:gitaly_reference_transactions_primary_wins` feature flag.
-- In GitLab 13.2, enable the `:gitaly_reference_transactions` feature flag.
-- In GitLab 13.1, enable the `:gitaly_reference_transactions` and `:gitaly_hooks_rpc`
-  feature flags.
+[Strong consistency](index.md#strong-consistency) is the default from GitLab 14.0.
+For configuration information on earlier versions, refer to documentation:
 
-Changing feature flags requires [access to the Rails console](../feature_flags.md#start-the-gitlab-rails-console).
-In the Rails console, enable or disable the flags as required. For example:
-
-```ruby
-Feature.enable(:gitaly_reference_transactions)
-Feature.disable(:gitaly_reference_transactions_primary_wins)
-```
+- On your GitLab instance at `/help`.
+- The [13.12 documentation](https://docs.gitlab.com/13.12/ee/administration/gitaly/praefect.html#strong-consistency).
 
-For information on monitoring strong consistency, see the
-[relevant documentation](index.md#monitor-gitaly-cluster).
+For information on monitoring strong consistency, see the Gitaly Cluster
+[Prometheus metrics documentation](index.md#monitor-gitaly-cluster).
 
 ## Configure replication factor
 

doc/administration/gitaly/troubleshooting.md

@@ -328,6 +328,27 @@ experiencing [clock drift](https://en.wikipedia.org/wiki/Clock_drift).
 Please ensure that the GitLab and Gitaly nodes are synchronized and use an NTP time
 server to keep them synchronized if possible.
 
+### Health check warnings
+
+The following warning in `/var/log/gitlab/praefect/current` can be ignored.
+
+```plaintext
+"error":"full method name not found: /grpc.health.v1.Health/Check",
+"msg":"error when looking up method info"
+```
+
+### File not found errors
+
+The following errors in `/var/log/gitlab/gitaly/current` can be ignored.
+They are caused by the GitLab Rails application checking for specific files
+that do not exist in a repository.
+
+```plaintext
+"error":"not found: .gitlab/route-map.yml"
+"error":"not found: Dockerfile"
+"error":"not found: .gitlab-ci.yml"
+```
+
 ## Troubleshoot Praefect (Gitaly Cluster)
 
 The following sections provide possible solutions to Gitaly Cluster errors.
@@ -409,3 +430,21 @@ This indicates that the virtual storage name used in the
 [`git_data_dirs` setting](praefect.md#gitaly) for GitLab.
 
 Resolve this by matching the virtual storage names used in Praefect and GitLab configuration.
+
+### Gitaly Cluster performance issues on cloud platforms
+
+Praefect does not require a lot of CPU or memory, and can run on small virtual machines.
+Cloud services may place other limits on the resources that small VMs can use, such as
+disk IO and network traffic.
+
+Praefect nodes generate a lot of network traffic. The following symptoms can be observed
+if their network bandwidth has been throttled by the cloud service.
+
+- Poor performance of Git operations
+- High network latency
+- High memory use by Praefect
+
+Possible solutions:
+
+- Provision larger VMs to gain access to larger network traffic allowances.
+- Use your cloud service's monitoring and logging to check that the Praefect nodes are not exhausting their traffic allowances.

doc/administration/operations/moving_repositories.md

@@ -42,7 +42,12 @@ To move repositories into a [Gitaly Cluster](../gitaly/index.md#gitaly-cluster)
 WARNING:
 Repositories can be **permanently deleted** by a call to `/projects/:project_id/repository_storage_moves`
 that attempts to move a project already stored in a Gitaly Cluster back into that cluster.
-See [this issue for more details](https://gitlab.com/gitlab-org/gitaly/-/issues/3752).
+See [this issue for more details](https://gitlab.com/gitlab-org/gitaly/-/issues/3752). This was fixed in
+GitLab 14.3.0 and backported to
+[14.2.4](https://about.gitlab.com/releases/2021/09/17/gitlab-14-2-4-released/),
+[14.1.6](https://about.gitlab.com/releases/2021/09/27/gitlab-14-1-6-released/),
+[14.0.11](https://about.gitlab.com/releases/2021/09/27/gitlab-14-0-11-released/), and
+[13.12.12](https://about.gitlab.com/releases/2021/09/22/gitlab-13-12-12-released/).
 
 Each repository is made read-only for the duration of the move. The repository is not writable
 until the move has completed.