Update praefect dataloss documentation with new output format

Praefect has added support for variable replication factor and
repository specific primaries. This commit updates the dataloss
subcommand documentation with the new output format and explains
the various fields outputted with each repository.

doc/administration/gitaly/praefect.md

@@ -1054,6 +1054,55 @@ To monitor strong consistency, you can use the following Prometheus metrics:
 - `gitaly_hook_transaction_voting_delay_seconds`: Client-side delay introduced
   by waiting for the transaction to be committed.
 
+## Replication factor
+
+Replication factor is the number of copies Praefect maintains of a given repository. A higher
+replication factor offers better redundancy and distribution of read workload, but also results
+in a higher storage cost. By default, Praefect replicates repositories to every storage in a
+virtual storage.
+
+### Variable replication factor
+
+WARNING:
+The feature is not production ready yet. After you set a replication factor, you can't unset it
+without manually modifying database state. Variable replication factor requires you to enable
+repository-specific primaries by configuring the `per_repository` primary election strategy. The election
+strategy is not production ready yet.
+
+Praefect supports configuring a replication factor on a per-repository basis, by assigning
+specific storage nodes to host a repository.
+
+[In an upcoming release](https://gitlab.com/gitlab-org/gitaly/-/issues/3362), we intend to
+support configuring a default replication factor for a virtual storage. The default replication factor
+is applied to every newly-created repository.
+
+Prafect does not store the actual replication factor, but assigns enough storages to host the repository
+so the desired replication factor is met. If a storage node is later removed from the virtual storage,
+the replication factor of repositories assigned to the storage is decreased accordingly.
+
+The only way to configure a repository's replication factor is the `set-replication-factor`
+sub-command. `set-replication-factor` automatically assigns or unassigns random storage nodes as necessary to
+reach the desired replication factor. The repository's primary node is always assigned
+first and is never unassigned.
+
+```shell
+sudo /opt/gitlab/embedded/bin/praefect -config /var/opt/gitlab/praefect/config.toml set-replication-factor -virtual-storage <virtual-storage> -repository <relative-path> -replication-factor <replication-factor>
+```
+
+- `-virtual-storage` is the virtual storage the repository is located in.
+- `-repository` is the repository's relative path in the storage.
+- `-replication-factor` is the desired replication factor of the repository. The minimum value is
+  `1`, as the primary needs a copy of the repository. The maximum replication factor is the number of
+  storages in the virtual storage.
+
+On success, the assigned host storages are printed. For example:
+
+```shell
+$ sudo /opt/gitlab/embedded/bin/praefect -config /var/opt/gitlab/praefect/config.toml set-replication-factor -virtual-storage default -repository @hashed/3f/db/3fdba35f04dc8c462986c992bcf875546257113072a909c162f7e470e581e278.git -replication-factor 2
+
+current assignments: gitaly-1, gitaly-2
+```
+
 ## Automatic failover and leader election
 
 Praefect regularly checks the health of each backend Gitaly node. This
@@ -1111,15 +1160,14 @@ useful for identifying potential data loss after a failover. The following param
 available:
 
 - `-virtual-storage` that specifies which virtual storage to check. The default behavior is to
-  display outdated replicas of read-only repositories as they generally require administrator
-  action.
+  display outdated replicas of read-only repositories as they might require administrator action.
 - In GitLab 13.3 and later, `-partially-replicated` that specifies whether to display a list of
   [outdated replicas of writable repositories](#outdated-replicas-of-writable-repositories).
 
 NOTE:
 `dataloss` is still in beta and the output format is subject to change.
 
-To check for outdated replicas of read-only repositories, run:
+To check for repositories with outdated primaries, run:
 
 ```shell
 sudo /opt/gitlab/embedded/bin/praefect -config /var/opt/gitlab/praefect/config.toml dataloss [-virtual-storage <virtual-storage>]
@@ -1131,24 +1179,45 @@ Every configured virtual storage is checked if none is specified:
 sudo /opt/gitlab/embedded/bin/praefect -config /var/opt/gitlab/praefect/config.toml dataloss
 ```
 
-The number of potentially unapplied changes to repositories is listed for each replica. Listed
-repositories might have the latest changes but it is not guaranteed. Only outdated replicas of
-read-only repositories are listed by default. For example:
+Repositories which have assigned storage nodes that contain an outdated copy of the repository are listed
+in the output. A number of useful information is printed for each repository:
+
+- A repository's relative path to the storage directory identifies each repository and groups the related
+  information.
+- The repository's current status is printed in parentheses next to the disk path. If the repository's primary
+  is outdated, the repository is in `read-only` mode and can't accept writes. Otherwise, the mode is `writable`.
+- The primary field lists the repository's current primary. If the repository has no primary, the field shows
+  `No Primary`.
+- The In-Sync Storages lists replicas which have replicated the latest successful write and all writes
+  preceding it.
+- The Outdated Storages lists replicas which contain an outdated copy of the repository. Replicas which have no copy
+  of the repository but should contain it are also listed here. The maximum number of changes the replica is missing
+  is listed next to replica. It's important to notice that the outdated replicas may be fully up to date or contain
+  later changes but Praefect can't guarantee it.
+
+Whether a replica is assigned to host the repository is listed with each replica's status. `assigned host` is printed
+next to replicas which are assigned to store the repository. The text is omitted if the replica contains a copy of
+the repository but is not assigned to store the repository. Such replicas won't be kept in-sync by Praefect but may
+act as replication sources to bring assigned replicas up to date.
+
+Example output:
 
 ```shell
 Virtual storage: default
-  Primary: gitaly-3
   Outdated repositories:
-    @hashed/2c/62/2c624232cdd221771294dfbb310aca000a0df6ac8b66b696d90ef06fdefb64a3.git (read-only):
-      gitaly-2 is behind by 1 change or less
-      gitaly-3 is behind by 2 changes or less
+    @hashed/3f/db/3fdba35f04dc8c462986c992bcf875546257113072a909c162f7e470e581e278.git (read-only):
+      Primary: gitaly-1
+      In-Sync Storages:
+        gitaly-2, assigned host
+      Outdated Storages:
+        gitaly-1 is behind by 3 changes or less, assigned host
+        gitaly-3 is behind by 3 changes or less
 ```
 
 A confirmation is printed out when every repository is writable. For example:
 
 ```shell
 Virtual storage: default
-  Primary: gitaly-1
   All repositories are writable!
 ```
 
@@ -1156,8 +1225,8 @@ Virtual storage: default
 
 > [Introduced](https://gitlab.com/gitlab-org/gitaly/-/issues/3019) in GitLab 13.3.
 
-To also list information for outdated replicas of writable repositories, use the
-`-partially-replicated` parameter.
+To also list information of repositories whose primary is up to date but one or more assigned
+replicas are outdated, use the `-partially-replicated` flag.
 
 A repository is writable if the primary has the latest changes. Secondaries might be temporarily
 outdated while they are waiting to replicate the latest changes.
@@ -1170,21 +1239,23 @@ Example output:
 
 ```shell
 Virtual storage: default
-  Primary: gitaly-3
   Outdated repositories:
-    @hashed/2c/62/2c624232cdd221771294dfbb310aca000a0df6ac8b66b696d90ef06fdefb64a3.git (read-only):
-      gitaly-2 is behind by 1 change or less
-      gitaly-3 is behind by 2 changes or less
-    @hashed/4b/22/4b227777d4dd1fc61c6f884f48641d02b4d121d3fd328cb08b5531fcacdabf8a.git (writable):
-      gitaly-2 is behind by 1 change or less
+    @hashed/3f/db/3fdba35f04dc8c462986c992bcf875546257113072a909c162f7e470e581e278.git (writable):
+      Primary: gitaly-1
+      In-Sync Storages:
+        gitaly-1, assigned host
+      Outdated Storages:
+        gitaly-2 is behind by 3 changes or less, assigned host
+        gitaly-3 is behind by 3 changes or less
 ```
 
-With the `-partially-replicated` flag set, a confirmation is printed out if every replica is fully up to date.
+With the `-partially-replicated` flag set, a confirmation is printed out if every assigned replica is fully up to
+date.
+
 For example:
 
 ```shell
 Virtual storage: default
-  Primary: gitaly-1
   All repositories are up to date!
 ```