Docs - Flesh out Praefect documentation

doc/administration/gitaly/praefect.md

@@ -26,33 +26,42 @@ For this document, the following network topology is assumed:
 graph TB
   GitLab --> Gitaly;
   GitLab --> Praefect;
-  Praefect --> Preafect-Git-1;
-  Praefect --> Preafect-Git-2;
-  Praefect --> Preafect-Git-3;
+  Praefect --> Praefect-Gitaly-1;
+  Praefect --> Praefect-Gitaly-2;
+  Praefect --> Praefect-Gitaly-3;
 ```
 
 Where `GitLab` is the collection of clients that can request Git operations.
-`Gitaly` is a Gitaly server before using Praefect. The Praefect node has two
+`Gitaly` is a Gitaly server before using Praefect. The Praefect node has three
 storage nodes attached. Praefect itself doesn't store data, but connects to
-three Gitaly nodes, `Praefect-Git-1`,  `Praefect-Git-2`, and `Praefect-Git-3`.
-There should be no knowledge other than with Praefect about the existence of
-the `Praefect-Git-X` nodes.
+three Gitaly nodes, `Praefect-Gitaly-1`,  `Praefect-Gitaly-2`, and `Praefect-Gitaly-3`.
 
-### Setup
+Praefect may be enabled on its own node or can be run on the GitLab server.
+In the example below we will use a separate server, but the optimal configuration
+for Praefect is still being determined.
 
-In this setup guide, the Gitaly node will be added first, then Praefect, and
-lastly we update the GitLab configuration.
+Praefect will handle all Gitaly RPC requests to its child nodes. However, the child nodes
+will still need to communicate with the GitLab server via its internal API for authentication
+purposes.
 
-#### Gitaly
+### Setup
 
-In their own machine, configure the Gitaly server as described in the
-[gitaly documentation](index.md#3-gitaly-server-configuration).
+In this setup guide we will start by configuring Praefect, then its child
+Gitaly nodes, and lastly the GitLab server configuration.
 
 #### Praefect
 
-Next, Praefect has to be enabled on its own node.
+On the Praefect node we disable all other services, including Gitaly. We list each
+Gitaly node that will be connected to Praefect under `praefect['storage_nodes']`.
 
-##### Disable other services
+In the example below, the Gitaly nodes are named `praefect-gitaly-N`. Note that one
+node is designated as primary by setting the primary to `true`.
+
+`praefect['auth_token']` is the token used to authenticate with the GitLab server,
+just like `gitaly['auth_token']` is used for a standard Gitaly server.
+
+The `token` field under each storage listed in `praefect['storage_nodes']` is used
+to authenticate each child Gitaly node with Praefect.
 
 ```ruby
 # /etc/gitlab/gitlab.rb
@@ -76,58 +85,101 @@ primary, by setting the primary to `true`:
 ```ruby
 # /etc/gitlab/gitlab.rb
 
+# Prevent database connections during 'gitlab-ctl reconfigure'
+gitlab_rails['rake_cache_clear'] = false
+gitlab_rails['auto_migrate'] = false
+
+praefect['enable'] = true
+
+# Make Praefect accept connections on all network interfaces. You must use
+# firewalls to restrict access to this address/port.
+praefect['listen_addr'] = '0.0.0.0:2305'
+
 # virtual_storage_name must match the same storage name given to praefect in git_data_dirs
 praefect['virtual_storage_name'] = 'praefect'
-praefect['auth_token'] = 'abc123secret'
-praefect['enable'] = true
-praefect['storage_nodes'] = [
-  {
-    'storage' => 'praefect-git-1',
-    'address' => 'tcp://praefect-git-1.internal',
-    'token'   => 'xyz123secret',
+
+# Authentication token to ensure only authorized servers can communicate with
+# Praefect server
+praefect['auth_token'] = 'praefect-token'
+praefect['storage_nodes'] = {
+  'praefect-gitaly-1' => {
+    'address' => 'tcp://praefect-git-1.internal:8075',
+    'token'   => 'praefect-gitaly-token',
     'primary' => true
   },
-  {
-    'storage' => 'praefect-git-2',
-    'address' => 'tcp://praefect-git-2.internal',
-    'token'   => 'xyz456secret',
+  'praefect-gitaly-2' => {
+    'address' => 'tcp://praefect-git-2.internal:8075',
+    'token'   => 'praefect-gitaly-token'
   },
-  {
-    'storage' => 'praefect-git-3',
-    'address' => 'tcp://praefect-git-3.internal',
-    'token'   => 'xyz789secret',
+  'praefect-gitaly-3' => {
+    'address' => 'tcp://praefect-git-3.internal:8075',
+    'token'   => 'praefect-gitaly-token'
   }
-]
+}
 ```
 
 Save the file and [reconfigure Praefect](../restart_gitlab.md#omnibus-gitlab-reconfigure).
 
-#### GitLab
+#### Gitaly
 
-When Praefect is running, it should be exposed as a storage to GitLab. This
-is done through setting the `git_data_dirs`.
+Next we will configure each Gitaly server assigned to Praefect.  Configuration for these
+is the same as a normal standalone Gitaly server, except that we use storage names and
+auth tokens from Praefect instead of GitLab.
 
-##### On a fresh GitLab installation
+Below is an example configuration for `praefect-gitaly-1`, the only difference for the
+other Gitaly nodes is the storage name under `git_data_dirs`.
 
-On a fresh Gitlab installation, set up the `default` storage to point to praefect:
+Note that `gitaly['auth_token']` matches the `token` value listed under `praefect['storage_nodes']`
+on the Praefect node.
 
 ```ruby
+# /etc/gitlab/gitlab.rb
+
+# Avoid running unnecessary services on the Gitaly server
+postgresql['enable'] = false
+redis['enable'] = false
+nginx['enable'] = false
+prometheus['enable'] = false
+unicorn['enable'] = false
+sidekiq['enable'] = false
+gitlab_workhorse['enable'] = false
+
+# Prevent database connections during 'gitlab-ctl reconfigure'
+gitlab_rails['rake_cache_clear'] = false
+gitlab_rails['auto_migrate'] = false
+
+# Configure the gitlab-shell API callback URL. Without this, `git push` will
+# fail. This can be your 'front door' GitLab URL or an internal load
+# balancer.
+# Don't forget to copy `/etc/gitlab/gitlab-secrets.json` from web server to Gitaly server.
+gitlab_rails['internal_api_url'] = 'https://gitlab.example.com'
+
+# Authentication token to ensure only authorized servers can communicate with
+# Gitaly server
+gitaly['auth_token'] = 'praefect-gitaly-token'
+
+# Make Gitaly accept connections on all network interfaces. You must use
+# firewalls to restrict access to this address/port.
+# Comment out following line if you only want to support TLS connections
+gitaly['listen_addr'] = "0.0.0.0:8075"
+
 git_data_dirs({
-  "default" => {
-    "gitaly_address" => "tcp://praefect.internal:2305"
-  },
+  "praefect-gitaly-1" => {
+    "path" => "/var/opt/gitlab/git-data"
+  }
 })
 ```
 
-##### On an existing GitLab instance
+Note that just as with a standard Gitaly server, `/etc/gitlab/gitlab-secrets.json` must
+be copied from the GitLab server to the Gitaly node for authentication purposes.
 
-On an existing GitLab instance, the `default` storage is already being served by a
-Gitaly node, so an additional storage can be added. `praefect` is chosen in the example
-below, but it can be any name as long as it matches the `virtual_storage_name` in the
-praefect attributes above.
+For more information on Gitaly server configuration, see our [gitaly documentation](index.md#3-gitaly-server-configuration).
 
-Assuming the default storage
-configuration is used, there would be two storages available to GitLab:
+#### GitLab
+
+When Praefect is running, it should be exposed as a storage to GitLab. This
+is done through setting the `git_data_dirs`. Assuming the default storage
+is present, there should be two storages available to GitLab:
 
 ```ruby
 git_data_dirs({
@@ -138,6 +190,28 @@ git_data_dirs({
     "gitaly_address" => "tcp://praefect.internal:2305"
   }
 })
+
+gitlab_rails['gitaly_token'] = 'praefect-token'
 ```
 
+Note that the storage name used is the same as the `praefect['virtual_storage_name']` set
+on the Praefect node.
+
+Also, the `gitlab_rails['gitaly_token']` matches the value of `praefect['auth_token']`
+on Praefect.
+
 Restart GitLab using `gitlab-ctl restart` on the GitLab node.
+
+### Testing Praefect
+
+To test Praefect, first set it as the default storage node for new projects
+using **Admin Area > Settings > Repository > Repository storage**. Next,
+create a new project and check the "Initialize repository with a README" box.
+
+If you receive a 503 error, check `/var/log/gitlab/gitlab-rails/production.log`.
+A `GRPC::Unavailable (14:failed to connect to all addresses)` error indicates
+that GitLab was unable to connect to Praefect.
+
+If the project is created but the README is not, then ensure that the
+`/etc/gitlab/gitlab-secrets.json` file from the GitLab server has been copied
+to the Gitaly servers.