Skip to content

G
gitlab-ce
Commit c30132ac authored by Achilleas Pipinellis's avatar Achilleas Pipinellis
Browse files
Options

Merge branch 'docs/jramsay/gitaly-ha' into 'master' 

Improve Gitaly HA docs

Closes gitaly#2470

See merge request gitlab-org/gitlab!25402
parents 51d962d2 6239d350
Show whitespace changes
Inline Side-by-side
Showing with 461 additions and 295 deletions
doc/administration/gitaly/praefect.md
View file @ c30132ac
# Praefect # Praefect: High Availability
NOTE: **Note:** Praefect is an experimental service, and for testing purposes only at NOTE: **Note:** Praefect is an experimental service, and data loss is likely.
this time.
Praefect is an optional reverse-proxy for [Gitaly](../index.md) to manage a Praefect is an optional reverse-proxy for [Gitaly](../index.md) to manage a
cluster of Gitaly nodes for high availability through replication. cluster of Gitaly nodes for high availability. Initially, high availability
If a Gitaly node becomes unavailable, it will be possible to fail over to a be implemented through asynchronous replication. If a Gitaly node becomes
warm Gitaly replica. unavailable, it will be possible to fail over to a warm Gitaly replica.
The first minimal version will support: The first minimal version will support:
- Eventual consistency of the secondary replicas. - Eventual consistency of the secondary replicas.
- Manual fail over from the primary to the secondary. - Automatic fail over from the primary to the secondary.
- Reporting of possible data loss if replication queue is non empty.
Follow the [HA Gitaly epic](https://gitlab.com/groups/gitlab-org/-/epics/1489) Follow the [HA Gitaly epic](https://gitlab.com/groups/gitlab-org/-/epics/1489)
for updates and roadmap. for updates and roadmap.
## Omnibus ## Requirements for configuring Gitaly for High Availability
### Architecture NOTE: **Note:** this reference architecture is not highly available because
Praefect is a single point of failure.
The most common architecture for Praefect is simplified in the diagram below: The minimal [alpha](https://about.gitlab.com/handbook/product/#alpha-beta-ga)
reference architecture additionally requires:
```mermaid - 1 Praefect node
graph TB - 1 PostgreSQL server (PostgreSQL 9.6 or newer)
GitLab --> Praefect; - 3 Gitaly nodes (1 primary, 2 secondary)
Praefect --- PostgreSQL;
Praefect --> Gitaly1; ![Alpha architecture diagram](img/praefect_architecture_v12_9.png)
Praefect --> Gitaly2;
Praefect --> Gitaly3; See the [design
``` document](https://gitlab.com/gitlab-org/gitaly/-/blob/master/doc/design_ha.md)
for implementation details.
## Setup Instructions
If you [installed](https://about.gitlab.com/install/) GitLab using the Omnibus
package (highly recommended), follow the steps below:
1. [Preparation](#preparation)
1. [Configuring the Praefect database](#postgresql)
1. [Configuring the Praefect proxy/router](#praefect)
1. [Configuring each Gitaly node](#gitaly) (once for each Gitaly node)
1. [Updating the GitLab server configuration](#gitlab)
Where `GitLab` is the collection of clients that can request Git operations. ### Preparation
The Praefect node has three storage nodes attached. Praefect itself doesn't
store data, but connects to three Gitaly nodes, `Gitaly-1`, `Gitaly-2`, and `Gitaly-3`.
In order to keep track of replication state, Praefect relies on a Before beginning, you should already have a working GitLab instance. [Learn how
PostgreSQL database. This database is a single point of failure so you to install GitLab](https://about.gitlab.com/install/).
should use a highly available PostgreSQL server for this. GitLab
itself needs a HA PostgreSQL server too, so you could optionally co-locate the Praefect
SQL database on the PostgreSQL server you use for the rest of GitLab.
Praefect may be enabled on its own node or can be run on the GitLab server. Provision a PostgreSQL server (PostgreSQL 9.6 or newer). Configuration through
In the example below we will use a separate server, but the optimal configuration the GitLab Omnibus distribution is not yet supported. Follow this
for Praefect is still being determined. [issue](https://gitlab.com/gitlab-org/gitaly/issues/2476) for updates.
Praefect will handle all Gitaly RPC requests to its child nodes. However, the child nodes Prepare all your new nodes by [installing
will still need to communicate with the GitLab server via its internal API for authentication GitLab](https://about.gitlab.com/install/).
purposes.
### Setup - 1 Praefect node (minimal storage required)
- 3 Gitaly nodes (high CPU, high memory, fast storage)
In this setup guide we will start by configuring Praefect, then its child You will need the IP/host address for each node.
Gitaly nodes, and lastly the GitLab server configuration.
1. `POSTGRESQL_SERVER_ADDRESS`: the IP/host address of the PostgreSQL server
1. `PRAEFECT_SERVER_ADDRESS`: the IP/host address of the Praefect server
1. `GITALY_SERVER_ADDRESS`: the IP/host address of each Gitaly node
#### Secrets #### Secrets
We need to manage the following secrets and make them match across hosts: The communication between components is secured with different secrets, which
are described below. Before you begin, generate a unique secret for each, and
1. `GITLAB_SHELL_SECRET_TOKEN`: this is used by Git hooks to make make note of it. This will make it easy to replace these placeholder tokens
callback HTTP API requests to GitLab when accepting a Git push. This with secure tokens as you complete the setup process.
secret is shared with GitLab Shell for legacy reasons.
1. `PRAEFECT_EXTERNAL_TOKEN`: repositories hosted on your Praefect 1. `GITLAB_SHELL_SECRET_TOKEN`: this is used by Git hooks to make callback HTTP
cluster can only be accessed by Gitaly clients that carry this API requests to GitLab when accepting a Git push. This secret is shared with
token. GitLab Shell for legacy reasons.
1. `PRAEFECT_INTERNAL_TOKEN`: this token is used for replication 1. `PRAEFECT_EXTERNAL_TOKEN`: repositories hosted on your Praefect cluster can
traffic inside your Praefect cluster. This is distinct from only be accessed by Gitaly clients that carry this token.
`PRAEFECT_EXTERNAL_TOKEN` because Gitaly clients must not be able to 1. `PRAEFECT_INTERNAL_TOKEN`: this token is used for replication traffic inside
access internal nodes of the Praefect cluster directly; that could your Praefect cluster. This is distinct from `PRAEFECT_EXTERNAL_TOKEN`
lead to data loss. because Gitaly clients must not be able to access internal nodes of the
Praefect cluster directly; that could lead to data loss.
1. `PRAEFECT_SQL_PASSWORD`: this password is used by Praefect to connect to 1. `PRAEFECT_SQL_PASSWORD`: this password is used by Praefect to connect to
PostgreSQL. PostgreSQL.
We will note in the instructions below where these secrets are required. We will note in the instructions below where these secrets are required.
#### Network addresses ### PostgreSQL
1. `POSTGRESQL_SERVER_ADDRESS`: the host name or IP address of your PostgreSQL server NOTE: **Note:** don't reuse the GitLab application database for the Praefect
database.
#### PostgreSQL To complete this section you will need:
To set up a Praefect cluster you need a highly available PostgreSQL - 1 Praefect node
server. You need PostgreSQL 9.6 or newer. Praefect needs to have a SQL - 1 PostgreSQL server (PostgreSQL 9.6 or newer)
user with the right to create databases. - An SQL user with permissions to create databases
In the instructions below we assume you have administrative access to During this section, we will configure the PostgreSQL server, from the Praefect
your PostgreSQL server via `psql`. Depending on your environment, you node, using `psql` which is installed by GitLab Omnibus.
may also be able to do this via the web interface of your cloud
platform, or via your configuration management system, etc.
Below we assume that you have administrative access as the `postgres` 1. SSH into the **Praefect** node and login as root:
user. First open a `psql` session as the `postgres` user:
```shell ```shell
/opt/gitlab/embedded/bin/psql -h POSTGRESQL_SERVER_ADDRESS -U postgres -d template1 sudo -i
``` ```
Once you are connected, run the following command. Replace 1. Connect to the PostgreSQL server with administrative access. This is likely
`PRAEFECT_SQL_PASSWORD` with the actual (random) password you the `postgres` user. The database `template1` is used because it is created
generated for the `praefect` SQL user: by default on all PostgreSQL servers.
```sql ```shell
CREATE ROLE praefect WITH LOGIN CREATEDB PASSWORD 'PRAEFECT_SQL_PASSWORD'; /opt/gitlab/embedded/bin/psql -U postgres -d template1 -h POSTGRESQL_SERVER_ADDRESS
\q ```
```
Now connect as the `praefect` user to create the database. This has Create a new user `praefect` which will be used by Praefect. Replace
the side effect of verifying that you have access: `PRAEFECT_SQL_PASSWORD` with the strong password you generated in the
preparation step.
```shell ```sql
/opt/gitlab/embedded/bin/psql -h POSTGRESQL_SERVER_ADDRESS -U praefect -d template1 CREATE ROLE praefect WITH LOGIN CREATEDB PASSWORD 'PRAEFECT_SQL_PASSWORD';
``` ```
Once you have connected as the `praefect` user, run: 1. Reconnect to the PostgreSQL server, this time as the `praefect` user:
```sql ```shell
CREATE DATABASE praefect_production WITH ENCODING=UTF8; /opt/gitlab/embedded/bin/psql -U praefect -d template1 -h POSTGRESQL_SERVER_ADDRESS
\q ```
```
Create a new database `praefect_production`. By creating the database while
connected as the `praefect` user, we are confident they have access.
```sql
CREATE DATABASE praefect_production WITH ENCODING=UTF8;
```
The database used by Praefect is now configured.
### Praefect
To complete this section you will need:
- [Configured PostgreSQL server](#postgresql), including:
- IP/host address (`POSTGRESQL_SERVER_ADDRESS`)
- password (`PRAEFECT_SQL_PASSWORD`)
Praefect should be run on a dedicated node. Do not run Praefect on the
application server, or a Gitaly node.
1. SSH into the **Praefect** node and login as root:
```shell
sudo -i
```
1. Disable all other services by editing `/etc/gitlab/gitlab.rb`:
#### Praefect ```ruby
# Disable all other services on the Praefect node
postgresql['enable'] = false
redis['enable'] = false
nginx['enable'] = false
prometheus['enable'] = false
grafana['enable'] = false
unicorn['enable'] = false
sidekiq['enable'] = false
gitlab_workhorse['enable'] = false
gitaly['enable'] = false
On the Praefect node we disable all other services, including Gitaly. We list each # Enable only the Praefect service
Gitaly node that will be connected to Praefect as members of the `praefect` hash in `praefect['virtual_storages']`. praefect['enable'] = true
In the example below, the Gitaly nodes are named `gitaly-N`. Note that one # Prevent database connections during 'gitlab-ctl reconfigure'
node is designated as primary by setting the primary to `true`. gitlab_rails['rake_cache_clear'] = false
gitlab_rails['auto_migrate'] = false
```
If you are using an uncrypted connection to Postgres, set `praefect['database_sslmode']` to false. 1. Configure **Praefect** to listen on network interfaces by editing
`/etc/gitlab/gitlab.rb`:
If you are using an encrypted connection with a client certificate, ```ruby
`praefect['database_sslcert']` and `praefect['database_sslkey']` will need to be set. # Make Praefect accept connections on all network interfaces.
If you are using a custom CA, also set `praefect['database_sslrootcert']`: # Use firewalls to restrict access to this address/port.
praefect['listen_addr'] = '0.0.0.0:2305'
```
```ruby 1. Configure a strong `auth_token` for **Praefect** by editing
# /etc/gitlab/gitlab.rb on praefect server `/etc/gitlab/gitlab.rb`. This will be needed by clients outside the cluster
(like GitLab Shell) to communicate with the Praefect cluster :
# Avoid running unnecessary services on the Gitaly server ```ruby
postgresql['enable'] = false praefect['auth_token'] = 'PRAEFECT_EXTERNAL_TOKEN'
redis['enable'] = false ```
nginx['enable'] = false
prometheus['enable'] = false
grafana['enable'] = false
unicorn['enable'] = false
sidekiq['enable'] = false
gitlab_workhorse['enable'] = false
gitaly['enable'] = false
# Prevent database connections during 'gitlab-ctl reconfigure' 1. Configure **Praefect** to connect to the PostgreSQL database by editing
gitlab_rails['rake_cache_clear'] = false `/etc/gitlab/gitlab.rb`.
gitlab_rails['auto_migrate'] = false
praefect['enable'] = true You will need to replace `POSTGRESQL_SERVER_ADDRESS` with the IP/host address
of the database, and `PRAEFECT_SQL_PASSWORD` with the strong password set
above.
# Make Praefect accept connections on all network interfaces. You must use ```ruby
# firewalls to restrict access to this address/port. praefect['database_host'] = 'POSTGRESQL_SERVER_ADDRESS'
praefect['listen_addr'] = '0.0.0.0:2305' praefect['database_port'] = 5432
praefect['database_user'] = 'praefect'
praefect['database_password'] = 'PRAEFECT_SQL_PASSWORD'
praefect['database_dbname'] = 'praefect_production'
```
# Replace PRAEFECT_EXTERNAL_TOKEN with a real secret If you want to use a TLS client certificate, the options below can be used:
praefect['auth_token'] = 'PRAEFECT_EXTERNAL_TOKEN'
# Replace each instance of PRAEFECT_INTERNAL_TOKEN below with a real ```ruby
# secret, distinct from PRAEFECT_EXTERNAL_TOKEN. # Connect to PostreSQL using a TLS client certificate
# Name of storage hash must match storage name in git_data_dirs on GitLab server. # praefect['database_sslcert'] = '/path/to/client-cert'
praefect['virtual_storages'] = { # praefect['database_sslkey'] = '/path/to/client-key'
# Trust a custom certificate authority
# praefect['database_sslrootcert'] = '/path/to/rootcert'
```
By default Praefect will refuse to make an unencrypted connection to
PostgreSQL. You can override this by uncommenting the following line:
```ruby
# praefect['database_sslmode'] = 'disable'
```
1. Configure the **Praefect** cluster to connect to each Gitaly node in the
cluster by editing `/etc/gitlab/gitlab.rb`.
In the example below we have configured one cluster named `praefect`. This
cluster has three Gitaly nodes `gitaly-1`, `gitaly-2`, and `gitaly-3`, which
will be replicas of each other.
Replace `PRAEFECT_INTERNAL_TOKEN` with a strong secret, which will be used by
Praefect when communicating with Gitaly nodes in the cluster. This token is
distinct from the `PRAEFECT_EXTERNAL_TOKEN`.
Replace `GITALY_HOST` with the IP/host address of the each Gitaly node.
More Gitaly nodes can be added to the cluster to increase the number of
replicas. More clusters can also be added for very large GitLab instances.
NOTE: **Note:** The `gitaly-1` node is currently denoted the primary. This
can be used to manually fail from one node to another. This will be removed
in the future to allow for automatic failover.
```ruby
# Name of storage hash must match storage name in git_data_dirs on GitLab
# server ('praefect') and in git_data_dirs on Gitaly nodes ('gitaly-1')
praefect['virtual_storages'] = {
'praefect' => { 'praefect' => {
'gitaly-1' => { 'gitaly-1' => {
# Replace GITALY_URL_OR_IP below with the real address to connect to. 'address' => 'tcp://GITALY_HOST:8075',
'address' => 'tcp://GITALY_URL_OR_IP:8075',
'token' => 'PRAEFECT_INTERNAL_TOKEN', 'token' => 'PRAEFECT_INTERNAL_TOKEN',
'primary' => true 'primary' => true
}, },
'gitaly-2' => { 'gitaly-2' => {
# Replace GITALY_URL_OR_IP below with the real address to connect to. 'address' => 'tcp://GITALY_HOST:8075',
'address' => 'tcp://GITALY_URL_OR_IP:8075',
'token' => 'PRAEFECT_INTERNAL_TOKEN' 'token' => 'PRAEFECT_INTERNAL_TOKEN'
}, },
'gitaly-3' => { 'gitaly-3' => {
# Replace GITALY_URL_OR_IP below with the real address to connect to. 'address' => 'tcp://GITALY_HOST:8075',
'address' => 'tcp://GITALY_URL_OR_IP:8075',
'token' => 'PRAEFECT_INTERNAL_TOKEN' 'token' => 'PRAEFECT_INTERNAL_TOKEN'
} }
} }
} }
```
# Replace POSTGRESQL_SERVER below with a real IP/host address of the database.
praefect['database_host'] = 'POSTGRESQL_SERVER_ADDRESS'
praefect['database_port'] = 5432
praefect['database_user'] = 'praefect'
# Replace PRAEFECT_SQL_PASSWORD below with a real password of the database.
praefect['database_password'] = 'PRAEFECT_SQL_PASSWORD'
praefect['database_dbname'] = 'praefect_production'
# Uncomment the line below if you do not want to use an encrypted
# connection to PostgreSQL
# praefect['database_sslmode'] = 'disable'
# Uncomment and modify these lines if you are using a TLS client
# certificate to connect to PostgreSQL
# praefect['database_sslcert'] = '/path/to/client-cert'
# praefect['database_sslkey'] = '/path/to/client-key'
# Uncomment and modify this line if your PostgreSQL server uses a custom
# CA
# praefect['database_sslrootcert'] = '/path/to/rootcert'
```
Replace `POSTGRESQL_SERVER_ADDRESS`, `PRAEFECT_EXTERNAL_TOKEN`, `PRAEFECT_INTERNAL_TOKEN`, 1. Save the changes to `/etc/gitlab/gitlab.rb` and [reconfigure Praefect](../restart_gitlab.md#omnibus-gitlab-reconfigure):
and `PRAEFECT_SQL_PASSWORD` with their respective values.
Save the file and reconfigure Praefect: ```shell
sudo gitlab-ctl reconfigure
```
```shell 1. Verify that Praefect can reach PostgreSQL:
sudo gitlab-ctl reconfigure
```
After you reconfigure, verify that Praefect can reach PostgreSQL: ```shell
sudo -u git /opt/gitlab/embedded/bin/praefect -config /var/opt/gitlab/praefect/config.toml sql-ping
```
```shell If the check fails, make sure you have followed the steps correctly. If you
sudo -u git /opt/gitlab/embedded/bin/praefect -config /var/opt/gitlab/praefect/config.toml sql-ping edit `/etc/gitlab/gitlab.rb`, remember to run `sudo gitlab-ctl reconfigure`
``` again before trying the `sql-ping` command.
### Gitaly
NOTE: **Note:** Complete these steps for **each** Gitaly node.
To complete this section you will need:
- [Configured Praefect node](#praefect)
- 3 (or more) servers, with GitLab installed, to be configured as Gitaly nodes.
These should be dedicated nodes, do not run other services on these nodes.
Every Gitaly server assigned to the Praefect cluster needs to be configured. The
configuration is the same as a normal [standalone Gitaly server](../index.md),
except:
- the storage names are exposed to Praefect, not GitLab
- the secret token is shared with Praefect, not GitLab
The configuration of all Gitaly nodes in the Praefect cluster can be identical,
because we rely on Praefect to route operations correctly.
Particular attention should be shown to:
- the `gitaly['auth_token']` configured in this section must match the `token`
value under `praefect['virtual_storages']` on the Praefect node. This was set
in the [previous section](#praefect). This document uses the placeholder
`PRAEFECT_INTERNAL_TOKEN` throughout.
- the storage names in `git_data_dirs` configured in this section must match the
storage names under `praefect['virtual_storages']` on the Praefect node. This
was set in the [previous section](#praefect). This document uses `gitaly-1`,
`gitaly-2`, and `gitaly-3` as Gitaly storage names.
If the check fails, make sure you have followed the steps correctly. If you edit `/etc/gitlab/gitlab.rb`, For more information on Gitaly server configuration, see our [Gitaly
remember to run `sudo gitlab-ctl reconfigure` again before trying the documentation](index.md#3-gitaly-server-configuration).
`sql-ping` command.
1. SSH into the **Gitaly** node and login as root:
#### Gitaly
```shell
Next we will configure each Gitaly server assigned to Praefect. Configuration for these sudo -i
is the same as a normal standalone Gitaly server, except that we use storage names and ```
auth tokens from Praefect instead of GitLab.
1. Disable all other services by editing `/etc/gitlab/gitlab.rb`:
Below is an example configuration for `gitaly-1`, the only difference for the
other Gitaly nodes is the storage name under `git_data_dirs`. ```ruby
# Disable all other services on the Praefect node
Note that `gitaly['auth_token']` matches the `token` value listed under `praefect['virtual_storages']` postgresql['enable'] = false
on the Praefect node. redis['enable'] = false
nginx['enable'] = false
```ruby prometheus['enable'] = false
# /etc/gitlab/gitlab.rb on gitaly node inside praefect cluster grafana['enable'] = false
unicorn['enable'] = false
# Avoid running unnecessary services on the Gitaly server sidekiq['enable'] = false
postgresql['enable'] = false gitlab_workhorse['enable'] = false
redis['enable'] = false prometheus_monitoring['enable'] = false
nginx['enable'] = false
prometheus['enable'] = false # Enable only the Praefect service
grafana['enable'] = false gitaly['enable'] = true
unicorn['enable'] = false
sidekiq['enable'] = false # Prevent database connections during 'gitlab-ctl reconfigure'
gitlab_workhorse['enable'] = false gitlab_rails['rake_cache_clear'] = false
prometheus_monitoring['enable'] = false gitlab_rails['auto_migrate'] = false
```
# Prevent database connections during 'gitlab-ctl reconfigure'
gitlab_rails['rake_cache_clear'] = false 1. Configure **Gitaly** to listen on network interfaces by editing
gitlab_rails['auto_migrate'] = false `/etc/gitlab/gitlab.rb`:
# Replace GITLAB_SHELL_SECRET_TOKEN below with real secret ```ruby
gitlab_shell['secret_token'] = 'GITLAB_SHELL_SECRET_TOKEN' # Make Gitaly accept connections on all network interfaces.
# Use firewalls to restrict access to this address/port.
# Configure the gitlab-shell API callback URL. Without this, `git push` will gitaly['listen_addr'] = '0.0.0.0:8075'
# fail. This can be your 'front door' GitLab URL or an internal load ```
# balancer.
# Possible values could be: 'http://10.23.101.53', 'https://gitlab.example.com', 1. Configure a strong `auth_token` for **Gitaly** by editing
# etc. Please replace GITLAB_SERVER_ADDRESS with proper value and change schema `/etc/gitlab/gitlab.rb`. This will be needed by clients to communicate with
# to 'https' in case you use encrypted connection. this Gitaly nodes. Typically, this token will be the same for all Gitaly
gitlab_rails['internal_api_url'] = 'http://GITLAB_SERVER_ADDRESS' nodes.
# Replace PRAEFECT_INTERNAL_TOKEN below with a real secret. ```ruby
gitaly['auth_token'] = 'PRAEFECT_INTERNAL_TOKEN' gitaly['auth_token'] = 'PRAEFECT_INTERNAL_TOKEN'
```
# Make Gitaly accept connections on all network interfaces. You must use
# firewalls to restrict access to this address/port. 1. Configure the GitLab Shell `secret_token`, and `internal_api_url` which are
# Comment out following line if you only want to support TLS connections needed for `git push` operations.
gitaly['listen_addr'] = "0.0.0.0:8075"
If you have already configured [Gitaly on its own server](../index.md)
git_data_dirs({
# Update this to the name of this Gitaly server which will be later ```ruby
# exposed in the UI under "Admin area > Gitaly" gitlab_shell['secret_token'] = 'GITLAB_SHELL_SECRET_TOKEN'
# Configure the gitlab-shell API callback URL. Without this, `git push` will
# fail. This can be your front door GitLab URL or an internal load balancer.
# Examples: 'https://example.gitlab.com', 'http://1.2.3.4'
gitlab_rails['internal_api_url'] = 'GITLAB_SERVER_URL'
```
1. Configure the storage location for Git data by setting `git_data_dirs` in
`/etc/gitlab/gitlab.rb`. Each Gitaly node should have a unique storage name
(eg `gitaly-1`).
Instead of configuring `git_data_dirs` uniquely for each Gitaly node, it is
often easier to have include the configuration for all Gitaly nodes on every
Gitaly node. This is supported because the Praefect `virtual_storages`
configuration maps each storage name (eg `gitaly-1`) to a specific node, and
requests are routed accordingly. This means every Gitaly node in your fleet
can share the same configuration.
```ruby
# You can include the data dirs for all nodes in the same config, because
# Praefect will only route requests according to the addresses provided in the
# prior step.
git_data_dirs({
"gitaly-1" => { "gitaly-1" => {
"path" => "/var/opt/gitlab/git-data" "path" => "/var/opt/gitlab/git-data"
},
"gitaly-2" => {
"path" => "/var/opt/gitlab/git-data"
},
"gitaly-3" => {
"path" => "/var/opt/gitlab/git-data"
} }
}) })
``` ```
Replace `GITLAB_SHELL_SECRET_TOKEN` and `PRAEFECT_INTERNAL_TOKEN` 1. Save the changes to `/etc/gitlab/gitlab.rb` and [reconfigure Gitaly](../restart_gitlab.md#omnibus-gitlab-reconfigure):
with their respective values.
For more information on Gitaly server configuration, see our [Gitaly documentation](index.md#3-gitaly-server-configuration). ```shell
sudo gitlab-ctl reconfigure
```
When finished editing the configuration file for each Gitaly server, run the **Complete these steps for each Gitaly node!**
reconfigure command to put changes into effect:
```shell After all Gitaly nodes are configured, you can run the Praefect connection
sudo gitlab-ctl reconfigure
```
When all Gitaly servers are configured, you can run the Praefect connection
checker to verify Praefect can connect to all Gitaly servers in the Praefect checker to verify Praefect can connect to all Gitaly servers in the Praefect
config. This can be done by running the following command on the Praefect config.
server:
```shell 1. SSH into the **Praefect** node and run the Praefect connection checker:
sudo /opt/gitlab/embedded/bin/praefect -config /var/opt/gitlab/praefect/config.toml dial-nodes
``` ```shell
sudo /opt/gitlab/embedded/bin/praefect -config /var/opt/gitlab/praefect/config.toml dial-nodes
```
### GitLab
To complete this section you will need:
- [Configured Praefect node](#praefect)
- [Configured Gitaly nodes](#gitaly)
The Praefect cluster needs to be exposed as a storage location to the GitLab
application. This is done by updating the `git_data_dirs`.
#### GitLab Particular attention should be shown to:
When Praefect is running, it should be exposed as a storage to GitLab. This - the storage name added to `git_data_dirs` in this section must match the
is done through setting the `git_data_dirs`. Assuming the default storage storage name under `praefect['virtual_storages']` on the Praefect node. This
is present, there should be two storages available to GitLab: was set in the [Praefect](#praefect) section of this guide. This document uses
`praefect` as the Praefect storage name.
```ruby 1. SSH into the **GitLab** node and login as root:
# /etc/gitlab/gitlab.rb on gitlab server
# Replace PRAEFECT_URL_OR_IP below with real address Praefect can be accessed at. ```shell
# Replace PRAEFECT_EXTERNAL_TOKEN below with real secret. sudo -i
git_data_dirs({ ```
1. Add the Praefect cluster as a storage location by editing
`/etc/gitlab/gitlab.rb`.
You will need to replace:
- `PRAEFECT_URL_OR_IP` with the IP/host address of the Praefect node
- `PRAEFECT_EXTERNAL_TOKEN` with the real secret
```ruby
git_data_dirs({
"default" => { "default" => {
"path" => "/var/opt/gitlab/git-data" "path" => "/var/opt/gitlab/git-data"
}, },
...@@ -323,37 +465,61 @@ git_data_dirs({ ...@@ -323,37 +465,61 @@ git_data_dirs({
"gitaly_address" => "tcp://PRAEFECT_URL_OR_IP:2305", "gitaly_address" => "tcp://PRAEFECT_URL_OR_IP:2305",
"gitaly_token" => 'PRAEFECT_EXTERNAL_TOKEN' "gitaly_token" => 'PRAEFECT_EXTERNAL_TOKEN'
} }
}) })
```
# Replace GITLAB_SHELL_SECRET_TOKEN below with real secret 1. Configure the `gitlab_shell['secret_token']` so that callbacks from Gitaly
gitlab_shell['secret_token'] = 'GITLAB_SHELL_SECRET_TOKEN' nodes during a `git push` are properly authenticated by editing
`/etc/gitlab/gitlab.rb`:
# Possible values could be: 'http://10.23.101.53', 'https://gitlab.example.com', You will need to replace `GITLAB_SHELL_SECRET_TOKEN` with the real secret.
# etc. Please replace GITLAB_SERVER_ADDRESS with proper value and change schema
# to 'https' in case you use encrypted connection. For more info please refer
# to https://docs.gitlab.com/omnibus/settings/configuration.html#configuring-the-external-url-for-gitlab
external_url "http://<GITLAB_SERVER_ADDRESS>"
```
Replace `GITLAB_SHELL_SECRET_TOKEN` and `PRAEFECT_EXTERNAL_TOKEN` ```ruby
with their respective values. gitlab_shell['secret_token'] = 'GITLAB_SHELL_SECRET_TOKEN'
```
Note that the storage name used is the same as the `praefect['virtual_storage_name']` set 1. Save the changes to `/etc/gitlab/gitlab.rb` and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure):
on the Praefect node.
Save your changes and reconfigure GitLab: ```shell
sudo gitlab-ctl reconfigure
```
```shell 1. Verify that GitLab can reach Praefect:
sudo gitlab-ctl reconfigure
``` ```shell
sudo gitlab-rake gitlab:gitaly:check
```
1. Update the **Repository storage** settings from **Admin Area > Settings >
Repository > Repository storage** to make the newly configured Praefect
cluster the storage location for new Git repositories.
- Deselect the **default** storage location
- Select the **praefect** storage location
1. Verify everything is still working by creating a new project. Check the
"Initialize repository with a README" box so that there is content in the
repository that viewed. If the project is created, and you can see the
README file, it works!
Run `sudo gitlab-rake gitlab:gitaly:check` to confirm that GitLab can reach Praefect. Congratulations! You have configured a highly available Praefect cluster, and
### Testing Praefect ## Migrating existing repositories to Praefect
If your GitLab instance already has repositories, these won't be migrated
automatically.
Repositories may be moved from one storage location using the [Repository
API](../../api/projects.html#edit-project):
```shell
curl --request PUT \
--header "PRIVATE-TOKEN: <your_access_token>" \
--data "repository_storage=praefect" \
https://example.gitlab.com/api/v4/projects/123
```
To test Praefect, first set it as the default storage node for new projects ## Debugging Praefect
using **Admin Area > Settings > Repository > Repository storage**. Next,
create a new project and check the "Initialize repository with a README" box.
If you receive an error, check `/var/log/gitlab/gitlab-rails/production.log`. If you receive an error, check `/var/log/gitlab/gitlab-rails/production.log`.
......
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment
GitLab Nexedi Edition | About GitLab | About Nexedi | 沪ICP备2021021310号-2 | 沪ICP备2021021310号-7