@@ -567,6 +567,127 @@ Particular attention should be shown to:
Congratulations! You have configured a highly available Praefect cluster.
### Failover
There are two ways to do a failover from one internal Gitaly node to another as the primary. Manually, or automatically.
As an example, in this `config.toml` we have one virtual storage named "default" with two internal Gitaly nodes behind it.
One is deemed the "primary". This means that read and write traffic will go to `internal_storage_0`, and writes
will get replicated to `internal_storage_1`:
```toml
socket_path="/path/to/Praefect.socket"
# failover_enabled will enable automatic failover
failover_enabled=false
[logging]
format="json"
level="info"
[[virtual_storage]]
name="default"
[[virtual_storage.node]]
name="internal_storage_0"
address="tcp://localhost:9999"
primary=true
token="supersecret"
[[virtual_storage.node]]
name="internal_storage_1"
address="tcp://localhost:9998"
token="supersecret"
```
#### Manual failover
In order to failover from using one internal Gitaly node to using another, a manual failover step can be used. Unless `failover_enabled` is set to `true`
in the `config.toml`, the only way to fail over from one primary to using another node as the primary is to do a manual failover.
1. Move `primary = true` from the current `[[virtual_storage.node]]` to another node in `/etc/gitlab/gitlab.rb`:
```ruby
praefect['virtual_storages']={
'praefect'=>{
'gitaly-1'=>{
'address'=>'tcp://GITALY_HOST:8075',
'token'=>'PRAEFECT_INTERNAL_TOKEN',
# no longer the primary
},
'gitaly-2'=>{
'address'=>'tcp://GITALY_HOST:8075',
'token'=>'PRAEFECT_INTERNAL_TOKEN',
# this is the new primary
'primary'=>true
},
'gitaly-3'=>{
'address'=>'tcp://GITALY_HOST:8075',
'token'=>'PRAEFECT_INTERNAL_TOKEN',
}
}
}
```
1. Save the file and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure).
On a restart, Praefect will send write traffic to `internal_storage_1`. `internal_storage_0` is the new secondary now,
and replication jobs will be created to replicate repository data to `internal_storage_0`**from**`internal_storage_1`
#### Automatic failover
When automatic failover is enabled, Praefect will do automatic detection of the health of internal Gitaly nodes. If the
primary has a certain amount of healthchecks fail, it will decide to promote one of the secondaries to be primary, and
demote the primary to be a secondary.
1. To enable automatic failover, edit `/etc/gitlab/gitlab.rb`:
```ruby
# failover_enabled turns on automatic failover
praefect['failover_enabled']=true
praefect['virtual_storages']={
'praefect'=>{
'gitaly-1'=>{
'address'=>'tcp://GITALY_HOST:8075',
'token'=>'PRAEFECT_INTERNAL_TOKEN',
'primary'=>true
},
'gitaly-2'=>{
'address'=>'tcp://GITALY_HOST:8075',
'token'=>'PRAEFECT_INTERNAL_TOKEN'
},
'gitaly-3'=>{
'address'=>'tcp://GITALY_HOST:8075',
'token'=>'PRAEFECT_INTERNAL_TOKEN'
}
}
}
```
1. Save the file and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure).
Below is the picture when Praefect starts up with the config.toml above:
