Skip to content

G
gitlab-ce
Commit 3cca7d4d authored by Achilleas Pipinellis's avatar Achilleas Pipinellis
Browse files
Options

Merge branch 'repmgr-with-password-docs' into 'master' 

Add notes on authentication to PG HA document

Closes omnibus-gitlab#2735

See merge request gitlab-org/gitlab-ee!2851
parents bd08d33f a6df68c0
Show whitespace changes
Inline Side-by-side
Showing with 462 additions and 277 deletions
doc/administration/high_availability/alpha_database.md
View file @ 3cca7d4d
# Configuring Databases for GitLab HA # Omnibus GitLab PostgreSQL High Availability
> Note: GitLab HA requires an Enterprise Edition Premium license
**Warning** > Available in [Omnibus GitLab Enterprise Edition Premium](https://about.gitlab.com/gitlab-ee/).
This functionality should be considered beta, use with caution.
**Warning** CAUTION: **Warning:**
This functionality should be considered **beta**, use with caution.
## Overview ## Overview
GitLab supports multiple options for its database backend GitLab supports multiple options for its database backend:
* Using the Omnibus GitLab package to configure PG in HA setup (EEP only). This document contains directions for EEP users.
* Using GitLab with an [externally managed PostgreSQL service](../external_database.md). This could be a cloud provider, or your own service.
or for a non-HA option
* Using the Omnibus Gitlab CE/EES package with a [single PostgreSQL instance](http://docs.gitlab.com/omnibus/settings/database.html).
## Configure Omnibus GitLab package database HA (Enterprise Edition Premium) 1. Using the Omnibus GitLab package to configure PG in HA setup (Enterprise Premium only).
1. Using GitLab with an [externally managed PostgreSQL service](../external_database.md).
This could be a cloud provider, your own service, or for a non-HA option.
1. Using the Omnibus Gitlab Community or Enterprise Starter Edition packages with
a [single PostgreSQL instance](http://docs.gitlab.com/omnibus/settings/database.html).
This document focuses on the first option.
## Preparation
The recommended configuration for a PostgreSQL HA setup requires:
### Preparation - A minimum of three consul server nodes
- A minimum of two database nodes
- Each node will run the following services:
- PostgreSQL - The database itself
- repmgrd - A service to monitor, and handle failover in case of a master failure
- Consul - Used for service discovery, to alert other nodes when failover occurs
- At least one separate node for running the `pgbouncer` service.
## Required information
**Network information for all nodes**
- DNS names - By default, `repmgr` and `pgbouncer` use DNS to locate nodes
- IP address - PostgreSQL does not listen on any network interface by default.
It needs to know which IP address to listen on in order to use the network
interface. It can be set to `0.0.0.0` to listen on all interfaces. It cannot
be set to the loopack address `127.0.0.1`
- Network Address - PostgreSQL access is controlled based on the network source.
This can be in subnet (i.e. `192.168.0.0/255.255.255.0`) or CIDR (i.e.
`192.168.0.0/24`) form.
**User information for `pgbouncer` service**
- The service runs as the same user as the database, default of `gitlab-psql`
- The service will have a regular database user account generated for it
- Default username is `pgbouncer`. In the rest of the documentation we will
refer to this username as `PGBOUNCER_USERNAME`
- Password for `pgbouncer` service. In the rest of the documentation we will
refer to this password as `PGBOUNCER_PASSWORD`
- Password hash for `pgbouncer` service generated from the `pgbouncer` username
and password pair with:
```sh
echo -n 'PASSWORD+USERNAME' | md5sum
```
The recommended configuration for a PostgreSQL HA setup requires: In the rest of the documentation we will refer to this hash as `PGBOUNCER_PASSWORD_HASH`
* A minimum of three consul server nodes - This password will be stored in the following locations:
* A minimum of two database nodes - `/etc/gitlab/gitlab.rb`: hashed, and in plain text
* Each node will run the following services - `/var/opt/gitlab/pgbouncer/pg_auth`: hashed
* postgresql -- The database itself
* repmgrd -- A service to monitor, and handle failover in case of a master failure **User information for the Repmgr service**
* consul -- Used for service discovery, to alert other nodes when failover occurs
* At least one separate node for running the `pgbouncer` service. - The service runs under the same system account as the database by default.
- The service requires a superuser database account be generated for it. This
#### Required information defaults to `gitlab_repmgr`
* Network information for all nodes **User information for the Consul service**
* DNS names -- By default, `repmgr` and `pgbouncer` use DNS to locate nodes
* IP address -- PostgreSQL does not listen on any network interface by default. It needs to know which IP address to listen on in order to use the network interface. It can be set to `0.0.0.0` to listen on all interfaces. It cannot be set to the loopack address 127.0.0.1 - The consul service runs under a dedicated system account by default,
* Network Address -- PostgreSQL access is controlled based on the network source. This can be in subnet (i.e. 192.168.0.0/255.255.255.0) or CIDR (i.e. 192.168.0.0/24) form. `gitlab-consul`. In the rest of the documentation we will refer to this
* User information for `pgbouncer` service username as `CONSUL_USERNAME`
* The service runs as the same user as the database, default of `gitlab-psql` - There will be a database user created with read only access to the repmgr
* The service will have a regular database user account generated for it database
* Default username is `pgbouncer`. In the rest of the documentation we will refer to this username as `PGBOUNCER_USERNAME` - Password for the database user. In the rest of the documentation we will
* Password for `pgbouncer` service. In the rest of the documentation we will refer to this password as `PGBOUNCER_PASSWORD` refer to this password as `CONSUL_DATABASE_PASSWORD`
* Password hash for `pgbouncer` service - Password hash for `gitlab-consul` service generated from the `gitlab-consul`
* This should be generated from `pgbouncer` username and password pair username and password pair with:
* Generate the hash with:
`` ```sh
$ echo -n 'PASSWORD+USERNAME' | md5sum echo -n 'PASSWORD+USERNAME' | md5sum
`` ```
* In the rest of the documentation we will refer to this hash as `PGBOUNCER_PASSWORD_HASH`
* This password will be stored in the following locations In the rest of the documentation we will refer to this hash as `CONSUL_PASSWORD_HASH`
* `/etc/gitlab/gitlab.rb`: hashed, and in plain text - This password will be stored in the following locations:
* `/var/opt/gitlab/pgbouncer/pg_auth`: hashed - `/etc/gitlab/gitlab.rb`: hashed
* User information for the Repmgr service - `/var/opt/gitlab/pgbouncer/pg_auth`: hashed
* The service runs under the same system account as the database by default. - `/var/opt/gitlab/gitlab-consul/.pgpass`: plaintext
* The service requires a superuser database account be generated for it. This defaults to `gitlab_repmgr`
* User information for the Consul service **The number of nodes in the database cluster**
* The consul service runs under a dedicated system account by default, `gitlab-consul`. In the rest of the documentation we will refer to this username as `CONSUL_USERNAME`
* There will be a database user created with read only access to the repmgr database When configuring PostgreSQL, we will set `max_wal_senders` to one more than
* Password for the database user. In the rest of the documentation we will refer to this password as `CONSUL_DATABASE_PASSWORD` this number. This is used to prevent replication from using up all of the
* Password hash for `gitlab-consul` service available database connections.
* This should be generated from `gitlab-consul` username and password pair
* Generate the hash with: ## Installing Omnibus GitLab
``
$ echo -n 'PASSWORD+USERNAME' | md5sum First, make sure to [download/install](https://about.gitlab.com/installation)
`` GitLab Omnibus **on each node**.
* In the rest of the documentation we will refer to this hash as `CONSUL_PASSWORD_HASH`
* This password will be stored in the following locations Just follow **steps 1 and 2**, do not complete any other steps shown in the
* '/etc/gitlab/gitlab.rb`: hashed page above.
* '/var/opt/gitlab/pgbouncer/pg_auth': hashed
* '/var/opt/gitlab/gitlab-consul/.pgpass': plaintext ## Initial node configuration
* The number of nodes in the database cluster.
* When configuring PostgreSQL, we will set `max_wal_senders` to one more than this number. This is used to prevent replication from using up all of the available database connections. Each node needs to be configured to run only the services it needs.
### Installation ### Configuring the Consul server nodes
#### On each node On each Consul node perform the following:
1. Download/install GitLab Omnibus using **steps 1 and 2** from
[GitLab downloads](https://about.gitlab.com/downloads). Do not complete other 1. Edit `/etc/gitlab/gitlab.rb`:
steps on the download page.
```ruby
#### Configuration # Disable all components except Consul
Each node needs to be configured to run only the services it needs. Create an `/etc/gitlab/gitlab.rb` on each node which looks like the following, then run `gitlab-ctl reconfigure` bootstrap['enable'] = false
gitlab_rails['auto_migrate'] = false
##### On each consul server node gitaly['enable'] = false
```ruby gitlab_workhorse['enable'] = false
# Disable all components except Consul mailroom['enable'] = false
bootstrap['enable'] = false nginx['enable'] = false
gitaly['enable'] = false postgresql['enable'] = false
gitlab_workhorse['enable'] = false redis['enable'] = false
mailroom['enable'] = false sidekiq['enable'] = false
nginx['enable'] = false unicorn['enable'] = false
postgresql['enable'] = false
redis['enable'] = false consul['enable'] = true
sidekiq['enable'] = false # START user configuration
unicorn['enable'] = false # Please set the real values as explained in Required Information section
#
consul['enable'] = true consul['configuration'] = {
# START user configuration
# Please set the real values as explained in Required Information section
#
consul['configuration'] = {
server: true, server: true,
retry_join: %w(NAMES OR IPS OF ALL CONSUL NODES) retry_join: %w(NAMES OR IPS OF ALL CONSUL NODES)
} }
# #
# END user configuration # END user configuration
``` ```
##### On each database node 1. [Reconfigure GitLab] for the changes to take effect.
```ruby
# Disable all components except PostgreSQL ### Configuring the Database nodes
postgresql['enable'] = true
bootstrap['enable'] = false On each database node perform the following:
nginx['enable'] = false
unicorn['enable'] = false 1. Edit `/etc/gitlab/gitlab.rb`:
sidekiq['enable'] = false
redis['enable'] = false ```ruby
gitaly['enable'] = false # Disable all components except PostgreSQL
gitlab_workhorse['enable'] = false postgresql['enable'] = true
mailroom['enable'] = false bootstrap['enable'] = false
nginx['enable'] = false
# PostgreSQL configuration unicorn['enable'] = false
postgresql['listen_address'] = '0.0.0.0' sidekiq['enable'] = false
postgresql['trust_auth_cidr_addresses'] = %w(127.0.0.0/24) redis['enable'] = false
postgresql['md5_auth_cidr_addresses'] = %w(0.0.0.0/0) gitaly['enable'] = false
postgresql['hot_standby'] = 'on' gitlab_workhorse['enable'] = false
postgresql['wal_level'] = 'replica' mailroom['enable'] = false
postgresql['shared_preload_libraries'] = 'repmgr_funcs'
# PostgreSQL configuration
# repmgr configuration postgresql['listen_address'] = '0.0.0.0'
repmgr['enable'] = true postgresql['trust_auth_cidr_addresses'] = %w(127.0.0.0/24)
postgresql['md5_auth_cidr_addresses'] = %w(0.0.0.0/0)
# Disable automatic database migrations postgresql['hot_standby'] = 'on'
gitlab_rails['auto_migrate'] = false postgresql['wal_level'] = 'replica'
postgresql['shared_preload_libraries'] = 'repmgr_funcs'
# Enable the consul agent
consul['enable'] = true # repmgr configuration
consul['services'] = %w(postgresql) repmgr['enable'] = true
# START user configuration # Disable automatic database migrations
# Please set the real values as explained in Required Information section gitlab_rails['auto_migrate'] = false
#
postgresql['pgbouncer_user'] = 'PGBOUNCER_USER' # Enable the consul agent
postgresql['pgbouncer_user_password'] = 'PGBOUNCER_PASSWORD_HASH' # This is the hash generated in the preparation section consul['enable'] = true
postgresql['max_wal_senders'] = X consul['services'] = %w(postgresql)
repmgr['trust_auth_cidr_addresses'] = %w(XXX.XXX.XXX.XXX/YY) # This should be the CIDR of the network(s) your database nodes are on
consul['configuration'] = { # START user configuration
# Please set the real values as explained in Required Information section
#
postgresql['pgbouncer_user'] = 'PGBOUNCER_USER'
postgresql['pgbouncer_user_password'] = 'PGBOUNCER_PASSWORD_HASH' # This is the hash generated in the preparation section
postgresql['max_wal_senders'] = X
repmgr['trust_auth_cidr_addresses'] = %w(XXX.XXX.XXX.XXX/YY) # This should be the CIDR of the network(s) your database nodes are on
consul['configuration'] = {
retry_join: %w(NAMES OR IPS OF ALL CONSUL NODES) retry_join: %w(NAMES OR IPS OF ALL CONSUL NODES)
} }
# #
# END user configuration # END user configuration
``` ```
##### On the pgbouncer node 1. [Reconfigure GitLab] for the changes to take effect.
Ensure the following attributes are set
```ruby ### Configuring the Pgbouncer node
# Disable all components except Pgbouncer
postgresql['enable'] = false 1. Edit `/etc/gitlab/gitlab.rb`:
bootstrap['enable'] = false
nginx['enable'] = false ```ruby
unicorn['enable'] = false # Disable all components except Pgbouncer
sidekiq['enable'] = false postgresql['enable'] = false
redis['enable'] = false bootstrap['enable'] = false
gitaly['enable'] = false nginx['enable'] = false
gitlab_workhorse['enable'] = false unicorn['enable'] = false
mailroom['enable'] = false sidekiq['enable'] = false
pgbouncer['enable'] = true redis['enable'] = false
gitaly['enable'] = false
# Configure pgbouncer gitlab_workhorse['enable'] = false
pgbouncer['listen_address'] = '0.0.0.0' mailroom['enable'] = false
pgbouncer['enable'] = true
# Enable the consul agent
consul['enable'] = true # Configure pgbouncer
consul['watchers'] = %w(postgresql) pgbouncer['admin_users'] = %w(pgbouncer gitlab-consul)
pgbouncer['listen_address'] = '0.0.0.0'
# START user configuration
# Please set the real values as explained in Required Information section # Enable the consul agent
# consul['enable'] = true
consul['configuration'] = { consul['watchers'] = %w(postgresql)
# START user configuration
# Please set the real values as explained in Required Information section
#
pgbouncer['users'] = {
'gitlab-consul': {
password: 'CONSUL_PASSWORD_HASH'
},
'pgbouncer': {
password: 'PGBOUNCER_PASSWORD_HASH'
}
}
consul['configuration'] = {
retry_join: %w(NAMES OR IPS OF ALL CONSUL NODES) retry_join: %w(NAMES OR IPS OF ALL CONSUL NODES)
} }
# #
# END user configuration # END user configuration
``` ```
##### Application node(s) 1. [Reconfigure GitLab] for the changes to take effect.
These will be the nodes running the gitlab-rails service. You may have other attributes set, but the following need to be set
```ruby
gitlab_rails['db_host'] = 'PGBOUNCER_NODE'
gitlab_rails['db_port'] = 6432
```
#### Post-configuration ### Configuring the Application nodes
After reconfigure successfully runs, the following steps must be completed to get the cluster up and running
#### Consul server nodes These will be the nodes running the `gitlab-rails` service. You may have other
1. Verify the nodes are all communicating attributes set, but the following need to be set.
```
# consul members 1. Edit `/etc/gitlab/gitlab.rb`:
Node Address Status Type Build Protocol DC
NODE_ONE XXX.XXX.XXX.YYY:8301 alive server 0.9.2 2 gitlab_cluster ```ruby
NODE_TWO XXX.XXX.XXX.YYY:8301 alive server 0.9.2 2 gitlab_cluster gitlab_rails['db_host'] = 'PGBOUNCER_NODE'
NODE_THREE XXX.XXX.XXX.YYY:8301 alive server 0.9.2 2 gitlab_cluster gitlab_rails['db_port'] = 6432
``` ```
##### On the primary database node 1. [Reconfigure GitLab] for the changes to take effect.
1. Open a database prompt: ## Node post-configuration
``` After reconfigure successfully runs, the following steps must be completed to
$ gitlab-psql -d gitlabhq_production get the cluster up and running.
# Output:
### Consul post-configuration
Verify the nodes are all communicating:
```sh
sudo consul members
```
The output should be similar to:
psql (DB_VERSION) ```
Type "help" for help. Node Address Status Type Build Protocol DC
NODE_ONE XXX.XXX.XXX.YYY:8301 alive server 0.9.2 2 gitlab_cluster
NODE_TWO XXX.XXX.XXX.YYY:8301 alive server 0.9.2 2 gitlab_cluster
NODE_THREE XXX.XXX.XXX.YYY:8301 alive server 0.9.2 2 gitlab_cluster
```
### Primary database node post-configuration
1. Open a database prompt:
gitlabhq_production=# ```sh
sudo gitlab-psql -d gitlabhq_production
``` ```
1. Enable the `pg_trgm` extension: 1. Enable the `pg_trgm` extension:
```
```sh
gitlabhq_production=# CREATE EXTENSION pg_trgm; gitlabhq_production=# CREATE EXTENSION pg_trgm;
# Output: # Output:
...@@ -227,105 +293,222 @@ After reconfigure successfully runs, the following steps must be completed to ge ...@@ -227,105 +293,222 @@ After reconfigure successfully runs, the following steps must be completed to ge
``` ```
1. Exit the database prompt by typing `\q` and Enter. 1. Exit the database prompt by typing `\q` and Enter.
1. Verify the cluster is initialized with one node:
```sh
sudo gitlab-ctl repmgr cluster show
```
The output should be similar to the following:
1. Verify the cluster is initialized with one node
``` ```
# gitlab-ctl repmgr cluster show
Role | Name | Upstream | Connection String Role | Name | Upstream | Connection String
----------+-------------|----------|---------------------------------------- ----------+----------|----------|----------------------------------------
* master | HOSTNAME | | host=HOSTNAME user=gitlab_repmgr dbname=gitlab_repmgr * master | HOSTNAME | | host=HOSTNAME user=gitlab_repmgr dbname=gitlab_repmgr
``` ```
##### On each standby node ### Standby nodes post-configuration
1. Setup the repmgr standby
1. Setup the repmgr standby:
```sh
sudo gitlab-ctl repmgr standby setup MASTER_NODE
``` ```
# gitlab-ctl repmgr standby setup MASTER_NODE
1. Verify the node now appears in the cluster:
```sh
sudo gitlab-ctl repmgr cluster show
``` ```
1. Verify the node now appears in the cluster The output should be similar to the following:
``` ```
# gitlab-ctl repmgr cluster show
Role | Name | Upstream | Connection String Role | Name | Upstream | Connection String
----------+------------|------------|------------------------------------------------ ----------+---------|-----------|------------------------------------------------
* master | MASTER | | host=MASTER_HOSTNAME user=gitlab_repmgr dbname=gitlab_repmgr * master | MASTER | | host=MASTER_HOSTNAME user=gitlab_repmgr dbname=gitlab_repmgr
standby | STANDBY | MASTER | host=STANDBY_HOSTNAME user=gitlab_repmgr dbname=gitlab_repmgr standby | STANDBY | MASTER | host=STANDBY_HOSTNAME user=gitlab_repmgr dbname=gitlab_repmgr
``` ```
##### On the pgbouncer node ### Pgbouncer node post-configuration
1. Create a `.pgpass` file user for the `CONSUL_USER` account to be able to reload pgbouncer
1. Create a `.pgpass` file user for the `CONSUL_USER` account to be able to
reload pgbouncer. Confirm the password twice when asked:
```sh
sudo gitlab-ctl write-pgpass --host PGBOUNCER_HOST --database pgbouncer --user pgbouncer --hostuser gitlab-consul
``` ```
# gitlab-ctl write-pgpass --host PGBOUNCER_HOSE --database pgbouncer --user gitlab-consul
Please enter password: **** 1. Ensure the node is talking to the current master:
Confirm password: ****
```sh
sudo /opt/gitlab/embedded/bin/psql -h 127.0.0.1 -p 6432 -d pgbouncer pgbouncer # You will be prompted for PGBOUNCER_PASSWORD
``` ```
1. Ensure the node is talking to the current master Then run:
```sh
show databases ; show clients ;
```
The output should be similar to the following:
``` ```
# /opt/gitlab/embedded/bin/psql -h 127.0.0.1 -p 6432 -d pgbouncer pgbouncer # You will be prompted for PGBOUNCER_PASSWORD
pgbouncer=# show databases ; show clients ;
name | host | port | database | force_user | pool_size | reserve_pool | pool_mode | max_connections | current_connections name | host | port | database | force_user | pool_size | reserve_pool | pool_mode | max_connections | current_connections
---------------------+-------------+------+---------------------+------------+-----------+--------------+-----------+-----------------+--------------------- ---------------------+-------------+------+---------------------+------------+-----------+--------------+-----------+-----------------+---------------------
gitlabhq_production | MASTER_HOST | 5432 | gitlabhq_production | | 20 | 0 | | 0 | 0 gitlabhq_production | MASTER_HOST | 5432 | gitlabhq_production | | 20 | 0 | | 0 | 0
pgbouncer | | 6432 | pgbouncer | pgbouncer | 2 | 0 | statement | 0 | 0 pgbouncer | | 6432 | pgbouncer | pgbouncer | 2 | 0 | statement | 0 | 0
(2 rows) (2 rows)
type | user | database | state | addr | port | local_addr | local_port | connect_time | request_time | ptr | link type | user | database | state | addr | port | local_addr | local_port | connect_time | request_time | ptr | link | remote_pid | tls
| remote_pid | tls ------+-----------+---------------------+---------+----------------+-------+------------+------------+---------------------+---------------------+-----------+------+------------+-----
------+-----------+---------------------+---------+----------------+-------+------------+------------+---------------------+---------------------+-----------+----- C | (nouser) | gitlabhq_production | waiting | IP_OF_APP_NODE | 56512 | 127.0.0.1 | 6432 | 2017-08-21 18:08:51 | 2017-08-21 18:08:51 | 0x22b3700 | | 0 |
-+------------+----- C | pgbouncer | pgbouncer | active | 127.0.0.1 | 56846 | 127.0.0.1 | 6432 | 2017-08-21 18:09:59 | 2017-08-21 18:10:48 | 0x22b3880 | | 0 |
C | (nouser) | gitlabhq_production | waiting | IP_OF_APP_NODE | 56512 | 127.0.0.1 | 6432 | 2017-08-21 18:08:51 | 2017-08-21 18:08:51 | 0x22b3700 |
| 0 |
C | pgbouncer | pgbouncer | active | 127.0.0.1 | 56846 | 127.0.0.1 | 6432 | 2017-08-21 18:09:59 | 2017-08-21 18:10:48 | 0x22b3880 |
| 0 |
(2 rows) (2 rows)
1. It may be necessary to manually run migrations.
``` ```
# gitlab-rake db:migrate
1. It may be necessary to manually run migrations:
```sh
sudo gitlab-rake gitlab:db:configure
``` ```
#### Server running ## Ensure GitLab is running
At this point, your GitLab instance should be up and running, verify you are able to login, and create issues and merge requests.
At this point, your GitLab instance should be up and running. Verify you are
able to login, and create issues and merge requests.
### Failover procedure ## Failover procedure
By default, if the master database fails, repmgrd should promote one of the standby nodes to master automatically, and consul will update pgbouncer with the new master.
By default, if the master database fails, `repmgrd` should promote one of the
standby nodes to master automatically, and consul will update pgbouncer with
the new master.
If you need to failover manually, you have two options: If you need to failover manually, you have two options:
1. Shutdown the current master database
```
# gitlab-ctl stop postgresql
```
The automated failover process will see this and failover to one of the standby nodes.
1. Manually failover **Shutdown the current master database**
1. Ensure the old master node is not still active.
1. Login to the server that should become the new master and run the following Run:
```
# gitlab-ctl repmgr standby promote
```
1. If there are any other standby servers in the cluster, have them follow the new master server ```sh
sudo gitlab-ctl stop postgresql
```
The automated failover process will see this and failover to one of the
standby nodes.
**Or perform a manual failover**
1. Ensure the old master node is not still active.
1. Login to the server that should become the new master and run:
```sh
sudo gitlab-ctl repmgr standby promote
``` ```
# gitlab-ctl repmgr standby follow NEW_MASTER
1. If there are any other standby servers in the cluster, have them follow
the new master server:
```sh
sudo gitlab-ctl repmgr standby follow NEW_MASTER
``` ```
### Restore procedure ## Restore procedure
If a node fails, it can be removed from the cluster, or added back as a standby after it has been restored to service.
* If you want to remove the node from the cluster, on any other node in the cluster, run: If a node fails, it can be removed from the cluster, or added back as a standby
after it has been restored to service.
- If you want to remove the node from the cluster, on any other node in the
cluster, run:
```sh
sudo gitlab-ctl repmgr standby unregister --node=X
``` ```
# gitlab-ctl repmgr standby unregister --node=X # X should be the value of node in repmgr.conf on the old server
where X is be the value of node in `repmgr.conf` on the old server.
- To add the node as a standby server:
```sh
sudo gitlab-ctl repmgr standby follow NEW_MASTER
sudo gitlab-ctl restart repmgrd
``` ```
* To add the node as a standby server[^1] CAUTION: **Warning:** When the server is brought back online, and before
you switch it to a standby node, repmgr will report that there are two masters.
If there are any clients that are still attempting to write to the old master,
this will cause a split, and the old master will need to be resynced from
scratch by performing a `standby setup NEW_MASTER`.
## Alternate configurations
### Database authorization
By default, we give any host on the database network the permission to perform
repmgr operations using PostgreSQL's `trust` method. If you do not want this
level of trust, there are alternatives.
You can trust only the specific nodes that will be database clusters, or you
can require md5 authentication.
#### Trust specific addresses
If you know the IP address, or FQDN of all database and pgbouncer nodes in the
cluster, you can trust only those nodes.
In `/etc/gitlab/gitlab.rb` on all of the database nodes, set
`repmgr['trust_auth_cidr_addresses']` to an array of strings containing all of
the addresses.
If setting to a node's FQDN, they must have a corresponding PTR record in DNS.
If setting to a node's IP address, specify it as `XXX.XXX.XXX.XXX/32`.
For example:
```ruby
repmgr['trust_auth_cidr_addresses'] = %w(192.168.1.44/32 db2.example.com)
```
#### MD5 Authentication
If you are running on an untrusted network, repmgr can use md5 authentication
with a [.pgpass file](https://www.postgresql.org/docs/9.6/static/libpq-pgpass.html)
to authenticate.
You can specify by IP address, FQDN, or by subnet, using the same format as in
the previous section:
1. On the current master node, create a password for the `gitlab` and
`gitlab_repmgr` user:
```sh
sudo gitlab-psql -d template1
template1=# \password gitlab_repmgr
Enter password: ****
Confirm password: ****
template1=# \password gitlab
``` ```
# gitlab-ctl repmgr standby follow NEW_MASTER
# gitlab-ctl restart repmgrd 1. On each database node:
1. Edit `/etc/gitlab/gitlab.rb`:
1. Ensure `repmgr['trust_auth_cidr_addresses']` is **not** set
1. Set `postgresql['md5_auth_cidr_addresses']` to the desired value
1. Set `postgresql['sql_replication_user'] = 'gitlab_repmgr'`
1. Reconfigure with `gitlab-ctl reconfigure`
1. Restart postgresql with `gitlab-ctl restart postgresql`
1. Create a `.pgpass` file. Enter the `gitlab_repmgr` password twice to
when asked:
```sh
sudo gitlab-ctl write-pgpass --user gitlab_repmgr --hostuser gitlab-psql --database '*'
``` ```
[^1]: **Warning**: When the server is brought back online, and before you switch it to a standby node, repmgr will report that there are two masters. 1. On each pgbouncer node, edit `/etc/gitlab/gitlab.rb`:
If there are any clients that are still attempting to write to the old master, this will cause a split, and the old master will need to be resynced from scratch by performing a `standby setup NEW_MASTER`. 1. Ensure `gitlab_rails['db_password']` is set to the plaintext password for
the `gitlab` database user
1. [Reconfigure GitLab] for the changes to take effect
--- ---
...@@ -335,3 +518,5 @@ Read more on high-availability configuration: ...@@ -335,3 +518,5 @@ Read more on high-availability configuration:
1. [Configure NFS](nfs.md) 1. [Configure NFS](nfs.md)
1. [Configure the GitLab application servers](gitlab.md) 1. [Configure the GitLab application servers](gitlab.md)
1. [Configure the load balancers](load_balancer.md) 1. [Configure the load balancers](load_balancer.md)
[reconfigure GitLab]: ../restart_gitlab.md#omnibus-gitlab-reconfigure
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment
GitLab Nexedi Edition | About GitLab | About Nexedi | 沪ICP备2021021310号-2 | 沪ICP备2021021310号-7