Skip to content

G
gitlab-ce
Commit 52954d5e authored by Achilleas Pipinellis's avatar Achilleas Pipinellis
Browse files
Options

Split database steps to Omnibus and source

parent 02568941
Hide whitespace changes
Inline Side-by-side
Showing with 13 additions and 255 deletions
doc/gitlab-geo/database.md
View file @ 52954d5e
# GitLab Geo database replication # GitLab Geo database replication
>**Note:**
This is the documentation for the Omnibus GitLab packages. For installations
from source, follow the
[**database replication for installations from source**](database_source.md) guide.
This document describes the minimal steps you have to take in order to This document describes the minimal steps you have to take in order to
replicate your GitLab database into another server. You may have to change replicate your GitLab database into another server. You may have to change
some values according to your database setup, how big it is, etc. some values according to your database setup, how big it is, etc.
<!-- START doctoc generated TOC please keep comment here to allow auto update -->
<!-- DON'T EDIT THIS SECTION, INSTEAD RE-RUN doctoc TO UPDATE -->
**Table of Contents**
- [PostgreSQL replication](#postgresql-replication)
- [Prerequisites](#prerequisites)
- [Step 1. Configure the primary server](#step-1-configure-the-primary-server)
- [Step 2. Configure the secondary server](#step-2-configure-the-secondary-server)
- [Step 3. Initiate the replication process](#step-3-initiate-the-replication-process)
- [Next steps](#next-steps)
- [MySQL replication](#mysql-replication)
<!-- END doctoc generated TOC please keep comment here to allow auto update -->
## PostgreSQL replication ## PostgreSQL replication
The GitLab primary node where the write operations happen will connect to The GitLab primary node where the write operations happen will connect to
...@@ -44,8 +35,6 @@ The following guide assumes that: ...@@ -44,8 +35,6 @@ The following guide assumes that:
### Step 1. Configure the primary server ### Step 1. Configure the primary server
**For Omnibus installations**
1. SSH into your GitLab **primary** server and login as root: 1. SSH into your GitLab **primary** server and login as root:
``` ```
...@@ -118,70 +107,8 @@ The following guide assumes that: ...@@ -118,70 +107,8 @@ The following guide assumes that:
public IP. public IP.
1. Continue to [set up the secondary server](#step-2-configure-the-secondary-server). 1. Continue to [set up the secondary server](#step-2-configure-the-secondary-server).
---
**For installations from source**
1. SSH into your database **primary** server and login as root:
```
sudo -i
```
1. Create a replication user named `gitlab_replicator`:
```bash
sudo -u postgres psql -c "CREATE USER gitlab_replicator REPLICATION ENCRYPTED PASSWORD 'thepassword';"
```
1. Edit `postgresql.conf` to configure the primary server for streaming replication
(for Debian/Ubuntu that would be `/etc/postgresql/9.x/main/postgresql.conf`):
```bash
listen_address = '1.2.3.4'
wal_level = hot_standby
max_wal_senders = 5
checkpoint_segments = 10
wal_keep_segments = 10
hot_standby = on
```
See the Omnibus notes above for more details of `listen_address`.
Edit the `wal` values as you see fit.
1. Set the access control on the primary to allow TCP connections using the
server's public IP and set the connection from the secondary to require a
password. Edit `pg_hba.conf` (for Debian/Ubuntu that would be
`/etc/postgresql/9.x/main/pg_hba.conf`):
```bash
host all all 127.0.0.1/32 trust
host all all 1.2.3.4/32 trust
host replication gitlab_replicator 5.6.7.8/32 md5
```
Where `1.2.3.4` is the public IP address of the primary server, and `5.6.7.8`
the public IP address of the secondary one. If you want to add another
secondary, add one more row like the replication one and change the IP
address:
```bash
host all all 127.0.0.1/32 trust
host all all 1.2.3.4/32 trust
host replication gitlab_replicator 5.6.7.8/32 md5
host replication gitlab_replicator 11.22.33.44/32 md5
```
1. Restart PostgreSQL for the changes to take effect.
1. Now that the PostgreSQL server is set up to accept remote connections, run
`netstat -plnt` to make sure that PostgreSQL is listening to the server's
public IP.
### Step 2. Configure the secondary server ### Step 2. Configure the secondary server
**For Omnibus installations**
1. SSH into your GitLab **secondary** server and login as root: 1. SSH into your GitLab **secondary** server and login as root:
``` ```
...@@ -217,46 +144,6 @@ The following guide assumes that: ...@@ -217,46 +144,6 @@ The following guide assumes that:
1. [Reconfigure GitLab][] for the changes to take effect. 1. [Reconfigure GitLab][] for the changes to take effect.
1. Continue to [initiate the replication process](#step-3-initiate-the-replication-process). 1. Continue to [initiate the replication process](#step-3-initiate-the-replication-process).
---
**For installations from source**
1. SSH into your database **secondary** server and login as root:
```
sudo -i
```
1. Test that the remote connection to the primary server works:
```
sudo -u postgres psql -h 1.2.3.4 -U gitlab_replicator -d gitlabhq_production -W
```
When prompted enter the password you set in the first step for the
`gitlab_replicator` user. If all worked correctly, you should see the
database prompt.
1. Exit the PostgreSQL console:
```
\q
```
1. Edit `postgresql.conf` to configure the secondary for streaming replication
(for Debian/Ubuntu that would be `/etc/postgresql/9.x/main/postgresql.conf`):
```bash
wal_level = hot_standby
max_wal_senders = 5
checkpoint_segments = 10
wal_keep_segments = 10
hot_standby = on
```
1. Restart PostgreSQL for the changes to take effect.
1. Continue to [initiate the replication process](#step-3-initiate-the-replication-process).
### Step 3. Initiate the replication process ### Step 3. Initiate the replication process
Below we provide a script that connects to the primary server, replicates the Below we provide a script that connects to the primary server, replicates the
......
doc/gitlab-geo/database_source.md
View file @ 52954d5e
# GitLab Geo database replication # GitLab Geo database replication
>**Note:**
This is the documentation for installations from source. For installations
using the Omnibus GitLab packages, follow the
[**database replication for Omnibus GitLab**](database.md) guide.
This document describes the minimal steps you have to take in order to This document describes the minimal steps you have to take in order to
replicate your GitLab database into another server. You may have to change replicate your GitLab database into another server. You may have to change
some values according to your database setup, how big it is, etc. some values according to your database setup, how big it is, etc.
<!-- START doctoc generated TOC please keep comment here to allow auto update -->
<!-- DON'T EDIT THIS SECTION, INSTEAD RE-RUN doctoc TO UPDATE -->
**Table of Contents**
- [PostgreSQL replication](#postgresql-replication)
- [Prerequisites](#prerequisites)
- [Step 1. Configure the primary server](#step-1-configure-the-primary-server)
- [Step 2. Configure the secondary server](#step-2-configure-the-secondary-server)
- [Step 3. Initiate the replication process](#step-3-initiate-the-replication-process)
- [Next steps](#next-steps)
- [MySQL replication](#mysql-replication)
<!-- END doctoc generated TOC please keep comment here to allow auto update -->
## PostgreSQL replication ## PostgreSQL replication
The GitLab primary node where the write operations happen will connect to The GitLab primary node where the write operations happen will connect to
...@@ -44,84 +35,6 @@ The following guide assumes that: ...@@ -44,84 +35,6 @@ The following guide assumes that:
### Step 1. Configure the primary server ### Step 1. Configure the primary server
**For Omnibus installations**
1. SSH into your GitLab **primary** server and login as root:
```
sudo -i
```
1. Omnibus GitLab has already a replication user called `gitlab_replicator`.
You must set its password manually. Replace `thepassword` with a strong
password:
```bash
sudo -u gitlab-psql /opt/gitlab/embedded/bin/psql -h /var/opt/gitlab/postgresql \
-d template1 \
-c "ALTER USER gitlab_replicator WITH ENCRYPTED PASSWORD 'thepassword'"
```
1. Edit `/etc/gitlab/gitlab.rb` and add the following:
```ruby
postgresql['listen_address'] = "1.2.3.4"
postgresql['trust_auth_cidr_addresses'] = ['127.0.0.1/32','1.2.3.4/32']
postgresql['md5_auth_cidr_addresses'] = ['5.6.7.8/32']
postgresql['sql_replication_user'] = "gitlab_replicator"
postgresql['wal_level'] = "hot_standby"
postgresql['max_wal_senders'] = 10
postgresql['wal_keep_segments'] = 10
postgresql['hot_standby'] = "on"
```
Where `1.2.3.4` is the public IP address of the primary server, and `5.6.7.8`
the public IP address of the secondary one.
For security reasons, PostgreSQL by default only listens on the local
interface (e.g. 127.0.0.1). However, GitLab Geo needs to communicate
between the primary and secondary nodes over a common network, such as a
corporate LAN or the public Internet. For this reason, we need to
configure PostgreSQL to listen on more interfaces.
The `listen_address` option opens PostgreSQL up to external connections
with the interface corresponding to the given IP. See [the PostgreSQL
documentation](https://www.postgresql.org/docs/9.6/static/runtime-config-connection.html)
for more details.
Note that if you are running GitLab Geo with a cloud provider (e.g. Amazon
Web Services), the internal interface IP (as provided by `ifconfig`) may
be different from the public IP address. For example, suppose you have a
nodes with the following configuration:
|Node Type|Internal IP|External IP|
|---------|-----------|-----------|
|Primary|10.1.5.3|54.193.124.100|
|Secondary|10.1.10.5|54.193.100.155|
In this case, for `1.2.3.4` use the internal IP of the primary node: 10.1.5.3.
For `5.6.7.8`, use the external of the secondary node: 54.193.100.155.
If you want to add another secondary, the relevant setting would look like:
```ruby
postgresql['md5_auth_cidr_addresses'] = ['5.6.7.8/32','11.22.33.44/32']
```
Edit the `wal` values as you see fit.
1. Check to make sure your firewall rules are set so that the secondary nodes
can access port 5432 on the primary node.
1. Save the file and [reconfigure GitLab][] for the changes to take effect.
1. Now that the PostgreSQL server is set up to accept remote connections, run
`netstat -plnt` to make sure that PostgreSQL is listening to the server's
public IP.
1. Continue to [set up the secondary server](#step-2-configure-the-secondary-server).
---
**For installations from source**
1. SSH into your database **primary** server and login as root: 1. SSH into your database **primary** server and login as root:
``` ```
...@@ -180,47 +93,6 @@ The following guide assumes that: ...@@ -180,47 +93,6 @@ The following guide assumes that:
### Step 2. Configure the secondary server ### Step 2. Configure the secondary server
**For Omnibus installations**
1. SSH into your GitLab **secondary** server and login as root:
```
sudo -i
```
1. Test that the remote connection to the primary server works:
```
sudo -u gitlab-psql /opt/gitlab/embedded/bin/psql -h 1.2.3.4 -U gitlab_replicator -d gitlabhq_production -W
```
When prompted enter the password you set in the first step for the
`gitlab_replicator` user. If all worked correctly, you should see the
database prompt.
1. Exit the PostgreSQL console:
```
\q
```
1. Edit `/etc/gitlab/gitlab.rb` and add the following:
```ruby
postgresql['wal_level'] = "hot_standby"
postgresql['max_wal_senders'] = 10
postgresql['wal_keep_segments'] = 10
postgresql['hot_standby'] = "on"
gitlab_rails['auto_migrate'] = false # prevents migrations to be executed on the secondary server
```
1. [Reconfigure GitLab][] for the changes to take effect.
1. Continue to [initiate the replication process](#step-3-initiate-the-replication-process).
---
**For installations from source**
1. SSH into your database **secondary** server and login as root: 1. SSH into your database **secondary** server and login as root:
``` ```
...@@ -262,9 +134,8 @@ The following guide assumes that: ...@@ -262,9 +134,8 @@ The following guide assumes that:
Below we provide a script that connects to the primary server, replicates the Below we provide a script that connects to the primary server, replicates the
database and creates the needed files for replication. database and creates the needed files for replication.
The directories used are the defaults that are set up in Omnibus. If you have The directories used are the defaults for Debian/Ubuntu. If you have changed
changed any defaults or are using a source installation, configure it as you any defaults, configure it as you see fit replacing the directories and paths.
see fit replacing the directories and paths.
>**Warning:** >**Warning:**
Make sure to run this on the **secondary** server as it removes all PostgreSQL's Make sure to run this on the **secondary** server as it removes all PostgreSQL's
...@@ -336,7 +207,7 @@ The replication process is now over. ...@@ -336,7 +207,7 @@ The replication process is now over.
Now that the database replication is done, the next step is to configure GitLab. Now that the database replication is done, the next step is to configure GitLab.
[➤ GitLab Geo configuration](configuration.md) [➤ GitLab Geo configuration](configuration_source.md)
## MySQL replication ## MySQL replication
......
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment
GitLab Nexedi Edition | About GitLab | About Nexedi | 沪ICP备2021021310号-2 | 沪ICP备2021021310号-7