After a failover, it is possible to fail back to the demoted primary to
After a failover, it is possible to fail back to the demoted **primary** node to
restore your original configuration. This process consists of two steps:
restore your original configuration. This process consists of two steps:
1. Making the old primary a secondary
1. Making the old **primary** node a **secondary** node.
1. Promoting a secondary to a primary
1. Promoting a **secondary** node to a **primary** node.
> *Warning:* If you have any doubts about the consistency of the data on this node, we recommend to set up it from scratch.
CAUTION: **Caution:**
If you have any doubts about the consistency of the data on this node, we recommend setting it up from scratch.
## Configure the former primary to be a secondary
## Configure the former **primary** node to be a **secondary** node
Since the former primary will be out of sync with the current primary, the first step is
Since the former **primary** node will be out of sync with the current **primary** node, the first step is to bring the former **primary** node up to date. Note, deletion of data stored on disk like
to bring the former primary up to date. Note, deletion of data stored on disk like
repositories and uploads will not be replayed when bringing the former **primary** node back
repositories and uploads will not be replayed when bringing the former primary in back
into sync, which may result in increased disk usage.
into sync, which may result in increased disk usage.
Alternatively, you can [set up a new secondary GitLab instance][setup-geo] to avoid this.
Alternatively, you can [set up a new **secondary** GitLab instance][setup-geo] to avoid this.
To bring the former primary up to date:
To bring the former **primary** node up to date:
1. SSH into the former primary that has fallen behind
1. SSH into the former **primary** node that has fallen behind.
1. Make sure all the services are up:
1. Make sure all the services are up:
```bash
```sh
sudo gitlab-ctl start
sudo gitlab-ctl start
```
```
> **Note 1:** If you [disabled primary permanently][disaster-recovery-disable-primary],
> **Note 1:** If you [disabled the **primary** node permanently][disaster-recovery-disable-primary],
> you need to undo those steps now. For Debian/Ubuntu you just need to run
> you need to undo those steps now. For Debian/Ubuntu you just need to run
> `sudo systemctl enable gitlab-runsvdir`. For CentOS 6, you need to install
> `sudo systemctl enable gitlab-runsvdir`. For CentOS 6, you need to install
> the GitLab instance from scratch and set it up as a secondary node by
> the GitLab instance from scratch and set it up as a **secondary** node by
> following [Setup instructions][setup-geo]. In this case you don't need to follow the next step.
> following [Setup instructions][setup-geo]. In this case, you don't need to follow the next step.
>
>
> **Note 2:** If you [changed the DNS records](index.md#step-4-optional-updating-the-primary-domains-dns-record)
> **Note 2:** If you [changed the DNS records](index.md#step-4-optional-updating-the-primary-domains-dns-record)
> for this node during disaster recovery procedure you may need to [block
> for this node during disaster recovery procedure you may need to [block
...
@@ -37,25 +37,25 @@ To bring the former primary up to date:
...
@@ -37,25 +37,25 @@ To bring the former primary up to date:
> during this procedure.
> during this procedure.
1.[Setup database replication][database-replication]. Note that in this
1.[Setup database replication][database-replication]. Note that in this
case, primary refers to the current primary, and secondary refers to the
case, **primary** node refers to the current **primary** node, and **secondary** node refers to the
former primary.
former **primary** node.
If you have lost your original primary, follow the
If you have lost your original **primary** node, follow the
[setup instructions][setup-geo] to set up a new secondary.
[setup instructions][setup-geo] to set up a new **secondary** node.
## Promote the secondary to primary
## Promote the **secondary** node to **primary** node
When the initial replication is complete and the primary and secondary are
When the initial replication is complete and the **primary** node and **secondary** node are
closely in sync, you can do a [planned failover].
closely in sync, you can do a [planned failover].
## Restore the secondary node
## Restore the **secondary** node
If your objective is to have two nodes again, you need to bring your secondary
If your objective is to have two nodes again, you need to bring your **secondary**
node back online as well by repeating the first step
node back online as well by repeating the first step
([configure the former primary to be a secondary](#configure-the-former-primary-to-be-a-secondary))
([configure the former **primary** node to be a **secondary** node](#configure-the-former-primary-node-to-be-a-secondary-node))
@@ -419,9 +419,9 @@ data before running `pg_basebackup`.
...
@@ -419,9 +419,9 @@ data before running `pg_basebackup`.
the instructions above are carefully written to ensure protection against
the instructions above are carefully written to ensure protection against
both passive eavesdroppers and active "man-in-the-middle" attackers.
both passive eavesdroppers and active "man-in-the-middle" attackers.
- Change the `--slot-name` to the name of the replication slot
- Change the `--slot-name` to the name of the replication slot
to be used on the primary database. The script will attempt to create the
to be used on the **primary** database. The script will attempt to create the
replication slot automatically if it does not exist.
replication slot automatically if it does not exist.
- If you're repurposing an old server into a Geo secondary, you'll need to
- If you're repurposing an old server into a Geo **secondary** node, you'll need to
add `--force` to the command line.
add `--force` to the command line.
- When not in a production machine you can disable backup step if you
- When not in a production machine you can disable backup step if you
really sure this is what you want by adding `--skip-backup`
really sure this is what you want by adding `--skip-backup`
...
@@ -433,19 +433,19 @@ The replication process is now complete.
...
@@ -433,19 +433,19 @@ The replication process is now complete.
[PGBouncer](http://pgbouncer.github.io/) may be used with GitLab Geo to pool
[PGBouncer](http://pgbouncer.github.io/) may be used with GitLab Geo to pool
PostgreSQL connections. We recommend using PGBouncer if you use GitLab in a
PostgreSQL connections. We recommend using PGBouncer if you use GitLab in a
high-availability configuration with a cluster of nodes supporting a Geo
high-availability configuration with a cluster of nodes supporting a Geo
primary and another cluster of nodes supporting a Geo secondary. For more
**primary** node and another cluster of nodes supporting a Geo **secondary** node. For more
information, see the [Omnibus HA](https://docs.gitlab.com/ee/administration/high_availability/database.html#configure-using-omnibus-for-high-availability)
information, see the [Omnibus HA](https://docs.gitlab.com/ee/administration/high_availability/database.html#configure-using-omnibus-for-high-availability)
documentation.
documentation.
For a Geo secondary to work properly with PGBouncer in front of the database,
For a Geo **secondary** node to work properly with PGBouncer in front of the database,
it will need a separate read-only user to make [PostgreSQL FDW queries][FDW]
it will need a separate read-only user to make [PostgreSQL FDW queries][FDW]
work:
work:
1. On the primary Geo database, enter the PostgreSQL on the console as an
1. On the **primary** Geo database, enter the PostgreSQL on the console as an
admin user. If you are using an Omnibus-managed database, log onto the primary
admin user. If you are using an Omnibus-managed database, log onto the **primary**
@@ -193,9 +193,9 @@ There is an [issue where support is being discussed](https://gitlab.com/gitlab-o
...
@@ -193,9 +193,9 @@ There is an [issue where support is being discussed](https://gitlab.com/gitlab-o
`secondary.geo.example.com`, you may use `secondary_example` as the slot
`secondary.geo.example.com`, you may use `secondary_example` as the slot
name.
name.
1. Create the replication slot on the primary:
1. Create the replication slot on the **primary** node:
```bash
```sh
$ sudo -u postgres psql -c "SELECT * FROM pg_create_physical_replication_slot('secondary_example');"
$ sudo -u postgres psql -c "SELECT * FROM pg_create_physical_replication_slot('secondary_example');"
slot_name | xlog_position
slot_name | xlog_position
------------------+---------------
------------------+---------------
...
@@ -209,16 +209,15 @@ There is an [issue where support is being discussed](https://gitlab.com/gitlab-o
...
@@ -209,16 +209,15 @@ There is an [issue where support is being discussed](https://gitlab.com/gitlab-o
### Step 2. Configure the secondary server
### Step 2. Configure the secondary server
Follow the first steps in ["configure the secondary server"][database-replication],
Follow the first steps in ["configure the secondary server"][database-replication] and note that since you are installing from source, the username and
but note that since you are installing from source, the username and
group listed as `gitlab-psql` in those steps should be replaced by `postgres`
group listed as `gitlab-psql` in those steps should be replaced by `postgres`
instead. After completing the "Test that the `gitlab-psql` user can connect to
instead. After completing the "Test that the `gitlab-psql` user can connect to
the primary's database" step, continue here:
the **primary** node's database" step, continue here:
1. Edit `postgresql.conf` to configure the secondary for streaming replication
1. Edit `postgresql.conf` to configure the secondary for streaming replication
(for Debian/Ubuntu that would be `/etc/postgresql/9.*/main/postgresql.conf`):
(for Debian/Ubuntu that would be `/etc/postgresql/9.*/main/postgresql.conf`):
```bash
```sh
wal_level = hot_standby
wal_level = hot_standby
max_wal_senders = 5
max_wal_senders = 5
checkpoint_segments = 10
checkpoint_segments = 10
...
@@ -235,9 +234,9 @@ and recover automatically from some replication issues. Follow the steps below t
...
@@ -235,9 +234,9 @@ and recover automatically from some replication issues. Follow the steps below t
the tracking database.
the tracking database.
1. On the secondary node, run the following command to create `database_geo.yml` with the
1. On the secondary node, run the following command to create `database_geo.yml` with the
information of your secondary PostgreSQL instance:
information of your secondary PostgreSQL instance:
@@ -7,22 +7,22 @@ The requirements are listed [on the index page](index.md#requirements-for-runnin
...
@@ -7,22 +7,22 @@ The requirements are listed [on the index page](index.md#requirements-for-runnin
## Can I use Geo in a disaster recovery situation?
## Can I use Geo in a disaster recovery situation?
Yes, but there are limitations to what we replicate (see
Yes, but there are limitations to what we replicate (see
[What data is replicated to a secondary node?](#what-data-is-replicated-to-a-secondary-node)).
[What data is replicated to a **secondary** node?](#what-data-is-replicated-to-a-secondary-node)).
Read the documentation for [Disaster Recovery](../disaster_recovery/index.md).
Read the documentation for [Disaster Recovery](../disaster_recovery/index.md).
## What data is replicated to a secondary node?
## What data is replicated to a **secondary** node?
We currently replicate project repositories, LFS objects, generated
We currently replicate project repositories, LFS objects, generated
attachments / avatars and the whole database. This means user accounts,
attachments / avatars and the whole database. This means user accounts,
issues, merge requests, groups, project data, etc., will be available for
issues, merge requests, groups, project data, etc., will be available for
query.
query.
## Can I git push to a secondary node?
## Can I git push to a **secondary** node?
Yes! Pushing directly to a **secondary** node (for both HTTP and SSH, including git-lfs) was [introduced](https://about.gitlab.com/2018/09/22/gitlab-11-3-released/) in [GitLab Premium](https://about.gitlab.com/pricing/#self-managed) 11.3.
Yes! Pushing directly to a **secondary** node (for both HTTP and SSH, including git-lfs) was [introduced](https://about.gitlab.com/2018/09/22/gitlab-11-3-released/) in [GitLab Premium](https://about.gitlab.com/pricing/#self-managed) 11.3.
## How long does it take to have a commit replicated to a secondary node?
## How long does it take to have a commit replicated to a **secondary** node?
All replication operations are asynchronous and are queued to be dispatched. Therefore, it depends on a lot of
All replication operations are asynchronous and are queued to be dispatched. Therefore, it depends on a lot of
factors including the amount of traffic, how big your commit is, the
factors including the amount of traffic, how big your commit is, the
...
@@ -30,8 +30,8 @@ connectivity between your nodes, your hardware, etc.
...
@@ -30,8 +30,8 @@ connectivity between your nodes, your hardware, etc.
## What if the SSH server runs at a different port?
## What if the SSH server runs at a different port?
That's totally fine. We use HTTP(s) to fetch repository changes from the **primary** node to**secondary** nodes.
That's totally fine. We use HTTP(s) to fetch repository changes from the **primary** node to all**secondary** nodes.
## Is this possible to set up a Docker Registry for a secondary node that mirrors the one on a primary node?
## Is this possible to set up a Docker Registry for a **secondary** node that mirrors the one on the **primary** node?
Yes. See [Docker Registry for a secondary Geo node](docker_registry.md).
Yes. See [Docker Registry for a **secondary** node](docker_registry.md).
The topology above assumes that the primary and secondary Geo clusters
The topology above assumes that the **primary** and **secondary** Geo clusters
are located in two separate locations, on their own virtual network
are located in two separate locations, on their own virtual network
with private IP addresses. The network is configured such that all machines within
with private IP addresses. The network is configured such that all machines within
one geographic location can communicate with each other using their private IP addresses.
one geographic location can communicate with each other using their private IP addresses.
...
@@ -20,14 +20,14 @@ network topology of your deployment.
...
@@ -20,14 +20,14 @@ network topology of your deployment.
The only external way to access the two Geo deployments is by HTTPS at
The only external way to access the two Geo deployments is by HTTPS at
`gitlab.us.example.com` and `gitlab.eu.example.com` in the example above.
`gitlab.us.example.com` and `gitlab.eu.example.com` in the example above.
> **Note:** The primary and secondary Geo deployments must be able to
NOTE: **Note:**
communicate to each other over HTTPS.
The **primary** and **secondary** Geo deployments must be able to communicate to each other over HTTPS.
## Redis and PostgreSQL High Availability
## Redis and PostgreSQL High Availability
The primary and secondary Redis and PostgreSQL should be configured
The **primary** and **secondary** Redis and PostgreSQL should be configured
for high availability. Because of the additional complexity involved
for high availability. Because of the additional complexity involved
in setting up this configuration for PostgreSQL and Redis
in setting up this configuration for PostgreSQL and Redis,
it is not covered by this Geo HA documentation.
it is not covered by this Geo HA documentation.
For more information about setting up a highly available PostgreSQL cluster and Redis cluster using the omnibus package see the high availability documentation for
For more information about setting up a highly available PostgreSQL cluster and Redis cluster using the omnibus package see the high availability documentation for
...
@@ -35,7 +35,7 @@ For more information about setting up a highly available PostgreSQL cluster and
...
@@ -35,7 +35,7 @@ For more information about setting up a highly available PostgreSQL cluster and
@@ -30,7 +30,7 @@ Implementing Geo provides the following benefits:
...
@@ -30,7 +30,7 @@ Implementing Geo provides the following benefits:
- Reduce from minutes to seconds the time taken for your distributed developers to clone and fetch large repositories and projects.
- Reduce from minutes to seconds the time taken for your distributed developers to clone and fetch large repositories and projects.
- Enable all of your developers to contribute ideas and work in parallel, no matter where they are.
- Enable all of your developers to contribute ideas and work in parallel, no matter where they are.
- Balance the load between your **primary** and **secondary** nodes, or offload your automated tests to the**secondary** node.
- Balance the load between your **primary** and **secondary** nodes, or offload your automated tests to a**secondary** node.
In addition, it:
In addition, it:
...
@@ -42,8 +42,8 @@ In addition, it:
...
@@ -42,8 +42,8 @@ In addition, it:
Geo provides:
Geo provides:
- Read-only **secondary** nodes: Maintain one **primary** GitLab node while still enabling a read-only **secondary** node for each of your distributed teams.
- Read-only **secondary** nodes: Maintain one **primary** GitLab node while still enabling read-only **secondary** nodes for each of your distributed teams.
- Authentication system hooks: The **secondary** node receives all authentication data (like user accounts and logins) from the **primary** instance.
- Authentication system hooks: **Secondary** nodes receives all authentication data (like user accounts and logins) from the **primary** instance.
- An intuitive UI: **Secondary** nodes utilize the same web interface your team has grown accustomed to. In addition, there are visual notifications that block write operations and make it clear that a user is on a **secondary** node.
- An intuitive UI: **Secondary** nodes utilize the same web interface your team has grown accustomed to. In addition, there are visual notifications that block write operations and make it clear that a user is on a **secondary** node.
## How it works
## How it works
...
@@ -75,12 +75,12 @@ The following diagram illustrates the underlying architecture of Geo.
...
@@ -75,12 +75,12 @@ The following diagram illustrates the underlying architecture of Geo.
In this diagram:
In this diagram:
- There is the **primary** node and the details of one **secondary** node.
- There is the **primary** node and the details of one **secondary** node.
- Writes to the database can only be performed on the **primary** node. A **secondary** node receives database
- Writes to the database can only be performed on the **primary** node. A **secondary** node receives database
updates via PostgreSQL streaming replication.
updates via PostgreSQL streaming replication.
- If present, the [LDAP server](#ldap) should be configured to replicate for [Disaster Recovery](../disaster_recovery/index.md) scenarios.
- If present, the [LDAP server](#ldap) should be configured to replicate for [Disaster Recovery](../disaster_recovery/index.md) scenarios.
- A **secondary** node performs different type of synchronizations against the **primary** node, using a special
- A **secondary** node performs different type of synchronizations against the **primary** node, using a special
authorization protected by JWT:
authorization protected by JWT:
- Repositories are cloned/updated via Git over HTTPS.
- Repositories are cloned/updated via Git over HTTPS.
- Attachments, LFS objects, and other files are downloaded via HTTPS using a private API endpoint.
- Attachments, LFS objects, and other files are downloaded via HTTPS using a private API endpoint.
From the perspective of a user performing Git operations:
From the perspective of a user performing Git operations:
...
@@ -89,6 +89,7 @@ From the perspective of a user performing Git operations:
...
@@ -89,6 +89,7 @@ From the perspective of a user performing Git operations:
-**Secondary** nodes are read-only but proxy Git push operations to the **primary** node. This makes **secondary** nodes appear to support push operations themselves.
-**Secondary** nodes are read-only but proxy Git push operations to the **primary** node. This makes **secondary** nodes appear to support push operations themselves.
To simplify the diagram, some necessary components are omitted. Note that:
To simplify the diagram, some necessary components are omitted. Note that:
- Git over SSH requires [`gitlab-shell`](https://gitlab.com/gitlab-org/gitlab-shell) and OpenSSH.
- Git over SSH requires [`gitlab-shell`](https://gitlab.com/gitlab-org/gitlab-shell) and OpenSSH.
- Git over HTTPS required [`gitlab-workhorse`](https://gitlab.com/gitlab-org/gitlab-workhorse).
- Git over HTTPS required [`gitlab-workhorse`](https://gitlab.com/gitlab-org/gitlab-workhorse).
...
@@ -97,7 +98,7 @@ Note that a **secondary** node needs two different PostgreSQL databases:
...
@@ -97,7 +98,7 @@ Note that a **secondary** node needs two different PostgreSQL databases:
- A read-only database instance that streams data from the main GitLab database.
- A read-only database instance that streams data from the main GitLab database.
-[Another database instance](#geo-tracking-database) used internally by the **secondary** node to record what data has been replicated.
-[Another database instance](#geo-tracking-database) used internally by the **secondary** node to record what data has been replicated.
In secondary nodes, there is an additional daemon: [Geo Log Cursor](#geo-log-cursor).
In **secondary** nodes, there is an additional daemon: [Geo Log Cursor](#geo-log-cursor).
## Requirements for running Geo
## Requirements for running Geo
...
@@ -115,12 +116,12 @@ The following are required to run Geo:
...
@@ -115,12 +116,12 @@ The following are required to run Geo:
The following table lists basic ports that must be open between the **primary** and **secondary** nodes for Geo.
The following table lists basic ports that must be open between the **primary** and **secondary** nodes for Geo.
See the full list of ports used by GitLab in [Package defaults](https://docs.gitlab.com/omnibus/package-information/defaults.html)
See the full list of ports used by GitLab in [Package defaults](https://docs.gitlab.com/omnibus/package-information/defaults.html)
...
@@ -134,9 +135,12 @@ If you wish to terminate SSL at the GitLab application server instead, use TCP p
...
@@ -134,9 +135,12 @@ If you wish to terminate SSL at the GitLab application server instead, use TCP p
### LDAP
### LDAP
We recommend that if you use LDAP on your **primary** node, you also set up a secondary LDAP server for the **secondary** node. Otherwise, users will not be able to perform Git operations over HTTP(s) on the **secondary** node using HTTP Basic Authentication. However, Git via SSH and personal access tokens will still work.
We recommend that if you use LDAP on your **primary** node, you also set up secondary LDAP servers on each **secondary** node. Otherwise, users will not be able to perform Git operations over HTTP(s) on the **secondary** node using HTTP Basic Authentication. However, Git via SSH and personal access tokens will still work.
NOTE: **Note:**
It is possible for all **secondary** nodes to share an LDAP server, but additional latency can be an issue. Also, consider what LDAP server will be available in a [disaster recovery](../disaster_recovery/index.md) scenario if a **secondary** node is promoted to be a **primary** node.
Check with your LDAP provider for instructions on how to set up replication. For example, OpenLDAP provides [these instructions](https://www.openldap.org/doc/admin24/replication.html).
Check for instructions on how to set up replication in your LDAP service. Instructions will be different depending on the software or service used. For example, OpenLDAP provides [these instructions](https://www.openldap.org/doc/admin24/replication.html).
### Geo Tracking Database
### Geo Tracking Database
...
@@ -153,7 +157,7 @@ The tracking database requires the `postgres_fdw` extension.
...
@@ -153,7 +157,7 @@ The tracking database requires the `postgres_fdw` extension.
This daemon:
This daemon:
- Reads a log of events replicated by the **primary** node to the secondary database instance.
- Reads a log of events replicated by the **primary** node to the **secondary** database instance.
- Updates the Geo Tracking Database instance with changes that need to be executed.
- Updates the Geo Tracking Database instance with changes that need to be executed.
When something is marked to be updated in the tracking database instance, asynchronous jobs running on the **secondary** node will execute the required operations and update the state.
When something is marked to be updated in the tracking database instance, asynchronous jobs running on the **secondary** node will execute the required operations and update the state.
## What virtualization requirements have been defined for the application?
## What virtualization requirements have been defined for the application?
- Nothing Geo-specific, but everything in GitLab needs to have full
- Nothing Geo-specific, but everything in GitLab needs to have full
functionality in such an environment.
functionality in such an environment.
### What aspects of the product may or may not be hosted via the cloud computing model?
### What aspects of the product may or may not be hosted via the cloud computing model?
- GitLab is “cloud native” and this applies to Geo as much as to the rest of the
- GitLab is “cloud native” and this applies to Geo as much as to the rest of the
product. Deployment in clouds is a common and supported scenario.
product. Deployment in clouds is a common and supported scenario.
## If applicable, what approach(es) to cloud computing will be taken (Managed Hosting versus "Pure" Cloud, a "full machine" approach such as AWS-EC2 versus a "hosted database" approach such as AWS-RDS and Azure, etc)?
## If applicable, what approach(es) to cloud computing will be taken (Managed Hosting versus "Pure" Cloud, a "full machine" approach such as AWS-EC2 versus a "hosted database" approach such as AWS-RDS and Azure, etc)?
...
@@ -184,103 +184,97 @@ product. Deployment in clouds is a common and supported scenario.
...
@@ -184,103 +184,97 @@ product. Deployment in clouds is a common and supported scenario.
### How will database connection strings, encryption keys, and other sensitive components be stored, accessed, and protected from unauthorized detection?
### How will database connection strings, encryption keys, and other sensitive components be stored, accessed, and protected from unauthorized detection?
- There are some Geo-specific values. Some are shared secrets which must be
- There are some Geo-specific values. Some are shared secrets which must be
securely transmitted from the primary to the secondary at setup time. Our
securely transmitted from the **primary** node to the **secondary** node at setup time. Our
documentation recommends transmitting them from the primary to the system
documentation recommends transmitting them from the **primary** node to the system
administrator via SSH, and then back out to the secondary in the same manner.
administrator via SSH, and then back out to the **secondary** node in the same manner.
In particular, this includes the PostgreSQL replication credentials and a secret
In particular, this includes the PostgreSQL replication credentials and a secret
key (`db_key_base`) which is used to decrypt certain columns in the database.
key (`db_key_base`) which is used to decrypt certain columns in the database.
The `db_key_base` secret is stored unencrypted on the filesystem, in
The `db_key_base` secret is stored unencrypted on the filesystem, in
`/etc/gitlab/gitlab-secrets.json`, along with a number of other secrets. There is
`/etc/gitlab/gitlab-secrets.json`, along with a number of other secrets. There is
no at-rest protection for them.
no at-rest protection for them.
## Data Processing
## Data Processing
### What data entry paths does the application support?
### What data entry paths does the application support?
- Data is entered via the web application exposed by GitLab itself. Some data is
- Data is entered via the web application exposed by GitLab itself. Some data is
also entered using system administration commands on the GitLab servers (e.g.,
also entered using system administration commands on the GitLab servers (e.g.,
`gitlab-ctl set-primary-node`).
`gitlab-ctl set-primary-node`).
- Secondaries also receive inputs via PostgreSQL streaming replication from the
-**Secondary** nodes also receive inputs via PostgreSQL streaming replication from the **primary** node.
primary.
### What data output paths does the application support?
### What data output paths does the application support?
-Primaries output via PostgreSQL streaming replication to the secondary.
-**Primary** nodes output via PostgreSQL streaming replication to the **secondary** node.
Otherwise, principally via the web application exposed by GitLab itself, and via
Otherwise, principally via the web application exposed by GitLab itself, and via
SSH `git clone` operations initiated by the end-user.
SSH `git clone` operations initiated by the end-user.
### How does data flow across the application's internal components?
### How does data flow across the application's internal components?
-Secondaries and primaries interact via HTTP/HTTPS (secured with JSON web
-**Secondary** nodes and **primary** nodes interact via HTTP/HTTPS (secured with JSON web
tokens) and via PostgreSQL streaming replication.
tokens) and via PostgreSQL streaming replication.
- Within a primary or secondary, the SSOT is the filesystem and the database
- Within a **primary** node or **secondary** node, the SSOT is the filesystem and the database
(including Geo tracking database on secondary). The various internal components
(including Geo tracking database on **secondary** node). The various internal components
are orchestrated to make alterations to these stores.
are orchestrated to make alterations to these stores.
### What data input validation requirements have been defined?
### What data input validation requirements have been defined?
-Secondaries must have a faithful replication of the primary’s data.
-**Secondary** nodes must have a faithful replication of the **primary** node’s data.
### What data does the application store and how?
### What data does the application store and how?
- Git repositories and files, tracking information related to the them, and the
- Git repositories and files, tracking information related to the them, and the GitLab database contents.
GitLab database contents.
### What data is or may need to be encrypted and what key management requirements have been defined?
### What data is or may need to be encrypted and what key management requirements have been defined?
- Neither primaries or secondaries encrypt Git repository or filesystem data at
- Neither **primary** nodes or **secondary** nodes encrypt Git repository or filesystem data at
rest. A subset of database columns are encrypted at rest using the `db_otp_key`
rest. A subset of database columns are encrypted at rest using the `db_otp_key`.
-a static secret shared across all hosts in a GitLab deployment.
-A static secret shared across all hosts in a GitLab deployment.
- In transit, data should be encrypted, although the application does permit
- In transit, data should be encrypted, although the application does permit
communication to proceed unencrypted. The two main transits are the secondary’s
communication to proceed unencrypted. The two main transits are the **secondary** node’s
replication process for PostgreSQL, and for git repositories/files. Both should
replication process for PostgreSQL, and for git repositories/files. Both should
be protected using TLS, with the keys for that managed via Omnibus per existing
be protected using TLS, with the keys for that managed via Omnibus per existing
configuration for end-user access to GitLab.
configuration for end-user access to GitLab.
### What capabilities exist to detect the leakage of sensitive data?
### What capabilities exist to detect the leakage of sensitive data?
- Comprehensive system logs exist, tracking every connection to GitLab and
- Comprehensive system logs exist, tracking every connection to GitLab and PostgreSQL.
PostgreSQL.
### What encryption requirements have been defined for data in transit - including transmission over WAN, LAN, SecureFTP, or publicly accessible protocols such as http: and https:?
### What encryption requirements have been defined for data in transit - including transmission over WAN, LAN, SecureFTP, or publicly accessible protocols such as http: and https:?
- Data must have the option to be encrypted in transit, and be secure against
- Data must have the option to be encrypted in transit, and be secure against
both passive and active attack (e.g., MITM attacks should not be possible).
both passive and active attack (e.g., MITM attacks should not be possible).
## Access
## Access
### What user privilege levels does the application support?
### What user privilege levels does the application support?
- Geo adds one type of privilege: secondaries can access a special Geo API to
- Geo adds one type of privilege: **secondary** nodes can access a special Geo API to
download files over HTTP/HTTPS, and to clone repositories using HTTP/HTTPS.
download files over HTTP/HTTPS, and to clone repositories using HTTP/HTTPS.
### What user identification and authentication requirements have been defined?
### What user identification and authentication requirements have been defined?
-Geo secondaries identify to Geo primaries via OAuth or JWT authentication
-**Secondary** nodes identify to Geo **primary** nodes via OAuth or JWT authentication
based on the shared database (HTTP access) or a PostgreSQL replication user (for
based on the shared database (HTTP access) or a PostgreSQL replication user (for
database replication). The database replication also requires IP-based access
database replication). The database replication also requires IP-based access
controls to be defined.
controls to be defined.
### What user authorization requirements have been defined?
### What user authorization requirements have been defined?
- Secondaries must only be able to *read* data. They are not currently able to
-**Secondary** nodes must only be able to *read* data. They are not currently able to mutate data on the **primary** node.
mutate data on the primary.
### What session management requirements have been defined?
### What session management requirements have been defined?
- Geo JWTs are defined to last for only two minutes before needing to be
- Geo JWTs are defined to last for only two minutes before needing to be regenerated.
regenerated.
### What access requirements have been defined for URI and Service calls?
### What access requirements have been defined for URI and Service calls?
- A Geo secondary makes many calls to the primary's API. This is how file
-**Secondary** nodes make many calls to the **primary** node's API. This is how file
replication proceeds, for instance. This endpoint is only accessible with a JWT
replication proceeds, for instance. This endpoint is only accessible with a JWT token.
token.
- The **primary** node also makes calls to the **secondary** node to get status information.
- The primary also makes calls to the secondary to get status information.
## Application Monitoring
## Application Monitoring
### What application auditing requirements have been defined? How are audit and debug logs accessed, stored, and secured?
### What application auditing requirements have been defined? How are audit and debug logs accessed, stored, and secured?
- Structured JSON log is written to the filesystem, and can also be ingested
- Structured JSON log is written to the filesystem, and can also be ingested
In order to update the Geo nodes when a new GitLab version is released,
In order to update the Geo nodes when a new GitLab version is released,
all you need to do is update GitLab itself:
all you need to do is update GitLab itself:
1. Log into each node (primary and secondaries)
1. Log into each node (**primary** and **secondary** nodes).
1.[Update GitLab][update]
1.[Update GitLab][update].
1.[Update tracking database on secondary node](#update-tracking-database-on-secondary-node) when
1.[Update tracking database on **secondary** node](#update-tracking-database-on-secondary-node) when
the tracking database is enabled.
the tracking database is enabled.
1.[Test](#check-status-after-updating)primary and secondary nodes, and check version in each.
1.[Test](#check-status-after-updating)**primary** and **secondary** nodes, and check version in each.
## Upgrading to GitLab 10.8
## Upgrading to GitLab 10.8
Before 10.8, broadcast messages would not propagate without flushing the cache on the secondaries.
Before 10.8, broadcast messages would not propagate without flushing the cache on the **secondary** nodes. This has been fixed in 10.8, but requires one last cache flush on each **secondary** node:
This has been fixed in 10.8, but requires one last cache flush on each secondary node:
```bash
```sh
sudo gitlab-rake cache:clear
sudo gitlab-rake cache:clear
```
```
## Upgrading to GitLab 10.6
## Upgrading to GitLab 10.6
In 10.4 we started to recommend that you define a password for database user (`gitlab`).
In 10.4, we started to recommend that you define a password for database user (`gitlab`).
We now require this change as we use this password to enable the Foreign Data Wrapper, as a way to optimize
We now require this change as we use this password to enable the Foreign Data Wrapper, as a way to optimize
the Geo Tracking Database. We are also improving security by disabling the use of **trust**
the Geo Tracking Database. We are also improving security by disabling the use of **trust**
authentication method.
authentication method.
1.**[primary]** Login to your primary node and run:
1.**[primary]** Login to your **primary** node and run:
```bash
```sh
gitlab-ctl pg-password-md5 gitlab
gitlab-ctl pg-password-md5 gitlab
# Enter password: mypassword
# Enter password: mypassword
# Confirm password: mypassword
# Confirm password: mypassword
...
@@ -58,14 +57,14 @@ authentication method.
...
@@ -58,14 +57,14 @@ authentication method.
postgresql['trust_auth_cidr_addresses'] = ['127.0.0.1/32','1.2.3.4/32'] # <- Remove this
postgresql['trust_auth_cidr_addresses'] = ['127.0.0.1/32','1.2.3.4/32'] # <- Remove this
```
```
1.**[primary]** Reconfigure and restart
1.**[primary]** Reconfigure and restart:
```bash
```sh
sudo gitlab-ctl reconfigure
sudo gitlab-ctl reconfigure
sudo gitlab-ctl restart
sudo gitlab-ctl restart
```
```
1.**[secondary]** Login to all secondary nodes and edit `/etc/gitlab/gitlab.rb`:
1.**[secondary]** Login to all **secondary** nodes and edit `/etc/gitlab/gitlab.rb`:
```ruby
```ruby
# Fill with the hash generated by `gitlab-ctl pg-password-md5 gitlab`
# Fill with the hash generated by `gitlab-ctl pg-password-md5 gitlab`
...
@@ -90,35 +89,35 @@ authentication method.
...
@@ -90,35 +89,35 @@ authentication method.
postgresql['trust_auth_cidr_addresses'] = ['127.0.0.1/32','5.6.7.8/32'] # <- Remove this
postgresql['trust_auth_cidr_addresses'] = ['127.0.0.1/32','5.6.7.8/32'] # <- Remove this
```
```
1.**[secondary]** Reconfigure and restart
1.**[secondary]** Reconfigure and restart:
```bash
```sh
sudo gitlab-ctl reconfigure
sudo gitlab-ctl reconfigure
sudo gitlab-ctl restart
sudo gitlab-ctl restart
```
```
## Upgrading to GitLab 10.5
## Upgrading to GitLab 10.5
For Geo Disaster Recovery to work with minimum downtime, your Geo secondary
For Geo Disaster Recovery to work with minimum downtime, your **secondary** node
should use the same set of secrets as the primary. However, setup instructions
should use the same set of secrets as the **primary** node. However, setup instructions
prior to the 10.5 release only synchronized the `db_key_base` secret.
prior to the 10.5 release only synchronized the `db_key_base` secret.
To rectify this error on existing installations, you should **overwrite** the
To rectify this error on existing installations, you should **overwrite** the
contents of `/etc/gitlab/gitlab-secrets.json` on the secondary node with the
contents of `/etc/gitlab/gitlab-secrets.json` on each **secondary** node with the
contents of `/etc/gitlab/gitlab-secrets.json` on the primary node, then run the
contents of `/etc/gitlab/gitlab-secrets.json` on the **primary** node, then run the
following command on the secondary node:
following command on each **secondary** node:
```bash
```sh
sudo gitlab-ctl reconfigure
sudo gitlab-ctl reconfigure
```
```
If you do not perform this step, you may find that two-factor authentication
If you do not perform this step, you may find that two-factor authentication
[is broken following DR](faq.md#i-followed-the-disaster-recovery-instructions-and-now-two-factor-auth-is-broken).
[is broken following DR](faq.md#i-followed-the-disaster-recovery-instructions-and-now-two-factor-auth-is-broken).
To prevent SSH requests to the newly promoted primary node from failing
To prevent SSH requests to the newly promoted **primary** node from failing
due to SSH host key mismatch when updating the primary domain's DNS record
due to SSH host key mismatch when updating the **primary** node domain's DNS record
you should perform the step to [Manually replicate primary SSH host keys][configuration-replicate-ssh] in each
you should perform the step to [Manually replicate **primary** SSH host keys][configuration-replicate-ssh] in each
secondary node.
**secondary** node.
## Upgrading to GitLab 10.4
## Upgrading to GitLab 10.4
...
@@ -145,7 +144,7 @@ file synchronization is **also** insecure in these cases!
...
@@ -145,7 +144,7 @@ file synchronization is **also** insecure in these cases!
Support for TLS-secured PostgreSQL replication has been added. If you are
Support for TLS-secured PostgreSQL replication has been added. If you are
currently using PostgreSQL replication across the open internet without an
currently using PostgreSQL replication across the open internet without an
external means of securing the connection (e.g., a site-to-site VPN), then you
external means of securing the connection (e.g., a site-to-site VPN), then you
should immediately reconfigure your primary and secondary PostgreSQL instances
should immediately reconfigure your **primary** and **secondary** PostgreSQL instances
according to the [updated instructions][database].
according to the [updated instructions][database].
If you *are* securing the connections externally and wish to continue doing so,
If you *are* securing the connections externally and wish to continue doing so,
...
@@ -158,8 +157,8 @@ Support for replicating repositories and wikis over HTTP/HTTPS has been added.
...
@@ -158,8 +157,8 @@ Support for replicating repositories and wikis over HTTP/HTTPS has been added.
Replicating over SSH has been deprecated, and support for this option will be
Replicating over SSH has been deprecated, and support for this option will be
removed in a future release.
removed in a future release.
To switch to HTTP/HTTPS replication, log into the primary node as an admin and visit
To switch to HTTP/HTTPS replication, log into the **primary** node as an admin and visit
**Admin Area > Geo** (`/admin/geo_nodes`). For each secondary listed,
**Admin Area > Geo** (`/admin/geo/nodes`). For each **secondary** node listed,
press the "Edit" button, change the "Repository cloning" setting from
press the "Edit" button, change the "Repository cloning" setting from
"SSH (deprecated)" to "HTTP/HTTPS", and press "Save changes". This should take
"SSH (deprecated)" to "HTTP/HTTPS", and press "Save changes". This should take
effect immediately.
effect immediately.
...
@@ -169,9 +168,9 @@ default setting.
...
@@ -169,9 +168,9 @@ default setting.
After you've verified that HTTP/HTTPS replication is working, you should remove
After you've verified that HTTP/HTTPS replication is working, you should remove
the now-unused SSH keys from your secondaries, as they may cause problems if the
the now-unused SSH keys from your secondaries, as they may cause problems if the
secondary if ever promoted to a primary:
**secondary** node if ever promoted to a **primary** node:
1.**[secondary]** Login to **all** your secondary nodes and run:
1.**[secondary]** Login to **all** your **secondary** nodes and run: