To check the health of Geo **secondary** nodes, we use a checksum over the list of
Git references and their values. The checksum includes `HEAD`, `heads`, `tags`,
`notes`, and GitLab-specific references to ensure true consistency. If two nodes
have the same checksum, then they definitely hold the same references. We compute
the checksum for every node after every update to make sure that they are all
in sync.
## Repository re-verification
> [Introduced](https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/8550) in GitLab Enterprise Edition 11.6. Available in [GitLab Premium](https://about.gitlab.com/pricing/).
Due to bugs or transient infrastructure failures, it is possible for Git
repositories to change unexpectedly without being marked for verification.
Geo constantly reverifies the repositories to ensure the integrity of the
data. The default and recommended re-verification interval is 7 days, though
an interval as short as 1 day can be set. Shorter intervals reduce risk but
increase load and vice versa.
Navigate to the **Admin Area > Geo** dashboard on the **primary** node, and
click the **Edit** button for the **primary** node to customize the minimum
# Bring a demoted primary node back online **[PREMIUM ONLY]**
After a failover, it is possible to fail back to the demoted **primary** node to
restore your original configuration. This process consists of two steps:
1. Making the old **primary** node a **secondary** node.
1. Promoting a **secondary** node to a **primary** node.
CAUTION: **Caution:**
If you have any doubts about the consistency of the data on this node, we recommend setting it up from scratch.
## Configure the former **primary** node to be a **secondary** node
Since the former **primary** node will be out of sync with the current **primary** node, the first step is to bring the former **primary** node up to date. Note, deletion of data stored on disk like
repositories and uploads will not be replayed when bringing the former **primary** node back
into sync, which may result in increased disk usage.
Alternatively, you can [set up a new **secondary** GitLab instance][setup-geo] to avoid this.
To bring the former **primary** node up to date:
1. SSH into the former **primary** node that has fallen behind.
1. Make sure all the services are up:
```sh
sudo gitlab-ctl start
```
> **Note 1:** If you [disabled the **primary** node permanently][disaster-recovery-disable-primary],
> you need to undo those steps now. For Debian/Ubuntu you just need to run
> `sudo systemctl enable gitlab-runsvdir`. For CentOS 6, you need to install
> the GitLab instance from scratch and set it up as a **secondary** node by
> following [Setup instructions][setup-geo]. In this case, you don't need to follow the next step.
>
> **Note 2:** If you [changed the DNS records](index.md#step-4-optional-updating-the-primary-domain-dns-record)
> for this node during disaster recovery procedure you may need to [block
> all the writes to this node](https://gitlab.com/gitlab-org/gitlab-ee/blob/master/doc/gitlab-geo/planned-failover.md#block-primary-traffic)
> during this procedure.
1.[Setup database replication][database-replication]. Note that in this
case, **primary** node refers to the current **primary** node, and **secondary** node refers to the
former **primary** node.
If you have lost your original **primary** node, follow the
[setup instructions][setup-geo] to set up a new **secondary** node.
## Promote the **secondary** node to **primary** node
When the initial replication is complete and the **primary** node and **secondary** node are
closely in sync, you can do a [planned failover].
## Restore the **secondary** node
If your objective is to have two nodes again, you need to bring your **secondary**
node back online as well by repeating the first step
([configure the former **primary** node to be a **secondary** node](#configure-the-former-primary-node-to-be-a-secondary-node))
If any files differ, replace the content on the **secondary** node with the
content from the **primary** node.
### Ensure Geo replication is up-to-date
The maintenance window won't end until Geo replication and verification is
completely finished. To keep the window as short as possible, you should
ensure these processes are close to 100% as possible during active use.
Navigate to the **Admin Area > Geo** dashboard on the **secondary** node to
review status. Replicated objects (shown in green) should be close to 100%,
and there should be no failures (shown in red). If a large proportion of
objects aren't yet replicated (shown in grey), consider giving the node more
time to complete
If any objects are failing to replicate, this should be investigated before
scheduling the maintenance window. Following a planned failover, anything that
failed to replicate will be **lost**.
You can use the [Geo status API](https://docs.gitlab.com/ee/api/geo_nodes.html#retrieve-project-sync-or-verification-failures-that-occurred-on-the-current-node) to review failed objects and
the reasons for failure.
A common cause of replication failures is the data being missing on the
**primary** node - you can resolve these failures by restoring the data from backup,
or removing references to the missing data.
### Verify the integrity of replicated data
This [content was moved to another location][background-verification].
### Notify users of scheduled maintenance
On the **primary** node, navigate to **Admin Area > Messages**, add a broadcast
message. You can check under **Admin Area > Geo** to estimate how long it
will take to finish syncing. An example message would be:
> A scheduled maintenance will take place at XX:XX UTC. We expect it to take
> less than 1 hour.
## Prevent updates to the **primary** node
Until a [read-only mode][ce-19739] is implemented, updates must be prevented
from happening manually. Note that your **secondary** node still needs read-only
access to the **primary** node during the maintenance window.
1. At the scheduled time, using your cloud provider or your node's firewall, block
all HTTP, HTTPS and SSH traffic to/from the **primary** node, **except** for your IP and
the **secondary** node's IP.
For instance, you might run the following commands on the server(s) making up your **primary** node:
## Enable FDW support for the Geo Tracking Database (improves performance)
##
geo_secondary['db_fdw'] = true
```
For external PostgreSQL instances, see [additional instructions](external_database.md).
If you bring a former **primary** node back online to serve as a **secondary** node, then you also need to remove `roles ['geo_primary_role']` or `geo_primary_role['enable'] = true`.
1. Reconfigure GitLab for the changes to take effect:
```sh
gitlab-ctl reconfigure
```
1. Restart PostgreSQL for the IP change to take effect and reconfigure again:
```sh
gitlab-ctl restart postgresql
gitlab-ctl reconfigure
```
This last reconfigure will provision the FDW configuration and enable it.
### Step 3. Initiate the replication process
Below we provide a script that connects the database on the **secondary** node to
the database on the **primary** node, replicates the database, and creates the
needed files for streaming replication.
The directories used are the defaults that are set up in Omnibus. If you have
changed any defaults or are using a source installation, configure it as you
see fit replacing the directories and paths.
CAUTION: **Warning:**
Make sure to run this on the **secondary** server as it removes all PostgreSQL's
data before running `pg_basebackup`.
1. SSH into your GitLab **secondary** server and login as root:
```sh
sudo -i
```
1. Choose a database-friendly name to use for your **secondary** node to
use as the replication slot name. For example, if your domain is
`secondary.geo.example.com`, you may use `secondary_example` as the slot
name as shown in the commands below.
1. Execute the command below to start a backup/restore and begin the replication
CAUTION: **Warning:** Each Geo **secondary** node must have its own unique replication slot name.
Using the same slot name between two secondaries will break PostgreSQL replication.
```sh
gitlab-ctl replicate-geo-database \
--slot-name=<secondary_node_name> \
--host=<primary_node_ip>
```
When prompted, enter the _plaintext_ password you set up for the `gitlab_replicator`
user in the first step.
This command also takes a number of additional options. You can use `--help`
to list them all, but here are a couple of tips:
- If PostgreSQL is listening on a non-standard port, add `--port=` as well.
- If your database is too large to be transferred in 30 minutes, you will need
to increase the timeout, e.g., `--backup-timeout=3600` if you expect the
initial replication to take under an hour.
- Pass `--sslmode=disable` to skip PostgreSQL TLS authentication altogether
(e.g., you know the network path is secure, or you are using a site-to-site
VPN). This is **not** safe over the public Internet!
- You can read more details about each `sslmode` in the
[PostgreSQL documentation][pg-docs-ssl];
the instructions above are carefully written to ensure protection against
both passive eavesdroppers and active "man-in-the-middle" attackers.
- Change the `--slot-name` to the name of the replication slot
to be used on the **primary** database. The script will attempt to create the
replication slot automatically if it does not exist.
- If you're repurposing an old server into a Geo **secondary** node, you'll need to
add `--force` to the command line.
- When not in a production machine you can disable backup step if you
really sure this is what you want by adding `--skip-backup`
The replication process is now complete.
## PGBouncer support (optional)
[PGBouncer](http://pgbouncer.github.io/) may be used with GitLab Geo to pool
PostgreSQL connections. We recommend using PGBouncer if you use GitLab in a
high-availability configuration with a cluster of nodes supporting a Geo
**primary** node and another cluster of nodes supporting a Geo **secondary** node. For more
information, see the [Omnibus HA](https://docs.gitlab.com/ee/administration/high_availability/database.html#configure-using-omnibus-for-high-availability)
documentation.
For a Geo **secondary** node to work properly with PGBouncer in front of the database,
it will need a separate read-only user to make [PostgreSQL FDW queries][FDW]
work:
1. On the **primary** Geo database, enter the PostgreSQL on the console as an
admin user. If you are using an Omnibus-managed database, log onto the **primary**
node that is running the PostgreSQL database (the default Omnibus database name is gitlabhq_production):
```sh
sudo \
-u gitlab-psql /opt/gitlab/embedded/bin/psql \
-h /var/opt/gitlab/postgresql gitlabhq_production
```
1. Then create the read-only user:
```sql
-- NOTE: Use the password defined earlier
CREATE USER gitlab_geo_fdw WITH password 'mypassword';
GRANT CONNECT ON DATABASE gitlabhq_production to gitlab_geo_fdw;
GRANT USAGE ON SCHEMA public TO gitlab_geo_fdw;
GRANT SELECT ON ALL TABLES IN SCHEMA public TO gitlab_geo_fdw;
GRANT SELECT ON ALL SEQUENCES IN SCHEMA public TO gitlab_geo_fdw;
-- Tables created by "gitlab" should be made read-only for "gitlab_geo_fdw"
-- automatically.
ALTER DEFAULT PRIVILEGES FOR USER gitlab IN SCHEMA public GRANT SELECT ON TABLES TO gitlab_geo_fdw;
ALTER DEFAULT PRIVILEGES FOR USER gitlab IN SCHEMA public GRANT SELECT ON SEQUENCES TO gitlab_geo_fdw;
```
1. On the **secondary** nodes, change `/etc/gitlab/gitlab.rb`:
1. Restart PostgreSQL for the changes to take effect.
1. Choose a database-friendly name to use for your secondary to use as the
replication slot name. For example, if your domain is
`secondary.geo.example.com`, you may use `secondary_example` as the slot
name.
1. Create the replication slot on the **primary** node:
```sh
$ sudo -u postgres psql -c "SELECT * FROM pg_create_physical_replication_slot('secondary_example');"
slot_name | xlog_position
------------------+---------------
secondary_example |
(1 row)
```
1. Now that the PostgreSQL server is set up to accept remote connections, run
`netstat -plnt` to make sure that PostgreSQL is listening to the server's
public IP.
### Step 2. Configure the secondary server
Follow the first steps in ["configure the secondary server"][database-replication] and note that since you are installing from source, the username and
group listed as `gitlab-psql` in those steps should be replaced by `postgres`
instead. After completing the "Test that the `gitlab-psql` user can connect to
the **primary** node's database" step, continue here:
1. Edit `postgresql.conf` to configure the secondary for streaming replication
(for Debian/Ubuntu that would be `/etc/postgresql/9.*/main/postgresql.conf`):
```sh
wal_level = hot_standby
max_wal_senders = 5
checkpoint_segments = 10
wal_keep_segments = 10
hot_standby = on
```
1. Restart PostgreSQL for the changes to take effect.
#### Enable tracking database on the secondary server
Geo secondary nodes use a tracking database to keep track of replication status
and recover automatically from some replication issues. Follow the steps below to create
the tracking database.
1. On the secondary node, run the following command to create `database_geo.yml` with the
information of your secondary PostgreSQL instance:
The requirements are listed [on the index page](index.md#requirements-for-running-geo)
## How does Geo know which projects to sync?
On each **secondary** node, there is a read-only replicated copy of the GitLab database.
A **secondary** node also has a tracking database where it stores which projects have been synced.
Geo compares the two databases to find projects that are not yet tracked.
At the start, this tracking database is empty, so Geo will start trying to update from every project that it can see in the GitLab database.
For each project to sync:
1. Geo will issue a `git fetch geo --mirror` to get the latest information from the **primary** node.
If there are no changes, the sync will be fast and end quickly. Otherwise, it will pull the latest commits.
1. The **secondary** node will update the tracking database to store the fact that it has synced projects A, B, C, etc.
1. Repeat until all projects are synced.
When someone pushes a commit to the **primary** node, it generates an event in the GitLab database that the repository has changed.
The **secondary** node sees this event, marks the project in question as dirty, and schedules the project to be resynced.
To ensure that problems with pipelines (for example, syncs failing too many times or jobs being lost) don't permanently stop projects syncing, Geo also periodically checks the tracking database for projects that are marked as dirty. This check happens when
the number of concurrent syncs falls below `repos_max_capacity` and there are no new projects waiting to be synced.
Geo also has a checksum feature which runs a SHA256 sum across all the Git references to the SHA values.
If the refs don't match between the **primary** node and the **secondary** node, then the **secondary** node will mark that project as dirty and try to resync it.
So even if we have an outdated tracking database, the validation should activate and find discrepancies in the repository state and resync.
## Can I use Geo in a disaster recovery situation?
Yes, but there are limitations to what we replicate (see
[What data is replicated to a **secondary** node?](#what-data-is-replicated-to-a-secondary-node)).
Read the documentation for [Disaster Recovery](../disaster_recovery/index.md).
## What data is replicated to a **secondary** node?
We currently replicate project repositories, LFS objects, generated
attachments / avatars and the whole database. This means user accounts,
issues, merge requests, groups, project data, etc., will be available for
query.
## Can I git push to a **secondary** node?
Yes! Pushing directly to a **secondary** node (for both HTTP and SSH, including git-lfs) was [introduced](https://about.gitlab.com/2018/09/22/gitlab-11-3-released/) in [GitLab Premium](https://about.gitlab.com/pricing/#self-managed) 11.3.
## How long does it take to have a commit replicated to a **secondary** node?
All replication operations are asynchronous and are queued to be dispatched. Therefore, it depends on a lot of
factors including the amount of traffic, how big your commit is, the
connectivity between your nodes, your hardware, etc.
## What if the SSH server runs at a different port?
That's totally fine. We use HTTP(s) to fetch repository changes from the **primary** node to all **secondary** nodes.
## Is this possible to set up a Docker Registry for a **secondary** node that mirrors the one on the **primary** node?
Yes. See [Docker Registry for a **secondary** node](docker_registry.md).
The topology above assumes that the **primary** and **secondary** Geo clusters
are located in two separate locations, on their own virtual network
with private IP addresses. The network is configured such that all machines within
one geographic location can communicate with each other using their private IP addresses.
The IP addresses given are examples and may be different depending on the
network topology of your deployment.
The only external way to access the two Geo deployments is by HTTPS at
`gitlab.us.example.com` and `gitlab.eu.example.com` in the example above.
NOTE: **Note:**
The **primary** and **secondary** Geo deployments must be able to communicate to each other over HTTPS.
## Redis and PostgreSQL High Availability
The **primary** and **secondary** Redis and PostgreSQL should be configured
for high availability. Because of the additional complexity involved
in setting up this configuration for PostgreSQL and Redis,
it is not covered by this Geo HA documentation.
For more information about setting up a highly available PostgreSQL cluster and Redis cluster using the omnibus package see the high availability documentation for
[PostgreSQL](../../high_availability/database.md) and
Geo is the solution for widely distributed development teams.
## Overview
Fetching large repositories can take a long time for teams located far from a single GitLab instance.
Geo provides local, read-only instances of your GitLab instances, reducing the time it takes to clone and fetch large repositories and speeding up development.
> - Geo is part of [GitLab Premium](https://about.gitlab.com/pricing/#self-managed).
> - Introduced in GitLab Enterprise Edition 8.9.
> - We recommend you use:
> - At least GitLab Enterprise Edition 10.0 for basic Geo features.
> - The latest version for a better experience.
> - Make sure that all nodes run the same GitLab version.
> - Geo requires PostgreSQL 9.6 and Git 2.9, in addition to GitLab's usual [minimum requirements](../../../install/requirements.md).
> - Using Geo in combination with [High Availability](../../high_availability/README.md) is considered **Generally Available** (GA) in GitLab [GitLab Premium](https://about.gitlab.com/pricing/) 10.4.
For a video introduction to Geo, see [Introduction to GitLab Geo - GitLab Features](https://www.youtube.com/watch?v=-HDLxSjEh6w).
CAUTION: **Caution:**
Geo undergoes significant changes from release to release. Upgrades **are** supported and [documented](#updating-geo), but you should ensure that you're using the right version of the documentation for your installation.
To make sure you're using the right version of the documentation, navigate to [the source version of this page on GitLab.com](https://gitlab.com/gitlab-org/gitlab-ee/blob/master/doc/administration/geo/replication/index.md) and choose the appropriate release from the **Switch branch/tag** dropdown. For example, [`v11.2.3-ee`](https://gitlab.com/gitlab-org/gitlab-ee/blob/v11.2.3-ee/doc/administration/geo/replication/index.md).
## Use cases
Implementing Geo provides the following benefits:
- Reduce from minutes to seconds the time taken for your distributed developers to clone and fetch large repositories and projects.
- Enable all of your developers to contribute ideas and work in parallel, no matter where they are.
- Balance the load between your **primary** and **secondary** nodes, or offload your automated tests to a **secondary** node.
In addition, it:
- Can be used for cloning and fetching projects, in addition to reading any data available in the GitLab web interface (see [current limitations](#current-limitations)).
- Overcomes slow connections between distant offices, saving time by improving speed for distributed teams.
- Helps reducing the loading time for automated tasks, custom integrations, and internal workflows.
- Can quickly fail over to a **secondary** node in a [disaster recovery](../disaster_recovery/index.md) scenario.
- Allows [planned failover](../disaster_recovery/planned_failover.md) to a **secondary** node.
Geo provides:
- Read-only **secondary** nodes: Maintain one **primary** GitLab node while still enabling read-only **secondary** nodes for each of your distributed teams.
- Authentication system hooks: **Secondary** nodes receives all authentication data (like user accounts and logins) from the **primary** instance.
- An intuitive UI: **Secondary** nodes utilize the same web interface your team has grown accustomed to. In addition, there are visual notifications that block write operations and make it clear that a user is on a **secondary** node.
## How it works
Your Geo instance can be used for cloning and fetching projects, in addition to reading any data. This will make working with large repositories over large distances much faster.
When Geo is enabled, the:
- Original instance is known as the **primary** node.
- Replicated read-only nodes are known as **secondary** nodes.
Keep in mind that:
-**Secondary** nodes talk to the **primary** node to:
- Get user data for logins (API).
- Replicate repositories, LFS Objects, and Attachments (HTTPS + JWT).
- Since GitLab Premium 10.0, the **primary** node no longer talks to **secondary** nodes to notify for changes (API).
- Pushing directly to a **secondary** node (for both HTTP and SSH, including git-lfs) was [introduced](https://about.gitlab.com/2018/09/22/gitlab-11-3-released/) in [GitLab Premium](https://about.gitlab.com/pricing/#self-managed) 11.3.
- There are [limitations](#current-limitations) in the current implementation.
### Architecture
The following diagram illustrates the underlying architecture of Geo.
In this diagram:
- There is the **primary** node and the details of one **secondary** node.
- Writes to the database can only be performed on the **primary** node. A **secondary** node receives database
updates via PostgreSQL streaming replication.
- If present, the [LDAP server](#ldap) should be configured to replicate for [Disaster Recovery](../disaster_recovery/index.md) scenarios.
- A **secondary** node performs different type of synchronizations against the **primary** node, using a special
authorization protected by JWT:
- Repositories are cloned/updated via Git over HTTPS.
- Attachments, LFS objects, and other files are downloaded via HTTPS using a private API endpoint.
From the perspective of a user performing Git operations:
- The **primary** node behaves as a full read-write GitLab instance.
-**Secondary** nodes are read-only but proxy Git push operations to the **primary** node. This makes **secondary** nodes appear to support push operations themselves.
To simplify the diagram, some necessary components are omitted. Note that:
- Git over SSH requires [`gitlab-shell`](https://gitlab.com/gitlab-org/gitlab-shell) and OpenSSH.
- Git over HTTPS required [`gitlab-workhorse`](https://gitlab.com/gitlab-org/gitlab-workhorse).
Note that a **secondary** node needs two different PostgreSQL databases:
- A read-only database instance that streams data from the main GitLab database.
-[Another database instance](#geo-tracking-database) used internally by the **secondary** node to record what data has been replicated.
In **secondary** nodes, there is an additional daemon: [Geo Log Cursor](#geo-log-cursor).
## Requirements for running Geo
The following are required to run Geo:
- An operating system that supports OpenSSH 6.9+ (needed for
[fast lookup of authorized SSH keys in the database](../../operations/fast_ssh_key_lookup.md))
The following operating systems are known to ship with a current version of OpenSSH:
-[CentOS](https://www.centos.org) 7.4+
-[Ubuntu](https://www.ubuntu.com) 16.04+
- PostgreSQL 9.6+ with [FDW](https://www.postgresql.org/docs/9.6/postgres-fdw.html) support and [Streaming Replication](https://wiki.postgresql.org/wiki/Streaming_Replication)
- Git 2.9+
### Firewall rules
The following table lists basic ports that must be open between the **primary** and **secondary** nodes for Geo.
See the full list of ports used by GitLab in [Package defaults](https://docs.gitlab.com/omnibus/package-information/defaults.html)
NOTE: **Note:**
[Web terminal](../../../ci/environments.md#web-terminals) support requires your load balancer to correctly handle WebSocket connections.
When using HTTP or HTTPS proxying, your load balancer must be configured to pass through the `Connection` and `Upgrade` hop-by-hop headers. See the [web terminal](../../integration/terminal.md) integration guide for more details.
NOTE: **Note:**
When using HTTPS protocol for port 443, you will need to add an SSL certificate to the load balancers.
If you wish to terminate SSL at the GitLab application server instead, use TCP protocol.
### LDAP
We recommend that if you use LDAP on your **primary** node, you also set up secondary LDAP servers on each **secondary** node. Otherwise, users will not be able to perform Git operations over HTTP(s) on the **secondary** node using HTTP Basic Authentication. However, Git via SSH and personal access tokens will still work.
NOTE: **Note:**
It is possible for all **secondary** nodes to share an LDAP server, but additional latency can be an issue. Also, consider what LDAP server will be available in a [disaster recovery](../disaster_recovery/index.md) scenario if a **secondary** node is promoted to be a **primary** node.
Check for instructions on how to set up replication in your LDAP service. Instructions will be different depending on the software or service used. For example, OpenLDAP provides [these instructions](https://www.openldap.org/doc/admin24/replication.html).
### Geo Tracking Database
The tracking database instance is used as metadata to control what needs to be updated on the disk of the local instance. For example:
- Download new assets.
- Fetch new LFS Objects.
- Fetch changes from a repository that has recently been updated.
Because the replicated database instance is read-only, we need this additional database instance for each **secondary** node.
The tracking database requires the `postgres_fdw` extension.
### Geo Log Cursor
This daemon:
- Reads a log of events replicated by the **primary** node to the **secondary** database instance.
- Updates the Geo Tracking Database instance with changes that need to be executed.
When something is marked to be updated in the tracking database instance, asynchronous jobs running on the **secondary** node will execute the required operations and update the state.
This new architecture allows GitLab to be resilient to connectivity issues between the nodes. It doesn't matter how long the **secondary** node is disconnected from the **primary** node as it will be able to replay all the events in the correct order and become synchronized with the **primary** node again.
## Setup instructions
These instructions assume you have a working instance of GitLab. They guide you through:
1. Making your existing instance the **primary** node.
1. Adding **secondary** nodes.
CAUTION: **Caution:**
The steps below should be followed in the order they appear. **Make sure the GitLab version is the same on all nodes.**
### Using Omnibus GitLab
If you installed GitLab using the Omnibus packages (highly recommended):
1.[Install GitLab Enterprise Edition](https://about.gitlab.com/installation/) on the server that will serve as the **secondary** node. Do not create an account or log in to the new **secondary** node.
1.[Upload the GitLab License](https://docs.gitlab.com/ee/user/admin_area/license.html) on the **primary** node to unlock Geo. The license must be for [GitLab Premium](https://about.gitlab.com/pricing/) or higher.
1.[Set up the database replication](database.md)(`primary(read-write)<-> secondary (read-only)` topology).
1. [Configure fast lookup of authorized SSH keys in the database](../../operations/fast_ssh_key_lookup.md). This step is required and needs to be done on **both** the **primary** and **secondary** nodes.
1. [Configure GitLab](configuration.md) to set the **primary** and **secondary** nodes.
1. Optional: [Configure a secondary LDAP server](../../auth/ldap.md) for the **secondary** node. See [notes on LDAP](#ldap).
1. [Follow the "Using a Geo Server" guide](using_a_geo_server.md).
### Using GitLab installed from source (Deprecated)
NOTE: **Note:**
In GitLab 11.5, support for using Geo in GitLab source installations was deprecated and will be removed in a future release. Please consider [migrating to GitLab Omnibus install](https://docs.gitlab.com/omnibus/update/convert_to_omnibus.html).
If you installed GitLab from source:
1. [Install GitLab Enterprise Edition](../../../install/installation.md) on the server that will serve as the **secondary** node. Do not create an account or log in to the new **secondary** node.
1. [Upload the GitLab License](https://docs.gitlab.com/ee/user/admin_area/license.html) on the **primary** node to unlock Geo. The license must be for [GitLab Premium](https://about.gitlab.com/pricing/) or higher.
1. [Set up the database replication](database_source.md) (`primary (read-write) <-> secondary (read-only)` topology).
1. [Configure fast lookup of authorized SSH keys in the database](../../operations/fast_ssh_key_lookup.md). Do this step for **both** **primary** and **secondary** nodes.
1. [Configure GitLab](configuration_source.md) to set the **primary** and **secondary** nodes.
1. [Follow the "Using a Geo Server" guide](using_a_geo_server.md).
## Post-installation documentation
After installing GitLab on the **secondary** nodes and performing the initial configuration, see the following documentation for post-installation information.
- [Geo configuration (source)](configuration_source.md). Configuring Geo in GitLab source installations was **deprecated** in GitLab 11.5.
### Updating Geo
For information on how to update your Geo nodes to the latest GitLab version, see [Updating the Geo nodes](updating_the_geo_nodes.md).
### Configuring Geo high availability
For information on configuring Geo for high availability, see [Geo High Availability](high_availability.md).
### Configuring Geo with Object Storage
For information on configuring Geo with object storage, see [Geo with Object storage](object_storage.md).
### Disaster Recovery
For information on using Geo in disaster recovery situations to mitigate data-loss and restore services, see [Disaster Recovery](../disaster_recovery/index.md).
### Replicating the Container Registry
For more information on how to replicate the Container Registry, see [Docker Registry for a **secondary** node](docker_registry.md).
### Security Review
For more information on Geo security, see [Geo security review](security_review.md).
### Tuning Geo
For more information on tuning Geo, see [Tuning Geo](tuning.md).
## Remove Geo node
For more information on removing a Geo node, see [Removing **secondary** Geo nodes](remove_geo_node.md).
## Current limitations
CAUTION: **Caution:**
This list of limitations only reflects the latest version of GitLab. If you are using an older version, extra limitations may be in place.
- Pushing directly to a **secondary** node redirects (for HTTP) or proxies (for SSH) the request to the **primary** node instead of [handling it directly](https://gitlab.com/gitlab-org/gitlab-ee/issues/1381), except when using Git over HTTP with credentials embedded within the URI. For example, `https://user:password@secondary.tld`.
- The **primary** node has to be online for OAuth login to happen. Existing sessions and Git are not affected.
- The installation takes multiple manual steps that together can take about an hour depending on circumstances. We are working on improving this experience. See [gitlab-org/omnibus-gitlab#2978](https://gitlab.com/gitlab-org/omnibus-gitlab/issues/2978) for details.
- Real-time updates of issues/merge requests (for example, via long polling) doesn't work on the **secondary** node.
- [Selective synchronization](configuration.md#selective-synchronization) applies only to files and repositories. Other datasets are replicated to the **secondary** node in full, making it inappropriate for use as an access control mechanism.
- Object pools for forked project deduplication work only on the **primary** node, and are duplicated on the **secondary** node.
- [External merge request diffs](../../merge_request_diffs.md) will not be replicated if they are on-disk, and viewing merge requests will fail. However, external MR diffs in object storage **are** supported. The default configuration (in-database) does work.
### Limitations on replication
Only the following items are replicated to the **secondary** node:
- All database content. For example, snippets, epics, issues, merge requests, groups, and project metadata.
- Project repositories.
- Project wiki repositories.
- User uploads. For example, attachments to issues, merge requests, epics, and avatars.
- CI job artifacts and traces.
DANGER: **DANGER**
Data not on this list is unavailable on the **secondary** node. Failing over without manually replicating data not on this list will cause the data to be **lost**.
### Examples of data not replicated
Take special note that these examples of GitLab features are both:
If you wish to use them on a **secondary** node, or to execute a failover successfully, you will need to replicate their data using some other means.
## Frequently Asked Questions
For answers to common questions, see the [Geo FAQ](faq.md).
## Log files
Since GitLab 9.5, Geo stores structured log messages in a `geo.log` file. For Omnibus installations, this file is at `/var/log/gitlab/gitlab-rails/geo.log`.
This file contains information about when Geo attempts to sync repositories and files. Each line in the file contains a separate JSON entry that can be ingested into Elasticsearch, Splunk, etc.
**Secondary** nodes can be removed from the Geo cluster using the Geo admin page of the **primary** node. To remove a **secondary** node:
1. Navigate to **Admin Area > Geo** (`/admin/geo/nodes`).
1. Click the **Remove** button for the **secondary** node you want to remove.
1. Confirm by clicking **Remove** when the prompt appears.
Once removed from the Geo admin page, you must stop and uninstall the **secondary** node:
1. On the **secondary** node, stop GitLab:
```bash
sudo gitlab-ctl stop
```
1. On the **secondary** node, uninstall GitLab:
```bash
# Stop gitlab and remove its supervision process
sudo gitlab-ctl uninstall
# Debian/Ubuntu
sudo dpkg --remove gitlab-ee
# Redhat/Centos
sudo rpm --erase gitlab-ee
```
Once GitLab has been uninstalled from the **secondary** node, the replication slot must be dropped from the **primary** node's database as follows:
1. On the **primary** node, start a PostgreSQL console session:
```bash
sudo gitlab-psql
```
NOTE: **Note:**
Using `gitlab-rails dbconsole` will not work, because managing replication slots requires superuser permissions.
1. Find the name of the relevant replication slot. This is the slot that is specified with `--slot-name` when running the replicate command: `gitlab-ctl replicate-geo-database`.
```sql
SELECT * FROM pg_replication_slots;
```
1. Remove the replication slot for the **secondary** node:
The following security review of the Geo feature set focuses on security
aspects of the feature as they apply to customers running their own GitLab
instances. The review questions are based in part on the [application security architecture](https://www.owasp.org/index.php/Application_Security_Architecture_Cheat_Sheet)
questions from [owasp.org](https://www.owasp.org).
## Business Model
### What geographic areas does the application service?
- This varies by customer. Geo allows customers to deploy to multiple areas,
and they get to choose where they are.
- Region and node selection is entirely manual.
## Data Essentials
### What data does the application receive, produce, and process?
- Geo streams almost all data held by a GitLab instance between sites. This
includes full database replication, most files (user-uploaded attachments,
etc) and repository + wiki data. In a typical configuration, this will
happen across the public Internet, and be TLS-encrypted.
- PostgreSQL replication is TLS-encrypted.
- See also: [only TLSv1.2 should be supported](https://gitlab.com/gitlab-org/omnibus-gitlab/issues/2948)
### How can the data be classified into categories according to its sensitivity?
- GitLab’s model of sensitivity is centered around public vs. internal vs.
private projects. Geo replicates them all indiscriminately. “Selective sync”
exists for files and repositories (but not database content), which would permit
only less-sensitive projects to be replicated to a **secondary** node if desired.
- See also: [developing a data classification policy](https://gitlab.com/gitlab-com/security/issues/4).
### What data backup and retention requirements have been defined for the application?
- Geo is designed to provide replication of a certain subset of the application
data. It is part of the solution, rather than part of the problem.
## End-Users
### Who are the application's end‐users?
-**Secondary** nodes are created in regions that are distant (in terms of
Internet latency) from the main GitLab installation (the **primary** node). They are
intended to be used by anyone who would ordinarily use the **primary** node, who finds
that the **secondary** node is closer to them (in terms of Internet latency).
### How do the end‐users interact with the application?
-**Secondary** nodes provide all the interfaces a **primary** node does
(notably a HTTP/HTTPS web application, and HTTP/HTTPS or SSH git repository
access), but is constrained to read-only activities. The principal use case is
envisioned to be cloning git repositories from the **secondary** node in favor of the
**primary** node, but end-users may use the GitLab web interface to view projects,
issues, merge requests, snippets, etc.
### What security expectations do the end‐users have?
- The replication process must be secure. It would typically be unacceptable to
transmit the entire database contents or all files and repositories across the
public Internet in plaintext, for instance.
-**Secondary** nodes must have the same access controls over its content as the
**primary** node - unauthenticated users must not be able to gain access to privileged
information on the **primary** node by querying the **secondary** node.
- Attackers must not be able to impersonate the **secondary** node to the **primary** node, and
thus gain access to privileged information.
## Administrators
### Who has administrative capabilities in the application?
- Nothing Geo-specific. Any user where `admin: true` is set in the database is
considered an admin with super-user privileges.
- See also: [more granular access control](https://gitlab.com/gitlab-org/gitlab-ce/issues/32730)
(not geo-specific)
- Much of Geo’s integration (database replication, for instance) must be
configured with the application, typically by system administrators.
### What administrative capabilities does the application offer?
-**Secondary** nodes may be added, modified, or removed by users with
administrative access.
- The replication process may be controlled (start/stop) via the Sidekiq
administrative controls.
## Network
### What details regarding routing, switching, firewalling, and load‐balancing have been defined?
- Geo requires the **primary** node and **secondary** node to be able to communicate with each
other across a TCP/IP network. In particular, the **secondary** nodes must be able to
access HTTP/HTTPS and PostgreSQL services on the **primary** node.
### What core network devices support the application?
- Varies from customer to customer.
### What network performance requirements exist?
- Maximum replication speeds between **primary** node and **secondary** node is limited by the
available bandwidth between sites. No hard requirements exist - time to complete
replication (and ability to keep up with changes on the **primary** node) is a function
of the size of the data set, tolerance for latency, and available network
capacity.
### What private and public network links support the application?
- Customers choose their own networks. As sites are intended to be
geographically separated, it is envisioned that replication traffic will pass
over the public Internet in a typical deployment, but this is not a requirement.
## Systems
### What operating systems support the application?
- Geo imposes no additional restrictions on operating system (see the
[GitLab installation](https://about.gitlab.com/installation/) page for more
details), however we recommend using the operating systems listed in the [Geo documentation](index.md#requirements-for-running-geo).
### What details regarding required OS components and lock‐down needs have been defined?
- The recommended installation method (Omnibus) packages most components itself.
A from-source installation method exists. Both are documented at
- There are significant dependencies on the system-installed OpenSSH daemon (Geo
requires users to set up custom authentication methods) and the omnibus or
system-provided PostgreSQL daemon (it must be configured to listen on TCP,
additional users and replication slots must be added, etc).
- The process for dealing with security updates (for example, if there is a
significant vulnerability in OpenSSH or other services, and the customer
wants to patch those services on the OS) is identical to the non-Geo
situation: security updates to OpenSSH would be provided to the user via the
usual distribution channels. Geo introduces no delay there.
## Infrastructure Monitoring
### What network and system performance monitoring requirements have been defined?
- None specific to Geo.
### What mechanisms exist to detect malicious code or compromised application components?
- None specific to Geo.
### What network and system security monitoring requirements have been defined?
- None specific to Geo.
## Virtualization and Externalization
### What aspects of the application lend themselves to virtualization?
- All.
## What virtualization requirements have been defined for the application?
- Nothing Geo-specific, but everything in GitLab needs to have full
functionality in such an environment.
### What aspects of the product may or may not be hosted via the cloud computing model?
- GitLab is “cloud native” and this applies to Geo as much as to the rest of the
product. Deployment in clouds is a common and supported scenario.
## If applicable, what approach(es) to cloud computing will be taken (Managed Hosting versus "Pure" Cloud, a "full machine" approach such as AWS-EC2 versus a "hosted database" approach such as AWS-RDS and Azure, etc)?
- To be decided by our customers, according to their operational needs.
## Environment
### What frameworks and programming languages have been used to create the application?
- Ruby on Rails, Ruby.
### What process, code, or infrastructure dependencies have been defined for the application?
- Nothing specific to Geo.
### What databases and application servers support the application?
- PostgreSQL >= 9.6, Redis, Sidekiq, Unicorn.
### How will database connection strings, encryption keys, and other sensitive components be stored, accessed, and protected from unauthorized detection?
- There are some Geo-specific values. Some are shared secrets which must be
securely transmitted from the **primary** node to the **secondary** node at setup time. Our
documentation recommends transmitting them from the **primary** node to the system
administrator via SSH, and then back out to the **secondary** node in the same manner.
In particular, this includes the PostgreSQL replication credentials and a secret
key (`db_key_base`) which is used to decrypt certain columns in the database.
The `db_key_base` secret is stored unencrypted on the filesystem, in
`/etc/gitlab/gitlab-secrets.json`, along with a number of other secrets. There is
no at-rest protection for them.
## Data Processing
### What data entry paths does the application support?
- Data is entered via the web application exposed by GitLab itself. Some data is
also entered using system administration commands on the GitLab servers (e.g.,
`gitlab-ctl set-primary-node`).
-**Secondary** nodes also receive inputs via PostgreSQL streaming replication from the **primary** node.
### What data output paths does the application support?
-**Primary** nodes output via PostgreSQL streaming replication to the **secondary** node.
Otherwise, principally via the web application exposed by GitLab itself, and via
SSH `git clone` operations initiated by the end-user.
### How does data flow across the application's internal components?
-**Secondary** nodes and **primary** nodes interact via HTTP/HTTPS (secured with JSON web
tokens) and via PostgreSQL streaming replication.
- Within a **primary** node or **secondary** node, the SSOT is the filesystem and the database
(including Geo tracking database on **secondary** node). The various internal components
are orchestrated to make alterations to these stores.
### What data input validation requirements have been defined?
-**Secondary** nodes must have a faithful replication of the **primary** node’s data.
### What data does the application store and how?
- Git repositories and files, tracking information related to the them, and the GitLab database contents.
### What data is or may need to be encrypted and what key management requirements have been defined?
- Neither **primary** nodes or **secondary** nodes encrypt Git repository or filesystem data at
rest. A subset of database columns are encrypted at rest using the `db_otp_key`.
- A static secret shared across all hosts in a GitLab deployment.
- In transit, data should be encrypted, although the application does permit
communication to proceed unencrypted. The two main transits are the **secondary** node’s
replication process for PostgreSQL, and for git repositories/files. Both should
be protected using TLS, with the keys for that managed via Omnibus per existing
configuration for end-user access to GitLab.
### What capabilities exist to detect the leakage of sensitive data?
- Comprehensive system logs exist, tracking every connection to GitLab and PostgreSQL.
### What encryption requirements have been defined for data in transit - including transmission over WAN, LAN, SecureFTP, or publicly accessible protocols such as http: and https:?
- Data must have the option to be encrypted in transit, and be secure against
both passive and active attack (e.g., MITM attacks should not be possible).
## Access
### What user privilege levels does the application support?
- Geo adds one type of privilege: **secondary** nodes can access a special Geo API to
download files over HTTP/HTTPS, and to clone repositories using HTTP/HTTPS.
### What user identification and authentication requirements have been defined?
-**Secondary** nodes identify to Geo **primary** nodes via OAuth or JWT authentication
based on the shared database (HTTP access) or a PostgreSQL replication user (for
database replication). The database replication also requires IP-based access
controls to be defined.
### What user authorization requirements have been defined?
-**Secondary** nodes must only be able to *read* data. They are not currently able to mutate data on the **primary** node.
### What session management requirements have been defined?
- Geo JWTs are defined to last for only two minutes before needing to be regenerated.
- Geo JWTs are generated for one of the following specific scopes:
- Geo API access.
- Git access.
- LFS and File ID.
- Upload and File ID.
- Job Artifact and File ID.
### What access requirements have been defined for URI and Service calls?
-**Secondary** nodes make many calls to the **primary** node's API. This is how file
replication proceeds, for instance. This endpoint is only accessible with a JWT token.
- The **primary** node also makes calls to the **secondary** node to get status information.
## Application Monitoring
### What application auditing requirements have been defined? How are audit and debug logs accessed, stored, and secured?
- Structured JSON log is written to the filesystem, and can also be ingested
Reconfigure in order to recreate the folders and make sure permissions and ownership
are correctly
```sh
gitlab-ctl reconfigure
```
1. Reset the Tracking Database
```sh
gitlab-rake geo:db:reset
```
1. Restart previously stopped services
```sh
gitlab-ctl start
```
## How do I fix a "Foreign Data Wrapper (FDW) is not configured" error?
When setting up Geo, you might see this warning in the `gitlab-rake
gitlab:geo:check` output:
```
GitLab Geo tracking database Foreign Data Wrapper schema is up-to-date? ... foreign data wrapper is not configured
```
There are a few key points to remember:
1. The FDW settings are configured on the Geo **tracking** database.
1. The configured foreign server enables a login to the Geo
**secondary**, read-only database.
By default, the Geo secondary and tracking database are running on the
same host on different ports. That is, 5432 and 5431 respectively.
### Checking configuration
NOTE: **Note:**
The following steps are for Omnibus installs only. Using Geo with source-based installs [is deprecated](index.md#using-gitlab-installed-from-source-deprecated).
To check the configuration:
1. Enter the database console:
```sh
gitlab-geo-psql
```
1. Check whether any tables are present. If everything is working, you
should see something like this:
```sql
gitlabhq_geo_production=# SELECT * from information_schema.foreign_tables;
Depending on which version of Geo you are updating to/from, there may be
different steps.
## General update steps
In order to update the Geo nodes when a new GitLab version is released,
all you need to do is update GitLab itself:
1. Log into each node (**primary** and **secondary** nodes).
1.[Update GitLab][update].
1.[Update tracking database on **secondary** node](#update-tracking-database-on-secondary-node) when
the tracking database is enabled.
1.[Test](#check-status-after-updating)**primary** and **secondary** nodes, and check version in each.
## Upgrading to GitLab 10.8
Before 10.8, broadcast messages would not propagate without flushing the cache on the **secondary** nodes. This has been fixed in 10.8, but requires one last cache flush on each **secondary** node:
```sh
sudo gitlab-rake cache:clear
```
## Upgrading to GitLab 10.6
In 10.4, we started to recommend that you define a password for database user (`gitlab`).
We now require this change as we use this password to enable the Foreign Data Wrapper, as a way to optimize
the Geo Tracking Database. We are also improving security by disabling the use of **trust**
authentication method.
1.**[primary]** Login to your **primary** node and run:
```sh
gitlab-ctl pg-password-md5 gitlab
# Enter password: <your_password_here>
# Confirm password: <your_password_here>
# fca0b89a972d69f00eb3ec98a5838484
```
Copy the generated hash and edit `/etc/gitlab/gitlab.rb`:
```ruby
# Fill with the hash generated by `gitlab-ctl pg-password-md5 gitlab`
Still in the configuration file, locate and remove the `trust_auth_cidr_address`:
```ruby
postgresql['trust_auth_cidr_addresses'] = ['127.0.0.1/32','5.6.7.8/32'] # <- Remove this
```
1.**[secondary]** Reconfigure and restart:
```sh
sudo gitlab-ctl reconfigure
sudo gitlab-ctl restart
```
## Upgrading to GitLab 10.5
For Geo Disaster Recovery to work with minimum downtime, your **secondary** node
should use the same set of secrets as the **primary** node. However, setup instructions
prior to the 10.5 release only synchronized the `db_key_base` secret.
To rectify this error on existing installations, you should **overwrite** the
contents of `/etc/gitlab/gitlab-secrets.json` on each **secondary** node with the
contents of `/etc/gitlab/gitlab-secrets.json` on the **primary** node, then run the
following command on each **secondary** node:
```sh
sudo gitlab-ctl reconfigure
```
If you do not perform this step, you may find that two-factor authentication
[is broken following DR](../disaster_recovery/index.html#i-followed-the-disaster-recovery-instructions-and-now-two-factor-auth-is-broken).
To prevent SSH requests to the newly promoted **primary** node from failing
due to SSH host key mismatch when updating the **primary** node domain's DNS record
you should perform the step to [Manually replicate **primary** SSH host keys](configuration.md#step-2-manually-replicate-the-primary-nodes-ssh-host-keys) in each
**secondary** node.
## Upgrading to GitLab 10.4
There are no Geo-specific steps to take!
## Upgrading to GitLab 10.3
### Support for SSH repository synchronization removed
In GitLab 10.2, synchronizing secondaries over SSH was deprecated. In 10.3,
support is removed entirely. All installations will switch to the HTTP/HTTPS
cloning method instead. Before upgrading, ensure that all your Geo nodes are
configured to use this method and that it works for your installation. In
particular, ensure that [Git access over HTTP/HTTPS is enabled](configuration.md#step-6-enable-git-access-over-httphttps).
Synchronizing repositories over the public Internet using HTTP is insecure, so
you should ensure that you have HTTPS configured before upgrading. Note that
file synchronization is **also** insecure in these cases!
## Upgrading to GitLab 10.2
### Secure PostgreSQL replication
Support for TLS-secured PostgreSQL replication has been added. If you are
currently using PostgreSQL replication across the open internet without an
external means of securing the connection (e.g., a site-to-site VPN), then you
should immediately reconfigure your **primary** and **secondary** PostgreSQL instances
according to the [updated instructions][database].
If you *are* securing the connections externally and wish to continue doing so,
ensure you include the new option `--sslmode=prefer` in future invocations of
`gitlab-ctl replicate-geo-database`.
### HTTPS repository sync
Support for replicating repositories and wikis over HTTP/HTTPS has been added.
Replicating over SSH has been deprecated, and support for this option will be
removed in a future release.
To switch to HTTP/HTTPS replication, log into the **primary** node as an admin and visit
**Admin Area > Geo** (`/admin/geo/nodes`). For each **secondary** node listed,
press the "Edit" button, change the "Repository cloning" setting from
"SSH (deprecated)" to "HTTP/HTTPS", and press "Save changes". This should take
effect immediately.
Any new secondaries should be created using HTTP/HTTPS replication - this is the
default setting.
After you've verified that HTTP/HTTPS replication is working, you should remove
the now-unused SSH keys from your secondaries, as they may cause problems if the
**secondary** node if ever promoted to a **primary** node:
1.**[secondary]** Login to **all** your **secondary** nodes and run:
[//]:#(Please update EE::GitLab::GeoGitAccess::GEO_SERVER_DOCS_URL if this file is moved)
# Using a Geo Server **[PREMIUM ONLY]**
After you set up the [database replication and configure the Geo nodes][req], use your closest GitLab node as you would a normal standalone GitLab instance.
Pushing directly to a **secondary** node (for both HTTP, SSH including git-lfs) was [introduced](https://about.gitlab.com/2018/09/22/gitlab-11-3-released/) in [GitLab Premium](https://about.gitlab.com/pricing/#self-managed) 11.3.
Example of the output you will see when pushing to a **secondary** node:
```bash
$ git push
> GitLab: You're pushing to a Geo secondary.
> GitLab: We'll help you by proxying this request to the primary: ssh://git@primary.geo/user/repo.git
