Commit dc0af3dd authored by Ben Prescott's avatar Ben Prescott

PgBouncer setup must include backups, restore improvements

- PgBouncer setup should mention backups
- Bypass PgBouncer: links to detail about backup/upgrades
- Sequenced information into restore steps for Omnibus
- Added secrets doctor rake task to restore steps
parent 08830d77
...@@ -66,6 +66,12 @@ This content has been moved to a [new location](replication_and_failover.md#conf ...@@ -66,6 +66,12 @@ This content has been moved to a [new location](replication_and_failover.md#conf
1. At this point, your instance should connect to the database through PgBouncer. If you are having issues, see the [Troubleshooting](#troubleshooting) section 1. At this point, your instance should connect to the database through PgBouncer. If you are having issues, see the [Troubleshooting](#troubleshooting) section
## Backups
Do not backup or restore GitLab through a PgBouncer connection: this will cause a GitLab outage.
[Read more about this and how to reconfigure backups](../../raketasks/backup_restore.md#backup-and-restore-for-installations-using-pgbouncer).
## Enable Monitoring ## Enable Monitoring
> [Introduced](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/3786) in GitLab 12.0. > [Introduced](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/3786) in GitLab 12.0.
...@@ -150,7 +156,10 @@ ote_pid | tls ...@@ -150,7 +156,10 @@ ote_pid | tls
## Procedure for bypassing PgBouncer ## Procedure for bypassing PgBouncer
Some database changes have to be done directly, and not through PgBouncer. This includes database restores and GitLab upgrades (because of the database migrations). Some database changes have to be done directly, and not through PgBouncer.
Read more about the affected tasks: [database restores](../../raketasks/backup_restore.md#backup-and-restore-for-installations-using-pgbouncer)
and [GitLab upgrades](https://docs.gitlab.com/omnibus/update/README.html#use-postgresql-ha).
1. To find the primary node, run the following on a database node: 1. To find the primary node, run the following on a database node:
......
...@@ -577,6 +577,12 @@ PostgreSQL's `trust_auth_cidr_addresses` in `gitlab.rb` on your database nodes. ...@@ -577,6 +577,12 @@ PostgreSQL's `trust_auth_cidr_addresses` in `gitlab.rb` on your database nodes.
[PgBouncer error `ERROR: pgbouncer cannot connect to server`](#pgbouncer-error-error-pgbouncer-cannot-connect-to-server) [PgBouncer error `ERROR: pgbouncer cannot connect to server`](#pgbouncer-error-error-pgbouncer-cannot-connect-to-server)
in the Troubleshooting section before proceeding. in the Troubleshooting section before proceeding.
### Backups
Do not backup or restore GitLab through a PgBouncer connection: this will cause a GitLab outage.
[Read more about this and how to reconfigure backups](../../raketasks/backup_restore.md#backup-and-restore-for-installations-using-pgbouncer).
### Ensure GitLab is running ### Ensure GitLab is running
At this point, your GitLab instance should be up and running. Verify you're able At this point, your GitLab instance should be up and running. Verify you're able
......
...@@ -890,7 +890,7 @@ Restoring repositories: ...@@ -890,7 +890,7 @@ Restoring repositories:
Deleting tmp directories...[DONE] Deleting tmp directories...[DONE]
``` ```
Next, restore `/home/git/gitlab/.secret` if necessary, as previously mentioned. Next, restore `/home/git/gitlab/.secret` if necessary, [as previously mentioned](#restore-prerequisites).
Restart GitLab: Restart GitLab:
...@@ -943,8 +943,16 @@ permissions on your Registry directory. This is a [known issue](https://gitlab.c ...@@ -943,8 +943,16 @@ permissions on your Registry directory. This is a [known issue](https://gitlab.c
On GitLab 12.2 or later, you can use `gitlab-backup restore` to avoid this On GitLab 12.2 or later, you can use `gitlab-backup restore` to avoid this
issue. issue.
Next, restore `/etc/gitlab/gitlab-secrets.json` if necessary, as previously If there's a GitLab version mismatch between your backup tar file and the
mentioned. installed version of GitLab, the restore command aborts with an error
message. Install the [correct GitLab version](https://packages.gitlab.com/gitlab/),
and then try again.
NOTE:
There is a known issue with restore not working with `pgbouncer`. [Read more about backup and restore with `pgbouncer`](#backup-and-restore-for-installations-using-pgbouncer).
Next, restore `/etc/gitlab/gitlab-secrets.json` if necessary,
[as previously mentioned](#restore-prerequisites).
Reconfigure, restart and check GitLab: Reconfigure, restart and check GitLab:
...@@ -954,13 +962,13 @@ sudo gitlab-ctl restart ...@@ -954,13 +962,13 @@ sudo gitlab-ctl restart
sudo gitlab-rake gitlab:check SANITIZE=true sudo gitlab-rake gitlab:check SANITIZE=true
``` ```
If there's a GitLab version mismatch between your backup tar file and the On GitLab 13.1 and later, check [database values can be decrypted](../administration/raketasks/doctor.md)
installed version of GitLab, the restore command aborts with an error especially if `/etc/gitlab/gitlab-secrets.json` was restored, or if a different server is
message. Install the [correct GitLab version](https://packages.gitlab.com/gitlab/), the target for the restore.
and then try again.
NOTE: ```shell
There is a known issue with restore not working with `pgbouncer`. [Read more about backup and restore with `pgbouncer`](#backup-and-restore-for-installations-using-pgbouncer). sudo gitlab-rake gitlab:doctor:secrets
```
### Restore for Docker image and GitLab Helm chart installations ### Restore for Docker image and GitLab Helm chart installations
...@@ -1068,6 +1076,13 @@ following error message is shown: ...@@ -1068,6 +1076,13 @@ following error message is shown:
ActiveRecord::StatementInvalid: PG::UndefinedTable ActiveRecord::StatementInvalid: PG::UndefinedTable
``` ```
Each time the GitLab backup runs, GitLab will start generating 500 errors and errors about missing
tables will [be logged by PostgreSQL](../administration/logs.md#postgresql-logs):
```plaintext
ERROR: relation "tablename" does not exist at character 123
```
This happens because the task uses `pg_dump`, which [sets a null search This happens because the task uses `pg_dump`, which [sets a null search
path and explicitly includes the schema in every SQL query](https://gitlab.com/gitlab-org/gitlab/-/issues/23211) path and explicitly includes the schema in every SQL query](https://gitlab.com/gitlab-org/gitlab/-/issues/23211)
to address [CVE-2018-1058](https://www.postgresql.org/about/news/postgresql-103-968-9512-9417-and-9322-released-1834/). to address [CVE-2018-1058](https://www.postgresql.org/about/news/postgresql-103-968-9512-9417-and-9322-released-1834/).
......
...@@ -195,7 +195,7 @@ However, for this to work there are the following requirements: ...@@ -195,7 +195,7 @@ However, for this to work there are the following requirements:
9.3. 9.3.
- You have to use [post-deployment - You have to use [post-deployment
migrations](../development/post_deployment_migrations.md) (included in migrations](../development/post_deployment_migrations.md) (included in
zero downtime update steps below). [zero downtime update steps below](#steps)).
- You are using PostgreSQL. Starting from GitLab 12.1, MySQL is not supported. - You are using PostgreSQL. Starting from GitLab 12.1, MySQL is not supported.
- Multi-node GitLab instance. Single-node instances may experience brief interruptions - Multi-node GitLab instance. Single-node instances may experience brief interruptions
[as services restart (Puma in particular)](https://docs.gitlab.com/omnibus/update/README.html#single-node-deployment). [as services restart (Puma in particular)](https://docs.gitlab.com/omnibus/update/README.html#single-node-deployment).
......
Markdown is supported
0%
or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment