Commit dc0af3dd authored by Ben Prescott's avatar Ben Prescott

PgBouncer setup must include backups, restore improvements

- PgBouncer setup should mention backups
- Bypass PgBouncer: links to detail about backup/upgrades
- Sequenced information into restore steps for Omnibus
- Added secrets doctor rake task to restore steps
parent 08830d77
......@@ -66,6 +66,12 @@ This content has been moved to a [new location](replication_and_failover.md#conf
1. At this point, your instance should connect to the database through PgBouncer. If you are having issues, see the [Troubleshooting](#troubleshooting) section
## Backups
Do not backup or restore GitLab through a PgBouncer connection: this will cause a GitLab outage.
[Read more about this and how to reconfigure backups](../../raketasks/backup_restore.md#backup-and-restore-for-installations-using-pgbouncer).
## Enable Monitoring
> [Introduced](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/3786) in GitLab 12.0.
......@@ -150,7 +156,10 @@ ote_pid | tls
## Procedure for bypassing PgBouncer
Some database changes have to be done directly, and not through PgBouncer. This includes database restores and GitLab upgrades (because of the database migrations).
Some database changes have to be done directly, and not through PgBouncer.
Read more about the affected tasks: [database restores](../../raketasks/backup_restore.md#backup-and-restore-for-installations-using-pgbouncer)
and [GitLab upgrades](https://docs.gitlab.com/omnibus/update/README.html#use-postgresql-ha).
1. To find the primary node, run the following on a database node:
......
......@@ -577,6 +577,12 @@ PostgreSQL's `trust_auth_cidr_addresses` in `gitlab.rb` on your database nodes.
[PgBouncer error `ERROR: pgbouncer cannot connect to server`](#pgbouncer-error-error-pgbouncer-cannot-connect-to-server)
in the Troubleshooting section before proceeding.
### Backups
Do not backup or restore GitLab through a PgBouncer connection: this will cause a GitLab outage.
[Read more about this and how to reconfigure backups](../../raketasks/backup_restore.md#backup-and-restore-for-installations-using-pgbouncer).
### Ensure GitLab is running
At this point, your GitLab instance should be up and running. Verify you're able
......
......@@ -890,7 +890,7 @@ Restoring repositories:
Deleting tmp directories...[DONE]
```
Next, restore `/home/git/gitlab/.secret` if necessary, as previously mentioned.
Next, restore `/home/git/gitlab/.secret` if necessary, [as previously mentioned](#restore-prerequisites).
Restart GitLab:
......@@ -943,8 +943,16 @@ permissions on your Registry directory. This is a [known issue](https://gitlab.c
On GitLab 12.2 or later, you can use `gitlab-backup restore` to avoid this
issue.
Next, restore `/etc/gitlab/gitlab-secrets.json` if necessary, as previously
mentioned.
If there's a GitLab version mismatch between your backup tar file and the
installed version of GitLab, the restore command aborts with an error
message. Install the [correct GitLab version](https://packages.gitlab.com/gitlab/),
and then try again.
NOTE:
There is a known issue with restore not working with `pgbouncer`. [Read more about backup and restore with `pgbouncer`](#backup-and-restore-for-installations-using-pgbouncer).
Next, restore `/etc/gitlab/gitlab-secrets.json` if necessary,
[as previously mentioned](#restore-prerequisites).
Reconfigure, restart and check GitLab:
......@@ -954,13 +962,13 @@ sudo gitlab-ctl restart
sudo gitlab-rake gitlab:check SANITIZE=true
```
If there's a GitLab version mismatch between your backup tar file and the
installed version of GitLab, the restore command aborts with an error
message. Install the [correct GitLab version](https://packages.gitlab.com/gitlab/),
and then try again.
On GitLab 13.1 and later, check [database values can be decrypted](../administration/raketasks/doctor.md)
especially if `/etc/gitlab/gitlab-secrets.json` was restored, or if a different server is
the target for the restore.
NOTE:
There is a known issue with restore not working with `pgbouncer`. [Read more about backup and restore with `pgbouncer`](#backup-and-restore-for-installations-using-pgbouncer).
```shell
sudo gitlab-rake gitlab:doctor:secrets
```
### Restore for Docker image and GitLab Helm chart installations
......@@ -1068,6 +1076,13 @@ following error message is shown:
ActiveRecord::StatementInvalid: PG::UndefinedTable
```
Each time the GitLab backup runs, GitLab will start generating 500 errors and errors about missing
tables will [be logged by PostgreSQL](../administration/logs.md#postgresql-logs):
```plaintext
ERROR: relation "tablename" does not exist at character 123
```
This happens because the task uses `pg_dump`, which [sets a null search
path and explicitly includes the schema in every SQL query](https://gitlab.com/gitlab-org/gitlab/-/issues/23211)
to address [CVE-2018-1058](https://www.postgresql.org/about/news/postgresql-103-968-9512-9417-and-9322-released-1834/).
......
......@@ -195,7 +195,7 @@ However, for this to work there are the following requirements:
9.3.
- You have to use [post-deployment
migrations](../development/post_deployment_migrations.md) (included in
zero downtime update steps below).
[zero downtime update steps below](#steps)).
- You are using PostgreSQL. Starting from GitLab 12.1, MySQL is not supported.
- Multi-node GitLab instance. Single-node instances may experience brief interruptions
[as services restart (Puma in particular)](https://docs.gitlab.com/omnibus/update/README.html#single-node-deployment).
......
Markdown is supported
0%
or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment