PgBouncer setup must include backups, restore improvements

- PgBouncer setup should mention backups
- Bypass PgBouncer: links to detail about backup/upgrades
- Sequenced information into restore steps for Omnibus
- Added secrets doctor rake task to restore steps

doc/administration/postgresql/pgbouncer.md

@@ -66,6 +66,12 @@ This content has been moved to a [new location](replication_and_failover.md#conf
 
 1. At this point, your instance should connect to the database through PgBouncer. If you are having issues, see the [Troubleshooting](#troubleshooting) section
 
+## Backups
+
+Do not backup or restore GitLab through a PgBouncer connection: this will cause a GitLab outage.
+
+[Read more about this and how to reconfigure backups](../../raketasks/backup_restore.md#backup-and-restore-for-installations-using-pgbouncer).
+
 ## Enable Monitoring
 
 > [Introduced](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/3786) in GitLab 12.0.
@@ -150,7 +156,10 @@ ote_pid | tls
 
 ## Procedure for bypassing PgBouncer
 
-Some database changes have to be done directly, and not through PgBouncer. This includes database restores and GitLab upgrades (because of the database migrations).
+Some database changes have to be done directly, and not through PgBouncer.
+
+Read more about the affected tasks: [database restores](../../raketasks/backup_restore.md#backup-and-restore-for-installations-using-pgbouncer)
+and [GitLab upgrades](https://docs.gitlab.com/omnibus/update/README.html#use-postgresql-ha).
 
 1. To find the primary node, run the following on a database node:
 

doc/administration/postgresql/replication_and_failover.md

@@ -577,6 +577,12 @@ PostgreSQL's `trust_auth_cidr_addresses` in `gitlab.rb` on your database nodes.
 [PgBouncer error `ERROR:  pgbouncer cannot connect to server`](#pgbouncer-error-error-pgbouncer-cannot-connect-to-server)
 in the Troubleshooting section before proceeding.
 
+### Backups
+
+Do not backup or restore GitLab through a PgBouncer connection: this will cause a GitLab outage.
+
+[Read more about this and how to reconfigure backups](../../raketasks/backup_restore.md#backup-and-restore-for-installations-using-pgbouncer).
+
 ### Ensure GitLab is running
 
 At this point, your GitLab instance should be up and running. Verify you're able

doc/raketasks/backup_restore.md

@@ -890,7 +890,7 @@ Restoring repositories:
 Deleting tmp directories...[DONE]
 ```
 
-Next, restore `/home/git/gitlab/.secret` if necessary, as previously mentioned.
+Next, restore `/home/git/gitlab/.secret` if necessary, [as previously mentioned](#restore-prerequisites).
 
 Restart GitLab:
 
@@ -943,8 +943,16 @@ permissions on your Registry directory. This is a [known issue](https://gitlab.c
 On GitLab 12.2 or later, you can use `gitlab-backup restore` to avoid this
 issue.
 
-Next, restore `/etc/gitlab/gitlab-secrets.json` if necessary, as previously
-mentioned.
+If there's a GitLab version mismatch between your backup tar file and the
+installed version of GitLab, the restore command aborts with an error
+message. Install the [correct GitLab version](https://packages.gitlab.com/gitlab/),
+and then try again.
+
+NOTE:
+There is a known issue with restore not working with `pgbouncer`. [Read more about backup and restore with `pgbouncer`](#backup-and-restore-for-installations-using-pgbouncer).
+
+Next, restore `/etc/gitlab/gitlab-secrets.json` if necessary,
+[as previously mentioned](#restore-prerequisites).
 
 Reconfigure, restart and check GitLab:
 
@@ -954,13 +962,13 @@ sudo gitlab-ctl restart
 sudo gitlab-rake gitlab:check SANITIZE=true
 ```
 
-If there's a GitLab version mismatch between your backup tar file and the
-installed version of GitLab, the restore command aborts with an error
-message. Install the [correct GitLab version](https://packages.gitlab.com/gitlab/),
-and then try again.
+On GitLab 13.1 and later, check [database values can be decrypted](../administration/raketasks/doctor.md)
+especially if `/etc/gitlab/gitlab-secrets.json` was restored, or if a different server is
+the target for the restore.
 
-NOTE:
-There is a known issue with restore not working with `pgbouncer`. [Read more about backup and restore with `pgbouncer`](#backup-and-restore-for-installations-using-pgbouncer).
+```shell
+sudo gitlab-rake gitlab:doctor:secrets
+```
 
 ### Restore for Docker image and GitLab Helm chart installations
 
@@ -1068,6 +1076,13 @@ following error message is shown:
 ActiveRecord::StatementInvalid: PG::UndefinedTable
 ```
 
+Each time the GitLab backup runs, GitLab will start generating 500 errors and errors about missing
+tables will [be logged by PostgreSQL](../administration/logs.md#postgresql-logs):
+
+```plaintext
+ERROR: relation "tablename" does not exist at character 123
+```
+
 This happens because the task uses `pg_dump`, which [sets a null search
 path and explicitly includes the schema in every SQL query](https://gitlab.com/gitlab-org/gitlab/-/issues/23211)
 to address [CVE-2018-1058](https://www.postgresql.org/about/news/postgresql-103-968-9512-9417-and-9322-released-1834/).

doc/update/README.md

@@ -195,7 +195,7 @@ However, for this to work there are the following requirements:
    9.3.
 - You have to use [post-deployment
    migrations](../development/post_deployment_migrations.md) (included in
-   zero downtime update steps below).
+   [zero downtime update steps below](#steps)).
 - You are using PostgreSQL. Starting from GitLab 12.1, MySQL is not supported.
 - Multi-node GitLab instance. Single-node instances may experience brief interruptions
   [as services restart (Puma in particular)](https://docs.gitlab.com/omnibus/update/README.html#single-node-deployment).