Merge branch 'cat-add-doctor-secrets-docs' into 'master'

Document the doctor:secrets rake task

Closes gitlab-com/support/toolbox/snippets#64

See merge request gitlab-org/gitlab!35393

doc/administration/raketasks/doctor.md

+# Doctor Rake tasks **(CORE ONLY)**
+
+This is a collection of tasks to help investigate and repair
+problems caused by data integrity issues.
+
+## Verify database values can be decrypted using the current secrets
+
+> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/20069) in GitLab 13.1.
+
+This task runs through all possible encrypted values in the
+database, verifying that they are decryptable using the current
+secrets file (`gitlab-secrets.json`).
+
+Automatic resolution is not yet implemented. If you have values that
+cannot be decrypted, you can follow steps to reset them, see our
+docs on what to do [when the secrets file is lost](../../raketasks/backup_restore.md#when-the-secrets-file-is-lost).
+
+NOTE: **Note:**
+This can take a very long time, depending on the size of your
+database, as it checks all rows in all tables.
+
+**Omnibus Installation**
+
+```shell
+sudo gitlab-rake gitlab:doctor:secrets
+```
+
+**Source Installation**
+
+```shell
+bundle exec rake gitlab:doctor:secrets RAILS_ENV=production
+```
+
+**Example output**
+
+<!-- vale gitlab.SentenceSpacing = NO -->
+```plaintext
+I, [2020-06-11T17:17:54.951815 #27148]  INFO -- : Checking encrypted values in the database
+I, [2020-06-11T17:18:12.677708 #27148]  INFO -- : - ApplicationSetting failures: 0
+I, [2020-06-11T17:18:12.823692 #27148]  INFO -- : - User failures: 0
+[...] other models possibly containing encrypted data
+I, [2020-06-11T17:18:14.938335 #27148]  INFO -- : - Group failures: 1
+I, [2020-06-11T17:18:15.559162 #27148]  INFO -- : - Operations::FeatureFlagsClient failures: 0
+I, [2020-06-11T17:18:15.575533 #27148]  INFO -- : - ScimOauthAccessToken failures: 0
+I, [2020-06-11T17:18:15.575678 #27148]  INFO -- : Total: 1 row(s) affected
+I, [2020-06-11T17:18:15.575711 #27148]  INFO -- : Done!
+```
+<!-- vale gitlab.SentenceSpacing = YES -->
+
+### Verbose mode
+
+In order to get more detailed information about which
+rows and columns cannot be decrypted, you can pass a VERBOSE
+environment variable:
+
+**Omnibus Installation**
+
+```shell
+sudo gitlab-rake gitlab:doctor:secrets VERBOSE=1
+```
+
+**Source Installation**
+
+```shell
+bundle exec rake gitlab:doctor:secrets RAILS_ENV=production VERBOSE=1
+```
+
+**Example verbose output**
+
+<!-- vale gitlab.SentenceSpacing = NO -->
+```plaintext
+I, [2020-06-11T17:17:54.951815 #27148]  INFO -- : Checking encrypted values in the database
+I, [2020-06-11T17:18:12.677708 #27148]  INFO -- : - ApplicationSetting failures: 0
+I, [2020-06-11T17:18:12.823692 #27148]  INFO -- : - User failures: 0
+[...] other models possibly containing encrypted data
+D, [2020-06-11T17:19:53.224344 #27351] DEBUG -- : > Something went wrong for Group[10].runners_token: Validation failed: Route can't be blank
+I, [2020-06-11T17:19:53.225178 #27351]  INFO -- : - Group failures: 1
+D, [2020-06-11T17:19:53.225267 #27351] DEBUG -- :   - Group[10]: runners_token
+I, [2020-06-11T17:18:15.559162 #27148]  INFO -- : - Operations::FeatureFlagsClient failures: 0
+I, [2020-06-11T17:18:15.575533 #27148]  INFO -- : - ScimOauthAccessToken failures: 0
+I, [2020-06-11T17:18:15.575678 #27148]  INFO -- : Total: 1 row(s) affected
+I, [2020-06-11T17:18:15.575711 #27148]  INFO -- : Done!
+```
+<!-- vale gitlab.SentenceSpacing = YES -->

doc/administration/troubleshooting/gitlab_rails_cheat_sheet.md

@@ -334,23 +334,7 @@ end
 
 ### Find mirrors with "bad decrypt" errors
 
-```ruby
-total = 0
-bad = []
-ProjectImportData.find_each do |data|
-  begin
-    total += 1
-    data.credentials
-  rescue => e
-    bad << data
-  end
-end
-
-puts "Bad count: #{bad.count} / #{total}"
-bad.each do |repo|
-  puts Project.find(repo.project_id).full_path
-end; bad.count
-```
+This content has been converted to a Rake task, see the [Doctor Rake tasks docs](../raketasks/doctor.md).
 
 ### Transfer mirror users and tokens to a single service account
 
@@ -769,18 +753,9 @@ area on disk. It remains to be seen exactly how or whether the deletion is usefu
 
 ### Bad Decrypt Script (for encrypted variables)
 
-See <https://gitlab.com/snippets/1730735/raw>.
-
-This script will go through all the encrypted variables and count how many are not able
-to be decrypted. Might be helpful to run on multiple nodes to see which `gitlab-secrets.json`
-file is most up to date:
-
-```shell
-wget -O /tmp/bad-decrypt.rb https://gitlab.com/snippets/1730735/raw
-gitlab-rails runner /tmp/bad-decrypt.rb
-```
+This content has been converted to a Rake task, see the [Doctor Rake tasks docs](../raketasks/doctor.md).
 
-If `ProjectImportData Bad count:` is detected and the decision is made to delete the
+As an example of repairing, if `ProjectImportData Bad count:` is detected and the decision is made to delete the
 encrypted credentials to allow manual reentry:
 
 ```ruby
@@ -811,16 +786,18 @@ encrypted credentials to allow manual reentry:
 If `User OTP Secret Bad count:` is detected. For each user listed disable/enable
 two-factor authentication.
 
-### Decrypt Script for encrypted tokens
-
-This script will search for all encrypted tokens that are causing decryption errors,
-and update or reset as needed:
+The following script will search in some of the tables for encrypted tokens that are
+causing decryption errors, and update or reset as needed:
 
 ```shell
 wget -O /tmp/encrypted-tokens.rb https://gitlab.com/snippets/1876342/raw
 gitlab-rails runner /tmp/encrypted-tokens.rb
 ```
 
+### Decrypt Script for encrypted tokens
+
+This content has been converted to a Rake task, see the [Doctor Rake tasks docs](../raketasks/doctor.md).
+
 ## Geo
 
 ### Artifacts

doc/raketasks/README.md

@@ -20,6 +20,7 @@ The following are available Rake tasks:
 | [Back up and restore](backup_restore.md)                                                            | Back up, restore, and migrate GitLab instances between servers.                           |
 | [Clean up](cleanup.md)                                                                              | Clean up unneeded items from GitLab instances.                                            |
 | [Development](../development/rake_tasks.md)                                                         | Tasks for GitLab contributors.                                                            |
+| [Doctor tasks](../administration/raketasks/doctor.md)                                               | Checks for data integrity issues.                                                         |
 | [Elasticsearch](../integration/elasticsearch.md#gitlab-elasticsearch-rake-tasks) **(STARTER ONLY)** | Maintain Elasticsearch in a GitLab instance.                                              |
 | [Enable namespaces](features.md)                                                                    | Enable usernames and namespaces for user projects.                                        |
 | [General maintenance](../administration/raketasks/maintenance.md)                                   | General maintenance and self-check tasks.                                                 |

doc/raketasks/backup_restore.md

@@ -963,6 +963,9 @@ experience some unexpected behavior such as:
 - Stuck jobs.
 - 500 errors.
 
+You can check whether you have undecryptable values in the database using
+the [Secrets Doctor Rake task](../administration/raketasks/doctor.md).
+
 In this case, you are required to reset all the tokens for CI/CD variables
 and Runner Authentication, which is described in more detail below. After
 resetting the tokens, you should be able to visit your project and the jobs