Commit a46dfff8 authored by Achilleas Pipinellis's avatar Achilleas Pipinellis

Merge branch 'bbodenmiller-master-patch-63776' into 'master'

Merge integrity check Rake task & doctor docs

See merge request gitlab-org/gitlab!77505
parents dfa2694d 0373b085
...@@ -201,6 +201,84 @@ The LDAP check Rake task tests the bind DN and password credentials ...@@ -201,6 +201,84 @@ The LDAP check Rake task tests the bind DN and password credentials
executed as part of the `gitlab:check` task, but can run independently. executed as part of the `gitlab:check` task, but can run independently.
See [LDAP Rake Tasks - LDAP Check](ldap.md#check) for details. See [LDAP Rake Tasks - LDAP Check](ldap.md#check) for details.
## Verify database values can be decrypted using the current secrets
> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/20069) in GitLab 13.1.
This task runs through all possible encrypted values in the
database, verifying that they are decryptable using the current
secrets file (`gitlab-secrets.json`).
Automatic resolution is not yet implemented. If you have values that
cannot be decrypted, you can follow steps to reset them, see our
docs on what to do [when the secrets file is lost](../../raketasks/backup_restore.md#when-the-secrets-file-is-lost).
This can take a very long time, depending on the size of your
database, as it checks all rows in all tables.
**Omnibus Installation**
```shell
sudo gitlab-rake gitlab:doctor:secrets
```
**Source Installation**
```shell
bundle exec rake gitlab:doctor:secrets RAILS_ENV=production
```
**Example output**
```plaintext
I, [2020-06-11T17:17:54.951815 #27148] INFO -- : Checking encrypted values in the database
I, [2020-06-11T17:18:12.677708 #27148] INFO -- : - ApplicationSetting failures: 0
I, [2020-06-11T17:18:12.823692 #27148] INFO -- : - User failures: 0
[...] other models possibly containing encrypted data
I, [2020-06-11T17:18:14.938335 #27148] INFO -- : - Group failures: 1
I, [2020-06-11T17:18:15.559162 #27148] INFO -- : - Operations::FeatureFlagsClient failures: 0
I, [2020-06-11T17:18:15.575533 #27148] INFO -- : - ScimOauthAccessToken failures: 0
I, [2020-06-11T17:18:15.575678 #27148] INFO -- : Total: 1 row(s) affected
I, [2020-06-11T17:18:15.575711 #27148] INFO -- : Done!
```
### Verbose mode
To get more detailed information about which rows and columns can't be
decrypted, you can pass a `VERBOSE` environment variable:
**Omnibus Installation**
```shell
sudo gitlab-rake gitlab:doctor:secrets VERBOSE=1
```
**Source Installation**
```shell
bundle exec rake gitlab:doctor:secrets RAILS_ENV=production VERBOSE=1
```
**Example verbose output**
<!-- vale gitlab.SentenceSpacing = NO -->
```plaintext
I, [2020-06-11T17:17:54.951815 #27148] INFO -- : Checking encrypted values in the database
I, [2020-06-11T17:18:12.677708 #27148] INFO -- : - ApplicationSetting failures: 0
I, [2020-06-11T17:18:12.823692 #27148] INFO -- : - User failures: 0
[...] other models possibly containing encrypted data
D, [2020-06-11T17:19:53.224344 #27351] DEBUG -- : > Something went wrong for Group[10].runners_token: Validation failed: Route can't be blank
I, [2020-06-11T17:19:53.225178 #27351] INFO -- : - Group failures: 1
D, [2020-06-11T17:19:53.225267 #27351] DEBUG -- : - Group[10]: runners_token
I, [2020-06-11T17:18:15.559162 #27148] INFO -- : - Operations::FeatureFlagsClient failures: 0
I, [2020-06-11T17:18:15.575533 #27148] INFO -- : - ScimOauthAccessToken failures: 0
I, [2020-06-11T17:18:15.575678 #27148] INFO -- : Total: 1 row(s) affected
I, [2020-06-11T17:18:15.575711 #27148] INFO -- : Done!
```
<!-- vale gitlab.SentenceSpacing = YES -->
## Troubleshooting ## Troubleshooting
The following are solutions to problems you might discover using the Rake tasks documented The following are solutions to problems you might discover using the Rake tasks documented
......
--- ---
stage: Enablement redirect_to: 'check.md#verify-database-values-can-be-decrypted-using-the-current-secrets'
group: Geo remove_date: '2022-03-04'
info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments
--- ---
# Doctor Rake tasks **(FREE SELF)** This document was moved to [another location](check.md#verify-database-values-can-be-decrypted-using-the-current-secrets).
This is a collection of tasks to help investigate and repair <!-- This redirect file can be deleted after 2022-03-04. -->
problems caused by data integrity issues. <!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/#move-or-rename-a-page -->
## Verify database values can be decrypted using the current secrets
> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/20069) in GitLab 13.1.
This task runs through all possible encrypted values in the
database, verifying that they are decryptable using the current
secrets file (`gitlab-secrets.json`).
Automatic resolution is not yet implemented. If you have values that
cannot be decrypted, you can follow steps to reset them, see our
docs on what to do [when the secrets file is lost](../../raketasks/backup_restore.md#when-the-secrets-file-is-lost).
This can take a very long time, depending on the size of your
database, as it checks all rows in all tables.
**Omnibus Installation**
```shell
sudo gitlab-rake gitlab:doctor:secrets
```
**Source Installation**
```shell
bundle exec rake gitlab:doctor:secrets RAILS_ENV=production
```
**Example output**
```plaintext
I, [2020-06-11T17:17:54.951815 #27148] INFO -- : Checking encrypted values in the database
I, [2020-06-11T17:18:12.677708 #27148] INFO -- : - ApplicationSetting failures: 0
I, [2020-06-11T17:18:12.823692 #27148] INFO -- : - User failures: 0
[...] other models possibly containing encrypted data
I, [2020-06-11T17:18:14.938335 #27148] INFO -- : - Group failures: 1
I, [2020-06-11T17:18:15.559162 #27148] INFO -- : - Operations::FeatureFlagsClient failures: 0
I, [2020-06-11T17:18:15.575533 #27148] INFO -- : - ScimOauthAccessToken failures: 0
I, [2020-06-11T17:18:15.575678 #27148] INFO -- : Total: 1 row(s) affected
I, [2020-06-11T17:18:15.575711 #27148] INFO -- : Done!
```
### Verbose mode
To get more detailed information about which rows and columns can't be
decrypted, you can pass a `VERBOSE` environment variable:
**Omnibus Installation**
```shell
sudo gitlab-rake gitlab:doctor:secrets VERBOSE=1
```
**Source Installation**
```shell
bundle exec rake gitlab:doctor:secrets RAILS_ENV=production VERBOSE=1
```
**Example verbose output**
<!-- vale gitlab.SentenceSpacing = NO -->
```plaintext
I, [2020-06-11T17:17:54.951815 #27148] INFO -- : Checking encrypted values in the database
I, [2020-06-11T17:18:12.677708 #27148] INFO -- : - ApplicationSetting failures: 0
I, [2020-06-11T17:18:12.823692 #27148] INFO -- : - User failures: 0
[...] other models possibly containing encrypted data
D, [2020-06-11T17:19:53.224344 #27351] DEBUG -- : > Something went wrong for Group[10].runners_token: Validation failed: Route can't be blank
I, [2020-06-11T17:19:53.225178 #27351] INFO -- : - Group failures: 1
D, [2020-06-11T17:19:53.225267 #27351] DEBUG -- : - Group[10]: runners_token
I, [2020-06-11T17:18:15.559162 #27148] INFO -- : - Operations::FeatureFlagsClient failures: 0
I, [2020-06-11T17:18:15.575533 #27148] INFO -- : - ScimOauthAccessToken failures: 0
I, [2020-06-11T17:18:15.575678 #27148] INFO -- : Total: 1 row(s) affected
I, [2020-06-11T17:18:15.575711 #27148] INFO -- : Done!
```
<!-- vale gitlab.SentenceSpacing = YES -->
...@@ -120,6 +120,8 @@ You may also have a look at our troubleshooting guides for: ...@@ -120,6 +120,8 @@ You may also have a look at our troubleshooting guides for:
- [GitLab](../index.md#troubleshooting) - [GitLab](../index.md#troubleshooting)
- [Omnibus GitLab](https://docs.gitlab.com/omnibus/index.html#troubleshooting) - [Omnibus GitLab](https://docs.gitlab.com/omnibus/index.html#troubleshooting)
Additionally you should also [verify database values can be decrypted using the current secrets](check.md#verify-database-values-can-be-decrypted-using-the-current-secrets).
To run `gitlab:check`, run: To run `gitlab:check`, run:
**Omnibus Installation** **Omnibus Installation**
......
...@@ -526,7 +526,7 @@ master f05321a5b5728bd8a89b7bf530aa44043c951dce...7d02e575fd790e76 ...@@ -526,7 +526,7 @@ master f05321a5b5728bd8a89b7bf530aa44043c951dce...7d02e575fd790e76
### Find mirrors with "bad decrypt" errors ### Find mirrors with "bad decrypt" errors
This content has been converted to a Rake task, see the [Doctor Rake tasks docs](../raketasks/doctor.md). This content has been converted to a Rake task, see [verify database values can be decrypted using the current secrets](../raketasks/check.md#verify-database-values-can-be-decrypted-using-the-current-secrets).
### Transfer mirror users and tokens to a single service account ### Transfer mirror users and tokens to a single service account
...@@ -1073,7 +1073,7 @@ area on disk. It remains to be seen exactly how or whether the deletion is usefu ...@@ -1073,7 +1073,7 @@ area on disk. It remains to be seen exactly how or whether the deletion is usefu
### Bad Decrypt Script (for encrypted variables) ### Bad Decrypt Script (for encrypted variables)
This content has been converted to a Rake task, see the [Doctor Rake tasks docs](../raketasks/doctor.md). This content has been converted to a Rake task, see [verify database values can be decrypted using the current secrets](../raketasks/check.md#verify-database-values-can-be-decrypted-using-the-current-secrets).
As an example of repairing, if `ProjectImportData Bad count:` is detected and the decision is made to delete the As an example of repairing, if `ProjectImportData Bad count:` is detected and the decision is made to delete the
encrypted credentials to allow manual reentry: encrypted credentials to allow manual reentry:
...@@ -1116,7 +1116,7 @@ gitlab-rails runner /tmp/encrypted-tokens.rb ...@@ -1116,7 +1116,7 @@ gitlab-rails runner /tmp/encrypted-tokens.rb
### Decrypt Script for encrypted tokens ### Decrypt Script for encrypted tokens
This content has been converted to a Rake task, see the [Doctor Rake tasks docs](../raketasks/doctor.md). This content has been converted to a Rake task, see [verify database values can be decrypted using the current secrets](../raketasks/check.md#verify-database-values-can-be-decrypted-using-the-current-secrets).
## Geo ## Geo
......
...@@ -990,7 +990,7 @@ sudo gitlab-ctl restart ...@@ -990,7 +990,7 @@ sudo gitlab-ctl restart
sudo gitlab-rake gitlab:check SANITIZE=true sudo gitlab-rake gitlab:check SANITIZE=true
``` ```
In GitLab 13.1 and later, check [database values can be decrypted](../administration/raketasks/doctor.md) In GitLab 13.1 and later, check [database values can be decrypted](../administration/raketasks/check.md#verify-database-values-can-be-decrypted-using-the-current-secrets)
especially if `/etc/gitlab/gitlab-secrets.json` was restored, or if a different server is especially if `/etc/gitlab/gitlab-secrets.json` was restored, or if a different server is
the target for the restore. the target for the restore.
...@@ -1359,8 +1359,8 @@ Use the information in the following sections at your own risk. ...@@ -1359,8 +1359,8 @@ Use the information in the following sections at your own risk.
#### Verify that all values can be decrypted #### Verify that all values can be decrypted
You can determine if your database contains values that can't be decrypted by using the You can determine if your database contains values that can't be decrypted by using a
[Secrets Doctor Rake task](../administration/raketasks/doctor.md). [Rake task](../administration/raketasks/check.md#verify-database-values-can-be-decrypted-using-the-current-secrets).
#### Take a backup #### Take a backup
......
...@@ -26,7 +26,6 @@ The following Rake tasks are available for use with GitLab: ...@@ -26,7 +26,6 @@ The following Rake tasks are available for use with GitLab:
| [Back up and restore](backup_restore.md) | Back up, restore, and migrate GitLab instances between servers. | | [Back up and restore](backup_restore.md) | Back up, restore, and migrate GitLab instances between servers. |
| [Clean up](cleanup.md) | Clean up unneeded items from GitLab instances. | | [Clean up](cleanup.md) | Clean up unneeded items from GitLab instances. |
| [Development](../development/rake_tasks.md) | Tasks for GitLab contributors. | | [Development](../development/rake_tasks.md) | Tasks for GitLab contributors. |
| [Doctor tasks](../administration/raketasks/doctor.md) | Checks for data integrity issues. |
| [Elasticsearch](../integration/elasticsearch.md#gitlab-advanced-search-rake-tasks) | Maintain Elasticsearch in a GitLab instance. | | [Elasticsearch](../integration/elasticsearch.md#gitlab-advanced-search-rake-tasks) | Maintain Elasticsearch in a GitLab instance. |
| [Enable namespaces](features.md) | Enable usernames and namespaces for user projects. | | [Enable namespaces](features.md) | Enable usernames and namespaces for user projects. |
| [General maintenance](../administration/raketasks/maintenance.md) | General maintenance and self-check tasks. | | [General maintenance](../administration/raketasks/maintenance.md) | General maintenance and self-check tasks. |
...@@ -34,7 +33,7 @@ The following Rake tasks are available for use with GitLab: ...@@ -34,7 +33,7 @@ The following Rake tasks are available for use with GitLab:
| [GitHub import](../administration/raketasks/github_import.md) | Retrieve and import repositories from GitHub. | | [GitHub import](../administration/raketasks/github_import.md) | Retrieve and import repositories from GitHub. |
| [Import repositories](import.md) | Import bare repositories into your GitLab instance. | | [Import repositories](import.md) | Import bare repositories into your GitLab instance. |
| [Import large project exports](../development/import_project.md#importing-via-a-rake-task) | Import large GitLab [project exports](../user/project/settings/import_export.md). | | [Import large project exports](../development/import_project.md#importing-via-a-rake-task) | Import large GitLab [project exports](../user/project/settings/import_export.md). |
| [Integrity checks](../administration/raketasks/check.md) | Check the integrity of repositories, files, and LDAP. | | [Integrity checks](../administration/raketasks/check.md) | Check the integrity of repositories, files, LDAP, and more. |
| [LDAP maintenance](../administration/raketasks/ldap.md) | [LDAP](../administration/auth/ldap/index.md)-related tasks. | | [LDAP maintenance](../administration/raketasks/ldap.md) | [LDAP](../administration/auth/ldap/index.md)-related tasks. |
| [List repositories](list_repos.md) | List all GitLab-managed Git repositories on disk. | | [List repositories](list_repos.md) | List all GitLab-managed Git repositories on disk. |
| [Migrate snippets to Git](migrate_snippets.md) | Migrate GitLab Snippets to Git repositories, and show the migration status. | | [Migrate snippets to Git](migrate_snippets.md) | Migrate GitLab Snippets to Git repositories, and show the migration status. |
......
...@@ -35,7 +35,7 @@ to ensure the major components of GitLab are working: ...@@ -35,7 +35,7 @@ to ensure the major components of GitLab are working:
sudo gitlab-rake gitlab:check sudo gitlab-rake gitlab:check
``` ```
1. Confirm that encrypted database values [can be decrypted](../administration/raketasks/doctor.md#verify-database-values-can-be-decrypted-using-the-current-secrets): 1. Confirm that encrypted database values [can be decrypted](../administration/raketasks/check.md#verify-database-values-can-be-decrypted-using-the-current-secrets):
```shell ```shell
sudo gitlab-rake gitlab:doctor:secrets sudo gitlab-rake gitlab:doctor:secrets
......
Markdown is supported
0%
or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment