From cd1cc23153ed8115bc565f62b5a9f4eddc0942ca Mon Sep 17 00:00:00 2001 From: GitLab Bot Date: Wed, 8 Jul 2020 13:55:03 +0000 Subject: Add latest changes from gitlab-org/gitlab@13-1-stable-ee --- doc/administration/raketasks/doctor.md | 84 ++++++++++++++++++++++ .../troubleshooting/gitlab_rails_cheat_sheet.md | 41 +++-------- doc/raketasks/README.md | 1 + doc/raketasks/backup_restore.md | 3 + 4 files changed, 97 insertions(+), 32 deletions(-) create mode 100644 doc/administration/raketasks/doctor.md (limited to 'doc') diff --git a/doc/administration/raketasks/doctor.md b/doc/administration/raketasks/doctor.md new file mode 100644 index 00000000000..2c1b6928663 --- /dev/null +++ b/doc/administration/raketasks/doctor.md @@ -0,0 +1,84 @@ +# Doctor Rake tasks **(CORE ONLY)** + +This is a collection of tasks to help investigate and repair +problems caused by data integrity issues. + +## Verify database values can be decrypted using the current secrets + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/20069) in GitLab 13.1. + +This task runs through all possible encrypted values in the +database, verifying that they are decryptable using the current +secrets file (`gitlab-secrets.json`). + +Automatic resolution is not yet implemented. If you have values that +cannot be decrypted, you can follow steps to reset them, see our +docs on what to do [when the secrets file is lost](../../raketasks/backup_restore.md#when-the-secrets-file-is-lost). + +NOTE: **Note:** +This can take a very long time, depending on the size of your +database, as it checks all rows in all tables. + +**Omnibus Installation** + +```shell +sudo gitlab-rake gitlab:doctor:secrets +``` + +**Source Installation** + +```shell +bundle exec rake gitlab:doctor:secrets RAILS_ENV=production +``` + +**Example output** + + +```plaintext +I, [2020-06-11T17:17:54.951815 #27148] INFO -- : Checking encrypted values in the database +I, [2020-06-11T17:18:12.677708 #27148] INFO -- : - ApplicationSetting failures: 0 +I, [2020-06-11T17:18:12.823692 #27148] INFO -- : - User failures: 0 +[...] other models possibly containing encrypted data +I, [2020-06-11T17:18:14.938335 #27148] INFO -- : - Group failures: 1 +I, [2020-06-11T17:18:15.559162 #27148] INFO -- : - Operations::FeatureFlagsClient failures: 0 +I, [2020-06-11T17:18:15.575533 #27148] INFO -- : - ScimOauthAccessToken failures: 0 +I, [2020-06-11T17:18:15.575678 #27148] INFO -- : Total: 1 row(s) affected +I, [2020-06-11T17:18:15.575711 #27148] INFO -- : Done! +``` + + +### Verbose mode + +In order to get more detailed information about which +rows and columns cannot be decrypted, you can pass a VERBOSE +environment variable: + +**Omnibus Installation** + +```shell +sudo gitlab-rake gitlab:doctor:secrets VERBOSE=1 +``` + +**Source Installation** + +```shell +bundle exec rake gitlab:doctor:secrets RAILS_ENV=production VERBOSE=1 +``` + +**Example verbose output** + + +```plaintext +I, [2020-06-11T17:17:54.951815 #27148] INFO -- : Checking encrypted values in the database +I, [2020-06-11T17:18:12.677708 #27148] INFO -- : - ApplicationSetting failures: 0 +I, [2020-06-11T17:18:12.823692 #27148] INFO -- : - User failures: 0 +[...] other models possibly containing encrypted data +D, [2020-06-11T17:19:53.224344 #27351] DEBUG -- : > Something went wrong for Group[10].runners_token: Validation failed: Route can't be blank +I, [2020-06-11T17:19:53.225178 #27351] INFO -- : - Group failures: 1 +D, [2020-06-11T17:19:53.225267 #27351] DEBUG -- : - Group[10]: runners_token +I, [2020-06-11T17:18:15.559162 #27148] INFO -- : - Operations::FeatureFlagsClient failures: 0 +I, [2020-06-11T17:18:15.575533 #27148] INFO -- : - ScimOauthAccessToken failures: 0 +I, [2020-06-11T17:18:15.575678 #27148] INFO -- : Total: 1 row(s) affected +I, [2020-06-11T17:18:15.575711 #27148] INFO -- : Done! +``` + diff --git a/doc/administration/troubleshooting/gitlab_rails_cheat_sheet.md b/doc/administration/troubleshooting/gitlab_rails_cheat_sheet.md index 33af356b37d..c911c617210 100644 --- a/doc/administration/troubleshooting/gitlab_rails_cheat_sheet.md +++ b/doc/administration/troubleshooting/gitlab_rails_cheat_sheet.md @@ -320,23 +320,7 @@ end ### Find mirrors with "bad decrypt" errors -```ruby -total = 0 -bad = [] -ProjectImportData.find_each do |data| - begin - total += 1 - data.credentials - rescue => e - bad << data - end -end - -puts "Bad count: #{bad.count} / #{total}" -bad.each do |repo| - puts Project.find(repo.project_id).full_path -end; bad.count -``` +This content has been converted to a Rake task, see the [Doctor Rake tasks docs](../raketasks/doctor.md). ### Transfer mirror users and tokens to a single service account @@ -755,18 +739,9 @@ area on disk. It remains to be seen exactly how or whether the deletion is usefu ### Bad Decrypt Script (for encrypted variables) -See . - -This script will go through all the encrypted variables and count how many are not able -to be decrypted. Might be helpful to run on multiple nodes to see which `gitlab-secrets.json` -file is most up to date: - -```shell -wget -O /tmp/bad-decrypt.rb https://gitlab.com/snippets/1730735/raw -gitlab-rails runner /tmp/bad-decrypt.rb -``` +This content has been converted to a Rake task, see the [Doctor Rake tasks docs](../raketasks/doctor.md). -If `ProjectImportData Bad count:` is detected and the decision is made to delete the +As an example of repairing, if `ProjectImportData Bad count:` is detected and the decision is made to delete the encrypted credentials to allow manual reentry: ```ruby @@ -797,16 +772,18 @@ encrypted credentials to allow manual reentry: If `User OTP Secret Bad count:` is detected. For each user listed disable/enable two-factor authentication. -### Decrypt Script for encrypted tokens - -This script will search for all encrypted tokens that are causing decryption errors, -and update or reset as needed: +The following script will search in some of the tables for encrypted tokens that are +causing decryption errors, and update or reset as needed: ```shell wget -O /tmp/encrypted-tokens.rb https://gitlab.com/snippets/1876342/raw gitlab-rails runner /tmp/encrypted-tokens.rb ``` +### Decrypt Script for encrypted tokens + +This content has been converted to a Rake task, see the [Doctor Rake tasks docs](../raketasks/doctor.md). + ## Geo ### Artifacts diff --git a/doc/raketasks/README.md b/doc/raketasks/README.md index 1d8aad25d57..eaaf1ebed99 100644 --- a/doc/raketasks/README.md +++ b/doc/raketasks/README.md @@ -20,6 +20,7 @@ The following are available Rake tasks: | [Back up and restore](backup_restore.md) | Back up, restore, and migrate GitLab instances between servers. | | [Clean up](cleanup.md) | Clean up unneeded items from GitLab instances. | | [Development](../development/rake_tasks.md) | Tasks for GitLab contributors. | +| [Doctor tasks](../administration/raketasks/doctor.md) | Checks for data integrity issues. | | [Elasticsearch](../integration/elasticsearch.md#gitlab-elasticsearch-rake-tasks) **(STARTER ONLY)** | Maintain Elasticsearch in a GitLab instance. | | [Enable namespaces](features.md) | Enable usernames and namespaces for user projects. | | [General maintenance](../administration/raketasks/maintenance.md) | General maintenance and self-check tasks. | diff --git a/doc/raketasks/backup_restore.md b/doc/raketasks/backup_restore.md index e83f51dc84e..18c1cba54f7 100644 --- a/doc/raketasks/backup_restore.md +++ b/doc/raketasks/backup_restore.md @@ -941,6 +941,9 @@ experience some unexpected behavior such as: - Stuck jobs. - 500 errors. +You can check whether you have undecryptable values in the database using +the [Secrets Doctor Rake task](../administration/raketasks/doctor.md). + In this case, you are required to reset all the tokens for CI/CD variables and Runner Authentication, which is described in more detail below. After resetting the tokens, you should be able to visit your project and the jobs -- cgit v1.2.3