diff options
Diffstat (limited to 'doc/administration/raketasks/storage.md')
| -rw-r--r-- | doc/administration/raketasks/storage.md | 100 |
1 files changed, 98 insertions, 2 deletions
diff --git a/doc/administration/raketasks/storage.md b/doc/administration/raketasks/storage.md index 912cf260a03..689ed5c5824 100644 --- a/doc/administration/raketasks/storage.md +++ b/doc/administration/raketasks/storage.md @@ -79,7 +79,7 @@ In GitLab 13.0, [hashed storage](../repository_storage_types.md#hashed-storage) is enabled by default and the legacy storage is deprecated. GitLab 14.0 eliminates support for legacy storage. If you're on GitLab 13.0 and later, switching new projects to legacy storage is not possible. -The option to choose between hashed and legacy storage in the admin area has +The option to choose between hashed and legacy storage in the Admin Area has been disabled. This task must be run on any machine that has Rails/Sidekiq configured, and the task @@ -132,7 +132,7 @@ In GitLab 13.0, [hashed storage](../repository_storage_types.md#hashed-storage) is enabled by default and the legacy storage is deprecated. GitLab 14.0 eliminates support for legacy storage. If you're on GitLab 13.0 and later, switching new projects to legacy storage is not possible. -The option to choose between hashed and legacy storage in the admin area has +The option to choose between hashed and legacy storage in the Admin Area has been disabled. This task schedules all your existing projects and associated attachments to be rolled back to the @@ -169,3 +169,99 @@ Any error or warning is logged in Sidekiq's log file. If you have a Geo setup, the rollback is not reflected automatically on the **secondary** node. You may need to wait for a backfill operation to kick-in and remove the remaining repositories from the special `@hashed/` folder manually. + +## Troubleshooting + +The Rake task might not be able to complete the migration to hashed storage. +Checks on the instance will continue to report that there is legacy data: + +```plaintext +* Found 1 projects using Legacy Storage +- janedoe/testproject (id: 1234) +``` + +If you have a subscription, [raise a ticket with GitLab support](https://support.gitlab.com) +as most of the fixes are relatively high risk, involving running code on the Rails console. + +### Read only projects + +If you have [set projects read only](../troubleshooting/gitlab_rails_cheat_sheet.md#make-a-project-read-only-can-only-be-done-in-the-console) +they might fail to migrate. + +1. [Start a Rails console](../operations/rails_console.md#starting-a-rails-console-session). + +1. Check if the project is read only: + + ```ruby + project = Project.find_by_full_path('janedoe/testproject') + project.repository_read_only + ``` + +1. If it returns `true` (not `nil` or `false`), set it writable: + + ```ruby + project.update!(repository_read_only: false) + ``` + +1. [Re-run the migration Rake task](#migrate-to-hashed-storage). + +1. Set the project read-only again: + + ```ruby + project.update!(repository_read_only: true) + ``` + +### Projects pending deletion + +Check the project details in the Admin Area. If deleting the project failed +it will show as `Marked For Deletion At ..`, `Scheduled Deletion At ..` and +`pending removal`, but the dates will not be recent. + +Delete the project using the Rails console: + +1. [Start a Rails console](../operations/rails_console.md#starting-a-rails-console-session). + +1. With the following code, select the project to be deleted and account to action it: + + ```ruby + project = Project.find_by_full_path('janedoe/testproject') + user = User.find_by_username('admin_handle') + puts "\nproject selected for deletion is:\nID: #{project.id}\nPATH: #{project.full_path}\nNAME: #{project.name}\n\n" + ``` + + - Replace `janedoe/testproject` with your project path from the Rake take output or from the Admin Area. + - Replace `admin_handle` with the handle of an instance administrator or with `root`. + - Verify the output before proceeding. **There are no other checks performed**. + +1. [Destroy the project](../troubleshooting/gitlab_rails_cheat_sheet.md#destroy-a-project) **immediately**: + + ```ruby + Projects::DestroyService.new(project, user).execute + ``` + +If destroying the project generates a stack trace relating to encryption or the error `OpenSSL::Cipher::CipherError`: + +1. [Verify your GitLab secrets](check.md#verify-database-values-can-be-decrypted-using-the-current-secrets). + +1. If the affected projects have secrets that cannot be decrypted it will be necessary to remove those specific secrets. + [Our documentation for dealing with lost secrets](../../raketasks/backup_restore.md#when-the-secrets-file-is-lost) + is for loss of all secrets, but it's possible for specific projects to be affected. For example, + to [reset specific runner registration tokens](../../raketasks/backup_restore.md#reset-runner-registration-tokens) + for a specific project ID: + + ```sql + UPDATE projects SET runners_token = null, runners_token_encrypted = null where id = 1234; + ``` + +### `Repository cannot be moved from` errors in Sidekiq log + +Projects might fail to migrate with errors in the Sidekiq log: + +```shell +# grep 'Repository cannot be moved' /var/log/gitlab/sidekiq/current +{"severity":"ERROR","time":"2021-02-29T02:29:02.021Z","message":"Repository cannot be moved from 'janedoe/testproject' to '@hashed<value>' (PROJECT_ID=1234)"} +``` + +This might be caused by [a bug](https://gitlab.com/gitlab-org/gitlab/-/issues/259605) in the original code for hashed storage migration. + +[There is a workaround for projects still affected by this issue](https://gitlab.com/-/snippets/2039252). |