diff options
Diffstat (limited to 'doc/administration/raketasks')
| -rw-r--r-- | doc/administration/raketasks/check.md | 48 | ||||
| -rw-r--r-- | doc/administration/raketasks/ldap.md | 2 | ||||
| -rw-r--r-- | doc/administration/raketasks/maintenance.md | 12 | ||||
| -rw-r--r-- | doc/administration/raketasks/storage.md | 270 |
4 files changed, 68 insertions, 264 deletions
diff --git a/doc/administration/raketasks/check.md b/doc/administration/raketasks/check.md index 9ced19b53b7..ec28b6bee67 100644 --- a/doc/administration/raketasks/check.md +++ b/doc/administration/raketasks/check.md @@ -283,6 +283,54 @@ I, [2020-06-11T17:18:15.575711 #27148] INFO -- : Done! <!-- vale gitlab.SentenceSpacing = YES --> +## Reset encrypted tokens when they can't be recovered + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/131893) in GitLab 16.6. + +WARNING: +This operation is dangerous and can result in data-loss. Proceed with extreme caution. +You must have knowledge about GitLab internals before you perform this operation. + +In some cases, encrypted tokens can no longer be recovered and cause issues. +Most often, runner registration tokens for groups and projects might be broken on very large instances. + +To reset broken tokens: + +1. Identify the database models that have broken encrypted tokens. For example, it can be `Group` and `Project`. +1. Identify the broken tokens. For example `runners_token`. +1. To reset broken tokens, run `gitlab:doctor:reset_encrypted_tokens` with `VERBOSE=true MODEL_NAMES=Model1,Model2 TOKEN_NAMES=broken_token1,broken_token2`. For example: + + ```shell + VERBOSE=true MODEL_NAMES=Project,Group TOKEN_NAMES=runners_token bundle exec rake gitlab:doctor:reset_encrypted_tokens + ``` + + You will see every action this task would try to perform: + + ```plain + I, [2023-09-26T16:20:23.230942 #88920] INFO -- : Resetting runners_token on Project, Group if they can not be read + I, [2023-09-26T16:20:23.230975 #88920] INFO -- : Executing in DRY RUN mode, no records will actually be updated + D, [2023-09-26T16:20:30.151585 #88920] DEBUG -- : > Fix Project[1].runners_token + I, [2023-09-26T16:20:30.151617 #88920] INFO -- : Checked 1/9 Projects + D, [2023-09-26T16:20:30.151873 #88920] DEBUG -- : > Fix Project[3].runners_token + D, [2023-09-26T16:20:30.152975 #88920] DEBUG -- : > Fix Project[10].runners_token + I, [2023-09-26T16:20:30.152992 #88920] INFO -- : Checked 11/29 Projects + I, [2023-09-26T16:20:30.153230 #88920] INFO -- : Checked 21/29 Projects + I, [2023-09-26T16:20:30.153882 #88920] INFO -- : Checked 29 Projects + D, [2023-09-26T16:20:30.195929 #88920] DEBUG -- : > Fix Group[22].runners_token + I, [2023-09-26T16:20:30.196125 #88920] INFO -- : Checked 1/19 Groups + D, [2023-09-26T16:20:30.196192 #88920] DEBUG -- : > Fix Group[25].runners_token + D, [2023-09-26T16:20:30.197557 #88920] DEBUG -- : > Fix Group[82].runners_token + I, [2023-09-26T16:20:30.197581 #88920] INFO -- : Checked 11/19 Groups + I, [2023-09-26T16:20:30.198455 #88920] INFO -- : Checked 19 Groups + I, [2023-09-26T16:20:30.198462 #88920] INFO -- : Done! + ``` + +1. If you are confident that this operation resets the correct tokens, disable dry-run mode and run the operation again: + + ```shell + DRY_RUN=false VERBOSE=true MODEL_NAMES=Project,Group TOKEN_NAMES=runners_token bundle exec rake gitlab:doctor:reset_encrypted_tokens + ``` + ## Troubleshooting The following are solutions to problems you might discover using the Rake tasks documented diff --git a/doc/administration/raketasks/ldap.md b/doc/administration/raketasks/ldap.md index e6700288631..d8909440ceb 100644 --- a/doc/administration/raketasks/ldap.md +++ b/doc/administration/raketasks/ldap.md @@ -1,6 +1,6 @@ --- stage: Govern -group: Authentication and Authorization +group: Authentication info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- diff --git a/doc/administration/raketasks/maintenance.md b/doc/administration/raketasks/maintenance.md index cdb70ca715b..724dcc2046a 100644 --- a/doc/administration/raketasks/maintenance.md +++ b/doc/administration/raketasks/maintenance.md @@ -454,3 +454,15 @@ main: == [advisory_lock_connection] object_id: 173580, pg_backend_pid: 5532 ``` The messages returned are informational and can be ignored. + +### PostgreSQL socket errors when executing the `gitlab:env:info` Rake task + +After running `sudo gitlab-rake gitlab:env:info` on Gitaly or other non-Rails nodes , you might see the following error: + +```plaintext +PG::ConnectionBad: could not connect to server: No such file or directory +Is the server running locally and accepting +connections on Unix domain socket "/var/opt/gitlab/postgresql/.s.PGSQL.5432"? +``` + +This is because, in a multi-node environment, the `gitlab:env:info` Rake task should only be executed on the nodes running **GitLab Rails**. diff --git a/doc/administration/raketasks/storage.md b/doc/administration/raketasks/storage.md index ce931e78c2b..33ddcd96de6 100644 --- a/doc/administration/raketasks/storage.md +++ b/doc/administration/raketasks/storage.md @@ -1,267 +1,11 @@ --- -stage: Systems -group: Distribution -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments +redirect_to: '../repository_storage_paths.md' +remove_date: '2024-01-11' --- -# Repository storage Rake tasks **(FREE SELF)** +This document was moved to [another location](../repository_storage_paths.md). -This is a collection of Rake tasks to help you list and migrate -existing projects and their attachments to the new -[hashed storage](../repository_storage_paths.md) that GitLab -uses to organize the Git data. - -## List projects and attachments - -The following Rake tasks lists the projects and attachments that are -available on legacy and hashed storage. - -### On legacy storage - -To have a summary and then a list of projects and their attachments using legacy storage: - -- Linux package installations: - - ```shell - # Projects - sudo gitlab-rake gitlab:storage:legacy_projects - sudo gitlab-rake gitlab:storage:list_legacy_projects - - # Attachments - sudo gitlab-rake gitlab:storage:legacy_attachments - sudo gitlab-rake gitlab:storage:list_legacy_attachments - ``` - -- Self-compiled installations: - - ```shell - # Projects - sudo -u git -H bundle exec rake gitlab:storage:legacy_projects RAILS_ENV=production - sudo -u git -H bundle exec rake gitlab:storage:list_legacy_projects RAILS_ENV=production - - # Attachments - sudo -u git -H bundle exec rake gitlab:storage:legacy_attachments RAILS_ENV=production - sudo -u git -H bundle exec rake gitlab:storage:list_legacy_attachments RAILS_ENV=production - ``` - -### On hashed storage - -To have a summary and then a list of projects and their attachments using hashed storage: - -- Linux package installations: - - ```shell - # Projects - sudo gitlab-rake gitlab:storage:hashed_projects - sudo gitlab-rake gitlab:storage:list_hashed_projects - - # Attachments - sudo gitlab-rake gitlab:storage:hashed_attachments - sudo gitlab-rake gitlab:storage:list_hashed_attachments - ``` - -- Self-compiled installations: - - ```shell - # Projects - sudo -u git -H bundle exec rake gitlab:storage:hashed_projects RAILS_ENV=production - sudo -u git -H bundle exec rake gitlab:storage:list_hashed_projects RAILS_ENV=production - - # Attachments - sudo -u git -H bundle exec rake gitlab:storage:hashed_attachments RAILS_ENV=production - sudo -u git -H bundle exec rake gitlab:storage:list_hashed_attachments RAILS_ENV=production - ``` - -## Migrate to hashed storage - -WARNING: -In GitLab 13.0, [hashed storage](../repository_storage_paths.md#hashed-storage) -is enabled by default and the legacy storage is deprecated. -GitLab 14.0 eliminates support for legacy storage. If you're on GitLab -13.0 and later, switching new projects to legacy storage is not possible. -The option to choose between hashed and legacy storage in the Admin Area has -been disabled. - -This task must be run on any machine that has Rails/Sidekiq configured, and the task -schedules all your existing projects and attachments associated with it to be -migrated to the **Hashed** storage type: - -- Linux package installations: - - ```shell - sudo gitlab-rake gitlab:storage:migrate_to_hashed - ``` - -- Self-compiled installations: - - ```shell - sudo -u git -H bundle exec rake gitlab:storage:migrate_to_hashed RAILS_ENV=production - ``` - -If you have any existing integration, you may want to do a small rollout first, -to validate. You can do so by specifying an ID range with the operation by using -the environment variables `ID_FROM` and `ID_TO`. For example, to limit the rollout -to project IDs 50 to 100 in a Linux package installation: - -```shell -sudo gitlab-rake gitlab:storage:migrate_to_hashed ID_FROM=50 ID_TO=100 -``` - -To monitor the progress in GitLab: - -1. On the left sidebar, select **Search or go to**. -1. Select **Admin Area**. -1. On the left sidebar, select **Monitoring > Background Jobs**. -1. Watch how long the `hashed_storage:hashed_storage_project_migrate` queue - takes to finish. After it reaches zero, you can confirm every project - has been migrated by running the commands above. - -If you find it necessary, you can run the previous migration script again to schedule missing projects. - -Any error or warning is logged in Sidekiq's log file. - -If [Geo](../geo/index.md) is enabled, each project that is successfully migrated -generates an event to replicate the changes on any **secondary** nodes. - -You only need the `gitlab:storage:migrate_to_hashed` Rake task to migrate your repositories, but there are -[additional commands](#list-projects-and-attachments) to help you inspect projects and attachments in both legacy and hashed storage. - -## Rollback from hashed storage to legacy storage - -WARNING: -In GitLab 13.0, [hashed storage](../repository_storage_paths.md#hashed-storage) -is enabled by default and the legacy storage is deprecated. -GitLab 14.0 eliminates support for legacy storage. If you're on GitLab -13.0 and later, switching new projects to legacy storage is not possible. -The option to choose between hashed and legacy storage in the Admin Area has -been disabled. - -This task schedules all your existing projects and associated attachments to be rolled back to the -legacy storage type. - -- Linux package installations: - - ```shell - sudo gitlab-rake gitlab:storage:rollback_to_legacy - ``` - -- Self-compiled installations: - - ```shell - sudo -u git -H bundle exec rake gitlab:storage:rollback_to_legacy RAILS_ENV=production - ``` - -If you have any existing integration, you may want to do a small rollback first, -to validate. You can do so by specifying an ID range with the operation by using -the environment variables `ID_FROM` and `ID_TO`. For example, to limit the rollout -to project IDs 50 to 100 in a Linux package installation: - -```shell -sudo gitlab-rake gitlab:storage:rollback_to_legacy ID_FROM=50 ID_TO=100 -``` - -You can monitor the progress in the **Admin Area > Monitoring > Background Jobs** page. -On the **Queues** tab, you can watch the `hashed_storage:hashed_storage_project_rollback` queue to see how long the process takes to finish. - -After it reaches zero, you can confirm every project has been rolled back by running the commands above. -If some projects weren't rolled back, you can run this rollback script again to schedule further rollbacks. -Any error or warning is logged in Sidekiq's log file. - -If you have a Geo setup, the rollback is not reflected automatically -on the **secondary** node. You may need to wait for a backfill operation to kick-in and remove -the remaining repositories from the special `@hashed/` folder manually. - -## Troubleshooting - -The Rake task might not be able to complete the migration to hashed storage. -Checks on the instance will continue to report that there is legacy data: - -```plaintext -* Found 1 projects using Legacy Storage -- janedoe/testproject (id: 1234) -``` - -If you have a subscription, [raise a ticket with GitLab support](https://support.gitlab.com) -as most of the fixes are relatively high risk, involving running code on the Rails console. - -### Read only projects - -If you have set projects as read only they might fail to migrate. - -1. [Start a Rails console](../operations/rails_console.md#starting-a-rails-console-session). - -1. Check if the project is read only: - - ```ruby - project = Project.find_by_full_path('janedoe/testproject') - project.repository_read_only - ``` - -1. If it returns `true` (not `nil` or `false`), set it writable: - - ```ruby - project.update!(repository_read_only: false) - ``` - -1. [Re-run the migration Rake task](#migrate-to-hashed-storage). - -1. Set the project read-only again: - - ```ruby - project.update!(repository_read_only: true) - ``` - -### Projects pending deletion - -Check the project details in the Admin Area. If deleting the project failed -it will show as `Marked For Deletion At ..`, `Scheduled Deletion At ..` and -`pending removal`, but the dates will not be recent. - -Delete the project using the Rails console: - -1. [Start a Rails console](../operations/rails_console.md#starting-a-rails-console-session). - -1. With the following code, select the project to be deleted and account to action it: - - ```ruby - project = Project.find_by_full_path('janedoe/testproject') - user = User.find_by_username('admin_handle') - puts "\nproject selected for deletion is:\nID: #{project.id}\nPATH: #{project.full_path}\nNAME: #{project.name}\n\n" - ``` - - - Replace `janedoe/testproject` with your project path from the Rake take output or from the Admin Area. - - Replace `admin_handle` with the handle of an instance administrator or with `root`. - - Verify the output before proceeding. **There are no other checks performed**. - -1. [Destroy the project](../../user/project/working_with_projects.md#delete-a-project-using-console) **immediately**: - - ```ruby - Projects::DestroyService.new(project, user).execute - ``` - -If destroying the project generates a stack trace relating to encryption or the error `OpenSSL::Cipher::CipherError`: - -1. [Verify your GitLab secrets](check.md#verify-database-values-can-be-decrypted-using-the-current-secrets). - -1. If the affected projects have secrets that cannot be decrypted it will be necessary to remove those specific secrets. - [Our documentation for dealing with lost secrets](../../administration/backup_restore/backup_gitlab.md#when-the-secrets-file-is-lost) - is for loss of all secrets, but it's possible for specific projects to be affected. For example, - to [reset specific runner registration tokens](../../administration/backup_restore/backup_gitlab.md#reset-runner-registration-tokens) - for a specific project ID: - - ```sql - UPDATE projects SET runners_token = null, runners_token_encrypted = null where id = 1234; - ``` - -### `Repository cannot be moved from` errors in Sidekiq log - -Projects might fail to migrate with errors in the Sidekiq log: - -```shell -# grep 'Repository cannot be moved' /var/log/gitlab/sidekiq/current -{"severity":"ERROR","time":"2021-02-29T02:29:02.021Z","message":"Repository cannot be moved from 'janedoe/testproject' to '@hashed<value>' (PROJECT_ID=1234)"} -``` - -This might be caused by [a bug](https://gitlab.com/gitlab-org/gitlab/-/issues/259605) in the original code for hashed storage migration. - -[There is a workaround for projects still affected by this issue](https://gitlab.com/-/snippets/2039252). +<!-- This redirect file can be deleted after <2024-01-11>. --> +<!-- Redirects that point to other docs in the same project expire in three months. --> +<!-- Redirects that point to docs in a different project or site (link is not relative and starts with `https:`) expire in one year. --> +<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/redirects.html --> |