From b76ae638462ab0f673e5915986070518dd3f9ad3 Mon Sep 17 00:00:00 2001 From: GitLab Bot Date: Thu, 19 Aug 2021 09:08:42 +0000 Subject: Add latest changes from gitlab-org/gitlab@14-2-stable-ee --- .../operations/extra_sidekiq_processes.md | 52 +-------- .../operations/extra_sidekiq_routing.md | 8 +- .../operations/fast_ssh_key_lookup.md | 2 +- .../operations/moving_repositories.md | 126 ++++++++++++++++++++- 4 files changed, 133 insertions(+), 55 deletions(-) (limited to 'doc/administration/operations') diff --git a/doc/administration/operations/extra_sidekiq_processes.md b/doc/administration/operations/extra_sidekiq_processes.md index 1f195bcc378..2cc4e3a4551 100644 --- a/doc/administration/operations/extra_sidekiq_processes.md +++ b/doc/administration/operations/extra_sidekiq_processes.md @@ -95,14 +95,14 @@ To view the Sidekiq processes in GitLab: ## Negate settings -To have the additional Sidekiq processes work on every queue **except** the ones +To have the Sidekiq process work on every queue **except** the ones you list. In this example, we exclude all import-related jobs from a Sidekiq node: -1. After you follow the steps for [starting extra processes](#start-multiple-processes), - edit `/etc/gitlab/gitlab.rb` and add: +1. Edit `/etc/gitlab/gitlab.rb` and add: ```ruby sidekiq['negate'] = true + sidekiq['queue_selector'] = true sidekiq['queue_groups'] = [ "feature_category=importers" ] @@ -140,50 +140,6 @@ sidekiq['queue_groups'] = [ ] ``` -### Disable Sidekiq cluster - -WARNING: -Sidekiq cluster is [scheduled](https://gitlab.com/gitlab-com/gl-infra/scalability/-/issues/240) -to be the only way to start Sidekiq in GitLab 14.0. - -By default, the Sidekiq service runs `sidekiq-cluster`. To disable this behavior, -add the following to the Sidekiq configuration: - -```ruby -sidekiq['enable'] = true -sidekiq['cluster'] = false -``` - -All of the aforementioned configuration options for `sidekiq` -are available. By default, they are configured as follows: - -```ruby -sidekiq['queue_selector'] = false -sidekiq['interval'] = nil -sidekiq['max_concurrency'] = 50 -sidekiq['min_concurrency'] = nil -sidekiq['negate'] = false -sidekiq['queue_groups'] = ['*'] -sidekiq['shutdown_timeout'] = 25 -``` - -`sidekiq_cluster` must be disabled if you decide to configure the -cluster as above. - -When disabling `sidekiq_cluster`, you must copy your configuration for -`sidekiq_cluster`over to `sidekiq`. Anything configured for -`sidekiq_cluster` is overridden by the options for `sidekiq` when -setting `sidekiq['cluster'] = true`. - -When using this feature, the service called `sidekiq` is now -running `sidekiq-cluster`. - -The [concurrency](#manage-concurrency) and other options configured -for Sidekiq are respected. - -By default, logs for `sidekiq-cluster` go to `/var/log/gitlab/sidekiq` -like regular Sidekiq logs. - ## Ignore all import queues When [importing from GitHub](../../user/project/import/github.md) or @@ -264,7 +220,7 @@ being equal to `max_concurrency`. Running a single Sidekiq process is the default in GitLab 12.10 and earlier. WARNING: -Running Sidekiq directly is scheduled to be removed in GitLab +Running Sidekiq directly was removed in GitLab [14.0](https://gitlab.com/gitlab-com/gl-infra/scalability/-/issues/240). 1. Edit `/etc/gitlab/gitlab.rb` and add: diff --git a/doc/administration/operations/extra_sidekiq_routing.md b/doc/administration/operations/extra_sidekiq_routing.md index 80540b7ba46..6938f8a7012 100644 --- a/doc/administration/operations/extra_sidekiq_routing.md +++ b/doc/administration/operations/extra_sidekiq_routing.md @@ -103,9 +103,11 @@ based on a subset of worker attributes: - `worker_name` - the worker name. The other attributes are typically more useful as they are more general, but this is available in case a particular worker needs to be selected. -- `name` - the queue name. The other attributes are typically more useful as - they are more general, but this is available in case a particular queue needs - to be selected. +- `name` - the queue name generated from the worker name. The other attributes + are typically more useful as they are more general, but this is available in + case a particular queue needs to be selected. Because this is generated from + the worker name, it does not change based on the result of other routing + rules. - `resource_boundary` - if the queue is bound by `cpu`, `memory`, or `unknown`. For example, the `ProjectExportWorker` is memory bound as it has to load data in memory before saving it for export. diff --git a/doc/administration/operations/fast_ssh_key_lookup.md b/doc/administration/operations/fast_ssh_key_lookup.md index bb0756cf948..5c1271486c0 100644 --- a/doc/administration/operations/fast_ssh_key_lookup.md +++ b/doc/administration/operations/fast_ssh_key_lookup.md @@ -48,7 +48,7 @@ lookups via database lookup. As part of [setting up Geo](../geo/index.md#setup-instructions), you are required to follow the steps outlined below for both the primary and -secondary nodes, but note that the `Write to "authorized keys" file` checkbox +secondary nodes, but the `Write to "authorized keys" file` checkbox only needs to be unchecked on the primary node since it is reflected automatically on the secondary if database replication is working. diff --git a/doc/administration/operations/moving_repositories.md b/doc/administration/operations/moving_repositories.md index fba20da9aea..765cf64e735 100644 --- a/doc/administration/operations/moving_repositories.md +++ b/doc/administration/operations/moving_repositories.md @@ -7,8 +7,7 @@ type: reference # Moving repositories managed by GitLab **(FREE SELF)** -Sometimes you need to move all repositories managed by GitLab to -another file system or another server. +You can move all repositories managed by GitLab to another file system or another server. ## Moving data within a GitLab instance @@ -28,7 +27,128 @@ For more information, see: querying and scheduling snippet repository moves. - [The API documentation](../../api/group_repository_storage_moves.md) details the endpoints for querying and scheduling group repository moves **(PREMIUM SELF)**. -- [Migrate to Gitaly Cluster](../gitaly/praefect.md#migrate-to-gitaly-cluster). +- [Migrate to Gitaly Cluster](../gitaly/index.md#migrate-to-gitaly-cluster). + +### Move Repositories + +GitLab repositories can be associated with projects, groups, and snippets. Each of these types +have a separate API to schedule the respective repositories to move. To move all repositories +on a GitLab instance, each of these types must be scheduled to move for each storage. + +WARNING: +To move repositories into a [Gitaly Cluster](../gitaly/index.md#gitaly-cluster) in GitLab versions +13.12 to 14.1, you must [enable the `gitaly_replicate_repository_direct_fetch` feature flag](../feature_flags.md). + +Each repository is made read-only for the duration of the move. The repository is not writable +until the move has completed. + +To move repositories: + +1. Ensure all storages are accessible to the GitLab instance. In this example, these are + `` and ``. +1. [Configure repository storage weights](../repository_storage_paths.md#configure-where-new-repositories-are-stored) + so that the new storages receives all new projects. This stops new projects from being created + on existing storages while the migration is in progress. +1. Schedule repository moves for: + - [Projects](#bulk-schedule-project-moves). + - [Snippets](#bulk-schedule-snippet-moves). + - [Groups](#bulk-schedule-group-moves). **(PREMIUM SELF)** + +### Bulk schedule project moves + +Use the API to schedule project moves: + +1. [Schedule repository storage moves for all projects on a storage shard](../../api/project_repository_storage_moves.md#schedule-repository-storage-moves-for-all-projects-on-a-storage-shard) + using the API. For example: + + ```shell + curl --request POST --header "Private-Token: " \ + --header "Content-Type: application/json" \ + --data '{"source_storage_name":"","destination_storage_name":""}' \ + "https://gitlab.example.com/api/v4/project_repository_storage_moves" + ``` + +1. [Query the most recent repository moves](../../api/project_repository_storage_moves.md#retrieve-all-project-repository-storage-moves) + using the API. The response indicates either: + - The moves have completed successfully. The `state` field is `finished`. + - The moves are in progress. Re-query the repository move until it completes successfully. + - The moves have failed. Most failures are temporary and are solved by rescheduling the move. + +1. After the moves are complete, use the API to [query projects](../../api/projects.md#list-all-projects) and confirm that all projects have moved. None of the projects should be returned with the + `repository_storage` field set to the old storage. For example: + + ```shell + curl --header "Private-Token: " --header "Content-Type: application/json" \ + "https://gitlab.example.com/api/v4/projects?repository_storage=" + ``` + + Alternatively use [the rails console](../operations/rails_console.md) to confirm that all + projects have moved. Run the following in the rails console: + + ```ruby + ProjectRepository.for_repository_storage('') + ``` + +1. Repeat for each storage as required. + +### Bulk schedule snippet moves + +Use the API to schedule snippet moves: + +1. [Schedule repository storage moves for all snippets on a storage shard](../../api/snippet_repository_storage_moves.md#schedule-repository-storage-moves-for-all-snippets-on-a-storage-shard). For example: + + ```shell + curl --request POST --header "PRIVATE-TOKEN: " \ + --header "Content-Type: application/json" \ + --data '{"source_storage_name":"","destination_storage_name":""}' \ + "https://gitlab.example.com/api/v4/snippet_repository_storage_moves" + ``` + +1. [Query the most recent repository moves](../../api/snippet_repository_storage_moves.md#retrieve-all-snippet-repository-storage-moves) +The response indicates either: + - The moves have completed successfully. The `state` field is `finished`. + - The moves are in progress. Re-query the repository move until it completes successfully. + - The moves have failed. Most failures are temporary and are solved by rescheduling the move. + +1. After the moves are complete, use [the rails console](../operations/rails_console.md) to confirm + that all snippets have moved. No snippets should be returned for the original storage. Run the + following in the rails console: + + ```ruby + SnippetRepository.for_repository_storage('') + ``` + +1. Repeat for each storage as required. + +### Bulk schedule group moves **(PREMIUM SELF)** + +Use the API to schedule group moves: + +1. [Schedule repository storage moves for all groups on a storage shard](../../api/group_repository_storage_moves.md#schedule-repository-storage-moves-for-all-groups-on-a-storage-shard) +. For example: + + ```shell + curl --request POST --header "PRIVATE-TOKEN: " \ + --header "Content-Type: application/json" \ + --data '{"source_storage_name":"","destination_storage_name":""}' \ + "https://gitlab.example.com/api/v4/group_repository_storage_moves" + ``` + +1. [Query the most recent repository moves](../../api/group_repository_storage_moves.md#retrieve-all-group-repository-storage-moves) +. The response indicates either: + - The moves have completed successfully. The `state` field is `finished`. + - The moves are in progress. Re-query the repository move until it completes successfully. + - The moves have failed. Most failures are temporary and are solved by rescheduling the move. + +1. After the moves are complete, use [the rails console](../operations/rails_console.md) to confirm + that all groups have moved. No groups should be returned for the original storage. Run the + following in the rails console: + + ```ruby + GroupWikiRepository.for_repository_storage('') + ``` + +1. Repeat for each storage as required. ## Migrating to another GitLab instance -- cgit v1.2.3