Add latest changes from gitlab-org/gitlab@16-6-stable-eev16.6.0-rc42

3 files changed, 57 insertions, 39 deletions

diff --git a/doc/administration/sidekiq/index.md b/doc/administration/sidekiq/index.md
index 10fadc40a82..0a7974c9622 100644
--- a/doc/administration/sidekiq/index.md
+++ b/doc/administration/sidekiq/index.md
@@ -95,27 +95,8 @@ Updates to example must be made at:
 -->
 
    ```ruby
-   ########################################
-   #####        Services Disabled       ###
-   ########################################
-   #
-   # When running GitLab on just one server, you have a single `gitlab.rb`
-   # to enable all services you want to run.
-   # When running GitLab on N servers, you have N `gitlab.rb` files.
-   # Enable only the services you want to run on each
-   # specific server, while disabling all others.
-   #
-   gitaly['enable'] = false
-   postgresql['enable'] = false
-   redis['enable'] = false
-   nginx['enable'] = false
-   puma['enable'] = false
-   gitlab_workhorse['enable'] = false
-   prometheus['enable'] = false
-   alertmanager['enable'] = false
-   grafana['enable'] = false
-   gitlab_exporter['enable'] = false
-   gitlab_kas['enable'] = false
+   # https://docs.gitlab.com/omnibus/roles/#sidekiq-roles
+   roles(["sidekiq_role"])
 
    ##
    ## To maintain uniformity of links across nodes, the
@@ -375,20 +356,6 @@ To enable LDAP with the synchronization worker for Sidekiq:
 
 If you use [SAML Group Sync](../../user/group/saml_sso/group_sync.md), you must configure [SAML Groups](../../integration/saml.md#configure-users-based-on-saml-group-membership) on all your Sidekiq nodes.
 
-## Disable Rugged
-
-Calls into Rugged, Ruby bindings for `libgit2`, [lock the Sidekiq processes (GVL)](https://silverhammermba.github.io/emberb/c/#c-in-ruby-threads),
-blocking all jobs on that worker from proceeding. If Rugged calls performed by Sidekiq are slow, this can cause significant delays in
-background task processing.
-
-By default, Rugged is used when Git repository data is stored on local storage or on an NFS mount.
-Using Rugged is recommended when using NFS, but if
-you are using local storage, disabling Rugged can improve Sidekiq performance:
-
-```shell
-sudo gitlab-rake gitlab:features:disable_rugged
-```
-
 ## Related topics
 
 - [Extra Sidekiq processes](extra_sidekiq_processes.md)
diff --git a/doc/administration/sidekiq/processing_specific_job_classes.md b/doc/administration/sidekiq/processing_specific_job_classes.md
index 696b0b9444c..74cbb6ca89b 100644
--- a/doc/administration/sidekiq/processing_specific_job_classes.md
+++ b/doc/administration/sidekiq/processing_specific_job_classes.md
@@ -179,14 +179,16 @@ nodes. In this example, we exclude all import-related jobs from a Sidekiq node.
    sudo gitlab-ctl reconfigure
    ```
 
-### Migrating from queue selectors to routing rules
+## Migrating from queue selectors to routing rules
 
 We recommend GitLab deployments add more Sidekiq processes listening to all queues, as in the
 [Reference Architectures](../reference_architectures/index.md). For very large-scale deployments, we recommend
 [routing rules](#routing-rules) instead of [queue selectors](#queue-selectors-deprecated). We use routing rules on GitLab.com as
 it helps to lower the load on Redis.
 
-To migrate from queue selectors to routing rules:
+### Single node setup
+
+To migrate from queue selectors to routing rules in a [single node setup](../reference_architectures/index.md#standalone-non-ha):
 
 1. Open `/etc/gitlab/gitlab.rb`.
 1. Set `sidekiq['queue_selector']` to `false`.
@@ -213,9 +215,11 @@ NOTE:
 It is important to run the Rake task immediately after reconfiguring GitLab.
 After reconfiguring GitLab, existing jobs are not processed until the Rake task starts to migrate the jobs.
 
+#### Migration example
+
 The following example better illustrates the migration process above:
 
-1. Check the following content of `/etc/gitlab/gitlab.rb`:
+1. In `/etc/gitlab/gitlab.rb`, check the `urgency` queries in the `sidekiq['queue_groups']`. For example:
 
    ```ruby
    sidekiq['routing_rules'] = []
@@ -228,7 +232,7 @@ The following example better illustrates the migration process above:
    ]
    ```
 
-1. Update `/etc/gitlab/gitlab.rb` to use routing rules:
+1. Use these same `urgency` queries to update `/etc/gitlab/gitlab.rb` to use routing rules:
 
    ```ruby
    sidekiq['min_concurrency'] = 20
@@ -270,6 +274,31 @@ in a queue group entry is 1, while `min_concurrency` is set to `0`, and `max_con
 concurrency is set to `2` instead. A concurrency of `2` might be too low in most cases, except for very highly-CPU
 bound tasks.
 
+### Multiple node setup
+
+For a multiple node setup:
+
+- Reconfigure all GitLab Rails and Sidekiq nodes with the same `sidekiq['routing_rules']` setting.
+- Alternate between GitLab Rails and Sidekiq nodes as you update and reconfigure the nodes. This ensures the newly configured Sidekiq is ready to consume jobs from the new set of
+  queues during the migration. Otherwise, the new jobs hang until the end of the migration.
+
+Consider the following example of three GitLab Rails nodes and two Sidekiq nodes. To migrate from queue selectors to routing rules:
+
+1. In Sidekiq 1, follow all steps but one in [single node setup](#single-node-setup).
+   **Do not** run the Rake task to [migrate existing jobs](sidekiq_job_migration.md).
+1. Configure the external load balancer to remove Rails 1 from accepting traffic. This step ensures Rails 1 is not serving any request while the Rails process is restarting. For more information, see [issue 428794](https://gitlab.com/gitlab-org/gitlab/-/issues/428794#note_1619505870).
+1. In Rails 1, update `/etc/gitlab/gitlab.rb` to use the same `sidekiq['routing_rules']` setting as Sidekiq 1.
+   Only `sidekiq['routing_rules']` is required in Rails nodes.
+1. Configure the external load balancer to register Rails 1 back.
+1. Repeat steps 1 to 4 for Sidekiq 2 and Rails 2.
+1. Repeat steps 2 to 4 for Rails 3.
+1. If there are more Sidekiq nodes than Rails nodes, follow step 1 on the remaining Sidekiq nodes.
+1. Run the Rake task to [migrate existing jobs](sidekiq_job_migration.md):
+
+   ```shell
+   sudo gitlab-rake gitlab:sidekiq:migrate_jobs:retry gitlab:sidekiq:migrate_jobs:schedule gitlab:sidekiq:migrate_jobs:queued
+   ```
+
 <!--- end_remove -->
 
 ## Worker matching query
diff --git a/doc/administration/sidekiq/sidekiq_troubleshooting.md b/doc/administration/sidekiq/sidekiq_troubleshooting.md
index 9ae2a59251a..2990110150f 100644
--- a/doc/administration/sidekiq/sidekiq_troubleshooting.md
+++ b/doc/administration/sidekiq/sidekiq_troubleshooting.md
@@ -536,6 +536,28 @@ The list of available jobs can be found in the [workers](https://gitlab.com/gitl
 
 For more information about Sidekiq jobs, see the [Sidekiq-cron](https://github.com/sidekiq-cron/sidekiq-cron#work-with-job) documentation.
 
+## Disabling cron jobs
+
+You can disable any Sidekiq cron jobs by visiting the [Monitoring section in the Admin area](../admin_area.md#monitoring-section). You can also perform the same action using the command line and [Rails Runner](../operations/rails_console.md#using-the-rails-runner).
+
+To disable all cron jobs:
+
+```shell
+sudo gitlab-rails runner 'Sidekiq::Cron::Job.all.map(&:disable!)'
+```
+
+To enable all cron jobs:
+
+```shell
+sudo gitlab-rails runner 'Sidekiq::Cron::Job.all.map(&:enable!)'
+```
+
+If you wish to enable only a subset of the jobs at a time you can use name matching. For example, to enable only jobs with `geo` in the name:
+
+```shell
+ sudo gitlab-rails runner 'Sidekiq::Cron::Job.all.select{ |j| j.name.match("geo") }.map(&:disable!)'
+```
+
 ## Clearing a Sidekiq job deduplication idempotency key
 
 Occasionally, jobs that are expected to run (for example, cron jobs) are observed to not run at all. When checking the logs, there might be instances where jobs are seen to not run with a `"job_status": "deduplicated"`.