Welcome to mirror list, hosted at ThFree Co, Russian Federation.

gitlab.com/gitlab-org/gitlab-foss.git - Unnamed repository; edit this file 'description' to name the repository.
summaryrefslogtreecommitdiff
path: root/doc
diff options
context:
space:
mode:
authorGitLab Bot <gitlab-bot@gitlab.com>2022-11-17 18:11:26 +0300
committerGitLab Bot <gitlab-bot@gitlab.com>2022-11-17 18:11:26 +0300
commit4e752429e6173020567f9509f1fa993cc82a258a (patch)
tree49648f91db0d7849065d2d8897757f7de815c773 /doc
parent255831389a5080bb61242b3b50426918c4e1a5aa (diff)
Add latest changes from gitlab-org/gitlab@master
Diffstat (limited to 'doc')
-rw-r--r--doc/administration/geo/replication/tuning.md2
-rw-r--r--doc/administration/geo/setup/database.md3
-rw-r--r--doc/administration/operations/index.md1
-rw-r--r--doc/administration/sidekiq/extra_sidekiq_processes.md286
-rw-r--r--doc/administration/sidekiq/extra_sidekiq_routing.md166
-rw-r--r--doc/administration/sidekiq/index.md2
-rw-r--r--doc/administration/sidekiq/processing_specific_job_classes.md240
-rw-r--r--doc/administration/sidekiq/sidekiq_job_migration.md17
-rw-r--r--doc/architecture/blueprints/ci_data_decay/pipeline_partitioning.md7
-rw-r--r--doc/ci/environments/index.md13
-rw-r--r--doc/development/sidekiq/index.md4
-rw-r--r--doc/index.md23
-rw-r--r--doc/integration/advanced_search/elasticsearch.md8
-rw-r--r--doc/integration/advanced_search/elasticsearch_troubleshooting.md2
-rw-r--r--doc/update/index.md6
15 files changed, 354 insertions, 426 deletions
diff --git a/doc/administration/geo/replication/tuning.md b/doc/administration/geo/replication/tuning.md
index ab9263ad344..4dc3ba93d66 100644
--- a/doc/administration/geo/replication/tuning.md
+++ b/doc/administration/geo/replication/tuning.md
@@ -30,7 +30,7 @@ However, this may not lead to more downloads in parallel unless the number of
available Sidekiq threads is also increased. For example, if repository synchronization
concurrency is increased from 25 to 50, you may also want to increase the number
of Sidekiq threads from 25 to 50. See the
-[Sidekiq concurrency documentation](../../sidekiq/extra_sidekiq_processes.md#number-of-threads)
+[Sidekiq concurrency documentation](../../sidekiq/extra_sidekiq_processes.md#concurrency)
for more details.
## Repository re-verification
diff --git a/doc/administration/geo/setup/database.md b/doc/administration/geo/setup/database.md
index 86caf5306b5..1ee7f0665cd 100644
--- a/doc/administration/geo/setup/database.md
+++ b/doc/administration/geo/setup/database.md
@@ -519,6 +519,9 @@ data before running `pg_basebackup`.
add `--force` to the command line.
- When not in a production machine you can disable backup step if you
really sure this is what you want by adding `--skip-backup`
+ - If you are using PgBouncer, you need to target the database host directly.
+ - If you are using Patroni on your primary site, you must target the current leader host.
+ - If you are using a load balancer proxy (for example HAProxy) and it is targeting the Patroni leader for the primary, you should target the load balancer proxy instead.
The replication process is now complete.
diff --git a/doc/administration/operations/index.md b/doc/administration/operations/index.md
index d18f41becd5..7b495a5dd58 100644
--- a/doc/administration/operations/index.md
+++ b/doc/administration/operations/index.md
@@ -16,7 +16,6 @@ Keep your GitLab instance up and running smoothly.
- [Sidekiq MemoryKiller](sidekiq_memory_killer.md): Configure Sidekiq MemoryKiller
to restart Sidekiq.
- [Multiple Sidekiq processes](extra_sidekiq_processes.md): Configure multiple Sidekiq processes to ensure certain queues always have dedicated workers, no matter the number of jobs that must be processed. **(FREE SELF)**
-- [Sidekiq routing rules](extra_sidekiq_routing.md): Configure the routing rules to route a job from a worker to a desirable queue. **(FREE SELF)**
- [Puma](puma.md): Understand Puma and puma-worker-killer.
- Speed up SSH operations by
[Authorizing SSH users via a fast, indexed lookup to the GitLab database](fast_ssh_key_lookup.md), and/or
diff --git a/doc/administration/sidekiq/extra_sidekiq_processes.md b/doc/administration/sidekiq/extra_sidekiq_processes.md
index feaaa55aa59..d5007e9a3e9 100644
--- a/doc/administration/sidekiq/extra_sidekiq_processes.md
+++ b/doc/administration/sidekiq/extra_sidekiq_processes.md
@@ -6,91 +6,41 @@ info: To determine the technical writer assigned to the Stage/Group associated w
# Run multiple Sidekiq processes **(FREE SELF)**
-GitLab allows you to start multiple Sidekiq processes.
-These processes can be used to consume a dedicated set
-of queues. This can be used to ensure certain queues always have dedicated
-workers, no matter the number of jobs to be processed.
+GitLab allows you to start multiple Sidekiq processes to process background jobs
+at a higher rate on a single instance. By default, Sidekiq starts one worker
+process and only uses a single core.
NOTE:
The information in this page applies only to Omnibus GitLab.
-## Available Sidekiq queues
-
-For a list of the existing Sidekiq queues, check the following files:
-
-- [Queues for both GitLab Community and Enterprise Editions](https://gitlab.com/gitlab-org/gitlab/-/blob/master/app/workers/all_queues.yml)
-- [Queues for GitLab Enterprise Editions only](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/app/workers/all_queues.yml)
-
-Each entry in the above files represents a queue on which Sidekiq processes
-can be started.
-
## Start multiple processes
> - [Introduced](https://gitlab.com/gitlab-org/omnibus-gitlab/-/merge_requests/4006) in GitLab 12.10, starting multiple processes with Sidekiq cluster.
> - [Sidekiq cluster moved](https://gitlab.com/groups/gitlab-com/gl-infra/-/epics/181) to GitLab Free in 12.10.
> - [Sidekiq cluster became default](https://gitlab.com/gitlab-org/omnibus-gitlab/-/merge_requests/4140) in GitLab 13.0.
-When starting multiple processes, the number of processes should
-equal (and **not** exceed) the number of CPU cores you want to
-dedicate to Sidekiq. Each Sidekiq process can use only 1 CPU
-core, subject to the available workload and concurrency settings.
+When starting multiple processes, the number of processes should at most equal
+(and **not** exceed) the number of CPU cores you want to dedicate to Sidekiq.
+The Sidekiq worker process uses no more than one CPU core.
-To start multiple processes:
+To start multiple processes, use the `sidekiq['queue_groups']` array setting to
+specify how many processes to create using `sidekiq-cluster` and which queues
+they should handle. Each item in the array equates to one additional Sidekiq
+process, and values in each item determine the queues it works on. In the vast
+majority of cases, all processes should listen to all queues (see
+[processing specific job classes](processing_specific_job_classes.md) for more
+details).
-1. Using the `sidekiq['queue_groups']` array setting, specify how many processes to
- create using `sidekiq-cluster` and which queue they should handle.
- Each item in the array equates to one additional Sidekiq
- process, and values in each item determine the queues it works on.
+For example, to create four Sidekiq processes, each listening
+to all available queues:
- For example, the following setting creates three Sidekiq processes, one to run on
- `elastic_commit_indexer`, one to run on `mailers`, and one process running on all queues:
+1. Edit `/etc/gitlab/gitlab.rb`:
```ruby
- sidekiq['queue_groups'] = [
- "elastic_commit_indexer",
- "mailers",
- "*"
- ]
+ sidekiq['queue_groups'] = ['*'] * 4
```
- To have an additional Sidekiq process handle multiple queues, add multiple
- queue names to its item delimited by commas. For example:
-
- ```ruby
- sidekiq['queue_groups'] = [
- "elastic_commit_indexer, elastic_association_indexer",
- "mailers",
- "*"
- ]
- ```
-
- [In GitLab 12.9](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/26594) and
- later, the special queue name `*` means all queues. This starts two
- processes, each handling all queues:
-
- ```ruby
- sidekiq['queue_groups'] = [
- "*",
- "*"
- ]
- ```
-
- `*` which matches all workers.
- As a result, the wildcard query must stay at the end of the list or the rules after it are ignored.
-
- `*` cannot be combined with concrete queue names - `*, mailers`
- just handles the `mailers` queue.
-
- When `sidekiq-cluster` is only running on a single node, make sure that at least
- one process is running on all queues using `*`. This ensures a process
- automatically picks up jobs in queues created in the future,
- including queues that have dedicated processes.
-
- If `sidekiq-cluster` is running on more than one node, you can also use
- [`--negate`](#negate-settings) and list all the queues that are already being
- processed.
-
-1. Save the file and reconfigure GitLab for the changes to take effect:
+1. Save the file and reconfigure GitLab:
```shell
sudo gitlab-ctl reconfigure
@@ -101,125 +51,38 @@ To view the Sidekiq processes in GitLab:
1. On the top bar, select **Main menu > Admin**.
1. On the left sidebar, select **Monitoring > Background Jobs**.
-## Negate settings
+## Concurrency
-To have the Sidekiq process work on every queue **except** the ones
-you list. In this example, we exclude all import-related jobs from a Sidekiq node:
+By default each process defined under `sidekiq` starts with a number of threads
+that equals the number of queues, plus one spare thread, up to a maximum of 50.
+For example, a process that handles all queues will use 50 threads by default.
-1. Edit `/etc/gitlab/gitlab.rb` and add:
-
- ```ruby
- sidekiq['negate'] = true
- sidekiq['queue_selector'] = true
- sidekiq['queue_groups'] = [
- "feature_category=importers"
- ]
- ```
-
-1. Save the file and reconfigure GitLab for the changes to take effect:
-
- ```shell
- sudo gitlab-ctl reconfigure
- ```
-
-## Queue selector
-
-> - [Introduced](https://gitlab.com/gitlab-com/gl-infra/scalability/-/issues/45) in GitLab 12.8.
-> - [Sidekiq cluster, including queue selector, moved](https://gitlab.com/groups/gitlab-com/gl-infra/-/epics/181) to GitLab Free in 12.10.
-> - [Renamed from `experimental_queue_selector` to `queue_selector`](https://gitlab.com/gitlab-com/gl-infra/scalability/-/issues/147) in GitLab 13.6.
-
-In addition to selecting queues by name, as above, the `queue_selector` option
-allows queue groups to be selected in a more general way using a
-[worker matching query](extra_sidekiq_routing.md#worker-matching-query). After `queue_selector`
-is set, all `queue_groups` must follow the aforementioned syntax.
-
-In `/etc/gitlab/gitlab.rb`:
-
-```ruby
-sidekiq['enable'] = true
-sidekiq['queue_selector'] = true
-sidekiq['queue_groups'] = [
- # Run all non-CPU-bound queues that are high urgency
- 'resource_boundary!=cpu&urgency=high',
- # Run all continuous integration and pages queues that are not high urgency
- 'feature_category=continuous_integration,pages&urgency!=high',
- # Run all queues
- '*'
-]
-```
-
-## Ignore all import queues
-
-When [importing from GitHub](../../user/project/import/github.md) or
-other sources, Sidekiq might use all of its resources to perform those
-operations. To set up two separate `sidekiq-cluster` processes, where
-one only processes imports and the other processes all other queues:
-
-1. Edit `/etc/gitlab/gitlab.rb` and add:
-
- ```ruby
- sidekiq['enable'] = true
- sidekiq['queue_selector'] = true
- sidekiq['queue_groups'] = [
- "feature_category=importers",
- "feature_category!=importers"
- ]
- ```
-
-1. Save the file and reconfigure GitLab for the changes to take effect:
-
- ```shell
- sudo gitlab-ctl reconfigure
- ```
-
-## Number of threads
-
-By default each process defined under `sidekiq` starts with a
-number of threads that equals the number of queues, plus one spare thread.
-For example, a process that handles the `process_commit` and `post_receive`
-queues uses three threads in total.
-
-These thread run inside a single Ruby process, and each process
-can only use a single CPU core. The usefulness of threading depends
-on the work having some external dependencies to wait on, like database queries or
-HTTP requests. Most Sidekiq deployments benefit from this threading, and when
-running fewer queues in a process, increasing the thread count might be
-even more desirable to make the most effective use of CPU resources.
+These threads run inside a single Ruby process, and each process can only use a
+single CPU core. The usefulness of threading depends on the work having some
+external dependencies to wait on, like database queries or HTTP requests. Most
+Sidekiq deployments benefit from this threading.
### Manage thread counts explicitly
-The correct maximum thread count (also called concurrency) depends on the workload.
-Typical values range from `1` for highly CPU-bound tasks to `15` or higher for mixed
-low-priority work. A reasonable starting range is `15` to `25` for a non-specialized
-deployment.
+The correct maximum thread count (also called concurrency) depends on the
+workload. Typical values range from `5` for highly CPU-bound tasks to `15` or
+higher for mixed low-priority work. A reasonable starting range is `15` to `25`
+for a non-specialized deployment.
-You can find example values used by GitLab.com by searching for `concurrency:` in
-[the Helm charts](https://gitlab.com/gitlab-com/gl-infra/k8s-workloads/gitlab-com/-/blob/master/releases/gitlab/values/gprd.yaml.gotmpl).
-The values vary according to the work each specific deployment of Sidekiq does.
-Any other specialized deployments with processes dedicated to specific queues should
-have the concurrency tuned according to:
-have the concurrency tuned according to:
+We only recommend setting explicit concurrency by setting `min_concurrency` and
+`max_concurrency` to the same value. The two values are kept for backwards
+compatibility reasons, but for more predictable results, use the same value.
-- The CPU usage of each type of process.
-- The throughput achieved.
-
-Each thread requires a Redis connection, so adding threads may increase Redis
-latency and potentially cause client timeouts. See the
-[Sidekiq documentation about Redis](https://github.com/mperham/sidekiq/wiki/Using-Redis) for more
-details.
-
-#### When running Sidekiq cluster (default)
+For example, to set the concurrency to `20`:
-Running Sidekiq cluster is the default in GitLab 13.0 and later.
-
-1. Edit `/etc/gitlab/gitlab.rb` and add:
+1. Edit `/etc/gitlab/gitlab.rb`:
```ruby
- sidekiq['min_concurrency'] = 15
- sidekiq['max_concurrency'] = 25
+ sidekiq['min_concurrency'] = 20
+ sidekiq['max_concurrency'] = 20
```
-1. Save the file and reconfigure GitLab for the changes to take effect:
+1. Save the file and reconfigure GitLab:
```shell
sudo gitlab-ctl reconfigure
@@ -231,50 +94,45 @@ the other. Setting `min_concurrency` to `0` disables the limit.
For each queue group, let `N` be one more than the number of queues. The
concurrency is set to:
+1. `min_concurrency`, if it's equal to `max_concurrency`.
1. `N`, if it's between `min_concurrency` and `max_concurrency`.
1. `max_concurrency`, if `N` exceeds this value.
1. `min_concurrency`, if `N` is less than this value.
-If `min_concurrency` is equal to `max_concurrency`, then this value is used
-regardless of the number of queues.
-
When `min_concurrency` is greater than `max_concurrency`, it is treated as
being equal to `max_concurrency`.
-#### When running a single Sidekiq process
-
-Running a single Sidekiq process is the default in GitLab 12.10 and earlier.
-
-WARNING:
-Running Sidekiq directly was removed in GitLab
-[14.0](https://gitlab.com/gitlab-com/gl-infra/scalability/-/issues/240).
-
-1. Edit `/etc/gitlab/gitlab.rb` and add:
-
- ```ruby
- sidekiq['cluster'] = false
- sidekiq['concurrency'] = 25
- ```
-
-1. Save the file and reconfigure GitLab for the changes to take effect:
+You can find example values used by GitLab.com by searching for `concurrency:`
+in [the Helm charts](https://gitlab.com/gitlab-com/gl-infra/k8s-workloads/gitlab-com/-/blob/master/releases/gitlab/values/gprd.yaml.gotmpl).
+The values vary according to the work each specific deployment of Sidekiq does.
+Any other specialized deployments with processes dedicated to specific queues
+should have the concurrency tuned according to:
- ```shell
- sudo gitlab-ctl reconfigure
- ```
+- The CPU usage of each type of process.
+- The throughput achieved.
-This sets the concurrency (number of threads) for the Sidekiq process.
+Each thread requires a Redis connection, so adding threads may increase Redis
+latency and potentially cause client timeouts. See the [Sidekiq documentation about Redis](https://github.com/mperham/sidekiq/wiki/Using-Redis)
+for more details.
## Modify the check interval
-To modify `sidekiq-cluster`'s health check interval for the additional Sidekiq processes:
+To modify Sidekiq's health check interval for the additional Sidekiq
+processes:
-1. Edit `/etc/gitlab/gitlab.rb` and add (the value can be any integer number of seconds):
+1. Edit `/etc/gitlab/gitlab.rb`:
```ruby
sidekiq['interval'] = 5
```
-1. Save the file and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect.
+ The value can be any integer number of seconds.
+
+1. Save the file and reconfigure GitLab:
+
+ ```shell
+ sudo gitlab-ctl reconfigure
+ ```
## Troubleshoot using the CLI
@@ -291,6 +149,9 @@ takes arguments using the following syntax:
/opt/gitlab/embedded/service/gitlab-rails/bin/sidekiq-cluster [QUEUE,QUEUE,...] [QUEUE, ...]
```
+The `--dryrun` argument allows viewing the command to be executed without
+actually starting it.
+
Each separate argument denotes a group of queues that have to be processed by a
Sidekiq process. Multiple queues can be processed by the same process by
separating them with a comma instead of a space.
@@ -301,29 +162,6 @@ explicitly list all the queue names. For more information about queue namespaces
see the relevant section in the
[Sidekiq development documentation](../../development/sidekiq/index.md#queue-namespaces).
-For example, say you want to start 2 extra processes: one to process the
-`process_commit` queue, and one to process the `post_receive` queue. This can be
-done as follows:
-
-```shell
-/opt/gitlab/embedded/service/gitlab-rails/bin/sidekiq-cluster process_commit post_receive
-```
-
-If you instead want to start one process processing both queues, you'd use the
-following syntax:
-
-```shell
-/opt/gitlab/embedded/service/gitlab-rails/bin/sidekiq-cluster process_commit,post_receive
-```
-
-If you want to have one Sidekiq process dealing with the `process_commit` and
-`post_receive` queues, and one process to process the `gitlab_shell` queue,
-you'd use the following:
-
-```shell
-/opt/gitlab/embedded/service/gitlab-rails/bin/sidekiq-cluster process_commit,post_receive gitlab_shell
-```
-
### Monitor the `sidekiq-cluster` command
The `sidekiq-cluster` command does not terminate once it has started the desired
diff --git a/doc/administration/sidekiq/extra_sidekiq_routing.md b/doc/administration/sidekiq/extra_sidekiq_routing.md
index 56c51beb758..d1d65498fcc 100644
--- a/doc/administration/sidekiq/extra_sidekiq_routing.md
+++ b/doc/administration/sidekiq/extra_sidekiq_routing.md
@@ -1,163 +1,11 @@
---
-stage: Systems
-group: Distribution
-info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments
+redirect_to: 'processing_specific_job_classes.md#routing-rules'
+remove_date: '2023-02-01'
---
-# Queue routing rules **(FREE SELF)**
+This document was moved to [another location](processing_specific_job_classes.md#routing-rules).
-When the number of Sidekiq jobs increases to a certain scale, the system faces
-some scalability issues. One of them is that the length of the queue tends to get
-longer. High-urgency jobs have to wait longer until other less urgent jobs
-finish. This head-of-line blocking situation may eventually affect the
-responsiveness of the system, especially critical actions. In another scenario,
-the performance of some jobs is degraded due to other long running or CPU-intensive jobs
-(computing or rendering ones) in the same machine.
-
-To counter the aforementioned issues, one effective solution is to split
-Sidekiq jobs into different queues and assign machines handling each queue
-exclusively. For example, all CPU-intensive jobs could be routed to the
-`cpu-bound` queue and handled by a fleet of CPU optimized instances. The queue
-topology differs between companies depending on the workloads and usage
-patterns. Therefore, GitLab supports a flexible mechanism for the
-administrator to route the jobs based on their characteristics.
-
-As an alternative to [Queue selector](extra_sidekiq_processes.md#queue-selector), which
-configures Sidekiq cluster to listen to a specific set of workers or queues,
-GitLab also supports routing a job from a worker to the desired queue when it
-is scheduled. Sidekiq clients try to match a job against a configured list of
-routing rules. Rules are evaluated from first to last, and as soon as we find a
-match for a given worker we stop processing for that worker (first match wins).
-If the worker doesn't match any rule, it falls back to the queue name generated
-from the worker name.
-
-By default, if the routing rules are not configured (or denoted with an empty
-array), all the jobs are routed to the queue generated from the worker name.
-
-## Example configuration
-
-In `/etc/gitlab/gitlab.rb`:
-
-```ruby
-sidekiq['routing_rules'] = [
- # Do not re-route workers that require their own queue
- ['tags=needs_own_queue', nil],
- # Route all non-CPU-bound workers that are high urgency to `high-urgency` queue
- ['resource_boundary!=cpu&urgency=high', 'high-urgency'],
- # Route all database, gitaly and global search workers that are throttled to `throttled` queue
- ['feature_category=database,gitaly,global_search&urgency=throttled', 'throttled'],
- # Route all workers having contact with outside work to a `network-intenstive` queue
- ['has_external_dependencies=true|feature_category=hooks|tags=network', 'network-intensive'],
- # Route all import workers to the queues generated by the worker name, for
- # example, JiraImportWorker to `jira_import`, SVNWorker to `svn_worker`
- ['feature_category=import', nil],
- # Wildcard matching, route the rest to `default` queue
- ['*', 'default']
-]
-```
-
-The routing rules list is an order-matter array of tuples of query and
-corresponding queue:
-
-- The query is following a [worker matching query](#worker-matching-query) syntax.
-- The `<queue_name>` must be a valid Sidekiq queue name. If the queue name
- is `nil`, or an empty string, the worker is routed to the queue generated
- by the name of the worker instead.
-
-The query supports wildcard matching `*`, which matches all workers. As a
-result, the wildcard query must stay at the end of the list or the rules after it
-are ignored.
-
-NOTE:
-Mixing queue routing rules and queue selectors requires care to
-ensure all jobs that are scheduled and picked up by appropriate Sidekiq
-workers.
-
-## Worker matching query
-
-GitLab provides a query syntax to match a worker based on its
-attributes. This query syntax is employed by both
-[Queue routing rules](#queue-routing-rules) and
-[Queue selector](extra_sidekiq_processes.md#queue-selector). A query includes two
-components:
-
-- Attributes that can be selected.
-- Operators used to construct a query.
-
-### Available attributes
-
-> [Introduced](https://gitlab.com/gitlab-com/gl-infra/scalability/-/issues/261) in GitLab 13.1 (`tags`).
-
-Queue matching query works upon the worker attributes, described in
-[Sidekiq style guide](../../development/sidekiq/index.md). We support querying
-based on a subset of worker attributes:
-
-- `feature_category` - the
- [GitLab feature category](https://about.gitlab.com/direction/maturity/#category-maturity) the
- queue belongs to. For example, the `merge` queue belongs to the
- `source_code_management` category.
-- `has_external_dependencies` - whether or not the queue connects to external
- services. For example, all importers have this set to `true`.
-- `urgency` - how important it is that this queue's jobs run
- quickly. Can be `high`, `low`, or `throttled`. For example, the
- `authorized_projects` queue is used to refresh user permissions, and
- is `high` urgency.
-- `worker_name` - the worker name. Use this attribute to select a specific worker.
-- `name` - the queue name generated from the worker name. Use this attribute to select a specific queue. Because this is generated from
- the worker name, it does not change based on the result of other routing
- rules.
-- `resource_boundary` - if the queue is bound by `cpu`, `memory`, or
- `unknown`. For example, the `ProjectExportWorker` is memory bound as it has
- to load data in memory before saving it for export.
-- `tags` - short-lived annotations for queues. These are expected to frequently
- change from release to release, and may be removed entirely.
-
-`has_external_dependencies` is a boolean attribute: only the exact
-string `true` is considered true, and everything else is considered
-false.
-
-`tags` is a set, which means that `=` checks for intersecting sets, and
-`!=` checks for disjoint sets. For example, `tags=a,b` selects queues
-that have tags `a`, `b`, or both. `tags!=a,b` selects queues that have
-neither of those tags.
-
-The attributes of each worker are hard-coded in the source code. For
-convenience, we generate a
-[list of all available attributes in GitLab Community Edition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/app/workers/all_queues.yml)
-and a
-[list of all available attributes in GitLab Enterprise Edition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/app/workers/all_queues.yml).
-
-### Available operators
-
-`queue_selector` supports the following operators, listed from highest
-to lowest precedence:
-
-- `|` - the logical `OR` operator. For example, `query_a|query_b` (where `query_a`
- and `query_b` are queries made up of the other operators here) includes
- queues that match either query.
-- `&` - the logical `AND` operator. For example, `query_a&query_b` (where
- `query_a` and `query_b` are queries made up of the other operators here) will
- only include queues that match both queries.
-- `!=` - the `NOT IN` operator. For example, `feature_category!=issue_tracking`
- excludes all queues from the `issue_tracking` feature category.
-- `=` - the `IN` operator. For example, `resource_boundary=cpu` includes all
- queues that are CPU bound.
-- `,` - the concatenate set operator. For example,
- `feature_category=continuous_integration,pages` includes all queues from
- either the `continuous_integration` category or the `pages` category. This
- example is also possible using the OR operator, but allows greater brevity, as
- well as being lower precedence.
-
-The operator precedence for this syntax is fixed: it's not possible to make `AND`
-have higher precedence than `OR`.
-
-[In GitLab 12.9](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/26594) and
-later, as with the standard queue group syntax above, a single `*` as the
-entire queue group selects all queues.
-
-### Migration
-
-After the Sidekiq routing rules are changed, administrators must take care
-with the migration to avoid losing jobs entirely, especially in a system with
-long queues of jobs. The migration can be done by following the migration steps
-mentioned in [Sidekiq job migration](sidekiq_job_migration.md)
+<!-- This redirect file can be deleted after <2023-02-01>. -->
+<!-- Redirects that point to other docs in the same project expire in three months. -->
+<!-- Redirects that point to docs in a different project or site (link is not relative and starts with `https:`) expire in one year. -->
+<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/redirects.html -->
diff --git a/doc/administration/sidekiq/index.md b/doc/administration/sidekiq/index.md
index f17c248e60e..d0cb124236c 100644
--- a/doc/administration/sidekiq/index.md
+++ b/doc/administration/sidekiq/index.md
@@ -398,7 +398,7 @@ sudo gitlab-rake gitlab:features:disable_rugged
## Related topics
- [Extra Sidekiq processes](extra_sidekiq_processes.md)
-- [Extra Sidekiq routing](extra_sidekiq_routing.md)
+- [Processing specific job classes](processing_specific_job_classes.md)
- [Sidekiq health checks](sidekiq_health_check.md)
- [Using the GitLab-Sidekiq chart](https://docs.gitlab.com/charts/charts/gitlab/sidekiq/)
diff --git a/doc/administration/sidekiq/processing_specific_job_classes.md b/doc/administration/sidekiq/processing_specific_job_classes.md
new file mode 100644
index 00000000000..18ffc8a8865
--- /dev/null
+++ b/doc/administration/sidekiq/processing_specific_job_classes.md
@@ -0,0 +1,240 @@
+---
+stage: Systems
+group: Distribution
+info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments
+---
+
+# Processing specific job classes
+
+WARNING:
+These are advanced settings. While they are used on GitLab.com, most GitLab
+instances should add more processes that all listen to all queues. This is the
+same approach we take in our [Reference Architectures](../reference_architectures/index.md).
+
+GitLab has two options for creating Sidekiq processes that only handle specific
+job classes:
+
+1. [Routing rules](#routing-rules) are used on GitLab.com. They direct jobs
+ inside the application to queue names configured by administrators. This
+ lowers the load on Redis, which is important on very large-scale deployments.
+1. [Queue selectors](#queue-selectors) perform the job selection outside the
+ application, when starting the Sidekiq process. This was used on GitLab.com
+ until September 2021, and is retained for compatibility reasons.
+
+Both of these use the same [worker matching query](#worker-matching-query)
+syntax. While they can technically be used together, most deployments should
+choose one or the other; there is no particular benefit in combining them.
+
+Routing rules must be the same across all GitLab nodes as they are part of the
+application configuration. Queue selectors can be different across GitLab nodes
+because they only change the arguments to the launched Sidekiq process.
+
+## Routing rules
+
+> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/59604) in GitLab 13.12.
+> - [Default routing rule value](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/97908) added in GitLab 15.4.
+
+We recommend most GitLab instances using routing rules to manage their Sidekiq
+queues. This allows administrators to choose single queue names for groups of
+job classes based on their attributes. The syntax is an ordered array of pairs of `[query, queue]`:
+
+1. The query is a [worker matching query](#worker-matching-query).
+1. The queue name must be a valid Sidekiq queue name. If the queue name
+ is `nil`, or an empty string, the worker is routed to the queue generated
+ by the name of the worker instead. (See [list of available job classes](#list-of-available-job-classes)
+ for more information).
+ The queue name does not have to match any existing queue name in the
+ list of available job classes.
+1. The first query matching a worker is chosen for that worker; later rules are
+ ignored.
+
+### Routing rules migration
+
+After the Sidekiq routing rules are changed, administrators must take care with
+the migration to avoid losing jobs entirely, especially in a system with long
+queues of jobs. The migration can be done by following the migration steps
+mentioned in [Sidekiq job migration](sidekiq_job_migration.md).
+
+### Detailed example
+
+This is a comprehensive example intended to show different possibilities. It is
+not a recommendation.
+
+1. Edit `/etc/gitlab/gitlab.rb`:
+
+ ```ruby
+ sidekiq['routing_rules'] = [
+ # Route all non-CPU-bound workers that are high urgency to `high-urgency` queue
+ ['resource_boundary!=cpu&urgency=high', 'high-urgency'],
+ # Route all database, gitaly and global search workers that are throttled to `throttled` queue
+ ['feature_category=database,gitaly,global_search&urgency=throttled', 'throttled'],
+ # Route all workers having contact with outside world to a `network-intenstive` queue
+ ['has_external_dependencies=true|feature_category=hooks|tags=network', 'network-intensive'],
+ # Route all import workers to the queues generated by the worker name, for
+ # example, JiraImportWorker to `jira_import`, SVNWorker to `svn_worker`
+ ['feature_category=import', 'import'],
+ # Wildcard matching, route the rest to `default` queue
+ ['*', 'default']
+ ]
+ ```
+
+ The `queue_groups` can then be set to match these generated queue names. For
+ instance:
+
+ ```ruby
+ sidekiq['queue_selector'] = false
+ sidekiq['queue_groups'] = [
+ # Run two high-urgency processes
+ 'high-urgency',
+ 'high-urgency',
+ # Run one process for throttled, network-intensive, import
+ 'throttled,network-intensive,import',
+ # Run one 'catchall' process on the default queue
+ 'default'
+ ]
+ ```
+
+1. Save the file and reconfigure GitLab:
+
+ ```shell
+ sudo gitlab-ctl reconfigure
+ ```
+
+## Queue selectors
+
+> - [Introduced](https://gitlab.com/gitlab-com/gl-infra/scalability/-/issues/45) in GitLab 12.8.
+> - [Sidekiq cluster, including queue selector, moved](https://gitlab.com/groups/gitlab-com/gl-infra/-/epics/181) to GitLab Free in 12.10.
+> - [Renamed from `experimental_queue_selector` to `queue_selector`](https://gitlab.com/gitlab-com/gl-infra/scalability/-/issues/147) in GitLab 13.6.
+
+The `queue_selector` option allows queue groups to be selected in a more general
+way using a [worker matching query](#worker-matching-query). After
+`queue_selector` is set, all `queue_groups` must follow the aforementioned
+syntax.
+
+### Using queue selectors
+
+1. Edit `/etc/gitlab/gitlab.rb`:
+
+ ```ruby
+ sidekiq['enable'] = true
+ sidekiq['routing_rules'] = [['*', nil]]
+ sidekiq['queue_selector'] = true
+ sidekiq['queue_groups'] = [
+ # Run all non-CPU-bound queues that are high urgency
+ 'resource_boundary!=cpu&urgency=high',
+ # Run all continuous integration and pages queues that are not high urgency
+ 'feature_category=continuous_integration,pages&urgency!=high',
+ # Run all queues
+ '*'
+ ]
+ ```
+
+1. Save the file and reconfigure GitLab:
+
+ ```shell
+ sudo gitlab-ctl reconfigure
+ ```
+
+### Negate settings
+
+This allows you to have the Sidekiq process work on every queue **except** the
+ones you list. This is generally only used when there are multiple Sidekiq
+nodes. In this example, we exclude all import-related jobs from a Sidekiq node.
+
+1. Edit `/etc/gitlab/gitlab.rb`:
+
+ ```ruby
+ sidekiq['routing_rules'] = [['*', nil]]
+ sidekiq['negate'] = true
+ sidekiq['queue_selector'] = true
+ sidekiq['queue_groups'] = [
+ "feature_category=importers"
+ ]
+ ```
+
+1. Save the file and reconfigure GitLab:
+
+ ```shell
+ sudo gitlab-ctl reconfigure
+ ```
+
+## Worker matching query
+
+GitLab provides a query syntax to match a worker based on its attributes. This
+query syntax is employed by both [routing rules](#routing-rules) and
+[queue selectors](#queue-selectors). A query includes two components:
+
+- Attributes that can be selected.
+- Operators used to construct a query.
+
+### Available attributes
+
+> [Introduced](https://gitlab.com/gitlab-com/gl-infra/scalability/-/issues/261) in GitLab 13.1 (`tags`).
+
+Queue matching query works upon the worker attributes, described in
+[Sidekiq style guide](../../development/sidekiq/index.md). We support querying
+based on a subset of worker attributes:
+
+- `feature_category` - the
+ [GitLab feature category](https://about.gitlab.com/direction/maturity/#category-maturity) the
+ queue belongs to. For example, the `merge` queue belongs to the
+ `source_code_management` category.
+- `has_external_dependencies` - whether or not the queue connects to external
+ services. For example, all importers have this set to `true`.
+- `urgency` - how important it is that this queue's jobs run
+ quickly. Can be `high`, `low`, or `throttled`. For example, the
+ `authorized_projects` queue is used to refresh user permissions, and
+ is `high` urgency.
+- `worker_name` - the worker name. Use this attribute to select a specific worker. Find all available names in [the job classes lists](#list-of-available-job-classes) below.
+- `name` - the queue name generated from the worker name. Use this attribute to select a specific queue. Because this is generated from
+ the worker name, it does not change based on the result of other routing
+ rules.
+- `resource_boundary` - if the queue is bound by `cpu`, `memory`, or
+ `unknown`. For example, the `ProjectExportWorker` is memory bound as it has
+ to load data in memory before saving it for export.
+- `tags` - short-lived annotations for queues. These are expected to frequently
+ change from release to release, and may be removed entirely.
+
+`has_external_dependencies` is a boolean attribute: only the exact
+string `true` is considered true, and everything else is considered
+false.
+
+`tags` is a set, which means that `=` checks for intersecting sets, and
+`!=` checks for disjoint sets. For example, `tags=a,b` selects queues
+that have tags `a`, `b`, or both. `tags!=a,b` selects queues that have
+neither of those tags.
+
+### Available operators
+
+Routing rules and queue selectors support the following operators, listed from
+highest to lowest precedence:
+
+- `|` - the logical `OR` operator. For example, `query_a|query_b` (where `query_a`
+ and `query_b` are queries made up of the other operators here) includes
+ queues that match either query.
+- `&` - the logical `AND` operator. For example, `query_a&query_b` (where
+ `query_a` and `query_b` are queries made up of the other operators here) will
+ only include queues that match both queries.
+- `!=` - the `NOT IN` operator. For example, `feature_category!=issue_tracking`
+ excludes all queues from the `issue_tracking` feature category.
+- `=` - the `IN` operator. For example, `resource_boundary=cpu` includes all
+ queues that are CPU bound.
+- `,` - the concatenate set operator. For example,
+ `feature_category=continuous_integration,pages` includes all queues from
+ either the `continuous_integration` category or the `pages` category. This
+ example is also possible using the OR operator, but allows greater brevity, as
+ well as being lower precedence.
+
+The operator precedence for this syntax is fixed: it's not possible to make `AND`
+have higher precedence than `OR`.
+
+As with the standard queue group syntax above, a single `*` as the
+entire queue group selects all queues.
+
+### List of available job classes
+
+For a list of the existing Sidekiq job classes and queues, check the following
+files:
+
+- [Queues for all GitLab editions](https://gitlab.com/gitlab-org/gitlab/-/blob/master/app/workers/all_queues.yml)
+- [Queues for GitLab Enterprise Editions only](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/app/workers/all_queues.yml)
diff --git a/doc/administration/sidekiq/sidekiq_job_migration.md b/doc/administration/sidekiq/sidekiq_job_migration.md
index f61021ad4e7..d79c13c8f29 100644
--- a/doc/administration/sidekiq/sidekiq_job_migration.md
+++ b/doc/administration/sidekiq/sidekiq_job_migration.md
@@ -17,24 +17,27 @@ If the Sidekiq routing rules are changed, administrators need to take care with
1. Listen to both the old and new queues.
1. Update the routing rules.
-1. Wait until there are no publishers dispatching jobs to the old queues.
-1. Run the [Rake tasks for future jobs](#future-jobs).
-1. Wait for the old queues to be empty.
+1. [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect.
+1. Run the [Rake tasks for migrating queued and future jobs](#migrate-queued-and-future-jobs).
1. Stop listening to the old queues.
-## Future jobs
+## Migrate queued and future jobs
Step 4 involves rewriting some Sidekiq job data for jobs that are already stored in Redis, but due to run in future. There are two sets of jobs to run in future: scheduled jobs and jobs to be retried. We provide a separate Rake task to migrate each set:
- `gitlab:sidekiq:migrate_jobs:retry` for jobs to be retried.
- `gitlab:sidekiq:migrate_jobs:scheduled` for scheduled jobs.
-Most of the time, running both at the same time is the correct choice. There are two separate tasks to allow for more fine-grained control where needed. To run both at once:
+Queued jobs that are yet to be run can also be migrated with a Rake task:
+
+- `gitlab:sidekiq:migrate_jobs:queued` for queued jobs to be performed asynchronously.
+
+Most of the time, running all three at the same time is the correct choice. There are three separate tasks to allow for more fine-grained control where needed. To run all three at once:
```shell
# omnibus-gitlab
-sudo gitlab-rake gitlab:sidekiq:migrate_jobs:retry gitlab:sidekiq:migrate_jobs:schedule
+sudo gitlab-rake gitlab:sidekiq:migrate_jobs:retry gitlab:sidekiq:migrate_jobs:schedule gitlab:sidekiq:migrate_jobs:queued
# source installations
-bundle exec rake gitlab:sidekiq:migrate_jobs:retry gitlab:sidekiq:migrate_jobs:schedule RAILS_ENV=production
+bundle exec rake gitlab:sidekiq:migrate_jobs:retry gitlab:sidekiq:migrate_jobs:schedule gitlab:sidekiq:migrate_jobs:queued RAILS_ENV=production
```
diff --git a/doc/architecture/blueprints/ci_data_decay/pipeline_partitioning.md b/doc/architecture/blueprints/ci_data_decay/pipeline_partitioning.md
index d61412ae1ed..9987118b6bc 100644
--- a/doc/architecture/blueprints/ci_data_decay/pipeline_partitioning.md
+++ b/doc/architecture/blueprints/ci_data_decay/pipeline_partitioning.md
@@ -248,10 +248,9 @@ smart enough to move rows between partitions on its own.
A partitioned table is called a **routing** table and it will use the `p_`
prefix which should help us with building automated tooling for query analysis.
-A table partition will be called **partition** and it can use the a
-physical partition ID as suffix, leaded by a `p` letter, for example
-`ci_builds_p101`. Existing CI tables will become **zero partitions** of the
-new routing tables. Depending on the chosen
+A table partition will be called **partition** and it can use the a physical
+partition ID as suffix, for example `ci_builds_101`. Existing CI tables will
+become **zero partitions** of the new routing tables. Depending on the chosen
[partitioning strategy](#how-do-we-want-to-partition-cicd-data) for a given
table, it is possible to have many logical partitions per one physical partition.
diff --git a/doc/ci/environments/index.md b/doc/ci/environments/index.md
index c4672b9dc7e..383127e651a 100644
--- a/doc/ci/environments/index.md
+++ b/doc/ci/environments/index.md
@@ -45,15 +45,20 @@ Deployments show up in this list only after a deployment job has created them.
## Search environments
-> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/10754) in GitLab 15.5.
+> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/10754) in GitLab 15.5.
+> - [Searching environments within a folder](https://gitlab.com/gitlab-org/gitlab/-/issues/373850) was introduced in GitLab 15.7 with [Feature flag `enable_environments_search_within_folder`](https://gitlab.com/gitlab-org/gitlab/-/issues/382108). Disabled by default.
To search environments by name:
1. On the top bar, select **Main menu > Projects** and find your project.
1. On the left sidebar, select **Deployments > Environments**.
-1. In the search bar, enter your search term. Matching applies from the
- beginning of the environment name. For example, `devel` matches the
- environment name `development`, but `elop` does not.
+1. In the search bar, enter your search term.
+ - The length of your **search term should be 3 or more characters**.
+ - Matching applies from the beginning of the environment name.
+ - For example, `devel` matches the environment name `development`, but `elop` does not.
+ - For environments with a folder name format, matching applies after the base folder name.
+ - For example when the name is `review/test-app`, search term `test` matches `review/test-app`.
+ - Also searching with the folder name prefixed like `review/test` matches `review/test-app`.
## Types of environments
diff --git a/doc/development/sidekiq/index.md b/doc/development/sidekiq/index.md
index a95e94cdd34..979432a2e5b 100644
--- a/doc/development/sidekiq/index.md
+++ b/doc/development/sidekiq/index.md
@@ -27,7 +27,7 @@ There are pages with additional detail on the following topics:
All workers should include `ApplicationWorker` instead of `Sidekiq::Worker`,
which adds some convenience methods and automatically sets the queue based on
-the [routing rules](../../administration/sidekiq/extra_sidekiq_routing.md#queue-routing-rules).
+the [routing rules](../../administration/sidekiq/processing_specific_job_classes.md#routing-rules).
## Retries
@@ -88,7 +88,7 @@ error rate.
Previously, each worker had its own queue, which was automatically set based on the
worker class name. For a worker named `ProcessSomethingWorker`, the queue name
would be `process_something`. You can now route workers to a specific queue using
-[queue routing rules](../../administration/sidekiq/extra_sidekiq_routing.md#queue-routing-rules).
+[queue routing rules](../../administration/sidekiq/processing_specific_job_classes.md#routing-rules).
In GDK, new workers are routed to a queue named `default`.
If you're not sure what queue a worker uses,
diff --git a/doc/index.md b/doc/index.md
index c359ec7b639..88b8c653aae 100644
--- a/doc/index.md
+++ b/doc/index.md
@@ -20,23 +20,16 @@ description: 'Learn how to use and administer GitLab, the most scalable Git-base
# GitLab Docs
-Welcome to [GitLab](https://about.gitlab.com/) documentation.
+Welcome to the GitLab documentation!
-Here you can access the complete documentation for GitLab, the single application for the
-[entire DevOps lifecycle](#the-entire-devops-lifecycle).
-
-## Overview
-
-No matter how you use GitLab, we have documentation for you.
-
-| Essential documentation | Essential documentation |
+| | |
|:------------------------|:------------------------|
-| [**User documentation**](user/index.md)<br>Discover features and concepts for GitLab users. | [**Administrator documentation**](administration/index.md)<br/>Everything GitLab self-managed administrators need to know. |
-| [**Contributing to GitLab**](#contributing-to-gitlab)<br/>At GitLab, everyone can contribute! | [**New to Git and GitLab?**](tutorials/index.md)<br/>We have the resources to get you started. |
-| [**Build an integration with GitLab**](#build-an-integration-with-gitlab)<br/>Consult our integration documentation. | [**Coming to GitLab from another platform?**](#coming-to-gitlab-from-another-platform)<br/>Consult our guides. |
-| [**Install GitLab**](https://about.gitlab.com/install/)<br/>Installation options for different platforms. | [**Customers**](subscriptions/index.md)<br/>Information for new and existing customers. |
-| [**Update GitLab**](update/index.md)<br/>Update your GitLab self-managed instance to the latest version. | [**Reference Architectures**](administration/reference_architectures/index.md)<br/>GitLab reference architectures. |
-| [**GitLab releases**](https://about.gitlab.com/releases/)<br/>What's new in GitLab. | |
+| [**Use GitLab**](user/index.md)<br>Get started with GitLab features and functionality. | [**Administer GitLab**](administration/index.md)<br/>Administer a self-managed GitLab instance. |
+| [**New to Git and GitLab?**](tutorials/index.md)<br/>Start learning about Git and GitLab. | [**Contribute to GitLab development**](#contributing-to-gitlab)<br/>Create new GitLab functionality and documentation. |
+| [**Coming to GitLab from another platform?**](#coming-to-gitlab-from-another-platform)<br/>Learn how to move to GitLab. | [**Build an integration with GitLab**](#build-an-integration-with-gitlab)<br/>Integrate with Jira and other common applications. |
+| [**Choose a subscription**](subscriptions/index.md)<br/>Determine which subscription tier makes sense for you. | [**Install GitLab**](https://about.gitlab.com/install/)<br/>Install GitLab on different platforms. |
+| [**Reference architectures**](administration/reference_architectures/index.md)<br/>View recommended deployments at scale. | [**Update GitLab**](update/index.md)<br/>Update your GitLab self-managed instance to the latest version. |
+| [**GitLab releases**](https://about.gitlab.com/releases/)<br/>See what's new in GitLab. | |
## Popular topics
diff --git a/doc/integration/advanced_search/elasticsearch.md b/doc/integration/advanced_search/elasticsearch.md
index 94493aa6958..741dccc31fe 100644
--- a/doc/integration/advanced_search/elasticsearch.md
+++ b/doc/integration/advanced_search/elasticsearch.md
@@ -783,8 +783,8 @@ additional process dedicated to indexing a set of queues (or queue group). This
ensure that indexing queues always have a dedicated worker, while the rest of the queues have
another dedicated worker to avoid contention.
-For this purpose, use the [queue selector](../../administration/sidekiq/extra_sidekiq_processes.md#queue-selector)
-option that allows a more general selection of queue groups using a [worker matching query](../../administration/sidekiq/extra_sidekiq_routing.md#worker-matching-query).
+For this purpose, use the [queue selectors](../../administration/sidekiq/processing_specific_job_classes.md#queue-selectors)
+option that allows a more general selection of queue groups using a [worker matching query](../../administration/sidekiq/processing_specific_job_classes.md#worker-matching-query).
To handle these two queue groups, we generally recommend one of the following two options. You can either:
@@ -804,8 +804,8 @@ To create both an indexing and a non-indexing Sidekiq process in one node:
```ruby
sidekiq['enable'] = true
- sidekiq['queue_selector'] = true
- sidekiq['queue_groups'] = [
+ sidekiq['queue_selector'] = true
+ sidekiq['queue_groups'] = [
"feature_category=global_search",
"feature_category!=global_search"
]
diff --git a/doc/integration/advanced_search/elasticsearch_troubleshooting.md b/doc/integration/advanced_search/elasticsearch_troubleshooting.md
index aa6613d6f1a..11d8ad0a596 100644
--- a/doc/integration/advanced_search/elasticsearch_troubleshooting.md
+++ b/doc/integration/advanced_search/elasticsearch_troubleshooting.md
@@ -248,7 +248,7 @@ sudo gitlab-rake gitlab:elastic:clear_locked_projects
If `ElasticCommitIndexerWorker` Sidekiq workers are failing with this error during indexing, it usually means that Elasticsearch is unable to keep up with the concurrency of indexing request. To address change the following settings:
- To decrease the indexing throughput you can decrease `Bulk request concurrency` (see [Advanced Search settings](elasticsearch.md#advanced-search-configuration)). This is set to `10` by default, but you change it to as low as 1 to reduce the number of concurrent indexing operations.
-- If changing `Bulk request concurrency` didn't help, you can use the [queue selector](../../administration/sidekiq/extra_sidekiq_processes.md#queue-selector) option to [limit indexing jobs only to specific Sidekiq nodes](elasticsearch.md#index-large-instances-with-dedicated-sidekiq-nodes-or-processes), which should reduce the number of indexing requests.
+- If changing `Bulk request concurrency` didn't help, you can use the [queue selector](../../administration/sidekiq/processing_specific_job_classes.md#queue-selectors) option to [limit indexing jobs only to specific Sidekiq nodes](elasticsearch.md#index-large-instances-with-dedicated-sidekiq-nodes-or-processes), which should reduce the number of indexing requests.
### Indexing is very slow or fails with `rejected execution of coordinating operation` messages
diff --git a/doc/update/index.md b/doc/update/index.md
index dbac4304897..81911ef3d82 100644
--- a/doc/update/index.md
+++ b/doc/update/index.md
@@ -477,7 +477,7 @@ and [Helm Chart deployments](https://docs.gitlab.com/charts/). They come with ap
### 15.5.0
-- GitLab 15.4.0 introduced a default [Sidekiq routing rule](../administration/sidekiq/extra_sidekiq_routing.md) that routes all jobs to the `default` queue. For instances using [queue selectors](../administration/sidekiq/extra_sidekiq_processes.md#queue-selector), this will cause [performance problems](https://gitlab.com/gitlab-com/gl-infra/scalability/-/issues/1991) as some Sidekiq processes will be idle.
+- GitLab 15.4.0 introduced a default [Sidekiq routing rule](../administration/sidekiq/extra_sidekiq_routing.md) that routes all jobs to the `default` queue. For instances using [queue selectors](../administration/sidekiq/processing_specific_job_classes.md#queue-selectors), this will cause [performance problems](https://gitlab.com/gitlab-com/gl-infra/scalability/-/issues/1991) as some Sidekiq processes will be idle.
- The default routing rule has been reverted in 15.5.4, so upgrading to that version or later will return to the previous behavior.
- If a GitLab instance now listens only to the `default` queue (which is not currently recommended), it will be required to add this routing rule back in `/etc/gitlab/gitlab.rb`:
@@ -490,7 +490,7 @@ and [Helm Chart deployments](https://docs.gitlab.com/charts/). They come with ap
- GitLab 15.4.0 includes a [batched background migration](#batched-background-migrations) to [remove incorrect values from `expire_at` in `ci_job_artifacts` table](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/89318).
This migration might take hours or days to complete on larger GitLab instances.
- By default, Gitaly and Praefect nodes use the time server at `pool.ntp.org`. If your instance can not connect to `pool.ntp.org`, [configure the `NTP_HOST` variable](../administration/gitaly/praefect.md#customize-time-server-setting).
-- GitLab 15.4.0 introduced a default [Sidekiq routing rule](../administration/sidekiq/extra_sidekiq_routing.md) that routes all jobs to the `default` queue. For instances using [queue selectors](../administration/sidekiq/extra_sidekiq_processes.md#queue-selector), this will cause [performance problems](https://gitlab.com/gitlab-com/gl-infra/scalability/-/issues/1991) as some Sidekiq processes will be idle.
+- GitLab 15.4.0 introduced a default [Sidekiq routing rule](../administration/sidekiq/extra_sidekiq_routing.md) that routes all jobs to the `default` queue. For instances using [queue selectors](../administration/sidekiq/processing_specific_job_classes.md#queue-selectors), this will cause [performance problems](https://gitlab.com/gitlab-com/gl-infra/scalability/-/issues/1991) as some Sidekiq processes will be idle.
- The default routing rule has been reverted in 15.4.5, so upgrading to that version or later will return to the previous behavior.
- If a GitLab instance now listens only to the `default` queue (which is not currently recommended), it will be required to add this routing rule back in `/etc/gitlab/gitlab.rb`:
@@ -514,7 +514,7 @@ and [Helm Chart deployments](https://docs.gitlab.com/charts/). They come with ap
[upgraded to 15.1](#1510) before upgrading to 15.2 (and later) due to a
configuration change in Rails that can result in inconsistent ETag key
generation.
-- Some Sidekiq workers were renamed in this release. To avoid any disruption, [run the Rake tasks to migrate any pending jobs](../administration/sidekiq/sidekiq_job_migration.md#future-jobs) before starting the upgrade to GitLab 15.2.0.
+- Some Sidekiq workers were renamed in this release. To avoid any disruption, [run the Rake tasks to migrate any pending jobs](../administration/sidekiq/sidekiq_job_migration.md#migrate-queued-and-future-jobs) before starting the upgrade to GitLab 15.2.0.
- Gitaly now executes its binaries in a [runtime location](https://gitlab.com/gitlab-org/gitaly/-/merge_requests/4670). By default on Omnibus GitLab,
this path is `/var/opt/gitlab/gitaly/run/`. If this location is mounted with `noexec`, merge requests generate the following error: