diff options
Diffstat (limited to 'doc/administration/sidekiq')
-rw-r--r-- | doc/administration/sidekiq/extra_sidekiq_routing.md | 11 | ||||
-rw-r--r-- | doc/administration/sidekiq/index.md | 43 | ||||
-rw-r--r-- | doc/administration/sidekiq/processing_specific_job_classes.md | 27 | ||||
-rw-r--r-- | doc/administration/sidekiq/sidekiq_memory_killer.md | 4 |
4 files changed, 41 insertions, 44 deletions
diff --git a/doc/administration/sidekiq/extra_sidekiq_routing.md b/doc/administration/sidekiq/extra_sidekiq_routing.md deleted file mode 100644 index d1d65498fcc..00000000000 --- a/doc/administration/sidekiq/extra_sidekiq_routing.md +++ /dev/null @@ -1,11 +0,0 @@ ---- -redirect_to: 'processing_specific_job_classes.md#routing-rules' -remove_date: '2023-02-01' ---- - -This document was moved to [another location](processing_specific_job_classes.md#routing-rules). - -<!-- This redirect file can be deleted after <2023-02-01>. --> -<!-- Redirects that point to other docs in the same project expire in three months. --> -<!-- Redirects that point to docs in a different project or site (link is not relative and starts with `https:`) expire in one year. --> -<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/redirects.html --> diff --git a/doc/administration/sidekiq/index.md b/doc/administration/sidekiq/index.md index bf858476c0c..6fb12aa6ef9 100644 --- a/doc/administration/sidekiq/index.md +++ b/doc/administration/sidekiq/index.md @@ -9,7 +9,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w You can configure an external Sidekiq instance by using the Sidekiq that's bundled in the GitLab package. Sidekiq requires connection to the Redis, PostgreSQL, and Gitaly instances. -## Configure TCP access for PostgreSQL, Gitaly, and Redis +## Configure TCP access for PostgreSQL, Gitaly, and Redis on the GitLab instance By default, GitLab uses UNIX sockets and is not set up to communicate via TCP. To change this: @@ -32,12 +32,20 @@ By default, GitLab uses UNIX sockets and is not set up to communicate via TCP. T ## Gitaly - # Make Gitaly accept connections on all network interfaces - gitaly['listen_addr'] = "0.0.0.0:8075" - ## Set up the Gitaly token as a form of authentication since you are accessing Gitaly over the network - ## https://docs.gitlab.com/ee/administration/gitaly/configure_gitaly.html#about-the-gitaly-token - gitaly['auth_token'] = 'abc123secret' - praefect['auth_token'] = 'abc123secret' + gitaly['configuration'] = { + # ... + # + # Make Gitaly accept connections on all network interfaces + listen_addr: '0.0.0.0:8075', + auth: { + ## Set up the Gitaly token as a form of authentication since you are accessing Gitaly over the network + ## https://docs.gitlab.com/ee/administration/gitaly/configure_gitaly.html#about-the-gitaly-token + token: 'abc123secret', + }, + } + + gitaly['auth_token'] = '' + praefect['configuration'][:auth][:token] = 'abc123secret' gitlab_rails['gitaly_token'] = 'abc123secret' ## Redis configuration @@ -48,7 +56,6 @@ By default, GitLab uses UNIX sockets and is not set up to communicate via TCP. T redis['password'] = 'redis-password-goes-here' gitlab_rails['redis_password'] = 'redis-password-goes-here' - gitlab_rails['auto_migrate'] = false ``` 1. Run `reconfigure`: @@ -63,18 +70,6 @@ By default, GitLab uses UNIX sockets and is not set up to communicate via TCP. T sudo gitlab-ctl restart postgresql ``` -1. After the restart, set `auto_migrate` to `true` or comment to use the default settings: - - ```ruby - gitlab_rails['auto_migrate'] = true - ``` - -1. Run `reconfigure` again: - - ```shell - sudo gitlab-ctl reconfigure - ``` - ## Set up Sidekiq instance 1. SSH into the Sidekiq server. @@ -170,7 +165,7 @@ Updates to example must be made at: # Replace <database_host> and <database_password> gitlab_rails['db_host'] = '<database_host>' - gitlab_rails['db_port'] = '5432' + gitlab_rails['db_port'] = 5432 gitlab_rails['db_password'] = '<database_password>' ## Prevent database migrations from running on upgrade automatically gitlab_rails['auto_migrate'] = false @@ -257,7 +252,7 @@ To configure the metrics server: ```ruby sidekiq['metrics_enabled'] = true sidekiq['listen_address'] = "localhost" - sidekiq['listen_port'] = "8082" + sidekiq['listen_port'] = 8082 # Optionally log all the metrics server logs to log/sidekiq_exporter.log sidekiq['exporter_log_enabled'] = true @@ -299,7 +294,7 @@ To make health checks available from `localhost:8092`: ```ruby sidekiq['health_checks_enabled'] = true sidekiq['health_checks_listen_address'] = "localhost" - sidekiq['health_checks_listen_port'] = "8092" + sidekiq['health_checks_listen_port'] = 8092 ``` 1. Reconfigure GitLab: @@ -325,7 +320,7 @@ To enable LDAP with the synchronization worker for Sidekiq: 1. Edit `/etc/gitlab/gitlab.rb`: - ```ruby + ```ruby gitlab_rails['ldap_enabled'] = true gitlab_rails['prevent_ldap_sign_in'] = false gitlab_rails['ldap_servers'] = { diff --git a/doc/administration/sidekiq/processing_specific_job_classes.md b/doc/administration/sidekiq/processing_specific_job_classes.md index 61a154c8db2..26c2804f130 100644 --- a/doc/administration/sidekiq/processing_specific_job_classes.md +++ b/doc/administration/sidekiq/processing_specific_job_classes.md @@ -8,7 +8,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w WARNING: These are advanced settings. While they are used on GitLab.com, most GitLab -instances should add more processes that all listen to all queues. This is the +instances should only add more processes that listen to all queues. This is the same approach we take in our [Reference Architectures](../reference_architectures/index.md). GitLab has two options for creating Sidekiq processes that only handle specific @@ -17,7 +17,7 @@ job classes: 1. [Routing rules](#routing-rules) are used on GitLab.com. They direct jobs inside the application to queue names configured by administrators. This lowers the load on Redis, which is important on very large-scale deployments. -1. [Queue selectors](#queue-selectors) perform the job selection outside the +1. [Queue selectors](#queue-selectors-deprecated) perform the job selection outside the application, when starting the Sidekiq process. This was used on GitLab.com until September 2021, and is retained for compatibility reasons. @@ -106,11 +106,19 @@ not a recommendation. sudo gitlab-ctl reconfigure ``` -## Queue selectors +<!--- start_remove The following content will be removed on remove_date: '2024-08-22' --> + +## Queue selectors (deprecated) > - [Introduced](https://gitlab.com/gitlab-com/gl-infra/scalability/-/issues/45) in GitLab 12.8. > - [Sidekiq cluster, including queue selector, moved](https://gitlab.com/groups/gitlab-com/gl-infra/-/epics/181) to GitLab Free in 12.10. > - [Renamed from `experimental_queue_selector` to `queue_selector`](https://gitlab.com/gitlab-com/gl-infra/scalability/-/issues/147) in GitLab 13.6. +> - [Deprecated](https://gitlab.com/gitlab-org/gitlab/-/issues/390787) in GitLab 15.9. + +WARNING: +This feature was [deprecated](https://gitlab.com/gitlab-org/gitlab/-/issues/390787) in GitLab 15.9 +and is planned for removal in 17.0. Most instances should have [all processes to listen to all queues](extra_sidekiq_processes.md#start-multiple-processes). +Another alternative is to use [routing rules](#routing-rules) (be warned this is an advanced setting). This change is a breaking change. The `queue_selector` option allows queue groups to be selected in a more general way using a [worker matching query](#worker-matching-query). After @@ -141,7 +149,12 @@ syntax. sudo gitlab-ctl reconfigure ``` -### Negate settings +### Negate settings (deprecated) + +WARNING: +This feature was [deprecated](https://gitlab.com/gitlab-org/gitlab/-/issues/390787) in GitLab 15.9 +and is planned for removal in 17.0. Most instances should have [all processes to listen to all queues](extra_sidekiq_processes.md#start-multiple-processes). +Another alternative is to use [routing rules](#routing-rules) (be warned this is an advanced setting). This change is a breaking change. This allows you to have the Sidekiq process work on every queue **except** the ones you list. This is generally only used when there are multiple Sidekiq @@ -168,7 +181,7 @@ nodes. In this example, we exclude all import-related jobs from a Sidekiq node. We recommend GitLab deployments add more Sidekiq processes listening to all queues, as in the [Reference Architectures](../reference_architectures/index.md). For very large-scale deployments, we recommend -[routing rules](#routing-rules) instead of [queue selectors](#queue-selectors). We use routing rules on GitLab.com as +[routing rules](#routing-rules) instead of [queue selectors](#queue-selectors-deprecated). We use routing rules on GitLab.com as it helps to lower the load on Redis. To migrate from queue selectors to routing rules: @@ -255,11 +268,13 @@ in a queue group entry is 1, while `min_concurrency` is set to `0`, and `max_con concurrency is set to `2` instead. A concurrency of `2` might be too low in most cases, except for very highly-CPU bound tasks. +<!--- end_remove --> + ## Worker matching query GitLab provides a query syntax to match a worker based on its attributes. This query syntax is employed by both [routing rules](#routing-rules) and -[queue selectors](#queue-selectors). A query includes two components: +[queue selectors](#queue-selectors-deprecated). A query includes two components: - Attributes that can be selected. - Operators used to construct a query. diff --git a/doc/administration/sidekiq/sidekiq_memory_killer.md b/doc/administration/sidekiq/sidekiq_memory_killer.md index cb27d44a2e6..63d93919129 100644 --- a/doc/administration/sidekiq/sidekiq_memory_killer.md +++ b/doc/administration/sidekiq/sidekiq_memory_killer.md @@ -56,9 +56,7 @@ Sidekiq memory limits are controlled using environment variables. If jobs do not finish during that time, all currently running jobs are interrupted with a `SIGTERM` signal sent to the Sidekiq process. -- `GITLAB_MEMORY_WATCHDOG_ENABLED`: enabled by default. Set the `GITLAB_MEMORY_WATCHDOG_ENABLED` to false, to use legacy - Daemon Sidekiq Memory Killer implementation used prior GitLab 15.9. Support for setting `GITLAB_MEMORY_WATCHDOG_ENABLED` - will be removed in GitLab 16.0. +- `GITLAB_MEMORY_WATCHDOG_ENABLED`: enabled by default. Set the `GITLAB_MEMORY_WATCHDOG_ENABLED` to false, to disable Watchdog from running. ### Monitor worker restarts |