diff options
Diffstat (limited to 'doc/development/sidekiq')
| -rw-r--r-- | doc/development/sidekiq/compatibility_across_updates.md | 13 | ||||
| -rw-r--r-- | doc/development/sidekiq/idempotent_jobs.md | 5 | ||||
| -rw-r--r-- | doc/development/sidekiq/index.md | 16 | ||||
| -rw-r--r-- | doc/development/sidekiq/logging.md | 5 | ||||
| -rw-r--r-- | doc/development/sidekiq/worker_attributes.md | 6 |
5 files changed, 19 insertions, 26 deletions
diff --git a/doc/development/sidekiq/compatibility_across_updates.md b/doc/development/sidekiq/compatibility_across_updates.md index 96a3573d11a..1d369b5a970 100644 --- a/doc/development/sidekiq/compatibility_across_updates.md +++ b/doc/development/sidekiq/compatibility_across_updates.md @@ -18,18 +18,17 @@ several possible situations: ## Adding new workers -On GitLab.com, we [do not currently have a Sidekiq deployment in the -canary stage](https://gitlab.com/gitlab-org/gitlab/-/issues/19239). This -means that a new worker than can be scheduled from an HTTP endpoint may +On GitLab.com, we +[do not currently have a Sidekiq deployment in the canary stage](https://gitlab.com/gitlab-org/gitlab/-/issues/19239). +This means that a new worker than can be scheduled from an HTTP endpoint may be scheduled from canary but not run on Sidekiq until the full production deployment is complete. This can be several hours later than scheduling the job. For some workers, this will not be a problem. For -others - particularly [latency-sensitive -jobs](worker_attributes.md#latency-sensitive-jobs) - this will result in a poor user -experience. +others - particularly [latency-sensitive jobs](worker_attributes.md#latency-sensitive-jobs) - +this will result in a poor user experience. This only applies to new worker classes when they are first introduced. -As we recommend [using feature flags](../feature_flags/) as a general +As we recommend [using feature flags](../feature_flags/index.md) as a general development process, it's best to control the entire change (including scheduling of the new Sidekiq worker) with a feature flag. diff --git a/doc/development/sidekiq/idempotent_jobs.md b/doc/development/sidekiq/idempotent_jobs.md index a5ae8737ad1..5d1ebce763e 100644 --- a/doc/development/sidekiq/idempotent_jobs.md +++ b/doc/development/sidekiq/idempotent_jobs.md @@ -78,9 +78,8 @@ GitLab supports two deduplication strategies: - `until_executing`, which is the default strategy - `until_executed` -More [deduplication strategies have been -suggested](https://gitlab.com/gitlab-com/gl-infra/scalability/-/issues/195). If -you are implementing a worker that could benefit from a different +More [deduplication strategies have been suggested](https://gitlab.com/gitlab-com/gl-infra/scalability/-/issues/195). +If you are implementing a worker that could benefit from a different strategy, please comment in the issue. #### Until Executing diff --git a/doc/development/sidekiq/index.md b/doc/development/sidekiq/index.md index c9906c4c768..003f54d48b5 100644 --- a/doc/development/sidekiq/index.md +++ b/doc/development/sidekiq/index.md @@ -9,7 +9,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w We use [Sidekiq](https://github.com/mperham/sidekiq) as our background job processor. These guides are for writing jobs that will work well on GitLab.com and be consistent with our existing worker classes. For -information on administering GitLab, see [configuring Sidekiq](../../administration/sidekiq.md). +information on administering GitLab, see [configuring Sidekiq](../../administration/sidekiq/index.md). There are pages with additional detail on the following topics: @@ -27,12 +27,11 @@ There are pages with additional detail on the following topics: All workers should include `ApplicationWorker` instead of `Sidekiq::Worker`, which adds some convenience methods and automatically sets the queue based on -the [routing rules](../../administration/operations/extra_sidekiq_routing.md#queue-routing-rules). +the [routing rules](../../administration/sidekiq/extra_sidekiq_routing.md#queue-routing-rules). ## Retries -Sidekiq defaults to using [25 -retries](https://github.com/mperham/sidekiq/wiki/Error-Handling#automatic-job-retry), +Sidekiq defaults to using [25 retries](https://github.com/mperham/sidekiq/wiki/Error-Handling#automatic-job-retry), with back-off between each retry. 25 retries means that the last retry would happen around three weeks after the first attempt (assuming all 24 prior retries failed). @@ -64,7 +63,7 @@ error rate. Previously, each worker had its own queue, which was automatically set based on the worker class name. For a worker named `ProcessSomethingWorker`, the queue name would be `process_something`. You can now route workers to a specific queue using -[queue routing rules](../../administration/operations/extra_sidekiq_routing.md#queue-routing-rules). +[queue routing rules](../../administration/sidekiq/extra_sidekiq_routing.md#queue-routing-rules). In GDK, new workers are routed to a queue named `default`. If you're not sure what queue a worker uses, @@ -75,7 +74,7 @@ After adding a new worker, run `bin/rake gitlab:sidekiq:all_queues_yml:generate` to regenerate `app/workers/all_queues.yml` or `ee/app/workers/all_queues.yml` so that it can be picked up by -[`sidekiq-cluster`](../../administration/operations/extra_sidekiq_processes.md) +[`sidekiq-cluster`](../../administration/sidekiq/extra_sidekiq_processes.md) in installations that don't use routing rules. To learn more about potential changes, read [Use routing rules by default and deprecate queue selectors for self-managed](https://gitlab.com/groups/gitlab-com/gl-infra/-/epics/596). @@ -176,11 +175,10 @@ available in Sidekiq. There are possible workarounds such as: Some jobs have a weight declared. This is only used when running Sidekiq in the default execution mode - using -[`sidekiq-cluster`](../../administration/operations/extra_sidekiq_processes.md) +[`sidekiq-cluster`](../../administration/sidekiq/extra_sidekiq_processes.md) does not account for weights. -As we are [moving towards using `sidekiq-cluster` in -Free](https://gitlab.com/gitlab-org/gitlab/-/issues/34396), newly-added +As we are [moving towards using `sidekiq-cluster` in Free](https://gitlab.com/gitlab-org/gitlab/-/issues/34396), newly-added workers do not need to have weights specified. They can use the default weight, which is 1. diff --git a/doc/development/sidekiq/logging.md b/doc/development/sidekiq/logging.md index 474ea5de951..b461047ea47 100644 --- a/doc/development/sidekiq/logging.md +++ b/doc/development/sidekiq/logging.md @@ -11,8 +11,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w > [Introduced](https://gitlab.com/gitlab-com/gl-infra/scalability/-/issues/9) in GitLab 12.8. To have some more information about workers in the logs, we add -[metadata to the jobs in the form of an -`ApplicationContext`](../logging.md#logging-context-metadata-through-rails-or-grape-requests). +[metadata to the jobs in the form of an `ApplicationContext`](../logging.md#logging-context-metadata-through-rails-or-grape-requests). In most cases, when scheduling a job from a request, this context is already deducted from the request and added to the scheduled job. @@ -128,7 +127,7 @@ blocks: ## Arguments logging -As of GitLab 13.6, Sidekiq job arguments are logged by default, unless [`SIDEKIQ_LOG_ARGUMENTS`](../../administration/troubleshooting/sidekiq.md#log-arguments-to-sidekiq-jobs) +As of GitLab 13.6, Sidekiq job arguments are logged by default, unless [`SIDEKIQ_LOG_ARGUMENTS`](../../administration/sidekiq/sidekiq_troubleshooting.md#log-arguments-to-sidekiq-jobs) is disabled. By default, the only arguments logged are numeric arguments, because diff --git a/doc/development/sidekiq/worker_attributes.md b/doc/development/sidekiq/worker_attributes.md index 6820627f761..a1d24d0c392 100644 --- a/doc/development/sidekiq/worker_attributes.md +++ b/doc/development/sidekiq/worker_attributes.md @@ -86,13 +86,11 @@ but that always reduces work. To do this, we want to calculate the expected increase in total execution time and RPS (throughput) for the new shard. We can get these values from: -- The [Queue Detail - dashboard](https://dashboards.gitlab.net/d/sidekiq-queue-detail/sidekiq-queue-detail) +- The [Queue Detail dashboard](https://dashboards.gitlab.net/d/sidekiq-queue-detail/sidekiq-queue-detail) has values for the queue itself. For a new queue, we can look for queues that have similar patterns or are scheduled in similar circumstances. -- The [Shard Detail - dashboard](https://dashboards.gitlab.net/d/sidekiq-shard-detail/sidekiq-shard-detail) +- The [Shard Detail dashboard](https://dashboards.gitlab.net/d/sidekiq-shard-detail/sidekiq-shard-detail) has Total Execution Time and Throughput (RPS). The Shard Utilization panel displays if there is currently any excess capacity for this shard. |