diff options
Diffstat (limited to 'doc/development/sidekiq_style_guide.md')
| -rw-r--r-- | doc/development/sidekiq_style_guide.md | 86 |
1 files changed, 70 insertions, 16 deletions
diff --git a/doc/development/sidekiq_style_guide.md b/doc/development/sidekiq_style_guide.md index 24570cfc07b..13ae39997bc 100644 --- a/doc/development/sidekiq_style_guide.md +++ b/doc/development/sidekiq_style_guide.md @@ -1,3 +1,9 @@ +--- +stage: none +group: unassigned +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +--- + # Sidekiq Style Guide This document outlines various guidelines that should be followed when adding or @@ -21,7 +27,10 @@ After adding a new queue, run `bin/rake gitlab:sidekiq:all_queues_yml:generate` to regenerate `app/workers/all_queues.yml` or `ee/app/workers/all_queues.yml` so that it can be picked up by -[`sidekiq-cluster`](../administration/operations/extra_sidekiq_processes.md). +[`sidekiq-cluster`](../administration/operations/extra_sidekiq_processes.md). +Additionally, run +`bin/rake gitlab:sidekiq:sidekiq_queues_yml:generate` to regenerate +`config/sidekiq_queues.yml`. ## Queue Namespaces @@ -112,7 +121,6 @@ As a general rule, a worker can be considered idempotent if: A good example of that would be a cache expiration worker. -NOTE: **Note:** A job scheduled for an idempotent worker will automatically be [deduplicated](#deduplication) when an unstarted job with the same arguments is already in the queue. @@ -152,7 +160,6 @@ end It's encouraged to only have the `idempotent!` call in the top-most worker class, even if the `perform` method is defined in another class or module. -NOTE: **Note:** If the worker class is not marked as idempotent, a cop will fail. Consider skipping the cop if you're not confident your job can safely run multiple times. @@ -165,6 +172,22 @@ job. The work is skipped because the same work would be done by the job that was scheduled first; by the time the second job executed, the first job would do nothing. +#### Strategies + +GitLab supports two deduplication strategies: + +- `until_executing` +- `until_executed` + +More [deduplication strategies have been +suggested](https://gitlab.com/gitlab-com/gl-infra/scalability/-/issues/195). If +you are implementing a worker that could benefit from a different +strategy, please comment in the issue. + +##### Until Executing + +This strategy takes a lock when a job is added to the queue, and removes that lock before the job starts. + For example, `AuthorizedProjectsWorker` takes a user ID. When the worker runs, it recalculates a user's authorizations. GitLab schedules this job each time an action potentially changes a user's @@ -173,10 +196,47 @@ same time, the second job can be skipped if the first job hasn't begun, because when the first job runs, it creates the authorizations for both projects. +```ruby +module AuthorizedProjectUpdate + class UserRefreshOverUserRangeWorker + include ApplicationWorker + + deduplicate :until_executing + idempotent! + + # ... + end +end +``` + +##### Until Executed + +This strategy takes a lock when a job is added to the queue, and removes that lock after the job finishes. +It can be used to prevent jobs from running simultaneously multiple times. + +```ruby +module Ci + class BuildTraceChunkFlushWorker + include ApplicationWorker + + deduplicate :until_executed + idempotent! + + # ... + end +end +``` + +#### Scheduling jobs in the future + GitLab doesn't skip jobs scheduled in the future, as we assume that the state will have changed by the time the job is scheduled to -execute. If you do want to deduplicate jobs scheduled in the future -this can be specified on the worker as follows: +execute. Deduplication of jobs scheduled in the feature is possible +for both `until_executed` and `until_executing` strategies. + +If you do want to deduplicate jobs scheduled in the future, +this can be specified on the worker by passing `including_scheduled: true` argument +when defining deduplication strategy: ```ruby module AuthorizedProjectUpdate @@ -191,11 +251,7 @@ module AuthorizedProjectUpdate end ``` -This strategy is called `until_executing`. More [deduplication -strategies have been -suggested](https://gitlab.com/gitlab-com/gl-infra/scalability/-/issues/195). If -you are implementing a worker that could benefit from a different -strategy, please comment in the issue. +#### Troubleshooting If the automatic deduplication were to cause issues in certain queues. This can be temporarily disabled by enabling a feature flag @@ -429,9 +485,7 @@ class ExternalDependencyWorker end ``` -NOTE: **Note:** -Note that a job cannot be both high urgency and have -external dependencies. +A job cannot be both high urgency and have external dependencies. ## CPU-bound and Memory-bound Workers @@ -645,8 +699,8 @@ blocks: ## Arguments logging -When [`SIDEKIQ_LOG_ARGUMENTS`](../administration/troubleshooting/sidekiq.md#log-arguments-to-sidekiq-jobs) -is enabled, Sidekiq job arguments will be logged. +As of GitLab 13.6, Sidekiq job arguments will be logged by default, unless [`SIDEKIQ_LOG_ARGUMENTS`](../administration/troubleshooting/sidekiq.md#log-arguments-to-sidekiq-jobs) +is disabled. By default, the only arguments logged are numeric arguments, because arguments of other types could contain sensitive information. To @@ -767,7 +821,7 @@ This approach requires multiple releases. ##### Parameter hash This approach will not require multiple releases if an existing worker already -utilizes a parameter hash. +uses a parameter hash. 1. Use a parameter hash in the worker to allow future flexibility. |