From a683d38a36aadac5c244299d37325424d85fa9e5 Mon Sep 17 00:00:00 2001 From: GitLab Bot Date: Mon, 26 Oct 2020 18:08:27 +0000 Subject: Add latest changes from gitlab-org/gitlab@master --- doc/development/sidekiq_style_guide.md | 63 ++++++++++++++++++++++++++++++---- 1 file changed, 56 insertions(+), 7 deletions(-) (limited to 'doc') diff --git a/doc/development/sidekiq_style_guide.md b/doc/development/sidekiq_style_guide.md index 24570cfc07b..b341c31e21d 100644 --- a/doc/development/sidekiq_style_guide.md +++ b/doc/development/sidekiq_style_guide.md @@ -165,6 +165,22 @@ job. The work is skipped because the same work would be done by the job that was scheduled first; by the time the second job executed, the first job would do nothing. +#### Strategies + +GitLab supports two deduplication strategies: + +- `until_executing` +- `until_executed` + +More [deduplication strategies have been +suggested](https://gitlab.com/gitlab-com/gl-infra/scalability/-/issues/195). If +you are implementing a worker that could benefit from a different +strategy, please comment in the issue. + +##### Until Executing + +This strategy takes a lock when a job is added to the queue, and removes that lock before the job starts. + For example, `AuthorizedProjectsWorker` takes a user ID. When the worker runs, it recalculates a user's authorizations. GitLab schedules this job each time an action potentially changes a user's @@ -173,10 +189,47 @@ same time, the second job can be skipped if the first job hasn't begun, because when the first job runs, it creates the authorizations for both projects. +```ruby +module AuthorizedProjectUpdate + class UserRefreshOverUserRangeWorker + include ApplicationWorker + + deduplicate :until_executing + idempotent! + + # ... + end +end +``` + +##### Until Executed + +This strategy takes a lock when a job is added to the queue, and removes that lock after the job finishes. +It can be used to prevent jobs from running simultaneously multiple times. + +```ruby +module Ci + class BuildTraceChunkFlushWorker + include ApplicationWorker + + deduplicate :until_executed + idempotent! + + # ... + end +end +``` + +#### Scheduling jobs in the future + GitLab doesn't skip jobs scheduled in the future, as we assume that the state will have changed by the time the job is scheduled to -execute. If you do want to deduplicate jobs scheduled in the future -this can be specified on the worker as follows: +execute. Deduplication of jobs scheduled in the feature is possible +for both `until_executed` and `until_executing` strategies. + +If you do want to deduplicate jobs scheduled in the future, +this can be specified on the worker by passing `including_scheduled: true` argument +when defining deduplication strategy: ```ruby module AuthorizedProjectUpdate @@ -191,11 +244,7 @@ module AuthorizedProjectUpdate end ``` -This strategy is called `until_executing`. More [deduplication -strategies have been -suggested](https://gitlab.com/gitlab-com/gl-infra/scalability/-/issues/195). If -you are implementing a worker that could benefit from a different -strategy, please comment in the issue. +#### Troubleshooting If the automatic deduplication were to cause issues in certain queues. This can be temporarily disabled by enabling a feature flag -- cgit v1.2.3