diff options
Diffstat (limited to 'doc/development/feature_flags/index.md')
| -rw-r--r-- | doc/development/feature_flags/index.md | 45 |
1 files changed, 45 insertions, 0 deletions
diff --git a/doc/development/feature_flags/index.md b/doc/development/feature_flags/index.md index 87d2da016d6..30837ac8f3f 100644 --- a/doc/development/feature_flags/index.md +++ b/doc/development/feature_flags/index.md @@ -144,6 +144,14 @@ An `experiment` feature flag should conform to the same standards as a `developm although the interface has some differences. An experiment feature flag should have a rollout issue, created using the [Experiment Tracking template](https://gitlab.com/gitlab-org/gitlab/-/blob/master/.gitlab/issue_templates/Experiment%20Rollout.md). More information can be found in the [experiment guide](../experiment_guide/index.md). +### `worker` type + +`worker` feature flags are used for controlling Sidekiq workers behavior, such as deferring Sidekiq jobs. + +`worker` feature flags likely do not have any YAML definition as the name could be dynamically generated using +the worker name itself, e.g. `defer_sidekiq_jobs_AuthorizedProjectsWorker`. Some examples for using `worker` type feature +flags can be found in [deferring Sidekiq jobs](#deferring-sidekiq-jobs). + ## Feature flag definition and validation > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/229161) in GitLab 13.3. @@ -698,3 +706,40 @@ test is written to enable/disable a feature flag explicitly. When a feature flag is changed on Staging or on GitLab.com, a Slack message will be posted to the `#qa-staging` or `#qa-production` channels to inform the pipeline triage DRI so that they can more easily determine if any failures are related to a feature flag change. However, if you are working on a change you can help to avoid unexpected failures by [confirming that the end-to-end tests pass with a feature flag enabled.](../testing_guide/end_to_end/feature_flags.md#confirming-that-end-to-end-tests-pass-with-a-feature-flag-enabled) + +## Controlling Sidekiq worker behavior with feature flags + +Feature flags with [`worker` type](#worker-type) can be used to control the behavior of a Sidekiq worker. + +### Deferring Sidekiq jobs + +Feature flags with the format of `defer_sidekiq_jobs_{WorkerName}` delay the execution of the worker +by scheduling the job at a later time. +Deferring jobs can be useful during an incident where contentious behavior from +worker instances are saturating infrastructure resources (such as database and database connection pool). +The implementation can be found at [DeferJobs Sidekiq server middleware](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/sidekiq_middleware/defer_jobs.rb). + +NOTE: +Jobs are deferred indefinitely as long as the feature flag is enabled. It is important to disable the +feature flag after the worker is deemed safe to continue processing. + +When set to true, 100% of the jobs are deferred. When you want processing to resume, you can +use a **percentage of time** rollout. For example: + +```shell +# defer 100% of the jobs +/chatops run feature set defer_sidekiq_jobs_SlowRunningWorker true + +# defer 99% of the jobs, only letting 1% processed +/chatops run feature set defer_sidekiq_jobs_SlowRunningWorker 99 + +# defer 50% of the jobs +/chatops run feature set defer_sidekiq_jobs_SlowRunningWorker 50 + +# stop deferring the jobs, jobs are being processed normally +/chatops run feature set defer_sidekiq_jobs_SlowRunningWorker false +``` + +NOTE: +The percentage of time value denotes the percentage of time the jobs are being deferred (instead of being processed). +For example, setting to `99` means only 1% of the jobs are being processed at random. |