diff options
Diffstat (limited to 'doc/development/feature_flags')
| -rw-r--r-- | doc/development/feature_flags/controls.md | 18 | ||||
| -rw-r--r-- | doc/development/feature_flags/development.md | 594 | ||||
| -rw-r--r-- | doc/development/feature_flags/index.md | 657 | ||||
| -rw-r--r-- | doc/development/feature_flags/process.md | 177 |
4 files changed, 627 insertions, 819 deletions
diff --git a/doc/development/feature_flags/controls.md b/doc/development/feature_flags/controls.md index 21ac152c469..fc327a2defc 100644 --- a/doc/development/feature_flags/controls.md +++ b/doc/development/feature_flags/controls.md @@ -36,7 +36,7 @@ easier to measure the impact of both separately. The GitLab feature library (using [Flipper](https://github.com/jnunemaker/flipper), and covered in the [Feature -Flags process](process.md) guide) supports rolling out changes to a percentage of +Flags process](https://about.gitlab.com/handbook/product-development-flow/feature-flag-lifecycle) guide) supports rolling out changes to a percentage of time to users. This in turn can be controlled using [GitLab ChatOps](../../ci/chatops/index.md). For an up to date list of feature flag commands please see [the source @@ -240,7 +240,7 @@ To disable a feature flag that has been enabled for a specific project you can r /chatops run feature set --group=gitlab-org some_feature false ``` -You cannot selectively disable feature flags for a specific project/group/user without applying a [specific method of implementing](development.md#selectively-disable-by-actor) the feature flags. +You cannot selectively disable feature flags for a specific project/group/user without applying a [specific method of implementing](index.md#selectively-disable-by-actor) the feature flags. ### Feature flag change logging @@ -276,17 +276,19 @@ and reduces confidence in our testing suite covering all possible combinations. Additionally, a feature flag overwritten in some of the environments can result in undefined and untested system behavior. -To remove a feature flag: +To remove a feature flag, open **one merge request** to make the changes. In the MR: -1. Open a new merge request with the ~"feature flag" label so - release managers are aware the changes are hidden behind a feature flag. +1. Add the ~"feature flag" label so release managers are aware the changes are hidden behind a feature flag. 1. If the merge request has to be picked into a stable branch, add the appropriate `~"Pick into X.Y"` label, for example `~"Pick into 13.0"`. - See [the feature flag process](process.md#including-a-feature-behind-feature-flag-in-the-final-release) + See [the feature flag process](https://about.gitlab.com/handbook/product-development-flow/feature-flag-lifecycle#including-a-feature-behind-feature-flag-in-the-final-release) for further details. -1. Remove all references to the feature flag from the codebase. +1. Remove all references to the feature flag from the codebase, including tests. 1. Remove the YAML definition for the feature from the repository. -1. Clean up the feature flag from all environments with `/chatops run feature delete some_feature`. + +Once the above MR has been merged, you should: + +1. [Clean up the feature flag from all environments](#cleanup-chatops) with `/chatops run feature delete some_feature`. 1. Close the rollout issue for the feature flag after the feature flag is removed from the codebase. ### Cleanup ChatOps diff --git a/doc/development/feature_flags/development.md b/doc/development/feature_flags/development.md index 0cdfa3e68d7..79efd6d5502 100644 --- a/doc/development/feature_flags/development.md +++ b/doc/development/feature_flags/development.md @@ -1,593 +1,7 @@ --- -type: reference, dev -stage: none -group: Development -info: "See the Technical Writers assigned to Development Guidelines: https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments-to-development-guidelines" +redirect_to: 'index.md' --- -# Developing with feature flags - -This document provides guidelines on how to use feature flags -in the GitLab codebase to conditionally enable features -and test them. - -Features that are developed and merged behind a feature flag -should not include a changelog entry. The entry should be added either in the merge -request removing the feature flag or the merge request where the default value of -the feature flag is set to enabled. If the feature contains any database migrations, it -*should* include a changelog entry for the database changes. - -WARNING: -All newly-introduced feature flags should be [disabled by default](process.md#feature-flags-in-gitlab-development). - -NOTE: -This document is the subject of continued work as part of an epic to [improve internal usage of Feature Flags](https://gitlab.com/groups/gitlab-org/-/epics/3551). Raise any suggestions as new issues and attach them to the epic. - -## Risk of a broken master (main) branch - -Feature flags **must** be used in the MR that introduces them. Not doing so causes a -[broken master](https://about.gitlab.com/handbook/engineering/workflow/#broken-master) scenario due -to the `rspec:feature-flags` job that only runs on the `master` branch. - -## Types of feature flags - -Choose a feature flag type that matches the expected usage. - -### `development` type - -`development` feature flags are short-lived feature flags, -used so that unfinished code can be deployed in production. - -A `development` feature flag should have a rollout issue, -ideally created using the [Feature Flag Roll Out template](https://gitlab.com/gitlab-org/gitlab/-/blob/master/.gitlab/issue_templates/Feature%20Flag%20Roll%20Out.md). - -This is the default type used when calling `Feature.enabled?`. - -### `ops` type - -`ops` feature flags are long-lived feature flags that control operational aspects -of GitLab product behavior. For example, feature flags that disable features that might -have a performance impact, like special Sidekiq worker behavior. - -`ops` feature flags likely do not have rollout issues, as it is hard to -predict when they are enabled or disabled. - -To use `ops` feature flags, you must append `type: :ops` to `Feature.enabled?` -invocations: - -```ruby -# Check if feature flag is enabled -Feature.enabled?(:my_ops_flag, project, type: :ops) - -# Check if feature flag is disabled -Feature.disabled?(:my_ops_flag, project, type: :ops) - -# Push feature flag to Frontend -push_frontend_feature_flag(:my_ops_flag, project, type: :ops) -``` - -### `experiment` type - -`experiment` feature flags are used for A/B testing on GitLab.com. - -An `experiment` feature flag should conform to the same standards as a `development` feature flag, -although the interface has some differences. More information can be found in the [experiment guide](../experiment_guide/index.md). - -## Feature flag definition and validation - -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/229161) in GitLab 13.3. - -During development (`RAILS_ENV=development`) or testing (`RAILS_ENV=test`) all feature flag usage is being strictly validated. - -This process is meant to ensure consistent feature flag usage in the codebase. All feature flags **must**: - -- Be known. Only use feature flags that are explicitly defined. -- Not be defined twice. They have to be defined either in FOSS or EE, but not both. -- Use a valid and consistent `type:` across all invocations. -- Use the same `default_enabled:` across all invocations. -- Have an owner. - -All feature flags known to GitLab are self-documented in YAML files stored in: - -- [`config/feature_flags`](https://gitlab.com/gitlab-org/gitlab/-/tree/master/config/feature_flags) -- [`ee/config/feature_flags`](https://gitlab.com/gitlab-org/gitlab/-/tree/master/ee/config/feature_flags) - -Each feature flag is defined in a separate YAML file consisting of a number of fields: - -| Field | Required | Description | -|---------------------|----------|----------------------------------------------------------------| -| `name` | yes | Name of the feature flag. | -| `type` | yes | Type of feature flag. | -| `default_enabled` | yes | The default state of the feature flag that is strictly validated, with `default_enabled:` passed as an argument. | -| `introduced_by_url` | no | The URL to the Merge Request that introduced the feature flag. | -| `rollout_issue_url` | no | The URL to the Issue covering the feature flag rollout. | -| `group` | no | The [group](https://about.gitlab.com/handbook/product/categories/#devops-stages) that owns the feature flag. | - -NOTE: -All validations are skipped when running in `RAILS_ENV=production`. - -## Create a new feature flag - -The GitLab codebase provides [`bin/feature-flag`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/bin/feature-flag), -a dedicated tool to create new feature flag definitions. -The tool asks various questions about the new feature flag, then creates -a YAML definition in `config/feature_flags` or `ee/config/feature_flags`. - -Only feature flags that have a YAML definition file can be used when running the development or testing environments. - -```shell -$ bin/feature-flag my_feature_flag ->> Specify the group introducing the feature flag, like `group::apm`: -?> group::memory - ->> URL of the MR introducing the feature flag (enter to skip): -?> https://gitlab.com/gitlab-org/gitlab/-/merge_requests/38602 - ->> Open this URL and fill in the rest of the details: -https://gitlab.com/gitlab-org/gitlab/-/issues/new?issue%5Btitle%5D=%5BFeature+flag%5D+Rollout+of+%60test-flag%60&issuable_template=Feature+Flag+Roll+Out - ->> URL of the rollout issue (enter to skip): -?> https://gitlab.com/gitlab-org/gitlab/-/issues/232533 -create config/feature_flags/development/my_feature_flag.yml ---- -name: my_feature_flag -introduced_by_url: https://gitlab.com/gitlab-org/gitlab/-/merge_requests/38602 -rollout_issue_url: https://gitlab.com/gitlab-org/gitlab/-/issues/232533 -group: group::memory -type: development -default_enabled: false -``` - -NOTE: -To create a feature flag that is only used in EE, add the `--ee` flag: `bin/feature-flag --ee` - -## Delete a feature flag - -See [cleaning up feature flags](controls.md#cleaning-up) for more information about -deleting feature flags. - -## Develop with a feature flag - -There are two main ways of using Feature Flags in the GitLab codebase: - -- [Backend code (Rails)](#backend) -- [Frontend code (VueJS)](#frontend) - -### Backend - -The feature flag interface is defined in [`lib/feature.rb`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/feature.rb). -This interface provides a set of methods to check if the feature flag is enabled or disabled: - -```ruby -if Feature.enabled?(:my_feature_flag, project) - # execute code if feature flag is enabled -else - # execute code if feature flag is disabled -end - -if Feature.disabled?(:my_feature_flag, project) - # execute code if feature flag is disabled -end -``` - -In rare cases you may want to make a feature enabled by default. If so, explain the reasoning -in the merge request. Use `default_enabled: true` when checking the feature flag state: - -```ruby -if Feature.enabled?(:feature_flag, project, default_enabled: true) - # execute code if feature flag is enabled -else - # execute code if feature flag is disabled -end - -if Feature.disabled?(:my_feature_flag, project, default_enabled: true) - # execute code if feature flag is disabled -end -``` - -If not specified, `default_enabled` is `false`. - -To force reading the `default_enabled` value from the relative YAML definition file, use -`default_enabled: :yaml`: - -```ruby -if Feature.enabled?(:feature_flag, project, default_enabled: :yaml) - # execute code if feature flag is enabled -end -``` - -```ruby -if Feature.disabled?(:feature_flag, project, default_enabled: :yaml) - # execute code if feature flag is disabled -end -``` - -This allows to use the same feature flag check across various parts of the codebase and -maintain the status of `default_enabled` in the YAML definition file which is the SSOT. - -If `default_enabled: :yaml` is used, a YAML definition is expected or an error is raised -in development or test environment, while returning `false` on production. - -If not specified, the default feature flag type for `Feature.enabled?` and `Feature.disabled?` -is `type: development`. For all other feature flag types, you must specify the `type:`: - -```ruby -if Feature.enabled?(:feature_flag, project, type: :ops) - # execute code if ops feature flag is enabled -else - # execute code if ops feature flag is disabled -end - -if Feature.disabled?(:my_feature_flag, project, type: :ops) - # execute code if feature flag is disabled -end -``` - -WARNING: -Don't use feature flags at application load time. For example, using the `Feature` class in -`config/initializers/*` or at the class level could cause an unexpected error. This error occurs -because a database that a feature flag adapter might depend on doesn't exist at load time -(especially for fresh installations). Checking for the database's existence at the caller isn't -recommended, as some adapters don't require a database at all (for example, the HTTP adapter). The -feature flag setup check must be abstracted in the `Feature` namespace. This approach also requires -application reload when the feature flag changes. You must therefore ask SREs to reload the -Web/API/Sidekiq fleet on production, which takes time to fully rollout/rollback the changes. For -these reasons, use environment variables (for example, `ENV['YOUR_FEATURE_NAME']`) or `gitlab.yml` -instead. - -Here's an example of a pattern that you should avoid: - -```ruby -class MyClass - if Feature.enabled?(:...) - new_process - else - legacy_process - end -end -``` - -### Frontend - -Use the `push_frontend_feature_flag` method for frontend code, which is -available to all controllers that inherit from `ApplicationController`. You can use -this method to expose the state of a feature flag, for example: - -```ruby -before_action do - # Prefer to scope it per project or user e.g. - push_frontend_feature_flag(:vim_bindings, project) -end - -def index - # ... -end - -def edit - # ... -end -``` - -You can then check the state of the feature flag in JavaScript as follows: - -```javascript -if ( gon.features.vimBindings ) { - // ... -} -``` - -The name of the feature flag in JavaScript is always camelCase, -so checking for `gon.features.vim_bindings` would not work. - -See the [Vue guide](../fe_guide/vue.md#accessing-feature-flags) for details about -how to access feature flags in a Vue component. - -In rare cases you may want to make a feature enabled by default. If so, explain the reasoning -in the merge request. Use `default_enabled: true` when checking the feature flag state: - -```ruby -before_action do - # Prefer to scope it per project or user e.g. - push_frontend_feature_flag(:vim_bindings, project, default_enabled: true) -end -``` - -If not specified, the default feature flag type for `push_frontend_feature_flag` -is `type: development`. For all other feature flag types, you must specify the `type:`: - -```ruby -before_action do - push_frontend_feature_flag(:vim_bindings, project, type: :ops) -end -``` - -### Feature actors - -**It is strongly advised to use actors with feature flags.** Actors provide a simple -way to enable a feature flag only for a given project, group or user. This makes debugging -easier, as you can filter logs and errors for example, based on actors. This also makes it possible -to enable the feature on the `gitlab-org` or `gitlab-com` groups first, while the rest of -the users aren't impacted. - -Actors also provide an easy way to do a percentage rollout of a feature in a sticky way. -If a 1% rollout enabled a feature for a specific actor, that actor will continue to have the feature enabled at -10%, 50%, and 100%. - -GitLab currently supports the following models as feature flag actors: - -- `User` -- `Project` -- `Group` - -The actor is a second parameter of the `Feature.enabled?` call. The -same actor type must be used consistently for all invocations of `Feature.enabled?`. - -```ruby -Feature.enabled?(:feature_flag, project) -Feature.enabled?(:feature_flag, group) -Feature.enabled?(:feature_flag, user) -``` - -#### Selectively disable by actor - -By default you cannot selectively disable a feature flag by actor. - -```shell -# This will not work how you would expect. -/chatops run feature set some_feature true -/chatops run feature set --project=gitlab-org/gitlab some_feature false -``` - -However, if you add two feature flags, you can write your conditional statement in such a way that the equivalent selective disable is possible. - -```ruby -Feature.enabled?(:a_feature, project) && Feature.disabled?(:a_feature_override, project) -``` - -```shell -# This will enable a feature flag globally, except for gitlab-org/gitlab -/chatops run feature set a_feature true -/chatops run feature set --project=gitlab-org/gitlab a_feature_override true -``` - -### Enable additional objects as actors - -To use feature gates based on actors, the model needs to respond to -`flipper_id`. For example, to enable for the Foo model: - -```ruby -class Foo < ActiveRecord::Base - include FeatureGate -end -``` - -Only models that `include FeatureGate` or expose `flipper_id` method can be -used as an actor for `Feature.enabled?`. - -### Feature flags for licensed features - -You can't use a feature flag with the same name as a licensed feature name, because -it would cause a naming collision. This was [widely discussed and removed](https://gitlab.com/gitlab-org/gitlab/-/issues/259611) -because it is confusing. - -To check for licensed features, add a dedicated feature flag under a different name -and check it explicitly, for example: - -```ruby -Feature.enabled?(:licensed_feature_feature_flag, project) && - project.feature_available?(:licensed_feature) -``` - -### Feature groups - -Feature groups must be defined statically in `lib/feature.rb` (in the -`.register_feature_groups` method), but their implementation can obviously be -dynamic (querying the DB, for example). - -Once defined in `lib/feature.rb`, you can to activate a -feature for a given feature group via the [`feature_group` parameter of the features API](../../api/features.md#set-or-create-a-feature) - -### Enabling a feature flag locally (in development) - -In the rails console (`rails c`), enter the following command to enable a feature flag: - -```ruby -Feature.enable(:feature_flag_name) -``` - -Similarly, the following command disables a feature flag: - -```ruby -Feature.disable(:feature_flag_name) -``` - -You can also enable a feature flag for a given gate: - -```ruby -Feature.enable(:feature_flag_name, Project.find_by_full_path("root/my-project")) -``` - -### Removing a feature flag locally (in development) - -When manually enabling or disabling a feature flag from the Rails console, its default value gets overwritten. -This can cause confusion when changing the flag's `default_enabled` attribute. - -To reset the feature flag to the default status, you can remove it in the rails console (`rails c`) -as follows: - -```ruby -Feature.remove(:feature_flag_name) -``` - -## Feature flags in tests - -Introducing a feature flag into the codebase creates an additional code path that should be tested. -It is strongly advised to test all code affected by a feature flag, both when **enabled** and **disabled** -to ensure the feature works properly. - -When using the testing environment, all feature flags are enabled by default. - -WARNING: -This does not apply to end-to-end (QA) tests, which [do not disable feature flags by default](#end-to-end-qa-tests). There is a different [process for using feature flags in end-to-end tests](../testing_guide/end_to_end/feature_flags.md). - -To disable a feature flag in a test, use the `stub_feature_flags` -helper. For example, to globally disable the `ci_live_trace` feature -flag in a test: - -```ruby -stub_feature_flags(ci_live_trace: false) - -Feature.enabled?(:ci_live_trace) # => false -``` - -If you wish to set up a test where a feature flag is enabled only -for some actors and not others, you can specify this in options -passed to the helper. For example, to enable the `ci_live_trace` -feature flag for a specific project: - -```ruby -project1, project2 = build_list(:project, 2) - -# Feature will only be enabled for project1 -stub_feature_flags(ci_live_trace: project1) - -Feature.enabled?(:ci_live_trace) # => false -Feature.enabled?(:ci_live_trace, project1) # => true -Feature.enabled?(:ci_live_trace, project2) # => false -``` - -The behavior of FlipperGate is as follows: - -1. You can enable an override for a specified actor to be enabled. -1. You can disable (remove) an override for a specified actor, - falling back to the default state. -1. There's no way to model that you explicitly disabled a specified actor. - -```ruby -Feature.enable(:my_feature) -Feature.disable(:my_feature, project1) -Feature.enabled?(:my_feature) # => true -Feature.enabled?(:my_feature, project1) # => true - -Feature.disable(:my_feature2) -Feature.enable(:my_feature2, project1) -Feature.enabled?(:my_feature2) # => false -Feature.enabled?(:my_feature2, project1) # => true -``` - -### `have_pushed_frontend_feature_flags` - -Use `have_pushed_frontend_feature_flags` to test if [`push_frontend_feature_flag`](#frontend) -has added the feature flag to the HTML. - -For example, - -```ruby -stub_feature_flags(value_stream_analytics_path_navigation: false) - -visit group_analytics_cycle_analytics_path(group) - -expect(page).to have_pushed_frontend_feature_flags(valueStreamAnalyticsPathNavigation: false) -``` - -### `stub_feature_flags` vs `Feature.enable*` - -It is preferred to use `stub_feature_flags` to enable feature flags -in the testing environment. This method provides a simple and well described -interface for simple use cases. - -However, in some cases more complex behavior needs to be tested, -like percentage rollouts of feature flags. This can be done using -`.enable_percentage_of_time` or `.enable_percentage_of_actors`: - -```ruby -# Good: feature needs to be explicitly disabled, as it is enabled by default if not defined -stub_feature_flags(my_feature: false) -stub_feature_flags(my_feature: true) -stub_feature_flags(my_feature: project) -stub_feature_flags(my_feature: [project, project2]) - -# Bad -Feature.enable(:my_feature_2) - -# Good: enable my_feature for 50% of time -Feature.enable_percentage_of_time(:my_feature_3, 50) - -# Good: enable my_feature for 50% of actors/gates/things -Feature.enable_percentage_of_actors(:my_feature_4, 50) -``` - -Each feature flag that has a defined state is persisted -during test execution time: - -```ruby -Feature.persisted_names.include?('my_feature') => true -Feature.persisted_names.include?('my_feature_2') => true -Feature.persisted_names.include?('my_feature_3') => true -Feature.persisted_names.include?('my_feature_4') => true -``` - -### Stubbing actor - -When you want to enable a feature flag for a specific actor only, -you can stub its representation. A gate that is passed -as an argument to `Feature.enabled?` and `Feature.disabled?` must be an object -that includes `FeatureGate`. - -In specs you can use the `stub_feature_flag_gate` method that allows you to -quickly create a custom actor: - -```ruby -gate = stub_feature_flag_gate('CustomActor') - -stub_feature_flags(ci_live_trace: gate) - -Feature.enabled?(:ci_live_trace) # => false -Feature.enabled?(:ci_live_trace, gate) # => true -``` - -You can also disable a feature flag for a specific actor: - -```ruby -gate = stub_feature_flag_gate('CustomActor') - -stub_feature_flags(ci_live_trace: false, thing: gate) -``` - -### Controlling feature flags engine in tests - -Our Flipper engine in the test environment works in a memory mode `Flipper::Adapters::Memory`. -`production` and `development` modes use `Flipper::Adapters::ActiveRecord`. - -You can control whether the `Flipper::Adapters::Memory` or `ActiveRecord` mode is being used. - -#### `stub_feature_flags: true` (default and preferred) - -In this mode Flipper is configured to use `Flipper::Adapters::Memory` and mark all feature -flags to be on-by-default and persisted on a first use. This overwrites the `default_enabled:` -of `Feature.enabled?` and `Feature.disabled?` returning always `true` unless feature flag -is persisted. - -Make sure behavior under feature flag doesn't go untested in some non-specific contexts. - -### `stub_feature_flags: false` - -This disables a memory-stubbed flipper, and uses `Flipper::Adapters::ActiveRecord` -a mode that is used by `production` and `development`. - -You should use this mode only when you really want to tests aspects of Flipper -with how it interacts with `ActiveRecord`. - -### End-to-end (QA) tests - -Toggling feature flags works differently in end-to-end (QA) tests. The end-to-end test framework does not have direct access to -Rails or the database, so it can't use Flipper. Instead, it uses [the public API](../../api/features.md#set-or-create-a-feature). Each end-to-end test can [enable or disable a feature flag during the test](../testing_guide/end_to_end/feature_flags.md). Alternatively, you can enable or disable a feature flag before one or more tests when you [run them from your GitLab repository's `qa` directory](https://gitlab.com/gitlab-org/gitlab/tree/master/qa#running-tests-with-a-feature-flag-enabled-or-disabled), or if you [run the tests via GitLab QA](https://gitlab.com/gitlab-org/gitlab-qa/-/blob/master/docs/what_tests_can_be_run.md#running-tests-with-a-feature-flag-enabled). - -[As noted above, feature flags are not enabled by default in end-to-end tests.](#feature-flags-in-tests) -This means that end-to-end tests will run with feature flags in the default state implemented in the source -code, or with the feature flag in its current state on the GitLab instance under test, unless the -test is written to enable/disable a feature flag explicitly. - -When a feature flag is changed on Staging or on GitLab.com, a Slack message will be posted to the `#qa-staging` or `#qa-production` channels to inform -the pipeline triage DRI so that they can more easily determine if any failures are related to a feature flag change. However, if you are working on a change you can -help to avoid unexpected failures by [confirming that the end-to-end tests pass with a feature flag enabled.](../testing_guide/end_to_end/feature_flags.md#confirming-that-end-to-end-tests-pass-with-a-feature-flag-enabled) +This document was moved to [another location](index.md). +<!-- This redirect file can be deleted after 2021-06-01. --> +<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/#move-or-rename-a-page -->
\ No newline at end of file diff --git a/doc/development/feature_flags/index.md b/doc/development/feature_flags/index.md index 4890cc5da35..5c98dc2e473 100644 --- a/doc/development/feature_flags/index.md +++ b/doc/development/feature_flags/index.md @@ -1,75 +1,636 @@ --- +type: reference, dev stage: none group: Development info: "See the Technical Writers assigned to Development Guidelines: https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments-to-development-guidelines" --- -# Feature flags in development of GitLab +# Developing with feature flags **NOTE**: The documentation below covers feature flags used by GitLab to deploy its own features, which **is not** the same as the [feature flags offered as part of the product](../../operations/feature_flags.md). -## When to use feature flags +This document provides guidelines on how to use feature flags +in the GitLab codebase to conditionally enable features +and test them. -Developers are required to use feature flags for changes that could affect availability of existing GitLab functionality (if it only affects the new feature you're making that is probably acceptable). -Such changes include: +Features that are developed and merged behind a feature flag +should not include a changelog entry. The entry should be added either in the merge +request removing the feature flag or the merge request where the default value of +the feature flag is set to enabled. If the feature contains any database migrations, it +*should* include a changelog entry for the database changes. -1. New features in high traffic areas (e.g. a new merge request widget, new option in issues/epics, new CI functionality). -1. Complex performance improvements that may require additional testing in production (e.g. rewriting complex queries, changes to frequently used API endpoints). -1. Invasive changes to the user interface (e.g. introducing a new navigation bar, removal of a sidebar, UI element change in issues or MR interface). -1. Introducing dependencies on third-party services (e.g. adding support for importing projects). -1. Changes to features that can cause data corruption or cause data loss (e.g. features processing repository data or user uploaded content). +WARNING: +All newly-introduced feature flags should be [disabled by default](https://about.gitlab.com/handbook/product-development-flow/feature-flag-lifecycle/#feature-flags-in-gitlab-development). -Situations where you might consider not using a feature flag: +NOTE: +This document is the subject of continued work as part of an epic to [improve internal usage of Feature Flags](https://gitlab.com/groups/gitlab-org/-/epics/3551). Raise any suggestions as new issues and attach them to the epic. -1. Adding a new API endpoint -1. Introducing new features in low traffic areas (e.g. adding a new export functionality in the admin area/group settings/project settings) -1. Non-invasive frontend changes (e.g. changing the color of a button, or moving a UI element in a low traffic area) +## Feature flags in GitLab development -In all cases, those working on the changes should ask themselves: +The following highlights should be considered when deciding if feature flags +should be leveraged: -> Why do I need to add a feature flag? If I don't add one, what options do I have to control the impact on application reliability, and user experience? +- By default, the feature flags should be **off**. +- Feature flags should remain in the codebase for as short period as possible + to reduce the need for feature flag accounting. +- The person operating with feature flags is responsible for clearly communicating + the status of a feature behind the feature flag with responsible stakeholders. The + issue description should be updated with the feature flag name and whether it is + defaulted on or off as soon it is evident that a feature flag is needed. +- Merge requests that make changes hidden behind a feature flag, or remove an + existing feature flag because a feature is deemed stable must have the + ~"feature flag" label assigned. +- When development of a feature will be spread across multiple merge + requests, you can use the following workflow: -For perspective on why we limit our use of feature flags please see -<i class="fa fa-youtube-play youtube" aria-hidden="true"></i> [Feature flags only when needed](https://www.youtube.com/watch?v=DQaGqyolOd8). + 1. [Create a new feature flag](#create-a-new-feature-flag) + which is **off** by default, in the first merge request which uses the flag. + Flags [should not be added separately](#risk-of-a-broken-master-main-branch). + 1. Submit incremental changes via one or more merge requests, ensuring that any + new code added can only be reached if the feature flag is **on**. + You can keep the feature flag enabled on your local GDK during development. + 1. When the feature is ready to be tested, enable the feature flag for + a specific project and ensure that there are no issues with the implementation. + 1. When the feature is ready to be announced, create a merge request that adds + documentation about the feature, including [documentation for the feature flag itself](../documentation/feature_flags.md), + and a changelog entry. In the same merge request either flip the feature flag to + be **on by default** or remove it entirely in order to enable the new behavior. -In case you are uncertain if a feature flag is necessary, simply ask about this in an early merge request, and those reviewing the changes will likely provide you with an answer. +One might be tempted to think that feature flags will delay the release of a +feature by at least one month (= one release). This is not the case. A feature +flag does not have to stick around for a specific amount of time +(e.g. at least one release), instead they should stick around until the feature +is deemed stable. Stable means it works on GitLab.com without causing any +problems, such as outages. -When using a feature flag for UI elements, make sure to _also_ use a feature -flag for the underlying backend code, if there is any. This ensures there is -absolutely no way to use the feature until it is enabled. +## Risk of a broken master (main) branch -## How to use Feature Flags +Feature flags **must** be used in the MR that introduces them. Not doing so causes a +[broken master](https://about.gitlab.com/handbook/engineering/workflow/#broken-master) scenario due +to the `rspec:feature-flags` job that only runs on the `master` branch. -Feature flags can be used to gradually deploy changes, regardless of whether -they are new features or performance improvements. By using feature flags, -you can determine the impact of GitLab-directed changes, while still being able -to disable those changes without having to revert an entire release. +## Types of feature flags -For an overview about starting with feature flags in GitLab development, -use this [training template](https://gitlab.com/gitlab-com/www-gitlab-com/-/blob/master/.gitlab/issue_templates/feature-flag-training.md). +Choose a feature flag type that matches the expected usage. -Before using feature flags for GitLab development, review the following development guides: +### `development` type -1. [Process for using features flags](process.md): When you should use - feature flags in the development of GitLab, what's the cost of using them, - and how to include them in a release. -1. [Developing with feature flags](development.md): Learn about the types of - feature flags, their definition and validation, how to create them, frontend and - backend details, and other information. -1. [Documenting features deployed behind feature flags](../documentation/feature_flags.md): - How to document features deployed behind feature flags, and how to update the - documentation for features' flags when their states change. -1. [Controlling feature flags](controls.md): Learn the process for deploying - a new feature, enabling it on GitLab.com, communicating the change, - logging, and cleaning up. +`development` feature flags are short-lived feature flags, +used so that unfinished code can be deployed in production. -User guides: +A `development` feature flag should have a rollout issue, +ideally created using the [Feature Flag Roll Out template](https://gitlab.com/gitlab-org/gitlab/-/blob/master/.gitlab/issue_templates/Feature%20Flag%20Roll%20Out.md). -1. [How GitLab administrators can enable and disable features behind flags](../../administration/feature_flags.md): - An explanation for GitLab administrators about how they can - enable or disable GitLab features behind feature flags. -1. [What "features deployed behind flags" means to the GitLab user](../../user/feature_flags.md): - An explanation for GitLab users regarding how certain features - might not be available to them until they are enabled. +This is the default type used when calling `Feature.enabled?`. + +### `ops` type + +`ops` feature flags are long-lived feature flags that control operational aspects +of GitLab product behavior. For example, feature flags that disable features that might +have a performance impact, like special Sidekiq worker behavior. + +`ops` feature flags likely do not have rollout issues, as it is hard to +predict when they are enabled or disabled. + +To use `ops` feature flags, you must append `type: :ops` to `Feature.enabled?` +invocations: + +```ruby +# Check if feature flag is enabled +Feature.enabled?(:my_ops_flag, project, type: :ops) + +# Check if feature flag is disabled +Feature.disabled?(:my_ops_flag, project, type: :ops) + +# Push feature flag to Frontend +push_frontend_feature_flag(:my_ops_flag, project, type: :ops) +``` + +### `experiment` type + +`experiment` feature flags are used for A/B testing on GitLab.com. + +An `experiment` feature flag should conform to the same standards as a `development` feature flag, +although the interface has some differences. An experiment feature flag should have a rollout issue, +ideally created using the [Experiment Tracking template](https://gitlab.com/gitlab-org/gitlab/-/blob/master/.gitlab/issue_templates/experiment_tracking_template.md). More information can be found in the [experiment guide](../experiment_guide/index.md). + +## Feature flag definition and validation + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/229161) in GitLab 13.3. + +During development (`RAILS_ENV=development`) or testing (`RAILS_ENV=test`) all feature flag usage is being strictly validated. + +This process is meant to ensure consistent feature flag usage in the codebase. All feature flags **must**: + +- Be known. Only use feature flags that are explicitly defined. +- Not be defined twice. They have to be defined either in FOSS or EE, but not both. +- Use a valid and consistent `type:` across all invocations. +- Use the same `default_enabled:` across all invocations. +- Have an owner. + +All feature flags known to GitLab are self-documented in YAML files stored in: + +- [`config/feature_flags`](https://gitlab.com/gitlab-org/gitlab/-/tree/master/config/feature_flags) +- [`ee/config/feature_flags`](https://gitlab.com/gitlab-org/gitlab/-/tree/master/ee/config/feature_flags) + +Each feature flag is defined in a separate YAML file consisting of a number of fields: + +| Field | Required | Description | +|---------------------|----------|----------------------------------------------------------------| +| `name` | yes | Name of the feature flag. | +| `type` | yes | Type of feature flag. | +| `default_enabled` | yes | The default state of the feature flag that is strictly validated, with `default_enabled:` passed as an argument. | +| `introduced_by_url` | no | The URL to the Merge Request that introduced the feature flag. | +| `rollout_issue_url` | no | The URL to the Issue covering the feature flag rollout. | +| `group` | no | The [group](https://about.gitlab.com/handbook/product/categories/#devops-stages) that owns the feature flag. | + +NOTE: +All validations are skipped when running in `RAILS_ENV=production`. + +## Create a new feature flag + +The GitLab codebase provides [`bin/feature-flag`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/bin/feature-flag), +a dedicated tool to create new feature flag definitions. +The tool asks various questions about the new feature flag, then creates +a YAML definition in `config/feature_flags` or `ee/config/feature_flags`. + +Only feature flags that have a YAML definition file can be used when running the development or testing environments. + +```shell +$ bin/feature-flag my_feature_flag +>> Specify the group introducing the feature flag, like `group::apm`: +?> group::memory + +>> URL of the MR introducing the feature flag (enter to skip): +?> https://gitlab.com/gitlab-org/gitlab/-/merge_requests/38602 + +>> Open this URL and fill in the rest of the details: +https://gitlab.com/gitlab-org/gitlab/-/issues/new?issue%5Btitle%5D=%5BFeature+flag%5D+Rollout+of+%60test-flag%60&issuable_template=Feature+Flag+Roll+Out + +>> URL of the rollout issue (enter to skip): +?> https://gitlab.com/gitlab-org/gitlab/-/issues/232533 +create config/feature_flags/development/my_feature_flag.yml +--- +name: my_feature_flag +introduced_by_url: https://gitlab.com/gitlab-org/gitlab/-/merge_requests/38602 +rollout_issue_url: https://gitlab.com/gitlab-org/gitlab/-/issues/232533 +group: group::memory +type: development +default_enabled: false +``` + +NOTE: +To create a feature flag that is only used in EE, add the `--ee` flag: `bin/feature-flag --ee` + +## Delete a feature flag + +See [cleaning up feature flags](controls.md#cleaning-up) for more information about +deleting feature flags. + +## Develop with a feature flag + +There are two main ways of using Feature Flags in the GitLab codebase: + +- [Backend code (Rails)](#backend) +- [Frontend code (VueJS)](#frontend) + +### Backend + +The feature flag interface is defined in [`lib/feature.rb`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/feature.rb). +This interface provides a set of methods to check if the feature flag is enabled or disabled: + +```ruby +if Feature.enabled?(:my_feature_flag, project) + # execute code if feature flag is enabled +else + # execute code if feature flag is disabled +end + +if Feature.disabled?(:my_feature_flag, project) + # execute code if feature flag is disabled +end +``` + +In rare cases you may want to make a feature enabled by default. If so, explain the reasoning +in the merge request. Use `default_enabled: true` when checking the feature flag state: + +```ruby +if Feature.enabled?(:feature_flag, project, default_enabled: true) + # execute code if feature flag is enabled +else + # execute code if feature flag is disabled +end + +if Feature.disabled?(:my_feature_flag, project, default_enabled: true) + # execute code if feature flag is disabled +end +``` + +If not specified, `default_enabled` is `false`. + +To force reading the `default_enabled` value from the relative YAML definition file, use +`default_enabled: :yaml`: + +```ruby +if Feature.enabled?(:feature_flag, project, default_enabled: :yaml) + # execute code if feature flag is enabled +end +``` + +```ruby +if Feature.disabled?(:feature_flag, project, default_enabled: :yaml) + # execute code if feature flag is disabled +end +``` + +This allows to use the same feature flag check across various parts of the codebase and +maintain the status of `default_enabled` in the YAML definition file which is the SSOT. + +If `default_enabled: :yaml` is used, a YAML definition is expected or an error is raised +in development or test environment, while returning `false` on production. + +If not specified, the default feature flag type for `Feature.enabled?` and `Feature.disabled?` +is `type: development`. For all other feature flag types, you must specify the `type:`: + +```ruby +if Feature.enabled?(:feature_flag, project, type: :ops) + # execute code if ops feature flag is enabled +else + # execute code if ops feature flag is disabled +end + +if Feature.disabled?(:my_feature_flag, project, type: :ops) + # execute code if feature flag is disabled +end +``` + +WARNING: +Don't use feature flags at application load time. For example, using the `Feature` class in +`config/initializers/*` or at the class level could cause an unexpected error. This error occurs +because a database that a feature flag adapter might depend on doesn't exist at load time +(especially for fresh installations). Checking for the database's existence at the caller isn't +recommended, as some adapters don't require a database at all (for example, the HTTP adapter). The +feature flag setup check must be abstracted in the `Feature` namespace. This approach also requires +application reload when the feature flag changes. You must therefore ask SREs to reload the +Web/API/Sidekiq fleet on production, which takes time to fully rollout/rollback the changes. For +these reasons, use environment variables (for example, `ENV['YOUR_FEATURE_NAME']`) or `gitlab.yml` +instead. + +Here's an example of a pattern that you should avoid: + +```ruby +class MyClass + if Feature.enabled?(:...) + new_process + else + legacy_process + end +end +``` + +### Frontend + +Use the `push_frontend_feature_flag` method for frontend code, which is +available to all controllers that inherit from `ApplicationController`. You can use +this method to expose the state of a feature flag, for example: + +```ruby +before_action do + # Prefer to scope it per project or user e.g. + push_frontend_feature_flag(:vim_bindings, project) +end + +def index + # ... +end + +def edit + # ... +end +``` + +You can then check the state of the feature flag in JavaScript as follows: + +```javascript +if ( gon.features.vimBindings ) { + // ... +} +``` + +The name of the feature flag in JavaScript is always camelCase, +so checking for `gon.features.vim_bindings` would not work. + +See the [Vue guide](../fe_guide/vue.md#accessing-feature-flags) for details about +how to access feature flags in a Vue component. + +In rare cases you may want to make a feature enabled by default. If so, explain the reasoning +in the merge request. Use `default_enabled: true` when checking the feature flag state: + +```ruby +before_action do + # Prefer to scope it per project or user e.g. + push_frontend_feature_flag(:vim_bindings, project, default_enabled: true) +end +``` + +If not specified, the default feature flag type for `push_frontend_feature_flag` +is `type: development`. For all other feature flag types, you must specify the `type:`: + +```ruby +before_action do + push_frontend_feature_flag(:vim_bindings, project, type: :ops) +end +``` + +### Feature actors + +**It is strongly advised to use actors with feature flags.** Actors provide a simple +way to enable a feature flag only for a given project, group or user. This makes debugging +easier, as you can filter logs and errors for example, based on actors. This also makes it possible +to enable the feature on the `gitlab-org` or `gitlab-com` groups first, while the rest of +the users aren't impacted. + +Actors also provide an easy way to do a percentage rollout of a feature in a sticky way. +If a 1% rollout enabled a feature for a specific actor, that actor will continue to have the feature enabled at +10%, 50%, and 100%. + +GitLab currently supports the following models as feature flag actors: + +- `User` +- `Project` +- `Group` + +The actor is a second parameter of the `Feature.enabled?` call. The +same actor type must be used consistently for all invocations of `Feature.enabled?`. + +```ruby +Feature.enabled?(:feature_flag, project) +Feature.enabled?(:feature_flag, group) +Feature.enabled?(:feature_flag, user) +``` + +#### Selectively disable by actor + +By default you cannot selectively disable a feature flag by actor. + +```shell +# This will not work how you would expect. +/chatops run feature set some_feature true +/chatops run feature set --project=gitlab-org/gitlab some_feature false +``` + +However, if you add two feature flags, you can write your conditional statement in such a way that the equivalent selective disable is possible. + +```ruby +Feature.enabled?(:a_feature, project) && Feature.disabled?(:a_feature_override, project) +``` + +```shell +# This will enable a feature flag globally, except for gitlab-org/gitlab +/chatops run feature set a_feature true +/chatops run feature set --project=gitlab-org/gitlab a_feature_override true +``` + +### Enable additional objects as actors + +To use feature gates based on actors, the model needs to respond to +`flipper_id`. For example, to enable for the Foo model: + +```ruby +class Foo < ActiveRecord::Base + include FeatureGate +end +``` + +Only models that `include FeatureGate` or expose `flipper_id` method can be +used as an actor for `Feature.enabled?`. + +### Feature flags for licensed features + +You can't use a feature flag with the same name as a licensed feature name, because +it would cause a naming collision. This was [widely discussed and removed](https://gitlab.com/gitlab-org/gitlab/-/issues/259611) +because it is confusing. + +To check for licensed features, add a dedicated feature flag under a different name +and check it explicitly, for example: + +```ruby +Feature.enabled?(:licensed_feature_feature_flag, project) && + project.feature_available?(:licensed_feature) +``` + +### Feature groups + +Feature groups must be defined statically in `lib/feature.rb` (in the +`.register_feature_groups` method), but their implementation can obviously be +dynamic (querying the DB, for example). + +Once defined in `lib/feature.rb`, you can to activate a +feature for a given feature group via the [`feature_group` parameter of the features API](../../api/features.md#set-or-create-a-feature) + +### Enabling a feature flag locally (in development) + +In the rails console (`rails c`), enter the following command to enable a feature flag: + +```ruby +Feature.enable(:feature_flag_name) +``` + +Similarly, the following command disables a feature flag: + +```ruby +Feature.disable(:feature_flag_name) +``` + +You can also enable a feature flag for a given gate: + +```ruby +Feature.enable(:feature_flag_name, Project.find_by_full_path("root/my-project")) +``` + +### Removing a feature flag locally (in development) + +When manually enabling or disabling a feature flag from the Rails console, its default value gets overwritten. +This can cause confusion when changing the flag's `default_enabled` attribute. + +To reset the feature flag to the default status, you can remove it in the rails console (`rails c`) +as follows: + +```ruby +Feature.remove(:feature_flag_name) +``` + +## Feature flags in tests + +Introducing a feature flag into the codebase creates an additional code path that should be tested. +It is strongly advised to test all code affected by a feature flag, both when **enabled** and **disabled** +to ensure the feature works properly. + +When using the testing environment, all feature flags are enabled by default. + +WARNING: +This does not apply to end-to-end (QA) tests, which [do not disable feature flags by default](#end-to-end-qa-tests). There is a different [process for using feature flags in end-to-end tests](../testing_guide/end_to_end/feature_flags.md). + +To disable a feature flag in a test, use the `stub_feature_flags` +helper. For example, to globally disable the `ci_live_trace` feature +flag in a test: + +```ruby +stub_feature_flags(ci_live_trace: false) + +Feature.enabled?(:ci_live_trace) # => false +``` + +If you wish to set up a test where a feature flag is enabled only +for some actors and not others, you can specify this in options +passed to the helper. For example, to enable the `ci_live_trace` +feature flag for a specific project: + +```ruby +project1, project2 = build_list(:project, 2) + +# Feature will only be enabled for project1 +stub_feature_flags(ci_live_trace: project1) + +Feature.enabled?(:ci_live_trace) # => false +Feature.enabled?(:ci_live_trace, project1) # => true +Feature.enabled?(:ci_live_trace, project2) # => false +``` + +The behavior of FlipperGate is as follows: + +1. You can enable an override for a specified actor to be enabled. +1. You can disable (remove) an override for a specified actor, + falling back to the default state. +1. There's no way to model that you explicitly disabled a specified actor. + +```ruby +Feature.enable(:my_feature) +Feature.disable(:my_feature, project1) +Feature.enabled?(:my_feature) # => true +Feature.enabled?(:my_feature, project1) # => true + +Feature.disable(:my_feature2) +Feature.enable(:my_feature2, project1) +Feature.enabled?(:my_feature2) # => false +Feature.enabled?(:my_feature2, project1) # => true +``` + +### `have_pushed_frontend_feature_flags` + +Use `have_pushed_frontend_feature_flags` to test if [`push_frontend_feature_flag`](#frontend) +has added the feature flag to the HTML. + +For example, + +```ruby +stub_feature_flags(value_stream_analytics_path_navigation: false) + +visit group_analytics_cycle_analytics_path(group) + +expect(page).to have_pushed_frontend_feature_flags(valueStreamAnalyticsPathNavigation: false) +``` + +### `stub_feature_flags` vs `Feature.enable*` + +It is preferred to use `stub_feature_flags` to enable feature flags +in the testing environment. This method provides a simple and well described +interface for simple use cases. + +However, in some cases more complex behavior needs to be tested, +like percentage rollouts of feature flags. This can be done using +`.enable_percentage_of_time` or `.enable_percentage_of_actors`: + +```ruby +# Good: feature needs to be explicitly disabled, as it is enabled by default if not defined +stub_feature_flags(my_feature: false) +stub_feature_flags(my_feature: true) +stub_feature_flags(my_feature: project) +stub_feature_flags(my_feature: [project, project2]) + +# Bad +Feature.enable(:my_feature_2) + +# Good: enable my_feature for 50% of time +Feature.enable_percentage_of_time(:my_feature_3, 50) + +# Good: enable my_feature for 50% of actors/gates/things +Feature.enable_percentage_of_actors(:my_feature_4, 50) +``` + +Each feature flag that has a defined state is persisted +during test execution time: + +```ruby +Feature.persisted_names.include?('my_feature') => true +Feature.persisted_names.include?('my_feature_2') => true +Feature.persisted_names.include?('my_feature_3') => true +Feature.persisted_names.include?('my_feature_4') => true +``` + +### Stubbing actor + +When you want to enable a feature flag for a specific actor only, +you can stub its representation. A gate that is passed +as an argument to `Feature.enabled?` and `Feature.disabled?` must be an object +that includes `FeatureGate`. + +In specs you can use the `stub_feature_flag_gate` method that allows you to +quickly create a custom actor: + +```ruby +gate = stub_feature_flag_gate('CustomActor') + +stub_feature_flags(ci_live_trace: gate) + +Feature.enabled?(:ci_live_trace) # => false +Feature.enabled?(:ci_live_trace, gate) # => true +``` + +You can also disable a feature flag for a specific actor: + +```ruby +gate = stub_feature_flag_gate('CustomActor') + +stub_feature_flags(ci_live_trace: false, thing: gate) +``` + +### Controlling feature flags engine in tests + +Our Flipper engine in the test environment works in a memory mode `Flipper::Adapters::Memory`. +`production` and `development` modes use `Flipper::Adapters::ActiveRecord`. + +You can control whether the `Flipper::Adapters::Memory` or `ActiveRecord` mode is being used. + +#### `stub_feature_flags: true` (default and preferred) + +In this mode Flipper is configured to use `Flipper::Adapters::Memory` and mark all feature +flags to be on-by-default and persisted on a first use. This overwrites the `default_enabled:` +of `Feature.enabled?` and `Feature.disabled?` returning always `true` unless feature flag +is persisted. + +Make sure behavior under feature flag doesn't go untested in some non-specific contexts. + +### `stub_feature_flags: false` + +This disables a memory-stubbed flipper, and uses `Flipper::Adapters::ActiveRecord` +a mode that is used by `production` and `development`. + +You should use this mode only when you really want to tests aspects of Flipper +with how it interacts with `ActiveRecord`. + +### End-to-end (QA) tests + +Toggling feature flags works differently in end-to-end (QA) tests. The end-to-end test framework does not have direct access to +Rails or the database, so it can't use Flipper. Instead, it uses [the public API](../../api/features.md#set-or-create-a-feature). Each end-to-end test can [enable or disable a feature flag during the test](../testing_guide/end_to_end/feature_flags.md). Alternatively, you can enable or disable a feature flag before one or more tests when you [run them from your GitLab repository's `qa` directory](https://gitlab.com/gitlab-org/gitlab/tree/master/qa#running-tests-with-a-feature-flag-enabled-or-disabled), or if you [run the tests via GitLab QA](https://gitlab.com/gitlab-org/gitlab-qa/-/blob/master/docs/what_tests_can_be_run.md#running-tests-with-a-feature-flag-enabled). + +[As noted above, feature flags are not enabled by default in end-to-end tests.](#feature-flags-in-tests) +This means that end-to-end tests will run with feature flags in the default state implemented in the source +code, or with the feature flag in its current state on the GitLab instance under test, unless the +test is written to enable/disable a feature flag explicitly. + +When a feature flag is changed on Staging or on GitLab.com, a Slack message will be posted to the `#qa-staging` or `#qa-production` channels to inform +the pipeline triage DRI so that they can more easily determine if any failures are related to a feature flag change. However, if you are working on a change you can +help to avoid unexpected failures by [confirming that the end-to-end tests pass with a feature flag enabled.](../testing_guide/end_to_end/feature_flags.md#confirming-that-end-to-end-tests-pass-with-a-feature-flag-enabled) diff --git a/doc/development/feature_flags/process.md b/doc/development/feature_flags/process.md index 65a6d01e0a2..247dafe9f0b 100644 --- a/doc/development/feature_flags/process.md +++ b/doc/development/feature_flags/process.md @@ -1,177 +1,8 @@ --- -type: reference, dev -stage: none -group: Development -info: "See the Technical Writers assigned to Development Guidelines: https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments-to-development-guidelines" +redirect_to: 'https://about.gitlab.com/handbook/product-development-flow/feature-flag-lifecycle/' --- -# Feature flags process +This document was moved to [another location](https://about.gitlab.com/handbook/product-development-flow/feature-flag-lifecycle/). -## Feature flags for user applications - -This document only covers feature flags used in the development of GitLab -itself. Feature flags in deployed user applications can be found at -[Feature Flags feature documentation](../../operations/feature_flags.md). - -## Feature flags in GitLab development - -The following highlights should be considered when deciding if feature flags -should be leveraged: - -- By default, the feature flags should be **off**. -- Feature flags should remain in the codebase for as short period as possible - to reduce the need for feature flag accounting. -- The person operating with feature flags is responsible for clearly communicating - the status of a feature behind the feature flag with responsible stakeholders. The - issue description should be updated with the feature flag name and whether it is - defaulted on or off as soon it is evident that a feature flag is needed. -- Merge requests that make changes hidden behind a feature flag, or remove an - existing feature flag because a feature is deemed stable must have the - ~"feature flag" label assigned. -- When development of a feature will be spread across multiple merge - requests, you can use the following workflow: - - 1. [Create a new feature flag](development.md#create-a-new-feature-flag) - which is **off** by default, in the first merge request which uses the flag. - Flags [should not be added separately](development.md#risk-of-a-broken-master-main-branch). - 1. Submit incremental changes via one or more merge requests, ensuring that any - new code added can only be reached if the feature flag is **on**. - You can keep the feature flag enabled on your local GDK during development. - 1. When the feature is ready to be tested, enable the feature flag for - a specific project and ensure that there are no issues with the implementation. - 1. When the feature is ready to be announced, create a merge request that adds - documentation about the feature, including [documentation for the feature flag itself](../documentation/feature_flags.md), - and a changelog entry. In the same merge request either flip the feature flag to - be **on by default** or remove it entirely in order to enable the new behavior. - -One might be tempted to think that feature flags will delay the release of a -feature by at least one month (= one release). This is not the case. A feature -flag does not have to stick around for a specific amount of time -(e.g. at least one release), instead they should stick around until the feature -is deemed stable. Stable means it works on GitLab.com without causing any -problems, such as outages. - -Please also read the [development guide for feature flags](development.md). - -### Including a feature behind feature flag in the final release - -In order to build a final release and present the feature for self-managed -users, the feature flag should be at least defaulted to **on**. If the feature -is deemed stable and there is confidence that removing the feature flag is safe, -consider removing the feature flag altogether. It's _strongly_ recommended that -the feature flag is [enabled **globally** on **production**](controls.md#enabling-a-feature-for-gitlabcom) for **at least one day** -before making this decision. Unexpected bugs are sometimes discovered during this period. - -The process for enabling features that are disabled by default can take 5-6 days -from when the merge request is first reviewed to when the change is deployed to -GitLab.com. However, it is recommended to allow 10-14 days for this activity to -account for unforeseen problems. - -Feature flags must be [documented according to their state (enabled/disabled)](../documentation/feature_flags.md), -and when the state changes, docs **must** be updated accordingly. - -NOTE: -Take into consideration that such action can make the feature available on -GitLab.com shortly after the change to the feature flag is merged. - -Changing the default state or removing the feature flag has to be done before -the 22nd of the month, _at least_ 3-4 working days before, in order for the change -to be included in the final self-managed release. - -In addition to this, the feature behind feature flag should: - -- Run in all GitLab.com environments for a sufficient period of time. This time - period depends on the feature behind the feature flag, but as a general rule of - thumb 2-4 working days should be sufficient to gather enough feedback. -- The feature should be exposed to all users within the GitLab.com plan during - the above mentioned period of time. Exposing the feature to a smaller percentage - or only a group of users might not expose a sufficient amount of information to aid in - making a decision on feature stability. - -While rare, release managers may decide to reject picking or revert a change in -a stable branch, even when feature flags are used. This might be necessary if -the changes are deemed problematic, too invasive, or there simply isn't enough -time to properly measure how the changes behave on GitLab.com. - -### The cost of feature flags - -When reading the above, one might be tempted to think this procedure is going to -add a lot of work. Fortunately, this is not the case, and we'll show why. For -this example we'll specify the cost of the work to do as a number, ranging from -0 to infinity. The greater the number, the more expensive the work is. The cost -does _not_ translate to time, it's just a way of measuring complexity of one -change relative to another. - -Let's say we are building a new feature, and we have determined that the cost of -this is 10. We have also determined that the cost of adding a feature flag check -in a variety of places is 1. If we do not use feature flags, and our feature -works as intended, our total cost is 10. This however is the best case scenario. -Optimizing for the best case scenario is guaranteed to lead to trouble, whereas -optimizing for the worst case scenario is almost always better. - -To illustrate this, let's say our feature causes an outage, and there's no -immediate way to resolve it. This means we'd have to take the following steps to -resolve the outage: - -1. Revert the release. -1. Perform any cleanups that might be necessary, depending on the changes that - were made. -1. Revert the commit, ensuring the "master" branch remains stable. This is - especially necessary if solving the problem can take days or even weeks. -1. Pick the revert commit into the appropriate stable branches, ensuring we - don't block any future releases until the problem is resolved. - -As history has shown, these steps are time consuming, complex, often involve -many developers, and worst of all: our users will have a bad experience using -GitLab.com until the problem is resolved. - -Now let's say that all of this has an associated cost of 10. This means that in -the worst case scenario, which we should optimize for, our total cost is now 20. - -If we had used a feature flag, things would have been very different. We don't -need to revert a release, and because feature flags are disabled by default we -don't need to revert and pick any Git commits. In fact, all we have to do is -disable the feature, and in the worst case, perform cleanup. Let's say that -the cost of this is 2. In this case, our best case cost is 11: 10 to build the -feature, and 1 to add the feature flag. The worst case cost is now 13: - -- 10 to build the feature. -- 1 to add the feature flag. -- 2 to disable and clean up. - -Here we can see that in the best case scenario the work necessary is only a tiny -bit more compared to not using a feature flag. Meanwhile, the process of -reverting our changes has been made significantly and reliably cheaper. - -In other words, feature flags do not slow down the development process. Instead, -they speed up the process as managing incidents now becomes _much_ easier. Once -continuous deployments are easier to perform, the time to iterate on a feature -is reduced even further, as you no longer need to wait weeks before your changes -are available on GitLab.com. - -### The benefits of feature flags - -It may seem like feature flags are configuration, which goes against our [convention-over-configuration](https://about.gitlab.com/handbook/product/product-principles/#convention-over-configuration) -principle. However, configuration is by definition something that is user-manageable. -Feature flags are not intended to be user-editable. Instead, they are intended as a tool for Engineers -and Site Reliability Engineers to use to de-risk their changes. Feature flags are the shim that gets us -to Continuous Delivery with our monorepo and without having to deploy the entire codebase on every change. -Feature flags are created to ensure that we can safely rollout our work on our terms. -If we use Feature Flags as a configuration, we are doing it wrong and are indeed in violation of our -principles. If something needs to be configured, we should intentionally make it configuration from the -first moment. - -Some of the benefits of using development-type feature flags are: - -1. It enables Continuous Delivery for GitLab.com. -1. It significantly reduces Mean-Time-To-Recovery. -1. It helps engineers to monitor and reduce the impact of their changes gradually, at any scale, - allowing us to be more metrics-driven and execute good DevOps practices, [shifting some responsibility "left"](https://devops.com/why-its-time-for-site-reliability-engineering-to-shift-left/). -1. Controlled feature rollout timing: without feature flags, we would need to wait until a specific - deployment was complete (which at GitLab could be at any time). -1. Increased psychological safety: when a feature flag is used, an engineer has the confidence that if anything goes wrong they can quickly disable the code and minimize the impact of a change that might be risky. -1. Improved throughput: when a change is less risky because a flag exists, theoretical tests about - scalability can potentially become unnecessary or less important. This allows an engineer to - potentially test a feature on a small project, monitor the impact, and proceed. The alternative might - be to build complex benchmarks locally, or on staging, or on another GitLab deployment, which has a - large impact on the time it can take to build and release a feature. +<!-- This redirect file can be deleted after 2021-06-01. --> +<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/#move-or-rename-a-page -->
\ No newline at end of file |