1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
|
---
type: reference, dev
stage: none
group: Development
info: "See the Technical Writers assigned to Development Guidelines: https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments-to-development-guidelines"
---
# Use ChatOps to enable and disable feature flags
NOTE:
This document explains how to contribute to the development of the GitLab product.
If you want to use feature flags to show and hide functionality in your own applications,
view [this feature flags information](../../operations/feature_flags.md) instead.
To turn on/off features behind feature flags in any of the
GitLab-provided environments, like staging and production, you need to
have access to the [ChatOps](../chatops_on_gitlabcom.md) bot. The ChatOps bot
is currently running on the ops instance, which is different from
[GitLab.com](https://gitlab.com) or [`dev.gitlab.org`](https://dev.gitlab.org).
Follow the ChatOps document to [request access](../chatops_on_gitlabcom.md#requesting-access).
After you are added to the project test if your access propagated,
run:
```shell
/chatops run feature --help
```
## Rolling out changes
When the changes are deployed to the environments it is time to start
rolling out the feature to our users. The exact procedure of rolling out a
change is unspecified, as this can vary from change to change. However, in
general we recommend rolling out changes incrementally, instead of enabling them
for everybody right away. We also recommend you to _not_ enable a feature
_before_ the code is being deployed.
This allows you to separate rolling out a feature from a deploy, making it
easier to measure the impact of both separately.
The GitLab feature library (using
[Flipper](https://github.com/jnunemaker/flipper), and covered in the
[Feature Flags process](https://about.gitlab.com/handbook/product-development-flow/feature-flag-lifecycle/) guide) supports rolling out changes to a percentage of
time to users. This in turn can be controlled using [GitLab ChatOps](../../ci/chatops/index.md).
For an up to date list of feature flag commands please see
[the source code](https://gitlab.com/gitlab-com/chatops/blob/master/lib/chatops/commands/feature.rb).
Note that all the examples in that file must be preceded by
`/chatops run`.
If you get an error "Whoops! This action is not allowed. This incident
will be reported." that means your Slack account is not allowed to
change feature flags or you do not have access.
### Enabling a feature for pre-production testing
As a first step in a feature rollout, you should enable the feature on
[`staging.gitlab.com`](https://staging.gitlab.com)
and [`dev.gitlab.org`](https://dev.gitlab.org).
These two environments have different scopes.
`dev.gitlab.org` is a production CE environment that has internal GitLab Inc.
traffic and is used for some development and other related work.
`staging.gitlab.com` has a smaller subset of GitLab.com database and repositories
and does not have regular traffic. Staging is an EE instance and can give you
a (very) rough estimate of how your feature will look and behave on GitLab.com.
Both of these instances are connected to Sentry so make sure you check the projects
there for any exceptions while testing your feature after enabling the feature flag.
For these pre-production environments, the commands should be run in a
Slack channel for the stage the feature is relevant to. For example, use the
`#s_monitor` channel for features developed by the Monitor stage, Health
group.
To enable a feature for 25% of the time, run the following in Slack:
```shell
/chatops run feature set new_navigation_bar 25 --random --dev
/chatops run feature set new_navigation_bar 25 --random --staging
```
### Enabling a feature for GitLab.com
When a feature has successfully been
[enabled on a pre-production](#enabling-a-feature-for-pre-production-testing)
environment and verified as safe and working, you can roll out the
change to GitLab.com (production).
#### Communicate the change
Some feature flag changes on GitLab.com should be communicated with
parts of the company. The developer responsible needs to determine
whether this is necessary and the appropriate level of communication.
This depends on the feature and what sort of impact it might have.
Guidelines:
- Consider notifying `#support_gitlab-com` beforehand. So in case if the feature has any side effects on user experience, they can mitigate and disable the feature flag to reduce some impact.
- If the feature meets the requirements for creating a [Change Management](https://about.gitlab.com/handbook/engineering/infrastructure/change-management/#feature-flags-and-the-change-management-process) issue, create a Change Management issue per [criticality guidelines](https://about.gitlab.com/handbook/engineering/infrastructure/change-management/#change-request-workflows).
- For simple, low-risk, easily reverted features, proceed and [enable the feature in `#production`](#process).
- For support requests to toggle feature flags for specific groups or projects, please follow the process outlined in the [support workflows](https://about.gitlab.com/handbook/support/workflows/saas_feature_flags.html).
#### Process
When enabling a feature flag rollout, the system will automatically block the
ChatOps command from succeeding if there are active `"severity::1"` or `~"severity::2"`
incidents or in-progress change issues, for example:
```shell
/chatops run feature set gitaly_lfs_pointers_pipeline true
- Production checks fail!
- active incidents
2021-06-29 Canary deployment failing QA tests
```
Before enabling a feature flag, verify that you are not violating any [Production Change Lock periods](https://about.gitlab.com/handbook/engineering/infrastructure/change-management/#production-change-lock-pcl) and are in compliance with the [Feature Flags and the Change Management Process](https://about.gitlab.com/handbook/engineering/infrastructure/change-management/#feature-flags-and-the-change-management-process).
The following `/chatops` commands should be performed in the Slack
`#production` channel.
When you begin to enable the feature, please link to the relevant
Feature Flag Rollout Issue within a Slack thread of the first `/chatops`
command you make so people can understand the change if they need to.
To enable a feature for 25% of the time, run the following in Slack:
```shell
/chatops run feature set new_navigation_bar 25 --random
```
This sets a feature flag to `true` based on the following formula:
```ruby
feature_flag_state = rand < (25 / 100.0)
```
This will enable the feature for GitLab.com, with `new_navigation_bar` being the
name of the feature.
This command does *not* enable the feature for 25% of the total users.
Instead, when the feature is checked with `enabled?`, it will return `true` 25% of the time.
To enable a feature for 25% of actors such as users, projects, or groups,
run the following in Slack:
```shell
/chatops run feature set some_feature 25 --actors
```
This sets a feature flag to `true` based on the following formula:
```ruby
feature_flag_state = Zlib.crc32("some_feature<Actor>:#{actor.id}") % (100 * 1_000) < 25 * 1_000
# where <Actor>: is a `User`, `Group`, `Project` and actor is an instance
```
During development, based on the nature of the feature, an actor choice
should be made.
For user focused features:
```ruby
Feature.enabled?(:feature_cool_avatars, current_user)
```
For group or namespace level features:
```ruby
Feature.enabled?(:feature_cooler_groups, group)
```
For project level features:
```ruby
Feature.enabled?(:feature_ice_cold_projects, project)
```
If you are not certain what percentages to use, use the following steps:
1. 25%
1. 50%
1. 75%
1. 100%
Between every step you'll want to wait a little while and monitor the
appropriate graphs on <https://dashboards.gitlab.net>. The exact time to wait
may differ. For some features a few minutes is enough, while for others you may
want to wait several hours or even days. This is entirely up to you, just make
sure it is clearly communicated to your team, and the Production team if you
anticipate any potential problems.
Feature gates can also be actor based, for example a feature could first be
enabled for only the `gitlab` project. The project is passed by supplying a
`--project` flag:
```shell
/chatops run feature set --project=gitlab-org/gitlab some_feature true
```
For groups the `--group` flag is available:
```shell
/chatops run feature set --group=gitlab-org some_feature true
```
Note that `--group` does not work with user namespaces. To enable a feature flag for a
generic namespace (including groups) use `--namespace`:
```shell
/chatops run feature set --namespace=gitlab-org some_feature true
/chatops run feature set --namespace=myusername some_feature true
```
Note that actor-based gates are applied before percentages. For example, considering the
`group/project` as `gitlab-org/gitlab` and a given example feature as `some_feature`, if
you run these 2 commands:
```shell
/chatops run feature set --project=gitlab-org/gitlab some_feature true
/chatops run feature set some_feature 25 --actors
```
Then `some_feature` will be enabled for both 25% of actors and always when interacting with
`gitlab-org/gitlab`. This is a good idea if the feature flag development makes use of group
actors.
```ruby
Feature.enabled?(:some_feature, group)
```
Multiple actors can be passed together in a comma-separated form:
```shell
/chatops run feature set --project=gitlab-org/gitlab,example-org/example-project some_feature true
/chatops run feature set --group=gitlab-org,example-org some_feature true
/chatops run feature set --namespace=gitlab-org,example-org some_feature true
```
Lastly, to verify that the feature is deemed stable in as many cases as possible,
you should fully roll out the feature by enabling the flag **globally** by running:
```shell
/chatops run feature set some_feature true
```
This changes the feature flag state to be **enabled** always, which overrides the
existing gates (for example, `--group=gitlab-org`) in the above processes.
Note, that if an actor based feature gate is present, switching the
`default_enabled` attribute of the YAML definition from `false` to `true`
will not have any effect. The feature gate must be deleted first.
For example, a feature flag is set via ChatOps:
```shell
/chatops run feature set --project=gitlab-org/gitlab some_feature true
```
When the `default_enabled` attribute in the YAML definition is switched to
`true`, the feature gate must be deleted to have the desired effect:
```shell
/chatops run feature delete some_feature
```
##### Percentage of actors vs percentage of time rollouts
If you want to make sure a feature is always on or off for users, use a **Percentage of actors**
rollout. Avoid using percentage of _time_ rollouts in this case.
A percentage of _time_ rollout can introduce inconsistent behavior when `Feature.enabled?`
is used multiple times in the code because the feature flag value is randomized each time
`Feature.enabled?` is called on your code path.
##### Disabling feature flags
To disable a feature flag that has been globally enabled you can run:
```shell
/chatops run feature set some_feature false
```
To disable a feature flag that has been enabled for a specific project you can run:
```shell
/chatops run feature set --project=gitlab-org/gitlab some_feature false
```
You cannot selectively disable feature flags for a specific project/group/user without applying a [specific method of implementing](index.md#selectively-disable-by-actor) the feature flags.
If a feature flag is disabled via ChatOps, that will take precedence over the `default_enabled` value in the YAML. In other words, you could have a feature enabled for on-premise installations but not for GitLab.com.
### Feature flag change logging
#### ChatOps level
Any feature flag change that affects GitLab.com (production) via [ChatOps](https://gitlab.com/gitlab-com/chatops)
is automatically logged in an issue.
The issue is created in the
[gl-infra/feature-flag-log](https://gitlab.com/gitlab-com/gl-infra/feature-flag-log/-/issues?scope=all&state=closed)
project, and it will at minimum log the Slack handle of person enabling
a feature flag, the time, and the name of the flag being changed.
The issue is then also posted to the GitLab internal
[Grafana dashboard](https://dashboards.gitlab.net/) as an annotation
marker to make the change even more visible.
Changes to the issue format can be submitted in the
[ChatOps project](https://gitlab.com/gitlab-com/chatops).
#### Instance level
Any feature flag change that affects any GitLab instance is automatically logged in
[features_json.log](../../administration/logs/index.md#features_jsonlog).
You can search the change history in [Kibana](https://about.gitlab.com/handbook/support/workflows/kibana.html).
You can also access the feature flag change history for GitLab.com [in Kibana](https://log.gprd.gitlab.net/goto/d060337c017723084c6d97e09e591fc6).
## Cleaning up
A feature flag should be removed as soon as it is no longer needed. Each additional
feature flag in the codebase increases the complexity of the application
and reduces confidence in our testing suite covering all possible combinations.
Additionally, a feature flag overwritten in some of the environments can result
in undefined and untested system behavior.
`development` type feature flags should have a short life-cycle because their purpose
is for rolling out a persistent change. `development` feature flags that are older
than 2 milestones are reported to engineering managers. The
[report tool](https://gitlab.com/gitlab-org/gitlab-feature-flag-alert) runs on a
monthly basis. For example, see [the report for December 2021](https://gitlab.com/gitlab-org/quality/triage-reports/-/issues/5480).
If a `development` feature flag is still present in the codebase after 6 months we should
take one of the following actions:
- Enable the feature flag by default and remove it.
- Convert it to an instance, group, or project setting.
- Revert the changes if it's still disabled and not needed anymore.
To remove a feature flag, open **one merge request** to make the changes. In the MR:
1. Add the ~"feature flag" label so release managers are aware of the removal.
1. If the merge request has to be picked into a stable branch, add the
appropriate `~"Pick into X.Y"` label, for example `~"Pick into 13.0"`.
See [the feature flag process](https://about.gitlab.com/handbook/product-development-flow/feature-flag-lifecycle/#including-a-feature-behind-feature-flag-in-the-final-release)
for further details.
1. Remove all references to the feature flag from the codebase, including tests.
1. Remove the YAML definition for the feature from the repository.
Once the above MR has been merged, you should:
1. [Clean up the feature flag from all environments](#cleanup-chatops) with `/chatops run feature delete some_feature`.
1. Close the rollout issue for the feature flag after the feature flag is removed from the codebase.
### Cleanup ChatOps
When a feature gate has been removed from the codebase, the feature
record still exists in the database that the flag was deployed too.
The record can be deleted once the MR is deployed to each environment:
```shell
/chatops run feature delete some_feature --dev
/chatops run feature delete some_feature --staging
```
Then, you can delete it from production after the MR is deployed to prod:
```shell
/chatops run feature delete some_feature
```
|