diff options
| author | John Cai <jcai@gitlab.com> | 2022-11-04 15:53:41 +0300 |
|---|---|---|
| committer | John Cai <jcai@gitlab.com> | 2022-11-04 15:53:41 +0300 |
| commit | 061bcd50aa4e5bae26265f8f039e6786b059053d (patch) | |
| tree | ff85981d4f564f8f373a46b93896c51fb30e5bb0 | |
| parent | f75740430e51520d3edcd22065285cec050d2b74 (diff) | |
| parent | b1249ff46b228c350a7641ecd9eba4d35617e401 (diff) | |
Merge branch 'jc-update-process-ff' into 'master'
PROCESS: Instruct gradual rollout for feature flags
See merge request https://gitlab.com/gitlab-org/gitaly/-/merge_requests/4997
Merged-by: John Cai <jcai@gitlab.com>
Approved-by: Patrick Steinhardt <psteinhardt@gitlab.com>
| -rw-r--r-- | .gitlab/issue_templates/Feature Flag Roll Out.md | 2 | ||||
| -rw-r--r-- | doc/PROCESS.md | 18 |
2 files changed, 14 insertions, 6 deletions
diff --git a/.gitlab/issue_templates/Feature Flag Roll Out.md b/.gitlab/issue_templates/Feature Flag Roll Out.md index 6eab09d89..163d85276 100644 --- a/.gitlab/issue_templates/Feature Flag Roll Out.md +++ b/.gitlab/issue_templates/Feature Flag Roll Out.md @@ -46,7 +46,7 @@ I.e. - [ ] Verify the feature flag was used by checking Prometheus metric [`gitaly_feature_flag_checks_total`](https://prometheus.gstg.gitlab.net/graph?g0.expr=sum%20by%20(flag)%20(rate(gitaly_feature_flag_checks_total%5B5m%5D))&g0.tab=1&g0.stacked=0&g0.range_input=1h) - [ ] Enable on production - [ ] Is the required code deployed on production? ([howto](https://gitlab.com/gitlab-org/gitaly/-/blob/master/doc/PROCESS.md#is-the-required-code-deployed)) - - [ ] Enable on production in `#production` ([howto](https://gitlab.com/gitlab-org/gitaly/-/blob/master/doc/PROCESS.md#enable-in-production)) + - [ ] Progressively enable in production ([howto](https://gitlab.com/gitlab-org/gitaly/-/blob/master/doc/PROCESS.md#enable-in-production)) - [ ] Add ~"featureflag::production" to this issue - [ ] Verify the feature flag was used by checking Prometheus metric [`gitaly_feature_flag_checks_total`](https://prometheus.gprd.gitlab.net/graph?g0.expr=sum%20by%20(flag)%20(rate(gitaly_feature_flag_checks_total%5B5m%5D))&g0.tab=1&g0.stacked=0&g0.range_input=1h) - [ ] Default-enable the feature flag (optional, only required if backwards-compatibility concerns exist) diff --git a/doc/PROCESS.md b/doc/PROCESS.md index c7b190b48..dd7a477c6 100644 --- a/doc/PROCESS.md +++ b/doc/PROCESS.md @@ -152,7 +152,7 @@ repository, and manually test from there. Manually use the feature in whatever way exercises the code paths being enabled. -Then enable `X` on staging, with: +Then enable `X` on staging, run the following in the `#staging` Slack channel: ```shell /chatops run feature set gitaly_X --staging @@ -176,12 +176,20 @@ environment? Good! #### Steps -To enable your `X` feature at 5/25/50 percent, run: +Feature flags must be rolled out in production systems gradually to +reduce risk of incidents. Use percentage based actors instead of enabling a +feature flag fully. The concrete percentages depend on the scope of the feature +flag as well as its inherent risk. In general, add more fine-grained steps the +higher the risk and the broader the scope of the gated feature. + +To enable your `X` feature at 5/25/50 percent, run the following in the +`#production` Slack channel. As Gitaly does not reliably support feature +flag actors, you need to do a fully-randomized rollout: ```shell -/chatops run feature set gitaly_X 5 -/chatops run feature set gitaly_X 25 -/chatops run feature set gitaly_X 50 +/chatops run feature set gitaly_X 5 --random +/chatops run feature set gitaly_X 25 --random +/chatops run feature set gitaly_X 50 --random ``` And then finally when you're happy it works properly do: |