diff options
Diffstat (limited to 'doc/development/testing_guide/end_to_end/feature_flags.md')
| -rw-r--r-- | doc/development/testing_guide/end_to_end/feature_flags.md | 26 |
1 files changed, 21 insertions, 5 deletions
diff --git a/doc/development/testing_guide/end_to_end/feature_flags.md b/doc/development/testing_guide/end_to_end/feature_flags.md index b4ec9e8ccd3..cb4c8e8a6e8 100644 --- a/doc/development/testing_guide/end_to_end/feature_flags.md +++ b/doc/development/testing_guide/end_to_end/feature_flags.md @@ -30,7 +30,7 @@ feature flag is under test. - Format: `feature_flag: { name: 'feature_flag_name', scope: :project }` - When `scope` is set to `:global`, the test will be **skipped on all live .com environments**. This is to avoid issues with feature flag changes affecting other tests or users on that environment. -- When `scope` is set to any other value (such as `:project`, `:group` or `:user`), or if no `scope` is specified, the test will only be **skipped on canary and production**. +- When `scope` is set to any other value (such as `:project`, `:group` or `:user`), or if no `scope` is specified, the test will only be **skipped on canary, production, and preprod**. This is due to the fact that administrator access is not available there. **WARNING:** You are strongly advised to first try and [enable feature flags only for a group, project, user](../../feature_flags/index.md#feature-actors), @@ -192,10 +192,26 @@ End-to-end tests should pass with a feature flag enabled before it is enabled on ### Automatic test execution when a feature flag definition changes -If a merge request adds or edits a [feature flag definition file](../../feature_flags/index.md#feature-flag-definition-and-validation), -two `package-and-qa` jobs will be included automatically in the merge request pipeline. One job will enable the defined -feature flag and the other will disable it. The jobs execute the same suite of tests to confirm that they pass with if -the feature flag is either enabled or disabled. +There are two ways to confirm that end-to-end tests pass: + +- If a merge request adds or edits a [feature flag definition file](../../feature_flags/index.md#feature-flag-definition-and-validation), + two `package-and-qa` jobs (`package-and-qa-ff-enabled` and `package-and-qa-ff-disabled`) are included automatically in the merge request + pipeline. One job enables the defined feature flag and the other job disables it. The jobs execute the same suite of tests to confirm + that they pass with the feature flag either enabled or disabled. +- In some cases, if `package-and-qa` hasn't been triggered automatically, or if it has run the tests with the default feature flag values + (which might not be desired), you can create a Draft MR that enables the feature flag to ensure that all E2E tests pass with the feature + flag enabled. + +### Troubleshooting end-to-end test failures with feature flag enabled + +If enabling the feature flag results in E2E test failures, you can browse the artifacts in the failed pipeline to see screenshots of the failed tests. After which, you can either: + +- Identify tests that need to be updated and contact the relevant [counterpart Software Engineer in Test](https://about.gitlab.com/handbook/engineering/quality/#individual-contributors) responsible for updating the tests or assisting another engineer to do so. However, if a change does not go through [quad-planning](https://about.gitlab.com/handbook/engineering/quality/quality-engineering/quad-planning/) and a required test update is not made, test failures could block deployment. +- Run the failed tests [locally](https://gitlab.com/gitlab-org/gitlab/-/tree/master/qa#run-the-end-to-end-tests-in-a-local-development-environment) + with the [feature flag enabled](https://gitlab.com/gitlab-org/gitlab/-/tree/master/qa#running-tests-with-a-feature-flag-enabled-or-disabled). + This option requires considerable amount of setup, but you'll be able to see what the browser is doing as it's running the failed + tests, which can help debug the problem faster. You can also refer to the [Troubleshooting Guide for E2E tests](troubleshooting.md) for + support for common blockers. ### Test execution during feature development |