diff options
Diffstat (limited to 'doc/development/testing_guide/end_to_end')
-rw-r--r-- | doc/development/testing_guide/end_to_end/best_practices.md | 53 | ||||
-rw-r--r-- | doc/development/testing_guide/end_to_end/feature_flags.md | 57 |
2 files changed, 102 insertions, 8 deletions
diff --git a/doc/development/testing_guide/end_to_end/best_practices.md b/doc/development/testing_guide/end_to_end/best_practices.md index 36cb49256a6..866a949d795 100644 --- a/doc/development/testing_guide/end_to_end/best_practices.md +++ b/doc/development/testing_guide/end_to_end/best_practices.md @@ -310,3 +310,56 @@ end # Using native mouse click events in the case of a mask/overlay click_element_coordinates(:title) ``` + +## Ensure `expect` statements wait efficiently + +In general, we use an `expect` statement to check that something _is_ as we expect it. For example: + +```ruby +Page::Project::Pipeline::Show.perform do |pipeline| + expect(pipeline).to have_job("a_job") +end +``` + +### Ensure `expect` checks for negation efficiently + +However, sometimes we want to check that something is _not_ as we _don't_ want it to be. In other +words, we want to make sure something is absent. In such a case we should use an appropriate +predicate method that returns quickly, rather than waiting for a state that won't appear. + +It's most efficient to use a predicate method that returns immediately when there is no job, or waits +until it disappears: + +```ruby +# Good +Page::Project::Pipeline::Show.perform do |pipeline| + expect(pipeline).to have_no_job("a_job") +end +``` + +### Problematic alternatives + +Alternatively, if we want to check that a job doesn't exist it might be tempting to use `not_to`: + +```ruby +# Bad +Page::Project::Pipeline::Show.perform do |pipeline| + expect(pipeline).not_to have_job("a_job") +end +``` + +For this statement to pass, `have_job("a_job")` has to return `false` so that `not_to` can negate it. +The problem is that `have_job("a_job")` waits up to ten seconds for `"a job"` to appear before +returning `false`. Under the expected condition this test will take ten seconds longer than it needs to. + +Instead, we could force no wait: + +```ruby +# Not as bad but potentially flaky +Page::Project::Pipeline::Show.perform do |pipeline| + expect(pipeline).not_to have_job("a_job", wait: 0) +end +``` + +The problem is that if `"a_job"` is present and we're waiting for it to disappear, this statement +will fail. diff --git a/doc/development/testing_guide/end_to_end/feature_flags.md b/doc/development/testing_guide/end_to_end/feature_flags.md index 87a9738b313..e571774167d 100644 --- a/doc/development/testing_guide/end_to_end/feature_flags.md +++ b/doc/development/testing_guide/end_to_end/feature_flags.md @@ -1,29 +1,70 @@ # Testing with feature flags -To run a specific test with a feature flag enabled you can use the `QA::Runtime::Feature` class to enable and disable feature flags ([via the API](../../../api/features.md)). +To run a specific test with a feature flag enabled you can use the `QA::Runtime::Feature` class to +enable and disable feature flags ([via the API](../../../api/features.md)). -Note that administrator authorization is required to change feature flags. `QA::Runtime::Feature` will automatically authenticate as an administrator as long as you provide an appropriate access token via `GITLAB_QA_ADMIN_ACCESS_TOKEN` (recommended), or provide `GITLAB_ADMIN_USERNAME` and `GITLAB_ADMIN_PASSWORD`. +Note that administrator authorization is required to change feature flags. `QA::Runtime::Feature` +will automatically authenticate as an administrator as long as you provide an appropriate access +token via `GITLAB_QA_ADMIN_ACCESS_TOKEN` (recommended), or provide `GITLAB_ADMIN_USERNAME` +and `GITLAB_ADMIN_PASSWORD`. -Please be sure to include the tag `:requires_admin` so that the test can be skipped in environments where admin access is not available. +Please be sure to include the tag `:requires_admin` so that the test can be skipped in environments +where admin access is not available. + +CAUTION: **Caution:** +You are strongly advised to [enable feature flags only for a group, project, user](../../feature_flags/development.md#feature-actors), +or [feature group](../../feature_flags/development.md#feature-groups). This makes it possible to +test a feature in a shared environment without affecting other users. + +For example, the code below would enable a feature flag named `:feature_flag_name` for the project +created by the test: ```ruby RSpec.describe "with feature flag enabled", :requires_admin do + let(:project) { Resource::Project.fabricate_via_api! } + before do - Runtime::Feature.enable('feature_flag_name') + Runtime::Feature.enable(:feature_flag_name, project: project) end it "feature flag test" do - # Execute a test with a feature flag enabled + # Execute the test with the feature flag enabled. + # It will only affect the project created in this test. end after do - Runtime::Feature.disable('feature_flag_name') + Runtime::Feature.disable(:feature_flag_name, project: project) end end ``` +Note that the `enable` and `disable` methods first set the flag and then check that the updated +value is returned by the API. + +Similarly, you can enable a feature for a group, user, or feature group: + +```ruby +group = Resource::Group.fabricate_via_api! +Runtime::Feature.enable(:feature_flag_name, group: group) + +user = Resource::User.fabricate_via_api! +Runtime::Feature.enable(:feature_flag_name, user: user) + +feature_group = "a_feature_group" +Runtime::Feature.enable(:feature_flag_name, feature_group: feature_group) +``` + +If no scope is provided, the feature flag will be set instance-wide: + +```ruby +# This will affect all users! +Runtime::Feature.enable(:feature_flag_name) +``` + ## Running a scenario with a feature flag enabled -It's also possible to run an entire scenario with a feature flag enabled, without having to edit existing tests or write new ones. +It's also possible to run an entire scenario with a feature flag enabled, without having to edit +existing tests or write new ones. -Please see the [QA README](https://gitlab.com/gitlab-org/gitlab/tree/master/qa#running-tests-with-a-feature-flag-enabled) for details. +Please see the [QA README](https://gitlab.com/gitlab-org/gitlab/tree/master/qa#running-tests-with-a-feature-flag-enabled) +for details. |