diff options
author | GitLab Bot <gitlab-bot@gitlab.com> | 2021-05-19 18:44:42 +0300 |
---|---|---|
committer | GitLab Bot <gitlab-bot@gitlab.com> | 2021-05-19 18:44:42 +0300 |
commit | 4555e1b21c365ed8303ffb7a3325d773c9b8bf31 (patch) | |
tree | 5423a1c7516cffe36384133ade12572cf709398d /doc/development/testing_guide/flaky_tests.md | |
parent | e570267f2f6b326480d284e0164a6464ba4081bc (diff) |
Add latest changes from gitlab-org/gitlab@13-12-stable-eev13.12.0-rc42
Diffstat (limited to 'doc/development/testing_guide/flaky_tests.md')
-rw-r--r-- | doc/development/testing_guide/flaky_tests.md | 27 |
1 files changed, 27 insertions, 0 deletions
diff --git a/doc/development/testing_guide/flaky_tests.md b/doc/development/testing_guide/flaky_tests.md index a9af8f03d63..6b1c7a7eb58 100644 --- a/doc/development/testing_guide/flaky_tests.md +++ b/doc/development/testing_guide/flaky_tests.md @@ -76,6 +76,33 @@ For instance `RETRIES=1 bin/rspec ...` would retry the failing examples once. - [Replace FFaker factory data with sequences](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/29643): <https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/10184> - [Transient failure in spec/finders/issues_finder_spec.rb](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/30211#note_26707685): <https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/10404> +### Order-dependent flaky tests + +These flaky tests can fail depending on the order they run with other tests. For example: + +- <https://gitlab.com/gitlab-org/gitlab/-/issues/327668> + +To identify the tests that lead to such failure, we can use `rspec --bisect`, +which would give us the minimal test combination to reproduce the failure: + +```shell +rspec --bisect ee/spec/services/ee/merge_requests/update_service_spec.rb ee/spec/services/ee/notes/quick_actions_service_spec.rb ee/spec/services/epic_links/create_service_spec.rb ee/spec/services/ee/issuable/bulk_update_service_spec.rb +Bisect started using options: "ee/spec/services/ee/merge_requests/update_service_spec.rb ee/spec/services/ee/notes/quick_actions_service_spec.rb ee/spec/services/epic_links/create_service_spec.rb ee/spec/services/ee/issuable/bulk_update_service_spec.rb" +Running suite to find failures... (2 minutes 18.4 seconds) +Starting bisect with 3 failing examples and 144 non-failing examples. +Checking that failure(s) are order-dependent... failure appears to be order-dependent + +Round 1: bisecting over non-failing examples 1-144 . ignoring examples 1-72 (1 minute 11.33 seconds) +... +Round 7: bisecting over non-failing examples 132-133 . ignoring example 132 (43.78 seconds) +Bisect complete! Reduced necessary non-failing examples from 144 to 1 in 8 minutes 31 seconds. + +The minimal reproduction command is: + rspec ./ee/spec/services/ee/issuable/bulk_update_service_spec.rb[1:2:1:1:1:1,1:2:1:2:1:1,1:2:1:3:1] ./ee/spec/services/epic_links/create_service_spec.rb[1:1:2:2:6:4] +``` + +We can reproduce the test failure with the reproduction command above. If we change the order of the tests, the test would pass. + ### Time-sensitive flaky tests - <https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/10046> |