diff options
Diffstat (limited to 'doc/development/testing_guide/flaky_tests.md')
-rw-r--r-- | doc/development/testing_guide/flaky_tests.md | 48 |
1 files changed, 24 insertions, 24 deletions
diff --git a/doc/development/testing_guide/flaky_tests.md b/doc/development/testing_guide/flaky_tests.md index 5015c613557..9ec0a8e803f 100644 --- a/doc/development/testing_guide/flaky_tests.md +++ b/doc/development/testing_guide/flaky_tests.md @@ -48,7 +48,7 @@ examples in a JSON report file on `master` (`retrieve-tests-metadata` and `updat is detected in any other branch (`flaky-examples-check` job). In the future, the `flaky-examples-check` job will not be allowed to fail. -This was originally implemented in: <https://gitlab.com/gitlab-org/gitlab-foss/merge_requests/13021>. +This was originally implemented in: <https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/13021>. If you want to enable retries locally, you can use the `RETRIES` env variable. For instance `RETRIES=1 bin/rspec ...` would retry the failing examples once. @@ -58,52 +58,52 @@ For instance `RETRIES=1 bin/rspec ...` would retry the failing examples once. ## Problems we had in the past at GitLab -- [`rspec-retry` is biting us when some API specs fail](https://gitlab.com/gitlab-org/gitlab-foss/issues/29242): <https://gitlab.com/gitlab-org/gitlab-foss/merge_requests/9825> -- [Sporadic RSpec failures due to `PG::UniqueViolation`](https://gitlab.com/gitlab-org/gitlab-foss/issues/28307#note_24958837): <https://gitlab.com/gitlab-org/gitlab-foss/merge_requests/9846> - - Follow-up: <https://gitlab.com/gitlab-org/gitlab-foss/merge_requests/10688> - - [Capybara.reset_session! should be called before requests are blocked](https://gitlab.com/gitlab-org/gitlab-foss/issues/33779): <https://gitlab.com/gitlab-org/gitlab-foss/merge_requests/12224> +- [`rspec-retry` is biting us when some API specs fail](https://gitlab.com/gitlab-org/gitlab-foss/issues/29242): <https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/9825> +- [Sporadic RSpec failures due to `PG::UniqueViolation`](https://gitlab.com/gitlab-org/gitlab-foss/issues/28307#note_24958837): <https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/9846> + - Follow-up: <https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/10688> + - [Capybara.reset_session! should be called before requests are blocked](https://gitlab.com/gitlab-org/gitlab-foss/issues/33779): <https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/12224> - FFaker generates funky data that tests are not ready to handle (and tests should be predictable so that's bad!): - - [Make `spec/mailers/notify_spec.rb` more robust](https://gitlab.com/gitlab-org/gitlab-foss/issues/20121): <https://gitlab.com/gitlab-org/gitlab-foss/merge_requests/10015> - - [Transient failure in `spec/requests/api/commits_spec.rb`](https://gitlab.com/gitlab-org/gitlab-foss/issues/27988#note_25342521): <https://gitlab.com/gitlab-org/gitlab-foss/merge_requests/9944> - - [Replace FFaker factory data with sequences](https://gitlab.com/gitlab-org/gitlab-foss/issues/29643): <https://gitlab.com/gitlab-org/gitlab-foss/merge_requests/10184> - - [Transient failure in spec/finders/issues_finder_spec.rb](https://gitlab.com/gitlab-org/gitlab-foss/issues/30211#note_26707685): <https://gitlab.com/gitlab-org/gitlab-foss/merge_requests/10404> + - [Make `spec/mailers/notify_spec.rb` more robust](https://gitlab.com/gitlab-org/gitlab-foss/issues/20121): <https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/10015> + - [Transient failure in `spec/requests/api/commits_spec.rb`](https://gitlab.com/gitlab-org/gitlab-foss/issues/27988#note_25342521): <https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/9944> + - [Replace FFaker factory data with sequences](https://gitlab.com/gitlab-org/gitlab-foss/issues/29643): <https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/10184> + - [Transient failure in spec/finders/issues_finder_spec.rb](https://gitlab.com/gitlab-org/gitlab-foss/issues/30211#note_26707685): <https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/10404> ### Time-sensitive flaky tests -- <https://gitlab.com/gitlab-org/gitlab-foss/merge_requests/10046> -- <https://gitlab.com/gitlab-org/gitlab-foss/merge_requests/10306> +- <https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/10046> +- <https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/10306> ### Array order expectation -- <https://gitlab.com/gitlab-org/gitlab-foss/merge_requests/10148> +- <https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/10148> ### Feature tests -- [Be sure to create all the data the test need before starting exercise](https://gitlab.com/gitlab-org/gitlab-foss/issues/32622#note_31128195): <https://gitlab.com/gitlab-org/gitlab-foss/merge_requests/12059> -- [Bis](https://gitlab.com/gitlab-org/gitlab-foss/issues/34609#note_34048715): <https://gitlab.com/gitlab-org/gitlab-foss/merge_requests/12604> -- [Bis](https://gitlab.com/gitlab-org/gitlab-foss/issues/34698#note_34276286): <https://gitlab.com/gitlab-org/gitlab-foss/merge_requests/12664> -- [Assert against the underlying database state instead of against a page's content](https://gitlab.com/gitlab-org/gitlab-foss/issues/31437): <https://gitlab.com/gitlab-org/gitlab-foss/merge_requests/10934> +- [Be sure to create all the data the test need before starting exercise](https://gitlab.com/gitlab-org/gitlab-foss/issues/32622#note_31128195): <https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/12059> +- [Bis](https://gitlab.com/gitlab-org/gitlab-foss/issues/34609#note_34048715): <https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/12604> +- [Bis](https://gitlab.com/gitlab-org/gitlab-foss/issues/34698#note_34276286): <https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/12664> +- [Assert against the underlying database state instead of against a page's content](https://gitlab.com/gitlab-org/gitlab-foss/issues/31437): <https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/10934> - In JS tests, shifting elements can cause Capybara to misclick when the element moves at the exact time Capybara sends the click - - [Dropdowns rendering upward or downward due to window size and scroll position](https://gitlab.com/gitlab-org/gitlab/merge_requests/17660) - - [Lazy loaded images can cause Capybara to misclick](https://gitlab.com/gitlab-org/gitlab/merge_requests/18713) -- [Triggering JS events before the event handlers are set up](https://gitlab.com/gitlab-org/gitlab/merge_requests/18742) + - [Dropdowns rendering upward or downward due to window size and scroll position](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/17660) + - [Lazy loaded images can cause Capybara to misclick](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/18713) +- [Triggering JS events before the event handlers are set up](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/18742) #### Capybara viewport size related issues -- [Transient failure of spec/features/issues/filtered_search/filter_issues_spec.rb](https://gitlab.com/gitlab-org/gitlab-foss/issues/29241#note_26743936): <https://gitlab.com/gitlab-org/gitlab-foss/merge_requests/10411> +- [Transient failure of spec/features/issues/filtered_search/filter_issues_spec.rb](https://gitlab.com/gitlab-org/gitlab-foss/issues/29241#note_26743936): <https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/10411> #### Capybara JS driver related issues -- [Don't wait for AJAX when no AJAX request is fired](https://gitlab.com/gitlab-org/gitlab-foss/issues/30461): <https://gitlab.com/gitlab-org/gitlab-foss/merge_requests/10454> -- [Bis](https://gitlab.com/gitlab-org/gitlab-foss/issues/34647): <https://gitlab.com/gitlab-org/gitlab-foss/merge_requests/12626> +- [Don't wait for AJAX when no AJAX request is fired](https://gitlab.com/gitlab-org/gitlab-foss/issues/30461): <https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/10454> +- [Bis](https://gitlab.com/gitlab-org/gitlab-foss/issues/34647): <https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/12626> #### PhantomJS / WebKit related issues -- Memory is through the roof! (TL;DR: Load images but block images requests!): <https://gitlab.com/gitlab-org/gitlab-foss/merge_requests/12003> +- Memory is through the roof! (TL;DR: Load images but block images requests!): <https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/12003> #### Capybara expectation times out -- [Test imports a project (via Sidekiq) that is growing over time, leading to timeouts when the import takes longer than 60 seconds](https://gitlab.com/gitlab-org/gitlab/merge_requests/22599) +- [Test imports a project (via Sidekiq) that is growing over time, leading to timeouts when the import takes longer than 60 seconds](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/22599) ## Resources |