diff options
| author | Mek Stittri <mstittri@gitlab.com> | 2019-01-22 13:03:37 +0300 |
|---|---|---|
| committer | Mek Stittri <mstittri@gitlab.com> | 2019-01-22 13:03:37 +0300 |
| commit | 3de51c11b5835527ab0cc618a43da1d2f17e4169 (patch) | |
| tree | 17ab9b8b605a5fb7de0aeac330a2f16d1f2d7a55 /doc | |
| parent | c23534c50f7debf219b282fa3d5777fe750927b0 (diff) | |
| parent | 41fba9575774d9a966b506073dbf3e2d0ab19ec2 (diff) | |
Merge branch 'docs-best-practice-with-quarantined-tests' into 'master'
Document better the process around quarantined tests
See merge request gitlab-org/gitlab-ce!24452
Diffstat (limited to 'doc')
| -rw-r--r-- | doc/development/testing_guide/flaky_tests.md | 33 |
1 files changed, 27 insertions, 6 deletions
diff --git a/doc/development/testing_guide/flaky_tests.md b/doc/development/testing_guide/flaky_tests.md index f6bc9fb0979..58a2e53b486 100644 --- a/doc/development/testing_guide/flaky_tests.md +++ b/doc/development/testing_guide/flaky_tests.md @@ -7,16 +7,37 @@ eventually. ## Quarantined tests -Tests can be put in quarantine by assigning `:quarantine` metadata. This means -they will be skipped unless run with `--tag quarantine`. This can be used for -tests that are expected to fail while a fix is in progress (similar to how -[`skip` or `pending`](https://relishapp.com/rspec/rspec-core/v/3-8/docs/pending-and-skipped-examples) - can be used). +When a test frequently fails in `master`, +[a ~"broken master" issue](https://about.gitlab.com/handbook/engineering/workflow/#broken-master) +should be created. +If the test cannot be fixed in a timely fashion, there is an impact on the +productivity of all the developers, so it should be placed in quarantine by +assigning the `:quarantine` metadata. -``` +This means it will be skipped unless run with `--tag quarantine`: + +```shell bin/rspec --tag quarantine ``` +**Before putting a test in quarantine, you should make sure that a +~"broken master" issue exists for it so it won't stay in quarantine forever.** + +Once a test is in quarantine, there are 3 choices: + +- Should the test be fixed (i.e. get rid of its flakiness)? +- Should the test be moved to a lower level of testing? +- Should the test be removed entirely (e.g. because there's already a + lower-level test, or it's duplicating another same-level test, or it's testing + too much etc.)? + +### Quarantine tests on the CI + +Quarantined tests are run on the CI in dedicated jobs that are allowed to fail: + +- `rspec-pg-quarantine` and `rspec-mysql-quarantine` (CE & EE) +- `rspec-pg-quarantine-ee` and `rspec-mysql-quarantine-ee` (EE only) + ## Automatic retries and flaky tests detection On our CI, we use [rspec-retry] to automatically retry a failing example a few |