diff options
author | Rémy Coutable <remy@rymai.me> | 2019-01-17 11:46:51 +0300 |
---|---|---|
committer | Rémy Coutable <remy@rymai.me> | 2019-01-17 13:23:36 +0300 |
commit | e6dea31365aba993ab3bbaa0e4e3d96e5b2213a5 (patch) | |
tree | 6e0b317d92dae8795562f98e2ad63ee1bc614254 /doc/development/testing_guide/flaky_tests.md | |
parent | f598daf28490d6cb131d060bc1a4d13447352f4c (diff) |
Document better the process around quarantined tests
Signed-off-by: Rémy Coutable <remy@rymai.me>
Diffstat (limited to 'doc/development/testing_guide/flaky_tests.md')
-rw-r--r-- | doc/development/testing_guide/flaky_tests.md | 33 |
1 files changed, 27 insertions, 6 deletions
diff --git a/doc/development/testing_guide/flaky_tests.md b/doc/development/testing_guide/flaky_tests.md index f6bc9fb0979..88b862d2a3b 100644 --- a/doc/development/testing_guide/flaky_tests.md +++ b/doc/development/testing_guide/flaky_tests.md @@ -7,16 +7,37 @@ eventually. ## Quarantined tests -Tests can be put in quarantine by assigning `:quarantine` metadata. This means -they will be skipped unless run with `--tag quarantine`. This can be used for -tests that are expected to fail while a fix is in progress (similar to how -[`skip` or `pending`](https://relishapp.com/rspec/rspec-core/v/3-8/docs/pending-and-skipped-examples) - can be used). +When a test frequently fails in `master`, +[a ~"broken master" issue](https://about.gitlab.com/handbook/engineering/workflow/#broken-master) +should be created. +If the test cannot be fixed in a timely fashion, there is an impact on the +productivity of all the developers, so you can decide to put it in quarantine by +assigning the `:quarantine` metadata. -``` +This means it will be skipped unless run with `--tag quarantine`: + +```shell bin/rspec --tag quarantine ``` +**Before putting a test in quarantine, you should make sure that a +~"broken master" issue exists for it so it won't stay in quarantine forever.** + +Once a test is in quarantine, there are 3 choices: + +- Should the test be fixed (i.e. get rid of its flakiness)? +- Should the test be moved to a lower level of testing? +- Should the test be removed entirely (e.g. because there's already a + lower-level test, or it's duplicating another same-level test, or it's testing + too much etc.)? + +### Quarantine tests on the CI + +Quarantined tests are run on the CI in dedicated jobs that are allowed to fail: + +- `rspec-pg-quarantine` and `rspec-mysql-quarantine` (CE & EE) +- `rspec-pg-quarantine-ee` and `rspec-mysql-quarantine-ee` (EE only) + ## Automatic retries and flaky tests detection On our CI, we use [rspec-retry] to automatically retry a failing example a few |