diff options
Diffstat (limited to 'doc/development/pipelines.md')
| -rw-r--r-- | doc/development/pipelines.md | 71 |
1 files changed, 17 insertions, 54 deletions
diff --git a/doc/development/pipelines.md b/doc/development/pipelines.md index 0fe48fe8b9e..820299a426b 100644 --- a/doc/development/pipelines.md +++ b/doc/development/pipelines.md @@ -404,7 +404,7 @@ graph LR A Merge Request author may choose to opt-out of the fail fast mechanism by doing one of the following: -- Including `[SKIP RSPEC FAIL-FAST]` in the Merge Request title. +- Adding the `pipeline:skip-rspec-fail-fast` label to the merge request - Starting the `dont-interrupt-me` job found in the `sync` stage of a Merge Request pipeline. The `rspec fail-fast` is a no-op if there are more than 10 test files related to the @@ -421,58 +421,18 @@ This experiment is only enabled when the CI/CD variable `RSPEC_FAIL_FAST_ENABLED The test files related to the Merge Request are determined using the [`test_file_finder`](https://gitlab.com/gitlab-org/ci-cd/test_file_finder) gem. We are using a custom mapping between source file to test files, maintained in the `tests.yml` file. -### RSpec minimal job experiment +### RSpec minimal jobs -As part of the objective to improve overall pipeline duration, we are experimenting with a minimal set of RSpec tests. -The purpose of this experiment is to verify if we are able to run a minimal set of RSpec tests in a Merge Request pipeline, -without resulting in increased number of broken main branch. +Before a merge request is approved, the pipeline will run a minimal set of RSpec tests that are related to the merge request changes. +This is to reduce the pipeline cost and shorten the job duration. To identify the minimal set of tests needed, we use [Crystalball gem](https://github.com/toptal/crystalball) to create a test mapping. The test mapping contains a map of each source files to a list of test files which is dependent of the source file. This mapping is currently generated using a combination of test coverage tracing and a static mapping. In the `detect-tests` job, we use this mapping to identify the minimal tests needed for the current Merge Request. -In this experiment, each `rspec` job is accompanied with a `minimal` version. -For example, `rspec unit` job has a corresponding `rspec unit minimal` job. -During the experiment, each Merge Request pipeline will contain both versions of the job, running in parallel. - -To illustrate this: - -```mermaid -graph LR - A --"artifact: list of test files"--> C1 & D1 & E1 & F1 - - subgraph "prepare stage"; - A["detect-tests"]; - end - - subgraph "test stage"; - C["rspec migration"]; - C1["rspec migration minimal"]; - D["rspec unit"]; - D1["rspec unit minimal"]; - E["rspec integration"]; - E1["rspec integration minimal"]; - F["rspec system"]; - F1["rspec system minimal"]; - end -``` - -The result of both set of jobs in the pipeline is then compared to identify any false positive. -A list of such pipeline can be found in [Sisense](https://app.periscopedata.com/app/gitlab/496118/Engineering-Productivity-Sandbox?widget=10492739&udv=833427). - -A false positive is defined as a pipeline where the `minimal` jobs passed, but the non-`minimal` jobs failed. -This indicates that the changeset resulted in a test failure, which was not detected by the `minimal` jobs. -Consequently, this signifies a gap in the test mapping used, which would need to be rectified. - -#### Findings - -After a round of the experiment done in December 2020, -we discovered that it was challenging to achieve a mapping that gives high confidence at the moment, -because of 2 reasons: - -- Each identified gap in the test mapping is unique, each needing its own investigation and improvement to the creation of the test mapping. -- There is a large number of flaky tests which added a lot of noise in the data, slowing down the investigation process. +After a merge request has been approved, the pipeline would contain the full RSpec tests. This will ensure that all tests +have been run before a merge request is merged. ### PostgreSQL versions testing @@ -508,8 +468,9 @@ for more information. We have dedicated jobs for each [testing level](testing_guide/testing_levels.md) and each job runs depending on the changes made in your merge request. -If you want to force all the RSpec jobs to run regardless of your changes, you can include `RUN ALL RSPEC` in your merge -request title. +If you want to force all the RSpec jobs to run regardless of your changes, you can add the `pipeline:run-all-rspec` label to the merge request. + +> Forcing all jobs on docs only related MRs would not have the prerequisite jobs and would lead to errors ### Review app jobs @@ -521,7 +482,7 @@ The `* as-if-foss` jobs allows the GitLab test suite "as-if-FOSS", meaning as if of the `gitlab-org/gitlab-foss` project. These jobs are only created in the following cases: - `gitlab-org/security/gitlab` merge requests. -- Merge requests which include `RUN AS-IF-FOSS` in their title. +- Merge requests with the `pipeline:run-as-if-foss` label - Merge requests that changes the CI configuration. The `* as-if-foss` jobs are run in addition to the regular EE-context jobs. They have the `FOSS_ONLY='1'` variable @@ -566,7 +527,7 @@ request, be sure to start the `dont-interrupt-me` job before pushing. - `update-assets-compile-test-cache`, defined in [`.gitlab/ci/frontend.gitlab-ci.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/.gitlab/ci/frontend.gitlab-ci.yml). - `update-yarn-cache`, defined in [`.gitlab/ci/frontend.gitlab-ci.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/.gitlab/ci/frontend.gitlab-ci.yml). - `update-storybook-yarn-cache`, defined in [`.gitlab/ci/frontend.gitlab-ci.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/.gitlab/ci/frontend.gitlab-ci.yml). -1. These jobs can also be forced to run in merge requests whose title include `UPDATE CACHE` (this can be useful to warm the caches in a MR that updates the cache keys). +1. These jobs can also be forced to run in merge requests with the `pipeline:update-cache` label (this can be useful to warm the caches in a MR that updates the cache keys). ### Artifacts strategy @@ -676,7 +637,9 @@ that is deployed in stage `review`. The default image is defined in [`.gitlab-ci.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/.gitlab-ci.yml). <!-- vale gitlab.Spelling = NO --> + It includes Ruby, Go, Git, Git LFS, Chrome, Node, Yarn, PostgreSQL, and Graphics Magick. + <!-- vale gitlab.Spelling = YES --> The images used in our pipelines are configured in the @@ -767,9 +730,9 @@ and included in `rules` definitions via [YAML anchors](../ci/yaml/index.md#ancho | `if-auto-deploy-branches` | Matches if the current branch is an auto-deploy one. | | | `if-master-or-tag` | Matches if the pipeline is for the `master` or `main` branch or for a tag. | | | `if-merge-request` | Matches if the pipeline is for a merge request. | | -| `if-merge-request-title-as-if-foss` | Matches if the pipeline is for a merge request and the MR title includes "RUN AS-IF-FOSS". | | -| `if-merge-request-title-update-caches` | Matches if the pipeline is for a merge request and the MR title includes "UPDATE CACHE". | | -| `if-merge-request-title-run-all-rspec` | Matches if the pipeline is for a merge request and the MR title includes "RUN ALL RSPEC". | | +| `if-merge-request-title-as-if-foss` | Matches if the pipeline is for a merge request and the MR has label ~"pipeline:run-as-if-foss" | | +| `if-merge-request-title-update-caches` | Matches if the pipeline is for a merge request and the MR has label ~"pipeline:update-cache". | | +| `if-merge-request-title-run-all-rspec` | Matches if the pipeline is for a merge request and the MR has label ~"pipeline:run-all-rspec". | | | `if-security-merge-request` | Matches if the pipeline is for a security merge request. | | | `if-security-schedule` | Matches if the pipeline is for a security scheduled pipeline. | | | `if-nightly-master-schedule` | Matches if the pipeline is for a `master` scheduled pipeline with `$NIGHTLY` set. | | @@ -782,7 +745,7 @@ and included in `rules` definitions via [YAML anchors](../ci/yaml/index.md#ancho | `if-dot-com-ee-schedule` | Limits jobs to scheduled pipelines for the `gitlab-org/gitlab` project on GitLab.com. | | | `if-cache-credentials-schedule` | Limits jobs to scheduled pipelines with the `$CI_REPO_CACHE_CREDENTIALS` variable set. | | | `if-rspec-fail-fast-disabled` | Limits jobs to pipelines with `$RSPEC_FAIL_FAST_ENABLED` CI/CD variable not set to `"true"`. | | -| `if-rspec-fail-fast-skipped` | Matches if the pipeline is for a merge request and the MR title includes "SKIP RSPEC FAIL-FAST". | | +| `if-rspec-fail-fast-skipped` | Matches if the pipeline is for a merge request and the MR has label ~"pipeline:skip-rspec-fail-fast". | | | `if-security-pipeline-merge-result` | Matches if the pipeline is for a security merge request triggered by `@gitlab-release-tools-bot`. | | <!-- vale gitlab.Substitutions = YES --> |