diff options
Diffstat (limited to 'doc/development/pipelines.md')
| -rw-r--r-- | doc/development/pipelines.md | 631 |
1 files changed, 339 insertions, 292 deletions
diff --git a/doc/development/pipelines.md b/doc/development/pipelines.md index dd45091a31b..45982d6075b 100644 --- a/doc/development/pipelines.md +++ b/doc/development/pipelines.md @@ -6,8 +6,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w # Pipelines for the GitLab project -Pipelines for [`gitlab-org/gitlab`](https://gitlab.com/gitlab-org/gitlab) and [`gitlab-org/gitlab-foss`](https://gitlab.com/gitlab-org/gitlab-foss) (as well as the -`dev` instance's mirrors) are configured in the usual +Pipelines for [`gitlab-org/gitlab`](https://gitlab.com/gitlab-org/gitlab) (as well as the `dev` instance's) is configured in the usual [`.gitlab-ci.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/.gitlab-ci.yml) which itself includes files under [`.gitlab/ci/`](https://gitlab.com/gitlab-org/gitlab/-/tree/master/.gitlab/ci) @@ -17,29 +16,206 @@ We're striving to [dogfood](https://about.gitlab.com/handbook/engineering/#dogfo GitLab [CI/CD features and best-practices](../ci/yaml/index.md) as much as possible. -## Overview +## Minimal test jobs before a merge request is approved -Pipelines for the GitLab project are created using the [`workflow:rules` keyword](../ci/yaml/index.md#workflow) -feature of the GitLab CI/CD. +**To reduce the pipeline cost and shorten the job duration, before a merge request is approved, the pipeline will run a minimal set of RSpec & Jest tests that are related to the merge request changes.** -Pipelines are always created for the following scenarios: +After a merge request has been approved, the pipeline would contain the full RSpec & Jest tests. This will ensure that all tests +have been run before a merge request is merged. -- `main` branch, including on schedules, pushes, merges, and so on. -- Merge requests. -- Tags. -- Stable, `auto-deploy`, and security branches. +### RSpec minimal jobs -Pipeline creation is also affected by the following CI/CD variables: +#### Determining related RSpec test files in a merge request -- If `$FORCE_GITLAB_CI` is set, pipelines are created. -- If `$GITLAB_INTERNAL` is not set, pipelines are not created. +To identify the minimal set of tests needed, we use the [`test_file_finder` gem](https://gitlab.com/gitlab-org/ci-cd/test_file_finder), with two strategies: -No pipeline is created in any other cases (for example, when pushing a branch with no -MR for it). +- dynamic mapping from test coverage tracing (generated via the [Crystalball gem](https://github.com/toptal/crystalball)) + ([see where it's used](https://gitlab.com/gitlab-org/gitlab/-/blob/47d507c93779675d73a05002e2ec9c3c467cd698/tooling/bin/find_tests#L15)) +- static mapping maintained in the [`tests.yml` file](https://gitlab.com/gitlab-org/gitlab/-/blob/master/tests.yml) for special cases that cannot + be mapped via coverage tracing ([see where it's used](https://gitlab.com/gitlab-org/gitlab/-/blob/47d507c93779675d73a05002e2ec9c3c467cd698/tooling/bin/find_tests#L12)) -The source of truth for these workflow rules is defined in [`.gitlab-ci.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/.gitlab-ci.yml). +The test mappings contain a map of each source files to a list of test files which is dependent of the source file. + +In the `detect-tests` job, we use this mapping to identify the minimal tests needed for the current merge request. + +#### Exceptional cases + +In addition, there are a few circumstances where we would always run the full RSpec tests: + +- when the `pipeline:run-all-rspec` label is set on the merge request +- when the merge request is created by an automation (e.g. Gitaly update or MR targeting a stable branch) +- when any CI config file is changed (i.e. `.gitlab-ci.yml` or `.gitlab/ci/**/*`) + +### Jest minimal jobs + +#### Determining related Jest test files in a merge request + +To identify the minimal set of tests needed, we pass a list of all the changed files into `jest` using the [`--findRelatedTests`](https://jestjs.io/docs/cli#--findrelatedtests-spaceseparatedlistofsourcefiles) option. +In this mode, `jest` would resolve all the dependencies of related to the changed files, which include test files that have these files in the dependency chain. + +#### Exceptional cases + +In addition, there are a few circumstances where we would always run the full Jest tests: + +- when the `pipeline:run-all-rspec` label is set on the merge request +- when the merge request is created by an automation (e.g. Gitaly update or MR targeting a stable branch) +- when any CI config file is changed (i.e. `.gitlab-ci.yml` or `.gitlab/ci/**/*`) +- when any frontend "core" file is changed (i.e. `package.json`, `yarn.lock`, `babel.config.js`, `jest.config.*.js`, `config/helpers/**/*.js`) +- when any vendored JavaScript file is changed (i.e. `vendor/assets/javascripts/**/*`) +- when any backend file is changed ([see the patterns list for details](https://gitlab.com/gitlab-org/gitlab/-/blob/3616946936c1adbd9e754c1bd06f86ba670796d8/.gitlab/ci/rules.gitlab-ci.yml#L205-216)) + +## Fail-fast job in merge request pipelines + +To provide faster feedback when a merge request breaks existing tests, we are experimenting with a +fail-fast mechanism. + +An `rspec fail-fast` job is added in parallel to all other `rspec` jobs in a merge +request pipeline. This job runs the tests that are directly related to the changes +in the merge request. + +If any of these tests fail, the `rspec fail-fast` job fails, triggering a +`fail-pipeline-early` job to run. The `fail-pipeline-early` job: + +- Cancels the currently running pipeline and all in-progress jobs. +- Sets pipeline to have status `failed`. + +For example: + +```mermaid +graph LR + subgraph "prepare stage"; + A["detect-tests"] + end + + subgraph "test stage"; + B["jest"]; + C["rspec migration"]; + D["rspec unit"]; + E["rspec integration"]; + F["rspec system"]; + G["rspec fail-fast"]; + end + + subgraph "post-test stage"; + Z["fail-pipeline-early"]; + end + + A --"artifact: list of test files"--> G + G --"on failure"--> Z +``` + +The `rspec fail-fast` is a no-op if there are more than 10 test files related to the +merge request. This prevents `rspec fail-fast` duration from exceeding the average +`rspec` job duration and defeating its purpose. + +This number can be overridden by setting a CI/CD variable named `RSPEC_FAIL_FAST_TEST_FILE_COUNT_THRESHOLD`. + +## Test jobs + +We have dedicated jobs for each [testing level](testing_guide/testing_levels.md) and each job runs depending on the +changes made in your merge request. +If you want to force all the RSpec jobs to run regardless of your changes, you can add the `pipeline:run-all-rspec` label to the merge request. + +WARNING: +Forcing all jobs on docs only related MRs would not have the prerequisite jobs and would lead to errors + +### Test suite parallelization + +Our current RSpec tests parallelization setup is as follows: + +1. The `retrieve-tests-metadata` job in the `prepare` stage ensures we have a + `knapsack/report-master.json` file: + - The `knapsack/report-master.json` file is fetched from the latest `main` pipeline which runs `update-tests-metadata` + (for now it's the 2-hourly scheduled master pipeline), if it's not here we initialize the file with `{}`. +1. Each `[rspec|rspec-ee] [unit|integration|system|geo] n m` job are run with + `knapsack rspec` and should have an evenly distributed share of tests: + - It works because the jobs have access to the `knapsack/report-master.json` + since the "artifacts from all previous stages are passed by default". + - the jobs set their own report path to + `"knapsack/${TEST_TOOL}_${TEST_LEVEL}_${DATABASE}_${CI_NODE_INDEX}_${CI_NODE_TOTAL}_report.json"`. + - if knapsack is doing its job, test files that are run should be listed under + `Report specs`, not under `Leftover specs`. +1. The `update-tests-metadata` job (which only runs on scheduled pipelines for + [the canonical project](https://gitlab.com/gitlab-org/gitlab) takes all the + `knapsack/rspec*_pg_*.json` files and merge them all together into a single + `knapsack/report-master.json` file that is saved as artifact. + +After that, the next pipeline uses the up-to-date `knapsack/report-master.json` file. + +### Monitoring + +The GitLab test suite is [monitored](performance.md#rspec-profiling) for the `main` branch, and any branch +that includes `rspec-profile` in their name. + +### Logging + +- Rails logging to `log/test.log` is disabled by default in CI [for + performance reasons](https://jtway.co/speed-up-your-rails-test-suite-by-6-in-1-line-13fedb869ec4). To override this setting, provide the + `RAILS_ENABLE_TEST_LOG` environment variable. + +## Review app jobs + +Consult the [Review Apps](testing_guide/review_apps.md) dedicated page for more information. + +## As-if-FOSS jobs + +The `* as-if-foss` jobs run the GitLab test suite "as if FOSS", meaning as if the jobs would run in the context +of the `gitlab-org/gitlab-foss` project. These jobs are only created in the following cases: + +- when the `pipeline:run-as-if-foss` label is set on the merge request +- when the merge request is created in the `gitlab-org/security/gitlab` project +- when any CI config file is changed (i.e. `.gitlab-ci.yml` or `.gitlab/ci/**/*`) + +The `* as-if-foss` jobs are run in addition to the regular EE-context jobs. They have the `FOSS_ONLY='1'` variable +set and get the `ee/` folder removed before the tests start running. + +The intent is to ensure that a change doesn't introduce a failure after the `gitlab-org/gitlab` project is synced to +the `gitlab-org/gitlab-foss` project. + +## As-if-JH jobs + +The `* as-if-jh` jobs run the GitLab test suite "as if JiHu", meaning as if the jobs would run in the context +of [the `gitlab-jh/gitlab` project](jh_features_review.md). These jobs are only created in the following cases: + +- when the `pipeline:run-as-if-jh` label is set on the merge request +- when the `pipeline:run-all-rspec` label is set on the merge request +- when any code or backstage file is changed +- when any startup CSS file is changed + +The `* as-if-jh` jobs are run in addition to the regular EE-context jobs. The `jh/` folder is added before the tests start running. + +The intent is to ensure that a change doesn't introduce a failure after the `gitlab-org/gitlab` project is synced to +the `gitlab-jh/gitlab` project. + +## PostgreSQL versions testing + +Our test suite runs against PG12 as GitLab.com runs on PG12 and +[Omnibus defaults to PG12 for new installs and upgrades](../administration/package_information/postgresql_versions.md). + +We do run our test suite against PG11 and PG13 on nightly scheduled pipelines. + +We also run our test suite against PG11 upon specific database library changes in MRs and `main` pipelines (with the `rspec db-library-code pg11` job). + +### Current versions testing + +| Where? | PostgreSQL version | +| ------ | ------------------ | +| MRs | 12, 11 for DB library changes | +| `main` (non-scheduled pipelines) | 12, 11 for DB library changes | +| 2-hourly scheduled pipelines | 12, 11 for DB library changes | +| `nightly` scheduled pipelines | 12, 11, 13 | + +### Long-term plan + +We follow the [PostgreSQL versions shipped with Omnibus GitLab](../administration/package_information/postgresql_versions.md): + +| PostgreSQL version | 14.1 (July 2021) | 14.2 (August 2021) | 14.3 (September 2021) | 14.4 (October 2021) | 14.5 (November 2021) | 14.6 (December 2021) | +| -------------------| ---------------------- | ---------------------- | ---------------------- | ---------------------- | ---------------------- | ---------------------- | +| PG12 | MRs/`2-hour`/`nightly` | MRs/`2-hour`/`nightly` | MRs/`2-hour`/`nightly` | MRs/`2-hour`/`nightly` | MRs/`2-hour`/`nightly` | MRs/`2-hour`/`nightly` | +| PG11 | `nightly` | `nightly` | `nightly` | `nightly` | `nightly` | `nightly` | +| PG13 | `nightly` | `nightly` | `nightly` | `nightly` | `nightly` | `nightly` | -### Pipelines for Merge Requests +## Pipelines types for merge requests In general, pipelines for an MR fall into one or more of the following types, depending on the changes made in the MR: @@ -53,7 +229,7 @@ We use the [`rules:`](../ci/yaml/index.md#rules) and [`needs:`](../ci/yaml/index to determine the jobs that need to be run in a pipeline. Note that an MR that includes multiple types of changes would have a pipelines that include jobs from multiple types (for example, a combination of docs-only and code-only pipelines). -#### Documentation only MR pipeline +### Documentation only MR pipeline [Reference pipeline](https://gitlab.com/gitlab-org/gitlab/-/pipelines/250546928): @@ -71,7 +247,7 @@ graph LR end ``` -#### Code-only MR pipeline +### Code-only MR pipeline [Reference pipeline](https://gitlab.com/gitlab-org/gitlab/pipelines/136295694) @@ -102,7 +278,6 @@ graph RL; 1-16["brakeman-sast"]; 1-17["eslint-sast"]; 1-18["kubesec-sast"]; - 1-19["nodejs-scan-sast"]; 1-20["secrets-sast"]; 1-21["static-analysis (14 minutes)"]; click 1-21 "https://app.periscopedata.com/app/gitlab/652085/Engineering-Productivity---Pipeline-Build-Durations?widget=6914471&udv=0" @@ -123,7 +298,7 @@ graph RL; 2_1-1 & 2_1-2 & 2_1-3 & 2_1-4 --> 1-6; end - 2_2-2["rspec frontend_fixture/rspec-ee frontend_fixture (7 minutes)"]; + 2_2-2["rspec-all frontend_fixture (7 minutes)"]; class 2_2-2 criticalPath; click 2_2-2 "https://app.periscopedata.com/app/gitlab/652085/Engineering-Productivity---Pipeline-Build-Durations?widget=7910143&udv=0" 2_2-4["memory-on-boot (3.5 minutes)"]; @@ -155,7 +330,7 @@ graph RL; 3_1-1["jest (14.5 minutes)"]; class 3_1-1 criticalPath; click 3_1-1 "https://app.periscopedata.com/app/gitlab/652085/Engineering-Productivity---Pipeline-Build-Durations?widget=6914204&udv=0" - subgraph "Needs `rspec frontend_fixture/rspec-ee frontend_fixture`"; + subgraph "Needs `rspec-all frontend_fixture`"; 3_1-1 --> 2_2-2; end @@ -173,7 +348,7 @@ graph RL; end ``` -#### Frontend-only MR pipeline +### Frontend-only MR pipeline [Reference pipeline](https://gitlab.com/gitlab-org/gitlab/pipelines/134661039): @@ -204,7 +379,6 @@ graph RL; 1-16["brakeman-sast"]; 1-17["eslint-sast"]; 1-18["kubesec-sast"]; - 1-19["nodejs-scan-sast"]; 1-20["secrets-sast"]; 1-21["static-analysis (14 minutes)"]; click 1-21 "https://app.periscopedata.com/app/gitlab/652085/Engineering-Productivity---Pipeline-Build-Durations?widget=6914471&udv=0" @@ -226,7 +400,7 @@ graph RL; 2_1-1 & 2_1-2 & 2_1-3 & 2_1-4 --> 1-6; end - 2_2-2["rspec frontend_fixture/rspec-ee frontend_fixture (7 minutes)"]; + 2_2-2["rspec-all frontend_fixture (7 minutes)"]; class 2_2-2 criticalPath; click 2_2-2 "https://app.periscopedata.com/app/gitlab/652085/Engineering-Productivity---Pipeline-Build-Durations?widget=7910143&udv=0" 2_2-4["memory-on-boot (3.5 minutes)"]; @@ -266,7 +440,7 @@ graph RL; 3_1-1["jest (14.5 minutes)"]; class 3_1-1 criticalPath; click 3_1-1 "https://app.periscopedata.com/app/gitlab/652085/Engineering-Productivity---Pipeline-Build-Durations?widget=6914204&udv=0" - subgraph "Needs `rspec frontend_fixture/rspec-ee frontend_fixture`"; + subgraph "Needs `rspec-all frontend_fixture`"; 3_1-1 --> 2_2-2; end @@ -299,7 +473,7 @@ graph RL; end ``` -#### QA-only MR pipeline +### QA-only MR pipeline [Reference pipeline](https://gitlab.com/gitlab-org/gitlab/pipelines/134645109): @@ -330,7 +504,6 @@ graph RL; 1-16["brakeman-sast"]; 1-17["eslint-sast"]; 1-18["kubesec-sast"]; - 1-19["nodejs-scan-sast"]; 1-20["secrets-sast"]; 1-21["static-analysis (14 minutes)"]; click 1-21 "https://app.periscopedata.com/app/gitlab/652085/Engineering-Productivity---Pipeline-Build-Durations?widget=6914471&udv=0" @@ -358,261 +531,53 @@ graph RL; end ``` -### Fail-fast pipeline in Merge Requests - -To provide faster feedback when a Merge Request breaks existing tests, we are experimenting with a -fail-fast mechanism. - -An `rspec fail-fast` job is added in parallel to all other `rspec` jobs in a Merge -Request pipeline. This job runs the tests that are directly related to the changes -in the Merge Request. - -If any of these tests fail, the `rspec fail-fast` job fails, triggering a -`fail-pipeline-early` job to run. The `fail-pipeline-early` job: - -- Cancels the currently running pipeline and all in-progress jobs. -- Sets pipeline to have status `failed`. - -For example: - -```mermaid -graph LR - subgraph "prepare stage"; - A["detect-tests"] - end - - subgraph "test stage"; - B["jest"]; - C["rspec migration"]; - D["rspec unit"]; - E["rspec integration"]; - F["rspec system"]; - G["rspec fail-fast"]; - end - - subgraph "post-test stage"; - Z["fail-pipeline-early"]; - end - - A --"artifact: list of test files"--> G - G --"on failure"--> Z -``` - -A Merge Request author may choose to opt-out of the fail fast mechanism by doing one of the following: - -- Adding the `pipeline:skip-rspec-fail-fast` label to the merge request -- Starting the `dont-interrupt-me` job found in the `sync` stage of a Merge Request pipeline. - -The `rspec fail-fast` is a no-op if there are more than 10 test files related to the -Merge Request. This prevents `rspec fail-fast` duration from exceeding the average -`rspec` job duration and defeating its purpose. - -This number can be overridden by setting a CI/CD variable named `RSPEC_FAIL_FAST_TEST_FILE_COUNT_THRESHOLD`. - -NOTE: -This experiment is only enabled when the CI/CD variable `RSPEC_FAIL_FAST_ENABLED=true` is set. - -#### Determining related test files in a Merge Request - -The test files related to the Merge Request are determined using the [`test_file_finder`](https://gitlab.com/gitlab-org/ci-cd/test_file_finder) gem. -We are using a custom mapping between source file to test files, maintained in the `tests.yml` file. - -### RSpec minimal jobs - -Before a merge request is approved, the pipeline will run a minimal set of RSpec tests that are related to the merge request changes. -This is to reduce the pipeline cost and shorten the job duration. - -To identify the minimal set of tests needed, we use [Crystalball gem](https://github.com/toptal/crystalball) to create a test mapping. -The test mapping contains a map of each source files to a list of test files which is dependent of the source file. -This mapping is currently generated using a combination of test coverage tracing and a static mapping. -In the `detect-tests` job, we use this mapping to identify the minimal tests needed for the current Merge Request. - -After a merge request has been approved, the pipeline would contain the full RSpec tests. This will ensure that all tests -have been run before a merge request is merged. - -### Jest minimal jobs - -Before a merge request is approved, the pipeline will run a minimal set of Jest tests that are related to the merge request changes. -This is to reduce the pipeline cost and shorten the job duration. - -To identify the minimal set of tests needed, we pass a list of all the changed files into `jest` using the [`--findRelatedTests`](https://jestjs.io/docs/cli#--findrelatedtests-spaceseparatedlistofsourcefiles) option. -In this mode, `jest` would resolve all the dependencies of related to the changed files, which include test files that have these files in the dependency chain. - -After a merge request has been approved, the pipeline would contain the full Jest tests. This will ensure that all tests -have been run before a merge request is merged. - -In addition, there are a few circumstances where we would always run the full Jest tests: - -- when `package.json`, `yarn.lock`, `jest` config changes -- when vendored JavaScript is changed -- when `.graphql` files are changed - -### PostgreSQL versions testing - -Our test suite runs against PG12 as GitLab.com runs on PG12 and -[Omnibus defaults to PG12 for new installs and upgrades](../administration/package_information/postgresql_versions.md), -Our test suite is currently running against PG11, since GitLab.com still runs on PG11. - -We do run our test suite against PG11 on nightly scheduled pipelines as well as upon specific -database library changes in MRs and `main` pipelines (with the `rspec db-library-code pg11` job). - -#### Current versions testing - -| Where? | PostgreSQL version | -| ------ | ------------------ | -| MRs | 12, 11 for DB library changes | -| `main` (non-scheduled pipelines) | 12, 11 for DB library changes | -| 2-hourly scheduled pipelines | 12, 11 for DB library changes | -| `nightly` scheduled pipelines | 12, 11 | - -#### Long-term plan - -We follow the [PostgreSQL versions shipped with Omnibus GitLab](../administration/package_information/postgresql_versions.md): - -| PostgreSQL version | 13.11 (April 2021) | 13.12 (May 2021) | 14.0 (June 2021?) | -| -------------------| ---------------------- | ---------------------- | ---------------------- | -| PG12 | `nightly` | MRs/`2-hour`/`nightly` | MRs/`2-hour`/`nightly` | -| PG11 | MRs/`2-hour`/`nightly` | `nightly` | `nightly` | - -### Test jobs - -Consult [GitLab tests in the Continuous Integration (CI) context](testing_guide/ci.md) -for more information. - -We have dedicated jobs for each [testing level](testing_guide/testing_levels.md) and each job runs depending on the -changes made in your merge request. -If you want to force all the RSpec jobs to run regardless of your changes, you can add the `pipeline:run-all-rspec` label to the merge request. - -> Forcing all jobs on docs only related MRs would not have the prerequisite jobs and would lead to errors - -### Review app jobs - -Consult the [Review Apps](testing_guide/review_apps.md) dedicated page for more information. - -### As-if-FOSS jobs - -The `* as-if-foss` jobs allows the GitLab test suite "as-if-FOSS", meaning as if the jobs would run in the context -of the `gitlab-org/gitlab-foss` project. These jobs are only created in the following cases: - -- `gitlab-org/security/gitlab` merge requests. -- Merge requests with the `pipeline:run-as-if-foss` label -- Merge requests that changes the CI configuration. - -The `* as-if-foss` jobs are run in addition to the regular EE-context jobs. They have the `FOSS_ONLY='1'` variable -set and get their EE-specific folders removed before the tests start running. - -The intent is to ensure that a change doesn't introduce a failure after the `gitlab-org/gitlab` project is synced to -the `gitlab-org/gitlab-foss` project. - -## Performance - -### Interruptible pipelines - -By default, all jobs are [interruptible](../ci/yaml/index.md#interruptible), except the -`dont-interrupt-me` job which runs automatically on `main`, and is `manual` -otherwise. - -If you want a running pipeline to finish even if you push new commits to a merge -request, be sure to start the `dont-interrupt-me` job before pushing. - -### Caching strategy - -1. All jobs must only pull caches by default. -1. All jobs must be able to pass with an empty cache. In other words, caches are only there to speed up jobs. -1. We currently have several different cache definitions defined in - [`.gitlab/ci/global.gitlab-ci.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/.gitlab/ci/global.gitlab-ci.yml), - with fixed keys: - - `.setup-test-env-cache` - - `.rails-cache` - - `.static-analysis-cache` - - `.coverage-cache` - - `.danger-review-cache` - - `.qa-cache` - - `.yarn-cache` - - `.assets-compile-cache` (the key includes `${NODE_ENV}` so it's actually two different caches). -1. These cache definitions are composed of [multiple atomic caches](../ci/caching/index.md#use-multiple-caches). -1. Only the following jobs, running in 2-hourly scheduled pipelines, are pushing (that is, updating) to the caches: - - `update-setup-test-env-cache`, defined in [`.gitlab/ci/rails.gitlab-ci.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/.gitlab/ci/rails.gitlab-ci.yml). - - `update-gitaly-binaries-cache`, defined in [`.gitlab/ci/rails.gitlab-ci.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/.gitlab/ci/rails.gitlab-ci.yml). - - `update-static-analysis-cache`, defined in [`.gitlab/ci/rails.gitlab-ci.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/.gitlab/ci/rails.gitlab-ci.yml). - - `update-qa-cache`, defined in [`.gitlab/ci/qa.gitlab-ci.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/.gitlab/ci/qa.gitlab-ci.yml). - - `update-assets-compile-production-cache`, defined in [`.gitlab/ci/frontend.gitlab-ci.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/.gitlab/ci/frontend.gitlab-ci.yml). - - `update-assets-compile-test-cache`, defined in [`.gitlab/ci/frontend.gitlab-ci.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/.gitlab/ci/frontend.gitlab-ci.yml). - - `update-yarn-cache`, defined in [`.gitlab/ci/frontend.gitlab-ci.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/.gitlab/ci/frontend.gitlab-ci.yml). - - `update-storybook-yarn-cache`, defined in [`.gitlab/ci/frontend.gitlab-ci.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/.gitlab/ci/frontend.gitlab-ci.yml). -1. These jobs can also be forced to run in merge requests with the `pipeline:update-cache` label (this can be useful to warm the caches in a MR that updates the cache keys). - -### Artifacts strategy +## CI configuration internals -We limit the artifacts that are saved and retrieved by jobs to the minimum in order to reduce the upload/download time and costs, as well as the artifacts storage. +### Workflow rules -### Pre-clone step +Pipelines for the GitLab project are created using the [`workflow:rules` keyword](../ci/yaml/index.md#workflow) +feature of the GitLab CI/CD. -The `gitlab-org/gitlab` project on GitLab.com uses a [pre-clone step](https://gitlab.com/gitlab-org/gitlab/-/issues/39134) -to seed the project with a recent archive of the repository. This is done for -several reasons: +Pipelines are always created for the following scenarios: -- It speeds up builds because a 800 MB download only takes seconds, as opposed to a full Git clone. -- It significantly reduces load on the file server, as smaller deltas mean less time spent in `git pack-objects`. +- `main` branch, including on schedules, pushes, merges, and so on. +- Merge requests. +- Tags. +- Stable, `auto-deploy`, and security branches. -The pre-clone step works by using the `CI_PRE_CLONE_SCRIPT` variable -[defined by GitLab.com shared runners](../ci/runners/build_cloud/linux_build_cloud.md#pre-clone-script). +Pipeline creation is also affected by the following CI/CD variables: -The `CI_PRE_CLONE_SCRIPT` is currently defined as a project CI/CD variable: +- If `$FORCE_GITLAB_CI` is set, pipelines are created. +- If `$GITLAB_INTERNAL` is not set, pipelines are not created. -```shell -( - echo "Downloading archived master..." - wget -O /tmp/gitlab.tar.gz https://storage.googleapis.com/gitlab-ci-git-repo-cache/project-278964/gitlab-master-shallow.tar.gz +No pipeline is created in any other cases (for example, when pushing a branch with no +MR for it). - if [ ! -f /tmp/gitlab.tar.gz ]; then - echo "Repository cache not available, cloning a new directory..." - exit - fi +The source of truth for these workflow rules is defined in [`.gitlab-ci.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/.gitlab-ci.yml). - rm -rf $CI_PROJECT_DIR - echo "Extracting tarball into $CI_PROJECT_DIR..." - mkdir -p $CI_PROJECT_DIR - cd $CI_PROJECT_DIR - tar xzf /tmp/gitlab.tar.gz - rm -f /tmp/gitlab.tar.gz - chmod a+w $CI_PROJECT_DIR -) -``` +### Default image -The first step of the script downloads `gitlab-master.tar.gz` from -Google Cloud Storage. There is a [GitLab CI job named `cache-repo`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/.gitlab/ci/cache-repo.gitlab-ci.yml#L5) -that is responsible for keeping that archive up-to-date. Every two hours -on a scheduled pipeline, it does the following: +The default image is defined in [`.gitlab-ci.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/.gitlab-ci.yml). -1. Creates a fresh clone of the `gitlab-org/gitlab` repository on GitLab.com. -1. Saves the data as a `.tar.gz`. -1. Uploads it into the Google Cloud Storage bucket. +<!-- vale gitlab.Spelling = NO --> -When a CI job runs with this configuration, the output looks something like this: +It includes Ruby, Go, Git, Git LFS, Chrome, Node, Yarn, PostgreSQL, and Graphics Magick. -```shell -$ eval "$CI_PRE_CLONE_SCRIPT" -Downloading archived master... -Extracting tarball into /builds/group/project... -Fetching changes... -Reinitialized existing Git repository in /builds/group/project/.git/ -``` +<!-- vale gitlab.Spelling = YES --> -Note that the `Reinitialized existing Git repository` message shows that -the pre-clone step worked. The runner runs `git init`, which -overwrites the Git configuration with the appropriate settings to fetch -from the GitLab repository. +The images used in our pipelines are configured in the +[`gitlab-org/gitlab-build-images`](https://gitlab.com/gitlab-org/gitlab-build-images) +project, which is push-mirrored to [`gitlab/gitlab-build-images`](https://dev.gitlab.org/gitlab/gitlab-build-images) +for redundancy. -`CI_REPO_CACHE_CREDENTIALS` contains the Google Cloud service account -JSON for uploading to the `gitlab-ci-git-repo-cache` bucket. (If you're a -GitLab Team Member, find credentials in the -[GitLab shared 1Password account](https://about.gitlab.com/handbook/security/#1password-for-teams). +The current version of the build images can be found in the +["Used by GitLab section"](https://gitlab.com/gitlab-org/gitlab-build-images/blob/master/.gitlab-ci.yml). -Note that this bucket should be located in the same continent as the -runner, or [you can incur network egress charges](https://cloud.google.com/storage/pricing). +### Default variables -## CI configuration internals +In addition to the [predefined CI/CD variables](../ci/variables/predefined_variables.md), +each pipeline includes default variables defined in +[`.gitlab-ci.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/.gitlab-ci.yml). ### Stages @@ -644,24 +609,6 @@ that is deployed in stage `review`. [an issue with the deployment](https://gitlab.com/gitlab-org/gitlab/-/issues/233458)). - `notify`: This stage includes jobs that notify various failures to Slack. -### Default image - -The default image is defined in [`.gitlab-ci.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/.gitlab-ci.yml). - -<!-- vale gitlab.Spelling = NO --> - -It includes Ruby, Go, Git, Git LFS, Chrome, Node, Yarn, PostgreSQL, and Graphics Magick. - -<!-- vale gitlab.Spelling = YES --> - -The images used in our pipelines are configured in the -[`gitlab-org/gitlab-build-images`](https://gitlab.com/gitlab-org/gitlab-build-images) -project, which is push-mirrored to [`gitlab/gitlab-build-images`](https://dev.gitlab.org/gitlab/gitlab-build-images) -for redundancy. - -The current version of the build images can be found in the -["Used by GitLab section"](https://gitlab.com/gitlab-org/gitlab-build-images/blob/master/.gitlab-ci.yml). - ### Dependency Proxy Some of the jobs are using images from Docker Hub, where we also use @@ -681,12 +628,6 @@ Projects in the `gitlab-org` group pull from the Dependency Proxy, while forks that reside on any other personal namespaces or groups fall back to Docker Hub unless `${GITLAB_DEPENDENCY_PROXY}` is also defined there. -### Default variables - -In addition to the [predefined CI/CD variables](../ci/variables/predefined_variables.md), -each pipeline includes default variables defined in -[`.gitlab-ci.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/.gitlab-ci.yml). - ### Common job definitions Most of the jobs [extend from a few CI definitions](../ci/yaml/index.md#extends) @@ -756,8 +697,6 @@ and included in `rules` definitions via [YAML anchors](../ci/yaml/index.md#ancho | `if-dot-com-gitlab-org-and-security-tag` | Limit jobs creation to tags for the `gitlab-org` and `gitlab-org/security` groups on GitLab.com. | | | `if-dot-com-ee-schedule` | Limits jobs to scheduled pipelines for the `gitlab-org/gitlab` project on GitLab.com. | | | `if-cache-credentials-schedule` | Limits jobs to scheduled pipelines with the `$CI_REPO_CACHE_CREDENTIALS` variable set. | | -| `if-rspec-fail-fast-disabled` | Limits jobs to pipelines with `$RSPEC_FAIL_FAST_ENABLED` CI/CD variable not set to `"true"`. | | -| `if-rspec-fail-fast-skipped` | Matches if the pipeline is for a merge request and the MR has label ~"pipeline:skip-rspec-fail-fast". | | | `if-security-pipeline-merge-result` | Matches if the pipeline is for a security merge request triggered by `@gitlab-release-tools-bot`. | | <!-- vale gitlab.Substitutions = YES --> @@ -783,6 +722,114 @@ and included in `rules` definitions via [YAML anchors](../ci/yaml/index.md#ancho | `code-qa-patterns` | Combination of `code-patterns` and `qa-patterns`. | | `code-backstage-qa-patterns` | Combination of `code-patterns`, `backstage-patterns`, and `qa-patterns`. | +## Performance + +### Interruptible pipelines + +By default, all jobs are [interruptible](../ci/yaml/index.md#interruptible), except the +`dont-interrupt-me` job which runs automatically on `main`, and is `manual` +otherwise. + +If you want a running pipeline to finish even if you push new commits to a merge +request, be sure to start the `dont-interrupt-me` job before pushing. + +### Caching strategy + +1. All jobs must only pull caches by default. +1. All jobs must be able to pass with an empty cache. In other words, caches are only there to speed up jobs. +1. We currently have several different cache definitions defined in + [`.gitlab/ci/global.gitlab-ci.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/.gitlab/ci/global.gitlab-ci.yml), + with fixed keys: + - `.setup-test-env-cache` + - `.rails-cache` + - `.static-analysis-cache` + - `.coverage-cache` + - `.danger-review-cache` + - `.qa-cache` + - `.yarn-cache` + - `.assets-compile-cache` (the key includes `${NODE_ENV}` so it's actually two different caches). +1. These cache definitions are composed of [multiple atomic caches](../ci/caching/index.md#use-multiple-caches). +1. Only the following jobs, running in 2-hourly scheduled pipelines, are pushing (that is, updating) to the caches: + - `update-setup-test-env-cache`, defined in [`.gitlab/ci/rails.gitlab-ci.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/.gitlab/ci/rails.gitlab-ci.yml). + - `update-gitaly-binaries-cache`, defined in [`.gitlab/ci/rails.gitlab-ci.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/.gitlab/ci/rails.gitlab-ci.yml). + - `update-static-analysis-cache`, defined in [`.gitlab/ci/rails.gitlab-ci.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/.gitlab/ci/rails.gitlab-ci.yml). + - `update-qa-cache`, defined in [`.gitlab/ci/qa.gitlab-ci.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/.gitlab/ci/qa.gitlab-ci.yml). + - `update-assets-compile-production-cache`, defined in [`.gitlab/ci/frontend.gitlab-ci.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/.gitlab/ci/frontend.gitlab-ci.yml). + - `update-assets-compile-test-cache`, defined in [`.gitlab/ci/frontend.gitlab-ci.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/.gitlab/ci/frontend.gitlab-ci.yml). + - `update-yarn-cache`, defined in [`.gitlab/ci/frontend.gitlab-ci.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/.gitlab/ci/frontend.gitlab-ci.yml). + - `update-storybook-yarn-cache`, defined in [`.gitlab/ci/frontend.gitlab-ci.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/.gitlab/ci/frontend.gitlab-ci.yml). +1. These jobs can also be forced to run in merge requests with the `pipeline:update-cache` label (this can be useful to warm the caches in a MR that updates the cache keys). + +### Artifacts strategy + +We limit the artifacts that are saved and retrieved by jobs to the minimum in order to reduce the upload/download time and costs, as well as the artifacts storage. + +### Pre-clone step + +The `gitlab-org/gitlab` project on GitLab.com uses a [pre-clone step](https://gitlab.com/gitlab-org/gitlab/-/issues/39134) +to seed the project with a recent archive of the repository. This is done for +several reasons: + +- It speeds up builds because a 800 MB download only takes seconds, as opposed to a full Git clone. +- It significantly reduces load on the file server, as smaller deltas mean less time spent in `git pack-objects`. + +The pre-clone step works by using the `CI_PRE_CLONE_SCRIPT` variable +[defined by GitLab.com shared runners](../ci/runners/build_cloud/linux_build_cloud.md#pre-clone-script). + +The `CI_PRE_CLONE_SCRIPT` is currently defined as a project CI/CD variable: + +```shell +( + echo "Downloading archived master..." + wget -O /tmp/gitlab.tar.gz https://storage.googleapis.com/gitlab-ci-git-repo-cache/project-278964/gitlab-master-shallow.tar.gz + + if [ ! -f /tmp/gitlab.tar.gz ]; then + echo "Repository cache not available, cloning a new directory..." + exit + fi + + rm -rf $CI_PROJECT_DIR + echo "Extracting tarball into $CI_PROJECT_DIR..." + mkdir -p $CI_PROJECT_DIR + cd $CI_PROJECT_DIR + tar xzf /tmp/gitlab.tar.gz + rm -f /tmp/gitlab.tar.gz + chmod a+w $CI_PROJECT_DIR +) +``` + +The first step of the script downloads `gitlab-master.tar.gz` from +Google Cloud Storage. There is a [GitLab CI job named `cache-repo`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/.gitlab/ci/cache-repo.gitlab-ci.yml#L5) +that is responsible for keeping that archive up-to-date. Every two hours +on a scheduled pipeline, it does the following: + +1. Creates a fresh clone of the `gitlab-org/gitlab` repository on GitLab.com. +1. Saves the data as a `.tar.gz`. +1. Uploads it into the Google Cloud Storage bucket. + +When a CI job runs with this configuration, the output looks something like this: + +```shell +$ eval "$CI_PRE_CLONE_SCRIPT" +Downloading archived master... +Extracting tarball into /builds/group/project... +Fetching changes... +Reinitialized existing Git repository in /builds/group/project/.git/ +``` + +Note that the `Reinitialized existing Git repository` message shows that +the pre-clone step worked. The runner runs `git init`, which +overwrites the Git configuration with the appropriate settings to fetch +from the GitLab repository. + +`CI_REPO_CACHE_CREDENTIALS` contains the Google Cloud service account +JSON for uploading to the `gitlab-ci-git-repo-cache` bucket. (If you're a +GitLab Team Member, find credentials in the +[GitLab shared 1Password account](https://about.gitlab.com/handbook/security/#1password-for-teams). + +Note that this bucket should be located in the same continent as the +runner, or [you can incur network egress charges](https://cloud.google.com/storage/pricing). + --- [Return to Development documentation](index.md) |