diff options
Diffstat (limited to 'doc/development/pipelines')
| -rw-r--r-- | doc/development/pipelines/index.md | 122 | ||||
| -rw-r--r-- | doc/development/pipelines/internals.md | 25 | ||||
| -rw-r--r-- | doc/development/pipelines/performance.md | 39 |
3 files changed, 136 insertions, 50 deletions
diff --git a/doc/development/pipelines/index.md b/doc/development/pipelines/index.md index c4dfda9466a..91f4ae702ac 100644 --- a/doc/development/pipelines/index.md +++ b/doc/development/pipelines/index.md @@ -13,7 +13,7 @@ which itself includes files under for easier maintenance. We're striving to [dogfood](https://about.gitlab.com/handbook/engineering/development/principles/#dogfooding) -GitLab [CI/CD features and best-practices](../../ci/yaml/index.md) +GitLab [CI/CD features and best-practices](../../ci/index.md) as much as possible. ## Predictive test jobs before a merge request is approved @@ -268,6 +268,9 @@ the specific list of rules. If you want to force `e2e:package-and-test` to run regardless of your changes, you can add the `pipeline:run-all-e2e` label to the merge request. +The [`e2e:test-on-gdk`](../testing_guide/end_to_end/index.md#using-the-test-on-gdk-job) child pipeline runs `:reliable` +E2E specs automatically for all `code patterns changes`. See `.qa:rules:e2e-blocking` [`rules.gitlab-ci.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/.gitlab/ci/rules.gitlab-ci.yml) for specific set of rules. + Consult the [End-to-end Testing](../testing_guide/end_to_end/index.md) dedicated page for more information. ### Review app jobs @@ -297,6 +300,23 @@ set and get the `ee/` folder removed before the tests start running. The intent is to ensure that a change doesn't introduce a failure after `gitlab-org/gitlab` is synced to `gitlab-org/gitlab-foss`. +#### As-if-FOSS cross project downstream pipeline + +As an alternative to the `* as-if-foss` jobs, we can also run a cross project +FOSS pipeline exactly in the `gitlab-org/gitlab-foss` project. We trigger it +in the following cases: + +- when the `pipeline:run-as-if-foss-cross-project` label is set on the merge request + +This is still working-in-progress to replace the `* as-if-foss` jobs. The +goal is to simplify pipeline rules and make it more clear about the intention. + +##### Tokens set in the project variables + +- `AS_IF_FOSS_TOKEN`: This is a [GitLab FOSS](https://gitlab.com/gitlab-org/gitlab-foss) + project token with `developer` role and `write_repository` permission, + to push generated `as-if-foss/*` branch. + ### As-if-JH cross project downstream pipeline #### What it is @@ -396,7 +416,8 @@ flowchart TD - `ADD_JH_FILES_TOKEN`: This is a [GitLab JH mirror](https://gitlab.com/gitlab-org/gitlab-jh-mirrors/gitlab) project token with `read_api` permission, to be able to download JiHu files. - `AS_IF_JH_TOKEN`: This is a [GitLab JH validation](https://gitlab.com/gitlab-org-sandbox/gitlab-jh-validation) - project token with `write_repository` permission, to push generated `as-if-jh/*` branch. + project token with `developer` role and `write_repository` permission, + to push generated `as-if-jh/*` branch. ##### How we generate the as-if-JH branch @@ -610,30 +631,30 @@ Exceptions to this general guideline should be motivated and documented. ### Ruby versions testing -We're running Ruby 3.0 on GitLab.com, as well as for the default branch. -To prepare for the next Ruby version, we run merge requests in Ruby 3.1. +We're running Ruby 3.1 on GitLab.com, as well as for the default branch. +To prepare for the next Ruby version, we will run merge requests in Ruby 3.2, +starting on February 2024. Please see the roadmap at +[Ruby 3.2 epic](https://gitlab.com/groups/gitlab-org/-/epics/9684#plan) +for more details. -This takes effects at the time when -[Run merge requests in Ruby 3.1 by default](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/134290) -is merged. See -[Ruby 3.1 epic](https://gitlab.com/groups/gitlab-org/-/epics/10034) -for the roadmap to fully make Ruby 3.1 the default. +To make sure all supported Ruby versions are working, we also run our test +suite on dedicated 2-hourly scheduled pipelines for each supported versions. -To make sure both Ruby versions are working, we also run our test suite -against both Ruby 3.0 and Ruby 3.1 on dedicated 2-hourly scheduled pipelines. +For merge requests, you can add the following labels to run the respective +Ruby version only: -For merge requests, you can add the `pipeline:run-in-ruby3_0` label to switch -the Ruby version to 3.0. When you do this, the test suite will no longer run -in Ruby 3.1 (default for merge requests). +- `pipeline:run-in-ruby3_0` +- `pipeline:run-in-ruby3_1` +- `pipeline:run-in-ruby3_2` -When the pipeline is running in a Ruby version not considered default, an -additional job `verify-default-ruby` will also run and always fail to remind -us to remove the label and run in default Ruby before merging the merge -request. At the moment both Ruby 3.0 and Ruby 3.1 are considered default. +Note that when you do this, the test suite will no longer run in the default +Ruby version for merge requests. In this case, an additional job +`verify-default-ruby` will also run and always fail to remind us to remove +the label and run in default Ruby before merging the merge request. This should let us: -- Test changes for Ruby 3.1 +- Test changes for any supported Ruby versions - Make sure it will not break anything when it's merged into the default branch ### PostgreSQL versions testing @@ -649,24 +670,27 @@ We also run our test suite against PostgreSQL 13 upon specific database library | Where? | PostgreSQL version | Ruby version | |--------------------------------------------------------------------------------------------------|-------------------------------------------------|-----------------------| -| Merge requests | 14 (default version), 13 for DB library changes | 3.1 | -| `master` branch commits | 14 (default version), 13 for DB library changes | 3.0 (default version) | -| `maintenance` scheduled pipelines for the `master` branch (every even-numbered hour) | 14 (default version), 13 for DB library changes | 3.0 (default version) | -| `maintenance` scheduled pipelines for the `ruby3_1` branch (every odd-numbered hour), see below. | 14 (default version), 13 for DB library changes | 3.1 | -| `nightly` scheduled pipelines for the `master` branch | 14 (default version), 13, 15 | 3.0 (default version) | - -There are 2 pipeline schedules used for testing Ruby 3.1. One is triggering a -pipeline in `ruby3_1-sync` branch, which updates the `ruby3_1` branch with latest -`master`, and no pipelines will be triggered by this push. The other schedule -is triggering a pipeline in `ruby3_1` 5 minutes after it, which is considered -the maintenance schedule to run test suites and update cache. - -The `ruby3_1` branch must not have any changes. The branch is only there to set -`RUBY_VERSION` to `3.1` in the maintenance pipeline schedule. - -The `gitlab` job in the `ruby3_1-sync` branch uses a `gitlab-org/gitlab` project -token with `write_repository` scope and `Maintainer` role with no expiration. -The token is stored in the `RUBY3_1_SYNC_TOKEN` variable in `gitlab-org/gitlab`. +| Merge requests | 14 (default version), 13 for DB library changes | 3.1 (default version) | +| `master` branch commits | 14 (default version), 13 for DB library changes | 3.1 (default version) | +| `maintenance` scheduled pipelines for the `master` branch (every even-numbered hour at XX:05) | 14 (default version), 13 for DB library changes | 3.1 (default version) | +| `maintenance` scheduled pipelines for the `ruby3_0` branch (every odd-numbered hour at XX:40) | 14 (default version), 13 for DB library changes | 3.0 | +| `maintenance` scheduled pipelines for the `ruby3_2` branch (every odd-numbered hour at XX:10) | 14 (default version), 13 for DB library changes | 3.2 | +| `nightly` scheduled pipelines for the `master` branch | 14 (default version), 13, 15 | 3.1 (default version) | + +For each current Ruby versions we're testing against with, we run +maintenance scheduled pipelines every 2 hours on their respective `ruby\d_\d` +branches. All these branches must not have any changes. These branches are +only there to run pipelines with their respective Ruby versions in the +scheduled maintenance pipelines. + +Additionally, we have scheduled pipelines running on `ruby-sync` branch also +every 2 hours, updating all the `ruby\d_\d` branches to be up-to-date with +the default branch `master`. No pipelines will be triggered by this push. + +The `gitlab` job in the `ruby-sync` branch uses a `gitlab-org/gitlab` project +token named `RUBY_SYNC` with `write_repository` scope and `Maintainer` role, +expiring on 2024-12-01. The token is stored in the `RUBY_SYNC_TOKEN` variable +in the pipeline schedule for `ruby-sync` branch. ### Redis versions testing @@ -692,9 +716,9 @@ We also run tests with a single database in nightly scheduled pipelines, and in Single database tests run in two modes: 1. **Single database with one connection**. Where GitLab connects to all the tables using one connection pool. -This runs through all the jobs that end with `-single-db` + This runs through all the jobs that end with `-single-db` 1. **Single database with two connections**. Where GitLab connects to `gitlab_main`, `gitlab_ci` database tables -using different database connections. This runs through all the jobs that end with `-single-db-ci-connection`. + using different database connections. This runs through all the jobs that end with `-single-db-ci-connection`. If you want to force tests to run with a single database, you can add the `pipeline:run-single-db` label to the merge request. @@ -747,28 +771,29 @@ graph LR ### Backend pipeline -[Reference pipeline](https://gitlab.com/gitlab-org/gitlab/-/pipelines/433316063). +[Reference pipeline](https://gitlab.com/gitlab-org/gitlab/-/pipelines/1118782302). ```mermaid graph RL; classDef criticalPath fill:#f66; - 1-3["compile-test-assets (5.5 minutes)"]; - class 1-3 criticalPath; + 1-1["clone-gitlab-repo (1 minute)"]; + 1-3["compile-test-assets (3 minutes)"]; click 1-3 "https://app.periscopedata.com/app/gitlab/652085/Engineering-Productivity---Pipeline-Build-Durations?widget=6914317&udv=0" - 1-6["setup-test-env (3.6 minutes)"]; + 1-6["setup-test-env (4 minutes)"]; + class 1-6 criticalPath; click 1-6 "https://app.periscopedata.com/app/gitlab/652085/Engineering-Productivity---Pipeline-Build-Durations?widget=6914315&udv=0" - 1-14["retrieve-tests-metadata"]; + 1-14["retrieve-tests-metadata (50 seconds)"]; click 1-14 "https://app.periscopedata.com/app/gitlab/652085/Engineering-Productivity---Pipeline-Build-Durations?widget=8356697&udv=0" - 1-15["detect-tests"]; + 1-15["detect-tests (1 minute)"]; click 1-15 "https://app.periscopedata.com/app/gitlab/652085/EP---Jobs-Durations?widget=10113603&udv=1005715" - 2_5-1["rspec & db jobs (24 minutes)"]; + 2_5-1["rspec & db jobs (30~50 minutes)"]; class 2_5-1 criticalPath; click 2_5-1 "https://app.periscopedata.com/app/gitlab/652085/Engineering-Productivity---Pipeline-Build-Durations" - 2_5-1 --> 1-3 & 1-6 & 1-14 & 1-15; + 2_5-1 --> 1-1 & 1-3 & 1-6 & 1-14 & 1-15; - ac-1["rspec:artifact-collector (2 minutes)<br/>(workaround for 'needs' limitation)"]; + ac-1["rspec:artifact-collector (30 seconds)<br/>(workaround for 'needs' limitation)"]; class ac-1 criticalPath; ac-1 --> 2_5-1; @@ -781,7 +806,6 @@ graph RL; class 4_3-1 criticalPath; click 4_3-1 "https://app.periscopedata.com/app/gitlab/652085/EP---Jobs-Durations?widget=13446492&udv=1005715" 4_3-1 --> 3_2-1; - ``` ### Review app pipeline diff --git a/doc/development/pipelines/internals.md b/doc/development/pipelines/internals.md index 1fdb2014c1f..16d0bfdfa30 100644 --- a/doc/development/pipelines/internals.md +++ b/doc/development/pipelines/internals.md @@ -48,6 +48,25 @@ from using `$FORCE_GITLAB_CI`. - [JiHu validation pipeline](https://about.gitlab.com/handbook/ceo/chief-of-staff-team/jihu-support/jihu-validation-pipelines.html) - [Gitaly downstream GitLab pipeline](https://gitlab.com/gitlab-org/gitaly/-/issues/4615) +See the next section for how we can enable pipelines without using +`$FORCE_GITLAB_CI`. + +#### Alternative to `$FORCE_GITLAB_CI` + +Essentially, we use different variables to enable different pipelines. +An example doing this is `$START_AS_IF_FOSS`. When we want to trigger a +cross project FOSS pipeline, we set `$START_AS_IF_FOSS`, along with a set of +other variables like `$ENABLE_RSPEC_UNIT`, `$ENABLE_RSPEC_SYSTEM`, and so on +so forth to enable each jobs we want to run in the as-if-foss cross project +downstream pipeline. + +The advantage of this over `$FORCE_GITLAB_CI` is that we have full control +over how we want to run the pipeline because `$START_AS_IF_FOSS` is only used +for this purpose, and changing how the pipeline behaves under this variable +will not affect other types of pipelines, while using `$FORCE_GITLAB_CI` we +do not know what exactly the pipeline is because it's used for multiple +purposes. + ## Default image The default image is defined in [`.gitlab-ci.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/.gitlab-ci.yml). @@ -149,6 +168,7 @@ that are scoped to a single [configuration keyword](../../ci/yaml/index.md#job-k |------------------|-------------| | `.default-retry` | Allows a job to [retry](../../ci/yaml/index.md#retry) upon `unknown_failure`, `api_failure`, `runner_system_failure`, `job_execution_timeout`, or `stuck_or_timeout_failure`. | | `.default-before_script` | Allows a job to use a default `before_script` definition suitable for Ruby/Rails tasks that may need a database running (for example, tests). | +| `.repo-from-artifacts` | Allows a job to fetch the repository from artifacts in `clone-gitlab-repo` instead of cloning. This should reduce GitLab.com Gitaly load and also slightly improve the speed because downloading from artifacts is faster than cloning. Note that this should be avoided to be used with jobs having `needs: []` because otherwise it'll start later and we normally want all jobs to start as soon as possible. Use this only on jobs which has other dependencies so that we don't wait longer than just cloning. Note that this behavior can be controlled via `CI_FETCH_REPO_GIT_STRATEGY`. See [Fetch repository via artifacts instead of cloning/fetching from Gitaly](performance.md#fetch-repository-via-artifacts-instead-of-cloningfetching-from-gitaly) for more details. | | `.setup-test-env-cache` | Allows a job to use a default `cache` definition suitable for setting up test environment for subsequent Ruby/Rails tasks. | | `.ruby-cache` | Allows a job to use a default `cache` definition suitable for Ruby tasks. | | `.static-analysis-cache` | Allows a job to use a default `cache` definition suitable for static analysis tasks. | @@ -199,6 +219,7 @@ and included in `rules` definitions via [YAML anchors](../../ci/yaml/yaml_optimi | `if-merge-request-title-as-if-foss` | Matches if the pipeline is for a merge request and the MR has label ~"pipeline:run-as-if-foss" | | | `if-merge-request-title-update-caches` | Matches if the pipeline is for a merge request and the MR has label ~"pipeline:update-cache". | | | `if-merge-request-labels-run-all-rspec` | Matches if the pipeline is for a merge request and the MR has label ~"pipeline:run-all-rspec". | | +| `if-merge-request-labels-run-cs-evaluation` | Matches if the pipeline is for a merge request and the MR has label ~"pipeline:run-CS-evaluation". | | | `if-security-merge-request` | Matches if the pipeline is for a security merge request. | | | `if-security-schedule` | Matches if the pipeline is for a security scheduled pipeline. | | | `if-nightly-master-schedule` | Matches if the pipeline is for a `master` scheduled pipeline with `$NIGHTLY` set. | | @@ -414,6 +435,8 @@ For this scenario, you have to: - `scripts/merge-simplecov` - `spec/simplecov_env_core.rb` - `spec/simplecov_env.rb` + - `prepare-as-if-foss-env` for: + - `scripts/setup/generate-as-if-foss-env.rb` Additionally, `scripts/utils.sh` is always downloaded from the API when this pattern is used (this file contains the code for `.fast-no-clone-job`). @@ -425,7 +448,7 @@ projects, only one of the following tags should be added to a job: - `gitlab-org`: Jobs randomly use privileged and unprivileged runners. - `gitlab-org-docker`: Jobs must use a privileged runner. If you need [Docker-in-Docker support](../../ci/docker/using_docker_build.md#use-docker-in-docker), -use `gitlab-org-docker` instead of `gitlab-org`. + use `gitlab-org-docker` instead of `gitlab-org`. The `gitlab-org-docker` tag is added by the `.use-docker-in-docker` job definition above. diff --git a/doc/development/pipelines/performance.md b/doc/development/pipelines/performance.md index 0cd4f4270fe..d9019e6053b 100644 --- a/doc/development/pipelines/performance.md +++ b/doc/development/pipelines/performance.md @@ -29,6 +29,45 @@ This works well for the following reasons: - We use [shallow clone](../../ci/pipelines/settings.md#limit-the-number-of-changes-fetched-during-clone) to avoid downloading the full Git history for every job. +### Fetch repository via artifacts instead of cloning/fetching from Gitaly + +Lately we see errors from Gitaly look like this: (see [the issue](https://gitlab.com/gitlab-org/gitlab/-/issues/435456)) + +```plaintext +fatal: remote error: GitLab is currently unable to handle this request due to load. +``` + +While GitLab.com uses [pack-objects cache](../../administration/gitaly/configure_gitaly.md#pack-objects-cache), +sometimes the load is still too heavy for Gitaly to handle, and +[thundering herds](https://gitlab.com/gitlab-org/gitlab/-/issues/423830) can +also be a concern that we have a lot of jobs cloning the repository around +the same time. + +To mitigate and reduce loads for Gitaly, we changed some jobs to fetch the +repository from artifacts in a job instead of all cloning from Gitaly at once. + +For now this applies to most of the RSpec jobs, which has the most concurrent +jobs in most pipelines. This also slightly improved the speed because fetching +from the artifacts is also slightly faster than cloning, at the cost of saving +more artifacts for each pipeline. + +Based on the numbers on 2023-12-20 at [Fetch repo from artifacts for RSpec jobs](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/140330), +the extra storage cost was about 280M for each pipeline, and we save 15 seconds +for each RSpec jobs. + +We do not apply this to jobs having no other job dependencies because we don't +want to delay any jobs from starting. + +This behavior can be controlled by variable `CI_FETCH_REPO_GIT_STRATEGY`: + +- Set to `none` means jobs using `.repo-from-artifacts` fetch repository from + artifacts in job `clone-gitlab-repo` rather than cloning. +- Set to `clone` means jobs using `.repo-from-artifacts` clone repository + as usual. Job `clone-gitlab-repo` does not run in this case. + +To disable it, set `CI_FETCH_REPO_GIT_STRATEGY` to `clone`. To enable it, +set `CI_FETCH_REPO_GIT_STRATEGY` to `none`. + ## Caching strategy 1. All jobs must only pull caches by default. |