From 163a7046ac76eb4109184e82ce0af911633e6626 Mon Sep 17 00:00:00 2001 From: GitLab Bot Date: Tue, 18 Feb 2020 15:08:51 +0000 Subject: Add latest changes from gitlab-org/gitlab@master --- doc/development/pipelines.md | 236 +++++++++++-------------------------------- 1 file changed, 59 insertions(+), 177 deletions(-) (limited to 'doc/development/pipelines.md') diff --git a/doc/development/pipelines.md b/doc/development/pipelines.md index 29208c1bf76..e228845bc72 100644 --- a/doc/development/pipelines.md +++ b/doc/development/pipelines.md @@ -64,180 +64,62 @@ Most of the jobs [extend from a few CI definitions](../ci/yaml/README.md#extends that are scoped to a single [configuration parameter](../ci/yaml/README.md#configuration-parameters). -These common definitions are: - -- `.default-tags`: Ensures a job has the `gitlab-org` tag to ensure it's using - our dedicated runners. -- `.default-retry`: Allows a job to [retry](../ci/yaml/README.md#retry) upon `unknown_failure`, `api_failure`, - `runner_system_failure`, `job_execution_timeout`, or `stuck_or_timeout_failure`. -- `.default-before_script`: Allows a job to use a default `before_script` definition - suitable for Ruby/Rails tasks that may need a database running (e.g. tests). -- `.default-cache`: Allows a job to use a default `cache` definition suitable for - Ruby/Rails and frontend tasks. -- `.default-only`: Restricts the cases where a job is created. This currently - includes `master`, `/^[\d-]+-stable(-ee)?$/` (stable branches), - `/^\d+-\d+-auto-deploy-\d+$/` (auto-deploy branches), `/^security\//` (security branches), `merge_requests`, `tags`. - Note that jobs won't be created for branches with this default configuration. -- `.only:variables-canonical-dot-com`: Only creates a job if the project is - located under . -- `.only:variables_refs-canonical-dot-com-schedules`: Same as - `.only:variables-canonical-dot-com` but add the condition that pipeline is scheduled. -- `.except:refs-deploy`: Don't create a job if the `ref` is an auto-deploy branch. -- `.except:refs-master-tags-stable-deploy`: Don't create a job if the `ref` is one of: - - `master` - - a tag - - a stable branch - - an auto-deploy branch -- `.only:kubernetes`: Only creates a job if a Kubernetes integration is enabled - on the project. -- `.only-review`: This extends from: - - `.only:variables-canonical-dot-com` - - `.only:kubernetes` - - `.except:refs-master-tags-stable-deploy` -- `.only-review-schedules`: This extends from: - - `.only:variables_refs-canonical-dot-com-schedules` - - `.only:kubernetes` - - `.except:refs-deploy` -- `.use-pg9`: Allows a job to use the `postgres:9.6` and `redis:alpine` services. -- `.use-pg10`: Allows a job to use the `postgres:10.9` and `redis:alpine` services. -- `.use-pg9-ee`: Same as `.use-pg9` but also use the - `docker.elastic.co/elasticsearch/elasticsearch:5.6.12` services. -- `.use-pg10-ee`: Same as `.use-pg10` but also use the - `docker.elastic.co/elasticsearch/elasticsearch:5.6.12` services. -- `.only-ee`: Only creates a job for the `gitlab` or `gitlab-ee` project. -- `.only-ee-as-if-foss`: Same as `.only-ee` but simulate the FOSS project by - setting the `FOSS_ONLY='1'` environment variable. - -## Changes detection - -If a job extends from `.default-only` (and most of the jobs should), it can restrict -the cases where it should be created -[based on the changes](../ci/yaml/README.md#onlychangesexceptchanges) -from a commit or MR by extending from the following CI definitions: - -- `.only:changes-code`: Allows a job to only be created upon code-related changes. -- `.only:changes-qa`: Allows a job to only be created upon QA-related changes. -- `.only:changes-docs`: Allows a job to only be created upon docs-related changes. -- `.only:changes-graphql`: Allows a job to only be created upon GraphQL-related changes. -- `.only:changes-code-backstage`: Allows a job to only be created upon code-related or backstage-related (e.g. Danger, RuboCop, specs) changes. -- `.only:changes-code-qa`: Allows a job to only be created upon code-related or QA-related changes. -- `.only:changes-code-backstage-qa`: Allows a job to only be created upon code-related, backstage-related (e.g. Danger, RuboCop, specs) or QA-related changes. - -**See -for the list of exact patterns.** - -## Rules conditions and changes patterns - -We're making use of the [`rules` keyword](https://docs.gitlab.com/ee/ci/yaml/#rules) but we're currently -duplicating the `if` conditions and `changes` patterns lists since they cannot be shared across -`include`d files as we do with `extends`. - -**If you update an `if` condition or `changes` -patterns list, make sure to mass-update those across all the CI config files (i.e. `.gitlab/ci/*.yml`).** - -### Canonical/security namespace merge requests only - -This condition limits jobs creation to merge requests under the `gitlab-org/` top-level group -on GitLab.com only (i.e. this won't run for `master`, stable or auto-deploy branches). -This is similar to the `.only:variables-canonical-dot-com` + `only:refs: [merge_requests]` -CI definitions. - -The definition for `if-canonical-dot-com-gitlab-org-groups-merge-request` can be -seen in . - -### Canonical/security namespace tags only - -This condition limits jobs creation to tags under the `gitlab-org/` top-level group -on GitLab.com only. -This is similar to the `.only:variables-canonical-dot-com` + `only:refs: [tags]` CI definition: - -The definition for `if-canonical-dot-com-gitlab-org-groups-tag` can be seen in -. - -### Canonical namespace `master` only - -This condition limits jobs creation to `master` pipelines for the `gitlab-org` top-level group -on GitLab.com only. -This is similar to the `.only:variables-canonical-dot-com` + `only:refs: [master]` CI definition: - -The definition for `if-canonical-dot-com-gitlab-org-group-master-refs` can be -seen in . - -### Canonical namespace schedules only - -This condition limits jobs creation to scheduled pipelines for the `gitlab-org` top-level group -on GitLab.com only. -This is similar to the `.only:variables-canonical-dot-com` + `only:refs: [schedules]` CI definition: - -The definition for `if-canonical-dot-com-gitlab-org-group-schedule` can be seen -in . - -### Not canonical/security namespace - -This condition matches if the project isn't in the canonical/security namespace. -Useful to **not** create a job if the project is a fork, or in other words, when -a job should only run in the canonical projects. - -The definition for `if-not-canonical-namespace` can be seen in -. - -### Not EE - -This condition matches if the project isn't EE. Useful to **not** create a job if -the project is GitLab, or in other words, when a job should only run in the GitLab -FOSS project. - -The definition for `if-not-ee` can be seen in -. - -### Default refs only - -This condition is the equivalent of `.default-only`. - -The definition for `if-default-refs` can be seen in -. - -### `master` refs only - -This condition is the equivalent of `only:refs: [master]`. - -The definition for `if-master-refs` can be seen in -. - -### Code changes patterns - -Similar patterns as for `.only:changes-code`: - -The definition for `code-patterns` can be seen in -. - -### QA changes patterns - -Similar patterns as for `.only:changes-qa`: - -The definition for `qa-patterns` can be seen in -. - -### Docs changes patterns - -Similar patterns as for `.only:changes-docs`: - -The definition for `docs-patterns` can be seen in -. - -### Code and QA changes patterns - -Similar patterns as for `.only:changes-code-qa`: - -The definition for `code-qa-patterns` can be seen in -. - -### Code, backstage and QA changes patterns - -Similar patterns as for `.only:changes-code-backstage-qa`: - -The definition for `code-backstage-qa-patterns` can be seen in -. +| Job definitions | Description | +|------------------|-------------| +| `.default-tags` | Ensures a job has the `gitlab-org` tag to ensure it's using our dedicated runners. | +| `.default-retry` | Allows a job to [retry](../ci/yaml/README.md#retry) upon `unknown_failure`, `api_failure`, `runner_system_failure`, `job_execution_timeout`, or `stuck_or_timeout_failure`. | +| `.default-before_script` | Allows a job to use a default `before_script` definition suitable for Ruby/Rails tasks that may need a database running (e.g. tests). | +| `.default-cache` | Allows a job to use a default `cache` definition suitable for Ruby/Rails and frontend tasks. | +| `.use-pg9` | Allows a job to use the `postgres:9.6.17` and `redis:alpine` services. | +| `.use-pg10` | Allows a job to use the `postgres:10.12` and `redis:alpine` services. | +| `.use-pg9-ee` | Same as `.use-pg9` but also use the `docker.elastic.co/elasticsearch/elasticsearch:6.4.2` services. | +| `.use-pg10-ee` | Same as `.use-pg10` but also use the `docker.elastic.co/elasticsearch/elasticsearch:6.4.2` services. | +| `.as-if-foss` | Simulate the FOSS project by setting the `FOSS_ONLY='1'` environment variable. | + +## `rules`, `if:` conditions and `changes:` patterns + +We're using the [`rules` keyword](../ci/yaml/README.md#rules) extensively. + +All `rules` definitions are defined in +, +then included in individual jobs via [`extends`](../ci/yaml/README.md#extends). + +The `rules` definitions are composed of `if:` conditions and `changes:` patterns, +which are also defined in + +and included in `rules` definitions via [YAML anchors](../ci/yaml/README.md#anchors) + +### `if:` conditions + +| `if:` conditions | Description | Notes | +|------------------|-------------|-------| +| `if-not-canonical-namespace` | Matches if the project isn't in the canonical (`gitlab-org/`) or security (`gitlab-org/security`) namespace. | Use to create a job for forks (by using `when: on_success\|manual`), or **not** create a job for forks (by using `when: never`). | +| `if-not-ee` | Matches if the project isn't EE (i.e. project name isn't `gitlab` or `gitlab-ee`). | Use to create a job only in the FOSS project (by using `when: on_success|manual`), or **not** create a job if the project is EE (by using `when: never`). | +| `if-not-foss` | Matches if the project isn't FOSS (i.e. project name isn't `gitlab-foss`, `gitlab-ce`, or `gitlabhq`). | Use to create a job only in the EE project (by using `when: on_success|manual`), or **not** create a job if the project is FOSS (by using `when: never`). | +| `if-default-refs` | Matches if the pipeline is for `master`, `/^[\d-]+-stable(-ee)?$/` (stable branches), `/^\d+-\d+-auto-deploy-\d+$/` (auto-deploy branches), `/^security\//` (security branches), merge requests, and tags. | Note that jobs won't be created for branches with this default configuration. | +| `if-master-refs` | Matches if the current branch is `master`. | | +| `if-master-or-tag` | Matches if the pipeline is for the `master` branch or for a tag. | | +| `if-merge-request` | Matches if the pipeline is for a merge request. | | +| `if-dot-com-gitlab-org-schedule` | Limits jobs creation to scheduled pipelines for the `gitlab-org` group on GitLab.com. | | +| `if-dot-com-gitlab-org-master` | Limits jobs creation to the `master` branch for the `gitlab-org` group on GitLab.com. | | +| `if-dot-com-gitlab-org-merge-request` | Limits jobs creation to merge requests for the `gitlab-org` group on GitLab.com. | | +| `if-dot-com-gitlab-org-and-security-tag` | Limits job creation to tags for the `gitlab-org` and `gitlab-org/security` groups on GitLab.com. | | +| `if-dot-com-gitlab-org-and-security-merge-request` | Limit jobs creation to merge requests for the `gitlab-org` and `gitlab-org/security` groups on GitLab.com. | | +| `if-dot-com-ee-schedule` | Limits jobs to scheduled pipelines for the `gitlab-org/gitlab` project on GitLab.com. | | +| `if-cache-credentials-schedule` | Limits jobs to scheduled pipelines with the `$CI_REPO_CACHE_CREDENTIALS` variable set. | | + +### `changes:` patterns + +| `changes:` patterns | Description | +|------------------------------|--------------------------------------------------------------------------| +| `yaml-patterns` | Only create job for YAML-related changes. | +| `docs-patterns` | Only create job for docs-related changes. | +| `backstage-patterns` | Only create job for backstage-related changes. | +| `code-patterns` | Only create job for code-related changes. | +| `qa-patterns` | Only create job for QA-related changes. | +| `code-backstage-patterns` | Combination of `code-patterns` and `backstage-patterns`. | +| `code-qa-patterns` | Combination of `code-patterns` and `qa-patterns`. | +| `code-backstage-qa-patterns` | Combination of `code-patterns`, `backstage-patterns`, and `qa-patterns`. | ## Directed acyclic graph @@ -254,11 +136,11 @@ graph RL; F[build-qa-image]; G[review-deploy]; I["karma, jest, webpack-dev-server, static-analysis"]; - I2["karma-foss, jest-foss
(EE default refs only)"]; + I2["karma-as-if-foss, jest-as-if-foss
(EE default refs only)"]; J["compile-assets pull-push-cache
(master only)"]; - J2["compile-assets pull-push-cache foss
(EE master only)"]; + J2["compile-assets pull-push-cache as-if-foss
(EE master only)"]; K[compile-assets pull-cache]; - K2["compile-assets pull-cache foss
(EE default refs only)"]; + K2["compile-assets pull-cache as-if-foss
(EE default refs only)"]; M[coverage]; N["pages (master only)"]; Q[package-and-qa]; -- cgit v1.2.3