Add latest changes from gitlab-org/gitlab@master

1 files changed, 59 insertions, 177 deletions

diff --git a/doc/development/pipelines.md b/doc/development/pipelines.md
index 29208c1bf76..e228845bc72 100644
--- a/doc/development/pipelines.md
+++ b/doc/development/pipelines.md
@@ -64,180 +64,62 @@ Most of the jobs [extend from a few CI definitions](../ci/yaml/README.md#extends
 that are scoped to a single
 [configuration parameter](../ci/yaml/README.md#configuration-parameters).
 
-These common definitions are:
-
-- `.default-tags`: Ensures a job has the `gitlab-org` tag to ensure it's using
-  our dedicated runners.
-- `.default-retry`: Allows a job to [retry](../ci/yaml/README.md#retry) upon `unknown_failure`, `api_failure`,
-  `runner_system_failure`, `job_execution_timeout`, or `stuck_or_timeout_failure`.
-- `.default-before_script`: Allows a job to use a default `before_script` definition
-  suitable for Ruby/Rails tasks that may need a database running (e.g. tests).
-- `.default-cache`: Allows a job to use a default `cache` definition suitable for
-  Ruby/Rails and frontend tasks.
-- `.default-only`: Restricts the cases where a job is created. This currently
-  includes `master`, `/^[\d-]+-stable(-ee)?$/` (stable branches),
-  `/^\d+-\d+-auto-deploy-\d+$/` (auto-deploy branches), `/^security\//` (security branches), `merge_requests`, `tags`.
-  Note that jobs won't be created for branches with this default configuration.
-- `.only:variables-canonical-dot-com`: Only creates a job if the project is
-  located under <https://gitlab.com/gitlab-org>.
-- `.only:variables_refs-canonical-dot-com-schedules`: Same as
-  `.only:variables-canonical-dot-com` but add the condition that pipeline is scheduled.
-- `.except:refs-deploy`: Don't create a job if the `ref` is an auto-deploy branch.
-- `.except:refs-master-tags-stable-deploy`: Don't create a job if the `ref` is one of:
-  - `master`
-  - a tag
-  - a stable branch
-  - an auto-deploy branch
-- `.only:kubernetes`: Only creates a job if a Kubernetes integration is enabled
-  on the project.
-- `.only-review`: This extends from:
-  - `.only:variables-canonical-dot-com`
-  - `.only:kubernetes`
-  - `.except:refs-master-tags-stable-deploy`
-- `.only-review-schedules`: This extends from:
-  - `.only:variables_refs-canonical-dot-com-schedules`
-  - `.only:kubernetes`
-  - `.except:refs-deploy`
-- `.use-pg9`: Allows a job to use the `postgres:9.6` and `redis:alpine` services.
-- `.use-pg10`: Allows a job to use the `postgres:10.9` and `redis:alpine` services.
-- `.use-pg9-ee`: Same as `.use-pg9` but also use the
-  `docker.elastic.co/elasticsearch/elasticsearch:5.6.12` services.
-- `.use-pg10-ee`: Same as `.use-pg10` but also use the
-  `docker.elastic.co/elasticsearch/elasticsearch:5.6.12` services.
-- `.only-ee`: Only creates a job for the `gitlab` or `gitlab-ee` project.
-- `.only-ee-as-if-foss`: Same as `.only-ee` but simulate the FOSS project by
-  setting the `FOSS_ONLY='1'` environment variable.
-
-## Changes detection
-
-If a job extends from `.default-only` (and most of the jobs should), it can restrict
-the cases where it should be created
-[based on the changes](../ci/yaml/README.md#onlychangesexceptchanges)
-from a commit or MR by extending from the following CI definitions:
-
-- `.only:changes-code`: Allows a job to only be created upon code-related changes.
-- `.only:changes-qa`: Allows a job to only be created upon QA-related changes.
-- `.only:changes-docs`: Allows a job to only be created upon docs-related changes.
-- `.only:changes-graphql`: Allows a job to only be created upon GraphQL-related changes.
-- `.only:changes-code-backstage`: Allows a job to only be created upon code-related or backstage-related (e.g. Danger, RuboCop, specs) changes.
-- `.only:changes-code-qa`: Allows a job to only be created upon code-related or QA-related changes.
-- `.only:changes-code-backstage-qa`: Allows a job to only be created upon code-related, backstage-related (e.g. Danger, RuboCop, specs) or QA-related changes.
-
-**See <https://gitlab.com/gitlab-org/gitlab/blob/master/.gitlab/ci/global.gitlab-ci.yml>
-for the list of exact patterns.**
-
-## Rules conditions and changes patterns
-
-We're making use of the [`rules` keyword](https://docs.gitlab.com/ee/ci/yaml/#rules) but we're currently
-duplicating the `if` conditions and `changes` patterns lists since they cannot be shared across
-`include`d files as we do with `extends`.
-
-**If you update an `if` condition or `changes`
-patterns list, make sure to mass-update those across all the CI config files (i.e. `.gitlab/ci/*.yml`).**
-
-### Canonical/security namespace merge requests only
-
-This condition limits jobs creation to merge requests under the `gitlab-org/` top-level group
-on GitLab.com only (i.e. this won't run for `master`, stable or auto-deploy branches).
-This is similar to the `.only:variables-canonical-dot-com` + `only:refs: [merge_requests]`
-CI definitions.
-
-The definition for `if-canonical-dot-com-gitlab-org-groups-merge-request` can be
-seen in <https://gitlab.com/gitlab-org/gitlab/-/blob/master/.gitlab/ci/docs.gitlab-ci.yml>.
-
-### Canonical/security namespace tags only
-
-This condition limits jobs creation to tags under the `gitlab-org/` top-level group
-on GitLab.com only.
-This is similar to the `.only:variables-canonical-dot-com` + `only:refs: [tags]` CI definition:
-
-The definition for `if-canonical-dot-com-gitlab-org-groups-tag` can be seen in
-<https://gitlab.com/gitlab-org/gitlab/-/blob/master/.gitlab/ci/cng.gitlab-ci.yml>.
-
-### Canonical namespace `master` only
-
-This condition limits jobs creation to `master` pipelines for the `gitlab-org` top-level group
-on GitLab.com only.
-This is similar to the `.only:variables-canonical-dot-com` + `only:refs: [master]` CI definition:
-
-The definition for `if-canonical-dot-com-gitlab-org-group-master-refs` can be
-seen in <https://gitlab.com/gitlab-org/gitlab/-/blob/master/.gitlab/ci/pages.gitlab-ci.yml>.
-
-### Canonical namespace schedules only
-
-This condition limits jobs creation to scheduled pipelines for the `gitlab-org` top-level group
-on GitLab.com only.
-This is similar to the `.only:variables-canonical-dot-com` + `only:refs: [schedules]` CI definition:
-
-The definition for `if-canonical-dot-com-gitlab-org-group-schedule` can be seen
-in <https://gitlab.com/gitlab-org/gitlab/-/blob/master/.gitlab/ci/qa.gitlab-ci.yml>.
-
-### Not canonical/security namespace
-
-This condition matches if the project isn't in the canonical/security namespace.
-Useful to **not** create a job if the project is a fork, or in other words, when
-a job should only run in the canonical projects.
-
-The definition for `if-not-canonical-namespace` can be seen in
-<https://gitlab.com/gitlab-org/gitlab/-/blob/master/.gitlab/ci/frontend.gitlab-ci.yml>.
-
-### Not EE
-
-This condition matches if the project isn't EE. Useful to **not** create a job if
-the project is GitLab, or in other words, when a job should only run in the GitLab
-FOSS project.
-
-The definition for `if-not-ee` can be seen in
-<https://gitlab.com/gitlab-org/gitlab/-/blob/master/.gitlab/ci/frontend.gitlab-ci.yml>.
-
-### Default refs only
-
-This condition is the equivalent of `.default-only`.
-
-The definition for `if-default-refs` can be seen in
-<https://gitlab.com/gitlab-org/gitlab/-/blob/master/.gitlab/ci/frontend.gitlab-ci.yml>.
-
-### `master` refs only
-
-This condition is the equivalent of `only:refs: [master]`.
-
-The definition for `if-master-refs` can be seen in
-<https://gitlab.com/gitlab-org/gitlab/-/blob/master/.gitlab/ci/frontend.gitlab-ci.yml>.
-
-### Code changes patterns
-
-Similar patterns as for `.only:changes-code`:
-
-The definition for `code-patterns` can be seen in
-<https://gitlab.com/gitlab-org/gitlab/-/blob/master/.gitlab/ci/qa.gitlab-ci.yml>.
-
-### QA changes patterns
-
-Similar patterns as for `.only:changes-qa`:
-
-The definition for `qa-patterns` can be seen in
-<https://gitlab.com/gitlab-org/gitlab/-/blob/master/.gitlab/ci/qa.gitlab-ci.yml>.
-
-### Docs changes patterns
-
-Similar patterns as for `.only:changes-docs`:
-
-The definition for `docs-patterns` can be seen in
-<https://gitlab.com/gitlab-org/gitlab/-/blob/master/.gitlab/ci/docs.gitlab-ci.yml>.
-
-### Code and QA changes patterns
-
-Similar patterns as for `.only:changes-code-qa`:
-
-The definition for `code-qa-patterns` can be seen in
-<https://gitlab.com/gitlab-org/gitlab/-/blob/master/.gitlab/ci/review.gitlab-ci.yml>.
-
-### Code, backstage and QA changes patterns
-
-Similar patterns as for `.only:changes-code-backstage-qa`:
-
-The definition for `code-backstage-qa-patterns` can be seen in
-<https://gitlab.com/gitlab-org/gitlab/-/blob/master/.gitlab/ci/frontend.gitlab-ci.yml>.
+| Job definitions  | Description |
+|------------------|-------------|
+| `.default-tags` | Ensures a job has the `gitlab-org` tag to ensure it's using our dedicated runners. |
+| `.default-retry` | Allows a job to [retry](../ci/yaml/README.md#retry) upon `unknown_failure`, `api_failure`, `runner_system_failure`, `job_execution_timeout`, or `stuck_or_timeout_failure`. |
+| `.default-before_script` | Allows a job to use a default `before_script` definition suitable for Ruby/Rails tasks that may need a database running (e.g. tests). |
+| `.default-cache` | Allows a job to use a default `cache` definition suitable for Ruby/Rails and frontend tasks. |
+| `.use-pg9` | Allows a job to use the `postgres:9.6.17` and `redis:alpine` services. |
+| `.use-pg10` | Allows a job to use the `postgres:10.12` and `redis:alpine` services. |
+| `.use-pg9-ee` | Same as `.use-pg9` but also use the `docker.elastic.co/elasticsearch/elasticsearch:6.4.2` services. |
+| `.use-pg10-ee` | Same as `.use-pg10` but also use the `docker.elastic.co/elasticsearch/elasticsearch:6.4.2` services. |
+| `.as-if-foss` | Simulate the FOSS project by setting the `FOSS_ONLY='1'` environment variable. |
+
+## `rules`, `if:` conditions and `changes:` patterns
+
+We're using the [`rules` keyword](../ci/yaml/README.md#rules) extensively.
+
+All `rules` definitions are defined in
+<https://gitlab.com/gitlab-org/gitlab/-/blob/master/.gitlab/ci/rules.gitlab-ci.yml>,
+then included in individual jobs via [`extends`](../ci/yaml/README.md#extends).
+
+The `rules` definitions are composed of `if:` conditions and `changes:` patterns,
+which are also defined in
+<https://gitlab.com/gitlab-org/gitlab/-/blob/master/.gitlab/ci/rules.gitlab-ci.yml>
+and included in `rules` definitions via [YAML anchors](../ci/yaml/README.md#anchors)
+
+### `if:` conditions
+
+| `if:` conditions | Description | Notes |
+|------------------|-------------|-------|
+| `if-not-canonical-namespace`                                 | Matches if the project isn't in the canonical (`gitlab-org/`) or security (`gitlab-org/security`) namespace. | Use to create a job for forks (by using `when: on_success\|manual`), or **not** create a job for forks (by using `when: never`). |
+| `if-not-ee`                                                  | Matches if the project isn't EE (i.e. project name isn't `gitlab` or `gitlab-ee`). | Use to create a job only in the FOSS project (by using `when: on_success|manual`), or **not** create a job if the project is EE (by using `when: never`). |
+| `if-not-foss`                                                | Matches if the project isn't FOSS (i.e. project name isn't `gitlab-foss`, `gitlab-ce`, or `gitlabhq`). | Use to create a job only in the EE project (by using `when: on_success|manual`), or **not** create a job if the project is FOSS (by using `when: never`). |
+| `if-default-refs`                                            | Matches if the pipeline is for `master`, `/^[\d-]+-stable(-ee)?$/` (stable branches), `/^\d+-\d+-auto-deploy-\d+$/` (auto-deploy branches), `/^security\//` (security branches), merge requests, and tags. | Note that jobs won't be created for branches with this default configuration. |
+| `if-master-refs`                                             | Matches if the current branch is `master`. | |
+| `if-master-or-tag`                                           | Matches if the pipeline is for the `master` branch or for a tag. | |
+| `if-merge-request`                                           | Matches if the pipeline is for a merge request. | |
+| `if-dot-com-gitlab-org-schedule`                             | Limits jobs creation to scheduled pipelines for the `gitlab-org` group on GitLab.com. | |
+| `if-dot-com-gitlab-org-master`                               | Limits jobs creation to the `master` branch for the `gitlab-org` group on GitLab.com. | |
+| `if-dot-com-gitlab-org-merge-request`                        | Limits jobs creation to merge requests for the `gitlab-org` group on GitLab.com. | |
+| `if-dot-com-gitlab-org-and-security-tag`                     | Limits job creation to tags for the `gitlab-org` and `gitlab-org/security` groups on GitLab.com. | |
+| `if-dot-com-gitlab-org-and-security-merge-request`           | Limit jobs creation to merge requests for the `gitlab-org` and `gitlab-org/security` groups on GitLab.com. | |
+| `if-dot-com-ee-schedule`                                     | Limits jobs to scheduled pipelines for the `gitlab-org/gitlab` project on GitLab.com. | |
+| `if-cache-credentials-schedule`                              | Limits jobs to scheduled pipelines with the `$CI_REPO_CACHE_CREDENTIALS` variable set. | |
+
+### `changes:` patterns
+
+| `changes:` patterns          | Description                                                              |
+|------------------------------|--------------------------------------------------------------------------|
+| `yaml-patterns`              | Only create job for YAML-related changes.                                |
+| `docs-patterns`              | Only create job for docs-related changes.                                |
+| `backstage-patterns`         | Only create job for backstage-related changes.                           |
+| `code-patterns`              | Only create job for code-related changes.                                |
+| `qa-patterns`                | Only create job for QA-related changes.                                  |
+| `code-backstage-patterns`    | Combination of `code-patterns` and `backstage-patterns`.                 |
+| `code-qa-patterns`           | Combination of `code-patterns` and `qa-patterns`.                        |
+| `code-backstage-qa-patterns` | Combination of `code-patterns`, `backstage-patterns`, and `qa-patterns`. |
 
 ## Directed acyclic graph
 
@@ -254,11 +136,11 @@ graph RL;
   F[build-qa-image];
   G[review-deploy];
   I["karma, jest, webpack-dev-server, static-analysis"];
-  I2["karma-foss, jest-foss<br/>(EE default refs only)"];
+  I2["karma-as-if-foss, jest-as-if-foss<br/>(EE default refs only)"];
   J["compile-assets pull-push-cache<br/>(master only)"];
-  J2["compile-assets pull-push-cache foss<br/>(EE master only)"];
+  J2["compile-assets pull-push-cache as-if-foss<br/>(EE master only)"];
   K[compile-assets pull-cache];
-  K2["compile-assets pull-cache foss<br/>(EE default refs only)"];
+  K2["compile-assets pull-cache as-if-foss<br/>(EE default refs only)"];
   M[coverage];
   N["pages (master only)"];
   Q[package-and-qa];