Add latest changes from gitlab-org/gitlab@13-0-stable-ee

1 files changed, 432 insertions, 103 deletions

diff --git a/doc/development/pipelines.md b/doc/development/pipelines.md
index 9ba6dfc110a..39ca846c1cc 100644
--- a/doc/development/pipelines.md
+++ b/doc/development/pipelines.md
@@ -19,6 +19,9 @@ The current stages are:
   <https://gitlab.com/gitlab-org/gitlab-foss>.
 - `prepare`: This stage includes jobs that prepare artifacts that are needed by
   jobs in subsequent stages.
+- `build-images`: This stage includes jobs that prepare docker images
+  that are needed by jobs in subsequent stages or downstream pipelines.
+- `fixtures`: This stage includes jobs that prepare fixtures needed by frontend tests.
 - `test`: This stage includes most of the tests, DB/migration jobs, and static analysis jobs.
 - `post-test`: This stage includes jobs that build reports or gather data from
   the `test` stage's jobs (e.g. coverage, Knapsack metadata etc.).
@@ -30,7 +33,6 @@ The current stages are:
   that is deployed in the previous stage.
 - `post-qa`: This stage includes jobs that build reports or gather data from
   the `qa` stage's jobs (e.g. Review App performance report).
-- `notification`: This stage includes jobs that sends notifications about pipeline status.
 - `pages`: This stage includes a job that deploys the various reports as
   GitLab Pages (e.g. <https://gitlab-org.gitlab.io/gitlab/coverage-ruby/>,
   <https://gitlab-org.gitlab.io/gitlab/coverage-javascript/>,
@@ -68,12 +70,9 @@ that are scoped to a single [configuration parameter](../ci/yaml/README.md#confi
 | `.default-retry` | Allows a job to [retry](../ci/yaml/README.md#retry) upon `unknown_failure`, `api_failure`, `runner_system_failure`, `job_execution_timeout`, or `stuck_or_timeout_failure`. |
 | `.default-before_script` | Allows a job to use a default `before_script` definition suitable for Ruby/Rails tasks that may need a database running (e.g. tests). |
 | `.default-cache` | Allows a job to use a default `cache` definition suitable for Ruby/Rails and frontend tasks. |
-| `.use-pg9` | Allows a job to use the `postgres:9.6.17` and `redis:alpine` services. |
-| `.use-pg10` | Allows a job to use the `postgres:10.12` and `redis:alpine` services. |
 | `.use-pg11` | Allows a job to use the `postgres:11.6` and `redis:alpine` services. |
-| `.use-pg9-ee` | Same as `.use-pg9` but also use the `docker.elastic.co/elasticsearch/elasticsearch:6.4.2` services. |
-| `.use-pg10-ee` | Same as `.use-pg10` but also use the `docker.elastic.co/elasticsearch/elasticsearch:6.4.2` services. |
 | `.use-pg11-ee` | Same as `.use-pg11` but also use the `docker.elastic.co/elasticsearch/elasticsearch:6.4.2` services. |
+| `.use-kaniko` | Allows a job to use the `kaniko` tool to build Docker images. |
 | `.as-if-foss` | Simulate the FOSS project by setting the `FOSS_ONLY='1'` environment variable. |
 
 ## `workflow:rules`
@@ -129,11 +128,12 @@ and included in `rules` definitions via [YAML anchors](../ci/yaml/README.md#anch
 
 | `changes:` patterns          | Description                                                              |
 |------------------------------|--------------------------------------------------------------------------|
+| `ci-patterns`                | Only create job for CI config-related changes.                           |
 | `yaml-patterns`              | Only create job for YAML-related changes.                                |
 | `docs-patterns`              | Only create job for docs-related changes.                                |
-| `frontend-dependency-patterns` | Only create job when frontend dependencies are updated (i.e. `package.json`, and `yarn.lock`). changes.                                |
+| `frontend-dependency-patterns` | Only create job when frontend dependencies are updated (i.e. `package.json`, and `yarn.lock`). changes. |
 | `frontend-patterns`          | Only create job for frontend-related changes.                           |
-| `backstage-patterns`         | Only create job for backstage-related changes (i.e. Danger, fixtures, RuboCop, specs).                           |
+| `backstage-patterns`         | Only create job for backstage-related changes (i.e. Danger, fixtures, RuboCop, specs). |
 | `code-patterns`              | Only create job for code-related changes.                                |
 | `qa-patterns`                | Only create job for QA-related changes.                                  |
 | `code-backstage-patterns`    | Combination of `code-patterns` and `backstage-patterns`.                 |
@@ -151,112 +151,360 @@ request, be sure to start the `dont-interrupt-me` job before pushing.
 
 ## PostgreSQL versions testing
 
+### Current versions testing
+
+| Where? | PostgreSQL version |
+| ------ | ------ |
+| MRs | 11 |
+| `master` (non-scheduled pipelines) | 11 |
+| 2-hourly scheduled pipelines | 11 |
+
+### Long-term plan
+
 We follow the [PostgreSQL versions shipped with Omnibus GitLab](https://docs.gitlab.com/omnibus/package-information/postgresql_versions.html):
 
-|        | 12.10 (April 2020) | 13.0 (May 2020) | 13.1 (June 2020) | 13.2 (July 2020) | 13.3 (August 2020) | 13.4, 13.5   | 13.6 (November 2020) | 14.0 (May 2021?) |
+| PostgreSQL version | 12.10 (April 2020) | 13.0 (May 2020) | 13.1 (June 2020) | 13.2 (July 2020) | 13.3 (August 2020) | 13.4, 13.5   | 13.6 (November 2020) | 14.0 (May 2021?) |
 | ------ | ------------------ | --------------- | ---------------- | ---------------- | ------------------ | ------------ | -------------------- | ---------------- |
-| PG9.6  | nightly            | -               | -                | -                | -                  | -            | -                    | -                |
-| PG10   | `master`           | -               | -                | -                | -                  | -            | -                    | -                |
-| PG11   | MRs/`master`       | MRs/`master`    | MRs/`master`     | MRs/`master`     | MRs/`master`       | MRs/`master` | nightly              | -                |
-| PG12   | -                  | -               | -                | -                | `master`           | `master`     | MRs/`master`         | `master`         |
-| PG13   | -                  | -               | -                | -                | -                  | -            | -                    | MRs/`master`     |
+| PG9.6  | MRs/`master`/`2-hour`/`nightly` | -               | -                | -                | -                  | -            | -                    | -                |
+| PG10   | `nightly`           | -               | -                | -                | -                  | -            | -                    | -                |
+| PG11   | `master`/`2-hour` | MRs/`master`/`2-hour`/`nightly` | MRs/`master`/`2-hour`/`nightly` | MRs/`master`/`2-hour`/`nightly` | MRs/`master`/`2-hour`/`nightly` | MRs/`master`/`2-hour`/`nightly` | `nightly`              | -                |
+| PG12   | -                  | -               | -                | -                | `master`/`2-hour` | `master`/`2-hour` | MRs/`master`/`2-hour`/`nightly`         | `master`/`2-hour` |
+| PG13   | -                  | -               | -                | -                | -                  | -            | -                    | MRs/`master`/`2-hour`/`nightly`     |
+
+## Pipeline types
+
+Since we use the [`rules:`](../ci/yaml/README.md#rules) and [`needs:`](../ci/yaml/README.md#needs) keywords extensively,
+we have four main pipeline types which are described below. Note that an MR that includes multiple types of changes would
+have a pipelines that include jobs from multiple types (e.g. a combination of docs-only and code-only pipelines).
+
+### Docs-only MR pipeline
+
+Reference pipeline: <https://gitlab.com/gitlab-org/gitlab/pipelines/135236627>
+
+```mermaid
+graph LR
+  subgraph "No needed jobs";
+    1-1["danger-review (3.5 minutes)"];
+    click 1-1 "https://app.periscopedata.com/app/gitlab/652085/Engineering-Productivity---Pipeline-Build-Durations?widget=8100542&udv=0"
+    1-50["docs lint (6.75 minutes)"];
+    click 1-50 "https://app.periscopedata.com/app/gitlab/652085/Engineering-Productivity---Pipeline-Build-Durations?widget=8356757&udv=0"
+  end
+```
+
+### Code-only MR pipeline
+
+Reference pipeline: <https://gitlab.com/gitlab-org/gitlab/pipelines/136295694>
+
+```mermaid
+graph RL;
+  classDef criticalPath fill:#f66;
+
+  subgraph "No needed jobs";
+    1-1["danger-review (3.5 minutes)"];
+    click 1-1 "https://app.periscopedata.com/app/gitlab/652085/Engineering-Productivity---Pipeline-Build-Durations?widget=8100542&udv=0"
+    1-2["build-qa-image (3.4 minutes)"];
+    click 1-2 "https://app.periscopedata.com/app/gitlab/652085/Engineering-Productivity---Pipeline-Build-Durations?widget=6914325&udv=0"
+    1-3["compile-assets pull-cache (9.06 minutes)"];
+    click 1-3 "https://app.periscopedata.com/app/gitlab/652085/Engineering-Productivity---Pipeline-Build-Durations?widget=6914317&udv=0"
+    1-4["compile-assets pull-cache as-if-foss (8.35 minutes)"];
+    click 1-4 "https://app.periscopedata.com/app/gitlab/652085/Engineering-Productivity---Pipeline-Build-Durations?widget=8356616&udv=0"
+    1-5["gitlab:assets:compile pull-cache (22 minutes)"];
+    click 1-5 "https://app.periscopedata.com/app/gitlab/652085/Engineering-Productivity---Pipeline-Build-Durations?widget=6914312&udv=0"
+    1-6["setup-test-env (8.22 minutes)"];
+    click 1-6 "https://app.periscopedata.com/app/gitlab/652085/Engineering-Productivity---Pipeline-Build-Durations?widget=6914315&udv=0"
+    1-7["review-stop-failed-deployment"];
+    1-8["dependency_scanning"];
+    1-9["qa:internal, qa:internal-as-if-foss"];
+    1-11["qa:selectors, qa:selectors-as-if-foss"];
+    1-14["retrieve-tests-metadata (1.5 minutes)"];
+    click 1-14 "https://app.periscopedata.com/app/gitlab/652085/Engineering-Productivity---Pipeline-Build-Durations?widget=8356697&udv=0"
+    1-15["code_quality"];
+    1-16["brakeman-sast"];
+    1-17["eslint-sast"];
+    1-18["kubesec-sast"];
+    1-19["nodejs-scan-sast"];
+    1-20["secrets-sast"];
+
+    class 1-3 criticalPath;
+    class 1-6 criticalPath;
+  end
+
+  2_1-1["graphql-reference-verify (5 minutes)"];
+  click 2_1-1 "https://app.periscopedata.com/app/gitlab/652085/Engineering-Productivity---Pipeline-Build-Durations?widget=8356715&udv=0"
+  2_1-2["memory-static (4.75 minutes)"];
+  click 2_1-2 "https://app.periscopedata.com/app/gitlab/652085/Engineering-Productivity---Pipeline-Build-Durations?widget=8356721&udv=0"
+  2_1-3["run-dev-fixtures (5 minutes)"];
+  click 2_1-3 "https://app.periscopedata.com/app/gitlab/652085/Engineering-Productivity---Pipeline-Build-Durations?widget=8356729&udv=0"
+  2_1-4["run-dev-fixtures-ee (5 minutes)"];
+  click 2_1-4 "https://app.periscopedata.com/app/gitlab/652085/Engineering-Productivity---Pipeline-Build-Durations?widget=8356731&udv=0"
+  subgraph "Needs `setup-test-env`";
+    2_1-1 & 2_1-2 & 2_1-3 & 2_1-4 --> 1-6;
+  end
+
+  2_2-1["static-analysis (17 minutes)"];
+  click 2_2-1 "https://app.periscopedata.com/app/gitlab/652085/Engineering-Productivity---Pipeline-Build-Durations?widget=6914471&udv=0"
+  2_2-2["frontend-fixtures (17.2 minutes)"];
+  class 2_2-2 criticalPath;
+  click 2_2-2 "https://app.periscopedata.com/app/gitlab/652085/Engineering-Productivity---Pipeline-Build-Durations?widget=7910143&udv=0"
+  2_2-3["frontend-fixtures-as-if-foss (8.75 minutes)"];
+  click 2_2-3 "https://app.periscopedata.com/app/gitlab/652085/Engineering-Productivity---Pipeline-Build-Durations?widget=7910154&udv=0"
+  2_2-4["memory-on-boot (7.19 minutes)"];
+  click 2_2-4 "https://app.periscopedata.com/app/gitlab/652085/Engineering-Productivity---Pipeline-Build-Durations?widget=8356727&udv=0"
+  2_2-5["webpack-dev-server (6.1 minutes)"];
+  click 2_2-5 "https://app.periscopedata.com/app/gitlab/652085/Engineering-Productivity---Pipeline-Build-Durations?widget=8404303&udv=0"
+  subgraph "Needs `setup-test-env` & `compile-assets`";
+    2_2-1 & 2_2-2 & 2_2-4 & 2_2-5 --> 1-6 & 1-3;
+    2_2-3 --> 1-6 & 1-4;
+  end
+
+  2_3-1["build-assets-image (2.5 minutes)"];
+  subgraph "Needs `gitlab:assets:compile`";
+    2_3-1 --> 1-5
+  end
+
+  2_4-1["package-and-qa (manual)"];
+  subgraph "Needs `build-qa-image`";
+    2_4-1 --> 1-2;
+    click 2_4-1 "https://app.periscopedata.com/app/gitlab/652085/Engineering-Productivity---Pipeline-Build-Durations?widget=6914305&udv=0"
+  end
+
+  2_5-1["rspec & db jobs (12-22 minutes)"];
+  subgraph "Needs `compile-assets`, `setup-test-env`, & `retrieve-tests-metadata`";
+    2_5-1 --> 1-3 & 1-6 & 1-14;
+    class 2_5-1 criticalPath;
+    click 2_5-1 "https://app.periscopedata.com/app/gitlab/652085/Engineering-Productivity---Pipeline-Build-Durations"
+  end
+
+  3_1-1["jest (15 minutes)"];
+  class 3_1-1 criticalPath;
+  click 3_1-1 "https://app.periscopedata.com/app/gitlab/652085/Engineering-Productivity---Pipeline-Build-Durations?widget=6914204&udv=0"
+  3_1-2["karma (8 minutes)"];
+  click 3_1-3 "https://app.periscopedata.com/app/gitlab/652085/Engineering-Productivity---Pipeline-Build-Durations?widget=6914200&udv=0"
+  3_1-3["jest-as-if-foss (19.7 minutes)"];
+  click 3_1-3 "https://app.periscopedata.com/app/gitlab/652085/Engineering-Productivity---Pipeline-Build-Durations?widget=6914205&udv=0"
+  3_1-4["karma-as-if-foss (7.5 minutes)"];
+  click 3_1-4 "https://app.periscopedata.com/app/gitlab/652085/Engineering-Productivity---Pipeline-Build-Durations?widget=6914203&udv=0"
+  subgraph "Needs `frontend-fixtures`";
+    3_1-1 & 3_1-2 --> 2_2-2;
+    3_1-3 & 3_1-4 --> 2_2-3;
+  end
+
+  3_2-1["rspec:coverage (6.5 minutes)"];
+  subgraph "Depends on `rspec` jobs";
+    3_2-1 -.->|"(don't use needs because of limitations)"| 2_5-1;
+    class 3_2-1 criticalPath;
+    click 3_2-1 "https://app.periscopedata.com/app/gitlab/652085/Engineering-Productivity---Pipeline-Build-Durations?widget=7248745&udv=0"
+  end
+
+  4_1-1["coverage-frontend (3.6 minutes)"];
+  subgraph "Needs `jest`";
+    4_1-1 --> 3_1-1;
+    class 4_1-1 criticalPath;
+    click 4_1-1 "https://app.periscopedata.com/app/gitlab/652085/Engineering-Productivity---Pipeline-Build-Durations?widget=7910777&udv=0"
+  end
+```
+
+### Frontend-only MR pipeline
+
+Reference pipeline: <https://gitlab.com/gitlab-org/gitlab/pipelines/134661039>
+
+```mermaid
+graph RL;
+  classDef criticalPath fill:#f66;
+
+  subgraph "No needed jobs";
+    1-1["danger-review (3.5 minutes)"];
+    click 1-1 "https://app.periscopedata.com/app/gitlab/652085/Engineering-Productivity---Pipeline-Build-Durations?widget=8100542&udv=0"
+    1-2["build-qa-image (3.4 minutes)"];
+    click 1-2 "https://app.periscopedata.com/app/gitlab/652085/Engineering-Productivity---Pipeline-Build-Durations?widget=6914325&udv=0"
+    1-3["compile-assets pull-cache (9.06 minutes)"];
+    click 1-3 "https://app.periscopedata.com/app/gitlab/652085/Engineering-Productivity---Pipeline-Build-Durations?widget=6914317&udv=0"
+    1-4["compile-assets pull-cache as-if-foss (8.35 minutes)"];
+    click 1-4 "https://app.periscopedata.com/app/gitlab/652085/Engineering-Productivity---Pipeline-Build-Durations?widget=8356616&udv=0"
+    1-5["gitlab:assets:compile pull-cache (22 minutes)"];
+    click 1-5 "https://app.periscopedata.com/app/gitlab/652085/Engineering-Productivity---Pipeline-Build-Durations?widget=6914312&udv=0"
+    1-6["setup-test-env (8.22 minutes)"];
+    click 1-6 "https://app.periscopedata.com/app/gitlab/652085/Engineering-Productivity---Pipeline-Build-Durations?widget=6914315&udv=0"
+    1-7["review-stop-failed-deployment"];
+    1-8["dependency_scanning"];
+    1-9["qa:internal, qa:internal-as-if-foss"];
+    1-11["qa:selectors, qa:selectors-as-if-foss"];
+    1-14["retrieve-tests-metadata (1.5 minutes)"];
+    click 1-14 "https://app.periscopedata.com/app/gitlab/652085/Engineering-Productivity---Pipeline-Build-Durations?widget=8356697&udv=0"
+    1-15["code_quality"];
+    1-16["brakeman-sast"];
+    1-17["eslint-sast"];
+    1-18["kubesec-sast"];
+    1-19["nodejs-scan-sast"];
+    1-20["secrets-sast"];
+
+    class 1-3 criticalPath;
+    class 1-5 criticalPath;
+    class 1-6 criticalPath;
+  end
+
+  2_1-1["graphql-reference-verify (5 minutes)"];
+  click 2_1-1 "https://app.periscopedata.com/app/gitlab/652085/Engineering-Productivity---Pipeline-Build-Durations?widget=8356715&udv=0"
+  2_1-2["memory-static (4.75 minutes)"];
+  click 2_1-2 "https://app.periscopedata.com/app/gitlab/652085/Engineering-Productivity---Pipeline-Build-Durations?widget=8356721&udv=0"
+  2_1-3["run-dev-fixtures (5 minutes)"];
+  click 2_1-3 "https://app.periscopedata.com/app/gitlab/652085/Engineering-Productivity---Pipeline-Build-Durations?widget=8356729&udv=0"
+  2_1-4["run-dev-fixtures-ee (5 minutes)"];
+  click 2_1-4 "https://app.periscopedata.com/app/gitlab/652085/Engineering-Productivity---Pipeline-Build-Durations?widget=8356731&udv=0"
+  subgraph "Needs `setup-test-env`";
+    2_1-1 & 2_1-2 & 2_1-3 & 2_1-4 --> 1-6;
+  end
+
+  2_2-1["static-analysis (17 minutes)"];
+  click 2_2-1 "https://app.periscopedata.com/app/gitlab/652085/Engineering-Productivity---Pipeline-Build-Durations?widget=6914471&udv=0"
+  2_2-2["frontend-fixtures (17.2 minutes)"];
+  class 2_2-2 criticalPath;
+  click 2_2-2 "https://app.periscopedata.com/app/gitlab/652085/Engineering-Productivity---Pipeline-Build-Durations?widget=7910143&udv=0"
+  2_2-3["frontend-fixtures-as-if-foss (8.75 minutes)"];
+  click 2_2-3 "https://app.periscopedata.com/app/gitlab/652085/Engineering-Productivity---Pipeline-Build-Durations?widget=7910154&udv=0"
+  2_2-4["memory-on-boot (7.19 minutes)"];
+  click 2_2-4 "https://app.periscopedata.com/app/gitlab/652085/Engineering-Productivity---Pipeline-Build-Durations?widget=8356727&udv=0"
+  2_2-5["webpack-dev-server (6.1 minutes)"];
+  click 2_2-5 "https://app.periscopedata.com/app/gitlab/652085/Engineering-Productivity---Pipeline-Build-Durations?widget=8404303&udv=0"
+  subgraph "Needs `setup-test-env` & `compile-assets`";
+    2_2-1 & 2_2-2 & 2_2-4 & 2_2-5 --> 1-6 & 1-3;
+    2_2-3 --> 1-6 & 1-4;
+  end
+
+  2_3-1["build-assets-image (2.5 minutes)"];
+  class 2_3-1 criticalPath;
+  subgraph "Needs `gitlab:assets:compile`";
+    2_3-1 --> 1-5
+  end
 
-## Directed acyclic graph
+  2_4-1["package-and-qa (manual)"];
+  subgraph "Needs `build-qa-image` & `build-assets-image`";
+    2_4-1 --> 1-2 & 2_3-1;
+    click 2_4-1 "https://app.periscopedata.com/app/gitlab/652085/Engineering-Productivity---Pipeline-Build-Durations?widget=6914305&udv=0"
+  end
+
+  2_5-1["rspec & db jobs (12-22 minutes)"];
+  subgraph "Needs `compile-assets`, `setup-test-env, & `retrieve-tests-metadata`";
+    2_5-1 --> 1-3 & 1-6 & 1-14;
+    class 2_5-1 criticalPath;
+    click 2_5-1 "https://app.periscopedata.com/app/gitlab/652085/Engineering-Productivity---Pipeline-Build-Durations"
+  end
+
+  2_6-1["review-build-cng (27.3 minutes)"];
+  subgraph "Needs `build-assets-image`";
+    2_6-1 --> 2_3-1;
+    class 2_6-1 criticalPath;
+    click 2_6-1 "https://app.periscopedata.com/app/gitlab/652085/Engineering-Productivity---Pipeline-Build-Durations?widget=6914314&udv=0"
+  end
+
+  3_1-1["jest (15 minutes)"];
+  class 3_1-1 criticalPath;
+  click 3_1-1 "https://app.periscopedata.com/app/gitlab/652085/Engineering-Productivity---Pipeline-Build-Durations?widget=6914204&udv=0"
+  3_1-2["karma (8 minutes)"];
+  click 3_1-3 "https://app.periscopedata.com/app/gitlab/652085/Engineering-Productivity---Pipeline-Build-Durations?widget=6914200&udv=0"
+  3_1-3["jest-as-if-foss (19.7 minutes)"];
+  click 3_1-3 "https://app.periscopedata.com/app/gitlab/652085/Engineering-Productivity---Pipeline-Build-Durations?widget=6914205&udv=0"
+  3_1-4["karma-as-if-foss (7.5 minutes)"];
+  click 3_1-4 "https://app.periscopedata.com/app/gitlab/652085/Engineering-Productivity---Pipeline-Build-Durations?widget=6914203&udv=0"
+  subgraph "Needs `frontend-fixtures`";
+    3_1-1 & 3_1-3 --> 2_2-2;
+    3_1-2 & 3_1-4 --> 2_2-3;
+  end
+
+  3_2-1["rspec:coverage (6.5 minutes)"];
+  subgraph "Depends on `rspec` jobs";
+    3_2-1 -.->|"(don't use needs because of limitations)"| 2_5-1;
+    class 3_2-1 criticalPath;
+    click 3_2-1 "https://app.periscopedata.com/app/gitlab/652085/Engineering-Productivity---Pipeline-Build-Durations?widget=7248745&udv=0"
+  end
 
-We're using the [`needs:`](../ci/yaml/README.md#needs) keyword to
-execute jobs out of order for the following jobs:
+  4_1-1["coverage-frontend (3.6 minutes)"];
+  subgraph "Needs `jest`";
+    4_1-1 --> 3_1-1;
+    class 4_1-1 criticalPath;
+    click 4_1-1 "https://app.periscopedata.com/app/gitlab/652085/Engineering-Productivity---Pipeline-Build-Durations?widget=7910777&udv=0"
+  end
+
+  3_3-1["review-deploy (6 minutes)"];
+  subgraph "Played by `review-build-cng`";
+    3_3-1 --> 2_6-1;
+    class 3_3-1 criticalPath;
+    click 3_3-1 "https://app.periscopedata.com/app/gitlab/652085/Engineering-Productivity---Pipeline-Build-Durations?widget=6721130&udv=0"
+  end
+
+  4_2-1["review-qa-smoke (8 minutes)"];
+  click 4_2-1 "https://app.periscopedata.com/app/gitlab/652085/Engineering-Productivity---Pipeline-Build-Durations?widget=6729805&udv=0"
+  4_2-2["review-performance (4 minutes)"];
+  click 4_2-2 "https://app.periscopedata.com/app/gitlab/652085/Engineering-Productivity---Pipeline-Build-Durations?widget=8356817&udv=0"
+  4_2-3["dast (18 minutes)"];
+  click 4_2-3 "https://app.periscopedata.com/app/gitlab/652085/Engineering-Productivity---Pipeline-Build-Durations?widget=8356819&udv=0"
+  class 4_2-3 criticalPath;
+  subgraph "Played by `review-deploy`";
+    4_2-1 & 4_2-2 & 4_2-3 -.->|"(don't use needs because of limitations)"| 3_3-1;
+  end
+```
+
+### QA-only MR pipeline
+
+Reference pipeline: <https://gitlab.com/gitlab-org/gitlab/pipelines/134645109>
 
 ```mermaid
 graph RL;
-  A[setup-test-env];
-  B["gitlab:assets:compile pull-push-cache<br/>(canonical master only)"];
-  C["gitlab:assets:compile pull-cache<br/>(canonical default refs only)"];
-  D["cache gems<br/>(master and tags only)"];
-  E[review-build-cng];
-  F[build-qa-image];
-  G[review-deploy];
-  I["karma, jest"];
-  I2["karma-as-if-foss, jest-as-if-foss<br/>(EE default refs only)"];
-  J["compile-assets pull-push-cache<br/>(master only)"];
-  J2["compile-assets pull-push-cache as-if-foss<br/>(EE master only)"];
-  K[compile-assets pull-cache];
-  K2["compile-assets pull-cache as-if-foss<br/>(EE default refs only)"];
-  U[frontend-fixtures];
-  U2["frontend-fixtures-as-if-foss<br/>(EE default refs only)"];
-  V["webpack-dev-server, static-analysis"];
-  M[coverage];
-  O[coverage-frontend];
-  N["pages (master only)"];
-  Q[package-and-qa];
-  S["RSpec<br/>(e.g. rspec unit pg10)"]
-  T[retrieve-tests-metadata];
-  QA["qa:internal, qa:selectors"];
-  QA2["qa:internal-as-if-foss, qa:selectors-as-if-foss<br/>(EE default refs only)"];
-  X["docs lint, code_quality, sast, dependency_scanning, danger-review"];
-
-subgraph "`prepare` stage"
-    A
-    B
-    C
-    F
-    K
-    K2
-    J
-    J2
-    T
-    end
-
-subgraph "`fixture` stage"
-    U -.-> |needs and depends on| A;
-    U -.-> |needs and depends on| K;
-    U2 -.-> |needs and depends on| A;
-    U2 -.-> |needs and depends on| K2;
-    end
-
-subgraph "`test` stage"
-    D -.-> |needs| A;
-    I -.-> |needs and depends on| U;
-    I2 -.-> |needs and depends on| U2;
-    L -.-> |needs and depends on| A;
-    S -.-> |needs and depends on| A;
-    S -.-> |needs and depends on| K;
-    S -.-> |needs and depends on| T;
-    L["db:*, gitlab:setup, graphql-docs-verify, downtime_check"] -.-> |needs| A;
-    V -.-> |needs and depends on| K;
-    X -.-> |needs| T;
-    QA -.-> |needs| T;
-    QA2 -.-> |needs| T;
-    end
-
-subgraph "`post-test` stage"
-    M --> |happens after| S
-    O --> |needs `jest`| I
-    end
-
-subgraph "`review-prepare` stage"
-    E -.-> |needs| C;
-    end
-
-subgraph "`review` stage"
-    G -.-> |needs| E
-    end
-
-subgraph "`qa` stage"
-    Q -.-> |needs| C;
-    Q -.-> |needs| F;
-    QA1["review-qa-smoke, review-qa-all, review-performance, dast"] -.-> |needs| G;
-    end
-
-subgraph "`post-qa` stage"
-  PQA1["parallel-spec-reports"] -.-> |depends on `review-qa-all`| QA1;
+  classDef criticalPath fill:#f66;
+
+  subgraph "No needed jobs";
+    1-1["danger-review (3.5 minutes)"];
+    click 1-1 "https://app.periscopedata.com/app/gitlab/652085/Engineering-Productivity---Pipeline-Build-Durations?widget=8100542&udv=0"
+    1-2["build-qa-image (3.4 minutes)"];
+    click 1-2 "https://app.periscopedata.com/app/gitlab/652085/Engineering-Productivity---Pipeline-Build-Durations?widget=6914325&udv=0"
+    1-3["compile-assets pull-cache (9.06 minutes)"];
+    click 1-3 "https://app.periscopedata.com/app/gitlab/652085/Engineering-Productivity---Pipeline-Build-Durations?widget=6914317&udv=0"
+    1-4["compile-assets pull-cache as-if-foss (8.35 minutes)"];
+    click 1-4 "https://app.periscopedata.com/app/gitlab/652085/Engineering-Productivity---Pipeline-Build-Durations?widget=8356616&udv=0"
+    1-5["gitlab:assets:compile pull-cache (22 minutes)"];
+    click 1-5 "https://app.periscopedata.com/app/gitlab/652085/Engineering-Productivity---Pipeline-Build-Durations?widget=6914312&udv=0"
+    1-6["setup-test-env (8.22 minutes)"];
+    click 1-6 "https://app.periscopedata.com/app/gitlab/652085/Engineering-Productivity---Pipeline-Build-Durations?widget=6914315&udv=0"
+    1-7["review-stop-failed-deployment"];
+    1-8["dependency_scanning"];
+    1-9["qa:internal, qa:internal-as-if-foss"];
+    1-11["qa:selectors, qa:selectors-as-if-foss"];
+    1-14["retrieve-tests-metadata (1.5 minutes)"];
+    click 1-14 "https://app.periscopedata.com/app/gitlab/652085/Engineering-Productivity---Pipeline-Build-Durations?widget=8356697&udv=0"
+    1-15["code_quality"];
+    1-16["brakeman-sast"];
+    1-17["eslint-sast"];
+    1-18["kubesec-sast"];
+    1-19["nodejs-scan-sast"];
+    1-20["secrets-sast"];
+
+    class 1-5 criticalPath;
+  end
+
+  2_1-1["graphql-reference-verify (5 minutes)"];
+  click 2_1-1 "https://app.periscopedata.com/app/gitlab/652085/Engineering-Productivity---Pipeline-Build-Durations?widget=8356715&udv=0"
+  subgraph "Needs `setup-test-env`";
+    2_1-1 --> 1-6;
+  end
+
+  2_2-1["static-analysis (17 minutes)"];
+  click 2_2-1 "https://app.periscopedata.com/app/gitlab/652085/Engineering-Productivity---Pipeline-Build-Durations?widget=6914471&udv=0"
+  subgraph "Needs `setup-test-env` & `compile-assets`";
+    2_2-1 --> 1-6 & 1-3;
+  end
+
+  2_3-1["build-assets-image (2.5 minutes)"];
+  subgraph "Needs `gitlab:assets:compile`";
+    2_3-1 --> 1-5
+    class 2_3-1 criticalPath;
   end
 
-subgraph "`pages` stage"
-    N -.-> |depends on| C;
-    N -.-> |depends on karma| I;
-    N -.-> |depends on| M;
-    N --> |happens after| PQA1
-    end
+  2_4-1["package-and-qa (108 minutes)"];
+  subgraph "Needs `build-qa-image` & `build-assets-image`";
+    2_4-1 --> 1-2 & 2_3-1;
+    class 2_4-1 criticalPath;
+    click 2_4-1 "https://app.periscopedata.com/app/gitlab/652085/Engineering-Productivity---Pipeline-Build-Durations?widget=6914305&udv=0"
+  end
 ```
 
 ## Test jobs
@@ -268,6 +516,87 @@ for more information.
 
 Consult the [Review Apps](testing_guide/review_apps.md) dedicated page for more information.
 
+## As-if-FOSS jobs
+
+The `* as-if-foss` jobs allows to run GitLab's test suite "as-if-FOSS", meaning as if the jobs would run in the context
+of the `gitlab-org/gitlab-foss` project. These jobs are only created in the following cases:
+
+- `master` commits (pushes and scheduled pipelines).
+- `gitlab-org/security/gitlab` merge requests.
+- Merge requests which include `RUN AS-IF-FOSS` in their title.
+- Merge requests that changes the CI config.
+
+The `* as-if-foss` jobs have the `FOSS_ONLY='1'` variable set and gets their EE-specific
+folders removed before the tests start running.
+
+The intent is to ensure that a change won't introduce a failure once the `gitlab-org/gitlab` project will be synced to
+the `gitlab-org/gitlab-foss` project.
+
+## Pre-clone step
+
+The `gitlab-org/gitlab` project on GitLab.com uses a [pre-clone step](https://gitlab.com/gitlab-org/gitlab/issues/39134)
+to seed the project with a recent archive of the repository. This is done for
+several reasons:
+
+- It speeds up builds because a 800 MB download only takes seconds, as opposed to a full Git clone.
+- It significantly reduces load on the file server, as smaller deltas mean less time spent in `git pack-objects`.
+
+The pre-clone step works by using the `CI_PRE_CLONE_SCRIPT` variable
+[defined by GitLab.com shared runners](../user/gitlab_com/index.md#pre-clone-script).
+
+The `CI_PRE_CLONE_SCRIPT` is currently defined as a project CI/CD
+variable:
+
+```shell
+echo "Downloading archived master..."
+wget -O /tmp/gitlab.tar.gz https://storage.googleapis.com/gitlab-ci-git-repo-cache/project-278964/gitlab-master.tar.gz
+
+if [ ! -f /tmp/gitlab.tar.gz ]; then
+    echo "Repository cache not available, cloning a new directory..."
+    exit
+fi
+
+rm -rf $CI_PROJECT_DIR
+echo "Extracting tarball into $CI_PROJECT_DIR..."
+mkdir -p $CI_PROJECT_DIR
+cd $CI_PROJECT_DIR
+tar xzf /tmp/gitlab.tar.gz
+rm -f /tmp/gitlab.tar.gz
+chmod a+w $CI_PROJECT_DIR
+```
+
+The first step of the script downloads `gitlab-master.tar.gz` from
+Google Cloud Storage. There is a [GitLab CI job named `cache-repo`](https://gitlab.com/gitlab-org/gitlab/blob/master/.gitlab/ci/cache-repo.gitlab-ci.yml#L5)
+that is responsible for keeping that archive up-to-date. Every two hours
+on a scheduled pipeline, it does the following:
+
+1. Creates a fresh clone of the `gitlab-org/gitlab` repository on GitLab.com.
+1. Saves the data as a `.tar.gz`.
+1. Uploads it into the Google Cloud Storage bucket.
+
+When a CI job runs with this configuration, you'll see something like
+this:
+
+```shell
+$ eval "$CI_PRE_CLONE_SCRIPT"
+Downloading archived master...
+Extracting tarball into /builds/group/project...
+Fetching changes...
+Reinitialized existing Git repository in /builds/group/project/.git/
+```
+
+Note that the `Reinitialized existing Git repository` message shows that
+the pre-clone step worked. The runner runs `git init`, which
+overwrites the Git configuration with the appropriate settings to fetch
+from the GitLab repository.
+
+`CI_REPO_CACHE_CREDENTIALS` contains the Google Cloud service account
+JSON for uploading to the `gitlab-ci-git-repo-cache` bucket. These
+credentials are stored in the 1Password GitLab.com Production vault.
+
+Note that this bucket should be located in the same continent as the
+runner, or [network egress charges will apply](https://cloud.google.com/storage/pricing).
+
 ---
 
 [Return to Development documentation](README.md)