Welcome to mirror list, hosted at ThFree Co, Russian Federation.

gitlab.com/gitlab-org/gitlab-foss.git - Unnamed repository; edit this file 'description' to name the repository.
summaryrefslogtreecommitdiff
diff options
context:
space:
mode:
Diffstat (limited to 'doc/development/testing_guide/review_apps.md')
-rw-r--r--doc/development/testing_guide/review_apps.md50
1 files changed, 26 insertions, 24 deletions
diff --git a/doc/development/testing_guide/review_apps.md b/doc/development/testing_guide/review_apps.md
index 5edf18725c0..b074a9e34f3 100644
--- a/doc/development/testing_guide/review_apps.md
+++ b/doc/development/testing_guide/review_apps.md
@@ -21,18 +21,23 @@ For any of the following scenarios, the `start-review-app-pipeline` job would be
- for scheduled pipelines
- the MR has the `pipeline:run-review-app` label set
-## QA runs on review apps
+## E2E test runs on review apps
-On every [pipeline](https://gitlab.com/gitlab-org/gitlab/pipelines/125315730) in the `qa` stage (which comes after the
-`review` stage), the `review-qa-smoke` and `review-qa-reliable` jobs are automatically started. The `review-qa-smoke` runs
-the QA smoke suite and the `review-qa-reliable` executes E2E tests identified as [reliable](https://about.gitlab.com/handbook/engineering/quality/quality-engineering/reliable-tests/).
+On every pipeline in the `qa` stage (which comes after the `review` stage), the `review-qa-smoke` and `review-qa-blocking` jobs are automatically started.
-`review-qa-*` jobs ensure that end-to-end tests for the changes in the merge request pass in a live environment. This shifts the identification of e2e failures from an environment on the path to production to the merge request, to prevent breaking features on GitLab.com or costly GitLab.com deployment blockers. `review-qa-*` failures should be investigated with counterpart SET involvement if needed to help determine the root cause of the error.
+`qa` stage consists of following jobs:
-You can also manually start the `review-qa-all`: it runs the full QA suite.
+- `review-qa-smoke`: small and fast subset of tests to validate core functionality of GitLab.
+- `review-qa-blocking`: subset of tests identified as [reliable](https://about.gitlab.com/handbook/engineering/quality/quality-engineering/reliable-tests/). These tests are
+ considered stable and are not allowed to fail.
+- `review-qa-non-blocking`: rest of the e2e tests that can be triggered manually.
+
+`review-qa-*` jobs ensure that end-to-end tests for the changes in the merge request pass in a live environment. This shifts the identification of e2e failures from an environment
+on the path to production to the merge request to prevent breaking features on GitLab.com or costly GitLab.com deployment blockers. If needed, `review-qa-*` failures should be
+investigated with an SET (software engineer in test) counterpart to help determine the root cause of the error.
After the end-to-end test runs have finished, [Allure reports](https://github.com/allure-framework/allure2) are generated and published by
-the `allure-report-qa-smoke`, `allure-report-qa-reliable`, and `allure-report-qa-all` jobs. A comment with links to the reports are added to the merge request.
+the `e2e-test-report` job. A comment with links to the reports is added to the merge request.
Errors can be found in the `gitlab-review-apps` Sentry project and [filterable by review app URL](https://sentry.gitlab.net/gitlab/gitlab-review-apps/?query=url%3A%22https%3A%2F%2Fgitlab-review-require-ve-u92nn2.gitlab-review.app%2F%22) or [commit SHA](https://sentry.gitlab.net/gitlab/gitlab-review-apps/releases/6095b501da7/all-events/).
@@ -129,31 +134,28 @@ resource.labels.pod_name:"review-qa-raise-e-12chm0-migrations"
```mermaid
graph TD
A["build-qa-image, compile-production-assets<br/>(canonical default refs only)"];
+ B1[start-review-app-pipeline];
B[review-build-cng];
- C[review-deploy];
+ C["review-deploy<br><br>Helm deploys the review app using the Cloud<br/>Native images built by the CNG-mirror pipeline.<br><br>Cloud Native images are deployed to the `review-apps`<br>Kubernetes (GKE) cluster, in the GCP `gitlab-review-apps` project."];
D[CNG-mirror];
- E[review-qa-smoke, review-qa-reliable];
+ E[review-qa-smoke, review-qa-blocking, review-qa-non-blocking<br><br>gitlab-qa runs the e2e tests against the review app.];
- A -->|once the `prepare` stage is done| B
- B -.->|triggers a CNG-mirror pipeline and wait for it to be done| D
- D -.->|polls until completed| B
- B -->|once the `review-build-cng` job is done| C
- C -->|once the `review-deploy` job is done| E
+ A --> B1
+ B1 --> B
+ B -.->|triggers a CNG-mirror pipeline| D
+ D -.->|depends on the multi-project pipeline| B
+ B --> C
+ C --> E
-subgraph "1. gitlab `prepare` stage"
+subgraph "1. gitlab-org/gitlab parent pipeline"
A
+ B1
end
-subgraph "2. gitlab `review-prepare` stage"
+subgraph "2. gitlab-org/gitlab child pipeline"
B
- end
-
-subgraph "3. gitlab `review` stage"
- C["review-deploy<br><br>Helm deploys the review app using the Cloud<br/>Native images built by the CNG-mirror pipeline.<br><br>Cloud Native images are deployed to the `review-apps`<br>Kubernetes (GKE) cluster, in the GCP `gitlab-review-apps` project."]
- end
-
-subgraph "4. gitlab `qa` stage"
- E[review-qa-smoke, review-qa-reliable<br><br>gitlab-qa runs the smoke and reliable suites against the review app.]
+ C
+ E
end
subgraph "CNG-mirror pipeline"