diff options
Diffstat (limited to 'doc/development/testing_guide/end_to_end/index.md')
| -rw-r--r-- | doc/development/testing_guide/end_to_end/index.md | 96 |
1 files changed, 54 insertions, 42 deletions
diff --git a/doc/development/testing_guide/end_to_end/index.md b/doc/development/testing_guide/end_to_end/index.md index d981f9bcbba..e6da4771e55 100644 --- a/doc/development/testing_guide/end_to_end/index.md +++ b/doc/development/testing_guide/end_to_end/index.md @@ -22,13 +22,13 @@ a black-box testing framework for the API and the UI. ### Testing nightly builds We run scheduled pipelines each night to test nightly builds created by Omnibus. -You can find these nightly pipelines at `https://gitlab.com/gitlab-org/quality/nightly/pipelines` +You can find these pipelines at <https://gitlab.com/gitlab-org/quality/nightly/pipelines> (need Developer access permissions). Results are reported in the `#qa-nightly` Slack channel. ### Testing staging We run scheduled pipelines each night to test staging. -You can find these nightly pipelines at `https://gitlab.com/gitlab-org/quality/staging/pipelines` +You can find these pipelines at <https://gitlab.com/gitlab-org/quality/staging/pipelines> (need Developer access permissions). Results are reported in the `#qa-staging` Slack channel. ### Testing code in merge requests @@ -36,64 +36,76 @@ You can find these nightly pipelines at `https://gitlab.com/gitlab-org/quality/s #### Using the `package-and-qa` job It is possible to run end-to-end tests for a merge request, eventually being run in -a pipeline in the [`gitlab-qa-mirror`](https://gitlab.com/gitlab-org/gitlab-qa-mirror/) project, -by triggering the `package-and-qa` manual action in the `test` stage (not +a pipeline in the [`gitlab-org/gitlab-qa-mirror`](https://gitlab.com/gitlab-org/gitlab-qa-mirror) project, +by triggering the `package-and-qa` manual action in the `qa` stage (not available for forks). -**This runs end-to-end tests against a custom CE and EE (with an Ultimate license) -Omnibus package built from your merge request's changes.** +**This runs end-to-end tests against a custom EE (with an Ultimate license) +Docker image built from your merge request's changes.** -Manual action that starts end-to-end tests is also available in merge requests -in [Omnibus GitLab](https://gitlab.com/gitlab-org/omnibus-gitlab). - -Below you can read more about how to use it and how does it work. +Manual action that starts end-to-end tests is also available +in [`gitlab-org/omnibus-gitlab` merge requests](https://docs.gitlab.com/omnibus/build/team_member_docs.html#i-have-an-mr-in-the-omnibus-gitlab-project-and-want-a-package-or-docker-image-to-test-it). #### How does it work? -Currently, we are using _multi-project pipeline_-like approach to run QA +Currently, we are using _multi-project pipeline_-like approach to run end-to-end pipelines. ```mermaid -graph LR - A1 -.->|1. Triggers an omnibus-gitlab-mirror pipeline and wait for it to be done| A2 - B2[`Trigger-qa` stage<br>`Trigger:qa-test` job] -.->|2. Triggers a gitlab-qa-mirror pipeline and wait for it to be done| A3 - -subgraph "gitlab-foss/gitlab pipeline" - A1[`test` stage<br>`package-and-qa` job] +graph TB + A1 -.->|once done, can be triggered| A2 + A2 -.->|1. Triggers an `omnibus-gitlab-mirror` pipeline<br>and wait for it to be done| B1 + B2[`Trigger-qa` stage<br>`Trigger:qa-test` job] -.->|2. Triggers a `gitlab-qa-mirror` pipeline<br>and wait for it to be done| C1 + +subgraph "`gitlab-org/gitlab` pipeline" + A1[`build-images` stage<br>`build-qa-image` and `build-assets-image` jobs] + A2[`qa` stage<br>`package-and-qa` job] end -subgraph "omnibus-gitlab pipeline" - A2[`Trigger-docker` stage<br>`Trigger:gitlab-docker` job] -->|once done| B2 +subgraph "`gitlab-org/build/omnibus-gitlab-mirror` pipeline" + B1[`Trigger-docker` stage<br>`Trigger:gitlab-docker` job] -->|once done| B2 end -subgraph "gitlab-qa-mirror pipeline" - A3>QA jobs run] -.->|3. Reports back the pipeline result to the `package-and-qa` job<br>and post the result on the original commit tested| A1 +subgraph "`gitlab-org/gitlab-qa-mirror` pipeline" + C1>End-to-end jobs run] end ``` -1. Developer triggers a manual action, that can be found in GitLab merge - requests. This starts a chain of pipelines in multiple projects. - -1. The script being executed triggers a pipeline in - [Omnibus GitLab Mirror](https://gitlab.com/gitlab-org/build/omnibus-gitlab-mirror) - and waits for the resulting status. We call this a _status attribution_. - -1. GitLab packages are being built in the [Omnibus GitLab Mirror](https://gitlab.com/gitlab-org/build/omnibus-gitlab-mirror) - pipeline. Packages are then pushed to its Container Registry. +1. In the [`gitlab-org/gitlab` pipeline](https://gitlab.com/gitlab-org/gitlab): + 1. Developer triggers the `package-and-qa` manual action (available once the `build-qa-image` and + `build-assets-image` jobs are done), that can be found in GitLab merge + requests. This starts a chain of pipelines in multiple projects. + 1. The script being executed triggers a pipeline in + [`gitlab-org/build/omnibus-gitlab-mirror`](https://gitlab.com/gitlab-org/build/omnibus-gitlab-mirror) + and polls for the resulting status. We call this a _status attribution_. -1. When packages are ready, and available in the registry, a final step in the - [Omnibus GitLab Mirror](https://gitlab.com/gitlab-org/build/omnibus-gitlab-mirror) pipeline, triggers a new - GitLab QA pipeline (those with access can view them at `https://gitlab.com/gitlab-org/gitlab-qa-mirror/pipelines`). It also waits for a resulting status. +1. In the [`gitlab-org/build/omnibus-gitlab-mirror` pipeline](https://gitlab.com/gitlab-org/build/omnibus-gitlab-mirror): + 1. Docker image is being built and pushed to its Container Registry. + 1. Finally, the `Trigger:qa-test` job triggers a new end-to-end pipeline in + [`gitlab-org/gitlab-qa-mirror`](https://gitlab.com/gitlab-org/gitlab-qa-mirror/pipelines) and polls for the resulting status. -1. GitLab QA pulls images from the registry, spins-up containers and runs tests - against a test environment that has been just orchestrated by the `gitlab-qa` - tool. +1. In the [`gitlab-org/gitlab-qa-mirror` pipeline](https://gitlab.com/gitlab-org/gitlab-qa-mirror): + 1. Container for the Docker image stored in the [`gitlab-org/build/omnibus-gitlab-mirror`](https://gitlab.com/gitlab-org/build/omnibus-gitlab-mirror) registry is spun-up. + 1. End-to-end tests are run with the `gitlab-qa` executable, which spin up a container for the end-to-end image from the [`gitlab-org/gitlab`](https://gitlab.com/gitlab-org/gitlab) registry. -1. The result of the GitLab QA pipeline is being - propagated upstream, through Omnibus, back to the GitLab merge request. +1. The result of the [`gitlab-org/gitlab-qa-mirror` pipeline](https://gitlab.com/gitlab-org/gitlab-qa-mirror) is being + propagated upstream (through polling from upstream pipelines), through [`gitlab-org/build/omnibus-gitlab-mirror`](https://gitlab.com/gitlab-org/build/omnibus-gitlab-mirror), back to the [`gitlab-org/gitlab`](https://gitlab.com/gitlab-org/gitlab) merge request. Please note, we plan to [add more specific information](https://gitlab.com/gitlab-org/quality/team-tasks/-/issues/156) -about the tests included in each job/scenario that runs in `gitlab-qa-mirror`. +about the tests included in each job/scenario that runs in `gitlab-org/gitlab-qa-mirror`. + +NOTE: +You may have noticed that we use `gitlab-org/build/omnibus-gitlab-mirror` instead of +`gitlab-org/omnibus-gitlab`, and `gitlab-org/gitlab-qa-mirror` instead of `gitlab-org/gitlab-qa`. +This is due to technical limitations in the GitLab permission model: the ability to run a pipeline +against a protected branch is controlled by the ability to push/merge to this branch. +This means that for developers to be able to trigger a pipeline for the default branch in +`gitlab-org/omnibus-gitlab`/`gitlab-org/gitlab-qa`, they would need to have Maintainer permission in those projects. +For security reasons and implications, we couldn't open up the default branch to all the Developers. +Hence we created these mirrors where Developers and Maintainers are allowed to push/merge to the default branch. +This problem was discovered in <https://gitlab.com/gitlab-org/gitlab-qa/-/issues/63#note_107175160> and the "mirror" +work-around was suggested in <https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/4717>. +A feature proposal to segregate access control regarding running pipelines from ability to push/merge was also created at <https://gitlab.com/gitlab-org/gitlab/-/issues/24585>. #### With Pipeline for Merged Results @@ -160,9 +172,9 @@ See [Review Apps](../review_apps.md) for more details about Review Apps. ## How do I run the tests? If you are not [testing code in a merge request](#testing-code-in-merge-requests), -there are two main options for running the tests. If you simply want to run -the existing tests against a live GitLab instance or against a pre-built Docker image -you can use the [GitLab QA orchestrator](https://gitlab.com/gitlab-org/gitlab-qa/tree/master/README.md). See also [examples +there are two main options for running the tests. If you want to run +the existing tests against a live GitLab instance or against a pre-built Docker image, +use the [GitLab QA orchestrator](https://gitlab.com/gitlab-org/gitlab-qa/tree/master/README.md). See also [examples of the test scenarios you can run via the orchestrator](https://gitlab.com/gitlab-org/gitlab-qa/blob/master/docs/what_tests_can_be_run.md#examples). On the other hand, if you would like to run against a local development GitLab |