diff options
Diffstat (limited to 'doc/development/testing_guide/end_to_end')
5 files changed, 77 insertions, 38 deletions
diff --git a/doc/development/testing_guide/end_to_end/best_practices.md b/doc/development/testing_guide/end_to_end/best_practices.md index 00b843ffdbe..bfda94b1f1d 100644 --- a/doc/development/testing_guide/end_to_end/best_practices.md +++ b/doc/development/testing_guide/end_to_end/best_practices.md @@ -415,8 +415,8 @@ except(page).to have_no_text('hidden') Unfortunately, that's not automatically the case for the predicate methods that we add to our [page objects](page_objects.md). We need to [create our own negatable matchers](https://relishapp.com/rspec/rspec-expectations/v/3-9/docs/custom-matchers/define-a-custom-matcher#matcher-with-separate-logic-for-expect().to-and-expect().not-to). -The initial example uses the `have_job` matcher which is derived from the [`has_job?` predicate -method of the `Page::Project::Pipeline::Show` page object](https://gitlab.com/gitlab-org/gitlab/-/blob/87864b3047c23b4308f59c27a3757045944af447/qa/qa/page/project/pipeline/show.rb#L53). +The initial example uses the `have_job` matcher which is derived from the +[`has_job?` predicate method of the `Page::Project::Pipeline::Show` page object](https://gitlab.com/gitlab-org/gitlab/-/blob/87864b3047c23b4308f59c27a3757045944af447/qa/qa/page/project/pipeline/show.rb#L53). To create a negatable matcher, we use `has_no_job?` for the negative case: ```ruby diff --git a/doc/development/testing_guide/end_to_end/feature_flags.md b/doc/development/testing_guide/end_to_end/feature_flags.md index cb4c8e8a6e8..33f73304a26 100644 --- a/doc/development/testing_guide/end_to_end/feature_flags.md +++ b/doc/development/testing_guide/end_to_end/feature_flags.md @@ -217,8 +217,8 @@ If enabling the feature flag results in E2E test failures, you can browse the ar If an end-to-end test enables a feature flag, the end-to-end test suite can be used to test changes in a merge request by running the `package-and-qa` job in the merge request pipeline. If the feature flag and relevant changes have already been merged, you can confirm that the tests -pass on the default branch. The end-to-end tests run on the default branch every two hours, and the results are posted to a [Test -Session Report, which is available in the testcase-sessions project](https://gitlab.com/gitlab-org/quality/testcase-sessions/-/issues?label_name%5B%5D=found%3Amain). +pass on the default branch. The end-to-end tests run on the default branch every two hours, and the results are posted to a +[Test Session Report, which is available in the testcase-sessions project](https://gitlab.com/gitlab-org/quality/testcase-sessions/-/issues?label_name%5B%5D=found%3Amain). If the relevant tests do not enable the feature flag themselves, you can check if the tests will need to be updated by opening a draft merge request that enables the flag by default via a [feature flag definition file](../../feature_flags/index.md#feature-flag-definition-and-validation). diff --git a/doc/development/testing_guide/end_to_end/index.md b/doc/development/testing_guide/end_to_end/index.md index 06359d612ad..989d090d581 100644 --- a/doc/development/testing_guide/end_to_end/index.md +++ b/doc/development/testing_guide/end_to_end/index.md @@ -140,8 +140,8 @@ a flaky test we first want to make sure that it's no longer flaky. We can do that using the `ce:custom-parallel` and `ee:custom-parallel` jobs. Both are manual jobs that you can configure using custom variables. When clicking the name (not the play icon) of one of the parallel jobs, -you are prompted to enter variables. You can use any of [the variables -that can be used with `gitlab-qa`](https://gitlab.com/gitlab-org/gitlab-qa/blob/master/docs/what_tests_can_be_run.md#supported-gitlab-environment-variables) +you are prompted to enter variables. You can use any of +[the variables that can be used with `gitlab-qa`](https://gitlab.com/gitlab-org/gitlab-qa/blob/master/docs/what_tests_can_be_run.md#supported-gitlab-environment-variables) as well as these: | Variable | Description | @@ -150,8 +150,9 @@ as well as these: | `QA_TESTS` | The tests to run (no default, which means run all the tests in the scenario). Use file paths as you would when running tests via RSpec, for example, `qa/specs/features/ee/browser_ui` would include all the `EE` UI tests. | | `QA_RSPEC_TAGS` | The RSpec tags to add (no default) | -For now, [manual jobs with custom variables don't use the same variable -when retried](https://gitlab.com/gitlab-org/gitlab/-/issues/31367), so if you want to run the same tests multiple times, +For now, +[manual jobs with custom variables don't use the same variable when retried](https://gitlab.com/gitlab-org/gitlab/-/issues/31367), +so if you want to run the same tests multiple times, specify the same variables in each `custom-parallel` job (up to as many of the 10 available jobs that you want to run). @@ -164,8 +165,8 @@ automatically started: it runs the QA smoke suite against the You can also manually start the `review-qa-all`: it runs the full QA suite against the [Review App](../review_apps.md). -**This runs end-to-end tests against a Review App based on [the official GitLab -Helm chart](https://gitlab.com/gitlab-org/charts/gitlab/), itself deployed with custom +**This runs end-to-end tests against a Review App based on +[the official GitLab Helm chart](https://gitlab.com/gitlab-org/charts/gitlab/), itself deployed with custom [Cloud Native components](https://gitlab.com/gitlab-org/build/CNG) built from your merge request's changes.** See [Review Apps](../review_apps.md) for more details about Review Apps. @@ -235,7 +236,7 @@ Each type of scheduled pipeline generates a static link for the latest test repo - [`staging-sanity`](https://storage.googleapis.com/gitlab-qa-allure-reports/staging-sanity/master/index.html) - [`staging-sanity-no-admin`](https://storage.googleapis.com/gitlab-qa-allure-reports/staging-sanity-no-admin/master/index.html) - [`canary-sanity`](https://storage.googleapis.com/gitlab-qa-allure-reports/canary-sanity/master/index.html) -- [`production`](https://storage.googleapis.com/gitlab-qa-allure-reports/production/master/index.html) +- [`production`](https://storage.googleapis.com/gitlab-qa-allure-reports/production-full/master/index.html) - [`production-sanity`](https://storage.googleapis.com/gitlab-qa-allure-reports/production-sanity/master/index.html) ## How do I run the tests? @@ -243,8 +244,8 @@ Each type of scheduled pipeline generates a static link for the latest test repo If you are not [testing code in a merge request](#testing-code-in-merge-requests), there are two main options for running the tests. If you want to run the existing tests against a live GitLab instance or against a pre-built Docker image, -use the [GitLab QA orchestrator](https://gitlab.com/gitlab-org/gitlab-qa/tree/master/README.md). See also [examples -of the test scenarios you can run via the orchestrator](https://gitlab.com/gitlab-org/gitlab-qa/blob/master/docs/what_tests_can_be_run.md#examples). +use the [GitLab QA orchestrator](https://gitlab.com/gitlab-org/gitlab-qa/tree/master/README.md). See also +[examples of the test scenarios you can run via the orchestrator](https://gitlab.com/gitlab-org/gitlab-qa/blob/master/docs/what_tests_can_be_run.md#examples). On the other hand, if you would like to run against a local development GitLab environment, you can use the [GitLab Development Kit (GDK)](https://gitlab.com/gitlab-org/gitlab-development-kit/). @@ -262,8 +263,8 @@ architecture. See the [documentation about it](https://gitlab.com/gitlab-org/git Once you decided where to put [test environment orchestration scenarios](https://gitlab.com/gitlab-org/gitlab-qa/tree/master/lib/gitlab/qa/scenario) and [instance-level scenarios](https://gitlab.com/gitlab-org/gitlab-foss/tree/master/qa/qa/specs/features), take a look at the [GitLab QA README](https://gitlab.com/gitlab-org/gitlab/-/tree/master/qa/README.md), -the [GitLab QA orchestrator README](https://gitlab.com/gitlab-org/gitlab-qa/tree/master/README.md), and [the already existing -instance-level scenarios](https://gitlab.com/gitlab-org/gitlab-foss/tree/master/qa/qa/specs/features). +the [GitLab QA orchestrator README](https://gitlab.com/gitlab-org/gitlab-qa/tree/master/README.md), +and [the already existing instance-level scenarios](https://gitlab.com/gitlab-org/gitlab-foss/tree/master/qa/qa/specs/features). ### Consider **not** writing an end-to-end test diff --git a/doc/development/testing_guide/end_to_end/rspec_metadata_tests.md b/doc/development/testing_guide/end_to_end/rspec_metadata_tests.md index 591d03db7b8..322f2412e5b 100644 --- a/doc/development/testing_guide/end_to_end/rspec_metadata_tests.md +++ b/doc/development/testing_guide/end_to_end/rspec_metadata_tests.md @@ -30,6 +30,7 @@ This is a partial list of the [RSpec metadata](https://relishapp.com/rspec/rspec | `:ldap_no_tls` | The test requires a GitLab instance to be configured to use an external LDAP server with TLS not enabled. | | `:ldap_tls` | The test requires a GitLab instance to be configured to use an external LDAP server with TLS enabled. | | `:mattermost` | The test requires a GitLab Mattermost service on the GitLab instance. | +| `:metrics` | The test requires a GitLab instance where [dedicated metrics exporters](../../../administration/monitoring/prometheus/web_exporter.md) are running alongside Puma and Sidekiq. | | `:mixed_env` | The test should only be executed in environments that have a paired canary version available through traffic routing based on the existence of the `gitlab_canary=true` cookie. Tests in this category are switching the cookie mid-test to validate mixed deployment environments. | | `:object_storage` | The test requires a GitLab instance to be configured to use multiple [object storage types](../../../administration/object_storage.md). Uses MinIO as the object storage server. | | `:only` | The test is only to be run in specific execution contexts. See [test execution context selection](execution_context_selection.md) for more information. | diff --git a/doc/development/testing_guide/end_to_end/running_tests_that_require_special_setup.md b/doc/development/testing_guide/end_to_end/running_tests_that_require_special_setup.md index 438294161ac..322f108783f 100644 --- a/doc/development/testing_guide/end_to_end/running_tests_that_require_special_setup.md +++ b/doc/development/testing_guide/end_to_end/running_tests_that_require_special_setup.md @@ -8,11 +8,20 @@ info: To determine the technical writer assigned to the Stage/Group associated w ## Jenkins spec -The [`jenkins_build_status_spec`](https://gitlab.com/gitlab-org/gitlab/-/blob/163c8a8c814db26d11e104d1cb2dcf02eb567dbe/qa/qa/specs/features/ee/browser_ui/3_create/jenkins/jenkins_build_status_spec.rb) spins up a Jenkins instance in a Docker container based on an image stored in the [GitLab-QA container registry](https://gitlab.com/gitlab-org/gitlab-qa/container_registry). -The Docker image it uses is preconfigured with some base data and plugins. -The test then configures the GitLab plugin in Jenkins with a URL of the GitLab instance that are used -to run the tests. Unfortunately, the GitLab Jenkins plugin does not accept ports so `http://localhost:3000` would -not be accepted. Therefore, this requires us to run GitLab on port 80 or inside a Docker container. +The [`jenkins_build_status_spec`](https://gitlab.com/gitlab-org/gitlab/-/blob/24a86debf49f3aed6f2ecfd6e8f9233b3a214181/qa/qa/specs/features/browser_ui/3_create/jenkins/jenkins_build_status_spec.rb) +spins up a Jenkins instance in a Docker container with the Jenkins GitLab plugin pre-installed. Due to a license restriction we are unable to distribute this image. +To build a QA compatible image, please visit the [third party images project](https://gitlab.com/gitlab-org/quality/third-party-docker-public), where third party Dockerfiles can be found. +The project also has instructions for forking and building the images automatically in CI. + +Some extra environment variables for the location of the forked repository are also needed. + +- `QA_THIRD_PARTY_DOCKER_REGISTRY` (the container registry where the repository/images are hosted, eg `registry.gitlab.com`) +- `QA_THIRD_PARTY_DOCKER_REPOSITORY` (the base repository path where the images are hosted, eg `registry.gitlab.com/<project path>`) +- `QA_THIRD_PARTY_DOCKER_USER` (a username that has access to the container registry for this repository) +- `QA_THIRD_PARTY_DOCKER_PASSWORD` (a password/token for the username to authenticate with) + +The test configures the GitLab plugin in Jenkins with a URL of the GitLab instance that are used +to run the tests. Bi-directional networking is needed between a GitLab instance and Jenkins, so GitLab can also be started in a Docker container. To start a Docker container for GitLab based on the nightly image: @@ -21,34 +30,25 @@ docker run \ --publish 80:80 \ --name gitlab \ --hostname localhost \ + --network test gitlab/gitlab-ee:nightly ``` To run the tests from the `/qa` directory: ```shell -WEBDRIVER_HEADLESS=false bin/qa Test::Instance::All http://localhost -- qa/specs/features/ee/browser_ui/3_create/jenkins/jenkins_build_status_spec.rb +export QA_THIRD_PARTY_DOCKER_REGISTRY=<registry> +export QA_THIRD_PARTY_DOCKER_REPOSITORY=<repository> +export QA_THIRD_PARTY_DOCKER_USER=<user with registry access> +export QA_THIRD_PARTY_DOCKER_PASSWORD=<password for user> +export WEBDRIVER_HEADLESS=0 +bin/qa Test::Instance::All http://localhost -- qa/specs/features/ee/browser_ui/3_create/jenkins/jenkins_build_status_spec.rb ``` The test automatically spins up a Docker container for Jenkins and tear down once the test completes. -However, if you need to run Jenkins manually outside of the tests, use this command: - -```shell -docker run \ - --hostname localhost \ - --name jenkins-server \ - --env JENKINS_HOME=jenkins_home \ - --publish 8080:8080 \ - registry.gitlab.com/gitlab-org/gitlab-qa/jenkins-gitlab:version1 -``` - -Jenkins is available on `http://localhost:8080`. - -Administrator username is `admin` and password is `password`. - -It is worth noting that this is not an orchestrated test. It is [tagged with the `:orchestrated` meta](https://gitlab.com/gitlab-org/gitlab/-/blob/163c8a8c814db26d11e104d1cb2dcf02eb567dbe/qa/qa/specs/features/ee/browser_ui/3_create/jenkins/jenkins_build_status_spec.rb#L5) -only to prevent it from running in the pipelines for live environments such as Staging. +If you need to run Jenkins manually outside of the tests, please refer to the README for the +[third party images project](https://gitlab.com/gitlab-org/quality/third-party-docker-public/-/blob/main/jenkins/README.md) ### Troubleshooting @@ -385,6 +385,43 @@ To run the LDAP tests on your local with TLS disabled, follow these steps: GITLAB_LDAP_USERNAME="tanuki" GITLAB_LDAP_PASSWORD="password" QA_LOG_LEVEL=debug WEBDRIVER_HEADLESS=false bin/qa Test::Instance::All http://localhost qa/specs/features/browser_ui/1_manage/login/log_into_gitlab_via_ldap_spec.rb ``` +## SMTP tests + +Tests that are tagged with `:smtp` meta tag are orchestrated tests that ensure email notifications are received by a user. + +These tests require a GitLab instance with SMTP enabled and integrated with an SMTP server, [MailHog](https://github.com/mailhog/MailHog). + +To run these tests locally against the GDK: + +1. Add these settings to your `gitlab.yml` file: + + ```yaml + smtp: + enabled: true + address: "mailhog.test" + port: 1025 + ``` + +1. Start MailHog in a Docker container: + + ```shell + docker network create test && docker run \ + --network test \ + --hostname mailhog.test \ + --name mailhog \ + --publish 1025:1025 \ + --publish 8025:8025 \ + mailhog/mailhog:v1.0.0 + ``` + +1. Run the test from [`gitlab/qa`](https://gitlab.com/gitlab-org/gitlab/-/tree/d5447ebb5f99d4c72780681ddf4dc25b0738acba/qa) directory: + + ```shell + QA_LOG_LEVEL=debug WEBDRIVER_HEADLESS=false bin/qa Test::Instance::All http://localhost:3000 qa/specs/features/browser_ui/2_plan/email/trigger_email_notification_spec.rb -- --tag orchestrated + ``` + +For instructions on how to run these tests using the `gitlab-qa` gem, please refer to [the GitLab QA documentation](https://gitlab.com/gitlab-org/gitlab-qa/-/blob/master/docs/what_tests_can_be_run.md#testintegrationsmtp-ceeefull-image-address). + ## Guide to the mobile suite ### What are mobile tests |