diff options
Diffstat (limited to 'doc/development/testing_guide/end_to_end')
6 files changed, 141 insertions, 77 deletions
diff --git a/doc/development/testing_guide/end_to_end/environment_selection.md b/doc/development/testing_guide/end_to_end/environment_selection.md index bcdf0e104dd..2192d9c4ed4 100644 --- a/doc/development/testing_guide/end_to_end/environment_selection.md +++ b/doc/development/testing_guide/end_to_end/environment_selection.md @@ -1,68 +1,8 @@ --- -stage: none -group: unassigned -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +redirect_to: 'execution_context_selection.md' --- -# Environment selection +This file was moved to [another location](execution_context_selection.md). -Some tests are designed to be run against specific environments or [pipelines](https://about.gitlab.com/handbook/engineering/quality/guidelines/debugging-qa-test-failures/#scheduled-qa-test-pipelines). -We can specify what environments or pipelines to run tests against using the `only` metadata. - -## Available switches - -| Switch | Function | Type | -| -------| ------- | ----- | -| `tld` | Set the top-level domain matcher | `String` | -| `subdomain` | Set the subdomain matcher | `Array` or `String` | -| `domain` | Set the domain matcher | `String` | -| `production` | Match against production | `Static` | -| `pipeline` | Match against a pipeline | `Array` or `Static`| - -WARNING: -You cannot specify `:production` and `{ <switch>: 'value' }` simultaneously. -These options are mutually exclusive. If you want to specify production, you -can control the `tld` and `domain` independently. - -## Examples - -| Environment or pipeline | Key | Matches (regex for environments, string matching for pipelines) | -| ---------------- | --- | --------------- | -| `any` | `` | `.+.com` | -| `gitlab.com` | `only: :production` | `gitlab.com` | -| `staging.gitlab.com` | `only: { subdomain: :staging }` | `(staging).+.com` | -| `gitlab.com and staging.gitlab.com` | `only: { subdomain: /(staging.)?/, domain: 'gitlab' }` | `(staging.)?gitlab.com` | -| `dev.gitlab.org` | `only: { tld: '.org', domain: 'gitlab', subdomain: 'dev' }` | `(dev).gitlab.org` | -| `staging.gitlab.com & domain.gitlab.com` | `only: { subdomain: %i[staging domain] }` | `(staging|domain).+.com` | -| `nightly` | `only: { pipeline: :nightly }` | "nightly" | -| `nightly`, `canary` | `only: { pipeline: [:nightly, :canary] }` | ["nightly"](https://gitlab.com/gitlab-org/quality/nightly) and ["canary"](https://gitlab.com/gitlab-org/quality/canary) | - -```ruby -RSpec.describe 'Area' do - it 'runs in any environment or pipeline' do; end - - it 'runs only in production environment', only: :production do; end - - it 'runs only in staging environment', only: { subdomain: :staging } do; end - - it 'runs in dev environment', only: { tld: '.org', domain: 'gitlab', subdomain: 'dev' } do; end - - it 'runs in prod and staging environments', only: { subdomain: /(staging.)?/, domain: 'gitlab' } {} - - it 'runs only in nightly pipeline', only: { pipeline: :nightly } do; end - - it 'runs in nightly and canary pipelines', only: { pipeline: [:nightly, :canary] } do; end -end -``` - -If the test has a `before` or `after`, you must add the `only` metadata -to the outer `RSpec.describe`. - -If you want to run an `only: { :pipeline }` tagged test on your local GDK make sure either the `CI_PROJECT_NAME` CI/CD variable is unset, or that the `CI_PROJECT_NAME` variable matches the specified pipeline in the `only: { :pipeline }` tag, or just delete the `only: { :pipeline }` tag. - -## Quarantining a test for a specific environment - -Similarly to specifying that a test should only run against a specific environment, it's also possible to quarantine a -test only when it runs against a specific environment. The syntax is exactly the same, except that the `only: { ... }` -hash is nested in the [`quarantine: { ... }`](https://about.gitlab.com/handbook/engineering/quality/guidelines/debugging-qa-test-failures/#quarantining-tests) hash. -For instance, `quarantine: { only: { subdomain: :staging } }` only quarantines the test when run against staging. +<!-- This redirect file can be deleted after <2021-08-14>. --> +<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/#move-or-rename-a-page --> diff --git a/doc/development/testing_guide/end_to_end/execution_context_selection.md b/doc/development/testing_guide/end_to_end/execution_context_selection.md new file mode 100644 index 00000000000..8456cfa8314 --- /dev/null +++ b/doc/development/testing_guide/end_to_end/execution_context_selection.md @@ -0,0 +1,122 @@ +--- +stage: none +group: unassigned +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +--- + +# Execution context selection + +Some tests are designed to be run against specific environments, or in specific [pipelines](https://about.gitlab.com/handbook/engineering/quality/guidelines/debugging-qa-test-failures/#scheduled-qa-test-pipelines) or jobs. We can specify the test execution context using the `only` and `except` metadata. + +## Available switches + +| Switch | Function | Type | +| -------| ------- | ----- | +| `tld` | Set the top-level domain matcher | `String` | +| `subdomain` | Set the subdomain matcher | `Array` or `String` | +| `domain` | Set the domain matcher | `String` | +| `production` | Match the production environment | `Static` | +| `pipeline` | Match a pipeline | `Array` or `Static`| +| `job` | Match a job | `Array` or `Static`| + +WARNING: +You cannot specify `:production` and `{ <switch>: 'value' }` simultaneously. +These options are mutually exclusive. If you want to specify production, you +can control the `tld` and `domain` independently. + +## Examples + +### Only + +Run tests in only the specified context. + +Matches use: + +- Regex for environments. +- String matching for pipelines. +- Regex or string matching for jobs. + +| Test execution context | Key | Matches | +| ---------------- | --- | --------------- | +| `gitlab.com` | `only: :production` | `gitlab.com` | +| `staging.gitlab.com` | `only: { subdomain: :staging }` | `(staging).+.com` | +| `gitlab.com and staging.gitlab.com` | `only: { subdomain: /(staging.)?/, domain: 'gitlab' }` | `(staging.)?gitlab.com` | +| `dev.gitlab.org` | `only: { tld: '.org', domain: 'gitlab', subdomain: 'dev' }` | `(dev).gitlab.org` | +| `staging.gitlab.com and domain.gitlab.com` | `only: { subdomain: %i[staging domain] }` | `(staging\|domain).+.com` | +| The `nightly` pipeline | `only: { pipeline: :nightly }` | "nightly" | +| The `nightly` and `canary` pipelines | `only: { pipeline: [:nightly, :canary] }` | ["nightly"](https://gitlab.com/gitlab-org/quality/nightly) and ["canary"](https://gitlab.com/gitlab-org/quality/canary) | +| The `ee:instance` job | `only: { job: 'ee:instance' }` | The `ee:instance` job in any pipeline | +| Any `quarantine` job | `only: { job: '.*quarantine' }` | Any job ending in `quarantine` in any pipeline | + +```ruby +RSpec.describe 'Area' do + it 'runs in any environment or pipeline' do; end + it 'runs only in production environment', only: :production do; end + + it 'runs only in staging environment', only: { subdomain: :staging } do; end + + it 'runs in dev environment', only: { tld: '.org', domain: 'gitlab', subdomain: 'dev' } do; end + + it 'runs in prod and staging environments', only: { subdomain: /(staging.)?/, domain: 'gitlab' } {} + + it 'runs only in nightly pipeline', only: { pipeline: :nightly } do; end + + it 'runs in nightly and canary pipelines', only: { pipeline: [:nightly, :canary] } do; end +end +``` + +### Except + +Run tests in their typical contexts _except_ as specified. + +Matches use: + +- Regex for environments. +- String matching for pipelines. +- Regex or string matching for jobs. + +| Test execution context | Key | Matches | +| ---------------- | --- | --------------- | +| `gitlab.com` | `except: :production` | `gitlab.com` | +| `staging.gitlab.com` | `except: { subdomain: :staging }` | `(staging).+.com` | +| `gitlab.com and staging.gitlab.com` | `except: { subdomain: /(staging.)?/, domain: 'gitlab' }` | `(staging.)?gitlab.com` | +| `dev.gitlab.org` | `except: { tld: '.org', domain: 'gitlab', subdomain: 'dev' }` | `(dev).gitlab.org` | +| `staging.gitlab.com and domain.gitlab.com` | `except: { subdomain: %i[staging domain] }` | `(staging\|domain).+.com` | +| The `nightly` pipeline | `except: { pipeline: :nightly }` | ["nightly"](https://gitlab.com/gitlab-org/quality/nightly) | +| The `nightly` and `canary` pipelines | `except: { pipeline: [:nightly, :canary] }` | ["nightly"](https://gitlab.com/gitlab-org/quality/nightly) and ["canary"](https://gitlab.com/gitlab-org/quality/canary) | +| The `ee:instance` job | `except: { job: 'ee:instance' }` | The `ee:instance` job in any pipeline | +| Any `quarantine` job | `except: { job: '.*quarantine' }` | Any job ending in `quarantine` in any pipeline | + +```ruby +RSpec.describe 'Area' do + it 'runs in any execution context except the production environment', except: :production do; end + + it 'runs in any execution context except the staging environment', except: { subdomain: :staging } do; end + + it 'runs in any execution context except the nightly pipeline', except: { pipeline: :nightly } do; end + + it 'runs in any execution context except the ee:instance job', except: { job: 'ee:instance' } do; end +end +``` + +## Usage notes + +If the test has a `before` or `after` block, you must add the `only` or `except` metadata to the outer `RSpec.describe` block. + +To run a test tagged with `only` on your local GitLab instance, you can do one of the following: + +- Make sure you **do not** have the `CI_PROJECT_NAME` or `CI_JOB_NAME` environment variables set. +- Set the appropriate variable to match the metadata. For example, if the metadata is `only: { pipeline: :nightly }` then set `CI_PROJECT_NAME=nightly`. If the metadata is `only: { job: 'ee:instance' }` then set `CI_JOB_NAME=ee:instance`. +- Temporarily remove the metadata. + +To run a test tagged with `except` locally, you can either: + +- Make sure you **do not** have the `CI_PROJECT_NAME` or `CI_JOB_NAME` environment variables set. +- Temporarily remove the metadata. + +## Quarantine a test for a specific environment + +Similarly to specifying that a test should only run against a specific environment, it's also possible to quarantine a +test only when it runs against a specific environment. The syntax is exactly the same, except that the `only: { ... }` +hash is nested in the [`quarantine: { ... }`](https://about.gitlab.com/handbook/engineering/quality/guidelines/debugging-qa-test-failures/#quarantining-tests) hash. +For example, `quarantine: { only: { subdomain: :staging } }` only quarantines the test when run against `staging`. diff --git a/doc/development/testing_guide/end_to_end/index.md b/doc/development/testing_guide/end_to_end/index.md index 6ab288b0525..eca649b73a5 100644 --- a/doc/development/testing_guide/end_to_end/index.md +++ b/doc/development/testing_guide/end_to_end/index.md @@ -23,13 +23,13 @@ a black-box testing framework for the API and the UI. We run scheduled pipelines each night to test nightly builds created by Omnibus. You can find these pipelines at <https://gitlab.com/gitlab-org/quality/nightly/pipelines> -(need Developer access permissions). Results are reported in the `#qa-nightly` Slack channel. +(requires the Developer role). Results are reported in the `#qa-nightly` Slack channel. ### Testing staging We run scheduled pipelines each night to test staging. You can find these pipelines at <https://gitlab.com/gitlab-org/quality/staging/pipelines> -(need Developer access permissions). Results are reported in the `#qa-staging` Slack channel. +(requires the Developer role). Results are reported in the `#qa-staging` Slack channel. ### Testing code in merge requests @@ -100,7 +100,7 @@ You may have noticed that we use `gitlab-org/build/omnibus-gitlab-mirror` instea This is due to technical limitations in the GitLab permission model: the ability to run a pipeline against a protected branch is controlled by the ability to push/merge to this branch. This means that for developers to be able to trigger a pipeline for the default branch in -`gitlab-org/omnibus-gitlab`/`gitlab-org/gitlab-qa`, they would need to have the +`gitlab-org/omnibus-gitlab`/`gitlab-org/gitlab-qa`, they would need to have the [Maintainer role](../../../user/permissions.md) for those projects. For security reasons and implications, we couldn't open up the default branch to all the Developers. Hence we created these mirrors where Developers and Maintainers are allowed to push/merge to the default branch. @@ -212,6 +212,7 @@ Continued reading: - [Testing with feature flags](feature_flags.md) - [Flows](flows.md) - [RSpec metadata/tags](rspec_metadata_tests.md) +- [Execution context selection](execution_context_selection.md) ## Where can I ask for help? diff --git a/doc/development/testing_guide/end_to_end/resources.md b/doc/development/testing_guide/end_to_end/resources.md index b6aef123c64..0819a2f7b54 100644 --- a/doc/development/testing_guide/end_to_end/resources.md +++ b/doc/development/testing_guide/end_to_end/resources.md @@ -50,7 +50,7 @@ create the resource via the public GitLab API: - `#api_post_path`: The `POST` path to create a new resource. - `#api_post_body`: The `POST` body (as a Ruby hash) to create a new resource. -> Be aware that many API resources are [paginated](../../../api/README.md#pagination). +> Be aware that many API resources are [paginated](../../../api/index.md#pagination). > If you don't find the results you expect, check if there is more that one page of results. Let's take the `Shirt` resource class, and add these three API methods: diff --git a/doc/development/testing_guide/end_to_end/rspec_metadata_tests.md b/doc/development/testing_guide/end_to_end/rspec_metadata_tests.md index 8a929737ebe..7f541f1be3f 100644 --- a/doc/development/testing_guide/end_to_end/rspec_metadata_tests.md +++ b/doc/development/testing_guide/end_to_end/rspec_metadata_tests.md @@ -14,6 +14,7 @@ This is a partial list of the [RSpec metadata](https://relishapp.com/rspec/rspec | Tag | Description | |-----|-------------| | `:elasticsearch` | The test requires an Elasticsearch service. It is used by the [instance-level scenario](https://gitlab.com/gitlab-org/gitlab-qa#definitions) [`Test::Integration::Elasticsearch`](https://gitlab.com/gitlab-org/gitlab/-/blob/72b62b51bdf513e2936301cb6c7c91ec27c35b4d/qa/qa/ee/scenario/test/integration/elasticsearch.rb) to include only tests that require Elasticsearch. | +| `:except` | The test is to be run in their typical execution contexts _except_ as specified. See [test execution context selection](execution_context_selection.md) for more information. | | `:geo` | The test requires two GitLab Geo instances - a primary and a secondary - to be spun up. | | `:gitaly_cluster` | The test runs against a GitLab instance where repositories are stored on redundant Gitaly nodes behind a Praefect node. All nodes are [separate containers](../../../administration/gitaly/praefect.md#requirements-for-configuring-a-gitaly-cluster). Tests that use this tag have a longer setup time since there are three additional containers that need to be started. | | `:github` | The test requires a GitHub personal access token. | @@ -26,10 +27,10 @@ This is a partial list of the [RSpec metadata](https://relishapp.com/rspec/rspec | `:ldap_tls` | The test requires a GitLab instance to be configured to use an external LDAP server with TLS enabled. | | `:mattermost` | The test requires a GitLab Mattermost service on the GitLab instance. | | `:object_storage` | The test requires a GitLab instance to be configured to use multiple [object storage types](../../../administration/object_storage.md). Uses MinIO as the object storage server. | -| `:only` | The test is only to be run against specific environments or pipelines. See [Environment selection](environment_selection.md) for more information. | +| `:only` | The test is only to be run in specific execution contexts. See [test execution context selection](execution_context_selection.md) for more information. | | `:orchestrated` | The GitLab instance under test may be [configured by `gitlab-qa`](https://gitlab.com/gitlab-org/gitlab-qa/-/blob/master/docs/what_tests_can_be_run.md#orchestrated-tests) to be different to the default GitLab configuration, or `gitlab-qa` may launch additional services in separate Docker containers, or both. Tests tagged with `:orchestrated` are excluded when testing environments where we can't dynamically modify the GitLab configuration (for example, Staging). | | `:packages` | The test requires a GitLab instance that has the [Package Registry](../../../administration/packages/#gitlab-package-registry-administration) enabled. | -| `:quarantine` | The test has been [quarantined](https://about.gitlab.com/handbook/engineering/quality/guidelines/debugging-qa-test-failures/#quarantining-tests), runs in a separate job that only includes quarantined tests, and is allowed to fail. The test is skipped in its regular job so that if it fails it doesn't hold up the pipeline. Note that you can also [quarantine a test only when it runs against specific environment](environment_selection.md#quarantining-a-test-for-a-specific-environment). | +| `:quarantine` | The test has been [quarantined](https://about.gitlab.com/handbook/engineering/quality/guidelines/debugging-qa-test-failures/#quarantining-tests), runs in a separate job that only includes quarantined tests, and is allowed to fail. The test is skipped in its regular job so that if it fails it doesn't hold up the pipeline. Note that you can also [quarantine a test only when it runs in a specific context](execution_context_selection.md#quarantine-a-test-for-a-specific-environment). | | `:relative_url` | The test requires a GitLab instance to be installed under a [relative URL](../../../install/relative_url.md). | | `:reliable` | The test has been [promoted to a reliable test](https://about.gitlab.com/handbook/engineering/quality/guidelines/reliable-tests/#promoting-an-existing-test-to-reliable) meaning it passes consistently in all pipelines, including merge requests. | | `:repository_storage` | The test requires a GitLab instance to be configured to use multiple [repository storage paths](../../../administration/repository_storage_paths.md). Paired with the `:orchestrated` tag. | diff --git a/doc/development/testing_guide/end_to_end/running_tests_that_require_special_setup.md b/doc/development/testing_guide/end_to_end/running_tests_that_require_special_setup.md index 859b8f950e3..f200d6c682a 100644 --- a/doc/development/testing_guide/end_to_end/running_tests_that_require_special_setup.md +++ b/doc/development/testing_guide/end_to_end/running_tests_that_require_special_setup.md @@ -27,7 +27,7 @@ docker run \ To run the tests from the `/qa` directory: ```shell -CHROME_HEADLESS=false bin/qa Test::Instance::All http://localhost -- qa/specs/features/ee/browser_ui/3_create/jenkins/jenkins_build_status_spec.rb +WEBDRIVER_HEADLESS=false bin/qa Test::Instance::All http://localhost -- qa/specs/features/ee/browser_ui/3_create/jenkins/jenkins_build_status_spec.rb ``` The test automatically spins up a Docker container for Jenkins and tear down once the test completes. @@ -131,7 +131,7 @@ sudo nginx -s reload You could then run the tests from the `/qa` directory: ```shell -CHROME_HEADLESS=false bin/qa Test::Instance::All http://gitlab-gitaly-cluster.test -- --tag gitaly_cluster +WEBDRIVER_HEADLESS=false bin/qa Test::Instance::All http://gitlab-gitaly-cluster.test -- --tag gitaly_cluster ``` Once you have finished testing you can stop and remove the Docker containers: @@ -160,13 +160,13 @@ You might see NGINX issues when you run `gdk start` or `gdk restart`. In that ca Navigate to the folder in `/your-gdk/gitlab/qa` and issue the command: ```shell -QA_DEBUG=true CHROME_HEADLESS=false GITLAB_ADMIN_USERNAME=rootusername GITLAB_ADMIN_PASSWORD=rootpassword GITLAB_QA_ACCESS_TOKEN=your_token_here GITLAB_QA_ADMIN_ACCESS_TOKEN=your_token_here CLUSTER_API_URL=https://kubernetes.docker.internal:6443 bundle exec bin/qa Test::Instance::All https://[YOUR-PORT].qa-tunnel.gitlab.info/ -- qa/specs/features/browser_ui/8_monitor/all_monitor_core_features_spec.rb --tag kubernetes --tag orchestrated --tag requires_admin +QA_DEBUG=true WEBDRIVER_HEADLESS=false GITLAB_ADMIN_USERNAME=rootusername GITLAB_ADMIN_PASSWORD=rootpassword GITLAB_QA_ACCESS_TOKEN=your_token_here GITLAB_QA_ADMIN_ACCESS_TOKEN=your_token_here CLUSTER_API_URL=https://kubernetes.docker.internal:6443 bundle exec bin/qa Test::Instance::All https://[YOUR-PORT].qa-tunnel.gitlab.info/ -- qa/specs/features/browser_ui/8_monitor/all_monitor_core_features_spec.rb --tag kubernetes --tag orchestrated --tag requires_admin ``` The following includes more information on the command: -`QA_DEBUG` - Set to `true` to verbosely log page object actions. --`CHROME_HEADLESS` - When running locally, set to `false` to allow Chrome tests to be visible - watch your tests being run. +-`WEBDRIVER_HEADLESS` - When running locally, set to `false` to allow browser tests to be visible - watch your tests being run. -`GITLAB_ADMIN_USERNAME` - Admin username to use when adding a license. -`GITLAB_ADMIN_PASSWORD` - Admin password to use when adding a license. -`GITLAB_QA_ACCESS_TOKEN` and `GITLAB_QA_ADMIN_ACCESS_TOKEN` - A valid personal access token with the `api` scope. This is used for API access during tests, and is used in the version that staging is currently running. The `ADMIN_ACCESS_TOKEN` is from a user with admin access. Used for API access as an admin during tests. @@ -279,7 +279,7 @@ Geo end-to-end tests can run locally against a [Geo GDK setup](https://gitlab.co Run from the [`qa/` directory](https://gitlab.com/gitlab-org/gitlab/-/blob/f7272b77e80215c39d1ffeaed27794c220dbe03f/qa) with both GDK Geo primary and Geo secondary instances running: ```shell -CHROME_HEADLESS=false bundle exec bin/qa QA::EE::Scenario::Test::Geo --primary-address http://localhost:3001 --secondary-address http://localhost:3002 --without-setup +WEBDRIVER_HEADLESS=false bundle exec bin/qa QA::EE::Scenario::Test::Geo --primary-address http://localhost:3001 --secondary-address http://localhost:3002 --without-setup ``` ### Using Geo in Docker @@ -455,7 +455,7 @@ To run the LDAP tests on your local with TLS enabled, follow these steps: 1. Run an LDAP test from [`gitlab/qa`](https://gitlab.com/gitlab-org/gitlab/-/tree/d5447ebb5f99d4c72780681ddf4dc25b0738acba/qa) directory: ```shell - GITLAB_LDAP_USERNAME="tanuki" GITLAB_LDAP_PASSWORD="password" QA_DEBUG=true CHROME_HEADLESS=false bin/qa Test::Instance::All https://gitlab.test qa/specs/features/browser_ui/1_manage/login/log_into_gitlab_via_ldap_spec.rb + GITLAB_LDAP_USERNAME="tanuki" GITLAB_LDAP_PASSWORD="password" QA_DEBUG=true WEBDRIVER_HEADLESS=false bin/qa Test::Instance::All https://gitlab.test qa/specs/features/browser_ui/1_manage/login/log_into_gitlab_via_ldap_spec.rb ``` ### Running LDAP tests with TLS disabled @@ -483,5 +483,5 @@ To run the LDAP tests on your local with TLS disabled, follow these steps: 1. Run an LDAP test from [`gitlab/qa`](https://gitlab.com/gitlab-org/gitlab/-/tree/d5447ebb5f99d4c72780681ddf4dc25b0738acba/qa) directory: ```shell - GITLAB_LDAP_USERNAME="tanuki" GITLAB_LDAP_PASSWORD="password" QA_DEBUG=true CHROME_HEADLESS=false bin/qa Test::Instance::All http://localhost qa/specs/features/browser_ui/1_manage/login/log_into_gitlab_via_ldap_spec.rb + GITLAB_LDAP_USERNAME="tanuki" GITLAB_LDAP_PASSWORD="password" QA_DEBUG=true WEBDRIVER_HEADLESS=false bin/qa Test::Instance::All http://localhost qa/specs/features/browser_ui/1_manage/login/log_into_gitlab_via_ldap_spec.rb ``` |