Welcome to mirror list, hosted at ThFree Co, Russian Federation.

gitlab.com/gitlab-org/gitlab-foss.git - Unnamed repository; edit this file 'description' to name the repository.
summaryrefslogtreecommitdiff
diff options
context:
space:
mode:
authorGitLab Bot <gitlab-bot@gitlab.com>2020-11-19 11:27:35 +0300
committerGitLab Bot <gitlab-bot@gitlab.com>2020-11-19 11:27:35 +0300
commit7e9c479f7de77702622631cff2628a9c8dcbc627 (patch)
treec8f718a08e110ad7e1894510980d2155a6549197 /doc/development/testing_guide
parente852b0ae16db4052c1c567d9efa4facc81146e88 (diff)
Add latest changes from gitlab-org/gitlab@13-6-stable-eev13.6.0-rc42
Diffstat (limited to 'doc/development/testing_guide')
-rw-r--r--doc/development/testing_guide/best_practices.md8
-rw-r--r--doc/development/testing_guide/ci.md6
-rw-r--r--doc/development/testing_guide/end_to_end/beginners_guide.md9
-rw-r--r--doc/development/testing_guide/end_to_end/best_practices.md76
-rw-r--r--doc/development/testing_guide/end_to_end/dynamic_element_validation.md6
-rw-r--r--doc/development/testing_guide/end_to_end/environment_selection.md11
-rw-r--r--doc/development/testing_guide/end_to_end/feature_flags.md6
-rw-r--r--doc/development/testing_guide/end_to_end/flows.md6
-rw-r--r--doc/development/testing_guide/end_to_end/index.md6
-rw-r--r--doc/development/testing_guide/end_to_end/page_objects.md6
-rw-r--r--doc/development/testing_guide/end_to_end/resources.md6
-rw-r--r--doc/development/testing_guide/end_to_end/rspec_metadata_tests.md25
-rw-r--r--doc/development/testing_guide/end_to_end/running_tests_that_require_special_setup.md12
-rw-r--r--doc/development/testing_guide/end_to_end/style_guide.md6
-rw-r--r--doc/development/testing_guide/flaky_tests.md14
-rw-r--r--doc/development/testing_guide/frontend_testing.md119
-rw-r--r--doc/development/testing_guide/index.md6
-rw-r--r--doc/development/testing_guide/review_apps.md6
-rw-r--r--doc/development/testing_guide/smoke.md6
-rw-r--r--doc/development/testing_guide/testing_levels.md6
-rw-r--r--doc/development/testing_guide/testing_migrations_guide.md4
-rw-r--r--doc/development/testing_guide/testing_rake_tasks.md6
22 files changed, 283 insertions, 73 deletions
diff --git a/doc/development/testing_guide/best_practices.md b/doc/development/testing_guide/best_practices.md
index 2c1d70a005e..dabb18c1f75 100644
--- a/doc/development/testing_guide/best_practices.md
+++ b/doc/development/testing_guide/best_practices.md
@@ -361,7 +361,7 @@ Finished in 34.51 seconds (files took 0.76702 seconds to load)
1 example, 0 failures
```
-Note: `live_debug` only works on JavaScript enabled specs.
+`live_debug` only works on JavaScript enabled specs.
#### Run `:js` spec in a visible browser
@@ -584,9 +584,9 @@ this trait should be either fixed to not rely on Sidekiq processing jobs, or the
`:sidekiq_might_not_need_inline` trait should be updated to `:sidekiq_inline` if
the processing of background jobs is needed/expected.
-NOTE: **Note:**
-The usage of `perform_enqueued_jobs` is only useful for testing delayed mail
-deliveries since our Sidekiq workers aren't inheriting from `ApplicationJob` / `ActiveJob::Base`.
+The usage of `perform_enqueued_jobs` is useful only for testing delayed mail
+deliveries, because our Sidekiq workers aren't inheriting from `ApplicationJob`
+/ `ActiveJob::Base`.
#### DNS
diff --git a/doc/development/testing_guide/ci.md b/doc/development/testing_guide/ci.md
index 8091142410c..618f9010b4d 100644
--- a/doc/development/testing_guide/ci.md
+++ b/doc/development/testing_guide/ci.md
@@ -1,3 +1,9 @@
+---
+stage: none
+group: unassigned
+info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers
+---
+
# GitLab tests in the Continuous Integration (CI) context
## Test suite parallelization on the CI
diff --git a/doc/development/testing_guide/end_to_end/beginners_guide.md b/doc/development/testing_guide/end_to_end/beginners_guide.md
index a1883f44170..ef0bd9902e1 100644
--- a/doc/development/testing_guide/end_to_end/beginners_guide.md
+++ b/doc/development/testing_guide/end_to_end/beginners_guide.md
@@ -1,3 +1,9 @@
+---
+stage: none
+group: unassigned
+info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers
+---
+
# Beginner's guide to writing end-to-end tests
In this tutorial, you will learn about the creation of end-to-end (_e2e_) tests
@@ -52,7 +58,6 @@ Check both [GitLab Community Edition](https://gitlab-org.gitlab.io/gitlab-foss/c
for previously-written tests for this feature. For analyzing the code coverage,
you must understand which application files implement specific features.
-NOTE: **Note:**
In this tutorial we're writing a login end-to-end test, even though it has been
sufficiently covered by lower-level testing, because it's the first step for most
end-to-end flows, and is easiest to understand.
@@ -68,7 +73,6 @@ under the stage.
![DevOps lifecycle by stages](img/gl-devops-lifecycle-by-stage-numbers_V12_10.png)
-NOTE: **Note:**
If the test is Enterprise Edition only, the test will be created in the `features/ee`
directory, but follow the same DevOps lifecycle format.
@@ -204,7 +208,6 @@ end
1. Check if the user avatar appears in the top navigation.
1. Check if the user avatar *does not* appear in the top navigation.
-NOTE: **Note:**
Behind the scenes, `be_signed_in` is a
[predicate matcher](https://relishapp.com/rspec/rspec-expectations/v/3-8/docs/built-in-matchers/predicate-matchers)
that [implements checking the user avatar](https://gitlab.com/gitlab-org/gitlab/-/blob/master/qa/qa/page/main/menu.rb#L74).
diff --git a/doc/development/testing_guide/end_to_end/best_practices.md b/doc/development/testing_guide/end_to_end/best_practices.md
index 58bae749dc5..4d12a0f79cb 100644
--- a/doc/development/testing_guide/end_to_end/best_practices.md
+++ b/doc/development/testing_guide/end_to_end/best_practices.md
@@ -1,6 +1,11 @@
+---
+stage: none
+group: Development
+info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers
+---
+
# End-to-end testing Best Practices
-NOTE: **Note:**
This is a tailored extension of the Best Practices [found in the testing guide](../best_practices.md).
## Link a test to its test-case issue
@@ -263,7 +268,7 @@ We don't run tests that require Administrator access against our Production envi
When you add a new test that requires Administrator access, apply the RSpec metadata `:requires_admin` so that the test will not be included in the test suites executed against Production and other environments on which we don't want to run those tests.
-Note: When running tests locally or configuring a pipeline, the environment variable `QA_CAN_TEST_ADMIN_FEATURES` can be set to `false` to skip tests that have the `:requires_admin` tag.
+When running tests locally or configuring a pipeline, the environment variable `QA_CAN_TEST_ADMIN_FEATURES` can be set to `false` to skip tests that have the `:requires_admin` tag.
## Prefer `Commit` resource over `ProjectPush`
@@ -288,7 +293,6 @@ Resource::Repository::ProjectPush.fabricate! do |push|
end
```
-NOTE: **Note:**
A few exceptions for using a `ProjectPush` would be when your test calls for testing SSH integration or
using the Git CLI.
@@ -321,39 +325,69 @@ In general, we use an `expect` statement to check that something _is_ as we expe
```ruby
Page::Project::Pipeline::Show.perform do |pipeline|
- expect(pipeline).to have_job("a_job")
+ expect(pipeline).to have_job('a_job')
end
```
-### Ensure `expect` checks for negation efficiently
+### Create negatable matchers to speed `expect` checks
However, sometimes we want to check that something is _not_ as we _don't_ want it to be. In other
-words, we want to make sure something is absent. In such a case we should use an appropriate
-predicate method that returns quickly, rather than waiting for a state that won't appear.
+words, we want to make sure something is absent. For unit tests and feature specs,
+we commonly use `not_to`
+because RSpec's built-in matchers are negatable, as are Capybara's, which means the following two statements are
+equivalent.
+
+```ruby
+except(page).not_to have_text('hidden')
+except(page).to have_no_text('hidden')
+```
+
+Unfortunately, that's not automatically the case for the predicate methods that we add to our
+[page objects](page_objects.md). We need to [create our own negatable matchers](https://relishapp.com/rspec/rspec-expectations/v/3-9/docs/custom-matchers/define-a-custom-matcher#matcher-with-separate-logic-for-expect().to-and-expect().not-to).
+
+The initial example uses the `have_job` matcher which is derived from the [`has_job?` predicate
+method of the `Page::Project::Pipeline::Show` page object](https://gitlab.com/gitlab-org/gitlab/-/blob/87864b3047c23b4308f59c27a3757045944af447/qa/qa/page/project/pipeline/show.rb#L53).
+To create a negatable matcher, we use `has_no_job?` for the negative case:
+
+```ruby
+RSpec::Matchers.define :have_job do |job_name|
+ match do |page_object|
+ page_object.has_job?(job_name)
+ end
+
+ match_when_negated do |page_object|
+ page_object.has_no_job?(job_name)
+ end
+end
+```
-It's most efficient to use a predicate method that returns immediately when there is no job, or waits
-until it disappears:
+And then the two `expect` statements in the following example are equivalent:
```ruby
-# Good
Page::Project::Pipeline::Show.perform do |pipeline|
- expect(pipeline).to have_no_job("a_job")
+ expect(pipeline).not_to have_job('a_job')
+ expect(pipeline).to have_no_job('a_job')
end
```
-### Problematic alternatives
+[See this merge request for a real example of adding a custom matcher](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/46302).
-Alternatively, if we want to check that a job doesn't exist it might be tempting to use `not_to`:
+NOTE: **Note:**
+We need to create custom negatable matchers only for the predicate methods we've added to the test framework, and only if we're using `not_to`. If we use `to have_no_*` a negatable matcher is not necessary.
+
+### Why we need negatable matchers
+
+Consider the following code, but assume that we _don't_ have a custom negatable matcher for `have_job`.
```ruby
# Bad
Page::Project::Pipeline::Show.perform do |pipeline|
- expect(pipeline).not_to have_job("a_job")
+ expect(pipeline).not_to have_job('a_job')
end
```
-For this statement to pass, `have_job("a_job")` has to return `false` so that `not_to` can negate it.
-The problem is that `have_job("a_job")` waits up to ten seconds for `"a job"` to appear before
+For this statement to pass, `have_job('a_job')` has to return `false` so that `not_to` can negate it.
+The problem is that `have_job('a_job')` waits up to ten seconds for `'a job'` to appear before
returning `false`. Under the expected condition this test will take ten seconds longer than it needs to.
Instead, we could force no wait:
@@ -361,9 +395,13 @@ Instead, we could force no wait:
```ruby
# Not as bad but potentially flaky
Page::Project::Pipeline::Show.perform do |pipeline|
- expect(pipeline).not_to have_job("a_job", wait: 0)
+ expect(pipeline).not_to have_job('a_job', wait: 0)
end
```
-The problem is that if `"a_job"` is present and we're waiting for it to disappear, this statement
-will fail.
+The problem is that if `'a_job'` is present and we're waiting for it to disappear, this statement will fail.
+
+Neither problem is present if we create a custom negatable matcher because the `has_no_job?` predicate method
+would be used, which would wait only as long as necessary for the job to disappear.
+
+Lastly, negatable matchers are preferred over using matchers of the form `have_no_*` because it's a common and familiar practice to negate matchers using `not_to`. If we facilitate that practice by adding negatable matchers, we make it easier for subsequent test authors to write efficient tests.
diff --git a/doc/development/testing_guide/end_to_end/dynamic_element_validation.md b/doc/development/testing_guide/end_to_end/dynamic_element_validation.md
index 32b1c304a9a..871b3f80c18 100644
--- a/doc/development/testing_guide/end_to_end/dynamic_element_validation.md
+++ b/doc/development/testing_guide/end_to_end/dynamic_element_validation.md
@@ -1,3 +1,9 @@
+---
+stage: none
+group: unassigned
+info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers
+---
+
# Dynamic Element Validation
We devised a solution to solve common test automation problems such as the dreaded `NoSuchElementException`.
diff --git a/doc/development/testing_guide/end_to_end/environment_selection.md b/doc/development/testing_guide/end_to_end/environment_selection.md
index 325f251b280..f5e3e99b79e 100644
--- a/doc/development/testing_guide/end_to_end/environment_selection.md
+++ b/doc/development/testing_guide/end_to_end/environment_selection.md
@@ -1,3 +1,9 @@
+---
+stage: none
+group: unassigned
+info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers
+---
+
# Environment selection
Some tests are designed to be run against specific environments or [pipelines](https://about.gitlab.com/handbook/engineering/quality/guidelines/debugging-qa-test-failures/#scheduled-qa-test-pipelines).
@@ -42,14 +48,13 @@ RSpec.describe 'Area' do
it 'runs in dev environment', only: { tld: '.org', domain: 'gitlab', subdomain: 'dev' } do; end
it 'runs in prod and staging environments', only: { subdomain: /(staging.)?/, domain: 'gitlab' } {}
-
+
it 'runs only in nightly pipeline', only: { pipeline: :nightly } do; end
-
+
it 'runs in nightly and canary pipelines', only: { pipeline: [:nightly, :canary] } do; end
end
```
-NOTE: **Note:**
If the test has a `before` or `after`, you must add the `only` metadata
to the outer `RSpec.describe`.
diff --git a/doc/development/testing_guide/end_to_end/feature_flags.md b/doc/development/testing_guide/end_to_end/feature_flags.md
index e571774167d..2ff1c9f6dc3 100644
--- a/doc/development/testing_guide/end_to_end/feature_flags.md
+++ b/doc/development/testing_guide/end_to_end/feature_flags.md
@@ -1,3 +1,9 @@
+---
+stage: none
+group: unassigned
+info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers
+---
+
# Testing with feature flags
To run a specific test with a feature flag enabled you can use the `QA::Runtime::Feature` class to
diff --git a/doc/development/testing_guide/end_to_end/flows.md b/doc/development/testing_guide/end_to_end/flows.md
index fb1d82914aa..291d8bd5319 100644
--- a/doc/development/testing_guide/end_to_end/flows.md
+++ b/doc/development/testing_guide/end_to_end/flows.md
@@ -1,3 +1,9 @@
+---
+stage: none
+group: unassigned
+info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers
+---
+
# Flows in GitLab QA
Flows are frequently used sequences of actions. They are a higher level
diff --git a/doc/development/testing_guide/end_to_end/index.md b/doc/development/testing_guide/end_to_end/index.md
index f61eab5c8f3..1d144359ed8 100644
--- a/doc/development/testing_guide/end_to_end/index.md
+++ b/doc/development/testing_guide/end_to_end/index.md
@@ -1,3 +1,9 @@
+---
+stage: none
+group: unassigned
+info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers
+---
+
# End-to-end Testing
## What is end-to-end testing?
diff --git a/doc/development/testing_guide/end_to_end/page_objects.md b/doc/development/testing_guide/end_to_end/page_objects.md
index 6ce44b2d359..7eacaf4b08a 100644
--- a/doc/development/testing_guide/end_to_end/page_objects.md
+++ b/doc/development/testing_guide/end_to_end/page_objects.md
@@ -1,3 +1,9 @@
+---
+stage: none
+group: unassigned
+info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers
+---
+
# Page objects in GitLab QA
In GitLab QA we are using a known pattern, called _Page Objects_.
diff --git a/doc/development/testing_guide/end_to_end/resources.md b/doc/development/testing_guide/end_to_end/resources.md
index b8a093c54c6..d73bae331d5 100644
--- a/doc/development/testing_guide/end_to_end/resources.md
+++ b/doc/development/testing_guide/end_to_end/resources.md
@@ -1,3 +1,9 @@
+---
+stage: none
+group: unassigned
+info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers
+---
+
# Resource class in GitLab QA
Resources are primarily created using Browser UI steps, but can also
diff --git a/doc/development/testing_guide/end_to_end/rspec_metadata_tests.md b/doc/development/testing_guide/end_to_end/rspec_metadata_tests.md
index 3a1303d9c0c..8e99cf18ea0 100644
--- a/doc/development/testing_guide/end_to_end/rspec_metadata_tests.md
+++ b/doc/development/testing_guide/end_to_end/rspec_metadata_tests.md
@@ -1,3 +1,9 @@
+---
+stage: none
+group: unassigned
+info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers
+---
+
# RSpec metadata for end-to-end tests
This is a partial list of the [RSpec metadata](https://relishapp.com/rspec/rspec-core/docs/metadata/user-defined-metadata)
@@ -10,7 +16,7 @@ This is a partial list of the [RSpec metadata](https://relishapp.com/rspec/rspec
| `:elasticsearch` | The test requires an Elasticsearch service. It is used by the [instance-level scenario](https://gitlab.com/gitlab-org/gitlab-qa#definitions) [`Test::Integration::Elasticsearch`](https://gitlab.com/gitlab-org/gitlab/-/blob/72b62b51bdf513e2936301cb6c7c91ec27c35b4d/qa/qa/ee/scenario/test/integration/elasticsearch.rb) to include only tests that require Elasticsearch. |
| `:gitaly_cluster` | The test will run against a GitLab instance where repositories are stored on redundant Gitaly nodes behind a Praefect node. All nodes are [separate containers](../../../administration/gitaly/praefect.md#requirements-for-configuring-a-gitaly-cluster). Tests that use this tag have a longer setup time since there are three additional containers that need to be started. |
| `:jira` | The test requires a Jira Server. [GitLab-QA](https://gitlab.com/gitlab-org/gitlab-qa) will provision the Jira Server in a Docker container when the `Test::Integration::Jira` test scenario is run.
-| `:kubernetes` | The test includes a GitLab instance that is configured to be run behind an SSH tunnel, allowing a TLS-accessible GitLab. This test will also include provisioning of at least one Kubernetes cluster to test against. *This tag is often be paired with `:orchestrated`.* |
+| `:kubernetes` | The test includes a GitLab instance that is configured to be run behind an SSH tunnel, allowing a TLS-accessible GitLab. This test will also include provisioning of at least one Kubernetes cluster to test against. _This tag is often be paired with `:orchestrated`._ |
| `:only` | The test is only to be run against specific environments or pipelines. See [Environment selection](environment_selection.md) for more information. |
| `:orchestrated` | The GitLab instance under test may be [configured by `gitlab-qa`](https://gitlab.com/gitlab-org/gitlab-qa/-/blob/master/docs/what_tests_can_be_run.md#orchestrated-tests) to be different to the default GitLab configuration, or `gitlab-qa` may launch additional services in separate Docker containers, or both. Tests tagged with `:orchestrated` are excluded when testing environments where we can't dynamically modify GitLab's configuration (for example, Staging). |
| `:quarantine` | The test has been [quarantined](https://about.gitlab.com/handbook/engineering/quality/guidelines/debugging-qa-test-failures/#quarantining-tests), will run in a separate job that only includes quarantined tests, and is allowed to fail. The test will be skipped in its regular job so that if it fails it will not hold up the pipeline. Note that you can also [quarantine a test only when it runs against specific environment](environment_selection.md#quarantining-a-test-for-a-specific-environment). |
@@ -19,3 +25,20 @@ This is a partial list of the [RSpec metadata](https://relishapp.com/rspec/rspec
| `:runner` | The test depends on and will set up a GitLab Runner instance, typically to run a pipeline. |
| `:skip_live_env` | The test will be excluded when run against live deployed environments such as Staging, Canary, and Production. |
| `:testcase` | The link to the test case issue in the [Quality Testcases project](https://gitlab.com/gitlab-org/quality/testcases/). |
+| `:mattermost` | The test requires a GitLab Mattermost service on the GitLab instance. |
+| `:ldap_no_server` | The test requires a GitLab instance to be configured to use LDAP. To be used with the `:orchestrated` tag. It does not spin up an LDAP server at orchestration time. Instead, it creates the LDAP server at runtime. |
+| `:ldap_no_tls` | The test requires a GitLab instance to be configured to use an external LDAP server with TLS not enabled. |
+| `:ldap_tls` | The test requires a GitLab instance to be configured to use an external LDAP server with TLS enabled. |
+| `:object_storage` | The test requires a GitLab instance to be configured to use multiple [object storage types](../../../administration/object_storage.md). Uses MinIO as the object storage server. |
+| `:smtp` | The test requires a GitLab instance to be configured to use an SMTP server. Tests SMTP notification email delivery from GitLab by using MailHog. |
+| `:group_saml` | The test requires a GitLab instance that has SAML SSO enabled at the group level. Interacts with an external SAML identity provider. Paired with the `:orchestrated` tag. |
+| `:instance_saml` | The test requires a GitLab instance that has SAML SSO enabled at the instance level. Interacts with an external SAML identity provider. Paired with the `:orchestrated` tag. |
+| `:skip_signup_disabled` | The test uses UI to sign up a new user and will be skipped in any environment that does not allow new user registration via the UI. |
+| `:smoke` | The test belongs to the test suite which verifies basic functionality of a GitLab instance.|
+| `:github` | The test requires a GitHub personal access token. |
+| `:repository_storage` | The test requires a GitLab instance to be configured to use multiple [repository storage paths](../../../administration/repository_storage_paths.md). Paired with the `:orchestrated` tag. |
+| `:geo` | The test requires two GitLab Geo instances - a primary and a secondary - to be spun up. |
+| `:relative_url` | The test requires a GitLab instance to be installed under a [relative URL](../../../install/relative_url.md). |
+| `:requires_git_protocol_v2` | The test requires that Git protocol version 2 is enabled on the server. It's assumed to be enabled by default but if not the test can be skipped by setting `QA_CAN_TEST_GIT_PROTOCOL_V2` to `false`. |
+| `:requires_praefect` | The test requires that the GitLab instance uses [Gitaly Cluster](../../../administration/gitaly/praefect.md) (a.k.a. Praefect) as the repository storage . It's assumed to be used by default but if not the test can be skipped by setting `QA_CAN_TEST_PRAEFECT` to `false`. |
+| `:packages` | The test requires a GitLab instance that has the [Package Registry](../../../administration/packages/#gitlab-package-registry-administration) enabled. |
diff --git a/doc/development/testing_guide/end_to_end/running_tests_that_require_special_setup.md b/doc/development/testing_guide/end_to_end/running_tests_that_require_special_setup.md
index 658839fcf96..df4b833526c 100644
--- a/doc/development/testing_guide/end_to_end/running_tests_that_require_special_setup.md
+++ b/doc/development/testing_guide/end_to_end/running_tests_that_require_special_setup.md
@@ -1,3 +1,9 @@
+---
+stage: none
+group: unassigned
+info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers
+---
+
# Running tests that require special setup
## Jenkins spec
@@ -232,7 +238,11 @@ run: ./services/tunnel_gitlab: (pid 2650) 10875s, normally down; run: log: (pid
run: ./services/tunnel_registry: (pid 2651) 10875s, normally down; run: log: (pid 65155) 177993s
```
-Also, restarting Docker and then, on the Terminal, issue the command `docker login https://[YOUR-REGISTRY-PORT].qa-tunnel.gitlab.info:443` and use the GDK credentials to login. Note that the Registry port and GDK port are not the same. When configuring Auto DevOps in GDK, the `gdk reconfigure` command outputs the port of the Registry:
+Also, restarting Docker and then, on the Terminal, issue the command
+`docker login https://[YOUR-REGISTRY-PORT].qa-tunnel.gitlab.info:443` and use
+the GDK credentials to sign in. Note that the Registry port and GDK port aren't
+the same. When configuring Auto DevOps in GDK, the `gdk reconfigure` command
+outputs the port of the Registry:
```shell
*********************************************
diff --git a/doc/development/testing_guide/end_to_end/style_guide.md b/doc/development/testing_guide/end_to_end/style_guide.md
index 645c2633831..e6c96bca93e 100644
--- a/doc/development/testing_guide/end_to_end/style_guide.md
+++ b/doc/development/testing_guide/end_to_end/style_guide.md
@@ -1,3 +1,9 @@
+---
+stage: none
+group: unassigned
+info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers
+---
+
# Style guide for writing end-to-end tests
This document describes the conventions used at GitLab for writing End-to-end (E2E) tests using the GitLab QA project.
diff --git a/doc/development/testing_guide/flaky_tests.md b/doc/development/testing_guide/flaky_tests.md
index 7aed908c4f6..47ed11d76a2 100644
--- a/doc/development/testing_guide/flaky_tests.md
+++ b/doc/development/testing_guide/flaky_tests.md
@@ -1,3 +1,9 @@
+---
+stage: none
+group: unassigned
+info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers
+---
+
# Flaky tests
## What's a flaky test?
@@ -12,7 +18,13 @@ When a test frequently fails in `master`,
should be created.
If the test cannot be fixed in a timely fashion, there is an impact on the
productivity of all the developers, so it should be placed in quarantine by
-assigning the `:quarantine` metadata.
+assigning the `:quarantine` metadata with the issue URL.
+
+```ruby
+it 'should succeed', quarantine: 'https://gitlab.com/gitlab-org/gitlab/-/issues/12345' do
+ expect(response).to have_gitlab_http_status(:ok)
+end
+```
This means it will be skipped unless run with `--tag quarantine`:
diff --git a/doc/development/testing_guide/frontend_testing.md b/doc/development/testing_guide/frontend_testing.md
index 730f8d5ad7d..28fe63f1fb4 100644
--- a/doc/development/testing_guide/frontend_testing.md
+++ b/doc/development/testing_guide/frontend_testing.md
@@ -1,3 +1,9 @@
+---
+stage: none
+group: unassigned
+info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers
+---
+
# Frontend testing standards and style guidelines
There are two types of test suites you'll encounter while developing frontend code
@@ -24,7 +30,6 @@ We have started to migrate frontend tests to the [Jest](https://jestjs.io) testi
Jest tests can be found in `/spec/frontend` and `/ee/spec/frontend` in EE.
-NOTE: **Note:**
Most examples have a Jest and Karma example. See the Karma examples only as explanation to what's going on in the code, should you stumble over some use cases during your discovery. The Jest examples are the one you should follow.
## Karma test suite
@@ -110,6 +115,37 @@ describe('Component', () => {
Remember that the performance of each test depends on the environment.
+### Timout error due to async components
+
+If your component is fetching some other components asynchroneously based on some conditions, it might happen so that your Jest suite for this component will become flaky timing out from time to time.
+
+```javascript
+// ide.vue
+export default {
+ components: {
+ 'error-message': () => import('./error_message.vue'),
+ 'gl-button': () => import('@gitlab/ui/src/components/base/button/button.vue'),
+ ...
+};
+```
+
+To address this issue, you can "help" Jest by stubbing the async components so that Jest would not need to fetch those asynchroneously at the run-time.
+
+```javascript
+// ide_spec.js
+import { GlButton } from '@gitlab/ui';
+import ErrorMessage from '~/ide/components/error_message.vue';
+...
+return shallowMount(ide, {
+ ...
+ stubs: {
+ ErrorMessage,
+ GlButton,
+ ...
+ },
+})
+```
+
## What and how to test
Before jumping into more gritty details about Jest-specific workflows like mocks and spies, we should briefly cover what to test with Jest.
@@ -198,47 +234,33 @@ Following you'll find some general common practices you will find as part of our
When it comes to querying DOM elements in your tests, it is best to uniquely and semantically target
the element.
-Preferentially, this is done by targeting text the user actually sees using [DOM Testing Library](https://testing-library.com/docs/dom-testing-library/intro).
+Preferentially, this is done by targeting what the user actually sees using [DOM Testing Library](https://testing-library.com/docs/dom-testing-library/intro).
When selecting by text it is best to use [`getByRole` or `findByRole`](https://testing-library.com/docs/dom-testing-library/api-queries#byrole)
as these enforce accessibility best practices as well. The examples below demonstrate the order of preference.
-Sometimes this cannot be done feasibly. In these cases, adding test attributes to simplify the
-selectors might be the best option.
+When writing Vue component unit tests, it can be wise to query children by component, so that the unit test can focus on comprehensive value coverage
+rather than dealing with the complexity of a child component's behavior.
+
+Sometimes, neither of the above are feasible. In these cases, adding test attributes to simplify the selectors might be the best option. A list of
+possible selectors include:
- A semantic attribute like `name` (also verifies that `name` was setup properly)
- A `data-testid` attribute ([recommended by maintainers of `@vue/test-utils`](https://github.com/vuejs/vue-test-utils/issues/1498#issuecomment-610133465))
- a Vue `ref` (if using `@vue/test-utils`)
```javascript
-import { mount, shallowMount } from '@vue/test-utils'
import { getByRole, getByText } from '@testing-library/dom'
-let wrapper
-let el
-
-const createComponent = (mountFn = shallowMount) => {
- wrapper = mountFn(Component)
- el = wrapper.vm.$el // reference to the container element
-}
-
-beforeEach(() => {
- createComponent()
-})
-
-
+// In this example, `wrapper` is a `@vue/test-utils` wrapper returned from `mount` or `shallowMount`.
it('exists', () => {
- // Best
-
- // NOTE: both mount and shallowMount work as long as a DOM element is available
- // Finds a properly formatted link with an accessible name of "Click Me"
- getByRole(el, 'link', { name: /Click Me/i })
- getByRole(el, 'link', { name: 'Click Me' })
- // Finds any element with the text "Click Me"
- getByText(el, 'Click Me')
- // Regex is also available
- getByText(el, /Click Me/i)
-
- // Good
+ // Best (especially for integration tests)
+ getByRole(wrapper.element, 'link', { name: /Click Me/i })
+ getByRole(wrapper.element, 'link', { name: 'Click Me' })
+ getByText(wrapper.element, 'Click Me')
+ getByText(wrapper.element, /Click Me/i)
+
+ // Good (especially for unit tests)
+ wrapper.find(FooComponent);
wrapper.find('input[name=foo]');
wrapper.find('[data-testid="foo"]');
wrapper.find({ ref: 'foo'});
@@ -249,14 +271,6 @@ it('exists', () => {
wrapper.find('.qa-foo-component');
wrapper.find('[data-qa-selector="foo"]');
});
-
-// Good
-it('exists', () => {
- wrapper.find(FooComponent);
- wrapper.find('input[name=foo]');
- wrapper.find('[data-testid="foo"]');
- wrapper.find({ ref: 'foo'});
-});
```
It is not recommended that you add `.js-*` classes just for testing purposes. Only do this if there are no other feasible options available.
@@ -327,7 +341,6 @@ it('tests a promise rejection', async () => {
You can also simply return a promise from the test function.
-NOTE: **Note:**
Using the `done` and `done.fail` callbacks is discouraged when working with
promises. They should only be used when testing callback-based code.
@@ -777,7 +790,7 @@ While you work on a test suite, you may want to run these specs in watch mode, s
# Watch and rerun all specs matching the name icon
yarn jest --watch icon
-# Watch and rerun one specifc file
+# Watch and rerun one specific file
yarn jest --watch path/to/spec/file.spec.js
```
@@ -917,6 +930,32 @@ it.each([
);
```
+**Note**: only use template literal block if pretty print is **not** needed for spec output. For example, empty strings, nested objects etc.
+
+For example, when testing the difference between an empty search string and a non-empty search string, the use of the array block syntax with the pretty print option would be preferred. That way the differences between an empty string e.g. `''` and a non-empty string e.g. `'search string'` would be visible in the spec output. Whereas with a template literal block, the empty string would be shown as a space, which could lead to a confusing developer experience
+
+```javascript
+// bad
+it.each`
+ searchTerm | expected
+ ${''} | ${{ issue: { users: { nodes: [] } } }}
+ ${'search term'} | ${{ issue: { other: { nested: [] } } }}
+`('when search term is $searchTerm, it returns $expected', ({ searchTerm, expected }) => {
+ expect(search(searchTerm)).toEqual(expected)
+});
+
+// good
+it.each([
+ ['', { issue: { users: { nodes: [] } } }],
+ ['search term', { issue: { other: { nested: [] } } }],
+])('when search term is %p, expect to return %p',
+ (searchTerm, expected) => {
+ expect(search(searchTerm)).toEqual(expected)
+ }
+);
+
+```
+
```javascript
// test suite with tagged template literal block
describe.each`
diff --git a/doc/development/testing_guide/index.md b/doc/development/testing_guide/index.md
index a61a700594c..840c8c9206c 100644
--- a/doc/development/testing_guide/index.md
+++ b/doc/development/testing_guide/index.md
@@ -1,3 +1,9 @@
+---
+stage: none
+group: unassigned
+info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers
+---
+
# Testing standards and style guidelines
This document describes various guidelines and best practices for automated
diff --git a/doc/development/testing_guide/review_apps.md b/doc/development/testing_guide/review_apps.md
index 61d3299cabf..5dcc7e7091e 100644
--- a/doc/development/testing_guide/review_apps.md
+++ b/doc/development/testing_guide/review_apps.md
@@ -1,3 +1,9 @@
+---
+stage: none
+group: unassigned
+info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers
+---
+
# Review Apps
Review Apps are automatically deployed by [the
diff --git a/doc/development/testing_guide/smoke.md b/doc/development/testing_guide/smoke.md
index 5a144b43478..76484dd193b 100644
--- a/doc/development/testing_guide/smoke.md
+++ b/doc/development/testing_guide/smoke.md
@@ -1,3 +1,9 @@
+---
+stage: none
+group: unassigned
+info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers
+---
+
# Smoke Tests
It is imperative in any testing suite that we have Smoke Tests. In short, smoke
diff --git a/doc/development/testing_guide/testing_levels.md b/doc/development/testing_guide/testing_levels.md
index 88d7cf9ca08..d9e7edfa0c8 100644
--- a/doc/development/testing_guide/testing_levels.md
+++ b/doc/development/testing_guide/testing_levels.md
@@ -1,3 +1,9 @@
+---
+stage: none
+group: unassigned
+info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers
+---
+
# Testing levels
![Testing priority triangle](img/testing_triangle.png)
diff --git a/doc/development/testing_guide/testing_migrations_guide.md b/doc/development/testing_guide/testing_migrations_guide.md
index a5bcb651d71..b944c7ed406 100644
--- a/doc/development/testing_guide/testing_migrations_guide.md
+++ b/doc/development/testing_guide/testing_migrations_guide.md
@@ -1,4 +1,7 @@
---
+stage: none
+group: unassigned
+info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers
type: reference
---
@@ -236,6 +239,5 @@ describe Gitlab::BackgroundMigration::ArchiveLegacyTraces, schema: 2018052915262
end
```
-NOTE: **Note:**
These tests do not run within a database transaction, as we use a deletion database
cleanup strategy. Do not depend on a transaction being present.
diff --git a/doc/development/testing_guide/testing_rake_tasks.md b/doc/development/testing_guide/testing_rake_tasks.md
index db8ca87e9f8..384739d1adb 100644
--- a/doc/development/testing_guide/testing_rake_tasks.md
+++ b/doc/development/testing_guide/testing_rake_tasks.md
@@ -1,3 +1,9 @@
+---
+stage: none
+group: unassigned
+info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers
+---
+
# Testing Rake tasks
To make testing Rake tasks a little easier, there is a helper that can be included