diff options
Diffstat (limited to 'doc/development/testing_guide')
13 files changed, 159 insertions, 117 deletions
diff --git a/doc/development/testing_guide/best_practices.md b/doc/development/testing_guide/best_practices.md index ea36214f6b7..79a72981e3f 100644 --- a/doc/development/testing_guide/best_practices.md +++ b/doc/development/testing_guide/best_practices.md @@ -422,8 +422,8 @@ Use the coverage reports to ensure your tests cover 100% of your code. ### System / Feature tests NOTE: -Before writing a new system test, [please consider **not** -writing one](testing_levels.md#consider-not-writing-a-system-test)! +Before writing a new system test, +[please consider **not** writing one](testing_levels.md#consider-not-writing-a-system-test)! - Feature specs should be named `ROLE_ACTION_spec.rb`, such as `user_changes_password_spec.rb`. @@ -909,8 +909,8 @@ By default, Sidekiq jobs are enqueued into a jobs array and aren't processed. If a test queues Sidekiq jobs and need them to be processed, the `:sidekiq_inline` trait can be used. -The `:sidekiq_might_not_need_inline` trait was added when [Sidekiq inline mode was -changed to fake mode](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/15479) +The `:sidekiq_might_not_need_inline` trait was added when +[Sidekiq inline mode was changed to fake mode](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/15479) to all the tests that needed Sidekiq to actually process jobs. Tests with this trait should be either fixed to not rely on Sidekiq processing jobs, or their `:sidekiq_might_not_need_inline` trait should be updated to `:sidekiq_inline` if @@ -1239,8 +1239,7 @@ The `match_schema` matcher allows validating that the subject matches a a JSON string or a JSON-compatible data structure. `match_response_schema` is a convenience matcher for using with a -response object. from a [request -spec](testing_levels.md#integration-tests). +response object. from a [request spec](testing_levels.md#integration-tests). Examples: diff --git a/doc/development/testing_guide/contract/consumer_tests.md b/doc/development/testing_guide/contract/consumer_tests.md index df7c9ee0abd..46f4f446ad9 100644 --- a/doc/development/testing_guide/contract/consumer_tests.md +++ b/doc/development/testing_guide/contract/consumer_tests.md @@ -6,7 +6,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w # Writing consumer tests -This tutorial guides you through writing a consumer test from scratch. To start, the consumer tests are written using [`jest-pact`](https://github.com/pact-foundation/jest-pact) that builds on top of [`pact-js`](https://github.com/pact-foundation/pact-js). This tutorial shows you how to write a consumer test for the `/discussions.json` endpoint, which is actually `/:namespace_name/:project_name/-/merge_requests/:id/discussions.json`. +This tutorial guides you through writing a consumer test from scratch. To start, the consumer tests are written using [`jest-pact`](https://github.com/pact-foundation/jest-pact) that builds on top of [`pact-js`](https://github.com/pact-foundation/pact-js). This tutorial shows you how to write a consumer test for the `/discussions.json` REST API endpoint, which is actually `/:namespace_name/:project_name/-/merge_requests/:id/discussions.json`. For an example of a GraphQL consumer test, see [`spec/contracts/consumer/specs/project/pipeline/show.spec.js`](https://gitlab.com/gitlab-org/gitlab/-/tree/master/spec/contracts/consumer/specs/project/pipeline/show.spec.js). ## Create the skeleton @@ -24,7 +24,7 @@ To learn more about how the contract test directory is structured, see the contr The Pact consumer test is defined through the `pactWith` function that takes `PactOptions` and the `PactFn`. ```javascript -const { pactWith } = require('jest-pact'); +import { pactWith } from 'jest-pact'; pactWith(PactOptions, PactFn); ``` @@ -34,7 +34,7 @@ pactWith(PactOptions, PactFn); `PactOptions` with `jest-pact` introduces [additional options](https://github.com/pact-foundation/jest-pact/blob/dce370c1ab4b7cb5dff12c4b62246dc229c53d0e/README.md#defaults) that build on top of the ones [provided in `pact-js`](https://github.com/pact-foundation/pact-js#constructor). In most cases, you define the `consumer`, `provider`, `log`, and `dir` options for these tests. ```javascript -const { pactWith } = require('jest-pact'); +import { pactWith } from 'jest-pact'; pactWith( { @@ -54,7 +54,7 @@ To learn more about how to name the consumers and providers, see contract testin The `PactFn` is where your tests are defined. This is where you set up the mock provider and where you can use the standard Jest methods like [`Jest.describe`](https://jestjs.io/docs/api#describename-fn), [`Jest.beforeEach`](https://jestjs.io/docs/api#beforeeachfn-timeout), and [`Jest.it`](https://jestjs.io/docs/api#testname-fn-timeout). For more information, see [https://jestjs.io/docs/api](https://jestjs.io/docs/api). ```javascript -const { pactWith } = require('jest-pact'); +import { pactWith } from 'jest-pact'; pactWith( { @@ -70,7 +70,7 @@ pactWith( }); - it('return a successful body', () => { + it('return a successful body', async () => { }); }); @@ -92,8 +92,8 @@ For this tutorial, define four attributes for the `Interaction`: After you define the `Interaction`, add that interaction to the mock provider by calling `addInteraction`. ```javascript -const { pactWith } = require('jest-pact'); -const { Matchers } = require('@pact-foundation/pact'); +import { pactWith } from 'jest-pact'; +import { Matchers } from '@pact-foundation/pact'; pactWith( { @@ -132,7 +132,7 @@ pactWith( provider.addInteraction(interaction); }); - it('return a successful body', () => { + it('return a successful body', async () => { }); }); @@ -142,38 +142,36 @@ pactWith( ### Response body `Matchers` -Notice how we use `Matchers` in the `body` of the expected response. This allows us to be flexible enough to accept different values but still be strict enough to distinguish between valid and invalid values. We must ensure that we have a tight definition that is neither too strict nor too lax. Read more about the [different types of `Matchers`](https://github.com/pact-foundation/pact-js#using-the-v3-matching-rules). +Notice how we use `Matchers` in the `body` of the expected response. This allows us to be flexible enough to accept different values but still be strict enough to distinguish between valid and invalid values. We must ensure that we have a tight definition that is neither too strict nor too lax. Read more about the [different types of `Matchers`](https://github.com/pact-foundation/pact-js/blob/master/docs/matching.md). We are currently using the V2 matching rules. ## Write the test After the mock provider is set up, you can write the test. For this test, you make a request and expect a particular response. -First, set up the client that makes the API request. To do that, create `spec/contracts/consumer/endpoints/project/merge_requests.js` and add the following API request. +First, set up the client that makes the API request. To do that, create `spec/contracts/consumer/resources/api/project/merge_requests.js` and add the following API request. If the endpoint is a GraphQL, then we create it under `spec/contracts/consumer/resources/graphql` instead. ```javascript -const axios = require('axios'); - -exports.getDiscussions = (endpoint) => { - const url = endpoint.url; - - return axios - .request({ - method: 'GET', - baseURL: url, - url: '/gitlab-org/gitlab-qa/-/merge_requests/1/discussions.json', - headers: { Accept: '*/*' }, - }) - .then((response) => response.data); -}; +import axios from 'axios'; + +export async function getDiscussions(endpoint) { + const { url } = endpoint; + + return axios({ + method: 'GET', + baseURL: url, + url: '/gitlab-org/gitlab-qa/-/merge_requests/1/discussions.json', + headers: { Accept: '*/*' }, + }) +} ``` After that's set up, import it to the test file and call it to make the request. Then, you can make the request and define your expectations. ```javascript -const { pactWith } = require('jest-pact'); -const { Matchers } = require('@pact-foundation/pact'); +import { pactWith } from 'jest-pact'; +import { Matchers } from '@pact-foundation/pact'; -const { getDiscussions } = require('../endpoints/project/merge_requests'); +import { getDiscussions } from '../../../resources/api/project/merge_requests'; pactWith( { @@ -211,17 +209,17 @@ pactWith( }; }); - it('return a successful body', () => { - return getDiscussions({ + it('return a successful body', async () => { + const discussions = await getDiscussions({ url: provider.mockService.baseUrl, - }).then((discussions) => { - expect(discussions).toEqual(Matchers.eachLike({ - id: 'fd73763cbcbf7b29eb8765d969a38f7d735e222a', - project_id: 6954442, - ... - resolved: true - })); }); + + expect(discussions).toEqual(Matchers.eachLike({ + id: 'fd73763cbcbf7b29eb8765d969a38f7d735e222a', + project_id: 6954442, + ... + resolved: true + })); }); }); }, @@ -237,7 +235,7 @@ As you may have noticed, the request and response definitions can get large. Thi Create a file under `spec/contracts/consumer/fixtures/project/merge_request` called `discussions.fixture.js` where you will place the `request` and `response` definitions. ```javascript -const { Matchers } = require('@pact-foundation/pact'); +import { Matchers } from '@pact-foundation/pact'; const body = Matchers.eachLike({ id: Matchers.string('fd73763cbcbf7b29eb8765d969a38f7d735e222a'), @@ -254,11 +252,15 @@ const Discussions = { headers: { 'Content-Type': 'application/json; charset=utf-8', }, - body: body, + body, }, - request: { + scenario: { + state: 'a merge request with discussions exists', uponReceiving: 'a request for discussions', + }, + + request: { withRequest: { method: 'GET', path: '/gitlab-org/gitlab-qa/-/merge_requests/1/discussions.json', @@ -275,36 +277,41 @@ exports.Discussions = Discussions; With all of that moved to the `fixture`, you can simplify the test to the following: ```javascript -const { pactWith } = require('jest-pact'); +import { pactWith } from 'jest-pact'; + +import { Discussions } from '../../../fixtures/project/merge_request/discussions.fixture'; +import { getDiscussions } from '../../../resources/api/project/merge_requests'; -const { Discussions } = require('../fixtures/discussions.fixture'); -const { getDiscussions } = require('../endpoints/project/merge_requests'); +const CONSUMER_NAME = 'MergeRequest#show'; +const PROVIDER_NAME = 'Merge Request Discussions Endpoint'; +const CONSUMER_LOG = '../logs/consumer.log'; +const CONTRACT_DIR = '../contracts/project/merge_request/show'; pactWith( { - consumer: 'MergeRequest#show', - provider: 'Merge Request Discussions Endpoint', - log: '../logs/consumer.log', - dir: '../contracts/project/merge_request/show', + consumer: CONSUMER_NAME, + provider: PROVIDER_NAME, + log: CONSUMER_LOG, + dir: CONTRACT_DIR, }, (provider) => { - describe('Merge Request Discussions Endpoint', () => { + describe(PROVIDER_NAME, () => { beforeEach(() => { const interaction = { - state: 'a merge request with discussions exists', + ...Discussions.scenario, ...Discussions.request, willRespondWith: Discussions.success, }; - return provider.addInteraction(interaction); + provider.addInteraction(interaction); }); - it('return a successful body', () => { - return getDiscussions({ + it('return a successful body', async () => { + const discussions = await getDiscussions({ url: provider.mockService.baseUrl, - }).then((discussions) => { - expect(discussions).toEqual(Discussions.body); }); + + expect(discussions).toEqual(Discussions.body); }); }); }, diff --git a/doc/development/testing_guide/contract/index.md b/doc/development/testing_guide/contract/index.md index 8e12eea2874..30a4adaca44 100644 --- a/doc/development/testing_guide/contract/index.md +++ b/doc/development/testing_guide/contract/index.md @@ -28,14 +28,14 @@ Before running the consumer tests, go to `spec/contracts/consumer` and run `npm ### Run the provider tests -Before running the provider tests, make sure your GDK (GitLab Development Kit) is fully set up and running. You can follow the setup instructions detailed in the [GDK repository](https://gitlab.com/gitlab-org/gitlab-development-kit/-/tree/main). To run the provider tests, you use Rake tasks that are defined in [`./lib/tasks/contracts.rake`](https://gitlab.com/gitlab-org/gitlab/-/tree/master/lib/tasks/contracts.rake). To get a list of all the Rake tasks related to the provider tests, run `bundle exec rake -T contracts`. For example: +Before running the provider tests, make sure your GDK (GitLab Development Kit) is fully set up and running. You can follow the setup instructions detailed in the [GDK repository](https://gitlab.com/gitlab-org/gitlab-development-kit/-/tree/main). To run the provider tests, you use Rake tasks that can be found in [`./lib/tasks/contracts`](https://gitlab.com/gitlab-org/gitlab/-/tree/master/lib/tasks/contracts). To get a list of all the Rake tasks related to the provider tests, run `bundle exec rake -T contracts`. For example: ```shell $ bundle exec rake -T contracts -rake contracts:mr:pact:verify:diffs # Verify provider against the consumer pacts for diffs -rake contracts:mr:pact:verify:discussions # Verify provider against the consumer pacts for discussions -rake contracts:mr:pact:verify:metadata # Verify provider against the consumer pacts for metadata -rake contracts:mr:test:merge_request[contract_mr] # Run all merge request contract tests +rake contracts:merge_requests:pact:verify:diffs_batch # Verify provider against the consumer pacts for diffs_batch +rake contracts:merge_requests:pact:verify:diffs_metadata # Verify provider against the consumer pacts for diffs_metadata +rake contracts:merge_requests:pact:verify:discussions # Verify provider against the consumer pacts for discussions +rake contracts:merge_requests:test:merge_requests[contract_merge_requests] # Run all merge request contract tests ``` ## Test suite folder structure and naming conventions @@ -50,11 +50,11 @@ Having an organized and sensible folder structure for the test suite makes it ea The consumer tests are grouped according to the different pages in the application. Each file contains various types of requests found in a page. As such, the consumer test files are named using the Rails standards of how pages are referenced. For example, the project pipelines page would be the `Project::Pipeline#index` page so the equivalent consumer test would be located in `consumer/specs/project/pipelines/index.spec.js`. -When defining the location to output the contract generated by the test, we want to follow the same file structure which would be `contracts/project/pipelines/` for this example. This is the structure in `consumer/endpoints` and `consumer/fixtures` as well. +When defining the location to output the contract generated by the test, we want to follow the same file structure which would be `contracts/project/pipelines/` for this example. This is the structure in `consumer/resources` and `consumer/fixtures` as well. #### Provider tests -The provider tests are grouped similarly to our controllers. Each of these tests contains various tests for an API endpoint. For example, the API endpoint to get a list of pipelines for a project would be located in `provider/pact_helpers/project/pipelines/get_list_project_pipelines_helper.rb`. The provider states are structured the same way. +The provider tests are grouped similarly to our controllers. Each of these tests contains various tests for an API endpoint. For example, the API endpoint to get a list of pipelines for a project would be located in `provider/pact_helpers/project/pipelines/get_list_project_pipelines_helper.rb`. The provider states are grouped according to the different pages in the application similar to the consumer tests. ### Naming conventions diff --git a/doc/development/testing_guide/end_to_end/best_practices.md b/doc/development/testing_guide/end_to_end/best_practices.md index 00b843ffdbe..bfda94b1f1d 100644 --- a/doc/development/testing_guide/end_to_end/best_practices.md +++ b/doc/development/testing_guide/end_to_end/best_practices.md @@ -415,8 +415,8 @@ except(page).to have_no_text('hidden') Unfortunately, that's not automatically the case for the predicate methods that we add to our [page objects](page_objects.md). We need to [create our own negatable matchers](https://relishapp.com/rspec/rspec-expectations/v/3-9/docs/custom-matchers/define-a-custom-matcher#matcher-with-separate-logic-for-expect().to-and-expect().not-to). -The initial example uses the `have_job` matcher which is derived from the [`has_job?` predicate -method of the `Page::Project::Pipeline::Show` page object](https://gitlab.com/gitlab-org/gitlab/-/blob/87864b3047c23b4308f59c27a3757045944af447/qa/qa/page/project/pipeline/show.rb#L53). +The initial example uses the `have_job` matcher which is derived from the +[`has_job?` predicate method of the `Page::Project::Pipeline::Show` page object](https://gitlab.com/gitlab-org/gitlab/-/blob/87864b3047c23b4308f59c27a3757045944af447/qa/qa/page/project/pipeline/show.rb#L53). To create a negatable matcher, we use `has_no_job?` for the negative case: ```ruby diff --git a/doc/development/testing_guide/end_to_end/feature_flags.md b/doc/development/testing_guide/end_to_end/feature_flags.md index cb4c8e8a6e8..33f73304a26 100644 --- a/doc/development/testing_guide/end_to_end/feature_flags.md +++ b/doc/development/testing_guide/end_to_end/feature_flags.md @@ -217,8 +217,8 @@ If enabling the feature flag results in E2E test failures, you can browse the ar If an end-to-end test enables a feature flag, the end-to-end test suite can be used to test changes in a merge request by running the `package-and-qa` job in the merge request pipeline. If the feature flag and relevant changes have already been merged, you can confirm that the tests -pass on the default branch. The end-to-end tests run on the default branch every two hours, and the results are posted to a [Test -Session Report, which is available in the testcase-sessions project](https://gitlab.com/gitlab-org/quality/testcase-sessions/-/issues?label_name%5B%5D=found%3Amain). +pass on the default branch. The end-to-end tests run on the default branch every two hours, and the results are posted to a +[Test Session Report, which is available in the testcase-sessions project](https://gitlab.com/gitlab-org/quality/testcase-sessions/-/issues?label_name%5B%5D=found%3Amain). If the relevant tests do not enable the feature flag themselves, you can check if the tests will need to be updated by opening a draft merge request that enables the flag by default via a [feature flag definition file](../../feature_flags/index.md#feature-flag-definition-and-validation). diff --git a/doc/development/testing_guide/end_to_end/index.md b/doc/development/testing_guide/end_to_end/index.md index 06359d612ad..989d090d581 100644 --- a/doc/development/testing_guide/end_to_end/index.md +++ b/doc/development/testing_guide/end_to_end/index.md @@ -140,8 +140,8 @@ a flaky test we first want to make sure that it's no longer flaky. We can do that using the `ce:custom-parallel` and `ee:custom-parallel` jobs. Both are manual jobs that you can configure using custom variables. When clicking the name (not the play icon) of one of the parallel jobs, -you are prompted to enter variables. You can use any of [the variables -that can be used with `gitlab-qa`](https://gitlab.com/gitlab-org/gitlab-qa/blob/master/docs/what_tests_can_be_run.md#supported-gitlab-environment-variables) +you are prompted to enter variables. You can use any of +[the variables that can be used with `gitlab-qa`](https://gitlab.com/gitlab-org/gitlab-qa/blob/master/docs/what_tests_can_be_run.md#supported-gitlab-environment-variables) as well as these: | Variable | Description | @@ -150,8 +150,9 @@ as well as these: | `QA_TESTS` | The tests to run (no default, which means run all the tests in the scenario). Use file paths as you would when running tests via RSpec, for example, `qa/specs/features/ee/browser_ui` would include all the `EE` UI tests. | | `QA_RSPEC_TAGS` | The RSpec tags to add (no default) | -For now, [manual jobs with custom variables don't use the same variable -when retried](https://gitlab.com/gitlab-org/gitlab/-/issues/31367), so if you want to run the same tests multiple times, +For now, +[manual jobs with custom variables don't use the same variable when retried](https://gitlab.com/gitlab-org/gitlab/-/issues/31367), +so if you want to run the same tests multiple times, specify the same variables in each `custom-parallel` job (up to as many of the 10 available jobs that you want to run). @@ -164,8 +165,8 @@ automatically started: it runs the QA smoke suite against the You can also manually start the `review-qa-all`: it runs the full QA suite against the [Review App](../review_apps.md). -**This runs end-to-end tests against a Review App based on [the official GitLab -Helm chart](https://gitlab.com/gitlab-org/charts/gitlab/), itself deployed with custom +**This runs end-to-end tests against a Review App based on +[the official GitLab Helm chart](https://gitlab.com/gitlab-org/charts/gitlab/), itself deployed with custom [Cloud Native components](https://gitlab.com/gitlab-org/build/CNG) built from your merge request's changes.** See [Review Apps](../review_apps.md) for more details about Review Apps. @@ -235,7 +236,7 @@ Each type of scheduled pipeline generates a static link for the latest test repo - [`staging-sanity`](https://storage.googleapis.com/gitlab-qa-allure-reports/staging-sanity/master/index.html) - [`staging-sanity-no-admin`](https://storage.googleapis.com/gitlab-qa-allure-reports/staging-sanity-no-admin/master/index.html) - [`canary-sanity`](https://storage.googleapis.com/gitlab-qa-allure-reports/canary-sanity/master/index.html) -- [`production`](https://storage.googleapis.com/gitlab-qa-allure-reports/production/master/index.html) +- [`production`](https://storage.googleapis.com/gitlab-qa-allure-reports/production-full/master/index.html) - [`production-sanity`](https://storage.googleapis.com/gitlab-qa-allure-reports/production-sanity/master/index.html) ## How do I run the tests? @@ -243,8 +244,8 @@ Each type of scheduled pipeline generates a static link for the latest test repo If you are not [testing code in a merge request](#testing-code-in-merge-requests), there are two main options for running the tests. If you want to run the existing tests against a live GitLab instance or against a pre-built Docker image, -use the [GitLab QA orchestrator](https://gitlab.com/gitlab-org/gitlab-qa/tree/master/README.md). See also [examples -of the test scenarios you can run via the orchestrator](https://gitlab.com/gitlab-org/gitlab-qa/blob/master/docs/what_tests_can_be_run.md#examples). +use the [GitLab QA orchestrator](https://gitlab.com/gitlab-org/gitlab-qa/tree/master/README.md). See also +[examples of the test scenarios you can run via the orchestrator](https://gitlab.com/gitlab-org/gitlab-qa/blob/master/docs/what_tests_can_be_run.md#examples). On the other hand, if you would like to run against a local development GitLab environment, you can use the [GitLab Development Kit (GDK)](https://gitlab.com/gitlab-org/gitlab-development-kit/). @@ -262,8 +263,8 @@ architecture. See the [documentation about it](https://gitlab.com/gitlab-org/git Once you decided where to put [test environment orchestration scenarios](https://gitlab.com/gitlab-org/gitlab-qa/tree/master/lib/gitlab/qa/scenario) and [instance-level scenarios](https://gitlab.com/gitlab-org/gitlab-foss/tree/master/qa/qa/specs/features), take a look at the [GitLab QA README](https://gitlab.com/gitlab-org/gitlab/-/tree/master/qa/README.md), -the [GitLab QA orchestrator README](https://gitlab.com/gitlab-org/gitlab-qa/tree/master/README.md), and [the already existing -instance-level scenarios](https://gitlab.com/gitlab-org/gitlab-foss/tree/master/qa/qa/specs/features). +the [GitLab QA orchestrator README](https://gitlab.com/gitlab-org/gitlab-qa/tree/master/README.md), +and [the already existing instance-level scenarios](https://gitlab.com/gitlab-org/gitlab-foss/tree/master/qa/qa/specs/features). ### Consider **not** writing an end-to-end test diff --git a/doc/development/testing_guide/end_to_end/rspec_metadata_tests.md b/doc/development/testing_guide/end_to_end/rspec_metadata_tests.md index 591d03db7b8..322f2412e5b 100644 --- a/doc/development/testing_guide/end_to_end/rspec_metadata_tests.md +++ b/doc/development/testing_guide/end_to_end/rspec_metadata_tests.md @@ -30,6 +30,7 @@ This is a partial list of the [RSpec metadata](https://relishapp.com/rspec/rspec | `:ldap_no_tls` | The test requires a GitLab instance to be configured to use an external LDAP server with TLS not enabled. | | `:ldap_tls` | The test requires a GitLab instance to be configured to use an external LDAP server with TLS enabled. | | `:mattermost` | The test requires a GitLab Mattermost service on the GitLab instance. | +| `:metrics` | The test requires a GitLab instance where [dedicated metrics exporters](../../../administration/monitoring/prometheus/web_exporter.md) are running alongside Puma and Sidekiq. | | `:mixed_env` | The test should only be executed in environments that have a paired canary version available through traffic routing based on the existence of the `gitlab_canary=true` cookie. Tests in this category are switching the cookie mid-test to validate mixed deployment environments. | | `:object_storage` | The test requires a GitLab instance to be configured to use multiple [object storage types](../../../administration/object_storage.md). Uses MinIO as the object storage server. | | `:only` | The test is only to be run in specific execution contexts. See [test execution context selection](execution_context_selection.md) for more information. | diff --git a/doc/development/testing_guide/end_to_end/running_tests_that_require_special_setup.md b/doc/development/testing_guide/end_to_end/running_tests_that_require_special_setup.md index 438294161ac..322f108783f 100644 --- a/doc/development/testing_guide/end_to_end/running_tests_that_require_special_setup.md +++ b/doc/development/testing_guide/end_to_end/running_tests_that_require_special_setup.md @@ -8,11 +8,20 @@ info: To determine the technical writer assigned to the Stage/Group associated w ## Jenkins spec -The [`jenkins_build_status_spec`](https://gitlab.com/gitlab-org/gitlab/-/blob/163c8a8c814db26d11e104d1cb2dcf02eb567dbe/qa/qa/specs/features/ee/browser_ui/3_create/jenkins/jenkins_build_status_spec.rb) spins up a Jenkins instance in a Docker container based on an image stored in the [GitLab-QA container registry](https://gitlab.com/gitlab-org/gitlab-qa/container_registry). -The Docker image it uses is preconfigured with some base data and plugins. -The test then configures the GitLab plugin in Jenkins with a URL of the GitLab instance that are used -to run the tests. Unfortunately, the GitLab Jenkins plugin does not accept ports so `http://localhost:3000` would -not be accepted. Therefore, this requires us to run GitLab on port 80 or inside a Docker container. +The [`jenkins_build_status_spec`](https://gitlab.com/gitlab-org/gitlab/-/blob/24a86debf49f3aed6f2ecfd6e8f9233b3a214181/qa/qa/specs/features/browser_ui/3_create/jenkins/jenkins_build_status_spec.rb) +spins up a Jenkins instance in a Docker container with the Jenkins GitLab plugin pre-installed. Due to a license restriction we are unable to distribute this image. +To build a QA compatible image, please visit the [third party images project](https://gitlab.com/gitlab-org/quality/third-party-docker-public), where third party Dockerfiles can be found. +The project also has instructions for forking and building the images automatically in CI. + +Some extra environment variables for the location of the forked repository are also needed. + +- `QA_THIRD_PARTY_DOCKER_REGISTRY` (the container registry where the repository/images are hosted, eg `registry.gitlab.com`) +- `QA_THIRD_PARTY_DOCKER_REPOSITORY` (the base repository path where the images are hosted, eg `registry.gitlab.com/<project path>`) +- `QA_THIRD_PARTY_DOCKER_USER` (a username that has access to the container registry for this repository) +- `QA_THIRD_PARTY_DOCKER_PASSWORD` (a password/token for the username to authenticate with) + +The test configures the GitLab plugin in Jenkins with a URL of the GitLab instance that are used +to run the tests. Bi-directional networking is needed between a GitLab instance and Jenkins, so GitLab can also be started in a Docker container. To start a Docker container for GitLab based on the nightly image: @@ -21,34 +30,25 @@ docker run \ --publish 80:80 \ --name gitlab \ --hostname localhost \ + --network test gitlab/gitlab-ee:nightly ``` To run the tests from the `/qa` directory: ```shell -WEBDRIVER_HEADLESS=false bin/qa Test::Instance::All http://localhost -- qa/specs/features/ee/browser_ui/3_create/jenkins/jenkins_build_status_spec.rb +export QA_THIRD_PARTY_DOCKER_REGISTRY=<registry> +export QA_THIRD_PARTY_DOCKER_REPOSITORY=<repository> +export QA_THIRD_PARTY_DOCKER_USER=<user with registry access> +export QA_THIRD_PARTY_DOCKER_PASSWORD=<password for user> +export WEBDRIVER_HEADLESS=0 +bin/qa Test::Instance::All http://localhost -- qa/specs/features/ee/browser_ui/3_create/jenkins/jenkins_build_status_spec.rb ``` The test automatically spins up a Docker container for Jenkins and tear down once the test completes. -However, if you need to run Jenkins manually outside of the tests, use this command: - -```shell -docker run \ - --hostname localhost \ - --name jenkins-server \ - --env JENKINS_HOME=jenkins_home \ - --publish 8080:8080 \ - registry.gitlab.com/gitlab-org/gitlab-qa/jenkins-gitlab:version1 -``` - -Jenkins is available on `http://localhost:8080`. - -Administrator username is `admin` and password is `password`. - -It is worth noting that this is not an orchestrated test. It is [tagged with the `:orchestrated` meta](https://gitlab.com/gitlab-org/gitlab/-/blob/163c8a8c814db26d11e104d1cb2dcf02eb567dbe/qa/qa/specs/features/ee/browser_ui/3_create/jenkins/jenkins_build_status_spec.rb#L5) -only to prevent it from running in the pipelines for live environments such as Staging. +If you need to run Jenkins manually outside of the tests, please refer to the README for the +[third party images project](https://gitlab.com/gitlab-org/quality/third-party-docker-public/-/blob/main/jenkins/README.md) ### Troubleshooting @@ -385,6 +385,43 @@ To run the LDAP tests on your local with TLS disabled, follow these steps: GITLAB_LDAP_USERNAME="tanuki" GITLAB_LDAP_PASSWORD="password" QA_LOG_LEVEL=debug WEBDRIVER_HEADLESS=false bin/qa Test::Instance::All http://localhost qa/specs/features/browser_ui/1_manage/login/log_into_gitlab_via_ldap_spec.rb ``` +## SMTP tests + +Tests that are tagged with `:smtp` meta tag are orchestrated tests that ensure email notifications are received by a user. + +These tests require a GitLab instance with SMTP enabled and integrated with an SMTP server, [MailHog](https://github.com/mailhog/MailHog). + +To run these tests locally against the GDK: + +1. Add these settings to your `gitlab.yml` file: + + ```yaml + smtp: + enabled: true + address: "mailhog.test" + port: 1025 + ``` + +1. Start MailHog in a Docker container: + + ```shell + docker network create test && docker run \ + --network test \ + --hostname mailhog.test \ + --name mailhog \ + --publish 1025:1025 \ + --publish 8025:8025 \ + mailhog/mailhog:v1.0.0 + ``` + +1. Run the test from [`gitlab/qa`](https://gitlab.com/gitlab-org/gitlab/-/tree/d5447ebb5f99d4c72780681ddf4dc25b0738acba/qa) directory: + + ```shell + QA_LOG_LEVEL=debug WEBDRIVER_HEADLESS=false bin/qa Test::Instance::All http://localhost:3000 qa/specs/features/browser_ui/2_plan/email/trigger_email_notification_spec.rb -- --tag orchestrated + ``` + +For instructions on how to run these tests using the `gitlab-qa` gem, please refer to [the GitLab QA documentation](https://gitlab.com/gitlab-org/gitlab-qa/-/blob/master/docs/what_tests_can_be_run.md#testintegrationsmtp-ceeefull-image-address). + ## Guide to the mobile suite ### What are mobile tests diff --git a/doc/development/testing_guide/frontend_testing.md b/doc/development/testing_guide/frontend_testing.md index d91c53823e2..2845dde9a24 100644 --- a/doc/development/testing_guide/frontend_testing.md +++ b/doc/development/testing_guide/frontend_testing.md @@ -51,15 +51,12 @@ The default timeout for Jest is set in If your test exceeds that time, it fails. If you cannot improve the performance of the tests, you can increase the timeout -for a specific test using -[`setTestTimeout`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/spec/frontend/__helpers__/timeout.js). +for a specific test using [`jest.setTimeout`](https://jestjs.io/docs/27.x/jest-object#jestsettimeouttimeout) ```javascript -import { setTestTimeout } from 'helpers/timeout'; - describe('Component', () => { it('does something amazing', () => { - setTestTimeout(500); + jest.setTimeout(500); // ... }); }); @@ -466,7 +463,7 @@ it('waits for an Ajax call', () => { #### Vue rendering -Use [`nextTick()`](https://vuejs.org/v2/api/#Vue-nextTick) to wait until a Vue component is +Use [`nextTick()`](https://v2.vuejs.org/v2/api/#Vue-nextTick) to wait until a Vue component is re-rendered. **in Jest:** diff --git a/doc/development/testing_guide/index.md b/doc/development/testing_guide/index.md index fa9f1f1ac3e..cd7c70e2eaa 100644 --- a/doc/development/testing_guide/index.md +++ b/doc/development/testing_guide/index.md @@ -9,8 +9,8 @@ info: To determine the technical writer assigned to the Stage/Group associated w This document describes various guidelines and best practices for automated testing of the GitLab project. -It is meant to be an _extension_ of the [Thoughtbot testing -style guide](https://github.com/thoughtbot/guides/tree/master/testing-rspec). If +It is meant to be an _extension_ of the +[Thoughtbot testing style guide](https://github.com/thoughtbot/guides/tree/master/testing-rspec). If this guide defines a rule that contradicts the Thoughtbot guide, this guide takes precedence. Some guidelines may be repeated verbatim to stress their importance. diff --git a/doc/development/testing_guide/review_apps.md b/doc/development/testing_guide/review_apps.md index 532bb9fcdef..b272d23522e 100644 --- a/doc/development/testing_guide/review_apps.md +++ b/doc/development/testing_guide/review_apps.md @@ -25,7 +25,7 @@ For any of the following scenarios, the `start-review-app-pipeline` job would be On every [pipeline](https://gitlab.com/gitlab-org/gitlab/pipelines/125315730) in the `qa` stage (which comes after the `review` stage), the `review-qa-smoke` and `review-qa-reliable` jobs are automatically started. The `review-qa-smoke` runs -the QA smoke suite and the `review-qa-reliable` executes E2E tests identified as [reliable](https://about.gitlab.com/handbook/engineering/quality/quality-engineering/reliable-tests). +the QA smoke suite and the `review-qa-reliable` executes E2E tests identified as [reliable](https://about.gitlab.com/handbook/engineering/quality/quality-engineering/reliable-tests/). `review-qa-*` jobs ensure that end-to-end tests for the changes in the merge request pass in a live environment. This shifts the identification of e2e failures from an environment on the path to production to the merge request, to prevent breaking features on GitLab.com or costly GitLab.com deployment blockers. `review-qa-*` failures should be investigated with counterpart SET involvement if needed to help determine the root cause of the error. diff --git a/doc/development/testing_guide/testing_levels.md b/doc/development/testing_guide/testing_levels.md index 02f32a031dc..c1bf3609b53 100644 --- a/doc/development/testing_guide/testing_levels.md +++ b/doc/development/testing_guide/testing_levels.md @@ -115,7 +115,7 @@ graph RL Testing the value of a constant means copying it, resulting in extra effort without additional confidence that the value is correct. - **Vue components**: Computed properties, methods, and lifecycle hooks can be considered an implementation detail of components, are implicitly covered by component tests, and don't need to be tested. - For more information, see the [official Vue guidelines](https://vue-test-utils.vuejs.org/guides/#getting-started). + For more information, see the [official Vue guidelines](https://v1.test-utils.vuejs.org/guides/#getting-started). #### What to mock in unit tests @@ -208,7 +208,7 @@ graph RL Similar to unit tests, background operations cannot be stopped or waited on. This means they continue running in the following tests and cause side effects. - **Child components**: Every component is tested individually, so child components are mocked. - See also [`shallowMount()`](https://vue-test-utils.vuejs.org/api/#shallowmount) + See also [`shallowMount()`](https://v1.test-utils.vuejs.org/api/#shallowmount) #### What *not* to mock in component tests diff --git a/doc/development/testing_guide/testing_migrations_guide.md b/doc/development/testing_guide/testing_migrations_guide.md index d71788e21f3..261a4f4a27e 100644 --- a/doc/development/testing_guide/testing_migrations_guide.md +++ b/doc/development/testing_guide/testing_migrations_guide.md @@ -317,8 +317,8 @@ To test these you usually have to: - Verify that the expected jobs were scheduled, with the correct set of records, the correct batch size, interval, etc. -The behavior of the background migration itself needs to be verified in a [separate -test for the background migration class](#example-background-migration-test). +The behavior of the background migration itself needs to be verified in a +[separate test for the background migration class](#example-background-migration-test). This spec tests the [`db/post_migrate/20210701111909_backfill_issues_upvotes_count.rb`](https://gitlab.com/gitlab-org/gitlab/-/blob/v14.1.0-ee/db/post_migrate/20210701111909_backfill_issues_upvotes_count.rb) |