diff options
Diffstat (limited to 'doc/development/testing_guide')
| -rw-r--r-- | doc/development/testing_guide/best_practices.md | 8 | ||||
| -rw-r--r-- | doc/development/testing_guide/contract/consumer_tests.md | 44 | ||||
| -rw-r--r-- | doc/development/testing_guide/contract/index.md | 26 | ||||
| -rw-r--r-- | doc/development/testing_guide/contract/provider_tests.md | 52 | ||||
| -rw-r--r-- | doc/development/testing_guide/end_to_end/index.md | 19 | ||||
| -rw-r--r-- | doc/development/testing_guide/end_to_end/resources.md | 180 | ||||
| -rw-r--r-- | doc/development/testing_guide/frontend_testing.md | 4 | ||||
| -rw-r--r-- | doc/development/testing_guide/img/testing_triangle.png | bin | 32902 -> 13854 bytes | |||
| -rw-r--r-- | doc/development/testing_guide/testing_migrations_guide.md | 2 |
9 files changed, 87 insertions, 248 deletions
diff --git a/doc/development/testing_guide/best_practices.md b/doc/development/testing_guide/best_practices.md index aee3e2871c2..e27d4911158 100644 --- a/doc/development/testing_guide/best_practices.md +++ b/doc/development/testing_guide/best_practices.md @@ -155,6 +155,8 @@ To avoid creation, it is worth bearing in mind that: Use [Factory Doctor](https://test-prof.evilmartians.io/#/profilers/factory_doctor) to find cases where database persistence is not needed in a given test. +Examples of factories optimization [1](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/106796), [2](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/105329). + ```shell # run test for path FDOC=1 bin/rspec spec/[path]/[to]/[spec].rb @@ -831,7 +833,7 @@ To resolve, use `let`, or change the factory to not use stubs. ### Time-sensitive tests -[`ActiveSupport::Testing::TimeHelpers`](https://api.rubyonrails.org/v6.0.3.1/classes/ActiveSupport/Testing/TimeHelpers.html) +[`ActiveSupport::Testing::TimeHelpers`](https://api.rubyonrails.org/classes/ActiveSupport/Testing/TimeHelpers.html) can be used to verify things that are time-sensitive. Any test that exercises or verifies something time-sensitive should make use of these helpers to prevent transient test failures. @@ -852,7 +854,7 @@ end #### RSpec helpers You can use the `:freeze_time` and `:time_travel_to` RSpec metadata tag helpers to help reduce the amount of -boilerplate code needed to wrap entire specs with the [`ActiveSupport::Testing::TimeHelpers`](https://api.rubyonrails.org/v6.0.3.1/classes/ActiveSupport/Testing/TimeHelpers.html) +boilerplate code needed to wrap entire specs with the [`ActiveSupport::Testing::TimeHelpers`](https://api.rubyonrails.org/classes/ActiveSupport/Testing/TimeHelpers.html) methods. ```ruby @@ -872,7 +874,7 @@ end ``` [Under the hood](https://gitlab.com/gitlab-org/gitlab/-/blob/master/spec/support/time_travel.rb), these helpers use the `around(:each)` hook and the block syntax of the -[`ActiveSupport::Testing::TimeHelpers`](https://api.rubyonrails.org/v6.0.3.1/classes/ActiveSupport/Testing/TimeHelpers.html) +[`ActiveSupport::Testing::TimeHelpers`](https://api.rubyonrails.org/classes/ActiveSupport/Testing/TimeHelpers.html) methods: ```ruby diff --git a/doc/development/testing_guide/contract/consumer_tests.md b/doc/development/testing_guide/contract/consumer_tests.md index 9c72e6835bd..39cc34d6153 100644 --- a/doc/development/testing_guide/contract/consumer_tests.md +++ b/doc/development/testing_guide/contract/consumer_tests.md @@ -6,11 +6,11 @@ info: To determine the technical writer assigned to the Stage/Group associated w # Writing consumer tests -This tutorial guides you through writing a consumer test from scratch. To start, the consumer tests are written using [`jest-pact`](https://github.com/pact-foundation/jest-pact) that builds on top of [`pact-js`](https://github.com/pact-foundation/pact-js). This tutorial shows you how to write a consumer test for the `/discussions.json` REST API endpoint, which is actually `/:namespace_name/:project_name/-/merge_requests/:id/discussions.json`. For an example of a GraphQL consumer test, see [`spec/contracts/consumer/specs/project/pipeline/show.spec.js`](https://gitlab.com/gitlab-org/gitlab/-/tree/master/spec/contracts/consumer/specs/project/pipeline/show.spec.js). +This tutorial guides you through writing a consumer test from scratch. To start, the consumer tests are written using [`jest-pact`](https://github.com/pact-foundation/jest-pact) that builds on top of [`pact-js`](https://github.com/pact-foundation/pact-js). This tutorial shows you how to write a consumer test for the `/discussions.json` REST API endpoint, which is `/:namespace_name/:project_name/-/merge_requests/:id/discussions.json`, that is called in the `MergeRequests#show` page. For an example of a GraphQL consumer test, see [`spec/contracts/consumer/specs/project/pipelines/show.spec.js`](https://gitlab.com/gitlab-org/gitlab/-/tree/master/spec/contracts/consumer/specs/project/pipelines/show.spec.js). ## Create the skeleton -Start by creating the skeleton of a consumer test. Create a file under `spec/contracts/consumer/specs/project/merge_request` called `discussions.spec.js`. +Start by creating the skeleton of a consumer test. Since this is for a request in the `MergeRequests#show` page, under `spec/contracts/consumer/specs/project/merge_requests`, create a file called `show.spec.js`. Then, populate it with the following function and parameters: - [`pactWith`](#the-pactwith-function) @@ -38,10 +38,10 @@ import { pactWith } from 'jest-pact'; pactWith( { - consumer: 'MergeRequest#show', - provider: 'Merge Request Discussions Endpoint', + consumer: 'MergeRequests#show', + provider: 'GET discussions', log: '../logs/consumer.log', - dir: '../contracts/project/merge_request/show', + dir: '../contracts/project/merge_requests/show', }, PactFn ); @@ -58,14 +58,14 @@ import { pactWith } from 'jest-pact'; pactWith( { - consumer: 'MergeRequest#show', - provider: 'Merge Request Discussions Endpoint', + consumer: 'MergeRequests#show', + provider: 'GET discussions', log: '../logs/consumer.log', - dir: '../contracts', + dir: '../contracts/project/merge_requests/show', }, (provider) => { - describe('Merge Request Discussions Endpoint', () => { + describe('GET discussions', () => { beforeEach(() => { }); @@ -97,14 +97,14 @@ import { Matchers } from '@pact-foundation/pact'; pactWith( { - consumer: 'MergeRequest#show', - provider: 'Merge Request Discussions Endpoint', + consumer: 'MergeRequests#show', + provider: 'GET discussions', log: '../logs/consumer.log', - dir: '../contracts/project/merge_request/show', + dir: '../contracts/project/merge_requests/show', }, (provider) => { - describe('Merge Request Discussions Endpoint', () => { + describe('GET discussions', () => { beforeEach(() => { const interaction = { state: 'a merge request with discussions exists', @@ -175,14 +175,14 @@ import { getDiscussions } from '../../../resources/api/project/merge_requests'; pactWith( { - consumer: 'MergeRequest#show', - provider: 'Merge Request Discussions Endpoint', + consumer: 'MergeRequests#show', + provider: 'GET discussions', log: '../logs/consumer.log', - dir: '../contracts/project/merge_request/show', + dir: '../contracts/project/merge_requests/show', }, (provider) => { - describe('Merge Request Discussions Endpoint', () => { + describe('GET discussions', () => { beforeEach(() => { const interaction = { state: 'a merge request with discussions exists', @@ -232,7 +232,7 @@ There we have it! The consumer test is now set up. You can now try [running this As you may have noticed, the request and response definitions can get large. This results in the test being difficult to read, with a lot of scrolling to find what you want. You can make the test easier to read by extracting these out to a `fixture`. -Create a file under `spec/contracts/consumer/fixtures/project/merge_request` called `discussions.fixture.js` where you will place the `request` and `response` definitions. +Create a file under `spec/contracts/consumer/fixtures/project/merge_requests` called `discussions.fixture.js` where you will place the `request` and `response` definitions. ```javascript import { Matchers } from '@pact-foundation/pact'; @@ -279,13 +279,13 @@ With all of that moved to the `fixture`, you can simplify the test to the follow ```javascript import { pactWith } from 'jest-pact'; -import { Discussions } from '../../../fixtures/project/merge_request/discussions.fixture'; +import { Discussions } from '../../../fixtures/project/merge_requests/discussions.fixture'; import { getDiscussions } from '../../../resources/api/project/merge_requests'; -const CONSUMER_NAME = 'MergeRequest#show'; -const PROVIDER_NAME = 'Merge Request Discussions Endpoint'; +const CONSUMER_NAME = 'MergeRequests#show'; +const PROVIDER_NAME = 'GET discussions'; const CONSUMER_LOG = '../logs/consumer.log'; -const CONTRACT_DIR = '../contracts/project/merge_request/show'; +const CONTRACT_DIR = '../contracts/project/merge_requests/show'; pactWith( { diff --git a/doc/development/testing_guide/contract/index.md b/doc/development/testing_guide/contract/index.md index 08a21e58a52..31d68bb9f4f 100644 --- a/doc/development/testing_guide/contract/index.md +++ b/doc/development/testing_guide/contract/index.md @@ -24,7 +24,7 @@ The contracts themselves are stored in [`/spec/contracts/contracts`](https://git ### Run the consumer tests -Before running the consumer tests, go to `spec/contracts/consumer` and run `npm install`. To run all the consumer tests, you just need to run `npm test -- /specs`. Otherwise, to run a specific spec file, replace `/specs` with the specific spec filename. +Before running the consumer tests, go to `spec/contracts/consumer` and run `npm install`. To run all the consumer tests, you just need to run `npm run jest:contract -- /specs`. Otherwise, to run a specific spec file, replace `/specs` with the specific spec filename. Running the consumer test will create the contract that the provider test uses to verify the actual API behavior. You can also run tests from the root directory of the project, using the command `yarn jest:contract`. @@ -40,6 +40,14 @@ rake contracts:merge_requests:pact:verify:discussions rake contracts:merge_requests:test:merge_requests[contract_merge_requests] # Run all merge request contract tests ``` +#### Verify the contracts in Pact Broker + +By default, the Rake tasks will verify the locally stored contracts. In order to verify the contracts published in the Pact Broker, we need to set the `PACT_BROKER` environment variable to `true`. It is important to point out here that the file path and file name of the provider test is what is used to find the contract in the Pact Broker which is why it is important to make sure the [provider test naming conventions](#provider-naming) are followed. + +## Publish contracts to Pact Broker + +The contracts generated by the consumer test can be published to a hosted Pact Broker by going to `spec/contracts` and running the `publish-contracts.sh` script. + ## Test suite folder structure and naming conventions To keep the consumer and provider test suite organized and maintainable, it's important that tests are organized, also that consumers and providers are named consistently. Therefore, it's important to adhere to the following conventions. @@ -50,31 +58,33 @@ Having an organized and sensible folder structure for the test suite makes it ea #### Consumer tests -The consumer tests are grouped according to the different pages in the application. Each file contains various types of requests found in a page. As such, the consumer test files are named using the Rails standards of how pages are referenced. For example, the project pipelines page would be the `Project::Pipeline#index` page so the equivalent consumer test would be located in `consumer/specs/project/pipelines/index.spec.js`. +The consumer tests are grouped according to the different pages in the application. Each file contains various types of requests found in a page. As such, the consumer test files are named using the Rails standards of how pages are referenced. For example, the project pipelines page would be the `Project::Pipelines#index` page so the equivalent consumer test would be located in `consumer/specs/project/pipelines/index.spec.js`. When defining the location to output the contract generated by the test, we want to follow the same file structure which would be `contracts/project/pipelines/` for this example. This is the structure in `consumer/resources` and `consumer/fixtures` as well. +The naming of the folders must also be pluralized to match how they are called in the Rails naming standard. + #### Provider tests The provider tests are grouped similarly to our controllers. Each of these tests contains various tests for an API endpoint. For example, the API endpoint to get a list of pipelines for a project would be located in `provider/pact_helpers/project/pipelines/get_list_project_pipelines_helper.rb`. The provider states are grouped according to the different pages in the application similar to the consumer tests. ### Naming conventions -When writing the consumer and provider tests, there are parts where a name is required for the consumer and provider. Since there are no restrictions imposed by Pact on how these should be named, a naming convention is important to keep it easy for us to figure out which consumer and provider tests are involved during debugging. Pact also uses the consumer and provider names to generate the generated contracts in the `#{consumer_name}-#{provider_name}` format. +When writing the consumer and provider tests, there are parts where a name is required for the consumer and provider. Since there are no restrictions imposed by Pact on how these should be named, a naming convention is important to keep it easy for us to figure out which consumer and provider tests are involved during debugging. Pact also uses the consumer and provider names to create the locally stored contract file names in the `#{consumer_name}-#{provider_name}` format. #### Consumer naming -As mentioned in the [folder structure section](#consumer-tests), consumer tests are grouped according to the different pages in the application. As such, consumer names should follow the same naming format using the Rails standard. For example, the consumer test for `Project::Pipeline#index` would be `ProjectPipeline#index` as the consumer name. Since Pact uses this name to name the contracts it generates, the colons (`::`) are dropped as colons are not valid characters in file names. +As mentioned in the [folder structure section](#consumer-tests), consumer tests are grouped according to the different pages in the application. As such, consumer names should follow the same naming format using the Rails standard. For example, the consumer test for `Project::Pipelines#index` would be under the `project` folder and will be called `Pipelines#index` as the consumer name. #### Provider naming -These are the API endpoints that provides the data to the consumer so they are named according to the API endpoint they pertain to. Be mindful that this name is as descriptive as possible. For example, if we're writing a test for the `GET /groups/:id/projects` endpoint, we don't want to name it "Projects endpoint" as there is a `GET /projects` endpoint as well that also fetches a list of projects the user has access to across all of GitLab. An easy way to name them is by checking out our [API documentation](../../../api/api_resources.md) and naming it the same way it is named in there. So the [`GET /groups/:id/projects`](../../../api/groups.md#list-a-groups-projects) would be called `List a group’s projects` and [`GET /projects`](../../../api/projects.md#list-all-projects) would be called `List all projects`. Subsequently, the test files are named `list_a_groups_projects_helper.rb` and `list_all_projects_helper.rb` respectively. +These are the API endpoints that provides the data to the consumer so they are named according to the API endpoint they pertain to. Be mindful that this begins with the HTTP request method and the rest of the name is as descriptive as possible. For example, if we're writing a test for the `GET /groups/:id/projects` endpoint, we don't want to name it "GET projects endpoint" as there is a `GET /projects` endpoint as well that also fetches a list of projects the user has access to across all of GitLab. To choose an appropriate name, we can start by checking out our [API documentation](../../../api/api_resources.md) and naming it the same way it is named in there while making sure to keep the name in sentence case. So the [`GET /groups/:id/projects`](../../../api/groups.md#list-a-groups-projects) would be called `GET list a group's projects` and [`GET /projects`](../../../api/projects.md#list-all-projects) would be called `GET list all projects`. Subsequently, the test files are named `get_list_a_groups_projects_helper.rb` and `get_list_all_projects_helper.rb` respectively. -There are some cases where the provider being tested may not be documented so, in those cases, fall back to choosing a name that is as descriptive as possible to ensure it's easy to tell what the provider is for. +There are some cases where the provider being tested may not be documented so, in those cases, fall back to starting with the HTTP request method followed by a name that is as descriptive as possible to ensure it's easy to tell what the provider is for. #### Conventions summary | Tests | Folder structure | Naming convention | | ----- | ---------------- | ----------------- | -| Consumer Test | Follows the Rails reference standards. For example, `Project::Pipeline#index` would be `consumer/specs/project/pipelines/index.spec.js` | Follows the Rails naming standard. For example, `Project::Pipeline#index` would be `ProjectPipeline#index` | -| Provider Test | Grouped like the Rails controllers. For example, [`List project pipelines` API endpoint](../../../api/pipelines.md#list-project-pipelines) would be `provider/pact_helpers/project/pipelines/provider/pact_helpers/project/pipelines/get_list_project_pipelines_helper.rb` | Follows the API documentation naming scheme. For example, [`GET /projects/:id/pipelines`](../../../api/pipelines.md#list-project-pipelines) would be called `List project pipelines`. | +| Consumer Test | Follows the Rails reference standards. For example, `Project::Pipelines#index` would be `consumer/specs/project/pipelines/index.spec.js` | Follows the Rails naming standard. For example, `Project::Pipelines#index` would be `Pipelines#index` within the `project` folder. | +| Provider Test | Grouped like the Rails controllers. For example, [`GET list project pipelines` API endpoint](../../../api/pipelines.md#list-project-pipelines) would be `provider/pact_helpers/project/pipelines/provider/pact_helpers/project/pipelines/get_list_project_pipelines_helper.rb` | Follows the API documentation naming scheme in sentence case. For example, [`GET /projects/:id/pipelines`](../../../api/pipelines.md#list-project-pipelines) would be called `GET list project pipelines`. | diff --git a/doc/development/testing_guide/contract/provider_tests.md b/doc/development/testing_guide/contract/provider_tests.md index 1772ed9384e..3ce9f91a307 100644 --- a/doc/development/testing_guide/contract/provider_tests.md +++ b/doc/development/testing_guide/contract/provider_tests.md @@ -10,7 +10,7 @@ This tutorial guides you through writing a provider test from scratch. It is a c ## Create the skeleton -Provider tests are quite simple. The goal is to set up the test data and then link that with the corresponding contract. Start by creating a file called `discussions_helper.rb` under `spec/contracts/provider/pact_helpers/project/merge_request`. Note that the files are called `helpers` to match how they are called by Pact in the Rake tasks, which are set up at the end of this tutorial. +Provider tests are quite simple. The goal is to set up the test data and then link that with the corresponding contract. Start by creating a file called `get_discussions_helper.rb` under `spec/contracts/provider/pact_helpers/project/merge_request`. Note that the files are called `helpers` to match how they are called by Pact in the Rake tasks, which are set up at the end of this tutorial. To learn more about how the contract test directory is structured, see the contract testing [test suite folder structure](index.md#test-suite-folder-structure). @@ -23,7 +23,7 @@ require_relative '../../../spec_helper' module Provider module DiscussionsHelper - Pact.service_provider 'Merge Request Discussions Endpoint' do + Pact.service_provider 'GET discussions' do end end @@ -39,8 +39,8 @@ require_relative '../../../spec_helper' module Provider module DiscussionsHelper - Pact.service_provider 'Merge Request Discussions Endpoint' do - honours_pact_with 'MergeRequest#show' do + Pact.service_provider 'GET discussions' do + honours_pact_with 'MergeRequests#show' do end end @@ -59,10 +59,10 @@ require_relative '../../../spec_helper' module Provider module DiscussionsHelper - Pact.service_provider 'Merge Request Discussions Endpoint' do + Pact.service_provider 'GET discussions' do app { Environment::Test.app } - honours_pact_with 'MergeRequest#show' do + honours_pact_with 'MergeRequests#show' do end end @@ -79,11 +79,11 @@ require_relative '../../../spec_helper' module Provider module DiscussionsHelper - Pact.service_provider 'Merge Request Discussions Endpoint' do + Pact.service_provider 'GET discussions' do app { Environment::Test.app } - honours_pact_with 'MergeRequest#show' do - pact_uri '../contracts/project/merge_request/show/mergerequest#show-merge_request_discussions_endpoint.json' + honours_pact_with 'MergeRequests#show' do + pact_uri '../contracts/project/merge_requests/show/mergerequests#show-merge_request_discussions_endpoint.json' end end end @@ -92,20 +92,25 @@ end ## Add / update the Rake tasks -Now that you have a test created, you must create Rake tasks that run this test. The Rake tasks are defined in [`lib/tasks/contracts.rake`](https://gitlab.com/gitlab-org/gitlab/-/tree/master/lib/tasks/contracts.rake) where we have individual Rake tasks to run individual specs, but also Rake tasks that run a group of tests. +Now that you have a test created, you must create Rake tasks that run this test. The Rake tasks are defined in [`lib/tasks/contracts/merge_requests.rake`](https://gitlab.com/gitlab-org/gitlab/-/tree/master/lib/tasks/contracts/merge_requests.rake) where we have individual Rake tasks to run individual tests, but also Rake tasks that run a group of tests. -Under the `contracts:mr` namespace, introduce the Rake task to run this new test specifically. In it, call `pact.uri` to define the location of the contract and the provider test that tests that contract. Notice here that `pact_uri` has a parameter called `pact_helper`. This is why the provider tests are called `_helper.rb`. +Under the `contracts:merge_requests` namespace, introduce the Rake task to run this new test specifically. In it, call `pact.uri` to define the location of the contract and the provider test that tests that contract. Notice here that `pact_uri` has a parameter called `pact_helper`. This is why the provider tests are called `_helper.rb`. ```ruby -Pact::VerificationTask.new(:discussions) do |pact| +Pact::VerificationTask.new(:get_discussions) do |pact| + provider = File.expand_path('../../../spec/contracts/provider', __dir__) + pact_helper_location = "pact_helpers/project/merge_requests/show/get_discussions_helper.rb" + pact.uri( - "#{contracts}/contracts/project/merge_request/show/merge_request#show-merge_request_discussions_endpoint.json", - pact_helper: "#{provider}/pact_helpers/project/merge_request/discussions_helper.rb" + Provider::ContractSourceHelper.contract_location(:rake, pact_helper_location), + pact_helper: "#{provider}/#{pact_helper_location}" ) end ``` -At the same time, add your new `:discussions` Rake task to be included in the `test:merge_request` Rake task. In that Rake task, there is an array defined (`%w[metadata diffs]`). You must add `discussions` in that list. +[`Provider::ContractSourceHelper`](https://gitlab.com/gitlab-org/gitlab/-/tree/master/spec/contracts/provider/helpers/contract_source_helper.rb) is a helper module that has the `#contract_location` method which parses `pact_helper_location` and determines where the contract is stored locally or on the Pact Broker depending on the `requester` passed in. + +At the same time, add your new `:get_discussions` Rake task to be included in the `test:merge_requests` Rake task. In that Rake task, there is an array defined (`%w[get_diffs_batch get_diffs_metadata]`). You must add `get_discussions` in that list. ## Create test data @@ -113,7 +118,7 @@ As the last step, create the test data that allows the provider test to return t You can read more about [provider states](https://docs.pact.io/implementation_guides/ruby/provider_states). We can do global provider states but for this tutorial, the provider state is for one specific `state`. -To create the test data, create `discussions_state.rb` under `spec/contracts/provider/states/project/merge_request`. Be sure to also import this state file in the `discussions_helper.rb` file. +To create the test data, create `show_state.rb` under `spec/contracts/provider/states/project/merge_requests`. Be sure to also import this state file in the `get_discussions_helper.rb` file. ### Default user in `spec/contracts/provider/spec_helper.rb` @@ -141,7 +146,7 @@ Any further modifications to the user that's needed can be done through the indi In the state file, you must define which consumer this provider state is for. You can do that with `provider_states_for`. Make sure that the `name` provided matches the name defined for the consumer. ```ruby -Pact.provider_states_for 'MergeRequest#show' do +Pact.provider_states_for 'MergeRequests#show' do end ``` @@ -150,7 +155,7 @@ end In the `provider_states_for` block, you then define the state the test data is for. These states are also defined in the consumer test. In this case, there is a `'a merge request with discussions exists'` state. ```ruby -Pact.provider_states_for "MergeRequest#show" do +Pact.provider_states_for "MergeRequests#show" do provider_state "a merge request with discussions exists" do end @@ -162,7 +167,7 @@ end This is where you define the test data creation steps. Use `FactoryBot` to create the data. As you create the test data, you can keep [running the provider test](index.md#run-the-provider-tests) to check on the status of the test and figure out what else is missing in your data setup. ```ruby -Pact.provider_states_for "MergeRequest#show" do +Pact.provider_states_for "MergeRequests#show" do provider_state "a merge request with discussions exists" do set_up do user = User.find_by(name: Provider::UsersHelper::CONTRACT_USER_NAME) @@ -187,20 +192,19 @@ Now that the provider state file is created, you need to import the state file t # frozen_string_literal: true require_relative '../../../spec_helper' -require_relative '../../../states/project/merge_request/discussions_state' +require_relative '../../../states/project/merge_requests/show_state' module Provider module DiscussionsHelper - Pact.service_provider "/merge_request/discussions" do + Pact.service_provider "GET discussions" do app { Environments::Test.app } honours_pact_with 'Merge Request#show' do - pact_uri '../contracts/project/merge_request/show/merge_request#show-merge_request_discussions_endpoint.json' + pact_uri '../contracts/project/merge_requests/show/mergerequests#show-merge_request_discussions_endpoint.json' end end end end - ``` -And there we have it. The provider test for `discussions_helper.rb` should now pass with this. +And there we have it. The provider test for `get_discussions_helper.rb` should now pass with this. diff --git a/doc/development/testing_guide/end_to_end/index.md b/doc/development/testing_guide/end_to_end/index.md index 55d725ba4ae..8369dcb0740 100644 --- a/doc/development/testing_guide/end_to_end/index.md +++ b/doc/development/testing_guide/end_to_end/index.md @@ -207,11 +207,10 @@ For additional test results visibility, tests that run on pipelines generate and host [Allure](https://github.com/allure-framework/allure2) test reports. The `QA` framework is using the [Allure RSpec](https://github.com/allure-framework/allure-ruby/blob/master/allure-rspec/README.md) -gem to generate source files for the `Allure` test report. An additional job -in the pipeline: +gem to generate source files for the `Allure` test report. An additional job in the pipeline: - Fetches these source files from all test jobs. -- Generates and uploads the report to the `GCS` bucket `gitlab-qa-allure-report` under the project `gitlab-qa-resources`. +- Generates and uploads the report to the `S3` bucket `gitlab-qa-allure-report` located in `AWS` group project `eng-quality-ops-ci-cd-shared-infra`. A common CI template for report uploading is stored in [`allure-report.yml`](https://gitlab.com/gitlab-org/quality/pipeline-common/-/blob/master/ci/allure-report.yml). @@ -230,13 +229,13 @@ a link to the current test report. Each type of scheduled pipeline generates a static link for the latest test report according to its stage: -- [`master`](https://storage.googleapis.com/gitlab-qa-allure-reports/e2e-package-and-test/master/index.html) -- [`staging-full`](https://storage.googleapis.com/gitlab-qa-allure-reports/staging-full/master/index.html) -- [`staging-sanity`](https://storage.googleapis.com/gitlab-qa-allure-reports/staging-sanity/master/index.html) -- [`staging-sanity-no-admin`](https://storage.googleapis.com/gitlab-qa-allure-reports/staging-sanity-no-admin/master/index.html) -- [`canary-sanity`](https://storage.googleapis.com/gitlab-qa-allure-reports/canary-sanity/master/index.html) -- [`production`](https://storage.googleapis.com/gitlab-qa-allure-reports/production-full/master/index.html) -- [`production-sanity`](https://storage.googleapis.com/gitlab-qa-allure-reports/production-sanity/master/index.html) +- [`master`](http://gitlab-qa-allure-reports.s3.amazonaws.com/e2e-package-and-test/master/index.html) +- [`staging-full`](http://gitlab-qa-allure-reports.s3.amazonaws.com/staging-full/master/index.html) +- [`staging-sanity`](http://gitlab-qa-allure-reports.s3.amazonaws.com/staging-sanity/master/index.html) +- [`staging-sanity-no-admin`](http://gitlab-qa-allure-reports.s3.amazonaws.com/staging-sanity-no-admin/master/index.html) +- [`canary-sanity`](http://gitlab-qa-allure-reports.s3.amazonaws.com/canary-sanity/master/index.html) +- [`production`](http://gitlab-qa-allure-reports.s3.amazonaws.com/production-full/master/index.html) +- [`production-sanity`](http://gitlab-qa-allure-reports.s3.amazonaws.com/production-sanity/master/index.html) ## How do you run the tests? diff --git a/doc/development/testing_guide/end_to_end/resources.md b/doc/development/testing_guide/end_to_end/resources.md index 6e29e9b9dff..37d354bf60c 100644 --- a/doc/development/testing_guide/end_to_end/resources.md +++ b/doc/development/testing_guide/end_to_end/resources.md @@ -8,17 +8,9 @@ info: To determine the technical writer assigned to the Stage/Group associated w Resources are primarily created using Browser UI steps, but can also be created via the API or the CLI. -A typical resource class is used to create a new resource that can be used in a single test. However, several tests can -end up creating the same kind of resource and use it in ways that mean it could have been -used by more than one test. Creating a new resource each time is not efficient. Therefore, we can also create reusable -resources that are created once and can then be used by many tests. - -In the following section the content focuses on single-use resources, however it also applies to reusable resources. -Information specific to [reusable resources is detailed below](#reusable-resources). - ## How to properly implement a resource class? -All non-reusable resource classes should inherit from `Resource::Base`. +All resource classes should inherit from `Resource::Base`. There is only one mandatory method to implement to define a resource class. This is the `#fabricate!` method, which is used to build the resource via the @@ -398,176 +390,6 @@ end In this case, the result is similar to calling `Resource::Shirt.fabricate!`. -## Reusable resources - -Reusable resources are created by the first test that needs a particular kind of resource, and then any test that needs -the same kind of resource can reuse it instead of creating a new one. - -The `ReusableProject` resource is an example of this class: - -```ruby -module QA - module Resource - class ReusableProject < Project # A reusable resource inherits from the resource class that we want to be able to reuse. - prepend Reusable # The Reusable module mixes in some methods that help implement reuse. - - def initialize - super # A ReusableProject is a Project so it should be initialized as one. - - # Some Project attributes aren't valid and need to be overridden. For example, a ReusableProject keeps its name once it's created, - # so we don't add a random string to the name specified. - @add_name_uuid = false - - # It has a default name, and a different name can be specified when a resource is first created. However, the same name must be - # provided any time that instance of the resource is used. - @name = "reusable_project" - - # Several instances of a ReusableProject can exists as long as each is identified via a unique value for `reuse_as`. - @reuse_as = :default_project - end - - # All reusable resource classes must validate that an instance meets the conditions that allow reuse. For example, - # by confirming that the name specified for the instance is valid and doesn't conflict with other instances. - def validate_reuse_preconditions - raise ResourceReuseError unless reused_name_valid? - end - - # Internally we identify an instance of a reusable resource by a unique value of `@reuse_as`, but in GitLab the - # resource has one or more attributes that must also be unique. This method lists those attributes and allows the - # test framework to check that each instance of a reusable resource has values that match the associated values - # in Gitlab. - def unique_identifiers - [:name, :path] - end - end - end -end -``` - -Reusable resources aren't removed immediately when `remove_via_api!` is called. Instead, they're removed after the test -suite completes. To do so each class must be registered with `QA::Resource::ReusableCollection` in `qa/spec/spec_helper.rb` -as in the example below. Registration allows `QA::Resource::ReusableCollection` to keep track of each instance of each -registered class, and to delete them all in the `config.after(:suite)` hook. - -```ruby -config.before(:suite) do |suite| - QA::Resource::ReusableCollection.register_resource_classes do |collection| - QA::Resource::ReusableProject.register(collection) - end -end -``` - -Consider some examples of how a reusable resource is used: - -```ruby -# This will create a project. -default_project = Resource::ReusableProject.fabricate_via_api! -default_project.name # => "reusable_project" -default_project.reuse_as # => :default_project -``` - -Then in another test we could reuse the project: - -```ruby -# This will fetch the project created above rather than creating a new one. -default_project_again = Resource::ReusableProject.fabricate_via_api! -default_project_again.name # => "reusable_project" -default_project_again.reuse_as # => :default_project -``` - -We can also create another project that we want to change in a way that might not be suitable for tests using the -default project: - -```ruby -project_with_member = Resource::ReusableProject.fabricate_via_api! do |project| - project.name = "project-with-member" - project.reuse_as = :project_with_member -end - -project_with_member.add_member(user) -``` - -Another test can reuse that project: - -```ruby -project_still_has_member = Resource::ReusableProject.fabricate_via_api! do |project| - project.name = "project-with-member" - project.reuse_as = :project_with_member -end - -expect(project_still_has_member).to have_member(user) -``` - -However, if we don't provide the name again an error will be raised: - -```ruby -Resource::ReusableProject.fabricate_via_api! do |project| - project.reuse_as = :project_with_member -end - -# => ResourceReuseError will be raised because it will try to use the default name, "reusable_project", which doesn't -# match the name specified when the project was first fabricated. -``` - -### Validating reusable resources - -Reusable resources can speed up test suites by avoiding the cost of creating the same resource again and again. However, -that can cause problems if a test makes changes to a resource that prevent it from being reused as expected by later -tests. That can lead to order-dependent test failures that can be difficult to troubleshoot. - -For example, the default project created by `QA::Resource::ReusableProject` has `auto_devops_enabled` set to `false` -(inherited from `QA::Resource::Project`). If a test reuses that project and enables Auto DevOps, subsequent tests that reuse -the project will fail if they expect Auto DevOps to be disabled. - -We try to avoid that kind of trouble by validating reusable resources after a test suite. If the environment variable -`QA_VALIDATE_RESOURCE_REUSE` is set to `true` the test framework will check each reusable resource to verify that none -of the attributes they were created with have been changed. It does that by creating a new resource using the same -attributes that were used to create the original resource. It then compares the new resource to the original and raises -an error if any attributes don't match. - -#### Implementation - -When you implement a new type of reusable resource there are two `private` methods you must implement so the resource -can be validated. They are: - -- `reference_resource`: creates a new instance of the resource that can be compared with the one that was used during the tests. -- `unique_identifiers`: returns an array of attributes that allow the resource to be identified (for example, name) and that are therefore -expected to differ when comparing the reference resource with the resource reused in the tests. - -The following example shows the implementation of those two methods in `QA::Resource::ReusableProject`. - -```ruby -# Creates a new project that can be compared to a reused project, using the attributes of the original. -# -# @return [QA::Resource] a new instance of Resource::ReusableProject that should be a copy of the original resource -def reference_resource - # These are the attributes that the reused resource was created with - attributes = self.class.resources[reuse_as][:attributes] - - # Two projects can't have the same path, and since we typically use the same value for the name and path, we assign - # a unique name and path to the reference resource. - name = "reference_resource_#{SecureRandom.hex(8)}_for_#{attributes.delete(:name)}" - - Project.fabricate_via_api! do |project| - self.class.resources[reuse_as][:attributes].each do |attribute_name, attribute_value| - project.instance_variable_set("@#{attribute_name}", attribute_value) if attribute_value - end - project.name = name - project.path = name - project.path_with_namespace = "#{project.group.full_path}/#{project.name}" - end -end - -# The attributes of the resource that should be the same whenever a test wants to reuse a project. -# -# @return [Array<Symbol>] the attribute names. -def unique_identifiers - # As noted above, path must be unique, and since we typically use the same value for both, we treat name and path - # as unique. These attributes are ignored when we compare the reference and reused resources. - [:name, :path] -end -``` - ### Resources cleanup We have a mechanism to [collect](https://gitlab.com/gitlab-org/gitlab/-/blob/44345381e89d6bbd440f7b4c680d03e8b75b86de/qa/qa/tools/test_resource_data_processor.rb#L32) diff --git a/doc/development/testing_guide/frontend_testing.md b/doc/development/testing_guide/frontend_testing.md index 2fa5fdeab7d..85d807eceb1 100644 --- a/doc/development/testing_guide/frontend_testing.md +++ b/doc/development/testing_guide/frontend_testing.md @@ -1368,7 +1368,7 @@ You can also prefix this command with `WEBDRIVER_HEADLESS=0` which will run the ```ruby require 'spec_helper' ``` - + Import any other relevant module. 1. Create a global scope for RSpec to define our tests, just like what we do in jest with the initial describe block. @@ -1414,7 +1414,7 @@ Most feature tests at least require you to create a user, because you want to be This creates a variable that holds the newly created user and we can use `create` because we imported the `spec_helper`. -However, we have not done anything with this user yet because it's just a variable. So, in the `before do` block of the spec, we could sign in with the user so that every spec starts with a signed in user. +However, we have not done anything with this user yet because it's just a variable. So, in the `before do` block of the spec, we could sign in with the user so that every spec starts with an authenticated user. ```ruby let(:user) { create(:user) } diff --git a/doc/development/testing_guide/img/testing_triangle.png b/doc/development/testing_guide/img/testing_triangle.png Binary files differindex 3ac4955eaff..747bad1d52d 100644 --- a/doc/development/testing_guide/img/testing_triangle.png +++ b/doc/development/testing_guide/img/testing_triangle.png diff --git a/doc/development/testing_guide/testing_migrations_guide.md b/doc/development/testing_guide/testing_migrations_guide.md index b276a7e2a3a..1b1fdcca003 100644 --- a/doc/development/testing_guide/testing_migrations_guide.md +++ b/doc/development/testing_guide/testing_migrations_guide.md @@ -17,6 +17,8 @@ a database schema. - If your migration is a data migration then it **must** have a migration test. - Other migrations may have a migration test if necessary. +We don't enforce tests on post migrations that only perform schema changes. + ## How does it work? Adding a `:migration` tag to a test signature enables some custom RSpec |
