diff options
Diffstat (limited to 'doc/development/testing_guide')
16 files changed, 666 insertions, 33 deletions
diff --git a/doc/development/testing_guide/best_practices.md b/doc/development/testing_guide/best_practices.md index 7ae49d33e91..eda1c8c3d10 100644 --- a/doc/development/testing_guide/best_practices.md +++ b/doc/development/testing_guide/best_practices.md @@ -215,6 +215,39 @@ In this case, the `total time` and `top-level time` numbers match more closely: 8 8 0.0477s 0.0477s 0.0477s namespace ``` +#### Stubbing methods within factories + +You should avoid using `allow(object).to receive(:method)` in factories, as this makes the factory unable to be used with `let_it_be`. + +Instead, you can use `stub_method` to stub the method: + +```ruby + before(:create) do |user, evaluator| + # Stub a method. + stub_method(user, :some_method) { 'stubbed!' } + # Or with arguments, including named ones + stub_method(user, :some_method) { |var1| "Returning #{var1}!" } + stub_method(user, :some_method) { |var1: 'default'| "Returning #{var1}!" } + end + + # Un-stub the method. + # This may be useful where the stubbed object is created with `let_it_be` + # and you want to reset the method between tests. + after(:create) do |user, evaluator| + restore_original_method(user, :some_method) + # or + restore_original_methods(user) + end +``` + +NOTE: +`stub_method` does not work when used in conjunction with `let_it_be_with_refind`. This is because `stub_method` will stub a method on an instance and `let_it_be_with_refind` will create a new instance of the object for each run. + +`stub_method` does not support method existence and method arity checks. + +WARNING: +`stub_method` is supposed to be used in factories only. It's strongly discouraged to be used elsewhere. Please consider using [RSpec's mocks](https://relishapp.com/rspec/rspec-mocks/v/3-10/docs/basics) if available. + #### Identify slow tests Running a spec with profiling is a good way to start optimizing a spec. This can @@ -981,7 +1014,7 @@ is used to delete data in all indices in between examples to ensure a clean inde Note that Elasticsearch indexing uses [`Gitlab::Redis::SharedState`](../../../ee/development/redis.md#gitlabrediscachesharedstatequeues). Therefore, the Elasticsearch traits dynamically use the `:clean_gitlab_redis_shared_state` trait. -You do NOT need to add `:clean_gitlab_redis_shared_state` manually. +You do not need to add `:clean_gitlab_redis_shared_state` manually. Specs using Elasticsearch require that you: @@ -1305,7 +1338,7 @@ GitLab uses [factory_bot](https://github.com/thoughtbot/factory_bot) as a test f See [issue #262624](https://gitlab.com/gitlab-org/gitlab/-/issues/262624) for further context. - Factories don't have to be limited to `ActiveRecord` objects. [See example](https://gitlab.com/gitlab-org/gitlab-foss/commit/0b8cefd3b2385a21cfed779bd659978c0402766d). -- Factories and their traits should produce valid objects that are [verified by specs](https://gitlab.com/gitlab-org/gitlab/-/blob/master/spec/factories_spec.rb). +- Factories and their traits should produce valid objects that are [verified by specs](https://gitlab.com/gitlab-org/gitlab/-/blob/master/spec/models/factories_spec.rb). - Avoid the use of [`skip_callback`](https://api.rubyonrails.org/classes/ActiveSupport/Callbacks/ClassMethods.html#method-i-skip_callback) in factories. See [issue #247865](https://gitlab.com/gitlab-org/gitlab/-/issues/247865) for details. diff --git a/doc/development/testing_guide/contract/consumer_tests.md b/doc/development/testing_guide/contract/consumer_tests.md new file mode 100644 index 00000000000..b4d6882a655 --- /dev/null +++ b/doc/development/testing_guide/contract/consumer_tests.md @@ -0,0 +1,308 @@ +--- +stage: none +group: Development +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +--- + +# Writing consumer tests + +This tutorial guides you through writing a consumer test from scratch. To start, the consumer tests are written using [`jest-pact`](https://github.com/pact-foundation/jest-pact) that builds on top of [`pact-js`](https://github.com/pact-foundation/pact-js). This tutorial shows you how to write a consumer test for the `/discussions.json` endpoint, which is actually `/:namespace_name/:project_name/-/merge_requests/:id/discussions.json`. + +## Create the skeleton + +Start by creating the skeleton of a consumer test. Create a file under `spec/contracts/consumer/specs` called `discussions.spec.js`. +Then, populate it with the following function and parameters: + +- [`pactWith`](#the-pactwith-function) +- [`PactOptions`](#the-pactoptions-parameter) +- [`PactFn`](#the-pactfn-parameter) + +### The `pactWith` function + +The Pact consumer test is defined through the `pactWith` function that takes `PactOptions` and the `PactFn`. + +```javascript +const { pactWith } = require('jest-pact'); + +pactWith(PactOptions, PactFn); +``` + +### The `PactOptions` parameter + +`PactOptions` with `jest-pact` introduces [additional options](https://github.com/pact-foundation/jest-pact/blob/dce370c1ab4b7cb5dff12c4b62246dc229c53d0e/README.md#defaults) that build on top of the ones [provided in `pact-js`](https://github.com/pact-foundation/pact-js#constructor). In most cases, you define the `consumer`, `provider`, `log`, and `dir` options for these tests. + +```javascript +const { pactWith } = require('jest-pact'); + +pactWith( + { + consumer: 'Merge Request Page', + provider: 'Merge Request Discussions Endpoint', + log: '../logs/consumer.log', + dir: '../contracts', + }, + PactFn +); +``` + +### The `PactFn` parameter + +The `PactFn` is where your tests are defined. This is where you set up the mock provider and where you can use the standard Jest methods like [`Jest.describe`](https://jestjs.io/docs/api#describename-fn), [`Jest.beforeEach`](https://jestjs.io/docs/api#beforeeachfn-timeout), and [`Jest.it`](https://jestjs.io/docs/api#testname-fn-timeout). For more information, see [https://jestjs.io/docs/api](https://jestjs.io/docs/api). + +```javascript +const { pactWith } = require('jest-pact'); + +pactWith( + { + consumer: 'Merge Request Page', + provider: 'Merge Request Discussions Endpoint', + log: '../logs/consumer.log', + dir: '../contracts', + }, + + (provider) => { + describe('Discussions Endpoint', () => { + beforeEach(() => { + + }); + + it('return a successful body', () => { + + }); + }); + }, +); +``` + +## Set up the mock provider + +Before you run your test, set up the mock provider that handles the specified requests and returns a specified response. To do that, define the state and the expected request and response in an [`Interaction`](https://github.com/pact-foundation/pact-js/blob/master/src/dsl/interaction.ts). + +For this tutorial, define four attributes for the `Interaction`: + +1. `state`: A description of what the prerequisite state is before the request is made. +1. `uponReceiving`: A description of what kind of request this `Interaction` is handling. +1. `withRequest`: Where you define the request specifications. It contains the request `method`, `path`, and any `headers`, `body`, or `query`. +1. `willRespondWith`: Where you define the expected response. It contains the response `status`, `headers`, and `body`. + +After you define the `Interaction`, add that interaction to the mock provider by calling `addInteraction`. + +```javascript +const { pactWith } = require('jest-pact'); +const { Matchers } = require('@pact-foundation/pact'); + +pactWith( + { + consumer: 'Merge Request Page', + provider: 'Merge Request Discussions Endpoint', + log: '../logs/consumer.log', + dir: '../contracts', + }, + + (provider) => { + describe('Discussions Endpoint', () => { + beforeEach(() => { + const interaction = { + state: 'a merge request with discussions exists', + uponReceiving: 'a request for discussions', + withRequest: { + method: 'GET', + path: '/gitlab-org/gitlab-qa/-/merge_requests/1/discussions.json', + headers: { + Accept: '*/*', + }, + }, + willRespondWith: { + status: 200, + headers: { + 'Content-Type': 'application/json; charset=utf-8', + }, + body: Matchers.eachLike({ + id: Matchers.string('fd73763cbcbf7b29eb8765d969a38f7d735e222a'), + project_id: Matchers.integer(6954442), + ... + resolved: Matchers.boolean(true) + }), + }, + }; + provider.addInteraction(interaction); + }); + + it('return a successful body', () => { + + }); + }); + }, +); +``` + +### Response body `Matchers` + +Notice how we use `Matchers` in the `body` of the expected response. This allows us to be flexible enough to accept different values but still be strict enough to distinguish between valid and invalid values. We must ensure that we have a tight definition that is neither too strict nor too lax. Read more about the [different types of `Matchers`](https://github.com/pact-foundation/pact-js#using-the-v3-matching-rules). + +## Write the test + +After the mock provider is set up, you can write the test. For this test, you make a request and expect a particular response. + +First, set up the client that makes the API request. To do that, either create or find an existing file under `spec/contracts/consumer/endpoints` and add the following API request. + +```javascript +const axios = require('axios'); + +exports.getDiscussions = (endpoint) => { + const url = endpoint.url; + + return axios + .request({ + method: 'GET', + baseURL: url, + url: '/gitlab-org/gitlab-qa/-/merge_requests/1/discussions.json', + headers: { Accept: '*/*' }, + }) + .then((response) => response.data); +}; +``` + +After that's set up, import it to the test file and call it to make the request. Then, you can make the request and define your expectations. + +```javascript +const { pactWith } = require('jest-pact'); +const { Matchers } = require('@pact-foundation/pact'); + +const { getDiscussions } = require('../endpoints/merge_requests'); + +pactWith( + { + consumer: 'Merge Request Page', + provider: 'Merge Request Discussions Endpoint', + log: '../logs/consumer.log', + dir: '../contracts', + }, + + (provider) => { + describe('Discussions Endpoint', () => { + beforeEach(() => { + const interaction = { + state: 'a merge request with discussions exists', + uponReceiving: 'a request for discussions', + withRequest: { + method: 'GET', + path: '/gitlab-org/gitlab-qa/-/merge_requests/1/discussions.json', + headers: { + Accept: '*/*', + }, + }, + willRespondWith: { + status: 200, + headers: { + 'Content-Type': 'application/json; charset=utf-8', + }, + body: Matchers.eachLike({ + id: Matchers.string('fd73763cbcbf7b29eb8765d969a38f7d735e222a'), + project_id: Matchers.integer(6954442), + ... + resolved: Matchers.boolean(true) + }), + }, + }; + }); + + it('return a successful body', () => { + return getDiscussions({ + url: provider.mockService.baseUrl, + }).then((discussions) => { + expect(discussions).toEqual(Matchers.eachLike({ + id: 'fd73763cbcbf7b29eb8765d969a38f7d735e222a', + project_id: 6954442, + ... + resolved: true + })); + }); + }); + }); + }, +); +``` + +There we have it! The consumer test is now set up. You can now try [running this test](index.md#run-the-consumer-tests). + +## Improve test readability + +As you may have noticed, the request and response definitions can get large. This results in the test being difficult to read, with a lot of scrolling to find what you want. You can make the test easier to read by extracting these out to a `fixture`. + +Create a file under `spec/contracts/consumer/fixtures` called `discussions.fixture.js`. You place the `request` and `response` definitions here. + +```javascript +const { Matchers } = require('@pact-foundation/pact'); + +const body = Matchers.eachLike({ + id: Matchers.string('fd73763cbcbf7b29eb8765d969a38f7d735e222a'), + project_id: Matchers.integer(6954442), + ... + resolved: Matchers.boolean(true) +}); + +const Discussions = { + body: Matchers.extractPayload(body), + + success: { + status: 200, + headers: { + 'Content-Type': 'application/json; charset=utf-8', + }, + body: body, + }, + + request: { + uponReceiving: 'a request for discussions', + withRequest: { + method: 'GET', + path: '/gitlab-org/gitlab-qa/-/merge_requests/1/discussions.json', + headers: { + Accept: '*/*', + }, + }, + }, +}; + +exports.Discussions = Discussions; +``` + +With all of that moved to the `fixture`, you can simplify the test to the following: + +```javascript +const { pactWith } = require('jest-pact'); + +const { Discussions } = require('../fixtures/discussions.fixture'); +const { getDiscussions } = require('../endpoints/merge_requests'); + +pactWith( + { + consumer: 'Merge Request Page', + provider: 'Merge Request Discussions Endpoint', + log: '../logs/consumer.log', + dir: '../contracts', + }, + + (provider) => { + describe('Discussions Endpoint', () => { + beforeEach(() => { + const interaction = { + state: 'a merge request with discussions exists', + ...Discussions.request, + willRespondWith: Discussions.success, + }; + return provider.addInteraction(interaction); + }); + + it('return a successful body', () => { + return getDiscussions({ + url: provider.mockService.baseUrl, + }).then((discussions) => { + expect(discussions).toEqual(Discussions.body); + }); + }); + }); + }, +); +``` diff --git a/doc/development/testing_guide/contract/index.md b/doc/development/testing_guide/contract/index.md new file mode 100644 index 00000000000..6556bd85624 --- /dev/null +++ b/doc/development/testing_guide/contract/index.md @@ -0,0 +1,39 @@ +--- +stage: none +group: Development +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +--- + +# Contract testing + +Contract tests consist of two parts — consumer tests and provider tests. A simple example of a consumer and provider relationship is between the frontend and backend. The frontend would be the consumer and the backend is the provider. The frontend consumes the API that is provided by the backend. The test helps ensure that these two sides follow an agreed upon contract and any divergence from the contract triggers a meaningful conversation to prevent breaking changes from slipping through. + +Consumer tests are similar to unit tests with each spec defining a requests and an expected mock responses and creating a contract based on those definitions. On the other hand, provider tests are similar to integration tests as each spec takes the request defined in the contract and runs that request against the actual service which is then matched against the contract to validate the contract. + +You can check out the existing contract tests at: + +- [`spec/contracts/consumer/specs`](https://gitlab.com/gitlab-org/gitlab/-/tree/master/spec/contracts/consumer/specs) for the consumer tests. +- [`spec/contracts/provider/specs`](https://gitlab.com/gitlab-org/gitlab/-/tree/master/spec/contracts/provider/specs) for the provider tests. + +The contracts themselves are stored in [`/spec/contracts/contracts`](https://gitlab.com/gitlab-org/gitlab/-/tree/master/spec/contracts/contracts) at the moment. The plan is to use [PactBroker](https://docs.pact.io/pact_broker/docker_images) hosted in AWS or another similar service. + +## Write the tests + +- [Writing consumer tests](consumer_tests.md) +- [Writing provider tests](provider_tests.md) + +### Run the consumer tests + +Before running the consumer tests, go to `spec/contracts/consumer` and run `npm install`. To run all the consumer tests, you just need to run `npm test -- /specs`. Otherwise, to run a specific spec file, replace `/specs` with the specific spec filename. + +### Run the provider tests + +Before running the provider tests, make sure your GDK (GitLab Development Kit) is fully set up and running. You can follow the setup instructions detailed in the [GDK repository](https://gitlab.com/gitlab-org/gitlab-development-kit/-/tree/main). To run the provider tests, you use Rake tasks that are defined in [`./lib/tasks/contracts.rake`](https://gitlab.com/gitlab-org/gitlab/-/tree/master/lib/tasks/contracts.rake). To get a list of all the Rake tasks related to the provider tests, run `bundle exec rake -T contracts`. For example: + +```shell +$ bundle exec rake -T contracts +rake contracts:mr:pact:verify:diffs # Verify provider against the consumer pacts for diffs +rake contracts:mr:pact:verify:discussions # Verify provider against the consumer pacts for discussions +rake contracts:mr:pact:verify:metadata # Verify provider against the consumer pacts for metadata +rake contracts:mr:test:merge_request[contract_mr] # Run all merge request contract tests +``` diff --git a/doc/development/testing_guide/contract/provider_tests.md b/doc/development/testing_guide/contract/provider_tests.md new file mode 100644 index 00000000000..0da5bcb4aef --- /dev/null +++ b/doc/development/testing_guide/contract/provider_tests.md @@ -0,0 +1,177 @@ +--- +stage: none +group: Development +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +--- + +# Writing provider tests + +This tutorial guides you through writing a provider test from scratch. It is a continuation of the [consumer test tutorial](consumer_tests.md). To start, the provider tests are written using [`pact-ruby`](https://github.com/pact-foundation/pact-ruby). In this tutorial, you write a provider test that addresses the contract generated by `discussions.spec.js`. + +## Create the skeleton + +Provider tests are quite simple. The goal is to set up the test data and then link that with the corresponding contract. Start by creating a file called `discussions_helper.rb` under `spec/contracts/provider/specs`. Note that the files are called `helpers` to match how they are called by Pact in the Rake tasks, which are set up at the end of this tutorial. + +### The `service_provider` block + +The `service_provider` block is where the provider test is defined. For this block, put in a description of the service provider. Name it exactly as it is called in the contracts that are derived from the consumer tests. + +```ruby +require_relative '../spec_helper' + +module Provider + module DiscussionsHelper + Pact.service_provider 'Merge Request Discussions Endpoint' do + + end + end +end +``` + +### The `honours_pact_with` block + +The `honours_pact_with` block describes which consumer this provider test is addressing. Similar to the `service_provider` block, name this exactly the same as it's called in the contracts that are derived from the consumer tests. + +```ruby +require_relative '../spec_helper' + +module Provider + module DiscussionsHelper + Pact.service_provider 'Merge Request Discussions Endpoint' do + honours_pact_with 'Merge Request Page' do + + end + end + end +end +``` + +## Configure the test app + +For the provider tests to verify the contracts, you must hook it up to a test app that makes the actual request and return a response to verify against the contract. To do this, configure the `app` the test uses as `Environment::Test.app`, which is defined in [`spec/contracts/provider/environments/test.rb`](https://gitlab.com/gitlab-org/gitlab/-/tree/master/spec/contracts/provider/environments/test.rb). + +```ruby +require_relative '../spec_helper' + +module Provider + module DiscussionsHelper + Pact.service_provider 'Merge Request Discussions Endpoint' do + app { Environment::Test.app } + + honours_pact_with 'Merge Request Page' do + + end + end + end +end +``` + +## Define the contract to verify + +Now that the test app is configured, all that is left is to define which contract this provider test is verifying. To do this, set the `pact_uri`. + +```ruby +require_relative '../spec_helper' + +module Provider + module DiscussionsHelper + Pact.service_provider 'Merge Request Discussions Endpoint' do + app { Environment::Test.app } + + honours_pact_with 'Merge Request Page' do + pact_uri '../contracts/merge_request_page-merge_request_discussions_endpoint.json' + end + end + end +end +``` + +## Add / update the Rake tasks + +Now that you have a test created, you must create Rake tasks that run this test. The Rake tasks are defined in [`lib/tasks/contracts.rake`](https://gitlab.com/gitlab-org/gitlab/-/tree/master/lib/tasks/contracts.rake) where we have individual Rake tasks to run individual specs, but also Rake tasks that run a group of tests. + +Under the `contracts:mr` namespace, introduce the Rake task to run this new test specifically. In it, call `pact.uri` to define the location of the contract and the provider test that tests that contract. Notice here that `pact_uri` has a parameter called `pact_helper`. This is why the provider tests are called `_helper.rb`. + +```ruby +Pact::VerificationTask.new(:discussions) do |pact| + pact.uri( + "#{contracts}/contracts/merge_request_page-merge_request_discussions_endpoint.json", + pact_helper: "#{provider}/specs/discussions_helper.rb" + ) +end +``` + +At the same time, add your new `:discussions` Rake task to be included in the `test:merge_request` Rake task. In that Rake task, there is an array defined (`%w[metadata diffs]`). You must add `discussions` in that list. + +## Create test data + +As the last step, create the test data that allows the provider test to return the contract's expected response. You might wonder why you create the test data last. It's really a matter of preference. With the test already configured, you can easily run the test to verify and make sure all the necessary test data are created to produce the expected response. + +You can read more about [provider states](https://docs.pact.io/implementation_guides/ruby/provider_states). We can do global provider states but for this tutorial, the provider state is for one specific `state`. + +To create the test data, create `discussions_state.rb` under `spec/contracts/provider/states`. As a quick aside, make sure to also import this state file in the `discussions_helper.rb` file. + +### Default user in `spec/contracts/provider/spec_helper.rb` + +Before you create the test data, note that a default user is created in the [`spec_helper`](https://gitlab.com/gitlab-org/gitlab/-/tree/master/spec/contracts/provider/spec_helper.rb), which is the user being used for the test runs. This user is configured using `RSpec.configure`, as Pact actually is built on top of RSpec. This step allows us to configure the user before any of the test runs. + +```ruby +RSpec.configure do |config| + config.include Devise::Test::IntegrationHelpers + config.before do + user = FactoryBot.create(:user, name: "Contract Test").tap do |user| + user.current_sign_in_at = Time.current + end + sign_in user + end +end +``` + +Any further modifications to the user that's needed can be done through the individual provider state files. + +### The `provider_states_for` block + +In the state file, you must define which consumer this provider state is for. You can do that with `provider_states_for`. Make sure that the `name` provided matches the name defined for the consumer. + +```ruby +Pact.provider_states_for 'Merge Request Page' do +end +``` + +### The `provider_state` block + +In the `provider_states_for` block, you then define the state the test data is for. These states are also defined in the consumer test. In this case, there is a `'a merge request with discussions exists'` state. + +```ruby +Pact.provider_states_for "Merge Request Page" do + provider_state "a merge request with discussions exists" do + + end +end +``` + +### The `set_up` block + +This is where you define the test data creation steps. Use `FactoryBot` to create the data. As you create the test data, you can keep [running the provider test](index.md#run-the-provider-tests) to check on the status of the test and figure out what else is missing in your data setup. + +```ruby +Pact.provider_states_for "Merge Request Page" do + provider_state "a merge request with discussions exists" do + set_up do + user = User.find_by(name: Provider::UsersHelper::CONTRACT_USER_NAME) + namespace = create(:namespace, name: 'gitlab-org') + project = create(:project, name: 'gitlab-qa', namespace: namespace) + + project.add_maintainer(user) + + merge_request = create(:merge_request_with_diffs, id: 1, source_project: project, author: user) + + create(:discussion_note_on_merge_request, noteable: merge_request, project: project, author: user) + end + end +end +``` + +Note the `Provider::UsersHelper::CONTRACT_USER_NAME` here to fetch a user is a user that is from the [`spec_helper`](https://gitlab.com/gitlab-org/gitlab/-/tree/master/spec/contracts/provider/spec_helper.rb) that sets up a user before any of these tests run. + +And with that, the provider tests for `discussion_helper.rb` should now pass with this. diff --git a/doc/development/testing_guide/end_to_end/beginners_guide.md b/doc/development/testing_guide/end_to_end/beginners_guide.md index 39a3e3445ea..a13011d0101 100644 --- a/doc/development/testing_guide/end_to_end/beginners_guide.md +++ b/doc/development/testing_guide/end_to_end/beginners_guide.md @@ -343,8 +343,8 @@ Before running the spec, make sure that: To run the spec, run the following command: -```ruby -GITLAB_PASSWORD=<GDK root password> bundle exec bin/qa Test::Instance::All http://localhost:3000 -- <test_file> +```shell +GITLAB_PASSWORD=<GDK root password> bundle exec rspec <test_file> ``` Where `<test_file>` is: @@ -352,6 +352,8 @@ Where `<test_file>` is: - `qa/specs/features/browser_ui/1_manage/login/log_in_spec.rb` when running the Login example. - `qa/specs/features/browser_ui/2_plan/issue/create_issue_spec.rb` when running the Issue example. +Additional information on test execution and possible options are described in ["QA framework README"](https://gitlab.com/gitlab-org/gitlab/-/blob/master/qa/README.md#run-the-end-to-end-tests-in-a-local-development-environment) + ## End-to-end test merge request template When submitting a new end-to-end test, use the ["New End to End Test"](https://gitlab.com/gitlab-org/gitlab/-/blob/master/.gitlab/merge_request_templates/New%20End%20To%20End%20Test.md) diff --git a/doc/development/testing_guide/end_to_end/best_practices.md b/doc/development/testing_guide/end_to_end/best_practices.md index bd9896934c7..85f8beeacad 100644 --- a/doc/development/testing_guide/end_to_end/best_practices.md +++ b/doc/development/testing_guide/end_to_end/best_practices.md @@ -189,9 +189,9 @@ Attach the `:aggregate_failures` metadata to the example if multiple expectation it 'searches', :aggregate_failures do Page::Search::Results.perform do |search| expect(search).to have_file_in_project(template[:file_name], project.name) - + search.switch_to_code - + expect(search).to have_file_with_content(template[:file_name], content[0..33]) end end @@ -208,6 +208,54 @@ it 'searches' do end ``` +## Avoid multiple actions in `expect do ... raise_error` blocks + +When you wrap multiple actions in a single `expect do ... end.not_to raise_error` or `expect do ... end.to raise_error` block, +it can be hard to debug the actual cause of the failure, because of how the logs are printed. Important information can be truncated +or missing altogether. + +For example, if you encapsulate some actions and expectations in a private method in the test, like `expect_owner_permissions_allow_delete_issue`: + +```ruby +it "has Owner role with Owner permissions" do + Page::Dashboard::Projects.perform do |projects| + projects.filter_by_name(project.name) + + expect(projects).to have_project_with_access_role(project.name, 'Owner') + end + + expect_owner_permissions_allow_delete_issue +end +``` + +Then, in the method itself: + +```ruby +#=> Good +def expect_owner_permissions_allow_delete_issue + issue.visit! + + Page::Project::Issue::Show.perform(&:delete_issue) + + Page::Project::Issue::Index.perform do |index| + expect(index).not_to have_issue(issue) + end +end + +#=> Bad +def expect_owner_permissions_allow_delete_issue + expect do + issue.visit! + + Page::Project::Issue::Show.perform(&:delete_issue) + + Page::Project::Issue::Index.perform do |index| + expect(index).not_to have_issue(issue) + end + end.not_to raise_error +end +``` + ## Prefer to split tests across multiple files Our framework includes a couple of parallelization mechanisms that work by executing spec files in parallel. diff --git a/doc/development/testing_guide/end_to_end/capybara_to_chemlab_migration_guide.md b/doc/development/testing_guide/end_to_end/capybara_to_chemlab_migration_guide.md index 9c7e0ef73a8..a71e076b57f 100644 --- a/doc/development/testing_guide/end_to_end/capybara_to_chemlab_migration_guide.md +++ b/doc/development/testing_guide/end_to_end/capybara_to_chemlab_migration_guide.md @@ -12,7 +12,7 @@ Given the view: ```html <form id="my-form"> - <label for="first-name">First name</label> + <label for="first-name">First name</label> <input type="text" name="first-name" data-qa-selector="first_name" /> <label for="last-name">Last name</label> @@ -26,7 +26,7 @@ Given the view: <label for="password">Password</label> <input type="password" name="password" data-qa-selector="password" /> - + <input type="submit" value="Continue" data-qa-selector="continue"/> </form> ``` diff --git a/doc/development/testing_guide/end_to_end/feature_flags.md b/doc/development/testing_guide/end_to_end/feature_flags.md index b4ec9e8ccd3..cb4c8e8a6e8 100644 --- a/doc/development/testing_guide/end_to_end/feature_flags.md +++ b/doc/development/testing_guide/end_to_end/feature_flags.md @@ -30,7 +30,7 @@ feature flag is under test. - Format: `feature_flag: { name: 'feature_flag_name', scope: :project }` - When `scope` is set to `:global`, the test will be **skipped on all live .com environments**. This is to avoid issues with feature flag changes affecting other tests or users on that environment. -- When `scope` is set to any other value (such as `:project`, `:group` or `:user`), or if no `scope` is specified, the test will only be **skipped on canary and production**. +- When `scope` is set to any other value (such as `:project`, `:group` or `:user`), or if no `scope` is specified, the test will only be **skipped on canary, production, and preprod**. This is due to the fact that administrator access is not available there. **WARNING:** You are strongly advised to first try and [enable feature flags only for a group, project, user](../../feature_flags/index.md#feature-actors), @@ -192,10 +192,26 @@ End-to-end tests should pass with a feature flag enabled before it is enabled on ### Automatic test execution when a feature flag definition changes -If a merge request adds or edits a [feature flag definition file](../../feature_flags/index.md#feature-flag-definition-and-validation), -two `package-and-qa` jobs will be included automatically in the merge request pipeline. One job will enable the defined -feature flag and the other will disable it. The jobs execute the same suite of tests to confirm that they pass with if -the feature flag is either enabled or disabled. +There are two ways to confirm that end-to-end tests pass: + +- If a merge request adds or edits a [feature flag definition file](../../feature_flags/index.md#feature-flag-definition-and-validation), + two `package-and-qa` jobs (`package-and-qa-ff-enabled` and `package-and-qa-ff-disabled`) are included automatically in the merge request + pipeline. One job enables the defined feature flag and the other job disables it. The jobs execute the same suite of tests to confirm + that they pass with the feature flag either enabled or disabled. +- In some cases, if `package-and-qa` hasn't been triggered automatically, or if it has run the tests with the default feature flag values + (which might not be desired), you can create a Draft MR that enables the feature flag to ensure that all E2E tests pass with the feature + flag enabled. + +### Troubleshooting end-to-end test failures with feature flag enabled + +If enabling the feature flag results in E2E test failures, you can browse the artifacts in the failed pipeline to see screenshots of the failed tests. After which, you can either: + +- Identify tests that need to be updated and contact the relevant [counterpart Software Engineer in Test](https://about.gitlab.com/handbook/engineering/quality/#individual-contributors) responsible for updating the tests or assisting another engineer to do so. However, if a change does not go through [quad-planning](https://about.gitlab.com/handbook/engineering/quality/quality-engineering/quad-planning/) and a required test update is not made, test failures could block deployment. +- Run the failed tests [locally](https://gitlab.com/gitlab-org/gitlab/-/tree/master/qa#run-the-end-to-end-tests-in-a-local-development-environment) + with the [feature flag enabled](https://gitlab.com/gitlab-org/gitlab/-/tree/master/qa#running-tests-with-a-feature-flag-enabled-or-disabled). + This option requires considerable amount of setup, but you'll be able to see what the browser is doing as it's running the failed + tests, which can help debug the problem faster. You can also refer to the [Troubleshooting Guide for E2E tests](troubleshooting.md) for + support for common blockers. ### Test execution during feature development diff --git a/doc/development/testing_guide/end_to_end/page_objects.md b/doc/development/testing_guide/end_to_end/page_objects.md index 85ab4d479f9..c93e8c9d13f 100644 --- a/doc/development/testing_guide/end_to_end/page_objects.md +++ b/doc/development/testing_guide/end_to_end/page_objects.md @@ -150,7 +150,7 @@ In our case, `data-qa-selector="login_field"`, `data-qa-selector="password_field ```haml = f.text_field :login, class: "form-control top", autofocus: "autofocus", autocapitalize: "off", autocorrect: "off", required: true, title: "This field is required.", data: { qa_selector: 'login_field' } = f.password_field :password, class: "form-control bottom", required: true, title: "This field is required.", data: { qa_selector: 'password_field' } -= f.submit "Sign in", class: "btn btn-success", data: { qa_selector: 'sign_in_button' } += f.submit "Sign in", class: "btn btn-confirm", data: { qa_selector: 'sign_in_button' } ``` Things to note: diff --git a/doc/development/testing_guide/end_to_end/resources.md b/doc/development/testing_guide/end_to_end/resources.md index e2b005d8a1b..dacc428aec6 100644 --- a/doc/development/testing_guide/end_to_end/resources.md +++ b/doc/development/testing_guide/end_to_end/resources.md @@ -531,7 +531,7 @@ When you implement a new type of reusable resource there are two `private` metho can be validated. They are: - `reference_resource`: creates a new instance of the resource that can be compared with the one that was used during the tests. -- `unique_identifiers`: returns an array of attributes that allow the resource to be identified (e.g., name) and that are therefore +- `unique_identifiers`: returns an array of attributes that allow the resource to be identified (for example, name) and that are therefore expected to differ when comparing the reference resource with the resource reused in the tests. The following example shows the implementation of those two methods in `QA::Resource::ReusableProject`. diff --git a/doc/development/testing_guide/end_to_end/rspec_metadata_tests.md b/doc/development/testing_guide/end_to_end/rspec_metadata_tests.md index 0163f2e648c..591d03db7b8 100644 --- a/doc/development/testing_guide/end_to_end/rspec_metadata_tests.md +++ b/doc/development/testing_guide/end_to_end/rspec_metadata_tests.md @@ -15,13 +15,14 @@ This is a partial list of the [RSpec metadata](https://relishapp.com/rspec/rspec |-----------------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | `:elasticsearch` | The test requires an Elasticsearch service. It is used by the [instance-level scenario](https://gitlab.com/gitlab-org/gitlab-qa#definitions) [`Test::Integration::Elasticsearch`](https://gitlab.com/gitlab-org/gitlab/-/blob/72b62b51bdf513e2936301cb6c7c91ec27c35b4d/qa/qa/ee/scenario/test/integration/elasticsearch.rb) to include only tests that require Elasticsearch. | | `:except` | The test is to be run in their typical execution contexts _except_ as specified. See [test execution context selection](execution_context_selection.md) for more information. | -| `:feature_flag` | The test uses a feature flag and therefore requires an administrator account to run. When `scope` is set to `:global`, the test will be skipped on all live .com environments. Otherwise, it will be skipped only on Canary and Production. See [testing with feature flags](../../../development/testing_guide/end_to_end/feature_flags.md) for more details. | +| `:feature_flag` | The test uses a feature flag and therefore requires an administrator account to run. When `scope` is set to `:global`, the test will be skipped on all live .com environments. Otherwise, it will be skipped only on Canary, Production, and Preprod. See [testing with feature flags](../../../development/testing_guide/end_to_end/feature_flags.md) for more details. | | `:geo` | The test requires two GitLab Geo instances - a primary and a secondary - to be spun up. | | `:gitaly_cluster` | The test runs against a GitLab instance where repositories are stored on redundant Gitaly nodes behind a Praefect node. All nodes are [separate containers](../../../administration/gitaly/praefect.md#requirements). Tests that use this tag have a longer setup time since there are three additional containers that need to be started. | | `:github` | The test requires a GitHub personal access token. | | `:group_saml` | The test requires a GitLab instance that has SAML SSO enabled at the group level. Interacts with an external SAML identity provider. Paired with the `:orchestrated` tag. | | `:instance_saml` | The test requires a GitLab instance that has SAML SSO enabled at the instance level. Interacts with an external SAML identity provider. Paired with the `:orchestrated` tag. | | `:integrations` | This aims to test the available [integrations](../../../user/project/integrations/index.md#available-integrations). The test requires Docker to be installed in the run context. It will provision the containers and can be run against a local instance or using the `gitlab-qa` scenario `Test::Integration::Integrations` | +| `:issue`, `:issue_${num}` | Optional links to issues which might be related to the spec. Helps keep track of related issues and can also be used by tools that create test reports. Currently added automatically to `Allure` test report. Multiple tags can be used by adding an optional numeric suffix like `issue_1`, `issue_2` etc. | | `:service_ping_disabled` | The test interacts with the GitLab configuration service ping at the instance level to turn Admin Area setting service ping checkbox on or off. This tag will have the test run only in the `service_ping_disabled` job and must be paired with the `:orchestrated` and `:requires_admin` tags. | | `:jira` | The test requires a Jira Server. [GitLab-QA](https://gitlab.com/gitlab-org/gitlab-qa) provisions the Jira Server in a Docker container when the `Test::Integration::Jira` test scenario is run. | | `:kubernetes` | The test includes a GitLab instance that is configured to be run behind an SSH tunnel, allowing a TLS-accessible GitLab. This test also includes provisioning of at least one Kubernetes cluster to test against. _This tag is often be paired with `:orchestrated`._ | @@ -42,6 +43,7 @@ This is a partial list of the [RSpec metadata](https://relishapp.com/rspec/rspec | `:requires_git_protocol_v2` | The test requires that Git protocol version 2 is enabled on the server. It's assumed to be enabled by default but if not the test can be skipped by setting `QA_CAN_TEST_GIT_PROTOCOL_V2` to `false`. | | `:requires_praefect` | The test requires that the GitLab instance uses [Gitaly Cluster](../../../administration/gitaly/praefect.md) (a.k.a. Praefect) as the repository storage . It's assumed to be used by default but if not the test can be skipped by setting `QA_CAN_TEST_PRAEFECT` to `false`. | | `:runner` | The test depends on and sets up a GitLab Runner instance, typically to run a pipeline. | +| `:sanity_feature_flags` | The test verifies the functioning of the feature flag handling part of the test framework | | `:skip_live_env` | The test is excluded when run against live deployed environments such as Staging, Canary, and Production. | | `:skip_fips_env` | The test is excluded when run against an environment in FIPS mode. | | `:skip_signup_disabled` | The test uses UI to sign up a new user and is skipped in any environment that does not allow new user registration via the UI. | @@ -49,4 +51,3 @@ This is a partial list of the [RSpec metadata](https://relishapp.com/rspec/rspec | `:smtp` | The test requires a GitLab instance to be configured to use an SMTP server. Tests SMTP notification email delivery from GitLab by using MailHog. | | `:testcase` | The link to the test case issue in the [GitLab Project Test Cases](https://gitlab.com/gitlab-org/gitlab/-/quality/test_cases). | | `:transient` | The test tests transient bugs. It is excluded by default. | -| `:issue`, `:issue_${num}` | Optional links to issues which might be related to the spec. Helps keep track of related issues and can also be used by tools that create test reports. Currently added automatically to `Allure` test report. Multiple tags can be used by adding an optional numeric suffix like `issue_1`, `issue_2` etc. | diff --git a/doc/development/testing_guide/end_to_end/running_tests_that_require_special_setup.md b/doc/development/testing_guide/end_to_end/running_tests_that_require_special_setup.md index 599e1104b72..438294161ac 100644 --- a/doc/development/testing_guide/end_to_end/running_tests_that_require_special_setup.md +++ b/doc/development/testing_guide/end_to_end/running_tests_that_require_special_setup.md @@ -153,7 +153,7 @@ Examples of tests which require a runner: Example: ```shell -docker run \ +docker run \ --detach \ --hostname interface_ip_address \ --publish 80:80 \ @@ -274,7 +274,7 @@ Geo requires an EE license. To visit the Geo sites in your browser, you need a r 1. To run end-to-end tests from your local GDK, run the [`EE::Scenario::Test::Geo` scenario](https://gitlab.com/gitlab-org/gitlab/-/blob/f7272b77e80215c39d1ffeaed27794c220dbe03f/qa/qa/ee/scenario/test/geo.rb) from the [`gitlab/qa/` directory](https://gitlab.com/gitlab-org/gitlab/-/blob/f7272b77e80215c39d1ffeaed27794c220dbe03f/qa). Include `--without-setup` to skip the Geo configuration steps. ```shell - QA_DEBUG=true GITLAB_QA_ACCESS_TOKEN=[add token here] GITLAB_QA_ADMIN_ACCESS_TOKEN=[add token here] bundle exec bin/qa QA::EE::Scenario::Test::Geo \ + QA_LOG_LEVEL=debug GITLAB_QA_ACCESS_TOKEN=[add token here] GITLAB_QA_ADMIN_ACCESS_TOKEN=[add token here] bundle exec bin/qa QA::EE::Scenario::Test::Geo \ --primary-address http://gitlab-primary.geo \ --secondary-address http://gitlab-secondary.geo \ --without-setup @@ -283,7 +283,7 @@ Geo requires an EE license. To visit the Geo sites in your browser, you need a r If the containers need to be configured first (for example, if you used the `--no-tests` option in the previous step), run the `QA::EE::Scenario::Test::Geo scenario` as shown below to first do the Geo configuration steps, and then run Geo end-to-end tests. Make sure that `EE_LICENSE` is (still) defined in your shell session. ```shell - QA_DEBUG=true bundle exec bin/qa QA::EE::Scenario::Test::Geo \ + QA_LOG_LEVEL=debug bundle exec bin/qa QA::EE::Scenario::Test::Geo \ --primary-address http://gitlab-primary.geo \ --primary-name gitlab-primary \ --secondary-address http://gitlab-secondary.geo \ @@ -354,7 +354,7 @@ To run the LDAP tests on your local with TLS enabled, follow these steps: 1. Run an LDAP test from [`gitlab/qa`](https://gitlab.com/gitlab-org/gitlab/-/tree/d5447ebb5f99d4c72780681ddf4dc25b0738acba/qa) directory: ```shell - GITLAB_LDAP_USERNAME="tanuki" GITLAB_LDAP_PASSWORD="password" QA_DEBUG=true WEBDRIVER_HEADLESS=false bin/qa Test::Instance::All https://gitlab.test qa/specs/features/browser_ui/1_manage/login/log_into_gitlab_via_ldap_spec.rb + GITLAB_LDAP_USERNAME="tanuki" GITLAB_LDAP_PASSWORD="password" QA_LOG_LEVEL=debug WEBDRIVER_HEADLESS=false bin/qa Test::Instance::All https://gitlab.test qa/specs/features/browser_ui/1_manage/login/log_into_gitlab_via_ldap_spec.rb ``` ### Running LDAP tests with TLS disabled @@ -382,7 +382,7 @@ To run the LDAP tests on your local with TLS disabled, follow these steps: 1. Run an LDAP test from [`gitlab/qa`](https://gitlab.com/gitlab-org/gitlab/-/tree/d5447ebb5f99d4c72780681ddf4dc25b0738acba/qa) directory: ```shell - GITLAB_LDAP_USERNAME="tanuki" GITLAB_LDAP_PASSWORD="password" QA_DEBUG=true WEBDRIVER_HEADLESS=false bin/qa Test::Instance::All http://localhost qa/specs/features/browser_ui/1_manage/login/log_into_gitlab_via_ldap_spec.rb + GITLAB_LDAP_USERNAME="tanuki" GITLAB_LDAP_PASSWORD="password" QA_LOG_LEVEL=debug WEBDRIVER_HEADLESS=false bin/qa Test::Instance::All http://localhost qa/specs/features/browser_ui/1_manage/login/log_into_gitlab_via_ldap_spec.rb ``` ## Guide to the mobile suite diff --git a/doc/development/testing_guide/end_to_end/troubleshooting.md b/doc/development/testing_guide/end_to_end/troubleshooting.md index 951fb056a4c..76d19dc0159 100644 --- a/doc/development/testing_guide/end_to_end/troubleshooting.md +++ b/doc/development/testing_guide/end_to_end/troubleshooting.md @@ -25,12 +25,12 @@ WEBDRIVER_HEADLESS=false bundle exec bin/qa Test::Instance::All http://localhost Sometimes a test might fail and the failure stack trace doesn't provide enough information to determine what went wrong. You can get more information by enabling -debug logs by setting `QA_DEBUG=true`, to see what the test framework is attempting. +debug logs by setting `QA_LOG_LEVEL=debug`, to see what the test framework is attempting. For example: ```shell cd gitlab/qa -QA_DEBUG=true bundle exec bin/qa Test::Instance::All http://localhost:3000 +QA_LOG_LEVEL=debug bundle exec bin/qa Test::Instance::All http://localhost:3000 ``` The test framework then outputs many logs showing the actions taken during diff --git a/doc/development/testing_guide/index.md b/doc/development/testing_guide/index.md index 2e00a00c454..fa9f1f1ac3e 100644 --- a/doc/development/testing_guide/index.md +++ b/doc/development/testing_guide/index.md @@ -70,4 +70,8 @@ Everything you should know about how to run end-to-end tests using Everything you should know about how to test migrations. +## [Contract tests](contract/index.md) + +Introduction to contract testing, how to run the tests, and how to write them. + [Return to Development documentation](../index.md) diff --git a/doc/development/testing_guide/review_apps.md b/doc/development/testing_guide/review_apps.md index ff4b77dec2c..f1083c23406 100644 --- a/doc/development/testing_guide/review_apps.md +++ b/doc/development/testing_guide/review_apps.md @@ -6,13 +6,13 @@ info: To determine the technical writer assigned to the Stage/Group associated w # Review Apps -Review Apps are deployed using the `start-review-app-pipeline` job. This job triggers a child pipeline containing a series of jobs to perform the various tasks needed to deploy a Review App. +Review Apps are deployed using the `start-review-app-pipeline` job which triggers a child pipeline containing a series of jobs to perform the various tasks needed to deploy a Review App.  For any of the following scenarios, the `start-review-app-pipeline` job would be automatically started: -- for merge requests with CI config changes +- for merge requests with CI configuration changes - for merge requests with frontend changes - for merge requests with changes to `{,ee/,jh/}{app/controllers}/**/*` - for merge requests with changes to `{,ee/,jh/}{app/models}/**/*` @@ -27,6 +27,8 @@ On every [pipeline](https://gitlab.com/gitlab-org/gitlab/pipelines/125315730) in `review` stage), the `review-qa-smoke` and `review-qa-reliable` jobs are automatically started. The `review-qa-smoke` runs the QA smoke suite and the `review-qa-reliable` executes E2E tests identified as [reliable](https://about.gitlab.com/handbook/engineering/quality/quality-engineering/reliable-tests). +`review-qa-*` jobs ensure that end-to-end tests for the changes in the merge request pass in a live environment. This shifts the identification of e2e failures from an environment on the path to production to the merge request, to prevent breaking features on GitLab.com or costly GitLab.com deployment blockers. `review-qa-*` failures should be investigated with counterpart SET involvement if needed to help determine the root cause of the error. + You can also manually start the `review-qa-all`: it runs the full QA suite. After the end-to-end test runs have finished, [Allure reports](https://github.com/allure-framework/allure2) are generated and published by @@ -34,6 +36,10 @@ the `allure-report-qa-smoke`, `allure-report-qa-reliable`, and `allure-report-qa Errors can be found in the `gitlab-review-apps` Sentry project and [filterable by Review App URL](https://sentry.gitlab.net/gitlab/gitlab-review-apps/?query=url%3A%22https%3A%2F%2Fgitlab-review-require-ve-u92nn2.gitlab-review.app%2F%22) or [commit SHA](https://sentry.gitlab.net/gitlab/gitlab-review-apps/releases/6095b501da7/all-events/). +### Bypass failed review app deployment to merge a broken `master` fix + +Maintainers can elect to use the [process for merging during broken `master`](https://about.gitlab.com/handbook/engineering/workflow/#instructions-for-the-maintainer) if a customer-critical merge request is blocked by pipelines failing due to review app deployment failures. + ## Performance Metrics On every [pipeline](https://gitlab.com/gitlab-org/gitlab/pipelines/125315730) in the `qa` stage, the @@ -94,8 +100,8 @@ the GitLab handbook information for the [shared 1Password account](https://about 1. Make sure you [have access to the cluster](#get-access-to-the-gcp-review-apps-cluster) and the `container.pods.exec` permission first. 1. [Filter Workloads by your Review App slug](https://console.cloud.google.com/kubernetes/workload?project=gitlab-review-apps). For example, `review-qa-raise-e-12chm0`. 1. Find and open the `toolbox` Deployment. For example, `review-qa-raise-e-12chm0-toolbox`. -1. Click on the Pod in the "Managed pods" section. For example, `review-qa-raise-e-12chm0-toolbox-d5455cc8-2lsvz`. -1. Click on the `KUBECTL` dropdown, then `Exec` -> `toolbox`. +1. Select the Pod in the "Managed pods" section. For example, `review-qa-raise-e-12chm0-toolbox-d5455cc8-2lsvz`. +1. Select the `KUBECTL` dropdown, then `Exec` -> `toolbox`. 1. Replace `-c toolbox -- ls` with `-it -- gitlab-rails console` from the default command or - Run `kubectl exec --namespace review-qa-raise-e-12chm0 review-qa-raise-e-12chm0-toolbox-d5455cc8-2lsvz -it -- gitlab-rails console` and @@ -107,8 +113,8 @@ the GitLab handbook information for the [shared 1Password account](https://about 1. Make sure you [have access to the cluster](#get-access-to-the-gcp-review-apps-cluster) and the `container.pods.getLogs` permission first. 1. [Filter Workloads by your Review App slug](https://console.cloud.google.com/kubernetes/workload?project=gitlab-review-apps). For example, `review-qa-raise-e-12chm0`. 1. Find and open the `migrations` Deployment. For example, `review-qa-raise-e-12chm0-migrations.1`. -1. Click on the Pod in the "Managed pods" section. For example, `review-qa-raise-e-12chm0-migrations.1-nqwtx`. -1. Click on the `Container logs` link. +1. Select the Pod in the "Managed pods" section. For example, `review-qa-raise-e-12chm0-migrations.1-nqwtx`. +1. Select `Container logs`. Alternatively, you could use the [Logs Explorer](https://console.cloud.google.com/logs/query;query=?project=gitlab-review-apps) which provides more utility to search logs. An example query for a pod name is as follows: @@ -199,7 +205,7 @@ subgraph "CNG-mirror pipeline" issue with a link to your merge request. Note that the deployment failure can reveal an actual problem introduced in your merge request (that is, this isn't necessarily a transient failure)! -- If the `review-qa-smoke` or `review-qa-reliable` job keeps failing, +- If the `review-qa-smoke` or `review-qa-reliable` job keeps failing (note that we already retry them once), please check the job's logs: you could discover an actual problem introduced in your merge request. You can also download the artifacts to see screenshots of the page at the time the failures occurred. If you don't find the cause of the @@ -237,7 +243,7 @@ due to this [known issue on the Kubernetes executor for GitLab Runner](https://g ### Helm The Helm version used is defined in the -[`registry.gitlab.com/gitlab-org/gitlab-build-images:gitlab-helm3-kubectl1.14` image](https://gitlab.com/gitlab-org/gitlab-build-images/-/blob/master/Dockerfile.gitlab-helm3-kubectl1.14#L7) +[`registry.gitlab.com/gitlab-org/gitlab-build-images:gitlab-helm3.5-kubectl1.17` image](https://gitlab.com/gitlab-org/gitlab-build-images/-/blob/master/Dockerfile.gitlab-helm3.5-kubectl1.17#L6) used by the `review-deploy` and `review-stop` jobs. ## Diagnosing unhealthy Review App releases diff --git a/doc/development/testing_guide/testing_levels.md b/doc/development/testing_guide/testing_levels.md index 9ca2d0db93c..02f32a031dc 100644 --- a/doc/development/testing_guide/testing_levels.md +++ b/doc/development/testing_guide/testing_levels.md @@ -55,7 +55,6 @@ records should use stubs/doubles as much as possible. | `lib/` | `spec/lib/` | RSpec | | | `lib/tasks/` | `spec/tasks/` | RSpec | | | `rubocop/` | `spec/rubocop/` | RSpec | | -| `spec/factories` | `spec/factories_spec.rb` | RSpec | | ### Frontend unit tests |