diff options
author | GitLab Bot <gitlab-bot@gitlab.com> | 2021-05-19 18:44:42 +0300 |
---|---|---|
committer | GitLab Bot <gitlab-bot@gitlab.com> | 2021-05-19 18:44:42 +0300 |
commit | 4555e1b21c365ed8303ffb7a3325d773c9b8bf31 (patch) | |
tree | 5423a1c7516cffe36384133ade12572cf709398d /doc/development/testing_guide | |
parent | e570267f2f6b326480d284e0164a6464ba4081bc (diff) |
Add latest changes from gitlab-org/gitlab@13-12-stable-eev13.12.0-rc42
Diffstat (limited to 'doc/development/testing_guide')
-rw-r--r-- | doc/development/testing_guide/best_practices.md | 33 | ||||
-rw-r--r-- | doc/development/testing_guide/end_to_end/beginners_guide.md | 5 | ||||
-rw-r--r-- | doc/development/testing_guide/end_to_end/capybara_to_chemlab_migration_guide.md | 148 | ||||
-rw-r--r-- | doc/development/testing_guide/end_to_end/img/gl-capybara_V13_12.png | bin | 0 -> 19201 bytes | |||
-rw-r--r-- | doc/development/testing_guide/end_to_end/img/gl-chemlab_V13_12.png | bin | 0 -> 17753 bytes | |||
-rw-r--r-- | doc/development/testing_guide/end_to_end/page_objects.md | 2 | ||||
-rw-r--r-- | doc/development/testing_guide/end_to_end/running_tests_that_require_special_setup.md | 8 | ||||
-rw-r--r-- | doc/development/testing_guide/flaky_tests.md | 27 | ||||
-rw-r--r-- | doc/development/testing_guide/frontend_testing.md | 12 |
9 files changed, 216 insertions, 19 deletions
diff --git a/doc/development/testing_guide/best_practices.md b/doc/development/testing_guide/best_practices.md index 828e9925d46..c3125f52cf2 100644 --- a/doc/development/testing_guide/best_practices.md +++ b/doc/development/testing_guide/best_practices.md @@ -168,7 +168,7 @@ can be used: ```ruby RSpec.describe API::Search, factory_default: :keep do - let_it_be(:namespace) { create_default(:namespace).freeze } + let_it_be(:namespace) { create_default(:namespace) } ``` Then every project we create uses this `namespace`, without us having to pass @@ -176,9 +176,9 @@ it as `namespace: namespace`. In order to make it work along with `let_it_be`, ` must be explicitly specified. That keeps the default factory for every example in a suite instead of recreating it for each example. -Objects created inside a `factory_default: :keep`, and using -`create_default` inside a `let_it_be` should be frozen to prevent accidental reliance -between test examples. +To prevent accidental reliance between test examples, objects created +with `create_default` are +[frozen](https://gitlab.com/gitlab-org/gitlab/-/blob/master/spec/support/factory_default.rb). Maybe we don't need to create 208 different projects - we can create one and reuse it. In addition, we can see that only about 1/3 of the @@ -186,7 +186,7 @@ projects we create are ones we ask for (76/208). There is benefit in setting a default value for projects as well: ```ruby - let_it_be(:project) { create_default(:project).freeze } + let_it_be(:project) { create_default(:project) } ``` In this case, the `total time` and `top-level time` numbers match more closely: @@ -451,7 +451,7 @@ expect(page).to have_current_path 'gitlab/gitlab-test/-/issues' expect(page).to have_title 'Not Found' -# acceptable when a more specific matcher above is not possible +# acceptable when a more specific matcher above is not possible expect(page).to have_css 'h2', text: 'Issue title' expect(page).to have_css 'p', text: 'Issue description', exact: true expect(page).to have_css '[data-testid="weight"]', text: 2 @@ -895,6 +895,27 @@ When you want to ensure that no event got called, you can use `expect_no_snowplo end ``` +#### Test Snowplow context against the schema + +The [Snowplow schema matcher](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/60480) +helps to reduce validation errors by testing Snowplow context against the JSON schema. +The schema matcher accepts the following parameters: + +- `schema path` +- `context` + +To add a schema matcher spec: + +1. Add a new schema to the [Iglu repository](https://gitlab.com/gitlab-org/iglu), + then copy the same schema to the `spec/fixtures/product_intelligence/` directory. +1. In the copied schema, remove the `"$schema"` key and value. We do not need it for specs + and the spec fails if we keep the key, as it tries to look for the schema in the URL. +1. Use the following snippet to call the schema matcher: + + ```ruby + match_snowplow_context_schema(schema_path: '<filename from step 1>', context: <Context Hash> ) + ``` + ### Table-based / Parameterized tests This style of testing is used to exercise one piece of code with a comprehensive diff --git a/doc/development/testing_guide/end_to_end/beginners_guide.md b/doc/development/testing_guide/end_to_end/beginners_guide.md index 29f6c93d65a..7cde2cad300 100644 --- a/doc/development/testing_guide/end_to_end/beginners_guide.md +++ b/doc/development/testing_guide/end_to_end/beginners_guide.md @@ -210,7 +210,7 @@ end Behind the scenes, `be_signed_in` is a [predicate matcher](https://relishapp.com/rspec/rspec-expectations/v/3-8/docs/built-in-matchers/predicate-matchers) -that [implements checking the user avatar](https://gitlab.com/gitlab-org/gitlab/-/blob/master/qa/qa/page/main/menu.rb#L74). +that [implements checking the user avatar](https://gitlab.com/gitlab-org/gitlab/-/blob/master/qa/qa/page/main/menu.rb#L92). ## De-duplicate your code @@ -339,11 +339,12 @@ Before running the spec, make sure that: - No additional [RSpec metadata tags](rspec_metadata_tests.md) have been applied. - Your working directory is `qa/` within your GDK GitLab installation. - Your GitLab instance-level settings are default. If you changed the default settings, some tests might have unexpected results. +- Because the GDK requires a password change on first login, you must include the GDK password for `root` user To run the spec, run the following command: ```ruby -bundle exec bin/qa Test::Instance::All http://localhost:3000 -- <test_file> +GITLAB_PASSWORD=<GDK root password> bundle exec bin/qa Test::Instance::All http://localhost:3000 -- <test_file> ``` Where `<test_file>` is: diff --git a/doc/development/testing_guide/end_to_end/capybara_to_chemlab_migration_guide.md b/doc/development/testing_guide/end_to_end/capybara_to_chemlab_migration_guide.md new file mode 100644 index 00000000000..9c7e0ef73a8 --- /dev/null +++ b/doc/development/testing_guide/end_to_end/capybara_to_chemlab_migration_guide.md @@ -0,0 +1,148 @@ +--- +stage: none +group: unassigned +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +--- + +# Migration Guide Capybara → Chemlab + +Given the view: + +*_form.html* + +```html +<form id="my-form"> + <label for="first-name">First name</label> + <input type="text" name="first-name" data-qa-selector="first_name" /> + + <label for="last-name">Last name</label> + <input type="text" name="last-name" data-qa-selector="last_name" /> + + <label for="company-name">Company name</label> + <input type="text" name="company-name" data-qa-selector="company_name" /> + + <label for="user-name">User name</label> + <input type="text" name="user-name" data-qa-selector="user_name" /> + + <label for="password">Password</label> + <input type="password" name="password" data-qa-selector="password" /> + + <input type="submit" value="Continue" data-qa-selector="continue"/> +</form> +``` + +| Capybara | Chemlab | +| ------ | ----- | +| ![before](img/gl-capybara_V13_12.png) | ![after](img/gl-chemlab_V13_12.png) | + +<!-- +```ruby +# frozen_string_literal: true + +module QA + module Page + class Form < Page::Base + view '_form.html' do + element :first_name + element :last_name + element :company_name + element :user_name + element :password + element :continue + end + end + end +end +``` +```ruby +# frozen_string_literal: true + +module QA + module Page + class Form < Chemlab::Page + text_field :first_name + text_field :last_name + text_field :company_name + text_field :user_name + text_field :password + + button :continue + end + end +end +``` +--> + +## Key Differences + +### Page Library Design vs Page Object Design + +Page Objects as implemented in the existing framework require you to define methods to perform actions on elements. (Usually one-liners) + +```ruby +def set_first_name(first_name) + fill_element(:first_name, first_name) +end + +def click_continue + click_element(:continue) +end + +it 'sets first name and clicks continue' do + Page::Form.perform do |form| + form.set_first_name('First Name') + form.click_continue + end +end +``` + +Page Libraries make this more efficient by providing methods based on the page's elements, making extra methods unnecessary. + +```ruby +it 'sets first name and clicks continue' do + Page::Form.perform do |form| + form.first_name = 'First Name' # sets the first_name + form.continue # clicks Continue + end +end +``` + +Consider if we needed to validate the text of the `First name` field using Capybara. We'd need to add a one-liner to fetch the text: + +```ruby +def get_first_name + find_element(:first_name).text +end + +Page::Form.perform do |form| + form.set_first_name('First Name') + expect(form.get_first_name).to eq('First Name') + form.click_continue +end +``` + +Instead, because the page library automatically creates methods from page elements, we can fetch the text by calling `first_name` without writing code to define the method ourselves: + +```ruby +Page::Form.perform do |form| + form.first_name = 'First Name' + expect(form.first_name).to eq('First Name') + form.continue +end +``` + +### Element Naming Convention + +Since the element type is preserved within the Page Library, there is no need to specify a `_field` or `_button` suffix to the data-qa-selector. + +```html +<!-- Before --> +<input type="text" name="first-name" data-qa-selector="first_name_field" /> +<input type="submit" name="continue" value="Continue" data-qa-selector="continue_button" /> + +<!-- After --> +<input type="text" name="first-name" data-qa-selector="first_name" /> +<input type="submit" name="continue" value="Continue" data-qa-selector="continue" /> +``` + +This makes it much easier for Developers to write tests and contributes to testability since we can write the Page Library while we look at the UI. diff --git a/doc/development/testing_guide/end_to_end/img/gl-capybara_V13_12.png b/doc/development/testing_guide/end_to_end/img/gl-capybara_V13_12.png Binary files differnew file mode 100644 index 00000000000..9ceccd39025 --- /dev/null +++ b/doc/development/testing_guide/end_to_end/img/gl-capybara_V13_12.png diff --git a/doc/development/testing_guide/end_to_end/img/gl-chemlab_V13_12.png b/doc/development/testing_guide/end_to_end/img/gl-chemlab_V13_12.png Binary files differnew file mode 100644 index 00000000000..489a043f52e --- /dev/null +++ b/doc/development/testing_guide/end_to_end/img/gl-chemlab_V13_12.png diff --git a/doc/development/testing_guide/end_to_end/page_objects.md b/doc/development/testing_guide/end_to_end/page_objects.md index b124ac430f6..9ffa7ea4f77 100644 --- a/doc/development/testing_guide/end_to_end/page_objects.md +++ b/doc/development/testing_guide/end_to_end/page_objects.md @@ -260,7 +260,7 @@ These modules must: These steps ensure the sanity selectors check detect problems properly. For example, `qa/qa/ee/page/merge_request/show.rb` adds EE-specific methods to `qa/qa/page/merge_request/show.rb` (with -`QA::Page::MergeRequest::Show.prepend_if_ee('QA::EE::Page::MergeRequest::Show')`) and following is how it's implemented +`QA::Page::MergeRequest::Show.prepend_mod_with('Page::MergeRequest::Show', namespace: QA)`) and following is how it's implemented (only showing the relevant part and referring to the 4 steps described above with inline comments): ```ruby diff --git a/doc/development/testing_guide/end_to_end/running_tests_that_require_special_setup.md b/doc/development/testing_guide/end_to_end/running_tests_that_require_special_setup.md index ea48a3aa8b8..549ab95a5d1 100644 --- a/doc/development/testing_guide/end_to_end/running_tests_that_require_special_setup.md +++ b/doc/development/testing_guide/end_to_end/running_tests_that_require_special_setup.md @@ -147,10 +147,10 @@ docker rm gitlab-gitaly-cluster praefect postgres gitaly3 gitaly2 gitaly1 To run the Monitor tests locally, against the GDK, please follow the preparation steps below: -1. Complete the [Prerequisites](https://gitlab.com/gitlab-org/gitlab-development-kit/-/blob/master/doc/howto/auto_devops/index.md#prerequisites-for-gitlab-team-members-only), at least through step 5. Note that the monitor tests do not require permissions to work with GKE because they use [k3s as a Kubernetes cluster provider](https://github.com/rancher/k3s). +1. Complete the [Prerequisites](https://gitlab.com/gitlab-org/gitlab-development-kit/-/blob/main/doc/howto/auto_devops/index.md#prerequisites-for-gitlab-team-members-only), at least through step 5. Note that the monitor tests do not require permissions to work with GKE because they use [k3s as a Kubernetes cluster provider](https://github.com/rancher/k3s). 1. The test setup deploys the app in a Kubernetes cluster, using the Auto DevOps deployment strategy. -To enable Auto DevOps in GDK, follow the [associated setup](https://gitlab.com/gitlab-org/gitlab-development-kit/-/blob/master/doc/howto/auto_devops/index.md#setup) instructions. If you have problems, review the [troubleshooting guide](https://gitlab.com/gitlab-org/gitlab-development-kit/-/blob/master/doc/howto/auto_devops/tips_and_troubleshooting.md) or reach out to the `#gdk` channel in the internal GitLab Slack. -1. Do [secure your GitLab instance](https://gitlab.com/gitlab-org/gitlab-development-kit/-/blob/master/doc/howto/auto_devops/index.md#secure-your-gitlab-instance) since it is now publicly accessible on `https://[YOUR-PORT].qa-tunnel.gitlab.info`. +To enable Auto DevOps in GDK, follow the [associated setup](https://gitlab.com/gitlab-org/gitlab-development-kit/-/blob/main/doc/howto/auto_devops/index.md#setup) instructions. If you have problems, review the [troubleshooting guide](https://gitlab.com/gitlab-org/gitlab-development-kit/-/blob/main/doc/howto/auto_devops/tips_and_troubleshooting.md) or reach out to the `#gdk` channel in the internal GitLab Slack. +1. Do [secure your GitLab instance](https://gitlab.com/gitlab-org/gitlab-development-kit/-/blob/main/doc/howto/auto_devops/index.md#secure-your-gitlab-instance) since it is now publicly accessible on `https://[YOUR-PORT].qa-tunnel.gitlab.info`. 1. Install the Kubernetes command line tool known as `kubectl`. Use the [official installation instructions](https://kubernetes.io/docs/tasks/tools/). You might see NGINX issues when you run `gdk start` or `gdk restart`. In that case, run `sft login` to revalidate your credentials and regain access the QA Tunnel. @@ -272,7 +272,7 @@ You can free some memory with either of the following commands: `docker prune sy ## Geo tests -Geo end-to-end tests can run locally against a [Geo GDK setup](https://gitlab.com/gitlab-org/gitlab-development-kit/-/blob/master/doc/howto/geo.md) or on Geo spun up in Docker containers. +Geo end-to-end tests can run locally against a [Geo GDK setup](https://gitlab.com/gitlab-org/gitlab-development-kit/-/blob/main/doc/howto/geo.md) or on Geo spun up in Docker containers. ### Using Geo GDK diff --git a/doc/development/testing_guide/flaky_tests.md b/doc/development/testing_guide/flaky_tests.md index a9af8f03d63..6b1c7a7eb58 100644 --- a/doc/development/testing_guide/flaky_tests.md +++ b/doc/development/testing_guide/flaky_tests.md @@ -76,6 +76,33 @@ For instance `RETRIES=1 bin/rspec ...` would retry the failing examples once. - [Replace FFaker factory data with sequences](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/29643): <https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/10184> - [Transient failure in spec/finders/issues_finder_spec.rb](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/30211#note_26707685): <https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/10404> +### Order-dependent flaky tests + +These flaky tests can fail depending on the order they run with other tests. For example: + +- <https://gitlab.com/gitlab-org/gitlab/-/issues/327668> + +To identify the tests that lead to such failure, we can use `rspec --bisect`, +which would give us the minimal test combination to reproduce the failure: + +```shell +rspec --bisect ee/spec/services/ee/merge_requests/update_service_spec.rb ee/spec/services/ee/notes/quick_actions_service_spec.rb ee/spec/services/epic_links/create_service_spec.rb ee/spec/services/ee/issuable/bulk_update_service_spec.rb +Bisect started using options: "ee/spec/services/ee/merge_requests/update_service_spec.rb ee/spec/services/ee/notes/quick_actions_service_spec.rb ee/spec/services/epic_links/create_service_spec.rb ee/spec/services/ee/issuable/bulk_update_service_spec.rb" +Running suite to find failures... (2 minutes 18.4 seconds) +Starting bisect with 3 failing examples and 144 non-failing examples. +Checking that failure(s) are order-dependent... failure appears to be order-dependent + +Round 1: bisecting over non-failing examples 1-144 . ignoring examples 1-72 (1 minute 11.33 seconds) +... +Round 7: bisecting over non-failing examples 132-133 . ignoring example 132 (43.78 seconds) +Bisect complete! Reduced necessary non-failing examples from 144 to 1 in 8 minutes 31 seconds. + +The minimal reproduction command is: + rspec ./ee/spec/services/ee/issuable/bulk_update_service_spec.rb[1:2:1:1:1:1,1:2:1:2:1:1,1:2:1:3:1] ./ee/spec/services/epic_links/create_service_spec.rb[1:1:2:2:6:4] +``` + +We can reproduce the test failure with the reproduction command above. If we change the order of the tests, the test would pass. + ### Time-sensitive flaky tests - <https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/10046> diff --git a/doc/development/testing_guide/frontend_testing.md b/doc/development/testing_guide/frontend_testing.md index 7289e66a045..911fbd43989 100644 --- a/doc/development/testing_guide/frontend_testing.md +++ b/doc/development/testing_guide/frontend_testing.md @@ -890,8 +890,8 @@ helper method. For example: describe GraphQL::Query, type: :request do include GraphqlHelpers - all_releases_query_path = 'releases/queries/all_releases.query.graphql' - fragment_paths = ['releases/queries/release.fragment.graphql'] + all_releases_query_path = 'releases/graphql/queries/all_releases.query.graphql' + fragment_paths = ['releases/graphql/fragments/release.fragment.graphql'] before(:all) do clean_frontend_fixtures('graphql/releases/') @@ -908,7 +908,7 @@ end ``` This will create a new fixture located at -`tmp/tests/frontend/fixtures-ee/graphql/releases/queries/all_releases.query.graphql.json`. +`tmp/tests/frontend/fixtures-ee/graphql/releases/graphql/queries/all_releases.query.graphql.json`. You will need to provide the paths to all fragments used by the query. `get_graphql_query_as_string` reads all of the provided file paths and returns @@ -1151,8 +1151,8 @@ Both functions run `callback` on the next tick after the requests finish (using ### `shallowMountExtended` and `mountExtended` -The `shallowMountExtended` and `mountExtended` utilities provide you with the ability to perform -any of the available [DOM Testing Library queries](https://testing-library.com/docs/queries/about) +The `shallowMountExtended` and `mountExtended` utilities provide you with the ability to perform +any of the available [DOM Testing Library queries](https://testing-library.com/docs/queries/about) by prefixing them with `find` or `findAll`. ```javascript @@ -1302,7 +1302,7 @@ A good guideline to follow: the more complex the component you may want to steer - To capture large data structures just to have something - To just have some kind of test written -- To capture highly volatile ui elements without stubbing them (Think of GitLab UI version updates) +- To capture highly volatile UI elements without stubbing them (Think of GitLab UI version updates) --- |