diff options
Diffstat (limited to 'doc/development/testing_guide')
| -rw-r--r-- | doc/development/testing_guide/best_practices.md | 86 | ||||
| -rw-r--r-- | doc/development/testing_guide/end_to_end/beginners_guide.md | 4 | ||||
| -rw-r--r-- | doc/development/testing_guide/end_to_end/best_practices.md | 106 | ||||
| -rw-r--r-- | doc/development/testing_guide/end_to_end/index.md | 39 | ||||
| -rw-r--r-- | doc/development/testing_guide/end_to_end/rspec_metadata_tests.md | 1 | ||||
| -rw-r--r-- | doc/development/testing_guide/frontend_testing.md | 193 | ||||
| -rw-r--r-- | doc/development/testing_guide/index.md | 4 | ||||
| -rw-r--r-- | doc/development/testing_guide/review_apps.md | 2 | ||||
| -rw-r--r-- | doc/development/testing_guide/testing_levels.md | 13 |
9 files changed, 219 insertions, 229 deletions
diff --git a/doc/development/testing_guide/best_practices.md b/doc/development/testing_guide/best_practices.md index ba7312b760f..79664490368 100644 --- a/doc/development/testing_guide/best_practices.md +++ b/doc/development/testing_guide/best_practices.md @@ -54,7 +54,7 @@ When using spring and guard together, use `SPRING=1 bundle exec guard` instead t > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/47767) in GitLab 13.7. -We've enabled [deprecation warnings](https://ruby-doc.org/core-2.7.2/Warning.html) +We've enabled [deprecation warnings](https://ruby-doc.org/core-2.7.4/Warning.html) by default when running specs. Making these warnings more visible to developers helps upgrading to newer Ruby versions. @@ -367,26 +367,34 @@ If needed, you can scope interactions within a specific area of the page by usin As you will likely be scoping to an element such as a `div`, which typically does not have a label, you may use a `data-testid` selector in this case. +##### Externalized contents + +Test expectations against externalized contents should call the same +externalizing method to match the translation. For example, you should use the `_` +method in Ruby and `__` method in JavaScript. + +See [Internationalization for GitLab - Test files](../i18n/externalization.md#test-files) for details. + ##### Actions Where possible, use more specific [actions](https://rubydoc.info/github/teamcapybara/capybara/master/Capybara/Node/Actions), such as the ones below. ```ruby # good -click_button 'Submit review' +click_button _('Submit review') -click_link 'UI testing docs' +click_link _('UI testing docs') -fill_in 'Search projects', with: 'gitlab' # fill in text input with text +fill_in _('Search projects'), with: 'gitlab' # fill in text input with text -select 'Last updated', from: 'Sort by' # select an option from a select input +select _('Last updated'), from: 'Sort by' # select an option from a select input -check 'Checkbox label' -uncheck 'Checkbox label' +check _('Checkbox label') +uncheck _('Checkbox label') -choose 'Radio input label' +choose _('Radio input label') -attach_file('Attach a file', '/path/to/file.png') +attach_file(_('Attach a file'), '/path/to/file.png') # bad - interactive elements must have accessible names, so # we should be able to use one of the specific actions above @@ -403,17 +411,17 @@ Where possible, use more specific [finders](https://rubydoc.info/github/teamcapy ```ruby # good -find_button 'Submit review' -find_button 'Submit review', disabled: true +find_button _('Submit review') +find_button _('Submit review'), disabled: true -find_link 'UI testing docs' -find_link 'UI testing docs', href: docs_url +find_link _('UI testing docs') +find_link _('UI testing docs'), href: docs_url -find_field 'Search projects' -find_field 'Search projects', with: 'gitlab' # find the input field with text -find_field 'Search projects', disabled: true -find_field 'Checkbox label', checked: true -find_field 'Checkbox label', unchecked: true +find_field _('Search projects') +find_field _('Search projects'), with: 'gitlab' # find the input field with text +find_field _('Search projects'), disabled: true +find_field _('Checkbox label'), checked: true +find_field _('Checkbox label'), unchecked: true # acceptable when finding a element that is not a button, link, or field find('[data-testid="element"]') @@ -425,31 +433,31 @@ Where possible, use more specific [matchers](https://rubydoc.info/github/teamcap ```ruby # good -expect(page).to have_button 'Submit review' -expect(page).to have_button 'Submit review', disabled: true -expect(page).to have_button 'Notifications', class: 'is-checked' # assert the "Notifications" GlToggle is checked +expect(page).to have_button _('Submit review') +expect(page).to have_button _('Submit review'), disabled: true +expect(page).to have_button _('Notifications'), class: 'is-checked' # assert the "Notifications" GlToggle is checked -expect(page).to have_link 'UI testing docs' -expect(page).to have_link 'UI testing docs', href: docs_url # assert the link has an href +expect(page).to have_link _('UI testing docs') +expect(page).to have_link _('UI testing docs'), href: docs_url # assert the link has an href -expect(page).to have_field 'Search projects' -expect(page).to have_field 'Search projects', disabled: true -expect(page).to have_field 'Search projects', with: 'gitlab' # assert the input field has text +expect(page).to have_field _('Search projects') +expect(page).to have_field _('Search projects'), disabled: true +expect(page).to have_field _('Search projects'), with: 'gitlab' # assert the input field has text -expect(page).to have_checked_field 'Checkbox label' -expect(page).to have_unchecked_field 'Radio input label' +expect(page).to have_checked_field _('Checkbox label') +expect(page).to have_unchecked_field _('Radio input label') -expect(page).to have_select 'Sort by' -expect(page).to have_select 'Sort by', selected: 'Last updated' # assert the option is selected -expect(page).to have_select 'Sort by', options: ['Last updated', 'Created date', 'Due date'] # assert an exact list of options -expect(page).to have_select 'Sort by', with_options: ['Created date', 'Due date'] # assert a partial list of options +expect(page).to have_select _('Sort by') +expect(page).to have_select _('Sort by'), selected: 'Last updated' # assert the option is selected +expect(page).to have_select _('Sort by'), options: ['Last updated', 'Created date', 'Due date'] # assert an exact list of options +expect(page).to have_select _('Sort by'), with_options: ['Created date', 'Due date'] # assert a partial list of options -expect(page).to have_text 'Some paragraph text.' -expect(page).to have_text 'Some paragraph text.', exact: true # assert exact match +expect(page).to have_text _('Some paragraph text.') +expect(page).to have_text _('Some paragraph text.'), exact: true # assert exact match expect(page).to have_current_path 'gitlab/gitlab-test/-/issues' -expect(page).to have_title 'Not Found' +expect(page).to have_title _('Not Found') # acceptable when a more specific matcher above is not possible expect(page).to have_css 'h2', text: 'Issue title' @@ -653,6 +661,12 @@ let_it_be_with_refind(:project) { create(:project) } let_it_be(:project, refind: true) { create(:project) } ``` +Note that `let_it_be` cannot be used with factories that has stubs, such as `allow`. +The reason is that `let_it_be` happens in a `before(:all)` block, and RSpec does not +allow stubs in `before(:all)`. +See this [issue](https://gitlab.com/gitlab-org/gitlab/-/issues/340487) for more details. +To resolve, use `let`, or change the factory to not use stubs. + ### Time-sensitive tests [`ActiveSupport::Testing::TimeHelpers`](https://api.rubyonrails.org/v6.0.3.1/classes/ActiveSupport/Testing/TimeHelpers.html) @@ -1197,6 +1211,8 @@ GitLab uses [factory_bot](https://github.com/thoughtbot/factory_bot) as a test f - Factories don't have to be limited to `ActiveRecord` objects. [See example](https://gitlab.com/gitlab-org/gitlab-foss/commit/0b8cefd3b2385a21cfed779bd659978c0402766d). - Factories and their traits should produce valid objects that are [verified by specs](https://gitlab.com/gitlab-org/gitlab/-/blob/master/spec/factories_spec.rb). +- Avoid the use of [`skip_callback`](https://api.rubyonrails.org/classes/ActiveSupport/Callbacks/ClassMethods.html#method-i-skip_callback) in factories. + See [issue #247865](https://gitlab.com/gitlab-org/gitlab/-/issues/247865) for details. ### Fixtures diff --git a/doc/development/testing_guide/end_to_end/beginners_guide.md b/doc/development/testing_guide/end_to_end/beginners_guide.md index e0f0e9e7089..7370cc5771b 100644 --- a/doc/development/testing_guide/end_to_end/beginners_guide.md +++ b/doc/development/testing_guide/end_to_end/beginners_guide.md @@ -349,8 +349,8 @@ GITLAB_PASSWORD=<GDK root password> bundle exec bin/qa Test::Instance::All http: Where `<test_file>` is: -- `qa/specs/features/browser_ui/1_manage/login/login_spec.rb` when running the Login example. -- `qa/specs/features/browser_ui/2_plan/issues/issue_spec.rb` when running the Issue example. +- `qa/specs/features/browser_ui/1_manage/login/log_in_spec.rb` when running the Login example. +- `qa/specs/features/browser_ui/2_plan/issues/create_issue_spec.rb` when running the Issue example. ## End-to-end test merge request template diff --git a/doc/development/testing_guide/end_to_end/best_practices.md b/doc/development/testing_guide/end_to_end/best_practices.md index 74c02d19d0a..a3caa8bf2b3 100644 --- a/doc/development/testing_guide/end_to_end/best_practices.md +++ b/doc/development/testing_guide/end_to_end/best_practices.md @@ -8,27 +8,33 @@ info: To determine the technical writer assigned to the Stage/Group associated w This is a tailored extension of the Best Practices [found in the testing guide](../best_practices.md). -## Link a test to its test-case issue +## Class and module naming -Every test should have a corresponding issue in the [Quality Test Cases project](https://gitlab.com/gitlab-org/quality/testcases/). -It's recommended that you reuse the issue created to plan the test. If one does not already exist you -can create the issue yourself. Alternatively, you can run the test in a pipeline that has reporting -enabled and the test-case issue reporter will automatically create a new issue. +The QA framework uses [Zeitwerk](https://github.com/fxn/zeitwerk) for class and module autoloading. The default Zeitwerk [inflector](https://github.com/fxn/zeitwerk#zeitwerkinflector) simply converts snake_cased file names to PascalCased module or class names. It is advised to stick to this pattern to avoid manual maintenance of inflections. -Whether you create a new test-case issue or one is created automatically, you will need to manually add -a `testcase` RSpec metadata tag. In most cases, a single test will be associated with a single test-case -issue ([see below for exceptions](#exceptions)). +In case custom inflection logic is needed, custom inflectors are added in the [qa.rb](https://gitlab.com/gitlab-org/gitlab/-/blob/master/qa/qa.rb) file in the `loader.inflector.inflect` method invocation. + +## Link a test to its test case + +Every test should have a corresponding test case as well as a results issue in the [Quality Test Cases project](https://gitlab.com/gitlab-org/quality/testcases/). +It's recommended that you reuse the issue created to plan the test as the results issue. If a test case or results issue does not already exist you +can create them yourself. Alternatively, you can run the test in a pipeline that has reporting +enabled and the test-case reporter will automatically create a new test case and/or results issue and link the results issue to it's corresponding test case. + +Whether you create a new test case or one is created automatically, you will need to manually add +a `testcase` RSpec metadata tag. In most cases, a single test will be associated with a single test case + ([see below for exceptions](#exceptions)). For example: ```ruby RSpec.describe 'Stage' do describe 'General description of the feature under test' do - it 'test name', testcase: 'https://gitlab.com/gitlab-org/quality/testcases/-/issues/:issue_id' do + it 'test name', testcase: 'https://gitlab.com/gitlab-org/quality/testcases/-/quality/test_cases/:test_case_id' do ... end - it 'another test', testcase: 'https://gitlab.com/gitlab-org/quality/testcases/-/issues/:another_issue_id' do + it 'another test', testcase: 'https://gitlab.com/gitlab-org/quality/testcases/-/quality/test_cases/:another_test_case_id' do ... end end @@ -38,10 +44,10 @@ end ### Exceptions Most tests are defined by a single line of a `spec` file, which is why those tests can be linked to a -single test-case issue via the `testcase` tag. +single test case via the `testcase` tag. -However, some tests don't have a one-to-one relationship between a line of a `spec` file and a test-case -issue. This is because some tests are defined in a way that means a single line is associated with +However, some tests don't have a one-to-one relationship between a line of a `spec` file and a test case. +This is because some tests are defined in a way that means a single line is associated with multiple tests, including: - Parallelized tests. @@ -49,13 +55,13 @@ multiple tests, including: - Tests in shared examples that include more than one example. In those and similar cases we can't assign a single `testcase` tag and so we rely on the test-case -reporter to programmatically determine the correct test-case issue based on the name and description of -the test. In such cases, the test-case reporter will automatically create a test-case issue the first time -the test runs, if no issue exists already. +reporter to programmatically determine the correct test case based on the name and description of +the test. In such cases, the test-case reporter will automatically create a test case and/or results issue +the first time the test runs, if none exist already. -In such a case, if you create the issue yourself or want to reuse an existing issue, +In such a case, if you create the test case or results issue yourself or want to reuse an existing issue, you must use this [end-to-end test issue template](https://gitlab.com/gitlab-org/quality/testcases/-/blob/master/.gitlab/issue_templates/End-to-end%20Test.md) -to format the issue description. +to format the issue description. (Note you must copy/paste this for test cases as templates aren't currently available.) To illustrate, there are two tests in the shared examples in [`qa/specs/features/ee/browser_ui/3_create/repository/restrict_push_protected_branch_spec.rb`](https://gitlab.com/gitlab-org/gitlab/-/blob/47b17db82c38ab704a23b5ba5d296ea0c6a732c8/qa/qa/specs/features/ee/browser_ui/3_create/repository/restrict_push_protected_branch_spec.rb): @@ -84,9 +90,9 @@ RSpec.describe 'Create' do end ``` -There would be two associated test-case issues, one for each shared example, with the following content: +There would be two associated test cases, one for each shared example, with the following content: -[Test 1](https://gitlab.com/gitlab-org/quality/testcases/-/issues/600): +[Test 1 Test Case](https://gitlab.com/gitlab-org/quality/testcases/-/quality/test_cases/1491): ````markdown ```markdown @@ -105,10 +111,66 @@ pushes and merges ./qa/specs/features/ee/browser_ui/3_create/repository/restrict_push_protected_branch_spec.rb +### DO NOT EDIT BELOW THIS LINE + +Active and historical test results: + +https://gitlab.com/gitlab-org/quality/testcases/-/issues/600 + +``` +```` + +[Test 1 Results Issue](https://gitlab.com/gitlab-org/quality/testcases/-/issues/600): + +````markdown +```markdown +Title: browser_ui/3_create/repository/restrict_push_protected_branch_spec.rb | Create Restricted +protected branch push and merge when only one user is allowed to merge and push to a protected +branch behaves like only user with access pushes and merges selecte... + +Description: +### Full description + +Create Restricted protected branch push and merge when only one user is allowed to merge and push +to a protected branch behaves like only user with access pushes and merges selected developer user +pushes and merges + +### File path + +./qa/specs/features/ee/browser_ui/3_create/repository/restrict_push_protected_branch_spec.rb + +``` +```` + +[Test 2 Test Case](https://gitlab.com/gitlab-org/quality/testcases/-/quality/test_cases/602): + +````markdown +```markdown +Title: browser_ui/3_create/repository/restrict_push_protected_branch_spec.rb | Create Restricted +protected branch push and merge when only one user is allowed to merge and push to a protected +branch behaves like only user with access pushes and merges unselec... + +Description: +### Full description + +Create Restricted protected branch push and merge when only one user is allowed to merge and push +to a protected branch behaves like only user with access pushes and merges unselected maintainer +user fails to push + +### File path + +./qa/specs/features/ee/browser_ui/3_create/repository/restrict_push_protected_branch_spec.rb + +### DO NOT EDIT BELOW THIS LINE + +Active and historical test results: + +https://gitlab.com/gitlab-org/quality/testcases/-/issues/602 + ``` ```` -[Test 2](https://gitlab.com/gitlab-org/quality/testcases/-/issues/602): +[Test 2 Results Issue](https://gitlab.com/gitlab-org/quality/testcases/-/issues/602): ````markdown ```markdown @@ -342,7 +404,7 @@ end When something requires waiting to be matched, use `eventually_` matchers with clear wait duration definition. -`Eventually` matchers use the following naming pattern: `eventually_${rspec_matcher_name}`. They are defined in [eventually_matcher.rb](https://gitlab.com/gitlab-org/gitlab/-/blob/master/qa/spec/support/matchers/eventually_matcher.rb). +`Eventually` matchers use the following naming pattern: `eventually_${rspec_matcher_name}`. They are defined in [eventually_matcher.rb](https://gitlab.com/gitlab-org/gitlab/-/blob/master/qa/qa/support/matchers/eventually_matcher.rb). ```ruby expect { async_value }.to eventually_eq(value).within(max_duration: 120, max_attempts: 60, reload_page: page) diff --git a/doc/development/testing_guide/end_to_end/index.md b/doc/development/testing_guide/end_to_end/index.md index f4b01c64385..36c0f5adf00 100644 --- a/doc/development/testing_guide/end_to_end/index.md +++ b/doc/development/testing_guide/end_to_end/index.md @@ -170,6 +170,45 @@ Helm chart](https://gitlab.com/gitlab-org/charts/gitlab/), itself deployed with See [Review Apps](../review_apps.md) for more details about Review Apps. +## Test reports + +### Allure report + +For additional test results visibility, tests that run on pipelines generate +and host [Allure](https://github.com/allure-framework/allure2) test reports. + +The `QA` framework is using the [Allure RSpec](https://github.com/allure-framework/allure-ruby/blob/master/allure-rspec/README.md) +gem to generate source files for the `Allure` test report. An additional job +in the pipeline: + +- Fetches these source files from all test jobs. +- Generates and uploads the report to the `GCS` bucket `gitlab-qa-allure-report` under the project `gitlab-qa-resources`. + +A common CI template for report uploading is stored in +[`allure-report.yml`](https://gitlab.com/gitlab-org/quality/pipeline-common/-/blob/master/ci/allure-report.yml). + +#### Merge requests + +When these tests are executed in the scope of merge requests, the `Allure` report is +uploaded to the `GCS` bucket and comment is added linking to their respective reports. + +#### Scheduled pipelines + +Scheduled pipelines for these tests contain a `generate-allure-report` job under the `Report` stage. They also output +a link to the current test report. + +#### Static report links + +Each type of scheduled pipeline generates a static link for the latest test report according to its stage: + +- [`master`](https://storage.googleapis.com/gitlab-qa-allure-reports/package-and-qa/master/index.html) +- [`staging-full`](https://storage.googleapis.com/gitlab-qa-allure-reports/staging-full/master/index.html) +- [`staging-sanity`](https://storage.googleapis.com/gitlab-qa-allure-reports/staging-sanity/master/index.html) +- [`staging-sanity-no-admin`](https://storage.googleapis.com/gitlab-qa-allure-reports/staging-sanity-no-admin/master/index.html) +- [`canary-sanity`](https://storage.googleapis.com/gitlab-qa-allure-reports/canary-sanity/master/index.html) +- [`production`](https://storage.googleapis.com/gitlab-qa-allure-reports/production/master/index.html) +- [`production-sanity`](https://storage.googleapis.com/gitlab-qa-allure-reports/production-sanity/master/index.html) + ## How do I run the tests? If you are not [testing code in a merge request](#testing-code-in-merge-requests), diff --git a/doc/development/testing_guide/end_to_end/rspec_metadata_tests.md b/doc/development/testing_guide/end_to_end/rspec_metadata_tests.md index 3a016c0e95c..b6e92367f89 100644 --- a/doc/development/testing_guide/end_to_end/rspec_metadata_tests.md +++ b/doc/development/testing_guide/end_to_end/rspec_metadata_tests.md @@ -44,3 +44,4 @@ This is a partial list of the [RSpec metadata](https://relishapp.com/rspec/rspec | `:smtp` | The test requires a GitLab instance to be configured to use an SMTP server. Tests SMTP notification email delivery from GitLab by using MailHog. | | `:testcase` | The link to the test case issue in the [Quality Test Cases project](https://gitlab.com/gitlab-org/quality/testcases/). | | `:transient` | The test tests transient bugs. It is excluded by default. | +| `:issue`, `:issue_${num}` | Optional links to issues which might be related to the spec. Helps keeping track of related issues and can also be used by tools that create test reports. Currently added automatically to `Allure` test report. Multiple tags can be used by adding optional number postfix like `issue_1`, `issue_2` etc. | diff --git a/doc/development/testing_guide/frontend_testing.md b/doc/development/testing_guide/frontend_testing.md index 3af806d8f57..76687db3a3f 100644 --- a/doc/development/testing_guide/frontend_testing.md +++ b/doc/development/testing_guide/frontend_testing.md @@ -7,7 +7,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w # Frontend testing standards and style guidelines There are two types of test suites encountered while developing frontend code -at GitLab. We use Karma with Jasmine and Jest for JavaScript unit and integration testing, +at GitLab. We use Jest for JavaScript unit and integration testing, and RSpec feature tests with Capybara for e2e (end-to-end) integration testing. Unit and feature tests need to be written for all new features. @@ -28,46 +28,9 @@ If you are looking for a guide on Vue component testing, you can jump right away We use Jest to write frontend unit and integration tests. Jest tests can be found in `/spec/frontend` and `/ee/spec/frontend` in EE. -## Karma test suite - -While GitLab has switched over to [Jest](https://jestjs.io), Karma tests still exist in our -application because some of our specs require a browser and can't be easily migrated to Jest. -Those specs intend to eventually drop Karma in favor of either Jest or RSpec. You can track this migration -in the [related epic](https://gitlab.com/groups/gitlab-org/-/epics/4900). - -[Karma](http://karma-runner.github.io/) is a test runner which uses -[Jasmine](https://jasmine.github.io/) as its test framework. Jest also uses Jasmine as foundation, -that's why it's looking quite similar. - -Karma tests live in `spec/javascripts/` and `/ee/spec/javascripts` in EE. - -`app/assets/javascripts/behaviors/autosize.js` -might have a corresponding `spec/javascripts/behaviors/autosize_spec.js` file. - -Keep in mind that in a CI environment, these tests are run in a headless -browser and you don't have access to certain APIs, such as -[`Notification`](https://developer.mozilla.org/en-US/docs/Web/API/notification), -which have to be stubbed. - -### Differences to Karma - -- Jest runs in a Node.js environment, not in a browser. [An issue exists](https://gitlab.com/gitlab-org/gitlab/-/issues/26982) for running Jest tests in a browser. -- Because Jest runs in a Node.js environment, it uses [jsdom](https://github.com/jsdom/jsdom) by default. See also its [limitations](#limitations-of-jsdom) below. -- Jest does not have access to Webpack loaders or aliases. - The aliases used by Jest are defined in its [own configuration](https://gitlab.com/gitlab-org/gitlab/-/blob/master/jest.config.js). -- All calls to `setTimeout` and `setInterval` are mocked away. See also [Jest Timer Mocks](https://jestjs.io/docs/timer-mocks). -- `rewire` is not required because Jest supports mocking modules. See also [Manual Mocks](https://jestjs.io/docs/manual-mocks). -- No [context object](https://jasmine.github.io/tutorials/your_first_suite#section-The_%3Ccode%3Ethis%3C/code%3E_keyword) is passed to tests in Jest. - This means sharing `this.something` between `beforeEach()` and `it()` for example does not work. - Instead you should declare shared variables in the context that they are needed (via `const` / `let`). -- The following cause tests to fail in Jest: - - Unmocked requests. - - Unhandled Promise rejections. - - Calls to `console.warn`, including warnings from libraries like Vue. - ### Limitations of jsdom -As mentioned [above](#differences-to-karma), Jest uses jsdom instead of a browser for running tests. +Jest uses jsdom instead of a browser for running tests. This comes with a number of limitations, namely: - [No scrolling support](https://github.com/jsdom/jsdom/blob/15.1.1/lib/jsdom/browser/Window.js#L623-L625) @@ -387,8 +350,7 @@ Sometimes we have to test time-sensitive code. For example, recurring events tha If the application itself is waiting for some time, mock await the waiting. In Jest this is already [done by default](https://gitlab.com/gitlab-org/gitlab/-/blob/a2128edfee799e49a8732bfa235e2c5e14949c68/jest.config.js#L47) -(see also [Jest Timer Mocks](https://jestjs.io/docs/timer-mocks)). In Karma you can use the -[Jasmine mock clock](https://jasmine.github.io/api/2.9/Clock.html). +(see also [Jest Timer Mocks](https://jestjs.io/docs/timer-mocks)). ```javascript const doSomethingLater = () => { @@ -409,20 +371,6 @@ it('does something', () => { }); ``` -**in Karma:** - -```javascript -it('does something', () => { - jasmine.clock().install(); - - doSomethingLater(); - jasmine.clock().tick(4000); - - expect(something).toBe('done'); - jasmine.clock().uninstall(); -}); -``` - ### Mocking the current location in Jest NOTE: @@ -476,8 +424,7 @@ it('passes', () => { Sometimes a test needs to wait for something to happen in the application before it continues. Avoid using [`setTimeout`](https://developer.mozilla.org/en-US/docs/Web/API/WindowOrWorkerGlobalScope/setTimeout) -because it makes the reason for waiting unclear and if used within Karma with a time larger than zero it slows down our test suite. -Instead use one of the following approaches. +because it makes the reason for waiting unclear. Instead use one of the following approaches. #### Promises and Ajax calls @@ -505,19 +452,6 @@ it('waits for an Ajax call', async () => { }); ``` -**in Karma:** - -```javascript -it('waits for an Ajax call', done => { - askTheServer() - .then(() => { - expect(something).toBe('done'); - }) - .then(done) - .catch(done.fail); -}); -``` - If you are not able to register handlers to the `Promise`, for example because it is executed in a synchronous Vue life cycle hook, take a look at the [waitFor](#wait-until-axios-requests-finish) helpers or you can flush all pending `Promise`s: **in Jest:** @@ -548,22 +482,6 @@ it('renders something', () => { }); ``` -**in Karma:** - -```javascript -it('renders something', done => { - wrapper.setProps({ value: 'new value' }); - - wrapper.vm - .$nextTick() - .then(() => { - expect(wrapper.text()).toBe('new value'); - }) - .then(done) - .catch(done.fail); -}); -``` - #### Events If the application triggers an event that you need to wait for in your test, register an event handler which contains @@ -776,8 +694,6 @@ TBU ### Stubbing and Mocking -Jasmine provides stubbing and mocking capabilities. There are some subtle differences in how to use it within Karma and Jest. - Stubs or spies are often used synonymously. In Jest it's quite easy thanks to the `.spyOn` method. [Official docs](https://jestjs.io/docs/jest-object#jestspyonobject-methodname) The more challenging part are mocks, which can be used for functions or even dependencies. @@ -835,7 +751,6 @@ For running the frontend tests, you need the following commands: - `rake frontend:fixtures` (re-)generates [fixtures](#frontend-test-fixtures). Make sure that fixtures are up-to-date before running tests that require them. - `yarn jest` runs Jest tests. -- `yarn karma` runs Karma tests. ### Live testing and focused testing -- Jest @@ -860,49 +775,36 @@ yarn jest ./path/to/folder/ yarn jest term ``` -### Live testing and focused testing -- Karma +## Frontend test fixtures -Karma allows something similar, but it's way more costly. +Frontend fixtures are files containing responses from backend controllers. These responses can be either HTML +generated from HAML templates or JSON payloads. Frontend tests that rely on these responses are +often using fixtures to validate correct integration with the backend code. -Running Karma with `yarn run karma-start` compiles the JavaScript -assets and runs a server at `http://localhost:9876/` where it automatically -runs the tests on any browser which connects to it. You can enter that URL on -multiple browsers at once to have it run the tests on each in parallel. +### Use fixtures -While Karma is running, any changes you make instantly trigger a recompile -and retest of the **entire test suite**, so you can see instantly if you've broken -a test with your changes. You can use [Jasmine focused](https://jasmine.github.io/2.5/focused_specs.html) or -excluded tests (with `fdescribe` or `xdescribe`) to get Karma to run only the -tests you want while you're working on a specific feature, but make sure to -remove these directives when you commit your code. +Jest uses `spec/frontend/__helpers__/fixtures.js` to import fixtures in tests. -It is also possible to only run Karma on specific folders or files by filtering -the run tests via the argument `--filter-spec` or short `-f`: +The following are examples of tests that work for Jest: -```shell -# Run all files -yarn karma-start -# Run specific spec files -yarn karma-start --filter-spec profile/account/components/update_username_spec.js -# Run specific spec folder -yarn karma-start --filter-spec profile/account/components/ -# Run all specs which path contain vue_shared or vie -yarn karma-start -f vue_shared -f vue_mr_widget -``` +```javascript +it('makes a request', () => { + const responseBody = getJSONFixture('some/fixture.json'); // loads spec/frontend/fixtures/some/fixture.json + axiosMock.onGet(endpoint).reply(200, responseBody); -You can also use glob syntax to match files. Remember to put quotes around the -glob otherwise your shell may split it into multiple arguments: + myButton.click(); -```shell -# Run all specs named `file_spec` within the IDE subdirectory -yarn karma -f 'spec/javascripts/ide/**/file_spec.js' -``` + // ... +}); -## Frontend test fixtures +it('uses some HTML element', () => { + loadFixtures('some/page.html'); // loads spec/frontend/fixtures/some/page.html and adds it to the DOM -Frontend fixtures are files containing responses from backend controllers. These responses can be either HTML -generated from HAML templates or JSON payloads. Frontend tests that rely on these responses are -often using fixtures to validate correct integration with the backend code. + const element = document.getElementById('#my-id'); + + // ... +}); +``` ### Generate fixtures @@ -961,34 +863,6 @@ This will create a new fixture located at You can import the JSON fixture in a Jest test using the `getJSONFixture` method [as described below](#use-fixtures). -### Use fixtures - -Jest and Karma test suites import fixtures in different ways: - -- The Karma test suite are served by [jasmine-jquery](https://github.com/velesin/jasmine-jquery). -- Jest use `spec/frontend/__helpers__/fixtures.js`. - -The following are examples of tests that work for both Karma and Jest: - -```javascript -it('makes a request', () => { - const responseBody = getJSONFixture('some/fixture.json'); // loads spec/frontend/fixtures/some/fixture.json - axiosMock.onGet(endpoint).reply(200, responseBody); - - myButton.click(); - - // ... -}); - -it('uses some HTML element', () => { - loadFixtures('some/page.html'); // loads spec/frontend/fixtures/some/page.html and adds it to the DOM - - const element = document.getElementById('#my-id'); - - // ... -}); -``` - ## Data-driven tests Similar to [RSpec's parameterized tests](best_practices.md#table-based--parameterized-tests), @@ -1139,13 +1013,10 @@ Main information on frontend testing levels can be found in the [Testing Levels Tests relevant for frontend development can be found at the following places: -- `spec/javascripts/`, for Karma tests - `spec/frontend/`, for Jest tests - `spec/features/`, for RSpec tests -RSpec runs complete [feature tests](testing_levels.md#frontend-feature-tests), while the Jest and Karma directories contain [frontend unit tests](testing_levels.md#frontend-unit-tests), [frontend component tests](testing_levels.md#frontend-component-tests), and [frontend integration tests](testing_levels.md#frontend-integration-tests). - -All tests in `spec/javascripts/` are intended to be migrated to `spec/frontend/` (see also [#52483](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/52483)). +RSpec runs complete [feature tests](testing_levels.md#frontend-feature-tests), while the Jest directories contain [frontend unit tests](testing_levels.md#frontend-unit-tests), [frontend component tests](testing_levels.md#frontend-component-tests), and [frontend integration tests](testing_levels.md#frontend-integration-tests). Before May 2018, `features/` also contained feature tests run by Spinach. These tests were removed from the codebase in May 2018 ([#23036](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/23036)). @@ -1161,6 +1032,16 @@ If you introduce new helpers, place them in that directory. We have a helper available to make testing actions easier, as per [official documentation](https://vuex.vuejs.org/guide/testing.html): ```javascript +// prefer using like this, a single object argument so parameters are obvious from reading the test +await testAction({ + action: actions.actionName, + payload: { deleteListId: 1 }, + state: { lists: [1, 2, 3] }, + expectedMutations: [ { type: types.MUTATION} ], + expectedActions: [], +}); + +// old way, don't do this for new tests testAction( actions.actionName, // action { }, // params to be passed to action @@ -1177,8 +1058,6 @@ testAction( ); ``` -Check an example in [`spec/frontend/ide/stores/actions_spec.js`](https://gitlab.com/gitlab-org/gitlab/-/blob/fdc7197609dfa7caeb1d962042a26248e49f27da/spec/frontend/ide/stores/actions_spec.js#L392). - ### Wait until Axios requests finish <!-- vale gitlab.Spelling = NO --> diff --git a/doc/development/testing_guide/index.md b/doc/development/testing_guide/index.md index 889dc45d6e3..015d8a92a4d 100644 --- a/doc/development/testing_guide/index.md +++ b/doc/development/testing_guide/index.md @@ -19,7 +19,7 @@ importance. GitLab is built on top of [Ruby on Rails](https://rubyonrails.org/), and we're using [RSpec](https://github.com/rspec/rspec-rails#feature-specs) for all the backend tests, with [Capybara](https://github.com/teamcapybara/capybara) for end-to-end integration testing. -On the frontend side, we're using [Jest](https://jestjs.io/) and [Karma](http://karma-runner.github.io/)/[Jasmine](https://jasmine.github.io/) for JavaScript unit and +On the frontend side, we're using [Jest](https://jestjs.io/) for JavaScript unit and integration testing. Following are two great articles that everyone should read to understand what @@ -40,7 +40,7 @@ system tests, parameterized tests etc. ## [Frontend testing standards and style guidelines](frontend_testing.md) -Everything you should know about how to write good Frontend tests: Karma, +Everything you should know about how to write good Frontend tests: Jest, testing promises, stubbing etc. ## [Flaky tests](flaky_tests.md) diff --git a/doc/development/testing_guide/review_apps.md b/doc/development/testing_guide/review_apps.md index cf757aad870..72d63fd8194 100644 --- a/doc/development/testing_guide/review_apps.md +++ b/doc/development/testing_guide/review_apps.md @@ -175,7 +175,7 @@ For GitLab Team Members only. If you want to sign in to the review app, review the GitLab handbook information for the [shared 1Password account](https://about.gitlab.com/handbook/security/#1password-for-teams). - The default username is `root`. -- The password can be found in the 1Password secure note named `gitlab-{ce,ee} Review App's root password`. +- The password can be found in the 1Password login item named `GitLab EE Review App`. ### Enable a feature flag for my Review App diff --git a/doc/development/testing_guide/testing_levels.md b/doc/development/testing_guide/testing_levels.md index 3a4a28702c7..29cdfab713e 100644 --- a/doc/development/testing_guide/testing_levels.md +++ b/doc/development/testing_guide/testing_levels.md @@ -31,7 +31,7 @@ records should use stubs/doubles as much as possible. | Code path | Tests path | Testing engine | Notes | | --------- | ---------- | -------------- | ----- | -| `app/assets/javascripts/` | `spec/javascripts/`, `spec/frontend/` | Karma & Jest | More details in the [Frontend Testing guide](frontend_testing.md) section. | +| `app/assets/javascripts/` | `spec/frontend/` | Jest | More details in the [Frontend Testing guide](frontend_testing.md) section. | | `app/finders/` | `spec/finders/` | RSpec | | | `app/graphql/` | `spec/graphql/` | RSpec | | | `app/helpers/` | `spec/helpers/` | RSpec | | @@ -233,7 +233,7 @@ They're useful to test permissions, redirections, what view is rendered etc. | `app/controllers/` | `spec/requests/`, `spec/controllers` | RSpec | Request specs are preferred over legacy controller specs. | | `app/mailers/` | `spec/mailers/` | RSpec | | | `lib/api/` | `spec/requests/api/` | RSpec | | -| `app/assets/javascripts/` | `spec/javascripts/`, `spec/frontend/` | Karma & Jest | [More details below](#frontend-integration-tests) | +| `app/assets/javascripts/` | `spec/frontend/` | Jest | [More details below](#frontend-integration-tests) | ### Frontend integration tests @@ -322,13 +322,6 @@ controller.instance_variable_set(:@user, user) and use methods [deprecated in Rails 5](https://gitlab.com/gitlab-org/gitlab/-/issues/16260). -### About Karma - -Karma is both in the Unit tests and the Integration tests category. Karma provides an environment to -run JavaScript tests, so you can either run unit tests (e.g. test a single -JavaScript method), or integration tests (e.g. test a component that is composed -of multiple components). - ## White-box tests at the system level (formerly known as System / Feature tests) Formal definitions: @@ -551,7 +544,7 @@ and the basic idea is that the cost of a test includes: There are cases where the behavior you are testing is not worth the time spent running the full application, for example, if you are testing styling, animation, edge cases or small actions that don't involve the backend, -you should write an integration test using Jasmine. +you should write an integration test using [Frontend integration tests](https://gitlab.com/gitlab-org/gitlab/-/blob/master/spec/frontend_integration/README.md). --- |