Welcome to mirror list, hosted at ThFree Co, Russian Federation.

gitlab.com/gitlab-org/gitlab-foss.git - Unnamed repository; edit this file 'description' to name the repository.
summaryrefslogtreecommitdiff
diff options
context:
space:
mode:
authorGitLab Bot <gitlab-bot@gitlab.com>2020-09-19 04:45:44 +0300
committerGitLab Bot <gitlab-bot@gitlab.com>2020-09-19 04:45:44 +0300
commit85dc423f7090da0a52c73eb66faf22ddb20efff9 (patch)
tree9160f299afd8c80c038f08e1545be119f5e3f1e1 /doc/development/testing_guide
parent15c2c8c66dbe422588e5411eee7e68f1fa440bb8 (diff)
Add latest changes from gitlab-org/gitlab@13-4-stable-ee
Diffstat (limited to 'doc/development/testing_guide')
-rw-r--r--doc/development/testing_guide/best_practices.md279
-rw-r--r--doc/development/testing_guide/end_to_end/beginners_guide.md4
-rw-r--r--doc/development/testing_guide/end_to_end/best_practices.md146
-rw-r--r--doc/development/testing_guide/end_to_end/rspec_metadata_tests.md7
-rw-r--r--doc/development/testing_guide/end_to_end/running_tests_that_require_special_setup.md259
-rw-r--r--doc/development/testing_guide/frontend_testing.md68
-rw-r--r--doc/development/testing_guide/index.md2
-rw-r--r--doc/development/testing_guide/review_apps.md7
8 files changed, 699 insertions, 73 deletions
diff --git a/doc/development/testing_guide/best_practices.md b/doc/development/testing_guide/best_practices.md
index b60a26c29b5..6ef9be381b4 100644
--- a/doc/development/testing_guide/best_practices.md
+++ b/doc/development/testing_guide/best_practices.md
@@ -1,3 +1,11 @@
+---
+type: reference, dev
+stage: none
+group: Development
+info: "See the Technical Writers assigned to Development Guidelines: https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments-to-development-guidelines"
+description: "GitLab development guidelines - testing best practices."
+---
+
# Testing best practices
## Test Design
@@ -15,21 +23,6 @@ manifest themselves within our code. When designing our tests, take time to revi
our test design. We can find some helpful heuristics documented in the Handbook in the
[Test Engineering](https://about.gitlab.com/handbook/engineering/quality/test-engineering/#test-heuristics) section.
-## Test speed
-
-GitLab has a massive test suite that, without [parallelization](ci.md#test-suite-parallelization-on-the-ci), can take hours
-to run. It's important that we make an effort to write tests that are accurate
-and effective _as well as_ fast.
-
-Here are some things to keep in mind regarding test performance:
-
-- `instance_double` and `spy` are faster than `FactoryBot.build(...)`
-- `FactoryBot.build(...)` and `.build_stubbed` are faster than `.create`.
-- Don't `create` an object when `build`, `build_stubbed`, `attributes_for`,
- `spy`, or `instance_double` will do. Database persistence is slow!
-- Don't mark a feature as requiring JavaScript (through `:js` in RSpec) unless it's _actually_ required for the test
- to be valid. Headless browser testing is slow!
-
## RSpec
To run RSpec tests:
@@ -57,13 +50,218 @@ bundle exec guard
When using spring and guard together, use `SPRING=1 bundle exec guard` instead to make use of spring.
-Use [Factory Doctor](https://test-prof.evilmartians.io/#/profilers/factory_doctor) to find cases on un-necessary database manipulation, which can cause slow tests.
+### Test speed
+
+GitLab has a massive test suite that, without [parallelization](ci.md#test-suite-parallelization-on-the-ci), can take hours
+to run. It's important that we make an effort to write tests that are accurate
+and effective _as well as_ fast.
+
+Test performance is important to maintaining quality and velocity, and has a
+direct impact on CI build times and thus fixed costs. We want thorough, correct,
+and fast tests. Here you can find some information about tools and techniques
+available to you to achieve that.
+
+#### Don't request capabilities you don't need
+
+We make it easy to add capabilities to our examples by annotating the example or
+a parent context. Examples of these are:
+
+- `:js` in feature specs, which runs a full JavaScript capable headless browser.
+- `:clean_gitlab_redis_cache` which provides a clean Redis cache to the examples.
+- `:request_store` which provides a request store to the examples.
+
+Obviously we should reduce test dependencies, and avoiding
+capabilities also reduces the amount of set-up needed.
+
+`:js` is particularly important to avoid. This must only be used if the feature
+test requires JavaScript reactivity in the browser, since using a headless
+browser is much slower than parsing the HTML response from the app.
+
+#### Optimize factory usage
+
+A common cause of slow tests is excessive creation of objects, and thus
+computation and DB time. Factories are essential to development, but they can
+make inserting data into the DB so easy that we may be able to optimize.
+
+The two basic techniques to bear in mind here are:
+
+- **Reduce**: avoid creating objects, and avoid persisting them.
+- **Reuse**: shared objects, especially nested ones we do not examine, can generally be shared.
+
+To avoid creation, it is worth bearing in mind that:
+
+- `instance_double` and `spy` are faster than `FactoryBot.build(...)`.
+- `FactoryBot.build(...)` and `.build_stubbed` are faster than `.create`.
+- Don't `create` an object when `build`, `build_stubbed`, `attributes_for`,
+ `spy`, or `instance_double` will do. Database persistence is slow!
+
+Use [Factory Doctor](https://test-prof.evilmartians.io/#/profilers/factory_doctor) to find cases where database persistence is not needed in a given test.
```shell
# run test for path
FDOC=1 bin/rspec spec/[path]/[to]/[spec].rb
```
+A common change is to use `build` or `build_stubbed` instead of `create`:
+
+```ruby
+# Old
+let(:project) { create(:project) }
+
+# New
+let(:project) { build(:project) }
+```
+
+[Factory Profiler](https://test-prof.evilmartians.io/#/profilers/factory_prof) can help to identify repetitive database persistence via factories.
+
+```shell
+# run test for path
+FPROF=1 bin/rspec spec/[path]/[to]/[spec].rb
+
+# to visualize with a flamegraph
+FPROF=flamegraph bin/rspec spec/[path]/[to]/[spec].rb
+```
+
+A common change is to use [`let_it_be`](#common-test-setup):
+
+```ruby
+# Old
+let(:project) { create(:project) }
+
+# New
+let_it_be(:project) { create(:project) }
+```
+
+A common cause of a large number of created factories is [factory cascades](https://github.com/test-prof/test-prof/blob/master/docs/profilers/factory_prof.md#factory-flamegraph), which result when factories create and recreate associations.
+They can be identified by a noticeable difference between `total time` and `top-level time` numbers:
+
+```plaintext
+ total top-level total time time per call top-level time name
+
+ 208 0 9.5812s 0.0461s 0.0000s namespace
+ 208 76 37.4214s 0.1799s 13.8749s project
+```
+
+The table above shows us that we never create any `namespace` objects explicitly
+(`top-level == 0`) - they are all created implicitly for us. But we still end up
+with 208 of them (one for each project) and this takes 9.5 seconds.
+
+In order to reuse a single object for all calls to a named factory in implicit parent associations,
+[`FactoryDefault`](https://github.com/test-prof/test-prof/blob/master/docs/recipes/factory_default.md)
+can be used:
+
+```ruby
+ let_it_be(:namespace) { create_default(:namespace) }
+```
+
+Then every project we create will use this `namespace`, without us having to pass
+it as `namespace: namespace`.
+
+Maybe we don't need to create 208 different projects - we
+can create one and reuse it. In addition, we can see that only about 1/3 of the
+projects we create are ones we ask for (76/208), so there is benefit in setting
+a default value for projects as well:
+
+```ruby
+ let_it_be(:project) { create_default(:project) }
+```
+
+In this case, the `total time` and `top-level time` numbers match more closely:
+
+```plaintext
+ total top-level total time time per call top-level time name
+
+ 31 30 4.6378s 0.1496s 4.5366s project
+ 8 8 0.0477s 0.0477s 0.0477s namespace
+```
+
+#### Identify slow tests
+
+Running a spec with profiling is a good way to start optimizing a spec. This can
+be done with:
+
+```shell
+bundle exec rspec --profile -- path/to/spec_file.rb
+```
+
+Which includes information like the following:
+
+```plaintext
+Top 10 slowest examples (10.69 seconds, 7.7% of total time):
+ Issue behaves like an editable mentionable creates new cross-reference notes when the mentionable text is edited
+ 1.62 seconds ./spec/support/shared_examples/models/mentionable_shared_examples.rb:164
+ Issue relative positioning behaves like a class that supports relative positioning .move_nulls_to_end manages to move nulls to the end, stacking if we cannot create enough space
+ 1.39 seconds ./spec/support/shared_examples/models/relative_positioning_shared_examples.rb:88
+ Issue relative positioning behaves like a class that supports relative positioning .move_nulls_to_start manages to move nulls to the end, stacking if we cannot create enough space
+ 1.27 seconds ./spec/support/shared_examples/models/relative_positioning_shared_examples.rb:180
+ Issue behaves like an editable mentionable behaves like a mentionable extracts references from its reference property
+ 0.99253 seconds ./spec/support/shared_examples/models/mentionable_shared_examples.rb:69
+ Issue behaves like an editable mentionable behaves like a mentionable creates cross-reference notes
+ 0.94987 seconds ./spec/support/shared_examples/models/mentionable_shared_examples.rb:101
+ Issue behaves like an editable mentionable behaves like a mentionable when there are cached markdown fields sends in cached markdown fields when appropriate
+ 0.94148 seconds ./spec/support/shared_examples/models/mentionable_shared_examples.rb:86
+ Issue behaves like an editable mentionable when there are cached markdown fields when the markdown cache is stale persists the refreshed cache so that it does not have to be refreshed every time
+ 0.92833 seconds ./spec/support/shared_examples/models/mentionable_shared_examples.rb:153
+ Issue behaves like an editable mentionable when there are cached markdown fields refreshes markdown cache if necessary
+ 0.88153 seconds ./spec/support/shared_examples/models/mentionable_shared_examples.rb:130
+ Issue behaves like an editable mentionable behaves like a mentionable generates a descriptive back-reference
+ 0.86914 seconds ./spec/support/shared_examples/models/mentionable_shared_examples.rb:65
+ Issue#related_issues returns only authorized related issues for given user
+ 0.84242 seconds ./spec/models/issue_spec.rb:335
+
+Finished in 2 minutes 19 seconds (files took 1 minute 4.42 seconds to load)
+277 examples, 0 failures, 1 pending
+```
+
+From this result, we can see the most expensive examples in our spec, giving us
+a place to start. The fact that the most expensive examples here are in
+shared examples means that any reductions are likely to have a larger impact as
+they are called in multiple places.
+
+#### Avoid repeating expensive actions
+
+While isolated examples are very clear, and help serve the purpose of specs as
+specification, the following example shows how we can combine expensive
+actions:
+
+```ruby
+subject { described_class.new(arg_0, arg_1) }
+
+it 'creates an event' do
+ expect { subject.execute }.to change(Event, :count).by(1)
+end
+
+it 'sets the frobulance' do
+ expect { subject.execute }.to change { arg_0.reset.frobulance }.to('wibble')
+end
+
+it 'schedules a background job' do
+ expect(BackgroundJob).to receive(:perform_async)
+
+ subject.execute
+end
+```
+
+If the call to `subject.execute` is expensive, then we are repeating the same
+action just to make different assertions. We can reduce this repetition by
+combining the examples:
+
+```ruby
+it 'performs the expected side-effects' do
+ expect(BackgroundJob).to receive(:perform_async)
+
+ expect { subject.execute }
+ .to change(Event, :count).by(1)
+ .and change { arg_0.frobulance }.to('wibble')
+end
+```
+
+Be careful doing this, as this sacrifices clarity and test independence for
+performance gains.
+
+When combining tests, consider using `:aggregate_failures`, so that the full
+results are available, and not just the first failure.
+
### General guidelines
- Use a single, top-level `RSpec.describe ClassName` block.
@@ -229,9 +427,9 @@ spec itself, but the former is preferred.
It takes around one second to load tests that are using `fast_spec_helper`
instead of 30+ seconds in case of a regular `spec_helper`.
-### `let` variables
+### `subject` and `let` variables
-GitLab's RSpec suite has made extensive use of `let`(along with it strict, non-lazy
+GitLab's RSpec suite has made extensive use of `let`(along with its strict, non-lazy
version `let!`) variables to reduce duplication. However, this sometimes [comes at the cost of clarity](https://thoughtbot.com/blog/lets-not),
so we need to set some guidelines for their use going forward:
@@ -250,6 +448,9 @@ so we need to set some guidelines for their use going forward:
- `let!` variables should be used only in case if strict evaluation with defined
order is required, otherwise `let` will suffice. Remember that `let` is lazy and won't
be evaluated until it is referenced.
+- Avoid referencing `subject` in examples. Use a named subject `subject(:name)`, or a `let` variable instead, so
+ the variable has a contextual name.
+- If the `subject` is never referenced inside examples, then it's acceptable to define the `subject` without a name.
### Common test setup
@@ -468,6 +669,48 @@ for modifications. If you have no other choice, an `around` block similar to the
example for global variables, above, can be used, but this should be avoided if
at all possible.
+#### Test Snowplow events
+
+CAUTION: **Warning:**
+Snowplow performs **runtime type checks** by using the [contracts gem](https://rubygems.org/gems/contracts).
+Since Snowplow is **by default disabled in tests and development**, it can be hard to
+**catch exceptions** when mocking `Gitlab::Tracking`.
+
+To catch runtime errors due to type checks, you can enable Snowplow in tests by marking the spec with
+`:snowplow` and use the `expect_snowplow_event` helper which will check for
+calls to `Gitlab::Tracking#event`.
+
+```ruby
+describe '#show', :snowplow do
+ it 'tracks snowplow events' do
+ get :show
+
+ expect_snowplow_event(
+ category: 'Experiment',
+ action: 'start',
+ )
+ expect_snowplow_event(
+ category: 'Experiment',
+ action: 'sent',
+ property: 'property',
+ label: 'label'
+ )
+ end
+end
+```
+
+When you want to ensure that no event got called, you can use `expect_no_snowplow_event`.
+
+```ruby
+ describe '#show', :snowplow do
+ it 'does not track any snowplow events' do
+ get :show
+
+ expect_no_snowplow_event
+ end
+ end
+```
+
### Table-based / Parameterized tests
This style of testing is used to exercise one piece of code with a comprehensive
diff --git a/doc/development/testing_guide/end_to_end/beginners_guide.md b/doc/development/testing_guide/end_to_end/beginners_guide.md
index 15a9b4406ab..c552c44c864 100644
--- a/doc/development/testing_guide/end_to_end/beginners_guide.md
+++ b/doc/development/testing_guide/end_to_end/beginners_guide.md
@@ -84,7 +84,7 @@ See the [`RSpec.describe` outer block](#the-outer-rspecdescribe-block)
CAUTION: **Deprecation notice:**
The outer `context` [was deprecated](https://gitlab.com/gitlab-org/quality/team-tasks/-/issues/550) in `13.2`
-in adherance to RSpec 4.0 specifications. Use `RSpec.describe` instead.
+in adherence to RSpec 4.0 specifications. Use `RSpec.describe` instead.
### The outer `RSpec.describe` block
@@ -287,7 +287,7 @@ Note the following important points:
- Our test fabricates only what it needs, when it needs it.
- The issue is fabricated through the API to save time.
- GitLab prefers `let()` over instance variables. See
- [best practices](../best_practices.md#let-variables).
+ [best practices](../best_practices.md#subject-and-let-variables).
- `be_closed` is not implemented in `page/project/issue/show.rb` yet, but will be
implemented in the next step.
diff --git a/doc/development/testing_guide/end_to_end/best_practices.md b/doc/development/testing_guide/end_to_end/best_practices.md
index 3b193721143..36cb49256a6 100644
--- a/doc/development/testing_guide/end_to_end/best_practices.md
+++ b/doc/development/testing_guide/end_to_end/best_practices.md
@@ -1,7 +1,128 @@
# End-to-end testing Best Practices
NOTE: **Note:**
-This is an tailored extension of the Best Practices [found in the testing guide](../best_practices.md).
+This is a tailored extension of the Best Practices [found in the testing guide](../best_practices.md).
+
+## Link a test to its test-case issue
+
+Every test should have a corresponding issue in the [Quality Testcases project](https://gitlab.com/gitlab-org/quality/testcases/).
+It's recommended that you reuse the issue created to plan the test. If one does not already exist you
+can create the issue yourself. Alternatively, you can run the test in a pipeline that has reporting
+enabled and the test-case issue reporter will automatically create a new issue.
+
+Whether you create a new test-case issue or one is created automatically, you will need to manually add
+a `testcase` RSpec metadata tag. In most cases, a single test will be associated with a single test-case
+issue ([see below for exceptions](#exceptions)).
+
+For example:
+
+```ruby
+RSpec.describe 'Stage' do
+ describe 'General description of the feature under test' do
+ it 'test name', testcase: 'https://gitlab.com/gitlab-org/quality/testcases/-/issues/:issue_id' do
+ ...
+ end
+
+ it 'another test', testcase: 'https://gitlab.com/gitlab-org/quality/testcases/-/issues/:another_issue_id' do
+ ...
+ end
+ end
+end
+```
+
+### Exceptions
+
+Most tests are defined by a single line of a `spec` file, which is why those tests can be linked to a
+single test-case issue via the `testcase` tag.
+
+However, some tests don't have a one-to-one relationship between a line of a `spec` file and a test-case
+issue. This is because some tests are defined in a way that means a single line is associated with
+multiple tests, including:
+
+- Parallelized tests.
+- Templated tests.
+- Tests in shared examples that include more than one example.
+
+In those and similar cases we can't assign a single `testcase` tag and so we rely on the test-case
+reporter to programmatically determine the correct test-case issue based on the name and description of
+the test. In such cases, the test-case reporter will automatically create a test-case issue the first time
+the test runs, if no issue exists already.
+
+In such a case, if you create the issue yourself or want to reuse an existing issue,
+you must use this [end-to-end test issue template](https://gitlab.com/gitlab-org/quality/testcases/-/blob/master/.gitlab/issue_templates/End-to-end%20Test.md)
+to format the issue description.
+
+To illustrate, there are two tests in the shared examples in [`qa/specs/features/ee/browser_ui/3_create/repository/restrict_push_protected_branch_spec.rb`](https://gitlab.com/gitlab-org/gitlab/-/blob/47b17db82c38ab704a23b5ba5d296ea0c6a732c8/qa/qa/specs/features/ee/browser_ui/3_create/repository/restrict_push_protected_branch_spec.rb):
+
+```ruby
+shared_examples 'only user with access pushes and merges' do
+ it 'unselected maintainer user fails to push' do
+ ...
+ end
+
+ it 'selected developer user pushes and merges' do
+ ...
+ end
+end
+```
+
+Consider the following test that includes the shared examples:
+
+```ruby
+RSpec.describe 'Create' do
+ describe 'Restricted protected branch push and merge' do
+ context 'when only one user is allowed to merge and push to a protected branch' do
+ ...
+ it_behaves_like 'only user with access pushes and merges'
+ end
+ end
+end
+```
+
+There would be two associated test-case issues, one for each shared example, with the following content:
+
+[Test 1](https://gitlab.com/gitlab-org/quality/testcases/-/issues/600):
+
+````markdown
+```markdown
+Title: browser_ui/3_create/repository/restrict_push_protected_branch_spec.rb | Create Restricted
+protected branch push and merge when only one user is allowed to merge and push to a protected
+branch behaves like only user with access pushes and merges selecte...
+
+Description:
+### Full description
+
+Create Restricted protected branch push and merge when only one user is allowed to merge and push
+to a protected branch behaves like only user with access pushes and merges selected developer user
+pushes and merges
+
+### File path
+
+./qa/specs/features/ee/browser_ui/3_create/repository/restrict_push_protected_branch_spec.rb
+
+```
+````
+
+[Test 2](https://gitlab.com/gitlab-org/quality/testcases/-/issues/602):
+
+````markdown
+```markdown
+Title: browser_ui/3_create/repository/restrict_push_protected_branch_spec.rb | Create Restricted
+protected branch push and merge when only one user is allowed to merge and push to a protected
+branch behaves like only user with access pushes and merges unselec...
+
+Description:
+### Full description
+
+Create Restricted protected branch push and merge when only one user is allowed to merge and push
+to a protected branch behaves like only user with access pushes and merges unselected maintainer
+user fails to push
+
+### File path
+
+./qa/specs/features/ee/browser_ui/3_create/repository/restrict_push_protected_branch_spec.rb
+```
+````
## Prefer API over UI
@@ -166,3 +287,26 @@ end
NOTE: **Note:**
A few exceptions for using a `ProjectPush` would be when your test calls for testing SSH integration or
using the Git CLI.
+
+## Preferred method to blur elements
+
+To blur an element, the preferred method is to click another element that does not alter the test state.
+If there's a mask that blocks the page elements, such as may occur with some dropdowns,
+use WebDriver's native mouse events to simulate a click event on the coordinates of an element. Use the following method: `click_element_coordinates`.
+
+Avoid clicking the `body` for blurring elements such as inputs and dropdowns because it clicks the center of the viewport.
+This action can also unintentionally click other elements, altering the test state and causing it to fail.
+
+```ruby
+# Clicking another element to blur an input
+def add_issue_to_epic(issue_url)
+ find_element(:issue_actions_split_button).find('button', text: 'Add an issue').click
+ fill_element :add_issue_input, issue_url
+ # Clicking the title blurs the input
+ click_element :title
+ click_element :add_issue_button
+end
+
+# Using native mouse click events in the case of a mask/overlay
+click_element_coordinates(:title)
+```
diff --git a/doc/development/testing_guide/end_to_end/rspec_metadata_tests.md b/doc/development/testing_guide/end_to_end/rspec_metadata_tests.md
index 4059c1960e2..a9f54b53e5a 100644
--- a/doc/development/testing_guide/end_to_end/rspec_metadata_tests.md
+++ b/doc/development/testing_guide/end_to_end/rspec_metadata_tests.md
@@ -8,13 +8,14 @@ This is a partial list of the [RSpec metadata](https://relishapp.com/rspec/rspec
| Tag | Description |
|-----|-------------|
| `:elasticsearch` | The test requires an Elasticsearch service. It is used by the [instance-level scenario](https://gitlab.com/gitlab-org/gitlab-qa#definitions) [`Test::Integration::Elasticsearch`](https://gitlab.com/gitlab-org/gitlab/-/blob/72b62b51bdf513e2936301cb6c7c91ec27c35b4d/qa/qa/ee/scenario/test/integration/elasticsearch.rb) to include only tests that require Elasticsearch. |
+| `:gitaly_cluster` | The test will run against a GitLab instance where repositories are stored on redundant Gitaly nodes behind a Praefect node. All nodes are [separate containers](../../../administration/gitaly/praefect.md#requirements-for-configuring-a-gitaly-cluster). Tests that use this tag have a longer setup time since there are three additional containers that need to be started. |
+| `:jira` | The test requires a Jira Server. [GitLab-QA](https://gitlab.com/gitlab-org/gitlab-qa) will provision the Jira Server in a Docker container when the `Test::Integration::Jira` test scenario is run.
| `:kubernetes` | The test includes a GitLab instance that is configured to be run behind an SSH tunnel, allowing a TLS-accessible GitLab. This test will also include provisioning of at least one Kubernetes cluster to test against. *This tag is often be paired with `:orchestrated`.* |
+| `:only` | The test is only to be run against specific environments. See [Environment selection](environment_selection.md) for more information. |
| `:orchestrated` | The GitLab instance under test may be [configured by `gitlab-qa`](https://gitlab.com/gitlab-org/gitlab-qa/-/blob/master/docs/what_tests_can_be_run.md#orchestrated-tests) to be different to the default GitLab configuration, or `gitlab-qa` may launch additional services in separate Docker containers, or both. Tests tagged with `:orchestrated` are excluded when testing environments where we can't dynamically modify GitLab's configuration (for example, Staging). |
| `:quarantine` | The test has been [quarantined](https://about.gitlab.com/handbook/engineering/quality/guidelines/debugging-qa-test-failures/#quarantining-tests), will run in a separate job that only includes quarantined tests, and is allowed to fail. The test will be skipped in its regular job so that if it fails it will not hold up the pipeline. Note that you can also [quarantine a test only when it runs against specific environment](environment_selection.md#quarantining-a-test-for-a-specific-environment). |
| `:reliable` | The test has been [promoted to a reliable test](https://about.gitlab.com/handbook/engineering/quality/guidelines/reliable-tests/#promoting-an-existing-test-to-reliable) meaning it passes consistently in all pipelines, including merge requests. |
| `:requires_admin` | The test requires an admin account. Tests with the tag are excluded when run against Canary and Production environments. |
| `:runner` | The test depends on and will set up a GitLab Runner instance, typically to run a pipeline. |
-| `:gitaly_ha` | The test will run against a GitLab instance where repositories are stored on redundant Gitaly nodes behind a Praefect node. All nodes are [separate containers](../../../administration/gitaly/praefect.md#requirements-for-configuring-a-gitaly-cluster). Tests that use this tag have a longer setup time since there are three additional containers that need to be started. |
| `:skip_live_env` | The test will be excluded when run against live deployed environments such as Staging, Canary, and Production. |
-| `:jira` | The test requires a Jira Server. [GitLab-QA](https://gitlab.com/gitlab-org/gitlab-qa) will provision the Jira Server in a Docker container when the `Test::Integration::Jira` test scenario is run.
-| `:only` | The test is only to be run against specific environments. See [Environment selection](environment_selection.md) for more information. |
+| `:testcase` | The link to the test case issue in the [Quality Testcases project](https://gitlab.com/gitlab-org/quality/testcases/). |
diff --git a/doc/development/testing_guide/end_to_end/running_tests_that_require_special_setup.md b/doc/development/testing_guide/end_to_end/running_tests_that_require_special_setup.md
index 2cf2bb5b1d0..7ac0a00fcff 100644
--- a/doc/development/testing_guide/end_to_end/running_tests_that_require_special_setup.md
+++ b/doc/development/testing_guide/end_to_end/running_tests_that_require_special_setup.md
@@ -134,3 +134,262 @@ Once you have finished testing you can stop and remove the Docker containers:
docker stop gitlab-gitaly-ha praefect postgres gitaly3 gitaly2 gitaly1
docker rm gitlab-gitaly-ha praefect postgres gitaly3 gitaly2 gitaly1
```
+
+## Guide to run and debug Monitor tests
+
+### How to set up
+
+To run the Monitor tests locally, against the GDK, please follow the preparation steps below:
+
+1. Complete the [Prerequisites](https://gitlab.com/gitlab-org/gitlab-development-kit/-/blob/master/doc/howto/auto_devops/index.md#prerequisites-for-gitlab-team-members-only), at least through step 5. Note that the monitor tests do not require permissions to work with GKE because they use [k3s as a Kubernetes cluster provider](https://github.com/rancher/k3s).
+1. The test setup deploys the app in a Kubernetes cluster, using the Auto DevOps deployment strategy.
+To enable Auto DevOps in GDK, follow the [associated setup](https://gitlab.com/gitlab-org/gitlab-development-kit/-/blob/master/doc/howto/auto_devops/index.md#setup) instructions. If you have problems, review the [troubleshooting guide](https://gitlab.com/gitlab-org/gitlab-development-kit/-/blob/master/doc/howto/auto_devops/tips_and_troubleshooting.md) or reach out to the `#gdk` channel in the internal GitLab Slack.
+1. Do [secure your GitLab instance](https://gitlab.com/gitlab-org/gitlab-development-kit/-/blob/master/doc/howto/auto_devops/index.md#secure-your-gitlab-instance) since it is now publicly accessible on `https://[YOUR-PORT].qa-tunnel.gitlab.info`.
+1. Install the Kubernetes command line tool known as `kubectl`. Use the [official installation instructions](https://kubernetes.io/docs/tasks/tools/install-kubectl/).
+
+You might see NGINX issues when you run `gdk start` or `gdk restart`. In that case, run `sft login` to revalidate your credentials and regain access the QA Tunnel.
+
+### How to run
+
+Navigate to the folder in `/your-gdk/gitlab/qa` and issue the command:
+
+```shell
+QA_DEBUG=true CHROME_HEADLESS=false GITLAB_ADMIN_USERNAME=rootusername GITLAB_ADMIN_PASSWORD=rootpassword GITLAB_QA_ACCESS_TOKEN=your_token_here GITLAB_QA_ADMIN_ACCESS_TOKEN=your_token_here CLUSTER_API_URL=https://kubernetes.docker.internal:6443 bundle exec bin/qa Test::Instance::All https://[YOUR-PORT].qa-tunnel.gitlab.info/ -- qa/specs/features/browser_ui/8_monitor/all_monitor_core_features_spec.rb --tag kubernetes --tag orchestrated --tag requires_admin
+```
+
+The following includes more information on the command:
+
+-`QA_DEBUG` - Set to `true` to verbosely log page object actions.
+-`CHROME_HEADLESS` - When running locally, set to `false` to allow Chrome tests to be visible - watch your tests being run.
+-`GITLAB_ADMIN_USERNAME` - Admin username to use when adding a license.
+-`GITLAB_ADMIN_PASSWORD` - Admin password to use when adding a license.
+-`GITLAB_QA_ACCESS_TOKEN` and `GITLAB_QA_ADMIN_ACCESS_TOKEN` - A valid personal access token with the `api` scope. This is used for API access during tests, and is used in the version that staging is currently running. The `ADMIN_ACCESS_TOKEN` is from a user with admin access. Used for API access as an admin during tests.
+-`CLUSTER_API_URL` - Use the address `https://kubernetes.docker.internal:6443` . This address is used to enable the cluster to be network accessible while deploying using Auto DevOps.
+-`https://[YOUR-PORT].qa-tunnel.gitlab.info/` - The address of your local GDK
+-`qa/specs/features/browser_ui/8_monitor/all_monitor_core_features_spec.rb` - The path to the monitor core specs
+-`--tag` - the meta-tags used to filter the specs correctly
+
+At the moment of this writing, there are two specs which run monitor tests:
+
+-`qa/specs/features/browser_ui/8_monitor/all_monitor_core_features_spec.rb` - has the specs of features in GitLab Core
+-`qa/specs/features/ee/browser_ui/8_monitor/all_monitor_features_spec.rb` - has the specs of features for paid GitLab (Enterprise Edition)
+
+### How to debug
+
+The monitor tests follow this setup flow:
+
+1. Creates a k3s cluster on your local machine.
+1. Creates a project that has Auto DevOps enabled and uses an Express template (NodeJS) for the app to be deployed.
+1. Associates the created cluster to the project and installs GitLab Runner, Prometheus and Ingress which are the needed components for a successful deployment.
+1. Creates a CI pipeline with 2 jobs (`build` and `production`) to deploy the app on the Kubernetes cluster.
+1. Goes to Operation > Metrics menu to verify data is being received and the app is being monitored successfully.
+
+The test requires a number of components. The setup requires time to collect the metrics of a real deployment.
+The complexity of the setup may lead to problems unrelated to the app. The following sections include common strategies to debug possible issues.
+
+#### Deployment with Auto DevOps
+
+When debugging issues in the CI or locally in the CLI, open the Kubernetes job in the pipeline.
+In the job log window, click on the top right icon labeled as *"Show complete raw"* to reveal raw job logs.
+You can now search through the logs for *Job log*, which matches delimited sections like this one:
+
+```shell
+------- Job log: -------
+```
+
+A Job log is a subsection within these logs, related to app deployment. We use two jobs: `build` and `production`.
+You can find the root causes of deployment failures in these logs, which can compromise the entire test.
+If a `build` job fails, the `production` job won't run, and the test fails.
+
+The long test setup does not take screenshots of failures, which is a known [issue](https://gitlab.com/gitlab-org/quality/team-tasks/-/issues/270).
+However, if the spec fails (after a successful deployment) then you should be able to find screenshots which display the feature failure.
+To access them in CI, go to the main job log window, look on the left side panel's Job artifacts section, and click Browse.
+
+#### Common issues
+
+**Container Registry**
+
+When enabling Auto DevOps in the GDK, you may see issues with the Container Registry, which stores
+images of the app to be deployed.
+
+You can access the Registry is available by opening an existing project. On the left hand menu,
+select `Packages & Registries > Container Registries`. If the Registry is available, this page should load normally.
+
+Also, the Registry should be running in Docker:
+
+```shell
+$ docker ps
+
+CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
+f035f339506c registry.gitlab.com/gitlab-org/build/cng/gitlab-container-registry:v2.9.1-gitlab "/bin/sh -c 'exec /b…" 3 hours ago Up 3 hours 0.0.0.0:5000->5000/tcp jovial_proskuriakova
+```
+
+The `gdk status` command shows if the registry is running:
+
+```shell
+run: ./services/registry: (pid 2662) 10875s, normally down; run: log: (pid 65148) 177993s
+run: ./services/tunnel_gitlab: (pid 2650) 10875s, normally down; run: log: (pid 65154) 177993s
+run: ./services/tunnel_registry: (pid 2651) 10875s, normally down; run: log: (pid 65155) 177993s
+```
+
+Also, restarting Docker and then, on the Terminal, issue the command `docker login https://[YOUR-REGISTRY-PORT].qa-tunnel.gitlab.info:443` and use the GDK credentials to login. Note that the Registry port and GDK port are not the same. When configuring Auto DevOps in GDK, the `gdk reconfigure` command outputs the port of the Registry:
+
+```shell
+*********************************************
+Tunnel URLs
+
+GitLab: https://[PORT].qa-tunnel.gitlab.info
+Registry: https://[PORT].qa-tunnel.gitlab.info
+*********************************************
+```
+
+These Tunnel URLs are used by the QA SSH Tunnel generated when enabling Auto DevOps on the GDK.
+
+**Pod Eviction**
+
+Pod eviction happens when a node in a Kubernetes cluster is running out of memory or disk. After many local deployments this issue can happen. The UI shows that installing Prometheus, GitLab Runner and Ingress failed. How to be sure it is an Eviction? While the test is running, open another Terminal window and debug the current Kubernetes cluster by `kubectl get pods --all-namespaces`. If you observe that Pods have *Evicted status* such as the install-runner here:
+
+```shell
+$ kubectl get pods --all-namespaces
+
+NAMESPACE NAME READY STATUS RESTARTS AGE
+gitlab-managed-apps install-ingress 0/1 Pending 0 25s
+gitlab-managed-apps install-prometheus 0/1 Pending 0 12s
+gitlab-managed-apps install-runner 0/1 Evicted 0 75s
+```
+
+You can free some memory with either of the following commands: `docker prune system` or `docker prune volume`.
+
+## Geo tests
+
+Geo end-to-end tests can run locally against a [Geo GDK setup](https://gitlab.com/gitlab-org/gitlab-development-kit/-/blob/master/doc/howto/geo.md) or on Geo spun up in Docker containers.
+
+### Using Geo GDK
+
+Run from the [`qa/` directory](https://gitlab.com/gitlab-org/gitlab/-/blob/f7272b77e80215c39d1ffeaed27794c220dbe03f/qa) with both GDK Geo primary and Geo secondary instances running:
+
+```shell
+CHROME_HEADLESS=false bundle exec bin/qa QA::EE::Scenario::Test::Geo --primary-address http://localhost:3001 --secondary-address http://localhost:3002 --without-setup
+```
+
+### Using Geo in Docker
+
+You can use [GitLab-QA Orchestrator](https://gitlab.com/gitlab-org/gitlab-qa) to orchestrate two GitLab containers and configure them as a Geo setup.
+
+Geo requires an EE license. To visit the Geo sites in your browser, you will need a reverse proxy server (for example, [NGINX](https://www.nginx.com/)).
+
+1. Export your EE license
+
+ ```shell
+ export EE_LICENSE=$(cat <path/to/your/gitlab_license>)
+ ```
+
+1. (Optional) Pull the GitLab image
+
+ This step is optional because pulling the Docker image is part of the [`Test::Integration::Geo` orchestrated scenario](https://gitlab.com/gitlab-org/gitlab-qa/-/blob/d8c5c40607c2be0eda58bbca1b9f534b00889a0b/lib/gitlab/qa/scenario/test/integration/geo.rb). However, it's easier to monitor the download progress if you pull the image first, and the scenario will skip this step after checking that the image is up to date.
+
+ ```shell
+ # For the most recent nightly image
+ docker pull gitlab/gitlab-ee:nightly
+
+ # For a specific release
+ docker pull gitlab/gitlab-ee:13.0.10-ee.0
+
+ # For a specific image
+ docker pull registry.gitlab.com/gitlab-org/build/omnibus-gitlab-mirror/gitlab-ee:examplesha123456789
+ ```
+
+1. Run the [`Test::Integration::Geo` orchestrated scenario](https://gitlab.com/gitlab-org/gitlab-qa/-/blob/d8c5c40607c2be0eda58bbca1b9f534b00889a0b/lib/gitlab/qa/scenario/test/integration/geo.rb) with the `--no-teardown` option to build the GitLab containers, configure the Geo setup, and run Geo end-to-end tests. Running the tests after the Geo setup is complete is optional; the containers will keep running after you stop the tests.
+
+ ```shell
+ # Using the most recent nightly image
+ gitlab-qa Test::Integration::Geo EE --no-teardown
+
+ # Using a specific GitLab release
+ gitlab-qa Test::Integration::Geo EE:13.0.10-ee.0 --no-teardown
+
+ # Using a full image address
+ GITLAB_QA_ACCESS_TOKEN=your-token-here gitlab-qa Test::Integration::Geo registry.gitlab.com/gitlab-org/build/omnibus-gitlab-mirror/gitlab-ee:examplesha123456789 --no-teardown
+ ```
+
+ You can use the `--no-tests` option to build the containers only, and then run the [`EE::Scenario::Test::Geo` scenario](https://gitlab.com/gitlab-org/gitlab/-/blob/f7272b77e80215c39d1ffeaed27794c220dbe03f/qa/qa/ee/scenario/test/geo.rb) from your GDK to complete setup and run tests. However, there might be configuration issues if your GDK and the containers are based on different GitLab versions. With the `--no-teardown` option, GitLab-QA uses the same GitLab version for the GitLab containers and the GitLab QA container used to configure the Geo setup.
+
+1. To visit the Geo sites in your browser, proxy requests to the hostnames used inside the containers. NGINX is used as the reverse proxy server for this example.
+
+ _Map the hostnames to the local IP in `/etc/hosts` file on your machine:_
+
+ ```plaintext
+ 127.0.0.1 gitlab-primary.geo gitlab-secondary.geo
+ ```
+
+ _Note the assigned ports:_
+
+ ```shell
+ $ docker port gitlab-primary
+
+ 80/tcp -> 0.0.0.0:32768
+
+ $ docker port gitlab-secondary
+
+ 80/tcp -> 0.0.0.0:32769
+ ```
+
+ _Configure the reverse proxy server with the assigned ports in `nginx.conf` file (usually found in `/usr/local/etc/nginx` on a Mac):_
+
+ ```plaintext
+ server {
+ server_name gitlab-primary.geo;
+ location / {
+ proxy_pass http://localhost:32768; # Change port to your assigned port
+ proxy_set_header Host gitlab-primary.geo;
+ }
+ }
+
+ server {
+ server_name gitlab-secondary.geo;
+ location / {
+ proxy_pass http://localhost:32769; # Change port to your assigned port
+ proxy_set_header Host gitlab-secondary.geo;
+ }
+ }
+ ```
+
+ _Start or reload the reverse proxy server:_
+
+ ```shell
+ sudo nginx
+ # or
+ sudo nginx -s reload
+ ```
+
+1. To run end-to-end tests from your local GDK, run the [`EE::Scenario::Test::Geo` scenario](https://gitlab.com/gitlab-org/gitlab/-/blob/f7272b77e80215c39d1ffeaed27794c220dbe03f/qa/qa/ee/scenario/test/geo.rb) from the [`gitlab/qa/` directory](https://gitlab.com/gitlab-org/gitlab/-/blob/f7272b77e80215c39d1ffeaed27794c220dbe03f/qa). Include `--without-setup` to skip the Geo configuration steps.
+
+ ```shell
+ QA_DEBUG=true GITLAB_QA_ACCESS_TOKEN=[add token here] GITLAB_QA_ADMIN_ACCESS_TOKEN=[add token here] bundle exec bin/qa QA::EE::Scenario::Test::Geo \
+ --primary-address http://gitlab-primary.geo \
+ --secondary-address http://gitlab-secondary.geo \
+ --without-setup
+ ```
+
+ If the containers need to be configured first (for example, if you used the `--no-tests` option in the previous step), run the `QA::EE::Scenario::Test::Geo scenario` as shown below to first do the Geo configuration steps, and then run Geo end-to-end tests. Make sure that `EE_LICENSE` is (still) defined in your shell session.
+
+ ```shell
+ QA_DEBUG=true bundle exec bin/qa QA::EE::Scenario::Test::Geo \
+ --primary-address http://gitlab-primary.geo \
+ --primary-name gitlab-primary \
+ --secondary-address http://gitlab-secondary.geo \
+ --secondary-name gitlab-secondary
+ ```
+
+1. Stop and remove containers
+
+ ```shell
+ docker stop gitlab-primary gitlab-secondary
+ docker rm gitlab-primary gitlab-secondary
+ ```
+
+#### Notes
+
+- You can find the full image address from a pipeline by [following these instructions](https://about.gitlab.com/handbook/engineering/quality/guidelines/tips-and-tricks/#running-gitlab-qa-pipeline-against-a-specific-gitlab-release). You might be prompted to set the `GITLAB_QA_ACCESS_TOKEN` variable if you specify the full image address.
+- You can increase the wait time for replication by setting `GEO_MAX_FILE_REPLICATION_TIME` and `GEO_MAX_DB_REPLICATION_TIME`. The default is 120 seconds.
+- To save time during tests, create a Personal Access Token with API access on the Geo primary node, and pass that value in as `GITLAB_QA_ACCESS_TOKEN` and `GITLAB_QA_ADMIN_ACCESS_TOKEN`.
diff --git a/doc/development/testing_guide/frontend_testing.md b/doc/development/testing_guide/frontend_testing.md
index 83d03097466..30e78766dde 100644
--- a/doc/development/testing_guide/frontend_testing.md
+++ b/doc/development/testing_guide/frontend_testing.md
@@ -230,7 +230,7 @@ it('exists', () => {
// Best
// NOTE: both mount and shallowMount work as long as a DOM element is available
- // Finds a properly formatted link with an accessable name of "Click Me"
+ // Finds a properly formatted link with an accessible name of "Click Me"
getByRole(el, 'link', { name: /Click Me/i })
getByRole(el, 'link', { name: 'Click Me' })
// Finds any element with the text "Click Me"
@@ -321,80 +321,56 @@ it('tests a promise', async () => {
});
it('tests a promise rejection', async () => {
- expect.assertions(1);
- try {
- await user.getUserName(1);
- } catch (e) {
- expect(e).toEqual({
- error: 'User with 1 not found.',
- });
- }
+ await expect(user.getUserName(1)).rejects.toThrow('User with 1 not found.');
});
```
-You can also work with Promise chains. In this case, you can make use of the `done` callback and `done.fail` in case an error occurred. Following are some examples:
+You can also simply return a promise from the test function.
+
+NOTE: **Note:**
+Using the `done` and `done.fail` callbacks is discouraged when working with
+promises. They should only be used when testing callback-based code.
**Bad**:
```javascript
-// missing done callback
+// missing return
it('tests a promise', () => {
promise.then(data => {
expect(data).toBe(asExpected);
});
});
-// missing catch
-it('tests a promise', done => {
- promise
- .then(data => {
- expect(data).toBe(asExpected);
- })
- .then(done);
-});
-
-// use done.fail in asynchronous tests
+// uses done/done.fail
it('tests a promise', done => {
promise
.then(data => {
expect(data).toBe(asExpected);
})
.then(done)
- .catch(fail);
-});
-
-// missing catch
-it('tests a promise rejection', done => {
- promise
- .catch(error => {
- expect(error).toBe(expectedError);
- })
- .then(done);
+ .catch(done.fail);
});
```
**Good**:
```javascript
-// handling success
-it('tests a promise', done => {
- promise
+// verifying a resolved promise
+it('tests a promise', () => {
+ return promise
.then(data => {
expect(data).toBe(asExpected);
- })
- .then(done)
- .catch(done.fail);
+ });
});
-// failure case
-it('tests a promise rejection', done => {
- promise
- .then(done.fail)
- .catch(error => {
- expect(error).toBe(expectedError);
- })
- .then(done)
- .catch(done.fail);
+// verifying a resolved promise using Jest's `resolves` matcher
+it('tests a promise', () => {
+ return expect(promise).resolves.toBe(asExpected);
+});
+
+// verifying a rejected promise using Jest's `rejects` matcher
+it('tests a promise rejection', () => {
+ return expect(promise).rejects.toThrow(expectedError);
});
```
diff --git a/doc/development/testing_guide/index.md b/doc/development/testing_guide/index.md
index 0d470e0e737..a61a700594c 100644
--- a/doc/development/testing_guide/index.md
+++ b/doc/development/testing_guide/index.md
@@ -4,7 +4,7 @@ This document describes various guidelines and best practices for automated
testing of the GitLab project.
It is meant to be an _extension_ of the [thoughtbot testing
-style guide](https://github.com/thoughtbot/guides/tree/master/style/testing). If
+style guide](https://github.com/thoughtbot/guides/tree/master/testing-rspec). If
this guide defines a rule that contradicts the thoughtbot guide, this guide
takes precedence. Some guidelines may be repeated verbatim to stress their
importance.
diff --git a/doc/development/testing_guide/review_apps.md b/doc/development/testing_guide/review_apps.md
index 68816ccfe45..61d3299cabf 100644
--- a/doc/development/testing_guide/review_apps.md
+++ b/doc/development/testing_guide/review_apps.md
@@ -165,8 +165,11 @@ This will grant you the following permissions for:
### Log into my Review App
-The default username is `root` and its password can be found in the 1Password
-secure note named `gitlab-{ce,ee} Review App's root password`.
+For GitLab Team Members only. If you want to sign in to the review app, review
+the GitLab handbook information for the [shared 1Password account](https://about.gitlab.com/handbook/security/#1password-for-teams).
+
+- The default username is `root`.
+- The password can be found in the 1Password secure note named `gitlab-{ce,ee} Review App's root password`.
### Enable a feature flag for my Review App