diff options
Diffstat (limited to 'doc/development/testing_guide/end_to_end')
6 files changed, 107 insertions, 26 deletions
diff --git a/doc/development/testing_guide/end_to_end/beginners_guide.md b/doc/development/testing_guide/end_to_end/beginners_guide.md index b81379d89b2..78e6af4a347 100644 --- a/doc/development/testing_guide/end_to_end/beginners_guide.md +++ b/doc/development/testing_guide/end_to_end/beginners_guide.md @@ -34,9 +34,8 @@ For more information, see [End-to-end testing Best Practices](best_practices.md) ## Determine if end-to-end tests are needed -Check the code coverage of a specific feature before writing end-to-end tests, -for both [GitLab Community Edition](https://gitlab-org.gitlab.io/gitlab-foss/coverage-ruby/#_AllFiles) -and [GitLab Enterprise Edition](https://gitlab-org.gitlab.io/gitlab/coverage-ruby/#_AllFiles) projects. +Check the code coverage of a specific feature before writing end-to-end tests +for the [GitLab](https://gitlab-org.gitlab.io/gitlab/coverage-ruby/#_AllFiles) project. Does sufficient test coverage exist at the unit, feature, or integration levels? If you answered *yes*, then you *don't* need an end-to-end test. @@ -53,9 +52,8 @@ For information about the distribution of tests per level in GitLab, see the feature and the lower-level tests. WARNING: -Check both [GitLab Community Edition](https://gitlab-org.gitlab.io/gitlab-foss/coverage-ruby/#_AllFiles) and -[GitLab Enterprise Edition](https://gitlab-org.gitlab.io/gitlab/coverage-ruby/#_AllFiles) coverage projects -for previously-written tests for this feature. For analyzing the code coverage, +Check the [GitLab](https://gitlab-org.gitlab.io/gitlab/coverage-ruby/#_AllFiles) coverage project +for previously written tests for this feature. To analyze code coverage, you must understand which application files implement specific features. In this tutorial we're writing a login end-to-end test, even though it has been @@ -87,7 +85,7 @@ file `basic_login_spec.rb`. See the [`RSpec.describe` outer block](#the-outer-rspecdescribe-block) WARNING: -The outer `context` [was deprecated](https://gitlab.com/gitlab-org/quality/team-tasks/-/issues/550) in `13.2` +The outer `context` [was deprecated](https://gitlab.com/gitlab-org/quality/quality-engineering/team-tasks/-/issues/550) in `13.2` in adherence to RSpec 4.0 specifications. Use `RSpec.describe` instead. ### The outer `RSpec.describe` block @@ -225,7 +223,7 @@ end 1. Check if the user avatar *does not* appear in the top navigation. Behind the scenes, `be_signed_in` is a -[predicate matcher](https://relishapp.com/rspec/rspec-expectations/v/3-8/docs/built-in-matchers/predicate-matchers) +[predicate matcher](https://rspec.info/features/3-12/rspec-expectations/built-in-matchers/predicates/) that [implements checking the user avatar](https://gitlab.com/gitlab-org/gitlab/-/blob/master/qa/qa/page/main/menu.rb#L92). ## De-duplicate your code diff --git a/doc/development/testing_guide/end_to_end/best_practices.md b/doc/development/testing_guide/end_to_end/best_practices.md index 777a9f47339..c1f06bb9a66 100644 --- a/doc/development/testing_guide/end_to_end/best_practices.md +++ b/doc/development/testing_guide/end_to_end/best_practices.md @@ -16,9 +16,9 @@ In case custom inflection logic is needed, custom inflectors are added in the [q ## Link a test to its test case -Every test should have a corresponding test case in the [GitLab project Test Cases](https://gitlab.com/gitlab-org/gitlab/-/quality/test_cases) as well as a results issue in the [Quality Test Cases project](https://gitlab.com/gitlab-org/quality/testcases/-/issues). +Every test should have a corresponding test case in the [GitLab project test cases](https://gitlab.com/gitlab-org/gitlab/-/quality/test_cases) as well as a results issue in the [Quality Test Cases project](https://gitlab.com/gitlab-org/quality/testcases/-/issues). If a test case issue does not yet exist, any GitLab team member can create a new test case in -the [Test Cases](https://gitlab.com/gitlab-org/gitlab/-/quality/test_cases) GitLab project +the **[CI/CD > Test cases](https://gitlab.com/gitlab-org/gitlab/-/quality/test_cases)** page of the GitLab project with a placeholder title. After the test case URL is linked to a test in the code, when the test is run in a pipeline that has reporting enabled, the `report-results` script automatically updates the test case and the results issue. @@ -413,7 +413,7 @@ except(page).to have_no_text('hidden') ``` Unfortunately, that's not automatically the case for the predicate methods that we add to our -[page objects](page_objects.md). We need to [create our own negatable matchers](https://relishapp.com/rspec/rspec-expectations/v/3-9/docs/custom-matchers/define-a-custom-matcher#matcher-with-separate-logic-for-expect().to-and-expect().not-to). +[page objects](page_objects.md). We need to [create our own negatable matchers](https://rspec.info/features/3-12/rspec-expectations/custom-matchers/define-matcher/). The initial example uses the `have_job` matcher which is derived from the [`has_job?` predicate method of the `Page::Project::Pipeline::Show` page object](https://gitlab.com/gitlab-org/gitlab/-/blob/87864b3047c23b4308f59c27a3757045944af447/qa/qa/page/project/pipeline/show.rb#L53). diff --git a/doc/development/testing_guide/end_to_end/execution_context_selection.md b/doc/development/testing_guide/end_to_end/execution_context_selection.md index 23b8ab7287b..4d83884f4d0 100644 --- a/doc/development/testing_guide/end_to_end/execution_context_selection.md +++ b/doc/development/testing_guide/end_to_end/execution_context_selection.md @@ -34,7 +34,8 @@ Matches use: - Regex for environments. - String matching for pipelines. -- Regex or string matching for jobs. +- Regex or string matching for jobs +- Lambda or truthy/falsey value for generic condition | Test execution context | Key | Matches | | ---------------- | --- | --------------- | @@ -47,6 +48,7 @@ Matches use: | The `nightly` and `canary` pipelines | `only: { pipeline: [:nightly, :canary] }` | ["nightly"](https://gitlab.com/gitlab-org/quality/nightly) and ["canary"](https://gitlab.com/gitlab-org/quality/canary) | | The `ee:instance` job | `only: { job: 'ee:instance' }` | The `ee:instance` job in any pipeline | | Any `quarantine` job | `only: { job: '.*quarantine' }` | Any job ending in `quarantine` in any pipeline | +| Any run where condition evaluates to a truthy value | `only: { condition: -> { ENV['TEST_ENV'] == 'true' } }` | Any run where `TEST_ENV` is set to true ```ruby RSpec.describe 'Area' do @@ -62,6 +64,8 @@ RSpec.describe 'Area' do it 'runs only in nightly pipeline', only: { pipeline: :nightly } do; end it 'runs in nightly and canary pipelines', only: { pipeline: [:nightly, :canary] } do; end + + it 'runs in specific environment matching condition', only: { condition: -> { ENV['TEST_ENV'] == 'true' } } do; end end ``` @@ -73,7 +77,8 @@ Matches use: - Regex for environments. - String matching for pipelines. -- Regex or string matching for jobs. +- Regex or string matching for jobs +- Lambda or truthy/falsey value for generic condition | Test execution context | Key | Matches | | ---------------- | --- | --------------- | @@ -86,6 +91,7 @@ Matches use: | The `nightly` and `canary` pipelines | `except: { pipeline: [:nightly, :canary] }` | ["nightly"](https://gitlab.com/gitlab-org/quality/nightly) and ["canary"](https://gitlab.com/gitlab-org/quality/canary) | | The `ee:instance` job | `except: { job: 'ee:instance' }` | The `ee:instance` job in any pipeline | | Any `quarantine` job | `except: { job: '.*quarantine' }` | Any job ending in `quarantine` in any pipeline | +| Any run except where condition evaluates to a truthy value | `except: { condition: -> { ENV['TEST_ENV'] == 'true' } }` | Any run where `TEST_ENV` is not set to true ```ruby RSpec.describe 'Area' do @@ -96,6 +102,8 @@ RSpec.describe 'Area' do it 'runs in any execution context except the nightly pipeline', except: { pipeline: :nightly } do; end it 'runs in any execution context except the ee:instance job', except: { job: 'ee:instance' } do; end + + it 'runs in specific environment not matching condition', except: { condition: -> { ENV['TEST_ENV'] == 'true' } } do; end end ``` diff --git a/doc/development/testing_guide/end_to_end/resources.md b/doc/development/testing_guide/end_to_end/resources.md index f3e072ffa21..f9e136a86df 100644 --- a/doc/development/testing_guide/end_to_end/resources.md +++ b/doc/development/testing_guide/end_to_end/resources.md @@ -249,7 +249,7 @@ end ``` The `populate` method iterates through its arguments and call each -attribute respectively. Here `populate(:brand)` has the same effect as +attribute. Here `populate(:brand)` has the same effect as just `brand`. Using the populate method makes the intention clearer. With this, it ensures we construct the data right after we create the diff --git a/doc/development/testing_guide/end_to_end/rspec_metadata_tests.md b/doc/development/testing_guide/end_to_end/rspec_metadata_tests.md index c1389b3ac0e..a42d5e3df1d 100644 --- a/doc/development/testing_guide/end_to_end/rspec_metadata_tests.md +++ b/doc/development/testing_guide/end_to_end/rspec_metadata_tests.md @@ -6,7 +6,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w # RSpec metadata for end-to-end tests -This is a partial list of the [RSpec metadata](https://relishapp.com/rspec/rspec-core/docs/metadata/user-defined-metadata) +This is a partial list of the [RSpec metadata](https://rspec.info/features/3-12/rspec-core/metadata/user-defined/) (a.k.a. tags) that are used in our end-to-end tests. <!-- Please keep the tags in alphabetical order --> @@ -52,5 +52,5 @@ This is a partial list of the [RSpec metadata](https://relishapp.com/rspec/rspec | `:skip_signup_disabled` | The test uses UI to sign up a new user and is skipped in any environment that does not allow new user registration via the UI. | | `:smoke` | The test belongs to the test suite which verifies basic functionality of a GitLab instance. | | `:smtp` | The test requires a GitLab instance to be configured to use an SMTP server. Tests SMTP notification email delivery from GitLab by using MailHog. | -| `:testcase` | The link to the test case issue in the [GitLab Project Test Cases](https://gitlab.com/gitlab-org/gitlab/-/quality/test_cases). | +| `:testcase` | The link to the test case issue in the [GitLab Project test cases](https://gitlab.com/gitlab-org/gitlab/-/quality/test_cases). | | `:transient` | The test tests transient bugs. It is excluded by default. | diff --git a/doc/development/testing_guide/end_to_end/running_tests_that_require_special_setup.md b/doc/development/testing_guide/end_to_end/running_tests_that_require_special_setup.md index 4a947e59d5f..adf7bccb7fb 100644 --- a/doc/development/testing_guide/end_to_end/running_tests_that_require_special_setup.md +++ b/doc/development/testing_guide/end_to_end/running_tests_that_require_special_setup.md @@ -219,7 +219,7 @@ Geo requires an EE license. To visit the Geo sites in your browser, you need a r # Using a full image address GITLAB_QA_ACCESS_TOKEN=your-token-here gitlab-qa Test::Integration::Geo registry.gitlab.com/gitlab-org/build/omnibus-gitlab-mirror/gitlab-ee:examplesha123456789 --no-teardown - ``` + ``` You can use the `--no-tests` option to build the containers only, and then run the [`EE::Scenario::Test::Geo` scenario](https://gitlab.com/gitlab-org/gitlab/-/blob/f7272b77e80215c39d1ffeaed27794c220dbe03f/qa/qa/ee/scenario/test/geo.rb) from your GDK to complete setup and run tests. However, there might be configuration issues if your GDK and the containers are based on different GitLab versions. With the `--no-teardown` option, GitLab-QA uses the same GitLab version for the GitLab containers and the GitLab QA container used to configure the Geo setup. @@ -369,15 +369,15 @@ To run the LDAP tests on your local with TLS disabled, follow these steps: 1. Run the GitLab container: - ```shell - sudo docker run \ - --hostname localhost \ - --net test \ - --publish 443:443 --publish 80:80 --publish 22:22 \ - --name gitlab \ - --env GITLAB_OMNIBUS_CONFIG="gitlab_rails['ldap_enabled'] = true; gitlab_rails['ldap_servers'] = {\"main\"=>{\"label\"=>\"LDAP\", \"host\"=>\"ldap-server.test\", \"port\"=>389, \"uid\"=>\"uid\", \"bind_dn\"=>\"cn=admin,dc=example,dc=org\", \"password\"=>\"admin\", \"encryption\"=>\"plain\", \"verify_certificates\"=>false, \"base\"=>\"dc=example,dc=org\", \"user_filter\"=>\"\", \"group_base\"=>\"ou=Global Groups,dc=example,dc=org\", \"admin_group\"=>\"AdminGroup\", \"external_groups\"=>\"\", \"sync_ssh_keys\"=>false}}; gitlab_rails['ldap_sync_worker_cron'] = '* * * * *'; gitlab_rails['ldap_group_sync_worker_cron'] = '* * * * *'; " \ - gitlab/gitlab-ee:latest - ``` + ```shell + sudo docker run \ + --hostname localhost \ + --net test \ + --publish 443:443 --publish 80:80 --publish 22:22 \ + --name gitlab \ + --env GITLAB_OMNIBUS_CONFIG="gitlab_rails['ldap_enabled'] = true; gitlab_rails['ldap_servers'] = {\"main\"=>{\"label\"=>\"LDAP\", \"host\"=>\"ldap-server.test\", \"port\"=>389, \"uid\"=>\"uid\", \"bind_dn\"=>\"cn=admin,dc=example,dc=org\", \"password\"=>\"admin\", \"encryption\"=>\"plain\", \"verify_certificates\"=>false, \"base\"=>\"dc=example,dc=org\", \"user_filter\"=>\"\", \"group_base\"=>\"ou=Global Groups,dc=example,dc=org\", \"admin_group\"=>\"AdminGroup\", \"external_groups\"=>\"\", \"sync_ssh_keys\"=>false}}; gitlab_rails['ldap_sync_worker_cron'] = '* * * * *'; gitlab_rails['ldap_group_sync_worker_cron'] = '* * * * *'; " \ + gitlab/gitlab-ee:latest + ``` 1. Run an LDAP test from [`gitlab/qa`](https://gitlab.com/gitlab-org/gitlab/-/tree/d5447ebb5f99d4c72780681ddf4dc25b0738acba/qa) directory: @@ -428,6 +428,8 @@ For instructions on how to run these tests using the `gitlab-qa` gem, please ref Tests that are tagged with `:mobile` can be run against specified mobile devices using cloud emulator/simulator services. +These tests run in the [nightly pipeline](https://gitlab.com/gitlab-org/quality/nightly/-/pipelines) in the `ce:remote_mobile_safari` job. + ### How to run mobile tests with Sauce Labs Running directly against an environment like staging is not recommended because Sauce Labs test logs expose credentials. Therefore, it is best practice and the default to use a tunnel. @@ -527,3 +529,76 @@ end ``` When running mobile tests for phone layouts, both `remote_mobile_device_name` and `mobile_layout` are `true` but when using a tablet layout, only `remote_mobile_device_name` is true. This is because phone layouts have more menus closed by default such as how both tablets and phones have the left nav closed but unlike phone layouts, tablets have the regular top navigation bar, not the mobile one. So in the case where the navigation being edited needs to be used in tablet layouts as well, use `remote_mobile_device_name` instead of `mobile_layout?` when prepending so it will use it if it's a tablet layout as well. + +## Targeting canary vs non-canary components in live environments + +Use the `QA_COOKIES` ENV variable to have the entire test target a `canary` (`staging-canary` or `canary`) or `non-canary` (`staging` or `production`) environment. + +Locally, that would mean prepending the ENV variable to your call to bin/qa. To target the `canary` version of that environment: + +```shell +QA_COOKIES="gitlab_canary=true" WEBDRIVER_HEADLESS=false bin/qa Test::Instance::Staging <YOUR SPECIFIC TAGS OR TESTS> +``` + +Alternatively, you may set the cookie to `false` to ensure the `non-canary` version is targeted. + +You can also export the cookie for your current session to avoid prepending it each time: + +```shell +export QA_COOKIES="gitlab_canary=true" +``` + +### Updating the cookie within a running spec + +Within a specific test, you can target either the `canary` or `non-canary` nodes within live environments, such as `staging` and `production`. + +For example, to switch back and forth between the two environments, you could utilize the `target_canary` method: + +```ruby +it 'tests toggling between canary and non-canary nodes' do + Runtime::Browser.visit(:gitlab, Page::Main::Login) + + # After starting the browser session, use the target_canary method ... + + Runtime::Browser::Session.target_canary(true) + Flow::Login.sign_in + + verify_session_on_canary(true) + + Runtime::Browser::Session.target_canary(false) + + # Refresh the page ... + + verify_session_on_canary(false) + + # Log out and clean up ... +end + +def verify_session_on_canary(enable_canary) + Page::Main::Menu.perform do |menu| + aggregate_failures 'testing session log in' do + expect(menu.canary?).to be(enable_canary) + end + end +end +``` + +You can verify whether GitLab is appropriately redirecting your session to the `canary` or `non-canary` nodes with the `menu.canary?` method. + +The above spec is verbose, written specifically this way to ensure the idea behind the implementation is clear. We recommend following the practices detailed within our [Beginner's guide to writing end-to-end tests](beginners_guide.md). + +## OpenID Connect (OIDC) tests + +To run the [`login_via_oidc_with_gitlab_as_idp_spec`](https://gitlab.com/gitlab-org/gitlab/-/blob/188e2c876a17a097448d7f3ed35bdf264fed0d3b/qa/qa/specs/features/browser_ui/1_manage/login/login_via_oidc_with_gitlab_as_idp_spec.rb) on your local machine: + +1. Make sure your GDK is set to run on a non-localhost address such as `gdk.test:3000`. +1. Configure a [loopback interface to 172.16.123.1](https://gitlab.com/gitlab-org/gitlab-development-kit/-/blob/6fe7b46403229f12ab6d903f99b024e0b82cb94a/doc/howto/local_network.md#create-loopback-interface). +1. Make sure Docker Desktop or Rancher Desktop is running. +1. Add an entry to your `/etc/hosts` file for `gitlab-oidc-consumer.bridge` pointing to `127.0.0.1`. +1. From the `qa` directory, run the following command. To set the GitLab image you want to use, update the `RELEASE` variable. For example, to use the latest EE image, set `RELEASE` to `gitlab/gitlab-ee:latest`: + + ```shell + bundle install + + RELEASE_REGISTRY_URL='registry.gitlab.com' RELEASE_REGISTRY_USERNAME='<your_gitlab_username>' RELEASE_REGISTRY_PASSWORD='<your_gitlab_personal_access_token>' RELEASE='registry.gitlab.com/gitlab-org/build/omnibus-gitlab-mirror/gitlab-ee:1d5a644145dfe901ea7648d825f8f9f3006d0acf' GITLAB_QA_ADMIN_ACCESS_TOKEN="<your_gdk_admin_personal_access_token>" QA_DEBUG=true CHROME_HEADLESS=false bundle exec bin/qa Test::Instance::All http://gdk.test:3000 qa/specs/features/browser_ui/1_manage/login/login_via_oidc_with_gitlab_as_idp_spec.rb + ``` |