diff options
Diffstat (limited to 'qa/README.md')
| -rw-r--r-- | qa/README.md | 144 |
1 files changed, 81 insertions, 63 deletions
diff --git a/qa/README.md b/qa/README.md index 724638d13c0..dbc70f55f1c 100644 --- a/qa/README.md +++ b/qa/README.md @@ -39,78 +39,73 @@ have an instance available you can follow the instructions below to use the [GitLab Development Kit (GDK)](https://gitlab.com/gitlab-org/gitlab-development-kit). This is the recommended option if you would like to contribute to the tests. -Note: GitLab QA uses [Selenium WebDriver](https://www.seleniumhq.org/) via -[Cabybara](http://teamcapybara.github.io/capybara/), and by default it targets Chrome as -the browser to use. You will need to have Chrome (or Chromium) and -[chromedriver](https://chromedriver.chromium.org/) installed / in your `$PATH`. +Note that tests are using `Chrome` web browser by default so it should be installed and present in `PATH`. + +## CI + +Tests are executed in merge request pipelines as part of the development lifecycle. + +- [Review app environment](../doc/development/testing_guide/review_apps.md) +- [package-and-qa](../doc/development/testing_guide/end_to_end/index.md#testing-code-in-merge-requests) + +### Logging + +By default tests on CI use `info` log level. `debug` level is still available in case of failure debugging. Logs are stored in jobs artifacts. ### Writing tests - [Writing tests from scratch tutorial](../doc/development/testing_guide/end_to_end/beginners_guide.md) - - [Best practices](../doc/development/testing_guide/best_practices.md) - - [Using page objects](../doc/development/testing_guide/end_to_end/page_objects.md) - - [Guidelines](../doc/development/testing_guide/index.md) - - [Tests with special setup for local environments](../doc/development/testing_guide/end_to_end/running_tests_that_require_special_setup.md) + - [Best practices](../doc/development/testing_guide/best_practices.md) + - [Using page objects](../doc/development/testing_guide/end_to_end/page_objects.md) + - [Guidelines](../doc/development/testing_guide/index.md) + - [Tests with special setup for local environments](../doc/development/testing_guide/end_to_end/running_tests_that_require_special_setup.md) ### Run the end-to-end tests in a local development environment -Follow the GDK instructions to [install](https://gitlab.com/gitlab-org/gitlab-development-kit/blob/main/doc/index.md) your local GitLab development environment. - -Once you have GDK running, switch to the `qa` directory. E.g., if you setup -GDK to develop in the main `gitlab-ce` repo, the GitLab source code will be -in a `gitlab` directory and so the end-to-end test code will be in `gitlab/qa`. +1. Follow the instructions to [install GDK](https://gitlab.com/gitlab-org/gitlab-development-kit/blob/main/doc/index.md), your local GitLab development environment. -From there you can run the tests. For example, the -following call would login to the GDK instance and run all specs in -`qa/specs/features`: +1. Navigate to the QA folder and run the following commands. +```bash +cd gitlab-development-kit/gitlab/qa +bundle install +export WEBDRIVER_HEADLESS=false +export GITLAB_INITIAL_ROOT_PASSWORD={your current root user's password} ``` -# Make sure to install the dependencies first with `bundle install` -bundle exec bin/qa Test::Instance::All http://localhost:3000 -``` +1. Most tests that do not require special setup could simply be run with the following command. However, tests that are tagged with `:orchestrated` tag require special setup. These tests can only be run with [bin/qa](https://gitlab.com/gitlab-org/gitlab/-/blob/master/qa/README.md#running-tests-with-a-custom-binqa-test-runner) script. -Note: If you want to run tests requiring SSH against GDK, you -will need to [modify your GDK setup](https://gitlab.com/gitlab-org/gitlab-qa/blob/master/docs/run_qa_against_gdk.md). +```bash +bundle exec rspec <path/to/spec.rb> +``` -Note: When you log into your GDK instance of GitLab for the first time, the root password requires a change. -GitLab QA expects the default initial password to be used in tests; see all default values listed in -[Supported GitLab environment variables](https://gitlab.com/gitlab-org/gitlab-qa/-/blob/master/docs/what_tests_can_be_run.md#supported-gitlab-environment-variables). -If you have changed your root password, you must set the `GITLAB_INITIAL_ROOT_PASSWORD` environment -variable. +1. For test that are tagged with `:orchestrated`, [re-configure IP address in GDK](https://gitlab.com/gitlab-org/gitlab-qa/-/blob/master/docs/run_qa_against_gdk.md#run-qa-tests-against-your-gdk-setup) to run QA tests. Once you have reconfigured GDK, ensure GitLab is running successfully on the IP address configured, then run the following command: -``` -export GITLAB_INITIAL_ROOT_PASSWORD="<GDK root password>" +```bash +bundle exec bin/qa Test::Instance::All {GDK IP ADDRESS} ``` +- Note: If you want to run tests requiring SSH against GDK, you will need to [modify your GDK setup](https://gitlab.com/gitlab-org/gitlab-qa/blob/master/docs/run_qa_against_gdk.md). +- Note: If this is your first time running GDK, you can use the password pre-set for `root`. [See supported GitLab environment variables](https://gitlab.com/gitlab-org/gitlab-qa/-/blob/master/docs/what_tests_can_be_run.md#supported-gitlab-environment-variables). If you have changed your `root` password, use that when exporting `GITLAB_INITIAL_ROOT_PASSWORD`. + #### Running EE tests When running EE tests you'll need to have a license available. GitLab engineers can [request a license](https://about.gitlab.com/handbook/developer-onboarding/#working-on-gitlab-ee). Once you have the license file you can export it as an environment variable and then the framework can use it. If you do so it will be installed automatically. -``` +```shell export EE_LICENSE=$(cat /path/to/gitlab_license) ``` -### Running specific tests +#### Running specific tests You can also supply specific tests to run as another parameter. For example, to run the repository-related specs, you can execute: +```shell +bundle exec rspec qa/specs/features/browser_ui/3_create/repository ``` -bundle exec bin/qa Test::Instance::All http://localhost:3000 -- qa/specs/features/browser_ui/3_create/repository -``` - -Since the arguments would be passed to `rspec`, you could use all `rspec` -options there. For example, passing `--backtrace` and also line number: - -``` -bundle exec bin/qa Test::Instance::All http://localhost:3000 -- qa/specs/features/browser_ui/3_create/merge_request/create_merge_request_spec.rb:6 --backtrace -``` - -Note that the separator `--` is required; all subsequent options will be -ignored by the QA framework and passed to `rspec`. #### Running tests for transient bugs @@ -118,10 +113,18 @@ A suite of tests have been written to test for [transient bugs](https://about.gi Those tests are tagged `:transient` and therefore can be run via: ```shell -bundle exec bin/qa Test::Instance::All http://localhost:3000 -- --tag transient +bundle exec rspec --tag transient ``` -### Overriding the authenticated user +#### Overriding gitlab address + +When running tests against GDK, the default address is `http://127.0.0.1:3000`. This value can be overridden by providing environment variable `QA_GITLAB_URL`: + +```shell +QA_GITLAB_URL=https://gdk.test:3000 bundle exec rspec +``` + +#### Overriding the authenticated user Unless told otherwise, the QA tests will run as the default `root` user seeded by the GDK. @@ -129,8 +132,8 @@ by the GDK. If you need to authenticate as a different user, you can provide the `GITLAB_USERNAME` and `GITLAB_PASSWORD` environment variables: -``` -GITLAB_USERNAME=jsmith GITLAB_PASSWORD=password bundle exec bin/qa Test::Instance::All https://gitlab.example.com +```shell +GITLAB_USERNAME=jsmith GITLAB_PASSWORD=password bundle exec rspec ``` Some QA tests require logging in as an admin user. By default, the QA @@ -140,21 +143,21 @@ If you need to authenticate with different admin credentials, you can provide the `GITLAB_ADMIN_USERNAME` and `GITLAB_ADMIN_PASSWORD` environment variables: -``` -GITLAB_ADMIN_USERNAME=admin GITLAB_ADMIN_PASSWORD=myadminpassword GITLAB_USERNAME=jsmith GITLAB_PASSWORD=password bundle exec bin/qa Test::Instance::All https://gitlab.example.com +```shell +GITLAB_ADMIN_USERNAME=admin GITLAB_ADMIN_PASSWORD=myadminpassword GITLAB_USERNAME=jsmith GITLAB_PASSWORD=password bundle exec rspec ``` If your user doesn't have permission to default sandbox group `gitlab-qa-sandbox`, you could also use another sandbox group by giving `GITLAB_SANDBOX_NAME`: -``` -GITLAB_USERNAME=jsmith GITLAB_PASSWORD=password GITLAB_SANDBOX_NAME=jsmith-qa-sandbox bundle exec bin/qa Test::Instance::All https://gitlab.example.com +```shell +GITLAB_USERNAME=jsmith GITLAB_PASSWORD=password GITLAB_SANDBOX_NAME=jsmith-qa-sandbox bundle exec rspec ``` All [supported environment variables are here](https://gitlab.com/gitlab-org/gitlab-qa/blob/master/docs/what_tests_can_be_run.md#supported-environment-variables). -### Sending additional cookies +#### Sending additional cookies The environment variable `QA_COOKIES` can be set to send additional cookies on every request. This is necessary on gitlab.com to direct traffic to the @@ -162,6 +165,21 @@ canary fleet. To do this set `QA_COOKIES="gitlab_canary=true"`. To set multiple cookies, separate them with the `;` character, for example: `QA_COOKIES="cookie1=value;cookie2=value2"` +#### Headless browser + +By default tests use headless browser. To override that, `WEBDRIVER_HEADLESS` must be set to `false`: + +```shell +WEBDRIVER_HEADLESS=false bundle exec rspec +``` + +#### Log level + +By default, the tests use the `info` log level. To change the test's log level, the environment variable `QA_LOG_LEVEL` can be set: + +```shell +QA_LOG_LEVEL=debug bundle exec rspec +``` ### Building a Docker image to test @@ -182,20 +200,19 @@ tests that are expected to fail while a fix is in progress (similar to how [`skip` or `pending`](https://relishapp.com/rspec/rspec-core/v/3-8/docs/pending-and-skipped-examples) can be used). -``` -bundle exec bin/qa Test::Instance::All http://localhost:3000 -- --tag quarantine +```shell +bundle exec rspec --tag quarantine ``` -If `quarantine` is used with other tags, tests will only be run if they have at -least one of the tags other than `quarantine`. This is different from how RSpec -tags usually work, where all tags are inclusive. +### Running tests with a custom bin/qa test runner -For example, suppose one test has `:smoke` and `:quarantine` metadata, and -another test has `:ldap` and `:quarantine` metadata. If the tests are run with -`--tag smoke --tag quarantine`, only the first test will run. The test with -`:ldap` will not run even though it also has `:quarantine`. +`bin/qa` is an additional custom wrapper script that abstracts away some of the more complicated setups that some tests require. This option requires test scenario and test instance's Gitlab address to be specified in the command. For example, to run any `Instance` scenario test, the following command can be used: + +```shell +bundle exec bin/qa Test::Instance::All http://localhost:3000 +``` -### Running tests with a feature flag enabled or disabled +#### Running tests with a feature flag enabled or disabled Tests can be run with a feature flag enabled or disabled by using the command-line option `--enable-feature FEATURE_FLAG` or `--disable-feature FEATURE_FLAG`. @@ -203,7 +220,7 @@ option `--enable-feature FEATURE_FLAG` or `--disable-feature FEATURE_FLAG`. For example, to enable the feature flag that enforces Gitaly request limits, you would use the command: -``` +```shell bundle exec bin/qa Test::Instance::All http://localhost:3000 --enable-feature gitaly_enforce_requests_limits ``` @@ -215,9 +232,10 @@ feature flag again. Similarly, to disable the feature flag that enforces Gitaly request limits, you would use the command: -``` +```shell bundle exec bin/qa Test::Instance::All http://localhost:3000 --disable-feature gitaly_enforce_requests_limits ``` + This will instruct the QA framework to disable the `gitaly_enforce_requests_limits` feature flag ([via the API](https://docs.gitlab.com/ee/api/features.html)) if not already disabled, run all the tests in the `Test::Instance::All` scenario, and then enable the |