Welcome to mirror list, hosted at ThFree Co, Russian Federation.

gitlab.com/gitlab-org/gitlab-foss.git - Unnamed repository; edit this file 'description' to name the repository.
summaryrefslogtreecommitdiff
diff options
context:
space:
mode:
authorGitLab Bot <gitlab-bot@gitlab.com>2021-10-20 11:43:02 +0300
committerGitLab Bot <gitlab-bot@gitlab.com>2021-10-20 11:43:02 +0300
commitd9ab72d6080f594d0b3cae15f14b3ef2c6c638cb (patch)
tree2341ef426af70ad1e289c38036737e04b0aa5007 /doc/development/testing_guide
parentd6e514dd13db8947884cd58fe2a9c2a063400a9b (diff)
Add latest changes from gitlab-org/gitlab@14-4-stable-eev14.4.0-rc42
Diffstat (limited to 'doc/development/testing_guide')
-rw-r--r--doc/development/testing_guide/best_practices.md2
-rw-r--r--doc/development/testing_guide/ci.md46
-rw-r--r--doc/development/testing_guide/end_to_end/best_practices.md38
-rw-r--r--doc/development/testing_guide/end_to_end/dynamic_element_validation.md2
-rw-r--r--doc/development/testing_guide/end_to_end/feature_flags.md2
-rw-r--r--doc/development/testing_guide/end_to_end/index.md2
-rw-r--r--doc/development/testing_guide/end_to_end/page_objects.md2
-rw-r--r--doc/development/testing_guide/end_to_end/rspec_metadata_tests.md3
-rw-r--r--doc/development/testing_guide/end_to_end/running_tests_that_require_special_setup.md12
-rw-r--r--doc/development/testing_guide/flaky_tests.md4
-rw-r--r--doc/development/testing_guide/frontend_testing.md23
-rw-r--r--doc/development/testing_guide/img/review-app-parent-pipeline.pngbin0 -> 136842 bytes
-rw-r--r--doc/development/testing_guide/index.md2
-rw-r--r--doc/development/testing_guide/review_apps.md174
14 files changed, 161 insertions, 151 deletions
diff --git a/doc/development/testing_guide/best_practices.md b/doc/development/testing_guide/best_practices.md
index 79664490368..52e89a10556 100644
--- a/doc/development/testing_guide/best_practices.md
+++ b/doc/development/testing_guide/best_practices.md
@@ -68,7 +68,7 @@ SILENCE_DEPRECATIONS=1 bin/rspec spec/models/project_spec.rb
### Test speed
-GitLab has a massive test suite that, without [parallelization](ci.md#test-suite-parallelization-on-the-ci), can take hours
+GitLab has a massive test suite that, without [parallelization](../pipelines.md#test-suite-parallelization), can take hours
to run. It's important that we make an effort to write tests that are accurate
and effective _as well as_ fast.
diff --git a/doc/development/testing_guide/ci.md b/doc/development/testing_guide/ci.md
index e3fccdcee34..de024084c9c 100644
--- a/doc/development/testing_guide/ci.md
+++ b/doc/development/testing_guide/ci.md
@@ -1,45 +1,9 @@
---
-stage: none
-group: unassigned
-info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments
+redirect_to: '../pipelines.md'
+remove_date: '2022-01-12'
---
-# GitLab tests in the Continuous Integration (CI) context
+This file was moved to [another location](../pipelines.md).
-## Test suite parallelization on the CI
-
-Our current CI parallelization setup is as follows:
-
-1. The `retrieve-tests-metadata` job in the `prepare` stage ensures we have a
- `knapsack/report-master.json` file:
- - The `knapsack/report-master.json` file is fetched from the latest `main` pipeline which runs `update-tests-metadata`
- (for now it's the 2-hourly scheduled master pipeline), if it's not here we initialize the file with `{}`.
-1. Each `[rspec|rspec-ee] [unit|integration|system|geo] n m` job are run with
- `knapsack rspec` and should have an evenly distributed share of tests:
- - It works because the jobs have access to the `knapsack/report-master.json`
- since the "artifacts from all previous stages are passed by default".
- - the jobs set their own report path to
- `"knapsack/${TEST_TOOL}_${TEST_LEVEL}_${DATABASE}_${CI_NODE_INDEX}_${CI_NODE_TOTAL}_report.json"`.
- - if knapsack is doing its job, test files that are run should be listed under
- `Report specs`, not under `Leftover specs`.
-1. The `update-tests-metadata` job (which only runs on scheduled pipelines for
- [the canonical project](https://gitlab.com/gitlab-org/gitlab) takes all the
- `knapsack/rspec*_pg_*.json` files and merge them all together into a single
- `knapsack/report-master.json` file that is saved as artifact.
-
-After that, the next pipeline uses the up-to-date `knapsack/report-master.json` file.
-
-## Monitoring
-
-The GitLab test suite is [monitored](../performance.md#rspec-profiling) for the `main` branch, and any branch
-that includes `rspec-profile` in their name.
-
-## CI setup
-
-- Rails logging to `log/test.log` is disabled by default in CI [for
- performance reasons](https://jtway.co/speed-up-your-rails-test-suite-by-6-in-1-line-13fedb869ec4). To override this setting, provide the
- `RAILS_ENABLE_TEST_LOG` environment variable.
-
----
-
-[Return to Testing documentation](index.md)
+<!-- This redirect file can be deleted after <2022-01-12>. -->
+<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/#move-or-rename-a-page -->
diff --git a/doc/development/testing_guide/end_to_end/best_practices.md b/doc/development/testing_guide/end_to_end/best_practices.md
index a3caa8bf2b3..9491c89c2a0 100644
--- a/doc/development/testing_guide/end_to_end/best_practices.md
+++ b/doc/development/testing_guide/end_to_end/best_practices.md
@@ -245,7 +245,11 @@ end
## Prefer `aggregate_failures` when there are back-to-back expectations
-In cases where there must be multiple (back-to-back) expectations within a test case, it is preferable to use `aggregate_failures`.
+See [Prefer aggregate failures when there are multiple expectations](#prefer-aggregate_failures-when-there-are-multiple-expectations)
+
+## Prefer `aggregate_failures` when there are multiple expectations
+
+In cases where there must be multiple expectations within a test case, it is preferable to use `aggregate_failures`.
This allows you to group a set of expectations and see all the failures altogether, rather than having the test being aborted on the first failure.
@@ -270,6 +274,32 @@ Page::Search::Results.perform do |search|
end
```
+Attach the `:aggregate_failures` metadata to the example if multiple expectations are separated by statements.
+
+```ruby
+#=> Good
+it 'searches', :aggregate_failures do
+ Page::Search::Results.perform do |search|
+ expect(search).to have_file_in_project(template[:file_name], project.name)
+
+ search.switch_to_code
+
+ expect(search).to have_file_with_content(template[:file_name], content[0..33])
+ end
+end
+
+#=> Bad
+it 'searches' do
+ Page::Search::Results.perform do |search|
+ expect(search).to have_file_in_project(template[:file_name], project.name)
+
+ search.switch_to_code
+
+ expect(search).to have_file_with_content(template[:file_name], content[0..33])
+ end
+end
+```
+
## Prefer to split tests across multiple files
Our framework includes a couple of parallelization mechanisms that work by executing spec files in parallel.
@@ -333,11 +363,11 @@ after(:all) do
end
```
-## Tag tests that require Administrator access
+## Tag tests that require the Administrator role
-We don't run tests that require Administrator access against our Production environments.
+We don't run tests that require the Administrator role against our Production environments.
-When you add a new test that requires Administrator access, apply the RSpec metadata `:requires_admin` so that the test will not be included in the test suites executed against Production and other environments on which we don't want to run those tests.
+When you add a new test that requires the Administrator role, apply the RSpec metadata `:requires_admin` so that the test will not be included in the test suites executed against Production and other environments on which we don't want to run those tests.
When running tests locally or configuring a pipeline, the environment variable `QA_CAN_TEST_ADMIN_FEATURES` can be set to `false` to skip tests that have the `:requires_admin` tag.
diff --git a/doc/development/testing_guide/end_to_end/dynamic_element_validation.md b/doc/development/testing_guide/end_to_end/dynamic_element_validation.md
index 6c504e6fa28..8770a5d33cd 100644
--- a/doc/development/testing_guide/end_to_end/dynamic_element_validation.md
+++ b/doc/development/testing_guide/end_to_end/dynamic_element_validation.md
@@ -52,7 +52,7 @@ Simply put, a required element is a visible HTML element that appears on a UI co
"Visible" can be defined as
-- Not having any CSS preventing its display. E.g.: `display: none` or `width: 0px; height: 0px;`
+- Not having any CSS preventing its display, for example, `display: none` or `width: 0px; height: 0px;`
- Being able to be interacted with by the user
"UI component" can be defined as
diff --git a/doc/development/testing_guide/end_to_end/feature_flags.md b/doc/development/testing_guide/end_to_end/feature_flags.md
index c9acb2e9371..994ee3f253c 100644
--- a/doc/development/testing_guide/end_to_end/feature_flags.md
+++ b/doc/development/testing_guide/end_to_end/feature_flags.md
@@ -15,7 +15,7 @@ token via `GITLAB_QA_ADMIN_ACCESS_TOKEN` (recommended), or provide `GITLAB_ADMIN
and `GITLAB_ADMIN_PASSWORD`.
Please be sure to include the tag `:requires_admin` so that the test can be skipped in environments
-where admin access is not available.
+where administrator access is not available.
WARNING:
You are strongly advised to [enable feature flags only for a group, project, user](../../feature_flags/index.md#feature-actors),
diff --git a/doc/development/testing_guide/end_to_end/index.md b/doc/development/testing_guide/end_to_end/index.md
index 36c0f5adf00..b097d6b0729 100644
--- a/doc/development/testing_guide/end_to_end/index.md
+++ b/doc/development/testing_guide/end_to_end/index.md
@@ -184,7 +184,7 @@ in the pipeline:
- Fetches these source files from all test jobs.
- Generates and uploads the report to the `GCS` bucket `gitlab-qa-allure-report` under the project `gitlab-qa-resources`.
-A common CI template for report uploading is stored in
+A common CI template for report uploading is stored in
[`allure-report.yml`](https://gitlab.com/gitlab-org/quality/pipeline-common/-/blob/master/ci/allure-report.yml).
#### Merge requests
diff --git a/doc/development/testing_guide/end_to_end/page_objects.md b/doc/development/testing_guide/end_to_end/page_objects.md
index 9ffa7ea4f77..85ab4d479f9 100644
--- a/doc/development/testing_guide/end_to_end/page_objects.md
+++ b/doc/development/testing_guide/end_to_end/page_objects.md
@@ -105,7 +105,7 @@ code but **this is deprecated** in favor of the above method for two reasons:
- Consistency: there is only one way to define an element
- Separation of concerns: QA uses dedicated `data-qa-*` attributes instead of reusing code
- or classes used by other components (e.g. `js-*` classes etc.)
+ or classes used by other components (for example, `js-*` classes etc.)
```ruby
view 'app/views/my/view.html.haml' do
diff --git a/doc/development/testing_guide/end_to_end/rspec_metadata_tests.md b/doc/development/testing_guide/end_to_end/rspec_metadata_tests.md
index b6e92367f89..3749511fef5 100644
--- a/doc/development/testing_guide/end_to_end/rspec_metadata_tests.md
+++ b/doc/development/testing_guide/end_to_end/rspec_metadata_tests.md
@@ -26,6 +26,7 @@ This is a partial list of the [RSpec metadata](https://relishapp.com/rspec/rspec
| `:ldap_no_tls` | The test requires a GitLab instance to be configured to use an external LDAP server with TLS not enabled. |
| `:ldap_tls` | The test requires a GitLab instance to be configured to use an external LDAP server with TLS enabled. |
| `:mattermost` | The test requires a GitLab Mattermost service on the GitLab instance. |
+| `:mixed_env` | The test should only be executed in environments that have a paired canary version available through traffic routing based on the existence of the `gitlab_canary=true` cookie. Tests in this category are switching the cookie mid-test to validate mixed deployment environments. |
| `:object_storage` | The test requires a GitLab instance to be configured to use multiple [object storage types](../../../administration/object_storage.md). Uses MinIO as the object storage server. |
| `:only` | The test is only to be run in specific execution contexts. See [test execution context selection](execution_context_selection.md) for more information. |
| `:orchestrated` | The GitLab instance under test may be [configured by `gitlab-qa`](https://gitlab.com/gitlab-org/gitlab-qa/-/blob/master/docs/what_tests_can_be_run.md#orchestrated-tests) to be different to the default GitLab configuration, or `gitlab-qa` may launch additional services in separate Docker containers, or both. Tests tagged with `:orchestrated` are excluded when testing environments where we can't dynamically modify the GitLab configuration (for example, Staging). |
@@ -34,7 +35,7 @@ This is a partial list of the [RSpec metadata](https://relishapp.com/rspec/rspec
| `:relative_url` | The test requires a GitLab instance to be installed under a [relative URL](../../../install/relative_url.md). |
| `:reliable` | The test has been [promoted to a reliable test](https://about.gitlab.com/handbook/engineering/quality/guidelines/reliable-tests/#promoting-an-existing-test-to-reliable) meaning it passes consistently in all pipelines, including merge requests. |
| `:repository_storage` | The test requires a GitLab instance to be configured to use multiple [repository storage paths](../../../administration/repository_storage_paths.md). Paired with the `:orchestrated` tag. |
-| `:requires_admin` | The test requires an admin account. Tests with the tag are excluded when run against Canary and Production environments. |
+| `:requires_admin` | The test requires an administrator account. Tests with the tag are excluded when run against Canary and Production environments. |
| `:requires_git_protocol_v2` | The test requires that Git protocol version 2 is enabled on the server. It's assumed to be enabled by default but if not the test can be skipped by setting `QA_CAN_TEST_GIT_PROTOCOL_V2` to `false`. |
| `:requires_praefect` | The test requires that the GitLab instance uses [Gitaly Cluster](../../../administration/gitaly/praefect.md) (a.k.a. Praefect) as the repository storage . It's assumed to be used by default but if not the test can be skipped by setting `QA_CAN_TEST_PRAEFECT` to `false`. |
| `:runner` | The test depends on and sets up a GitLab Runner instance, typically to run a pipeline. |
diff --git a/doc/development/testing_guide/end_to_end/running_tests_that_require_special_setup.md b/doc/development/testing_guide/end_to_end/running_tests_that_require_special_setup.md
index 46a3053c267..eadd0ef49a0 100644
--- a/doc/development/testing_guide/end_to_end/running_tests_that_require_special_setup.md
+++ b/doc/development/testing_guide/end_to_end/running_tests_that_require_special_setup.md
@@ -45,7 +45,7 @@ docker run \
Jenkins is available on `http://localhost:8080`.
-Admin username is `admin` and password is `password`.
+Administrator username is `admin` and password is `password`.
It is worth noting that this is not an orchestrated test. It is [tagged with the `:orchestrated` meta](https://gitlab.com/gitlab-org/gitlab/-/blob/163c8a8c814db26d11e104d1cb2dcf02eb567dbe/qa/qa/specs/features/ee/browser_ui/3_create/jenkins/jenkins_build_status_spec.rb#L5)
only to prevent it from running in the pipelines for live environments such as Staging.
@@ -167,9 +167,9 @@ The following includes more information on the command:
-`QA_DEBUG` - Set to `true` to verbosely log page object actions.
-`WEBDRIVER_HEADLESS` - When running locally, set to `false` to allow browser tests to be visible - watch your tests being run.
--`GITLAB_ADMIN_USERNAME` - Admin username to use when adding a license.
--`GITLAB_ADMIN_PASSWORD` - Admin password to use when adding a license.
--`GITLAB_QA_ACCESS_TOKEN` and `GITLAB_QA_ADMIN_ACCESS_TOKEN` - A valid personal access token with the `api` scope. This is used for API access during tests, and is used in the version that staging is currently running. The `ADMIN_ACCESS_TOKEN` is from a user with admin access. Used for API access as an admin during tests.
+-`GITLAB_ADMIN_USERNAME` - Administrator username to use when adding a license.
+-`GITLAB_ADMIN_PASSWORD` - Administrator password to use when adding a license.
+-`GITLAB_QA_ACCESS_TOKEN` and `GITLAB_QA_ADMIN_ACCESS_TOKEN` - A valid personal access token with the `api` scope. This is used for API access during tests, and is used in the version that staging is currently running. The `ADMIN_ACCESS_TOKEN` is from a user with administrator access. Used for API access as an administrator during tests.
-`CLUSTER_API_URL` - Use the address `https://kubernetes.docker.internal:6443` . This address is used to enable the cluster to be network accessible while deploying using Auto DevOps.
-`https://[YOUR-PORT].qa-tunnel.gitlab.info/` - The address of your local GDK
-`qa/specs/features/browser_ui/8_monitor/all_monitor_core_features_spec.rb` - The path to the monitor core specs
@@ -410,9 +410,9 @@ Tests that are tagged with `:ldap_tls` and `:ldap_no_tls` meta are orchestrated
These tests spin up a Docker container [(`osixia/openldap`)](https://hub.docker.com/r/osixia/openldap) running an instance of [OpenLDAP](https://www.openldap.org/).
The container uses fixtures [checked into the GitLab-QA repository](https://gitlab.com/gitlab-org/gitlab-qa/-/tree/9ffb9ad3be847a9054967d792d6772a74220fb42/fixtures/ldap) to create
-base data such as users and groups including the admin group. The password for [all users](https://gitlab.com/gitlab-org/gitlab-qa/-/blob/9ffb9ad3be847a9054967d792d6772a74220fb42/fixtures/ldap/2_add_users.ldif) including [the `tanuki` user](https://gitlab.com/gitlab-org/gitlab-qa/-/blob/9ffb9ad3be847a9054967d792d6772a74220fb42/fixtures/ldap/tanuki.ldif) is `password`.
+base data such as users and groups including the administrator group. The password for [all users](https://gitlab.com/gitlab-org/gitlab-qa/-/blob/9ffb9ad3be847a9054967d792d6772a74220fb42/fixtures/ldap/2_add_users.ldif) including [the `tanuki` user](https://gitlab.com/gitlab-org/gitlab-qa/-/blob/9ffb9ad3be847a9054967d792d6772a74220fb42/fixtures/ldap/tanuki.ldif) is `password`.
-A GitLab instance is also created in a Docker container based on our [General LDAP setup](../../../administration/auth/ldap/index.md#general-ldap-setup) documentation.
+A GitLab instance is also created in a Docker container based on our [LDAP setup](../../../administration/auth/ldap/index.md) documentation.
Tests that are tagged `:ldap_tls` enable TLS on GitLab using the certificate [checked into the GitLab-QA repository](https://gitlab.com/gitlab-org/gitlab-qa/-/tree/9ffb9ad3be847a9054967d792d6772a74220fb42/tls_certificates/gitlab).
diff --git a/doc/development/testing_guide/flaky_tests.md b/doc/development/testing_guide/flaky_tests.md
index bfcd68dbaf3..9489020de5d 100644
--- a/doc/development/testing_guide/flaky_tests.md
+++ b/doc/development/testing_guide/flaky_tests.md
@@ -37,9 +37,9 @@ bin/rspec --tag quarantine
Once a test is in quarantine, there are 3 choices:
-- Should the test be fixed (i.e. get rid of its flakiness)?
+- Should the test be fixed (that is, get rid of its flakiness)?
- Should the test be moved to a lower level of testing?
-- Should the test be removed entirely (e.g. because there's already a
+- Should the test be removed entirely (for example, because there's already a
lower-level test, or it's duplicating another same-level test, or it's testing
too much etc.)?
diff --git a/doc/development/testing_guide/frontend_testing.md b/doc/development/testing_guide/frontend_testing.md
index 76687db3a3f..0e721ba2760 100644
--- a/doc/development/testing_guide/frontend_testing.md
+++ b/doc/development/testing_guide/frontend_testing.md
@@ -146,7 +146,7 @@ it('does not display a dropdown if no metricTypes exist', () => {
});
```
-Keep an eye out for these kinds of tests, as they just make updating logic more fragile and tedious than it needs to be. This is also true for other libraries. A rule of thumb here is: if you are checking a `wrapper.vm` property, you should probably stop and rethink the test to check the rendered template instead.
+Keep an eye out for these kinds of tests, as they just make updating logic more fragile and tedious than it needs to be. This is also true for other libraries. A suggestion here is: if you are checking a `wrapper.vm` property, you should probably stop and rethink the test to check the rendered template instead.
Some more examples can be found in the [Frontend unit tests section](testing_levels.md#frontend-unit-tests)
@@ -783,20 +783,25 @@ often using fixtures to validate correct integration with the backend code.
### Use fixtures
-Jest uses `spec/frontend/__helpers__/fixtures.js` to import fixtures in tests.
-
-The following are examples of tests that work for Jest:
+To import a JSON fixture, `import` it using the `test_fixtures` alias.
```javascript
+import responseBody from 'test_fixtures/some/fixture.json' // loads spec/frontend/fixtures/some/fixture.json
+
it('makes a request', () => {
- const responseBody = getJSONFixture('some/fixture.json'); // loads spec/frontend/fixtures/some/fixture.json
axiosMock.onGet(endpoint).reply(200, responseBody);
myButton.click();
// ...
});
+```
+
+For other fixtures, Jest uses `spec/frontend/__helpers__/fixtures.js` to import them in tests.
+The following are examples of tests that work for Jest:
+
+```javascript
it('uses some HTML element', () => {
loadFixtures('some/page.html'); // loads spec/frontend/fixtures/some/page.html and adds it to the DOM
@@ -843,10 +848,6 @@ describe GraphQL::Query, type: :request do
all_releases_query_path = 'releases/graphql/queries/all_releases.query.graphql'
- before(:all) do
- clean_frontend_fixtures('graphql/releases/')
- end
-
it "graphql/#{all_releases_query_path}.json" do
query = get_graphql_query_as_string(all_releases_query_path)
@@ -860,7 +861,7 @@ end
This will create a new fixture located at
`tmp/tests/frontend/fixtures-ee/graphql/releases/graphql/queries/all_releases.query.graphql.json`.
-You can import the JSON fixture in a Jest test using the `getJSONFixture` method
+You can import the JSON fixture in a Jest test using the `test_fixtures` alias
[as described below](#use-fixtures).
## Data-driven tests
@@ -998,7 +999,7 @@ it like so:
import Subject from '~/feature/the_subject.vue';
// Force Jest to transpile and cache
-// eslint-disable-next-line import/order, no-unused-vars
+// eslint-disable-next-line no-unused-vars
import _Thing from '~/feature/path/to/thing.vue';
```
diff --git a/doc/development/testing_guide/img/review-app-parent-pipeline.png b/doc/development/testing_guide/img/review-app-parent-pipeline.png
new file mode 100644
index 00000000000..5686d5f6ebe
--- /dev/null
+++ b/doc/development/testing_guide/img/review-app-parent-pipeline.png
Binary files differ
diff --git a/doc/development/testing_guide/index.md b/doc/development/testing_guide/index.md
index 015d8a92a4d..2e00a00c454 100644
--- a/doc/development/testing_guide/index.md
+++ b/doc/development/testing_guide/index.md
@@ -48,7 +48,7 @@ testing promises, stubbing etc.
What are flaky tests, the different kind of flaky tests we encountered, and what
we do about them.
-## [GitLab tests in the Continuous Integration (CI) context](ci.md)
+## [GitLab pipelines](../pipelines.md)
How GitLab test suite is run in the CI context: setup, caches, artifacts,
parallelization, monitoring.
diff --git a/doc/development/testing_guide/review_apps.md b/doc/development/testing_guide/review_apps.md
index 72d63fd8194..4091f213a8f 100644
--- a/doc/development/testing_guide/review_apps.md
+++ b/doc/development/testing_guide/review_apps.md
@@ -6,8 +6,98 @@ info: To determine the technical writer assigned to the Stage/Group associated w
# Review Apps
-Review Apps are automatically deployed by [the
-pipeline](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/6665).
+Review Apps are deployed using the `start-review-app-pipeline` job. This job triggers a child pipeline containing a series of jobs to perform the various tasks needed to deploy a Review App.
+
+![start-review-app-pipeline job](img/review-app-parent-pipeline.png)
+
+For any of the following scenarios, the `start-review-app-pipeline` job would be automatically started:
+
+- for merge requests with CI config changes
+- for merge requests with frontend changes
+- for merge requests with QA changes
+- for scheduled pipelines
+
+## QA runs on Review Apps
+
+On every [pipeline](https://gitlab.com/gitlab-org/gitlab/pipelines/125315730) in the `qa` stage (which comes after the
+`review` stage), the `review-qa-smoke` job is automatically started and it runs
+the QA smoke suite.
+
+You can also manually start the `review-qa-all`: it runs the full QA suite.
+
+After the end-to-end test runs have finished, [Allure reports](https://github.com/allure-framework/allure2) are generated and published by
+the `allure-report-qa-smoke` and `allure-report-qa-all` jobs. A comment with links to the reports are added to the merge request.
+
+## Performance Metrics
+
+On every [pipeline](https://gitlab.com/gitlab-org/gitlab/pipelines/125315730) in the `qa` stage, the
+`review-performance` job is automatically started: this job does basic
+browser performance testing using a
+[Sitespeed.io Container](../../user/project/merge_requests/browser_performance_testing.md).
+
+## How to
+
+### Get access to the GCP Review Apps cluster
+
+You need to [open an access request (internal link)](https://gitlab.com/gitlab-com/access-requests/-/issues/new)
+for the `gcp-review-apps-dev` GCP group and role.
+
+This grants you the following permissions for:
+
+- [Retrieving pod logs](#dig-into-a-pods-logs). Granted by [Viewer (`roles/viewer`)](https://cloud.google.com/iam/docs/understanding-roles#kubernetes-engine-roles).
+- [Running a Rails console](#run-a-rails-console). Granted by [Kubernetes Engine Developer (`roles/container.pods.exec`)](https://cloud.google.com/iam/docs/understanding-roles#kubernetes-engine-roles).
+
+### Log into my Review App
+
+For GitLab Team Members only. If you want to sign in to the review app, review
+the GitLab handbook information for the [shared 1Password account](https://about.gitlab.com/handbook/security/#1password-for-teams).
+
+- The default username is `root`.
+- The password can be found in the 1Password login item named `GitLab EE Review App`.
+
+### Enable a feature flag for my Review App
+
+1. Open your Review App and log in as documented above.
+1. Create a personal access token.
+1. Enable the feature flag using the [Feature flag API](../../api/features.md).
+
+### Find my Review App slug
+
+1. Open the `review-deploy` job.
+1. Look for `** Deploying review-*`.
+1. For instance for `** Deploying review-1234-abc-defg... **`,
+ your Review App slug would be `review-1234-abc-defg` in this case.
+
+### Run a Rails console
+
+1. Make sure you [have access to the cluster](#get-access-to-the-gcp-review-apps-cluster) and the `container.pods.exec` permission first.
+1. [Filter Workloads by your Review App slug](https://console.cloud.google.com/kubernetes/workload?project=gitlab-review-apps),
+ e.g. `review-qa-raise-e-12chm0`.
+1. Find and open the `task-runner` Deployment, e.g. `review-qa-raise-e-12chm0-task-runner`.
+1. Click on the Pod in the "Managed pods" section, e.g. `review-qa-raise-e-12chm0-task-runner-d5455cc8-2lsvz`.
+1. Click on the `KUBECTL` dropdown, then `Exec` -> `task-runner`.
+1. Replace `-c task-runner -- ls` with `-it -- gitlab-rails console` from the
+ default command or
+ - Run `kubectl exec --namespace review-qa-raise-e-12chm0 review-qa-raise-e-12chm0-task-runner-d5455cc8-2lsvz -it -- gitlab-rails console` and
+ - Replace `review-qa-raise-e-12chm0-task-runner-d5455cc8-2lsvz`
+ with your Pod's name.
+
+### Dig into a Pod's logs
+
+1. Make sure you [have access to the cluster](#get-access-to-the-gcp-review-apps-cluster) and the `container.pods.getLogs` permission first.
+1. [Filter Workloads by your Review App slug](https://console.cloud.google.com/kubernetes/workload?project=gitlab-review-apps),
+ e.g. `review-qa-raise-e-12chm0`.
+1. Find and open the `migrations` Deployment, e.g.
+ `review-qa-raise-e-12chm0-migrations.1`.
+1. Click on the Pod in the "Managed pods" section, e.g.
+ `review-qa-raise-e-12chm0-migrations.1-nqwtx`.
+1. Click on the `Container logs` link.
+
+Alternatively, you could use the [Logs Explorer](https://console.cloud.google.com/logs/query;query=?project=gitlab-review-apps) which provides more utility to search logs. An example query for a pod name is as follows:
+
+```shell
+resource.labels.pod_name:"review-qa-raise-e-12chm0-migrations"
+```
## How does it work?
@@ -89,7 +179,7 @@ subgraph "CNG-mirror pipeline"
**Additional notes:**
-- If the `review-deploy` job keep failing (note that we already retry it twice),
+- If the `review-deploy` job keeps failing (and a manual retry didn't help),
please post a message in the `#g_qe_engineering_productivity` channel and/or create a `~"Engineering Productivity"` `~"ep::review apps"` `~bug`
issue with a link to your merge request. Note that the deployment failure can
reveal an actual problem introduced in your merge request (i.e. this isn't
@@ -105,7 +195,7 @@ subgraph "CNG-mirror pipeline"
stop a Review App manually, and is also started by GitLab once a merge
request's branch is deleted after being merged.
- The Kubernetes cluster is connected to the `gitlab` projects using the
- [GitLab Kubernetes integration](../../user/project/clusters/index.md). This basically
+ [GitLab Kubernetes integration](../../user/infrastructure/clusters/index.md). This basically
allows to have a link to the Review App directly from the merge request widget.
### Auto-stopping of Review Apps
@@ -126,24 +216,6 @@ The `review-gcp-cleanup` job that automatically runs in scheduled pipelines
(and is manual in merge request) removes any dangling GCP network resources
that were not removed along with the Kubernetes resources.
-## QA runs
-
-On every [pipeline](https://gitlab.com/gitlab-org/gitlab/pipelines/125315730) in the `qa` stage (which comes after the
-`review` stage), the `review-qa-smoke` job is automatically started and it runs
-the QA smoke suite.
-
-You can also manually start the `review-qa-all`: it runs the full QA suite.
-
-After the end-to-end test runs have finished, [Allure reports](https://github.com/allure-framework/allure2) are generated and published by
-the `allure-report-qa-smoke` and `allure-report-qa-all` jobs. A comment with links to the reports are added to the merge request.
-
-## Performance Metrics
-
-On every [pipeline](https://gitlab.com/gitlab-org/gitlab/pipelines/125315730) in the `qa` stage, the
-`review-performance` job is automatically started: this job does basic
-browser performance testing using a
-[Sitespeed.io Container](../../user/project/merge_requests/browser_performance_testing.md).
-
## Cluster configuration
The cluster is configured via Terraform in the [`engineering-productivity-infrastructure`](https://gitlab.com/gitlab-org/quality/engineering-productivity-infrastructure) project.
@@ -157,64 +229,6 @@ The Helm version used is defined in the
[`registry.gitlab.com/gitlab-org/gitlab-build-images:gitlab-helm3-kubectl1.14` image](https://gitlab.com/gitlab-org/gitlab-build-images/-/blob/master/Dockerfile.gitlab-helm3-kubectl1.14#L7)
used by the `review-deploy` and `review-stop` jobs.
-## How to
-
-### Get access to the GCP Review Apps cluster
-
-You need to [open an access request (internal link)](https://gitlab.com/gitlab-com/access-requests/-/issues/new)
-for the `gcp-review-apps-dev` GCP group and role.
-
-This grants you the following permissions for:
-
-- [Retrieving pod logs](#dig-into-a-pods-logs). Granted by [Viewer (`roles/viewer`)](https://cloud.google.com/iam/docs/understanding-roles#kubernetes-engine-roles).
-- [Running a Rails console](#run-a-rails-console). Granted by [Kubernetes Engine Developer (`roles/container.pods.exec`)](https://cloud.google.com/iam/docs/understanding-roles#kubernetes-engine-roles).
-
-### Log into my Review App
-
-For GitLab Team Members only. If you want to sign in to the review app, review
-the GitLab handbook information for the [shared 1Password account](https://about.gitlab.com/handbook/security/#1password-for-teams).
-
-- The default username is `root`.
-- The password can be found in the 1Password login item named `GitLab EE Review App`.
-
-### Enable a feature flag for my Review App
-
-1. Open your Review App and log in as documented above.
-1. Create a personal access token.
-1. Enable the feature flag using the [Feature flag API](../../api/features.md).
-
-### Find my Review App slug
-
-1. Open the `review-deploy` job.
-1. Look for `** Deploying review-*`.
-1. For instance for `** Deploying review-1234-abc-defg... **`,
- your Review App slug would be `review-1234-abc-defg` in this case.
-
-### Run a Rails console
-
-1. Make sure you [have access to the cluster](#get-access-to-the-gcp-review-apps-cluster) and the `container.pods.exec` permission first.
-1. [Filter Workloads by your Review App slug](https://console.cloud.google.com/kubernetes/workload?project=gitlab-review-apps),
- e.g. `review-qa-raise-e-12chm0`.
-1. Find and open the `task-runner` Deployment, e.g. `review-qa-raise-e-12chm0-task-runner`.
-1. Click on the Pod in the "Managed pods" section, e.g. `review-qa-raise-e-12chm0-task-runner-d5455cc8-2lsvz`.
-1. Click on the `KUBECTL` dropdown, then `Exec` -> `task-runner`.
-1. Replace `-c task-runner -- ls` with `-it -- gitlab-rails console` from the
- default command or
- - Run `kubectl exec --namespace review-qa-raise-e-12chm0 review-qa-raise-e-12chm0-task-runner-d5455cc8-2lsvz -it -- gitlab-rails console` and
- - Replace `review-qa-raise-e-12chm0-task-runner-d5455cc8-2lsvz`
- with your Pod's name.
-
-### Dig into a Pod's logs
-
-1. Make sure you [have access to the cluster](#get-access-to-the-gcp-review-apps-cluster) and the `container.pods.getLogs` permission first.
-1. [Filter Workloads by your Review App slug](https://console.cloud.google.com/kubernetes/workload?project=gitlab-review-apps),
- e.g. `review-qa-raise-e-12chm0`.
-1. Find and open the `migrations` Deployment, e.g.
- `review-qa-raise-e-12chm0-migrations.1`.
-1. Click on the Pod in the "Managed pods" section, e.g.
- `review-qa-raise-e-12chm0-migrations.1-nqwtx`.
-1. Click on the `Container logs` link.
-
## Diagnosing unhealthy Review App releases
If [Review App Stability](https://app.periscopedata.com/app/gitlab/496118/Engineering-Productivity-Sandbox?widget=6690556&udv=785399)