Add latest changes from gitlab-org/gitlab@14-8-stable-eev14.8.0-rc42

10 files changed, 142 insertions, 27 deletions

diff --git a/doc/development/testing_guide/end_to_end/best_practices.md b/doc/development/testing_guide/end_to_end/best_practices.md
index 543feaa967c..405ff40ef6a 100644
--- a/doc/development/testing_guide/end_to_end/best_practices.md
+++ b/doc/development/testing_guide/end_to_end/best_practices.md
@@ -363,11 +363,11 @@ after(:all) do
 end
 ```
 
-## Tag tests that require the Administrator role
+## Tag tests that require administrator access
 
-We don't run tests that require the Administrator role against our Production environments.
+We don't run tests that require administrator access against our Production environments.
 
-When you add a new test that requires the Administrator role, apply the RSpec metadata `:requires_admin` so that the test will not be included in the test suites executed against Production and other environments on which we don't want to run those tests.
+When you add a new test that requires administrator access, apply the RSpec metadata `:requires_admin` so that the test will not be included in the test suites executed against Production and other environments on which we don't want to run those tests.
 
 When running tests locally or configuring a pipeline, the environment variable `QA_CAN_TEST_ADMIN_FEATURES` can be set to `false` to skip tests that have the `:requires_admin` tag.
 
diff --git a/doc/development/testing_guide/end_to_end/execution_context_selection.md b/doc/development/testing_guide/end_to_end/execution_context_selection.md
index 8456cfa8314..0fdcf0c8c3b 100644
--- a/doc/development/testing_guide/end_to_end/execution_context_selection.md
+++ b/doc/development/testing_guide/end_to_end/execution_context_selection.md
@@ -120,3 +120,5 @@ Similarly to specifying that a test should only run against a specific environme
 test only when it runs against a specific environment. The syntax is exactly the same, except that the `only: { ... }`
 hash is nested in the [`quarantine: { ... }`](https://about.gitlab.com/handbook/engineering/quality/guidelines/debugging-qa-test-failures/#quarantining-tests) hash.
 For example, `quarantine: { only: { subdomain: :staging } }` only quarantines the test when run against `staging`.
+
+The quarantine feature can be explicitly disabled with the `DISABLE_QUARANTINE` environment variable. This can be useful when running tests locally.
diff --git a/doc/development/testing_guide/end_to_end/feature_flags.md b/doc/development/testing_guide/end_to_end/feature_flags.md
index 48157a961e1..238b1ccd8f9 100644
--- a/doc/development/testing_guide/end_to_end/feature_flags.md
+++ b/doc/development/testing_guide/end_to_end/feature_flags.md
@@ -144,6 +144,14 @@ end
 After the feature flag is removed, clean up the resource class and delete the variable.
 All methods should use the condition procedures of the now-default state.
 
+## Managing flakiness due to caching
+
+All application settings, and all feature flags, are cached inside GitLab for one minute.
+All caching is disabled during testing, except on static environments.
+
+When a test changes a feature flag, it can cause flaky behavior if elements are visible only with an
+active feature flag. To circumvent this behavior, add a wait for elements behind a feature flag.
+
 ## Running a scenario with a feature flag enabled
 
 It's also possible to run an entire scenario with a feature flag enabled, without having to edit
diff --git a/doc/development/testing_guide/end_to_end/index.md b/doc/development/testing_guide/end_to_end/index.md
index 1fc9bc8258a..dc989acbdcc 100644
--- a/doc/development/testing_guide/end_to_end/index.md
+++ b/doc/development/testing_guide/end_to_end/index.md
@@ -101,19 +101,19 @@ This is due to technical limitations in the GitLab permission model: the ability
 against a protected branch is controlled by the ability to push/merge to this branch.
 This means that for developers to be able to trigger a pipeline for the default branch in
 `gitlab-org/omnibus-gitlab`/`gitlab-org/gitlab-qa`, they would need to have the
-[Maintainer role](../../../user/permissions.md) for those projects.
+Maintainer role for those projects.
 For security reasons and implications, we couldn't open up the default branch to all the Developers.
 Hence we created these mirrors where Developers and Maintainers are allowed to push/merge to the default branch.
 This problem was discovered in <https://gitlab.com/gitlab-org/gitlab-qa/-/issues/63#note_107175160> and the "mirror"
 work-around was suggested in <https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/4717>.
 A feature proposal to segregate access control regarding running pipelines from ability to push/merge was also created at <https://gitlab.com/gitlab-org/gitlab/-/issues/24585>.
 
-#### With Pipeline for Merged Results
+#### With merged results pipelines
 
-In a Pipeline for Merged Results, the pipeline runs on a new ref that contains the merge result of the source and target branch.
+In a merged results pipeline, the pipeline runs on a new ref that contains the merge result of the source and target branch.
 However, this ref is not available to the `gitlab-qa-mirror` pipeline.
 
-For this reason, the end-to-end tests on a Pipeline for Merged Results would use the head of the merge request source branch.
+For this reason, the end-to-end tests on a merged results pipeline would use the head of the merge request source branch.
 
 ```mermaid
 graph LR
@@ -126,7 +126,7 @@ A --> C
 B --> C
 
 A --> E["E2E tests"]
-C --> D["Pipeline for merged results"]
+C --> D["Merged results pipeline"]
  ```
 
 ##### Running custom tests
diff --git a/doc/development/testing_guide/end_to_end/resources.md b/doc/development/testing_guide/end_to_end/resources.md
index abba1d51f07..e2b005d8a1b 100644
--- a/doc/development/testing_guide/end_to_end/resources.md
+++ b/doc/development/testing_guide/end_to_end/resources.md
@@ -431,11 +431,32 @@ module QA
       def validate_reuse_preconditions
         raise ResourceReuseError unless reused_name_valid?
       end
+
+      # Internally we identify an instance of a reusable resource by a unique value of `@reuse_as`, but in GitLab the
+      # resource has one or more attributes that must also be unique. This method lists those attributes and allows the
+      # test framework to check that each instance of a reusable resource has values that match the associated values
+      # in Gitlab.
+      def unique_identifiers
+        [:name, :path]
+      end
     end
   end
 end
 ```
 
+Reusable resources aren't removed immediately when `remove_via_api!` is called. Instead, they're removed after the test
+suite completes. To do so each class must be registered with `QA::Resource::ReusableCollection` in `qa/spec/spec_helper.rb`
+as in the example below. Registration allows `QA::Resource::ReusableCollection` to keep track of each instance of each
+registered class, and to delete them all in the `config.after(:suite)` hook.
+
+```ruby
+config.before(:suite) do |suite|
+  QA::Resource::ReusableCollection.register_resource_classes do |collection|
+    QA::Resource::ReusableProject.register(collection)
+  end
+end
+```
+
 Consider some examples of how a reusable resource is used:
 
 ```ruby
@@ -488,6 +509,65 @@ end
 # match the name specified when the project was first fabricated.
 ```
 
+### Validating reusable resources
+
+Reusable resources can speed up test suites by avoiding the cost of creating the same resource again and again. However,
+that can cause problems if a test makes changes to a resource that prevent it from being reused as expected by later
+tests. That can lead to order-dependent test failures that can be difficult to troubleshoot.
+
+For example, the default project created by `QA::Resource::ReusableProject` has `auto_devops_enabled` set to `false`
+(inherited from `QA::Resource::Project`). If a test reuses that project and enables Auto DevOps, subsequent tests that reuse
+the project will fail if they expect Auto DevOps to be disabled.
+
+We try to avoid that kind of trouble by validating reusable resources after a test suite. If the environment variable
+`QA_VALIDATE_RESOURCE_REUSE` is set to `true` the test framework will check each reusable resource to verify that none
+of the attributes they were created with have been changed. It does that by creating a new resource using the same
+attributes that were used to create the original resource. It then compares the new resource to the original and raises
+an error if any attributes don't match.
+
+#### Implementation
+
+When you implement a new type of reusable resource there are two `private` methods you must implement so the resource
+can be validated. They are:
+
+- `reference_resource`: creates a new instance of the resource that can be compared with the one that was used during the tests.
+- `unique_identifiers`: returns an array of attributes that allow the resource to be identified (e.g., name) and that are therefore
+expected to differ when comparing the reference resource with the resource reused in the tests.
+
+The following example shows the implementation of those two methods in `QA::Resource::ReusableProject`.
+
+```ruby
+# Creates a new project that can be compared to a reused project, using the attributes of the original.
+#
+# @return [QA::Resource] a new instance of Resource::ReusableProject that should be a copy of the original resource
+def reference_resource
+  # These are the attributes that the reused resource was created with
+  attributes = self.class.resources[reuse_as][:attributes]
+
+  # Two projects can't have the same path, and since we typically use the same value for the name and path, we assign
+  # a unique name and path to the reference resource.
+  name = "reference_resource_#{SecureRandom.hex(8)}_for_#{attributes.delete(:name)}"
+
+  Project.fabricate_via_api! do |project|
+    self.class.resources[reuse_as][:attributes].each do |attribute_name, attribute_value|
+      project.instance_variable_set("@#{attribute_name}", attribute_value) if attribute_value
+    end
+    project.name = name
+    project.path = name
+    project.path_with_namespace = "#{project.group.full_path}/#{project.name}"
+  end
+end
+
+# The attributes of the resource that should be the same whenever a test wants to reuse a project.
+#
+# @return [Array<Symbol>] the attribute names.
+def unique_identifiers
+  # As noted above, path must be unique, and since we typically use the same value for both, we treat name and path
+  # as unique. These attributes are ignored when we compare the reference and reused resources.
+  [:name, :path]
+end
+```
+
 ## Where to ask for help?
 
 If you need more information, ask for help on `#quality` channel on Slack
diff --git a/doc/development/testing_guide/end_to_end/rspec_metadata_tests.md b/doc/development/testing_guide/end_to_end/rspec_metadata_tests.md
index bb214976926..07f7e9582ee 100644
--- a/doc/development/testing_guide/end_to_end/rspec_metadata_tests.md
+++ b/doc/development/testing_guide/end_to_end/rspec_metadata_tests.md
@@ -20,6 +20,7 @@ This is a partial list of the [RSpec metadata](https://relishapp.com/rspec/rspec
 | `:github`         | The test requires a GitHub personal access token. |
 | `:group_saml`     | The test requires a GitLab instance that has SAML SSO enabled at the group level. Interacts with an external SAML identity provider. Paired with the `:orchestrated` tag. |
 | `:instance_saml`  | The test requires a GitLab instance that has SAML SSO enabled at the instance level. Interacts with an external SAML identity provider. Paired with the `:orchestrated` tag. |
+| `:integrations`   | This aims to test the available [integrations](../../../user/project/integrations/overview.md#integrations-listing). The test requires Docker to be installed in the run context. It will provision the containers and can be run against a local instance or using the `gitlab-qa` scenario `Test::Integration::Integrations` | 
 | `:service_ping_disabled`  | The test interacts with the GitLab configuration service ping at the instance level to turn admin setting service ping checkbox on or off. This tag will have the test run only in the `service_ping_disabled` job and must be paired with the `:orchestrated` and `:requires_admin` tags. |
 | `:jira`           | The test requires a Jira Server. [GitLab-QA](https://gitlab.com/gitlab-org/gitlab-qa) provisions the Jira Server in a Docker container when the `Test::Integration::Jira` test scenario is run.
 | `:kubernetes`     | The test includes a GitLab instance that is configured to be run behind an SSH tunnel, allowing a TLS-accessible GitLab. This test also includes provisioning of at least one Kubernetes cluster to test against. _This tag is often be paired with `:orchestrated`._ |
diff --git a/doc/development/testing_guide/end_to_end/running_tests_that_require_special_setup.md b/doc/development/testing_guide/end_to_end/running_tests_that_require_special_setup.md
index ef3e0624395..7fb95769fc2 100644
--- a/doc/development/testing_guide/end_to_end/running_tests_that_require_special_setup.md
+++ b/doc/development/testing_guide/end_to_end/running_tests_that_require_special_setup.md
@@ -141,6 +141,34 @@ docker stop gitlab-gitaly-cluster praefect postgres gitaly3 gitaly2 gitaly1
 docker rm gitlab-gitaly-cluster praefect postgres gitaly3 gitaly2 gitaly1
 ```
 
+## Tests that require a runner
+
+To execute tests that use a runner without errors, while creating the GitLab Docker instance the `--hostname` parameter in the Docker `run` command should be given a specific interface IP address or a non-loopback hostname accessible from the runner container. Having `localhost` (or `127.0.0.1`) as the GitLab hostname won't work (unless the GitLab Runner is created with the Docker network as `host`)
+
+Examples of tests which require a runner:
+
+- `qa/qa/specs/features/ee/browser_ui/13_secure/create_merge_request_with_secure_spec.rb`
+- `qa/qa/specs/features/browser_ui/4_verify/runner/register_runner_spec.rb`
+
+Example:
+
+```shell
+docker run \ 
+  --detach \
+  --hostname interface_ip_address \
+  --publish 80:80 \
+  --name gitlab \
+  --restart always \
+  --volume ~/ee_volume/config:/etc/gitlab \
+  --volume ~/ee_volume/logs:/var/log/gitlab \
+  --volume ~/ee_volume/data:/var/opt/gitlab \
+  --shm-size 256m \
+  gitlab/gitlab-ee:latest
+```
+
+Where `interface_ip_address` is your local network's interface IP, which you can find with the `ifconfig` command.
+The same would apply to GDK running with the instance address as `localhost` too.
+
 ## Guide to run and debug Monitor tests
 
 ### How to set up
diff --git a/doc/development/testing_guide/flaky_tests.md b/doc/development/testing_guide/flaky_tests.md
index d2e68ea7715..09fc8f4d33e 100644
--- a/doc/development/testing_guide/flaky_tests.md
+++ b/doc/development/testing_guide/flaky_tests.md
@@ -14,14 +14,14 @@ eventually.
 ## Quarantined tests
 
 When a test frequently fails in `main`,
-[a ~"master:broken" issue](https://about.gitlab.com/handbook/engineering/workflow/#broken-master)
-should be created.
+create [a ~"failure::flaky-test" issue](https://about.gitlab.com/handbook/engineering/workflow/#broken-master).
+
 If the test cannot be fixed in a timely fashion, there is an impact on the
-productivity of all the developers, so it should be placed in quarantine by
-assigning the `:quarantine` metadata with the issue URL.
+productivity of all the developers, so it should be quarantined by
+assigning the `:quarantine` metadata with the issue URL, and add the `~"quarantined test"` label to the issue.
 
 ```ruby
-it 'should succeed', quarantine: 'https://gitlab.com/gitlab-org/gitlab/-/issues/12345' do
+it 'succeeds', quarantine: 'https://gitlab.com/gitlab-org/gitlab/-/issues/12345' do
   expect(response).to have_gitlab_http_status(:ok)
 end
 ```
@@ -32,23 +32,13 @@ This means it is skipped unless run with `--tag quarantine`:
 bin/rspec --tag quarantine
 ```
 
-**Before putting a test in quarantine, you should make sure that a
-~"master:broken" issue exists for it so it doesn't stay in quarantine forever.**
-
 Once a test is in quarantine, there are 3 choices:
 
-- Should the test be fixed (that is, get rid of its flakiness)?
-- Should the test be moved to a lower level of testing?
-- Should the test be removed entirely (for example, because there's already a
+- Fix the test (that is, get rid of its flakiness).
+- Move the test to a lower level of testing.
+- Remove the test entirely (for example, because there's already a
   lower-level test, or it's duplicating another same-level test, or it's testing
-  too much etc.)?
-
-### Quarantine tests on the CI
-
-Quarantined tests are run on the CI in dedicated jobs that are allowed to fail:
-
-- `rspec-pg-quarantine` (CE & EE)
-- `rspec-pg-quarantine-ee` (EE only)
+  too much etc.).
 
 ## Automatic retries and flaky tests detection
 
@@ -116,6 +106,7 @@ reproduction.
   - [Lazy loaded images can cause Capybara to mis-click](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/18713)
 - [Triggering JS events before the event handlers are set up](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/18742)
 - [Wait for the image to be lazy-loaded when asserting on a Markdown image's `src` attribute](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/25408)
+- [Avoid asserting against flash notice banners](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/79432)
 
 #### Capybara viewport size related issues
 
diff --git a/doc/development/testing_guide/frontend_testing.md b/doc/development/testing_guide/frontend_testing.md
index eb35227a50c..d03a4976a8c 100644
--- a/doc/development/testing_guide/frontend_testing.md
+++ b/doc/development/testing_guide/frontend_testing.md
@@ -748,6 +748,8 @@ Consult the [official Jest docs](https://jestjs.io/docs/jest-object#mock-modules
 
 ## Running Frontend Tests
 
+Before generating fixtures, make sure you have a running GDK instance.
+
 For running the frontend tests, you need the following commands:
 
 - `rake frontend:fixtures` (re-)generates [fixtures](#frontend-test-fixtures). Make sure that
diff --git a/doc/development/testing_guide/review_apps.md b/doc/development/testing_guide/review_apps.md
index aef83109b9b..27d5ae70ed7 100644
--- a/doc/development/testing_guide/review_apps.md
+++ b/doc/development/testing_guide/review_apps.md
@@ -15,6 +15,7 @@ For any of the following scenarios, the `start-review-app-pipeline` job would be
 - for merge requests with CI config changes
 - for merge requests with frontend changes
 - for merge requests with changes to `{,ee/,jh/}{app/controllers}/**/*`
+- for merge requests with changes to `{,ee/,jh/}{app/models}/**/*`
 - for merge requests with QA changes
 - for scheduled pipelines
 - the MR has the `pipeline:run-review-app` label set
@@ -30,6 +31,8 @@ You can also manually start the `review-qa-all`: it runs the full QA suite.
 After the end-to-end test runs have finished, [Allure reports](https://github.com/allure-framework/allure2) are generated and published by
 the `allure-report-qa-smoke`, `allure-report-qa-reliable`, and `allure-report-qa-all` jobs. A comment with links to the reports are added to the merge request.
 
+Errors can be found in the `gitlab-review-apps` Sentry project and [filterable by Review App URL](https://sentry.gitlab.net/gitlab/gitlab-review-apps/?query=url%3A%22https%3A%2F%2Fgitlab-review-require-ve-u92nn2.gitlab-review.app%2F%22) or [commit SHA](https://sentry.gitlab.net/gitlab/gitlab-review-apps/releases/6095b501da7/all-events/).
+
 ## Performance Metrics
 
 On every [pipeline](https://gitlab.com/gitlab-org/gitlab/pipelines/125315730) in the `qa` stage, the