Add latest changes from gitlab-org/gitlab@14-9-stable-eev14.9.0-rc42

5 files changed, 47 insertions, 255 deletions

diff --git a/doc/development/testing_guide/end_to_end/best_practices.md b/doc/development/testing_guide/end_to_end/best_practices.md
index 405ff40ef6a..e0f6cbe632d 100644
--- a/doc/development/testing_guide/end_to_end/best_practices.md
+++ b/doc/development/testing_guide/end_to_end/best_practices.md
@@ -17,13 +17,16 @@ In case custom inflection logic is needed, custom inflectors are added in the [q
 ## Link a test to its test case
 
 Every test should have a corresponding test case in the [GitLab project Test Cases](https://gitlab.com/gitlab-org/gitlab/-/quality/test_cases) as well as a results issue in the [Quality Test Cases project](https://gitlab.com/gitlab-org/quality/testcases/-/issues).
-It's recommended that you reuse the issue created to plan the test as the results issue. If a test case or results issue does not already exist you
-can create them yourself. Alternatively, you can run the test in a pipeline that has reporting
-enabled and the test-case reporter will automatically create a new test case and/or results issue and link the results issue to it's corresponding test case.
+If a test case issue does not yet exist you can create one yourself. To do so, create a new
+issue in the [Test Cases](https://gitlab.com/gitlab-org/gitlab/-/quality/test_cases) GitLab project
+with a placeholder title. After the test case URL is linked to a test in the code, when the test is
+run in a pipeline that has reporting enabled, the `report-results` script automatically updates the
+test case and the results issue.
+If a results issue does not yet exist, the `report-results` script automatically creates one and
+links it to its corresponding test case.
 
-Whether you create a new test case or one is created automatically, you will need to manually add
-a `testcase` RSpec metadata tag. In most cases, a single test will be associated with a single test case
- ([see below for exceptions](#exceptions)).
+To link a test case to a test in the code, you must manually add a `testcase` RSpec metadata tag.
+In most cases, a single test is associated with a single test case.
 
 For example:
 
@@ -41,7 +44,7 @@ RSpec.describe 'Stage' do
 end
 ```
 
-### Exceptions
+### For shared tests
 
 Most tests are defined by a single line of a `spec` file, which is why those tests can be linked to a
 single test case via the `testcase` tag.
@@ -54,24 +57,19 @@ multiple tests, including:
 - Templated tests.
 - Tests in shared examples that include more than one example.
 
-In those and similar cases we can't assign a single `testcase` tag and so we rely on the test-case
-reporter to programmatically determine the correct test case based on the name and description of
-the test. In such cases, the test-case reporter will automatically create a test case and/or results issue
-the first time the test runs, if none exist already.
+In those and similar cases we need to include the test case link by other means.
 
-In such a case, if you create the test case or results issue yourself or want to reuse an existing issue,
-you must use this [end-to-end test issue template](https://gitlab.com/gitlab-org/quality/testcases/-/blob/master/.gitlab/issue_templates/End-to-end%20Test.md)
-to format the issue description. (Note you must copy/paste this for test cases as templates aren't currently available.)
-
-To illustrate, there are two tests in the shared examples in [`qa/specs/features/ee/browser_ui/3_create/repository/restrict_push_protected_branch_spec.rb`](https://gitlab.com/gitlab-org/gitlab/-/blob/47b17db82c38ab704a23b5ba5d296ea0c6a732c8/qa/qa/specs/features/ee/browser_ui/3_create/repository/restrict_push_protected_branch_spec.rb):
+To illustrate, there are two tests in the shared examples in [`qa/specs/features/ee/browser_ui/3_create/repository/restrict_push_protected_branch_spec.rb`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/qa/qa/specs/features/ee/browser_ui/3_create/repository/restrict_push_protected_branch_spec.rb):
 
 ```ruby
-shared_examples 'only user with access pushes and merges' do
-  it 'unselected maintainer user fails to push' do
+shared_examples 'unselected maintainer' do |testcase|
+  it 'user fails to push', testcase: testcase do
     ...
   end
+end
 
-  it 'selected developer user pushes and merges' do
+shared_examples 'selected developer' do |testcase|
+  it 'user pushes and merges', testcase: testcase do
     ...
   end
 end
@@ -84,112 +82,22 @@ RSpec.describe 'Create' do
   describe 'Restricted protected branch push and merge' do
     context 'when only one user is allowed to merge and push to a protected branch' do
       ...
-      it_behaves_like 'only user with access pushes and merges'
-    end
-  end
-end
-```
-
-There would be two associated test cases, one for each shared example, with the following content:
-
-[Test 1 Test Case](https://gitlab.com/gitlab-org/gitlab/-/quality/test_cases/347774):
-
-````markdown
-```markdown
-Title: browser_ui/3_create/repository/restrict_push_protected_branch_spec.rb | Create Restricted
-protected branch push and merge when only one user is allowed to merge and push to a protected
-branch behaves like only user with access pushes and merges selecte...
-
-Description:
-### Full description
-
-Create Restricted protected branch push and merge when only one user is allowed to merge and push
-to a protected branch behaves like only user with access pushes and merges selected developer user
-pushes and merges
-
-### File path
-
-./qa/specs/features/ee/browser_ui/3_create/repository/restrict_push_protected_branch_spec.rb
-
-### DO NOT EDIT BELOW THIS LINE
-
-Active and historical test results:
-
-https://gitlab.com/gitlab-org/quality/testcases/-/issues/2177
-
-```
-````
-
-[Test 1 Results Issue](https://gitlab.com/gitlab-org/quality/testcases/-/issues/2177):
-
-````markdown
-```markdown
-Title: browser_ui/3_create/repository/restrict_push_protected_branch_spec.rb | Create Restricted
-protected branch push and merge when only one user is allowed to merge and push to a protected
-branch behaves like only user with access pushes and merges selecte...
-
-Description:
-### Full description
-
-Create Restricted protected branch push and merge when only one user is allowed to merge and push
-to a protected branch behaves like only user with access pushes and merges selected developer user
-pushes and merges
-
-### File path
-
-./qa/specs/features/ee/browser_ui/3_create/repository/restrict_push_protected_branch_spec.rb
 
-```
-````
-
-[Test 2 Test Case](https://gitlab.com/gitlab-org/gitlab/-/quality/test_cases/347775):
-
-````markdown
-```markdown
-Title: browser_ui/3_create/repository/restrict_push_protected_branch_spec.rb | Create Restricted
-protected branch push and merge when only one user is allowed to merge and push to a protected
-branch behaves like only user with access pushes and merges unselec...
-
-Description:
-### Full description
-
-Create Restricted protected branch push and merge when only one user is allowed to merge and push
-to a protected branch behaves like only user with access pushes and merges unselected maintainer
-user fails to push
-
-### File path
-
-./qa/specs/features/ee/browser_ui/3_create/repository/restrict_push_protected_branch_spec.rb
-
-### DO NOT EDIT BELOW THIS LINE
-
-Active and historical test results:
+      it_behaves_like 'unselected maintainer', 'https://gitlab.com/gitlab-org/gitlab/-/quality/test_cases/347775'
+      it_behaves_like 'selected developer', 'https://gitlab.com/gitlab-org/gitlab/-/quality/test_cases/347774'
+    end
 
-https://gitlab.com/gitlab-org/quality/testcases/-/issues/2176
+    context 'when only one group is allowed to merge and push to a protected branch' do
+      ...
 
+      it_behaves_like 'unselected maintainer', 'https://gitlab.com/gitlab-org/gitlab/-/quality/test_cases/347772'
+      it_behaves_like 'selected developer', 'https://gitlab.com/gitlab-org/gitlab/-/quality/test_cases/347773'
+    end
+  end
+end
 ```
-````
-
-[Test 2 Results Issue](https://gitlab.com/gitlab-org/quality/testcases/-/issues/2176):
-
-````markdown
-```markdown
-Title: browser_ui/3_create/repository/restrict_push_protected_branch_spec.rb | Create Restricted
-protected branch push and merge when only one user is allowed to merge and push to a protected
-branch behaves like only user with access pushes and merges unselec...
-
-Description:
-### Full description
-
-Create Restricted protected branch push and merge when only one user is allowed to merge and push
-to a protected branch behaves like only user with access pushes and merges unselected maintainer
-user fails to push
 
-### File path
-
-./qa/specs/features/ee/browser_ui/3_create/repository/restrict_push_protected_branch_spec.rb
-```
-````
+We recommend creating four associated test cases, two for each shared example.
 
 ## Prefer API over UI
 
@@ -518,3 +426,12 @@ Neither problem is present if we create a custom negatable matcher because the `
 would be used, which would wait only as long as necessary for the job to disappear.
 
 Lastly, negatable matchers are preferred over using matchers of the form `have_no_*` because it's a common and familiar practice to negate matchers using `not_to`. If we facilitate that practice by adding negatable matchers, we make it easier for subsequent test authors to write efficient tests.
+
+## Use logger over puts
+
+We currently use Rails `logger` to handle logs in both GitLab QA application and end-to-end tests.
+This provides additional functionalities when compared with `puts`, such as:
+
+- Ability to specify the logging level.
+- Ability to tag similar logs.
+- Auto-formatting log messages.
diff --git a/doc/development/testing_guide/end_to_end/feature_flags.md b/doc/development/testing_guide/end_to_end/feature_flags.md
index 238b1ccd8f9..c3e3f117c2b 100644
--- a/doc/development/testing_guide/end_to_end/feature_flags.md
+++ b/doc/development/testing_guide/end_to_end/feature_flags.md
@@ -130,7 +130,7 @@ You can set this variable inside the `fabricate_via_api` call. For a consistent
 - Add the word `activated` to the end of a variable's name.
 - Inside the `initialize` method, set the variable's default value.
 
-For example: 
+For example:
 
 ```ruby
 def initialize
diff --git a/doc/development/testing_guide/end_to_end/rspec_metadata_tests.md b/doc/development/testing_guide/end_to_end/rspec_metadata_tests.md
index 07f7e9582ee..f9b505a8271 100644
--- a/doc/development/testing_guide/end_to_end/rspec_metadata_tests.md
+++ b/doc/development/testing_guide/end_to_end/rspec_metadata_tests.md
@@ -20,7 +20,7 @@ This is a partial list of the [RSpec metadata](https://relishapp.com/rspec/rspec
 | `:github`         | The test requires a GitHub personal access token. |
 | `:group_saml`     | The test requires a GitLab instance that has SAML SSO enabled at the group level. Interacts with an external SAML identity provider. Paired with the `:orchestrated` tag. |
 | `:instance_saml`  | The test requires a GitLab instance that has SAML SSO enabled at the instance level. Interacts with an external SAML identity provider. Paired with the `:orchestrated` tag. |
-| `:integrations`   | This aims to test the available [integrations](../../../user/project/integrations/overview.md#integrations-listing). The test requires Docker to be installed in the run context. It will provision the containers and can be run against a local instance or using the `gitlab-qa` scenario `Test::Integration::Integrations` | 
+| `:integrations`   | This aims to test the available [integrations](../../../user/project/integrations/overview.md#integrations-listing). The test requires Docker to be installed in the run context. It will provision the containers and can be run against a local instance or using the `gitlab-qa` scenario `Test::Integration::Integrations` |
 | `:service_ping_disabled`  | The test interacts with the GitLab configuration service ping at the instance level to turn admin setting service ping checkbox on or off. This tag will have the test run only in the `service_ping_disabled` job and must be paired with the `:orchestrated` and `:requires_admin` tags. |
 | `:jira`           | The test requires a Jira Server. [GitLab-QA](https://gitlab.com/gitlab-org/gitlab-qa) provisions the Jira Server in a Docker container when the `Test::Integration::Jira` test scenario is run.
 | `:kubernetes`     | The test includes a GitLab instance that is configured to be run behind an SSH tunnel, allowing a TLS-accessible GitLab. This test also includes provisioning of at least one Kubernetes cluster to test against. _This tag is often be paired with `:orchestrated`._ |
@@ -42,6 +42,7 @@ This is a partial list of the [RSpec metadata](https://relishapp.com/rspec/rspec
 | `:requires_praefect`   | The test requires that the GitLab instance uses [Gitaly Cluster](../../../administration/gitaly/praefect.md) (a.k.a. Praefect) as the repository storage . It's assumed to be used by default but if not the test can be skipped by setting `QA_CAN_TEST_PRAEFECT` to `false`. |
 | `:runner`         | The test depends on and sets up a GitLab Runner instance, typically to run a pipeline. |
 | `:skip_live_env`  | The test is excluded when run against live deployed environments such as Staging, Canary, and Production. |
+| `:skip_fips_env`  | The test is excluded when run against an environment in FIPS mode. |
 | `:skip_signup_disabled` | The test uses UI to sign up a new user and is skipped in any environment that does not allow new user registration via the UI. |
 | `:smoke`          | The test belongs to the test suite which verifies basic functionality of a GitLab instance.|
 | `:smtp`           | The test requires a GitLab instance to be configured to use an SMTP server. Tests SMTP notification email delivery from GitLab by using MailHog. |
diff --git a/doc/development/testing_guide/end_to_end/running_tests_that_require_special_setup.md b/doc/development/testing_guide/end_to_end/running_tests_that_require_special_setup.md
index 7fb95769fc2..49a9124253d 100644
--- a/doc/development/testing_guide/end_to_end/running_tests_that_require_special_setup.md
+++ b/doc/development/testing_guide/end_to_end/running_tests_that_require_special_setup.md
@@ -169,135 +169,6 @@ docker run \
 Where `interface_ip_address` is your local network's interface IP, which you can find with the `ifconfig` command.
 The same would apply to GDK running with the instance address as `localhost` too.
 
-## Guide to run and debug Monitor tests
-
-### How to set up
-
-To run the Monitor tests locally, against the GDK, please follow the preparation steps below:
-
-1. Complete the [Prerequisites](https://gitlab.com/gitlab-org/gitlab-development-kit/-/blob/main/doc/howto/auto_devops/index.md#prerequisites-for-gitlab-team-members-only), at least through step 5. Note that the monitor tests do not require permissions to work with GKE because they use [k3s as a Kubernetes cluster provider](https://github.com/rancher/k3s).
-1. The test setup deploys the app in a Kubernetes cluster, using the Auto DevOps deployment strategy.
-To enable Auto DevOps in GDK, follow the [associated setup](https://gitlab.com/gitlab-org/gitlab-development-kit/-/blob/main/doc/howto/auto_devops/index.md#setup) instructions. If you have problems, review the [troubleshooting guide](https://gitlab.com/gitlab-org/gitlab-development-kit/-/blob/main/doc/howto/auto_devops/tips_and_troubleshooting.md) or reach out to the `#gdk` channel in the internal GitLab Slack.
-1. Do [secure your GitLab instance](https://gitlab.com/gitlab-org/gitlab-development-kit/-/blob/main/doc/howto/auto_devops/index.md#secure-your-gitlab-instance) since it is now publicly accessible on `https://[YOUR-PORT].qa-tunnel.gitlab.info`.
-1. Install the Kubernetes command line tool known as `kubectl`. Use the [official installation instructions](https://kubernetes.io/docs/tasks/tools/).
-
-You might see NGINX issues when you run `gdk start` or `gdk restart`. In that case, run `sft login` to revalidate your credentials and regain access the QA Tunnel.
-
-### How to run
-
-Navigate to the folder in `/your-gdk/gitlab/qa` and issue the command:
-
-```shell
-QA_DEBUG=true WEBDRIVER_HEADLESS=false GITLAB_ADMIN_USERNAME=rootusername GITLAB_ADMIN_PASSWORD=rootpassword GITLAB_QA_ACCESS_TOKEN=your_token_here GITLAB_QA_ADMIN_ACCESS_TOKEN=your_token_here CLUSTER_API_URL=https://kubernetes.docker.internal:6443 bundle exec bin/qa Test::Instance::All https://[YOUR-PORT].qa-tunnel.gitlab.info/ -- qa/specs/features/browser_ui/8_monitor/all_monitor_core_features_spec.rb --tag kubernetes --tag orchestrated --tag requires_admin
-```
-
-The following includes more information on the command:
-
-- `QA_DEBUG` - Set to `true` to verbosely log page object actions.
-- `WEBDRIVER_HEADLESS` - When running locally, set to `false` to allow browser tests to be visible - watch your tests being run.
-- `GITLAB_ADMIN_USERNAME` - Administrator username to use when adding a license.
-- `GITLAB_ADMIN_PASSWORD` - Administrator password to use when adding a license.
-- `GITLAB_QA_ACCESS_TOKEN` and `GITLAB_QA_ADMIN_ACCESS_TOKEN` - A valid personal access token with the `api` scope. This is used for API access during tests, and is used in the version that staging is currently running. The `ADMIN_ACCESS_TOKEN` is from a user with administrator access. Used for API access as an administrator during tests.
-- `CLUSTER_API_URL` - Use the address `https://kubernetes.docker.internal:6443` . This address is used to enable the cluster to be network accessible while deploying using Auto DevOps.
-- `https://[YOUR-PORT].qa-tunnel.gitlab.info/` - The address of your local GDK
-- `qa/specs/features/browser_ui/8_monitor/all_monitor_core_features_spec.rb` - The path to the monitor core specs
-- `--tag` - the meta-tags used to filter the specs correctly
-
-At the moment of this writing, there are two specs which run monitor tests:
-
-- `qa/specs/features/browser_ui/8_monitor/all_monitor_core_features_spec.rb` - has the specs of features in GitLab Free
-- `qa/specs/features/ee/browser_ui/8_monitor/all_monitor_features_spec.rb` - has the specs of features for paid GitLab (Enterprise Edition)
-
-### How to debug
-
-The monitor tests follow this setup flow:
-
-1. Creates a k3s cluster on your local machine.
-1. Creates a project that has Auto DevOps enabled and uses an Express template (NodeJS) for the app to be deployed.
-1. Associates the created cluster to the project and installs GitLab Runner, Prometheus and Ingress which are the needed components for a successful deployment.
-1. Creates a CI pipeline with 2 jobs (`build` and `production`) to deploy the app on the Kubernetes cluster.
-1. Goes to Operation > Metrics menu to verify data is being received and the app is being monitored successfully.
-
-The test requires a number of components. The setup requires time to collect the metrics of a real deployment.
-The complexity of the setup may lead to problems unrelated to the app. The following sections include common strategies to debug possible issues.
-
-#### Deployment with Auto DevOps
-
-When debugging issues in the CI or locally in the CLI, open the Kubernetes job in the pipeline.
-In the job log window, click on the top right icon labeled as *"Show complete raw"* to reveal raw job logs.
-You can now search through the logs for *Job log*, which matches delimited sections like this one:
-
-```shell
-------- Job log: -------
-```
-
-A Job log is a subsection within these logs, related to app deployment. We use two jobs: `build` and `production`.
-You can find the root causes of deployment failures in these logs, which can compromise the entire test.
-If a `build` job fails, the `production` job doesn't run, and the test fails.
-
-The long test setup does not take screenshots of failures, which is a known [issue](https://gitlab.com/gitlab-org/quality/team-tasks/-/issues/270).
-However, if the spec fails (after a successful deployment) then you should be able to find screenshots which display the feature failure.
-To access them in CI, go to the main job log window, look on the left side panel's Job artifacts section, and click Browse.
-
-#### Common issues
-
-**Container Registry**
-
-When enabling Auto DevOps in the GDK, you may see issues with the Container Registry, which stores
-images of the app to be deployed.
-
-You can access the Registry is available by opening an existing project. On the left hand menu,
-select `Packages & Registries > Container Registries`. If the Registry is available, this page should load normally.
-
-Also, the Registry should be running in Docker:
-
-```shell
-$ docker ps
-
-CONTAINER ID        IMAGE                                                                              COMMAND                  CREATED             STATUS              PORTS                    NAMES
-f035f339506c        registry.gitlab.com/gitlab-org/build/cng/gitlab-container-registry:v2.9.1-gitlab   "/bin/sh -c 'exec /b…"   3 hours ago         Up 3 hours          0.0.0.0:5000->5000/tcp   jovial_proskuriakova
-```
-
-The `gdk status` command shows if the registry is running:
-
-```shell
-run: ./services/registry: (pid 2662) 10875s, normally down; run: log: (pid 65148) 177993s
-run: ./services/tunnel_gitlab: (pid 2650) 10875s, normally down; run: log: (pid 65154) 177993s
-run: ./services/tunnel_registry: (pid 2651) 10875s, normally down; run: log: (pid 65155) 177993s
-```
-
-Also, restarting Docker and then, on the Terminal, issue the command
-`docker login https://[YOUR-REGISTRY-PORT].qa-tunnel.gitlab.info:443` and use
-the GDK credentials to sign in. Note that the Registry port and GDK port aren't
-the same. When configuring Auto DevOps in GDK, the `gdk reconfigure` command
-outputs the port of the Registry:
-
-```shell
-*********************************************
-Tunnel URLs
-
-GitLab: https://[PORT].qa-tunnel.gitlab.info
-Registry: https://[PORT].qa-tunnel.gitlab.info
-*********************************************
-```
-
-These Tunnel URLs are used by the QA SSH Tunnel generated when enabling Auto DevOps on the GDK.
-
-**Pod Eviction**
-
-Pod eviction happens when a node in a Kubernetes cluster is running out of memory or disk. After many local deployments this issue can happen. The UI shows that installing Prometheus, GitLab Runner and Ingress failed. How to be sure it is an Eviction? While the test is running, open another Terminal window and debug the current Kubernetes cluster by `kubectl get pods --all-namespaces`. If you observe that Pods have *Evicted status* such as the install-runner here:
-
-```shell
-$ kubectl get pods --all-namespaces
-
-NAMESPACE             NAME                                      READY   STATUS    RESTARTS   AGE
-gitlab-managed-apps   install-ingress                           0/1     Pending   0          25s
-gitlab-managed-apps   install-prometheus                        0/1     Pending   0          12s
-gitlab-managed-apps   install-runner                            0/1     Evicted   0          75s
-```
-
-You can free some memory with either of the following commands: `docker prune system` or `docker prune volume`.
-
 ## Geo tests
 
 Geo end-to-end tests can run locally against a [Geo GDK setup](https://gitlab.com/gitlab-org/gitlab-development-kit/-/blob/main/doc/howto/geo.md) or on Geo spun up in Docker containers.
diff --git a/doc/development/testing_guide/flaky_tests.md b/doc/development/testing_guide/flaky_tests.md
index 09fc8f4d33e..3c8df7d3416 100644
--- a/doc/development/testing_guide/flaky_tests.md
+++ b/doc/development/testing_guide/flaky_tests.md
@@ -117,14 +117,17 @@ reproduction.
 - [Don't wait for AJAX when no AJAX request is fired](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/30461): <https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/10454>
 - [Bis](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/34647): <https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/12626>
 
-#### PhantomJS / WebKit related issues
-
-- Memory is through the roof! (Load images but block images requests!): <https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/12003>
-
 #### Capybara expectation times out
 
 - [Test imports a project (via Sidekiq) that is growing over time, leading to timeouts when the import takes longer than 60 seconds](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/22599)
 
+### Hanging specs
+
+If a spec hangs, it might be caused by a [bug in Rails](https://github.com/rails/rails/issues/34310):
+
+- <https://gitlab.com/gitlab-org/gitlab/-/merge_requests/81112>
+- <https://gitlab.com/gitlab-org/gitlab/-/issues/337039>
+
 ## Resources
 
 - [Flaky Tests: Are You Sure You Want to Rerun Them?](https://semaphoreci.com/blog/2017/04/20/flaky-tests.html)