Add latest changes from gitlab-org/gitlab@master

5 files changed, 369 insertions, 3 deletions

diff --git a/doc/user/packages/package_registry/dependency_proxy/index.md b/doc/user/packages/package_registry/dependency_proxy/index.md
new file mode 100644
index 00000000000..e1d968e63c0
--- /dev/null
+++ b/doc/user/packages/package_registry/dependency_proxy/index.md
@@ -0,0 +1,293 @@
+---
+stage: Package
+group: Package Registry
+info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments
+---
+
+# Dependency proxy for packages **(PREMIUM ALL BETA)**
+
+> - [Introduced](https://gitlab.com/groups/gitlab-org/-/epics/3610) in GitLab 16.6 [with a flag](../../../../administration/feature_flags.md) named `packages_dependency_proxy_maven`. Disabled by default.
+> - This feature is in [Beta](../../../../policy/experiment-beta-support.md).
+
+FLAG:
+On self-managed GitLab, by default this feature is not available. To make it available, an administrator can [enable the feature flag](../../../../administration/feature_flags.md) named `packages_dependency_proxy_maven`.
+On GitLab.com, this feature is not available.
+The feature is not ready for production use.
+
+The GitLab dependency proxy for packages is a local proxy for frequently pulled packages.
+It is implemented as a pull-through cache that works at the project level.
+
+Packages are pulled from the upstream package registry and automatically published to the
+project's package registry. Subsequent identical requests are fulfilled with the project's
+package registry. You can use the dependency proxy for packages to reduce unnecessary traffic
+to the upstream registry.
+
+## Enable the dependency proxy
+
+To use the dependency proxy for packages, ensure your project is configured properly,
+and that users who pull from the cache have the necessary authentication:
+
+1. In the global configuration, if the following features are disabled, enable them:
+   - The [`package` feature](../../../../administration/packages/index.md#enable-or-disable-the-package-registry). Enabled by default.
+   - The [`dependency_proxy` feature](../../../../administration/packages/dependency_proxy.md#turn-on-the-dependency-proxy). Enabled by default.
+1. In the project settings, if the [`package` feature](../../package_registry/index.md#disable-the-package-registry)
+   is disabled, enable it. It is enabled by default.
+1. [Add an authentication method](#configure-a-client). The dependency proxy supports the same [authentication methods](../index.md#authenticate-with-the-registry) as the package registry:
+   - [Personal access token](../../../profile/personal_access_tokens.md)
+   - [Project deploy token](../../../project/deploy_tokens/index.md)
+   - [Group deploy token](../../../project/deploy_tokens/index.md)
+   - [Job token](../../../../ci/jobs/ci_job_token.md)
+
+## Advanced caching
+
+When possible, the dependency proxy for packages uses advanced caching to store packages in the project's package registry.
+
+Advanced caching verifies the coherence between the project's package registry
+and the upstream package registry. If the upstream registry has updated files,
+the dependency proxy uses them to update the cached files.
+
+When advanced caching is not supported, the dependency proxy falls back to the default behavior:
+
+- If the requested file is found in the project's package registry, it is returned.
+- If the file is not found, it is fetched from the upstream package registry.
+
+Advanced caching support depends on how the upstream package registry
+responds to dependency proxy requests, and on
+which package format you use.
+
+::Tabs
+
+:::TabTitle Maven
+
+| Package registry                                                                                                                         | Advanced caching supported? |
+|------------------------------------------------------------------------------------------------------------------------------------------|-----------------------------|
+| [GitLab](../../maven_repository/index.md)                                                                                                | **{check-circle}** Yes      |
+| [Maven Central](https://mvnrepository.com/repos/central)                                                                                 | **{check-circle}** Yes      |
+| [Artifactory](https://jfrog.com/integration/maven-repository/)                                                                           | **{check-circle}** Yes      |
+| [Sonatype Nexus](https://help.sonatype.com/repomanager3/nexus-repository-administration/formats/maven-repositories)                      | **{check-circle}** Yes      |
+| [GitHub Packages](https://docs.github.com/en/packages/working-with-a-github-packages-registry/working-with-the-apache-maven-registry)    | **{dotted-circle}** No      |
+
+::EndTabs
+
+### Permissions
+
+When the dependency proxy pulls a file, the following occurs:
+
+1. The dependency proxy searches for a file in the project's package registry.
+   This is a read operation.
+1. The dependency proxy might publish a package file to the project's package registry.
+   This is a write operation.
+
+Whether both steps are executed depends on user permissions.
+The dependency proxy uses the [same permissions as the package registry](../index.md#package-registry-visibility-permissions).
+
+| Project visibility | Minimum [role](../../../../user/permissions.md#roles) | Can read package files? | Can write package files? | Behavior |
+|--------------------|-------------------------------------------------------|-------------------------|--------------------------|----------|
+| Public             | Anonymous                                             | **{dotted-circle}** No  | **{dotted-circle}** No   | Request rejected. |
+| Public             | Guest                                                 | **{check-circle}** Yes  | **{dotted-circle}** No   | Package file returned from either the cache or the remote registry. |
+| Public             | Developer                                             | **{check-circle}** Yes  | **{check-circle}** Yes   | Package file returned from either the cache or the remote registry. The file is published to the cache. |
+| Internal           | Anonymous                                             | **{dotted-circle}** No  | **{dotted-circle}** No   | Request rejected |
+| Internal           | Guest                                                 | **{check-circle}** Yes  | **{dotted-circle}** No   | Package file returned from either the cache or the remote registry. |
+| Internal           | Developer                                             | **{check-circle}** Yes  | **{check-circle}** Yes   | Package file returned from either the cache or the remote registry. The file is published to the cache. |
+| Private            | Anonymous                                             | **{dotted-circle}** No  | **{dotted-circle}** No   | Request rejected |
+| Private            | Reporter                                              | **{check-circle}** Yes  | **{dotted-circle}** No   | Package file returned from either the cache or the remote registry. |
+| Internal           | Developer                                             | **{check-circle}** Yes  | **{check-circle}** Yes   | Package file returned from either the cache or the remote registry. The file is published to the cache. |
+
+At a minimum, any user who can use the dependency proxy can also use the project's package registry.
+
+To ensure the cache is properly filled over time, you should make sure a user with at least the Developer role pulls packages with the dependency proxy.
+
+## Configure a client
+
+Configuring a client for the dependency proxy is similar to configuring a client for the [package registry](../supported_functionality.md#pulling-packages).
+
+### For Maven packages
+
+For Maven packages, [all clients supported by the package registry](../../maven_repository/index.md) are supported by the dependency proxy:
+
+- `mvn`
+- `gradle`
+- `sbt`
+
+For authentication, the Maven dependency proxy access all authentication methods accepted by the [Maven package registry](../../maven_repository/index.md#edit-the-client-configuration).
+You should use the [Basic HTTP authentication](../../maven_repository/index.md#basic-http-authentication) method as it is less complex.
+
+To configure the client:
+
+1. Follow the instructions in [Basic HTTP authentication](../../maven_repository/index.md#basic-http-authentication).
+
+   Make sure you use the endpoint URL `https://gitlab.example.com/api/v4/projects/<project_id>/dependency_proxy/packages/maven`.
+
+1. Complete the configuration for your client:
+
+::Tabs
+
+:::TabTitle mvn
+
+In the `pom.xml` file add a `repository` element:
+
+```xml
+<repositories>
+  <repository>
+    <id>gitlab-maven</id>
+    <url>https://gitlab.example.com/api/v4/projects/<project_id>/dependency_proxy/packages/maven</url>
+  </repository>
+</repositories>
+```
+
+Where:
+
+- `<project_id>` is the ID of the project to be used as a dependency proxy.
+- `<id>` contains the name of the `<server>` used in the [authentication configuration](../../maven_repository/index.md#basic-http-authentication).
+
+By default, Maven Central is checked first through the [Super POM](https://maven.apache.org/guides/introduction/introduction-to-the-pom.html#Super_POM).
+However, you might want to force `mvn` to check the GitLab endpoint first. To do this, follow the instructions from the [request forward](../../maven_repository/index.md#additional-configuration-for-mvn).
+
+:::TabTitle gradle
+
+Add a `repositories` section to your [`build.gradle`](https://docs.gradle.org/current/userguide/tutorial_using_tasks.html) file.
+
+- In Groovy DSL:
+
+  ```groovy
+  repositories {
+      maven {
+          url "https://gitlab.example.com/api/v4/projects/<project_id>/dependency_proxy/packages/maven"
+          name "GitLab"
+          credentials(PasswordCredentials) {
+              username = 'REPLACE_WITH_NAME'
+              password = gitLabPrivateToken
+          }
+          authentication {
+              basic(BasicAuthentication)
+          }
+      }
+  }
+  ```
+
+- In Kotlin DSL:
+
+  ```kotlin
+  repositories {
+      maven {
+          url = uri("https://gitlab.example.com/api/v4/projects/<project_id>/dependency_proxy/packages/maven")
+          name = "GitLab"
+          credentials(BasicAuthentication::class) {
+              username = "REPLACE_WITH_NAME"
+              password = findProperty("gitLabPrivateToken") as String?
+          }
+          authentication {
+              create("basic", BasicAuthentication::class)
+          }
+      }
+  }
+  ```
+
+In this example:
+
+- `<project_id>` is the ID of the project to be used as a dependency proxy.
+- `REPLACE_WITH_NAME` is explained in the [Basic HTTP authentication](../../maven_repository/index.md#basic-http-authentication) section.
+
+:::TabTitle sbt
+
+In your [`build.sbt`](https://www.scala-sbt.org/1.x/docs/Directories.html#sbt+build+definition+files), add the following lines:
+
+```scala
+resolvers += ("gitlab" at "https://gitlab.example.com/api/v4/projects/<project_id>/dependency_proxy/packages/maven")
+
+credentials += Credentials("GitLab Packages Registry", "<host>", "<name>", "<token>")
+```
+
+In this example:
+
+- `<project_id>` is the ID of the project to be used as a dependency proxy.
+- `<host>` is the host present in the `<endpoint url>` without the protocol scheme or the port. Example: `gitlab.example.com`.
+- `<name>` and `<token>` are explained in the [Basic HTTP authentication](../../maven_repository/index.md#basic-http-authentication) section.
+
+::EndTabs
+
+## Configure the remote registry
+
+The dependency proxy must be configured with:
+
+- The URL of the remote package registry.
+- Optional. The required credentials.
+
+To set those parameters:
+
+1. Use the related [GraphQL mutation](../../../../api/graphql/reference/index.md#mutationupdatedependencyproxypackagessettings).
+
+   A frontend to display and manage these settings is proposed by [issue 410726](https://gitlab.com/gitlab-org/gitlab/-/issues/410726).
+
+1. Complete the configuration for your package format:
+
+::Tabs
+
+:::TabTitle Maven
+
+Any Maven package registry can be connected to the dependency proxy. You can
+authorize the connection with the Maven package registry username and password.
+
+To set or update the remote Maven package registry, update the following fields on the settings object:
+
+- `mavenExternalRegistryUrl` - The URL of the remote registry.
+- `mavenExternalRegistryUsername` - Optional. The username to use with the remote registry.
+- `mavenExternalRegistryPassword` - Optional. The password to use with the remote registry.
+
+Example of the GraphQL mutation:
+
+```graphql
+mutation {
+  updateDependencyProxyPackagesSettings(input: {  projectPath : <project full path>, enabled: true, mavenExternalRegistryUrl: <remote registry url>, mavenExternalRegistryUsername: <username>, mavenExternalRegistryPassword: <password> }) {
+    dependencyProxyPackagesSetting {
+      enabled
+      mavenExternalRegistryUrl
+      mavenExternalRegistryUsername
+    }
+    errors
+  }
+}
+```
+
+::EndTabs
+
+## Troubleshooting
+
+### Manual file pull errors
+
+You can pull files manually with cURL.
+However, you might encounter one of the following responses:
+
+- `404 Not Found` - The dependency proxy setting object was not found because it doesn't exist, or because the [requirements](#enable-the-dependency-proxy) were not fulfilled.
+- `401 Unauthorized` - The user was properly authenticated but did not have the proper permissions to access the dependency proxy object.
+- `403 Forbidden` - There was an issue with the [GitLab license level](#enable-the-dependency-proxy).
+- `502 Bad Gateway` - The remote package registry could not fulfill the file request. Verify the [dependency proxy settings](#configure-the-remote-registry).
+- `504 Gateway Timeout` - The remote package registry timed out. Verify the [dependency proxy settings](#configure-the-remote-registry).
+
+::Tabs
+
+:::TabTitle Maven
+
+```shell
+curl --verbose "https://<username>:<personal access token>@gitlab.example.com/api/v4/projects/<project_id>/dependency_proxy/packages/maven/<group id and artifact id>/<version>/<file_name>"
+```
+
+- `<username>` and `<personal access token>` are the credentials to access the dependency proxy of the GitLab instance.
+- `<project_id>` is the project ID.
+- `<group id and artifact id>` are the [Maven package group ID and artifact ID](https://maven.apache.org/pom.html#Maven_Coordinates) joined with a forward slash.
+- `<version>` is the package version.
+- `file_name` is the exact name of the file.
+
+For example, given a package with:
+
+- group ID: `com.my_company`.
+- artifact ID: `my_package`.
+- version: `1.2.3`.
+
+The request to manually pull a package is:
+
+```shell
+curl --verbose "https://<username>:<personal access token>@gitlab.example.com/api/v4/projects/<project_id>/dependency_proxy/packages/maven/com/my_company/my_package/1.2.3/my_package-1.2.3.pom"
+```
+
+::EndTabs
diff --git a/qa/.devcontainer/devcontainer.json b/qa/.devcontainer/devcontainer.json
new file mode 100644
index 00000000000..99c705269c1
--- /dev/null
+++ b/qa/.devcontainer/devcontainer.json
@@ -0,0 +1,40 @@
+// this configuration starts docker container with qa code attached to docker test network
+// this assumes omnibus instance has been started beforehand using `gitlab-qa` gem
+{
+  "name": "gitlab-qa",
+  "privileged": true,
+  "build": {
+    "dockerfile": "../Dockerfile",
+    "context": "../../",
+    "target": "dev"
+  },
+  "runArgs": [
+    "--net=test"
+  ],
+  "mounts": [
+    {
+      "type": "bind",
+      "source": "/var/run/docker.sock",
+      "target": "/var/run/docker.sock"
+    }
+  ],
+  "containerEnv": {
+    "CHROME_DISABLE_DEV_SHM": "true",
+    "GITLAB_QA_ADMIN_ACCESS_TOKEN": "ypCa3Dzb23o5nvsixwPA",
+    "QA_GITLAB_URL": "${localEnv:QA_GITLAB_URL}",
+    "QA_SMOCKER_HOST": "smocker"
+  },
+  "onCreateCommand": "echo \"export QA_GITLAB_URL=${QA_GITLAB_URL:-http://$(docker ps | grep -m 1 'gitlab' | awk '{ print $NF }').test}\" >> /root/.bashrc",
+  "updateContentCommand": "cp .solargraph.yml.example .solargraph.yml",
+  "customizations": {
+    "vscode": {
+      "extensions": [
+        "castwide.solargraph"
+      ],
+      "settings": {
+        "solargraph.diagnostics": true,
+        "solargraph.formatting": true
+      }
+    }
+  }
+}
diff --git a/qa/.solargraph.yml.example b/qa/.solargraph.yml.example
new file mode 100644
index 00000000000..41508b2162d
--- /dev/null
+++ b/qa/.solargraph.yml.example
@@ -0,0 +1,17 @@
+---
+include:
+- "**/*.rb"
+exclude:
+- "spec/**/*"
+- qa/specs/features/**/*
+- ".bundle/**/*"
+require: []
+domains: []
+reporters:
+- rubocop  # diagnostics
+- require_not_found
+formatter:
+  rubocop: # formatting
+require_paths: []
+plugins: []
+max_files: 20000
diff --git a/qa/Dockerfile b/qa/Dockerfile
index 92866ec66cc..1a6a670bf96 100644
--- a/qa/Dockerfile
+++ b/qa/Dockerfile
@@ -1,7 +1,7 @@
-ARG DOCKER_VERSION=20.10.14
-ARG CHROME_VERSION=106
-ARG QA_BUILD_TARGET=ee
+ARG DOCKER_VERSION=24.0.5
+ARG CHROME_VERSION=113
 ARG RUBY_VERSION=3.0
+ARG QA_BUILD_TARGET=ee
 
 FROM registry.gitlab.com/gitlab-org/gitlab-build-images/debian-bullseye-ruby-${RUBY_VERSION}:bundler-2.3-git-2.36-lfs-2.9-chrome-${CHROME_VERSION}-docker-${DOCKER_VERSION}-gcloud-383-kubectl-1.23 AS foss
 LABEL maintainer="GitLab Quality Department <quality@gitlab.com>"
@@ -70,4 +70,9 @@ ONBUILD COPY ./jh/qa /home/gitlab/jh/qa
 ONBUILD COPY ./jh/lib /home/gitlab/jh/lib
 ONBUILD COPY ./jh/config/feature_flags /home/gitlab/jh/config/feature_flags
 
+# Add solargraph gem for devcontainer
+# Solargraph is only present in parent Gemfile so we just install it manually
+FROM ee as dev
+RUN gem install solargraph --force
+
 FROM $QA_BUILD_TARGET
diff --git a/qa/README.md b/qa/README.md
index ee9f1128f70..da845108b18 100644
--- a/qa/README.md
+++ b/qa/README.md
@@ -104,11 +104,22 @@ bundle exec rspec <path/to/spec.rb>
 ```
 
 Note:
+
 - If you want to run tests requiring SSH against GDK, you will need to [modify your GDK setup](https://gitlab.com/gitlab-org/gitlab-qa/blob/master/docs/run_qa_against_gdk.md).
 - If this is your first time running GDK, you can use the password pre-set for `root`. [See supported GitLab environment variables](https://gitlab.com/gitlab-org/gitlab-qa/-/blob/master/docs/what_tests_can_be_run.md#supported-gitlab-environment-variables). If you have changed your `root` password, export the password as `GITLAB_INITIAL_ROOT_PASSWORD`.
 - By default the tests will run in a headless browser. If you'd like to watch the test execution, you can export `WEBDRIVER_HEADLESS=false`.
 - Tests that are tagged `:orchestrated` require special setup (e.g., custom GitLab configuration, or additional services such as LDAP). All [orchestrated tests can be run via `gitlab-qa`](https://gitlab.com/gitlab-org/gitlab-qa/-/blob/master/docs/what_tests_can_be_run.md). There are also [setup instructions](https://docs.gitlab.com/ee/development/testing_guide/end_to_end/running_tests_that_require_special_setup.html) for running some of those tests against GDK or another local GitLab instance.
 
+#### Remote development
+
+For [VSCode](https://code.visualstudio.com/) user, [.devcontainer](.devcontainer/devcontainer.json) defines configuration to develop E2E tests inside a Docker container which by default is attached to the same network as environments started by [`gitlab-qa`](https://gitlab.com/gitlab-org/gitlab-qa) gem. For more information on how to use `dev containers`, see [tutorial](https://code.visualstudio.com/docs/devcontainers/tutorial).
+
+This is useful when developing E2E tests that require GitLab instance with specific omnibus configuration. Typical workflow example:
+
+- Start `GitLab` omnibus instance with specific configuration without running tests, for example: `gitlab-qa Test::Integration::Import EE --no-tests`. For available configurations, see [docs](https://gitlab.com/gitlab-org/gitlab-qa/-/blob/master/docs/what_tests_can_be_run.md)
+- Start dev container from `VSCode` environment
+- Develop and run tests from within the container which will automatically execute against started GitLab instance
+
 #### Generic command for a typical GDK installation
 
 The following is an example command you can use if you have configured GDK to run on a specific IP address and port, with a username, password, and personal access token that aren't the defaults, and you would like the test framework to show debug logs: