Add latest changes from gitlab-org/gitlab@master

10 files changed, 180 insertions, 34 deletions

diff --git a/doc/administration/packages/container_registry.md b/doc/administration/packages/container_registry.md
index 6ef1a3ec607..d3415913bab 100644
--- a/doc/administration/packages/container_registry.md
+++ b/doc/administration/packages/container_registry.md
@@ -144,7 +144,7 @@ otherwise you will run into conflicts.
 
 1. Validate using:
 
-   ```sh
+   ```shell
    openssl s_client -showcerts -servername gitlab.example.com -connect gitlab.example.com:443 > cacert.pem
    ```
 
@@ -156,7 +156,7 @@ If your certificate provider provides the CA Bundle certificates, append them to
 1. Open `/home/git/gitlab/config/gitlab.yml`, find the `registry` entry and
    configure it with the following settings:
 
-   ```
+   ```yaml
    registry:
      enabled: true
      host: gitlab.example.com
@@ -408,7 +408,7 @@ when you [deployed your docker registry](https://docs.docker.com/registry/deploy
 
 `s3` storage driver example:
 
-```yml
+```yaml
 storage:
   s3:
     accesskey: 's3-access-key'
@@ -616,7 +616,7 @@ Before diving in to the following sections, here's some basic troubleshooting:
 If you're using a self-signed certificate with your Container Registry, you
 might encounter issues during the CI jobs like the following:
 
-```
+```plaintext
 Error response from daemon: Get registry.example.com/v1/users/: x509: certificate signed by unknown authority
 ```
 
@@ -666,7 +666,7 @@ Container Registry > Authorization token duration (minutes)**.
 When using AWS S3 with the GitLab registry, an error may occur when pushing
 large images. Look in the Registry log for the following error:
 
-```
+```plaintext
 level=error msg="response completed with error" err.code=unknown err.detail="unexpected EOF" err.message="unknown error"
 ```
 
@@ -813,7 +813,7 @@ diagnose a problem with the S3 setup.
 A user attempted to enable an S3-backed Registry. The `docker login` step went
 fine. However, when pushing an image, the output showed:
 
-```text
+```plaintext
 The push refers to a repository [s3-testing.myregistry.com:4567/root/docker-test/docker-image]
 dc5e59c14160: Pushing [==================================================>] 14.85 kB
 03c20c1a019a: Pushing [==================================================>] 2.048 kB
@@ -859,27 +859,27 @@ The following installation instructions assume you are running Ubuntu:
    Enter <kbd>CTRL</kbd>-<kbd>C</kbd> to quit.
 1. Install the certificate from `~/.mitmproxy` to your system:
 
-   ```sh
+   ```shell
    sudo cp ~/.mitmproxy/mitmproxy-ca-cert.pem /usr/local/share/ca-certificates/mitmproxy-ca-cert.crt
    sudo update-ca-certificates
    ```
 
 If successful, the output should indicate that a certificate was added:
 
-```sh
+```shell
 Updating certificates in /etc/ssl/certs... 1 added, 0 removed; done.
 Running hooks in /etc/ca-certificates/update.d....done.
 ```
 
 To verify that the certificates are properly installed, run:
 
-```sh
+```shell
 mitmproxy --port 9000
 ```
 
 This will run mitmproxy on port `9000`. In another window, run:
 
-```sh
+```shell
 curl --proxy http://localhost:9000 https://httpbin.org/status/200
 ```
 
@@ -892,7 +892,7 @@ For Docker to connect through a proxy, you must start the Docker daemon with the
 proper environment variables. The easiest way is to shutdown Docker (e.g. `sudo initctl stop docker`)
 and then run Docker by hand. As root, run:
 
-```sh
+```shell
 export HTTP_PROXY="http://localhost:9000"
 export HTTPS_PROXY="https://localhost:9000"
 docker daemon --debug
@@ -905,7 +905,7 @@ This will launch the Docker daemon and proxy all connections through mitmproxy.
 Now that we have mitmproxy and Docker running, we can attempt to login and push
 a container image. You may need to run as root to do this. For example:
 
-```sh
+```shell
 docker login s3-testing.myregistry.com:4567
 docker push s3-testing.myregistry.com:4567/root/docker-test/docker-image
 ```
diff --git a/doc/development/README.md b/doc/development/README.md
index d551e6f471e..c30b7cbc558 100644
--- a/doc/development/README.md
+++ b/doc/development/README.md
@@ -54,6 +54,7 @@ description: 'Learn how to contribute to GitLab.'
 - [How to dump production data to staging](db_dump.md)
 - [Working with the GitHub importer](github_importer.md)
 - [Import/Export development documentation](import_export.md)
+- [Test Import Project](import_project.md)
 - [Elasticsearch integration docs](elasticsearch.md)
 - [Working with Merge Request diffs](diffs.md)
 - [Kubernetes integration guidelines](kubernetes.md)
diff --git a/doc/development/import_project.md b/doc/development/import_project.md
new file mode 100644
index 00000000000..06c0bd02262
--- /dev/null
+++ b/doc/development/import_project.md
@@ -0,0 +1,153 @@
+# Test Import Project
+
+For testing, we can import our own [GitLab CE](https://gitlab.com/gitlab-org/gitlab-foss/) project (named `gitlabhq` in this case) under a group named `qa-perf-testing`. Project tarballs that can be used for testing can be found over on the [performance-data](https://gitlab.com/gitlab-org/quality/performance-data) project. A different project could be used if required.
+
+There are several options for importing the project into your GitLab environment. They are detailed as follows with the assumption that the recommended group `qa-perf-testing` and project `gitlabhq` are being set up.
+
+## Importing the project
+
+There are several ways to import a project.
+
+### Importing via UI
+
+The first option is to simply [import the Project tarball file via the GitLab UI](../user/project/settings/import_export.md#importing-the-project):
+
+1. Create the group `qa-perf-testing`
+1. Import the [GitLab FOSS project tarball](https://gitlab.com/gitlab-org/quality/performance-data/raw/master/gitlabhq_export.tar.gz) into the Group.
+
+It should take up to 15 minutes for the project to fully import. You can head to the project's main page for the current status.
+
+NOTE: **Note:** This method ignores all the errors silently (including the ones related to `GITALY_DISABLE_REQUEST_LIMITS`) and is used by GitLab's users. For development and testing, check the other methods below.
+
+### Importing via the `import-project` script
+
+A convenient script, [`bin/import-project`](https://gitlab.com/gitlab-org/quality/performance/blob/master/bin/import-project), is provided with [performance](https://gitlab.com/gitlab-org/quality/performance) project to import the Project tarball into a GitLab environment via API from the terminal.
+
+Note that to use the script, it will require some preparation if you haven't done so already:
+
+1. First, set up [`Ruby`](https://www.ruby-lang.org/en/documentation/installation/) and [`Ruby Bundler`](https://bundler.io) if they aren't already available on the machine.
+1. Next, install the required Ruby Gems via Bundler with `bundle install`.
+
+For details how to use `bin/import-project`, run:
+
+```sh
+bin/import-project --help
+```
+
+The process should take up to 15 minutes for the project to import fully. The script will keep checking periodically for the status and exit once import has completed.
+
+### Importing via GitHub
+
+There is also an option to [import the project via GitHub](../user/project/import/github.md):
+
+1. Create the group `qa-perf-testing`
+1. Import the GitLab FOSS repository that's [mirrored on GitHub](https://github.com/gitlabhq/gitlabhq) into the group via the UI.
+
+This method will take longer to import than the other methods and will depend on several factors. It's recommended to use the other methods.
+
+### Importing via a rake task
+
+[`import.rake`](https://gitlab.com/gitlab-org/gitlab/blob/master/lib/tasks/gitlab/import_export/import.rake) was introduced for importing large GitLab project exports.
+
+As part of this script we also disable direct and background upload to avoid situations where a huge archive is being uploaded to GCS (while being inside a transaction, which can cause idle transaction timeouts).
+
+We can simply run this script from the terminal:
+
+```sh
+bundle exec rake "gitlab:import_export:import[root, root, testingprojectimport, /path/to/file.tar.gz]"
+```
+
+### Importing via the Rails console
+
+The last option is to import a project using a Rails console:
+
+1. Start a Ruby on Rails console:
+
+   ```sh
+   # Omnibus GitLab
+   gitlab-rails console
+
+   # For installations from source
+   sudo -u git -H bundle exec rails console RAILS_ENV=production
+   ```
+
+1. Create a project and run `ProjectTreeRestorer`:
+
+   ```ruby
+   shared_class = Struct.new(:export_path) do
+     def error(message)
+       raise message
+     end
+   end
+
+   user = User.first
+
+   shared = shared_class.new(path)
+
+   project = Projects::CreateService.new(user, { name: name, namespace: user.namespace }).execute
+   begin
+     #Enable Request store
+     RequestStore.begin!
+     Gitlab::ImportExport::ProjectTreeRestorer.new(user: user, shared: shared, project: project).restore
+   ensure
+     RequestStore.end!
+     RequestStore.clear!
+   end
+   ```
+
+1. In case you need the repository as well, you can restore it using:
+
+   ```ruby
+   repo_path = File.join(shared.export_path, Gitlab::ImportExport.project_bundle_filename)
+
+   Gitlab::ImportExport::RepoRestorer.new(path_to_bundle: repo_path,
+                                          shared: shared,
+                                          project: project).restore
+   ```
+
+    We are storing all import failures in the `import_failures` data table.
+
+    To make sure that the project import finished without any issues, check:
+
+    ```ruby
+    project.import_failures.all
+    ```
+
+## Performance testing
+
+For Performance testing, we should:
+
+- Import a quite large project, [`gitlabhq`](https://gitlab.com/gitlab-org/quality/performance-data#gitlab-performance-test-framework-data) should be a good example.
+- Measure the execution time of `ProjectTreeRestorer`.
+- Count the number of executed SQL queries during the restore.
+- Observe the number of GC cycles happening.
+
+You can use this [snippet](https://gitlab.com/gitlab-org/gitlab/snippets/1924954), which will restore the project, and measure the execution time of `ProjectTreeRestorer`, number of SQL queries and number of GC cycles happening.
+
+You can execute the script from the `gdk/gitlab` directory like this:
+
+```sh
+bundle exec rails r  /path_to_sript/script.rb project_name /path_to_extracted_project request_store_enabled
+```
+
+## Troubleshooting
+
+In this section we'll detail any known issues we've seen when trying to import a project and how to manage them.
+
+### Gitaly calls error when importing
+
+If you're attempting to import a large project into a development environment, you may see Gitaly throw an error about too many calls or invocations, for example:
+
+```
+Error importing repository into qa-perf-testing/gitlabhq - GitalyClient#call called 31 times from single request. Potential n+1?
+```
+
+This is due to a [n+1 calls limit being set for development setups](gitaly.md#toomanyinvocationserror-errors). You can work around this by setting `GITALY_DISABLE_REQUEST_LIMITS=1` as an environment variable, restarting your development environment and importing again.
+
+## Access token setup
+
+Many of the tests also require a GitLab Personal Access Token. This is due to numerous endpoints themselves requiring authentication.
+
+[The official GitLab docs detail how to create this token](../user/profile/personal_access_tokens.md#creating-a-personal-access-token). The tests require that the token is generated by an admin user and that it has the `API` and `read_repository` permissions.
+
+Details on how to use the Access Token with each type of test are found in their respective documentation.
diff --git a/doc/user/analytics/code_review_analytics.md b/doc/user/analytics/code_review_analytics.md
index cd2b0dd2bf7..0b501b1a56d 100644
--- a/doc/user/analytics/code_review_analytics.md
+++ b/doc/user/analytics/code_review_analytics.md
@@ -47,13 +47,3 @@ and trends between them. For example:
 
 - On [Starter or Bronze tier](https://about.gitlab.com/pricing/) and above.
 - By users with Reporter access and above.
-
-## Disable with feature flag
-
-Code Review Analytics is [currently enabled by a feature flag](https://gitlab.com/gitlab-org/gitlab/issues/194165)
-that defaults to ON, meaning the feature is available. If you experience performance problems or
-otherwise wish to disable the feature, a GitLab administrator can execute a command in a Rails console:
-
-```ruby
-Feature.disable(:code_review_analytics)
-```
diff --git a/doc/user/analytics/index.md b/doc/user/analytics/index.md
index 07d756af582..3117a5dfbca 100644
--- a/doc/user/analytics/index.md
+++ b/doc/user/analytics/index.md
@@ -15,8 +15,7 @@ Once enabled, click on **Analytics** from the top navigation bar.
 
 From the centralized analytics workspace, the following analytics are available:
 
-- [Code Review Analytics](code_review_analytics.md), enabled with the `code_review_analytics`
-  [feature flag](../../development/feature_flags/development.md#enabling-a-feature-flag-in-development). **(STARTER)**
+- [Code Review Analytics](code_review_analytics.md). **(STARTER)**
 - [Cycle Analytics](cycle_analytics.md), enabled with the `cycle_analytics`
   [feature flag](../../development/feature_flags/development.md#enabling-a-feature-flag-in-development). **(PREMIUM)**
 - [Productivity Analytics](productivity_analytics.md), enabled with the `productivity_analytics`
diff --git a/doc/user/application_security/dependency_scanning/index.md b/doc/user/application_security/dependency_scanning/index.md
index 0c1896e210d..3f725089932 100644
--- a/doc/user/application_security/dependency_scanning/index.md
+++ b/doc/user/application_security/dependency_scanning/index.md
@@ -58,14 +58,13 @@ The following languages and dependency managers are supported.
 | Java ([Gradle](https://gradle.org/)) | yes | [gemnasium](https://gitlab.com/gitlab-org/security-products/gemnasium) |
 | Java ([Maven](https://maven.apache.org/)) | yes | [gemnasium](https://gitlab.com/gitlab-org/security-products/gemnasium) |
 | JavaScript ([npm](https://www.npmjs.com/), [yarn](https://yarnpkg.com/en/)) | yes | [gemnasium](https://gitlab.com/gitlab-org/security-products/gemnasium), [Retire.js](https://retirejs.github.io/retire.js/)         |
-| Go ([Golang](https://golang.org/)) | not currently ([issue](https://gitlab.com/gitlab-org/gitlab/issues/7132 "Dependency Scanning for Go")) | not available |
 | PHP ([Composer](https://getcomposer.org/))  | yes | [gemnasium](https://gitlab.com/gitlab-org/security-products/gemnasium) |
 | Python ([pip](https://pip.pypa.io/en/stable/)) | yes | [gemnasium](https://gitlab.com/gitlab-org/security-products/gemnasium) |
 | Python ([Pipfile](https://pipenv.kennethreitz.org/en/latest/basics/)) | not currently ([issue](https://gitlab.com/gitlab-org/gitlab/issues/11756 "Pipfile.lock support for Dependency Scanning"))| not available |
 | Python ([poetry](https://poetry.eustace.io/)) | not currently ([issue](https://gitlab.com/gitlab-org/gitlab/issues/7006 "Support Poetry in Dependency Scanning")) | not available |
 | Ruby ([gem](https://rubygems.org/)) | yes | [gemnasium](https://gitlab.com/gitlab-org/security-products/gemnasium), [bundler-audit](https://github.com/rubysec/bundler-audit) |
 | Scala ([sbt](https://www.scala-sbt.org/)) | yes | [gemnasium](https://gitlab.com/gitlab-org/security-products/gemnasium) |
-| Go ([go](https://golang.org/)) | yes (alpha) | [gemnasium](https://gitlab.com/gitlab-org/security-products/gemnasium) |
+| Go ([Golang](https://golang.org/)) | yes ([alpha](https://gitlab.com/gitlab-org/gitlab/issues/7132)) | [gemnasium](https://gitlab.com/gitlab-org/security-products/gemnasium) |
 
 ## Configuration
 
diff --git a/doc/user/application_security/license_compliance/index.md b/doc/user/application_security/license_compliance/index.md
index 97804a451b9..fb6fa0632f6 100644
--- a/doc/user/application_security/license_compliance/index.md
+++ b/doc/user/application_security/license_compliance/index.md
@@ -52,7 +52,7 @@ The following languages and package managers are supported.
 | JavaScript | [Bower](https://bower.io/), [npm](https://www.npmjs.com/), [yarn](https://yarnpkg.com/) ([experimental support](https://github.com/pivotal/LicenseFinder#experimental-project-types)) |[License Finder](https://github.com/pivotal/LicenseFinder)|
 | Go         | [Godep](https://github.com/tools/godep), go get ([experimental support](https://github.com/pivotal/LicenseFinder#experimental-project-types)), gvt ([experimental support](https://github.com/pivotal/LicenseFinder#experimental-project-types)), glide ([experimental support](https://github.com/pivotal/LicenseFinder#experimental-project-types)), dep ([experimental support](https://github.com/pivotal/LicenseFinder#experimental-project-types)), trash ([experimental support](https://github.com/pivotal/LicenseFinder#experimental-project-types))  and govendor ([experimental support](https://github.com/pivotal/LicenseFinder#experimental-project-types)), [go mod](https://github.com/golang/go/wiki/Modules) ([experimental support](https://github.com/pivotal/LicenseFinder#experimental-project-types))   |[License Finder](https://github.com/pivotal/LicenseFinder)|
 | Java       | [Gradle](https://gradle.org/), [Maven](https://maven.apache.org/) |[License Finder](https://github.com/pivotal/LicenseFinder)|
-| .NET       | [Nuget](https://www.nuget.org/)                                   |[License Finder](https://github.com/pivotal/LicenseFinder)|
+| .NET      | [Nuget](https://www.nuget.org/) (.NET Framework is supported via the [mono project](https://www.mono-project.com/). Windows specific dependencies are not supported at this time.)  |[License Finder](https://github.com/pivotal/LicenseFinder)|
 | Python     | [pip](https://pip.pypa.io/en/stable/)                             |[License Finder](https://github.com/pivotal/LicenseFinder)|
 | Ruby       | [gem](https://rubygems.org/)                                      |[License Finder](https://github.com/pivotal/LicenseFinder)|
 | Erlang     | [rebar](https://www.rebar3.org/) ([experimental support](https://github.com/pivotal/LicenseFinder#experimental-project-types))|[License Finder](https://github.com/pivotal/LicenseFinder)|
diff --git a/doc/user/application_security/sast/index.md b/doc/user/application_security/sast/index.md
index cd1dabb6ef4..9a6f61c9c8c 100644
--- a/doc/user/application_security/sast/index.md
+++ b/doc/user/application_security/sast/index.md
@@ -54,6 +54,8 @@ this is enabled by default.
 Privileged mode is not necessary if you've [disabled Docker in Docker
 for SAST](#disabling-docker-in-docker-for-sast)
 
+CAUTION: **Caution:** Our SAST jobs currently expect a Linux container type. Windows containers are not yet supported.
+
 CAUTION: **Caution:**
 If you use your own Runners, make sure that the Docker version you have installed
 is **not** `19.03.00`. See [troubleshooting information](#error-response-from-daemon-error-processing-tar-file-docker-tar-relocation-error) for details.
diff --git a/doc/user/permissions.md b/doc/user/permissions.md
index 985c1babdb5..57873494a48 100644
--- a/doc/user/permissions.md
+++ b/doc/user/permissions.md
@@ -37,11 +37,12 @@ usernames. A GitLab administrator can configure the GitLab instance to
 NOTE: **Note:**
 In GitLab 11.0, the Master role was renamed to Maintainer.
 
-While Maintainer is the highest project-level role, some actions can only be performed by a personal namespace or group owner.
+While Maintainer is the highest project-level role, some actions can only be performed by a personal namespace or group owner,
+or an instance admin, who receives all permissions.
 
 The following table depicts the various user permission levels in a project.
 
-| Action                                            | Guest   | Reporter   | Developer   |Maintainer| Owner  |
+| Action                                            | Guest   | Reporter   | Developer   |Maintainer| Owner* |
 |---------------------------------------------------|---------|------------|-------------|----------|--------|
 | Download project                                  | ✓ (*1*) | ✓          | ✓           | ✓        | ✓      |
 | Leave comments                                    | ✓ (*1*) | ✓          | ✓           | ✓        | ✓      |
@@ -138,10 +139,11 @@ The following table depicts the various user permission levels in a project.
 | Force push to protected branches (*4*)            |         |            |             |          |        |
 | Remove protected branches (*4*)                   |         |            |             |          |        |
 
-- (*1*): Guest users are able to perform this action on public and internal projects, but not private projects.
-- (*2*): Guest users can only view the confidential issues they created themselves
-- (*3*): If **Public pipelines** is enabled in **Project Settings > CI/CD**
-- (*4*): Not allowed for Guest, Reporter, Developer, Maintainer, or Owner. See [Protected Branches](./project/protected_branches.md).
+\* Owner permission is only available at the group or personal namespace level (and for instance admins) and is inherited by its projects.  
+(*1*): Guest users are able to perform this action on public and internal projects, but not private projects.  
+(*2*): Guest users can only view the confidential issues they created themselves.  
+(*3*): If **Public pipelines** is enabled in **Project Settings > CI/CD**.  
+(*4*): Not allowed for Guest, Reporter, Developer, Maintainer, or Owner. See [Protected Branches](./project/protected_branches.md).
 
 ## Project features permissions
 
diff --git a/doc/user/search/index.md b/doc/user/search/index.md
index d7ca43b1164..580406c5fe2 100644
--- a/doc/user/search/index.md
+++ b/doc/user/search/index.md
@@ -89,7 +89,7 @@ Some filters can be added multiple times. These include but are not limited to a
 
 ![multiple assignees filtering](img/multiple_assignees.png)
 
-### Shortcut
+## Shortcut
 
 You'll also find a shortcut on the search field on the top-right of the project's dashboard to
 quickly access issues and merge requests created or assigned to you within that project: