Add latest changes from gitlab-org/gitlab@12-10-stable-ee

11 files changed, 183 insertions, 31 deletions

diff --git a/doc/administration/file_hooks.md b/doc/administration/file_hooks.md
index 7d935af8e37..21ade36a2a5 100644
--- a/doc/administration/file_hooks.md
+++ b/doc/administration/file_hooks.md
@@ -23,22 +23,22 @@ see the [system hooks] documentation.
 
 ## Setup
 
-The file hooks must be placed directly into the `plugins` directory, subdirectories
+The file hooks must be placed directly into the `file_hooks` directory, subdirectories
 will be ignored. There is an
-[`example` directory inside `plugins`](https://gitlab.com/gitlab-org/gitlab/tree/master/plugins/examples)
+[`example` directory inside `file_hooks`](https://gitlab.com/gitlab-org/gitlab/tree/master/file_hooks/examples)
 where you can find some basic examples.
 
 Follow the steps below to set up a custom hook:
 
 1. On the GitLab server, navigate to the plugin directory.
    For an installation from source the path is usually
-   `/home/git/gitlab/plugins/`. For Omnibus installs the path is
-   usually `/opt/gitlab/embedded/service/gitlab-rails/plugins`.
+   `/home/git/gitlab/file_hooks/`. For Omnibus installs the path is
+   usually `/opt/gitlab/embedded/service/gitlab-rails/file_hooks`.
 
     For [highly available] configurations, your hook file should exist on each
     application server.
 
-1. Inside the `plugins` directory, create a file with a name of your choice,
+1. Inside the `file_hooks` directory, create a file with a name of your choice,
    without spaces or special characters.
 1. Make the hook file executable and make sure it's owned by the Git user.
 1. Write the code to make the file hook function as expected. That can be
@@ -106,9 +106,9 @@ bundle exec rake file_hooks:validate RAILS_ENV=production
 Example of output:
 
 ```plaintext
-Validating file hooks from /plugins directory
-* /home/git/gitlab/plugins/save_to_file.clj succeed (zero exit code)
-* /home/git/gitlab/plugins/save_to_file.rb failure (non-zero exit code)
+Validating file hooks from /file_hooks directory
+* /home/git/gitlab/file_hooks/save_to_file.clj succeed (zero exit code)
+* /home/git/gitlab/file_hooks/save_to_file.rb failure (non-zero exit code)
 ```
 
 [system hooks]: ../system_hooks/system_hooks.md
diff --git a/doc/administration/geo/disaster_recovery/index.md b/doc/administration/geo/disaster_recovery/index.md
index 0a5c39665f4..6f417f955ac 100644
--- a/doc/administration/geo/disaster_recovery/index.md
+++ b/doc/administration/geo/disaster_recovery/index.md
@@ -248,6 +248,10 @@ The data can be removed with the following command:
 sudo rm -rf /var/opt/gitlab/geo-postgresql
 ```
 
+If you have any `geo_secondary[]` configuration options enabled in your `gitlab.rb`
+file, these can be safely commented out or removed, and then [reconfigure GitLab](../../restart_gitlab.md#omnibus-gitlab-reconfigure)
+for the changes to take effect.
+
 ## Promoting secondary Geo replica in multi-secondary configurations
 
 If you have more than one **secondary** node and you need to promote one of them, we suggest you follow
diff --git a/doc/administration/geo/replication/datatypes.md b/doc/administration/geo/replication/datatypes.md
index a1f511fe2a5..3431df3ed1f 100644
--- a/doc/administration/geo/replication/datatypes.md
+++ b/doc/administration/geo/replication/datatypes.md
@@ -145,6 +145,7 @@ successfully, you must replicate their data using some other means.
 | [Maven Repository](../../../user/packages/maven_repository/index.md) | [No](https://gitlab.com/groups/gitlab-org/-/epics/2346)  | No                                                      |                                                                                                            |
 | [Conan Repository](../../../user/packages/conan_repository/index.md) | [No](https://gitlab.com/groups/gitlab-org/-/epics/2346)  | No                                                      |                                                                                                            |
 | [NuGet Repository](../../../user/packages/nuget_repository/index.md) | [No](https://gitlab.com/groups/gitlab-org/-/epics/2346)  | No                                                      |                                                                                                            |
+| [PyPi Repository](../../../user/packages/pypi_repository/index.md) | [No](https://gitlab.com/groups/gitlab-org/-/epics/2554)  | No                                                      |                                                                                                            |
 | [External merge request diffs](../../merge_request_diffs.md)         | [No](https://gitlab.com/gitlab-org/gitlab/issues/33817)  | No                                                      |                                                                                                            |
 | Content in object storage                                            | **Yes**                                                  | No                                                      |                                                                                                            |
 
diff --git a/doc/administration/geo/replication/external_database.md b/doc/administration/geo/replication/external_database.md
index fd14c100ffb..e305718580e 100644
--- a/doc/administration/geo/replication/external_database.md
+++ b/doc/administration/geo/replication/external_database.md
@@ -45,13 +45,17 @@ Skip to the [Configure secondary application node](#configure-secondary-applicat
 The [geo_primary_role](https://docs.gitlab.com/omnibus/roles/#gitlab-geo-roles)
 configures the **primary** node's database to be replicated by making changes to
 `pg_hba.conf` and `postgresql.conf`. Make the following configuration changes
-manually to your external database configuration:
+manually to your external database configuration and ensure that you restart PostgreSQL
+afterwards for the changes to take effect:
 
 ```plaintext
 ##
 ## Geo Primary Role
 ## - pg_hba.conf
 ##
+host    all         all               <trusted primary IP>/32       md5
+host    replication gitlab_replicator <trusted primary IP>/32       md5
+host    all         all               <trusted secondary IP>/32     md5
 host    replication gitlab_replicator <trusted secondary IP>/32     md5
 ```
 
@@ -60,7 +64,6 @@ host    replication gitlab_replicator <trusted secondary IP>/32     md5
 ## Geo Primary Role
 ## - postgresql.conf
 ##
-sql_replication_user = gitlab_replicator
 wal_level = hot_standby
 max_wal_senders = 10
 wal_keep_segments = 50
@@ -72,8 +75,19 @@ hot_standby = on
 
 ### Manually configure the replica database
 
-Make the following configuration changes manually to your `postgresql.conf`
-of external replica database:
+Make the following configuration changes manually to your `pg_hba.conf` and `postgresql.conf`
+of your external replica database and ensure that you restart PostgreSQL afterwards
+for the changes to take effect:
+
+```plaintext
+##
+## Geo Secondary Role
+## - pg_hba.conf
+##
+host    all         all               <trusted secondary IP>/32     md5
+host    replication gitlab_replicator <trusted secondary IP>/32     md5
+host    all         all               <trusted primary IP>/24       md5
+```
 
 ```plaintext
 ##
@@ -145,6 +159,19 @@ outbound rules do not apply to RDS PostgreSQL databases. Therefore, you need to
 rule to the read-replica's security group allowing any TCP traffic from
 the tracking database on port 5432.
 
+1. Ensure that your secondary node can communicate with your tracking database by
+   manually changing the `pg_hba.conf` that is associated with your tracking database.
+   Remember to restart PostgreSQL afterwards for the changes to take effect:
+
+    ```plaintext
+    ##
+    ## Geo Tracking Database Role
+    ## - pg_hba.conf
+    ##
+    host    all         all               <trusted tracking IP>/32      md5
+    host    all         all               <trusted secondary IP>/32     md5
+    ```
+
 1. SSH into a GitLab **secondary** server and login as root:
 
    ```shell
diff --git a/doc/administration/geo/replication/img/geo_node_dashboard.png b/doc/administration/geo/replication/img/geo_node_dashboard.png
index 0d3dc5895af..53ca4767c5b 100644
--- a/doc/administration/geo/replication/img/geo_node_dashboard.png
+++ b/doc/administration/geo/replication/img/geo_node_dashboard.png
diff --git a/doc/administration/gitaly/praefect.md b/doc/administration/gitaly/praefect.md
index d1d0c358dc6..19a69dc348c 100644
--- a/doc/administration/gitaly/praefect.md
+++ b/doc/administration/gitaly/praefect.md
@@ -300,6 +300,60 @@ application server, or a Gitaly node.
    edit `/etc/gitlab/gitlab.rb`, remember to run `sudo gitlab-ctl reconfigure`
    again before trying the `sql-ping` command.
 
+#### Automatic failover
+
+When automatic failover is enabled, Praefect will do automatic detection of the health of internal Gitaly nodes. If the
+primary has a certain amount of health checks fail, it will decide to promote one of the secondaries to be primary, and
+demote the primary to be a secondary.
+
+1. To enable automatic failover, edit `/etc/gitlab/gitlab.rb`:
+
+   ```ruby
+   # failover_enabled turns on automatic failover
+   praefect['failover_enabled'] = true
+   praefect['virtual_storages'] = {
+     'praefect' => {
+       'gitaly-1' => {
+         'address' => 'tcp://GITALY_HOST:8075',
+         'token'   => 'PRAEFECT_INTERNAL_TOKEN',
+         'primary' => true
+       },
+       'gitaly-2' => {
+         'address' => 'tcp://GITALY_HOST:8075',
+         'token'   => 'PRAEFECT_INTERNAL_TOKEN'
+       },
+       'gitaly-3' => {
+         'address' => 'tcp://GITALY_HOST:8075',
+         'token'   => 'PRAEFECT_INTERNAL_TOKEN'
+       }
+     }
+   }
+   ```
+
+1. Save the file and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure).
+
+Below is the picture when Praefect starts up with the config.toml above:
+
+```mermaid
+graph TD
+  A[Praefect] -->|Mutator RPC| B(internal_storage_0)
+  B --> |Replication|C[internal_storage_1]
+```
+
+Let's say suddenly `internal_storage_0` goes down. Praefect will detect this and
+automatically switch over to `internal_storage_1`, and `internal_storage_0` will serve as a secondary:
+
+```mermaid
+graph TD
+  A[Praefect] -->|Mutator RPC| B(internal_storage_1)
+  B --> |Replication|C[internal_storage_0]
+```
+
+NOTE: **Note:**: Currently this feature is supported for setups that only have 1 Praefect instance. Praefect instances running,
+for example behind a load balancer, `failover_enabled` should be disabled. The reason is The reason is because there
+is no coordination that currently happens across different Praefect instances, so there could be a situation where
+two Praefect instances think two different Gitaly nodes are the primary.
+
 ### Gitaly
 
 NOTE: **Note:** Complete these steps for **each** Gitaly node.
@@ -697,6 +751,31 @@ during a failover. Follow issue
 It is likely that we will implement support for Consul, and a cloud native
 strategy in the future.
 
+## Identifying Impact of a Primary Node Failure
+
+When a primary Gitaly node fails, there is a chance of dataloss. Dataloss can occur if there were outstanding replication jobs the secondaries did not manage to process before the failure. The Praefect `dataloss` subcommand helps identify these cases by counting the number of dead replication jobs for each repository within a given timeframe.
+
+```shell
+sudo /opt/gitlab/embedded/bin/praefect -config /var/opt/gitlab/praefect/config.toml dataloss -from <rfc3339-time> -to <rfc3339-time>
+```
+
+If the timeframe is not specified, dead replication jobs from the last six hours are counted:
+
+```shell
+sudo /opt/gitlab/embedded/bin/praefect -config /var/opt/gitlab/praefect/config.toml dataloss
+
+Failed replication jobs between [2020-01-02 00:00:00 +0000 UTC, 2020-01-02 06:00:00 +0000 UTC):
+example/repository-1: 1 jobs
+example/repository-2: 4 jobs
+example/repository-3: 2 jobs
+```
+
+To specify a timeframe in UTC, run:
+
+```shell
+sudo /opt/gitlab/embedded/bin/praefect -config /var/opt/gitlab/praefect/config.toml dataloss -from 2020-01-02T00:00:00+00:00 -to 2020-01-02T00:02:00+00:00
+```
+
 ## Backend Node Recovery
 
 When a Praefect backend node fails and is no longer able to
diff --git a/doc/administration/high_availability/redis.md b/doc/administration/high_availability/redis.md
index a1e5482e5dc..1e19e7e6c01 100644
--- a/doc/administration/high_availability/redis.md
+++ b/doc/administration/high_availability/redis.md
@@ -8,10 +8,15 @@ type: reference
 
 The following are the requirements for providing your own Redis instance:
 
-- GitLab 12.0 and later requires Redis version 3.2 or higher. Version 3.2 or higher is recommend as this is
-  what ships with the GitLab Omnibus package. Older Redis versions do not
-  support an optional count argument to SPOP which is now required for
-  [Merge Trains](../../ci/merge_request_pipelines/pipelines_for_merged_results/merge_trains/index.md).
+- Redis version 5.0 or higher is recommended, as this is what ships with
+  Omnibus GitLab packages starting with GitLab 12.7.
+- Support for Redis 3.2 is deprecated with GitLab 12.10 and will be completely
+  removed in GitLab 13.0.
+- GitLab 12.0 and later requires Redis version 3.2 or higher. Older Redis
+  versions do not support an optional count argument to SPOP which is now
+  required for [Merge Trains](../../ci/merge_request_pipelines/pipelines_for_merged_results/merge_trains/index.md).
+- In addition, if Redis 4 or later is available, GitLab makes use of certain
+  commands like `UNLINK` and `USAGE` which were introduced only in Redis 4.
 - Standalone Redis or Redis high availability with Sentinel are supported. Redis
   Cluster is not supported.
 - Managed Redis from cloud providers such as AWS ElastiCache will work. If these
diff --git a/doc/administration/logs.md b/doc/administration/logs.md
index c43406fb647..e4a3514a539 100644
--- a/doc/administration/logs.md
+++ b/doc/administration/logs.md
@@ -30,17 +30,21 @@ Line breaks have been added to this example for legibility:
   "controller":"Projects::IssuesController",
   "action":"show",
   "status":200,
-  "duration":229.03,
-  "view":174.07,
-  "db":13.24,
   "time":"2017-08-08T20:15:54.821Z",
   "params":[{"key":"param_key","value":"param_value"}],
   "remote_ip":"18.245.0.1",
   "user_id":1,
   "username":"admin",
-  "gitaly_calls":76,
-  "gitaly_duration":7.41,
-  "queue_duration": 112.47
+  "queue_duration_s":0.0,
+  "gitaly_calls":16,
+  "gitaly_duration_s":0.16,
+  "redis_calls":115,
+  "redis_duration_s":0.13,
+  "correlation_id":"O1SdybnnIq7",
+  "cpu_s":17.50,
+  "db_duration_s":0.08,
+  "view_duration_s":2.39,
+  "duration_s":20.54
 }
 ```
 
@@ -48,12 +52,15 @@ This example was a GET request for a specific
 issue. Each line also contains performance data, with times in
 milliseconds:
 
-1. `duration`: total time taken to retrieve the request
-1. `queue_duration`: total time that the request was queued inside GitLab Workhorse
-1. `view`: total time taken inside the Rails views
-1. `db`: total time to retrieve data from the database
+1. `duration_s`: total time taken to retrieve the request
+1. `queue_duration_s`: total time that the request was queued inside GitLab Workhorse
+1. `view_duration_s`: total time taken inside the Rails views
+1. `db_duration_s`: total time to retrieve data from PostgreSQL
+1. `redis_duration_s`: total time to retrieve data from Redis
+1. `cpu_s`: total time spent on CPU
+1. `gitaly_duration_s`: total time taken by Gitaly calls
 1. `gitaly_calls`: total number of calls made to Gitaly
-1. `gitaly_duration`: total time taken by Gitaly calls
+1. `redis_calls`: total number of calls made to Redis
 
 User clone and fetch activity using HTTP transport appears in this log as `action: git_upload_pack`.
 
@@ -73,9 +80,6 @@ NOTE: **Note:** Starting with GitLab 12.5, if an error occurs, an
   "controller": "Admin::DashboardController",
   "action": "index",
   "status": 500,
-  "duration": 2584.11,
-  "view": 0,
-  "db": 9.21,
   "time": "2019-11-14T13:12:46.156Z",
   "params": [],
   "remote_ip": "127.0.0.1",
@@ -84,7 +88,16 @@ NOTE: **Note:** Starting with GitLab 12.5, if an error occurs, an
   "ua": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.14; rv:70.0) Gecko/20100101 Firefox/70.0",
   "queue_duration": 274.35,
   "correlation_id": "KjDVUhNvvV3",
-  "cpu_s": 2.837645135999999,
+  "queue_duration_s":0.0,
+  "gitaly_calls":16,
+  "gitaly_duration_s":0.16,
+  "redis_calls":115,
+  "redis_duration_s":0.13,
+  "correlation_id":"O1SdybnnIq7",
+  "cpu_s":17.50,
+  "db_duration_s":0.08,
+  "view_duration_s":2.39,
+  "duration_s":20.54
   "exception.class": "NameError",
   "exception.message": "undefined local variable or method `adsf' for #<Admin::DashboardController:0x00007ff3c9648588>",
   "exception.backtrace": [
diff --git a/doc/administration/packages/index.md b/doc/administration/packages/index.md
index d14726d33de..f826741d66f 100644
--- a/doc/administration/packages/index.md
+++ b/doc/administration/packages/index.md
@@ -8,6 +8,7 @@ The Packages feature allows GitLab to act as a repository for the following:
 
 | Software repository | Description | Available in GitLab version |
 | ------------------- | ----------- | --------------------------- |
+| [PyPi Repository](../../user/packages/pypi_repository/index.md) | The GitLab PyPi Repository enables every project in GitLab to have its own space to store [PyPi](https://pypi.org/) packages. | 12.10+ |
 | [NuGet Repository](../../user/packages/nuget_repository/index.md) | The GitLab NuGet Repository enables every project in GitLab to have its own space to store [NuGet](https://www.nuget.org/) packages. | 12.8+ |
 | [Conan Repository](../../user/packages/conan_repository/index.md) | The GitLab Conan Repository enables every project in GitLab to have its own space to store [Conan](https://conan.io/) packages. | 12.4+ |
 | [Maven Repository](../../user/packages/maven_repository/index.md) | The GitLab Maven Repository enables every project in GitLab to have its own space to store [Maven](https://maven.apache.org/) packages. | 11.3+ |
diff --git a/doc/administration/raketasks/praefect.md b/doc/administration/raketasks/praefect.md
new file mode 100644
index 00000000000..4f9cf0e10ba
--- /dev/null
+++ b/doc/administration/raketasks/praefect.md
@@ -0,0 +1,22 @@
+# Praefect Rake Tasks
+
+> [Introduced]( https://gitlab.com/gitlab-org/gitlab/-/merge_requests/28369) in GitLab 12.10.
+
+## Replica checksums
+
+Prints out checksums of the repository of a given project_id on the primary as well as secondary internal Gitaly nodes.
+
+NOTE: **Note:**
+This only is relevant and works for projects that have been created on a praefect storage. See the [Praefect Documentation](../gitaly/praefect.md) for configuring Praefect.
+
+**Omnibus Installation**
+
+```shell
+sudo gitlab-rake "gitlab:praefect:replicas[project_id]"
+```
+
+**Source Installation**
+
+```shell
+sudo -u git -H bundle exec rake "gitlab:praefect:replicas[project_id]" RAILS_ENV=production
+```
diff --git a/doc/administration/troubleshooting/img/AzureAD-scim_attribute_mapping.png b/doc/administration/troubleshooting/img/AzureAD-scim_attribute_mapping.png
index 807936423e5..5ad82003eed 100644
--- a/doc/administration/troubleshooting/img/AzureAD-scim_attribute_mapping.png
+++ b/doc/administration/troubleshooting/img/AzureAD-scim_attribute_mapping.png