Add latest changes from gitlab-org/gitlab@master

7 files changed, 177 insertions, 12 deletions

diff --git a/doc/.vale/gitlab/ElementDescriptors.yml b/doc/.vale/gitlab/ElementDescriptors.yml
new file mode 100644
index 00000000000..254da16d00c
--- /dev/null
+++ b/doc/.vale/gitlab/ElementDescriptors.yml
@@ -0,0 +1,14 @@
+---
+# Suggestion: gitlab.ElementDescriptors
+#
+# Suggests the correct way to describe elements in a form.
+#
+# For a list of all options, see https://errata-ai.github.io/vale/styles/
+extends: substitution
+message: 'When describing elements, %s "%s".'
+link: https://docs.gitlab.com/ee/development/documentation/styleguide/index.html#language
+level: suggestion
+ignorecase: true
+swap:
+  button: 'if possible, rewrite to not use'
+  area: 'use "section" instead of'
diff --git a/doc/administration/object_storage.md b/doc/administration/object_storage.md
index a24de2a0817..6c77576cb27 100644
--- a/doc/administration/object_storage.md
+++ b/doc/administration/object_storage.md
@@ -74,6 +74,7 @@ because it does not require a shared folder.
 
 Consolidated object storage configuration can't be used for backups or
 Mattermost. See the [full table for a complete list](#storage-specific-configuration).
+However, backups can be configured with [server side encryption](../raketasks/backup_restore.md#s3-encrypted-buckets) separately.
 
 Enabling consolidated object storage enables object storage for all object
 types. If you want to use local storage for specific object types, you can
diff --git a/doc/administration/packages/container_registry.md b/doc/administration/packages/container_registry.md
index 1134ef8ac77..1067474c8b4 100644
--- a/doc/administration/packages/container_registry.md
+++ b/doc/administration/packages/container_registry.md
@@ -1054,6 +1054,80 @@ PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin
 
 You may want to add the `-m` flag to [remove untagged manifests and unreferenced layers](#removing-untagged-manifests-and-unreferenced-layers).
 
+## Configuring GitLab and Registry to run on separate nodes (Omnibus GitLab)
+
+By default, package assumes that both services are running on the same node.
+In order to get GitLab and Registry to run on a separate nodes, separate configuration
+is necessary for Registry and GitLab.
+
+### Configuring Registry
+
+Below you will find configuration options you should set in `/etc/gitlab/gitlab.rb`,
+for Registry to run separately from GitLab:
+
+- `registry['registry_http_addr']`, default [set programmatically](https://gitlab.com/gitlab-org/omnibus-gitlab/blob/10-3-stable/files/gitlab-cookbooks/gitlab/libraries/registry.rb#L50). Needs to be reachable by web server (or LB).
+- `registry['token_realm']`, default [set programmatically](https://gitlab.com/gitlab-org/omnibus-gitlab/blob/10-3-stable/files/gitlab-cookbooks/gitlab/libraries/registry.rb#L53). Specifies the endpoint to use to perform authentication, usually the GitLab URL.
+  This endpoint needs to be reachable by user.
+- `registry['http_secret']`, [random string](https://gitlab.com/gitlab-org/omnibus-gitlab/blob/10-3-stable/files/gitlab-cookbooks/gitlab/libraries/registry.rb#L32). A random piece of data used to sign state that may be stored with the client to protect against tampering.
+- `registry['internal_key']`, default [automatically generated](https://gitlab.com/gitlab-org/omnibus-gitlab/blob/10-3-stable/files/gitlab-cookbooks/gitlab/recipes/gitlab-rails.rb#L113-119). Contents of the key that GitLab uses to sign the tokens. They key gets created on the Registry server, but it won't be used there.
+- `gitlab_rails['registry_key_path']`, default [set programmatically](https://gitlab.com/gitlab-org/omnibus-gitlab/blob/10-3-stable/files/gitlab-cookbooks/gitlab/recipes/gitlab-rails.rb#L35). This is the path where `internal_key` contents will be written to disk.
+- `registry['internal_certificate']`, default [automatically generated](https://gitlab.com/gitlab-org/omnibus-gitlab/blob/10-3-stable/files/gitlab-cookbooks/registry/recipes/enable.rb#L60-66). Contents of the certificate that GitLab uses to sign the tokens.
+- `registry['rootcertbundle']`, default [set programmatically](https://gitlab.com/gitlab-org/omnibus-gitlab/blob/10-3-stable/files/gitlab-cookbooks/registry/recipes/enable.rb#L60). Path to certificate. This is the path where `internal_certificate`
+  contents will be written to disk.
+- `registry['health_storagedriver_enabled']`, default [set programmatically](https://gitlab.com/gitlab-org/omnibus-gitlab/blob/10-7-stable/files/gitlab-cookbooks/gitlab/libraries/registry.rb#L88). Configure whether health checks on the configured storage driver are enabled.
+- `gitlab_rails['registry_issuer']`, [default value](https://gitlab.com/gitlab-org/omnibus-gitlab/blob/10-3-stable/files/gitlab-cookbooks/gitlab/attributes/default.rb#L153). This setting needs to be set the same between Registry and GitLab.
+
+### Configuring GitLab
+
+Below you will find configuration options you should set in `/etc/gitlab/gitlab.rb`,
+for GitLab to run separately from Registry:
+
+- `gitlab_rails['registry_enabled']`, must be set to `true`. This setting will
+  signal to GitLab that it should allow Registry API requests.
+- `gitlab_rails['registry_api_url']`, default [set programmatically](https://gitlab.com/gitlab-org/omnibus-gitlab/blob/10-3-stable/files/gitlab-cookbooks/gitlab/libraries/registry.rb#L52). This is the Registry URL used internally that users do not need to interact with, `registry['registry_http_addr']` with scheme.
+- `gitlab_rails['registry_host']`, eg. `registry.gitlab.example`. Registry endpoint without the scheme, the address that gets shown to the end user.
+- `gitlab_rails['registry_port']`. Registry endpoint port, visible to the end user.
+- `gitlab_rails['registry_issuer']` must match the issuer in the Registry configuration.
+- `gitlab_rails['registry_key_path']`, path to the key that matches the certificate on the
+  Registry side.
+- `gitlab_rails['internal_key']`, contents of the key that GitLab uses to sign the tokens.
+
+## Architecture of GitLab Container Registry
+
+The GitLab registry is what users use to store their own Docker images.
+Because of that the Registry is client facing, meaning that we expose it directly
+on the web server (or load balancers, LB for short).
+
+![GitLab Registry diagram](img/gitlab-registry-architecture.png)
+
+The flow described by the diagram above:
+
+1. A user runs `docker login registry.gitlab.example` on their client. This reaches the web server (or LB) on port 443.
+1. Web server connects to the Registry backend pool (by default, using port 5000). Since the user
+   didn’t provide a valid token, the Registry returns a 401 HTTP code and the URL (`token_realm` from
+   Registry configuration) where to get one. This points to the GitLab API.
+1. The Docker client then connects to the GitLab API and obtains a token.
+1. The API signs the token with the registry key and hands it to the Docker client
+1. The Docker client now logs in again with the token received from the API. It can now push and pull Docker images.
+
+Reference: <https://docs.docker.com/registry/spec/auth/token/>
+
+### Communication between GitLab and Registry
+
+Registry doesn’t have a way to authenticate users internally so it relies on
+GitLab to validate credentials. The connection between Registry and GitLab is
+TLS encrypted. The key is used by GitLab to sign the tokens while the certificate
+is used by Registry to validate the signature. By default, a self-signed certificate key pair is generated
+for all installations. This can be overridden as needed.
+
+GitLab interacts with the Registry using the Registry private key. When a Registry
+request goes out, a new short-living (10 minutes) namespace limited token is generated
+and signed with the private key.
+The Registry then verifies that the signature matches the registry certificate
+specified in its configuration and allows the operation.
+GitLab background jobs processing (through Sidekiq) also interacts with Registry.
+These jobs talk directly to Registry in order to handle image deletion.
+
 ## Troubleshooting
 
 Before diving in to the following sections, here's some basic troubleshooting:
@@ -1122,7 +1196,7 @@ CI/CD > Container Registry > Authorization token duration (minutes)**.
 
 ### Docker login attempt fails with: 'token signed by untrusted key'
 
-[Registry relies on GitLab to validate credentials](https://docs.gitlab.com/omnibus/architecture/registry/).
+[Registry relies on GitLab to validate credentials](#architecture-of-gitlab-container-registry)
 If the registry fails to authenticate valid login attempts, you get the following error message:
 
 ```shell
diff --git a/doc/administration/packages/img/gitlab-registry-architecture.png b/doc/administration/packages/img/gitlab-registry-architecture.png
new file mode 100644
index 00000000000..742678d5e11
--- /dev/null
+++ b/doc/administration/packages/img/gitlab-registry-architecture.png
diff --git a/doc/development/service_ping/metrics_dictionary.md b/doc/development/service_ping/metrics_dictionary.md
index 7c2bf41794d..8dc2d2255d1 100644
--- a/doc/development/service_ping/metrics_dictionary.md
+++ b/doc/development/service_ping/metrics_dictionary.md
@@ -22,6 +22,9 @@ All metrics are stored in YAML files:
 
 - [`config/metrics`](https://gitlab.com/gitlab-org/gitlab/-/tree/master/config/metrics)
 
+WARNING:
+Only metrics with a metric definition YAML are added to the Service Ping JSON payload.
+
 Each metric is defined in a separate YAML file consisting of a number of fields:
 
 | Field               | Required | Additional information                                         |
diff --git a/doc/raketasks/backup_restore.md b/doc/raketasks/backup_restore.md
index cddf62644d4..b6f772dee17 100644
--- a/doc/raketasks/backup_restore.md
+++ b/doc/raketasks/backup_restore.md
@@ -407,6 +407,67 @@ For Omnibus GitLab packages:
 1. [Reconfigure GitLab](../administration/restart_gitlab.md#omnibus-gitlab-reconfigure)
    for the changes to take effect
 
+##### S3 Encrypted Buckets
+
+AWS supports these [modes for server side encryption](https://docs.aws.amazon.com/AmazonS3/latest/userguide/serv-side-encryption.html):
+
+- Amazon S3-Managed Keys (SSE-S3)
+- Customer Master Keys (CMKs) Stored in AWS Key Management Service (SSE-KMS)
+- Customer-Provided Keys (SSE-C)
+
+Use your mode of choice with GitLab. Each mode has similar, but slightly
+different, configuration methods.
+
+###### SSE-S3
+
+To enable SSE-S3, in the backup storage options set the `server_side_encryption`
+field to `AES256`. For example, in Omnibus GitLab:
+
+```ruby
+gitlab_rails['backup_upload_storage_options'] = {
+  'server_side_encryption' => 'AES256'
+}
+```
+
+###### SSE-KMS
+
+To enable SSE-KMS, you'll need the [KMS key via its Amazon Resource Name (ARN)
+in the `arn:aws:kms:region:acct-id:key/key-id` format](https://docs.aws.amazon.com/AmazonS3/latest/userguide/UsingKMSEncryption.html). Under the `backup_upload_storage_options` config setting, set:
+
+- `server_side_encryption` to `aws:kms`.
+- `server_side_encryption_kms_key_id` to the ARN of the key.
+
+For example, in Omnibus GitLab:
+
+```ruby
+gitlab_rails['backup_upload_storage_options'] = {
+  'server_side_encryption' => 'aws:kms',
+  'server_side_encryption_kms_key_id' => 'arn:aws:<YOUR KMS KEY ID>:'
+}
+```
+
+###### SSE-C
+
+SSE-C requires you to set these encryption options:
+
+- `backup_encryption`: AES256.
+- `backup_encryption_key`: Unencoded, 32-byte (256 bits) key. The upload fails if this isn't exactly 32 bytes.
+
+For example, in Omnibus GitLab:
+
+```ruby
+gitlab_rails['backup_encryption'] = 'AES256'
+gitlab_rails['backup_encryption_key'] = '<YOUR 32-BYTE KEY HERE>'
+```
+
+If the key contains binary characters and cannot be encoded in UTF-8,
+instead, specify the key with the `GITLAB_BACKUP_ENCRYPTION_KEY` environment variable.
+For example:
+
+```ruby
+gitlab_rails['env'] = { 'GITLAB_BACKUP_ENCRYPTION_KEY' => "\xDE\xAD\xBE\xEF" * 8 }
+```
+
 ##### Digital Ocean Spaces
 
 This example can be used for a bucket in Amsterdam (AMS3):
@@ -458,15 +519,25 @@ For installations from source:
            # use_iam_profile: 'true'
          # The remote 'directory' to store your backups. For S3, this would be the bucket name.
          remote_directory: 'my.s3.bucket'
-         # Turns on AWS Server-Side Encryption with Amazon S3-Managed Keys for backups, this is optional
-         # encryption: 'AES256'
-         # Turns on AWS Server-Side Encryption with Amazon Customer-Provided Encryption Keys for backups, this is optional
-         #   This should be set to the encryption key for Amazon S3 to use to encrypt or decrypt your data.
-         #   'encryption' must also be set in order for this to have any effect.
-         #   To avoid storing the key on disk, the key can also be specified via the `GITLAB_BACKUP_ENCRYPTION_KEY` environment variable.
-         # encryption_key: '<key>'
          # Specifies Amazon S3 storage class to use for backups, this is optional
          # storage_class: 'STANDARD'
+         #
+         # Turns on AWS Server-Side Encryption with Amazon Customer-Provided Encryption Keys for backups, this is optional
+         #   'encryption' must be set in order for this to have any effect.
+         #   'encryption_key' should be set to the 256-bit encryption key for Amazon S3 to use to encrypt or decrypt.
+         #   To avoid storing the key on disk, the key can also be specified via the `GITLAB_BACKUP_ENCRYPTION_KEY`  your data.
+         # encryption: 'AES256'
+         # encryption_key: '<key>'
+         #
+         #
+         # Turns on AWS Server-Side Encryption with Amazon S3-Managed keys (optional)
+         # https://docs.aws.amazon.com/AmazonS3/latest/userguide/serv-side-encryption.html
+         # For SSE-S3, set 'server_side_encryption' to 'AES256'.
+         # For SS3-KMS, set 'server_side_encryption' to 'aws:kms'. Set
+         # 'server_side_encryption_kms_key_id' to the ARN of customer master key.
+         # storage_options:
+         #   server_side_encryption: 'aws:kms'
+         #   server_side_encryption_kms_key_id: 'arn:aws:kms:YOUR-KEY-ID-HERE'
    ```
 
 1. [Restart GitLab](../administration/restart_gitlab.md#installations-from-source)
diff --git a/doc/user/analytics/index.md b/doc/user/analytics/index.md
index 7cb5db4379a..5b7e6e39187 100644
--- a/doc/user/analytics/index.md
+++ b/doc/user/analytics/index.md
@@ -75,12 +75,12 @@ in one place.
 
 [Learn more about instance-level analytics](../admin_area/analytics/index.md).
 
-## Group-level analytics **(PREMIUM)**
+## Group-level analytics
 
 > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/195979) in GitLab 12.8.
 > - Moved to GitLab Premium in 13.9.
 
-The following analytics features are available at the group level:
+GitLab provides several analytics features at the group level. Some of these features require you to use a higher tier than GitLab Free.
 
 - [Application Security](../application_security/security_dashboard/#group-security-dashboard)
 - [Contribution](../group/contribution_analytics/index.md)
@@ -93,7 +93,7 @@ The following analytics features are available at the group level:
 
 ## Project-level analytics
 
-The following analytics features are available at the project level:
+You can use GitLab to review analytics at the project level. Some of these features require you to use a higher tier than GitLab Free.
 
 - [Application Security](../application_security/security_dashboard/#project-security-dashboard)
 - [CI/CD](ci_cd_analytics.md)
@@ -105,8 +105,10 @@ The following analytics features are available at the project level:
 - [Repository](repository_analytics.md)
 - [Value Stream](value_stream_analytics.md)
 
-## User-configurable analytics **(ULTIMATE)**
+## User-configurable analytics
 
 The following analytics features are available for users to create personalized views:
 
 - [Application Security](../application_security/security_dashboard/#security-center)
+
+Be sure to review the documentation page for this feature for GitLab tier requirements.