Merge branch '52568-external-mr-diffs' into 'master'

Allow merge request diffs to be placed into an object store

Closes #52568

See merge request gitlab-org/gitlab-ce!24276

3 files changed, 157 insertions, 0 deletions

diff --git a/doc/administration/index.md b/doc/administration/index.md
index 0b673d61139..184754cd467 100644
--- a/doc/administration/index.md
+++ b/doc/administration/index.md
@@ -48,6 +48,7 @@ Learn how to install, configure, update, and maintain your GitLab instance.
 - [Third party offers](../user/admin_area/settings/third_party_offers.md)
 - [Compliance](compliance.md): A collection of features from across the application that you may configure to help ensure that your GitLab instance and DevOps workflow meet compliance standards.
 - [Diff limits](../user/admin_area/diff_limits.md): Configure the diff rendering size limits of branch comparison pages.
+- [Merge request diffs](merge_request_diffs.md): Configure the diffs shown on merge requests
 - [Broadcast Messages](../user/admin_area/broadcast_messages.md): Send messages to GitLab users through the UI.
 
 #### Customizing GitLab's appearance
diff --git a/doc/administration/merge_request_diffs.md b/doc/administration/merge_request_diffs.md
new file mode 100644
index 00000000000..94620c3d3a0
--- /dev/null
+++ b/doc/administration/merge_request_diffs.md
@@ -0,0 +1,154 @@
+# Merge request diffs administration
+
+> **Notes:**
+> - External merge request diffs introduced in GitLab 11.8
+
+Merge request diffs are size-limited copies of diffs associated with merge
+requests. When viewing a merge request, diffs are sourced from these copies
+wherever possible as a performance optimization.
+
+By default, merge request diffs are stored in the database, in a table named
+`merge_request_diff_files`. Larger installations may find this table grows too
+large, in which case, switching to external storage is recommended.
+
+### Using external storage
+
+Merge request diffs can be stored on disk, or in object storage. In general, it
+is better to store the diffs in the database than on disk.
+
+To enable external storage of merge request diffs:
+
+---
+
+**In Omnibus installations:**
+
+1. Edit `/etc/gitlab/gitlab.rb` and add the following line:
+
+    ```ruby
+    gitlab_rails['external_diffs_enabled'] = true
+    ```
+
+1. _The external diffs will be stored in in
+   `/var/opt/gitlab/gitlab-rails/shared/external-diffs`._  To change the path,
+   for example to `/mnt/storage/external-diffs`, edit `/etc/gitlab/gitlab.rb`
+   and add the following line:
+
+    ```ruby
+    gitlab_rails['external_diffs_storage_path'] = "/mnt/storage/external-diffs"
+    ```
+
+1. Save the file and [reconfigure GitLab][] for the changes to take effect.
+
+---
+
+**In installations from source:**
+
+1. Edit `/home/git/gitlab/config/gitlab.yml` and add or amend the following
+   lines:
+
+    ```yaml
+    external_diffs:
+      enabled: true
+    ```
+
+1. _The external diffs will be stored in 
+   `/home/git/gitlab/shared/external-diffs`._ To change the path,  for example
+   to `/mnt/storage/external-diffs`, edit `/home/git/gitlab/config/gitlab.yml`
+   and add or amend the following lines:
+
+    ```yaml
+    external_diffs:
+      enabled: true
+      storage_path: /mnt/storage/external-diffs
+    ```
+
+1. Save the file and [restart GitLab][] for the changes to take effect.
+
+### Using object storage
+
+Instead of storing the external diffs on disk, we recommended you use an object
+store like AWS S3 instead. This configuration relies on valid AWS credentials to
+be configured already.
+
+### Object Storage Settings
+
+For source installations, these settings are nested under `external_diffs:` and
+then `object_store:`. On omnibus installs, they are prefixed by
+`external_diffs_object_store_`.
+
+| Setting | Description | Default |
+|---------|-------------|---------|
+| `enabled` | Enable/disable object storage | `false` |
+| `remote_directory` | The bucket name where external diffs will be stored| |
+| `direct_upload` | Set to true to enable direct upload of external diffs without the need of local shared storage. Option may be removed once we decide to support only single storage for all files. | `false` |
+| `background_upload` | Set to false to disable automatic upload. Option may be removed once upload is direct to S3 | `true` |
+| `proxy_download` | Set to true to enable proxying all files served. Option allows to reduce egress traffic as this allows clients to download directly from remote storage instead of proxying all data | `false` |
+| `connection` | Various connection options described below | |
+
+#### S3 compatible connection settings
+
+The connection settings match those provided by [Fog](https://github.com/fog), and are as follows:
+
+| Setting | Description | Default |
+|---------|-------------|---------|
+| `provider` | Always `AWS` for compatible hosts | AWS |
+| `aws_access_key_id` | AWS credentials, or compatible | |
+| `aws_secret_access_key` | AWS credentials, or compatible | |
+| `aws_signature_version` | AWS signature version to use. 2 or 4 are valid options. Digital Ocean Spaces and other providers may need 2. | 4 |
+| `region` | AWS region | us-east-1 |
+| `host` | S3 compatible host for when not using AWS, e.g. `localhost` or `storage.example.com` | s3.amazonaws.com |
+| `endpoint` | Can be used when configuring an S3 compatible service such as [Minio](https://www.minio.io), by entering a URL such as `http://127.0.0.1:9000` | (optional) |
+| `path_style` | Set to true to use `host/bucket_name/object` style paths instead of `bucket_name.host/object`. Leave as false for AWS S3 | false |
+| `use_iam_profile` | Set to true to use IAM profile instead of access keys | false
+
+**In Omnibus installations:**
+
+1. Edit `/etc/gitlab/gitlab.rb` and add the following lines by replacing with
+   the values you want:
+
+    ```ruby
+    gitlab_rails['external_diffs_enabled'] = true
+    gitlab_rails['external_diffs_object_store_enabled'] = true
+    gitlab_rails['external_diffs_object_store_remote_directory'] = "external-diffs"
+    gitlab_rails['external_diffs_object_store_connection'] = {
+      'provider' => 'AWS',
+      'region' => 'eu-central-1',
+      'aws_access_key_id' => 'AWS_ACCESS_KEY_ID',
+      'aws_secret_access_key' => 'AWS_SECRET_ACCESS_KEY'
+    }
+    ```
+
+    NOTE: if you are using AWS IAM profiles, be sure to omit the
+    AWS access key and secret access key/value pairs. For example:
+
+    ```ruby
+    gitlab_rails['external_diffs_object_store_connection'] = {
+      'provider' => 'AWS',
+      'region' => 'eu-central-1',
+      'use_iam_profile' => true
+    }
+    ```
+
+1. Save the file and [reconfigure GitLab][] for the changes to take effect.
+
+---
+
+**In installations from source:**
+
+1. Edit `/home/git/gitlab/config/gitlab.yml` and add or amend the following
+   lines:
+
+    ```yaml
+    external_diffs:
+      enabled: true
+      object_store:
+        enabled: true
+        remote_directory: "external-diffs" # The bucket name
+        connection:
+          provider: AWS # Only AWS supported at the moment
+          aws_access_key_id: AWS_ACCESS_KEY_ID
+          aws_secret_access_key: AWS_SECRET_ACCESS_KEY
+          region: eu-central-1
+    ```
+
+1. Save the file and [restart GitLab][] for the changes to take effect.
diff --git a/doc/development/file_storage.md b/doc/development/file_storage.md
index b90dc90e424..597812c8c49 100644
--- a/doc/development/file_storage.md
+++ b/doc/development/file_storage.md
@@ -18,6 +18,7 @@ There are many places where file uploading is used, according to contexts:
   - Issues/MR/Notes Legacy Markdown attachments
   - CI Artifacts (archive, metadata, trace)
   - LFS Objects
+  - Merge request diffs
 
 
 ## Disk storage
@@ -37,6 +38,7 @@ they are still not 100% standardized. You can see them below:
 | Issues/MR/Notes Legacy Markdown attachments | no     | uploads/-/system/note/attachment/:id/:filename              | `AttachmentUploader`   | Note       |
 | CI Artifacts (CE)                     | yes    | shared/artifacts/:disk_hash[0..1]/:disk_hash[2..3]/:disk_hash/:year_:month_:date/:job_id/:job_artifact_id (:disk_hash is SHA256 digest of project_id) | `JobArtifactUploader`  | Ci::JobArtifact  |
 | LFS Objects  (CE)                     | yes    | shared/lfs-objects/:hex/:hex/:object_hash                   | `LfsObjectUploader`    | LfsObject  |
+| External merge request diffs          | yes    | shared/external-diffs/merge_request_diffs/mr-:parent_id/diff-:id | `ExternalDiffUploader` | MergeRequestDiff |
 
 CI Artifacts and LFS Objects behave differently in CE and EE. In CE they inherit the `GitlabUploader`
 while in EE they inherit the `ObjectStorage` and store files in and S3 API compatible object store.