diff options
Diffstat (limited to 'doc/development/uploads')
-rw-r--r-- | doc/development/uploads/index.md | 35 | ||||
-rw-r--r-- | doc/development/uploads/working_with_uploads.md | 28 |
2 files changed, 16 insertions, 47 deletions
diff --git a/doc/development/uploads/index.md b/doc/development/uploads/index.md index 41ec71451fb..b5509f5934e 100644 --- a/doc/development/uploads/index.md +++ b/doc/development/uploads/index.md @@ -40,21 +40,9 @@ When using object storage, administrators can control how those files are moved This move can happen in one of these ways: - [Rails controller upload](#rails-controller-upload). -- [Background upload](#background-upload). - [Direct upload](#direct-upload). -These strategies activate as per the following `<feature>.object_store.*` settings: - -| | `background_upload` = `false` | `background_upload` = `true` | -| ------------------------- | ----------------------------- | ------------------------------- | -| `direct_upload` = `false` | Controller upload | Background upload | -| `direct_upload` = `true` | Direct upload | Direct upload (takes precedence)| - Individual Sidekiq workers might also store files in object storage, which is not something we cover here. -More importantly, `background_upload` does not imply _all files are uploaded by Sidekiq._ -Sidekiq workers that store files in object storage could still exist when this setting is `false`. -Those cases are never user-initiated uploads, but they might occur in response to another user-initiated -action, such as exporting a GitLab repository. Finally, Workhorse assists most user-initiated uploads using an upload buffering mechanism to keep slow work out of Rails controllers. This mechanism is explained in [Workhorse assisted uploads](#workhorse-assisted-uploads), @@ -98,12 +86,12 @@ GitLab to the object store provider. As mentioned above, there are three differe this HTTP request is sent. - [Rails controller upload](#rails-controller-upload). -- [Background upload](#background-upload). - [Direct upload](#direct-upload). +- [Workhorse assisted uploads](#workhorse-assisted-uploads). ### Rails controller upload -When neither background upload nor direct upload are available, Rails uploads the file to object storage +When direct upload is not available, Rails uploads the file to object storage as part of the controller `create` action. Which controller is responsible depends on the kind of file uploaded. A Rails controller upload is very similar to uploading to local storage. The main difference: Rails must @@ -115,25 +103,6 @@ keep some of the costly I/O work out of Ruby and Rails. Direct upload does a bet This strategy is only suitable for small file uploads, as it is subject to Puma's 60 second request timeout. -### Background upload - -WARNING: -This strategy is deprecated in GitLab 14.9 and later, and is scheduled to [be removed in GitLab 15.0](https://gitlab.com/gitlab-org/gitlab/-/issues/26600). - -With background uploads enabled: - -1. Files are uploaded as if they were to reside in local storage. -1. When Rails saves the upload metadata and the transaction completes, a Sidekiq job is scheduled. -1. The Sidekiq job transfers the file to the object store bucket. - - If the job completes, the upload record is updated to reflect the file's new location. - - If the job fails or gets lost, the upload stays in local storage and has the lifecycle of a normal local storage upload. - -As Rails and Sidekiq must cooperate to move the file to its final destination, it requires shared -storage and as such is unsuitable for CNG installations. We do not use background upload in GitLab SaaS. - -As background upload is an extension of local storage, it benefits from the same [Workhorse assistance](#workhorse-assisted-uploads) to -keep costly I/O work out of Ruby and Rails. - ### Direct upload Direct upload is the recommended way to move large files into object storage in CNG installations like GitLab SaaS. diff --git a/doc/development/uploads/working_with_uploads.md b/doc/development/uploads/working_with_uploads.md index c88762e6bd5..a3951fb4c7e 100644 --- a/doc/development/uploads/working_with_uploads.md +++ b/doc/development/uploads/working_with_uploads.md @@ -8,9 +8,9 @@ info: To determine the technical writer assigned to the Stage/Group associated w ## Recommendations -- When creating an uploader, [make it a subclass](#where-should-i-store-my-files) of `AttachmentUploader` +- When creating an uploader, [make it a subclass](#where-should-you-store-your-files) of `AttachmentUploader` - Add your uploader to the [tables](#tables) in this document -- Do not add [new object storage buckets](#where-should-i-store-my-files) +- Do not add [new object storage buckets](#where-should-you-store-your-files) - Implement [direct upload](#implementing-direct-upload-support) - If you need to process your uploads, decide [where to do that](#processing-uploads) @@ -19,14 +19,14 @@ info: To determine the technical writer assigned to the Stage/Group associated w - [CarrierWave Uploaders](#carrierwave-uploaders) - [GitLab modifications to CarrierWave](#gitlab-modifications-to-carrierwave) -## Where should I store my files? +## Where should you store your files? CarrierWave Uploaders determine where files get stored. When you create a new Uploader class you are deciding where to store the files of your new feature. First of all, ask yourself if you need a new Uploader class. It is OK -to use the same Uploader class for different mountpoints or different +to use the same Uploader class for different mount points or different models. If you do want or need your own Uploader class then you should make it @@ -160,8 +160,8 @@ we modified it. The central concept of CarrierWave is the **Uploader** class. The Uploader defines where files get stored, and optionally contains validation and processing logic. To use an Uploader you must associate -it with a text column on an ActiveRecord model. This called "mounting" -and the column is called the "mountpoint". For example: +it with a text column on an ActiveRecord model. This is called "mounting" +and the column is called `mountpoint`. For example: ```ruby class Project < ApplicationRecord @@ -169,8 +169,8 @@ class Project < ApplicationRecord end ``` -Now if I upload an avatar called `tanuki.png` the idea is that in the -`projects.avatar` column for my project, CarrierWave stores the string +Now if you upload an avatar called `tanuki.png` the idea is that in the +`projects.avatar` column for your project, CarrierWave stores the string `tanuki.png`, and that the AttachmentUploader class contains the configuration data and directory schema. For example if the project ID is 123, the actual file may be in @@ -179,12 +179,12 @@ The directory `/var/opt/gitlab/gitlab-rails/uploads/-/system/project/avatar/123/` was chosen by the Uploader using among others configuration (`/var/opt/gitlab/gitlab-rails/uploads`), the model name (`project`), -the model ID (`123`) and the mountpoint (`avatar`). +the model ID (`123`) and the mount point (`avatar`). > The Uploader determines the individual storage directory of your -> upload. The mountpoint column in your model contains the filename. +> upload. The `mountpoint` column in your model contains the filename. -You never access the mountpoint column directly because CarrierWave +You never access the `mountpoint` column directly because CarrierWave defines a getter and setter on your model that operates on file handle objects. @@ -213,7 +213,7 @@ CarrierWave has 2 storage engines: |CarrierWave class|GitLab name|Description| |---|---|---| -|`CarrierWave::Storage::File`|`ObjectStorage::Store::LOCAL` |Local files, accessed through the Ruby stdlib| +|`CarrierWave::Storage::File`|`ObjectStorage::Store::LOCAL` |Local files, accessed through the Ruby `stdlib` | | `CarrierWave::Storage::Fog`|`ObjectStorage::Store::REMOTE`|Cloud files, accessed through the [Fog gem](https://github.com/fog/fog)| GitLab uses both of these engines, depending on configuration. @@ -227,8 +227,8 @@ storage engine file by file. An Uploader is associated with two storage areas: regular storage and cache storage. Each has its own storage engine. If you assign a file -to a mountpoint setter (`project.avatar = -File.open('/tmp/tanuki.png')`) you will copy/move the file to cache +to a mount point setter (`project.avatar = File.open('/tmp/tanuki.png')`) +you will copy/move the file to cache storage as a side effect via the `cache!` method. To persist the file you must somehow call the `store!` method. This either happens via [ActiveRecord callbacks](https://github.com/carrierwaveuploader/carrierwave/blob/v1.3.2/lib/carrierwave/orm/activerecord.rb#L55) |