diff options
Diffstat (limited to 'doc/development/uploads')
| -rw-r--r-- | doc/development/uploads/background.md | 157 | ||||
| -rw-r--r-- | doc/development/uploads/implementation.md | 193 | ||||
| -rw-r--r-- | doc/development/uploads/index.md | 160 | ||||
| -rw-r--r-- | doc/development/uploads/working_with_uploads.md | 82 |
4 files changed, 210 insertions, 382 deletions
diff --git a/doc/development/uploads/background.md b/doc/development/uploads/background.md index e68e4127b57..1ad1aec23f2 100644 --- a/doc/development/uploads/background.md +++ b/doc/development/uploads/background.md @@ -1,154 +1,11 @@ --- -stage: none -group: unassigned -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +redirect_to: 'index.md' +remove_date: '2022-07-25' --- -# Uploads guide: Why GitLab uses custom upload logic +This document was moved to [another location](index.md). -This page is for developers trying to better understand the history behind GitLab uploads and the -technical challenges associated with uploads. - -## Problem description - -GitLab and [GitLab Workhorse](https://gitlab.com/gitlab-org/gitlab-workhorse) use special rules for handling file uploads, -because in an ordinary Rails application file uploads can become expensive as files grow in size. -Rails often sacrifices performance to provide a better developer experience, including how it handles -`multipart/form-post` uploads. In any Rack server, Rails applications included, when such a request arrives at the application server, -several things happen: - -1. A [Rack middleware](https://github.com/rack/rack/blob/main/lib/rack/multipart.rb) intercepts the request and parses the request body. -1. The middleware writes each file in the multipart request to a temporary directory on disk. -1. A `params` hash is constructed with entries pointing to the respective files on disk. -1. A Rails controller acts on the file contents. - -While this is convenient for developers, it is costly for the Ruby server process to buffer large files on disk. -Because of Ruby's [global interpreter lock](https://en.wikipedia.org/wiki/Global_interpreter_lock), -only a single thread of execution of a given Ruby process can be on CPU. This means the amount of CPU -time spent doing this is not available to other worker threads serving user requests. -Buffering files to disk also means spending more time in I/O routines and mode switches, which are expensive operations. - -The following diagram shows how GitLab handled such a request prior to putting optimizations in place. - -```mermaid -graph TB - subgraph "load balancers" - LB(Proxy) - end - - subgraph "Shared storage" - nfs(NFS) - end - - subgraph "redis cluster" - r(persisted redis) - end - LB-- 1 -->Workhorse - - subgraph "web or API fleet" - Workhorse-- 2 -->rails - end - rails-- "3 (write files)" -->nfs - rails-- "4 (schedule a job)" -->r - - subgraph sidekiq - s(sidekiq) - end - s-- "5 (fetch a job)" -->r - s-- "6 (read files)" -->nfs -``` - -We went through two major iterations of our uploads architecture to improve on these problems: - -1. [Moving disk buffering to Workhorse.](#moving-disk-buffering-to-workhorse) -1. [Uploading to Object Storage from Workhorse.](#moving-to-object-storage-and-direct-uploads) - -### Moving disk buffering to Workhorse - -To address the performance issues resulting from buffering files in Ruby, we moved this logic to Workhorse instead, -our reverse proxy fronting the GitLab Rails application. -Workhorse is written in Go, and is much better at dealing with stream processing and I/O than Rails. - -There are two parts to this implementation: - -1. In Workhorse, a request handler detects `multipart/form-data` content in an incoming user request. - If such a request is detected, Workhorse hijacks the request body before forwarding it to Rails. - Workhorse writes all files to disk, rewrites the multipart form fields to point to the new locations, signs the - request, then forwards it to Rails. -1. In Rails, a [custom multipart Rack middleware](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/middleware/multipart.rb) - identifies any signed multipart requests coming from Workhorse and prepares the `params` hash Rails - would expect, now pointing to the files cached by Workhorse. This makes it a drop-in replacement for `Rack::Multipart`. - -The diagram below shows how GitLab handles such a request today: - -```mermaid -graph TB - subgraph "load balancers" - LB(HA Proxy) - end - - subgraph "Shared storage" - nfs(NFS) - end - - subgraph "redis cluster" - r(persisted redis) - end - LB-- 1 -->Workhorse - - subgraph "web or API fleet" - Workhorse-- "3 (without files)" -->rails - end - Workhorse -- "2 (write files)" -->nfs - rails-- "4 (schedule a job)" -->r - - subgraph sidekiq - s(sidekiq) - end - s-- "5 (fetch a job)" -->r - s-- "6 (read files)" -->nfs -``` - -While this "one-size-fits-all" solution greatly improves performance for multipart uploads without compromising -developer ergonomics, it severely limits GitLab [availability](#availability-challenges) -and [scalability](#scalability-challenges). - -#### Availability challenges - -Moving file buffering to Workhorse addresses the immediate performance problems stemming from Ruby not being good at -handling large file uploads. However, a remaining issue of this solution is its reliance on attached storage, -whether via ordinary hard drives or network attached storage like NFS. -NFS is a [single point of failure](https://en.wikipedia.org/wiki/Single_point_of_failure), and is unsuitable for -deploying GitLab in highly available, cloud native environments. - -#### Scalability challenges - -NFS is not a part of cloud native installations, such as those running in Kubernetes. -In Kubernetes, machine boundaries translate to pods, and without network-attached storage, disk-buffered uploads -must be written directly to the pod's file system. - -Using disk buffering presents us with a scalability challenge here. If Workhorse can only -write files to a pod's private file system, then these files are inaccessible outside of this particular pod. -With disk buffering, a Rails controller will accept a file upload and enqueue it for upload in a Sidekiq -background job. Therefore, Sidekiq requires access to these files. -However, in a cloud native environment all Sidekiq instances run on separate pods, so they are -not able to access files buffered to disk on a web server pod. - -Therefore, all features that involve Sidekiq uploading disk-buffered files severely limit the scalability of GitLab. - -## Moving to object storage and direct uploads - -To address these availability and scalability problems, -instead of buffering files to disk, we have added support for uploading files directly -from Workhorse to a given destination. While it remains possible to upload to local or network-attached storage -this way, you should use a highly available -[object store](https://en.wikipedia.org/wiki/Object_storage), -such as AWS S3, Google GCS, or Azure, for scalability reasons. - -With direct uploads, Workhorse does not buffer files to disk. Instead, it first authorizes the request with -the Rails application to find out where to upload it, then streams the file directly to its ultimate destination. - -To learn more about how disk buffering and direct uploads are implemented, see: - -- [How uploads work technically](implementation.md) -- [Adding new uploads](working_with_uploads.md) +<!-- This redirect file can be deleted after <2022-07-25>. --> +<!-- Redirects that point to other docs in the same project expire in three months. --> +<!-- Redirects that point to docs in a different project or site (for example, link is not relative and starts with `https:`) expire in one year. --> +<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/redirects.html --> diff --git a/doc/development/uploads/implementation.md b/doc/development/uploads/implementation.md index 13a875cd1af..1ad1aec23f2 100644 --- a/doc/development/uploads/implementation.md +++ b/doc/development/uploads/implementation.md @@ -1,190 +1,11 @@ --- -stage: none -group: unassigned -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +redirect_to: 'index.md' +remove_date: '2022-07-25' --- -# Uploads guide: How uploads work technically +This document was moved to [another location](index.md). -This page is for developers trying to better understand what kinds of uploads exist in GitLab and how they are implemented. - -## Kinds of uploads and how to choose between them - -We can identify three major use-cases for an upload: - -1. **storage:** if we are uploading for storing a file (like artifacts, packages, or discussion attachments). In this case [direct upload](#direct-upload) is the proper level as it's the less resource-intensive operation. Additional information can be found on [File Storage in GitLab](../file_storage.md). -1. **in-controller/synchronous processing:** if we allow processing **small files** synchronously, using [disk buffered upload](#disk-buffered-upload) may speed up development. -1. **Sidekiq/asynchronous processing:** Asynchronous processing must implement [direct upload](#direct-upload), the reason being that it's the only way to support Cloud Native deployments without a shared NFS. - -Selecting the proper acceleration is a tradeoff between speed of development and operational costs. - -For more details about currently broken feature see [epic &1802](https://gitlab.com/groups/gitlab-org/-/epics/1802). - -### Handling repository uploads - -Some features involves Git repository uploads without using a regular Git client. -Some examples are uploading a repository file from the web interface and [design management](../../user/project/issues/design_management.md). - -Those uploads requires the rails controller to act as a Git client in lieu of the user. -Those operation falls into _in-controller/synchronous processing_ category, but we have no warranties on the file size. - -In case of a LFS upload, the file pointer is committed synchronously, but file upload to object storage is performed asynchronously with Sidekiq. - -## Upload encodings - -By upload encoding we mean how the file is included within the incoming request. - -We have three kinds of file encoding in our uploads: - -1. <i class="fa fa-check-circle"></i> **multipart**: `multipart/form-data` is the most common, a file is encoded as a part of a multipart encoded request. -1. <i class="fa fa-check-circle"></i> **body**: some APIs uploads files as the whole request body. -1. <i class="fa fa-times-circle"></i> **JSON**: some JSON APIs upload files as base64-encoded strings. This requires a change to GitLab Workhorse, - which is tracked [in this issue](https://gitlab.com/gitlab-org/gitlab/-/issues/325068). - -## Uploading technologies - -By uploading technologies we mean how all the involved services interact with each other. - -GitLab supports 3 kinds of uploading technologies, here follows a brief description with a sequence diagram for each one. Diagrams are not meant to be exhaustive. - -### Rack Multipart upload - -This is the default kind of upload, and it's the most expensive in terms of resources. - -In this case, Workhorse is unaware of files being uploaded and acts as a regular proxy. - -When a multipart request reaches the rails application, `Rack::Multipart` leaves behind temporary files in `/tmp` and uses valuable Ruby process time to copy files around. - -```mermaid -sequenceDiagram - participant c as Client - participant w as Workhorse - participant r as Rails - - activate c - c ->>+w: POST /some/url/upload - w->>+r: POST /some/url/upload - - r->>r: save the incoming file on /tmp - r->>r: read the file for processing - - r-->>-c: request result - deactivate c - deactivate w -``` - -### Disk buffered upload - -This kind of upload avoids wasting resources caused by handling upload writes to `/tmp` in rails. - -This optimization is not active by default on REST API requests. - -When enabled, Workhorse looks for files in multipart MIME requests, uploading -any it finds to a temporary file on shared storage. The MIME data in the request -is replaced with the path to the corresponding file before it is forwarded to -Rails. - -To prevent abuse of this feature, Workhorse signs the modified request with a -special header, stating which entries it modified. Rails ignores any -unsigned path entries. - -```mermaid -sequenceDiagram - participant c as Client - participant w as Workhorse - participant r as Rails - participant s as NFS - - activate c - c ->>+w: POST /some/url/upload - - w->>+s: save the incoming file on a temporary location - s-->>-w: request result - - w->>+r: POST /some/url/upload - Note over w,r: file was replaced with its location<br>and other metadata - - opt requires async processing - r->>+redis: schedule a job - redis-->>-r: job is scheduled - end - - r-->>-c: request result - deactivate c - w->>-w: cleanup - - opt requires async processing - activate sidekiq - sidekiq->>+redis: fetch a job - redis-->>-sidekiq: job - - sidekiq->>+s: read file - s-->>-sidekiq: file - - sidekiq->>sidekiq: process file - - deactivate sidekiq - end -``` - -### Direct upload - -This is the more advanced acceleration technique we have in place. - -Workhorse asks Rails for temporary pre-signed object storage URLs and directly uploads to object storage. - -In this setup, an extra Rails route must be implemented in order to handle authorization. Examples of this can be found in: - -- [`Projects::LfsStorageController`](https://gitlab.com/gitlab-org/gitlab/-/blob/cc723071ad337573e0360a879cbf99bc4fb7adb9/app/controllers/projects/lfs_storage_controller.rb) - and [its routes](https://gitlab.com/gitlab-org/gitlab/-/blob/cc723071ad337573e0360a879cbf99bc4fb7adb9/config/routes/git_http.rb#L31-32). -- [API endpoints for uploading packages](../packages.md#file-uploads). - -Direct upload falls back to _disk buffered upload_ when `direct_upload` is disabled inside the [object storage setting](../../administration/uploads.md#object-storage-settings). -The answer to the `/authorize` call contains only a file system path. - -```mermaid -sequenceDiagram - participant c as Client - participant w as Workhorse - participant r as Rails - participant os as Object Storage - - activate c - c ->>+w: POST /some/url/upload - - w ->>+r: POST /some/url/upload/authorize - Note over w,r: this request has an empty body - r-->>-w: presigned OS URL - - w->>+os: PUT file - Note over w,os: file is stored on a temporary location. Rails select the destination - os-->>-w: request result - - w->>+r: POST /some/url/upload - Note over w,r: file was replaced with its location<br>and other metadata - - r->>+os: move object to final destination - os-->>-r: request result - - opt requires async processing - r->>+redis: schedule a job - redis-->>-r: job is scheduled - end - - r-->>-c: request result - deactivate c - w->>-w: cleanup - - opt requires async processing - activate sidekiq - sidekiq->>+redis: fetch a job - redis-->>-sidekiq: job - - sidekiq->>+os: get object - os-->>-sidekiq: file - - sidekiq->>sidekiq: process file - - deactivate sidekiq - end -``` +<!-- This redirect file can be deleted after <2022-07-25>. --> +<!-- Redirects that point to other docs in the same project expire in three months. --> +<!-- Redirects that point to docs in a different project or site (for example, link is not relative and starts with `https:`) expire in one year. --> +<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/redirects.html --> diff --git a/doc/development/uploads/index.md b/doc/development/uploads/index.md index c486f2d3689..b8326489d40 100644 --- a/doc/development/uploads/index.md +++ b/doc/development/uploads/index.md @@ -6,9 +6,159 @@ info: To determine the technical writer assigned to the Stage/Group associated w # Uploads development guide -Uploads are an integral part of many GitLab features. To understand how GitLab handles uploads, refer to -the following pages: +Uploads are an integral part of many GitLab features. To understand how GitLab handles uploads, this page +provides an overview of the key mechanisms for transferring files to a storage destination. -- [Why GitLab uses custom upload logic.](background.md) -- [How uploads work technically.](implementation.md) -- [How to add new uploads.](working_with_uploads.md) +GitLab uploads are configured by feature. All features that involve uploads provide the same configuration options, +but they can be configured independently of one another. For example, Git LFS uploads can be configured +independently of CI/CD build artifact uploads, but they both offer the same set of settings keys. These settings +govern how an upload is processed, which can have a dramatic impact on performance and scalability. + +This page summarizes the upload settings that are important in deciding how such files are handled. The sections +that follow then describe each of these mechanisms in more detail. + +## How upload settings drive upload flow + +Before we examine individual upload strategies in more detail, let's examine a high-level +breakdown of which upload settings map to each of these strategies. + +Upload settings themselves are documented in [Uploads administration](../../administration/uploads.md). +Here, we focus on how these settings drive the internals of GitLab upload logic. +At the top level, we distinguish between two **destinations** for uploaded files: + +- [**Local storage**](#local-storage) - Files are stored on a volume attached to the web server node. +- [**Object storage**](#object-storage) - Files are stored in a remote object store bucket. + +In this table, `x.y.z` specifies the path taken through `gitlab.yml`: + +| Setting | Value | Behavior | +| -------------------------------------- | ------- | ------------------------------- | +| `<feature>.object_store.enabled` | `false` | Files are stored locally in `<feature>.storage_path` | +| `<feature>.object_store.enabled` | `true` | Files are stored remotely in `<feature>.object_store.remote_directory` | + +When using object storage, administrators can control how those files are moved into the respective bucket. +This move can happen in one of these ways: + +- [Rails controller upload](#rails-controller-upload). +- [Background upload](#background-upload). +- [Direct upload](#direct-upload). + +These strategies activate as per the following `<feature>.object_store.*` settings: + +| | `background_upload` = `false` | `background_upload` = `true` | +| ------------------------- | ----------------------------- | ------------------------------- | +| `direct_upload` = `false` | Controller upload | Background upload | +| `direct_upload` = `true` | Direct upload | Direct upload (takes precedence)| + +Individual Sidekiq workers might also store files in object storage, which is not something we cover here. +More importantly, `background_upload` does not imply _all files are uploaded by Sidekiq._ +Sidekiq workers that store files in object storage could still exist when this setting is `false`. +Those cases are never user-initiated uploads, but they might occur in response to another user-initiated +action, such as exporting a GitLab repository. + +Finally, Workhorse assists most user-initiated uploads using an upload buffering mechanism to keep slow work out of Rails controllers. +This mechanism is explained in [Workhorse assisted uploads](#workhorse-assisted-uploads), +as it runs orthogonal to much of what we discuss beforehand. + +We now look at each case in more detail. + +## Local storage + +Local storage is the simplest path an upload can take. It was how GitLab treated uploads in its early days. +It assumes a storage volume (like a disk or network attached storage) is accessible +to the Rails application at `storage_path`. This file path is relative to the Rails root directory and, +like any upload setting, configurable per feature. + +When a client sends a file upload, Workhorse first buffers the file to disk, a mechanism explained in more +detail in [Workhorse assisted uploads](#workhorse-assisted-uploads). When the request reaches the Rails +application, the file already exists on local storage, so Rails merely has to move it to the specified +directory to finalize the transaction. + +Local storage cannot be used with cloud-native GitLab (CNG) installations. It is therefore not used for +GitLab SaaS either. + +## Object storage + +To provide horizontally scalable storage, you must use an object store provider such as: + +- Amazon AWS. +- Google Cloud Storage (GCS). +- Azure Cloud Storage. + +Using object storage provides two main benefits: + +- Ease of adding more storage capacity: cloud providers do this for you automatically. +- Enabling horizontal scaling of your GitLab installation: multiple GitLab application servers can access the same data + when it is stored in object storage. + +CNG installations including GitLab SaaS always use object storage (GCS in the case of GitLab SaaS.) + +A challenge with uploading to a remote object store is that it includes an outgoing HTTP request from +GitLab to the object store provider. As mentioned above, there are three different strategies available for how +this HTTP request is sent. + +- [Rails controller upload](#rails-controller-upload). +- [Background upload](#background-upload). +- [Direct upload](#direct-upload). + +### Rails controller upload + +When neither background upload nor direct upload are available, Rails uploads the file to object storage +as part of the controller `create` action. Which controller is responsible depends on the kind of file uploaded. + +A Rails controller upload is very similar to uploading to local storage. The main difference: Rails must +send an HTTP request to the object store. This happens via the [CarrierWave Fog](https://github.com/carrierwaveuploader/carrierwave#fog) +uploader. + +As with local storage, this strategy benefits from [Workhorse assistance](#workhorse-assisted-uploads) to +keep some of the costly I/O work out of Ruby and Rails. Direct upload does a better job at this because it also keeps the HTTP PUT requests to object storage outside Puma. + +This strategy is only suitable for small file uploads, as it is subject to Puma's 60 second request timeout. + +### Background upload + +WARNING: +This strategy is deprecated in GitLab 14.9 and later, and is scheduled to [be removed in GitLab 15.0](https://gitlab.com/gitlab-org/gitlab/-/issues/26600). + +With background uploads enabled: + +1. Files are uploaded as if they were to reside in local storage. +1. When Rails saves the upload metadata and the transaction completes, a Sidekiq job is scheduled. +1. The Sidekiq job transfers the file to the object store bucket. + - If the job completes, the upload record is updated to reflect the file's new location. + - If the job fails or gets lost, the upload stays in local storage and has the lifecycle of a normal local storage upload. + +As Rails and Sidekiq must cooperate to move the file to its final destination, it requires shared +storage and as such is unsuitable for CNG installations. We do not use background upload in GitLab SaaS. + +As background upload is an extension of local storage, it benefits from the same [Workhorse assistance](#workhorse-assisted-uploads) to +keep costly I/O work out of Ruby and Rails. + +### Direct upload + +Direct upload is the recommended way to move large files into object storage in CNG installations like GitLab SaaS. + +With direct upload enabled, Workhorse: + +1. Authorizes the request with Rails. +1. Establishes a connection with the object store itself to transfer the file to a temporary location. +1. When the transfer is complete, Workhorse finalizes the request with Rails. Rails issues an object store copy operation to put the file in its final location. +1. Completes the upload by deleting the temporary file in object storage. + +This strategy is a different form of [Workhorse assistance](#workhorse-assisted-uploads). It does not rely on shared storage that is accessible by both Workhorse and Puma. + +Of all existing upload strategies, direct upload is best able to handle large (gigabyte) uploads. However, because Puma still does an object storage copy operation, which takes time proportional to the size of the upload, there remains a possibility of hitting Puma timeouts. + +## Workhorse assisted uploads + +Most uploads receive assistance from Workhorse in some way. + +- Often, Workhorse buffers the upload to a temporary file. Workhorse adds metadata to the request to tell + Puma the name and location of the temporary file. This requires shared temporary storage between Workhorse and Puma. + All GitLab installations (including CNG) have this shared temporary storage. +- Workhorse sometimes pre-processes the file. For example, for CI artifact uploads, Workhorse creates a separate index + of the contents of the ZIP file. By doing this in Workhorse we bypass the Puma request timeout. + Compared to Sidekiq background processing, this has the advantage that the user does not see an intermediate state + where GitLab accepts the file but has not yet processed it. +- With direct upload, Workhorse can both pre-process the file and upload it to object storage. + Uploading a large file to object storage takes time; by doing this in Workhorse we avoid the Puma request timeout. diff --git a/doc/development/uploads/working_with_uploads.md b/doc/development/uploads/working_with_uploads.md index 99c04888804..4e907530a9f 100644 --- a/doc/development/uploads/working_with_uploads.md +++ b/doc/development/uploads/working_with_uploads.md @@ -6,7 +6,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w # Uploads guide: Adding new uploads -In this section, we describe how to add a new upload route [accelerated](implementation.md#uploading-technologies) by Workhorse for [body and multipart](implementation.md#upload-encodings) encoded uploads. +Here, we describe how to add a new upload route [accelerated](index.md#workhorse-assisted-uploads) by Workhorse. Upload routes belong to one of these categories: @@ -15,31 +15,31 @@ Upload routes belong to one of these categories: 1. GraphQL API: uploads handled by a GraphQL resolve function. WARNING: -GraphQL uploads do not support [direct upload](implementation.md#direct-upload) yet. Depending on the use case, the feature may not work on installations without NFS (like GitLab.com or Kubernetes installations). Uploading to object storage inside the GraphQL resolve function may result in timeout errors. For more details please follow [issue #280819](https://gitlab.com/gitlab-org/gitlab/-/issues/280819). +GraphQL uploads do not support [direct upload](index.md#direct-upload). Depending on the use case, the feature may not work on installations without NFS (like GitLab.com or Kubernetes installations). Uploading to object storage inside the GraphQL resolve function may result in timeout errors. For more details, follow [issue #280819](https://gitlab.com/gitlab-org/gitlab/-/issues/280819). ## Update Workhorse for the new route -For both the Rails controller and Grape API uploads, Workhorse has to be updated in order to get the +For both the Rails controller and Grape API uploads, Workhorse must be updated to get the support for the new upload route. 1. Open a new issue in the [Workhorse tracker](https://gitlab.com/gitlab-org/gitlab-workhorse/-/issues/new) describing precisely the new upload route: - The route's URL. - - The [upload encoding](implementation.md#upload-encodings). + - The upload encoding. - If possible, provide a dump of the upload request. 1. Implement and get the MR merged for this issue above. -1. Ask the Maintainers of [Workhorse](https://gitlab.com/gitlab-org/gitlab-workhorse) to create a new release. You can do that in the MR - directly during the maintainer review or ask for it in the `#workhorse` Slack channel. +1. Ask the Maintainers of [Workhorse](https://gitlab.com/gitlab-org/gitlab-workhorse) to create a new release. You can do that in the merge request + directly during the maintainer review, or ask for it in the `#workhorse` Slack channel. 1. Bump the [Workhorse version file](https://gitlab.com/gitlab-org/gitlab/-/blob/master/GITLAB_WORKHORSE_VERSION) to the version you have from the previous points, or bump it in the same merge request that contains - the Rails changes (see [Implementing the new route with a Rails controller](#implementing-the-new-route-with-a-rails-controller) or [Implementing the new route with a Grape API endpoint](#implementing-the-new-route-with-a-grape-api-endpoint) below). + the Rails changes. Refer to [Implementing the new route with a Rails controller](#implementing-the-new-route-with-a-rails-controller) or [Implementing the new route with a Grape API endpoint](#implementing-the-new-route-with-a-grape-api-endpoint) below. ## Implementing the new route with a Rails controller -For a Rails controller upload, we usually have a [multipart](implementation.md#upload-encodings) upload and there are a +For a Rails controller upload, we usually have a `multipart/form-data` upload and there are a few things to do: 1. The upload is available under the parameter name you're using. For example, it could be an `artifact` - or a nested parameter such as `user[avatar]`. Let's say that we have the upload under the + or a nested parameter such as `user[avatar]`. If you have the upload under the `file` parameter, reading `params[:file]` should get you an [`UploadedFile`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/uploaded_file.rb) instance. 1. Generally speaking, it's a good idea to check if the instance is from the [`UploadedFile`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/uploaded_file.rb) class. For example, see how we checked [that the parameter is indeed an `UploadedFile`](https://gitlab.com/gitlab-org/gitlab/-/commit/ea30fe8a71bf16ba07f1050ab4820607b5658719#51c0cc7a17b7f12c32bc41cfab3649ff2739b0eb_79_77). @@ -53,7 +53,7 @@ builds automatically for you. ## Implementing the new route with a Grape API endpoint -For a Grape API upload, we can have [body or a multipart](implementation.md#upload-encodings) upload. Things are slightly more complicated: two endpoints are needed. One for the +For a Grape API upload, we can have a body or multipart upload. Things are slightly more complicated: two endpoints are needed. One for the Workhorse pre-upload authorization and one for accepting the upload metadata from Workhorse: 1. Implement an endpoint with the URL + `/authorize` suffix that will: @@ -70,8 +70,8 @@ use `requires :file, type: ::API::Validations::Types::WorkhorseFile`. - Check that the request is coming from Workhorse with the `require_gitlab_workhorse!` from the [API helpers](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/api/helpers.rb). - Check the user permissions. - - The remaining code of the processing. This is where the code must be reading the parameter (for -our example, it would be `params[:file]`). + - The remaining code of the processing. In this step, the code must read the parameter. For +our example, it would be `params[:file]`. WARNING: **Do not** call `UploadedFile#from_params` directly! Do not build an [`UploadedFile`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/uploaded_file.rb) @@ -124,40 +124,40 @@ Therefore, document new uploads here by slotting them into the following tables: ### CarrierWave integration -| File | Carrierwave usage | Categorized | +| File | CarrierWave usage | Categorized | |---------------------------------------------------------|----------------------------------------------------------------------------------|---------------------| -| `app/models/project.rb` | `include Avatarable` | :white_check_mark: | -| `app/models/projects/topic.rb` | `include Avatarable` | :white_check_mark: | -| `app/models/group.rb` | `include Avatarable` | :white_check_mark: | -| `app/models/user.rb` | `include Avatarable` | :white_check_mark: | -| `app/models/terraform/state_version.rb` | `include FileStoreMounter` | :white_check_mark: | -| `app/models/ci/job_artifact.rb` | `include FileStoreMounter` | :white_check_mark: | -| `app/models/ci/pipeline_artifact.rb` | `include FileStoreMounter` | :white_check_mark: | -| `app/models/pages_deployment.rb` | `include FileStoreMounter` | :white_check_mark: | -| `app/models/lfs_object.rb` | `include FileStoreMounter` | :white_check_mark: | -| `app/models/dependency_proxy/blob.rb` | `include FileStoreMounter` | :white_check_mark: | -| `app/models/dependency_proxy/manifest.rb` | `include FileStoreMounter` | :white_check_mark: | -| `app/models/packages/composer/cache_file.rb` | `include FileStoreMounter` | :white_check_mark: | -| `app/models/packages/package_file.rb` | `include FileStoreMounter` | :white_check_mark: | -| `app/models/concerns/packages/debian/component_file.rb` | `include FileStoreMounter` | :white_check_mark: | +| `app/models/project.rb` | `include Avatarable` | **{check-circle}** Yes | +| `app/models/projects/topic.rb` | `include Avatarable` | **{check-circle}** Yes | +| `app/models/group.rb` | `include Avatarable` | **{check-circle}** Yes | +| `app/models/user.rb` | `include Avatarable` | **{check-circle}** Yes | +| `app/models/terraform/state_version.rb` | `include FileStoreMounter` | **{check-circle}** Yes | +| `app/models/ci/job_artifact.rb` | `include FileStoreMounter` | **{check-circle}** Yes | +| `app/models/ci/pipeline_artifact.rb` | `include FileStoreMounter` | **{check-circle}** Yes | +| `app/models/pages_deployment.rb` | `include FileStoreMounter` | **{check-circle}** Yes | +| `app/models/lfs_object.rb` | `include FileStoreMounter` | **{check-circle}** Yes | +| `app/models/dependency_proxy/blob.rb` | `include FileStoreMounter` | **{check-circle}** Yes | +| `app/models/dependency_proxy/manifest.rb` | `include FileStoreMounter` | **{check-circle}** Yes | +| `app/models/packages/composer/cache_file.rb` | `include FileStoreMounter` | **{check-circle}** Yes | +| `app/models/packages/package_file.rb` | `include FileStoreMounter` | **{check-circle}** Yes | +| `app/models/concerns/packages/debian/component_file.rb` | `include FileStoreMounter` | **{check-circle}** Yes | | `ee/app/models/issuable_metric_image.rb` | `include FileStoreMounter` | | | `ee/app/models/vulnerabilities/remediation.rb` | `include FileStoreMounter` | | | `ee/app/models/vulnerabilities/export.rb` | `include FileStoreMounter` | | -| `app/models/packages/debian/project_distribution.rb` | `include Packages::Debian::Distribution` | :white_check_mark: | -| `app/models/packages/debian/group_distribution.rb` | `include Packages::Debian::Distribution` | :white_check_mark: | -| `app/models/packages/debian/project_component_file.rb` | `include Packages::Debian::ComponentFile` | :white_check_mark: | -| `app/models/packages/debian/group_component_file.rb` | `include Packages::Debian::ComponentFile` | :white_check_mark: | -| `app/models/merge_request_diff.rb` | `mount_uploader :external_diff, ExternalDiffUploader` | :white_check_mark: | -| `app/models/note.rb` | `mount_uploader :attachment, AttachmentUploader` | :white_check_mark: | -| `app/models/appearance.rb` | `mount_uploader :logo, AttachmentUploader` | :white_check_mark: | -| `app/models/appearance.rb` | `mount_uploader :header_logo, AttachmentUploader` | :white_check_mark: | -| `app/models/appearance.rb` | `mount_uploader :favicon, FaviconUploader` | :white_check_mark: | +| `app/models/packages/debian/project_distribution.rb` | `include Packages::Debian::Distribution` | **{check-circle}** Yes | +| `app/models/packages/debian/group_distribution.rb` | `include Packages::Debian::Distribution` | **{check-circle}** Yes | +| `app/models/packages/debian/project_component_file.rb` | `include Packages::Debian::ComponentFile` | **{check-circle}** Yes | +| `app/models/packages/debian/group_component_file.rb` | `include Packages::Debian::ComponentFile` | **{check-circle}** Yes | +| `app/models/merge_request_diff.rb` | `mount_uploader :external_diff, ExternalDiffUploader` | **{check-circle}** Yes | +| `app/models/note.rb` | `mount_uploader :attachment, AttachmentUploader` | **{check-circle}** Yes | +| `app/models/appearance.rb` | `mount_uploader :logo, AttachmentUploader` | **{check-circle}** Yes | +| `app/models/appearance.rb` | `mount_uploader :header_logo, AttachmentUploader` | **{check-circle}** Yes | +| `app/models/appearance.rb` | `mount_uploader :favicon, FaviconUploader` | **{check-circle}** Yes | | `app/models/project.rb` | `mount_uploader :bfg_object_map, AttachmentUploader` | | -| `app/models/import_export_upload.rb` | `mount_uploader :import_file, ImportExportUploader` | :white_check_mark: | -| `app/models/import_export_upload.rb` | `mount_uploader :export_file, ImportExportUploader` | :white_check_mark: | +| `app/models/import_export_upload.rb` | `mount_uploader :import_file, ImportExportUploader` | **{check-circle}** Yes | +| `app/models/import_export_upload.rb` | `mount_uploader :export_file, ImportExportUploader` | **{check-circle}** Yes | | `app/models/ci/deleted_object.rb` | `mount_uploader :file, DeletedObjectUploader` | | -| `app/models/design_management/action.rb` | `mount_uploader :image_v432x230, DesignManagement::DesignV432x230Uploader` | :white_check_mark: | -| `app/models/concerns/packages/debian/distribution.rb` | `mount_uploader :signed_file, Packages::Debian::DistributionReleaseFileUploader` | :white_check_mark: | -| `app/models/bulk_imports/export_upload.rb` | `mount_uploader :export_file, ExportUploader` | :white_check_mark: | +| `app/models/design_management/action.rb` | `mount_uploader :image_v432x230, DesignManagement::DesignV432x230Uploader` | **{check-circle}** Yes | +| `app/models/concerns/packages/debian/distribution.rb` | `mount_uploader :signed_file, Packages::Debian::DistributionReleaseFileUploader` | **{check-circle}** Yes | +| `app/models/bulk_imports/export_upload.rb` | `mount_uploader :export_file, ExportUploader` | **{check-circle}** Yes | | `ee/app/models/user_permission_export_upload.rb` | `mount_uploader :file, AttachmentUploader` | | | `app/models/ci/secure_file.rb` | `include FileStoreMounter` | | |