| Age | Commit message (Collapse) | Author |
|---|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
rails_helper.rb's only logic was to require
spec_helper.rb.
|
|
|
|
We don't need to find the filename from the remote URL
|
|
When file_mover_spec.rb ran, it would initialize fog-aws with
`Fog::AWS::Storage::Real` service instead of `Fog::AWS::Storage::Mock`
because `Fog.mock!` was not called. Ensure that we use
`stub_uploads_object_storage` to prevent that from happening.
Closes https://gitlab.com/gitlab-org/gitlab-ce/issues/64083
|
|
|
|
It persist temporary personal snippets under
user/:id namespaces temporarily while creating
a upload record to track it. If an user gets removed
while it's still a tmp upload, it also gets removed.
If the tmp upload is sent, the upload gets moved to
personal_snippets/:id as before. The upload record
also gets updated to the new model type as well.
|
|
We've already migrated all the legacy artifacts to the new realm,
which is ci_job_artifacts table.
It's time to remove the old code base that is no longer used.
|
|
https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/24550 fixed the
case where the wrong path on disk was being searched, but it
inadvertently ommitted the `/uploads/-/system` prefix when rendering the
Markdown for personal snippet uploads when they were stored directly in
object storage.
A personal snippet path is stored using FileUploader#upload_path.
The format for the path:
Local storage: :random_hex/:filename.
Object storage: personal_snippet/:id/:random_hex/:filename.
upload_paths represent the possible paths for a given identifier,
which will vary depending on whether the file is stored in local or
object storage. upload_path should match an element in upload_paths.
base_dir represents the path seen by the user in Markdown, and it
should always be prefixed with uploads/-/system.
store_dirs represent the paths that are actually used on disk. For
object storage, this should omit the prefix /uploads/-/system.
For example, consider the requested path
/uploads/-/system/personal_snippet/172/ff4ad5c2/file.png.
For local storage:
base_dir: uploads/-/system/personal_snippet/172
upload_path: ff4ad5c2/file.png
upload_paths: ["ff4ad5c2/file.png", "personal_snippet/172/ff4ad5c2/file.png"].
store_dirs: {1=>"uploads/-/system/personal_snippet/172/ff4ad5c2",
2=>"personal_snippet/172/ff4ad5c2"}
For object storage:
upload_path: personal_snippet/172/ff4ad5c2/file.png
Closes https://gitlab.com/gitlab-org/gitlab-ce/issues/61671
|
|
String#underscore isn't particularly slow, but it's possible for us to
call it many times in a users autocomplete request, with mostly-static
values ('User', 'Group', etc.). We can memoise this and save a
surprising amount of time (around 10% of the total request time in some
cases).
|
|
Unlike uploads that have been uploaded with Tempfile, the project
import/export archives are stored in a temporary cache directory and
remain there if:
1. Object storage is enabled
2. `move_to_store` is set to `true`.
CarrierWave will leave these files there until disk space runs out or a
clean step is run manually.
If `move_to_store` is set to `false`, CarrierWave will remove the files
after storing them. However, unlike a local file, with object storage,
the file is still copied, so setting `move_to_store` to `true`
doesn't buy us anything.
To ensure files are cleaned up, we can just inherit from the
GitlabUploader implementation of `move_to_store`, which returns `true`
if it's a local file, `false` otherwise.
Closes https://gitlab.com/gitlab-org/gitlab-ce/issues/60656
|
|
We've seen a significant performance penalty when using
`BatchLoader#__replace_with!`. This defines methods on the batch loader
that proxy to the 'real' object using send. The alternative is
`method_missing`, which is slower. However, we've noticed that
`method_missing` can be faster if:
1. The objects being loaded have a large interface.
2. We don't call too many methods on the loaded object.
Avatar uploads meet both criteria above, so let's use the newly-released
feature in https://github.com/exAspArk/batch-loader/pull/45.
Closes https://gitlab.com/gitlab-org/gitlab-ce/issues/60903
|
|
- rewords examples starting with 'should'
- rewords examples starting with 'it'
Note: I had to manually fixup "onlies" to "only"
|
|
With object storage enabled, calling `#filename` on an upload does this:
1. Call the `#filename` method on the CarrierWave object.
2. Generate the URL for that object.
3. If the uploader isn't public, do so by generating an authenticated
URL, including signing that request.
That's all correct behaviour, but for the case where we use `#filename`,
it's typically to generate a GitLab URL. That URL doesn't need to be
signed because we do our own auth.
Signing the URLs can be very expensive, especially in batch (say, we
need to get the avatar URLs for 150 users in one request). It's all
unnecessary work. If we used the `RecordsUploads` concern, we have
already recorded a `path` in the database. That `path` is actually
generated from CarrierWave's `#filename` at upload time, so we don't
need to recompute it - we can just use it and strip off the prefix if
it's available.
On a sample users autocomplete URL, at least 10% of the time before this
change went to signing URLs. After this change, we spend no time in URL
signing, and still get the correct results.
|
|
Set ACL of uploaded objects to not-public (fog-aws) or no ACL at all
(fog-google). Value is ignored by other supported backends (fog-aliyun,
fog-openstack, fog-rackspace)
This allows uploads to GCS with ACL support disabled.
|
|
Previously one could move any temp/ sub folder around.
|
|
Currently we pass temp file path to FileMover
|
|
|
|
Signed-off-by: Rémy Coutable <remy@rymai.me>
|
|
Previously, an HTTP request for
`/uploads/-/system/personal_snippet/:snippet_id/:hash/:filename` would
look for an uploader of `PersonalFileUploader` class and use
`PersonalFileUploader#upload_paths` to search the datbase for one of the
following paths:
1. `:hash/:filename`
2. `uploads/-/system/personal_snippet/:id/:hash/:filename`
However, when the upload were stored in object storage,
`PersonalFileUploader#store_dirs` stored the path as:
`personal_snippet/:snippet_id/:hash`
The extraneous `uploads/-/system` prefix prevented the path from being
matched, and uploads in object storage would return a 404 error. Uploads
in local storage would work fine.
To fix this, we set the `#base_dir` properly so that `#upload_paths`
generates the right value for object storage. Note that this also makes
`#store_dirs` do the right thing in `FileUploader`.
Closes https://gitlab.com/gitlab-org/gitlab-ce/issues/52595
|
|
Specifically, the `ListAllMyBuckets` permission.
This works if you know the directory exists.
See more:
* https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/23981
* https://stackoverflow.com/a/12288581/1992201
|
