| Age | Commit message (Collapse) | Author |
|---|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
Queries for Upload should be scoped by model
See merge request gitlab/gitlabhq!3229
|
|
None of this code can be reached any more, so it can all be removed
|
|
|
|
Suggests to use a JSON structured log instead
Related to https://gitlab.com/gitlab-org/gitlab-ce/issues/54102
|
|
We don't need to find the filename from the remote URL
|
|
|
|
It persist temporary personal snippets under
user/:id namespaces temporarily while creating
a upload record to track it. If an user gets removed
while it's still a tmp upload, it also gets removed.
If the tmp upload is sent, the upload gets moved to
personal_snippets/:id as before. The upload record
also gets updated to the new model type as well.
|
|
We've already migrated all the legacy artifacts to the new realm,
which is ci_job_artifacts table.
It's time to remove the old code base that is no longer used.
|
|
https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/24550 fixed the
case where the wrong path on disk was being searched, but it
inadvertently ommitted the `/uploads/-/system` prefix when rendering the
Markdown for personal snippet uploads when they were stored directly in
object storage.
A personal snippet path is stored using FileUploader#upload_path.
The format for the path:
Local storage: :random_hex/:filename.
Object storage: personal_snippet/:id/:random_hex/:filename.
upload_paths represent the possible paths for a given identifier,
which will vary depending on whether the file is stored in local or
object storage. upload_path should match an element in upload_paths.
base_dir represents the path seen by the user in Markdown, and it
should always be prefixed with uploads/-/system.
store_dirs represent the paths that are actually used on disk. For
object storage, this should omit the prefix /uploads/-/system.
For example, consider the requested path
/uploads/-/system/personal_snippet/172/ff4ad5c2/file.png.
For local storage:
base_dir: uploads/-/system/personal_snippet/172
upload_path: ff4ad5c2/file.png
upload_paths: ["ff4ad5c2/file.png", "personal_snippet/172/ff4ad5c2/file.png"].
store_dirs: {1=>"uploads/-/system/personal_snippet/172/ff4ad5c2",
2=>"personal_snippet/172/ff4ad5c2"}
For object storage:
upload_path: personal_snippet/172/ff4ad5c2/file.png
Closes https://gitlab.com/gitlab-org/gitlab-ce/issues/61671
|
|
String#underscore isn't particularly slow, but it's possible for us to
call it many times in a users autocomplete request, with mostly-static
values ('User', 'Group', etc.). We can memoise this and save a
surprising amount of time (around 10% of the total request time in some
cases).
|
|
Unlike uploads that have been uploaded with Tempfile, the project
import/export archives are stored in a temporary cache directory and
remain there if:
1. Object storage is enabled
2. `move_to_store` is set to `true`.
CarrierWave will leave these files there until disk space runs out or a
clean step is run manually.
If `move_to_store` is set to `false`, CarrierWave will remove the files
after storing them. However, unlike a local file, with object storage,
the file is still copied, so setting `move_to_store` to `true`
doesn't buy us anything.
To ensure files are cleaned up, we can just inherit from the
GitlabUploader implementation of `move_to_store`, which returns `true`
if it's a local file, `false` otherwise.
Closes https://gitlab.com/gitlab-org/gitlab-ce/issues/60656
|
|
|
|
Prepares us for upgrade to Rails 5.2
|
|
Drop legacy artifacts usage as there are no leftovers
See merge request gitlab-org/gitlab-ce!24294
|
|
With object storage enabled, calling `#filename` on an upload does this:
1. Call the `#filename` method on the CarrierWave object.
2. Generate the URL for that object.
3. If the uploader isn't public, do so by generating an authenticated
URL, including signing that request.
That's all correct behaviour, but for the case where we use `#filename`,
it's typically to generate a GitLab URL. That URL doesn't need to be
signed because we do our own auth.
Signing the URLs can be very expensive, especially in batch (say, we
need to get the avatar URLs for 150 users in one request). It's all
unnecessary work. If we used the `RecordsUploads` concern, we have
already recorded a `path` in the database. That `path` is actually
generated from CarrierWave's `#filename` at upload time, so we don't
need to recompute it - we can just use it and strip off the prefix if
it's available.
On a sample users autocomplete URL, at least 10% of the time before this
change went to signing URLs. After this change, we spend no time in URL
signing, and still get the correct results.
|
|
Legacy artifacts have been correctly migrated to new place -
ci_job_artifacts. Now it's time to remove the related code, but before
that we should ensure it doesn't break anything by using feature flag.
|
|
Set ACL of uploaded objects to not-public (fog-aws) or no ACL at all
(fog-google). Value is ignored by other supported backends (fog-aliyun,
fog-openstack, fog-rackspace)
This allows uploads to GCS with ACL support disabled.
|
|
Previously one could move any temp/ sub folder around.
|
|
|
|
Previously, an HTTP request for
`/uploads/-/system/personal_snippet/:snippet_id/:hash/:filename` would
look for an uploader of `PersonalFileUploader` class and use
`PersonalFileUploader#upload_paths` to search the datbase for one of the
following paths:
1. `:hash/:filename`
2. `uploads/-/system/personal_snippet/:id/:hash/:filename`
However, when the upload were stored in object storage,
`PersonalFileUploader#store_dirs` stored the path as:
`personal_snippet/:snippet_id/:hash`
The extraneous `uploads/-/system` prefix prevented the path from being
matched, and uploads in object storage would return a 404 error. Uploads
in local storage would work fine.
To fix this, we set the `#base_dir` properly so that `#upload_paths`
generates the right value for object storage. Note that this also makes
`#store_dirs` do the right thing in `FileUploader`.
Closes https://gitlab.com/gitlab-org/gitlab-ce/issues/52595
|
|
`spec/features/uploads/user_uploads_file_to_note_spec.rb` was failing in
master because MySQL detected a deadlock when a DELETE and INSERT for
the same indexed item occurred within a transaction in the `uploads`
table. Due to InnoDB's next-key locking algorithm
(innodb_locks_unsafe_for_binlog in
https://dev.mysql.com/doc/refman/5.5/en/innodb-parameters.html), InnoDB
sets an exclusive lock for any of the indexed records it encounters, so
the INSERT will fail until the DELETE is committed.
To fix this, we just disable the transaction for MySQL and keep
it for PostgreSQL.
Closes https://gitlab.com/gitlab-org/gitlab-ce/issues/55161
|
