diff options
Diffstat (limited to 'doc/administration/lfs/index.md')
| -rw-r--r-- | doc/administration/lfs/index.md | 129 |
1 files changed, 59 insertions, 70 deletions
diff --git a/doc/administration/lfs/index.md b/doc/administration/lfs/index.md index d2f220e3795..3fe6a94ef13 100644 --- a/doc/administration/lfs/index.md +++ b/doc/administration/lfs/index.md @@ -56,7 +56,7 @@ In `config/gitlab.yml`: ## Storing LFS objects in remote object storage You can store LFS objects in remote object storage. This allows you -to offload reads and writes to the local disk, and free up disk space significantly. +to reduce reads and writes to the local disk, and free up disk space significantly. GitLab is tightly integrated with `Fog`, so you can refer to its [documentation](http://fog.io/about/provider_documentation.html) to check which storage services can be integrated with GitLab. You can also use external object storage in a private local network. For example, @@ -98,32 +98,6 @@ See [the available connection settings for different providers](../object_storag Here is a configuration example with S3. -### Manual uploading to an object storage - -There are two ways to manually do the same thing as automatic uploading (described above). - -**Option 1: Rake task** - -```shell -gitlab-rake gitlab:lfs:migrate -``` - -**Option 2: Rails console** - -Log into the Rails console: - -```shell -sudo gitlab-rails console -``` - -Upload LFS files manually - -```ruby -LfsObject.where(file_store: [nil, 1]).find_each do |lfs_object| - lfs_object.file.migrate!(ObjectStorage::Store::REMOTE) if lfs_object.file.file.exists? -end -``` - ### S3 for Omnibus installations On Omnibus GitLab installations, the settings are prefixed by `lfs_object_store_`: @@ -146,32 +120,10 @@ On Omnibus GitLab installations, the settings are prefixed by `lfs_object_store_ ``` 1. Save the file, and then [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. -1. Migrate any existing local LFS objects to the object storage: - - ```shell - gitlab-rake gitlab:lfs:migrate - ``` - - This migrates existing LFS objects to object storage. New LFS objects +1. [Migrate any existing local LFS objects to the object storage](#migrating-to-object-storage). + New LFS objects are forwarded to object storage unless `gitlab_rails['lfs_object_store_background_upload']` and `gitlab_rails['lfs_object_store_direct_upload']` is set to `false`. -1. (Optional) Verify all files migrated properly. - From [PostgreSQL console](https://docs.gitlab.com/omnibus/settings/database.html#connecting-to-the-bundled-postgresql-database) - (`sudo gitlab-psql -d gitlabhq_production`) verify `objectstg` below (where `file_store=2`) has count of all artifacts: - - ```shell - gitlabhq_production=# SELECT count(*) AS total, sum(case when file_store = '1' then 1 else 0 end) AS filesystem, sum(case when file_store = '2' then 1 else 0 end) AS objectstg FROM lfs_objects; - - total | filesystem | objectstg - ------+------------+----------- - 2409 | 0 | 2409 - ``` - - Verify no files on disk in `artifacts` folder: - - ```shell - sudo find /var/opt/gitlab/gitlab-rails/shared/lfs-objects -type f | grep -v tmp/cache | wc -l - ``` ### S3 for installations from source @@ -199,31 +151,68 @@ For source installations the settings are nested under `lfs:` and then ``` 1. Save the file, and then [restart GitLab](../restart_gitlab.md#installations-from-source) for the changes to take effect. -1. Migrate any existing local LFS objects to the object storage: +1. [Migrate any existing local LFS objects to the object storage](#migrating-to-object-storage). + New LFS objects + are forwarded to object storage unless + `background_upload` and `direct_upload` is set to `false`. - ```shell - sudo -u git -H bundle exec rake gitlab:lfs:migrate RAILS_ENV=production - ``` +### Migrating to object storage - This migrates existing LFS objects to object storage. New LFS objects - are forwarded to object storage unless `background_upload` and `direct_upload` is set to - `false`. -1. (Optional) Verify all files migrated properly. - From PostgreSQL console (`sudo -u git -H psql -d gitlabhq_production`) verify `objectstg` below (where `file_store=2`) has count of all artifacts: +**Option 1: Rake task** - ```shell - gitlabhq_production=# SELECT count(*) AS total, sum(case when file_store = '1' then 1 else 0 end) AS filesystem, sum(case when file_store = '2' then 1 else 0 end) AS objectstg FROM lfs_objects; +After [configuring the object storage](#storing-lfs-objects-in-remote-object-storage), use the following task to +migrate existing LFS objects from the local storage to the remote storage. +The processing is done in a background worker and requires **no downtime**. - total | filesystem | objectstg - ------+------------+----------- - 2409 | 0 | 2409 - ``` +For Omnibus GitLab: + +```shell +sudo gitlab-rake "gitlab:lfs:migrate" +``` - Verify no files on disk in `artifacts` folder: +For installations from source: - ```shell - sudo find /var/opt/gitlab/gitlab-rails/shared/lfs-objects -type f | grep -v tmp/cache | wc -l - ``` +```shell +RAILS_ENV=production sudo -u git -H bundle exec rake gitlab:lfs:migrate +``` + +You can optionally track progress and verify that all packages migrated successfully using the +[PostgreSQL console](https://docs.gitlab.com/omnibus/settings/database.html#connecting-to-the-bundled-postgresql-database): + +- `sudo gitlab-rails dbconsole` for Omnibus GitLab instances. +- `sudo -u git -H psql -d gitlabhq_production` for source-installed instances. + +Verify `objectstg` below (where `store=2`) has count of all LFS objects: + +```shell +gitlabhq_production=# SELECT count(*) AS total, sum(case when file_store = '1' then 1 else 0 end) AS filesystem, sum(case when file_store = '2' then 1 else 0 end) AS objectstg FROM lfs_objects; + +total | filesystem | objectstg +------+------------+----------- + 2409 | 0 | 2409 +``` + +Verify that there are no files on disk in the `objects` folder: + +```shell +sudo find /var/opt/gitlab/gitlab-rails/shared/lfs-objects -type f | grep -v tmp | wc -l +``` + +**Option 2: Rails console** + +Log into the Rails console: + +```shell +sudo gitlab-rails console +``` + +Upload LFS files manually + +```ruby +LfsObject.where(file_store: [nil, 1]).find_each do |lfs_object| + lfs_object.file.migrate!(ObjectStorage::Store::REMOTE) if lfs_object.file.file.exists? +end +``` ### Migrating back to local storage |