diff options
author | GitLab Bot <gitlab-bot@gitlab.com> | 2020-04-01 21:07:56 +0300 |
---|---|---|
committer | GitLab Bot <gitlab-bot@gitlab.com> | 2020-04-01 21:07:56 +0300 |
commit | 33aa02e7a38d8dfc5e470dd5d776c8d4ce5b2dd5 (patch) | |
tree | 8cd2bc4711d3a017d839760c7fbea267e2ba4951 /doc/administration/lfs | |
parent | 1219a9dce91f4edbc135dfc08299b4122b4825a8 (diff) |
Add latest changes from gitlab-org/gitlab@master
Diffstat (limited to 'doc/administration/lfs')
-rw-r--r-- | doc/administration/lfs/img/lfs-icon.png | bin | 4317 -> 0 bytes | |||
-rw-r--r-- | doc/administration/lfs/lfs_administration.md | 2 | ||||
-rw-r--r-- | doc/administration/lfs/manage_large_binaries_with_git_lfs.md | 272 | ||||
-rw-r--r-- | doc/administration/lfs/migrate_from_git_annex_to_git_lfs.md | 255 |
4 files changed, 6 insertions, 523 deletions
diff --git a/doc/administration/lfs/img/lfs-icon.png b/doc/administration/lfs/img/lfs-icon.png Binary files differdeleted file mode 100644 index eef9a14187a..00000000000 --- a/doc/administration/lfs/img/lfs-icon.png +++ /dev/null diff --git a/doc/administration/lfs/lfs_administration.md b/doc/administration/lfs/lfs_administration.md index a7ff624e1c6..10ff15b1ff4 100644 --- a/doc/administration/lfs/lfs_administration.md +++ b/doc/administration/lfs/lfs_administration.md @@ -4,7 +4,7 @@ disqus_identifier: 'https://docs.gitlab.com/ee/workflow/lfs/lfs_administration.h # GitLab Git LFS Administration -Documentation on how to use Git LFS are under [Managing large binary files with Git LFS doc](manage_large_binaries_with_git_lfs.md). +Documentation on how to use Git LFS are under [Managing large binary files with Git LFS doc](../../topics/git/lfs/index.md). ## Requirements diff --git a/doc/administration/lfs/manage_large_binaries_with_git_lfs.md b/doc/administration/lfs/manage_large_binaries_with_git_lfs.md index 27cf27fddc9..4656bccf5e6 100644 --- a/doc/administration/lfs/manage_large_binaries_with_git_lfs.md +++ b/doc/administration/lfs/manage_large_binaries_with_git_lfs.md @@ -1,273 +1,5 @@ --- -disqus_identifier: 'https://docs.gitlab.com/ee/workflow/lfs/manage_large_binaries_with_git_lfs.html' +redirect_to: '../../topics/git/lfs/index.md' --- -# Git LFS - -Managing large files such as audio, video and graphics files has always been one -of the shortcomings of Git. The general recommendation is to not have Git repositories -larger than 1GB to preserve performance. - -![Git LFS tracking status](img/lfs-icon.png) - -An LFS icon is shown on files tracked by Git LFS to denote if a file is stored -as a blob or as an LFS pointer. - -## How it works - -Git LFS client talks with the GitLab server over HTTPS. It uses HTTP Basic Authentication -to authorize client requests. Once the request is authorized, Git LFS client receives -instructions from where to fetch or where to push the large file. - -## GitLab server configuration - -Documentation for GitLab instance administrators is under [LFS administration doc](lfs_administration.md). - -## Requirements - -- Git LFS is supported in GitLab starting with version 8.2 -- Git LFS must be enabled under project settings -- [Git LFS client](https://git-lfs.github.com) version 1.0.1 and up - -## Known limitations - -- Git LFS v1 original API is not supported since it was deprecated early in LFS - development -- When SSH is set as a remote, Git LFS objects still go through HTTPS -- Any Git LFS request will ask for HTTPS credentials to be provided so a good Git - credentials store is recommended -- Git LFS always assumes HTTPS so if you have GitLab server on HTTP you will have - to add the URL to Git config manually (see [troubleshooting](#troubleshooting)) - -NOTE: **Note:** -With 8.12 GitLab added LFS support to SSH. The Git LFS communication -still goes over HTTP, but now the SSH client passes the correct credentials -to the Git LFS client, so no action is required by the user. - -## Using Git LFS - -Lets take a look at the workflow when you need to check large files into your Git -repository with Git LFS. For example, if you want to upload a very large file and -check it into your Git repository: - -```shell -git clone git@gitlab.example.com:group/project.git -git lfs install # initialize the Git LFS project -git lfs track "*.iso" # select the file extensions that you want to treat as large files -``` - -Once a certain file extension is marked for tracking as a LFS object you can use -Git as usual without having to redo the command to track a file with the same extension: - -```shell -cp ~/tmp/debian.iso ./ # copy a large file into the current directory -git add . # add the large file to the project -git commit -am "Added Debian iso" # commit the file meta data -git push origin master # sync the git repo and large file to the GitLab server -``` - -**Make sure** that `.gitattributes` is tracked by Git. Otherwise Git -LFS will not be working properly for people cloning the project: - -```shell -git add .gitattributes -``` - -Cloning the repository works the same as before. Git automatically detects the -LFS-tracked files and clones them via HTTP. If you performed the `git clone` -command with a SSH URL, you have to enter your GitLab credentials for HTTP -authentication. - -```shell -git clone git@gitlab.example.com:group/project.git -``` - -If you already cloned the repository and you want to get the latest LFS object -that are on the remote repository, eg. for a branch from origin: - -```shell -git lfs fetch origin master -``` - -### Migrate an existing repo to Git LFS - -Read the documentation on how to [migrate an existing Git repo with Git LFS](../../topics/git/migrate_to_git_lfs/index.md). - -### Removing objects from LFS - -To remove objects from LFS: - -1. Use [BFG-Cleaner](../../user/project/repository/reducing_the_repo_size_using_git.md#using-the-bfg-repo-cleaner) or [filter-branch](../../user/project/repository/reducing_the_repo_size_using_git.md#using-git-filter-branch) to remove the objects from the repository. -1. Delete the relevant LFS lines for the objects you have removed from your `.gitattributes` file and commit those changes. - -## File Locking - -> [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/issues/35856) in GitLab 10.5. - -The first thing to do before using File Locking is to tell Git LFS which -kind of files are lockable. The following command will store PNG files -in LFS and flag them as lockable: - -```shell -git lfs track "*.png" --lockable -``` - -After executing the above command a file named `.gitattributes` will be -created or updated with the following content: - -```shell -*.png filter=lfs diff=lfs merge=lfs -text lockable -``` - -You can also register a file type as lockable without using LFS -(In order to be able to lock/unlock a file you need a remote server that implements the LFS File Locking API), -in order to do that you can edit the `.gitattributes` file manually: - -```shell -*.pdf lockable -``` - -After a file type has been registered as lockable, Git LFS will make -them readonly on the file system automatically. This means you will -need to lock the file before editing it. - -### Managing Locked Files - -Once you're ready to edit your file you need to lock it first: - -```shell -git lfs lock images/banner.png -Locked images/banner.png -``` - -This will register the file as locked in your name on the server: - -```shell -git lfs locks -images/banner.png joe ID:123 -``` - -Once you have pushed your changes, you can unlock the file so others can -also edit it: - -```shell -git lfs unlock images/banner.png -``` - -You can also unlock by id: - -```shell -git lfs unlock --id=123 -``` - -If for some reason you need to unlock a file that was not locked by you, -you can use the `--force` flag as long as you have a `maintainer` access on -the project: - -```shell -git lfs unlock --id=123 --force -``` - -## Troubleshooting - -### error: Repository or object not found - -There are a couple of reasons why this error can occur: - -- You don't have permissions to access certain LFS object - -Check if you have permissions to push to the project or fetch from the project. - -- Project is not allowed to access the LFS object - -LFS object you are trying to push to the project or fetch from the project is not -available to the project anymore. Probably the object was removed from the server. - -- Local Git repository is using deprecated LFS API - -### Invalid status for `<url>` : 501 - -Git LFS will log the failures into a log file. -To view this log file, while in project directory: - -```shell -git lfs logs last -``` - -If the status `error 501` is shown, it is because: - -- Git LFS is not enabled in project settings. Check your project settings and - enable Git LFS. - -- Git LFS support is not enabled on the GitLab server. Check with your GitLab - administrator why Git LFS is not enabled on the server. See - [LFS administration documentation](lfs_administration.md) for instructions - on how to enable LFS support. - -- Git LFS client version is not supported by GitLab server. Check your Git LFS - version with `git lfs version`. Check the Git config of the project for traces - of deprecated API with `git lfs -l`. If `batch = false` is set in the config, - remove the line and try to update your Git LFS client. Only version 1.0.1 and - newer are supported. - -### getsockopt: connection refused - -If you push a LFS object to a project and you receive an error similar to: -`Post <URL>/info/lfs/objects/batch: dial tcp IP: getsockopt: connection refused`, -the LFS client is trying to reach GitLab through HTTPS. However, your GitLab -instance is being served on HTTP. - -This behaviour is caused by Git LFS using HTTPS connections by default when a -`lfsurl` is not set in the Git config. - -To prevent this from happening, set the lfs url in project Git config: - -```shell -git config --add lfs.url "http://gitlab.example.com/group/project.git/info/lfs" -``` - -### Credentials are always required when pushing an object - -NOTE: **Note:** -With 8.12 GitLab added LFS support to SSH. The Git LFS communication -still goes over HTTP, but now the SSH client passes the correct credentials -to the Git LFS client, so no action is required by the user. - -Given that Git LFS uses HTTP Basic Authentication to authenticate the user pushing -the LFS object on every push for every object, user HTTPS credentials are required. - -By default, Git has support for remembering the credentials for each repository -you use. This is described in [Git credentials man pages](https://git-scm.com/docs/gitcredentials). - -For example, you can tell Git to remember the password for a period of time in -which you expect to push the objects: - -```shell -git config --global credential.helper 'cache --timeout=3600' -``` - -This will remember the credentials for an hour after which Git operations will -require re-authentication. - -If you are using OS X you can use `osxkeychain` to store and encrypt your credentials. -For Windows, you can use `wincred` or Microsoft's [Git Credential Manager for Windows](https://github.com/Microsoft/Git-Credential-Manager-for-Windows/releases). - -More details about various methods of storing the user credentials can be found -on [Git Credential Storage documentation](https://git-scm.com/book/en/v2/Git-Tools-Credential-Storage). - -### LFS objects are missing on push - -GitLab checks files to detect LFS pointers on push. If LFS pointers are detected, GitLab tries to verify that those files already exist in LFS on GitLab. - -Verify that LFS is installed locally and consider a manual push with `git lfs push --all`. - -If you are storing LFS files outside of GitLab you can disable LFS on the project by setting `lfs_enabled: false` with the [projects API](../../api/projects.md#edit-project). - -### Hosting LFS objects externally - -It is possible to host LFS objects externally by setting a custom LFS url with `git config -f .lfsconfig lfs.url https://example.com/<project>.git/info/lfs`. - -You might choose to do this if you are using an appliance like a Sonatype Nexus to store LFS data. If you choose to use an external LFS store, -GitLab will not be able to verify LFS objects which means that pushes will fail if you have GitLab LFS support enabled. - -To stop push failure, LFS support can be disabled in the [Project settings](../../user/project/settings/index.md). This means you will lose GitLab LFS value-adds (Verifying LFS objects, UI integration for LFS). +This document was moved to [another location](../../topics/git/lfs/index.md). diff --git a/doc/administration/lfs/migrate_from_git_annex_to_git_lfs.md b/doc/administration/lfs/migrate_from_git_annex_to_git_lfs.md index bfe18f697bf..dd98a50e353 100644 --- a/doc/administration/lfs/migrate_from_git_annex_to_git_lfs.md +++ b/doc/administration/lfs/migrate_from_git_annex_to_git_lfs.md @@ -1,254 +1,5 @@ -# Migration guide from Git Annex to Git LFS - ->**Note:** -Git Annex support [has been removed][issue-remove-annex] in GitLab Enterprise -Edition 9.0 (2017/03/22). - -Both [Git Annex][] and [Git LFS][] are tools to manage large files in Git. - -## History - -Git Annex [was introduced in GitLab Enterprise Edition 7.8][post-3], at a time -where Git LFS didn't yet exist. A few months later, GitLab brought support for -Git LFS in [GitLab 8.2][post-2] and is available for both Community and -Enterprise editions. - -## Differences between Git Annex and Git LFS - -Some items below are general differences between the two protocols and some are -ones that GitLab developed. - -- Git Annex works only through SSH, whereas Git LFS works both with SSH and HTTPS - (SSH support was added in GitLab 8.12). -- Annex files are stored in a sub-directory of the normal repositories, whereas - LFS files are stored outside of the repositories in a place you can define. -- Git Annex requires a more complex setup, but has much more options than Git - LFS. You can compare the commands each one offers by running `man git-annex` - and `man git-lfs`. -- Annex files cannot be browsed directly in GitLab's interface, whereas LFS - files can. - -## Migration steps - ->**Note:** -Since Git Annex files are stored in a sub-directory of the normal repositories -(`.git/annex/objects`) and LFS files are stored outside of the repositories, -they are not compatible as they are using a different scheme. Therefore, the -migration has to be done manually per repository. - -There are basically two steps you need to take in order to migrate from Git -Annex to Git LFS. - -### TL; DR - -If you know what you are doing and want to skip the reading, this is what you -need to do (we assume you have [git-annex enabled](../git_annex.md#using-gitlab-git-annex) in your -repository and that you have made backups in case something goes wrong). -Fire up a terminal, navigate to your Git repository and: - -1. Disable `git-annex`: - - ```shell - git annex sync --content - git annex direct - git annex uninit - git annex indirect - ``` - -1. Enable `git-lfs`: - - ```shell - git lfs install - git lfs track <files> - git add . - git commit -m "commit message" - git push - ``` - -### Disabling Git Annex in your repo - -Before changing anything, make sure you have a backup of your repository first. -There are a couple of ways to do that, but you can simply clone it to another -local path and maybe push it to GitLab if you want a remote backup as well. -Here you'll find a guide on -[how to back up a **git-annex** repository to an external hard drive][bkp-ext-drive]. - -Since Annex files are stored as objects with symlinks and cannot be directly -modified, we need to first remove those symlinks. - -NOTE: **Note:** -Make sure the you read about the [`direct` mode][annex-direct] as it contains -useful information that may fit in your use case. Note that `annex direct` is -deprecated in Git Annex version 6, so you may need to upgrade your repository -if the server also has Git Annex 6 installed. Read more in the -[Git Annex troubleshooting tips](../git_annex.md#troubleshooting-tips) section. - -1. Backup your repository - - ```shell - cd repository - git annex sync --content - cd .. - git clone repository repository-backup - cd repository-backup - git annex get - cd .. - ``` - -1. Use `annex direct`: - - ```shell - cd repository - git annex direct - ``` - - The output should be similar to this: - - ```shell - commit - On branch master - Your branch is up-to-date with 'origin/master'. - nothing to commit, working tree clean - ok - direct debian.iso ok - direct ok - ``` - -1. Disable Git Annex with [`annex uninit`][uninit]: - - ```shell - git annex uninit - ``` - - The output should be similar to this: - - ```shell - unannex debian.iso ok - Deleted branch git-annex (was 2534d2c). - ``` - - This will `unannex` every file in the repository, leaving the original files. - -1. Switch back to `indirect` mode: - - ```shell - git annex indirect - ``` - - The output should be similar to this: - - ```shell - (merging origin/git-annex into git-annex...) - (recording state in git...) - commit (recording state in git...) - - ok - (recording state in git...) - [master fac3194] commit before switching to indirect mode - 1 file changed, 1 deletion(-) - delete mode 120000 alpine-virt-3.4.4-x86_64.iso - ok - indirect ok - ok - ``` - +--- +redirect_to: '../../topics/git/lfs/migrate_from_git_annex_to_git_lfs.md' --- -At this point, you have two options. Either add, commit and push the files -directly back to GitLab or switch to Git LFS. We will tackle the LFS switch in -the next section. - -### Enabling Git LFS in your repo - -Git LFS is enabled by default on all GitLab products (GitLab CE, GitLab EE, -GitLab.com), therefore, you don't need to do anything server-side. - -1. First, make sure you have `git-lfs` installed locally: - - ```shell - git lfs help - ``` - - If the terminal doesn't prompt you with a full response on `git-lfs` commands, - [install the Git LFS client][install-lfs] first. - -1. Inside the repo, run the following command to initiate LFS: - - ```shell - git lfs install - ``` - -1. Enable `git-lfs` for the group of files you want to track. You - can track specific files, all files containing the same extension, or an - entire directory: - - ```shell - git lfs track images/01.png # per file - git lfs track **/*.png # per extension - git lfs track images/ # per directory - ``` - - Once you do that, run `git status` and you'll see `.gitattributes` added - to your repo. It collects all file patterns that you chose to track via - `git-lfs`. - -1. Add the files, commit and push them to GitLab: - - ```shell - git add . - git commit -m "commit message" - git push - ``` - - If your remote is set up with HTTP, you will be asked to enter your login - credentials. If you have [2FA enabled](../../user/profile/account/two_factor_authentication.md), make sure to use a - [personal access token](../../user/profile/account/two_factor_authentication.md#personal-access-tokens) - instead of your password. - -## Removing the Git Annex branches - -After the migration finishes successfully, you can remove all `git-annex` -related branches from your repository. - -On GitLab, navigate to your project's **Repository ➔ Branches** and delete all -branches created by Git Annex: `git-annex`, and all under `synced/`. - -![repository branches](img/git-annex-branches.png) - -You can also do this on the command line with: - -```shell -git branch -d synced/master -git branch -d synced/git-annex -git push origin :synced/master -git push origin :synced/git-annex -git push origin :git-annex -git remote prune origin -``` - -If there are still some Annex objects inside your repository (`.git/annex/`) -or references inside `.git/config`, run `annex uninit` again: - -```shell -git annex uninit -``` - -## Further Reading - -- (Blog Post) [Getting Started with Git FLS][post-1] -- (Blog Post) [Announcing LFS Support in GitLab][post-2] -- (Blog Post) [GitLab Annex Solves the Problem of Versioning Large Binaries with Git][post-3] -- (GitLab Docs) [Git Annex](../git_annex.md) -- (GitLab Docs) [Git LFS](manage_large_binaries_with_git_lfs.md) - -[annex-direct]: https://git-annex.branchable.com/direct_mode/ -[bkp-ext-drive]: https://www.thomas-krenn.com/en/wiki/Git-annex_Repository_on_an_External_Hard_Drive -[Git Annex]: http://git-annex.branchable.com/ -[Git LFS]: https://git-lfs.github.com/ -[install-lfs]: https://git-lfs.github.com/ -[issue-remove-annex]: https://gitlab.com/gitlab-org/gitlab/issues/1648 -[lfs-track]: https://about.gitlab.com/blog/2017/01/30/getting-started-with-git-lfs-tutorial/#tracking-files-with-lfs -[post-1]: https://about.gitlab.com/blog/2017/01/30/getting-started-with-git-lfs-tutorial/ -[post-2]: https://about.gitlab.com/blog/2015/11/23/announcing-git-lfs-support-in-gitlab/ -[post-3]: https://about.gitlab.com/blog/2015/02/17/gitlab-annex-solves-the-problem-of-versioning-large-binaries-with-git/ -[uninit]: https://git-annex.branchable.com/git-annex-uninit/ +This document was moved to [another location](../../topics/git/lfs/migrate_from_git_annex_to_git_lfs.md). |