From 9dc93a4519d9d5d7be48ff274127136236a3adb3 Mon Sep 17 00:00:00 2001 From: GitLab Bot Date: Tue, 20 Apr 2021 23:50:22 +0000 Subject: Add latest changes from gitlab-org/gitlab@13-11-stable-ee --- doc/administration/nfs.md | 61 ++++++++++++++++++++++++++++++++++------------- 1 file changed, 45 insertions(+), 16 deletions(-) (limited to 'doc/administration/nfs.md') diff --git a/doc/administration/nfs.md b/doc/administration/nfs.md index 59b18f7a74a..52948732766 100644 --- a/doc/administration/nfs.md +++ b/doc/administration/nfs.md @@ -14,17 +14,19 @@ Pages](https://gitlab.com/gitlab-org/gitlab-pages/-/issues/196). For data objects such as LFS, Uploads, Artifacts, etc., an [Object Storage service](object_storage.md) is recommended over NFS where possible, due to better performance. -WARNING: -From GitLab 13.0, using NFS for Git repositories is deprecated. -From GitLab 14.0, technical support for NFS for Git repositories -will no longer be provided. Upgrade to [Gitaly Cluster](gitaly/praefect.md) -as soon as possible. - File system performance can impact overall GitLab performance, especially for actions that read or write to Git repositories. For steps you can use to test file system performance, see [File system Performance Benchmarking](operations/filesystem_benchmarking.md). +## Gitaly and NFS deprecation + +WARNING: +From GitLab 14.0, enhancements and bug fixes for NFS for Git repositories will no longer be +considered and customer technical support will be considered out of scope. +[Read more about Gitaly and NFS](gitaly/index.md#nfs-deprecation-notice) and +[the correct mount options to use](#upgrade-to-gitaly-cluster-or-disable-caching-if-experiencing-data-loss). + ## Known kernel version incompatibilities RedHat Enterprise Linux (RHEL) and CentOS v7.7 and v7.8 ship with kernel @@ -169,11 +171,11 @@ apt-get install nfs-common Here is an example snippet to add to `/etc/fstab`: ```plaintext -10.1.0.1:/var/opt/gitlab/.ssh /var/opt/gitlab/.ssh nfs4 defaults,vers=4.1,hard,rsize=1048576,wsize=1048576,noatime,nofail,lookupcache=positive 0 2 -10.1.0.1:/var/opt/gitlab/gitlab-rails/uploads /var/opt/gitlab/gitlab-rails/uploads nfs4 defaults,vers=4.1,hard,rsize=1048576,wsize=1048576,noatime,nofail,lookupcache=positive 0 2 -10.1.0.1:/var/opt/gitlab/gitlab-rails/shared /var/opt/gitlab/gitlab-rails/shared nfs4 defaults,vers=4.1,hard,rsize=1048576,wsize=1048576,noatime,nofail,lookupcache=positive 0 2 -10.1.0.1:/var/opt/gitlab/gitlab-ci/builds /var/opt/gitlab/gitlab-ci/builds nfs4 defaults,vers=4.1,hard,rsize=1048576,wsize=1048576,noatime,nofail,lookupcache=positive 0 2 -10.1.0.1:/var/opt/gitlab/git-data /var/opt/gitlab/git-data nfs4 defaults,vers=4.1,hard,rsize=1048576,wsize=1048576,noatime,nofail,lookupcache=positive 0 2 +10.1.0.1:/var/opt/gitlab/.ssh /var/opt/gitlab/.ssh nfs4 defaults,vers=4.1,hard,rsize=1048576,wsize=1048576,noatime,nofail,_netdev,lookupcache=positive 0 2 +10.1.0.1:/var/opt/gitlab/gitlab-rails/uploads /var/opt/gitlab/gitlab-rails/uploads nfs4 defaults,vers=4.1,hard,rsize=1048576,wsize=1048576,noatime,nofail,_netdev,lookupcache=positive 0 2 +10.1.0.1:/var/opt/gitlab/gitlab-rails/shared /var/opt/gitlab/gitlab-rails/shared nfs4 defaults,vers=4.1,hard,rsize=1048576,wsize=1048576,noatime,nofail,_netdev,lookupcache=positive 0 2 +10.1.0.1:/var/opt/gitlab/gitlab-ci/builds /var/opt/gitlab/gitlab-ci/builds nfs4 defaults,vers=4.1,hard,rsize=1048576,wsize=1048576,noatime,nofail,_netdev,lookupcache=positive 0 2 +10.1.0.1:/var/opt/gitlab/git-data /var/opt/gitlab/git-data nfs4 defaults,vers=4.1,hard,rsize=1048576,wsize=1048576,noatime,nofail,_netdev,lookupcache=positive 0 2 ``` You can view information and options set for each of the mounted NFS file @@ -187,6 +189,8 @@ Note there are several options that you should consider using: | `nofail` | Don't halt boot process waiting for this mount to become available | `lookupcache=positive` | Tells the NFS client to honor `positive` cache results but invalidates any `negative` cache results. Negative cache results cause problems with Git. Specifically, a `git push` can fail to register uniformly across all NFS clients. The negative cache causes the clients to 'remember' that the files did not exist previously. | `hard` | Instead of `soft`. [Further details](#soft-mount-option). +| `cto` | `cto` is the default option, which you should use. Do not use `nocto`. [Further details](#nocto-mount-option). +| `_netdev` | Wait to mount filesystem until network is online. See also the [`high_availability['mountpoint']`](https://docs.gitlab.com/omnibus/settings/configuration.html#only-start-omnibus-gitlab-services-after-a-given-file-system-is-mounted) option. #### `soft` mount option @@ -223,6 +227,25 @@ the mount point. Use `SIGKILL` (`kill -9`) to deal with hung processes. The `intr` option [stopped working in the 2.6 kernel](https://access.redhat.com/solutions/157873). +#### `nocto` mount option + +Do not use `nocto`. Instead, use `cto`, which is the default. + +When using `nocto`, the dentry cache is always used, up to `acdirmax` seconds (attribute cache time) from the time it's created. + +This results in stale dentry cache issues with multiple clients, where each client can see a different (cached) +version of a directory. + +From the [Linux man page](https://linux.die.net/man/5/nfs), the important parts: + +> If the nocto option is specified, the client uses a non-standard heuristic to determine when files on the server have changed. +> +> Using the nocto option may improve performance for read-only mounts, but should be used only if the data on the server changes only occasionally. + +We have noticed this behavior in an issue about [refs not found after a push](https://gitlab.com/gitlab-org/gitlab/-/issues/326066), +where newly added loose refs can be seen as missing on a different client with a local dentry cache, as +[described in this issue](https://gitlab.com/gitlab-org/gitlab/-/issues/326066#note_539436931). + ### A single NFS mount It's recommended to nest all GitLab data directories within a mount, that allows automatic @@ -346,12 +369,18 @@ sudo ufw allow from to any port nfs ### Upgrade to Gitaly Cluster or disable caching if experiencing data loss WARNING: -From GitLab 13.0, using NFS for Git repositories is deprecated. In GitLab 14.0, -support for NFS for Git repositories is scheduled to be removed. Upgrade to -[Gitaly Cluster](gitaly/praefect.md) as soon as possible. +From GitLab 13.0, using NFS for Git repositories is deprecated. +As of GitLab 14.0, NFS-related issues with Gitaly will no longer be addressed. Read +more about [Gitaly and NFS deprecation](gitaly/index.md#nfs-deprecation-notice). Customers and users have reported data loss on high-traffic repositories when using NFS for Git repositories. -For example, we have seen [inconsistent updates after a push](https://gitlab.com/gitlab-org/gitaly/-/issues/2589). The problem may be partially mitigated by adjusting caching using the following NFS client mount options: +For example, we have seen: + +- [Inconsistent updates after a push](https://gitlab.com/gitlab-org/gitaly/-/issues/2589). +- `git ls-remote` [returning the wrong (or no branches)](https://gitlab.com/gitlab-org/gitaly/-/issues/3083) +causing Jenkins to intermittently re-run all pipelines for a repository. + +The problem may be partially mitigated by adjusting caching using the following NFS client mount options: | Setting | Description | | ------- | ----------- | @@ -362,7 +391,7 @@ For example, we have seen [inconsistent updates after a push](https://gitlab.com WARNING: The `actimeo=0` and `noac` options both result in a significant reduction in performance, possibly leading to timeouts. You may be able to avoid timeouts and data loss using `actimeo=0` and `lookupcache=positive` _without_ `noac`, however -we expect the performance reduction will still be significant. As noted above, we strongly recommend upgrading to +we expect the performance reduction will still be significant. Upgrade to [Gitaly Cluster](gitaly/praefect.md) as soon as possible. ### Avoid using cloud-based file systems -- cgit v1.2.3