diff options
| author | Patrick Steinhardt <psteinhardt@gitlab.com> | 2021-09-09 15:07:32 +0300 |
|---|---|---|
| committer | Patrick Steinhardt <psteinhardt@gitlab.com> | 2021-09-09 15:53:04 +0300 |
| commit | 6ebcf23f1aa77d21c98789a9823a6fde362a09c4 (patch) | |
| tree | 2eddd7995d09d614e75304d466038a070f551aac /Makefile | |
| parent | e7f5668a0d77fc4c1ba0d329ea8c8e758e73ff4f (diff) | |
Makefile: Apply Git patches to speed up fetches
For quite some time we're aware of the fact that mirror-fetches into
repositories with many refs are exceedingly slow. Most importantly, this
issue poses problems for our replication strategy where replication jobs
take so much time that replication targets are likely to be out of date
immediately after they have received a replication jobs because the
primary node has received additional mutators while the replication
target was fetching changes.
To address this problem, we have upstreamed a patch series into git.git
which speeds fetches up somewhat. Most importantly, this patch series
optimizes the way git-fetch(1) enumerates refs by making better use of
the commit-graph. The result is that mirror-fetches in the benchmarking
repository gitlab-org/gitlab have been sped up from originally 56s to
25s.
While it is unlikely that this speedup alone will fix our replication
issue, it is definitely an important step towards improving the
situation.
Changelog: performance
Diffstat (limited to 'Makefile')
| -rw-r--r-- | Makefile | 19 |
1 files changed, 18 insertions, 1 deletions
@@ -125,13 +125,30 @@ ifeq ($(origin GIT_PATCHES),undefined) GIT_PATCHES += 0005-commit-graph-split-out-function-to-search-commit-pos.patch GIT_PATCHES += 0006-revision-avoid-hitting-packfiles-when-commits-are-in.patch + # Due to a bug, fetches with `--quiet` were slower than those without + # because Git formatted each reference into the output buffer even though + # it wasn't used. This has been merged into next via 2440a8a2aa (Merge + # branch 'ps/fetch-omit-formatting-under-quiet' into next, 2021-09-01) + GIT_PATCHES += 0007-fetch-skip-formatting-updated-refs-with-quiet.patch + + # This patch set speeds up fetches, most importantly by making better use + # of the commit graph. They have been merged into next via 99f865125d + # (Merge branch 'ps/fetch-optim' into next, 2021-09-08). + GIT_PATCHES += 0008-fetch-speed-up-lookup-of-want-refs-via-commit-graph.patch + GIT_PATCHES += 0009-fetch-avoid-unpacking-headers-in-object-existence-ch.patch + GIT_PATCHES += 0010-connected-refactor-iterator-to-return-next-object-ID.patch + GIT_PATCHES += 0011-fetch-pack-optimize-loading-of-refs-via-commit-graph.patch + GIT_PATCHES += 0012-fetch-refactor-fetch-refs-to-be-more-extendable.patch + GIT_PATCHES += 0013-fetch-merge-fetching-and-consuming-refs.patch + GIT_PATCHES += 0014-fetch-avoid-second-connectivity-check-if-we-already-.patch + # This extra version has two intentions: first, it allows us to detect # capabilities of the command at runtime. Second, it helps admins to # discover which version is currently in use. As such, this version must be # incremented whenever a new patch is added above. When no patches exist, # then this should be undefined. Otherwise, it must be set to at least # `gl1` given that `0` is the "default" GitLab patch level. - GIT_EXTRA_VERSION := gl1 + GIT_EXTRA_VERSION := gl2 endif ifeq ($(origin GIT_BUILD_OPTIONS),undefined) |