| Age | Commit message (Collapse) | Author |
|---|
|
Soon we will be looking to implement server-side backups. To do this,
all the RPCs used for creating a backup will need to be converted to
localrepo.
|
|
tools/gotestsum: Update module gotest.tools/gotestsum to v1.10.0
See merge request https://gitlab.com/gitlab-org/gitaly/-/merge_requests/5649
Merged-by: Toon Claes <toon@gitlab.com>
Approved-by: Toon Claes <toon@gitlab.com>
Co-authored-by: GitLab Renovate Bot <gitlab-bot@gitlab.com>
|
|
tools/goimports: Update module golang.org/x/tools to v0.8.0
See merge request https://gitlab.com/gitlab-org/gitaly/-/merge_requests/5643
Merged-by: Toon Claes <toon@gitlab.com>
Approved-by: Sami Hiltunen <shiltunen@gitlab.com>
Co-authored-by: GitLab Renovate Bot <gitlab-bot@gitlab.com>
|
|
praefect: Start switch to the new cli implementation
See merge request https://gitlab.com/gitlab-org/gitaly/-/merge_requests/5616
Merged-by: Toon Claes <toon@gitlab.com>
Approved-by: John Cai <jcai@gitlab.com>
Approved-by: Toon Claes <toon@gitlab.com>
Reviewed-by: Toon Claes <toon@gitlab.com>
Reviewed-by: Pavlo Strokov <pstrokov@gitlab.com>
Co-authored-by: Pavlo Strokov <pstrokov@gitlab.com>
|
|
tools: Rewrite Ruby scripts in Go
See merge request https://gitlab.com/gitlab-org/gitaly/-/merge_requests/5635
Merged-by: Will Chandler <wchandler@gitlab.com>
Approved-by: Will Chandler <wchandler@gitlab.com>
Reviewed-by: Will Chandler <wchandler@gitlab.com>
Reviewed-by: John Cai <jcai@gitlab.com>
Co-authored-by: Toon Claes <toon@gitlab.com>
|
|
|
|
praefect: Use localhost listener for DB in test
See merge request https://gitlab.com/gitlab-org/gitaly/-/merge_requests/5641
Merged-by: Will Chandler <wchandler@gitlab.com>
Approved-by: Will Chandler <wchandler@gitlab.com>
Reviewed-by: Will Chandler <wchandler@gitlab.com>
Co-authored-by: Toon Claes <toon@gitlab.com>
|
|
tools/golangci-lint: Update module golang.org/x/tools to v0.8.0
See merge request https://gitlab.com/gitlab-org/gitaly/-/merge_requests/5644
Merged-by: Will Chandler <wchandler@gitlab.com>
Approved-by: Patrick Steinhardt <psteinhardt@gitlab.com>
Approved-by: Will Chandler <wchandler@gitlab.com>
Co-authored-by: GitLab Renovate Bot <gitlab-bot@gitlab.com>
|
|
Ruby isn't used no more, so we can use a build image that has no Ruby.
Issue: https://gitlab.com/gitlab-org/gitaly/-/issues/4636
|
|
We've been using a small Ruby oneliner to get panics from the full
output of the test suite. But we're removing all use of Ruby, so we
replace it with a small Go tool.
|
|
Instead of using Ruby, build the praefect config file with `envsubst`.
|
|
Replace the Ruby script _support/test-boot with a small tool written in
Go.
Issue: https://gitlab.com/gitlab-org/gitaly/-/issues/4636
|
|
Praefect has a bunch of sub-commands besides its main service function.
All of them are managed in a very simple, but inflexible way. Also,
the implementation differs from what is used in the Gitaly. To
standardize approach of implementing cli Praefect is refactored to
utilise github.com/urfave/cli package for it.
Currently, the only two commands were switched to the new approach.
All the other commands will be migrated in upcoming changes.
Part of: https://gitlab.com/gitlab-org/gitaly/-/issues/5001
|
|
[ci skip]
|
|
[ci skip]
|
|
This fixes leaking goroutines, because it was failing to look up the
"stub" host.
Issue: https://gitlab.com/gitlab-org/gitaly/-/issues/4916
Changelog: fixed
|
|
git/housekeeping: Reduce frequency of full repacks
Closes #2774
See merge request https://gitlab.com/gitlab-org/gitaly/-/merge_requests/5640
Merged-by: Quang-Minh Nguyen <qmnguyen@gitlab.com>
Approved-by: Justin Tobler <jtobler@gitlab.com>
Co-authored-by: Patrick Steinhardt <psteinhardt@gitlab.com>
|
|
ruby: Update dependency google-protobuf to '~> 3.22.3'
See merge request https://gitlab.com/gitlab-org/gitaly/-/merge_requests/5645
Merged-by: Patrick Steinhardt <psteinhardt@gitlab.com>
Approved-by: Patrick Steinhardt <psteinhardt@gitlab.com>
Co-authored-by: GitLab Renovate Bot <gitlab-bot@gitlab.com>
|
|
Remove trace2 feature flags
See merge request https://gitlab.com/gitlab-org/gitaly/-/merge_requests/5624
Merged-by: James Fargher <proglottis@gmail.com>
Approved-by: James Fargher <proglottis@gmail.com>
Reviewed-by: Will Chandler <wchandler@gitlab.com>
Reviewed-by: Quang-Minh Nguyen <qmnguyen@gitlab.com>
Co-authored-by: Quang-Minh Nguyen <qmnguyen@gitlab.com>
|
|
|
|
|
|
diff: FindChangedPathsRequest
See merge request https://gitlab.com/gitlab-org/gitaly/-/merge_requests/5564
Merged-by: Patrick Steinhardt <psteinhardt@gitlab.com>
Approved-by: Patrick Steinhardt <psteinhardt@gitlab.com>
Reviewed-by: Patrick Steinhardt <psteinhardt@gitlab.com>
Co-authored-by: Toon Claes <toon@gitlab.com>
|
|
Due to different reasons we need to perform regular full repacks both in
normal repositories and in object pools. These full repacks are guided
by a cooldown period so that we'll perform them only in case the last
full repack is longer ago than the cooldown period.
For object pool, the reason we do full repacks is to refresh deltas so
that they again honor our delta islands. This is not all that important
to users and should not be noticeable in general when we do this less
frequently. Consequentially, we only perform a full repack once every
week.
For normal repositories full repacks are mostly done in order to
guarantee that objects will get evicted into cruft packs so that they
can be expired and thus deleted. This _is_ something that both we and
our customers care about given that it can be directly equated to disk
space that is required. It is thus prudent that we perform this on a
more-regular basis so that objects get deleted quickly.
That being said, there is interplay between the stale object grace
period (which is 14 days) and the cooldown periods (which is 1 day).
Effectively, assuming that a repository gets daily optimization jobs,
and with the knowledge in mind that we need to perform two full repacks
in order to evict an unreachable object, objects will get deleted after
14 to 15 days:
1. The first full repack on days 0 to 1 will evict the unreachable
object into a cruft pack.
2. We wait 14 days and will thus land either on day 14 or 15.
3. We perform a second full repack to expire the object part of the
cruft pack.
This interplay between both periods is important, because it means that
we can do compromises between tuning the cooldown period and stale
object grace period without actually impacting the median time to
deletion:
- Increasing the cooldown period means we need to perform less
regular full repacks, thus saving on resources. Conversely,
decreasing the cooldown means more regular full repacks and thus
using more resources.
- Increasing the grace period means we'll have a longer time to
avoid racy access to Git objects with the downside of more disk
space use. Decreasing the grace period means we are more likely to
hit racy access to Git objects, but evict objects and thus save
disk space more regularly.
Now optimizing the cooldown period is something we're very keen to do
because it directly impacts how much resources we and our customers need
to provision for machines. On the other hand, the grace period is mostly
there to avoid racy access to Git objects, and two weeks feels excessive
for that.
So long story short, this commit changes our strategy to increase the
full repack cooldown period to 5 days instead of 1 day while decreasing
the stale object grace period from 14 days to 7 days to counteract the
longer time-to-deletion for stale objects. This means objects will get
deleted 12 to 17 days afer becoming unreachable, with a median value of
14.5 days. This is the exact same median value as previously, so the
time-to-deletion should not change in practice. But on the other hand,
it does allow us to greatly save on compute resources by reducing the
frequency we perform full repacks to one fifth.
Furthermore, as the repack cooldown period for normal repositories and
object pools are almost the same now, let's merge them so that we have
one less special case to think about.
Changelog: changed
|
|
We're using cooldown periods to figure out whether it's time for another
full repack or not. So we will only consider doing a full repack in case
the previous full repack is longer ago than the cooldown period.
This is a value we can and want to fine-tune as we gain experience with
geometric repacking. So let's these into a constant that we can use in
our tests so that we don't have to adjust them every time we change the
cooldown period.
|
|
`gitaly_export_trace2_pack_objects_metrics` enables Gitaly to capture
metrics from git-pack-objects. Those metrics are exposed in two forms:
log fields and Prometheus metrics. This flag was rolled out in
Production successfully. This commit removes this flag.
|
|
`gitaly_export_trace2_tracing` allows Gitaly enables trace2 and export
collected events as distributed tracing spans. After roll out that flag,
everything was fine. GitLab.com didn't observe any performance impact.
It's time to remove that flag. Even if it is removed, the exporter still
enables trace2 only if the current code path has tracing enabled and
sampled.
|
|
praefect: Implementation of the new configuration validation
Closes #4650
See merge request https://gitlab.com/gitlab-org/gitaly/-/merge_requests/5601
Merged-by: Quang-Minh Nguyen <qmnguyen@gitlab.com>
Approved-by: Quang-Minh Nguyen <qmnguyen@gitlab.com>
Reviewed-by: Patrick Steinhardt <psteinhardt@gitlab.com>
Reviewed-by: Quang-Minh Nguyen <qmnguyen@gitlab.com>
Co-authored-by: Pavlo Strokov <pstrokov@gitlab.com>
|
|
All configuration types have validation methods that can be used
to check the values set. It is a final change that assembles all
these validations with validation of the other fields used by
Config type. Now 'praefect configuration validate' subcommand
returns all validation errors of the configuration.
Changelog: addition
Closes: https://gitlab.com/gitlab-org/gitaly/-/issues/4650
|
|
The check IsPositive is not that useful when we have other
comparison function. Also, it confuses as 0 is not a positive
number. It is replaced with usage of the GreaterOrEqual where
value for comparison is set to 0. The IsPositive is removed
and all call sites are changed to use GreaterOrEqual.
|
|
The new 'Validate' method validates values of the
'Yamux' type. It will be used in the later changes.
The 'AcceptBacklog' field now has 'uint' type as it is not allowed
to use negative values for it.
|
|
The new 'Validate' method validates values of the
'RepositoriesCleanup' type. It will be used in the later changes.
The 'RepositoriesInBatch' field now has 'uint' type as it is
not allowed to use negative values for it.
|
|
The new 'Validate' method validates values of the
'VirtualStorage' type. It will be used in the later changes.
Package 'cfgerror' extended with new 'NotEmptySlice()' function
to check the slice is not empty.
|
|
The new 'Validate' method validates values of the
'Node' type. It will be used in the later changes.
|
|
The new 'Validate' method validates values of the
'Replication' type. It will be used in the later changes.
Package 'cfgerror' extended with new value comparison checks:
LessThan, GreaterThan and GreaterOrEqual.
|
|
The new 'Validate' method validates values of the
'Reconciliation' type. It will be used in the later changes.
|
|
The new 'Validate' method validates values of the
'BackgroundVerification' type. It will be used in the later changes.
|
|
The new 'Validate' method validates values of the 'Failover' type and
returns all found errors together. If 'Enabled' field is set to
'false' no validation done as it won't be used and doesn't affect
the service anyhow. It will be used in the later changes.
The cfgerror now has a new 'IsSupportedValue' check.
|
|
The 'FailoverEnabled' field of the 'Config' was deprecated long time
ago. It was kept for backwards compatibility with omnibus. The change
done in https://gitlab.com/gitlab-org/omnibus-gitlab/-/merge_requests/4048
allows us to drop it completely.
|
|
doc: Add a doc about using Jaeger for local development
See merge request https://gitlab.com/gitlab-org/gitaly/-/merge_requests/5628
Merged-by: Quang-Minh Nguyen <qmnguyen@gitlab.com>
Approved-by: Evan Read <eread@gitlab.com>
Approved-by: Justin Tobler <jtobler@gitlab.com>
Reviewed-by: Evan Read <eread@gitlab.com>
Reviewed-by: Quang-Minh Nguyen <qmnguyen@gitlab.com>
Reviewed-by: Justin Tobler <jtobler@gitlab.com>
|
|
commit: Implement SHA256 compatibility for ListLastCommitsForTree
See merge request https://gitlab.com/gitlab-org/gitaly/-/merge_requests/5627
Merged-by: Justin Tobler <jtobler@gitlab.com>
Approved-by: Justin Tobler <jtobler@gitlab.com>
Approved-by: Will Chandler <wchandler@gitlab.com>
Co-authored-by: Patrick Steinhardt <psteinhardt@gitlab.com>
|
|
Distributed tracing and Jaeger are powerful tool. They are integrated
deep into Gitaly and other services at GitLab. Unfortunately, the
adoption at GitLab is not high. This commit adds a development doc to
get started with using Jaeger for Gitaly local development.
|
|
|
|
go: Update module google.golang.org/grpc to v1.54.0
See merge request https://gitlab.com/gitlab-org/gitaly/-/merge_requests/5581
Merged-by: John Cai <jcai@gitlab.com>
Approved-by: John Cai <jcai@gitlab.com>
Approved-by: Will Chandler <wchandler@gitlab.com>
Co-authored-by: GitLab Renovate Bot <gitlab-bot@gitlab.com>
|
|
tools/dlv: Update module github.com/go-delve/delve to v1.20.2
See merge request https://gitlab.com/gitlab-org/gitaly/-/merge_requests/5637
Merged-by: John Cai <jcai@gitlab.com>
Approved-by: John Cai <jcai@gitlab.com>
Approved-by: Will Chandler <wchandler@gitlab.com>
Co-authored-by: GitLab Renovate Bot <gitlab-bot@gitlab.com>
|
|
Revert "Merge branch 'jmd-update-error-message-concurrency-queue' into 'master'"
See merge request https://gitlab.com/gitlab-org/gitaly/-/merge_requests/5631
Merged-by: John Cai <jcai@gitlab.com>
Approved-by: John Cai <jcai@gitlab.com>
Approved-by: Christian Couder <chriscool@tuxfamily.org>
Co-authored-by: John McDonnell <jmcdonnell@gitlab.com>
|
|
The paths field in FindChangedPathsResponse didn't have any comment, so
we add some.
|
|
By default the FindChangedPaths RPC uses flag `-m` for git-diff-tree(1)
under the hood. This is usually what most callers want. But in some
cases they only want to see changes that are different from all the
parents.
The option `-c` for git-diff-tree(1) exists for this purpose. These
changes add a mode field to the RPC request so the caller can choose
between either command line flag.
Issue: https://gitlab.com/gitlab-org/gitaly/-/issues/4827
Changelog: added
|
|
Stop using the seeded repository and set up a repo in test, this should
make it easier to understand what each test case wants to test and
prepares tests for SHA256.
|
|
go: Update module golang.org/x/sys to v0.7.0
See merge request https://gitlab.com/gitlab-org/gitaly/-/merge_requests/5630
Merged-by: Justin Tobler <jtobler@gitlab.com>
Approved-by: Quang-Minh Nguyen <qmnguyen@gitlab.com>
Approved-by: Justin Tobler <jtobler@gitlab.com>
Co-authored-by: GitLab Renovate Bot <gitlab-bot@gitlab.com>
|
|
git: Remove sidecar Git configuration from command factory
See merge request https://gitlab.com/gitlab-org/gitaly/-/merge_requests/5636
Merged-by: Toon Claes <toon@gitlab.com>
Approved-by: Toon Claes <toon@gitlab.com>
Co-authored-by: Patrick Steinhardt <psteinhardt@gitlab.com>
|
