| Age | Commit message (Collapse) | Author |
|---|
|
|
|
go: Update module github.com/pelletier/go-toml/v2 to v2.0.8
See merge request https://gitlab.com/gitlab-org/gitaly/-/merge_requests/5855
Merged-by: James Fargher <proglottis@gmail.com>
Approved-by: Will Chandler <wchandler@gitlab.com>
Approved-by: James Fargher <proglottis@gmail.com>
Co-authored-by: GitLab Renovate Bot <gitlab-bot@gitlab.com>
|
|
conflicts: Introduce and use `conflictFilesWithGitMergeTree`
See merge request https://gitlab.com/gitlab-org/gitaly/-/merge_requests/5853
Merged-by: Will Chandler <wchandler@gitlab.com>
Approved-by: Will Chandler <wchandler@gitlab.com>
Reviewed-by: John Cai <jcai@gitlab.com>
Reviewed-by: karthik nayak <knayak@gitlab.com>
Co-authored-by: Karthik Nayak <knayak@gitlab.com>
|
|
Edit some Gitaly CLI command text
See merge request https://gitlab.com/gitlab-org/gitaly/-/merge_requests/5850
Merged-by: Justin Tobler <jtobler@gitlab.com>
Approved-by: Justin Tobler <jtobler@gitlab.com>
Reviewed-by: Justin Tobler <jtobler@gitlab.com>
Co-authored-by: Evan Read <eread@gitlab.com>
|
|
go: Use Go 1.19 as minimum required version
Closes #5324
See merge request https://gitlab.com/gitlab-org/gitaly/-/merge_requests/5831
Merged-by: Sami Hiltunen <shiltunen@gitlab.com>
Approved-by: Quang-Minh Nguyen <qmnguyen@gitlab.com>
Reviewed-by: Patrick Steinhardt <psteinhardt@gitlab.com>
Reviewed-by: Quang-Minh Nguyen <qmnguyen@gitlab.com>
Co-authored-by: Patrick Steinhardt <psteinhardt@gitlab.com>
|
|
ref: Cleanup deprecated method buildLocalBranch and proto field
See merge request https://gitlab.com/gitlab-org/gitaly/-/merge_requests/5851
Merged-by: karthik nayak <knayak@gitlab.com>
Approved-by: karthik nayak <knayak@gitlab.com>
Reviewed-by: Quang-Minh Nguyen <qmnguyen@gitlab.com>
Co-authored-by: Xing Xin <xingxin.xx@bytedance.com>
|
|
The `branches` field is actually deprecated and no longer referenced
during the rolling out of `simplify_find_local_branches_response`, but
it seems we missed the deprecation marking and removal process for the
proto.
Signed-off-by: Xing Xin <xingxin.xx@bytedance.com>
|
|
The current implementation of `ListConflictFiles` uses Git2Go to find
the file conflicts, parses this data and returns the information to the
client.
As part of the plan to move away from Git2Go, we introduce a new
function `conflictFilesWithGitMergeTree` which uses git-merge-tree(1) to
find the conflicted files. We then use this implementation behind the
`ListConflictFilesMergeTree` featureflag.
|
|
Add a new flag `ListConflictFilesMergeTree`, which will be used to
enable the use of git-merge-tree(1) for the ListConflictFiles RPC.
|
|
The `MergeTreeConflictError` doesn't contain the mode information of the
conflict files. Let's add this information as its needed by
`ListConflictFiles`.
|
|
The `Merge()` function iterates over a map and does a lookup on blobs.
This works, but iterating over a map doesn't guarantee the order of
iteration. Matter of fact, go randomizes map iteration to avoid sloppy
programming.
The iteration order doesn't really matter for the functionality of the
function. The only problem is it makes testing non-deterministic. Since
we're about to rid this code with time and to make the upcoming tests
deterministic, lets replace the map with a struct slice.
|
|
tests: Extend SHA256 test coverage
See merge request https://gitlab.com/gitlab-org/gitaly/-/merge_requests/5856
Merged-by: Quang-Minh Nguyen <qmnguyen@gitlab.com>
Approved-by: karthik nayak <knayak@gitlab.com>
Co-authored-by: Patrick Steinhardt <psteinhardt@gitlab.com>
|
|
Enable testing with the SHA256 object format in the Gitaly repository
service by dropping seed repositories and generating test data at
runtime. This change only covers those tests which are trivially
adaptable to the new object format.
|
|
Enable testing with the SHA256 object format in the Gitaly ref
service by dropping seed repositories and generating test data at
runtime. This change only covers those tests which are trivially
adaptable to the new object format.
|
|
Enable testing with the SHA256 object format in the Gitaly hook
service by dropping seed repositories and generating test data at
runtime. This change only covers those tests which are trivially
adaptable to the new object format.
|
|
Enable testing with the SHA256 object format in the Gitaly commit
service by dropping seed repositories and generating test data at
runtime. This change only covers those tests which are trivially
adaptable to the new object format.
|
|
Enable testing with the SHA256 object format for Gitaly server related
functionality. These tests don't rely on the repository contents, so the
change is as simple as removing the seed repository.
|
|
Enable testing with the SHA256 object format for all Praefect related
packages. Given that most of the tests don't go down to the repository
level the majority of tests only need to drop the seed repository in
order to work correctly with SHA256.
|
|
go: Update module github.com/sirupsen/logrus to v1.9.2
See merge request https://gitlab.com/gitlab-org/gitaly/-/merge_requests/5837
Merged-by: Toon Claes <toon@gitlab.com>
Approved-by: Pavlo Strokov <pstrokov@gitlab.com>
Approved-by: Toon Claes <toon@gitlab.com>
Co-authored-by: GitLab Renovate Bot <gitlab-bot@gitlab.com>
|
|
into 'master'
go: Update github.com/gl-gitaly/go-license-detector/v4 digest to 18a439e
See merge request https://gitlab.com/gitlab-org/gitaly/-/merge_requests/5832
Merged-by: James Fargher <proglottis@gmail.com>
Approved-by: James Fargher <proglottis@gmail.com>
Co-authored-by: Toon Claes <toon@gitlab.com>
Co-authored-by: GitLab Renovate Bot <gitlab-bot@gitlab.com>
|
|
|
|
|
|
Validate custom hook archive prior to accepting a transaction
Closes #5126
See merge request https://gitlab.com/gitlab-org/gitaly/-/merge_requests/5835
Merged-by: Justin Tobler <jtobler@gitlab.com>
Approved-by: Justin Tobler <jtobler@gitlab.com>
Approved-by: Toon Claes <toon@gitlab.com>
Co-authored-by: Sami Hiltunen <shiltunen@gitlab.com>
|
|
|
|
Upgrade to grpc-tools v1.55.0
See merge request https://gitlab.com/gitlab-org/gitaly/-/merge_requests/5852
Merged-by: Toon Claes <toon@gitlab.com>
Approved-by: Toon Claes <toon@gitlab.com>
Co-authored-by: Stan Hu <stanhu@gmail.com>
|
|
Remove unused feature flag pack_objects_limiting_remote_ip
See merge request https://gitlab.com/gitlab-org/gitaly/-/merge_requests/5828
Merged-by: Quang-Minh Nguyen <qmnguyen@gitlab.com>
Approved-by: Quang-Minh Nguyen <qmnguyen@gitlab.com>
Co-authored-by: Sami Hiltunen <shiltunen@gitlab.com>
|
|
'4736-clean-up-stale-locks-when-applying-wal-in-transactionmanager' into 'master'
gitaly: Cleanup stale reflocks in `prepareReferenceTransaction`
Closes #4736
See merge request https://gitlab.com/gitlab-org/gitaly/-/merge_requests/5758
Merged-by: Sami Hiltunen <shiltunen@gitlab.com>
Approved-by: Sami Hiltunen <shiltunen@gitlab.com>
Reviewed-by: Sami Hiltunen <shiltunen@gitlab.com>
Reviewed-by: karthik nayak <knayak@gitlab.com>
Co-authored-by: Karthik Nayak <knayak@gitlab.com>
|
|
Switch all of our tools to use Go v1.19 as minimum Go version. While not
strictly necessary, it makes for a more coherent picture of which Go
version Gitaly uses as a whole.
|
|
We have globally migrated to build all GitLab with Go 1.19, and many of
the components have already upgraded to Go 1.19 as their minimum
supported version. Let's follow suit and do the same by bumping our
minimum required Go version to v1.19.
|
|
We're about to bump the minimum required Go version to 1.19, so this
commit drops test jobs which exercise Go 1.18 and adds test jobs for the
new Go 1.20 release.
Note that we're not yet introducing tests with Go 1.20 for our FIPS test
jobs given that there is no FIPS-enabled image for Go 1.20 yet.
|
|
The shared object for our Gitaly-specific linter is implemented in its
own module with a standalone `go.mod` file. Consequentially, our normal
linting run won't process its sources as golangci-lint only supports
linting a single module at a time. To work around this, we thus have a
separate Makefile target that is supposed to run golangci-lint inside of
that separate module.
With Go 1.20, this linting target started to fail in our CI though with
the following error:
```
./packed_binaries.go:19:3: invalid go:embed: build system did not supply embed configuration
```
The error is confusing as the linter isn't even supposed to process the
reported file. But as it turns out, our linting Makefile target switches
directories and executes golangci-lint in separate shells, which causes
us to wrongly lint the main Gitaly module again. And given that this
Makefile target doesn't have our binaries as a dependency they might not
yet have been built, which then causes the `go:embed` line to complain.
The reason this started to fail is an upstream regression that was
introduced into Go via c0ac33c006 (cmd/go: don't compute Embed fields if
they're not needed, 2023-02-20). With this change, Go doesn't seem to
compute the `Embed` fields at all anymore, and thus golangci-lint starts
to complain.
Fix this bug by changing the directory in the same shell as we run the
linter command.
|
|
In the transaction manager we have a set of tests that verify whether
asynchronous deletion of repositories works as expected. This test has
started to indeterministically deadlock with Go 1.20.
Bisecting this regression points to upstream commit 8477562ce5
(cmd/compile: be more careful about pointer incrementing in range loops,
2022-11-11). This commit changes how pointers are incremented in loops
so that there never is a Go pointer that points past the backing array.
This is done by using a `uintptr` to track the next array member at the
end of the loop, which is thus not getting treated as a valid pointer by
Go.
Interestingly, the regression is fixed with 0384235a15 (cmd/compile:
don't mark uintptr->unsafe.Pointer conversion unsafe points,
2023-01-11), which changes semantics around conversions between
`uintptr` and unsafe pointers. Previously, code preceding any such
conversion is considered to be an unsafe point in the code flow. This
had the consequence that Go didn't allow preemption of Goroutines
between any such conversion and a preceding function call. And in
combination with the above commit that introduces the regression it
could now happen that a complete loop is considered to be unsafe where
it previously was safe for preemption. The second commit then fixes the
issue because it starts to not treat `uintptr` to unsafe pointer
conversions as unsafe anymore.
We seemingly have such a case in our transaction manager tests where a
loop is now busy spinning without ever being preempted anymore when
trying to delete repositories asynchronously. It is not exactly clear
where this is happening though, but we can seemingly work around the
deadlock by changing a non-ranged loop to stop busy spinning.
Fix the deadlock by converting the `admitted` field of transaction from
a boolean to a channel. This allows us to wait for transactions to be
admitted without busy spinning and thus avoids the deadlock. It is more
of a workaround than a proper fix, but this should be good enough for
now given that the compiler regression is about to be fixed with Go
1.21 anyway.
|
|
The `addPackRefsInhibitor` function allows users to add an inhibitor to
`git-pack-refs(1)`. In the function we check if `git-pack-refs(1)` is
currently/still running (in a loop) and allow the user to exit early via
context cancellation.
There is race condition here wherein we read from the
`state.packRefsDone` channel without a lock. This is a race condition,
since, the cleanup function of `tryRunningPackRefs` could also set
`state.packRefsDone` to nil concurrently. And reading from a nil channel
causes a panic.
We solve this race by using the local variable `packRefsDone`, which is
set after obtaining a lock. This way the local variable could be closed
but will not be `nil`, since only `state.packRefsDone` is set to `nil`.
|
|
When running `prepareReferenceTransaction`, there is a possibility that
it fails due to the existence of stale reference locks in the repository.
Since TransactionManager is the only process apart from housekeeping
which writes into the repository, we should be okay with cleaning these
stale reference locks.
So in `prepareReferenceTransaction`, if we encounter a stale reference
lock, we add an inhibitor on housekeeping from running git-pack-refs(1).
This is to avoid new reference locks from being created. Then we clear
the existing stale reference locks before continuing further.
To enable this we expose the internal function `addPackRefsInhibitor` as
`AddPackRefsInhibitor` in the housekeeping package via the
`HousekeepingManager`.
We also add a new step type: `ModifyFiles` in `TestTransactionManager`
which allows us to create files in the repository.
|
|
In the `TransactionManager` we want to cleanup stale lock files if we
run into them (but not when housekeeping is running). To do this we need
the `housekeeping.RepositoryManager`, let's add this field to the
TransactionManager.
The following commit[s] will utilize the field.
|
|
This matches the upgrade to the grpc v1.55.0 gem in GitLab Rails:
https://gitlab.com/gitlab-org/gitlab/-/merge_requests/121899
|
|
The interface `repository` was added to allow us to add hooks for
testing. In the previous commit, we removed this dependency. Let's
remove the interface altogether and pass `localrepo.Repo` everywhere. We
need this for the upcoming commits wherein we'll integrate
`housekeeping.Manager` which expects us to pass a `localrepo.Repo`
variable.
|
|
We use the `repositoryHook` to add a hook before the `ResolveRevision`
function is called. In reality the use-case is to add a hook before the
log is stored into the database.
In `1ff6dcd1` we introduced a framework for panics in the hooks. Let's
use this to have a `beforeStoreLogEntry` hook instead and remove the
`ResolveRevision` hook. This is more inline with what we actually want
to test.
Overall, this is done so we can remove the `repositoryHook` and remove
the `repository` interface altogether.
|
|
The method buildLocalBranch constructs FindLocalBranchResponse, which is
required by FindLocalBranchesResponse. We already finished the rolling
process of replacing FindLocalBranchResponse with the shared message
Branch(ff simplify_find_local_branches_response), so this method is no
longer needed.
Signed-off-by: Xing Xin <xingxin.xx@bytedance.com>
|
|
Revert "Merge branch 'revert-git-merge-tree' into 'master'"
See merge request https://gitlab.com/gitlab-org/gitaly/-/merge_requests/5847
Merged-by: Patrick Steinhardt <psteinhardt@gitlab.com>
Approved-by: Patrick Steinhardt <psteinhardt@gitlab.com>
Co-authored-by: Karthik Nayak <knayak@gitlab.com>
|
|
This reverts commit 2df9d7f637a21e0b90c875726fcbc2f21e0d43d3, reversing
changes made to 75a157036114fea3c51eed343fa4cf482ccdb0ba.
|
|
praefect: Drop feature flag that fixes routing with additional repos
Closes #5134
See merge request https://gitlab.com/gitlab-org/gitaly/-/merge_requests/5838
Merged-by: James Fargher <proglottis@gmail.com>
Approved-by: Will Chandler <wchandler@gitlab.com>
Approved-by: James Fargher <proglottis@gmail.com>
Co-authored-by: Patrick Steinhardt <psteinhardt@gitlab.com>
|
|
conflicts: Introduce and use `conflictFilesWithGitMergeTree`
See merge request https://gitlab.com/gitlab-org/gitaly/-/merge_requests/5843
Merged-by: Will Chandler <wchandler@gitlab.com>
Approved-by: John Cai <jcai@gitlab.com>
Approved-by: Will Chandler <wchandler@gitlab.com>
Reviewed-by: karthik nayak <knayak@gitlab.com>
Co-authored-by: Karthik Nayak <knayak@gitlab.com>
|
|
Remove unnecessary validation from default branch updates with WAL
Closes #4822
See merge request https://gitlab.com/gitlab-org/gitaly/-/merge_requests/5839
Merged-by: karthik nayak <knayak@gitlab.com>
Approved-by: Pavlo Strokov <pstrokov@gitlab.com>
Co-authored-by: Sami Hiltunen <shiltunen@gitlab.com>
|
|
The current implementation of `ListConflictFiles` uses Git2Go to find
the file conflicts, parses this data and returns the information to the
client.
As part of the plan to move away from Git2Go, we introduce a new
function `conflictFilesWithGitMergeTree` which uses git-merge-tree(1) to
find the conflicted files. We then use this implementation behind the
`ListConflictFilesMergeTree` featureflag.
|
|
git: Extend locking timeout for packed-refs to decrease contention
Closes #5160
See merge request https://gitlab.com/gitlab-org/gitaly/-/merge_requests/5833
Merged-by: Justin Tobler <jtobler@gitlab.com>
Approved-by: Justin Tobler <jtobler@gitlab.com>
Approved-by: karthik nayak <knayak@gitlab.com>
Co-authored-by: Patrick Steinhardt <psteinhardt@gitlab.com>
|
|
TransactionManager is currently not validating the custom hook archive
in any way prior to accepting and logging the transaction. This could
lead to invalid hook archive being logged which would prevent applying
the log entry of the transaction leading to transaction processing halting.
This commit verifies the archive by extracting it on the disk to the
staging directory prior to logging. These extracted files can then later
be also used for computing a vote to Praefect from the hook files. For
now, no other validation is performed than just ensuring the hooks can
be extracted on the disk. This matches the behavior in SetCustomHooks
which also doesn't verify anything. This is something we should improve
later. However, if the hooks extract successfully, the log processing won't
fail because of them. If the hooks fail to execute, it will fail the
hook execution but that's outside of the scope of the TransactionManager.
The hooks can still be fixed by committing new ones.
|
|
cli/praefect: Fix flaky tests for the metadata subcommand
Closes #5144
See merge request https://gitlab.com/gitlab-org/gitaly/-/merge_requests/5805
Merged-by: Toon Claes <toon@gitlab.com>
Approved-by: Toon Claes <toon@gitlab.com>
Approved-by: Pavlo Strokov <pstrokov@gitlab.com>
Co-authored-by: Patrick Steinhardt <psteinhardt@gitlab.com>
|
|
Add a new flag `ListConflictFilesMergeTree`, which will be used to
enable the use of git-merge-tree(1) for the ListConflictFiles RPC.
|
|
The `MergeTreeConflictError` doesn't contain the mode information of the
conflict files. Let's add this information as its needed by
`ListConflictFiles`.
|
