| Age | Commit message (Collapse) | Author |
|---|
|
|
|
As we are going to be able to offload blobs in some packfiles that might
be stored on HDDs to save storage cost, we should have some repository
path infrastructure to find the path where the new packfiles will be and
to know if a repo uses blob offloading.
|
|
localrepo: Add tests for ResolveRevision
See merge request https://gitlab.com/gitlab-org/gitaly/-/merge_requests/6498
Merged-by: James Fargher <jfargher@gitlab.com>
Approved-by: James Fargher <jfargher@gitlab.com>
Reviewed-by: karthik nayak <knayak@gitlab.com>
Co-authored-by: Toon Claes <toon@gitlab.com>
|
|
Git: Bump the minimum version to 2.42
See merge request https://gitlab.com/gitlab-org/gitaly/-/merge_requests/6486
Merged-by: karthik nayak <knayak@gitlab.com>
Approved-by: Will Chandler <wchandler@gitlab.com>
Approved-by: Quang-Minh Nguyen <qmnguyen@gitlab.com>
Reviewed-by: Will Chandler <wchandler@gitlab.com>
Reviewed-by: Quang-Minh Nguyen <qmnguyen@gitlab.com>
|
|
There was no test coverage for the ResolveRevision() function, add them.
Label: maintenance::pipelines
|
|
It's common practice to mark all tests to run in parallel with
t.Parallel(). Add this to all existing tests in refs_test.go and make
the required code change to allow parallel execution.
|
|
localrepo: Quarantine flaky `TestRepo_FetchInternal` test
See merge request https://gitlab.com/gitlab-org/gitaly/-/merge_requests/6491
Merged-by: Justin Tobler <jtobler@gitlab.com>
Approved-by: Will Chandler <wchandler@gitlab.com>
|
|
Handle repository creations transactionally
See merge request https://gitlab.com/gitlab-org/gitaly/-/merge_requests/6492
Merged-by: Justin Tobler <jtobler@gitlab.com>
Approved-by: Justin Tobler <jtobler@gitlab.com>
Approved-by: karthik nayak <knayak@gitlab.com>
Reviewed-by: Sami Hiltunen <shiltunen@gitlab.com>
Co-authored-by: Sami Hiltunen <shiltunen@gitlab.com>
|
|
Since we've updated the minimum version of Git to v2.42.0, we no longer
need to check if git-cat-file(1) supports the '-Z' option. So let's get
rid of the helper function and code around it.
|
|
As we have stopped building git 2.41, let's bump the minimum version on
the CI to 2.42.
|
|
In the previous release we had removed the flag to use 2.41. In this
release we can stop building it and remove it entirely.
|
|
Let's bump the minimum version of Git to 2.42 with this bring in
the following important changes:
* git-pack-refs(1) learned to take --include and --exclude, which allows
us to skip packing certain references. This is important in the context
of the write-ahead log so that we don't pack write-ahead log
references.
* SHA256 is no longer marked as experimental.
* Git has learned to more efficiently exclude references, which should
help repositories which have many hidden references.
* git-rev-list(1) has learned to take pseudo-refs like --all and --not
in --stdin mode.
* git-cat-file(1) has learned a new -Z switch, which uses NUL delimiters
for both stdin and stdout.
|
|
Update to latest gitlab-dangerfiles Gem version
See merge request https://gitlab.com/gitlab-org/gitaly/-/merge_requests/6493
Merged-by: Toon Claes <toon@gitlab.com>
Approved-by: Toon Claes <toon@gitlab.com>
Co-authored-by: Evan Read <eread@gitlab.com>
|
|
proto: Add documentation for `ConflictsService`
See merge request https://gitlab.com/gitlab-org/gitaly/-/merge_requests/6484
Merged-by: karthik nayak <knayak@gitlab.com>
Approved-by: karthik nayak <knayak@gitlab.com>
Reviewed-by: Justin Tobler <jtobler@gitlab.com>
Reviewed-by: karthik nayak <knayak@gitlab.com>
Co-authored-by: Justin Tobler <jtobler@gitlab.com>
|
|
|
|
featureflag: Default enable `ReplicateRepositoryObjectPool`
See merge request https://gitlab.com/gitlab-org/gitaly/-/merge_requests/6487
Merged-by: Justin Tobler <jtobler@gitlab.com>
Approved-by: Will Chandler <wchandler@gitlab.com>
|
|
With the TransactionManager now supporting creating a repository in
a transaction, let's enable the repository creating RPCs to run
transactionally.
Some tests needed minor adjustments as they assert storage details
that change with transactions.
|
|
The `TestRepo_FetchInternal/nonexistent_revision` test is flaky and
causing consistent pipeline failures. Quarantine the test until a proper
fix can be implemented.
|
|
The `ReplicateRepositoryObjectPool` feature flag enables object pool
replication via the `ReplicateRepository` RPC when the client sets the
`ReplicateObjectDeduplicationNetworkMembership` option in the request.
Set this feature flag to be enabled by default.
|
|
Start commands directly in correct cgroups
See merge request https://gitlab.com/gitlab-org/gitaly/-/merge_requests/6470
Merged-by: Will Chandler <wchandler@gitlab.com>
Approved-by: Will Chandler <wchandler@gitlab.com>
Reviewed-by: Will Chandler <wchandler@gitlab.com>
Reviewed-by: Sami Hiltunen <shiltunen@gitlab.com>
Reviewed-by: Quang-Minh Nguyen <qmnguyen@gitlab.com>
Co-authored-by: Sami Hiltunen <shiltunen@gitlab.com>
|
|
Gitaly is currently first launching commands, and then adding them
to the correct cgroup. This has the downside that the commands get
to execute without the appropriate limits being applied for some
time before Gitaly manages to configure the Cgroup. This commit
changes Gitaly to launch commands directly in the correct cgroup.
This gets rid of the brief window with incorrect limits, and should
also be more efficient than moving processes between cgroups.
The fix only applies when using cgroups version 2 with Linux 5.7
or later.
Changelog: fixed
|
|
This commit exposes a method from the cgroup manager to configure a
command directly into a cgroup. This is done by configuring the cgroup
in the SysProcAttr so the command gets spawned directly in the cgroup.
The functionality is only available with cgroups version 2 and kernel
version 5.7 or later.
|
|
This commit adds a utility function for detecting whether the running
kernel is at least at given version. This can be used to determine
whether functionality in some later kernels can be used safely by
Gitaly.
Version detection is for now only implemented for Linux as we don't
have use cases elsewhere yet.
|
|
Remove name resolving error test
See merge request https://gitlab.com/gitlab-org/gitaly/-/merge_requests/6488
Merged-by: Will Chandler <wchandler@gitlab.com>
Approved-by: Justin Tobler <jtobler@gitlab.com>
Approved-by: Will Chandler <wchandler@gitlab.com>
Co-authored-by: Sami Hiltunen <shiltunen@gitlab.com>
|
|
We have a test asserting a specific error is returned when attempting
to resolve a non-existent name. The 'not-existing.com' address used
in the test has since been registered and no longer results in a
resolving error. Remove the test to have the pipeline pass. We're
probably fine even if we don't test for this specific case.
|
|
Some of the protobuf message types in the `ConflictsService` lack
documentation. Add the missing documentation.
|
|
[ci skip]
|
|
repository: Update references atomically in `FetchRemote`
See merge request https://gitlab.com/gitlab-org/gitaly/-/merge_requests/6432
Merged-by: Quang-Minh Nguyen <qmnguyen@gitlab.com>
Approved-by: Quang-Minh Nguyen <qmnguyen@gitlab.com>
Reviewed-by: Sami Hiltunen <shiltunen@gitlab.com>
Reviewed-by: Quang-Minh Nguyen <qmnguyen@gitlab.com>
Reviewed-by: Justin Tobler <jtobler@gitlab.com>
Co-authored-by: Justin Tobler <jtobler@gitlab.com>
|
|
Fix TestUserCreateBranch_successful writing into shared repository
See merge request https://gitlab.com/gitlab-org/gitaly/-/merge_requests/6485
Merged-by: Toon Claes <toon@gitlab.com>
Approved-by: Toon Claes <toon@gitlab.com>
Approved-by: Justin Tobler <jtobler@gitlab.com>
Co-authored-by: Sami Hiltunen <shiltunen@gitlab.com>
|
|
Fix broken test performing concurrent writes to shared repo
See merge request https://gitlab.com/gitlab-org/gitaly/-/merge_requests/6483
Merged-by: Toon Claes <toon@gitlab.com>
Approved-by: Toon Claes <toon@gitlab.com>
Approved-by: Will Chandler <wchandler@gitlab.com>
Co-authored-by: Sami Hiltunen <shiltunen@gitlab.com>
|
|
backup: Add support for incremental server-side backups
See merge request https://gitlab.com/gitlab-org/gitaly/-/merge_requests/6475
Merged-by: Quang-Minh Nguyen <qmnguyen@gitlab.com>
Approved-by: Quang-Minh Nguyen <qmnguyen@gitlab.com>
Co-authored-by: James Liu <jliu@gitlab.com>
|
|
Set default branch from manifest on restore
See merge request https://gitlab.com/gitlab-org/gitaly/-/merge_requests/6482
Merged-by: Toon Claes <toon@gitlab.com>
Approved-by: Toon Claes <toon@gitlab.com>
Co-authored-by: James Fargher <jfargher@gitlab.com>
|
|
Incremental backups are currently supported only for client-side
repository backups, i.e. if the `REPOSITORIES_SERVER_SIDE=true` env
var is not set when running the gitlab:backup:create Rake task. This is
because we're not providing this flag in the request to the
BackupRepository RPC. This commit:
- Modifies the BackupRepositoryRequest message to include an
`incremental` flag and update the surrounding documentation.
- Threads the `incremental` flag through the gRPC request and into the
backup manager on the other end.
- Updates some of the tests to exercise the flag.
Changelog: added
|
|
The majority of setup, assertion, and teardown logic for the two tests
for the `gitaly-backup create` command are shared. Refactor the two
separate test functions into a single function utilising table tests,
since we'll be adding additional tests soon.
|
|
Since backup manifest files explicitly setup the object format to use,
these tests can be enabled for sha256 testing.
The other cases will be converted to sha256 when the pointer and legacy
layouts are replaced.
|
|
The setup functions here were sporadically using t instead of tb. The t
in scope here is from the parent test which means assertions and
skipping would apply to the wrong test.
|
|
TestUserCreateBranch_startPoint is sharing a test repository between
multiple subtests and writing concurrently into the repository from
both subtests. Transactions expect the TransactionManager to be the
only writer to the repository, and concurrent writes may cause flakes
in the tests. Set up a clean repository for each subtest.
|
|
TestUserCreateBranch_successful is setting up a shared repository in
the tests. It then proceeds to concurrently write to the shared
repository in the subtests. This leads to indeterministic results as
the branch may or may not be already created by one of the other
subtests. Fix this by running each of the tests agains their own
repository. As all tests are now running agains their own repositories,
we know longer need to delete the created branch at the end of the test.
|
|
The `FetchRemote` RPC fetches a set of refspecs from a remote repository
into a target repository. To be atomic, reference updates should only be
performed if transaction quorum is achieved with participating
repository replicas. Furthermore, when a subset of references diverge,
partial reference updates should be applied.
To accommodate this, the `FetchRemote` RPC now supports quarantining
fetched objects and now performs a reference-transaction only on updated
references.
|
|
In a future change, the `FetchRemote` RPC will introduce atomic
reference updates behind a feature flag. Split out current transaction
logic into a separate function to make swapping implementations easier.
|
|
When performing `git-fetch(1)` with the `porcelain` option, references
are printed to stdout in an easy-to-parse format. This change introduces
a `localrepo.FetchRemote()` configuration to enabled `porcelain` output.
|
|
When performing `git-fetch(1)` with the `dry-run` option, reference
updates are not performed. In a future commit, reference updates need to
be captured without committing the changes. This change introduces a
configuration to `localrepo.FetchRemote()` to enable dry-runs.
|
|
In a future commit, updated references need to be extracted from
`git-fetch(1)` output performed during `FetchRemote`. This change
introduces configuration to `localrepo.FetchRemote()` to configure
stdout.
|
|
When `git-fetch(1)` is executed with the `porcelain` option, reference
updates are printed to stdout in an easy-to-parse format. This change
introduces a scanner to parse this output for reference information.
|
|
Introduce transaction managing middleware
See merge request https://gitlab.com/gitlab-org/gitaly/-/merge_requests/6458
Merged-by: Sami Hiltunen <shiltunen@gitlab.com>
Approved-by: Will Chandler <wchandler@gitlab.com>
Reviewed-by: Will Chandler <wchandler@gitlab.com>
Reviewed-by: Sami Hiltunen <shiltunen@gitlab.com>
|
|
TestUserRebaseToRef_failure is sharing a test repository between the
subtests. The subtests are ran concurrently. Each of the subtests are
resetting the valid target ref on beginning. As the tests are ran
concurrently, they're all modifying the repository concurrently. This
won't work anymore with transactions enabled and will cause flakes.
As none of the tests should modify the repository, stop resetting the
target ref at the beginning of each test. If one of the tests fails and
modifies the ref, the other subtests could fail as well. This can be
improved later by actually setting up independent test state for each
of the tests. For now, fix the flake.
|
|
Write-ahead log repository creations
See merge request https://gitlab.com/gitlab-org/gitaly/-/merge_requests/6435
Merged-by: Sami Hiltunen <shiltunen@gitlab.com>
Approved-by: Quang-Minh Nguyen <qmnguyen@gitlab.com>
Reviewed-by: Sami Hiltunen <shiltunen@gitlab.com>
Reviewed-by: Quang-Minh Nguyen <qmnguyen@gitlab.com>
Reviewed-by: karthik nayak <knayak@gitlab.com>
|
|
go: Update module github.com/go-enry/go-enry/v2 to v2.8.6
See merge request https://gitlab.com/gitlab-org/gitaly/-/merge_requests/6430
Merged-by: Quang-Minh Nguyen <qmnguyen@gitlab.com>
Approved-by: Quang-Minh Nguyen <qmnguyen@gitlab.com>
Co-authored-by: GitLab Renovate Bot <gitlab-bot@gitlab.com>
|
|
Previously backups would try and guess what to set HEAD to based off of
the bundle file. If the manifest file has the default branch to use,
then we no longer have to guess. In that case turn off the guessing
logic.
|
|
Adds a test for the worst case scenario where the backup was of an empty
repository. In this case there is no bundle file to derive the default
branch.
Eventually manifest files will have blank paths to indicate, for
example, that there is no bundle or no hooks. Until then the code
expects that all paths will be set, even if the destination file does
not exist.
Changelog: changed
|
