| Age | Commit message (Collapse) | Author |
|---|
|
RunPraefectServer is intended to be used only for testing purposes
so we should exclude it from builds to avoid any logic in the file
from leaking into the actual build.
|
|
Fix flaky TestGetSpawnToken_CommandStats test
See merge request https://gitlab.com/gitlab-org/gitaly/-/merge_requests/6147
Merged-by: karthik nayak <knayak@gitlab.com>
Approved-by: karthik nayak <knayak@gitlab.com>
Reviewed-by: Quang-Minh Nguyen <qmnguyen@gitlab.com>
Reviewed-by: karthik nayak <knayak@gitlab.com>
Co-authored-by: Quang-Minh Nguyen <qmnguyen@gitlab.com>
|
|
go: Update module github.com/rubenv/sql-migrate to v1.5.2
See merge request https://gitlab.com/gitlab-org/gitaly/-/merge_requests/6152
Merged-by: Quang-Minh Nguyen <qmnguyen@gitlab.com>
Approved-by: Quang-Minh Nguyen <qmnguyen@gitlab.com>
Co-authored-by: GitLab Renovate Bot <gitlab-bot@gitlab.com>
|
|
This test aims to test the metrics of spawn tokens. One particularly
interesting metric is the `gitaly_spawn_token_waiting_length` gauge. It
measures the length of the queue waiting for a spawn token.
In the test, we set up two main and secondary goroutines. The spawn
token parallelism is set to 1. The main goroutine acquires the goroutine
first, second one in parallel follows. However, the second goroutine may
not start on time, or the acquiring function hasn't been called at the
time the metrics are asserted.
Even after the second goroutine starts, there is a small possibility for
the metric collection to run before the second goroutine lines up for
the token. There is no way to get a precise signal when the second
goroutine is waiting. Hence, the only solution is to poll for the metric
until it changes.
|
|
commit: Implement SHA256 support for CommitSignatures
See merge request https://gitlab.com/gitlab-org/gitaly/-/merge_requests/6156
Merged-by: Quang-Minh Nguyen <qmnguyen@gitlab.com>
Approved-by: karthik nayak <knayak@gitlab.com>
Approved-by: Quang-Minh Nguyen <qmnguyen@gitlab.com>
Co-authored-by: Patrick Steinhardt <psteinhardt@gitlab.com>
|
|
go: Update golang.org/x/exp digest to b0cb94b
See merge request https://gitlab.com/gitlab-org/gitaly/-/merge_requests/6161
Merged-by: Quang-Minh Nguyen <qmnguyen@gitlab.com>
Approved-by: karthik nayak <knayak@gitlab.com>
Approved-by: Quang-Minh Nguyen <qmnguyen@gitlab.com>
Co-authored-by: GitLab Renovate Bot <gitlab-bot@gitlab.com>
|
|
counter: Add repository counter middleware
See merge request https://gitlab.com/gitlab-org/gitaly/-/merge_requests/6071
Merged-by: Quang-Minh Nguyen <qmnguyen@gitlab.com>
Approved-by: karthik nayak <knayak@gitlab.com>
Approved-by: Quang-Minh Nguyen <qmnguyen@gitlab.com>
Reviewed-by: Steve Xuereb <sxuereb@gitlab.com>
Reviewed-by: James Fargher <jfargher@gitlab.com>
Reviewed-by: Will Chandler <wchandler@gitlab.com>
Reviewed-by: Quang-Minh Nguyen <qmnguyen@gitlab.com>
Reviewed-by: karthik nayak <knayak@gitlab.com>
Co-authored-by: Will Chandler <wchandler@gitlab.com>
|
|
Fallback to 'direct' for GOPROXY when 'https://proxy.golang.org' is unavailable
See merge request https://gitlab.com/gitlab-org/gitaly/-/merge_requests/6162
Merged-by: Justin Tobler <jtobler@gitlab.com>
Approved-by: Stan Hu <stanhu@gmail.com>
Approved-by: Justin Tobler <jtobler@gitlab.com>
Reviewed-by: Justin Tobler <jtobler@gitlab.com>
Co-authored-by: Nao Hashizume <nhashizume@gitlab.com>
|
|
|
|
'master'
conflicts: Remove git2go implementation of `ResolveConflicts`
Closes #5436 and #5395
See merge request https://gitlab.com/gitlab-org/gitaly/-/merge_requests/6122
Merged-by: Justin Tobler <jtobler@gitlab.com>
Approved-by: John Cai <jcai@gitlab.com>
Approved-by: Justin Tobler <jtobler@gitlab.com>
Reviewed-by: Justin Tobler <jtobler@gitlab.com>
Reviewed-by: karthik nayak <knayak@gitlab.com>
Co-authored-by: Karthik Nayak <knayak@gitlab.com>
|
|
repository: Refactor `CreateRepositoryFromBundle` tests
See merge request https://gitlab.com/gitlab-org/gitaly/-/merge_requests/6111
Merged-by: karthik nayak <knayak@gitlab.com>
Approved-by: karthik nayak <knayak@gitlab.com>
Reviewed-by: Patrick Steinhardt <psteinhardt@gitlab.com>
Reviewed-by: Justin Tobler <jtobler@gitlab.com>
Reviewed-by: karthik nayak <knayak@gitlab.com>
Co-authored-by: Justin Tobler <jtobler@gitlab.com>
|
|
operations: Remove SquashInGit feature flag
See merge request https://gitlab.com/gitlab-org/gitaly/-/merge_requests/6145
Merged-by: Will Chandler <wchandler@gitlab.com>
Approved-by: Will Chandler <wchandler@gitlab.com>
Co-authored-by: John Cai <jcai@gitlab.com>
|
|
|
|
Now that the git implementation has been working fine for two weeks,
let's remove the feature flag.
Also, now that the mergeWithGit2go function is unused, we can remove
that as well.
|
|
To get an estimate of the existing repositories on disk, we add a
startup task to walk all configured storages and update the
`gitaly_repository_count` metric with the number of repositories
found. This is run in a goroutine to prevent walking a very large
storage from blocking Gitaly's startup.
We avoid reporting this metric to collectors until after the walker has
finished to avoid having the value jump suddently. However, the walker
will still race with new repository creations received while it is
running and the metric will not be fully deterministic. For example:
- Repo `@hashed/ff/ff/ffff...` is created immediately after Gitaly
starts up and the metric is incremented.
- The storage walker reaches `@hashed/ff/ff/ffff...` and increments the
metric a second time.
We could drop events receiving during the walk, but it's equally likely
that the walker has already finished the new repo's directory before the
event arrives and we would then undercount. Walking should finish
relatively quickly, so this will probably not be too much of an issue in
practice.
The walker uses `ValidateRepository` to confirm if a directory is a git
repository, so in addition to the work done by `filepath.WalkDir` we are
running five stat(2) calls per repository:
- The storage's root directory
- The directory being checked
- Its objects directory
- Its refs directory
- Its HEAD file
- Its packed-refs file
A non-repository directory would perform three `stat`s, storage,
repository, and a failed check for objects.
Changelog: added
|
|
All repository creation and removal is done via `repoutil`, so
hooking `RepositoryCounter` in here will allow us to track changes
without dealing with the idiosyncracies of the many RPCs that may
create repositories.
Add a `RepositoryCounter` as a parameter to the `repoutil.Create` and
`repoutil.Remove` functions and wire it up to necessary services.
The exception to this is the `RemoveAll` RPC which deletes an entire
storage, we use the counter directly here.
|
|
Gitaly currently does not expose a way to determine the number of
repositories present on a storage. In a cluster an admin could query the
Praefect database, but this is undocumented and is not available for
non-cluster nodes.
Introduce a `RepositoryCounter` to manage the new
`gitaly_total_repositories_count` gauge. For our purposes object pools
are considered to be separate repositories and no effort is made to
associate them with their children.
The metric will include the storage name and the first relevant
directory of the repository path as `prefix`. The latter is included to
allow admins to understand which types of repositories are present. For
a standalone gitaly node `prefix` would be the first directory up
from the storage, e.g. `@hashed`, `@pools`, or `@snippets`. For a Gitaly
Cluster node it would be the child directory of `@cluster`, e.g.
`repositories` or `pools`.
|
|
The tests for the `TestCreateRepositoryFromBundle` RPC use seeded
repositories preventing execution during SHA256 repository testing. The
tests still fail with SHA256 because the Git repository initialized by
the RPC always uses the SHA1 object format.
Refactor these tests to be table driven and remove seeded repositories.
|
|
In d54bd2a5, the `CreateRepositoryFromBundle` RPC was modified to no
longer invoke the `sanitizedError` function, but was left even though it
is now only invoked by its associated test `TestSanitizedError`.
Remove this function and its test.
|
|
The `CreateRepositoryFromBundle` RPC and its supporting data types do
not currently have any documentation in their Protobuf definitions. Add
this missing documenation.
|
|
We will need to reference the '@cluster' prefix when counting
repositories on disk, but this is not set as a constant currently.
Expose the root path used in storages and join the pool and
repository paths off of that.
|
|
limiter: Implement Cgroup resource watchers
See merge request https://gitlab.com/gitlab-org/gitaly/-/merge_requests/6129
Merged-by: Justin Tobler <jtobler@gitlab.com>
Approved-by: Justin Tobler <jtobler@gitlab.com>
Reviewed-by: Quang-Minh Nguyen <qmnguyen@gitlab.com>
Reviewed-by: John Cai <jcai@gitlab.com>
Reviewed-by: Justin Tobler <jtobler@gitlab.com>
Co-authored-by: Quang-Minh Nguyen <qmnguyen@gitlab.com>
|
|
ci: Add variable to force pipeline to run
See merge request https://gitlab.com/gitlab-org/gitaly/-/merge_requests/6110
Merged-by: Toon Claes <toon@gitlab.com>
Approved-by: Toon Claes <toon@gitlab.com>
Co-authored-by: Will Chandler <wchandler@gitlab.com>
|
|
Makefile: Remove Git v2.40 configuration
See merge request https://gitlab.com/gitlab-org/gitaly/-/merge_requests/6154
Merged-by: Toon Claes <toon@gitlab.com>
Approved-by: Toon Claes <toon@gitlab.com>
Co-authored-by: Justin Tobler <jtobler@gitlab.com>
|
|
Implement functionality to automatically configure repositories with transaction's quarantine
See merge request https://gitlab.com/gitlab-org/gitaly/-/merge_requests/6032
Merged-by: Patrick Steinhardt <psteinhardt@gitlab.com>
Approved-by: Patrick Steinhardt <psteinhardt@gitlab.com>
Approved-by: Quang-Minh Nguyen <qmnguyen@gitlab.com>
Reviewed-by: Patrick Steinhardt <psteinhardt@gitlab.com>
Reviewed-by: Sami Hiltunen <shiltunen@gitlab.com>
Reviewed-by: Quang-Minh Nguyen <qmnguyen@gitlab.com>
Co-authored-by: Sami Hiltunen <shiltunen@gitlab.com>
|
|
'master'
Migrate some more RPCs to make them SHA256 compatible
See merge request https://gitlab.com/gitlab-org/gitaly/-/merge_requests/6134
Merged-by: Patrick Steinhardt <psteinhardt@gitlab.com>
Approved-by: Patrick Steinhardt <psteinhardt@gitlab.com>
Reviewed-by: Patrick Steinhardt <psteinhardt@gitlab.com>
Reviewed-by: Quang-Minh Nguyen <qmnguyen@gitlab.com>
Reviewed-by: karthik nayak <knayak@gitlab.com>
Co-authored-by: Karthik Nayak <knayak@gitlab.com>
|
|
Implement support for the SHA256 object format in the CommitSignatures
RPC.
Changelog: added
|
|
In order to extract commit signatures we scan the commit message for
"gpgsig" fields. Starting from any line with such a prefix we then scan
until we find a line that terminates the signature, e.g. an empty line.
The logic to do this is buggy though as we didn't include the trailing
space that separates the field name from its data. Consequentially, we
currently recognize all fields that start with "gpgsig" as signature
data, even if they have an arbitrary suffix like "-garbage". This suffix
is instead appended to the signed commit data that we extract.
Fix this bug by checking for the space. While at it, prepare the code
for SHA256 support already by reformatting it a bit. Signatures in
SHA256 repositories use the "gpgsig-sha256" field instead, which is how
this bug was indeed found.
Changelog: fixed
|
|
Refactor CommitSignatures tests to be fully table-driven and generate
their test data at runtime. This prepares us for adding SHA256 support
to this RPC.
|
|
Since the tests in `TestCountCommits` no longer use seeded repositories,
we can make them SHA256 compatible.
|
|
The tests in `count_commits_test.go` use seeded repositories and old
convention of testing. Migrate all the tests to table driven tests and
get rid of seeded repositories.
|
|
Since the tests in `TestCommitsByMessage` no longer use seeded
repositories, we can make them SHA256 compatible.
|
|
The tests in `commits_by_message_test.go` use seeded repositories and
old convention of testing. Migrate all the tests to table driven tests
and get rid of seeded repositories.
|
|
Commit signatures are usually denoted by a "gpgsig" field in the commit.
When using the SHA256 object format though, Git uses "gpgsig-sha256" as
field name instead. We do not recognize this field name in the catfile
package yet and will thus fail to parse such commits.
Add the logic to detect this field as well as a test to verify that we
can parse such commits as expected.
|
|
cgroup: using a noop manager on linux without cgroup
Closes #4511
See merge request https://gitlab.com/gitlab-org/gitaly/-/merge_requests/6150
Merged-by: karthik nayak <knayak@gitlab.com>
Approved-by: Quang-Minh Nguyen <qmnguyen@gitlab.com>
Approved-by: karthik nayak <knayak@gitlab.com>
Reviewed-by: Quang-Minh Nguyen <qmnguyen@gitlab.com>
Reviewed-by: karthik nayak <knayak@gitlab.com>
Co-authored-by: ZheNing Hu <adlternative@gmail.com>
|
|
This commit implements Cgroup memory resource watcher to monitor the
memory usage of the parent Cgroup. When the usage exceeds 90% of the
memory limit or the cgroup is under OOM, the watcher considers it as a
backoff event.
We target the parent cgroup only for the sake of simplicity. Observing
the memory usage of repository cgroups adds a lot of overhead. In
addition, when the parent cgroup reaches its limit, all commands are
affected. The impact of repository cgroup exceeding limit is local to
some certain repositories.
|
|
This commit implements Cgroup CPU research watcher for notifying CPU
throttling of cgroup. When configured, the adaptive calculator polls
this watcher periodically. When it returns a backoff event, the
calculator adjusts the concurrency limit accordingly.
Cgroup doesn't have an easy way to gauge how bad the CPU is percentage.
Hence, this watcher compares the recorded total throttled time between
two polls. If the throttled time exceeds 50% of the observation window,
it returns a backoff event. The watcher uses `throttled_time` (CgroupV1)
or `throttled_usec` (CgroupV2) stats from the cgroup manager.
|
|
This commit implements Stats() function to cgroup managers. This
function collects the usage statistics of the cgroups managed by the
cgroup manager. This information is fetched from cgroupfs files,
especially `cpu.stat`, `memory.events`, and some `memory.*` files.
|
|
This commit adds the Ready() function to the cgroup manager. This
function is to check if the manager is configured properly and ready to
use.
|
|
Cgroup testing suite has an utility to mock underlying cgroupfs files.
This mocking utility hard-codes the values or accepts a very limited set
of configs. This commits adds a map that allows this utility to mock
custom values of files.
|
|
Due to the lack of consideration for non-cgroup scenarios
on Linux systems, Gitaly exits with the error message
"unknown cgroup version", which hinders the usage for
these Linux users. Therefore, using noop implementation
of the cgroup manager for Linux systems without cgroup
is necessary to solve this issue.
Signed-off-by: ZheNing Hu <adlternative@gmail.com>
|
|
Operations: Remove MergeToRefWithGit feature flag
See merge request https://gitlab.com/gitlab-org/gitaly/-/merge_requests/6153
Merged-by: Justin Tobler <jtobler@gitlab.com>
Approved-by: Justin Tobler <jtobler@gitlab.com>
Co-authored-by: John Cai <jcai@gitlab.com>
|
|
Now that the git implemenmtation of MergeToRef has been running for 3
weeks without issue, let's remove the feature flag.
|
|
Now Git v2.41 is the default, remove Makefile configuration to build and
install Git v2.40.
|
|
For debugging CI it is often useful to be able to run a pipeline outside
of the context of an MR. The GitLab project has a `FORCE_GITLAB_CI`
variable that forces a pipeline to run even when the workflow
requirements are not met.
Add an equivalent `FORCE_GITALY_CI` variable to allow us to manually
run pipelines as needed.
|
|
[ci skip]
|
|
|
|
operations: make UserCreateBranch SHA256 compatible
See merge request https://gitlab.com/gitlab-org/gitaly/-/merge_requests/6109
Merged-by: John Cai <jcai@gitlab.com>
Approved-by: John Cai <jcai@gitlab.com>
Reviewed-by: Sami Hiltunen <shiltunen@gitlab.com>
Co-authored-by: Karthik Nayak <knayak@gitlab.com>
|
|
Since the tests in `TestCommitIsAncestor` no longer use seeded
repositories, we can make them SHA256 compatible.
|
|
The tests in `isancestor_test.go` use seeded repositories and old
convention of testing. Migrate all the tests to table driven tests and
get rid of seeded repositories.
|
