| Age | Commit message (Collapse) | Author |
|---|
|
In the past we required a workaround for Linguist so that it picked up
the correct Git executables. We have since converted the code to not use
`git-linguist` anymore, but to use our own `gitaly-linguist` wrapper.
And in fact, while `git-linguist` did use Git to verify that the repo
exists, the Linguist Gem exclusively uses Rugged to access repos. As a
consequence, `gitaly-linguist` doesn't use Git at all anymore.
Remove the workaround to clean up the code. Note that we'd detect the
case where we did unexpectedly execute Git here because we inject our
own `PATH` in the testhelper code that has broken `git` wrappers in it.
So if we did still execute it we'd see that tests failed.
|
|
We're using the `git-linguist` binary to derive programming-language
statics for a repository. This binary is using Rugged to read a given
commit and analyze all blobs referenced by the root tree so that it can
return accumulated lines of counts for every language.
Unfortunately, `git-linguist` reads in gitconfig files by default with
no escape hatch, which sabotages our efforts to get gitconfig-clean in
the Gitaly codebase. We are thus forced to implement our own wrapper
script around the Linguist Gem that allows us to ignore the gitconfig in
Rugged.
Do so and implement a new `gitaly-linguist` binary that mostly mirrors
what `git-linguist` is doing. This allows us to override the Rugged
search path and point it to `/dev/null` so that it won't read any
gitconfig files at all.
Note that we do not use e.g. Gitaly's own gitconfig as computed by the
Git command factory. Ultimately, the Linguist Gem does not read any Git
configuration that would change its behaviour, so it would be overkill
to do so.
Changelog: changed
|
|
When running Ruby commands we need to make sure to expose certain
environment variables to the spawned processes so that they have a valid
Ruby execution environment. This logic is required both by Linguist and
by the Ruby server, but they duplicate the logic.
Create a new `env.AllowedRubyEnvironment()` helper function to unify
this knowledge into a single place. While at it, remove the logic to
handle the generic environment variables `HOME` and `PATH` from
Linguist: they are already injected by our `command` package.
|
|
Makefile: Fix install target using doubly-prefixed Gitaly paths
See merge request gitlab-org/gitaly!4553
|
|
The Go build chain doesn't write a GNU build ID into executables in case
the executable format is not ELF. This is most importantly the case on
macOS, which means that our `replace-buildid` tool fails to work on that
platform because it cannot find the GNU build ID.
Let's fix this by only trying to replace the build ID on Linux.
|
|
With 17f3bf27b (Makefile: Use pattern rules to build Go binaries,
2022-05-10), we have converted the `GITALY_EXECUTABLES` variable to
contain absolute paths instead of only the executables' basenames. The
`install` target was still trying to prefix these binaries though, which
ultimately led to a path which had the binary directory's prefix twice
and thus broke installation.
Fix this bug by not reapplying the prefix.
Changelog: fixed
|
|
into 'master'""
This reverts commit 933109e3e849358daf78c4618d07091b3f36068f.
|
|
'master'
praefect: Remove logic to handle maintenance-style replication events
Closes #4185
See merge request gitlab-org/gitaly!4575
|
|
Upgrade Gitaly module to 15.0
See merge request gitlab-org/gitaly!4493
|
|
Makefile: Fix protoc compile option to disable building tests
See merge request gitlab-org/gitaly!4579
|
|
This commit changes the major version in the package name from v14 to
v15
Updating go.mod & go.sum with new module name v15
Update Makefile to bump major version to v15
Update the gitaly package name in the Makefile. Also update
gitaly-git2go-v14 -> gitaly-git2go-v15. We need to keep
gitaly-git2go-v14 for a release however, for zero downtime upgrades.
This pulls directly from a sha that is v14.
Update package name from v14->v15 for auth, client, cmd, internal packages
This commit changes the package name from v14 to v15 in go and proto
files in the internal, auth, client, cmd packages.
proto: Update major package number in package name
tools: Change major version number in package name from v14 to v15
gitaly-git2go: Change the package name from v14 to v15
update module updater for v15
Update the documentation for the module updater to reflect v15
|
|
[ci skip]
|
|
The Protobuf project has seemingly renamed their CMake options to all
have a `protobuf_` prefix. While the deault options we set still work,
this broke the `-DBUILD_TESTS=OFF` switch.
Fix this and adapt accordingly to significantly reduce build times of
this project. On my machine, it decreases from about 58.9s to 22.8s.
|
|
[ci skip]
|
|
Remove the fallback logic that is handling maintenance-style replication
events. We don't generate any such events since v15.0 anymore, and we do
have a migration in place that ensures that any stale events are removed
from the replication queue. Meaning that we now shouldn't ever see any
such event in the wild anymore.
|
|
In release 15.0 we have stopped creating maintenance-style replication
events in favor of using a best-effort strategy for handling repository
maintenance. And while we still have the logic in place to dequeue these
events, they're currently ignored by Praefect altogether.
We're about to remove this fallback logic now, so let's make clear that
the replication queue really doesn't have any maintenance-style events
left by adding a migration that prunes any stale ones.
|
|
supervisor: Fix leaking logrus Goroutine on spawn failure
Closes #3927
See merge request gitlab-org/gitaly!4567
|
|
cmd/praefect: Improve style of test for removal of replicaton jobs
Closes #3944
See merge request gitlab-org/gitaly!4569
|
|
proto: Add DeleteRefErrors
See merge request gitlab-org/gitaly!4501
|
|
operations: Fix flaky test for known-to-fail requests in UserApplyPatch
Closes #4161
See merge request gitlab-org/gitaly!4573
|
|
cgroups: Handle nil repo
See merge request gitlab-org/gitaly!4572
|
|
The test we have which verifies that removal of replication jobs works
as expected for the `praefect remove-repository` subcommand is quite
hard to read. Furthermore, many of the conditions aren't as strict as
they could theoretically be.
Refactor the test to match modern coding style and tighten it to assert
conditions that should be true. This hopefully helps to nail down why
our CI is hanging from time to time in this job.
|
|
operations: Remove SquashUsingMerge feature flag
See merge request gitlab-org/gitaly!4563
|
|
We have observed that one of our tests for `UserApplyPatch` is flaky:
from time to time we receive an unexpected EOF while streaming the patch
data to the server side. As it turns out, this flakiness is our own
fault: we send a request header that is known to cause failure, which
means that the server will return an error as soon at it realizes that.
This is effectively causing a race between the client, which is trying
to send the patch data to the server, and the server which is realizing
the error and thus aborting the RPC call.
Fix this flake by stopping on EOF so that we can test for the real error
condition via `CloseAndRecv()`.
|
|
If a command's repo is nil, handle it gracefully by adding the command
to a cgroup by using the command arguments as a key for the circular
hash.
While we're at it, fix a test that was doing the checksum incorrectly.
This didn't make a difference in the test however, because we were using
the groupID we calculated ourselves to check for the location of the
cgroup file.
Changelog: fixed
|
|
featureflag: Remove ConcurrencyQueueMaxWait feature flag
See merge request gitlab-org/gitaly!4564
|
|
commandstatshandler: Fix flaky test caused by Git version check
Closes #4239
See merge request gitlab-org/gitaly!4568
|
|
ruby: Reduce usage of Git commands
See merge request gitlab-org/gitaly!4559
|
|
In our commandstatshandler we're asserting that when executing RPCs, we
record the number of spawned commands. To do so we use an RPC that
spawns a Git command. Due to recent changes in our Git command factory
we had to adapt the test though as we are now additionaly spawning a
git-version(1) command to auto-detect the version.
This change is causing flakiness though because we cache the Git
version per binary that is executed: we don't reuse the context, and
when running with bundled Git we now thus randomly use either the old or
the new bundled Git binaries based on a feature flag that uses a random
value. So based on whether this value is the same across both RPC calls
or not we may or may not re-execute git-version(1).
Fix this by reusing the same context for both RPC calls to guarantee
that the feature flag has the same state.
|
|
We cannot really exhaustively test some of our feature flags because
they're hit in almost every code path. To still heuristically hit both
code paths with the feature flag turned on and off we're thus using a
random value instead.
This randomness is seeded with the current time in milliseconds. Which
effectively means that even if a test uses multiple different contexts,
the likelihood that both will see the same feature flag value is rather
high because it's likely they'll run in the same millisecond. This is
hiding flakiness that happens in case a test implicitly relies on the
fact that the feature flag value must be the same across subtests when
by accident we're using multiple different contexts.
Let's inject more randomness to cause such tests to be a lot more flaky
than they are right now by instead using nanosecond resolution for the
randomness' seed.
|
|
rubyserver: Ensure we sync objects if there is no Rugged gitconfig
Closes #4232
See merge request gitlab-org/gitaly!4560
|
|
For a long time we have observed leaking Goroutines in our supervisor
test which verifies that the circuit breaker kicks in in case spawning
of the process fails. As it turns out, this is a real issue: we set both
stdout and stderr of the new command to an `io.PipeWriter` created for
logging purposes. And because the Go runtime doesn't automatically close
these writers for us, we must make sure to do so ourselves. And while we
do that in case the command has been spawned successfully, we don't
handle the case where spawning has failed and thus don't close it at all
here.
Fix this Goroutine leakage by closing the `io.PipeWriter` in case we
fail to start the command.
Changelog: fixed
|
|
We use different loggers for both stdout and stderr of processes spawned
by our supervisor. And while these loggers aren't any different, this
means that we now also have two Goroutines copying the process's output
into the loggers.
Let's improve this and only use a single Goroutine by reusing the same
log writer for both streams.
|
|
featureflag: Turn RunCommandsInCGroup to default false
See merge request gitlab-org/gitaly!4566
|
|
|
|
We can remove this feature flag now that we've been running in
production with this flag on for a few weeks now without issue.
|
|
We want to return a structured error from DeleteRefs when certain errors
happen. Currently we are returning nil for the error, which leads to
unnecessary Praefect replication jobs.
|
|
git: Sync internal namespaces with distros' settings
Closes #4237
See merge request gitlab-org/gitaly!4562
|
|
Update "Support Request" template in line with Gitaly's intake workflow.
See merge request gitlab-org/gitaly!4552
|
|
Handle repository creations, deletions and renames atomically
Closes #4003 and #3485
See merge request gitlab-org/gitaly!4101
|
|
The `#head_symbolic_ref` function is using git-symbolic-ref(1) to read
the repo's `HEAD` symbolic reference. We're slowly phasing out all
executions of the Git command line though, and we can easily implement
it via Rugged.
Convert the function to use Rugged to read the reference. This allows us
to get rid of the now-unused `#run_git` function, and furthermore we
don't have to inject configuration for git-symbolic-ref(1) into the
sidecar anymore.
|
|
Now that we don't fetch from remote repositories anymore there are no
callers left that use the `RemoteRepository` class. Remove it.
|
|
The `start_branch` parameter from `#with_branch` could be used to
specify which branch the new branch should have been created from in
case it didn't exist yet. The only caller that still sets this is
setting it to the same value as `branch_name` though.
Remove this parameter and instead consolidate it into `branch_name` to
simplify the code.
|
|
Remove the unused `force` parameter from `#with_branch`.
|
|
Remove the unused `start_sha` parameter from `#with_branch`.
|
|
The `#with_branch` function can optionally fetch from a remote repo in
case the requested branch or commit does not yet exist in the local one.
It only has a single caller left though, and that caller doesn't ever
set the `start_repository` parameter to anything. Consequentially, we
know that it's always going to be the local repo anyway, making the
functionality unused.
Remove the ability to fetch from a start repository and the now-unused
`#fetch_sha` function and stop injecting configuration for git-fetch(1)
into the sidecar.
|
|
The Gitaly config exposes a `rugged_git_config_search_path` config that
allows the administrator to configure where Rugged should search for the
gitconfig it is supposed to use. While this option rather feels like an
edge case, it is in fact currently mandatory to set this configuration
to a gitconfig that sets `core.fsyncObjectFiles=true`. If there is no
such configuration, then the result is that Rugged will not flush newly
written objects to disk, which can easily lead to repository corruption.
Luckily, both CNG and Omnibus know to set up this config as expected.
But this brings its own problem with it: we have `core.fsyncObjectFiles`
set in a central location now by distributions, which also means that
Git as spawned by our Git command factory picks it up. But because that
option has been deprecated in Git v2.36.0 this means that all Git
commands would now print a warning. So we're between a rock and a hard
place now: we must make sure that the configuration is set in Rugged,
but it must not be present in the Git configuration.
Given that this is the only configuration that Rugged really needs, and
given that it must always be present or we otherwise corrupt repos, we
can fix this issue by simply writing our own Gitconfig file in case
`rugged_git_config_search_path` is unset. Like this, we can stop setting
the search path in distributions and eventually deprecate its use.
|
|
When stopping the Ruby server, we need to first stop all of the workers.
But because we first stop the worker's monitor before we actually stop
the worker's process we're not consuming the event that is generated on
shutdown of the worker anymore. This leads to a Goroutine leak because
we're blocked waiting for the monitor to consume the event, but that's
never going to happen.
Fix this Goroutine leak by first stopping the process, and only stopping
the monitor after that to ensure the event can still be read. This is in
preparation for a set of tests we're going to add in the Ruby server
which would otherwise trigger this leak.
|
|
ruby: Fix diverging Git configuration in Go and Ruby
Closes #4236 and #4230
See merge request gitlab-org/gitaly!4557
|
|
[ci skip]
|
