| Age | Commit message (Collapse) | Author |
|---|
|
[ci skip]
|
|
hooks: Call reference-transaction hook from Ruby HooksService
Closes #3034
See merge request gitlab-org/gitaly!2566
|
|
Update Nokogiri gem to v1.10.10
See merge request gitlab-org/gitaly!2567
|
|
Remove server scoped handling from coordinator
Closes #2974
See merge request gitlab-org/gitaly!2546
|
|
Introduce Locator abstraction to diff service
See merge request gitlab-org/gitaly!2559
|
|
Export GL_REPOSITORY and GL_PROJECT_PATH in `git archive` call
See merge request gitlab-org/gitaly!2557
|
|
When modifying Git references, we want to perform voting on the change
that's about to be made to enable strong consistency. While we're
currently emulating this behaviour mostly via the pre-receive hook,
eventually we'll want to fully migrate to the new reference-transaction
hook. This buys us more granularity, allows us to capture reference
updates across all commands and also allows us to abort any update.
To reach this goal, there's two steps we need to do: first we need to
always invoke Git with appropriate environment variables to make our
reference-transaction hook implementation aware of the ongoing
transaction. And second we'll need to adjust places where we manually
call out to Git hooks.
This commit addresses the second part for our Ruby sidecar, invoking the
reference-transaction hook in the hooks service. As the hook
implementation is currently hidden behind a feature flag, this commit
also sets up a separate feature flag for Ruby. If set, it knows to set
up the correct environment variables required by our hook implementation
to execute the actual logic.
|
|
Add fuzz testing to objectinfo parser
See merge request gitlab-org/gitaly!2481
|
|
Port OperationService.UserMergeBranch to Go
Closes #3069
See merge request gitlab-org/gitaly!2540
|
|
GitLab EE/CE has been updated to use v1.10.10, so this reduces gem
duplication.
|
|
To support bundling Git LFS objects within a `git archive` call, we will
need a custom LFS smudge filter that will download the LFS data. To do
this, we will need to make an API call request to check the the
requested LFS object ID actually belongs to the LFS object network, so
we need to export `GL_REPOSITORY` to the environment. `GL_PROJECT_PATH`
may be useful for debugging purposes.
|
|
In order to break dependency on the shared global config.Config
variable the Locator interface is injected into the service.
Calls that eventually lead to global config.Config var were
removed.
Part of: https://gitlab.com/gitlab-org/gitaly/-/issues/2699
|
|
Fix logging of replication events processing.
See merge request gitlab-org/gitaly!2547
|
|
This commit ports the OperationsService.UserMergeBranch RPC from Ruby to
Go, which is hidden behind a feature flag for now.
|
|
Bump labkit dependency to get mutex profiling
See merge request gitlab-org/gitaly!2562
|
|
|
|
This is part of dogfooding GitLab coverage fuzzing feature
|
|
With the reference-transaction hook having been battle-tested in
production without any issue reports, it seems safe to enable voting via
that hook by default now. So let's do it.
|
|
We should have an explicit markers of replication event
processing start and stop with the final state of the event.
It will help understand if processing of replication event
was done properly or some interruption happened as well
as identify events that take too long to be processed.
The starting log entry includes event because after successful processing
it will be removed and there is no way to check its target/source
nodes and repository. All replication log entries for event has
'correlation_id' so it is easy to get all related log entries by scanning
for it.
Part of: https://gitlab.com/gitlab-org/gitaly/-/issues/3047
|
|
transactions: Enable majority-wins voting strategy by default
See merge request gitlab-org/gitaly!2529
|
|
This commit removes server scoped request handling from the coordinator
as there are no server scoped RPCs Praefect should direct to Gitaly
nodes.
|
|
In order to check whether a reference exists inside of a Git repository,
we're executing `git log --max-count=1 $REFERENCE`. Using porcelain
commands for scripting tasks is not recommended by the Git project and
also typically a lot slower compared to using plumbing commands. As a
result, our use of git-log(1) is both surprising and inefficient here.
Convert the code to use `git rev-parse --verify $REFERENCE`. A simple
benchmark looping around `repo.ContainsRef(ctx, "refs/heads/master"`
shows that this buys us roughly a 50% performance boost compared to the
previous implementation.
The following benchmark uses git-log(1):
goos: linux
goarch: amd64
pkg: gitlab.com/gitlab-org/gitaly/internal/git
BenchmarkLocalRepository_ContainsRef-6 909 1339892 ns/op
PASS
ok gitlab.com/gitlab-org/gitaly/internal/git 1.422s
And this one uses git-rev-parse(1):
goos: linux
goarch: amd64
pkg: gitlab.com/gitlab-org/gitaly/internal/git
BenchmarkLocalRepository_ContainsRef-6 1431 815856 ns/op
PASS
ok gitlab.com/gitlab-org/gitaly/internal/git 1.325s
|
|
Change timeformat to not omit trailing 0s
See merge request gitlab-org/gitaly!2526
|
|
Per https://golang.org/pkg/time/#Time.Format:
"If the fraction in the layout is 9s, trailing zeros are dropped."
With 9's, at the top of the second the time printed in logs goes from something like:
2020-09-07 01:02:03.999Z
to
2020-09-07 01:02:04Z
then back to
2020-09-07 01:02:04.001Z
This can be problematic for parsers (e.g. fluentd) which assume a consistent format including the milliseconds (currently we're dropping about 1/1000 of the gitaly logs on gitlab.com because of this). By using 0s in the format string we always get the sub-second portion, and log parsing will be kept happier
|
|
Upgrade sentry-raven and Faraday to v1.0.1
See merge request gitlab-org/gitaly!2533
|
|
Include grpc_service in gitaly_service_client_requests_total metric
See merge request gitlab-org/gitaly!2536
|
|
This mirrors the changes made in GitLab EE/CE in
https://gitlab.com/gitlab-org/gitlab/-/merge_requests/41623 and
reduces Gem duplication.
This commit also fixes the Sentry
tests. https://github.com/getsentry/raven-ruby/pull/928 removed the
extraneous exception message in the event, so we can drop this.
|
|
This change is part of a larger effort to move away from using the hooks
being implemented in Ruby, and moves the execution to Go to speed up the
hooks.
The prereceive hook was already enabled by default, meaning that the
change made doesn't influence anyone that hasn't explicity turned it
off. Given there were no bug reports, the old implementation is removed
as fallback.
|
|
|
|
Improved SQL to get up to date storages for repository.
See merge request gitlab-org/gitaly!2514
|
|
Pass correlation_id over to gitaly-ssh
See merge request gitlab-org/gitaly!2530
|
|
When gitaly-ssh is called it doesn't receive correlation_id.
That is why it is hard to understand what to what chain of calls
this call is related.
correlation_id passed as env var 'CORRELATION_ID' to gitaly-ssh
process. So it would be picked by it and used in outgoing requests.
In order to cover other missing parts where correlation_id should
be passed the client.DialContext method includes interceptors for it
by default. If correlation_id is present in the context.Context used
to invoke the method it will be passed to the remote.
Part of: https://gitlab.com/gitlab-org/gitaly/-/issues/3047
|
|
In order to get our feet wet with transactions, we initially enabled
transactions with a primary-wins voting strategy. This means that as
long as the primary casts a vote, the transaction is guaranteed to
be committed. This of course doesn't buy us any consistency guarantees,
but it did allow initial testing of the code without too much risk.
Given that primary-wins feature flag has been disabled in production for
some time now without any outfall at all, let's change its default value
to "disabled" to enable the majority-wins voting strategy by default.
|
|
Upgrade the Git version used by our Dockerfile to v2.28.0.
|
|
Makefile: Avoid Git2Go being linked against stale libgit2
See merge request gitlab-org/gitaly!2525
|
|
Even though we're rebuilding libgit2.a on Makefile changes, this won't
update Git2Go. As a result, there may be a stale version of Git2Go in
the module cache that's linked against an old static archive of libgit2.
Fix this issue by always rebuilding Git2Go in case libgit2 gets rebuilt.
|
|
DowngradeAttemptedError is not correctly handled in defaultReplicator.
This commit fixes the issue and adds logging for cases when a downgrade
was actually attempted.
|
|
Fix stale connections to Praefect due to keepalive policy
See merge request gitlab-org/gitaly!2511
|
|
Transactional voting via reference-transaction hook
See merge request gitlab-org/gitaly!2509
|
|
The SQL query to get up to date storages of the repository got
updated to one that shows better results. The benchmark test
added to check impact in future. The optimization is needed
as the resource consumption by the Postgresql instance increases
the more read operations praefect receives to serve.
It addresses the problem only partially as we should reduce
the load to database more with applying other optimizations.
Part of: https://gitlab.com/gitlab-org/gitaly/-/issues/3053
|
|
Replication job acknowledge removes 'completed' and 'dead' events.
Closes #2941
See merge request gitlab-org/gitaly!2457
|
|
hooks: Remove update feature flag
See merge request gitlab-org/gitaly!2501
|
|
Migration script to remove all redundant rows from replication_queue.
It is safe to re-run migration if it fails. It is designed to remove
rows in batches by 100000 each iteration. Each removal happens in
separate transaction. The script removes only 'dead' and 'completed'
replication events.
Closes: https://gitlab.com/gitlab-org/gitaly/-/issues/2941
|
|
|
|
Automatic repository reconciliation
Closes #2717
See merge request gitlab-org/gitaly!2462
|
|
[ci skip]
|
|
The reference-transaction hook included in Git v2.28.0 is currently
broken with interleaving hook calls, where the wrong hook would start to
be executed with parameters of the reference-transaction hook. As a
result, the commit introducing it got reverted.
At the point of writing, there is no release of Git including the fixed
version of that hook yet, and chances are good we'll have to wait
another two months until Git v2.29.0. To unblock ourselves, a workaround
has been added in c9d0ae5a (gitaly-hooks: Fixup reference-transaction
hook name based on arguments, 2020-08-31). Together with the testcase in
this commit's predecessor, we can now include the reference-transaction
hook again and be sure to not trigger the original issue anymore.
This reverts commit 19e2caa3a8a9fe390b568dd8d2b2a565be6094a7 and wires
up the reference-transaction hook again.
|
|
Fixup reference-transaction hook name based on arguments
See merge request gitlab-org/gitaly!2506
|
|
Replication jobs are currently only attempted three times at maximum
until they are dropped. After a replication job is dropped, its changes
are not going to be replicated until another replication job is triggered
form another mutator operation or an administrator runs reconcile manually.
Repositories can stay outdated for an indefinite time.
This commit addresses the problem by adding a background goroutine that
automatically schedules replication jobs from storages which have a fully
up to date copy of the repository to storages with outdated copies of the
repository. Replication jobs are only scheduled from and to storages which
are currently healthy to avoid scheduling replication jobs that are not
going to be successfully completed. If there is a replication job already
targeting the repository, a redundant reconciliation job won't be scheduled.
|
|
While the reference-transaction hook was released with Git v2.28.0, it's
got a bug which will cause Git to accidentally execute the wrong hook if
there's been interleaving calls to both the reference-transaction and
another hook. This bug currently prohibits us to use the hook for
reference transaction voting until a new Git version is released with
the upstreamed fix. Given that this could be as late as release v2.29.0,
it's potentially still quite a while until we can use the hook.
As a temporary workaround, we may manually fixup hook names in our
gitaly-hooks binary. GitLab systems only have at most four hooks
installed: reference-transaction, pre-receive, update and post-receive.
Because of how these hooks get called, we should be able to tell the
case when Git meant to execute the reference-transaction hook instead of
any of the other three by inspecting the hook's arguments:
- reference-transaction will receive as first argument one of
prepared, committed or aborted
- pre-receive never gets any arguments
- update will get as first argument the reference that is to be
updated in its fully-qualified form (e.g. refs/heads/master) and
thus should never clash with any of the three above verbs
- post-receive never gets any arguments
So none of these hooks except for the reference-transaction hook will
ever get executed with one of prepared, committed or aborted. If they
were executed with such an argument, we can be sure that Git wanted to
execute the reference-transaction hook instead.
Let's work around the bug by mangling arguments in case we're called as
a hook with exactly a sinlge argument which is either "prepared",
"committed" or "aborted".
|
