| Age | Commit message (Collapse) | Author |
|---|
|
Hooks up the error tracker in the node manager so it checks if a certain
backend node has reached a threshold of errors. If it has, then it will
be deemed unhealthy.
|
|
Fix semantic merge conflict
See merge request gitlab-org/gitaly!2409
|
|
|
|
Remove Praefect primary from config
See merge request gitlab-org/gitaly!2392
|
|
|
|
|
|
Praefect: storage scoped RPCs not handled properly
Closes #2442
See merge request gitlab-org/gitaly!2234
|
|
Scope of the FindRemoteRepository RPC call changed to STORAGE.
Storage value of the incoming message can be changed by new
'SetStorage' method.
Human-readable string for 'Scope' type with default string formatting.
Closes: https://gitlab.com/gitlab-org/gitaly/-/issues/2442
|
|
Support dry-run cherry-picks and reverts
See merge request gitlab-org/gitaly!2382
|
|
lines.Send() only support byte delimiter
Closes #2846
See merge request gitlab-org/gitaly!2402
|
|
Add old path to NumStats protobuf output
See merge request gitlab-org/gitaly!2395
|
|
Add Git v2.28.0-rc1 to the test matrix
See merge request gitlab-org/gitaly!2399
|
|
Document hook execution paths
See merge request gitlab-org/gitaly!2391
|
|
|
|
This will make it possible to solve some N+1 issues in the
/internal/allowed endpoint.
Relates to https://gitlab.com/gitlab-org/gitlab/-/issues/227572
|
|
|
|
[ci skip]
|
|
|
|
|
|
[ci skip]
|
|
During the work on pagination for FindAllLocalBranches in
e3b1d6e5d2379977de477a2614f960d4f19e6dea I ran into buggy behaviour when multi
line delimiters were used in `lines.Send()`. One example: if a delimiter is
twice the same character, it would unsuccessfully detect the last delimiter and
thus send invalid results to a client.
Luckily there's no call site that actually uses multi byte delimiters,
which allows this change to remove support for it.
|
|
In order to let us allow the new reference-transaction hook, let's
add Git to v2.28.0-rc1 to the test matrix. As there is currently only a
release candidate available, the conversion does not yet go all-in but
adds a single test job, only.
|
|
Fix service registration after start
See merge request gitlab-org/gitaly!2398
|
|
Expose transactions to callers directly
See merge request gitlab-org/gitaly!2396
|
|
Fix push options not working with Gitaly post-receive hook
Closes #2970
See merge request gitlab-org/gitaly!2394
|
|
Praefect: include PgBouncer in CI
Closes #2816
See merge request gitlab-org/gitaly!2378
|
|
Fix context cancellation issues with datastore
See merge request gitlab-org/gitaly!2397
|
|
This will perform a cherry-pick or revert with the specified arguments,
but not actually create a resulting commit. This allows a user to check
for a `CreateTreeError` on an operation but not modify the repository.
One example use case is in the Delivery team. We want to know if a
security merge request targeting `master` will cherry-pick cleanly to
the previous stable branches. If it does, we can simply cherry-pick it
prior to release; if it doesn't, we need to inform the developer that
they need to create a backport.
|
|
|
|
|
|
|
|
When the context for `StartHealthUpdate()` got canceled, we do not want
to return an error but instead just stop the health updating loop and
return. We need to treat context cancellation specially when executing
SQL queries, though, as the pq driver will not return `context.Canceled`
or `context.DeadlineExceeded` errors, but instead a custom error code
"query_canceled".
Fix detection of context cancellation by checking for any `pq.Error`s
with the mentioned error code.
|
|
One of the datastore tests verifies that if we trigger the health
updater twice, that the job lock's timestamp got updated. The way we
run the health updater is racy, though, as we do not verify that it
actually exits before the test case finishes. As a result, it may
execute queries against the database while the follwing test truncates
all tables already.
Let's fix this race by synchronizing shutdown of the Goroutine.
|
|
Repository state tracking stores
See merge request gitlab-org/gitaly!2380
|
|
It's not allowed to register any more services with a GRPC server after
it's been started already. But due to a race, we may do exactly that
when registering new mocked SmartHTTP servers: `NewServerWithHealth()`
not only creates a new GRPC server with the health service registered,
but already starts it. As we still need to register the SmartHTTP server
afterwards, we're thus racing against the server's `Serve()` Goroutine.
Fix this race by assembling the server manually before calling
`Serve()` on it.
|
|
The Ruby implementation passes `push_options[]` because it uses a
Content-Type of `application/x-www-form-urlencoded`, but the Go
gitlab-shell client passes a body with `Content-Type` of
`application/json`. Because the JSON payload is parsed literally, the
key needs to be `push_options` instead of `push_options[]`.
Closes https://gitlab.com/gitlab-org/gitaly/-/issues/2970
|
|
The transaction state is currently returned whenever one calls the
transaction cleanup function. This is an artifact caused by the
transaction type not being publicly visible earlier, but it starts to
become a hinderance when implementing new features and querying
transacton state. So let's just stop returning the state from the
cleanup function and ask callers to retrieve the state by themselves.
|
|
Right now, `RegisterTransaction()` will return the registered
transaction's identifier instead of the transaction itself. This makes
it kind of unwiedly to retrieve further information about the the
transaction. Now that the `Transaction` type became public, let's
instead return the transaction itself instead of a proxy value.
|
|
Currently, users of transactions only interact with a transaction via
the proxy of its identifier. While it worked just fine in the beginning
where transactions didn't really do a lot, they start to become more
complex and thus carry more information that may be of interest to the
user.
So let's make the transaction type public as a preparatory step to
returning it directly when registering a transaction instead of
returning its identifier, only. In order to avoid abuse of transactions,
only accessors are made public.
|
|
Currently, all state about a transaction is reported on transaction
cancellation. While this is the natural point in time when it should
happen, it is an awkward interface to have the state be returned by the
cancellation function. So let's split up cancellation of transactions
and state retrieval such that callers interestend in the state need to
call `transaction.State()` after the transaction was cancelled.
|
|
Subtransactions histogram
See merge request gitlab-org/gitaly!2390
|
|
Support parsing of Git release-candidate versions
See merge request gitlab-org/gitaly!2389
|
|
Adds an in-memory and a Postgres implemenation of repository
state tracking stores. The two tables added are:
1. `repositories` which contains the expected state of repositories
in the virtual storage.
2. `storage_repositories` which contains the state of the repositories
on the physical storages.
Cross-referencing these two makes it easier to identify outdated
repositories.
This commit implements only the stores without hooking them up to the
rest of the code.
|
|
Wire up the new subtransactions histogram metric for the transaction
manager.
|
|
With our new support for subtransactions, we cannot quite tell how many
votes there are per created transaction. While we do know how many
subtransactions there are, we cannot correlate a single transaction with
its subtransactions. This is quite an interesting metric though to see
how often the reference-transaction hook gets executed per transaction
and to better estimate the overhead of reaching out to the transaction
manager repeatedly.
Let's add a new histgoram that allows us to record the number of
subtransactions per transaction.
|
|
In some places, we need to check whether Git supports a certain feature
or not. As Git doesn't usually provide a direct way to test for a
feature, we instead use the version number as a proxy. This works just
fine for normal releases by just parsing the semantic version number
as a tuple of integers in the format "major.minor.patch". But we do not
correctly cope with release candidates like "2.28.0.rc0", returning an
error.
Fix this by expecting an optional fourth touple, setting a flag inside
the version structure that says that it's a release candidate.
Furthermore, we also need to extend the version comparison code: a
release candidate is always older than its respective release.
Note that we do not parse the actual release candidate number. As
numbering is zero-indexed, we cannot simply use "0" as a placeholder for
a non-release-candidate, and everything else will require additional
logic. As it's quite unlikely that we'll ever need to verify whether
a given version is a specific release candidate (new features aren't
usually introduced in between rc's and they shouldn't run in production
anyway), adding this additional logic doesn't seem to be worth it at
all.
|
|
Allow multiple votes per transaction
Closes #2939
See merge request gitlab-org/gitaly!2386
|
|
Update design documentation for transactions
See merge request gitlab-org/gitaly!2385
|
|
Exported errors in the transaction code didn't yet have any
documentation, which this commit amends now.
|
|
Right now, we're using the pre-receive hook to call into transactions on
the Gitaly nodes. This hook is well-understood to be executed once and
only once per action, and as a result the current implementation of
transactions allowed for a single vote via this pre-receive hook, only.
The pre-receive hook was a stop-gap implementation of the real mechanism
we wanted to eventually use, though, which is the reference-transaction
hook that's going to be released as part of git-core v2.28. While the
input to both hooks is the same and thus no changes to the actual voting
logic should be required, the most important difference is that the
reference-transaction hook may be invoked arbitrarily many times for
each Git command. E.g. a non-atomic push will execute the hook as many
times as there are updated references.
This surfaces a current design limitation of the transaction mechanism
as it is implemented in Gitaly: as a transaction only allows for a
single vote per node, it's inherently incompatible with the semantics
introduce by the reference-transaction hook. This is why we now extend
the transaction mechanism to allow a sequence of votes instead by
introducing subtransactions.
Subtransactions encapsulate all the voting logic that was previously
part of the transaction, which is casting the vote and collecting the
votes to establish whether quorum was reached. Transactions now start to
be a wrapper of a set of transactions, which transparantly creates new
subtransactions as required whenever a node casts a vote. Whenever a
node casts a vote, one of three things may now happen:
1. A subtransaction exists where the node's status is "aborted". This
means that any one of the subtransactions failed and thus the
complete transaction needs to be labelled as "aborted" for the
node. It will receive an error and is not allowed to cast any
votes anymore.
2. A subtransaction exists where the node's state is "undecided". As
this means that the node simply didn't cast a vote, it will cast
its vote for the oldest undecided subtransaction and wait for
quorum to be reached.
3. All existing subtransactions are in "committed" state for the
given node. We'll thus create a new subtransaction, cast our vote
for this new transaction and wait for quorum to be reached.
To evaluate the complete transaction's outcome, we need to establish
whether for a given node, all created subtransactions are in state
"committed". Only if that's the case will the transaction be treated as
successful for the node.
With this logic, we're now able to transparently allow a voter to cast a
sequence of votes.
|
