| Age | Commit message (Collapse) | Author |
|---|
|
Since the inception of our `make git` target we have always passed the
`NO_INSTALL_HARDLINKS=YesPlease` build option. This option causes the
Git build system to create copies of the Git binaries instead of hard
links.
It is not clear why we should need to set this option though as it only
increases the size of the installed distribution by about 25MB without
much merit. So let's stop passing the build option so that Git's build
system can decide for itself how it should install the binaries.
|
|
Building either our full Git distribution or the bundled Git versions
always uses a `.version` file to verify whether we need to rebuild the
target or not: only if the commit, build options or patches have been
modified since the last time the `.version` file was generated will we
actually rebuild Git. This is an important optimization and works quite
well.
One case where it doesn't work well right now though is when building
different versions of distributed Git. Depending on which version the
user wants to build via `GIT_VERSION`, we use different directories to
build Git and thus also use different `.version` files. This has a catch
though: because the `.version` file will not be updated again when both
Git versions have already been built successfully, we now think that we
don't need to rebuild Git because its `Makefile` is still newer than the
`.version` file when we switch between versions.
Fix this bug by always using the same directory to build distributed
Git, regardless of its version. While this means more rebuilds, it's
simple and correct.
|
|
Makefile: Drop bundled Git v2.33.1.gl3
See merge request gitlab-org/gitaly!4495
|
|
Instrument verification worker for observability
Closes #4097
See merge request gitlab-org/gitaly!4473
|
|
smarthttp: Add finalizing vote in PostReceivePack
Closes #4110
See merge request gitlab-org/gitaly!4488
|
|
operations: Remove feature flag for transactional voting in UserSquash
Closes #4119
See merge request gitlab-org/gitaly!4496
|
|
Ignore verification columns for read-only cache updates
Closes #4159
See merge request gitlab-org/gitaly!4468
|
|
When all updates are rejected, there will be no votes cast. This leads
to an unnecessary replication job. In this case, we want to cast a last
but empty vote. This is the same as how SSHReceivePack works.
Changelog: changed
|
|
sidechannel: lower yamux close timeout
See merge request gitlab-org/gitaly!4491
|
|
This is a preperatory commit so that PostReceivePack can do transaction
voting in the RPC handler. This adds a transaction manager to the server
struct to be used by PostReceivePack.
|
|
With af4ea3258 (operations: Fix missing votes on squashed commits,
2022-03-18) we have introduced the use of quarantine directories in the
`UserSquash()` RPC to enable transactional voting. This change in
behaviour has been rolled out to production, and the change was enabled
by default in e8413304d (operations: Always use structured errors for
UserSquash, 2022-03-21).
We haven't seen any issues with this change. Remove the feature flag to
always enable it.
Changelog: removed
|
|
We have finished the migration to bundled Git v2.35.1.gl1 in v14.10. Due
to concerns with zero-downtime upgrades we couldn't yet remove the old
version though. But now that we have waited for a release we can finally
remove the old version.
Remove the infrastructure to build and install bundled Git v2.33.1.gl3.
Changelog: removed
|
|
The test repository helper in our rspecs is used to create temporary
repositories for testing purposes. The Git version it uses for this
purpose depends on a set of environment variables, but in case it runs
with bundled Git it needs to some more bootstrapping of an execution
environment in order to be able to properly clone the repository.
The bundled Git version is currently hard-coded to `gitaly-git`, which
we're about to phase out in favor of `gitaly-git-v2.35.1.gl1`. With the
removal of the old version the helper is thus about to break.
Fix this by switching over to the newer bundled Git version. While we
could do fancy things like injecting the version or auto-detecting it,
it doesn't really feel worth it over simply hard-coding it. We're
ultimately going to retire the sidecar over the next few releases, so we
shouldn't spend more time than necessary on maintaining it.
|
|
repository: RestoreCustomHooks to do transaction voting
Closes #4081
See merge request gitlab-org/gitaly!4481
|
|
[ci skip]
|
|
Lower the yamux StreamCloseTimeout setting from 5 minutes to 1 second
for sidechannel connections.
The motivation for changing this is to make it easier to test client
side cancelation in Workhorse tests.
|
|
Fix typo in documentation of UserCommitFilesRequestHeader
See merge request gitlab-org/gitaly!4422
|
|
Remove Maintenance routing feature flag
See merge request gitlab-org/gitaly!4486
|
|
[ci skip]
|
|
The verification worker is already collecting the metrics on how
many jobs it's processing. Praefect is still lacking a metric for
the overall verification queue depth which would be very helpful
in determining how much work is there left for the verification to
be completed. This commit adds a collector for the verification
queue depth. The metric is collected for each storage separately.
The metric differentiates between replicas that are unverified and
replicas that have been verified but the verification has expired.
|
|
This commit adds metrics on the verification worker so the process
becomes more observable. The metrics track the number of dequeued
jobs per storage and then number of completed jobs per storage with
their results. The number of stale leases releases is also tracked.
This gives insight into how the verification work is progressing.
|
|
Release expired verification leases periodically
See merge request gitlab-org/gitaly!4478
|
|
[ci skip]
|
|
RestoreCustomHooks changes data in a repository, so it should do voting.
Otherwise, each time we create a replication job for it, which is
useless because we don't replicate hooks.
Add transactional voting to the RestoreCustomHooks RPC by integrating
with the new LockingDirectory.
Changelog: changed
|
|
Sometimes it's useful to be able to lock a directory before modifying
it. This is similar to LockingFileWriter, except it just gives the
ability to lock a directory so another process cannot also lock the
directory.
This will be immediately useful in RestoreCustomHooks where we are about
ot add transactional voting to it. With transactional voting, the
semantics are that the `Prepared` vote is made once the data that is
about to be changed is locked.
This will enable us to lock the custom hooks directory.
Changelog: added
|
|
Since the maintenance routing feature flag has been running in
production without issue, we can now remove it.
Changelog: changed
|
|
The background verifier sets a lease time on a replica when it picks
it up for verification. If the worker dies for some reason, the lease
will remain in place and no other worker will pick up the replica for
verification again until the lease is cleared. The lease itself tells
the maximum time the worker itself would be working on the replica.
After it has been passed, it would be safe for another worker to pick
up the replica for verification again. This commit adds a background
goroutine that periodically releases expired leases so other workers
can take up the work if the original worker failed and did not release
the lease. The 'verificaton_leases' index is added so the query can
efficiently find the replicas with leases acquired to find the stale
ones.
|
|
Update version of danger-files dependency
See merge request gitlab-org/gitaly!4485
|
|
|
|
docs: Document Gitaly backpressure
See merge request gitlab-org/gitaly!4469
|
|
There are a number of knobs in Gitaly to tune backpressure Gitaly can impose on services that call it. This commit documents these.
|
|
Add support for FIPS encryption
See merge request gitlab-org/gitaly!4482
|
|
repocleaner: Allow NewWalker to receive grace period parameter
Closes #4164
See merge request gitlab-org/gitaly!4474
|
|
A default grace period of 6 hours is sufficient for the subcommand.
|
|
Implement 'praefect verify' subcommand
Closes #4091
See merge request gitlab-org/gitaly!4463
|
|
Praefect periodically verifies the repository metadata in the background.
The interval may be too long especially if there was an incident, say
a disk failure. After recovering the Gitaly node, the disk may be completely
empty or may contains an older snapshot which does not contain all expected
repositories. In such cases, it would be great if Praefect could be manually
instructed to verify the storage again as soon as possible rather than waiting
for the next scheduled verification interval to pass. This commit adds the
'praefect verify' subcommand that allows for doing that. It takes in either a
repository id, a virtual storage or a (virtual storage, storage) tuple and
marks all replicas matching the selector as being unverified. Praefect's
metadata verifier will then prioritize verifying these repositories over other
repositories pending verification. This allows administrators to speed up
the verification process and thus recovery.
Changelog: added
|
|
With the introduction of metadata verification, Praefect needs a tool
to manually mark a repository as needing verification immediately rather
than after the specified verification interval has passed. That tool will
require a new RPC that it can call achieve its goal. This commit adds the
proto definitions for MarkUnverified RPC which can be called to either
mark a single repository by ID, a whole virtual storage, or a whole storage
as needing verification.
Changelog: added
|
|
proto: Add LimitError as a structured error
See merge request gitlab-org/gitaly!4476
|
|
This commit adds support of using a FIPS-validated SSL library with
compiled Go executables when `FIPS_MODE=1 make` is run. A Go compiler
that supports BoringSSL either directly (e.g. the `dev.boringcrypto`
branch) or with a dynamically linked OpenSSL
(e.g. https://github.com/golang-fips/go) is required.
This is similar to the changes to support FIPS in GitLab Runner and in
GitLab Pages:
https://gitlab.com/gitlab-org/gitlab-pages/-/merge_requests/716
Changelog: added
|
|
Makefile: Make GITALY_EXECUTABLES deferred again
See merge request gitlab-org/gitaly!4477
|
|
Add new examples for concurrency and rate limiters
See merge request gitlab-org/gitaly!4472
|
|
Recently, in b5c9c7efc (Makefile: Rename find_commands to
GITALY_EXECUTABLES, 2022-03-25), we've changed the variable that holds
the names of all Gitaly executable to be an immediate variable. While
this is a good idea in general, it causes trouble in CI. In CI the
compiled executables are put in cache, but the source files are not. So
when files are pulled from cache, and any make target is built, it will
expand GITALY_EXECUTABLES. Now source files are not pulled from the
cache, so the `cmd` directory is missing. And therefore we revert it
back to be a deferred variable.
|
|
When Gitaly enforces a limit, either due rate limiting or concurrency
limiting, it needs to be able to return an error to its clients to
provide context into why it failed so that clients can then inform its
callers of why the call failed.
Changelog: added
|
|
Expose last verification time in 'praefect metadata'
Closes #4092
See merge request gitlab-org/gitaly!4466
|
|
Read-only cache receives invalidations on record updates via triggers
in Postgres. Currently the notifications are sent for any modification
to the records. The verification related columns are not relevant to
the operation of the cache so this commit ignores the changes to the
columns in the triggers.
Changelog: changed
|
|
Administrator's may want to know when Praefect has last verified a
replica. This commit exposes that information via the 'praefect metadata'
command.
Changelog: changed
|
|
GetRepositoryMetadata fetches a repository's metadata from the
database. This commit expands the query to also fetch the newly added
verified_at column so we can expose it in the 'praefect metadata'
command to the admins.
|
|
Administrators may want to know when a replica has been last verified
by Praefect. GetRepositoryMetadata RPC is called by the 'metadata'
sub-command to retrieve infromation about a repository and its
replicas from Praefect's database. This commit adds the proto
definitions for exposing the last verification time of replicas to
the metadata sub-command.
Changelog: changed
|
|
Initial implementation of a metadata verifier
See merge request gitlab-org/gitaly!4459
|
|
This commit wires the metadata verifier in Praefect's main so it can
actually be configured for use. It's default disabled still as it still
is missing some functionality that should be in place before generally
enabling it, for example tooling like metrics, integration in to the
'praefect metadata' tool and a background routine to release stale leases.
Changelog: added
|
