| Age | Commit message (Collapse) | Author |
|---|
|
This can be used by Geo to authenticate against a primary node
Changelog: changed
|
|
|
|
log: Fix formatting via gofumpt
See merge request gitlab-org/gitaly!4368
|
|
Fix formatting in the log package, which is currently not clean based on
what gofumpt says.
|
|
git: Remove support for the Ruby hooks directory
Closes #4006
See merge request gitlab-org/gitaly!4356
|
|
gitaly-git2go: Stop using stderr
See merge request gitlab-org/gitaly!4343
|
|
Previously stderr was used to send plaintext errors back to the caller.
Since we are looking to use stderr for logging, we need to stop sending
errors this way. Here we use the universal gob response error field
instead.
Client side must continue reading errors from both gob and stderr for a
release so that we can safely upgrade.
|
|
Now that gob is required for all git2go commands we can hoist the
initialisation of the gob encoding so that later all errors can be
encoded over gob instead of using stderr.
|
|
Now that one release has passed we can safely remove the previously
deprecated field.
|
|
into 'master'
git: Convert internal fetches to use sidechannel
Closes #4042
See merge request gitlab-org/gitaly!4358
|
|
proto: Deprecate fine-grained repository maintenance RPCs
See merge request gitlab-org/gitaly!4362
|
|
[ci skip]
|
|
repository: Ignore missing refs in CreateBundleFromRefList
Closes #4000
See merge request gitlab-org/gitaly!4364
|
|
gitaly: Fix missing housekeeping manager dependency
See merge request gitlab-org/gitaly!4365
|
|
git: Simplify setup of pack-generation configuration
See merge request gitlab-org/gitaly!4361
|
|
'master'
operations: Enable use of detailed errors for UserMergeBranch
Closes #3757
See merge request gitlab-org/gitaly!4357
|
|
While we set up the housekeeping manager when starting Gitaly, we don't
inject it into the service dependencies. As a result, Gitaly will not
have it set up during normal runtime operations and will thus trigger a
panic whenever trying to access it. This went unnoticed because our test
setup for the Gitaly server is different from the setup we use when
running tests.
Fix this bug by injecting the housekeeping manager as expected
Changelog: fixed
|
|
housekeeping: Create central stateful architecture which handles repository optimizations
See merge request gitlab-org/gitaly!4348
|
|
This RPC is used to create backup bundles. The list of refs passed in is
collected by a call to ListRefs. If a ref is deleted between these two
calls then the `git bundle create` call made by CreateBundleFromRefList
would fail with an error.
We are not interested in missing refs for backups because there will be
no need to restore a now deleted ref. However, ignoring missing refs
will mean that the list of refs saved for incremental backups will be
out-of-date compared to the bundle. There should be no problem creating
incremental backups because we only need the object ID that the ref used
to be pointed at. So the ref names would only be used to detect ref
deletions from the previous backup on restore
(https://gitlab.com/gitlab-org/gitaly/-/issues/3857), any extra refs in
the ref list can simply be ignored.
Changelog: fixed
|
|
log: Disable gRPC HealthCheck message logging
Closes #3428
See merge request gitlab-org/gitaly!4363
|
|
Create support request issue template
See merge request gitlab-org/gitaly!3836
|
|
gRPC HealthCheck messages are quite noisy, dominate the log volume in
Gitaly, and usually are not that useful. We now disable them by
default and add documentation on how to enable them.
Closes https://gitlab.com/gitlab-org/gitaly/-/issues/3428
Changelog: changed
|
|
|
|
Historically we had and still have lots of fine-grained RPCs which
perform repository maintenance in Gitaly. This meant that control of how
repositories are packed does not reside in Gitaly though, and this is
starting to become a bigger problem because we cannot easily iterate on
how repositories should be maintained. We have thus recently extended
the black-box-style maintenance RPC OptimizeRepository with the intent
of making it the single source of truth for how repositories are packed.
Deprecate the old fine-grained repository maintenance RPCs. They will be
removed as soon as all callers have been migrated to OptimizeRepository,
which will then allow us to improve the general repository maintenance
strategy.
Changelog: deprecated
|
|
Document behaviour of the OptimizeRepository RPC call.
|
|
Fix typo in the documentation of the PruneUnreachableObjects RPC
definition.
|
|
We use a flag to tag all Git commands which may potentially generate a
packfile, where this information is then used to inject specific config
into these commands which changes the way the packfiles are generated.
This is needlessly roundabout: we can just append this configuration to
the per-command options.
Convert the code to do this to make it easier to understand.
|
|
datastore: Clean completed & dead replication jobs
Closes #3665
See merge request gitlab-org/gitaly!4353
|
|
It is currently hard to observe which optimizations were executed for
repositories and how long they typically take. This commit introduces
two new metrics to better track this information, which also allows us
to easily write tests for OptimizeRepository.
|
|
git: Backport patches to speed up git-fetch(1) in repos with many refs
Closes #4050
See merge request gitlab-org/gitaly!4355
|
|
With commit b5722938b (Merge branch 'jv-ssh-sidechannel-2' into
'master', 2022-01-18) we have introduced a new RPC which allows us to
serve SSH-based fetches with a sidechannel. Instead of using gRPC as a
means for communicating between client and server we instead use a
simple protocol based on pktlines, which allows us to avoid the overhead
introduced by gRPC message serialization and deserialization.
For now, use of the sidechannel is only enabled for external clients
which come in via gitlab-shell. This commit adds the necessary infra to
also use the sidechannel for internal fetches, which most importantly
includes the RPCs ReplicateRepository, FetchSourceBranch and a subset of
RPCs in the OperationService which know to fetch from a remote
repository.
The new code is guarded by a feature flag.
Changelog: performance
|
|
The Ruby hooks directory does not exist anymore, and instead we're using
symlinks to the gitaly-hooks binary. Update our documentation to reflect
that change.
|
|
When we introduced the hook symlinks which directly point to the
gitaly-hooks binary we also had to adapt gitaly-hooks such that it can
continue to resolve the hook name it ought to execute. Previously, this
name was derived via the first argument, which was passed to us via a
wrapper script. This was not possible anymore though given that there
is no wrapper script anymore. Instead, gitaly-hooks gained support for
resolving the hook name via its zeroth argument, which is the name of
the symlink we're executing and thus contains the hook name, as well.
We had to retain backwards-compatibility with the old way of resolving
the hook name though to not break hook executions during zero-downtime
upgrades.
Now that the old code is gone though we do not need the compatibility
code anymore, so this commit removes it.
|
|
The GITALY_BIN_DIR environment variable was used to derive the location
of the gitaly-hooks executable when we were still using the Ruby wrapper
scripts. We now exclusively use symlinks for this though, and thus the
variable is not used anymore.
Stop injecting the GITALY_BIN_DIR environment variable.
|
|
Historically, our Git hooks were located in the Ruby directory as a set
of wrapper scripts which derive the actual location of the gitaly-hooks
binary via the `GITALY_BIN_DIR` environment variable. This has been
replaced with a new mode though, where we instead create a temporary
directory which contains symlinks for all of the hooks which directly
point to gitaly-hooks. This is done both to remove the indirection via
another shell script, and more importantly to get rid of our dependency
on the Ruby directory.
This new mode has been rolled out as default-enabled in 14.8, so we can
now remove support for the old Ruby hooks directory in this release.
Changelog: removed
|
|
Tests for the gitaly-hooks binary are still using the Ruby wrapper
scripts. These are deprecated though, where the new way to execute the
binary is instead via a temporary directory created at runtime which
only contains symlinks to the binary.
Refactor the tests to execute the binary directly now that we're about
to remove the Ruby hook scripts. This also requires us to move to the
new way to derive which hook is about to be executed: the hook name is
not derived via the first argument anymore, but instead via the zeroth
argument which is set to the name of the symlink.
|
|
The test-boot script verifies that Gitaly comes up correctly and is used
as a very basic smoke test in our CI. As part of the bringup it writes
the gitlab-shell secret into the `ruby/git-hooks` directory such that
Gitaly is able to find the secret, or otherwise it would fail to boot.
This directory is going away, so we need to find a better location. And
we do in fact already have one: we can just write it into a subdirectory
of the temporary directory which already contains all the other runtime
data required for a successful boot. Convert the script to do so.
|
|
With UserMergeBranch we have introduced a first test balloon for the use
of detailed gRPC errors which inject information about why specifically
an RPC call has failed via embedded structured errors. While it has
taken quite a while to roll this out, Rails has now been converted to
use it. The result is that we have much more robust error handling,
where Rails can now use the structured data to handle the error itself
better without having to distinguish errors by matching messages.
This change has been rolled out to production on January 28th, and we
didn't observe any issues with it. Remove the feature flag guarding the
code.
|
|
We have upstreamed two patches for git-fetch(1) which speed up mirror
fetches in repositories with hundreds of millions of references. The
first patch makes better use of the commit-graph when computing common
objects, and the second patch will cause us to skip computing the output
width when fetching with the `--quiet` flag.
The patches were tested with our notorious problem repo `www-gitlab-com`
with speedups of about 25% when doing mirror-fetches.
Backport them into our own Git version now that they have been merged
via 2b331293fb (Merge branch 'ps/fetch-optim-with-commit-graph' into
next, 2022-02-14).
Changelog: performance
|
|
Now that we have the housekeeping manager globally injected as required,
move the logic to clean up repositories and optimize them to this
manager.
|
|
We're eventually going to move the housekeeping package to keep state
such that we can improve the heuristics used for optimizing repos. This
will allow us to e.g. take into account the last time a repository was
optimized to not optimize repositories too often, or avoid packing the
same repository twice concurrently.
Implment a new housekeeping manager structure that will be the home of
this functionality and inject it as required. This manager isn't yet
doing anything.
|
|
We want to allow for OptimizeRepository to be called without invoking an
RPC in order to allow us to call it in more contexts. Now that all its
dependent parts are part of the housekeeping package this commit moves
the logic of OptimizeRepository in there, too.
|
|
We want to allow for OptimizeRepository to be called without invoking an
RPC in order to allow us to call it in more contexts. As a preparatory
step, we thus have to move all functionality which is invoked by it into
a package that is independent of gRPC services.
Move the logic handling repacking of objects into the housekeeping
package as part of this move.
|
|
We want to allow for OptimizeRepository to be called without invoking an
RPC in order to allow us to call it in more contexts. As a preparatory
step, we thus have to move all functionality which is invoked by it into
a package that is independent of gRPC services.
Move the writing of the commit-graph into the housekeeping package as
part of this move.
|
|
We want to allow for OptimizeRepository to be called without invoking an
RPC in order to allow us to call it in more contexts. As a preparatory
step, we thus have to move all functionality which is invoked by it into
a package that is independent of gRPC services.
Move the cleanup of worktrees into the housekeeping package as part of
this move.
|
|
We're about to move handling of all repository implementations into the
housekeeping package. Prepare for this by making OptimizeRepository not
depend on the RepositoryService server anymore.
|
|
coordinator: Fix error comparison causing excessive replication jobs
Closes #4045
See merge request gitlab-org/gitaly!4349
|
|
Propagate NotFound error returned by GetRepoPath
See merge request gitlab-org/gitaly!4338
|
|
Previous attempts [1] & [2] were made to avoid completed and dead jobs
being left over in the replication queue. But only in [3] there was made
sure no new jobs were generated.
So now no more of these stale are created, we can run a migration that
wipes all stale records from the database. This change in fact repeats
the `DELETE` query from the migration added in [2].
1. 8f8ae3025 (Replication job acknowledge removes 'completed' and 'dead'
events., 2020-08-05)
2. 2d3fc8063 (Replication job acknowledge removes 'completed' and 'dead'
events., 2020-09-03)
3. 8c4da135f (replication: Remove 'dead' stale jobs., 2021-10-20)
Changelog: performance
Issue: https://gitlab.com/gitlab-org/gitaly/-/issues/3665
|
|
We are currently wrapping some of the errors returned by GetRepoPath,
which makes them lose the gRPC NotFound code.
This leads to a gRPC error code of Unknown, which counts towards our
error SLO, paging the on-call.
This patch removes the wrapping, ensuring the error code is propagated
to the client, and accounted for appropriately in logs and metrics.
Changelog: fixed
|
