| Age | Commit message (Collapse) | Author |
|---|
|
- Add a new RPC TestStream to gitaly/proto, which uses the StreamRPC protocol
- Put all the pieces together so that Gitaly can accept StreamRPC calls
and handle the TestStream RPC over TCP, TLS, Unix sockets
- The first iteration should include metrics, logging and authentication
Changelog: added
|
|
|
|
Port UserApplyPatch to Go
Closes #3076
See merge request gitlab-org/gitaly!3066
|
|
This commit adds a Go port of UserApplyPatch.
UserApplyPatch is an RPC that parses a stream of patches in unix
mailbox format, applies them sequentially on top of the base commit
and sets the target branch to point to the resulting commit.
The original Ruby implementation basically just streamed the patches
to `git am`. The Go port uses the same approach although returns more
descriptive errors when the command fails.
The RPC does not support setting a base branch. The Go implementation
follows the same default branch discovery for determining the base
commit as the Ruby implementation. Notably, neither support creating
the first branch in the repository.
This was initially implemented using libgit2. During the implementation,
a bug was discovered in the patching routine. As the fix wasn't merged
upstream in a long time, we've implemented this using a worktree for now
to unblock the efforts to remove the Ruby sidecar.
|
|
Remove UpdateRemoteMirror Go port's feature flag
See merge request gitlab-org/gitaly!3693
|
|
UpdateRemoteMirror's Go port has been running successfully in production
and was default enabled in 14.1. This commit removes the feature flag for
so we can proceed removing the Ruby implementation in a later release.
Changelog: removed
|
|
`git am` is going to be used for applying the patches in the Go port
of UserApplyPatch. This commit adds the command description to our
list of git commands.
|
|
newSquashWorktreePath generates a worktree name to use for a squash
operation. As we'll soon need to generate worktree names for git am
operations, this commit generalizes the function so we can reuse it
for generating the worktree paths.
|
|
UserApplyPatch' test coverage is very lacking at the moment. The
original tests are using gitlab/gitlab-test and applying various
patches to them. Some of the target branches have changed since then
with the patches not applying cleanly anymore.
This commit extends the suite with a bunch of new tests to nail down
the behavior in various cases. The test suite generates the state
needed on the fly to avoid testing against the moving target of
gitlab/gitlab-test.
|
|
[ci skip]
|
|
Remove orphan repository replicas
Closes #3480
See merge request gitlab-org/gitaly!3674
|
|
Bump gitlab-shell dependency
See merge request gitlab-org/gitaly!3695
|
|
When repository removal operation fails for secondaries for
max attempts to process the job the repositories remain on
the disk with no use. It happens because the entry is removed
from repositories table when removal operation runs for the primary
storage. That is why we should extend our reconciler to handle
removal of those orphan repositories stored on the secondary gitaly
nodes. Because of the delete_replica_unique_index it is possible to
schedule only a single removal operation for one reconciliation run.
But eventually all repositories will be removed. In order to return
a single storage into proper state faster we use ordering for the
scheduling, it also simplifies testing.
Closes: https://gitlab.com/gitlab-org/gitaly/-/issues/3480
Changelog: fixed
|
|
git: Fix ordering of `--end-of-options` and mark commands as supporting the separator
See merge request gitlab-org/gitaly!3676
|
|
Changelog: changed
|
|
blob: Remove LFSPointersPipeline feature flag
Closes #3681
See merge request gitlab-org/gitaly!3692
|
|
Add pagination to GetTreeEntries RPC
See merge request gitlab-org/gitaly!3611
|
|
Adjust trivial cases of commands which support `--end-of-options`
without requiring any changes in our codebase. All other commands where
we cannot enable this separator are documented, stating why we must keep
the flag.
|
|
The git-config(1) command is currently labelled as not supporting the
`--end-of-options` separator, even though Git supports this just fine.
There is currently only a single callsite which keeps us from marking
the command as supporting `--end-of-options`, which is the
implementation of `DeleteConfig()`. What we do there is that we abuse a
`git.ValueFlag` to pass `--unset-all $value`, even though this is in
fact a flag plus a positional argument. This causes Git to misinterpret
the command, where it thinks that `--end-of-options` is a value pattern
to use for `--unset-all`.
Fix the callsite to pass the value as a proper positional argument and
mark the command as supporting the separator.
|
|
The git-clone(1) command is currently labelled as not supporting the
`--end-of-options` separator, even though Git supports this just fine.
What it doesn't support though is intermixed use of this separator with
the `--` separator, but this is exactly how we use it in several of the
callsites because we pass arguments as post-separator arguments.
Adjust the callsites to not pass the revision as a post-separator
argument, but instead as a normal argument. By marking this command as
supporting `--end-of-options`, this has exactly the same semantics given
that both separators can be used interchangeably in this context, but it
now automatically injects the separator.
|
|
The git-ls-tree(1) command is currently labelled as not supporting the
`--end-of-options` separator, even though Git supports this just fine.
What it doesn't support though is intermixed use of this separator with
the `--` separator, but this is exactly how we use it in one of the
callsites.
Adjust the callsites to not pass the revision as a post-separator
argument, but instead as a normal argument. By marking this command as
supporting `--end-of-options`, this has exactly the same semantics given
that both separators can be used interchangeably in this context, but it
now automatically injects the separator.
|
|
While git-for-each-ref(1) is currently marked as not supporting the
`--end-of-options` separator, it is perfectly capable of supporting it.
One callsite that keeps us from using the separator though is a custom
command implementation for git-for-each-ref(1), which has been
introduced via e54e89905 (Add PostArgFlags, 2019-10-22). This
implementation overrides the validation of arguments such that we can
use flags after positional arguments, which is of course incompatible
with `--end-of-options`. It's unclear why this is done though: the only
callsite of this custom implementation uses it to move the `--contains`
flag after the other reference pattern argument, but git-for-each-ref(1)
is perfectly happy to have that flag as a leading flag.
Remove the custom implementation of the `git.Cmd` interface. With this
code gone, we can now mark the command as supporting `--end-of-options`.
|
|
Git commands support two kinds of separators to disambiguate command
line options: `--end-of-options` tells it that from here on, option
parsing has concluded and that all remaining options are to be
considered normal positional command line parameters. And `--` typically
separates revisions from paths for positional arguments, like for
example in git-log(1). A command using both would thus e.g. like the
following:
$ git log --follow --end-of-options HEAD -- Makefile
As it turns out, our Git DSL puts both types of separators in the same
place though, ending up with the following equivalent instead:
$ git log --follow HEAD --end-of-options -- Makefile
This command does work the same in this case, but it is not how it's
intended to work given that we may now potentially interpret "HEAD" as
an option. While we do validate that no positional arguments start with
leading dashes, this is rather about safety in depth.
Fix this bug and move `--end-of-options` immediately after all flags and
before the positional arguments.
Note that this change requires us to label git-rev-list(1) as a command
which doesn't support `--end-of-options`: we use `--all` and `--not`,
whose order interacts with other positional arguments. But if the
separator is specified, then the command will not even accept pseudo
revisions as parameters anymore. We're still safe against injection
though given that we have custom logic in place to verify the command's
positional arguments.
|
|
The LFSPointersPipeline feature flag, which enables finding LFS Pointers
via the new gitpipe package, has been default enabled in 5c43dfe6f
(featureflag: Default-enable LFS pointers pipeline, 2021-07-08). Given
that this commit is part of v14.1.0, we can now remove the feature flag
altogether.
Changelog: changed
|
|
Enable ResolveConflictsWithHooks feature flag by default
See merge request gitlab-org/gitaly!3639
|
|
If a page limit larger than 0 is set, this RPC will only retrieve that
number of entries from the list.
If a page token is set, this RPC will return entries after this OID.
This reduces the amount of recursive calls needed to flatten a path.
Relates to https://gitlab.com/gitlab-org/gitaly/-/issues/2704
Changelog: added
|
|
This fix was shipped in GitLab 14.1 behind a feature flag and should be
enabled by default in GitLab 14.2 since it fixes the issue described in
https://gitlab.com/gitlab-org/gitlab/-/issues/18917.
Changelog: changed
|
|
documentation: Changelog creation instructions
See merge request gitlab-org/gitaly!3679
|
|
|
|
HEAD: default to main for new repositories
Closes gitlab#331906
See merge request gitlab-org/gitaly!3537
|
|
git2go: Implement support for object quarantine directories
See merge request gitlab-org/gitaly!3680
|
|
[ci skip]
|
|
coordinator: Only schedule replication for differing error states
See merge request gitlab-org/gitaly!3660
|
|
operations: Squash via patches if commits are not directly related
See merge request gitlab-org/gitaly!3685
|
|
Add sort capability for GetTreeEntries
See merge request gitlab-org/gitaly!3671
|
|
In commit 640b4de09 (operations: Implement squashing without worktrees,
2021-07-19), we've introduced a new squashing implementation which
creates commits via git-commit-tree(1) based on the observation that a
squashed merge of two commits which have a direct parent-child relation
is exactly the same as a new commit with the parent's parent and the
child's tree. As it turns out, the assumption that squashes are always
performed on a line of history where child and parent are on the same
first-parent-chain isn't correct though, and thus causes test failures
in QA.
Fix the issue by falling back to patch-based squashing in the case where
the start commit is not an ancestor of the end commit.
Changelog: fixed
|
|
gitpipe: Smallish tweaks
See merge request gitlab-org/gitaly!3681
|
|
Bump grpc to 1.38.0
Closes #3627
See merge request gitlab-org/gitaly!3659
|
|
Now that we can make Git2Go use object quarantine directories, this
commit converts ResolveConflicts as a first proof of concept to set up
object quarantine directories. Like this, we'll stage both objects
fetched from the target directory as well as the resolved commit in a
temporary quarantine directory. Only if we successfully compute the
commit and execute the pre-receive, update and reference-transaction
hook will the objects then be migrated into the target repository.
This change helps both correctness (rejected objects don't end up in the
repository) and performance (authentication checks can now use the
quarantine directory to speed up enumeration of objects).
Changelog: performance
|
|
When executing Git commands, Git will look for a bunch of environment
variables which influence how the repository will be opened. Next to
environment variables like GIT_CEILING_DIRECTORIES and
GIT_DISCOVER_ACROSS_FILESYSTEM, which both influence the lookup itself,
it'll also consider variables which modify locations of the Git
directory via GIT_DIR or of object directories via GIT_OBJECT_DIRECTORY
and GIT_ALTERNATE_OBJECT_DIRECTORIES. In Gitaly, we use the latter in
object quarantine directories such that we can locate objects which are
not part of main object directory.
While Git uses these environment variables by default, libgit2 and thus
Git2Go do not. As a result, our gitaly-git2go binary will not correctly
honor quarantine directories but only use the "normal" repository
structures. While this hasn't yet been a problem given that the binary
never runs in any context where we do have an object quarantine, we'll
eventually want to use manual quarantine directories in this context
such that we can stage, check and migrate new objects with Git2Go, too.
To support this, we need to instruct libgit2 to honor above environment
variables: if they're unset, then it'll behave the same as before. If
set, then libgit2 will not do default discovery of the repository but
instead use values as set in the environment.
Given that there shouldn't be any RPCs using gitaly-git2go which make
use of object quarantine directories, this change shouldn't have any
user-visible effect yet. Users of this new capability will land at a
later point in time when we start to make use of manual quarantine
directories in Git2Go-based RPCs.
|
|
In order to make it easier to change the way how we open Git2Go
repositories across all gitaly-git2go subcommands, this commit unifies
the logic into a single `git2goutil.OpenRepository()` function.
No functional change is expected from this commit given that it still
does the same thing as all codesites did before.
|
|
While we already know to inject info about alternate object directories
into all Git commands, we don't do the same for the gitaly-git2go
command. As a result, if we receive a request which has those alternate
directories set up via the repository protobuf, then we wouldn't honor
them at all.
Prepare for a fix by injecting necessary environment variables into the
spawned gitaly-git2go executable. No functional change is expected from
this given that we still use `git2go.OpenRepository()`, which doesn't
honor environment variables at all.
|
|
In order to support use of alternative object directories for a
repository, we'll need to inject it into subcommands such that we can
extract relevant information.
Add a `repository.GitRepo` parameter to all subcommands to enable for
this. No functional change is expected from this commit given that we do
not yet use the repository.
|
|
Both `run()` and `runWithGob()` are implemented as standalone functions,
even though they would neatly fit as a receiver function of the
`Executor` struct. Let's thus refactor them to be receiver functions,
which will also make it easier to inject additional dependencies without
having to change signatures of these functions.
|
|
We have two ways of serializing parameters for git2go subcommands, once
via JSON and once via gob. They're both located in separate files, even
though they're thematically related.
Refactor the code and move them into a single "serialization" code unit.
|
|
We're about to extend dependencies required to run gitaly-git2go
subcommands, which will make calling them tedious. This commit thus
refactors the submodule subcommand to be implemented on top of the
`Executor` structure, which encapsulates all required dependencies and
thus avoids the additional complexity.
|
|
We're about to extend dependencies required to run gitaly-git2go
subcommands, which will make calling them tedious. This commit thus
refactors the set-config subcommand to be implemented on top of the
`Executor` structure, which encapsulates all required dependencies and
thus avoids the additional complexity.
|
|
We're about to extend dependencies required to run gitaly-git2go
subcommands, which will make calling them tedious. This commit thus
refactors the revert subcommand to be implemented on top of the
`Executor` structure, which encapsulates all required dependencies and
thus avoids the additional complexity.
|
|
We're about to extend dependencies required to run gitaly-git2go
subcommands, which will make calling them tedious. This commit thus
refactors the resolve-conflicts subcommand to be implemented on top of
the `Executor` structure, which encapsulates all required dependencies
and thus avoids the additional complexity.
|
|
We're about to extend dependencies required to run gitaly-git2go
subcommands, which will make calling them tedious. This commit thus
refactors the rebase subcommand to be implemented on top of the
`Executor` structure, which encapsulates all required dependencies and
thus avoids the additional complexity.
|
