| Age | Commit message (Collapse) | Author |
|---|
|
Enforce that RPC definitions must have a comment and add a placeholder
for all instances where such a comment is missing.
|
|
The Protoc plugins we use are hidden away deep into the `proto/`
directory, which makes it very hard to discover them when one doesn't
already know about their existence. Let's move them into a new top-level
`tools/` directory.
|
|
There is no real reason why the `protoc-gen-gitaly-lint` package
requires another internal package to provide the actual logic.
Furthermore, we want to move this plugin into a top-level `tools`
directory to make it easier to discover.
Absorb the `linter` package to make it easier to move the code around.
|
|
The `protoc-gen-gitaly` plugin provides two different functionalities:
- It lints our Protobuf definitions to make sure they conform to our
coding guidelines.
- It generates a `protolist.go` file, which contains a list of all
Protobuf files.
This is conflating two otherwise unrelated concerns, and thus makes the
divide between our `proto` and `lint-proto` Makefile targets a bit
blurry. Furthermore, it is creating a cyclic dependency: the linting
logic requires that the Protobuf files must have already been generated,
but we can only generate them when we have plugin compiled.
Prepare for a fix by splitting up functionality of this plugin into two
separate plugins: `protoc-gen-gitaly-lint` is responsible for linting
our definitions, while `protoc-gen-gitaly-protolist` will generate the
list of Protobuf files.
|
|
proto: Add CherryPickError type for UserCherryPick structured errors
See merge request gitlab-org/gitaly!4497
|
|
In UserCherryPick, there are some error conditions where instead of
returning an error, we return OK and embed the error in the response.
Instead of doing this, we want to return structured errors.
This change adds the proto definition for this error.
|
|
commit: Various fixes for `CheckObjectsExist()`
See merge request gitlab-org/gitaly!4510
|
|
The `PackObjectsHook()` RPC is deprecated and its implementation has
been removed in 473e3540f (Deprecate PackObjectsHook, 2021-10-25).
Now that v15.0 is close, let's also remove the Protobuf definitions.
Changelog: removed
|
|
The `CheckObjectsExist()` RPC is a streaming RPC: the caller may send
one or more requests to check sets of revisions. This is done such that
this RPC call also works in the context where millions of revisions need
to be checked in a single call, which could otherwise exceed the gRPC
memory limitations.
The current behaviour is a tad weird though: revisions sent as part of
the initial request are completely ignored. This doesn't feel right as
the caller would now be forced to _always_ send at least two messages.
It's also not documented to be the case, and last but not least all our
other streaming RPCs don't do it this way either.
So let's fix this behaviour and also honor revisions sent of the initial
request.
Changelog: fixed
|
|
This makes the naming of all proto files consistent.
|
|
|
|
Fix typo in documentation of UserCommitFilesRequestHeader
See merge request gitlab-org/gitaly!4422
|
|
With the introduction of metadata verification, Praefect needs a tool
to manually mark a repository as needing verification immediately rather
than after the specified verification interval has passed. That tool will
require a new RPC that it can call achieve its goal. This commit adds the
proto definitions for MarkUnverified RPC which can be called to either
mark a single repository by ID, a whole virtual storage, or a whole storage
as needing verification.
Changelog: added
|
|
proto: Add LimitError as a structured error
See merge request gitlab-org/gitaly!4476
|
|
When Gitaly enforces a limit, either due rate limiting or concurrency
limiting, it needs to be able to return an error to its clients to
provide context into why it failed so that clients can then inform its
callers of why the call failed.
Changelog: added
|
|
Administrators may want to know when a replica has been last verified
by Praefect. GetRepositoryMetadata RPC is called by the 'metadata'
sub-command to retrieve infromation about a repository and its
replicas from Praefect's database. This commit adds the proto
definitions for exposing the last verification time of replicas to
the metadata sub-command.
Changelog: changed
|
|
When pushing commits to a repository, access checks are run. In order to
use the quarantine directory, we need a way to filter out revisions that
a repository already has in the case that a packfile sends over objects
that already exists on the server. In this case, we don't need to check
the access.
Add an RPC that when given a list of revisions, returns the ones that
already exist in the repository, and the ones that do not exist in the
repository.
Changelog: added
|
|
|
|
proto: Introduce new "maintenance" RPC type
Closes #4079
See merge request gitlab-org/gitaly!4399
|
|
In order to make a decision on which repositories to clean up,
Praefect's repocleaner will match repository paths with what it sees in
the database. If it doesn't see a record in the database, that means
Praefect doesn't know about this repository.
However, it could be the case that a repository has just been created
and the record for it doesn't yet exist in the database.
In order for repocleaner to know, it needs the information to be
returned from Gitaly.
Add a field in the RPC response, and return the last modified date on the
refs/ dir so we can use it as a proxy for when the repository was
created.
|
|
We've got all the infrastructure in place now to handle maintenance
operations specially in our infrastructure. Mark all RPCs which are
maintenance-related as maintenance operations to switch them to use the
new infrastucture.
Note that we've still got a feature flag which toggles between old and
new routing behaviour for these RPCs.
Changelog: changed
|
|
Gitaly classifies RPC calls into two different types:
- Accessors are all RPCs which access data without modifying the
on-disk state of the repository.
- Mutators perform changes in repositories to modify the state as
visible to the user.
Our maintenance RPCs don't fit into either category though: they modify
on-disk data, but the end result as visible to the user shouldn't be
changed by this. They should instead be considered as idempotent with
regards to user-visible behaviour.
Right now those RPCs are labelled as mutators. This is not a good fit
though: we have to disable transactional voting because it doesn't make
sense to vote on the on-disk representation of a repository, which is
not deterministic. As a consequence, we only route those requests to the
primary node and replicate the change afterwards, which is not an ideal
strategy.
Introduce a new RPC type for these maintenance-style RPCs. This will
easily allow us to implement behaviour that's different from both
accessors and mutators in a subsequent commit.
Note that this commit doesn't yet change the type of any of our RPC
calls. First, we must adapt our architecture to handle the new type.
|
|
With d88d29185 (operations: Return error from UserSquash when resolving
revisions fails, 2022-03-01), we have converted UserSquash to return
structured errors in some cases where it previously returned no errors
at all. Instead, the RPC used to embed the error in the response struct
via the `GitError` field. This has made us blind for actual issues which
happen in production, and furthermore it trips Praefect's logic which
decides whether we need to create repliction jobs or not.
Rails has been adapted already in c7bcaadf193 (gitaly_client: Handle
detailed errors for UserSquash, 2022-03-02), and the changes have been
rolled out to production. While we required one follow-up fix to the new
error codes we returned in 6c63998ad (operations: Fix wrong error code
when UserSquash conflicts, 2022-03-11), the rollout has otherwise been
successful.
Remove the feature flag altogether and deprecated the `GitError` field,
which isn't used anymore.
Changelog: changed
|
|
Signed-off-by: Kartik <kartikprajapati987@gmail.com>
|
|
Changelog: added
|
|
This commit introduces these changes by creating a new
UserRebaseConfirmableError Protobuf message which contains all potential
structured errors we want to return from the UserSquash RPC.
Changelog: added
|
|
In GitLab rails, there is code to ensure that the default branch on a
new repository is set. This is currently done with an extra WriteRef
call. This is done in a racy way, which causes problems with transaction
voting. To fix this, we can simplify the logic altogether and avoid
adding another roundtrip, add a parameter in CreateRepositoryRequest
to pass a default_branch.
Changelog: changed
|
|
With e542a7d87 (operations: Implement rich errors for UserMergeBranch
access checks, 2021-08-30) we have introduced the ode to return proper
errors when access checks failed via gRPC's rich error model. The
feature flag has been remove with a23b34894 (operations: Enable use of
detailed errors for UserMergeBranch, 2022-02-16) , which means that we
don't ever set the `PreReceiveError` field of the
UserMergeBranchResponse message anymore.
Deprecate the field such that we can eventually remove it.
Changelog: deprecated
|
|
The PreReceiveError field in the UserMergeToRefResponse isn't ever set
because the RPC call doesn't in fact perform any access checks at all.
Digging back through history this doesn't seem to be an oversight in the
Go port, but was the case in the Ruby implementation already. While the
initial implementation of UserMergeToRef in 8fbc337b8 (Allow merging to
a ref without changing the target branch, 2019-01-25) still used to call
hooks, this was explicitly changed with 19fdfa12a (Skip hooks for
UserMergeToRef RPC, 2019-06-14) to not do so anymore. The field thus
hasn't been set anymore for more than two years by now.
Deprecate the PreReceiveError field so that we can eventually remove it.
Changelog: deprecated
|
|
This commit marks the RefTransactionService with the intercepted
option to denote it is handled by Praefect. Such services do not
need operation type annotations on the RPCs as they do not go through
the proxy. This should have no behavioral changes and mostly just
cleans up the code a bit.
|
|
Add skip_content param to wiki find page call
See merge request gitlab-org/gitaly!4373
|
|
|
|
We're about to introduce support for structured errors in the UserSquash
RPC. We cannot roll out this change even with a feature flag though
because some of our testing infrastructure will always test code with
all feature flags enabled. We thus have to land the protobuf changes
independently such that we can adjust downstream callers.
This commit introduces these changes by creating a new UserSquashError
Protobuf message which contains all potential structured errors we want
to return from the UserSquash RPC.
Changelog: added
|
|
Originally when UserSquash was still using worktrees to do its work, we
used a squash ID which enabled us to reuse worktrees. We have since
converted the RPC to do the rebase in-memory though, which allowed us to
get rid of the use of worktrees and thus make the squash ID obsolete. It
was thus deprecated in 112d74720 (operations: Deprecate `SquashId`,
2021-08-20), and callsites in Rails have been converted in fc5b05874f9
(gitaly_client: Drop use of squash ID, 2021-09-21). There aren't any
other callsites left.
Remove the `squash_id` altogether.
Changelog: removed
|
|
Enable CloneRepositoryFromURL to authenticate with custom token
See merge request gitlab-org/gitaly!4239
|
|
Comments for prune field of GarbageCollectRequest indicates that if set
true, dangling objects that are not modified in the last 24 hours would
be dropped. But the time gap is 30 minutes actually.
Commit ab6bc21a1140d0cab98855a7cd7fe81d166a6950 tuned the time gap from
24 hours to 30 minutes, but the proto comments are missed at that time.
Signed-off-by: blanet <moweng.xx@alibaba-inc.com>
|
|
This can be used by Geo to authenticate against a primary node
Changelog: changed
|
|
|
|
Historically we had and still have lots of fine-grained RPCs which
perform repository maintenance in Gitaly. This meant that control of how
repositories are packed does not reside in Gitaly though, and this is
starting to become a bigger problem because we cannot easily iterate on
how repositories should be maintained. We have thus recently extended
the black-box-style maintenance RPC OptimizeRepository with the intent
of making it the single source of truth for how repositories are packed.
Deprecate the old fine-grained repository maintenance RPCs. They will be
removed as soon as all callers have been migrated to OptimizeRepository,
which will then allow us to improve the general repository maintenance
strategy.
Changelog: deprecated
|
|
Document behaviour of the OptimizeRepository RPC call.
|
|
Fix typo in the documentation of the PruneUnreachableObjects RPC
definition.
|
|
When rewriting the repository's history with the BFG Repo-Cleaner, then
we potentially accumulate lots and lots of unreachable objects in the
repository's object database. By default, we'd clean up those objects
after two weeks, which is a rather long time to sit on such a huge
number of objects. To fix this usecase we have thus gained a `prune`
parameter in our GarbageCollect RPC call: if set, then we prune
unreachable objects if they haven't been accessed during the last 30
mintues.
The problem with this though is that GarbageCollect does a lot more than
only pruning objects: it may end up packing objects or objects, writing
commit-graphs, write bitmaps or some other things. All of these are
things we want to control ourselves though, but we instead let git-gc(1)
dictate how the repository is packed.
We're thus about to deprecate all RPCs which directly influence how a
repository is packed in favor of OptimizeRepository: this is our "black
box" RPC that, from the viewpoint of the caller, does something with the
repository to make it great again. And this is by design: callers should
not control the way Gitaly handles repository maintenance.
This highlights the need though for a new RPC call which _only_ prunes
objects which have become unreachable to disentangle it from repository
maintenance tasks. This commit thus introduces PruneUnreachableObjects,
a new RPC which does exactly that: any unreachable loose object that
hasn't been touched in the last 30 minutes is going to be pruned.
Note that to make this work correctly, the caller has to do two RPC
calls: the first RPC call to OptimizeRepository is required to unpack
unreachable loose objects, and 30 minutes later they may prune these
objects with a second call to PruneUnreachableObjects.
This is no different from right now, even though it's hidden away and
(naturally) used incorrectly by Rails: GarbageCollect would need to be
called twice, first to explode unreachable objects into loose objects
and then second with `prune=true` to prune them after half an hour. This
is because Git will only ever consider loose objects for pruning, and
the grace period is determined by inspecting its access time. So the way
Rails does this is broken, and the new RPC call doesn't change that
fact. This is a separate story though and nothing we can fix in Gitaly:
we must retain the grace period to avoid repository corruption.
Changelog: added
|
|
Add HTTP Host to all requests that use URLs
See merge request gitlab-org/security/gitaly!46
|
|
|
|
Back during the days when rebases were implemented via worktrees we
tried to amortize costs of creating the worktrees by reusing them. To
identify a worktree for reuse, callers passed a rebase ID identifying
it, where no two concurrent rebase requests must have used the same ID.
Nowadays though rebases are done in-memory without the use of worktrees,
which is a lot more efficient, especially in big repositories. As a
consequence, the rebase ID doesn't carry any value, and the RPC call
`IsRebaseInProgress()` doesn't exist anymore either. It's thus useless,
but we still require it to be passed.
Stop validating the rebase ID and deprecate the field to phase out
support for it.
Changelog: deprecated
|
|
cmd/praefect: Check of the system clock synchronization
Closes gitlab#342574
See merge request gitlab-org/gitaly!4225
|
|
feat: Detect SSH signed objects
See merge request gitlab-org/gitaly!4255
|
|
It is not possible for RPC clients to tell when RPCs have failed to
update a reference. Add a new error detail type which represents such
failures to prepare for passing down this information.
Changelog: added
|
|
Support for signing Git objects with SSH keys was added in Git v2.34.0.
This change allows Gitaly to detect when an object was signed with an
SSH key and indicates the signature type of that object as `SSH`.
Changelog: added
|
|
Because check of the authentication token depends on the time
we need to make sure it is synced on the praefect machine and
all gitaly machines that belong to the cluster. That is why
a new check point is added to the 'check' sub-command of the
praefect binary. The task should be run on the praefect node
and doesn't require praefect to be up and running. It is possible
to configure the URL of the NTP service and acceptable time
offset via env variables NTP_HOST and DRIFT_THRESHOLD.
The check has fatal severity because the cluster won't work
correctly if auth checks fail continuously for each request.
Closes: https://gitlab.com/gitlab-org/gitlab/-/issues/342574
Changelog: added
|
