| Age | Commit message (Collapse) | Author |
|---|
|
On many cloud providers, cadvisor runs and keeps track of cgroups. To
have Gitaly expose its own metrics is redundant at this point. Allow
users to configure whether or not they would like Gitaly to collect its
own metrics about cgroups.
Changelog: added
|
|
Change the metric names to be more accurate. The memory.failcnt tracks
how often the kernel has tried to reclaim memory. While this maps
closely to the number of killed processes, they are not exactly the same
thing.
The cgroups cpu usage metrics are a running total. Add "total" at the
end of the metric to indicate this.
Changelog: changed
|
|
datastore: Fix migration that prunes maintenance-style replication jobs
Closes #4275
See merge request gitlab-org/gitaly!4608
|
|
operations: Use structured errors in UserDeleteBranch
See merge request gitlab-org/gitaly!4605
|
|
operations: Remove deprecated error fields
See merge request gitlab-org/gitaly!4610
|
|
Makefile: Update our tooling
See merge request gitlab-org/gitaly!4611
|
|
Makefile: Run bundle install in frozen mode
See merge request gitlab-org/gitaly!4613
|
|
catfile: Add tests which exercise the request queue
See merge request gitlab-org/gitaly!4598
|
|
While some of our tool versions require the `v` prefix of their tag to
be present, others don't. This is inconsistent and a tag confusing.
Unify this to always require the prefix. This also gives us more
flexibility given that it's now easily possible to e.g. override the
version with a commit ID.
|
|
Update protoc to v21.1. Note that this is quite a jump from our current
v3.17.3. This isn't all that scary, but a result of upstream changing
their versioning schema to drop the leading `3`. Our generated Go files
remain unchanged.
|
|
Update protoc-gen-go-grpc to v1.2.0. The most notable change is that we
now have headers in our generated gRPC services that indicate the
versions used to generate them. Other than that there are no changes in
our generated files.
|
|
Update protoc-gen-go to v1.28.0. No noticable changes come with this
release, and most notably there are no changes in our generated Go
files.
|
|
Update go-licenses to v1.2.1. One notable change is that the module now
prints out warnings for files it doesn't recognize.
|
|
Update gotestsum to v1.8.1. There aren't any noteworthy changes in our
context.
|
|
Update goimports to v0.1.11. The most important change in our context is
that it now uses a cache, which "should speed up goimports -w *.go
dramatically in cases where imports need to be added" (0133cac3
(cmd/goimports: reuse cached state, 2019-05-06)).
|
|
Update protolint to v0.38.1. The only interesting change is support for
Protocol buffer 3's optional fields, but we don't currently use them
anyway.
|
|
Update to the most recent version of golangci-lint, which is v1.46.1. No
interesting new linters have been added since v1.44.2.
|
|
Recently security concerns popped up about gems that might get
compromised and get replaced with malicious content.
To mitigate this, the Bundler team recommends[1] using bundler in
--frozen or --deployment mode in CI and during deploys.
We address this by exporting BUNDLE_FROZEN=true in the environment of
the Makefile. This applies to all bundle commands ran by make, and will
affect local development and CI.
For omnibus deployments the fix is applied in
omnibus-gitlab@4b18021f9 (Merge branch 'bundle-frozen' into 'master',
2022-05-24). The CNG changes are being made in
https://gitlab.com/gitlab-org/build/CNG/-/merge_requests/1053.
[1]: https://github.com/rubygems/rubygems.org/security/advisories/GHSA-hccv-rwq6-vh79
Issue: https://gitlab.com/gitlab-org/gitaly/-/issues/4225
Changelog: security
|
|
UserDeleteBranch is returning successfully in some error cases. This is
a pattern we want to get rid of, so let's convert the RPC to instead use
the structured errors via the new `UserDeleteBranchError` message type.
This fixes some cases where we know to create replication jobs because
the `UserDeleteBranch` RPC returns successfully, but didn't cast a
single vote.
Changelog: fixed
|
|
gitaly-lfs-smudge: Implement support for running as a long-running filter process
Closes #4153
See merge request gitlab-org/gitaly!4595
|
|
Wire up support for running gitaly-lfs-smudge as a long-running filter
process to speed up conversion of LFS pointers into their converted
objects. This should result in a signficant performance boost when
creating archives for large repositories with many LFS pointers and
avoid the "process-spawn storms" we saw in the past whenever such an
archive was requested.
Changelog: performance
|
|
Originally, Git only ever supported smudging the contents of a single
blob by piping that blob to a filter process directly. This meant that
for every blob that needs converting, we have to spawn a full process.
Naturally, this doesn't scale well when repositories grow and when they
have many blobs that do need converting. Git has thus introduced a
long-running filter protocol: instead of spawning one process per blob
that needs converting, Git will only spawn a single server and then
communicate with that server via pktlines.
Implement support for this protocol in gitaly-lfs-smudge so that we can
start using it in our `GetArchive()` RPC.
|
|
Until now, gitaly-lfs-smudge only supports converting a single LFS
pointer into its object. This is inefficient though, and Git has an
alternative protocol that allows us to convert multiple LFS pointers in
a single session via the long-running filter protocol.
Introduce a new driver type so that we can easily discern which type of
filter Git is expecting: either the old one-shot "filter", or the newer
"process" type.
For now, this functionality is neither implemented nor used.
|
|
We're injecting the correlation ID into the Git process that'll create
the archive for us in the `GetArchive()` RPC. This is not required
though because we always inject the correlation ID via the command
package because we use LabKit's environment injector.
Remove the duplicate logic.
|
|
We're currently always injecting configuration required for the LFS
smudge filter, even if the caller of `GetArchive()` didn't ask us to
convert LFS pointers.
Refactor the code to only inject the configuration when required.
|
|
There is only a single caller of `smudgeOneObject()`, and that caller
will immediately bubble up any errors returned by this function. It is
thus needless to return an `io.NopCloser()` in case `smudgeOneObject()`
itself fails.
Return `nil` instead of the `io.NopCloser()` to clarify the code flow.
|
|
Rename the smudging functions to be better prepared for when we add
support for the long-running filter process protocol. While at it, this
commit also renames the hosting file to be called `smudge.go` instead of
`lfs_smudge.go` to avoid some repetition.
|
|
The GitLab HTTP client is currently created closely before performing
the request to retrieve the LFS pointers' smudged contents. This has
been fine until now given that we only ever requested a single LFS
pointer across the whole lifetime of the command. As we are about to
implement the long-running filter protocol though this will change so
that we may request an unbounded number of LFS pointers in a single
session.
Move out creation of the HTTP client so that it can be reused across
calls to `smudgeContents()`. This avoids some overhead like re-reading
contents of the GitLab secret from disk.
Note that this is essentially a change in behaviour: previously, if we
failed to create the HTTP client, then we'd have just silently returned
the blob's clean contents. This easily papers over misconfigurations
though and is thus not all that great. Now we do return errors in case
we failed to create the client, which means that any such misconfigured
instance may now error. This feels like a reasonable change though:
better to complain about errors than to silently swallow then.
|
|
The context is currently created when smudging contents. We're about to
add another code path to the `run()` function though that will also need
a context.
Move creation of the context higher up into `run()` so that it can be
shared across both code paths.
|
|
While the pktline package already provides the `Data()` function to
retrieve a pktline's payload, this function doesn't perform any sanity
checks on retrieved data at all.
Add a new function `Payload()` that verifies that the pktline header's
claimed length matches the actual data's length. While it would be
preferable to just change `Data()`, chances are high that this would
cause unintended side effects.
|
|
The `GitError` field of the `UserSquash` response has been deprecated in
e8413304d (operations: Always use structured errors for UserSquash,
2022-03-21) and is never set anymore. Let's remove it.
|
|
The `PreReceiveError` field of the `UserMergeToRef` response has been
deprecated in 9aa0347a2 (proto: Deprecate PreReceiveError field in
UserMergeToRefResponse, 2022-03-07) and is never set anymore. Let's
remove it.
|
|
The `PreReceiveError` field of the `UserMergeBranch` response has been
deprecated in 151bdfd96 (proto: Deprecate PreReceiveError field in
UserMergeBranchResponse, 2022-03-07) and is never set anymore. Let's
remove it.
|
|
[ci skip]
|
|
Enable feature flag exact_pagination_token_match by default
See merge request gitlab-org/gitaly!4606
|
|
proto: Introduce new UserDeleteBranchError message
See merge request gitlab-org/gitaly!4604
|
|
We 41ca8534e (praefect: Prune stale maintenance-style replication
events, 2022-05-20), we have introduced a migration that prunes old
maintenance-style replication jobs from our replication queue. That
migration was a bit too naive though because it assumed that we had
mostly drained the replication queue of such jobs already and that there
aren't any stale locks which referenced these jobs. Naturally, the real
world shows that such stale locks exist. And because these locks have a
foreign-key constraint on the job without cascading deletes we fail to
prune any such maintenance-style job that is referenced by a lock.
Fix this issue by deleting job locks before deleting the jobs. Also,
let's handle the case where there are acquired job locks that should be
unlocked because all job are removed.
Changelog: fixed
|
|
We're about to convert UserDeleteBranch to use structured errors. As a
first step we introduce a new UserDeleteBranchError message that hosts
all errors that are expected failures and which may thus be handled by
the client. The RPC is not yet converted to use the new error types
because we first have to update clients to handle them.
|
|
Improve test coverage for UserDeleteBranch to exercise failures caused
by concurrent writes to the same references and when the call to Rails'
`/internal/allowed` interface fails for access checks.
|
|
Contributes to https://gitlab.com/gitlab-org/gitaly/-/issues/3817
Changelog: changed
|
|
Modernize test names to match our current coding style more closely.
|
|
Modernize error messages returned by UserDeleteBranch to match our
modern coding style more closely.
|
|
Remove comment stating that we don't verify the branch name and/or user
in `UserCreateBranch()` while we do in `UserDeleteBranch()`: we do it in
both, so this comment is clearly stale.
|
|
The UserDeleteBranch RPC is missing documentation. Add it to document
its behaviour and expected error cases.
|
|
chore: Update Gemfile.lock to use bundler v2.3.15
See merge request gitlab-org/gitaly!4603
|
|
gitaly-lfs-smudge: Update git-lfs module and dependencies
See merge request gitlab-org/gitaly!4600
|
|
This is just to minimize the versions of bundler used for development.
The GDK runs `support/bundle-install` in this directory to obtain the
version of bundler needed.
This relates to https://gitlab.com/gitlab-org/gitlab/-/issues/364373.
|
|
We do not currently have tests which directly exercise the request queue
at a low level. Add some to improve our test coverage and assert that it
continues to work alright for several edge cases.
|
|
[ci skip]
|
|
Regenerated via `make notice`
|
