| Age | Commit message (Collapse) | Author |
|---|
|
grpc/backchannel: Simplify logging with yamux
See merge request https://gitlab.com/gitlab-org/gitaly/-/merge_requests/6330
Merged-by: Quang-Minh Nguyen <qmnguyen@gitlab.com>
Approved-by: Sami Hiltunen <shiltunen@gitlab.com>
Approved-by: Quang-Minh Nguyen <qmnguyen@gitlab.com>
Co-authored-by: Patrick Steinhardt <psteinhardt@gitlab.com>
|
|
|
|
We're currently passing in an `io.WriteCloser()` to Yamux that is then
being used as logger. This has the consequence that we're quite tied to
the actual implementation of the logger because `WriterLevel()` is not
something that's typically available in other logging libraries. But
more importantly, it means that all log messages are now written at the
error level, which isn't quite what we want.
The current setup is actually a limitation of the Yamux package itself:
we can either pass in a log writer, or a `log.Logger`. We don't really
want either though. This limitation has been fixed upstream via 13a7bc0
(Fix hashicorp/yamux#72, abstract logger into interface, 2019-04-15),
which converts starts to accept an interface instead of a `log.Logger`.
The library didn't release a new version since then though, and given
that this is change has been implemented four years ago, it seems quite
unlikely that a new release will be created soonish. That being said
though, upgrading to that unstable version should be fine as the only
change between our current v0.1.1 and that commit is the change to use
that logger interface.
Let's upgrade Yamux to the version including this change and refactor
our code to pass the `logrus.FieldLogger` directly instead of passing
an `io.Writer`.
|
|
Remove the now-unused gitaly-git2o command as well as its supporting
infrastructure.
|
|
|
|
|
|
|
|
cgroup: Add support for cgroups v2
See merge request https://gitlab.com/gitlab-org/gitaly/-/merge_requests/5547
Merged-by: Quang-Minh Nguyen <qmnguyen@gitlab.com>
Approved-by: karthik nayak <knayak@gitlab.com>
Approved-by: Quang-Minh Nguyen <qmnguyen@gitlab.com>
Reviewed-by: Steve Xuereb <sxuereb@gitlab.com>
Reviewed-by: Quang-Minh Nguyen <qmnguyen@gitlab.com>
Reviewed-by: karthik nayak <knayak@gitlab.com>
Co-authored-by: ZheNing Hu <adlternative@gmail.com>
|
|
Due to the lack of cgroup V2 support in Gitaly, some operating systems
that use cgroup V2 may not be able to use cgroups in Gitaly properly.
Therefore, this patch adds support for cgroup V2 in Gitaly, which maintains
a similar interface to cgroup V1 but removes the
"gitaly_cgroup_memory_reclaim_attempts_total" metric and modifies the label
of the "gitaly_cgroup_procs_total" metric compared to V1.
Signed-off-by: ZheNing Hu <adlternative@gmail.com>
|
|
|
|
|
|
|
|
|
|
Badger has bumped its major version from v3 series to v4. This bump is
mostly uninteresting, but there are two interesting changes in there:
- They have upgraded the minimum required Go version to 1.19. This
has kept us from upgrading Badger given that we still supported Go
1.18. This has changed with a17fb7823 (go.mod: Use Go 1.19 as
minimum required version, 2023-05-23) though, so we're unblocked
with the upgrade.
- They have changed their version schema from CalVer to SemVer,
meaning that future releases should be more indicative of real
major changes.
Other than that there are no real changes that would be interesting to
us.
|
|
datastore: Upgrade the pgx library to v5
See merge request https://gitlab.com/gitlab-org/gitaly/-/merge_requests/5819
Merged-by: James Fargher <proglottis@gmail.com>
Approved-by: James Fargher <proglottis@gmail.com>
Reviewed-by: James Fargher <proglottis@gmail.com>
Co-authored-by: Patrick Steinhardt <psteinhardt@gitlab.com>
|
|
The pgtype dependency has received a major upgrade to v5. This release
brings a dramatically refactored API that makes more use of generics in
many places. Unfortunately, we can't really use them as they do not work
with the generic `database/sql` interfaces, which we're using.
Our `StringArray` helper type depends on `pgtype.TextArray`, which has
been removed in favor of generics. Adapt the type accordingly.
|
|
go: Update module github.com/beevik/ntp to v0.3.3
See merge request https://gitlab.com/gitlab-org/gitaly/-/merge_requests/5824
Merged-by: Will Chandler <wchandler@gitlab.com>
Approved-by: James Fargher <proglottis@gmail.com>
Approved-by: Will Chandler <wchandler@gitlab.com>
Co-authored-by: GitLab Renovate Bot <gitlab-bot@gitlab.com>
|
|
|
|
Update the cgroups dependency to major version 3. This change includes a
bunch of renames where upstream moved the cgroups implementations into
different packages. Adapt our code accordingly.
|
|
Upgrade the pgx library to v5. The most notable change that we need to
adapt to is that the old "prefer_simple_protocol=true" connection option
has been reworked to "default_query_exec_mode".
Other than that, the major release has also introduced support for
generics. Unfortunately, we cannot really use them given that we use the
`database/sql` interface instead of the native pgx one. Ideally, we'd
thus migrate to use the native interface, but we can't easily do that
because the `rubenv/sql-migrate` library depends on the `database/sql`
interface directly.
Note that this does not yet migrate the pgtypes dependency. This will be
done in the next commit.
|
|
|
|
go: Fix incompatibility of gopsutil dependency with macOS
See merge request https://gitlab.com/gitlab-org/gitaly/-/merge_requests/5570
Merged-by: Quang-Minh Nguyen <qmnguyen@gitlab.com>
Approved-by: Will Chandler <wchandler@gitlab.com>
Approved-by: Quang-Minh Nguyen <qmnguyen@gitlab.com>
Approved-by: Lukas 'Eipi' Eipert <leipert@gitlab.com>
Reviewed-by: Patrick Steinhardt <psteinhardt@gitlab.com>
Co-authored-by: Patrick Steinhardt <psteinhardt@gitlab.com>
|
|
We have received a report that users of macOS are seeing a lot of
compiler warnings complaining about TARGET_OS_MAC being undefined:
# github.com/shirou/gopsutil/v3/cpu
.../pkg/mod/github.com/shirou/gopsutil/v3@v3.21.2/cpu/cpu_darwin_cgo.go:13:5: warning: 'TARGET_OS_MAC' is not defined, evaluates to 0 [-Wundef-prefix=TARGET_OS_]
This warnings is caused by f47b8851318d ([clang] Enable errors for
undefined TARGET_OS_ macros in Darwin driver, 2020-07-06) in the Clang
compiler, which causes it to emit warnings by default whenever the
compiler sees an undefined `TARGET_OS_` macro in the Darwin driver. And
this change is completely sensible, as the macro being undefined would
mean that the code that the platform-dependent code is not enabled as it
ought to be.
So the actual bug is in the gopsutil package, which forgot to include
the `<TargetConditionals.h>` header. This was fixed in gopsutil v3.21.3,
but we are still pulling in v3.21.2.
Fix this bug by upgrading to the most recent bug release of the v3.21
release train, which is v3.21.12.
Changelog: fixed
|
|
|
|
|
|
|
|
|
|
|
|
Gitaly's Main is currently a bit of a mess with CLI parsing logic
mixed with other code. There's no real support for adding properly
subcommands or generating automatically help texts. This commit
introduces `urfave/cli` as the framework of choice in Gitaly to
structure the CLI interfaces. This gives all of the features we need
out of the box so we don't have to implement them ourselves.
Backwards compatibility is maintained with successful invocations
of the Gitaly resulting in launching the `check` subcommand, `-version`
printing or launching the main server daemon. Erroring invocations are
not guaranteed to be backwards compatible, as the help text is different
and in some cases the exit code as well. This should generally be fine
as it's unlikely anyone would be relying on failing invocations in scripts
other than tests.
The commands' existing code is only minimally changed to support the CLI
framework and still access globals like stdout and stderr or may exit
abruptly via os.Exit. This was done to keep the change minimal but we
should later restructure the commands not to use any of the globals from
the environment and structure the control flow to better fit the framework.
|
|
In d1c74493a (Add golangci-lint custom linter infrastructure,
2023-02-15), we have added a Gitaly-specific custom linter that is
plugged into golangci-lint. For the time being, we agreed to just make
it use the same `go.mod` as the main module, mostly because we wanted
`make test` to also execute its tests.
This is causing problems already, though. The Renovate bot has decided
to immediately try and upgrade the new dependency on golang.org/x/tools
to the most recent version. But this causes CI failures now because
golangci-lint requires all plugins to always use the same version of
that package as it uses itself.
Fix this by moving the Gitaly-specific linting infrastructure into a
subpackage of `tools/golangci-lint` and make it reuse the `go.mod` file.
This ensures that they always use compatible versions and that the main
production-level `go.mod` file is not tied to a specific dependency
version.
|
|
Include go-license-detector performance patch in fork
See merge request https://gitlab.com/gitlab-org/gitaly/-/merge_requests/5348
Merged-by: Toon Claes <toon@gitlab.com>
Approved-by: Toon Claes <toon@gitlab.com>
Co-authored-by: Sami Hiltunen <shiltunen@gitlab.com>
|
|
This commit adds the infrastructure to support custom linters.
Unfortunately, golangci-lint doesn't support dynamic linter. It means
that we need to compile each linter with `-buildmode=plugin` flag. The
compiled artifacts are then configured in `.golangci.yml` file. New
linter implementation follows the guide in this lint:
https://golangci-lint.run/contributing/new-linters/
Implementing this linter requires the following steps:
* Add analyzers implemting the checks to tools/gitaly-linters package
* Add the new checks to `lint.go` file
* Re-compile via `make gitaly-linters` or `make lint`
|
|
The library is using a regex to split files from possible first
lines of licenses. It uses normalized first line's from known
licenses to build the regex. The problem is that App-s2p.txt
license's first line normalizes into an empty line, which causes
the regex to match every line beginning and ending. Each split is
treated as a possible license candidate and analyzed. This has a
drastic effect on performance as any large files that look like
possible licenses will have the logic run for each line. This
commit solves the problem by dropping the problematic license
from the first line regex. Proper fix would be better but this
allows us to test the effect on performance faster.
This changes our fork to point to a version that contains the fix
for the performance issue. The fork is otherwise the same.
|
|
Currently there is not a standardized approach to quarantining flaky
tests. This change implements `testhelper.SkipQuarantinedTest` which can
be invoked to skip specified tests during execution.
|
|
gRPC supports a built-in DNS resolver. This resolver works quite well
in most scenarios. It has some drawbacks:
- After the DNS is resolved for the first time, the resolver does not
refresh the list of addresses until the client connection triggers the
resolver actively. Client connection does so when it detects some of its
subchannels are unavailable permanently. It means as soon as the client
connection is stable, the client is not aware of new hosts added to
the cluster via DNS service discovery. This behavior leads to unexpected
stickiness and workload skew, especially after a failover.
- The support for SRV record is in a weird state. This type of record
is only supported when grpclb load balancing strategy is enabled. This
strategy is deprecated, unfortunately. Its behavior is also not as we
expected. In short-term, we would like to use round-robin strategy. In
longer term, we may have a custom strategy for Raft-based cluster. Thus,
SRV service discovery is crucial in the future.
- The resolver detects service config via TXT record if any. While this
option is convenient for a generic grpc setting, it does not make sense
for Gitaly. So, we should get rid of it.
This commit implements a custom DNS resolver. This resolver has some
major features:
- Resolve DNS service discovery via A records
- Periodically refresh the DNS (5 minutes by default)
- Update DNS state only if it detects real changes
- Support logging
Service discovery via SRV records is not supported in this version to
keep the backward compatibility with Ruby clients.
|
|
|
|
Gitaly is currently lacking in transaction control. Each write coming
in launches their own Git commands which operate on the repository
concurrently. This makes transaction management difficult. It's
difficult to optimize the writes as they are being done from multiple
locations without synchronization. The concurrent writers may step on
each others toes and surface lock conflicts to the users. Recovering from
crashes is also difficult as Gitaly is not logging the modifications it is
about to perform and thus loses the transaction state on crashes. There's
also no clear notion of ordering which further complicates replication
related matters. It's not easy to say which writes a repository is missing
and which not.
We've recently designed a new replication architecture for Gitaly. The
new architecture relies on a replicated write-ahead log. The write-ahead
log defines a clear order of writes and aids in crash recovery. A single
writer will be operating on a repository which makes further optimizations
such as write batching easier.
This commit implements the first steps towards the new architecture by
implementing the TransactionManager. The TransactionManager will be
responsible for transaction management of a single repository. It will
be the single goroutine that writes into a repository and is invoked by
all other locations in the code that wish to write. It will also be
responsible for synchronizing reads by ensuring they see the changes they
are supposed to see.
TransactionManager implementation introduced here does not contain the
full implementation but aims to provide a basis for future iteration. For
now, it implements basic write processing with a write-ahead log. It
processes writes one-by-one by verifying references, logging the changes
and finally applying the changes to the repository. It also supports
recovering from the write-ahead log should the log processing be interrupted.
The reference verification behavior can be tuned on a per transaction level
to match behavior Git's `--atomic` or `--force` push flags.
The TransactionManager stores the state related to the write-ahead log in
BadgerDB, which is a key-value store that will be local to each Gitaly
storage. The values are marshaled protocol buffer messages.
This iteration is mostly concerned with the reference updating logic.
Pack files are not handled yet as aren't the internal references they need.
Symbolic references, namely for updating the default branch, are not handled
yet either. The writes are processed one by one and are acknowledged after
applying them to the repository. Given that, there's not separate logic needed
for read synchronization yet either.
The goal here is to set the initial interface and log processing, and to lock
down the reference updating logic with tests so we can later on safely start
iterating on the internals of the TransactionManager and start adding support
for the missing functionality.
|
|
|
|
|
|
|
|
|
|
Upstream has released libgit2 v1.5.0 on July 14th already, but so far we
had been blocked from upgrading as Git2go didn't yet support it. This
has now changed with the release of Git2go v34, so let's upgrade both so
that we run with the latest version.
Notable upstream changes include:
- Initial work on support for SHA256 as object hash.
- Fixes for CVE 2022-24765 and CVE 2022-29187, which could lead to
arbitrary code execution in repositories not owned by the current
user. libgit2 was not directly impacted, but the fixes now align
behaviour with Git and thus refuses to open repositories owned by
a different user.
- Several fixes for MIDX files, which might be beneficial with our
plans to start using them.
- The rename-detection limit for merges was bumped from 200 to 1000
files to match Git's behaviour.
There are many other changes, but due to our limited use of libgit2 most
of them aren't relevant to us.
|
|
|
|
|
|
Instead of using the external 'client' package from 'gitlab-shell', use
the internally created one 'internal/gitlab/client'. This drops the
dependency on 'gitlab-shell'.
Run 'go mod tidy' to remove the dependency on 'gitlab-shell' from our
'go.{mod, sum}' files.
|
|
With this change we upgrade go-toml to v2. This includes changing the
import paths to include v2.
|
|
|
|
go: Update gitlab-shell dependency to v14.8.0
See merge request gitlab-org/gitaly!4690
|
|
|
|
This change adds an alternative implementation of linguist.Stats using
go-enry as a pure Go solution. The code is behind a default disabled
feature flag 'go_language_stats'.
Issue: https://gitlab.com/gitlab-org/gitaly/-/issues/2571
Changelog: performance
|
