| Age | Commit message (Collapse) | Author |
|---|
|
Changelog: fixed
|
|
|
|
The new "v14" version of the Gitaly module is named to match
the next GitLab release. The module versioning is needed in
order to pull gitaly as a dependency in other projects. The
change updates all imports to include v14 version. The go.mod
file was modified as well after go mod tidy execution. And
the changes in dependency licenses are reflected in the NOTICE
file.
Part of: https://gitlab.com/gitlab-org/gitaly/-/issues/3177
|
|
Signed-off-by: Takuya Noguchi <takninnovationresearch@gmail.com>
|
|
We have been bitten by Goroutine leaks in the GRPC proxying code in some
cases, but until now we didn't have any meaningful way to detect this
kind of leak.
Fix the test gap by using go.uber.org/goleak to detect any leaked
Goroutines after the test. This nicely demonstrates that we currently do
have a Goroutine leak in the proxying code in `forwardClienToServers()`.
|
|
When Gitaly needs to cast a vote for a reference transaction, it
currently relies on Praefect to provide address information and tokens
needed to dial to the Praefect for voting. This can be problematic as
Praefect's listening socket may not be reachable by the Gitaly. Configuring
TLS provides additional challenges, as Gitaly would need to have the certificates
to identify the Praefect.
Praefect already establishes a connection to Gitaly, so we can avoid these problems
by piggybacking Gitaly's votes through that same connection. This piggybacking could
be implemented as persistent bidi stream that Praefect calls Gitaly with. However, this
leaves a lot of the complexities managed by gRPC up to us, such as request routing,
concurrency handling, cancellations and request packing.
In order to leverage gRPC for the above, we can instead multiplex the network
connection established by Praefect in order to run multiple gRPC/HTTP2 sessions in it.
This allows for dialing from Gitaly to a gRPC server running on Praefect's end of the
established connection. This allows us to rely on simply using gRPC in the usual fashion
to communicate with the transaction service running in Praefect.
The implementation has components in the client and the server.
The client implements a function, a ServerFactory, that returns a server that should
serve on the client's end of the connection. It implements a ClientHanshake which multiplexes
the connection and starts the backchannel server on it. The gRPC ClientConn itself looks like
normal ClientConn to the rest of the code.
The server implements most of the functionality in gRPC's ServerHandshake. When a connection
is established, the server peeks the stream to see if the client has indicated it supports
multiplexing. Client indicates this by sending magic bytes to the server after establishing
the connection. If the client is not multiplexing aware, the server handles the connection
as it does usually. If the client is multiplexing aware, the server establishes a multiplexing
session, dials the client's backchannel server and starts serving the connection as usual.
The ServerHandshake injects an identifier which can be accessed through the RPC handler's
context. The ID can be used to retrieve the peer's backchannel connection when it is needed,
namely when the votes need to be cast. Connections are uniquely identified by an incrementing
counter, so no ID can ever refer to the wrong peer.
|
|
Recently, we've had increasing reports about Gitaly not correctly
coming up, where it never starts to accept incoming RPCs. It's unclear
yet what the root cause is, and no reliable reproudcer was found for
this issue yet. But it's clear that something's going on there.
One of the Goroutine dumps provided in this context shows that tableflip
was still running after dozens of minutes, which seems to indicate that
an upgrade went wrong somehow. Digging through changes made in the
tableflip library made me notice commit cae714b (ensure that inherited
files are nonblocking, 2021-01-22), which talks about indefinite hangs
caused by Go clearing `O_NONBLOCK` on exec. While this really is a
shot in the dark and quite unlikely to be the root cause, upgrading our
version of tableflip to get hold of that commit is easy enough to do.
Upgrade the tableflip module to v1.2.2. Most important changes between
the 4baec98 (Add ListenConfig as Option to Upgrader, 2020-05-07) and
v1.2.2 are a bump of the minimum required Go version to v1.14 and the
`O_NONBLOCK` change.
|
|
This commit executes `go mod tidy` to clean up unused modules.
|
|
When parsing an LFS pointer, the git-lfs library now uses stronger
validation of the OID in
https://github.com/git-lfs/git-lfs/pull/4421. We update the package to
take advantage of that.
|
|
|
|
`grpc.start_time` and `grpc.request.deadline` are inserted and formatted
by go-grpc-middleware. We pull in
https://github.com/grpc-ecosystem/go-grpc-middleware/pull/398 and
configure the timestamps with the Gitaly format.
Closes https://gitlab.com/gitlab-org/gitaly/-/issues/3451
|
|
|
|
Related to https://gitlab.com/gitlab-com/gl-infra/infrastructure/-/issues/11648
|
|
Gitaly-git2go apply subcommand
See merge request gitlab-org/gitaly!2676
|
|
With enabled distributed_reads feature each read operation leads to
a database query execution to get state of the storages for particular
repository. More read calls leads to more database access operations,
so the pressure to it increases in linear (or even worse).
To mitigate this problem it was decided to introduce an in-memory cache
added before accessing the database. Invalidation happens on receiving
notification events from the database. The events are send by the
triggers attached to the repositories (delete) and storage_repositories
(insert, delete, update) tables.
To monitor the cache a new counter was added: gitaly_praefect_uptodate_storages_cache_access_total.
It tracks amount of cache hits, misses and populates and evicts per virtual
repository. And to track an error rate of the notifications processing the
gitaly_praefect_uptodate_storages_errors_total was added with type set to one of:
retrieve, notification_decode.
Closes: https://gitlab.com/gitlab-org/gitaly/-/issues/3053
|
|
Upgrades git2go to v30.0.18 which contains the bindings for ApplyToTree.
|
|
LabKit has reached a 1.0.0 milestone :tada:
See https://gitlab.com/gitlab-org/labkit/-/releases/v1.0.0
|
|
To support configuring client certificates in TLS connections to the
GitLab internal API, the entire configuration is needed.
|
|
|
|
|
|
|
|
GitLab-Shell and Gitaly share code to call out to the internal API from
GitLab-Rails. That means it's hard to easily differentiate the requests
from one another. This change sets the UserAgent to `gitaly/` postfixed
with the current version.
|
|
This commit adds a new binary, `gitaly-lfs-smudge`, that will be
used to include Git LFS blobs inside a project archive file.
The smudge filter will eventually be called by specifying the `-c
filter.lfs.smudge` option when the `include_lfs_blob` flag is enabled.
The filter works as follows:
1. Read the LFS pointer from STDIN.
2. Decode the OID, if one is available.
3. Make an internal API call to Workhorse/Rails for this OID.
4. Write the contents of the file to STDOUT.
If any errors are encountered, log an error and return a non-zero exit
code.
|
|
Use github.com/pelletier/go-toml as its a more active project.
Close https://gitlab.com/gitlab-org/gitaly/-/issues/3182
|
|
Previously the `correlation_id` was not passed along the `X-Request-Id`
header to the GitLab API requests
(e.g. `/api/v4/internal/{pre_receive,post_receive}`). Now we extract the
`CORRELATION_ID` environment variable, store it in the context for the
gitaly-hooks binary, and pass the context along to the API requests.
This requires an update to gitlab-shell to support this:
https://gitlab.com/gitlab-org/gitlab-shell/-/merge_requests/413
Fixes https://gitlab.com/gitlab-org/gitaly/-/issues/2725
|
|
|
|
Upgrades the package version to fix a potential vulnerability.
For details, see: https://nvd.nist.gov/vuln/detail/CVE-2020-14040
|
|
Fix Git hooks when GitLab relative URL path and UNIX socket in use
See merge request gitlab-org/gitaly!2485
|
|
With commit a6091637 (Merge branch 'pks-revert-git2go' into 'master',
2020-07-31), our new Git2Go dependency was reverted again due to several
issues:
- Some distributions didn't have a recent enough version of
CMake available. As a result, building libgit2 failed. This was
fixed by bumping the build images' CMake version.
- Build tags didn't work as expected and thus we ended up not using
them at all. This was fixed by passing build flags via the
Makefile. As this causes e.g. `go test ./...` to fail with a
linking error due to build tags not being specified and the
PKG_CONFIG_PATH not being set up as required. To work around this
issue and not break developer workflows, gitaly-git2go will now
only be built if build tags required for git2go are set.
- Invalidation of libgit2.a didn't work as required. E.g. if the
Makefile changes, it wasn't rebuilt and it was thus impossible to
provide fixes for any broken setups if libgit2.a was built
already. This was fixed by adding a build dependency on the
Makefile.
All in all, this should fix all currently known problems with building
libgit2.
|
|
Previously if GitLab were configured to use a relative URL
(e.g. `/gitlab`) and the Gitaly `gitlab.url` configuration used the
http+unix:// scheme, the hooks would not be able to contact the API
server. We add an explicit `relative_url_root` parameter to make it
possible for all connections to go through Workhorse.
This commit depends on changes in
https://gitlab.com/gitlab-org/gitlab-shell/-/merge_requests/406.
NOTE: This only fixes the Git hooks that are implemented in Go. The Ruby
gitlab-shell hooks don't appear to have ever work with GitLab
installations using a relative URL when a UNIX domain socket is
used. Omnibus installations bypass Workhorse and talk directly to the
Web server.
Relates to https://gitlab.com/gitlab-org/gitlab-shell/-/issues/476
|
|
This reverts commit 4f3253e9758affac7b7a8e7f74e97117299c4002, reversing
changes made to 40b90823b0d55561059d27249e02db426b428786.
|
|
This adds a build target for libgit2 as a prerequisite dependency for
Git2Go. The build is configured as a static archive with minimal
dependencies such that the only required shared library is the system's
libc.
Instead of building libgit2 manually, we could've also gone the way of
fetching the Git2Go repository and using its scripts. This has multiple
problems though. First, the build scripts aren't flexible and will
default to auto-detection of system-provided libraries. As a result, the
generated archive will usually have more dependencies than only libc.
Second, we'd have to use a "replace" statement for our go.mod file.
While that's not a problem by itself, it will cause several targets like
e.g. linting, verification and others to depend on the presence of this
Git2Go repository.
The approach of building libgit2 manually is thus preferable, as it
gives us both more flexibility and less dependencies on libgit2 in our
build process.
|
|
Using gitlab-shell's internal api client library, make calls to the
gitlab api and port the ruby pre-receive fully to go.
|
|
9738045a3 added a dependency to gitlab-shell@v1.9.8. This version of
gitlab-shell in turn depends on gitaly@v1.68.0 which depends on
cloudflare/tableflip@v0.0.0-20190329062924-8392f1641731. This commit
adds the missing checksum for that specific revision of tableflip.
|
|
Add gitlab api access
See merge request gitlab-org/gitaly!2156
|
|
|
|
|
|
|
|
This adds the subcommand `praefect sql-migrate-status` to show
whether a migration has been applied.
|
|
Bumps Labkit version in order to be able to start
Gitaly / Praefect with the Go Profiler.
Once Gitaly is compiled with continuous profiling,
the profiler configuration can be set via GITLAB_CONTINUOUS_PROFILING
environment variable:
```shell
GITLAB_CONTINUOUS_PROFILING="stackdriver?service=gitaly&service_version=1.0.1&project_id=test-123 ./binary"
```
|
|
Since the gitaly-remote command was added in 22eb30d2 (Add gitaly-remote
binary to add or update remote URL, 2019-03-27), it didn't catch much
interest and is not really used anywhere. It does require us to do quite
some build hacks, though, because of its dependency on git2go and thus
indirectly on libgit2, and in fact building gitaly-remote does not
currently work. Furthermore, it breaks our gosec-sast job that performs
scanning for security vulnerabilities.
So given that we'd require additional hacks to keep gitaly-remote while
nobody really uses it, let's instead remove it and the git2go/libgit2
dependencies altogether.
|
|
This change adds a correlation ID to persisted replication jobs.
Additionally, it ensures that all RPC's related to a replication job
provide the correlation ID in the context metadata.
|
|
The Gitaly project was on 2.25: https://github.com/go-yaml/yaml/commits/v2.2.5
this change bumps it to 2.28:
https://github.com/go-yaml/yaml/commits/v2.2.8
|
|
|
|
|
|
|
|
This reverts commit 3b68223057bed9d6e6111e7882daa3d0b679c316, reversing
changes made to 1b130a414536204204b6f998e3bbb4b01d79c6b5.
|
|
|
|
|
|
|
