Age | Commit message (Collapse) | Author |
|
This change implements the license finding in Go using
github.com/go-enry/go-license-detector.
Issue: https://gitlab.com/gitlab-org/gitaly/-/issues/3078
Changelog: performance
|
|
StorageServiceSink allows to use desired storage engine based on the
url value provided for its creation. The set of storage engines now
includes: AWS S3, Google Cloud Storage and Azure Blob Storage.
The initialization of a particular client uses a set of env vars that
should be set prior the StorageServiceSink creation. Each storage provider
has its own set of required and optional parameters.
We use a memory storage provider in tests to verify the implementation
works as expected.
Part of: https://gitlab.com/gitlab-org/gitaly/-/issues/3633
|
|
Changelog: changed
|
|
|
|
The new "v14" version of the Gitaly module is named to match
the next GitLab release. The module versioning is needed in
order to pull gitaly as a dependency in other projects. The
change updates all imports to include v14 version. The go.mod
file was modified as well after go mod tidy execution. And
the changes in dependency licenses are reflected in the NOTICE
file.
Part of: https://gitlab.com/gitlab-org/gitaly/-/issues/3177
|
|
Signed-off-by: Takuya Noguchi <takninnovationresearch@gmail.com>
|
|
Add TestNewNonExistent to help detecting goroutine leaks when
spawning a non-existent command with nil stderr.
When we `New` a command with nil stderr, a default stderr would
be created to help logging possible cmd errors, and in the
meanwhile we use io.Pipe to implement log size and line size
limiting. However, if we encounter an error from the latter
`cmd.Start()`, the PipeWriter will be left unclosed and related
goroutine leaks.
Signed-off-by: blanet <bupt_xingxin@163.com>
|
|
When Gitaly needs to cast a vote for a reference transaction, it
currently relies on Praefect to provide address information and tokens
needed to dial to the Praefect for voting. This can be problematic as
Praefect's listening socket may not be reachable by the Gitaly. Configuring
TLS provides additional challenges, as Gitaly would need to have the certificates
to identify the Praefect.
Praefect already establishes a connection to Gitaly, so we can avoid these problems
by piggybacking Gitaly's votes through that same connection. This piggybacking could
be implemented as persistent bidi stream that Praefect calls Gitaly with. However, this
leaves a lot of the complexities managed by gRPC up to us, such as request routing,
concurrency handling, cancellations and request packing.
In order to leverage gRPC for the above, we can instead multiplex the network
connection established by Praefect in order to run multiple gRPC/HTTP2 sessions in it.
This allows for dialing from Gitaly to a gRPC server running on Praefect's end of the
established connection. This allows us to rely on simply using gRPC in the usual fashion
to communicate with the transaction service running in Praefect.
The implementation has components in the client and the server.
The client implements a function, a ServerFactory, that returns a server that should
serve on the client's end of the connection. It implements a ClientHanshake which multiplexes
the connection and starts the backchannel server on it. The gRPC ClientConn itself looks like
normal ClientConn to the rest of the code.
The server implements most of the functionality in gRPC's ServerHandshake. When a connection
is established, the server peeks the stream to see if the client has indicated it supports
multiplexing. Client indicates this by sending magic bytes to the server after establishing
the connection. If the client is not multiplexing aware, the server handles the connection
as it does usually. If the client is multiplexing aware, the server establishes a multiplexing
session, dials the client's backchannel server and starts serving the connection as usual.
The ServerHandshake injects an identifier which can be accessed through the RPC handler's
context. The ID can be used to retrieve the peer's backchannel connection when it is needed,
namely when the votes need to be cast. Connections are uniquely identified by an incrementing
counter, so no ID can ever refer to the wrong peer.
|
|
When parsing an LFS pointer, the git-lfs library now uses stronger
validation of the OID in
https://github.com/git-lfs/git-lfs/pull/4421. We update the package to
take advantage of that.
|
|
|
|
|
|
Our hooks used to be implemented via gitlab-shell, but all production
code relating to this is gone by now as we have moved to our own
gitaly-hooks executable. There is still testhelpers related to the
gitlab-shell configuration file though, which muddy the water as they
are seemingly required while in fact they're not used at all.
Remove all code related to the gitlab-shell configuration to clean this
up. Note that the gitlab-shell secret is still required and thus not
removed, as it is used to authenticate against the GitLab API.
|
|
Related to https://gitlab.com/gitlab-com/gl-infra/infrastructure/-/issues/11648
|
|
With enabled distributed_reads feature each read operation leads to
a database query execution to get state of the storages for particular
repository. More read calls leads to more database access operations,
so the pressure to it increases in linear (or even worse).
To mitigate this problem it was decided to introduce an in-memory cache
added before accessing the database. Invalidation happens on receiving
notification events from the database. The events are send by the
triggers attached to the repositories (delete) and storage_repositories
(insert, delete, update) tables.
To monitor the cache a new counter was added: gitaly_praefect_uptodate_storages_cache_access_total.
It tracks amount of cache hits, misses and populates and evicts per virtual
repository. And to track an error rate of the notifications processing the
gitaly_praefect_uptodate_storages_errors_total was added with type set to one of:
retrieve, notification_decode.
Closes: https://gitlab.com/gitlab-org/gitaly/-/issues/3053
|
|
|
|
The tool https://github.com/google/go-licenses we use for generating
NOTICE file has some fixes, especially important is one made in
https://github.com/google/go-licenses/pull/19 that add case insensitivity
to regexp to find the license file.
With the new version of the tool we have a fix to our NOTICE file.
The library github.com/git-lfs/go-ntlm/ntlm has a license file named
'License' and it was not handled properly with old versions of
go-licenses tool.
|
|
This commit adds a new binary, `gitaly-lfs-smudge`, that will be
used to include Git LFS blobs inside a project archive file.
The smudge filter will eventually be called by specifying the `-c
filter.lfs.smudge` option when the `include_lfs_blob` flag is enabled.
The filter works as follows:
1. Read the LFS pointer from STDIN.
2. Decode the OID, if one is available.
3. Make an internal API call to Workhorse/Rails for this OID.
4. Write the contents of the file to STDOUT.
If any errors are encountered, log an error and return a non-zero exit
code.
|
|
Use github.com/pelletier/go-toml as its a more active project.
Close https://gitlab.com/gitlab-org/gitaly/-/issues/3182
|
|
With commit a6091637 (Merge branch 'pks-revert-git2go' into 'master',
2020-07-31), our new Git2Go dependency was reverted again due to several
issues:
- Some distributions didn't have a recent enough version of
CMake available. As a result, building libgit2 failed. This was
fixed by bumping the build images' CMake version.
- Build tags didn't work as expected and thus we ended up not using
them at all. This was fixed by passing build flags via the
Makefile. As this causes e.g. `go test ./...` to fail with a
linking error due to build tags not being specified and the
PKG_CONFIG_PATH not being set up as required. To work around this
issue and not break developer workflows, gitaly-git2go will now
only be built if build tags required for git2go are set.
- Invalidation of libgit2.a didn't work as required. E.g. if the
Makefile changes, it wasn't rebuilt and it was thus impossible to
provide fixes for any broken setups if libgit2.a was built
already. This was fixed by adding a build dependency on the
Makefile.
All in all, this should fix all currently known problems with building
libgit2.
|
|
The license file is currently getting executed without any build tags.
As a result, this causes us to miss a few dependencies which are thus
missing in our NOTICE file.
Fix the issue by passing tags to go-licenses and update our NOTICE file
to include missing licenses.
|
|
This reverts commit 4f3253e9758affac7b7a8e7f74e97117299c4002, reversing
changes made to 40b90823b0d55561059d27249e02db426b428786.
|
|
Update our NOTICE file to add the license for our new Git2Go dependency.
|
|
Add gitlab api access
See merge request gitlab-org/gitaly!2156
|
|
|
|
|
|
|
|
This adds the subcommand `praefect sql-migrate-status` to show
whether a migration has been applied.
|
|
Since the gitaly-remote command was added in 22eb30d2 (Add gitaly-remote
binary to add or update remote URL, 2019-03-27), it didn't catch much
interest and is not really used anywhere. It does require us to do quite
some build hacks, though, because of its dependency on git2go and thus
indirectly on libgit2, and in fact building gitaly-remote does not
currently work. Furthermore, it breaks our gosec-sast job that performs
scanning for security vulnerabilities.
So given that we'd require additional hacks to keep gitaly-remote while
nobody really uses it, let's instead remove it and the git2go/libgit2
dependencies altogether.
|
|
|
|
|
|
|
|
|
|
This reverts commit 3b68223057bed9d6e6111e7882daa3d0b679c316, reversing
changes made to 1b130a414536204204b6f998e3bbb4b01d79c6b5.
|
|
|
|
|
|
|
|
|
|
Add filesystem metadata file on startup
See merge request gitlab-org/gitaly!1289
|
|
Write a metadata file on startup with a unique id. Also add
a field to the ServerInfoResponse message that returns this unique id
as well as the path to the file on the server
|
|
|
|
|
|
This change absorbs the previously vendored library grpc-proxy into
the Gitaly project for breaking changes needed for Praefect
requirements. There were slight modifications needed to make the
project work within Gitaly:
- Go import paths were rewritten to refer to the new Gitaly location
- Test service code was moved to testdata folder to prevent CI linter warnings
- CI-lint overrides were added to exclude inherited technical debt (see follow up issue #1663)
- Removal of unused/ineffective code detected by linter.
|
|
Signed-off-by: Mateusz Nowotyński <maxmati4@gmail.com>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|