| Age | Commit message (Collapse) | Author |
|---|
|
In order to generate our `NOTICE` file, we first export all Go sources
referenced by our project into a directory and then walk that directory
to gather any licenses. The result is that the generated `NOTICE` file
contains the licenses of all our dependencies.
Ideally, the `NOTICE` file wouldn't need to contain any licenses that we
have as part of our own sources. These may for example include licenses
of test files, but also licenses of vendored dependencies. But because
we are already bundling their licenses into our sources in the first
place it shouldn't be necessary to also put them into our `NOTICE` file.
In fact, it seems like we already intended to skip soaking up any such
license files in our own tree: we initially determine the module path of
ourselves, and then return early in `filepath.Walk()` when hitting a
directory path that matches. We only return a `nil` pointer though,
which means we only skip that single directory. That doesn't seem to
make a whole lot of sense though, and it feels like the intent was to
instead skip walking that whole directory.
So let's fix this by returning `filepath.SkipDir` instead so that we
never collect licenses part of our own source tree.
|
|
Add RateLimiting
Closes #4026
See merge request gitlab-org/gitaly!4427
|
|
Introduce a simple rate limiter that limits the number of requests a
minute that an RPC can allow.
If the feature flag is enabled, the middleware will drop any request
that bursts the per second limit of the RPC. Otherwise, it will only emit
metrics so we can first have some data on the traffic profile.
|
|
Upgrade go-licenses to v1.0.0. Most importantly, this release contains a
fix to the way project sources are copied into the target directory so
that we don't end up with files and directories which cannot be modified
[1]. The workaround we have in our codebase which fixes the permissions
can be removed as a consequence.
Note that this causes us to pick up an additional file that's part of
our own sources as licensed file because it's a test verifying that
licenses can be parsed alright, and thus it contains a license body
itself. It's known though that go-licenses behaves weird already though:
it copies the complete sourcecode of projects into the target directory
in case it detects that the license requires it. So adding one more
weird edge case to our NOTICE file doesn't really matter. Together with
the fact that we can remove the workaround we previously had this seems
like an acceptable tradeoff.
[1]: https://github.com/google/go-licenses/issues/11
|
|
cmd/gitaly-git2go: Upgrade to libgit2 v1.3.0
See merge request gitlab-org/gitaly!4273
|
|
Upgrade libgit2 to v1.3.0 and git2go to v33.
Changelog: changed
|
|
Because check of the authentication token depends on the time
we need to make sure it is synced on the praefect machine and
all gitaly machines that belong to the cluster. That is why
a new check point is added to the 'check' sub-command of the
praefect binary. The task should be run on the praefect node
and doesn't require praefect to be up and running. It is possible
to configure the URL of the NTP service and acceptable time
offset via env variables NTP_HOST and DRIFT_THRESHOLD.
The check has fatal severity because the cluster won't work
correctly if auth checks fail continuously for each request.
Closes: https://gitlab.com/gitlab-org/gitlab/-/issues/342574
Changelog: added
|
|
As we are switched to the new listener implementation
this change removes old one as not used anymore and
as well removes dependency on the lib/pq package.
|
|
This change adds jackc/pgx driver as a replacement
for the lib/pq driver. To reduce code changes to
minimum we are using github.com/jackc/pgx/v4/stdlib
at first as it provides support of the standard
database/sql package, but still uses pgx underneath.
The configuration 'binary_parameters' has a different
name in pgx that is why we replace it, but it still does
the same thing.
|
|
libgit2 v1.2.0 has been released on September 1st. Next to a bunch of
new features and performance improvements, this release also contains a
fix for unpacking objects from packfiles which sometimes led to errors.
This is an error we have seen once or twice in the past.
Upgrade our version of libgit2 to v1.2.0, along with the correspongind
update of Git2Go to v32 and Rugged to v1.2.0.
|
|
This change implements the license finding in Go using
github.com/go-enry/go-license-detector.
Issue: https://gitlab.com/gitlab-org/gitaly/-/issues/3078
Changelog: performance
|
|
StorageServiceSink allows to use desired storage engine based on the
url value provided for its creation. The set of storage engines now
includes: AWS S3, Google Cloud Storage and Azure Blob Storage.
The initialization of a particular client uses a set of env vars that
should be set prior the StorageServiceSink creation. Each storage provider
has its own set of required and optional parameters.
We use a memory storage provider in tests to verify the implementation
works as expected.
Part of: https://gitlab.com/gitlab-org/gitaly/-/issues/3633
|
|
Changelog: changed
|
|
|
|
The new "v14" version of the Gitaly module is named to match
the next GitLab release. The module versioning is needed in
order to pull gitaly as a dependency in other projects. The
change updates all imports to include v14 version. The go.mod
file was modified as well after go mod tidy execution. And
the changes in dependency licenses are reflected in the NOTICE
file.
Part of: https://gitlab.com/gitlab-org/gitaly/-/issues/3177
|
|
Signed-off-by: Takuya Noguchi <takninnovationresearch@gmail.com>
|
|
Add TestNewNonExistent to help detecting goroutine leaks when
spawning a non-existent command with nil stderr.
When we `New` a command with nil stderr, a default stderr would
be created to help logging possible cmd errors, and in the
meanwhile we use io.Pipe to implement log size and line size
limiting. However, if we encounter an error from the latter
`cmd.Start()`, the PipeWriter will be left unclosed and related
goroutine leaks.
Signed-off-by: blanet <bupt_xingxin@163.com>
|
|
When Gitaly needs to cast a vote for a reference transaction, it
currently relies on Praefect to provide address information and tokens
needed to dial to the Praefect for voting. This can be problematic as
Praefect's listening socket may not be reachable by the Gitaly. Configuring
TLS provides additional challenges, as Gitaly would need to have the certificates
to identify the Praefect.
Praefect already establishes a connection to Gitaly, so we can avoid these problems
by piggybacking Gitaly's votes through that same connection. This piggybacking could
be implemented as persistent bidi stream that Praefect calls Gitaly with. However, this
leaves a lot of the complexities managed by gRPC up to us, such as request routing,
concurrency handling, cancellations and request packing.
In order to leverage gRPC for the above, we can instead multiplex the network
connection established by Praefect in order to run multiple gRPC/HTTP2 sessions in it.
This allows for dialing from Gitaly to a gRPC server running on Praefect's end of the
established connection. This allows us to rely on simply using gRPC in the usual fashion
to communicate with the transaction service running in Praefect.
The implementation has components in the client and the server.
The client implements a function, a ServerFactory, that returns a server that should
serve on the client's end of the connection. It implements a ClientHanshake which multiplexes
the connection and starts the backchannel server on it. The gRPC ClientConn itself looks like
normal ClientConn to the rest of the code.
The server implements most of the functionality in gRPC's ServerHandshake. When a connection
is established, the server peeks the stream to see if the client has indicated it supports
multiplexing. Client indicates this by sending magic bytes to the server after establishing
the connection. If the client is not multiplexing aware, the server handles the connection
as it does usually. If the client is multiplexing aware, the server establishes a multiplexing
session, dials the client's backchannel server and starts serving the connection as usual.
The ServerHandshake injects an identifier which can be accessed through the RPC handler's
context. The ID can be used to retrieve the peer's backchannel connection when it is needed,
namely when the votes need to be cast. Connections are uniquely identified by an incrementing
counter, so no ID can ever refer to the wrong peer.
|
|
When parsing an LFS pointer, the git-lfs library now uses stronger
validation of the OID in
https://github.com/git-lfs/git-lfs/pull/4421. We update the package to
take advantage of that.
|
|
|
|
|
|
Our hooks used to be implemented via gitlab-shell, but all production
code relating to this is gone by now as we have moved to our own
gitaly-hooks executable. There is still testhelpers related to the
gitlab-shell configuration file though, which muddy the water as they
are seemingly required while in fact they're not used at all.
Remove all code related to the gitlab-shell configuration to clean this
up. Note that the gitlab-shell secret is still required and thus not
removed, as it is used to authenticate against the GitLab API.
|
|
Related to https://gitlab.com/gitlab-com/gl-infra/infrastructure/-/issues/11648
|
|
With enabled distributed_reads feature each read operation leads to
a database query execution to get state of the storages for particular
repository. More read calls leads to more database access operations,
so the pressure to it increases in linear (or even worse).
To mitigate this problem it was decided to introduce an in-memory cache
added before accessing the database. Invalidation happens on receiving
notification events from the database. The events are send by the
triggers attached to the repositories (delete) and storage_repositories
(insert, delete, update) tables.
To monitor the cache a new counter was added: gitaly_praefect_uptodate_storages_cache_access_total.
It tracks amount of cache hits, misses and populates and evicts per virtual
repository. And to track an error rate of the notifications processing the
gitaly_praefect_uptodate_storages_errors_total was added with type set to one of:
retrieve, notification_decode.
Closes: https://gitlab.com/gitlab-org/gitaly/-/issues/3053
|
|
|
|
The tool https://github.com/google/go-licenses we use for generating
NOTICE file has some fixes, especially important is one made in
https://github.com/google/go-licenses/pull/19 that add case insensitivity
to regexp to find the license file.
With the new version of the tool we have a fix to our NOTICE file.
The library github.com/git-lfs/go-ntlm/ntlm has a license file named
'License' and it was not handled properly with old versions of
go-licenses tool.
|
|
This commit adds a new binary, `gitaly-lfs-smudge`, that will be
used to include Git LFS blobs inside a project archive file.
The smudge filter will eventually be called by specifying the `-c
filter.lfs.smudge` option when the `include_lfs_blob` flag is enabled.
The filter works as follows:
1. Read the LFS pointer from STDIN.
2. Decode the OID, if one is available.
3. Make an internal API call to Workhorse/Rails for this OID.
4. Write the contents of the file to STDOUT.
If any errors are encountered, log an error and return a non-zero exit
code.
|
|
Use github.com/pelletier/go-toml as its a more active project.
Close https://gitlab.com/gitlab-org/gitaly/-/issues/3182
|
|
With commit a6091637 (Merge branch 'pks-revert-git2go' into 'master',
2020-07-31), our new Git2Go dependency was reverted again due to several
issues:
- Some distributions didn't have a recent enough version of
CMake available. As a result, building libgit2 failed. This was
fixed by bumping the build images' CMake version.
- Build tags didn't work as expected and thus we ended up not using
them at all. This was fixed by passing build flags via the
Makefile. As this causes e.g. `go test ./...` to fail with a
linking error due to build tags not being specified and the
PKG_CONFIG_PATH not being set up as required. To work around this
issue and not break developer workflows, gitaly-git2go will now
only be built if build tags required for git2go are set.
- Invalidation of libgit2.a didn't work as required. E.g. if the
Makefile changes, it wasn't rebuilt and it was thus impossible to
provide fixes for any broken setups if libgit2.a was built
already. This was fixed by adding a build dependency on the
Makefile.
All in all, this should fix all currently known problems with building
libgit2.
|
|
The license file is currently getting executed without any build tags.
As a result, this causes us to miss a few dependencies which are thus
missing in our NOTICE file.
Fix the issue by passing tags to go-licenses and update our NOTICE file
to include missing licenses.
|
|
This reverts commit 4f3253e9758affac7b7a8e7f74e97117299c4002, reversing
changes made to 40b90823b0d55561059d27249e02db426b428786.
|
|
Update our NOTICE file to add the license for our new Git2Go dependency.
|
|
Add gitlab api access
See merge request gitlab-org/gitaly!2156
|
|
|
|
|
|
|
|
This adds the subcommand `praefect sql-migrate-status` to show
whether a migration has been applied.
|
|
Since the gitaly-remote command was added in 22eb30d2 (Add gitaly-remote
binary to add or update remote URL, 2019-03-27), it didn't catch much
interest and is not really used anywhere. It does require us to do quite
some build hacks, though, because of its dependency on git2go and thus
indirectly on libgit2, and in fact building gitaly-remote does not
currently work. Furthermore, it breaks our gosec-sast job that performs
scanning for security vulnerabilities.
So given that we'd require additional hacks to keep gitaly-remote while
nobody really uses it, let's instead remove it and the git2go/libgit2
dependencies altogether.
|
|
|
|
|
|
|
|
|
|
This reverts commit 3b68223057bed9d6e6111e7882daa3d0b679c316, reversing
changes made to 1b130a414536204204b6f998e3bbb4b01d79c6b5.
|
|
|
|
|
|
|
|
|
|
Add filesystem metadata file on startup
See merge request gitlab-org/gitaly!1289
|
|
Write a metadata file on startup with a unique id. Also add
a field to the ServerInfoResponse message that returns this unique id
as well as the path to the file on the server
|
|
|
