| Age | Commit message (Collapse) | Author |
|---|
|
|
|
|
|
|
|
|
|
The script to build the Gitaly ruby gem compares the VERSION file with
the current tag (via `git describe --tags`), this is a safety check so
that we don't build an publish non-tagged releases.
But this also blocks us from building the gem for local testing. So add
a new option `---skip-verify-tag` to the build script which can skip
this verification.
Since we're adding OptionParser also move the previously used argument
to an option.
|
|
|
|
|
|
|
|
|
|
into 'master'
tools/protolint: Update module github.com/yoheimuta/protolint to v0.42.0
See merge request https://gitlab.com/gitlab-org/gitaly/-/merge_requests/5028
Merged-by: Patrick Steinhardt <psteinhardt@gitlab.com>
Approved-by: Justin Tobler <jtobler@gitlab.com>
Approved-by: Patrick Steinhardt <psteinhardt@gitlab.com>
Co-authored-by: GitLab Renovate Bot <gitlab-bot@gitlab.com>
|
|
These tests know nothing of object hashes and so are safe to run.
|
|
While we were able to get rid of all custom Protobuf definitions used
for tests, this is definitely not possible for our tests that verify the
behaviour of `protoc-gen-gitaly-lint`. Its whole purpose is to lint
invalid Protobuf definitions, and we of course ain't got any in our
normal gRPC services.
We can still make the generation of this test-specific Protobuf code
easier by moving the definitions into a new `proto/testproto` directory
so that they live in the same filesystem hierarchy as the other Protobuf
definitions. This both allows us to unify the call to protoc and will
allow us to easily change the directory where the code is generated in
in a subsequent commit.
|
|
|
|
|
|
The helper functions in `run.rb` had in the past been used by multiple
different scripts. Nowadays there is only a single user though, so let's
inline it.
|
|
The `generate-proto-ruby` script is not only generating the Ruby sources
anymore, but also builds the `gitaly` Gem. It wasn't renamed in the
preceding commit though as Git doesn't detect the rename by default,
which would have made the diff hard to read.
Rename the file now to the more fitting `build-proto-gem`.
|
|
The Ruby sidecar is slowly nearing its end: right now, it only
implements a single RPC that is in the process of being migrated to Go.
This means we'll start to soon remove all the Ruby infrastructure in the
Gitaly project.
Part of the infrastructure is the Ruby code generated from our Protobuf
definitions. And while we won't need that code ourselves anymore, it is
important for clients written in Ruby that want to interface with us. As
such, the Gitaly project still has to release the `gitaly` Gem. In the
past, we built the Gem by simply soaking up the code in `ruby/proto`.
But as it does not make any sense for us to keep the Ruby code around
anymore this strategy will fail once we have removed it.
Merge the `generate-proto-ruby` any `publish-gem` scripts into a single
script that:
1. Generates all required Ruby sources into a temporary directory.
2. Writes the `gitaly.gemspec` file into the same directory.
3. Builds the Gitaly Gem from those temporary files.
Like this, all code is generated ad-hoc and doesn't need to be committed
to the repository anymore.
Change the Makefile's `proto` target to not generate Ruby code anymore.
Instead, there are two new Makefile targets `build-proto-gem` and
`publish-proto-gem` that are responsible for building and publishing the
Gem, respecfively.
The generated Ruby code is thus stale now and will be removed in a
subsequent commit.
|
|
On every version bump the GitLab Release Tools bot updates the version
in two different files: first in our top-level `VERSION` file, and
second in the Ruby Protobuf files at `ruby/proto/version.rb`. Both files
basically contain the same information, but serve different purposes.
We're about to drop our generated Ruby Protobuf code though in favor of
creating the Protobuf Gem ad-hoc. This also means it doesn't make a
whole lot of sense anymore to carry around the version file in our tree.
Consequentially, we've adjusted the Release Tools bot to stop writing
the `version.rb` file completely. Instead, we now generate the file via
the information we already have in our `VERSION` file.
|
|
The `generate-proto-ruby` script assumes a specific output directory
that's always fixed at `ruby/proto`. We're about to drop our in-tree
copy of generated Ruby code in favor of using the `gitaly` Gem, which
will require us to generate the code in a temporary directory.
Refactor the code to be less dependent on the output directory. Like
this we can easily change the location where we generate the code into.
Right now the location is unchanged though, which also demonstrates that
there is no change in the generated code.
|
|
We've got multiple scripts that are required to generate Ruby code from
our Protobuf definitions in the `_support` directory. This has multiple
smells:
- It's out-of-line with all the other tools, which nowadays are
located in the `tools` directory.
- It's hard to discover and find out which parts logically form a
unit.
- We are reusing the Gemfile of the Ruby sidecar to pin the
`grpc-tools` dependency to a specific version.
Move the tooling into its own `tools/protogem` directory that's got its
own Gemfile to fix these points. This also allows us to auto-update
dependencies via the Renovate bot like we do for our other tools.
|
|
|
|
golangci-lint doesn't work with Go 1.17 anymore, since it has
dependencies which set a minimum version of Go 1.18. This breaks `make
lint` locally since the tool errors out during installation.
Update the go.mod to `go 1.18` and run `go mod tidy` in the package.
|
|
Upgrade the protoc compiler to the latest version at the time of writing
this commit, which is v21.7. There are no relevant changes, except for
improved formatting of Go comments according to recent changes in gofmt.
Regenerate the Protobuf code.
|
|
|
|
The upgrade to Viper v1.13 has introduced a change to lower-case all
config entries in arrays via 5247643 (Recurse into arrays when
converting keys to lowercase, 2022-06-23). This in turn breaks our
ability to pass the `allowTypesBefore` argument to the Revive linter as
it performs an exact match for the arguments.
Let's unblock the upgrade to golangci-lint v1.50.0 by excluding Viper
v1.13 for the time being.
|
|
|
|
into 'master'
tools/protoc-gen-go: Update module google.golang.org/protobuf to v1.28.1
See merge request https://gitlab.com/gitlab-org/gitaly/-/merge_requests/4934
Merged-by: Justin Tobler <jtobler@gitlab.com>
Approved-by: James Fargher <proglottis@gmail.com>
Approved-by: Justin Tobler <jtobler@gitlab.com>
Co-authored-by: GitLab Renovate Bot <gitlab-bot@gitlab.com>
|
|
into 'master'
tools/protolint: Update module github.com/yoheimuta/protolint to v0.41.0
See merge request https://gitlab.com/gitlab-org/gitaly/-/merge_requests/4926
Merged-by: Justin Tobler <jtobler@gitlab.com>
Approved-by: Quang-Minh Nguyen <qmnguyen@gitlab.com>
Approved-by: Justin Tobler <jtobler@gitlab.com>
Co-authored-by: GitLab Renovate Bot <gitlab-bot@gitlab.com>
|
|
|
|
|
|
The new tools mechanism is great, this commit just changes the package
name for external tools from `gofumpt` to its own module name, to fix a
small nits from MR#4910.
Signed-off-by: blanet <moweng.xx@alibaba-inc.com>
|
|
Update go-licenses to v1.4.0 by executing the following commands:
```
$ cd tools/go-licenses
$ go get github.com/google/go-licenses@latest
$ go mod tidy
```
There are no noteworthy updates in this
release.
|
|
Update gotestsum to v1.8.2 by executing the following commands:
```
$ cd tools/gotestsum
$ go get gotest.tools/gotestsum@latest
$ go mod tidy
```
There are no noteworthy updates in this release.
|
|
Right now we track versions of our Go tooling directly in our Makefile.
While this is simple, it has several drawbacks:
- We're susceptible to supply-chain attacks in case an adversary
manages to replace the code used to build any of our tools.
- We cannot use proper dependencies in our Makefile, which adds the
need for `*.version` files.
- It is hard to build the tools outside of our Makefile as we don't
have a way to properly pull in the correct version.
- Upgrading our tooling requires us to manually hunt down new
releases for all of our tools.
We can fix these issues by following the approach that is efficially
recommended by the Go project [1]: every tool has its own Go module in
`tools/` with a "tool.go" file that imports the tool of interest. Like
this we can use Go's normal tooling to keep track of versions:
- We record hashes of the tool's sources as well as all of its
dependencies, making supply-chain attacks almost impossible.
- We can now provide proper dependencies in our Makefile: every tool
depends on "tool.go", "go.mod" and "go.sum". If any of them
changes we need to rebuild.
- The tools can be installed in the correct version simply by using
`go install` with the correct `go.mod` file.
- Upgrading tools is as simple as running `go get -u`, so no more
manual hunting for new versions.
While these benefits are great on their own already, we can go even
further with this refactoring: now that each tool has its own `go.mod`
file we can adapt the Renovate bot to pick up these files. This means
that we don't have to remember upgrading at all anymore, but instead the
bot will automatically upgrade them for us.
[1]: https://github.com/golang/go/wiki/Modules#how-can-i-track-tool-dependencies-for-a-module
|
|
Reformat surces with the newly upgraded gofumpt v0.4.0.
|
|
We're about to add the ability to test with SHA256 hashes. We assume
none of the tests work with this object format.
With this change we add the build constraint to not run any test when
the tag 'gitaly_test_sha256' is set.
|
|
Update protoc to v21.1. Note that this is quite a jump from our current
v3.17.3. This isn't all that scary, but a result of upstream changing
their versioning schema to drop the leading `3`. Our generated Go files
remain unchanged.
|
|
Update protoc-gen-go-grpc to v1.2.0. The most notable change is that we
now have headers in our generated gRPC services that indicate the
versions used to generate them. Other than that there are no changes in
our generated files.
|
|
Update protoc-gen-go to v1.28.0. No noticable changes come with this
release, and most notably there are no changes in our generated Go
files.
|
|
into 'master'""
This reverts commit 933109e3e849358daf78c4618d07091b3f36068f.
|
|
This commit changes the major version in the package name from v14 to
v15
Updating go.mod & go.sum with new module name v15
Update Makefile to bump major version to v15
Update the gitaly package name in the Makefile. Also update
gitaly-git2go-v14 -> gitaly-git2go-v15. We need to keep
gitaly-git2go-v14 for a release however, for zero downtime upgrades.
This pulls directly from a sha that is v14.
Update package name from v14->v15 for auth, client, cmd, internal packages
This commit changes the package name from v14 to v15 in go and proto
files in the internal, auth, client, cmd packages.
proto: Update major package number in package name
tools: Change major version number in package name from v14 to v15
gitaly-git2go: Change the package name from v14 to v15
update module updater for v15
Update the documentation for the module updater to reflect v15
|
|
'master'"
This reverts commit 94a955f7bac56cb8f524f43a7773038e6e341585, reversing
changes made to 708408a8ad99f942c9cfd40f43ec11b961d31846.
|
|
Back when we added support for GNU build IDs to our binaries we started
building our Go binaries twice: the first time we do it so that we can
derive a deterministic GNU build ID from the Go build ID, and the second
time to embed that derived GNU build ID into the final binary. This has
two problems:
1. We build the binary twice, and even though Go caches most of the
build process this still significantly slows down incremental
builds of our binaries.
2. We're rebuilding the binary in-place by overwriting the binary
with no GNU build ID with the one that contains the GNU build ID.
While I'm not a 100% sure, this seems to leads to issues from
time to time where the resulting Go binary may be invalid when
the build got cancelled at the wrong point in time. This then
broke subsequent rebuilds of the binary.
Ideally, we wouldn't have to care about generating a deterministic GNU
build ID at all. But unfortunately, the only part of Go's build infra
that supports them is `go build`, so we have no easy way to avoid the
rebuild.
Instead, we can use a very ugly workaround though: when building the
binary, we embed a fixed GNU build ID with a known string and put this
binary into an intermediate location. We now derive the GNU build ID
from that intermediate binary, but instead of rebuilding it we simply
replace the known GNU build ID with the derived GNU build ID. Like this
we don't have to rebuild the binary but still get the same end result as
before.
This is implemented via a new naive Go tool that does this replacement
for us. Note that we cannot use e.g. sed(1) for this, and we don't want
to start depending on new tools like xxd(1). The tool is simple enough
though and allows us to have some additional safeguards to verify that
we are unlikely to wreak havoc on the binary.
|
|
In order to generate our `NOTICE` file, we first export all Go sources
referenced by our project into a directory and then walk that directory
to gather any licenses. The result is that the generated `NOTICE` file
contains the licenses of all our dependencies.
Ideally, the `NOTICE` file wouldn't need to contain any licenses that we
have as part of our own sources. These may for example include licenses
of test files, but also licenses of vendored dependencies. But because
we are already bundling their licenses into our sources in the first
place it shouldn't be necessary to also put them into our `NOTICE` file.
In fact, it seems like we already intended to skip soaking up any such
license files in our own tree: we initially determine the module path of
ourselves, and then return early in `filepath.Walk()` when hitting a
directory path that matches. We only return a `nil` pointer though,
which means we only skip that single directory. That doesn't seem to
make a whole lot of sense though, and it feels like the intent was to
instead skip walking that whole directory.
So let's fix this by returning `filepath.SkipDir` instead so that we
never collect licenses part of our own source tree.
|
|
Move the `noticegen` tool into the top-level `tools/` directory so that
all of our custom build tools are in one place. This also makes its
sources discoverable for our formatter.
|
|
Move the `module-updater` tool into the top-level `tools/` directory so
that all of our custom build tools are in one place. This also makes its
sources discoverable for our formatter.
|
|
The Protoc plugins we use are hidden away deep into the `proto/`
directory, which makes it very hard to discover them when one doesn't
already know about their existence. Let's move them into a new top-level
`tools/` directory.
|
