| Age | Commit message (Collapse) | Author |
|---|
|
The ubi image used for FIPS testing does not have Python3 installed by
default. Install it as part of the `before_script` steps.
|
|
|
|
|
|
Update to Ruby v3.2.2 to match GitLab.
|
|
As we have stopped building git 2.41, let's bump the minimum version on
the CI to 2.42.
|
|
|
|
We're currently running TransactionManager's tests with SHA256 as
part of the SHA256 enabled tests. We're not running the `test-wal`
and `test-with-praefect-wal` jobs with SHA256. Those jobs will wire
up transaction management into the test server and run all of our
service tests with WAL enabled. Let's run them also as part of the
SHA256 tests to ensure the service tests work fine with WAL and
SHA256.
|
|
Now that we have dropped gitaly-git2go there is no need to build libgit2
anymore. Drop it from our Makefile.
|
|
Add support for Go 1.21 both in our CI system and in our asdf version
file.
|
|
We specify about a dozen different versions in our CI definitions, and
many of these versions are repeated in multiple locations. At times,
this makes it hard to update one of these versions while making sure
that all of the places have been updated that should be.
Unfortunately, it is not possible to have YAML anchors that refer to a
single plain string, only. That is, the following is not possible:
```yaml
.versions:
git: "v2.41.0" &version_git
```
We are lucky though, because at GitLab we have noticed that this is
indeed an issue and have introduced the `!reference` extension that is
available in GitLab 13.9 and later. So let's unify all our versions into
a common dictionary so that we can easily reference them. This has two
major advantages, namely that all used versions are defined in a single
spot and that the versions are deduplicated.
|
|
For debugging CI it is often useful to be able to run a pipeline outside
of the context of an MR. The GitLab project has a `FORCE_GITLAB_CI`
variable that forces a pipeline to run even when the workflow
requirements are not met.
Add an equivalent `FORCE_GITALY_CI` variable to allow us to manually
run pipelines as needed.
|
|
.gitlab-ci.yml: Allow retry of jobs
See merge request https://gitlab.com/gitlab-org/gitaly/-/merge_requests/6135
Merged-by: Justin Tobler <jtobler@gitlab.com>
Approved-by: Justin Tobler <jtobler@gitlab.com>
Reviewed-by: Justin Tobler <jtobler@gitlab.com>
Co-authored-by: John Cai <jcai@gitlab.com>
|
|
d1eec76f3 (ci: Rename rules to be more consistent, 2023-07-19)
refactored the rules to be more consistent. However, in the process it
removed the ability to re-try jobs manually. Add a rule that allows
that.
|
|
ci: Mark all Rails specs jobs as manual
See merge request https://gitlab.com/gitlab-org/gitaly/-/merge_requests/6116
Merged-by: Toon Claes <toon@gitlab.com>
Approved-by: Toon Claes <toon@gitlab.com>
Co-authored-by: Patrick Steinhardt <psteinhardt@gitlab.com>
|
|
ci: Drop optimization to run tests with code changes only
See merge request https://gitlab.com/gitlab-org/gitaly/-/merge_requests/6115
Merged-by: Patrick Steinhardt <psteinhardt@gitlab.com>
Approved-by: Quang-Minh Nguyen <qmnguyen@gitlab.com>
|
|
We have recently added a set of jobs that allow us to execute Rails
specs. We only marked a subset of these as manual though, which is why
we now observe test failures in our scheduled pipelines.
Mark all of these jobs as manual to fix this.
|
|
We have recently started an experiment where we changed our tests jobs
to only run in case there are some actual code changes. This change has
proven to be very fragile as we had many contexts now where we didn't
run any test jobs despite some code changes.
The original intent of this change was to skip running tests when we
know that there were only documentation changes. Our CI rules don't have
such a condition though, which means that we are forced to express the
opposite rule that there were only code changes. This has the downside
though that we now fail open (that is we err on the side of running too
few CI jobs) instead of failing closed (which would be to rather run too
many CI jobs). This is not a good failure mode though as it is now quite
easy for us to miss test failures introduced by a merge request.
Drop the optimization again and always run tests, regardless of which
files have changed.
|
|
Currently the `rails-specs` jobs rely on the `CI_MERGE_REQUEST_IID` and
`CI_MERGE_REQUEST_SOURCE_BRANCH_SHA` variables to execute. However, if a
pipeline is run outside of the context of an MR then these variables are
undefined and the jobs will fail.
To resolve this, rely on the pipeline iid to uniquely identify the
branch created in 'gitlab-org/gitlab'.
|
|
Use Golang 1.20 as minimum required version
See merge request https://gitlab.com/gitlab-org/gitaly/-/merge_requests/6056
Merged-by: karthik nayak <knayak@gitlab.com>
Approved-by: karthik nayak <knayak@gitlab.com>
Reviewed-by: Patrick Steinhardt <psteinhardt@gitlab.com>
Reviewed-by: Justin Tobler <jtobler@gitlab.com>
Reviewed-by: karthik nayak <knayak@gitlab.com>
Co-authored-by: Justin Tobler <jtobler@gitlab.com>
Co-authored-by: Nao Hashizume <nhashizume@gitlab.com>
|
|
|
|
Our list of files that relate to code changes is missing some items:
- `go.mod` and `go.sum` files in our `tools/` directory. This rule
is important because changes in our tooling may have a ripple
effect.
- `*.go` files in our root directory, like the `packed_binaries.go`
file.
Change all of the recursive patterns as well as the `go.*` pattern to
match both in the root directory as well as in recursive directories so
that this becomes less fragile.
|
|
When a job is skipped because no code changed there is no way to run the
job manually anymore. Fix this by adding an optional `manual` step to
the pipeline for merge requests.
|
|
Rename the rules `rules_should_test_all` and `rules_should_test_code` to
`rules_run_always` and `rules_run_on_code_change` to better match other
rules. Furthermore, the imperative mode of the new names feels a bit
easier to reason about.
|
|
.gitlab-ci.yml: Use environment to clean up branch
See merge request https://gitlab.com/gitlab-org/gitaly/-/merge_requests/6065
Merged-by: Justin Tobler <jtobler@gitlab.com>
Approved-by: Lin Jen-Shin <jen-shin@gitlab.com>
Approved-by: Justin Tobler <jtobler@gitlab.com>
Reviewed-by: Lin Jen-Shin <jen-shin@gitlab.com>
Reviewed-by: Quang-Minh Nguyen <qmnguyen@gitlab.com>
Co-authored-by: John Cai <jcai@gitlab.com>
|
|
|
|
In eb9491a0ce (ci: Allow `test:macos` job to fail, 2023-04-28), we have
adapted our macOS job to fail. This was because we were frequently
observing flakes and errors due to various reasons.
This has since been fixed by migrating to GitLab's new macOS runners and
speeding up their tests significantly by using a RAM disk. Since then,
the job is significantly more stable and finishes faster than e.g. the
FIPS jobs.
Let's require the macOS job to succeed again. While not required for
production systems, it is important for many of GitLab's developers as
the majority does indeed use macOS on their development machines.
|
|
.gitlab-ci.yml: Test code only
See merge request https://gitlab.com/gitlab-org/gitaly/-/merge_requests/6082
Merged-by: John Cai <jcai@gitlab.com>
Approved-by: Justin Tobler <jtobler@gitlab.com>
Reviewed-by: Justin Tobler <jtobler@gitlab.com>
|
|
With 9b1ef44c, the GitLab Rails suite of tests was introduced to the
Gitaly pipeline and required to pass in order for merges to be accepted.
Unfortunately, the Rails spec jobs are frequently flaky thus blocking
merges.
Set the `start-rails-specs` CI job to only run if executed manually.
Also remove the rules from `rails-specs` and `rails-specs-cleanup` so
they execute automatically when `start-rails-specs` is triggered.
|
|
Fix a bug where we are skipping tests if a change includes *any*
non-code related changes. Use the `changes` keyword to only include code
level changes for tests that care about code.
|
|
For the CI pipeline to run correctly the `rail-specs` job needs the
`start-rails-specs` job to run and the `rails-specs-cleanup` job needs
the `rails-specs` job to run. These jobs have mismatched rules though
preventing them from executing in the same context, which can lead to
pipeline errors.
Make the CI rules for the three jobs all use `rules_run_on_merge` to
ensure they run in the same context.
|
|
Adding a job to run the full rails suite of tests. This allows
developers to trigger the job to see if their changes pass the rails
test suite.
We have three jobs to accomplish this. First, start-rails-specs will
create a branch and modify the GITALY_SERVER_VERSION on it. Then,
rails-specs will run the Rails test suite on that branch. Finally,
rails-specs-cleanup will delete that branch.
|
|
We don't need the full pipeline to run for things in the .gitlab/
folder, nor any documentation in .md files. Create a rule to skip tests
if we detect such files.
|
|
This change sets the required minimum version of Git to v2.41 so new Git
features can be used.
|
|
Back when we migrated to Go 1.20 we didn't yet have a FIPS-enabled image
for that Go version available. Now we do, so let's update our CI job to
use the new Go version.
|
|
Bump version of markdownlint-cli2 in project
See merge request https://gitlab.com/gitlab-org/gitaly/-/merge_requests/5959
Merged-by: Patrick Steinhardt <psteinhardt@gitlab.com>
Approved-by: Patrick Steinhardt <psteinhardt@gitlab.com>
Co-authored-by: Evan Read <eread@gitlab.com>
|
|
|
|
|
|
|
|
We see some ENOSPC errors in our test jobs since we have migrated them
to use a tmpfs. Seemingly, 2GB of storage is not sufficient.
Increase the tmpfs partition to 4GB instead. Given that the large
runners have 32GB of RAM the machine should trivially handle this.
|
|
Now that we're using a larger machine for running tests it is more
likely that we hit the connection limit for PgBouncer due to higher
concurrency of tests. Let's bump the limit to avoid this problem.
|
|
GitLab has recently launched a set of SaaS runners which are a lot more
beefy than the GitLab-group-specific runners: while our group-specific
runners only have two CPU cores, the large SaaS runners have eight.
Given that our tests are neatly parallelized more cores directly
translate into faster job execution.
Migrate to these large SaaS in order to decrease job times
significantly, from an (uncached) 16 minutes to 7 minutes.
|
|
It seems like part of our slowness in CI systems boils down to all the
data we're writing. The I/O seemingly causes quite some disk contention
and thus slows the tests down.
Convert our tests to use tmpfs. For uncached test runs this decreases
the test execution time from around 579 to 509 seconds, which is more
than a minute and thus considered a (small) win.
|
|
Gatekeeper is a security feature of macOS intended to secure the
operating system against executing malicious code. This feature works by
checking signatures of executables against known-malicious code and
verifying signatures. But while this may be a good idea for normal
desktop and server usage, it doesn't help us much in the context of CI
pipelines. Quite on the contrary: as our Gitaly test suite is quite
heavy on executing binaries the check even causes a quite noticeable
slowdown.
Disable Gatekeeper in our macOS job. This reduces test times from around
600 seconds to 480 seconds.
|
|
The macOS-based test jobs are extremely slow, typically taking about 55
minutes or more. It seems like the root cause for this is the ton of I/O
that we perform during our tests, where the disk becomes our bottleneck.
Convert the job to use a RAM disk for our test data. This results in a
significant speedup of test execution from around 2500 seconds to only
around 600 seconds.
|
|
The optional macOS job is quite slow and only ever runs when a merge
request is about to get merged. It is thus not useful to collect
coverage info here, as it wouldn't typically be visible in the merge
request anyway.
Stop collecting coverage information to speed it up.
|
|
Starting with 77c67bb94 (Generate coverage report from all test runs,
2023-05-21), we generate coverage information by default. The runtime
overhead of recording the data is comparatively negligible with a 3%
slowdown. But interestingly, it does take quite a while to generate the
resulting Coverage report.
Fix this by making generation of coverage info optional with the newly
introduced `TEST_WITH_COVERAGE` variable. This allows us to only compute
the data as needed and provides a very noticable speedup:
Benchmark 1: make test-go
Time (mean ± σ): 69.015 s ± 1.001 s [User: 521.524 s, System: 171.512 s]
Range (min … max): 67.865 s … 69.690 s 3 runs
Benchmark 2: make test-go TEST_WITH_COVERAGE=YesPlease
Time (mean ± σ): 120.624 s ± 1.185 s [User: 673.246 s, System: 207.668 s]
Range (min … max): 119.470 s … 121.838 s 3 runs
Summary
'make test-go' ran
1.75 ± 0.03 times faster than 'make test-go TEST_WITH_COVERAGE=YesPlease'
As we do indeed want to have coverage information available in our CI
systems the variable is set in our test jobs by default.
|
|
The variable `TEST_WITH_SHA256` controls whether Gitaly's test suite
should run with SHA1 or SHA256. It is quite long and thus unhandy to
write, and doesn't match the remaining test-related variables we have in
our Makefile.
Rename the variable to `TEST_WITH_SHA256` for brevity and consistency.
|
|
Our macOS job spews turns of the following warning:
ld: warning: object file (/Users/gitlab/builds/djQLaC-y/1/gitlab-org/gitaly/_build/deps/libgit2/install/lib/libgit2.a(patch_parse.c.o)) was built for newer macOS version (12.6) than being linked (12.0)
This warning seems to come from CGo when we try to link gitaly-git2go
against libgit2.a, where CGo and CMake seemingly chose different
deployment targets. While seemingly harmless, those warnings are quite
annoying given that they make the CI output basically unreadable.
Fix this warning by setting the MACOSX_DEPLOYMENT_TARGET variable, which
should enforce a consistent target across the build.
|
|
GitLab has recently deprecated the Intel-based macOS runners. As we are
still using them as part of our pipeline the consequence is that the
pipelines are now broken.
Fix this by updating to the new M1 runners.
|
|
With the move to M1-based macOS runners the Postgres database directory
is about to be moved. Instead of relying on any hardcoded locations,
let's convert the CI job to initialize our own database at a location
controlled by us.
|
