| Age | Commit message (Collapse) | Author |
|---|
|
While we already have caches for build-time dependencies and Ruby Gems,
we don't have any for Go modules. As a result, each job needs to fetch
and build Go modules anew, which adds to the time it takes for these
jobs to finish.
Add a new Go module cache and include it as required.
|
|
The Ruby cache is pointing to `ruby/vendor`, but we're not in fact
fetching our Gems into this directory anymore. Fix this by setting
`BUNDLE_PATH`.
|
|
We're currently using a single cache for both our build-time
dependencies like Git and libgit2, and for our Ruby Gems. Ideally, this
cache would also include our Go modules, but adding this in is busting
the size limitations.
Prepare for introducing a separate Go modules cache by splitting up the
Ruby and dependency caches. Like this, each job can include only the
caches it really needs, which reduces the time it takes to fetch and
decompress the archives. And furthermore, by splitting up the caches, we
reduce their respective sizes such that we can then easily introduce
another cache for Go modules, only.
|
|
In many of our jobs, we execute `git version` to report the version of
Git that was in use. In some of our jobs this is misleading though given
that we report the system-installed version of Git, which is not the one
that's used to execute our tests. And in other cases it's broken because
we execute the self-built version before calling `make git`, which only
works if we've been able to restore the cache containing this binary.
Remove reporting of the Git version altogether. We can derive which Git
version is in use by checking `GIT_VERSION` and our Makefile. While at
it, fix the logic which tries to delete our Git installation when
testing bundled Git in case the installation doesn't exist.
|
|
We frequently see issues in CI where we're seemingly stuck terminating
connections to the database when using PgBouncer. While the root cause
is not clear at this point in time, PgBouncer has fixed a bug in v1.16.0
where connections were left hanging and didn't get cancelled as expected
when the pool size was full -- which may or may not be related to the
issues we're seeing.
Upgrade the PgBouncer version to v1.16.1 to potentially fix the issue.
|
|
The PgBouncer test job requires both Postgres and PgBouncer to run
concurrently. While the former runs as a service, the latter doesn't.
Instead, we run it manually and let it daemonize into the background via
the start script. This has the downside that we require PgBouncer to be
part of the build image, and furthermore it's not good practice.
Convert PgBouncer to instead run as a service.
|
|
Use gotestsum instead of go-junit-report
See merge request gitlab-org/gitaly!4151
|
|
In the Makefile, GIT_VERSION is overwritten when it is default. This
makes the junit filename no longer match what is specified in
gitlab-ci.yml. Instead here we are simplifying down to a static
filename. There shouldn't be any risk of the report being overwritten
since CI runs each permutation of the test matrix in separate
containers.
|
|
praefect: Track database schema in Git
See merge request gitlab-org/gitaly!4130
|
|
Signed-off-by: Balasankar "Balu" C <balasankar@gitlab.com>
|
|
Add a new test job that exercises our ability to use bundled Git
binaries to drive our test suite.
|
|
Given that our Praefect database schema is seeded by migrations, there
is no single place which gives an overview over the complete schema.
While this is information that can be easily obtained by connecting to a
Praefect database, this workaround doesn't really work across different
Praefect versions given that one would have to re-seed the database for
each version one is about to investigate. This makes it hard to get info
about the current database schema and to compare changes to the schema
between different versions.
Introduce a new script "generate-praefect-schema" to fix this issue. The
script connects to a Postgres server, creates a new database, seeds the
database by executing `praefect sql-migrate` and then dumps the
resulting schema. Furthermore, a new Makefile target automates this and
writes the dump into our `_support` directory.
The intent is that whenever the database changes, the author of those
changes must update the schema in our `_support` folder such that it's
easily possible to diff the schema across different versions in our Git
history.
To ensure that the schema is always up to date, a new CI job is added
which performs the dump and then checks that no changes occurred.
|
|
Specify the location of the junit report in the test_template anchor so
every job using the anchor has it.
|
|
The junit report is mentioned as normal artifact *and* report. This is
not needed, so remove it from the regular artifacts.
|
|
While we have a global default image, this default image is not
parameterized with the Ruby and Go version. Because of this, we have to
redefine the image in multiple locations, while we don't in others. This
is confusing, and ultimately completely unnecessary: we can just add the
variables in the default image given that we also specify both Ruby and
Go versions as default variables anyway.
Remove the duplicate image definitions and instead parameterize the
default image.
|
|
Nowadays, all tests use the Postgres database even when not using the
"test-with-praefect" target. As a result, each test job we have defined
includes both the "test_definition" and the "postgres_definition".
Remove this duplication by inlining the "postgres_definition" into the
"test_definition". While at it, also inline the "pgbouncer_definition"
into the only job which is using it.
|
|
The Praefect smoke test job sets up the Postgres service on its own,
even though we have the `postgres_definition` which would already go
most of the way of setting it up correctly.
Convert the test to use the shared Postgres service definition instead
to avoid code duplication. The only thing that needs to change for the
definition is that we didn't have a database name defined at all, which
is added as part of this commit.
|
|
The "coverage" job is executing tests just as all the other targets we
have, but it doesn't include the "test_definition". Fix it to instead
include "test_definition" such that we can iterate more easily on all
test jobs at once. This also lets us get rid of the stage definition
given that it's already part of the test definition.
|
|
The verify job runs, amongst other things, the lint target in the
Makefile. So the job to run `make lint` is redundant, therefore this
change removes the lint job.
|
|
We have two jobs which execute the "verify" and the "proto" targets,
respectively to check whether Go modules are tidy, whether the NOTICE
file is up-to-date, whether Protobuf files have been generated and to
run linting. The "verify" target does all the "proto" target already
does though, so the second job to explicitly call this target is
duplicating functionality we already have. The only additional thing it
brings to the table is that it will upload changed Protobuf files as
artifacts if this target fails.
Merge the "proto" job into the "verify" job to fix this duplication.
|
|
We don't use the "publish" stage at all. Drop it.
|
|
Job names we have are quite inconsistent, making it hard to spot jobs of
specific stages and jobs which are related to each other. Rename them to
have a common stage-specific prefix.
Ideally, we'd do the same for "analyze"-style jobs like linting or
static analysis. Most of these are included via external templates
though, so changing their names wouldn't work.
|
|
The CI jobs we create have grown all over the place without a strict
schema. This makes it harder than necessary to spot related jobs, e.g.
all jobs which execute Gitaly tests.
Reorder the jobs by stage and by functionality to make this easier.
|
|
Our CI jobs all test against Postgres 12. Given that our minimum
required Postgres version is currently v11, this has allowed us to
introduce an incompatibility with Postgres 11 into our codebase which
wen't undetected by CI.
Add another job which also tests against the minimum required Postgres
version to avoid this in the future.
|
|
We needlessly include the default versions of some of our components in
job matrices. This makes it harder to spot which locations need updates
and which locations don't.
Remove these default versions to avoid this.
|
|
When the merged result pipeline is triggered the child pipeline clones
Gitaly. But it does not fetch the merged result revision, because
there's no branch referencing it. Ideally we should change the
downstream pipeline so it fetches that revision [1], but for now just
checkout the HEAD revision on the merge request source branch.
Now because we don't know whether the pipeline is triggered on a merged
result, merge train, or default branch, it might be that
CI_MERGE_REQUEST_SOURCE_BRANCH_SHA is not defined. To overcome this the
job is rewritten to use the rules definitions. This allows us to set
variables depending on some conditions.
This change needs `allow_failure: true` because it would otherwise stop
the pipeline [2].
1. https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/6482
2. https://docs.gitlab.com/ee/ci/yaml/index.html#rulesallow_failure
Issue: https://gitlab.com/gitlab-org/gitaly/-/issues/3852
|
|
A child QA pipeline is triggered with ALTERNATIVE_SOURCES=true and a
GITALY_SERVER_VERSION set to a specific revision. This makes QA expect
the revision to be on gitlab.com/gitlab-org/gitaly.git, but in some
cases that's not guaranteed (i.e. changes made to the security-release
fork).
This change passes on GITALY_SERVER_ALTERNATIVE_REPO so the QA project
pulls Gitaly from wherever the pipeline was triggered and will find the
revision specified.
|
|
We've grown a bunch of patches which get applied to Git such that we can
land improvements in production faster. These patches get applied by
default, and in fact they also get applied when the user specifies a
different Git version than the default one. This is quite annoying given
that the patches are unlikely to apply on an arbitrary Git version, and
it thus requires callers to always pass `GIT_PATCHES=` in order to not
apply patches when using a custom version. Furthermore, it's a tad
dangerous: if the user explicitly requests a different Git version and
doesn't override GIT_PATCHES, then we'd also amend the current extra
version to indicate the Gitaly patch level.
Fix this issue by only applying patches by default in case the default
Git version is in use. Users can still explicitly request patching by
setting GIT_APPLY_DEFAULT_PATCHES, if they want to.
|
|
Even though the default version of Git is nowadays specified by our
Makefile and built ad-hoc, we still specify the same default version in
our CI instructions. This is needless effort, and sure enough we have
already diverged in that CI uses Git v2.31.0 while the default version
is Git v2.31.1.
With the recent change to interpret an empty GIT_VERSION to mean the
default version we can fix this though: in theory, we could just set the
GIT_VERSION to the empty string by default in our CI instructions and
then we'd automatically get the default Git version. It's a bit unhandy
though given that we'd now end up with weird cache and artifact names
because the Git version is embedded in their names. Instead, introduce
another way to request the default Git version which is to pass
GIT_VERSION=default.
While at it, this commit fixes another bug with setting the Git version:
while we correctly handle the case where the Git version is passed via
environment variables and tweak it accordingly, this doesn't work when
invoking `make GIT_VERSION=default`. This is because by default, make(1)
does not assign to variables which have been explicitly defined by the
user. This issue is fixed by adding an explicit `override` directive.
|
|
Our default cache definition which is includde by most of our jobs
doesn't specify a cache policy, which thus means that by default they'll
use the push-pull policy. This doesn't make all that much sense though:
we have preparatory "build" jobs which build sources, and these should
be the only ones updating the cache.
Refactor the cache definition to use a "pull" policy by default and
explicitly override it in the build jobs. While at it, this commit also
deduplicates the different definitions by making better use of anchors.
|
|
Back when we introduced the libgit2 and git build targets, we generated
a `Makefile.sha256` file to see whether contents of the Makefile have
changed and thus whether we need to rebuild these targets. This has
since been replaced with a much finer-grained mechanism which records
the version of each of our build targets separately such that only those
targets are rebuild which have actually changed. Our CI cache still
refers to the old file path though, which is nowadays unused.
Remove references to `Makefile.sha256`.
|
|
The backwards_compatibility CI job runs the tests with the latest
migrations against the test suite from the previous version. While
testing the backwards compatibility of the migrations is important,
the current approach has problems that make it unfeasible in practice.
1. The CI job figures out the previous version by reading the VERSION
file from Gitaly's repo. This version file appears to be unupdated
in Gitaly's master branch. As it is, it points to 14.3.0-rc2 when
in fact it should point to some release candidate of 14.4. This means
the tests are not against the latest release but the one before that.
2. Some assertions the tests make fail with newer migrations even if
everything would be fine with a real setup. Some tests haven't setup
their test state to fully match the production setup, and thus fail
when the newer migrations are applied. In the same fashion, some tests
fail with newer migrations even if they've setup their test state
correctly. They may however make assertions on how the state looks like
after running some code, yet the new migrations cause the state to look
different from the asserted state. As an example, Praefect's deletion
handling was changed in gitlab-org/gitaly!3906. The new code removes all
records of a repository when the repository is deleted. The old code
testing deletion fails as it still expects the records to be in place.
The new code is backwards compatible though and would process deletion
jobs scheduled by the old code.
While testing for backwards compatibility is important, this commit removes
the job as it's not clear how to solve the problems. This unblocks further
fixes from being merged.
|
|
The database backwards-compatibility job is testing whether an old
version of Praefect continues to work correctly in case it's running
with a database which has been migrated to a newer schema. To do so, we
check out the previous minor release, but restore migrations from the
current version. While this works alright in case there are only changes
to the migrations themselves (e.g. only migrations have been added), it
falls apart when the setup of migrations itself changes.
An upcoming change will remove setup of the migration package in
"migrations.go" and move it into local state. Naturally, if we run an
older version which does not yet set up that local state in the other
packages with a "migrations.go" file which doesn't set up the table name
anymore, the only result can be failure.
Fix the issue by not modifying "migrations.go" anymore.
|
|
We're about to drop support for Git versions older than v2.33.0. Drop
tests for these versions from the CI definitions.
|
|
We're about to phase out support for Go 1.15, so this commit drops
all build jobs which use this version.
|
|
Our CI job which generates code navigation info has been broken for
quite a while already. Remove it given that it has repeatedly been
breaking in the past without anybody caring too much.
|
|
We have recently started to grow a set of Git patches on top of our own
Git version. While the intention is that those patches are only applied
for the default Git version of ours, where Gitaly must support Git with
and without these patches, we have indeed been applying those patches to
all our currently tested Git versions. In fact, it's by pure chance only
that all patches apply to v2.31.1, v2.32.0 and v2.33.0.
We're about to add some patches which do not apply on the older Git
versions and which thus break those builds. This commit thus fixes the
CI job definitions for non-v2.33.0 version by explicitly overriding the
GIT_PATCHES variable such that our Makefile won't set up the default set
of patches anymore.
|
|
Back in ce3f2ee42 (Lint: split up "make lint" and "make lint-strict"
configs, 2021-01-19), we have split up our linting rules into a "normal"
set which needs to be satisfied for each CI run and a "strict" set which
only runs on scheduled pipelines. I highly doubt that anybody ever takes
a look at the strict linter job: there are no MRs which clean up any of
its additional violations.
Let's remove this job and be done with it. Anybody wishing to fix up
linting rules can just play with our normal linting rules and is
unlikely to use CI for this anyway.
|
|
|
|
Upgrade the default Git version to v2.33.0, released on August 16th.
This Git version has been tested as part of our nightly pipelines for
quite some time already given that we always test "master" and "next"
branches.
Changelog: changed
|
|
Drop GIT_VERSION in matrix jobs when it only redeclares the default
version of Git which is already defined as a top-level variable anyway.
|
|
The newer shared library contains CPU instructions
that are incompatible with some customer machines.
This means we also have to stop building against
Ruby 3 for now since the update was required for
that to work.
Changelog: fixed
|
|
Now that our tests by default also include our Postgres tests, running
`make test` always requires a database to be set up. While our normal
test jobs does, the nightly job doesn't and thus always fails right now.
Fix this issue by merging in the Postgres template into the nightly job.
|
|
The test CI job duplicates the Postgres database setup which is alreay
provided by the Postgres template. Let's deduplicate this by including
this template.
|
|
Split up Postgres and PgBouncer templates such that we can reuse the
Postgres template for our jobs without having to include PgBouncer.
|
|
Use gitlab-dangerfiles for review roulette
See merge request gitlab-org/gitaly!3734
|
|
Now that we have a gemfile it's easier to use the standard ruby image
and install our dependencies to run danger.
|
|
We should keep version of the database up to date with
the officially supported version of it for the GitLab.
The source is https://docs.gitlab.com/omnibus/package-information/postgresql_versions.html
That is why we update is to the 12.6 version for the
gitaly project.
|
|
As we always require a PostgreSQL DB instance to be
running during the test execution it makes no more
sense to guard SQL dependent tests with a special
build tag. The change removes the flag from the code.
As we modify files that has a lint issues we should
fix them as well otherwise the lint job will fail.
We still need praefect_sql_test CI job because it
verifies the service works properly with PgBouncer
instance between service and database.
|
|
To get more realistic tests we replace in-memory implementation
of the replication events queue with Postgres based.
It will require Postgres database to be available during unit-test
execution, but now it is not a problem as it is required for the
test-with-postgres task already. The caller is now responsible for
providing PGPORT, PGHOST and PGUSER env vars properly configured
before the test run.
As the tests run in parallel for each packages and each package has
its own database there is no problem of concurrent data change
until we have parallel tests running inside one package.
The name of the database is generated based on the package name
where from the setup function is called.
|
