| Age | Commit message (Collapse) | Author |
|---|
|
Each replication queue worker dequeues up to 10 events at a time
and executes them sequentially. If the first job in the batch is
slow, the remaining jobs end up blocked. This commit changes a
queue worker to dequeue only one event at a time, leading to finer
grained locking of events. This should allow for other free workers
to pick up events that are not yet being handled.
|
|
Metric for the number of read-only repositories
See merge request gitlab-org/gitaly!2426
|
|
Adds a Prometheus collector for RepositoryStore to expose the number
of read-only repositories within a virtual storage.
Since the state of the repositories is in the database, the current
approach leads to redundant work as each Praefect will query the
database for the number of read-only repositories. This could be
fixed later by extracting an exporter from the metrics that use the
database as their source of truth.
Alternatively, synchronization between Praefect nodes could be added
in order to coordinate which Praefect should perform the potentially
expensive queries for metrics.
|
|
Configuration of gitaly nodes health checks
See merge request gitlab-org/gitaly!2440
|
|
As we are moving towards enabling all feature flags and
configuration options for praefect and gitaly in tests by
default we require a configurable bootstrap and check intervals
for the health check request praefect executes on gitaly nodes.
This needed because of 'test-with-praefect' verification we
have where praefect is used in front of the gitaly server in
order to verify the integration is works properly.
With this settings we are able to minimize bootstrap interval
to reduce time needed to await until praefect would be ready
to serve requests. Also monitoring interval configuration allows
to reduce time required to get a gitaly node become unhealthy.
Part of: https://gitlab.com/gitlab-org/gitaly/-/issues/2702
|
|
Revert introduction of libgit2 build dependency
See merge request gitlab-org/gitaly!2437
|
|
This reverts commit 4f3253e9758affac7b7a8e7f74e97117299c4002, reversing
changes made to 40b90823b0d55561059d27249e02db426b428786.
|
|
This reverts commit bf70b7bdcef2da641c946bd79be37a49d8ccded9, reversing
changes made to 27ef86959e802d78ff51ec8b73e3d15220ff77c5.
|
|
|
|
Makefile: disable iconv support for libgit2
See merge request gitlab-org/gitaly!2434
|
|
On macOS, libgit2 will by default compile with support for iconv. As a
result, we thus have another runtime dependency on libiconv, which is
not something we want.
Fix this by explicitly disabling support for iconv.
|
|
Improve query to identify up to date storages for reads distribution
Closes #2944
See merge request gitlab-org/gitaly!2372
|
|
Retrieve up to date storages that can server read operation for the
repository in order to distribute reads across all healthy storages
of the virtual storage.
Closes: https://gitlab.com/gitlab-org/gitaly/-/issues/2944
|
|
|
|
Add build target for Git2Go
See merge request gitlab-org/gitaly!2413
|
|
PgBouncer deployment with terraform
Closes #2975
See merge request gitlab-org/gitaly!2418
|
|
Remove virtual storage wide read-only mode
See merge request gitlab-org/gitaly!2431
|
|
With the introduction of per repository read-only mode, this commit
removes the previous virtual storage wide read-only mode
implementation.
|
|
Resulting name of the IP for the SQL connection renamed
to praefect_pgbouncer_ip.
Closes: https://gitlab.com/gitlab-org/gitaly/-/issues/2975
|
|
Fix PostReceive reftx hook test
See merge request gitlab-org/gitaly!2430
|
|
Generate data loss report from repository generation info
See merge request gitlab-org/gitaly!2403
|
|
Assignment of the dedicated IP address to PgBouncer instance
in order to narrow the set of public IP addresses allowed to
connect to PostgreSQL instance.
Closes: https://gitlab.com/gitlab-org/gitaly/-/issues/2975
|
|
[ci skip]
|
|
Changes dataloss subcommand to report data loss based on repository
generations. This allows us to report more accurately, including
how many writes a repository is behind at maximum.
The report is not 100% accurate as generations map to mutator calls and
some mutator calls do not indicate unreplicated writes. These are mutators
are mainly for repository housekeeping and do make any changes visible to the
client. The accuracy can be improved by tagging these housekeeping mutators
later as such and ignoring them when it comes to incrementing the generation
of a repository.
Generations guarantee the repository was at least on the stored generation.
As such, the tool might produce false positives in case a replication was
successful but the database record failed to be updated. In case of a failover,
some of the repositories reported as outdated may contain the latest changes.
Since generations are only recorded for repositories that have received
new mutator RPCs, this won't report data loss from repositories that have
failed replication prior to the generations being introduced.
|
|
Add GetOutdatedRepositories to repository store
See merge request gitlab-org/gitaly!2416
|
|
Adds a method for getting outdated repositories for reporting purposes.
|
|
Terraform requires a new module to be installed: pgbouncer
In order to install all required modules automatically
the `terraform get` command added to main script.
The command installs all missed dependencies recursively.
Closes: https://gitlab.com/gitlab-org/gitaly/-/issues/2975
|
|
Use of 'locator' instead of func dependent on global config
See merge request gitlab-org/gitaly!2429
|
|
Update our NOTICE file to add the license for our new Git2Go dependency.
|
|
This adds a new gitaly-git2go command that is currently intended as a
proof-of-concept for our Git2Go build framework. It thus doesn't do much
except demonstrating it's possible to link against Git2Go and its
bundled lbigit2 static archive and use its functionality. At a later
point, this is going to be the entrypoint for any Git functionality that
we want to implement via Git2Go.
|
|
The new libgit2 build target will be run by CI jobs in multiple stages.
As builds require a non-negligible amount of time, it adds up and thus
slows down our CI pipeline.
Let's work against this by using a file-keyed cache. As it's impossible
to use multiple caches, we also have to convert the existing cache for
Ruby to be. The result is thus keyed by both the Gemfile.lock and
Makefile, as both are definitive sources for what's being built.
|
|
This adds a build target for libgit2 as a prerequisite dependency for
Git2Go. The build is configured as a static archive with minimal
dependencies such that the only required shared library is the system's
libc.
Instead of building libgit2 manually, we could've also gone the way of
fetching the Git2Go repository and using its scripts. This has multiple
problems though. First, the build scripts aren't flexible and will
default to auto-detection of system-provided libraries. As a result, the
generated archive will usually have more dependencies than only libc.
Second, we'd have to use a "replace" statement for our go.mod file.
While that's not a problem by itself, it will cause several targets like
e.g. linting, verification and others to depend on the presence of this
Git2Go repository.
The approach of building libgit2 manually is thus preferable, as it
gives us both more flexibility and less dependencies on libgit2 in our
build process.
|
|
Right now, we add build flags to most every Go command in order to
enable a certain set of features. This is quite tedious, and naturally
there are some invocations of Go commands which do not currently have
those tags set. As a result, some may be built inconsistently, but more
importantly manually running e.g. `go test ./...` will not use any of
those build tags at all.
There's an obscure way to configure build tags such that they're always
on, which is the "ctxt.BuildTags" file. It is mostly undocumented,
except for one small excerpt in `go doc build`:
During a particular build, the following words are satisfied:
- the target operating system, as spelled by runtime.GOOS
...
- "go1.14", from Go version 1.14 onward
- any additional words listed in ctxt.BuildTags
So it does at least seem to be officially supported, even though it's
not that well known.
Let's use it to avoid any inconsistencies. Furthermore, it will help us
when reintroducing Git2Go into our build dependencies, which will make
specifying some build tags mandatory.
|
|
Add accept-dataloss sub-command
See merge request gitlab-org/gitaly!2415
|
|
While TestPostReceiveWithReferenceTransactionHook succeeds when run as
part of the smarthttp testsuite, it fails when run standalone. The issue
is that the test didn't set up the hooks override and thus hooks cannot
be located correctly in case the test is run standalone. In case it's
run as part of the smarthttp suite, it succeeds because previous tests
failed to correctly restore the old value.
Fix the test by correctly setting up `hooks.Override` and correctly
restoring its value after the tests.
|
|
Update of PostgeSQL instance version
See merge request gitlab-org/gitaly!2428
|
|
Remove remote_branches_ls_remote feature flag
See merge request gitlab-org/gitaly!2417
|
|
This behavior has been battle-tested on gitlab.com and included in the
13.2 on-premises release defaulted to on.
Pending any last-minute catastrophic failures, we can safely remove the
old fetch-based behavior.
|
|
Migration of 'commit' service to use 'locator' injected
dependency to resolve repo path/location.
Part of: https://gitlab.com/gitlab-org/gitaly/-/issues/2699
|
|
As the target PostgeSQL version currently is 11 we should
use this version for tests and demo.
|
|
Adds 'accept-dataloss' sub-command which allows designating a new
authoritative storage for a repository. This might be necessary if
a repository is in read-only mode and recovering the data is not
viable. The current version of the repository on the authoritative
storage is considered latest and versions on the other physical
storages will be overwritten with it.
|
|
Enforce read-only mode per repository
Closes #2783 and #2862
See merge request gitlab-org/gitaly!2405
|
|
Fix Praefect not starting with unhealthy Gitalys
Closes #2847
See merge request gitlab-org/gitaly!2422
|
|
Fixes Praefect not starting successfully when dialing any of the
configured Gitaly nodes fails.
|
|
Queue replication jobs in case a transaction is unused
Closes #2955
See merge request gitlab-org/gitaly!2404
|
|
Propagate correlation_ids to Gitaly Ruby
See merge request gitlab-org/gitaly!1461
|
|
Support passing extra []grpc.DialOption to connection pool
See merge request gitlab-org/gitaly!2401
|
|
Remove GO111MODULE environment variable
See merge request gitlab-org/gitaly!2420
|
|
|
|
misc: Remove John from danger automation
See merge request gitlab-org/gitaly!2419
|
