| Age | Commit message (Collapse) | Author |
|---|
|
This commit changes log package's Configure function to take the
loggers as a parameter and fixes up call sites to pass the global
loggers explicitly. This aims to make the function more testable.
|
|
Adds a configuration option for the default replication factor to
use for new repositories in a given virtual storage. This allows
Praefect to configure new repositories directly with the desired
replication factor instead of the administrator having to manually
set the replication factor of a repository after it has been created
already.
|
|
ElectionStrategy constants in Praefect's config are untyped. This
commit types them as ElectionStrategy.
|
|
With enabled distributed_reads feature each read operation leads to
a database query execution to get state of the storages for particular
repository. More read calls leads to more database access operations,
so the pressure to it increases in linear (or even worse).
To mitigate this problem it was decided to introduce an in-memory cache
added before accessing the database. Invalidation happens on receiving
notification events from the database. The events are send by the
triggers attached to the repositories (delete) and storage_repositories
(insert, delete, update) tables.
To monitor the cache a new counter was added: gitaly_praefect_uptodate_storages_cache_access_total.
It tracks amount of cache hits, misses and populates and evicts per virtual
repository. And to track an error rate of the notifications processing the
gitaly_praefect_uptodate_storages_errors_total was added with type set to one of:
retrieve, notification_decode.
Closes: https://gitlab.com/gitlab-org/gitaly/-/issues/3053
|
|
Praefect's config validation tests duplicate the meaning of a valid
config. This commit removes that duplication by letting each test
case make changes on a valid config instead of defining a fully
valid config each time.
|
|
Praefect currently does not validate election strategy configuration.
This commit adds validation for the config option.
|
|
Use github.com/pelletier/go-toml as its a more active project.
Close https://gitlab.com/gitlab-org/gitaly/-/issues/3182
|
|
In order to omit misunderstanding in configuration the old settings
remain without changes and to specify a direct connection a new
settings were added: host_no_proxy and port_no_proxy.
Connections should use old host and port settings as before without
changes. If there is a connection that requires a direct link to
database it should use new settings: host_no_proxy and port_no_proxy.
Closes: https://gitlab.com/gitlab-org/gitaly/-/issues/3119
|
|
or to the PgBouncer instance set in front of it. That is why
proxy_host and proxy_port configuration settings were added.
All connections need to be done via proxy if it is configured.
Exception to it is a listening connection that must be a direct
connection to the database instance.
Closes: https://gitlab.com/gitlab-org/gitaly/-/issues/3119
|
|
Enables automatic reconciliation with by default with a five minute
interval.
|
|
Since the introduction of Praefect, our code layout started to become
confusing: while Praefect code lives in `internal/praefect`,
Gitaly-specific code is all over the place and not neatly singled out.
This makes it hard at times to tell apart Praefect- and Gitaly-specific
from generic code.
To improve the situation, this commit thus moves most of the server
specific code into a new `internal/gitaly` package. Currently, this is
the `internal/config`, `internal/server`, `internal/service` and
`internal/rubyserver` packages, which are all main components of Gitaly.
The move was realized with the following script:
#!/bin/sh
mkdir -p internal/gitaly
git mv internal/{config,server,service,rubyserver} internal/gitaly/
find . -name '*.go' -exec sed -i \
-e 's|gitlab-org/gitaly/internal/rubyserver|gitlab-org/gitaly/internal/gitaly/rubyserver|' \
-e 's|gitlab-org/gitaly/internal/server|gitlab-org/gitaly/internal/gitaly/server|' \
-e 's|gitlab-org/gitaly/internal/service|gitlab-org/gitaly/internal/gitaly/service|' \
-e 's|gitlab-org/gitaly/internal/config|gitlab-org/gitaly/internal/gitaly/config|' {} \;
In addition to that, some minor adjustments were needed for tests which
used relative paths.
|
|
Automatic repository reconciliation
Closes #2717
See merge request gitlab-org/gitaly!2462
|
|
Replication jobs are currently only attempted three times at maximum
until they are dropped. After a replication job is dropped, its changes
are not going to be replicated until another replication job is triggered
form another mutator operation or an administrator runs reconcile manually.
Repositories can stay outdated for an indefinite time.
This commit addresses the problem by adding a background goroutine that
automatically schedules replication jobs from storages which have a fully
up to date copy of the repository to storages with outdated copies of the
repository. Replication jobs are only scheduled from and to storages which
are currently healthy to avoid scheduling replication jobs that are not
going to be successfully completed. If there is a replication job already
targeting the repository, a redundant reconciliation job won't be scheduled.
|
|
Hooks up the error tracker in the node manager so it checks if a certain
backend node has reached a threshold of errors. If it has, then it will
be deemed unhealthy.
|
|
Adds a configuration option for controlling the replicator's job
dequeueing batch size.
|
|
As we are moving towards enabling all feature flags and
configuration options for praefect and gitaly in tests by
default we require a configurable bootstrap and check intervals
for the health check request praefect executes on gitaly nodes.
This needed because of 'test-with-praefect' verification we
have where praefect is used in front of the gitaly server in
order to verify the integration is works properly.
With this settings we are able to minimize bootstrap interval
to reduce time needed to await until praefect would be ready
to serve requests. Also monitoring interval configuration allows
to reduce time required to get a gitaly node become unhealthy.
Part of: https://gitlab.com/gitlab-org/gitaly/-/issues/2702
|
|
With the introduction of per repository read-only mode, this commit
removes the previous virtual storage wide read-only mode
implementation.
|
|
|
|
Praefect: include PgBouncer in CI
Closes #2816
See merge request gitlab-org/gitaly!2378
|
|
Adds an in-memory and a Postgres implemenation of repository
state tracking stores. The two tables added are:
1. `repositories` which contains the expected state of repositories
in the virtual storage.
2. `storage_repositories` which contains the state of the repositories
on the physical storages.
Cross-referencing these two makes it easier to identify outdated
repositories.
This commit implements only the stores without hooking them up to the
rest of the code.
|
|
Run 'praefect_sql_test' with a new docker image that includes
PgBouncer. All interaction with PostgreSQL database for this
job are going throw a PgBouncer.
The connection throw PgBouncer has some limitations that is
why `default_timeout_ms` configuration option was removed
and 'statement_timeout' is not set for the connection.
The version of the PostgreSQL was also upgraded to 11.8.
Closes: https://gitlab.com/gitlab-org/gitaly/-/issues/2816
|
|
Praefect allowed to use TLS address to listen on in
combination with certificate and its key to be used for it.
'metadata' package refactored, so TLS address can be passed
into the Gitaly.
Documentation fixed and example file filled with proper vals.
Closes: https://gitlab.com/gitlab-org/gitaly/-/issues/1698
|
|
Introduces server factory for creating gRPC servers.
Praefect gRPC server created by separate function and
can be reused in tests to check routing.
Part of: https://gitlab.com/gitlab-org/gitaly/-/issues/1698
|
|
|
|
failover: Default to enabling SQL strategy
Closes #2682
See merge request gitlab-org/gitaly!2218
|
|
Given the SQL strategy is a reasonable default, and tested well enough,
this change makes the SQL strategy the preferred strategy. This was
`local`, which creates dataloss scenarios when multiple Praefects are
behind a load balancer.
Additionally, this changes behaviour around the read only feature. This
would default to being enabled. Currently availability is favoured over
consistency. Now this is flipped.
Fixes: https://gitlab.com/gitlab-org/gitaly/-/issues/2682
|
|
After removal of Datastore struct it make sense to move
entities of the models package into config package.
As it is has only a configuration purpose.
As well 'node' configuration removed from Config as it is
not used anymore.
Closes: https://gitlab.com/gitlab-org/gitaly/-/issues/2613
|
|
In order to exclude usage of Datastore struct as not actual anymore
migration to replication queue is node and set of virtual storages
is provided from outside in time of creation of replication manager.
Part of: https://gitlab.com/gitlab-org/gitaly/-/issues/2613
|
|
In order to reduce usage of global configuration
Bootstrap struct changed to accept duration as
parameter.
That was a cause of adding a new configuration setting
to praefect configuration file, because previously it
used default value of 1m configured for gitaly.
|
|
The reads distribution feature depends on the `distributed_reads`
feature flag, so there is no `distributed_reads_enabled`
configuration parameter anymore.
|
|
Thew PostgreSQL based implementation of the replication queue
is enabled by default.
In order to simplify unit testing the in-memory implementation
resides and can be used in case `memory_queue_enabled = true`
configuration option provided.
Closes: https://gitlab.com/gitlab-org/gitaly/-/issues/2615
|
|
Read distribution is inherently request scoped, and as such the same
infrasture could be leveraged compared to other RPCs with feature flags.
Being scoped on the request, set by Rails.
This change removes the config value, and adds the feature flag.
Relates to: https://gitlab.com/gitlab-org/gitaly/-/merge_requests/2162
Closes: https://gitlab.com/gitlab-org/gitlab-development-kit/-/issues/925
Closes: https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/5361
|
|
Praefect: horizontal scaling of a single shard MVC
Closes #2650
See merge request gitlab-org/gitaly!2162
|
|
Random distribution of reads to up to date gitaly nodes.
Up to date status of node verified based on the state of the
replication queue. Backoff strategy is to use primary node
if there are no up to date secondaries or an error occurred.
This feature can be enabled with 'distribution_of_reads_enabled'.
Closes: https://gitlab.com/gitlab-org/gitaly/-/issues/2650
|
|
Removes the feature-flag controlling read-only enforcement. The
feature flag only controlled enforcement in RPCs, making future
upgrades more difficult as some storages could already be in
read-only mode when the feature flag is removed. The feature flag
is replaced with a config toggle that also controls whether the
storages should be set read-only on a failover.
|
|
Praefect uses a .toml file as a source of configuration.
Before praefect starts handle requests the verification
must be executed to check if configuration is correct.
All `gitaly`-s must use distinct addresses.
Any virtual node must have at least one gitaly behind it.
Each virtual node must be named.
|
|
Run a query for new primary selection that calculates a replication
queue "size" that uses a heuristic for number of incomplete jobs
(dead, in_progress, ready, failed) since the last completed replication job.
|
|
|
|
This commit adds the following strategy to enable redundant Praefect
nodes to run simultaenously:
1. Every Praefect node periodically (every second) performs a health
check RPC with a Gitaly node.
2. For each node, Praefect updates a row in a new table (`node_status`)
with the following information:
a. The name of the Praefect instance (`praefect_name`)
b. The name of the virtual storage name (`shard_name`)
c. The name of the Gitaly storage name (`storage_name`)
d. The timestamp of the last time Praefect tried to reach that node
(`last_contact_attempt_at`)
e. The timestamp of the last successful health check (`last_seen_active_at`)
3. Periodically every Praefect node does a `SELECT` from `node_status`
to determine **healthy nodes**. A healthy node is defined by:
a. A node that has a recent successful error check (e.g. one in the last 10 s).
b. A majority of the available Praefect nodes have entries that match the two above.
4. To determine the majority, we use a lightweight service discovery
protocol: a Praefect node is deemed a voting member if the
`praefect_name` has a recent `last_contact_attempt_at` in the
`node_status` table. The name is derived from a combination of the
hostname and listening port/socket.
5. The primary of each shard is listed in the `shard_primaries`. If the
current primary is in the healthy node list, then no election needs to
be done.
6. Otherwise, if there is no primary or it is unhealthy, any Praefect
node can elect a new primary by choosing candidate from the healthy node
list and inserting a row into the table.
Closes https://gitlab.com/gitlab-org/gitaly/-/issues/2547
|
|
Implementation of the replication events queue now can be
switched using `postgres_queue_enabled` between in-memory
and Postgres.
`Datastore` changed from interface to struct as there is no
single struct implementation for it anymore.
Closes: https://gitlab.com/gitlab-org/gitaly/-/issues/2166
|
|
PgBouncer limits usage of placeholders in queries, because
it prepares statement first and execute it afterwards.
The problem could be if statement will be prepared using
one session, but executed using another session.
Binary format will issue only single operation without
preparing statement.
Part of: https://gitlab.com/gitlab-org/gitaly/-/issues/2166
|
|
To enable feature flag toggling in node manager, have a set of shards
that is not monitored and stays static. When an incoming request has the
feature flag, it will get directed to the version that does failover.
If it does not have the feature flag, it will get directed to the
primary as though the node manager did nothing.
|
|
|
|
|
|
Move prometheus config to its own package
See merge request gitlab-org/gitaly!1676
|
|
Praefect and Gitaly both need the prometheus config, so move it to
internal/config/prometheus so Praefect doesn't have to import the gitaly
config.
|
|
Both Gitaly and Praefect use the bootstrap package and need to play well
with gitaly-wrapper. This changes praefect to use the same env vars as
gitaly. It also moves the env var retrieval inside the bootstrap
constructor.
|
|
|
|
|
|
Refactored configs so that both praefect and gitaly can share logging
and sentry config structs
|
