| Age | Commit message (Collapse) | Author |
|---|
|
This commit dependency injects a shared backchannel registry to the
components that need it, namely the GitalyServerFactory and transaction
manager.
|
|
This commit injects the multiplexing handshaker from Praefect's main
to the dialing locations. This allows us to later plug in a backchannel
server easily. This commit has no changes to the functionality itself.
|
|
This commit wires up a PrimaryGetter and Connections into Praefect's
Info server as they'll be needed when converting the methods to support
variable replication factor and repository specific primaries.
ReplicationFactorSetter is also converted into AssignmentStore so the
methods can acccess assignments later.
|
|
Implementation of the linguist package relies on the
package-private state: the use of the colorMap variable.
It is filled with a hook mechanism after validation of the
gitaly configuration. The problem is that this operation
invokes ruby code to get it done and this call is pretty
expensive. It also doesn't allow us to convert tests to
parallel as it causes race conditions in attempt to fill
the global variable.
The solution to this problem is creation of the struct
instance that encapsulates the state and can be instantiated
only when needed. The change also includes minification in
generating test configuration that does not require to make
a ruby call and uses a stub file for initialisation. The
behaviour could be returned back to the normal with usage of
the WithActualLinguist option for configuration creation.
The change also includes renames of the test functions for
the linguist package. The change speeds up execution of the
tests: before 0m43.907s and after 0m21.659s on my local.
Part of: https://gitlab.com/gitlab-org/gitaly/-/issues/2699
|
|
This commit removes DirectStorageProvider as it is just a proxy to
call GetConsistentStorages on the RepositoryStore.
|
|
It's currently not possible to use our git DSL in the testhelper package
because of an import cycle between the testhelper and git package. To
fix this import cycle, we thus move test-related git helpers into the
gittest package.
This commit moves the repository helpers. As we're already touching all
sites which use these helpers anyway, it also aligns functions to have
more consistent naming.
|
|
Server factory should not manage gitaly-ruby server internally
as it completely independent entity and should be provided only
as a dependency into other components.
The change also restructure creation of the gRPC servers by the
factory, so that doesn't rely on the specific services. This gives
us un opportunity to re-use setup code in the tests and remove
the helpers that are not complete compared to actual production used.
|
|
Factory extended with additional constructor parameters to
reduce code coupling and make it dependent on abstractions.
Two server constructors combined into one that has a single
additional parameter to control tls usage.
Part of: https://gitlab.com/gitlab-org/gitaly/-/issues/2699
|
|
We'll need the transaction manager in the RepositoryService in order to
enable transactional voting for RPCs which modify the repository state.
So let's inject the manager to make it available.
|
|
The `testhelper.GetTemporaryGitalySocketFileName()` function currently
returns an error. Given that all callers use `require.NoError()` on that
error, let's convert it to instead receive a `testing.TB` and not return
an error.
|
|
The default minimum TLS version in Go is currently TLS 1.0. Because
there are known weaknesses and vulnerabilities in both TLS 1.0 and
1.1, GitLab has announced the end of support for those versions on
October 2018 already [1]. But Gitaly never followed through and still
accepts the old TLS versions.
Fix this by raising the minimum required TLS version to TLS 1.2 for both
client- and server-side connections.
[1]: https://about.gitlab.com/blog/2018/10/15/gitlab-to-deprecate-older-tls/
|
|
We're currently using `NewClientTLSFromCert()`, which is inflexible and
doesn't allow us to tweak details of the resulting TLS transport
credentials. This commit converts them to use an equivalent invocation
of `NewTLS()` instead, which allows us to specify the `tls.Config`
directly.
No change in behaviour is expected from this commit.
|
|
As we are trying to remove dependencies on the global vars
this change removes usage of the config.Config from the
GitalyServerFactory and substitutes it with passed in config.Cfg
parameter. As GitalyServerFactory is used in different parts of the
service it requires changes in a couple of other places to pass the
parameter from top to bottom.
Part of: https://gitlab.com/gitlab-org/gitaly/-/issues/2699
|
|
As the strategy of interacting with the repository
changes there is no more need to provide RepositoryStore
into Mgr as a dependency as it has no usage.
This commit removes RepositoryStore from the list of the
input parameters of the constructor function and from the
list of fields of the Mgr struct.
|
|
This reverts commit 09c6d25de370446ac855a8241d8f821ed3f1ceec
|
|
gitaly node is a primary. For mutator operations it used to
define from what node the response will be returned back to
the client. For the read operations it is used to redirect request
to or as a fallback option for reads distribution in case it
is enabled. The default strategy for defining the primary is
an 'sql' which means the primary is tracked inside Postgres
database and praefect issues select statement into it each time
it needs to define current primary. It creates a high load
on the database when there are too many read operations (the
outcome of the performance testing).
To resolve this problem we change logic of retrieving set of
up to date storages to return all storages including primary.
So now we don't need to know the current primary and use any
storage that has latest generation of the repository to serve
the requests. As this information is cached by the in-memory
cache praefect won't create a high load on the database anymore.
This change also makes check IsLatestGeneration for the primary
node redundant as it won't be present in the set of consistent
storages if its generation not the latest one.
Closes: https://gitlab.com/gitlab-org/gitaly/-/issues/3337
|
|
This commit implements an RPC on Praefect's Info service to allow
setting a repository's replication factor.
|
|
To prepare for removing the MemoryRepositoryStore, this commit
removes its uses in tests. Mostly the tests need something that works,
which is when DisableRepositoryStore is used. When a test is testing
a particular scenario with the RepositoryStore, a mock is provided
instead. Ideally we'd use the Postgres implementation in these cases
but hooking it in requires some additional work as the test setup
overwrites home directory which breaks the discovery of GDK's Postgres.
|
|
In order to remove dependency on the global config.Config var we
need to break dependencies between Config and it's initialization
and validation functions. Most of them now represented as a methods
on the config.Cfg type. It allows us to setup required Cfg struct
for the tests to run without need to sync changes done on the
shared global Config var. The tests that were heavily dependent
on the Config var now depend on the struct initialized inside
the test instead.
Part of: https://gitlab.com/gitlab-org/gitaly/-/issues/2699
Blocks: https://gitlab.com/gitlab-org/gitaly/-/issues/3166
|
|
It is a next step in including cached storages provider in order to
support reads distribution across gitalies. On each invocation it
queries the passed in dependency and combine the result with existing
primary. The resulted list is used by the manager to decide where request
should be routed for processing. In a follow up MR it will be extended
with expiration cache to reduce load on database as accessing it on each
read operation is not efficient.
Part of: https://gitlab.com/gitlab-org/gitaly/-/issues/3053
|
|
Dependency inject connection pool in to the operations server to
share connections between multiple services.
|
|
Various tests are binding to other IP addresses than localhost.
Ideally the tests should not open public sockets during the test run.
Additionally, this causes macOS firewall to ask for permission on each
test run.
|
|
Routing logic is currently not pluggable as it is part of the coordinator
and depends on the NodeManager. In preparation for variable replication
factor and per repository primary, this commit extracts the routing logic
in to a separate component which allows for plugging in alternative
implementation. There should be no behavior difference, except for eagerly
loading consistent secondaries on repository scoped mutators regardless of
whether transactions are enabled or not. This should be fine though as
transactions are enabled by default.
|
|
Since the introduction of Praefect, our code layout started to become
confusing: while Praefect code lives in `internal/praefect`,
Gitaly-specific code is all over the place and not neatly singled out.
This makes it hard at times to tell apart Praefect- and Gitaly-specific
from generic code.
To improve the situation, this commit thus moves most of the server
specific code into a new `internal/gitaly` package. Currently, this is
the `internal/config`, `internal/server`, `internal/service` and
`internal/rubyserver` packages, which are all main components of Gitaly.
The move was realized with the following script:
#!/bin/sh
mkdir -p internal/gitaly
git mv internal/{config,server,service,rubyserver} internal/gitaly/
find . -name '*.go' -exec sed -i \
-e 's|gitlab-org/gitaly/internal/rubyserver|gitlab-org/gitaly/internal/gitaly/rubyserver|' \
-e 's|gitlab-org/gitaly/internal/server|gitlab-org/gitaly/internal/gitaly/server|' \
-e 's|gitlab-org/gitaly/internal/service|gitlab-org/gitaly/internal/gitaly/service|' \
-e 's|gitlab-org/gitaly/internal/config|gitlab-org/gitaly/internal/gitaly/config|' {} \;
In addition to that, some minor adjustments were needed for tests which
used relative paths.
|
|
Hooks up the error tracker in the node manager so it checks if a certain
backend node has reached a threshold of errors. If it has, then it will
be deemed unhealthy.
|
|
Right now, setup of metrics used in the transaction manager is split
across multiple locations. This makes the process of adding new metrics
more involved than it needs to be and is a source of bugs in case any of
those locations is not updated.
Improve the situation by moving setup of metrics into the transaction
manager. Metrics are exposed by implementing the Collector interface and
registering the transaction manager itself as a metric.
|
|
It is decided to go with all features enabled by default behaviour
for tests. This should help us identify potential problems with
feature flag combinations enabled in production. To verify implementation
without feature flag enabled it should be disabled explicitly in the
test.
Part of: https://gitlab.com/gitlab-org/gitaly/-/issues/2702
|
|
Retrieve up to date storages that can server read operation for the
repository in order to distribute reads across all healthy storages
of the virtual storage.
Closes: https://gitlab.com/gitlab-org/gitaly/-/issues/2944
|
|
Repository state tracking integration
Closes #2866
See merge request gitlab-org/gitaly!2379
|
|
|
|
Tracking the expected and the actual repository states within a virtual
storage is currently done by searching through the replication queue. This
requires many variables to be taken in to account such as timings between
different jobs and the job history of source nodes. To make the tracking
easier, this commit adds two tables to record the latest state of
repositories across the cluster:
1. `repositories` table contains the expected state of a repository
within a virtual storage.
2. `storage_repositories` table contains the state of the repository
on a given storage that is part of a virtual storage.
Cross-referencing `storage_repositories` with `repositories` makes it
straightforward to figure out repositories which are in the expected
state. If a repository on a storage is not in the expected state,
appropriate corrective actions can be scheduled by diffing the expected
record with the record of the stale storage.
Each repository has a generation number which increases monotonically
for each write. The generation number can be used to deduce whether
the repository has the latest changes or not. The generation number
guarantees the repository is at least on the generation stored but it
may also be on a later generation if an update was partially applied.
To prevent the generation number from referring to outdated data,
repository downgrades are rejected. Generation numbers get propagated
via replication jobs which again guarantee the repository will be at
least on the generation included in the job.
After the upgrade, there won't be any repositories in the tables and
there might be replication jobs which do not have a generation number.
To account for this, the downgrade protection is only applied to
repositories which have a stored generation number, ensuring existing
replication jobs during cluster upgrade are still accepted. As an
upgraded primary receives new writes, the repository entries will be
added to the tables and replication jobs with correct generation numbers
scheduled.
|
|
Praefect allowed to use TLS address to listen on in
combination with certificate and its key to be used for it.
'metadata' package refactored, so TLS address can be passed
into the Gitaly.
Documentation fixed and example file filled with proper vals.
Closes: https://gitlab.com/gitlab-org/gitaly/-/issues/1698
|
|
Introduces server factory for creating gRPC servers.
Praefect gRPC server created by separate function and
can be reused in tests to check routing.
Part of: https://gitlab.com/gitlab-org/gitaly/-/issues/1698
|
