| Age | Commit message (Collapse) | Author |
|---|
|
We're about to introduce a second playbook for cluster creation, so
let's rename the generic "playbook.yml" to "configure.yml".
|
|
Both `destroy-demo-cluster` and `print-info` are simple wrappers around
terraform. Using Ruby for that is overkill.
Let's convert both scripts to shell onliners.
|
|
Setup the omnibus gitaly as a storage "internal"
|
|
With PATH now containing a broken version of Git by design, we need to
adjust all callers of Git to use the real executable instead of the
broken one. The script `check-mod-tidy` is one of those which needs
adjusting.
Given it's such a tiny script, let's just inline it into our Makefile
and have it use the `${GIT}` variable.
|
|
Starting with commit 36437f18 (PgBouncer deployment with terraform,
2020-07-29), we've introduced a new dependency on the pgbouncer module
for Terraform. To download the module, the commit in question added a
call to `terraform get`, which causes us to download the module. But as
we're only executing `terraform init` in case no `.terraform` directory
exists and `terraform get` will unconditionally create that directory,
the result is that `terraform init` will never be executed.
Fix the issue by just removing the call to `terraform get`.
Initialization will fetch the pgbouncer module anyway, so there's no
need to do it explicitly.
|
|
Right now, our Terraform scripts only handle creation of machines in
GCP, but not their respective configuration. As it's a task we're doing
rather frequently which takes some time, this commit creates an Ansible
playbook which automates this task.
On creation of the cluster, our scripts now automatically generate a
`hosts.ini` file containing all necessary connection information. With
this file, the admin may now run `./configure-demo-cluster`, which will
invoke Ansible and automatically generate and apply configuration for
each of the nodes, restarting services as required. The task can be run
repeatedly in order to update configuration on the target nodes.
|
|
PgBouncer deployment with terraform
Closes #2975
See merge request gitlab-org/gitaly!2418
|
|
Resulting name of the IP for the SQL connection renamed
to praefect_pgbouncer_ip.
Closes: https://gitlab.com/gitlab-org/gitaly/-/issues/2975
|
|
Assignment of the dedicated IP address to PgBouncer instance
in order to narrow the set of public IP addresses allowed to
connect to PostgreSQL instance.
Closes: https://gitlab.com/gitlab-org/gitaly/-/issues/2975
|
|
Terraform requires a new module to be installed: pgbouncer
In order to install all required modules automatically
the `terraform get` command added to main script.
The command installs all missed dependencies recursively.
Closes: https://gitlab.com/gitlab-org/gitaly/-/issues/2975
|
|
As the target PostgeSQL version currently is 11 we should
use this version for tests and demo.
|
|
In order to verify usage of PgBouncer in front of Postgres
database PgBouncer included into terraform deployment.
It uses separate machine with internal IP that is accessible
to Praefect instances.
Cloud SQL authorized networks changed to '0.0.0.0/0' because
it is not possible to use PgBouncer IP for it, as PgBouncer
requires IP of Cloud SQL instance in setup (circular dependency).
The output of 'praefect_postgresql_ip' is a private IP of the
PgBouncer instance that should be used instead of a public Cloud
SQL instance to proxy SQL requests.
Closes: https://gitlab.com/gitlab-org/gitaly/-/issues/2975
|
|
|
|
|
|
We currently use templating to generate the Makefile, which causes us to
split up build information across a set of multiple different files. The
initial motivation for this was two-fold, where one reason was to make
the Makefile more predictable and the second was performance. The
downside of generating Makefiles is added complexity, though, and that
reading Makefiles got kind of awkward because of huge generated lines.
This commit restructures our Makefile in order to get rid of the
templating scheme in order to unify build information into a single
place again. To make them predictable, the new Makefile makes strong use
of variables that hold various information about paths. Furthermore, to
not pay a performance penalty, the dynamic parts of discovering
packages, source files and commands was moved into callable functions
that only get executed when the current target actually requires that
kind of information.
The resulting Makefile seems easier to manage in the long term and
doesn't seem to perform any worse than the generated Makefile. It'll
also make future refactoring and improvements a lot easier to implement.
|
|
|
|
|
|
Fix build-git target on musl-based systems
See merge request gitlab-org/gitaly!2171
|
|
- update Docker base image
- update CI testing matrix
Part of: https://gitlab.com/gitlab-org/gitaly/-/issues/2829
|
|
|
|
|
|
Support bindir in Makefile
Closes gitlab#213467
See merge request gitlab-org/gitaly!2167
|
|
|
|
Election of the new primary node must take into account
relation of the node to the proper virtual storage.
To speed up SQL operations index were introduced based on
'virtual_storage' and 'target_node_storage' values.
As an addition down migration was fixed to remove new type.
|
|
By default, Git will assume that the regex implementation of libc
support the `REG_STARTEND` feature. As this feature started out as a
non-standard glibc extension, it's not available to all systems. Most
notably, musl libc doesn't implement it as it tries to be as strictly
POSIX compliant as it makes sense.
Fix the issue by amending NO_REGEX=YesPlease to Git's build options.
This will cause it to use its own embedded regex implementation which
obviously supports this feature.
|
|
We currently hardcode the number of jobs to "8", which may be optimal on
some platforms, but definitely not on most. Some systems with a smaller
amount of CPU cores may thus be overloaded while others with a lot more
cores will be underloaded.
Let's improve this by using the number of cores available to the system,
plus one.
|
|
The `grpc_tools` gem was installed using a `gem` invocation, which lead
to different versions of the same gem being used among developers. By
using the Gemfile for version management this should be resolved. The
gem is added as development dependency, and as such will not make its
way to production.
To support using bundler, `bundle exec` had to be used in the support
script, which required the execution context to be changed to the ruby
sub-directory. `Dir.chdir` is not thread safe, but currently I think
it's fair to assume this is not required.
|
|
git: Default to latest supported version to build
See merge request gitlab-org/gitaly!2131
|
|
Add load balancer and multiple Praefect nodes
Closes #2639
See merge request gitlab-org/gitaly!2098
|
|
Gitaly's Makefile can build Git, and currently defaults to master. For
local development this is inconvenient given there might be differences
which could lead to issues which are hard to debug.
The downside of this change, is that it locks the version, and requires
the Gitaly team to bump this version along with other Git version
upgrades.
|
|
Add build-git target to Makefile
See merge request gitlab-org/gitaly!1532
|
|
The new 'download-git' target can be used like this:
$ make GIT_REV=test-next download-git
to download a tarball from:
https://gitlab.com/gitlab-org/gitlab-git/-/jobs/artifacts/test-next/raw/git_full_bins.tgz?job=build
|
|
In 86e90ec2711025a69ab3bb696783de324fdf9b51, support was added for PG
provisioning through terraform. The readme did still include the manual
steps however.
|
|
JUnit reports should be captured on test failures
See merge request gitlab-org/gitaly!2087
|
|
Praefect now stores state in SQL so that multiple nodes can be run
behind a load balancer. This updates the Terraform configuration to
reflect this.
|
|
The update to gRPC v1.27.0 introduced issues with various setups, most
notably with CentOS 6 due to an old compiler toolchain and musl-based
Linux distributions like Alpine Linux because of non-POSIX-compliant use
of symbols. As a result, the equivalent update in the GitLab repository
was reverted already. Let's do the same for Gitaly to fix these issues.
This reverts commit 998db420d6b68b81e1ac9974854a700e7985db91, reversing
changes made to 9f35085fd2f69bf0a892746037da7e2c9511c1f6.
|
|
|
|
Add support for multiple Praefects in Terraform config
See merge request gitlab-org/gitaly!1993
|
|
This commit adds the following strategy to enable redundant Praefect
nodes to run simultaenously:
1. Every Praefect node periodically (every second) performs a health
check RPC with a Gitaly node.
2. For each node, Praefect updates a row in a new table (`node_status`)
with the following information:
a. The name of the Praefect instance (`praefect_name`)
b. The name of the virtual storage name (`shard_name`)
c. The name of the Gitaly storage name (`storage_name`)
d. The timestamp of the last time Praefect tried to reach that node
(`last_contact_attempt_at`)
e. The timestamp of the last successful health check (`last_seen_active_at`)
3. Periodically every Praefect node does a `SELECT` from `node_status`
to determine **healthy nodes**. A healthy node is defined by:
a. A node that has a recent successful error check (e.g. one in the last 10 s).
b. A majority of the available Praefect nodes have entries that match the two above.
4. To determine the majority, we use a lightweight service discovery
protocol: a Praefect node is deemed a voting member if the
`praefect_name` has a recent `last_contact_attempt_at` in the
`node_status` table. The name is derived from a combination of the
hostname and listening port/socket.
5. The primary of each shard is listed in the `shard_primaries`. If the
current primary is in the healthy node list, then no election needs to
be done.
6. Otherwise, if there is no primary or it is unhealthy, any Praefect
node can elect a new primary by choosing candidate from the healthy node
list and inserting a row into the table.
Closes https://gitlab.com/gitlab-org/gitaly/-/issues/2547
|
|
This will help test multiple Praefect nodes to test SQL leader
elections. Currently `count` is set to 1; it can be increased
to 3 or more.
|
|
lint (static code analysis) enhancements
See merge request gitlab-org/gitaly!2033
|
|
lint must use Go1.13 as it is highest supported version.
Fix deprecated use of `grpc_logrus.Extract`.
New linters introduced and replacement of old ones in favor
of new ones with same purpose.
Fix of new linter errors.
|
|
- update Docker base image
- update CI testing matrix
Part of: https://gitlab.com/gitlab-org/gitaly/issues/2497
|
|
In 5109ed2d, pipefail was turned on in order to produce a correct
exit code on test failure. This caused the JUnit reporter not
to run on test failures. This commit turns off pipefail, ensuring
the JUnit reporter runs even after tests fail and returns the
exit code of 'go test'.
|
|
|
|
Create Gitaly HA PostgreSQL instance with Terraform
See merge request gitlab-org/gitaly!1990
|
|
Currently the test output is only written in to the junit report.
This commit modifies the makefile to tee the output also to console.
|
|
Add stackdriver profiler support with LabKit
See merge request gitlab-org/gitaly!1842
|
|
|
|
Terraform: deduplicate gitaly instance definition
See merge request gitlab-org/gitaly!2002
|
