diff options
Diffstat (limited to 'doc/administration/gitaly/praefect.md')
| -rw-r--r-- | doc/administration/gitaly/praefect.md | 171 |
1 files changed, 137 insertions, 34 deletions
diff --git a/doc/administration/gitaly/praefect.md b/doc/administration/gitaly/praefect.md index 9038675a28f..83c9aa3f013 100644 --- a/doc/administration/gitaly/praefect.md +++ b/doc/administration/gitaly/praefect.md @@ -26,34 +26,42 @@ For this document, the following network topology is assumed: graph TB GitLab --> Gitaly; GitLab --> Praefect; - Praefect --> Preafect-Git-1; - Praefect --> Preafect-Git-2; - Praefect --> Preafect-Git-3; + Praefect --> Praefect-Gitaly-1; + Praefect --> Praefect-Gitaly-2; + Praefect --> Praefect-Gitaly-3; ``` Where `GitLab` is the collection of clients that can request Git operations. -`Gitaly` is a Gitaly server before using Praefect. The Praefect node has two +`Gitaly` is a Gitaly server before using Praefect. The Praefect node has three storage nodes attached. Praefect itself doesn't store data, but connects to -three Gitaly nodes, `Praefect-Git-1`, `Praefect-Git-2`, and `Praefect-Git-3`. -There should be no knowledge other than with Praefect about the existence of -the `Praefect-Git-X` nodes. +three Gitaly nodes, `Praefect-Gitaly-1`, `Praefect-Gitaly-2`, and `Praefect-Gitaly-3`. -### Setup +Praefect may be enabled on its own node or can be run on the GitLab server. +In the example below we will use a separate server, but the optimal configuration +for Praefect is still being determined. -In this setup guide, the Gitaly node will be added first, then Praefect, and -lastly we update the GitLab configuration. +Praefect will handle all Gitaly RPC requests to its child nodes. However, the child nodes +will still need to communicate with the GitLab server via its internal API for authentication +purposes. -#### Gitaly +### Setup -In their own machine, configure the Gitaly server as described in the -[gitaly documentation](index.md#3-gitaly-server-configuration). +In this setup guide we will start by configuring Praefect, then its child +Gitaly nodes, and lastly the GitLab server configuration. #### Praefect -Next, Praefect has to be enabled on its own node. Disable all other services, -and add each Gitaly node that will be connected to Praefect. In the example below, -the Gitaly nodes are named `praefect-git-X`. Note that one node is designated as -primary, by setting the primary to `true`: +On the Praefect node we disable all other services, including Gitaly. We list each +Gitaly node that will be connected to Praefect under `praefect['storage_nodes']`. + +In the example below, the Gitaly nodes are named `praefect-gitaly-N`. Note that one +node is designated as primary by setting the primary to `true`. + +`praefect['auth_token']` is the token used to authenticate with the GitLab server, +just like `gitaly['auth_token']` is used for a standard Gitaly server. + +The `token` field under each storage listed in `praefect['storage_nodes']` is used +to authenticate each child Gitaly node with Praefect. ```ruby # /etc/gitlab/gitlab.rb @@ -67,38 +75,111 @@ unicorn['enable'] = false sidekiq['enable'] = false gitlab_workhorse['enable'] = false gitaly['enable'] = false +``` + +##### Set up Praefect and its Gitaly nodes + +In the example below, the Gitaly nodes are named `praefect-git-X`. Note that one node is designated as +primary, by setting the primary to `true`: + +```ruby +# /etc/gitlab/gitlab.rb + +# Prevent database connections during 'gitlab-ctl reconfigure' +gitlab_rails['rake_cache_clear'] = false +gitlab_rails['auto_migrate'] = false + +praefect['enable'] = true + +# Make Praefect accept connections on all network interfaces. You must use +# firewalls to restrict access to this address/port. +praefect['listen_addr'] = '0.0.0.0:2305' # virtual_storage_name must match the same storage name given to praefect in git_data_dirs praefect['virtual_storage_name'] = 'praefect' -praefect['auth_token'] = 'super_secret_abc' -praefect['enable'] = true -praefect['storage_nodes'] = [ - { - 'storage' => 'praefect-git-1', - 'address' => 'tcp://praefect-git-1.internal', - 'token' => 'token1', + +# Authentication token to ensure only authorized servers can communicate with +# Praefect server +praefect['auth_token'] = 'praefect-token' +praefect['storage_nodes'] = { + 'praefect-gitaly-1' => { + 'address' => 'tcp://praefect-git-1.internal:8075', + 'token' => 'praefect-gitaly-token', 'primary' => true }, - { - 'storage' => 'praefect-git-2', - 'address' => 'tcp://praefect-git-2.internal', - 'token' => 'token2' + 'praefect-gitaly-2' => { + 'address' => 'tcp://praefect-git-2.internal:8075', + 'token' => 'praefect-gitaly-token' }, - { - 'storage' => 'praefect-git-3', - 'address' => 'tcp://praefect-git-3.internal', - 'token' => 'token3' + 'praefect-gitaly-3' => { + 'address' => 'tcp://praefect-git-3.internal:8075', + 'token' => 'praefect-gitaly-token' } -] +} ``` Save the file and [reconfigure Praefect](../restart_gitlab.md#omnibus-gitlab-reconfigure). +#### Gitaly + +Next we will configure each Gitaly server assigned to Praefect. Configuration for these +is the same as a normal standalone Gitaly server, except that we use storage names and +auth tokens from Praefect instead of GitLab. + +Below is an example configuration for `praefect-gitaly-1`, the only difference for the +other Gitaly nodes is the storage name under `git_data_dirs`. + +Note that `gitaly['auth_token']` matches the `token` value listed under `praefect['storage_nodes']` +on the Praefect node. + +```ruby +# /etc/gitlab/gitlab.rb + +# Avoid running unnecessary services on the Gitaly server +postgresql['enable'] = false +redis['enable'] = false +nginx['enable'] = false +prometheus['enable'] = false +unicorn['enable'] = false +sidekiq['enable'] = false +gitlab_workhorse['enable'] = false + +# Prevent database connections during 'gitlab-ctl reconfigure' +gitlab_rails['rake_cache_clear'] = false +gitlab_rails['auto_migrate'] = false + +# Configure the gitlab-shell API callback URL. Without this, `git push` will +# fail. This can be your 'front door' GitLab URL or an internal load +# balancer. +# Don't forget to copy `/etc/gitlab/gitlab-secrets.json` from web server to Gitaly server. +gitlab_rails['internal_api_url'] = 'https://gitlab.example.com' + +# Authentication token to ensure only authorized servers can communicate with +# Gitaly server +gitaly['auth_token'] = 'praefect-gitaly-token' + +# Make Gitaly accept connections on all network interfaces. You must use +# firewalls to restrict access to this address/port. +# Comment out following line if you only want to support TLS connections +gitaly['listen_addr'] = "0.0.0.0:8075" + +git_data_dirs({ + "praefect-gitaly-1" => { + "path" => "/var/opt/gitlab/git-data" + } +}) +``` + +Note that just as with a standard Gitaly server, `/etc/gitlab/gitlab-secrets.json` must +be copied from the GitLab server to the Gitaly node for authentication purposes. + +For more information on Gitaly server configuration, see our [gitaly documentation](index.md#3-gitaly-server-configuration). + #### GitLab When Praefect is running, it should be exposed as a storage to GitLab. This is done through setting the `git_data_dirs`. Assuming the default storage -configuration is used, there would be two storages available to GitLab: +is present, there should be two storages available to GitLab: ```ruby git_data_dirs({ @@ -109,6 +190,28 @@ git_data_dirs({ "gitaly_address" => "tcp://praefect.internal:2305" } }) + +gitlab_rails['gitaly_token'] = 'praefect-token' ``` +Note that the storage name used is the same as the `praefect['virtual_storage_name']` set +on the Praefect node. + +Also, the `gitlab_rails['gitaly_token']` matches the value of `praefect['auth_token']` +on Praefect. + Restart GitLab using `gitlab-ctl restart` on the GitLab node. + +### Testing Praefect + +To test Praefect, first set it as the default storage node for new projects +using **Admin Area > Settings > Repository > Repository storage**. Next, +create a new project and check the "Initialize repository with a README" box. + +If you receive a 503 error, check `/var/log/gitlab/gitlab-rails/production.log`. +A `GRPC::Unavailable (14:failed to connect to all addresses)` error indicates +that GitLab was unable to connect to Praefect. + +If the project is created but the README is not, then ensure that the +`/etc/gitlab/gitlab-secrets.json` file from the GitLab server has been copied +to the Gitaly servers. |