diff options
Diffstat (limited to 'doc/administration/gitaly')
-rw-r--r-- | doc/administration/gitaly/index.md | 458 | ||||
-rw-r--r-- | doc/administration/gitaly/praefect.md | 72 | ||||
-rw-r--r-- | doc/administration/gitaly/reference.md | 9 |
3 files changed, 287 insertions, 252 deletions
diff --git a/doc/administration/gitaly/index.md b/doc/administration/gitaly/index.md index 14b0a6bd450..1469ed64004 100644 --- a/doc/administration/gitaly/index.md +++ b/doc/administration/gitaly/index.md @@ -1,14 +1,25 @@ +--- +stage: Create +group: Gitaly +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +type: reference +--- + # Gitaly -[Gitaly](https://gitlab.com/gitlab-org/gitaly) is the service that -provides high-level RPC access to Git repositories. Without it, no other -components can read or write Git data. GitLab components that access Git -repositories (GitLab Rails, GitLab Shell, GitLab Workhorse, etc.) act as clients -to Gitaly. End users do not have direct access to Gitaly. +[Gitaly](https://gitlab.com/gitlab-org/gitaly) is the service that provides high-level RPC access to +Git repositories. Without it, no GitLab components can read or write Git data. + +In the Gitaly documentation: -On this page, *Gitaly server* refers to a standalone node that only runs Gitaly -and *Gitaly client* is a GitLab Rails app node that runs all other processes -except Gitaly. +- **Gitaly server** refers to any node that runs Gitaly itself. +- **Gitaly client** refers to any node that runs a process that makes requests of the + Gitaly server. Processes include, but are not limited to: + - [GitLab Rails application](https://gitlab.com/gitlab-org/gitlab). + - [GitLab Shell](https://gitlab.com/gitlab-org/gitlab-shell). + - [GitLab Workhorse](https://gitlab.com/gitlab-org/gitlab-workhorse). + +GitLab end users do not have direct access to Gitaly. CAUTION: **Caution:** From GitLab 13.0, using NFS for Git repositories is deprecated. In GitLab 14.0, @@ -17,19 +28,20 @@ support for NFS for Git repositories is scheduled to be removed. Upgrade to ## Architecture -Here's a high-level architecture overview of how Gitaly is used. +The following is a high-level architecture overview of how Gitaly is used. ![Gitaly architecture diagram](img/architecture_v12_4.png) -## Configuring Gitaly +## Configure Gitaly The Gitaly service itself is configured via a [TOML configuration file](reference.md). -If you want to change any of its settings: +To change Gitaly settings: **For Omnibus GitLab** -1. Edit `/etc/gitlab/gitlab.rb` and add or change the [Gitaly settings](https://gitlab.com/gitlab-org/omnibus-gitlab/blob/1dd07197c7e5ae23626aad5a4a070a800b670380/files/gitlab-config-template/gitlab.rb.template#L1622-1676). +1. Edit `/etc/gitlab/gitlab.rb` and add or change the + [Gitaly settings](https://gitlab.com/gitlab-org/omnibus-gitlab/blob/1dd07197c7e5ae23626aad5a4a070a800b670380/files/gitlab-config-template/gitlab.rb.template#L1622-1676). 1. Save the file and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure). **For installations from source** @@ -37,92 +49,121 @@ If you want to change any of its settings: 1. Edit `/home/git/gitaly/config.toml` and add or change the [Gitaly settings](https://gitlab.com/gitlab-org/gitaly/blob/master/config.toml.example). 1. Save the file and [restart GitLab](../restart_gitlab.md#installations-from-source). -## Running Gitaly on its own server +## Run Gitaly on its own server + +By default, Gitaly is run on the same server as Gitaly clients and is +[configured as above](#configure-gitaly). Single-server installations are best served by +this default configuration used by: -This is an optional way to deploy Gitaly which can benefit GitLab -installations that are larger than a single machine. Most -installations will be better served with the default configuration -used by Omnibus and the GitLab source installation guide. -Follow transition to Gitaly on its own server, [Gitaly servers will need to be upgraded before other servers in your cluster](https://docs.gitlab.com/omnibus/update/#upgrading-gitaly-servers). +- [Omnibus GitLab](https://docs.gitlab.com/omnibus/). +- The GitLab [source installation guide](../../install/installation.md). -Starting with GitLab 11.4, Gitaly is able to serve all Git requests without -requiring a shared NFS mount for Git repository data. -Between 11.4 and 11.8 the exception was the -[Elasticsearch indexer](https://gitlab.com/gitlab-org/gitlab-elasticsearch-indexer). -But since 11.8 the indexer uses Gitaly for data access as well. NFS can still -be leveraged for redundancy on block level of the Git data. But only has to -be mounted on the Gitaly server. +However, Gitaly can be deployed to its own server, which can benefit GitLab installations that span +multiple machines. -From GitLab v11.8 to v12.2, it is possible to use Elasticsearch in conjunction with -a Gitaly setup that isn't utilising NFS. In order to use Elasticsearch in this -scenario, the [new repository indexer](../../integration/elasticsearch.md#elasticsearch-repository-indexer) -needs to be enabled in your GitLab configuration. [Since GitLab v12.3](https://gitlab.com/gitlab-org/gitlab/issues/6481), -the new indexer becomes the default and no configuration is required. +NOTE: **Note:** +When configured to run on their own servers, Gitaly servers +[must be upgraded](https://docs.gitlab.com/omnibus/update/#upgrading-gitaly-servers) before Gitaly +clients in your cluster. + +The process for setting up Gitaly on its own server is: + +1. [Install Gitaly](#install-gitaly). +1. [Configure authentication](#configure-authentication). +1. [Configure Gitaly servers](#configure-gitaly-servers). +1. [Configure Gitaly clients](#configure-gitaly-clients). + +When running Gitaly on its own server, note the following regarding GitLab versions: + +- From GitLab 11.4, Gitaly was able to serve all Git requests without requiring a shared NFS mount + for Git repository data, except for the + [Elasticsearch indexer](https://gitlab.com/gitlab-org/gitlab-elasticsearch-indexer). +- From GitLab 11.8, the Elasticsearch indexer uses Gitaly for data access as well. NFS can still be + leveraged for redundancy on block-level Git data, but only has to be mounted on the Gitaly + servers. +- From GitLab 11.8 to 12.2, it is possible to use Elasticsearch in a Gitaly setup that doesn't use + NFS. In order to use Elasticsearch in these versions, the + [repository indexer](../../integration/elasticsearch.md#elasticsearch-repository-indexer) + must be enabled in your GitLab configuration. +- [Since GitLab 12.3](https://gitlab.com/gitlab-org/gitlab/-/issues/6481), the new indexer is + the default and no configuration is required. ### Network architecture -The following list depicts what the network architecture of Gitaly is: +The following list depicts the network architecture of Gitaly: - GitLab Rails shards repositories into [repository storages](../repository_storage_paths.md). -- `/config/gitlab.yml` contains a map from storage names to - `(Gitaly address, Gitaly token)` pairs. -- the `storage name` -\> `(Gitaly address, Gitaly token)` map in - `/config/gitlab.yml` is the single source of truth for the Gitaly network - topology. +- `/config/gitlab.yml` contains a map from storage names to `(Gitaly address, Gitaly token)` pairs. +- The `storage name` -\> `(Gitaly address, Gitaly token)` map in `/config/gitlab.yml` is the single + source of truth for the Gitaly network topology. - A `(Gitaly address, Gitaly token)` corresponds to a Gitaly server. - A Gitaly server hosts one or more storages. -- A GitLab server can use one or more Gitaly servers. -- Gitaly addresses must be specified in such a way that they resolve - correctly for ALL Gitaly clients. -- Gitaly clients are: Puma/Unicorn, Sidekiq, GitLab Workhorse, - GitLab Shell, Elasticsearch Indexer, and Gitaly itself. +- A Gitaly client can use one or more Gitaly servers. +- Gitaly addresses must be specified in such a way that they resolve correctly for **all** Gitaly + clients. +- Gitaly clients are: + - Puma or Unicorn. + - Sidekiq. + - GitLab Workhorse. + - GitLab Shell. + - Elasticsearch indexer. + - Gitaly itself. - A Gitaly server must be able to make RPC calls **to itself** via its own `(Gitaly address, Gitaly token)` pair as specified in `/config/gitlab.yml`. -- Gitaly servers must not be exposed to the public internet as Gitaly's network - traffic is unencrypted by default. The use of firewall is highly recommended - to restrict access to the Gitaly server. Another option is to - [use TLS](#tls-support). -- Authentication is done through a static token which is shared among the Gitaly - and GitLab Rails nodes. +- Authentication is done through a static token which is shared among the Gitaly and GitLab Rails + nodes. + +DANGER: **Danger:** +Gitaly servers must not be exposed to the public internet as Gitaly's network traffic is unencrypted +by default. The use of firewall is highly recommended to restrict access to the Gitaly server. +Another option is to [use TLS](#tls-support). + +In the following sections, we describe how to configure two Gitaly servers with secret token +`abc123secret`: -Below we describe how to configure two Gitaly servers one at -`gitaly1.internal` and the other at `gitaly2.internal` -with secret token `abc123secret`. We assume -your GitLab installation has three repository storages: `default`, -`storage1` and `storage2`. You can use as little as just one server with one -repository storage if desired. +- `gitaly1.internal`. +- `gitaly2.internal`. -Note: **Note:** The token referred to throughout the Gitaly documentation is -just an arbitrary password selected by the administrator. It is unrelated to -tokens created for the GitLab API or other similar web API tokens. +We assume your GitLab installation has three repository storages: -### 1. Installation +- `default`. +- `storage1`. +- `storage2`. -First install Gitaly on each Gitaly server using either -Omnibus GitLab or install it from source: +You can use as few as one server with one repository storage if desired. -- For Omnibus GitLab: [Download/install](https://about.gitlab.com/install/) the Omnibus GitLab - package you want using **steps 1 and 2** from the GitLab downloads page but - **_do not_** provide the `EXTERNAL_URL=` value. -- From source: [Install Gitaly](../../install/installation.md#install-gitaly). +NOTE: **Note:** +The token referred to throughout the Gitaly documentation is just an arbitrary password selected by +the administrator. It is unrelated to tokens created for the GitLab API or other similar web API +tokens. + +### Install Gitaly + +Install Gitaly on each Gitaly server using either Omnibus GitLab or install it from source: + +- For Omnibus GitLab, [download and install](https://about.gitlab.com/install/) the Omnibus GitLab + package you want but **do not** provide the `EXTERNAL_URL=` value. +- To install from source, follow the steps at + [Install Gitaly](../../install/installation.md#install-gitaly). -### 2. Authentication +### Configure authentication -Gitaly and GitLab use two shared secrets for authentication, one to authenticate gRPC requests -to Gitaly, and a second for authentication callbacks from GitLab-Shell to the GitLab internal API. +Gitaly and GitLab use two shared secrets for authentication: + +- One to authenticate gRPC requests to Gitaly. +- A second for authentication callbacks from GitLab Shell to the GitLab internal API. **For Omnibus GitLab** To configure the Gitaly token: -1. On the client server, edit `/etc/gitlab/gitlab.rb`: +1. On the Gitaly clients, edit `/etc/gitlab/gitlab.rb`: ```ruby gitlab_rails['gitaly_token'] = 'abc123secret' ``` 1. Save the file and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure). - 1. On the Gitaly server, edit `/etc/gitlab/gitlab.rb`: ```ruby @@ -131,22 +172,24 @@ To configure the Gitaly token: 1. [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure). -There are two ways to configure the GitLab-Shell token: +There are two ways to configure the GitLab Shell token. -1. Copy `/etc/gitlab/gitlab-secrets.json` from the client server to same path on the Gitaly server. -1. [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure). +Method 1: + +1. Copy `/etc/gitlab/gitlab-secrets.json` from the Gitaly client to same path on the Gitaly servers + (and any other Gitaly clients). +1. [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) on Gitaly servers. -**OR** +Method 2: -1. On the client server, edit `/etc/gitlab/gitlab.rb`: +1. On the Gitaly clients, edit `/etc/gitlab/gitlab.rb`: ```ruby gitlab_shell['secret_token'] = 'shellsecret' ``` 1. Save the file and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure). - -1. On the Gitaly server, edit `/etc/gitlab/gitlab.rb`: +1. On the Gitaly servers, edit `/etc/gitlab/gitlab.rb`: ```ruby gitlab_shell['secret_token'] = 'shellsecret' @@ -156,9 +199,9 @@ There are two ways to configure the GitLab-Shell token: **For installations from source** -1. Copy `/home/git/gitlab/.gitlab_shell_secret` from the client server to the same path on the Gitaly -server. -1. On the client server, edit `/home/git/gitlab/config/gitlab.yml`: +1. Copy `/home/git/gitlab/.gitlab_shell_secret` from the Gitaly client to the same path on the + Gitaly servers (and any other Gitaly clients). +1. On the Gitaly clients, edit `/home/git/gitlab/config/gitlab.yml`: ```yaml gitlab: @@ -167,17 +210,22 @@ server. ``` 1. Save the file and [restart GitLab](../restart_gitlab.md#installations-from-source). +1. On the Gitaly servers, edit `/home/git/gitaly/config.toml`: + + ```toml + [auth] + token = 'abc123secret' + ``` -### 3. Gitaly server configuration +1. Save the file and [restart GitLab](../restart_gitlab.md#installations-from-source). -Next, on the Gitaly servers, you need to configure storage paths, enable -the network listener and configure the token. +### Configure Gitaly servers -NOTE: **Note:** If you want to reduce the risk of downtime when you enable -authentication you can temporarily disable enforcement, see [the -documentation on configuring Gitaly -authentication](https://gitlab.com/gitlab-org/gitaly/blob/master/doc/configuration/README.md#authentication) -. +On the Gitaly servers, you must configure storage paths and enable the network listener. + +If you want to reduce the risk of downtime when you enable authentication, you can temporarily +disable enforcement. For more information, see the documentation on configuring +[Gitaly authentication](https://gitlab.com/gitlab-org/gitaly/blob/master/doc/configuration/README.md#authentication). **For Omnibus GitLab** @@ -199,6 +247,7 @@ authentication](https://gitlab.com/gitlab-org/gitaly/blob/master/doc/configurati sidekiq['enable'] = false gitlab_workhorse['enable'] = false grafana['enable'] = false + gitlab_exporter['enable'] = false # If you run a separate monitoring node you can disable these services alertmanager['enable'] = false @@ -211,7 +260,6 @@ authentication](https://gitlab.com/gitlab-org/gitaly/blob/master/doc/configurati # prometheus['monitor_kubernetes'] = false # If you don't want to run monitoring services uncomment the following (not recommended) - # gitlab_exporter['enable'] = false # node_exporter['enable'] = false # Prevent database connections during 'gitlab-ctl reconfigure' @@ -221,7 +269,7 @@ authentication](https://gitlab.com/gitlab-org/gitaly/blob/master/doc/configurati # Configure the gitlab-shell API callback URL. Without this, `git push` will # fail. This can be your 'front door' GitLab URL or an internal load # balancer. - # Don't forget to copy `/etc/gitlab/gitlab-secrets.json` from web server to Gitaly server. + # Don't forget to copy `/etc/gitlab/gitlab-secrets.json` from Gitaly client to Gitaly server. gitlab_rails['internal_api_url'] = 'https://gitlab.example.com' # Make Gitaly accept connections on all network interfaces. You must use @@ -230,7 +278,7 @@ authentication](https://gitlab.com/gitlab-org/gitaly/blob/master/doc/configurati gitaly['listen_addr'] = "0.0.0.0:8075" ``` -1. Append the following to `/etc/gitlab/gitlab.rb` for each respective server: +1. Append the following to `/etc/gitlab/gitlab.rb` for each respective Gitaly server: <!-- updates to following example must also be made at @@ -262,27 +310,24 @@ authentication](https://gitlab.com/gitlab-org/gitaly/blob/master/doc/configurati 1. Save the file and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure). 1. Run `sudo /opt/gitlab/embedded/service/gitlab-shell/bin/check -config /opt/gitlab/embedded/service/gitlab-shell/config.yml` -to confirm that Gitaly can perform callbacks to the internal API. + to confirm that Gitaly can perform callbacks to the GitLab internal API. **For installations from source** -1. On the client node(s), edit `/home/git/gitaly/config.toml`: +1. Edit `/home/git/gitaly/config.toml`: ```toml listen_addr = '0.0.0.0:8075' internal_socket_dir = '/var/opt/gitlab/gitaly' - [auth] - token = 'abc123secret' - [logging] format = 'json' level = 'info' dir = '/var/log/gitaly' ``` -1. Append the following to `/home/git/gitaly/config.toml` for each respective server: +1. Append the following to `/home/git/gitaly/config.toml` for each respective Gitaly server: On `gitaly1.internal`: @@ -304,34 +349,51 @@ to confirm that Gitaly can perform callbacks to the internal API. path = '/srv/gitlab/git-data/repositories' ``` -1. On each Gitaly server, edit `/home/git/gitlab-shell/config.yml`: +1. Edit `/home/git/gitlab-shell/config.yml`: ```yaml gitlab_url: https://gitlab.example.com ``` -1. Save the file and [restart GitLab](../restart_gitlab.md#installations-from-source). +1. Save the files and [restart GitLab](../restart_gitlab.md#installations-from-source). 1. Run `sudo -u git /home/git/gitlab-shell/bin/check -config /home/git/gitlab-shell/config.yml` -to confirm that Gitaly can perform callbacks to the internal API. + to confirm that Gitaly can perform callbacks to the GitLab internal API. + +### Configure Gitaly clients -### 4. Converting clients to use the Gitaly server +As the final step, you must update Gitaly clients to switch from using local Gitaly service to use +the Gitaly servers you just configured. -As the final step, you need to update the client machines to switch from using -their local Gitaly service to the new Gitaly server you just configured. This -is a risky step because if there is any sort of network, firewall, or name -resolution problem preventing your GitLab server from reaching the Gitaly server, -then all Gitaly requests will fail. +This can be risky because anything that prevents your Gitaly clients from reaching the Gitaly +servers will cause all Gitaly requests to fail. For example, any sort of network, firewall, or name +resolution problems. -Additionally, you need to -[disable Rugged if previously manually enabled](../high_availability/nfs.md#improving-nfs-performance-with-gitlab). +Additionally, you must [disable Rugged](../high_availability/nfs.md#improving-nfs-performance-with-gitlab) +if previously enabled manually. -We assume that your `gitaly1.internal` Gitaly server can be reached at -`gitaly1.internal:8075` from your GitLab server, and that Gitaly server -can read and write to `/mnt/gitlab/default` and `/mnt/gitlab/storage1`. +Gitaly makes the following assumptions: -We assume also that your `gitaly2.internal` Gitaly server can be reached at -`gitaly2.internal:8075` from your GitLab server, and that Gitaly server -can read and write to `/mnt/gitlab/storage2`. +- Your `gitaly1.internal` Gitaly server can be reached at `gitaly1.internal:8075` from your Gitaly + clients, and that Gitaly server can read and write to `/mnt/gitlab/default` and + `/mnt/gitlab/storage1`. +- Your `gitaly2.internal` Gitaly server can be reached at `gitaly2.internal:8075` from your Gitaly + clients, and that Gitaly server can read and write to `/mnt/gitlab/storage2`. +- Your `gitaly1.internal` and `gitaly2.internal` Gitaly servers can reach each other. + +Note you can't a use mixed setup, with at least one of your Gitaly servers configured as a local +server with the `path` setting provided. This is because other Gitaly instances can't communicate +with it. The following setup is _incorrect_, because: + +- You must replace `path` with `gitaly_address` containing a proper value. +- The address must be reachable from the other two addresses provided. + +```ruby +git_data_dirs({ + 'default' => { 'gitaly_address' => 'tcp://gitaly1.internal:8075' }, + 'storage1' => { 'path' => '/var/opt/gitlab/git-data' }, + 'storage2' => { 'gitaly_address' => 'tcp://gitaly2.internal:8075' }, +}) +``` **For Omnibus GitLab** @@ -346,7 +408,8 @@ can read and write to `/mnt/gitlab/storage2`. ``` 1. Save the file and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure). -1. Run `sudo gitlab-rake gitlab:gitaly:check` to confirm the client can connect to Gitaly. +1. Run `sudo gitlab-rake gitlab:gitaly:check` to confirm the Gitaly client can connect to Gitaly + servers. 1. Tail the logs to see the requests: ```shell @@ -373,39 +436,38 @@ can read and write to `/mnt/gitlab/storage2`. ``` NOTE: **Note:** - `/some/dummy/path` should be set to a local folder that exists, however no - data will be stored in this folder. This will no longer be necessary after - [this issue](https://gitlab.com/gitlab-org/gitaly/issues/1282) is resolved. + `/some/dummy/path` should be set to a local folder that exists, however no data will be stored in + this folder. This will no longer be necessary after + [this issue](https://gitlab.com/gitlab-org/gitaly/-/issues/1282) is resolved. 1. Save the file and [restart GitLab](../restart_gitlab.md#installations-from-source). -1. Run `sudo -u git -H bundle exec rake gitlab:gitaly:check RAILS_ENV=production` to -confirm the client can connect to Gitaly. +1. Run `sudo -u git -H bundle exec rake gitlab:gitaly:check RAILS_ENV=production` to confirm the + Gitaly client can connect to Gitaly servers. 1. Tail the logs to see the requests: ```shell tail -f /home/git/gitlab/log/gitaly.log ``` -When you tail the Gitaly logs on your Gitaly server you should see requests -coming in. One sure way to trigger a Gitaly request is to clone a repository -from your GitLab server over HTTP. +When you tail the Gitaly logs on your Gitaly server, you should see requests coming in. One sure way +to trigger a Gitaly request is to clone a repository from GitLab over HTTP or HTTPS. DANGER: **Danger:** -If you have [Server hooks](../server_hooks.md) configured, -either per repository or globally, you must move these to the Gitaly node. -If you have multiple Gitaly nodes, copy your server hook(s) to all nodes. +If you have [server hooks](../server_hooks.md) configured, either per repository or globally, you +must move these to the Gitaly servers. If you have multiple Gitaly servers, copy your server hooks +to all Gitaly servers. ### Disabling the Gitaly service in a cluster environment If you are running Gitaly [as a remote -service](#running-gitaly-on-its-own-server) you may want to disable +service](#run-gitaly-on-its-own-server) you may want to disable the local Gitaly service that runs on your GitLab server by default. Disabling Gitaly only makes sense when you run GitLab in a custom cluster configuration, where different services run on different machines. Disabling Gitaly on all machines in the cluster is not a valid configuration. -To disable Gitaly on a client node: +To disable Gitaly on a GitLab server: **For Omnibus GitLab** @@ -436,13 +498,16 @@ with a Gitaly instance that listens for secure connections you will need to use scheme in the `gitaly_address` of the corresponding storage entry in the GitLab configuration. You will need to bring your own certificates as this isn't provided automatically. -The certificate, or its certificate authority, must be installed on all Gitaly -nodes (including the Gitaly node using the certificate) and on all client nodes +The certificate corresponding to each Gitaly server will need to be installed +on that Gitaly server. + +Additionally the certificate, or its certificate authority, must be installed on all Gitaly servers +(including the Gitaly server using the certificate) and on all Gitaly clients that communicate with it following the procedure described in -[GitLab custom certificate configuration](https://docs.gitlab.com/omnibus/settings/ssl.html#install-custom-public-certificates). +[GitLab custom certificate configuration](https://docs.gitlab.com/omnibus/settings/ssl.html#install-custom-public-certificates) (and repeated below). NOTE: **Note** -The self-signed certificate must specify the address you use to access the +The certificate must specify the address you use to access the Gitaly server. If you are addressing the Gitaly server by a hostname, you can either use the Common Name field for this, or add it as a Subject Alternative Name. If you are addressing the Gitaly server by its IP address, you must add it @@ -459,7 +524,14 @@ To configure Gitaly with TLS: **For Omnibus GitLab** -1. On the client node(s), edit `/etc/gitlab/gitlab.rb` as follows: +1. Create certificates for Gitaly servers. +1. On the Gitaly clients, copy the certificates, or their certificate authority, into the `/etc/gitlab/trusted-certs`: + + ```shell + sudo cp cert.pem /etc/gitlab/trusted-certs/ + ``` + +1. On the Gitaly clients, edit `git_data_dirs` in `/etc/gitlab/gitlab.rb` as follows: ```ruby git_data_dirs({ @@ -467,19 +539,10 @@ To configure Gitaly with TLS: 'storage1' => { 'gitaly_address' => 'tls://gitaly1.internal:9999' }, 'storage2' => { 'gitaly_address' => 'tls://gitaly2.internal:9999' }, }) - - gitlab_rails['gitaly_token'] = 'abc123secret' - gitlab_shell['secret_token'] = 'shellsecret' ``` -1. Save the file and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) on client node(s). -1. On the client node(s), copy the cert into the `/etc/gitlab/trusted-certs`: - - ```shell - sudo cp cert.pem /etc/gitlab/trusted-certs/ - ``` - -1. On the Gitaly server, create the `/etc/gitlab/ssl` directory and copy your key and certificate there: +1. Save the file and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure). +1. On the Gitaly servers, create the `/etc/gitlab/ssl` directory and copy your key and certificate there: ```shell sudo mkdir -p /etc/gitlab/ssl @@ -488,14 +551,14 @@ To configure Gitaly with TLS: sudo chmod 644 key.pem cert.pem ``` -1. Copy the cert to `/etc/gitlab/trusted-certs` so Gitaly will trust the cert when -calling into itself: +1. Copy all Gitaly server certificates, or their certificate authority, to `/etc/gitlab/trusted-certs` so Gitaly server will trust the certificate when +calling into itself or other Gitaly servers: - ```shell - sudo cp /etc/gitlab/ssl/cert.pem /etc/gitlab/trusted-certs/ - ``` + ```shell + sudo cp cert1.pem cert2.pem /etc/gitlab/trusted-certs/ + ``` -1. On the Gitaly server node(s), edit `/etc/gitlab/gitlab.rb` and add: +1. Edit `/etc/gitlab/gitlab.rb` and add: <!-- updates to following example must also be made at @@ -508,23 +571,23 @@ calling into itself: gitaly['key_path'] = "/etc/gitlab/ssl/key.pem" ``` -1. Save the file and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) on Gitaly server node(s). +1. Save the file and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure). 1. (Optional) After [verifying that all Gitaly traffic is being served over TLS](#observe-type-of-gitaly-connections), you can improve security by disabling non-TLS connections by commenting out or deleting `gitaly['listen_addr']` in `/etc/gitlab/gitlab.rb`, saving the file, - and [reconfiguring GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) - on Gitaly server node(s). + and [reconfiguring GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure). **For installations from source** -1. On the client node(s), add the cert to the system trusted certs: +1. Create certificates for Gitaly servers. +1. On the Gitaly clients, copy the certificates into the system trusted certificates: ```shell sudo cp cert.pem /usr/local/share/ca-certificates/gitaly.crt sudo update-ca-certificates ``` -1. On the client node(s), edit `/home/git/gitlab/config/gitlab.yml` as follows: +1. On the Gitaly clients, edit `storages` in `/home/git/gitlab/config/gitlab.yml` as follows: ```yaml gitlab: @@ -539,28 +602,21 @@ calling into itself: storage2: gitaly_address: tls://gitaly2.internal:9999 path: /some/dummy/path - - gitaly: - token: 'abc123secret' ``` NOTE: **Note:** `/some/dummy/path` should be set to a local folder that exists, however no data will be stored in this folder. This will no longer be necessary after - [this issue](https://gitlab.com/gitlab-org/gitaly/issues/1282) is resolved. + [this issue](https://gitlab.com/gitlab-org/gitaly/-/issues/1282) is resolved. -1. Save the file and[restart GitLab](../restart_gitlab.md#installations-from-source) -on client node(s). -1. Copy `/home/git/gitlab/.gitlab_shell_secret` from the client server to the same -path on the Gitaly server. -1. On the Gitaly server, create or edit `/etc/default/gitlab` and add: +1. Save the file and [restart GitLab](../restart_gitlab.md#installations-from-source). +1. On the Gitaly servers, create or edit `/etc/default/gitlab` and add: ```shell export SSL_CERT_DIR=/etc/gitlab/ssl ``` -1. Save the file. -1. Create the `/etc/gitlab/ssl` directory and copy your key and certificate there: +1. On the Gitaly servers, create the `/etc/gitlab/ssl` directory and copy your key and certificate there: ```shell sudo mkdir -p /etc/gitlab/ssl @@ -569,15 +625,14 @@ path on the Gitaly server. sudo chmod 644 key.pem cert.pem ``` -1. On the Gitaly server, add the cert to the system trusted certs so Gitaly will trust it -when calling into itself: +1. Copy all Gitaly server certificates, or their certificate authority, to the system trusted certificates so Gitaly server will trust the certificate when calling into itself or other Gitaly servers. ```shell sudo cp cert.pem /usr/local/share/ca-certificates/gitaly.crt sudo update-ca-certificates ``` -1. On the Gitaly server node(s), edit `/home/git/gitaly/config.toml` and add: +1. Edit `/home/git/gitaly/config.toml` and add: ```toml tls_listen_addr = '0.0.0.0:9999' @@ -587,12 +642,11 @@ when calling into itself: key_path = '/etc/gitlab/ssl/key.pem' ``` -1. Save the file and [restart GitLab](../restart_gitlab.md#installations-from-source) on Gitaly server node(s). +1. Save the file and [restart GitLab](../restart_gitlab.md#installations-from-source). 1. (Optional) After [verifying that all Gitaly traffic is being served over TLS](#observe-type-of-gitaly-connections), you can improve security by disabling non-TLS connections by commenting out or deleting `listen_addr` in `/home/git/gitaly/config.toml`, saving the file, - and [restarting GitLab](../restart_gitlab.md#installations-from-source) - on Gitaly server node(s). + and [restarting GitLab](../restart_gitlab.md#installations-from-source). ### Observe type of Gitaly connections @@ -673,8 +727,8 @@ gitaly['concurrency'] = [ This will limit the number of in-flight RPC calls for the given RPC's. The limit is applied per repository. In the example above, each on the -Gitaly server can have at most 20 simultaneous PostUploadPack calls in -flight, and the same for SSHUploadPack. If another request comes in for +Gitaly server can have at most 20 simultaneous `PostUploadPack` calls in +flight, and the same for `SSHUploadPack`. If another request comes in for a repository that has used up its 20 slots, that request will get queued. @@ -686,7 +740,7 @@ reporting about the concurrency limiter. In Prometheus, look for the `gitaly_rate_limiting_seconds` metrics. The name of the Prometheus metric is not quite right because this is a -concurrency limiter, not a rate limiter. If a client makes 1000 requests +concurrency limiter, not a rate limiter. If a Gitaly client makes 1000 requests in a row in a very short timespan, the concurrency will not exceed 1, and this mechanism (the concurrency limiter) will do nothing. @@ -697,7 +751,7 @@ downtime, or causes outages, or both. If you are careful, though, you *can* rotate Gitaly credentials without a service interruption. This procedure also works if you are running GitLab on a single server. -In that case, "Gitaly servers" and "Gitaly clients" refers to the same +In that case, "Gitaly server" and "Gitaly client" refers to the same machine. ### 1. Monitor current authentication behavior @@ -723,7 +777,7 @@ The only non-zero number should have `enforced="true",status="ok"`. If you have other non-zero numbers, something is wrong in your configuration. -The 'status="ok"' number reflects your current request rate. In the example +The `status="ok"` number reflects your current request rate. In the example above, Gitaly is handling about 4000 requests per second. Now you have established that you can monitor the Gitaly authentication @@ -894,9 +948,9 @@ which is still under development as of December 2019. ## Troubleshooting Gitaly -### Checking versions when using standalone Gitaly nodes +### Checking versions when using standalone Gitaly servers -When using standalone Gitaly nodes, you must make sure they are the same version +When using standalone Gitaly servers, you must make sure they are the same version as GitLab to ensure full compatibility. Check **Admin Area > Gitaly Servers** on your GitLab instance and confirm all Gitaly Servers are `Up to date`. @@ -931,8 +985,8 @@ gitaly-debug -h remote: GitLab: 401 Unauthorized ``` -You will need to sync your `gitlab-secrets.json` file with your GitLab -app nodes. +You will need to sync your `gitlab-secrets.json` file with your Gitaly clients (GitLab +app nodes). ### Client side gRPC logs @@ -975,11 +1029,11 @@ If you're running Gitaly on its own server and notice that users can successfully clone and fetch repositories (via both SSH and HTTPS), but can't push to them or make changes to the repository in the web UI without getting a `401 Unauthorized` message, then it's possible Gitaly is failing to authenticate -with the other nodes due to having the [wrong secrets file](#3-gitaly-server-configuration). +with the Gitaly client due to having the [wrong secrets file](#configure-gitaly-servers). Confirm the following are all true: -- When any user performs a `git push` to any repository on this Gitaly node, it +- When any user performs a `git push` to any repository on this Gitaly server, it fails with the following error (note the `401 Unauthorized`): ```shell @@ -993,7 +1047,8 @@ Confirm the following are all true: UI, it immediately fails with a red `401 Unauthorized` banner. - Creating a new project and [initializing it with a README](../../gitlab-basics/create-project.md#blank-projects) successfully creates the project but doesn't create the README. -- When [tailing the logs](https://docs.gitlab.com/omnibus/settings/logs.html#tail-logs-in-a-console-on-the-server) on an app node and reproducing the error, you get `401` errors +- When [tailing the logs](https://docs.gitlab.com/omnibus/settings/logs.html#tail-logs-in-a-console-on-the-server) + on a Gitaly client and reproducing the error, you get `401` errors when reaching the `/api/v4/internal/allowed` endpoint: ```shell @@ -1055,14 +1110,16 @@ Confirm the following are all true: [IP] - - [18/Jul/2019:00:30:14 +0000] "POST /api/v4/internal/allowed HTTP/1.1" 401 30 "" "Ruby" ``` -To fix this problem, confirm that your [`gitlab-secrets.json` file](#3-gitaly-server-configuration) -on the Gitaly node matches the one on all other nodes. If it doesn't match, -update the secrets file on the Gitaly node to match the others, then -[reconfigure the node](../restart_gitlab.md#omnibus-gitlab-reconfigure). +To fix this problem, confirm that your [`gitlab-secrets.json` file](#configure-gitaly-servers) +on the Gitaly server matches the one on Gitaly client. If it doesn't match, +update the secrets file on the Gitaly server to match the Gitaly client, then +[reconfigure](../restart_gitlab.md#omnibus-gitlab-reconfigure). ### Command line tools cannot connect to Gitaly -If you are having trouble connecting to a Gitaly node with command line (CLI) tools, and certain actions result in a `14: Connect Failed` error message, it means that gRPC cannot reach your Gitaly node. +If you are having trouble connecting to a Gitaly server with command line (CLI) tools, +and certain actions result in a `14: Connect Failed` error message, +it means that gRPC cannot reach your Gitaly server. Verify that you can reach Gitaly via TCP: @@ -1070,18 +1127,22 @@ Verify that you can reach Gitaly via TCP: sudo gitlab-rake gitlab:tcp_check[GITALY_SERVER_IP,GITALY_LISTEN_PORT] ``` -If the TCP connection fails, check your network settings and your firewall rules. If the TCP connection succeeds, your networking and firewall rules are correct. +If the TCP connection fails, check your network settings and your firewall rules. +If the TCP connection succeeds, your networking and firewall rules are correct. -If you use proxy servers in your command line environment, such as Bash, these can interfere with your gRPC traffic. +If you use proxy servers in your command line environment, such as Bash, these +can interfere with your gRPC traffic. -If you use Bash or a compatible command line environment, run the following commands to determine whether you have proxy servers configured: +If you use Bash or a compatible command line environment, run the following commands +to determine whether you have proxy servers configured: ```shell echo $http_proxy echo $https_proxy ``` -If either of these variables have a value, your Gitaly CLI connections may be getting routed through a proxy which cannot connect to Gitaly. +If either of these variables have a value, your Gitaly CLI connections may be +getting routed through a proxy which cannot connect to Gitaly. To remove the proxy setting, run the following commands (depending on which variables had values): @@ -1092,20 +1153,21 @@ unset https_proxy ### Gitaly not listening on new address after reconfiguring -When updating the `gitaly['listen_addr']` or `gitaly['prometheus_listen_addr']` values, Gitaly may continue to listen on the old address after a `sudo gitlab-ctl reconfigure`. +When updating the `gitaly['listen_addr']` or `gitaly['prometheus_listen_addr']` +values, Gitaly may continue to listen on the old address after a `sudo gitlab-ctl reconfigure`. -When this occurs, performing a `sudo gitlab-ctl restart` will resolve the issue. This will no longer be necessary after [this issue](https://gitlab.com/gitlab-org/gitaly/issues/2521) is resolved. +When this occurs, performing a `sudo gitlab-ctl restart` will resolve the issue. This will no longer be necessary after [this issue](https://gitlab.com/gitlab-org/gitaly/-/issues/2521) is resolved. -### Permission denied errors appearing in Gitaly logs when accessing repositories from a standalone Gitaly node +### Permission denied errors appearing in Gitaly logs when accessing repositories from a standalone Gitaly server If this error occurs even though file permissions are correct, it's likely that -the Gitaly node is experiencing +the Gitaly server is experiencing [clock drift](https://en.wikipedia.org/wiki/Clock_drift). -Please ensure that the GitLab and Gitaly nodes are synchronized and use an NTP time +Please ensure that the Gitaly clients and servers are synchronized and use an NTP time server to keep them synchronized if possible. ### Praefect -Praefect is an experimental daemon that allows for replication of the Git data. -It can be setup with omnibus, [as explained here](./praefect.md). +Praefect is a router and transaction manager for Gitaly, and a required +component for running a Gitaly Cluster. For more information see [Gitaly Cluster](praefect.md). diff --git a/doc/administration/gitaly/praefect.md b/doc/administration/gitaly/praefect.md index 0ea83f0e090..3d4e606205e 100644 --- a/doc/administration/gitaly/praefect.md +++ b/doc/administration/gitaly/praefect.md @@ -1,4 +1,7 @@ --- +stage: Create +group: Gitaly +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers type: reference --- @@ -89,7 +92,7 @@ to install GitLab](https://about.gitlab.com/install/). Provision a PostgreSQL server (PostgreSQL 11 or newer). Configuration through the Omnibus GitLab distribution is not yet supported. Follow this -[issue](https://gitlab.com/gitlab-org/gitaly/issues/2476) for updates. +[issue](https://gitlab.com/gitlab-org/gitaly/-/issues/2476) for updates. Prepare all your new nodes by [installing GitLab](https://about.gitlab.com/install/). @@ -319,37 +322,6 @@ application server, or a Gitaly node. } ``` -1. Enable the database replication queue: - - ```ruby - praefect['postgres_queue_enabled'] = true - ``` - - In the next release, database replication queue will be enabled by default. - See [issue #2615](https://gitlab.com/gitlab-org/gitaly/-/issues/2615). - -1. Enable automatic failover by editing `/etc/gitlab/gitlab.rb`: - - ```ruby - praefect['failover_enabled'] = true - praefect['failover_election_strategy'] = 'sql' - ``` - - When automatic failover is enabled, Praefect checks the health of internal - Gitaly nodes. If the primary has a certain amount of health checks fail, it - will promote one of the secondaries to be primary, and demote the primary to - be a secondary. - - NOTE: **Note:** Database leader election will be [enabled by default in the - future](https://gitlab.com/gitlab-org/gitaly/-/issues/2682). - - Caution, **automatic failover** favors availability over consistency and will - cause data loss if changes have not been replicated to the newly elected - primary. In the next release, leader election will [prefer to promote up to - date replicas](https://gitlab.com/gitlab-org/gitaly/-/issues/2642), and it - will be an option to favor consistency by marking [out-of-date repositories - read-only](https://gitlab.com/gitlab-org/gitaly/-/issues/2630). - 1. Save the changes to `/etc/gitlab/gitlab.rb` and [reconfigure Praefect](../restart_gitlab.md#omnibus-gitlab-reconfigure): @@ -409,7 +381,7 @@ Particular attention should be shown to: `gitaly-2`, and `gitaly-3` as Gitaly storage names. For more information on Gitaly server configuration, see our [Gitaly -documentation](index.md#3-gitaly-server-configuration). +documentation](index.md#configure-gitaly-servers). 1. SSH into the **Gitaly** node and login as root: @@ -424,16 +396,18 @@ documentation](index.md#3-gitaly-server-configuration). postgresql['enable'] = false redis['enable'] = false nginx['enable'] = false - prometheus['enable'] = false grafana['enable'] = false puma['enable'] = false sidekiq['enable'] = false gitlab_workhorse['enable'] = false prometheus_monitoring['enable'] = false - # Enable only the Praefect service + # Enable only the Gitaly service gitaly['enable'] = true + # Enable Prometheus if needed + prometheus['enable'] = true + # Prevent database connections during 'gitlab-ctl reconfigure' gitlab_rails['rake_cache_clear'] = false gitlab_rails['auto_migrate'] = false @@ -682,8 +656,8 @@ Particular attention should be shown to: Repository > Repository storage** to make the newly configured Praefect cluster the storage location for new Git repositories. - - Deselect the **default** storage location - - Select the **praefect** storage location + - The default option is unchecked. + - The Praefect option is checked. ![Update repository storage](img/praefect_storage_v12_10.png) @@ -744,7 +718,7 @@ Praefect regularly checks the health of each backend Gitaly node. This information can be used to automatically failover to a new primary node if the current primary node is found to be unhealthy. -- **PostgreSQL (recommended):** Enabled by setting +- **PostgreSQL (recommended):** Enabled by default, and equivalent to: `praefect['failover_election_strategy'] = sql`. This configuration option will allow multiple Praefect nodes to coordinate via the PostgreSQL database to elect a primary Gitaly node. This configuration @@ -755,24 +729,21 @@ current primary node is found to be unhealthy. reconfigured in `/etc/gitlab/gitlab.rb` on the Praefect node. Modify the `praefect['virtual_storages']` field by moving the `primary = true` to promote a different Gitaly node to primary. In the steps above, `gitaly-1` was set to - the primary. -- **Memory:** Enabled by setting `praefect['failover_enabled'] = true` in - `/etc/gitlab/gitlab.rb` on the Praefect node. If a sufficient number of health + the primary. Requires `praefect['failover_enabled'] = false` in the configuration. +- **Memory:** Enabled by setting `praefect['failover_election_strategy'] = 'local'` + in `/etc/gitlab/gitlab.rb` on the Praefect node. If a sufficient number of health checks fail for the current primary backend Gitaly node, and new primary will be elected. **Do not use with multiple Praefect nodes!** Using with multiple Praefect nodes is likely to result in a split brain. -NOTE: **Note:**: Praefect does not yet account for replication lag on -the secondaries during the election process, so data loss can occur -during a failover. Follow issue -[#2642](https://gitlab.com/gitlab-org/gitaly/-/issues/2642) for updates. - It is likely that we will implement support for Consul, and a cloud native strategy in the future. ## Identifying Impact of a Primary Node Failure -When a primary Gitaly node fails, there is a chance of data loss. Data loss can occur if there were outstanding replication jobs the secondaries did not manage to process before the failure. The Praefect `dataloss` sub-command helps identify these cases by counting the number of dead replication jobs for each repository within a given time frame. +When a primary Gitaly node fails, there is a chance of data loss. Data loss can occur if there were outstanding replication jobs the secondaries did not manage to process before the failure. The `dataloss` Praefect sub-command helps identify these cases by counting the number of dead replication jobs for each repository. This command must be executed on a Praefect node. + +A time frame to search can be specified with `-from` and `-to`: ```shell sudo /opt/gitlab/embedded/bin/praefect -config /var/opt/gitlab/praefect/config.toml dataloss -from <rfc3339-time> -to <rfc3339-time> @@ -784,9 +755,7 @@ If the time frame is not specified, dead replication jobs from the last six hour sudo /opt/gitlab/embedded/bin/praefect -config /var/opt/gitlab/praefect/config.toml dataloss Failed replication jobs between [2020-01-02 00:00:00 +0000 UTC, 2020-01-02 06:00:00 +0000 UTC): -example/repository-1: 1 jobs -example/repository-2: 4 jobs -example/repository-3: 2 jobs +@hashed/fa/53/fa539965395b8382145f8370b34eab249cf610d2d6f2943c95b9b9d08a63d4a3.git: 2 jobs ``` To specify a time frame in UTC, run: @@ -797,7 +766,8 @@ sudo /opt/gitlab/embedded/bin/praefect -config /var/opt/gitlab/praefect/config.t ### Checking repository checksums -To check a project's checksums across all nodes, the Praefect replicas Rake task can be used: +To check a project's repository checksums across on all Gitaly nodes, the +replicas Rake task can be run on the main GitLab node: ```shell sudo gitlab-rake "gitlab:praefect:replicas[project_id]" diff --git a/doc/administration/gitaly/reference.md b/doc/administration/gitaly/reference.md index e718d8953ca..52fd6fa6900 100644 --- a/doc/administration/gitaly/reference.md +++ b/doc/administration/gitaly/reference.md @@ -1,4 +1,7 @@ --- +stage: Create +group: Gitaly +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers type: reference --- @@ -12,7 +15,7 @@ The configuration file is passed as an argument to the `gitaly` executable. This is usually done by either Omnibus GitLab or your [init](https://en.wikipedia.org/wiki/Init) script. -An [example config file](https://gitlab.com/gitlab-org/gitaly/blob/master/config.toml.example) +An [example configuration file](https://gitlab.com/gitlab-org/gitaly/blob/master/config.toml.example) can be found in the Gitaly project. ## Format @@ -157,7 +160,7 @@ sum(rate(gitaly_catfile_cache_total{type="hit"}[5m])) / sum(rate(gitaly_catfile_ A Gitaly process uses one or more `gitaly-ruby` helper processes to execute RPC's implemented in Ruby instead of Go. The `[gitaly-ruby]` -section of the config file contains settings for these helper processes. +section of the configuration file contains settings for these helper processes. These processes are known to occasionally suffer from memory leaks. Gitaly restarts its `gitaly-ruby` helpers when their memory exceeds the @@ -190,7 +193,7 @@ For historical reasons the Git hooks that allow GitLab to validate and react to Git pushes. Because Gitaly "owns" Git pushes, GitLab Shell must therefore be installed alongside Gitaly. This will be [simplified in the -future](https://gitlab.com/gitlab-org/gitaly/issues/1226). +future](https://gitlab.com/gitlab-org/gitaly/-/issues/1226). | Name | Type | Required | Description | | ---- | ---- | -------- | ----------- | |