diff options
Diffstat (limited to 'doc/administration/gitaly')
| -rw-r--r-- | doc/administration/gitaly/configure_gitaly.md | 19 | ||||
| -rw-r--r-- | doc/administration/gitaly/faq.md | 8 | ||||
| -rw-r--r-- | doc/administration/gitaly/index.md | 10 | ||||
| -rw-r--r-- | doc/administration/gitaly/praefect.md | 100 | ||||
| -rw-r--r-- | doc/administration/gitaly/recovery.md | 2 | ||||
| -rw-r--r-- | doc/administration/gitaly/troubleshooting.md | 9 |
6 files changed, 93 insertions, 55 deletions
diff --git a/doc/administration/gitaly/configure_gitaly.md b/doc/administration/gitaly/configure_gitaly.md index a0c959d5de9..88efd1885db 100644 --- a/doc/administration/gitaly/configure_gitaly.md +++ b/doc/administration/gitaly/configure_gitaly.md @@ -28,6 +28,12 @@ The following configuration options are also available: - Configuring the [number of `gitaly-ruby` workers](#configure-number-of-gitaly-ruby-workers). - Limiting [RPC concurrency](#limit-rpc-concurrency). +## About the Gitaly token + +The token referred to throughout the Gitaly documentation is just an arbitrary password selected by +the administrator. It is unrelated to tokens created for the GitLab API or other similar web API +tokens. + ## Run Gitaly on its own server By default, Gitaly is run on the same server as Gitaly clients and is @@ -116,11 +122,6 @@ We assume your GitLab installation has three repository storages: You can use as few as one server with one repository storage if desired. -NOTE: -The token referred to throughout the Gitaly documentation is just an arbitrary password selected by -the administrator. It is unrelated to tokens created for the GitLab API or other similar web API -tokens. - ### Install Gitaly Install Gitaly on each Gitaly server using either Omnibus GitLab or install it from source: @@ -476,7 +477,7 @@ example: ```ruby git_data_dirs({ 'default' => { 'gitaly_address' => 'tcp://gitaly1.internal:8075' }, - # Address of the GitLab server that has Gitaly running on it + # Address of the GitLab server that also has Gitaly running on it 'storage1' => { 'gitaly_address' => 'tcp://gitlab.internal:8075', 'path' => '/mnt/gitlab/git-data' }, 'storage2' => { 'gitaly_address' => 'tcp://gitaly2.internal:8075' }, }) @@ -565,12 +566,6 @@ Note the following: - You can configure Gitaly servers with both an unencrypted listening address `listen_addr` and an encrypted listening address `tls_listen_addr` at the same time. This allows you to gradually transition from unencrypted to encrypted traffic if necessary. -- When running Praefect sub-commands such as `dial-nodes` and `list-untracked-repositories` from the command line with Gitaly TLS enabled, you must set - the `SSL_CERT_DIR` or `SSL_CERT_FILE` environment variable so that the Gitaly certificate is trusted. For example: - - ```shell - sudo SSL_CERT_DIR=/etc/gitlab/trusted_certs /opt/gitlab/embedded/bin/praefect -config /var/opt/gitlab/praefect/config.toml dial-nodes - ``` To configure Gitaly with TLS: diff --git a/doc/administration/gitaly/faq.md b/doc/administration/gitaly/faq.md index f79b9555c10..b0a88e8adc9 100644 --- a/doc/administration/gitaly/faq.md +++ b/doc/administration/gitaly/faq.md @@ -23,10 +23,10 @@ Gitaly Cluster and [Geo](../geo/index.md) both provide redundancy. However the r The following table outlines the major differences between Gitaly Cluster and Geo: -| Tool | Nodes | Locations | Latency tolerance | Failover | Consistency | Provides redundancy for | -|:---------------|:---------|:----------|:-------------------|:----------------------------------------------------------------------------|:-----------------------------------------|:------------------------| -| Gitaly Cluster | Multiple | Single | Approximately 1 ms | [Automatic](praefect.md#automatic-failover-and-primary-election-strategies) | [Strong](index.md#strong-consistency) | Data storage in Git | -| Geo | Multiple | Multiple | Up to one minute | [Manual](../geo/disaster_recovery/index.md) | Eventual | Entire GitLab instance | +| Tool | Nodes | Locations | Latency tolerance | Failover | Consistency | Provides redundancy for | +|:---------------|:---------|:----------|:------------------------------------------------------------------------------------------------------|:----------------------------------------------------------------------------|:--------------------------------------|:------------------------| +| Gitaly Cluster | Multiple | Single | [Less than 1 second, ideally single-digit milliseconds](praefect.md#network-latency-and-connectivity) | [Automatic](praefect.md#automatic-failover-and-primary-election-strategies) | [Strong](index.md#strong-consistency) | Data storage in Git | +| Geo | Multiple | Multiple | Up to one minute | [Manual](../geo/disaster_recovery/index.md) | Eventual | Entire GitLab instance | For more information, see: diff --git a/doc/administration/gitaly/index.md b/doc/administration/gitaly/index.md index 4d60832720b..ea24e901943 100644 --- a/doc/administration/gitaly/index.md +++ b/doc/administration/gitaly/index.md @@ -290,7 +290,7 @@ including [horizontally distributing reads](https://gitlab.com/groups/gitlab-org #### Distributed reads -> - Introduced in GitLab 13.1 in [beta](https://about.gitlab.com/handbook/product/gitlab-the-product/#alpha-beta-ga) with feature flag `gitaly_distributed_reads` set to disabled. +> - Introduced in GitLab 13.1 in [beta](../../policy/alpha-beta-support.md#beta-features) with feature flag `gitaly_distributed_reads` set to disabled. > - [Made generally available and enabled by default](https://gitlab.com/gitlab-org/gitaly/-/issues/2951) in GitLab 13.3. > - [Disabled by default](https://gitlab.com/gitlab-org/gitaly/-/issues/3178) in GitLab 13.5. > - [Enabled by default](https://gitlab.com/gitlab-org/gitaly/-/issues/3334) in GitLab 13.8. @@ -316,8 +316,8 @@ You can [monitor distribution of reads](#monitor-gitaly-cluster) using Prometheu #### Strong consistency -> - Introduced in GitLab 13.1 in [alpha](https://about.gitlab.com/handbook/product/gitlab-the-product/#alpha-beta-ga), disabled by default. -> - Entered [beta](https://about.gitlab.com/handbook/product/gitlab-the-product/#alpha-beta-ga) in GitLab 13.2, disabled by default. +> - Introduced in GitLab 13.1 in [alpha](../../policy/alpha-beta-support.md#alpha-features), disabled by default. +> - Entered [beta](../../policy/alpha-beta-support.md#beta-features) in GitLab 13.2, disabled by default. > - In GitLab 13.3, disabled unless primary-wins voting strategy is disabled. > - From GitLab 13.4, enabled by default. > - From GitLab 13.5, you must use Git v2.28.0 or higher on Gitaly nodes to enable strong consistency. @@ -529,6 +529,10 @@ To monitor [strong consistency](#strong-consistency), you can use the following - `gitaly_hook_transaction_voting_delay_seconds`, the client-side delay introduced by waiting for the transaction to be committed. +To monitor the number of repositories that have no healthy, up-to-date replicas: + +- `gitaly_praefect_unavailable_repositories` + You can also monitor the [Praefect logs](../logs.md#praefect-logs). #### Database metrics `/db_metrics` endpoint diff --git a/doc/administration/gitaly/praefect.md b/doc/administration/gitaly/praefect.md index e2db30b8358..8f17835b8a3 100644 --- a/doc/administration/gitaly/praefect.md +++ b/doc/administration/gitaly/praefect.md @@ -39,10 +39,26 @@ NOTE: If not set in GitLab, feature flags are read as false from the console and Praefect uses their default value. The default value depends on the GitLab version. -### Network connectivity +### Network latency and connectivity -Gitaly Cluster [components](index.md#components) need to communicate with each other over many -routes. Your firewall rules must allow the following for Gitaly Cluster to function properly: +Network latency for Gitaly Cluster should ideally be measurable in single-digit milliseconds. Latency is particularly +important for: + +- Gitaly node health checks. Nodes must be able to respond within 1 second. +- Reference transactions that enforce [strong consistency](index.md#strong-consistency). Lower latencies mean Gitaly + nodes can agree on changes faster. + +Achieving acceptable latency between Gitaly nodes: + +- On physical networks generally means high bandwidth, single location connections. +- On the cloud generally means within the same region, including allowing cross availability zone replication. These links + are designed for this type of synchronization. Latency of less than 2ms should be sufficient for Gitaly Cluster. + +If you can't provide low network latencies for replication (for example, between distant locations), consider Geo. For +more information, see [How does Gitaly Cluster compare to Geo](faq.md#how-does-gitaly-cluster-compare-to-geo). + +Gitaly Cluster [components](index.md#components) communicate with each other over many routes. Your firewall rules must +allow the following for Gitaly Cluster to function properly: | From | To | Default port | TLS port | |:-----------------------|:-----------------------|:-------------|:---------| @@ -254,11 +270,11 @@ reads distribution caching is enabled by configuration To reduce PostgreSQL resource consumption, we recommend setting up and configuring [PgBouncer](https://www.pgbouncer.org/) in front of the PostgreSQL instance. To do -this, you must point Praefect to PgBouncer by setting Praefect database parameters: +this, you must point Praefect to PgBouncer by setting database parameters on Praefect configuration: ```ruby praefect['database_host'] = PGBOUNCER_HOST -praefect['database_port'] = 5432 +praefect['database_port'] = 6432 praefect['database_user'] = 'praefect' praefect['database_password'] = PRAEFECT_SQL_PASSWORD praefect['database_dbname'] = 'praefect_production' @@ -273,40 +289,21 @@ Praefect requires an additional connection to the PostgreSQL that supports the this feature is only available with `session` pool mode (`pool_mode = session`). It is not supported in `transaction` pool mode (`pool_mode = transaction`). -For the additional connection, you must either: +To configure the additional connection, you must either: -- Connect Praefect directly to PostgreSQL and bypass PgBouncer. - Configure a new PgBouncer database that uses to the same PostgreSQL database endpoint, but with different pool mode. That is, `pool_mode = session`. +- Connect Praefect directly to PostgreSQL and bypass PgBouncer. -Praefect can be configured to use different connection parameters for direct access -to PostgreSQL. This is the connection that supports the `LISTEN` feature. - -Here is an example of Praefect that bypasses PgBouncer and directly connects to PostgreSQL: - -```ruby -praefect['database_direct_host'] = POSTGRESQL_HOST -praefect['database_direct_port'] = 5432 - -# Use the following to override parameters of direct database connection. -# Comment out where the parameters are the same for both connections. - -praefect['database_direct_user'] = 'praefect' -praefect['database_direct_password'] = PRAEFECT_SQL_PASSWORD -praefect['database_direct_dbname'] = 'praefect_production' -#praefect['database_direct_sslmode'] = '...' -#praefect['database_direct_sslcert'] = '...' -#praefect['database_direct_sslkey'] = '...' -#praefect['database_direct_sslrootcert'] = '...' -``` +#### Configure a new PgBouncer database with `pool_mode = session` -We recommend using PgBouncer with `session` pool mode instead. You can use the +We recommend using PgBouncer with `session` pool mode. You can use the [bundled PgBouncer](../postgresql/pgbouncer.md) or use an external PgBouncer and [configure it manually](https://www.pgbouncer.org/config.html). -The following example uses the bundled PgBouncer and sets up two separate connection pools, +The following example uses the bundled PgBouncer and sets up two separate connection pools on PostgreSQL host, one in `session` pool mode and the other in `transaction` pool mode. For this example to work, -you need to prepare PostgreSQL server with [setup instruction](#manual-database-setup): +you need to prepare PostgreSQL server as documented in [in the setup instructions](#manual-database-setup): ```ruby pgbouncer['databases'] = { @@ -373,6 +370,29 @@ your databases manually and configuring an external PgBouncer, you must include its password in the file used by PgBouncer. For example, `userlist.txt` if the [`auth_file`](https://www.pgbouncer.org/config.html#auth_file) configuration option is set. For more details, consult the PgBouncer documentation. +#### Configure Praefect to connect directly to PostgreSQL + +As an alternative to configuring PgBouncer with `session` pool mode, Praefect can be configured to use different connection parameters for direct access +to PostgreSQL. This is the connection that supports the `LISTEN` feature. + +An example of Praefect configuration that bypasses PgBouncer and directly connects to PostgreSQL: + +```ruby +praefect['database_direct_host'] = POSTGRESQL_HOST +praefect['database_direct_port'] = 5432 + +# Use the following to override parameters of direct database connection. +# Comment out where the parameters are the same for both connections. + +praefect['database_direct_user'] = 'praefect' +praefect['database_direct_password'] = PRAEFECT_SQL_PASSWORD +praefect['database_direct_dbname'] = 'praefect_production' +#praefect['database_direct_sslmode'] = '...' +#praefect['database_direct_sslcert'] = '...' +#praefect['database_direct_sslkey'] = '...' +#praefect['database_direct_sslrootcert'] = '...' +``` + ### Praefect > [Introduced](https://gitlab.com/gitlab-org/gitaly/-/issues/2634) in GitLab 13.4, Praefect nodes can no longer be designated as `primary`. @@ -570,15 +590,15 @@ On the **Praefect** node: edit `/etc/gitlab/gitlab.rb`, remember to run `sudo gitlab-ctl reconfigure` again before trying the `sql-ping` command. -#### Enabling TLS support +#### Enable TLS support > [Introduced](https://gitlab.com/gitlab-org/gitaly/-/issues/1698) in GitLab 13.2. Praefect supports TLS encryption. To communicate with a Praefect instance that listens for secure connections, you must: -- Use a `tls://` URL scheme in the `gitaly_address` of the corresponding storage entry - in the GitLab configuration. +- Ensure Gitaly is [configured for TLS](configure_gitaly.md#enable-tls-support) and use a `tls://` URL scheme in the `gitaly_address` + of the corresponding storage entry in the GitLab configuration. - Bring your own certificates because this isn't provided automatically. The certificate corresponding to each Praefect server must be installed on that Praefect server. @@ -594,6 +614,13 @@ Note the following: - Hostname, you can either use the Common Name field for this, or add it as a Subject Alternative Name. - IP address, you must add it as a Subject Alternative Name to the certificate. +- When running Praefect sub-commands such as `dial-nodes` and `list-untracked-repositories` from the command line with + [Gitaly TLS enabled](configure_gitaly.md#enable-tls-support), you must set the `SSL_CERT_DIR` or `SSL_CERT_FILE` + environment variable so that the Gitaly certificate is trusted. For example: + + ```shell + sudo SSL_CERT_DIR=/etc/gitlab/trusted_certs /opt/gitlab/embedded/bin/praefect -config /var/opt/gitlab/praefect/config.toml dial-nodes + ``` - You can configure Praefect servers with both an unencrypted listening address `listen_addr` and an encrypted listening address `tls_listen_addr` at the same time. @@ -639,7 +666,7 @@ To configure Praefect with TLS: ```ruby git_data_dirs({ "default" => { - "gitaly_address" => 'tls://PRAEFECT_LOADBALANCER_HOST:2305', + "gitaly_address" => 'tls://PRAEFECT_LOADBALANCER_HOST:3305', "gitaly_token" => 'PRAEFECT_EXTERNAL_TOKEN' } }) @@ -957,7 +984,10 @@ Particular attention should be shown to: balancer. - `PRAEFECT_EXTERNAL_TOKEN` with the real secret - If you are using TLS, the `gitaly_address` should begin with `tls://`. + If you are using TLS: + + - The `gitaly_address` should begin with `tls://` instead. + - The port should be changed to `3305`. ```ruby git_data_dirs({ diff --git a/doc/administration/gitaly/recovery.md b/doc/administration/gitaly/recovery.md index e1b9a73908d..9210c8f08d3 100644 --- a/doc/administration/gitaly/recovery.md +++ b/doc/administration/gitaly/recovery.md @@ -28,7 +28,7 @@ To minimize data loss in GitLab 13.0 to 14.0, Gitaly Cluster: ### Read-only mode -> - Introduced in GitLab 13.0 as [generally available](https://about.gitlab.com/handbook/product/gitlab-the-product/#generally-available-ga). +> - Introduced in GitLab 13.0 as [generally available](../../policy/alpha-beta-support.md#generally-available-ga). > - Between GitLab 13.0 and GitLab 13.2, read-only mode applied to the whole virtual storage and occurred whenever failover occurred. > - [In GitLab 13.3 and later](https://gitlab.com/gitlab-org/gitaly/-/issues/2862), read-only mode applies on a per-repository basis and only occurs if a new primary is out of date. new primary. If the failed primary contained unreplicated writes, [data loss can occur](#check-for-data-loss). diff --git a/doc/administration/gitaly/troubleshooting.md b/doc/administration/gitaly/troubleshooting.md index fdd281c1a90..ff136b44340 100644 --- a/doc/administration/gitaly/troubleshooting.md +++ b/doc/administration/gitaly/troubleshooting.md @@ -435,6 +435,15 @@ Here are common errors and potential causes: - Praefect cannot reach one or more of its child Gitaly nodes. Try running the Praefect connection checker to diagnose. +### Praefect database experiencing high CPU load + +Some common reasons for the Praefect database to experience elevated CPU usage include: + +- Prometheus metrics scrapes [running an expensive query](https://gitlab.com/gitlab-org/gitaly/-/issues/3796). If you have GitLab 14.2 + or above, set `praefect['separate_database_metrics'] = true` in `gitlab.rb`. +- [Read distribution caching](praefect.md#reads-distribution-caching) is disabled, increasing the number of queries made to the + database when user traffic is high. Ensure read distribution caching is enabled. + ### Determine primary Gitaly node To determine the primary node of a repository: |