diff options
Diffstat (limited to 'doc/administration/gitaly')
| -rw-r--r-- | doc/administration/gitaly/configure_gitaly.md | 86 | ||||
| -rw-r--r-- | doc/administration/gitaly/faq.md | 90 | ||||
| -rw-r--r-- | doc/administration/gitaly/index.md | 62 | ||||
| -rw-r--r-- | doc/administration/gitaly/monitoring.md | 1 | ||||
| -rw-r--r-- | doc/administration/gitaly/praefect.md | 68 | ||||
| -rw-r--r-- | doc/administration/gitaly/recovery.md | 2 | ||||
| -rw-r--r-- | doc/administration/gitaly/troubleshooting.md | 11 |
7 files changed, 188 insertions, 132 deletions
diff --git a/doc/administration/gitaly/configure_gitaly.md b/doc/administration/gitaly/configure_gitaly.md index 6888a2abe9a..26aa750900b 100644 --- a/doc/administration/gitaly/configure_gitaly.md +++ b/doc/administration/gitaly/configure_gitaly.md @@ -67,7 +67,7 @@ When running Gitaly on its own server, note the following regarding GitLab versi servers. - From GitLab 11.8 to 12.2, it is possible to use Elasticsearch in a Gitaly setup that doesn't use NFS. To use Elasticsearch in these versions, the - [repository indexer](../../integration/elasticsearch.md#elasticsearch-repository-indexer) + [repository indexer](../../integration/advanced_search/elasticsearch.md#elasticsearch-repository-indexer) must be enabled in your GitLab configuration. - [In GitLab 12.3 and later](https://gitlab.com/gitlab-org/gitlab/-/issues/6481), the new indexer is the default and no configuration is required. @@ -825,14 +825,14 @@ information, see the [relevant documentation](monitoring.md#monitor-gitaly-concu ## Control groups -> - Introduced in GitLab 13.10. +> Introduced in GitLab 13.10. Gitaly shells out to Git for many of its operations. Git can consume a lot of resources for certain operations, especially for large repositories. Control groups (cgroups) in Linux allow limits to be imposed on how much memory and CPU can be consumed. See the [`cgroups` Linux man page](https://man7.org/linux/man-pages/man7/cgroups.7.html) for more information. -cgroups can be useful for protecting the system against resource exhaustion because of overconsumption of memory and CPU. +cgroups can be useful for protecting the system against resource exhaustion because of overcomsumption of memory and CPU. Gitaly has built-in cgroups control. When configured, Gitaly assigns Git processes to a cgroup based on the repository the Git command is operating in. @@ -856,7 +856,59 @@ Using cgroups allows the kernel to kill these operations before they hog up all ### Configure cgroups in Gitaly -To configure cgroups in Gitaly, add `gitaly['cgroups']` to `/etc/gitlab/gitlab.rb`. For +Two ways of configuring cgroups are available. + +#### Configure cgroups (new method) + +> This method of configuring cgroups introduced in GitLab 15.1. + +FLAG: +On self-managed GitLab, by default this method of configuring cgroups is not available. To make it available, ask an administrator to +[enable the feature flag](../feature_flags.md) named `gitaly_run_cmds_in_cgroup`. + +Gitaly creates a pool of cgroups that are isolated based on the repository used in the Git command to be placed under one of these cgroups. + +To configure cgroups in Gitaly, add `gitaly['cgroups']` to `/etc/gitlab/gitlab.rb`. + +For example: + +```ruby +# in /etc/gitlab/gitlab.rb +gitaly['cgroups_mountpoint'] = "/sys/fs/cgroup" +gitaly['cgroups_hierarchy_root'] =>"gitaly" +gitaly['cgroups_memory_bytes'] = 64424509440, # 60gb +gitaly['cgroups_cpu_shares'] = 1024 +gitaly['cgroups_repositories_count'] => 1000, +gitaly['cgroups_repositories_memory_bytes'] => 32212254720 # 20gb +gitaly['cgroups_repositories_cpu_shares'] => 512 +``` + +- `cgroups_mountpoint` is where the parent cgroup directory is mounted. Defaults to `/sys/fs/cgroup`. +- `cgroups_hierarchy_root` is the parent cgroup under which Gitaly creates groups, and + is expected to be owned by the user and group Gitaly runs as. Omnibus GitLab + creates the set of directories `mountpoint/<cpu|memory>/hierarchy_root` + when Gitaly starts. +- `cgroups_memory_bytes` is the total memory limit that is imposed collectively on all + Git processes that Gitaly spawns. 0 implies no limit. +- `cgroups_cpu_shares` is the cpu limit that is imposed collectively on all Git + processes that Gitaly spawns. 0 implies no limit. The maximum is 1024 shares, + which represents 100% of CPU. +- `cgroups_repositories_count` is the number of cgroups in the cgroups pool. Each time a new Git + command is spawned, Gitaly assigns it to one of these cgroups based + on the repository the command is for. A circular hashing algorithm assigns + Git commands to these cgroups, so a Git command for a repository is + always assigned to the same cgroup. +- `cgroups_repositories_memory_bytes` is the total memory limit that is imposed collectively on all + Git processes that Gitaly spawns. 0 implies no limit. This value cannot exceed + that of the top level `cgroups_memory_bytes`. +- `cgroups_repositories_cpu_shares` is the CPU limit that is imposed collectively on all Git + processes Gitaly spawns. 0 implies no limit. The maximum is 1024 shares, + which represents 100% of CPU. This value cannot exceed that of the top + level`cgroups_cpu_shares`. + +#### Configure cgroups (legacy method) + +To configure cgroups in Gitaly for GitLab versions using the legacy method, add `gitaly['cgroups']` to `/etc/gitlab/gitlab.rb`. For example: ```ruby @@ -868,7 +920,6 @@ gitaly['cgroups_memory_limit'] = 32212254720 gitaly['cgroups_memory_enabled'] = true gitaly['cgroups_cpu_shares'] = 1024 gitaly['cgroups_cpu_enabled'] = true - ``` - `cgroups_count` is the number of cgroups created. Each time a new @@ -883,9 +934,28 @@ gitaly['cgroups_cpu_enabled'] = true - `cgroups_memory_enabled` enables or disables the memory limit on cgroups. - `cgroups_memory_bytes` is the total memory limit each cgroup imposes on the processes added to it. - `cgroups_cpu_enabled` enables or disables the CPU limit on cgroups. -- `cgroups_cpu_shares` is the CPU limit each cgroup imposes on the processes added to it. The maximum is 1024 shares, - which represents 100% of CPU. - which represents 100% of CPU. +- `cgroups_cpu_shares` is the CPU limit each cgroup imposes on the processes added to it. The maximum is 1024 shares, which represents 100% of CPU. + +### Configuring oversubscription + +In the previous example using the new configuration method: + +- The top level memory limit is capped at 60gb. +- Each of the 1000 cgroups in the repositories pool is capped at 20gb. + +This is called "oversubscription". Each cgroup in the pool has a much larger capacity than 1/1000th +of the top-level memory limit. + +This strategy has two main benefits: + +- It gives the host protection from overall memory starvation (OOM), because the top-level + cgroup's memory limit can be set to a threshold smaller than the host's + capacity. Processes outside of that cgroup are not at risk of OOM. +- It allows each individual cgroup in the pool to burst up to a generous upper + bound (in this example 20 GB) that is smaller than the parent cgroup's limit, + but substantially larger than 1/N of the parent's limit. In this example, up + to 3 child cgroups can concurrently burst up to their max. In general, all + 1000 cgroups would use much less than the 20 GB. ## Background Repository Optimization diff --git a/doc/administration/gitaly/faq.md b/doc/administration/gitaly/faq.md index a5c2c7d1469..f6571295200 100644 --- a/doc/administration/gitaly/faq.md +++ b/doc/administration/gitaly/faq.md @@ -1,90 +1,6 @@ --- -stage: Create -group: Gitaly -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +redirect_to: 'index.md' +remove_date: '2022-08-24' --- -# Frequently asked questions **(FREE SELF)** - -The following are answers to frequently asked questions about Gitaly and Gitaly Cluster. For -troubleshooting information, see [Troubleshooting Gitaly and Gitaly Cluster](troubleshooting.md). - -## How does Gitaly Cluster compare to Geo? - -Gitaly Cluster and [Geo](../geo/index.md) both provide redundancy. However the redundancy of: - -- Gitaly Cluster provides fault tolerance for data storage and is invisible to the user. Users are - not aware when Gitaly Cluster is used. -- Geo provides [replication](../geo/index.md) and [disaster recovery](../geo/disaster_recovery/index.md) for - an entire instance of GitLab. Users know when they are using Geo for - [replication](../geo/index.md). Geo [replicates multiple data types](../geo/replication/datatypes.md#limitations-on-replicationverification), - including Git data. - -The following table outlines the major differences between Gitaly Cluster and Geo: - -| Tool | Nodes | Locations | Latency tolerance | Failover | Consistency | Provides redundancy for | -|:---------------|:---------|:----------|:------------------------------------------------------------------------------------------------------|:----------------------------------------------------------------------------|:--------------------------------------|:------------------------| -| Gitaly Cluster | Multiple | Single | [Less than 1 second, ideally single-digit milliseconds](praefect.md#network-latency-and-connectivity) | [Automatic](praefect.md#automatic-failover-and-primary-election-strategies) | [Strong](index.md#strong-consistency) | Data storage in Git | -| Geo | Multiple | Multiple | Up to one minute | [Manual](../geo/disaster_recovery/index.md) | Eventual | Entire GitLab instance | - -For more information, see: - -- Geo [use cases](../geo/index.md#use-cases). -- Geo [architecture](../geo/index.md#architecture). - -## Are there instructions for migrating to Gitaly Cluster? - -Yes! For more information, see [Migrating to Gitaly Cluster](index.md#migrating-to-gitaly-cluster). - -## What are some repository storage recommendations? - -The size of the required storage can vary between instances and depends on the set -[replication factor](index.md#replication-factor). You might want to include implementing -repository storage redundancy. - -For a replication factor: - -- Of `1`: NFS, Gitaly, and Gitaly Cluster have roughly the same storage requirements. -- More than `1`: The amount of required storage is `used space * replication factor`. `used space` - should include any planned future growth. - -## What are some Praefect database storage requirements? - -The requirements are relatively low because the database contains only metadata of: - -- Where repositories are located. -- Some queued work. - -It depends on the number of repositories, but a useful minimum is 5-10 GB, similar to the main -GitLab application database. - -## Can the GitLab application database and the Praefect database be on the same servers? - -Yes, however Praefect should have it's own database server when using Omnibus GitLab PostgreSQL. If -there is a failover, Praefect isn't aware and starts to fail as the database it's trying to use would -either: - -- Be unavailable. -- In read-only mode. - -A future solution may allow for Praefect and Omnibus GitLab databases on the same PostgreSQL server. -For more information, see the relevant: - -- [Omnibus GitLab issue](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/5919). -- [Gitaly issue](https://gitlab.com/gitlab-org/gitaly/-/issues/3398). - -## Is PgBouncer required for the Praefect database? - -No, because the number of connections Praefect makes is low. You can use the same PgBouncer instance -for both the GitLab application database and the Praefect database if you wish. - -## Are there any special considerations for Gitaly Cluster when PostgreSQL is upgraded? - -There are no special requirements. Gitaly Cluster requires PostgreSQL version 11 or later. - -## Praefect database tables are empty? - -These tables are created per the [specific configuration section](praefect.md#postgresql). - -If you find you have an empty Praefect database table, see the -[relevant troubleshooting section](troubleshooting.md#relation-does-not-exist-errors). +This document was moved to [another location](index.md). diff --git a/doc/administration/gitaly/index.md b/doc/administration/gitaly/index.md index f43f9f5d6c1..160313073a5 100644 --- a/doc/administration/gitaly/index.md +++ b/doc/administration/gitaly/index.md @@ -71,7 +71,7 @@ the current status of these issues, please refer to the referenced issues and ep | Issue | Summary | How to avoid | |:--------------------------------------------------------------------------------------|:------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|:--------------------------------| | Gitaly Cluster + Geo - Issues retrying failed syncs | If Gitaly Cluster is used on a Geo secondary site, repositories that have failed to sync could continue to fail when Geo tries to resync them. Recovering from this state requires assistance from support to run manual steps. Work is in-progress to update Gitaly Cluster to [identify repositories with a unique and persistent identifier](https://gitlab.com/gitlab-org/gitaly/-/issues/3485), which is expected to resolve the issue. | No known solution at this time. | -| Praefect unable to insert data into the database due to migrations not being applied after an upgrade | If the database is not kept up to date with completed migrations, then the Praefect node is unable to perform normal operation. | Make sure the Praefect database is up and running with all migrations completed (For example: `/opt/gitlab/embedded/bin/praefect -config /var/opt/gitlab/praefect/config.toml sql-migrate-status` should show a list of all applied migrations). Consider [requesting live upgrade assistance](https://about.gitlab.com/support/scheduling-upgrade-assistance.html) so your upgrade plan can be reviewed by support. | +| Praefect unable to insert data into the database due to migrations not being applied after an upgrade | If the database is not kept up to date with completed migrations, then the Praefect node is unable to perform normal operation. | Make sure the Praefect database is up and running with all migrations completed (For example: `/opt/gitlab/embedded/bin/praefect -config /var/opt/gitlab/praefect/config.toml sql-migrate-status` should show a list of all applied migrations). Consider [requesting live upgrade assistance](https://about.gitlab.com/support/scheduling-upgrade-assistance/) so your upgrade plan can be reviewed by support. | | Restoring a Gitaly Cluster node from a snapshot in a running cluster | Because the Gitaly Cluster runs with consistent state, introducing a single node that is behind will result in the cluster not being able to reconcile the nodes data and other nodes data | Don't restore a single Gitaly Cluster node from a backup snapshot. If you must restore from backup, it's best to snapshot all Gitaly Cluster nodes at the same time and take a database dump of the Praefect database. | ### Snapshot backup and recovery limitations @@ -206,6 +206,29 @@ WARNING: If complete cluster failure occurs, disaster recovery plans should be executed. These can affect the RPO and RTO discussed above. +### Comparison to Geo + +Gitaly Cluster and [Geo](../geo/index.md) both provide redundancy. However the redundancy of: + +- Gitaly Cluster provides fault tolerance for data storage and is invisible to the user. Users are + not aware when Gitaly Cluster is used. +- Geo provides [replication](../geo/index.md) and [disaster recovery](../geo/disaster_recovery/index.md) for + an entire instance of GitLab. Users know when they are using Geo for + [replication](../geo/index.md). Geo [replicates multiple data types](../geo/replication/datatypes.md#limitations-on-replicationverification), + including Git data. + +The following table outlines the major differences between Gitaly Cluster and Geo: + +| Tool | Nodes | Locations | Latency tolerance | Failover | Consistency | Provides redundancy for | +|:---------------|:---------|:----------|:------------------------------------------------------------------------------------------------------|:----------------------------------------------------------------------------|:--------------------------------------|:------------------------| +| Gitaly Cluster | Multiple | Single | [Less than 1 second, ideally single-digit milliseconds](praefect.md#network-latency-and-connectivity) | [Automatic](praefect.md#automatic-failover-and-primary-election-strategies) | [Strong](index.md#strong-consistency) | Data storage in Git | +| Geo | Multiple | Multiple | Up to one minute | [Manual](../geo/disaster_recovery/index.md) | Eventual | Entire GitLab instance | + +For more information, see: + +- Geo [use cases](../geo/index.md#use-cases). +- Geo [architecture](../geo/index.md#architecture). + ### Virtual storage Virtual storage makes it viable to have a single repository storage in GitLab to simplify repository @@ -320,9 +343,9 @@ Use the [`praefect metadata`](troubleshooting.md#view-repository-metadata) subco - The virtual storage and relative path. - The repository ID. -The repository on disk also contains the project path in the Git configuration file. The configuration file can be used to determine -the project's location even if the repository's metadata has been deleted. Follow the -[instructions in hashed storage's documentation](../repository_storage_types.md#from-hashed-path-to-project-name). +The repository on disk also contains the project path in the Git configuration file. The configuration +file can be used to determine the project path even if the repository's metadata has been deleted. +Follow the [instructions in hashed storage's documentation](../repository_storage_types.md#from-hashed-path-to-project-name). #### Atomicity of operations @@ -330,7 +353,7 @@ Gitaly Cluster uses the PostgreSQL metadata store with the storage layout to ens deletion, and move operations. The disk operations can't be atomically applied across multiple storages. However, PostgreSQL guarantees the atomicity of the metadata operations. Gitaly Cluster models the operations in a manner that the failing operations always leave the metadata consistent. The disks may contain stale state even after successful operations. This is expected and the leftover state -won't intefere with future operations but may use up disk space unnecessarily until a clean up is performed. +won't interfere with future operations but may use up disk space unnecessarily until a clean up is performed. There is on-going work on a [background crawler](https://gitlab.com/gitlab-org/gitaly/-/issues/3719) that cleans up the leftover repositories from the storages. @@ -366,7 +389,7 @@ relative path of the repository in the metadata store. ### Moving beyond NFS Engineering support for NFS for Git repositories is deprecated. Technical support is planned to be unavailable starting -November 22, 2022. Please see our [statement of support](https://about.gitlab.com/support/statement-of-support.html#gitaly-and-nfs) +November 22, 2022. Please see our [statement of support](https://about.gitlab.com/support/statement-of-support/#gitaly-and-nfs) for more details. [Network File System (NFS)](https://en.wikipedia.org/wiki/Network_File_System) @@ -495,24 +518,31 @@ For configuration information, see [Configure replication factor](praefect.md#co For more information on configuring Gitaly Cluster, see [Configure Gitaly Cluster](praefect.md). -## Migrating to Gitaly Cluster +### Upgrade Gitaly Cluster + +To upgrade a Gitaly Cluster, follow the documentation for +[zero downtime upgrades](../../update/zero_downtime.md#gitaly-or-gitaly-cluster). -See the [Before deploying Gitaly Cluster](#before-deploying-gitaly-cluster) section before continuing. The basic process -for migrating to Gitaly Cluster involves: +## Migrate to Gitaly Cluster + +WARNING: +Some [known issues](#known-issues) exist in Gitaly Cluster. Review the following information before you continue. + +Before migrating to Gitaly Cluster: + +- Review [Before deploying Gitaly Cluster](#before-deploying-gitaly-cluster). +- Upgrade to the latest possible version of GitLab, to take advantage of improvements and bug fixes. + +To migrate to Gitaly Cluster: 1. Create the required storage. Refer to - [repository storage recommendations](faq.md#what-are-some-repository-storage-recommendations). + [repository storage recommendations](praefect.md#repository-storage-recommendations). 1. Create and configure [Gitaly Cluster](praefect.md). 1. [Move the repositories](../operations/moving_repositories.md#move-repositories). To migrate to Gitaly Cluster, existing repositories stored outside Gitaly Cluster must be moved. There is no automatic migration but the moves can be scheduled with the GitLab API. -WARNING: -Some [known database inconsistency issues](#known-issues) exist in Gitaly Cluster. We recommend you -remain on your current service for now. - -NOTE: -GitLab requires a `default` repository storage to be configured. +Even if you don't use the `default` repository storage, you must ensure it is configured. [Read more about this limitation](configure_gitaly.md#gitlab-requires-a-default-repository-storage). ## Migrate off Gitaly Cluster diff --git a/doc/administration/gitaly/monitoring.md b/doc/administration/gitaly/monitoring.md index 17f94f912ee..a0a2e43569c 100644 --- a/doc/administration/gitaly/monitoring.md +++ b/doc/administration/gitaly/monitoring.md @@ -155,6 +155,7 @@ The following metrics are available from the `/metrics` endpoint: - `gitaly_praefect_node_latency_bucket`, a histogram measuring the latency in Gitaly returning health check information to Praefect. This indicates Praefect connection saturation. Available in GitLab 12.10 and later. +- `gitaly_praefect_connections_total`, the total number of connections to Praefect. [Introduced](https://gitlab.com/gitlab-org/gitaly/-/merge_requests/4220) in GitLab 14.7. To monitor [strong consistency](index.md#strong-consistency), you can use the following Prometheus metrics: diff --git a/doc/administration/gitaly/praefect.md b/doc/administration/gitaly/praefect.md index fb9f3e9c361..f6166d713b1 100644 --- a/doc/administration/gitaly/praefect.md +++ b/doc/administration/gitaly/praefect.md @@ -19,9 +19,6 @@ Configure Gitaly Cluster using either: Smaller GitLab installations may need only [Gitaly itself](index.md). -To upgrade a Gitaly Cluster, follow the documentation for -[zero downtime upgrades](../../update/zero_downtime.md#gitaly-cluster). - ## Requirements The minimum recommended configuration for a Gitaly Cluster requires: @@ -54,7 +51,7 @@ Achieving acceptable latency between Gitaly nodes: are designed for this type of synchronization. Latency of less than 2ms should be sufficient for Gitaly Cluster. If you can't provide low network latencies for replication (for example, between distant locations), consider Geo. For -more information, see [How does Gitaly Cluster compare to Geo](faq.md#how-does-gitaly-cluster-compare-to-geo). +more information, see [Comparison to Geo](index.md#comparison-to-geo). Gitaly Cluster [components](index.md#components) communicate with each other over many routes. Your firewall rules must allow the following for Gitaly Cluster to function properly: @@ -64,6 +61,7 @@ allow the following for Gitaly Cluster to function properly: | GitLab | Praefect load balancer | `2305` | `3305` | | Praefect load balancer | Praefect | `2305` | `3305` | | Praefect | Gitaly | `8075` | `9999` | +| Praefect | GitLab (internal API) | `80` | `443` | | Gitaly | GitLab (internal API) | `80` | `443` | | Gitaly | Praefect load balancer | `2305` | `3305` | | Gitaly | Praefect | `2305` | `3305` | @@ -74,6 +72,16 @@ Gitaly does not directly connect to Praefect. However, requests from Gitaly to t load balancer may still be blocked unless firewalls on the Praefect nodes allow traffic from the Gitaly nodes. +### Praefect database storage + +The requirements are relatively low because the database contains only metadata of: + +- Where repositories are located. +- Some queued work. + +It depends on the number of repositories, but a useful minimum is 5-10 GB, similar to the main +GitLab application database. + ## Setup Instructions If you [installed](https://about.gitlab.com/install/) GitLab using the Omnibus GitLab package @@ -168,6 +176,18 @@ The following options are available: - Use a cloud-managed PostgreSQL service. AWS [Relational Database Service](https://aws.amazon.com/rds/) is recommended. +Setting up PostgreSQL creates empty Praefect tables. For more information, see the +[relevant troubleshooting section](troubleshooting.md#relation-does-not-exist-errors). + +#### Running GitLab and Praefect databases on the same server + +The GitLab application database and the Praefect database can be run on the same server. However, Praefect should have +its own database server when using Omnibus GitLab PostgreSQL. If there is a failover, Praefect isn't aware and starts to +fail as the database it's trying to use would either: + +- Be unavailable. +- In read-only mode. + #### Manual database setup To complete this section you need: @@ -278,8 +298,12 @@ reads distribution caching is enabled by configuration #### Use PgBouncer To reduce PostgreSQL resource consumption, we recommend setting up and configuring -[PgBouncer](https://www.pgbouncer.org/) in front of the PostgreSQL instance. To do -this, you must point Praefect to PgBouncer by setting database parameters on Praefect configuration: +[PgBouncer](https://www.pgbouncer.org/) in front of the PostgreSQL instance. However, PgBouncer isn't required because +Praefect makes a low number of connections. If you choose to use PgBouncer, you can use the same PgBouncer instance for +both the GitLab application database and the Praefect database. + +To configure PgBouncer in front of the PostgreSQL instance, you must point Praefect to PgBouncer by setting database +parameters on Praefect configuration: ```ruby praefect['database_host'] = PGBOUNCER_HOST @@ -475,7 +499,7 @@ Updates to example must be made at: # Some metrics run queries against the database. Enabling separate database metrics allows # these metrics to be collected when the metrics are - # scraped on a separate /db_metrics endpoint. + # scraped on a separate /db_metrics endpoint. praefect['separate_database_metrics'] = true ``` @@ -519,7 +543,7 @@ Updates to example must be made at: WARNING: If you have data on an already existing storage called `default`, you should configure the virtual storage with another name and - [migrate the data to the Gitaly Cluster storage](index.md#migrating-to-gitaly-cluster) + [migrate the data to the Gitaly Cluster storage](index.md#migrate-to-gitaly-cluster) afterwards. Replace `PRAEFECT_INTERNAL_TOKEN` with a strong secret, which is used by @@ -632,12 +656,8 @@ and on all Praefect clients that communicate with it following the procedure des Note the following: -- The certificate must specify the address you use to access the Praefect server. If - addressing the Praefect server by: - - - Hostname, you can either use the Common Name field for this, or add it as a Subject - Alternative Name. - - IP address, you must add it as a Subject Alternative Name to the certificate. +- The certificate must specify the address you use to access the Praefect server. You must add the hostname or IP + address as a Subject Alternative Name to the certificate. - When running Praefect sub-commands such as `dial-nodes` and `list-untracked-repositories` from the command line with [Gitaly TLS enabled](configure_gitaly.md#enable-tls-support), you must set the `SSL_CERT_DIR` or `SSL_CERT_FILE` environment variable so that the Gitaly certificate is trusted. For example: @@ -651,6 +671,8 @@ Note the following: This allows you to do a gradual transition from unencrypted to encrypted traffic, if necessary. + To disable the unencrypted listener, set `praefect['listen_addr'] = nil`. + To configure Praefect with TLS: **For Omnibus GitLab** @@ -992,7 +1014,7 @@ Particular attention should be shown to: WARNING: If you have existing data stored on the default Gitaly storage, - you should [migrate the data your Gitaly Cluster storage](index.md#migrating-to-gitaly-cluster) + you should [migrate the data your Gitaly Cluster storage](index.md#migrate-to-gitaly-cluster) first. ```ruby @@ -1107,7 +1129,7 @@ Particular attention should be shown to: #### Use TCP for existing GitLab instances When adding Gitaly Cluster to an existing Gitaly instance, the existing Gitaly storage -must use a TCP address. If `gitaly_address` is not specified, then a Unix socket is used, +must be listening on TCP/TLS. If `gitaly_address` is not specified, then a Unix socket is used, which prevents the communication with the cluster. For example: @@ -1116,7 +1138,7 @@ For example: git_data_dirs({ 'default' => { 'gitaly_address' => 'tcp://old-gitaly.internal:8075' }, 'cluster' => { - 'gitaly_address' => 'tcp://<PRAEFECT_LOADBALANCER_HOST>:2305', + 'gitaly_address' => 'tls://<PRAEFECT_LOADBALANCER_HOST>:3305', 'gitaly_token' => '<praefect_external_token>' } }) @@ -1223,6 +1245,18 @@ You can configure: If `default_replication_factor` is unset, the repositories are always replicated on every node defined in `virtual_storages`. If a new node is introduced to the virtual storage, both new and existing repositories are replicated to the node automatically. +### Repository storage recommendations + +The size of the required storage can vary between instances and depends on the set +[replication factor](index.md#replication-factor). You might want to include implementing +repository storage redundancy. + +For a replication factor: + +- Of `1`: NFS, Gitaly, and Gitaly Cluster have roughly the same storage requirements. +- More than `1`: The amount of required storage is `used space * replication factor`. `used space` + should include any planned future growth. + ## Repository verification > [Introduced](https://gitlab.com/gitlab-org/gitaly/-/issues/4080) in GitLab 15.0. diff --git a/doc/administration/gitaly/recovery.md b/doc/administration/gitaly/recovery.md index 6de2acf1792..ecd68234b01 100644 --- a/doc/administration/gitaly/recovery.md +++ b/doc/administration/gitaly/recovery.md @@ -359,7 +359,7 @@ Add the `-older-than` option to avoid showing repositories that are the process Praefect database. Replace `<duration>` with a time duration (for example, `5s`, `10m`, or `1h`). Defaults to `6h`. ```shell -sudo /opt/gitlab/embedded/bin/praefect -config /var/opt/gitlab/praefect/config.toml list-untracked-repositories -older-than <duration> +sudo /opt/gitlab/embedded/bin/praefect -config /var/opt/gitlab/praefect/config.toml list-untracked-repositories -older-than <duration> ``` Only repositories with a creation time before the specified duration are considered. diff --git a/doc/administration/gitaly/troubleshooting.md b/doc/administration/gitaly/troubleshooting.md index c79ed1d1707..a1f542ddac8 100644 --- a/doc/administration/gitaly/troubleshooting.md +++ b/doc/administration/gitaly/troubleshooting.md @@ -8,9 +8,6 @@ info: To determine the technical writer assigned to the Stage/Group associated w Refer to the information below when troubleshooting Gitaly and Gitaly Cluster. -Before troubleshooting, see the Gitaly and Gitaly Cluster -[frequently asked questions](faq.md). - ## Troubleshoot Gitaly The following sections provide possible solutions to Gitaly errors. @@ -349,6 +346,14 @@ that do not exist in a repository. "error":"not found: .gitlab-ci.yml" ``` +### Git pushes are slow when Dynatrace is enabled + +Dynatrace can cause the `/opt/gitlab/embedded/bin/gitaly-hooks` reference transaction hook, +to take several seconds to start up and shut down. `gitaly-hooks` is executed twice when users +push, which causes a significant delay. + +If Git pushes are too slow when Dynatrace is enabled, disable Dynatrace. + ## Troubleshoot Praefect (Gitaly Cluster) The following sections provide possible solutions to Gitaly Cluster errors. |