diff options
Diffstat (limited to 'doc/administration/gitaly')
| -rw-r--r-- | doc/administration/gitaly/configure_gitaly.md | 166 | ||||
| -rw-r--r-- | doc/administration/gitaly/img/gitaly_adaptive_concurrency_limit.png | bin | 0 -> 36052 bytes | |||
| -rw-r--r-- | doc/administration/gitaly/index.md | 86 | ||||
| -rw-r--r-- | doc/administration/gitaly/monitoring.md | 41 | ||||
| -rw-r--r-- | doc/administration/gitaly/recovery.md | 54 | ||||
| -rw-r--r-- | doc/administration/gitaly/troubleshooting.md | 37 |
6 files changed, 274 insertions, 110 deletions
diff --git a/doc/administration/gitaly/configure_gitaly.md b/doc/administration/gitaly/configure_gitaly.md index f62f0a5a4e2..15ace9c4ed9 100644 --- a/doc/administration/gitaly/configure_gitaly.md +++ b/doc/administration/gitaly/configure_gitaly.md @@ -27,6 +27,7 @@ The following configuration options are also available: - Enabling [TLS support](#enable-tls-support). - Limiting [RPC concurrency](#limit-rpc-concurrency). +- Limiting [pack-objects concurrency](#limit-pack-objects-concurrency). ## About the Gitaly token @@ -361,7 +362,7 @@ Configure Gitaly server in one of two ways: WARNING: If directly copying repository data from a GitLab server to Gitaly, ensure that the metadata file, default path `/var/opt/gitlab/git-data/repositories/.gitaly-metadata`, is not included in the transfer. -Copying this file causes GitLab to use the [Rugged patches](index.md#direct-access-to-git-in-gitlab) for repositories hosted on the Gitaly server, +Copying this file causes GitLab to use the direct disk access to repositories hosted on the Gitaly server, leading to `Error creating pipeline` and `Commit not found` errors, or stale data. ### Configure Gitaly clients @@ -665,6 +666,8 @@ Configure Gitaly with TLS in one of two ways: ``` 1. Save the file and [reconfigure GitLab](../restart_gitlab.md#reconfigure-a-linux-package-installation). +1. Run `sudo gitlab-rake gitlab:gitaly:check` on the Gitaly client (for example, the + Rails application) to confirm it can connect to Gitaly servers. 1. Verify Gitaly traffic is being served over TLS by [observing the types of Gitaly connections](#observe-type-of-gitaly-connections). 1. Optional. Improve security by: @@ -751,6 +754,43 @@ Configure Gitaly with TLS in one of two ways: ::EndTabs +#### Update the certificates + +To update the Gitaly certificates after initial configuration: + +::Tabs + +:::TabTitle Linux package (Omnibus) + +If the content of your SSL certificates under the `/etc/gitlab/ssl` directory have been updated, but no configuration changes have been made to +`/etc/gitlab/gitlab.rb`, then reconfiguring GitLab doesn’t affect Gitaly. Instead, you must restart Gitaly manually for the certificates to be loaded +by the Gitaly process: + +```shell +sudo gitlab-ctl restart gitaly +``` + +If you change or update the certificates in `/etc/gitlab/trusted-certs` without making changes to the `/etc/gitlab/gitlab.rb` file, you must: + +1. [Reconfigure GitLab](../restart_gitlab.md#reconfigure-a-linux-package-installation) so the symlinks for the trusted certificates are updated. +1. Restart Gitaly manually for the certificates to be loaded by the Gitaly process: + + ```shell + sudo gitlab-ctl restart gitaly + ``` + +:::TabTitle Self-compiled (source) + +If the content of your SSL certificates under the `/etc/gitlab/ssl` directory have been updated, you must +[restart GitLab](../restart_gitlab.md#self-compiled-installations) for the certificates to be loaded by the Gitaly process. + +If you change or update the certificates in `/usr/local/share/ca-certificates`, you must: + +1. Run `sudo update-ca-certificates` to update the system's trusted store. +1. [Restart GitLab](../restart_gitlab.md#self-compiled-installations) for the certificates to be loaded by the Gitaly process. + +::EndTabs + ### Observe type of Gitaly connections For information on observing the type of Gitaly connections being served, see the @@ -866,6 +906,126 @@ When the pack-object cache is enabled, pack-objects limiting kicks in only if th You can observe the behavior of this queue using Gitaly logs and Prometheus. For more information, see [Monitor Gitaly pack-objects concurrency limiting](monitoring.md#monitor-gitaly-pack-objects-concurrency-limiting). +## Adaptive concurrency limiting + +> [Introduced](https://gitlab.com/groups/gitlab-org/-/epics/10734) in GitLab 16.6. + +Gitaly supports two concurrency limits: + +- An [RPC concurrency limit](#limit-rpc-concurrency), which allow you to configure a maximum number of simultaneous in-flight requests for each + Gitaly RPC. The limit is scoped by RPC and repository. +- A [Pack-objects concurrency limit](#limit-pack-objects-concurrency), which restricts the number of concurrent Git data transfer request by IP. + +If this limit is exceeded, either: + +- The request is put in a queue. +- The request is rejected if the queue is full or if the request remains in the queue for too long. + +Both of these concurrency limits can be configured statically. Though static limits can yield good protection results, they have some drawbacks: + +- Static limits are not good for all usage patterns. There is no one-size-fits-all value. If the limit is too low, big repositories are + negatively impacted. If the limit is too high, the protection is essentially lost. +- It's tedious to maintain a sane value for the concurrency limit, especially when the workload of each repository changes over time. +- A request can be rejected even though the server is idle because the rate doesn't factor in the load on the server. + +You can overcome all of these drawbacks and keep the benefits of concurrency limiting by configuring adaptive concurrency limits. Adaptive +concurrency limits are optional and build on the two concurrency limiting types. It uses Additive Increase/Multiplicative Decrease (AIMD) +algorithm. Each adaptive limit: + +- Gradually increases up to a certain upper limit during typical process functioning. +- Quickly decreases when the host machine has a resource problem. + +This mechanism provides some headroom for the machine to "breathe" and speeds up current inflight requests. + + + +The adaptive limiter calibrates the limits every 30 seconds and: + +- Increases the limits by one until reaching the upper limit. +- Decreases the limits by half when the top-level cgroup has either memory usage that exceeds 90%, excluding highly-evictable page caches, + or CPU throttled for 50% or more of the observation time. + +Otherwise, the limits increase by one until reaching the upper bound. For more information about technical implementation +of this system, please refer to [this blueprint](../../architecture/blueprints/gitaly_adaptive_concurrency_limit/index.md). + +Adaptive limiting is enabled for each RPC or pack-objects cache individually. However, limits are calibrated at the same time. + +### Enable adaptiveness for RPC concurrency + +Prerequisites: + +- Because adaptive limiting depends on [control groups](#control-groups), control groups must be enabled before using adaptive limiting. + +The following is an example to configure an adaptive limit for RPC concurrency: + +```ruby +# in /etc/gitlab/gitlab.rb +gitaly['configuration'] = { + # ... + concurrency: [ + { + rpc: '/gitaly.SmartHTTPService/PostUploadPackWithSidechannel', + max_queue_wait: '1s', + max_queue_size: 10, + adaptive: true, + min_limit: 10, + initial_limit: 20, + max_limit: 40 + }, + { + rpc: '/gitaly.SSHService/SSHUploadPackWithSidechannel', + max_queue_wait: '10s', + max_queue_size: 20, + adaptive: true, + min_limit: 10, + initial_limit: 50, + max_limit: 100 + }, + ], +} +``` + +In this example: + +- `adaptive` sets whether the adaptiveness is enabled. If set, the `max_per_repo` value is ignored in favor of the following configuration. +- `initial_limit` is the per-repository concurrency limit to use when Gitaly starts. +- `max_limit` is the minimum per-repository concurrency limit of the configured RPC. Gitaly increases the current limit + until it reaches this number. +- `min_limit` is the is the minimum per-repository concurrency limit of the configured RPC. When the host machine has a resource problem, + Gitaly quickly reduces the limit until reaching this value. + +For more information, see [RPC concurrency](#limit-rpc-concurrency). + +### Enable adaptiveness for pack-objects concurrency + +Prerequisites: + +- Because adaptive limiting depends on [control groups](#control-groups), control groups must be enabled before using adaptive limiting. + +The following is an example to configure an adaptive limit for pack-objects concurrency: + +```ruby +# in /etc/gitlab/gitlab.rb +gitaly['pack_objects_limiting'] = { + 'max_queue_length' => 200, + 'max_queue_wait' => '60s', + 'adaptive' => true, + 'min_limit' => 10, + 'initial_limit' => 20, + 'max_limit' => 40 +} +``` + +In this example: + +- `adaptive` sets whether the adaptiveness is enabled. If set, the value of `max_concurrency` is ignored in favor of the following configuration. +- `initial_limit` is the per-IP concurrency limit to use when Gitaly starts. +- `max_limit` is the minimum per-IP concurrency limit for pack-objects. Gitaly increases the current limit until it reaches this number. +- `min_limit` is the is the minimum per-IP concurrency limit for pack-objects. When the host machine has a resources problem, Gitaly quickly + reduces the limit until it reaches this value. + +For more information, see [pack-objects concurrency](#limit-pack-objects-concurrency). + ## Control groups WARNING: @@ -1673,7 +1833,9 @@ Gitaly fails to start up if either: ## Configure server-side backups -> [Introduced](https://gitlab.com/gitlab-org/gitaly/-/issues/4941) in GitLab 16.3. +> - [Introduced](https://gitlab.com/gitlab-org/gitaly/-/issues/4941) in GitLab 16.3. +> - Server-side support for restoring a specified backup instead of the latest backup [introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/132188) in GitLab 16.6. +> - Server-side support for creating incremental backups [introduced](https://gitlab.com/gitlab-org/gitaly/-/merge_requests/6475) in GitLab 16.6. Repository backups can be configured so that the Gitaly node that hosts each repository is responsible for creating the backup and streaming it to diff --git a/doc/administration/gitaly/img/gitaly_adaptive_concurrency_limit.png b/doc/administration/gitaly/img/gitaly_adaptive_concurrency_limit.png Binary files differnew file mode 100644 index 00000000000..ce6bb1a8dfc --- /dev/null +++ b/doc/administration/gitaly/img/gitaly_adaptive_concurrency_limit.png diff --git a/doc/administration/gitaly/index.md b/doc/administration/gitaly/index.md index 46f6a5829c8..6784ff4d970 100644 --- a/doc/administration/gitaly/index.md +++ b/doc/administration/gitaly/index.md @@ -587,92 +587,6 @@ off Gitaly Cluster to a sharded Gitaly instance: 1. [Move the repositories](../operations/moving_repositories.md#moving-repositories) to the newly created storage. You can move them by shard or by group, which gives you the opportunity to spread them over multiple Gitaly servers. -## Direct access to Git in GitLab - -Direct access to Git uses code in GitLab known as the "Rugged patches". - -Before Gitaly existed, what are now Gitaly clients accessed Git repositories directly, either: - -- On a local disk in the case of a single-machine Linux package installation. -- Using NFS in the case of a horizontally-scaled GitLab installation. - -In addition to running plain `git` commands, GitLab used a Ruby library called -[Rugged](https://github.com/libgit2/rugged). Rugged is a wrapper around -[libgit2](https://libgit2.org/), a stand-alone implementation of Git in the form of a C library. - -Over time it became clear that Rugged, particularly in combination with -[Unicorn](https://yhbt.net/unicorn/), is extremely efficient. Because `libgit2` is a library and -not an external process, there was very little overhead between: - -- GitLab application code that tried to look up data in Git repositories. -- The Git implementation itself. - -Because the combination of Rugged and Unicorn was so efficient, the GitLab application code ended up -with lots of duplicate Git object lookups. For example, looking up the default branch commit a dozen -times in one request. We could write inefficient code without poor performance. - -When we migrated these Git lookups to Gitaly calls, we suddenly had a much higher fixed cost per Git -lookup. Even when Gitaly is able to re-use an already-running `git` process (for example, to look up -a commit), you still have: - -- The cost of a network roundtrip to Gitaly. -- Inside Gitaly, a write/read roundtrip on the Unix pipes that connect Gitaly to the `git` process. - -Using GitLab.com to measure, we reduced the number of Gitaly calls per request until we no longer felt -the efficiency loss of losing Rugged. It also helped that we run Gitaly itself directly on the Git -file servers, rather than by using NFS mounts. This gave us a speed boost that counteracted the -negative effect of not using Rugged anymore. - -Unfortunately, other deployments of GitLab could not remove NFS like we did on GitLab.com, and they -got the worst of both worlds: - -- The slowness of NFS. -- The increased inherent overhead of Gitaly. - -The code removed from GitLab during the Gitaly migration project affected these deployments. As a -performance workaround for these NFS-based deployments, we re-introduced some of the old Rugged -code. This re-introduced code is informally referred to as the "Rugged patches". - -### Automatic detection - -> Automatic detection for Rugged [disabled](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/95445) in GitLab 15.3. - -FLAG: -On self-managed GitLab, by default automatic detection of whether Rugged should be used (per storage) is not available. -To make it available, an administrator can [disable the feature flag](../../administration/feature_flags.md) named -`skip_rugged_auto_detect`. - -The Ruby methods that perform direct Git access are behind -[feature flags](../../development/gitaly.md#legacy-rugged-code), disabled by default. It wasn't -convenient to set feature flags to get the best performance, so we added an automatic mechanism that -enables direct Git access. - -When GitLab calls a function that has a "Rugged patch", it performs two checks: - -- Is the feature flag for this patch set in the database? If so, the feature flag setting controls - the GitLab use of "Rugged patch" code. -- If the feature flag is not set, GitLab tries accessing the file system underneath the - Gitaly server directly. If it can, it uses the "Rugged patch": - - If using Puma and [thread count](../../install/requirements.md#puma-threads) is set - to `1`. - -The result of these checks is cached. - -To see if GitLab can access the repository file system directly, we use the following heuristic: - -- Gitaly ensures that the file system has a metadata file in its root with a UUID in it. -- Gitaly reports this UUID to GitLab by using the `ServerInfo` RPC. -- GitLab Rails tries to read the metadata file directly. If it exists, and if the UUIDs match, - assume we have direct access. - -Direct Git access is: - -- [Disabled](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/95445) by default in GitLab 15.3 and later for - compatibility with [Praefect-generated replica paths](#praefect-generated-replica-paths-gitlab-150-and-later). It - can be enabled if Rugged [feature flags](../../development/gitaly.md#legacy-rugged-code) are enabled. -- Enabled by default in GitLab 15.2 and earlier because it fills in the correct repository paths in the GitLab - configuration file `config/gitlab.yml`. This satisfies the UUID check. - ### Transition to Gitaly Cluster For the sake of removing complexity, we must remove direct Git access in GitLab. However, we can't diff --git a/doc/administration/gitaly/monitoring.md b/doc/administration/gitaly/monitoring.md index cbf5722f2c5..5d8de42666b 100644 --- a/doc/administration/gitaly/monitoring.md +++ b/doc/administration/gitaly/monitoring.md @@ -90,6 +90,47 @@ In Prometheus, look for the following metrics: - `gitaly_pack_objects_queued` indicates how many requests for pack-objects processes are waiting due to the concurrency limit being reached. - `gitaly_pack_objects_acquiring_seconds` indicates how long a request for a pack-object process has to wait due to concurrency limits before being processed. +## Monitor Gitaly adaptive concurrency limiting + +> [Introduced](https://gitlab.com/groups/gitlab-org/-/epics/10734) in GitLab 16.6. + +You can observe specific behavior of [adaptive concurrency limiting](configure_gitaly.md#adaptive-concurrency-limiting) using Gitaly logs and Prometheus. + +In the [Gitaly logs](../logs/index.md#gitaly-logs), you can identify logs related to the adaptive concurrency limiting when the current limits are adjusted. +You can filter the content of the logs (`msg`) for "Multiplicative decrease" and "Additive increase" messages. + +| Log Field | Description | +|:---|:---| +| `limit` | The name of the limit being adjusted. | +| `previous_limit` | The previous limit before it was increased or decreased. | +| `new_limit` | The new limit after it was increased or decreased. | +| `watcher` | The resource watcher that decided the node is under pressure. For example: `CgroupCpu` or `CgroupMemory`. | +| `reason` | The reason behind limit adjustment. | +| `stats.*` | Some statistics behind an adjustment decision. They are for debugging purposes. | + +Example log: + +```json +{ + "msg": "Multiplicative decrease", + "limit": "pack-objects", + "new_limit": 14, + "previous_limit": 29, + "reason": "cgroup CPU throttled too much", + "watcher": "CgroupCpu", + "stats.time_diff": 15.0, + "stats.throttled_duration": 13.0, + "stat.sthrottled_threshold": 0.5 +} +``` + +In Prometheus, look for the following metrics: + +- `gitaly_concurrency_limiting_current_limit` The current limit value of an adaptive concurrency limit. +- `gitaly_concurrency_limiting_watcher_errors_total` indicates the total number of watcher errors while fetching resource metrics. +- `gitaly_concurrency_limiting_backoff_events_total` indicates the total number of backoff events, which are when the limits being + adjusted due to resource pressure. + ## Monitor Gitaly cgroups You can observe the status of [control groups (cgroups)](configure_gitaly.md#control-groups) using Prometheus: diff --git a/doc/administration/gitaly/recovery.md b/doc/administration/gitaly/recovery.md index 45bde083a1a..6779823c941 100644 --- a/doc/administration/gitaly/recovery.md +++ b/doc/administration/gitaly/recovery.md @@ -15,12 +15,17 @@ You can add and replace Gitaly nodes on a Gitaly Cluster. ### Add new Gitaly nodes -To add a new Gitaly node to a Gitaly Cluster that has [replication factor](praefect.md#configure-replication-factor): +The steps to add a new Gitaly node to a Gitaly Cluster depend on whether a [custom replication factor](praefect.md#configure-replication-factor) is set. -- Set, set the [replication factor](praefect.md#configure-replication-factor) for each repository using `set-replication-factor` Praefect command. New repositories are - replicated based on [replication factor](praefect.md#configure-replication-factor). Praefect doesn't automatically replicate existing repositories to the new Gitaly node. -- Not set, add the new node in your [Praefect configuration](praefect.md#praefect) under `praefect['virtual_storages']`. Praefect automatically replicates all data to any - new Gitaly node added to the configuration. +#### Custom replication factor + +If a custom replication factor is set, set the [replication factor](praefect.md#configure-replication-factor) for each repository using the +`set-replication-factor` Praefect command. New repositories are replicated based on the [replication factor](praefect.md#configure-replication-factor). Praefect doesn't automatically replicate existing repositories to the new Gitaly node. + +#### Default replication factor + +If the default replication factor is used, add the new node in your [Praefect configuration](praefect.md#praefect) under `praefect['virtual_storages']`. +Praefect automatically replicates all data to any new Gitaly node added to the configuration. ### Replace an existing Gitaly node @@ -33,32 +38,37 @@ To use the same name for the replacement node, use [repository verifier](praefec #### With a node with a different name -To use a different name for the replacement node for a Gitaly Cluster that has [replication factor](praefect.md#configure-replication-factor): +The steps use a different name for the replacement node for a Gitaly Cluster depend on if a [custom replication factor](praefect.md#configure-replication-factor) +is set. -- Set, use [`praefect set-replication-factor`](praefect.md#configure-replication-factor) to set the replication factor per repository again to get new storage assigned. - For example: +##### Custom replication factor set - ```shell - $ sudo /opt/gitlab/embedded/bin/praefect -config /var/opt/gitlab/praefect/config.toml set-replication-factor -virtual-storage default -relative-path @hashed/3f/db/3fdba35f04dc8c462986c992bcf875546257113072a909c162f7e470e581e278.git -replication-factor 2 +If a custom replication factor is set, use [`praefect set-replication-factor`](praefect.md#configure-replication-factor) to set the replication factor per repository again to get new storage assigned. For example: - current assignments: gitaly-1, gitaly-2 - ``` +```shell +$ sudo /opt/gitlab/embedded/bin/praefect -config /var/opt/gitlab/praefect/config.toml set-replication-factor -virtual-storage default -relative-path @hashed/3f/db/3fdba35f04dc8c462986c992bcf875546257113072a909c162f7e470e581e278.git -replication-factor 2 + +current assignments: gitaly-1, gitaly-2 +``` + +To reassign all repositories from the old storage to the new one, after configuring the new Gitaly node: - To reassign all repositories from the old storage to the new one, after configuring the new Gitaly node: +1. Connect to Praefect database: - 1. Connect to Praefect database: + ```shell + /opt/gitlab/embedded/bin/psql -h <psql host> -U <user> -d <database name> + ``` - ```shell - /opt/gitlab/embedded/bin/psql -h <psql host> -U <user> -d <database name> - ``` +1. Update the `repository_assignments` table to replace the old Gitaly node name (for example, `old-gitaly`) with the new Gitaly node name + (for example, `new-gitaly`): - 1. Update `repository_assignments` table to replace the old Gitaly node name (for example, `old-gitaly`) with the new Gitaly node name (for example, `new-gitaly`): + ```sql + UPDATE repository_assignments SET storage='new-gitaly' WHERE storage='old-gitaly'; + ``` - ```sql - UPDATE repository_assignments SET storage='new-gitaly' WHERE storage='old-gitaly'; - ``` +##### Default replication factor -- Not set, replace the node in the configuration. The old node's state remains in the Praefect database but it is ignored. +If the default replication factor is used, replace the node in the configuration. The old node's state remains in the Praefect database but it is ignored. ## Primary node failure diff --git a/doc/administration/gitaly/troubleshooting.md b/doc/administration/gitaly/troubleshooting.md index 556bc29b76f..17687cbb181 100644 --- a/doc/administration/gitaly/troubleshooting.md +++ b/doc/administration/gitaly/troubleshooting.md @@ -387,6 +387,43 @@ If Git pushes are too slow when Dynatrace is enabled, disable Dynatrace. One way to resolve this is to make sure the entry is correct for the GitLab internal API URL configured in `gitlab.rb` with `gitlab_rails['internal_api_url']`. +### Changes (diffs) don't load for new merge requests when using Gitaly TLS + +After enabling [Gitaly with TLS](configure_gitaly.md#enable-tls-support), changes (diffs) for new merge requests are not generated +and you see the following message in GitLab: + +```plaintext +Building your merge request... This page will update when the build is complete +``` + +Gitaly must be able to connect to itself to complete some operations. If the Gitaly certificate is not trusted by the Gitaly server, +merge request diffs can't be generated. + +If Gitaly can't connect to itself, you see messages in the [Gitaly logs](../../administration/logs/index.md#gitaly-logs) like the following messages: + +```json +{ + "level":"warning", + "msg":"[core] [Channel #16 SubChannel #17] grpc: addrConn.createTransport failed to connect to {Addr: \"ext-gitaly.example.com:9999\", ServerName: \"ext-gitaly.example.com:9999\", }. Err: connection error: desc = \"transport: authentication handshake failed: tls: failed to verify certificate: x509: certificate signed by unknown authority\"", + "pid":820, + "system":"system", + "time":"2023-11-06T05:40:04.169Z" +} +{ + "level":"info", + "msg":"[core] [Server #3] grpc: Server.Serve failed to create ServerTransport: connection error: desc = \"ServerHandshake(\\\"x.x.x.x:x\\\") failed: wrapped server handshake: remote error: tls: bad certificate\"", + "pid":820, + "system":"system", + "time":"2023-11-06T05:40:04.169Z" +} +``` + +To resolve the problem, ensure that you have added your Gitaly certificate to the `/etc/gitlab/trusted-certs` folder on the Gitaly server +and: + +1. [Reconfigure GitLab](../restart_gitlab.md#reconfigure-a-linux-package-installation) so the certificates are symlinked +1. Restart Gitaly manually `sudo gitlab-ctl restart gitaly` for the certificates to be loaded by the Gitaly process. + ## Gitaly fails to fork processes stored on `noexec` file systems Because of changes [introduced](https://gitlab.com/gitlab-org/omnibus-gitlab/-/merge_requests/5999) in GitLab 14.10, applying the `noexec` option to a mount |
