diff options
Diffstat (limited to 'doc/administration/gitaly')
| -rw-r--r-- | doc/administration/gitaly/concurrency_limiting.md | 6 | ||||
| -rw-r--r-- | doc/administration/gitaly/configure_gitaly.md | 35 | ||||
| -rw-r--r-- | doc/administration/gitaly/praefect.md | 4 | ||||
| -rw-r--r-- | doc/administration/gitaly/reference.md | 260 | ||||
| -rw-r--r-- | doc/administration/gitaly/troubleshooting.md | 21 |
5 files changed, 69 insertions, 257 deletions
diff --git a/doc/administration/gitaly/concurrency_limiting.md b/doc/administration/gitaly/concurrency_limiting.md index 321bb9efe20..2b1a2bbef64 100644 --- a/doc/administration/gitaly/concurrency_limiting.md +++ b/doc/administration/gitaly/concurrency_limiting.md @@ -18,7 +18,7 @@ Enabling limits on your environment should be done with caution and only in select circumstances, such as to protect against unexpected traffic. When reached, limits _do_ result in disconnects that negatively impact users. For consistent and stable performance, you should first explore other options such as -adjusting node specifications, and [reviewing large repositories](../../user/project/repository/managing_large_repositories.md) or workloads. +adjusting node specifications, and [reviewing large repositories](../../user/project/repository/monorepos/index.md) or workloads. ## Limit RPC concurrency @@ -30,7 +30,7 @@ When cloning or pulling repositories, various RPCs run in the background. In par These RPCs can consume a large amount of resources, which can have a significant impact in situations such as: - Unexpectedly high traffic. -- Running against [large repositories](../../user/project/repository/managing_large_repositories.md) that don't follow best practices. +- Running against [large repositories](../../user/project/repository/monorepos/index.md) that don't follow best practices. You can limit these processes from overwhelming your Gitaly server in these scenarios using the concurrency limits in the Gitaly configuration file. For example: @@ -94,7 +94,7 @@ number of in-flight pack-object processes per remote IP address. WARNING: Only enable these limits on your environment with caution and only in select circumstances, such as to protect against unexpected traffic. When reached, these limits disconnect users. For consistent and stable performance, you should first explore other options such as adjusting node specifications, and -[reviewing large repositories](../../user/project/repository/managing_large_repositories.md) or workloads. +[reviewing large repositories](../../user/project/repository/monorepos/index.md) or workloads. Example configuration: diff --git a/doc/administration/gitaly/configure_gitaly.md b/doc/administration/gitaly/configure_gitaly.md index d89413b2cf4..eb620ff7413 100644 --- a/doc/administration/gitaly/configure_gitaly.md +++ b/doc/administration/gitaly/configure_gitaly.md @@ -13,7 +13,7 @@ Configure Gitaly in one of two ways: :::TabTitle Linux package (Omnibus) 1. Edit `/etc/gitlab/gitlab.rb` and add or change the - [Gitaly settings](https://gitlab.com/gitlab-org/omnibus-gitlab/blob/1dd07197c7e5ae23626aad5a4a070a800b670380/files/gitlab-config-template/gitlab.rb.template#L1622-1676). + [Gitaly settings](https://gitlab.com/gitlab-org/gitaly/-/blob/master/config.toml.example). 1. Save the file and [reconfigure GitLab](../restart_gitlab.md#reconfigure-a-linux-package-installation). :::TabTitle Self-compiled (source) @@ -574,7 +574,7 @@ Enabling limits on your environment should be done with caution and only in select circumstances, such as to protect against unexpected traffic. When reached, limits _do_ result in disconnects that negatively impact users. For consistent and stable performance, you should first explore other options such as -adjusting node specifications, and [reviewing large repositories](../../user/project/repository/managing_large_repositories.md) or workloads. +adjusting node specifications, and [reviewing large repositories](../../user/project/repository/monorepos/index.md) or workloads. When enabling cgroups for memory, you should ensure that no swap is configured on the Gitaly nodes as processes may switch to using that instead of being terminated. This situation could lead to notably compromised @@ -1019,9 +1019,9 @@ meaning that unique requests do not get written into the cache. If you: - Increase this number, your cache hit rate goes down and the -cache uses less disk space. + cache uses less disk space. - Decrease this number, your cache hit -rate goes up and the cache uses more disk space. + rate goes up and the cache uses more disk space. You should set `min_occurrences` to `1`. On GitLab.com, going from 0 to 1 saved us 50% cache disk space while barely affecting @@ -1089,6 +1089,33 @@ Example: } ``` +## `cat-file` cache + +A lot of Gitaly RPCs need to look up Git objects from repositories. +Most of the time we use `git cat-file --batch` processes for that. For +better performance, Gitaly can re-use these `git cat-file` processes +across RPC calls. Previously used processes are kept around in a +["Git cat-file cache"](https://about.gitlab.com/blog/2019/07/08/git-performance-on-nfs/#enter-cat-file-cache). +To control how much system resources this uses, we have a maximum number of +cat-file processes that can go into the cache. + +The default limit is 100 `cat-file`s, which constitute a pair of +`git cat-file --batch` and `git cat-file --batch-check` processes. If +you see errors about "too many open files", or an +inability to create new processes, you may want to lower this limit. + +Ideally, the number should be large enough to handle standard +traffic. If you raise the limit, you should measure the cache hit ratio +before and after. If the hit ratio does not improve, the higher limit is +probably not making a meaningful difference. Here is an example +Prometheus query to see the hit rate: + +```plaintext +sum(rate(gitaly_catfile_cache_total{type="hit"}[5m])) / sum(rate(gitaly_catfile_cache_total{type=~"(hit)|(miss)"}[5m])) +``` + +Configure the `cat-file` cache in the [Gitaly configuration file](reference.md). + ## Repository consistency checks Gitaly runs repository consistency checks: diff --git a/doc/administration/gitaly/praefect.md b/doc/administration/gitaly/praefect.md index 66e220dacc2..112c441164d 100644 --- a/doc/administration/gitaly/praefect.md +++ b/doc/administration/gitaly/praefect.md @@ -949,8 +949,8 @@ You can also appoint an authoritative name server by setting it in this format: 1. Save the file and [reconfigure](../restart_gitlab.md#reconfigure-a-linux-package-installation). 1. On the Praefect clients (except Gitaly servers), edit `git_data_dirs` in -`/etc/gitlab/gitlab.rb` as follows. Replace `PRAEFECT_SERVICE_DISCOVERY_ADDRESS` -with Praefect service discovery address, such as `praefect.service.consul`. + `/etc/gitlab/gitlab.rb` as follows. Replace `PRAEFECT_SERVICE_DISCOVERY_ADDRESS` + with Praefect service discovery address, such as `praefect.service.consul`. ```ruby git_data_dirs({ diff --git a/doc/administration/gitaly/reference.md b/doc/administration/gitaly/reference.md index 946be6c8af3..169b6e00813 100644 --- a/doc/administration/gitaly/reference.md +++ b/doc/administration/gitaly/reference.md @@ -4,256 +4,24 @@ group: Gitaly info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://handbook.gitlab.com/handbook/product/ux/technical-writing/#assignments --- -# Gitaly reference **(FREE SELF)** +# Example configuration files **(FREE SELF)** -Gitaly is configured via a [TOML](https://github.com/toml-lang/toml) -configuration file. Unlike self-compiled installations, in Linux package installations you -would not edit this file directly. For Linux package installations, the default file location is `/var/opt/gitlab/gitaly/config.toml`. +Gitaly and Gitaly Cluster are configured by using configuration files. The default location of the configuration files +depends on the type of installation you have: -The configuration file is passed as an argument to the `gitaly` executable, which is usually done by either your Linux -package installation or your [init](https://en.wikipedia.org/wiki/Init) script. +- For Linux package installations, the default location for Gitaly and Gitaly Cluster configuration is in the + `/etc/gitlab/gitlab.rb` Ruby file. +- For self-compiled, the default location for Gitaly and Gitaly Cluster configuration is in the + `/home/git/gitaly/config.toml` and `/home/git/gitaly/config.prafect.toml` TOML files. -An [example configuration file](https://gitlab.com/gitlab-org/gitaly/blob/master/config.toml.example) -can be found in the Gitaly project. +You can find example TOML configuration files in the `gitaly` project for: -## Format +- Gitaly: <https://gitlab.com/gitlab-org/gitaly/-/blob/master/config.toml.example> +- Gitaly Cluster: <https://gitlab.com/gitlab-org/gitaly/-/blob/master/config.praefect.toml.example> -At the top level, `config.toml` defines the items described on the table below. +If you are configuring a Linux package installation, you must convert the examples into Ruby to use them. -| Name | Type | Required | Description | -| ---- | ---- | -------- | ----------- | -| `socket_path` | string | yes (if `listen_addr` is not set) | A path which Gitaly should open a Unix socket. | -| `listen_addr` | string | yes (if `socket_path` is not set) | TCP address for Gitaly to listen on. | -| `tls_listen_addr` | string | no | TCP over TLS address for Gitaly to listen on. | -| `bin_dir` | string | yes | Directory containing Gitaly executables. | -| `prometheus_listen_addr` | string | no | TCP listen address for Prometheus metrics. If not set, no Prometheus listener is started. | +For more information on: -For example: - -```toml -socket_path = "/home/git/gitlab/tmp/sockets/private/gitaly.socket" -listen_addr = "localhost:9999" -tls_listen_addr = "localhost:8888" -bin_dir = "/home/git/gitaly" -prometheus_listen_addr = "localhost:9236" -``` - -### Authentication - -Gitaly can be configured to reject requests that do not contain a -specific bearer token in their headers, which is a security measure to -be used when serving requests over TCP: - -```toml -[auth] -# A non-empty token enables authentication. -token = "the secret token" -``` - -Authentication is disabled when the token setting in `config.toml` is absent or -an empty string. - -It is possible to temporarily disable authentication with the `transitioning` -setting. This allows you to monitor if all clients are -authenticating correctly without causing a service outage for clients -that are still to be configured correctly: - -```toml -[auth] -token = "the secret token" -transitioning = true -``` - -WARNING: -Remember to disable `transitioning` when you are done -changing your token settings. - -All authentication attempts are counted in Prometheus under -the [`gitaly_authentications_total` metric](monitoring.md#queries). - -### TLS - -Gitaly supports TLS encryption. You need to bring your own certificates as -this isn't provided automatically. - -| Name | Type | Required | Description | -| ---- | ---- | -------- | ----------- | -| `certificate_path` | string | no | Path to the certificate. | -| `key_path` | string | no | Path to the key. | - -```toml -tls_listen_addr = "localhost:8888" - -[tls] -certificate_path = '/home/git/cert.cert' -key_path = '/home/git/key.pem' -``` - -[Read more](tls_support.md) about TLS in Gitaly. - -### Storage - -GitLab repositories are grouped into directories known as storages, such as -`/home/git/repositories`. They contain bare repositories managed -by GitLab with names, such as `default`. - -These names and paths are also defined in the `gitlab.yml` configuration file of -GitLab. When you run Gitaly on the same machine as GitLab (the default -and recommended configuration) storage paths defined in the Gitaly `config.toml` -must match those in `gitlab.yml`. - -| Name | Type | Required | Description | -| ---- | ---- | -------- | ----------- | -| `storage` | array | yes | An array of storage shards. | -| `path` | string | yes | The path to the storage shard. | -| `name` | string | yes | The name of the storage shard. | - -For example: - -```toml -[[storage]] -path = "/path/to/storage/repositories" -name = "my_shard" - -[[storage]] -path = "/path/to/other/repositories" -name = "other_storage" -``` - -### Git - -The following values can be set in the `[git]` section of the configuration file. - -| Name | Type | Required | Description | -| ---- | ---- | -------- | ----------- | -| `bin_path` | string | no | Path to Git binary. If not set, is resolved using `PATH`. | -| `catfile_cache_size` | integer | no | Maximum number of cached [cat-file processes](#cat-file-cache). Default is `100`. | -| `signing_key` | string | no | Path to [GPG signing key](configure_gitaly.md#configure-commit-signing-for-gitlab-ui-commits). If not set, Gitaly doesn't sign commits made using the UI. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/19185) in GitLab 15.4. | - -#### `cat-file` cache - -A lot of Gitaly RPCs need to look up Git objects from repositories. -Most of the time we use `git cat-file --batch` processes for that. For -better performance, Gitaly can re-use these `git cat-file` processes -across RPC calls. Previously used processes are kept around in a -["Git cat-file cache"](https://about.gitlab.com/blog/2019/07/08/git-performance-on-nfs/#enter-cat-file-cache). -To control how much system resources this uses, we have a maximum number of -cat-file processes that can go into the cache. - -The default limit is 100 `cat-file`s, which constitute a pair of -`git cat-file --batch` and `git cat-file --batch-check` processes. If -you are seeing errors complaining about "too many open files", or an -inability to create new processes, you may want to lower this limit. - -Ideally, the number should be large enough to handle standard -traffic. If you raise the limit, you should measure the cache hit ratio -before and after. If the hit ratio does not improve, the higher limit is -probably not making a meaningful difference. Here is an example -Prometheus query to see the hit rate: - -```plaintext -sum(rate(gitaly_catfile_cache_total{type="hit"}[5m])) / sum(rate(gitaly_catfile_cache_total{type=~"(hit)|(miss)"}[5m])) -``` - -#### Custom Hooks - -> Method of configuring custom hooks directory documented here is preferred from GitLab 16.4. `[gitlab-shell] dir` configuration is no longer required. - -Gitaly supports [custom Git hooks](../server_hooks.md) that are -used to perform tasks based on changes performed in any repository. The custom -hooks directory is configured in the `[hooks]` section: - -| Name | Type | Required | Description | -| ---- | ---- | -------- | ----------- | -| `custom_hooks_dir` | string | no | The directory where custom Git hooks are installed. | - -Example: - -```toml -[hooks] -custom_hooks_dir = "/home/git/custom-hooks" -``` - -If left unset, no custom hooks are used. - -### GitLab - -> Method of configuring custom hooks directory documented here is preferred from GitLab 16.4. `[gitlab-shell] dir` configuration is no longer required. - -Gitaly must connect to the GitLab application to perform access -checks when a user performs a change. The parameters to connect to GitLab must -be configured in the `[gitlab]` section. - -| Name | Type | Required | Description | -| ---- | ---- | -------- | ----------- | -| `url` | string | yes | The URL of the GitLab server. -| `secret` | string | no | The secret token used to authenticate with GitLab. | -| `secret_file` | string | no | The path of the file containing the secret token used to authenticate with GitLab. | - -Only one of `secret` or `secret_file` can be configured. - -### Prometheus - -You can optionally configure Gitaly to record histogram latencies on GRPC method -calls in Prometheus. - -| Name | Type | Required | Description | -| ---- | ---- | -------- | ----------- | -| `grpc_latency_buckets` | array | no | Prometheus stores each observation in a bucket, which means you'd get an approximation of latency. Optimizing the buckets gives more control over the accuracy of the approximation. | - -Example: - -```toml -prometheus_listen_addr = "localhost:9236" - -[prometheus] -grpc_latency_buckets = [0.001, 0.005, 0.025, 0.1, 0.5, 1.0, 10.0, 30.0, 60.0, 300.0, 1500.0] -``` - -### Logging - -The following values configure logging in Gitaly under the `[logging]` section. - -| Name | Type | Required | Description | -| ---- | ---- | -------- | ----------- | -| `format` | string | no | Log format: `text` or `json`. Default: `text`. | -| `level` | string | no | Log level: `debug`, `info`, `warn`, `error`, `fatal`, or `panic`. Default: `info`. | -| `sentry_dsn` | string | no | Sentry DSN (Data Source Name) for exception monitoring. | -| `sentry_environment` | string | no | [Sentry Environment](https://docs.sentry.io/product/sentry-basics/environments/) for exception monitoring. | - -While the main Gitaly application logs go to `stdout`, there are some extra log -files that go to a configured directory, like the GitLab Shell logs. -GitLab Shell does not support `panic` or `trace` level logs: - -- `panic` falls back to `error`. -- `trace` falls back to `debug`. -- Any other invalid log levels default to `info`. - -Example: - -```toml -[logging] -level = "warn" -dir = "/home/gitaly/logs" -format = "json" -sentry_dsn = "https://<key>:<secret>@sentry.io/<project>" -ruby_sentry_dsn = "https://<key>:<secret>@sentry.io/<project>" -``` - -## Concurrency - -You can adjust the `concurrency` of each RPC endpoint. - -| Name | Type | Required | Description | -| ---- | ---- | -------- | ----------- | -| `concurrency` | array | yes | An array of RPC endpoints. | -| `rpc` | string | no | The name of the RPC endpoint (`/gitaly.RepositoryService/GarbageCollect`). | -| `max_per_repo` | integer | no | Concurrency per RPC per repository. | - -Example: - -```toml -[[concurrency]] -rpc = "/gitaly.RepositoryService/GarbageCollect" -max_per_repo = 1 -``` +- Configuring Gitaly, see [Configure Gitaly](configure_gitaly.md). +- Configuring Gitaly Cluster, see [Configure Gitaly Cluster](praefect.md). diff --git a/doc/administration/gitaly/troubleshooting.md b/doc/administration/gitaly/troubleshooting.md index 7d0255a3efb..6c355f79191 100644 --- a/doc/administration/gitaly/troubleshooting.md +++ b/doc/administration/gitaly/troubleshooting.md @@ -224,7 +224,7 @@ the application might be fetching this secret from a different file. Your Gitaly If that setting is missing, GitLab defaults to using `.gitlab_shell_secret` under `/opt/gitlab/embedded/service/gitlab-rails/.gitlab_shell_secret`. -### Repository pushes fail +### Repository pushes fail with `401 Unauthorized` and `JWT::VerificationError` When attempting `git push`, you can see: @@ -240,7 +240,7 @@ When attempting `git push`, you can see: } ``` -This error occurs when the GitLab server has been upgraded to GitLab 15.5 or later but Gitaly has not yet been upgraded. +This combination of errors occurs when the GitLab server has been upgraded to GitLab 15.5 or later but Gitaly has not yet been upgraded. From GitLab 15.5, GitLab [authenticates with GitLab Shell using a JWT token instead of a shared secret](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/86148). You should follow the [recommendations on upgrading external Gitaly](../../update/plan_your_upgrade.md#external-gitaly) and upgrade Gitaly before the GitLab @@ -441,6 +441,23 @@ To resolve this, remove the `noexec` option from the file system mount. An alter Because Gitaly commit signing is headless and not associated with a specific user, the GPG signing key must be created without a passphrase, or the passphrase must be removed before export. +### Gitaly logs show errors in `info` messages + +Because of a bug [introduced](https://gitlab.com/gitlab-org/gitaly/-/merge_requests/6201) in GitLab 16.3, additional entries were written to the +[Gitaly logs](../logs/index.md#gitaly-logs). These log entries contained `"level":"info"` but the `msg` string appeared to contain an error. + +For example: + +```json +{"level":"info","msg":"[core] [Server #3] grpc: Server.Serve failed to create ServerTransport: connection error: desc = \"ServerHandshake(\\\"x.x.x.x:x\\\") failed: wrapped server handshake: EOF\"","pid":6145,"system":"system","time":"2023-12-14T21:20:39.999Z"} +``` + +The reason for this log entry is that the underlying gRPC library sometimes output verbose transportation logs. These log entries appear to be errors but are, in general, +safe to ignore. + +This bug was [fixed](https://gitlab.com/gitlab-org/gitaly/-/merge_requests/6513/) in GitLab 16.4.5, 16.5.5, and 16.6.0, which prevents these types of messages from +being written to the Gitaly logs. + ## Troubleshoot Praefect (Gitaly Cluster) The following sections provide possible solutions to Gitaly Cluster errors. |