Add latest changes from gitlab-org/gitlab@master

2 files changed, 42 insertions, 247 deletions

diff --git a/doc/administration/gitaly/configure_gitaly.md b/doc/administration/gitaly/configure_gitaly.md
index d89413b2cf4..829455283b0 100644
--- a/doc/administration/gitaly/configure_gitaly.md
+++ b/doc/administration/gitaly/configure_gitaly.md
@@ -13,7 +13,7 @@ Configure Gitaly in one of two ways:
 :::TabTitle Linux package (Omnibus)
 
 1. Edit `/etc/gitlab/gitlab.rb` and add or change the
-   [Gitaly settings](https://gitlab.com/gitlab-org/omnibus-gitlab/blob/1dd07197c7e5ae23626aad5a4a070a800b670380/files/gitlab-config-template/gitlab.rb.template#L1622-1676).
+   [Gitaly settings](https://gitlab.com/gitlab-org/gitaly/-/blob/master/config.toml.example).
 1. Save the file and [reconfigure GitLab](../restart_gitlab.md#reconfigure-a-linux-package-installation).
 
 :::TabTitle Self-compiled (source)
@@ -1089,6 +1089,33 @@ Example:
 }
 ```
 
+## `cat-file` cache
+
+A lot of Gitaly RPCs need to look up Git objects from repositories.
+Most of the time we use `git cat-file --batch` processes for that. For
+better performance, Gitaly can re-use these `git cat-file` processes
+across RPC calls. Previously used processes are kept around in a
+["Git cat-file cache"](https://about.gitlab.com/blog/2019/07/08/git-performance-on-nfs/#enter-cat-file-cache).
+To control how much system resources this uses, we have a maximum number of
+cat-file processes that can go into the cache.
+
+The default limit is 100 `cat-file`s, which constitute a pair of
+`git cat-file --batch` and `git cat-file --batch-check` processes. If
+you see errors about "too many open files", or an
+inability to create new processes, you may want to lower this limit.
+
+Ideally, the number should be large enough to handle standard
+traffic. If you raise the limit, you should measure the cache hit ratio
+before and after. If the hit ratio does not improve, the higher limit is
+probably not making a meaningful difference. Here is an example
+Prometheus query to see the hit rate:
+
+```plaintext
+sum(rate(gitaly_catfile_cache_total{type="hit"}[5m])) / sum(rate(gitaly_catfile_cache_total{type=~"(hit)|(miss)"}[5m]))
+```
+
+Configure the `cat-file` cache in the [Gitaly configuration file](reference.md).
+
 ## Repository consistency checks
 
 Gitaly runs repository consistency checks:
diff --git a/doc/administration/gitaly/reference.md b/doc/administration/gitaly/reference.md
index 946be6c8af3..169b6e00813 100644
--- a/doc/administration/gitaly/reference.md
+++ b/doc/administration/gitaly/reference.md
@@ -4,256 +4,24 @@ group: Gitaly
 info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://handbook.gitlab.com/handbook/product/ux/technical-writing/#assignments
 ---
 
-# Gitaly reference **(FREE SELF)**
+# Example configuration files **(FREE SELF)**
 
-Gitaly is configured via a [TOML](https://github.com/toml-lang/toml)
-configuration file. Unlike self-compiled installations, in Linux package installations you
-would not edit this file directly. For Linux package installations, the default file location is `/var/opt/gitlab/gitaly/config.toml`.
+Gitaly and Gitaly Cluster are configured by using configuration files. The default location of the configuration files
+depends on the type of installation you have:
 
-The configuration file is passed as an argument to the `gitaly` executable, which is usually done by either your Linux
-package installation or your [init](https://en.wikipedia.org/wiki/Init) script.
+- For Linux package installations, the default location for Gitaly and Gitaly Cluster configuration is in the
+  `/etc/gitlab/gitlab.rb` Ruby file.
+- For self-compiled, the default location for Gitaly and Gitaly Cluster configuration is in the
+  `/home/git/gitaly/config.toml` and `/home/git/gitaly/config.prafect.toml` TOML files.
 
-An [example configuration file](https://gitlab.com/gitlab-org/gitaly/blob/master/config.toml.example)
-can be found in the Gitaly project.
+You can find example TOML configuration files in the `gitaly` project for:
 
-## Format
+- Gitaly: <https://gitlab.com/gitlab-org/gitaly/-/blob/master/config.toml.example>
+- Gitaly Cluster: <https://gitlab.com/gitlab-org/gitaly/-/blob/master/config.praefect.toml.example>
 
-At the top level, `config.toml` defines the items described on the table below.
+If you are configuring a Linux package installation, you must convert the examples into Ruby to use them.
 
-| Name | Type | Required | Description |
-| ---- | ---- | -------- | ----------- |
-| `socket_path` | string | yes (if  `listen_addr` is not set) | A path which Gitaly should open a Unix socket. |
-| `listen_addr` | string | yes (if `socket_path` is not set) | TCP address for Gitaly to listen on. |
-| `tls_listen_addr` | string | no | TCP over TLS address for Gitaly to listen on. |
-| `bin_dir`    | string | yes    | Directory containing Gitaly executables. |
-| `prometheus_listen_addr` | string | no | TCP listen address for Prometheus metrics. If not set, no Prometheus listener is started. |
+For more information on:
 
-For example:
-
-```toml
-socket_path = "/home/git/gitlab/tmp/sockets/private/gitaly.socket"
-listen_addr = "localhost:9999"
-tls_listen_addr = "localhost:8888"
-bin_dir = "/home/git/gitaly"
-prometheus_listen_addr = "localhost:9236"
-```
-
-### Authentication
-
-Gitaly can be configured to reject requests that do not contain a
-specific bearer token in their headers, which is a security measure to
-be used when serving requests over TCP:
-
-```toml
-[auth]
-# A non-empty token enables authentication.
-token = "the secret token"
-```
-
-Authentication is disabled when the token setting in `config.toml` is absent or
-an empty string.
-
-It is possible to temporarily disable authentication with the `transitioning`
-setting. This allows you to monitor if all clients are
-authenticating correctly without causing a service outage for clients
-that are still to be configured correctly:
-
-```toml
-[auth]
-token = "the secret token"
-transitioning = true
-```
-
-WARNING:
-Remember to disable `transitioning` when you are done
-changing your token settings.
-
-All authentication attempts are counted in Prometheus under
-the [`gitaly_authentications_total` metric](monitoring.md#queries).
-
-### TLS
-
-Gitaly supports TLS encryption. You need to bring your own certificates as
-this isn't provided automatically.
-
-| Name | Type | Required | Description |
-| ---- | ---- | -------- | ----------- |
-| `certificate_path` | string | no | Path to the certificate. |
-| `key_path` | string | no | Path to the key. |
-
-```toml
-tls_listen_addr = "localhost:8888"
-
-[tls]
-certificate_path = '/home/git/cert.cert'
-key_path = '/home/git/key.pem'
-```
-
-[Read more](tls_support.md) about TLS in Gitaly.
-
-### Storage
-
-GitLab repositories are grouped into directories known as storages, such as
-`/home/git/repositories`. They contain bare repositories managed
-by GitLab with names, such as `default`.
-
-These names and paths are also defined in the `gitlab.yml` configuration file of
-GitLab. When you run Gitaly on the same machine as GitLab (the default
-and recommended configuration) storage paths defined in the Gitaly `config.toml`
-must match those in `gitlab.yml`.
-
-| Name | Type | Required | Description |
-| ---- | ---- | -------- | ----------- |
-| `storage` | array | yes | An array of storage shards. |
-| `path` | string | yes | The path to the storage shard. |
-| `name` | string | yes | The name of the storage shard. |
-
-For example:
-
-```toml
-[[storage]]
-path = "/path/to/storage/repositories"
-name = "my_shard"
-
-[[storage]]
-path = "/path/to/other/repositories"
-name = "other_storage"
-```
-
-### Git
-
-The following values can be set in the `[git]` section of the configuration file.
-
-| Name | Type | Required | Description |
-| ---- | ---- | -------- | ----------- |
-| `bin_path` | string | no | Path to Git binary. If not set, is resolved using `PATH`. |
-| `catfile_cache_size` | integer | no | Maximum number of cached [cat-file processes](#cat-file-cache). Default is `100`. |
-| `signing_key` | string | no | Path to [GPG signing key](configure_gitaly.md#configure-commit-signing-for-gitlab-ui-commits). If not set, Gitaly doesn't sign commits made using the UI. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/19185) in GitLab 15.4. |
-
-#### `cat-file` cache
-
-A lot of Gitaly RPCs need to look up Git objects from repositories.
-Most of the time we use `git cat-file --batch` processes for that. For
-better performance, Gitaly can re-use these `git cat-file` processes
-across RPC calls. Previously used processes are kept around in a
-["Git cat-file cache"](https://about.gitlab.com/blog/2019/07/08/git-performance-on-nfs/#enter-cat-file-cache).
-To control how much system resources this uses, we have a maximum number of
-cat-file processes that can go into the cache.
-
-The default limit is 100 `cat-file`s, which constitute a pair of
-`git cat-file --batch` and `git cat-file --batch-check` processes. If
-you are seeing errors complaining about "too many open files", or an
-inability to create new processes, you may want to lower this limit.
-
-Ideally, the number should be large enough to handle standard
-traffic. If you raise the limit, you should measure the cache hit ratio
-before and after. If the hit ratio does not improve, the higher limit is
-probably not making a meaningful difference. Here is an example
-Prometheus query to see the hit rate:
-
-```plaintext
-sum(rate(gitaly_catfile_cache_total{type="hit"}[5m])) / sum(rate(gitaly_catfile_cache_total{type=~"(hit)|(miss)"}[5m]))
-```
-
-#### Custom Hooks
-
-> Method of configuring custom hooks directory documented here is preferred from GitLab 16.4. `[gitlab-shell] dir` configuration is no longer required.
-
-Gitaly supports [custom Git hooks](../server_hooks.md) that are
-used to perform tasks based on changes performed in any repository. The custom
-hooks directory is configured in the `[hooks]` section:
-
-| Name | Type | Required | Description |
-| ---- | ---- | -------- | ----------- |
-| `custom_hooks_dir` | string | no | The directory where custom Git hooks are installed. |
-
-Example:
-
-```toml
-[hooks]
-custom_hooks_dir = "/home/git/custom-hooks"
-```
-
-If left unset, no custom hooks are used.
-
-### GitLab
-
-> Method of configuring custom hooks directory documented here is preferred from GitLab 16.4. `[gitlab-shell] dir` configuration is no longer required.
-
-Gitaly must connect to the GitLab application to perform access
-checks when a user performs a change. The parameters to connect to GitLab must
-be configured in the `[gitlab]` section.
-
-| Name | Type | Required | Description |
-| ---- | ---- | -------- | ----------- |
-| `url` | string | yes | The URL of the GitLab server.
-| `secret` | string | no | The secret token used to authenticate with GitLab. |
-| `secret_file` | string | no | The path of the file containing the secret token used to authenticate with GitLab. |
-
-Only one of `secret` or `secret_file` can be configured.
-
-### Prometheus
-
-You can optionally configure Gitaly to record histogram latencies on GRPC method
-calls in Prometheus.
-
-| Name | Type | Required | Description |
-| ---- | ---- | -------- | ----------- |
-| `grpc_latency_buckets` | array | no | Prometheus stores each observation in a bucket, which means you'd get an approximation of latency. Optimizing the buckets gives more control over the accuracy of the approximation. |
-
-Example:
-
-```toml
-prometheus_listen_addr = "localhost:9236"
-
-[prometheus]
-grpc_latency_buckets = [0.001, 0.005, 0.025, 0.1, 0.5, 1.0, 10.0, 30.0, 60.0, 300.0, 1500.0]
-```
-
-### Logging
-
-The following values configure logging in Gitaly under the `[logging]` section.
-
-| Name | Type | Required | Description |
-| ---- | ---- | -------- | ----------- |
-| `format` | string | no | Log format: `text` or `json`. Default: `text`. |
-| `level`  | string | no | Log level: `debug`, `info`, `warn`, `error`, `fatal`, or `panic`. Default: `info`. |
-| `sentry_dsn` | string | no | Sentry DSN (Data Source Name) for exception monitoring. |
-| `sentry_environment` | string | no | [Sentry Environment](https://docs.sentry.io/product/sentry-basics/environments/) for exception monitoring. |
-
-While the main Gitaly application logs go to `stdout`, there are some extra log
-files that go to a configured directory, like the GitLab Shell logs.
-GitLab Shell does not support `panic` or `trace` level logs:
-
-- `panic` falls back to `error`.
-- `trace` falls back to `debug`.
-- Any other invalid log levels default to `info`.
-
-Example:
-
-```toml
-[logging]
-level = "warn"
-dir = "/home/gitaly/logs"
-format = "json"
-sentry_dsn = "https://<key>:<secret>@sentry.io/<project>"
-ruby_sentry_dsn = "https://<key>:<secret>@sentry.io/<project>"
-```
-
-## Concurrency
-
-You can adjust the `concurrency` of each RPC endpoint.
-
-| Name | Type | Required | Description |
-| ---- | ---- | -------- | ----------- |
-| `concurrency` | array | yes | An array of RPC endpoints. |
-| `rpc` | string | no | The name of the RPC endpoint (`/gitaly.RepositoryService/GarbageCollect`). |
-| `max_per_repo` | integer | no | Concurrency per RPC per repository. |
-
-Example:
-
-```toml
-[[concurrency]]
-rpc = "/gitaly.RepositoryService/GarbageCollect"
-max_per_repo = 1
-```
+- Configuring Gitaly, see [Configure Gitaly](configure_gitaly.md).
+- Configuring Gitaly Cluster, see [Configure Gitaly Cluster](praefect.md).