Welcome to mirror list, hosted at ThFree Co, Russian Federation.

gitlab.com/gitlab-org/gitlab-foss.git - Unnamed repository; edit this file 'description' to name the repository.
summaryrefslogtreecommitdiff
path: root/doc/administration/gitaly
diff options
context:
space:
mode:
Diffstat (limited to 'doc/administration/gitaly')
-rw-r--r--doc/administration/gitaly/configure_gitaly.md156
-rw-r--r--doc/administration/gitaly/index.md72
-rw-r--r--doc/administration/gitaly/monitoring.md4
-rw-r--r--doc/administration/gitaly/praefect.md28
-rw-r--r--doc/administration/gitaly/recovery.md4
-rw-r--r--doc/administration/gitaly/troubleshooting.md21
6 files changed, 157 insertions, 128 deletions
diff --git a/doc/administration/gitaly/configure_gitaly.md b/doc/administration/gitaly/configure_gitaly.md
index 4b2832bebc0..5b868c274cd 100644
--- a/doc/administration/gitaly/configure_gitaly.md
+++ b/doc/administration/gitaly/configure_gitaly.md
@@ -280,8 +280,9 @@ Updates to example must be made at:
```
1. Save the file and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure).
-1. Run `sudo /opt/gitlab/embedded/bin/gitaly-hooks check /var/opt/gitlab/gitaly/config.toml`
- to confirm that Gitaly can perform callbacks to the GitLab internal API.
+1. Confirm that Gitaly can perform callbacks to the GitLab internal API:
+ - For GitLab 15.3 and later, run `sudo /opt/gitlab/embedded/bin/gitaly check /var/opt/gitlab/gitaly/config.toml`.
+ - For GitLab 15.2 and earlier, run `sudo /opt/gitlab/embedded/bin/gitaly-hooks check /var/opt/gitlab/gitaly/config.toml`.
**For installations from source**
@@ -330,8 +331,9 @@ Updates to example must be made at:
```
1. Save the files and [restart GitLab](../restart_gitlab.md#installations-from-source).
-1. Run `sudo -u git /home/git/gitaly/gitaly-hooks check /home/git/gitaly/config.toml`
- to confirm that Gitaly can perform callbacks to the GitLab internal API.
+1. Confirm that Gitaly can perform callbacks to the GitLab internal API:
+ - For GitLab 15.3 and later, run `sudo /opt/gitlab/embedded/bin/gitaly check /var/opt/gitlab/gitaly/config.toml`.
+ - For GitLab 15.2 and earlier, run `sudo /opt/gitlab/embedded/bin/gitaly-hooks check /var/opt/gitlab/gitaly/config.toml`.
WARNING:
If directly copying repository data from a GitLab server to Gitaly, ensure that the metadata file,
@@ -792,7 +794,7 @@ gitaly['concurrency'] = [
- `max_per_repo` is the maximum number of in-flight RPC calls for the given RPC per repository.
- `max_queue_time` is the maximum amount of time a request can wait in the concurrency queue to
be picked up by Gitaly.
-- `max_queue_size` is the maximum size the concurrency queue can grow to before requests are rejected by
+- `max_queue_size` is the maximum size the concurrency queue (per RPC method) can grow to before requests are rejected by
Gitaly.
This limits the number of in-flight RPC calls for the given RPCs. The limit is applied per
@@ -811,26 +813,12 @@ information, see the [relevant documentation](monitoring.md#monitor-gitaly-concu
## Control groups
FLAG:
-On self-managed GitLab, by default cgroups are not available. To make it available, ask an administrator to
+On self-managed GitLab, by default repository cgroups are not available. To make it available, ask an administrator to
[enable the feature flag](../feature_flags.md) named `gitaly_run_cmds_in_cgroup`.
-Gitaly shells out to Git for many of its operations. Git can consume a lot of resources for certain operations,
-especially for large repositories.
-
Control groups (cgroups) in Linux allow limits to be imposed on how much memory and CPU can be consumed.
See the [`cgroups` Linux man page](https://man7.org/linux/man-pages/man7/cgroups.7.html) for more information.
-cgroups can be useful for protecting the system against resource exhaustion because of overcomsumption of memory and CPU.
-
-Gitaly has built-in cgroups control. When configured, Gitaly assigns Git
-processes to a cgroup based on the repository the Git command is operating in.
-Each cgroup has a memory and CPU limit. When a cgroup reaches its:
-
-- Memory limit, the kernel looks through the processes for a candidate to kill.
-- CPU limit, processes are not killed, but the processes are prevented from consuming more CPU than allowed.
-
-The main reason to configure cgroups for your GitLab installation is that it
-protects against system resource starvation due to a few large repositories or
-bad actors.
+cgroups can be useful for protecting the system against resource exhaustion because of over consumption of memory and CPU.
Some Git operations are expensive by nature. `git clone`, for instance,
spawns a `git-upload-pack` process on the server that can consume a lot of memory
@@ -838,33 +826,33 @@ for large repositories. For example, a client that keeps on cloning a
large repository over and over again. This situation could potentially use up all of the
memory on a server, causing other operations to fail for other users.
-There are many ways someone can create a repository that can consume large amounts of memory when cloned or downloaded.
+A repository can consume large amounts of memory for many reasons when cloned or downloaded.
Using cgroups allows the kernel to kill these operations before they hog up all system resources.
-### Configure cgroups in Gitaly
+Gitaly shells out to Git for many of its operations. Git can consume a lot of resources for certain operations,
+especially for large repositories.
-Two ways of configuring cgroups are available.
+Gitaly has built-in cgroups control. When configured, Gitaly assigns Git processes to a cgroup based on the repository
+the Git command is operating in. These cgroups are called repository cgroups. Each repository cgroup:
-#### Configure cgroups (new method)
+- Has a memory and CPU limit.
+- Contains the Git processes for a single repository.
+- Uses a consistent hash to ensure a Git process for a given repository always ends up in the same cgroup.
-> This method of configuring cgroups introduced in GitLab 15.1.
+When a repository cgroup reaches its:
-Gitaly creates a pool of cgroups that are isolated based on the repository used in the Git command to be placed under one of these cgroups.
+- Memory limit, the kernel looks through the processes for a candidate to kill.
+- CPU limit, processes are not killed, but the processes are prevented from consuming more CPU than allowed.
-To configure cgroups in Gitaly, add `gitaly['cgroups']` to `/etc/gitlab/gitlab.rb`.
+You configure repository cgroups for your GitLab installation to protect against system resource starvation from a few
+large repositories or bad actors.
-For example:
+### Configure repository cgroups (new method)
-```ruby
-# in /etc/gitlab/gitlab.rb
-gitaly['cgroups_mountpoint'] = "/sys/fs/cgroup"
-gitaly['cgroups_hierarchy_root'] =>"gitaly"
-gitaly['cgroups_memory_bytes'] = 64424509440, # 60gb
-gitaly['cgroups_cpu_shares'] = 1024
-gitaly['cgroups_repositories_count'] => 1000,
-gitaly['cgroups_repositories_memory_bytes'] => 32212254720 # 20gb
-gitaly['cgroups_repositories_cpu_shares'] => 512
-```
+> This method of configuring repository cgroups was introduced in GitLab 15.1.
+
+To configure repository cgroups in Gitaly using the new method, use the following settings for the new configuration method
+to `gitaly['cgroups']` in `/etc/gitlab/gitlab.rb`:
- `cgroups_mountpoint` is where the parent cgroup directory is mounted. Defaults to `/sys/fs/cgroup`.
- `cgroups_hierarchy_root` is the parent cgroup under which Gitaly creates groups, and
@@ -873,7 +861,7 @@ gitaly['cgroups_repositories_cpu_shares'] => 512
when Gitaly starts.
- `cgroups_memory_bytes` is the total memory limit that is imposed collectively on all
Git processes that Gitaly spawns. 0 implies no limit.
-- `cgroups_cpu_shares` is the cpu limit that is imposed collectively on all Git
+- `cgroups_cpu_shares` is the CPU limit that is imposed collectively on all Git
processes that Gitaly spawns. 0 implies no limit. The maximum is 1024 shares,
which represents 100% of CPU.
- `cgroups_repositories_count` is the number of cgroups in the cgroups pool. Each time a new Git
@@ -881,30 +869,30 @@ gitaly['cgroups_repositories_cpu_shares'] => 512
on the repository the command is for. A circular hashing algorithm assigns
Git commands to these cgroups, so a Git command for a repository is
always assigned to the same cgroup.
-- `cgroups_repositories_memory_bytes` is the total memory limit that is imposed collectively on all
- Git processes that Gitaly spawns. 0 implies no limit. This value cannot exceed
- that of the top level `cgroups_memory_bytes`.
-- `cgroups_repositories_cpu_shares` is the CPU limit that is imposed collectively on all Git
- processes Gitaly spawns. 0 implies no limit. The maximum is 1024 shares,
- which represents 100% of CPU. This value cannot exceed that of the top
- level`cgroups_cpu_shares`.
-
-#### Configure cgroups (legacy method)
+- `cgroups_repositories_memory_bytes` is the total memory limit imposed on all Git processes contained in a repository cgroup.
+ 0 implies no limit. This value cannot exceed that of the top level `cgroups_memory_bytes`.
+- `cgroups_repositories_cpu_shares` is the CPU limit that is imposed on all Git processes contained in a repository cgroup.
+ 0 implies no limit. The maximum is 1024 shares, which represents 100% of CPU.
+ This value cannot exceed that of the top level`cgroups_cpu_shares`.
-To configure cgroups in Gitaly for GitLab versions using the legacy method, add `gitaly['cgroups']` to `/etc/gitlab/gitlab.rb`. For
-example:
+For example:
```ruby
# in /etc/gitlab/gitlab.rb
-gitaly['cgroups_count'] = 1000
gitaly['cgroups_mountpoint'] = "/sys/fs/cgroup"
-gitaly['cgroups_hierarchy_root'] = "gitaly"
-gitaly['cgroups_memory_limit'] = 32212254720
-gitaly['cgroups_memory_enabled'] = true
+gitaly['cgroups_hierarchy_root'] => "gitaly"
+gitaly['cgroups_memory_bytes'] = 64424509440, # 60gb
gitaly['cgroups_cpu_shares'] = 1024
-gitaly['cgroups_cpu_enabled'] = true
+gitaly['cgroups_repositories_count'] => 1000,
+gitaly['cgroups_repositories_memory_bytes'] => 32212254720 # 20gb
+gitaly['cgroups_repositories_cpu_shares'] => 512
```
+### Configure repository cgroups (legacy method)
+
+To configure repository cgroups in Gitaly using the legacy method, use the following settings
+in `/etc/gitlab/gitlab.rb`:
+
- `cgroups_count` is the number of cgroups created. Each time a new
command is spawned, Gitaly assigns it to one of these cgroups based
on the command line arguments of the command. A circular hashing algorithm assigns
@@ -917,7 +905,21 @@ gitaly['cgroups_cpu_enabled'] = true
- `cgroups_memory_enabled` enables or disables the memory limit on cgroups.
- `cgroups_memory_bytes` is the total memory limit each cgroup imposes on the processes added to it.
- `cgroups_cpu_enabled` enables or disables the CPU limit on cgroups.
-- `cgroups_cpu_shares` is the CPU limit each cgroup imposes on the processes added to it. The maximum is 1024 shares, which represents 100% of CPU.
+- `cgroups_cpu_shares` is the CPU limit each cgroup imposes on the processes added to it. The maximum is 1024 shares,
+ which represents 100% of CPU.
+
+For example:
+
+```ruby
+# in /etc/gitlab/gitlab.rb
+gitaly['cgroups_count'] = 1000
+gitaly['cgroups_mountpoint'] = "/sys/fs/cgroup"
+gitaly['cgroups_hierarchy_root'] = "gitaly"
+gitaly['cgroups_memory_limit'] = 32212254720
+gitaly['cgroups_memory_enabled'] = true
+gitaly['cgroups_cpu_shares'] = 1024
+gitaly['cgroups_cpu_enabled'] = true
+```
### Configuring oversubscription
@@ -926,16 +928,15 @@ In the previous example using the new configuration method:
- The top level memory limit is capped at 60gb.
- Each of the 1000 cgroups in the repositories pool is capped at 20gb.
-This is called "oversubscription". Each cgroup in the pool has a much larger capacity than 1/1000th
+This configuration leads to "oversubscription". Each cgroup in the pool has a much larger capacity than 1/1000th
of the top-level memory limit.
This strategy has two main benefits:
-- It gives the host protection from overall memory starvation (OOM), because the top-level
- cgroup's memory limit can be set to a threshold smaller than the host's
- capacity. Processes outside of that cgroup are not at risk of OOM.
+- It gives the host protection from overall memory starvation (OOM), because the memory limit of the top-level cgroup
+ can be set to a threshold smaller than the host's capacity. Processes outside of that cgroup are not at risk of OOM.
- It allows each individual cgroup in the pool to burst up to a generous upper
- bound (in this example 20 GB) that is smaller than the parent cgroup's limit,
+ bound (in this example 20 GB) that is smaller than the limit of the parent cgroup,
but substantially larger than 1/N of the parent's limit. In this example, up
to 3 child cgroups can concurrently burst up to their max. In general, all
1000 cgroups would use much less than the 20 GB.
@@ -1141,8 +1142,7 @@ gitaly['pack_objects_cache_enabled'] = true
#### `enabled` defaults to `false`
The cache is disabled by default. This is because in some cases, it
-can create an [extreme
-increase](https://gitlab.com/gitlab-com/gl-infra/production/-/issues/4010#note_534564684)
+can create an [extreme increase](https://gitlab.com/gitlab-com/gl-infra/production/-/issues/4010#note_534564684)
in the number of bytes written to disk. On GitLab.com, we have verified
that our repository storage disks can handle this extra workload, but
we felt we cannot assume this is true everywhere.
@@ -1302,14 +1302,26 @@ process repositories that do not pass consistency checks.
For Omnibus GitLab installations, edit `/etc/gitlab/gitlab.rb` and set the
following keys (in this example, to disable the `hasDotgit` consistency check):
-```ruby
-ignored_git_errors = ["hasDotgit = ignore"]
-omnibus_gitconfig['system'] = {
- "fsck" => ignored_git_errors,
- "fetch.fsck" => ignored_git_errors,
- "receive.fsck" => ignored_git_errors,
-}
-```
+- In [GitLab 15.3](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/6800) and later:
+
+ ```ruby
+ gitaly['gitconfig'] = [
+ { key: "fsck.hasDotgit", value: "ignore" },
+ { key: "fetch.fsck.hasDotgit", value: "ignore" },
+ { key: "receive.fsck.hasDotgit", value: "ignore "},
+ ]
+ ```
+
+- In GitLab 15.2 and earlier (legacy method):
+
+ ```ruby
+ ignored_git_errors = ["hasDotgit = ignore"]
+ omnibus_gitconfig['system'] = {
+ "fsck" => ignored_git_errors,
+ "fetch.fsck" => ignored_git_errors,
+ "receive.fsck" => ignored_git_errors,
+ }
+ ```
For source installs, edit the Gitaly configuration (`gitaly.toml`) to do the
equivalent:
diff --git a/doc/administration/gitaly/index.md b/doc/administration/gitaly/index.md
index c543f62f135..b053da7ac9b 100644
--- a/doc/administration/gitaly/index.md
+++ b/doc/administration/gitaly/index.md
@@ -70,7 +70,7 @@ the current status of these issues, please refer to the referenced issues and ep
| Issue | Summary | How to avoid |
|:--------------------------------------------------------------------------------------|:------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|:--------------------------------|
-| Gitaly Cluster + Geo - Issues retrying failed syncs | If Gitaly Cluster is used on a Geo secondary site, repositories that have failed to sync could continue to fail when Geo tries to resync them. Recovering from this state requires assistance from support to run manual steps. Work is in-progress to update Gitaly Cluster to [identify repositories with a unique and persistent identifier](https://gitlab.com/gitlab-org/gitaly/-/issues/3485), which is expected to resolve the issue. | No known solution at this time. |
+| Gitaly Cluster + Geo - Issues retrying failed syncs | If Gitaly Cluster is used on a Geo secondary site, repositories that have failed to sync could continue to fail when Geo tries to resync them. Recovering from this state requires assistance from support to run manual steps. | No known solution prior to GitLab 15.0. In GitLab 15.0 to 15.2, enable the [`gitaly_praefect_generated_replica_paths` feature flag](#praefect-generated-replica-paths-gitlab-150-and-later). In GitLab 15.3, the feature flag is enabled by default. |
| Praefect unable to insert data into the database due to migrations not being applied after an upgrade | If the database is not kept up to date with completed migrations, then the Praefect node is unable to perform normal operation. | Make sure the Praefect database is up and running with all migrations completed (For example: `/opt/gitlab/embedded/bin/praefect -config /var/opt/gitlab/praefect/config.toml sql-migrate-status` should show a list of all applied migrations). Consider [requesting upgrade assistance](https://about.gitlab.com/support/scheduling-upgrade-assistance/) so your upgrade plan can be reviewed by support. |
| Restoring a Gitaly Cluster node from a snapshot in a running cluster | Because the Gitaly Cluster runs with consistent state, introducing a single node that is behind will result in the cluster not being able to reconcile the nodes data and other nodes data | Don't restore a single Gitaly Cluster node from a backup snapshot. If you must restore from backup, it's best to snapshot all Gitaly Cluster nodes at the same time and take a database dump of the Praefect database. |
@@ -80,16 +80,31 @@ Gitaly Cluster does not support snapshot backups. Snapshot backups can cause iss
out of sync with the disk storage. Because of how Praefect rebuilds the replication metadata of Gitaly disk information
during a restore, we recommend using the [official backup and restore Rake tasks](../../raketasks/backup_restore.md).
-If you are unable to use this method, please contact customer support for restoration help.
+The [incremental backup method](../../raketasks/backup_gitlab.md#incremental-repository-backups)
+can be used to speed up Gitaly Cluster backups.
-We are tracking in [this issue](https://gitlab.com/gitlab-org/gitlab/-/issues/351383) improvements to the
-[official backup and restore Rake tasks](../../raketasks/backup_restore.md) to add support for incremental backups. For
-more information, see [this epic](https://gitlab.com/groups/gitlab-org/-/epics/2094).
+If you are unable to use either method, please contact customer support for restoration help.
### What to do if you are on Gitaly Cluster experiencing an issue or limitation
Please contact customer support for immediate help in restoration or recovery.
+## Directly accessing repositories
+
+GitLab doesn't advise directly accessing Gitaly repositories stored on disk with a Git client or any other tool,
+because Gitaly is being continuously improved and changed. These improvements may invalidate
+your assumptions, resulting in performance degradation, instability, and even data loss. For example:
+
+- Gitaly has optimizations such as the [`info/refs` advertisement cache](https://gitlab.com/gitlab-org/gitaly/blob/master/doc/design_diskcache.md),
+ that rely on Gitaly controlling and monitoring access to repositories by using the official gRPC
+ interface.
+- [Gitaly Cluster](#gitaly-cluster) has optimizations, such as fault tolerance and
+ [distributed reads](#distributed-reads), that depend on the gRPC interface and database
+ to determine repository state.
+
+WARNING:
+Accessing Git repositories directly is done at your own risk and is not supported.
+
## Gitaly
The following shows GitLab set up to use direct access to Gitaly:
@@ -141,6 +156,11 @@ Gitaly comes pre-configured with Omnibus GitLab, which is a configuration
GitLab installations for more than 2000 active users performing daily Git write operation may be
best suited by using Gitaly Cluster.
+### Backing up repositories
+
+When backing up or syncing repositories using tools other than GitLab, you must [prevent writes](../../raketasks/backup_restore.md#prevent-writes-and-copy-the-git-repository-data)
+while copying repository data.
+
## Gitaly Cluster
Git storage is provided through the Gitaly service in GitLab, and is essential to the operation of
@@ -297,11 +317,12 @@ follow the [hashed storage](../repository_storage_types.md#hashed-storage) schem
#### Praefect-generated replica paths (GitLab 15.0 and later)
-> Introduced in GitLab 15.0 behind [a feature flag](https://gitlab.com/gitlab-org/gitaly/-/issues/4218) named `gitaly_praefect_generated_replica_paths`. Disabled by default.
+> - [Introduced](https://gitlab.com/gitlab-org/gitaly/-/issues/4218) in GitLab 15.0 [with a flag](../feature_flags.md) named `gitaly_praefect_generated_replica_paths`. Disabled by default.
+> - [Enabled on GitLab.com](https://gitlab.com/gitlab-org/gitaly/-/issues/4218) in GitLab 15.2.
+> - [Enabled on self-managed](https://gitlab.com/gitlab-org/gitaly/-/merge_requests/4809) in GitLab 15.3.
FLAG:
-On self-managed GitLab, by default this feature is not available. To make it available, ask an administrator to [enable the feature flag](../feature_flags.md)
-named `gitaly_praefect_generated_replica_paths`. On GitLab.com, this feature is available but can be configured by GitLab.com administrators only. The feature is not ready for production use.
+On self-managed GitLab, by default this feature is available. To hide the feature, ask an administrator to [disable the feature flag](../feature_flags.md) named `gitaly_praefect_generated_replica_paths`. On GitLab.com, this feature is available but can be configured by GitLab.com administrators only.
When Gitaly Cluster creates a repository, it assigns the repository a unique and permanent ID called the _repository ID_. The repository ID is
internal to Gitaly Cluster and doesn't relate to any IDs elsewhere in GitLab. If a repository is removed from Gitaly Cluster and later moved
@@ -525,12 +546,9 @@ To upgrade a Gitaly Cluster, follow the documentation for
### Downgrade Gitaly Cluster to a previous version
-If you need to roll back a Gitaly Cluster to an earlier version, some Praefect database migrations may need to be reverted. In a cluster with:
+If you need to roll back a Gitaly Cluster to an earlier version, some Praefect database migrations may need to be reverted.
-- A single Praefect node, this happens when GitLab itself is downgraded.
-- Multiple Praefect nodes, additional steps are required.
-
-To downgrade a Gitaly Cluster with multiple Praefect nodes:
+To downgrade a Gitaly Cluster (assuming multiple Praefect nodes):
1. Stop the Praefect service on all Praefect nodes:
@@ -565,7 +583,7 @@ To downgrade a Gitaly Cluster with multiple Praefect nodes:
gitlab-ctl start praefect
```
-## Migrate to Gitaly Cluster
+### Migrate to Gitaly Cluster
WARNING:
Some [known issues](#known-issues) exist in Gitaly Cluster. Review the following information before you continue.
@@ -587,7 +605,7 @@ To migrate to Gitaly Cluster:
Even if you don't use the `default` repository storage, you must ensure it is configured.
[Read more about this limitation](configure_gitaly.md#gitlab-requires-a-default-repository-storage).
-## Migrate off Gitaly Cluster
+### Migrate off Gitaly Cluster
If the limitations and tradeoffs of Gitaly Cluster are found to be not suitable for your environment, you can Migrate
off Gitaly Cluster to a sharded Gitaly instance:
@@ -596,22 +614,6 @@ off Gitaly Cluster to a sharded Gitaly instance:
1. [Move the repositories](../operations/moving_repositories.md#move-repositories) to the newly created storage. You can
move them by shard or by group, which gives you the opportunity to spread them over multiple Gitaly servers.
-## Do not bypass Gitaly
-
-GitLab doesn't advise directly accessing Gitaly repositories stored on disk with a Git client,
-because Gitaly is being continuously improved and changed. These improvements may invalidate
-your assumptions, resulting in performance degradation, instability, and even data loss. For example:
-
-- Gitaly has optimizations such as the [`info/refs` advertisement cache](https://gitlab.com/gitlab-org/gitaly/blob/master/doc/design_diskcache.md),
- that rely on Gitaly controlling and monitoring access to repositories by using the official gRPC
- interface.
-- [Gitaly Cluster](#gitaly-cluster) has optimizations, such as fault tolerance and
- [distributed reads](#distributed-reads), that depend on the gRPC interface and database
- to determine repository state.
-
-WARNING:
-Accessing Git repositories directly is done at your own risk and is not supported.
-
## Direct access to Git in GitLab
Direct access to Git uses code in GitLab known as the "Rugged patches".
@@ -683,8 +685,12 @@ To see if GitLab can access the repository file system directly, we use the foll
- GitLab Rails tries to read the metadata file directly. If it exists, and if the UUID's match,
assume we have direct access.
-Direct Git access is enable by default in Omnibus GitLab because it fills in the correct repository
-paths in the GitLab configuration file `config/gitlab.yml`. This satisfies the UUID check.
+Versions of GitLab 15.3 and later disable direct Git access by default.
+
+For versions of GitLab prior to 15.3, direct Git access is enabled by
+default in Omnibus GitLab because it fills in the correct repository
+paths in the GitLab configuration file `config/gitlab.yml`. This
+satisfies the UUID check.
### Transition to Gitaly Cluster
diff --git a/doc/administration/gitaly/monitoring.md b/doc/administration/gitaly/monitoring.md
index 0dd9e0d7128..a435fff74fc 100644
--- a/doc/administration/gitaly/monitoring.md
+++ b/doc/administration/gitaly/monitoring.md
@@ -34,7 +34,7 @@ of requests dropped due to request limiting. The `reason` label indicates why a
You can observe specific behavior of [concurrency-queued requests](configure_gitaly.md#limit-rpc-concurrency) using
the Gitaly logs and Prometheus:
-- In the [Gitaly logs](../logs.md#gitaly-logs), look for the string (or structured log field)
+- In the [Gitaly logs](../logs/index.md#gitaly-logs), look for the string (or structured log field)
`acquire_ms`. Messages that have this field are reporting about the concurrency limiter.
- In Prometheus, look for the following metrics:
- `gitaly_concurrency_limiting_in_progress` indicates how many concurrent requests are
@@ -188,7 +188,7 @@ To monitor [repository verification](praefect.md#repository-verification), use t
- `gitaly_praefect_stale_verification_leases_released_total`, the number of stale verification leases
released.
-You can also monitor the [Praefect logs](../logs.md#praefect-logs).
+You can also monitor the [Praefect logs](../logs/index.md#praefect-logs).
### Database metrics `/db_metrics` endpoint
diff --git a/doc/administration/gitaly/praefect.md b/doc/administration/gitaly/praefect.md
index 5635898293b..488e6a0df5f 100644
--- a/doc/administration/gitaly/praefect.md
+++ b/doc/administration/gitaly/praefect.md
@@ -97,8 +97,8 @@ If you [installed](https://about.gitlab.com/install/) GitLab using the Omnibus G
### Preparation
-Before beginning, you should already have a working GitLab instance. [Learn how
-to install GitLab](https://about.gitlab.com/install/).
+Before beginning, you should already have a working GitLab instance.
+[Learn how to install GitLab](https://about.gitlab.com/install/).
Provision a PostgreSQL server. We recommend using the PostgreSQL that is shipped
with Omnibus GitLab and use it to configure the PostgreSQL database. You can use an
@@ -331,8 +331,8 @@ To configure the additional connection, you must either:
#### Configure a new PgBouncer database with `pool_mode = session`
We recommend using PgBouncer with `session` pool mode. You can use the
-[bundled PgBouncer](../postgresql/pgbouncer.md) or use an external PgBouncer and [configure it
-manually](https://www.pgbouncer.org/config.html).
+[bundled PgBouncer](../postgresql/pgbouncer.md) or use an external PgBouncer and
+[configure it manually](https://www.pgbouncer.org/config.html).
The following example uses the bundled PgBouncer and sets up two separate connection pools on PostgreSQL host,
one in `session` pool mode and the other in `transaction` pool mode. For this example to work,
@@ -620,8 +620,8 @@ Updates to example must be made at:
gitlab-ctl reconfigure
```
-1. To ensure that Praefect [has updated its Prometheus listen
- address](https://gitlab.com/gitlab-org/gitaly/-/issues/2734),
+1. To ensure that Praefect
+ [has updated its Prometheus listen address](https://gitlab.com/gitlab-org/gitaly/-/issues/2734),
[restart Praefect](../restart_gitlab.md#omnibus-gitlab-restart):
```shell
@@ -928,8 +928,8 @@ For more information on Gitaly server configuration, see our
gitlab-ctl reconfigure
```
-1. To ensure that Gitaly [has updated its Prometheus listen
- address](https://gitlab.com/gitlab-org/gitaly/-/issues/2734),
+1. To ensure that Gitaly
+ [has updated its Prometheus listen address](https://gitlab.com/gitlab-org/gitaly/-/issues/2734),
[restart Gitaly](../restart_gitlab.md#omnibus-gitlab-restart):
```shell
@@ -1103,10 +1103,8 @@ Particular attention should be shown to:
```
1. Verify on each Gitaly node the Git Hooks can reach GitLab. On each Gitaly node run:
-
- ```shell
- /opt/gitlab/embedded/bin/gitaly-hooks check /var/opt/gitlab/gitaly/config.toml
- ```
+ - For GitLab 15.3 and later, run `sudo /opt/gitlab/embedded/bin/gitaly check /var/opt/gitlab/gitaly/config.toml`.
+ - For GitLab 15.2 and earlier, run `sudo /opt/gitlab/embedded/bin/gitaly-hooks check /var/opt/gitlab/gitaly/config.toml`.
1. Verify that GitLab can reach Praefect:
@@ -1151,8 +1149,7 @@ running multiple Gitaly storages.
### Grafana
Grafana is included with GitLab, and can be used to monitor your Praefect
-cluster. See [Grafana Dashboard
-Service](https://docs.gitlab.com/omnibus/settings/grafana.html)
+cluster. See [Grafana Dashboard Service](https://docs.gitlab.com/omnibus/settings/grafana.html)
for detailed documentation.
To get started quickly:
@@ -1320,8 +1317,7 @@ praefect['background_verification_verification_interval'] = '0'
WARNING:
Deletions are disabled by default due to a race condition with repository renames that can cause incorrect
deletions. This is especially prominent in Geo instances as Geo performs more renames than instances without Geo.
-See [Handle repository creations, deletions and renames atomically](https://gitlab.com/gitlab-org/gitaly/-/merge_requests/4101)
-for progress on a fix. We do not recommend enabling the deletions until this is fixed.
+You should enable deletions only if the [`gitaly_praefect_generated_replica_paths` feature flag](index.md#praefect-generated-replica-paths-gitlab-150-and-later) is enabled.
By default, the worker does not delete invalid metadata records but simply logs them and outputs Prometheus
metrics for them.
diff --git a/doc/administration/gitaly/recovery.md b/doc/administration/gitaly/recovery.md
index 551a7ab289a..bd4846a986d 100644
--- a/doc/administration/gitaly/recovery.md
+++ b/doc/administration/gitaly/recovery.md
@@ -296,6 +296,7 @@ sudo /opt/gitlab/embedded/bin/praefect -config /var/opt/gitlab/praefect/config.t
> - [Introduced](https://gitlab.com/gitlab-org/gitaly/-/merge_requests/3767) in GitLab 14.3.
> - [Introduced](https://gitlab.com/gitlab-org/gitaly/-/merge_requests/4054) in GitLab 14.6, support for dry-run mode.
+> - [Introduced](https://gitlab.com/gitlab-org/gitaly/-/merge_requests/4715) in GitLab 15.3, support for removing repositories from Praefect's database.
The `remove-repository` Praefect sub-command removes a repository from a Gitaly Cluster, and all state associated with a given repository including:
@@ -310,6 +311,9 @@ sudo /opt/gitlab/embedded/bin/praefect -config /var/opt/gitlab/praefect/config.t
- Replace `<virtual-storage>` with the name of the virtual storage containing the repository.
- Replace `<repository>` with the relative path of the repository to remove.
+- In GitLab 15.3 and later, add `-db-only` to remove the Praefect database entry without removing the on-disk repository. Use this option to remove orphaned database entries and to
+ protect on-disk repository data from deletion when a valid repository is accidentally specified. If the database entry is accidentally deleted, re-track the repository with the
+ [`track-repository` command](#manually-track-repositories).
- In GitLab 14.6 and later, add `-apply` to run the command outside of dry-run mode and remove the repository. For example:
```shell
diff --git a/doc/administration/gitaly/troubleshooting.md b/doc/administration/gitaly/troubleshooting.md
index fb3a31d61b8..e308fa9da43 100644
--- a/doc/administration/gitaly/troubleshooting.md
+++ b/doc/administration/gitaly/troubleshooting.md
@@ -13,7 +13,7 @@ Refer to the information below when troubleshooting Gitaly and Gitaly Cluster.
The following sections provide possible solutions to Gitaly errors.
See also [Gitaly timeout](../../user/admin_area/settings/gitaly_timeouts.md) settings,
-and our advice on [parsing the `gitaly/current` file](../troubleshooting/log_parsing.md#parsing-gitalycurrent).
+and our advice on [parsing the `gitaly/current` file](../logs/log_parsing.md#parsing-gitalycurrent).
### Check versions when using standalone Gitaly servers
@@ -24,6 +24,17 @@ as GitLab to ensure full compatibility:
1. On the left sidebar, select **Overview > Gitaly Servers**.
1. Confirm all Gitaly servers indicate that they are up to date.
+### Find storage resource details
+
+You can run the following commands in a [Rails console](../operations/rails_console.md#starting-a-rails-console-session)
+to determine the available and used space on a Gitaly storage:
+
+```ruby
+Gitlab::GitalyClient::ServerService.new("default").storage_disk_statistics
+# For Gitaly Cluster
+Gitlab::GitalyClient::ServerService.new("<storage name>").disk_statistics
+```
+
### Use `gitaly-debug`
The `gitaly-debug` command provides "production debugging" tools for Gitaly and Git
@@ -444,15 +455,15 @@ If you receive an error, check `/var/log/gitlab/gitlab-rails/production.log`.
Here are common errors and potential causes:
- 500 response code
- - **ActionView::Template::Error (7:permission denied)**
+ - `ActionView::Template::Error (7:permission denied)`
- `praefect['auth_token']` and `gitlab_rails['gitaly_token']` do not match on the GitLab server.
- - **Unable to save project. Error: 7:permission denied**
+ - `Unable to save project. Error: 7:permission denied`
- Secret token in `praefect['storage_nodes']` on GitLab server does not match the
value in `gitaly['auth_token']` on one or more Gitaly servers.
- 503 response code
- - **GRPC::Unavailable (14:failed to connect to all addresses)**
+ - `GRPC::Unavailable (14:failed to connect to all addresses)`
- GitLab was unable to reach Praefect.
- - **GRPC::Unavailable (14:all SubCons are in TransientFailure...)**
+ - `GRPC::Unavailable (14:all SubCons are in TransientFailure...)`
- Praefect cannot reach one or more of its child Gitaly nodes. Try running
the Praefect connection checker to diagnose.
generated by cgit v1.2.3 (git 2.25.1) at 2024-08-12 21:34:16 +0300