Add latest changes from gitlab-org/gitlab@15-3-stable-eev15.3.0-rc42

1 files changed, 84 insertions, 72 deletions

diff --git a/doc/administration/gitaly/configure_gitaly.md b/doc/administration/gitaly/configure_gitaly.md
index 4b2832bebc0..5b868c274cd 100644
--- a/doc/administration/gitaly/configure_gitaly.md
+++ b/doc/administration/gitaly/configure_gitaly.md
@@ -280,8 +280,9 @@ Updates to example must be made at:
    ```
 
 1. Save the file and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure).
-1. Run `sudo /opt/gitlab/embedded/bin/gitaly-hooks check /var/opt/gitlab/gitaly/config.toml`
-   to confirm that Gitaly can perform callbacks to the GitLab internal API.
+1. Confirm that Gitaly can perform callbacks to the GitLab internal API:
+   - For GitLab 15.3 and later, run `sudo /opt/gitlab/embedded/bin/gitaly check /var/opt/gitlab/gitaly/config.toml`.
+   - For GitLab 15.2 and earlier, run `sudo /opt/gitlab/embedded/bin/gitaly-hooks check /var/opt/gitlab/gitaly/config.toml`.
 
 **For installations from source**
 
@@ -330,8 +331,9 @@ Updates to example must be made at:
    ```
 
 1. Save the files and [restart GitLab](../restart_gitlab.md#installations-from-source).
-1. Run `sudo -u git /home/git/gitaly/gitaly-hooks check /home/git/gitaly/config.toml`
-   to confirm that Gitaly can perform callbacks to the GitLab internal API.
+1. Confirm that Gitaly can perform callbacks to the GitLab internal API:
+   - For GitLab 15.3 and later, run `sudo /opt/gitlab/embedded/bin/gitaly check /var/opt/gitlab/gitaly/config.toml`.
+   - For GitLab 15.2 and earlier, run `sudo /opt/gitlab/embedded/bin/gitaly-hooks check /var/opt/gitlab/gitaly/config.toml`.
 
 WARNING:
 If directly copying repository data from a GitLab server to Gitaly, ensure that the metadata file,
@@ -792,7 +794,7 @@ gitaly['concurrency'] = [
 - `max_per_repo` is the maximum number of in-flight RPC calls for the given RPC per repository.
 - `max_queue_time` is the maximum amount of time a request can wait in the concurrency queue to
   be picked up by Gitaly.
-- `max_queue_size` is the maximum size the concurrency queue can grow to before requests are rejected by
+- `max_queue_size` is the maximum size the concurrency queue (per RPC method) can grow to before requests are rejected by
   Gitaly.
 
 This limits the number of in-flight RPC calls for the given RPCs. The limit is applied per
@@ -811,26 +813,12 @@ information, see the [relevant documentation](monitoring.md#monitor-gitaly-concu
 ## Control groups
 
 FLAG:
-On self-managed GitLab, by default cgroups are not available. To make it available, ask an administrator to
+On self-managed GitLab, by default repository cgroups are not available. To make it available, ask an administrator to
 [enable the feature flag](../feature_flags.md) named `gitaly_run_cmds_in_cgroup`.
 
-Gitaly shells out to Git for many of its operations. Git can consume a lot of resources for certain operations,
-especially for large repositories.
-
 Control groups (cgroups) in Linux allow limits to be imposed on how much memory and CPU can be consumed.
 See the [`cgroups` Linux man page](https://man7.org/linux/man-pages/man7/cgroups.7.html) for more information.
-cgroups can be useful for protecting the system against resource exhaustion because of overcomsumption of memory and CPU.
-
-Gitaly has built-in cgroups control. When configured, Gitaly assigns Git
-processes to a cgroup based on the repository the Git command is operating in.
-Each cgroup has a memory and CPU limit. When a cgroup reaches its:
-
-- Memory limit, the kernel looks through the processes for a candidate to kill.
-- CPU limit, processes are not killed, but the processes are prevented from consuming more CPU than allowed.
-
-The main reason to configure cgroups for your GitLab installation is that it
-protects against system resource starvation due to a few large repositories or
-bad actors.
+cgroups can be useful for protecting the system against resource exhaustion because of over consumption of memory and CPU.
 
 Some Git operations are expensive by nature. `git clone`, for instance,
 spawns a `git-upload-pack` process on the server that can consume a lot of memory
@@ -838,33 +826,33 @@ for large repositories. For example, a client that keeps on cloning a
 large repository over and over again. This situation could potentially use up all of the
 memory on a server, causing other operations to fail for other users.
 
-There are many ways someone can create a repository that can consume large amounts of memory when cloned or downloaded.
+A repository can consume large amounts of memory for many reasons when cloned or downloaded.
 Using cgroups allows the kernel to kill these operations before they hog up all system resources.
 
-### Configure cgroups in Gitaly
+Gitaly shells out to Git for many of its operations. Git can consume a lot of resources for certain operations,
+especially for large repositories.
 
-Two ways of configuring cgroups are available.
+Gitaly has built-in cgroups control. When configured, Gitaly assigns Git processes to a cgroup based on the repository
+the Git command is operating in. These cgroups are called repository cgroups. Each repository cgroup:
 
-#### Configure cgroups (new method)
+- Has a memory and CPU limit.
+- Contains the Git processes for a single repository.
+- Uses a consistent hash to ensure a Git process for a given repository always ends up in the same cgroup.
 
-> This method of configuring cgroups introduced in GitLab 15.1.
+When a repository cgroup reaches its:
 
-Gitaly creates a pool of cgroups that are isolated based on the repository used in the Git command to be placed under one of these cgroups.
+- Memory limit, the kernel looks through the processes for a candidate to kill.
+- CPU limit, processes are not killed, but the processes are prevented from consuming more CPU than allowed.
 
-To configure cgroups in Gitaly, add `gitaly['cgroups']` to `/etc/gitlab/gitlab.rb`.
+You configure repository cgroups for your GitLab installation to protect against system resource starvation from a few
+large repositories or bad actors.
 
-For example:
+### Configure repository cgroups (new method)
 
-```ruby
-# in /etc/gitlab/gitlab.rb
-gitaly['cgroups_mountpoint'] = "/sys/fs/cgroup"
-gitaly['cgroups_hierarchy_root'] =>"gitaly"
-gitaly['cgroups_memory_bytes'] = 64424509440,  # 60gb
-gitaly['cgroups_cpu_shares'] = 1024
-gitaly['cgroups_repositories_count'] => 1000,
-gitaly['cgroups_repositories_memory_bytes'] => 32212254720 # 20gb
-gitaly['cgroups_repositories_cpu_shares'] => 512
-```
+> This method of configuring repository cgroups was introduced in GitLab 15.1.
+
+To configure repository cgroups in Gitaly using the new method, use the following settings for the new configuration method
+to `gitaly['cgroups']` in `/etc/gitlab/gitlab.rb`:
 
 - `cgroups_mountpoint` is where the parent cgroup directory is mounted. Defaults to `/sys/fs/cgroup`.
 - `cgroups_hierarchy_root` is the parent cgroup under which Gitaly creates groups, and
@@ -873,7 +861,7 @@ gitaly['cgroups_repositories_cpu_shares'] => 512
    when Gitaly starts.
 - `cgroups_memory_bytes` is the total memory limit that is imposed collectively on all
    Git processes that Gitaly spawns. 0 implies no limit.
-- `cgroups_cpu_shares` is the cpu limit that is imposed collectively on all Git
+- `cgroups_cpu_shares` is the CPU limit that is imposed collectively on all Git
    processes that Gitaly spawns. 0 implies no limit. The maximum is 1024 shares,
    which represents 100% of CPU.
 - `cgroups_repositories_count` is the number of cgroups in the cgroups pool. Each time a new Git
@@ -881,30 +869,30 @@ gitaly['cgroups_repositories_cpu_shares'] => 512
   on the repository the command is for. A circular hashing algorithm assigns
   Git commands to these cgroups, so a Git command for a repository is
   always assigned to the same cgroup.
-- `cgroups_repositories_memory_bytes` is the total memory limit that is imposed collectively on all
-  Git processes that Gitaly spawns. 0 implies no limit. This value cannot exceed
-  that of the top level `cgroups_memory_bytes`.
-- `cgroups_repositories_cpu_shares` is the CPU limit that is imposed collectively on all Git
-  processes Gitaly spawns. 0 implies no limit. The maximum is 1024 shares,
-  which represents 100% of CPU. This value cannot exceed that of the top
-  level`cgroups_cpu_shares`.
-
-#### Configure cgroups (legacy method)
+- `cgroups_repositories_memory_bytes` is the total memory limit imposed on all Git processes contained in a repository cgroup.
+  0 implies no limit. This value cannot exceed that of the top level `cgroups_memory_bytes`.
+- `cgroups_repositories_cpu_shares` is the CPU limit that is imposed on all Git processes contained in a repository cgroup.
+  0 implies no limit. The maximum is 1024 shares, which represents 100% of CPU.
+  This value cannot exceed that of the top level`cgroups_cpu_shares`.
 
-To configure cgroups in Gitaly for GitLab versions using the legacy method, add `gitaly['cgroups']` to `/etc/gitlab/gitlab.rb`. For
-example:
+For example:
 
 ```ruby
 # in /etc/gitlab/gitlab.rb
-gitaly['cgroups_count'] = 1000
 gitaly['cgroups_mountpoint'] = "/sys/fs/cgroup"
-gitaly['cgroups_hierarchy_root'] = "gitaly"
-gitaly['cgroups_memory_limit'] = 32212254720
-gitaly['cgroups_memory_enabled'] = true
+gitaly['cgroups_hierarchy_root'] => "gitaly"
+gitaly['cgroups_memory_bytes'] = 64424509440,  # 60gb
 gitaly['cgroups_cpu_shares'] = 1024
-gitaly['cgroups_cpu_enabled'] = true
+gitaly['cgroups_repositories_count'] => 1000,
+gitaly['cgroups_repositories_memory_bytes'] => 32212254720 # 20gb
+gitaly['cgroups_repositories_cpu_shares'] => 512
 ```
 
+### Configure repository cgroups (legacy method)
+
+To configure repository cgroups in Gitaly using the legacy method, use the following settings
+in `/etc/gitlab/gitlab.rb`:
+
 - `cgroups_count` is the number of cgroups created. Each time a new
    command is spawned, Gitaly assigns it to one of these cgroups based
    on the command line arguments of the command. A circular hashing algorithm assigns
@@ -917,7 +905,21 @@ gitaly['cgroups_cpu_enabled'] = true
 - `cgroups_memory_enabled` enables or disables the memory limit on cgroups.
 - `cgroups_memory_bytes` is the total memory limit each cgroup imposes on the processes added to it.
 - `cgroups_cpu_enabled` enables or disables the CPU limit on cgroups.
-- `cgroups_cpu_shares` is the CPU limit each cgroup imposes on the processes added to it. The maximum is 1024 shares, which represents 100% of CPU.
+- `cgroups_cpu_shares` is the CPU limit each cgroup imposes on the processes added to it. The maximum is 1024 shares,
+  which represents 100% of CPU.
+
+For example:
+
+```ruby
+# in /etc/gitlab/gitlab.rb
+gitaly['cgroups_count'] = 1000
+gitaly['cgroups_mountpoint'] = "/sys/fs/cgroup"
+gitaly['cgroups_hierarchy_root'] = "gitaly"
+gitaly['cgroups_memory_limit'] = 32212254720
+gitaly['cgroups_memory_enabled'] = true
+gitaly['cgroups_cpu_shares'] = 1024
+gitaly['cgroups_cpu_enabled'] = true
+```
 
 ### Configuring oversubscription
 
@@ -926,16 +928,15 @@ In the previous example using the new configuration method:
 - The top level memory limit is capped at 60gb.
 - Each of the 1000 cgroups in the repositories pool is capped at 20gb.
 
-This is called "oversubscription". Each cgroup in the pool has a much larger capacity than 1/1000th
+This configuration leads to "oversubscription". Each cgroup in the pool has a much larger capacity than 1/1000th
 of the top-level memory limit.
 
 This strategy has two main benefits:
 
-- It gives the host protection from overall memory starvation (OOM), because the top-level
-  cgroup's memory limit can be set to a threshold smaller than the host's
-  capacity. Processes outside of that cgroup are not at risk of OOM.
+- It gives the host protection from overall memory starvation (OOM), because the memory limit of the top-level cgroup
+  can be set to a threshold smaller than the host's capacity. Processes outside of that cgroup are not at risk of OOM.
 - It allows each individual cgroup in the pool to burst up to a generous upper
-  bound (in this example 20 GB) that is smaller than the parent cgroup's limit,
+  bound (in this example 20 GB) that is smaller than the limit of the parent cgroup,
   but substantially larger than 1/N of the parent's limit. In this example, up
   to 3 child cgroups can concurrently burst up to their max. In general, all
   1000 cgroups would use much less than the 20 GB.
@@ -1141,8 +1142,7 @@ gitaly['pack_objects_cache_enabled'] = true
 #### `enabled` defaults to `false`
 
 The cache is disabled by default. This is because in some cases, it
-can create an [extreme
-increase](https://gitlab.com/gitlab-com/gl-infra/production/-/issues/4010#note_534564684)
+can create an [extreme increase](https://gitlab.com/gitlab-com/gl-infra/production/-/issues/4010#note_534564684)
 in the number of bytes written to disk. On GitLab.com, we have verified
 that our repository storage disks can handle this extra workload, but
 we felt we cannot assume this is true everywhere.
@@ -1302,14 +1302,26 @@ process repositories that do not pass consistency checks.
 For Omnibus GitLab installations, edit `/etc/gitlab/gitlab.rb` and set the
 following keys (in this example, to disable the `hasDotgit` consistency check):
 
-```ruby
-ignored_git_errors = ["hasDotgit = ignore"]
-omnibus_gitconfig['system'] = {
-        "fsck" => ignored_git_errors,
-        "fetch.fsck" => ignored_git_errors,
-        "receive.fsck" => ignored_git_errors,
-}
-```
+- In [GitLab 15.3](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/6800) and later:
+
+  ```ruby
+  gitaly['gitconfig'] = [
+   { key: "fsck.hasDotgit", value: "ignore" },
+   { key: "fetch.fsck.hasDotgit", value: "ignore" },
+   { key: "receive.fsck.hasDotgit", value: "ignore "},
+  ]
+  ```
+
+- In GitLab 15.2 and earlier (legacy method):
+
+  ```ruby
+  ignored_git_errors = ["hasDotgit = ignore"]
+  omnibus_gitconfig['system'] = {
+    "fsck" => ignored_git_errors,
+    "fetch.fsck" => ignored_git_errors,
+    "receive.fsck" => ignored_git_errors,
+  }
+  ```
 
 For source installs, edit the Gitaly configuration (`gitaly.toml`) to do the
 equivalent: