7 files changed, 395 insertions, 94 deletions

diff --git a/doc/administration/gitaly/configure_gitaly.md b/doc/administration/gitaly/configure_gitaly.md
index b26ae84eef0..9f3afae0e9f 100644
--- a/doc/administration/gitaly/configure_gitaly.md
+++ b/doc/administration/gitaly/configure_gitaly.md
@@ -276,11 +276,11 @@ Configure Gitaly server in one of two ways:
       storage: [
          {
             name: 'default',
-            path: '/var/opt/gitlab/git-data',
+            path: '/var/opt/gitlab/git-data/repositories',
          },
          {
             name: 'storage1',
-            path: '/mnt/gitlab/git-data',
+            path: '/mnt/gitlab/git-data/repositories',
          },
       ],
    }
@@ -294,7 +294,7 @@ Configure Gitaly server in one of two ways:
       storage: [
          {
             name: 'storage2',
-            path: '/srv/gitlab/git-data',
+            path: '/srv/gitlab/git-data/repositories',
          },
       ],
    }
@@ -517,7 +517,7 @@ gitaly['configuration'] = {
   storage: [
     {
       name: 'storage1',
-      path: '/mnt/gitlab/git-data',
+      path: '/mnt/gitlab/git-data/repositories',
     },
   ],
 }
@@ -786,13 +786,13 @@ gitaly['configuration'] = {
       {
          rpc: '/gitaly.SmartHTTPService/PostUploadPackWithSidechannel',
          max_per_repo: 20,
-         max_queue_time: '1s',
+         max_queue_wait: '1s',
          max_queue_size: 10,
       },
       {
          rpc: '/gitaly.SSHService/SSHUploadPackWithSidechannel',
          max_per_repo: 20,
-         max_queue_time: '1s',
+         max_queue_wait: '1s',
          max_queue_size: 10,
       },
    ],
@@ -801,7 +801,7 @@ gitaly['configuration'] = {
 
 - `rpc` is the name of the RPC to set a concurrency limit for per repository.
 - `max_per_repo` is the maximum number of in-flight RPC calls for the given RPC per repository.
-- `max_queue_time` is the maximum amount of time a request can wait in the concurrency queue to
+- `max_queue_wait` is the maximum amount of time a request can wait in the concurrency queue to
   be picked up by Gitaly.
 - `max_queue_size` is the maximum size the concurrency queue (per RPC method) can grow to before requests are rejected by
   Gitaly.
@@ -822,6 +822,50 @@ When these limits are reached, users are disconnected.
 You can observe the behavior of this queue using the Gitaly logs and Prometheus. For more
 information, see the [relevant documentation](monitoring.md#monitor-gitaly-concurrency-limiting).
 
+## Limit pack-objects concurrency
+
+> - [Introduced](https://gitlab.com/groups/gitlab-org/-/epics/7891) in GitLab 15.11 [with a flag](../../administration/feature_flags.md) named `gitaly_pack_objects_limiting_remote_ip`. Disabled by default.
+> - [Generally available](https://gitlab.com/gitlab-org/gitaly/-/merge_requests/5772) in GitLab 16.0. Feature flag `gitaly_pack_objects_limiting_remote_ip` removed.
+
+Gitaly triggers `git-pack-objects` processes when handling both SSH and HTTPS traffic to clone or pull repositories. These processes generate a `pack-file` and can
+consume a significant amount of resources, especially in situations such as unexpectedly high traffic or concurrent pulls from a large repository. On GitLab.com, we also
+observe problems with clients that have slow internet connections.
+
+You can limit these processes from overwhelming your Gitaly server by setting pack-objects concurrency limits in the Gitaly configuration file. This setting limits the
+number of in-flight pack-object processes per remote IP address.
+
+WARNING:
+Only enable these limits on your environment with caution and only in select circumstances, such as to protect against unexpected traffic. When reached, these limits
+disconnect users. For consistent and stable performance, you should first explore other options such as adjusting node specifications, and
+[reviewing large repositories](../../user/project/repository/managing_large_repositories.md) or workloads.
+
+Example configuration:
+
+```ruby
+# in /etc/gitlab/gitlab.rb
+gitaly['pack_objects_limiting'] = {
+   'max_concurrency' => 15,
+   'max_queue_length' => 200,
+   'max_queue_wait' => '60s',
+}
+```
+
+- `max_concurrency` is the maximum number of in-flight pack-object processes per key.
+- `max_queue_length` is the maximum size the concurrency queue (per key) can grow to before requests are rejected by Gitaly.
+- `max_queue_wait` is the maximum amount of time a request can wait in the concurrency queue to be picked up by Gitaly.
+
+In the example above:
+
+- Each remote IP can have at most 15 simultaneous pack-object processes in flight on a Gitaly node.
+- If another request comes in from an IP that has used up its 15 slots, that request gets queued.
+- If a request waits in the queue for more than 1 minute, it is rejected with an error.
+- If the queue grows beyond 200, subsequent requests are rejected with an error.
+
+When the pack-object cache is enabled, pack-objects limiting kicks in only if the cache is missed. For more, see [Pack-objects cache](#pack-objects-cache).
+
+You can observe the behavior of this queue using Gitaly logs and Prometheus. For more information, see
+[Monitor Gitaly pack-objects concurrency limiting](monitoring.md#monitor-gitaly-pack-objects-concurrency-limiting).
+
 ## Control groups
 
 WARNING:
@@ -831,10 +875,6 @@ When reached, limits _do_ result in disconnects that negatively impact users.
 For consistent and stable performance, you should first explore other options such as
 adjusting node specifications, and [reviewing large repositories](../../user/project/repository/managing_large_repositories.md) or workloads.
 
-FLAG:
-On self-managed GitLab, by default repository cgroups are not available. To make it available, an administrator can
-[enable the feature flag](../feature_flags.md) named `gitaly_run_cmds_in_cgroup`.
-
 When enabling cgroups for memory, you should ensure that no swap is configured on the Gitaly nodes as
 processes may switch to using that instead of being terminated. This situation could lead to notably compromised
 performance.
@@ -1288,47 +1328,34 @@ the cache hit rate.
 
 ### Observe the cache
 
-The cache can be observed [using metrics](monitoring.md#pack-objects-cache) and in the following logged
-information:
+> Logs for pack-objects caching was [changed](https://gitlab.com/gitlab-org/gitaly/-/merge_requests/5719) in GitLab 16.0.
 
-|Message|Fields|Description|
-|:---|:---|:---|
-|`generated bytes`|`bytes`, `cache_key`|Logged when an entry was added to the cache|
-|`served bytes`|`bytes`, `cache_key`|Logged when an entry was read from the cache|
+You can observe the cache [using metrics](monitoring.md#pack-objects-cache) and in the following logged information. These logs are part of the gRPC logs and can
+be discovered when a call is executed.
+
+| Field | Description |
+|:---|:---|
+| `pack_objects_cache.hit` | Indicates whether the current pack-objects cache was hit (`true` or `false`) |
+| `pack_objects_cache.key` | Cache key used for the pack-objects cache |
+| `pack_objects_cache.generated_bytes` | Size (in bytes) of the new cache being written |
+| `pack_objects_cache.served_bytes` | Size (in bytes) of the cache being served |
+| `pack_objects.compression_statistics` | Statistics regarding pack-objects generation |
+| `pack_objects.enumerate_objects_ms` | Total time (in ms) spent enumerating objects sent by clients |
+| `pack_objects.prepare_pack_ms` | Total time (in ms) spent preparing the packfile before sending it back to the client |
+| `pack_objects.write_pack_file_ms` | Total time (in ms) spent sending back the packfile to the client. Highly dependent on the client's internet connection |
+| `pack_objects.written_object_count` | Total number of objects Gitaly sends back to the client |
 
 In the case of a:
 
-- Cache miss, Gitaly logs both a `generated bytes` and a `served bytes` message.
-- Cache hit, Gitaly logs only a `served bytes` message.
+- Cache miss, Gitaly logs both a `pack_objects_cache.generated_bytes` and `pack_objects_cache.served_bytes` message. Gitaly also logs some more detailed statistics of
+  pack-object generation.
+- Cache hit, Gitaly logs only a `pack_objects_cache.served_bytes` message.
 
 Example:
 
 ```json
 {
   "bytes":26186490,
-  "cache_key":"1b586a2698ca93c2529962e85cda5eea8f0f2b0036592615718898368b462e19",
-  "correlation_id":"01F1MY8JXC3FZN14JBG1H42G9F",
-  "grpc.meta.deadline_type":"none",
-  "grpc.method":"PackObjectsHook",
-  "grpc.request.fullMethod":"/gitaly.HookService/PackObjectsHook",
-  "grpc.request.glProjectPath":"root/gitlab-workhorse",
-  "grpc.request.glRepository":"project-2",
-  "grpc.request.repoPath":"@hashed/d4/73/d4735e3a265e16eee03f59718b9b5d03019c07d8b6c51f90da3a666eec13ab35.git",
-  "grpc.request.repoStorage":"default",
-  "grpc.request.topLevelGroup":"@hashed",
-  "grpc.service":"gitaly.HookService",
-  "grpc.start_time":"2021-03-25T14:57:52.747Z",
-  "level":"info",
-  "msg":"generated bytes",
-  "peer.address":"@",
-  "pid":20961,
-  "span.kind":"server",
-  "system":"grpc",
-  "time":"2021-03-25T14:57:53.543Z"
-}
-{
-  "bytes":26186490,
-  "cache_key":"1b586a2698ca93c2529962e85cda5eea8f0f2b0036592615718898368b462e19",
   "correlation_id":"01F1MY8JXC3FZN14JBG1H42G9F",
   "grpc.meta.deadline_type":"none",
   "grpc.method":"PackObjectsHook",
@@ -1341,12 +1368,23 @@ Example:
   "grpc.service":"gitaly.HookService",
   "grpc.start_time":"2021-03-25T14:57:52.747Z",
   "level":"info",
-  "msg":"served bytes",
+  "msg":"finished unary call with code OK",
   "peer.address":"@",
   "pid":20961,
   "span.kind":"server",
   "system":"grpc",
-  "time":"2021-03-25T14:57:53.543Z"
+  "time":"2021-03-25T14:57:53.543Z",
+  "pack_objects.compression_statistics": "Total 145991 (delta 68), reused 6 (delta 2), pack-reused 145911",
+  "pack_objects.enumerate_objects_ms": 170,
+  "pack_objects.prepare_pack_ms": 7,
+  "pack_objects.write_pack_file_ms": 786,
+  "pack_objects.written_object_count": 145991,
+  "pack_objects_cache.generated_bytes": 49533030,
+  "pack_objects_cache.hit": "false",
+  "pack_objects_cache.key": "123456789",
+  "pack_objects_cache.served_bytes": 49533030,
+  "peer.address": "127.0.0.1",
+  "pid": 8813,
 }
 ```
 
@@ -1496,7 +1534,13 @@ value = "/etc/gitlab/instance_wide_ignored_git_blobs.txt"
 
 ## Configure commit signing for GitLab UI commits
 
-> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/19185) in GitLab 15.4.
+> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/19185) in GitLab 15.4.
+> - Displaying **Verified** badge for signed GitLab UI commits [introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/124218) in GitLab 16.3 [with a flag](../feature_flags.md) named `gitaly_gpg_signing`. Disabled by default.
+
+FLAG:
+On self-managed GitLab, by default this feature is not available. To make it available,
+an administrator can [enable the feature flag](../feature_flags.md) named `gitaly_gpg_signing`.
+On GitLab.com, this feature is not available.
 
 By default, Gitaly doesn't sign commits made using GitLab UI. For example, commits made using:
 
@@ -1513,13 +1557,21 @@ Configure Gitaly to sign commits made with the GitLab UI in one of two ways:
 
 :::TabTitle Linux package (Omnibus)
 
-1. [Create a GPG key](../../user/project/repository/gpg_signed_commits/index.md#create-a-gpg-key)
-   and export it. For optimal performance, consider using an EdDSA key.
+1. [Create a GPG key](../../user/project/repository/signed_commits/gpg.md#create-a-gpg-key)
+   and export it, or [create an SSH key](../../user/ssh.md#generate-an-ssh-key-pair). For optimal performance, use an EdDSA key.
+
+   Export GPG key:
 
    ```shell
    gpg --export-secret-keys <ID> > signing_key.gpg
    ```
 
+   Or create an SSH key (with no passphrase):
+
+   ```shell
+   ssh-keygen -t ed25519 -f signing_key.ssh
+   ```
+
 1. On the Gitaly nodes, copy the key into `/etc/gitlab/gitaly/`.
 1. Edit `/etc/gitlab/gitlab.rb` and configure `gitaly['gpg_signing_key_path']`:
 
@@ -1537,13 +1589,21 @@ Configure Gitaly to sign commits made with the GitLab UI in one of two ways:
 
 :::TabTitle Self-compiled (source)
 
-1. [Create a GPG key](../../user/project/repository/gpg_signed_commits/index.md#create-a-gpg-key)
-   and export it. For optimal performance, consider using an EdDSA key.
+1. [Create a GPG key](../../user/project/repository/signed_commits/gpg.md#create-a-gpg-key)
+   and export it, or [create an SSH key](../../user/ssh.md#generate-an-ssh-key-pair). For optimal performance, use an EdDSA key.
+
+   Export GPG key:
 
    ```shell
    gpg --export-secret-keys <ID> > signing_key.gpg
    ```
 
+   Or create an SSH key (with no passphrase):
+
+   ```shell
+   ssh-keygen -t ed25519 -f signing_key.ssh
+   ```
+
 1. On the Gitaly nodes, copy the key into `/etc/gitlab`.
 1. Edit `/home/git/gitaly/config.toml` and configure `signing_key`:
 
@@ -1610,3 +1670,172 @@ Gitaly fails to start up if either:
 
 - The configuration command fails.
 - The output produced by the command cannot be parsed as valid JSON.
+
+## Configure server-side backups
+
+> [Introduced](https://gitlab.com/gitlab-org/gitaly/-/issues/4941) in GitLab 16.3.
+
+Repository backups can be configured so that the Gitaly node that hosts each
+repository is responsible for creating the backup and streaming it to
+object storage. This helps reduce the network resources required to create and
+restore a backup.
+
+Each Gitaly node must be configured to connect to object storage for backups.
+
+After configuring server-side backups, you can
+[create a server-side repository backup](../backup_restore/backup_gitlab.md#create-server-side-repository-backups).
+
+### Configure Azure Blob storage
+
+How you configure Azure Blob storage for backups depends on the type of installation you have. For self-compiled installations, you must set
+the `AZURE_STORAGE_ACCOUNT` and `AZURE_STORAGE_KEY` environment variables outside of GitLab.
+
+::Tabs
+
+:::TabTitle Linux package (Omnibus)
+
+Edit `/etc/gitlab/gitlab.rb` and configure the `go_cloud_url`:
+
+```ruby
+gitaly['env'] = {
+    'AZURE_STORAGE_ACCOUNT' => 'azure_storage_account',
+    'AZURE_STORAGE_KEY' => 'azure_storage_key' # or 'AZURE_STORAGE_SAS_TOKEN'
+}
+gitaly['configuration'] = {
+    backup: {
+        go_cloud_url: 'azblob://gitaly-backups'
+    }
+}
+```
+
+:::TabTitle Self-compiled (source)
+
+Edit `/home/git/gitaly/config.toml` and configure `go_cloud_url`:
+
+```toml
+[backup]
+go_cloud_url = "azblob://gitaly-backups"
+```
+
+::EndTabs
+
+### Configure Google Cloud storage
+
+Google Cloud storage (GCP) authenticates using Application Default Credentials. Set up Application Default Credentials on each Gitaly server using either:
+
+- The [`gcloud auth application-default login`](https://cloud.google.com/sdk/gcloud/reference/auth/application-default/login) command.
+- The `GOOGLE_APPLICATION_CREDENTIALS` environment variable. For self-compiled installations, set the environment
+  variable outside of GitLab.
+
+For more information, see [Application Default Credentials](https://cloud.google.com/docs/authentication/provide-credentials-adc).
+
+The destination bucket is configured using the `go_cloud_url` option.
+
+::Tabs
+
+:::TabTitle Linux package (Omnibus)
+
+Edit `/etc/gitlab/gitlab.rb` and configure the `go_cloud_url`:
+
+```ruby
+gitaly['env'] = {
+    'GOOGLE_APPLICATION_CREDENTIALS' => '/path/to/service.json'
+}
+gitaly['configuration'] = {
+    backup: {
+        go_cloud_url: 'gs://gitaly-backups'
+    }
+}
+```
+
+:::TabTitle Self-compiled (source)
+
+Edit `/home/git/gitaly/config.toml` and configure `go_cloud_url`:
+
+```toml
+[backup]
+go_cloud_url = "gs://gitaly-backups"
+```
+
+::EndTabs
+
+### Configure S3 storage
+
+To configure S3 storage authentication:
+
+- If you authenticate with the AWS CLI, you can use the default AWS session.
+- Otherwise, you can use the `AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY` environment variables. For self-compiled installations, set the environment
+  variables outside of GitLab.
+
+For more information, see [AWS Session documentation](https://docs.aws.amazon.com/sdk-for-go/api/aws/session/).
+
+The destination bucket and region are configured using the `go_cloud_url` option.
+
+::Tabs
+
+:::TabTitle Linux package (Omnibus)
+
+Edit `/etc/gitlab/gitlab.rb` and configure the `go_cloud_url`:
+
+```ruby
+gitaly['env'] = {
+    'AWS_ACCESS_KEY_ID' => 'aws_access_key_id',
+    'AWS_SECRET_ACCESS_KEY' => 'aws_secret_access_key'
+}
+gitaly['configuration'] = {
+    backup: {
+        go_cloud_url: 's3://gitaly-backups?region=us-west-1'
+    }
+}
+```
+
+:::TabTitle Self-compiled (source)
+
+Edit `/home/git/gitaly/config.toml` and configure `go_cloud_url`:
+
+```toml
+[backup]
+go_cloud_url = "s3://gitaly-backups?region=us-west-1"
+```
+
+::EndTabs
+
+#### Configure S3-compatible servers
+
+S3-compatible servers such as MinIO are configured similarly to S3 with the addition of the `endpoint` parameter.
+
+The following parameters are supported:
+
+- `region`: The AWS region.
+- `endpoint`: The endpoint URL.
+- `disabledSSL`: A value of `true` disables SSL.
+- `s3ForcePathStyle`: A value of `true` forces path-style addressing.
+
+::Tabs
+
+:::TabTitle Linux package (Omnibus)
+
+Edit `/etc/gitlab/gitlab.rb` and configure the `go_cloud_url`:
+
+```ruby
+gitaly['env'] = {
+    'AWS_ACCESS_KEY_ID' => 'minio_access_key_id',
+    'AWS_SECRET_ACCESS_KEY' => 'minio_secret_access_key'
+}
+gitaly['configuration'] = {
+    backup: {
+        go_cloud_url: 's3://gitaly-backups?region=minio&endpoint=my.minio.local:8080&disableSSL=true&s3ForcePathStyle=true'
+    }
+}
+```
+
+:::TabTitle Self-compiled (source)
+
+Edit `/home/git/gitaly/config.toml` and configure `go_cloud_url`:
+
+```toml
+[backup]
+go_cloud_url = "s3://gitaly-backups?region=minio&endpoint=my.minio.local:8080&disableSSL=true&s3ForcePathStyle=true"
+```
+
+::EndTabs
diff --git a/doc/administration/gitaly/index.md b/doc/administration/gitaly/index.md
index db11ac8c769..46f6a5829c8 100644
--- a/doc/administration/gitaly/index.md
+++ b/doc/administration/gitaly/index.md
@@ -25,6 +25,7 @@ Gitaly implements a client-server architecture:
   - [GitLab Shell](https://gitlab.com/gitlab-org/gitlab-shell)
   - [GitLab Workhorse](https://gitlab.com/gitlab-org/gitlab-workhorse)
   - [GitLab Elasticsearch Indexer](https://gitlab.com/gitlab-org/gitlab-elasticsearch-indexer)
+  - [GitLab Zoekt Indexer](https://gitlab.com/gitlab-org/gitlab-zoekt-indexer)
 
 Gitaly manages only Git repository access for GitLab. Other types of GitLab data aren't accessed
 using Gitaly.
@@ -57,7 +58,7 @@ If you have not yet migrated to Gitaly Cluster, you have two options:
 - A sharded Gitaly instance.
 - Gitaly Cluster.
 
-Contact your [Customer Success Manager](https://about.gitlab.com/job-families/sales/customer-success-management/) or customer support if you have any questions.
+Contact your [Customer Success Manager](https://handbook.gitlab.com/job-families/sales/customer-success-management/) or customer support if you have any questions.
 
 ### Known issues
 
@@ -326,7 +327,7 @@ conflicts that could occur due to partially applied operations.
 
 Repositories are stored in the storages at the relative path determined by the [Gitaly client](#gitaly-architecture). These paths can be
 identified by them not beginning with the `@cluster` prefix. The relative paths
-follow the [hashed storage](../repository_storage_types.md#hashed-storage) schema.
+follow the [hashed storage](../repository_storage_paths.md#hashed-storage) schema.
 
 #### Praefect-generated replica paths (GitLab 15.0 and later)
 
@@ -377,7 +378,7 @@ Use the [`praefect metadata`](troubleshooting.md#view-repository-metadata) subco
 
 The repository on disk also contains the project path in the Git configuration file. The configuration
 file can be used to determine the project path even if the repository's metadata has been deleted.
-Follow the [instructions in hashed storage's documentation](../repository_storage_types.md#from-hashed-path-to-project-name).
+Follow the [instructions in hashed storage's documentation](../repository_storage_paths.md#from-hashed-path-to-project-name).
 
 #### Atomicity of operations
 
diff --git a/doc/administration/gitaly/monitoring.md b/doc/administration/gitaly/monitoring.md
index 00d0499faa2..cbf5722f2c5 100644
--- a/doc/administration/gitaly/monitoring.md
+++ b/doc/administration/gitaly/monitoring.md
@@ -31,18 +31,64 @@ of requests dropped due to request limiting. The `reason` label indicates why a
 
 ## Monitor Gitaly concurrency limiting
 
-You can observe specific behavior of [concurrency-queued requests](configure_gitaly.md#limit-rpc-concurrency) using
-the Gitaly logs and Prometheus:
-
-- In the [Gitaly logs](../logs/index.md#gitaly-logs), look for the string (or structured log field)
-  `acquire_ms`. Messages that have this field are reporting about the concurrency limiter.
-- In Prometheus, look for the following metrics:
-  - `gitaly_concurrency_limiting_in_progress` indicates how many concurrent requests are
-    being processed.
-  - `gitaly_concurrency_limiting_queued` indicates how many requests for an RPC for a given
-    repository are waiting due to the concurrency limit being reached.
-  - `gitaly_concurrency_limiting_acquiring_seconds` indicates how long a request has to
-    wait due to concurrency limits before being processed.
+You can observe specific behavior of [concurrency-queued requests](configure_gitaly.md#limit-rpc-concurrency) using Gitaly logs and Prometheus.
+
+In the [Gitaly logs](../logs/index.md#gitaly-logs), you can identify logs related to the pack-objects concurrency limiting with entries such as:
+
+| Log Field | Description |
+| --- | --- |
+| `limit.concurrency_queue_length` | Indicates the current length of the queue specific to the RPC type of the ongoing call. It provides insight into the number of requests waiting to be processed due to concurrency limits.                                       |
+| `limit.concurrency_queue_ms`     | Represents the duration, in milliseconds, that a request has spent waiting in the queue due to the limit on concurrent RPCs. This field helps understand the impact of concurrency limits on request processing times.           |
+| `limit.concurrency_dropped`      | If the request is dropped due to limits being reached, this field specifies the reason: either `max_time` (request waited in the queue longer than the maximum allowed time) or `max_size` (the queue reached its maximum size). |
+| `limit.limiting_key`             | Identifies the key used for limiting.  |
+| `limit.limiting_type`            | Specifies the type of process being limited. In this context, it's `per-rpc`, indicating that the concurrency limiting is applied on a per-RPC basis.                                                                            |
+
+For example:
+
+```json
+{
+  "limit .concurrency_queue_length": 1,
+  "limit .concurrency_queue_ms": 0,
+  "limit.limiting_key": "@hashed/79/02/7902699be42c8a8e46fbbb450172651786b22c56a189f7625a6da49081b2451.git",
+  "limit.limiting_type": "per-rpc"
+}
+```
+
+In Prometheus, look for the following metrics:
+
+- `gitaly_concurrency_limiting_in_progress` indicates how many concurrent requests are being processed.
+- `gitaly_concurrency_limiting_queued` indicates how many requests for an RPC for a given repository are waiting due to the concurrency limit being reached.
+- `gitaly_concurrency_limiting_acquiring_seconds` indicates how long a request has to wait due to concurrency limits before being processed.
+
+## Monitor Gitaly pack-objects concurrency limiting
+
+You can observe specific behavior of [pack-objects limiting](configure_gitaly.md#limit-pack-objects-concurrency) using Gitaly logs and Prometheus.
+
+In the [Gitaly logs](../logs/index.md#gitaly-logs), you can identify logs related to the pack-objects concurrency limiting with entries such as:
+
+| Log Field | Description |
+|:---|:---|
+| `limit.concurrency_queue_length` | Current length of the queue for the pack-objects processes. Indicates the number of requests that are waiting to be processed because the limit on concurrent processes has been reached. |
+| `limit.concurrency_queue_ms` | Time a request has spent waiting in the queue, in milliseconds. Indicates how long a request has had to wait because of the limits on concurrency. |
+| `limit.limiting_key` | Remote IP of the sender. |
+| `limit.limiting_type` | Type of process being limited. In this case, `pack-objects`. |
+
+Example configuration:
+
+```json
+{
+  "limit .concurrency_queue_length": 1,
+  "limit .concurrency_queue_ms": 0,
+  "limit.limiting_key": "1.2.3.4",
+  "limit.limiting_type": "pack-objects"
+}
+```
+
+In Prometheus, look for the following metrics:
+
+- `gitaly_pack_objects_in_progress` indicates how many pack-objects processes are being processed concurrently.
+- `gitaly_pack_objects_queued` indicates how many requests for pack-objects processes are waiting due to the concurrency limit being reached.
+- `gitaly_pack_objects_acquiring_seconds` indicates how long a request for a pack-object process has to wait due to concurrency limits before being processed.
 
 ## Monitor Gitaly cgroups
 
diff --git a/doc/administration/gitaly/praefect.md b/doc/administration/gitaly/praefect.md
index 6eefd0a7bb5..0297f295e6f 100644
--- a/doc/administration/gitaly/praefect.md
+++ b/doc/administration/gitaly/praefect.md
@@ -1133,15 +1133,15 @@ For more information on Gitaly server configuration, see our
       storage: [
         {
           name: 'gitaly-1',
-          path: '/var/opt/gitlab/git-data',
+          path: '/var/opt/gitlab/git-data/repositories',
         },
         {
           name: 'gitaly-2',
-          path: '/var/opt/gitlab/git-data',
+          path: '/var/opt/gitlab/git-data/repositories',
         },
         {
           name: 'gitaly-3',
-          path: '/var/opt/gitlab/git-data',
+          path: '/var/opt/gitlab/git-data/repositories',
         },
       ],
    }
@@ -1337,7 +1337,7 @@ Particular attention should be shown to:
 
 1. Check that the Praefect storage is configured to store new repositories:
 
-   1. On the left sidebar, expand the top-most chevron (**{chevron-down}**).
+   1. On the left sidebar, select **Search or go to**.
    1. Select **Admin Area**.
    1. On the left sidebar, select **Settings > Repository**.
    1. Expand the **Repository storage** section.
@@ -1498,9 +1498,12 @@ For a replication factor:
 > [Introduced](https://gitlab.com/gitlab-org/gitaly/-/issues/4080) in GitLab 15.0.
 
 Praefect stores metadata about the repositories in a database. If the repositories are modified on disk
-without going through Praefect, the metadata can become inaccurate. Because the metadata is used for replication
-and routing decisions, any inaccuracies may cause problems. Praefect contains a background worker that
-periodically verifies the metadata against the actual state on the disks. The worker:
+without going through Praefect, the metadata can become inaccurate. For example if a Gitaly node is
+rebuilt, rather than being replaced with a new node, repository verification ensures this is detected.
+
+The metadata is used for replication and routing decisions, so any inaccuracies may cause problems.
+Praefect contains a background worker that periodically verifies the metadata against the actual state on the disks.
+The worker:
 
 1. Picks up a batch of replicas to verify on healthy storages. The replicas are either unverified or have exceeded
    the configured verification interval. Replicas that have never been verified are prioritized, followed by
@@ -1512,8 +1515,8 @@ periodically verifies the metadata against the actual state on the disks. The wo
 
 The worker acquires an exclusive verification lease on each of the replicas it is about to verify. This avoids multiple
 workers from verifying the same replica concurrently. The worker releases the leases when it has completed its check.
-Praefect contains a background goroutine that releases stale leases every 10 seconds when workers are terminated for
-some reason without releasing the lease.
+If workers are terminated for some reason without releasing the lease, Praefect contains a background goroutine
+that releases stale leases every 10 seconds.
 
 The worker logs each of the metadata removals prior to executing them. The `perform_deletions` key
 indicates whether the invalid metadata records are actually deleted or not. For example:
diff --git a/doc/administration/gitaly/recovery.md b/doc/administration/gitaly/recovery.md
index aa487917ef0..45bde083a1a 100644
--- a/doc/administration/gitaly/recovery.md
+++ b/doc/administration/gitaly/recovery.md
@@ -39,7 +39,7 @@ To use a different name for the replacement node for a Gitaly Cluster that has [
   For example:
 
   ```shell
-  $ sudo /opt/gitlab/embedded/bin/praefect -config /var/opt/gitlab/praefect/config.toml set-replication-factor -virtual-storage default -repository @hashed/3f/db/3fdba35f04dc8c462986c992bcf875546257113072a909c162f7e470e581e278.git -replication-factor 2
+  $ sudo /opt/gitlab/embedded/bin/praefect -config /var/opt/gitlab/praefect/config.toml set-replication-factor -virtual-storage default -relative-path @hashed/3f/db/3fdba35f04dc8c462986c992bcf875546257113072a909c162f7e470e581e278.git -replication-factor 2
 
   current assignments: gitaly-1, gitaly-2
   ```
@@ -224,7 +224,7 @@ repository so care must be taken.
 
 ```shell
 sudo /opt/gitlab/embedded/bin/praefect -config /var/opt/gitlab/praefect/config.toml accept-dataloss
--virtual-storage <virtual-storage> -repository <relative-path> -authoritative-storage <storage-name>
+-virtual-storage <virtual-storage> -relative-path <relative-path> -authoritative-storage <storage-name>
 ```
 
 ### Enable writes or accept data loss
@@ -237,7 +237,7 @@ Praefect provides the following subcommands to re-enable writes or accept data l
 of the up-to-date nodes back online, you might have to accept data loss:
 
 ```shell
-sudo /opt/gitlab/embedded/bin/praefect -config /var/opt/gitlab/praefect/config.toml accept-dataloss -virtual-storage <virtual-storage> -repository <relative-path> -authoritative-storage <storage-name>
+sudo /opt/gitlab/embedded/bin/praefect -config /var/opt/gitlab/praefect/config.toml accept-dataloss -virtual-storage <virtual-storage> -relative-path <relative-path> -authoritative-storage <storage-name>
 ```
 
 When accepting data loss, Praefect:
@@ -338,7 +338,7 @@ The `remove-repository` Praefect sub-command removes a repository from a Gitaly
 In GitLab 14.6 and later, by default, the command operates in dry-run mode. In earlier versions, the command didn't support dry-run mode. For example:
 
 ```shell
-sudo /opt/gitlab/embedded/bin/praefect -config /var/opt/gitlab/praefect/config.toml remove-repository -virtual-storage <virtual-storage> -repository <repository>
+sudo /opt/gitlab/embedded/bin/praefect -config /var/opt/gitlab/praefect/config.toml remove-repository -virtual-storage <virtual-storage> -relative-path <repository>
 ```
 
 - Replace `<virtual-storage>` with the name of the virtual storage containing the repository.
@@ -349,7 +349,7 @@ sudo /opt/gitlab/embedded/bin/praefect -config /var/opt/gitlab/praefect/config.t
 - In GitLab 14.6 and later, add `-apply` to run the command outside of dry-run mode and remove the repository. For example:
 
   ```shell
-  sudo /opt/gitlab/embedded/bin/praefect -config /var/opt/gitlab/praefect/config.toml remove-repository -virtual-storage <virtual-storage> -repository <repository> -apply
+  sudo /opt/gitlab/embedded/bin/praefect -config /var/opt/gitlab/praefect/config.toml remove-repository -virtual-storage <virtual-storage> -relative-path <repository> -apply
   ```
 
 - `-virtual-storage` is the virtual storage the repository is located in. Virtual storages are configured in `/etc/gitlab/gitlab.rb` under `praefect['configuration']['virtual_storage]` and looks like the following:
@@ -372,7 +372,7 @@ sudo /opt/gitlab/embedded/bin/praefect -config /var/opt/gitlab/praefect/config.t
 
   In this example, the virtual storage to specify is `default` or `storage-1`.
 
-- `-repository` is the repository's relative path in the storage [beginning with `@hashed`](../repository_storage_types.md#hashed-storage).
+- `-repository` is the repository's relative path in the storage [beginning with `@hashed`](../repository_storage_paths.md#hashed-storage).
   For example:
 
   ```plaintext
@@ -440,7 +440,7 @@ inaccessible.
 The `track-repository` Praefect sub-command adds repositories on disk to the Praefect tracking database to be tracked.
 
 ```shell
-sudo /opt/gitlab/embedded/bin/praefect -config /var/opt/gitlab/praefect/config.toml track-repository -virtual-storage <virtual-storage> -authoritative-storage <storage-name> -repository <repository> -replicate-immediately
+sudo /opt/gitlab/embedded/bin/praefect -config /var/opt/gitlab/praefect/config.toml track-repository -virtual-storage <virtual-storage> -authoritative-storage <storage-name> -relative-path <repository> -replica-path <disk_path> -replicate-immediately
 ```
 
 - `-virtual-storage` is the virtual storage the repository is located in. Virtual storages are configured in `/etc/gitlab/gitlab.rb` under `praefect['configuration'][:virtual_storage]` and looks like the following:
@@ -463,13 +463,14 @@ sudo /opt/gitlab/embedded/bin/praefect -config /var/opt/gitlab/praefect/config.t
 
   In this example, the virtual storage to specify is `default` or `storage-1`.
 
-- `-repository` is the repository's relative path in the storage [beginning with `@hashed`](../repository_storage_types.md#hashed-storage).
+- `-relative-path` is the relative path in the virtual storage. Usually [beginning with `@hashed`](../repository_storage_paths.md#hashed-storage).
   For example:
 
   ```plaintext
   @hashed/f5/ca/f5ca38f748a1d6eaf726b8a42fb575c3c71f1864a8143301782de13da2d9202b.git
   ```
 
+- `-replica-path` is the relative path on physical storage. Can start with [`@cluster` or match `relative_path`](../repository_storage_paths.md#gitaly-cluster-storage).
 - `-authoritative-storage` is the storage we want Praefect to treat as the primary. Required if
   [per-repository replication](praefect.md#configure-replication-factor) is set as the replication strategy.
 - `-replicate-immediately`, available in GitLab 14.6 and later, causes the command to replicate the repository to its secondaries immediately.
@@ -523,8 +524,8 @@ If any entry fails these checks, the command aborts prior to attempting to track
     For example:
 
     ```json
-    {"relative_path":"@hashed/f5/ca/f5ca38f748a1d6eaf726b8a42fb575c3c71f1864a8143301782de13da2d9202b.git","authoritative_storage":"gitaly-1","virtual_storage":"default"}
-    {"relative_path":"@hashed/f8/9f/f89f8d0e735a91c5269ab08d72fa27670d000e7561698d6e664e7b603f5c4e40.git","authoritative_storage":"gitaly-2","virtual_storage":"default"}
+    {"relative_path":"@hashed/f5/ca/f5ca38f748a1d6eaf726b8a42fb575c3c71f1864a8143301782de13da2d9202b.git","replica_path":"@cluster/fe/d3/1","authoritative_storage":"gitaly-1","virtual_storage":"default"}
+    {"relative_path":"@hashed/f8/9f/f89f8d0e735a91c5269ab08d72fa27670d000e7561698d6e664e7b603f5c4e40.git","replica_path":"@cluster/7b/28/2","authoritative_storage":"gitaly-2","virtual_storage":"default"}
     ```
 
 - `-replicate-immediately`, causes the command to replicate the repository to its secondaries immediately.
diff --git a/doc/administration/gitaly/reference.md b/doc/administration/gitaly/reference.md
index aa04c9a92c4..5b26c93cea8 100644
--- a/doc/administration/gitaly/reference.md
+++ b/doc/administration/gitaly/reference.md
@@ -155,25 +155,43 @@ Prometheus query to see the hit rate:
 sum(rate(gitaly_catfile_cache_total{type="hit"}[5m])) / sum(rate(gitaly_catfile_cache_total{type=~"(hit)|(miss)"}[5m]))
 ```
 
-### GitLab Shell
+#### Custom Hooks
 
-For historical reasons
-[GitLab Shell](https://gitlab.com/gitlab-org/gitlab-shell) contains
-the Git hooks that allow GitLab to validate and react to Git pushes.
-Because Gitaly "owns" Git pushes, GitLab Shell must therefore be
-installed alongside Gitaly.
+> Method of configuring custom hooks directory documented here is preferred from GitLab 16.4. `[gitlab-shell] dir` configuration is no longer required.
+
+Gitaly supports [custom Git hooks](../server_hooks.md) that are
+used to perform tasks based on changes performed in any repository. The custom
+hooks directory is configured in the `[hooks]` section:
 
 | Name | Type | Required | Description |
 | ---- | ---- | -------- | ----------- |
-| `dir` | string | yes | The directory where GitLab Shell is installed.|
+| `custom_hooks_dir` | string | no | The directory where custom Git hooks are installed. |
 
 Example:
 
 ```toml
-[gitlab-shell]
-dir = "/home/git/gitlab-shell"
+[hooks]
+custom_hooks_dir = "/home/git/custom-hooks"
 ```
 
+If left unset, no custom hooks are used.
+
+### GitLab
+
+> Method of configuring custom hooks directory documented here is preferred from GitLab 16.4. `[gitlab-shell] dir` configuration is no longer required.
+
+Gitaly must connect to the GitLab application to perform access
+checks when a user performs a change. The parameters to connect to GitLab must
+be configured in the `[gitlab]` section.
+
+| Name | Type | Required | Description |
+| ---- | ---- | -------- | ----------- |
+| `url` | string | yes | The URL of the GitLab server.
+| `secret` | string | no | The secret token used to authenticate with GitLab. |
+| `secret_file` | string | no | The path of the file containing the secret token used to authenticate with GitLab. |
+
+Only one of `secret` or `secret_file` can be configured.
+
 ### Prometheus
 
 You can optionally configure Gitaly to record histogram latencies on GRPC method
diff --git a/doc/administration/gitaly/troubleshooting.md b/doc/administration/gitaly/troubleshooting.md
index 3d8110e1dab..556bc29b76f 100644
--- a/doc/administration/gitaly/troubleshooting.md
+++ b/doc/administration/gitaly/troubleshooting.md
@@ -20,7 +20,7 @@ and our advice on [parsing the `gitaly/current` file](../logs/log_parsing.md#par
 When using standalone Gitaly servers, you must make sure they are the same version
 as GitLab to ensure full compatibility:
 
-1. On the left sidebar, expand the top-most chevron (**{chevron-down}**).
+1. On the left sidebar, select **Search or go to**.
 1. Select **Admin Area**.
 1. On the left sidebar, select **Overview > Gitaly Servers**.
 1. Confirm all Gitaly servers indicate that they are up to date.
@@ -269,7 +269,7 @@ git push origin +refs/heads/*:refs/heads/* +refs/tags/*:refs/tags/*
 
 Any other namespaces that the administrator wants to push can be included there as well via additional patterns.
 
-### Command line tools cannot connect to Gitaly
+### Command-line tools cannot connect to Gitaly
 
 gRPC cannot reach your Gitaly server if:
 
@@ -556,6 +556,9 @@ You can retrieve a repository's metadata by its Praefect-assigned repository ID:
 sudo /opt/gitlab/embedded/bin/praefect -config /var/opt/gitlab/praefect/config.toml metadata -repository-id <repository-id>
 ```
 
+When the physical path on the physical storage starts with `@cluster`, you can
+[find the repository ID in the physical path](index.md#praefect-generated-replica-paths-gitlab-150-and-later).
+
 You can also retrieve a repository's metadata by its virtual storage and relative path:
 
 ```shell