Add latest changes from gitlab-org/gitlab@master

2 files changed, 129 insertions, 46 deletions

diff --git a/doc/administration/gitaly/configure_gitaly.md b/doc/administration/gitaly/configure_gitaly.md
index b840be1285c..63f2150c936 100644
--- a/doc/administration/gitaly/configure_gitaly.md
+++ b/doc/administration/gitaly/configure_gitaly.md
@@ -822,6 +822,47 @@ When these limits are reached, users are disconnected.
 You can observe the behavior of this queue using the Gitaly logs and Prometheus. For more
 information, see the [relevant documentation](monitoring.md#monitor-gitaly-concurrency-limiting).
 
+## Limit pack-objects concurrency
+
+Gitaly triggers `git-pack-objects` processes when handling both SSH and HTTPS traffic to clone or pull repositories. These processes generate a `pack-file` and can
+consume a significant amount of resources, especially in situations such as unexpectedly high traffic or concurrent pulls from a large repository. On GitLab.com, we also
+observe problems with clients that have slow internet connections.
+
+You can limit these processes from overwhelming your Gitaly server by setting pack-objects concurrency limits in the Gitaly configuration file. This setting limits the
+number of in-flight pack-object processes per remote IP address.
+
+WARNING:
+Only enable these limits on your environment with caution and only in select circumstances, such as to protect against unexpected traffic. When reached, these limits
+disconnect users. For consistent and stable performance, you should first explore other options such as adjusting node specifications, and
+[reviewing large repositories](../../user/project/repository/managing_large_repositories.md) or workloads.
+
+Example configuration:
+
+```ruby
+# in /etc/gitlab/gitlab.rb
+gitaly['pack_objects_limiting'] = {
+   'max_concurrency' => 15,
+   'max_queue_length' => 200,
+   'max_queue_wait' => '60s',
+}
+```
+
+- `max_concurrency` is the maximum number of in-flight pack-object processes per key.
+- `max_queue_length` is the maximum size the concurrency queue (per key) can grow to before requests are rejected by Gitaly.
+- `max_queue_wait` is the maximum amount of time a request can wait in the concurrency queue to be picked up by Gitaly.
+
+In the example above:
+
+- Each remote IP can have at most 15 simultaneous pack-object processes in flight on a Gitaly node.
+- If another request comes in from an IP that has used up its 15 slots, that request gets queued.
+- If a request waits in the queue for more than 1 minute, it is rejected with an error.
+- If the queue grows beyond 200, subsequent requests are rejected with an error.
+
+When the pack-object cache is enabled, pack-objects limiting kicks in only if the cache is missed. For more, see [Pack-objects cache](#pack-objects-cache).
+
+You can observe the behavior of this queue using Gitaly logs and Prometheus. For more information, see
+[Monitor Gitaly pack-objects concurrency limiting](monitoring.md#monitor-gitaly-pack-objects-concurrency-limiting).
+
 ## Control groups
 
 WARNING:
@@ -1288,47 +1329,32 @@ the cache hit rate.
 
 ### Observe the cache
 
-The cache can be observed [using metrics](monitoring.md#pack-objects-cache) and in the following logged
-information:
-
-|Message|Fields|Description|
-|:---|:---|:---|
-|`generated bytes`|`bytes`, `cache_key`|Logged when an entry was added to the cache|
-|`served bytes`|`bytes`, `cache_key`|Logged when an entry was read from the cache|
+You can observe the cache [using metrics](monitoring.md#pack-objects-cache) and in the following logged information. These logs are part of the gRPC logs and can
+be discovered when a call is executed.
+
+| Field | Description |
+|:---|:---|
+| `pack_objects_cache.hit` | Indicates whether the current pack-objects cache was hit (`true` or `false`) |
+| `pack_objects_cache.key` | Cache key used for the pack-objects cache |
+| `pack_objects_cache.generated_bytes` | Size (in bytes) of the new cache being written |
+| `pack_objects_cache.served_bytes` | Size (in bytes) of the cache being served |
+| `pack_objects.compression_statistics` | Statistics regarding pack-objects generation |
+| `pack_objects.enumerate_objects_ms` | Total time (in ms) spent enumerating objects sent by clients |
+| `pack_objects.prepare_pack_ms` | Total time (in ms) spent preparing the packfile before sending it back to the client |
+| `pack_objects.write_pack_file_ms` | Total time (in ms) spent sending back the packfile to the client. Highly dependent on the client's internet connection |
+| `pack_objects.written_object_count` | Total number of objects Gitaly sends back to the client |
 
 In the case of a:
 
-- Cache miss, Gitaly logs both a `generated bytes` and a `served bytes` message.
-- Cache hit, Gitaly logs only a `served bytes` message.
+- Cache miss, Gitaly logs both a `pack_objects_cache.generated_bytes` and `pack_objects_cache.served_bytes` message. Gitaly also logs some more detailed statistics of
+  pack-object generation.
+- Cache hit, Gitaly logs only a `pack_objects_cache.served_bytes` message.
 
 Example:
 
 ```json
 {
   "bytes":26186490,
-  "cache_key":"1b586a2698ca93c2529962e85cda5eea8f0f2b0036592615718898368b462e19",
-  "correlation_id":"01F1MY8JXC3FZN14JBG1H42G9F",
-  "grpc.meta.deadline_type":"none",
-  "grpc.method":"PackObjectsHook",
-  "grpc.request.fullMethod":"/gitaly.HookService/PackObjectsHook",
-  "grpc.request.glProjectPath":"root/gitlab-workhorse",
-  "grpc.request.glRepository":"project-2",
-  "grpc.request.repoPath":"@hashed/d4/73/d4735e3a265e16eee03f59718b9b5d03019c07d8b6c51f90da3a666eec13ab35.git",
-  "grpc.request.repoStorage":"default",
-  "grpc.request.topLevelGroup":"@hashed",
-  "grpc.service":"gitaly.HookService",
-  "grpc.start_time":"2021-03-25T14:57:52.747Z",
-  "level":"info",
-  "msg":"generated bytes",
-  "peer.address":"@",
-  "pid":20961,
-  "span.kind":"server",
-  "system":"grpc",
-  "time":"2021-03-25T14:57:53.543Z"
-}
-{
-  "bytes":26186490,
-  "cache_key":"1b586a2698ca93c2529962e85cda5eea8f0f2b0036592615718898368b462e19",
   "correlation_id":"01F1MY8JXC3FZN14JBG1H42G9F",
   "grpc.meta.deadline_type":"none",
   "grpc.method":"PackObjectsHook",
@@ -1341,12 +1367,23 @@ Example:
   "grpc.service":"gitaly.HookService",
   "grpc.start_time":"2021-03-25T14:57:52.747Z",
   "level":"info",
-  "msg":"served bytes",
+  "msg":"finished unary call with code OK",
   "peer.address":"@",
   "pid":20961,
   "span.kind":"server",
   "system":"grpc",
-  "time":"2021-03-25T14:57:53.543Z"
+  "time":"2021-03-25T14:57:53.543Z",
+  "pack_objects.compression_statistics": "Total 145991 (delta 68), reused 6 (delta 2), pack-reused 145911",
+  "pack_objects.enumerate_objects_ms": 170,
+  "pack_objects.prepare_pack_ms": 7,
+  "pack_objects.write_pack_file_ms": 786,
+  "pack_objects.written_object_count": 145991,
+  "pack_objects_cache.generated_bytes": 49533030,
+  "pack_objects_cache.hit": "false",
+  "pack_objects_cache.key": "123456789",
+  "pack_objects_cache.served_bytes": 49533030,
+  "peer.address": "127.0.0.1",
+  "pid": 8813,
 }
 ```
 
diff --git a/doc/administration/gitaly/monitoring.md b/doc/administration/gitaly/monitoring.md
index 00d0499faa2..cbf5722f2c5 100644
--- a/doc/administration/gitaly/monitoring.md
+++ b/doc/administration/gitaly/monitoring.md
@@ -31,18 +31,64 @@ of requests dropped due to request limiting. The `reason` label indicates why a
 
 ## Monitor Gitaly concurrency limiting
 
-You can observe specific behavior of [concurrency-queued requests](configure_gitaly.md#limit-rpc-concurrency) using
-the Gitaly logs and Prometheus:
-
-- In the [Gitaly logs](../logs/index.md#gitaly-logs), look for the string (or structured log field)
-  `acquire_ms`. Messages that have this field are reporting about the concurrency limiter.
-- In Prometheus, look for the following metrics:
-  - `gitaly_concurrency_limiting_in_progress` indicates how many concurrent requests are
-    being processed.
-  - `gitaly_concurrency_limiting_queued` indicates how many requests for an RPC for a given
-    repository are waiting due to the concurrency limit being reached.
-  - `gitaly_concurrency_limiting_acquiring_seconds` indicates how long a request has to
-    wait due to concurrency limits before being processed.
+You can observe specific behavior of [concurrency-queued requests](configure_gitaly.md#limit-rpc-concurrency) using Gitaly logs and Prometheus.
+
+In the [Gitaly logs](../logs/index.md#gitaly-logs), you can identify logs related to the pack-objects concurrency limiting with entries such as:
+
+| Log Field | Description |
+| --- | --- |
+| `limit.concurrency_queue_length` | Indicates the current length of the queue specific to the RPC type of the ongoing call. It provides insight into the number of requests waiting to be processed due to concurrency limits.                                       |
+| `limit.concurrency_queue_ms`     | Represents the duration, in milliseconds, that a request has spent waiting in the queue due to the limit on concurrent RPCs. This field helps understand the impact of concurrency limits on request processing times.           |
+| `limit.concurrency_dropped`      | If the request is dropped due to limits being reached, this field specifies the reason: either `max_time` (request waited in the queue longer than the maximum allowed time) or `max_size` (the queue reached its maximum size). |
+| `limit.limiting_key`             | Identifies the key used for limiting.  |
+| `limit.limiting_type`            | Specifies the type of process being limited. In this context, it's `per-rpc`, indicating that the concurrency limiting is applied on a per-RPC basis.                                                                            |
+
+For example:
+
+```json
+{
+  "limit .concurrency_queue_length": 1,
+  "limit .concurrency_queue_ms": 0,
+  "limit.limiting_key": "@hashed/79/02/7902699be42c8a8e46fbbb450172651786b22c56a189f7625a6da49081b2451.git",
+  "limit.limiting_type": "per-rpc"
+}
+```
+
+In Prometheus, look for the following metrics:
+
+- `gitaly_concurrency_limiting_in_progress` indicates how many concurrent requests are being processed.
+- `gitaly_concurrency_limiting_queued` indicates how many requests for an RPC for a given repository are waiting due to the concurrency limit being reached.
+- `gitaly_concurrency_limiting_acquiring_seconds` indicates how long a request has to wait due to concurrency limits before being processed.
+
+## Monitor Gitaly pack-objects concurrency limiting
+
+You can observe specific behavior of [pack-objects limiting](configure_gitaly.md#limit-pack-objects-concurrency) using Gitaly logs and Prometheus.
+
+In the [Gitaly logs](../logs/index.md#gitaly-logs), you can identify logs related to the pack-objects concurrency limiting with entries such as:
+
+| Log Field | Description |
+|:---|:---|
+| `limit.concurrency_queue_length` | Current length of the queue for the pack-objects processes. Indicates the number of requests that are waiting to be processed because the limit on concurrent processes has been reached. |
+| `limit.concurrency_queue_ms` | Time a request has spent waiting in the queue, in milliseconds. Indicates how long a request has had to wait because of the limits on concurrency. |
+| `limit.limiting_key` | Remote IP of the sender. |
+| `limit.limiting_type` | Type of process being limited. In this case, `pack-objects`. |
+
+Example configuration:
+
+```json
+{
+  "limit .concurrency_queue_length": 1,
+  "limit .concurrency_queue_ms": 0,
+  "limit.limiting_key": "1.2.3.4",
+  "limit.limiting_type": "pack-objects"
+}
+```
+
+In Prometheus, look for the following metrics:
+
+- `gitaly_pack_objects_in_progress` indicates how many pack-objects processes are being processed concurrently.
+- `gitaly_pack_objects_queued` indicates how many requests for pack-objects processes are waiting due to the concurrency limit being reached.
+- `gitaly_pack_objects_acquiring_seconds` indicates how long a request for a pack-object process has to wait due to concurrency limits before being processed.
 
 ## Monitor Gitaly cgroups