Add latest changes from gitlab-org/gitlab@15-3-stable-eev15.3.0-rc42

9 files changed, 1700 insertions, 0 deletions

diff --git a/doc/administration/logs/img/database-query-dialog_v14_3.png b/doc/administration/logs/img/database-query-dialog_v14_3.png
new file mode 100644
index 00000000000..197cfa17da2
--- /dev/null
+++ b/doc/administration/logs/img/database-query-dialog_v14_3.png
diff --git a/doc/administration/logs/img/network_monitor_xid.png b/doc/administration/logs/img/network_monitor_xid.png
new file mode 100644
index 00000000000..fa64fd1509c
--- /dev/null
+++ b/doc/administration/logs/img/network_monitor_xid.png
diff --git a/doc/administration/logs/img/obtaining-a-session-cookie-for-request_v14_3.png b/doc/administration/logs/img/obtaining-a-session-cookie-for-request_v14_3.png
new file mode 100644
index 00000000000..a63ebf9ecf2
--- /dev/null
+++ b/doc/administration/logs/img/obtaining-a-session-cookie-for-request_v14_3.png
diff --git a/doc/administration/logs/img/paste-request-id-into-progress-bar_v14_3.png b/doc/administration/logs/img/paste-request-id-into-progress-bar_v14_3.png
new file mode 100644
index 00000000000..a19585d7a89
--- /dev/null
+++ b/doc/administration/logs/img/paste-request-id-into-progress-bar_v14_3.png
diff --git a/doc/administration/logs/img/select-request-id-from-request-selector-drop-down-menu_v14_3.png b/doc/administration/logs/img/select-request-id-from-request-selector-drop-down-menu_v14_3.png
new file mode 100644
index 00000000000..b8314056c9b
--- /dev/null
+++ b/doc/administration/logs/img/select-request-id-from-request-selector-drop-down-menu_v14_3.png
diff --git a/doc/administration/logs/img/view-pg-details_v14_3.png b/doc/administration/logs/img/view-pg-details_v14_3.png
new file mode 100644
index 00000000000..1fe12462e4e
--- /dev/null
+++ b/doc/administration/logs/img/view-pg-details_v14_3.png
diff --git a/doc/administration/logs/index.md b/doc/administration/logs/index.md
new file mode 100644
index 00000000000..d3529de08db
--- /dev/null
+++ b/doc/administration/logs/index.md
@@ -0,0 +1,1182 @@
+---
+stage: Monitor
+group: Respond
+info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments
+---
+
+# Log system **(FREE SELF)**
+
+GitLab has an advanced log system where everything is logged, so you
+can analyze your instance using various system log files. In addition to
+system log files, GitLab Enterprise Edition provides [Audit Events](../audit_events.md).
+
+System log files are typically plain text in a standard log file format.
+This guide talks about how to read and use these system log files.
+
+Read more about the log system and using the logs:
+
+- [Customize logging on Omnibus GitLab installations](https://docs.gitlab.com/omnibus/settings/logs.html)
+including adjusting log retention, log forwarding,
+switching logs from JSON to plain text logging, and more.
+- [How to parse and analyze JSON logs](../logs/log_parsing.md).
+
+## Log Levels
+
+Each log message has an assigned log level that indicates its importance and verbosity.
+Each logger has an assigned minimum log level.
+A logger emits a log message only if its log level is equal to or above the minimum log level.
+
+The following log levels are supported:
+
+| Level | Name      |
+|:------|:----------|
+| 0     | `DEBUG`   |
+| 1     | `INFO`    |
+| 2     | `WARN`    |
+| 3     | `ERROR`   |
+| 4     | `FATAL`   |
+| 5     | `UNKNOWN` |
+
+GitLab loggers emit all log messages because they are set to `DEBUG` by default.
+
+### Override default log level
+
+You can override the minimum log level for GitLab loggers using the `GITLAB_LOG_LEVEL` environment variable.
+Valid values are either a value of `0` to `5`, or the name of the log level.
+
+Example:
+
+```shell
+GITLAB_LOG_LEVEL=info
+```
+
+For some services, other log levels are in place that are not affected by this setting.
+Some of these services have their own environment variables to override the log level. For example:
+
+| Service              | Log level | Environment variable |
+|:---------------------|:----------|:---------------------|
+| GitLab API           | `INFO`    |                      |
+| GitLab Cleanup       | `INFO`    | `DEBUG`              |
+| GitLab Doctor        | `INFO`    | `VERBOSE`            |
+| GitLab Export        | `INFO`    | `EXPORT_DEBUG`       |
+| GitLab Geo           | `INFO`    |                      |
+| GitLab Import        | `INFO`    | `IMPORT_DEBUG`       |
+| GitLab QA Runtime    | `INFO`    | `QA_LOG_LEVEL`       |
+| Google APIs          | `INFO`    |                      |
+| Rack Timeout         | `ERROR`   |                      |
+| Sidekiq (server)     | `INFO`    |                      |
+| Snowplow Tracker     | `FATAL`   |                      |
+| gRPC Client (Gitaly) | `WARN`    | `GRPC_LOG_LEVEL`     |
+
+## Log Rotation
+
+The logs for a given service may be managed and rotated by:
+
+- `logrotate`
+- `svlogd` (`runit`'s service logging daemon)
+- `logrotate` and `svlogd`
+- Or not at all
+
+The following table includes information about what's responsible for managing and rotating logs for
+the included services. Logs
+[managed by `svlogd`](https://docs.gitlab.com/omnibus/settings/logs.html#runit-logs)
+are written to a file called `current`. The `logrotate` service built into GitLab
+[manages all logs](https://docs.gitlab.com/omnibus/settings/logs.html#logrotate)
+except those captured by `runit`.
+
+| Log type                                        | Managed by logrotate    | Managed by svlogd/runit |
+|:------------------------------------------------|:------------------------|:------------------------|
+| [Alertmanager Logs](#alertmanager-logs)         | **{dotted-circle}** No  | **{check-circle}** Yes  |
+| [Crond Logs](#crond-logs)                       | **{dotted-circle}** No  | **{check-circle}** Yes  |
+| [Gitaly](#gitaly-logs)                          | **{check-circle}** Yes  | **{check-circle}** Yes  |
+| [GitLab Exporter for Omnibus](#gitlab-exporter) | **{dotted-circle}** No  | **{check-circle}** Yes  |
+| [GitLab Pages Logs](#pages-logs)                | **{check-circle}** Yes  | **{check-circle}** Yes  |
+| GitLab Rails                                    | **{check-circle}** Yes  | **{dotted-circle}** No  |
+| [GitLab Shell Logs](#gitlab-shelllog)           | **{check-circle}** Yes  | **{dotted-circle}** No  |
+| [Grafana Logs](#grafana-logs)                   | **{dotted-circle}** No  | **{check-circle}** Yes  |
+| [LogRotate Logs](#logrotate-logs)               | **{dotted-circle}** No  | **{check-circle}** Yes  |
+| [Mailroom](#mail_room_jsonlog-default)          | **{check-circle}** Yes  | **{check-circle}** Yes  |
+| [NGINX](#nginx-logs)                            | **{check-circle}** Yes  | **{check-circle}** Yes  |
+| [PostgreSQL Logs](#postgresql-logs)             | **{dotted-circle}** No  | **{check-circle}** Yes  |
+| [Praefect Logs](#praefect-logs)                 | **{dotted-circle}** Yes | **{check-circle}** Yes  |
+| [Prometheus Logs](#prometheus-logs)             | **{dotted-circle}** No  | **{check-circle}** Yes  |
+| [Puma](#puma-logs)                              | **{check-circle}** Yes  | **{check-circle}** Yes  |
+| [Redis Logs](#redis-logs)                       | **{dotted-circle}** No  | **{check-circle}** Yes  |
+| [Registry Logs](#registry-logs)                 | **{dotted-circle}** No  | **{check-circle}** Yes  |
+| [Workhorse Logs](#workhorse-logs)               | **{check-circle}** Yes  | **{check-circle}** Yes  |
+
+## `production_json.log`
+
+Depending on your installation method, this file is located at:
+
+- Omnibus GitLab: `/var/log/gitlab/gitlab-rails/production_json.log`
+- Installations from source: `/home/git/gitlab/log/production_json.log`
+
+It contains a structured log for Rails controller requests received from
+GitLab, thanks to [Lograge](https://github.com/roidrage/lograge/).
+Requests from the API are logged to a separate file in `api_json.log`.
+
+Each line contains JSON that can be ingested by services like Elasticsearch and Splunk.
+Line breaks were added to examples for legibility:
+
+```json
+{
+  "method":"GET",
+  "path":"/gitlab/gitlab-foss/issues/1234",
+  "format":"html",
+  "controller":"Projects::IssuesController",
+  "action":"show",
+  "status":200,
+  "time":"2017-08-08T20:15:54.821Z",
+  "params":[{"key":"param_key","value":"param_value"}],
+  "remote_ip":"18.245.0.1",
+  "user_id":1,
+  "username":"admin",
+  "queue_duration_s":0.0,
+  "gitaly_calls":16,
+  "gitaly_duration_s":0.16,
+  "redis_calls":115,
+  "redis_duration_s":0.13,
+  "redis_read_bytes":1507378,
+  "redis_write_bytes":2920,
+  "correlation_id":"O1SdybnnIq7",
+  "cpu_s":17.50,
+  "db_duration_s":0.08,
+  "view_duration_s":2.39,
+  "duration_s":20.54,
+  "pid": 81836,
+  "worker_id":"puma_0"
+}
+```
+
+This example was a GET request for a specific
+issue. Each line also contains performance data, with times in
+seconds:
+
+- `duration_s`: Total time to retrieve the request
+- `queue_duration_s`: Total time the request was queued inside GitLab Workhorse
+- `view_duration_s`: Total time inside the Rails views
+- `db_duration_s`: Total time to retrieve data from PostgreSQL
+- `cpu_s`: Total time spent on CPU
+- `gitaly_duration_s`: Total time by Gitaly calls
+- `gitaly_calls`: Total number of calls made to Gitaly
+- `redis_calls`: Total number of calls made to Redis
+- `redis_duration_s`: Total time to retrieve data from Redis
+- `redis_read_bytes`: Total bytes read from Redis
+- `redis_write_bytes`: Total bytes written to Redis
+- `redis_<instance>_calls`: Total number of calls made to a Redis instance
+- `redis_<instance>_duration_s`: Total time to retrieve data from a Redis instance
+- `redis_<instance>_read_bytes`: Total bytes read from a Redis instance
+- `redis_<instance>_write_bytes`: Total bytes written to a Redis instance
+- `pid`: The worker's Linux process ID (changes when workers restart)
+- `worker_id`: The worker's logical ID (does not change when workers restart)
+
+User clone and fetch activity using HTTP transport appears in the log as `action: git_upload_pack`.
+
+In addition, the log contains the originating IP address,
+(`remote_ip`), the user's ID (`user_id`), and username (`username`).
+
+Some endpoints (such as `/search`) may make requests to Elasticsearch if using
+[Advanced Search](../../user/search/advanced_search.md). These
+additionally log `elasticsearch_calls` and `elasticsearch_call_duration_s`,
+which correspond to:
+
+- `elasticsearch_calls`: Total number of calls to Elasticsearch
+- `elasticsearch_duration_s`: Total time taken by Elasticsearch calls
+- `elasticsearch_timed_out_count`: Total number of calls to Elasticsearch that
+   timed out and therefore returned partial results
+
+ActionCable connection and subscription events are also logged to this file and they follow the
+previous format. The `method`, `path`, and `format` fields are not applicable, and are always empty.
+The ActionCable connection or channel class is used as the `controller`.
+
+```json
+{
+  "method":null,
+  "path":null,
+  "format":null,
+  "controller":"IssuesChannel",
+  "action":"subscribe",
+  "status":200,
+  "time":"2020-05-14T19:46:22.008Z",
+  "params":[{"key":"project_path","value":"gitlab/gitlab-foss"},{"key":"iid","value":"1"}],
+  "remote_ip":"127.0.0.1",
+  "user_id":1,
+  "username":"admin",
+  "ua":"Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:76.0) Gecko/20100101 Firefox/76.0",
+  "correlation_id":"jSOIEynHCUa",
+  "duration_s":0.32566
+}
+```
+
+NOTE:
+Starting with GitLab 12.5, if an error occurs, an
+`exception` field is included with `class`, `message`, and
+`backtrace`. Previous versions included an `error` field instead of
+`exception.class` and `exception.message`. For example:
+
+```json
+{
+  "method": "GET",
+  "path": "/admin",
+  "format": "html",
+  "controller": "Admin::DashboardController",
+  "action": "index",
+  "status": 500,
+  "time": "2019-11-14T13:12:46.156Z",
+  "params": [],
+  "remote_ip": "127.0.0.1",
+  "user_id": 1,
+  "username": "root",
+  "ua": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.14; rv:70.0) Gecko/20100101 Firefox/70.0",
+  "queue_duration": 274.35,
+  "correlation_id": "KjDVUhNvvV3",
+  "queue_duration_s":0.0,
+  "gitaly_calls":16,
+  "gitaly_duration_s":0.16,
+  "redis_calls":115,
+  "redis_duration_s":0.13,
+  "correlation_id":"O1SdybnnIq7",
+  "cpu_s":17.50,
+  "db_duration_s":0.08,
+  "view_duration_s":2.39,
+  "duration_s":20.54,
+  "pid": 81836,
+  "worker_id": "puma_0",
+  "exception.class": "NameError",
+  "exception.message": "undefined local variable or method `adsf' for #<Admin::DashboardController:0x00007ff3c9648588>",
+  "exception.backtrace": [
+    "app/controllers/admin/dashboard_controller.rb:11:in `index'",
+    "ee/app/controllers/ee/admin/dashboard_controller.rb:14:in `index'",
+    "ee/lib/gitlab/ip_address_state.rb:10:in `with'",
+    "ee/app/controllers/ee/application_controller.rb:43:in `set_current_ip_address'",
+    "lib/gitlab/session.rb:11:in `with_session'",
+    "app/controllers/application_controller.rb:450:in `set_session_storage'",
+    "app/controllers/application_controller.rb:444:in `set_locale'",
+    "ee/lib/gitlab/jira/middleware.rb:19:in `call'"
+  ]
+}
+```
+
+## `production.log`
+
+Depending on your installation method, this file is located at:
+
+- Omnibus GitLab: `/var/log/gitlab/gitlab-rails/production.log`
+- Installations from source: `/home/git/gitlab/log/production.log`
+
+It contains information about all performed requests. You can see the
+URL and type of request, IP address, and what parts of code were
+involved to service this particular request. Also, you can see all SQL
+requests performed, and how much time each took. This task is
+more useful for GitLab contributors and developers. Use part of this log
+file when you're reporting bugs. For example:
+
+```plaintext
+Started GET "/gitlabhq/yaml_db/tree/master" for 168.111.56.1 at 2015-02-12 19:34:53 +0200
+Processing by Projects::TreeController#show as HTML
+  Parameters: {"project_id"=>"gitlabhq/yaml_db", "id"=>"master"}
+
+  ... [CUT OUT]
+
+  Namespaces"."created_at" DESC, "namespaces"."id" DESC LIMIT 1 [["id", 26]]
+  CACHE (0.0ms) SELECT  "members".* FROM "members"  WHERE "members"."source_type" = 'Project' AND "members"."type" IN ('ProjectMember') AND "members"."source_id" = $1 AND "members"."source_type" = $2 AND "members"."user_id" = 1  ORDER BY "members"."created_at" DESC, "members"."id" DESC LIMIT 1  [["source_id", 18], ["source_type", "Project"]]
+  CACHE (0.0ms) SELECT  "members".* FROM "members"  WHERE "members"."source_type" = 'Project' AND "members".
+  (1.4ms) SELECT COUNT(*) FROM "merge_requests"  WHERE "merge_requests"."target_project_id" = $1 AND ("merge_requests"."state" IN ('opened','reopened')) [["target_project_id", 18]]
+  Rendered layouts/nav/_project.html.haml (28.0ms)
+  Rendered layouts/_collapse_button.html.haml (0.2ms)
+  Rendered layouts/_flash.html.haml (0.1ms)
+  Rendered layouts/_page.html.haml (32.9ms)
+Completed 200 OK in 166ms (Views: 117.4ms | ActiveRecord: 27.2ms)
+```
+
+In this example, the server processed an HTTP request with URL
+`/gitlabhq/yaml_db/tree/master` from IP `168.111.56.1` at `2015-02-12 19:34:53 +0200`.
+The request was processed by `Projects::TreeController`.
+
+## `api_json.log`
+
+Depending on your installation method, this file is located at:
+
+- Omnibus GitLab: `/var/log/gitlab/gitlab-rails/api_json.log`
+- Installations from source: `/home/git/gitlab/log/api_json.log`
+
+It helps you see requests made directly to the API. For example:
+
+```json
+{
+  "time":"2018-10-29T12:49:42.123Z",
+  "severity":"INFO",
+  "duration":709.08,
+  "db":14.59,
+  "view":694.49,
+  "status":200,
+  "method":"GET",
+  "path":"/api/v4/projects",
+  "params":[{"key":"action","value":"git-upload-pack"},{"key":"changes","value":"_any"},{"key":"key_id","value":"secret"},{"key":"secret_token","value":"[FILTERED]"}],
+  "host":"localhost",
+  "remote_ip":"::1",
+  "ua":"Ruby",
+  "route":"/api/:version/projects",
+  "user_id":1,
+  "username":"root",
+  "queue_duration":100.31,
+  "gitaly_calls":30,
+  "gitaly_duration":5.36,
+  "pid": 81836,
+  "worker_id": "puma_0",
+  ...
+}
+```
+
+This entry shows an internal endpoint accessed to check whether an
+associated SSH key can download the project in question by using a `git fetch` or
+`git clone`. In this example, we see:
+
+- `duration`: Total time in milliseconds to retrieve the request
+- `queue_duration`: Total time in milliseconds the request was queued inside GitLab Workhorse
+- `method`: The HTTP method used to make the request
+- `path`: The relative path of the query
+- `params`: Key-value pairs passed in a query string or HTTP body (sensitive parameters, such as passwords and tokens, are filtered out)
+- `ua`: The User-Agent of the requester
+
+## `application.log`
+
+Depending on your installation method, this file is located at:
+
+- Omnibus GitLab: `/var/log/gitlab/gitlab-rails/application.log`
+- Installations from source: `/home/git/gitlab/log/application.log`
+
+It helps you discover events happening in your instance such as user creation
+and project deletion. For example:
+
+```plaintext
+October 06, 2014 11:56: User "Administrator" (admin@example.com) was created
+October 06, 2014 11:56: Documentcloud created a new project "Documentcloud / Underscore"
+October 06, 2014 11:56: Gitlab Org created a new project "Gitlab Org / Gitlab Ce"
+October 07, 2014 11:25: User "Claudie Hodkiewicz" (nasir_stehr@olson.co.uk)  was removed
+October 07, 2014 11:25: Project "project133" was removed
+```
+
+## `application_json.log`
+
+> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/22812) in GitLab 12.7.
+
+Depending on your installation method, this file is located at:
+
+- Omnibus GitLab: `/var/log/gitlab/gitlab-rails/application_json.log`
+- Installations from source: `/home/git/gitlab/log/application_json.log`
+
+It contains the JSON version of the logs in `application.log`, like this example:
+
+```json
+{
+  "severity":"INFO",
+  "time":"2020-01-14T13:35:15.466Z",
+  "correlation_id":"3823a1550b64417f9c9ed8ee0f48087e",
+  "message":"User \"Administrator\" (admin@example.com) was created"
+}
+{
+  "severity":"INFO",
+  "time":"2020-01-14T13:35:15.466Z",
+  "correlation_id":"78e3df10c9a18745243d524540bd5be4",
+  "message":"Project \"project133\" was removed"
+}
+```
+
+## `integrations_json.log`
+
+Depending on your installation method, this file is located at:
+
+- Omnibus GitLab: `/var/log/gitlab/gitlab-rails/integrations_json.log`
+- Installations from source: `/home/git/gitlab/log/integrations_json.log`
+
+It contains information about [integration](../../user/project/integrations/index.md)
+activities, such as Jira, Asana, and irker services. It uses JSON format,
+like this example:
+
+```json
+{
+  "severity":"ERROR",
+  "time":"2018-09-06T14:56:20.439Z",
+  "service_class":"Integrations::Jira",
+  "project_id":8,
+  "project_path":"h5bp/html5-boilerplate",
+  "message":"Error sending message",
+  "client_url":"http://jira.gitlap.com:8080",
+  "error":"execution expired"
+}
+{
+  "severity":"INFO",
+  "time":"2018-09-06T17:15:16.365Z",
+  "service_class":"Integrations::Jira",
+  "project_id":3,
+  "project_path":"namespace2/project2",
+  "message":"Successfully posted",
+  "client_url":"http://jira.example.com"
+}
+```
+
+## `kubernetes.log`
+
+Depending on your installation method, this file is located at:
+
+- Omnibus GitLab: `/var/log/gitlab/gitlab-rails/kubernetes.log`
+- Installations from source: `/home/git/gitlab/log/kubernetes.log`
+
+It logs information related to the Kubernetes Integration, including errors
+during installing cluster applications on your managed Kubernetes
+clusters.
+
+Each line contains JSON that can be ingested by services like Elasticsearch and Splunk.
+Line breaks have been added to the following example for clarity:
+
+```json
+{
+  "severity":"ERROR",
+  "time":"2018-11-23T15:14:54.652Z",
+  "exception":"Kubeclient::HttpError",
+  "error_code":401,
+  "service":"Clusters::Applications::CheckInstallationProgressService",
+  "app_id":14,
+  "project_ids":[1],
+  "group_ids":[],
+  "message":"Unauthorized"
+}
+{
+  "severity":"ERROR",
+  "time":"2018-11-23T15:42:11.647Z",
+  "exception":"Kubeclient::HttpError",
+  "error_code":null,
+  "service":"Clusters::Applications::InstallService",
+  "app_id":2,
+  "project_ids":[19],
+  "group_ids":[],
+  "message":"SSL_connect returned=1 errno=0 state=error: certificate verify failed (unable to get local issuer certificate)"
+}
+```
+
+## `git_json.log`
+
+Depending on your installation method, this file is located at:
+
+- Omnibus GitLab: `/var/log/gitlab/gitlab-rails/git_json.log`
+- Installations from source: `/home/git/gitlab/log/git_json.log`
+
+After GitLab version 12.2, this file was renamed from `githost.log` to
+`git_json.log` and stored in JSON format.
+
+GitLab has to interact with Git repositories, but in some rare cases
+something can go wrong. If this happens, you need to know exactly what
+happened. This log file contains all failed requests from GitLab to Git
+repositories. In the majority of cases this file is useful for developers
+only. For example:
+
+```json
+{
+   "severity":"ERROR",
+   "time":"2019-07-19T22:16:12.528Z",
+   "correlation_id":"FeGxww5Hj64",
+   "message":"Command failed [1]: /usr/bin/git --git-dir=/Users/vsizov/gitlab-development-kit/gitlab/tmp/tests/gitlab-satellites/group184/gitlabhq/.git --work-tree=/Users/vsizov/gitlab-development-kit/gitlab/tmp/tests/gitlab-satellites/group184/gitlabhq merge --no-ff -mMerge branch 'feature_conflict' into 'feature' source/feature_conflict\n\nerror: failed to push some refs to '/Users/vsizov/gitlab-development-kit/repositories/gitlabhq/gitlab_git.git'"
+}
+```
+
+## `audit_json.log` **(FREE)**
+
+NOTE:
+GitLab Free tracks a small number of different audit events.
+GitLab Premium tracks many more.
+
+Depending on your installation method, this file is located at:
+
+- Omnibus GitLab: `/var/log/gitlab/gitlab-rails/audit_json.log`
+- Installations from source: `/home/git/gitlab/log/audit_json.log`
+
+Changes to group or project settings and memberships (`target_details`)
+are logged to this file. For example:
+
+```json
+{
+  "severity":"INFO",
+  "time":"2018-10-17T17:38:22.523Z",
+  "author_id":3,
+  "entity_id":2,
+  "entity_type":"Project",
+  "change":"visibility",
+  "from":"Private",
+  "to":"Public",
+  "author_name":"John Doe4",
+  "target_id":2,
+  "target_type":"Project",
+  "target_details":"namespace2/project2"
+}
+```
+
+## Sidekiq Logs
+
+NOTE:
+In Omnibus GitLab `12.10` or earlier, the Sidekiq log is at `/var/log/gitlab/gitlab-rails/sidekiq.log`.
+
+For Omnibus GitLab installations, some Sidekiq logs are in `/var/log/gitlab/sidekiq/current`
+and as follows.
+
+### `sidekiq.log`
+
+Depending on your installation method, this file is located at:
+
+- Omnibus GitLab: `/var/log/gitlab/sidekiq/current`
+- Installations from source: `/home/git/gitlab/log/sidekiq.log`
+
+GitLab uses background jobs for processing tasks which can take a long
+time. All information about processing these jobs are written down to
+this file. For example:
+
+```plaintext
+2014-06-10T07:55:20Z 2037 TID-tm504 ERROR: /opt/bitnami/apps/discourse/htdocs/vendor/bundle/ruby/1.9.1/gems/redis-3.0.7/lib/redis/client.rb:228:in `read'
+2014-06-10T18:18:26Z 14299 TID-55uqo INFO: Booting Sidekiq 3.0.0 with redis options {:url=>"redis://localhost:6379/0", :namespace=>"sidekiq"}
+```
+
+Instead of the previous format, you can opt to generate JSON logs for
+Sidekiq. For example:
+
+```json
+{
+  "severity":"INFO",
+  "time":"2018-04-03T22:57:22.071Z",
+  "queue":"cronjob:update_all_mirrors",
+  "args":[],
+  "class":"UpdateAllMirrorsWorker",
+  "retry":false,
+  "queue_namespace":"cronjob",
+  "jid":"06aeaa3b0aadacf9981f368e",
+  "created_at":"2018-04-03T22:57:21.930Z",
+  "enqueued_at":"2018-04-03T22:57:21.931Z",
+  "pid":10077,
+  "worker_id":"sidekiq_0",
+  "message":"UpdateAllMirrorsWorker JID-06aeaa3b0aadacf9981f368e: done: 0.139 sec",
+  "job_status":"done",
+  "duration":0.139,
+  "completed_at":"2018-04-03T22:57:22.071Z",
+  "db_duration":0.05,
+  "db_duration_s":0.0005,
+  "gitaly_duration":0,
+  "gitaly_calls":0
+}
+```
+
+For Omnibus GitLab installations, add the configuration option:
+
+```ruby
+sidekiq['log_format'] = 'json'
+```
+
+For installations from source, edit the `gitlab.yml` and set the Sidekiq
+`log_format` configuration option:
+
+```yaml
+  ## Sidekiq
+  sidekiq:
+    log_format: json
+```
+
+### `sidekiq_client.log`
+
+> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/26586) in GitLab 12.9.
+
+Depending on your installation method, this file is located at:
+
+- Omnibus GitLab: `/var/log/gitlab/gitlab-rails/sidekiq_client.log`
+- Installations from source: `/home/git/gitlab/log/sidekiq_client.log`
+
+This file contains logging information about jobs before Sidekiq starts
+processing them, such as before being enqueued.
+
+This log file follows the same structure as
+[`sidekiq.log`](#sidekiqlog), so it is structured as JSON if
+you've configured this for Sidekiq as mentioned above.
+
+## `gitlab-shell.log`
+
+GitLab Shell is used by GitLab for executing Git commands and provide SSH
+access to Git repositories.
+
+### For GitLab versions 12.10 and up
+
+Information containing `git-{upload-pack,receive-pack}` requests is at
+`/var/log/gitlab/gitlab-shell/gitlab-shell.log`. Information about hooks to
+GitLab Shell from Gitaly is at `/var/log/gitlab/gitaly/current`.
+
+Example log entries for `/var/log/gitlab/gitlab-shell/gitlab-shell.log`:
+
+```json
+{
+  "duration_ms": 74.104,
+  "level": "info",
+  "method": "POST",
+  "msg": "Finished HTTP request",
+  "time": "2020-04-17T20:28:46Z",
+  "url": "http://127.0.0.1:8080/api/v4/internal/allowed"
+}
+{
+  "command": "git-upload-pack",
+  "git_protocol": "",
+  "gl_project_path": "root/example",
+  "gl_repository": "project-1",
+  "level": "info",
+  "msg": "executing git command",
+  "time": "2020-04-17T20:28:46Z",
+  "user_id": "user-1",
+  "username": "root"
+}
+```
+
+Example log entries for `/var/log/gitlab/gitaly/current`:
+
+```json
+{
+  "method": "POST",
+  "url": "http://127.0.0.1:8080/api/v4/internal/allowed",
+  "duration": 0.058012959,
+  "gitaly_embedded": true,
+  "pid": 16636,
+  "level": "info",
+  "msg": "finished HTTP request",
+  "time": "2020-04-17T20:29:08+00:00"
+}
+{
+  "method": "POST",
+  "url": "http://127.0.0.1:8080/api/v4/internal/pre_receive",
+  "duration": 0.031022552,
+  "gitaly_embedded": true,
+  "pid": 16636,
+  "level": "info",
+  "msg": "finished HTTP request",
+  "time": "2020-04-17T20:29:08+00:00"
+}
+```
+
+### For GitLab versions 12.5 through 12.9
+
+For GitLab 12.5 to 12.9, depending on your installation method, this
+file is located at:
+
+- Omnibus GitLab: `/var/log/gitlab/gitaly/gitlab-shell.log`
+- Installation from source: `/home/git/gitaly/gitlab-shell.log`
+
+Example log entries:
+
+```json
+{
+  "method": "POST",
+  "url": "http://127.0.0.1:8080/api/v4/internal/post_receive",
+  "duration": 0.031809164,
+  "gitaly_embedded": true,
+  "pid": 27056,
+  "level": "info",
+  "msg": "finished HTTP request",
+  "time": "2020-04-17T16:24:38+00:00"
+}
+```
+
+### For GitLab 12.5 and earlier
+
+For GitLab 12.5 and earlier, the file is at `/var/log/gitlab/gitlab-shell/gitlab-shell.log`.
+
+Example log entries:
+
+```plaintext
+I, [2015-02-13T06:17:00.671315 #9291]  INFO -- : Adding project root/example.git at </var/opt/gitlab/git-data/repositories/root/dcdcdcdcd.git>.
+I, [2015-02-13T06:17:00.679433 #9291]  INFO -- : Moving existing hooks directory and symlinking global hooks directory for /var/opt/gitlab/git-data/repositories/root/example.git.
+```
+
+User clone/fetch activity using SSH transport appears in this log as
+`executing git command <gitaly-upload-pack...`.
+
+## Gitaly Logs
+
+This file is in `/var/log/gitlab/gitaly/current` and is produced by [runit](http://smarden.org/runit/).
+`runit` is packaged with Omnibus GitLab and a brief explanation of its purpose
+is available [in the Omnibus GitLab documentation](https://docs.gitlab.com/omnibus/architecture/#runit).
+[Log files are rotated](http://smarden.org/runit/svlogd.8.html), renamed in
+Unix timestamp format, and `gzip`-compressed (like `@1584057562.s`).
+
+### `grpc.log`
+
+This file is at `/var/log/gitlab/gitlab-rails/grpc.log` for Omnibus GitLab
+packages. Native [gRPC](https://grpc.io/) logging used by Gitaly.
+
+### `gitaly_ruby_json.log`
+
+> [Introduced](https://gitlab.com/gitlab-org/gitaly/-/merge_requests/2678) in GitLab 13.6.
+
+This file is at `/var/log/gitlab/gitaly/gitaly_ruby_json.log` and is
+produced by [`gitaly-ruby`](../gitaly/reference.md#gitaly-ruby). It contains an
+access log of gRPC calls made by Gitaly to `gitaly-ruby`.
+
+### `gitaly_hooks.log`
+
+This file is at `/var/log/gitlab/gitaly/gitaly_hooks.log` and is
+produced by `gitaly-hooks` command. It also contains records about
+failures received during processing of the responses from GitLab API.
+
+## Puma Logs
+
+### `puma_stdout.log`
+
+Depending on your installation method, this file is located at:
+
+- Omnibus GitLab: `/var/log/gitlab/puma/puma_stdout.log`
+- Installations from source: `/home/git/gitlab/log/puma_stdout.log`
+
+### `puma_stderr.log`
+
+Depending on your installation method, this file is located at:
+
+- Omnibus GitLab: `/var/log/gitlab/puma/puma_stderr.log`
+- Installations from source: `/home/git/gitlab/log/puma_stderr.log`
+
+## `repocheck.log`
+
+Depending on your installation method, this file is located at:
+
+- Omnibus GitLab: `/var/log/gitlab/gitlab-rails/repocheck.log`
+- Installations from source: `/home/git/gitlab/log/repocheck.log`
+
+It logs information whenever a [repository check is run](../repository_checks.md)
+on a project.
+
+## `importer.log`
+
+Depending on your installation method, this file is located at:
+
+- Omnibus GitLab: `/var/log/gitlab/gitlab-rails/importer.log`
+- Installations from source: `/home/git/gitlab/log/importer.log`
+
+It logs the progress of the import process.
+
+## `exporter.log`
+
+> Introduced in GitLab 13.1.
+
+Depending on your installation method, this file is located at:
+
+- Omnibus GitLab: `/var/log/gitlab/gitlab-rails/exporter.log`
+- Installations from source: `/home/git/gitlab/log/exporter.log`
+
+It logs the progress of the export process.
+
+## `features_json.log`
+
+> [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/59587) in GitLab 13.7.
+
+Depending on your installation method, this file is located at:
+
+- Omnibus GitLab: `/var/log/gitlab/gitlab-rails/features_json.log`
+- Installations from source: `/home/git/gitlab/log/features_json.log`
+
+The modification events from [Feature flags in development of GitLab](../../development/feature_flags/index.md)
+are recorded in this file. For example:
+
+```json
+{"severity":"INFO","time":"2020-11-24T02:30:59.860Z","correlation_id":null,"key":"cd_auto_rollback","action":"enable","extra.thing":"true"}
+{"severity":"INFO","time":"2020-11-24T02:31:29.108Z","correlation_id":null,"key":"cd_auto_rollback","action":"enable","extra.thing":"true"}
+{"severity":"INFO","time":"2020-11-24T02:31:29.129Z","correlation_id":null,"key":"cd_auto_rollback","action":"disable","extra.thing":"false"}
+{"severity":"INFO","time":"2020-11-24T02:31:29.177Z","correlation_id":null,"key":"cd_auto_rollback","action":"enable","extra.thing":"Project:1"}
+{"severity":"INFO","time":"2020-11-24T02:31:29.183Z","correlation_id":null,"key":"cd_auto_rollback","action":"disable","extra.thing":"Project:1"}
+{"severity":"INFO","time":"2020-11-24T02:31:29.188Z","correlation_id":null,"key":"cd_auto_rollback","action":"enable_percentage_of_time","extra.percentage":"50"}
+{"severity":"INFO","time":"2020-11-24T02:31:29.193Z","correlation_id":null,"key":"cd_auto_rollback","action":"disable_percentage_of_time"}
+{"severity":"INFO","time":"2020-11-24T02:31:29.198Z","correlation_id":null,"key":"cd_auto_rollback","action":"enable_percentage_of_actors","extra.percentage":"50"}
+{"severity":"INFO","time":"2020-11-24T02:31:29.203Z","correlation_id":null,"key":"cd_auto_rollback","action":"disable_percentage_of_actors"}
+{"severity":"INFO","time":"2020-11-24T02:31:29.329Z","correlation_id":null,"key":"cd_auto_rollback","action":"remove"}
+```
+
+## `auth.log`
+
+> Introduced in GitLab 12.0.
+
+Depending on your installation method, this file is located at:
+
+- Omnibus GitLab: `/var/log/gitlab/gitlab-rails/auth.log`
+- Installations from source: `/home/git/gitlab/log/auth.log`
+
+This log records:
+
+- Requests over the [Rate Limit](../../user/admin_area/settings/rate_limits_on_raw_endpoints.md) on raw endpoints.
+- [Protected paths](../../user/admin_area/settings/protected_paths.md) abusive requests.
+- In GitLab versions [12.3](https://gitlab.com/gitlab-org/gitlab/-/issues/29239) and later,
+  user ID and username, if available.
+
+## `graphql_json.log`
+
+> [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/59587) in GitLab 12.0.
+
+Depending on your installation method, this file is located at:
+
+- Omnibus GitLab: `/var/log/gitlab/gitlab-rails/graphql_json.log`
+- Installations from source: `/home/git/gitlab/log/graphql_json.log`
+
+GraphQL queries are recorded in the file. For example:
+
+```json
+{"query_string":"query IntrospectionQuery{__schema {queryType { name },mutationType { name }}}...(etc)","variables":{"a":1,"b":2},"complexity":181,"depth":1,"duration_s":7}
+```
+
+## `migrations.log`
+
+> Introduced in GitLab 12.3.
+
+Depending on your installation method, this file is located at:
+
+- Omnibus GitLab: `/var/log/gitlab/gitlab-rails/migrations.log`
+- Installations from source: `/home/git/gitlab/log/migrations.log`
+
+## `mail_room_json.log` (default)
+
+> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/19186) in GitLab 12.6.
+
+Depending on your installation method, this file is located at:
+
+- Omnibus GitLab: `/var/log/gitlab/mailroom/current`
+- Installations from source: `/home/git/gitlab/log/mail_room_json.log`
+
+This structured log file records internal activity in the `mail_room` gem.
+Its name and path are configurable, so the name and path may not match the above.
+
+## Reconfigure logs
+
+Reconfigure log files are in `/var/log/gitlab/reconfigure` for Omnibus GitLab
+packages. Installations from source don't have reconfigure logs. A reconfigure log
+is populated whenever `gitlab-ctl reconfigure` is run manually or as part of an upgrade.
+
+Reconfigure logs files are named according to the UNIX timestamp of when the reconfigure
+was initiated, such as `1509705644.log`
+
+## `sidekiq_exporter.log` and `web_exporter.log`
+
+If Prometheus metrics and the Sidekiq Exporter are both enabled, Sidekiq
+starts a Web server and listen to the defined port (default:
+`8082`). By default, Sidekiq Exporter access logs are disabled but can
+be enabled based on your installation method:
+
+- Omnibus GitLab: Use the `sidekiq['exporter_log_enabled'] = true`
+  option in `/etc/gitlab/gitlab.rb`
+- Installations from source: Use the `sidekiq_exporter.log_enabled` option
+  in `gitlab.yml`
+
+When enabled, depending on your installation method, this file is located at:
+
+- Omnibus GitLab: `/var/log/gitlab/gitlab-rails/sidekiq_exporter.log`
+- Installations from source: `/home/git/gitlab/log/sidekiq_exporter.log`
+
+If Prometheus metrics and the Web Exporter are both enabled, Puma
+starts a Web server and listen to the defined port (default: `8083`), and access logs
+are generated in a location based on your installation method:
+
+- Omnibus GitLab: `/var/log/gitlab/gitlab-rails/web_exporter.log`
+- Installations from source: `/home/git/gitlab/log/web_exporter.log`
+
+## `database_load_balancing.log` **(PREMIUM SELF)**
+
+> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/15442) in GitLab 12.3.
+
+Contains details of GitLab [Database Load Balancing](../postgresql/database_load_balancing.md).
+Depending on your installation method, this file is located at:
+
+- Omnibus GitLab: `/var/log/gitlab/gitlab-rails/database_load_balancing.log`
+- Installations from source: `/home/git/gitlab/log/database_load_balancing.log`
+
+## `elasticsearch.log` **(PREMIUM SELF)**
+
+> Introduced in GitLab 12.6.
+
+This file logs information related to the Elasticsearch Integration, including
+errors during indexing or searching Elasticsearch. Depending on your installation
+method, this file is located at:
+
+- Omnibus GitLab: `/var/log/gitlab/gitlab-rails/elasticsearch.log`
+- Installations from source: `/home/git/gitlab/log/elasticsearch.log`
+
+Each line contains JSON that can be ingested by services like Elasticsearch and Splunk.
+Line breaks have been added to the following example line for clarity:
+
+```json
+{
+  "severity":"DEBUG",
+  "time":"2019-10-17T06:23:13.227Z",
+  "correlation_id":null,
+  "message":"redacted_search_result",
+  "class_name":"Milestone",
+  "id":2,
+  "ability":"read_milestone",
+  "current_user_id":2,
+  "query":"project"
+}
+```
+
+## `exceptions_json.log`
+
+> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/17819) in GitLab 12.6.
+
+This file logs the information about exceptions being tracked by
+`Gitlab::ErrorTracking`, which provides a standard and consistent way of
+[processing rescued exceptions](https://gitlab.com/gitlab-org/gitlab/-/blob/master/doc/development/logging.md#exception-handling).
+Depending on your installation method, this file is located at:
+
+- Omnibus GitLab: `/var/log/gitlab/gitlab-rails/exceptions_json.log`
+- Installations from source: `/home/git/gitlab/log/exceptions_json.log`
+
+Each line contains JSON that can be ingested by Elasticsearch. For example:
+
+```json
+{
+  "severity": "ERROR",
+  "time": "2019-12-17T11:49:29.485Z",
+  "correlation_id": "AbDVUrrTvM1",
+  "extra.project_id": 55,
+  "extra.relation_key": "milestones",
+  "extra.relation_index": 1,
+  "exception.class": "NoMethodError",
+  "exception.message": "undefined method `strong_memoize' for #<Gitlab::ImportExport::RelationFactory:0x00007fb5d917c4b0>",
+  "exception.backtrace": [
+    "lib/gitlab/import_export/relation_factory.rb:329:in `unique_relation?'",
+    "lib/gitlab/import_export/relation_factory.rb:345:in `find_or_create_object!'"
+  ]
+}
+```
+
+## `service_measurement.log`
+
+> Introduced in GitLab 13.0.
+
+Depending on your installation method, this file is located at:
+
+- Omnibus GitLab: `/var/log/gitlab/gitlab-rails/service_measurement.log`
+- Installations from source: `/home/git/gitlab/log/service_measurement.log`
+
+It contains only a single structured log with measurements for each service execution.
+It contains measurements such as the number of SQL calls, `execution_time`, `gc_stats`, and `memory usage`.
+
+For example:
+
+```json
+{ "severity":"INFO", "time":"2020-04-22T16:04:50.691Z","correlation_id":"04f1366e-57a1-45b8-88c1-b00b23dc3616","class":"Projects::ImportExport::ExportService","current_user":"John Doe","project_full_path":"group1/test-export","file_path":"/path/to/archive","gc_stats":{"count":{"before":127,"after":127,"diff":0},"heap_allocated_pages":{"before":10369,"after":10369,"diff":0},"heap_sorted_length":{"before":10369,"after":10369,"diff":0},"heap_allocatable_pages":{"before":0,"after":0,"diff":0},"heap_available_slots":{"before":4226409,"after":4226409,"diff":0},"heap_live_slots":{"before":2542709,"after":2641420,"diff":98711},"heap_free_slots":{"before":1683700,"after":1584989,"diff":-98711},"heap_final_slots":{"before":0,"after":0,"diff":0},"heap_marked_slots":{"before":2542704,"after":2542704,"diff":0},"heap_eden_pages":{"before":10369,"after":10369,"diff":0},"heap_tomb_pages":{"before":0,"after":0,"diff":0},"total_allocated_pages":{"before":10369,"after":10369,"diff":0},"total_freed_pages":{"before":0,"after":0,"diff":0},"total_allocated_objects":{"before":24896308,"after":24995019,"diff":98711},"total_freed_objects":{"before":22353599,"after":22353599,"diff":0},"malloc_increase_bytes":{"before":140032,"after":6650240,"diff":6510208},"malloc_increase_bytes_limit":{"before":25804104,"after":25804104,"diff":0},"minor_gc_count":{"before":94,"after":94,"diff":0},"major_gc_count":{"before":33,"after":33,"diff":0},"remembered_wb_unprotected_objects":{"before":34284,"after":34284,"diff":0},"remembered_wb_unprotected_objects_limit":{"before":68568,"after":68568,"diff":0},"old_objects":{"before":2404725,"after":2404725,"diff":0},"old_objects_limit":{"before":4809450,"after":4809450,"diff":0},"oldmalloc_increase_bytes":{"before":140032,"after":6650240,"diff":6510208},"oldmalloc_increase_bytes_limit":{"before":68537556,"after":68537556,"diff":0}},"time_to_finish":0.12298400001600385,"number_of_sql_calls":70,"memory_usage":"0.0 MiB","label":"process_48616"}
+```
+
+## `geo.log` **(PREMIUM SELF)**
+
+Geo stores structured log messages in a `geo.log` file. For Omnibus GitLab
+installations, this file is at `/var/log/gitlab/gitlab-rails/geo.log`.
+
+This file contains information about when Geo attempts to sync repositories
+and files. Each line in the file contains a separate JSON entry that can be
+ingested into (for example, Elasticsearch or Splunk).
+
+For example:
+
+```json
+{"severity":"INFO","time":"2017-08-06T05:40:16.104Z","message":"Repository update","project_id":1,"source":"repository","resync_repository":true,"resync_wiki":true,"class":"Gitlab::Geo::LogCursor::Daemon","cursor_delay_s":0.038}
+```
+
+This message shows that Geo detected that a repository update was needed for project `1`.
+
+## `update_mirror_service_json.log`
+
+Depending on your installation method, this file is located at:
+
+- Omnibus GitLab: `/var/log/gitlab/gitlab-rails/update_mirror_service_json.log`
+- Installations from source: `/home/git/gitlab/log/update_mirror_service_json.log`
+
+This file contains information about LFS errors that occurred during project mirroring.
+While we work to move other project mirroring errors into this log, the [general log](#productionlog)
+can be used.
+
+```json
+{
+   "severity":"ERROR",
+   "time":"2020-07-28T23:29:29.473Z",
+   "correlation_id":"5HgIkCJsO53",
+   "user_id":"x",
+   "project_id":"x",
+   "import_url":"https://mirror-source/group/project.git",
+   "error_message":"The LFS objects download list couldn't be imported. Error: Unauthorized"
+}
+```
+
+## Registry Logs
+
+For Omnibus GitLab installations, Container Registry logs are in `/var/log/gitlab/registry/current`.
+
+## NGINX Logs
+
+For Omnibus GitLab installations, NGINX logs are in:
+
+- `/var/log/gitlab/nginx/gitlab_access.log`: A log of requests made to GitLab
+- `/var/log/gitlab/nginx/gitlab_error.log`: A log of NGINX errors for GitLab
+- `/var/log/gitlab/nginx/gitlab_pages_access.log`: A log of requests made to Pages static sites
+- `/var/log/gitlab/nginx/gitlab_pages_error.log`: A log of NGINX errors for Pages static sites
+- `/var/log/gitlab/nginx/gitlab_registry_access.log`: A log of requests made to the Container Registry
+- `/var/log/gitlab/nginx/gitlab_registry_error.log`: A log of NGINX errors for the Container Registry
+- `/var/log/gitlab/nginx/gitlab_mattermost_access.log`: A log of requests made to Mattermost
+- `/var/log/gitlab/nginx/gitlab_mattermost_error.log`: A log of NGINX errors for Mattermost
+
+Below is the default GitLab NGINX access log format:
+
+```plaintext
+$remote_addr - $remote_user [$time_local] "$request" $status $body_bytes_sent "$http_referer" "$http_user_agent"
+```
+
+## Pages Logs
+
+For Omnibus GitLab installations, Pages logs are in `/var/log/gitlab/gitlab-pages/current`.
+
+For example:
+
+```json
+{
+  "level": "info",
+  "msg": "GitLab Pages Daemon",
+  "revision": "52b2899",
+  "time": "2020-04-22T17:53:12Z",
+  "version": "1.17.0"
+}
+{
+  "level": "info",
+  "msg": "URL: https://gitlab.com/gitlab-org/gitlab-pages",
+  "time": "2020-04-22T17:53:12Z"
+}
+{
+  "gid": 998,
+  "in-place": false,
+  "level": "info",
+  "msg": "running the daemon as unprivileged user",
+  "time": "2020-04-22T17:53:12Z",
+  "uid": 998
+}
+```
+
+## Mattermost Logs
+
+For Omnibus GitLab installations, Mattermost logs are in these locations:
+
+- `/var/log/gitlab/mattermost/mattermost.log`
+- `/var/log/gitlab/mattermost/current`
+
+## Workhorse Logs
+
+For Omnibus GitLab installations, Workhorse logs are in `/var/log/gitlab/gitlab-workhorse/current`.
+
+## PostgreSQL Logs
+
+For Omnibus GitLab installations, PostgreSQL logs are in `/var/log/gitlab/postgresql/current`.
+
+## Prometheus Logs
+
+For Omnibus GitLab installations, Prometheus logs are in `/var/log/gitlab/prometheus/current`.
+
+## Redis Logs
+
+For Omnibus GitLab installations, Redis logs are in `/var/log/gitlab/redis/current`.
+
+## Alertmanager Logs
+
+For Omnibus GitLab installations, Alertmanager logs are in `/var/log/gitlab/alertmanager/current`.
+
+<!-- vale gitlab.Spelling = NO -->
+
+## Crond Logs
+
+For Omnibus GitLab installations, crond logs are in `/var/log/gitlab/crond/`.
+
+<!-- vale gitlab.Spelling = YES -->
+
+## Grafana Logs
+
+For Omnibus GitLab installations, Grafana logs are in `/var/log/gitlab/grafana/current`.
+
+## LogRotate Logs
+
+For Omnibus GitLab installations, `logrotate` logs are in `/var/log/gitlab/logrotate/current`.
+
+## GitLab Monitor Logs
+
+For Omnibus GitLab installations, GitLab Monitor logs are in `/var/log/gitlab/gitlab-monitor/`.
+
+## GitLab Exporter
+
+For Omnibus GitLab installations, GitLab Exporter logs are in `/var/log/gitlab/gitlab-exporter/current`.
+
+## GitLab agent server
+
+For Omnibus GitLab installations, GitLab agent server logs are
+in `/var/log/gitlab/gitlab-kas/current`.
+
+## Praefect Logs
+
+For Omnibus GitLab installations, Praefect logs are in `/var/log/gitlab/praefect/`.
+
+GitLab also tracks [Prometheus metrics for Praefect](../gitaly/monitoring.md#monitor-gitaly-cluster).
+
+## Backup log
+
+> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/63832) in GitLab 14.1.
+
+For Omnibus installations, the backup log is located at `/var/log/gitlab/gitlab-rails/backup_json.log`.
+
+This log is populated when a [GitLab backup is created](../../raketasks/backup_restore.md). You can use this log to understand how the backup process performed.
+
+## Performance bar stats
+
+> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/48149) in GitLab 13.7.
+
+Depending on your installation method, this file is located at:
+
+- Omnibus GitLab: `/var/log/gitlab/gitlab-rails/performance_bar_json.log`
+- Installations from source: `/home/git/gitlab/log/performance_bar_json.log`
+
+Performance bar statistics (currently only duration of SQL queries) are recorded
+in that file. For example:
+
+```json
+{"severity":"INFO","time":"2020-12-04T09:29:44.592Z","correlation_id":"33680b1490ccd35981b03639c406a697","filename":"app/models/ci/pipeline.rb","method_path":"app/models/ci/pipeline.rb:each_with_object","request_id":"rYHomD0VJS4","duration_ms":26.889,"count":2,"query_type": "active-record"}
+```
+
+These statistics are logged on .com only, disabled on self-deployments.
+
+## Gathering logs
+
+When [troubleshooting](../troubleshooting/index.md) issues that aren't localized to one of the
+previously listed components, it's helpful to simultaneously gather multiple logs and statistics
+from a GitLab instance.
+
+NOTE:
+GitLab Support often asks for one of these, and maintains the required tools.
+
+### Briefly tail the main logs
+
+If the bug or error is readily reproducible, save the main GitLab logs
+[to a file](../troubleshooting/linux_cheat_sheet.md#files-and-directories) while reproducing the
+problem a few times:
+
+```shell
+sudo gitlab-ctl tail | tee /tmp/<case-ID-and-keywords>.log
+```
+
+Conclude the log gathering with <kbd>Control</kbd> + <kbd>C</kbd>.
+
+### GitLabSOS
+
+If performance degradations or cascading errors occur that can't readily be attributed to one
+of the previously listed GitLab components, [GitLabSOS](https://gitlab.com/gitlab-com/support/toolbox/gitlabsos/)
+can provide a broader perspective of the GitLab instance. For more details and instructions
+to run it, read [the GitLabSOS documentation](https://gitlab.com/gitlab-com/support/toolbox/gitlabsos/#gitlabsos).
+
+### Fast-stats
+
+[Fast-stats](https://gitlab.com/gitlab-com/support/toolbox/fast-stats) is a tool
+for creating and comparing performance statistics from GitLab logs.
+For more details and instructions to run it, read the
+[documentation for fast-stats](https://gitlab.com/gitlab-com/support/toolbox/fast-stats#usage).
+
+## Find relevant log entries with a correlation ID
+
+Most requests have a log ID that can be used to [find relevant log entries](tracing_correlation_id.md).
diff --git a/doc/administration/logs/log_parsing.md b/doc/administration/logs/log_parsing.md
new file mode 100644
index 00000000000..20b439af49b
--- /dev/null
+++ b/doc/administration/logs/log_parsing.md
@@ -0,0 +1,316 @@
+---
+stage: Systems
+group: Distribution
+info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments
+---
+
+# Parsing GitLab logs with `jq` **(FREE SELF)**
+
+We recommend using log aggregation and search tools like Kibana and Splunk whenever possible,
+but if they are not available you can still quickly parse
+[GitLab logs](../logs/index.md) in JSON format
+(the default in GitLab 12.0 and later) using [`jq`](https://stedolan.github.io/jq/).
+
+NOTE:
+Specifically for summarizing error events and basic usage statistics,
+the GitLab Support Team provides the specialised
+[`fast-stats` tool](https://gitlab.com/gitlab-com/support/toolbox/fast-stats/#when-to-use-it).
+
+## What is JQ?
+
+As noted in its [manual](https://stedolan.github.io/jq/manual/), `jq` is a command-line JSON processor. The following examples
+include use cases targeted for parsing GitLab log files.
+
+## Parsing Logs
+
+The examples listed below address their respective log files by
+their relative Omnibus paths and default filenames.
+Find the respective full paths in the [GitLab logs sections](../logs/index.md#production_jsonlog).
+
+### General Commands
+
+#### Pipe colorized `jq` output into `less`
+
+```shell
+jq . <FILE> -C | less -R
+```
+
+#### Search for a term and pretty-print all matching lines
+
+```shell
+grep <TERM> <FILE> | jq .
+```
+
+#### Skip invalid lines of JSON
+
+```shell
+jq -cR 'fromjson?' file.json | jq <COMMAND>
+```
+
+By default `jq` errors out when it encounters a line that is not valid JSON.
+This skips over all invalid lines and parses the rest.
+
+#### Print a JSON log's time range
+
+```shell
+cat log.json | (head -1; tail -1) | jq '.time'
+```
+
+Use `zcat` if the file has been rotated and compressed:
+
+```shell
+zcat @400000006026b71d1a7af804.s | (head -1; tail -1) | jq '.time'
+
+zcat some_json.log.25.gz | (head -1; tail -1) | jq '.time'
+```
+
+#### Get activity for correlation ID across multiple JSON logs in chronological order
+
+```shell
+grep -hR <correlationID> | jq -c -R 'fromjson?' | jq -C -s 'sort_by(.time)'  | less -R
+```
+
+### Parsing `gitlab-rails/production_json.log` and `gitlab-rails/api_json.log`
+
+#### Find all requests with a 5XX status code
+
+```shell
+jq 'select(.status >= 500)' <FILE>
+```
+
+#### Top 10 slowest requests
+
+```shell
+jq -s 'sort_by(-.duration_s) | limit(10; .[])' <FILE>
+```
+
+#### Find and pretty print all requests related to a project
+
+```shell
+grep <PROJECT_NAME> <FILE> | jq .
+```
+
+#### Find all requests with a total duration > 5 seconds
+
+```shell
+jq 'select(.duration_s > 5000)' <FILE>
+```
+
+#### Find all project requests with more than 5 rugged calls
+
+```shell
+grep <PROJECT_NAME> <FILE> | jq 'select(.rugged_calls > 5)'
+```
+
+#### Find all requests with a Gitaly duration > 10 seconds
+
+```shell
+jq 'select(.gitaly_duration_s > 10000)' <FILE>
+```
+
+#### Find all requests with a queue duration > 10 seconds
+
+```shell
+jq 'select(.queue_duration_s > 10000)' <FILE>
+```
+
+#### Top 10 requests by # of Gitaly calls
+
+```shell
+jq -s 'map(select(.gitaly_calls != null)) | sort_by(-.gitaly_calls) | limit(10; .[])' <FILE>
+```
+
+### Parsing `gitlab-rails/production_json.log`
+
+#### Print the top three controller methods by request volume and their three longest durations
+
+```shell
+jq -s -r 'group_by(.controller+.action) | sort_by(-length) | limit(3; .[]) | sort_by(-.duration_s) | "CT: \(length)\tMETHOD: \(.[0].controller)#\(.[0].action)\tDURS: \(.[0].duration_s),  \(.[1].duration_s),  \(.[2].duration_s)"' production_json.log
+```
+
+**Example output**
+
+```plaintext
+CT: 2721   METHOD: SessionsController#new  DURS: 844.06,  713.81,  704.66
+CT: 2435   METHOD: MetricsController#index DURS: 299.29,  284.01,  158.57
+CT: 1328   METHOD: Projects::NotesController#index DURS: 403.99,  386.29,  384.39
+```
+
+### Parsing `gitlab-rails/api_json.log`
+
+#### Print top three routes with request count and their three longest durations
+
+```shell
+jq -s -r 'group_by(.route) | sort_by(-length) | limit(3; .[]) | sort_by(-.duration_s) | "CT: \(length)\tROUTE: \(.[0].route)\tDURS: \(.[0].duration_s),  \(.[1].duration_s),  \(.[2].duration_s)"' api_json.log
+```
+
+**Example output**
+
+```plaintext
+CT: 2472 ROUTE: /api/:version/internal/allowed   DURS: 56402.65,  38411.43,  19500.41
+CT: 297  ROUTE: /api/:version/projects/:id/repository/tags       DURS: 731.39,  685.57,  480.86
+CT: 190  ROUTE: /api/:version/projects/:id/repository/commits    DURS: 1079.02,  979.68,  958.21
+```
+
+### Print top API user agents
+
+```shell
+jq --raw-output '[.route, .ua] | @tsv' api_json.log | sort | uniq -c | sort -n
+```
+
+**Example output**:
+
+```plaintext
+  89 /api/:version/usage_data/increment_unique_users  # plus browser details
+ 567 /api/:version/jobs/:id/trace       gitlab-runner # plus version details
+1234 /api/:version/internal/allowed     GitLab-Shell
+```
+
+This sample response seems normal. A custom tool or script might be causing a high load
+if the output contains many:
+
+- Third party libraries like `python-requests` or `curl`.
+- [GitLab CLI clients](https://about.gitlab.com/partners/technology-partners/#cli-clients).
+
+You can also [use `fast-stats top`](#parsing-gitlab-logs-with-jq) to extract performance statistics.
+
+### Parsing `gitlab-workhorse/current`
+
+### Print top Workhorse user agents
+
+```shell
+jq --raw-output '[.uri, .user_agent] | @tsv' current | sort | uniq -c | sort -n
+```
+
+**Example output**:
+
+```plaintext
+  89 /api/graphql # plus browser details
+ 567 /api/v4/internal/allowed   GitLab-Shell
+1234 /api/v4/jobs/request       gitlab-runner # plus version details
+```
+
+Similar to the [API `ua` data](#print-top-api-user-agents),
+deviations from this common order might indicate scripts that could be optimized.
+
+The performance impact of runners checking for new jobs can be reduced by increasing
+[the `check_interval` setting](https://docs.gitlab.com/runner/configuration/advanced-configuration.html#the-global-section),
+for example.
+
+### Parsing `gitlab-rails/geo.log`
+
+#### Find most common Geo sync errors
+
+If [the `geo:status` Rake task](../geo/replication/troubleshooting.md#sync-status-rake-task)
+repeatedly reports that some items never reach 100%,
+the following command helps to focus on the most common errors.
+
+```shell
+jq --raw-output 'select(.severity == "ERROR") | [.project_path, .message] | @tsv' geo.log | sort | uniq -c | sort | tail
+```
+
+### Parsing `gitaly/current`
+
+Use the following examples to [troubleshoot Gitaly](../gitaly/troubleshooting.md).
+
+#### Find all Gitaly requests sent from web UI
+
+```shell
+jq 'select(."grpc.meta.client_name" == "gitlab-web")' current
+```
+
+#### Find all failed Gitaly requests
+
+```shell
+jq 'select(."grpc.code" != null and ."grpc.code" != "OK")' current
+```
+
+#### Find all requests that took longer than 30 seconds
+
+```shell
+jq 'select(."grpc.time_ms" > 30000)' current
+```
+
+#### Print top ten projects by request volume and their three longest durations
+
+```shell
+jq --raw-output --slurp '
+  map(
+    select(
+      ."grpc.request.glProjectPath" != null
+      and ."grpc.request.glProjectPath" != ""
+      and ."grpc.time_ms" != null
+    )
+  )
+  | group_by(."grpc.request.glProjectPath")
+  | sort_by(-length)
+  | limit(10; .[])
+  | sort_by(-."grpc.time_ms")
+  | [
+      length,
+      .[0]."grpc.time_ms",
+      .[1]."grpc.time_ms",
+      .[2]."grpc.time_ms",
+      .[0]."grpc.request.glProjectPath"
+    ]
+  | @sh' current \
+| awk 'BEGIN { printf "%7s %10s %10s %10s\t%s\n", "CT", "MAX DURS", "", "", "PROJECT" }
+  { printf "%7u %7u ms, %7u ms, %7u ms\t%s\n", $1, $2, $3, $4, $5 }'
+```
+
+**Example output**
+
+```plaintext
+   CT    MAX DURS                              PROJECT
+  206    4898 ms,    1101 ms,    1032 ms      'groupD/project4'
+  109    1420 ms,     962 ms,     875 ms      'groupEF/project56'
+  663     106 ms,      96 ms,      94 ms      'groupABC/project123'
+  ...
+```
+
+#### Find all projects affected by a fatal Git problem
+
+```shell
+grep "fatal: " current | \
+    jq '."grpc.request.glProjectPath"' | \
+    sort | uniq
+```
+
+### Parsing `gitlab-shell/gitlab-shell.log`
+
+For investigating Git calls via SSH, from [GitLab 12.10](https://gitlab.com/gitlab-org/gitlab-shell/-/merge_requests/367).
+
+Find the top 20 calls by project and user:
+
+```shell
+jq --raw-output --slurp '
+  map(
+    select(
+      .username != null and
+      .gl_project_path !=null
+    )
+  )
+  | group_by(.username+.gl_project_path)
+  | sort_by(-length)
+  | limit(20; .[])
+  | "count: \(length)\tuser: \(.[0].username)\tproject: \(.[0].gl_project_path)" ' \
+  gitlab-shell.log
+```
+
+Find the top 20 calls by project, user, and command:
+
+```shell
+jq --raw-output --slurp '
+  map(
+    select(
+      .command  != null and
+      .username != null and
+      .gl_project_path !=null
+    )
+  )
+  | group_by(.username+.gl_project_path+.command)
+  | sort_by(-length)
+  | limit(20; .[])
+  | "count: \(length)\tcommand: \(.[0].command)\tuser: \(.[0].username)\tproject: \(.[0].gl_project_path)" ' \
+  gitlab-shell.log
+```
diff --git a/doc/administration/logs/tracing_correlation_id.md b/doc/administration/logs/tracing_correlation_id.md
new file mode 100644
index 00000000000..1d3e4955d3a
--- /dev/null
+++ b/doc/administration/logs/tracing_correlation_id.md
@@ -0,0 +1,202 @@
+---
+stage: Systems
+group: Distribution
+info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments
+---
+
+# Find relevant log entries with a correlation ID **(FREE SELF)**
+
+GitLab instances log a unique request tracking ID (known as the
+"correlation ID") for most requests. Each individual request to GitLab gets
+its own correlation ID, which then gets logged in each GitLab component's logs for that
+request. This makes it easier to trace behavior in a
+distributed system. Without this ID it can be difficult or
+impossible to match correlating log entries.
+
+## Identify the correlation ID for a request
+
+The correlation ID is logged in structured logs under the key `correlation_id`
+and in all response headers GitLab sends under the header `x-request-id`.
+You can find your correlation ID by searching in either place.
+
+### Getting the correlation ID in your browser
+
+You can use your browser's developer tools to monitor and inspect network
+activity with the site that you're visiting. See the links below for network monitoring
+documentation for some popular browsers.
+
+- [Network Monitor - Firefox Developer Tools](https://developer.mozilla.org/en-US/docs/Tools/Network_Monitor)
+- [Inspect Network Activity In Chrome DevTools](https://developer.chrome.com/docs/devtools/network/)
+- [Safari Web Development Tools](https://developer.apple.com/safari/tools/)
+- [Microsoft Edge Network panel](https://docs.microsoft.com/en-us/microsoft-edge/devtools-guide-chromium/network/)
+
+To locate a relevant request and view its correlation ID:
+
+1. Enable persistent logging in your network monitor. Some actions in GitLab redirect you quickly after you submit a form, so this helps capture all relevant activity.
+1. To help isolate the requests you are looking for, you can filter for `document` requests.
+1. Select the request of interest to view further detail.
+1. Go to the **Headers** section and look for **Response Headers**. There you should find an `x-request-id` header with a
+value that was randomly generated by GitLab for the request.
+
+See the following example:
+
+![Firefox's network monitor showing an request ID header](img/network_monitor_xid.png)
+
+### Getting the correlation ID from your logs
+
+Another approach to finding the correct correlation ID is to search or watch
+your logs and find the `correlation_id` value for the log entry that you're
+watching for.
+
+For example, let's say that you want learn what's happening or breaking when
+you reproduce an action in GitLab. You could tail the GitLab logs, filtering
+to requests by your user, and then watch the requests until you see what you're
+interested in.
+
+### Getting the correlation ID from curl
+
+If you're using `curl` then you can use the verbose option to show request and response headers, as well as other debug information.
+
+```shell
+➜  ~ curl --verbose "https://gitlab.example.com/api/v4/projects"
+# look for a line that looks like this
+< x-request-id: 4rAMkV3gof4
+```
+
+#### Using jq
+
+This example uses [jq](https://stedolan.github.io/jq/) to filter results and
+display values we most likely care about.
+
+```shell
+sudo gitlab-ctl tail gitlab-rails/production_json.log | jq 'select(.username == "bob") | "User: \(.username), \(.method) \(.path), \(.controller)#\(.action), ID: \(.correlation_id)"'
+```
+
+```plaintext
+"User: bob, GET /root/linux, ProjectsController#show, ID: U7k7fh6NpW3"
+"User: bob, GET /root/linux/commits/master/signatures, Projects::CommitsController#signatures, ID: XPIHpctzEg1"
+"User: bob, GET /root/linux/blob/master/README, Projects::BlobController#show, ID: LOt9hgi1TV4"
+```
+
+#### Using grep
+
+This example uses only `grep` and `tr`, which are more likely to be installed than `jq`.
+
+```shell
+sudo gitlab-ctl tail gitlab-rails/production_json.log | grep '"username":"bob"' | tr ',' '\n' | egrep 'method|path|correlation_id'
+```
+
+```plaintext
+{"method":"GET"
+"path":"/root/linux"
+"username":"bob"
+"correlation_id":"U7k7fh6NpW3"}
+{"method":"GET"
+"path":"/root/linux/commits/master/signatures"
+"username":"bob"
+"correlation_id":"XPIHpctzEg1"}
+{"method":"GET"
+"path":"/root/linux/blob/master/README"
+"username":"bob"
+"correlation_id":"LOt9hgi1TV4"}
+```
+
+## Searching your logs for the correlation ID
+
+Once you have the correlation ID you can start searching for relevant log
+entries. You can filter the lines by the correlation ID itself.
+Combining a `find` and `grep` should be sufficient to find the entries you are looking for.
+
+```shell
+# find <gitlab log directory> -type f -mtime -0 exec grep '<correlation ID>' '{}' '+'
+find /var/log/gitlab -type f -mtime 0 -exec grep 'LOt9hgi1TV4' '{}' '+'
+```
+
+```plaintext
+/var/log/gitlab/gitlab-workhorse/current:{"correlation_id":"LOt9hgi1TV4","duration_ms":2478,"host":"gitlab.domain.tld","level":"info","method":"GET","msg":"access","proto":"HTTP/1.1","referrer":"https://gitlab.domain.tld/root/linux","remote_addr":"68.0.116.160:0","remote_ip":"[filtered]","status":200,"system":"http","time":"2019-09-17T22:17:19Z","uri":"/root/linux/blob/master/README?format=json\u0026viewer=rich","user_agent":"Mozilla/5.0 (Mac) Gecko Firefox/69.0","written_bytes":1743}
+/var/log/gitlab/gitaly/current:{"correlation_id":"LOt9hgi1TV4","grpc.code":"OK","grpc.meta.auth_version":"v2","grpc.meta.client_name":"gitlab-web","grpc.method":"FindCommits","grpc.request.deadline":"2019-09-17T22:17:47Z","grpc.request.fullMethod":"/gitaly.CommitService/FindCommits","grpc.request.glProjectPath":"root/linux","grpc.request.glRepository":"project-1","grpc.request.repoPath":"@hashed/6b/86/6b86b273ff34fce19d6b804eff5a3f5747ada4eaa22f1d49c01e52ddb7875b4b.git","grpc.request.repoStorage":"default","grpc.request.topLevelGroup":"@hashed","grpc.service":"gitaly.CommitService","grpc.start_time":"2019-09-17T22:17:17Z","grpc.time_ms":2319.161,"level":"info","msg":"finished streaming call with code OK","peer.address":"@","span.kind":"server","system":"grpc","time":"2019-09-17T22:17:19Z"}
+/var/log/gitlab/gitlab-rails/production_json.log:{"method":"GET","path":"/root/linux/blob/master/README","format":"json","controller":"Projects::BlobController","action":"show","status":200,"duration":2448.77,"view":0.49,"db":21.63,"time":"2019-09-17T22:17:19.800Z","params":[{"key":"viewer","value":"rich"},{"key":"namespace_id","value":"root"},{"key":"project_id","value":"linux"},{"key":"id","value":"master/README"}],"remote_ip":"[filtered]","user_id":2,"username":"bob","ua":"Mozilla/5.0 (Mac) Gecko Firefox/69.0","queue_duration":3.38,"gitaly_calls":1,"gitaly_duration":0.77,"rugged_calls":4,"rugged_duration_ms":28.74,"correlation_id":"LOt9hgi1TV4"}
+```
+
+### Searching in distributed architectures
+
+If you have done some horizontal scaling in your GitLab infrastructure, then
+you must search across _all_ of your GitLab nodes. You can do this with
+some sort of log aggregation software like Loki, ELK, Splunk, or others.
+
+You can use a tool like Ansible or PSSH (parallel SSH) that can execute identical commands across your servers in
+parallel, or craft your own solution.
+
+### Viewing the request in the Performance Bar
+
+You can use the [performance bar](../monitoring/performance/performance_bar.md) to view interesting data including calls made to SQL and Gitaly.
+
+To view the data, the correlation ID of the request must match the same session as the user
+viewing the performance bar. For API requests, this means that you must perform the request
+using the session cookie of the signed-in user.
+
+For example, if you want to view the database queries executed for the following API endpoint:
+
+```shell
+https://gitlab.com/api/v4/groups/2564205/projects?with_security_reports=true&page=1&per_page=1
+```
+
+First, enable the **Developer Tools** panel. See [Getting the correlation ID in your browser](#getting-the-correlation-id-in-your-browser) for details on how to do this.
+
+After developer tools have been enabled, obtain a session cookie as follows:
+
+1. Visit <https://gitlab.com> while logged in.
+1. Optional. Select **Fetch/XHR** request filter in the **Developer Tools** panel. This step is described for Google Chrome developer tools and is not strictly necessary, it just makes it easier to find the correct request.
+1. Select the `results?request_id=<some-request-id>` request on the left hand side.
+1. The session cookie is displayed under the `Request Headers` section of the `Headers` panel. Right-click on the cookie value and select `Copy value`.
+
+![Obtaining a session cookie for request](img/obtaining-a-session-cookie-for-request_v14_3.png)
+
+You have the value of the session cookie copied to your clipboard, for example:
+
+```shell
+experimentation_subject_id=<subject-id>; _gitlab_session=<session-id>; event_filter=all; visitor_id=<visitor-id>; perf_bar_enabled=true; sidebar_collapsed=true; diff_view=inline; sast_entry_point_dismissed=true; auto_devops_settings_dismissed=true; cf_clearance=<cf-clearance>; collapsed_gutter=false; frequently_used_emojis=clap,thumbsup,rofl,tada,eyes,bow
+```
+
+Use the value of the session cookie to craft an API request by pasting it into a custom header of a `curl` request:
+
+```shell
+$ curl --include "https://gitlab.com/api/v4/groups/2564205/projects?with_security_reports=true&page=1&per_page=1" \
+--header 'cookie: experimentation_subject_id=<subject-id>; _gitlab_session=<session-id>; event_filter=all; visitor_id=<visitor-id>; perf_bar_enabled=true; sidebar_collapsed=true; diff_view=inline; sast_entry_point_dismissed=true; auto_devops_settings_dismissed=true; cf_clearance=<cf-clearance>; collapsed_gutter=false; frequently_used_emojis=clap,thumbsup,rofl,tada,eyes,bow'
+
+  date: Tue, 28 Sep 2021 03:55:33 GMT
+  content-type: application/json
+  ...
+  x-request-id: 01FGN8P881GF2E5J91JYA338Y3
+  ...
+  [
+    {
+      "id":27497069,
+      "description":"Analyzer for images used on live K8S containers based on Starboard"
+    },
+    "container_registry_image_prefix":"registry.gitlab.com/gitlab-org/security-products/analyzers/cluster-image-scanning",
+    "..."
+  ]
+```
+
+The response contains the data from the API endpoint, and a `correlation_id` value, returned in the `x-request-id` header, as described in the [Identify the correlation ID for a request](#identify-the-correlation-id-for-a-request) section.
+
+You can then view the database details for this request:
+
+1. Paste the `x-request-id` value into the `request details` field of the [performance bar](../monitoring/performance/performance_bar.md) and press <kbd>Enter/Return</kbd>. This example uses the `x-request-id` value `01FGN8P881GF2E5J91JYA338Y3`, returned by the above response:
+
+   ![Paste request ID into progress bar](img/paste-request-id-into-progress-bar_v14_3.png)
+
+1. A new request is inserted into the `Request Selector` dropdown on the right-hand side of the Performance Bar. Select the new request to view the metrics of the API request:
+
+   ![Select request ID from request selector drop down menu](img/select-request-id-from-request-selector-drop-down-menu_v14_3.png)
+
+   <!-- vale gitlab.Substitutions = NO -->
+1. Select the `pg` link in the Progress Bar to view the database queries executed by the API request:
+
+   ![View pg database details](img/view-pg-details_v14_3.png)
+   <!-- vale gitlab.Substitutions = YES -->
+
+   The database query dialog is displayed:
+
+   ![Database query dialog](img/database-query-dialog_v14_3.png)