diff options
author | GitLab Bot <gitlab-bot@gitlab.com> | 2023-05-17 19:05:49 +0300 |
---|---|---|
committer | GitLab Bot <gitlab-bot@gitlab.com> | 2023-05-17 19:05:49 +0300 |
commit | 43a25d93ebdabea52f99b05e15b06250cd8f07d7 (patch) | |
tree | dceebdc68925362117480a5d672bcff122fb625b /doc/administration/logs | |
parent | 20c84b99005abd1c82101dfeff264ac50d2df211 (diff) |
Add latest changes from gitlab-org/gitlab@16-0-stable-eev16.0.0-rc42
Diffstat (limited to 'doc/administration/logs')
-rw-r--r-- | doc/administration/logs/index.md | 78 | ||||
-rw-r--r-- | doc/administration/logs/log_parsing.md | 20 |
2 files changed, 79 insertions, 19 deletions
diff --git a/doc/administration/logs/index.md b/doc/administration/logs/index.md index eab4c9b7d83..7fab97f76da 100644 --- a/doc/administration/logs/index.md +++ b/doc/administration/logs/index.md @@ -96,6 +96,7 @@ except those captured by `runit`. | [LogRotate logs](#logrotate-logs) | **{dotted-circle}** No | **{check-circle}** Yes | | [Mailroom](#mail_room_jsonlog-default) | **{check-circle}** Yes | **{check-circle}** Yes | | [NGINX](#nginx-logs) | **{check-circle}** Yes | **{check-circle}** Yes | +| [PgBouncer logs](#pgbouncer-logs) | **{dotted-circle}** No | **{check-circle}** Yes | | [PostgreSQL logs](#postgresql-logs) | **{dotted-circle}** No | **{check-circle}** Yes | | [Praefect logs](#praefect-logs) | **{dotted-circle}** Yes | **{check-circle}** Yes | | [Prometheus logs](#prometheus-logs) | **{dotted-circle}** No | **{check-circle}** Yes | @@ -180,7 +181,7 @@ In addition, the log contains the originating IP address, (`remote_ip`), the user's ID (`user_id`), and username (`username`). Some endpoints (such as `/search`) may make requests to Elasticsearch if using -[Advanced Search](../../user/search/advanced_search.md). These +[advanced search](../../user/search/advanced_search.md). These additionally log `elasticsearch_calls` and `elasticsearch_call_duration_s`, which correspond to: @@ -349,15 +350,17 @@ the `view_duration_s` is calculated by [`duration_s - db_duration_s`](https://gi Therefore, `view_duration_s` can be affected by multiple different factors, like read-write process on Redis or external HTTP, not only the serialization process. -## `application.log` +## `application.log` (deprecated) + +> [Deprecated](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/111046) in GitLab 15.10. Depending on your installation method, this file is located at: - Omnibus GitLab: `/var/log/gitlab/gitlab-rails/application.log` - Installations from source: `/home/git/gitlab/log/application.log` -It helps you discover events happening in your instance such as user creation -and project deletion. For example: +It contains a less structured version of the logs in +[`application_json.log`](#application_jsonlog), like this example: ```plaintext October 06, 2014 11:56: User "Administrator" (admin@example.com) was created @@ -376,7 +379,8 @@ Depending on your installation method, this file is located at: - Omnibus GitLab: `/var/log/gitlab/gitlab-rails/application_json.log` - Installations from source: `/home/git/gitlab/log/application_json.log` -It contains the JSON version of the logs in `application.log`, like this example: +It helps you discover events happening in your instance such as user creation +and project deletion. For example: ```json { @@ -426,7 +430,7 @@ like this example: } ``` -## `kubernetes.log` (DEPRECATED) +## `kubernetes.log` (deprecated) > [Deprecated](https://gitlab.com/groups/gitlab-org/configure/-/epics/8) in GitLab 14.5. @@ -686,14 +690,6 @@ Unix timestamp format, and `gzip`-compressed (like `@1584057562.s`). This file is at `/var/log/gitlab/gitlab-rails/grpc.log` for Omnibus GitLab packages. Native [gRPC](https://grpc.io/) logging used by Gitaly. -### `gitaly_ruby_json.log` - -> [Introduced](https://gitlab.com/gitlab-org/gitaly/-/merge_requests/2678) in GitLab 13.6. - -This file is at `/var/log/gitlab/gitaly/gitaly_ruby_json.log` and is -produced by [`gitaly-ruby`](../gitaly/reference.md#gitaly-ruby). It contains an -access log of gRPC calls made by Gitaly to `gitaly-ruby`. - ### `gitaly_hooks.log` This file is at `/var/log/gitlab/gitaly/gitaly_hooks.log` and is @@ -733,7 +729,7 @@ Depending on your installation method, this file is located at: - Omnibus GitLab: `/var/log/gitlab/gitlab-rails/importer.log` - Installations from source: `/home/git/gitlab/log/importer.log` -It logs the progress of the import process. +This file logs the progress of [project imports and migrations](../../user/project/import/index.md). ## `exporter.log` @@ -771,6 +767,24 @@ are recorded in this file. For example: {"severity":"INFO","time":"2020-11-24T02:31:29.329Z","correlation_id":null,"key":"cd_auto_rollback","action":"remove"} ``` +## `ci_resource_groups_json.log` + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/384180) in GitLab 15.9. + +Depending on your installation method, this file is located at: + +- Omnibus GitLab: `/var/log/gitlab/gitlab-rails/ci_resource_group_json.log` +- Installations from source: `/home/git/gitlab/log/ci_resource_group_json.log` + +It contains information about [resource group](../../ci/resource_groups/index.md) acquisition. For example: + +```json +{"severity":"INFO","time":"2023-02-10T23:02:06.095Z","correlation_id":"01GRYS10C2DZQ9J1G12ZVAD4YD","resource_group_id":1,"processable_id":288,"message":"attempted to assign resource to processable","success":true} +{"severity":"INFO","time":"2023-02-10T23:02:08.945Z","correlation_id":"01GRYS138MYEG32C0QEWMC4BDM","resource_group_id":1,"processable_id":288,"message":"attempted to release resource from processable","success":true} +``` + +The examples show the `resource_group_id`, `processable_id`, `message`, and `success` fields for each entry. + ## `auth.log` > Introduced in GitLab 12.0. @@ -787,6 +801,28 @@ This log records: - In GitLab versions [12.3](https://gitlab.com/gitlab-org/gitlab/-/issues/29239) and later, user ID and username, if available. +## `auth_json.log` + +Depending on your installation method, this file is located at: + +- Omnibus GitLab: `/var/log/gitlab/gitlab-rails/auth_json.log` +- Installations from source: `/home/git/gitlab/log/auth_json.log` + +This file contains the JSON version of the logs in `auth.log`, for example: + +```json +{ + "severity":"ERROR", + "time":"2023-04-19T22:14:25.893Z", + "correlation_id":"01GYDSAKAN2SPZPAMJNRWW5H8S", + "message":"Rack_Attack", + "env":"blocklist", + "remote_ip":"x.x.x.x", + "request_method":"GET", + "path":"/group/project.git/info/refs?service=git-upload-pack" +} +``` + ## `graphql_json.log` > [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/59587) in GitLab 12.0. @@ -811,6 +847,8 @@ Depending on your installation method, this file is located at: - Omnibus GitLab: `/var/log/gitlab/gitlab-rails/migrations.log` - Installations from source: `/home/git/gitlab/log/migrations.log` +This file logs the progress of [database migrations](../raketasks/maintenance.md#display-status-of-database-migrations). + ## `mail_room_json.log` (default) > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/19186) in GitLab 12.6. @@ -1016,9 +1054,13 @@ For Omnibus GitLab installations, NGINX logs are in: Below is the default GitLab NGINX access log format: ```plaintext -$remote_addr - $remote_user [$time_local] "$request" $status $body_bytes_sent "$http_referer" "$http_user_agent" +'$remote_addr - $remote_user [$time_local] "$request" $status $body_bytes_sent "$http_referer" "$http_user_agent"' ``` +The `$request` and `$http_referer` are +[filtered](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/support/nginx/gitlab) +for sensitive query string parameters such as secret tokens. + ## Pages logs For Omnibus GitLab installations, Pages logs are in `/var/log/gitlab/gitlab-pages/current`. @@ -1059,6 +1101,10 @@ For Omnibus GitLab installations, Mattermost logs are in these locations: For Omnibus GitLab installations, Workhorse logs are in `/var/log/gitlab/gitlab-workhorse/current`. +## PgBouncer logs + +For Omnibus GitLab installations, PgBouncer logs are in `/var/log/gitlab/pgbouncer/current`. + ## PostgreSQL logs For Omnibus GitLab installations, PostgreSQL logs are in `/var/log/gitlab/postgresql/current`. diff --git a/doc/administration/logs/log_parsing.md b/doc/administration/logs/log_parsing.md index 84c517e6120..d9520ea9bc0 100644 --- a/doc/administration/logs/log_parsing.md +++ b/doc/administration/logs/log_parsing.md @@ -158,7 +158,7 @@ CT: 297 ROUTE: /api/:version/projects/:id/repository/tags DURS: 731.39, CT: 190 ROUTE: /api/:version/projects/:id/repository/commits DURS: 1079.02, 979.68, 958.21 ``` -### Print top API user agents +#### Print top API user agents ```shell jq --raw-output '[.route, .ua] | @tsv' api_json.log | sort | uniq -c | sort -n @@ -180,9 +180,20 @@ if the output contains many: You can also [use `fast-stats top`](#parsing-gitlab-logs-with-jq) to extract performance statistics. +### Parsing `gitlab-rails/importer.log` + +To troubleshoot [project imports](../../administration/raketasks/project_import_export.md) or +[migrations](../../user/project/import/index.md), run this command: + +```shell +jq 'select(.project_path == "<namespace>/<project>").error_messages' importer.log +``` + +For common issues, see [troubleshooting](../../administration/raketasks/project_import_export.md#troubleshooting). + ### Parsing `gitlab-workhorse/current` -### Print top Workhorse user agents +#### Print top Workhorse user agents ```shell jq --raw-output '[.uri, .user_agent] | @tsv' current | sort | uniq -c | sort -n @@ -212,9 +223,12 @@ repeatedly reports that some items never reach 100%, the following command helps to focus on the most common errors. ```shell -jq --raw-output 'select(.severity == "ERROR") | [.project_path, .message] | @tsv' geo.log | sort | uniq -c | sort | tail +jq --raw-output 'select(.severity == "ERROR") | [.project_path, .class, .message, .error] | @tsv' geo.log | sort | uniq -c | sort | tail ``` +Refer to our [Geo troubleshooting page](../geo/replication/troubleshooting.md) +for advice about specific error messages. + ### Parsing `gitaly/current` Use the following examples to [troubleshoot Gitaly](../gitaly/troubleshooting.md). |