diff options
author | GitLab Bot <gitlab-bot@gitlab.com> | 2020-02-20 21:08:51 +0300 |
---|---|---|
committer | GitLab Bot <gitlab-bot@gitlab.com> | 2020-02-20 21:08:51 +0300 |
commit | df2eda3f14dccb703bd7054d4ddde7803cb1fe7e (patch) | |
tree | 0200bb0def01cde22da3bc4c9ed0a7b0f91d6b50 /doc | |
parent | b9bac6dbf78a5a7976fba14aaeef96bdeb0da612 (diff) |
Add latest changes from gitlab-org/gitlab@master
Diffstat (limited to 'doc')
-rw-r--r-- | doc/administration/audit_events.md | 58 | ||||
-rw-r--r-- | doc/administration/logs.md | 224 | ||||
-rw-r--r-- | doc/administration/monitoring/prometheus/index.md | 38 | ||||
-rw-r--r-- | doc/api/merge_trains.md | 80 | ||||
-rw-r--r-- | doc/development/adding_database_indexes.md | 2 | ||||
-rw-r--r-- | doc/development/migration_style_guide.md | 19 | ||||
-rw-r--r-- | doc/user/project/clusters/kubernetes_pod_logs.md | 10 |
7 files changed, 323 insertions, 108 deletions
diff --git a/doc/administration/audit_events.md b/doc/administration/audit_events.md index cf017c0167b..2f4079396e6 100644 --- a/doc/administration/audit_events.md +++ b/doc/administration/audit_events.md @@ -3,28 +3,28 @@ GitLab offers a way to view the changes made within the GitLab server for owners and administrators on a [paid plan][ee]. GitLab system administrators can also take advantage of the logs located on the -filesystem, see [the logs system documentation](logs.md) for more details. +filesystem. See [the logs system documentation](logs.md) for more details. ## Overview -**Audit Events** is a tool for GitLab owners and administrators to be -able to track important events such as who performed certain actions and the -time they happened. These actions could be, for example, change a user +**Audit Events** is a tool for GitLab owners and administrators +to track important events such as who performed certain actions and the +time they happened. For example, these actions could be a change to a user permission level, who added a new user, or who removed a user. -## Use-cases +## Use cases -- Check who the person was that changed the permission level of a particular - user for a project in GitLab. -- Use it to track which users have access to a certain group of projects - in GitLab, and who gave them that permission level. +- Check who changed the permission level of a particular + user for a GitLab project. +- Track which users have access to a certain group of projects + in GitLab, and who gave them that permission level. ## List of events There are two kinds of events logged: -- Events scoped to the group or project, used by group / project managers - to look up who made what change. +- Events scoped to the group or project, used by group and project managers + to look up who made a change. - Instance events scoped to the whole GitLab instance, used by your Compliance team to perform formal audits. @@ -36,9 +36,9 @@ You need Owner [permissions] to view the group Audit Events page. To view a group's audit events, navigate to **Group > Settings > Audit Events**. From there, you can see the following actions: -- Group name/path changed +- Group name or path changed - Group repository size limit changed -- Group created/deleted +- Group created or deleted - Group changed visibility - User was added to group and with which [permissions] - Permissions changes of a user assigned to a group @@ -48,11 +48,11 @@ From there, you can see the following actions: - [Project shared with group](../user/project/members/share_project_with_groups.md) and with which [permissions] - Removal of a previously shared group with a project -- LFS enabled/disabled +- LFS enabled or disabled - Shared runners minutes limit changed -- Membership lock enabled/disabled -- Request access enabled/disabled -- 2FA enforcement/grace period changed +- Membership lock enabled or disabled +- Request access enabled or disabled +- 2FA enforcement or grace period changed - Roles allowed to create project changed Group events can also be accessed via the [Group Audit Events API](../api/audit_events.md#group-audit-events-starter) @@ -65,8 +65,8 @@ You need Maintainer [permissions] or higher to view the project Audit Events pag To view a project's audit events, navigate to **Project > Settings > Audit Events**. From there, you can see the following actions: -- Added/removed deploy keys -- Project created/deleted/renamed/moved(transferred)/changed path +- Added or removed deploy keys +- Project created, deleted, renamed, moved(transferred), changed path - Project changed visibility level - User was added to project and with which [permissions] - Permission changes of a user assigned to a project @@ -75,7 +75,7 @@ From there, you can see the following actions: - Project repository was downloaded - Project was archived - Project was unarchived -- Added/removed/updated protected branches +- Added, removed, or updated protected branches - Release was added to a project - Release was updated - Release milestone associations changed @@ -94,20 +94,20 @@ In addition to the group and project events, the following user actions are also recorded: - Failed Logins -- Sign-in events and the authentication type (standard, LDAP, OmniAuth, etc.) +- Sign-in events and the authentication type (such as standard, LDAP, or OmniAuth) - Added SSH key -- Added/removed email +- Added or removed email - Changed password - Ask for password reset - Grant OAuth access -- Started/stopped user impersonation +- Started or stopped user impersonation - Changed username ([introduced](https://gitlab.com/gitlab-org/gitlab/issues/7797) in GitLab 12.8) - User was deleted ([introduced](https://gitlab.com/gitlab-org/gitlab/issues/251) in GitLab 12.8) - User was added ([introduced](https://gitlab.com/gitlab-org/gitlab/issues/251) in GitLab 12.8) - User was blocked via Admin Area ([introduced](https://gitlab.com/gitlab-org/gitlab/issues/251) in GitLab 12.8) -It is possible to filter particular actions by choosing an audit data type from -the filter dropdown box. You can further filter by specific group, project or user +It's possible to filter particular actions by choosing an audit data type from +the filter dropdown box. You can further filter by specific group, project, or user (for authentication events). ![audit log](img/audit_log.png) @@ -116,8 +116,8 @@ Instance events can also be accessed via the [Instance Audit Events API](../api/ ### Missing events -Some events are not being tracked in Audit Events. Please see the following -epics for more detail on which events are not being tracked and our progress +Some events are not tracked in Audit Events. See the following +epics for more detail on which events are not being tracked, and our progress on adding these events into GitLab: - [Project settings and activity](https://gitlab.com/groups/gitlab-org/-/epics/474) @@ -129,8 +129,8 @@ on adding these events into GitLab: #### Repository push The current architecture of audit events is not prepared to receive a very high amount of records. -It may make your project/admin audit logs UI very busy and the disk space consumed by the -`audit_events` Postgres table will increase considerably. Thus, it's disabled by default +It may make the user interface for your project or audit logs very busy, and the disk space consumed by the +`audit_events` Postgres table will increase considerably. It's disabled by default to prevent performance degradations on GitLab instances with very high Git write traffic. In an upcoming release, Audit Logs for Git push events will be enabled diff --git a/doc/administration/logs.md b/doc/administration/logs.md index b98f02ccc98..d224ba51bb3 100644 --- a/doc/administration/logs.md +++ b/doc/administration/logs.md @@ -1,10 +1,9 @@ # Log system -GitLab has an advanced log system where everything is logged so that you +GitLab has an advanced log system where everything is logged, so you can analyze your instance using various system log files. In addition to -system log files, GitLab Enterprise Edition comes with Audit Events. -Find more about them [in Audit Events -documentation](audit_events.md) +system log files, GitLab Enterprise Edition provides Audit Events. +Find more about them [in Audit Events documentation](audit_events.md). System log files are typically plain text in a standard log file format. This guide talks about how to read and use these system log files. @@ -13,21 +12,40 @@ This guide talks about how to read and use these system log files. This file lives in `/var/log/gitlab/gitlab-rails/production_json.log` for Omnibus GitLab packages or in `/home/git/gitlab/log/production_json.log` for -installations from source. (When GitLab is running in an environment -other than production, the corresponding logfile is shown here.) +installations from source. When GitLab is running in an environment +other than production, the corresponding logfile is shown here. It contains a structured log for Rails controller requests received from GitLab, thanks to [Lograge](https://github.com/roidrage/lograge/). Note that requests from the API are logged to a separate file in `api_json.log`. -Each line contains a JSON line that can be ingested by Elasticsearch, Splunk, etc. For example: +Each line contains a JSON line that can be ingested by services like Elasticsearch and Splunk. +Line breaks have been added to this example for legibility: ```json -{"method":"GET","path":"/gitlab/gitlab-foss/issues/1234","format":"html","controller":"Projects::IssuesController","action":"show","status":200,"duration":229.03,"view":174.07,"db":13.24,"time":"2017-08-08T20:15:54.821Z","params":[{"key":"param_key","value":"param_value"}],"remote_ip":"18.245.0.1","user_id":1,"username":"admin","gitaly_calls":76,"gitaly_duration":7.41,"queue_duration": 112.47} +{ + "method":"GET", + "path":"/gitlab/gitlab-foss/issues/1234", + "format":"html", + "controller":"Projects::IssuesController", + "action":"show", + "status":200, + "duration":229.03, + "view":174.07, + "db":13.24, + "time":"2017-08-08T20:15:54.821Z", + "params":[{"key":"param_key","value":"param_value"}], + "remote_ip":"18.245.0.1", + "user_id":1, + "username":"admin", + "gitaly_calls":76, + "gitaly_duration":7.41, + "queue_duration": 112.47 +} ``` -In this example, you can see this was a GET request for a specific -issue. Notice each line also contains performance data. All times are in +This example was a GET request for a specific +issue. Each line also contains performance data, with times in milliseconds: 1. `duration`: total time taken to retrieve the request @@ -37,10 +55,10 @@ milliseconds: 1. `gitaly_calls`: total number of calls made to Gitaly 1. `gitaly_duration`: total time taken by Gitaly calls -User clone/fetch activity using http transport appears in this log as `action: git_upload_pack`. +User clone and fetch activity using HTTP transport appears in this log as `action: git_upload_pack`. -In addition, the log contains the IP address from which the request originated -(`remote_ip`) as well as the user's ID (`user_id`), and username (`username`). +In addition, the log contains the originating IP address, +(`remote_ip`),the user's ID (`user_id`), and username (`username`). NOTE: **Note:** Starting with GitLab 12.5, if an error occurs, an `exception` field is included with `class`, `message`, and @@ -90,11 +108,11 @@ installations from source. (When GitLab is running in an environment other than production, the corresponding logfile is shown here.) It contains information about all performed requests. You can see the -URL and type of request, IP address and what exactly parts of code were -involved to service this particular request. Also you can see all SQL -request that have been performed and how much time it took. This task is +URL and type of request, IP address, and what parts of code were +involved to service this particular request. Also, you can see all SQL +requests performed, and how much time each took. This task is more useful for GitLab contributors and developers. Use part of this log -file when you are going to report bug. For example: +file when you're reporting bugs. For example: ```plaintext Started GET "/gitlabhq/yaml_db/tree/master" for 168.111.56.1 at 2015-02-12 19:34:53 +0200 @@ -114,26 +132,44 @@ Processing by Projects::TreeController#show as HTML Completed 200 OK in 166ms (Views: 117.4ms | ActiveRecord: 27.2ms) ``` -In this example we can see that server processed an HTTP request with URL -`/gitlabhq/yaml_db/tree/master` from IP 168.111.56.1 at 2015-02-12 -19:34:53 +0200. Also we can see that request was processed by -`Projects::TreeController`. +In this example, the server processed an HTTP request with URL +`/gitlabhq/yaml_db/tree/master` from IP `168.111.56.1` at `2015-02-12 19:34:53 +0200`. +The request was processed by `Projects::TreeController`. ## `api_json.log` > Introduced in GitLab 10.0. This file lives in -`/var/log/gitlab/gitlab-rails/api_json.log` for Omnibus GitLab packages or in +`/var/log/gitlab/gitlab-rails/api_json.log` for Omnibus GitLab packages, or in `/home/git/gitlab/log/api_json.log` for installations from source. It helps you see requests made directly to the API. For example: ```json -{"time":"2018-10-29T12:49:42.123Z","severity":"INFO","duration":709.08,"db":14.59,"view":694.49,"status":200,"method":"GET","path":"/api/v4/projects","params":[{"key":"action","value":"git-upload-pack"},{"key":"changes","value":"_any"},{"key":"key_id","value":"secret"},{"key":"secret_token","value":"[FILTERED]"}],"host":"localhost","remote_ip":"::1","ua":"Ruby","route":"/api/:version/projects","user_id":1,"username":"root","queue_duration":100.31,"gitaly_calls":30,"gitaly_duration":5.36} +{ + "time":"2018-10-29T12:49:42.123Z", + "severity":"INFO", + "duration":709.08, + "db":14.59, + "view":694.49, + "status":200, + "method":"GET", + "path":"/api/v4/projects", + "params":[{"key":"action","value":"git-upload-pack"},{"key":"changes","value":"_any"},{"key":"key_id","value":"secret"},{"key":"secret_token","value":"[FILTERED]"}], + "host":"localhost", + "remote_ip":"::1", + "ua":"Ruby", + "route":"/api/:version/projects", + "user_id":1, + "username":"root", + "queue_duration":100.31, + "gitaly_calls":30, + "gitaly_duration":5.36 +} ``` -This entry above shows an access to an internal endpoint to check whether an +This entry shows an access to an internal endpoint to check whether an associated SSH key can download the project in question via a `git fetch` or `git clone`. In this example, we see: @@ -141,7 +177,7 @@ associated SSH key can download the project in question via a `git fetch` or 1. `queue_duration`: total time in milliseconds that the request was queued inside GitLab Workhorse 1. `method`: The HTTP method used to make the request 1. `path`: The relative path of the query -1. `params`: Key-value pairs passed in a query string or HTTP body. Sensitive parameters (e.g. passwords, tokens, etc.) are filtered out. +1. `params`: Key-value pairs passed in a query string or HTTP body. Sensitive parameters (such as passwords and tokens) are filtered out. 1. `ua`: The User-Agent of the requester ## `application.log` @@ -172,8 +208,18 @@ installations from source. It contains the JSON version of the logs in `application.log` like the example below: ``` json -{"severity":"INFO","time":"2020-01-14T13:35:15.466Z","correlation_id":"3823a1550b64417f9c9ed8ee0f48087e","message":"User \"Administrator\" (admin@example.com) was created"} -{"severity":"INFO","time":"2020-01-14T13:35:15.466Z","correlation_id":"78e3df10c9a18745243d524540bd5be4","message":"Project \"project133\" was removed"} +{ + "severity":"INFO", + "time":"2020-01-14T13:35:15.466Z", + "correlation_id":"3823a1550b64417f9c9ed8ee0f48087e", + "message":"User \"Administrator\" (admin@example.com) was created" +} +{ + "severity":"INFO", + "time":"2020-01-14T13:35:15.466Z", + "correlation_id":"78e3df10c9a18745243d524540bd5be4", + "message":"Project \"project133\" was removed" +} ``` ## `integrations_json.log` @@ -182,11 +228,28 @@ This file lives in `/var/log/gitlab/gitlab-rails/integrations_json.log` for Omnibus GitLab packages or in `/home/git/gitlab/log/integrations_json.log` for installations from source. -It contains information about [integrations](../user/project/integrations/project_services.md) activities such as Jira, Asana and Irker services. It uses JSON format like the example below: +It contains information about [integrations](../user/project/integrations/project_services.md) activities such as Jira, Asana, and Irker services. It uses JSON format like the example below: ```json -{"severity":"ERROR","time":"2018-09-06T14:56:20.439Z","service_class":"JiraService","project_id":8,"project_path":"h5bp/html5-boilerplate","message":"Error sending message","client_url":"http://jira.gitlap.com:8080","error":"execution expired"} -{"severity":"INFO","time":"2018-09-06T17:15:16.365Z","service_class":"JiraService","project_id":3,"project_path":"namespace2/project2","message":"Successfully posted","client_url":"http://jira.example.com"} +{ + "severity":"ERROR", + "time":"2018-09-06T14:56:20.439Z", + "service_class":"JiraService", + "project_id":8, + "project_path":"h5bp/html5-boilerplate", + "message":"Error sending message", + "client_url":"http://jira.gitlap.com:8080", + "error":"execution expired" +} +{ + "severity":"INFO", + "time":"2018-09-06T17:15:16.365Z", + "service_class":"JiraService", + "project_id":3, + "project_path":"namespace2/project2", + "message":"Successfully posted", + "client_url":"http://jira.example.com" +} ``` ## `kubernetes.log` @@ -202,12 +265,32 @@ It logs information related to the Kubernetes Integration including errors during installing cluster applications on your GitLab managed Kubernetes clusters. -Each line contains a JSON line that can be ingested by Elasticsearch, Splunk, -etc. For example: +Each line contains a JSON line that can be ingested by services like Elasticsearch and Splunk. +Line breaks have been added to the following example for clarity: ```json -{"severity":"ERROR","time":"2018-11-23T15:14:54.652Z","exception":"Kubeclient::HttpError","error_code":401,"service":"Clusters::Applications::CheckInstallationProgressService","app_id":14,"project_ids":[1],"group_ids":[],"message":"Unauthorized"} -{"severity":"ERROR","time":"2018-11-23T15:42:11.647Z","exception":"Kubeclient::HttpError","error_code":null,"service":"Clusters::Applications::InstallService","app_id":2,"project_ids":[19],"group_ids":[],"message":"SSL_connect returned=1 errno=0 state=error: certificate verify failed (unable to get local issuer certificate)"} +{ + "severity":"ERROR", + "time":"2018-11-23T15:14:54.652Z", + "exception":"Kubeclient::HttpError", + "error_code":401, + "service":"Clusters::Applications::CheckInstallationProgressService", + "app_id":14, + "project_ids":[1], + "group_ids":[], + "message":"Unauthorized" +} +{ + "severity":"ERROR", + "time":"2018-11-23T15:42:11.647Z", + "exception":"Kubeclient::HttpError", + "error_code":null, + "service":"Clusters::Applications::InstallService", + "app_id":2, + "project_ids":[19], + "group_ids":[], + "message":"SSL_connect returned=1 errno=0 state=error: certificate verify failed (unable to get local issuer certificate)" +} ``` ## `git_json.log` @@ -220,8 +303,8 @@ NOTE: **Note:** After 12.2, this file was renamed from `githost.log` to `git_json.log` and stored in JSON format. -GitLab has to interact with Git repositories but in some rare cases -something can go wrong and in this case you will know what exactly +GitLab has to interact with Git repositories, but in some rare cases +something can go wrong, and in this case you will know what exactly happened. This log file contains all failed requests from GitLab to Git repositories. In the majority of cases this file will be useful for developers only. For example: @@ -244,9 +327,20 @@ installations from source. Changes to group or project settings are logged to this file. For example: ```json -{"severity":"INFO","time":"2018-10-17T17:38:22.523Z","author_id":3,"entity_id":2,"entity_type":"Project","change":"visibility","from":"Private","to":"Public","author_name":"John Doe4","target_id":2,"target_type":"Project","target_details":"namespace2/project2"} -{"severity":"INFO","time":"2018-10-17T17:38:22.830Z","author_id":5,"entity_id":3,"entity_type":"Project","change":"name","from":"John Doe7 / project3","to":"John Doe7 / new name","author_name":"John Doe6","target_id":3,"target_type":"Project","target_details":"namespace3/project3"} -{"severity":"INFO","time":"2018-10-17T17:38:23.175Z","author_id":7,"entity_id":4,"entity_type":"Project","change":"path","from":"","to":"namespace4/newpath","author_name":"John Doe8","target_id":4,"target_type":"Project","target_details":"namespace4/newpath"} +{ + "severity":"INFO", + "time":"2018-10-17T17:38:22.523Z", + "author_id":3, + "entity_id":2, + "entity_type":"Project", + "change":"visibility", + "from":"Private", + "to":"Public", + "author_name":"John Doe4", + "target_id":2, + "target_type":"Project", + "target_details":"namespace2/project2" +} ``` ## `sidekiq.log` @@ -268,7 +362,27 @@ Instead of the format above, you can opt to generate JSON logs for Sidekiq. For example: ```json -{"severity":"INFO","time":"2018-04-03T22:57:22.071Z","queue":"cronjob:update_all_mirrors","args":[],"class":"UpdateAllMirrorsWorker","retry":false,"queue_namespace":"cronjob","jid":"06aeaa3b0aadacf9981f368e","created_at":"2018-04-03T22:57:21.930Z","enqueued_at":"2018-04-03T22:57:21.931Z","pid":10077,"message":"UpdateAllMirrorsWorker JID-06aeaa3b0aadacf9981f368e: done: 0.139 sec","job_status":"done","duration":0.139,"completed_at":"2018-04-03T22:57:22.071Z","db_duration":0.05,"db_duration_s":0.0005,"gitaly_duration":0,"gitaly_calls":0} +{ + "severity":"INFO", + "time":"2018-04-03T22:57:22.071Z", + "queue":"cronjob:update_all_mirrors", + "args":[], + "class":"UpdateAllMirrorsWorker", + "retry":false, + "queue_namespace":"cronjob", + "jid":"06aeaa3b0aadacf9981f368e", + "created_at":"2018-04-03T22:57:21.930Z", + "enqueued_at":"2018-04-03T22:57:21.931Z", + "pid":10077, + "message":"UpdateAllMirrorsWorker JID-06aeaa3b0aadacf9981f368e: done: 0.139 sec", + "job_status":"done", + "duration":0.139, + "completed_at":"2018-04-03T22:57:22.071Z", + "db_duration":0.05, + "db_duration_s":0.0005, + "gitaly_duration":0, + "gitaly_calls":0 +} ``` For Omnibus GitLab installations, add the configuration option: @@ -292,8 +406,8 @@ This file lives in `/var/log/gitlab/gitaly/gitlab-shell.log` for Omnibus GitLab packages or in `/home/git/gitaly/gitlab-shell.log` for installations from source. -NOTE: **Note** -For GitLab 12.5 and earlier the file lives in `/var/log/gitlab/gitlab-shell/gitlab-shell.log`. +NOTE: **Note:** +For GitLab 12.5 and earlier, the file lives in `/var/log/gitlab/gitlab-shell/gitlab-shell.log`. GitLab Shell is used by GitLab for executing Git commands and provide SSH access to Git repositories. For example: @@ -363,7 +477,7 @@ This log records: - [Protected paths] abusive requests. NOTE: **Note:** -From [%12.1](https://gitlab.com/gitlab-org/gitlab-foss/issues/62756), user id and username are also +From [%12.1](https://gitlab.com/gitlab-org/gitlab-foss/issues/62756), user ID and username are also recorded on this log, if available. ## `graphql_json.log` @@ -411,13 +525,13 @@ was initiated, such as `1509705644.log` ## `sidekiq_exporter.log` and `web_exporter.log` If Prometheus metrics and the Sidekiq Exporter are both enabled, Sidekiq will -start a Web server and listen to the defined port (default: 8082). Access logs +start a Web server and listen to the defined port (default: `8082`). Access logs will be generated in `/var/log/gitlab/gitlab-rails/sidekiq_exporter.log` for Omnibus GitLab packages or in `/home/git/gitlab/log/sidekiq_exporter.log` for installations from source. If Prometheus metrics and the Web Exporter are both enabled, Unicorn/Puma will -start a Web server and listen to the defined port (default: 8083). Access logs +start a Web server and listen to the defined port (default: `8083`). Access logs will be generated in `/var/log/gitlab/gitlab-rails/web_exporter.log` for Omnibus GitLab packages or in `/home/git/gitlab/log/web_exporter.log` for installations from source. @@ -427,7 +541,7 @@ installations from source. > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/15442) in GitLab 12.3. Contains details of GitLab's [Database Load Balancing](database_load_balancing.md). -It is stored at: +It's stored at: - `/var/log/gitlab/gitlab-rails/database_load_balancing.log` for Omnibus GitLab packages. - `/home/git/gitlab/log/database_load_balancing.log` for installations from source. @@ -444,11 +558,21 @@ from source. It logs information related to the Elasticsearch Integration including errors during indexing or searching Elasticsearch. -Each line contains a JSON line that can be ingested by Elasticsearch, Splunk, -etc. For example: +Each line contains a JSON line that can be ingested by services like Elasticsearch and Splunk. +Line breaks have been added to the following example line for clarity: ```json -{"severity":"DEBUG","time":"2019-10-17T06:23:13.227Z","correlation_id":null,"message":"redacted_search_result","class_name":"Milestone","id":2,"ability":"read_milestone","current_user_id":2,"query":"project"} +{ + "severity":"DEBUG", + "time":"2019-10-17T06:23:13.227Z", + "correlation_id":null, + "message":"redacted_search_result", + "class_name":"Milestone", + "id":2, + "ability":"read_milestone", + "current_user_id":2, + "query":"project" +} ``` ## `exceptions_json.log` @@ -460,7 +584,7 @@ This file lives in packages or in `/home/git/gitlab/log/exceptions_json.log` for installations from source. -It logs the information about exceptions being tracked by `Gitlab::ErrorTracking` which provides standard and consistent way of [processing rescued exceptions](https://gitlab.com/gitlab-org/gitlab/blob/master/doc/development/logging.md#exception-handling). +It logs the information about exceptions being tracked by `Gitlab::ErrorTracking` which provides a standard and consistent way of [processing rescued exceptions](https://gitlab.com/gitlab-org/gitlab/blob/master/doc/development/logging.md#exception-handling). Each line contains a JSON line that can be ingested by Elasticsearch. For example: diff --git a/doc/administration/monitoring/prometheus/index.md b/doc/administration/monitoring/prometheus/index.md index 03ea7656d83..87114c154a2 100644 --- a/doc/administration/monitoring/prometheus/index.md +++ b/doc/administration/monitoring/prometheus/index.md @@ -7,7 +7,7 @@ > they got added. For installations from source you will have to install them > yourself. Over subsequent releases additional GitLab metrics will be captured. > - Prometheus services are on by default with GitLab 9.0. -> - Prometheus and its exporters do not authenticate users, and will be available +> - Prometheus and its exporters don't authenticate users, and will be available > to anyone who can access them. [Prometheus] is a powerful time-series monitoring service, providing a flexible @@ -18,7 +18,7 @@ access to high quality time-series monitoring of GitLab services. ## Overview Prometheus works by periodically connecting to data sources and collecting their -performance metrics via the [various exporters](#bundled-software-metrics). To view +performance metrics through the [various exporters](#bundled-software-metrics). To view and work with the monitoring data, you can either [connect directly to Prometheus](#viewing-performance-metrics) or utilize a dashboard tool like [Grafana](https://grafana.com). @@ -26,11 +26,11 @@ dashboard tool like [Grafana](https://grafana.com). ## Configuring Prometheus NOTE: **Note:** -For installations from source you'll have to install and configure it yourself. +For installations from source, you'll have to install and configure it yourself. Prometheus and its exporters are on by default, starting with GitLab 9.0. Prometheus will run as the `gitlab-prometheus` user and listen on -`http://localhost:9090`. By default Prometheus is only accessible from the GitLab server itself. +`http://localhost:9090`. By default, Prometheus is only accessible from the GitLab server itself. Each exporter will be automatically set up as a monitoring target for Prometheus, unless individually disabled. @@ -51,7 +51,7 @@ To disable Prometheus and all of its exporters, as well as any added in the futu NOTE: **Note:** The following change was added in [GitLab Omnibus 8.17][1261]. Although possible, it's not recommended to change the port Prometheus listens -on as this might affect or conflict with other services running on the GitLab +on, as this might affect or conflict with other services running on the GitLab server. Proceed at your own risk. In order to access Prometheus from outside the GitLab server you will need to @@ -65,7 +65,7 @@ To change the address/port that Prometheus listens on: prometheus['listen_address'] = 'localhost:9090' ``` - Replace `localhost:9090` with the address/port you want Prometheus to + Replace `localhost:9090` with the address or port you want Prometheus to listen on. If you would like to allow access to Prometheus to hosts other than `localhost`, leave out the host, or use `0.0.0.0` to allow public access: @@ -106,7 +106,7 @@ prometheus['scrape_configs'] = [ ### Using an external Prometheus server NOTE: **Note:** -Prometheus and most exporters do not support authentication. We do not recommend exposing them outside the local network. +Prometheus and most exporters don't support authentication. We don't recommend exposing them outside the local network. A few configuration changes are required to allow GitLab to be monitored by an external Prometheus server. External servers are recommended for highly available deployments of GitLab with multiple nodes. @@ -151,7 +151,7 @@ To use an external Prometheus server: } ``` -1. [Reconfigure GitLab][reconfigure] to apply the changes +1. [Reconfigure GitLab][reconfigure] to apply the changes. 1. Edit the Prometheus server's configuration file. 1. Add each node's exporters to the Prometheus server's [scrape target configuration](https://prometheus.io/docs/prometheus/latest/configuration/configuration/#%3Cscrape_config%3E). @@ -223,9 +223,9 @@ some workarounds: using a separate FQDN, using server IP, using a separate brows having [NGINX proxy it][nginx-custom-config]. The performance data collected by Prometheus can be viewed directly in the -Prometheus console or through a compatible dashboard tool. +Prometheus console, or through a compatible dashboard tool. The Prometheus interface provides a [flexible query language](https://prometheus.io/docs/prometheus/latest/querying/basics/) -to work with the collected data where you can visualize their output. +to work with the collected data where you can visualize the output. For a more fully featured dashboard, Grafana can be used and has [official support for Prometheus][prom-grafana]. @@ -238,23 +238,23 @@ Sample Prometheus queries: ## Prometheus as a Grafana data source -Grafana allows you to import Prometheus performance metrics as a data source -and render the metrics as graphs and dashboards which is helpful with visualisation. +Grafana allows you to import Prometheus performance metrics as a data source, +and render the metrics as graphs and dashboards, which is helpful with visualization. To add a Prometheus dashboard for a single server GitLab setup: 1. Create a new data source in Grafana. -1. Name your data source i.e GitLab. +1. Name your data source (such as GitLab). 1. Select `Prometheus` in the type dropdown box. -1. Add your Prometheus listen address as the URL and set access to `Browser`. +1. Add your Prometheus listen address as the URL, and set access to `Browser`. 1. Set the HTTP method to `GET`. -1. Save & Test your configuration to verify that it works. +1. Save and test your configuration to verify that it works. ## GitLab metrics > Introduced in GitLab 9.3. -GitLab monitors its own internal service metrics, and makes them available at the `/-/metrics` endpoint. Unlike other exporters, this endpoint requires authentication as it is available on the same URL and port as user traffic. +GitLab monitors its own internal service metrics, and makes them available at the `/-/metrics` endpoint. Unlike other exporters, this endpoint requires authentication as it's available on the same URL and port as user traffic. [➔ Read more about the GitLab Metrics.](gitlab_metrics.md) @@ -265,8 +265,8 @@ export Prometheus metrics. ### Node exporter -The node exporter allows you to measure various machine resources such as -memory, disk and CPU utilization. +The node exporter allows you to measure various machine resources, such as +memory, disk, and CPU utilization. [➔ Read more about the node exporter.](node_exporter.md) @@ -310,7 +310,7 @@ If your GitLab server is running within Kubernetes, Prometheus will collect metr To disable the monitoring of Kubernetes: 1. Edit `/etc/gitlab/gitlab.rb`. -1. Add or find and uncomment the following line and set it to `false`: +1. Add (or find and uncomment) the following line and set it to `false`: ```ruby prometheus['monitor_kubernetes'] = false diff --git a/doc/api/merge_trains.md b/doc/api/merge_trains.md new file mode 100644 index 00000000000..7fff09474df --- /dev/null +++ b/doc/api/merge_trains.md @@ -0,0 +1,80 @@ +# Merge Trains API **(PREMIUM)** + +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/issues/36146) in GitLab 12.9. +> - Using this API you can consume GitLab's [Merge Train](../ci/merge_request_pipelines/pipelines_for_merged_results/merge_trains/index.md) entries. + +Every API call to merge trains must be authenticated with Developer or higher [permissions](link-to-permissions-doc). + +If a user is not a member of a project and the project is private, a `GET` request on that project will result to a `404` status code. + +If Merge Trains is not available for the project, a `403` status code will return. + +## Merge Trains API pagination + +By default, `GET` requests return 20 results at a time because the API results +are paginated. + +Read more on [pagination](README.md#pagination). + +## List Merge Trains for a project + +Get all Merge Trains of the requested project: + +```txt +GET /projects/:id/merge_trains +GET /projects/:id/merge_trains?scope=complete +``` + +| Attribute | Type | Required | Description | +| ------------------- | ---------------- | ---------- | --------------------------------------------------------------------------------------------------------------------------- | +| `id` | integer/string | yes | The ID or [URL-encoded path of the project](README.md#namespaced-path-encoding). | +| `scope` | string | no | Return Merge Trains filtered by the given scope. Available scopes are `active` (to be merged) and `complete` (have been merged). | +| `sort` | string | no | Return Merge Trains sorted in `asc` or `desc` order. Default is `desc`. | + +```shell +curl --header "PRIVATE-TOKEN: <your_access_token>" https://gitlab.example.com/api/v4/projects/1/merge_trains +``` + +Example response: + +```json +[ + { + "id": 110, + "merge_request": { + "id": 126, + "iid": 59, + "project_id": 20, + "title": "Test MR 1580978354", + "description": "", + "state": "merged", + "created_at": "2020-02-06T08:39:14.883Z", + "updated_at": "2020-02-06T08:40:57.038Z", + "web_url": "http://local.gitlab.test:8181/root/merge-train-race-condition/-/merge_requests/59" + }, + "user": { + "id": 1, + "name": "Administrator", + "username": "root", + "state": "active", + "avatar_url": "https://www.gravatar.com/avatar/e64c7d89f26bd1972efa854d13d7dd61?s=80&d=identicon", + "web_url": "http://local.gitlab.test:8181/root" + }, + "pipeline": { + "id": 246, + "sha": "bcc17a8ffd51be1afe45605e714085df28b80b13", + "ref": "refs/merge-requests/59/train", + "status": "success", + "created_at": "2020-02-06T08:40:42.410Z", + "updated_at": "2020-02-06T08:40:46.912Z", + "web_url": "http://local.gitlab.test:8181/root/merge-train-race-condition/pipelines/246" + }, + "created_at": "2020-02-06T08:39:47.217Z", + "updated_at": "2020-02-06T08:40:57.720Z", + "target_branch": "feature-1580973432", + "status": "merged", + "merged_at": "2020-02-06T08:40:57.719Z", + "duration": 70 + } +] +``` diff --git a/doc/development/adding_database_indexes.md b/doc/development/adding_database_indexes.md index 3ddb15fa290..01b621b6631 100644 --- a/doc/development/adding_database_indexes.md +++ b/doc/development/adding_database_indexes.md @@ -73,7 +73,7 @@ especially the case for small tables. If a table is expected to grow in size and you expect your query has to filter out a lot of rows you may want to consider adding an index. If the table size is -very small (e.g. only a handful of rows) or any existing indexes filter out +very small (e.g. less than `1,000` records) or any existing indexes filter out enough rows you may _not_ want to add a new index. ## Maintenance Overhead diff --git a/doc/development/migration_style_guide.md b/doc/development/migration_style_guide.md index 2c9ad8c00cf..c15e51c2b9d 100644 --- a/doc/development/migration_style_guide.md +++ b/doc/development/migration_style_guide.md @@ -304,10 +304,16 @@ combining it with other operations that don't require `disable_ddl_transaction!` ## Adding indexes -If you need to add a unique index, please keep in mind there is the possibility -of existing duplicates being present in the database. This means that should -always _first_ add a migration that removes any duplicates, before adding the -unique index. +Before adding an index, consider if this one is necessary. There are situations in which an index +might not be required, like: + +- The table is small (less than `1,000` records) and it's not expected to exponentially grow in size. +- Any existing indexes filter out enough rows. +- The reduction in query timings after the index is added is not significant. + +Additionally, wide indexes are not required to match all filter criteria of queries, we just need +to cover enough columns so that the index lookup has a small enough selectivity. Please review our +[Adding Database indexes](adding_database_indexes.md) guide for more details. When adding an index to a non-empty table make sure to use the method `add_concurrent_index` instead of the regular `add_index` method. @@ -334,6 +340,11 @@ class MyMigration < ActiveRecord::Migration[4.2] end ``` +If you need to add a unique index, please keep in mind there is the possibility +of existing duplicates being present in the database. This means that should +always _first_ add a migration that removes any duplicates, before adding the +unique index. + For a small table (such as an empty one or one with less than `1,000` records), it is recommended to use `add_index` in a single-transaction migration, combining it with other operations that don't require `disable_ddl_transaction!`. diff --git a/doc/user/project/clusters/kubernetes_pod_logs.md b/doc/user/project/clusters/kubernetes_pod_logs.md index a36b712ae76..76622380e92 100644 --- a/doc/user/project/clusters/kubernetes_pod_logs.md +++ b/doc/user/project/clusters/kubernetes_pod_logs.md @@ -29,7 +29,7 @@ You can access them in two ways. > [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/22011) in GitLab 12.5. -Go to **Operations > Pod logs** on the sidebar menu. +Go to **{cloud-gear}** **Operations > Pod logs** on the sidebar menu. ![Sidebar menu](img/sidebar_menu_pod_logs_v12_5.png) @@ -37,7 +37,7 @@ Go to **Operations > Pod logs** on the sidebar menu. Logs can be displayed by clicking on a specific pod from [Deploy Boards](../deploy_boards.md): -1. Go to **Operations > Environments** and find the environment which contains the desired pod, like `production`. +1. Go to **{cloud-gear}** **Operations > Environments** and find the environment which contains the desired pod, like `production`. 1. On the **Environments** page, you should see the status of the environment's pods with [Deploy Boards](../deploy_boards.md). 1. When mousing over the list of pods, a tooltip will appear with the exact pod name and status. ![Deploy Boards pod list](img/pod_logs_deploy_board.png) @@ -45,7 +45,7 @@ Logs can be displayed by clicking on a specific pod from [Deploy Boards](../depl ### Logs view -The logs view will contain the last 500 lines for a pod, and has control to filter via: +The logs view will contain the last 500 lines for a pod, and has control to filter through: - Pods. - [From GitLab 12.4](https://gitlab.com/gitlab-org/gitlab/issues/5769), environments. @@ -62,14 +62,14 @@ Support for historical data is coming [in a future release](https://gitlab.com/g When you enable [Elastic Stack](../../clusters/applications.md#elastic-stack) on your cluster, you can filter by date. -Click on "Show last" to see the available options. +Click on **Show last** to see the available options. ### Full text search > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/21656) in GitLab 12.7. When you enable [Elastic Stack](../../clusters/applications.md#elastic-stack) on your cluster, -you can search the content of your logs via a search bar. +you can search the content of your logs through a search bar. The search is passed on to Elasticsearch using the [simple_query_string](https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-simple-query-string-query.html) Elasticsearch function, which supports the following operators: |