diff options
| author | GitLab Bot <gitlab-bot@gitlab.com> | 2020-07-20 15:26:25 +0300 |
|---|---|---|
| committer | GitLab Bot <gitlab-bot@gitlab.com> | 2020-07-20 15:26:25 +0300 |
| commit | a09983ae35713f5a2bbb100981116d31ce99826e (patch) | |
| tree | 2ee2af7bd104d57086db360a7e6d8c9d5d43667a /doc/administration/monitoring | |
| parent | 18c5ab32b738c0b6ecb4d0df3994000482f34bd8 (diff) | |
Add latest changes from gitlab-org/gitlab@13-2-stable-ee
Diffstat (limited to 'doc/administration/monitoring')
14 files changed, 231 insertions, 178 deletions
diff --git a/doc/administration/monitoring/gitlab_self_monitoring_project/img/self_monitoring_default_dashboard.png b/doc/administration/monitoring/gitlab_self_monitoring_project/img/self_monitoring_default_dashboard.png Binary files differnew file mode 100644 index 00000000000..1d61823ce41 --- /dev/null +++ b/doc/administration/monitoring/gitlab_self_monitoring_project/img/self_monitoring_default_dashboard.png diff --git a/doc/administration/monitoring/gitlab_self_monitoring_project/index.md b/doc/administration/monitoring/gitlab_self_monitoring_project/index.md index d05fb803c5c..44ba26296b9 100644 --- a/doc/administration/monitoring/gitlab_self_monitoring_project/index.md +++ b/doc/administration/monitoring/gitlab_self_monitoring_project/index.md @@ -26,7 +26,7 @@ of the project shows some basic resource usage charts, such as CPU and memory us of each server in [Omnibus GitLab](https://docs.gitlab.com/omnibus/) installations. You can also use the project to configure your own -[custom metrics](../../../user/project/integrations/prometheus.md#adding-custom-metrics) using +[custom metrics](../../../operations/metrics/index.md#adding-custom-metrics) using metrics exposed by the [GitLab exporter](../prometheus/gitlab_metrics.md#metrics-available). ## Creating the self monitoring project @@ -47,6 +47,19 @@ project. If you create the project again, it will be created in its default stat It can take a few seconds for it to be deleted. 1. After the project is deleted, GitLab displays a message confirming your action. +## Dashboards available in Omnibus GitLab + +Omnibus GitLab provides a dashboard that displays CPU and memory usage +of each GitLab server. To select the servers to be displayed in the +panels, provide a regular expression in the **Instance label regex** field. +The dashboard uses metrics available in +[Omnibus GitLab](https://docs.gitlab.com/omnibus/) installations. + + + +You can also +[create your own dashboards](../../../operations/metrics/dashboards/index.md#defining-custom-dashboards-per-project). + ## Connection to Prometheus The project will be automatically configured to connect to the @@ -60,18 +73,18 @@ you should ## Taking action on Prometheus alerts **(ULTIMATE)** -You can [add a webhook](../../../user/project/integrations/prometheus.md#external-prometheus-instances) +You can [add a webhook](../../../operations/metrics/alerts.md#external-prometheus-instances) to the Prometheus configuration in order for GitLab to receive notifications of any alerts. Once the webhook is setup, you can -[take action on incoming alerts](../../../user/project/integrations/prometheus.md#taking-action-on-incidents-ultimate). +[take action on incoming alerts](../../../operations/metrics/alerts.md#trigger-actions-from-alerts-ultimate). ## Adding custom metrics to the self monitoring project You can add custom metrics in the self monitoring project by: -1. [Duplicating](../../../user/project/integrations/prometheus.md#duplicating-a-gitlab-defined-dashboard) the default dashboard. -1. [Editing](../../../user/project/integrations/prometheus.md#view-and-edit-the-source-file-of-a-custom-dashboard) the newly created dashboard file and configuring it with [dashboard YAML properties](../../../user/project/integrations/prometheus.md#dashboard-yaml-properties). +1. [Duplicating](../../../operations/metrics/dashboards/index.md#duplicating-a-gitlab-defined-dashboard) the default dashboard. +1. [Editing](../../../operations/metrics/dashboards/index.md#view-and-edit-the-source-file-of-a-custom-dashboard) the newly created dashboard file and configuring it with [dashboard YAML properties](../../../operations/metrics/dashboards/yaml.md). ## Troubleshooting diff --git a/doc/administration/monitoring/performance/grafana_configuration.md b/doc/administration/monitoring/performance/grafana_configuration.md index 4dbe3aed84e..96f1377fb73 100644 --- a/doc/administration/monitoring/performance/grafana_configuration.md +++ b/doc/administration/monitoring/performance/grafana_configuration.md @@ -6,85 +6,90 @@ info: To determine the technical writer assigned to the Stage/Group associated w # Grafana Configuration -[Grafana](https://grafana.com/) is a tool that allows you to visualize time -series metrics through graphs and dashboards. GitLab writes performance data to Prometheus -and Grafana will allow you to query to display useful graphs. +[Grafana](https://grafana.com/) is a tool that enables you to visualize time +series metrics through graphs and dashboards. GitLab writes performance data to Prometheus, +and Grafana allows you to query the data to display useful graphs. ## Installation -[Omnibus GitLab can help you install Grafana (recommended)](https://docs.gitlab.com/omnibus/settings/grafana.html) +Omnibus GitLab can [help you install Grafana (recommended)](https://docs.gitlab.com/omnibus/settings/grafana.html) or Grafana supplies package repositories (Yum/Apt) for easy installation. See [Grafana installation documentation](https://grafana.com/docs/grafana/latest/installation/) for detailed steps. NOTE: **Note:** Before starting Grafana for the first time, set the admin user -and password in `/etc/grafana/grafana.ini`. Otherwise, the default password -will be `admin`. +and password in `/etc/grafana/grafana.ini`. If you don't, the default password +is `admin`. ## Configuration -Login as the admin user. Expand the menu by clicking the Grafana logo in the -top left corner. Choose 'Data Sources' from the menu. Then, click 'Add new' -in the top bar. - - - - +1. Log in to Grafana as the admin user. +1. Expand the menu by clicking the Grafana logo in the top left corner. +1. Choose **Data Sources** from the menu. +1. Click **Add new** in the top bar: +  +1. Edit the data source to fit your needs: +  +1. Click **Save**. ## Import Dashboards -You can now import a set of default dashboards that will give you a good -start on displaying useful information. GitLab has published a set of default -[Grafana dashboards](https://gitlab.com/gitlab-org/grafana-dashboards) to get you started. Clone the -repository or download a zip/tarball, then follow these steps to import each -JSON file. - -Open the dashboard dropdown menu and click 'Import' - - - -Click 'Choose file' and browse to the location where you downloaded or cloned -the dashboard repository. Pick one of the JSON files to import. - - - -Once the dashboard is imported, be sure to click save icon in the top bar. If -you do not save the dashboard after importing it will be removed when you -navigate away. - - +You can now import a set of default dashboards to start displaying useful information. +GitLab has published a set of default +[Grafana dashboards](https://gitlab.com/gitlab-org/grafana-dashboards) to get you started. +Clone the repository, or download a ZIP file or tarball, then follow these steps to import each +JSON file individually: + +1. Log in to Grafana as the admin user. +1. Open the dashboard dropdown menu and click **Import**: +  +1. Click **Choose file**, and browse to the location where you downloaded or + cloned the dashboard repository. Select a JSON file to import: +  +1. After the dashboard is imported, click the **Save dashboard** icon in the top bar: +  + + NOTE: **Note:** + If you don't save the dashboard after importing it, the dashboard is removed + when you navigate away from the page. Repeat this process for each dashboard you wish to import. -Alternatively you can automatically import all the dashboards into your Grafana -instance. See the README of the [Grafana dashboards](https://gitlab.com/gitlab-org/grafana-dashboards) -repository for more information on this process. +Alternatively, you can import all the dashboards into your Grafana +instance. For more information about this process, see the +[README of the Grafana dashboards](https://gitlab.com/gitlab-org/grafana-dashboards) +repository. ## Integration with GitLab UI > [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/61005) in GitLab 12.1. -If you have set up Grafana, you can enable a link to access it easily from the sidebar: +After setting up Grafana, you can enable a link to access it easily from the +GitLab sidebar: -1. Go to the **Admin Area > Settings > Metrics and profiling**. +1. Navigate to the **{admin}** **Admin Area > Settings > Metrics and profiling**. 1. Expand **Metrics - Grafana**. -1. Check the "Enable access to Grafana" checkbox. -1. If Grafana is enabled through Omnibus GitLab and on the same server, - leave **Grafana URL** unchanged. It should be `/-/grafana`. - - In any other case, enter the full URL of the Grafana instance. +1. Check the **Enable access to Grafana** checkbox. +1. Configure the **Grafana URL**: + - *If Grafana is enabled through Omnibus GitLab and on the same server,* + leave **Grafana URL** unchanged. It should be `/-/grafana`. + - *Otherwise,* enter the full URL of the Grafana instance. 1. Click **Save changes**. -1. The new link will be available in the **Admin Area > Monitoring > Metrics Dashboard**. + +GitLab displays your link in the **{admin}** **Admin Area > Monitoring > Metrics Dashboard**. ## Security Update -Users running GitLab version 12.0 or later should immediately upgrade to one of the following security releases due to a known vulnerability with the embedded Grafana dashboard: +Users running GitLab version 12.0 or later should immediately upgrade to one of the +following security releases due to a known vulnerability with the embedded Grafana dashboard: - 12.0.6 - 12.1.6 -After upgrading, the Grafana dashboard will be disabled and the location of your existing Grafana data will be changed from `/var/opt/gitlab/grafana/data/` to `/var/opt/gitlab/grafana/data.bak.#{Date.today}/`. +After upgrading, the Grafana dashboard is disabled, and the location of your +existing Grafana data is changed from `/var/opt/gitlab/grafana/data/` to +`/var/opt/gitlab/grafana/data.bak.#{Date.today}/`. To prevent the data from being relocated, you can run the following command prior to upgrading: @@ -100,19 +105,23 @@ sudo mv /var/opt/gitlab/grafana/data.bak.xxxx/ /var/opt/gitlab/grafana/data/ However, you should **not** reinstate your old data _except_ under one of the following conditions: -1. If you are certain that you changed your default admin password when you enabled Grafana -1. If you run GitLab in a private network, accessed only by trusted users, and your Grafana login page has not been exposed to the internet +1. If you're certain that you changed your default admin password when you enabled Grafana. +1. If you run GitLab in a private network, accessed only by trusted users, and your + Grafana login page has not been exposed to the internet. -If you require access to your old Grafana data but do not meet one of these criteria, you may consider: +If you require access to your old Grafana data but don't meet one of these criteria, you may consider: 1. Reinstating it temporarily. 1. [Exporting the dashboards](https://grafana.com/docs/grafana/latest/reference/export_import/#exporting-a-dashboard) you need. 1. Refreshing the data and [re-importing your dashboards](https://grafana.com/docs/grafana/latest/reference/export_import/#importing-a-dashboard). DANGER: **Danger:** -This poses a temporary vulnerability while your old Grafana data is in use and the decision to do so should be weighed carefully with your need to access existing data and dashboards. +These actions pose a temporary vulnerability while your old Grafana data is in use. +Deciding to take any of these actions should be weighed carefully with your need to access +existing data and dashboards. -For more information and further mitigation details, please refer to our [blog post on the security release](https://about.gitlab.com/releases/2019/08/12/critical-security-release-gitlab-12-dot-1-dot-6-released/). +For more information and further mitigation details, please refer to our +[blog post on the security release](https://about.gitlab.com/releases/2019/08/12/critical-security-release-gitlab-12-dot-1-dot-6-released/). --- diff --git a/doc/administration/monitoring/performance/img/performance_bar_configuration_settings.png b/doc/administration/monitoring/performance/img/performance_bar_configuration_settings.png Binary files differdeleted file mode 100644 index fafc50cd000..00000000000 --- a/doc/administration/monitoring/performance/img/performance_bar_configuration_settings.png +++ /dev/null diff --git a/doc/administration/monitoring/performance/img/performance_bar_frontend.png b/doc/administration/monitoring/performance/img/performance_bar_frontend.png Binary files differindex 32a241e032b..9cc874c883a 100644 --- a/doc/administration/monitoring/performance/img/performance_bar_frontend.png +++ b/doc/administration/monitoring/performance/img/performance_bar_frontend.png diff --git a/doc/administration/monitoring/performance/img/performance_bar_gitaly_calls.png b/doc/administration/monitoring/performance/img/performance_bar_gitaly_calls.png Binary files differindex 2c43201cbd0..6caa2869d9b 100644 --- a/doc/administration/monitoring/performance/img/performance_bar_gitaly_calls.png +++ b/doc/administration/monitoring/performance/img/performance_bar_gitaly_calls.png diff --git a/doc/administration/monitoring/performance/img/performance_bar_gitaly_threshold.png b/doc/administration/monitoring/performance/img/performance_bar_gitaly_threshold.png Binary files differindex 4e42d904cdf..386558da813 100644 --- a/doc/administration/monitoring/performance/img/performance_bar_gitaly_threshold.png +++ b/doc/administration/monitoring/performance/img/performance_bar_gitaly_threshold.png diff --git a/doc/administration/monitoring/performance/img/performance_bar_redis_calls.png b/doc/administration/monitoring/performance/img/performance_bar_redis_calls.png Binary files differindex ecb2dffbf8d..f2df8c794db 100644 --- a/doc/administration/monitoring/performance/img/performance_bar_redis_calls.png +++ b/doc/administration/monitoring/performance/img/performance_bar_redis_calls.png diff --git a/doc/administration/monitoring/performance/img/performance_bar_request_selector_warning.png b/doc/administration/monitoring/performance/img/performance_bar_request_selector_warning.png Binary files differindex 74711387ffe..3872c8b41b2 100644 --- a/doc/administration/monitoring/performance/img/performance_bar_request_selector_warning.png +++ b/doc/administration/monitoring/performance/img/performance_bar_request_selector_warning.png diff --git a/doc/administration/monitoring/performance/img/performance_bar_rugged_calls.png b/doc/administration/monitoring/performance/img/performance_bar_rugged_calls.png Binary files differindex 210f80a713d..0340ca1b2f7 100644 --- a/doc/administration/monitoring/performance/img/performance_bar_rugged_calls.png +++ b/doc/administration/monitoring/performance/img/performance_bar_rugged_calls.png diff --git a/doc/administration/monitoring/performance/performance_bar.md b/doc/administration/monitoring/performance/performance_bar.md index c681dedbca8..8002a9ab296 100644 --- a/doc/administration/monitoring/performance/performance_bar.md +++ b/doc/administration/monitoring/performance/performance_bar.md @@ -6,71 +6,83 @@ info: To determine the technical writer assigned to the Stage/Group associated w # Performance Bar -A Performance Bar can be displayed, to dig into the performance of a page. When -activated, it looks as follows: +You can display the GitLab Performance Bar to see statistics for the performance +of a page. When activated, it looks as follows:  -It allows you to see (from left to right): +From left to right, it displays: -- the current host serving the page -- time taken and number of DB queries; click through for details of these queries +- **Current Host**: the current host serving the page. +- **Database queries**: the time taken (in milliseconds) and the total number + of database queries, displayed in the format `00ms / 00pg`. Click to display + a modal window with more details:  -- time taken and number of [Gitaly](../../gitaly/index.md) calls; click through for details of these calls +- **Gitaly calls**: the time taken (in milliseconds) and the total number of + [Gitaly](../../gitaly/index.md) calls. Click to display a modal window with more + details:  -- time taken and number of [Rugged](../../high_availability/nfs.md#improving-nfs-performance-with-gitlab) calls; click through for details of these calls +- **Rugged calls**: the time taken (in milliseconds) and the total number of + [Rugged](../../high_availability/nfs.md#improving-nfs-performance-with-gitlab) calls. + Click to display a modal window with more details:  -- time taken and number of Redis calls; click through for details of these calls +- **Redis calls**: the time taken (in milliseconds) and the total number of + Redis calls. Click to display a modal window with more details:  -- total load timings of the page; click through for details of these calls. Values in the following order: - - Backend - Time that the actual base page took to load - - [First Contentful Paint](hhttps://web.dev/first-contentful-paint/) - Time until something was visible to the user - - [DomContentLoaded](https://developers.google.com/web/fundamentals/performance/critical-rendering-path/measure-crp) Event - - Number of Requests that the page loaded -  -- a link to add a request's details to the performance bar; the request can be - added by its full URL (authenticated as the current user), or by the value of - its `X-Request-Id` header -- a link to download the raw JSON used to generate the Performance Bar reports - -On the far right is a request selector that allows you to view the same metrics -(excluding the page timing and line profiler) for any requests made while the -page was open. Only the first two requests per unique URL are captured. +- **Elasticsearch calls**: the time taken (in milliseconds) and the total number of + Elasticsearch calls. Click to display a modal window with more details. +- **Load timings** of the page: if your browser supports load timings (Chromium + and Chrome) several values in milliseconds, separated by slashes. + Click to display a modal window with more details. The values, from left to right: + - **Backend**: time needed for the base page to load. + - [**First Contentful Paint**](https://web.dev/first-contentful-paint/): + Time until something was visible to the user. + - [**DomContentLoaded**](https://developers.google.com/web/fundamentals/performance/critical-rendering-path/measure-crp) Event. + - **Total number of requests** the page loaded: +  +- **Trace**: If Jaeger is integrated, **Trace** links to a Jaeger tracing page + with the current request's `correlation_id` included. +- **+**: A link to add a request's details to the performance bar. The request + can be added by its full URL (authenticated as the current user), or by the value of + its `X-Request-Id` header. +- **Download**: a link to download the raw JSON used to generate the Performance Bar reports. +- **Request Selector**: a select box displayed on the right-hand side of the + Performance Bar which enables you to view these metrics for any requests made while + the current page was open. Only the first two requests per unique URL are captured. ## Request warnings -For requests exceeding predefined limits, a warning icon will be shown -next to the failing metric, along with an explanation. In this example, -the Gitaly call duration exceeded the threshold: +Requests exceeding predefined limits display a warning **{warning}** icon and +explanation next to the failing metric. In this example, the Gitaly call duration +exceeded the threshold:  -If any requests on the current page generated warnings, the icon will -appear next to the request selector: +If any requests on the current page generated warnings, the warning icon displays +next to the **Request selector**:  -And requests with warnings are indicated in the request selector with a -`(!)` after their path: +Requests with warnings display `(!)` after their path in the **Request selector**:  ## Enable the Performance Bar via the Admin panel -GitLab Performance Bar is disabled by default. To enable it for a given group, -navigate to **Admin Area > Settings > Metrics and profiling** -(`admin/application_settings/metrics_and_profiling`), and expand the section -**Profiling - Performance bar**. +The GitLab Performance Bar is disabled by default. To enable it for a given group: -The only required setting you need to set is the full path of the group that -will be allowed to display the Performance Bar. -Make sure _Enable the Performance Bar_ is checked and hit -**Save** to save the changes. +1. Sign in as a user with Administrator [permissions](../../../user/permissions.md). +1. In the menu bar, click the **{admin}** **Admin Area** icon. +1. Navigate to **{settings}** **Settings > Metrics and profiling** + (`admin/application_settings/metrics_and_profiling`), and expand the section + **Profiling - Performance bar**. +1. Click **Enable access to the Performance Bar**. +1. In the **Allowed group** field, provide the full path of the group allowed + to access the GitLab Performance Bar. +1. Click **Save changes**. -Once the Performance Bar is enabled, you will need to press the [<kbd>p</kbd> + -<kbd>b</kbd> keyboard shortcut](../../../user/shortcuts.md) to actually -display it. +## Keyboard shortcut for the Performance Bar -You can toggle the Bar using the same shortcut. - - +After enabling the GitLab Performance Bar, press the [<kbd>p</kbd> + +<kbd>b</kbd> keyboard shortcut](../../../user/shortcuts.md) to display it, and +again to hide it. diff --git a/doc/administration/monitoring/prometheus/gitlab_exporter.md b/doc/administration/monitoring/prometheus/gitlab_exporter.md index 3effca4a2bf..686ed14ba42 100644 --- a/doc/administration/monitoring/prometheus/gitlab_exporter.md +++ b/doc/administration/monitoring/prometheus/gitlab_exporter.md @@ -9,27 +9,25 @@ info: To determine the technical writer assigned to the Stage/Group associated w >- Available since [Omnibus GitLab 8.17](https://gitlab.com/gitlab-org/omnibus-gitlab/-/merge_requests/1132). >- Renamed from `GitLab monitor exporter` to `GitLab exporter` in [GitLab 12.3](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/16511). -The [GitLab exporter](https://gitlab.com/gitlab-org/gitlab-exporter) allows you to -measure various GitLab metrics, pulled from Redis and the database, in Omnibus GitLab +The [GitLab exporter](https://gitlab.com/gitlab-org/gitlab-exporter) enables you to +measure various GitLab metrics pulled from Redis and the database in Omnibus GitLab instances. NOTE: **Note:** -For installations from source you'll have to install and configure it yourself. +For installations from source you must install and configure it yourself. To enable the GitLab exporter in an Omnibus GitLab instance: -1. [Enable Prometheus](index.md#configuring-prometheus) -1. Edit `/etc/gitlab/gitlab.rb` -1. Add or find and uncomment the following line, making sure it's set to `true`: +1. [Enable Prometheus](index.md#configuring-prometheus). +1. Edit `/etc/gitlab/gitlab.rb`. +1. Add, or find and uncomment, the following line, making sure it's set to `true`: ```ruby gitlab_exporter['enable'] = true ``` 1. Save the file and [reconfigure GitLab](../../restart_gitlab.md#omnibus-gitlab-reconfigure) - for the changes to take effect + for the changes to take effect. -Prometheus will now automatically begin collecting performance data from -the GitLab exporter exposed under `localhost:9168`. - -[← Back to the main Prometheus page](index.md) +Prometheus automatically begins collecting performance data from +the GitLab exporter exposed at `localhost:9168`. diff --git a/doc/administration/monitoring/prometheus/gitlab_metrics.md b/doc/administration/monitoring/prometheus/gitlab_metrics.md index f3084b1732e..fa5e5da80c3 100644 --- a/doc/administration/monitoring/prometheus/gitlab_metrics.md +++ b/doc/administration/monitoring/prometheus/gitlab_metrics.md @@ -14,18 +14,18 @@ To enable the GitLab Prometheus metrics: 1. [Restart GitLab](../../restart_gitlab.md#omnibus-gitlab-restart) for the changes to take effect. NOTE: **Note:** -For installations from source you'll have to configure it yourself. +For installations from source you must configure it yourself. ## Collecting the metrics GitLab monitors its own internal service metrics, and makes them available at the `/-/metrics` endpoint. Unlike other [Prometheus](https://prometheus.io) exporters, to access -it, the client IP address needs to be [explicitly allowed](../ip_whitelist.md). +the metrics, the client IP address must be [explicitly allowed](../ip_whitelist.md). For [Omnibus GitLab](https://docs.gitlab.com/omnibus/) and Chart installations, these metrics are enabled and collected as of [GitLab 9.4](https://gitlab.com/gitlab-org/omnibus-gitlab/-/merge_requests/1702). -For source installations or earlier versions, these metrics must be enabled +For source installations, these metrics must be enabled manually and collected by a Prometheus server. For enabling and viewing metrics from Sidekiq nodes, see [Sidekiq metrics](#sidekiq-metrics). @@ -40,7 +40,7 @@ The following metrics are available: | `gitlab_banzai_cacheless_render_real_duration_seconds` | Histogram | 9.4 | Duration of rendering Markdown into HTML when cached output does not exist | `controller`, `action` | | `gitlab_cache_misses_total` | Counter | 10.2 | Cache read miss | `controller`, `action` | | `gitlab_cache_operation_duration_seconds` | Histogram | 10.2 | Cache access time | | -| `gitlab_cache_operations_total` | Counter | 12.2 | Cache operations by controller/action | `controller`, `action`, `operation` | +| `gitlab_cache_operations_total` | Counter | 12.2 | Cache operations by controller or action | `controller`, `action`, `operation` | | `gitlab_ci_pipeline_creation_duration_seconds` | Histogram | 13.0 | Time in seconds it takes to create a CI/CD pipeline | | | `gitlab_ci_pipeline_size_builds` | Histogram | 13.1 | Total number of builds within a pipeline grouped by a pipeline source | `source` | | `job_waiter_started_total` | Counter | 12.9 | Number of batches of jobs started where a web request is waiting for the jobs to complete | `worker` | @@ -92,16 +92,16 @@ The following metrics are available: | `gitlab_view_rendering_duration_seconds` | Histogram | 10.2 | Duration for views (histogram) | `controller`, `action`, `view` | | `http_requests_total` | Counter | 9.4 | Rack request count | `method` | | `http_request_duration_seconds` | Histogram | 9.4 | HTTP response time from rack middleware | `method`, `status` | -| `gitlab_transaction_db_count_total` | Counter | 13.1 | Counter for total number of sql calls | `controller`, `action` | -| `gitlab_transaction_db_write_count_total` | Counter | 13.1 | Counter for total number of write sql calls | `controller`, `action` | -| `gitlab_transaction_db_cached_count_total` | Counter | 13.1 | Counter for total number of cached sql calls | `controller`, `action` | +| `gitlab_transaction_db_count_total` | Counter | 13.1 | Counter for total number of SQL calls | `controller`, `action` | +| `gitlab_transaction_db_write_count_total` | Counter | 13.1 | Counter for total number of write SQL calls | `controller`, `action` | +| `gitlab_transaction_db_cached_count_total` | Counter | 13.1 | Counter for total number of cached SQL calls | `controller`, `action` | | `http_redis_requests_duration_seconds` | Histogram | 13.1 | Redis requests duration during web transactions | `controller`, `action` | | `http_redis_requests_total` | Counter | 13.1 | Redis requests count during web transactions | `controller`, `action` | | `http_elasticsearch_requests_duration_seconds` **(STARTER)** | Histogram | 13.1 | Elasticsearch requests duration during web transactions | `controller`, `action` | | `http_elasticsearch_requests_total` **(STARTER)** | Counter | 13.1 | Elasticsearch requests count during web transactions | `controller`, `action` | | `pipelines_created_total` | Counter | 9.4 | Counter of pipelines created | | | `rack_uncaught_errors_total` | Counter | 9.4 | Rack connections handling uncaught errors count | | -| `user_session_logins_total` | Counter | 9.4 | Counter of how many users have logged in | | +| `user_session_logins_total` | Counter | 9.4 | Counter of how many users have logged in since GitLab was started or restarted | | | `upload_file_does_not_exist` | Counter | 10.7 in EE, 11.5 in CE | Number of times an upload record could not find its file | | | `failed_login_captcha_total` | Gauge | 11.0 | Counter of failed CAPTCHA attempts during login | | | `successful_login_captcha_total` | Gauge | 11.0 | Counter of successful CAPTCHA attempts during login | | @@ -120,7 +120,7 @@ The following metrics can be controlled by feature flags: ## Sidekiq metrics Sidekiq jobs may also gather metrics, and these metrics can be accessed if the -Sidekiq exporter is enabled (for example, using the `monitoring.sidekiq_exporter` +Sidekiq exporter is enabled: for example, using the `monitoring.sidekiq_exporter` configuration option in `gitlab.yml`. These metrics are served from the `/metrics` path on the configured port. @@ -173,6 +173,7 @@ configuration option in `gitlab.yml`. These metrics are served from the | `geo_repositories_retrying_verification_count` | Gauge | 11.2 | Number of repositories verification failures that Geo is actively trying to correct on secondary | `url` | | `geo_wikis_retrying_verification_count` | Gauge | 11.2 | Number of wikis verification failures that Geo is actively trying to correct on secondary | `url` | | `global_search_bulk_cron_queue_size` | Gauge | 12.10 | Number of database records waiting to be synchronized to Elasticsearch | | +| `global_search_awaiting_indexing_queue_size` | Gauge | 13.2 | Number of database updates waiting to be synchronized to Elasticsearch while indexing is paused | | | `package_files_count` | Gauge | 13.0 | Number of package files on primary | `url` | | `package_files_checksummed_count` | Gauge | 13.0 | Number of package files checksummed on primary | `url` | | `package_files_checksum_failed_count` | Gauge | 13.0 | Number of package files failed to calculate the checksum on primary @@ -187,16 +188,16 @@ The following metrics are available: ## Connection pool metrics -These metrics record the status of the database [connection pools](https://api.rubyonrails.org/classes/ActiveRecord/ConnectionAdapters/ConnectionPool.html). +These metrics record the status of the database +[connection pools](https://api.rubyonrails.org/classes/ActiveRecord/ConnectionAdapters/ConnectionPool.html), +and the metrics all have these labels: -They all have these labels: - -1. `class` - the Ruby class being recorded. - 1. `ActiveRecord::Base` is the main database connection. - 1. `Geo::TrackingBase` is the connection to the Geo tracking database, if - enabled. -1. `host` - the host name used to connect to the database. -1. `port` - the port used to connect to the database. +- `class` - the Ruby class being recorded. + - `ActiveRecord::Base` is the main database connection. + - `Geo::TrackingBase` is the connection to the Geo tracking database, if + enabled. +- `host` - the host name used to connect to the database. +- `port` - the port used to connect to the database. | Metric | Type | Since | Description | |:----------------------------------------------|:------|:------|:--------------------------------------------------| @@ -251,15 +252,29 @@ When Puma is used instead of Unicorn, the following metrics are available: | `puma_idle_threads` | Gauge | 12.0 | Number of spawned threads which are not processing a request | | `puma_killer_terminations_total` | Gauge | 12.0 | Number of workers terminated by PumaWorkerKiller | +## Redis metrics + +These client metrics are meant to complement Redis server metrics. +These metrics are broken down per [Redis +instance](https://docs.gitlab.com/omnibus/settings/redis.html#running-with-multiple-redis-instances). +These metrics all have a `storage` label which indicates the Redis +instance (`cache`, `shared_state` etc.). + +| Metric | Type | Since | Description | +|:--------------------------------- |:------- |:----- |:----------- | +| `gitlab_redis_client_exceptions_total` | Counter | 13.2 | Number of Redis client exceptions, broken down by exception class | +| `gitlab_redis_client_requests_total` | Counter | 13.2 | Number of Redis client requests | +| `gitlab_redis_client_requests_duration_seconds` | Histogram | 13.2 | Redis request latency, excluding blocking commands | + ## Metrics shared directory GitLab's Prometheus client requires a directory to store metrics data shared between multi-process services. Those files are shared among all instances running under Unicorn server. The directory must be accessible to all running Unicorn's processes, or -metrics won't function correctly. +metrics can't function correctly. This directory's location is configured using environment variable `prometheus_multiproc_dir`. For best performance, create this directory in `tmpfs`. If GitLab is installed using [Omnibus GitLab](https://docs.gitlab.com/omnibus/) -and `tmpfs` is available, then the metrics directory will be configured for you. +and `tmpfs` is available, then GitLab configures the metrics directory for you. diff --git a/doc/administration/monitoring/prometheus/index.md b/doc/administration/monitoring/prometheus/index.md index 1e233b890a2..f0ad0a1a2e6 100644 --- a/doc/administration/monitoring/prometheus/index.md +++ b/doc/administration/monitoring/prometheus/index.md @@ -145,6 +145,12 @@ To use an external Prometheus server: gitlab_rails['monitoring_whitelist'] = ['127.0.0.0/8', '192.168.0.1'] ``` +1. On **all** GitLab Rails(Puma/Unicorn, Sidekiq) servers, set the Prometheus server IP address and listen port. For example: + + ```ruby + gitlab_rails['prometheus_address'] = '192.168.0.1:9090' + ``` + 1. To scrape NGINX metrics, you'll also need to configure NGINX to allow the Prometheus server IP. For example: @@ -165,54 +171,54 @@ To use an external Prometheus server: ```yaml scrape_configs: - - job_name: nginx - static_configs: - - targets: - - 1.1.1.1:8060 - - job_name: redis - static_configs: - - targets: - - 1.1.1.1:9121 - - job_name: postgres - static_configs: - - targets: - - 1.1.1.1:9187 - - job_name: node - static_configs: - - targets: - - 1.1.1.1:9100 - - job_name: gitlab-workhorse - static_configs: - - targets: - - 1.1.1.1:9229 - - job_name: gitlab-rails - metrics_path: "/-/metrics" - static_configs: - - targets: - - 1.1.1.1:8080 - - job_name: gitlab-sidekiq - static_configs: - - targets: - - 1.1.1.1:8082 - - job_name: gitlab_exporter_database - metrics_path: "/database" - static_configs: - - targets: - - 1.1.1.1:9168 - - job_name: gitlab_exporter_sidekiq - metrics_path: "/sidekiq" - static_configs: - - targets: - - 1.1.1.1:9168 - - job_name: gitlab_exporter_process - metrics_path: "/process" - static_configs: - - targets: - - 1.1.1.1:9168 - - job_name: gitaly - static_configs: - - targets: - - 1.1.1.1:9236 + - job_name: nginx + static_configs: + - targets: + - 1.1.1.1:8060 + - job_name: redis + static_configs: + - targets: + - 1.1.1.1:9121 + - job_name: postgres + static_configs: + - targets: + - 1.1.1.1:9187 + - job_name: node + static_configs: + - targets: + - 1.1.1.1:9100 + - job_name: gitlab-workhorse + static_configs: + - targets: + - 1.1.1.1:9229 + - job_name: gitlab-rails + metrics_path: "/-/metrics" + static_configs: + - targets: + - 1.1.1.1:8080 + - job_name: gitlab-sidekiq + static_configs: + - targets: + - 1.1.1.1:8082 + - job_name: gitlab_exporter_database + metrics_path: "/database" + static_configs: + - targets: + - 1.1.1.1:9168 + - job_name: gitlab_exporter_sidekiq + metrics_path: "/sidekiq" + static_configs: + - targets: + - 1.1.1.1:9168 + - job_name: gitlab_exporter_process + metrics_path: "/process" + static_configs: + - targets: + - 1.1.1.1:9168 + - job_name: gitaly + static_configs: + - targets: + - 1.1.1.1:9236 ``` 1. Reload the Prometheus server. |
