1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
|
---
stage: Systems
group: Distribution
info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments
---
# Monitoring GitLab with Prometheus **(FREE SELF)**
[Prometheus](https://prometheus.io) is a powerful time-series monitoring service, providing a flexible
platform for monitoring GitLab and other software products.
GitLab provides out-of-the-box monitoring with Prometheus, providing easy
access to high quality time-series monitoring of GitLab services.
Prometheus and the various exporters listed in this page are bundled in the
Omnibus GitLab package. Check each exporter's documentation for the timeline
they got added. For installations from source you must install them
yourself. Over subsequent releases additional GitLab metrics are captured.
Prometheus services are on by default.
Prometheus and its exporters don't authenticate users, and are available to anyone who can access
them.
## How Prometheus works
Prometheus works by periodically connecting to data sources and collecting their
performance metrics through the [various exporters](#bundled-software-metrics). To view
and work with the monitoring data, you can either
[connect directly to Prometheus](#viewing-performance-metrics) or use a
dashboard tool like [Grafana](https://grafana.com).
## Configuring Prometheus
For installations from source, you must install and configure it yourself.
Prometheus and its exporters are on by default.
Prometheus runs as the `gitlab-prometheus` user and listen on
`http://localhost:9090`. By default, Prometheus is only accessible from the GitLab server itself.
Each exporter is automatically set up as a
monitoring target for Prometheus, unless individually disabled.
To disable Prometheus and all of its exporters, as well as any added in the future:
1. Edit `/etc/gitlab/gitlab.rb`
1. Add or find and uncomment the following line, making sure it's set to `false`:
```ruby
prometheus_monitoring['enable'] = false
```
1. Save the file and [reconfigure GitLab](../../restart_gitlab.md#reconfigure-a-linux-package-installation) for the changes to
take effect.
### Changing the port and address Prometheus listens on
WARNING:
Although possible, it's not recommended to change the port Prometheus listens
on, as this might affect or conflict with other services running on the GitLab
server. Proceed at your own risk.
To access Prometheus from outside the GitLab server,
change the address/port that Prometheus
listens on:
1. Edit `/etc/gitlab/gitlab.rb`
1. Add or find and uncomment the following line:
```ruby
prometheus['listen_address'] = 'localhost:9090'
```
Replace `localhost:9090` with the address or port you want Prometheus to
listen on. If you would like to allow access to Prometheus to hosts other
than `localhost`, leave out the host, or use `0.0.0.0` to allow public access:
```ruby
prometheus['listen_address'] = ':9090'
# or
prometheus['listen_address'] = '0.0.0.0:9090'
```
1. Save the file and [reconfigure GitLab](../../restart_gitlab.md#reconfigure-a-linux-package-installation) for the changes to
take effect
### Adding custom scrape configurations
You can configure additional scrape targets for the Omnibus GitLab-bundled
Prometheus by editing `prometheus['scrape_configs']` in `/etc/gitlab/gitlab.rb`
using the [Prometheus scrape target configuration](https://prometheus.io/docs/prometheus/latest/configuration/configuration/#%3Cscrape_config%3E)
syntax.
Here is an example configuration to scrape `http://1.1.1.1:8060/probe?param_a=test¶m_b=additional_test`:
```ruby
prometheus['scrape_configs'] = [
{
'job_name': 'custom-scrape',
'metrics_path': '/probe',
'params' => {
'param_a' => ['test'],
'param_b' => ['additional_test']
},
'static_configs' => [
'targets' => ['1.1.1.1:8060'],
],
},
]
```
### Standalone Prometheus using Omnibus GitLab
The Omnibus GitLab package can be used to configure a standalone Monitoring node running Prometheus and [Grafana](../performance/grafana_configuration.md).
The steps below are the minimum necessary to configure a Monitoring node running Prometheus and Grafana with Omnibus GitLab:
1. SSH into the Monitoring node.
1. [Install](https://about.gitlab.com/install/) the Omnibus GitLab
package you want using **steps 1 and 2** from the GitLab downloads page, but
do not follow the remaining steps.
1. Make sure to collect the IP addresses or DNS records of the Consul server nodes, for the next step.
1. Edit `/etc/gitlab/gitlab.rb` and add the contents:
```ruby
roles ['monitoring_role']
external_url 'http://gitlab.example.com'
# Prometheus
prometheus['listen_address'] = '0.0.0.0:9090'
prometheus['monitor_kubernetes'] = false
# Grafana
grafana['enable'] = true
grafana['admin_password'] = 'toomanysecrets'
grafana['disable_login_form'] = false
# Enable service discovery for Prometheus
consul['enable'] = true
consul['monitoring_service_discovery'] = true
consul['configuration'] = {
retry_join: %w(10.0.0.1 10.0.0.2 10.0.0.3), # The addresses can be IPs or FQDNs
}
# Nginx - For Grafana access
nginx['enable'] = true
```
1. Run `sudo gitlab-ctl reconfigure` to compile the configuration.
The next step is to tell all the other nodes where the monitoring node is:
1. Edit `/etc/gitlab/gitlab.rb`, and add, or find and uncomment the following line:
```ruby
gitlab_rails['prometheus_address'] = '10.0.0.1:9090'
```
Where `10.0.0.1:9090` is the IP address and port of the Prometheus node.
1. Save the file and [reconfigure GitLab](../../restart_gitlab.md#reconfigure-a-linux-package-installation) for the changes to
take effect.
After monitoring using Service Discovery is enabled with `consul['monitoring_service_discovery'] = true`,
ensure that `prometheus['scrape_configs']` is not set in `/etc/gitlab/gitlab.rb`. Setting both
`consul['monitoring_service_discovery'] = true` and `prometheus['scrape_configs']` in `/etc/gitlab/gitlab.rb` results in errors.
### Using an external Prometheus server
WARNING:
Prometheus and most exporters don't support authentication. We don't recommend exposing them outside the local network.
A few configuration changes are required to allow GitLab to be monitored by an external Prometheus server. External servers are recommended for [GitLab deployments with multiple nodes](../../reference_architectures/index.md).
To use an external Prometheus server:
1. Edit `/etc/gitlab/gitlab.rb`.
1. Disable the bundled Prometheus:
```ruby
prometheus['enable'] = false
```
1. Set each bundled service's [exporter](#bundled-software-metrics) to listen on a network address, for example:
```ruby
node_exporter['listen_address'] = '0.0.0.0:9100'
gitlab_workhorse['prometheus_listen_addr'] = "0.0.0.0:9229"
# Rails nodes
gitlab_exporter['listen_address'] = '0.0.0.0'
gitlab_exporter['listen_port'] = '9168'
registry['debug_addr'] = '0.0.0.0:5001'
# Sidekiq nodes
sidekiq['listen_address'] = '0.0.0.0'
# Redis nodes
redis_exporter['listen_address'] = '0.0.0.0:9121'
# PostgreSQL nodes
postgres_exporter['listen_address'] = '0.0.0.0:9187'
# Gitaly nodes
gitaly['configuration'] = {
# ...
prometheus_listen_addr: '0.0.0.0:9236',
}
# Pgbouncer nodes
pgbouncer_exporter['listen_address'] = '0.0.0.0:9188'
```
1. Install and set up a dedicated Prometheus instance, if necessary, using the [official installation instructions](https://prometheus.io/docs/prometheus/latest/installation/).
1. On **all** GitLab Rails(Puma, Sidekiq) servers, set the Prometheus server IP address and listen port. For example:
```ruby
gitlab_rails['prometheus_address'] = '192.168.0.1:9090'
```
1. To scrape NGINX metrics, you must also configure NGINX to allow the Prometheus server
IP. For example:
```ruby
nginx['status']['options'] = {
"server_tokens" => "off",
"access_log" => "off",
"allow" => "192.168.0.1",
"deny" => "all",
}
```
1. To allow the Prometheus server to fetch from the [GitLab metrics](#gitlab-metrics) endpoint, add the Prometheus
server IP address to the [monitoring IP allowlist](../ip_allowlist.md):
```ruby
gitlab_rails['monitoring_whitelist'] = ['127.0.0.0/8', '192.168.0.1']
```
1. As we are setting each bundled service's [exporter](#bundled-software-metrics) to listen on a network address,
update the firewall on the instance to only allow traffic from your Prometheus IP for the exporters enabled. A full reference list of exporter services and their respective ports can be found [here](../../package_information/defaults.md#ports).
1. [Reconfigure GitLab](../../restart_gitlab.md#reconfigure-a-linux-package-installation) to apply the changes.
1. Edit the Prometheus server's configuration file.
1. Add each node's exporters to the Prometheus server's
[scrape target configuration](https://prometheus.io/docs/prometheus/latest/configuration/configuration/#%3Cscrape_config%3E).
For example, a sample snippet using `static_configs`:
```yaml
scrape_configs:
- job_name: nginx
static_configs:
- targets:
- 1.1.1.1:8060
- job_name: redis
static_configs:
- targets:
- 1.1.1.1:9121
- job_name: postgres
static_configs:
- targets:
- 1.1.1.1:9187
- job_name: node
static_configs:
- targets:
- 1.1.1.1:9100
- job_name: gitlab-workhorse
static_configs:
- targets:
- 1.1.1.1:9229
- job_name: gitlab-rails
metrics_path: "/-/metrics"
scheme: https
static_configs:
- targets:
- 1.1.1.1
- job_name: gitlab-sidekiq
static_configs:
- targets:
- 1.1.1.1:8082
- job_name: gitlab_exporter_database
metrics_path: "/database"
static_configs:
- targets:
- 1.1.1.1:9168
- job_name: gitlab_exporter_sidekiq
metrics_path: "/sidekiq"
static_configs:
- targets:
- 1.1.1.1:9168
- job_name: gitaly
static_configs:
- targets:
- 1.1.1.1:9236
```
WARNING:
The `gitlab-rails` job in the snippet assumes that GitLab is reachable through HTTPS. If your
deployment doesn't use HTTPS, the job configuration is adapted to use the `http` scheme and port
80.
1. Reload the Prometheus server.
### Configure the storage retention size
Prometheus has several custom flags to configure local storage:
- `storage.tsdb.retention.time`: when to remove old data. Defaults to `15d`. Overrides
`storage.tsdb.retention` if this flag is set to anything other than the default.
- `storage.tsdb.retention.size`: (experimental) the maximum number of bytes of storage blocks to
retain. The oldest data is removed first. Defaults to `0` (disabled). This flag is experimental
and may change in future releases. Units supported: `B`, `KB`, `MB`, `GB`, `TB`, `PB`, `EB`. For
example, `512MB`.
To configure the storage retention size:
1. Edit `/etc/gitlab/gitlab.rb`:
```ruby
prometheus['flags'] = {
'storage.tsdb.path' => "/var/opt/gitlab/prometheus/data",
'storage.tsdb.retention.time' => "7d",
'storage.tsdb.retention.size' => "2GB",
'config.file' => "/var/opt/gitlab/prometheus/prometheus.yml"
}
```
1. Reconfigure GitLab:
```shell
sudo gitlab-ctl reconfigure
```
## Viewing performance metrics
You can visit `http://localhost:9090` for the dashboard that Prometheus offers by default.
If SSL has been enabled on your GitLab instance, you may not be able to access
Prometheus on the same browser as GitLab if using the same FQDN due to [HSTS](https://en.wikipedia.org/wiki/HTTP_Strict_Transport_Security).
[A test project exists](https://gitlab.com/gitlab-org/multi-user-prometheus) to provide access via GitLab, but in the interim there are
some workarounds: using a separate FQDN, using server IP, using a separate browser for Prometheus, resetting HSTS, or
having [NGINX proxy it](https://docs.gitlab.com/omnibus/settings/nginx.html#inserting-custom-nginx-settings-into-the-gitlab-server-block).
The performance data collected by Prometheus can be viewed directly in the
Prometheus console, or through a compatible dashboard tool.
The Prometheus interface provides a [flexible query language](https://prometheus.io/docs/prometheus/latest/querying/basics/)
to work with the collected data where you can visualize the output.
For a more fully featured dashboard, Grafana can be used and has
[official support for Prometheus](https://prometheus.io/docs/visualization/grafana/).
Sample Prometheus queries:
- **% Memory available:** `((node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes) or ((node_memory_MemFree_bytes + node_memory_Buffers_bytes + node_memory_Cached_bytes) / node_memory_MemTotal_bytes)) * 100`
- **% CPU utilization:** `1 - avg without (mode,cpu) (rate(node_cpu_seconds_total{mode="idle"}[5m]))`
- **Data transmitted:** `rate(node_network_transmit_bytes_total{device!="lo"}[5m])`
- **Data received:** `rate(node_network_receive_bytes_total{device!="lo"}[5m])`
## Prometheus as a Grafana data source
Grafana allows you to import Prometheus performance metrics as a data source,
and render the metrics as graphs and dashboards, which is helpful with visualization.
To add a Prometheus dashboard for a single server GitLab setup:
1. Create a new data source in Grafana.
1. Name your data source (such as GitLab).
1. Select `Prometheus` in the type dropdown list.
1. Add your Prometheus listen address as the URL, and set access to `Browser`.
1. Set the HTTP method to `GET`.
1. Save and test your configuration to verify that it works.
## GitLab metrics
GitLab monitors its own internal service metrics, and makes them available at the `/-/metrics` endpoint. Unlike other exporters, this endpoint requires authentication as it's available on the same URL and port as user traffic.
Read more about the [GitLab Metrics](gitlab_metrics.md).
## Bundled software metrics
Many of the GitLab dependencies bundled in Omnibus GitLab are preconfigured to
export Prometheus metrics.
### Node exporter
The node exporter allows you to measure various machine resources, such as
memory, disk, and CPU utilization.
[Read more about the node exporter](node_exporter.md).
### Web exporter
The web exporter is a dedicated metrics server that allows splitting end-user and Prometheus traffic
into two separate applications to improve performance and availability.
[Read more about the web exporter](web_exporter.md).
### Redis exporter
The Redis exporter allows you to measure various Redis metrics.
[Read more about the Redis exporter](redis_exporter.md).
### PostgreSQL exporter
The PostgreSQL exporter allows you to measure various PostgreSQL metrics.
[Read more about the PostgreSQL exporter](postgres_exporter.md).
### PgBouncer exporter
The PgBouncer exporter allows you to measure various PgBouncer metrics.
[Read more about the PgBouncer exporter](pgbouncer_exporter.md).
### Registry exporter
The Registry exporter allows you to measure various Registry metrics.
[Read more about the Registry exporter](registry_exporter.md).
### GitLab exporter
The GitLab exporter allows you to measure various GitLab metrics, pulled from Redis and the database.
[Read more about the GitLab exporter](gitlab_exporter.md).
## Configuring Prometheus to monitor Kubernetes
If your GitLab server is running within Kubernetes, Prometheus collects metrics from the Nodes and [annotated Pods](https://prometheus.io/docs/prometheus/latest/configuration/configuration/#kubernetes_sd_config) in the cluster, including performance data on each container. This is particularly helpful if your CI/CD environments run in the same cluster, as you can use the [Prometheus project integration](../../../user/project/integrations/prometheus.md) to monitor them.
To disable the monitoring of Kubernetes:
1. Edit `/etc/gitlab/gitlab.rb`.
1. Add (or find and uncomment) the following line and set it to `false`:
```ruby
prometheus['monitor_kubernetes'] = false
```
1. Save the file and [reconfigure GitLab](../../restart_gitlab.md#reconfigure-a-linux-package-installation) for the changes to
take effect.
## Troubleshooting
### `/var/opt/gitlab/prometheus` consumes too much disk space
If you are **not** using Prometheus monitoring:
1. [Disable Prometheus](index.md#configuring-prometheus).
1. Delete the data under `/var/opt/gitlab/prometheus`.
If you are using Prometheus monitoring:
1. Stop Prometheus (deleting data while it's running can cause data corruption):
```shell
gitlab-ctl stop prometheus
```
1. Delete the data under `/var/opt/gitlab/prometheus/data`.
1. Start the service again:
```shell
gitlab-ctl start prometheus
```
1. Verify the service is up and running:
```shell
gitlab-ctl status prometheus
```
1. Optional. [Configure the storage retention size](index.md#configure-the-storage-retention-size).
### Monitoring node not receiving data
If the monitoring node is not receiving any data, check that the exporters are capturing data:
```shell
curl "http[s]://localhost:<EXPORTER LISTENING PORT>/metrics"
```
or
```shell
curl "http[s]://localhost:<EXPORTER LISTENING PORT>/-/metrics"
```
|