diff options
author | Aleksei Lipniagov <alipniagov@gitlab.com> | 2019-08-09 14:56:22 +0300 |
---|---|---|
committer | Aleksei Lipniagov <alipniagov@gitlab.com> | 2019-08-12 13:52:15 +0300 |
commit | 3f9815865c38aeec0007b4ef3c636be51967b661 (patch) | |
tree | 0790f5bb23fedf11d706bf0abfd203045ad97e2f | |
parent | 7a48b4d6b0f3e92887407768d639d13489d0d495 (diff) |
Fix metric files being wiped after the app starts
When we hit our app with the initial request, in `warmup`,
some metrics already being created as well as corresponding files.
If we do `multiproc_file_dir` cleanup after that, we delete the files
from the dir while keeping them in memory which leads to the incorrect
behavior: the metric is being updated in in-memory, while is not present
in the db, not sent to Prometheus as the result.
-rw-r--r-- | changelogs/unreleased/65278-fix-puma-master-counter-wipe.yml | 5 | ||||
-rw-r--r-- | config.ru | 15 | ||||
-rw-r--r-- | config/initializers/7_prometheus_metrics.rb | 19 |
3 files changed, 20 insertions, 19 deletions
diff --git a/changelogs/unreleased/65278-fix-puma-master-counter-wipe.yml b/changelogs/unreleased/65278-fix-puma-master-counter-wipe.yml new file mode 100644 index 00000000000..fb9d6fa251d --- /dev/null +++ b/changelogs/unreleased/65278-fix-puma-master-counter-wipe.yml @@ -0,0 +1,5 @@ +--- +title: Fix active metric files being wiped after the app starts +merge_request: 31668 +author: +type: fixed diff --git a/config.ru b/config.ru index 6f6fb85d8fa..d3c6e304998 100644 --- a/config.ru +++ b/config.ru @@ -17,7 +17,22 @@ end require ::File.expand_path('../config/environment', __FILE__) + +# The following is necessary to ensure stale Prometheus metrics don't accumulate over time. +# It needs to be done as early as here to ensure metrics files aren't deleted. +# After we hit our app in `warmup`, first metrics and corresponding files already being created, +# for example in `lib/gitlab/metrics/requests_rack_middleware.rb`. +def cleanup_prometheus_multiproc_dir + if dir = ::Prometheus::Client.configuration.multiprocess_files_dir + old_metrics = Dir[File.join(dir, '*.db')] + + FileUtils.rm_rf(old_metrics) + end +end + warmup do |app| + cleanup_prometheus_multiproc_dir + client = Rack::MockRequest.new(app) client.get('/') end diff --git a/config/initializers/7_prometheus_metrics.rb b/config/initializers/7_prometheus_metrics.rb index 3f2691dde95..70e5dcd042e 100644 --- a/config/initializers/7_prometheus_metrics.rb +++ b/config/initializers/7_prometheus_metrics.rb @@ -51,22 +51,3 @@ if !Rails.env.test? && Gitlab::Metrics.prometheus_metrics_enabled? end end end - -def cleanup_prometheus_multiproc_dir - # The following is necessary to ensure stale Prometheus metrics don't - # accumulate over time. It needs to be done in this hook as opposed to - # inside an init script to ensure metrics files aren't deleted after new - # unicorn workers start after a SIGUSR2 is received. - if dir = ::Prometheus::Client.configuration.multiprocess_files_dir - old_metrics = Dir[File.join(dir, '*.db')] - FileUtils.rm_rf(old_metrics) - end -end - -Gitlab::Cluster::LifecycleEvents.on_master_start do - cleanup_prometheus_multiproc_dir -end - -Gitlab::Cluster::LifecycleEvents.on_master_restart do - cleanup_prometheus_multiproc_dir -end |