From 91b752dce63147bc99d7784d3d37865efb5e9352 Mon Sep 17 00:00:00 2001 From: Yorick Peterse Date: Thu, 19 Jul 2018 17:16:47 +0200 Subject: Respond to DB health in background migrations This changes the BackgroundMigration worker so it checks for the health of the DB before performing a background migration. This in turn allows us to reduce the minimum interval, without having to worry about blowing things up if we schedule too many migrations. In this setup, the BackgroundMigration worker will reschedule jobs as long as the database is considered to be in an unhealthy state. Once the database has recovered, the migration can be performed. To determine if the database is in a healthy state, we look at the replication lag of any replication slots defined on the primary. If the lag is deemed to great (100 MB by default) for too many slots, the migration is rescheduled for a later point in time. The health checking code is hidden behind a feature flag, allowing us to disable it if necessary. --- lib/gitlab/background_migration.rb | 6 +++++- lib/gitlab/database/migration_helpers.rb | 4 ++-- 2 files changed, 7 insertions(+), 3 deletions(-) (limited to 'lib') diff --git a/lib/gitlab/background_migration.rb b/lib/gitlab/background_migration.rb index d3f66877672..36c85dec544 100644 --- a/lib/gitlab/background_migration.rb +++ b/lib/gitlab/background_migration.rb @@ -46,7 +46,11 @@ module Gitlab # arguments - The arguments to pass to the background migration's "perform" # method. def self.perform(class_name, arguments) - const_get(class_name).new.perform(*arguments) + migration_class_for(class_name).new.perform(*arguments) + end + + def self.migration_class_for(class_name) + const_get(class_name) end end end diff --git a/lib/gitlab/database/migration_helpers.rb b/lib/gitlab/database/migration_helpers.rb index 4fe5b4cc835..f39b3b6eb5b 100644 --- a/lib/gitlab/database/migration_helpers.rb +++ b/lib/gitlab/database/migration_helpers.rb @@ -979,8 +979,8 @@ into similar problems in the future (e.g. when new tables are created). # To not overload the worker too much we enforce a minimum interval both # when scheduling and performing jobs. - if delay_interval < BackgroundMigrationWorker::MIN_INTERVAL - delay_interval = BackgroundMigrationWorker::MIN_INTERVAL + if delay_interval < BackgroundMigrationWorker.minimum_interval + delay_interval = BackgroundMigrationWorker.minimum_interval end model_class.each_batch(of: batch_size) do |relation, index| -- cgit v1.2.3