diff options
author | Yorick Peterse <yorickpeterse@gmail.com> | 2018-07-19 18:16:47 +0300 |
---|---|---|
committer | Yorick Peterse <yorickpeterse@gmail.com> | 2018-08-06 16:20:36 +0300 |
commit | 91b752dce63147bc99d7784d3d37865efb5e9352 (patch) | |
tree | 447dcd9dc5efcb14af5439f247d87938daf845dc /app/models/postgresql | |
parent | 5f742eb95a0080343167469ccabfeccd3630007d (diff) |
Respond to DB health in background migrations
This changes the BackgroundMigration worker so it checks for the health
of the DB before performing a background migration. This in turn allows
us to reduce the minimum interval, without having to worry about blowing
things up if we schedule too many migrations.
In this setup, the BackgroundMigration worker will reschedule jobs as
long as the database is considered to be in an unhealthy state. Once the
database has recovered, the migration can be performed.
To determine if the database is in a healthy state, we look at the
replication lag of any replication slots defined on the primary. If the
lag is deemed to great (100 MB by default) for too many slots, the
migration is rescheduled for a later point in time.
The health checking code is hidden behind a feature flag, allowing us to
disable it if necessary.
Diffstat (limited to 'app/models/postgresql')
-rw-r--r-- | app/models/postgresql/replication_slot.rb | 32 |
1 files changed, 32 insertions, 0 deletions
diff --git a/app/models/postgresql/replication_slot.rb b/app/models/postgresql/replication_slot.rb new file mode 100644 index 00000000000..70c7432e6b5 --- /dev/null +++ b/app/models/postgresql/replication_slot.rb @@ -0,0 +1,32 @@ +# frozen_string_literal: true + +module Postgresql + class ReplicationSlot < ActiveRecord::Base + self.table_name = 'pg_replication_slots' + + # Returns true if the lag observed across all replication slots exceeds a + # given threshold. + # + # max - The maximum replication lag size, in bytes. Based on GitLab.com + # statistics it takes between 1 and 5 seconds to replicate around + # 100 MB of data. + def self.lag_too_great?(max = 100.megabytes) + lag_function = "#{Gitlab::Database.pg_wal_lsn_diff}" \ + "(#{Gitlab::Database.pg_current_wal_insert_lsn}(), restart_lsn)::bigint" + + # We force the use of a transaction here so the query always goes to the + # primary, even when using the EE DB load balancer. + sizes = transaction { pluck(lag_function) } + too_great = sizes.count { |size| size >= max } + + # If too many replicas are falling behind too much, the availability of a + # GitLab instance might suffer. To prevent this from happening we require + # at least 1 replica to have data recent enough. + if sizes.any? && too_great.positive? + (sizes.length - too_great) <= 1 + else + false + end + end + end +end |