Welcome to mirror list, hosted at ThFree Co, Russian Federation.

gitlab.com/gitlab-org/gitlab-foss.git - Unnamed repository; edit this file 'description' to name the repository.
summaryrefslogtreecommitdiff
diff options
context:
space:
mode:
authorYorick Peterse <yorickpeterse@gmail.com>2019-04-29 15:16:03 +0300
committerYorick Peterse <yorickpeterse@gmail.com>2019-06-17 18:09:05 +0300
commit8469f59d786be6762908f62d642625790999cb9b (patch)
tree03f99c605e465673e017e775a623ffa5b62f18d3 /lib/gitlab/background_migration
parent0f777a8d49568d3f5ffdd4b75b594f07e9fbd2f0 (diff)
Backport the EE schema and migrations to CE
This backports all EE schema changes to CE, including EE migrations, ensuring both use the same schema. == Updated tests A spec related to ghost and support bot users had to be modified to make it pass. The spec in question assumes that the "support_bot" column exists when defining the spec. In the single codebase setup this is not the case, as the column is backported in a later migration. Any attempt to use a different schema version or use of "around" blocks to conditionally disable specs won't help, as reverting the backport migration would also drop the "support_bot" column. Removing the "support_bot" tests entirely appears to be the only solution. We also need to update some foreign key tests now that we have backported the EE columns. Fortunately, these changes are very minor. == Backporting migrations This commit moves EE specific migrations (except those for the Geo tracking database) and related files to CE, and also removes any traces of the ee/db directory. Some migrations had to be modified or removed, as they no longer work with the schema being backported. These migrations were all quite old, so we opted for removing them where modifying them would take too much time and effort. Some old migrations were modified in EE, while also existing in CE. In these cases we took the EE code, and in one case removed them entirely. It's not worth spending time trying to merge these changes somehow as we plan to remove old migrations around the release of 12.0, see https://gitlab.com/gitlab-org/gitlab-ce/issues/59177 for more details.
Diffstat (limited to 'lib/gitlab/background_migration')
-rw-r--r--lib/gitlab/background_migration/create_gpg_key_subkeys_from_gpg_keys.rb56
-rw-r--r--lib/gitlab/background_migration/delete_diff_files.rb81
-rw-r--r--lib/gitlab/background_migration/deserialize_merge_request_diffs_and_commits.rb149
-rw-r--r--lib/gitlab/background_migration/merge_request_assignees_migration_progress_check.rb43
-rw-r--r--lib/gitlab/background_migration/populate_external_pipeline_source.rb50
-rw-r--r--lib/gitlab/background_migration/populate_import_state.rb39
-rw-r--r--lib/gitlab/background_migration/populate_merge_request_metrics_with_events_data.rb132
-rw-r--r--lib/gitlab/background_migration/populate_merge_request_metrics_with_events_data_improved.rb99
-rw-r--r--lib/gitlab/background_migration/redact_links.rb51
-rw-r--r--lib/gitlab/background_migration/redact_links/redactable.rb21
-rw-r--r--lib/gitlab/background_migration/rollback_import_state_data.rb40
-rw-r--r--lib/gitlab/background_migration/schedule_diff_files_deletion.rb44
12 files changed, 43 insertions, 762 deletions
diff --git a/lib/gitlab/background_migration/create_gpg_key_subkeys_from_gpg_keys.rb b/lib/gitlab/background_migration/create_gpg_key_subkeys_from_gpg_keys.rb
deleted file mode 100644
index da8265a3a5f..00000000000
--- a/lib/gitlab/background_migration/create_gpg_key_subkeys_from_gpg_keys.rb
+++ /dev/null
@@ -1,56 +0,0 @@
-# frozen_string_literal: true
-# rubocop:disable Style/Documentation
-
-class Gitlab::BackgroundMigration::CreateGpgKeySubkeysFromGpgKeys
- class GpgKey < ActiveRecord::Base
- self.table_name = 'gpg_keys'
-
- include EachBatch
- include ShaAttribute
-
- sha_attribute :primary_keyid
- sha_attribute :fingerprint
-
- has_many :subkeys, class_name: 'GpgKeySubkey'
- end
-
- class GpgKeySubkey < ActiveRecord::Base
- self.table_name = 'gpg_key_subkeys'
-
- include ShaAttribute
-
- sha_attribute :keyid
- sha_attribute :fingerprint
- end
-
- def perform(gpg_key_id)
- gpg_key = GpgKey.find_by(id: gpg_key_id)
-
- return if gpg_key.nil?
- return if gpg_key.subkeys.any?
-
- create_subkeys(gpg_key)
- update_signatures(gpg_key)
- end
-
- private
-
- def create_subkeys(gpg_key)
- gpg_subkeys = Gitlab::Gpg.subkeys_from_key(gpg_key.key)
-
- gpg_subkeys[gpg_key.primary_keyid.upcase]&.each do |subkey_data|
- gpg_key.subkeys.build(keyid: subkey_data[:keyid], fingerprint: subkey_data[:fingerprint])
- end
-
- # Improve latency by doing all INSERTs in a single call
- GpgKey.transaction do
- gpg_key.save!
- end
- end
-
- def update_signatures(gpg_key)
- return unless gpg_key.subkeys.exists?
-
- InvalidGpgSignatureUpdateWorker.perform_async(gpg_key.id)
- end
-end
diff --git a/lib/gitlab/background_migration/delete_diff_files.rb b/lib/gitlab/background_migration/delete_diff_files.rb
deleted file mode 100644
index 664ead1af44..00000000000
--- a/lib/gitlab/background_migration/delete_diff_files.rb
+++ /dev/null
@@ -1,81 +0,0 @@
-# frozen_string_literal: true
-# rubocop:disable Style/Documentation
-
-module Gitlab
- module BackgroundMigration
- class DeleteDiffFiles
- class MergeRequestDiff < ActiveRecord::Base
- self.table_name = 'merge_request_diffs'
-
- belongs_to :merge_request
- has_many :merge_request_diff_files
- end
-
- class MergeRequestDiffFile < ActiveRecord::Base
- self.table_name = 'merge_request_diff_files'
- end
-
- DEAD_TUPLES_THRESHOLD = 50_000
- VACUUM_WAIT_TIME = 5.minutes
-
- def perform(ids)
- @ids = ids
-
- # We should reschedule until deadtuples get in a desirable
- # state (e.g. < 50_000). That may take more than one reschedule.
- #
- if should_wait_deadtuple_vacuum?
- reschedule
- return
- end
-
- prune_diff_files
- end
-
- def should_wait_deadtuple_vacuum?
- return false unless Gitlab::Database.postgresql?
-
- diff_files_dead_tuples_count >= DEAD_TUPLES_THRESHOLD
- end
-
- private
-
- def reschedule
- BackgroundMigrationWorker.perform_in(VACUUM_WAIT_TIME, self.class.name.demodulize, [@ids])
- end
-
- def diffs_collection
- MergeRequestDiff.where(id: @ids)
- end
-
- def diff_files_dead_tuples_count
- dead_tuple =
- execute_statement("SELECT n_dead_tup FROM pg_stat_all_tables "\
- "WHERE relname = 'merge_request_diff_files'")[0]
-
- dead_tuple&.fetch('n_dead_tup', 0).to_i
- end
-
- def prune_diff_files
- removed = 0
- updated = 0
-
- MergeRequestDiff.transaction do
- updated = diffs_collection.update_all(state: 'without_files')
- removed = MergeRequestDiffFile.where(merge_request_diff_id: @ids).delete_all
- end
-
- log_info("Removed #{removed} merge_request_diff_files rows, "\
- "updated #{updated} merge_request_diffs rows")
- end
-
- def execute_statement(sql)
- ActiveRecord::Base.connection.execute(sql)
- end
-
- def log_info(message)
- Rails.logger.info("BackgroundMigration::DeleteDiffFiles - #{message}")
- end
- end
- end
-end
diff --git a/lib/gitlab/background_migration/deserialize_merge_request_diffs_and_commits.rb b/lib/gitlab/background_migration/deserialize_merge_request_diffs_and_commits.rb
deleted file mode 100644
index 58df74cfa9b..00000000000
--- a/lib/gitlab/background_migration/deserialize_merge_request_diffs_and_commits.rb
+++ /dev/null
@@ -1,149 +0,0 @@
-# frozen_string_literal: true
-# rubocop:disable Metrics/MethodLength
-# rubocop:disable Metrics/AbcSize
-# rubocop:disable Style/Documentation
-
-module Gitlab
- module BackgroundMigration
- class DeserializeMergeRequestDiffsAndCommits
- attr_reader :diff_ids, :commit_rows, :file_rows
-
- class Error < StandardError
- def backtrace
- cause.backtrace
- end
- end
-
- class MergeRequestDiff < ActiveRecord::Base
- self.table_name = 'merge_request_diffs'
- end
-
- BUFFER_ROWS = 1000
- DIFF_FILE_BUFFER_ROWS = 100
-
- def perform(start_id, stop_id)
- merge_request_diffs = MergeRequestDiff
- .select(:id, :st_commits, :st_diffs)
- .where('st_commits IS NOT NULL OR st_diffs IS NOT NULL')
- .where(id: start_id..stop_id)
-
- reset_buffers!
-
- merge_request_diffs.each do |merge_request_diff|
- commits, files = single_diff_rows(merge_request_diff)
-
- diff_ids << merge_request_diff.id
- commit_rows.concat(commits)
- file_rows.concat(files)
-
- if diff_ids.length > BUFFER_ROWS ||
- commit_rows.length > BUFFER_ROWS ||
- file_rows.length > DIFF_FILE_BUFFER_ROWS
-
- flush_buffers!
- end
- end
-
- flush_buffers!
- rescue => e
- Rails.logger.info("#{self.class.name}: failed for IDs #{merge_request_diffs.map(&:id)} with #{e.class.name}")
-
- raise Error.new(e.inspect)
- end
-
- private
-
- def reset_buffers!
- @diff_ids = []
- @commit_rows = []
- @file_rows = []
- end
-
- def flush_buffers!
- if diff_ids.any?
- commit_rows.each_slice(BUFFER_ROWS).each do |commit_rows_slice|
- bulk_insert('merge_request_diff_commits', commit_rows_slice)
- end
-
- file_rows.each_slice(DIFF_FILE_BUFFER_ROWS).each do |file_rows_slice|
- bulk_insert('merge_request_diff_files', file_rows_slice)
- end
-
- MergeRequestDiff.where(id: diff_ids).update_all(st_commits: nil, st_diffs: nil)
- end
-
- reset_buffers!
- end
-
- def bulk_insert(table, rows)
- Gitlab::Database.bulk_insert(table, rows)
- rescue ActiveRecord::RecordNotUnique
- ids = rows.map { |row| row[:merge_request_diff_id] }.uniq.sort
-
- Rails.logger.info("#{self.class.name}: rows inserted twice for IDs #{ids}")
- end
-
- def single_diff_rows(merge_request_diff)
- sha_attribute = Gitlab::Database::ShaAttribute.new
- commits = YAML.load(merge_request_diff.st_commits) rescue []
- commits ||= []
-
- commit_rows = commits.map.with_index do |commit, index|
- commit_hash = commit.to_hash.with_indifferent_access.except(:parent_ids)
- sha = commit_hash.delete(:id)
-
- commit_hash.merge(
- merge_request_diff_id: merge_request_diff.id,
- relative_order: index,
- sha: sha_attribute.serialize(sha)
- )
- end
-
- diffs = YAML.load(merge_request_diff.st_diffs) rescue []
- diffs = [] unless valid_raw_diffs?(diffs)
-
- file_rows = diffs.map.with_index do |diff, index|
- diff_hash = diff.to_hash.with_indifferent_access.merge(
- binary: false,
- merge_request_diff_id: merge_request_diff.id,
- relative_order: index
- )
-
- diff_hash.tap do |hash|
- diff_text = hash[:diff]
-
- hash[:too_large] = !!hash[:too_large]
-
- hash[:a_mode] ||= guess_mode(hash[:new_file], hash[:diff])
- hash[:b_mode] ||= guess_mode(hash[:deleted_file], hash[:diff])
-
- # Compatibility with old diffs created with Psych.
- if diff_text.encoding == Encoding::BINARY && !diff_text.ascii_only?
- hash[:binary] = true
- hash[:diff] = [diff_text].pack('m0')
- end
- end
- end
-
- [commit_rows, file_rows]
- end
-
- # This doesn't have to be 100% accurate, because it's only used for
- # display - it won't change file modes in the repository. Submodules are
- # created as 600, regular files as 644.
- def guess_mode(file_missing, diff)
- return '0' if file_missing
-
- diff.include?('Subproject commit') ? '160000' : '100644'
- end
-
- # Unlike MergeRequestDiff#valid_raw_diff?, don't count Rugged objects as
- # valid, because we don't render them usefully anyway.
- def valid_raw_diffs?(diffs)
- return false unless diffs.respond_to?(:each)
-
- diffs.all? { |diff| diff.is_a?(Hash) }
- end
- end
- end
-end
diff --git a/lib/gitlab/background_migration/merge_request_assignees_migration_progress_check.rb b/lib/gitlab/background_migration/merge_request_assignees_migration_progress_check.rb
new file mode 100644
index 00000000000..e948cedaad5
--- /dev/null
+++ b/lib/gitlab/background_migration/merge_request_assignees_migration_progress_check.rb
@@ -0,0 +1,43 @@
+# frozen_string_literal: true
+
+module Gitlab
+ module BackgroundMigration
+ # rubocop: disable Style/Documentation
+ class MergeRequestAssigneesMigrationProgressCheck
+ include Gitlab::Utils::StrongMemoize
+
+ RESCHEDULE_DELAY = 3.hours
+ WORKER = 'PopulateMergeRequestAssigneesTable'.freeze
+ DeadJobsError = Class.new(StandardError)
+
+ def perform
+ raise DeadJobsError, "Only dead background jobs in the queue for #{WORKER}" if !ongoing? && dead_jobs?
+
+ if ongoing?
+ BackgroundMigrationWorker.perform_in(RESCHEDULE_DELAY, self.class.name)
+ else
+ Feature.enable(:multiple_merge_request_assignees)
+ end
+ end
+
+ private
+
+ def dead_jobs?
+ strong_memoize(:dead_jobs) do
+ migration_klass.dead_jobs?(WORKER)
+ end
+ end
+
+ def ongoing?
+ strong_memoize(:ongoing) do
+ migration_klass.exists?(WORKER) || migration_klass.retrying_jobs?(WORKER)
+ end
+ end
+
+ def migration_klass
+ Gitlab::BackgroundMigration
+ end
+ end
+ # rubocop: enable Style/Documentation
+ end
+end
diff --git a/lib/gitlab/background_migration/populate_external_pipeline_source.rb b/lib/gitlab/background_migration/populate_external_pipeline_source.rb
deleted file mode 100644
index 036fe641757..00000000000
--- a/lib/gitlab/background_migration/populate_external_pipeline_source.rb
+++ /dev/null
@@ -1,50 +0,0 @@
-# frozen_string_literal: true
-# rubocop:disable Style/Documentation
-
-module Gitlab
- module BackgroundMigration
- class PopulateExternalPipelineSource
- module Migratable
- class Pipeline < ActiveRecord::Base
- self.table_name = 'ci_pipelines'
-
- def self.sources
- {
- unknown: nil,
- push: 1,
- web: 2,
- trigger: 3,
- schedule: 4,
- api: 5,
- external: 6
- }
- end
- end
-
- class CommitStatus < ActiveRecord::Base
- self.table_name = 'ci_builds'
- self.inheritance_column = :_type_disabled
-
- scope :has_pipeline, -> { where('ci_builds.commit_id=ci_pipelines.id') }
- scope :of_type, -> (type) { where('type=?', type) }
- end
- end
-
- def perform(start_id, stop_id)
- external_pipelines(start_id, stop_id)
- .update_all(source: Migratable::Pipeline.sources[:external])
- end
-
- private
-
- def external_pipelines(start_id, stop_id)
- Migratable::Pipeline.where(id: (start_id..stop_id))
- .where(
- 'EXISTS (?) AND NOT EXISTS (?)',
- Migratable::CommitStatus.of_type('GenericCommitStatus').has_pipeline.select(1),
- Migratable::CommitStatus.of_type('Ci::Build').has_pipeline.select(1)
- )
- end
- end
- end
-end
diff --git a/lib/gitlab/background_migration/populate_import_state.rb b/lib/gitlab/background_migration/populate_import_state.rb
deleted file mode 100644
index 695a2a713c5..00000000000
--- a/lib/gitlab/background_migration/populate_import_state.rb
+++ /dev/null
@@ -1,39 +0,0 @@
-# frozen_string_literal: true
-
-module Gitlab
- module BackgroundMigration
- # This background migration creates all the records on the
- # import state table for projects that are considered imports or forks
- class PopulateImportState
- def perform(start_id, end_id)
- move_attributes_data_to_import_state(start_id, end_id)
- rescue ActiveRecord::RecordNotUnique
- retry
- end
-
- def move_attributes_data_to_import_state(start_id, end_id)
- Rails.logger.info("#{self.class.name} - Moving import attributes data to project mirror data table: #{start_id} - #{end_id}")
-
- ActiveRecord::Base.connection.execute <<~SQL
- INSERT INTO project_mirror_data (project_id, status, jid, last_error)
- SELECT id, import_status, import_jid, import_error
- FROM projects
- WHERE projects.import_status != 'none'
- AND projects.id BETWEEN #{start_id} AND #{end_id}
- AND NOT EXISTS (
- SELECT id
- FROM project_mirror_data
- WHERE project_id = projects.id
- )
- SQL
-
- ActiveRecord::Base.connection.execute <<~SQL
- UPDATE projects
- SET import_status = 'none'
- WHERE import_status != 'none'
- AND id BETWEEN #{start_id} AND #{end_id}
- SQL
- end
- end
- end
-end
diff --git a/lib/gitlab/background_migration/populate_merge_request_metrics_with_events_data.rb b/lib/gitlab/background_migration/populate_merge_request_metrics_with_events_data.rb
deleted file mode 100644
index d89ce358bb9..00000000000
--- a/lib/gitlab/background_migration/populate_merge_request_metrics_with_events_data.rb
+++ /dev/null
@@ -1,132 +0,0 @@
-# frozen_string_literal: true
-# rubocop:disable Style/Documentation
-
-module Gitlab
- module BackgroundMigration
- class PopulateMergeRequestMetricsWithEventsData
- def perform(min_merge_request_id, max_merge_request_id)
- insert_metrics_for_range(min_merge_request_id, max_merge_request_id)
- update_metrics_with_events_data(min_merge_request_id, max_merge_request_id)
- end
-
- # Inserts merge_request_metrics records for merge_requests without it for
- # a given merge request batch.
- def insert_metrics_for_range(min, max)
- metrics_not_exists_clause =
- <<-SQL.strip_heredoc
- NOT EXISTS (SELECT 1 FROM merge_request_metrics
- WHERE merge_request_metrics.merge_request_id = merge_requests.id)
- SQL
-
- MergeRequest.where(metrics_not_exists_clause).where(id: min..max).each_batch do |batch|
- select_sql = batch.select(:id, :created_at, :updated_at).to_sql
-
- execute("INSERT INTO merge_request_metrics (merge_request_id, created_at, updated_at) #{select_sql}")
- end
- end
-
- def update_metrics_with_events_data(min, max)
- if Gitlab::Database.postgresql?
- # Uses WITH syntax in order to update merged and closed events with a single UPDATE.
- # WITH is not supported by MySQL.
- update_events_for_range(min, max)
- else
- update_merged_events_for_range(min, max)
- update_closed_events_for_range(min, max)
- end
- end
-
- private
-
- # Updates merge_request_metrics latest_closed_at, latest_closed_by_id and merged_by_id
- # based on the latest event records on events table for a given merge request batch.
- def update_events_for_range(min, max)
- sql = <<-SQL.strip_heredoc
- WITH events_for_update AS (
- SELECT DISTINCT ON (target_id, action) target_id, action, author_id, updated_at
- FROM events
- WHERE target_id BETWEEN #{min} AND #{max}
- AND target_type = 'MergeRequest'
- AND action IN (#{Event::CLOSED},#{Event::MERGED})
- ORDER BY target_id, action, id DESC
- )
- UPDATE merge_request_metrics met
- SET latest_closed_at = latest_closed.updated_at,
- latest_closed_by_id = latest_closed.author_id,
- merged_by_id = latest_merged.author_id
- FROM (SELECT * FROM events_for_update WHERE action = #{Event::CLOSED}) AS latest_closed
- FULL OUTER JOIN
- (SELECT * FROM events_for_update WHERE action = #{Event::MERGED}) AS latest_merged
- USING (target_id)
- WHERE target_id = merge_request_id;
- SQL
-
- execute(sql)
- end
-
- # Updates merge_request_metrics latest_closed_at, latest_closed_by_id based on the latest closed
- # records on events table for a given merge request batch.
- def update_closed_events_for_range(min, max)
- sql =
- <<-SQL.strip_heredoc
- UPDATE merge_request_metrics metrics,
- (#{select_events(min, max, Event::CLOSED)}) closed_events
- SET metrics.latest_closed_by_id = closed_events.author_id,
- metrics.latest_closed_at = closed_events.updated_at #{where_matches_closed_events};
- SQL
-
- execute(sql)
- end
-
- # Updates merge_request_metrics merged_by_id based on the latest merged
- # records on events table for a given merge request batch.
- def update_merged_events_for_range(min, max)
- sql =
- <<-SQL.strip_heredoc
- UPDATE merge_request_metrics metrics,
- (#{select_events(min, max, Event::MERGED)}) merged_events
- SET metrics.merged_by_id = merged_events.author_id #{where_matches_merged_events};
- SQL
-
- execute(sql)
- end
-
- def execute(sql)
- @connection ||= ActiveRecord::Base.connection
- @connection.execute(sql)
- end
-
- def select_events(min, max, action)
- select_max_event_id = <<-SQL.strip_heredoc
- SELECT max(id)
- FROM events
- WHERE action = #{action}
- AND target_type = 'MergeRequest'
- AND target_id BETWEEN #{min} AND #{max}
- GROUP BY target_id
- SQL
-
- <<-SQL.strip_heredoc
- SELECT author_id, updated_at, target_id
- FROM events
- WHERE id IN(#{select_max_event_id})
- SQL
- end
-
- def where_matches_closed_events
- <<-SQL.strip_heredoc
- WHERE metrics.merge_request_id = closed_events.target_id
- AND metrics.latest_closed_at IS NULL
- AND metrics.latest_closed_by_id IS NULL
- SQL
- end
-
- def where_matches_merged_events
- <<-SQL.strip_heredoc
- WHERE metrics.merge_request_id = merged_events.target_id
- AND metrics.merged_by_id IS NULL
- SQL
- end
- end
- end
-end
diff --git a/lib/gitlab/background_migration/populate_merge_request_metrics_with_events_data_improved.rb b/lib/gitlab/background_migration/populate_merge_request_metrics_with_events_data_improved.rb
deleted file mode 100644
index 37592d67dd9..00000000000
--- a/lib/gitlab/background_migration/populate_merge_request_metrics_with_events_data_improved.rb
+++ /dev/null
@@ -1,99 +0,0 @@
-# frozen_string_literal: true
-# rubocop:disable Style/Documentation
-
-module Gitlab
- module BackgroundMigration
- class PopulateMergeRequestMetricsWithEventsDataImproved
- CLOSED_EVENT_ACTION = 3
- MERGED_EVENT_ACTION = 7
-
- def perform(min_merge_request_id, max_merge_request_id)
- insert_metrics_for_range(min_merge_request_id, max_merge_request_id)
- update_metrics_with_events_data(min_merge_request_id, max_merge_request_id)
- end
-
- # Inserts merge_request_metrics records for merge_requests without it for
- # a given merge request batch.
- def insert_metrics_for_range(min, max)
- metrics_not_exists_clause =
- <<-SQL.strip_heredoc
- NOT EXISTS (SELECT 1 FROM merge_request_metrics
- WHERE merge_request_metrics.merge_request_id = merge_requests.id)
- SQL
-
- MergeRequest.where(metrics_not_exists_clause).where(id: min..max).each_batch do |batch|
- select_sql = batch.select(:id, :created_at, :updated_at).to_sql
-
- execute("INSERT INTO merge_request_metrics (merge_request_id, created_at, updated_at) #{select_sql}")
- end
- end
-
- def update_metrics_with_events_data(min, max)
- if Gitlab::Database.postgresql?
- psql_update_metrics_with_events_data(min, max)
- else
- mysql_update_metrics_with_events_data(min, max)
- end
- end
-
- def psql_update_metrics_with_events_data(min, max)
- update_sql = <<-SQL.strip_heredoc
- UPDATE merge_request_metrics
- SET (latest_closed_at,
- latest_closed_by_id) =
- ( SELECT updated_at,
- author_id
- FROM events
- WHERE target_id = merge_request_id
- AND target_type = 'MergeRequest'
- AND action = #{CLOSED_EVENT_ACTION}
- ORDER BY id DESC
- LIMIT 1 ),
- merged_by_id =
- ( SELECT author_id
- FROM events
- WHERE target_id = merge_request_id
- AND target_type = 'MergeRequest'
- AND action = #{MERGED_EVENT_ACTION}
- ORDER BY id DESC
- LIMIT 1 )
- WHERE merge_request_id BETWEEN #{min} AND #{max}
- SQL
-
- execute(update_sql)
- end
-
- def mysql_update_metrics_with_events_data(min, max)
- closed_updated_at_subquery = mysql_events_select(:updated_at, CLOSED_EVENT_ACTION)
- closed_author_id_subquery = mysql_events_select(:author_id, CLOSED_EVENT_ACTION)
- merged_author_id_subquery = mysql_events_select(:author_id, MERGED_EVENT_ACTION)
-
- update_sql = <<-SQL.strip_heredoc
- UPDATE merge_request_metrics
- SET latest_closed_at = (#{closed_updated_at_subquery}),
- latest_closed_by_id = (#{closed_author_id_subquery}),
- merged_by_id = (#{merged_author_id_subquery})
- WHERE merge_request_id BETWEEN #{min} AND #{max}
- SQL
-
- execute(update_sql)
- end
-
- def mysql_events_select(column, action)
- <<-SQL.strip_heredoc
- SELECT #{column} FROM events
- WHERE target_id = merge_request_id
- AND target_type = 'MergeRequest'
- AND action = #{action}
- ORDER BY id DESC
- LIMIT 1
- SQL
- end
-
- def execute(sql)
- @connection ||= ActiveRecord::Base.connection
- @connection.execute(sql)
- end
- end
- end
-end
diff --git a/lib/gitlab/background_migration/redact_links.rb b/lib/gitlab/background_migration/redact_links.rb
deleted file mode 100644
index 92256e59a6c..00000000000
--- a/lib/gitlab/background_migration/redact_links.rb
+++ /dev/null
@@ -1,51 +0,0 @@
-# frozen_string_literal: true
-# rubocop:disable Style/Documentation
-
-require_relative 'redact_links/redactable'
-
-module Gitlab
- module BackgroundMigration
- class RedactLinks
- class Note < ActiveRecord::Base
- include EachBatch
- include ::Gitlab::BackgroundMigration::RedactLinks::Redactable
-
- self.table_name = 'notes'
- self.inheritance_column = :_type_disabled
- end
-
- class Issue < ActiveRecord::Base
- include EachBatch
- include ::Gitlab::BackgroundMigration::RedactLinks::Redactable
-
- self.table_name = 'issues'
- self.inheritance_column = :_type_disabled
- end
-
- class MergeRequest < ActiveRecord::Base
- include EachBatch
- include ::Gitlab::BackgroundMigration::RedactLinks::Redactable
-
- self.table_name = 'merge_requests'
- self.inheritance_column = :_type_disabled
- end
-
- class Snippet < ActiveRecord::Base
- include EachBatch
- include ::Gitlab::BackgroundMigration::RedactLinks::Redactable
-
- self.table_name = 'snippets'
- self.inheritance_column = :_type_disabled
- end
-
- def perform(model_name, field, start_id, stop_id)
- link_pattern = "%/sent_notifications/" + ("_" * 32) + "/unsubscribe%"
- model = "Gitlab::BackgroundMigration::RedactLinks::#{model_name}".constantize
-
- model.where("#{field} like ?", link_pattern).where(id: start_id..stop_id).each do |resource|
- resource.redact_field!(field)
- end
- end
- end
- end
-end
diff --git a/lib/gitlab/background_migration/redact_links/redactable.rb b/lib/gitlab/background_migration/redact_links/redactable.rb
deleted file mode 100644
index baab34221f1..00000000000
--- a/lib/gitlab/background_migration/redact_links/redactable.rb
+++ /dev/null
@@ -1,21 +0,0 @@
-# frozen_string_literal: true
-# rubocop:disable Style/Documentation
-
-module Gitlab
- module BackgroundMigration
- class RedactLinks
- module Redactable
- extend ActiveSupport::Concern
-
- def redact_field!(field)
- self[field].gsub!(%r{/sent_notifications/\h{32}/unsubscribe}, '/sent_notifications/REDACTED/unsubscribe')
-
- if self.changed?
- self.update_columns(field => self[field],
- "#{field}_html" => nil)
- end
- end
- end
- end
- end
-end
diff --git a/lib/gitlab/background_migration/rollback_import_state_data.rb b/lib/gitlab/background_migration/rollback_import_state_data.rb
deleted file mode 100644
index a7c986747d8..00000000000
--- a/lib/gitlab/background_migration/rollback_import_state_data.rb
+++ /dev/null
@@ -1,40 +0,0 @@
-# frozen_string_literal: true
-
-module Gitlab
- module BackgroundMigration
- # This background migration migrates all the data of import_state
- # back to the projects table for projects that are considered imports or forks
- class RollbackImportStateData
- def perform(start_id, end_id)
- move_attributes_data_to_project(start_id, end_id)
- end
-
- def move_attributes_data_to_project(start_id, end_id)
- Rails.logger.info("#{self.class.name} - Moving import attributes data to projects table: #{start_id} - #{end_id}")
-
- if Gitlab::Database.mysql?
- ActiveRecord::Base.connection.execute <<~SQL
- UPDATE projects, project_mirror_data
- SET
- projects.import_status = project_mirror_data.status,
- projects.import_jid = project_mirror_data.jid,
- projects.import_error = project_mirror_data.last_error
- WHERE project_mirror_data.project_id = projects.id
- AND project_mirror_data.id BETWEEN #{start_id} AND #{end_id}
- SQL
- else
- ActiveRecord::Base.connection.execute <<~SQL
- UPDATE projects
- SET
- import_status = project_mirror_data.status,
- import_jid = project_mirror_data.jid,
- import_error = project_mirror_data.last_error
- FROM project_mirror_data
- WHERE project_mirror_data.project_id = projects.id
- AND project_mirror_data.id BETWEEN #{start_id} AND #{end_id}
- SQL
- end
- end
- end
- end
-end
diff --git a/lib/gitlab/background_migration/schedule_diff_files_deletion.rb b/lib/gitlab/background_migration/schedule_diff_files_deletion.rb
deleted file mode 100644
index 609cf19187c..00000000000
--- a/lib/gitlab/background_migration/schedule_diff_files_deletion.rb
+++ /dev/null
@@ -1,44 +0,0 @@
-# frozen_string_literal: true
-# rubocop:disable Style/Documentation
-
-module Gitlab
- module BackgroundMigration
- class ScheduleDiffFilesDeletion
- class MergeRequestDiff < ActiveRecord::Base
- self.table_name = 'merge_request_diffs'
-
- belongs_to :merge_request
-
- include EachBatch
- end
-
- DIFF_BATCH_SIZE = 5_000
- INTERVAL = 5.minutes
- MIGRATION = 'DeleteDiffFiles'
-
- def perform
- diffs = MergeRequestDiff
- .from("(#{diffs_collection.to_sql}) merge_request_diffs")
- .where('merge_request_diffs.id != merge_request_diffs.latest_merge_request_diff_id')
- .select(:id)
-
- diffs.each_batch(of: DIFF_BATCH_SIZE) do |relation, index|
- ids = relation.pluck(:id)
-
- BackgroundMigrationWorker.perform_in(index * INTERVAL, MIGRATION, [ids])
- end
- end
-
- private
-
- def diffs_collection
- MergeRequestDiff
- .joins(:merge_request)
- .where("merge_requests.state = 'merged'")
- .where('merge_requests.latest_merge_request_diff_id IS NOT NULL')
- .where("merge_request_diffs.state NOT IN ('without_files', 'empty')")
- .select('merge_requests.latest_merge_request_diff_id, merge_request_diffs.id')
- end
- end
- end
-end