Add latest changes from gitlab-org/gitlab@master

11 files changed, 29 insertions, 34 deletions

diff --git a/doc/development/code_review.md b/doc/development/code_review.md
index 01e0dce8358..bdd2c424687 100644
--- a/doc/development/code_review.md
+++ b/doc/development/code_review.md
@@ -662,9 +662,10 @@ Enterprise Edition instance. This has some implications:
       - Regular migrations run before the new code is running on the instance.
       - [Post-deployment migrations](database/post_deployment_migrations.md) run _after_
         the new code is deployed, when the instance is configured to do that.
-      - [Background migrations](database/background_migrations.md) run in Sidekiq, and
-        should only be done for migrations that would take an extreme amount of
-        time at GitLab.com scale.
+      - [Batched background migrations](database/batched_background_migrations.md) run in Sidekiq, and
+        should be used for migrations that
+        [exceed the post-deployment migration time limit](migration_style_guide.md#how-long-a-migration-should-take)
+        GitLab.com scale.
 1. **Sidekiq workers** [cannot change in a backwards-incompatible way](sidekiq/compatibility_across_updates.md):
    1. Sidekiq queues are not drained before a deploy happens, so there are
       workers in the queue from the previous version of GitLab.
diff --git a/doc/development/database/add_foreign_key_to_existing_column.md b/doc/development/database/add_foreign_key_to_existing_column.md
index c92537859e1..1609e00531e 100644
--- a/doc/development/database/add_foreign_key_to_existing_column.md
+++ b/doc/development/database/add_foreign_key_to_existing_column.md
@@ -126,7 +126,7 @@ Validating the foreign key scans the whole table and makes sure that each relati
 Fortunately, this does not lock the source table (`users`) while running.
 
 NOTE:
-When using [background migrations](background_migrations.md), foreign key validation should happen in the next GitLab release.
+When using [batched background migrations](batched_background_migrations.md), foreign key validation should happen in the next GitLab release.
 
 Migration file for validating the foreign key:
 
diff --git a/doc/development/database/avoiding_downtime_in_migrations.md b/doc/development/database/avoiding_downtime_in_migrations.md
index bd17133fea1..57f5a66a9ee 100644
--- a/doc/development/database/avoiding_downtime_in_migrations.md
+++ b/doc/development/database/avoiding_downtime_in_migrations.md
@@ -296,7 +296,7 @@ when migrating a column in a large table (for example, `issues`). Background
 migrations spread the work / load over a longer time period, without slowing
 down deployments.
 
-For more information, see [the documentation on cleaning up background migrations](background_migrations.md#cleaning-up).
+For more information, see [the documentation on cleaning up batched background migrations](batched_background_migrations.md#cleaning-up).
 
 ## Adding Indexes
 
diff --git a/doc/development/database/batched_background_migrations.md b/doc/development/database/batched_background_migrations.md
index ea1de91c26c..947e92bf83f 100644
--- a/doc/development/database/batched_background_migrations.md
+++ b/doc/development/database/batched_background_migrations.md
@@ -670,3 +670,8 @@ You can view failures in two ways:
       ON transition_logs.batched_background_migration_job_id = jobs.id
       WHERE transition_logs.next_status = '2' AND migration.job_class_name = "CLASS_NAME";
      ```
+
+## Legacy background migrations
+
+Batched background migrations replaced the [legacy background migrations framework](background_migrations.md).
+Check that documentation in reference to any changes involving that framework.
diff --git a/doc/development/database/index.md b/doc/development/database/index.md
index 0fc49e58faa..87b1b4a9d87 100644
--- a/doc/development/database/index.md
+++ b/doc/development/database/index.md
@@ -26,7 +26,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w
 - [Different types of migrations](../migration_style_guide.md#choose-an-appropriate-migration-type)
 - [Create a regular migration](../migration_style_guide.md#create-a-regular-schema-migration), including creating new models
 - [Post-deployment migrations guidelines](post_deployment_migrations.md) and [how to create one](post_deployment_migrations.md#creating-migrations)
-- [Background migrations guidelines](background_migrations.md)
+- [Legacy Background migrations guidelines](background_migrations.md)
 - [Batched background migrations guidelines](batched_background_migrations.md)
 - [Deleting migrations](deleting_migrations.md)
 - [Running database migrations](database_debugging.md#migration-wrangling)
@@ -36,7 +36,6 @@ info: To determine the technical writer assigned to the Stage/Group associated w
 - [Migrations style guide](../migration_style_guide.md) for creating safe SQL migrations
 - [Testing Rails migrations](../testing_guide/testing_migrations_guide.md) guide
 - [Post deployment migrations](post_deployment_migrations.md)
-- [Background migrations](background_migrations.md)
 - [Swapping tables](swapping_tables.md)
 - [Deleting migrations](deleting_migrations.md)
 - [SQL guidelines](../sql.md) for working with SQL queries
diff --git a/doc/development/database/iterating_tables_in_batches.md b/doc/development/database/iterating_tables_in_batches.md
index 69139d85f44..73e9d08bf57 100644
--- a/doc/development/database/iterating_tables_in_batches.md
+++ b/doc/development/database/iterating_tables_in_batches.md
@@ -114,7 +114,7 @@ falling into an endless loop as described in following
 When dealing with data migrations the preferred way to iterate over a large volume of data is using
 `EachBatch`.
 
-A special case of data migration is a [background migration](background_migrations.md#scheduling)
+A special case of data migration is a [batched background migration](batched_background_migrations.md)
 where the actual data modification is executed in a background job. The migration code that
 determines the data ranges (slices) and schedules the background jobs uses `each_batch`.
 
diff --git a/doc/development/database/not_null_constraints.md b/doc/development/database/not_null_constraints.md
index a8bae22fcd3..9e11db074cb 100644
--- a/doc/development/database/not_null_constraints.md
+++ b/doc/development/database/not_null_constraints.md
@@ -202,7 +202,7 @@ end
 
 If you have to clean up a nullable column for a [high-traffic table](../migration_style_guide.md#high-traffic-tables)
 (for example, the `artifacts` in `ci_builds`), your background migration goes on for a while and
-it needs an additional [background migration cleaning up](background_migrations.md#cleaning-up)
+it needs an additional [batched background migration cleaning up](batched_background_migrations.md#cleaning-up)
 in the release after adding the data migration.
 
 In that rare case you need 3 releases end-to-end:
diff --git a/doc/development/database/strings_and_the_text_data_type.md b/doc/development/database/strings_and_the_text_data_type.md
index 097e378d066..a431268f89a 100644
--- a/doc/development/database/strings_and_the_text_data_type.md
+++ b/doc/development/database/strings_and_the_text_data_type.md
@@ -196,7 +196,7 @@ to add a background migration for the 13.0 milestone (current),
 `db/post_migrate/20200501000002_schedule_cap_title_length_on_issues.rb`:
 
 ```ruby
-class ScheduleCapTitleLengthOnIssues < Gitlab::Database::Migration[1.0]
+class ScheduleCapTitleLengthOnIssues < Gitlab::Database::Migration[2.0]
   # Info on how many records will be affected on GitLab.com
   # time each batch needs to run on average, etc ...
   BATCH_SIZE = 5000
@@ -207,30 +207,25 @@ class ScheduleCapTitleLengthOnIssues < Gitlab::Database::Migration[1.0]
 
   disable_ddl_transaction!
 
-  class Issue < ::ApplicationRecord
-    include EachBatch
-
-    self.table_name = 'issues'
-  end
-
   def up
-    queue_background_migration_jobs_by_range_at_intervals(
-      Issue.where('char_length(title_html) > 1024'),
-      ISSUES_MIGRATION,
-      DELAY_INTERVAL,
+    queue_batched_background_migration(
+      ISSUES_BACKGROUND_MIGRATION,
+      :issues,
+      :id,
+      job_interval: DELAY_INTERVAL,
       batch_size: BATCH_SIZE
     )
   end
 
   def down
-    # no-op : the part of the title_html after the limit is lost forever
+    delete_batched_background_migration(ISSUES_BACKGROUND_MIGRATION, :issues, :id, [])
   end
 end
 ```
 
 To keep this guide short, we skipped the definition of the background migration and only
 provided a high level example of the post-deployment migration that is used to schedule the batches.
-You can find more information on the guide about [background migrations](background_migrations.md)
+You can find more information on the guide about [batched background migrations](batched_background_migrations.md)
 
 #### Validate the text limit (next release)
 
@@ -278,7 +273,7 @@ end
 
 If you have to clean up a text column for a really [large table](https://gitlab.com/gitlab-org/gitlab/-/blob/master/rubocop/rubocop-migrations.yml#L3)
 (for example, the `artifacts` in `ci_builds`), your background migration goes on for a while and
-it needs an additional [background migration cleaning up](background_migrations.md#cleaning-up)
+it needs an additional [batched background migration cleaning up](batched_background_migrations.md#cleaning-up)
 in the release after adding the data migration.
 
 In that rare case you need 3 releases end-to-end:
diff --git a/doc/development/database_review.md b/doc/development/database_review.md
index 0b267f642ce..58776c5330c 100644
--- a/doc/development/database_review.md
+++ b/doc/development/database_review.md
@@ -240,9 +240,11 @@ Include in the MR description:
   - Check queries timing (If any): In a single transaction, cumulative query time executed in a migration
     needs to fit comfortably in `15s` - preferably much less than that - on GitLab.com.
   - For column removals, make sure the column has been [ignored in a previous release](database/avoiding_downtime_in_migrations.md#dropping-columns)
-- Check [background migrations](database/background_migrations.md):
+- Check [batched background migrations](database/batched_background_migrations.md):
   - Establish a time estimate for execution on GitLab.com. For historical purposes,
     it's highly recommended to include this estimation on the merge request description.
+    This can be the number of expected batches times the delay interval.
+  - Manually trigger the [database testing](database/database_migration_pipeline.md) job (`db:gitlabcom-database-testing`) in the `test` stage.
   - If a single `update` is below than `1s` the query can be placed
       directly in a regular migration (inside `db/migrate`).
   - Background migrations are normally used, but not limited to:
@@ -253,8 +255,6 @@ Include in the MR description:
     it's suggested to treat background migrations as
     [post migrations](migration_style_guide.md#choose-an-appropriate-migration-type):
     place them in `db/post_migrate` instead of `db/migrate`.
-  - If a migration [has tracking enabled](database/background_migrations.md#background-jobs-tracking),
-    ensure `mark_all_as_succeeded` is called even if no work is done.
 - Check [timing guidelines for migrations](migration_style_guide.md#how-long-a-migration-should-take)
 - Check migrations are reversible and implement a `#down` method
 - Check new table migrations:
diff --git a/doc/development/migration_style_guide.md b/doc/development/migration_style_guide.md
index c3b2c178d2b..34ffdda9fa3 100644
--- a/doc/development/migration_style_guide.md
+++ b/doc/development/migration_style_guide.md
@@ -275,7 +275,7 @@ in that limit. Singular query timings should fit within the [standard limit](dat
 In case you need to insert, update, or delete a significant amount of data, you:
 
 - Must disable the single transaction with `disable_ddl_transaction!`.
-- Should consider doing it in a [Background Migration](database/background_migrations.md).
+- Should consider doing it in a [batched background migration](database/batched_background_migrations.md).
 
 ## Migration helpers and versioning
 
@@ -1266,7 +1266,7 @@ by an integer. For example: `users` would turn into `users0`
 ## Using models in migrations (discouraged)
 
 The use of models in migrations is generally discouraged. As such models are
-[contraindicated for background migrations](database/background_migrations.md#isolation),
+[contraindicated for batched background migrations](database/batched_background_migrations.md#isolation),
 the model needs to be declared in the migration.
 
 If using a model in the migrations, you should first
diff --git a/doc/user/project/integrations/webhooks.md b/doc/user/project/integrations/webhooks.md
index b9035868a51..6857f50bb6e 100644
--- a/doc/user/project/integrations/webhooks.md
+++ b/doc/user/project/integrations/webhooks.md
@@ -236,12 +236,7 @@ For more information about supported events for Webhooks, go to [Webhook events]
 
 ## Delivery headers
 
-> `X-Gitlab-Instance` header introduced in GitLab 15.5 [with a flag](../../../administration/feature_flags.md) named `webhooks_gitlab_instance_header`. Disabled by default.
-
-FLAG:
-On self-managed GitLab, by default this feature is not available. To make it available,
-ask an administrator to [enable the feature flag](../../../administration/feature_flags.md) named `webhooks_gitlab_instance_header`.
-The feature is not ready for production use.
+> `X-Gitlab-Instance` header [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/31333) in GitLab 15.5.
 
 Webhook requests to your endpoint include the following headers: