diff options
author | GitLab Bot <gitlab-bot@gitlab.com> | 2022-10-05 06:09:43 +0300 |
---|---|---|
committer | GitLab Bot <gitlab-bot@gitlab.com> | 2022-10-05 06:09:43 +0300 |
commit | b1646969577dbafca1b5936c3aa9535ae17d8558 (patch) | |
tree | 7b8c7d67a447efe0daee4aa26bd8011bf5d59433 /doc/development | |
parent | 59429d48eb1cf7032cf12363b83a045743f02a1e (diff) |
Add latest changes from gitlab-org/gitlab@master
Diffstat (limited to 'doc/development')
-rw-r--r-- | doc/development/code_review.md | 7 | ||||
-rw-r--r-- | doc/development/database/add_foreign_key_to_existing_column.md | 2 | ||||
-rw-r--r-- | doc/development/database/avoiding_downtime_in_migrations.md | 2 | ||||
-rw-r--r-- | doc/development/database/batched_background_migrations.md | 5 | ||||
-rw-r--r-- | doc/development/database/index.md | 3 | ||||
-rw-r--r-- | doc/development/database/iterating_tables_in_batches.md | 2 | ||||
-rw-r--r-- | doc/development/database/not_null_constraints.md | 2 | ||||
-rw-r--r-- | doc/development/database/strings_and_the_text_data_type.md | 23 | ||||
-rw-r--r-- | doc/development/database_review.md | 6 | ||||
-rw-r--r-- | doc/development/migration_style_guide.md | 4 |
10 files changed, 28 insertions, 28 deletions
diff --git a/doc/development/code_review.md b/doc/development/code_review.md index 01e0dce8358..bdd2c424687 100644 --- a/doc/development/code_review.md +++ b/doc/development/code_review.md @@ -662,9 +662,10 @@ Enterprise Edition instance. This has some implications: - Regular migrations run before the new code is running on the instance. - [Post-deployment migrations](database/post_deployment_migrations.md) run _after_ the new code is deployed, when the instance is configured to do that. - - [Background migrations](database/background_migrations.md) run in Sidekiq, and - should only be done for migrations that would take an extreme amount of - time at GitLab.com scale. + - [Batched background migrations](database/batched_background_migrations.md) run in Sidekiq, and + should be used for migrations that + [exceed the post-deployment migration time limit](migration_style_guide.md#how-long-a-migration-should-take) + GitLab.com scale. 1. **Sidekiq workers** [cannot change in a backwards-incompatible way](sidekiq/compatibility_across_updates.md): 1. Sidekiq queues are not drained before a deploy happens, so there are workers in the queue from the previous version of GitLab. diff --git a/doc/development/database/add_foreign_key_to_existing_column.md b/doc/development/database/add_foreign_key_to_existing_column.md index c92537859e1..1609e00531e 100644 --- a/doc/development/database/add_foreign_key_to_existing_column.md +++ b/doc/development/database/add_foreign_key_to_existing_column.md @@ -126,7 +126,7 @@ Validating the foreign key scans the whole table and makes sure that each relati Fortunately, this does not lock the source table (`users`) while running. NOTE: -When using [background migrations](background_migrations.md), foreign key validation should happen in the next GitLab release. +When using [batched background migrations](batched_background_migrations.md), foreign key validation should happen in the next GitLab release. Migration file for validating the foreign key: diff --git a/doc/development/database/avoiding_downtime_in_migrations.md b/doc/development/database/avoiding_downtime_in_migrations.md index bd17133fea1..57f5a66a9ee 100644 --- a/doc/development/database/avoiding_downtime_in_migrations.md +++ b/doc/development/database/avoiding_downtime_in_migrations.md @@ -296,7 +296,7 @@ when migrating a column in a large table (for example, `issues`). Background migrations spread the work / load over a longer time period, without slowing down deployments. -For more information, see [the documentation on cleaning up background migrations](background_migrations.md#cleaning-up). +For more information, see [the documentation on cleaning up batched background migrations](batched_background_migrations.md#cleaning-up). ## Adding Indexes diff --git a/doc/development/database/batched_background_migrations.md b/doc/development/database/batched_background_migrations.md index ea1de91c26c..947e92bf83f 100644 --- a/doc/development/database/batched_background_migrations.md +++ b/doc/development/database/batched_background_migrations.md @@ -670,3 +670,8 @@ You can view failures in two ways: ON transition_logs.batched_background_migration_job_id = jobs.id WHERE transition_logs.next_status = '2' AND migration.job_class_name = "CLASS_NAME"; ``` + +## Legacy background migrations + +Batched background migrations replaced the [legacy background migrations framework](background_migrations.md). +Check that documentation in reference to any changes involving that framework. diff --git a/doc/development/database/index.md b/doc/development/database/index.md index 0fc49e58faa..87b1b4a9d87 100644 --- a/doc/development/database/index.md +++ b/doc/development/database/index.md @@ -26,7 +26,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w - [Different types of migrations](../migration_style_guide.md#choose-an-appropriate-migration-type) - [Create a regular migration](../migration_style_guide.md#create-a-regular-schema-migration), including creating new models - [Post-deployment migrations guidelines](post_deployment_migrations.md) and [how to create one](post_deployment_migrations.md#creating-migrations) -- [Background migrations guidelines](background_migrations.md) +- [Legacy Background migrations guidelines](background_migrations.md) - [Batched background migrations guidelines](batched_background_migrations.md) - [Deleting migrations](deleting_migrations.md) - [Running database migrations](database_debugging.md#migration-wrangling) @@ -36,7 +36,6 @@ info: To determine the technical writer assigned to the Stage/Group associated w - [Migrations style guide](../migration_style_guide.md) for creating safe SQL migrations - [Testing Rails migrations](../testing_guide/testing_migrations_guide.md) guide - [Post deployment migrations](post_deployment_migrations.md) -- [Background migrations](background_migrations.md) - [Swapping tables](swapping_tables.md) - [Deleting migrations](deleting_migrations.md) - [SQL guidelines](../sql.md) for working with SQL queries diff --git a/doc/development/database/iterating_tables_in_batches.md b/doc/development/database/iterating_tables_in_batches.md index 69139d85f44..73e9d08bf57 100644 --- a/doc/development/database/iterating_tables_in_batches.md +++ b/doc/development/database/iterating_tables_in_batches.md @@ -114,7 +114,7 @@ falling into an endless loop as described in following When dealing with data migrations the preferred way to iterate over a large volume of data is using `EachBatch`. -A special case of data migration is a [background migration](background_migrations.md#scheduling) +A special case of data migration is a [batched background migration](batched_background_migrations.md) where the actual data modification is executed in a background job. The migration code that determines the data ranges (slices) and schedules the background jobs uses `each_batch`. diff --git a/doc/development/database/not_null_constraints.md b/doc/development/database/not_null_constraints.md index a8bae22fcd3..9e11db074cb 100644 --- a/doc/development/database/not_null_constraints.md +++ b/doc/development/database/not_null_constraints.md @@ -202,7 +202,7 @@ end If you have to clean up a nullable column for a [high-traffic table](../migration_style_guide.md#high-traffic-tables) (for example, the `artifacts` in `ci_builds`), your background migration goes on for a while and -it needs an additional [background migration cleaning up](background_migrations.md#cleaning-up) +it needs an additional [batched background migration cleaning up](batched_background_migrations.md#cleaning-up) in the release after adding the data migration. In that rare case you need 3 releases end-to-end: diff --git a/doc/development/database/strings_and_the_text_data_type.md b/doc/development/database/strings_and_the_text_data_type.md index 097e378d066..a431268f89a 100644 --- a/doc/development/database/strings_and_the_text_data_type.md +++ b/doc/development/database/strings_and_the_text_data_type.md @@ -196,7 +196,7 @@ to add a background migration for the 13.0 milestone (current), `db/post_migrate/20200501000002_schedule_cap_title_length_on_issues.rb`: ```ruby -class ScheduleCapTitleLengthOnIssues < Gitlab::Database::Migration[1.0] +class ScheduleCapTitleLengthOnIssues < Gitlab::Database::Migration[2.0] # Info on how many records will be affected on GitLab.com # time each batch needs to run on average, etc ... BATCH_SIZE = 5000 @@ -207,30 +207,25 @@ class ScheduleCapTitleLengthOnIssues < Gitlab::Database::Migration[1.0] disable_ddl_transaction! - class Issue < ::ApplicationRecord - include EachBatch - - self.table_name = 'issues' - end - def up - queue_background_migration_jobs_by_range_at_intervals( - Issue.where('char_length(title_html) > 1024'), - ISSUES_MIGRATION, - DELAY_INTERVAL, + queue_batched_background_migration( + ISSUES_BACKGROUND_MIGRATION, + :issues, + :id, + job_interval: DELAY_INTERVAL, batch_size: BATCH_SIZE ) end def down - # no-op : the part of the title_html after the limit is lost forever + delete_batched_background_migration(ISSUES_BACKGROUND_MIGRATION, :issues, :id, []) end end ``` To keep this guide short, we skipped the definition of the background migration and only provided a high level example of the post-deployment migration that is used to schedule the batches. -You can find more information on the guide about [background migrations](background_migrations.md) +You can find more information on the guide about [batched background migrations](batched_background_migrations.md) #### Validate the text limit (next release) @@ -278,7 +273,7 @@ end If you have to clean up a text column for a really [large table](https://gitlab.com/gitlab-org/gitlab/-/blob/master/rubocop/rubocop-migrations.yml#L3) (for example, the `artifacts` in `ci_builds`), your background migration goes on for a while and -it needs an additional [background migration cleaning up](background_migrations.md#cleaning-up) +it needs an additional [batched background migration cleaning up](batched_background_migrations.md#cleaning-up) in the release after adding the data migration. In that rare case you need 3 releases end-to-end: diff --git a/doc/development/database_review.md b/doc/development/database_review.md index 0b267f642ce..58776c5330c 100644 --- a/doc/development/database_review.md +++ b/doc/development/database_review.md @@ -240,9 +240,11 @@ Include in the MR description: - Check queries timing (If any): In a single transaction, cumulative query time executed in a migration needs to fit comfortably in `15s` - preferably much less than that - on GitLab.com. - For column removals, make sure the column has been [ignored in a previous release](database/avoiding_downtime_in_migrations.md#dropping-columns) -- Check [background migrations](database/background_migrations.md): +- Check [batched background migrations](database/batched_background_migrations.md): - Establish a time estimate for execution on GitLab.com. For historical purposes, it's highly recommended to include this estimation on the merge request description. + This can be the number of expected batches times the delay interval. + - Manually trigger the [database testing](database/database_migration_pipeline.md) job (`db:gitlabcom-database-testing`) in the `test` stage. - If a single `update` is below than `1s` the query can be placed directly in a regular migration (inside `db/migrate`). - Background migrations are normally used, but not limited to: @@ -253,8 +255,6 @@ Include in the MR description: it's suggested to treat background migrations as [post migrations](migration_style_guide.md#choose-an-appropriate-migration-type): place them in `db/post_migrate` instead of `db/migrate`. - - If a migration [has tracking enabled](database/background_migrations.md#background-jobs-tracking), - ensure `mark_all_as_succeeded` is called even if no work is done. - Check [timing guidelines for migrations](migration_style_guide.md#how-long-a-migration-should-take) - Check migrations are reversible and implement a `#down` method - Check new table migrations: diff --git a/doc/development/migration_style_guide.md b/doc/development/migration_style_guide.md index c3b2c178d2b..34ffdda9fa3 100644 --- a/doc/development/migration_style_guide.md +++ b/doc/development/migration_style_guide.md @@ -275,7 +275,7 @@ in that limit. Singular query timings should fit within the [standard limit](dat In case you need to insert, update, or delete a significant amount of data, you: - Must disable the single transaction with `disable_ddl_transaction!`. -- Should consider doing it in a [Background Migration](database/background_migrations.md). +- Should consider doing it in a [batched background migration](database/batched_background_migrations.md). ## Migration helpers and versioning @@ -1266,7 +1266,7 @@ by an integer. For example: `users` would turn into `users0` ## Using models in migrations (discouraged) The use of models in migrations is generally discouraged. As such models are -[contraindicated for background migrations](database/background_migrations.md#isolation), +[contraindicated for batched background migrations](database/batched_background_migrations.md#isolation), the model needs to be declared in the migration. If using a model in the migrations, you should first |