diff options
Diffstat (limited to 'doc/update/background_migrations.md')
| -rw-r--r-- | doc/update/background_migrations.md | 365 |
1 files changed, 221 insertions, 144 deletions
diff --git a/doc/update/background_migrations.md b/doc/update/background_migrations.md index bf9f2df9e87..cbdcfef3bae 100644 --- a/doc/update/background_migrations.md +++ b/doc/update/background_migrations.md @@ -4,132 +4,119 @@ group: Database info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- -# Check for background migrations before upgrading +# Background migrations and upgrades **(FREE SELF)** -Certain releases may require different migrations to be -finished before you update to the newer version. +> - Batched background migrations [introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/51332) in GitLab 13.11 [with a flag](../user/feature_flags.md) named `execute_batched_migrations_on_schedule`. Disabled by default. +> - Feature flag `execute_batched_migrations_on_schedule` [enabled by default](https://gitlab.com/gitlab-org/gitlab/-/issues/329511) in GitLab 13.12. +> - For GitLab self-managed instances, GitLab administrators can opt to [disable it](../development/database/batched_background_migrations.md#enable-or-disable-background-migrations). -There are two kinds of migrations: +Certain releases may require different migrations to be finished before you +update to the newer version. Two kinds of migrations exist. They differ, and you +should check that both are complete before upgrading GitLab: -- [Background migrations](#background-migrations) -- [Batched background migrations](#batched-background-migrations) (available in GitLab 14.0 and later) +- [Batched background migrations](#batched-background-migrations), most + commonly used in GitLab 14.0 and later. +- [Background migrations](#background-migrations) that are not batched. + Most commonly used in GitLab 13.11 and earlier. -Background migrations and batched migrations are not the same, so you should check that both are -complete before updating. - -Decrease the time required to complete these migrations by increasing the number of +To decrease the time required to complete these migrations, increase the number of [Sidekiq workers](../administration/sidekiq/extra_sidekiq_processes.md) that can process jobs in the `background_migration` queue. -## Background migrations - -### Pending migrations - -**For Omnibus installations:** - -```shell -sudo gitlab-rails runner -e production 'puts Gitlab::BackgroundMigration.remaining' -sudo gitlab-rails runner -e production 'puts Gitlab::Database::BackgroundMigration::BatchedMigration.queued.count' -``` - -**For installations from source:** - -```shell -cd /home/git/gitlab -sudo -u git -H bundle exec rails runner -e production 'puts Gitlab::BackgroundMigration.remaining' -sudo -u git -H bundle exec rails runner -e production 'puts Gitlab::Database::BackgroundMigration::BatchedMigration.queued.count' -``` - -### Failed migrations - -**For Omnibus installations:** +## Batched background migrations -For GitLab 14.0-14.9: - -```shell -sudo gitlab-rails runner -e production 'puts Gitlab::Database::BackgroundMigration::BatchedMigration.failed.count' -``` - -For GitLab 14.10 and later: +To update database tables in batches, GitLab can use batched background migrations. These migrations +are created by GitLab developers and run automatically on upgrade. However, such migrations are +limited in scope to help with migrating some `integer` database columns to `bigint`. This is needed to +prevent integer overflow for some tables. -```shell -sudo gitlab-rails runner -e production 'puts Gitlab::Database::BackgroundMigration::BatchedMigration.with_status(:failed).count' -``` +Some installations [may need to run GitLab 14.0 for at least a day](index.md#1400) +to complete the database changes introduced by that upgrade. -**For installations from source:** +Batched background migrations are handled by Sidekiq and +[run in isolation](../development/database/batched_background_migrations.md#isolation), +so an instance can remain operational while the migrations are processed. However, +performance might degrade on larger instances that are heavily used while +batched background migrations are run. You should +[Actively monitor the Sidekiq status](../administration/admin_area.md#background-jobs) +until all migrations are completed. -For GitLab 14.0-14.9: +### Check the status of batched background migrations -```shell -cd /home/git/gitlab -sudo -u git -H bundle exec rails runner -e production 'puts Gitlab::Database::BackgroundMigration::BatchedMigration.failed.count' -``` +You can check the status of batched background migrations in the GitLab UI, or +by querying the database directly. Before you upgrade GitLab, all migrations must +have a `Finished` status. -For GitLab 14.10 and later: +If the migrations are not finished and you try to upgrade GitLab, you might +see this error: -```shell -cd /home/git/gitlab -sudo -u git -H bundle exec rails runner -e production 'puts Gitlab::Database::BackgroundMigration::BatchedMigration.with_status(:failed).count' +```plaintext +Expected batched background migration for the given configuration to be marked +as 'finished', but it is 'active': ``` -## Batched background migrations **(FREE SELF)** - -> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/51332) in GitLab 13.11, [behind a feature flag](../user/feature_flags.md), disabled by default. -> - [Enabled by default](https://gitlab.com/gitlab-org/gitlab/-/issues/329511) in GitLab 13.12. -> - Enabled on GitLab.com. -> - Recommended for production use. -> - For GitLab self-managed instances, GitLab administrators can opt to [disable it](#enable-or-disable-background-migrations). - -There can be [risks when disabling released features](../administration/feature_flags.md#risks-when-disabling-released-features). -Refer to this feature's version history for more details. - -To update database tables in batches, GitLab can use batched background migrations. These migrations -are created by GitLab developers and run automatically on upgrade. However, such migrations are -limited in scope to help with migrating some `integer` database columns to `bigint`. This is needed to -prevent integer overflow for some tables. +If you get this error, +[review the options](#database-migrations-failing-because-of-batched-background-migration-not-finished) for +how to complete the batched background migrations needed for the GitLab upgrade. -Some installations [may need to run GitLab 14.0 for at least a day](index.md#1400) to complete the database changes introduced by that upgrade. +#### From the GitLab UI -Batched background migrations are handled by Sidekiq and [run in isolation](../development/database/batched_background_migrations.md#isolation), so an instance can remain operational while the migrations are processed. However, there may be performance degradation on larger instances that are heavily used while batched background migrations are run, so it's a good idea to [actively monitor the Sidekiq status](../user/admin_area/index.md#background-jobs) until all migrations are completed. +Prerequisites: -### Check the status of batched background migrations +- You must have administrator access to the instance. To check the status of batched background migrations: 1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). 1. Select **Admin Area**. 1. Select **Monitoring > Background Migrations**. +1. Select **Queued** or **Finalizing** to see incomplete migrations, + and **Failed** for failed migrations. + +#### From the database -  +Prerequisites: -All migrations must have a `Finished` status before you upgrade GitLab. +- You must have administrator access to the instance. -The status of batched background migrations can also be queried directly in the database. +To query the database directly for the status of batched background migrations: -1. Log into a `psql` prompt according to the directions for your instance's installation method -(for example, `sudo gitlab-psql` for Omnibus installations). -1. Run the following query in the `psql` session to see details on incomplete batched background migrations: +1. Log into a `psql` prompt, according to the directions for your instance's + installation method. For example, `sudo gitlab-psql` for Linux package installations. +1. To see details on incomplete batched background migrations, run this query in + the `psql` session: ```sql - select job_class_name, table_name, column_name, job_arguments from batched_background_migrations where status <> 3; + SELECT + job_class_name, + table_name, + column_name, + job_arguments + FROM batched_background_migrations + WHERE status <> 3; ``` -If the migrations are not finished and you try to update to a later version, -GitLab prompts you with an error: +### Enable or disable advanced features -```plaintext -Expected batched background migration for the given configuration to be marked as 'finished', but it is 'active': -``` +Batched background migrations provide feature flags that enable you to customize +migrations or pause them entirely. These feature flags should only be disabled by +advanced users who understand the risks of doing so. -If you get this error, [check the batched background migration options](#database-migrations-failing-because-of-batched-background-migration-not-finished) to complete the upgrade. +#### Pause batched background migrations in GitLab 14.x -### Pause batched background migrations in GitLab 14.x +WARNING: +There can be [risks when disabling released features](../administration/feature_flags.md#risks-when-disabling-released-features). +Refer to each feature's version history for more details. To pause an ongoing batched background migration, -[disable the batched background migrations feature](#enable-or-disable-background-migrations). +[disable the batched background migrations feature](../development/database/batched_background_migrations.md#enable-or-disable-background-migrations). Disabling the feature completes the current batch of migrations, then waits to start the next batch until after the feature is enabled again. +Prerequisites: + +- You must have administrator access to the instance. + Use the following database queries to see the state of the current batched background migration: 1. Obtain the ID of the running migration: @@ -170,86 +157,166 @@ Use the following database queries to see the state of the current batched backg command above) to proceed with the batch when ready. On larger instances, background migrations can take as long as 48 hours to complete each batch. -### Automatic batch size optimization +#### Automatic batch size optimization -> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/60133) -> in GitLab 13.12, [behind a feature flag](../user/feature_flags.md), -> [enabled by default](https://gitlab.com/gitlab-org/gitlab/-/issues/329511). -> - Enabled on GitLab.com. -> - Recommended for production use. -> - For GitLab self-managed instances, GitLab administrators can opt to -> [disable it](#enable-or-disable-automatic-batch-size-optimization). +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/60133) in GitLab 13.2 [with a flag](../administration/feature_flags.md) named `optimize_batched_migrations`. Enabled by default. +WARNING: There can be [risks when disabling released features](../administration/feature_flags.md#risks-when-disabling-released-features). Refer to this feature's version history for more details. +FLAG: +On self-managed GitLab, by default this feature is available. To hide the feature, ask an administrator to [disable the feature flag](../administration/feature_flags.md) named `optimize_batched_migrations`. +On GitLab.com, this feature is available. + To maximize throughput of batched background migrations (in terms of the number of tuples updated per time unit), batch sizes are automatically adjusted based on how long the previous batches took to complete. -### Enable or disable automatic batch size optimization +#### Parallel execution -Automatic batch size optimization for batched background migrations is under development but ready for production use. -It is deployed behind a feature flag that is **enabled by default**. -[GitLab administrators with access to the GitLab Rails console](../administration/feature_flags.md) -can opt to disable it. +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/104027) in GitLab 15.7 [with a flag](../administration/feature_flags.md) named `batched_migrations_parallel_execution`. Disabled by default. +> - [Enabled on GitLab.com](https://gitlab.com/gitlab-org/gitlab/-/issues/372316) in GitLab 15.11. +> - [Generally available](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/120808) in GitLab 16.1. Feature flag `batched_migrations_parallel_execution` removed. -To enable it: +WARNING: +There can be [risks when disabling released features](../administration/feature_flags.md#risks-when-disabling-released-features). +Refer to this feature's version history for more details. + +To speed up the execution of batched background migrations, two migrations are executed at the same time. + +[GitLab administrators with access to the GitLab Rails console](../administration/feature_flags.md) can change +the number of batched background migrations executed in parallel: ```ruby -Feature.enable(:optimize_batched_migrations) +ApplicationSetting.update_all(database_max_running_batched_background_migrations: 4) ``` -To disable it: +### Fix and retry failed batched background migrations -```ruby -Feature.disable(:optimize_batched_migrations) +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/67504) in GitLab 14.3. + +If you [check the status](#check-the-status-of-batched-background-migrations) of batched background migrations, +some migrations might display in the **Failed** tab with a **failed** status: + + + +You must resolve all failed batched background migrations to upgrade to a newer +version of GitLab. + +To determine why the batched background migration failed, +[view the failure error logs](../development/database/batched_background_migrations.md#viewing-failure-error-logs) or: + +Prerequisites: + +- You must have administrator access to the instance. + +1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. Select **Admin Area**. +1. Select **Monitoring > Background Migrations**. +1. Select the **Failed** tab. This displays a list of failed batched background migrations. +1. Select the failed **Migration** to see the migration parameters and the jobs that failed. +1. Under **Failed jobs**, select each **ID** to see why the job failed. + +If you are a GitLab customer, consider opening a [Support Request](https://support.gitlab.com/hc/en-us/requests/new) +to debug why the batched background migrations failed. + +To correct the problem, you can retry the failed batched background migrations: + +Prerequisites: + +- You must have administrator access to the instance. + +1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. Select **Admin Area**. +1. Select **Monitoring > Background Migrations**. +1. Select the **Failed** tab. This displays a list of failed batched background migrations. +1. Select a failed batched background migration to retry by clicking on the retry button (**{retry}**). + +To monitor the retried batched background migrations, you can +[check the status of batched background migrations](#check-the-status-of-batched-background-migrations) +on a regular interval. + +## Background migrations + +In GitLab 13, background migrations were not batched. In GitLab 14 and later, this +type of migration was superseded by batched background migrations. + +### Check for pending background migrations + +To check for pending non-batched background migrations: + +::Tabs + +:::TabTitle Linux package (Omnibus) + +```shell +sudo gitlab-rails runner -e production 'puts Gitlab::BackgroundMigration.remaining' +sudo gitlab-rails runner -e production 'puts Gitlab::Database::BackgroundMigration::BatchedMigration.queued.count' ``` -### Parallel execution +:::TabTitle Self-compiled (source) -> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/104027) -> in GitLab 15.7, [behind a feature flag](../user/feature_flags.md), -> - [Enabled by default](https://gitlab.com/gitlab-org/gitlab/-/issues/372316) in GitLab 15.11. -> - Feature flag `batched_migrations_parallel_execution` removed in GitLab 16.1. +```shell +cd /home/git/gitlab +sudo -u git -H bundle exec rails runner -e production 'puts Gitlab::BackgroundMigration.remaining' +sudo -u git -H bundle exec rails runner -e production 'puts Gitlab::Database::BackgroundMigration::BatchedMigration.queued.count' +``` -There can be [risks when disabling released features](../administration/feature_flags.md#risks-when-disabling-released-features). -Refer to this feature's version history for more details. +::EndTabs -To speed up the execution of batched background migrations, two migrations are executed at the same time. +### Check for failed background migrations -[GitLab administrators with access to the GitLab Rails console](../administration/feature_flags.md) can change -the number of batched background migrations executed in parallel: +To check for non-batched background migrations that have failed: -```ruby -ApplicationSetting.update_all(database_max_running_batched_background_migrations: 4) +::Tabs + +:::TabTitle Linux package (Omnibus) + +For GitLab versions 14.10 and later: + +```shell +sudo gitlab-rails runner -e production 'puts Gitlab::Database::BackgroundMigration::BatchedMigration.with_status(:failed).count' ``` -## Troubleshooting +For GitLab versions 14.0-14.9: -### Enable or disable background migrations +```shell +sudo gitlab-rails runner -e production 'puts Gitlab::Database::BackgroundMigration::BatchedMigration.failed.count' +``` -In extremely limited circumstances, a GitLab administrator can disable either or -both of these [feature flags](../administration/feature_flags.md): +:::TabTitle Self-compiled (source) -- `execute_background_migrations` -- `execute_batched_migrations_on_schedule` +For GitLab versions 14.10 and later: -These flags are enabled by default. Disable them only as a last resort -to limit database operations in special circumstances, like database host maintenance. +```shell +cd /home/git/gitlab +sudo -u git -H bundle exec rails runner -e production 'puts Gitlab::Database::BackgroundMigration::BatchedMigration.with_status(:failed).count' +``` -WARNING: -Do not disable either of these flags unless you fully understand the ramifications. If you disable -the `execute_background_migrations` or `execute_batched_migrations_on_schedule` feature flag, -GitLab upgrades might fail and data loss might occur. +For GitLab versions 14.0-14.9: + +```shell +cd /home/git/gitlab +sudo -u git -H bundle exec rails runner -e production 'puts Gitlab::Database::BackgroundMigration::BatchedMigration.failed.count' +``` + +::EndTabs + +## Troubleshooting ### Database migrations failing because of batched background migration not finished -When updating to GitLab 14.2 or later there might be a database migration failing with a message like: +When updating to GitLab version 14.2 or later, database migrations might fail with a message like: ```plaintext StandardError: An error has occurred, all later migrations canceled: Expected batched background migration for the given configuration to be marked as 'finished', but it is 'active': - {:job_class_name=>"CopyColumnUsingBackgroundMigrationJob", :table_name=>"push_event_payloads", :column_name=>"event_id", :job_arguments=>[["event_id"], ["event_id_convert_to_bigint"]]} + {:job_class_name=>"CopyColumnUsingBackgroundMigrationJob", + :table_name=>"push_event_payloads", + :column_name=>"event_id", + :job_arguments=>[["event_id"], + ["event_id_convert_to_bigint"]] + } ``` First, check if you have followed the [version-specific upgrade instructions for 14.2](../update/index.md#1420). @@ -263,7 +330,7 @@ version and manually ensuring that the batched migrations complete successfully. #### Roll back and follow the required upgrade path -1. [Rollback and restore the previously installed version](../raketasks/backup_restore.md) +1. [Rollback and restore the previously installed version](../administration/backup_restore/index.md) 1. Update to either 14.0.5 or 14.1 **before** updating to 14.2+ 1. [Check the status](#check-the-status-of-batched-background-migrations) of the batched background migrations and make sure they are all marked as finished before attempting to upgrade again. If any remain marked as active, @@ -321,8 +388,14 @@ arguments. For example, if you receive an error similar to this: ```plaintext StandardError: An error has occurred, all later migrations canceled: -Expected batched background migration for the given configuration to be marked as 'finished', but it is 'active': - {:job_class_name=>"CopyColumnUsingBackgroundMigrationJob", :table_name=>"push_event_payloads", :column_name=>"event_id", :job_arguments=>[["event_id"], ["event_id_convert_to_bigint"]]} +Expected batched background migration for the given configuration to be marked as +'finished', but it is 'active': + {:job_class_name=>"CopyColumnUsingBackgroundMigrationJob", + :table_name=>"push_event_payloads", + :column_name=>"event_id", + :job_arguments=>[["event_id"], + ["event_id_convert_to_bigint"]] + } ``` Plug the arguments from the error message into the command: @@ -395,17 +468,19 @@ The following operations can disrupt your GitLab performance. They run a number Run the following check. If it returns non-zero and the count does not decrease over time, follow the rest of the steps in this section. ```shell -# For Omnibus installations: +# For Linux package installations: sudo gitlab-rails runner -e production 'puts Gitlab::BackgroundMigration.remaining' -# For installations from source: +# For self-compiled installations: cd /home/git/gitlab sudo -u git -H bundle exec rails runner -e production 'puts Gitlab::BackgroundMigration.remaining' ``` It is safe to re-execute the following commands, especially if you have 1000+ pending jobs which would likely overflow your runtime memory. -**For Omnibus installations** +::Tabs + +:::TabTitle Linux package (Omnibus) ```shell # Start the rails console @@ -417,7 +492,7 @@ pending_job_classes = scheduled_queue.select { |job| job["class"] == "Background pending_job_classes.each { |job_class| Gitlab::BackgroundMigration.steal(job_class) } ``` -**For installations from source** +:::TabTitle Self-compiled (source) ```shell # Start the rails console @@ -429,16 +504,14 @@ pending_job_classes = scheduled_queue.select { |job| job["class"] == "Background pending_job_classes.each { |job_class| Gitlab::BackgroundMigration.steal(job_class) } ``` +::EndTabs + ### Background migrations stuck in 'pending' state WARNING: The following operations can disrupt your GitLab performance. They run a number of Sidekiq jobs that perform various database or file updates. -- GitLab 13.6 introduced an issue where a background migration named - `BackfillJiraTrackerDeploymentType2` can be permanently stuck in a - **pending** state across upgrades. To clean up this stuck migration, see the - [13.6.0 version-specific instructions](index.md#1360). - GitLab 14.2 introduced an issue where a background migration named `BackfillDraftStatusOnMergeRequests` can be permanently stuck in a **pending** state across upgrades when the instance lacks records that match @@ -469,17 +542,19 @@ it returns non-zero and the count does not decrease over time, follow the rest of the steps in this section. ```shell -# For Omnibus installations: +# For Linux package installations: sudo gitlab-rails runner -e production 'puts Gitlab::Database::BackgroundMigrationJob.pending.count' -# For installations from source: +# For self-compiled installations: cd /home/git/gitlab sudo -u git -H bundle exec rails runner -e production 'puts Gitlab::Database::BackgroundMigrationJob.pending.count' ``` It is safe to re-attempt these migrations to clear them out from a pending status: -**For Omnibus installations** +::Tabs + +:::TabTitle Linux package (Omnibus) ```shell # Start the rails console @@ -493,7 +568,7 @@ Gitlab::Database::BackgroundMigrationJob.pending.find_each do |job| end ``` -**For installations from source** +:::TabTitle Self-compiled (source) ```shell # Start the rails console @@ -506,3 +581,5 @@ Gitlab::Database::BackgroundMigrationJob.pending.find_each do |job| puts "Result: #{result}" end ``` + +::EndTabs |