diff options
Diffstat (limited to 'doc/update/index.md')
| -rw-r--r-- | doc/update/index.md | 160 |
1 files changed, 103 insertions, 57 deletions
diff --git a/doc/update/index.md b/doc/update/index.md index b719c47ae26..bb7fdaa93c0 100644 --- a/doc/update/index.md +++ b/doc/update/index.md @@ -12,6 +12,11 @@ GitLab version is, if you're upgrading to a major version, and so on. Make sure to read the whole page as it contains information related to every upgrade method. +NOTE: +Upgrade GitLab to the latest available patch release, for example `13.8.8` rather than `13.8.0`. +This includes [versions you must stop at on the upgrade path](#upgrade-paths) as there may +be fixes for issues relating to the upgrade process. + The [maintenance policy documentation](../policy/maintenance.md) has additional information about upgrading, including: @@ -76,22 +81,44 @@ See the guide to [plan your GitLab upgrade](plan_your_upgrade.md). ## Checking for background migrations before upgrading -Certain major/minor releases may require different migrations to be -finished before you update to the newer version. +Certain releases may require different migrations to be +finished before you update to the newer version. Additionally check +[batched migrations](#batched-background-migrations) from GitLab 14.0. Decrease the time required to complete these migrations by increasing the number of [Sidekiq workers](../administration/operations/extra_sidekiq_processes.md) that can process jobs in the `background_migration` queue. -**For GitLab 14.0 and newer** +**For Omnibus installations:** + +```shell +sudo gitlab-rails runner -e production 'puts Gitlab::BackgroundMigration.remaining' +``` + +**For installations from source:** + +```shell +cd /home/git/gitlab +sudo -u git -H bundle exec rails runner -e production 'puts Gitlab::BackgroundMigration.remaining' +``` + +### Batched background migrations + +GitLab 14.0 introduced [batched background migrations](../user/admin_area/monitoring/background_migrations.md). -To check the status of [batched background migrations](../user/admin_area/monitoring/background_migrations.md): +Some installations [may need to run GitLab 14.0 for at least a day](#1400) to complete the database changes introduced by that upgrade. + +#### Check the status of batched background migrations + +To check the status of batched background migrations: 1. On the top bar, select **Menu > Admin**. 1. On the left sidebar, select **Monitoring > Background Migrations**.  +All migrations must have a `Finished` status before you upgrade GitLab. + The status of batched background migrations can also be queried directly in the database. 1. Log into a `psql` prompt according to the directions for your instance's installation method @@ -102,20 +129,14 @@ The status of batched background migrations can also be queried directly in the select job_class_name, table_name, column_name, job_arguments from batched_background_migrations where status <> 3; ``` -**For Omnibus installations** +If the migrations are not finished and you try to update to a later version, +GitLab prompts you with an error: -You can also run: - -```shell -sudo gitlab-rails runner -e production 'puts Gitlab::BackgroundMigration.remaining' +```plaintext +Expected batched background migration for the given configuration to be marked as 'finished', but it is 'active': ``` -**For installations from source** - -```shell -cd /home/git/gitlab -sudo -u git -H bundle exec rails runner -e production 'puts Gitlab::BackgroundMigration.remaining' -``` +If you get this error, [check the batched background migration options](../user/admin_area/monitoring/background_migrations.md#database-migrations-failing-because-of-batched-background-migration-not-finished) to complete the upgrade. ### What do I do if my background migrations are stuck? @@ -148,6 +169,10 @@ pending_job_classes = scheduled_queue.select { |job| job["class"] == "Background pending_job_classes.each { |job_class| Gitlab::BackgroundMigration.steal(job_class) } ``` +**Batched migrations (GitLab 14.0 and newer):** + +See [troubleshooting batched background migrations](../user/admin_area/monitoring/background_migrations.md#troubleshooting). + ## Dealing with running CI/CD pipelines and jobs If you upgrade your GitLab instance while the GitLab Runner is processing jobs, the trace updates fail. When GitLab is back online, the trace updates should self-heal. However, depending on the error, the GitLab Runner either retries or eventually terminates job handling. @@ -163,12 +188,10 @@ To address the above two scenario's, it is advised to do the following prior to ## Checking for pending Advanced Search migrations -This section is only applicable if you have enabled the [Elasticsearch -integration](../integration/elasticsearch.md). +This section is only applicable if you have enabled the [Elasticsearch integration](../integration/elasticsearch.md). -Major releases require all [Advanced Search -migrations](../integration/elasticsearch.md#advanced-search-migrations) to -be finished from the most recent minor release in your current version +Major releases require all [Advanced Search migrations](../integration/elasticsearch.md#advanced-search-migrations) +to be finished from the most recent minor release in your current version before the major version upgrade. You can find pending migrations by running the following command: @@ -187,8 +210,7 @@ sudo -u git -H bundle exec rake gitlab:elastic:list_pending_migrations ### What do I do if my Advanced Search migrations are stuck? -See [how to retry a halted -migration](../integration/elasticsearch.md#retry-a-halted-migration). +See [how to retry a halted migration](../integration/elasticsearch.md#retry-a-halted-migration). ## Upgrade paths @@ -200,7 +222,7 @@ Find where your version sits in the upgrade path below, and upgrade GitLab accordingly, while also consulting the [version-specific upgrade instructions](#version-specific-upgrading-instructions): -`8.11.Z` -> `8.12.0` -> `8.17.7` -> `9.5.10` -> `10.8.7` -> [`11.11.8`](#1200) -> `12.0.12` -> [`12.1.17`](#1210) -> `12.10.14` -> `13.0.14` -> [`13.1.11`](#1310) -> [`13.8.8`](#1388) -> [latest `13.12.Z`](https://about.gitlab.com/releases/categories/releases/) -> [latest `14.0.Z`](#1400) -> [latest `14.Y.Z`](https://about.gitlab.com/releases/categories/releases/) +`8.11.Z` -> `8.12.0` -> `8.17.7` -> `9.5.10` -> `10.8.7` -> [`11.11.8`](#1200) -> `12.0.12` -> [`12.1.17`](#1210) -> `12.10.14` -> `13.0.14` -> [`13.1.11`](#1310) -> [`13.8.8`](#1388) -> [`13.12.15`](https://about.gitlab.com/releases/categories/releases/) -> [latest `14.0.Z`](#1400) -> [latest `14.1.Z`](#1410) -> [latest `14.Y.Z`](https://about.gitlab.com/releases/categories/releases/) The following table, while not exhaustive, shows some examples of the supported upgrade paths. @@ -234,12 +256,11 @@ It's also important to ensure that any background migrations have been fully com before upgrading to a new major version. To see the current size of the `background_migration` queue, [Check for background migrations before upgrading](#checking-for-background-migrations-before-upgrading). -If you have enabled the [Elasticsearch -integration](../integration/elasticsearch.md), then ensure +If you have enabled the [Elasticsearch integration](../integration/elasticsearch.md), then ensure all Advanced Search migrations are completed in the last minor version within -your current version. Be sure to [check for pending Advanced Search -migrations](#checking-for-pending-advanced-search-migrations) before proceeding -with the major version upgrade. +your current version. Be sure to +[check for pending Advanced Search migrations](#checking-for-pending-advanced-search-migrations) +before proceeding with the major version upgrade. If your GitLab instance has any runners associated with it, it is very important to upgrade GitLab Runner to match the GitLab minor version that was @@ -307,6 +328,33 @@ NOTE: Specific information that follow related to Ruby and Git versions do not apply to [Omnibus installations](https://docs.gitlab.com/omnibus/) and [Helm Chart deployments](https://docs.gitlab.com/charts/). They come with appropriate Ruby and Git versions and are not using system binaries for Ruby and Git. There is no need to install Ruby or Git when utilizing these two approaches. +### 14.5.0 + +- When `make` is run, Gitaly builds are now created in `_build/bin` and no longer in the root directory of the source directory. If you +are using a source install, update paths to these binaries in your [systemd unit files](upgrading_from_source.md#configure-systemd-units) +or [init scripts](upgrading_from_source.md#configure-sysv-init-script) by [following the documentation](upgrading_from_source.md). + +- Connections between Workhorse and Gitaly use the Gitaly `backchannel` protocol by default. If you deployed a gRPC proxy between Workhorse and Gitaly, + Workhorse can no longer connect. As a workaround, [disable the temporary `workhorse_use_sidechannel`](../administration/feature_flags.md#enable-or-disable-the-feature) + feature flag. If you need a proxy between Workhorse and Gitaly, use a TCP proxy. If you have feedback about this change, please go to [this issue](https://gitlab.com/gitlab-com/gl-infra/scalability/-/issues/1301). + +- In 14.1 we introduced a background migration that changes how we store merge request diff commits + in order to significantly reduce the amount of storage needed. + In 14.5 we introduce a set of migrations that wrap up this process by making sure + that all remaining jobs over the `merge_request_diff_commits` table are completed. + These jobs will have already been processed in most cases so that no extra time is necessary during an upgrade to 14.5. + But if there are remaining jobs, the deployment may take a few extra minutes to complete. + + All merge request diff commits will automatically incorporate these changes, and there are no + additional requirements to perform the upgrade. + Existing data in the `merge_request_diff_commits` table remains unpacked until you run `VACUUM FULL merge_request_diff_commits`. + But note that the `VACUUM FULL` operation locks and rewrites the entire `merge_request_diff_commits` table, + so the operation takes some time to complete and it blocks access to this table until the end of the process. + We advise you to only run this command while GitLab is not actively used or it is taken offline for the duration of the process. + The time it takes to complete depends on the size of the table, which can be obtained by using `select pg_size_pretty(pg_total_relation_size('merge_request_diff_commits'));`. + + For more information, refer to [this issue](https://gitlab.com/gitlab-org/gitlab/-/issues/331823). + ### 14.4.0 Git 2.33.x and later is required. We recommend you use the @@ -314,9 +362,11 @@ Git 2.33.x and later is required. We recommend you use the ### 14.3.0 -Ruby 2.7.4 is required. Refer to [the Ruby installation instructions](../install/installation.md#2-ruby) +- [Instances running 14.0.0 - 14.0.4 should not upgrade directly to GitLab 14.2 or later](#upgrading-to-later-14y-releases). +- Ensure [batched background migrations finish](#batched-background-migrations) before upgrading + to 14.3.Z from earlier GitLab 14 releases. +- Ruby 2.7.4 is required. Refer to [the Ruby installation instructions](../install/installation.md#2-ruby) for how to proceed. - - GitLab 14.3.0 contains post-deployment migrations to [address Primary Key overflow risk for tables with an integer PK](https://gitlab.com/groups/gitlab-org/-/epics/4785) for the tables listed below: - [`ci_builds.id`](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/70245) @@ -337,13 +387,10 @@ for how to proceed. ### 14.2.0 -- Due to an issue where `BatchedBackgroundMigrationWorkers` were - [not working](https://gitlab.com/gitlab-org/charts/gitlab/-/issues/2785#note_614738345) - for self-managed instances, a [fix was created](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/65106) - and a [14.0.Z](#1400) version was released. If you haven't updated to 14.0.5, you need - to update to at least 14.1.0 that contains the same fix before you update to 14.2. +- [Instances running 14.0.0 - 14.0.4 should not upgrade directly to GitLab 14.2 or later](#upgrading-to-later-14y-releases). +- Ensure [batched background migrations finish](#batched-background-migrations) before upgrading + to 14.2.Z from earlier GitLab 14 releases. - GitLab 14.2.0 contains background migrations to [address Primary Key overflow risk for tables with an integer PK](https://gitlab.com/groups/gitlab-org/-/epics/4785) for the tables listed below: - - [`ci_build_needs`](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/65216) - [`ci_build_trace_chunks`](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/66123) - [`ci_builds_runner_session`](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/66433) @@ -366,35 +413,24 @@ for how to proceed. ### 14.1.0 -- Due to an issue where `BatchedBackgroundMigrationWorkers` were - [not working](https://gitlab.com/gitlab-org/charts/gitlab/-/issues/2785#note_614738345) - for self-managed instances, a [fix was created](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/65106) - and a [14.0.Z](#1400) version was released. If you haven't updated to 14.0.5, you need - to update to at least 14.1.0 that contains the same fix before you update to - a later version. - - After you update to 14.1.0, - [batched background migrations need to finish](../user/admin_area/monitoring/background_migrations.md#check-the-status-of-background-migrations) - before you update to a later version. - - If the migrations are not finished and you try to update to a later version, - you'll see an error like: - - ```plaintext - Expected batched background migration for the given configuration to be marked as 'finished', but it is 'active': - ``` - - See how to [resolve this error](../user/admin_area/monitoring/background_migrations.md#database-migrations-failing-because-of-batched-background-migration-not-finished). +- [Instances running 14.0.0 - 14.0.4 should not upgrade directly to GitLab 14.2 or later](#upgrading-to-later-14y-releases) + but can upgrade to 14.1.Z. +- It is not required for instances already running 14.0.5 (or higher) to stop at 14.1.Z. + 14.1 is included on the upgrade path for the broadest compatibility + with self-managed installations, and ensure 14.0.0-14.0.4 installations do not + encounter issues with [batched background migrations](#batched-background-migrations). ### 14.0.0 +- Database changes made by the upgrade to GitLab 14.0 can take hours or days to complete on larger GitLab instances. + These [batched background migrations](#batched-background-migrations) update whole database tables to mitigate primary key overflow and must be finished before upgrading to GitLab 14.2 or higher. - Due to an issue where `BatchedBackgroundMigrationWorkers` were [not working](https://gitlab.com/gitlab-org/charts/gitlab/-/issues/2785#note_614738345) for self-managed instances, a [fix was created](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/65106) - that requires an update to at least 14.0.5. + that requires an update to at least 14.0.5. The fix was also released in [14.1.0](#1410). After you update to 14.0.5 or a later 14.0 patch version, - [batched background migrations need to finish](../user/admin_area/monitoring/background_migrations.md#check-the-status-of-background-migrations) + [batched background migrations need to finish](#batched-background-migrations) before you update to a later version. If the migrations are not finished and you try to update to a later version, @@ -413,6 +449,16 @@ for how to proceed. You should instead follow a [supported upgrade path](#upgrade-paths). - The support of PostgreSQL 11 [has been dropped](../install/requirements.md#database). Make sure to [update your database](https://docs.gitlab.com/omnibus/settings/database.html#upgrade-packaged-postgresql-server) to version 12 before updating to GitLab 14.0. +#### Upgrading to later 14.Y releases + +- Instances running 14.0.0 - 14.0.4 should not upgrade directly to GitLab 14.2 or later, + because of [batched background migrations](#batched-background-migrations). + 1. Upgrade first to either: + - 14.0.5 or a later 14.0.Z patch release. + - 14.1.0 or a later 14.1.Z patch release. + 1. [Batched background migrations need to finish](#batched-background-migrations) + before you update to a later version [and may take longer than usual](#1400). + ### 13.11.0 Git 2.31.x and later is required. We recommend you use the |