diff options
Diffstat (limited to 'doc/update')
| -rw-r--r-- | doc/update/deprecations.md | 176 | ||||
| -rw-r--r-- | doc/update/index.md | 160 | ||||
| -rw-r--r-- | doc/update/package/index.md | 37 | ||||
| -rw-r--r-- | doc/update/patch_versions.md | 9 | ||||
| -rw-r--r-- | doc/update/upgrading_from_source.md | 35 | ||||
| -rw-r--r-- | doc/update/zero_downtime.md | 52 |
6 files changed, 358 insertions, 111 deletions
diff --git a/doc/update/deprecations.md b/doc/update/deprecations.md index e2af4f453c0..42bd41ed74b 100644 --- a/doc/update/deprecations.md +++ b/doc/update/deprecations.md @@ -17,16 +17,26 @@ sole discretion of GitLab Inc. <!-- vale off --> <!-- +DO NOT EDIT THIS PAGE DIRECTLY + This page is automatically generated from the YAML files in `/data/deprecations` by the rake task located at `lib/tasks/gitlab/docs/compile_deprecations.rake`. -Do not edit this page directly. +For deprecation authors (usually Product Managers and Engineering Managers): + +- To add a deprecation, use the example.yml file in `/data/deprecations/templates` as a template. +- For more information about authoring deprecations, check the the deprecation item guidance: + https://about.gitlab.com/handbook/marketing/blog/release-posts/#creating-a-deprecation-entry + +For deprecation reviewers (Technical Writers only): -To add a deprecation, use the example.yml file in `/data/deprecations/templates` as a template, -then run `bin/rake gitlab:docs:compile_deprecations`. +- To update the deprecation doc, run: `bin/rake gitlab:docs:compile_deprecations` +- To verify the deprecations doc is up to date, run: `bin/rake gitlab:docs:check_deprecations` +- For more information about updating the deprecation doc, see the deprecation doc update guidance: + https://about.gitlab.com/handbook/marketing/blog/release-posts/#update-the-deprecations-doc --> -## 14.4 +## 14.5 ### Rename Task Runner pod to Toolbox @@ -44,17 +54,17 @@ The [release-cli](https://gitlab.com/gitlab-org/release-cli) will be released as Announced: 2021-08-22 -## 15.0 +## 14.8 -### Legacy database configuration +### openSUSE Leap 15.2 packages -The syntax of [GitLabs database](https://docs.gitlab.com/omnibus/settings/database.html) -configuration located in `database.yml` is changing and the legacy format is deprecated. The legacy format -supported using a single PostgreSQL adapter, whereas the new format is changing to support multiple databases. The `main:` database needs to be defined as a first configuration item. +Distribution support and security updates for openSUSE Leap 15.2 are [ending December 2021](https://en.opensuse.org/Lifetime#openSUSE_Leap). -This deprecation mainly impacts users compiling GitLab from source because Omnibus will handle this configuration automatically. +Starting in 14.5 we are providing packages for openSUSE Leap 15.3, and will stop providing packages for openSUSE Leap 15.2 in the 14.8 milestone. -Announced: 2021-09-22 +Announced: 2021-11-22 + +## 15.0 ### Audit events for repository push events @@ -66,15 +76,39 @@ dramatically slow down GitLab instances. For this reason, they are being removed Announced: 2021-09-22 -### OmniAuth Kerberos gem +### Certificate-based integration with Kubernetes -The `omniauth-kerberos` gem will be removed in our next major release, GitLab 15.0. +[We are deprecating the certificate-based integration with Kubernetes](https://about.gitlab.com/blog/2021/11/15/deprecating-the-cert-based-kubernetes-integration/). +The timeline of removal of the integration from the product is not yet planned and we will communicate +more details as they emerge. The certificate-based integration will continue to receive security and +critical fixes, and features built on the integration will continue to work with supported Kubernetes +versions. We will provide migration plans in a future iteration. See [the list of features affected by this deprecation](https://docs.gitlab.com/ee/user/infrastructure/clusters/#deprecated-features). +For updates and details, follow this [epic](https://gitlab.com/groups/gitlab-org/configure/-/epics/8). -This gem has not been maintained and has very little usage. We therefore plan to remove support for this authentication method and recommend using the Kerberos [SPNEGO](https://en.wikipedia.org/wiki/SPNEGO) integration instead. You can follow the [upgrade instructions](https://docs.gitlab.com/ee/integration/kerberos.html#upgrading-from-password-based-to-ticket-based-kerberos-sign-ins) to upgrade from the `omniauth-kerberos` integration to the supported one. +For a more robust, secure, forthcoming, and reliable integration with Kubernetes, we recommend the use of the +[Kubernetes Agent](https://docs.gitlab.com/ee/user/clusters/agent/) to connect Kubernetes clusters with GitLab. -Note that we are not deprecating the Kerberos SPNEGO integration, only the old password-based Kerberos integration. +Announced: 2021-11-15 -Announced: 2021-09-22 +### Converting an instance (shared) runner to a project (specific) runner is deprecated + +In GitLab 15.0, we will remove the feature that enables you to convert an instance (shared) runner to a project (specific) runner. Users who need to add a runner to only a particular project can register a runner to the project directly. + +Announced: 2021-11-22 + +### Deprecate `Versions` on base `PackageType` + +As part of the work to create a [Package Registry GraphQL API](https://gitlab.com/groups/gitlab-org/-/epics/6318), the Package group deprecated the `Version` type for the basic `PackageType` type and moved it to [`PackageDetailsType`](https://docs.gitlab.com/ee/api/graphql/reference/index.html#packagedetailstype). + +In milestone 15.0, we will completely remove `Version` from `PackageType`. + +Announced: 2021-11-22 + +### Deprecate support for SLES 12 SP2 + +Long term service and support (LTSS) for SUSE Linux Enterprise Server (SLES) 12 SP2 [ended on March 31, 2021](https://www.suse.com/lifecycle/). The CA certificates on SP2 include the expired DST root certificate, and it's not getting new CA certificate package updates. We have implemented some [workarounds](https://gitlab.com/gitlab-org/gitlab-omnibus-builder/-/merge_requests/191), but we will not be able to continue to keep the build running properly. + +Announced: 2021-11-22 ### GitLab Serverless @@ -84,7 +118,23 @@ We decided to remove the GitLab Serverless features as they never really resonat Announced: 2021-09-22 -## 15.2 +### Known host required for GitLab Runner SSH executor + +In [GitLab 14.3](https://gitlab.com/gitlab-org/gitlab-runner/-/merge_requests/3074), we added a configuration setting in the GitLab Runner `config.toml` file. This setting, [`[runners.ssh.disable_strict_host_key_checking]`](https://docs.gitlab.com/runner/executors/ssh.html#security), controls whether or not to use strict host key checking with the SSH executor. + +In GitLab 15.0 and later, the default value for this configuration option will change from `true` to `false`. This means that strict host key checking will be enforced when using the GitLab Runner SSH executor. + +Announced: 2021-11-22 + +### Legacy database configuration + +The syntax of [GitLabs database](https://docs.gitlab.com/omnibus/settings/database.html) +configuration located in `database.yml` is changing and the legacy format is deprecated. The legacy format +supported using a single PostgreSQL adapter, whereas the new format is changing to support multiple databases. The `main:` database needs to be defined as a first configuration item. + +This deprecation mainly impacts users compiling GitLab from source because Omnibus will handle this configuration automatically. + +Announced: 2021-09-22 ### NFS for Git repository storage deprecated @@ -99,3 +149,95 @@ Gitaly Cluster offers tremendous benefits for our customers such as: We encourage customers currently using NFS for Git repositories to plan their migration by reviewing our documentation on [migrating to Gitaly Cluster](https://docs.gitlab.com/ee/administration/gitaly/index.html#migrate-to-gitaly-cluster). Announced: 2021-06-22 + +### OmniAuth Kerberos gem + +The `omniauth-kerberos` gem will be removed in our next major release, GitLab 15.0. + +This gem has not been maintained and has very little usage. We therefore plan to remove support for this authentication method and recommend using the Kerberos [SPNEGO](https://en.wikipedia.org/wiki/SPNEGO) integration instead. You can follow the [upgrade instructions](https://docs.gitlab.com/ee/integration/kerberos.html#upgrading-from-password-based-to-ticket-based-kerberos-sign-ins) to upgrade from the `omniauth-kerberos` integration to the supported one. + +Note that we are not deprecating the Kerberos SPNEGO integration, only the old password-based Kerberos integration. + +Announced: 2021-09-22 + +### Package pipelines in API payload is paginated + +A request to the API for `/api/v4/projects/:id/packages` returns a paginated result of packages. Each package lists all of its pipelines in this response. This is a performance concern, as it's possible for a package to have hundreds or thousands of associated pipelines. + +In milestone 15.0, we will remove the `pipelines` attribute from the API response. + +Announced: 2021-11-22 + +### REST API Runner will not contain `paused` + +Runner REST API will not return `paused` as a status in GitLab 15.0. + +Paused runners' status will only relate to runner contact status, such as: +`online`, `offline`, or `not_connected`. Status `paused` will not appear when the runner is +not active. + +When checking if a runner is `paused`, API users are advised to check the boolean attribute +`active` to be `false` instead. + +Announced: 2021-11-22 + +### Removal of `promote-db` command from `gitlab-ctl` + +In GitLab 14.5, we introduced the command `gitlab-ctl promote` to promote any Geo secondary node to a primary during a failover. This command replaces `gitlab-ctl promote-db` which is used to promote database nodes in multi-node Geo secondary sites. `gitlab-ctl promote-db` will continue to function as-is and be available until GitLab 15.0. We recommend that Geo customers begin testing the new `gitlab-ctl promote` command in their staging environments and incorporating the new command in their failover procedures. + +Announced: 2021-11-22 + +### Removal of `promote-to-primary-node` command from `gitlab-ctl` + +In GitLab 14.5, we introduced the command `gitlab-ctl promote` to promote any Geo secondary node to a primary during a failover. This command replaces `gitlab-ctl promote-to-primary-node` which was only usable for single-node Geo sites. `gitlab-ctl promote-to-primary-node` will continue to function as-is and be available until GitLab 15.0. We recommend that Geo customers begin testing the new `gitlab-ctl promote` command in their staging environments and incorporating the new command in their failover procedures. + +Announced: 2021-11-22 + +### Remove the `:dependency_proxy_for_private_groups` feature flag + +We added a feature flag because [GitLab-#11582](https://gitlab.com/gitlab-org/gitlab/-/issues/11582) changed how public groups use the Dependency Proxy. Prior to this change, you could use the Dependency Proxy without authentication. The change requires authentication to use the Dependency Proxy. + +In milestone 15.0, we will remove the feature flag entirely. Moving forward, you must authenticate when using the Dependency Proxy. + +Announced: 2021-11-22 + +### Remove the `pipelines` field from the `version` field + +In GraphQL, there are two `pipelines` fields that you can use in a [`PackageDetailsType`](https://docs.gitlab.com/ee/api/graphql/reference/#packagedetailstype) to get the pipelines for package versions: + +- The `versions` field's `pipelines` field. This returns all the pipelines associated with all the package's versions, which can pull an unbounded number of objects in memory and create performance concerns. +- The `pipelines` field of a specific `version`. This returns only the pipelines associated with that single package version. + +To mitigate possible performance problems, we will remove the `versions` field's `pipelines` field in milestone 15.0. Although you will no longer be able to get all pipelines for all versions of a package, you can still get the pipelines of a single version through the remaining `pipelines` field for that version. + +Announced: 2021-11-22 + +### Update to the Container Registry group-level API + +In milestone 15.0, support for the `tags` and `tags_count` parameters will be removed from the Container Registry API that [gets registry repositories from a group](../api/container_registry.md#within-a-group). + +The `GET /groups/:id/registry/repositories` endpoint will remain, but won't return any info about tags. To get the info about tags, you can use the existing `GET /registry/repositories/:id` endpoint, which will continue to support the `tags` and `tag_count` options as it does today. The latter must be called once per image repository. + +Announced: 2021-11-22 + +### Value Stream Analytics filtering calculation change + +We are changing how the date filter works in Value Stream Analytics. Instead of filtering by the time that the issue or merge request was created, the date filter will filter by the end event time of the given stage. This will result in completely different figures after this change has rolled out. + +If you monitor Value Stream Analytics metrics and rely on the date filter, to avoid losing data, you must save the data prior to this change. + +Announced: 2021-11-22 + +### `AuthenticationType` for `[runners.cache.s3]` must be explicitly assigned + +In GitLab 15.0 and later, to access the AWS S3 cache, you must specify the `AuthenticationType` for [`[runners.cache.s3]`](https://docs.gitlab.com/runner/configuration/advanced-configuration.html#the-runnerscaches3-section). The `AuthenticationType` must be `IAM` or `credentials`. + +Prior to 14.5, if you did not define the `AuthenticationType`, GitLab Runner chose a type for you. + +Announced: 2021-11-22 + +### defaultMergeCommitMessageWithDescription GraphQL API field will be removed in GitLab 15.0 + +The GraphQL API field `defaultMergeCommitMessageWithDescription` has been deprecated and will be removed in GitLab 15.0. For projects with a commit message template set, it will ignore the template. + +Announced: 2021-11-22 diff --git a/doc/update/index.md b/doc/update/index.md index b719c47ae26..bb7fdaa93c0 100644 --- a/doc/update/index.md +++ b/doc/update/index.md @@ -12,6 +12,11 @@ GitLab version is, if you're upgrading to a major version, and so on. Make sure to read the whole page as it contains information related to every upgrade method. +NOTE: +Upgrade GitLab to the latest available patch release, for example `13.8.8` rather than `13.8.0`. +This includes [versions you must stop at on the upgrade path](#upgrade-paths) as there may +be fixes for issues relating to the upgrade process. + The [maintenance policy documentation](../policy/maintenance.md) has additional information about upgrading, including: @@ -76,22 +81,44 @@ See the guide to [plan your GitLab upgrade](plan_your_upgrade.md). ## Checking for background migrations before upgrading -Certain major/minor releases may require different migrations to be -finished before you update to the newer version. +Certain releases may require different migrations to be +finished before you update to the newer version. Additionally check +[batched migrations](#batched-background-migrations) from GitLab 14.0. Decrease the time required to complete these migrations by increasing the number of [Sidekiq workers](../administration/operations/extra_sidekiq_processes.md) that can process jobs in the `background_migration` queue. -**For GitLab 14.0 and newer** +**For Omnibus installations:** + +```shell +sudo gitlab-rails runner -e production 'puts Gitlab::BackgroundMigration.remaining' +``` + +**For installations from source:** + +```shell +cd /home/git/gitlab +sudo -u git -H bundle exec rails runner -e production 'puts Gitlab::BackgroundMigration.remaining' +``` + +### Batched background migrations + +GitLab 14.0 introduced [batched background migrations](../user/admin_area/monitoring/background_migrations.md). -To check the status of [batched background migrations](../user/admin_area/monitoring/background_migrations.md): +Some installations [may need to run GitLab 14.0 for at least a day](#1400) to complete the database changes introduced by that upgrade. + +#### Check the status of batched background migrations + +To check the status of batched background migrations: 1. On the top bar, select **Menu > Admin**. 1. On the left sidebar, select **Monitoring > Background Migrations**.  +All migrations must have a `Finished` status before you upgrade GitLab. + The status of batched background migrations can also be queried directly in the database. 1. Log into a `psql` prompt according to the directions for your instance's installation method @@ -102,20 +129,14 @@ The status of batched background migrations can also be queried directly in the select job_class_name, table_name, column_name, job_arguments from batched_background_migrations where status <> 3; ``` -**For Omnibus installations** +If the migrations are not finished and you try to update to a later version, +GitLab prompts you with an error: -You can also run: - -```shell -sudo gitlab-rails runner -e production 'puts Gitlab::BackgroundMigration.remaining' +```plaintext +Expected batched background migration for the given configuration to be marked as 'finished', but it is 'active': ``` -**For installations from source** - -```shell -cd /home/git/gitlab -sudo -u git -H bundle exec rails runner -e production 'puts Gitlab::BackgroundMigration.remaining' -``` +If you get this error, [check the batched background migration options](../user/admin_area/monitoring/background_migrations.md#database-migrations-failing-because-of-batched-background-migration-not-finished) to complete the upgrade. ### What do I do if my background migrations are stuck? @@ -148,6 +169,10 @@ pending_job_classes = scheduled_queue.select { |job| job["class"] == "Background pending_job_classes.each { |job_class| Gitlab::BackgroundMigration.steal(job_class) } ``` +**Batched migrations (GitLab 14.0 and newer):** + +See [troubleshooting batched background migrations](../user/admin_area/monitoring/background_migrations.md#troubleshooting). + ## Dealing with running CI/CD pipelines and jobs If you upgrade your GitLab instance while the GitLab Runner is processing jobs, the trace updates fail. When GitLab is back online, the trace updates should self-heal. However, depending on the error, the GitLab Runner either retries or eventually terminates job handling. @@ -163,12 +188,10 @@ To address the above two scenario's, it is advised to do the following prior to ## Checking for pending Advanced Search migrations -This section is only applicable if you have enabled the [Elasticsearch -integration](../integration/elasticsearch.md). +This section is only applicable if you have enabled the [Elasticsearch integration](../integration/elasticsearch.md). -Major releases require all [Advanced Search -migrations](../integration/elasticsearch.md#advanced-search-migrations) to -be finished from the most recent minor release in your current version +Major releases require all [Advanced Search migrations](../integration/elasticsearch.md#advanced-search-migrations) +to be finished from the most recent minor release in your current version before the major version upgrade. You can find pending migrations by running the following command: @@ -187,8 +210,7 @@ sudo -u git -H bundle exec rake gitlab:elastic:list_pending_migrations ### What do I do if my Advanced Search migrations are stuck? -See [how to retry a halted -migration](../integration/elasticsearch.md#retry-a-halted-migration). +See [how to retry a halted migration](../integration/elasticsearch.md#retry-a-halted-migration). ## Upgrade paths @@ -200,7 +222,7 @@ Find where your version sits in the upgrade path below, and upgrade GitLab accordingly, while also consulting the [version-specific upgrade instructions](#version-specific-upgrading-instructions): -`8.11.Z` -> `8.12.0` -> `8.17.7` -> `9.5.10` -> `10.8.7` -> [`11.11.8`](#1200) -> `12.0.12` -> [`12.1.17`](#1210) -> `12.10.14` -> `13.0.14` -> [`13.1.11`](#1310) -> [`13.8.8`](#1388) -> [latest `13.12.Z`](https://about.gitlab.com/releases/categories/releases/) -> [latest `14.0.Z`](#1400) -> [latest `14.Y.Z`](https://about.gitlab.com/releases/categories/releases/) +`8.11.Z` -> `8.12.0` -> `8.17.7` -> `9.5.10` -> `10.8.7` -> [`11.11.8`](#1200) -> `12.0.12` -> [`12.1.17`](#1210) -> `12.10.14` -> `13.0.14` -> [`13.1.11`](#1310) -> [`13.8.8`](#1388) -> [`13.12.15`](https://about.gitlab.com/releases/categories/releases/) -> [latest `14.0.Z`](#1400) -> [latest `14.1.Z`](#1410) -> [latest `14.Y.Z`](https://about.gitlab.com/releases/categories/releases/) The following table, while not exhaustive, shows some examples of the supported upgrade paths. @@ -234,12 +256,11 @@ It's also important to ensure that any background migrations have been fully com before upgrading to a new major version. To see the current size of the `background_migration` queue, [Check for background migrations before upgrading](#checking-for-background-migrations-before-upgrading). -If you have enabled the [Elasticsearch -integration](../integration/elasticsearch.md), then ensure +If you have enabled the [Elasticsearch integration](../integration/elasticsearch.md), then ensure all Advanced Search migrations are completed in the last minor version within -your current version. Be sure to [check for pending Advanced Search -migrations](#checking-for-pending-advanced-search-migrations) before proceeding -with the major version upgrade. +your current version. Be sure to +[check for pending Advanced Search migrations](#checking-for-pending-advanced-search-migrations) +before proceeding with the major version upgrade. If your GitLab instance has any runners associated with it, it is very important to upgrade GitLab Runner to match the GitLab minor version that was @@ -307,6 +328,33 @@ NOTE: Specific information that follow related to Ruby and Git versions do not apply to [Omnibus installations](https://docs.gitlab.com/omnibus/) and [Helm Chart deployments](https://docs.gitlab.com/charts/). They come with appropriate Ruby and Git versions and are not using system binaries for Ruby and Git. There is no need to install Ruby or Git when utilizing these two approaches. +### 14.5.0 + +- When `make` is run, Gitaly builds are now created in `_build/bin` and no longer in the root directory of the source directory. If you +are using a source install, update paths to these binaries in your [systemd unit files](upgrading_from_source.md#configure-systemd-units) +or [init scripts](upgrading_from_source.md#configure-sysv-init-script) by [following the documentation](upgrading_from_source.md). + +- Connections between Workhorse and Gitaly use the Gitaly `backchannel` protocol by default. If you deployed a gRPC proxy between Workhorse and Gitaly, + Workhorse can no longer connect. As a workaround, [disable the temporary `workhorse_use_sidechannel`](../administration/feature_flags.md#enable-or-disable-the-feature) + feature flag. If you need a proxy between Workhorse and Gitaly, use a TCP proxy. If you have feedback about this change, please go to [this issue](https://gitlab.com/gitlab-com/gl-infra/scalability/-/issues/1301). + +- In 14.1 we introduced a background migration that changes how we store merge request diff commits + in order to significantly reduce the amount of storage needed. + In 14.5 we introduce a set of migrations that wrap up this process by making sure + that all remaining jobs over the `merge_request_diff_commits` table are completed. + These jobs will have already been processed in most cases so that no extra time is necessary during an upgrade to 14.5. + But if there are remaining jobs, the deployment may take a few extra minutes to complete. + + All merge request diff commits will automatically incorporate these changes, and there are no + additional requirements to perform the upgrade. + Existing data in the `merge_request_diff_commits` table remains unpacked until you run `VACUUM FULL merge_request_diff_commits`. + But note that the `VACUUM FULL` operation locks and rewrites the entire `merge_request_diff_commits` table, + so the operation takes some time to complete and it blocks access to this table until the end of the process. + We advise you to only run this command while GitLab is not actively used or it is taken offline for the duration of the process. + The time it takes to complete depends on the size of the table, which can be obtained by using `select pg_size_pretty(pg_total_relation_size('merge_request_diff_commits'));`. + + For more information, refer to [this issue](https://gitlab.com/gitlab-org/gitlab/-/issues/331823). + ### 14.4.0 Git 2.33.x and later is required. We recommend you use the @@ -314,9 +362,11 @@ Git 2.33.x and later is required. We recommend you use the ### 14.3.0 -Ruby 2.7.4 is required. Refer to [the Ruby installation instructions](../install/installation.md#2-ruby) +- [Instances running 14.0.0 - 14.0.4 should not upgrade directly to GitLab 14.2 or later](#upgrading-to-later-14y-releases). +- Ensure [batched background migrations finish](#batched-background-migrations) before upgrading + to 14.3.Z from earlier GitLab 14 releases. +- Ruby 2.7.4 is required. Refer to [the Ruby installation instructions](../install/installation.md#2-ruby) for how to proceed. - - GitLab 14.3.0 contains post-deployment migrations to [address Primary Key overflow risk for tables with an integer PK](https://gitlab.com/groups/gitlab-org/-/epics/4785) for the tables listed below: - [`ci_builds.id`](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/70245) @@ -337,13 +387,10 @@ for how to proceed. ### 14.2.0 -- Due to an issue where `BatchedBackgroundMigrationWorkers` were - [not working](https://gitlab.com/gitlab-org/charts/gitlab/-/issues/2785#note_614738345) - for self-managed instances, a [fix was created](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/65106) - and a [14.0.Z](#1400) version was released. If you haven't updated to 14.0.5, you need - to update to at least 14.1.0 that contains the same fix before you update to 14.2. +- [Instances running 14.0.0 - 14.0.4 should not upgrade directly to GitLab 14.2 or later](#upgrading-to-later-14y-releases). +- Ensure [batched background migrations finish](#batched-background-migrations) before upgrading + to 14.2.Z from earlier GitLab 14 releases. - GitLab 14.2.0 contains background migrations to [address Primary Key overflow risk for tables with an integer PK](https://gitlab.com/groups/gitlab-org/-/epics/4785) for the tables listed below: - - [`ci_build_needs`](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/65216) - [`ci_build_trace_chunks`](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/66123) - [`ci_builds_runner_session`](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/66433) @@ -366,35 +413,24 @@ for how to proceed. ### 14.1.0 -- Due to an issue where `BatchedBackgroundMigrationWorkers` were - [not working](https://gitlab.com/gitlab-org/charts/gitlab/-/issues/2785#note_614738345) - for self-managed instances, a [fix was created](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/65106) - and a [14.0.Z](#1400) version was released. If you haven't updated to 14.0.5, you need - to update to at least 14.1.0 that contains the same fix before you update to - a later version. - - After you update to 14.1.0, - [batched background migrations need to finish](../user/admin_area/monitoring/background_migrations.md#check-the-status-of-background-migrations) - before you update to a later version. - - If the migrations are not finished and you try to update to a later version, - you'll see an error like: - - ```plaintext - Expected batched background migration for the given configuration to be marked as 'finished', but it is 'active': - ``` - - See how to [resolve this error](../user/admin_area/monitoring/background_migrations.md#database-migrations-failing-because-of-batched-background-migration-not-finished). +- [Instances running 14.0.0 - 14.0.4 should not upgrade directly to GitLab 14.2 or later](#upgrading-to-later-14y-releases) + but can upgrade to 14.1.Z. +- It is not required for instances already running 14.0.5 (or higher) to stop at 14.1.Z. + 14.1 is included on the upgrade path for the broadest compatibility + with self-managed installations, and ensure 14.0.0-14.0.4 installations do not + encounter issues with [batched background migrations](#batched-background-migrations). ### 14.0.0 +- Database changes made by the upgrade to GitLab 14.0 can take hours or days to complete on larger GitLab instances. + These [batched background migrations](#batched-background-migrations) update whole database tables to mitigate primary key overflow and must be finished before upgrading to GitLab 14.2 or higher. - Due to an issue where `BatchedBackgroundMigrationWorkers` were [not working](https://gitlab.com/gitlab-org/charts/gitlab/-/issues/2785#note_614738345) for self-managed instances, a [fix was created](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/65106) - that requires an update to at least 14.0.5. + that requires an update to at least 14.0.5. The fix was also released in [14.1.0](#1410). After you update to 14.0.5 or a later 14.0 patch version, - [batched background migrations need to finish](../user/admin_area/monitoring/background_migrations.md#check-the-status-of-background-migrations) + [batched background migrations need to finish](#batched-background-migrations) before you update to a later version. If the migrations are not finished and you try to update to a later version, @@ -413,6 +449,16 @@ for how to proceed. You should instead follow a [supported upgrade path](#upgrade-paths). - The support of PostgreSQL 11 [has been dropped](../install/requirements.md#database). Make sure to [update your database](https://docs.gitlab.com/omnibus/settings/database.html#upgrade-packaged-postgresql-server) to version 12 before updating to GitLab 14.0. +#### Upgrading to later 14.Y releases + +- Instances running 14.0.0 - 14.0.4 should not upgrade directly to GitLab 14.2 or later, + because of [batched background migrations](#batched-background-migrations). + 1. Upgrade first to either: + - 14.0.5 or a later 14.0.Z patch release. + - 14.1.0 or a later 14.1.Z patch release. + 1. [Batched background migrations need to finish](#batched-background-migrations) + before you update to a later version [and may take longer than usual](#1400). + ### 13.11.0 Git 2.31.x and later is required. We recommend you use the diff --git a/doc/update/package/index.md b/doc/update/package/index.md index 776a7111188..27845caed76 100644 --- a/doc/update/package/index.md +++ b/doc/update/package/index.md @@ -262,21 +262,36 @@ unable to add the `commit_message_regex_change` column. This results in the [backport migration of EE tables](https://gitlab.com/gitlab-org/gitlab/-/blob/cf00e431024018ddd82158f8a9210f113d0f4dbc/db/migrate/20190402150158_backport_enterprise_schema.rb#L1619) not working correctly. The backport migration assumes that certain tables in the database do not exist when running CE. -To fix this issue, manually add the missing `commit_message_negative_regex` column and restart GitLab: +To fix this issue: -```shell -# Access psql -sudo gitlab-rails dbconsole +1. Start a database console: -# Add the missing column -ALTER TABLE push_rules ADD COLUMN commit_message_negative_regex VARCHAR; + In GitLab 14.2 and later: -# Exit psql -\q + ```shell + sudo gitlab-rails dbconsole --database main + ``` -# Restart GitLab -sudo gitlab-ctl restart -``` + In GitLab 14.1 and earlier: + + ```shell + sudo gitlab-rails dbconsole + ``` + +1. Manually add the missing `commit_message_negative_regex` column: + + ```sql + ALTER TABLE push_rules ADD COLUMN commit_message_negative_regex VARCHAR; + + # Exit psql + \q + ``` + +1. Restart GitLab: + + ```shell + sudo gitlab-ctl restart + ``` ### Error `Failed to connect to the internal GitLab API` on a separate GitLab Pages server diff --git a/doc/update/patch_versions.md b/doc/update/patch_versions.md index d09f19d143b..a2d672e00ac 100644 --- a/doc/update/patch_versions.md +++ b/doc/update/patch_versions.md @@ -20,6 +20,10 @@ It's useful to make a backup just in case things go south. Depending on the inst ### 1. Stop server ```shell +# For systems running systemd +sudo systemctl stop gitlab.target + +# For systems running SysV init sudo service gitlab stop ``` @@ -108,6 +112,11 @@ Please follow the [install instruction](../integration/elasticsearch.md#install- ### 9. Start application ```shell +# For systems running systemd +sudo systemctl start gitlab.target +sudo systemctl restart nginx.service + +# For systems running SysV init sudo service gitlab start sudo service nginx restart ``` diff --git a/doc/update/upgrading_from_source.md b/doc/update/upgrading_from_source.md index d882de62c06..4343b464ba6 100644 --- a/doc/update/upgrading_from_source.md +++ b/doc/update/upgrading_from_source.md @@ -54,6 +54,10 @@ sudo -u git -H bundle exec rake gitlab:backup:create RAILS_ENV=production ### 2. Stop server ```shell +# For systems running systemd +sudo systemctl stop gitlab.target + +# For systems running SysV init sudo service gitlab stop ``` @@ -229,7 +233,29 @@ ActionMailer::Base.delivery_method = :smtp See [`smtp_settings.rb.sample`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/initializers/smtp_settings.rb.sample#L13) as an example. -#### Init script +#### Configure systemd units + +If using the SysV init script, see [Configure SysV init script](#configure-sysv-init-script). + +Check if the systemd units have been updated: + +```shell +cd /home/git/gitlab + +git diff origin/PREVIOUS_BRANCH:lib/support/systemd origin/BRANCH:lib/support/systemd +``` + +Copy them over: + +```shell +sudo mkdir -p /usr/local/lib/systemd/system +sudo cp lib/support/systemd/* /usr/local/lib/systemd/system/ +sudo systemctl daemon-reload +``` + +#### Configure SysV init script + +If using systemd units, see [Configure systemd units](#configure-systemd-units). There might be new configuration options available for [`gitlab.default.example`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/support/init.d/gitlab.default.example). @@ -249,7 +275,7 @@ cd /home/git/gitlab sudo cp lib/support/init.d/gitlab /etc/init.d/gitlab ``` -For Ubuntu 16.04.1 LTS: +If you are using the init script on a system running systemd as init, because you have not switched to native systemd units yet, run: ```shell sudo systemctl daemon-reload @@ -341,6 +367,11 @@ sudo -u git -H make ### 15. Start application ```shell +# For systems running systemd +sudo systemctl start gitlab.target +sudo systemctl restart nginx.service + +# For systems running SysV init sudo service gitlab start sudo service nginx restart ``` diff --git a/doc/update/zero_downtime.md b/doc/update/zero_downtime.md index 7a74435267f..a311731cadd 100644 --- a/doc/update/zero_downtime.md +++ b/doc/update/zero_downtime.md @@ -1,17 +1,17 @@ --- stage: Enablement group: Distribution -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- # Zero downtime upgrades **(FREE SELF)** -Starting with GitLab 9.1.0 it's possible to upgrade to a newer major, minor, or -patch version of GitLab without having to take your GitLab instance offline. -However, for this to work there are the following requirements: +It's possible to upgrade to a newer major, minor, or patch version of GitLab +without having to take your GitLab instance offline. However, for this to work +there are the following requirements: -- You can only upgrade 1 minor release at a time. So from 9.1 to 9.2, not to - 9.3. If you skip releases, database modifications may be run in the wrong +- You can only upgrade one minor release at a time. So from 13.1 to 13.2, not to + 13.3. If you skip releases, database modifications may be run in the wrong sequence [and leave the database schema in a broken state](https://gitlab.com/gitlab-org/gitlab/-/issues/321542). - You have to use [post-deployment migrations](../development/post_deployment_migrations.md). - You are using PostgreSQL. Starting from GitLab 12.1, MySQL is not supported. @@ -36,10 +36,10 @@ to re-read any database changes that have been made by post-deployment migration Most of the time you can safely upgrade from a patch release to the next minor release if the patch release is not the latest. For example, upgrading from -9.1.1 to 9.2.0 should be safe even if 9.1.2 has been released. We do recommend +14.1.1 to 14.2.0 should be safe even if 14.1.2 has been released. We do recommend you check the release posts of any releases between your current and target version just in case they include any migrations that may require you to upgrade -1 release at a time. +one release at a time. Some releases may also include so called "background migrations". These migrations are performed in the background by Sidekiq and are often used for @@ -63,21 +63,21 @@ the migrations that are being performed. To help explain this, let's look at some examples: -**Example 1:** You are running a large GitLab installation using version 9.4.2, -which is the latest patch release of 9.4. When GitLab 9.5.0 is released this -installation can be safely upgraded to 9.5.0 without requiring downtime if the -requirements mentioned above are met. You can also skip 9.5.0 and upgrade to -9.5.1 after it's released, but you **can not** upgrade straight to 9.6.0; you -_have_ to first upgrade to a 9.5.Z release. - -**Example 2:** You are running a large GitLab installation using version 9.4.2, -which is the latest patch release of 9.4. GitLab 9.5 includes some background -migrations, and 10.0 requires these to be completed (processing any -remaining jobs for you). Skipping 9.5 is not possible without downtime, and due +**Example 1:** You are running a large GitLab installation using version 13.4.2, +which is the latest patch release of 13.4. When GitLab 13.5.0 is released this +installation can be safely upgraded to 13.5.0 without requiring downtime if the +requirements mentioned above are met. You can also skip 13.5.0 and upgrade to +13.5.1 after it's released, but you **can not** upgrade straight to 13.6.0; you +_have_ to first upgrade to a 13.5.Z release. + +**Example 2:** You are running a large GitLab installation using version 13.4.2, +which is the latest patch release of 13.4. GitLab 13.5 includes some background +migrations, and 14.0 requires these to be completed (processing any +remaining jobs for you). Skipping 13.5 is not possible without downtime, and due to the background migrations would require potentially hours of downtime depending on how long it takes for the background migrations to complete. To -work around this you have to upgrade to 9.5.Z first, then wait at least a -week before upgrading to 10.0. +work around this you have to upgrade to 13.5.Z first, then wait at least a +week before upgrading to 14.0. **Example 3:** You use MySQL as the database for GitLab. Any upgrade to a new major/minor release requires downtime. If a release includes any background @@ -89,7 +89,7 @@ meet the other online upgrade requirements mentioned above. Before following these instructions, note the following **important** information: -- You can only upgrade 1 minor release at a time. So from 13.6 to 13.7, not to 13.8. +- You can only upgrade one minor release at a time. So from 13.6 to 13.7, not to 13.8. If you attempt more than one minor release, the upgrade may fail. - On single-node Omnibus deployments, updates with no downtime are not possible when using Puma because Puma always requires a complete restart. This is because the @@ -100,6 +100,8 @@ Before following these instructions, note the following **important** informatio these instructions, **it is not possible to always achieve true zero downtime updates**. Users may see some connections timeout or be refused for a few minutes, depending on which services need to restart. +- On Omnibus deployments, the `/etc/gitlab/gitlab.rb` configuration file must **not** have + `gitlab_rails['auto_migrate'] = true`. 1. Create an empty file at `/etc/gitlab/skip-auto-reconfigure`. This prevents upgrades from running `gitlab-ctl reconfigure`, which by default automatically stops GitLab, runs all database migrations, and restarts GitLab. @@ -156,7 +158,7 @@ you've completed these steps. ## Multi-node / HA deployment -You can only upgrade 1 minor release at a time. So from 13.6 to 13.7, not to 13.8. +You can only upgrade one minor release at a time. So from 13.6 to 13.7, not to 13.8. If you attempt more than one minor release, the upgrade may fail. ### Use a load balancer in front of web (Puma) nodes @@ -208,7 +210,9 @@ load balancer to latest GitLab version. If you are an Enterprise Edition user, replace `gitlab-ce` with `gitlab-ee` in the above command. - 1. Get the regular migrations and latest code in place: + 1. Get the regular migrations and latest code in place. Before running this step, + the deploy node's `/etc/gitlab/gitlab.rb` configuration file must have + `gitlab_rails['auto_migrate'] = true` to permit regular migrations. ```shell sudo SKIP_POST_DEPLOYMENT_MIGRATIONS=true gitlab-ctl reconfigure |