Add latest changes from gitlab-org/gitlab@16-4-stable-eev16.4.0-rc42

8 files changed, 2954 insertions, 1824 deletions

diff --git a/doc/update/background_migrations.md b/doc/update/background_migrations.md
index cbdcfef3bae..5a4b19016f8 100644
--- a/doc/update/background_migrations.md
+++ b/doc/update/background_migrations.md
@@ -30,7 +30,7 @@ are created by GitLab developers and run automatically on upgrade. However, such
 limited in scope to help with migrating some `integer` database columns to `bigint`. This is needed to
 prevent integer overflow for some tables.
 
-Some installations [may need to run GitLab 14.0 for at least a day](index.md#1400)
+Some installations [may need to run GitLab 14.0 for at least a day](versions/gitlab_14_changes.md#1400)
 to complete the database changes introduced by that upgrade.
 
 Batched background migrations are handled by Sidekiq and
@@ -67,7 +67,7 @@ Prerequisites:
 
 To check the status of batched background migrations:
 
-1. On the left sidebar, expand the top-most chevron (**{chevron-down}**).
+1. On the left sidebar, select **Search or go to**.
 1. Select **Admin Area**.
 1. Select **Monitoring > Background Migrations**.
 1. Select **Queued** or **Finalizing** to see incomplete migrations,
@@ -190,26 +190,34 @@ the number of batched background migrations executed in parallel:
 ApplicationSetting.update_all(database_max_running_batched_background_migrations: 4)
 ```
 
-### Fix and retry failed batched background migrations
+### Resolve failed batched background migrations
+
+If a batched background migration fails, [fix and retry](#fix-and-retry-the-migration) it.
+If the migration continues to fail with an error, either:
+
+- [Finish the failed migration manually](#finish-a-failed-migration-manually)
+- [Mark the failed migration finished](#mark-a-failed-migration-finished)
+
+#### Fix and retry the migration
 
 > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/67504) in GitLab 14.3.
 
-If you [check the status](#check-the-status-of-batched-background-migrations) of batched background migrations,
-some migrations might display in the **Failed** tab with a **failed** status:
+All failed batched background migrations must be resolved to upgrade to a newer
+version of GitLab. If you [check the status](#check-the-status-of-batched-background-migrations)
+of batched background migrations, some migrations might display in the **Failed** tab
+with a **failed** status:
 
 ![failed batched background migrations table](img/batched_background_migrations_failed_v14_3.png)
 
-You must resolve all failed batched background migrations to upgrade to a newer
-version of GitLab.
-
 To determine why the batched background migration failed,
-[view the failure error logs](../development/database/batched_background_migrations.md#viewing-failure-error-logs) or:
+[view the failure error logs](../development/database/batched_background_migrations.md#viewing-failure-error-logs)
+or view error information in the UI.
 
 Prerequisites:
 
 - You must have administrator access to the instance.
 
-1. On the left sidebar, expand the top-most chevron (**{chevron-down}**).
+1. On the left sidebar, select **Search or go to**.
 1. Select **Admin Area**.
 1. Select **Monitoring > Background Migrations**.
 1. Select the **Failed** tab. This displays a list of failed batched background migrations.
@@ -219,13 +227,13 @@ Prerequisites:
 If you are a GitLab customer, consider opening a [Support Request](https://support.gitlab.com/hc/en-us/requests/new)
 to debug why the batched background migrations failed.
 
-To correct the problem, you can retry the failed batched background migrations:
+To correct the problem, you can retry the failed migration.
 
 Prerequisites:
 
 - You must have administrator access to the instance.
 
-1. On the left sidebar, expand the top-most chevron (**{chevron-down}**).
+1. On the left sidebar, select **Search or go to**.
 1. Select **Admin Area**.
 1. Select **Monitoring > Background Migrations**.
 1. Select the **Failed** tab. This displays a list of failed batched background migrations.
@@ -235,6 +243,111 @@ To monitor the retried batched background migrations, you can
 [check the status of batched background migrations](#check-the-status-of-batched-background-migrations)
 on a regular interval.
 
+#### Finish a failed migration manually
+
+> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/62634) in GitLab 14.1.
+
+To manually finish a batched background migration that failed with an error,
+use the information in the failure error logs or the database:
+
+::Tabs
+
+:::TabTitle From the failure error logs
+
+1. [View the failure error logs](../development/database/batched_background_migrations.md#viewing-failure-error-logs)
+   and look for an `An error has occurred, all later migrations canceled` error message, like this:
+
+   ```plaintext
+   StandardError: An error has occurred, all later migrations canceled:
+
+   Expected batched background migration for the given configuration to be marked as
+   'finished', but it is 'active':
+     {:job_class_name=>"CopyColumnUsingBackgroundMigrationJob",
+      :table_name=>"push_event_payloads",
+      :column_name=>"event_id",
+      :job_arguments=>[["event_id"],
+      ["event_id_convert_to_bigint"]]
+     }
+   ```
+
+1. Run the following command, replacing the values in angle brackets with the correct arguments:
+
+   ```shell
+   sudo gitlab-rake gitlab:background_migrations:finalize[<job_class_name>,<table_name>,<column_name>,'<job_arguments>']
+   ```
+
+   When dealing with multiple arguments, such as `[["id"],["id_convert_to_bigint"]]`, escape the
+   comma between each argument with a backslash <code>&#92;</code> to prevent an invalid character error.
+   For example, to finish the migration from the previous step:
+
+   ```shell
+   sudo gitlab-rake gitlab:background_migrations:finalize[CopyColumnUsingBackgroundMigrationJob,push_event_payloads,event_id,'[["event_id"]\, ["event_id_convert_to_bigint"]]']
+   ```
+
+:::TabTitle From the database
+
+  1. [Check the status](#check-the-status-of-batched-background-migrations) of the
+     migration in the database.
+  1. Use the query results to construct a migration command, replacing the values
+     in angle brackets with the correct arguments:
+
+     ```shell
+     sudo gitlab-rake gitlab:background_migrations:finalize[<job_class_name>,<table_name>,<column_name>,'<job_arguments>']
+     ```
+
+     For example, if the query returns this data:
+
+     - `job_class_name`: `CopyColumnUsingBackgroundMigrationJob`
+     - `table_name`: `events`
+     - `column_name`: `id`
+     - `job_arguments`: `[["id"], ["id_convert_to_bigint"]]`
+
+   When dealing with multiple arguments, such as `[["id"],["id_convert_to_bigint"]]`, escape the
+   comma between each argument with a backslash <code>&#92;</code> to prevent an invalid character error.
+   The command should be:
+
+   ```shell
+   sudo gitlab-rake gitlab:background_migrations:finalize[CopyColumnUsingBackgroundMigrationJob,events,id,'[["id"]\, ["id_convert_to_bigint"]]']
+   ```
+
+::EndTabs
+
+#### Mark a failed migration finished
+
+WARNING:
+[Contact GitLab Support](https://about.gitlab.com/support/#contact-support) before using
+these instructions. This action can cause data loss, and make your instance fail
+in ways that are difficult to recover from.
+
+There can be cases where the background migration fails: when jumping too many version upgrades,
+or backward-incompatible database schema changes. (For an example, see [issue 393216](https://gitlab.com/gitlab-org/gitlab/-/issues/393216)).
+Failed background migrations prevent further application upgrades.
+
+When the background migration is determined to be "safe" to skip, the migration can be manually marked finished:
+
+WARNING:
+Make sure you create a backup before proceeding.
+
+```ruby
+# Start the rails console
+
+connection = ApplicationRecord.connection # or Ci::ApplicationRecord.connection, depending on which DB was the migration scheduled
+
+Gitlab::Database::SharedModel.using_connection(connection) do
+  migration = Gitlab::Database::BackgroundMigration::BatchedMigration.find_for_configuration(
+    Gitlab::Database.gitlab_schemas_for_connection(connection),
+    'BackfillUserDetailsFields',
+    :users,
+    :id,
+    []
+  )
+
+  # mark all jobs completed
+  migration.batched_jobs.update_all(status: Gitlab::Database::BackgroundMigration::BatchedJob.state_machine.states['succeeded'].value)
+  migration.update_attribute(:status, Gitlab::Database::BackgroundMigration::BatchedMigration.state_machine.states[:finished].value)
+end
+```
+
 ## Background migrations
 
 In GitLab 13, background migrations were not batched. In GitLab 14 and later, this
@@ -303,6 +416,8 @@ sudo -u git -H bundle exec rails runner -e production 'puts Gitlab::Database::Ba
 
 ## Troubleshooting
 
+<!-- Linked from lib/gitlab/database/migrations/batched_background_migration_helpers.rb -->
+
 ### Database migrations failing because of batched background migration not finished
 
 When updating to GitLab version 14.2 or later, database migrations might fail with a message like:
@@ -319,8 +434,8 @@ Expected batched background migration for the given configuration to be marked a
   }
 ```
 
-First, check if you have followed the [version-specific upgrade instructions for 14.2](../update/index.md#1420).
-If you have, you can [manually finish the batched background migration](#manually-finishing-a-batched-background-migration).
+First, check if you have followed the [version-specific upgrade instructions for 14.2](../update/versions/gitlab_14_changes.md#1420).
+If you have, you can [manually finish the batched background migration](#finish-a-failed-migration-manually)).
 If you haven't, choose one of the following methods:
 
 1. [Rollback and upgrade](#roll-back-and-follow-the-required-upgrade-path) through one of the required
@@ -334,7 +449,7 @@ version and manually ensuring that the batched migrations complete successfully.
 1. Update to either 14.0.5 or 14.1 **before** updating to 14.2+
 1. [Check the status](#check-the-status-of-batched-background-migrations) of the batched background migrations and
 make sure they are all marked as finished before attempting to upgrade again. If any remain marked as active,
-you can [manually finish them](#manually-finishing-a-batched-background-migration).
+you can [manually finish them](#finish-a-failed-migration-manually).
 
 #### Roll forward and finish the migrations on the upgraded version
 
@@ -344,7 +459,7 @@ To run all the batched background migrations, it can take a significant amount o
 depending on the size of your GitLab installation.
 
 1. [Check the status](#check-the-status-of-batched-background-migrations) of the batched background migrations in the
-database, and [manually run them](#manually-finishing-a-batched-background-migration) with the appropriate
+database, and [manually run them](#finish-a-failed-migration-manually) with the appropriate
 arguments until the status query returns no rows.
 1. When the status of all of all them is marked as complete, re-run migrations for your installation.
 1. [Complete the database migrations](../administration/raketasks/maintenance.md#run-incomplete-database-migrations) from your GitLab upgrade:
@@ -368,89 +483,9 @@ version and wait for the batched background migrations to finish.
 
 1. [Check the status](#check-the-status-of-batched-background-migrations) of the batched background migration from
 the error message, and make sure it is listed as finished. If it is still active, either wait until it is done,
-or [manually finish it](#manually-finishing-a-batched-background-migration).
+or [manually finish it](#finish-a-failed-migration-manually).
 1. Re-run migrations for your installation, so the remaining post-deployment migrations finish.
 
-### Manually finishing a batched background migration
-
-> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/62634) in GitLab 14.1
-
-If you need to manually finish a batched background migration due to an
-error, you can run:
-
-```shell
-sudo gitlab-rake gitlab:background_migrations:finalize[<job_class_name>,<table_name>,<column_name>,'<job_arguments>']
-```
-
-Replace the values in angle brackets with the correct
-arguments. For example, if you receive an error similar to this:
-
-```plaintext
-StandardError: An error has occurred, all later migrations canceled:
-
-Expected batched background migration for the given configuration to be marked as
-'finished', but it is 'active':
-  {:job_class_name=>"CopyColumnUsingBackgroundMigrationJob",
-   :table_name=>"push_event_payloads",
-   :column_name=>"event_id",
-   :job_arguments=>[["event_id"],
-   ["event_id_convert_to_bigint"]]
-  }
-```
-
-Plug the arguments from the error message into the command:
-
-```shell
-sudo gitlab-rake gitlab:background_migrations:finalize[CopyColumnUsingBackgroundMigrationJob,push_event_payloads,event_id,'[["event_id"]\, ["event_id_convert_to_bigint"]]']
-```
-
-If you need to manually run a batched background migration to continue an upgrade, you can
-[check the status](#check-the-status-of-batched-background-migrations) in the database and get the
-arguments from the query results. For example, if the query returns this:
-
-```plaintext
-            job_class_name             | table_name | column_name |           job_arguments
----------------------------------------+------------+-------------+------------------------------------
- CopyColumnUsingBackgroundMigrationJob | events     | id          | [["id"], ["id_convert_to_bigint"]]
- ```
-
-The results from the query can be plugged into the command:
-
-```shell
-sudo gitlab-rake gitlab:background_migrations:finalize[CopyColumnUsingBackgroundMigrationJob,events,id,'[["id"]\, ["id_convert_to_bigint"]]']
-```
-
-#### Mark a batched migration finished
-
-There can be cases where the background migration fails: when jumping too many version upgrades,
-or backward-incompatible database schema changes. (For an example, see [issue 393216](https://gitlab.com/gitlab-org/gitlab/-/issues/393216)).
-Failed background migrations prevent further application upgrades.
-
-When the background migration is determined to be "safe" to skip, the migration can be manually marked finished:
-
-WARNING:
-Make sure you create a backup before proceeding.
-
-```ruby
-# Start the rails console
-
-connection = ApplicationRecord.connection # or Ci::ApplicationRecord.connection, depending on which DB was the migration scheduled
-
-Gitlab::Database::SharedModel.using_connection(connection) do
-  migration = Gitlab::Database::BackgroundMigration::BatchedMigration.find_for_configuration(
-    Gitlab::Database.gitlab_schemas_for_connection(connection),
-    'BackfillUserDetailsFields',
-    :users,
-    :id,
-    []
-  )
-
-  # mark all jobs completed
-  migration.batched_jobs.update_all(status: Gitlab::Database::BackgroundMigration::BatchedJob.state_machine.states['succeeded'].value)
-  migration.update_attribute(:status, Gitlab::Database::BackgroundMigration::BatchedMigration.state_machine.states[:finished].value)
-end
-```
-
 ### The `BackfillNamespaceIdForNamespaceRoute` batched migration job fails
 
 In GitLab 14.8, the `BackfillNamespaceIdForNamespaceRoute` batched background migration job
@@ -516,26 +551,26 @@ of Sidekiq jobs that perform various database or file updates.
   `BackfillDraftStatusOnMergeRequests` can be permanently stuck in a
   **pending** state across upgrades when the instance lacks records that match
   the migration's target. To clean up this stuck migration, see the
-  [14.2.0 version-specific instructions](index.md#1420).
+  [14.2.0 version-specific instructions](versions/gitlab_14_changes.md#1420).
 - GitLab 14.4 introduced an issue where a background migration named
   `PopulateTopicsTotalProjectsCountCache` can be permanently stuck in a
   **pending** state across upgrades when the instance lacks records that match
   the migration's target. To clean up this stuck migration, see the
-  [14.4.0 version-specific instructions](index.md#1440).
+  [14.4.0 version-specific instructions](versions/gitlab_14_changes.md#1440).
 - GitLab 14.5 introduced an issue where a background migration named
   `UpdateVulnerabilityOccurrencesLocation` can be permanently stuck in a
   **pending** state across upgrades when the instance lacks records that match
   the migration's target. To clean up this stuck migration, see the
-  [14.5.0 version-specific instructions](index.md#1450).
+  [14.5.0 version-specific instructions](versions/gitlab_14_changes.md#1450).
 - GitLab 14.8 introduced an issue where a background migration named
   `PopulateTopicsNonPrivateProjectsCount` can be permanently stuck in a
   **pending** state across upgrades. To clean up this stuck migration, see the
-  [14.8.0 version-specific instructions](index.md#1480).
+  [14.8.0 version-specific instructions](versions/gitlab_14_changes.md#1480).
 - GitLab 14.9 introduced an issue where a background migration named
   `ResetDuplicateCiRunnersTokenValuesOnProjects` can be permanently stuck in a
   **pending** state across upgrades when the instance lacks records that match
   the migration's target. To clean up this stuck migration, see the
-  [14.9.0 version-specific instructions](index.md#1490).
+  [14.9.0 version-specific instructions](versions/gitlab_14_changes.md#1490).
 
 For other background migrations stuck in pending, run the following check. If
 it returns non-zero and the count does not decrease over time, follow the rest
diff --git a/doc/update/deprecations.md b/doc/update/deprecations.md
index 08a5a3a7549..153119da1cc 100644
--- a/doc/update/deprecations.md
+++ b/doc/update/deprecations.md
@@ -46,6 +46,108 @@ For deprecation reviewers (Technical Writers only):
 {::options parse_block_html="true" /}
 
 <div class="js-deprecation-filters"></div>
+<div class="milestone-wrapper" data-milestone="18.0">
+
+## GitLab 18.0
+
+<div class="deprecation breaking-change" data-milestone="18.0">
+
+### GitLab Runner registration token in Runner Operator
+
+<div class="deprecation-notes">
+- Announced in GitLab <span class="milestone">15.6</span>
+- Removal in GitLab <span class="milestone">18.0</span> ([breaking change](https://docs.gitlab.com/ee/update/terminology.html#breaking-change))
+- To discuss this change or learn more, see the [deprecation issue](https://gitlab.com/gitlab-org/gitlab/-/issues/382077).
+</div>
+
+The [`runner-registration-token`](https://docs.gitlab.com/runner/install/operator.html#install-the-kubernetes-operator) parameter that uses the OpenShift and Kubernetes Vanilla Operator to install a runner on Kubernetes is deprecated. Authentication tokens will be used to register runners instead. Registration tokens, and support for certain configuration arguments,
+will be removed in GitLab 18.0. For more information, see [Migrating to the new runner registration workflow](../ci/runners/new_creation_workflow.md).
+The configuration arguments disabled for authentication tokens are:
+
+- `--locked`
+- `--access-level`
+- `--run-untagged`
+- `--tag-list`
+
+This change is a breaking change. You should use an [authentication token](../ci/runners/register_runner.md) in the `gitlab-runner register` command instead.
+
+</div>
+
+<div class="deprecation breaking-change" data-milestone="18.0">
+
+### Registration tokens and server-side runner arguments in `gitlab-runner register` command
+
+<div class="deprecation-notes">
+- Announced in GitLab <span class="milestone">15.6</span>
+- Removal in GitLab <span class="milestone">18.0</span> ([breaking change](https://docs.gitlab.com/ee/update/terminology.html#breaking-change))
+- To discuss this change or learn more, see the [deprecation issue](https://gitlab.com/gitlab-org/gitlab/-/issues/380872).
+</div>
+
+Registration tokens and certain configuration arguments in the command `gitlab-runner register` that [registers](https://docs.gitlab.com/runner/register/) a runner, are deprecated.
+Authentication tokens will be used to register runners instead. Registration tokens, and support for certain configuration arguments,
+will be removed in GitLab 18.0. For more information, see [Migrating to the new runner registration workflow](../ci/runners/new_creation_workflow.md).
+The configuration arguments disabled for authentication tokens are:
+
+- `--locked`
+- `--access-level`
+- `--run-untagged`
+- `--maximum-timeout`
+- `--paused`
+- `--tag-list`
+- `--maintenance-note`
+
+This change is a breaking change. You should [create a runner in the UI](../ci/runners/register_runner.md) to add configurations, and use the authentication token in the `gitlab-runner register` command instead.
+
+</div>
+
+<div class="deprecation breaking-change" data-milestone="18.0">
+
+### Support for REST API endpoints that reset runner registration tokens
+
+<div class="deprecation-notes">
+- Announced in GitLab <span class="milestone">15.7</span>
+- Removal in GitLab <span class="milestone">18.0</span> ([breaking change](https://docs.gitlab.com/ee/update/terminology.html#breaking-change))
+- To discuss this change or learn more, see the [deprecation issue](https://gitlab.com/gitlab-org/gitlab/-/issues/383341).
+</div>
+
+The support for runner registration tokens is deprecated. As a consequence, the REST API endpoints to reset a registration token are also deprecated and will
+return the HTTP `410 Gone` status code in GitLab 18.0.
+The deprecated endpoints are:
+
+- `POST /runners/reset_registration_token`
+- `POST /projects/:id/runners/reset_registration_token`
+- `POST /groups/:id/runners/reset_registration_token`
+
+We plan to implement a new method to bind runners to a GitLab instance
+as part of the new [GitLab Runner token architecture](https://docs.gitlab.com/ee/ci/runners/new_creation_workflow.html).
+The work is planned in [this epic](https://gitlab.com/groups/gitlab-org/-/epics/7633).
+This new architecture introduces a new method for registering runners and will eliminate the legacy
+[runner registration token](https://docs.gitlab.com/ee/security/token_overview.html#runner-registration-tokens).
+From GitLab 18.0 and later, the runner registration methods implemented by the new GitLab Runner token architecture will be the only supported methods.
+
+</div>
+
+<div class="deprecation breaking-change" data-milestone="18.0">
+
+### `runnerRegistrationToken` parameter for GitLab Runner Helm Chart
+
+<div class="deprecation-notes">
+- Announced in GitLab <span class="milestone">15.6</span>
+- Removal in GitLab <span class="milestone">18.0</span> ([breaking change](https://docs.gitlab.com/ee/update/terminology.html#breaking-change))
+- To discuss this change or learn more, see the [deprecation issue](https://gitlab.com/gitlab-org/gitlab/-/issues/381111).
+</div>
+
+The [`runnerRegistrationToken`](https://docs.gitlab.com/runner/install/kubernetes.html#required-configuration) parameter to use the GitLab Helm Chart to install a runner on Kubernetes is deprecated.
+
+We plan to implement a new method to bind runners to a GitLab instance leveraging `runnerToken`
+as part of the new [GitLab Runner token architecture](https://docs.gitlab.com/ee/ci/runners/new_creation_workflow.html).
+The work is planned in [this epic](https://gitlab.com/groups/gitlab-org/-/epics/7633).
+
+From GitLab 18.0 and later, the methods to register runners introduced by the new GitLab Runner token architecture will be the only supported methods.
+
+</div>
+</div>
+
 <div class="milestone-wrapper" data-milestone="17.0">
 
 ## GitLab 17.0
@@ -131,6 +233,30 @@ These three variables will be removed in GitLab 17.0.
 
 <div class="deprecation breaking-change" data-milestone="17.0">
 
+### Default CI/CD job token (`CI_JOB_TOKEN`) scope changed
+
+<div class="deprecation-notes">
+- Announced in GitLab <span class="milestone">15.9</span>
+- Removal in GitLab <span class="milestone">17.0</span> ([breaking change](https://docs.gitlab.com/ee/update/terminology.html#breaking-change))
+- To discuss this change or learn more, see the [deprecation issue](https://gitlab.com/gitlab-org/gitlab/-/issues/383084).
+</div>
+
+In GitLab 14.4 we introduced the ability to [limit your project's CI/CD job token](https://docs.gitlab.com/ee/ci/jobs/ci_job_token.html#limit-your-projects-job-token-access) (`CI_JOB_TOKEN`) access to make it more secure. You can prevent job tokens **from your project's** pipelines from being used to **access other projects**. When enabled with no other configuration, your pipelines cannot access other projects. To use the job token to access other projects from your pipeline, you must list those projects explicitly in the **Limit CI_JOB_TOKEN access** setting's allowlist, and you must be a maintainer in all the projects.
+
+The job token functionality was updated in 15.9 with a better security setting to [allow access to your project with a job token](https://docs.gitlab.com/ee/ci/jobs/ci_job_token.html#allow-access-to-your-project-with-a-job-token). When enabled with no other configuration, job tokens **from other projects** cannot **access your project**. Similar to the older setting, you can optionally allow other projects to access your project with a job token if you list those projects explicitly in the **Allow access to this project with a CI_JOB_TOKEN** setting's allowlist. With this new setting, you must be a maintainer in your own project, but only need to have the Guest role in the other projects.
+
+The **Limit** setting was deprecated in 16.0 in preference of the better **Allow access** setting and **Limit** setting was disabled by default for all new projects. From this point forward, if the **Limit** setting is disabled in any project, it will not be possible to re-enable this setting in 16.0 or later.
+
+In 17.0, we will remove the **Limit** setting completely, and set the **Allow access** setting to enabled for all projects. This change ensures a higher level of security between projects. If you currently use the **Limit** setting, you should update your projects to use the **Allow access** setting instead. If other projects access your project with a job token, you must add them to the **Allow access** allowlist.
+
+To prepare for this change, users on GitLab.com or self-managed GitLab 15.9 or later can enable the **Allow access** setting now and add the other projects. It will not be possible to disable the setting in 17.0 or later.
+
+In 16.3, the names of these settings were changed to clarify their meanings: the deprecated **Limit CI_JOB_TOKEN access** setting is now called **Limit access _from_ this project**, and the newer **Allow access to this project with a CI_JOB_TOKEN** setting is now called **Limit access _to_ this project**.
+
+</div>
+
+<div class="deprecation breaking-change" data-milestone="17.0">
+
 ### Deprecate Windows CMD in GitLab Runner
 
 <div class="deprecation-notes">
@@ -189,6 +315,20 @@ The GitLab Runner Kubernetes executor setting, `terminationGracePeriodSeconds`,
 
 <div class="deprecation breaking-change" data-milestone="17.0">
 
+### Deprecate change vulnerability status from the Developer role
+
+<div class="deprecation-notes">
+- Announced in GitLab <span class="milestone">16.4</span>
+- Removal in GitLab <span class="milestone">17.0</span> ([breaking change](https://docs.gitlab.com/ee/update/terminology.html#breaking-change))
+- To discuss this change or learn more, see the [deprecation issue](https://gitlab.com/gitlab-org/gitlab/-/issues/424133).
+</div>
+
+The ability for Developers to change the status of vulnerabilities is now deprecated.  We plan to make a breaking change in the upcoming GitLab 17.0 release to remove this ability from the Developer role.  Users who wish to continue to grant this permission to developers can [create a custom role](https://docs.gitlab.com/ee/user/permissions.html#custom-roles) for their developers and add in the `admin_vulnerability` permission to give them this access.
+
+</div>
+
+<div class="deprecation breaking-change" data-milestone="17.0">
+
 ### Deprecate field `hasSolutions` from GraphQL VulnerabilityType
 
 <div class="deprecation-notes">
@@ -273,6 +413,25 @@ To avoid any disruptions, you should replace `filepath` with `direct_asset_path`
 
 <div class="deprecation breaking-change" data-milestone="17.0">
 
+### Geo: Legacy replication details routes for designs and projects deprecated
+
+<div class="deprecation-notes">
+- Announced in GitLab <span class="milestone">16.4</span>
+- Removal in GitLab <span class="milestone">17.0</span> ([breaking change](https://docs.gitlab.com/ee/update/terminology.html#breaking-change))
+- To discuss this change or learn more, see the [deprecation issue](https://gitlab.com/gitlab-org/gitlab/-/issues/424002).
+</div>
+
+As part of the migration of legacy data types to the [Geo self-service framework](https://docs.gitlab.com/ee/development/geo/framework.html), the following replication details routes are deprecated:
+
+- Designs `/admin/geo/replication/designs` replaced by `/admin/geo/sites/<Geo Node/Site ID>/replication/design_management_repositories`
+- Projects `/admin/geo/replication/projects` replaced by `/admin/geo/sites/<Geo Node/Site ID>/replication/projects`
+
+From GitLab 16.4 to 17.0, lookups for the legacy routes will automatically be redirected to the new routes. We will remove the redirections in 17.0. Please update any bookmarks or scripts that may use the legacy routes.
+
+</div>
+
+<div class="deprecation breaking-change" data-milestone="17.0">
+
 ### GitLab Helm chart values `gitlab.kas.privateApi.*` are deprecated
 
 <div class="deprecation-notes">
@@ -310,29 +469,6 @@ are deprecated and will be removed from the GraphQL API. For installation instru
 
 <div class="deprecation breaking-change" data-milestone="17.0">
 
-### GitLab Runner registration token in Runner Operator
-
-<div class="deprecation-notes">
-- Announced in GitLab <span class="milestone">15.6</span>
-- Removal in GitLab <span class="milestone">17.0</span> ([breaking change](https://docs.gitlab.com/ee/update/terminology.html#breaking-change))
-- To discuss this change or learn more, see the [deprecation issue](https://gitlab.com/gitlab-org/gitlab/-/issues/382077).
-</div>
-
-The [`runner-registration-token`](https://docs.gitlab.com/runner/install/operator.html#install-the-kubernetes-operator) parameter that uses the OpenShift and Kubernetes Vanilla Operator to install a runner on Kubernetes is deprecated. Authentication tokens will be used to register runners instead. Registration tokens, and support for certain configuration arguments,
-will be removed in GitLab 17.0. For more information, see [Migrating to the new runner registration workflow](../ci/runners/new_creation_workflow.md).
-The configuration arguments disabled for authentication tokens are:
-
-- `--locked`
-- `--access-level`
-- `--run-untagged`
-- `--tag-list`
-
-This change is a breaking change. You should use an [authentication token](../ci/runners/register_runner.md) in the `gitlab-runner register` command instead.
-
-</div>
-
-<div class="deprecation breaking-change" data-milestone="17.0">
-
 ### GraphQL deprecation of `dependencyProxyTotalSizeInBytes` field
 
 <div class="deprecation-notes">
@@ -381,6 +517,20 @@ Use `totalIssueWeight` instead, introduced in GitLab 16.2.
 
 <div class="deprecation breaking-change" data-milestone="17.0">
 
+### GraphQL networkPolicies resource deprecated
+
+<div class="deprecation-notes">
+- Announced in GitLab <span class="milestone">14.8</span>
+- Removal in GitLab <span class="milestone">17.0</span> ([breaking change](https://docs.gitlab.com/ee/update/terminology.html#breaking-change))
+- To discuss this change or learn more, see the [deprecation issue](https://gitlab.com/gitlab-org/gitlab/-/issues/421440).
+</div>
+
+The `networkPolicies` [GraphQL resource](https://docs.gitlab.com/ee/api/graphql/reference/#projectnetworkpolicies) has been deprecated and will be removed in GitLab 17.0. Since GitLab 15.0 this field has returned no data.
+
+</div>
+
+<div class="deprecation breaking-change" data-milestone="17.0">
+
 ### GraphQL type, `RunnerMembershipFilter` renamed to `CiRunnerMembershipFilter`
 
 <div class="deprecation-notes">
@@ -410,6 +560,61 @@ In GitLab 17.0, the `DISABLED_WITH_OVERRIDE` value of the `SharedRunnersSetting`
 
 <div class="deprecation breaking-change" data-milestone="17.0">
 
+### HashiCorp Vault integration will no longer use CI_JOB_JWT by default
+
+<div class="deprecation-notes">
+- Announced in GitLab <span class="milestone">15.9</span>
+- Removal in GitLab <span class="milestone">17.0</span> ([breaking change](https://docs.gitlab.com/ee/update/terminology.html#breaking-change))
+- To discuss this change or learn more, see the [deprecation issue](https://gitlab.com/gitlab-org/gitlab/-/issues/366798).
+</div>
+
+As part of our effort to improve the security of your CI workflows using JWT and OIDC, the native HashiCorp integration is also being updated in GitLab 16.0. Any projects that use the [`secrets:vault`](https://docs.gitlab.com/ee/ci/yaml/#secretsvault) keyword to retrieve secrets from Vault will need to be [configured to use the ID tokens](https://docs.gitlab.com/ee/ci/secrets/id_token_authentication.html#configure-automatic-id-token-authentication). ID tokens were introduced in 15.7.
+
+To prepare for this change, use the new [`id_tokens`](https://docs.gitlab.com/ee/ci/yaml/#id_tokens)
+keyword and configure the `aud` claim. Ensure the bound audience is prefixed with `https://`.
+
+In GitLab 15.9 to 15.11, you can [enable the **Limit JSON Web Token (JWT) access**](https://docs.gitlab.com/ee/ci/secrets/id_token_authentication.html#enable-automatic-id-token-authentication)
+setting, which prevents the old tokens from being exposed to any jobs and enables
+[ID token authentication for the `secrets:vault` keyword](https://docs.gitlab.com/ee/ci/secrets/id_token_authentication.html#configure-automatic-id-token-authentication).
+
+In GitLab 16.0 and later:
+
+- This setting will be removed.
+- CI/CD jobs that use the `id_tokens` keyword can use ID tokens with `secrets:vault`,
+  and will not have any `CI_JOB_JWT*` tokens available.
+- Jobs that do not use the `id_tokens` keyword will continue to have the `CI_JOB_JWT*`
+  tokens available until GitLab 17.0.
+
+</div>
+
+<div class="deprecation breaking-change" data-milestone="17.0">
+
+### Internal Container Registry API tag deletion endpoint
+
+<div class="deprecation-notes">
+- Announced in GitLab <span class="milestone">16.4</span>
+- Removal in GitLab <span class="milestone">17.0</span> ([breaking change](https://docs.gitlab.com/ee/update/terminology.html#breaking-change))
+- To discuss this change or learn more, see the [deprecation issue](https://gitlab.com/gitlab-org/container-registry/-/issues/1094).
+</div>
+
+The [Docker Registry HTTP API V2 Spec](https://docs.docker.com/registry/spec/api/), later replaced by the [OCI Distribution Spec](https://github.com/opencontainers/distribution-spec/blob/main/spec.md) did not include a tag delete operation, and an unsafe and slow workaround (involving deleting manifests, not tags) had to be used to achieve the same end.
+
+Tag deletion is an important function, so we added a tag deletion operation to the GitLab Container Registry, extending the V2 API beyond the scope of the Docker and OCI distribution spec.
+
+Since then, the OCI Distribution Spec has had some updates and it now has a tag delete operation, using the [`DELETE /v2/<name>/manifests/<tag>` endpoint](https://github.com/opencontainers/distribution-spec/blob/main/spec.md#deleting-tags).
+
+This leaves the container registry with two endpoints that provide the exact same functionality. `DELETE /v2/<name>/tags/reference/<tag>` is the custom GitLab tag delete endpoint and `DELETE /v2/<name>/manifests/<tag>`, the OCI compliant tag delete endpoint introduced in GitLab 16.4.
+
+Support for the custom GitLab tag delete endpoint is deprecated in GitLab 16.4, and it will be removed in GitLab 17.0.
+
+This endpoint is used by the **internal** Container Registry application API, not the public [GitLab Container Registry API](https://docs.gitlab.com/ee/api/container_registry.html). No action should be required by the majority of container registry users. All the GitLab UI and API functionality related to tag deletions will remain intact as we transition to the new OCI-compliant endpoint.
+
+If you do access the internal container registry API and use the original tag deletion endpoint, you must update to the new endpoint.
+
+</div>
+
+<div class="deprecation breaking-change" data-milestone="17.0">
+
 ### Maintainer role providing the ability to change Package settings using GraphQL API
 
 <div class="deprecation-notes">
@@ -432,6 +637,48 @@ settings for the group using either the GitLab UI or GraphQL API.
 
 <div class="deprecation breaking-change" data-milestone="17.0">
 
+### Old versions of JSON web tokens are deprecated
+
+<div class="deprecation-notes">
+- Announced in GitLab <span class="milestone">15.9</span>
+- Removal in GitLab <span class="milestone">17.0</span> ([breaking change](https://docs.gitlab.com/ee/update/terminology.html#breaking-change))
+- To discuss this change or learn more, see the [deprecation issue](https://gitlab.com/gitlab-org/gitlab/-/issues/366798).
+</div>
+
+[ID tokens](https://docs.gitlab.com/ee/ci/secrets/id_token_authentication.html) with OIDC support
+were introduced in GitLab 15.7. These tokens are more configurable than the old JSON web tokens (JWTs), are OIDC compliant,
+and only available in CI/CD jobs that explictly have ID tokens configured.
+ID tokens are more secure than the old `CI_JOB_JWT*` JSON web tokens which are exposed in every job,
+and as a result these old JSON web tokens are deprecated:
+
+- `CI_JOB_JWT`
+- `CI_JOB_JWT_V1`
+- `CI_JOB_JWT_V2`
+
+To prepare for this change, configure your pipelines to use [ID tokens](https://docs.gitlab.com/ee/ci/yaml/index.html#id_tokens)
+instead of the deprecated tokens. For OIDC compliance, the `iss` claim now uses
+the fully qualified domain name, for example `https://example.com`, previously
+introduced with the `CI_JOB_JWT_V2` token.
+
+In GitLab 15.9 to 15.11, you can [enable the **Limit JSON Web Token (JWT) access**](https://docs.gitlab.com/ee/ci/secrets/id_token_authentication.html#enable-automatic-id-token-authentication)
+setting, which prevents the old tokens from being exposed to any jobs and enables
+[ID token authentication for the `secrets:vault` keyword](https://docs.gitlab.com/ee/ci/secrets/id_token_authentication.html#configure-automatic-id-token-authentication).
+
+In GitLab 16.0 and later:
+
+- This setting will be removed.
+- CI/CD jobs that use the `id_tokens` keyword can use ID tokens with `secrets:vault`,
+  and will not have any `CI_JOB_JWT*` tokens available.
+- Jobs that do not use the `id_tokens` keyword will continue to have the `CI_JOB_JWT*`
+  tokens available until GitLab 17.0.
+
+In GitLab 17.0, the deprecated tokens will be completely removed and will no longer
+be available in CI/CD jobs.
+
+</div>
+
+<div class="deprecation breaking-change" data-milestone="17.0">
+
 ### OmniAuth Facebook is deprecated
 
 <div class="deprecation-notes">
@@ -544,33 +791,6 @@ This change is a breaking change. You should [create a runner in the UI](../ci/r
 
 <div class="deprecation breaking-change" data-milestone="17.0">
 
-### Registration tokens and server-side runner arguments in `gitlab-runner register` command
-
-<div class="deprecation-notes">
-- Announced in GitLab <span class="milestone">15.6</span>
-- Removal in GitLab <span class="milestone">17.0</span> ([breaking change](https://docs.gitlab.com/ee/update/terminology.html#breaking-change))
-- To discuss this change or learn more, see the [deprecation issue](https://gitlab.com/gitlab-org/gitlab/-/issues/380872).
-</div>
-
-Registration tokens and certain configuration arguments in the command `gitlab-runner register` that [registers](https://docs.gitlab.com/runner/register/) a runner, are deprecated.
-Authentication tokens will be used to register runners instead. Registration tokens, and support for certain configuration arguments,
-will be removed in GitLab 17.0. For more information, see [Migrating to the new runner registration workflow](../ci/runners/new_creation_workflow.md).
-The configuration arguments disabled for authentication tokens are:
-
-- `--locked`
-- `--access-level`
-- `--run-untagged`
-- `--maximum-timeout`
-- `--paused`
-- `--tag-list`
-- `--maintenance-note`
-
-This change is a breaking change. You should [create a runner in the UI](../ci/runners/register_runner.md) to add configurations, and use the authentication token in the `gitlab-runner register` command instead.
-
-</div>
-
-<div class="deprecation breaking-change" data-milestone="17.0">
-
 ### Required Pipeline Configuration is deprecated
 
 <div class="deprecation-notes">
@@ -678,56 +898,43 @@ we'll be introducing support in [this epic](https://gitlab.com/groups/gitlab-org
 
 <div class="deprecation breaking-change" data-milestone="17.0">
 
-### Support for REST API endpoints that reset runner registration tokens
+### The GitLab legacy requirement IID is deprecated in favor of work item IID
 
 <div class="deprecation-notes">
-- Announced in GitLab <span class="milestone">15.7</span>
+- Announced in GitLab <span class="milestone">15.9</span>
 - Removal in GitLab <span class="milestone">17.0</span> ([breaking change](https://docs.gitlab.com/ee/update/terminology.html#breaking-change))
-- To discuss this change or learn more, see the [deprecation issue](https://gitlab.com/gitlab-org/gitlab/-/issues/383341).
+- To discuss this change or learn more, see the [deprecation issue](https://gitlab.com/gitlab-org/gitlab/-/issues/390263).
 </div>
 
-The support for runner registration tokens is deprecated. As a consequence, the REST API endpoints to reset a registration token are also deprecated and will
-return the HTTP `410 Gone` status code in GitLab 17.0.
-The deprecated endpoints are:
-
-- `POST /runners/reset_registration_token`
-- `POST /projects/:id/runners/reset_registration_token`
-- `POST /groups/:id/runners/reset_registration_token`
-
-We plan to implement a new method to bind runners to a GitLab instance
-as part of the new [GitLab Runner token architecture](https://docs.gitlab.com/ee/architecture/blueprints/runner_tokens/).
-The work is planned in [this epic](https://gitlab.com/groups/gitlab-org/-/epics/7633).
-This new architecture introduces a new method for registering runners and will eliminate the legacy
-[runner registration token](https://docs.gitlab.com/ee/security/token_overview.html#runner-registration-tokens).
-From GitLab 17.0 and later, the runner registration methods implemented by the new GitLab Runner token architecture will be the only supported methods.
+We will be transitioning to a new IID as a result of moving requirements to a [work item type](https://docs.gitlab.com/ee/development/work_items.html#work-items-and-work-item-types). Users should begin using the new IID as support for the legacy IID and existing formatting will end in GitLab 17.0. The legacy requirement IID remains available until its removal in GitLab 17.0.
 
 </div>
 
 <div class="deprecation breaking-change" data-milestone="17.0">
 
-### The GitLab legacy requirement IID is deprecated in favor of work item IID
+### The Visual Reviews tool is deprecated
 
 <div class="deprecation-notes">
-- Announced in GitLab <span class="milestone">15.9</span>
+- Announced in GitLab <span class="milestone">15.8</span>
 - Removal in GitLab <span class="milestone">17.0</span> ([breaking change](https://docs.gitlab.com/ee/update/terminology.html#breaking-change))
-- To discuss this change or learn more, see the [deprecation issue](https://gitlab.com/gitlab-org/gitlab/-/issues/390263).
+- To discuss this change or learn more, see the [deprecation issue](https://gitlab.com/gitlab-org/gitlab/-/issues/387751).
 </div>
 
-We will be transitioning to a new IID as a result of moving requirements to a [work item type](https://docs.gitlab.com/ee/development/work_items.html#work-items-and-work-item-types). Users should begin using the new IID as support for the legacy IID and existing formatting will end in GitLab 17.0. The legacy requirement IID remains available until its removal in GitLab 17.0.
+Due to limited customer usage and capabilities, the Visual Reviews feature for Review Apps is deprecated and will be removed. There is no planned replacement and users should stop using Visual Reviews before GitLab 17.0.
 
 </div>
 
 <div class="deprecation breaking-change" data-milestone="17.0">
 
-### The Visual Reviews tool is deprecated
+### The `ci_job_token_scope_enabled` projects API attribute is deprecated
 
 <div class="deprecation-notes">
-- Announced in GitLab <span class="milestone">15.8</span>
+- Announced in GitLab <span class="milestone">16.4</span>
 - Removal in GitLab <span class="milestone">17.0</span> ([breaking change](https://docs.gitlab.com/ee/update/terminology.html#breaking-change))
-- To discuss this change or learn more, see the [deprecation issue](https://gitlab.com/gitlab-org/gitlab/-/issues/387751).
+- To discuss this change or learn more, see the [deprecation issue](https://gitlab.com/gitlab-org/gitlab/-/issues/423091).
 </div>
 
-Due to limited customer usage and capabilities, the Visual Reviews feature for Review Apps is deprecated and will be removed. There is no planned replacement and users should stop using Visual Reviews before GitLab 17.0.
+GitLab 16.1 introduced [API endpoints for the job token scope](https://gitlab.com/gitlab-org/gitlab/-/issues/351740). In the [projects API](https://docs.gitlab.com/ee/api/projects.html), the `ci_job_token_scope_enabled` attribute is deprecated, and will be removed in 17.0. You should use the [job token scope APIs](https://docs.gitlab.com/ee/api/project_job_token_scopes.html) instead.
 
 </div>
 
@@ -830,26 +1037,6 @@ removed in 17.0.
 
 <div class="deprecation breaking-change" data-milestone="17.0">
 
-### `runnerRegistrationToken` parameter for GitLab Runner Helm Chart
-
-<div class="deprecation-notes">
-- Announced in GitLab <span class="milestone">15.6</span>
-- Removal in GitLab <span class="milestone">17.0</span> ([breaking change](https://docs.gitlab.com/ee/update/terminology.html#breaking-change))
-- To discuss this change or learn more, see the [deprecation issue](https://gitlab.com/gitlab-org/gitlab/-/issues/381111).
-</div>
-
-The [`runnerRegistrationToken`](https://docs.gitlab.com/runner/install/kubernetes.html#required-configuration) parameter to use the GitLab Helm Chart to install a runner on Kubernetes is deprecated.
-
-We plan to implement a new method to bind runners to a GitLab instance leveraging `runnerToken`
-as part of the new [GitLab Runner token architecture](https://docs.gitlab.com/ee/architecture/blueprints/runner_tokens/).
-The work is planned in [this epic](https://gitlab.com/groups/gitlab-org/-/epics/7633).
-
-From GitLab 17.0 and later, the methods to register runners introduced by the new GitLab Runner token architecture will be the only supported methods.
-
-</div>
-
-<div class="deprecation breaking-change" data-milestone="17.0">
-
 ### `sidekiq` delivery method for `incoming_email` and `service_desk_email` is deprecated
 
 <div class="deprecation-notes">
@@ -909,92 +1096,42 @@ Previous work helped [align the vulnerabilities calls for pipeline security tabs
 </div>
 </div>
 
-<div class="milestone-wrapper" data-milestone="16.5">
+<div class="milestone-wrapper" data-milestone="16.6">
 
-## GitLab 16.5
+## GitLab 16.6
 
-<div class="deprecation " data-milestone="16.5">
+<div class="deprecation breaking-change" data-milestone="16.6">
 
-### Adding non-LDAP synced members to a locked LDAP group is deprecated
+### Job token allowlist covers public and internal projects
 
 <div class="deprecation-notes">
-- Announced in GitLab <span class="milestone">16.0</span>
-- Removal in GitLab <span class="milestone">16.5</span>
-- To discuss this change or learn more, see the [deprecation issue](https://gitlab.com/gitlab-org/gitlab/-/issues/213311).
-</div>
-
-Enabling the `ldap_settings_unlock_groups_by_owners` feature flag allowed non-LDAP synced users to be added to a locked LDAP group. This [feature](https://gitlab.com/gitlab-org/gitlab/-/issues/1793) has always been disabled by default and behind a feature flag. We are removing this feature to keep continuity with our SAML integration, and because allowing non-synced group members defeats the "single source of truth" principle of using a directory service. Once this feature is removed, any LDAP group members that are not synced with LDAP will lose access to that group.
-
+- Announced in GitLab <span class="milestone">16.3</span>
+- Removal in GitLab <span class="milestone">16.6</span> ([breaking change](https://docs.gitlab.com/ee/update/terminology.html#breaking-change))
+- To discuss this change or learn more, see the [deprecation issue](https://gitlab.com/gitlab-org/gitlab/-/issues/420678).
 </div>
 
-<div class="deprecation breaking-change" data-milestone="16.5">
+Starting in 16.6, projects that are **public** or **internal** will no longer authorize job token requests from projects that are **not** on the project's allowlist when [**Limit access to this project**](https://docs.gitlab.com/ee/ci/jobs/ci_job_token.html#allow-access-to-your-project-with-a-job-token) is enabled.
 
-### HashiCorp Vault integration will no longer use CI_JOB_JWT by default
+If you have [public or internal](https://docs.gitlab.com/ee/user/public_access.html#change-project-visibility) projects with the **Limit access to this project** setting enabled, you must add any projects which make job token requests to your project's allowlist for continued authorization.
 
-<div class="deprecation-notes">
-- Announced in GitLab <span class="milestone">15.9</span>
-- Removal in GitLab <span class="milestone">16.5</span> ([breaking change](https://docs.gitlab.com/ee/update/terminology.html#breaking-change))
-- To discuss this change or learn more, see the [deprecation issue](https://gitlab.com/gitlab-org/gitlab/-/issues/366798).
+</div>
 </div>
 
-As part of our effort to improve the security of your CI workflows using JWT and OIDC, the native HashiCorp integration is also being updated in GitLab 16.0. Any projects that use the [`secrets:vault`](https://docs.gitlab.com/ee/ci/yaml/#secretsvault) keyword to retrieve secrets from Vault will need to be [configured to use the ID tokens](https://docs.gitlab.com/ee/ci/secrets/id_token_authentication.html#configure-automatic-id-token-authentication). ID tokens were introduced in 15.7.
-
-To prepare for this change, use the new [`id_tokens`](https://docs.gitlab.com/ee/ci/yaml/#id_tokens)
-keyword and configure the `aud` claim. Ensure the bound audience is prefixed with `https://`.
-
-In GitLab 15.9 to 15.11, you can [enable the **Limit JSON Web Token (JWT) access**](https://docs.gitlab.com/ee/ci/secrets/id_token_authentication.html#enable-automatic-id-token-authentication)
-setting, which prevents the old tokens from being exposed to any jobs and enables
-[ID token authentication for the `secrets:vault` keyword](https://docs.gitlab.com/ee/ci/secrets/id_token_authentication.html#configure-automatic-id-token-authentication).
-
-In GitLab 16.0 and later:
-
-- This setting will be removed.
-- CI/CD jobs that use the `id_tokens` keyword can use ID tokens with `secrets:vault`,
-  and will not have any `CI_JOB_JWT*` tokens available.
-- Jobs that do not use the `id_tokens` keyword will continue to have the `CI_JOB_JWT*`
-  tokens available until GitLab 16.5.
+<div class="milestone-wrapper" data-milestone="16.5">
 
-</div>
+## GitLab 16.5
 
-<div class="deprecation breaking-change" data-milestone="16.5">
+<div class="deprecation " data-milestone="16.5">
 
-### Old versions of JSON web tokens are deprecated
+### Adding non-LDAP synced members to a locked LDAP group is deprecated
 
 <div class="deprecation-notes">
-- Announced in GitLab <span class="milestone">15.9</span>
-- Removal in GitLab <span class="milestone">16.5</span> ([breaking change](https://docs.gitlab.com/ee/update/terminology.html#breaking-change))
-- To discuss this change or learn more, see the [deprecation issue](https://gitlab.com/gitlab-org/gitlab/-/issues/366798).
+- Announced in GitLab <span class="milestone">16.0</span>
+- Removal in GitLab <span class="milestone">16.5</span>
+- To discuss this change or learn more, see the [deprecation issue](https://gitlab.com/gitlab-org/gitlab/-/issues/213311).
 </div>
 
-[ID tokens](https://docs.gitlab.com/ee/ci/secrets/id_token_authentication.html) with OIDC support
-were introduced in GitLab 15.7. These tokens are more configurable than the old JSON web tokens (JWTs), are OIDC compliant,
-and only available in CI/CD jobs that explictly have ID tokens configured.
-ID tokens are more secure than the old `CI_JOB_JWT*` JSON web tokens which are exposed in every job,
-and as a result these old JSON web tokens are deprecated:
-
-- `CI_JOB_JWT`
-- `CI_JOB_JWT_V1`
-- `CI_JOB_JWT_V2`
-
-To prepare for this change, configure your pipelines to use [ID tokens](https://docs.gitlab.com/ee/ci/yaml/index.html#id_tokens)
-instead of the deprecated tokens. For OIDC compliance, the `iss` claim now uses
-the fully qualified domain name, for example `https://example.com`, previously
-introduced with the `CI_JOB_JWT_V2` token.
-
-In GitLab 15.9 to 15.11, you can [enable the **Limit JSON Web Token (JWT) access**](https://docs.gitlab.com/ee/ci/secrets/id_token_authentication.html#enable-automatic-id-token-authentication)
-setting, which prevents the old tokens from being exposed to any jobs and enables
-[ID token authentication for the `secrets:vault` keyword](https://docs.gitlab.com/ee/ci/secrets/id_token_authentication.html#configure-automatic-id-token-authentication).
-
-In GitLab 16.0 and later:
-
-- This setting will be removed.
-- CI/CD jobs that use the `id_tokens` keyword can use ID tokens with `secrets:vault`,
-  and will not have any `CI_JOB_JWT*` tokens available.
-- Jobs that do not use the `id_tokens` keyword will continue to have the `CI_JOB_JWT*`
-  tokens available until GitLab 16.5.
-
-In GitLab 16.5, the deprecated tokens will be completely removed and will no longer
-be available in CI/CD jobs.
+Enabling the `ldap_settings_unlock_groups_by_owners` feature flag allowed non-LDAP synced users to be added to a locked LDAP group. This [feature](https://gitlab.com/gitlab-org/gitlab/-/issues/1793) has always been disabled by default and behind a feature flag. We are removing this feature to keep continuity with our SAML integration, and because allowing non-synced group members defeats the "single source of truth" principle of using a directory service. Once this feature is removed, any LDAP group members that are not synced with LDAP will lose access to that group.
 
 </div>
 </div>
@@ -1038,9 +1175,19 @@ However, enabling the bundled Grafana will no longer work from GitLab 16.3.
 - To discuss this change or learn more, see the [deprecation issue](https://gitlab.com/gitlab-org/gitlab/-/issues/387561).
 </div>
 
-**Update:** We previously announced we would remove the existing License Compliance CI template in GitLab 16.0. However, due to performance issues with the [license scanning of CycloneDX files](https://docs.gitlab.com/ee/user/compliance/license_scanning_of_cyclonedx_files/) we will do this change in 16.3 instead.
+**Update:** We previously announced we would remove the existing License Compliance CI template in GitLab 16.0. However, due to performance issues with the [license scanning of CycloneDX files](https://docs.gitlab.com/ee/user/compliance/license_scanning_of_cyclonedx_files/) we will do this in 16.3 instead.
+
+The GitLab [**License Compliance**](https://docs.gitlab.com/ee/user/compliance/license_compliance/) CI/CD template is now deprecated and is scheduled for removal in the GitLab 16.3 release.
+
+To continue using GitLab for license compliance, remove the **License Compliance** template from your CI/CD pipeline and add the **Dependency Scanning** template. The **Dependency Scanning** template is now capable of gathering the required license information, so it is no longer necessary to run a separate license compliance job.
+
+Before you remove the **License Compliance** CI/CD template, verify that the instance has been upgraded to a version that supports the new method of license scanning.
+
+To begin using the Dependency Scanner quickly at scale, you may set up a scan execution policy at the group level to enforce the SBOM-based license scan for all projects in the group. Then, you may remove the inclusion of the `Jobs/License-Scanning.gitlab-ci.yml` template from your CI/CD configuration.
+
+If you wish to continue using the legacy license compliance feature, you can do so by setting the `LICENSE_MANAGEMENT_VERSION CI` variable to `4`. This variable can be set at the project, group, or instance level. This configuration change will allow you to continue using an existing version of license compliance without having to adopt the new approach.
 
-The GitLab [License Compliance](https://docs.gitlab.com/ee/user/compliance/license_compliance/) CI template is now deprecated and is scheduled for removal in the GitLab 16.1 release. Users who wish to continue using GitLab for License Compliance should remove the License Compliance template from their CI pipeline and add the [Dependency Scanning template](https://docs.gitlab.com/ee/user/application_security/dependency_scanning/#configuration). The Dependency Scanning template is now capable of gathering the required license information so it is no longer necessary to run a separate License Compliance job. The License Compliance CI template should not be removed prior to verifying that the `license_scanning_sbom_scanner` and `package_metadata_synchronization` flags are enabled for the instance and that the instance has been upgraded to a version that supports [the new method of license scanning](https://docs.gitlab.com/ee/user/compliance/license_scanning_of_cyclonedx_files/).
+Bugs and vulnerabilities in this legacy analyzer will no longer be fixed.
 
 | CI Pipeline Includes | GitLab <= 15.8 | 15.9 <= GitLab < 16.3 | GitLab >= 16.3 |
 | ------------- | ------------- | ------------- | ------------- |
@@ -1052,6 +1199,22 @@ The GitLab [License Compliance](https://docs.gitlab.com/ee/user/compliance/licen
 
 <div class="deprecation breaking-change" data-milestone="16.3">
 
+### RSA key size limits
+
+<div class="deprecation-notes">
+- Announced in GitLab <span class="milestone">16.3</span>
+- Removal in GitLab <span class="milestone">16.3</span> ([breaking change](https://docs.gitlab.com/ee/update/terminology.html#breaking-change))
+- To discuss this change or learn more, see the [deprecation issue](https://gitlab.com/groups/gitlab-org/-/epics/11186).
+</div>
+
+Go versions 1.20.7 and later add a `maxRSAKeySize` constant that limits RSA keys to a maximum of 8192 bits. As a result, RSA keys larger than 8192 bits will no longer work with GitLab. Any RSA keys larger than 8192 bits must be regenerated at a smaller size.
+
+You might notice this issue because your logs include an error like `tls: server sent certificate containing RSA key larger than 8192 bits`. To test the length of your key, use this command: `openssl rsa -in <your-key-file> -text -noout | grep "Key:"`.
+
+</div>
+
+<div class="deprecation breaking-change" data-milestone="16.3">
+
 ### Twitter OmniAuth login option is removed from GitLab.com
 
 <div class="deprecation-notes">
@@ -3376,7 +3539,7 @@ By default, all new applications expire access tokens after 2 hours. In GitLab 1
 had no expiration. In GitLab 15.0, an expiry will be automatically generated for any existing token that does not
 already have one.
 
-You should [opt in](https://docs.gitlab.com/ee/integration/oauth_provider.html#expiring-access-tokens) to expiring
+You should [opt in](https://docs.gitlab.com/ee/integration/oauth_provider.html#access-token-expiration) to expiring
 tokens before GitLab 15.0 is released:
 
 1. Edit the application.
diff --git a/doc/update/index.md b/doc/update/index.md
index 5a0fd0bdf31..b821be3deea 100644
--- a/doc/update/index.md
+++ b/doc/update/index.md
@@ -25,12 +25,9 @@ has additional information about upgrading, including:
 Depending on the installation method and your GitLab version, there are multiple
 official ways to upgrade GitLab:
 
-- [Linux packages (Omnibus)](#linux-packages-omnibus)
-- [Self-compiled installations](#self-compiled-installation)
-- [Docker installations](#installation-using-docker)
-- [Kubernetes (Helm) installations](#installation-using-helm)
+::Tabs
 
-### Linux packages (Omnibus)
+:::TabTitle Linux packages (Omnibus)
 
 The [package upgrade guide](package/index.md)
 contains the steps needed to upgrade a package installed by official GitLab
@@ -39,12 +36,27 @@ repositories.
 There are also instructions when you want to
 [upgrade to a specific version](package/index.md#upgrade-to-a-specific-version-using-the-official-repositories).
 
-### Self-compiled installation
+:::TabTitle Helm chart (Kubernetes)
+
+GitLab can be deployed into a Kubernetes cluster using Helm.
+Instructions on how to upgrade a cloud-native deployment are in
+[a separate document](https://docs.gitlab.com/charts/installation/upgrade.html).
+
+Use the [version mapping](https://docs.gitlab.com/charts/installation/version_mappings.html)
+from the chart version to GitLab version to determine the [upgrade path](#upgrade-paths).
+
+:::TabTitle Docker
+
+GitLab provides official Docker images for both Community and Enterprise
+editions, and they are based on the Omnibus package. See how to
+[install GitLab using Docker](../install/docker.md).
+
+:::TabTitle Self-compiled (source)
 
 - [Upgrading Community Edition and Enterprise Edition from source](upgrading_from_source.md) -
   The guidelines for upgrading Community Edition and Enterprise Edition from source.
 - [Patch versions](patch_versions.md) guide includes the steps needed for a
-  patch version, such as 13.2.0 to 13.2.1, and apply to both Community and Enterprise
+  patch version, such as 15.2.0 to 15.2.1, and apply to both Community and Enterprise
   Editions.
 
 In the past we used separate documents for the upgrading instructions, but we
@@ -54,20 +66,7 @@ can still be found in the Git repository:
 - [Old upgrading guidelines for Community Edition](https://gitlab.com/gitlab-org/gitlab-foss/tree/11-8-stable/doc/update)
 - [Old upgrading guidelines for Enterprise Edition](https://gitlab.com/gitlab-org/gitlab/-/tree/11-8-stable-ee/doc/update)
 
-### Installation using Docker
-
-GitLab provides official Docker images for both Community and Enterprise
-editions, and they are based on the Omnibus package. See how to
-[install GitLab using Docker](../install/docker.md).
-
-### Installation using Helm
-
-GitLab can be deployed into a Kubernetes cluster using Helm.
-Instructions on how to upgrade a cloud-native deployment are in
-[a separate document](https://docs.gitlab.com/charts/installation/upgrade.html).
-
-Use the [version mapping](https://docs.gitlab.com/charts/installation/version_mappings.html)
-from the chart version to GitLab version to determine the [upgrade path](#upgrade-paths).
+::EndTabs
 
 ## Plan your upgrade
 
@@ -188,9 +187,13 @@ When upgrading:
 
 1. Find where your version sits in the upgrade path:
 
-   - GitLab 14: [`14.0.12`](#1400) > [`14.3.6`](#1430) > [`14.9.5`](#1490) > [`14.10.5`](#14100).
-   - GitLab 15: [`15.0.5`](#1500) > [`15.1.6`](#1510) (for GitLab instances with multiple web nodes) > [`15.4.6`](#1540) > [`15.11.13`](#15110).
-   - GitLab 16: [`16.0.x`](versions/gitlab_16_changes.md#1600) (only [instances with lots of users](versions/gitlab_16_changes.md#long-running-user-type-data-change)) > [latest `16.Y.Z`](https://gitlab.com/gitlab-org/gitlab/-/releases).
+   - GitLab 14: [`14.0.12`](versions/gitlab_14_changes.md#1400) > [`14.3.6`](versions/gitlab_14_changes.md#1430) >
+     [`14.9.5`](versions/gitlab_14_changes.md#1490) > [`14.10.5`](versions/gitlab_14_changes.md#14100).
+   - GitLab 15: [`15.0.5`](versions/gitlab_15_changes.md#1500) > [`15.1.6`](versions/gitlab_15_changes.md#1510) (for
+     GitLab instances with multiple web nodes) > [`15.4.6`](versions/gitlab_15_changes.md#1540) >
+     [`15.11.13`](versions/gitlab_15_changes.md#15110).
+   - GitLab 16: [`16.0.x`](versions/gitlab_16_changes.md#1600) (only
+     [instances with lots of users](versions/gitlab_16_changes.md#long-running-user-type-data-change)) > [`16.3`](versions/gitlab_16_changes.md#1630) > [latest `16.Y.Z`](https://gitlab.com/gitlab-org/gitlab/-/releases).
 
 1. Check for [required upgrade stops](#required-upgrade-stops).
 1. Consult the [version-specific upgrade instructions](#version-specific-upgrading-instructions).
@@ -209,11 +212,9 @@ crucial database schema and migration patches may be included in the latest patc
 Required upgrade stops are versions of GitLab that you must upgrade to before upgrading to later versions. Required
 upgrade stops allow required background migrations to finish.
 
-During GitLab 16.x, we are scheduling two or three required upgrade stops. We will give at least two milestones of
-notice when we schedule a required upgrade stop.
+During GitLab 16.x, we are scheduling required upgrade stops beforehand so users can better plan out appropriate upgrade stops and downtime when necessary.
 
-The first planned required upgrade stop is scheduled for GitLab 16.3. If nothing is introduced requiring an upgrade stop,
-GitLab 16.3 will be treated as a regular upgrade.
+The first scheduled required upgrade stop has been announced for 16.3. When planning upgrades, please take this into account.
 
 ### Earlier GitLab versions
 
@@ -287,1499 +288,15 @@ and [Helm Chart deployments](https://docs.gitlab.com/charts/). They come with ap
 
 ### GitLab 16
 
-Before upgrading, see [GitLab 16 changes](versions/gitlab_16_changes.md).
-
-### 15.11.1
-
-- Many [project importers](../user/project/import/index.md) and [group importers](../user/group/import/index.md) now
-  require the Maintainer role instead of only requiring the Developer role. For more information, see the documentation
-  for any importers you use.
-- Geo: Some project imports do not initialize wiki repositories on project creation. Since the migration of project wikis to SSF, [missing wiki repositories are being incorrectly flagged as failing verification](https://gitlab.com/gitlab-org/gitlab/-/issues/409704). This is not a result of an actual replication/verification failure but an invalid internal state for these missing repositories inside Geo and results in errors in the logs and the verification progress reporting a failed state for these wiki repositories. If you have not imported projects you are not impacted by this issue.
-  - Impacted versions: GitLab versions 15.11.x, 16.0.x, and 16.1.0 - 16.1.2.
-  - Versions containing fix: GitLab 16.1.3 and later.
-- Geo: A [bug](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/7841) in the built-in `pg-upgrade` tool prevents upgrading the bundled PostgreSQL database to version 13. This leaves the secondary site in a broken state, and prevents upgrading the Geo installation to GitLab 16.x ([PostgreSQL 12 support has removed in 16.0](deprecations.md#postgresql-12-deprecated) and later releases). This occurs on secondary sites using the bundled PostgreSQL software, running both the secondary main Rails database and tracking database on the same node. There is a manual [workaround](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/7841#workaround) for those impacted until a fix is backported to 15.11.
-  - Impacted versions: GitLab versions 15.2 - 15.11
-  - Versions containing fix: 15.11.12 and later.
-  - Version 16.0 and later are not impacted. Note, 15.11 is a mandatory upgrade stop on the way to 16.0.
-
-### 15.11.0
-
-- **Upgrade to patch release 15.11.3 or later**. This avoids [issue 408304](https://gitlab.com/gitlab-org/gitlab/-/issues/408304) when upgrading from 15.5.0 and earlier.
-- Geo: Some project imports do not initialize wiki repositories on project creation. Since the migration of project wikis to SSF, [missing wiki repositories are being incorrectly flagged as failing verification](https://gitlab.com/gitlab-org/gitlab/-/issues/409704). This is not a result of an actual replication/verification failure but an invalid internal state for these missing repositories inside Geo and results in errors in the logs and the verification progress reporting a failed state for these wiki repositories. If you have not imported projects you are not impacted by this issue.
-  - Impacted versions: GitLab versions 15.11.x, 16.0.x, and 16.1.0 - 16.1.2.
-  - Versions containing fix: GitLab 16.1.3 and later.
-- Geo: A [bug](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/7841) in the built-in `pg-upgrade` tool prevents upgrading the bundled PostgreSQL database to version 13. This leaves the secondary site in a broken state, and prevents upgrading the Geo installation to GitLab 16.x ([PostgreSQL 12 support has removed in 16.0](deprecations.md#postgresql-12-deprecated) and later releases). This occurs on secondary sites using the bundled PostgreSQL software, running both the secondary main Rails database and tracking database on the same node. There is a manual [workaround](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/7841#workaround) for those impacted until a fix is backported to 15.11.
-  - Impacted versions: GitLab versions 15.2 - 15.11.11.
-  - Versions containing fix: 15.11.12 and later.
-  - Version 16.0 and later are not impacted. Note, 15.11 is a mandatory upgrade stop on the way to 16.0.
-
-### 15.11.x
-
-- A [bug](https://gitlab.com/gitlab-org/gitlab/-/issues/411604) can cause new LDAP users signing in for the first time to be assigned a username based on their email address instead of their LDAP username attribute. A manual workaround is to set `gitlab_rails['omniauth_auto_link_ldap_user'] = true`, or upgrade to GitLab 16.1 or later where the bug has been fixed.
-
-### 15.10.5
-
-- A [bug with Elastic Indexer Cron Workers](https://gitlab.com/gitlab-org/gitlab/-/issues/408214) can cause saturation in Sidekiq.
-  - When this issue occurs, merge request merges, pipelines, Slack notifications, and other events are not created or take a long time to occur.
-  - This issue may not manifest immediately as it can take up to a week before the Sidekiq is saturated enough.
-  - Elasticsearch does not need to be enabled for this to occur.
-  - To resolve this issue, upgrade to 15.11 or use the workaround in the issue.
-- Many [project importers](../user/project/import/index.md) and [group importers](../user/group/import/index.md) now
-  require the Maintainer role instead of only requiring the Developer role. For more information, see the documentation
-  for any importers you use.
-
-### 15.10.0
-
-- A [bug with Elastic Indexer Cron Workers](https://gitlab.com/gitlab-org/gitlab/-/issues/408214) can cause saturation in Sidekiq.
-  - When this issue occurs, merge request merges, pipelines, Slack notifications, and other events are not created or take a long time to occur.
-  - This issue may not manifest immediately as it can take up to a week before the Sidekiq is saturated enough.
-  - Elasticsearch does not need to be enabled for this to occur.
-  - To resolve this issue, upgrade to 15.11 or use the workaround in the issue.
-- Gitaly configuration changes significantly in Omnibus GitLab 16.0. You can begin migrating to the new structure in Omnibus GitLab 15.10 while backwards compatibility is
-  maintained in the lead up to Omnibus GitLab 16.0. [Read more about this change](#gitaly-omnibus-gitlab-configuration-structure-change).
-- You might encounter the following error while upgrading to GitLab 15.10 or later:
-
-  ```shell
-  STDOUT: rake aborted!
-  StandardError: An error has occurred, all later migrations canceled:
-  PG::CheckViolation: ERROR:  check constraint "check_70f294ef54" is violated by some row
-  ```
-
-  This error is caused by a [batched background migration introduced in GitLab 15.8](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/107701)
-  not being finalized before GitLab 15.10. To resolve this error:
-
-  1. Execute the following SQL statement using the database console (`sudo gitlab-psql` for Linux package installs):
-
-     ```sql
-     UPDATE oauth_access_tokens SET expires_in = '7200' WHERE expires_in IS NULL;
-     ```
-
-  1. [Re-run database migrations](../administration/raketasks/maintenance.md#run-incomplete-database-migrations).
-
-- You might also encounter the following error while upgrading to GitLab 15.10 or later:
-
-  ```shell
-  "exception.class": "ActiveRecord::StatementInvalid",
-  "exception.message": "PG::SyntaxError: ERROR:  zero-length delimited identifier at or near \"\"\"\"\nLINE 1: ...COALESCE(\"lock_version\", 0) + 1 WHERE \"ci_builds\".\"\" IN (SEL...\n
-  ```
-
-  This error is caused by a [batched background migration introduced in GitLab 14.9](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/81410)
-  not being finalized before upgrading to GitLab 15.10 or later. To resolve this error, it is safe to [mark the migration as complete](background_migrations.md#mark-a-batched-migration-finished):
-
-  ```ruby
-  # Start the rails console
-
-  connection = Ci::ApplicationRecord.connection
-
-  Gitlab::Database::SharedModel.using_connection(connection) do
-    migration = Gitlab::Database::BackgroundMigration::BatchedMigration.find_for_configuration(
-      Gitlab::Database.gitlab_schemas_for_connection(connection), 'NullifyOrphanRunnerIdOnCiBuilds', :ci_builds, :id, [])
-
-    # mark all jobs completed
-    migration.batched_jobs.update_all(status: Gitlab::Database::BackgroundMigration::BatchedJob.state_machine.states[:succeeded].value)
-    migration.update_attribute(:status, Gitlab::Database::BackgroundMigration::BatchedMigration.state_machine.states[:finished].value)
-  end
-  ```
-
-For more information, see [issue 415724](https://gitlab.com/gitlab-org/gitlab/-/issues/415724).
-
-### 15.9.0
-
-- A [bug with Elastic Indexer Cron Workers](https://gitlab.com/gitlab-org/gitlab/-/issues/408214) can cause saturation in Sidekiq.
-  - When this issue occurs, merge request merges, pipelines, Slack notifications, and other events are not created or take a long time to occur.
-  - This issue may not manifest immediately as it can take up to a week before the Sidekiq is saturated enough.
-  - Elasticsearch does not need to be enabled for this to occur.
-  - To resolve this issue, upgrade to 15.11 or use the workaround in the issue.
-- **Upgrade to patch release 15.9.3 or later**. This provides fixes for two database migration bugs:
-  - Patch releases 15.9.0, 15.9.1, 15.9.2 have [a bug that can cause data loss](#user-profile-data-loss-bug-in-159x) from the user profile fields.
-  - The second [bug fix](https://gitlab.com/gitlab-org/gitlab/-/issues/394760) ensures it is possible to upgrade directly from 15.4.x.
-- As part of the [CI Partitioning effort](../architecture/blueprints/ci_data_decay/pipeline_partitioning.md), a [new Foreign Key](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/107547) was added to `ci_builds_needs`. On GitLab instances with large CI tables, adding this constraint can take longer than usual.
-- Praefect's metadata verifier's [invalid metadata deletion behavior](../administration/gitaly/praefect.md#enable-deletions) is now enabled by default.
-
-  The metadata verifier processes replica records in the Praefect database and verifies the replicas actually exist on the Gitaly nodes. If the replica doesn't exist, its
-  metadata record is deleted. This enables Praefect to fix situations where a replica has a metadata record indicating it's fine but, in reality, it doesn't exist on disk.
-  After the metadata record is deleted, Praefect's reconciler schedules a replication job to recreate the replica.
-
-  Because of past issues with the state management logic, there may be invalid metadata records in the database. These could exist, for example, because of incomplete
-  deletions of repositories or partially completed renames. The verifier deletes these stale replica records of affected repositories. These repositories may show up as
-  unavailable repositories in the metrics and `praefect dataloss` sub-command because of the replica records being removed. If you encounter such repositories, remove
-  the repository using `praefect remove-repository` to remove the repository's remaining records.
-
-  You can find repositories with invalid metadata records prior in GitLab 15.0 and later by searching for the log records outputted by the verifier. [Read more about repository verification, and to see an example log entry](../administration/gitaly/praefect.md#repository-verification).
-- Praefect configuration changes significantly in Omnibus GitLab 16.0. You can begin migrating to the new structure in Omnibus GitLab 15.9 while backwards compatibility is
-  maintained in the lead up to Omnibus GitLab 16.0. [Read more about this change](#praefect-omnibus-gitlab-configuration-structure-change).
-- For **self-compiled (source) installations**, with the addition of `gitlab-sshd` the Kerberos headers are needed to build GitLab Shell.
-
-  ```shell
-  sudo apt install libkrb5-dev
-  ```
-
-### 15.8.2
-
-- Geo: We discovered an issue where [replication and verification of projects and wikis was not keeping up](https://gitlab.com/gitlab-org/gitlab/-/issues/387980) on small number of Geo installations. Your installation may be affected if you see some projects and/or wikis persistently in the "Queued" state for verification. This can lead to data loss after a failover.
-  - Affected versions: GitLab versions 15.6.x, 15.7.x, and 15.8.0 - 15.8.2.
-  - Versions containing fix: GitLab 15.8.3 and later.
-
-### 15.8.1
-
-- Geo: We discovered an issue where [replication and verification of projects and wikis was not keeping up](https://gitlab.com/gitlab-org/gitlab/-/issues/387980) on small number of Geo installations. Your installation may be affected if you see some projects and/or wikis persistently in the "Queued" state for verification. This can lead to data loss after a failover.
-  - Affected versions: GitLab versions 15.6.x, 15.7.x, and 15.8.0 - 15.8.2.
-  - Versions containing fix: GitLab 15.8.3 and later.
-- Due to [a bug introduced in GitLab 15.4](https://gitlab.com/gitlab-org/gitlab/-/issues/390155), if one or more Git repositories in Gitaly Cluster is [unavailable](../administration/gitaly/recovery.md#unavailable-repositories), then [Repository checks](../administration/repository_checks.md#repository-checks) and [Geo replication and verification](../administration/geo/index.md) stop running for all project or project wiki repositories in the affected Gitaly Cluster. The bug was fixed by [reverting the change in GitLab 15.9.0](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/110823). Before upgrading to this version, check if you have any "unavailable" repositories. See [the bug issue](https://gitlab.com/gitlab-org/gitlab/-/issues/390155) for more information.
-
-### 15.8.0
-
-- Git 2.38.0 and later is required by Gitaly. For self-compiled installations, you should use the [Git version provided by Gitaly](../install/installation.md#git).
-- Due to [a bug introduced in GitLab 15.4](https://gitlab.com/gitlab-org/gitlab/-/issues/390155), if one or more Git repositories in Gitaly Cluster is [unavailable](../administration/gitaly/recovery.md#unavailable-repositories), then [Repository checks](../administration/repository_checks.md#repository-checks) and [Geo replication and verification](../administration/geo/index.md) stop running for all project or project wiki repositories in the affected Gitaly Cluster. The bug was fixed by [reverting the change in GitLab 15.9.0](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/110823). Before upgrading to this version, check if you have any "unavailable" repositories. See [the bug issue](https://gitlab.com/gitlab-org/gitlab/-/issues/390155) for more information.
-- Geo: We discovered an issue where [replication and verification of projects and wikis was not keeping up](https://gitlab.com/gitlab-org/gitlab/-/issues/387980) on small number of Geo installations. Your installation may be affected if you see some projects and/or wikis persistently in the "Queued" state for verification. This can lead to data loss after a failover.
-  - Affected versions: GitLab versions 15.6.x, 15.7.x, and 15.8.0 - 15.8.2.
-  - Versions containing fix: GitLab 15.8.3 and later.
-
-### 15.7.6
-
-- Geo: We discovered an issue where [replication and verification of projects and wikis was not keeping up](https://gitlab.com/gitlab-org/gitlab/-/issues/387980) on small number of Geo installations. Your installation may be affected if you see some projects and/or wikis persistently in the "Queued" state for verification. This can lead to data loss after a failover.
-  - Affected versions: GitLab versions 15.6.x, 15.7.x, and 15.8.0 - 15.8.2.
-  - Versions containing fix: GitLab 15.8.3 and later.
-- Due to [a bug introduced in GitLab 15.4](https://gitlab.com/gitlab-org/gitlab/-/issues/390155), if one or more Git repositories in Gitaly Cluster is [unavailable](../administration/gitaly/recovery.md#unavailable-repositories), then [Repository checks](../administration/repository_checks.md#repository-checks) and [Geo replication and verification](../administration/geo/index.md) stop running for all project or project wiki repositories in the affected Gitaly Cluster. The bug was fixed by [reverting the change in GitLab 15.9.0](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/110823). Before upgrading to this version, check if you have any "unavailable" repositories. See [the bug issue](https://gitlab.com/gitlab-org/gitlab/-/issues/390155) for more information.
-
-### 15.7.5
-
-- Geo: We discovered an issue where [replication and verification of projects and wikis was not keeping up](https://gitlab.com/gitlab-org/gitlab/-/issues/387980) on small number of Geo installations. Your installation may be affected if you see some projects and/or wikis persistently in the "Queued" state for verification. This can lead to data loss after a failover.
-  - Affected versions: GitLab versions 15.6.x, 15.7.x, and 15.8.0 - 15.8.2.
-  - Versions containing fix: GitLab 15.8.3 and later.
-- Due to [a bug introduced in GitLab 15.4](https://gitlab.com/gitlab-org/gitlab/-/issues/390155), if one or more Git repositories in Gitaly Cluster is [unavailable](../administration/gitaly/recovery.md#unavailable-repositories), then [Repository checks](../administration/repository_checks.md#repository-checks) and [Geo replication and verification](../administration/geo/index.md) stop running for all project or project wiki repositories in the affected Gitaly Cluster. The bug was fixed by [reverting the change in GitLab 15.9.0](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/110823). Before upgrading to this version, check if you have any "unavailable" repositories. See [the bug issue](https://gitlab.com/gitlab-org/gitlab/-/issues/390155) for more information.
-
-### 15.7.4
-
-- Geo: We discovered an issue where [replication and verification of projects and wikis was not keeping up](https://gitlab.com/gitlab-org/gitlab/-/issues/387980) on small number of Geo installations. Your installation may be affected if you see some projects and/or wikis persistently in the "Queued" state for verification. This can lead to data loss after a failover.
-  - Affected versions: GitLab versions 15.6.x, 15.7.x, and 15.8.0 - 15.8.2.
-  - Versions containing fix: GitLab 15.8.3 and later.
-- Due to [a bug introduced in GitLab 15.4](https://gitlab.com/gitlab-org/gitlab/-/issues/390155), if one or more Git repositories in Gitaly Cluster is [unavailable](../administration/gitaly/recovery.md#unavailable-repositories), then [Repository checks](../administration/repository_checks.md#repository-checks) and [Geo replication and verification](../administration/geo/index.md) stop running for all project or project wiki repositories in the affected Gitaly Cluster. The bug was fixed by [reverting the change in GitLab 15.9.0](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/110823). Before upgrading to this version, check if you have any "unavailable" repositories. See [the bug issue](https://gitlab.com/gitlab-org/gitlab/-/issues/390155) for more information.
-
-### 15.7.3
-
-- Geo: We discovered an issue where [replication and verification of projects and wikis was not keeping up](https://gitlab.com/gitlab-org/gitlab/-/issues/387980) on small number of Geo installations. Your installation may be affected if you see some projects and/or wikis persistently in the "Queued" state for verification. This can lead to data loss after a failover.
-  - Affected versions: GitLab versions 15.6.x, 15.7.x, and 15.8.0 - 15.8.2.
-  - Versions containing fix: GitLab 15.8.3 and later.
-- Due to [a bug introduced in GitLab 15.4](https://gitlab.com/gitlab-org/gitlab/-/issues/390155), if one or more Git repositories in Gitaly Cluster is [unavailable](../administration/gitaly/recovery.md#unavailable-repositories), then [Repository checks](../administration/repository_checks.md#repository-checks) and [Geo replication and verification](../administration/geo/index.md) stop running for all project or project wiki repositories in the affected Gitaly Cluster. The bug was fixed by [reverting the change in GitLab 15.9.0](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/110823). Before upgrading to this version, check if you have any "unavailable" repositories. See [the bug issue](https://gitlab.com/gitlab-org/gitlab/-/issues/390155) for more information.
-
-### 15.7.2
-
-- Geo: [Container registry push events are rejected](https://gitlab.com/gitlab-org/gitlab/-/issues/386389) by the `/api/v4/container_registry_event/events` endpoint resulting in Geo secondary sites not being aware of updates to container registry images and subsequently not replicating the upgrades. Secondary sites may contain out of date container images after a failover as a consequence. This affects versions 15.6.0 - 15.6.6 and 15.7.0 - 15.7.2. If you're using Geo with container repositories, you are advised to upgrade to GitLab 15.6.7, 15.7.3, or 15.8.0 which contain a fix for this issue and avoid potential data loss after a failover.
-- Geo: We discovered an issue where [replication and verification of projects and wikis was not keeping up](https://gitlab.com/gitlab-org/gitlab/-/issues/387980) on small number of Geo installations. Your installation may be affected if you see some projects and/or wikis persistently in the "Queued" state for verification. This can lead to data loss after a failover.
-  - Affected versions: GitLab versions 15.6.x, 15.7.x, and 15.8.0 - 15.8.2.
-  - Versions containing fix: GitLab 15.8.3 and later.
-- Due to [a bug introduced in GitLab 15.4](https://gitlab.com/gitlab-org/gitlab/-/issues/390155), if one or more Git repositories in Gitaly Cluster is [unavailable](../administration/gitaly/recovery.md#unavailable-repositories), then [Repository checks](../administration/repository_checks.md#repository-checks) and [Geo replication and verification](../administration/geo/index.md) stop running for all project or project wiki repositories in the affected Gitaly Cluster. The bug was fixed by [reverting the change in GitLab 15.9.0](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/110823). Before upgrading to this version, check if you have any "unavailable" repositories. See [the bug issue](https://gitlab.com/gitlab-org/gitlab/-/issues/390155) for more information.
-
-### 15.7.1
-
-- Geo: [Container registry push events are rejected](https://gitlab.com/gitlab-org/gitlab/-/issues/386389) by the `/api/v4/container_registry_event/events` endpoint resulting in Geo secondary sites not being aware of updates to container registry images and subsequently not replicating the updates. Secondary sites may contain out of date container images after a failover as a consequence. This affects versions 15.6.0 - 15.6.6 and 15.7.0 - 15.7.2. If you're using Geo with container repositories, you are advised to upgrade to GitLab 15.6.7, 15.7.3, or 15.8.0 which contain a fix for this issue and avoid potential data loss after a failover.
-- Geo: We discovered an issue where [replication and verification of projects and wikis was not keeping up](https://gitlab.com/gitlab-org/gitlab/-/issues/387980) on small number of Geo installations. Your installation may be affected if you see some projects and/or wikis persistently in the "Queued" state for verification. This can lead to data loss after a failover.
-  - Affected versions: GitLab versions 15.6.x, 15.7.x, and 15.8.0 - 15.8.2.
-  - Versions containing fix: GitLab 15.8.3 and later.
-- Due to [a bug introduced in GitLab 15.4](https://gitlab.com/gitlab-org/gitlab/-/issues/390155), if one or more Git repositories in Gitaly Cluster is [unavailable](../administration/gitaly/recovery.md#unavailable-repositories), then [Repository checks](../administration/repository_checks.md#repository-checks) and [Geo replication and verification](../administration/geo/index.md) stop running for all project or project wiki repositories in the affected Gitaly Cluster. The bug was fixed by [reverting the change in GitLab 15.9.0](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/110823). Before upgrading to this version, check if you have any "unavailable" repositories. See [the bug issue](https://gitlab.com/gitlab-org/gitlab/-/issues/390155) for more information.
-
-### 15.7.0
-
-- This version validates a `NOT NULL DB` constraint on the `issues.work_item_type_id` column.
-  To upgrade to this version, no records with a `NULL` `work_item_type_id` should exist on the `issues` table.
-  There are multiple `BackfillWorkItemTypeIdForIssues` background migrations that will be finalized with
-  the `EnsureWorkItemTypeBackfillMigrationFinished` post-deploy migration.
-- GitLab 15.4.0 introduced a [batched background migration](background_migrations.md#batched-background-migrations) to
-  [backfill `namespace_id` values on issues table](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/91921). This
-  migration might take multiple hours or days to complete on larger GitLab instances. Make sure the migration
-  has completed successfully before upgrading to 15.7.0.
-- A database constraint is added, specifying that the `namespace_id` column on the issues
-  table has no `NULL` values.
-
-  - If the `namespace_id` batched background migration from 15.4 failed (see above) then the 15.7 upgrade
-    fails with a database migration error.
-
-  - On GitLab instances with large issues tables, validating this constraint causes the upgrade to take
-    longer than usual. All database changes need to complete within a one-hour limit:
-
-    ```plaintext
-    FATAL: Mixlib::ShellOut::CommandTimeout: rails_migration[gitlab-rails]
-    [..]
-    Mixlib::ShellOut::CommandTimeout: Command timed out after 3600s:
-    ```
-
-    A workaround exists to [complete the data change and the upgrade manually](package/index.md#mixlibshelloutcommandtimeout-rails_migrationgitlab-rails--command-timed-out-after-3600s).
-- The default Sidekiq `max_concurrency` has been changed to 20. This is now
-  consistent in our documentation and product defaults.
-
-  For example, previously:
-
-  - Linux package installation default (`sidekiq['max_concurrency']`): 50
-  - Self-compiled installation default: 50
-  - Helm chart default (`gitlab.sidekiq.concurrency`): 25
-
-  Reference architectures still use a default of 10 as this is set specifically
-  for those configurations.
-
-  Sites that have configured `max_concurrency` will not be affected by this change.
-  [Read more about the Sidekiq concurrency setting](../administration/sidekiq/extra_sidekiq_processes.md#concurrency).
-- GitLab Runner 15.7.0 introduced a breaking change that affects CI/CD jobs: [Correctly handle expansion of job file variables](https://gitlab.com/gitlab-org/gitlab-runner/-/merge_requests/3613).
-  Previously, job-defined variables that referred to
-  [file type variables](../ci/variables/index.md#use-file-type-cicd-variables)
-  were expanded to the value of the file variable (its content). This behavior did not
-  respect the typical rules of shell variable expansion. There was also the potential
-  that secrets or sensitive information could leak if the file variable and its
-  contents printed. For example, if they were printed in an echo output. For more information,
-  see [Understanding the file type variable expansion change in GitLab 15.7](https://about.gitlab.com/blog/2023/02/13/impact-of-the-file-type-variable-change-15-7/).
-- Geo: [Container registry push events are rejected](https://gitlab.com/gitlab-org/gitlab/-/issues/386389) by the `/api/v4/container_registry_event/events` endpoint resulting in Geo secondary sites not being aware of updates to container registry images and subsequently not replicating the updates. Secondary sites may contain out of date container images after a failover as a consequence. This affects versions 15.6.0 - 15.6.6 and 15.7.0 - 15.7.2. If you're using Geo with container repositories, you are advised to upgrade to GitLab 15.6.7, 15.7.3, or 15.8.0 which contain a fix for this issue and avoid potential data loss after a failover.
-- Due to [a bug introduced in GitLab 15.4](https://gitlab.com/gitlab-org/gitlab/-/issues/390155), if one or more Git repositories in Gitaly Cluster is [unavailable](../administration/gitaly/recovery.md#unavailable-repositories), then [Repository checks](../administration/repository_checks.md#repository-checks) and [Geo replication and verification](../administration/geo/index.md) stop running for all project or project wiki repositories in the affected Gitaly Cluster. The bug was fixed by [reverting the change in GitLab 15.9.0](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/110823). Before upgrading to this version, check if you have any "unavailable" repositories. See [the bug issue](https://gitlab.com/gitlab-org/gitlab/-/issues/390155) for more information.
-- Geo: We discovered an issue where [replication and verification of projects and wikis was not keeping up](https://gitlab.com/gitlab-org/gitlab/-/issues/387980) on small number of Geo installations. Your installation may be affected if you see some projects and/or wikis persistently in the "Queued" state for verification. This can lead to data loss after a failover.
-  - Affected versions: GitLab versions 15.6.x, 15.7.x, and 15.8.0 - 15.8.2.
-  - Versions containing fix: GitLab 15.8.3 and later.
-
-### 15.6.7
-
-- Geo: We discovered an issue where [replication and verification of projects and wikis was not keeping up](https://gitlab.com/gitlab-org/gitlab/-/issues/387980) on small number of Geo installations. Your installation may be affected if you see some projects and/or wikis persistently in the "Queued" state for verification. This can lead to data loss after a failover.
-  - Affected versions: GitLab versions 15.6.x, 15.7.x, and 15.8.0 - 15.8.2.
-  - Versions containing fix: GitLab 15.8.3 and later.
-- Due to [a bug introduced in GitLab 15.4](https://gitlab.com/gitlab-org/gitlab/-/issues/390155), if one or more Git repositories in Gitaly Cluster is [unavailable](../administration/gitaly/recovery.md#unavailable-repositories), then [Repository checks](../administration/repository_checks.md#repository-checks) and [Geo replication and verification](../administration/geo/index.md) stop running for all project or project wiki repositories in the affected Gitaly Cluster. The bug was fixed by [reverting the change in GitLab 15.9.0](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/110823). Before upgrading to this version, check if you have any "unavailable" repositories. See [the bug issue](https://gitlab.com/gitlab-org/gitlab/-/issues/390155) for more information.
-
-### 15.6.6
-
-- Geo: [Container registry push events are rejected](https://gitlab.com/gitlab-org/gitlab/-/issues/386389) by the `/api/v4/container_registry_event/events` endpoint resulting in Geo secondary sites not being aware of updates to container registry images and subsequently not replicating the updates. Secondary sites may contain out of date container images after a failover as a consequence. This affects versions 15.6.0 - 15.6.6 and 15.7.0 - 15.7.2. If you're using Geo with container repositories, you are advised to upgrade to GitLab 15.6.7, 15.7.3, or 15.8.0 which contain a fix for this issue and avoid potential data loss after a failover.
-- Geo: We discovered an issue where [replication and verification of projects and wikis was not keeping up](https://gitlab.com/gitlab-org/gitlab/-/issues/387980) on small number of Geo installations. Your installation may be affected if you see some projects and/or wikis persistently in the "Queued" state for verification. This can lead to data loss after a failover.
-  - Affected versions: GitLab versions 15.6.x, 15.7.x, and 15.8.0 - 15.8.2.
-  - Versions containing fix: GitLab 15.8.3 and later.
-- Due to [a bug introduced in GitLab 15.4](https://gitlab.com/gitlab-org/gitlab/-/issues/390155), if one or more Git repositories in Gitaly Cluster is [unavailable](../administration/gitaly/recovery.md#unavailable-repositories), then [Repository checks](../administration/repository_checks.md#repository-checks) and [Geo replication and verification](../administration/geo/index.md) stop running for all project or project wiki repositories in the affected Gitaly Cluster. The bug was fixed by [reverting the change in GitLab 15.9.0](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/110823). Before upgrading to this version, check if you have any "unavailable" repositories. See [the bug issue](https://gitlab.com/gitlab-org/gitlab/-/issues/390155) for more information.
-
-### 15.6.5
-
-- Geo: [Container registry push events are rejected](https://gitlab.com/gitlab-org/gitlab/-/issues/386389) by the `/api/v4/container_registry_event/events` endpoint resulting in Geo secondary sites not being aware of updates to container registry images and subsequently not replicating the updates. Secondary sites may contain out of date container images after a failover as a consequence. This affects versions 15.6.0 - 15.6.6 and 15.7.0 - 15.7.2. If you're using Geo with container repositories, you are advised to upgrade to GitLab 15.6.7, 15.7.3, or 15.8.0 which contain a fix for this issue and avoid potential data loss after a failover.
-- Geo: We discovered an issue where [replication and verification of projects and wikis was not keeping up](https://gitlab.com/gitlab-org/gitlab/-/issues/387980) on small number of Geo installations. Your installation may be affected if you see some projects and/or wikis persistently in the "Queued" state for verification. This can lead to data loss after a failover.
-  - Affected versions: GitLab versions 15.6.x, 15.7.x, and 15.8.0 - 15.8.2.
-  - Versions containing fix: GitLab 15.8.3 and later.
-- Due to [a bug introduced in GitLab 15.4](https://gitlab.com/gitlab-org/gitlab/-/issues/390155), if one or more Git repositories in Gitaly Cluster is [unavailable](../administration/gitaly/recovery.md#unavailable-repositories), then [Repository checks](../administration/repository_checks.md#repository-checks) and [Geo replication and verification](../administration/geo/index.md) stop running for all project or project wiki repositories in the affected Gitaly Cluster. The bug was fixed by [reverting the change in GitLab 15.9.0](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/110823). Before upgrading to this version, check if you have any "unavailable" repositories. See [the bug issue](https://gitlab.com/gitlab-org/gitlab/-/issues/390155) for more information.
-
-### 15.6.4
-
-- Geo: [Container registry push events are rejected](https://gitlab.com/gitlab-org/gitlab/-/issues/386389) by the `/api/v4/container_registry_event/events` endpoint resulting in Geo secondary sites not being aware of updates to container registry images and subsequently not replicating the updates. Secondary sites may contain out of date container images after a failover as a consequence. This affects versions 15.6.0 - 15.6.6, and 15.7.0 - 15.7.2. If you're using Geo with container repositories, you are advised to upgrade to GitLab 15.6.7, 15.7.3, or 15.8.0 which contain a fix for this issue and avoid potential data loss after a failover.
-- Geo: We discovered an issue where [replication and verification of projects and wikis was not keeping up](https://gitlab.com/gitlab-org/gitlab/-/issues/387980) on small number of Geo installations. Your installation may be affected if you see some projects and/or wikis persistently in the "Queued" state for verification. This can lead to data loss after a failover.
-  - Affected versions: GitLab versions 15.6.x, 15.7.x, and 15.8.0 - 15.8.2.
-  - Versions containing fix: GitLab 15.8.3 and later.
-- Due to [a bug introduced in GitLab 15.4](https://gitlab.com/gitlab-org/gitlab/-/issues/390155), if one or more Git repositories in Gitaly Cluster is [unavailable](../administration/gitaly/recovery.md#unavailable-repositories), then [Repository checks](../administration/repository_checks.md#repository-checks) and [Geo replication and verification](../administration/geo/index.md) stop running for all project or project wiki repositories in the affected Gitaly Cluster. The bug was fixed by [reverting the change in GitLab 15.9.0](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/110823). Before upgrading to this version, check if you have any "unavailable" repositories. See [the bug issue](https://gitlab.com/gitlab-org/gitlab/-/issues/390155) for more information.
-
-### 15.6.3
-
-- Geo: [Container registry push events are rejected](https://gitlab.com/gitlab-org/gitlab/-/issues/386389) by the `/api/v4/container_registry_event/events` endpoint resulting in Geo secondary sites not being aware of updates to container registry images and subsequently not replicating the updates. Secondary sites may contain out of date container images after a failover as a consequence. This affects versions 15.6.0 - 15.6.6 and 15.7.0 - 15.7.2. If you're using Geo with container repositories, you are advised to upgrade to GitLab 15.6.7, 15.7.3, or 15.8.0 which contain a fix for this issue and avoid potential data loss after a failover.
-- Geo: We discovered an issue where [replication and verification of projects and wikis was not keeping up](https://gitlab.com/gitlab-org/gitlab/-/issues/387980) on small number of Geo installations. Your installation may be affected if you see some projects and/or wikis persistently in the "Queued" state for verification. This can lead to data loss after a failover.
-  - Affected versions: GitLab versions 15.6.x, 15.7.x, and 15.8.0 - 15.8.2.
-  - Versions containing fix: GitLab 15.8.3 and later.
-- Due to [a bug introduced in GitLab 15.4](https://gitlab.com/gitlab-org/gitlab/-/issues/390155), if one or more Git repositories in Gitaly Cluster is [unavailable](../administration/gitaly/recovery.md#unavailable-repositories), then [Repository checks](../administration/repository_checks.md#repository-checks) and [Geo replication and verification](../administration/geo/index.md) stop running for all project or project wiki repositories in the affected Gitaly Cluster. The bug was fixed by [reverting the change in GitLab 15.9.0](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/110823). Before upgrading to this version, check if you have any "unavailable" repositories. See [the bug issue](https://gitlab.com/gitlab-org/gitlab/-/issues/390155) for more information.
-
-### 15.6.2
-
-- Geo: [Container registry push events are rejected](https://gitlab.com/gitlab-org/gitlab/-/issues/386389) by the `/api/v4/container_registry_event/events` endpoint resulting in Geo secondary sites not being aware of updates to container registry images and subsequently not replicating the updates. Secondary sites may contain out of date container images after a failover as a consequence. This affects versions 15.6.0 - 15.6.6 and 15.7.0 - 15.7.2. If you're using Geo with container repositories, you are advised to upgrade to GitLab 15.6.7, 15.7.3, or 15.8.0 which contain a fix for this issue and avoid potential data loss after a failover.
-- Geo: We discovered an issue where [replication and verification of projects and wikis was not keeping up](https://gitlab.com/gitlab-org/gitlab/-/issues/387980) on small number of Geo installations. Your installation may be affected if you see some projects and/or wikis persistently in the "Queued" state for verification. This can lead to data loss after a failover.
-  - Affected versions: GitLab versions 15.6.x, 15.7.x, and 15.8.0 - 15.8.2.
-  - Versions containing fix: GitLab 15.8.3 and later.
-- Due to [a bug introduced in GitLab 15.4](https://gitlab.com/gitlab-org/gitlab/-/issues/390155), if one or more Git repositories in Gitaly Cluster is [unavailable](../administration/gitaly/recovery.md#unavailable-repositories), then [Repository checks](../administration/repository_checks.md#repository-checks) and [Geo replication and verification](../administration/geo/index.md) stop running for all project or project wiki repositories in the affected Gitaly Cluster. The bug was fixed by [reverting the change in GitLab 15.9.0](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/110823). Before upgrading to this version, check if you have any "unavailable" repositories. See [the bug issue](https://gitlab.com/gitlab-org/gitlab/-/issues/390155) for more information.
-
-### 15.6.1
-
-- Geo: [Container registry push events are rejected](https://gitlab.com/gitlab-org/gitlab/-/issues/386389) by the `/api/v4/container_registry_event/events` endpoint resulting in Geo secondary sites not being aware of updates to container registry images and subsequently not replicating the updates. Secondary sites may contain out of date container images after a failover as a consequence. This affects versions 15.6.0 - 15.6.6 and 15.7.0 - 15.7.2. If you're using Geo with container repositories, you are advised to upgrade to GitLab 15.6.7, 15.7.3, or 15.8.0 which contain a fix for this issue and avoid potential data loss after a failover.
-- Geo: We discovered an issue where [replication and verification of projects and wikis was not keeping up](https://gitlab.com/gitlab-org/gitlab/-/issues/387980) on small number of Geo installations. Your installation may be affected if you see some projects and/or wikis persistently in the "Queued" state for verification. This can lead to data loss after a failover.
-  - Affected versions: GitLab versions 15.6.x, 15.7.x, and 15.8.0 - 15.8.2.
-  - Versions containing fix: GitLab 15.8.3 and later.
-- Due to [a bug introduced in GitLab 15.4](https://gitlab.com/gitlab-org/gitlab/-/issues/390155), if one or more Git repositories in Gitaly Cluster is [unavailable](../administration/gitaly/recovery.md#unavailable-repositories), then [Repository checks](../administration/repository_checks.md#repository-checks) and [Geo replication and verification](../administration/geo/index.md) stop running for all project or project wiki repositories in the affected Gitaly Cluster. The bug was fixed by [reverting the change in GitLab 15.9.0](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/110823). Before upgrading to this version, check if you have any "unavailable" repositories. See [the bug issue](https://gitlab.com/gitlab-org/gitlab/-/issues/390155) for more information.
-
-### 15.6.0
-
-- You should use one of the [officially supported PostgreSQL versions](../administration/package_information/postgresql_versions.md). Some database migrations can cause stability and performance issues with older PostgreSQL versions.
-- Git 2.37.0 and later is required by Gitaly. For self-compiled installations, you should use the [Git version provided by Gitaly](../install/installation.md#git).
-- A database change to modify the behavior of four indexes fails on instances
-  where these indexes do not exist:
-
-  ```plaintext
-  Caused by:
-  PG::UndefinedTable: ERROR:  relation "index_issues_on_title_trigram" does not exist
-  ```
-
-  The other three indexes are: `index_merge_requests_on_title_trigram`, `index_merge_requests_on_description_trigram`,
-  and `index_issues_on_description_trigram`.
-
-  This issue was [fixed in GitLab 15.7](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/105375) and backported
-  to GitLab 15.6.2. The issue can also be worked around:
-  [read about how to create these indexes](https://gitlab.com/gitlab-org/gitlab/-/issues/378343#note_1199863087).
-- Geo: [Container registry push events are rejected](https://gitlab.com/gitlab-org/gitlab/-/issues/386389) by the `/api/v4/container_registry_event/events` endpoint resulting in Geo secondary sites not being aware of updates to container registry images and subsequently not replicating the updates. Secondary sites may contain out of date container images after a failover as a consequence. This affects versions 15.6.0 - 15.6.6 and 15.7.0 - 15.7.2. If you're using Geo with container repositories, you are advised to upgrade to GitLab 15.6.7, 15.7.3, or 15.8.0 which contain a fix for this issue and avoid potential data loss after a failover.
-- Geo: We discovered an issue where [replication and verification of projects and wikis was not keeping up](https://gitlab.com/gitlab-org/gitlab/-/issues/387980) on small number of Geo installations. Your installation may be affected if you see some projects and/or wikis persistently in the "Queued" state for verification. This can lead to data loss after a failover.
-  - Affected versions: GitLab versions 15.6.x, 15.7.x, and 15.8.0 - 15.8.2.
-  - Versions containing fix: GitLab 15.8.3 and later.
-- Due to [a bug introduced in GitLab 15.4](https://gitlab.com/gitlab-org/gitlab/-/issues/390155), if one or more Git repositories in Gitaly Cluster is [unavailable](../administration/gitaly/recovery.md#unavailable-repositories), then [Repository checks](../administration/repository_checks.md#repository-checks) and [Geo replication and verification](../administration/geo/index.md) stop running for all project or project wiki repositories in the affected Gitaly Cluster. The bug was fixed by [reverting the change in GitLab 15.9.0](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/110823). Before upgrading to this version, check if you have any "unavailable" repositories. See [the bug issue](https://gitlab.com/gitlab-org/gitlab/-/issues/390155) for more information.
-
-### 15.5.5
-
-- Due to [a bug introduced in GitLab 15.4](https://gitlab.com/gitlab-org/gitlab/-/issues/390155), if one or more Git repositories in Gitaly Cluster is [unavailable](../administration/gitaly/recovery.md#unavailable-repositories), then [Repository checks](../administration/repository_checks.md#repository-checks) and [Geo replication and verification](../administration/geo/index.md) stop running for all project or project wiki repositories in the affected Gitaly Cluster. The bug was fixed by [reverting the change in GitLab 15.9.0](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/110823). Before upgrading to this version, check if you have any "unavailable" repositories. See [the bug issue](https://gitlab.com/gitlab-org/gitlab/-/issues/390155) for more information.
-
-### 15.5.4
-
-- Due to [a bug introduced in GitLab 15.4](https://gitlab.com/gitlab-org/gitlab/-/issues/390155), if one or more Git repositories in Gitaly Cluster is [unavailable](../administration/gitaly/recovery.md#unavailable-repositories), then [Repository checks](../administration/repository_checks.md#repository-checks) and [Geo replication and verification](../administration/geo/index.md) stop running for all project or project wiki repositories in the affected Gitaly Cluster. The bug was fixed by [reverting the change in GitLab 15.9.0](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/110823). Before upgrading to this version, check if you have any "unavailable" repositories. See [the bug issue](https://gitlab.com/gitlab-org/gitlab/-/issues/390155) for more information.
-
-### 15.5.3
-
-- GitLab 15.4.0 introduced a default [Sidekiq routing rule](../administration/sidekiq/processing_specific_job_classes.md#routing-rules) that routes all jobs to the `default` queue. For instances using [queue selectors](../administration/sidekiq/processing_specific_job_classes.md#queue-selectors-deprecated), this causes [performance problems](https://gitlab.com/gitlab-com/gl-infra/scalability/-/issues/1991) as some Sidekiq processes will be idle.
-  - The default routing rule has been reverted in 15.5.4, so upgrading to that version or later will return to the previous behavior.
-  - If a GitLab instance now listens only to the `default` queue (which is not currently recommended), it will be required to add this routing rule back in `/etc/gitlab/gitlab.rb`:
-
-    ```ruby
-    sidekiq['routing_rules'] = [['*', 'default']]
-    ```
-
-- Due to [a bug introduced in GitLab 15.4](https://gitlab.com/gitlab-org/gitlab/-/issues/390155), if one or more Git repositories in Gitaly Cluster is [unavailable](../administration/gitaly/recovery.md#unavailable-repositories), then [Repository checks](../administration/repository_checks.md#repository-checks) and [Geo replication and verification](../administration/geo/index.md) stop running for all project or project wiki repositories in the affected Gitaly Cluster. The bug was fixed by [reverting the change in GitLab 15.9.0](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/110823). Before upgrading to this version, check if you have any "unavailable" repositories. See [the bug issue](https://gitlab.com/gitlab-org/gitlab/-/issues/390155) for more information.
-
-### 15.5.2
-
-- GitLab 15.4.0 introduced a default [Sidekiq routing rule](../administration/sidekiq/processing_specific_job_classes.md#routing-rules) that routes all jobs to the `default` queue. For instances using [queue selectors](../administration/sidekiq/processing_specific_job_classes.md#queue-selectors-deprecated), this causes [performance problems](https://gitlab.com/gitlab-com/gl-infra/scalability/-/issues/1991) as some Sidekiq processes will be idle.
-  - The default routing rule has been reverted in 15.5.4, so upgrading to that version or later will return to the previous behavior.
-  - If a GitLab instance now listens only to the `default` queue (which is not currently recommended), it will be required to add this routing rule back in `/etc/gitlab/gitlab.rb`:
-
-    ```ruby
-    sidekiq['routing_rules'] = [['*', 'default']]
-    ```
-
-- Due to [a bug introduced in GitLab 15.4](https://gitlab.com/gitlab-org/gitlab/-/issues/390155), if one or more Git repositories in Gitaly Cluster is [unavailable](../administration/gitaly/recovery.md#unavailable-repositories), then [Repository checks](../administration/repository_checks.md#repository-checks) and [Geo replication and verification](../administration/geo/index.md) stop running for all project or project wiki repositories in the affected Gitaly Cluster. The bug was fixed by [reverting the change in GitLab 15.9.0](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/110823). Before upgrading to this version, check if you have any "unavailable" repositories. See [the bug issue](https://gitlab.com/gitlab-org/gitlab/-/issues/390155) for more information.
-
-### 15.5.1
-
-- GitLab 15.4.0 introduced a default [Sidekiq routing rule](../administration/sidekiq/processing_specific_job_classes.md#routing-rules) that routes all jobs to the `default` queue. For instances using [queue selectors](../administration/sidekiq/processing_specific_job_classes.md#queue-selectors-deprecated), this causes [performance problems](https://gitlab.com/gitlab-com/gl-infra/scalability/-/issues/1991) as some Sidekiq processes will be idle.
-  - The default routing rule has been reverted in 15.5.4, so upgrading to that version or later will return to the previous behavior.
-  - If a GitLab instance now listens only to the `default` queue (which is not currently recommended), it will be required to add this routing rule back in `/etc/gitlab/gitlab.rb`:
-
-    ```ruby
-    sidekiq['routing_rules'] = [['*', 'default']]
-    ```
-
-- Due to [a bug introduced in GitLab 15.4](https://gitlab.com/gitlab-org/gitlab/-/issues/390155), if one or more Git repositories in Gitaly Cluster is [unavailable](../administration/gitaly/recovery.md#unavailable-repositories), then [Repository checks](../administration/repository_checks.md#repository-checks) and [Geo replication and verification](../administration/geo/index.md) stop running for all project or project wiki repositories in the affected Gitaly Cluster. The bug was fixed by [reverting the change in GitLab 15.9.0](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/110823). Before upgrading to this version, check if you have any "unavailable" repositories. See [the bug issue](https://gitlab.com/gitlab-org/gitlab/-/issues/390155) for more information.
-
-### 15.5.0
-
-- GitLab 15.4.0 introduced a default [Sidekiq routing rule](../administration/sidekiq/processing_specific_job_classes.md#routing-rules) that routes all jobs to the `default` queue. For instances using [queue selectors](../administration/sidekiq/processing_specific_job_classes.md#queue-selectors-deprecated), this causes [performance problems](https://gitlab.com/gitlab-com/gl-infra/scalability/-/issues/1991) as some Sidekiq processes will be idle.
-  - The default routing rule has been reverted in 15.5.4, so upgrading to that version or later will return to the previous behavior.
-  - If a GitLab instance now listens only to the `default` queue (which is not currently recommended), it will be required to add this routing rule back in `/etc/gitlab/gitlab.rb`:
-
-    ```ruby
-    sidekiq['routing_rules'] = [['*', 'default']]
-    ```
-
-- Due to [a bug introduced in GitLab 15.4](https://gitlab.com/gitlab-org/gitlab/-/issues/390155), if one or more Git repositories in Gitaly Cluster is [unavailable](../administration/gitaly/recovery.md#unavailable-repositories), then [Repository checks](../administration/repository_checks.md#repository-checks) and [Geo replication and verification](../administration/geo/index.md) stop running for all project or project wiki repositories in the affected Gitaly Cluster. The bug was fixed by [reverting the change in GitLab 15.9.0](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/110823). Before upgrading to this version, check if you have any "unavailable" repositories. See [the bug issue](https://gitlab.com/gitlab-org/gitlab/-/issues/390155) for more information.
-
-### 15.4.6
-
-- Due to a [bug introduced in curl in GitLab 15.4.6](https://github.com/curl/curl/issues/10122), the [`no_proxy` environment variable may not work properly](../administration/geo/replication/troubleshooting.md#secondary-site-returns-received-http-code-403-from-proxy-after-connect). Either downgrade to GitLab 15.4.5, or upgrade to GitLab 15.5.7 or a later version.
-- Due to [a bug introduced in GitLab 15.4](https://gitlab.com/gitlab-org/gitlab/-/issues/390155), if one or more Git repositories in Gitaly Cluster is [unavailable](../administration/gitaly/recovery.md#unavailable-repositories), then [Repository checks](../administration/repository_checks.md#repository-checks) and [Geo replication and verification](../administration/geo/index.md) stop running for all project or project wiki repositories in the affected Gitaly Cluster. The bug was fixed by [reverting the change in GitLab 15.9.0](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/110823). Before upgrading to this version, check if you have any "unavailable" repositories. See [the bug issue](https://gitlab.com/gitlab-org/gitlab/-/issues/390155) for more information.
-
-### 15.4.5
-
-- Due to [a bug introduced in GitLab 15.4](https://gitlab.com/gitlab-org/gitlab/-/issues/390155), if one or more Git repositories in Gitaly Cluster is [unavailable](../administration/gitaly/recovery.md#unavailable-repositories), then [Repository checks](../administration/repository_checks.md#repository-checks) and [Geo replication and verification](../administration/geo/index.md) stop running for all project or project wiki repositories in the affected Gitaly Cluster. The bug was fixed by [reverting the change in GitLab 15.9.0](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/110823). Before upgrading to this version, check if you have any "unavailable" repositories. See [the bug issue](https://gitlab.com/gitlab-org/gitlab/-/issues/390155) for more information.
-
-### 15.4.4
-
-- Due to [a bug introduced in GitLab 15.4](https://gitlab.com/gitlab-org/gitlab/-/issues/390155), if one or more Git repositories in Gitaly Cluster is [unavailable](../administration/gitaly/recovery.md#unavailable-repositories), then [Repository checks](../administration/repository_checks.md#repository-checks) and [Geo replication and verification](../administration/geo/index.md) stop running for all project or project wiki repositories in the affected Gitaly Cluster. The bug was fixed by [reverting the change in GitLab 15.9.0](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/110823). Before upgrading to this version, check if you have any "unavailable" repositories. See [the bug issue](https://gitlab.com/gitlab-org/gitlab/-/issues/390155) for more information.
-
-### 15.4.3
-
-- Due to [a bug introduced in GitLab 15.4](https://gitlab.com/gitlab-org/gitlab/-/issues/390155), if one or more Git repositories in Gitaly Cluster is [unavailable](../administration/gitaly/recovery.md#unavailable-repositories), then [Repository checks](../administration/repository_checks.md#repository-checks) and [Geo replication and verification](../administration/geo/index.md) stop running for all project or project wiki repositories in the affected Gitaly Cluster. The bug was fixed by [reverting the change in GitLab 15.9.0](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/110823). Before upgrading to this version, check if you have any "unavailable" repositories. See [the bug issue](https://gitlab.com/gitlab-org/gitlab/-/issues/390155) for more information.
-
-### 15.4.2
-
-- A [license caching issue](https://gitlab.com/gitlab-org/gitlab/-/issues/376706) prevents some premium features of GitLab from working correctly if you add a new license. Workarounds for this issue:
-  - Restart all Rails, Sidekiq and Gitaly nodes after applying a new license. This clears the relevant license caches and allows all premium features to operate correctly.
-  - Upgrade to a version that is not affected by this issue. The following upgrade paths are available for affected versions:
-    - 15.2.5 --> 15.3.5
-    - 15.3.0 - 15.3.4 --> 15.3.5
-    - 15.4.1 --> 15.4.3
-- Due to [a bug introduced in GitLab 15.4](https://gitlab.com/gitlab-org/gitlab/-/issues/390155), if one or more Git repositories in Gitaly Cluster is [unavailable](../administration/gitaly/recovery.md#unavailable-repositories), then [Repository checks](../administration/repository_checks.md#repository-checks) and [Geo replication and verification](../administration/geo/index.md) stop running for all project or project wiki repositories in the affected Gitaly Cluster. The bug was fixed by [reverting the change in GitLab 15.9.0](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/110823). Before upgrading to this version, check if you have any "unavailable" repositories. See [the bug issue](https://gitlab.com/gitlab-org/gitlab/-/issues/390155) for more information.
-
-### 15.4.1
-
-- A [license caching issue](https://gitlab.com/gitlab-org/gitlab/-/issues/376706) prevents some premium features of GitLab from working correctly if you add a new license. Workarounds for this issue:
-  - Restart all Rails, Sidekiq and Gitaly nodes after applying a new license. This clears the relevant license caches and allows all premium features to operate correctly.
-  - Upgrade to a version that is not affected by this issue. The following upgrade paths are available for affected versions:
-    - 15.2.5 --> 15.3.5
-    - 15.3.0 - 15.3.4 --> 15.3.5
-    - 15.4.1 --> 15.4.3
-- Due to [a bug introduced in GitLab 15.4](https://gitlab.com/gitlab-org/gitlab/-/issues/390155), if one or more Git repositories in Gitaly Cluster is [unavailable](../administration/gitaly/recovery.md#unavailable-repositories), then [Repository checks](../administration/repository_checks.md#repository-checks) and [Geo replication and verification](../administration/geo/index.md) stop running for all project or project wiki repositories in the affected Gitaly Cluster. The bug was fixed by [reverting the change in GitLab 15.9.0](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/110823). Before upgrading to this version, check if you have any "unavailable" repositories. See [the bug issue](https://gitlab.com/gitlab-org/gitlab/-/issues/390155) for more information.
-
-### 15.4.0
-
-- GitLab 15.4.0 includes a [batched background migration](background_migrations.md#batched-background-migrations) to [remove incorrect values from `expire_at` in `ci_job_artifacts` table](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/89318).
-  This migration might take hours or days to complete on larger GitLab instances.
-- By default, Gitaly and Praefect nodes use the time server at `pool.ntp.org`. If your instance can not connect to `pool.ntp.org`, [configure the `NTP_HOST` variable](../administration/gitaly/praefect.md#customize-time-server-setting).
-- GitLab 15.4.0 introduced a default [Sidekiq routing rule](../administration/sidekiq/processing_specific_job_classes.md#routing-rules) that routes all jobs to the `default` queue. For instances using [queue selectors](../administration/sidekiq/processing_specific_job_classes.md#queue-selectors-deprecated), this causes [performance problems](https://gitlab.com/gitlab-com/gl-infra/scalability/-/issues/1991) as some Sidekiq processes will be idle.
-  - The default routing rule has been reverted in 15.4.5, so upgrading to that version or later will return to the previous behavior.
-  - If a GitLab instance now listens only to the `default` queue (which is not currently recommended), it will be required to add this routing rule back in `/etc/gitlab/gitlab.rb`:
-
-    ```ruby
-    sidekiq['routing_rules'] = [['*', 'default']]
-    ```
-
-- New Git repositories created in Gitaly cluster [no longer use the `@hashed` storage path](#change-to-praefect-generated-replica-paths-in-gitlab-153). Server
-  hooks for new repositories must be copied into a different location.
-- The structure of `/etc/gitlab/gitlab-secrets.json` was modified in [GitLab 15.4](https://gitlab.com/gitlab-org/omnibus-gitlab/-/merge_requests/6310),
-  and new configuration was added to `gitlab_pages`, `grafana`, and `mattermost` sections.
-  In a highly available or GitLab Geo environment, secrets need to be the same on all nodes.
-  If you're manually syncing the secrets file across nodes, or manually specifying secrets in
-  `/etc/gitlab/gitlab.rb`, make sure `/etc/gitlab/gitlab-secrets.json` is the same on all nodes.
-- GitLab 15.4.0 introduced a [batched background migration](background_migrations.md#batched-background-migrations) to
-  [backfill `namespace_id` values on issues table](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/91921). This
-  migration might take multiple hours or days to complete on larger GitLab instances. Make sure the migration
-  has completed successfully before upgrading to 15.7.0 or later.
-- Due to [a bug introduced in GitLab 15.4](https://gitlab.com/gitlab-org/gitlab/-/issues/390155), if one or more Git repositories in Gitaly Cluster is [unavailable](../administration/gitaly/recovery.md#unavailable-repositories), then [Repository checks](../administration/repository_checks.md#repository-checks) and [Geo replication and verification](../administration/geo/index.md) stop running for all project or project wiki repositories in the affected Gitaly Cluster. The bug was fixed by [reverting the change in GitLab 15.9.0](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/110823). Before upgrading to this version, check if you have any "unavailable" repositories. See [the bug issue](https://gitlab.com/gitlab-org/gitlab/-/issues/390155) for more information.
-- A redesigned sign-in page is enabled by default in GitLab 15.4 and later, with improvements shipping in later releases. For more information, see [epic 8557](https://gitlab.com/groups/gitlab-org/-/epics/8557).
-  It can be disabled with a feature flag. Start [a Rails console](../administration/operations/rails_console.md) and run:
-
-  ```ruby
-  Feature.disable(:restyle_login_page)
-  ```
-
-### 15.3.4
-
-A [license caching issue](https://gitlab.com/gitlab-org/gitlab/-/issues/376706) prevents some premium features of GitLab from working correctly if you add a new license. Workarounds for this issue:
-
-- Restart all Rails, Sidekiq and Gitaly nodes after applying a new license. This clears the relevant license caches and allows all premium features to operate correctly.
-- Upgrade to a version that is not affected by this issue. The following upgrade paths are available for affected versions:
-  - 15.2.5 --> 15.3.5
-  - 15.3.0 - 15.3.4 --> 15.3.5
-  - 15.4.1 --> 15.4.3
-
-### 15.3.3
-
-- In GitLab 15.3.3, [SAML Group Links](../api/groups.md#saml-group-links) API `access_level` attribute type changed to `integer`. See
-[the API documentation](../api/members.md).
-- A [license caching issue](https://gitlab.com/gitlab-org/gitlab/-/issues/376706) prevents some premium features of GitLab from working correctly if you add a new license. Workarounds for this issue:
-
-  - Restart all Rails, Sidekiq and Gitaly nodes after applying a new license. This clears the relevant license caches and allows all premium features to operate correctly.
-  - Upgrade to a version that is not affected by this issue. The following upgrade paths are available for affected versions:
-    - 15.2.5 --> 15.3.5
-    - 15.3.0 - 15.3.4 --> 15.3.5
-    - 15.4.1 --> 15.4.3
-
-### 15.3.2
-
-A [license caching issue](https://gitlab.com/gitlab-org/gitlab/-/issues/376706) prevents some premium features of GitLab from working correctly if you add a new license. Workarounds for this issue:
-
-- Restart all Rails, Sidekiq and Gitaly nodes after applying a new license. This clears the relevant license caches and allows all premium features to operate correctly.
-- Upgrade to a version that is not affected by this issue. The following upgrade paths are available for affected versions:
-  - 15.2.5 --> 15.3.5
-  - 15.3.0 - 15.3.4 --> 15.3.5
-  - 15.4.1 --> 15.4.3
-
-### 15.3.1
-
-A [license caching issue](https://gitlab.com/gitlab-org/gitlab/-/issues/376706) prevents some premium features of GitLab from working correctly if you add a new license. Workarounds for this issue:
-
-- Restart all Rails, Sidekiq and Gitaly nodes after applying a new license. This clears the relevant license caches and allows all premium features to operate correctly.
-- Upgrade to a version that is not affected by this issue. The following upgrade paths are available for affected versions:
-  - 15.2.5 --> 15.3.5
-  - 15.3.0 - 15.3.4 --> 15.3.5
-  - 15.4.1 --> 15.4.3
-
-### 15.3.0
-
-- [Incorrect deletion of object storage files on Geo secondary sites](https://gitlab.com/gitlab-org/gitlab/-/issues/371397) can occur in certain situations. See [Geo: Incorrect object storage LFS file deletion on secondary site issue in GitLab 15.0.0 to 15.3.2](#geo-incorrect-object-storage-lfs-file-deletion-on-secondary-sites-in-gitlab-1500-to-1532).
-- LFS transfers can [redirect to the primary from secondary site mid-session](https://gitlab.com/gitlab-org/gitlab/-/issues/371571) causing failed pull and clone requests when [Geo proxying](../administration/geo/secondary_proxy/index.md) is enabled. Geo proxying is enabled by default in GitLab 15.1 and later. See [Geo: LFS transfer redirect to primary from secondary site mid-session issue in GitLab 15.1.0 to 15.3.2](#geo-lfs-transfers-redirect-to-primary-from-secondary-site-mid-session-in-gitlab-1510-to-1532) for more details.
-- New Git repositories created in Gitaly cluster [no longer use the `@hashed` storage path](#change-to-praefect-generated-replica-paths-in-gitlab-153). Server
-  hooks for new repositories must be copied into a different location.
-- A [license caching issue](https://gitlab.com/gitlab-org/gitlab/-/issues/376706) prevents some premium features of GitLab from working correctly if you add a new license. Workarounds for this issue:
-
-  - Restart all Rails, Sidekiq and Gitaly nodes after applying a new license. This clears the relevant license caches and allows all premium features to operate correctly.
-  - Upgrade to a version that is not affected by this issue. The following upgrade paths are available for affected versions:
-    - 15.2.5 --> 15.3.5
-    - 15.3.0 - 15.3.4 --> 15.3.5
-    - 15.4.1 --> 15.4.3
-
-### 15.2.5
-
-A [license caching issue](https://gitlab.com/gitlab-org/gitlab/-/issues/376706) prevents some premium features of GitLab from working correctly if you add a new license. Workarounds for this issue:
-
-- Restart all Rails, Sidekiq and Gitaly nodes after applying a new license. This clears the relevant license caches and allows all premium features to operate correctly.
-- Upgrade to a version that is not affected by this issue. The following upgrade paths are available for affected versions:
-  - 15.2.5 --> 15.3.5
-  - 15.3.0 - 15.3.4 --> 15.3.5
-  - 15.4.1 --> 15.4.3
-
-### 15.2.0
-
-- GitLab installations that have multiple web nodes should be
-  [upgraded to 15.1](#1510) before upgrading to 15.2 (and later) due to a
-  configuration change in Rails that can result in inconsistent ETag key
-  generation.
-- Some Sidekiq workers were renamed in this release. To avoid any disruption, [run the Rake tasks to migrate any pending jobs](../administration/sidekiq/sidekiq_job_migration.md#migrate-queued-and-future-jobs) before starting the upgrade to GitLab 15.2.0.
-- Gitaly now executes its binaries in a [runtime location](https://gitlab.com/gitlab-org/gitaly/-/merge_requests/4670). By default on Omnibus GitLab,
-  this path is `/var/opt/gitlab/gitaly/run/`. If this location is mounted with `noexec`, merge requests generate the following error:
-
-  ```plaintext
-  fork/exec /var/opt/gitlab/gitaly/run/gitaly-<nnnn>/gitaly-git2go-v15: permission denied
-  ```
-
-  To resolve this, remove the `noexec` option from the file system mount. An alternative is to change the Gitaly runtime directory:
-
-  1. Add `gitaly['runtime_dir'] = '<PATH_WITH_EXEC_PERM>'` to `/etc/gitlab/gitlab.rb` and specify a location without `noexec` set.
-  1. Run `sudo gitlab-ctl reconfigure`.
-- [Incorrect deletion of object storage files on Geo secondary sites](https://gitlab.com/gitlab-org/gitlab/-/issues/371397) can occur in certain situations. See [Geo: Incorrect object storage LFS file deletion on secondary site issue in GitLab 15.0.0 to 15.3.2](#geo-incorrect-object-storage-lfs-file-deletion-on-secondary-sites-in-gitlab-1500-to-1532).
-- LFS transfers can [redirect to the primary from secondary site mid-session](https://gitlab.com/gitlab-org/gitlab/-/issues/371571) causing failed pull and clone requests when [Geo proxying](../administration/geo/secondary_proxy/index.md) is enabled. Geo proxying is enabled by default in GitLab 15.1 and later. See [Geo: LFS transfer redirect to primary from secondary site mid-session issue in GitLab 15.1.0 to 15.3.2](#geo-lfs-transfers-redirect-to-primary-from-secondary-site-mid-session-in-gitlab-1510-to-1532) for more details.
-
-### 15.1.0
-
-- If you run external PostgreSQL, particularly AWS RDS,
-  [check you have a PostgreSQL bug fix](#postgresql-segmentation-fault-issue)
-  to avoid the database crashing.
-- In GitLab 15.1.0, we are switching Rails `ActiveSupport::Digest` to use SHA256 instead of MD5.
-  This affects ETag key generation for resources such as raw Snippet file
-  downloads. To ensure consistent ETag key generation across multiple
-  web nodes when upgrading, all servers must first be upgraded to 15.1.Z before
-  upgrading to 15.2.0 or later:
-
-  1. Ensure all GitLab web nodes are running GitLab 15.1.Z.
-  1. [Enable the `active_support_hash_digest_sha256` feature flag](../administration/feature_flags.md#how-to-enable-and-disable-features-behind-flags) to switch `ActiveSupport::Digest` to use SHA256:
-
-     1. [Start the rails console](../administration/operations/rails_console.md)
-     1. Enable the feature flag:
-
-        ```ruby
-        Feature.enable(:active_support_hash_digest_sha256)
-        ```
-
-  1. Only then, continue to upgrade to later versions of GitLab.
-- Unauthenticated requests to the [`ciConfig` GraphQL field](../api/graphql/reference/index.md#queryciconfig) are no longer supported.
-  Before you upgrade to GitLab 15.1, add an [access token](../api/rest/index.md#authentication) to your requests.
-  The user creating the token must have [permission](../user/permissions.md) to create pipelines in the project.
-- [Incorrect deletion of object storage files on Geo secondary sites](https://gitlab.com/gitlab-org/gitlab/-/issues/371397) can occur in certain situations. See [Geo: Incorrect object storage LFS file deletion on secondary site issue in GitLab 15.0.0 to 15.3.2](#geo-incorrect-object-storage-lfs-file-deletion-on-secondary-sites-in-gitlab-1500-to-1532).
-- LFS transfers can [redirect to the primary from secondary site mid-session](https://gitlab.com/gitlab-org/gitlab/-/issues/371571) causing failed pull and clone requests when [Geo proxying](../administration/geo/secondary_proxy/index.md) is enabled. Geo proxying is enabled by default in GitLab 15.1 and later. See [Geo: LFS transfer redirect to primary from secondary site mid-session issue in GitLab 15.1.0 to 15.3.2](#geo-lfs-transfers-redirect-to-primary-from-secondary-site-mid-session-in-gitlab-1510-to-1532) for more details.
-
-### 15.0.0
-
-- Elasticsearch 6.8 [is no longer supported](../integration/advanced_search/elasticsearch.md#version-requirements). Before you upgrade to GitLab 15.0, [update Elasticsearch to any 7.x version](../integration/advanced_search/elasticsearch.md#upgrade-to-a-new-elasticsearch-major-version).
-- If you run external PostgreSQL, particularly AWS RDS,
-  [check you have a PostgreSQL bug fix](#postgresql-segmentation-fault-issue)
-  to avoid the database crashing.
-- The use of encrypted S3 buckets with storage-specific configuration is no longer supported after [removing support for using `background_upload`](deprecations.md#background-upload-for-object-storage).
-- The [certificate-based Kubernetes integration (DEPRECATED)](../user/infrastructure/clusters/index.md#certificate-based-kubernetes-integration-deprecated) is disabled by default, but you can be re-enable it through the [`certificate_based_clusters` feature flag](../administration/feature_flags.md#how-to-enable-and-disable-features-behind-flags) until GitLab 16.0.
-- When you use the GitLab Helm Chart project with a custom `serviceAccount`, ensure it has `get` and `list` permissions for the `serviceAccount` and `secret` resources.
-- The [`custom_hooks_dir`](../administration/server_hooks.md#create-global-server-hooks-for-all-repositories) setting for configuring global server hooks is now configured in
-  Gitaly. The previous implementation in GitLab Shell was removed in GitLab 15.0. With this change, global server hooks are stored only inside a subdirectory named after the
-  hook type. Global server hooks can no longer be a single hook file in the root of the custom hooks directory. For example, you must use `<custom_hooks_dir>/<hook_name>.d/*` rather
-  than `<custom_hooks_dir>/<hook_name>`.
-  - Use `gitaly['custom_hooks_dir']` in `gitlab.rb` ([introduced in 14.3](https://gitlab.com/gitlab-org/omnibus-gitlab/-/merge_requests/4208))
-    for Omnibus GitLab. This replaces `gitlab_shell['custom_hooks_dir']`.
-- [Incorrect deletion of object storage files on Geo secondary sites](https://gitlab.com/gitlab-org/gitlab/-/issues/371397) can occur in certain situations. See [Geo: Incorrect object storage LFS file deletion on secondary site issue in GitLab 15.0.0 to 15.3.2](#geo-incorrect-object-storage-lfs-file-deletion-on-secondary-sites-in-gitlab-1500-to-1532).
-- The `FF_GITLAB_REGISTRY_HELPER_IMAGE` [feature flag](../administration/feature_flags.md#enable-or-disable-the-feature) is removed and helper images are always pulled from GitLab Registry.
-- The `AES256-GCM-SHA384` SSL cipher is no longer allowed by NGINX.
-  See how you can [add the cipher back](https://docs.gitlab.com/omnibus/update/gitlab_15_changes.html#aes256-gcm-sha384-ssl-cipher-no-longer-allowed-by-default-by-nginx) to the allow list.
-- Support for more than one database has been added to GitLab. For **self-compiled (source) installations**,
-  `config/database.yml` must include a database name in the database configuration.
-  The `main: database` must be first. If an invalid or deprecated syntax is used, an error is generated
-  during application start:
-
-  ```plaintext
-  ERROR: This installation of GitLab uses unsupported 'config/database.yml'.
-  The main: database needs to be defined as a first configuration item instead of primary. (RuntimeError)
-  ```
-
-  Previously, the `config/database.yml` file looked like the following:
-
-  ```yaml
-  production:
-    adapter: postgresql
-    encoding: unicode
-    database: gitlabhq_production
-    ...
-  ```
-
-  Starting with GitLab 15.0, it must define a `main` database first:
-
-  ```yaml
-  production:
-    main:
-      adapter: postgresql
-      encoding: unicode
-      database: gitlabhq_production
-      ...
-  ```
-
-### 14.10.0
-
-- Before upgrading to GitLab 14.10, you must already have the latest 14.9.Z installed on your instance.
-  The upgrade to GitLab 14.10 executes a [concurrent index drop](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/84308) of unneeded
-  entries from the `ci_job_artifacts` database table. This could potentially run for multiple minutes, especially if the table has a lot of
-  traffic and the migration is unable to acquire a lock. It is advised to let this process finish as restarting may result in data loss.
-
-- If you run external PostgreSQL, particularly AWS RDS,
-  [check you have a PostgreSQL bug fix](#postgresql-segmentation-fault-issue)
-  to avoid the database crashing.
-
-- Upgrading to patch level 14.10.3 or later might encounter a one-hour timeout due to a long running database data change,
-  if it was not completed while running GitLab 14.9.
-
-  ```plaintext
-  FATAL: Mixlib::ShellOut::CommandTimeout: rails_migration[gitlab-rails]
-  (gitlab::database_migrations line 51) had an error:
-  [..]
-  Mixlib::ShellOut::CommandTimeout: Command timed out after 3600s:
-  ```
-
-  A workaround exists to [complete the data change and the upgrade manually](package/index.md#mixlibshelloutcommandtimeout-rails_migrationgitlab-rails--command-timed-out-after-3600s).
-
-### 14.9.0
-
-- Database changes made by the upgrade to GitLab 14.9 can take hours or days to complete on larger GitLab instances.
-  These [batched background migrations](background_migrations.md#batched-background-migrations) update whole database tables to ensure corresponding
-  records in `namespaces` table for each record in `projects` table.
-
-  After you upgrade to 14.9.0 or a later 14.9 patch version,
-  [batched background migrations must finish](background_migrations.md#batched-background-migrations)
-  before you upgrade to a later version.
-
-  If the migrations are not finished and you try to upgrade to a later version,
-  you see errors like:
-
-  ```plaintext
-  Expected batched background migration for the given configuration to be marked as 'finished', but it is 'active':
-  ```
-
-  Or
-
-  ```plaintext
-  Error executing action `run` on resource 'bash[migrate gitlab-rails database]'
-  ================================================================================
-
-  Mixlib::ShellOut::ShellCommandFailed
-  ------------------------------------
-  Command execution failed. STDOUT/STDERR suppressed for sensitive resource
-  ```
-
-- GitLab 14.9.0 includes a
-  [background migration `ResetDuplicateCiRunnersTokenValuesOnProjects`](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/79140)
-  that may remain stuck permanently in a **pending** state.
-
-  To clean up this stuck job, run the following in the [GitLab Rails Console](../administration/operations/rails_console.md):
-
-  ```ruby
-  Gitlab::Database::BackgroundMigrationJob.pending.where(class_name: "ResetDuplicateCiRunnersTokenValuesOnProjects").find_each do |job|
-    puts Gitlab::Database::BackgroundMigrationJob.mark_all_as_succeeded("ResetDuplicateCiRunnersTokenValuesOnProjects", job.arguments)
-  end
-  ```
-
-- If you run external PostgreSQL, particularly AWS RDS,
-  [check you have a PostgreSQL bug fix](#postgresql-segmentation-fault-issue)
-  to avoid the database crashing.
-
-### 14.8.0
-
-- If upgrading from a version earlier than 14.6.5, 14.7.4, or 14.8.2, review the [Critical Security Release: 14.8.2, 14.7.4, and 14.6.5](https://about.gitlab.com/releases/2022/02/25/critical-security-release-gitlab-14-8-2-released/) blog post.
-  Updating to 14.8.2 or later resets runner registration tokens for your groups and projects.
-- The agent server for Kubernetes [is enabled by default](https://about.gitlab.com/releases/2022/02/22/gitlab-14-8-released/#the-agent-server-for-kubernetes-is-enabled-by-default)
-  on Omnibus installations. If you run GitLab at scale,
-  such as [the reference architectures](../administration/reference_architectures/index.md),
-  you must disable the agent on the following server types, **if the agent is not required**.
-
-  - Praefect
-  - Gitaly
-  - Sidekiq
-  - Redis (if configured using `redis['enable'] = true` and not via `roles`)
-  - Container registry
-  - Any other server types based on `roles(['application_role'])`, such as the GitLab Rails nodes
-
-  [The reference architectures](../administration/reference_architectures/index.md) have been updated
-  with this configuration change and a specific role for standalone Redis servers.
-
-  Steps to disable the agent:
-
-  1. Add `gitlab_kas['enable'] = false` to `gitlab.rb`.
-  1. If the server is already upgraded to 14.8, run `gitlab-ctl reconfigure`.
-- GitLab 14.8.0 includes a
-[background migration `PopulateTopicsNonPrivateProjectsCount`](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/79140)
-that may remain stuck permanently in a **pending** state.
-
-  To clean up this stuck job, run the following in the [GitLab Rails Console](../administration/operations/rails_console.md):
-
-  ```ruby
-  Gitlab::Database::BackgroundMigrationJob.pending.where(class_name: "PopulateTopicsNonPrivateProjectsCount").find_each do |job|
-    puts Gitlab::Database::BackgroundMigrationJob.mark_all_as_succeeded("PopulateTopicsNonPrivateProjectsCount", job.arguments)
-  end
-  ```
-
-- If upgrading from a version earlier than 14.3.0, to avoid
-  [an issue with job retries](https://gitlab.com/gitlab-org/gitlab/-/issues/357822), first upgrade
-  to GitLab 14.7.x and make sure all batched migrations have finished.
-- If upgrading from version 14.3.0 or later, you might notice a failed
-  [batched migration](background_migrations.md#batched-background-migrations) named
-  `BackfillNamespaceIdForNamespaceRoute`. You can [ignore](https://gitlab.com/gitlab-org/gitlab/-/issues/357822)
-  this. Retry it after you upgrade to version 14.9.x.
-- If you run external PostgreSQL, particularly AWS RDS,
-  [check you have a PostgreSQL bug fix](#postgresql-segmentation-fault-issue)
-  to avoid the database crashing.
-
-### 14.7.0
-
-- See [LFS objects import and mirror issue in GitLab 14.6.0 to 14.7.2](#lfs-objects-import-and-mirror-issue-in-gitlab-1460-to-1472).
-- If upgrading from a version earlier than 14.6.5, 14.7.4, or 14.8.2, review the [Critical Security Release: 14.8.2, 14.7.4, and 14.6.5](https://about.gitlab.com/releases/2022/02/25/critical-security-release-gitlab-14-8-2-released/) blog post.
-  Updating to 14.7.4 or later resets runner registration tokens for your groups and projects.
-- GitLab 14.7 introduced a change where Gitaly expects persistent files in the `/tmp` directory.
-  When using the `noatime` mount option on `/tmp` in a node running Gitaly, most Linux distributions
-  run into [an issue with Git server hooks getting deleted](https://gitlab.com/gitlab-org/gitaly/-/issues/4113).
-  These conditions are present in the default Amazon Linux configuration.
-
-  If your Linux distribution manages files in `/tmp` with the `tmpfiles.d` service, you
-  can override the behavior of `tmpfiles.d` for the Gitaly files and avoid this issue:
-
-  ```shell
-  sudo printf "x /tmp/gitaly-%s-*\n" hooks git-exec-path >/etc/tmpfiles.d/gitaly-workaround.conf
-  ```
-
-  This issue is fixed in GitLab 14.10 and later when using the [Gitaly runtime directory](https://docs.gitlab.com/omnibus/update/gitlab_14_changes.html#gitaly-runtime-directory)
-  to specify a location to store persistent files.
-
-### 14.6.0
-
-- See [LFS objects import and mirror issue in GitLab 14.6.0 to 14.7.2](#lfs-objects-import-and-mirror-issue-in-gitlab-1460-to-1472).
-- If upgrading from a version earlier than 14.6.5, 14.7.4, or 14.8.2, review the [Critical Security Release: 14.8.2, 14.7.4, and 14.6.5](https://about.gitlab.com/releases/2022/02/25/critical-security-release-gitlab-14-8-2-released/) blog post.
-  Updating to 14.6.5 or later resets runner registration tokens for your groups and projects.
-
-### 14.5.0
-
-- When `make` is run, Gitaly builds are now created in `_build/bin` and no longer in the root directory of the source directory. If you
-are using a self-compiled installation, update paths to these binaries in your [systemd unit files](upgrading_from_source.md#configure-systemd-units)
-or [init scripts](upgrading_from_source.md#configure-sysv-init-script) by [following the documentation](upgrading_from_source.md).
-
-- Connections between Workhorse and Gitaly use the Gitaly `backchannel` protocol by default. If you deployed a gRPC proxy between Workhorse and Gitaly,
-  Workhorse can no longer connect. As a workaround, [disable the temporary `workhorse_use_sidechannel`](../administration/feature_flags.md#enable-or-disable-the-feature)
-  feature flag. If you need a proxy between Workhorse and Gitaly, use a TCP proxy. If you have feedback about this change, go to [this issue](https://gitlab.com/gitlab-com/gl-infra/scalability/-/issues/1301).
-
-- In 14.1 we introduced a background migration that changes how we store merge request diff commits,
-  to significantly reduce the amount of storage needed.
-  In 14.5 we introduce a set of migrations that wrap up this process by making sure
-  that all remaining jobs over the `merge_request_diff_commits` table are completed.
-  These jobs have already been processed in most cases so that no extra time is necessary during an upgrade to 14.5.
-  However, if there are remaining jobs or you haven't already upgraded to 14.1,
-  the deployment may take multiple hours to complete.
-
-  All merge request diff commits automatically incorporate these changes, and there are no
-  additional requirements to perform the upgrade.
-  Existing data in the `merge_request_diff_commits` table remains unpacked until you run `VACUUM FULL merge_request_diff_commits`.
-  However, the `VACUUM FULL` operation locks and rewrites the entire `merge_request_diff_commits` table,
-  so the operation takes some time to complete and it blocks access to this table until the end of the process.
-  We advise you to only run this command while GitLab is not actively used or it is taken offline for the duration of the process.
-  The time it takes to complete depends on the size of the table, which can be obtained by using `select pg_size_pretty(pg_total_relation_size('merge_request_diff_commits'));`.
-
-  For more information, refer to [this issue](https://gitlab.com/gitlab-org/gitlab/-/issues/331823).
-
-- GitLab 14.5.0 includes a
-  [background migration `UpdateVulnerabilityOccurrencesLocation`](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/72788)
-  that may remain stuck permanently in a **pending** state when the instance lacks records that match the migration's target.
-
-  To clean up this stuck job, run the following in the [GitLab Rails Console](../administration/operations/rails_console.md):
-
-  ```ruby
-  Gitlab::Database::BackgroundMigrationJob.pending.where(class_name: "UpdateVulnerabilityOccurrencesLocation").find_each do |job|
-    puts Gitlab::Database::BackgroundMigrationJob.mark_all_as_succeeded("UpdateVulnerabilityOccurrencesLocation", job.arguments)
-  end
-  ```
-
-- Upgrading to 14.5 (or later) [might encounter a one hour timeout](https://gitlab.com/gitlab-org/gitlab/-/issues/354211)
-  owing to a long running database data change.
-
-  ```plaintext
-  FATAL: Mixlib::ShellOut::CommandTimeout: rails_migration[gitlab-rails]
-  (gitlab::database_migrations line 51) had an error:
-  [..]
-  Mixlib::ShellOut::CommandTimeout: Command timed out after 3600s:
-  ```
-
-  [There is a workaround to complete the data change and the upgrade manually](package/index.md#mixlibshelloutcommandtimeout-rails_migrationgitlab-rails--command-timed-out-after-3600s)
-
-- As part of [enabling real-time issue assignees](https://gitlab.com/gitlab-org/gitlab/-/issues/330117), Action Cable is now enabled by default.
-  For **self-compiled (source) installations**, `config/cable.yml` is required to be present.
-
-  Configure this by running:
-
-  ```shell
-  cd /home/git/gitlab
-  sudo -u git -H cp config/cable.yml.example config/cable.yml
-
-  # Change the Redis socket path if you are not using the default Debian / Ubuntu configuration
-  sudo -u git -H editor config/cable.yml
-  ```
-
-### 14.4.4
-
-- For [zero-downtime upgrades](zero_downtime.md) on a GitLab cluster with separate Web and API nodes, you must enable the `paginated_tree_graphql_query` [feature flag](../administration/feature_flags.md#enable-or-disable-the-feature) _before_ upgrading GitLab Web nodes to 14.4.
-  This is because we [enabled `paginated_tree_graphql_query` by default in 14.4](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/70913/diffs), so if GitLab UI is on 14.4 and its API is on 14.3, the frontend has this feature enabled but the backend has it disabled. This results in the following error:
-
-  ```shell
-  bundle.esm.js:63 Uncaught (in promise) Error: GraphQL error: Field 'paginatedTree' doesn't exist on type 'Repository'
-  ```
-
-### 14.4.0
-
-- Git 2.33.x and later is required. We recommend you use the
-  [Git version provided by Gitaly](../install/installation.md#git).
-- See [Maintenance mode issue in GitLab 13.9 to 14.4](#maintenance-mode-issue-in-gitlab-139-to-144).
-- After enabling database load balancing by default in 14.4.0, we found an issue where
-  [cron jobs would not work if the connection to PostgreSQL was severed](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/73716),
-  as Sidekiq would continue using a bad connection. Geo and other features that rely on
-  cron jobs running regularly do not work until Sidekiq is restarted. We recommend
-  upgrading to GitLab 14.4.3 and later if this issue affects you.
-- After enabling database load balancing by default in 14.4.0, we found an issue where
-  [Database load balancing does not work with an AWS Aurora cluster](https://gitlab.com/gitlab-org/gitlab/-/issues/220617).
-  We recommend moving your databases from Aurora to RDS for PostgreSQL before
-  upgrading. Refer to [Moving GitLab databases to a different PostgreSQL instance](../administration/postgresql/moving.md).
-- GitLab 14.4.0 includes a
-[background migration `PopulateTopicsTotalProjectsCountCache`](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/71033)
-that may remain stuck permanently in a **pending** state when the instance lacks records that match the migration's target.
-
-  To clean up this stuck job, run the following in the [GitLab Rails Console](../administration/operations/rails_console.md):
-
-  ```ruby
-  Gitlab::Database::BackgroundMigrationJob.pending.where(class_name: "PopulateTopicsTotalProjectsCountCache").find_each do |job|
-    puts Gitlab::Database::BackgroundMigrationJob.mark_all_as_succeeded("PopulateTopicsTotalProjectsCountCache", job.arguments)
-  end
-  ```
-
-### 14.3.0
-
-- [Instances running 14.0.0 - 14.0.4 should not upgrade directly to GitLab 14.2 or later](#upgrading-to-later-14y-releases).
-- Ensure [batched background migrations finish](background_migrations.md#batched-background-migrations) before upgrading
-  to 14.3.Z from earlier GitLab 14 releases.
-- Ruby 2.7.4 is required. Refer to [the Ruby installation instructions](../install/installation.md#2-ruby)
-for how to proceed.
-- GitLab 14.3.0 contains post-deployment migrations to [address Primary Key overflow risk for tables with an integer PK](https://gitlab.com/groups/gitlab-org/-/epics/4785) for the tables listed below:
-
-  - [`ci_builds.id`](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/70245)
-  - [`ci_builds.stage_id`](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/66688)
-  - [`ci_builds_metadata`](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/65692)
-  - [`taggings`](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/66625)
-  - [`events`](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/64779)
-
-  If the migrations are executed as part of a no-downtime deployment, there's a risk of failure due to lock conflicts with the application logic, resulting in lock timeout or deadlocks. In each case, these migrations are safe to re-run until successful:
-
-  ```shell
-  # For Omnibus GitLab
-  sudo gitlab-rake db:migrate
-
-  # For source installations
-  sudo -u git -H bundle exec rake db:migrate RAILS_ENV=production
-  ```
-
-- After upgrading to 14.3, ensure that all the `MigrateMergeRequestDiffCommitUsers` background
-  migration jobs have completed before continuing with upgrading to GitLab 14.5 or later.
-  This is especially important if your GitLab instance has a large
-  `merge_request_diff_commits` table. Any pending
-  `MigrateMergeRequestDiffCommitUsers` background migration jobs are
-  foregrounded in GitLab 14.5, and may take a long time to complete.
-  You can check the count of pending jobs for
-  `MigrateMergeRequestDiffCommitUsers` by using the PostgreSQL console (or `sudo
-  gitlab-psql`):
-
-  ```sql
-  select status, count(*) from background_migration_jobs
-  where class_name = 'MigrateMergeRequestDiffCommitUsers' group by status;
-  ```
-
-  As jobs are completed, the database records change from `0` (pending) to `1`. If the number of
-  pending jobs doesn't decrease after a while, it's possible that the
-  `MigrateMergeRequestDiffCommitUsers` background migration jobs have failed. You
-  can check for errors in the Sidekiq logs:
-
-  ```shell
-  sudo grep MigrateMergeRequestDiffCommitUsers /var/log/gitlab/sidekiq/current | grep -i error
-  ```
-
-  If needed, you can attempt to run the `MigrateMergeRequestDiffCommitUsers` background
-  migration jobs manually in the [GitLab Rails Console](../administration/operations/rails_console.md).
-  This can be done using Sidekiq asynchronously, or by using a Rails process directly:
-
-  - Using Sidekiq to schedule jobs asynchronously:
-
-    ```ruby
-    # For the first run, only attempt to execute 1 migration. If successful, increase
-    # the limit for subsequent runs
-    limit = 1
-
-    jobs = Gitlab::Database::BackgroundMigrationJob.for_migration_class('MigrateMergeRequestDiffCommitUsers').pending.to_a
-
-    pp "#{jobs.length} jobs remaining"
-
-    jobs.first(limit).each do |job|
-      BackgroundMigrationWorker.perform_in(5.minutes, 'MigrateMergeRequestDiffCommitUsers', job.arguments)
-    end
-    ```
-
-    NOTE:
-    The queued jobs can be monitored using the Sidekiq admin panel, which can be accessed at the `/admin/sidekiq` endpoint URI.
-
-  - Using a Rails process to run jobs synchronously:
-
-    ```ruby
-    def process(concurrency: 1)
-      queue = Queue.new
-
-      Gitlab::Database::BackgroundMigrationJob
-        .where(class_name: 'MigrateMergeRequestDiffCommitUsers', status: 0)
-        .each { |job| queue << job }
-
-      concurrency
-        .times
-        .map do
-          Thread.new do
-            Thread.abort_on_exception = true
-
-            loop do
-              job = queue.pop(true)
-              time = Benchmark.measure do
-                Gitlab::BackgroundMigration::MigrateMergeRequestDiffCommitUsers
-                  .new
-                  .perform(*job.arguments)
-              end
-
-              puts "#{job.id} finished in #{time.real.round(2)} seconds"
-            rescue ThreadError
-              break
-            end
-          end
-        end
-        .each(&:join)
-    end
-
-    ActiveRecord::Base.logger.level = Logger::ERROR
-    process
-    ```
-
-    NOTE:
-    When using Rails to execute these background migrations synchronously, make sure that the machine running the process has sufficient resources to handle the task. If the process gets terminated, it's likely due to insufficient memory available. If your SSH session times out after a while, it might be necessary to run the previous code by using a terminal multiplexer like `screen` or `tmux`.
-
-- See [Maintenance mode issue in GitLab 13.9 to 14.4](#maintenance-mode-issue-in-gitlab-139-to-144).
-
-- You may see the following error when setting up two factor authentication (2FA) for accounts
-  that authenticate using an LDAP password:
-
-  ```plaintext
-  You must provide a valid current password
-  ```
-
-  - The error occurs because verification is incorrectly performed against accounts'
-    randomly generated internal GitLab passwords, not the LDAP passwords.
-  - This is [fixed in GitLab 14.5.0 and backported to 14.4.3](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/73538).
-  - Workarounds:
-    - Instead of upgrading to GitLab 14.3.x to comply with the supported upgrade path:
-      1. Upgrade to 14.4.5.
-      1. Make sure the [`MigrateMergeRequestDiffCommitUsers` background migration](#1430) has finished.
-      1. Upgrade to GitLab 14.5 or later.
-    - Reset the random password for affected accounts, using [the Rake task](../security/reset_user_password.md#use-a-rake-task):
-
-      ```plaintext
-      sudo gitlab-rake "gitlab:password:reset[user_handle]"
-      ```
-
-- If you encounter the error, `I18n::InvalidLocale: :en is not a valid locale`, when starting the application, follow the [patching](https://about.gitlab.com/handbook/support/workflows/patching_an_instance.html) process. Use [122978](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/122978) as the `mr_iid`.
-
-### 14.2.0
-
-- [Instances running 14.0.0 - 14.0.4 should not upgrade directly to GitLab 14.2 or later](#upgrading-to-later-14y-releases).
-- Ensure [batched background migrations finish](background_migrations.md#batched-background-migrations) before upgrading
-  to 14.2.Z from earlier GitLab 14 releases.
-- GitLab 14.2.0 contains background migrations to [address Primary Key overflow risk for tables with an integer PK](https://gitlab.com/groups/gitlab-org/-/epics/4785) for the tables listed below:
-  - [`ci_build_needs`](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/65216)
-  - [`ci_build_trace_chunks`](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/66123)
-  - [`ci_builds_runner_session`](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/66433)
-  - [`deployments`](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/67341)
-  - [`geo_job_artifact_deleted_events`](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/66763)
-  - [`push_event_payloads`](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/67299)
-  - `ci_job_artifacts`:
-    - [Finalize `job_id` conversion to `bigint` for `ci_job_artifacts`](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/67774)
-    - [Finalize `ci_job_artifacts` conversion to `bigint`](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/65601)
-
-  If the migrations are executed as part of a no-downtime deployment, there's a risk of failure due to lock conflicts with the application logic, resulting in lock timeout or deadlocks. In each case, these migrations are safe to re-run until successful:
-
-  ```shell
-  # For Omnibus GitLab
-  sudo gitlab-rake db:migrate
-
-  # For source installations
-  sudo -u git -H bundle exec rake db:migrate RAILS_ENV=production
-  ```
-
-- See [Maintenance mode issue in GitLab 13.9 to 14.4](#maintenance-mode-issue-in-gitlab-139-to-144).
-- GitLab 14.2.0 includes a
-  [background migration `BackfillDraftStatusOnMergeRequests`](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/67687)
-  that may remain stuck permanently in a **pending** state when the instance lacks records that match the migration's target.
-
-  To clean up this stuck job, run the following in the [GitLab Rails Console](../administration/operations/rails_console.md):
-
-  ```ruby
-  Gitlab::Database::BackgroundMigrationJob.pending.where(class_name: "BackfillDraftStatusOnMergeRequests").find_each do |job|
-    puts Gitlab::Database::BackgroundMigrationJob.mark_all_as_succeeded("BackfillDraftStatusOnMergeRequests", job.arguments)
-  end
-  ```
-
-- If you encounter the error, `I18n::InvalidLocale: :en is not a valid locale`, when starting the application, follow the [patching](https://about.gitlab.com/handbook/support/workflows/patching_an_instance.html) process. Use [123476](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/123476) as the `mr_iid`.
-
-### 14.1.0
-
-- [Instances running 14.0.0 - 14.0.4 should not upgrade directly to GitLab 14.2 or later](#upgrading-to-later-14y-releases)
-  but can upgrade to 14.1.Z.
-
-  It is not required for instances already running 14.0.5 (or later) to stop at 14.1.Z.
-  14.1 is included on the upgrade path for the broadest compatibility
-  with self-managed installations, and ensure 14.0.0-14.0.4 installations do not
-  encounter issues with [batched background migrations](background_migrations.md#batched-background-migrations).
-
-- Upgrading to GitLab [14.5](#1450) (or later) may take a lot longer if you do not upgrade to at least 14.1
-  first. The 14.1 merge request diff commits database migration can take hours to run, but runs in the
-  background while GitLab is in use. GitLab instances upgraded directly from 14.0 to 14.5 or later must
-  run the migration in the foreground and therefore take a lot longer to complete.
-
-- See [Maintenance mode issue in GitLab 13.9 to 14.4](#maintenance-mode-issue-in-gitlab-139-to-144).
-
-- If you encounter the error, `I18n::InvalidLocale: :en is not a valid locale`, when starting the application, follow the [patching](https://about.gitlab.com/handbook/support/workflows/patching_an_instance.html) process. Use [123475](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/123475) as the `mr_iid`.
-
-### 14.0.0
-
-Prerequisites:
-
-- The [GitLab 14.0 release post contains several important notes](https://about.gitlab.com/releases/2021/06/22/gitlab-14-0-released/#upgrade)
-  about pre-requisites including [using Patroni instead of repmgr](../administration/postgresql/replication_and_failover.md#switching-from-repmgr-to-patroni),
-  migrating [to hashed storage](../administration/raketasks/storage.md#migrate-to-hashed-storage),
-  and [to Puma](../administration/operations/puma.md).
-- The support of PostgreSQL 11 [has been dropped](../install/requirements.md#database). Make sure to [update your database](https://docs.gitlab.com/omnibus/settings/database.html#upgrade-packaged-postgresql-server) to version 12 before updating to GitLab 14.0.
-
-Long running batched background database migrations:
-
-- Database changes made by the upgrade to GitLab 14.0 can take hours or days to complete on larger GitLab instances.
-  These [batched background migrations](background_migrations.md#batched-background-migrations) update whole database tables to mitigate primary key overflow and must be finished before upgrading to GitLab 14.2 or later.
-- Due to an issue where `BatchedBackgroundMigrationWorkers` were
-  [not working](https://gitlab.com/gitlab-org/charts/gitlab/-/issues/2785#note_614738345)
-  for self-managed instances, a [fix was created](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/65106)
-  that requires an update to at least 14.0.5. The fix was also released in [14.1.0](#1410).
-
-  After you update to 14.0.5 or a later 14.0 patch version,
-  [batched background migrations must finish](background_migrations.md#batched-background-migrations)
-  before you upgrade to a later version.
-
-  If the migrations are not finished and you try to upgrade to a later version,
-  you see an error like:
-
-  ```plaintext
-  Expected batched background migration for the given configuration to be marked as 'finished', but it is 'active':
-  ```
-
-  See how to [resolve this error](background_migrations.md#database-migrations-failing-because-of-batched-background-migration-not-finished).
-
-Other issues:
-
-- In GitLab 13.3 some [pipeline processing methods were deprecated](https://gitlab.com/gitlab-org/gitlab/-/issues/218536)
-  and this code was completely removed in GitLab 14.0. If you plan to upgrade from
-  **GitLab 13.2 or older** directly to 14.0, this is [unsupported](#upgrading-to-a-new-major-version).
-  You should instead follow a [supported upgrade path](#upgrade-paths).
-- See [Maintenance mode issue in GitLab 13.9 to 14.4](#maintenance-mode-issue-in-gitlab-139-to-144).
-
-#### Upgrading to later 14.Y releases
-
-- Instances running 14.0.0 - 14.0.4 should not upgrade directly to GitLab 14.2 or later,
-  because of [batched background migrations](background_migrations.md#batched-background-migrations).
-  1. Upgrade first to either:
-     - 14.0.5 or a later 14.0.Z patch release.
-     - 14.1.0 or a later 14.1.Z patch release.
-  1. [Batched background migrations must finish](background_migrations.md#batched-background-migrations)
-     before you upgrade to a later version [and may take longer than usual](#1400).
-
-### User profile data loss bug in 15.9.x
-
-There is a database migration bug in 15.9.0, 15.9.1, and 15.9.2 that can cause data loss from the user profile fields `linkedin`, `twitter`, `skype`, `website_url`, `location`, and `organization`.
-
-This bug is fixed in patch releases 15.9.3 and later.
-
-The following upgrade path also works around the bug:
-
-1. Upgrade to GitLab 15.6.x, 15.7.x, or 15.8.x.
-1. [Ensure batched background migrations](background_migrations.md#batched-background-migrations) are complete.
-1. Upgrade to an earlier GitLab 15.9 patch release that doesn't have the bug fix.
-
-It is not then required to upgrade to 15.9.3 or later for this issue.
-
-[Read the issue](https://gitlab.com/gitlab-org/gitlab/-/issues/393216) for more information.
-
-### Gitaly: Omnibus GitLab configuration structure change
-
-Gitaly configuration structure in Omnibus GitLab [changes](https://gitlab.com/gitlab-org/gitaly/-/issues/4467) in GitLab 16.0 to be consistent with the Gitaly configuration
-structure used in self-compiled installations.
-
-As a result of this change, a single hash under `gitaly['configuration']` holds most Gitaly
-configuration. Some `gitaly['..']` configuration options will continue to be used by Omnibus GitLab 16.0 and later:
-
-- `enable`
-- `dir`
-- `bin_path`
-- `env_directory`
-- `env`
-- `open_files_ulimit`
-- `consul_service_name`
-- `consul_service_meta`
-
-Migrate by moving your existing configuration under the new structure. The new structure is supported from Omnibus GitLab 15.10.
-
-The new structure is documented below with the old keys described in a comment above the new keys. When applying the new structure to your configuration:
-
-1. Replace the `...` with the value from the old key.
-1. Skip any keys you haven't configured a value for previously.
-1. Remove the old keys from the configuration once migrated.
-1. Optional but recommended. Include a trailing comma for all hash keys so the hash remains valid when keys are re-ordered or additional keys are added.
-1. When configuring `storage` to replace `git_data_dirs`, you must append `repositories` to the path as documented below. If you omit this step, your Git repositories are
-   inaccessible until the configuration is fixed.
-
-  ```ruby
-gitaly['configuration'] = {
-  # gitaly['socket_path']
-  socket_path: ...,
-  # gitaly['runtime_dir']
-  runtime_dir: ...,
-  # gitaly['listen_addr']
-  listen_addr: ...,
-  # gitaly['prometheus_listen_addr']
-  prometheus_listen_addr: ...,
-  # gitaly['tls_listen_addr']
-  tls_listen_addr: ...,
-  tls: {
-    # gitaly['certificate_path']
-    certificate_path: ...,
-    # gitaly['key_path']
-    key_path: ...,
-  },
-  # gitaly['graceful_restart_timeout']
-  graceful_restart_timeout: ...,
-  logging: {
-    # gitaly['logging_level']
-    level: ...,
-    # gitaly['logging_format']
-    format: ...,
-    # gitaly['logging_sentry_dsn']
-    sentry_dsn: ...,
-    # gitaly['logging_ruby_sentry_dsn']
-    ruby_sentry_dsn: ...,
-    # gitaly['logging_sentry_environment']
-    sentry_environment: ...,
-    # gitaly['log_directory']
-    dir: ...,
-  },
-  prometheus: {
-    # gitaly['prometheus_grpc_latency_buckets']. The old value was configured as a string
-    # such as '[0, 1, 2]'. The new value must be an array like [0, 1, 2].
-    grpc_latency_buckets: ...,
-  },
-  auth: {
-    # gitaly['auth_token']
-    token: ...,
-    # gitaly['auth_transitioning']
-    transitioning: ...,
-  },
-  git: {
-    # gitaly['git_catfile_cache_size']
-    catfile_cache_size: ...,
-    # gitaly['git_bin_path']
-    bin_path: ...,
-    # gitaly['use_bundled_git']
-    use_bundled_binaries: ...,
-    # gitaly['gpg_signing_key_path']
-    signing_key: ...,
-    # gitaly['gitconfig']. This is still an array but the type of the elements have changed.
-    config: [
-      {
-        # Previously the elements contained 'section', and 'subsection' in addition to 'key'. Now
-        # these all should be concatenated into just 'key', separated by dots. For example,
-        # {section: 'first', subsection: 'middle', key: 'last', value: 'value'}, should become
-        # {key: 'first.middle.last', value: 'value'}.
-        key: ...,
-        value: ...,
-      },
-    ],
-  },
-  # Storage could previously be configured through either gitaly['storage'] or 'git_data_dirs'. Migrate
-  # the relevant configuration according to the instructions below.
-  # For 'git_data_dirs', migrate only the 'path' to the gitaly['configuration'] and leave the rest of it untouched.
-  storage: [
-    {
-      # gitaly['storage'][<index>]['name']
-      #
-      # git_data_dirs[<name>]. The storage name was configured as a key in the map.
-      name: ...,
-      # gitaly['storage'][<index>]['path']
-      #
-      # git_data_dirs[<name>]['path']. Use the value from git_data_dirs[<name>]['path'] and append '/repositories' to it.
-      #
-      # For example, if the path in 'git_data_dirs' was '/var/opt/gitlab/git-data', use
-      # '/var/opt/gitlab/git-data/repositories'. The '/repositories' extension was automatically
-      # appended to the path configured in `git_data_dirs`.
-      path: ...,
-    },
-  ],
-  hooks: {
-    # gitaly['custom_hooks_dir']
-    custom_hooks_dir: ...,
-  },
-  daily_maintenance: {
-    # gitaly['daily_maintenance_disabled']
-    disabled: ...,
-    # gitaly['daily_maintenance_start_hour']
-    start_hour: ...,
-    # gitaly['daily_maintenance_start_minute']
-    start_minute: ...,
-    # gitaly['daily_maintenance_duration']
-    duration: ...,
-    # gitaly['daily_maintenance_storages']
-    storages: ...,
-  },
-  cgroups: {
-    # gitaly['cgroups_mountpoint']
-    mountpoint: ...,
-    # gitaly['cgroups_hierarchy_root']
-    hierarchy_root: ...,
-    # gitaly['cgroups_memory_bytes']
-    memory_bytes: ...,
-    # gitaly['cgroups_cpu_shares']
-    cpu_shares: ...,
-    repositories: {
-      # gitaly['cgroups_repositories_count']
-      count: ...,
-      # gitaly['cgroups_repositories_memory_bytes']
-      memory_bytes: ...,
-      # gitaly['cgroups_repositories_cpu_shares']
-      cpu_shares: ...,
-    }
-  },
-  # gitaly['concurrency']. While the structure is the same, the string keys in the array elements
-  # should be replaced by symbols as elsewhere. {'key' => 'value'}, should become {key: 'value'}.
-  concurrency: ...,
-  # gitaly['rate_limiting']. While the structure is the same, the string keys in the array elements
-  # should be replaced by symbols as elsewhere. {'key' => 'value'}, should become {key: 'value'}.
-  rate_limiting: ...,
-  pack_objects_cache: {
-    # gitaly['pack_objects_cache_enabled']
-    enabled: ...,
-    # gitaly['pack_objects_cache_dir']
-    dir: ...,
-    # gitaly['pack_objects_cache_max_age']
-    max_age: ...,
-  }
-}
-```
-
-### Praefect: Omnibus GitLab configuration structure change
-
-Praefect configuration structure in Omnibus GitLab [changes](https://gitlab.com/gitlab-org/gitaly/-/issues/4467) in GitLab 16.0 to be consistent with the Praefect configuration
-structure used in self-compiled installations.
-
-As a result of this change, a single hash under `praefect['configuration']` holds most Praefect
-configuration. Some `praefect['..']` configuration options will continue to be used by Omnibus GitLab 16.0 and later:
-
-- `enable`
-- `dir`
-- `log_directory`
-- `env_directory`
-- `env`
-- `wrapper_path`
-- `auto_migrate`
-- `consul_service_name`
-
-Migrate by moving your existing configuration under the new structure. The new structure is supported from Omnibus GitLab 15.9.
-
-The new structure is documented below with the old keys described in a comment above the new keys. When applying the new structure to your configuration:
-
-1. Replace the `...` with the value from the old key.
-1. Skip any keys you haven't configured a value for previously.
-1. Remove the old keys from the configuration once migrated.
-1. Optional but recommended. Include a trailing comma for all hash keys so the hash remains valid when keys are re-ordered or additional keys are added.
-
-```ruby
-praefect['configuration'] = {
-  # praefect['listen_addr']
-  listen_addr: ...,
-  # praefect['socket_path']
-  socket_path: ...,
-  # praefect['prometheus_listen_addr']
-  prometheus_listen_addr: ...,
-  # praefect['tls_listen_addr']
-  tls_listen_addr: ...,
-  # praefect['separate_database_metrics']
-  prometheus_exclude_database_from_default_metrics: ...,
-  auth: {
-    # praefect['auth_token']
-    token: ...,
-    # praefect['auth_transitioning']
-    transitioning: ...,
-  },
-  logging: {
-    # praefect['logging_format']
-    format: ...,
-    # praefect['logging_level']
-    level: ...,
-  },
-  failover: {
-    # praefect['failover_enabled']
-    enabled: ...,
-  },
-  background_verification: {
-    # praefect['background_verification_delete_invalid_records']
-    delete_invalid_records: ...,
-    # praefect['background_verification_verification_interval']
-    verification_interval: ...,
-  },
-  reconciliation: {
-    # praefect['reconciliation_scheduling_interval']
-    scheduling_interval: ...,
-    # praefect['reconciliation_histogram_buckets']. The old value was configured as a string
-    # such as '[0, 1, 2]'. The new value must be an array like [0, 1, 2].
-    histogram_buckets: ...,
-  },
-  tls: {
-    # praefect['certificate_path']
-    certificate_path: ...,
-    # praefect['key_path']
-    key_path: ...,
-  },
-  database: {
-    # praefect['database_host']
-    host: ...,
-    # praefect['database_port']
-    port: ...,
-    # praefect['database_user']
-    user: ...,
-    # praefect['database_password']
-    password: ...,
-    # praefect['database_dbname']
-    dbname: ...,
-    # praefect['database_sslmode']
-    sslmode: ...,
-    # praefect['database_sslcert']
-    sslcert: ...,
-    # praefect['database_sslkey']
-    sslkey: ...,
-    # praefect['database_sslrootcert']
-    sslrootcert: ...,
-    session_pooled: {
-      # praefect['database_direct_host']
-      host: ...,
-      # praefect['database_direct_port']
-      port: ...,
-      # praefect['database_direct_user']
-      user: ...,
-      # praefect['database_direct_password']
-      password: ...,
-      # praefect['database_direct_dbname']
-      dbname: ...,
-      # praefect['database_direct_sslmode']
-      sslmode: ...,
-      # praefect['database_direct_sslcert']
-      sslcert: ...,
-      # praefect['database_direct_sslkey']
-      sslkey: ...,
-      # praefect['database_direct_sslrootcert']
-      sslrootcert: ...,
-    }
-  },
-  sentry: {
-    # praefect['sentry_dsn']
-    sentry_dsn: ...,
-    # praefect['sentry_environment']
-    sentry_environment: ...,
-  },
-  prometheus: {
-    # praefect['prometheus_grpc_latency_buckets']. The old value was configured as a string
-    # such as '[0, 1, 2]'. The new value must be an array like [0, 1, 2].
-    grpc_latency_buckets: ...,
-  },
-  # praefect['graceful_stop_timeout']
-  graceful_stop_timeout: ...,
-
-  # praefect['virtual_storages']. The old value was a hash map but the new value is an array.
-  virtual_storage: [
-    {
-      # praefect['virtual_storages'][VIRTUAL_STORAGE_NAME]. The name was previously the key in
-      # the 'virtual_storages' hash.
-      name: ...,
-      # praefect['virtual_storages'][VIRTUAL_STORAGE_NAME]['nodes'][NODE_NAME]. The old value was a hash map
-      # but the new value is an array.
-      node: [
-        {
-          # praefect['virtual_storages'][VIRTUAL_STORAGE_NAME]['nodes'][NODE_NAME]. Use NODE_NAME key as the
-          # storage.
-          storage: ...,
-          # praefect['virtual_storages'][VIRTUAL_STORAGE_NAME]['nodes'][NODE_NAME]['address'].
-          address: ...,
-          # praefect['virtual_storages'][VIRTUAL_STORAGE_NAME]['nodes'][NODE_NAME]['token'].
-          token: ...,
-        },
-      ],
-    }
-  ]
-}
-```
-
-### Change to Praefect-generated replica paths in GitLab 15.3
-
-New Git repositories created in Gitaly cluster no longer use the `@hashed` storage path.
-
-Praefect now generates replica paths for use by Gitaly cluster.
-This change is a pre-requisite for Gitaly cluster atomically creating, deleting, and
-renaming Git repositories.
-
-To identify the replica path, [query the Praefect repository metadata](../administration/gitaly/troubleshooting.md#view-repository-metadata)
-and pass the `@hashed` storage path to `-relative-path`.
-
-With this information, you can correctly install [server hooks](../administration/server_hooks.md).
-
-### Geo: LFS transfers redirect to primary from secondary site mid-session in GitLab 15.1.0 to 15.3.2
-
-LFS transfers can [redirect to the primary from secondary site mid-session](https://gitlab.com/gitlab-org/gitlab/-/issues/371571) causing failed pull and clone requests in GitLab 15.1.0 to 15.3.2 when [Geo proxying](../administration/geo/secondary_proxy/index.md) is enabled. Geo proxying is enabled by default in GitLab 15.1 and later.
-
-This issue is resolved in GitLab 15.3.3, so customers with the following configuration should upgrade to 15.3.3 or later:
-
-- LFS is enabled.
-- LFS objects are being replicated across Geo sites.
-- Repositories are being pulled by using a Geo secondary site.
-
-### Geo: Incorrect object storage LFS file deletion on secondary sites in GitLab 15.0.0 to 15.3.2
-
-[Incorrect deletion of object storage files on Geo secondary sites](https://gitlab.com/gitlab-org/gitlab/-/issues/371397)
-can occur in GitLab 15.0.0 to 15.3.2 in the following situations:
-
-- GitLab-managed object storage replication is disabled, and LFS objects are created while importing a project with object storage enabled.
-- GitLab-managed replication to sync object storage is enabled and subsequently disabled.
-
-This issue is resolved in 15.3.3. Customers who have both LFS enabled and LFS objects being replicated across Geo sites
-should upgrade directly to 15.3.3 to reduce the risk of data loss on secondary sites.
-
-### PostgreSQL segmentation fault issue
-
-If you run GitLab with external PostgreSQL, particularly AWS RDS, ensure you upgrade PostgreSQL
-to patch levels to a minimum of 12.7 or 13.3 before upgrading to GitLab 14.8 or later.
-
-[In 14.8](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/75511)
-for GitLab Enterprise Edition and [in 15.1](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/87983)
-for GitLab Community Edition a GitLab feature called Loose Foreign Keys was enabled.
-
-After it was enabled, we have had reports of unplanned PostgreSQL restarts caused
-by a database engine bug that causes a segmentation fault.
-
-Read more [in the issue](https://gitlab.com/gitlab-org/gitlab/-/issues/364763).
-
-### LFS objects import and mirror issue in GitLab 14.6.0 to 14.7.2
-
-When Geo is enabled, LFS objects fail to be saved for imported or mirrored projects.
-
-[This bug](https://gitlab.com/gitlab-org/gitlab/-/issues/352368) was fixed in GitLab 14.8.0 and backported into 14.7.3.
+Before upgrading to GitLab 16, see [GitLab 16 changes](versions/gitlab_16_changes.md).
 
-### Maintenance mode issue in GitLab 13.9 to 14.4
+### GitLab 15
 
-When [Maintenance mode](../administration/maintenance_mode/index.md) is enabled, users cannot sign in with SSO, SAML, or LDAP.
+Before upgrading to GitLab 15, see [GitLab 15 changes](versions/gitlab_15_changes.md).
 
-Users who were signed in before Maintenance mode was enabled, continue to be signed in. If the administrator who enabled Maintenance mode loses their session, then they can't disable Maintenance mode via the UI. In that case, you can [disable Maintenance mode via the API or Rails console](../administration/maintenance_mode/index.md#disable-maintenance-mode).
+### GitLab 14
 
-[This bug](https://gitlab.com/gitlab-org/gitlab/-/issues/329261) was fixed in GitLab 14.5.0 and backported into 14.4.3 and 14.3.5.
+Before upgrading to GitLab 14, see [GitLab 14 changes](versions/gitlab_14_changes.md).
 
 ## Miscellaneous
 
diff --git a/doc/update/package/index.md b/doc/update/package/index.md
index 148791dbc75..19d308bce6b 100644
--- a/doc/update/package/index.md
+++ b/doc/update/package/index.md
@@ -43,6 +43,14 @@ check the version your are upgrading to:
 - [GitLab 15](https://docs.gitlab.com/omnibus/update/gitlab_15_changes.html)
 - [GitLab 14](https://docs.gitlab.com/omnibus/update/gitlab_14_changes.html)
 
+### Earlier GitLab versions
+
+For version-specific information for earlier GitLab versions, see the [documentation archives](https://archives.docs.gitlab.com).
+The versions of the documentation in the archives contain version-specific information for even earlier versions of GitLab.
+
+For example, the [documentation for GitLab 15.11](https://archives.docs.gitlab.com/15.11/ee/update/package/#version-specific-changes)
+contains information on versions back to GitLab 11.
+
 ## Back up before upgrading
 
 The GitLab database is backed up before installing a newer GitLab version. You
diff --git a/doc/update/plan_your_upgrade.md b/doc/update/plan_your_upgrade.md
index 0bb760ec47e..25578dbef59 100644
--- a/doc/update/plan_your_upgrade.md
+++ b/doc/update/plan_your_upgrade.md
@@ -103,11 +103,7 @@ For the upgrade plan, start by creating an outline of a plan that best applies
 to your instance and then upgrade it for any relevant features you're using.
 
 - Generate an upgrade plan by reading and understanding the relevant documentation:
-  - upgrade based on the installation method:
-    - [Linux package (Omnibus)](index.md#linux-packages-omnibus)
-    - [Self-compiled](index.md#self-compiled-installation)
-    - [Docker](index.md#installation-using-docker)
-    - [Helm Charts](index.md#installation-using-helm)
+  - Upgrade based on the [installation method](index.md#upgrade-based-on-installation-method).
   - [Zero-downtime upgrades](zero_downtime.md) (if possible and desired)
   - [Convert from GitLab Community Edition to Enterprise Edition](package/convert_to_ee.md)
 - What version should you upgrade to:
@@ -122,7 +118,7 @@ to your instance and then upgrade it for any relevant features you're using.
   [turning on maintenance mode](../administration/maintenance_mode/index.md) during the
   upgrade.
 - About PostgreSQL:
-  1. On the left sidebar, expand the top-most chevron (**{chevron-down}**).
+  1. On the left sidebar, select **Search or go to**.
   1. Select **Admin Area**.
   1. Look for the version of PostgreSQL you are using.
     If [a PostgreSQL upgrade is needed](../administration/package_information/postgresql_versions.md),
diff --git a/doc/update/versions/gitlab_14_changes.md b/doc/update/versions/gitlab_14_changes.md
new file mode 100644
index 00000000000..700baf4ef09
--- /dev/null
+++ b/doc/update/versions/gitlab_14_changes.md
@@ -0,0 +1,909 @@
+---
+stage: Systems
+group: Distribution
+info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments
+---
+
+# GitLab 14 changes **(FREE SELF)**
+
+This page contains upgrade information for minor and patch versions of GitLab 14.
+Ensure you review these instructions for:
+
+- Your installation type.
+- All versions between your current version and your target version.
+
+For more information about upgrading GitLab Helm Chart, see [the release notes for 5.0](https://docs.gitlab.com/charts/releases/5_0.html).
+
+## 14.10.0
+
+- Before upgrading to GitLab 14.10, you must already have the latest 14.9.Z installed on your instance.
+  The upgrade to GitLab 14.10 executes a [concurrent index drop](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/84308) of unneeded
+  entries from the `ci_job_artifacts` database table. This could potentially run for multiple minutes, especially if the table has a lot of
+  traffic and the migration is unable to acquire a lock. It is advised to let this process finish as restarting may result in data loss.
+
+- If you run GitLab with external PostgreSQL, particularly AWS RDS, ensure you
+  upgrade PostgreSQL to patch levels to a minimum of 12.7 or 13.3 before
+  upgrading to GitLab 14.8 or later.
+
+  [In 14.8](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/75511)
+  for GitLab Enterprise Edition and [in 15.0](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/87983)
+  for GitLab Community Edition a GitLab feature called Loose Foreign Keys was enabled.
+
+  After it was enabled, we have had reports of unplanned PostgreSQL restarts caused
+  by a database engine bug that causes a segmentation fault.
+
+  For more information, see [issue 364763](https://gitlab.com/gitlab-org/gitlab/-/issues/364763).
+
+- Upgrading to patch level 14.10.3 or later might encounter a one-hour timeout due to a long running database data change,
+  if it was not completed while running GitLab 14.9.
+
+  ```plaintext
+  FATAL: Mixlib::ShellOut::CommandTimeout: rails_migration[gitlab-rails]
+  (gitlab::database_migrations line 51) had an error:
+  [..]
+  Mixlib::ShellOut::CommandTimeout: Command timed out after 3600s:
+  ```
+
+  A workaround exists to [complete the data change and the upgrade manually](../package/index.md#mixlibshelloutcommandtimeout-rails_migrationgitlab-rails--command-timed-out-after-3600s).
+
+### Linux package installations
+
+- In GitLab 14.10, Gitaly has introduced a new runtime directory. This directory is intended to hold all files and
+  directories Gitaly needs to create at runtime to operate correctly. This includes, for example, internal sockets, the
+  Git execution environment, or the temporary hooks directory.
+
+  This new configuration can be set via `gitaly['runtime_dir']`. It replaces the old `gitaly['internal_socket_dir']`
+  configuration. If the internal socket directory is not explicitly configured, sockets will be created in the runtime directory.
+
+  Support for `gitaly['internal_socket_dir']` will be removed in 15.0.
+
+## 14.9.0
+
+- Database changes made by the upgrade to GitLab 14.9 can take hours or days to complete on larger GitLab instances.
+  These [batched background migrations](../background_migrations.md#batched-background-migrations) update whole database tables to ensure corresponding
+  records in `namespaces` table for each record in `projects` table.
+
+  After you upgrade to 14.9.0 or a later 14.9 patch version,
+  [batched background migrations must finish](../background_migrations.md#batched-background-migrations)
+  before you upgrade to a later version.
+
+  If the migrations are not finished and you try to upgrade to a later version,
+  you see errors like:
+
+  ```plaintext
+  Expected batched background migration for the given configuration to be marked as 'finished', but it is 'active':
+  ```
+
+  Or
+
+  ```plaintext
+  Error executing action `run` on resource 'bash[migrate gitlab-rails database]'
+  ================================================================================
+
+  Mixlib::ShellOut::ShellCommandFailed
+  ------------------------------------
+  Command execution failed. STDOUT/STDERR suppressed for sensitive resource
+  ```
+
+- GitLab 14.9.0 includes a
+  [background migration `ResetDuplicateCiRunnersTokenValuesOnProjects`](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/79140)
+  that may remain stuck permanently in a **pending** state.
+
+  To clean up this stuck job, run the following in the [GitLab Rails Console](../../administration/operations/rails_console.md):
+
+  ```ruby
+  Gitlab::Database::BackgroundMigrationJob.pending.where(class_name: "ResetDuplicateCiRunnersTokenValuesOnProjects").find_each do |job|
+    puts Gitlab::Database::BackgroundMigrationJob.mark_all_as_succeeded("ResetDuplicateCiRunnersTokenValuesOnProjects", job.arguments)
+  end
+  ```
+
+- If you run GitLab with external PostgreSQL, particularly AWS RDS, ensure you
+  upgrade PostgreSQL to patch levels to a minimum of 12.7 or 13.3 before
+  upgrading to GitLab 14.8 or later.
+
+  [In 14.8](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/75511)
+  for GitLab Enterprise Edition and [in 15.0](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/87983)
+  for GitLab Community Edition a GitLab feature called Loose Foreign Keys was enabled.
+
+  After it was enabled, we have had reports of unplanned PostgreSQL restarts caused
+  by a database engine bug that causes a segmentation fault.
+
+  For more information, see [issue 364763](https://gitlab.com/gitlab-org/gitlab/-/issues/364763).
+
+### Geo installations **(PREMIUM SELF)**
+
+- **Do not** upgrade to GitLab 14.9.0. Instead, use 14.9.1 or later.
+
+  We've discovered an issue with Geo's CI verification feature that may
+  [cause job traces to be lost](https://gitlab.com/gitlab-com/gl-infra/production/-/issues/6664). This issue was fixed
+  in [the GitLab 14.9.1 patch release](https://about.gitlab.com/releases/2022/03/23/gitlab-14-9-1-released/).
+
+  If you have already upgraded to GitLab 14.9.0, you can disable the feature causing the issue by
+  [disabling the `geo_job_artifact_replication` feature flag](../../administration/feature_flags.md#how-to-enable-and-disable-features-behind-flags).
+
+## 14.8.0
+
+- If upgrading from a version earlier than 14.6.5, 14.7.4, or 14.8.2, review the [Critical Security Release: 14.8.2, 14.7.4, and 14.6.5](https://about.gitlab.com/releases/2022/02/25/critical-security-release-gitlab-14-8-2-released/) blog post.
+  Updating to 14.8.2 or later resets runner registration tokens for your groups and projects.
+- The agent server for Kubernetes [is enabled by default](https://about.gitlab.com/releases/2022/02/22/gitlab-14-8-released/#the-agent-server-for-kubernetes-is-enabled-by-default)
+  on Omnibus installations. If you run GitLab at scale,
+  such as [the reference architectures](../../administration/reference_architectures/index.md),
+  you must disable the agent on the following server types, **if the agent is not required**.
+
+  - Praefect
+  - Gitaly
+  - Sidekiq
+  - Redis (if configured using `redis['enable'] = true` and not via `roles`)
+  - Container registry
+  - Any other server types based on `roles(['application_role'])`, such as the GitLab Rails nodes
+
+  [The reference architectures](../../administration/reference_architectures/index.md) have been updated
+  with this configuration change and a specific role for standalone Redis servers.
+
+  Steps to disable the agent:
+
+  1. Add `gitlab_kas['enable'] = false` to `gitlab.rb`.
+  1. If the server is already upgraded to 14.8, run `gitlab-ctl reconfigure`.
+- GitLab 14.8.0 includes a
+[background migration `PopulateTopicsNonPrivateProjectsCount`](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/79140)
+that may remain stuck permanently in a **pending** state.
+
+  To clean up this stuck job, run the following in the [GitLab Rails Console](../../administration/operations/rails_console.md):
+
+  ```ruby
+  Gitlab::Database::BackgroundMigrationJob.pending.where(class_name: "PopulateTopicsNonPrivateProjectsCount").find_each do |job|
+    puts Gitlab::Database::BackgroundMigrationJob.mark_all_as_succeeded("PopulateTopicsNonPrivateProjectsCount", job.arguments)
+  end
+  ```
+
+- If upgrading from a version earlier than 14.3.0, to avoid
+  [an issue with job retries](https://gitlab.com/gitlab-org/gitlab/-/issues/357822), first upgrade
+  to GitLab 14.7.x and make sure all batched migrations have finished.
+- If upgrading from version 14.3.0 or later, you might notice a failed
+  [batched migration](../background_migrations.md#batched-background-migrations) named
+  `BackfillNamespaceIdForNamespaceRoute`. You can [ignore](https://gitlab.com/gitlab-org/gitlab/-/issues/357822)
+  this. Retry it after you upgrade to version 14.9.x.
+- If you run GitLab with external PostgreSQL, particularly AWS RDS, ensure you
+  upgrade PostgreSQL to patch levels to a minimum of 12.7 or 13.3 before
+  upgrading to GitLab 14.8 or later.
+
+  [In 14.8](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/75511)
+  for GitLab Enterprise Edition and [in 15.0](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/87983)
+  for GitLab Community Edition a GitLab feature called Loose Foreign Keys was enabled.
+
+  After it was enabled, we have had reports of unplanned PostgreSQL restarts caused
+  by a database engine bug that causes a segmentation fault.
+
+  For more information, see [issue 364763](https://gitlab.com/gitlab-org/gitlab/-/issues/364763).
+
+## 14.7.0
+
+- If upgrading from a version earlier than 14.6.5, 14.7.4, or 14.8.2, review the [Critical Security Release: 14.8.2, 14.7.4, and 14.6.5](https://about.gitlab.com/releases/2022/02/25/critical-security-release-gitlab-14-8-2-released/) blog post.
+  Updating to 14.7.4 or later resets runner registration tokens for your groups and projects.
+- GitLab 14.7 introduced a change where Gitaly expects persistent files in the `/tmp` directory.
+  When using the `noatime` mount option on `/tmp` in a node running Gitaly, most Linux distributions
+  run into [an issue with Git server hooks getting deleted](https://gitlab.com/gitlab-org/gitaly/-/issues/4113).
+  These conditions are present in the default Amazon Linux configuration.
+
+  If your Linux distribution manages files in `/tmp` with the `tmpfiles.d` service, you
+  can override the behavior of `tmpfiles.d` for the Gitaly files and avoid this issue:
+
+  ```shell
+  sudo printf "x /tmp/gitaly-%s-*\n" hooks git-exec-path >/etc/tmpfiles.d/gitaly-workaround.conf
+  ```
+
+  This issue is fixed in GitLab 14.10 and later when using the [Gitaly runtime directory](https://docs.gitlab.com/omnibus/update/gitlab_14_changes.html#gitaly-runtime-directory)
+  to specify a location to store persistent files.
+
+### Linux package installations
+
+- In GitLab 14.8, we are upgrading Redis from 6.0.16 to 6.2.6. This upgrade is expected to be fully backwards compatible.
+
+  If your instance has Redis HA with Sentinel, follow the upgrade steps documented in
+  [Redis HA (using Sentinel)](../zero_downtime.md#redis-ha-using-sentinel).
+
+### Geo installations **(PREMIUM SELF)**
+
+- LFS objects import and mirror issue in GitLab 14.6.0 to 14.7.2.
+  When Geo is enabled, LFS objects fail to be saved for imported or mirrored projects.
+  [This bug](https://gitlab.com/gitlab-org/gitlab/-/issues/352368) was fixed in GitLab 14.8.0 and backported into 14.7.3.
+- There is [an issue in GitLab 14.2 through 14.7](https://gitlab.com/gitlab-org/gitlab/-/issues/299819#note_822629467)
+  that affects Geo when the GitLab-managed object storage replication is used, causing blob object types to fail synchronization.
+
+  Since GitLab 14.2, verification failures result in synchronization failures and cause a resynchronization of these objects.
+
+  As verification is not yet implemented for files stored in object storage (see
+  [issue 13845](https://gitlab.com/gitlab-org/gitlab/-/issues/13845) for more details), this
+  results in a loop that consistently fails for all objects stored in object storage.
+
+  For information on how to fix this, see
+  [Troubleshooting - Failed syncs with GitLab-managed object storage replication](../../administration/geo/replication/troubleshooting.md#failed-syncs-with-gitlab-managed-object-storage-replication).
+
+## 14.6.0
+
+- If upgrading from a version earlier than 14.6.5, 14.7.4, or 14.8.2, review the [Critical Security Release: 14.8.2, 14.7.4, and 14.6.5](https://about.gitlab.com/releases/2022/02/25/critical-security-release-gitlab-14-8-2-released/) blog post.
+  Updating to 14.6.5 or later resets runner registration tokens for your groups and projects.
+
+### Geo installations **(PREMIUM SELF)**
+
+- LFS objects import and mirror issue in GitLab 14.6.0 to 14.7.2.
+  When Geo is enabled, LFS objects fail to be saved for imported or mirrored projects.
+  [This bug](https://gitlab.com/gitlab-org/gitlab/-/issues/352368) was fixed in GitLab 14.8.0 and backported into 14.7.3.
+- There is [an issue in GitLab 14.2 through 14.7](https://gitlab.com/gitlab-org/gitlab/-/issues/299819#note_822629467)
+  that affects Geo when the GitLab-managed object storage replication is used, causing blob object types to fail synchronization.
+
+  Since GitLab 14.2, verification failures result in synchronization failures and cause a resynchronization of these objects.
+
+  As verification is not yet implemented for files stored in object storage (see
+  [issue 13845](https://gitlab.com/gitlab-org/gitlab/-/issues/13845) for more details), this
+  results in a loop that consistently fails for all objects stored in object storage.
+
+  For information on how to fix this, see
+  [Troubleshooting - Failed syncs with GitLab-managed object storage replication](../../administration/geo/replication/troubleshooting.md#failed-syncs-with-gitlab-managed-object-storage-replication).
+
+## 14.5.0
+
+- Connections between Workhorse and Gitaly use the Gitaly `backchannel` protocol by default. If you deployed a gRPC proxy between Workhorse and Gitaly,
+  Workhorse can no longer connect. As a workaround, [disable the temporary `workhorse_use_sidechannel`](../../administration/feature_flags.md#enable-or-disable-the-feature)
+  feature flag. If you need a proxy between Workhorse and Gitaly, use a TCP proxy. If you have feedback about this change, go to [this issue](https://gitlab.com/gitlab-com/gl-infra/scalability/-/issues/1301).
+
+- In 14.1 we introduced a background migration that changes how we store merge request diff commits,
+  to significantly reduce the amount of storage needed.
+  In 14.5 we introduce a set of migrations that wrap up this process by making sure
+  that all remaining jobs over the `merge_request_diff_commits` table are completed.
+  These jobs have already been processed in most cases so that no extra time is necessary during an upgrade to 14.5.
+  However, if there are remaining jobs or you haven't already upgraded to 14.1,
+  the deployment may take multiple hours to complete.
+
+  All merge request diff commits automatically incorporate these changes, and there are no
+  additional requirements to perform the upgrade.
+  Existing data in the `merge_request_diff_commits` table remains unpacked until you run `VACUUM FULL merge_request_diff_commits`.
+  However, the `VACUUM FULL` operation locks and rewrites the entire `merge_request_diff_commits` table,
+  so the operation takes some time to complete and it blocks access to this table until the end of the process.
+  We advise you to only run this command while GitLab is not actively used or it is taken offline for the duration of the process.
+  The time it takes to complete depends on the size of the table, which can be obtained by using `select pg_size_pretty(pg_total_relation_size('merge_request_diff_commits'));`.
+
+  For more information, refer to [this issue](https://gitlab.com/gitlab-org/gitlab/-/issues/331823).
+
+- GitLab 14.5.0 includes a
+  [background migration `UpdateVulnerabilityOccurrencesLocation`](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/72788)
+  that may remain stuck permanently in a **pending** state when the instance lacks records that match the migration's target.
+
+  To clean up this stuck job, run the following in the [GitLab Rails Console](../../administration/operations/rails_console.md):
+
+  ```ruby
+  Gitlab::Database::BackgroundMigrationJob.pending.where(class_name: "UpdateVulnerabilityOccurrencesLocation").find_each do |job|
+    puts Gitlab::Database::BackgroundMigrationJob.mark_all_as_succeeded("UpdateVulnerabilityOccurrencesLocation", job.arguments)
+  end
+  ```
+
+- Upgrading to 14.5 (or later) [might encounter a one hour timeout](https://gitlab.com/gitlab-org/gitlab/-/issues/354211)
+  owing to a long running database data change.
+
+  ```plaintext
+  FATAL: Mixlib::ShellOut::CommandTimeout: rails_migration[gitlab-rails]
+  (gitlab::database_migrations line 51) had an error:
+  [..]
+  Mixlib::ShellOut::CommandTimeout: Command timed out after 3600s:
+  ```
+
+  [There is a workaround to complete the data change and the upgrade manually](../package/index.md#mixlibshelloutcommandtimeout-rails_migrationgitlab-rails--command-timed-out-after-3600s)
+
+- As part of [enabling real-time issue assignees](https://gitlab.com/gitlab-org/gitlab/-/issues/330117), Action Cable is now enabled by default.
+  For **self-compiled (source) installations**, `config/cable.yml` is required to be present.
+
+  Configure this by running:
+
+  ```shell
+  cd /home/git/gitlab
+  sudo -u git -H cp config/cable.yml.example config/cable.yml
+
+  # Change the Redis socket path if you are not using the default Debian / Ubuntu configuration
+  sudo -u git -H editor config/cable.yml
+  ```
+
+### Self-compiled installations
+
+- When `make` is run, Gitaly builds are now created in `_build/bin` and no longer in the root directory of the source directory. If you
+are using a self-compiled installation, update paths to these binaries in your [systemd unit files](../upgrading_from_source.md#configure-systemd-units)
+or [init scripts](../upgrading_from_source.md#configure-sysv-init-script) by [following the documentation](../upgrading_from_source.md).
+
+### Geo installations **(PREMIUM SELF)**
+
+- There is [an issue in GitLab 14.2 through 14.7](https://gitlab.com/gitlab-org/gitlab/-/issues/299819#note_822629467)
+  that affects Geo when the GitLab-managed object storage replication is used, causing blob object types to fail synchronization.
+
+  Since GitLab 14.2, verification failures result in synchronization failures and cause a resynchronization of these objects.
+
+  As verification is not yet implemented for files stored in object storage (see
+  [issue 13845](https://gitlab.com/gitlab-org/gitlab/-/issues/13845) for more details), this
+  results in a loop that consistently fails for all objects stored in object storage.
+
+  For information on how to fix this, see
+  [Troubleshooting - Failed syncs with GitLab-managed object storage replication](../../administration/geo/replication/troubleshooting.md#failed-syncs-with-gitlab-managed-object-storage-replication).
+
+## 14.4.4
+
+- For [zero-downtime upgrades](../zero_downtime.md) on a GitLab cluster with separate Web and API nodes, you must enable the `paginated_tree_graphql_query` [feature flag](../../administration/feature_flags.md#enable-or-disable-the-feature) _before_ upgrading GitLab Web nodes to 14.4.
+  This is because we [enabled `paginated_tree_graphql_query` by default in 14.4](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/70913/diffs), so if GitLab UI is on 14.4 and its API is on 14.3, the frontend has this feature enabled but the backend has it disabled. This results in the following error:
+
+  ```shell
+  bundle.esm.js:63 Uncaught (in promise) Error: GraphQL error: Field 'paginatedTree' doesn't exist on type 'Repository'
+  ```
+
+## 14.4.1 and 14.4.2
+
+### Geo installations **(PREMIUM SELF)**
+
+- There is [an issue in GitLab 14.4.0 through 14.4.2](#1440) that can affect
+  Geo and other features that rely on cronjobs. We recommend upgrading to GitLab 14.4.3 or later.
+
+## 14.4.0
+
+- Git 2.33.x and later is required. We recommend you use the
+  [Git version provided by Gitaly](../../install/installation.md#git).
+- When [Maintenance mode](../../administration/maintenance_mode/index.md) is
+  enabled, users cannot sign in with SSO, SAML, or LDAP.
+
+  Users who were signed in before Maintenance mode was enabled, continue to be
+  signed in. If the administrator who enabled Maintenance mode loses their
+  session, then they can't disable Maintenance mode via the UI. In that case,
+  you can
+  [disable Maintenance mode via the API or Rails console](../../administration/maintenance_mode/index.md#disable-maintenance-mode).
+
+  [This bug](https://gitlab.com/gitlab-org/gitlab/-/issues/329261) was fixed in
+  GitLab 14.5.0 and backported into 14.4.3 and 14.3.5.
+- After enabling database load balancing by default in 14.4.0, we found an issue where
+  [cron jobs would not work if the connection to PostgreSQL was severed](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/73716),
+  as Sidekiq would continue using a bad connection. Geo and other features that rely on
+  cron jobs running regularly do not work until Sidekiq is restarted. We recommend
+  upgrading to GitLab 14.4.3 and later if this issue affects you.
+- After enabling database load balancing by default in 14.4.0, we found an issue where
+  [Database load balancing does not work with an AWS Aurora cluster](https://gitlab.com/gitlab-org/gitlab/-/issues/220617).
+  We recommend moving your databases from Aurora to RDS for PostgreSQL before
+  upgrading. Refer to [Moving GitLab databases to a different PostgreSQL instance](../../administration/postgresql/moving.md).
+- GitLab 14.4.0 includes a
+[background migration `PopulateTopicsTotalProjectsCountCache`](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/71033)
+that may remain stuck permanently in a **pending** state when the instance lacks records that match the migration's target.
+
+  To clean up this stuck job, run the following in the [GitLab Rails Console](../../administration/operations/rails_console.md):
+
+  ```ruby
+  Gitlab::Database::BackgroundMigrationJob.pending.where(class_name: "PopulateTopicsTotalProjectsCountCache").find_each do |job|
+    puts Gitlab::Database::BackgroundMigrationJob.mark_all_as_succeeded("PopulateTopicsTotalProjectsCountCache", job.arguments)
+  end
+  ```
+
+### Linux package installations
+
+- In GitLab 14.4, the provided Grafana version is 7.5, this is a downgrade from the Grafana 8.1 version introduced in
+  GitLab 14.3. This was reverted to an Apache-licensed Grafana release to allow time to consider the implications of the
+  newer AGPL-licensed releases.
+
+  Users that have customized their Grafana install with plugins or library panels may experience errors in Grafana after
+  the downgrade. If the errors persist after a Grafana restart you may need to reset the Grafana db and re-add the
+  customizations. The Grafana database can be reset with `sudo gitlab-ctl reset-grafana`.
+
+### Geo installations **(PREMIUM SELF)**
+
+- There is [an issue in GitLab 14.2 through 14.7](https://gitlab.com/gitlab-org/gitlab/-/issues/299819#note_822629467)
+  that affects Geo when the GitLab-managed object storage replication is used, causing blob object types to fail synchronization.
+
+  Since GitLab 14.2, verification failures result in synchronization failures and cause a resynchronization of these objects.
+
+  As verification is not yet implemented for files stored in object storage (see
+  [issue 13845](https://gitlab.com/gitlab-org/gitlab/-/issues/13845) for more details), this
+  results in a loop that consistently fails for all objects stored in object storage.
+
+  For information on how to fix this, see
+  [Troubleshooting - Failed syncs with GitLab-managed object storage replication](../../administration/geo/replication/troubleshooting.md#failed-syncs-with-gitlab-managed-object-storage-replication).
+
+- There is [an issue in GitLab 14.4.0 through 14.4.2](#1440) that can affect
+  Geo and other features that rely on cronjobs. We recommend upgrading to GitLab 14.4.3 or later.
+
+## 14.3.0
+
+- [Instances running 14.0.0 - 14.0.4 should not upgrade directly to GitLab 14.2 or later](#upgrading-to-later-14y-releases).
+- Ensure [batched background migrations finish](../background_migrations.md#batched-background-migrations) before upgrading
+  to 14.3.Z from earlier GitLab 14 releases.
+- GitLab 14.3.0 contains post-deployment migrations to [address Primary Key overflow risk for tables with an integer PK](https://gitlab.com/groups/gitlab-org/-/epics/4785) for the tables listed below:
+
+  - [`ci_builds.id`](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/70245)
+  - [`ci_builds.stage_id`](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/66688)
+  - [`ci_builds_metadata`](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/65692)
+  - [`taggings`](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/66625)
+  - [`events`](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/64779)
+
+  If the migrations are executed as part of a no-downtime deployment, there's a risk of failure due to lock conflicts with the application logic, resulting in lock timeout or deadlocks. In each case, these migrations are safe to re-run until successful:
+
+  ```shell
+  # For Omnibus GitLab
+  sudo gitlab-rake db:migrate
+
+  # For source installations
+  sudo -u git -H bundle exec rake db:migrate RAILS_ENV=production
+  ```
+
+- After upgrading to 14.3, ensure that all the `MigrateMergeRequestDiffCommitUsers` background
+  migration jobs have completed before continuing with upgrading to GitLab 14.5 or later.
+  This is especially important if your GitLab instance has a large
+  `merge_request_diff_commits` table. Any pending
+  `MigrateMergeRequestDiffCommitUsers` background migration jobs are
+  foregrounded in GitLab 14.5, and may take a long time to complete.
+  You can check the count of pending jobs for
+  `MigrateMergeRequestDiffCommitUsers` by using the PostgreSQL console (or `sudo
+  gitlab-psql`):
+
+  ```sql
+  select status, count(*) from background_migration_jobs
+  where class_name = 'MigrateMergeRequestDiffCommitUsers' group by status;
+  ```
+
+  As jobs are completed, the database records change from `0` (pending) to `1`. If the number of
+  pending jobs doesn't decrease after a while, it's possible that the
+  `MigrateMergeRequestDiffCommitUsers` background migration jobs have failed. You
+  can check for errors in the Sidekiq logs:
+
+  ```shell
+  sudo grep MigrateMergeRequestDiffCommitUsers /var/log/gitlab/sidekiq/current | grep -i error
+  ```
+
+  If needed, you can attempt to run the `MigrateMergeRequestDiffCommitUsers` background
+  migration jobs manually in the [GitLab Rails Console](../../administration/operations/rails_console.md).
+  This can be done using Sidekiq asynchronously, or by using a Rails process directly:
+
+  - Using Sidekiq to schedule jobs asynchronously:
+
+    ```ruby
+    # For the first run, only attempt to execute 1 migration. If successful, increase
+    # the limit for subsequent runs
+    limit = 1
+
+    jobs = Gitlab::Database::BackgroundMigrationJob.for_migration_class('MigrateMergeRequestDiffCommitUsers').pending.to_a
+
+    pp "#{jobs.length} jobs remaining"
+
+    jobs.first(limit).each do |job|
+      BackgroundMigrationWorker.perform_in(5.minutes, 'MigrateMergeRequestDiffCommitUsers', job.arguments)
+    end
+    ```
+
+    NOTE:
+    The queued jobs can be monitored using the Sidekiq admin panel, which can be accessed at the `/admin/sidekiq` endpoint URI.
+
+  - Using a Rails process to run jobs synchronously:
+
+    ```ruby
+    def process(concurrency: 1)
+      queue = Queue.new
+
+      Gitlab::Database::BackgroundMigrationJob
+        .where(class_name: 'MigrateMergeRequestDiffCommitUsers', status: 0)
+        .each { |job| queue << job }
+
+      concurrency
+        .times
+        .map do
+          Thread.new do
+            Thread.abort_on_exception = true
+
+            loop do
+              job = queue.pop(true)
+              time = Benchmark.measure do
+                Gitlab::BackgroundMigration::MigrateMergeRequestDiffCommitUsers
+                  .new
+                  .perform(*job.arguments)
+              end
+
+              puts "#{job.id} finished in #{time.real.round(2)} seconds"
+            rescue ThreadError
+              break
+            end
+          end
+        end
+        .each(&:join)
+    end
+
+    ActiveRecord::Base.logger.level = Logger::ERROR
+    process
+    ```
+
+    NOTE:
+    When using Rails to execute these background migrations synchronously, make sure that the machine running the process has sufficient resources to handle the task. If the process gets terminated, it's likely due to insufficient memory available. If your SSH session times out after a while, it might be necessary to run the previous code by using a terminal multiplexer like `screen` or `tmux`.
+
+- When [Maintenance mode](../../administration/maintenance_mode/index.md) is
+  enabled, users cannot sign in with SSO, SAML, or LDAP.
+
+  Users who were signed in before Maintenance mode was enabled, continue to be
+  signed in. If the administrator who enabled Maintenance mode loses their
+  session, then they can't disable Maintenance mode via the UI. In that case,
+  you can
+  [disable Maintenance mode via the API or Rails console](../../administration/maintenance_mode/index.md#disable-maintenance-mode).
+
+  [This bug](https://gitlab.com/gitlab-org/gitlab/-/issues/329261) was fixed in
+  GitLab 14.5.0 and backported into 14.4.3 and 14.3.5.
+
+- You may see the following error when setting up two factor authentication (2FA) for accounts
+  that authenticate using an LDAP password:
+
+  ```plaintext
+  You must provide a valid current password
+  ```
+
+  - The error occurs because verification is incorrectly performed against accounts'
+    randomly generated internal GitLab passwords, not the LDAP passwords.
+  - This is [fixed in GitLab 14.5.0 and backported to 14.4.3](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/73538).
+  - Workarounds:
+    - Instead of upgrading to GitLab 14.3.x to comply with the supported upgrade path:
+      1. Upgrade to 14.4.5.
+      1. Make sure the [`MigrateMergeRequestDiffCommitUsers` background migration](#1430) has finished.
+      1. Upgrade to GitLab 14.5 or later.
+    - Reset the random password for affected accounts, using [the Rake task](../../security/reset_user_password.md#use-a-rake-task):
+
+      ```plaintext
+      sudo gitlab-rake "gitlab:password:reset[user_handle]"
+      ```
+
+- If you encounter the error, `I18n::InvalidLocale: :en is not a valid locale`, when starting the application, follow the [patching](https://about.gitlab.com/handbook/support/workflows/patching_an_instance.html) process. Use [122978](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/122978) as the `mr_iid`.
+
+### Self-compiled installations
+
+- Ruby 2.7.4 is required. Refer to [the Ruby installation instructions](../../install/installation.md#2-ruby)
+  for how to proceed.
+
+### Geo installations **(PREMIUM SELF)**
+
+- There is [an issue in GitLab 14.2 through 14.7](https://gitlab.com/gitlab-org/gitlab/-/issues/299819#note_822629467)
+  that affects Geo when the GitLab-managed object storage replication is used, causing blob object types to fail synchronization.
+
+  Since GitLab 14.2, verification failures result in synchronization failures and cause a resynchronization of these objects.
+
+  As verification is not yet implemented for files stored in object storage (see
+  [issue 13845](https://gitlab.com/gitlab-org/gitlab/-/issues/13845) for more details), this
+  results in a loop that consistently fails for all objects stored in object storage.
+
+  For information on how to fix this, see
+  [Troubleshooting - Failed syncs with GitLab-managed object storage replication](../../administration/geo/replication/troubleshooting.md#failed-syncs-with-gitlab-managed-object-storage-replication).
+
+- We found an [issue](https://gitlab.com/gitlab-org/gitlab/-/issues/336013) where the Container Registry replication
+  wasn't fully working if you used multi-arch images. In case of a multi-arch image, only the primary architecture
+  (for example `amd64`) would be replicated to the secondary site. This has been [fixed in GitLab 14.3](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/67624) and was backported to 14.2 and 14.1, but manual steps are required to force a re-sync.
+
+  You can check if you are affected by running:
+
+  ```shell
+  docker manifest inspect <SECONDARY_IMAGE_LOCATION> | jq '.mediaType'
+  ```
+
+  Where `<SECONDARY_IMAGE_LOCATION>` is a container image on your secondary site.
+  If the output matches `application/vnd.docker.distribution.manifest.list.v2+json`
+  (there can be a `mediaType` entry at several levels, we only care about the top level entry),
+  then you don't need to do anything.
+
+  Otherwise, for each **secondary** site, on a Rails application node, open a [Rails console](../../administration/operations/rails_console.md), and run the following:
+
+   ```ruby
+   list_type = 'application/vnd.docker.distribution.manifest.list.v2+json'
+
+   Geo::ContainerRepositoryRegistry.synced.each do |gcr|
+     cr = gcr.container_repository
+     primary = Geo::ContainerRepositorySync.new(cr)
+     cr.tags.each do |tag|
+       primary_manifest = JSON.parse(primary.send(:client).repository_raw_manifest(cr.path, tag.name))
+       next unless primary_manifest['mediaType'].eql?(list_type)
+
+       cr.delete_tag_by_name(tag.name)
+     end
+     primary.execute
+   end
+   ```
+
+  If you are running a version prior to 14.1 and are using Geo and multi-arch containers in your Container Registry,
+  we recommend [upgrading](../../administration/geo/replication/upgrading_the_geo_sites.md) to at least GitLab 14.1.
+
+## 14.2.0
+
+- [Instances running 14.0.0 - 14.0.4 should not upgrade directly to GitLab 14.2 or later](#upgrading-to-later-14y-releases).
+- Ensure [batched background migrations finish](../background_migrations.md#batched-background-migrations) before upgrading
+  to 14.2.Z from earlier GitLab 14 releases.
+- GitLab 14.2.0 contains background migrations to [address Primary Key overflow risk for tables with an integer PK](https://gitlab.com/groups/gitlab-org/-/epics/4785) for the tables listed below:
+  - [`ci_build_needs`](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/65216)
+  - [`ci_build_trace_chunks`](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/66123)
+  - [`ci_builds_runner_session`](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/66433)
+  - [`deployments`](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/67341)
+  - [`geo_job_artifact_deleted_events`](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/66763)
+  - [`push_event_payloads`](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/67299)
+  - `ci_job_artifacts`:
+    - [Finalize `job_id` conversion to `bigint` for `ci_job_artifacts`](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/67774)
+    - [Finalize `ci_job_artifacts` conversion to `bigint`](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/65601)
+
+  If the migrations are executed as part of a no-downtime deployment, there's a risk of failure due to lock conflicts with the application logic, resulting in lock timeout or deadlocks. In each case, these migrations are safe to re-run until successful:
+
+  ```shell
+  # For Omnibus GitLab
+  sudo gitlab-rake db:migrate
+
+  # For source installations
+  sudo -u git -H bundle exec rake db:migrate RAILS_ENV=production
+  ```
+
+- When [Maintenance mode](../../administration/maintenance_mode/index.md) is
+  enabled, users cannot sign in with SSO, SAML, or LDAP.
+
+  Users who were signed in before Maintenance mode was enabled, continue to be
+  signed in. If the administrator who enabled Maintenance mode loses their
+  session, then they can't disable Maintenance mode via the UI. In that case,
+  you can
+  [disable Maintenance mode via the API or Rails console](../../administration/maintenance_mode/index.md#disable-maintenance-mode).
+
+  [This bug](https://gitlab.com/gitlab-org/gitlab/-/issues/329261) was fixed in
+  GitLab 14.5.0 and backported into 14.4.3 and 14.3.5.
+- GitLab 14.2.0 includes a
+  [background migration `BackfillDraftStatusOnMergeRequests`](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/67687)
+  that may remain stuck permanently in a **pending** state when the instance lacks records that match the migration's target.
+
+  To clean up this stuck job, run the following in the [GitLab Rails Console](../../administration/operations/rails_console.md):
+
+  ```ruby
+  Gitlab::Database::BackgroundMigrationJob.pending.where(class_name: "BackfillDraftStatusOnMergeRequests").find_each do |job|
+    puts Gitlab::Database::BackgroundMigrationJob.mark_all_as_succeeded("BackfillDraftStatusOnMergeRequests", job.arguments)
+  end
+  ```
+
+- If you encounter the error, `I18n::InvalidLocale: :en is not a valid locale`, when starting the application, follow the [patching](https://about.gitlab.com/handbook/support/workflows/patching_an_instance.html) process. Use [123476](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/123476) as the `mr_iid`.
+
+### Geo installations **(PREMIUM SELF)**
+
+- There is [an issue in GitLab 14.2 through 14.7](https://gitlab.com/gitlab-org/gitlab/-/issues/299819#note_822629467)
+  that affects Geo when the GitLab-managed object storage replication is used, causing blob object types to fail synchronization.
+
+  Since GitLab 14.2, verification failures result in synchronization failures and cause a resynchronization of these objects.
+
+  As verification is not yet implemented for files stored in object storage (see
+  [issue 13845](https://gitlab.com/gitlab-org/gitlab/-/issues/13845) for more details), this
+  results in a loop that consistently fails for all objects stored in object storage.
+
+  For information on how to fix this, see
+  [Troubleshooting - Failed syncs with GitLab-managed object storage replication](../../administration/geo/replication/troubleshooting.md#failed-syncs-with-gitlab-managed-object-storage-replication).
+
+- We found an [issue](https://gitlab.com/gitlab-org/gitlab/-/issues/336013) where the Container Registry replication
+  wasn't fully working if you used multi-arch images. In case of a multi-arch image, only the primary architecture
+  (for example `amd64`) would be replicated to the secondary site. This has been [fixed in GitLab 14.3](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/67624) and was backported to 14.2 and 14.1, but manual steps are required to force a re-sync.
+
+  You can check if you are affected by running:
+
+  ```shell
+  docker manifest inspect <SECONDARY_IMAGE_LOCATION> | jq '.mediaType'
+  ```
+
+  Where `<SECONDARY_IMAGE_LOCATION>` is a container image on your secondary site.
+  If the output matches `application/vnd.docker.distribution.manifest.list.v2+json`
+  (there can be a `mediaType` entry at several levels, we only care about the top level entry),
+  then you don't need to do anything.
+
+  Otherwise, for each **secondary** site, on a Rails application node, open a [Rails console](../../administration/operations/rails_console.md), and run the following:
+
+   ```ruby
+   list_type = 'application/vnd.docker.distribution.manifest.list.v2+json'
+
+   Geo::ContainerRepositoryRegistry.synced.each do |gcr|
+     cr = gcr.container_repository
+     primary = Geo::ContainerRepositorySync.new(cr)
+     cr.tags.each do |tag|
+       primary_manifest = JSON.parse(primary.send(:client).repository_raw_manifest(cr.path, tag.name))
+       next unless primary_manifest['mediaType'].eql?(list_type)
+
+       cr.delete_tag_by_name(tag.name)
+     end
+     primary.execute
+   end
+   ```
+
+  If you are running a version prior to 14.1 and are using Geo and multi-arch containers in your Container Registry,
+  we recommend [upgrading](../../administration/geo/replication/upgrading_the_geo_sites.md) to at least GitLab 14.1.
+
+## 14.1.0
+
+- [Instances running 14.0.0 - 14.0.4 should not upgrade directly to GitLab 14.2 or later](#upgrading-to-later-14y-releases)
+  but can upgrade to 14.1.Z.
+
+  It is not required for instances already running 14.0.5 (or later) to stop at 14.1.Z.
+  14.1 is included on the upgrade path for the broadest compatibility
+  with self-managed installations, and ensure 14.0.0-14.0.4 installations do not
+  encounter issues with [batched background migrations](../background_migrations.md#batched-background-migrations).
+
+- Upgrading to GitLab [14.5](#1450) (or later) may take a lot longer if you do not upgrade to at least 14.1
+  first. The 14.1 merge request diff commits database migration can take hours to run, but runs in the
+  background while GitLab is in use. GitLab instances upgraded directly from 14.0 to 14.5 or later must
+  run the migration in the foreground and therefore take a lot longer to complete.
+
+- When [Maintenance mode](../../administration/maintenance_mode/index.md) is
+  enabled, users cannot sign in with SSO, SAML, or LDAP.
+
+  Users who were signed in before Maintenance mode was enabled, continue to be
+  signed in. If the administrator who enabled Maintenance mode loses their
+  session, then they can't disable Maintenance mode via the UI. In that case,
+  you can
+  [disable Maintenance mode via the API or Rails console](../../administration/maintenance_mode/index.md#disable-maintenance-mode).
+
+  [This bug](https://gitlab.com/gitlab-org/gitlab/-/issues/329261) was fixed in
+  GitLab 14.5.0 and backported into 14.4.3 and 14.3.5.
+
+- If you encounter the error, `I18n::InvalidLocale: :en is not a valid locale`, when starting the application, follow the [patching](https://about.gitlab.com/handbook/support/workflows/patching_an_instance.html) process. Use [123475](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/123475) as the `mr_iid`.
+
+### Geo installations **(PREMIUM SELF)**
+
+- We found an [issue](https://gitlab.com/gitlab-org/gitlab/-/issues/336013) where the Container Registry replication
+  wasn't fully working if you used multi-arch images. In case of a multi-arch image, only the primary architecture
+  (for example `amd64`) would be replicated to the secondary site. This has been [fixed in GitLab 14.3](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/67624) and was backported to 14.2 and 14.1, but manual steps are required to force a re-sync.
+
+  You can check if you are affected by running:
+
+  ```shell
+  docker manifest inspect <SECONDARY_IMAGE_LOCATION> | jq '.mediaType'
+  ```
+
+  Where `<SECONDARY_IMAGE_LOCATION>` is a container image on your secondary site.
+  If the output matches `application/vnd.docker.distribution.manifest.list.v2+json`
+  (there can be a `mediaType` entry at several levels, we only care about the top level entry),
+  then you don't need to do anything.
+
+  Otherwise, for each **secondary** site, on a Rails application node, open a [Rails console](../../administration/operations/rails_console.md), and run the following:
+
+   ```ruby
+   list_type = 'application/vnd.docker.distribution.manifest.list.v2+json'
+
+   Geo::ContainerRepositoryRegistry.synced.each do |gcr|
+     cr = gcr.container_repository
+     primary = Geo::ContainerRepositorySync.new(cr)
+     cr.tags.each do |tag|
+       primary_manifest = JSON.parse(primary.send(:client).repository_raw_manifest(cr.path, tag.name))
+       next unless primary_manifest['mediaType'].eql?(list_type)
+
+       cr.delete_tag_by_name(tag.name)
+     end
+     primary.execute
+   end
+   ```
+
+  If you are running a version prior to 14.1 and are using Geo and multi-arch containers in your Container Registry,
+  we recommend [upgrading](../../administration/geo/replication/upgrading_the_geo_sites.md) to at least GitLab 14.1.
+- We found an issue where [Primary sites cannot be removed from the UI](https://gitlab.com/gitlab-org/gitlab/-/issues/338231).
+
+  This bug only exists in the UI and does not block the removal of Primary sites using any other method.
+
+  If you are running an affected version and need to remove your Primary site, you can manually remove the Primary site
+  by using the [Geo Sites API](../../api/geo_nodes.md#delete-a-geo-node).
+
+## 14.0.0
+
+Prerequisites:
+
+- The [GitLab 14.0 release post contains several important notes](https://about.gitlab.com/releases/2021/06/22/gitlab-14-0-released/#upgrade)
+  about pre-requisites including [using Patroni instead of repmgr](../../administration/postgresql/replication_and_failover.md#switching-from-repmgr-to-patroni),
+  migrating [to hashed storage](../../administration/raketasks/storage.md#migrate-to-hashed-storage),
+  and [to Puma](../../administration/operations/puma.md).
+- The support of PostgreSQL 11 [has been dropped](../../install/requirements.md#database). Make sure to [update your database](https://docs.gitlab.com/omnibus/settings/database.html#upgrade-packaged-postgresql-server) to version 12 before updating to GitLab 14.0.
+
+Long running batched background database migrations:
+
+- Database changes made by the upgrade to GitLab 14.0 can take hours or days to complete on larger GitLab instances.
+  These [batched background migrations](../background_migrations.md#batched-background-migrations) update whole database tables to mitigate primary key overflow and must be finished before upgrading to GitLab 14.2 or later.
+- Due to an issue where `BatchedBackgroundMigrationWorkers` were
+  [not working](https://gitlab.com/gitlab-org/charts/gitlab/-/issues/2785#note_614738345)
+  for self-managed instances, a [fix was created](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/65106)
+  that requires an update to at least 14.0.5. The fix was also released in [14.1.0](#1410).
+
+  After you update to 14.0.5 or a later 14.0 patch version,
+  [batched background migrations must finish](../background_migrations.md#batched-background-migrations)
+  before you upgrade to a later version.
+
+  If the migrations are not finished and you try to upgrade to a later version,
+  you see an error like:
+
+  ```plaintext
+  Expected batched background migration for the given configuration to be marked as 'finished', but it is 'active':
+  ```
+
+  See how to [resolve this error](../background_migrations.md#database-migrations-failing-because-of-batched-background-migration-not-finished).
+
+Other issues:
+
+- In GitLab 13.3 some [pipeline processing methods were deprecated](https://gitlab.com/gitlab-org/gitlab/-/issues/218536)
+  and this code was completely removed in GitLab 14.0. If you plan to upgrade from
+  **GitLab 13.2 or older** directly to 14.0, this is [unsupported](../index.md#upgrading-to-a-new-major-version).
+  You should instead follow a [supported upgrade path](../index.md#upgrade-paths).
+- When [Maintenance mode](../../administration/maintenance_mode/index.md) is
+  enabled, users cannot sign in with SSO, SAML, or LDAP.
+
+  Users who were signed in before Maintenance mode was enabled, continue to be
+  signed in. If the administrator who enabled Maintenance mode loses their
+  session, then they can't disable Maintenance mode via the UI. In that case,
+  you can
+  [disable Maintenance mode via the API or Rails console](../../administration/maintenance_mode/index.md#disable-maintenance-mode).
+
+  [This bug](https://gitlab.com/gitlab-org/gitlab/-/issues/329261) was fixed in
+  GitLab 14.5.0 and backported into 14.4.3 and 14.3.5.
+- **In GitLab 13.12.2 and later**, users with expired passwords can no longer authenticate with API and Git using tokens because of
+  the [Insufficient Expired Password Validation](https://about.gitlab.com/releases/2021/06/01/security-release-gitlab-13-12-2-released/#insufficient-expired-password-validation)
+  security fix. If your users get authentication issues following the upgrade, check that their password is not expired:
+
+  1. [Connect to the PostgreSQL database](https://docs.gitlab.com/omnibus/settings/database.html#connecting-to-the-postgresql-database) and execute the
+     following query:
+
+     ```sql
+     select id,username,password_expires_at from users where password_expires_at < now();
+     ```
+
+  1. If the user is in the returned list, reset the `password_expires_at` for that user:
+
+     ```sql
+     update users set password_expires_at = null where username='<USERNAME>';
+     ```
+
+### Linux package installations
+
+- In GitLab 13.0, `sidekiq-cluster` was enabled by default and the `sidekiq` service ran `sidekiq-cluster` under the hood.
+  However, users could control this behavior using `sidekiq['cluster']` setting to run Sidekiq directly instead. Users
+  could also run `sidekiq-cluster` separately using the various `sidekiq_cluster[*]` settings available in `gitlab.rb`.
+  However these features were deprecated and are now being removed.
+
+  Starting with GitLab 14.0, `sidekiq-cluster` becomes the only way to run Sidekiq in Linux package installations. As
+  part of this process, support for the following settings in `gitlab.rb` is being removed:
+
+  - `sidekiq['cluster']` setting. Sidekiq can only be run using `sidekiq-cluster` now.
+  - `sidekiq_cluster[*]` settings. They should be set via respective `sidekiq[*]` counterparts.
+  - `sidekiq['concurrency']` setting. The limits should be controlled using the two settings `sidekiq['min_concurrency']`
+    and `sidekiq['max_concurrency']`.
+
+- In GitLab 13.0, Puma became the default web server for GitLab, but users were still able to continue using Unicorn if
+  needed. Starting with GitLab 14.0, Unicorn is no longer supported as a webserver for GitLab and is no longer shipped
+  with the Linux package.
+
+  Users must migrate to Puma following [the documentation](../../administration/operations/puma.md) to upgrade to GitLab
+  14.0.
+- The Consul version has been updated from 1.6.10 to 1.9.6 for Geo and multi-node PostgreSQL installs. Its important
+  that Consul nodes be upgraded and restarted one at a time.
+
+  For more information, see [Upgrade the Consul nodes](../../administration/consul.md#upgrade-the-consul-nodes).
+- Starting with GitLab 14.0, GitLab automatically generates a password for initial administrator user (`root`) and stores
+  this value to `/etc/gitlab/initial_root_password`.
+
+  For more information, see
+  [Set up the initial password](https://docs.gitlab.com/omnibus/installation/index.html#set-up-the-initial-password).
+- The binaries for PostgreSQL 11 and repmgr have been removed. Prior to upgrading, administrators of Linux package
+  installations must:
+  1. Ensure the installation is using [PostgreSQL 12](https://docs.gitlab.com/omnibus/settings/database.html#upgrade-packaged-postgresql-server).
+  1. If using repmgr, [convert to using patroni](../../administration/postgresql/replication_and_failover.md#switching-from-repmgr-to-patroni).
+- Two configuration options for Redis were deprecated in GitLab 13 and removed in GitLab 14:
+
+  - `redis_slave_role` is replaced with `redis_replica_role`.
+  - `redis['client_output_buffer_limit_slave']` is replaced with `redis['client_output_buffer_limit_replica']`.
+
+  Redis Cache nodes being upgraded from GitLab 13.12 to 14.0 that still refer to `redis_slave_role` in `gitlab.rb` will
+  encounter an error in the output of `gitlab-ctl reconfigure`:
+
+  ```plaintext
+  There was an error running gitlab-ctl reconfigure:
+
+  The following invalid roles have been set in 'roles': redis_slave_role
+  ```
+
+### Geo installations **(PREMIUM SELF)**
+
+- We found an issue where [Primary sites cannot be removed from the UI](https://gitlab.com/gitlab-org/gitlab/-/issues/338231).
+
+  This bug only exists in the UI and does not block the removal of Primary sites using any other method.
+
+  If you are running an affected version and need to remove your Primary site, you can manually remove the Primary site
+  by using the [Geo Sites API](../../api/geo_nodes.md#delete-a-geo-node).
+
+### Upgrading to later 14.Y releases
+
+- Instances running 14.0.0 - 14.0.4 should not upgrade directly to GitLab 14.2 or later,
+  because of [batched background migrations](../background_migrations.md#batched-background-migrations).
+  1. Upgrade first to either:
+     - 14.0.5 or a later 14.0.Z patch release.
+     - 14.1.0 or a later 14.1.Z patch release.
+  1. [Batched background migrations must finish](../background_migrations.md#batched-background-migrations)
+     before you upgrade to a later version [and may take longer than usual](#1400).
diff --git a/doc/update/versions/gitlab_15_changes.md b/doc/update/versions/gitlab_15_changes.md
new file mode 100644
index 00000000000..12cbf5c5ec1
--- /dev/null
+++ b/doc/update/versions/gitlab_15_changes.md
@@ -0,0 +1,921 @@
+---
+stage: Systems
+group: Distribution
+info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments
+---
+
+# GitLab 15 changes **(FREE SELF)**
+
+This page contains upgrade information for minor and patch versions of GitLab 15.
+Ensure you review these instructions for:
+
+- Your installation type.
+- All versions between your current version and your target version.
+
+For more information about upgrading GitLab Helm Chart, see [the release notes for 6.0](https://docs.gitlab.com/charts/releases/6_0.html).
+
+## 15.11.1
+
+- Many [project importers](../../user/project/import/index.md) and [group importers](../../user/group/import/index.md) now
+  require the Maintainer role instead of only requiring the Developer role. For more information, see the documentation
+  for any importers you use.
+
+## 15.11.0
+
+- **Upgrade to patch release 15.11.3 or later**. This avoids [issue 408304](https://gitlab.com/gitlab-org/gitlab/-/issues/408304) when upgrading from 15.5.0 and earlier.
+
+### Linux package installations
+
+In GitLab 15.11, PostgreSQL will automatically be upgraded to 13.x except for the following cases:
+
+- You are running the database in high availability using Patroni.
+- Your database nodes are part of a GitLab Geo configuration.
+- You have specifically [opted out](https://docs.gitlab.com/omnibus/settings/database.html#opt-out-of-automatic-postgresql-upgrades) from automatically upgrading PostgreSQL.
+- You have `postgresql['version'] = 12` in your `/etc/gitlab/gitlab.rb`.
+
+Fault-tolerant and Geo installations support manual upgrades to PostgreSQL 13,
+see [Packaged PostgreSQL deployed in an HA/Geo Cluster](https://docs.gitlab.com/omnibus/settings/database.html#packaged-postgresql-deployed-in-an-hageo-cluster).
+
+### Geo installations **(PREMIUM SELF)**
+
+- Some project imports do not initialize wiki repositories on project creation. See
+  [the details and workaround](gitlab_16_changes.md#wiki-repositories-not-initialized-on-project-creation).
+- `pg_upgrade` fails to upgrade the bundled PostregSQL database to version 13. See
+  [the details and workaround](#pg_upgrade-fails-to-upgrade-the-bundled-postregsql-database-to-version-13).
+
+#### `pg_upgrade` fails to upgrade the bundled PostregSQL database to version 13
+
+| Affected minor releases | Affected patch releases | Fixed in |
+|-------------------------|-------------------------|----------|
+| 15.2 - 15.10            | All                     | None     |
+| 15.11                   | 15.11.0 - 15.11.11      | 15.11.12 and later |
+
+A [bug](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/7841) in the
+built-in `pg-upgrade` tool prevents upgrading the bundled PostgreSQL database
+to version 13. This leaves the secondary site in a broken state, and prevents
+upgrading the Geo installation to GitLab 16.x
+([PostgreSQL 12 support has removed in 16.0](../deprecations.md#postgresql-12-deprecated) and later
+releases). This occurs on secondary sites using the bundled PostgreSQL
+software, running both the secondary main Rails database and tracking database
+on the same node. There is a manual
+[workaround](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/7841#workaround)
+if you can't upgrade to 15.11.12 and later.
+
+## 15.11.x
+
+- A [bug](https://gitlab.com/gitlab-org/gitlab/-/issues/411604) can cause new LDAP users signing in for the first time to be assigned a username based on their email address instead of their LDAP username attribute. A manual workaround is to set `gitlab_rails['omniauth_auto_link_ldap_user'] = true`, or upgrade to GitLab 16.1 or later where the bug has been fixed.
+
+## 15.10.5
+
+- A [bug with Elastic Indexer Cron Workers](https://gitlab.com/gitlab-org/gitlab/-/issues/408214) can cause saturation in Sidekiq.
+  - When this issue occurs, merge request merges, pipelines, Slack notifications, and other events are not created or take a long time to occur.
+  - This issue may not manifest immediately as it can take up to a week before the Sidekiq is saturated enough.
+  - Elasticsearch does not need to be enabled for this to occur.
+  - To resolve this issue, upgrade to 15.11 or use the workaround in the issue.
+- Many [project importers](../../user/project/import/index.md) and [group importers](../../user/group/import/index.md) now
+  require the Maintainer role instead of only requiring the Developer role. For more information, see the documentation
+  for any importers you use.
+
+## 15.10.0
+
+- A [bug with Elastic Indexer Cron Workers](https://gitlab.com/gitlab-org/gitlab/-/issues/408214) can cause saturation in Sidekiq.
+  - When this issue occurs, merge request merges, pipelines, Slack notifications, and other events are not created or take a long time to occur.
+  - This issue may not manifest immediately as it can take up to a week before the Sidekiq is saturated enough.
+  - Elasticsearch does not need to be enabled for this to occur.
+  - To resolve this issue, upgrade to 15.11 or use the workaround in the issue.
+- A [bug with zero-downtime reindexing](https://gitlab.com/gitlab-org/gitlab/-/issues/422938) can cause a `Couldn't load task status` error when you reindex. You might also get a `sliceId must be greater than 0 but was [-1]` error on the Elasticsearch host. As a workaround, consider [reindexing from scratch](../../integration/advanced_search/elasticsearch_troubleshooting.md#last-resort-to-recreate-an-index) or upgrading to GitLab 16.3.
+- Gitaly configuration changes significantly in Omnibus GitLab 16.0. You can begin migrating to the new structure in Omnibus GitLab 15.10 while backwards compatibility is
+  maintained in the lead up to Omnibus GitLab 16.0. [Read more about this change](gitlab_16_changes.md#gitaly-configuration-structure-change).
+- You might encounter the following error while upgrading to GitLab 15.10 or later:
+
+  ```shell
+  STDOUT: rake aborted!
+  StandardError: An error has occurred, all later migrations canceled:
+  PG::CheckViolation: ERROR:  check constraint "check_70f294ef54" is violated by some row
+  ```
+
+  This error is caused by a [batched background migration introduced in GitLab 15.8](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/107701)
+  not being finalized before GitLab 15.10. To resolve this error:
+
+  1. Execute the following SQL statement using the database console (`sudo gitlab-psql` for Linux package installs):
+
+     ```sql
+     UPDATE oauth_access_tokens SET expires_in = '7200' WHERE expires_in IS NULL;
+     ```
+
+  1. [Re-run database migrations](../../administration/raketasks/maintenance.md#run-incomplete-database-migrations).
+
+- You might also encounter the following error while upgrading to GitLab 15.10 or later:
+
+  ```shell
+  "exception.class": "ActiveRecord::StatementInvalid",
+  "exception.message": "PG::SyntaxError: ERROR:  zero-length delimited identifier at or near \"\"\"\"\nLINE 1: ...COALESCE(\"lock_version\", 0) + 1 WHERE \"ci_builds\".\"\" IN (SEL...\n
+  ```
+
+  This error is caused by a [batched background migration introduced in GitLab 14.9](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/81410)
+  not being finalized before upgrading to GitLab 15.10 or later. To resolve this error, it is safe to [mark the migration as complete](../background_migrations.md#mark-a-failed-migration-finished):
+
+  ```ruby
+  # Start the rails console
+
+  connection = Ci::ApplicationRecord.connection
+
+  Gitlab::Database::SharedModel.using_connection(connection) do
+    migration = Gitlab::Database::BackgroundMigration::BatchedMigration.find_for_configuration(
+      Gitlab::Database.gitlab_schemas_for_connection(connection), 'NullifyOrphanRunnerIdOnCiBuilds', :ci_builds, :id, [])
+
+    # mark all jobs completed
+    migration.batched_jobs.update_all(status: Gitlab::Database::BackgroundMigration::BatchedJob.state_machine.states[:succeeded].value)
+    migration.update_attribute(:status, Gitlab::Database::BackgroundMigration::BatchedMigration.state_machine.states[:finished].value)
+  end
+  ```
+
+  For more information, see [issue 415724](https://gitlab.com/gitlab-org/gitlab/-/issues/415724).
+
+### Geo installations **(PREMIUM SELF)**
+
+- `pg_upgrade` fails to upgrade the bundled PostregSQL database to version 13. See
+  [the details and workaround](#pg_upgrade-fails-to-upgrade-the-bundled-postregsql-database-to-version-13).
+
+## 15.9.0
+
+- A [bug with Elastic Indexer Cron Workers](https://gitlab.com/gitlab-org/gitlab/-/issues/408214) can cause saturation in Sidekiq.
+  - When this issue occurs, merge request merges, pipelines, Slack notifications, and other events are not created or take a long time to occur.
+  - This issue may not manifest immediately as it can take up to a week before the Sidekiq is saturated enough.
+  - Elasticsearch does not need to be enabled for this to occur.
+  - To resolve this issue, upgrade to 15.11 or use the workaround in the issue.
+- **Upgrade to patch release 15.9.3 or later**. This provides fixes for two database migration bugs:
+  - Patch releases 15.9.0, 15.9.1, 15.9.2 have a bug that can cause data loss
+    from the user profile fields `linkedin`, `twitter`, `skype`, `website_url`,
+    `location`, and `organization`. For more information, see
+    [issue 393216](https://gitlab.com/gitlab-org/gitlab/-/issues/393216).
+  - The second [bug fix](https://gitlab.com/gitlab-org/gitlab/-/issues/394760) ensures it is possible to upgrade directly from 15.4.x.
+- As part of the [CI Partitioning effort](../../architecture/blueprints/ci_data_decay/pipeline_partitioning.md), a [new Foreign Key](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/107547) was added to `ci_builds_needs`. On GitLab instances with large CI tables, adding this constraint can take longer than usual.
+- Praefect's metadata verifier's [invalid metadata deletion behavior](../../administration/gitaly/praefect.md#enable-deletions) is now enabled by default.
+
+  The metadata verifier processes replica records in the Praefect database and verifies the replicas actually exist on the Gitaly nodes. If the replica doesn't exist, its
+  metadata record is deleted. This enables Praefect to fix situations where a replica has a metadata record indicating it's fine but, in reality, it doesn't exist on disk.
+  After the metadata record is deleted, Praefect's reconciler schedules a replication job to recreate the replica.
+
+  Because of past issues with the state management logic, there may be invalid metadata records in the database. These could exist, for example, because of incomplete
+  deletions of repositories or partially completed renames. The verifier deletes these stale replica records of affected repositories. These repositories may show up as
+  unavailable repositories in the metrics and `praefect dataloss` sub-command because of the replica records being removed. If you encounter such repositories, remove
+  the repository using `praefect remove-repository` to remove the repository's remaining records.
+
+  You can find repositories with invalid metadata records prior in GitLab 15.0 and later by searching for the log records outputted by the verifier. [Read more about repository verification, and to see an example log entry](../../administration/gitaly/praefect.md#repository-verification).
+- Praefect configuration changes significantly in Omnibus GitLab 16.0. You can begin migrating to the new structure in Omnibus GitLab 15.9 while backwards compatibility is
+  maintained in the lead up to Omnibus GitLab 16.0. [Read more about this change](gitlab_16_changes.md#praefect-configuration-structure-change).
+
+### Self-compiled installations
+
+- For **self-compiled (source) installations**, with the addition of `gitlab-sshd` the Kerberos headers are needed to build GitLab Shell.
+
+  ```shell
+  sudo apt install libkrb5-dev
+  ```
+
+### Geo installations **(PREMIUM SELF)**
+
+- `pg_upgrade` fails to upgrade the bundled PostregSQL database to version 13. See
+  [the details and workaround](#pg_upgrade-fails-to-upgrade-the-bundled-postregsql-database-to-version-13).
+
+## 15.8.2
+
+### Geo installations **(PREMIUM SELF)**
+
+- We discovered an issue where [replication and verification of projects and wikis was not keeping up](https://gitlab.com/gitlab-org/gitlab/-/issues/387980) on small number of Geo installations. Your installation may be affected if you see some projects and/or wikis persistently in the "Queued" state for verification. This can lead to data loss after a failover.
+  - Affected versions: GitLab versions 15.6.x, 15.7.x, and 15.8.0 - 15.8.2.
+  - Versions containing fix: GitLab 15.8.3 and later.
+
+## 15.8.1
+
+- Due to [a bug introduced in GitLab 15.4](https://gitlab.com/gitlab-org/gitlab/-/issues/390155), if one or more Git repositories in Gitaly Cluster is [unavailable](../../administration/gitaly/recovery.md#unavailable-repositories), then [Repository checks](../../administration/repository_checks.md#repository-checks) and [Geo replication and verification](../../administration/geo/index.md) stop running for all project or project wiki repositories in the affected Gitaly Cluster. The bug was fixed by [reverting the change in GitLab 15.9.0](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/110823). Before upgrading to this version, check if you have any "unavailable" repositories. See [the bug issue](https://gitlab.com/gitlab-org/gitlab/-/issues/390155) for more information.
+
+### Geo installations **(PREMIUM SELF)**
+
+- We discovered an issue where [replication and verification of projects and wikis was not keeping up](https://gitlab.com/gitlab-org/gitlab/-/issues/387980) on small number of Geo installations. Your installation may be affected if you see some projects and/or wikis persistently in the "Queued" state for verification. This can lead to data loss after a failover.
+  - Affected versions: GitLab versions 15.6.x, 15.7.x, and 15.8.0 - 15.8.2.
+  - Versions containing fix: GitLab 15.8.3 and later.
+
+## 15.8.0
+
+- Git 2.38.0 and later is required by Gitaly. For self-compiled installations, you should use the [Git version provided by Gitaly](../../install/installation.md#git).
+- Due to [a bug introduced in GitLab 15.4](https://gitlab.com/gitlab-org/gitlab/-/issues/390155), if one or more Git repositories in Gitaly Cluster is [unavailable](../../administration/gitaly/recovery.md#unavailable-repositories), then [Repository checks](../../administration/repository_checks.md#repository-checks) and [Geo replication and verification](../../administration/geo/index.md) stop running for all project or project wiki repositories in the affected Gitaly Cluster. The bug was fixed by [reverting the change in GitLab 15.9.0](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/110823). Before upgrading to this version, check if you have any "unavailable" repositories. See [the bug issue](https://gitlab.com/gitlab-org/gitlab/-/issues/390155) for more information.
+
+### Geo installations **(PREMIUM SELF)**
+
+- `pg_upgrade` fails to upgrade the bundled PostregSQL database to version 13. See
+  [the details and workaround](#pg_upgrade-fails-to-upgrade-the-bundled-postregsql-database-to-version-13).
+- We discovered an issue where [replication and verification of projects and wikis was not keeping up](https://gitlab.com/gitlab-org/gitlab/-/issues/387980) on small number of Geo installations. Your installation may be affected if you see some projects and/or wikis persistently in the "Queued" state for verification. This can lead to data loss after a failover.
+  - Affected versions: GitLab versions 15.6.x, 15.7.x, and 15.8.0 - 15.8.2.
+  - Versions containing fix: GitLab 15.8.3 and later.
+
+## 15.7.6
+
+- Due to [a bug introduced in GitLab 15.4](https://gitlab.com/gitlab-org/gitlab/-/issues/390155), if one or more Git repositories in Gitaly Cluster is [unavailable](../../administration/gitaly/recovery.md#unavailable-repositories), then [Repository checks](../../administration/repository_checks.md#repository-checks) and [Geo replication and verification](../../administration/geo/index.md) stop running for all project or project wiki repositories in the affected Gitaly Cluster. The bug was fixed by [reverting the change in GitLab 15.9.0](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/110823). Before upgrading to this version, check if you have any "unavailable" repositories. See [the bug issue](https://gitlab.com/gitlab-org/gitlab/-/issues/390155) for more information.
+
+### Geo installations **(PREMIUM SELF)**
+
+- We discovered an issue where [replication and verification of projects and wikis was not keeping up](https://gitlab.com/gitlab-org/gitlab/-/issues/387980) on small number of Geo installations. Your installation may be affected if you see some projects and/or wikis persistently in the "Queued" state for verification. This can lead to data loss after a failover.
+  - Affected versions: GitLab versions 15.6.x, 15.7.x, and 15.8.0 - 15.8.2.
+  - Versions containing fix: GitLab 15.8.3 and later.
+
+## 15.7.5
+
+- Due to [a bug introduced in GitLab 15.4](https://gitlab.com/gitlab-org/gitlab/-/issues/390155), if one or more Git repositories in Gitaly Cluster is [unavailable](../../administration/gitaly/recovery.md#unavailable-repositories), then [Repository checks](../../administration/repository_checks.md#repository-checks) and [Geo replication and verification](../../administration/geo/index.md) stop running for all project or project wiki repositories in the affected Gitaly Cluster. The bug was fixed by [reverting the change in GitLab 15.9.0](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/110823). Before upgrading to this version, check if you have any "unavailable" repositories. See [the bug issue](https://gitlab.com/gitlab-org/gitlab/-/issues/390155) for more information.
+
+### Geo installations **(PREMIUM SELF)**
+
+- We discovered an issue where [replication and verification of projects and wikis was not keeping up](https://gitlab.com/gitlab-org/gitlab/-/issues/387980) on small number of Geo installations. Your installation may be affected if you see some projects and/or wikis persistently in the "Queued" state for verification. This can lead to data loss after a failover.
+  - Affected versions: GitLab versions 15.6.x, 15.7.x, and 15.8.0 - 15.8.2.
+  - Versions containing fix: GitLab 15.8.3 and later.
+
+## 15.7.4
+
+- Due to [a bug introduced in GitLab 15.4](https://gitlab.com/gitlab-org/gitlab/-/issues/390155), if one or more Git repositories in Gitaly Cluster is [unavailable](../../administration/gitaly/recovery.md#unavailable-repositories), then [Repository checks](../../administration/repository_checks.md#repository-checks) and [Geo replication and verification](../../administration/geo/index.md) stop running for all project or project wiki repositories in the affected Gitaly Cluster. The bug was fixed by [reverting the change in GitLab 15.9.0](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/110823). Before upgrading to this version, check if you have any "unavailable" repositories. See [the bug issue](https://gitlab.com/gitlab-org/gitlab/-/issues/390155) for more information.
+
+### Geo installations **(PREMIUM SELF)**
+
+- We discovered an issue where [replication and verification of projects and wikis was not keeping up](https://gitlab.com/gitlab-org/gitlab/-/issues/387980) on small number of Geo installations. Your installation may be affected if you see some projects and/or wikis persistently in the "Queued" state for verification. This can lead to data loss after a failover.
+  - Affected versions: GitLab versions 15.6.x, 15.7.x, and 15.8.0 - 15.8.2.
+  - Versions containing fix: GitLab 15.8.3 and later.
+
+## 15.7.3
+
+- Due to [a bug introduced in GitLab 15.4](https://gitlab.com/gitlab-org/gitlab/-/issues/390155), if one or more Git repositories in Gitaly Cluster is [unavailable](../../administration/gitaly/recovery.md#unavailable-repositories), then [Repository checks](../../administration/repository_checks.md#repository-checks) and [Geo replication and verification](../../administration/geo/index.md) stop running for all project or project wiki repositories in the affected Gitaly Cluster. The bug was fixed by [reverting the change in GitLab 15.9.0](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/110823). Before upgrading to this version, check if you have any "unavailable" repositories. See [the bug issue](https://gitlab.com/gitlab-org/gitlab/-/issues/390155) for more information.
+
+### Geo installations **(PREMIUM SELF)**
+
+- We discovered an issue where [replication and verification of projects and wikis was not keeping up](https://gitlab.com/gitlab-org/gitlab/-/issues/387980) on small number of Geo installations. Your installation may be affected if you see some projects and/or wikis persistently in the "Queued" state for verification. This can lead to data loss after a failover.
+  - Affected versions: GitLab versions 15.6.x, 15.7.x, and 15.8.0 - 15.8.2.
+  - Versions containing fix: GitLab 15.8.3 and later.
+
+## 15.7.2
+
+- Due to [a bug introduced in GitLab 15.4](https://gitlab.com/gitlab-org/gitlab/-/issues/390155), if one or more Git repositories in Gitaly Cluster is [unavailable](../../administration/gitaly/recovery.md#unavailable-repositories), then [Repository checks](../../administration/repository_checks.md#repository-checks) and [Geo replication and verification](../../administration/geo/index.md) stop running for all project or project wiki repositories in the affected Gitaly Cluster. The bug was fixed by [reverting the change in GitLab 15.9.0](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/110823). Before upgrading to this version, check if you have any "unavailable" repositories. See [the bug issue](https://gitlab.com/gitlab-org/gitlab/-/issues/390155) for more information.
+
+### Geo installations **(PREMIUM SELF)**
+
+- [Container registry push events are rejected](https://gitlab.com/gitlab-org/gitlab/-/issues/386389) by the `/api/v4/container_registry_event/events` endpoint resulting in Geo secondary sites not being aware of updates to container registry images and subsequently not replicating the upgrades. Secondary sites may contain out of date container images after a failover as a consequence. This affects versions 15.6.0 - 15.6.6 and 15.7.0 - 15.7.2. If you're using Geo with container repositories, you are advised to upgrade to GitLab 15.6.7, 15.7.3, or 15.8.0 which contain a fix for this issue and avoid potential data loss after a failover.
+- We discovered an issue where [replication and verification of projects and wikis was not keeping up](https://gitlab.com/gitlab-org/gitlab/-/issues/387980) on small number of Geo installations. Your installation may be affected if you see some projects and/or wikis persistently in the "Queued" state for verification. This can lead to data loss after a failover.
+  - Affected versions: GitLab versions 15.6.x, 15.7.x, and 15.8.0 - 15.8.2.
+  - Versions containing fix: GitLab 15.8.3 and later.
+
+## 15.7.1
+
+- Due to [a bug introduced in GitLab 15.4](https://gitlab.com/gitlab-org/gitlab/-/issues/390155), if one or more Git repositories in Gitaly Cluster is [unavailable](../../administration/gitaly/recovery.md#unavailable-repositories), then [Repository checks](../../administration/repository_checks.md#repository-checks) and [Geo replication and verification](../../administration/geo/index.md) stop running for all project or project wiki repositories in the affected Gitaly Cluster. The bug was fixed by [reverting the change in GitLab 15.9.0](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/110823). Before upgrading to this version, check if you have any "unavailable" repositories. See [the bug issue](https://gitlab.com/gitlab-org/gitlab/-/issues/390155) for more information.
+
+### Geo installations **(PREMIUM SELF)**
+
+- [Container registry push events are rejected](https://gitlab.com/gitlab-org/gitlab/-/issues/386389) by the `/api/v4/container_registry_event/events` endpoint resulting in Geo secondary sites not being aware of updates to container registry images and subsequently not replicating the updates. Secondary sites may contain out of date container images after a failover as a consequence. This affects versions 15.6.0 - 15.6.6 and 15.7.0 - 15.7.2. If you're using Geo with container repositories, you are advised to upgrade to GitLab 15.6.7, 15.7.3, or 15.8.0 which contain a fix for this issue and avoid potential data loss after a failover.
+- We discovered an issue where [replication and verification of projects and wikis was not keeping up](https://gitlab.com/gitlab-org/gitlab/-/issues/387980) on small number of Geo installations. Your installation may be affected if you see some projects and/or wikis persistently in the "Queued" state for verification. This can lead to data loss after a failover.
+  - Affected versions: GitLab versions 15.6.x, 15.7.x, and 15.8.0 - 15.8.2.
+  - Versions containing fix: GitLab 15.8.3 and later.
+
+## 15.7.0
+
+- This version validates a `NOT NULL DB` constraint on the `issues.work_item_type_id` column.
+  To upgrade to this version, no records with a `NULL` `work_item_type_id` should exist on the `issues` table.
+  There are multiple `BackfillWorkItemTypeIdForIssues` background migrations that will be finalized with
+  the `EnsureWorkItemTypeBackfillMigrationFinished` post-deploy migration.
+- GitLab 15.4.0 introduced a [batched background migration](../background_migrations.md#batched-background-migrations) to
+  [backfill `namespace_id` values on issues table](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/91921). This
+  migration might take multiple hours or days to complete on larger GitLab instances. Make sure the migration
+  has completed successfully before upgrading to 15.7.0.
+- A database constraint is added, specifying that the `namespace_id` column on the issues
+  table has no `NULL` values.
+
+  - If the `namespace_id` batched background migration from 15.4 failed (see above) then the 15.7 upgrade
+    fails with a database migration error.
+
+  - On GitLab instances with large issues tables, validating this constraint causes the upgrade to take
+    longer than usual. All database changes need to complete within a one-hour limit:
+
+    ```plaintext
+    FATAL: Mixlib::ShellOut::CommandTimeout: rails_migration[gitlab-rails]
+    [..]
+    Mixlib::ShellOut::CommandTimeout: Command timed out after 3600s:
+    ```
+
+    A workaround exists to [complete the data change and the upgrade manually](../package/index.md#mixlibshelloutcommandtimeout-rails_migrationgitlab-rails--command-timed-out-after-3600s).
+- The default Sidekiq `max_concurrency` has been changed to 20. This is now
+  consistent in our documentation and product defaults.
+
+  For example, previously:
+
+  - Linux package installation default (`sidekiq['max_concurrency']`): 50
+  - Self-compiled installation default: 50
+  - Helm chart default (`gitlab.sidekiq.concurrency`): 25
+
+  Reference architectures still use a default of 10 as this is set specifically
+  for those configurations.
+
+  Sites that have configured `max_concurrency` will not be affected by this change.
+  [Read more about the Sidekiq concurrency setting](../../administration/sidekiq/extra_sidekiq_processes.md#concurrency).
+- GitLab Runner 15.7.0 introduced a breaking change that affects CI/CD jobs: [Correctly handle expansion of job file variables](https://gitlab.com/gitlab-org/gitlab-runner/-/merge_requests/3613).
+  Previously, job-defined variables that referred to
+  [file type variables](../../ci/variables/index.md#use-file-type-cicd-variables)
+  were expanded to the value of the file variable (its content). This behavior did not
+  respect the typical rules of shell variable expansion. There was also the potential
+  that secrets or sensitive information could leak if the file variable and its
+  contents printed. For example, if they were printed in an echo output. For more information,
+  see [Understanding the file type variable expansion change in GitLab 15.7](https://about.gitlab.com/blog/2023/02/13/impact-of-the-file-type-variable-change-15-7/).
+- Due to [a bug introduced in GitLab 15.4](https://gitlab.com/gitlab-org/gitlab/-/issues/390155), if one or more Git repositories in Gitaly Cluster is [unavailable](../../administration/gitaly/recovery.md#unavailable-repositories), then [Repository checks](../../administration/repository_checks.md#repository-checks) and [Geo replication and verification](../../administration/geo/index.md) stop running for all project or project wiki repositories in the affected Gitaly Cluster. The bug was fixed by [reverting the change in GitLab 15.9.0](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/110823). Before upgrading to this version, check if you have any "unavailable" repositories. See [the bug issue](https://gitlab.com/gitlab-org/gitlab/-/issues/390155) for more information.
+
+### Geo installations **(PREMIUM SELF)**
+
+- `pg_upgrade` fails to upgrade the bundled PostregSQL database to version 13. See
+  [the details and workaround](#pg_upgrade-fails-to-upgrade-the-bundled-postregsql-database-to-version-13).
+- [Container registry push events are rejected](https://gitlab.com/gitlab-org/gitlab/-/issues/386389) by the `/api/v4/container_registry_event/events` endpoint resulting in Geo secondary sites not being aware of updates to container registry images and subsequently not replicating the updates. Secondary sites may contain out of date container images after a failover as a consequence. This affects versions 15.6.0 - 15.6.6 and 15.7.0 - 15.7.2. If you're using Geo with container repositories, you are advised to upgrade to GitLab 15.6.7, 15.7.3, or 15.8.0 which contain a fix for this issue and avoid potential data loss after a failover.
+- We discovered an issue where [replication and verification of projects and wikis was not keeping up](https://gitlab.com/gitlab-org/gitlab/-/issues/387980) on small number of Geo installations. Your installation may be affected if you see some projects and/or wikis persistently in the "Queued" state for verification. This can lead to data loss after a failover.
+  - Affected versions: GitLab versions 15.6.x, 15.7.x, and 15.8.0 - 15.8.2.
+  - Versions containing fix: GitLab 15.8.3 and later.
+
+## 15.6.7
+
+- Due to [a bug introduced in GitLab 15.4](https://gitlab.com/gitlab-org/gitlab/-/issues/390155), if one or more Git repositories in Gitaly Cluster is [unavailable](../../administration/gitaly/recovery.md#unavailable-repositories), then [Repository checks](../../administration/repository_checks.md#repository-checks) and [Geo replication and verification](../../administration/geo/index.md) stop running for all project or project wiki repositories in the affected Gitaly Cluster. The bug was fixed by [reverting the change in GitLab 15.9.0](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/110823). Before upgrading to this version, check if you have any "unavailable" repositories. See [the bug issue](https://gitlab.com/gitlab-org/gitlab/-/issues/390155) for more information.
+
+### Geo installations **(PREMIUM SELF)**
+
+- We discovered an issue where [replication and verification of projects and wikis was not keeping up](https://gitlab.com/gitlab-org/gitlab/-/issues/387980) on small number of Geo installations. Your installation may be affected if you see some projects and/or wikis persistently in the "Queued" state for verification. This can lead to data loss after a failover.
+  - Affected versions: GitLab versions 15.6.x, 15.7.x, and 15.8.0 - 15.8.2.
+  - Versions containing fix: GitLab 15.8.3 and later.
+
+## 15.6.6
+
+- Due to [a bug introduced in GitLab 15.4](https://gitlab.com/gitlab-org/gitlab/-/issues/390155), if one or more Git repositories in Gitaly Cluster is [unavailable](../../administration/gitaly/recovery.md#unavailable-repositories), then [Repository checks](../../administration/repository_checks.md#repository-checks) and [Geo replication and verification](../../administration/geo/index.md) stop running for all project or project wiki repositories in the affected Gitaly Cluster. The bug was fixed by [reverting the change in GitLab 15.9.0](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/110823). Before upgrading to this version, check if you have any "unavailable" repositories. See [the bug issue](https://gitlab.com/gitlab-org/gitlab/-/issues/390155) for more information.
+
+### Geo installations **(PREMIUM SELF)**
+
+- [Container registry push events are rejected](https://gitlab.com/gitlab-org/gitlab/-/issues/386389) by the `/api/v4/container_registry_event/events` endpoint resulting in Geo secondary sites not being aware of updates to container registry images and subsequently not replicating the updates. Secondary sites may contain out of date container images after a failover as a consequence. This affects versions 15.6.0 - 15.6.6 and 15.7.0 - 15.7.2. If you're using Geo with container repositories, you are advised to upgrade to GitLab 15.6.7, 15.7.3, or 15.8.0 which contain a fix for this issue and avoid potential data loss after a failover.
+- We discovered an issue where [replication and verification of projects and wikis was not keeping up](https://gitlab.com/gitlab-org/gitlab/-/issues/387980) on small number of Geo installations. Your installation may be affected if you see some projects and/or wikis persistently in the "Queued" state for verification. This can lead to data loss after a failover.
+  - Affected versions: GitLab versions 15.6.x, 15.7.x, and 15.8.0 - 15.8.2.
+  - Versions containing fix: GitLab 15.8.3 and later.
+
+## 15.6.5
+
+### Geo installations **(PREMIUM SELF)**
+
+- [Container registry push events are rejected](https://gitlab.com/gitlab-org/gitlab/-/issues/386389) by the `/api/v4/container_registry_event/events` endpoint resulting in Geo secondary sites not being aware of updates to container registry images and subsequently not replicating the updates. Secondary sites may contain out of date container images after a failover as a consequence. This affects versions 15.6.0 - 15.6.6 and 15.7.0 - 15.7.2. If you're using Geo with container repositories, you are advised to upgrade to GitLab 15.6.7, 15.7.3, or 15.8.0 which contain a fix for this issue and avoid potential data loss after a failover.
+- We discovered an issue where [replication and verification of projects and wikis was not keeping up](https://gitlab.com/gitlab-org/gitlab/-/issues/387980) on small number of Geo installations. Your installation may be affected if you see some projects and/or wikis persistently in the "Queued" state for verification. This can lead to data loss after a failover.
+  - Affected versions: GitLab versions 15.6.x, 15.7.x, and 15.8.0 - 15.8.2.
+  - Versions containing fix: GitLab 15.8.3 and later.
+- Due to [a bug introduced in GitLab 15.4](https://gitlab.com/gitlab-org/gitlab/-/issues/390155), if one or more Git repositories in Gitaly Cluster is [unavailable](../../administration/gitaly/recovery.md#unavailable-repositories), then [Repository checks](../../administration/repository_checks.md#repository-checks) and [Geo replication and verification](../../administration/geo/index.md) stop running for all project or project wiki repositories in the affected Gitaly Cluster. The bug was fixed by [reverting the change in GitLab 15.9.0](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/110823). Before upgrading to this version, check if you have any "unavailable" repositories. See [the bug issue](https://gitlab.com/gitlab-org/gitlab/-/issues/390155) for more information.
+
+## 15.6.4
+
+### Geo installations **(PREMIUM SELF)**
+
+- [Container registry push events are rejected](https://gitlab.com/gitlab-org/gitlab/-/issues/386389) by the `/api/v4/container_registry_event/events` endpoint resulting in Geo secondary sites not being aware of updates to container registry images and subsequently not replicating the updates. Secondary sites may contain out of date container images after a failover as a consequence. This affects versions 15.6.0 - 15.6.6, and 15.7.0 - 15.7.2. If you're using Geo with container repositories, you are advised to upgrade to GitLab 15.6.7, 15.7.3, or 15.8.0 which contain a fix for this issue and avoid potential data loss after a failover.
+- We discovered an issue where [replication and verification of projects and wikis was not keeping up](https://gitlab.com/gitlab-org/gitlab/-/issues/387980) on small number of Geo installations. Your installation may be affected if you see some projects and/or wikis persistently in the "Queued" state for verification. This can lead to data loss after a failover.
+  - Affected versions: GitLab versions 15.6.x, 15.7.x, and 15.8.0 - 15.8.2.
+  - Versions containing fix: GitLab 15.8.3 and later.
+- Due to [a bug introduced in GitLab 15.4](https://gitlab.com/gitlab-org/gitlab/-/issues/390155), if one or more Git repositories in Gitaly Cluster is [unavailable](../../administration/gitaly/recovery.md#unavailable-repositories), then [Repository checks](../../administration/repository_checks.md#repository-checks) and [Geo replication and verification](../../administration/geo/index.md) stop running for all project or project wiki repositories in the affected Gitaly Cluster. The bug was fixed by [reverting the change in GitLab 15.9.0](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/110823). Before upgrading to this version, check if you have any "unavailable" repositories. See [the bug issue](https://gitlab.com/gitlab-org/gitlab/-/issues/390155) for more information.
+
+## 15.6.3
+
+### Geo installations **(PREMIUM SELF)**
+
+- [Container registry push events are rejected](https://gitlab.com/gitlab-org/gitlab/-/issues/386389) by the `/api/v4/container_registry_event/events` endpoint resulting in Geo secondary sites not being aware of updates to container registry images and subsequently not replicating the updates. Secondary sites may contain out of date container images after a failover as a consequence. This affects versions 15.6.0 - 15.6.6 and 15.7.0 - 15.7.2. If you're using Geo with container repositories, you are advised to upgrade to GitLab 15.6.7, 15.7.3, or 15.8.0 which contain a fix for this issue and avoid potential data loss after a failover.
+- We discovered an issue where [replication and verification of projects and wikis was not keeping up](https://gitlab.com/gitlab-org/gitlab/-/issues/387980) on small number of Geo installations. Your installation may be affected if you see some projects and/or wikis persistently in the "Queued" state for verification. This can lead to data loss after a failover.
+  - Affected versions: GitLab versions 15.6.x, 15.7.x, and 15.8.0 - 15.8.2.
+  - Versions containing fix: GitLab 15.8.3 and later.
+- Due to [a bug introduced in GitLab 15.4](https://gitlab.com/gitlab-org/gitlab/-/issues/390155), if one or more Git repositories in Gitaly Cluster is [unavailable](../../administration/gitaly/recovery.md#unavailable-repositories), then [Repository checks](../../administration/repository_checks.md#repository-checks) and [Geo replication and verification](../../administration/geo/index.md) stop running for all project or project wiki repositories in the affected Gitaly Cluster. The bug was fixed by [reverting the change in GitLab 15.9.0](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/110823). Before upgrading to this version, check if you have any "unavailable" repositories. See [the bug issue](https://gitlab.com/gitlab-org/gitlab/-/issues/390155) for more information.
+
+## 15.6.2
+
+### Geo installations **(PREMIUM SELF)**
+
+- [Container registry push events are rejected](https://gitlab.com/gitlab-org/gitlab/-/issues/386389) by the `/api/v4/container_registry_event/events` endpoint resulting in Geo secondary sites not being aware of updates to container registry images and subsequently not replicating the updates. Secondary sites may contain out of date container images after a failover as a consequence. This affects versions 15.6.0 - 15.6.6 and 15.7.0 - 15.7.2. If you're using Geo with container repositories, you are advised to upgrade to GitLab 15.6.7, 15.7.3, or 15.8.0 which contain a fix for this issue and avoid potential data loss after a failover.
+- We discovered an issue where [replication and verification of projects and wikis was not keeping up](https://gitlab.com/gitlab-org/gitlab/-/issues/387980) on small number of Geo installations. Your installation may be affected if you see some projects and/or wikis persistently in the "Queued" state for verification. This can lead to data loss after a failover.
+  - Affected versions: GitLab versions 15.6.x, 15.7.x, and 15.8.0 - 15.8.2.
+  - Versions containing fix: GitLab 15.8.3 and later.
+- Due to [a bug introduced in GitLab 15.4](https://gitlab.com/gitlab-org/gitlab/-/issues/390155), if one or more Git repositories in Gitaly Cluster is [unavailable](../../administration/gitaly/recovery.md#unavailable-repositories), then [Repository checks](../../administration/repository_checks.md#repository-checks) and [Geo replication and verification](../../administration/geo/index.md) stop running for all project or project wiki repositories in the affected Gitaly Cluster. The bug was fixed by [reverting the change in GitLab 15.9.0](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/110823). Before upgrading to this version, check if you have any "unavailable" repositories. See [the bug issue](https://gitlab.com/gitlab-org/gitlab/-/issues/390155) for more information.
+
+## 15.6.1
+
+### Geo installations **(PREMIUM SELF)**
+
+- [Container registry push events are rejected](https://gitlab.com/gitlab-org/gitlab/-/issues/386389) by the `/api/v4/container_registry_event/events` endpoint resulting in Geo secondary sites not being aware of updates to container registry images and subsequently not replicating the updates. Secondary sites may contain out of date container images after a failover as a consequence. This affects versions 15.6.0 - 15.6.6 and 15.7.0 - 15.7.2. If you're using Geo with container repositories, you are advised to upgrade to GitLab 15.6.7, 15.7.3, or 15.8.0 which contain a fix for this issue and avoid potential data loss after a failover.
+- We discovered an issue where [replication and verification of projects and wikis was not keeping up](https://gitlab.com/gitlab-org/gitlab/-/issues/387980) on small number of Geo installations. Your installation may be affected if you see some projects and/or wikis persistently in the "Queued" state for verification. This can lead to data loss after a failover.
+  - Affected versions: GitLab versions 15.6.x, 15.7.x, and 15.8.0 - 15.8.2.
+  - Versions containing fix: GitLab 15.8.3 and later.
+- Due to [a bug introduced in GitLab 15.4](https://gitlab.com/gitlab-org/gitlab/-/issues/390155), if one or more Git repositories in Gitaly Cluster is [unavailable](../../administration/gitaly/recovery.md#unavailable-repositories), then [Repository checks](../../administration/repository_checks.md#repository-checks) and [Geo replication and verification](../../administration/geo/index.md) stop running for all project or project wiki repositories in the affected Gitaly Cluster. The bug was fixed by [reverting the change in GitLab 15.9.0](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/110823). Before upgrading to this version, check if you have any "unavailable" repositories. See [the bug issue](https://gitlab.com/gitlab-org/gitlab/-/issues/390155) for more information.
+
+## 15.6.0
+
+- You should use one of the [officially supported PostgreSQL versions](../../administration/package_information/postgresql_versions.md). Some database migrations can cause stability and performance issues with older PostgreSQL versions.
+- Git 2.37.0 and later is required by Gitaly. For self-compiled installations, you should use the [Git version provided by Gitaly](../../install/installation.md#git).
+- A database change to modify the behavior of four indexes fails on instances
+  where these indexes do not exist:
+
+  ```plaintext
+  Caused by:
+  PG::UndefinedTable: ERROR:  relation "index_issues_on_title_trigram" does not exist
+  ```
+
+  The other three indexes are: `index_merge_requests_on_title_trigram`, `index_merge_requests_on_description_trigram`,
+  and `index_issues_on_description_trigram`.
+
+  This issue was [fixed in GitLab 15.7](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/105375) and backported
+  to GitLab 15.6.2. The issue can also be worked around:
+  [read about how to create these indexes](https://gitlab.com/gitlab-org/gitlab/-/issues/378343#note_1199863087).
+
+### Linux package installations
+
+In GitLab 15.6, the [PostgreSQL versions shipped with `omnibus-gitlab` packages](../../administration/package_information/postgresql_versions.md)
+have been upgraded to 12.12 and 13.8. Unless
+[explicitly opted out](https://docs.gitlab.com/omnibus/settings/database.html#automatic-restart-when-the-postgresql-version-changes),
+this can cause an automatic restart of the PostgreSQL service, and can
+potentially cause downtime.
+
+### Geo installations **(PREMIUM SELF)**
+
+- `pg_upgrade` fails to upgrade the bundled PostregSQL database to version 13. See
+  [the details and workaround](#pg_upgrade-fails-to-upgrade-the-bundled-postregsql-database-to-version-13).
+- [Container registry push events are rejected](https://gitlab.com/gitlab-org/gitlab/-/issues/386389) by the `/api/v4/container_registry_event/events` endpoint resulting in Geo secondary sites not being aware of updates to container registry images and subsequently not replicating the updates. Secondary sites may contain out of date container images after a failover as a consequence. This affects versions 15.6.0 - 15.6.6 and 15.7.0 - 15.7.2. If you're using Geo with container repositories, you are advised to upgrade to GitLab 15.6.7, 15.7.3, or 15.8.0 which contain a fix for this issue and avoid potential data loss after a failover.
+- We discovered an issue where [replication and verification of projects and wikis was not keeping up](https://gitlab.com/gitlab-org/gitlab/-/issues/387980) on small number of Geo installations. Your installation may be affected if you see some projects and/or wikis persistently in the "Queued" state for verification. This can lead to data loss after a failover.
+  - Affected versions: GitLab versions 15.6.x, 15.7.x, and 15.8.0 - 15.8.2.
+  - Versions containing fix: GitLab 15.8.3 and later.
+- Due to [a bug introduced in GitLab 15.4](https://gitlab.com/gitlab-org/gitlab/-/issues/390155), if one or more Git repositories in Gitaly Cluster is [unavailable](../../administration/gitaly/recovery.md#unavailable-repositories), then [Repository checks](../../administration/repository_checks.md#repository-checks) and [Geo replication and verification](../../administration/geo/index.md) stop running for all project or project wiki repositories in the affected Gitaly Cluster. The bug was fixed by [reverting the change in GitLab 15.9.0](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/110823). Before upgrading to this version, check if you have any "unavailable" repositories. See [the bug issue](https://gitlab.com/gitlab-org/gitlab/-/issues/390155) for more information.
+
+## 15.5.5
+
+- Due to [a bug introduced in GitLab 15.4](https://gitlab.com/gitlab-org/gitlab/-/issues/390155), if one or more Git repositories in Gitaly Cluster is [unavailable](../../administration/gitaly/recovery.md#unavailable-repositories), then [Repository checks](../../administration/repository_checks.md#repository-checks) and [Geo replication and verification](../../administration/geo/index.md) stop running for all project or project wiki repositories in the affected Gitaly Cluster. The bug was fixed by [reverting the change in GitLab 15.9.0](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/110823). Before upgrading to this version, check if you have any "unavailable" repositories. See [the bug issue](https://gitlab.com/gitlab-org/gitlab/-/issues/390155) for more information.
+
+## 15.5.4
+
+- Due to [a bug introduced in GitLab 15.4](https://gitlab.com/gitlab-org/gitlab/-/issues/390155), if one or more Git repositories in Gitaly Cluster is [unavailable](../../administration/gitaly/recovery.md#unavailable-repositories), then [Repository checks](../../administration/repository_checks.md#repository-checks) and [Geo replication and verification](../../administration/geo/index.md) stop running for all project or project wiki repositories in the affected Gitaly Cluster. The bug was fixed by [reverting the change in GitLab 15.9.0](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/110823). Before upgrading to this version, check if you have any "unavailable" repositories. See [the bug issue](https://gitlab.com/gitlab-org/gitlab/-/issues/390155) for more information.
+
+## 15.5.3
+
+- GitLab 15.4.0 introduced a default [Sidekiq routing rule](../../administration/sidekiq/processing_specific_job_classes.md#routing-rules) that routes all jobs to the `default` queue. For instances using [queue selectors](../../administration/sidekiq/processing_specific_job_classes.md#queue-selectors-deprecated), this causes [performance problems](https://gitlab.com/gitlab-com/gl-infra/scalability/-/issues/1991) as some Sidekiq processes will be idle.
+  - The default routing rule has been reverted in 15.5.4, so upgrading to that version or later will return to the previous behavior.
+  - If a GitLab instance now listens only to the `default` queue (which is not currently recommended), it will be required to add this routing rule back in `/etc/gitlab/gitlab.rb`:
+
+    ```ruby
+    sidekiq['routing_rules'] = [['*', 'default']]
+    ```
+
+- Due to [a bug introduced in GitLab 15.4](https://gitlab.com/gitlab-org/gitlab/-/issues/390155), if one or more Git repositories in Gitaly Cluster is [unavailable](../../administration/gitaly/recovery.md#unavailable-repositories), then [Repository checks](../../administration/repository_checks.md#repository-checks) and [Geo replication and verification](../../administration/geo/index.md) stop running for all project or project wiki repositories in the affected Gitaly Cluster. The bug was fixed by [reverting the change in GitLab 15.9.0](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/110823). Before upgrading to this version, check if you have any "unavailable" repositories. See [the bug issue](https://gitlab.com/gitlab-org/gitlab/-/issues/390155) for more information.
+
+## 15.5.2
+
+- GitLab 15.4.0 introduced a default [Sidekiq routing rule](../../administration/sidekiq/processing_specific_job_classes.md#routing-rules) that routes all jobs to the `default` queue. For instances using [queue selectors](../../administration/sidekiq/processing_specific_job_classes.md#queue-selectors-deprecated), this causes [performance problems](https://gitlab.com/gitlab-com/gl-infra/scalability/-/issues/1991) as some Sidekiq processes will be idle.
+  - The default routing rule has been reverted in 15.5.4, so upgrading to that version or later will return to the previous behavior.
+  - If a GitLab instance now listens only to the `default` queue (which is not currently recommended), it will be required to add this routing rule back in `/etc/gitlab/gitlab.rb`:
+
+    ```ruby
+    sidekiq['routing_rules'] = [['*', 'default']]
+    ```
+
+- Due to [a bug introduced in GitLab 15.4](https://gitlab.com/gitlab-org/gitlab/-/issues/390155), if one or more Git repositories in Gitaly Cluster is [unavailable](../../administration/gitaly/recovery.md#unavailable-repositories), then [Repository checks](../../administration/repository_checks.md#repository-checks) and [Geo replication and verification](../../administration/geo/index.md) stop running for all project or project wiki repositories in the affected Gitaly Cluster. The bug was fixed by [reverting the change in GitLab 15.9.0](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/110823). Before upgrading to this version, check if you have any "unavailable" repositories. See [the bug issue](https://gitlab.com/gitlab-org/gitlab/-/issues/390155) for more information.
+
+## 15.5.1
+
+- GitLab 15.4.0 introduced a default [Sidekiq routing rule](../../administration/sidekiq/processing_specific_job_classes.md#routing-rules) that routes all jobs to the `default` queue. For instances using [queue selectors](../../administration/sidekiq/processing_specific_job_classes.md#queue-selectors-deprecated), this causes [performance problems](https://gitlab.com/gitlab-com/gl-infra/scalability/-/issues/1991) as some Sidekiq processes will be idle.
+  - The default routing rule has been reverted in 15.5.4, so upgrading to that version or later will return to the previous behavior.
+  - If a GitLab instance now listens only to the `default` queue (which is not currently recommended), it will be required to add this routing rule back in `/etc/gitlab/gitlab.rb`:
+
+    ```ruby
+    sidekiq['routing_rules'] = [['*', 'default']]
+    ```
+
+- Due to [a bug introduced in GitLab 15.4](https://gitlab.com/gitlab-org/gitlab/-/issues/390155), if one or more Git repositories in Gitaly Cluster is [unavailable](../../administration/gitaly/recovery.md#unavailable-repositories), then [Repository checks](../../administration/repository_checks.md#repository-checks) and [Geo replication and verification](../../administration/geo/index.md) stop running for all project or project wiki repositories in the affected Gitaly Cluster. The bug was fixed by [reverting the change in GitLab 15.9.0](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/110823). Before upgrading to this version, check if you have any "unavailable" repositories. See [the bug issue](https://gitlab.com/gitlab-org/gitlab/-/issues/390155) for more information.
+
+## 15.5.0
+
+- GitLab 15.4.0 introduced a default [Sidekiq routing rule](../../administration/sidekiq/processing_specific_job_classes.md#routing-rules) that routes all jobs to the `default` queue. For instances using [queue selectors](../../administration/sidekiq/processing_specific_job_classes.md#queue-selectors-deprecated), this causes [performance problems](https://gitlab.com/gitlab-com/gl-infra/scalability/-/issues/1991) as some Sidekiq processes will be idle.
+  - The default routing rule has been reverted in 15.5.4, so upgrading to that version or later will return to the previous behavior.
+  - If a GitLab instance now listens only to the `default` queue (which is not currently recommended), it will be required to add this routing rule back in `/etc/gitlab/gitlab.rb`:
+
+    ```ruby
+    sidekiq['routing_rules'] = [['*', 'default']]
+    ```
+
+- Due to [a bug introduced in GitLab 15.4](https://gitlab.com/gitlab-org/gitlab/-/issues/390155), if one or more Git repositories in Gitaly Cluster is [unavailable](../../administration/gitaly/recovery.md#unavailable-repositories), then [Repository checks](../../administration/repository_checks.md#repository-checks) and [Geo replication and verification](../../administration/geo/index.md) stop running for all project or project wiki repositories in the affected Gitaly Cluster. The bug was fixed by [reverting the change in GitLab 15.9.0](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/110823). Before upgrading to this version, check if you have any "unavailable" repositories. See [the bug issue](https://gitlab.com/gitlab-org/gitlab/-/issues/390155) for more information.
+
+### Geo installations **(PREMIUM SELF)**
+
+- `pg_upgrade` fails to upgrade the bundled PostregSQL database to version 13. See
+  [the details and workaround](#pg_upgrade-fails-to-upgrade-the-bundled-postregsql-database-to-version-13).
+
+## 15.4.6
+
+- Due to a [bug introduced in curl in GitLab 15.4.6](https://github.com/curl/curl/issues/10122), the [`no_proxy` environment variable may not work properly](../../administration/geo/replication/troubleshooting.md#secondary-site-returns-received-http-code-403-from-proxy-after-connect). Either downgrade to GitLab 15.4.5, or upgrade to GitLab 15.5.7 or a later version.
+- Due to [a bug introduced in GitLab 15.4](https://gitlab.com/gitlab-org/gitlab/-/issues/390155), if one or more Git repositories in Gitaly Cluster is [unavailable](../../administration/gitaly/recovery.md#unavailable-repositories), then [Repository checks](../../administration/repository_checks.md#repository-checks) and [Geo replication and verification](../../administration/geo/index.md) stop running for all project or project wiki repositories in the affected Gitaly Cluster. The bug was fixed by [reverting the change in GitLab 15.9.0](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/110823). Before upgrading to this version, check if you have any "unavailable" repositories. See [the bug issue](https://gitlab.com/gitlab-org/gitlab/-/issues/390155) for more information.
+
+## 15.4.5
+
+- Due to [a bug introduced in GitLab 15.4](https://gitlab.com/gitlab-org/gitlab/-/issues/390155), if one or more Git repositories in Gitaly Cluster is [unavailable](../../administration/gitaly/recovery.md#unavailable-repositories), then [Repository checks](../../administration/repository_checks.md#repository-checks) and [Geo replication and verification](../../administration/geo/index.md) stop running for all project or project wiki repositories in the affected Gitaly Cluster. The bug was fixed by [reverting the change in GitLab 15.9.0](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/110823). Before upgrading to this version, check if you have any "unavailable" repositories. See [the bug issue](https://gitlab.com/gitlab-org/gitlab/-/issues/390155) for more information.
+
+## 15.4.4
+
+- Due to [a bug introduced in GitLab 15.4](https://gitlab.com/gitlab-org/gitlab/-/issues/390155), if one or more Git repositories in Gitaly Cluster is [unavailable](../../administration/gitaly/recovery.md#unavailable-repositories), then [Repository checks](../../administration/repository_checks.md#repository-checks) and [Geo replication and verification](../../administration/geo/index.md) stop running for all project or project wiki repositories in the affected Gitaly Cluster. The bug was fixed by [reverting the change in GitLab 15.9.0](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/110823). Before upgrading to this version, check if you have any "unavailable" repositories. See [the bug issue](https://gitlab.com/gitlab-org/gitlab/-/issues/390155) for more information.
+
+## 15.4.3
+
+- Due to [a bug introduced in GitLab 15.4](https://gitlab.com/gitlab-org/gitlab/-/issues/390155), if one or more Git repositories in Gitaly Cluster is [unavailable](../../administration/gitaly/recovery.md#unavailable-repositories), then [Repository checks](../../administration/repository_checks.md#repository-checks) and [Geo replication and verification](../../administration/geo/index.md) stop running for all project or project wiki repositories in the affected Gitaly Cluster. The bug was fixed by [reverting the change in GitLab 15.9.0](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/110823). Before upgrading to this version, check if you have any "unavailable" repositories. See [the bug issue](https://gitlab.com/gitlab-org/gitlab/-/issues/390155) for more information.
+
+## 15.4.2
+
+- A [license caching issue](https://gitlab.com/gitlab-org/gitlab/-/issues/376706) prevents some premium features of GitLab from working correctly if you add a new license. Workarounds for this issue:
+  - Restart all Rails, Sidekiq and Gitaly nodes after applying a new license. This clears the relevant license caches and allows all premium features to operate correctly.
+  - Upgrade to a version that is not affected by this issue. The following upgrade paths are available for affected versions:
+    - 15.2.5 --> 15.3.5
+    - 15.3.0 - 15.3.4 --> 15.3.5
+    - 15.4.1 --> 15.4.3
+- Due to [a bug introduced in GitLab 15.4](https://gitlab.com/gitlab-org/gitlab/-/issues/390155), if one or more Git repositories in Gitaly Cluster is [unavailable](../../administration/gitaly/recovery.md#unavailable-repositories), then [Repository checks](../../administration/repository_checks.md#repository-checks) and [Geo replication and verification](../../administration/geo/index.md) stop running for all project or project wiki repositories in the affected Gitaly Cluster. The bug was fixed by [reverting the change in GitLab 15.9.0](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/110823). Before upgrading to this version, check if you have any "unavailable" repositories. See [the bug issue](https://gitlab.com/gitlab-org/gitlab/-/issues/390155) for more information.
+
+## 15.4.1
+
+- A [license caching issue](https://gitlab.com/gitlab-org/gitlab/-/issues/376706) prevents some premium features of GitLab from working correctly if you add a new license. Workarounds for this issue:
+  - Restart all Rails, Sidekiq and Gitaly nodes after applying a new license. This clears the relevant license caches and allows all premium features to operate correctly.
+  - Upgrade to a version that is not affected by this issue. The following upgrade paths are available for affected versions:
+    - 15.2.5 --> 15.3.5
+    - 15.3.0 - 15.3.4 --> 15.3.5
+    - 15.4.1 --> 15.4.3
+- Due to [a bug introduced in GitLab 15.4](https://gitlab.com/gitlab-org/gitlab/-/issues/390155), if one or more Git repositories in Gitaly Cluster is [unavailable](../../administration/gitaly/recovery.md#unavailable-repositories), then [Repository checks](../../administration/repository_checks.md#repository-checks) and [Geo replication and verification](../../administration/geo/index.md) stop running for all project or project wiki repositories in the affected Gitaly Cluster. The bug was fixed by [reverting the change in GitLab 15.9.0](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/110823). Before upgrading to this version, check if you have any "unavailable" repositories. See [the bug issue](https://gitlab.com/gitlab-org/gitlab/-/issues/390155) for more information.
+
+## 15.4.0
+
+- GitLab 15.4.0 includes a [batched background migration](../background_migrations.md#batched-background-migrations) to [remove incorrect values from `expire_at` in `ci_job_artifacts` table](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/89318).
+  This migration might take hours or days to complete on larger GitLab instances.
+- By default, Gitaly and Praefect nodes use the time server at `pool.ntp.org`. If your instance can not connect to `pool.ntp.org`, [configure the `NTP_HOST` variable](../../administration/gitaly/praefect.md#customize-time-server-setting).
+- GitLab 15.4.0 introduced a default [Sidekiq routing rule](../../administration/sidekiq/processing_specific_job_classes.md#routing-rules) that routes all jobs to the `default` queue. For instances using [queue selectors](../../administration/sidekiq/processing_specific_job_classes.md#queue-selectors-deprecated), this causes [performance problems](https://gitlab.com/gitlab-com/gl-infra/scalability/-/issues/1991) as some Sidekiq processes will be idle.
+  - The default routing rule has been reverted in 15.4.5, so upgrading to that version or later will return to the previous behavior.
+  - If a GitLab instance now listens only to the `default` queue (which is not currently recommended), it will be required to add this routing rule back in `/etc/gitlab/gitlab.rb`:
+
+    ```ruby
+    sidekiq['routing_rules'] = [['*', 'default']]
+    ```
+
+- The structure of `/etc/gitlab/gitlab-secrets.json` was modified in [GitLab 15.4](https://gitlab.com/gitlab-org/omnibus-gitlab/-/merge_requests/6310),
+  and new configuration was added to `gitlab_pages`, `grafana`, and `mattermost` sections.
+  In a highly available or GitLab Geo environment, secrets need to be the same on all nodes.
+  If you're manually syncing the secrets file across nodes, or manually specifying secrets in
+  `/etc/gitlab/gitlab.rb`, make sure `/etc/gitlab/gitlab-secrets.json` is the same on all nodes.
+- GitLab 15.4.0 introduced a [batched background migration](../background_migrations.md#batched-background-migrations) to
+  [backfill `namespace_id` values on issues table](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/91921). This
+  migration might take multiple hours or days to complete on larger GitLab instances. Make sure the migration
+  has completed successfully before upgrading to 15.7.0 or later.
+- Due to [a bug introduced in GitLab 15.4](https://gitlab.com/gitlab-org/gitlab/-/issues/390155), if one or more Git repositories in Gitaly Cluster is [unavailable](../../administration/gitaly/recovery.md#unavailable-repositories), then [Repository checks](../../administration/repository_checks.md#repository-checks) and [Geo replication and verification](../../administration/geo/index.md) stop running for all project or project wiki repositories in the affected Gitaly Cluster. The bug was fixed by [reverting the change in GitLab 15.9.0](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/110823). Before upgrading to this version, check if you have any "unavailable" repositories. See [the bug issue](https://gitlab.com/gitlab-org/gitlab/-/issues/390155) for more information.
+- A redesigned sign-in page is enabled by default in GitLab 15.4 and later, with improvements shipping in later releases. For more information, see [epic 8557](https://gitlab.com/groups/gitlab-org/-/epics/8557).
+  It can be disabled with a feature flag. Start [a Rails console](../../administration/operations/rails_console.md) and run:
+
+  ```ruby
+  Feature.disable(:restyle_login_page)
+  ```
+
+### Geo installations **(PREMIUM SELF)**
+
+- `pg_upgrade` fails to upgrade the bundled PostregSQL database to version 13. See
+  [the details and workaround](#pg_upgrade-fails-to-upgrade-the-bundled-postregsql-database-to-version-13).
+
+## 15.3.4
+
+A [license caching issue](https://gitlab.com/gitlab-org/gitlab/-/issues/376706) prevents some premium features of GitLab from working correctly if you add a new license. Workarounds for this issue:
+
+- Restart all Rails, Sidekiq and Gitaly nodes after applying a new license. This clears the relevant license caches and allows all premium features to operate correctly.
+- Upgrade to a version that is not affected by this issue. The following upgrade paths are available for affected versions:
+  - 15.2.5 --> 15.3.5
+  - 15.3.0 - 15.3.4 --> 15.3.5
+  - 15.4.1 --> 15.4.3
+
+## 15.3.3
+
+- In GitLab 15.3.3, [SAML Group Links](../../api/groups.md#saml-group-links) API `access_level` attribute type changed to `integer`. See
+[the API documentation](../../api/members.md).
+- A [license caching issue](https://gitlab.com/gitlab-org/gitlab/-/issues/376706) prevents some premium features of GitLab from working correctly if you add a new license. Workarounds for this issue:
+
+  - Restart all Rails, Sidekiq and Gitaly nodes after applying a new license. This clears the relevant license caches and allows all premium features to operate correctly.
+  - Upgrade to a version that is not affected by this issue. The following upgrade paths are available for affected versions:
+    - 15.2.5 --> 15.3.5
+    - 15.3.0 - 15.3.4 --> 15.3.5
+    - 15.4.1 --> 15.4.3
+
+## 15.3.2
+
+A [license caching issue](https://gitlab.com/gitlab-org/gitlab/-/issues/376706) prevents some premium features of GitLab from working correctly if you add a new license. Workarounds for this issue:
+
+- Restart all Rails, Sidekiq and Gitaly nodes after applying a new license. This clears the relevant license caches and allows all premium features to operate correctly.
+- Upgrade to a version that is not affected by this issue. The following upgrade paths are available for affected versions:
+  - 15.2.5 --> 15.3.5
+  - 15.3.0 - 15.3.4 --> 15.3.5
+  - 15.4.1 --> 15.4.3
+
+## 15.3.1
+
+A [license caching issue](https://gitlab.com/gitlab-org/gitlab/-/issues/376706) prevents some premium features of GitLab from working correctly if you add a new license. Workarounds for this issue:
+
+- Restart all Rails, Sidekiq and Gitaly nodes after applying a new license. This clears the relevant license caches and allows all premium features to operate correctly.
+- Upgrade to a version that is not affected by this issue. The following upgrade paths are available for affected versions:
+  - 15.2.5 --> 15.3.5
+  - 15.3.0 - 15.3.4 --> 15.3.5
+  - 15.4.1 --> 15.4.3
+
+## 15.3.0
+
+- New Git repositories created in Gitaly cluster no longer use the `@hashed`
+  storage path. Server hooks for new repositories must be copied into a
+  different location. Praefect now generates replica paths for use by Gitaly
+  cluster. This change is a pre-requisite for Gitaly cluster atomically
+  creating, deleting, and renaming Git repositories.
+
+  To identify the replica path,
+  [query the Praefect repository metadata](../../administration/gitaly/troubleshooting.md#view-repository-metadata)
+  and pass the `@hashed` storage path to `-relative-path`.
+
+  With this information, you can correctly install
+  [server hooks](../../administration/server_hooks.md).
+
+- A [license caching issue](https://gitlab.com/gitlab-org/gitlab/-/issues/376706) prevents some premium features of GitLab from working correctly if you add a new license. Workarounds for this issue:
+
+  - Restart all Rails, Sidekiq and Gitaly nodes after applying a new license. This clears the relevant license caches and allows all premium features to operate correctly.
+  - Upgrade to a version that is not affected by this issue. The following upgrade paths are available for affected versions:
+    - 15.2.5 --> 15.3.5
+    - 15.3.0 - 15.3.4 --> 15.3.5
+    - 15.4.1 --> 15.4.3
+
+### Geo installations **(PREMIUM SELF)**
+
+- `pg_upgrade` fails to upgrade the bundled PostregSQL database to version 13. See
+  [the details and workaround](#pg_upgrade-fails-to-upgrade-the-bundled-postregsql-database-to-version-13).
+- LFS transfers can redirect to the primary from secondary site mid-session. See
+  [the details and workaround](#lfs-transfers-redirect-to-primary-from-secondary-site-mid-session).
+- Incorrect object storage LFS files deletion on Geo secondary sites. See
+  [the details and workaround](#incorrect-object-storage-lfs-file-deletion-on-secondary-sites).
+
+#### LFS transfers redirect to primary from secondary site mid-session
+
+| Affected minor releases | Affected patch releases | Fixed in |
+|-------------------------|-------------------------|----------|
+| 15.1                    | All                     | None     |
+| 15.2                    | All                     | None     |
+| 15.3                    | 15.3.0 - 15.3.2         | 15.3.3 and later |
+
+LFS transfers can [redirect to the primary from secondary site mid-session](https://gitlab.com/gitlab-org/gitlab/-/issues/371571) causing failed pull and clone requests in GitLab 15.1.0 to 15.3.2 when [Geo proxying](../../administration/geo/secondary_proxy/index.md) is enabled. Geo proxying is enabled by default in GitLab 15.1 and later.
+
+This issue is resolved in GitLab 15.3.3, so customers with the following configuration should upgrade to 15.3.3 or later:
+
+- LFS is enabled.
+- LFS objects are being replicated across Geo sites.
+- Repositories are being pulled by using a Geo secondary site.
+
+#### Incorrect object storage LFS file deletion on secondary sites
+
+| Affected minor releases | Affected patch releases | Fixed in |
+|-------------------------|-------------------------|----------|
+| 15.0                    | All                     | None     |
+| 15.1                    | All                     | None     |
+| 15.2                    | All                     | None     |
+| 15.3                    | 15.3.0 - 15.3.2         | 15.3.3 and later |
+
+[Incorrect deletion of object storage files on Geo secondary sites](https://gitlab.com/gitlab-org/gitlab/-/issues/371397)
+can occur in GitLab 15.0.0 to 15.3.2 in the following situations:
+
+- GitLab-managed object storage replication is disabled, and LFS objects are created while importing a project with object storage enabled.
+- GitLab-managed replication to sync object storage is enabled and subsequently disabled.
+
+This issue is resolved in 15.3.3. Customers who have both LFS enabled and LFS objects being replicated across Geo sites
+should upgrade directly to 15.3.3 to reduce the risk of data loss on secondary sites.
+
+## 15.2.5
+
+A [license caching issue](https://gitlab.com/gitlab-org/gitlab/-/issues/376706) prevents some premium features of GitLab from working correctly if you add a new license. Workarounds for this issue:
+
+- Restart all Rails, Sidekiq and Gitaly nodes after applying a new license. This clears the relevant license caches and allows all premium features to operate correctly.
+- Upgrade to a version that is not affected by this issue. The following upgrade paths are available for affected versions:
+  - 15.2.5 --> 15.3.5
+  - 15.3.0 - 15.3.4 --> 15.3.5
+  - 15.4.1 --> 15.4.3
+
+## 15.2.0
+
+- GitLab installations that have multiple web nodes should be
+  [upgraded to 15.1](#1510) before upgrading to 15.2 (and later) due to a
+  configuration change in Rails that can result in inconsistent ETag key
+  generation.
+- Some Sidekiq workers were renamed in this release. To avoid any disruption, [run the Rake tasks to migrate any pending jobs](../../administration/sidekiq/sidekiq_job_migration.md#migrate-queued-and-future-jobs) before starting the upgrade to GitLab 15.2.0.
+- Gitaly now executes its binaries in a [runtime location](https://gitlab.com/gitlab-org/gitaly/-/merge_requests/4670). By default on Omnibus GitLab,
+  this path is `/var/opt/gitlab/gitaly/run/`. If this location is mounted with `noexec`, merge requests generate the following error:
+
+  ```plaintext
+  fork/exec /var/opt/gitlab/gitaly/run/gitaly-<nnnn>/gitaly-git2go-v15: permission denied
+  ```
+
+  To resolve this, remove the `noexec` option from the file system mount. An alternative is to change the Gitaly runtime directory:
+
+  1. Add `gitaly['runtime_dir'] = '<PATH_WITH_EXEC_PERM>'` to `/etc/gitlab/gitlab.rb` and specify a location without `noexec` set.
+  1. Run `sudo gitlab-ctl reconfigure`.
+
+### Geo installations **(PREMIUM SELF)**
+
+- `pg_upgrade` fails to upgrade the bundled PostregSQL database to version 13. See
+  [the details and workaround](#pg_upgrade-fails-to-upgrade-the-bundled-postregsql-database-to-version-13).
+- LFS transfers can redirect to the primary from secondary site mid-session. See
+  [the details and workaround](#lfs-transfers-redirect-to-primary-from-secondary-site-mid-session).
+- Incorrect object storage LFS files deletion on Geo secondary sites. See
+  [the details and workaround](#incorrect-object-storage-lfs-file-deletion-on-secondary-sites).
+
+## 15.1.0
+
+- In GitLab 15.1.0, we are switching Rails `ActiveSupport::Digest` to use SHA256 instead of MD5.
+  This affects ETag key generation for resources such as raw Snippet file
+  downloads. To ensure consistent ETag key generation across multiple
+  web nodes when upgrading, all servers must first be upgraded to 15.1.6 before
+  upgrading to 15.2.0 or later:
+
+  1. Ensure all GitLab web nodes are running GitLab 15.1.6.
+  1. If you run [GitLab on Kubernetes](https://docs.gitlab.com/charts/installation/) by using the cloud native GitLab Helm chart, make sure that all
+     webservice pods are running GitLab 15.1.Z:
+
+     ```shell
+     kubectl get pods -l app=webservice -o custom-columns=webservice-image:{.spec.containers[0].image},workhorse-image:{.spec.containers[1].image}
+     ```
+
+  1. [Enable the `active_support_hash_digest_sha256` feature flag](../../administration/feature_flags.md#how-to-enable-and-disable-features-behind-flags) to switch `ActiveSupport::Digest` to use SHA256:
+
+     1. [Start the rails console](../../administration/operations/rails_console.md)
+     1. Enable the feature flag:
+
+        ```ruby
+        Feature.enable(:active_support_hash_digest_sha256)
+        ```
+
+  1. Only then, continue to upgrade to later versions of GitLab.
+- Unauthenticated requests to the [`ciConfig` GraphQL field](../../api/graphql/reference/index.md#queryciconfig) are no longer supported.
+  Before you upgrade to GitLab 15.1, add an [access token](../../api/rest/index.md#authentication) to your requests.
+  The user creating the token must have [permission](../../user/permissions.md) to create pipelines in the project.
+
+### Geo installations **(PREMIUM SELF)**
+
+- [Geo proxying](../../administration/geo/secondary_proxy/index.md) was [enabled by default for different URLs](https://gitlab.com/gitlab-org/gitlab/-/issues/346112) in 15.1. This may be a breaking change. If needed, you may [disable Geo proxying](../../administration/geo/secondary_proxy/index.md#disable-geo-proxying). If you are using SAML with different URLs, you must modify your SAML configuration and your Identity Provider configuration. For more information, see the [Geo with Single Sign-On (SSO) documentation](../../administration/geo/replication/single_sign_on.md).
+- LFS transfers can redirect to the primary from secondary site mid-session. See
+  [the details and workaround](#lfs-transfers-redirect-to-primary-from-secondary-site-mid-session).
+- Incorrect object storage LFS files deletion on Geo secondary sites. See
+  [the details and workaround](#incorrect-object-storage-lfs-file-deletion-on-secondary-sites).
+
+## 15.0.0
+
+- Elasticsearch 6.8 [is no longer supported](../../integration/advanced_search/elasticsearch.md#version-requirements). Before you upgrade to GitLab 15.0, [update Elasticsearch to any 7.x version](../../integration/advanced_search/elasticsearch.md#upgrade-to-a-new-elasticsearch-major-version).
+- If you run GitLab with external PostgreSQL, particularly AWS RDS, ensure you
+  upgrade PostgreSQL to patch levels to a minimum of 12.7 or 13.3 before
+  upgrading to GitLab 14.8 or later.
+
+  [In 14.8](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/75511)
+  for GitLab Enterprise Edition and [in 15.0](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/87983)
+  for GitLab Community Edition a GitLab feature called Loose Foreign Keys was enabled.
+
+  After it was enabled, we have had reports of unplanned PostgreSQL restarts caused
+  by a database engine bug that causes a segmentation fault.
+
+  For more information, see [issue 364763](https://gitlab.com/gitlab-org/gitlab/-/issues/364763).
+
+- The use of encrypted S3 buckets with storage-specific configuration is no longer supported after [removing support for using `background_upload`](../deprecations.md#background-upload-for-object-storage).
+- The [certificate-based Kubernetes integration (DEPRECATED)](../../user/infrastructure/clusters/index.md#certificate-based-kubernetes-integration-deprecated) is disabled by default, but you can be re-enable it through the [`certificate_based_clusters` feature flag](../../administration/feature_flags.md#how-to-enable-and-disable-features-behind-flags) until GitLab 16.0.
+- When you use the GitLab Helm Chart project with a custom `serviceAccount`, ensure it has `get` and `list` permissions for the `serviceAccount` and `secret` resources.
+- The `FF_GITLAB_REGISTRY_HELPER_IMAGE` [feature flag](../../administration/feature_flags.md#enable-or-disable-the-feature) is removed and helper images are always pulled from GitLab Registry.
+
+### Linux package installations
+
+- The [`custom_hooks_dir`](../../administration/server_hooks.md#create-global-server-hooks-for-all-repositories) setting for configuring global server hooks is now configured in
+  Gitaly. The previous implementation in GitLab Shell was removed in GitLab 15.0. With this change, global server hooks are stored only inside a subdirectory named after the
+  hook type. Global server hooks can no longer be a single hook file in the root of the custom hooks directory. For example, you must use `<custom_hooks_dir>/<hook_name>.d/*` rather
+  than `<custom_hooks_dir>/<hook_name>`.
+  - Use `gitaly['custom_hooks_dir']` in `gitlab.rb` ([introduced in 14.3](https://gitlab.com/gitlab-org/omnibus-gitlab/-/merge_requests/4208))
+    for Omnibus GitLab. This replaces `gitlab_shell['custom_hooks_dir']`.
+- PostgreSQL 13.6 is being shipped as the default version for fresh installs and
+  12.10 for upgrades. You can manually upgrade to PostgreSQL 13.6 following the
+  [upgrade docs](https://docs.gitlab.com/omnibus/settings/database.html#gitlab-150-and-later).
+  Because of underlying structural changes, the running PostgreSQL process
+  **_must_** be restarted when it is upgraded before running database
+  migrations. If automatic restart is skipped, you must run the following
+  command before migrations are run:
+
+  ```shell
+  # If using PostgreSQL
+  sudo gitlab-ctl restart postgresql
+
+  # If using Patroni for Database replication
+  sudo gitlab-ctl restart patroni
+  ```
+
+  If PostgreSQL is not restarted, you might face
+  [errors related to loading libraries](https://docs.gitlab.com/omnibus/settings/database.html#could-not-load-library-plpgsqlso).
+
+- Starting with GitLab 15.0, `postgresql` and `geo-postgresql` services are
+  automatically restarted when the PostgreSQL version changes. Restarting
+  PostgreSQL services causes downtime due to the temporary unavailability of the
+  database for operations. While this restart is mandatory for proper functioning
+  of the Database services, you might want more control over when the PostgreSQL
+  is restarted. For that purpose, you can choose to skip the automatic restarts as
+  part of `gitlab-ctl reconfigure` and manually restart the services.
+
+  To skip automatic restarts as part of GitLab 15.0 upgrade, perform the following
+  steps before the upgrade:
+
+  1. Edit `/etc/gitlab/gitlab.rb` and add the following line:
+
+     ```ruby
+     # For PostgreSQL/Patroni
+     postgresql['auto_restart_on_version_change'] = false
+
+     # For Geo PostgreSQL
+     geo_postgresql['auto_restart_on_version_change'] = false
+     ```
+
+  1. Reconfigure GitLab:
+
+     ```shell
+     sudo gitlab-ctl reconfigure
+     ```
+
+  NOTE:
+  It is mandatory to restart PostgreSQL when underlying version changes, to avoid
+  errors like the [one related to loading necessary libraries](https://docs.gitlab.com/omnibus/settings/database.html#could-not-load-library-plpgsqlso)
+  that can cause downtime. So, if you skip the automatic restarts using the above
+  method, ensure that you restart the services manually before upgrading to GitLab
+  15.0.
+
+- Starting with GitLab 15.0, the `AES256-GCM-SHA384` SSL cipher will not be allowed by
+  NGINX by default. If you require this cipher (for example, if you use
+  [AWS's Classic Load Balancer](https://docs.aws.amazon.com/en_en/elasticloadbalancing/latest/classic/elb-ssl-security-policy.html#ssl-ciphers)),
+  you can add the cipher back to the allow list by following the steps below:
+
+  1. Edit `/etc/gitlab/gitlab.rb` and add the following line:
+
+     ```ruby
+     nginx['ssl_ciphers'] = "ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384:AES256-GCM-SHA384"
+     ```
+
+  1. Reconfigure GitLab:
+
+     ```shell
+     sudo gitlab-ctl reconfigure
+     ```
+
+- Support for Gitaly's internal socket path is removed.
+  In GitLab 14.10, Gitaly introduced a new directory that holds all runtime
+  data Gitaly requires to operate correctly. This new directory replaces the
+  old internal socket directory, and consequentially the usage of
+  `gitaly['internal_socket_dir']` was deprecated in favor of
+  `gitaly['runtime_dir']`.
+
+  The old `gitaly['internal_socket_dir']` configuration was removed in this release.
+
+- Background uploads settings for object storage are removed.
+  Object storage now preferentially uses direct uploads.
+
+  The following keys are no longer supported in `/etc/gitlab/gitlab.rb`:
+
+  - `gitlab_rails['artifacts_object_store_direct_upload']`
+  - `gitlab_rails['artifacts_object_store_background_upload']`
+  - `gitlab_rails['external_diffs_object_store_direct_upload']`
+  - `gitlab_rails['external_diffs_object_store_background_upload']`
+  - `gitlab_rails['lfs_object_store_direct_upload']`
+  - `gitlab_rails['lfs_object_store_background_upload']`
+  - `gitlab_rails['uploads_object_store_direct_upload']`
+  - `gitlab_rails['uploads_object_store_background_upload']`
+  - `gitlab_rails['packages_object_store_direct_upload']`
+  - `gitlab_rails['packages_object_store_background_upload']`
+  - `gitlab_rails['dependency_proxy_object_store_direct_upload']`
+  - `gitlab_rails['dependency_proxy_object_store_background_upload']`
+
+### Self-compiled installations
+
+- Support for more than one database has been added to GitLab. For **self-compiled (source) installations**,
+  `config/database.yml` must include a database name in the database configuration.
+  The `main: database` must be first. If an invalid or deprecated syntax is used, an error is generated
+  during application start:
+
+  ```plaintext
+  ERROR: This installation of GitLab uses unsupported 'config/database.yml'.
+  The main: database needs to be defined as a first configuration item instead of primary. (RuntimeError)
+  ```
+
+  Previously, the `config/database.yml` file looked like the following:
+
+  ```yaml
+  production:
+    adapter: postgresql
+    encoding: unicode
+    database: gitlabhq_production
+    ...
+  ```
+
+  Starting with GitLab 15.0, it must define a `main` database first:
+
+  ```yaml
+  production:
+    main:
+      adapter: postgresql
+      encoding: unicode
+      database: gitlabhq_production
+      ...
+  ```
+
+### Geo installations **(PREMIUM SELF)**
+
+- Incorrect object storage LFS files deletion on Geo secondary sites. See
+  [the details and workaround](#incorrect-object-storage-lfs-file-deletion-on-secondary-sites).
diff --git a/doc/update/versions/gitlab_16_changes.md b/doc/update/versions/gitlab_16_changes.md
index cd51ca2a518..04581a88a93 100644
--- a/doc/update/versions/gitlab_16_changes.md
+++ b/doc/update/versions/gitlab_16_changes.md
@@ -4,7 +4,7 @@ group: Distribution
 info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments
 ---
 
-# GitLab 16 changes
+# GitLab 16 changes **(FREE SELF)**
 
 This page contains upgrade information for minor and patch versions of GitLab 16.
 Ensure you review these instructions for:
@@ -12,11 +12,76 @@ Ensure you review these instructions for:
 - Your installation type.
 - All versions between your current version and your target version.
 
-Some GitLab installations must upgrade to GitLab 16.0 before upgrading to any other version. For more information, see
-[Long-running user type data change](#long-running-user-type-data-change).
-
 For more information about upgrading GitLab Helm Chart, see [the release notes for 7.0](https://docs.gitlab.com/charts/releases/7_0.html).
 
+## Issues to be aware of when upgrading from 15.11
+
+- Some GitLab installations must upgrade to GitLab 16.0 before upgrading to any other version. For more information, see
+  [Long-running user type data change](#long-running-user-type-data-change).
+- Other installations can skip 16.0, 16.1, and 16.2 as the first required stop on the upgrade path is 16.3. Review the notes for those intermediate
+  versions.
+- If your GitLab instance upgraded first to 15.11.0, 15.11.1, or 15.11.2 the database schema is incorrect.
+  Recommended: perform the workaround before upgrading to 16.x.
+  See [the details and workaround](#undefined-column-error-upgrading-to-162-or-later).
+- Linux package installations must change Gitaly and Praefect configuration structure before upgrading to GitLab 16.
+  **To avoid data loss** reconfigure Praefect first, and as part of the new configuration, disable metadata verification.
+  Read more:
+
+  - [Praefect configuration structure change](#praefect-configuration-structure-change).
+  - [Gitaly configuration structure change](#gitaly-configuration-structure-change).
+
+## 16.4.0
+
+- Updating a group path [received a bug fix](https://gitlab.com/gitlab-org/gitlab/-/issues/419289) that uses a database index introduced in 16.3.
+
+  If you upgrade to 16.4 from a version lower than 16.3, you must execute `ANALYZE packages_packages;` in the database before you use it.
+
+- You might encounter the following error while upgrading to GitLab 16.4 or later:
+
+  ```plaintext
+  main: == 20230830084959 ValidatePushRulesConstraints: migrating =====================
+  main: -- execute("SET statement_timeout TO 0")
+  main:    -> 0.0002s
+  main: -- execute("ALTER TABLE push_rules VALIDATE CONSTRAINT force_push_regex_size_constraint;")
+  main:    -> 0.0004s
+  main: -- execute("RESET statement_timeout")
+  main:    -> 0.0003s
+  main: -- execute("ALTER TABLE push_rules VALIDATE CONSTRAINT delete_branch_regex_size_constraint;")
+  rails aborted!
+  StandardError: An error has occurred, all later migrations canceled:
+
+  PG::CheckViolation: ERROR:  check constraint "delete_branch_regex_size_constraint" of relation "push_rules" is violated by some row
+  ```
+
+  These constraints might return an error:
+
+  - `author_email_regex_size_constraint`
+  - `branch_name_regex_size_constraint`
+  - `commit_message_negative_regex_size_constraint`
+  - `commit_message_regex_size_constraint`
+  - `delete_branch_regex_size_constraint`
+  - `file_name_regex_size_constraint`
+  - `force_push_regex_size_constraint`
+
+  To fix the error, find the records in the `push_rules` table that exceed the 511
+  character limit.
+
+  ```sql
+  ;; replace `delete_branch_regex` with a name of the field used in constraint
+  SELECT id FROM push_rules WHERE LENGTH(delete_branch_regex) > 511;
+  ```
+
+  Reduce the value length of the regex field for affected push rules records, then
+  retry the migration.
+
+### Self-compiled installations
+
+- A new method of configuring paths for the GitLab secret and custom hooks is preferred in GitLab 16.4 and later:
+  1. Update your configuration `[gitlab] secret_file` to [configure the path](../../administration/gitaly/reference.md#gitlab) to the GitLab secret token.
+  1. If you have custom hooks, update your configuration `[hooks] custom_hooks_dir` to [configure the path](../../administration/gitaly/reference.md#custom-hooks) to
+     server-side custom hooks.
+  1. Remove the `[gitlab-shell] dir` configuration.
+
 ## 16.3.0
 
 - For Go applications, [`crypto/tls`: verifying certificate chains containing large RSA keys is slow (CVE-2023-29409)](https://github.com/golang/go/issues/61460)
@@ -54,6 +119,17 @@ Specific information applies to Linux package installations:
   - [An informal explanation](https://gitlab.com/gitlab-org/gitlab/-/issues/416714#note_1482388504).
   - `omnibus-gitlab` [merge request 7035](https://gitlab.com/gitlab-org/omnibus-gitlab/-/merge_requests/7035), which introduces the environment variable.
 
+### Geo installations
+
+Specific information applies to installations using Geo:
+
+- Git pulls against a secondary Geo site are being proxied to the primary Geo site even when that secondary site is up to date. You are impacted if you are using Geo to accelerate remote users who make Git pull requests against a secondary Geo site.
+
+  - Impacted versions: 16.3.0 - 16.3.2
+  - Versions containing fix: 16.4.0
+
+  For more information, see [issue 425224](https://gitlab.com/gitlab-org/gitlab/-/issues/425224).
+
 ## 16.2.0
 
 - Legacy LDAP configuration settings may cause
@@ -62,11 +138,15 @@ Specific information applies to Linux package installations:
   in the `tls_options` hash, or use the legacy `gitlab_rails['ldap_host']` option.
   See the [configuration workarounds](https://gitlab.com/gitlab-org/gitlab/-/issues/419485#workarounds)
   for more details.
-- New job artifacts are not replicated if job artifacts are configured to be stored in object storage and `direct_upload` is enabled. This bug is fixed in GitLab versions 16.1.4,
-  16.2.3, 16.3.0, and later.
-  - Impacted versions: GitLab versions 16.1.0 - 16.1.3 and 16.2.0 - 16.2.2.
-  - If you deployed an affected version, after upgrading to a fixed GitLab version, follow [these instructions](https://gitlab.com/gitlab-org/gitlab/-/issues/419742#to-fix-data)
-    to resync the affected job artifacts.
+- If your GitLab database was created by or upgraded via versions 15.11.0 - 15.11.2 inclusive, upgrading to GitLab 16.2 fails with:
+
+  ```plaintext
+  PG::UndefinedColumn: ERROR:  column "id_convert_to_bigint" of relation "ci_build_needs" does not exist
+  LINE 1: ...db_config_name:main*/ UPDATE "ci_build_needs" SET "id_conver...
+  ```
+
+  See [the details and workaround](#undefined-column-error-upgrading-to-162-or-later).
+
 - You might encounter the following error while upgrading to GitLab 16.2 or later:
 
   ```plaintext
@@ -98,6 +178,17 @@ Specific information applies to Linux package installations:
 
 - Git 2.41.0 and later is required by Gitaly. You should use the [Git version provided by Gitaly](../../install/installation.md#git).
 
+### Geo installations
+
+Specific information applies to installations using Geo:
+
+- New job artifacts are not replicated by Geo if job artifacts are configured to be stored in object storage and `direct_upload` is enabled. This bug is fixed in GitLab versions 16.1.4,
+  16.2.3, 16.3.0, and later.
+  - Impacted versions: GitLab versions 16.1.0 - 16.1.3 and 16.2.0 - 16.2.2.
+  - While running an affected version, artifacts which appeared to become synced may actually be missing on the secondary site.
+    Affected artifacts are automatically resynced upon upgrade to 16.1.5, 16.2.5, 16.3.1, 16.4.0, or later.
+    You can [manually resync affected job artifacts](https://gitlab.com/gitlab-org/gitlab/-/issues/419742#to-fix-data) if needed.
+
 ## 16.1.0
 
 - A `BackfillPreparedAtMergeRequests` background migration is finalized with
@@ -106,11 +197,6 @@ Specific information applies to Linux package installations:
   [backfill `prepared_at` values on the `merge_requests` table](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/111865). This
   migration may take multiple days to complete on larger GitLab instances. Make sure the migration
   has completed successfully before upgrading to 16.1.0.
-- New job artifacts are not replicated if job artifacts are configured to be stored in object storage and `direct_upload` is enabled. This bug is fixed in GitLab versions 16.1.4,
-  16.2.3, 16.3.0, and later.
-  - Impacted versions: GitLab versions 16.1.0 - 16.1.3 and 16.2.0 - 16.2.2.
-  - If you deployed an affected version, after upgrading to a fixed GitLab version, follow [these instructions](https://gitlab.com/gitlab-org/gitlab/-/issues/419742#to-fix-data)
-    to resync the affected job artifacts.
 
 ### Self-compiled installations
 
@@ -118,23 +204,41 @@ Specific information applies to Linux package installations:
   [removed](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/118645). For more information, see the
   [`puma.rb.example`](https://gitlab.com/gitlab-org/gitlab/-/blob/16-0-stable-ee/config/puma.rb.example) file.
 
-### Geo installations
+### Geo installations **(PREMIUM SELF)**
 
 Specific information applies to installations using Geo:
 
-- Some project imports do not initialize wiki repositories on project creation. Because of the migration of project wikis to
-  SSF, [missing wiki repositories are being incorrectly flagged as failing verification](https://gitlab.com/gitlab-org/gitlab/-/issues/409704).
-  This issue is not a result of an actual replication/verification failure but an invalid internal state for these missing
-  repositories inside Geo and results in errors in the logs and the verification progress reporting a failed state for
-  these wiki repositories. If you have not imported projects you are not impacted by this issue.
-  - Impacted versions: GitLab versions 15.11.x, 16.0.x, and 16.1.0 - 16.1.2.
-  - Versions containing fix: GitLab 16.1.3 and later.
+- Some project imports do not initialize wiki repositories on project creation. See
+  [the details and workaround](#wiki-repositories-not-initialized-on-project-creation).
 - Because of the migration of project designs to SSF, [missing design repositories are being incorrectly flagged as failing verification](https://gitlab.com/gitlab-org/gitlab/-/issues/414279).
   This issue is not a result of an actual replication/verification failure but an invalid internal state for these missing
   repositories inside Geo and results in errors in the logs and the verification progress reporting a failed state for
   these design repositories. You could be impacted by this issue even if you have not imported projects.
-  - Impacted versions: GitLab versions 16.1.x.
-  - Versions containing fix: GitLab 16.2.0 and later.
+  - Impacted versions: GitLab versions 16.1.0 - 16.1.2
+  - Versions containing fix: GitLab 16.1.3 and later.
+- New job artifacts are not replicated by Geo if job artifacts are configured to be stored in object storage and `direct_upload` is enabled. This bug is fixed in GitLab versions 16.1.4,
+  16.2.3, 16.3.0, and later.
+  - Impacted versions: GitLab versions 16.1.0 - 16.1.3 and 16.2.0 - 16.2.2.
+  - While running an affected version, artifacts which appeared to become synced may actually be missing on the secondary site.
+    Affected artifacts are automatically resynced upon upgrade to 16.1.5, 16.2.5, 16.3.1, 16.4.0, or later.
+    You can [manually resync affected job artifacts](https://gitlab.com/gitlab-org/gitlab/-/issues/419742#to-fix-data) if needed.
+
+#### Wiki repositories not initialized on project creation
+
+| Affected minor releases | Affected patch releases | Fixed in |
+|-------------------------|-------------------------|----------|
+| 15.11                   | All                     | None     |
+| 16.0                    | All                     | None     |
+| 16.1                    | 16.1.0 - 16.1.2         | 16.1.3 and later |
+
+Some project imports do not initialize wiki repositories on project creation.
+Since the migration of project wikis to SSF,
+[missing wiki repositories are being incorrectly flagged as failing verification](https://gitlab.com/gitlab-org/gitlab/-/issues/409704).
+This is not a result of an actual replication/verification failure but an
+invalid internal state for these missing repositories inside Geo and results in
+errors in the logs and the verification progress reporting a failed state for
+these wiki repositories. If you have not imported projects you are not impacted
+by this issue.
 
 ## 16.0.0
 
@@ -168,18 +272,371 @@ Specific information applies to Linux package installations:
 
   Workaround is to make use of a different key type, or upgrade the client OpenSSH to a version >= 8.7.
 
-### Geo installations
+- [Migrate your Praefect configuration to the new structure](#praefect-configuration-structure-change)
+  to ensure all your `praefect['..']` settings continue to work in GitLab 16.0 and later.
+
+- [Migrate your Gitaly configuration to the new structure](#gitaly-configuration-structure-change)
+  to ensure all your `gitaly['..']` settings continue to work in GitLab 16.0 and later.
+
+### Geo installations **(PREMIUM SELF)**
 
 Specific information applies to installations using Geo:
 
-- Some project imports do not initialize wiki repositories on project creation. Because of the migration of project wikis to
-  SSF, [missing wiki repositories are being incorrectly flagged as failing verification](https://gitlab.com/gitlab-org/gitlab/-/issues/409704).
-  This issue is not a result of an actual replication/verification failure but an invalid internal state for these missing
-  repositories inside Geo and results in errors in the logs and the verification progress reporting a failed state for
-  these wiki repositories. If you have not imported projects you are not impacted by this issue.
+- Some project imports do not initialize wiki repositories on project creation. See
+  [the details and workaround](#wiki-repositories-not-initialized-on-project-creation).
+
+### Gitaly configuration structure change
+
+The Gitaly configuration structure in the Linux package
+[changes](https://gitlab.com/gitlab-org/gitaly/-/issues/4467) in GitLab 16.0
+to be consistent with the Gitaly configuration structure used in
+self-compiled installations.
+
+As a result of this change, a single hash under `gitaly['configuration']` holds most Gitaly
+configuration. Some `gitaly['..']` configuration options continue to be used by GitLab 16.0 and later:
+
+- `enable`
+- `dir`
+- `bin_path`
+- `env_directory`
+- `env`
+- `open_files_ulimit`
+- `consul_service_name`
+- `consul_service_meta`
+
+Migrate by moving your existing configuration under the new structure. The new structure is supported from GitLab 15.10.
+
+**Migrate to the new structure**
+
+WARNING:
+If you are running Gitaly cluster, [migrate Praefect to the new configuration structure **first**](#praefect-configuration-structure-change).
+Once this change is tested, proceed with your Gitaly nodes.
+If Gitaly is misconfigured as part of the configuration structure change, [repository verification](../../administration/gitaly/praefect.md#repository-verification)
+will [delete metadata required for Gitaly cluster to work](https://gitlab.com/gitlab-org/gitaly/-/issues/5529).
+To protect against configuration mistakes, temporarily disable repository verification in Praefect.
+
+1. If you're running Gitaly Cluster, ensure repository verification is disabled on all Praefect nodes.
+   Configure `verification_interval: 0`, and apply with `gitlab-ctl reconfigure`.
+1. When applying the new structure to your configuration
+   - Replace the `...` with the value from the old key.
+   - When configuring `storage` to replace `git_data_dirs`, **you must append `repositories` to the path** as documented below.
+     If you miss this out your Git repositories are inaccessible until the configuration is fixed.
+     This misconfiguration can cause metadata deletion, and is the reason for disabling repository verification.
+   - Skip any keys you haven't configured a value for previously.
+   - Recommended. Include a trailing comma for all hash keys so the hash remains valid when keys are re-ordered or additional keys are added.
+1. Apply the change with `gitlab-ctl reconfigure`.
+1. Test Git repository functionality in GitLab.
+1. Remove the old keys from the configuration once migrated, and then re-run `gitlab-ctl reconfigure`.
+1. Recommended, if you're running Gitaly Cluster. Reinstate Praefect [repository verification](../../administration/gitaly/praefect.md#repository-verification)
+   by removing `verification_interval: 0`.
+
+The new structure is documented below with the old keys described in a comment above the new keys.
+
+```ruby
+gitaly['configuration'] = {
+  # gitaly['socket_path']
+  socket_path: ...,
+  # gitaly['runtime_dir']
+  runtime_dir: ...,
+  # gitaly['listen_addr']
+  listen_addr: ...,
+  # gitaly['prometheus_listen_addr']
+  prometheus_listen_addr: ...,
+  # gitaly['tls_listen_addr']
+  tls_listen_addr: ...,
+  tls: {
+    # gitaly['certificate_path']
+    certificate_path: ...,
+    # gitaly['key_path']
+    key_path: ...,
+  },
+  # gitaly['graceful_restart_timeout']
+  graceful_restart_timeout: ...,
+  logging: {
+    # gitaly['logging_level']
+    level: ...,
+    # gitaly['logging_format']
+    format: ...,
+    # gitaly['logging_sentry_dsn']
+    sentry_dsn: ...,
+    # gitaly['logging_ruby_sentry_dsn']
+    ruby_sentry_dsn: ...,
+    # gitaly['logging_sentry_environment']
+    sentry_environment: ...,
+    # gitaly['log_directory']
+    dir: ...,
+  },
+  prometheus: {
+    # gitaly['prometheus_grpc_latency_buckets']. The old value was configured as a string
+    # such as '[0, 1, 2]'. The new value must be an array like [0, 1, 2].
+    grpc_latency_buckets: ...,
+  },
+  auth: {
+    # gitaly['auth_token']
+    token: ...,
+    # gitaly['auth_transitioning']
+    transitioning: ...,
+  },
+  git: {
+    # gitaly['git_catfile_cache_size']
+    catfile_cache_size: ...,
+    # gitaly['git_bin_path']
+    bin_path: ...,
+    # gitaly['use_bundled_git']
+    use_bundled_binaries: ...,
+    # gitaly['gpg_signing_key_path']
+    signing_key: ...,
+    # gitaly['gitconfig']. This is still an array but the type of the elements have changed.
+    config: [
+      {
+        # Previously the elements contained 'section', and 'subsection' in addition to 'key'. Now
+        # these all should be concatenated into just 'key', separated by dots. For example,
+        # {section: 'first', subsection: 'middle', key: 'last', value: 'value'}, should become
+        # {key: 'first.middle.last', value: 'value'}.
+        key: ...,
+        value: ...,
+      },
+    ],
+  },
+  # Storage could previously be configured through either gitaly['storage'] or 'git_data_dirs'. Migrate
+  # the relevant configuration according to the instructions below.
+  # For 'git_data_dirs', migrate only the 'path' to the gitaly['configuration'] and leave the rest of it untouched.
+  storage: [
+    {
+      # gitaly['storage'][<index>]['name']
+      #
+      # git_data_dirs[<name>]. The storage name was configured as a key in the map.
+      name: ...,
+      # gitaly['storage'][<index>]['path']
+      #
+      # git_data_dirs[<name>]['path']. Use the value from git_data_dirs[<name>]['path'] and append '/repositories' to it.
+      #
+      # For example, if the path in 'git_data_dirs' was '/var/opt/gitlab/git-data', use
+      # '/var/opt/gitlab/git-data/repositories'. The '/repositories' extension was automatically
+      # appended to the path configured in `git_data_dirs`.
+      path: ...,
+    },
+  ],
+  hooks: {
+    # gitaly['custom_hooks_dir']
+    custom_hooks_dir: ...,
+  },
+  daily_maintenance: {
+    # gitaly['daily_maintenance_disabled']
+    disabled: ...,
+    # gitaly['daily_maintenance_start_hour']
+    start_hour: ...,
+    # gitaly['daily_maintenance_start_minute']
+    start_minute: ...,
+    # gitaly['daily_maintenance_duration']
+    duration: ...,
+    # gitaly['daily_maintenance_storages']
+    storages: ...,
+  },
+  cgroups: {
+    # gitaly['cgroups_mountpoint']
+    mountpoint: ...,
+    # gitaly['cgroups_hierarchy_root']
+    hierarchy_root: ...,
+    # gitaly['cgroups_memory_bytes']
+    memory_bytes: ...,
+    # gitaly['cgroups_cpu_shares']
+    cpu_shares: ...,
+    repositories: {
+      # gitaly['cgroups_repositories_count']
+      count: ...,
+      # gitaly['cgroups_repositories_memory_bytes']
+      memory_bytes: ...,
+      # gitaly['cgroups_repositories_cpu_shares']
+      cpu_shares: ...,
+    }
+  },
+  # gitaly['concurrency']. While the structure is the same, the string keys in the array elements
+  # should be replaced by symbols as elsewhere. {'key' => 'value'}, should become {key: 'value'}.
+  concurrency: ...,
+  # gitaly['rate_limiting']. While the structure is the same, the string keys in the array elements
+  # should be replaced by symbols as elsewhere. {'key' => 'value'}, should become {key: 'value'}.
+  rate_limiting: ...,
+  pack_objects_cache: {
+    # gitaly['pack_objects_cache_enabled']
+    enabled: ...,
+    # gitaly['pack_objects_cache_dir']
+    dir: ...,
+    # gitaly['pack_objects_cache_max_age']
+    max_age: ...,
+  }
+}
+```
 
-  - Impacted versions: GitLab versions 15.11.x, 16.0.x, and 16.1.0 - 16.1.2.
-  - Versions containing fix: GitLab 16.1.3 and later.
+### Praefect configuration structure change
+
+The Praefect configuration structure in the Linux package
+[changes](https://gitlab.com/gitlab-org/gitaly/-/issues/4467) in GitLab 16.0
+to be consistent with the Praefect configuration structure used in
+self-compiled installations.
+
+As a result of this change, a single hash under `praefect['configuration']` holds most Praefect
+configuration. Some `praefect['..']` configuration options continue to be used by GitLab 16.0 and later:
+
+- `enable`
+- `dir`
+- `log_directory`
+- `env_directory`
+- `env`
+- `wrapper_path`
+- `auto_migrate`
+- `consul_service_name`
+
+Migrate by moving your existing configuration under the new structure. The new structure is supported from GitLab 15.9.
+
+**Migrate to the new structure**
+
+WARNING:
+Migrate Praefect to the new configuration structure **first**.
+Once this change is tested, [proceed with your Gitaly nodes](#gitaly-configuration-structure-change).
+If Gitaly is misconfigured as part of the configuration structure change, [repository verification](../../administration/gitaly/praefect.md#repository-verification)
+will [delete metadata required for Gitaly cluster to work](https://gitlab.com/gitlab-org/gitaly/-/issues/5529).
+To protect against configuration mistakes, temporarily disable repository verification in Praefect.
+
+1. When applying the new structure to your configuration:
+   - Replace the `...` with the value from the old key.
+   - Disable repository verification using `verification_interval: 0`, as shown below.
+   - Skip any keys you haven't configured a value for previously.
+   - Recommended. Include a trailing comma for all hash keys so the hash remains valid when keys are re-ordered or additional keys are added.
+1. Apply the change with `gitlab-ctl reconfigure`.
+1. Test Git repository functionality in GitLab.
+1. Remove the old keys from the configuration once migrated, and then re-run `gitlab-ctl reconfigure`.
+
+The new structure is documented below with the old keys described in a comment above the new keys.
+
+```ruby
+praefect['configuration'] = {
+  # praefect['listen_addr']
+  listen_addr: ...,
+  # praefect['socket_path']
+  socket_path: ...,
+  # praefect['prometheus_listen_addr']
+  prometheus_listen_addr: ...,
+  # praefect['tls_listen_addr']
+  tls_listen_addr: ...,
+  # praefect['separate_database_metrics']
+  prometheus_exclude_database_from_default_metrics: ...,
+  auth: {
+    # praefect['auth_token']
+    token: ...,
+    # praefect['auth_transitioning']
+    transitioning: ...,
+  },
+  logging: {
+    # praefect['logging_format']
+    format: ...,
+    # praefect['logging_level']
+    level: ...,
+  },
+  failover: {
+    # praefect['failover_enabled']
+    enabled: ...,
+  },
+  background_verification: {
+    # praefect['background_verification_delete_invalid_records']
+    delete_invalid_records: ...,
+    # praefect['background_verification_verification_interval']
+    #
+    # IMPORTANT:
+    # As part of reconfiguring Praefect, disable this feature.
+    # Read about this above.
+    #
+    verification_interval: 0,
+  },
+  reconciliation: {
+    # praefect['reconciliation_scheduling_interval']
+    scheduling_interval: ...,
+    # praefect['reconciliation_histogram_buckets']. The old value was configured as a string
+    # such as '[0, 1, 2]'. The new value must be an array like [0, 1, 2].
+    histogram_buckets: ...,
+  },
+  tls: {
+    # praefect['certificate_path']
+    certificate_path: ...,
+   # praefect['key_path']
+    key_path: ...,
+  },
+  database: {
+    # praefect['database_host']
+    host: ...,
+    # praefect['database_port']
+    port: ...,
+    # praefect['database_user']
+    user: ...,
+    # praefect['database_password']
+    password: ...,
+    # praefect['database_dbname']
+    dbname: ...,
+    # praefect['database_sslmode']
+    sslmode: ...,
+    # praefect['database_sslcert']
+    sslcert: ...,
+    # praefect['database_sslkey']
+    sslkey: ...,
+    # praefect['database_sslrootcert']
+    sslrootcert: ...,
+    session_pooled: {
+      # praefect['database_direct_host']
+      host: ...,
+      # praefect['database_direct_port']
+      port: ...,
+      # praefect['database_direct_user']
+      user: ...,
+      # praefect['database_direct_password']
+      password: ...,
+      # praefect['database_direct_dbname']
+      dbname: ...,
+      # praefect['database_direct_sslmode']
+      sslmode: ...,
+      # praefect['database_direct_sslcert']
+      sslcert: ...,
+      # praefect['database_direct_sslkey']
+      sslkey: ...,
+      # praefect['database_direct_sslrootcert']
+      sslrootcert: ...,
+    }
+  },
+  sentry: {
+    # praefect['sentry_dsn']
+    sentry_dsn: ...,
+    # praefect['sentry_environment']
+    sentry_environment: ...,
+  },
+  prometheus: {
+    # praefect['prometheus_grpc_latency_buckets']. The old value was configured as a string
+    # such as '[0, 1, 2]'. The new value must be an array like [0, 1, 2].
+    grpc_latency_buckets: ...,
+  },
+  # praefect['graceful_stop_timeout']
+  graceful_stop_timeout: ...,
+  # praefect['virtual_storages']. The old value was a hash map but the new value is an array.
+  virtual_storage: [
+    {
+      # praefect['virtual_storages'][VIRTUAL_STORAGE_NAME]. The name was previously the key in
+      # the 'virtual_storages' hash.
+      name: ...,
+      # praefect['virtual_storages'][VIRTUAL_STORAGE_NAME]['nodes'][NODE_NAME]. The old value was a hash map
+      # but the new value is an array.
+      node: [
+        {
+          # praefect['virtual_storages'][VIRTUAL_STORAGE_NAME]['nodes'][NODE_NAME]. Use NODE_NAME key as the
+          # storage.
+          storage: ...,
+          # praefect['virtual_storages'][VIRTUAL_STORAGE_NAME]['nodes'][NODE_NAME]['address'].
+          address: ...,
+          # praefect['virtual_storages'][VIRTUAL_STORAGE_NAME]['nodes'][NODE_NAME]['token'].
+          token: ...,
+        },
+      ],
+    }
+  ]
+}
+```
 
 ## Long-running user type data change
 
@@ -196,7 +653,7 @@ migration might take multiple days to complete on larger GitLab instances. Make
 has completed successfully before upgrading to 16.1.0 or later.
 
 GitLab 16.1 introduces the `FinalizeUserTypeMigration` migration which ensures the
-16.0 `MigrateHumanUserType` background migration is completed, making the 16.0 changes synchronously
+16.0 `MigrateHumanUserType` background migration is completed, executing the 16.0 change synchronously
 during the upgrade if it's not completed.
 
 GitLab 16.2 [implements a `NOT NULL` database constraint](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/122454)
@@ -219,3 +676,127 @@ an unusable state, generating `500` errors. The errors are caused by Sidekiq and
 application code that is incompatible with the database schema.
 
 At the end of the workaround process, Sidekiq and Puma are restarted to resolve that issue.
+
+## Undefined column error upgrading to 16.2 or later
+
+A bug in GitLab 15.11 incorrectly disabled a database change on self-managed instances.
+For more information, see [issue 408835](https://gitlab.com/gitlab-org/gitlab/-/issues/408835).
+
+If your GitLab instance upgraded first to 15.11.0, 15.11.1, or 15.11.2 the database schema is
+incorrect and upgrading to GitLab 16.2 or later fails with an error. A database change
+requires the earlier modification to be in place:
+
+```plaintext
+PG::UndefinedColumn: ERROR:  column "id_convert_to_bigint" of relation "ci_build_needs" does not exist
+LINE 1: ...db_config_name:main*/ UPDATE "ci_build_needs" SET "id_conver...
+```
+
+GitLab 15.11.3 shipped a fix for this bug, but it doesn't correct the problem on
+instances already running the earlier 15.11 releases.
+
+If you're not sure if an instance is affected, check for the column on the
+[database console](../../administration/troubleshooting/postgresql.md#start-a-database-console):
+
+```sql
+select pg_typeof (id_convert_to_bigint) from public.ci_build_needs limit 1;
+```
+
+If you need the workaround, this query fails:
+
+```plaintext
+ERROR:  column "id_convert_to_bigintd" does not exist
+LINE 1: select pg_typeof (id_convert_to_bigintd) from public.ci_buil...
+```
+
+Unaffected instances return:
+
+```plaintext
+ pg_typeof
+-----------
+ bigint
+```
+
+The workaround for this issue differs if your GitLab instance's database schema
+was recently created:
+
+| Installation version | Workaround |
+| -------------------- | ---------- |
+| 15.9 or earlier      | [15.9](#workaround-instance-created-with-159-or-earlier) |
+| 15.10                | [15.10](#workaround-instance-created-with-1510) |
+| 15.11                | [15.11](#workaround-instance-created-with-1511) |
+
+Most instances should use the 15.9 procedure. Only very new instances require the
+the 15.10 or 15.11 procedures. If you've migrated GitLab using backup and restore,
+the database schema comes from the original instance. Select the workaround based
+on the source instance.
+
+The commands in the following sections are for Linux package installations, and
+differ for other installation types:
+
+::Tabs
+
+:::TabTitle Docker
+
+- Omit `sudo`
+- Shell into the GitLab container and run the same commands:
+
+  ```shell
+  docker exec -it <container-id> bash
+  ```
+
+:::TabTitle Self-compiled (source)
+
+- Use `sudo -u git -H bundle exec rake` instead of `sudo gitlab-rake`
+- Run the SQL on [your PostgreSQL database console](../../administration/troubleshooting/postgresql.md#start-a-database-console)
+
+:::TabTitle Helm chart (Kubernetes)
+
+- Omit `sudo`.
+- Shell into the `toolbox` pod to run the Rake commands: `gitlab-rake` is in `/usr/local/bin` if not in the `PATH`.
+  - Refer to our [Kubernetes cheat sheet](https://docs.gitlab.com/charts/troubleshooting/kubernetes_cheat_sheet.html#gitlab-specific-kubernetes-information) for details.
+- Run the SQL on [your PostgreSQL database console](../../administration/troubleshooting/postgresql.md#start-a-database-console)
+
+::EndTabs
+
+### Workaround: instance created with 15.9 or earlier
+
+```shell
+# Restore schema
+sudo gitlab-psql -c "DELETE FROM schema_migrations WHERE version IN ('20230130175512', '20230130104819');"
+sudo gitlab-rake db:migrate:up VERSION=20230130175512
+sudo gitlab-rake db:migrate:up VERSION=20230130104819
+
+# Re-schedule background migrations
+sudo gitlab-rake db:migrate:down VERSION=20230130202201
+sudo gitlab-rake db:migrate:down VERSION=20230130110855
+sudo gitlab-rake db:migrate:up VERSION=20230130202201
+sudo gitlab-rake db:migrate:up VERSION=20230130110855
+```
+
+### Workaround: instance created with 15.10
+
+```shell
+# Restore schema for sent_notifications
+sudo gitlab-psql -c "DELETE FROM schema_migrations WHERE version = '20230130175512';"
+sudo gitlab-rake db:migrate:up VERSION=20230130175512
+
+# Re-schedule background migration for sent_notifications
+sudo gitlab-rake db:migrate:down VERSION=20230130202201
+sudo gitlab-rake db:migrate:up VERSION=20230130202201
+
+# Restore schema for ci_build_needs
+sudo gitlab-rake db:migrate:down VERSION=20230321163547
+sudo gitlab-psql -c "INSERT INTO schema_migrations (version) VALUES ('20230321163547');"
+```
+
+### Workaround: instance created with 15.11
+
+```shell
+# Restore schema for sent_notifications
+sudo gitlab-rake db:migrate:down VERSION=20230411153310
+sudo gitlab-psql -c "INSERT INTO schema_migrations (version) VALUES ('20230411153310');"
+
+# Restore schema for ci_build_needs
+sudo gitlab-rake db:migrate:down VERSION=20230321163547
+sudo gitlab-psql -c "INSERT INTO schema_migrations (version) VALUES ('20230321163547');"
+```