Add latest changes from gitlab-org/gitlab@16-6-stable-eev16.6.0-rc42

3 files changed, 386 insertions, 24 deletions

diff --git a/doc/update/deprecations.md b/doc/update/deprecations.md
index 3bb7f9816b4..333dad86086 100644
--- a/doc/update/deprecations.md
+++ b/doc/update/deprecations.md
@@ -102,6 +102,28 @@ This change is a breaking change. You should [create a runner in the UI](../ci/r
 
 <div class="deprecation breaking-change" data-milestone="18.0">
 
+### Running a single database is deprecated
+
+<div class="deprecation-notes">
+- Announced in GitLab <span class="milestone">16.1</span>
+- Removal in GitLab <span class="milestone">18.0</span> ([breaking change](https://docs.gitlab.com/ee/update/terminology.html#breaking-change))
+- To discuss this change or learn more, see the [deprecation issue](https://gitlab.com/gitlab-org/gitlab/-/issues/411239).
+</div>
+
+From GitLab 18.0, we will require a [separate database for CI features](https://gitlab.com/groups/gitlab-org/-/epics/7509).
+We recommend running both databases on the same Postgres instance(s) due to ease of management for most deployments.
+
+We are providing this as an informational advance notice but we do not recommend taking action yet.
+We will have another update communicated (as well as the deprecation note) when we recommend admins to start the migration process.
+
+This change provides additional scalability for the largest of GitLab instances, like GitLab.com.
+This change applies to all installation methods: Omnibus GitLab, GitLab Helm chart, GitLab Operator, GitLab Docker images, and installation from source.
+Before upgrading to GitLab 18.0, please ensure you have [migrated](https://docs.gitlab.com/ee/administration/postgresql/multiple_databases.html) to two databases.
+
+</div>
+
+<div class="deprecation breaking-change" data-milestone="18.0">
+
 ### Support for REST API endpoints that reset runner registration tokens
 
 <div class="deprecation-notes">
@@ -187,6 +209,26 @@ Because Cloud Native Buildpacks do not support automatic testing, the Auto Test
 
 <div class="deprecation breaking-change" data-milestone="17.0">
 
+### Breaking change to the Maven repository group permissions
+
+<div class="deprecation-notes">
+- Announced in GitLab <span class="milestone">16.6</span>
+- Removal in GitLab <span class="milestone">17.0</span> ([breaking change](https://docs.gitlab.com/ee/update/terminology.html#breaking-change))
+- To discuss this change or learn more, see the [deprecation issue](https://gitlab.com/gitlab-org/gitlab/-/issues/393933).
+</div>
+
+The Maven repository exposes an API endpoint at the group level that allows Maven clients to download files from a specific package. The package finder first locates the package within the group, and then finds the file within the package.
+However, there is a limitation that affects duplicate package names hosted in different projects. The Maven package finder always returns the most recent package, but the "most recent" filter depends on user permissions. It is possible for a user with different permissions in different projects to download the wrong Maven package.
+
+In GitLab 17.0, the package finder logic will be fixed so that the "most recent" package is the last updated name and version of a package in a group. User permissions will be checked after the most recent package is found.
+After the change, download requests for users without correct permissions will be rejected. If your workflow depends on the current bugged behavior, this fix will introduce a breaking change.
+
+The change will be introduced in GitLab 16.6 behind a feature flag. If you are interested in enabling the feature flag for your group, leave a comment in [issue 393933](https://gitlab.com/gitlab-org/gitlab/-/issues/393933).
+
+</div>
+
+<div class="deprecation breaking-change" data-milestone="17.0">
+
 ### CiRunner.projects default sort is changing to `id_desc`
 
 <div class="deprecation-notes">
@@ -217,6 +259,24 @@ the aliasing for the `CiRunnerUpgradeStatusType` type will be removed.
 
 <div class="deprecation breaking-change" data-milestone="17.0">
 
+### Container Registry support for the Swift and OSS storage drivers
+
+<div class="deprecation-notes">
+- Announced in GitLab <span class="milestone">16.6</span>
+- Removal in GitLab <span class="milestone">17.0</span> ([breaking change](https://docs.gitlab.com/ee/update/terminology.html#breaking-change))
+- To discuss this change or learn more, see the [deprecation issue](https://gitlab.com/gitlab-org/container-registry/-/issues/1141).
+</div>
+
+The container registry uses storage drivers to work with various object storage platforms. While each driver's code is relatively self-contained, there is a high maintenance burden for these drivers. Each driver implementation is unique and making changes to a driver requires a high level of domain expertise with that specific driver.
+
+As we look to reduce maintenance costs, we are deprecating support for OSS (Object Storage Service) and OpenStack Swift. Both have already been removed from the upstream Docker Distribution. This helps align the container registry with the broader GitLab product offering with regards to [object storage support](https://docs.gitlab.com/ee/administration/object_storage.html#supported-object-storage-providers).
+
+OSS has an [S3 compatibility mode](https://www.alibabacloud.com/help/en/oss/developer-reference/compatibility-with-amazon-s3), so consider using that if you can't migrate to a supported driver. Swift is [compatible with S3 API operations](https://docs.openstack.org/swift/latest/s3_compat.html), required by the S3 storage driver as well.
+
+</div>
+
+<div class="deprecation breaking-change" data-milestone="17.0">
+
 ### DAST ZAP advanced configuration variables deprecation
 
 <div class="deprecation-notes">
@@ -387,6 +447,24 @@ major release, GitLab 17.0. This gem sees very little use and is better suited f
 
 <div class="deprecation breaking-change" data-milestone="17.0">
 
+### File type variable expansion fixed in downstream pipelines
+
+<div class="deprecation-notes">
+- Announced in GitLab <span class="milestone">16.6</span>
+- Removal in GitLab <span class="milestone">17.0</span> ([breaking change](https://docs.gitlab.com/ee/update/terminology.html#breaking-change))
+- To discuss this change or learn more, see the [deprecation issue](https://gitlab.com/gitlab-org/gitlab/-/issues/419445).
+</div>
+
+Previously, if you tried to reference a [file type CI/CD variable](https://docs.gitlab.com/ee/ci/variables/#use-file-type-cicd-variables) in another CI/CD variable, the CI/CD variable would expand to contain the contents of the file. This behavior was incorrect because it did not comply with typical shell variable expansion rules. The CI/CD variable reference should expand to only contain the path to the file, not the contents of the file itself. This was [fixed for most use cases in GitLab 15.7](https://gitlab.com/gitlab-org/gitlab/-/issues/29407). Unfortunately, passing CI/CD variables to downstream pipelines was an edge case not yet fixed, but which will now be fixed in GitLab 17.0.
+
+With this change, a variable configured in the `.gitlab-ci.yml` file can reference a file variable and be passed to a downstream pipeline, and the file variable will be passed to the downstream pipeline as well. The downstream pipeline will expand the variable reference to the file path, not the file contents.
+
+This breaking change could disrupt user workflows that depend on expanding a file variable in a downstream pipeline.
+
+</div>
+
+<div class="deprecation breaking-change" data-milestone="17.0">
+
 ### Filepath field in Releases and Release Links APIs
 
 <div class="deprecation-notes">
@@ -560,6 +638,24 @@ In GitLab 17.0, the `DISABLED_WITH_OVERRIDE` value of the `SharedRunnersSetting`
 
 <div class="deprecation breaking-change" data-milestone="17.0">
 
+### GraphQL: deprecate support for `canDestroy` and `canDelete`
+
+<div class="deprecation-notes">
+- Announced in GitLab <span class="milestone">16.6</span>
+- Removal in GitLab <span class="milestone">17.0</span> ([breaking change](https://docs.gitlab.com/ee/update/terminology.html#breaking-change))
+- To discuss this change or learn more, see the [deprecation issue](https://gitlab.com/gitlab-org/gitlab/-/issues/390754).
+</div>
+
+The Package Registry user interface relies on the GitLab GraphQL API. To make it easy for everyone to contribute, it's important that the frontend is coded consistently across all GitLab product areas. Before GitLab 16.6, however, the Package Registry UI handled permissions differently from other areas of the product.
+
+In 16.6, we added a new `UserPermissions` field under the `Types::PermissionTypes::Package` type to align the Package Registry with the rest of GitLab. This new field replaces the `canDestroy` field under the `Package`, `PackageBase`, and `PackageDetailsType` types. It also replaces the field `canDelete` for `ContainerRepository`, `ContainerRepositoryDetails`, and `ContainerRepositoryTag`. In GitLab 17.0, the `canDestroy` and `canDelete` fields will be removed.
+
+This is a breaking change that will be completed in 17.0.
+
+</div>
+
+<div class="deprecation breaking-change" data-milestone="17.0">
+
 ### HashiCorp Vault integration will no longer use CI_JOB_JWT by default
 
 <div class="deprecation-notes">
@@ -615,6 +711,33 @@ If you do access the internal container registry API and use the original tag de
 
 <div class="deprecation breaking-change" data-milestone="17.0">
 
+### Legacy Geo Prometheus metrics
+
+<div class="deprecation-notes">
+- Announced in GitLab <span class="milestone">16.6</span>
+- Removal in GitLab <span class="milestone">17.0</span> ([breaking change](https://docs.gitlab.com/ee/update/terminology.html#breaking-change))
+- To discuss this change or learn more, see the [deprecation issue](https://gitlab.com/gitlab-org/gitlab/-/issues/430192).
+</div>
+
+Following the migration of projects to the [Geo self-service framework](https://docs.gitlab.com/ee/development/geo/framework.html) we have deprecated a number of [Prometheus](https://docs.gitlab.com/ee/administration/monitoring/prometheus/) metrics.
+The following Geo-related Prometheus metrics are deprecated and will be removed in 17.0.
+The table below lists the deprecated metrics and their respective replacements. The replacements are available in GitLab 16.3.0 and later.
+
+| Deprecated metric                        |  Replacement metric                            |
+| ---------------------------------------- | ---------------------------------------------- |
+| `geo_repositories_synced`                | `geo_project_repositories_synced`              |
+| `geo_repositories_failed`                | `geo_project_repositories_failed`              |
+| `geo_repositories_checksummed`           | `geo_project_repositories_checksummed`         |
+| `geo_repositories_checksum_failed`       | `geo_project_repositories_checksum_failed`     |
+| `geo_repositories_verified`              | `geo_project_repositories_verified`            |
+| `geo_repositories_verification_failed`   | `geo_project_repositories_verification_failed` |
+| `geo_repositories_checksum_mismatch`     |  None available                                |
+| `geo_repositories_retrying_verification` |  None available                                |
+
+</div>
+
+<div class="deprecation breaking-change" data-milestone="17.0">
+
 ### Maintainer role providing the ability to change Package settings using GraphQL API
 
 <div class="deprecation-notes">
@@ -757,6 +880,20 @@ PostgreSQL 14 will also be supported for instances that want to upgrade prior to
 
 <div class="deprecation breaking-change" data-milestone="17.0">
 
+### Proxy-based DAST deprecated
+
+<div class="deprecation-notes">
+- Announced in GitLab <span class="milestone">16.6</span>
+- Removal in GitLab <span class="milestone">17.0</span> ([breaking change](https://docs.gitlab.com/ee/update/terminology.html#breaking-change))
+- To discuss this change or learn more, see the [deprecation issue](https://gitlab.com/gitlab-org/gitlab/-/issues/430966).
+</div>
+
+As of GitLab 17.0, Proxy-based DAST will not be supported. Please migrate to Browser-based DAST to continue analyzing your projects for security findings via dynamic analysis.
+
+</div>
+
+<div class="deprecation breaking-change" data-milestone="17.0">
+
 ### Queue selector for running Sidekiq is deprecated
 
 <div class="deprecation-notes">
@@ -822,28 +959,6 @@ that is available now. We recommend this alternative solution because it provide
 
 <div class="deprecation breaking-change" data-milestone="17.0">
 
-### Running a single database is deprecated
-
-<div class="deprecation-notes">
-- Announced in GitLab <span class="milestone">16.1</span>
-- Removal in GitLab <span class="milestone">17.0</span> ([breaking change](https://docs.gitlab.com/ee/update/terminology.html#breaking-change))
-- To discuss this change or learn more, see the [deprecation issue](https://gitlab.com/gitlab-org/gitlab/-/issues/411239).
-</div>
-
-From GitLab 17.0, we will require a [separate database for CI features](https://gitlab.com/groups/gitlab-org/-/epics/7509).
-We recommend running both databases on the same Postgres instance(s) due to ease of management for most deployments.
-
-We are providing this as an informational advance notice but we do not recommend taking action yet.
-We will have another update communicated (as well as the deprecation note) when we recommend admins to start the migration process.
-
-This change provides additional scalability for the largest of GitLab instances, like GitLab.com.
-This change applies to all installation methods: Omnibus GitLab, GitLab Helm chart, GitLab Operator, GitLab Docker images, and installation from source.
-Before upgrading to GitLab 17.0, please ensure you have [migrated](https://docs.gitlab.com/ee/administration/postgresql/multiple_databases.html) to two databases.
-
-</div>
-
-<div class="deprecation breaking-change" data-milestone="17.0">
-
 ### Security policy field `newly_detected` is deprecated
 
 <div class="deprecation-notes">
@@ -892,8 +1007,6 @@ For updates and details about this deprecation, follow [this epic](https://gitla
 - To discuss this change or learn more, see the [deprecation issue](https://gitlab.com/gitlab-org/gitlab/-/issues/387898).
 </div>
 
-This deprecation is now superseded by another [deprecation notice](#running-a-single-database-is-deprecated).
-
 Previously, [GitLab's database](https://docs.gitlab.com/omnibus/settings/database.html)
 configuration had a single `main:` section. This is being deprecated. The new
 configuration has both a `main:` and a `ci:` section.
@@ -926,6 +1039,24 @@ we'll be introducing support in [this epic](https://gitlab.com/groups/gitlab-org
 
 <div class="deprecation breaking-change" data-milestone="17.0">
 
+### The GitHub importer Rake task
+
+<div class="deprecation-notes">
+- Announced in GitLab <span class="milestone">16.6</span>
+- Removal in GitLab <span class="milestone">17.0</span> ([breaking change](https://docs.gitlab.com/ee/update/terminology.html#breaking-change))
+- To discuss this change or learn more, see the [deprecation issue](https://gitlab.com/gitlab-org/gitlab/-/issues/428225).
+</div>
+
+In GitLab 16.6 the [GitHub importer Rake task](https://docs.gitlab.com/ee/administration/raketasks/github_import.html) is deprecated. The Rake task lacks several features that are supported by the API and is not actively maintained.
+
+In GitLab 17.0, the Rake task will be removed.
+
+Instead, GitHub repositories can be imported by using the [API](https://docs.gitlab.com/ee/api/import.html#import-repository-from-github) or the [UI](https://docs.gitlab.com/ee/user/project/import/github.html).
+
+</div>
+
+<div class="deprecation breaking-change" data-milestone="17.0">
+
 ### The GitLab legacy requirement IID is deprecated in favor of work item IID
 
 <div class="deprecation-notes">
@@ -1142,6 +1273,29 @@ Previous work helped [align the vulnerabilities calls for pipeline security tabs
 </div>
 </div>
 
+<div class="milestone-wrapper" data-milestone="16.9">
+
+## GitLab 16.9
+
+<div class="deprecation " data-milestone="16.9">
+
+### Deprecation of `lfs_check` feature flag
+
+<div class="deprecation-notes">
+- Announced in GitLab <span class="milestone">16.6</span>
+- Removal in GitLab <span class="milestone">16.9</span>
+- To discuss this change or learn more, see the [deprecation issue](https://gitlab.com/gitlab-org/gitlab/-/issues/233550).
+</div>
+
+In GitLab 16.9, we will remove the `lfs_check` feature flag. This feature flag was [introduced 4 years ago](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/60588) and controls whether the LFS integrity check is enabled. The feature flag is enabled by default, but some customers experienced performance issues with the LFS integrity check and explicitly disabled it.
+
+After [dramatically improving the performance](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/61355) of the LFS integrity check, we are ready to remove the feature flag. After the flag is removed, the feature will automatically be turned on for any environment in which it is currently disabled.
+
+If this feature flag is disabled for your environment, and you are concerned about performance issues, please enable it and monitor the performance before it is removed in 16.9. If you see any performance issues after enabling it, please let us know in [this feedback issue](https://gitlab.com/gitlab-org/gitlab/-/issues/233550).
+
+</div>
+</div>
+
 <div class="milestone-wrapper" data-milestone="16.8">
 
 ## GitLab 16.8
@@ -1206,6 +1360,33 @@ If you have [public or internal](https://docs.gitlab.com/ee/user/public_access.h
 Enabling the `ldap_settings_unlock_groups_by_owners` feature flag allowed non-LDAP synced users to be added to a locked LDAP group. This [feature](https://gitlab.com/gitlab-org/gitlab/-/issues/1793) has always been disabled by default and behind a feature flag. We are removing this feature to keep continuity with our SAML integration, and because allowing non-synced group members defeats the "single source of truth" principle of using a directory service. Once this feature is removed, any LDAP group members that are not synced with LDAP will lose access to that group.
 
 </div>
+
+<div class="deprecation breaking-change" data-milestone="16.5">
+
+### Geo: Housekeeping Rake tasks
+
+<div class="deprecation-notes">
+- Announced in GitLab <span class="milestone">16.3</span>
+- Removal in GitLab <span class="milestone">16.5</span> ([breaking change](https://docs.gitlab.com/ee/update/terminology.html#breaking-change))
+- To discuss this change or learn more, see the [deprecation issue](https://gitlab.com/gitlab-org/gitlab/-/issues/416384).
+</div>
+
+As part of the migration of the replication and verification to the
+[Geo self-service framework (SSF)](https://docs.gitlab.com/ee/development/geo/framework.html),
+the legacy replication for project repositories has been
+[removed](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/130565).
+As a result, the following Rake tasks that relied on legacy code have also been removed. The work invoked by these Rake tasks are now triggered automatically either periodically or based on trigger events.
+
+| Rake task | Replacement |
+| --------- | ----------- |
+| `geo:git:housekeeping:full_repack` | [Moved to UI](https://docs.gitlab.com/ee/administration/housekeeping.html#heuristical-housekeeping). No equivalent Rake task in the SSF. |
+| `geo:git:housekeeping:gc` | Always executed for new repositories, and then when it's needed. No equivalent Rake task in the SSF. |
+| `geo:git:housekeeping:incremental_repack` | Executed when needed. No equivalent Rake task in the SSF. |
+| `geo:run_orphaned_project_registry_cleaner` | Executed regularly by a registry [consistency worker](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/app/workers/geo/secondary/registry_consistency_worker.rb) which removes orphaned registries. No equivalent Rake task in the SSF. |
+| `geo:verification:repository:reset` | Moved to UI.  No equivalent Rake task in the SSF. |
+| `geo:verification:wiki:reset` | Moved to UI.  No equivalent Rake task in the SSF. |
+
+</div>
 </div>
 
 <div class="milestone-wrapper" data-milestone="16.3">
diff --git a/doc/update/versions/gitlab_15_changes.md b/doc/update/versions/gitlab_15_changes.md
index 019b8929a45..bd5efef8f1b 100644
--- a/doc/update/versions/gitlab_15_changes.md
+++ b/doc/update/versions/gitlab_15_changes.md
@@ -136,6 +136,7 @@ if you can't upgrade to 15.11.12 and later.
 
 - `pg_upgrade` fails to upgrade the bundled PostregSQL database to version 13. See
   [the details and workaround](#pg_upgrade-fails-to-upgrade-the-bundled-postregsql-database-to-version-13).
+- Cloning LFS objects from secondary site downloads from the primary site even when secondary is fully synced. See [the details and workaround](gitlab_16_changes.md#cloning-lfs-objects-from-secondary-site-downloads-from-the-primary-site-even-when-secondary-is-fully-synced).
 
 ## 15.9.0
 
@@ -181,6 +182,7 @@ if you can't upgrade to 15.11.12 and later.
 
 - `pg_upgrade` fails to upgrade the bundled PostregSQL database to version 13. See
   [the details and workaround](#pg_upgrade-fails-to-upgrade-the-bundled-postregsql-database-to-version-13).
+- Cloning LFS objects from secondary site downloads from the primary site even when secondary is fully synced. See [the details and workaround](gitlab_16_changes.md#cloning-lfs-objects-from-secondary-site-downloads-from-the-primary-site-even-when-secondary-is-fully-synced).
 
 ## 15.8.2
 
@@ -212,6 +214,7 @@ if you can't upgrade to 15.11.12 and later.
 - We discovered an issue where [replication and verification of projects and wikis was not keeping up](https://gitlab.com/gitlab-org/gitlab/-/issues/387980) on small number of Geo installations. Your installation may be affected if you see some projects and/or wikis persistently in the "Queued" state for verification. This can lead to data loss after a failover.
   - Affected versions: GitLab versions 15.6.x, 15.7.x, and 15.8.0 - 15.8.2.
   - Versions containing fix: GitLab 15.8.3 and later.
+- Cloning LFS objects from secondary site downloads from the primary site even when secondary is fully synced. See [the details and workaround](gitlab_16_changes.md#cloning-lfs-objects-from-secondary-site-downloads-from-the-primary-site-even-when-secondary-is-fully-synced).
 
 ## 15.7.6
 
@@ -324,6 +327,7 @@ if you can't upgrade to 15.11.12 and later.
   contents printed. For example, if they were printed in an echo output. For more information,
   see [Understanding the file type variable expansion change in GitLab 15.7](https://about.gitlab.com/blog/2023/02/13/impact-of-the-file-type-variable-change-15-7/).
 - Due to [a bug introduced in GitLab 15.4](https://gitlab.com/gitlab-org/gitlab/-/issues/390155), if one or more Git repositories in Gitaly Cluster is [unavailable](../../administration/gitaly/recovery.md#unavailable-repositories), then [Repository checks](../../administration/repository_checks.md#repository-checks) and [Geo replication and verification](../../administration/geo/index.md) stop running for all project or project wiki repositories in the affected Gitaly Cluster. The bug was fixed by [reverting the change in GitLab 15.9.0](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/110823). Before upgrading to this version, check if you have any "unavailable" repositories. See [the bug issue](https://gitlab.com/gitlab-org/gitlab/-/issues/390155) for more information.
+- Cloning LFS objects from secondary site downloads from the primary site even when secondary is fully synced. See [the details and workaround](gitlab_16_changes.md#cloning-lfs-objects-from-secondary-site-downloads-from-the-primary-site-even-when-secondary-is-fully-synced).
 
 ### Geo installations **(PREMIUM SELF)**
 
@@ -441,6 +445,7 @@ potentially cause downtime.
   - Affected versions: GitLab versions 15.6.x, 15.7.x, and 15.8.0 - 15.8.2.
   - Versions containing fix: GitLab 15.8.3 and later.
 - Due to [a bug introduced in GitLab 15.4](https://gitlab.com/gitlab-org/gitlab/-/issues/390155), if one or more Git repositories in Gitaly Cluster is [unavailable](../../administration/gitaly/recovery.md#unavailable-repositories), then [Repository checks](../../administration/repository_checks.md#repository-checks) and [Geo replication and verification](../../administration/geo/index.md) stop running for all project or project wiki repositories in the affected Gitaly Cluster. The bug was fixed by [reverting the change in GitLab 15.9.0](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/110823). Before upgrading to this version, check if you have any "unavailable" repositories. See [the bug issue](https://gitlab.com/gitlab-org/gitlab/-/issues/390155) for more information.
+- Cloning LFS objects from secondary site downloads from the primary site even when secondary is fully synced. See [the details and workaround](gitlab_16_changes.md#cloning-lfs-objects-from-secondary-site-downloads-from-the-primary-site-even-when-secondary-is-fully-synced).
 
 ## 15.5.5
 
@@ -502,6 +507,7 @@ potentially cause downtime.
 
 - `pg_upgrade` fails to upgrade the bundled PostregSQL database to version 13. See
   [the details and workaround](#pg_upgrade-fails-to-upgrade-the-bundled-postregsql-database-to-version-13).
+- Cloning LFS objects from secondary site downloads from the primary site even when secondary is fully synced. See [the details and workaround](gitlab_16_changes.md#cloning-lfs-objects-from-secondary-site-downloads-from-the-primary-site-even-when-secondary-is-fully-synced).
 
 ## 15.4.6
 
@@ -576,6 +582,7 @@ potentially cause downtime.
 
 - `pg_upgrade` fails to upgrade the bundled PostregSQL database to version 13. See
   [the details and workaround](#pg_upgrade-fails-to-upgrade-the-bundled-postregsql-database-to-version-13).
+- Cloning LFS objects from secondary site downloads from the primary site even when secondary is fully synced. See [the details and workaround](gitlab_16_changes.md#cloning-lfs-objects-from-secondary-site-downloads-from-the-primary-site-even-when-secondary-is-fully-synced).
 
 ## 15.3.4
 
@@ -666,6 +673,7 @@ This issue is resolved in GitLab 15.3.3, so customers with the following configu
 - LFS is enabled.
 - LFS objects are being replicated across Geo sites.
 - Repositories are being pulled by using a Geo secondary site.
+- Cloning LFS objects from secondary site downloads from the primary site even when secondary is fully synced. See [the details and workaround](gitlab_16_changes.md#cloning-lfs-objects-from-secondary-site-downloads-from-the-primary-site-even-when-secondary-is-fully-synced).
 
 #### Incorrect object storage LFS file deletion on secondary sites
 
@@ -722,6 +730,7 @@ A [license caching issue](https://gitlab.com/gitlab-org/gitlab/-/issues/376706)
   [the details and workaround](#lfs-transfers-redirect-to-primary-from-secondary-site-mid-session).
 - Incorrect object storage LFS files deletion on Geo secondary sites. See
   [the details and workaround](#incorrect-object-storage-lfs-file-deletion-on-secondary-sites).
+- Cloning LFS objects from secondary site downloads from the primary site even when secondary is fully synced. See [the details and workaround](gitlab_16_changes.md#cloning-lfs-objects-from-secondary-site-downloads-from-the-primary-site-even-when-secondary-is-fully-synced).
 
 ## 15.1.0
 
@@ -760,6 +769,7 @@ A [license caching issue](https://gitlab.com/gitlab-org/gitlab/-/issues/376706)
   [the details and workaround](#lfs-transfers-redirect-to-primary-from-secondary-site-mid-session).
 - Incorrect object storage LFS files deletion on Geo secondary sites. See
   [the details and workaround](#incorrect-object-storage-lfs-file-deletion-on-secondary-sites).
+- Cloning LFS objects from secondary site downloads from the primary site even when secondary is fully synced. See [the details and workaround](gitlab_16_changes.md#cloning-lfs-objects-from-secondary-site-downloads-from-the-primary-site-even-when-secondary-is-fully-synced).
 
 ## 15.0.0
 
diff --git a/doc/update/versions/gitlab_16_changes.md b/doc/update/versions/gitlab_16_changes.md
index 7c5dd8ae6ae..836f5d188c5 100644
--- a/doc/update/versions/gitlab_16_changes.md
+++ b/doc/update/versions/gitlab_16_changes.md
@@ -30,6 +30,52 @@ For more information about upgrading GitLab Helm Chart, see [the release notes f
   - [Praefect configuration structure change](#praefect-configuration-structure-change).
   - [Gitaly configuration structure change](#gitaly-configuration-structure-change).
 
+## 16.5.0
+
+- Git 2.42.0 and later is required by Gitaly. For self-compiled installations, you should use the [Git version provided by Gitaly](../../install/installation.md#git).
+
+### Geo installations
+
+Specific information applies to installations using Geo:
+
+- A number of Prometheus metrics were incorrectly removed in 16.3.0, which can break dashboards and alerting:
+
+  | Affected metric                          | Metric restored in 16.5.2 and later  | Replacement available in 16.3+                 |
+  | ---------------------------------------- | ------------------------------------ | ---------------------------------------------- |
+  | `geo_repositories_synced`                | Yes                                  | `geo_project_repositories_synced`              |
+  | `geo_repositories_failed`                | Yes                                  | `geo_project_repositories_failed`              |
+  | `geo_repositories_checksummed`           | Yes                                  | `geo_project_repositories_checksummed`         |
+  | `geo_repositories_checksum_failed`       | Yes                                  | `geo_project_repositories_checksum_failed`     |
+  | `geo_repositories_verified`              | Yes                                  | `geo_project_repositories_verified`            |
+  | `geo_repositories_verification_failed`   | Yes                                  | `geo_project_repositories_verification_failed` |
+  | `geo_repositories_checksum_mismatch`     | No                                   | None available                                 |
+  | `geo_repositories_retrying_verification` | No                                   | None available                                 |
+
+  - Impacted versions:
+    - 16.3.0 to 16.5.1
+  - Versions containing fix:
+    - 16.5.2 and later
+
+  For more information, see [issue 429617](https://gitlab.com/gitlab-org/gitlab/-/issues/429617).
+
+- [Object storage verification](https://about.gitlab.com/releases/2023/09/22/gitlab-16-4-released/#geo-verifies-object-storage) was added in GitLab 16.4. Due to an [issue](https://gitlab.com/gitlab-org/gitlab/-/issues/429242) some Geo installations are reporting high memory usage which can lead to the GitLab application on the primary becoming unresponsive. 
+
+  Your installation may be impacted if you have configured it to use [object storage](../../administration/object_storage.md) and have enabled [GitLab-managed object storage replication](../../administration/geo/replication/object_storage.md#enabling-gitlab-managed-object-storage-replication)
+
+  Until this is fixed, the workaround is to disable object storage verification.
+  Run the following command on one of the Rails nodes on the primary site:
+
+  ```shell
+  sudo gitlab-rails runner 'Feature.disable(:geo_object_storage_verification)'
+  ```
+
+  **Affected releases**:
+ 
+  | Affected minor releases | Affected patch releases | Fixed in |
+  | ------ | ------ | ------ |
+  | 16.4   | All    | None   |
+  | 16.5   | All    | None   |
+
 ## 16.4.0
 
 - Updating a group path [received a bug fix](https://gitlab.com/gitlab-org/gitlab/-/issues/419289) that uses a database index introduced in 16.3.
@@ -71,9 +117,33 @@ For more information about upgrading GitLab Helm Chart, see [the release notes f
   SELECT id FROM push_rules WHERE LENGTH(delete_branch_regex) > 511;
   ```
 
+  To find out if a push rule belongs to a project, group, or instance, run this script
+  in the [Rails console](../../administration/operations/rails_console.md#starting-a-rails-console-session):
+  
+  ```ruby
+  # replace `delete_branch_regex` with a name of the field used in constraint
+  long_rules = PushRule.where("length(delete_branch_regex) > 511")
+
+  array = long_rules.map do |lr|
+    if lr.project
+      "Push rule with ID #{lr.id} is configured in a project #{lr.project.full_name}"
+    elsif lr.group
+      "Push rule with ID #{lr.id} is configured in a group #{lr.group.full_name}"
+    else
+      "Push rule with ID #{lr.id} is configured on the instance level"
+    end
+  end
+
+  puts "Total long rules: #{array.count}"
+  puts array.join("\n")
+  ```
+
   Reduce the value length of the regex field for affected push rules records, then
   retry the migration.
 
+  If you have too many affected push rules, and you can't update them through the GitLab UI,
+  contact [GitLab support](https://about.gitlab.com/support/).
+
 ### Self-compiled installations
 
 - A new method of configuring paths for the GitLab secret and custom hooks is preferred in GitLab 16.4 and later:
@@ -82,6 +152,57 @@ For more information about upgrading GitLab Helm Chart, see [the release notes f
      server-side custom hooks.
   1. Remove the `[gitlab-shell] dir` configuration.
 
+### Geo installations
+
+Specific information applies to installations using Geo:
+
+- A number of Prometheus metrics were incorrectly removed in 16.3.0, which can break dashboards and alerting:
+
+  | Affected metric                          | Metric restored in 16.5.2 and later  | Replacement available in 16.3+                 |
+  | ---------------------------------------- | ------------------------------------ | ---------------------------------------------- |
+  | `geo_repositories_synced`                | Yes                                  | `geo_project_repositories_synced`              |
+  | `geo_repositories_failed`                | Yes                                  | `geo_project_repositories_failed`              |
+  | `geo_repositories_checksummed`           | Yes                                  | `geo_project_repositories_checksummed`         |
+  | `geo_repositories_checksum_failed`       | Yes                                  | `geo_project_repositories_checksum_failed`     |
+  | `geo_repositories_verified`              | Yes                                  | `geo_project_repositories_verified`            |
+  | `geo_repositories_verification_failed`   | Yes                                  | `geo_project_repositories_verification_failed` |
+  | `geo_repositories_checksum_mismatch`     | No                                   | None available                                 |
+  | `geo_repositories_retrying_verification` | No                                   | None available                                 |
+
+  - Impacted versions:
+    - 16.3.0 to 16.5.1
+  - Versions containing fix:
+    - 16.5.2 and later
+
+  For more information, see [issue 429617](https://gitlab.com/gitlab-org/gitlab/-/issues/429617).
+
+- [Object storage verification](https://about.gitlab.com/releases/2023/09/22/gitlab-16-4-released/#geo-verifies-object-storage) was added in GitLab 16.4. Due to an [issue](https://gitlab.com/gitlab-org/gitlab/-/issues/429242) some Geo installations are reporting high memory usage which can lead to the GitLab application on the primary becoming unresponsive. 
+
+  Your installation may be impacted if you have configured it to use [object storage](../../administration/object_storage.md) and have enabled [GitLab-managed object storage replication](../../administration/geo/replication/object_storage.md#enabling-gitlab-managed-object-storage-replication)
+
+  Until this is fixed, the workaround is to disable object storage verification.
+  Run the following command on one of the Rails nodes on the primary site:
+
+  ```shell
+  sudo gitlab-rails runner 'Feature.disable(:geo_object_storage_verification)'
+  ```
+
+  **Affected releases**:
+ 
+  | Affected minor releases | Affected patch releases | Fixed in |
+  | ------ | ------ | ------ |
+  | 16.4   | All    | None   |
+  | 16.5   | All    | None   |
+
+- An [issue](https://gitlab.com/gitlab-org/gitlab/-/issues/419370) with sync states getting stuck in pending state results in replication being stuck indefinitely for impacted items leading to risk of data loss in the event of a failover. This mostly impact repository syncs but can also can also affect container registry syncs. You are advised to upgrade to a fixed version to avoid risk of data loss.
+
+  **Affected releases**:
+
+  | Affected minor releases | Affected patch releases | Fixed in |
+  | ------ | ------ | ------ |
+  | 16.3   | 16.3.0 - 16.3.5    | 16.3.6   |
+  | 16.4   | 16.4.0 - 16.4.1    | 16.4.2   |
+
 ## 16.3.0
 
 - **Update to GitLab 16.3.5 or later**. This avoids [issue 425971](https://gitlab.com/gitlab-org/gitlab/-/issues/425971) that causes an excessive use of database disk space for GitLab 16.3.3 and 16.3.4.
@@ -149,6 +270,35 @@ Specific information applies to installations using Geo:
 
   For more information, see [issue 425224](https://gitlab.com/gitlab-org/gitlab/-/issues/425224).
 
+- A number of Prometheus metrics were incorrectly removed in 16.3.0, which can break dashboards and alerting:
+
+  | Affected metric                          | Metric restored in 16.5.2 and later  | Replacement available in 16.3+                 |
+  | ---------------------------------------- | ------------------------------------ | ---------------------------------------------- |
+  | `geo_repositories_synced`                | Yes                                  | `geo_project_repositories_synced`              |
+  | `geo_repositories_failed`                | Yes                                  | `geo_project_repositories_failed`              |
+  | `geo_repositories_checksummed`           | Yes                                  | `geo_project_repositories_checksummed`         |
+  | `geo_repositories_checksum_failed`       | Yes                                  | `geo_project_repositories_checksum_failed`     |
+  | `geo_repositories_verified`              | Yes                                  | `geo_project_repositories_verified`            |
+  | `geo_repositories_verification_failed`   | Yes                                  | `geo_project_repositories_verification_failed` |
+  | `geo_repositories_checksum_mismatch`     | No                                   | None available                                 |
+  | `geo_repositories_retrying_verification` | No                                   | None available                                 |
+
+  - Impacted versions:
+    - 16.3.0 to 16.5.1
+  - Versions containing fix:
+    - 16.5.2 and later
+
+  For more information, see [issue 429617](https://gitlab.com/gitlab-org/gitlab/-/issues/429617).
+
+- An [issue](https://gitlab.com/gitlab-org/gitlab/-/issues/419370) with sync states getting stuck in pending state results in replication being stuck indefinitely for impacted items leading to risk of data loss in the event of a failover. This mostly impact repository syncs but can also can also affect container registry syncs. You are advised to upgrade to a fixed version to avoid risk of data loss.
+
+  **Affected releases**:
+
+  | Affected minor releases | Affected patch releases | Fixed in |
+  | ------ | ------ | ------ |
+  | 16.3   | 16.3.0 - 16.3.5    | 16.3.6   |
+  | 16.4   | 16.4.0 - 16.4.1    | 16.4.2   |
+
 ## 16.2.0
 
 - Legacy LDAP configuration settings may cause
@@ -227,6 +377,24 @@ Specific information applies to installations using Geo:
     Affected artifacts are automatically resynced upon upgrade to 16.1.5, 16.2.5, 16.3.1, 16.4.0, or later.
     You can [manually resync affected job artifacts](https://gitlab.com/gitlab-org/gitlab/-/issues/419742#to-fix-data) if needed.
 
+#### Cloning LFS objects from secondary site downloads from the primary site even when secondary is fully synced
+
+A [bug](https://gitlab.com/gitlab-org/gitlab/-/issues/410413) in the Geo proxying logic for LFS objects meant that all LFS clone requests against a secondary site are proxied to the primary even if the secondary site is up-to-date. This can result in increased load on the primary site and longer access times for LFS objects for users cloning from the secondary site.
+
+In GitLab 15.1 proxying was enabled by default.
+
+You are not impacted:
+
+- If your installation is not configured to use LFS objects
+- If you do not use Geo to accelerate remote users
+- If you are using Geo to accelerate remote users but have disabled proxying
+
+| Affected minor releases | Affected patch releases | Fixed in |
+|-------------------------|-------------------------|----------|
+| 15.1 - 16.2             | All                     | 16.3 and later    |
+
+Workaround: A possible workaround is to [disable proxying](../../administration/geo/secondary_proxy/index.md#disable-geo-proxying). Note that the secondary site fails to serve LFS files that have not been replicated at the time of cloning.
+
 ## 16.1.0
 
 - A `BackfillPreparedAtMergeRequests` background migration is finalized with
@@ -273,6 +441,7 @@ Specific information applies to installations using Geo:
   - While running an affected version, artifacts which appeared to become synced may actually be missing on the secondary site.
     Affected artifacts are automatically resynced upon upgrade to 16.1.5, 16.2.5, 16.3.1, 16.4.0, or later.
     You can [manually resync affected job artifacts](https://gitlab.com/gitlab-org/gitlab/-/issues/419742#to-fix-data) if needed.
+  - Cloning LFS objects from secondary site downloads from the primary site even when secondary is fully synced. See [the details and workaround](#cloning-lfs-objects-from-secondary-site-downloads-from-the-primary-site-even-when-secondary-is-fully-synced).
 
 #### Wiki repositories not initialized on project creation
 
@@ -302,6 +471,7 @@ by this issue.
   [throw errors on startup](../../install/docker.md#threaderror-cant-create-thread-operation-not-permitted).
 - Starting with 16.0, GitLab self-managed installations now have two database connections by default, instead of one. This change doubles the number of PostgreSQL connections. It makes self-managed versions of GitLab behave similarly to GitLab.com, and is a step toward enabling a separate database for CI features for self-managed versions of GitLab. Before upgrading to 16.0, determine if you need to [increase max connections for PostgreSQL](https://docs.gitlab.com/omnibus/settings/database.html#configuring-multiple-database-connections).
   - This change applies to installation methods with Linux packages (Omnibus), GitLab Helm chart, GitLab Operator, GitLab Docker images, and self-compiled installations.
+- Container registry using Azure storage might be empty with zero tags. You can fix this by following the [breaking change instructions](../deprecations.md#azure-storage-driver-defaults-to-the-correct-root-prefix). 
 
 ### Linux package installations
 
@@ -334,6 +504,7 @@ Specific information applies to installations using Geo:
 
 - Some project imports do not initialize wiki repositories on project creation. See
   [the details and workaround](#wiki-repositories-not-initialized-on-project-creation).
+- Cloning LFS objects from secondary site downloads from the primary site even when secondary is fully synced. See [the details and workaround](#cloning-lfs-objects-from-secondary-site-downloads-from-the-primary-site-even-when-secondary-is-fully-synced).
 
 ### Gitaly configuration structure change