Add latest changes from gitlab-org/gitlab@14-8-stable-eev14.8.0-rc42

10 files changed, 115 insertions, 40 deletions

diff --git a/doc/administration/geo/glossary.md b/doc/administration/geo/glossary.md
index f8769d31ec7..a18e78a5e01 100644
--- a/doc/administration/geo/glossary.md
+++ b/doc/administration/geo/glossary.md
@@ -25,8 +25,8 @@ these definitions yet.
 | Site                      | One or a collection of nodes running a single GitLab application. A site can be single-node or multi-node.                                                                             | GitLab       | deployment, installation instance               |
 | Single-node site          | A specific configuration of GitLab that uses exactly one node.                                                                                                                     | GitLab       | single-server, single-instance
 | Multi-node site           | A specific configuration of GitLab that uses more than one node.                                                                                                                   | GitLab       | multi-server, multi-instance, high availability |
-| Primary site              | A GitLab site that is configured to be read and writable. There can only be a single primary site.                                                                                     | Geo-specific | Geo deployment, Primary node                    |
-| Secondary site(s)         | GitLab site that is configured to be read-only. There can be one or more secondary sites.                                                                                              | Geo-specific | Geo deployment, Secondary node                  |
+| Primary site              | A GitLab site whose data is being replicated by at least one secondary site. There can only be a single primary site.                                                                  | Geo-specific | Geo deployment, Primary node                    |
+| Secondary site(s)         | A GitLab site that is configured to replicate the data of a primary site. There can be one or more secondary sites.                                                                    | Geo-specific | Geo deployment, Secondary node                  |
 | Geo deployment            | A collection of two or more GitLab sites with exactly one primary site being replicated by one or more secondary sites.                                                                | Geo-specific |                                                 |
 | Reference architecture(s) | A [specified configuration of GitLab for a number of users](../reference_architectures/index.md), possibly including multiple nodes and multiple sites. | GitLab       |                                                 |
 | Promoting                 | Changing the role of a site from secondary to primary.                                                                                                                                 | Geo-specific |                                                 |
diff --git a/doc/administration/geo/index.md b/doc/administration/geo/index.md
index b5eb0ba5841..c4164284e97 100644
--- a/doc/administration/geo/index.md
+++ b/doc/administration/geo/index.md
@@ -204,6 +204,7 @@ This list of limitations only reflects the latest version of GitLab. If you are
 - Real-time updates of issues/merge requests (for example, via long polling) doesn't work on the **secondary** site.
 - GitLab Runners cannot register with a **secondary** site. Support for this is [planned for the future](https://gitlab.com/gitlab-org/gitlab/-/issues/3294).
 - [Selective synchronization](replication/configuration.md#selective-synchronization) only limits what repositories and files are replicated. The entire PostgreSQL data is still replicated. Selective synchronization is not built to accommodate compliance / export control use cases.
+- [Pages access control](../../user/project/pages/pages_access_control.md) doesn't work on secondaries. See [GitLab issue #9336](https://gitlab.com/gitlab-org/gitlab/-/issues/9336) for details.
 
 ### Limitations on replication/verification
 
diff --git a/doc/administration/geo/replication/datatypes.md b/doc/administration/geo/replication/datatypes.md
index 0fa08dcc9e0..5beb2479c57 100644
--- a/doc/administration/geo/replication/datatypes.md
+++ b/doc/administration/geo/replication/datatypes.md
@@ -39,7 +39,7 @@ verification methods:
 | Blobs    | User uploads _(object storage)_                 | Geo with API/Managed (*2*)            | _Not implemented_      |
 | Blobs    | LFS objects _(file system)_                     | Geo with API                          | SHA256 checksum        |
 | Blobs    | LFS objects _(object storage)_                  | Geo with API/Managed (*2*)            | _Not implemented_      |
-| Blobs    | CI job artifacts _(file system)_                | Geo with API                          | _Not implemented_      |
+| Blobs    | CI job artifacts _(file system)_                | Geo with API                          | SHA256 checksum      |
 | Blobs    | CI job artifacts _(object storage)_             | Geo with API/Managed (*2*)            | _Not implemented_      |
 | Blobs    | Archived CI build traces _(file system)_        | Geo with API                          | _Not implemented_      |
 | Blobs    | Archived CI build traces _(object storage)_     | Geo with API/Managed (*2*)            | _Not implemented_      |
diff --git a/doc/administration/geo/replication/geo_validation_tests.md b/doc/administration/geo/replication/geo_validation_tests.md
index ce1bd8a9d3c..70b3eaf5fea 100644
--- a/doc/administration/geo/replication/geo_validation_tests.md
+++ b/doc/administration/geo/replication/geo_validation_tests.md
@@ -118,7 +118,7 @@ The following are PostgreSQL upgrade validation tests we performed.
 [Verify Geo installation with PostgreSQL 13](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/6131):
 
 - Description: With PostgreSQL 13 available as an opt-in version in GitLab 14.1, we tested fresh installations of GitLab with Geo when PostgreSQL 13 is enabled.
-- Outcome: Successfully built an environment with Geo and PostgreSQL 13 using [GitLab Environment Toolkit](https://gitlab.com/gitlab-org/quality/gitlab-environment-toolkit) and performed Geo QA tests against the environment without failures.
+- Outcome: Successfully built an environment with Geo and PostgreSQL 13 using [GitLab Environment Toolkit](https://gitlab.com/gitlab-org/gitlab-environment-toolkit) and performed Geo QA tests against the environment without failures.
 
 ### September 2020
 
@@ -200,3 +200,12 @@ The following are additional validation tests we performed.
 
 - Description: Tested a Geo deployment with Gitaly clusters configured on both the primary and secondary Geo sites. Triggered automatic Gitaly cluster failover on the primary Geo site, and ran end-to-end Geo tests. Then triggered Gitaly cluster failover on the secondary Geo site, and re-ran the end-to-end Geo tests.
 - Outcome: Successful end-to-end tests before and after Gitaly cluster failover on the primary site, and before and after Gitaly cluster failover on the secondary site.
+
+### January 2022
+
+[Validate Object storage replication using Azure based object storage](https://gitlab.com/gitlab-org/gitlab/-/issues/348804#note_821294631):
+
+- Description: Tested the average time it takes for a single image to replicate from the primary object storage location to the secondary when using Azure based object storage replication and [GitLab based object storage replication](object_storage.md#enabling-gitlab-managed-object-storage-replication). This was tested by uploading a 1mb image to a project on the primary site every second for 60 seconds. The time was then measured until a image was available on the secondary site. This was achieved using a [Ruby Script](https://gitlab.com/gitlab-org/quality/geo-replication-tester).
+- Outcome: When using Azure based replication the average time for an image to replicate from the primary object storage to the secondary was recorded as 40 seconds, the longest replication time was 70 seconds and the quickest was 11 seconds. When using GitLab based replication the average time for replication to complete was 5 seconds, the longest replication time was 10 seconds and the quickest was 3 seconds.
+- Follow up issue:
+  - [Test and validate object storage replication performance on reference architectures](https://gitlab.com/gitlab-org/gitlab/-/issues/347314)
diff --git a/doc/administration/geo/replication/location_aware_git_url.md b/doc/administration/geo/replication/location_aware_git_url.md
index ce1af27611e..8689fac987f 100644
--- a/doc/administration/geo/replication/location_aware_git_url.md
+++ b/doc/administration/geo/replication/location_aware_git_url.md
@@ -7,6 +7,12 @@ type: howto
 
 # Location-aware Git remote URL with AWS Route53 **(PREMIUM SELF)**
 
+NOTE:
+Since GitLab 14.6,
+[GitLab Geo supports a location-aware URL including web UI and API traffic.](../secondary_proxy/location_aware_external_url.md)
+This configuration is recommended over the location-aware Git remote URL
+described in this document.
+
 You can provide GitLab users with a single remote URL that automatically uses
 the Geo site closest to them. This means users don't need to update their Git
 configuration to take advantage of closer Geo sites as they move.
@@ -18,12 +24,6 @@ Though these instructions use [AWS Route53](https://aws.amazon.com/route53/),
 other services such as [Cloudflare](https://www.cloudflare.com/) could be used
 as well.
 
-NOTE:
-You can also use a load balancer to distribute web UI or API traffic to
-[multiple Geo **secondary** sites](../../../user/admin_area/geo_nodes.md#multiple-secondary-sites-behind-a-load-balancer).
-Importantly, the **primary** site cannot yet be included. See the feature request
-[Support putting the **primary** behind a Geo node load balancer](https://gitlab.com/gitlab-org/gitlab/-/issues/10888).
-
 ## Prerequisites
 
 In this example, we have already set up:
diff --git a/doc/administration/geo/replication/troubleshooting.md b/doc/administration/geo/replication/troubleshooting.md
index a6ea41171a9..1371e5d84c8 100644
--- a/doc/administration/geo/replication/troubleshooting.md
+++ b/doc/administration/geo/replication/troubleshooting.md
@@ -115,35 +115,39 @@ http://secondary.example.com/
 
 To check if PostgreSQL replication is working, check if:
 
-- [Nodes are pointing to the correct database instance](#are-nodes-pointing-to-the-correct-database-instance).
-- [Geo can detect the current node correctly](#can-geo-detect-the-current-node-correctly).
+- [Sites are pointing to the correct database node](#are-sites-pointing-to-the-correct-database-node).
+- [Geo can detect the current site correctly](#can-geo-detect-the-current-site-correctly).
 
-#### Are nodes pointing to the correct database instance?
+#### Are sites pointing to the correct database node?
 
-You should make sure your **primary** Geo node points to the instance with
-writing permissions.
+You should make sure your **primary** Geo [site](../glossary.md) points to
+the database node that has write permissions.
 
-Any **secondary** nodes should point only to read-only instances.
+Any **secondary** sites should point only to read-only database nodes.
 
-#### Can Geo detect the current node correctly?
+#### Can Geo detect the current site correctly?
 
-Geo finds the current machine's Geo node name in `/etc/gitlab/gitlab.rb` by:
+Geo finds the current Puma or Sidekiq node's Geo [site](../glossary.md) name in
+`/etc/gitlab/gitlab.rb` with the following logic:
 
-- Using the `gitlab_rails['geo_node_name']` setting.
-- If that is not defined, using the `external_url` setting.
+1. Get the "Geo node name" (there is
+   [an issue to rename the settings to "Geo site name"](https://gitlab.com/gitlab-org/gitlab/-/issues/335944)):
+   - Omnibus GitLab: Get the `gitlab_rails['geo_node_name']` setting.
+   - GitLab Helm Charts: Get the `global.geo.nodeName` setting (see [Charts with GitLab Geo](https://docs.gitlab.com/charts/advanced/geo/index.html)).
+1. If that is not defined, then get the `external_url` setting.
 
-This name is used to look up the node with the same **Name** in the **Geo Nodes**
+This name is used to look up the Geo site with the same **Name** in the **Geo Sites**
 dashboard.
 
-To check if the current machine has a node name that matches a node in the
+To check if the current machine has a site name that matches a site in the
 database, run the check task:
 
 ```shell
 sudo gitlab-rake gitlab:geo:check
 ```
 
-It displays the current machine's node name and whether the matching database
-record is a **primary** or **secondary** node.
+It displays the current machine's site name and whether the matching database
+record is a **primary** or **secondary** site.
 
 ```plaintext
 This machine's Geo node name matches a database record ... yes, found a secondary node named "Shanghai"
@@ -158,6 +162,9 @@ This machine's Geo node name matches a database record ... no
   doc/administration/geo/replication/troubleshooting.md#can-geo-detect-the-current-node-correctly
 ```
 
+Learn more about recommended site names in the description of the Name field in
+[Geo Admin Area Common Settings](../../../user/admin_area/geo_nodes.md#common-settings).
+
 ### Message: `WARNING: oldest xmin is far in the past` and `pg_wal` size growing
 
 If a replication slot is inactive,
@@ -692,6 +699,8 @@ determine the actual replication status of Design repositories.
 
 ### Sync failure message: "Verification failed with: Error during verification: File is not checksummable"
 
+#### Missing files on the Geo primary site
+
 In GitLab 14.5 and earlier, certain data types which were missing on the Geo primary site were marked as "synced" on Geo secondary sites. This was because from the perspective of Geo secondary sites, the state matched the primary site and nothing more could be done on secondary sites.
 
 Secondaries would regularly try to sync these files again by using the "verification" feature:
@@ -745,6 +754,32 @@ This behavior affects only the following data types through GitLab 14.6:
 to make Geo visibly surface data loss risks. The sync/verification loop is
 therefore short-circuited. `last_sync_failure` is now set to `The file is missing on the Geo primary site`.
 
+#### Failed syncs with GitLab-managed object storage replication
+
+There is [an issue in GitLab 14.2 through 14.7](https://gitlab.com/gitlab-org/gitlab/-/issues/299819#note_822629467)
+that affects Geo when the GitLab-managed object storage replication is used, causing blob object types to fail synchronization.
+
+Since GitLab 14.2, verification failures result in synchronization failures and cause
+a re-synchronization of these objects.
+
+As verification is not implemented for files stored in object storage (see
+[issue 13845](https://gitlab.com/gitlab-org/gitlab/-/issues/13845) for more details), this
+results in a loop that consistently fails for all objects stored in object storage.
+
+You can work around this by marking the objects as synced and succeeded verification, however
+be aware that can also mark objects that may be
+[missing from the primary](#missing-files-on-the-geo-primary-site).
+
+To do that, enter the [Rails console](../../troubleshooting/navigating_gitlab_via_rails_console.md)
+and run:
+
+```ruby
+Gitlab::Geo.verification_enabled_replicator_classes.each do |klass|
+  updated = klass.registry_class.failed.where(last_sync_failure: "Verification failed with: Error during verification: File is not checksummable").update_all(verification_checksum: '0000000000000000000000000000000000000000', verification_state: 2, verification_failure: nil, verification_retry_at: nil, state: 2, last_sync_failure: nil, retry_at: nil, verification_retry_count: 0, retry_count: 0)
+  pp "Updated #{updated} #{klass.replicable_name_plural}"
+end
+```
+
 ## Fixing errors during a failover or when promoting a secondary to a primary node
 
 The following are possible error messages that might be encountered during failover or
diff --git a/doc/administration/geo/replication/using_a_geo_server.md b/doc/administration/geo/replication/using_a_geo_server.md
index 04c30514a89..62562a1149d 100644
--- a/doc/administration/geo/replication/using_a_geo_server.md
+++ b/doc/administration/geo/replication/using_a_geo_server.md
@@ -1,9 +1,9 @@
 ---
 redirect_to: '../../geo/replication/usage.md'
-remove_date: '2022-06-01'
+remove_date: '2022-02-01'
 ---
 
 This document was moved to [another location](../../geo/replication/usage.md).
 
-<!-- This redirect file can be deleted after 2022-06-01 -->
+<!-- This redirect file can be deleted after 2022-02-01 -->
 <!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/#move-or-rename-a-page -->
diff --git a/doc/administration/geo/replication/version_specific_updates.md b/doc/administration/geo/replication/version_specific_updates.md
index d3a132a6666..95833a290dd 100644
--- a/doc/administration/geo/replication/version_specific_updates.md
+++ b/doc/administration/geo/replication/version_specific_updates.md
@@ -8,7 +8,22 @@ info: To determine the technical writer assigned to the Stage/Group associated w
 
 Review this page for update instructions for your version. These steps
 accompany the [general steps](updating_the_geo_sites.md#general-update-steps)
-for updating Geo nodes.
+for updating Geo sites.
+
+## Updating to 14.2 through 14.7
+
+There is [an issue in GitLab 14.2 through 14.7](https://gitlab.com/gitlab-org/gitlab/-/issues/299819#note_822629467)
+that affects Geo when the GitLab-managed object storage replication is used, causing blob object types to fail synchronization.
+
+Since GitLab 14.2, verification failures result in synchronization failures and cause
+a resynchronization of these objects.
+
+As verification is not yet implemented for files stored in object storage (see
+[issue 13845](https://gitlab.com/gitlab-org/gitlab/-/issues/13845) for more details), this
+results in a loop that consistently fails for all objects stored in object storage.
+
+For information on how to fix this, see
+[Troubleshooting - Failed syncs with GitLab-managed object storage replication](troubleshooting.md#failed-syncs-with-gitlab-managed-object-storage-replication).
 
 ## Updating to 14.4
 
@@ -18,7 +33,7 @@ There is [an issue in GitLab 14.4.0 through 14.4.2](../../../update/index.md#144
 
 ### Multi-arch images
 
-We found an [issue](https://gitlab.com/gitlab-org/gitlab/-/issues/336013) where the Container Registry replication wasn't fully working if you used multi-arch images. In case of a multi-arch image, only the primary architecture (for example `amd64`) would be replicated to the secondary node. This has been [fixed in GitLab 14.3](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/67624) and was backported to 14.2 and 14.1, but manual steps are required to force a re-sync.
+We found an [issue](https://gitlab.com/gitlab-org/gitlab/-/issues/336013) where the Container Registry replication wasn't fully working if you used multi-arch images. In case of a multi-arch image, only the primary architecture (for example `amd64`) would be replicated to the secondary site. This has been [fixed in GitLab 14.3](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/67624) and was backported to 14.2 and 14.1, but manual steps are required to force a re-sync.
 
 You can check if you are affected by running:
 
@@ -26,12 +41,12 @@ You can check if you are affected by running:
 docker manifest inspect <SECONDARY_IMAGE_LOCATION> | jq '.mediaType'
 ```
 
-Where `<SECONDARY_IMAGE_LOCATION>` is a container image on your secondary node.
+Where `<SECONDARY_IMAGE_LOCATION>` is a container image on your secondary site.
 If the output matches `application/vnd.docker.distribution.manifest.list.v2+json`
 (there can be a `mediaType` entry at several levels, we only care about the top level entry),
 then you don't need to do anything.
 
-Otherwise, on all your **secondary** nodes, in a [Rails console](../../operations/rails_console.md), run the following:
+Otherwise, for each **secondary** site, on a Rails application node, open a [Rails console](../../operations/rails_console.md), and run the following:
 
  ```ruby
  list_type = 'application/vnd.docker.distribution.manifest.list.v2+json'
@@ -63,7 +78,7 @@ We found an issue where [Primary sites can not be removed from the UI](https://g
 
 This bug only exists in the UI and does not block the removal of Primary sites using any other method.
 
-If you are running an affected version and need to remove your Primary site, you can manually remove the Primary site by using the [Geo Nodes API](../../../api/geo_nodes.md#delete-a-geo-node).
+If you are running an affected version and need to remove your Primary site, you can manually remove the Primary site by using the [Geo Sites API](../../../api/geo_nodes.md#delete-a-geo-node).
 
 ### Geo Admin Area shows 'Unhealthy' after enabling Maintenance Mode
 
@@ -71,9 +86,9 @@ GitLab 13.9 through GitLab 14.3 are affected by a bug in which enabling [GitLab
 
 ## Updating to GitLab 13.12
 
-### Secondary nodes re-download all LFS files upon update
+### Secondary sites re-download all LFS files upon update
 
-We found an issue where [secondary nodes re-download all LFS files](https://gitlab.com/gitlab-org/gitlab/-/issues/334550) upon update. This bug:
+We found an issue where [secondary sites re-download all LFS files](https://gitlab.com/gitlab-org/gitlab/-/issues/334550) upon update. This bug:
 
 - Only applies to Geo secondary sites that have replicated LFS objects.
 - Is _not_ a data loss risk.
@@ -172,7 +187,7 @@ In GitLab 13.3, Geo removed the PostgreSQL [Foreign Data Wrapper](https://www.po
 dependency for the tracking database.
 
 The FDW server, user, and the extension is removed during the upgrade
-process on each secondary node. The GitLab settings related to the FDW in the
+process on each secondary site. The GitLab settings related to the FDW in the
 `/etc/gitlab/gitlab.rb`  have been deprecated and can be safely removed.
 
 There are some scenarios like using an external PostgreSQL instance for the
@@ -185,9 +200,9 @@ DROP EXTENSION IF EXISTS postgres_fdw;
 ```
 
 WARNING:
-In GitLab 13.3, promoting a secondary node to a primary while the secondary is
+In GitLab 13.3, promoting a secondary site to a primary while the secondary is
 paused fails. Do not pause replication before promoting a secondary. If the
-node is paused, be sure to resume before promoting. To avoid this issue,
+site is paused, be sure to resume before promoting. To avoid this issue,
 upgrade to GitLab 13.4 or later.
 
 WARNING:
@@ -198,9 +213,9 @@ contain a workaround if you run into errors during the failover.
 
 ## Updating to GitLab 13.2
 
-In GitLab 13.2, promoting a secondary node to a primary while the secondary is
+In GitLab 13.2, promoting a secondary site to a primary while the secondary is
 paused fails. Do not pause replication before promoting a secondary. If the
-node is paused, be sure to resume before promoting. To avoid this issue,
+site is paused, be sure to resume before promoting. To avoid this issue,
 upgrade to GitLab 13.4 or later.
 
 ## Updating to GitLab 13.0
@@ -375,5 +390,5 @@ For the recommended procedure, see the
 
 WARNING:
 This version is affected by a [bug that results in new LFS objects not being
-replicated to Geo secondary nodes](https://gitlab.com/gitlab-org/gitlab/-/issues/32696).
+replicated to Geo secondary sites](https://gitlab.com/gitlab-org/gitlab/-/issues/32696).
 The issue is fixed in GitLab 12.1. Be sure to upgrade to GitLab 12.1 or later.
diff --git a/doc/administration/geo/secondary_proxy/index.md b/doc/administration/geo/secondary_proxy/index.md
index ebd71757e91..c8dcbb81312 100644
--- a/doc/administration/geo/secondary_proxy/index.md
+++ b/doc/administration/geo/secondary_proxy/index.md
@@ -140,6 +140,18 @@ for details.
   To use TLS certificates with Let's Encrypt, you can manually point the domain to one of the Geo sites, generate
   the certificate, then copy it to all other sites.
 
+## Behavior of secondary sites when the primary Geo site is down
+
+Considering that web traffic is proxied to the primary, the behavior of the secondary sites differs when the primary
+site is inaccessible:
+
+- UI and API traffic return the same errors as the primary (or fail if the primary is not accessible at all), since they are proxied.
+- For repositories that already exist on the specific secondary site being accessed, Git read operations still work as expected,
+  including authentication through HTTP(s) or SSH.
+- Git operations for repositories that are not replicated to the secondary site return the same errors
+  as the primary site, since they are proxied.
+- All Git write operations return the same errors as the primary site, since they are proxied.
+
 ## Features accelerated by secondary Geo sites
 
 Most HTTP traffic sent to a secondary Geo site can be proxied to the primary Geo site. With this architecture,
diff --git a/doc/administration/geo/setup/database.md b/doc/administration/geo/setup/database.md
index 7f4c771c962..a8bebef8f44 100644
--- a/doc/administration/geo/setup/database.md
+++ b/doc/administration/geo/setup/database.md
@@ -40,7 +40,7 @@ instructions on setting up replication with a Patroni cluster.
 
 The GitLab **primary** node where the write operations happen connects to
 the **primary** database server, and **secondary** nodes
-connect to their own database servers (which are also read-only).
+connect to their own database servers (which are read-only).
 
 We recommend using [PostgreSQL replication slots](https://medium.com/@tk512/replication-slots-in-postgresql-b4b03d277c75)
 to ensure that the **primary** node retains all the data necessary for the **secondary** nodes to
@@ -893,6 +893,9 @@ Instead, follow the instructions below.
 A production-ready and secure setup requires at least three Consul nodes, two
 Patroni nodes and one PgBouncer node on the secondary site.
 
+Because of [omnibus-6587](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/6587), Consul can't track multiple
+services, so these need to be different than the nodes used for the Standby Cluster database.
+
 Be sure to use [password credentials](../../postgresql/replication_and_failover.md#database-authorization-for-patroni)
 and other database best practices.