diff options
| author | GitLab Bot <gitlab-bot@gitlab.com> | 2023-12-19 14:01:45 +0300 |
|---|---|---|
| committer | GitLab Bot <gitlab-bot@gitlab.com> | 2023-12-19 14:01:45 +0300 |
| commit | 9297025d0b7ddf095eb618dfaaab2ff8f2018d8b (patch) | |
| tree | 865198c01d1824a9b098127baa3ab980c9cd2c06 /doc/administration/geo | |
| parent | 6372471f43ee03c05a7c1f8b0c6ac6b8a7431dbe (diff) | |
Add latest changes from gitlab-org/gitlab@16-7-stable-eev16.7.0-rc42
Diffstat (limited to 'doc/administration/geo')
31 files changed, 236 insertions, 196 deletions
diff --git a/doc/administration/geo/disaster_recovery/background_verification.md b/doc/administration/geo/disaster_recovery/background_verification.md index 8aec77d9d88..a927450d80f 100644 --- a/doc/administration/geo/disaster_recovery/background_verification.md +++ b/doc/administration/geo/disaster_recovery/background_verification.md @@ -1,7 +1,7 @@ --- stage: Systems group: Geo -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://handbook.gitlab.com/handbook/product/ux/technical-writing/#assignments --- # Automatic background verification **(PREMIUM SELF)** @@ -27,9 +27,8 @@ the site more time before scheduling a planned failover. On the **primary** site: -1. On the left sidebar, select **Search or go to**. -1. Select **Admin Area**. -1. On the left sidebar, select **Geo > Sites**. +1. On the left sidebar, at the bottom, select **Admin Area**. +1. Select **Geo > Sites**. 1. Expand **Verification information** tab for that site to view automatic checksumming status for repositories and wikis. Successes are shown in green, pending work in gray, and failures in red. @@ -38,9 +37,8 @@ On the **primary** site: On the **secondary** site: -1. On the left sidebar, select **Search or go to**. -1. Select **Admin Area**. -1. On the left sidebar, select **Geo > Sites**. +1. On the left sidebar, at the bottom, select **Admin Area**. +1. Select **Geo > Sites**. 1. Expand **Verification information** tab for that site to view automatic checksumming status for repositories and wikis. Successes are shown in green, pending work in gray, and failures in red. @@ -67,9 +65,8 @@ increase load and vice versa. On the **primary** site: -1. On the left sidebar, select **Search or go to**. -1. Select **Admin Area**. -1. On the left sidebar, select **Geo > Sites**. +1. On the left sidebar, at the bottom, select **Admin Area**. +1. Select **Geo > Sites**. 1. Select **Edit** for the **primary** site to customize the minimum re-verification interval: @@ -101,8 +98,7 @@ sudo gitlab-rake geo:verification:wiki:reset If the **primary** and **secondary** sites have a checksum verification mismatch, the cause may not be apparent. To find the cause of a checksum mismatch: 1. On the **primary** site: - 1. On the left sidebar, select **Search or go to**. - 1. Select **Admin Area**. + 1. On the left sidebar, at the bottom, select **Admin Area**. 1. On the left sidebar, select **Overview > Projects**. 1. Find the project that you want to check the checksum differences and select its name. diff --git a/doc/administration/geo/disaster_recovery/bring_primary_back.md b/doc/administration/geo/disaster_recovery/bring_primary_back.md index 5f2cbd4d03b..be4f4d3f6a4 100644 --- a/doc/administration/geo/disaster_recovery/bring_primary_back.md +++ b/doc/administration/geo/disaster_recovery/bring_primary_back.md @@ -1,8 +1,7 @@ --- stage: Systems group: Geo -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments -type: howto +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://handbook.gitlab.com/handbook/product/ux/technical-writing/#assignments --- # Bring a demoted primary site back online **(PREMIUM SELF)** diff --git a/doc/administration/geo/disaster_recovery/index.md b/doc/administration/geo/disaster_recovery/index.md index 2f636dc6ba4..f97c892c3eb 100644 --- a/doc/administration/geo/disaster_recovery/index.md +++ b/doc/administration/geo/disaster_recovery/index.md @@ -1,7 +1,7 @@ --- stage: Systems group: Geo -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://handbook.gitlab.com/handbook/product/ux/technical-writing/#assignments --- # Disaster Recovery (Geo) **(PREMIUM SELF)** diff --git a/doc/administration/geo/disaster_recovery/planned_failover.md b/doc/administration/geo/disaster_recovery/planned_failover.md index cdc52b55c3e..01e5c33a726 100644 --- a/doc/administration/geo/disaster_recovery/planned_failover.md +++ b/doc/administration/geo/disaster_recovery/planned_failover.md @@ -1,8 +1,7 @@ --- stage: Systems group: Geo -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments -type: howto +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://handbook.gitlab.com/handbook/product/ux/technical-writing/#assignments --- # Disaster recovery for planned failover **(PREMIUM SELF)** @@ -45,7 +44,7 @@ be adapted for use with any other file-based data. ### Container registry By default, the container registry is not automatically replicated to secondary -sites and this needs to be manually configured, see [Container Registry for a secondary site](../replication/container_registry.md). +sites and this needs to be manually configured, see [container registry for a secondary site](../replication/container_registry.md). If you are using local storage on your current primary site for the container registry, you can `rsync` the container registry objects to the secondary @@ -154,9 +153,8 @@ ensure these processes are close to 100% as possible during active use. On the **secondary** site: -1. On the left sidebar, select **Search or go to**. -1. Select **Admin Area**. -1. On the left sidebar, select **Geo > Sites**. +1. On the left sidebar, at the bottom, select **Admin Area**. +1. Select **Geo > Sites**. Replicated objects (shown in green) should be close to 100%, and there should be no failures (shown in red). If a large proportion of objects aren't yet replicated (shown in gray), consider giving the site more @@ -183,9 +181,8 @@ This [content was moved to another location](background_verification.md). On the **primary** site: -1. On the left sidebar, select **Search or go to**. -1. Select **Admin Area**. -1. On the left sidebar, select **Messages**. +1. On the left sidebar, at the bottom, select **Admin Area**. +1. Select **Messages**. 1. Add a message notifying users on the maintenance window. You can check under **Geo > Sites** to estimate how long it takes to finish syncing. @@ -197,9 +194,8 @@ To ensure that all data is replicated to a secondary site, updates (write reques be disabled on the **primary** site: 1. Enable [maintenance mode](../../maintenance_mode/index.md) on the **primary** site. -1. On the left sidebar, select **Search or go to**. -1. Select **Admin Area**. -1. On the left sidebar, select **Monitoring > Background Jobs**. +1. On the left sidebar, at the bottom, select **Admin Area**. +1. Select **Monitoring > Background Jobs**. 1. On the Sidekiq dashboard, select **Cron**. 1. Select `Disable All` to disable non-Geo periodic background jobs. 1. Select `Enable` for the `geo_sidekiq_cron_config_worker` cron job. @@ -211,8 +207,7 @@ be disabled on the **primary** site: 1. If you are manually replicating any data not managed by Geo, trigger the final replication process now. 1. On the **primary** site: - 1. On the left sidebar, select **Search or go to**. - 1. Select **Admin Area**. + 1. On the left sidebar, at the bottom, select **Admin Area**. 1. On the left sidebar, select **Monitoring > Background Jobs**. 1. On the Sidekiq dashboard, select **Queues**, and wait for all queues except those with `geo` in the name to drop to 0. @@ -227,8 +222,7 @@ be disabled on the **primary** site: - The Geo log cursor is up to date (0 events behind). 1. On the **secondary** site: - 1. On the left sidebar, select **Search or go to**. - 1. Select **Admin Area**. + 1. On the left sidebar, at the bottom, select **Admin Area**. 1. On the left sidebar, select **Monitoring > Background Jobs**. 1. On the Sidekiq dashboard, select **Queues**, and wait for all the `geo` queues to drop to 0 queued and 0 running jobs. diff --git a/doc/administration/geo/disaster_recovery/runbooks/planned_failover_multi_node.md b/doc/administration/geo/disaster_recovery/runbooks/planned_failover_multi_node.md index 787e7c4669e..f24d2bba2a3 100644 --- a/doc/administration/geo/disaster_recovery/runbooks/planned_failover_multi_node.md +++ b/doc/administration/geo/disaster_recovery/runbooks/planned_failover_multi_node.md @@ -1,8 +1,7 @@ --- stage: Systems group: Geo -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments -type: howto +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://handbook.gitlab.com/handbook/product/ux/technical-writing/#assignments --- WARNING: @@ -65,9 +64,8 @@ promote a Geo replica and perform a failover. On the **secondary** site: -1. On the left sidebar, select **Search or go to**. -1. Select **Admin Area**. -1. On the left sidebar, select **Geo > Sites** to see its status. +1. On the left sidebar, at the bottom, select **Admin Area**. +1. Select **Geo > Sites** to see its status. Replicated objects (shown in green) should be close to 100%, and there should be no failures (shown in red). If a large proportion of objects aren't replicated (shown in gray), consider giving the site more @@ -105,8 +103,7 @@ follow these steps to avoid unnecessary data loss: [data not managed by Geo](../../replication/datatypes.md#limitations-on-replicationverification), trigger the final replication process now. 1. On the **primary** site: - 1. On the left sidebar, select **Search or go to**. - 1. Select **Admin Area**. + 1. On the left sidebar, at the bottom, select **Admin Area**. 1. On the left sidebar, select **Monitoring > Background Jobs**. 1. On the Sidekiq dashboard, select **Queues**, and wait for all queues except those with `geo` in the name to drop to 0. @@ -121,8 +118,7 @@ follow these steps to avoid unnecessary data loss: - The Geo log cursor is up to date (0 events behind). 1. On the **secondary** site: - 1. On the left sidebar, select **Search or go to**. - 1. Select **Admin Area**. + 1. On the left sidebar, at the bottom, select **Admin Area**. 1. On the left sidebar, select **Monitoring > Background Jobs**. 1. On the Sidekiq dashboard, select **Queues**, and wait for all the `geo` queues to drop to 0 queued and 0 running jobs. diff --git a/doc/administration/geo/disaster_recovery/runbooks/planned_failover_single_node.md b/doc/administration/geo/disaster_recovery/runbooks/planned_failover_single_node.md index e29b19ca040..0afd669e797 100644 --- a/doc/administration/geo/disaster_recovery/runbooks/planned_failover_single_node.md +++ b/doc/administration/geo/disaster_recovery/runbooks/planned_failover_single_node.md @@ -1,8 +1,7 @@ --- stage: Systems group: Geo -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments -type: howto +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://handbook.gitlab.com/handbook/product/ux/technical-writing/#assignments --- WARNING: @@ -115,8 +114,7 @@ follow these steps to avoid unnecessary data loss: connection. 1. On the **primary** site: - 1. On the left sidebar, select **Search or go to**. - 1. Select **Admin Area**. + 1. On the left sidebar, at the bottom, select **Admin Area**.. 1. On the left sidebar, select **Monitoring > Background Jobs**. 1. On the Sidekiq dashboard, select **Cron**. 1. Select `Disable All` to disable any non-Geo periodic background jobs. @@ -134,8 +132,7 @@ follow these steps to avoid unnecessary data loss: [data not managed by Geo](../../replication/datatypes.md#limitations-on-replicationverification), trigger the final replication process now. 1. On the **primary** site: - 1. On the left sidebar, select **Search or go to**. - 1. Select **Admin Area**. + 1. On the left sidebar, at the bottom, select **Admin Area**. 1. On the left sidebar, select **Monitoring > Background Jobs**. 1. On the Sidekiq dashboard, select **Queues**, and wait for all queues except those with `geo` in the name to drop to 0. @@ -150,8 +147,7 @@ follow these steps to avoid unnecessary data loss: - The Geo log cursor is up to date (0 events behind). 1. On the **secondary** site: - 1. On the left sidebar, select **Search or go to**. - 1. Select **Admin Area**. + 1. On the left sidebar, at the bottom, select **Admin Area**. 1. On the left sidebar, select **Monitoring > Background Jobs**. 1. On the Sidekiq dashboard, select **Queues**, and wait for all the `geo` queues to drop to 0 queued and 0 running jobs. diff --git a/doc/administration/geo/glossary.md b/doc/administration/geo/glossary.md index d0f94ba39f5..138bf6c444c 100644 --- a/doc/administration/geo/glossary.md +++ b/doc/administration/geo/glossary.md @@ -1,8 +1,7 @@ --- stage: Systems group: Geo -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments -type: howto +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://handbook.gitlab.com/handbook/product/ux/technical-writing/#assignments --- diff --git a/doc/administration/geo/index.md b/doc/administration/geo/index.md index e8b2cb38563..6609bbed5a6 100644 --- a/doc/administration/geo/index.md +++ b/doc/administration/geo/index.md @@ -1,7 +1,7 @@ --- stage: Systems group: Geo -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://handbook.gitlab.com/handbook/product/ux/technical-writing/#assignments --- # Geo **(PREMIUM SELF)** @@ -115,15 +115,16 @@ The following are required to run Geo: The following operating systems are known to ship with a current version of OpenSSH: - [CentOS](https://www.centos.org) 7.4 or later - [Ubuntu](https://ubuntu.com) 16.04 or later -- PostgreSQL 12 or 13 with [Streaming Replication](https://wiki.postgresql.org/wiki/Streaming_Replication) +- [Supported PostgreSQL versions](https://about.gitlab.com/handbook/engineering/development/enablement/data_stores/database/postgresql-upgrade-cadence.html) for your GitLab releases with [Streaming Replication](https://wiki.postgresql.org/wiki/Streaming_Replication). - Note,[PostgreSQL 12 is deprecated](../../update/deprecations.md#postgresql-12-deprecated) and is removed in GitLab 16.0. +- All sites must run [the same PostgreSQL versions](setup/database.md#postgresql-replication). + - Where possible, you should also use the same operating system version on all + Geo sites. If using different operating system versions between Geo sites, you + **must** [check OS locale data compatibility](replication/troubleshooting.md#check-os-locale-data-compatibility) + across Geo sites to avoid silent corruption of database indexes. - Git 2.9 or later - Git-lfs 2.4.2 or later on the user side when using LFS - All sites must run the same GitLab version. -- All sites must run [the same PostgreSQL versions](setup/database.md#postgresql-replication). - - If using different operating system versions between Geo sites, - [check OS locale data compatibility](replication/troubleshooting.md#check-os-locale-data-compatibility) - across Geo sites to avoid silent corruption of database indexes. - All sites must define the same [repository storages](../repository_storage_paths.md). Additionally, check the GitLab [minimum requirements](../../install/requirements.md), @@ -159,9 +160,8 @@ public URL of the primary site is used. To update the internal URL of the primary Geo site: -1. On the left sidebar, select **Search or go to**. -1. Select **Admin Area**. -1. On the left sidebar, select **Geo > Sites**. +1. On the left sidebar, at the bottom, select **Admin Area**. +1. Select **Geo > Sites**. 1. Select **Edit** on the primary site. 1. Change the **Internal URL**, then select **Save changes**. @@ -245,17 +245,29 @@ Linux package-managed database. External databases are not supported. In some circumstances, like during [upgrades](replication/upgrading_the_geo_sites.md) or a [planned failover](disaster_recovery/planned_failover.md), it is desirable to pause replication between the primary and secondary. -Pausing and resuming replication is done through a command-line tool from the node in the secondary site where the `postgresql` service is enabled. +If you plan to allow user activity on your secondary sites during the upgrade, +do not pause replication for a [zero downtime upgrade](../../update/zero_downtime.md). While paused, the secondary site gets more and more out-of-date. +One known effect is that more and more Git fetches get redirected or proxied to the primary site. There may be additional unknown effects. + +Pausing and resuming replication is done through a command-line tool from a specific node in the secondary site. Depending on your database architecture, +this will target either the `postgresql` or `patroni` service: + +- If you are using a single node for all services on your secondary site, you must run the commands on this single node. +- If you have a standalone PostgreSQL node on your secondary site, you must run the commands on this standalone PostgreSQL node. +- If your secondary site is using a Patroni cluster, you must run these commands on the secondary Patroni standby leader node. + +If you aren't using a single node for all services on your secondary site, ensure that the `/etc/gitlab/gitlab.rb` on your PostgreSQL or Patroni nodes +contains the configuration line `gitlab_rails['geo_node_name'] = 'node_name'`, where `node_name` is the same as the `geo_node_name` on the application node. -If `postgresql` is on a standalone database node, ensure that `gitlab.rb` on that node contains the configuration line `gitlab_rails['geo_node_name'] = 'node_name'`, where `node_name` is the same as the `geo_node_name` on the application node. +**To Pause: (from secondary site)** -**To Pause: (from secondary)** +Also, be aware that if PostgreSQL is restarted after pausing replication (either by restarting the VM or restarting the service with `gitlab-ctl restart postgresql`), PostgreSQL automatically resumes replication, which is something you wouldn't want during an upgrade or in a planned failover scenario. ```shell gitlab-ctl geo-replication-pause ``` -**To Resume: (from secondary)** +**To Resume: (from secondary site)** ```shell gitlab-ctl geo-replication-resume @@ -273,9 +285,9 @@ For information on configuring Geo with Object storage, see [Geo with Object sto For information on using Geo in disaster recovery situations to mitigate data-loss and restore services, see [Disaster Recovery](disaster_recovery/index.md). -### Replicating the Container Registry +### Replicating the container registry -For more information on how to replicate the Container Registry, see [Container Registry for a **secondary** site](replication/container_registry.md). +For more information on how to replicate the container registry, see [Container registry for a **secondary** site](replication/container_registry.md). ### Geo secondary proxy diff --git a/doc/administration/geo/replication/configuration.md b/doc/administration/geo/replication/configuration.md index 8722ab574c3..b50f86dcf47 100644 --- a/doc/administration/geo/replication/configuration.md +++ b/doc/administration/geo/replication/configuration.md @@ -1,8 +1,7 @@ --- stage: Systems group: Geo -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments -type: howto +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://handbook.gitlab.com/handbook/product/ux/technical-writing/#assignments --- # Geo configuration **(PREMIUM SELF)** @@ -217,8 +216,7 @@ In the following steps, replace `<ssh_host_key_path>` with the one you're using: ``` 1. Navigate to the Primary Node GitLab Instance: - 1. On the left sidebar, select **Search or go to**. - 1. Select **Admin Area**. + 1. On the left sidebar, at the bottom, select **Admin Area**. 1. On the left sidebar, select **Geo > Sites**. 1. Select **Add site**.  @@ -327,9 +325,8 @@ method to be enabled. This is enabled by default, but if converting an existing On the **primary** site: -1. On the left sidebar, select **Search or go to**. -1. Select **Admin Area**. -1. On the left sidebar, select **Settings > General**. +1. On the left sidebar, at the bottom, select **Admin Area**. +1. Select **Settings > General**. 1. Expand **Visibility and access controls**. 1. If using Git over SSH, then: 1. Ensure "Enabled Git access protocols" is set to "Both SSH and HTTP(S)". @@ -341,9 +338,8 @@ On the **primary** site: You can sign in to the **secondary** site with the same credentials you used with the **primary** site. After you sign in: -1. On the left sidebar, select **Search or go to**. -1. Select **Admin Area**. -1. On the left sidebar, select **Geo > Sites**. +1. On the left sidebar, at the bottom, select **Admin Area**. +1. Select **Geo > Sites**. 1. Verify that it's correctly identified as a **secondary** Geo site, and that Geo is enabled. diff --git a/doc/administration/geo/replication/container_registry.md b/doc/administration/geo/replication/container_registry.md index 65ecba0d04a..b9bee753089 100644 --- a/doc/administration/geo/replication/container_registry.md +++ b/doc/administration/geo/replication/container_registry.md @@ -1,17 +1,16 @@ --- stage: Systems group: Geo -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments -type: howto +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://handbook.gitlab.com/handbook/product/ux/technical-writing/#assignments --- -# Container Registry for a secondary site **(PREMIUM SELF)** +# Container registry for a secondary site **(PREMIUM SELF)** -You can set up a Container Registry on your **secondary** Geo site that mirrors the one on the **primary** Geo site. +You can set up a container registry on your **secondary** Geo site that mirrors the one on the **primary** Geo site. NOTE: -The Container Registry replication is used only for disaster recovery purposes. We do not recommend -pulling the Container Registry data from the secondary. For a feature proposal to implement it in the +The container registry replication is used only for disaster recovery purposes. We do not recommend +pulling the container registry data from the secondary. For a feature proposal to implement it in the future, see [Geo: Accelerate container images by serving read request from secondary site](https://gitlab.com/gitlab-org/gitlab/-/issues/365864) for details. ## Supported container registries @@ -40,7 +39,7 @@ For more information on supported registry storage drivers see Read the [Load balancing considerations](https://docs.docker.com/registry/deploying/#load-balancing-considerations) when deploying the Registry, and how to set up the storage driver for the GitLab integrated -[Container Registry](../../packages/container_registry.md#use-object-storage). +[container registry](../../packages/container_registry.md#use-object-storage). ### Registries that support OCI artifacts @@ -55,26 +54,26 @@ The following registries support OCI artifacts: For more information, see the [OCI Distribution Specification](https://github.com/opencontainers/distribution-spec). -## Configure Container Registry replication +## Configure container registry replication You can enable a storage-agnostic replication so it can be used for cloud or local storage. Whenever a new image is pushed to the **primary** site, each **secondary** site pulls it to its own container repository. -To configure Container Registry replication: +To configure container registry replication: 1. Configure the [**primary** site](#configure-primary-site). 1. Configure the [**secondary** site](#configure-secondary-site). -1. Verify Container Registry [replication](#verify-replication). +1. Verify container registry [replication](#verify-replication). ### Configure **primary** site -Make sure that you have Container Registry set up and working on +Make sure that you have container registry set up and working on the **primary** site before following the next steps. -To be able to replicate new container images, the Container Registry must send notification events to the -**primary** site for every push. The token shared between the Container Registry and the web nodes on the +To be able to replicate new container images, the container registry must send notification events to the +**primary** site for every push. The token shared between the container registry and the web nodes on the **primary** is used to make communication more secure. 1. SSH into your GitLab **primary** server and login as root (for GitLab HA, you only need a Registry node): @@ -124,17 +123,17 @@ To be able to replicate new container images, the Container Registry must send n ### Configure **secondary** site -Make sure you have Container Registry set up and working on +Make sure you have container registry set up and working on the **secondary** site before following the next steps. The following steps should be done on each **secondary** site you're expecting to see the container images replicated. Because we need to allow the **secondary** site to communicate securely with -the **primary** site Container Registry, we need to have a single key +the **primary** site container registry, we need to have a single key pair for all the sites. The **secondary** site uses this key to generate a short-lived JWT that is pull-only-capable to access the -**primary** site Container Registry. +**primary** site container registry. For each application and Sidekiq node on the **secondary** site: @@ -164,11 +163,10 @@ For each application and Sidekiq node on the **secondary** site: ### Verify replication -To verify Container Registry replication is working, on the **secondary** site: +To verify container registry replication is working, on the **secondary** site: -1. On the left sidebar, select **Search or go to**. -1. Select **Admin Area**. -1. On the left sidebar, select **Geo > Nodes**. +1. On the left sidebar, at the bottom, select **Admin Area**. +1. Select **Geo > Nodes**. The initial replication, or "backfill", is probably still in progress. You can monitor the synchronization process on each Geo site from the **primary** site's **Geo Nodes** dashboard in your browser. diff --git a/doc/administration/geo/replication/datatypes.md b/doc/administration/geo/replication/datatypes.md index 896501128f7..634a524e6f0 100644 --- a/doc/administration/geo/replication/datatypes.md +++ b/doc/administration/geo/replication/datatypes.md @@ -1,8 +1,7 @@ --- stage: Systems group: Geo -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments -type: howto +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://handbook.gitlab.com/handbook/product/ux/technical-writing/#assignments --- # Supported Geo data types **(PREMIUM SELF)** @@ -190,7 +189,7 @@ successfully, you must replicate their data using some other means. |Feature | Replicated (added in GitLab version) | Verified (added in GitLab version) | GitLab-managed object storage replication (added in GitLab version) | GitLab-managed object storage verification (added in GitLab version) | Notes | |:--------------------------------------------------------------------------------------------------------------|:------------------------------------------------------------------------|:---------------------------------------------------------------------------|:--------------------------------------------------------------------|:----------------------------------------------------------------|:------| |[Application data in PostgreSQL](../../postgresql/index.md) | **Yes** (10.2) | **Yes** (10.2) | Not applicable | Not applicable | | -|[Project repository](../../../user/project/repository/index.md) | **Yes** (10.2) | **Yes** (10.7) | Not applicable | Not applicable | Migrated to [self-service framework](../../../development/geo/framework.md) in 16.2. See GitLab issue [#367925](https://gitlab.com/gitlab-org/gitlab/-/issues/367925) for more details.<br /><br />Behind feature flag `geo_project_repository_replication`, enabled by default in (16.3).<br /><br /> All projects, including [archived projects](../../../user/project/settings/index.md#archive-a-project), are replicated. | +|[Project repository](../../../user/project/repository/index.md) | **Yes** (10.2) | **Yes** (10.7) | Not applicable | Not applicable | Migrated to [self-service framework](../../../development/geo/framework.md) in 16.2. See GitLab issue [#367925](https://gitlab.com/gitlab-org/gitlab/-/issues/367925) for more details.<br /><br />Behind feature flag `geo_project_repository_replication`, enabled by default in (16.3).<br /><br /> All projects, including [archived projects](../../../user/project/settings/migrate_projects.md#archive-a-project), are replicated. | |[Project wiki repository](../../../user/project/wiki/index.md) | **Yes** (10.2)<sup>2</sup> | **Yes** (10.7)<sup>2</sup> | Not applicable | Not applicable | Migrated to [self-service framework](../../../development/geo/framework.md) in 15.11. See GitLab issue [#367925](https://gitlab.com/gitlab-org/gitlab/-/issues/367925) for more details.<br /><br />Behind feature flag `geo_project_wiki_repository_replication`, enabled by default in (15.11). | |[Group wiki repository](../../../user/project/wiki/group.md) | [**Yes** (13.10)](https://gitlab.com/gitlab-org/gitlab/-/issues/208147) | [**Yes** (16.3)](https://gitlab.com/gitlab-org/gitlab/-/issues/323897) | Not applicable | Not applicable | Behind feature flag `geo_group_wiki_repository_replication`, enabled by default. | |[Uploads](../../uploads.md) | **Yes** (10.2) | **Yes** (14.6) | [**Yes** (15.1)](https://gitlab.com/groups/gitlab-org/-/epics/5551) | [**Yes** (16.4)](object_storage.md) | Replication is behind the feature flag `geo_upload_replication`, enabled by default. Verification was behind the feature flag `geo_upload_verification`, removed in 14.8. | @@ -200,10 +199,10 @@ successfully, you must replicate their data using some other means. |[CI job artifacts](../../../ci/jobs/job_artifacts.md) | **Yes** (10.4) | **Yes** (14.10) | [**Yes** (15.1)](https://gitlab.com/groups/gitlab-org/-/epics/5551) | [**Yes** (16.4)](object_storage.md) | Verification is behind the feature flag `geo_job_artifact_replication`, enabled by default in 14.10. | |[CI Pipeline Artifacts](https://gitlab.com/gitlab-org/gitlab/-/blob/master/app/models/ci/pipeline_artifact.rb) | [**Yes** (13.11)](https://gitlab.com/gitlab-org/gitlab/-/issues/238464) | [**Yes** (13.11)](https://gitlab.com/gitlab-org/gitlab/-/issues/238464) | [**Yes** (15.1)](https://gitlab.com/groups/gitlab-org/-/epics/5551) | [**Yes** (16.4)](object_storage.md) | Persists additional artifacts after a pipeline completes. | |[CI Secure Files](https://gitlab.com/gitlab-org/gitlab/-/blob/master/app/models/ci/secure_file.rb) | [**Yes** (15.3)](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/91430) | [**Yes** (15.3)](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/91430) | [**Yes** (15.3)](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/91430) | [**Yes** (16.4)](object_storage.md) | Verification is behind the feature flag `geo_ci_secure_file_replication`, enabled by default in 15.3. | -|[Container Registry](../../packages/container_registry.md) | **Yes** (12.3)<sup>1</sup> | **Yes** (15.10) | **Yes** (12.3)<sup>1</sup> | **Yes** (15.10) | See [instructions](container_registry.md) to set up the Container Registry replication. | +|[Container registry](../../packages/container_registry.md) | **Yes** (12.3)<sup>1</sup> | **Yes** (15.10) | **Yes** (12.3)<sup>1</sup> | **Yes** (15.10) | See [instructions](container_registry.md) to set up the container registry replication. | |[Terraform Module Registry](../../../user/packages/terraform_module_registry/index.md) | **Yes** (14.0) | **Yes** (14.0) | [**Yes** (15.1)](https://gitlab.com/groups/gitlab-org/-/epics/5551) | [**Yes** (16.4)](object_storage.md) | Behind feature flag `geo_package_file_replication`, enabled by default. | |[Project designs repository](../../../user/project/issues/design_management.md) | **Yes** (12.7) | **Yes** (16.1) | Not applicable | Not applicable | Designs also require replication of LFS objects and Uploads. | -|[Package Registry](../../../user/packages/package_registry/index.md) | **Yes** (13.2) | **Yes** (13.10) | [**Yes** (15.1)](https://gitlab.com/groups/gitlab-org/-/epics/5551) | [**Yes** (16.4)](object_storage.md) | Behind feature flag `geo_package_file_replication`, enabled by default. | +|[Package registry](../../../user/packages/package_registry/index.md) | **Yes** (13.2) | **Yes** (13.10) | [**Yes** (15.1)](https://gitlab.com/groups/gitlab-org/-/epics/5551) | [**Yes** (16.4)](object_storage.md) | Behind feature flag `geo_package_file_replication`, enabled by default. | |[Versioned Terraform State](../../terraform_state.md) | **Yes** (13.5) | **Yes** (13.12) | [**Yes** (15.1)](https://gitlab.com/groups/gitlab-org/-/epics/5551) | [**Yes** (16.4)](object_storage.md) | Replication is behind the feature flag `geo_terraform_state_version_replication`, enabled by default. Verification was behind the feature flag `geo_terraform_state_version_verification`, which was removed in 14.0. | |[External merge request diffs](../../merge_request_diffs.md) | **Yes** (13.5) | **Yes** (14.6) | [**Yes** (15.1)](https://gitlab.com/groups/gitlab-org/-/epics/5551) | [**Yes** (16.4)](object_storage.md) | Replication is behind the feature flag `geo_merge_request_diff_replication`, enabled by default. Verification was behind the feature flag `geo_merge_request_diff_verification`, removed in 14.7.| |[Versioned snippets](../../../user/snippets.md#versioned-snippets) | [**Yes** (13.7)](https://gitlab.com/groups/gitlab-org/-/epics/2809) | [**Yes** (14.2)](https://gitlab.com/groups/gitlab-org/-/epics/2810) | Not applicable | Not applicable | Verification was implemented behind the feature flag `geo_snippet_repository_verification` in 13.11, and the feature flag was removed in 14.2. | @@ -215,6 +214,7 @@ successfully, you must replicate their data using some other means. |[Elasticsearch integration](../../../integration/advanced_search/elasticsearch.md) | [Not planned](https://gitlab.com/gitlab-org/gitlab/-/issues/1186) | No | No | No | Not planned because further product discovery is required and Elasticsearch (ES) clusters can be rebuilt. Secondaries use the same ES cluster as the primary. | |[Dependency Proxy Images](../../../user/packages/dependency_proxy/index.md) | [**Yes** (15.7)](https://gitlab.com/groups/gitlab-org/-/epics/8833) | [**Yes** (15.7)](https://gitlab.com/groups/gitlab-org/-/epics/8833) | [**Yes** (15.7)](https://gitlab.com/groups/gitlab-org/-/epics/8833) | [**Yes** (16.4)](object_storage.md) | | |[Vulnerability Export](../../../user/application_security/vulnerability_report/index.md#export-vulnerability-details) | [Not planned](https://gitlab.com/groups/gitlab-org/-/epics/3111) | No | No | No | Not planned because they are ephemeral and sensitive information. They can be regenerated on demand. | +|Packages NPM metadata cache | [Not planned](https://gitlab.com/gitlab-org/gitlab/-/issues/408278) | No | No | No | Not planned because it would not notably improve disaster recovery capabilities nor response times at secondary sites. | <sup>1</sup> Migrated to [self-service framework](../../../development/geo/framework.md) in 15.5. See GitLab issue [#337436](https://gitlab.com/gitlab-org/gitlab/-/issues/337436) for more details. diff --git a/doc/administration/geo/replication/disable_geo.md b/doc/administration/geo/replication/disable_geo.md index 42c37a79ec2..597bb08c188 100644 --- a/doc/administration/geo/replication/disable_geo.md +++ b/doc/administration/geo/replication/disable_geo.md @@ -1,8 +1,7 @@ --- stage: Systems group: Geo -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments -type: howto +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://handbook.gitlab.com/handbook/product/ux/technical-writing/#assignments --- # Disabling Geo **(PREMIUM SELF)** @@ -36,9 +35,8 @@ to do that. To remove the **primary** site: 1. [Remove all secondary Geo sites](remove_geo_site.md) -1. On the left sidebar, select **Search or go to**. -1. Select **Admin Area**. -1. On the left sidebar, select **Geo > Nodes**. +1. On the left sidebar, at the bottom, select **Admin Area**. +1. Select **Geo > Nodes**. 1. Select **Remove** for the **primary** node. 1. Confirm by selecting **Remove** when the prompt appears. diff --git a/doc/administration/geo/replication/faq.md b/doc/administration/geo/replication/faq.md index 311cdeee5b9..f29d7967feb 100644 --- a/doc/administration/geo/replication/faq.md +++ b/doc/administration/geo/replication/faq.md @@ -1,7 +1,7 @@ --- stage: Systems group: Geo -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://handbook.gitlab.com/handbook/product/ux/technical-writing/#assignments --- # Geo Frequently Asked Questions **(PREMIUM SELF)** @@ -65,9 +65,9 @@ connectivity between your sites, and your hardware. That's totally fine. We use HTTP(s) to fetch repository changes from the **primary** site to all **secondary** sites. -## Is this possible to set up a Container Registry for a **secondary** site that mirrors the one on the **primary** site? +## Is this possible to set up a container registry for a **secondary** site that mirrors the one on the **primary** site? -Yes. See [Container Registry for a **secondary** site](container_registry.md). +Yes. See [container registry for a **secondary** site](container_registry.md). ## Can you sign in to a secondary site? diff --git a/doc/administration/geo/replication/geo_validation_tests.md b/doc/administration/geo/replication/geo_validation_tests.md index 7897635e78f..b4f4374fd9b 100644 --- a/doc/administration/geo/replication/geo_validation_tests.md +++ b/doc/administration/geo/replication/geo_validation_tests.md @@ -1,7 +1,7 @@ --- stage: Systems group: Geo -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://handbook.gitlab.com/handbook/product/ux/technical-writing/#assignments --- # Geo validation tests **(PREMIUM SELF)** diff --git a/doc/administration/geo/replication/location_aware_git_url.md b/doc/administration/geo/replication/location_aware_git_url.md index a3abc945288..6849a648991 100644 --- a/doc/administration/geo/replication/location_aware_git_url.md +++ b/doc/administration/geo/replication/location_aware_git_url.md @@ -1,8 +1,7 @@ --- stage: Systems group: Geo -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments -type: howto +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://handbook.gitlab.com/handbook/product/ux/technical-writing/#assignments --- # Location-aware Git remote URL with AWS Route53 **(PREMIUM SELF)** diff --git a/doc/administration/geo/replication/multiple_servers.md b/doc/administration/geo/replication/multiple_servers.md index 786030736f6..850ab9f0707 100644 --- a/doc/administration/geo/replication/multiple_servers.md +++ b/doc/administration/geo/replication/multiple_servers.md @@ -1,8 +1,7 @@ --- stage: Systems group: Geo -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments -type: howto +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://handbook.gitlab.com/handbook/product/ux/technical-writing/#assignments --- # Geo for multiple nodes **(PREMIUM SELF)** diff --git a/doc/administration/geo/replication/object_storage.md b/doc/administration/geo/replication/object_storage.md index 3b6b214d9e3..9d1cb4387a0 100644 --- a/doc/administration/geo/replication/object_storage.md +++ b/doc/administration/geo/replication/object_storage.md @@ -1,8 +1,7 @@ --- stage: Systems group: Geo -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments -type: howto +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://handbook.gitlab.com/handbook/product/ux/technical-writing/#assignments --- # Geo with Object storage **(PREMIUM SELF)** @@ -43,9 +42,8 @@ whether they are stored on the local file system or in object storage. To enable GitLab replication: -1. On the left sidebar, select **Search or go to**. -1. Select **Admin Area**. -1. On the left sidebar, select **Geo > Nodes**. +1. On the left sidebar, at the bottom, select **Admin Area**. +1. Select **Geo > Nodes**. 1. Select **Edit** on the **secondary** site. 1. In the **Synchronization Settings** section, find the **Allow this secondary node to replicate content on Object Storage** checkbox to enable it. diff --git a/doc/administration/geo/replication/remove_geo_site.md b/doc/administration/geo/replication/remove_geo_site.md index a55c05050a0..4787b949e2e 100644 --- a/doc/administration/geo/replication/remove_geo_site.md +++ b/doc/administration/geo/replication/remove_geo_site.md @@ -1,17 +1,15 @@ --- stage: Systems group: Geo -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments -type: howto +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://handbook.gitlab.com/handbook/product/ux/technical-writing/#assignments --- # Removing secondary Geo sites **(PREMIUM SELF)** **Secondary** sites can be removed from the Geo cluster using the Geo administration page of the **primary** site. To remove a **secondary** site: -1. On the left sidebar, select **Search or go to**. -1. Select **Admin Area**. -1. On the left sidebar, select **Geo > Nodes**. +1. On the left sidebar, at the bottom, select **Admin Area**. +1. Select **Geo > Nodes**. 1. For the **secondary** site you want to remove, select **Remove**. 1. Confirm by selecting **Remove** when the prompt appears. diff --git a/doc/administration/geo/replication/security_review.md b/doc/administration/geo/replication/security_review.md index e099c2b2e9d..a267ac23779 100644 --- a/doc/administration/geo/replication/security_review.md +++ b/doc/administration/geo/replication/security_review.md @@ -1,8 +1,7 @@ --- stage: Systems group: Geo -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments -type: howto +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://handbook.gitlab.com/handbook/product/ux/technical-writing/#assignments --- # Geo security review (Q&A) **(PREMIUM SELF)** diff --git a/doc/administration/geo/replication/single_sign_on.md b/doc/administration/geo/replication/single_sign_on.md index 15e24cdcefb..f8ca6209524 100644 --- a/doc/administration/geo/replication/single_sign_on.md +++ b/doc/administration/geo/replication/single_sign_on.md @@ -1,8 +1,7 @@ --- stage: Systems group: Geo -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments -type: howto +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://handbook.gitlab.com/handbook/product/ux/technical-writing/#assignments --- # Geo with Single Sign On (SSO) **(PREMIUM SELF)** @@ -15,7 +14,7 @@ This documentation only discusses Geo-specific SSO considerations and configurat [Instance-wide SAML](../../../integration/saml.md) must be working on your primary Geo site. -You only configure SAML on the primary site. Configuring `gitlab_rails['omniauth_providers']` in `gitlab.rb` in a secondary site has no effect. +You only configure SAML on the primary site. Configuring `gitlab_rails['omniauth_providers']` in `gitlab.rb` in a secondary site has no effect. The secondary site authenticates against the SAML provider configured on the primary site. Depending on the [URL type](#determine-the-type-of-url-your-secondary-site-uses) of the secondary site, [additional configuration](#saml-with-separate-url-with-proxying-enabled) might be needed on the primary site. ### Determine the type of URL your secondary site uses diff --git a/doc/administration/geo/replication/troubleshooting.md b/doc/administration/geo/replication/troubleshooting.md index dd021695800..6e60ae34a22 100644 --- a/doc/administration/geo/replication/troubleshooting.md +++ b/doc/administration/geo/replication/troubleshooting.md @@ -1,7 +1,7 @@ --- stage: Systems group: Geo -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://handbook.gitlab.com/handbook/product/ux/technical-writing/#assignments --- # Troubleshooting Geo **(PREMIUM SELF)** @@ -27,9 +27,8 @@ Before attempting more advanced troubleshooting: On the **primary** site: -1. On the left sidebar, select **Search or go to**. -1. Select **Admin Area**. -1. On the left sidebar, select **Geo > Sites**. +1. On the left sidebar, at the bottom, select **Admin Area**. +1. Select **Geo > Sites**. We perform the following health checks on each **secondary** site to help identify if something is wrong: @@ -199,7 +198,7 @@ http://secondary.example.com/ Last status report was: 1 minute ago ``` -There are up to three statuses for each item. For example, for `Project Repositories`, you see the following lines: +Each item can have up to three statuses. For example, for `Project Repositories`, you see the following lines: ```plaintext Project Repositories: succeeded 12345 / total 12345 (100%) @@ -466,9 +465,9 @@ For more information about recommended site names in the description of the Name ### Check OS locale data compatibility -If different operating systems or different operating system versions are deployed across Geo sites, you should perform a locale data compatibility check before setting up Geo. +If different operating systems or different operating system versions are deployed across Geo sites, you **must** perform a locale data compatibility check before setting up Geo. -Geo uses PostgreSQL and Streaming Replication to replicate data across Geo sites. PostgreSQL uses locale data provided by the operating system's C library for sorting text. If the locale data in the C library is incompatible across Geo sites, it can cause erroneous query results that lead to [incorrect behavior on secondary sites](https://gitlab.com/gitlab-org/gitlab/-/issues/360723). +Geo uses PostgreSQL and Streaming Replication to replicate data across Geo sites. PostgreSQL uses locale data provided by the operating system's C library for sorting text. If the locale data in the C library is incompatible across Geo sites, it causes erroneous query results that lead to [incorrect behavior on secondary sites](https://gitlab.com/gitlab-org/gitlab/-/issues/360723). For example, Ubuntu 18.04 (and earlier) and RHEL/Centos7 (and earlier) are incompatible with their later releases. See the [PostgreSQL wiki for more details](https://wiki.postgresql.org/wiki/Locale_data_changes). @@ -493,9 +492,9 @@ or the reverse order: 1-1 ``` -If the output is identical on all hosts, then they running compatible versions of locale data. +If the output is **identical** on all hosts, then they running compatible versions of locale data and you may proceed with Geo configuration. -If the output differs on some hosts, PostgreSQL replication does not work properly: indexes are corrupted on the database replicas. You should select operating system versions that are compatible. +If the output **differs** on any hosts, PostgreSQL replication will not work properly: indexes will become corrupted on the database replicas. You **must** select operating system versions that are compatible. A full index rebuild is required if the on-disk data is transferred 'at rest' to an operating system with an incompatible locale, or through replication. @@ -682,6 +681,19 @@ In GitLab 13.4, a seed project is added when GitLab is first installed. This mak on a new Geo secondary site. There is an [issue to account for seed projects](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/5618) when checking the database. +### Message: `FATAL: could not map anonymous shared memory: Cannot allocate memory` + +If you see this message, it means that the secondary site's PostgreSQL tries to request memory that is higher than the available memory. There is an [issue](https://gitlab.com/gitlab-org/gitlab/-/issues/381585) that tracks this problem. + +Example error message in Patroni logs (located at `/var/log/gitlab/patroni/current` for Linux package installations): + +```plaintext +2023-11-21_23:55:18.63727 FATAL: could not map anonymous shared memory: Cannot allocate memory +2023-11-21_23:55:18.63729 HINT: This error usually means that PostgreSQL's request for a shared memory segment exceeded available memory, swap space, or huge pages. To reduce the request size (currently 17035526144 bytes), reduce PostgreSQL's shared memory usage, perhaps by reducing shared_buffers or max_connections. +``` + +The workaround is to increase the memory available to the secondary site's PostgreSQL nodes to match the memory requirements of the primary site's PostgreSQL nodes. + ## Synchronization errors ### Reverify all uploads (or any SSF data type which is verified) @@ -726,16 +738,75 @@ If large repositories are affected by this problem, their resync may take a long time and cause significant load on your Geo sites, storage and network systems. -If you see the error message `Synchronization failed - Error syncing repository` along with `fatal: fsck error in packed object`, this indicates -a consistency check error when syncing the repository. +The following error message indicates a consistency check error when syncing the repository: + +```plaintext +Synchronization failed - Error syncing repository [..] fatal: fsck error in packed object +``` + +Several issues can trigger this error. For example, problems with email addresses: + +```plaintext +Error syncing repository: 13:fetch remote: "error: object <SHA>: badEmail: invalid author/committer line - bad email + fatal: fsck error in packed object + fatal: fetch-pack: invalid index-pack output +``` + +Another issue that can trigger this error is `object <SHA>: hasDotgit: contains '.git'`. Check the specific errors because you might have more than one problem across all +your repositories. + +A second synchronization error can also be caused by repository check issues: + +```plaintext +Error syncing repository: 13:Received RST_STREAM with error code 2. +``` + +These errors can be observed by [immediately syncing all failed repositories](#sync-all-failed-repositories-now). + +Removing the malformed objects causing consistency errors involves rewriting the repository history, which is usually not an option. + +To ignore these consistency checks, reconfigure Gitaly **on the secondary Geo sites** to ignore these `git fsck` issues. +The following configuration example: + +- [Uses the new configuration structure](../../../update/versions/gitlab_16_changes.md#gitaly-configuration-structure-change) required from GitLab 16.0. +- Ignores five common check failures. + +[The Gitaly documentation has more details](../../gitaly/configure_gitaly.md#repository-consistency-checks) +about other Git check failures and older versions of GitLab. + +```ruby +gitaly['configuration'] = { + git: { + config: [ + { key: "fsck.duplicateEntries", value: "ignore" }, + { key: "fsck.badFilemode", value: "ignore" }, + { key: "fsck.missingEmail", value: "ignore" }, + { key: "fsck.badEmail", value: "ignore" }, + { key: "fsck.hasDotgit", value: "ignore" }, + { key: "fetch.fsck.duplicateEntries", value: "ignore" }, + { key: "fetch.fsck.badFilemode", value: "ignore" }, + { key: "fetch.fsck.missingEmail", value: "ignore" }, + { key: "fetch.fsck.badEmail", value: "ignore" }, + { key: "fetch.fsck.hasDotgit", value: "ignore" }, + { key: "receive.fsck.duplicateEntries", value: "ignore" }, + { key: "receive.fsck.badFilemode", value: "ignore" }, + { key: "receive.fsck.missingEmail", value: "ignore" }, + { key: "receive.fsck.badEmail", value: "ignore" }, + { key: "receive.fsck.hasDotgit", value: "ignore" }, + ], + }, +} +``` + +GitLab 16.1 and later [include an enhancement](https://gitlab.com/gitlab-org/gitaly/-/merge_requests/5879) that might resolve some of these issues. -One example of a consistency error is: `error: object f4a87a3be694fbbd6e50a668a31a8513caeaafe3: hasDotgit: contains '.git`. +[Gitaly issue 5625](https://gitlab.com/gitlab-org/gitaly/-/issues/5625) proposes to ensure that Geo replicates repositories even if the source repository contains +problematic commits. -Removing the malformed objects causing consistency errors require rewriting the repository history, which is not always an option. However, -it's possible to override the consistency checks instead. To do that, follow -[the instructions in the Gitaly docs](../../gitaly/configure_gitaly.md#repository-consistency-checks). +#### Related error `does not appear to be a git repository` -You can also get the error message `Synchronization failed - Error syncing repository` along with the following log messages, this indicates that the expected `geo` remote is not present in the `.git/config` file +You can also get the error message `Synchronization failed - Error syncing repository` along with the following log messages. +This error indicates that the expected Geo remote is not present in the `.git/config` file of a repository on the secondary Geo site's file system: ```json @@ -839,7 +910,7 @@ This behavior affects only the following data types through GitLab 14.6: | Data type | From version | | ------------------------ | ------------ | -| Package Registry | 13.10 | +| Package registry | 13.10 | | CI Pipeline Artifacts | 13.11 | | Terraform State Versions | 13.12 | | Infrastructure Registry (renamed to Terraform Module Registry in GitLab 15.11) | 14.0 | @@ -953,6 +1024,11 @@ The following script: - Displays the project details and the reasons for the last failure. - Attempts to resync the repository. - Reports back if a failure occurs, and why. +- Might take some time to complete. Each repository check must complete + before reporting back the result. If your session times out, take measures + to allow the process to continue running such as starting a `screen` session, + or running it using [Rails runner](../../operations/rails_console.md#using-the-rails-runner) + and `nohup`. ```ruby Geo::ProjectRegistry.sync_failed('repository').find_each do |p| @@ -1450,9 +1526,8 @@ If you have updated the value of `external_url` in `/etc/gitlab/gitlab.rb` for t In this case, make sure to update the changed URL on all your sites: -1. On the left sidebar, select **Search or go to**. -1. Select **Admin Area**. -1. On the left sidebar, select **Geo > Sites**. +1. On the left sidebar, at the bottom, select **Admin Area**. +1. Select **Geo > Sites**. 1. Change the URL and save the change. ### Message: `ERROR: canceling statement due to conflict with recovery` during backup @@ -1556,9 +1631,8 @@ site's URL matches its external URL. On the **primary** site: -1. On the left sidebar, select **Search or go to**. -1. Select **Admin Area**. -1. On the left sidebar, select **Geo > Sites**. +1. On the left sidebar, at the bottom, select **Admin Area**. +1. Select **Geo > Sites**. 1. Find the affected **secondary** site and select **Edit**. 1. Ensure the **URL** field matches the value found in `/etc/gitlab/gitlab.rb` in `external_url "https://gitlab.example.com"` on the **Rails nodes of the secondary** site. diff --git a/doc/administration/geo/replication/tuning.md b/doc/administration/geo/replication/tuning.md index 3a4366cfd63..c02571e21a8 100644 --- a/doc/administration/geo/replication/tuning.md +++ b/doc/administration/geo/replication/tuning.md @@ -1,8 +1,7 @@ --- stage: Systems group: Geo -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments -type: howto +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://handbook.gitlab.com/handbook/product/ux/technical-writing/#assignments --- # Tuning Geo **(PREMIUM SELF)** @@ -14,9 +13,8 @@ in the background. On the **primary** site: -1. On the left sidebar, select **Search or go to**. -1. Select **Admin Area**. -1. On the left sidebar, select **Geo > Sites**. +1. On the left sidebar, at the bottom, select **Admin Area**. +1. Select **Geo > Sites**. 1. Select **Edit** of the secondary site you want to tune. 1. Under **Tuning settings**, there are several variables that can be tuned to improve the performance of Geo: diff --git a/doc/administration/geo/replication/upgrading_the_geo_sites.md b/doc/administration/geo/replication/upgrading_the_geo_sites.md index 6f02ef29f99..aae35d45bde 100644 --- a/doc/administration/geo/replication/upgrading_the_geo_sites.md +++ b/doc/administration/geo/replication/upgrading_the_geo_sites.md @@ -1,8 +1,7 @@ --- stage: Systems group: Geo -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments -type: howto +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://handbook.gitlab.com/handbook/product/ux/technical-writing/#assignments --- # Upgrading the Geo sites **(PREMIUM SELF)** diff --git a/doc/administration/geo/replication/usage.md b/doc/administration/geo/replication/usage.md index cfaa02e1a17..0594709b23d 100644 --- a/doc/administration/geo/replication/usage.md +++ b/doc/administration/geo/replication/usage.md @@ -1,7 +1,7 @@ --- stage: Systems group: Geo -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://handbook.gitlab.com/handbook/product/ux/technical-writing/#assignments --- <!-- Please update EE::GitLab::GeoGitAccess::GEO_SERVER_DOCS_URL if this file is moved) --> @@ -41,5 +41,5 @@ Go modules can be pulled from secondary sites, with a number of limitations: - Git configuration (using `insteadOf`) is needed to fetch data from the Geo secondary site. - For private projects, authentication details need to be specified in `~/.netrc`. -Read more in the -[working with projects `go get` documentation](../../../user/project/working_with_projects.md#fetch-go-modules-from-geo-secondary-sites). +For more information, see +[Using a project as a Go package](../../../user/project/use_project_as_go_package.md#fetch-go-modules-from-geo-secondary-sites). diff --git a/doc/administration/geo/secondary_proxy/index.md b/doc/administration/geo/secondary_proxy/index.md index f6572ded4a9..22bb7dd721d 100644 --- a/doc/administration/geo/secondary_proxy/index.md +++ b/doc/administration/geo/secondary_proxy/index.md @@ -1,8 +1,7 @@ --- stage: Systems group: Geo -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments -type: howto +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://handbook.gitlab.com/handbook/product/ux/technical-writing/#assignments --- # Geo proxying for secondary sites **(PREMIUM SELF)** @@ -115,7 +114,7 @@ for details. - Non-Rails requests are not proxied, so other services may need to use a separate, non-unified URL to ensure requests are always sent to the primary. These services include: - - GitLab Container Registry - [can be configured to use a separate domain](../../packages/container_registry.md#configure-container-registry-under-its-own-domain). + - GitLab container registry - [can be configured to use a separate domain](../../packages/container_registry.md#configure-container-registry-under-its-own-domain). - GitLab Pages - should always use a separate domain, as part of [the prerequisites for running GitLab Pages](../../pages/index.md#prerequisites). - With a unified URL, Let's Encrypt can't generate certificates unless it can reach both IPs through the same domain. diff --git a/doc/administration/geo/secondary_proxy/location_aware_external_url.md b/doc/administration/geo/secondary_proxy/location_aware_external_url.md index ae701bb8482..4ff349d8741 100644 --- a/doc/administration/geo/secondary_proxy/location_aware_external_url.md +++ b/doc/administration/geo/secondary_proxy/location_aware_external_url.md @@ -1,8 +1,7 @@ --- stage: Systems group: Geo -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments -type: howto +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://handbook.gitlab.com/handbook/product/ux/technical-writing/#assignments --- # Location-aware public URL **(PREMIUM SELF)** diff --git a/doc/administration/geo/setup/database.md b/doc/administration/geo/setup/database.md index 471bae72c5b..13615825a14 100644 --- a/doc/administration/geo/setup/database.md +++ b/doc/administration/geo/setup/database.md @@ -1,8 +1,7 @@ --- stage: Systems group: Geo -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments -type: howto +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://handbook.gitlab.com/handbook/product/ux/technical-writing/#assignments --- # Geo database replication **(PREMIUM SELF)** @@ -763,7 +762,7 @@ defaults default-server inter 3s fall 3 rise 2 on-marked-down shutdown-sessions frontend internal-postgresql-tcp-in - bind *:5000 + bind *:5432 mode tcp option tcplog @@ -900,6 +899,10 @@ For each node running a Patroni instance on the secondary site: gitlab_rails['auto_migrate'] = false ``` + When configuring `patroni['standby_cluster']['host']` and `patroni['standby_cluster']['port']`: + - `INTERNAL_LOAD_BALANCER_PRIMARY_IP` must point to the primary internal load balancer IP. + - `INTERNAL_LOAD_BALANCER_PRIMARY_PORT` must point to the frontend port [configured for the primary Patroni cluster leader](#step-2-configure-the-internal-load-balancer-on-the-primary-site). **Do not** use the PgBouncer frontend port. + 1. Reconfigure GitLab for the changes to take effect. This step is required to bootstrap PostgreSQL users and settings. @@ -952,8 +955,7 @@ If you want to run the Geo tracking database on a single node, see [Configure th A production-ready and secure setup for the tracking PostgreSQL DB requires at least three Consul nodes: two Patroni nodes, and one PgBouncer node on the secondary site. -Because of [issue 6587](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/6587), Consul can't track multiple -services, so these must be different than the nodes used for the Standby Cluster database. +Consul can track multiple services, so you can choose to reuse the nodes used for the Standby Cluster database, though the instructions below do not show how to combine configurations when reusing Consul nodes. Be sure to use [password credentials](../../postgresql/replication_and_failover.md#database-authorization-for-patroni) and other database best practices. diff --git a/doc/administration/geo/setup/external_database.md b/doc/administration/geo/setup/external_database.md index 6083d7331b6..2d0e9e45197 100644 --- a/doc/administration/geo/setup/external_database.md +++ b/doc/administration/geo/setup/external_database.md @@ -1,7 +1,7 @@ --- stage: Systems group: Geo -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://handbook.gitlab.com/handbook/product/ux/technical-writing/#assignments --- # Geo with external PostgreSQL instances **(PREMIUM SELF)** diff --git a/doc/administration/geo/setup/index.md b/doc/administration/geo/setup/index.md index f59dec17f8b..1868902cfcf 100644 --- a/doc/administration/geo/setup/index.md +++ b/doc/administration/geo/setup/index.md @@ -1,8 +1,7 @@ --- stage: Systems group: Geo -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments -type: howto +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://handbook.gitlab.com/handbook/product/ux/technical-writing/#assignments --- # Setting up Geo **(PREMIUM SELF)** @@ -33,14 +32,14 @@ If both Geo sites are based on the [1K reference architecture](../../reference_a If using external PostgreSQL services, for example Amazon RDS, follow [Set up Geo for two single-node sites (with external PostgreSQL services)](two_single_node_external_services.md). -Depending on your GitLab deployment, [additional configuration](#additional-configuration) for LDAP, object storage, and the Container Registry might be required. +Depending on your GitLab deployment, [additional configuration](#additional-configuration) for LDAP, object storage, and the container registry might be required. ### Multi-node Geo sites If one or more of your sites is using the [2K reference architecture](../../reference_architectures/2k_users.md) or larger, see [Configure Geo for multiple nodes](../replication/multiple_servers.md). -Depending on your GitLab deployment, [additional configuration](#additional-configuration) for LDAP, object storage, and the Container Registry might be required. +Depending on your GitLab deployment, [additional configuration](#additional-configuration) for LDAP, object storage, and the container registry might be required. ### General steps for reference @@ -50,7 +49,7 @@ Depending on your GitLab deployment, [additional configuration](#additional-conf 1. [Configure GitLab](../replication/configuration.md) to set the **primary** and **secondary** sites. 1. Follow the [Using a Geo Site](../replication/usage.md) guide. -Depending on your GitLab deployment, [additional configuration](#additional-configuration) for LDAP, object storage, and the Container Registry might be required. +Depending on your GitLab deployment, [additional configuration](#additional-configuration) for LDAP, object storage, and the container registry might be required. ### Additional configuration @@ -59,7 +58,7 @@ Depending on how you use GitLab, the following configuration might be required: - If the **primary** site uses object storage, [configure object storage replication](../replication/object_storage.md) for the **secondary** sites. - If you use LDAP, [configure a secondary LDAP server](../../auth/ldap/index.md) for the **secondary** sites. For more information, see [LDAP with Geo](../replication/single_sign_on.md#ldap). -- If you use the Container Registry, [configure the Container Registry for replication](../replication/container_registry.md) on the **primary** and **secondary** sites. +- If you use the container registry, [configure the container registry for replication](../replication/container_registry.md) on the **primary** and **secondary** sites. You should [configure unified URLs](../secondary_proxy/index.md#set-up-a-unified-url-for-geo-sites) to use a single, unified URL for all Geo sites. diff --git a/doc/administration/geo/setup/two_single_node_external_services.md b/doc/administration/geo/setup/two_single_node_external_services.md index 405a791fedc..a77bacb0587 100644 --- a/doc/administration/geo/setup/two_single_node_external_services.md +++ b/doc/administration/geo/setup/two_single_node_external_services.md @@ -1,7 +1,7 @@ --- stage: Systems group: Geo -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://handbook.gitlab.com/handbook/product/ux/technical-writing/#assignments --- # Set up Geo for two single-node sites (with external PostgreSQL services) **(PREMIUM SELF)** @@ -38,7 +38,7 @@ Prerequisites: ```ruby ## ## The unique identifier for the Geo site. See - ## https://docs.gitlab.com/ee/user/admin_area/geo_nodes.html#common-settings + ## https://docs.gitlab.com/ee/administration/geo_sites.html#common-settings ## gitlab_rails['geo_node_name'] = '<site_name_here>' ``` @@ -292,7 +292,7 @@ secondary site is a read-only copy. ```ruby ## ## The unique identifier for the Geo site. See - ## https://docs.gitlab.com/ee/user/admin_area/geo_nodes.html#common-settings + ## https://docs.gitlab.com/ee/administration/geo_sites.html#common-settings ## gitlab_rails['geo_node_name'] = '<secondary_site_name_here>' ``` @@ -306,8 +306,7 @@ secondary site is a read-only copy. ``` 1. Go to the primary node GitLab instance: - 1. On the left sidebar, select **Search or go to**. - 1. Select **Admin Area**. + 1. On the left sidebar, at the bottom, select **Admin Area**.. 1. Select **Geo > Sites**. 1. Select **Add site**. @@ -362,8 +361,7 @@ If you convert an existing site to Geo, you should check that the clone method i On the primary site: -1. On the left sidebar, select **Search or go to**. -1. Select **Admin Area**. +1. On the left sidebar, at the bottom, select **Admin Area**. 1. Select **Settings > General**. 1. Expand **Visibility and access controls**. 1. If you use Git over SSH: @@ -378,8 +376,7 @@ the primary site. After you sign in: -1. On the left sidebar, select **Search or go to**. -1. Select **Admin Area**. +1. On the left sidebar, at the bottom, select **Admin Area**. 1. Select **Geo > Sites**. 1. Verify that the site is correctly identified as a secondary Geo site, and that Geo is enabled. diff --git a/doc/administration/geo/setup/two_single_node_sites.md b/doc/administration/geo/setup/two_single_node_sites.md index da7ec455c6a..edcb2f467ff 100644 --- a/doc/administration/geo/setup/two_single_node_sites.md +++ b/doc/administration/geo/setup/two_single_node_sites.md @@ -1,7 +1,7 @@ --- stage: Systems group: Geo -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://handbook.gitlab.com/handbook/product/ux/technical-writing/#assignments --- # Set up Geo for two single-node sites **(PREMIUM SELF)** @@ -38,7 +38,7 @@ Prerequisites: ```ruby ## ## The unique identifier for the Geo site. See - ## https://docs.gitlab.com/ee/user/admin_area/geo_nodes.html#common-settings + ## https://docs.gitlab.com/ee/administration/geo_sites.html#common-settings ## gitlab_rails['geo_node_name'] = '<site_name_here>' ``` @@ -538,7 +538,7 @@ You must manually replicate the secret file across all of your secondary sites, ```ruby ## ## The unique identifier for the Geo site. See - ## https://docs.gitlab.com/ee/user/admin_area/geo_nodes.html#common-settings + ## https://docs.gitlab.com/ee/administration/geo_sites.html#common-settings ## gitlab_rails['geo_node_name'] = '<site_name_here>' ``` @@ -552,8 +552,7 @@ You must manually replicate the secret file across all of your secondary sites, ``` 1. Navigate to the primary node GitLab instance: - 1. On the left sidebar, select **Search or go to**. - 1. Select **Admin Area**. + 1. On the left sidebar, at the bottom, select **Admin Area**. 1. Select **Geo > Sites**. 1. Select **Add site**. @@ -608,8 +607,7 @@ If you convert an existing site to Geo, you should check that the clone method i On the primary site: -1. On the left sidebar, select **Search or go to**. -1. Select **Admin Area**. +1. On the left sidebar, at the bottom, select **Admin Area**. 1. Select **Settings > General**. 1. Expand **Visibility and access controls**. 1. If you use Git over SSH: @@ -624,8 +622,7 @@ the primary site. After you sign in: -1. On the left sidebar, select **Search or go to**. -1. Select **Admin Area**. +1. On the left sidebar, at the bottom, select **Admin Area**. 1. Select **Geo > Sites**. 1. Verify that the site is correctly identified as a secondary Geo site, and that Geo is enabled. |